TITLE: On Comparison of Adaptive Regularization Methods

AUTHORS: Sigurdur Sigurdsson, Lars Kai Hansen and Jan Larsen
Department of Mathematical Modelling, Building 321
Technical University of Denmark, DK-2800 Lyngby, Denmark
emails: siggi,lkhansen,jl@imm.dtu.dk
www: http://eivind.imm.dtu.dk


Modeling with flexible models, such as neural networks, requires careful control of the model complexity and generalization ability of the resulting model which finds expression in the ubiquitous bias-variance dilemma.

Regularization is a tool for optimizing the model structure reducing variance at the expense of introducing extra bias. The overall objective of adaptive regularization is to tune the amount of regularization ensuring minimal generalization error.

This paper investigates recently suggested adaptive regularization schemes. Some methods focus directly on minimizing an estimate of the generalization error (either algebraic or empirical), whereas others starts from different criteria, e.g., the Bayesian evidence.

We suggest various algorithm extensions and performed numerical experiments with linear models.

Appears in proc. of NNSP2000, Sydney, Australia, Dec. 11-13, 2000.