TITLE: Pruning from Adaptive Regularization
AUTHORS: Lars Kai Hansen & Carl Edward Rasmussen
ABSTRACT:
Inspired by the recent upsurge of interest in Bayesian methods
we consider adaptive regularization. A generalization based scheme for
adaptation of regularization parameters is introduced and compared to Bayesian
regularization. We show that pruning arises naturally within both adaptive
regularization schemes. As model example we have chosen the simplest possible:
estimating the mean of a random variable with known variance. Marked
similarities are found between the two methods in that they both involve a
``noise limit'', below which they regularize with infinite weight decay, \ie they
{\it prune}. However, pruning is not always beneficial. We show explicitly that
both methods in some cases may increase the generalization error. This
corresponds to situations where the underlying assumptions of the regularizer are
poorly matched to the environment.