TITLE: GENERALIZATION PERFORMANCE OF REGULARIZED NEURAL NETWORK MODELS

AUTHORS: Jan Larsen  Lars Kai Hansen

ABSTRACT:
Architecture optimization is a fundamental problem of neural network
modeling. The optimal architecture is defined as the one which minimizes the
generalization error. This paper addresses estimation of the
generalization
performance of regularized, complete neural network models. Regularization
normally
improves the generalization performance by restricting the model complexity.
A formula for the optimal weight decay regularizer is derived.
A regularized model may be characterized by an effective number of weights
(parameters); however, it is demonstrated that no simple definition is possible.
A novel estimator of the average generalization error (called \ems{FPER}) is
suggested and compared
to the Final Prediction Error (\ems{FPE}) and Generalized Prediction Error (\ems{GPE})
estimators. In addition, comparative numerical studies demonstrate the
qualities of the suggested estimator.


In J. Vlontzos, J.-N. Whang & E. Wilson (eds.): Proceedings of the 4th IEEE 
Workshop on Neural Networks for Signal Processing, IEEE Press, 1994, pp. 42-51.