TITLE: Design and Regularization of Neural Networks: The Optimal Use
       of A Validation Set

AUTHORS: J. Larsen, L.K. Hansen, C. Svarer* and M. Ohlsson
         CONNECT Department of Mathematical Modelling, Building 349
         Technical University of Denmark
         DK-2800 Lyngby, Denmark
         emails: jl@imm.dtu.dk, lkhansen@ei.dtu.dk, mo@imm.dtu.dk

         *Department of Neurology
         National University Hospital
         DK-2100 Copenhagen O, Denmark
         email: csvarer@pet.rh.dk

ABSTRACT:
In this paper we derive novel algorithms for estimation of regularization
parameters and for optimization of neural net architectures based on a
validation set. Regularization parameters are estimated using an iterative
gradient descent scheme. Architecture optimization is performed by approximative
combinatorial search among the relevant subsets of an initial neural network
architecture by employing a validation set based Optimal Brain Damage/Surgeon
(OBD/OBS) or a mean field combinatorial optimization approach.
Numerical results with linear models and feed-forward neural networks
demonstrate the viability of the methods.

To appear in Proceedings of the IEEE Workshop on Neural Networks for Signal
Processing VI, Piscataway, New Jersey: IEEE, 1996.