TITLE: Behaviour in 0 of the neural networks training cost
AUTHOR: Cyril Goutte
Department of Mathematical Modelling,
Technical University of Denmark, Lyngby, Denmark
cg@imm.dtu.dk
http://eivind.imm.dtu.dk
ABSTRACT:
We study the behaviour in zero of the derivatives of the
cost function used when training non-linear neural networks. It is
shown that a fair number of first, second and higher order derivatives
vanish in zero, validating the belief that 0 is a peculiar and
potentially harmful location. These calculations are related to
practical and theoretical aspects of neural networks training.
Key words: training cost derivatives, neural networks training,
numerical optimisation, regularisation
Preprint,
Neural Processing Letters, 8:2, pp. 107-116 (Kluwer Academic Publishers).
Download:
Postscript
(from IMM) or pdf
directly from Kluwer.