TITLE: Behaviour in 0 of the neural networks training cost

AUTHOR: Cyril Goutte
Department of Mathematical Modelling,
Technical University of Denmark, Lyngby, Denmark


We study the behaviour in zero of the derivatives of the cost function used when training non-linear neural networks. It is shown that a fair number of first, second and higher order derivatives vanish in zero, validating the belief that 0 is a peculiar and potentially harmful location. These calculations are related to practical and theoretical aspects of neural networks training.

Key words: training cost derivatives, neural networks training, numerical optimisation, regularisation

Preprint, Neural Processing Letters, 8:2, pp. 107-116 (Kluwer Academic Publishers).
Download: Postscript (from IMM) or pdf directly from Kluwer.