Information Geometry of Multilayer Neural Networks

Neural networks are adaptive and tractable nonlinear models for processing signals, and are widely used in applications. Learning takes place in the parameter space of a neural network. The parameter space is, however, not Euclidean but strongly curved, so that we need a geometrical view to understand behaviors of learning and to obtain reliable learning algorithms.

Information geometry provides a powerful tool, as well as intuitive insight, for this purpose. The present talk summarizes the geometrical structure of manifolds of neural networks and gives a new learning algorithm named the natural gradient learning method. We further study the geometrical singularities existing in the parameter space, which is called a neuromanifold. They cause serious slowing-down in learning, which is called plateaus in learning. We analyze the geometrical structures of singularities, and show why, and how, the new method works well when singularities exist.

The present talk gives a unified overview connecting neural networks, statistical inference, and information-geometry.