Title:   Training Recurrent Networks

Authors: Morten With Pedersen 
         CONNECT, Department of Mathematical Modelling, Building 321
         Technical University of Denmark
         DK-2800 Lyngby, Denmark
         Phones: + 45 45253920
         Fax: + 45 45872599
         email: mwp@imm.dtu.dk

Abstract:
Training recurrent networks is generally believed to be a difficult task.
Excessive training times and lack of convergence to an acceptable solution
are frequently reported. In this paper we seek to explain the reason for this 
from a numerical point of view and show how to avoid problems when training. 
In particular we investigate ill-conditioning, the need for and effect of 
regularization and illustrate the superiority of second-order methods for 
training. 


Extended summary version, submnitted for NNSP'97, Florida, September, 1997.