Advanced methods for non-parametric modelling

Carl Edward Rasmussen
Cyril Goutte
Roderick Murray-Smith


IMM, Section for digital signal processing

Preliminary program

In this course we will discuss various topics for non-linear, non-parametric modelling. We will discuss the fundamental issues involved in modelling and describe some prominent methods including both non-Bayesian and Bayesian approaches to neural networks. The choice of topics is not exhaustive, but rather governed by our experience and personal inclinations.

Place: The lectures will take place in building 305, room 205 (2nd floor). The assignments will be performed in the terminal-room in the basement of building 305 using the sections' linux machnine.

Time: The course will take place in week 3 (january 18-22). It will contain 11 lectures given in English, 1,5 hours each, and computer exercises for 3 to 4 hours in the afternoons. The course is based on test-book material as well as discussion of recent research papers.
Check also the administration page for the course.

Basic readings: These references are given as indication of the material covered last year. They will change in the coming weeks. This is meant as a list of work you can refer to after the course if you want to get more information about a given topic.

The datasets, Matlab functions and assignments are available on a separate webpage.

The topics covered will be: We will focus our attention on the following non-parametric models:
Monday 18 9:00-10:20
Course introduction
Numerical methods for model fitting
Introduction to Neural Networks
Generalisation and regularisation
Tuesday 19 9:00-13.00
Assignment 1: optimisation and NN
Kernel methods
Introduction to Radial Basis Function networks & Mixture models
Wednesday 20 9:00-13:00
Assignment 2: Kernel and RBF
Bayesian inference and Markov Chain Monte Carlo
Bayesian training of Neural networks
Thursday 21 9:00-13:00
Assignment 3: MCMC and NN
Gaussian Processes
Bayesian training of Gaussian Processes
Friday 22 9:00-13:00
Assignment 4: Gaussian Processes
Local models, effective degrees of freedom & equivalent kernels
Infinite Gaussian mixture models

Monday 18, 9.00- 9.30: Course introduction

Monday 18, 9.40-12.00: Numerical methods for model fitting

One-dimensional optimisation and line search, Multi-dimensional optimisation, steepest descent, Newton and quasi-Newton, conjugate gradient.

Monday 18, 14.00-15.20: Introduction to Neural Networks

The multi-layer perceptron (MLP), training of MLP, regularisation techniques, parameter selection (pruning).

Monday 18, 15.30-17.00: Generalisation and Regularisation

Well-posed and ill-posed problems, regularisation, example on linear models
Risk minimisation, generalisation, generalisation bounds, generalisation estimators, resampling.

Tuesday 19, 9.00-13.00: Assignment 1: Optimisation and NN (in Matlab)

Comparison of optimisation methods on a simple problem.
Training a neural network.
Experiments on over-fitting.

Tuesday 19, 14.00-15.20: Kernel methods

Kernel density estimation, kernel regression, local smoothing, bandwidth estimation, multivariate regression, variable metric.

Tuesday 19, 15.30-17.00: Introduction to Radial Basis Function networks & Mixture models

Multiple component approaches, Basis function models, Mixture models, Expectation Maximisation (EM)

Wednesday 20, 9.00-13.00: Assignment 2: Kernel and RBF (in Matlab)

Multivariate kernel regression.
Mixture of Gaussians for probability density function estimation.
Mixture of linear models for regression
RBF net for regression.

Wednesday 20, 14.00-15.20: Bayesian inference and MCMC

Wednesday 20, 15.30-17.00: Bayesian training of Neural networks

Thursday 21, 9.00-13.00: Assignment 3: MCMC and NN

Bayesian training of neural networks using Radford Neal's flexible Bayesian modelling software.

Thursday 21, 14.00-15.20: Gaussian Processes

Thursday 21, 15.30-17.00: Bayesian training of Gaussian Processes

Friday 22, 9.00-13.00: Assignment 4: Gaussian Processes (in Matlab)

MAP training of Gaussian Processes.
Bayesian trianing of GP using Radford Neal's fbm software.

Friday 22, 14.00-15.20: Local models, effective degrees of freedom & equivalent kernels

Mixture models where each mixture has a local linear function instead of a constant weight.
Effective degrees of freedom.
Equivalent kernel interpretation of linear-in-the-parameters identification of basis function models.

Friday 22, 15.30-17.00: Infinite Gaussian mixture models

If you access this page through frames, you can click or copy the following link in your browser

Last modified October 29, 1998
Write to the DSP, IMM webmaster at
© Copyright 1998 by Section for DSP, IMM.