TITLE: Bayesian Averaging is Well-Temperated
AUTHORS: Lars Kai Hansen
Department of Mathematical Modelling, Building 321
Technical University of Denmark, DK-2800 Lyngby, Denmark
email: lkhansen@imm.dtu.dk
www: http://eivind.imm.dtu.dk
ABSTRACT:
Bayesian predictions are stochastic just like predictions
of any other inference scheme that generalize from
a finite sample. While a simple variational argument shows
that Bayes averaging is generalization optimal
given that the prior matches the teacher parameter distribution
the situation is less clear if the teacher distribution is unknown.
I define a class of averaging procedures, the temperated likelihoods,
including both Bayes averaging with a uniform prior and maximum likelihood
estimation as special cases. I show that Bayes is generalization optimal
in this family for any teacher distribution
for two learning problems that are analytically tractable:
learning the mean of a Gaussian and asymptotics of smooth learners.
Submitted for NIPS*99, December 1999, Denver, USA