PRML Slides 3

PATTERN RECOGNITION
AND MACHINE LEARNING
CHAPTER 3: LINEAR MODELS FOR REGRESSION

Linear Basis Function Models (1)
Example: Polynomial Curve Fitting
Generally
where j(x) are known as basis functions.

Typically, 0(x) = 1, so that w0 acts as a bias.
In the simplest case, we use linear basis
functions : d(x) = xd.
Polynomial basis functions:
These are global; a small

change in x affect all basis
functions.
Gaussian basis functions:
These are local; a small change

in x only affect nearby basis
functions. j and s control
location and scale (width).
Sigmoidal basis functions:
where
Also these are local; a small

change in x only affect nearby
basis functions. j and s
control location and scale
(slope).
Maximum Likelihood and Least Squares (1)
Assume observations from a deterministic function

with added Gaussian noise:
where
which is the same as saying,
Given observed inputs, , and targets,

, we obtain the likelihood function
Taking the logarithm, we get
where
is the sum-of-squares error.

Computing the gradient and setting it to zero yields
Solving for w, we get The Moore-Penrose

pseudo-inverse, .
where
Geometry of Least Squares
Consider
N-dimensional
M-dimensional
S is spanned by .
wML minimizes the distance
between t and its orthogonal
projection on S, i.e. y.
Sequential Learning
Data items considered one at a time (a.k.a.
online learning); use stochastic (sequential)
gradient descent:
This is known as the least-mean-squares (LMS)

algorithm. Issue: how to choose ?
Regularized Least Squares (1)
Consider the error function:
Data term + Regularization term
With the sum-of-squares error function and a

quadratic regularizer, we get
is called the
regularization
which is minimized by coefficient.
With a more general regularizer, we have
Lasso Quadratic
Lasso tends to generate sparser solutions than a
quadratic
regularizer.
Multiple Outputs (1)
Analogously to the single output case we have:
Given observed inputs, , and targets,

, we obtain the log likelihood function
Multiple Outputs (2)
Maximizing with respect to W, we obtain
If we consider a single target variable, tk, we see that
where , which is identical with the

single output case.
The Bias-Variance Decomposition (1)
Recall the expected squared loss,
where
The second term of E[L] corresponds to the noise

inherent in the random variable t.
What about the first term?
Suppose we were given multiple data sets, each of
size N. Any particular data set, D, will give a
particular function y(x;D). We then have
Taking the expectation over D yields
Thus we can write
where
Example: 25 data sets from the sinusoidal, varying
the degree of regularization, .
The Bias-Variance Trade-off
From these plots, we note
that an over-regularized
model (large ) will have a
high bias, while an under-
regularized model (small )
will have a high variance.
Bayesian Linear Regression (1)
Define a conjugate prior over w
Combining this with the likelihood function and using

results for marginal and conditional Gaussian
distributions, gives the posterior
where
A common choice for the prior is
for which
Next we consider an example

0 data points observed
Prior Data Space

1 data point observed
Likelihood Posterior Data Space



Predictive Distribution (1)
Predict t for new values of x by integrating
over w:
where
Example: Sinusoidal data, 9 Gaussian basis functions,
1 data point
2 data points
4 data points
25 data points
Equivalent Kernel (1)
The predictive mean can be written
Equivalent kernel or
smoother matrix.
This is a weighted sum of the training data

target values, tn.
Weight of tn depends on distance between x and xn;

nearby xn carry more weight.
Non-local basis functions have local equivalent
kernels:
Polynomial Sigmoidal
The kernel as a covariance function: consider
We can avoid the use of basis functions and

define the kernel function directly, leading
to Gaussian Processes (Chapter 6).
for all values of x; however, the equivalent kernel

may be negative for some values of x.
Like all kernel functions, the equivalent kernel can be
expressed as an inner product:
where .
Bayesian Model Comparison (1)
How do we choose the right model?
Assume we want to compare models Mi, i=1, ,L,
using data D; this requires computing
Posterior Prior Model evidence or

marginal likelihood
Bayes Factor: ratio of evidence for two models

Having computed p(MijD), we can compute
the predictive (mixture) distribution
A simpler approximation, known as model

selection, is to use the model with the
highest evidence.
For a model with parameters w, we get the
model evidence by marginalizing over w
Note that
For a given model with a
single parameter, w, con-
sider the approximation
where the posterior is

assumed to be sharply
peaked.
Taking logarithms, we obtain
Negative
With M parameters, all assumed to have the same

ratio , we get
Negative and linear in M.

Matching data and model complexity
The Evidence Approximation (1)
The fully Bayesian predictive distribution is given by
but this integral is intractable. Approximate with
where is the mode of , which is assumed to

be sharply peaked; a.k.a. empirical Bayes, type II or gene-
ralized maximum likelihood, or evidence approximation.
From Bayes theorem we have
and if we assume p(,) to be flat we see that
General results for Gaussian integrals give

Example: sinusoidal data, M th degree polynomial,
Maximizing the Evidence Function (1)
To maximise w.r.t. and , we define the
eigenvector equation
Thus
has eigenvalues i + .
Maximizing the Evidence Function (2)
We can now differentiate w.r.t. and ,
and set the results to zero, to get
where
N.B. depends on both and .

Effective Number of Parameters (3)
w1 is not well
Likelihood determined by the
likelihood
w2 is well determined
by the likelihood
Prior is the number of well

determined parameters
Example: sinusoidal data, 9 Gaussian basis functions,
= 11.1.
= 11.1.
Test set error

= 11.1.
In the limit , = M and we can consider
using the easy-to-compute approximation
Limitations of Fixed Basis Functions
M basis function along each dimension of a
D-dimensional input space requires MD
basis functions: the curse of dimensionality.
In later chapters, we shall see how we can
get away with fewer basis functions, by
choosing these using the training data.

PRML Slides 3

Uploaded by

Copyright:

Available Formats

PRML Slides 3

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

PRML Slides 3

Uploaded by

Copyright:

Available Formats

PATTERN RECOGNITION

AND MACHINE LEARNING

CHAPTER 3: LINEAR MODELS FOR REGRESSION

where j(x) are known as basis functions.

These are global; a small

These are local; a small change

Also these are local; a small

Assume observations from a deterministic function

which is the same as saying,

Given observed inputs, , and targets,

Taking the logarithm, we get

is the sum-of-squares error.

Computing the gradient and setting it to zero yields

Solving for w, we get The Moore-Penrose

This is known as the least-mean-squares (LMS)

Data term + Regularization term

With the sum-of-squares error function and a

Given observed inputs, , and targets,

If we consider a single target variable, tk, we see that

where , which is identical with the

The second term of E[L] corresponds to the noise

Combining this with the likelihood function and using

Next we consider an example

Prior Data Space

Likelihood Posterior Data Space

Likelihood Posterior Data Space

Likelihood Posterior Data Space

This is a weighted sum of the training data

Weight of tn depends on distance between x and xn;

We can avoid the use of basis functions and

for all values of x; however, the equivalent kernel

Posterior Prior Model evidence or

Bayes Factor: ratio of evidence for two models

A simpler approximation, known as model

where the posterior is

With M parameters, all assumed to have the same

Negative and linear in M.

but this integral is intractable. Approximate with

where is the mode of , which is assumed to

and if we assume p(,) to be flat we see that

General results for Gaussian integrals give

N.B. depends on both and .

Prior is the number of well

Test set error

You might also like

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.