SDV

Nonparametric Estimation: Smoothing and
Visualization
Ronaldo Dias
Departamento de Estatstica - IMECC.
Universidade Estadual de Campinas.
Sao Paulo, Brasil.
E-mail address: dias@ime.unicamp.br
Preface
In recent years more and more data have been collected in order to extract information or to learn valuable characteristics about experiments, phenomena, observational facts, etc.. This is what its been called learning from data. Due to their complexity, several datasets have been analyzed by nonparametric approaches. This field
of Statistics impose minimum assumptions to get useful information from data. In
fact, nonparametric procedures, usually, let the data speak for themselves. This work
is a brief introduction to a few of the most useful procedures in the nonparametric estimation toward smoothing and data visualization. In particular, it describes the theory and the applications of nonparametric curve estimation (density and regression)
problems with emphasis in kernel, nearest neighbor, orthogonal series, smoothing
splines methods. The text is designed for undergraduate students in mathematical
sciences, engineering and economics. It requires at least one semester in calculus,
probability and mathematical statistics.
ii
Contents
1 Introduction
2 Kernel estimation
2.1
The Histogram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2
Kernel Density Estimation . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.1
The Nearest Neighbor Method . . . . . . . . . . . . . . . . . . .
10
2.3
Some Statistical Results for Kernel Density Estimation . . . . . . . . . .
12
2.4
Bandwidth Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
2.4.1
Reference to a Standard Distribution . . . . . . . . . . . . . . . .
14
2.4.2
Maximum likelihood Cross-Validation . . . . . . . . . . . . . . .
15
2.4.3
Least-Squares Cross-Validation . . . . . . . . . . . . . . . . . . .
18
Orthogonal series estimators . . . . . . . . . . . . . . . . . . . . . . . . .
19
2.5
3 Kernel nonparametric Regression Method
23
3.1
k-Nearest Neighbor (k-NN) . . . . . . . . . . . . . . . . . . . . . . . . .
25
3.2
Local Polynomial Regression: LOWESS . . . . . . . . . . . . . . . . . .
26
3.3
Penalized Maximum Likelihood Estimation . . . . . . . . . . . . . . . .
28
3.3.1
32
Computing Penalized Log-Likelihood Density Estimates . . . .
4 Spline Functions
35
4.1
Acquiring the Taste . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
4.2
Logspline Density Estimation . . . . . . . . . . . . . . . . . . . . . . . .
39
4.3
Splines Density Estimation: A Dimensionless Approach . . . . . . . . .
41
iii
CONTENTS
iv
5
The thin-plate spline on Rd
45
5.1
Additive Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
48
5.2
Generalized Cross-Validation Method for Splines nonparametric Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
50
Regression splines, P-splines and H-splines
53
6.1
Sequentially Adaptive H-splines . . . . . . . . . . . . . . . . . . . . . .
53
6.2
P-splines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
61
6.3
A Bayesian Approach to H-splines
62
Final Comments
. . . . . . . . . . . . . . . . . . . .
69
List of Figures
2.2.1 Naive estimate constructed from Old faithful geyser data with h = 0.1
2.2.2 Kernel density estimate constructed from Old faithful geyser data with
Gaussian kernel and h = 0.25 . . . . . . . . . . . . . . . . . . . . . . . .
2.2.3 Bandwidth effect on kernel density estimates. The data set income was
rescaled to have mean 1. . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.4 Effect of the smoothing parameter k on the estimates . . . . . . . . . .
11
2.4.5 Comparison of two bandwidths, (the sample standard deviation)

and R (the sample interquartile) for the mixture 0.7 N (2, 1) + 0.3
N (1, 1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
2.5.6 Effect of the smoothing parameter K on the orthogonal series method

for density estimation
. . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
3.0.1 Effect of bandwiths on Nadaraya-Watson kernel . . . . . . . . . . . . .
24
3.1.2 Effect of the smoothing parameter k on the k-NN regression estimates.
26
3.2.3 Effect of the smoothing parameter using LOWESS method. . . . . . . .
27
4.1.1 Basis Functions with 6 knots placed at x . . . . . . . . . . . . . . . . .
37
4.1.2 Basis Functions with 6 knots placed at x . . . . . . . . . . . . . . . . .
38
4.2.3 logspline density estimation for .5*N(0,1)+.5*N(5,1) . . . . . . . . . . .
41
4.3.4 Histogram, SSDE, Kernel and Logspline density estimates . . . . . . .
43
5.1.1 True, tensor product, gam non-adaptive and gam adaptive surfaces . .
50
5.2.2 Smoothing spline fitting with smoothing parameter obtained by GCV

method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
v
52
vi
LIST OF FIGURES
6.1.1 Spline least square fittings for different values of K . . . . . . . . . . . .
54
6.1.2 Five thousand replicates of y( x ) = exp( x ) sin (x/2) cos (x ) + . . .
56
6.1.3 Five thousand replicates of the affinity and the partial affinity for adaptive nonparametric regression using H-splines with the true curve. . .
58
6.1.4 Density estimates of the affinity based on five thousand replicates of

the curve yi = xi3 + i with i N (0, .5). Solid line is a density estimate
using beta model and dotted line is a nonparametric density estimate.
59
6.1.5 A comparison between smoothing splines (S-splines) and hybrid splines

(H-splines) methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
60
6.2.6 smooth spline and P-spline . . . . . . . . . . . . . . . . . . . . . . . . .
62
6.3.7 Estimation results: a) Bayesian estimate with a = 17 and (K ) = K3

(dotted line); b) (SS) smoothing splines estimate (dashed line). The
true regression function is also plotted (solid line). The SS estimate was
computed using the R function smooth.spline from which 4 degrees
of freedom were obtained and was computed by GCV. . . . . . . . .
67
6.3.8 One hundred estimates of the curve 6.3.7 and a Bayesian confidence
interval for the regression curve g(t) = exp(t2 /2) cos(4t) with t
[0, ]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
68
Chapter 1
Introduction
Probably, the most used procedure to describe a possible relationship among variables is the statistical technique known as regression analysis. It is always useful to
begin the study of regression analysis by making use of simple models. For this, assume that we have collected observations from a continuous variable Y at n values of
a predict variable T. Let (t j , y j ) be such that:
y j = g (t j ) + j ,
j = 1, . . . , n,
(1.0.1)
where the random variables j are uncorrelated with mean zero and variance 2 .
Moreover, g(t j ) are the values obtained from some unknown function g computed at
the points t1 , . . . , tn . In general, the function g is called regression function or regression
curve.
A parametric regression model assumes that the form of g is known up to a finite
number of parameters. That is, we can write a parametric regression model by,
y j = g (t j , ) + j ,
j = 1, . . . , n,
(1.0.2)
where = ( 1 , . . . , p )T R p . Thus, to determine from the data a curve g is equivalent to determine the vector of parameters . One may notice that, if g has a linear
p
form, i.e., g(t, ) = j=1 j x j (t), where { x j (t)} j=1 are the explanatory variables, e.g.,
as in polynomial regression x j (t) = t j1 , then we are dealing with a linear parametric
regression model.
1
CHAPTER 1. INTRODUCTION
Certainly, there are other methods of fitting curves to data. A collection of tech-
niques known as nonparametric regression, for example, allows great flexibility in

the possible form of the regression curve. In particular, assume no parametric form
for g. In fact, a nonparametric regression model makes the assumption that the regression curve belongs to some infinite collection of curves. For example, g can be
in the class of functions that are differentiable with square integrable second derivatives, etc. Consequently, in order to propose a nonparametric model one may just
need to choose an appropriate space of functions where he/she believes that the regression curve lies. This choice, usually, is motivated by the degree of the smoothness
of g. Then, one uses the data to determine an element of this function space that can
represent the unknown regression curve. Consequently, nonparametric techniques
rely more heavily on the data for information about g than their parametric counterparts. Unfortunately, nonparametric estimators have some disadvantages. In general, they are less efficient than the parametric estimators when the parametric model
is correctly specified. For most parametric estimators the risk will decay to zero at a
rate of n1 while nonparametric estimators decay at a rate of n , where the parameter (0, 1) depends on the smoothness of g. For example, when g is twice
differentiable the rate is usually, n4/5 . However, in the case where the parametric
model is incorrectly specified, ad hoc, the rate n1 cannot be achieved. In fact, the
parametric estimator does not even converge to the true regression curve.
Chapter 2
Kernel estimation
Suppose we have n independent measurements {(ti , yi )}in=1 , the regression equation is, in general, described as in (1.0.1). Note that the regression curve g is the
conditional expectation of the independent variable Y given the predict variable T,
that is, g(t) = E[Y | T = t]. When we try to approximate the mean response function
g, we concentrate on the average dependence of Y on T = t. This means that we try
to estimate the conditional mean curve
g(t) = E[Y | T = t] =
f TY (t, y)
dy,
f T (t)
(2.0.1)
where f TY (t, y) denotes the joint density of (T, Y ) and f T (t) the marginal density of
T. In order to provide an estimate g (t) of g we need to obtain estimates of f TY (t, y)
and f T (t). Consequently, density estimation methodologies will be described.
3
CHAPTER 2. KERNEL ESTIMATION
2.1
The Histogram
The histogram is one of the first, and one of the most common, methods of density estimation. It is important to bear in mind that the histogram is a smoothing
technique used to estimate the unknown density and hence it deserves some consideration.
Let us try to combine the data by counting how many data points fall into a small
interval of length h. This kind of interval is called a bin. Observe that the well known
dot plot of Box, Hunter and Hunter (1978) is a particular type of histogram where
h = 0.
Without loss of generality, we consider a bin centered at 0, namely the interval
[h/2, h/2) and let FX be the distribution function of X such that FX is absolutely
continuous with respect to a Lesbegue measure on R. Consequently the probability
that an observation of X will fall into the interval [h/2, h/2) is given by:
P(X [h/2, h/2)) =
Z h/2
h/2
f X ( x )dx,
where f X is the density of X.

A natural estimate of this probability is the relative frequency of the observations
in this interval, that is, we count the number of observations falling into the interval
and divide it by the total number of observations. In other words, given the data
X1 , . . . , Xn , we have:
P(X [h/2, h/2))
1
#{ Xi [h/2, h/2)}.
n
Now applying the mean value theorem for continuous bounded function we obtain,
P(X [h/2, h/2)) =
Z h/2
h/2
f ( x )dx = f ( )h,
with [h/2, h/2). Thus, we arrive at the following density estimate:

1
fh ( x ) =
#{ Xi [h/2, h/2)},
nh
for all x [h/2, h/2).
2.1. THE HISTOGRAM
Formally, suppose we observe random variables X1 , . . . , Xn whose unknown common density is f . Let k be the number of bins, and define Cj = [ x0 + ( j 1)h, x0 + jh),
j = 1, . . . , k. Now, take n j = in=1 I (Xi Cj ), where the function I ( x A) is defined
to be :
1 if x A
I ( x A) =
0 otherwise,
and, kj=1 n j = n. Then,
1
fh ( x ) =
nh
n j I ( x C j ),
j =1
for all x. Here, note that the density estimate fh depends upon the histogram bandwidth
h. By varying h we can have different shapes of fh . For example, if one increases h,
one is averaging over more data and the histogram appears to be smoother. When
h 0, the histogram becomes a very noisy representation of the data (needle-plot,
Hardle (1990)). The opposite, situation when h , the histogram, now, becomes
overly smooth (box-shaped). Thus, h is the smoothing parameter of this type of density estimate, and the question of how to choose the histogram bandwidth h turns out
to be an important question in representing the data via the histogram. For details on
how to estimate h see Hardle (1990).
2.2 Kernel Density Estimation
The motivation behind the histogram can be expanded quite naturally. For this consider a weight function,
1 , if | x |< 1
2
K(x) =
0, otherwise
and define the estimator,
1
f( x ) =
nh
K(
i =1
x Xi
).
h
(2.2.1)
We can see that f extends the idea of the histogram. Notice that this estimate just
places a box of side (width) 2h and height (2nh)1 on each observation and then
sums to obtain f. See Silverman (1986) for a discussion of this kind of estimator. It is
not difficult to verify that f is not a continuous function and has zero derivatives everywhere except on the jump points Xi h. Besides having the undesirable character
of nonsmoothness (Silverman (1986)), it could give a misleading impression to a untrained observer since its somewhat ragged character might suggest several different
bumps.
Figure 2.2.1 shows the nonsmooth character of the naive estimate. The data seem
to have two major modes. However, the naive estimator suggests several different
small bumps.
2.2. KERNEL DENSITY ESTIMATION
0.3
0.0
0.1
0.2
density estimate
0.4
0.5
0.6
x xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx x xx
x xx
xxxxxx xx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxx
Eruptions length
Figure 2.2.1: Naive estimate constructed from Old faithful geyser data with h = 0.1
To overcome some of these difficulties, assumptions have been introduced on the
function K. That is, K must be a nonnegative kernel function that satisfies the following property:
Z
K ( x )dx = 1.
In other words K ( x ) is a probability density function, as for instance, the Gaussian

density, it will follow from definition that f will itself be a probability density. In addition, f will inherit all the continuity and differentiability properties of the kernel K.
For example, if K is a Gaussian density then f will be a smooth curve with derivatives
of all orders.
Figure 2.2.2 exhibits the smooth properties of f when a Gaussian kernel is used.
0.3
0.1
0.2
density estimate
0.4
0.5
0.0
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xx x xx x
xxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Eruptions length
Figure 2.2.2: Kernel density estimate constructed from Old faithful geyser data with
Gaussian kernel and h = 0.25
Note that an estimate based on the kernel function places bumps on the observations and the shape of those bumps is determined by the kernel function K. The
bandwidth h sets the width around each observation and this bandwidth controls the
degree of smoothness of a density estimate. It is possible to verify that as h 0, the
estimate becomes a sum of Dirac delta functions at the observations while as h ,
it eliminates all the local roughness and possibly important details are missed.
The data for the Figure 2.2.3 which is labelled income were provided by Charles
Kooperberg. This data set consists of 7125 random samples of yearly net income in
the United Kingdom (Family Expenditure Survey, 1968-1983). The income data is
considerably large and so it is more of a challenge to computing resources and there
are severe outliers. The peak at 0.24 is due to the UK old age pension, which caused
0.8
Histogram of income data
0.4
0.0
0.2
Relative Frequency
0.6
h=R default
h=.12
h=.25
h=.5
10
12
transformed data
Figure 2.2.3: Bandwidth effect on kernel density estimates. The data set income was
rescaled to have mean 1.
many people to have nearly identical incomes. The width of the peak is about 0.02,
compared to the range 11.5 of the data. The rise of the density to the left of the peak
is very steep.
There is a vast (Silverman (1986)) literature on kernel density estimation studying its mathematical properties and proposing several algorithms to obtain estimates
based on it. This method of density estimation became, apart from the histogram,
the most commonly used estimator. However it has drawbacks when the underlying
density has long tails Silverman (1986). What causes this problem is the fact that the
bandwidth is fixed for all observations, not considering any local characteristic of the
data.
In order to solve this problem several other Kernel Density Estimation Methods
were proposed such as the nearest neighbor and the variable kernel. A detailed discussion and illustration of these methods can be found in Silverman (1986).
10
2.2.1 The Nearest Neighbor Method
The idea behind the nearest neighbor method is to adapt the amount of smoothing
to local characteristics of the data. The degree of smoothing is then controlled by an
integer k. Essentially, the nearest neighbor density estimator uses distances from x in
f ( x ) to the data point. For example, let d( x1 , x ) be the distance of data point x1 from
the point x, and for each x denote dk ( x ) as the distance from its kth nearest neighbor
among the data points x1 , . . . , xn .
The kth nearest neighbor density estimate is defined as,
f( x ) =
k
,
2ndk ( x )
where n is the sample size and, typically, k is chosen to be proportional to n1/2 .

In order to understand this definition, suppose that the density at x is f ( x ). Then,
one would expect about 2rn f ( x ) observations to fall in the interval [ x r, x + r ]
for each r > 0. Since, by definition, exactly k observations fall in the interval [ x
dk ( x ), x + dk ( x )], an estimate of the density at x may be obtained by putting
k = 2dk ( x )n f( x ).
Note that while estimators like histogram are based on the number of observations
falling in a box of fixed width centered at the point of interest, the nearest neighbor
estimate is inversely proportional to the size of the box needed to contain a given
number of observations. In the tail of the distribution, the distance dk ( x ) will be larger
than in the main part of the distribution, and so the problem of under-smoothing in
the tails should be reduced. Like the histogram the nearest neighbor estimate is not a
smooth curve. Moreover, the nearest neighbor estimate does not integrate to one and
the tails of f( x ) die away at rate x 1 , in other words extremely slowly. Hence, this
estimate is not appropriate if one is required to estimate the entire density. However,
it is possible to generalize the nearest neighbor estimator in a manner related to the
kernel estimate. The generalized kth nearest neighbor estimate is defined by,
f( x ) =
n
1
x Xi
K(
).
ndk ( x ) i =1
dk ( x )
11
Observe that the overall amount of smoothing is governed by the choice of k, but the
bandwidth used at any particular point depends on the density of observations near
that point. Again, we face the problems of discontinuity of at all the points where
the function dk ( x ) has discontinuous derivative. The precise integrability and tail
properties will depend on the exact form of the kernel.
Figure 2.2.4 shows the effect of the smoothing parameter k on the density estimate. Observe that as k increases rougher the density estimate becomes. This effect
is equivalent when h is approaching to zero in the kernel density estimator.
100 obs. from N(0.5,0.1)
density
True
k=40
k=30
k=20
0.3
0.4
0.5
0.6
0.7
data
Figure 2.2.4: Effect of the smoothing parameter k on the estimates
12
2.3 Some Statistical Results for Kernel Density Estimation

As starting point one might want to compute the expected value of f. For this,
suppose we have Xi , . . . , Xn i.i.d. random variables with common density f and let
K () be a probability density function defined on the real line. Then we have, for a
nonstochastic h
E[ f( x )] =
1 n
x Xi
E[K (
)]
nh i =1
h
x Xi
1
E[K (
)]
hZ
h
xu
1
) f (u)du
K(
=
h
Zh
K (y) f ( x + yh)dy.
Now, let h 0. We see that E[ f( x )] f ( x )

asymptotic unbiased estimator of f .
(2.3.1)
K (y)dy = f ( x ). Thus, f is an
To compute the bias of this estimator we have to make the assumption that the underlying density is twice differentiable and satisfies the following conditions PrakasaRao (1983):
Condition 1. supx K ( x ) M < ; | x |K ( x ) 0 as | x | .
Condition 2. K ( x ) = K ( x ), x (, ) with
x2 K ( x )dx < .
Then by using a Taylor expansion of f ( x + yh), the bias of f in estimating f is

h2
f (x)
b f [ f( x )] =
2
y2 K (y)dy + o(h2 ).
We observe that since we have assumed the kernel K is symmetric around zero, we
R
have that yK (y)h f ( x )dy = 0, and the bias is quadratic in h. Parzen (1962)
Using a similar approach we obtain :
2.4. BANDWIDTH SELECTION

Var f [ f( x )] =
1
2
nh kK k2 f ( x )
MSE f [ f( x )] =
1
nh
13
1
), where kK k22 =
+ o( nh
f ( x )kK k22 +
h4
4 ( f (x)
kK ( x )k2 dx
1
) + o ( h4 ) ,
y2 K (y)dy)2 + o( nh
where MSE f [ f] stands for mean squared error of the estimator f of f .

Hence, when the conditions h 0 and nh are assumed, the MSE f [ f] 0,
which means that the kernel density estimate is a consistent estimator of the underlying density f . Moreover, MSE balances variance and squared bias of the estimator
in such way that the variance term controls the under-smoothing and the bias term
controls over-smoothing. In other words, an attempt to reduce the bias increases the
variance, making the estimate too noisy (under-smooth). On the contrary, minimizing the variance leads to a very smooth estimate (over-smooth) with high bias.
2.4 Bandwidth Selection

It is natural to think of finding an optimal bandwidth by minimizing MSE f [ f] in
h > 0. Hardle(1990) shows that the asymptotic approximation, say, h for the optimal
bandwidth is
h =
1/5
f ( x )kK k22
R
n1/5 .
2
2
2
( f ( x )) ( y K (y)dy) n
(2.4.1)
The problem with this approach is that h depends on two unknown functions
f () and f (). An approach to overcome this problem uses a global measure that
can be defined as:
I MSE[ f] =
MSE f [ f( x )]dx
h4
1
kK k22 + (
nh
4
y2 K (y)dy)2 k f k22 + o(
1
) + o ( h 4 ).
nh
(2.4.2)
IMSE is the well known integrated mean squared error of a density estimate. The optimal value of h considering the IMSE is define as
hopt = arg min I MSE[ f].
h>0
14
it can be shown that,
hopt =
where c2 =
c22/5
Z
K2 ( x )dx
1/5
k f k22
1/5
n1/5 ,
(2.4.3)
y2 K (y)dy. Unfortunately, (2.4.3) still depends on the second derivative
of f , which measures the speed of fluctuations in the density of f .
2.4.1 Reference to a Standard Distribution
A very natural way to get around the problem of not knowing f is to use a
standard family of distributions to assign a value of the term k f k22 in expression
(2.4.3). For example, assume that a density f belongs to the Gaussian family with
mean and variance 2 , then
Z
( f ( x )) dx =
=
( ( x ))2 dx
3 1 5
2 0.212 5,
8
(2.4.4)
where ( x ) is the standard normal density. If one uses a Gaussian kernel, then
3
hopt = (4 )1/10 ( 1/2 )1/5 n1/5
8
4 1/5
=
n1/5 = 1.06 n1/5
3
(2.4.5)
Hence, in practice a possible choice for hopt is 1.06 n1/5 , where is the sample
standard deviation.
If we want to make this estimate more insensitive to outliers, we have to use a
more robust estimate for the scale parameter of the distribution. Let R be the sample
interquartile, then one possible choice for h is
R
) n1/5
((3/4) (1/4))
R
) n1/5 ,
= 1.06 min(,
1.349
h opt = 1.06 min(,
where is the standard normal distribution function.
(2.4.6)
15
Figure 2.4.5 exhibits how a robust estimate of the scale can help in choosing the
bandwidth. Note that by using R we have strong evidence that the underlying density has two modes.
0.30
Histogram of a mixture of two normal densities
0.15
0.00
0.05
0.10
Relative Frequency
0.20
0.25
True
sigmahat
interquartile
data
Figure 2.4.5: Comparison of two bandwidths, (the sample standard deviation) and
R (the sample interquartile) for the mixture 0.7 N (2, 1) + 0.3 N (1, 1).
2.4.2 Maximum likelihood Cross-Validation
Consider kernel density estimates f and suppose we want to test for a specific h
the hypothesis
f( x ) = f ( x )
vs.
f( x ) 6= f ( x ),
for a fixed x The likelihood ratio test would be based on the test statistic f ( x )/ f( x ).
For a good bandwidth this statistic should thus be close to 1. Alternatively, we would
16
expect E[log(
f (X)
)]
f( X )
to be close to 0. Thus, a good bandwidth, which is minimizing
this measure of accuracy, is in effect optimizing the Kullback-Leibler distance:

dKL ( f , f) =
log
f (x)
f( x )
f ( x )dx.
(2.4.7)
Of course, we are not able to compute dKL ( f , f) from the data, since we do not
know f . But from a theoretical point of view, we can investigate this distance for
the choice of an appropriate bandwidth h. When dKL ( f , f) is close to 0 this would
give the best agreement with the hypothesis f = f . Hence, we are looking for a
bandwidth h, which minimizes dKL ( f , f).
Suppose we are given a set of additional observations Xi , independent of the others. The likelihood for these observations is i f (Xi ). Substituting f in the likelihood
equation we have i f(Xi ) and the value of this statistic for different h would indicate
which value of h is preferable, since the logarithm of this statistic is close to dKL ( f , f).
Usually, we do not have additional observations. A way out of this dilemma is to base
the estimate f on the subset { X j } j6=i , and to calculate the likelihood for Xi . Denoting
the leave-one-out estimate
f(Xi ) = (n 1)1 h1 K (
j 6 =i
Hence,
i =1
Xi X j
).
h
f(Xi ) = (n 1)n hn K (
i =1 j 6 =i
Xi X j
).
h
(2.4.8)
However it is convenient to consider the logarithm of this statistic normalized

with the factor n1 to get the following procedure:
1 n
log[ f h,i (Xi )]
n i
=1
h
Xi X j i
1 n
=
log
K
(
) log[(n 1)h]
n i
h
=1
j 6 =i
CVKL (h) =
(2.4.9)
Naturally, we choose hKL such that:

hKL = arg max CVKL (h).
h
(2.4.10)
17
Since we assumed that Xi are i.i.d., the scores log fi (Xi ) are identically distributed
and so,
E[CVKL (h)] = E[log fi (Xi )].
Disregarding the leave-one-out effect, we can write
E[CVKL (h)] E
hZ
log f( x ) f ( x )dx
E[dKL ( f , f)] +
log[ f ( x )] f ( x )dx.
(2.4.11)
The second term of the right-hand side does not depend on h. Then, we can expect
that we approximate the optimal bandwidth that minimizes dKL ( f , f).
The Maximum likelihood cross validation has two shortcomings:
When we have identical observations in one point, we may obtain an infinite
value if CVKL (h) and hence we cannot define an optimal bandwidth.
Suppose we use a kernel function with finite support, e.g., the interval [1, 1]. If
an observation Xi is more separated from the other observations than the bandwidth
h, the likelihood fi (Xi ) becomes 0. Hence the score function reaches the value .
Maximizing CVKL (h) forces us to use a large bandwidth to prevent this degenerated
case. This might lead to slight over-smoothing for the other observations.
18
2.4.3 Least-Squares Cross-Validation
Consider an alternative distance between f h and f . The integrated squared error

(ISE)
d ISE (h) =
=
d ISE (h)
f ( x )dx =
( f h f )2 ( x )dx
f h2 ( x )dx
f h2 ( x )dx
2
2
( f h f )( x )dx +
f 2 ( x )dx
( f h f )( x )dx
(2.4.12)
R
For the last term, observe that ( f h f )( x )dx = E[ f h (Xi )] where the expectation is un-
derstood to be computed with respect to an additional and independent observation

X. For estimation of this term define the leave-one-out estimate
1 n
EX [ fh (X )] = f h,i (Xi )
n i =1
(2.4.13)
This leads to the Least-squares cross-validation:
CVLS (h) =
f h2 ( x )dx 2 f h,i (Xi )
(2.4.14)
i =1
The bandwidth minimizing this function is,

h LS = arg min CVLS (h).
h
This cross-validation function is called an unbiased cross-validation criterion, since,

1 n
E[CVLS (h)] = E[d ISE (h) + 2(EX [ f h (X )] E[ f h,i (Xi )]) k f k22
n i =1
= I MSE[ f h ] k f k22 .
(2.4.15)
An interesting question is, how good is the approximation of d ISE by CVLS . To investigate this define a sequence of bandwidths hn = h(X1 , . . . , Xn ) to be asymptotically
optimal, if
d ISE (hn )
1, a.s. when n .
infh>0 d ISE (h)
It can be shown that if the density f is bounded then h LS is asymptotically optimal.
Similarly to maximum likelihood cross-validation one can found in Hardle (1990) an
algorithm to compute the least-squares cross-validation.
2.5. ORTHOGONAL SERIES ESTIMATORS
19
2.5 Orthogonal series estimators
Orthogonal series estimators approach the density estimation problem from a

quite different point of view. While kernel estimators is close related to statistical
thinking orthogonal series relies on the ideas of approximation theory. Without loss
of generality let us assume that we are trying to estimate a density f on the interval
[0, 1]. The idea is to use the theory of orthogonal series method and then to reduce the
estimation procedure by estimating the coefficients of its Fourier expansion. Define
the sequence v ( x ) by
(x) = 1
2r1 ( x ) = 2 cos 2rx r = 1, 2, . . .
( x ) = 2 sin 2rx r = 1, 2, . . .
2r
It is well known that f can be represented as Fourier series i=0 ai i , where, for
each i 0,
ai =
f ( x )i ( x )dx.
(2.5.1)
Now, suppose that X is a random variable with density f . Then (2.5.1) can be
written
ai = Ei (X )
and so an unbiased estimator of f based on X1 , . . . , Xn is
ai =
1
n
i (Xj ).
j =1
Note that the i=1 a i i converges to a sum of delta functions at the observations, since
(x) =
1 n
( x Xi )
n i
=1
where is the Dirac delta function. Then for each i,

ai =
Z 1
0
( x )i ( x )dx
(2.5.2)
20
and hence the ai are exactly the Fourier coefficients of the function . The easiest to
way to smooth is to truncate the expansion ai i at some point. That is, choose K
and define a density estimate f by
f( x ) =
ai i (x).
(2.5.3)
i =1
Note that the amount of smoothing is determined by K. Small value of K implies in

over-smoothing, large value of K under-smoothing.
100 obs from N(.5,.1)
3
0
density
True
K=3
K=10
K=100
0.0
0.2
0.4
0.6
0.8
1.0
data
Figure 2.5.6: Effect of the smoothing parameter K on the orthogonal series method
for density estimation
A more general approach would be, choose a sequence of weights i , such that,
i 0 as i . Then
f( x ) =
i ai i (x).
i =0
The rate at which the weights i converge to zero will determine the amount of
smoothing. For non finite interval we can have weight functions a( x ) = e x
orthogonal functions ( x ) proportional to Hermite polynomials.
2 /2
and
2.5. ORTHOGONAL SERIES ESTIMATORS
21
The data in figure 2.5.6 were provided to me by Francisco Cribari-Neto and consists of the variation rate of ICMS (imposto sobre circulaca o de mercadorias e servicos)
tax for the city of Brasilia, D.F., from August 1994 to July 1999.
22
Chapter 3
Kernel nonparametric Regression
Method
Suppose we have i.i.d. observations {(Xi , Yi )}in=1 and the nonparametric regression
model given in equation (1.0.1). By equation (2.0.1) we know how to estimate the
denominator by using the kernel density estimation method. For the numerator one
can estimate the joint density using the multiplicative kernel
f h1 ,h2 ( x, y) =
1 n
Kh1 ( x Xi )Kh2 (y Yi ).
n i
=1
where, Kh1 ( x Xi ) = h11 K (( x Xi )/h1 ), Kh2 ( x Yi ) = h21 K (( x Yi )/h2 ). It is not

difficult to show that
Z
y f h1 ,h2 ( x, y)dy =
1 n
Kh1 ( x Xi )Yi .
n i
=1
Based on the methodology of kernel density estimation Nadaraya (1964) and Watson
(1964) suggested the following estimator gh for g.
gh ( x ) =
in=1 Kh ( x Xi )Yi
nj=1 Kh ( x X j )
(3.0.1)
In general, the kernel function Kh ( x ) = K (( x x j )/h) is taken as probability density

function symmetric around zero and parameter h is called smoothing parameter or
bandwidth.
23
CHAPTER 3. KERNEL NONPARAMETRIC REGRESSION METHOD
24
Now, consider the model (1.0.1) and let X1 , . . . , Xn be i.i.d. random variables with
density f X such that Xi is independent of i for all i = 1, . . . , n. Assume the conditions
given in Section 2.3 and suppose that f and g are twice continuously differentiable
in neighborhood of the point x. Then, if h 0 and nh as n , we
R
have gh g in probability. Moreover, suppose E[| i |2+ ] and |K ( x |2+ dx < ,
R
for some > 0, then nh ( gh E[ gh ]) N (0, ( f X ( x ))1 2 (K ( x ))2 dx ) in distribu-
tion, where N (, ) stands for a Gaussian distribution, (see details in Pagan and Ullah
(1999)).
As an example, figure 3.0.1 shows the effect of choosing h on the NadarayaWatson procedure. The data consist of the speed of cars and the distances taken
to stop. It is important to notice that the data were recorded in the 1920s. (These
datasets can be found in the software R) The Nadaraya-Watson kernel method can
be extended to the multivariate regression problem by considering the multidimen-
100
120
sional kernel density estimation method (see details in Scott (1992)).
60
40
20
0
dist
80
h=2
h=5
10
15
20
speed
Figure 3.0.1: Effect of bandwiths on Nadaraya-Watson kernel
25
3.1. K-NEAREST NEIGHBOR (K-NN)
25
3.1 k-Nearest Neighbor (k-NN)
One may notice that regression by kernels is based on local averaging of observations Yi in a fixed neighborhood of x. Instead of this fixed neighborhood, k-NN
employs varying neighborhoods in the X variable support. That is,
gk ( x ) =
1 n
Wki ( x )Yi ,
n i
=1
(3.1.1)
where,
n/k if i J
x
Wki ( x ) =
0
otherwise,
(3.1.2)
with Jx = {i : Xi is one of the k nearest observations to x }

It can be shown that the bias and variance of the k-NN estimator gk with weights
(3.1.2) are given by, for a fixed x
E[ gk ( x )] g( x )
1
[ g ( x ) f ( x ) + 2g ( x ) f ( x )](k/n)2
3
24( f ( x ))
(3.1.3)
2
.
k
(3.1.4)
and
Var [ gk ( x )]
We observe that the bias increasing and the variance is decreasing in the smoothing
parameter k. To balance this trade-off one should choose k n4/5 . For details, see
Hardle (1990).
Figure 3.1.2 shows the effect of the parameter k on the regression curve estimates.
Note that the curve estimate with k = 2 is less smoother than the curve estimate
with k = 1. The data set consist of the revenue passenger miles flown by commercial
airlines in the United States for each year from 1937 to 1960 and is available through
R package.
26
25000
30000
airmiles data
15000
0
5000
10000
airmiles
20000
Data
K=1
K=2
1940
1945
1950
1955
1960
Passengermiles flown by U.S. commercial airlines
Figure 3.1.2: Effect of the smoothing parameter k on the k-NN regression estimates.
3.2 Local Polynomial Regression: LOWESS
Cleveland (1979) proposed the algorithm LOWESS, locally weighted scatter plot
smoothing, as an outlier resistant method based on local polynomial fits. The basic
idea is to start with a local polynomial (a k-NN type fitting) least squares fit and then
to use robust methods to obtain the final fit. Specifically, one can first fit a polynomial
regression in a neighborhood of x, that is, find R p+1 which minimize
n
Wki
i =1
yi j x
j =0
2
(3.2.1)
where Wki denote k-NN weights. Compute the residuals i and the scale parameter = median(i ). Define robustness weights i = K (i /6 ), where K (u) =
(15/16)(1 u)2 , if |u| 1 and K (u) = 0, if otherwise. Then, fit a polynomial regression as in (3.2.1) but with weights (i Wki ( x )). Cleveland suggests that p = 1
provides good balance between computational ease and the need for flexibility to
3.2. LOCAL POLYNOMIAL REGRESSION: LOWESS
27
reproduce patterns in the data. In addition, the smoothing parameter can be determined by cross-validation as in (2.4.10). Note that when using the R function lowess
or loess, f acts as the smoothing parameter. Its relation to the k-NN nearest neighbor
is given by
k = n f ,
f (0, 1),
where n is the sample size.
120
lowess(cars)
60
40
20
0
dist
80
100
f = 2/3
f = .2
10
15
20
25
speed
Figure 3.2.3: Effect of the smoothing parameter using LOWESS method.
28
3.3 Penalized Maximum Likelihood Estimation

The method of penalized maximum likelihood in the context of density estimation
consist of estimating a density f by minimizing a penalized likelihood score L ( f ) +
J ( f ), where L ( f ) is a goodness-of-fit measure, and J ( f ) is a roughness penalty. This
section is developed considering historical results, beginning with Good and Gaskins
(1971), and ending with the most recent result given by Gu (1993).
The maximum likelihood (M.L.) method has been used as statistical standard procedure in the case where the underlying density f is known except by a finite number
of parameters. It is well known the M.L. has optimal properties (asymptotically unbiased and asymptotically normal distributed) to estimate the unknown parameters.
Thus, it would be interesting if such standard technique could be applied on a more
general scheme where there is no assumption on the form of the underlying density
by assuming f to belong to a pre-specified family of density functions.
Let X1 , . . . , Xn be i.i.d. random variables with unknown density f . The likelihood
function is given by:
L( f | X1 , . . . , Xn ) = f (Xi ).
i =1
The problem with this approach can be described by the following example. Recall fh ( x ) a kernel estimate, that is,
1
fh ( x ) =
nh
K(
i =1
x Xi
),
h
with h = h/c, where c is constant greater than 0, i.e., for the moment the bandwidth
is h/c. Let h be small enough such that |
Xi Xi
h/c |
> M > 0, and assume K has been
chosen so that K (u) = 0, if |u| > M. Then,

c
K (0 ).
fh (Xi ) =
nh
If c >
1
K (0)
then fh (Xi ) >
1
nh .
For fixed n, we can do this for all Xi simultaneously.
Thus,
L(
1 n
) .
nh
3.3. PENALIZED MAXIMUM LIKELIHOOD ESTIMATION
29
Letting h 0, we have L . That is, L( f | X1 , . . . , Xn ) does not have a finite maximum over the class of all densities. Hence, the likelihood function can be as large
as one wants it just by taking densities with the smoothing parameter approaching zero. Densities having this characteristic, e.g., bandwidth h 0, approximate
to delta functions and the likelihood function ends up to be a sum of spikes delta
functions. Therefore, without putting constraints on the class of all densities, the
maximum likelihood procedure cannot be used properly.
One possible way to overcome the problem described above is to consider a penalized log-likelihood function. The idea is to introduce a penalty term on the loglikelihood function such that this penalty term quantifies the smoothness of g =
log f .
Let us take, for instance, the functional J ( g) =
define the penalized log-likelihood function by
L ( g) =
( g )2 as a penalty term. Then
1 n
g(Xi ) J ( g) ,
n i
=1
(3.3.1)
where is the smoothing parameter which controls two conflicting goals, the fidelity
to the data given by in=1 g(Xi ) and the smoothness, given by the penalty term J ( g).
The pioneer work on penalized log-likelihood method is due to Good and Gaskins(1971), who suggested a Bayesian scheme with penalized log-likelihood (using
their notation) becomes:
= ( f ) = L( f ) ( f ) ,
where L = in=1 g(Xi ) and is the smoothness penalty.
R
R
In order to simplify the notation, let h have the same meaning as h( x )dx.
Now, consider the number of bumps in the density as the measure of roughness or
smoothness. The first approach was to take the penalty term proportional to Fishers
information, that is,
Z
( f ) = ( f )2 / f .
R
Now by setting f = 2 , ( f ) becomes ( )2 , and then replace f by in the penal-
ized likelihood equation. Doing that the constraint f 0 is eliminated and the other
R
R
constraint, f = 1, turns out to be equivalent to 2 = 1, with L2 (, ).
30
R
Good and Gaskins(1971) verified that when the penalty 4 ( )2 yielded density
curves having portions that looked too straight. This fact can be explained noting
that the curvature depends also on the second derivatives. Thus ( )2 should be
included on the penalty term. The final roughness functional proposed was:
( f ) = 4
( ) +
( )2 ,
with , satisfying,
3
22 + = 4 ,
4
(3.3.2)
where 2 is either an initially guessed value of the variance or it can be estimated

the sample variance based on the data.
According to Good and Gaskins (1971),
the basis for this constraint is the feeling that the class of normal distributions form
the smoothest class of distributions, the improper uniform distribution being limiting form. Moreover, they pointed out that some justification for this feeling is that
a normal distribution is the distribution of maximum entropy for a given mean and
R
variance. The integral ( )2 is also minimized for a given variance when f is nor-
mal (Good and Gaskins, 1971). They thought was reasonable to give the normal
distribution special consideration and decided to choose , such that (, ; f ) is

maximized by taking the mean equal to x and variance as iN=1 ( xi x )2 /N 1. That
R
R
3
is, if f ( x ) N (, 2 ) then ( )2 = 41 2 , ()2 = 16
4 and hence we have,
(, ; f ) =
1
N
log(22 ) 2
2
2
(xi )2 2 164 .
i =1
The score function (, ; f ) is maximized when = x and is such that,
N +
3
iN=1 ( xi x )2 2
+ 2 + 4 = 0.
2
(3.3.3)
If we put 2 = iN=1 ( xi x )2 /N 1, the equation (3.3.3) becomes,

4 ( N 1) + 22 +
3
= 4 N,
4
and so we have the constraint (3.3.2).

Pursuing the idea of Good and Gaskins, Silverman (1982) proposed a similar
method where the log density is estimated instead of the density itself. An advantage
31
of Silvermans approach is that using the logarithm of the density and the augmented
Penalized likelihood functional, any density estimates obtained will automatically be
positive and integrate to one. Specifically,
Let (m1 , . . . , mk ) be a sequence of natural numbers so that 1 ik=1 mi m,
where m > 0 is such that g(m1) exists and is continuous. Define a linear differential
operator D as:
D ( g) =
c(m1 , . . . , mk )( x1 )m1 . . . ( xk )mk (g) .
Now assume that at least one of the coefficients c(m1 , . . . , mk ) 6= 0 for mi = m.
Using this linear differential operator define a bilinear functional h, i by
h g1 , g2 i =
D ( g1 ) D ( g2 ) .
where the integral is taken over a open set with respect to Lebesgue measure.
Let S be the set of real functions g on for which:
the (m 1)th derivatives of g exist everywhere and are piecewise differentiable,
h g, gi < ,
e g < .
Given the data X1 , . . . , Xn i.i.d. with common density f , such that g = log f , g is the
solution, if it exists, of the optimization problem
max{
subject to
1 n
g(Xi ) h g, gi} ,
n i =1
2
e g = 1. And the density estimate f = e g , where the the null space of the
penalty term is the set { g S : h g, gi = 0}.
Note that the null space of h g, gi is an exponential family with at most (m 1)

R
parameters, for example, if h g, gi = ( g(3) )2 then g = log f is in an exponential
family with 2 parameters. See Silverman (1982).
Silverman presented an important result which makes the computation of the constrained optimization problem a relatively easy computational scheme of finding
32
the minimum of an unconstrained variational problem. Precisely, for any g in a class

of smooth functions (see details in Silverman (1982)) and for any fixed positive , let
1 n
0 ( g ) = g ( Xi ) +
n i =1
2
( g )2
and
1 n
( g ) = g ( Xi ) +
n i =1
e +
2
g
( g )2 .
Silverman proved that unconstrained minimum of ( g) is identical with the constrained minimum of 0 , if such a minimizer exists.
3.3.1 Computing Penalized Log-Likelihood Density Estimates
Based on Silvermans approach, OSullivan(1988) developed an algorithm which

is a fully automatic, data driven version of Silvermans estimator. Furthermore, the
estimators obtained by OSullivans algorithm are approximated by linear combination of basis functions. Similarly to the estimators given by Good and Gaskins(1971),
OSullivan proposed that cubic B-splines with knots at data points should be used as
the basis functions. A summary of definitions and properties of B-splines were given
in the section 4.
The basic idea of computing a density estimate provided by penalized likelihood
method is to construct approximations to it. Given x1 , . . . , xn , the realizations of random variables X1 , . . . , Xn , with common log density g. We are to solve a finite version
of (3.3.1) which are reasonable approximations to the infinite dimensional problem
(Thompson and Tapia, 1990, 121145). Good and Gaskins (1971) based their computational scheme on the fact that since L2 (, ) then for a given orthonormal
system of functions {n },
m.s.
an n g L2 ,
n =0
2
with
n =0 | an | < and { an } R. That is, in L can be arbitrarily approximated
by a linear combination of basis functions. In their paper, Hermite polynomials were
33
used as basis functions. Specifically:

n ( x ) = e x
2 /2
Hn ( x )2n/2 1/4 (n!)1/2 ,
where,
2
Hn ( x ) = (1)n e x (
d n x2
e ).
dx n
The log density estimator proposed by OSullivan (1988) is defined as the minimizer of
1 n
g ( xi ) +
n i =1
Z b
a
g(s)
ds +
Z b
a
( g(m) )2 ds,
(3.3.4)
for fixed > 0, and data points x1 , . . . , xn . The minimization is over a class of absolutely continuous functions on [ a, b] whose mth derivative is square integrable.
Computational advantages of this log density estimators using approximations
by cubic B-splines are:
It is a fully automatic procedure for selecting an appropriate value of the smoothing parameter , based on the AIC type criteria.
The banded structures induced by B-splines leads to an algorithm where the
computational cost is linear in the number of observations (data points).
It provides approximate pointwise Bayesian confidence intervals for the estimator.
A disadvantage of OSullivans work is that it does not provide any comparison of
performance with other available techniques.
We see that the previous computational framework is unidimensional, although
Silvermans approach can be extended to higher dimensions.
34
Chapter 4
Spline Functions
4.1 Acquiring the Taste
Due to their simple structure and good approximation properties, polynomials
are widely used in practice for approximating functions. For this propose, one usually divides the interval [ a, b] in the function support into sufficiently small subintervals of the form [ x0 , x1 ], . . . , [ xk , xk+1 ] and then uses a low degree polynomial pi for
approximation over each interval [ xi , xi +1 ], i = 0, . . . , k. This procedure produces a
piecewise polynomial approximating function s();
s( x ) = pi ( x ) on [ xi , xi +1 ], i = 0, . . . , k.
In the general case, the polynomial pieces pi ( x ) are constructed independently of
each other and therefore do not constitute a continuous function s( x ) on [ a, b]. This
is not desirable if the interest is on approximating a smooth function. Naturally, it is
necessary to require the polynomial pieces pi ( x ) to join smoothly at knots x1 , . . . , xk ,
and to have all derivatives up to a certain order, coincide at knots. As a result, we get
a smooth piecewise polynomial function, called a spline function.
Definition 4.1.1 The function s( x ) is called a spline function (or simply spline) of degree
r with knots at { xi }ik=1 if =: x0 < x1 < . . . < xk < xk+1 := , where =: x0 and
xk+1 := are set by definition,
35
CHAPTER 4. SPLINE FUNCTIONS
36
for each i = 0, . . . , k, s( x ) coincides on [ xi , xi +1 ] with a polynomial of degree not greater

than r;
s( x ), s ( x ), . . . , sr1 ( x ) are continuous functions on (, ).
The set Sr ( x1 , . . . , xk ) of spline functions is called spline space. Moreover, the spline
space is a linear space with dimension r + k + 1 (Schumaker (1981)).
Definition 4.1.2 For a given point x (a, b) the function
(t x )r if t > x
r
(t x )+ =
0
if t x
is called the truncated power function of degree r with knot x.
Hence, we can express any spline function as a linear combination of r + k + 1 basis

functions. For this, consider a set of interior knots { x1 , . . . , xk } and the basis functions
{1, t, t2 , . . . , tr , (t x1 )r+ , . . . , (t xk )r+ }. Thus, a spline function is given by,

r
s (t) =
i t
i =0
j =r +1
j (t x jr )r+
It would be interesting if we could have basis functions that make it easy to compute the spline functions. It can be shown that B-splines form a basis of spline spaces
Schumaker (1981). Also, B-splines have an important computational property, they
are splines which have smallest possible support. In other words, B-splines are zero
on a large set. Furthermore, a stable evaluation of B-splines with the aid of a recurrence relation is possible.
Definition 4.1.3 Let = { x j }{ jZ} be a nondecreasing sequence of knots. The i-th Bspline of order k for the knot sequence is defined by
Bkj (t) = ( xk+ j x j )[ x j , . . . , xk+ j ](t x j )k+1
for all
t R,
where, [ x j , . . . , xk+ j ](t x j )k+1 is (k 1)th divided difference of the function ( x x j )k+
evaluated at points x j , . . . , xk+ j .
4.1. ACQUIRING THE TASTE
37
From the Definition 4.1.3 we notice that Bkj (t) = 0 for all t 6 [ x j , x j+k ]. It follows
that only k B-splines have any particular interval [ x j , x j+1] in their support. That
is, of all the B-splines of order k for the knot sequence , only the k B-splines
Bkjk+1, Bkjk+2, . . . , Bkj might be nonzero on the interval [ x j , x j+1 ]. (See de Boor (1978)
for details). Moreover, Bkj (t) > 0 for all x ( x j , x j+k ) and jZ Bkj (t) = 1, that is, the
B-spline sequence Bkj consists of nonnegative functions which sum up to 1 and provides a partition of unity. Thus, a spline function can be written as linear combination
of B-splines,
s (t) =
j Bkj (t).
j Z
The value of the function s at point t is simply the value of the function jZ j Bkj (t)
0.0
0.2
0.4
Bsplines
0.6
0.8
1.0
which makes good sense since the latter sum has at most k nonzero terms.
0.0
0.2
0.4
0.6
0.8
1.0
Figure 4.1.1: Basis Functions with 6 knots placed at x

Figure 4.1.1 shows an example of B-splines basis and their compact support property. This property makes the computation of B-splines easier and numerically stable.
Of special interest is the set of natural splines of order 2m, m N, with k knots
at x j . A spline function is a natural spline of order 2m with knots at x1 , . . . , xk , if, in
38
addition to the properties implied by definition (4.1.1), it satisfies an extra condition:

s is polynomial of order m outside of [ x1 , xk ].
Consider the interval [ a, b] R and the knot sequence a := x0 < x1 < . . . < xk <
xk+1 := b. Then, N S 2m = {s S(P2m ) : s0 = s|[ a,x1 ) and sk = s|[ xk ,b) Pm }, is the
natural polynomial spline space of order 2m with knots at x1 , . . . , xk . The name natural
spline stems from the fact that, as a result of this extra condition, s satisfies the so
called natural boundary conditions s j (a) = s j (b) = 0, j = m, . . . , 2m 1.
Now, since the dimension of S(P2m ) is 2m + k and we have enforced 2m extra
conditions to define N S 2m , it is natural to expect the dimension of N S 2m to be k.
Actually, it is well known that N S 2m is linear space of dimension k. See details in

Schumaker (1981).
In some applications it may be possible to deal with natural splines by using a
basis for S(P2m ) and enforcing the end conditions. For other applications it is desir-
able to have a basis for N S 2m itself. To construct such a basis consisting of splines
with small supports we just need functions based on the usual B-splines. Particularly, when m = 2, we will be constructing basis functions for the Natural Cubic Spline
0.2
0.0
0.2
Natural Splines
0.4
0.6
Space, N S 4 . Figure 4.1.2 show an example of the natural splines basis.
0.0
0.2
0.4
0.6
0.8
1.0
Figure 4.1.2: Basis Functions with 6 knots placed at x
4.2. LOGSPLINE DENSITY ESTIMATION
39
4.2 Logspline Density Estimation

Kooperberg and Stone (1991) introduced another type of algorithm to estimate an
univariate density. This algorithm was based on the work of Stone (1990) and Stone
and Koo (1985) where the theory of the logspline family of functions was developed.
Consider an increasing sequence of knots {t j }Kj=1 , K 4, in R . Denote by S0
the set of real functions such that s is a cubic polynomial in each interval of the
form (, t1 ], [t1 , t2 ], . . . , [tK , ). Elements in S0 are the well-known cubic splines
with knots at {t j }Kj=1. Notice that S0 is a (K + 4)-dimensional linear space. Now, let
S S0 such that the dimension of S is K with functions s S linear on (, t1 ]
and on [tK , ). Thus, S has a basis of the form 1, B1 . . . , BK 1, such that B1 is linear
function with negative slope on (, t1 ] and B2 , . . . , BK 1 are constant functions on
the same interval. Similarly, BK 1 is linear function with positive slope on [tK , ) and
B1 , . . . , BK 2 are constant on the interval [tK , ).
Let be the parametric space of dimension p = K 1, such that for = (1 , . . . , p )
R p , 1 < 0 and p > 0. Then, define
c() = log(
and
K 1
exp( j B j ( x )dx ))
j =1
K 1
f ( x; ) = exp{ j B j ( x ) c()}.
j =1
The p-parametric exponential family f (, ), R p of positive twice differentiable density function on R is called logspline family and the corresponding loglikelihood function is given by
L() =
log f (x; );
The log-likelihood function L() is strictly concave and hence the maximum likelihood estimator of is unique, if it exists. We refer to f = f (, ) as the logspline
density estimate. Note that the estimation of makes logspline procedure not essentially nonparametric. Thus, estimation of by Newton-Raphson, together with small
40
numbers of basis function necessary to estimate a density, make the logspline algorithm extremely fast when it is compared with Gu (1993) algorithm for smoothing
spline density estimation.
In the Logspline approach the number of knots is the smoothing parameter. That
is, too many knots lead to a noisy estimate while too few knots give a very smooth
curve. Based on their experience of fitting logspline models, Kooperberg and Stone
provide a table with the number of knots based on the number of observations. No
indication was found that the number of knots takes in consideration the structure of
the data (number of modes, bumps, asymmetry, etc.). However, an objective criterion
for the choice of the number of knots, Stepwise Knot Deletion and Stepwise knot Addition,
are included in the logspline procedure.
For 1 j p, let B j be a linear combination of a truncated power basis ( x tk )3+
for the a knot sequence t1 , . . . , t p , that is,
B j ( x ) = j + j0 x + jk ( x tk )3+ .
k
Then
j Bj (x) = j j0 + jk j (x tk )3+ .
j
Then, for 1 k K Kooperberg and Stone (1991), show that

Let j j jk = Tk .
q
T
SE( k ) = Tk ( I ( ))1 k )
where I () is the Fisher information matrix obtained from the log-likelihood function.
The knots t1 and tK are considered permanent knots, and tk , 2 k K, are
nonpermanent knots. Then at any step delete (similarly for addition step) that knot
which has the smallest value of | Tk |/SE( Tk ). In this matter, we have a sequence
of models which ranges from 2 to p 1 knots. Now, denote by L m the log-likelihood
function of the mth model (2 m + 2 p 1) evaluated at the maximum likelihood

estimate for that model. To specify a stop criteria, Kooperberg and Stone make use
of the Akaike Information Criterion (AIC), that is, AIC,m = 2 L m + ( p m) and
that minimizes AIC3,m . There is no theoretical justification for choosing
choose m
4.3. SPLINES DENSITY ESTIMATION: A DIMENSIONLESS APPROACH
41
= 3. The choice was made, according to them, because this value of makes the
probability that f is bimodal when f is Gamma(5) to be about 0.1. Figure 4.2.3 shows
an example of logspline density estimation for a mixture of two normal densities.
0.20
Mixture of Normals
0.00
0.05
0.10
0.15
True
logspline
Figure 4.2.3: logspline density estimation for .5*N(0,1)+.5*N(5,1)

It would be interesting to have an algorithm which combines the low computational cost of logsplines (due to B-splines and the estimation of their coefficients) and
the performance of the automatic smoothing parameter selection developed by Gu
(1993).
4.3 Splines Density Estimation: A Dimensionless Approach
Let X1 , . . . , Xn a random sample from a probability density f on a finite domain

R
X . Assuming that f > 0 on X , one can make a logistic transformation f = e g /( e g ).
42
We know that this transformation is not one-to-one and Gu and Qiu (1993) proposed
R
side conditions on g such that g( x0 ) = 0, x0 X or X g = 0. Given those conditions
we have to find the minimizer of the penalized log-likelihood
1 n
g(Xi ) + log
n i =1
eg +
J ( g)
2
(4.3.1)
in a Hilbert space H, where J is a roughness penalty and is the smoothing parameter. The space H is such that the evaluation is continuous so that the first term in
(4.3.1) is continuous. The penalty term J is a seminorm in H with a null space J of

finite dimension M 1. By taking a finite dimensional J one prevents interpolation
(i.e. the empirical distribution) and a quadratic J makes easier the numerical solution of the variational problem (4.3.1). Since, H is an infinite dimensional space, the
minimizer of (4.3.1) is, in general, not computable. Thus, Gu and Qiu (1993) propose
calculating the solution of the variational problem in finite dimensional space, say,
Hn , where n is the sample size.

The performance of the smoothing spline estimator depends upon the choice of
the smoothing parameter . Gu (1993), suggested a performance-oriented iteration
procedure ( GCV-like procedure) which updates g and jointly according to a performance estimate. The performance is measured by a loss function which was taken
R
R
as a symmetrized Kullback-Leibler distance between e g / e g and e g0 / e g0 . Specif-
ically, if one solves the variational problem (4.3.1) in Hn by a standard Newton instead of calculating
Raphson procedure, then by starting from a current iterate g,
the next iterate with a fixed , one may choose a that minimizes the loss function.
Figure 4.3.4 exhibits the performance of SSDE for Buffalo Snow data. (This data
set can be found in R.)
4.3. SPLINES DENSITY ESTIMATION: A DIMENSIONLESS APPROACH
43
0.020
Buffalo snow data
0.0
0.005
0.010
0.015
SSDE
Logspline(d)
Kernel
20
40
60
80
100
120
data
Figure 4.3.4: Histogram, SSDE, Kernel and Logspline density estimates

Under this approach, one might ask the following questions:
Is it possible to estimate a density using K n basis functions instead of the
original n such that it reduces the computational cost of getting the solution (4.3.1)
significantly ? How good would such an approximation be ?
Dias (1998) provided some answers to those questions by using the basis functions
Bi ( x ) given in Definition (4.1.3) that can be easily extend to a multivariate case by a
tensor product.
44
Chapter 5
The thin-plate spline on Rd
There are many applications where a unknown function g of one or more variables and a set of measurements are given such that:
yi = Li g + i
(5.0.1)
where L1 , . . . , Ln are linear functionals defined on some linear space H containing g,

and 1 , . . . , n are measurement errors usually assumed to be independently, identically and normally distributed with mean zero and unknown variance 2 . Typically,
the Li will be point evaluation of the function g.
Straight forward least square fitting is often appropriate but it produces a function which is not sufficiently smooth for some data fitting problems. In such cases,
it may be better to look for a function which minimizes a criterion that involves
a combination of goodness of fit and an appropriate measure of smoothness. Let
t = ( x1 , . . . , xd ), ti = ( x1 (i ), . . . , xd (i )) for i = 1, . . . , n and the evaluation functionals
Li g = g(ti ), then the regression model (5.0.1) becomes,

yi = g( x1 (i ), . . . , xd (i )) + i .
(5.0.2)
The thin-plate smoothing spline is the solution to the following variational problem.
Find g H to minimize
L ( g ) =
1 n
d
(yi g(ti ))2 + Jm
( g)
n i
=1
45
(5.0.3)
CHAPTER 5. THE THIN-PLATE SPLINE ON RD
46
where is the smoothing parameter which controls the trade off between fidelity to
d . Note that, when is large a prethe data and smoothness with penalty term Jm
mium is being placed on smoothness and functions with large mth derivatives are
penalized. In fact, gives an mth order polynomial regression fit to the data.
Conversely, for small values of more emphasis is put on goodness-of-fit and the
limit case of 0, we have interpolation. In general, in smoothing spline non-
d is given by
parametric regression the penalty term Jm
d
Jm
( g)
m!
=
1 ! . . . , d !
1 + ...+ d = m
...
Z
2
m g
dx j .
x1 1 . . . xd d
j
The condition 2m d > 0 is necessary and sufficient in order to have bounded evaluation functionals in H, i.e., H is a reproducing kernel in Hilbert space. Moreover,
d is the M-dimensional space spanned by polythe null space of the penalty term Jm
nomials 1 , . . . , M of degree less or equal to m 1, e.g., i (t) = t j1 /( j 1)!, for

j = 1, . . . , m.
Wahba (1990) has shown that, if t1 , . . . , tn are such that least squares regression on
1 , . . . , M is unique, then (5.0.3) has a unique minimizer g , with representation
g ( t ) =
i =1
j =1
ci Em (t, ti ) + bj j (t)
= Qc + Tb
(5.0.4)
where, T is a n M matrix with entries j (tl ) for j = 1, . . . , M, l = 1, . . . , n and Q is

a n n matrix with entries Em (tl , ti ), for i = 1, . . . , n. The function Em is a Greens
function for the m-iterate Laplacian Wahba (1990), Silverman and Green (1994). For
1
example, when d = 1, Em (t, ti ) = (t ti )m
+ /(m 1)!. The coefficients c and b can
be determined by substituting (5.0.4) into (5.0.3). Thus, the optimization problem

(5.0.3) subject to T c = 0, is reduced to a linear system of equations which is solved
by standard matrix decomposition such as QR decomposition. The constraint T c =
d ( g ) is
0 is necessary to guarantee that when computing the penalty term at g , Jm
conditionally positive definite (See, Wahba (1990)). Efforts have been done in order
to reduce substantially the computational cost of solving smoothing splines fitting by
47
introducing the concept of H-splines (Luo and Wahba (1997) and Dias (1999)), where
the number of basis functions and act as the smoothing parameters.
A major conceptual problem with spline smoothing is that it is defined implicitly
as the solution to a variational problem rather than as an explicit formula involving
the data values. This difficulty can be resolved, at least approximately, by considering
how the estimate behaves on large data sets. It can be shown from the quadratic
nature of (5.0.3) that g is linear in the observations yi , in the sense that there exists a
weight function H (s, t) such that
n
yi H (s, ti ).
g ( s ) =
(5.0.5)
i =1
It is possible to obtain the asymptotic form of the weight function, and hence an
approximate explicit form of the estimate. For the sake of simplicity consider d = 1,
m = 2 and suppose that the design points have local density f (t) with respect to a
Lesbegue measure on R. Assuming the following conditions, (Silverman (1984)),
1. g H[ a, b].
2. There exists an absolutely continuous distribution function F on [ a, b] such that
Fn F uniformly as n .
3. f = F , 0 < inf[ a,b] f sup[ a,b] f < .
4. The density has bounded first derivative on [ a, b].
5. a(n) = sup[ a,b] | Fn F|, the smoothing parameter depends on n in such a way
that 0 and 1/4 a(n) 0 as n .
In particular, one can assume that the design points are regularly distributed with
density f ; that is, ti = F1 ((i 1/2)/n). Then, sup | Fn F| = (1/2)n1 so that
n4 and 0 for (5) to hold. Thus, as n ,
H (s, t) =
1 1
st
K(
),
f (t) h (t) h (t)
where the kernel function K is given by

K (u) =
1
exp(|u|/ 2) sin(|u|/ 2 + /4),
2
48
and the bandwidth h(t) satisfies
h(t) = 1/4 n1/4 f (t)1/4 .

Based on these formulas, we can see that the spline smoother is approximately a
convolution smoothing method but the data are not convolved with a kernel with
fixed bandwidth, in fact, h varies across the sample.
5.1 Additive Models

The additive model is a generalization of the usual linear regression model and
what has made it so popular for statistical inference is that the linear model is linear in the predictor variables (explanatory variables). Once we have fitted the linear
model we can examine the predictor variables separately, in the absence of interactions. Additive models are also linear in the predictor variables. An additive model
is defined by
p
yi = +
gj (t j ) + i
(5.1.1)
j =1
where t j are the predictor variables and as defined before in section 5, i are uncorrelated error measurements with E[i ] = 0 and Var [i ] = 2 . The functions g j are
unknown but assumed to be smooth functions lying in some metric space. Section
5 describes a general framework for defining and estimating general nonparametric
regression models which includes additive models as a special case. For this, suppose that is the space of the vector predictor t and assume the H is reproducing
kernel in Hilbert space. Hence H has the decomposition
p
H = H0 +
Hk
(5.1.2)
k=1
where H0 is spanned by 1 , . . . , M and Hk has the reproducing kernel Ek (, ), de-
fined in section 5. The space H0 is the space of functions that are not to be penalized
in the optimization. For example, recall equation (5.0.3) and let m = 2 then H0 is the
space of linear functions in t.
5.1. ADDITIVE MODELS
49
The optimization problem becomes: For a given set of predictors t1 , . . . , tn , find

the minimizer of
n
{yi gk (ti )}
i =1
k=0
k ||gk ||2Hk ,
(5.1.3)
k=1
with gk Hk . Then, the theory of reproducing kernel guarantees that a minimizer

exists and has the form
p
Qk c + Tb,
g =
(5.1.4)
k=1
where Qk and T are given in equation (5.0.4) and the vectors c and b are found by
minimizing the finite dimensional penalized least square criterion
p
||y Tb
Qk c|| + k ckT Qk c2.
k=1
(5.1.5)
k=1
This general problem (5.1.4) can potentially be solved by a backfitting type algorithm
as in Hastie and Tibshirani (1990).
Algorithm 5.1.1
(0)
1. Initialize g j = g j
for j = 0, . . . , p.
2. Cycle j = 0, . . . , p, . . . , j = 0, . . . , p, . . .
g j = S j (y g j (t j ))
j6=k
3. Continue (ii) until the individual functions do not change.
where y = (y1 , . . . , yn ), S j = Qk (Qk + k I )1 , for j = 1, . . . , p, and S0 = T (T T T )1 .

One may observe that omitting the constant term in (5.1.1) does not change the
resulting estimates.
An example of gam method is given in Figure 5.1.1
50
0.8
0.8
linea
0.6
0.6
r pre
0.4
dicto
0.2
1.5
0.2
1.0
0.0
0.5
1.0
0.0
0.5
0.5
0.0
0.0
0.5
x
0.4
0.5
x
0.0
1.0
0.0
1.0
1.50.5
0.8
0.8
linea
0.6
r pre
r pre
linea
0.6
0.4
0.4
dicto
dicto
0.2
0.2
r
1.0
0.0
1.0
0.0
0.5
0.5
x
0.0
0.0
0.5
0.5
x
0.0
1.0
0.0
1.0
Figure 5.1.1: True, tensor product, gam non-adaptive and gam adaptive surfaces
5.2 Generalized Cross-Validation Method for Splines nonparametric Regression

Without loss of generality, take d = 1 and m = 2. The solution of (5.0.3) depends strongly on the smoothing parameter. Craven and Wahba (1979) provide an
[k]
automatic data-driven procedure to estimate . For this, let g be the minimizer of

1
(yi g(ti ))2 +
n i
6=k
( g (u))2 du,
the optimization problem with the kth data point left out. Then following Wahbas
notation, the ordinary cross-validation function V0 () is defined as
V0 () =
1 n
[k]
(yk g (tk ))2 ,
n k
=1
(5.2.1)
and the leave-one-out estimate of is the minimizer of V0 (). To proceed, we need to

describe the influence matrix. It is not difficult to show (see Wahba (1990)) that, for
5.2. GENERALIZED CROSS-VALIDATION METHOD FOR SPLINES NONPARAMETRIC REGRES

fixed we have by (5.0.5) that g is linear in the observations yi , that is, in matrix
notation
g = H y.
At this stage, one may think that the computation of this problem is prohibitive but
Craven and Wahba (1979) give us a very useful mathematical identity, which will not
be proved here, but is
[k]
(yk g (tk )) = (yk g (tk ))/(1 hkk (),
(5.2.2)
where hkk () is the kth entry of H . By substituting (5.2.2) into (5.2.1) we obtain a
simplified form of V0 , that is,
V0 () =
1 n
(yk g (tk ))2 /(1 hkk ())2
n k
=1
(5.2.3)
The right hand of (5.2.3) is easier to compute than (5.2.1), however the GCV is even
easier. The generalized cross-validation (GCV) is method for choosing the smoothing
parameter , which is based on leaving-one-out, but it has two advantages. It is easy to
compute and it posses some important theoretical properties the would be impossible
to prove for leaving-one-out, although, as pointed out by Wahba, in many cases the
GCV and leaving-one-out estimates will give similar answers. The GCV function is
defined by
V () =
1
2
1 n
2
2
n ||( I H )y||
,
(
y
g
(
t
))
/
(
1
h
(
))
=
k
k
kk
1
2
n k
tr
(
I
H
]
[
=1
n
(5.2.4)
where h kk () = (1/n)tr ( H ), with tr ( H ) standing for the trace of H . Note that

V () is a weighted version of V0 (). In addition, if hkk () does not depend on k, then
V0 () = V () for all > 0.
It is important to observe that GCV is a predictive mean square error criteria. Note
that by defining the predictive mean square error T () as
T () =
1 n
(Li g Li g)2
n i =1
(5.2.5)
where, Li is the evaluation functional defined in section 4.3, the GCV estimate of is
the minimizer of (5.2.5). Consider the expected value of T (),
E[ T ()] =
1 n
E[(Li g Li g)2 ].
n i
=1
(5.2.6)
52
The GCV theorem Wahba (1990) says that if g is in a reproducing kernel Hilbert space
then there is a sequence of minimizers (n) of EV () that comes close to achieving
the minimum possible value of the expected mean square error, E[ T ()], using (n),
as n . That is, let the expectation inefficiency In be defined as
In =
E[ T ( (n))]
,
E[ T ( )]
where is the minimizer of E[ T ()]. Then, under mild conditions as such the ones
described and discussed by Golub, Heath and Wahba (1979) and Craven and Wahba
(1979), we have In 1 as n .
Figure 5.2.2 shows the scatter plot of the revenue passenger miles flown by commercial airlines in the United States for each year from 1937 to 1960. (This data can be
found in the software). The smoothing parameter was computed by GCV method
through the R function smooth.spline().
25000
30000
airmiles data
15000
0
5000
10000
airmiles
20000
Data
SS
1940
1945
1950
1955
1960
Passengermiles flown by U.S. commercial airlines
Figure 5.2.2: Smoothing spline fitting with smoothing parameter obtained by GCV
method
Chapter 6
Regression splines, P-splines and
H-splines
6.1 Sequentially Adaptive H-splines
In regression splines, the idea is to approximate g by a finite dimensional subspace
of W , a Sobolev space, spanned by basis functions B1 , . . . , BK , K n. That is,
K
g gK =
c j Bj ,
j =1
where the parameter K controls the flexibility of the fitting. A very common choice
for basis functions is the set of cubic B-splines (de Boor, 1978). The B-splines basis
functions provide numerically superior scheme of computation and have the main
feature that each B j has compact support. In practice, it means that we obtain a stable
evaluation of the resulting matrix with entries Bi,j = B j (ti ), for j = 1, . . . , K and i =
1, . . . , n is banded. Unfortunately, the main difficulty when working with regression
splines is to select the number and the positions of a sequence of breakpoints called
knots where the piecewise cubic polynomials are tied to enforce continuity and lower
order continuous derivatives. (See Schumaker (1972) for details. ) Regression splines
are attractive because of their computational scheme where standard linear model
techniques can be applied. But smoothness of the estimate cannot easily be varied
53
CHAPTER 6. REGRESSION SPLINES, P-SPLINES AND H-SPLINES
54
continuously as functions of a single smoothing parameter (Hastie and Tibshirani,

1990). In particular, when = 0 we have the regression spline case, where K is
the parameter that controls the flexibility of the fitting. To exemplify the action of
K on the estimated curve, let us consider an example by simulation with y(t) =
exp(t) sin (t/2) cos (t) + with N (0, .05). The curve estimates were obtained
by least square method with four different numbers of basis functions which are the
cubic B-splines. Figure 6.1.1 shows the effect of varying the number of basis functions
on the estimation of the true curve. Observe that small values of K make smoother
the estimate and hence over smoothing may occur. Large values of K may cause
under-smoothing.
100 obs. from y(x)=-exp(-x)*sin(pi*x/2)*cos(pi*x)+N(0,.025)
0.2
0.0
y
-0.2
-0.4
True
K=4
K=12
K=60
0.5
0.0
1.0
1.5
2.0
2.5
3.0
Figure 6.1.1: Spline least square fittings for different values of K
6.1. SEQUENTIALLY ADAPTIVE H-SPLINES
55
In the smoothing techniques the number of basis functions is chosen to be as large

as the number of observations and then let the choice of the smoothing parameter
contolling the smoothing (Bates and Wahba, 1982). Here a different approach is to be
taken. The H-splines method introduced by Dias (1994) in the case of nonparametric
density estimation, combines ideas from regression splines and smoothing splines
methods by finding the number of basis functions and the smoothing parameter iteratively according to a criterion that is described below. With the point evaluation
functionals Li g = g(ti ) the equation (6.3.2) becomes,
A ( g) =
(yi g(ti ))
i =1
( g )2 .
(6.1.1)
Assume that g gK = iK=1 ci Bi = Xc so that gK HK , where HK denotes the space
of natural cubic splines (NCS) spanned by the basis functions { Bi }iK=1 and X is a n K
matrix with entries Xij = Bi (t j ), for i = 1, . . . , K and j = 1, . . . , n. Then, the numerical

problem is to find a vector c = (c1 , . . . , cK )T that minimizes,
A (c) = ky Xck22 + c T c,
where is K K matrix with entries ij =
Bi (t) Bj (t)dt and y is the vector
(y1 , . . . , yn )T . Standard calculations (de Boor, 1978) provide c as a solution of the
following linear system (X T X + )c = X T y. Note that the linear system now involves K K matrices instead of using n n matrices which is the case of smoothing
splines. Both K and controls the trade off between smoothness and fidelity to the
data. Figure 6.1.2 shows, for > 0, an example of the relationship between K and
. Note that when the number of basis functions increases, the smoothing parameter
decreases to a point and then it increases with K. That is, for large values of K, the
smoothing parameter becomes larger in order to enforce smoothness.
56

Smoothing parameter x knots
0.010
0.005
smoohting paramter
0.015
10
20
30
40
50
knots
Figure 6.1.2: Five thousand replicates of y( x ) = exp( x ) sin(x/2) cos (x ) + .

Based on the facts described previously, the idea is to provide a procedure that estimates the smoothing parameter and the number of basis functions iteratively. Consider the following algorithm.
Algorithm 6.1.1
(1) Let K0 be the initial number of basis functions and fix 0 .
(2) Compute c0 by solving (X T X + 0 )c0 = X T y.
(3) Find which minimizes,
n1 in=1 (yi gK0 (ti ))2
GCV() =
,
1 n1 tr( A())
where A() = X (X T X + )1 X T .
57
(4) Compute gK0 , = A( )y.

(5) Increment the number of basis functions by one and repeat steps (2) to (4) in order to
get gK0 +1, .
(6) For a real number > 0, if a distance d( gK0 , , gK0 +1, ) < , stop the procedure. The
number can be determined empirically according to the particular distance d(, ).
Note that each time the number of basis functions K is incremented by one the
numerator of GCV changes and hence this procedure provides an optimal smoothing
parameter for the estimate gK based on K basis functions.
The aim is to find a criterion able to tell when to stop increasing the number of
basis functions. That is, to find the dimension of the natural cubic spline space where
one is looking for the approximation of the solution of (6.3.2). For this, let us define
the following transformation. Given any function in W22 [ a, b], take
then t g 0 and
g2
R
tg =
,
g2
t g = 1. For any functions f , g W22 [ a, b], define a pseudo distance
closely related to the square of the Hellinger distance,

d2 ( f , g ) =
Z q
tf
t g )2 = 2(1 ( f , g)),
where
( f , g) =
Z q
t f tg =
f 2 g2
R
=
f 2 g2
qR
| f g|
R ,
f 2 g2
is the affinity between f and g. It is not difficult to see that 0 ( f , g) 1, f , g

R
R
W22 [ a, b]. Note that d2 ( f , g) is minimum when ( f , g) = 1, i.e., ( f 2 g2 )1/2 = | f g|
only if | f | + | g| = 0 for some .
Increasing the number of basis functions K by one, the procedure will stop when
gK, gK +1, in the sense of the partial affinity,
| gK, gK +1, |
1,
( gK, , gK +1, ) = qR
g2 g2
K, K +1,
58
where the dependence of on K is omitted for the sake of simplicity. Simulations

were performed in order to verify the behavior of the affinity and the partial affinity. Figure 6.1.3 shows a typical example given by the underlying function y( x ) =
0 98
0 94
0 96
a n y
0.990
0.995
0 92
0.985
0 88
0 90
0 86
0.970
0.975
0.980
partial affinity
1 00
1.000
exp( x ) sin (x/2) cos (x ) + .
10
20
30
knots
40
50
10
20
30
40
50
kno s
Figure 6 1 3 Five thousand replicates of the affinity and the partial affinity for adaptive nonparametric regression using H-splines with the true curve
One may notice that the affinity is a concave function of the number of basis functions (knots) and the partial affinity approaches one quickly Moreover numerical
experiments have shown that the maximum of the affinity and the stabilization of
the partial affinity coincide That means increasing the K arbitrarily not only increases the computational cost but also does not provide the best fitted curve (in the
59
pseudo Hellinger norm).

It would be useful to have the distribution of the affinity between the true curve
and the estimate produced by the adaptive H-splines method. A previous study
(Dias, 1996) showed an empirical unimodal density with support on [0, 1] skewed
to the left suggesting a beta model. To illustrate, five thousand replicates with sample size 20,100,200 and 500 were taken from a test function yi = xi3 + i , where and
50
10
100
20
30
200
40
50
300
1 , . . . , n are i.i.d. N (0, 5).
0.90
0.94
0.98
0.990
0.998
affinity,
n=100
200
100
200
600
300
400
1000
500
affinity,
n=20
0.994
0.994
0.998
0.996
affinity,
n=200
0.998
1.000
affinity,
n=500
Figure 6.1.4: Density estimates of the affinity based on five thousand replicates of the
curve yi = xi3 + i with i N (0, .5). Solid line is a density estimate using beta model
and dotted line is a nonparametric density estimate.
Figure 6.1.4 shows that the empirical affinity distribution (unimodal, skewed to
60
the left with range between 0 and 1), a nonparametric density estimate using kernel
method and a parametric one using a beta model whose parameters were estimated
using method of the moments.
0.8
100 obs. from y(x)=exp(-2*x)*sin(2*pi*x)+N(0,.1)

o
TRUE
H-splines
S-splines
0.6
oo
oo
o
o
o
o
o
o
o
0.4
o
o o
o
o
0.2
o
o
o
oo
oo
0.0
o
o
o
o
o
o
o
o
oo
-0.2
oo o o
o
ooo
o
o
-0.4
oo
o o
o
o
o
o
o
o
o oo o oo oo
o
o
o
oo
oo o
oo
o
oo
o
o
oo
o
o
0.0
0.5
1.0
1.5
2.0
Figure 6.1.5: A comparison between smoothing splines (S-splines) and hybrid splines
(H-splines) methods.
Figure 6.1.5 shows that, in general, H-splines method has similar performance as
smoothing splines. But as mentioned before the H-splines approach solves a linear
system of order K while smoothing splines must have to solve a linear system of
order n K.
6.2. P-SPLINES
61
6.2 P-splines
The basic idea of P-splines proposed by Eilers and Marx (1996) is to use a considerable number of knots and to control the smoothness through a difference penalty
on coefficients of adjacent B-splines. For this, lets consider a simple regression model
y( x ) = g( x ) + , where is a random variable with symmetric distribution with mean
zero and finite variance. Assume that the regression curve g can be well approximate
by a linear combination of, without loss of generality, cubic B-splines, denoted by
B( x ) = B( x; 3). Specifically, Given n data points ( xi , yi ) on a set of K B-splines B j (.),
we take, g( xi ) = Kj=1 a j B j ( xi ). Now, the penalized least square problem becomes to
find a vector of coefficients a = (a1 . . . , aK ) that minimizes:
n
PLS(a) =
i =1
Yi a j B j ( xi )
j =1
o2
Z n K
o
(
a
B
x
)
j j i .
j =1
Following, de Boor (1978), we have that the second derivative

K
a j B j ( xi ; 3)
=h
2 a j B j ( x i ; 1 )
j =1
j =1
where h is the distance between knots and 2 a j = (a j ) = (a j a j1 ) The Psplines method penalizes the higher-order of the finite differences of the coefficients
of adjacent B-splines. That is,
n
i =1
Yi a j B j ( xi )
j =1
o2
(m (a j ))2 .
j = m +1
Eilers and Marx (1996) shows that the difference penalty is a good discrete approximation to the integrated square of the kth derivative and with this this penalty moments of the data are conserved and polynomial regression models occur as limits
for large values of .
Figure 6.2.6 shows a comparison of smooth.spline and P-spline estimates on simulated example.
62
sin(x*2*pi/10)+.5 + N(0,.71)
**
** *
* * **
*
**
*
* *
* ***
*
**
* * * **
*
* * **
*
*
* * *
*
*
*
* ** *
*
* *
*
*
*
*
***
*
*
*
*
*
*
*
*
*
*
*
*
* *
**
*
*
*
** *
*
*
1
y
True
smooth.spline
Pspline
*
*
*
**
* **
** *
*
*
**
* *
*
*
*
**
*
*
*
** *
** *
* **
*
* *
*
* *
*
*
*
**
*
*
*
**
* *
*
**
*
*
* *
*
*
**
*
*
* **
*
*
*
*
*
*
* *
*
* *
*
*
*
*
**
*
*
*
*
*
*
10
Figure 6.2.6: smooth spline and P-spline
6.3
A Bayesian Approach to H-splines
We have seen that there are several methods to estimate non-parametrically an

unknown regression curve g by using splines since the pioneer work of Craven and
Wahba (1979). Kimeldorf and Wahba (1970) and Wahba (1983) gave an attractive
Bayesian interpretation for an estimate g of the unknown curve g. They showed that
g can be viewed as a Bayes estimate of g with respect to a certain prior on the class of
all smooth functions. The Bayesian approach allows one not only to estimate the unknown function, but also to provide error bounds by constructing the corresponding,
Bayesian confidence intervals (Wahba, 1983). In this section we allow two smoothing
parameters as Luo and Wahba (1997) and Dias (1999) did. However, instead of going
through the difficulties of specifying them precisely in an ad-hoc manner, they are
allowed to vary according to prior information. In this way, the procedure becomes
more capable of providing an adequate fit.
6.3. A BAYESIAN APPROACH TO H-SPLINES
63
Suppose we have the following regression model,

y i = g ( ti ) + i
i = 1, . . . , n.
where i s are uncorrelated with a N (0, 2 ). Moreover, assume that the parametric
form of the regression curve g is unknown. Then the likelihood of g given the observations y is,
1
||y g||2 }.
(6.3.1)
22
The Bayesian justification of penalized maximum likelihood is to place a prior density
R
proportional to exp{ 2 ( g )2 } over the space of all smooth functions. (see details
ly ( g) (2 )n/2 exp{
in Silverman and Green (1994) and Kimeldorf and Wahba (1970)). However, an in-
finite dimensional case has a paradox alluded to by Wahba (1983). Silverman (1985)
proposed a finite dimensional Bayesian formulation to avoid the paradoxes and difficulties involved in the infinite dimensional case. For this, let gK, = iK=1 i Bix = XK K
with a knot sequence x placed at order statistics. A complete Bayesian approach
would assign prior distribution to the coefficients of the expansion, to the knot positions, to the number of knots and for 2 . A Bayesian approach to hybrid splines
non-parametric regression assigns priors for gK , K, and 2 . Given a realization
of K the interior knots are placed at order statistics. This well known procedure in
non-parametric regression reduces the computational cost substantially and avoids
trying to solve a difficult problem of optimizing the knot positions. Any other procedure has to take into account the fact that changes in the knot positions might cause
considerable change in the function g (see details in Wahba (1982) for ill-posed problems in splines non-parametric regression). Moreover, in theory the number of basis
functions (which is a linear function of the number of knots) can be as large as the
sample size. But then one has to solve a system of n equations instead of K. An
attempt to keep the computational cost down one might want to have K small as
possible and hence over-smoothing may occur. For any K large enough keeps the
balance between over-smoothing and under-smoothing. Thus the penalized likelihood becomes with g = gK ,
2 n/2
l p ( )
exp{ 2 ||y gK ||2 } exp{

2
2
( gK )2 },
(6.3.2)
64
where gK = gK, = XK K . The reason why we suppress the subindex in gK, will
be explained later in this section. Note that maximization of the penalized likelihood
is equivalent to minimization of (5.0.3). For this proposed Bayesian set up we have a
prior for ( gK , K, , 2 ) of the form,
p( gK , K, , 2 ) = p( gK |K, ) p(K, ) p(2 )
(6.3.3)
= p( gK |K, ) p(|K ) p(K ) p(2 ),

where
p( gK |K, ) exp{
( gK )2 },
2
1
p(2 ) 2 (u+1) exp{v2 },
( )
Z
for u > 0, v > 0,

p (K ) =
exp{ a} aK /K!
,
1 exp{ a}(1 + q )
K = 1, . . . , K
j
where q =
j= K +1 a /j!, and
p(|K ) = (K ) exp {(K )},

with any smooth function of K. It is well known that, for > 0, when the number
of basis functions increases the smoothing parameter decreases to a point and then it
increases with K. That is, for large values of K the smoothing parameter becomes
larger to enforce smoothness. (See details in Dias (1999).). Therefore, functions that
satisfy these requirements are recommended. In particular, a flexible class is given
by (K ) = K b exp(cK ) for suitably chosen hyperparameters b and c. However,
undesirably large values of K can be excluded through fixing K appropriately or be
made unlikely by fixing a accordingly. These large values of K can be controlled by
the hyperparameters a and K of the prior p(K ).
The choice of a is governed by the prior expectation of the structure of the underlying curve such as maxima, minima, inflection points etc. We suggest the reader to
follow some of the rules recommended by Wegman and Wright (1983). These recommendations are based on the assumption of fitting a cubic spline, the most popular
case and are summarized below.
65
1. Extrema should be center in intervals and inflection points should be located

near knot points.
2. No more than one extremum and one inflection point should fall between knots
(because a cubic could not fit more).
3. Knot points should be located at data points.
Note that gK = XK K is completely determined if we know the coefficients K .
Hence, the overall parameter space can be written as countable union of subspaces
K = { : = (K , ) (RK {1, . . . , K } [0, ]2 )} with = (K, , 2 ). Thus, the
posterior is given by
( | y) l p ( | y) p ( ).
(6.3.4)
In order to sample from the posterior ( |y) we have to consider the variation of
dimensionality of this problem. Hence one has to design move types between subspaces K . However, assigning a prior to gK , equivalently to the coefficients K ,
leads to a serious computational difficulty pointed out by Denison, Mallick and Smith
(1998) where a comparative study was developed. They suggest that the least square
estimates for the vector K leads to a non-significant deterioration in performance for
overall curve estimation. Similarly, given a realization of (K, , 2 ), we solve the penalized least square objective function (6.1.1) to obtain the estimates, K = K (y), for
the vector K and consequently we have an estimate g K = XK K . Thus, there is no
prior assigned for this vector of parameters, and so, we write gK = gK, . Having got
K , we approximate the marginal posterior (|y) by the conditional posterior
(|y, K ) l p (|y, K ) p(),
(6.3.5)
with
p() = p(K, , 2 ) = p(|K ) p(K ) p (2 ).
Note that if one assigns independent normal distributions to the parameters K it will
not be difficult to obtain the marginal posterior (|y) and approximations will not
be necessary. However, the results will be very similar.
66

To solve the problem of sampling from the posterior, (|y, K ), Dias and Gamer-
man (2002) used reversible jump methodology (Green (1995)). This technique is beyond the level of this book and it will not be explained here but the interested reader
will find the details of the algorithm in Dias and Gamerman (2002).
In figure 6.3.7 we present a simulated example to verify the estimates provided

by this approach. The final estimate is,
y + (ti ) =
1
100
100
y j (ti ).
j =1
Figure 6.3.8, exhibit approximate Bayesian confidence intervals for the true curve
regression f and it was computed as following. Let y(ti ) and y(ti ) be a particular
model and its estimate provided by this proposed method with i = 1, . . . , n where n
is the sample size. For each i = 1, . . . , n the fitting vectors (y1 (ti ), y2 (ti ), . . . , y100 (ti ))T
form random samples and from those vectors the lower and upper quantiles were
computed in order to obtain the confidence intervals.
Figure 6.3.7 exhibits an example of how useful a Bayesian approach to hybrid

splines non-parametric regression can be. It describes a situation where a prior information tells that the underlying curve has large curvature and the variance of the
error measurements is not too small and the traditional methods of smoothing, e.g.
smoothing splines, might not be able to capture all the structure of the true regression
curve. By using vague but proper priors to the smoothing parameters K and and
for the variance 2 this Bayesian approach provides a much better fitting than the
traditional smooth splines approach does.
67
50 observations from y=exp(t^2/2)*cos(4*pi*t)+N(0,.36)
*
*
*
y
0
* *
*
*
*
*
*
*
*
*
*
*
*
*
*
* *
*
*
*
*
True
SS estimate
Bayesian estimate
*
*
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Figure 6.3.7: Estimation results: a) Bayesian estimate with a = 17 and (K ) = K3

(dotted line); b) (SS) smoothing splines estimate (dashed line). The true regression
function is also plotted (solid line). The SS estimate was computed using the R function smooth.spline from which 4 degrees of freedom were obtained and was computed by GCV.
Figure 6.3.8 shows one hundred curves sampling from the posterior (after burnin) and approximate 95% Bayesian confidence interval for the regression curve g(t) =
exp(t2 /2) cos(4t) with t [0, ]. On the right panel of this figure we see the curve
estimate which is an approximation for the posterior mean and the percentiles curves
2.5% and 97.5%.
68
Average
.025 quantile
.975 quantile
-0.5
-1.0
-1.5
-1.5
-0.5
0.0
0.5
1.0
95% "Confidence" Interval
-1.0
0.0
0.5
1.0
The last 100 curves sampled
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Figure 6.3.8: One hundred estimates of the curve 6.3.7 and a Bayesian confidence
interval for the regression curve g(t) = exp(t2 /2) cos(4t) with t [0, ].
Chapter 7
Final Comments
Compared to parametric techniques nonparametric modeling has more flexibility

since it allows one to choose from an infinite dimensional class of functions where
the underlying regression curve is assumed to belong. In general, this type of choice
depends on the unknown smoothness of the true curve. But for most of the cases one
can assume mild restrictions such that a regression curve has an absolutely continuous first derivative and a square integrable second derivative. Nevertheless, nonparametric estimators are less efficient than the parametric ones when the parametric
model is valid. For many parametric estimators the mean square error goes to zero
with rate of n1 , while nonparametric estimators have rate of n , [0, 1], and
depends on the smoothness of the underlying curve. When the postulate parametric model is not valid, many parametric estimators cannot have, ad hoc, rate n1 . In
fact, those estimators will not converge to the true curve. One of the advantages of
the adaptive basis functions procedures, e.g., H-splines methods is the ability to vary
the amount of smoothing in response to the inhomogeneous curvature of the true
functions at different locations. Those methods have been very successful in capturing the structure of the unknown function. In general, nonparametric estimators are
good candidates when one does not know the form of the underlying curve.
69
70
CHAPTER 7. FINAL COMMENTS
Bibliography
Bates, D. and Wahba, G. (1982). Computational Methods for Generalized Cross-Validation
with large data sets, Academic Press, London.
Box, G. E. P., Hunter, W. G. and Hunter, J. S. (1978). Statistics for Experiments: An
Introduction to Design, Data Analysis, and Model Building, John Wiley and Sons
(New York, Chichester).
Cleveland, W. S. (1979). Robust locally weighted regression and smoothing scatterplots, J. Amer. Statist. Assoc. 74(368): 829836.
Craven, P. and Wahba, G. (1979). Smoothing noisy data with spline functions, Numerische Mathematik 31: 377403.
de Boor, C. (1978). A Practical Guide to Splines, Springer Verlag, New York.
Denison, D. G. T., Mallick, B. K. and Smith, A. F. M. (1998). Automatic bayesian curve
fitting, Journal of the Royal Statistical Society B 60: 363377.
Dias, R. (1994). Density estimation via h-splines, University of Wisconsin-Madison.
Ph.D. dissertation.
Dias, R. (1996). Sequential adaptive nonparametric regression via H-splines. Technical Report RP 43/96, University of Campinas, June 1996. Submitted.
Dias, R. (1998). Density estimation via hybrid splines, Journal of Statistical Computation
and Simulation 60: 277294.
71
BIBLIOGRAPHY
72
Dias, R. (1999). Sequential adaptive non parametric regression via H-splines, Communications in Statistics: Computations and Simulations 28: 501515.
Dias, R. and Gamerman, D. (2002). A Bayesian approach to hybrid splines nonparametric regression, Journal of Statistical Computation and Simulation. 72(4): 285297.
Eilers, P. H. C. and Marx, B. D. (1996). Flexible smoothing with B-splines and penalties, Statist. Sci. 11(2): 89121. With comments and a rejoinder by the authors.
Golub, G. H., Heath, M. and Wahba, G. (1979). Generalized cross-validation as a
method for choosing a good ridge parameter, Technometrics 21(2): 215223.
Good, I. J. and Gaskins, R. A. (1971). Nonparametric roughness penalties for probability densities, Biometrika 58: 255277.
Green, P. J. (1995). Reversible jump Markov Chain Monte Carlo computation and
bayesian model determination, Biometrika 82: 711732.
Gu, C. (1993). Smoothing spline density estimation: A dimensionless automatic algorithm, J. of the Amer. Statl. Assn. 88: 495504.
Gu, C. and Qiu, C. (1993). Smoothing spline density estimation:theory, Ann. of Statistics 21: 217234.
Hardle, W. (1990). Smoothing Techniques With Implementation in S, Springer-Verlag
(Berlin, New York).
Hastie, T. J. and Tibshirani, R. J. (1990). Generalized Additive Models, Chapman and
Hall.
Kimeldorf, G. S. and Wahba, G. (1970). A correspondence between Bayesian estimation on stochastic processes and smoothing by splines, The Annals of Mathematical Statistics 41: 495502.
Kooperberg, C. and Stone, C. J. (1991). A study of logspline density estimation, Computational Statistics and Data Analyis 12: 327347.
BIBLIOGRAPHY
73
Luo, Z. and Wahba, G. (1997). Hybrid adaptive splines, Journal of the American Statistical Association 92: 107116.
Nadaraya, E. A. (1964). On estimating regression, Theory of probability and its applications 10: 186190.
OSullivan, F. (1988). Fast computation of fully automated log-density and log-hazard
estimators, SIAM J. on Scientific and Statl. Computing 9: 363379.
Pagan, A. and Ullah, A. (1999). Nonparametric econometrics, Cambridge University
Press, Cambridge.
Parzen, E. (1962). On estimation of a probability density function and mode, Ann. of
Mathematical Stat. 33: 10651076.
Prakasa-Rao, B. L. S. (1983). Nonparametric Functional Estimation, Academic Press
(Duluth, London).
Schumaker, L. L. (1972). Spline Functions and Aproximation theory, Birkhauser.
Schumaker, L. L. (1981). Spline Functions: Basic Theory, WileyISci:NJ.
Scott, D. W. (1992). Multivariate Density Estimation. Theory, Practice, and Visualization,
John Wiley and Sons (New York, Chichester).
Silverman, B. W. (1982). On the estimation of a probability density function by the
maximum penalized likelihood method, Ann. of Statistics 10: 795810.
Silverman, B. W. (1984). Spline smoothing: The equivalent variable kernel method,
Ann. of Statistics 12: 898916.
Silverman, B. W. (1985). Some aspects of the spline smoothing approach to nonparametric regression curve fitting, Journal of the Royal Statistical Society, Series B,
Methodological 47: 121.
Silverman, B. W. (1986). Density Estimation for Statistics and Data Analysis, Chapman
and Hall (London).
BIBLIOGRAPHY
74
Silverman, B. W. and Green, P. J. (1994). Nonparametric Regression and Generalized

Linear Models, Chapman and Hall (London).
Stone, C. J. (1990). Large-sample inference for log-spline models, Ann. of Statistics
18: 717741.
Stone, C. J. and Koo, C.-Y. (1985). Logspline density estimation, Contemporary Mathematics pp. 1158.
Thompson, J. R. and Tapia, R. A. (1990). Nonparametric Function Estimation, Modeling
and Simulation, SIAM:PA.
Wahba, G. (1982). Constrained regularization for ill posed linear operator equations,
with applications in meteorology and medicine, in S. S. Gupta and J. O. Berger
(eds), Statistical Decision Theory and Related Topics III, in two volumes, Vol. 2, Academic:NY:Lnd, pp. 383418.
Wahba, G. (1983). Bayesian confidence intervals for the cross-validated smoothing
spline, JRSS-B, Methodological 45: 133150.
Wahba, G. (1990). Spline Models for Observational Data, SIAM:PA.
Watson, G. S. (1964). Smooth regression analysis, Sankya A 26: 359372.
Wegman, E., J. and Wright, I. W. (1983). Splines in statistics, Journal of the American
Statistical Association 78: 351365.

SDV

Uploaded by

Copyright:

Available Formats

SDV

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

SDV

Uploaded by

Copyright:

Available Formats

Nonparametric Estimation: Smoothing and

Kernel Density Estimation . . . . . . . . . . . . . . . . . . . . . . . . . .

The Nearest Neighbor Method . . . . . . . . . . . . . . . . . . .

Some Statistical Results for Kernel Density Estimation . . . . . . . . . .

Reference to a Standard Distribution . . . . . . . . . . . . . . . .

Maximum likelihood Cross-Validation . . . . . . . . . . . . . . .

Orthogonal series estimators . . . . . . . . . . . . . . . . . . . . . . . . .

3 Kernel nonparametric Regression Method

k-Nearest Neighbor (k-NN) . . . . . . . . . . . . . . . . . . . . . . . . .

Local Polynomial Regression: LOWESS . . . . . . . . . . . . . . . . . .

Penalized Maximum Likelihood Estimation . . . . . . . . . . . . . . . .

Computing Penalized Log-Likelihood Density Estimates . . . .

Acquiring the Taste . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Logspline Density Estimation . . . . . . . . . . . . . . . . . . . . . . . .

Splines Density Estimation: A Dimensionless Approach . . . . . . . . .

The thin-plate spline on Rd

Generalized Cross-Validation Method for Splines nonparametric Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Regression splines, P-splines and H-splines

Sequentially Adaptive H-splines . . . . . . . . . . . . . . . . . . . . . .

A Bayesian Approach to H-splines

2.2.4 Effect of the smoothing parameter k on the estimates . . . . . . . . . .

2.4.5 Comparison of two bandwidths, (the sample standard deviation)

2.5.6 Effect of the smoothing parameter K on the orthogonal series method

3.0.1 Effect of bandwiths on Nadaraya-Watson kernel . . . . . . . . . . . . .

3.1.2 Effect of the smoothing parameter k on the k-NN regression estimates.

3.2.3 Effect of the smoothing parameter using LOWESS method. . . . . . . .

4.1.1 Basis Functions with 6 knots placed at x . . . . . . . . . . . . . . . . .

4.1.2 Basis Functions with 6 knots placed at x . . . . . . . . . . . . . . . . .

4.2.3 logspline density estimation for .5*N(0,1)+.5*N(5,1) . . . . . . . . . . .

4.3.4 Histogram, SSDE, Kernel and Logspline density estimates . . . . . . .

5.2.2 Smoothing spline fitting with smoothing parameter obtained by GCV

6.1.2 Five thousand replicates of y( x ) = exp( x ) sin (x/2) cos (x ) + . . .

6.1.4 Density estimates of the affinity based on five thousand replicates of

6.1.5 A comparison between smoothing splines (S-splines) and hybrid splines

6.2.6 smooth spline and P-spline . . . . . . . . . . . . . . . . . . . . . . . . .

6.3.7 Estimation results: a) Bayesian estimate with a = 17 and (K ) = K3

as in polynomial regression x j (t) = t j1 , then we are dealing with a linear parametric

niques known as nonparametric regression, for example, allows great flexibility in

CHAPTER 2. KERNEL ESTIMATION

where f X is the density of X.

with [h/2, h/2). Thus, we arrive at the following density estimate:

2.1. THE HISTOGRAM

j = 1, . . . , k. Now, take n j = in=1 I (Xi Cj ), where the function I ( x A) is defined

and, kj=1 n j = n. Then,

CHAPTER 2. KERNEL ESTIMATION

2.2 Kernel Density Estimation

and define the estimator,

2.2. KERNEL DENSITY ESTIMATION

xxxxxx xx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxx

In other words K ( x ) is a probability density function, as for instance, the Gaussian

CHAPTER 2. KERNEL ESTIMATION

2.2. KERNEL DENSITY ESTIMATION

Histogram of income data

CHAPTER 2. KERNEL ESTIMATION

2.2.1 The Nearest Neighbor Method

where n is the sample size and, typically, k is chosen to be proportional to n1/2 .

2.2. KERNEL DENSITY ESTIMATION

100 obs. from N(0.5,0.1)

Figure 2.2.4: Effect of the smoothing parameter k on the estimates

4.2.3 logspline density estimation for .5N(0,1)+.5N(5,1) . . . . . . . . . . .