A Tutorial On Support Vector Regression

Statistics and Computing 14: 199–222, 2004
C 2004 Kluwer Academic Publishers. Manufactured in The Netherlands.
A tutorial on support vector regression∗

ALEX J. SMOLA and BERNHARD SCHÖLKOPF
RSISE, Australian National University, Canberra 0200, Australia
Alex.Smola@anu.edu.au
Max-Planck-Institut für biologische Kybernetik, 72076 Tübingen, Germany
Bernhard.Schoelkopf@tuebingen.mpg.de
Received July 2002 and accepted November 2003
In this tutorial we give an overview of the basic ideas underlying Support Vector (SV) machines for
function estimation. Furthermore, we include a summary of currently used algorithms for training
SV machines, covering both the quadratic (or convex) programming part and advanced methods for
dealing with large datasets. Finally, we mention some modifications and extensions that have been
applied to the standard SV algorithm, and discuss the aspect of regularization from a SV perspective.
Keywords: machine learning, support vector machines, regression estimation
1. Introduction (Vapnik and Lerner 1963, Vapnik and Chervonenkis 1964). As

such, it is firmly grounded in the framework of statistical learn-
The purpose of this paper is twofold. It should serve as a self- ing theory, or VC theory, which has been developed over the last
contained introduction to Support Vector regression for readers three decades by Vapnik and Chervonenkis (1974) and Vapnik
new to this rapidly developing field of research.1 On the other (1982, 1995). In a nutshell, VC theory characterizes properties
hand, it attempts to give an overview of recent developments in of learning machines which enable them to generalize well to
the field. unseen data.
To this end, we decided to organize the essay as follows. In its present form, the SV machine was largely developed
We start by giving a brief overview of the basic techniques in at AT&T Bell Laboratories by Vapnik and co-workers (Boser,
Sections 1, 2 and 3, plus a short summary with a number of Guyon and Vapnik 1992, Guyon, Boser and Vapnik 1993, Cortes
figures and diagrams in Section 4. Section 5 reviews current and Vapnik, 1995, Schölkopf, Burges and Vapnik 1995, 1996,
algorithmic techniques used for actually implementing SV Vapnik, Golowich and Smola 1997). Due to this industrial con-
machines. This may be of most interest for practitioners. text, SV research has up to date had a sound orientation towards
The following section covers more advanced topics such as real-world applications. Initial work focused on OCR (optical
extensions of the basic SV algorithm, connections between SV character recognition). Within a short period of time, SV clas-
machines and regularization and briefly mentions methods for sifiers became competitive with the best available systems for
carrying out model selection. We conclude with a discussion both OCR and object recognition tasks (Schölkopf, Burges and
of open questions and problems and current directions of SV Vapnik 1996, 1998a, Blanz et al. 1996, Schölkopf 1997). A
research. Most of the results presented in this review paper comprehensive tutorial on SV classifiers has been published by
already have been published elsewhere, but the comprehensive Burges (1998). But also in regression and time series predic-
presentations and some details are new. tion applications, excellent performances were soon obtained
(Müller et al. 1997, Drucker et al. 1997, Stitson et al. 1999,
Mattera and Haykin 1999). A snapshot of the state of the art
1.1. Historic background in SV learning was recently taken at the annual Neural In-
formation Processing Systems conference (Schölkopf, Burges,
The SV algorithm is a nonlinear generalization of the Gener- and Smola 1999a). SV learning has now evolved into an active
alized Portrait algorithm developed in Russia in the sixties2 area of research. Moreover, it is in the process of entering the
∗An extended version of this paper is available as NeuroCOLT Technical Report standard methods toolbox of machine learning (Haykin 1998,
TR-98-030. Cherkassky and Mulier 1998, Hearst et al. 1998). Schölkopf and
0960-3174
C 2004 Kluwer Academic Publishers
200 Smola and Schölkopf
Smola (2002) contains a more in-depth overview of SVM regres-
sion. Additionally, Cristianini and Shawe-Taylor (2000) and Her-
brich (2002) provide further details on kernels in the context of
classification.
1.2. The basic idea

Suppose we are given training data {(x1 , y1 ), . . . , (x , y )} ⊂
X × R, where X denotes the space of the input patterns (e.g. Fig. 1. The soft margin loss setting for a linear SVM (from Schölkopf
X = Rd ). These might be, for instance, exchange rates for some and Smola, 2002)
currency measured at subsequent days together with correspond-
ing econometric indicators. In ε-SV regression (Vapnik 1995), Figure 1 depicts the situation graphically. Only the points outside
our goal is to find a function f (x) that has at most ε deviation the shaded region contribute to the cost insofar, as the deviations
from the actually obtained targets yi for all the training data, and are penalized in a linear fashion. It turns out that in most cases
at the same time is as flat as possible. In other words, we do not the optimization problem (3) can be solved more easily in its dual
care about errors as long as they are less than ε, but will not formulation.4 Moreover, as we will see in Section 2, the dual for-
accept any deviation larger than this. This may be important if mulation provides the key for extending SV machine to nonlinear
you want to be sure not to lose more than ε money when dealing functions. Hence we will use a standard dualization method uti-
with exchange rates, for instance. lizing Lagrange multipliers, as described in e.g. Fletcher (1989).
For pedagogical reasons, we begin by describing the case of
linear functions f , taking the form 1.3. Dual problem and quadratic programs
f (x) = w, x + b with w ∈ X , b ∈ R (1) The key idea is to construct a Lagrange function from the ob-
jective function (it will be called the primal objective function
where · , · denotes the dot product in X . Flatness in the case
in the rest of this article) and the corresponding constraints, by
of (1) means that one seeks a small w. One way to ensure this is
introducing a dual set of variables. It can be shown that this
to minimize the norm,3 i.e. w2 = w, w. We can write this
function has a saddle point with respect to the primal and dual
problem as a convex optimization problem:
variables at the solution. For details see e.g. Mangasarian (1969),
minimize 1
w2 McCormick (1983), and Vanderbei (1997) and the explanations

2
yi − w, xi − b ≤ ε (2) in Section 5.2. We proceed as follows:
subject to
w, xi + b − yi ≤ ε 1
L := w2 + C (ξi + ξi∗ ) − (ηi ξi + ηi∗ ξi∗ )
2
The tacit assumption in (2) was that such a function f actually i=1 i=1
exists that approximates all pairs (xi , yi ) with ε precision, or in

other words, that the convex optimization problem is feasible. − αi (ε + ξi − yi + w, xi + b)
Sometimes, however, this may not be the case, or we also may i=1
want to allow for some errors. Analogously to the “soft mar-

gin” loss function (Bennett and Mangasarian 1992) which was − αi∗ (ε + ξi∗ + yi − w, xi − b) (5)
used in SV machines by Cortes and Vapnik (1995), one can in- i=1
troduce slack variables ξi , ξi∗ to cope with otherwise infeasible Here L is the Lagrangian and ηi , ηi∗ , αi , αi∗ are Lagrange multi-
constraints of the optimization problem (2). Hence we arrive at pliers. Hence the dual variables in (5) have to satisfy positivity
the formulation stated in Vapnik (1995). constraints, i.e.
1 (∗) (∗)
αi , ηi ≥ 0. (6)
minimize w2 + C (ξi + ξi∗ )
Note that by αi , we refer to αi and αi∗ .
2 (∗)
 i=1
 yi − w, xi − b ≤ ε + ξi
 (3) It follows from the saddle point condition that the partial
subject to w, xi + b − yi ≤ ε + ξi ∗ derivatives of L with respect to the primal variables (w, b, ξi , ξi∗ )

 have to vanish for optimality.
ξi , ξi∗ ≥0

The constant C > 0 determines the trade-off between the flat- ∂b L = (αi∗ − αi ) = 0 (7)
ness of f and the amount up to which deviations larger than i=1
ε are tolerated. This corresponds to dealing with a so called

ε-insensitive loss function |ξ |ε described by ∂w L = w − (αi − αi∗ )xi = 0 (8)

0 if |ξ | ≤ ε i=1
|ξ |ε := (4) (∗) (∗)
|ξ | − ε otherwise. ∂ξ (∗) L = C − αi − ηi = 0 (9)
i
A tutorial on support vector regression 201
Substituting (7), (8), and (9) into (5) yields the dual optimization In conjunction with an analogous analysis on αi∗ we have
problem. max{−ε + yi − w, xi | αi < C or αi∗ > 0} ≤ b ≤
 (16)

 1 min{−ε + yi − w, xi | αi > 0 or αi∗ < C}

 − (αi − αi∗ )(α j − α ∗j )xi , x j
 2 (∗)
maximize i, j=1 If some αi ∈ (0, C) the inequalities become equalities. See

 also Keerthi et al. (2001) for further means of choosing b.

 (αi + αi∗ ) + yi (αi − αi∗ )
 −ε (10) Another way of computing b will be discussed in the context
i=1 i=1

of interior point optimization (cf. Section 5). There b turns out
subject to (αi − αi∗ ) = 0 and αi , αi∗ ∈ [0, C] to be a by-product of the optimization process. Further consid-
i=1 erations shall be deferred to the corresponding section. See also
Keerthi et al. (1999) for further methods to compute the constant
In deriving (10) we already eliminated the dual variables ηi , ηi∗ offset.
(∗)
through condition (9) which can be reformulated as ηi = C − A final note has to be made regarding the sparsity of the SV
(∗)
αi . Equation (8) can be rewritten as follows expansion. From (12) it follows that only for | f (xi ) − yi | ≥ ε
the Lagrange multipliers may be nonzero, or in other words, for

w= (αi −αi∗ )xi , thus f (x) = (αi −αi∗ )xi , x + b. (11) all samples inside the ε–tube (i.e. the shaded region in Fig. 1)
i=1 i=1 the αi , αi∗ vanish: for | f (xi ) − yi | < ε the second factor in
(12) is nonzero, hence αi , αi∗ has to be zero such that the KKT
This is the so-called Support Vector expansion, i.e. w can be conditions are satisfied. Therefore we have a sparse expansion
completely described as a linear combination of the training of w in terms of xi (i.e. we do not need all xi to describe w). The
patterns xi . In a sense, the complexity of a function’s represen- examples that come with nonvanishing coefficients are called
tation by SVs is independent of the dimensionality of the input Support Vectors.
space X , and depends only on the number of SVs.
Moreover, note that the complete algorithm can be described
in terms of dot products between the data. Even when evalu- 2. Kernels
ating f (x) we need not compute w explicitly. These observa-
tions will come in handy for the formulation of a nonlinear 2.1. Nonlinearity by preprocessing
extension. The next step is to make the SV algorithm nonlinear. This, for
instance, could be achieved by simply preprocessing the training
patterns xi by a map : X → F into some feature space F,
1.4. Computing b as described in Aizerman, Braverman and Rozonoér (1964) and
So far we neglected the issue of computing b. The latter can be Nilsson (1965) and then applying the standard SV regression
done by exploiting the so called Karush–Kuhn–Tucker (KKT) algorithm. Let us have a brief look at an example given in Vapnik
conditions (Karush 1939, Kuhn and Tucker 1951). These state (1995).
that at the point of the solution the product between dual variables
and constraints has to vanish. Example 1 (Quadratic features in√ R2 ). Consider the map :
R → R with (x1 , x2 ) = (x1 , 2x1 x2 , x22 ). It is understood
2 3 2
αi (ε + ξi − yi + w, xi + b) = 0 that the subscripts in this case refer to the components of x ∈ R2 .

(12) Training a linear SV machine on the preprocessed features would
αi∗ (ε + ξi∗ + yi − w, xi − b) = 0 yield a quadratic function.
and While this approach seems reasonable in the particular ex-

ample above, it can easily become computationally infeasible
(C − αi )ξi = 0 for both polynomial features of higher order and higher di-
(13) mensionality, as the number of different monomial features
(C − αi∗ )ξi∗ = 0. of degree p is (d+ pp−1), where d = dim(X ). Typical values
for OCR tasks (with good performance) (Schölkopf, Burges
This allows us to make several useful conclusions. Firstly only
(∗) and Vapnik 1995, Schölkopf et al. 1997, Vapnik 1995) are
samples (xi , yi ) with corresponding αi = C lie outside the ε- p = 7, d = 28 · 28 = 784, corresponding to approximately
∗
insensitive tube. Secondly αi αi = 0, i.e. there can never be a set 3.7 · 1016 features.
of dual variables αi , αi∗ which are both simultaneously nonzero.
This allows us to conclude that
2.2. Implicit mapping via kernels
ε − yi + w, xi + b ≥ 0 and ξi = 0 if αi < C (14)
Clearly this approach is not feasible and we have to find a com-
ε − yi + w, xi + b ≤ 0 if αi > 0 (15) putationally cheaper way. The key observation (Boser, Guyon
and Vapnik 1992) is that for the feature map of example 2.1 we condition (Schölkopf, Burges and Smola 1999a). In the follow-
have ing we will call such functions k admissible SV kernels.
2 √ 2 √ 2
x1 , 2x1 x2 , x22 , x 1 , 2x 1 x 2 , x 2 = x, x 2 . (17) Corollary 3 (Positive linear combinations of kernels). Denote
As noted in the previous section, the SV algorithm only depends by k1 , k2 admissible SV kernels and c1 , c2 ≥ 0 then
on dot products between patterns xi . Hence it suffices to know k(x, x ) := c1 k1 (x, x ) + c2 k2 (x, x ) (22)
k(x, x ) := (x), (x ) rather than explicitly which allows
us to restate the SV optimization problem: is an admissible kernel. This follows directly from (21) by virtue
 of the linearity of integrals.


 1
−
 (αi − αi∗ )(α j − α ∗j )k(xi , x j )
2 i, j=1 More generally, one can show that the set of admissible ker-
maximize nels forms a convex cone, closed in the topology of pointwise



 −ε (α + α ∗
) + yi (αi − αi∗ ) (18) convergence (Berg, Christensen and Ressel 1984).
 i i
i=1 i=1

Corollary 4 (Integrals of kernels). Let s(x, x ) be a function
subject to (αi − αi∗ ) =0 and αi , αi∗ ∈ [0, C] on X × X such that
i=1

Likewise the expansion of f (11) may be written as k(x, x ) := s(x, z)s(x , z) dz (23)
X

w= (αi − αi∗ )(xi ) and exists. Then k is an admissible SV kernel.
i=1
This can be shown directly from (21) and (23) by rearranging the

f (x) = (αi − αi∗ )k(xi , x) + b. (19) order of integration. We now state a necessary and sufficient con-
i=1 dition for translation invariant kernels, i.e. k(x, x ) := k(x − x )
The difference to the linear case is that w is no longer given ex- as derived in Smola, Schölkopf and Müller (1998c).
plicitly. Also note that in the nonlinear setting, the optimization
problem corresponds to finding the flattest function in feature Theorem 5 (Products of kernels). Denote by k1 and k2 admis-
space, not in input space. sible SV kernels then
k(x, x ) := k1 (x, x )k2 (x, x ) (24)
2.3. Conditions for kernels is an admissible kernel.

The question that arises now is, which functions k(x, x ) corre-
This can be seen by an application of the “expansion part” of
spond to a dot product in some feature space F. The following
Mercer’s theorem to the kernels
k1 and k2 and observing that
theorem characterizes these functions (defined on X ).
each term in the double sum i, j λi1 λ2j ψi1 (x)ψi1 (x )ψ 2j (x)ψ 2j (x )
gives rise to a positive coefficient when checking (21).
Theorem 2 (Mercer 1909). Suppose k ∈ L ∞ (X 2 ) such that
the integral operator Tk : L 2 (X ) → L 2 (X ), Theorem 6 (Smola, Schölkopf and Müller 1998c). A transla-

tion invariant kernel k(x, x ) = k(x − x ) is an admissible SV
Tk f (·) := k(·, x) f (x)dµ(x) (20)
X kernels if and only if the Fourier transform

is positive (here µ denotes a measure on X with µ(X ) finite F[k](ω) = (2π )− 2
d
e−iω,x k(x)d x (25)
and supp(µ) = X ). Let ψ j ∈ L 2 (X ) be the eigenfunction of Tk X
associated with the eigenvalue λ j = 0 and normalized such that is nonnegative.
ψ j L 2 = 1 and let ψ j denote its complex conjugate. Then
1. (λ j (T )) j ∈ 1 . We will give a proof and some additional explanations to this
theorem in Section 7. It follows from interpolation theory
2. k(x, x ) = j∈N λ j ψ j (x)ψ j (x ) holds for almost all (x, x ),
where the series converges absolutely and uniformly for al- (Micchelli 1986) and the theory of regularization networks
most all (x, x ). (Girosi, Jones and Poggio 1993). For kernels of the dot-product
type, i.e. k(x, x ) = k(x, x ), there exist sufficient conditions
Less formally speaking this theorem means that if for being admissible.

k(x, x ) f (x) f (x ) d xd x ≥ 0 for all f ∈ L 2 (X ) (21) Theorem 7 (Burges 1999). Any kernel of dot-product type
X ×X k(x, x ) = k(x, x ) has to satisfy
holds we can write k(x, x ) as a dot product in some feature k(ξ ) ≥ 0, ∂ξ k(ξ ) ≥ 0 and ∂ξ k(ξ ) + ξ ∂ξ2 k(ξ ) ≥ 0 (26)
space. From this condition we can conclude some simple rules
for compositions of kernels, which then also satisfy Mercer’s for any ξ ≥ 0 in order to be an admissible SV kernel.
Note that the conditions in Theorem 7 are only necessary but B-splines of order 2n + 1, defined by the 2n + 1 convolution of
not sufficient. The rules stated above can be useful tools for the unit inverval, are also admissible. We shall postpone further
practitioners both for checking whether a kernel is an admissible considerations to Section 7 where the connection to regulariza-
SV kernel and for actually constructing new kernels. The general tion operators will be pointed out in more detail.
case is given by the following theorem.
Theorem 8 (Schoenberg 1942). A kernel of dot-product type

3. Cost functions
k(x, x ) = k(x, x ) defined on an infinite dimensional Hilbert
So far the SV algorithm for regression may seem rather strange
space, with a power series expansion
and hardly related to other existing methods of function esti-

∞
mation (e.g. Huber 1981, Stone 1985, Härdle 1990, Hastie and
k(t) = an t n (27)
n=0
Tibshirani 1990, Wahba 1990). However, once cast into a more
standard mathematical notation, we will observe the connec-
is admissible if and only if all an ≥ 0. tions to previous work. For the sake of simplicity we will, again,
only consider the linear case, as extensions to the nonlinear one
A slightly weaker condition applies for finite dimensional are straightforward by using the kernel method described in the
spaces. For further details see Berg, Christensen and Ressel previous chapter.
(1984) and Smola, Óvári and Williamson (2001).
3.1. The risk functional

2.4. Examples
Let us for a moment go back to the case of Section 1.2. There, we
In Schölkopf, Smola and Müller (1998b) it has been shown, by had some training data X := {(x1 , y1 ), . . . , (x , y )} ⊂ X × R.
explicitly computing the mapping, that homogeneous polyno- We will assume now, that this training set has been drawn iid
mial kernels k with p ∈ N and (independent and identically distributed) from some probabil-
ity distribution P(x, y). Our goal will be to find a function f
k(x, x ) = x, x p (28)
minimizing the expected risk (cf. Vapnik 1982)
are suitable SV kernels (cf. Poggio 1975). From this observation
one can conclude immediately (Boser, Guyon and Vapnik 1992, R[ f ] = c(x, y, f (x))d P(x, y) (33)
Vapnik 1995) that kernels of the type
(c(x, y, f (x)) denotes a cost function determining how we will
k(x, x ) = (x, x + c) p (29) penalize estimation errors) based on the empirical data X. Given
i.e. inhomogeneous polynomial kernels with p ∈ N, c ≥ 0 are that we do not know the distribution P(x, y) we can only use
admissible, too: rewrite k as a sum of homogeneous kernels and X for estimating a function f that minimizes R[ f ]. A possi-
apply Corollary 3. Another kernel, that might seem appealing ble approximation consists in replacing the integration by the
due to its resemblance to Neural Networks is the hyperbolic empirical estimate, to get the so called empirical risk functional
tangent kernel
1
Remp [ f ] := c(xi , yi , f (xi )). (34)
k(x, x ) = tanh(ϑ + κx, x ). (30) i=1
By applying Theorem 8 one can check that this kernel does A first attempt would be to find the empirical risk minimizer
not actually satisfy Mercer’s condition (Ovari 2000). Curiously, f 0 := argmin f ∈H Remp [ f ] for some function class H . However,
the kernel has been successfully used in practice; cf. Scholkopf if H is very rich, i.e. its “capacity” is very high, as for instance
(1997) for a discussion of the reasons. when dealing with few data in very high-dimensional spaces,
Translation invariant kernels k(x, x ) = k(x − x ) are this may not be a good idea, as it will lead to overfitting and thus
quite widespread. It was shown in Aizerman, Braverman and bad generalization properties. Hence one should add a capacity
Rozonoér (1964), Micchelli (1986) and Boser, Guyon and Vap- control term, in the SV case w2 , which leads to the regularized
nik (1992) that risk functional (Tikhonov and Arsenin 1977, Morozov 1984,
x−x 2 Vapnik 1982)
k(x, x ) = e− 2σ 2 (31)
λ
is an admissible SV kernel. Moreover one can show (Smola Rreg [ f ] := Remp [ f ] + w2 (35)
2
1996, Vapnik, Golowich and Smola 1997) that (1 X denotes the
indicator function on the set X and ⊗ the convolution operation) where λ > 0 is a so called regularization constant. Many
algorithms like regularization networks (Girosi, Jones and

k
k(x, x ) = B2n+1 (x − x ) with Bk := 1[− 12 , 12 ] (32) Poggio 1993) or neural networks with weight decay networks
i=1 (e.g. Bishop 1995) minimize an expression similar to (35).
3.2. Maximum likelihood and density models However, the cost function resulting from this reasoning might
be nonconvex. In this case one would have to find a convex
The standard setting in the SV case is, as already mentioned in
proxy in order to deal with the situation efficiently (i.e. to find
Section 1.2, the ε-insensitive loss
an efficient implementation of the corresponding optimization
c(x, y, f (x)) = |y − f (x)|ε . (36) problem).
If, on the other hand, we are given a specific cost function from
It is straightforward to show that minimizing (35) with the par-
a real world problem, one should try to find as close a proxy to
ticular loss function of (36) is equivalent to minimizing (3), the
this cost function as possible, as it is the performance wrt. this
only difference being that C = 1/(λ).
p particular cost function that matters ultimately.
Loss functions such like |y − f (x)|ε with p > 1 may not
Table 1 contains an overview over some common density
be desirable, as the superlinear increase leads to a loss of the
models and the corresponding loss functions as defined by
robustness properties of the estimator (Huber 1981): in those
(37).
cases the derivative of the cost function grows without bound.
The only requirement we will impose on c(x, y, f (x)) in the
For p < 1, on the other hand, c becomes nonconvex.
following is that for fixed x and y we have convexity in f (x).
For the case of c(x, y, f (x)) = (y − f (x))2 we recover the
This requirement is made, as we want to ensure the existence and
least mean squares fit approach, which, unlike the standard SV
uniqueness (for strict convexity) of a minimum of optimization
loss function, leads to a matrix inversion instead of a quadratic
problems (Fletcher 1989).
programming problem.
The question is which cost function should be used in (35). On
the one hand we will want to avoid a very complicated function c 3.3. Solving the equations
as this may lead to difficult optimization problems. On the other
For the sake of simplicity we will additionally assume c to
hand one should use that particular cost function that suits the
be symmetric and to have (at most) two (for symmetry) dis-
problem best. Moreover, under the assumption that the samples
continuities at ±ε, ε ≥ 0 in the first derivative, and to be
were generated by an underlying functional dependency plus
zero in the interval [−ε, ε]. All loss functions from Table 1
additive noise, i.e. yi = f true (xi ) + ξi with density p(ξ ), then the
belong to this class. Hence c will take on the following
optimal cost function in a maximum likelihood sense is
form.
c(x, y, f (x)) = − log p(y − f (x)). (37)
c(x, y, f (x)) = c̃(|y − f (x)|ε ) (41)
This can be seen as follows. The likelihood of an estimate
X f := {(x1 , f (x1 )), . . . , (x , f (x ))} (38) Note the similarity to Vapnik’s ε-insensitive loss. It is rather
straightforward to extend this special choice to more general
for additive noise and iid data is
convex cost functions. For nonzero cost functions in the inter-

val [−ε, ε] use an additional pair of slack variables. Moreover
p(X f | X) = p( f (xi ) | (xi , yi )) = p(yi − f (xi )). (39)
we might choose different cost functions c̃i , c̃i∗ and different
i=1 i=1
values of εi , εi∗ for each sample. At the expense of additional
Maximizing P(X f | X) is equivalent to minimizing − log P Lagrange multipliers in the dual formulation additional discon-
(X f | X). By using (37) we get tinuities also can be taken care of. Analogously to (3) we arrive at

a convex minimization problem (Smola and Schölkopf 1998a).
− log P(X f | X) = c(xi , yi , f (xi )). (40) To simplify notation we will stick to the one of (3) and use C
i=1
Table 1. Common loss functions and corresponding density models
Loss function Density model
ε-insensitive c(ξ ) = |ξ |ε p(ξ ) = 2(1+ε)

1
exp(−|ξ |ε )
Laplacian c(ξ ) = |ξ | p(ξ ) = 2 exp(−|ξ
1
|) 2
Gaussian c(ξ ) = 12 ξ 2 p(ξ ) = √12π exp − ξ2

1
(ξ )2 if |ξ | ≤ σ  exp − ξ 2 if |ξ | ≤ σ
Huber’s robust loss c(ξ ) = 2σ
p(ξ ) ∝ 2σ

|ξ | − σ
2
otherwise  exp σ − |ξ | otherwise
2
Polynomial c(ξ ) = 1
|ξ | p p(ξ ) = p
exp(−|ξ | p )
p
1 
2(1/ p)
(ξ ) p if |ξ | ≤ σ  exp − ξ p−1 p
if |ξ | ≤ σ
Piecewise polynomial c(ξ ) = pσ p−1
p(ξ ) ∝ pσ

|ξ | − σ p−1 p
otherwise  exp σ p−1 − |ξ | otherwise
p
instead of normalizing by λ and . Table 2. Terms of the convex optimization problem depending on the
choice of the loss function
1
minimize w2 + C (c̃(ξi ) + c̃(ξi∗ )) ε α C T (α)
2
 i=1
 yi − w, xi − b ≤ ε + ξi
 (42) ε-insensitive ε = 0 α ∈ [0, C] 0
subject to w, xi + b − yi ≤ ε + ξi∗ Laplacian ε =0 α ∈ [0, C] 0

 Gaussian ε =0 α ∈ [0, ∞) − 12 C −1 α 2
ξi , ξi∗ ≥0
Huber’s ε =0 α ∈ [0, C] − 12 σ C −1 α 2
Again, by standard Lagrange multiplier techniques, exactly in robust loss
1 p
the same manner as in the |·|ε case, one can compute the dual op- Polynomial ε=0 α ∈ [0, ∞) − p−1
p
C − p−1 α p−1
1 p
timization problem (the main difference is that the slack variable Piecewise ε=0 α ∈ [0, C] − p−1
p
σ C − p−1 α p−1
(∗)
terms c̃(ξi ) now have nonvanishing derivatives). We will omit polynomial
the indices i and ∗ , where applicable to avoid tedious notation.
This yields
 In the second case (ξ ≥ σ ) we have


−1
 (αi − αi∗ )(α j − α ∗j )xi , x j p−1 p−1

 T (ξ ) = ξ − σ − ξ = −σ (48)


2 i, j=1 p p


maximize + yi (αi − αi∗ ) − ε(αi + αi∗ ) and ξ = inf{ξ | C ≥ α} = σ , which, in turn yields α ∈ [0, C].

 Combining both cases we have

 i=1

 p−1

 +C
p p

 T (ξi ) + T (ξi∗ ) α ∈ [0, C] and T (α) = − σ C − p−1 α p−1 . (49)
i=1
p
 Table 2 contains a summary of the various conditions on α and



w= (αi − αi∗ )xi formulas for T (α) (strictly speaking T (ξ (α))) for different cost
where (43) functions.5 Note that the maximum slope of c̃ determines the

 i=1
T (ξ ) := c̃(ξ ) − ξ ∂ξ c̃(ξ ) region of feasibility of α, i.e. s := supξ ∈R+ ∂ξ c̃(ξ ) < ∞ leads to
 compact intervals [0, Cs] for α. This means that the influence



 (αi − αi∗ ) = 0 of a single pattern is bounded, leading to robust estimators


 i=1 (Huber 1972). One can also observe experimentally that the
subject to α ≤ C∂ξ c̃(ξ ) performance of a SV machine depends significantly on the cost



 ξ = inf{ξ | C∂ξ c̃ ≥ α} function used (Müller et al. 1997, Smola, Schölkopf and Müller


 1998b)
α, ξ ≥ 0
A cautionary remark is necessary regarding the use of cost
functions other than the ε-insensitive one. Unless ε = 0 we
3.4. Examples will lose the advantage of a sparse decomposition. This may
be acceptable in the case of few data, but will render the pre-
Let us consider the examples of Table 1. We will show explicitly diction step extremely slow otherwise. Hence one will have to
for two examples how (43) can be further simplified to bring it trade off a potential loss in prediction accuracy with faster pre-
into a form that is practically useful. In the ε-insensitive case, dictions. Note, however, that also a reduced set algorithm like
i.e. c̃(ξ ) = |ξ | we get in Burges (1996), Burges and Schölkopf (1997) and Schölkopf
et al. (1999b) or sparse decomposition techniques (Smola and
T (ξ ) = ξ − ξ · 1 = 0. (44)
Schölkopf 2000) could be applied to address this issue. In a
Morover one can conclude from ∂ξ c̃(ξ ) = 1 that Bayesian setting, Tipping (2000) has recently shown how an L 2
cost function can be used without sacrificing sparsity.
ξ = inf{ξ | C ≥ α} = 0 and α ∈ [0, C]. (45)
For the case of piecewise polynomial loss we have to distinguish 4. The bigger picture
two different cases: ξ ≤ σ and ξ > σ . In the first case we get
Before delving into algorithmic details of the implementation
1 1 p − 1 1− p p
T (ξ ) = ξ p − p−1 ξ p = − σ ξ (46) let us briefly review the basic properties of the SV algorithm
pσ p−1 σ p for regression as described so far. Figure 2 contains a graphical
overview over the different steps in the regression stage.
and ξ = inf{ξ | Cσ 1− p ξ p−1 ≥ α} = σ C − p−1 α p−1 and thus
1 1
The input pattern (for which a prediction is to be made) is

p−1 p p mapped into feature space by a map . Then dot products
T (ξ ) = − σ C − p−1 α p−1 . (47)
p are computed with the images of the training patterns under
ization operators. This will be explained in more detail in
Section 7.
Finally Fig. 4 shows the relation between approximation qual-
ity and sparsity of representation in the SV case. The lower the
precision required for approximating the original data, the fewer
SVs are needed to encode that. The non-SVs are redundant, i.e.
even without these patterns in the training set, the SV machine
would have constructed exactly the same function f . One might
think that this could be an efficient way of data compression,
namely by storing only the support patterns, from which the es-
timate can be reconstructed completely. However, this simple
analogy turns out to fail in the case of high-dimensional data,
and even more drastically in the presence of noise. In Vapnik,
Fig. 2. Architecture of a regression machine constructed by the SV Golowich and Smola (1997) one can see that even for moderate
algorithm approximation quality, the number of SVs can be considerably
high, yielding rates worse than the Nyquist rate (Nyquist 1928,
Shannon 1948).
the map . This corresponds to evaluating kernel functions
k(xi , x). Finally the dot products are added up using the weights
νi = αi − αi∗ . This, plus the constant term b yields the final 5. Optimization algorithms
prediction output. The process described here is very similar to
regression in a neural network, with the difference, that in the While there has been a large number of implementations of SV
SV case the weights in the input layer are a subset of the training algorithms in the past years, we focus on a few algorithms which
patterns. will be presented in greater detail. This selection is somewhat
Figure 3 demonstrates how the SV algorithm chooses the biased, as it contains these algorithms the authors are most fa-
flattest function among those approximating the original data miliar with. However, we think that this overview contains some
with a given precision. Although requiring flatness only in of the most effective ones and will be useful for practitioners
feature space, one can observe that the functions also are who would like to actually code a SV machine by themselves.
very flat in input space. This is due to the fact, that ker- But before doing so we will briefly cover major optimization
nels can be associated with flatness properties via regular- packages and strategies.
Fig. 3. Left to right: approximation of the function sinc x with precisions ε = 0.1, 0.2, and 0.5. The solid top and the bottom lines indicate the size
of the ε-tube, the dotted line in between is the regression
Fig. 4. Left to right: regression (solid line), datapoints (small dots) and SVs (big dots) for an approximation with ε = 0.1, 0.2, and 0.5. Note the
decrease in the number of SVs
5.1. Implementations subset selection techniques (see Section 5.5) to address this
problem.
Most commercially available packages for quadratic program-
ming can also be used to train SV machines. These are usually
numerically very stable general purpose codes, with special en- 5.2. Basic notions
hancements for large sparse systems. While the latter is a feature
Most algorithms rely on results from the duality theory in convex
that is not needed at all in SV problems (there the dot product
optimization. Although we already happened to mention some
matrix is dense and huge) they still can be used with good suc-
basic ideas in Section 1.2 we will, for the sake of convenience,
cess.6
briefly review without proof the core results. These are needed
in particular to derive an interior point algorithm. For details and
OSL: This package was written by IBM-Corporation (1992). It proofs (see e.g. Fletcher 1989).
uses a two phase algorithm. The first step consists of solving
Uniqueness: Every convex constrained optimization problem
a linear approximation of the QP problem by the simplex al-
has a unique minimum. If the problem is strictly convex then
gorithm (Dantzig 1962). Next a related very simple QP prob-
the solution is unique. This means that SVs are not plagued
lem is dealt with. When successive approximations are close
with the problem of local minima as Neural Networks are.7
enough together, the second subalgorithm, which permits a
Lagrange function: The Lagrange function is given by the pri-
quadratic objective and converges very rapidly from a good
mal objective function minus the sum of all products between
starting value, is used. Recently an interior point algorithm
constraints and corresponding Lagrange multipliers (cf. e.g.
was added to the software suite.
Fletcher 1989, Bertsekas 1995). Optimization can be seen
CPLEX by CPLEX-Optimization-Inc. (1994) uses a primal-dual
as minimzation of the Lagrangian wrt. the primal variables
logarithmic barrier algorithm (Megiddo 1989) instead with
and simultaneous maximization wrt. the Lagrange multipli-
predictor-corrector step (see e.g. Lustig, Marsten and Shanno
ers, i.e. dual variables. It has a saddle point at the solution.
1992, Mehrotra and Sun 1992).
Usually the Lagrange function is only a theoretical device to
MINOS by the Stanford Optimization Laboratory (Murtagh and
derive the dual objective function (cf. Section 1.2).
Saunders 1983) uses a reduced gradient algorithm in con-
Dual objective function: It is derived by minimizing the
junction with a quasi-Newton algorithm. The constraints are
Lagrange function with respect to the primal variables and
handled by an active set strategy. Feasibility is maintained
subsequent elimination of the latter. Hence it can be written
throughout the process. On the active constraint manifold, a
solely in terms of the dual variables.
quasi-Newton approximation is used.
Duality gap: For both feasible primal and dual variables the pri-
MATLAB: Until recently the matlab QP optimizer delivered only
mal objective function (of a convex minimization problem)
agreeable, although below average performance on classifi-
is always greater or equal than the dual objective function.
cation tasks and was not all too useful for regression tasks
Since SVMs have only linear constraints the constraint qual-
(for problems much larger than 100 samples) due to the fact
ifications of the strong duality theorem (Bazaraa, Sherali and
that one is effectively dealing with an optimization prob-
Shetty 1993, Theorem 6.2.4) are satisfied and it follows that
lem of size 2 where at least half of the eigenvalues of the
gap vanishes at optimality. Thus the duality gap is a measure
Hessian vanish. These problems seem to have been addressed
how close (in terms of the objective function) the current set
in version 5.3 / R11. Matlab now uses interior point codes.
of variables is to the solution.
LOQO by Vanderbei (1994) is another example of an interior
Karush–Kuhn–Tucker (KKT) conditions: A set of primal and
point code. Section 5.3 discusses the underlying strategies in
dual variables that is both feasible and satisfies the KKT
detail and shows how they can be adapted to SV algorithms.
conditions is the solution (i.e. constraint · dual variable = 0).
Maximum margin perceptron by Kowalczyk (2000) is an algo-
The sum of the violated KKT terms determines exactly the
rithm specifically tailored to SVs. Unlike most other tech-
size of the duality gap (that is, we simply compute the
niques it works directly in primal space and thus does not
constraint · Lagrangemultiplier part as done in (55)). This
have to take the equality constraint on the Lagrange multipli-
allows us to compute the latter quite easily.
ers into account explicitly.
A simple intuition is that for violated constraints the dual
Iterative free set methods The algorithm by Kaufman (Bunch,
variable could be increased arbitrarily, thus rendering the
Kaufman and Parlett 1976, Bunch and Kaufman 1977, 1980,
Lagrange function arbitrarily large. This, however, is in con-
Drucker et al. 1997, Kaufman 1999), uses such a technique
tradition to the saddlepoint property.
starting with all variables on the boundary and adding them as
the Karush Kuhn Tucker conditions become more violated.
5.3. Interior point algorithms
This approach has the advantage of not having to compute
the full dot product matrix from the beginning. Instead it is In a nutshell the idea of an interior point algorithm is to com-
evaluated on the fly, yielding a performance improvement pute the dual of the optimization problem (in our case the dual
in comparison to tackling the whole optimization problem dual of Rreg [ f ]) and solve both primal and dual simultaneously.
at once. However, also other algorithms can be modified by This is done by only gradually enforcing the KKT conditions
to iteratively find a feasible solution and to use the duality Assuming that the (dominant) convex part q(α) of the pri-
gap between primal and dual objective function to determine mal objective is quadratic, the q scales with τ 2 where as the
the quality of the current set of variables. The special flavour linear part scales with τ . However, since the linear term dom-
of algorithm we will describe is primal-dual path-following inates the objective function, the rescaled values are still a
(Vanderbei 1994). better starting point than α = 0. In practice a speedup of
In order to avoid tedious notation we will consider the slightly approximately 95% of the overall training time can be ob-
more general problem and specialize the result to the SVM later. served when using the sequential minimization algorithm,
It is understood that unless stated otherwise, variables like α cf. (Smola 1998). A similar reasoning can be applied when
denote vectors and αi denotes its i-th component. retraining with the same regularization parameter but differ-
1 ent (yet similar) width parameters of the kernel function. See
minimize q(α) + c, α Cristianini, Campbell and Shawe-Taylor (1998) for details
2 (50)
subject to Aα = b and l ≤ α ≤ u thereon in a different context.
with c, α, l, u ∈ Rn , A ∈ Rn·m , b ∈ Rm , the inequalities be- Monitoring convergence via the feasibility gap: In the case of
tween vectors holding componentwise and q(α) being a convex both primal and dual feasible variables the following con-
function of α. Now we will add slack variables to get rid of all nection between primal and dual objective function holds:

inequalities but the positivity constraints. This yields: Dual Obj. = Primal Obj. − (gi z i + si ti ) (55)
1 i
minimize q(α) + c, α
2 This can be seen immediately by the construction of the
subject to Aα = b, α − g = l, α + t = u, (51)
g, t ≥ 0, α free
Lagrange function. In Regression Estimation (with the ε-
insensitive loss function) one obtains for i gi z i + si ti
 
The dual of (51) is + max(0, f (xi ) − (yi + εi ))(C − αi∗ )
1   − min(0, f (xi ) − (yi + εi ))αi
∗ 

maximize (q(α) − ∂q(α), α) + b, y + l, z − u, s   . (56)
2  + max(0, (y − ε ∗
) − f (x ))(C − α ) 
i i i i
1 i
subject to ∂q(α) + c − (Ay) + s = z, s, z ≥ 0, y free − min(0, (yi − εi∗ ) − f (xi ))αi
2
(52) Thus convergence with respect to the point of the solution
can be expressed in terms of the duality gap. An effective
Moreover we get the KKT conditions, namely stopping rule is to require

gi z i = 0 and si ti = 0 for all i ∈ [1 . . . n]. (53) i gi z i + si ti
≤ εtol (57)
A necessary and sufficient condition for the optimal solution is |Primal Objective| + 1
that the primal/dual variables satisfy both the feasibility condi- for some precision εtol . This condition is much in the spirit of
tions of (51) and (52) and the KKT conditions (53). We pro- primal dual interior point path following algorithms, where
ceed to solve (51)–(53) iteratively. The details can be found in convergence is measured in terms of the number of significant
Appendix A. figures (which would be the decimal logarithm of (57)), a
convention that will also be adopted in the subsequent parts
5.4. Useful tricks of this exposition.
Before proceeding to further algorithms for quadratic optimiza-

tion let us briefly mention some useful tricks that can be applied 5.5. Subset selection algorithms
to all algorithms described subsequently and may have signif- The convex programming algorithms described so far can be
icant impact despite their simplicity. They are in part derived used directly on moderately sized (up to 3000) samples datasets
from ideas of the interior-point approach. without any further modifications. On large datasets, however, it
Training with different regularization parameters: For several is difficult, due to memory and cpu limitations, to compute the
reasons (model selection, controlling the number of support dot product matrix k(xi , x j ) and keep it in memory. A simple
vectors, etc.) it may happen that one has to train a SV ma- calculation shows that for instance storing the dot product matrix
chine with different regularization parameters C, but other- of the NIST OCR database (60.000 samples) at single precision
wise rather identical settings. If the parameters Cnew = τ Cold would consume 0.7 GBytes. A Cholesky decomposition thereof,
is not too different it is advantageous to use the rescaled val- which would additionally require roughly the same amount of
ues of the Lagrange multipliers (i.e. αi , αi∗ ) as a starting point memory and 64 Teraflops (counting multiplies and adds sepa-
for the new optimization problem. Rescaling is necessary to rately), seems unrealistic, at least at current processor speeds.
satisfy the modified constraints. One gets A first solution, which was introduced in Vapnik (1982) relies
on the observation that the solution can be reconstructed from
αnew = τ αold and likewise bnew = τ bold . (54) the SVs alone. Hence, if we knew the SV set beforehand, and
it fitted into memory, then we could directly solve the reduced tions an explicit solution of the restricted quadratic programming
problem. The catch is that we do not know the SV set before problem is impossible. Yet, one could derive an analogous non-
solving the problem. The solution is to start with an arbitrary quadratic convex optimization problem for general cost func-
subset, a first chunk that fits into memory, train the SV algorithm tions but at the expense of having to solve it numerically.
on it, keep the SVs and fill the chunk up with data the current The exposition proceeds as follows: first one has to derive
estimator would make errors on (i.e. data lying outside the ε- the (modified) boundary conditions for the constrained 2 indices
tube of the current regression). Then retrain the system and keep (i, j) subproblem in regression, next one can proceed to solve the
on iterating until after training all KKT-conditions are satisfied. optimization problem analytically, and finally one has to check,
The basic chunking algorithm just postponed the underlying which part of the selection rules have to be modified to make
problem of dealing with large datasets whose dot-product matrix the approach work for regression. Since most of the content is
cannot be kept in memory: it will occur for larger training set fairly technical it has been relegated to Appendix C.
sizes than originally, but it is not completely avoided. Hence The main difference in implementations of SMO for regres-
the solution is Osuna, Freund and Girosi (1997) to use only a sion can be found in the way the constant offset b is determined
subset of the variables as a working set and optimize the problem (Keerthi et al. 1999) and which criterion is used to select a new
with respect to them while freezing the other variables. This set of variables. We present one such strategy in Appendix C.3.
method is described in detail in Osuna, Freund and Girosi (1997), However, since selection strategies are the focus of current re-
Joachims (1999) and Saunders et al. (1998) for the case of pattern search we recommend that readers interested in implementing
recognition.8 the algorithm make sure they are aware of the most recent de-
An adaptation of these techniques to the case of regression velopments in this area.
with convex cost functions can be found in Appendix B. The Finally, we note that just as we presently describe a generaliza-
basic structure of the method is described by Algorithm 1. tion of SMO to regression estimation, other learning problems
can also benefit from the underlying ideas. Recently, a SMO
Algorithm 1.: Basic structure of a working set algorithm algorithm for training novelty detection systems (i.e. one-class
Initialize αi , αi∗ = 0 classification) has been proposed (Schölkopf et al. 2001).
Choose arbitrary working set Sw
repeat 6. Variations on a theme
Compute coupling terms (linear and constant) for Sw (see
Appendix A.3) There exists a large number of algorithmic modifications of the
Solve reduced optimization problem SV algorithm, to make it suitable for specific settings (inverse
Choose new Sw from variables αi , αi∗ not satisfying the problems, semiparametric settings), different ways of measuring
KKT conditions capacity and reductions to linear programming (convex com-
until working set Sw = ∅ binations) and different ways of controlling capacity. We will
mention some of the more popular ones.
5.6. Sequential minimal optimization

6.1. Convex combinations and 1 -norms
Recently an algorithm—Sequential Minimal Optimization
(SMO)—was proposed (Platt 1999) that puts chunking to the All the algorithms presented so far involved convex, and at
extreme by iteratively selecting subsets only of size 2 and op- best, quadratic programming. Yet one might think of reducing
timizing the target function with respect to them. It has been the problem to a case where linear programming techniques
reported to have good convergence properties and it is easily can be applied. This can be done in a straightforward fashion
implemented. The key point is that for a working set of 2 the (Mangasarian 1965, 1968, Weston et al. 1999, Smola, Schölkopf
optimization subproblem can be solved analytically without ex- and Rätsch 1999) for both SV pattern recognition and regression.
plicitly invoking a quadratic optimizer. The key is to replace (35) by
While readily derived for pattern recognition by Platt (1999), Rreg [ f ] := Remp [ f ] + λα1 (58)
one simply has to mimick the original reasoning to obtain an
extension to Regression Estimation. This is what will be done where α1 denotes the 1 norm in coefficient space. Hence one
in Appendix C (the pseudocode can be found in Smola and uses the SV kernel expansion (11)
Schölkopf (1998b)). The modifications consist of a pattern de-

pendent regularization, convergence control via the number of f (x) = αi k(xi , x) + b
significant figures, and a modified system of equations to solve i=1
the optimization problem in two variables for regression analyt- with a different way of controlling capacity by minimizing
ically.
Note that the reasoning only applies to SV regression with 1
Rreg [ f ] = c(xi , yi , f (xi )) + λ |αi |. (59)
the ε insensitive loss function—for most other convex cost func- i=1 i=1
For the ε-insensitive loss function this leads to a linear program- for some ν > 0. Hence (42) becomes (again carrying out the
ming problem. In the other cases, however, the problem still stays usual transformation between λ, and C)
a quadratic or general convex one, and therefore may not yield
1
∗
the desired computational advantage. Therefore we will limit minimize w + C
2
(c̃(ξi ) + c̃(ξi )) + νε
ourselves to the derivation of the linear programming problem 2 i=1
in the case of | · |ε cost function. Reformulating (59) yields  (61)

 yi − w, x i − b ≤ ε + ξ i

(αi + αi∗ ) + C (ξi + ξi∗ ) subject to w, xi + b − yi ≤ ε + ξi∗
minimize 

i=1 i=1 ξi , ξi∗ ≥ 0



We consider the standard | · |ε loss function. Computing the dual

 y − (α j − α ∗j )k(x j , xi ) − b ≤ ε + ξi


i of (62) yields
 j=1 
subject to 

 1


 (α j − α ∗j )k(x j , xi ) + b − yi ≤ ε + ξi∗  −
 2 (αi − αi∗ )(α j − α ∗j )k(xi , x j )



 j=1 maximize i, j=1
αi , αi∗ , ξi , ξi∗ ≥ 0 


 yi (αi − αi∗ )
+
Unlike in the classical SV case, the transformation into its dual i=1
 (62)
does not give any improvement in the structure of the optimiza-


tion problem. Hence it is best to minimize Rreg [ f ] directly, which 
 (αi − αi∗ ) = 0


can be achieved by a linear optimizer, (e.g. Dantzig 1962, Lustig,  i=1
Marsten and Shanno 1990, Vanderbei 1997). subject to

 (αi + αi∗ ) ≤ Cν
In (Weston et al. 1999) a similar variant of the linear SV ap- 



proach is used to estimate densities on a line. One can show  i=1 ∗
αi , αi ∈ [0, C]
(Smola et al. 2000) that one may obtain bounds on the gener-
alization error which exhibit even better rates (in terms of the Note that the optimization problem is thus very similar to the ε-
entropy numbers) than the classical SV case (Williamson, Smola SV one: the target function is even simpler (it is homogeneous),
and Schölkopf 1998). but there is an additional constraint. For information on how this
affects the implementation (cf. Chang and Lin 2001).
Besides having the advantage of being able to automatically
6.2. Automatic tuning of the insensitivity tube determine ε (63) also has another advantage. It can be used to
Besides standard model selection issues, i.e. how to specify the pre–specify the number of SVs:
trade-off between empirical error and model capacity there also
exists the problem of an optimal choice of a cost function. In Theorem 9 (Schölkopf et al. 2000).
particular, for the ε-insensitive cost function we still have the 1. ν is an upper bound on the fraction of errors.
problem of choosing an adequate parameter ε in order to achieve 2. ν is a lower bound on the fraction of SVs.
good performance with the SV machine. 3. Suppose the data has been generated iid from a distribution
Smola et al. (1998a) show the existence of a linear depen- p(x, y) = p(x) p(y | x) with a continuous conditional distri-
dency between the noise level and the optimal ε-parameter for bution p(y | x). With probability 1, asymptotically, ν equals
SV regression. However, this would require that we know some- the fraction of SVs and the fraction of errors.
thing about the noise model. This knowledge is not available in
general. Therefore, albeit providing theoretical insight, this find- Essentially, ν-SV regression improves upon ε-SV regression by
ing by itself is not particularly useful in practice. Moreover, if we allowing the tube width to adapt automatically to the data. What
really knew the noise model, we most likely would not choose is kept fixed up to this point, however, is the shape of the tube.
the ε-insensitive cost function but the corresponding maximum One can, however, go one step further and use parametric tube
likelihood loss function instead. models with non-constant width, leading to almost identical op-
There exists, however, a method to construct SV machines timization problems (Schölkopf et al. 2000).
that automatically adjust ε and moreover also, at least asymptot- Combining ν-SV regression with results on the asymptotical
ically, have a predetermined fraction of sampling points as SVs optimal choice of ε for a given noise model (Smola et al. 1998a)
(Schölkopf et al. 2000). We modify (35) such that ε becomes a leads to a guideline how to adjust ν provided the class of noise
variable of the optimization problem, including an extra term in models (e.g. Gaussian or Laplacian) is known.
the primal objective function which attempts to minimize ε. In
other words Remark 10 (Optimal choice of ν). Denote by p a probability
λ density with unit variance, and by P a famliy of noise models
minimize Rν [ f ] := Remp [ f ] + w2 + νε (60) generated from p by P := { p| p = σ1 p( σy )}. Moreover assume
2
by making problems seemingly easier yet reliable via a map into
some even higher dimensional space.
In this section we focus on the connections between SV
methods and previous techniques like Regularization Networks
(Girosi, Jones and Poggio 1993).9 In particular we will show
that SV machines are essentially Regularization Networks (RN)
with a clever choice of cost functions and that the kernels are
Green’s function of the corresponding regularization operators.
For a full exposition of the subject the reader is referred to Smola,
Schölkopf and Müller (1998c).
7.1. Regularization networks

Let us briefly review the basic concepts of RNs. As in (35)
we minimize a regularized risk functional. However, rather than
enforcing flatness in feature space we try to optimize some
smoothness criterion for the function in input space. Thus we
get
Fig. 5. Optimal ν and ε for various degrees of polynomial additive
noise λ
Rreg [ f ] := Remp [ f ] + P f 2 . (64)
2
that the data were drawn iid from p(x, y) = p(x) p(y − f (x))
with p(y − f (x)) continuous. Then under the assumption of Here P denotes a regularization operator in the sense of
uniform convergence, the asymptotically optimal value of ν is Tikhonov and Arsenin (1977), i.e. P is a positive semidefinite
ε operator mapping from the Hilbert space H of functions f under
ν = 1− p(t) dt consideration to a dot product space D such that the expression
−ε
P f · Pg is well defined for f, g ∈ H . For instance by choos-
τ
−2 ing a suitable operator that penalizes large variations of f one
where ε := argmin(p(−τ ) + p(τ )) 1 − p(t) dt (63)
τ −τ can reduce the well–known overfitting effect. Another possible
setting also might be an operator P mapping from L 2 (Rn ) into
For polynomial noise models, i.e. densities of type exp(−|ξ | p ) some Reproducing Kernel Hilbert Space (RKHS) (Aronszajn,
one may compute the corresponding (asymptotically) optimal 1950, Kimeldorf and Wahba 1971, Saitoh 1988, Schölkopf 1997,
values of ν. They are given in Fig. 5. For further details see Girosi 1998).
(Schölkopf et al. 2000, Smola 1998); an experimental validation Using an expansion of f in terms of some symmetric function
has been given by Chalimourda, Schölkopf and Smola (2000). k(xi , x j ) (note here, that k need not fulfill Mercer’s condition
We conclude this section by noting that ν-SV regression is and can be chosen arbitrarily since it is not used to define a
related to the idea of trimmed estimators. One can show that the regularization term),
regression is not influenced if we perturb points lying outside the
tube. Thus, the regression is essentially computed by discarding

a certain fraction of outliers, specified by ν, and computing the f (x) = αi k(xi , x) + b, (65)
i=1
regression estimate from the remaining points (Schölkopf et al.
2000). and the ε-insensitive cost function, this leads to a quadratic pro-
gramming problem similar to the one for SVs. Using
7. Regularization
Di j := (Pk)(xi , .) · (Pk)(x j , .) (66)
So far we were not concerned about the specific properties of
the map into feature space and used it only as a convenient we get α = D −1 K (β − β ∗ ), with β, β ∗ being the solution of
trick to construct nonlinear regression functions. In some cases
1 ∗
the map was just given implicitly by the kernel, hence the map minimize (β − β) KD−1 K (β ∗ − β)
itself and many of its properties have been neglected. A deeper 2

understanding of the kernel map would also be useful to choose −(β ∗ − β) y − ε (βi + βi∗ ) (67)
appropriate kernels for a specific task (e.g. by incorporating i=1

prior knowledge (Schölkopf et al. 1998a)). Finally the feature subject to (βi − βi∗ ) = 0 and βi , βi∗ ∈ [0, C].
map seems to defy the curse of dimensionality (Bellman 1961) i=1
Unfortunately, this setting of the problem does not preserve spar- with f˜(ω) denoting the Fourier transform of f (x), and P(ω) =
sity in terms of the coefficients, as a potentially sparse decom- P(−ω) real valued, nonnegative and converging to 0 for |ω| →
position in terms of βi and βi∗ is spoiled by D −1 K , which is not ∞ and := supp[P(ω)]. Small values of P(ω) correspond to
in general diagonal. a strong attenuation of the corresponding frequencies. Hence
small values of P(ω) for large ω are desirable since high fre-
7.2. Green’s functions quency components of f˜ correspond to rapid changes in f .
P(ω) describes the filter properties of P ∗ P. Note that no atten-
Comparing (10) with (67) leads to the question whether and un- uation takes place for P(ω) = 0 as these frequencies have been
der which condition the two methods might be equivalent and excluded from the integration domain.
therefore also under which conditions regularization networks For regularization operators defined in Fourier Space by (70)
might lead to sparse decompositions, i.e. only a few of the ex- one can show by exploiting P(ω) = P(−ω) = P(ω) that
pansion coefficients αi in f would differ from zero. A sufficient
1
condition is D = K and thus KD−1 K = K (if K does not have G(xi , x) = eiω(xi −x) P(ω) dω (71)
(2π )n/2 Rn
full rank we only need that KD−1 K = K holds on the image of
K ): is a corresponding Green’s function satisfying translational in-
variance, i.e.
k(xi , x j ) = (Pk)(xi , .) · (Pk)(x j , .) (68)
G(xi , x j ) = G(xi − x j ) and ˜
G(ω) = P(ω). (72)
Our goal now is to solve the following two problems:
This provides us with an efficient tool for analyzing SV kernels
1. Given a regularization operator P, find a kernel k such that a and the types of capacity control they exhibit. In fact the above
SV machine using k will not only enforce flatness in feature is a special case of Bochner’s theorem (Bochner 1959) stating
space, but also correspond to minimizing a regularized risk that the Fourier transform of a positive measure constitutes a
functional with P as regularizer. positive Hilbert Schmidt kernel.
2. Given an SV kernel k, find a regularization operator P such
that a SV machine using this kernel can be viewed as a Reg-
Example 2 (Gaussian kernels). Following the exposition of
ularization Network using P.
Yuille and Grzywacz (1988) as described in Girosi, Jones and
These two problems can be solved by employing the concept Poggio (1993), one can see that for
of Green’s functions as described in Girosi, Jones and Poggio σ 2m
(1993). These functions were introduced for the purpose of solv- P f 2 = d x m
( Ô m f (x))2 (73)
m m!2
ing differential equations. In our context it is sufficient to know
that the Green’s functions G xi (x) of P ∗ P satisfy with Ô 2m = m and Ô 2m+1 = ∇m , being the Laplacian
(P ∗ P G xi )(x) = δxi (x). (69) and ∇ the Gradient operator, we get Gaussians kernels (31).
Moreover, we can provide an equivalent representation of P
σ 2 ω2
Here, δxi (x) is the δ-distribution (not to be confused with the Kro- in terms of its Fourier properties, i.e. P(ω) = e− 2 up to a
necker symbol δi j ) which has the property that f · δxi = f (xi ). multiplicative constant.
The relationship between kernels and regularization operators is
formalized in the following proposition: Training an SV machine with Gaussian RBF kernels (Schölkopf
et al. 1997) corresponds to minimizing the specific cost func-
Proposition 1 (Smola, Schölkopf and Müller 1998b). Let P tion with a regularization operator of type (73). Recall that (73)
be a regularization operator, and G be the Green’s function of means that all derivatives of f are penalized (we have a pseudod-
P ∗ P. Then G is a Mercer Kernel such that D = K . SV machines ifferential operator) to obtain a very smooth estimate. This also
using G minimize risk functional (64) with P as regularization explains the good performance of SV machines in this case, as it
operator. is by no means obvious that choosing a flat function in some high
dimensional space will correspond to a simple function in low
In the following we will exploit this relationship in both ways: dimensional space, as shown in Smola, Schölkopf and Müller
to compute Green’s functions for a given regularization operator (1998c) for Dirichlet kernels.
P and to infer the regularizer, given a kernel k. The question that arises now is which kernel to choose. Let
us think about two extreme situations.
7.3. Translation invariant kernels 1. Suppose we already knew the shape of the power spectrum
Pow(ω) of the function we would like to estimate. In this case
Let us now more specifically consider regularization operators
we choose k such that k̃ matches the power spectrum (Smola
P̂ that may be written as multiplications in Fourier space
1998).
˜
1 ˜
f (ω)g(ω) 2. If we happen to know very little about the given data a gen-
P f · Pg = dω (70)
(2π) n/2
P(ω) eral smoothness assumption is a reasonable choice. Hence
we might want to choose a Gaussian kernel. If computing function class in all possible ways, the Covering Number which
time is important one might moreover consider kernels with is the number of elements from F that are needed to cover F with
compact support, e.g. using the Bq –spline kernels (cf. (32)). accuracy of at least ε, Entropy Numbers which are the functional
This choice will cause many matrix elements ki j = k(xi −x j ) inverse of Covering Numbers, and many more variants thereof
to vanish. (see e.g. Vapnik 1982, 1998, Devroye, Györfi and Lugosi 1996,
Williamson, Smola and Schölkopf 1998, Shawe-Taylor et al.
The usual scenario will be in between the two extreme cases and 1998).
we will have some limited prior knowledge available. For more
information on using prior knowledge for choosing kernels (see
Schölkopf et al. 1998a). 8. Conclusion
Due to the already quite large body of work done in the field of
7.4. Capacity control SV research it is impossible to write a tutorial on SV regression
which includes all contributions to this field. This also would
All the reasoning so far was based on the assumption that there be quite out of the scope of a tutorial and rather be relegated to
exist ways to determine model parameters like the regularization textbooks on the matter (see Schölkopf and Smola (2002) for a
constant λ or length scales σ of rbf–kernels. The model selec- comprehensive overview, Schölkopf, Burges and Smola (1999a)
tion issue itself would easily double the length of this review for a snapshot of the current state of the art, Vapnik (1998) for an
and moreover it is an area of active and rapidly moving research. overview on statistical learning theory, or Cristianini and Shawe-
Therefore we limit ourselves to a presentation of the basic con- Taylor (2000) for an introductory textbook). Still the authors
cepts and refer the interested reader to the original publications. hope that this work provides a not overly biased view of the state
It is important to keep in mind that there exist several fun- of the art in SV regression research. We deliberately omitted
damentally different approaches such as Minimum Description (among others) the following topics.
Length (cf. e.g. Rissanen 1978, Li and Vitányi 1993) which is
based on the idea that the simplicity of an estimate, and therefore
8.1. Missing topics
also its plausibility is based on the information (number of bits)
needed to encode it such that it can be reconstructed. Mathematical programming: Starting from a completely differ-
Bayesian estimation, on the other hand, considers the pos- ent perspective algorithms have been developed that are sim-
terior probability of an estimate, given the observations X = ilar in their ideas to SV machines. A good primer might
{(x1 , y1 ), . . . (x , y )}, an observation noise model, and a prior be (Bradley, Fayyad and Mangasarian 1998). (Also see
probability distribution p( f ) over the space of estimates Mangasarian 1965, 1969, Street and Mangasarian 1995). A
(parameters). It is given by Bayes Rule p( f | X ) p(X ) = comprehensive discussion of connections between mathe-
p(X | f ) p( f ). Since p(X ) does not depend on f , one can maxi- matical programming and SV machines has been given by
mize p(X | f ) p( f ) to obtain the so-called MAP estimate.10 As (Bennett 1999).
a rule of thumb, to translate regularized risk functionals into Density estimation: with SV machines (Weston et al. 1999,
Bayesian MAP estimation schemes, all one has to do is to con- Vapnik 1999). There one makes use of the fact that the cu-
sider exp(−Rreg [ f ]) = p( f | X ). For a more detailed discussion mulative distribution function is monotonically increasing,
(see e.g. Kimeldorf and Wahba 1970, MacKay 1991, Neal 1996, and that its values can be predicted with variable confidence
Rasmussen 1996, Williams 1998). which is adjusted by selecting different values of ε in the loss
A simple yet powerful way of model selection is cross valida- function.
tion. This is based on the idea that the expectation of the error Dictionaries: were originally introduced in the context of
on a subset of the training sample not used during training is wavelets by (Chen, Donoho and Saunders 1999) to allow
identical to the expected error itself. There exist several strate- for a large class of basis functions to be considered simulta-
gies such as 10-fold crossvalidation, leave-one out error (-fold neously, e.g. kernels with different widths. In the standard SV
crossvalidation), bootstrap and derived algorithms to estimate case this is hardly possible except by defining new kernels as
the crossvalidation error itself (see e.g. Stone 1974, Wahba 1980, linear combinations of differently scaled ones: choosing the
Efron 1982, Efron and Tibshirani 1994, Wahba 1999, Jaakkola regularization operator already determines the kernel com-
and Haussler 1999) for further details. pletely (Kimeldorf and Wahba 1971, Cox and O’Sullivan
Finally, one may also use uniform convergence bounds such 1990, Schölkopf et al. 2000). Hence one has to resort to lin-
as the ones introduced by Vapnik and Chervonenkis (1971). The ear programming (Weston et al. 1999).
basic idea is that one may bound with probability 1 − η (with Applications: The focus of this review was on methods and
η > 0) the expected risk R[ f ] by Remp [ f ] + (F, η), where theory rather than on applications. This was done to limit
is a confidence term depending on the class of functions F. the size of the exposition. State of the art, or even record
Several criteria for measuring the capacity of F exist, such as the performance was reported in Müller et al. (1997), Drucker
VC-Dimension which, in pattern recognition problems, is given et al. (1997), Stitson et al. (1999) and Mattera and Haykin
by the maximum number of points that can be separated by the (1999).
In many cases, it may be possible to achieve similar per- this exciting field, may find it useful to consult the web page
formance with neural network methods, however, only if www.kernel-machines.org.
many parameters are optimally tuned by hand, thus depend-
ing largely on the skill of the experimenter. Certainly, SV
machines are not a “silver bullet.” However, as they have Appendix A: Solving the interior-point
only few critical parameters (e.g. regularization and kernel equations
width), state-of-the-art results can be achieved with relatively
little effort. A.1. Path following
Rather than trying to satisfy (53) directly we will solve a modified
8.2. Open issues version thereof for some µ > 0 substituted on the rhs in the first
Being a very active field there exist still a number of open is- place and decrease µ while iterating.
sues that have to be addressed by future research. After that
the algorithmic development seems to have found a more sta- gi z i = µ, si ti = µ for all i ∈ [1 . . . n]. (74)
ble stage, one of the most important ones seems to be to find
tight error bounds derived from the specific properties of ker- Still it is rather difficult to solve the nonlinear system of equa-
nel functions. It will be of interest in this context, whether tions (51), (52), and (74) exactly. However we are not interested
SV machines, or similar approaches stemming from a lin- in obtaining the exact solution to the approximation (74). In-
ear programming regularizer, will lead to more satisfactory stead, we seek a somewhat more feasible solution for a given µ,
results. then decrease µ and repeat. This can be done by linearizing the
Moreover some sort of “luckiness framework” (Shawe-Taylor above system and solving the resulting equations by a predictor–
et al. 1998) for multiple model selection parameters, similar to corrector approach until the duality gap is small enough. The
multiple hyperparameters and automatic relevance detection in advantage is that we will get approximately equal performance
Bayesian statistics (MacKay 1991, Bishop 1995), will have to as by trying to solve the quadratic system directly, provided that
be devised to make SV machines less dependent on the skill of the terms in 2 are small enough.
the experimenter.
It is also worth while to exploit the bridge between regulariza- A(α + α) = b
tion operators, Gaussian processes and priors (see e.g. (Williams
α + α − g − g = l
1998)) to state Bayesian risk bounds for SV machines in order
to compare the predictions with the ones from VC theory. Op- α + α + t + t = u
timization techniques developed in the context of SV machines 1 1
also could be used to deal with large datasets in the Gaussian c + ∂α q(α) + ∂α2 q(α)α − (A(y + y))
2 2
process settings.
+ s + s = z + z
Prior knowledge appears to be another important question in
SV regression. Whilst invariances could be included in pattern (gi + gi )(z i + z i ) = µ
recognition in a principled way via the virtual SV mechanism
(si + si )(ti + ti ) = µ
and restriction of the feature space (Burges and Schölkopf 1997,
Schölkopf et al. 1998a), it is still not clear how (probably) more
Solving for the variables in we get
subtle properties, as required for regression, could be dealt with
efficiently.
Aα = b − Aα =: ρ
Reduced set methods also should be considered for speeding
up prediction (and possibly also training) phase for large datasets α − g = l − α + g =: ν
(Burges and Schölkopf 1997, Osuna and Girosi 1999, Schölkopf
α + t = u − α − t =: τ
et al. 1999b, Smola and Schölkopf 2000). This topic is of great
1
importance as data mining applications require algorithms that (Ay) + z − s − ∂α2 q(α)α
are able to deal with databases that are often at least one order of 2
magnitude larger (1 million samples) than the current practical 1
size for SV regression. = c − (Ay) + s − z + ∂α q(α) =: σ
2
Many more aspects such as more data dependent generaliza-
g −1 zg + z = µg −1 − z − g −1 gz =: γz
tion bounds, efficient training algorithms, automatic kernel se-
lection procedures, and many techniques that already have made t −1 st + s = µt −1 − s − t −1 ts =: γs
their way into the standard neural networks toolkit, will have to
be considered in the future. where g −1 denotes the vector (1/g1 , . . . , 1/gn ), and t analo-
Readers who are tempted to embark upon a more detailed gously. Moreover denote g −1 z and t −1 s the vector generated
exploration of these topics, and to contribute their own ideas to by the componentwise product of the two vectors. Solving for
g, t, z, s we get and subsequently restricts the solution to a feasible set

u
g = z −1 g(γz − z) z = g −1 z(ν̂ − α) x = max x,
100
t = s −1 t(γs − s) s = t −1 s(α − τ̂ )
(75) g = min(α − l, u)
where ν̂ := ν − z −1 gγz
τ̂ := τ − s −1 tγs t = min(u − α, u) (79)

1 u
Now we can formulate the reduced KKT–system (see (Vanderbei z = min ∂α q(α) + c − (Ay) + ,u
2 100
1994) for the quadratic case):
1 u
s = min − ∂α q(α) − c + (Ay) + ,u
−H A α σ − g −1 z ν̂ − t −1 s τ̂ 2 100
= (76)
A 0 y ρ (·) denotes the Heavyside function, i.e. (x) = 1 for x > 0
and (x) = 0 otherwise.
where H := ( 12 ∂α2 q(α) + g −1 z + t −1 s).
A.3. Special considerations for SV regression
A.2. Iteration strategies The algorithm described so far can be applied to both SV pattern
recognition and regression estimation. For the standard setting
For the predictor-corrector method we proceed as follows. In in pattern recognition we have
the predictor step solve the system of (75) and (76) with µ = 0
and all -terms on the rhs set to 0, i.e. γz = z, γs = s. The

q(α) = αi α j yi y j k(xi , x j ) (80)
values in are substituted back into the definitions for γz and i, j=0
γs and (75) and (76) are solved again in the corrector step. As the
quadratic part in (76) is not affected by the predictor–corrector and consequently ∂αi q(α) = 0, ∂α2i α j q(α) = yi y j k(xi , x j ), i.e.
steps, we only need to invert the quadratic matrix once. This is the Hessian is dense and the only thing we can do is compute
done best by manually pivoting for the H part, as it is positive its Cholesky factorization to compute (76). In the case of SV re-
definite. gression, however we have (with α := (α1 , . . . , α , α1∗ , . . . , α∗ ))
Next the values in obtained by such an iteration step are used

to update the corresponding values in α, s, t, z, . . . . To ensure q(α) = (αi − αi∗ )(α j − α ∗j )k(xi , x j )
that the variables meet the positivity constraints, the steplength i, j=1
ξ is chosen such that the variables move at most 1 − ε of their

initial distance to the boundaries of the positive orthant. Usually + 2C T (αi ) + T (αi∗ ) (81)
(Vanderbei 1994) one sets ε = 0.05. i=1
Another heuristic is used for computing µ, the parameter de- and therefore
termining how much the KKT-conditions should be enforced. d
Obviously it is our aim to reduce µ as fast as possible, however ∂αi q(α) = T (αi )
dαi
if we happen to choose it too small, the condition of the equa-
tions will worsen drastically. A setting that has proven to work d2
∂α2i α j q(α) = k(xi , x j ) + δi j T (αi ) (82)
robustly is dαi2
2 ∂α2i α∗j q(α) = −k(xi , x j )
g, z + s, t ξ −1
µ= . (77)
2n ξ + 10
and ∂α2∗ α∗ q(α), ∂α2∗ α j q(α) analogously. Hence we are dealing with
i j i
a matrix of type M := [ −K K +D −K ] where D, D are diagonal
The rationale behind (77) is to use the average of the satisfac- K +D
tion of the KKT conditions (74) as point of reference and then matrices. By applying an orthogonal transformation M can be
decrease µ rapidly if we are far enough away from the bound- inverted essentially by inverting an × matrix instead of a
aries of the positive orthant, to which all variables (except y) are 2 × 2 system. This is exactly the additional advantage one
constrained to. can gain from implementing the optimization algorithm directly
Finally one has to come up with good initial values. Analo- instead of using a general purpose optimizer. One can show that
gously to Vanderbei (1994) we choose a regularized version of for practical implementations (Smola, Schölkopf and Müller
(76) in order to determine the initial conditions. One solves 1998b) one can solve optimization problems using nearly ar-
bitrary convex cost functions as efficiently as the special case of

− 12 ∂α2 q(α) + 1 A α c ε-insensitive loss functions.
= (78) Finally note that due to the fact that we are solving the pri-
A 1 y b
mal and dual optimization problem simultaneously we are also
computing parameters corresponding to the initial SV optimiza- at sample xi , i.e.
tion problem. This observation is useful as it allows us to obtain

the constant term b directly, namely by setting b = y. (see Smola
m
(1998) for details). ϕi := yi − f (xi ) = yi − k(xi , x j )(αi − αi∗ ) +b . (84)
j=1
Appendix B: Solving the subset selection Rewriting the feasibility conditions (52) in terms of α yields
problem
2∂αi T (αi ) + ε − ϕi + si − z i = 0
B.1. Subset optimization problem (85)
2∂αi∗ T (αi∗ ) + ε + ϕi + si∗ − z i∗ = 0
We will adapt the exposition of Joachims (1999) to the case of
regression with convex cost functions. Without loss of general-
ity we will assume ε = 0 and α ∈ [0, C] (the other situations for all i ∈ {1, . . . , m} with z i , z i∗ , si , si∗ ≥ 0. A set of dual
can be treated as a special case). First we will extract a reduced feasible variables z, s is given by
optimization problem for the working set when all other vari-
ables are kept fixed. Denote Sw ⊂ {1, . . . , } the working set z i = max 2∂αi T (αi ) + ε − ϕi , 0
and S f := {1, . . . , }\Sw the fixed set. Writing (43) as an opti-
si = − min 2∂αi T (αi ) + ε − ϕi , 0
mization problem only in terms of Sw yields
(86)
 1 z i∗ = max 2∂αi∗ T (αi∗ ) + ε + ϕi , 0

 − (αi − αi∗ )(α j − α ∗j )xi , x j

 si∗ = − min 2∂αi∗ T (αi∗ ) + ε + ϕi , 0

 2



i, j∈Sw

∗ ∗
maximize + (αi − αi ) yi − (α j − α j )xi , x j Consequently the KKT conditions (53) can be translated into




i∈S w j∈S

f

 (−ε(αi + αi∗ ) + C(T (αi ) + T (αi∗ )))
 + αi z i = 0 and (C − αi )si = 0
i∈Sw
 (87)
 (αi − αi∗ ) = − (αi − αi∗ ) αi∗ z i∗ =0 and (C − αi∗ )si∗ = 0
subject to i∈Sw i∈S f

αi ∈ [0, C] All variables αi , αi∗ violating some of the conditions of (87) may
(83) be selected for further optimization. In most cases, especially in
the initial stage of the optimization algorithm, this set of pat-
Hence we only haveto update the linear terns is much larger than any practical size of Sw . Unfortunately
term by the coupling Osuna, Freund and Girosi (1997) contains little information on
with the fixed set − i∈Sw (αi − αi∗ ) j∈S f (α j − α ∗j )xi , x j and
how to select Sw . The heuristics presented here are an adaptation
the equality constraint by − i∈S f (αi − αi∗ ). It is easy to see
of Joachims (1999) to regression. See also Lin (2001) for details
that maximizing (83) also decreases (43) by exactly the same
on optimization for SVR.
amount. If we choose variables for which the KKT–conditions
are not satisfied the overall objective function tends to decrease
whilst still keeping all variables feasible. Finally it is bounded
from below. B.3. Selection rules
Even though this does not prove convergence (contrary to Similarly to a merit function approach (El-Bakry et al. 1996) the
statement in Osuna, Freund and Girosi (1997)) this algorithm idea is to select those variables that violate (85) and (87) most,
proves very useful in practice. It is one of the few methods (be- thus contribute most to the feasibility gap. Hence one defines a
sides (Kaufman 1999, Platt 1999)) that can deal with problems score variable ζi by
whose quadratic part does not completely fit into memory. Still
in practice one has to take special precautions to avoid stalling
of convergence (recent results of Chang, Hsu and Lin (1999) ζi := gi z i + si ti
indicate that under certain conditions a proof of convergence is = αi z i + αi∗ z i∗ + (C − αi )si + (C − αi∗ )si∗ (88)
possible). The crucial part is the one of Sw .

By construction, i ζi is the size of the feasibility gap (cf. (56)
for the case of ε-insensitive loss). By decreasing this gap, one
B.2. A note on optimality
approaches the the solution (upper bounded by the primal objec-
For convenience the KKT conditions are repeated in a slightly tive and lower bounded by the dual objective function). Hence,
modified form. Denote ϕi the error made by the current estimate the selection rule is to choose those patterns for which ζi is
largest. Some algorithms use C.2. Analytic solution for regression
ζi := αi (z i ) + αi∗ (z i∗ ) Next one has to solve the optimization problem analytically. We
make use of (84) and substitute the values of φi into the reduced
+ (C − αi )(si ) + (C − αi∗ )(si ) optimization problem (83). In particular we use
(89)
or ζi := (αi )z i + (αi∗ )z i∗
yi − (αi − αi∗ )K i j = ϕi + b + αiold − αi∗ old K i j .
+ (C − αi )si + (C − αi∗ )si . j∈ Sw j∈Sw
One can see that ζi = 0, ζi = 0, and ζi = 0 mutually imply each (91)

other. However, only ζi gives a measure for the contribution of
the variable i to the size of the feasibility gap. Moreover with the auxiliary variables γ = αi −αi∗ +α j −α ∗j and
Finally, note that heuristics like assigning sticky–flags (cf. η := (K ii + K j j − 2K i j ) one obtains the following constrained
Burges 1998) to variables at the boundaries, thus effec- optimization problem in i (after eliminating j, ignoring terms
tively solving smaller subproblems, or completely removing independent of α j , α ∗j and noting that this only holds for αi αi∗ =
the corresponding patterns from the training set while ac- α j α ∗j = 0):
counting for their couplings (Joachims 1999) can signifi- 1
maximize − (αi − αi∗ )2 η − ε(αi + αi∗ )(1 − s)
cantly decrease the size of the problem one has to solve and 2
thus result in a noticeable speedup. Also caching (Joachims
+ (αi − αi∗ ) φi − φ j + η αiold − αi∗ old (92)
1999, Kowalczyk 2000) of already computed entries of the
(∗)
dot product matrix may have a significant impact on the subject to αi ∈ [L (∗) , H (∗) ].
performance.
The unconstrained maximum of (92) with respect to αi or αi∗
can be found below.
Appendix C: Solving the SMO equations
C.1. Pattern dependent regularization (I) αi , α j αiold + η−1 (ϕi − ϕ j )
(II) αi , α ∗j αiold + η−1 (ϕi − ϕ j − 2ε)
Consider the constrained optimization problem (83) for two in- (III) αi∗ , α j αi∗ old − η−1 (ϕi − ϕ j + 2ε)
dices, say (i, j). Pattern dependent regularization means that Ci (IV) αi∗ , α ∗j αi∗ old − η−1 (ϕi − ϕ j )
may be different for every pattern (possibly even different for
αi and αi∗ ). Since at most two variables may become nonzero
at the same time and moreover we are dealing with a con- The problem is that we do not know beforehand which of the
strained optimization problem we may express everything in four quadrants (I)–(IV) contains the solution. However, by con-
terms of just one variable. From the summation constraint we sidering the sign of γ we can distinguish two cases: for γ > 0
obtain only (I)–(III) are possible, for γ < 0 the coefficients satisfy one
of the cases (II)–(IV). In case of γ = 0 only (II) and (III) have
to be considered. See also the diagram below.
(αi − αi∗ ) + (α j − α ∗j ) = αiold − αi∗ old + α old ∗ old
j − αj := γ
(90)
(∗) (∗) (∗)

for regression. Exploiting α j ∈ [0, C j ] yields αi ∈ [L , H ].
This is taking account of the fact that there may be only four
different pairs of nonzero variables: (αi , α j ), (αi∗ , α j ), (αi , α ∗j ),
and (αi∗ , α ∗j ). For convenience define an auxiliary variables s
such that s = 1 in the first and the last case and s = −1 other-
wise.
For γ > 0 it is best to start with quadrant (I), test whether the
αj α ∗j unconstrained solution hits one of the boundaries L , H and if so,
probe the corresponding adjacent quadrant (II) or (III). γ < 0
αi L max(0, γ − C j ) max(0, γ )
can be dealt with analogously.
H min(Ci , γ ) min(Ci , C ∗j + γ )
αi∗ L max(0, −γ ) max(0, −γ − C ∗j )
Due to numerical instabilities, it may happen that η < 0. In
H min(Ci∗ , −γ + C j ) min(Ci∗ , −γ ) that case η should be set to 0 and one has to solve (92) in a linear
fashion directly.11
C.3. Selection rule for regression Acknowledgments
Finally, one has to pick indices (i, j) such that the objective
This work has been supported in part by a grant of the DFG
function is maximized. Again, the reasoning of SMO (Platt 1999,
(Ja 379/71, Sm 62/1). The authors thank Peter Bartlett, Chris
Section 12.2.2) for classification will be mimicked. This means
Burges, Stefan Harmeling, Olvi Mangasarian, Klaus-Robert
that a two loop approach is chosen to maximize the objective
Müller, Vladimir Vapnik, Jason Weston, Robert Williamson, and
function. The outer loop iterates over all patterns violating the
Andreas Ziehe for helpful discussions and comments.
KKT conditions, first only over those with Lagrange multipliers
neither on the upper nor lower boundary, and once all of them
are satisfied, over all patterns violating the KKT conditions, to Notes
ensure self consistency on the complete dataset.12 This solves
the problem of choosing i. 1. Our use of the term ‘regression’ is somewhat lose in that it also includes
cases of function estimation where one minimizes errors other than the mean
Now for j: To make a large step towards the minimum, one square loss. This is done mainly for historical reasons (Vapnik, Golowich
looks for large steps in αi . As it is computationally expensive to and Smola 1997).
compute η for all possible pairs (i, j) one chooses the heuristic to 2. A similar approach, however using linear instead of quadratic programming,
maximize the absolute value of the numerator in the expressions was taken at the same time in the USA, mainly by Mangasarian (1965, 1968,
1969).
for αi and αi∗ , i.e. |ϕi − ϕ j | and |ϕi − ϕ j ± 2ε|. The index j 3. See Smola (1998) for an overview over other ways of specifying flatness of
corresponding to the maximum absolute value is chosen for this such functions.
purpose. 4. This is true as long as the dimensionality of w is much higher than the
number of observations. If this is not the case, specialized methods can
If this heuristic happens to fail, in other words if little progress offer considerable computational savings (Lee and Mangasarian 2001).
is made by this choice, all other indices j are looked at (this is 5. The table displays CT(α) instead of T (α) since the former can be plugged
what is called “second choice hierarcy” in Platt (1999) in the directly into the corresponding optimization equations.
following way: 6. The high price tag usually is the major deterrent for not using them. Moreover
one has to bear in mind that in SV regression, one may speed up the solution
considerably by exploiting the fact that the quadratic form has a special
1. All indices j corresponding to non–bound examples are structure or that there may exist rank degeneracies in the kernel matrix
looked at, searching for an example to make progress on. itself.
7. For large and noisy problems (e.g. 100.000 patterns and more with a substan-
2. In the case that the first heuristic was unsuccessful, all tial fraction of nonbound Lagrange multipliers) it is impossible to solve the
other samples are analyzed until an example is found where problem exactly: due to the size one has to use subset selection algorithms,
progress can be made. hence joint optimization over the training set is impossible. However, unlike
in Neural Networks, we can determine the closeness to the optimum. Note
3. If both previous steps fail proceed to the next i. that this reasoning only holds for convex cost functions.
8. A similar technique was employed by Bradley and Mangasarian (1998) in
For a more detailed discussion (see Platt 1999). Unlike interior the context of linear programming in order to deal with large datasets.
9. Due to length constraints we will not deal with the connection between
point algorithms SMO does not automatically provide a value Gaussian Processes and SVMs. See Williams (1998) for an excellent
for b. However this can be chosen like in Section 1.4 by having overview.
(∗)
a close look at the Lagrange multipliers αi obtained. 10. Strictly speaking, in Bayesian estimation one is not so much concerned about
the maximizer fˆ of p( f | X ) but rather about the posterior distribution of
f.
11. Negative values of η are theoretically impossible since k satisfies Mercer’s
C.4. Stopping criteria condition: 0 ≤ (xi ) − (x j )2 = K ii + K j j − 2K i j = η.
12. It is sometimes useful, especially when dealing with noisy data, to iterate
By essentially minimizing a constrained primal optimization over the complete KKT violating dataset already before complete self con-
problem one cannot ensure that the dual objective function in- sistency on the subset has been achieved. Otherwise much computational
resources are spent on making subsets self consistent that are not globally
creases with every iteration step.13 Nevertheless one knows that self consistent. This is the reason why in the pseudo code a global loop
the minimum value of the objective function lies in the interval is initiated already when only less than 10% of the non bound variables
[dual objectivei , primal objectivei ] for all steps i, hence also in changed.
the interval [(max j≤i dual objective j ), primal objectivei ]. One 13. It is still an open question how a subset selection optimization algorithm
could be devised that decreases both primal and dual objective function
uses the latter to determine the quality of the current solution. at the same time. The problem is that this usually involves a number of
The calculation of the primal objective function from the pre- dual variables of the order of the sample size, which makes this attempt
diction errors is straightforward. One uses unpractical.

(αi − αi∗ )(α j − α ∗j )ki j = − (αi − αi∗ )(ϕi + yi − b), References
i, j i
Aizerman M.A., Braverman É.M., and Rozonoér L.I. 1964. Theoretical
(93) foundations of the potential function method in pattern recognition
learning. Automation and Remote Control 25: 821–837.
i.e. the definition of ϕi to avoid the matrix–vector multiplication Aronszajn N. 1950. Theory of reproducing kernels. Transactions of the
with the dot product matrix. American Mathematical Society 68: 337–404.
Bazaraa M.S., Sherali H.D., and Shetty C.M. 1993. Nonlinear Program- Burges C.J.C. and Schölkopf B. 1997. Improving the accuracy and speed
ming: Theory and Algorithms, 2nd edition, Wiley. of support vector learning machines. In Mozer M.C., Jordan M.I.,
Bellman R.E. 1961. Adaptive Control Processes. Princeton University and Petsche T., (Eds.), Advances in Neural Information Processing
Press, Princeton, NJ. Systems 9, MIT Press, Cambridge, MA, pp. 375–381.
Bennett K. 1999. Combining support vector and mathematical program- Chalimourda A., Schölkopf B., and Smola A.J. 2004. Experimentally
ming methods for induction. In: Schölkopf B., Burges C.J.C., and optimal ν in support vector regression for different noise models
Smola A.J., (Eds.), Advances in Kernel Methods—SV Learning, and parameter settings. Neural Networks 17(1): 127–141.
MIT Press, Cambridge, MA, pp. 307–326. Chang C.-C., Hsu C.-W., and Lin C.-J. 1999. The analysis of decom-
Bennett K.P. and Mangasarian O.L. 1992. Robust linear program- position methods for support vector machines. In Proceeding of
ming discrimination of two linearly inseparable sets. Optimization IJCAI99, SVM Workshop.
Methods and Software 1: 23–34. Chang C.C. and Lin C.J. 2001. Training ν-support vector classi-
Berg C., Christensen J.P.R., and Ressel P. 1984. Harmonic Analysis on fiers: Theory and algorithms. Neural Computation 13(9): 2119–
Semigroups. Springer, New York. 2147.
Bertsekas D.P. 1995. Nonlinear Programming. Athena Scientific, Chen S., Donoho D., and Saunders M. 1999. Atomic decomposition by
Belmont, MA. basis pursuit. Siam Journal of Scientific Computing 20(1): 33–61.
Bishop C.M. 1995. Neural Networks for Pattern Recognition. Cherkassky V. and Mulier F. 1998. Learning from Data. John Wiley and
Clarendon Press, Oxford. Sons, New York.
Blanz V., Schölkopf B., Bülthoff H., Burges C., Vapnik V., and Vetter Cortes C. and Vapnik V. 1995. Support vector networks. Machine Learn-
T. 1996. Comparison of view-based object recognition algorithms ing 20: 273–297.
using realistic 3D models. In: von der Malsburg C., von Seelen Cox D. and O’Sullivan F. 1990. Asymptotic analysis of penalized like-
W., Vorbrüggen J.C., and Sendhoff B. (Eds.), Artificial Neural lihood and related estimators. Annals of Statistics 18: 1676–1695.
Networks ICANN’96, Berlin. Springer Lecture Notes in Computer CPLEX Optimization Inc. Using the CPLEX callable library. Manual,
Science, Vol. 1112, pp. 251–256. 1994.
Bochner S. 1959. Lectures on Fourier integral. Princeton Univ. Press, Cristianini N. and Shawe-Taylor J. 2000. An Introduction to Support
Princeton, New Jersey. Vector Machines. Cambridge University Press, Cambridge, UK.
Boser B.E., Guyon I.M., and Vapnik V.N. 1992. A training algorithm for Cristianini N., Campbell C., and Shawe-Taylor J. 1998. Multiplicative
optimal margin classifiers. In: Haussler D. (Ed.), Proceedings of updatings for support vector learning. NeuroCOLT Technical Re-
the Annual Conference on Computational Learning Theory. ACM port NC-TR-98-016, Royal Holloway College.
Press, Pittsburgh, PA, pp. 144–152. Dantzig G.B. 1962. Linear Programming and Extensions. Princeton
Bradley P.S., Fayyad U.M., and Mangasarian O.L. 1998. Data min- Univ. Press, Princeton, NJ.
ing: Overview and optimization opportunities. Technical Re- Devroye L., Györfi L., and Lugosi G. 1996. A Probabilistic Theory of
port 98–01, University of Wisconsin, Computer Sciences Depart- Pattern Recognition. Number 31 in Applications of mathematics.
ment, Madison, January. INFORMS Journal on Computing, to Springer, New York.
appear. Drucker H., Burges C.J.C., Kaufman L., Smola A., and Vapnik V. 1997.
Bradley P.S. and Mangasarian O.L. 1998. Feature selection via con- Support vector regression machines. In: Mozer M.C., Jordan M.I.,
cave minimization and support vector machines. In: Shavlik J. and Petsche T. (Eds.), Advances in Neural Information Processing
(Ed.), Proceedings of the International Conference on Machine Systems 9, MIT Press, Cambridge, MA, pp. 155–161.
Learning, Morgan Kaufmann Publishers, San Francisco, Califor- Efron B. 1982. The jacknife, the bootstrap, and other resampling plans.
nia, pp. 82–90. ftp://ftp.cs.wisc.edu/math-prog/tech-reports/98- SIAM, Philadelphia.
03.ps.Z. Efron B. and Tibshirani R.J. 1994. An Introduction to the Bootstrap.
Bunch J.R. and Kaufman L. 1977. Some stable methods for calculat- Chapman and Hall, New York.
ing inertia and solving symmetric linear systems. Mathematics of El-Bakry A., Tapia R., Tsuchiya R., and Zhang Y. 1996. On the formula-
Computation 31: 163–179. tion and theory of the Newton interior-point method for nonlinear
Bunch J.R. and Kaufman L. 1980. A computational method for the programming. J. Optimization Theory and Applications 89: 507–
indefinite quadratic programming problem. Linear Algebra and 541.
Its Applications, pp. 341–370, December. Fletcher R. 1989. Practical Methods of Optimization. John Wiley and
Bunch J.R., Kaufman L., and Parlett B. 1976. Decomposition of a sym- Sons, New York.
metric matrix. Numerische Mathematik 27: 95–109. Girosi F. 1998. An equivalence between sparse approximation and sup-
Burges C.J.C. 1996. Simplified support vector decision rules. In port vector machines. Neural Computation 10(6): 1455–1480.
L. Saitta (Ed.), Proceedings of the International Conference on Girosi F., Jones M., and Poggio T. 1993. Priors, stabilizers and ba-
Machine Learning, Morgan Kaufmann Publishers, San Mateo, sis functions: From regularization to radial, tensor and additive
CA, pp. 71–77. splines. A.I. Memo No. 1430, Artificial Intelligence Laboratory,
Burges C.J.C. 1998. A tutorial on support vector machines for pattern Massachusetts Institute of Technology.
recognition. Data Mining and Knowledge Discovery 2(2): 121– Guyon I., Boser B., and Vapnik V. 1993. Automatic capacity tuning of
167. very large VC-dimension classifiers. In: Hanson S.J., Cowan J.D.,
Burges C.J.C. 1999. Geometry and invariance in kernel based methods. and Giles C.L. (Eds.), Advances in Neural Information Processing
In Schölkopf B., Burges C.J.C., and Smola A.J., (Eds.), Advances Systems 5. Morgan Kaufmann Publishers, pp. 147–155.
in Kernel Methods—Support Vector Learning, MIT Press, Cam- Härdle W. 1990. Applied nonparametric regression, volume 19 of
bridge, MA, pp. 89–116. Econometric Society Monographs. Cambridge University Press.
Hastie T.J. and Tibshirani R.J. 1990. Generalized Additive Models, gramming. Princeton Technical Report SOR 90–03., Dept. of Civil
volume 43 of Monographs on Statistics and Applied Probability. Engineering and Operations Research, Princeton University.
Chapman and Hall, London. Lustig I.J., Marsten R.E., and Shanno D.F. 1992. On implement-
Haykin S. 1998. Neural Networks: A Comprehensive Foundation. 2nd ing Mehrotra’s predictor-corrector interior point method for lin-
edition. Macmillan, New York. ear programming. SIAM Journal on Optimization 2(3): 435–
Hearst M.A., Schölkopf B., Dumais S., Osuna E., and Platt J. 1998. 449.
Trends and controversies—support vector machines. IEEE Intel- MacKay D.J.C. 1991. Bayesian Methods for Adaptive Models. PhD
ligent Systems 13: 18–28. thesis, Computation and Neural Systems, California Institute of
Herbrich R. 2002. Learning Kernel Classifiers: Theory and Algorithms. Technology, Pasadena, CA.
MIT Press. Mangasarian O.L. 1965. Linear and nonlinear separation of patterns by
Huber P.J. 1972. Robust statistics: A review. Annals of Statistics linear programming. Operations Research 13: 444–452.
43: 1041. Mangasarian O.L. 1968. Multi-surface method of pattern separation.
Huber P.J. 1981. Robust Statistics. John Wiley and Sons, New York. IEEE Transactions on Information Theory IT-14: 801–807.
IBM Corporation. 1992. IBM optimization subroutine library guide Mangasarian O.L. 1969. Nonlinear Programming. McGraw-Hill, New
and reference. IBM Systems Journal, 31, SC23-0519. York.
Jaakkola T.S. and Haussler D. 1999. Probabilistic kernel regression Mattera D. and Haykin S. 1999. Support vector machines for dy-
models. In: Proceedings of the 1999 Conference on AI and Statis- namic reconstruction of a chaotic system. In: Schölkopf B., Burges
tics. C.J.C., and Smola A.J. (Eds.), Advances in Kernel Methods—
Joachims T. 1999. Making large-scale SVM learning practical. Support Vector Learning, MIT Press, Cambridge, MA, pp. 211–
In: Schölkopf B., Burges C.J.C., and Smola A.J. (Eds.), Ad- 242.
vances in Kernel Methods—Support Vector Learning, MIT Press, McCormick G.P. 1983. Nonlinear Programming: Theory, Algorithms,
Cambridge, MA, pp. 169–184. and Applications. John Wiley and Sons, New York.
Karush W. 1939. Minima of functions of several variables with inequal- Megiddo N. 1989. Progressin Mathematical Programming, chapter
ities as side constraints. Master’s thesis, Dept. of Mathematics, Pathways to the optimal set in linear programming, Springer, New
Univ. of Chicago. York, NY, pp. 131–158.
Kaufman L. 1999. Solving the quadratic programming problem arising Mehrotra S. and Sun J. 1992. On the implementation of a (primal-dual)
in support vector classification. In: Schölkopf B., Burges C.J.C., interior point method. SIAM Journal on Optimization 2(4): 575–
and Smola A.J. (Eds.), Advances in Kernel Methods—Support 601.
Vector Learning, MIT Press, Cambridge, MA, pp. 147–168 Mercer J. 1909. Functions of positive and negative type and their con-
Keerthi S.S., Shevade S.K., Bhattacharyya C., and Murthy K.R.K. 1999. nection with the theory of integral equations. Philosophical Trans-
Improvements to Platt’s SMO algorithm for SVM classifier design. actions of the Royal Society, London A 209: 415–446.
Technical Report CD-99-14, Dept. of Mechanical and Production Micchelli C.A. 1986. Algebraic aspects of interpolation. Proceedings
Engineering, Natl. Univ. Singapore, Singapore. of Symposia in Applied Mathematics 36: 81–102.
Keerthi S.S., Shevade S.K., Bhattacharyya C., and Murty K.R.K. 2001. Morozov V.A. 1984. Methods for Solving Incorrectly Posed Problems.
Improvements to platt’s SMO algorithm for SVM classifier design. Springer.
Neural Computation 13: 637–649. Müller K.-R., Smola A., Rätsch G., Schölkopf B., Kohlmorgen J., and
Kimeldorf G.S. and Wahba G. 1970. A correspondence between Vapnik V. 1997. Predicting time series with support vector ma-
Bayesian estimation on stochastic processes and smoothing by chines. In: Gerstner W., Germond A., Hasler M., and Nicoud J.-D.
splines. Annals of Mathematical Statistics 41: 495–502. (Eds.), Artificial Neural Networks ICANN’97, Berlin. Springer
Kimeldorf G.S. and Wahba G. 1971. Some results on Tchebycheffian Lecture Notes in Computer Science Vol. 1327 pp. 999–1004.
spline functions. J. Math. Anal. Applic. 33: 82–95. Murtagh B.A. and Saunders M.A. 1983. MINOS 5.1 user’s guide. Tech-
Kowalczyk A. 2000. Maximal margin perceptron. In: Smola A.J., nical Report SOL 83-20R, Stanford University, CA, USA, Revised
Bartlett P.L., Schölkopf B., and Schuurmans D. (Eds.), Advances 1987.
in Large Margin Classifiers, MIT Press, Cambridge, MA, pp. 75– Neal R. 1996. Bayesian Learning in Neural Networks. Springer.
113. Nilsson N.J. 1965. Learning machines: Foundations of Trainable Pattern
Kuhn H.W. and Tucker A.W. 1951. Nonlinear programming. In: Proc. Classifying Systems. McGraw-Hill.
2nd Berkeley Symposium on Mathematical Statistics and Proba- Nyquist. H. 1928. Certain topics in telegraph transmission theory.
bilistics, Berkeley. University of California Press, pp. 481–492. Trans. A.I.E.E., pp. 617–644.
Lee Y.J. and Mangasarian O.L. 2001. SSVM: A smooth support vector Osuna E., Freund R., and Girosi F. 1997. An improved training algo-
machine for classification. Computational optimization and Ap- rithm for support vector machines. In Principe J., Gile L., Morgan
plications 20(1): 5–22. N., and Wilson E. (Eds.), Neural Networks for Signal Processing
Li M. and Vitányi P. 1993. An introduction to Kolmogorov Complexity VII—Proceedings of the 1997 IEEE Workshop, pp. 276–285, New
and its applications. Texts and Monographs in Computer Science. York, IEEE.
Springer, New York. Osuna E. and Girosi F. 1999. Reducing the run-time complexity in
Lin C.J. 2001. On the convergence of the decomposition method for support vector regression. In: Schölkopf B., Burges C.J.C., and
support vector machines. IEEE Transactions on Neural Networks Smola A. J. (Eds.), Advances in Kernel Methods—Support Vector
12(6): 1288–1298. Learning, pp. 271–284, Cambridge, MA, MIT Press.
Lustig I.J., Marsten R.E., and Shanno D.F. 1990. On implementing Ovari Z. 2000. Kernels, eigenvalues and support vector machines. Hon-
Mehrotra’s predictor-corrector interior point method for linear pro- ours thesis, Australian National University, Canberra.
Platt J. 1999. Fast training of support vector machines using sequen- Gaussian kernels to radial basis function classifiers. IEEE Trans-
tial minimal optimization. In: Schölkopf B., Burges C.J.C., and actions on Signal Processing, 45: 2758–2765.
Smola A.J. (Eds.) Advances in Kernel Methods—Support Vector Shannon C.E. 1948. A mathematical theory of communication. Bell
Learning, pp. 185–208, Cambridge, MA, MIT Press. System Technical Journal, 27: 379–423, 623–656.
Poggio T. 1975. On optimal nonlinear associative recall. Biological Shawe-Taylor J., Bartlett P.L., Williamson R.C., and Anthony M.
Cybernetics, 19: 201–209. 1998. Structural risk minimization over data-dependent hierar-
Rasmussen C. 1996. Evaluation of Gaussian Processes and chies. IEEE Transactions on Information Theory, 44(5): 1926–
Other Methods for Non-Linear Regression. PhD thesis, 1940.
Department of Computer Science, University of Toronto, Smola A., Murata N., Schölkopf B., and Müller K.-R. 1998a. Asymp-
ftp://ftp.cs.toronto.edu/pub/carl/thesis.ps.gz. totically optimal choice of ε-loss for support vector machines.
Rissanen J. 1978. Modeling by shortest data description. Automatica, In: Niklasson L., Bodén M., and Ziemke T. (Eds.) Proceed-
14: 465–471. ings of the International Conference on Artificial Neural Net-
Saitoh S. 1988. Theory of Reproducing Kernels and its Applications. works, Perspectives in Neural Computing, pp. 105–110, Berlin,
Longman Scientific & Technical, Harlow, England. Springer.
Saunders C., Stitson M.O., Weston J., Bottou L., Schölkopf B., and Smola A., Schölkopf B., and Müller K.-R. 1998b. The connection be-
Smola A. 1998. Support vector machine—reference manual. Tech- tween regularization operators and support vector kernels. Neural
nical Report CSD-TR-98-03, Department of Computer Science, Networks, 11: 637–649.
Royal Holloway, University of London, Egham, UK. SVM avail- Smola A., Schölkopf B., and Müller K.-R. 1998c. General cost func-
able at http://svm.dcs.rhbnc.ac.uk/. tions for support vector regression. In: Downs T., Frean M.,
Schoenberg I. 1942. Positive definite functions on spheres. Duke and Gallagher M. (Eds.) Proc. of the Ninth Australian Conf. on
Math. J., 9: 96–108. Neural Networks, pp. 79–83, Brisbane, Australia. University of
Schölkopf B. 1997. Support Vector Learning. R. Oldenbourg Queensland.
Verlag, München. Doktorarbeit, TU Berlin. Download: Smola A., Schölkopf B., and Rätsch G. 1999. Linear programs for
http://www.kernel-machines.org. automatic accuracy control in regression. In: Ninth International
Schölkopf B., Burges C., and Vapnik V. 1995. Extracting support data Conference on Artificial Neural Networks, Conference Publica-
for a given task. In: Fayyad U.M. and Uthurusamy R. (Eds.), Pro- tions No. 470, pp. 575–580, London. IEE.
ceedings, First International Conference on Knowledge Discovery Smola. A.J. 1996. Regression estimation with support vector learning
& Data Mining, Menlo Park, AAAI Press. machines. Diplomarbeit, Technische Universität München.
Schölkopf B., Burges C., and Vapnik V. 1996. Incorporating invariances Smola A.J. 1998. Learning with Kernels. PhD thesis, Technische Uni-
in support vector learning machines. In: von der Malsburg C., von versität Berlin. GMD Research Series No. 25.
Seelen W., Vorbrüggen J. C., and Sendhoff B. (Eds.), Artificial Smola A.J., Elisseeff A., Schölkopf B., and Williamson R.C. 2000.
Neural Networks ICANN’96, pp. 47–52, Berlin, Springer Lecture Entropy numbers for convex combinations and MLPs. In Smola
Notes in Computer Science, Vol. 1112. A.J., Bartlett P.L., Schölkopf B., and Schuurmans D. (Eds.) Ad-
Schölkopf B., Burges C.J.C., and Smola A.J. 1999a. (Eds.) Ad- vances in Large Margin Classifiers, MIT Press, Cambridge, MA,
vances in Kernel Methods—Support Vector Learning. MIT Press, pp. 369–387.
Cambridge, MA. Smola A.J., Óvári Z.L., and Williamson R.C. 2001. Regularization with
Schölkopf B., Herbrich R., Smola A.J., and Williamson R.C. 2001. A dot-product kernels. In: Leen T.K., Dietterich T.G., and Tresp V.
generalized representer theorem. Technical Report 2000-81, Neu- (Eds.) Advances in Neural Information Processing Systems 13,
roCOLT, 2000. To appear in Proceedings of the Annual Conference MIT Press, pp. 308–314.
on Learning Theory, Springer (2001). Smola A.J. and Schölkopf B. 1998a. On a kernel-based method for
Schölkopf B., Mika S., Burges C., Knirsch P., Müller K.-R., Rätsch G., pattern recognition, regression, approximation and operator in-
and Smola A. 1999b. Input space vs. feature space in kernel-based version. Algorithmica, 22: 211–231.
methods. IEEE Transactions on Neural Networks, 10(5): 1000– Smola A.J. and Schölkopf B. 1998b. A tutorial on support vector re-
1017. gression. NeuroCOLT Technical Report NC-TR-98-030, Royal
Schölkopf B., Platt J., Shawe-Taylor J., Smola A.J. , and Williamson R.C. Holloway College, University of London, UK.
2001. Estimating the support of a high-dimensional distribution. Smola A.J. and Schölkopf B. 2000. Sparse greedy matrix approximation
Neural Computation, 13(7): 1443–1471. for machine learning. In: Langley P. (Ed.), Proceedings of the In-
Schölkopf B., Simard P., Smola A., and Vapnik V. 1998a. Prior knowl- ternational Conference on Machine Learning, Morgan Kaufmann
edge in support vector kernels. In: Jordan M.I., Kearns M.J., and Publishers, San Francisco, pp. 911–918.
Solla S.A. (Eds.) Advances in Neural Information Processing Sys- Stitson M., Gammerman A., Vapnik V., Vovk V., Watkins C., and
tems 10, MIT Press. Cambridge, MA, pp. 640–646. Weston J. 1999. Support vector regression with ANOVA decom-
Schölkopf B., Smola A., and Müller K.-R. 1998b. Nonlinear compo- position kernels. In: Schölkopf B., Burges C.J.C., and Smola A.J.
nent analysis as a kernel eigenvalue problem. Neural Computation, (Eds.), Advances in Kernel Methods—Support Vector Learning,
10: 1299–1319. MIT Press Cambridge, MA, pp. 285–292.
Schölkopf B., Smola A., Williamson R.C., and Bartlett P.L. 2000. New Stone C.J. 1985. Additive regression and other nonparametric models.
support vector algorithms. Neural Computation, 12: 1207–1245. Annals of Statistics, 13: 689–705.
Schölkopf B. and Smola A.J. 2002. Learning with Kernels. MIT Press. Stone M. 1974. Cross-validatory choice and assessment of statistical
Schölkopf B., Sung K., Burges C., Girosi F., Niyogi P., Poggio T., predictors (with discussion). Journal of the Royal Statistical Soci-
and Vapnik V. 1997. Comparing support vector machines with ety, B36: 111–147.
Street W.N. and Mangasarian O.L. 1995. Improved generalization via Vapnik V.N. 1982. Estimation of Dependences Based on Empirical
tolerant training. Technical Report MP-TR-95-11, University of Data. Springer, Berlin.
Wisconsin, Madison. Vapnik V.N. and Chervonenkis A.Y. 1971. On the uniform convergence
Tikhonov A.N. and Arsenin V.Y. 1977. Solution of Ill-posed problems. of relative frequencies of events to their probabilities. Theory of
V. H. Winston and Sons. Probability and its Applications, 16(2): 264–281.
Tipping M.E. 2000. The relevance vector machine. In: Solla S.A., Leen Wahba G. 1980. Spline bases, regularization, and generalized
T.K., and Müller K.-R. (Eds.), Advances in Neural Information cross-validation for solving approximation problems with large
Processing Systems 12, MIT Press, Cambridge, MA, pp. 652–658. quantities of noisy data. In: Ward J. and Cheney E. (Eds.), Proceed-
Vanderbei R.J. 1994. LOQO: An interior point code for quadratic pro- ings of the International Conference on Approximation theory in
gramming. TR SOR-94-15, Statistics and Operations Research, honour of George Lorenz, Academic Press, Austin, TX, pp. 8–10.
Princeton Univ., NJ. Wahba G. 1990. Spline Models for Observational Data, volume 59 of
Vanderbei R.J. 1997. LOQO user’s manual—version 3.10. Technical CBMS-NSF Regional Conference Series in Applied Mathematics.
Report SOR-97-08, Princeton University, Statistics and Oper- SIAM, Philadelphia.
ations Research, Code available at http://www.princeton.edu/ Wahba G. 1999. Support vector machines, reproducing kernel Hilbert
˜rvdb/. spaces and the randomized GACV. In: Schölkopf B., Burges
Vapnik V. 1995. The Nature of Statistical Learning Theory. Springer, C.J.C., and Smola A.J. (Eds.), Advances in Kernel Methods—
New York. Support Vector Learning, MIT Press, Cambridge, MA. pp. 69–88.
Vapnik V. 1998. Statistical Learning Theory. John Wiley and Sons, Weston J., Gammerman A., Stitson M., Vapnik V., Vovk V., and Watkins
New York. C. 1999. Support vector density estimation. In: Schölkopf B.,
Vapnik. V. 1999. Three remarks on the support vector method of Burges C.J.C., and Smola A.J. (Eds.) Advances in Kernel
function estimation. In: Schölkopf B., Burges C.J.C., and Smola Methods—Support Vector Learning, MIT Press, Cambridge,
A.J. (Eds.), Advances in Kernel Methods—Support Vector MA. pp. 293–306.
Learning, MIT Press, Cambridge, MA, pp. 25–42. Williams C.K.I. 1998. Prediction with Gaussian processes: From linear
Vapnik V. and Chervonenkis A. 1964. A note on one class of regression to linear prediction and beyond. In: Jordan M.I. (Ed.),
perceptrons. Automation and Remote Control, 25. Learning and Inference in Graphical Models, Kluwer Academic,
Vapnik V. and Chervonenkis A. 1974. Theory of Pattern Recognition pp. 599–621.
[in Russian]. Nauka, Moscow. (German Translation: Wapnik Williamson R.C., Smola A.J., and Schölkopf B. 1998. Generalization
W. & Tscherwonenkis A., Theorie der Zeichenerkennung, performance of regularization networks and support vector
Akademie-Verlag, Berlin, 1979). machines via entropy numbers of compact operators. Technical
Vapnik V., Golowich S., and Smola A. 1997. Support vector method Report 19, NeuroCOLT, http://www.neurocolt.com. Published
for function approximation, regression estimation, and signal in IEEE Transactions on Information Theory, 47(6): 2516–2532
processing. In: Mozer M.C., Jordan M.I., and Petsche T. (Eds.) (2001).
Advances in Neural Information Processing Systems 9, MA, MIT Yuille A. and Grzywacz N. 1988. The motion coherence theory.
Press, Cambridge. pp. 281–287. In: Proceedings of the International Conference on Computer
Vapnik V. and Lerner A. 1963. Pattern recognition using generalized Vision, IEEE Computer Society Press, Washington, DC, pp. 344–
portrait method. Automation and Remote Control, 24: 774–780. 354.
A Tutorial on ν-Support Vector Machines
Pai-Hsuen Chen1 , Chih-Jen Lin1 , and Bernhard Schölkopf2?

1
Department of Computer Science and Information Engineering
National Taiwan University
Taipei 106, Taiwan
2
Max Planck Institute for Biological Cybernetics, Tübingen, Germany
bernhard.schoelkopf@tuebingen.mpg.de
Abstract. We briefly describe the main ideas of statistical learning theory, sup-
port vector machines (SVMs), and kernel feature spaces. We place particular em-
phasis on a description of the so-called ν-SVM, including details of the algorithm
and its implementation, theoretical results, and practical applications.
1 An Introductory Example
Suppose we are given empirical data
(x1 , y1 ), . . . , (xm , ym ) ∈ X × {±1}. (1)
Here, the domain X is some nonempty set that the patterns xi are taken from; the yi
are called labels or targets.
Unless stated otherwise, indices i and j will always be understood to run over the
training set, i.e., i, j = 1, . . . , m.
Note that we have not made any assumptions on the domain X other than it being a
set. In order to study the problem of learning, we need additional structure. In learning,
we want to be able to generalize to unseen data points. In the case of pattern recognition,
this means that given some new pattern x ∈ X , we want to predict the corresponding
y ∈ {±1}. By this we mean, loosely speaking, that we choose y such that (x, y) is in
some sense similar to the training examples. To this end, we need similarity measures in
X and in {±1}. The latter is easy, as two target values can only be identical or different.
For the former, we require a similarity measure
k : X × X → R,
(x, x0 ) 7→ k(x, x0 ), (2)
i.e., a function that, given two examples x and x0 , returns a real number characterizing
their similarity. For reasons that will become clear later, the function k is called a kernel
([24], [1], [8]).
A type of similarity measure that is of particular mathematical appeal are dot prod-
ucts. For instance, given two vectors x, x0 ∈ RN , the canonical dot product is defined
?
Parts of the present article are based on [31].
as
N
X
(x · x0 ) := (x)i (x0 )i . (3)
i=1
Here, (x)i denotes the ith entry of x.

The geometrical interpretation of this dot product is that it computes the cosine of
the angle between the vectors x and x0 , provided they are normalized
p to length 1. More-
over, it allows computation of the length of a vector x as (x · x), and of the distance
between two vectors as the length of the difference vector. Therefore, being able to
compute dot products amounts to being able to carry out all geometrical constructions
that can be formulated in terms of angles, lengths and distances.
Note, however, that we have not made the assumption that the patterns live in a
dot product space. In order to be able to use a dot product as a similarity measure, we
therefore first need to transform them into some dot product space H, which need not
be identical to RN . To this end, we use a map
Φ:X →H
x 7→ x. (4)
The space H is called a feature space. To summarize, there are three benefits to trans-
form the data into H
1. It lets us define a similarity measure from the dot product in H,
k(x, x0 ) := (x · x0 ) = (Φ(x) · Φ(x0 )). (5)
2. It allows us to deal with the patterns geometrically, and thus lets us study learning
algorithm using linear algebra and analytic geometry.
3. The freedom to choose the mapping Φ will enable us to design a large variety of
learning algorithms. For instance, consider a situation where the inputs already live
in a dot product space. In that case, we could directly define a similarity measure
as the dot product. However, we might still choose to first apply a nonlinear map Φ
to change the representation into one that is more suitable for a given problem and
learning algorithm.
We are now in the position to describe a pattern recognition learning algorithm that
is arguable one of the simplest possible. The basic idea is to compute the means of the
two classes in feature space,
1 X
c+ = xi , (6)
m+
{i:yi =+1}
1 X
c− = xi , (7)
m−
{i:yi =−1}
where m+ and m− are the number of examples with positive and negative labels, re-
spectively (see Figure 1). We then assign a new point x to the class whose mean is
o
+
+ . o
w o c-
+ c
c+ x-c
+
x
Fig. 1. A simple geometric classification algorithm: given two classes of points (depicted by ‘o’
and ‘+’), compute their means c+ , c− and assign a test pattern x to the one whose mean is closer.
This can be done by looking at the dot product between x − c (where c = (c+ + c− )/2) and
w := c+ − c− , which changes sign as the enclosed angle passes through π/2. Note that the
corresponding decision boundary is a hyperplane (the dotted line) orthogonal to w (from [31]).
closer to it. This geometrical construction can be formulated in terms of dot products.
Half-way in between c+ and c− lies the point c := (c+ + c− )/2. We compute the class
of x by checking whether the vector connecting c and x encloses an angle smaller than
π/2 with the vector w := c+ − c− connecting the class means, in other words
y = sgn ((x − c) · w)
y = sgn ((x − (c+ + c− )/2) · (c+ − c− ))
= sgn ((x · c+ ) − (x · c− ) + b). (8)
Here, we have defined the offset
1¡ ¢
b := kc− k2 − kc+ k2 . (9)
2
It will be proved instructive to rewrite this expression in terms of the patterns xi in

the input domain X . To this end, note that we do not have a dot product in X , all we
have is the similarity measure k (cf. (5)). Therefore, we need to rewrite everything in
terms of the kernel k evaluated on input patterns. To this end, substitute (6) and (7) into
(8) to get the decision function
 
1 X 1 X
y = sgn  (x · xi ) − (x · xi ) + b
m+ m−
{i:yi =+1} {i:yi =−1}
 
1 X 1 X
= sgn  k(x, xi ) − k(x, xi ) + b . (10)
m+ m−
{i:yi =+1} {i:yi =−1}
Similarly, the offset becomes
 
1 1 X 1 X
b :=  2 k(xi , xj ) − k(xi , xj ) . (11)
2 m− m2+
{(i,j):yi =yj =−1} {(i,j):yi =yj =+1}
Let us consider one well-known special case of this type of classifier. Assume that the
class means have the same distance to the origin (hence b = 0), and that k can be viewed
as a density, i.e., it is positive and has integral 1,
Z
k(x, x0 )dx = 1 for all x0 ∈ X . (12)
X
In order to state this assumption, we have to require that we can define an integral on
X.
If the above holds true, then (10) corresponds to the so-called Bayes decision bound-
ary separating the two classes, subject to the assumption that the two classes were gen-
erated from two probability distributions that are correctly estimated by the Parzen
windows estimators of the two classes,
1 X
p1 (x) := k(x, xi ) (13)
m+
{i:yi =+1}
1 X
p2 (x) := k(x, xi ). (14)
m−
{i:yi =−1}
Given some point x, the label is then simply computed by checking which of the two,
p1 (x) or p2 (x), is larger, which directly leads to (10). Note that this decision is the best
we can do if we have no prior information about the probabilities of the two classes.
For further details, see [31].
The classifier (10) is quite close to the types of learning machines that we will
be interested in. It is linear in the feature space, and while in the input domain, it is
represented by a kernel expansion in terms of the training points. It is example-based
in the sense that the kernels are centered on the training examples, i.e., one of the two
arguments of the kernels is always a training example. The main points that the more
sophisticated techniques to be discussed later will deviate from (10) are in the selection
of the examples that the kernels are centered on, and in the weights that are put on the
individual data in the decision function. Namely, it will no longer be the case that all
training examples appear in the kernel expansion, and the weights of the kernels in the
expansion will no longer be uniform. In the feature space representation, this statement
corresponds to saying that we will study all normal vectors w of decision hyperplanes
that can be represented as linear combinations of the training examples. For instance,
we might want to remove the influence of patterns that are very far away from the
decision boundary, either since we expect that they will not improve the generalization
error of the decision function, or since we would like to reduce the computational cost
of evaluating the decision function (cf. (10)). The hyperplane will then only depend on
a subset of training examples, called support vectors.
2 Learning Pattern Recognition from Examples
With the above example in mind, let us now consider the problem of pattern recognition
in a more formal setting ([37], [38]), following the introduction of [30]. In two-class
pattern recognition, we seek to estimate a function
f : X → {±1} (15)
based on input-output training data (1). We assume that the data were generated inde-
pendently from some unknown (but fixed) probability distribution P (x, y). Our goal
is to learn a function that will correctly classify unseen examples (x, y), i.e., we want
f (x) = y for examples (x, y) that were also generated from P (x, y).
If we put no restriction on the class of functions that we choose our estimate f
from, however, even a function which does well on the training data, e.g. by satisfying
f (xi ) = yi for all i = 1, . . . , m, need not generalize well to unseen examples. To see
this, note that for each function f and any test set (x̄1 , ȳ1 ), . . . , (x̄m̄ , ȳm̄ ) ∈ RN ×{±1},
satisfying {x̄1 , . . . , x̄m̄ } ∩ {x1 , . . . , xm } = {}, there exists another function f ∗ such
that f ∗ (xi ) = f (xi ) for all i = 1, . . . , m, yet f ∗ (x̄i ) 6= f (x̄i ) for all i = 1, . . . , m̄.
As we are only given the training data, we have no means of selecting which of the two
functions (and hence which of the completely different sets of test label predictions) is
preferable. Hence, only minimizing the training error (or empirical risk),
m
1 X1
Remp [f ] = |f (xi ) − yi |, (16)
m i=1 2
does not imply a small test error (called risk), averaged over test examples drawn from
the underlying distribution P (x, y),
Z
1
R[f ] = |f (x) − y| dP (x, y). (17)
2
Statistical learning theory ([41], [37], [38], [39]), or VC (Vapnik-Chervonenkis) theory,
shows that it is imperative to restrict the class of functions that f is chosen from to one
which has a capacity that is suitable for the amount of available training data. VC theory
provides bounds on the test error. The minimization of these bounds, which depend on
both the empirical risk and the capacity of the function class, leads to the principle of
structural risk minimization ([37]). The best-known capacity concept of VC theory is
the VC dimension, defined as the largest number h of points that can be separated in
all possible ways using functions of the given class. An example of a VC bound is the
following: if h < m is the VC dimension of the class of functions that the learning
machine can implement, then for all functions of that class, with a probability of at
least 1 − η, the bound
µ ¶
h log(η)
R(f ) ≤ Remp (f ) + φ , (18)
m m
holds, where the confidence term φ is defined as
s ¡ ¢
µ ¶
h log(η) h log 2m h + 1 − log(η/4)
φ , = . (19)
m m m
Tighter bounds can be formulated in terms of other concepts, such as the annealed VC
entropy or the Growth function. These are usually considered to be harder to evaluate,
but they play a fundamental role in the conceptual part of VC theory ([38]). Alterna-
tive capacity concepts that can be used to formulate bounds include the fat shattering
dimension ([2]).
The bound (18) deserves some further explanatory remarks. Suppose we wanted to
learn a “dependency” where P (x, y) = P (x) · P (y), i.e., where the pattern x contains
no information about the label y, with uniform P (y). Given a training sample of fixed
size, we can then surely come up with a learning machine which achieves zero training
error (provided we have no examples contradicting each other). However, in order to
reproduce the random labelling, this machine will necessarily require a large VC di-
mension h. Thus, the confidence term (19), increasing monotonically with h, will be
large, and the bound (18) will not support possible hopes that due to the small training
error, we should expect a small test error. This makes it understandable how (18) can
hold independent of assumptions about the underlying distribution P (x, y): it always
holds (provided that h < m), but it does not always make a nontrivial prediction — a
bound on an error rate becomes void if it is larger than the maximum error rate. In order
to get nontrivial predictions from (18), the function space must be restricted such that
the capacity (e.g. VC dimension) is small enough (in relation to the available amount
of data).
3 Hyperplane Classifiers
In the present section, we shall describe a hyperplane learning algorithm that can be
performed in a dot product space (such as the feature space that we introduced previ-
ously). As described in the previous section, to design learning algorithms, one needs
to come up with a class of functions whose capacity can be computed.
[42] considered the class of hyperplanes
(w · x) + b = 0 w ∈ RN , b ∈ R, (20)
corresponding to decision functions
f (x) = sgn ((w · x) + b), (21)
and proposed a learning algorithm for separable problems, termed the Generalized Por-
trait, for constructing f from empirical data. It is based on two facts. First, among all
hyperplanes separating the data, there exists a unique one yielding the maximum margin
of separation between the classes,
max min{kx − xi k : x ∈ RN , (w · x) + b = 0, i = 1, . . . , m}. (22)
w,b
Second, the capacity decreases with increasing margin.
To construct this Optimal Hyperplane (cf. Figure 2), one solves the following opti-
mization problem:
1
minimize kwk2
w,b 2
subject to yi · ((w · xi ) + b) ≥ 1, i = 1, . . . , m. (23)
{x | (w . x) + b = +1}
{x | (w . x) + b = −1} Note:
◆
(w . x1) + b = +1
❍ ◆ x1 yi = +1 (w . x2) + b = −1
x2❍
=> (w . (x1−x2)) = 2
◆
. w . 2
yi = −1 w ◆ => ( )
||w|| (x1−x2) = ||w||
❍
❍
❍
{x | (w . x) + b = 0}
Fig. 2. A binary classification toy problem: separate balls from diamonds. The optimal hyperplane
is orthogonal to the shortest line connecting the convex hulls of the two classes (dotted), and
intersects it half-way between the two classes. The problem is separable, so there exists a weight
vector w and a threshold b such that yi · ((w · xi ) + b) > 0 (i = 1, . . . , m). Rescaling w and b
such that the point(s) closest to the hyperplane satisfy |(w · xi ) + b| = 1, we obtain a canonical
form (w, b) of the hyperplane, satisfying yi · ((w · xi ) + b) ≥ 1. Note that in this case, the margin,
measured perpendicularly to the hyperplane, equals 2/kwk. This can be seen by considering two
points x1 , x2 on opposite sides of the margin, i.e., (w · x1 ) + b = 1, (w · x2 ) + b = −1, and
projecting them onto the hyperplane normal vector w/kwk (from [29]).
A way to solve (23) is through its Lagrangian dual:

max(min L(w, b, α)), (24)
α≥0 w,b
where
Xm
1
L(w, b, α) = kwk2 − αi (yi · ((xi · w) + b) − 1) . (25)
2 i=1
The Lagrangian L has to be minimized with respect to the primal variables w and
b and maximized with respect to the dual variables αi . For a nonlinear problem like
(23), called the primal problem, there are several closely related problems of which
the Lagrangian dual is an important one. Under certain conditions, the primal and dual
problems have the same optimal objective values. Therefore, we can instead solve the
dual which may be an easier problem than the primal. In particular, we will see in
Section 4 that when working in feature spaces, solving the dual may be the only way to
train SVM.
Let us try to get some intuition for this primal-dual relation. Assume (w̄, b̄) is an
optimal solution of the primal with the optimal objective value γ = 12 kw̄k2 . Thus, no
(w, b) satisfies
1
kwk2 < γ and yi · ((w · xi ) + b) ≥ 1, i = 1, . . . , m. (26)
2
With (26), there is ᾱ ≥ 0 such that for all w, b
Xm
1
kwk2 − γ − ᾱi (yi · ((xi · w) + b) − 1) ≥ 0. (27)
2 i=1
We do not provide a rigorous proof here but details can be found in, for example, [5].
Note that for general convex programming this result requires some additional condi-
tions on constraints which are now satisfied by our simple linear inequalities.
Therefore, (27) implies
max min L(w, b, α) ≥ γ. (28)

α≥0 w,b
On the other hand, for any α,
min L(w, b, α) ≤ L(w̄, b̄, α),

w,b
so
1
max min L(w, b, α) ≤ max L(w̄, b̄, α) = kw̄k2 = γ. (29)
α≥0 w,b α≥0 2
Therefore, with (28), the inequality in (29) becomes an equality. This property is
the strong duality where the primal and dual have the same optimal objective value. In
addition, putting (w̄, b̄) into (27), with ᾱi ≥ 0 and yi · ((xi · w̄) + b̄) − 1 ≥ 0,
ᾱi · [yi ((xi · w̄) + b̄) − 1] = 0, i = 1, . . . , m, (30)
which is usually called the complementarity condition.

To simplify the dual, as L(w, b, α) is convex when α is fixed, for any given α,
∂ ∂
L(w, b, α) = 0, L(w, b, α) = 0, (31)
∂b ∂w
leads to
m
X
αi yi = 0 (32)
i=1
and
m
X
w= αi yi xi . (33)
i=1
As α is nowPgiven, we may wonder what (32) means. Pm From the definition of the La-
m
grangian, if i=1 αi yi 6= 0, we can decrease −b i=1 αi yi in L(w, b, α) as much as
we want. Therefore, by substituting (33) into (24), the dual problem can be written as
(P Pm Pm
m 1
i=1 αi − 2 i,j=1 αi αj yi yj (xi · xj ) if i=1 αi yi = 0,
max Pm (34)
α≥0 −∞ if i=1 αi yi 6= 0.
As −∞ is definitely not the maximal
Pm objective value of the dual, the dual optimal so-
lution does not happen when i=1 αi yi 6= 0. Therefore, the dual problem is simplified
to finding multipliers αi which
m
X m
1 X
maximize αi − αi αj yi yj (xi · xj ) (35)
α∈Rm 2 i,j=1
i=1
m
X
subject to αi ≥ 0, i = 1, . . . , m, and αi yi = 0. (36)
i=1
This is the dual SVM problem that we usually refer to. Note that (30), (32), αi ≥ 0∀i,
and (33), are called the Karush-Kuhn-Tucker (KKT) optimality conditions of the primal
problem. Except an abnormal situation where all optimal αi are zero, b can be computed
using (30).
The discussion from (31) to (33) implies that we can consider a different form of
dual problem:
maximize L(w, b, α)
w,b,α≥0
∂ ∂ (37)
subject to L(w, b, α) = 0, L(w, b, α) = 0.
∂b ∂w
This is the so called Wolfe dual for convex optimization, which is a very early work in
duality [45]. For convex and differentiable problems, it is equivalent to the Lagrangian
dual though the derivation of the Lagrangian dual more easily shows the strong duality
results. Some notes about the two duals are in, for example, [3, Section 5.4].
Following the above discussion, the hyperplane decision function can be written as
Ãm !
X
f (x) = sgn yi αi · (x · xi ) + b . (38)
i=1
The solution vector w thus has an expansion in terms of a subset of the training pat-
terns, namely those patterns whose αi is non-zero, called Support Vectors. By (30), the
Support Vectors lie on the margin (cf. Figure 2). All remaining examples of the training
set are irrelevant: their constraint (23) does not play a role in the optimization, and they
do not appear in the expansion (33). This nicely captures our intuition of the problem:
as the hyperplane (cf. Figure 2) is completely determined by the patterns closest to it,
the solution should not depend on the other examples.
The structure of the optimization problem closely resembles those that typically
arise in Lagrange’s formulation of mechanics. Also there, often only a subset of the
constraints become active. For instance, if we keep a ball in a box, then it will typically
roll into one of the corners. The constraints corresponding to the walls which are not
touched by the ball are irrelevant, the walls could just as well be removed.
Seen in this light, it is not too surprising that it is possible to give a mechanical in-
terpretation of optimal margin hyperplanes ([9]): If we assume that each support vector
xi exerts a perpendicular force of size αi and sign yi on a solid plane sheet lying along
the hyperplane, then the solution satisfies the requirements of mechanical stability. The
constraint (32) states that the P
forces on the sheet sum to zero; and (33) implies that the
torques also sum to zero, via i xi × yi αi · w/kwk = w × w/kwk = 0.
There are theoretical arguments supporting the good generalization performance of
the optimal hyperplane ([41], [37], [4], [33], [44]). In addition, it is computationally
attractive, since it can be constructed by solving a quadratic programming problem.
4 Optimal Margin Support Vector Classifiers
We now have all the tools to describe support vector machines ([38], [31]). Everything
in the last section was formulated in a dot product space. We think of this space as the
feature space H described in Section 1. To express the formulas in terms of the input
patterns living in X , we thus need to employ (5), which expresses the dot product of
bold face feature vectors x, x0 in terms of the kernel k evaluated on input patterns x, x0 ,
k(x, x0 ) = (x · x0 ). (39)
This can be done since all feature vectors only occurred in dot products. The weight
vector (cf. (33)) then becomes an expansion in feature space,1 and will thus typically
no longer correspond to the image of a single vector from input space. We thus obtain
decision functions of the more general form (cf. (38))
Ãm !
X
f (x) = sgn yi αi · (Φ(x) · Φ(xi )) + b
i=1
Ãm !
X
= sgn yi αi · k(x, xi ) + b , (40)
i=1
and the following quadratic program (cf. (35)):

m
X m
1 X
maximize W (α) = αi − αi αj yi yj k(xi , xj ) (41)
α∈Rm 2 i,j=1
i=1
m
X
subject to αi ≥ 0, i = 1, . . . , m, and αi yi = 0. (42)
i=1
Working in the feature space somewhat forces us to solve the dual problem instead
of the primal. The dual problem has the same number of variables as the number of
training data. However, the primal problem may have a lot more (even infinite) variables
depending on the dimensionality of the feature space (i.e. the length of Φ(x)). Though
our derivation of the dual problem in Section 3 considers problems in finite-dimensional
spaces, it can be directly extended to problems in Hilbert spaces [20].
1
This constitutes a special case of the so-called representer theorem, which states that under
fairly general conditions, the minimizers of objective functions which contain a penalizer in
terms of a norm in feature space will have kernel expansions ([43], [31]).
Fig. 3. Example of a Support Vector classifier found by using a radial basis function kernel
k(x, x0 ) = exp(−kx − x0 k2 ). Both coordinate axes range from -1 to +1. Circles and disks
are two classes of training examples; the middle line is the decision surface; the outer lines pre-
cisely meet the constraint (23). Note that the Support Vectors found by the algorithm (marked by
extra circles) are not centers of clusters, but examples which
P are critical for the given classifica-
tion task. Grey values code the modulus of the argument m i=1 yi αi · k(x, xi ) + b of the decision
function (40) (from [29]).)
5 Kernels
We now take a closer look at the issue of the similarity measure, or kernel, k. In this
section, we think of X as a subset of the vector space RN , (N ∈ N), endowed with the
canonical dot product (3).
5.1 Product Features
Suppose we are given patterns x ∈ RN where most information is contained in the dth
order products (monomials) of entries [x]j of x,
[x]j1 · · · · · [x]jd , (43)
where j1 , . . . , jd ∈ {1, . . . , N }. In that case, we might prefer to extract these product

features, and work in the feature space H of all products of d entries. In visual recog-
nition problems, where images are often represented as vectors, this would amount to
extracting features which are products of individual pixels.
For instance, in R2 , we can collect all monomial feature extractors of degree 2 in
the nonlinear map
Φ : R2 → H = R3 (44)
([x]1 , [x]2 ) 7→ ([x]21 , [x]22 , [x]1 [x]2 ). (45)
This approach works fine for small toy examples, but it fails for realistically sized prob-
lems: for N -dimensional input patterns, there exist
(N + d − 1)!
NH = (46)
d!(N − 1)!
different monomials (43), comprising a feature space H of dimensionality NH . For

instance, already 16 × 16 pixel input images and a monomial degree d = 5 yield a
dimensionality of 1010 .
In certain cases described below, there exists, however, a way of computing dot
products in these high-dimensional feature spaces without explicitly mapping into them:
by means of kernels nonlinear in the input space RN . Thus, if the subsequent process-
ing can be carried out using dot products exclusively, we are able to deal with the high
dimensionality.
5.2 Polynomial Feature Spaces Induced by Kernels
In order to compute dot products of the form (Φ(x) · Φ(x0 )), we employ kernel repre-
sentations of the form
k(x, x0 ) = (Φ(x) · Φ(x0 )), (47)
which allow us to compute the value of the dot product in H without having to carry
out the map Φ. This method was used by Boser et al. to extend the Generalized Por-
trait hyperplane classifier [41] to nonlinear Support Vector machines [8]. Aizerman et
al. called H the linearization space, and used in the context of the potential function
classification method to express the dot product between elements of H in terms of
elements of the input space [1].
What does k look like for the case of polynomial features? We start by giving an
example ([38]) for N = d = 2. For the map
Φ2 : ([x]1 , [x]2 ) 7→ ([x]21 , [x]22 , [x]1 [x]2 , [x]2 [x]1 ), (48)
dot products in H take the form
(Φ2 (x) · Φ2 (x0 )) = [x]21 [x0 ]21 + [x]22 [x0 ]22 + 2[x]1 [x]2 [x0 ]1 [x0 ]2 = (x · x0 )2 , (49)
i.e., the desired kernel k is simply the square of the dot product in input space. Note that
it is possible to modify (x · x0 )d such that it maps into the space of all monomials up to
degree d, defining ([38])
k(x, x0 ) = ((x · x0 ) + 1)d . (50)
5.3 Examples of Kernels
When considering feature maps, it is also possible to look at things the other way
around, and start with the kernel. Given a kernel function satisfying a mathematical
condition termed positive definiteness, it is possible to construct a feature space such
that the kernel computes the dot product in that feature space. This has been brought
to the attention of the machine learning community by [1], [8], and [38]. In functional
analysis, the issue has been studied under the heading of Reproducing kernel Hilbert
space (RKHS).
Besides (50), a popular choice of kernel is the Gaussian radial basis function ([1])
¡ ¢
k(x, x0 ) = exp −γkx − x0 k2 . (51)
An illustration is in Figure 3. For an overview of other kernels, see [31].
6 ν-Soft Margin Support Vector Classifiers

In practice, a separating hyperplane may not exist, e.g. if a high noise level causes a
large overlap of the classes. To allow for the possibility of examples violating (23), one
introduces slack variables ([15], [38], [32])
ξi ≥ 0, i = 1, . . . , m (52)
in order to relax the constraints to
yi · ((w · xi ) + b) ≥ 1 − ξi , i = 1, . . . , m. (53)
A classifier which generalizes well is then P

found by controlling both the classifier ca-
pacity (via kwk) and the sum of the slacks i ξi . The latter is done as it can be shown
to provide an upper bound on the number of training errors which leads to a convex
optimization problem.
One possible realization,called C-SVC, of a soft margin classifier is minimizing the
objective function
Xm
1 2
τ (w, ξ) = kwk + C ξi (54)
2 i=1
subject to the constraints (52) and (53), for some value of the constant C > 0 deter-
mining the trade-off. Here and below, we use boldface Greek letters as a shorthand
for corresponding vectors ξ = (ξ1 , . . . , ξm ). Incorporating kernels, and rewriting it
in terms of Lagrange multipliers, this again leads to the problem of maximizing (41),
subject to the constraints
m
X
0 ≤ αi ≤ C, i = 1, . . . , m, and αi yi = 0. (55)
i=1
The only difference from the separable case is the upper bound C on the Lagrange mul-
tipliers αi . This way, the influence of the individual patterns (which could be outliers)
gets limited. As above, the solution takes the form (40).
Another possible realization,called ν-SVC of a soft margin variant of the optimal
hyperplane uses the ν-parameterization ([32]). In it, the parameter C is replaced by a
parameter ν ∈ [0, 1] which is the lower and upper bound on the number of examples
that are support vectors and that lie on the wrong side of the hyperplane, respectively.
As a primal problem for this approach, termed the ν-SV classifier, we consider
m
1 1 X
minimize τ (w, ξ, ρ) = kwk2 − νρ + ξi (56)
w∈H,ξ∈Rm ,ρ,b∈R 2 m i=1
subject to yi (hxi , wi + b) ≥ ρ − ξi , i = 1, . . . , m (57)
and ξi ≥ 0, ρ ≥ 0. (58)
Note that no constant C appears in this formulation; instead, there is a parameter ν, and
also an additional variable ρ to be optimized. To understand the role of ρ, note that for
ξ = 0, the constraint (57) simply states that the two classes are separated by the margin
2ρ/kwk.
To explain the significance of ν, let us first introduce the term margin error: by this,
we denote training points with ξi > 0. These are points which either are errors, or lie
within the margin. Formally, the fraction of margin errors is
ρ 1
Remp [g] := |{i|yi g(xi ) < ρ}| . (59)
m
Here, g is used to denote the argument of the sign in the decision function (40): f =
sgn ◦g. We are now in a position to state a result that explains the significance of ν.
Proposition 1 ([32]). Suppose we run ν-SVC with kernel function k on some data with
the result that ρ > 0. Then
(i) ν is an upper bound on the fraction of margin errors (and hence also on the fraction
of training errors).
(ii) ν is a lower bound on the fraction of SVs.
(iii) Suppose the data (x1 , y1 ), . . . , (xm , ym ) were generated iid from a distribution
Pr(x, y) = Pr(x) Pr(y|x), such that neither Pr(x, y = 1) nor Pr(x, y = −1) con-
tains any discrete component. Suppose, moreover, that the kernel used is analytic
and non-constant. With probability 1, asymptotically, ν equals both the fraction of
SVs and the fraction of margin errors.
Before we get into the technical details of the dual derivation, let us take a look at
a toy example illustrating the influence of ν (Figure 4). The corresponding fractions of
SVs and margin errors are listed in table 1.
Let us next derive the dual of the ν-SV classification algorithm. We consider the
Lagrangian
m
1 1 X
L(w, ξ, b, ρ, α, β, δ) = kwk2 − νρ + ξi
2 m i=1
m
X
− (αi (yi (hxi , wi + b) − ρ + ξi ) + βi ξi − δρ), (60)
i=1
Fig. 4. Toy problem (task: to separate circles from disks) solved using ν-SV classification, with
parameter values ranging from ν = 0.1 (top left) to ν = 0.8 (bottom right). The larger we make
ν, the more points are allowed to lie inside the margin (depicted by dotted lines). Results are
shown for a Gaussian kernel, k(x, x0 ) = exp(−kx − x0 k2 ) (from [31]).
Table 1. Fractions of errors and SVs, along with the margins of class separation, for the toy
example in Figure 4.
Note that ν upper bounds the fraction of errors and lower bounds the fraction of SVs, and that
increasing ν, i.e., allowing more errors, increases the margin.
ν 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

fraction of errors 0.00 0.07 0.25 0.32 0.39 0.50 0.61 0.71
fraction of SVs 0.29 0.36 0.43 0.46 0.57 0.68 0.79 0.86
margin ρ/kwk 0.005 0.018 0.115 0.156 0.364 0.419 0.461 0.546
using multipliers αi , βi , δ ≥ 0. This function has to be minimized with respect to the

primal variables w, ξ, b, ρ, and maximized with respect to the dual variables α, β, δ.
Following the same derivation in (31)–(33), we compute the corresponding partial
derivatives and set them to 0, obtaining the following conditions:
m
X
w= αi yi xi , (61)
i=1
αi + βi = 1/m, (62)
m
X
αi yi = 0, (63)
i=1
m
X
αi − δ = ν. (64)
i=1
Again, in the SV expansion (61), the αi that are non-zero correspond to a constraint (57)
which is precisely met.
Substituting (61) and (62) into L, using αi , βi , δ ≥ 0, and incorporating kernels
for dot products, leaves us with the following quadratic optimization problem for ν-SV
classification:
m
1 X
maximize W (α) = − αi αj yi yj k(xi , xj ) (65)
α∈Rm 2 i,j=1
1
subject to 0 ≤ αi ≤ , (66)
m
m
X
αi yi = 0, (67)
i=1
Xm
αi ≥ ν. (68)
i=1
As above, the resulting decision function can be shown to take the form
Ãm !
X
f (x) = sgn αi yi k(x, xi ) + b . (69)
i=1
Compared with the C-SVC dual ((41), (55)), there are Pm two differences. First, there is
an additional constraint (68). Second, the linear term i=1 αi no longer appears in the
objective function (65). This has an interesting consequence: (65) is now quadratically
homogeneous in α. It is straightforward to verify that the same decision function is
obtained if we start with the primal function
Ã m
!
1 2 1 X
τ (w, ξ, ρ) = kwk + C −νρ + ξi , (70)
2 m i=1
i.e., if one does use C [31].

The computation of the threshold b and the margin parameter ρ will be discussed in
Section 7.4.
A connection to standard SV classification, and a somewhat surprising interpreta-
tion of the regularization parameter C, is described by the following result:
Proposition 2 (Connection ν-SVC — C-SVC [32]). If ν-SV classification leads to
ρ > 0, then C-SV classification, with C set a priori to 1/mρ, leads to the same decision
function.
For further details on the connection between ν-SVMs and C-SVMs, see [16, 6]. By
considering the optimal α as a function of parameters, a complete account is as follows:
Pm
Proposition 3 (Detailed connection ν-SVC — C-SVC [11]). i=1 αi /(Cm) by the
C-SVM is a well defined decreasing function of C. We can define
Pm Pm
i=1 αi i=1 αi
lim = νmin ≥ 0 and lim = νmax ≤ 1. (71)
C→∞ Cm C→0 Cm
Then,
1. νmax = 2 min(m+ , m− )/m.
2. For any ν > νmax , the dual ν-SVM is infeasible. That is, the set of feasible points is
empty. For any ν ∈ (νmin , νmax ], the optimal solution set of dual ν-SVM is the same
as that of either one or some C-SVM where these C form an interval. In addition,
the optimal objective value of ν-SVM is strictly positive. For any 0 ≤ ν ≤ νmin ,
dual ν-SVM is feasible with zero optimal objective value.
3. If the kernel matrix is positive definite, then νmin = 0.
Therefore, for a given problem and kernel, there is an interval [νmin , νmax ] of admissible
values for ν, with 0 ≤ νmin ≤ νmax ≤ 1. An illustration of the relation between ν and
C is in Figure 5.
0.9
0.8
0.7
0.6
0.5
ν
0.4
0.3
0.2
0.1
0
-3 -2 -1 0 1 2 3 4 5
log10 C
Fig. 5. The relation between ν and C (using the RBF kernel on the problem australian from the
Statlog collection [25])
It has been noted that ν-SVMs have an interesting interpretation in terms of reduced
convex hulls [16, 6]. One can show that for separable problems, one can obtain the opti-
mal margin separating hyperplane by forming the convex hulls of both classes, finding
the shortest connection between the two convex hulls (since the problem is separable,
they are disjoint), and putting the hyperplane halfway along that connection, orthogonal
to it. If a problem is non-separable, however, the convex hulls of the two classes will
no longer be disjoint. Therefore, it no longer makes sense to search for the shortest line
connecting them. In this situation, it seems natural to reduce the convex hulls in size,
by limiting the size of the coefficients ci in the convex sets
( ¯ )
X ¯ X
¯
C± := ci xi ¯ ci = 1, ci ≥ 0 . (72)
¯
yi =±1 yi =±1
to some value ν ∈ (0, 1). Intuitively, this amounts to limiting the influence of individual
points. It is possible to show that the ν-SVM formulation solves the problem of finding
the hyperplane orthogonal to the closest line connecting the reduced convex hulls [16].
We now move on to another aspect of soft margin classification. When we intro-
duced the slack variables, wePdid not attempt to justify the fact that in the P objective
m m
function, we used a penalizer i=1 ξi . Why not use another penalizer, such as i=1 ξip ,
for some p ≥ 0 [15]? For instance, p = 0 would yield a penalizer that exactly counts
the number of margin errors. Unfortunately, however, it is also a penalizer that leads to
a combinatorial optimization problem. Penalizers yielding optimization problems that
are particularly convenient, on the other hand, are obtained for p = 1 and p = 2. By
default, we use the former, as it possesses an additional property which is statistically
attractive. As the following proposition shows, linearity of the target function in the
slack variables ξi leads to a certain “outlier” resistance of the estimator. As above, we
use the shorthand xi for Φ(xi ).
Proposition 4 (Resistance of SV classification [32]). Suppose w can be expressed in
terms of the SVs which are not at bound,
m
X
w= γi xi (73)
i=1
with γi 6= 0 only if αi ∈ (0, 1/m) (where the αi are the coefficients of the dual solu-
tion). Then local movements of any margin error xj parallel to w do not change the
hyperplane.2
This result is about the stability of classifiers. Results have also shown that in general
p = 1 leads to fewer support vectors. Further results in support of the p = 1 case can
be seen in [34, 36].
Although proposition 1 shows that ν possesses an intuitive meaning, it is still un-
clear how to choose ν for a learning task. [35] proves that given R̄, a close upper bound
on the expected optimal Bayes risk, an asymptotically good estimate of the optimal
value of ν is 2R̄:
Proposition 5. If R[f ] is the expected risk defined in (17),
Rp := inf R[f ], (74)

f
and the kernel used by ν-SVM is universal, then for all ν > 2Rp and all ² > 0, there
exists a constant c > 0 such that
P (T = {(x1 , y1 ), . . . , (xm , ym )} | R[fTν ] ≤ ν − Rp + ²) ≥ 1 − e−cm . (75)
Quite a few popular kernels such as the Gaussian are universal. The definition of a
universal kernel can be seen in [35]. Here, fTν is the decision function obtained by
training ν-SVM on the data set T .
2
Note that the perturbation of the point is carried out in feature space. What it precisely corre-
sponds to in input space therefore depends on the specific kernel chosen.
Therefore, given an upper bound R̄ on Rp , the decision function with respect to ν = 2R̄
almost surely achieves a risk not larger than Rp + 2(R̄ − Rp ).
The selection of ν and kernel parameters can be done by estimating the performance
of support vector binary classifiers on data not yet observed. One such performance esti-
mate is the leave-one-out error, which is an almost unbiased estimate of the generaliza-
tion performance [22]. To compute this performance metric, a single point is excluded
from the training set, and the classifier is trained using the remaining points. It is then
determined whether this new classifier correctly labels the point that was excluded. The
process is repeated over the entire training set. Although theoretically attractive, this
estimate obviously entails a large computational cost.
Three estimates of the leave-one-out error for the ν-SV learning algorithm are pre-
sented in [17]. Of these three estimates, the general ν-SV bound is an upper bound on
the leave-one-out error, the restricted ν-SV estimate is an approximation that assumes
the sets of margin errors and support vectors on the margin to be constant, and the max-
imized target estimate is an approximation that assumes the sets of margin errors and
non-support vectors not to decrease. The derivation of the general ν-SV bound takes
a form similar to an upper bound described in [40] for the C-SV classifier, while the
restricted ν-SV estimate is based on a similar C-SV estimate proposed in [40, 26]: both
these estimates are based on the geometrical concept of the span, which is (roughly
speaking) a measure of how easily a particular point in the training sample can be
replaced by the other points used to define the classification function. No analogous
method exists in the C-SV case for the maximized target estimate.
7 Implementation of ν-SV Classifiers

We change the dual form of ν-SV classifiers to be a minimization problem:
m
1 X
minimize W (α) = αi αj yi yj k(xi , xj )
α∈Rm 2 i,j=1
1
subject to 0 ≤ αi ≤ , (76)
m
m
X
αi yi = 0,
i=1
Xm
αi = ν. (77)
i=1
[11] proves that for any given ν, there is at least an optimal solution which satisfies
eT α = ν. Therefore, it is sufficient to solve a simpler problem with the equality con-
straint (77).
Similar to C-SVC, the difficulty of solving (76) is that yi yj k(xi , xj ) are in gen-
eral not zero. Thus, for large data sets, the Hessian (second derivative) matrix of the
objective function cannot be stored in the computer memory, so traditional optimiza-
tion methods such as Newton or quasi Newton cannot be directly used. Currently, the
decomposition method is the most used approach to conquer this difficulty. Here, we
present the implementation in [11], which modifies the procedure for C-SVC.
7.1 The Decomposition Method
The decomposition method is an iterative process. In each step, the index set of variables
is partitioned to two sets B and N , where B is the working set. Then, in that iteration
variables corresponding to N are fixed while a sub-problem on variables corresponding
to B is minimized. The procedure is as follows:
Algorithm 1 (Decomposition method)

1. Given a number q ≤ l as the size of the working set. Find α1 as an initial feasible
solution of (76). Set k = 1.
2. If αk is an optimal solution of (76), stop. Otherwise, find a working set B ⊂
k k
{1, . . . , l} whose size is q . Define N ≡ {1, . . . , l}\B and αB and αN to be sub-
k
vectors of α corresponding to B and N , respectively.
3. Solve the following sub-problem with the variable αB :
1 X X
minimize αi αj yi yj k(xi , xj ) + αi αjk yi yj k(xi , xj )
αB ∈Rq 2
i∈B,j∈B i∈B,j∈N
1
subject to 0 ≤ αi ≤ , i ∈ B, (78)
X m
X
αi yi = − αik yi , (79)
i∈B i∈N
X X
αi = ν − αik . (80)
i∈B i∈N
k+1 k+1 k
4. Set αB to be the optimal solution of (78) and αN ≡ αN . Set k ← k + 1 and
goto Step 2.
Note that B is updated in each iteration. To simplify the notation, we simply use B
instead of B k .
7.2 Working Set Selection

An important issue of the decomposition method is the selection of the working set B.
Here, we consider an approach based on the violation of the KKT condition. Similar to
(30), by putting (61) into (57), one of the KKT conditions is
Xm
αi · [yi ( αj K(xi , xj ) + b) − ρ + ξi ] = 0, i = 1, . . . , m. (81)
j=1
1
Using 0 ≤ αi ≤ m, (81) can be rewritten as:
m
X 1
αj yi yj k(xi , xj ) + byi − ρ ≥ 0, if αi < ,
j=1
m
m
(82)
X
αj yi yj k(xi , xj ) + byi − ρ ≤ 0, if αi > 0.
j=1
That is, an α is optimal for the dual problem (76) if and only if α is feasible
Pm and satisfies
(81). Using the property that yi = ±1 and representing ∇W (α)i = j=1 αj yi yj K(xi , xj ),
(82) can be further written as
max ∇W (α)i ≤ ρ − b ≤
1 (α)
min ∇W (α)i and
i∈Iup 1
i∈Ilow (α)
(83)
max ∇W (α)i ≤ ρ + b ≤ min ∇W (α)i ,
−1
i∈Iup (α) −1
i∈Ilow (α)
where
1 1 1
Iup (α) := {i | αi > 0, yi = 1}, Ilow (α) := {i | αi < , yi = 1}, (84)
m
and
−1 1 −1
Iup (α) := {i | αi < , yi = −1}, Ilow (α) := {i | αi > 0, yi = −1}. (85)
m
1 1 −1 −1
We call any (i, j) ∈ Iup (α) × Ilow (α) or Iup (α) × Ilow (α) satisfying
yi ∇W (α)i > yj ∇W (α)j (86)
a violating pair as (83) is not satisfied. When α is not optimal yet, if any such a violating
pair is included in B, the optimal objective value of (78) is small than that at αk .
Therefore, the decomposition procedure has its objective value strictly decreasing from
one iteration to the next.
Therefore, a natural choice of B is to select all pairs which violate (83) the most.
To be more precise, we can set q to be an even integer and sequentially select q/2 pairs
1 1 −1 −1
{(i1 , j1 ), . . . , (iq/2 , jq/2 )} from ∈ Iup (α) × Ilow (α) or Iup (α) × Ilow (α) such that
yi1 ∇W (α)i1 − yj1 ∇W (α)j1 ≥ · · · ≥ yiq/2 ∇W (α)iq/2 − yjq/2 ∇W (α)jq/2 . (87)
This working set selection is merely an extension of that for C-SVC. The main
difference is that for C-SVM, (83) becomes only one inequality with b. Due to this
similarity, we believe that the convergence analysis of C-SVC [21] can be adapted here
though detailed proofs have not been written and published.
[11] considers the same working set selection. However, following the derivation
for C-SVC in [19], it is obtained using the concept of feasible directions in constrained
optimization. We feel that a derivation from the violation of the KKT condition is more
intuitive.
7.3 SMO-type Implementation

The Sequential Minimal Optimization (SMO) algorithm [28] is an extreme of the de-
composition method where, for C-SVC, the working set is restricted to only two el-
ements. The main advantage is that each two-variable sub-problem can be analyti-
cally solved, so numerical optimization software are not needed. For this method, at
least two elements are required for the working set. Otherwise, the equality constraint
P P
i∈B αi yi = − j∈N αjk yj leads to a fixed optimal objective value of the sub-problem.
Then, the decomposition procedure stays at the same point.
Now the dual of ν-SVC possesses two inequalities, so we may think that more
elements are needed for the working set. Indeed, two elements are still enough for the
case of ν-SVC. Note that (79) and (80) can be rewritten as
X ν X X ν X
αi yi = − αik yi and αi yi = − αik yi . (88)
2 2
i∈B,yi =1 i∈N,yi =1 i∈B,yi =−1 i∈N,yi =−1
Thus, if (i1 , j1 ) are selected as the working set selection using (87), yi1 = yj1 , so
(88) reduces to only one equality with two variables. Then, the sub-problem is still
guaranteed to be smaller than that at αk .
The comparison in [11] shows that using C and ν with the connection in proposition
3 and equivalent stopping condition, the performance of the SMO-type implementation
described here for C-SVM and ν-SVM are comparable.
7.4 The Calculation of b and ρ and Stopping Criteria

1 1
If at an optimal solution, 0 < αi < 1/m and yi = 1, then i ∈ Iup (α) and Ilow (α).
Thus, ρ − b = ∇W (α)i . Similarly, if there is another 0 < αj < 1/m and yj = −1,
then ρ + b = ∇W (α)j . Thus, solving two equalities gives b and ρ. In practice, we
average W (α)i to avoid numerical errors:
P
1
0<αi < m ,yi =1 ∇W (α)i
ρ−b= P , (89)
0<αi < 1 ,yi =1 1
m
If there are no components such that 0 < αi < 1/m, ρ − b (and ρ + b) can be any
number in the interval formed by (83). A common way is to select the middle point and
then still solves two linear equations
The stopping condition of the decomposition method can easily follow the new form
of the optimality condition (83):
¡
max − min ∇W (α)i + max
1 (α)
∇W (α)i ,
1
i∈Ilow (α) i∈Iup
¢ (90)
− min ∇W (α)i + max ∇W (α)i < ²,
i∈Ilow (α)
−1
i∈Iup (α)
−1
where ² > 0 is a chosen stopping tolerance.
8 Multi-Class ν-SV Classifiers
Though SVM was originally designed for two-class problems, several approaches have
been developed to extend SVM for multi-class data sets. In this section, we discuss the
extension of the “one-against-one” approach for multi-class ν-SVM.
Most approaches for multi-class SVM decompose the data set to several binary
problems. For example, the “one-against-one” approach trains a binary SVM for any
two classes of data and obtains a decision function. Thus, for a k-class problem, there
are k(k−1)/2 decision functions. In the prediction stage, a voting strategy is used where
the testing point is designated to be in a class with the maximum number of votes. In
[18], it was experimentally shown that for general problems, using C-SV classifier,
various multi-class approaches give similar accuracy. However, the “one-against-one”
method is more efficient for training. Here, we will focus on extending it for ν-SVM.
Multi-class methods must be considered together with parameter-selection strate-
gies. That is, we search for appropriate C and kernel parameters for constructing a
better model. In the following, we restrict the discussion on only the Gaussian (radius
2
basis function) kernel k(xi , xj ) = e−γkxi −xj k , so the kernel parameter is γ. With
the parameter selection considered, there are two ways to implement the “one-against-
one” method: First, for any two classes of data, the parameter selection is conducted
to have the best (C, γ). Thus, for the best model selected, each decision function has
its own (C, γ). For experiments here, the parameter selection of each binary SVM is
by a five-fold cross-validation. The second way is that for each (C, γ), an evaluation
criterion (e.g. cross-validation) combining with the “one-against-one” method is used
for estimating the performance of the model. A sequence of pre-selected (C, γ) is tried
to select the best model. Therefore, for each model, k(k − 1)/2 decision functions share
the same C and γ.
It is not very clear which one of the two implementations is better. On one hand, a
single parameter set may not be uniformly good for all k(k − 1)/2 decision functions.
On the other hand, as the overall accuracy is the final consideration, one parameter set
for one decision function may lead to over-fitting. [14] is the first to compare the two
approaches using C-SVM, where the preliminary results show that both give similar
accuracy.
For ν-SVM, each binary SVM using data from the ith and the jth classes has an
ij ij ij
admissible interval [νmin , νmax ], where νmax = 2 min(mi , mj )/(mi + mj ) according
to proposition 3. Here mi and mj are the number of data points in the ith and jth
classes, respectively. Thus, if all k(k − 1)/2 decision functions share the same ν, the
admissible interval is
ij ij
[max νmin , min νmax ]. (91)
i6=j i6=j
This set is non-empty if the kernel matrix is positive definite. The reason is that proposi-
ij ij
tion 3 implies νmin = 0, ∀i 6= j, so mini6=j νmax = 0. Therefore, unlike C of C-SVM,
which has a large valid range [0, ∞), for ν-SVM, we worry that the admissible interval
ij
may be too small. For example, if the data set is highly unbalanced, mini6=j νmin is very
small.
We redo the same comparison as that in [14] for ν-SVM. Results are in Table 2. We
consider multi-class problems tested in [18], where most of them are from the statlog
collection [25]. Except data sets dna, shuttle, letter, satimage, and usps, where test sets
are available, we separate each problem to 80% training and 20% testing. Then, cross
validation are conducted only on the training data. All other settings such as data scaling
are the same as those in [18]. Experiments are conducted using LIBSVM [10], which
solves both C-SVM and ν-SVM.
Results in Table 2 show no significant difference among the four implementations.
Note that some problems (e.g. shuttle) are highly unbalanced so the admissible interval
(91) is very small. Surprisingly, from such intervals, we can still find a suitable ν which
leads to a good model. This preliminary experiment indicates that in general the use of
“one-against-one” approach for multi-class ν-SVM is viable.
Table 2. Test accuracy (in percentage) of multi-class data sets by C-SVM and ν-SVM. The
columns “Common C”, “Different C”, “Common ν”, “Different ν” are testing accuracy of using
the same and different (C,γ), (or (ν,γ)) for all k(k − 1)/2 decision functions. The validation is
conducted on the following points of (C,γ): [2−5 , 2−3 , . . . , 215 ] × [2−15 , 2−13 , . . . , 23 ]. For ν-
SVM, the range of γ is the same but we validate a 10-point discretization of ν in the interval (91)
ij ij
or [νmin , νmax ], depending on whether k(k − 1)/2 decision functions share the same parameters
or not. For small problems (number of training data ≤ 1000), we do cross validation five times,
and then average the testing accuracy.
Data set Class No. # training # testing Common C Different C Common ν Different ν
vehicle 4 677 169 86.5 87.1 85.9 87.8
glass 6 171 43 72.2 70.7 73.0 69.3
iris 3 120 30 96.0 93.3 94.0 94.6
dna 3 2000 1186 95.6 95.1 95.0 94.8
segment 7 1848 462 98.3 97.2 96.7 97.6
shuttle 7 43500 14500 99.9 99.9 99.7 99.8
letter 26 15000 5000 97.9 97.7 97.9 96.8
vowel 11 423 105 98.1 97.7 98.3 96.0
satimage 6 4435 2000 91.9 92.2 92.1 91.9
wine 3 143 35 97.1 97.1 97.1 96.6
usps 10 7291 2007 95.3 95.2 95.3 94.8
We also present the contours of C-SVM and ν-SVM in Figure 8 using the approach
that all decision functions share the same (C, γ). In the contour of C-SVM, the x-
axis and y-axis are log2 C and log2 γ, respectively. For ν-SVM, the x-axis is ν in the
interval (91). Clearly, the good region of using ν-SVM is smaller. This confirms our
concern earlier, which motivated us to conduct experiments in this section. Fortunately,
points in this smaller good region still lead to models that are competitive with those by
C-SVM.
There are some ways to enlarge the admissible interval of ν. A work to extend
algorithm to the case of very small values of ν by allowing negative margins is [27].
For the upper bound, according to the above proposition 3, if the classes are balanced,
then the upper bound is 1. This leads to the idea to modify the algorithm by adjusting
the cost function such that the classes are balanced in terms of the cost, even if they are
not in terms of the mere numbers of training examples. An earlier discussion on such
formulations is at [12]. For example, we can consider the following formulation:
1 1 X 1 X
minimize τ (w, ξ, ρ) = kwk2 − νρ + ξi + ξi
w∈H,ξ∈Rm ,ρ,b∈R 2 2m+ i:y =1 2m− i:yi =−1
i
subject to yi (hxi , wi + b) ≥ ρ − ξi ,
and ξi ≥ 0, ρ ≥ 0.
satimage 91.5 satimage 91.5
91 91
90.5 90.5
90 90
89.5 89.5
4 89 4 89
88.5 88.5
2 2
0 0
-2 -2
-4 -4
-6 log(gamma) -6 log(gamma)
-8 -8
-10 -10
-12 -12
-14 -14
-16 -16
-5 0 5 10 15 0 0.1 0.2 0.3 0.4 0.5 0.6
log(C) NU
Fig. 6. 5-fold cross-validation accuracy of the data set satimage. Left: C-SVM, Right: ν-SVM
The dual is
m
1 X
maximize W (α) = − αi αj yi yj k(xi , xj )
α∈Rm 2 i,j=1
1
subject to 0 ≤ αi ≤ , if yi = 1,
2m+
1
0 ≤ αi ≤ , if yi = −1,
2m−
Xm m
X
αi yi = 0, αi ≥ ν.
i=1 i=1
Clearly,
Pm when all αi equals its corresponding upper bound, α is a feasible solution with
i=1 αi = 1 .
Another possibility is
X m
1 1
minimize τ (w, ξ, ρ) = kwk2 − νρ + ξi
w∈H,ξ∈Rm ,ρ,b∈R 2 2 min(m+ , m− ) i=1
subject to yi (hxi , wi + b) ≥ ρ − ξi ,
and ξi ≥ 0, ρ ≥ 0.
The dual is
m
1 X
maximize W (α) = − αi αj yi yj k(xi , xj )
α∈Rm 2 i,j=1
1
subject to 0 ≤ αi ≤ ,
2 min(m+ , m− )
m
X m
X
αi yi = 0, αi ≥ ν.
i=1 i=1
Then, the largest admissible ν is 1.

A slight modification of the implementation in Section 7 for the above formulations
is in [13].
9 Applications of ν-SV Classifiers

Researchers have applied ν-SVM on different applications. Some of them feel that it
is easier and more intuitive to deal with ν ∈ [0, 1] than C ∈ [0, ∞). Here, we briefly
summarize some work which use LIBSVM to solve ν-SVM.
In [7], researchers from HP Labs discuss the topics of personal email agent. Data
classification is an important component for which the authors use ν-SVM because they
think “the ν parameter is more intuitive than the C parameter.”
[23] applies machine learning methods to detect and localize boundaries of natural
images. Several classifiers are tested where, for SVM, the authors considered ν-SVM.
10 Conclusion
One of the most appealing features of kernel algorithms is the solid foundation pro-
vided by both statistical learning theory and functional analysis. Kernel methods let
us interpret (and design) learning algorithms geometrically in feature spaces nonlin-
early related to the input space, and combine statistics and geometry in a promising
way. Kernels provide an elegant framework for studying three fundamental issues of
machine learning:
– Similarity measures — the kernel can be viewed as a (nonlinear) similarity measure,
and should ideally incorporate prior knowledge about the problem at hand
– Data representation — as described above, kernels induce representations of the
data in a linear space
– Function class — due to the representer theorem, the kernel implicitly also deter-
mines the function class which is used for learning.
Support vector machines have been one of the major kernel methods for data classi-
fication. Its original form requires a parameter C ∈ [0, ∞), which controls the trade-off
between the classifier capacity and the training errors. Using the ν-parameterization,
the parameter C is replaced by a parameter ν ∈ [0, 1]. In this tutorial, we have given its
derivation and present possible advantages of using the ν-support vector classifier.
Acknowledgments
The authors thank Ingo Steinwart and Arthur Gretton for some helpful comments.
References
1. M. A. Aizerman, É.. M. Braverman, and L. I. Rozonoér. Theoretical foundations of the

potential function method in pattern recognition learning. Automation and Remote Control,
25:821–837, 1964.
2. N. Alon, S. Ben-David, N. Cesa-Bianchi, and D. Haussler. Scale-sensitive dimensions, uni-
form convergence, and learnability. Journal of the ACM, 44(4):615–631, 1997.
3. M. Avriel. Nonlinear Programming. Prentice-Hall Inc., New Jersey, 1976.
4. P. L. Bartlett and J. Shawe-Taylor. Generalization performance of support vector machines
and other pattern classifiers. In B. Schölkopf, C. J. C. Burges, and A. J. Smola, editors,
Advances in Kernel Methods — Support Vector Learning, pages 43–54, Cambridge, MA,
1999. MIT Press.
5. M. S. Bazaraa, H. D. Sherali, and C. M. Shetty. Nonlinear programming : theory and algo-
rithms. Wiley, second edition, 1993.
6. K. P. Bennett and E. J. Bredensteiner. Duality and geometry in SVM classifiers. In P. Langley,
editor, Proceedings of the 17th International Conference on Machine Learning, pages 57–64,
San Francisco, California, 2000. Morgan Kaufmann.
7. R. Bergman, M. Griss, and C. Staelin. A personal email assistant. Technical Report HPL-
2002-236, HP Laboratories, Palo Alto, CA, 2002.
8. B. E. Boser, I. M. Guyon, and V. Vapnik. A training algorithm for optimal margin classifiers.
In D. Haussler, editor, Proceedings of the 5th Annual ACM Workshop on Computational
Learning Theory, pages 144–152, Pittsburgh, PA, July 1992. ACM Press.
9. C. J. C. Burges and B. Schölkopf. Improving the accuracy and speed of support vector
learning machines. In M. Mozer, M. Jordan, and T. Petsche, editors, Advances in Neural
Information Processing Systems 9, pages 375–381, Cambridge, MA, 1997. MIT Press.
10. C.-C. Chang and C.-J. Lin. LIBSVM: a library for support vector machines, 2001. Software
available at http://www.csie.ntu.edu.tw/˜cjlin/libsvm.
11. C.-C. Chang and C.-J. Lin. Training ν-support vector classifiers: Theory and algorithms.
Neural Computation, 13(9):2119–2147, 2001.
12. H.-G. Chew, R. E. Bogner, and C.-C. Lim. Dual ν-support vector machine with error rate
and training size biasing. In Proceedings of ICASSP, pages 1269–72, 2001.
13. H. G. Chew, C. C. Lim, and R. E. Bogner. An implementation of training dual-nu support
vector machines. In Qi, Teo, and Yang, editors, Optimization and Control with Applications.
Kluwer, 2003.
14. K.-M. Chung, W.-C. Kao, C.-L. Sun, and C.-J. Lin. Decomposition methods for linear sup-
port vector machines. Technical report, Department of Computer Science and Information
Engineering, National Taiwan University, 2002.
15. C. Cortes and V. Vapnik. Support vector networks. Machine Learning, 20:273–297, 1995.
16. D. J. Crisp and C. J. C. Burges. A geometric interpretation of ν-SVM classifiers. In S. A.
Solla, T. K. Leen, and K.-R. Müller, editors, Advances in Neural Information Processing
Systems 12. MIT Press, 2000.
17. A. Gretton, R. Herbrich, O. Chapelle, B. Schölkopf, and P. J. W. Rayner. Estimating the
Leave-One-Out Error for Classification Learning with SVMs. Technical Report CUED/F-
INFENG/TR.424, Cambridge University Engineering Department, 2001.
18. C.-W. Hsu and C.-J. Lin. A comparison of methods for multi-class support vector machines.
IEEE Transactions on Neural Networks, 13(2):415–425, 2002.
19. T. Joachims. Making large–scale SVM learning practical. In B. Schölkopf, C. J. C. Burges,
and A. J. Smola, editors, Advances in Kernel Methods — Support Vector Learning, pages
169–184, Cambridge, MA, 1999. MIT Press.
20. C.-J. Lin. Formulations of support vector machines: a note from an optimization point of
view. Neural Computation, 13(2):307–317, 2001.
21. C.-J. Lin. On the convergence of the decomposition method for support vector machines.
IEEE Transactions on Neural Networks, 12(6):1288–1298, 2001.
22. Luntz, A. and Brailovsky, V. On estimation of characters obtained in statistical procedure of
recognition. Technicheskaya Kibernetica 3, 1969.
23. D. R. Martin, C. C. Fowlkes, and J. Malik. Learning to detect natural image boundaries using
brightness and texture. In Advances in Neural Information Processing Systems, volume 14,
2002.
24. J. Mercer. Functions of positive and negative type and their connection with the theory of
integral equations. Philosophical Transactions of the Royal Society, London, A 209:415–
446, 1909.
25. D. Michie, D. J. Spiegelhalter, and C. C. Taylor. Machine Learning, Neural and Sta-
tistical Classification. Prentice Hall, Englewood Cliffs, N.J., 1994. Data available at
http://www.ncc.up.pt/liacc/ML/statlog/datasets.html.
26. M. Opper and O. Winther. Gaussian processes and SVM: Mean field and leave-one-out
estimator. In A. Smola, P. Bartlett, B. Schölkopf, and D. Schuurmans, editors, Advances in
Large Margin Classifiers, Cambridge, MA, 2000. MIT Press.
27. F. Perez Cruz, J. Weston, D. J. L. Herrmann, and B. Schölkopf. Extension of the ν-svm
range for classification. In J. Suykens, G. Horvath, S. Basu, C. Micchelli, and J. Vandewalle,
editors, Advances in Learning Theory: Methods, Models and Applications, 190, pages 179–
196, Amsterdam, 2003. IOS Press.
28. J. C. Platt. Fast training of support vector machines using sequential minimal optimization.
In B. Schölkopf, C. J. C. Burges, and A. J. Smola, editors, Advances in Kernel Methods -
Support Vector Learning, Cambridge, MA, 1998. MIT Press.
29. B. Schölkopf. Support Vector Learning. R. Oldenbourg Verlag, München, 1997. Doktorar-
beit, Technische Universität Berlin. Available from http://www.kyb.tuebingen.mpg.de/∼bs.
30. B. Schölkopf, C. J. C. Burges, and A. J. Smola. Advances in Kernel Methods — Support
Vector Learning. MIT Press, Cambridge, MA, 1999.
31. B. Schölkopf and A. J. Smola. Learning with Kernels. MIT Press, Cambridge, MA, 2002.
32. B. Schölkopf, A. J. Smola, R. C. Williamson, and P. L. Bartlett. New support vector algo-
rithms. Neural Computation, 12:1207–1245, 2000.
33. A. J. Smola, P. L. Bartlett, B. Schölkopf, and D. Schuurmans. Advances in Large Margin
Classifiers. MIT Press, Cambridge, MA, 2000.
34. I. Steinwart. Support vector machines are universally consistent. Journal of Complexity,
18:768–791, 2002.
35. I. Steinwart. On the optimal parameter choice for ν-support vector machines. IEEE Trans-
actions on Pattern Analysis and Machine Intelligence, 2003. To appear.
36. I. Steinwart. Sparseness of support vector machines. Technical report, 2003.
37. V. Vapnik. Estimation of Dependences Based on Empirical Data (in Russian). Nauka,
Moscow, 1979. (English translation: Springer Verlag, New York, 1982).
38. V. Vapnik. The Nature of Statistical Learning Theory. Springer Verlag, New York, 1995.
39. V. Vapnik. Statistical Learning Theory. Wiley, NY, 1998.
40. V. Vapnik and O. Chapelle. Bounds on error expectation for support vector machines. Neural
Computation, 12(9):2013–2036, 2000.
41. V. Vapnik and A. Chervonenkis. Theory of Pattern Recognition (in Russian). Nauka,
Moscow, 1974. (German Translation: W. Wapnik & A. Tscherwonenkis, Theorie der Ze-
ichenerkennung, Akademie–Verlag, Berlin, 1979).
42. V. Vapnik and A. Lerner. Pattern recognition using generalized portrait method. Automation
and Remote Control, 24:774–780, 1963.
43. G. Wahba. Spline Models for Observational Data, volume 59 of CBMS-NSF Regional Con-
ference Series in Applied Mathematics. Society for Industrial and Applied Mathematics,
Philadelphia, Pennsylvania, 1990.
44. R. C. Williamson, A. J. Smola, and B. Schölkopf. Generalization performance of regulariza-
tion networks and support vector machines via entropy numbers of compact operators. IEEE
Transactions on Information Theory, 47(6):2516–2532, 2001.
45. P. Wolfe. A duality theorem for non-linear programming. Quartely of Applied Mathematics,
19:239–244, 1961.
Statistics and Computing 14: 199–222, 2004
C 2004 Kluwer Academic Publishers. Manufactured in The Netherlands.
A tutorial on support vector regression∗

ALEX J. SMOLA and BERNHARD SCHÖLKOPF
RSISE, Australian National University, Canberra 0200, Australia
Alex.Smola@anu.edu.au
Max-Planck-Institut für biologische Kybernetik, 72076 Tübingen, Germany
Bernhard.Schoelkopf@tuebingen.mpg.de
Received July 2002 and accepted November 2003
In this tutorial we give an overview of the basic ideas underlying Support Vector (SV) machines for
function estimation. Furthermore, we include a summary of currently used algorithms for training
SV machines, covering both the quadratic (or convex) programming part and advanced methods for
dealing with large datasets. Finally, we mention some modifications and extensions that have been
applied to the standard SV algorithm, and discuss the aspect of regularization from a SV perspective.
Keywords: machine learning, support vector machines, regression estimation
1. Introduction (Vapnik and Lerner 1963, Vapnik and Chervonenkis 1964). As

such, it is firmly grounded in the framework of statistical learn-
The purpose of this paper is twofold. It should serve as a self- ing theory, or VC theory, which has been developed over the last
contained introduction to Support Vector regression for readers three decades by Vapnik and Chervonenkis (1974) and Vapnik
new to this rapidly developing field of research.1 On the other (1982, 1995). In a nutshell, VC theory characterizes properties
hand, it attempts to give an overview of recent developments in of learning machines which enable them to generalize well to
the field. unseen data.
To this end, we decided to organize the essay as follows. In its present form, the SV machine was largely developed
We start by giving a brief overview of the basic techniques in at AT&T Bell Laboratories by Vapnik and co-workers (Boser,
Sections 1, 2 and 3, plus a short summary with a number of Guyon and Vapnik 1992, Guyon, Boser and Vapnik 1993, Cortes
figures and diagrams in Section 4. Section 5 reviews current and Vapnik, 1995, Schölkopf, Burges and Vapnik 1995, 1996,
algorithmic techniques used for actually implementing SV Vapnik, Golowich and Smola 1997). Due to this industrial con-
machines. This may be of most interest for practitioners. text, SV research has up to date had a sound orientation towards
The following section covers more advanced topics such as real-world applications. Initial work focused on OCR (optical
extensions of the basic SV algorithm, connections between SV character recognition). Within a short period of time, SV clas-
machines and regularization and briefly mentions methods for sifiers became competitive with the best available systems for
carrying out model selection. We conclude with a discussion both OCR and object recognition tasks (Schölkopf, Burges and
of open questions and problems and current directions of SV Vapnik 1996, 1998a, Blanz et al. 1996, Schölkopf 1997). A
research. Most of the results presented in this review paper comprehensive tutorial on SV classifiers has been published by
already have been published elsewhere, but the comprehensive Burges (1998). But also in regression and time series predic-
presentations and some details are new. tion applications, excellent performances were soon obtained
(Müller et al. 1997, Drucker et al. 1997, Stitson et al. 1999,
Mattera and Haykin 1999). A snapshot of the state of the art
1.1. Historic background in SV learning was recently taken at the annual Neural In-
formation Processing Systems conference (Schölkopf, Burges,
The SV algorithm is a nonlinear generalization of the Gener- and Smola 1999a). SV learning has now evolved into an active
alized Portrait algorithm developed in Russia in the sixties2 area of research. Moreover, it is in the process of entering the
∗An extended version of this paper is available as NeuroCOLT Technical Report standard methods toolbox of machine learning (Haykin 1998,
TR-98-030. Cherkassky and Mulier 1998, Hearst et al. 1998). Schölkopf and
0960-3174
C 2004 Kluwer Academic Publishers
Smola (2002) contains a more in-depth overview of SVM regres-
sion. Additionally, Cristianini and Shawe-Taylor (2000) and Her-
brich (2002) provide further details on kernels in the context of
classification.
1.2. The basic idea

Suppose we are given training data {(x1 , y1 ), . . . , (x , y )} ⊂
X × R, where X denotes the space of the input patterns (e.g. Fig. 1. The soft margin loss setting for a linear SVM (from Schölkopf
X = Rd ). These might be, for instance, exchange rates for some and Smola, 2002)
currency measured at subsequent days together with correspond-
ing econometric indicators. In ε-SV regression (Vapnik 1995), Figure 1 depicts the situation graphically. Only the points outside
our goal is to find a function f (x) that has at most ε deviation the shaded region contribute to the cost insofar, as the deviations
from the actually obtained targets yi for all the training data, and are penalized in a linear fashion. It turns out that in most cases
at the same time is as flat as possible. In other words, we do not the optimization problem (3) can be solved more easily in its dual
care about errors as long as they are less than ε, but will not formulation.4 Moreover, as we will see in Section 2, the dual for-
accept any deviation larger than this. This may be important if mulation provides the key for extending SV machine to nonlinear
you want to be sure not to lose more than ε money when dealing functions. Hence we will use a standard dualization method uti-
with exchange rates, for instance. lizing Lagrange multipliers, as described in e.g. Fletcher (1989).
For pedagogical reasons, we begin by describing the case of
linear functions f , taking the form 1.3. Dual problem and quadratic programs
f (x) = w, x + b with w ∈ X , b ∈ R (1) The key idea is to construct a Lagrange function from the ob-
jective function (it will be called the primal objective function
where · , · denotes the dot product in X . Flatness in the case
in the rest of this article) and the corresponding constraints, by
of (1) means that one seeks a small w. One way to ensure this is
introducing a dual set of variables. It can be shown that this
to minimize the norm,3 i.e. w2 = w, w. We can write this
function has a saddle point with respect to the primal and dual
problem as a convex optimization problem:
variables at the solution. For details see e.g. Mangasarian (1969),
minimize 1
w2 McCormick (1983), and Vanderbei (1997) and the explanations

2
yi − w, xi − b ≤ ε (2) in Section 5.2. We proceed as follows:
subject to
w, xi + b − yi ≤ ε 1
L := w2 + C (ξi + ξi∗ ) − (ηi ξi + ηi∗ ξi∗ )
2
The tacit assumption in (2) was that such a function f actually i=1 i=1
exists that approximates all pairs (xi , yi ) with ε precision, or in

other words, that the convex optimization problem is feasible. − αi (ε + ξi − yi + w, xi + b)
Sometimes, however, this may not be the case, or we also may i=1
want to allow for some errors. Analogously to the “soft mar-

gin” loss function (Bennett and Mangasarian 1992) which was − αi∗ (ε + ξi∗ + yi − w, xi − b) (5)
used in SV machines by Cortes and Vapnik (1995), one can in- i=1
troduce slack variables ξi , ξi∗ to cope with otherwise infeasible Here L is the Lagrangian and ηi , ηi∗ , αi , αi∗ are Lagrange multi-
constraints of the optimization problem (2). Hence we arrive at pliers. Hence the dual variables in (5) have to satisfy positivity
the formulation stated in Vapnik (1995). constraints, i.e.
1 (∗) (∗)
αi , ηi ≥ 0. (6)
minimize w2 + C (ξi + ξi∗ )
Note that by αi , we refer to αi and αi∗ .
2 (∗)
 i=1
 yi − w, xi − b ≤ ε + ξi
 (3) It follows from the saddle point condition that the partial
subject to w, xi + b − yi ≤ ε + ξi ∗ derivatives of L with respect to the primal variables (w, b, ξi , ξi∗ )

 have to vanish for optimality.
ξi , ξi∗ ≥0

The constant C > 0 determines the trade-off between the flat- ∂b L = (αi∗ − αi ) = 0 (7)
ness of f and the amount up to which deviations larger than i=1
ε are tolerated. This corresponds to dealing with a so called

ε-insensitive loss function |ξ |ε described by ∂w L = w − (αi − αi∗ )xi = 0 (8)

0 if |ξ | ≤ ε i=1
|ξ |ε := (4) (∗) (∗)
|ξ | − ε otherwise. ∂ξ (∗) L = C − αi − ηi = 0 (9)
i
Substituting (7), (8), and (9) into (5) yields the dual optimization In conjunction with an analogous analysis on αi∗ we have
problem. max{−ε + yi − w, xi | αi < C or αi∗ > 0} ≤ b ≤
 (16)

 1 min{−ε + yi − w, xi | αi > 0 or αi∗ < C}

 − (αi − αi∗ )(α j − α ∗j )xi , x j
 2 (∗)
maximize i, j=1 If some αi ∈ (0, C) the inequalities become equalities. See

 also Keerthi et al. (2001) for further means of choosing b.

 (αi + αi∗ ) + yi (αi − αi∗ )
 −ε (10) Another way of computing b will be discussed in the context
i=1 i=1

of interior point optimization (cf. Section 5). There b turns out
subject to (αi − αi∗ ) = 0 and αi , αi∗ ∈ [0, C] to be a by-product of the optimization process. Further consid-
i=1 erations shall be deferred to the corresponding section. See also
Keerthi et al. (1999) for further methods to compute the constant
In deriving (10) we already eliminated the dual variables ηi , ηi∗ offset.
(∗)
through condition (9) which can be reformulated as ηi = C − A final note has to be made regarding the sparsity of the SV
(∗)
αi . Equation (8) can be rewritten as follows expansion. From (12) it follows that only for | f (xi ) − yi | ≥ ε
the Lagrange multipliers may be nonzero, or in other words, for

w= (αi −αi∗ )xi , thus f (x) = (αi −αi∗ )xi , x + b. (11) all samples inside the ε–tube (i.e. the shaded region in Fig. 1)
i=1 i=1 the αi , αi∗ vanish: for | f (xi ) − yi | < ε the second factor in
(12) is nonzero, hence αi , αi∗ has to be zero such that the KKT
This is the so-called Support Vector expansion, i.e. w can be conditions are satisfied. Therefore we have a sparse expansion
completely described as a linear combination of the training of w in terms of xi (i.e. we do not need all xi to describe w). The
patterns xi . In a sense, the complexity of a function’s represen- examples that come with nonvanishing coefficients are called
tation by SVs is independent of the dimensionality of the input Support Vectors.
space X , and depends only on the number of SVs.
Moreover, note that the complete algorithm can be described
in terms of dot products between the data. Even when evalu- 2. Kernels
ating f (x) we need not compute w explicitly. These observa-
tions will come in handy for the formulation of a nonlinear 2.1. Nonlinearity by preprocessing
extension. The next step is to make the SV algorithm nonlinear. This, for
instance, could be achieved by simply preprocessing the training
patterns xi by a map : X → F into some feature space F,
1.4. Computing b as described in Aizerman, Braverman and Rozonoér (1964) and
So far we neglected the issue of computing b. The latter can be Nilsson (1965) and then applying the standard SV regression
done by exploiting the so called Karush–Kuhn–Tucker (KKT) algorithm. Let us have a brief look at an example given in Vapnik
conditions (Karush 1939, Kuhn and Tucker 1951). These state (1995).
that at the point of the solution the product between dual variables
and constraints has to vanish. Example 1 (Quadratic features in√ R2 ). Consider the map :
R → R with (x1 , x2 ) = (x1 , 2x1 x2 , x22 ). It is understood
2 3 2
αi (ε + ξi − yi + w, xi + b) = 0 that the subscripts in this case refer to the components of x ∈ R2 .

(12) Training a linear SV machine on the preprocessed features would
αi∗ (ε + ξi∗ + yi − w, xi − b) = 0 yield a quadratic function.
and While this approach seems reasonable in the particular ex-

ample above, it can easily become computationally infeasible
(C − αi )ξi = 0 for both polynomial features of higher order and higher di-
(13) mensionality, as the number of different monomial features
(C − αi∗ )ξi∗ = 0. of degree p is (d+ pp−1), where d = dim(X ). Typical values
for OCR tasks (with good performance) (Schölkopf, Burges
This allows us to make several useful conclusions. Firstly only
(∗) and Vapnik 1995, Schölkopf et al. 1997, Vapnik 1995) are
samples (xi , yi ) with corresponding αi = C lie outside the ε- p = 7, d = 28 · 28 = 784, corresponding to approximately
∗
insensitive tube. Secondly αi αi = 0, i.e. there can never be a set 3.7 · 1016 features.
of dual variables αi , αi∗ which are both simultaneously nonzero.
This allows us to conclude that
2.2. Implicit mapping via kernels
ε − yi + w, xi + b ≥ 0 and ξi = 0 if αi < C (14)
Clearly this approach is not feasible and we have to find a com-
ε − yi + w, xi + b ≤ 0 if αi > 0 (15) putationally cheaper way. The key observation (Boser, Guyon
and Vapnik 1992) is that for the feature map of example 2.1 we condition (Schölkopf, Burges and Smola 1999a). In the follow-
have ing we will call such functions k admissible SV kernels.
2 √ 2 √ 2
x1 , 2x1 x2 , x22 , x 1 , 2x 1 x 2 , x 2 = x, x 2 . (17) Corollary 3 (Positive linear combinations of kernels). Denote
As noted in the previous section, the SV algorithm only depends by k1 , k2 admissible SV kernels and c1 , c2 ≥ 0 then
on dot products between patterns xi . Hence it suffices to know k(x, x ) := c1 k1 (x, x ) + c2 k2 (x, x ) (22)
k(x, x ) := (x), (x ) rather than explicitly which allows
us to restate the SV optimization problem: is an admissible kernel. This follows directly from (21) by virtue
 of the linearity of integrals.


 1
−
 (αi − αi∗ )(α j − α ∗j )k(xi , x j )
2 i, j=1 More generally, one can show that the set of admissible ker-
maximize nels forms a convex cone, closed in the topology of pointwise



 −ε (α + α ∗
) + yi (αi − αi∗ ) (18) convergence (Berg, Christensen and Ressel 1984).
 i i
i=1 i=1

Corollary 4 (Integrals of kernels). Let s(x, x ) be a function
subject to (αi − αi∗ ) =0 and αi , αi∗ ∈ [0, C] on X × X such that
i=1

Likewise the expansion of f (11) may be written as k(x, x ) := s(x, z)s(x , z) dz (23)
X

w= (αi − αi∗ )(xi ) and exists. Then k is an admissible SV kernel.
i=1
This can be shown directly from (21) and (23) by rearranging the

f (x) = (αi − αi∗ )k(xi , x) + b. (19) order of integration. We now state a necessary and sufficient con-
i=1 dition for translation invariant kernels, i.e. k(x, x ) := k(x − x )
The difference to the linear case is that w is no longer given ex- as derived in Smola, Schölkopf and Müller (1998c).
plicitly. Also note that in the nonlinear setting, the optimization
problem corresponds to finding the flattest function in feature Theorem 5 (Products of kernels). Denote by k1 and k2 admis-
space, not in input space. sible SV kernels then
k(x, x ) := k1 (x, x )k2 (x, x ) (24)
2.3. Conditions for kernels is an admissible kernel.

The question that arises now is, which functions k(x, x ) corre-
This can be seen by an application of the “expansion part” of
spond to a dot product in some feature space F. The following
Mercer’s theorem to the kernels
k1 and k2 and observing that
theorem characterizes these functions (defined on X ).
each term in the double sum i, j λi1 λ2j ψi1 (x)ψi1 (x )ψ 2j (x)ψ 2j (x )
gives rise to a positive coefficient when checking (21).
Theorem 2 (Mercer 1909). Suppose k ∈ L ∞ (X 2 ) such that
the integral operator Tk : L 2 (X ) → L 2 (X ), Theorem 6 (Smola, Schölkopf and Müller 1998c). A transla-

tion invariant kernel k(x, x ) = k(x − x ) is an admissible SV
Tk f (·) := k(·, x) f (x)dµ(x) (20)
X kernels if and only if the Fourier transform

is positive (here µ denotes a measure on X with µ(X ) finite F[k](ω) = (2π )− 2
d
e−iω,x k(x)d x (25)
and supp(µ) = X ). Let ψ j ∈ L 2 (X ) be the eigenfunction of Tk X
associated with the eigenvalue λ j = 0 and normalized such that is nonnegative.
ψ j L 2 = 1 and let ψ j denote its complex conjugate. Then
1. (λ j (T )) j ∈ 1 . We will give a proof and some additional explanations to this
theorem in Section 7. It follows from interpolation theory
2. k(x, x ) = j∈N λ j ψ j (x)ψ j (x ) holds for almost all (x, x ),
where the series converges absolutely and uniformly for al- (Micchelli 1986) and the theory of regularization networks
most all (x, x ). (Girosi, Jones and Poggio 1993). For kernels of the dot-product
type, i.e. k(x, x ) = k(x, x ), there exist sufficient conditions
Less formally speaking this theorem means that if for being admissible.

k(x, x ) f (x) f (x ) d xd x ≥ 0 for all f ∈ L 2 (X ) (21) Theorem 7 (Burges 1999). Any kernel of dot-product type
X ×X k(x, x ) = k(x, x ) has to satisfy
holds we can write k(x, x ) as a dot product in some feature k(ξ ) ≥ 0, ∂ξ k(ξ ) ≥ 0 and ∂ξ k(ξ ) + ξ ∂ξ2 k(ξ ) ≥ 0 (26)
space. From this condition we can conclude some simple rules
for compositions of kernels, which then also satisfy Mercer’s for any ξ ≥ 0 in order to be an admissible SV kernel.
Note that the conditions in Theorem 7 are only necessary but B-splines of order 2n + 1, defined by the 2n + 1 convolution of
not sufficient. The rules stated above can be useful tools for the unit inverval, are also admissible. We shall postpone further
practitioners both for checking whether a kernel is an admissible considerations to Section 7 where the connection to regulariza-
SV kernel and for actually constructing new kernels. The general tion operators will be pointed out in more detail.
case is given by the following theorem.
Theorem 8 (Schoenberg 1942). A kernel of dot-product type

3. Cost functions
k(x, x ) = k(x, x ) defined on an infinite dimensional Hilbert
So far the SV algorithm for regression may seem rather strange
space, with a power series expansion
and hardly related to other existing methods of function esti-

∞
mation (e.g. Huber 1981, Stone 1985, Härdle 1990, Hastie and
k(t) = an t n (27)
n=0
Tibshirani 1990, Wahba 1990). However, once cast into a more
standard mathematical notation, we will observe the connec-
is admissible if and only if all an ≥ 0. tions to previous work. For the sake of simplicity we will, again,
only consider the linear case, as extensions to the nonlinear one
A slightly weaker condition applies for finite dimensional are straightforward by using the kernel method described in the
spaces. For further details see Berg, Christensen and Ressel previous chapter.
(1984) and Smola, Óvári and Williamson (2001).
3.1. The risk functional

2.4. Examples
Let us for a moment go back to the case of Section 1.2. There, we
In Schölkopf, Smola and Müller (1998b) it has been shown, by had some training data X := {(x1 , y1 ), . . . , (x , y )} ⊂ X × R.
explicitly computing the mapping, that homogeneous polyno- We will assume now, that this training set has been drawn iid
mial kernels k with p ∈ N and (independent and identically distributed) from some probabil-
ity distribution P(x, y). Our goal will be to find a function f
k(x, x ) = x, x p (28)
minimizing the expected risk (cf. Vapnik 1982)
are suitable SV kernels (cf. Poggio 1975). From this observation
one can conclude immediately (Boser, Guyon and Vapnik 1992, R[ f ] = c(x, y, f (x))d P(x, y) (33)
Vapnik 1995) that kernels of the type
(c(x, y, f (x)) denotes a cost function determining how we will
k(x, x ) = (x, x + c) p (29) penalize estimation errors) based on the empirical data X. Given
i.e. inhomogeneous polynomial kernels with p ∈ N, c ≥ 0 are that we do not know the distribution P(x, y) we can only use
admissible, too: rewrite k as a sum of homogeneous kernels and X for estimating a function f that minimizes R[ f ]. A possi-
apply Corollary 3. Another kernel, that might seem appealing ble approximation consists in replacing the integration by the
due to its resemblance to Neural Networks is the hyperbolic empirical estimate, to get the so called empirical risk functional
tangent kernel
1
Remp [ f ] := c(xi , yi , f (xi )). (34)
k(x, x ) = tanh(ϑ + κx, x ). (30) i=1
By applying Theorem 8 one can check that this kernel does A first attempt would be to find the empirical risk minimizer
not actually satisfy Mercer’s condition (Ovari 2000). Curiously, f 0 := argmin f ∈H Remp [ f ] for some function class H . However,
the kernel has been successfully used in practice; cf. Scholkopf if H is very rich, i.e. its “capacity” is very high, as for instance
(1997) for a discussion of the reasons. when dealing with few data in very high-dimensional spaces,
Translation invariant kernels k(x, x ) = k(x − x ) are this may not be a good idea, as it will lead to overfitting and thus
quite widespread. It was shown in Aizerman, Braverman and bad generalization properties. Hence one should add a capacity
Rozonoér (1964), Micchelli (1986) and Boser, Guyon and Vap- control term, in the SV case w2 , which leads to the regularized
nik (1992) that risk functional (Tikhonov and Arsenin 1977, Morozov 1984,
x−x 2 Vapnik 1982)
k(x, x ) = e− 2σ 2 (31)
λ
is an admissible SV kernel. Moreover one can show (Smola Rreg [ f ] := Remp [ f ] + w2 (35)
2
1996, Vapnik, Golowich and Smola 1997) that (1 X denotes the
indicator function on the set X and ⊗ the convolution operation) where λ > 0 is a so called regularization constant. Many
algorithms like regularization networks (Girosi, Jones and

k
k(x, x ) = B2n+1 (x − x ) with Bk := 1[− 12 , 12 ] (32) Poggio 1993) or neural networks with weight decay networks
i=1 (e.g. Bishop 1995) minimize an expression similar to (35).
3.2. Maximum likelihood and density models However, the cost function resulting from this reasoning might
be nonconvex. In this case one would have to find a convex
The standard setting in the SV case is, as already mentioned in
proxy in order to deal with the situation efficiently (i.e. to find
Section 1.2, the ε-insensitive loss
an efficient implementation of the corresponding optimization
c(x, y, f (x)) = |y − f (x)|ε . (36) problem).
If, on the other hand, we are given a specific cost function from
It is straightforward to show that minimizing (35) with the par-
a real world problem, one should try to find as close a proxy to
ticular loss function of (36) is equivalent to minimizing (3), the
this cost function as possible, as it is the performance wrt. this
only difference being that C = 1/(λ).
p particular cost function that matters ultimately.
Loss functions such like |y − f (x)|ε with p > 1 may not
Table 1 contains an overview over some common density
be desirable, as the superlinear increase leads to a loss of the
models and the corresponding loss functions as defined by
robustness properties of the estimator (Huber 1981): in those
(37).
cases the derivative of the cost function grows without bound.
The only requirement we will impose on c(x, y, f (x)) in the
For p < 1, on the other hand, c becomes nonconvex.
following is that for fixed x and y we have convexity in f (x).
For the case of c(x, y, f (x)) = (y − f (x))2 we recover the
This requirement is made, as we want to ensure the existence and
least mean squares fit approach, which, unlike the standard SV
uniqueness (for strict convexity) of a minimum of optimization
loss function, leads to a matrix inversion instead of a quadratic
problems (Fletcher 1989).
programming problem.
The question is which cost function should be used in (35). On
the one hand we will want to avoid a very complicated function c 3.3. Solving the equations
as this may lead to difficult optimization problems. On the other
For the sake of simplicity we will additionally assume c to
hand one should use that particular cost function that suits the
be symmetric and to have (at most) two (for symmetry) dis-
problem best. Moreover, under the assumption that the samples
continuities at ±ε, ε ≥ 0 in the first derivative, and to be
were generated by an underlying functional dependency plus
zero in the interval [−ε, ε]. All loss functions from Table 1
additive noise, i.e. yi = f true (xi ) + ξi with density p(ξ ), then the
belong to this class. Hence c will take on the following
optimal cost function in a maximum likelihood sense is
form.
c(x, y, f (x)) = − log p(y − f (x)). (37)
c(x, y, f (x)) = c̃(|y − f (x)|ε ) (41)
This can be seen as follows. The likelihood of an estimate
X f := {(x1 , f (x1 )), . . . , (x , f (x ))} (38) Note the similarity to Vapnik’s ε-insensitive loss. It is rather
straightforward to extend this special choice to more general
for additive noise and iid data is
convex cost functions. For nonzero cost functions in the inter-

val [−ε, ε] use an additional pair of slack variables. Moreover
p(X f | X) = p( f (xi ) | (xi , yi )) = p(yi − f (xi )). (39)
we might choose different cost functions c̃i , c̃i∗ and different
i=1 i=1
values of εi , εi∗ for each sample. At the expense of additional
Maximizing P(X f | X) is equivalent to minimizing − log P Lagrange multipliers in the dual formulation additional discon-
(X f | X). By using (37) we get tinuities also can be taken care of. Analogously to (3) we arrive at

a convex minimization problem (Smola and Schölkopf 1998a).
− log P(X f | X) = c(xi , yi , f (xi )). (40) To simplify notation we will stick to the one of (3) and use C
i=1
Table 1. Common loss functions and corresponding density models
Loss function Density model
ε-insensitive c(ξ ) = |ξ |ε p(ξ ) = 2(1+ε)

1
exp(−|ξ |ε )
Laplacian c(ξ ) = |ξ | p(ξ ) = 2 exp(−|ξ
1
|) 2
Gaussian c(ξ ) = 12 ξ 2 p(ξ ) = √12π exp − ξ2

1
(ξ )2 if |ξ | ≤ σ  exp − ξ 2 if |ξ | ≤ σ
Huber’s robust loss c(ξ ) = 2σ
p(ξ ) ∝ 2σ

|ξ | − σ
2
otherwise  exp σ − |ξ | otherwise
2
Polynomial c(ξ ) = 1
|ξ | p p(ξ ) = p
exp(−|ξ | p )
p
1 
2(1/ p)
(ξ ) p if |ξ | ≤ σ  exp − ξ p−1 p
if |ξ | ≤ σ
Piecewise polynomial c(ξ ) = pσ p−1
p(ξ ) ∝ pσ

|ξ | − σ p−1 p
otherwise  exp σ p−1 − |ξ | otherwise
p
instead of normalizing by λ and . Table 2. Terms of the convex optimization problem depending on the
choice of the loss function
1
minimize w2 + C (c̃(ξi ) + c̃(ξi∗ )) ε α C T (α)
2
 i=1
 yi − w, xi − b ≤ ε + ξi
 (42) ε-insensitive ε = 0 α ∈ [0, C] 0
subject to w, xi + b − yi ≤ ε + ξi∗ Laplacian ε =0 α ∈ [0, C] 0

 Gaussian ε =0 α ∈ [0, ∞) − 12 C −1 α 2
ξi , ξi∗ ≥0
Huber’s ε =0 α ∈ [0, C] − 12 σ C −1 α 2
Again, by standard Lagrange multiplier techniques, exactly in robust loss
1 p
the same manner as in the |·|ε case, one can compute the dual op- Polynomial ε=0 α ∈ [0, ∞) − p−1
p
C − p−1 α p−1
1 p
timization problem (the main difference is that the slack variable Piecewise ε=0 α ∈ [0, C] − p−1
p
σ C − p−1 α p−1
(∗)
terms c̃(ξi ) now have nonvanishing derivatives). We will omit polynomial
the indices i and ∗ , where applicable to avoid tedious notation.
This yields
 In the second case (ξ ≥ σ ) we have


−1
 (αi − αi∗ )(α j − α ∗j )xi , x j p−1 p−1

 T (ξ ) = ξ − σ − ξ = −σ (48)


2 i, j=1 p p


maximize + yi (αi − αi∗ ) − ε(αi + αi∗ ) and ξ = inf{ξ | C ≥ α} = σ , which, in turn yields α ∈ [0, C].

 Combining both cases we have

 i=1

 p−1

 +C
p p

 T (ξi ) + T (ξi∗ ) α ∈ [0, C] and T (α) = − σ C − p−1 α p−1 . (49)
i=1
p
 Table 2 contains a summary of the various conditions on α and



w= (αi − αi∗ )xi formulas for T (α) (strictly speaking T (ξ (α))) for different cost
where (43) functions.5 Note that the maximum slope of c̃ determines the

 i=1
T (ξ ) := c̃(ξ ) − ξ ∂ξ c̃(ξ ) region of feasibility of α, i.e. s := supξ ∈R+ ∂ξ c̃(ξ ) < ∞ leads to
 compact intervals [0, Cs] for α. This means that the influence



 (αi − αi∗ ) = 0 of a single pattern is bounded, leading to robust estimators


 i=1 (Huber 1972). One can also observe experimentally that the
subject to α ≤ C∂ξ c̃(ξ ) performance of a SV machine depends significantly on the cost



 ξ = inf{ξ | C∂ξ c̃ ≥ α} function used (Müller et al. 1997, Smola, Schölkopf and Müller


 1998b)
α, ξ ≥ 0
A cautionary remark is necessary regarding the use of cost
functions other than the ε-insensitive one. Unless ε = 0 we
3.4. Examples will lose the advantage of a sparse decomposition. This may
be acceptable in the case of few data, but will render the pre-
Let us consider the examples of Table 1. We will show explicitly diction step extremely slow otherwise. Hence one will have to
for two examples how (43) can be further simplified to bring it trade off a potential loss in prediction accuracy with faster pre-
into a form that is practically useful. In the ε-insensitive case, dictions. Note, however, that also a reduced set algorithm like
i.e. c̃(ξ ) = |ξ | we get in Burges (1996), Burges and Schölkopf (1997) and Schölkopf
et al. (1999b) or sparse decomposition techniques (Smola and
T (ξ ) = ξ − ξ · 1 = 0. (44)
Schölkopf 2000) could be applied to address this issue. In a
Morover one can conclude from ∂ξ c̃(ξ ) = 1 that Bayesian setting, Tipping (2000) has recently shown how an L 2
cost function can be used without sacrificing sparsity.
ξ = inf{ξ | C ≥ α} = 0 and α ∈ [0, C]. (45)
For the case of piecewise polynomial loss we have to distinguish 4. The bigger picture
two different cases: ξ ≤ σ and ξ > σ . In the first case we get
Before delving into algorithmic details of the implementation
1 1 p − 1 1− p p
T (ξ ) = ξ p − p−1 ξ p = − σ ξ (46) let us briefly review the basic properties of the SV algorithm
pσ p−1 σ p for regression as described so far. Figure 2 contains a graphical
overview over the different steps in the regression stage.
and ξ = inf{ξ | Cσ 1− p ξ p−1 ≥ α} = σ C − p−1 α p−1 and thus
1 1
The input pattern (for which a prediction is to be made) is

p−1 p p mapped into feature space by a map . Then dot products
T (ξ ) = − σ C − p−1 α p−1 . (47)
p are computed with the images of the training patterns under
ization operators. This will be explained in more detail in
Section 7.
Finally Fig. 4 shows the relation between approximation qual-
ity and sparsity of representation in the SV case. The lower the
precision required for approximating the original data, the fewer
SVs are needed to encode that. The non-SVs are redundant, i.e.
even without these patterns in the training set, the SV machine
would have constructed exactly the same function f . One might
think that this could be an efficient way of data compression,
namely by storing only the support patterns, from which the es-
timate can be reconstructed completely. However, this simple
analogy turns out to fail in the case of high-dimensional data,
and even more drastically in the presence of noise. In Vapnik,
Fig. 2. Architecture of a regression machine constructed by the SV Golowich and Smola (1997) one can see that even for moderate
algorithm approximation quality, the number of SVs can be considerably
high, yielding rates worse than the Nyquist rate (Nyquist 1928,
Shannon 1948).
the map . This corresponds to evaluating kernel functions
k(xi , x). Finally the dot products are added up using the weights
νi = αi − αi∗ . This, plus the constant term b yields the final 5. Optimization algorithms
prediction output. The process described here is very similar to
regression in a neural network, with the difference, that in the While there has been a large number of implementations of SV
SV case the weights in the input layer are a subset of the training algorithms in the past years, we focus on a few algorithms which
patterns. will be presented in greater detail. This selection is somewhat
Figure 3 demonstrates how the SV algorithm chooses the biased, as it contains these algorithms the authors are most fa-
flattest function among those approximating the original data miliar with. However, we think that this overview contains some
with a given precision. Although requiring flatness only in of the most effective ones and will be useful for practitioners
feature space, one can observe that the functions also are who would like to actually code a SV machine by themselves.
very flat in input space. This is due to the fact, that ker- But before doing so we will briefly cover major optimization
nels can be associated with flatness properties via regular- packages and strategies.
Fig. 3. Left to right: approximation of the function sinc x with precisions ε = 0.1, 0.2, and 0.5. The solid top and the bottom lines indicate the size
of the ε-tube, the dotted line in between is the regression
Fig. 4. Left to right: regression (solid line), datapoints (small dots) and SVs (big dots) for an approximation with ε = 0.1, 0.2, and 0.5. Note the
decrease in the number of SVs
5.1. Implementations subset selection techniques (see Section 5.5) to address this
problem.
Most commercially available packages for quadratic program-
ming can also be used to train SV machines. These are usually
numerically very stable general purpose codes, with special en- 5.2. Basic notions
hancements for large sparse systems. While the latter is a feature
Most algorithms rely on results from the duality theory in convex
that is not needed at all in SV problems (there the dot product
optimization. Although we already happened to mention some
matrix is dense and huge) they still can be used with good suc-
basic ideas in Section 1.2 we will, for the sake of convenience,
cess.6
briefly review without proof the core results. These are needed
in particular to derive an interior point algorithm. For details and
OSL: This package was written by IBM-Corporation (1992). It proofs (see e.g. Fletcher 1989).
uses a two phase algorithm. The first step consists of solving
Uniqueness: Every convex constrained optimization problem
a linear approximation of the QP problem by the simplex al-
has a unique minimum. If the problem is strictly convex then
gorithm (Dantzig 1962). Next a related very simple QP prob-
the solution is unique. This means that SVs are not plagued
lem is dealt with. When successive approximations are close
with the problem of local minima as Neural Networks are.7
enough together, the second subalgorithm, which permits a
Lagrange function: The Lagrange function is given by the pri-
quadratic objective and converges very rapidly from a good
mal objective function minus the sum of all products between
starting value, is used. Recently an interior point algorithm
constraints and corresponding Lagrange multipliers (cf. e.g.
was added to the software suite.
Fletcher 1989, Bertsekas 1995). Optimization can be seen
CPLEX by CPLEX-Optimization-Inc. (1994) uses a primal-dual
as minimzation of the Lagrangian wrt. the primal variables
logarithmic barrier algorithm (Megiddo 1989) instead with
and simultaneous maximization wrt. the Lagrange multipli-
predictor-corrector step (see e.g. Lustig, Marsten and Shanno
ers, i.e. dual variables. It has a saddle point at the solution.
1992, Mehrotra and Sun 1992).
Usually the Lagrange function is only a theoretical device to
MINOS by the Stanford Optimization Laboratory (Murtagh and
derive the dual objective function (cf. Section 1.2).
Saunders 1983) uses a reduced gradient algorithm in con-
Dual objective function: It is derived by minimizing the
junction with a quasi-Newton algorithm. The constraints are
Lagrange function with respect to the primal variables and
handled by an active set strategy. Feasibility is maintained
subsequent elimination of the latter. Hence it can be written
throughout the process. On the active constraint manifold, a
solely in terms of the dual variables.
quasi-Newton approximation is used.
Duality gap: For both feasible primal and dual variables the pri-
MATLAB: Until recently the matlab QP optimizer delivered only
mal objective function (of a convex minimization problem)
agreeable, although below average performance on classifi-
is always greater or equal than the dual objective function.
cation tasks and was not all too useful for regression tasks
Since SVMs have only linear constraints the constraint qual-
(for problems much larger than 100 samples) due to the fact
ifications of the strong duality theorem (Bazaraa, Sherali and
that one is effectively dealing with an optimization prob-
Shetty 1993, Theorem 6.2.4) are satisfied and it follows that
lem of size 2 where at least half of the eigenvalues of the
gap vanishes at optimality. Thus the duality gap is a measure
Hessian vanish. These problems seem to have been addressed
how close (in terms of the objective function) the current set
in version 5.3 / R11. Matlab now uses interior point codes.
of variables is to the solution.
LOQO by Vanderbei (1994) is another example of an interior
Karush–Kuhn–Tucker (KKT) conditions: A set of primal and
point code. Section 5.3 discusses the underlying strategies in
dual variables that is both feasible and satisfies the KKT
detail and shows how they can be adapted to SV algorithms.
conditions is the solution (i.e. constraint · dual variable = 0).
Maximum margin perceptron by Kowalczyk (2000) is an algo-
The sum of the violated KKT terms determines exactly the
rithm specifically tailored to SVs. Unlike most other tech-
size of the duality gap (that is, we simply compute the
niques it works directly in primal space and thus does not
constraint · Lagrangemultiplier part as done in (55)). This
have to take the equality constraint on the Lagrange multipli-
allows us to compute the latter quite easily.
ers into account explicitly.
A simple intuition is that for violated constraints the dual
Iterative free set methods The algorithm by Kaufman (Bunch,
variable could be increased arbitrarily, thus rendering the
Kaufman and Parlett 1976, Bunch and Kaufman 1977, 1980,
Lagrange function arbitrarily large. This, however, is in con-
Drucker et al. 1997, Kaufman 1999), uses such a technique
tradition to the saddlepoint property.
starting with all variables on the boundary and adding them as
the Karush Kuhn Tucker conditions become more violated.
5.3. Interior point algorithms
This approach has the advantage of not having to compute
the full dot product matrix from the beginning. Instead it is In a nutshell the idea of an interior point algorithm is to com-
evaluated on the fly, yielding a performance improvement pute the dual of the optimization problem (in our case the dual
in comparison to tackling the whole optimization problem dual of Rreg [ f ]) and solve both primal and dual simultaneously.
at once. However, also other algorithms can be modified by This is done by only gradually enforcing the KKT conditions
to iteratively find a feasible solution and to use the duality Assuming that the (dominant) convex part q(α) of the pri-
gap between primal and dual objective function to determine mal objective is quadratic, the q scales with τ 2 where as the
the quality of the current set of variables. The special flavour linear part scales with τ . However, since the linear term dom-
of algorithm we will describe is primal-dual path-following inates the objective function, the rescaled values are still a
(Vanderbei 1994). better starting point than α = 0. In practice a speedup of
In order to avoid tedious notation we will consider the slightly approximately 95% of the overall training time can be ob-
more general problem and specialize the result to the SVM later. served when using the sequential minimization algorithm,
It is understood that unless stated otherwise, variables like α cf. (Smola 1998). A similar reasoning can be applied when
denote vectors and αi denotes its i-th component. retraining with the same regularization parameter but differ-
1 ent (yet similar) width parameters of the kernel function. See
minimize q(α) + c, α Cristianini, Campbell and Shawe-Taylor (1998) for details
2 (50)
subject to Aα = b and l ≤ α ≤ u thereon in a different context.
with c, α, l, u ∈ Rn , A ∈ Rn·m , b ∈ Rm , the inequalities be- Monitoring convergence via the feasibility gap: In the case of
tween vectors holding componentwise and q(α) being a convex both primal and dual feasible variables the following con-
function of α. Now we will add slack variables to get rid of all nection between primal and dual objective function holds:

inequalities but the positivity constraints. This yields: Dual Obj. = Primal Obj. − (gi z i + si ti ) (55)
1 i
minimize q(α) + c, α
2 This can be seen immediately by the construction of the
subject to Aα = b, α − g = l, α + t = u, (51)
g, t ≥ 0, α free
Lagrange function. In Regression Estimation (with the ε-
insensitive loss function) one obtains for i gi z i + si ti
 
The dual of (51) is + max(0, f (xi ) − (yi + εi ))(C − αi∗ )
1   − min(0, f (xi ) − (yi + εi ))αi
∗ 

maximize (q(α) − ∂q(α), α) + b, y + l, z − u, s   . (56)
2  + max(0, (y − ε ∗
) − f (x ))(C − α ) 
i i i i
1 i
subject to ∂q(α) + c − (Ay) + s = z, s, z ≥ 0, y free − min(0, (yi − εi∗ ) − f (xi ))αi
2
(52) Thus convergence with respect to the point of the solution
can be expressed in terms of the duality gap. An effective
Moreover we get the KKT conditions, namely stopping rule is to require

gi z i = 0 and si ti = 0 for all i ∈ [1 . . . n]. (53) i gi z i + si ti
≤ εtol (57)
A necessary and sufficient condition for the optimal solution is |Primal Objective| + 1
that the primal/dual variables satisfy both the feasibility condi- for some precision εtol . This condition is much in the spirit of
tions of (51) and (52) and the KKT conditions (53). We pro- primal dual interior point path following algorithms, where
ceed to solve (51)–(53) iteratively. The details can be found in convergence is measured in terms of the number of significant
Appendix A. figures (which would be the decimal logarithm of (57)), a
convention that will also be adopted in the subsequent parts
5.4. Useful tricks of this exposition.
Before proceeding to further algorithms for quadratic optimiza-

tion let us briefly mention some useful tricks that can be applied 5.5. Subset selection algorithms
to all algorithms described subsequently and may have signif- The convex programming algorithms described so far can be
icant impact despite their simplicity. They are in part derived used directly on moderately sized (up to 3000) samples datasets
from ideas of the interior-point approach. without any further modifications. On large datasets, however, it
Training with different regularization parameters: For several is difficult, due to memory and cpu limitations, to compute the
reasons (model selection, controlling the number of support dot product matrix k(xi , x j ) and keep it in memory. A simple
vectors, etc.) it may happen that one has to train a SV ma- calculation shows that for instance storing the dot product matrix
chine with different regularization parameters C, but other- of the NIST OCR database (60.000 samples) at single precision
wise rather identical settings. If the parameters Cnew = τ Cold would consume 0.7 GBytes. A Cholesky decomposition thereof,
is not too different it is advantageous to use the rescaled val- which would additionally require roughly the same amount of
ues of the Lagrange multipliers (i.e. αi , αi∗ ) as a starting point memory and 64 Teraflops (counting multiplies and adds sepa-
for the new optimization problem. Rescaling is necessary to rately), seems unrealistic, at least at current processor speeds.
satisfy the modified constraints. One gets A first solution, which was introduced in Vapnik (1982) relies
on the observation that the solution can be reconstructed from
αnew = τ αold and likewise bnew = τ bold . (54) the SVs alone. Hence, if we knew the SV set beforehand, and
it fitted into memory, then we could directly solve the reduced tions an explicit solution of the restricted quadratic programming
problem. The catch is that we do not know the SV set before problem is impossible. Yet, one could derive an analogous non-
solving the problem. The solution is to start with an arbitrary quadratic convex optimization problem for general cost func-
subset, a first chunk that fits into memory, train the SV algorithm tions but at the expense of having to solve it numerically.
on it, keep the SVs and fill the chunk up with data the current The exposition proceeds as follows: first one has to derive
estimator would make errors on (i.e. data lying outside the ε- the (modified) boundary conditions for the constrained 2 indices
tube of the current regression). Then retrain the system and keep (i, j) subproblem in regression, next one can proceed to solve the
on iterating until after training all KKT-conditions are satisfied. optimization problem analytically, and finally one has to check,
The basic chunking algorithm just postponed the underlying which part of the selection rules have to be modified to make
problem of dealing with large datasets whose dot-product matrix the approach work for regression. Since most of the content is
cannot be kept in memory: it will occur for larger training set fairly technical it has been relegated to Appendix C.
sizes than originally, but it is not completely avoided. Hence The main difference in implementations of SMO for regres-
the solution is Osuna, Freund and Girosi (1997) to use only a sion can be found in the way the constant offset b is determined
subset of the variables as a working set and optimize the problem (Keerthi et al. 1999) and which criterion is used to select a new
with respect to them while freezing the other variables. This set of variables. We present one such strategy in Appendix C.3.
method is described in detail in Osuna, Freund and Girosi (1997), However, since selection strategies are the focus of current re-
Joachims (1999) and Saunders et al. (1998) for the case of pattern search we recommend that readers interested in implementing
recognition.8 the algorithm make sure they are aware of the most recent de-
An adaptation of these techniques to the case of regression velopments in this area.
with convex cost functions can be found in Appendix B. The Finally, we note that just as we presently describe a generaliza-
basic structure of the method is described by Algorithm 1. tion of SMO to regression estimation, other learning problems
can also benefit from the underlying ideas. Recently, a SMO
Algorithm 1.: Basic structure of a working set algorithm algorithm for training novelty detection systems (i.e. one-class
Initialize αi , αi∗ = 0 classification) has been proposed (Schölkopf et al. 2001).
Choose arbitrary working set Sw
repeat 6. Variations on a theme
Compute coupling terms (linear and constant) for Sw (see
Appendix A.3) There exists a large number of algorithmic modifications of the
Solve reduced optimization problem SV algorithm, to make it suitable for specific settings (inverse
Choose new Sw from variables αi , αi∗ not satisfying the problems, semiparametric settings), different ways of measuring
KKT conditions capacity and reductions to linear programming (convex com-
until working set Sw = ∅ binations) and different ways of controlling capacity. We will
mention some of the more popular ones.
5.6. Sequential minimal optimization

6.1. Convex combinations and 1 -norms
Recently an algorithm—Sequential Minimal Optimization
(SMO)—was proposed (Platt 1999) that puts chunking to the All the algorithms presented so far involved convex, and at
extreme by iteratively selecting subsets only of size 2 and op- best, quadratic programming. Yet one might think of reducing
timizing the target function with respect to them. It has been the problem to a case where linear programming techniques
reported to have good convergence properties and it is easily can be applied. This can be done in a straightforward fashion
implemented. The key point is that for a working set of 2 the (Mangasarian 1965, 1968, Weston et al. 1999, Smola, Schölkopf
optimization subproblem can be solved analytically without ex- and Rätsch 1999) for both SV pattern recognition and regression.
plicitly invoking a quadratic optimizer. The key is to replace (35) by
While readily derived for pattern recognition by Platt (1999), Rreg [ f ] := Remp [ f ] + λα1 (58)
one simply has to mimick the original reasoning to obtain an
extension to Regression Estimation. This is what will be done where α1 denotes the 1 norm in coefficient space. Hence one
in Appendix C (the pseudocode can be found in Smola and uses the SV kernel expansion (11)
Schölkopf (1998b)). The modifications consist of a pattern de-

pendent regularization, convergence control via the number of f (x) = αi k(xi , x) + b
significant figures, and a modified system of equations to solve i=1
the optimization problem in two variables for regression analyt- with a different way of controlling capacity by minimizing
ically.
Note that the reasoning only applies to SV regression with 1
Rreg [ f ] = c(xi , yi , f (xi )) + λ |αi |. (59)
the ε insensitive loss function—for most other convex cost func- i=1 i=1
For the ε-insensitive loss function this leads to a linear program- for some ν > 0. Hence (42) becomes (again carrying out the
ming problem. In the other cases, however, the problem still stays usual transformation between λ, and C)
a quadratic or general convex one, and therefore may not yield
1
∗
the desired computational advantage. Therefore we will limit minimize w + C
2
(c̃(ξi ) + c̃(ξi )) + νε
ourselves to the derivation of the linear programming problem 2 i=1
in the case of | · |ε cost function. Reformulating (59) yields  (61)

 yi − w, x i − b ≤ ε + ξ i

(αi + αi∗ ) + C (ξi + ξi∗ ) subject to w, xi + b − yi ≤ ε + ξi∗
minimize 

i=1 i=1 ξi , ξi∗ ≥ 0



We consider the standard | · |ε loss function. Computing the dual

 y − (α j − α ∗j )k(x j , xi ) − b ≤ ε + ξi


i of (62) yields
 j=1 
subject to 

 1


 (α j − α ∗j )k(x j , xi ) + b − yi ≤ ε + ξi∗  −
 2 (αi − αi∗ )(α j − α ∗j )k(xi , x j )



 j=1 maximize i, j=1
αi , αi∗ , ξi , ξi∗ ≥ 0 


 yi (αi − αi∗ )
+
Unlike in the classical SV case, the transformation into its dual i=1
 (62)
does not give any improvement in the structure of the optimiza-


tion problem. Hence it is best to minimize Rreg [ f ] directly, which 
 (αi − αi∗ ) = 0


can be achieved by a linear optimizer, (e.g. Dantzig 1962, Lustig,  i=1
Marsten and Shanno 1990, Vanderbei 1997). subject to

 (αi + αi∗ ) ≤ Cν
In (Weston et al. 1999) a similar variant of the linear SV ap- 



proach is used to estimate densities on a line. One can show  i=1 ∗
αi , αi ∈ [0, C]
(Smola et al. 2000) that one may obtain bounds on the gener-
alization error which exhibit even better rates (in terms of the Note that the optimization problem is thus very similar to the ε-
entropy numbers) than the classical SV case (Williamson, Smola SV one: the target function is even simpler (it is homogeneous),
and Schölkopf 1998). but there is an additional constraint. For information on how this
affects the implementation (cf. Chang and Lin 2001).
Besides having the advantage of being able to automatically
6.2. Automatic tuning of the insensitivity tube determine ε (63) also has another advantage. It can be used to
Besides standard model selection issues, i.e. how to specify the pre–specify the number of SVs:
trade-off between empirical error and model capacity there also
exists the problem of an optimal choice of a cost function. In Theorem 9 (Schölkopf et al. 2000).
particular, for the ε-insensitive cost function we still have the 1. ν is an upper bound on the fraction of errors.
problem of choosing an adequate parameter ε in order to achieve 2. ν is a lower bound on the fraction of SVs.
good performance with the SV machine. 3. Suppose the data has been generated iid from a distribution
Smola et al. (1998a) show the existence of a linear depen- p(x, y) = p(x) p(y | x) with a continuous conditional distri-
dency between the noise level and the optimal ε-parameter for bution p(y | x). With probability 1, asymptotically, ν equals
SV regression. However, this would require that we know some- the fraction of SVs and the fraction of errors.
thing about the noise model. This knowledge is not available in
general. Therefore, albeit providing theoretical insight, this find- Essentially, ν-SV regression improves upon ε-SV regression by
ing by itself is not particularly useful in practice. Moreover, if we allowing the tube width to adapt automatically to the data. What
really knew the noise model, we most likely would not choose is kept fixed up to this point, however, is the shape of the tube.
the ε-insensitive cost function but the corresponding maximum One can, however, go one step further and use parametric tube
likelihood loss function instead. models with non-constant width, leading to almost identical op-
There exists, however, a method to construct SV machines timization problems (Schölkopf et al. 2000).
that automatically adjust ε and moreover also, at least asymptot- Combining ν-SV regression with results on the asymptotical
ically, have a predetermined fraction of sampling points as SVs optimal choice of ε for a given noise model (Smola et al. 1998a)
(Schölkopf et al. 2000). We modify (35) such that ε becomes a leads to a guideline how to adjust ν provided the class of noise
variable of the optimization problem, including an extra term in models (e.g. Gaussian or Laplacian) is known.
the primal objective function which attempts to minimize ε. In
other words Remark 10 (Optimal choice of ν). Denote by p a probability
λ density with unit variance, and by P a famliy of noise models
minimize Rν [ f ] := Remp [ f ] + w2 + νε (60) generated from p by P := { p| p = σ1 p( σy )}. Moreover assume
2
by making problems seemingly easier yet reliable via a map into
some even higher dimensional space.
In this section we focus on the connections between SV
methods and previous techniques like Regularization Networks
(Girosi, Jones and Poggio 1993).9 In particular we will show
that SV machines are essentially Regularization Networks (RN)
with a clever choice of cost functions and that the kernels are
Green’s function of the corresponding regularization operators.
For a full exposition of the subject the reader is referred to Smola,
Schölkopf and Müller (1998c).
7.1. Regularization networks

Let us briefly review the basic concepts of RNs. As in (35)
we minimize a regularized risk functional. However, rather than
enforcing flatness in feature space we try to optimize some
smoothness criterion for the function in input space. Thus we
get
Fig. 5. Optimal ν and ε for various degrees of polynomial additive
noise λ
Rreg [ f ] := Remp [ f ] + P f 2 . (64)
2
that the data were drawn iid from p(x, y) = p(x) p(y − f (x))
with p(y − f (x)) continuous. Then under the assumption of Here P denotes a regularization operator in the sense of
uniform convergence, the asymptotically optimal value of ν is Tikhonov and Arsenin (1977), i.e. P is a positive semidefinite
ε operator mapping from the Hilbert space H of functions f under
ν = 1− p(t) dt consideration to a dot product space D such that the expression
−ε
P f · Pg is well defined for f, g ∈ H . For instance by choos-
τ
−2 ing a suitable operator that penalizes large variations of f one
where ε := argmin(p(−τ ) + p(τ )) 1 − p(t) dt (63)
τ −τ can reduce the well–known overfitting effect. Another possible
setting also might be an operator P mapping from L 2 (Rn ) into
For polynomial noise models, i.e. densities of type exp(−|ξ | p ) some Reproducing Kernel Hilbert Space (RKHS) (Aronszajn,
one may compute the corresponding (asymptotically) optimal 1950, Kimeldorf and Wahba 1971, Saitoh 1988, Schölkopf 1997,
values of ν. They are given in Fig. 5. For further details see Girosi 1998).
(Schölkopf et al. 2000, Smola 1998); an experimental validation Using an expansion of f in terms of some symmetric function
has been given by Chalimourda, Schölkopf and Smola (2000). k(xi , x j ) (note here, that k need not fulfill Mercer’s condition
We conclude this section by noting that ν-SV regression is and can be chosen arbitrarily since it is not used to define a
related to the idea of trimmed estimators. One can show that the regularization term),
regression is not influenced if we perturb points lying outside the
tube. Thus, the regression is essentially computed by discarding

a certain fraction of outliers, specified by ν, and computing the f (x) = αi k(xi , x) + b, (65)
i=1
regression estimate from the remaining points (Schölkopf et al.
2000). and the ε-insensitive cost function, this leads to a quadratic pro-
gramming problem similar to the one for SVs. Using
7. Regularization
Di j := (Pk)(xi , .) · (Pk)(x j , .) (66)
So far we were not concerned about the specific properties of
the map into feature space and used it only as a convenient we get α = D −1 K (β − β ∗ ), with β, β ∗ being the solution of
trick to construct nonlinear regression functions. In some cases
1 ∗
the map was just given implicitly by the kernel, hence the map minimize (β − β) KD−1 K (β ∗ − β)
itself and many of its properties have been neglected. A deeper 2

understanding of the kernel map would also be useful to choose −(β ∗ − β) y − ε (βi + βi∗ ) (67)
appropriate kernels for a specific task (e.g. by incorporating i=1

prior knowledge (Schölkopf et al. 1998a)). Finally the feature subject to (βi − βi∗ ) = 0 and βi , βi∗ ∈ [0, C].
map seems to defy the curse of dimensionality (Bellman 1961) i=1
Unfortunately, this setting of the problem does not preserve spar- with f˜(ω) denoting the Fourier transform of f (x), and P(ω) =
sity in terms of the coefficients, as a potentially sparse decom- P(−ω) real valued, nonnegative and converging to 0 for |ω| →
position in terms of βi and βi∗ is spoiled by D −1 K , which is not ∞ and := supp[P(ω)]. Small values of P(ω) correspond to
in general diagonal. a strong attenuation of the corresponding frequencies. Hence
small values of P(ω) for large ω are desirable since high fre-
7.2. Green’s functions quency components of f˜ correspond to rapid changes in f .
P(ω) describes the filter properties of P ∗ P. Note that no atten-
Comparing (10) with (67) leads to the question whether and un- uation takes place for P(ω) = 0 as these frequencies have been
der which condition the two methods might be equivalent and excluded from the integration domain.
therefore also under which conditions regularization networks For regularization operators defined in Fourier Space by (70)
might lead to sparse decompositions, i.e. only a few of the ex- one can show by exploiting P(ω) = P(−ω) = P(ω) that
pansion coefficients αi in f would differ from zero. A sufficient
1
condition is D = K and thus KD−1 K = K (if K does not have G(xi , x) = eiω(xi −x) P(ω) dω (71)
(2π )n/2 Rn
full rank we only need that KD−1 K = K holds on the image of
K ): is a corresponding Green’s function satisfying translational in-
variance, i.e.
k(xi , x j ) = (Pk)(xi , .) · (Pk)(x j , .) (68)
G(xi , x j ) = G(xi − x j ) and ˜
G(ω) = P(ω). (72)
Our goal now is to solve the following two problems:
This provides us with an efficient tool for analyzing SV kernels
1. Given a regularization operator P, find a kernel k such that a and the types of capacity control they exhibit. In fact the above
SV machine using k will not only enforce flatness in feature is a special case of Bochner’s theorem (Bochner 1959) stating
space, but also correspond to minimizing a regularized risk that the Fourier transform of a positive measure constitutes a
functional with P as regularizer. positive Hilbert Schmidt kernel.
2. Given an SV kernel k, find a regularization operator P such
that a SV machine using this kernel can be viewed as a Reg-
Example 2 (Gaussian kernels). Following the exposition of
ularization Network using P.
Yuille and Grzywacz (1988) as described in Girosi, Jones and
These two problems can be solved by employing the concept Poggio (1993), one can see that for
of Green’s functions as described in Girosi, Jones and Poggio σ 2m
(1993). These functions were introduced for the purpose of solv- P f 2 = d x m
( Ô m f (x))2 (73)
m m!2
ing differential equations. In our context it is sufficient to know
that the Green’s functions G xi (x) of P ∗ P satisfy with Ô 2m = m and Ô 2m+1 = ∇m , being the Laplacian
(P ∗ P G xi )(x) = δxi (x). (69) and ∇ the Gradient operator, we get Gaussians kernels (31).
Moreover, we can provide an equivalent representation of P
σ 2 ω2
Here, δxi (x) is the δ-distribution (not to be confused with the Kro- in terms of its Fourier properties, i.e. P(ω) = e− 2 up to a
necker symbol δi j ) which has the property that f · δxi = f (xi ). multiplicative constant.
The relationship between kernels and regularization operators is
formalized in the following proposition: Training an SV machine with Gaussian RBF kernels (Schölkopf
et al. 1997) corresponds to minimizing the specific cost func-
Proposition 1 (Smola, Schölkopf and Müller 1998b). Let P tion with a regularization operator of type (73). Recall that (73)
be a regularization operator, and G be the Green’s function of means that all derivatives of f are penalized (we have a pseudod-
P ∗ P. Then G is a Mercer Kernel such that D = K . SV machines ifferential operator) to obtain a very smooth estimate. This also
using G minimize risk functional (64) with P as regularization explains the good performance of SV machines in this case, as it
operator. is by no means obvious that choosing a flat function in some high
dimensional space will correspond to a simple function in low
In the following we will exploit this relationship in both ways: dimensional space, as shown in Smola, Schölkopf and Müller
to compute Green’s functions for a given regularization operator (1998c) for Dirichlet kernels.
P and to infer the regularizer, given a kernel k. The question that arises now is which kernel to choose. Let
us think about two extreme situations.
7.3. Translation invariant kernels 1. Suppose we already knew the shape of the power spectrum
Pow(ω) of the function we would like to estimate. In this case
Let us now more specifically consider regularization operators
we choose k such that k̃ matches the power spectrum (Smola
P̂ that may be written as multiplications in Fourier space
1998).
˜
1 ˜
f (ω)g(ω) 2. If we happen to know very little about the given data a gen-
P f · Pg = dω (70)
(2π) n/2
P(ω) eral smoothness assumption is a reasonable choice. Hence
we might want to choose a Gaussian kernel. If computing function class in all possible ways, the Covering Number which
time is important one might moreover consider kernels with is the number of elements from F that are needed to cover F with
compact support, e.g. using the Bq –spline kernels (cf. (32)). accuracy of at least ε, Entropy Numbers which are the functional
This choice will cause many matrix elements ki j = k(xi −x j ) inverse of Covering Numbers, and many more variants thereof
to vanish. (see e.g. Vapnik 1982, 1998, Devroye, Györfi and Lugosi 1996,
Williamson, Smola and Schölkopf 1998, Shawe-Taylor et al.
The usual scenario will be in between the two extreme cases and 1998).
we will have some limited prior knowledge available. For more
information on using prior knowledge for choosing kernels (see
Schölkopf et al. 1998a). 8. Conclusion
Due to the already quite large body of work done in the field of
7.4. Capacity control SV research it is impossible to write a tutorial on SV regression
which includes all contributions to this field. This also would
All the reasoning so far was based on the assumption that there be quite out of the scope of a tutorial and rather be relegated to
exist ways to determine model parameters like the regularization textbooks on the matter (see Schölkopf and Smola (2002) for a
constant λ or length scales σ of rbf–kernels. The model selec- comprehensive overview, Schölkopf, Burges and Smola (1999a)
tion issue itself would easily double the length of this review for a snapshot of the current state of the art, Vapnik (1998) for an
and moreover it is an area of active and rapidly moving research. overview on statistical learning theory, or Cristianini and Shawe-
Therefore we limit ourselves to a presentation of the basic con- Taylor (2000) for an introductory textbook). Still the authors
cepts and refer the interested reader to the original publications. hope that this work provides a not overly biased view of the state
It is important to keep in mind that there exist several fun- of the art in SV regression research. We deliberately omitted
damentally different approaches such as Minimum Description (among others) the following topics.
Length (cf. e.g. Rissanen 1978, Li and Vitányi 1993) which is
based on the idea that the simplicity of an estimate, and therefore
8.1. Missing topics
also its plausibility is based on the information (number of bits)
needed to encode it such that it can be reconstructed. Mathematical programming: Starting from a completely differ-
Bayesian estimation, on the other hand, considers the pos- ent perspective algorithms have been developed that are sim-
terior probability of an estimate, given the observations X = ilar in their ideas to SV machines. A good primer might
{(x1 , y1 ), . . . (x , y )}, an observation noise model, and a prior be (Bradley, Fayyad and Mangasarian 1998). (Also see
probability distribution p( f ) over the space of estimates Mangasarian 1965, 1969, Street and Mangasarian 1995). A
(parameters). It is given by Bayes Rule p( f | X ) p(X ) = comprehensive discussion of connections between mathe-
p(X | f ) p( f ). Since p(X ) does not depend on f , one can maxi- matical programming and SV machines has been given by
mize p(X | f ) p( f ) to obtain the so-called MAP estimate.10 As (Bennett 1999).
a rule of thumb, to translate regularized risk functionals into Density estimation: with SV machines (Weston et al. 1999,
Bayesian MAP estimation schemes, all one has to do is to con- Vapnik 1999). There one makes use of the fact that the cu-
sider exp(−Rreg [ f ]) = p( f | X ). For a more detailed discussion mulative distribution function is monotonically increasing,
(see e.g. Kimeldorf and Wahba 1970, MacKay 1991, Neal 1996, and that its values can be predicted with variable confidence
Rasmussen 1996, Williams 1998). which is adjusted by selecting different values of ε in the loss
A simple yet powerful way of model selection is cross valida- function.
tion. This is based on the idea that the expectation of the error Dictionaries: were originally introduced in the context of
on a subset of the training sample not used during training is wavelets by (Chen, Donoho and Saunders 1999) to allow
identical to the expected error itself. There exist several strate- for a large class of basis functions to be considered simulta-
gies such as 10-fold crossvalidation, leave-one out error (-fold neously, e.g. kernels with different widths. In the standard SV
crossvalidation), bootstrap and derived algorithms to estimate case this is hardly possible except by defining new kernels as
the crossvalidation error itself (see e.g. Stone 1974, Wahba 1980, linear combinations of differently scaled ones: choosing the
Efron 1982, Efron and Tibshirani 1994, Wahba 1999, Jaakkola regularization operator already determines the kernel com-
and Haussler 1999) for further details. pletely (Kimeldorf and Wahba 1971, Cox and O’Sullivan
Finally, one may also use uniform convergence bounds such 1990, Schölkopf et al. 2000). Hence one has to resort to lin-
as the ones introduced by Vapnik and Chervonenkis (1971). The ear programming (Weston et al. 1999).
basic idea is that one may bound with probability 1 − η (with Applications: The focus of this review was on methods and
η > 0) the expected risk R[ f ] by Remp [ f ] + (F, η), where theory rather than on applications. This was done to limit
is a confidence term depending on the class of functions F. the size of the exposition. State of the art, or even record
Several criteria for measuring the capacity of F exist, such as the performance was reported in Müller et al. (1997), Drucker
VC-Dimension which, in pattern recognition problems, is given et al. (1997), Stitson et al. (1999) and Mattera and Haykin
by the maximum number of points that can be separated by the (1999).
In many cases, it may be possible to achieve similar per- this exciting field, may find it useful to consult the web page
formance with neural network methods, however, only if www.kernel-machines.org.
many parameters are optimally tuned by hand, thus depend-
ing largely on the skill of the experimenter. Certainly, SV
machines are not a “silver bullet.” However, as they have Appendix A: Solving the interior-point
only few critical parameters (e.g. regularization and kernel equations
width), state-of-the-art results can be achieved with relatively
little effort. A.1. Path following
Rather than trying to satisfy (53) directly we will solve a modified
8.2. Open issues version thereof for some µ > 0 substituted on the rhs in the first
Being a very active field there exist still a number of open is- place and decrease µ while iterating.
sues that have to be addressed by future research. After that
the algorithmic development seems to have found a more sta- gi z i = µ, si ti = µ for all i ∈ [1 . . . n]. (74)
ble stage, one of the most important ones seems to be to find
tight error bounds derived from the specific properties of ker- Still it is rather difficult to solve the nonlinear system of equa-
nel functions. It will be of interest in this context, whether tions (51), (52), and (74) exactly. However we are not interested
SV machines, or similar approaches stemming from a lin- in obtaining the exact solution to the approximation (74). In-
ear programming regularizer, will lead to more satisfactory stead, we seek a somewhat more feasible solution for a given µ,
results. then decrease µ and repeat. This can be done by linearizing the
Moreover some sort of “luckiness framework” (Shawe-Taylor above system and solving the resulting equations by a predictor–
et al. 1998) for multiple model selection parameters, similar to corrector approach until the duality gap is small enough. The
multiple hyperparameters and automatic relevance detection in advantage is that we will get approximately equal performance
Bayesian statistics (MacKay 1991, Bishop 1995), will have to as by trying to solve the quadratic system directly, provided that
be devised to make SV machines less dependent on the skill of the terms in 2 are small enough.
the experimenter.
It is also worth while to exploit the bridge between regulariza- A(α + α) = b
tion operators, Gaussian processes and priors (see e.g. (Williams
α + α − g − g = l
1998)) to state Bayesian risk bounds for SV machines in order
to compare the predictions with the ones from VC theory. Op- α + α + t + t = u
timization techniques developed in the context of SV machines 1 1
also could be used to deal with large datasets in the Gaussian c + ∂α q(α) + ∂α2 q(α)α − (A(y + y))
2 2
process settings.
+ s + s = z + z
Prior knowledge appears to be another important question in
SV regression. Whilst invariances could be included in pattern (gi + gi )(z i + z i ) = µ
recognition in a principled way via the virtual SV mechanism
(si + si )(ti + ti ) = µ
and restriction of the feature space (Burges and Schölkopf 1997,
Schölkopf et al. 1998a), it is still not clear how (probably) more
Solving for the variables in we get
subtle properties, as required for regression, could be dealt with
efficiently.
Aα = b − Aα =: ρ
Reduced set methods also should be considered for speeding
up prediction (and possibly also training) phase for large datasets α − g = l − α + g =: ν
(Burges and Schölkopf 1997, Osuna and Girosi 1999, Schölkopf
α + t = u − α − t =: τ
et al. 1999b, Smola and Schölkopf 2000). This topic is of great
1
importance as data mining applications require algorithms that (Ay) + z − s − ∂α2 q(α)α
are able to deal with databases that are often at least one order of 2
magnitude larger (1 million samples) than the current practical 1
size for SV regression. = c − (Ay) + s − z + ∂α q(α) =: σ
2
Many more aspects such as more data dependent generaliza-
g −1 zg + z = µg −1 − z − g −1 gz =: γz
tion bounds, efficient training algorithms, automatic kernel se-
lection procedures, and many techniques that already have made t −1 st + s = µt −1 − s − t −1 ts =: γs
their way into the standard neural networks toolkit, will have to
be considered in the future. where g −1 denotes the vector (1/g1 , . . . , 1/gn ), and t analo-
Readers who are tempted to embark upon a more detailed gously. Moreover denote g −1 z and t −1 s the vector generated
exploration of these topics, and to contribute their own ideas to by the componentwise product of the two vectors. Solving for
g, t, z, s we get and subsequently restricts the solution to a feasible set

u
g = z −1 g(γz − z) z = g −1 z(ν̂ − α) x = max x,
100
t = s −1 t(γs − s) s = t −1 s(α − τ̂ )
(75) g = min(α − l, u)
where ν̂ := ν − z −1 gγz
τ̂ := τ − s −1 tγs t = min(u − α, u) (79)

1 u
Now we can formulate the reduced KKT–system (see (Vanderbei z = min ∂α q(α) + c − (Ay) + ,u
2 100
1994) for the quadratic case):
1 u
s = min − ∂α q(α) − c + (Ay) + ,u
−H A α σ − g −1 z ν̂ − t −1 s τ̂ 2 100
= (76)
A 0 y ρ (·) denotes the Heavyside function, i.e. (x) = 1 for x > 0
and (x) = 0 otherwise.
where H := ( 12 ∂α2 q(α) + g −1 z + t −1 s).
A.3. Special considerations for SV regression
A.2. Iteration strategies The algorithm described so far can be applied to both SV pattern
recognition and regression estimation. For the standard setting
For the predictor-corrector method we proceed as follows. In in pattern recognition we have
the predictor step solve the system of (75) and (76) with µ = 0
and all -terms on the rhs set to 0, i.e. γz = z, γs = s. The

q(α) = αi α j yi y j k(xi , x j ) (80)
values in are substituted back into the definitions for γz and i, j=0
γs and (75) and (76) are solved again in the corrector step. As the
quadratic part in (76) is not affected by the predictor–corrector and consequently ∂αi q(α) = 0, ∂α2i α j q(α) = yi y j k(xi , x j ), i.e.
steps, we only need to invert the quadratic matrix once. This is the Hessian is dense and the only thing we can do is compute
done best by manually pivoting for the H part, as it is positive its Cholesky factorization to compute (76). In the case of SV re-
definite. gression, however we have (with α := (α1 , . . . , α , α1∗ , . . . , α∗ ))
Next the values in obtained by such an iteration step are used

to update the corresponding values in α, s, t, z, . . . . To ensure q(α) = (αi − αi∗ )(α j − α ∗j )k(xi , x j )
that the variables meet the positivity constraints, the steplength i, j=1
ξ is chosen such that the variables move at most 1 − ε of their

initial distance to the boundaries of the positive orthant. Usually + 2C T (αi ) + T (αi∗ ) (81)
(Vanderbei 1994) one sets ε = 0.05. i=1
Another heuristic is used for computing µ, the parameter de- and therefore
termining how much the KKT-conditions should be enforced. d
Obviously it is our aim to reduce µ as fast as possible, however ∂αi q(α) = T (αi )
dαi
if we happen to choose it too small, the condition of the equa-
tions will worsen drastically. A setting that has proven to work d2
∂α2i α j q(α) = k(xi , x j ) + δi j T (αi ) (82)
robustly is dαi2
2 ∂α2i α∗j q(α) = −k(xi , x j )
g, z + s, t ξ −1
µ= . (77)
2n ξ + 10
and ∂α2∗ α∗ q(α), ∂α2∗ α j q(α) analogously. Hence we are dealing with
i j i
a matrix of type M := [ −K K +D −K ] where D, D are diagonal
The rationale behind (77) is to use the average of the satisfac- K +D
tion of the KKT conditions (74) as point of reference and then matrices. By applying an orthogonal transformation M can be
decrease µ rapidly if we are far enough away from the bound- inverted essentially by inverting an × matrix instead of a
aries of the positive orthant, to which all variables (except y) are 2 × 2 system. This is exactly the additional advantage one
constrained to. can gain from implementing the optimization algorithm directly
Finally one has to come up with good initial values. Analo- instead of using a general purpose optimizer. One can show that
gously to Vanderbei (1994) we choose a regularized version of for practical implementations (Smola, Schölkopf and Müller
(76) in order to determine the initial conditions. One solves 1998b) one can solve optimization problems using nearly ar-
bitrary convex cost functions as efficiently as the special case of

− 12 ∂α2 q(α) + 1 A α c ε-insensitive loss functions.
= (78) Finally note that due to the fact that we are solving the pri-
A 1 y b
mal and dual optimization problem simultaneously we are also
computing parameters corresponding to the initial SV optimiza- at sample xi , i.e.
tion problem. This observation is useful as it allows us to obtain

the constant term b directly, namely by setting b = y. (see Smola
m
(1998) for details). ϕi := yi − f (xi ) = yi − k(xi , x j )(αi − αi∗ ) +b . (84)
j=1
Appendix B: Solving the subset selection Rewriting the feasibility conditions (52) in terms of α yields
problem
2∂αi T (αi ) + ε − ϕi + si − z i = 0
B.1. Subset optimization problem (85)
2∂αi∗ T (αi∗ ) + ε + ϕi + si∗ − z i∗ = 0
We will adapt the exposition of Joachims (1999) to the case of
regression with convex cost functions. Without loss of general-
ity we will assume ε = 0 and α ∈ [0, C] (the other situations for all i ∈ {1, . . . , m} with z i , z i∗ , si , si∗ ≥ 0. A set of dual
can be treated as a special case). First we will extract a reduced feasible variables z, s is given by
optimization problem for the working set when all other vari-
ables are kept fixed. Denote Sw ⊂ {1, . . . , } the working set z i = max 2∂αi T (αi ) + ε − ϕi , 0
and S f := {1, . . . , }\Sw the fixed set. Writing (43) as an opti-
si = − min 2∂αi T (αi ) + ε − ϕi , 0
mization problem only in terms of Sw yields
(86)
 1 z i∗ = max 2∂αi∗ T (αi∗ ) + ε + ϕi , 0

 − (αi − αi∗ )(α j − α ∗j )xi , x j

 si∗ = − min 2∂αi∗ T (αi∗ ) + ε + ϕi , 0

 2



i, j∈Sw

∗ ∗
maximize + (αi − αi ) yi − (α j − α j )xi , x j Consequently the KKT conditions (53) can be translated into




i∈S w j∈S

f

 (−ε(αi + αi∗ ) + C(T (αi ) + T (αi∗ )))
 + αi z i = 0 and (C − αi )si = 0
i∈Sw
 (87)
 (αi − αi∗ ) = − (αi − αi∗ ) αi∗ z i∗ =0 and (C − αi∗ )si∗ = 0
subject to i∈Sw i∈S f

αi ∈ [0, C] All variables αi , αi∗ violating some of the conditions of (87) may
(83) be selected for further optimization. In most cases, especially in
the initial stage of the optimization algorithm, this set of pat-
Hence we only haveto update the linear terns is much larger than any practical size of Sw . Unfortunately
term by the coupling Osuna, Freund and Girosi (1997) contains little information on
with the fixed set − i∈Sw (αi − αi∗ ) j∈S f (α j − α ∗j )xi , x j and
how to select Sw . The heuristics presented here are an adaptation
the equality constraint by − i∈S f (αi − αi∗ ). It is easy to see
of Joachims (1999) to regression. See also Lin (2001) for details
that maximizing (83) also decreases (43) by exactly the same
on optimization for SVR.
amount. If we choose variables for which the KKT–conditions
are not satisfied the overall objective function tends to decrease
whilst still keeping all variables feasible. Finally it is bounded
from below. B.3. Selection rules
Even though this does not prove convergence (contrary to Similarly to a merit function approach (El-Bakry et al. 1996) the
statement in Osuna, Freund and Girosi (1997)) this algorithm idea is to select those variables that violate (85) and (87) most,
proves very useful in practice. It is one of the few methods (be- thus contribute most to the feasibility gap. Hence one defines a
sides (Kaufman 1999, Platt 1999)) that can deal with problems score variable ζi by
whose quadratic part does not completely fit into memory. Still
in practice one has to take special precautions to avoid stalling
of convergence (recent results of Chang, Hsu and Lin (1999) ζi := gi z i + si ti
indicate that under certain conditions a proof of convergence is = αi z i + αi∗ z i∗ + (C − αi )si + (C − αi∗ )si∗ (88)
possible). The crucial part is the one of Sw .

By construction, i ζi is the size of the feasibility gap (cf. (56)
for the case of ε-insensitive loss). By decreasing this gap, one
B.2. A note on optimality
approaches the the solution (upper bounded by the primal objec-
For convenience the KKT conditions are repeated in a slightly tive and lower bounded by the dual objective function). Hence,
modified form. Denote ϕi the error made by the current estimate the selection rule is to choose those patterns for which ζi is
largest. Some algorithms use C.2. Analytic solution for regression
ζi := αi (z i ) + αi∗ (z i∗ ) Next one has to solve the optimization problem analytically. We
make use of (84) and substitute the values of φi into the reduced
+ (C − αi )(si ) + (C − αi∗ )(si ) optimization problem (83). In particular we use
(89)
or ζi := (αi )z i + (αi∗ )z i∗
yi − (αi − αi∗ )K i j = ϕi + b + αiold − αi∗ old K i j .
+ (C − αi )si + (C − αi∗ )si . j∈ Sw j∈Sw
One can see that ζi = 0, ζi = 0, and ζi = 0 mutually imply each (91)

other. However, only ζi gives a measure for the contribution of
the variable i to the size of the feasibility gap. Moreover with the auxiliary variables γ = αi −αi∗ +α j −α ∗j and
Finally, note that heuristics like assigning sticky–flags (cf. η := (K ii + K j j − 2K i j ) one obtains the following constrained
Burges 1998) to variables at the boundaries, thus effec- optimization problem in i (after eliminating j, ignoring terms
tively solving smaller subproblems, or completely removing independent of α j , α ∗j and noting that this only holds for αi αi∗ =
the corresponding patterns from the training set while ac- α j α ∗j = 0):
counting for their couplings (Joachims 1999) can signifi- 1
maximize − (αi − αi∗ )2 η − ε(αi + αi∗ )(1 − s)
cantly decrease the size of the problem one has to solve and 2
thus result in a noticeable speedup. Also caching (Joachims
+ (αi − αi∗ ) φi − φ j + η αiold − αi∗ old (92)
1999, Kowalczyk 2000) of already computed entries of the
(∗)
dot product matrix may have a significant impact on the subject to αi ∈ [L (∗) , H (∗) ].
performance.
The unconstrained maximum of (92) with respect to αi or αi∗
can be found below.
Appendix C: Solving the SMO equations
C.1. Pattern dependent regularization (I) αi , α j αiold + η−1 (ϕi − ϕ j )
(II) αi , α ∗j αiold + η−1 (ϕi − ϕ j − 2ε)
Consider the constrained optimization problem (83) for two in- (III) αi∗ , α j αi∗ old − η−1 (ϕi − ϕ j + 2ε)
dices, say (i, j). Pattern dependent regularization means that Ci (IV) αi∗ , α ∗j αi∗ old − η−1 (ϕi − ϕ j )
may be different for every pattern (possibly even different for
αi and αi∗ ). Since at most two variables may become nonzero
at the same time and moreover we are dealing with a con- The problem is that we do not know beforehand which of the
strained optimization problem we may express everything in four quadrants (I)–(IV) contains the solution. However, by con-
terms of just one variable. From the summation constraint we sidering the sign of γ we can distinguish two cases: for γ > 0
obtain only (I)–(III) are possible, for γ < 0 the coefficients satisfy one
of the cases (II)–(IV). In case of γ = 0 only (II) and (III) have
to be considered. See also the diagram below.
(αi − αi∗ ) + (α j − α ∗j ) = αiold − αi∗ old + α old ∗ old
j − αj := γ
(90)
(∗) (∗) (∗)

for regression. Exploiting α j ∈ [0, C j ] yields αi ∈ [L , H ].
This is taking account of the fact that there may be only four
different pairs of nonzero variables: (αi , α j ), (αi∗ , α j ), (αi , α ∗j ),
and (αi∗ , α ∗j ). For convenience define an auxiliary variables s
such that s = 1 in the first and the last case and s = −1 other-
wise.
For γ > 0 it is best to start with quadrant (I), test whether the
αj α ∗j unconstrained solution hits one of the boundaries L , H and if so,
probe the corresponding adjacent quadrant (II) or (III). γ < 0
αi L max(0, γ − C j ) max(0, γ )
can be dealt with analogously.
H min(Ci , γ ) min(Ci , C ∗j + γ )
αi∗ L max(0, −γ ) max(0, −γ − C ∗j )
Due to numerical instabilities, it may happen that η < 0. In
H min(Ci∗ , −γ + C j ) min(Ci∗ , −γ ) that case η should be set to 0 and one has to solve (92) in a linear
fashion directly.11
C.3. Selection rule for regression Acknowledgments
Finally, one has to pick indices (i, j) such that the objective
This work has been supported in part by a grant of the DFG
function is maximized. Again, the reasoning of SMO (Platt 1999,
(Ja 379/71, Sm 62/1). The authors thank Peter Bartlett, Chris
Section 12.2.2) for classification will be mimicked. This means
Burges, Stefan Harmeling, Olvi Mangasarian, Klaus-Robert
that a two loop approach is chosen to maximize the objective
Müller, Vladimir Vapnik, Jason Weston, Robert Williamson, and
function. The outer loop iterates over all patterns violating the
Andreas Ziehe for helpful discussions and comments.
KKT conditions, first only over those with Lagrange multipliers
neither on the upper nor lower boundary, and once all of them
are satisfied, over all patterns violating the KKT conditions, to Notes
ensure self consistency on the complete dataset.12 This solves
the problem of choosing i. 1. Our use of the term ‘regression’ is somewhat lose in that it also includes
cases of function estimation where one minimizes errors other than the mean
Now for j: To make a large step towards the minimum, one square loss. This is done mainly for historical reasons (Vapnik, Golowich
looks for large steps in αi . As it is computationally expensive to and Smola 1997).
compute η for all possible pairs (i, j) one chooses the heuristic to 2. A similar approach, however using linear instead of quadratic programming,
maximize the absolute value of the numerator in the expressions was taken at the same time in the USA, mainly by Mangasarian (1965, 1968,
1969).
for αi and αi∗ , i.e. |ϕi − ϕ j | and |ϕi − ϕ j ± 2ε|. The index j 3. See Smola (1998) for an overview over other ways of specifying flatness of
corresponding to the maximum absolute value is chosen for this such functions.
purpose. 4. This is true as long as the dimensionality of w is much higher than the
number of observations. If this is not the case, specialized methods can
If this heuristic happens to fail, in other words if little progress offer considerable computational savings (Lee and Mangasarian 2001).
is made by this choice, all other indices j are looked at (this is 5. The table displays CT(α) instead of T (α) since the former can be plugged
what is called “second choice hierarcy” in Platt (1999) in the directly into the corresponding optimization equations.
following way: 6. The high price tag usually is the major deterrent for not using them. Moreover
one has to bear in mind that in SV regression, one may speed up the solution
considerably by exploiting the fact that the quadratic form has a special
1. All indices j corresponding to non–bound examples are structure or that there may exist rank degeneracies in the kernel matrix
looked at, searching for an example to make progress on. itself.
7. For large and noisy problems (e.g. 100.000 patterns and more with a substan-
2. In the case that the first heuristic was unsuccessful, all tial fraction of nonbound Lagrange multipliers) it is impossible to solve the
other samples are analyzed until an example is found where problem exactly: due to the size one has to use subset selection algorithms,
progress can be made. hence joint optimization over the training set is impossible. However, unlike
in Neural Networks, we can determine the closeness to the optimum. Note
3. If both previous steps fail proceed to the next i. that this reasoning only holds for convex cost functions.
8. A similar technique was employed by Bradley and Mangasarian (1998) in
For a more detailed discussion (see Platt 1999). Unlike interior the context of linear programming in order to deal with large datasets.
9. Due to length constraints we will not deal with the connection between
point algorithms SMO does not automatically provide a value Gaussian Processes and SVMs. See Williams (1998) for an excellent
for b. However this can be chosen like in Section 1.4 by having overview.
(∗)
a close look at the Lagrange multipliers αi obtained. 10. Strictly speaking, in Bayesian estimation one is not so much concerned about
the maximizer fˆ of p( f | X ) but rather about the posterior distribution of
f.
11. Negative values of η are theoretically impossible since k satisfies Mercer’s
C.4. Stopping criteria condition: 0 ≤ (xi ) − (x j )2 = K ii + K j j − 2K i j = η.
12. It is sometimes useful, especially when dealing with noisy data, to iterate
By essentially minimizing a constrained primal optimization over the complete KKT violating dataset already before complete self con-
problem one cannot ensure that the dual objective function in- sistency on the subset has been achieved. Otherwise much computational
resources are spent on making subsets self consistent that are not globally
creases with every iteration step.13 Nevertheless one knows that self consistent. This is the reason why in the pseudo code a global loop
the minimum value of the objective function lies in the interval is initiated already when only less than 10% of the non bound variables
[dual objectivei , primal objectivei ] for all steps i, hence also in changed.
the interval [(max j≤i dual objective j ), primal objectivei ]. One 13. It is still an open question how a subset selection optimization algorithm
could be devised that decreases both primal and dual objective function
uses the latter to determine the quality of the current solution. at the same time. The problem is that this usually involves a number of
The calculation of the primal objective function from the pre- dual variables of the order of the sample size, which makes this attempt
diction errors is straightforward. One uses unpractical.

(αi − αi∗ )(α j − α ∗j )ki j = − (αi − αi∗ )(ϕi + yi − b), References
i, j i
Aizerman M.A., Braverman É.M., and Rozonoér L.I. 1964. Theoretical
(93) foundations of the potential function method in pattern recognition
learning. Automation and Remote Control 25: 821–837.
i.e. the definition of ϕi to avoid the matrix–vector multiplication Aronszajn N. 1950. Theory of reproducing kernels. Transactions of the
with the dot product matrix. American Mathematical Society 68: 337–404.
Bazaraa M.S., Sherali H.D., and Shetty C.M. 1993. Nonlinear Program- Burges C.J.C. and Schölkopf B. 1997. Improving the accuracy and speed
ming: Theory and Algorithms, 2nd edition, Wiley. of support vector learning machines. In Mozer M.C., Jordan M.I.,
Bellman R.E. 1961. Adaptive Control Processes. Princeton University and Petsche T., (Eds.), Advances in Neural Information Processing
Press, Princeton, NJ. Systems 9, MIT Press, Cambridge, MA, pp. 375–381.
Bennett K. 1999. Combining support vector and mathematical program- Chalimourda A., Schölkopf B., and Smola A.J. 2004. Experimentally
ming methods for induction. In: Schölkopf B., Burges C.J.C., and optimal ν in support vector regression for different noise models
Smola A.J., (Eds.), Advances in Kernel Methods—SV Learning, and parameter settings. Neural Networks 17(1): 127–141.
MIT Press, Cambridge, MA, pp. 307–326. Chang C.-C., Hsu C.-W., and Lin C.-J. 1999. The analysis of decom-
Bennett K.P. and Mangasarian O.L. 1992. Robust linear program- position methods for support vector machines. In Proceeding of
ming discrimination of two linearly inseparable sets. Optimization IJCAI99, SVM Workshop.
Methods and Software 1: 23–34. Chang C.C. and Lin C.J. 2001. Training ν-support vector classi-
Berg C., Christensen J.P.R., and Ressel P. 1984. Harmonic Analysis on fiers: Theory and algorithms. Neural Computation 13(9): 2119–
Semigroups. Springer, New York. 2147.
Bertsekas D.P. 1995. Nonlinear Programming. Athena Scientific, Chen S., Donoho D., and Saunders M. 1999. Atomic decomposition by
Belmont, MA. basis pursuit. Siam Journal of Scientific Computing 20(1): 33–61.
Bishop C.M. 1995. Neural Networks for Pattern Recognition. Cherkassky V. and Mulier F. 1998. Learning from Data. John Wiley and
Clarendon Press, Oxford. Sons, New York.
Blanz V., Schölkopf B., Bülthoff H., Burges C., Vapnik V., and Vetter Cortes C. and Vapnik V. 1995. Support vector networks. Machine Learn-
T. 1996. Comparison of view-based object recognition algorithms ing 20: 273–297.
using realistic 3D models. In: von der Malsburg C., von Seelen Cox D. and O’Sullivan F. 1990. Asymptotic analysis of penalized like-
W., Vorbrüggen J.C., and Sendhoff B. (Eds.), Artificial Neural lihood and related estimators. Annals of Statistics 18: 1676–1695.
Networks ICANN’96, Berlin. Springer Lecture Notes in Computer CPLEX Optimization Inc. Using the CPLEX callable library. Manual,
Science, Vol. 1112, pp. 251–256. 1994.
Bochner S. 1959. Lectures on Fourier integral. Princeton Univ. Press, Cristianini N. and Shawe-Taylor J. 2000. An Introduction to Support
Princeton, New Jersey. Vector Machines. Cambridge University Press, Cambridge, UK.
Boser B.E., Guyon I.M., and Vapnik V.N. 1992. A training algorithm for Cristianini N., Campbell C., and Shawe-Taylor J. 1998. Multiplicative
optimal margin classifiers. In: Haussler D. (Ed.), Proceedings of updatings for support vector learning. NeuroCOLT Technical Re-
the Annual Conference on Computational Learning Theory. ACM port NC-TR-98-016, Royal Holloway College.
Press, Pittsburgh, PA, pp. 144–152. Dantzig G.B. 1962. Linear Programming and Extensions. Princeton
Bradley P.S., Fayyad U.M., and Mangasarian O.L. 1998. Data min- Univ. Press, Princeton, NJ.
ing: Overview and optimization opportunities. Technical Re- Devroye L., Györfi L., and Lugosi G. 1996. A Probabilistic Theory of
port 98–01, University of Wisconsin, Computer Sciences Depart- Pattern Recognition. Number 31 in Applications of mathematics.
ment, Madison, January. INFORMS Journal on Computing, to Springer, New York.
appear. Drucker H., Burges C.J.C., Kaufman L., Smola A., and Vapnik V. 1997.
Bradley P.S. and Mangasarian O.L. 1998. Feature selection via con- Support vector regression machines. In: Mozer M.C., Jordan M.I.,
cave minimization and support vector machines. In: Shavlik J. and Petsche T. (Eds.), Advances in Neural Information Processing
(Ed.), Proceedings of the International Conference on Machine Systems 9, MIT Press, Cambridge, MA, pp. 155–161.
Learning, Morgan Kaufmann Publishers, San Francisco, Califor- Efron B. 1982. The jacknife, the bootstrap, and other resampling plans.
nia, pp. 82–90. ftp://ftp.cs.wisc.edu/math-prog/tech-reports/98- SIAM, Philadelphia.
03.ps.Z. Efron B. and Tibshirani R.J. 1994. An Introduction to the Bootstrap.
Bunch J.R. and Kaufman L. 1977. Some stable methods for calculat- Chapman and Hall, New York.
ing inertia and solving symmetric linear systems. Mathematics of El-Bakry A., Tapia R., Tsuchiya R., and Zhang Y. 1996. On the formula-
Computation 31: 163–179. tion and theory of the Newton interior-point method for nonlinear
Bunch J.R. and Kaufman L. 1980. A computational method for the programming. J. Optimization Theory and Applications 89: 507–
indefinite quadratic programming problem. Linear Algebra and 541.
Its Applications, pp. 341–370, December. Fletcher R. 1989. Practical Methods of Optimization. John Wiley and
Bunch J.R., Kaufman L., and Parlett B. 1976. Decomposition of a sym- Sons, New York.
metric matrix. Numerische Mathematik 27: 95–109. Girosi F. 1998. An equivalence between sparse approximation and sup-
Burges C.J.C. 1996. Simplified support vector decision rules. In port vector machines. Neural Computation 10(6): 1455–1480.
L. Saitta (Ed.), Proceedings of the International Conference on Girosi F., Jones M., and Poggio T. 1993. Priors, stabilizers and ba-
Machine Learning, Morgan Kaufmann Publishers, San Mateo, sis functions: From regularization to radial, tensor and additive
CA, pp. 71–77. splines. A.I. Memo No. 1430, Artificial Intelligence Laboratory,
Burges C.J.C. 1998. A tutorial on support vector machines for pattern Massachusetts Institute of Technology.
recognition. Data Mining and Knowledge Discovery 2(2): 121– Guyon I., Boser B., and Vapnik V. 1993. Automatic capacity tuning of
167. very large VC-dimension classifiers. In: Hanson S.J., Cowan J.D.,
Burges C.J.C. 1999. Geometry and invariance in kernel based methods. and Giles C.L. (Eds.), Advances in Neural Information Processing
In Schölkopf B., Burges C.J.C., and Smola A.J., (Eds.), Advances Systems 5. Morgan Kaufmann Publishers, pp. 147–155.
in Kernel Methods—Support Vector Learning, MIT Press, Cam- Härdle W. 1990. Applied nonparametric regression, volume 19 of
bridge, MA, pp. 89–116. Econometric Society Monographs. Cambridge University Press.
Hastie T.J. and Tibshirani R.J. 1990. Generalized Additive Models, gramming. Princeton Technical Report SOR 90–03., Dept. of Civil
volume 43 of Monographs on Statistics and Applied Probability. Engineering and Operations Research, Princeton University.
Chapman and Hall, London. Lustig I.J., Marsten R.E., and Shanno D.F. 1992. On implement-
Haykin S. 1998. Neural Networks: A Comprehensive Foundation. 2nd ing Mehrotra’s predictor-corrector interior point method for lin-
edition. Macmillan, New York. ear programming. SIAM Journal on Optimization 2(3): 435–
Hearst M.A., Schölkopf B., Dumais S., Osuna E., and Platt J. 1998. 449.
Trends and controversies—support vector machines. IEEE Intel- MacKay D.J.C. 1991. Bayesian Methods for Adaptive Models. PhD
ligent Systems 13: 18–28. thesis, Computation and Neural Systems, California Institute of
Herbrich R. 2002. Learning Kernel Classifiers: Theory and Algorithms. Technology, Pasadena, CA.
MIT Press. Mangasarian O.L. 1965. Linear and nonlinear separation of patterns by
Huber P.J. 1972. Robust statistics: A review. Annals of Statistics linear programming. Operations Research 13: 444–452.
43: 1041. Mangasarian O.L. 1968. Multi-surface method of pattern separation.
Huber P.J. 1981. Robust Statistics. John Wiley and Sons, New York. IEEE Transactions on Information Theory IT-14: 801–807.
IBM Corporation. 1992. IBM optimization subroutine library guide Mangasarian O.L. 1969. Nonlinear Programming. McGraw-Hill, New
and reference. IBM Systems Journal, 31, SC23-0519. York.
Jaakkola T.S. and Haussler D. 1999. Probabilistic kernel regression Mattera D. and Haykin S. 1999. Support vector machines for dy-
models. In: Proceedings of the 1999 Conference on AI and Statis- namic reconstruction of a chaotic system. In: Schölkopf B., Burges
tics. C.J.C., and Smola A.J. (Eds.), Advances in Kernel Methods—
Joachims T. 1999. Making large-scale SVM learning practical. Support Vector Learning, MIT Press, Cambridge, MA, pp. 211–
In: Schölkopf B., Burges C.J.C., and Smola A.J. (Eds.), Ad- 242.
vances in Kernel Methods—Support Vector Learning, MIT Press, McCormick G.P. 1983. Nonlinear Programming: Theory, Algorithms,
Cambridge, MA, pp. 169–184. and Applications. John Wiley and Sons, New York.
Karush W. 1939. Minima of functions of several variables with inequal- Megiddo N. 1989. Progressin Mathematical Programming, chapter
ities as side constraints. Master’s thesis, Dept. of Mathematics, Pathways to the optimal set in linear programming, Springer, New
Univ. of Chicago. York, NY, pp. 131–158.
Kaufman L. 1999. Solving the quadratic programming problem arising Mehrotra S. and Sun J. 1992. On the implementation of a (primal-dual)
in support vector classification. In: Schölkopf B., Burges C.J.C., interior point method. SIAM Journal on Optimization 2(4): 575–
and Smola A.J. (Eds.), Advances in Kernel Methods—Support 601.
Vector Learning, MIT Press, Cambridge, MA, pp. 147–168 Mercer J. 1909. Functions of positive and negative type and their con-
Keerthi S.S., Shevade S.K., Bhattacharyya C., and Murthy K.R.K. 1999. nection with the theory of integral equations. Philosophical Trans-
Improvements to Platt’s SMO algorithm for SVM classifier design. actions of the Royal Society, London A 209: 415–446.
Technical Report CD-99-14, Dept. of Mechanical and Production Micchelli C.A. 1986. Algebraic aspects of interpolation. Proceedings
Engineering, Natl. Univ. Singapore, Singapore. of Symposia in Applied Mathematics 36: 81–102.
Keerthi S.S., Shevade S.K., Bhattacharyya C., and Murty K.R.K. 2001. Morozov V.A. 1984. Methods for Solving Incorrectly Posed Problems.
Improvements to platt’s SMO algorithm for SVM classifier design. Springer.
Neural Computation 13: 637–649. Müller K.-R., Smola A., Rätsch G., Schölkopf B., Kohlmorgen J., and
Kimeldorf G.S. and Wahba G. 1970. A correspondence between Vapnik V. 1997. Predicting time series with support vector ma-
Bayesian estimation on stochastic processes and smoothing by chines. In: Gerstner W., Germond A., Hasler M., and Nicoud J.-D.
splines. Annals of Mathematical Statistics 41: 495–502. (Eds.), Artificial Neural Networks ICANN’97, Berlin. Springer
Kimeldorf G.S. and Wahba G. 1971. Some results on Tchebycheffian Lecture Notes in Computer Science Vol. 1327 pp. 999–1004.
spline functions. J. Math. Anal. Applic. 33: 82–95. Murtagh B.A. and Saunders M.A. 1983. MINOS 5.1 user’s guide. Tech-
Kowalczyk A. 2000. Maximal margin perceptron. In: Smola A.J., nical Report SOL 83-20R, Stanford University, CA, USA, Revised
Bartlett P.L., Schölkopf B., and Schuurmans D. (Eds.), Advances 1987.
in Large Margin Classifiers, MIT Press, Cambridge, MA, pp. 75– Neal R. 1996. Bayesian Learning in Neural Networks. Springer.
113. Nilsson N.J. 1965. Learning machines: Foundations of Trainable Pattern
Kuhn H.W. and Tucker A.W. 1951. Nonlinear programming. In: Proc. Classifying Systems. McGraw-Hill.
2nd Berkeley Symposium on Mathematical Statistics and Proba- Nyquist. H. 1928. Certain topics in telegraph transmission theory.
bilistics, Berkeley. University of California Press, pp. 481–492. Trans. A.I.E.E., pp. 617–644.
Lee Y.J. and Mangasarian O.L. 2001. SSVM: A smooth support vector Osuna E., Freund R., and Girosi F. 1997. An improved training algo-
machine for classification. Computational optimization and Ap- rithm for support vector machines. In Principe J., Gile L., Morgan
plications 20(1): 5–22. N., and Wilson E. (Eds.), Neural Networks for Signal Processing
Li M. and Vitányi P. 1993. An introduction to Kolmogorov Complexity VII—Proceedings of the 1997 IEEE Workshop, pp. 276–285, New
and its applications. Texts and Monographs in Computer Science. York, IEEE.
Springer, New York. Osuna E. and Girosi F. 1999. Reducing the run-time complexity in
Lin C.J. 2001. On the convergence of the decomposition method for support vector regression. In: Schölkopf B., Burges C.J.C., and
support vector machines. IEEE Transactions on Neural Networks Smola A. J. (Eds.), Advances in Kernel Methods—Support Vector
12(6): 1288–1298. Learning, pp. 271–284, Cambridge, MA, MIT Press.
Lustig I.J., Marsten R.E., and Shanno D.F. 1990. On implementing Ovari Z. 2000. Kernels, eigenvalues and support vector machines. Hon-
Mehrotra’s predictor-corrector interior point method for linear pro- ours thesis, Australian National University, Canberra.
Platt J. 1999. Fast training of support vector machines using sequen- Gaussian kernels to radial basis function classifiers. IEEE Trans-
tial minimal optimization. In: Schölkopf B., Burges C.J.C., and actions on Signal Processing, 45: 2758–2765.
Smola A.J. (Eds.) Advances in Kernel Methods—Support Vector Shannon C.E. 1948. A mathematical theory of communication. Bell
Learning, pp. 185–208, Cambridge, MA, MIT Press. System Technical Journal, 27: 379–423, 623–656.
Poggio T. 1975. On optimal nonlinear associative recall. Biological Shawe-Taylor J., Bartlett P.L., Williamson R.C., and Anthony M.
Cybernetics, 19: 201–209. 1998. Structural risk minimization over data-dependent hierar-
Rasmussen C. 1996. Evaluation of Gaussian Processes and chies. IEEE Transactions on Information Theory, 44(5): 1926–
Other Methods for Non-Linear Regression. PhD thesis, 1940.
Department of Computer Science, University of Toronto, Smola A., Murata N., Schölkopf B., and Müller K.-R. 1998a. Asymp-
ftp://ftp.cs.toronto.edu/pub/carl/thesis.ps.gz. totically optimal choice of ε-loss for support vector machines.
Rissanen J. 1978. Modeling by shortest data description. Automatica, In: Niklasson L., Bodén M., and Ziemke T. (Eds.) Proceed-
14: 465–471. ings of the International Conference on Artificial Neural Net-
Saitoh S. 1988. Theory of Reproducing Kernels and its Applications. works, Perspectives in Neural Computing, pp. 105–110, Berlin,
Longman Scientific & Technical, Harlow, England. Springer.
Saunders C., Stitson M.O., Weston J., Bottou L., Schölkopf B., and Smola A., Schölkopf B., and Müller K.-R. 1998b. The connection be-
Smola A. 1998. Support vector machine—reference manual. Tech- tween regularization operators and support vector kernels. Neural
nical Report CSD-TR-98-03, Department of Computer Science, Networks, 11: 637–649.
Royal Holloway, University of London, Egham, UK. SVM avail- Smola A., Schölkopf B., and Müller K.-R. 1998c. General cost func-
able at http://svm.dcs.rhbnc.ac.uk/. tions for support vector regression. In: Downs T., Frean M.,
Schoenberg I. 1942. Positive definite functions on spheres. Duke and Gallagher M. (Eds.) Proc. of the Ninth Australian Conf. on
Math. J., 9: 96–108. Neural Networks, pp. 79–83, Brisbane, Australia. University of
Schölkopf B. 1997. Support Vector Learning. R. Oldenbourg Queensland.
Verlag, München. Doktorarbeit, TU Berlin. Download: Smola A., Schölkopf B., and Rätsch G. 1999. Linear programs for
http://www.kernel-machines.org. automatic accuracy control in regression. In: Ninth International
Schölkopf B., Burges C., and Vapnik V. 1995. Extracting support data Conference on Artificial Neural Networks, Conference Publica-
for a given task. In: Fayyad U.M. and Uthurusamy R. (Eds.), Pro- tions No. 470, pp. 575–580, London. IEE.
ceedings, First International Conference on Knowledge Discovery Smola. A.J. 1996. Regression estimation with support vector learning
& Data Mining, Menlo Park, AAAI Press. machines. Diplomarbeit, Technische Universität München.
Schölkopf B., Burges C., and Vapnik V. 1996. Incorporating invariances Smola A.J. 1998. Learning with Kernels. PhD thesis, Technische Uni-
in support vector learning machines. In: von der Malsburg C., von versität Berlin. GMD Research Series No. 25.
Seelen W., Vorbrüggen J. C., and Sendhoff B. (Eds.), Artificial Smola A.J., Elisseeff A., Schölkopf B., and Williamson R.C. 2000.
Neural Networks ICANN’96, pp. 47–52, Berlin, Springer Lecture Entropy numbers for convex combinations and MLPs. In Smola
Notes in Computer Science, Vol. 1112. A.J., Bartlett P.L., Schölkopf B., and Schuurmans D. (Eds.) Ad-
Schölkopf B., Burges C.J.C., and Smola A.J. 1999a. (Eds.) Ad- vances in Large Margin Classifiers, MIT Press, Cambridge, MA,
vances in Kernel Methods—Support Vector Learning. MIT Press, pp. 369–387.
Cambridge, MA. Smola A.J., Óvári Z.L., and Williamson R.C. 2001. Regularization with
Schölkopf B., Herbrich R., Smola A.J., and Williamson R.C. 2001. A dot-product kernels. In: Leen T.K., Dietterich T.G., and Tresp V.
generalized representer theorem. Technical Report 2000-81, Neu- (Eds.) Advances in Neural Information Processing Systems 13,
roCOLT, 2000. To appear in Proceedings of the Annual Conference MIT Press, pp. 308–314.
on Learning Theory, Springer (2001). Smola A.J. and Schölkopf B. 1998a. On a kernel-based method for
Schölkopf B., Mika S., Burges C., Knirsch P., Müller K.-R., Rätsch G., pattern recognition, regression, approximation and operator in-
and Smola A. 1999b. Input space vs. feature space in kernel-based version. Algorithmica, 22: 211–231.
methods. IEEE Transactions on Neural Networks, 10(5): 1000– Smola A.J. and Schölkopf B. 1998b. A tutorial on support vector re-
1017. gression. NeuroCOLT Technical Report NC-TR-98-030, Royal
Schölkopf B., Platt J., Shawe-Taylor J., Smola A.J. , and Williamson R.C. Holloway College, University of London, UK.
2001. Estimating the support of a high-dimensional distribution. Smola A.J. and Schölkopf B. 2000. Sparse greedy matrix approximation
Neural Computation, 13(7): 1443–1471. for machine learning. In: Langley P. (Ed.), Proceedings of the In-
Schölkopf B., Simard P., Smola A., and Vapnik V. 1998a. Prior knowl- ternational Conference on Machine Learning, Morgan Kaufmann
edge in support vector kernels. In: Jordan M.I., Kearns M.J., and Publishers, San Francisco, pp. 911–918.
Solla S.A. (Eds.) Advances in Neural Information Processing Sys- Stitson M., Gammerman A., Vapnik V., Vovk V., Watkins C., and
tems 10, MIT Press. Cambridge, MA, pp. 640–646. Weston J. 1999. Support vector regression with ANOVA decom-
Schölkopf B., Smola A., and Müller K.-R. 1998b. Nonlinear compo- position kernels. In: Schölkopf B., Burges C.J.C., and Smola A.J.
nent analysis as a kernel eigenvalue problem. Neural Computation, (Eds.), Advances in Kernel Methods—Support Vector Learning,
10: 1299–1319. MIT Press Cambridge, MA, pp. 285–292.
Schölkopf B., Smola A., Williamson R.C., and Bartlett P.L. 2000. New Stone C.J. 1985. Additive regression and other nonparametric models.
support vector algorithms. Neural Computation, 12: 1207–1245. Annals of Statistics, 13: 689–705.
Schölkopf B. and Smola A.J. 2002. Learning with Kernels. MIT Press. Stone M. 1974. Cross-validatory choice and assessment of statistical
Schölkopf B., Sung K., Burges C., Girosi F., Niyogi P., Poggio T., predictors (with discussion). Journal of the Royal Statistical Soci-
and Vapnik V. 1997. Comparing support vector machines with ety, B36: 111–147.
Street W.N. and Mangasarian O.L. 1995. Improved generalization via Vapnik V.N. 1982. Estimation of Dependences Based on Empirical
tolerant training. Technical Report MP-TR-95-11, University of Data. Springer, Berlin.
Wisconsin, Madison. Vapnik V.N. and Chervonenkis A.Y. 1971. On the uniform convergence
Tikhonov A.N. and Arsenin V.Y. 1977. Solution of Ill-posed problems. of relative frequencies of events to their probabilities. Theory of
V. H. Winston and Sons. Probability and its Applications, 16(2): 264–281.
Tipping M.E. 2000. The relevance vector machine. In: Solla S.A., Leen Wahba G. 1980. Spline bases, regularization, and generalized
T.K., and Müller K.-R. (Eds.), Advances in Neural Information cross-validation for solving approximation problems with large
Processing Systems 12, MIT Press, Cambridge, MA, pp. 652–658. quantities of noisy data. In: Ward J. and Cheney E. (Eds.), Proceed-
Vanderbei R.J. 1994. LOQO: An interior point code for quadratic pro- ings of the International Conference on Approximation theory in
gramming. TR SOR-94-15, Statistics and Operations Research, honour of George Lorenz, Academic Press, Austin, TX, pp. 8–10.
Princeton Univ., NJ. Wahba G. 1990. Spline Models for Observational Data, volume 59 of
Vanderbei R.J. 1997. LOQO user’s manual—version 3.10. Technical CBMS-NSF Regional Conference Series in Applied Mathematics.
Report SOR-97-08, Princeton University, Statistics and Oper- SIAM, Philadelphia.
ations Research, Code available at http://www.princeton.edu/ Wahba G. 1999. Support vector machines, reproducing kernel Hilbert
˜rvdb/. spaces and the randomized GACV. In: Schölkopf B., Burges
Vapnik V. 1995. The Nature of Statistical Learning Theory. Springer, C.J.C., and Smola A.J. (Eds.), Advances in Kernel Methods—
New York. Support Vector Learning, MIT Press, Cambridge, MA. pp. 69–88.
Vapnik V. 1998. Statistical Learning Theory. John Wiley and Sons, Weston J., Gammerman A., Stitson M., Vapnik V., Vovk V., and Watkins
New York. C. 1999. Support vector density estimation. In: Schölkopf B.,
Vapnik. V. 1999. Three remarks on the support vector method of Burges C.J.C., and Smola A.J. (Eds.) Advances in Kernel
function estimation. In: Schölkopf B., Burges C.J.C., and Smola Methods—Support Vector Learning, MIT Press, Cambridge,
A.J. (Eds.), Advances in Kernel Methods—Support Vector MA. pp. 293–306.
Learning, MIT Press, Cambridge, MA, pp. 25–42. Williams C.K.I. 1998. Prediction with Gaussian processes: From linear
Vapnik V. and Chervonenkis A. 1964. A note on one class of regression to linear prediction and beyond. In: Jordan M.I. (Ed.),
perceptrons. Automation and Remote Control, 25. Learning and Inference in Graphical Models, Kluwer Academic,
Vapnik V. and Chervonenkis A. 1974. Theory of Pattern Recognition pp. 599–621.
[in Russian]. Nauka, Moscow. (German Translation: Wapnik Williamson R.C., Smola A.J., and Schölkopf B. 1998. Generalization
W. & Tscherwonenkis A., Theorie der Zeichenerkennung, performance of regularization networks and support vector
Akademie-Verlag, Berlin, 1979). machines via entropy numbers of compact operators. Technical
Vapnik V., Golowich S., and Smola A. 1997. Support vector method Report 19, NeuroCOLT, http://www.neurocolt.com. Published
for function approximation, regression estimation, and signal in IEEE Transactions on Information Theory, 47(6): 2516–2532
processing. In: Mozer M.C., Jordan M.I., and Petsche T. (Eds.) (2001).
Advances in Neural Information Processing Systems 9, MA, MIT Yuille A. and Grzywacz N. 1988. The motion coherence theory.
Press, Cambridge. pp. 281–287. In: Proceedings of the International Conference on Computer
Vapnik V. and Lerner A. 1963. Pattern recognition using generalized Vision, IEEE Computer Society Press, Washington, DC, pp. 344–
portrait method. Automation and Remote Control, 24: 774–780. 354.

A Tutorial On Support Vector Regression

Uploaded by

Copyright:

Available Formats

A Tutorial On Support Vector Regression

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

A Tutorial On Support Vector Regression

Uploaded by

Copyright:

Available Formats

Statistics and Computing 14: 199–222, 2004

C 2004 Kluwer Academic Publishers. Manufactured in The Netherlands.

A tutorial on support vector regression∗

Received July 2002 and accepted November 2003

Keywords: machine learning, support vector machines, regression estimation

1. Introduction (Vapnik and Lerner 1963, Vapnik and Chervonenkis 1964). As

1.2. The basic idea

αi (ε + ξi − yi + w, xi  + b) = 0 that the subscripts in this case refer to the components of x ∈ R2 .

and While this approach seems reasonable in the particular ex-

Theorem 8 (Schoenberg 1942). A kernel of dot-product type

3.1. The risk functional

Table 1. Common loss functions and corresponding density models

Loss function Density model

ε-insensitive c(ξ ) = |ξ |ε p(ξ ) = 2(1+ε)

The input pattern (for which a prediction is to be made) is

Before proceeding to further algorithms for quadratic optimiza-

5.6. Sequential minimal optimization

7.1. Regularization networks

ξ is chosen such that the variables move at most 1 − ε of their

One can see that ζi = 0, ζi = 0, and ζi = 0 mutually imply each (91)

(∗) (∗) (∗)

Pai-Hsuen Chen1 , Chih-Jen Lin1 , and Bernhard Schölkopf2?

Suppose we are given empirical data

(x1 , y1 ), . . . , (xm , ym ) ∈ X × {±1}. (1)

Here, (x)i denotes the ith entry of x.

1. It lets us define a similarity measure from the dot product in H,

k(x, x0 ) := (x · x0 ) = (Φ(x) · Φ(x0 )). (5)

Here, we have defined the offset

It will be proved instructive to rewrite this expression in terms of the patterns xi in

A way to solve (23) is through its Lagrangian dual:

max min L(w, b, α) ≥ γ. (28)

On the other hand, for any α,

min L(w, b, α) ≤ L(w̄, b̄, α),

ᾱi · [yi ((xi · w̄) + b̄) − 1] = 0, i = 1, . . . , m, (30)

which is usually called the complementarity condition.

4 Optimal Margin Support Vector Classifiers

and the following quadratic program (cf. (35)):

5.1 Product Features

[x]j1 · · · · · [x]jd , (43)

where j1 , . . . , jd ∈ {1, . . . , N }. In that case, we might prefer to extract these product

different monomials (43), comprising a feature space H of dimensionality NH . For

5.2 Polynomial Feature Spaces Induced by Kernels

Φ2 : ([x]1 , [x]2 ) 7→ ([x]21 , [x]22 , [x]1 [x]2 , [x]2 [x]1 ), (48)

dot products in H take the form

An illustration is in Figure 3. For an overview of other kernels, see [31].

6 ν-Soft Margin Support Vector Classifiers

in order to relax the constraints to

A classifier which generalizes well is then P

ν 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

using multipliers αi , βi , δ ≥ 0. This function has to be minimized with respect to the

i.e., if one does use C [31].

Rp := inf R[f ], (74)

P (T = {(x1 , y1 ), . . . , (xm , ym )} | R[fTν ] ≤ ν − Rp + ²) ≥ 1 − e−cm . (75)

7 Implementation of ν-SV Classifiers

Algorithm 1 (Decomposition method)

7.2 Working Set Selection

yi ∇W (α)i > yj ∇W (α)j (86)

yi1 ∇W (α)i1 − yj1 ∇W (α)j1 ≥ · · · ≥ yiq/2 ∇W (α)iq/2 − yjq/2 ∇W (α)jq/2 . (87)

αi (ε + ξi − yi + w, xi + b) = 0 that the subscripts in this case refer to the components of x ∈ R2 .

αi (ε + ξi − yi + w, xi + b) = 0 that the subscripts in this case refer to the components of x ∈ R2 .