A simplex method for function minimization

By J. A. Nelder and R. Meadf

A method is described for the minimization of a function of n variables, which depends on the
comparison of function values at the (n 4- 1) vertices of a general simplex, followed by the
replacement of the vertex with the highest value by another point. The simplex adapts itself to
the local landscape, and contracts on to thefinalminimum. The method is shown to be effective
and computationally compact. A procedure is given for the estimation of the Hessian matrix in
the neighbourhood of the minimum, needed in statistical estimation problems.

Spendley et al. (1962) introduced an ingenious idea for of p from Ph with [P*p] = oc[PhP]. If y* lies between
tracking optimum operating conditions by evaluating yh and yh then Ph is replaced by P* and we start again
the output from a system at a set of points forming a with the new simplex.
simplex in the factor-space, and continually forming If y* <yh i.e. if reflection has produced a new
new simplices by reflecting one point in the hyperplane

Downloaded from http://comjnl.oxfordjournals.org/ by guest on June 26, 2013

minimum, then we expand P* to P** by the relation
of the remaining points. This idea is clearly applicable
to the problem of minimizing a mathematical function P** = yP* + (1 - y)P.
of several variables, as was recognized by these authors. The expansion coefficient y, which is greater than unity,
However, they assumed that the relative steps to be made is the ratio of the distance [P**P] to [P*P]. If
in varying the factors were known, and this makes their y** < yt we replace Ph by P** and restart the process;
strategy rather rigid for general use. In the method to but if y** > y, then we have a failed expansion, and we
be described the simplex adapts itself to the local land- replace Ph by P* before restarting.
scape, elongating down long inclined planes, changing
direction on encountering a valley at an angle, and If on reflecting P to P* we find that y* > y, for all
contracting in the neighbourhood of a minimum. The / ^= h, i.e. that replacing P by P* leaves y* the maximum,
criterion for stopping the process has been chosen with then we define a new Ph to be either the old Ph or P*,
an eye to its use for statistical problems involving the whichever has the lower y value, and form
maximization of a likelihood function, in which the P** = pPh + (1 - ft?.
unknown parameters enter non-linearly.
The contraction coefficient j8 lies _between_0 and 1 and
is the ratio of the distance [P**P] to [PP]. We then
The method accept P** for Ph and restart, unless y** > min (yh, y*),
We consider, initially, the minimization of a function i.e. the contracted point is worse than the better of Ph
of n variables, without constraints. Po, Pu . . . Pn are and P*. For such a failed contraction we replace all
the (n + 1) points in n-dimensional space defining the the P,'s by (P, + P,)/2 and restart the process.
current "simplex." [The simplex will not, of course, be A failed expansion may be thought of as resulting
regular in general.] We write yt for the function value from a lucky foray into a valley (P*) but at an angle to
at Ph and define the valley so that P** is well up on the opposite slope.
A failed contraction is much rarer, but can occur when
h as the suffix such that yh = max (.>>,) [h for "high"] a valley is curved and one point of the simplex is much
and farther from the valley bottom than the others; con-
/ as the suffix such that yl = min (j>,) [/ for "low"]. traction may then cause the reflected point to move
away from the valley bottom instead of towards it.
Further we define p as the centroid of the points with Further contractions are then useless. The action pro-
i # h, and write [P/Pj] for the distance from P, to Pj. posed contracts the simplex towards the lowest point,
At each stage in the process Ph is replaced by a new and will eventually bring all points into the valley. The
point; three operations are usedreflection, contraction, coefficients a, jS, y give the factor by which the volume
and expansion. These are defined as follows: the of the simplex is changed by the operations of reflection,
reflection of Ph is denoted by P*, and its co-ordinates contraction or expansion respectively. The complete
are defined by the relation method is given as a flow diagram in Fig. 1.
P* = (1 + *)P - ocPh A final point concerns the criterion used for halting
the procedure. The criterion adopted is somewhat
where a is a positive constant, the reflection coefficient. different from that used by Powell (1964) in that it is
Thus P* is on the line joining Ph and P, on the far side concerned with the variation in the y values over the
t National Vegetable Research Station, Wellesbourne, Warwick.
308
 Function minimization
INTER
Calculate i n i t i a l p^ ami y.
i Determine h, calculate P
Form P = (i+a)P - a P h
Calculate y
is
i s y* < y 1 ? ' -No- is y* > y., i ^ h ?

Yes
I Replace P h by P*
No
Form P** = (i+yJP* -TP Form P** = Ph+(A-)P
Calculate y " Calculate y*
is
K y " < yi ? No- -Yes

Yes No Replace a l l

Replace P h by P**

Downloaded from http://comjnl.oxfordjournals.org/ by guest on June 26, 2013

Replace P. by P Replace Ph by P*

Has minimum been

reached ? -)EXIT

-No
Fig. 1.Flow diagram

simplex rather than with changes in the x's. The form be made to it. Clearly either technique can deal with
chosen is to compare the "standard error" of the y's individual limitations on the range of any number of
in the form */{Y,(y, ~y)2/n} with a pre-set value, and x's. Constraints involving more than one x can be
to stop when it falls below this value. The success of included using the second technique provided that an
the criterion depends on the simplex not becoming too initial simplex can be found inside the permitted region,
small in relation to the curvature of the surface until the from which to start the process. Linear constraints that
final minimum is reached. The reasoning behind the reduce the dimensionality of the field of search can be
criterion is that in statistical problems where one is included by choosing the initial simplex to satisfy
concerned with finding the minimum of a negative the constraints and reducing the dimensions accord-
likelihood surface (or of a sum-of-squares surface) the ingly. Thus to minimize y f(xu x2, x3) subject to
curvature near the minimum gives the information x, + x2 + x3 = X, we could choose an initial simplex
available on the unknown parameters. If the curvature with vertices (X, 0, 0), (0, X, 0), and (0, 0, X), treating
is slight the sampling variance of the estimates will be the search as being in two dimensions. In particular,
large so there is no sense in finding the co-ordinates of any x, may be held constant by setting its value to that
the minimum very accurately, while if the curvature is constant for all vertices of the initial simplex.
marked there is justification for pinning down the
minimum more exactly.
Results
Constraints on the volume to be searched Three functions, all of which have been used before
for testing minimization procedures, were used to test
If, for example, one of the x, must be non-negative in the method. The functions, all of which have a minimum
a minimization problem, then our method may be of zero, were:
adapted in one of two ways. The scale of the x con-
cerned can be transformed, e.g., by using the logarithm, (1) Rosenbrock's parabolic valley (Rosenbrock (I960))
so that negative values are excluded, or the function can y = 100(^2 x])2 + (1 x,)2, starting point
be modified to take a large positive value for all negative (-1-2,1).
x. In the latter case any trespassing by the simplex over
the border will be followed automatically by contraction (2) Powell's quartic function (Powell (1962))
moves which will eventually keep it inside. In either y = (xi + l(hr2)2 + 5(x3 - x4)2 + (x2 - 2x3)4
case an actual minimum with x = 0 would be inaccess- + 10(x, - xA)\
ible in general, though arbitrarily close approaches could starting point (3, 1, 0, 1).
309
 Function minimization
(3) Fletcher and Powell's helical valley (Fletcher and Table 1
Powell (1963))
Mean and minimum numbers of evaluations for function 2
y = 100[x3 - lOflfr,, x 2 )] 2
+ W(A + 4) - i]2 + A MEAN NUMBER
where 2-nQ(xu x2) = arctan (x2lx{), x, > 0
STRATEGY
= 77 + arctan (*2/*i), *i < 0 STEP-LENGTH
0, h 2) 0. h 3)(1, h 4) (1, h 2) 0, i, 2)
starting point (1, 0, 0).
The stopping criterion used was s/^L^y, j) 2 /n}<10~ 8 . 0-25 225 234 282 255 344
The function value at the centroid of the final simplex 0-5 210 234 254 300 335
usually deviated from the true minimum by less than 1 216 229 260 283 343
10~ 8 ; a sample of runs gave 2-5 x 10~9 as the geo- 2 216 239 250 264 253
metric mean of this deviation. A difficulty encountered 4 226 241 251 249 347
in testing the procedure was that the size and orientation
of the initial simplex had an effect on the speed of con- Mean 219 235 259 270 322
vergence and consequently several initial step-lengths
and several arrangements of the initial simplex were

Downloaded from http://comjnl.oxfordjournals.org/ by guest on June 26, 2013

used for each trial (the arrangements consisted of two MINIMUM NUMBER
forms of the initial simplex, regular, as in Spendley et
al.'s original method, and axial from the starting point,
combined with several orientations). The first set of 0-25 191 205 241 172 189
trials investigated the different strategies (values of a, 0-5 181 181 190 190 233
p and y) of the simplex method of minimization, and 1 180 191 210 150 216
the second compared the results for the best strategy 2 180 194 201 174 117
with those of Powell (1964), which are among the best 4 184 170 217 152 217
previously obtained.
An initial trial with function (1) used all combinations Mean 183 188 212 168 194
of a = i, f, 1; ft = , $-, i; y = 2, 3, 4; and initial step-
lengths i, 1, and 2. The main result was that the lower
values of a and p gave generally slower convergence. used in comparing the method with Powell's (1964)
In a second trial with function (1) six main strategies technique.
a = 1, /S = \ or \, and y = 2, 3, 4 with two additional
strategies ($, \, 3) and (f, -J-, 4) were all tried for three Comparison with Powell's results
initial step-lengths i, 1, 2, and eight arrangements of the For this comparison we used the same convergence
initial simplex. This trial showed that the additional criterion; smaller values of the criterion would not be
strategies were more variable in performance and, on justifiable when it is remembered that most functions
the average, slower to converge than the six main met with in practice are likely to have a rounding-off
strategies. These six strategies gave very similar results, error of this order. We used all three functions with a
as was shown by an analysis of variance of the number range of initial step-lengths 0-1(0-1)1(0-2)3 and eight
of function evaluations, the mean square for strategies initial simplex arrangements. Table 2 shows the results
being 197 compared with a residual mean square (after of these trials.
removing size and orientation effects) of 463. Apart from the smaller step-lengths the effect of step-
Using function (2) five strategies were tried: a = 1, length on number of evaluations required is not very
P = i, Y = 2, 3 or 4, and a = 1, p = or , y = 3. large and it is reasonable to use the mean number of
The latter two strategies gave very variable results, as is evaluations over all the unasterisked step-lengths for
indicated in Table 1 which shows the mean and minimum comparison with other methods. The mean values
numbers of evaluations required for convergence over obtained are 144 evaluations for function (1), 216 for
eight arrangements. function (2), and 228 for function (3). Powell's results
Analyzing the results for the best three strategies gave for functions (1) and (2) give the evaluations required to
a standard error of difference between the strategy mean reach our mean final value of 2-5 x 10~9 as 150 and
numbers of 7-0 (based on 70 degrees of freedom), 235 respectively.
showing that the results for the three strategies are all An EMA version of Powell's method was written and
significantly different at the 95 % level. for function (3) it was found that the initial step-length
Using function (3) for the same trial confirmed these had a considerable effect on the speed of convergence.
conclusions, the last two strategies converging to false Furthermore the optimum initial step-lengths for the
minima on several occasions. two methods were different. For initial step-lengths in
It was clear, therefore, that the simple strategy the range 0-1 to 0 95, Powell's method gave numbers
a = 1, p = i, y 2 was the best, and these values were of function values between 177 and 375 (sample of six).
310
 Function minimization
Table 2 Discussion
The method presented in this paper differs from most
Mean number of evaluations required for convergence for previously put forward in being based neither on
different step-lengths for functions (1), (2) and (3) gradients (first-order derivatives) nor on quadratic
forms (second-order derivatives). As can be seen in
FUNCTION Fig. 1, the algebraic operations on the x, are all linear
STEP-LENGTH
(1) (2) (3) (e.g. as in determining P from Ph and P), while those
on the y-t are concerned with finding the maximum or
minimum of a finite set of quantities. This latter type
01 165* 262* 310* of operation is non-linear and accounts for the ability
0-2 161* 220 256 of the method to locate a minimum with arbitrary
0-3 159* 214 276 accuracy. The remark of Spendley et al. (1962) that
0-4 162* 215 256 "Continued application of the simplex procedure, with
0-5 143 210 230 progressively reduced step size, is inherently as self-
0-6 142 206 268 defeating as any other linear technique," is thus some-
0-7 151 210 264 what misleading, since the use made of the simplex
0-8 150 231 231 at any stage is not to estimate parameters in a
0-9 157 223 236 regression equation (in which case only a linear model

Downloaded from http://comjnl.oxfordjournals.org/ by guest on June 26, 2013

10 147 216 213 could be fitted) but to guide the direction of the next
move.
1-2 145 217 183 Out method is highly opportunist, in that the least
1-4 145 208 194 possible information is used at each stage and no
1-6 154 217 188 account is kept of past positions. No assumptions are
1-8 138 225 213 made about the surface except that it is continuous and
20 144 216 207 has a unique minimum in the area of the search. It
2-2 130| 197 206 might thus be expected that when the curvature of the
2-4 128 224 224 landscape (as measured by the Hessian matrix of second-
2-6 134 223 209 order partial derivatives) is changing rapidly, the present
2-8 135 241 233 method will do well when compared with methods which
30 148 191 234 depend on arguments applicable to quadratic forms.
Conversely in the neighbourhood of a minimum, when
* See text. the Hessian matrix is relatively stable, it may do worse.
t Omitting arrangements where the true minimum was a vertex This expectation is borne out by the results obtained
of the initial simplex. on the test functions. However, the positions of minima
are often needed with only limited accuracy, so that final
There appears, therefore, to be little difference between rapid convergence is not essential. This is especially
the two methods to this degree of accuracy as far as true in statistical problems where the surface may be,
our evidence goes. However, there are indications that, e.g., a sum of squares of residuals, and the position of
with less severe convergence criteria, greater differences the minimum is a function of the random errors in the
would have been found. This is suggested by the observations. An important property of our method is
progress of the two methods on three typical runs, as that it will converge even when the initial simplex
shown in Table 3; for all three functions our method straddles two or more valleys, a property which is not
establishes an initial advantage over Powell's. For shared by, e.g., Powell's method.
function (2) this advantage is maintained approximately A general problem encountered by all minimization
at the same level for the remainder of the run; for methods is that of false convergence at a point other
functions (1) and (3) the advantage is further increased than the minimum. This difficulty has been found in
up to the final stage when Powell's method rapidly using the simplex method on a four-dimensional surface
closes the gap. having a long, curved, valley with extremely steep sides;
along the valley bottom the function varies considerably
Effect of number of variables compared with the accuracy to which the minimum
function value is required. On meeting the valley, the
To investigate the effect of increasing the number of simplex may need to undergo a drastic change in both
variables, the sum of fourth powers was used as a test size and shape, in the course of which the variation
function, starting at (1, 1, 1, . . ., 1). For functions with between the function values at the simplex vertices may
2 to 10 variables it was found that the relationship become small even when compared with a convergence
between the number of variables k and the mean number criterion of one-tenth of the required accuracy. Merely
of evaluations N for convergence (using a final value of refining the convergence criterion would often involve
approximately 2-5 x 10~9) was well described by needless evaluations, and in other more extreme cases
N=3-I6(k \2 -11 could still be ineffective.
311
 Function minimization
Table 3
Values of the three functions on 3 typical runs, for simplex method and Powell's (1964) method

NUMBER OF FUNCTION
EVALUATION (1) (2) (3)

Simplex Powell Simplex Powell Simplex Powell

1 2-4 X 101 2-4 X IO1 2-2 X IO2 2-2 X IO2 2-5 X IO3 2-5 X IO3
20 2-6 X 10 3-6 X 10 1-2 X IO1 3-1 X IO1 1-9 X 10 1-9 X IO2
40 9-7 X IO- 1 2-2 X 10 3-0 X io- 12 4-2 X 10 1-2 X 10 3-2 X IO1
60 3-8 X IO- 1 1-1 X 10 8-7 X io- 2 1-5 X 10 7-6 X io- 22 7-7 X 10
80 8-1 X io- 32 4-6 X io- 11 1-7 X io- 3 2-3 X io- 13 3-0 X io- 2 2-0 X 10
100 1-3 X io- 6 10 X io- 2 1-2 X IO- 6-4 X io- 3 2-0 X io- 3 7-6 X io- 11
120 8-4 X io- 9 1-7 X io- 4 6-7 X io- 56 1-1 X io- 4 4-4 X io- 3-1 X io-
140 7-7 X io- 1 0 10 X io- 9 3-9 X io- 1-3 X io- 1-3 X io- 4 2-3 X io- 1
150 3-3 X io- 1-2 X io-
io- 76 io- 56 io- 67 io- 32

Downloaded from http://comjnl.oxfordjournals.org/ by guest on June 26, 2013

160 1-2 X 2-1 X 10 X 6-6 X
180 5-9 X io- 2-1 X io- 7 1-7 X io- 3-6 X io- 4
200 2-6 X io- 18 0 1-3 X io- 8 3-7 X io- 18 0 11 X io- 9
220 9-3 X io- 1-6 X io- 2-7 X io- 7-2 X io-

The values for Powell's method were obtained by logarithmic interpolation of the function values at the end of each iteration.
Data for functions (1) and (2) from Powell (1964), data for function (3) from our EMA program of his method.

Powell (1964) suggested a more complex convergence values are sufficiently close is convergence allowed.
criterion, for this general problem, based on perturbing The simplex method is computationally compact; on
the first minimum found and repeating the method to the Orion computer the basic routine (without final
find a second minimum, followed by exploration along printing) contains less than 350 instructions, and the
the line joining the two. An alternative technique, more great majority of orders are additions and subtractions
suited to our convergence criterion in terms of function or simple logical orders. There are few multiplications,
variation, is to continue after the first convergence for a and no divisions at all except on entering and leaving
prescribed number of evaluations, to test for convergence the routine.
again and, if the second test proves successful, to com- Copies of the routine, written in Extended Mercury
pare the two "converged" function values. Only if these Autocode, are available from the authors.

Appendix
The Hessian matrix at the minimum

The minimization method proposed, being independent points. If the original points of the simplex are used to
of the properties of quadratic forms, does not yield any define a set of oblique axes with co-ordinates xh then
estimate of the Hessian matrix of second derivatives at the points may be taken as
the minimum. This matrix is, of course, the information
matrix in statistical problems when the function being (0, 0, 0, . . . 0)
minimized is minus the log. likelihood, and its inverse is (1, 0, 0, . . . 0)
the sample variance-covariance matrix of the estimates. (0, 1, 0, . . . 0)
A convenient way of utilizing a quadratic surface to
estimate the minimum when the simplex is close to that and (0, 0, 0, . . . 1).
minimum was given by Spendley et al. (1962) and their
method can be readily extended to give the required If the quadratic approximation to the function in the
variance-covariance matrix of the estimates. neighbourhood of the minimum is written as
If the (n + 1) points of the simplex in n dimensions y = aQ
are given by Po, P{, . . . Pn, then Spendley et al. form
the "half-way points" Pu = (P, + Pj)l2, i =j and fit a or in vector form as
quadratic surface to the combined set of (M + 1)( + 2)/2 y = a0 + la'x + x'Bx,
312
 Function minimization
then the coefficients are estimated as so that the variance-covariance matrix is given by

a; = 2y0i - (y, + 3yo)/2, i=\,...,n If normal equal-variance independent errors are involved
ba = 2(j>,- + y0- 2y0l), i=l,...,n and the sum of squares of residuals is minimized, then
bu = 2(yu +y0- yOi - y0J), i ^ j , this matrix must be multiplied by 2a2, where as usual
where y, is the function value at P ; and ya that at P l7 . a2 would be estimated by ymin/(N n), N being the
The estimated minimum is then given by total number of observations, and n the number of
parameters fitted.
In estimating B numerically it is necessary to steer a
course between two hazards. In one the simplex is so
and the information matrix is just B. small that (yu + j 0 yOi y0J) consists largely of
If Pi denotes the co-ordinates of .P, in the original rounding-off errors incurred in calculating the y's. In
system, and if Q is the n x n matrix whose ith column the other the simplex is so large that the quadratic
is />, p0, then the minimum is estimated to be at approximation is poor, and the b's are correspondingly
biased. If the method given in this paper is used, the
Pmin = Po former hazard will usually be the important one, and
_ nrt-i'a. it may be necessary to enlarge the final simplex before
adding the extra points. A possible way of doing this

Downloaded from http://comjnl.oxfordjournals.org/ by guest on June 26, 2013

The minimum value of the function is estimated to be is to double the distance of each point /", from the
centroid until the corresponding function value exceeds
ymin = Qo a'B~la. that at the centroid by more than a given constant.
The information matrix in the original co-ordinate The choice of this would depend on the rounding-off
system is given by error attaching to the evaluation of the function, and
would need to be at least 103 times that rounding error,
(Q-'YBQ - 1 if acceptable estimates of the b's were to be obtained.
References
FLETCHER, R., and POWELL, M. J. D. (1963). "A rapidly convergent descent method for minimization," The Computer Journal
Vol. 6, p. 163.
POWELL, M. J. D. (1962). "An iterative method for finding stationary value of a function of several variables," The Computer
Journal, Vol. 5, p. 147.
POWELL, M. J. D. (1964). "An efficient method for finding the minimum of a function of several variables without calculating
derivatives," The Computer Journal, Vol. 7, p. 155.
ROSENBROCK, H. (1960). "An automatic method for finding the greatest or least value of a function," The Computer Journal,
Vol. 3, p. 175.
SPENDLEY, W., HEXT, G. R., and HIMSWORTH, F. R. (1962). "Sequential Application of Simplex Designs in Optimisation and
Evolutionary Operation," Technometrics, Vol. 4, p. 441.
Correspondence
To the Editor, Suppose T[R] is a Boolean function taking a routine
The Computer Journal. (or program) R with no formal or free variables as its
argument and that for all R, T[R] True if R terminates
An impossible program if run and that T[R] = False if R does not terminate.
Consider the routine P defined as follows
Sir,
A well-known piece of folk-lore among programmers rec routine P
holds that it is impossible to write a program which can
examine any other program and tell, in every case, if it L :if T[P]gotoL
will terminate or get into a closed loop when it is run. Return
I have never actually seen a proof of this in print, and
though Alan Turing once gave me a verbal proof (in a If T[P] = True the routine P will loop, and it will
railway carriage on the way to a Conference at the only terminate if T[P] = False. In each case T[P] has
NPL in 1953), I unfortunately and promptly forgot the exactly the wrong value, and this contradiction shows
details. This left me with an uneasy feeling that the that the function T cannot exist.
proof must be long or complicated, but in fact it is so
short and simple that it may be of interest to casual Yours faithfully,
readers. The version below uses CPL, but not in any Churchill College, C. STRACHEY.
essential way. Cambridge.
313