Chapter Two

GENERAL PRINCIPLES
IN RANDOM VARIATE GENERATION

1. INTRODUCTION.
In thls chapter we lntroduce the reader to the fundamental prlnclples In
non-unlform random varlate generatlon. Thls chapter 1s a must for the serlous
reader. On Its own I t can be used as part of a course In simulation.
i
These baslc prlnclples apply often, but not always, to both contlnuous and
dlscrete random varlables. For a structured development I t 1s perhaps best to
develop the materlal accordlng to the guldlng prlnclple rather than accordlng to
the type of random variable Involved. The reader 1s also cautloned that we do
not make any recomrnendatlons at thls palnt about generators for varlous dlstrl-
butlons. All the examples found In thls chapter are of a dldactlcal nature, and
the most lmportant famllles of dlstrlbutlons wlll be studled In chapters IX,X,XI In
more detall.

2. THE INVERSION METHOD.

2.1. The inversion principle.

The lnverslon method 1s based upon the followlng property:
28 II.2.INVERSION METHOD
~ ~ ~~

Theorem 2.1.
Let F be a contlnuous dlstrlbutlon functlon on R wlth lnverse F-' deflned
by
F - ' ( u ) = 1nf {a::F(a:)=u, O < U <I}.
If U 1s a unlform [O,l]random varlable, then F - ' ( U ) has dlstrlbutlon functlon
F . Also, If X has dlstrlbutlon functlon F , then F ( X ) 1s unlformly dlstrlbuted
on [0,1].

Proof of Theorem 2.1.

The flrst statement follows after notlng that for all a: ER ,
P ( F - ' ( U ) < s ) = P(1nf { g : F ( y ) = U } j z )
=P(U<F(s)) =F ( z ) .
I

The second statement follows from the fact that for all O<u <1,
P ( F ( X ) l u )= P ( X L F - ' ( u ) )
= F ( F - l ( u )) = u I.

Theorem 2.1 can be used to generate random varlates wlth an arbltrary con-
tlnuous dlstrlbutlon functlon F provlded that F-' 1s expllcltly known. The fas-
ter the Inverse can be computed, the faster we can compute X from a glven unl-
form [0,1] random varlate U . Formally, we have

The inversion method

random variate U .
Generate a uniform [0,1]
RETURN X +F-'( U )

In the next table, we glve a few lmportant examples. Often, the formulas for
11.2.INVERSION METHOD 29

F-'( U )can be slmpllfled, by noting for example that 1-U 1s dlstrlbuted as U .

-~
Density f (F ) F(x) I X=F-'(u) I Simplified form
Exponential(X) I I

Cauchy(a)
U 1 1 X 1
-+-arctan(-) atan(A( U--)) atan(n u )
n(x a+$) 2 A U 2

Rayleigh(u)
--22 22
_I_

-2e
U
a02 , X ~ O 1-e m2 ad-log( 1- u) a m

I I I
Triangular on(0,a ) I

I Tail of Rayleigh I I I 1

I I I
Pareto( a ,b )

There are many areas In random varlate generatlon where the lnverslon
method 1s of partlcular Importance. We clte four examples:

Example 2.1. Generating correlated random variates.

. When two random varlates X and Y are needed wlth dlstrlbutlon functlons
F and G respectlvely, then these can be obtalned as F-'( U )and G-'(V )where
U and V are unlform [0,1] random varlates. If U and V are dependent, then so
are F - ' ( U ) and C - ' ( V ) . Maxlmal correlatlon Is achleved by uslng V = U , and
maxlmal negatlve correlatlon 1s obtalned by settlng v=-U. Whlle other
methods may be avallable for generatlng X and/or Y lndlvldually, few methods
allow the flexlblllty of controlllng the correlatlon as descrlbed here. In varlance
reductlon, negatlvely correlated random varlates are very useful (see e.g. Ham-
mersley and Handscomb, 1964, or Bratley, Fox and Schrage, 1984).
30 II.2.IhTVERSION METHOD

Example 2.2. Generating maxima.

To generate X=max(X,, . . . , x, ), where the Xi's are lld random varlates
wlth dlstrlbutlon functlon F , we could:
(I) Generate X , , . . . , X , , and take the maslmum.
(11) Generate a unlform [0,1]random varlate U and And the solutlon X of
Pyx) = u.
(111) Generate V , a random varlate dlstrlbuted as the maximum of n lld unl-
form [0,1] random varlates, and And the solutlon x' of F (X) = V.
Thus, the elegant solutlons (11) and (111) lnvolve lnverslon.

Example 2.3. Generating all order statistics.

A sample X ( , ) , . . . , X(n)of order statlstlcs of a sequence X , , . . . , X , of
lld random varlables wlth dlstrlbutlon functlon F can be obtalned a s
F-'(U(,)), . . . , F-l(U(n$ where the U ( i ) ' s are t h e order statlstlcs of a unlform
sample. ks we wlll see further on, thls 1s often more efnclent than generatlng the
Xi sample and sortlng It.

Example 2.4. A general purpose generator.

The lnverslon method 1s the only truly unlversal method: If all we can do 1s
compute F (a: ) for all 2 , and we have enough (Le., lnflnlte) tlme on our hands,
then we can generate random varlates wlth dlstrlbutlon functlon F . All the
other methods described In thls book requlre addltlonal lnformatlon In one form
or another.
11.2 .INVERSION METHOD 31

2.2. Inversion by numerical solution of F(X)=U.

The lnverslon method 1s exact when an expllclt form of F-’ 1s known. In
other cases, we mtist solve the equatlon F (X) = U numerlcally, and thls
requlres an lnflnlte amount of tlme when F 1s contlnuous. Any stopplng rule
that we use wlth the numerlcal method leads necessarlly t o an lnexact algorlthm.
In thls sectlon we wlll brlefly descrlbe a few numerlcal lnverslon algorlthms and
stopplng rules. Desplte the fact that the algorlthms are lnexact, there are sltua-
tlons In whlch we are vlrtually forced to use numerlcal lnverslon, and I t 1s lmpor-
tant to compare dlfferent lnverslon algorlthms from varlous polnts of vlew.
In what follows, X 1s the (unknown, but exact) solutlon of F (X) = U ,and
X * 1s the value returned by the numerlcal lnverslon algorlthm. A stopplng rule
whlch lnslsts that I X*-X I <6 for some small 6>0 1s not reallstlc because for
large values of X , thls would probably lmply that the number of slgnlflcant
dlglts 1s greater than the bullt-ln llmlt dlctated by the wordslze of the computer.
A second cholce for our stopplng rule would by I F (X*)-F ( X ) I < E where
, E>O
1s a small number. Slnce all F values are In the range [0,1],we do not face the
above-mentloned problem any more, were I t not for the fact that small varlatlons
In X can lead to large varlatlons In F (X>values. Thus, I t Is posslble that even
the smallest reallzable lncrement ln X yields a change In F ( X ) that exceeds the
glven constant E . A thlrd posslblllty for our stopplng rule would be
I X*-X I <6 I X I where the value of 6 1s determined by the wordslze of the
computer. Whlle thls addresses the problem of relatlve accuracy correctly, I t wlll
lead to more accuracy than 1s orlnarlly requlred for values of X near 0. Thus, no
stopplng rule seems universally recommendable. If we know that x takes values
In [-1,1], then the rule I X*-X I <6 seems both practlcal and amenable to
theoretlcal analysls. Let us flrst see what we could do when the support of F falls
outslde [-1,1].
Let h :I? +(-1,l) be a strlctly monotone contlnuous transformatlon.
Assume now that we obtaln X * by the followlng method:

Let Y* be the numerical solution of F ( h - ’ ( y ) ) = u , where u is a uniform [0,1]random

variable and y* is such that it is within 6 of the exact solution Y of the given equation.
x* +- h-‘(Y*)

Here we used the fact that Y has dlstrlbutlon functlon F ( h - ’ ( y ) ) , I y I <1 .

Let us now look at what happens to the accuracy of the solutlon. A varlatlon of
dy on the value of y leads t o varlatlon of h - ” ( y ) dx = h-”(h (5))dx on the
correspondlng value of x . The expected varlatlon thus 1s about equal to v6
where
1
v = E(h-”(h ( X ) ) )= E(-
h f ( X )) *
32 II.2.INVERSION METHOD

Unfortunately, the best transformatlon h , 1.e. the one that mlnlmlzes V , depends
upon the dlstrlbutlon of x.
We can glve the reader some lnslght In how to
choose h by an example. Conslder for example the class if transformatlons
h(x)=
x -m
s+lx-m I '
where s >O and mER are constants. Thus, we have
h-'(y)=m+sy/(l-Iy I),and
v = E (-(s
1
S
+ I X-m I 12) = s + 2 (~I x - m I )+'E
S
((x-m 12) .
For symmetrlc random varlables x,
thls expresslon 1s mlnlmlzed by settlng
m =O and s = d m . For asymmetrlc X ,the mlnlmlzatlon problem 1s very
dlfflcult. The next best thlng we could do 1s mlnlmlze a good upper bound for V ,
such as the one provlded by applylng the Cauchy-Schwarz lnequallty,

Thls upper bound 1s mlnlmal when

m =E(X),s =d-.
The upper bound for V then becomes 4 d m . Thls approach requlres elther
.
exact values or good approxlmatlons for m and s We refer to Exerclse 1 for a
detalled comparlson of the average accuracy of thls method wlth that of the
dlrect solutlon of F ( X ) = U glven that the same stopplng rule 1s used.
We wlll dlscuss three popular nurnerlcal lnverslon algorlthms for F ( X ) = U :

The bisection method

Find an initial interval [ a ,b ] t o which the solution belongs.

REPEAT
X+(a + b )/2
IF F ( X ) < u
THEN a+X
ELSE b +X
UNTIL b -a 526
RETURN x
II.2.INVERSION METHOD 33

The secant method (regula falsi method)

Find an interval [ a , b ] to which the solution belongs.

REPEAT

IF F ( X ) < U
THEN a t X
ELSE b +X
UNTIL b -a 56
RETURN x

The Newton-Raphson method

Choose an initial guess X

REPEAT

UNTIL stopping rule is satisfled. (Note: f is the density corresponding to F .)

RETURN X .

In the flrst two methods, we need an lnltlal lnterval [ a ,b ] known t o contaln

the solutlon. If the user knows functlons G and H such that
G ( x ) L F ( a : ) L H ( x ) for all x, then we could start wlth
[ a , b ] = [ G - ' ( U ) , H - ' ( U ) ] . In particular, If the support of F 1s known, then we
can set [ a , b ] equal t o It. Because I t 1s lmportant t o have reasonably small lnter-
vals, any a prlorl lnformatlon should be used to select [ a ,b 1. For example, If F
has varlance o2 and Is symrnetrlc about 0, then by Cantelll's extenslon of
Chebyshev's lnequallty,
X2
F(x)L (a: >o) .
X 2+02

Thls suggests that when U > -,1 we take

2
34 II.2.INVERSION METHOD

1
When uI2
-
we ,
argue by symmetry. Thus, lnformatlon about moments and
quantlles of F can be valuable for lnltlal guesswork. For the Newton-Raphson
method, we can often take an arbltrary polnt such as 0 a s our lnltlal guess.
The actual cholce of an algorlthm depends upon many factors such as
(1) Guaranteed convergence.
(11) Speed of convergence.
(111) A prlorl lnformatlon.
(lv) Knowledge of the denslty f .
If f 1s not expllcltly known, then the Newton-Raphson method should be
1
avolded because the approxlmatlon of f (z ) by -(F (a: +6)-F (z )) 1s rather lnac-
6
curate because of cancelatlon errors.
Only the blsectlon method 1s guaranteed t o converge In all cases. If
F ( X ) = U has a unlque solutlon, then the secant method converges too. By "con-
vergence" we mean of course that the returned varlable x*
would approach the
exact solutlon X if we would let the number of lteratlons tend to 03. The
Newton-Raphson method converges when F 1s convex or concave. Often, the
denslty f 1s unlmodal wlth peak at m . Then, clearly, F 1s convex on (-m,m 1,
and concave on [ m , m ) , and the Newton-Raphson method started at m con-
verges.
Let us conslder the speed of convergence now. For the blsectlon method
started at [ a , b ] = [g U ) , g , ( U)](where g 1,g2 are glven functlons), we need N
lteratlons If and only if
2N-1 < g 2 ( U ) - g 1 ( U )5 2N .

The solutlon of thls 1s

where log+ 1s the posltlve part of the logarlthm wlth base 2. From thls expres-
slon, we retaln that E ( N )can be lnflnlte for some long-talled dlstrlbutlons. If the
solutlon 1s known to belong t o [-1,1], then we have determlnlstlcally,

N 5 l+log (L).
+ 6

And In all cases in whlch E ( N ) < 00, we have as 610, E(N)-log(-). 1 Essen-
s
tlally, addlng one blt of accuracy t o the solutlon 1s equlvalent t o addlng one
lteratlon. As an example, let us take 6 = lo-', whlch corresponds t o the stan-
dard cholce for problems wlth solutlons In [-1,1] when a 32-blt computer 1s used.
The value of N In that case 1s In the nelghborhood of 24, and thls 1s often lnac-
ceptable.
The secant and Newton-Raphson methods are both faster, albelt less robust,
than the blsectlon method. For a good dlscusslon of the convergence and rate of
convergence of the W e n methods, we refer to Ostrowskl (1973). Let us merely
11.2 .INVERSION METHOD 35

state one of the results for E ( N ) ,the quantlty of lnterest to us, where N 1s the
number of lteratlons needed to get to wlthln 6 of the solutlon (note that thls 1s
lmposslble to verlfy when an algorlthm 1s runnlng !). Also, let F be the dlstrlbu-
tlon functlon correspondlng to a unlmodal denslty wlth absolutely bounded
derlvatlve f'. The Newton-Raphson method started at the mode converges, and
for some number N o dependlng only upon F (but posslbly 00) we have

where all logarlthms are base 2. For the secant method, a slmllar statement can
1
be made but the base should be replaced by the golden ratlo, -(l+&). In both
2
cases, the Influence of 6 on the average number of lteratlons 1s practlcally nll, and
the asyrnptotlc expresslon for E ( N ) 1s smaller than In the blsectlon method
(when 610). Obvlously, the secant and Newton-Raphson methods are not unlver-
sally faster than the blsectlon method. For ways of acceleratlng these methods,
see for example Ostrowskl (1973, Appendlx I, Appendlx G).

2.3. Explicit approximations.

When F-' 1s not expllcltly known, I t can sometlmes be well approxlmated by
another expllcltly known functlon g (U). In lteratlve methods, the stopplng rule
usually takes care of the accuracy problem. Now, by resortlng to a one-step pro-
cedure, we squarely put the burden of verlfylng the accuracy of the solutlon on
the shoulders of the theoretlclan. Also, we should define once agaln what we
mean by accuracy (see Devroye (1982) for a crltlcal dlscusslon of varlous
deflnltlons). Iteratlve methods can be notorlously slow, but thls 1s a small prlce
to pay for thelr conclseness, slmpllclty, flexlblllty and accuracy. The four maln
llmltatlons of the dlrect approxlmatlon method are:
(1) The approxlmatlon 1s valid for a glven F : to use I t when F changes fre-
quently durlng the slmulatlon experlment would probably requlre extraordl-
nary set-up tlmes.
(11) The functlon g must be stored. For example, g 1s often a ratlo of two poly-
nomlals, In whlch case all the coemclents must be put In a long table.
(111) The accuracy of the approxlmatlon 1s fixed. If a better accuracy 1s needed,
the entlre functlon g must be replaced. Thls happens for example when one
swltches to a computer wlth a larger wordslze. In other words, future Corn-
puter upgrades wlll be expensive.
(lv) Certaln functlons cannot be approxlmated very well by standard aPProx1-
matlon technlques, except posslbly by lnacceptably compllcated functions.
Also, approxlmatlons are dlmcult to develop for multlparameter famllles Of
functlons.
36 II.2.INVERSION METHOD

How one actually goes about deslgnlng approxlmatlons g wlll not be

explalned here. For example, we could start from a very rough approxlmatlon of
F - l 1 and then expllcltly compute the functlon that corresponds t o one or two or
a Axed number of Newton-Raphson lteratlons. Thls 1s not systematlc enough In
general. A spllne method was developed In Kohrt (1980) and Ahrens and Kohrt
(1981). In the general Ilterature, one can And many examples of approxlmatlons
by ratlos of polynomlals. For example, for the lnverse of the normal dlstrlbutlon
functlon, Odeh and Evans (1974) suggest

4 4
where A (a: )= ai z i t and B (a: )= bi x i , and the coefllclents are as shown In
I =o i=O
the table below:

-0.322232431088 0.0993484626060
-1.0 0.588581570495
-0.342242088547 0.531103462366
-0.0204231210245 0.103537752850
I4 I -0.0000453642210148 I 0.0038560700634

1
For u In the range [-,1-10-20], we take -g (l-u
) l and for u In the two tlny left-
2
over lntervals near 0 and 1, the approxlmatlon should not be used. Rougher
approxlmatlons can be found In Hastlngs (1955) and Balley (1981). Balley’s
approxlmatlon requlres fewer constants and 1s very fast. The approxlmatlon of
Beasley and Sprlnger (1977) 1s also very fast, although not as accurate as the
Odeh-Evans approxlmatlon glven here. Slmllar methods exlst for the lnverslon of
beta and gamma dlstrlbutlon functlons.

2.4. Exercises.
1. Most stopplng rules for the numerlcal lteratlve solutlon of F (X)=U are of
the type b -a 56 where [ a , b ] 1s an lnterval contalnlng the solutlon X , and
6>0 1s a small number. These algorlthms may never halt If for some u ,
there 1s an lnterval of solutlons of F(X)=u (thls applles especlally to the
secant method). Let A be the set of all u for whlch we have for some
x < y F (x )=F ( y )=u .
Show that P ( U EA )=O, 1.e. the probablllty of
endlng up In an lnAnlte loop 1s zero. Thus, we can safely llft the restrlctlon
lmposed throughout thls sectlon that F (X)=u has one solutlon for all u .
2. Show that the secant method converges if F (X)=U has one solutlon for
the glven value of U .
3. Show that If F(O)=O and F 1s concave on [O,oo),then the Newton-Raphson
method started at 0 converges.
II.2.INVERSION METHOD 37

4. Student’s t distribution with 3 degrees of freedom.

Consider the denslty
2
f (x)=
n(l+Z 2)2 ’
and the correspondlng dlstrlbutlon functlon
1 1 X
F (z)= -+-(arc
2 7 r
tan z +-1+x2 > .
These functions deflne the t dlstrlbutlon with 3 degrees of freedom. Else-
where we wlll see very efflclent methods for generating random varlates from
thls dlstrlbutlon. Nevertheless, because F-l is not known expllcltly (except
perhaps as an inflnlte series), thls distribution can be used t o illustrate many
points made In the text. Note A r s t that the dlstrlbutlon is symmetric about
0. Prove flrst that
1 1
-+-arc tan x 5 1 2
F ( z ) 5 -+-arc tan x
2 7 r 2 7 r

Thus, for U >,‘ 2 the solutlon of F ( X ) = U lles In the lnterval

[tan(-(7r U--)),tan(n(
1 U-?))I
1
2 2
Using thls lnterval as a startlng lnterval, compare and tlme the blsectlon
method, the secant method and the Newton-Raphson method (In the latter
method, start at 0 and keep iterating untll X does not change In value any
further). Flnally, assume that we have an efflclent Cauchy random varlate
generator at our disposal. Recalling that a Cauchy random varlable c IS
1
dlstrlbuted as tan(n(U--)), show that we can generate X by solving the
2
e quat ion
arc tan X +- X = arc tan C ,
1+x2
and by startlng wlth lnltlal lnterval

when c>O (use symmetry In the other case). Prove that thls Is a valid
method.
5. Develop a general purpose random varlate generator whlch 1s based upon
lnverslon by the Newton-Raphson method, and assumes only that F and the
correspondlng denslty f can be computed at all points, and that 1 1s uni-
modal. Verlfy that your method 1s convergent. Allow the user t o W e c ! m a
mode If thls information 1s avallable.
 I

38 II.2.INVERSION METHOD

6. Wrlte general purpose generators for the blsectlon and secant methods In
whlch the user speclfles an lnltlal lnterval [g u),g,(V ) ] .
7. Dlscuss how you would solve F (x)=u for X by the blsectlon method If no
lnltlal lnterval 1s avallable. In a Arst stage, you could look for an lnterval
[ a ,b ] whlch contalns the solutlon x . In a second stage, you proceed by ordl-
nary blsectlon untll the lnterval’s length drops below 6. Show that regardless
of how you organlze the orlglnal search (thls could be by looklng at adjacent
lntervals of equal length, or adJacent lntervals wlth geometrlcally lncreaslng
lengths, or adJacent lntervals growlng as 2,22,222,...),the expected tlme taken
by the entlre algorlthm 1s 00 whenever E (log, I X I )=m. Show that for
extrapolatory search, I t 1s not a bad strategy to double the lnterval slzes.
Finally, exhlblt a dlstrlbutlon for whlch the glven expected search tlme 1s 00.
(Note that for such dlstrlbutlons, the expected number of blts needed to
represent the lnteger portlon Is lnflnlte.)
8. An exponential class of distributions. Conslder the dlstrlbutlon func-
tlon F (a: )=1-e -An(z) where A, (a:)= 5
:=1
ai x i for a: 20 and A, (a:)=O for
z <O. Assume that all coefflclents ai are nonnegatlve and that a ,>O. If U
1s a unlform [0,1] random varlate, and E 1s an exponentlal random varlate,
then I t 1s easy to see that the solutlon of l-e-An(X)=U 1s dlstrlbuted as the
solutlon of A , ( X ) = E . The baslc Newton-Raphson step for the solutlon of
the second equatlon 1s

xc-x- A n (X1-E
An’(X) *

Slnce a ,>O and A , 1s convex, any startlng polnt X 20 wlll yleld a conver-
gent sequence of values. We can thus start at X” =O or at =E / a (whlchx
1s the flrst value obtalned In the Newton-Raphson sequence started at 0).
Compare thls algorlthm wlth the algorlthm In whlch X 1s generated as

where E , , . . . , E, are lld exponentlal random varlates.

9. Adaptive inversion. Conslder the sltuatlon in whlch we need to generate
a sequence of n lld random varlables wlth continuous dlstrlbutlon functlon
F by the method of lnverslon. The generated couples (x,,u,), ... are stored
(X,=F-l(U,) and U l 1s unlform [O,l]).Deflne an algorlthm based upon a
dynamlc hash table for the Vi’sIn whlch the table 1s used to And a good
startlng lnterval for lnverslon. Implement, and compare thls adaptlve
method wlth memoryless algorithms (Yuen, 1981).
10. Truncated distributions. Let X be a random varlable wlth dlstrlbutlon
functlon F . Deflne the truncated random variable Y by Its dlstrlbutlon
II.2.INVERSION METHOD 39

functlon

0 x <a
G(x) =

1 x >6

Here -cos&< 6 500. Show that Y can be generated as

F-l(F ( a )+U ( F (6 )-F ( a ))) where U 1s a unlform [0,1] random varlate.
11. Find a monotonlcally decreaslng denslty f on [O,co)such that the Newton-
Raphson procedure started at X=O needs N steps to get wlthln 6 of the
solutlon of F ( X ) = U where N 1s a random varlable wlth mean E ( N )=00
for all 6>0.
12. T h e logistic distribution. A random varlable X 1s sald t o have the logls-
t l c dlstrlbutlon wlth parameters a ER and 6 > O when
1 .
F(x) =
--2 -a
b
l+e
It Is obvlous t h a t a 1s a translatlon parameter and t h a t b 1s a scale parame-
ter. The standardlzed loglstlc dlstrlbutlon has a =o,b =l. The denslty 1s
e-$
f (XI= = F(x)(l-F(x)) .
(l+e+ l2
The loglstlc denslty 1s symmetrlc about 0 and resembles In several respects
the normal denslty. Show the followlng:
A. When U 1s unlformly dlstrlbuted on [0,1], then x=log(-)
U has the
1- u
standard loglstlc dlstrlbutlon.
r,

B.
u
-
1- u
1s dlstrlbuted as the ratlo of two lld exponentlal random varlables.

C. We say that a random varlable 2' has the extremal value dlstrlbutlon
with parameter a when F ( x ) = ~ - ' ~ * If . X 1s dlstrlbuted as Z with
parameter Y where Y 1s exponentlally dlstrlbuted, then X has the
standard loglstlc dlstrlbutlon.
n2 7n4
D. E ( X 2 ) = - , and E(X4)=-.
3 15
E. If X1,X2are lndependent extremal value dlstrlbuted random varlables
wlth the same parameter a , then Xl-x, has a loglstlc dlstrlbutlon.

..
40 II.3.REJECTION METHOD

3. THE REJECTION METHOD.

3.1. Definition.
The rejectlon method 1s based upon the followlng fundamental property of
densltles:

Theorem 3.1.
Let X be a random vector with denslty f on R d , and let U be an lndepen-
1 dent unlform [0,1]random varlable. Then (x,cuf (x)) 1s unlformly dlstrlbuted
on A ={(x ,u ):x ER , O s u 5 c f (x)}, where c > O 1s an arbltrary constant.
Vlce versa, If ( X , U ) 1s a random vector In R d-C1 unlformly dlstrlbuted on A ,
then X has denslty f on R d .

Proof of Theorem 3.1.

For the flrst statement, take a Borel set B & A , and let B, be the sectlon of
B at x , 1.e. B, ={ u :(x ,u )EB }. By Tonelll's theorem,

Since the area of A 1s c , we have shown the flrst part of the Theorem. The
second part follows If we can show that for all Borel sets B of R d ,
P ( X E B )=Sj (x ) dx (recall the deflnltlon of a denslty). But
B
P ( X E B ) = P ( ( X , U ) E B ,= ((x,u):sEB ,o<u Scf (x)})
du dx

whlch was to be shown.

11.3.REJECTION METHOD 41

Theorem 3.2.
Let x1,x2,... be a sequence of lld random vectors taklng values In R d , and
let A C R be a Borel set such that P ( X , E A ) = p BO. Let Y be the flrst Xi
taklng values In A . Then Y has a dlstrlbutlon that 1s determlned by
P ( X , E A nB )
P ( Y E B )= , B Borel set of R
' P
In partlcular, If x, ,
1s unlformly dlstrlbuted In A where A , 2 A , then Y 1s unl-
formly dlstrlbuted In A .

Proof of Theorem 3.2.'

For arbltrary Borel sets B , we observe that
03
P ( Y E B ) = C P ( X , @ i , . . . ,Xi-14A,X;EBnA)
t =1
00
= (1-p )j-lP (X,EA n B )
i=1
1

whlch was to be shown. If XI1s unlformly dlstrlbuted In A , , then

This concludes the proof of Theorem 3.2.

The baslc verslon of the reJectlon algorlthm assumes the existence of a den-
slty g and the knowledge of a constant c 21 such that
f (XI L W(X) (all 2 ) .

Random varlates wlth denslty f on R can be obtalned as follows:

42 II.3.REJECTION METHOD

The rejection method

REPEAT
Generate two independent random variates x (with density g on R ) and U (uni-
formly distributed on [0,1]).

UNTIL UT 51
RETURN x

By Theorem 3.1, ( X , c U g ( X ) )(where X and U are as explalned In the flrst llne

of the REPEAT loop) 1s unlformly dlstrlbuted under the curve of cg In R d + l .
By Theorem 3.2, we conclude that the random varlate ( x , c U g (x))
generated by
thls algorlthm (1.e. at tlme of exlt) 1s unlformly dlstrlbuted under the curve of f .
By the second part of Theorem 3.1, we can then conclude that Its d-dlmenslonal
proJectlon X must have denslty f .
The three thlngs we need before we can apply the reJectlon algorlthm are (1)
a domlnatlng denslty g ; (11) a slmple method for generatlng random varlates wlth
.
denslty g ; and (111) knowledge of c Often, (1) and (111) can be satlsfled by a prlorl
lnspectlon of the analytlcal form of f . Baslcally, g must have heavler talls and
.
sharper lnflnlte peaks than f In some sltuatlons, we can determlne c g for entlre
.
classes of densltles f The domlnatlng curves c g should always be plcked wlth
care: not only do we need a slmple generator for g (requlrement (ll)), but we
’
must make sure that the cornputatlon of -1s slmple. Flnally, c g must be
f (XI
such that the algorlthm 1s efflclent.
Let N be the number of lteratlons In the algorlthm, Le. the number of pairs
( X , u )requlred before the algorlthm halts. We have

where

Thus, E ( N ) = - =1 c , E (N2)=---2 1
and Var(N)=-- - c -c . In other
P P2 p P2
words, E ( N ) 1s one over the probablllty of acceptlng X . From thls we conclude
that we should keep c as small as posslble. Note that the dlstrlbutlon of N 1s
geometrlc wlth parameter p =-
1

C
. Thls 1s good, because the probabllltles
11.3 .REJECTION METHOD 43

P ( N =z' ) decrease monotonlcally, and at an exponentlal rate (note that

P ( N > i ) = ( l - p 1' 5 e + 1.
The reJectlon method has an almost unllmlted potentlal. We have glven up
the prlnclple that one unlform (0,1]random varlate ylelds one varlate X (as In
the lnverslon method), but what we recelve In return 1s a powerful, slmple and
exact algorlthm.

Example 3.1. Bounded densities of compact support.

Let cMaa ,b be the class of all densltles on [ a ,b ] bounded by M .Any such
denslty 1s clearly bounded by M .Thus, the rejectlon algorlthm can be used wlth
unlform domlnatlng denslty g ( x ) = ( b -a )-' ( a 5. 5 b ), and the constant c
becomes M ( b -a ). Formally, we have

The rejection method for C M , ~ , ~

REPEAT
Generate two lndependent unlform [o,i] random varlates U and V .
Set X+a+(b-a)V.
UNTIL U M < f ( X )
RETURNX

The reader should be warned here that thls algorlthm can be horrlbly lnemclent,
and that the cholce of a constant domlnatlng curve should be avolded except In a
few cases.

3.2. Development of good rejection algorithms.

Generally speaklng, g 1s chosen from a class of easy densltles. Thls class
lncludes the unlform denslty, trlangular densltles, and most densltles that can be
Generated qulckly by the lnverslon method. The sltuatlon usually dlctates whlch
densltles are consldered as "easy". There are two maJor technlques for determln-
lng c and g In the lnequallty f L c g : one could flrst study the form of f and
apply one of many analytlcal devtces for obtalnlng lnequalltles. Many of these
are Illustrated throughout thls book (collectlng them In a speclal chapter would
have forced us to dupllcate too much materlal). Whlle thls approach glves often
44 11.3.REJECTION METHOD

qulck results (see Example 3.2 below), I t 1s ad hoc, and depends a lot on the
mathematlcal background and lnslght of the deslgner. In a second approach,
whlch 1s also lllustrated In thls sectlon, one starts wlth a famlly of domlnatlng
densltles g and chooses the denslty wlthln that class for whlch c 1s smallest.
Thls approach 1s more structured but could sometlmes lead to dlmcult optlmlza-
tlon problems.

Example 3.2. A normal generator by rejection from the Laplace density.

Let f be the normal denslty. Obtalnlng an upper bound for bolls down
to obtainlng a lower bound for -.X22 But we have of course

1
-(
2

Thus,
--2 2 1 e 21- 1 Z I
-
1
= cg (x )
6e 2 < -
-6 7

where g (x ) = f e - l I 1s the Laplace denslty, and c = 1s the rejectlon

2
constant. Thls suggests the followlng algorlthm:

A normal generator by the rejection method

REPEAT
Generate an exponential random variate x and two independent uniform [OJ] ran-
1
dom variates U and V . If U <- set X t - X (X is now distributed as a Laplace
2’
random variate).
X2
1 f-IXI 1 -2
UNTIL v7ze 5xe.
RETURN X

1
The condltlon In the UNTIL statement can be cleaned up. The constant -
6
cancels out on left and rlght hand sldes. It 1s also better to take logarithms on
both sldes. Flnally, we can move the slgn change to the RETURN statement
because there 1s no need for a slgn change of a random varlate that wlll be
rejected. The random varlate U can also be avolded by the trlck lmplemented In
the algorlthm glven below.
II.3.REJECTION METHOD 45

A normal generator by rejection from the Laplace density

REPEAT
Generate an exponential random variate X and an independent uniform [-l,l] ran-
dom variate V .
UNTlL (x-1)25-21~g( I vI)
RETURN X-X sign ( V )

For glven densltles f and g , the rejectlon constant c should be at least

equal t o

We cannot loose anythlng by settlng c equal t o thls supremum, because thls

lnsures us that the curves of f and c g touch each other somewhere. Instead of
letting g be determlned by some lnequallty whlch we happen t o come across as
In Example 3.2, I t Is often wlser to take the best ge In a famlly of densltles
parametrlzed by 8. Here 8 should be thought of as a subset of R k (In whlch case
we say that there are k parameters). Deflne the optlmal rejectlon constant by

The optlmal 8 1s that for whlch c e 1s mlnlmal, Le. for whlch c 8 1s closest to 1.
We wlll now lllustrate thls optlmlzatlon process by an example. For the sake
.
of argument, we take once agaln the normal denslty f The famlly of domlnat-
lng densltles 1s the Cauchy famlly wlth scale parameter 8:

There 1s no need t o conslder a translatlon parameter as well because both f and

the Cauchy densltles are unlmodal wlth peak at 0. Let us flrst compute the
optlmal reJectlon constant c 8 . We wlll prove that
46 11.3.REJECTION METHOD

We argue as follows: f / g o 1s maxlmal when log(f / g o ) 1s maxlmal. Settlng the

derlvatlve wlth respect to x of log( f / g e) equal to 0 ylelds the equatlon
2x
-x+- =0
e2+x 2

Thls glves the values z=O and ~ = f d 2 - 8 (the latter case can only happen
A.
~

when 1 9 ~ 5 2 ) . At x = O , f / g o takes the value 8 At z=&&F ,f /go

6Q
6 -
takes the value -e . It 1s easy to see that for e<&, the maxlmum of f /go
1s attalned at z
is attalned at x =O.
=&m et9
and the mlnlmum at x
concludes the verlflcatlon
Thls
=O. For e>&, the maxlmum
of the expresslon for c 8.
The remalnder of the optlmlzatlon 1s slmple. The functlon c e has only one
mlnlmum, at 8=1.
dltlon of acceptance Uc eg &X)Lf
The mlnlmal value 1s c &.
(x)can be rewrltten as
Wlth thls value, the con-

X2
--- <-
1 e--T
e T l+x2 - 6 9

or as
&-
U 5 (1+X2)-e -$
2

A normal generator by rejection from the Cauchy density

[SET-UP]

a+-
dT
2
[GENERATOR]
REPEAT
Generate two independent uniform [0,1]
random variates u and v
Set X + t a n ( r V ) , S+x2 (xis now Cauchy distributed).
_-
s
UNTIL U l a ( l + S ) e
RETURN x

The algorlthm derlved here, though I t has a reJectlon constant near 1 . 4 1s no

match for most normal generators developed further on. The reason for thls 1s
that we need falrly expenslve Cauchy random varlates, plus the evaluatlon of exp
In the acceptance step.
II.3.REJECTION METHOD 47

3.3. Generalizations of the rejection method.

Some generallzatlons of the rejectlon method are lmportant enough t o war-
rant speclal treatment ln thls key chapter. The flrst generallzatlon concerns the
following case:
f ($1 = c g ( z ) $(z) 9

where the functlon $ 1s [O,l]-valued, g 1s an easy denslty and c 1s a normallza-

tion constant at least equal to 1. The rejectlon algorlthm for thls case can be
rewrltten as follows:

The rejection method

REPEAT
Generate independent random variates X , U where X has density g and U is uni-
formly distributed on [O,l].
UNTIL U <$(z )
RETURN x

Vaduva (1977) observed that for speclal forms of $, there 1s another way of
proceedlng. Thls occurs when $=l-Q where \k Is a dlstrlbutlon functlon of an
easy denslty.

Vaduva’s generalization of the rejection method

REPEAT
Generate two independent random variates x,Y ,where x has density g and Y has
distribution function 9.
UNTIL x< Y
RETURN x

For $=*,we need t o replace X 5Y In the acceptance step by X 2Y .

Theorem 3.3.
Vaduva’s rejectlon method produces a random varlate X wlth denslty
1 =cg (l-q), and the rejectlon constant (the expected number of lteratlons) Is c .
48 II.3.REJECTION METHOD

Proof of Theorem 3.3.

We prove thls by showlng that Vaduva's algorlthm 1s entlrely equlvalent to
the orlglnal reJectlon algorlthm. Note that the condltlon of acceptance, X <_ Y Is
wlth probablllty one satlsfled If and only If l-\k(X)zl-\k( Y). But by the proba-
blllty lntegral transform, we know that l-\k(y) 1s dlstrlbuted as U , a unlform
[0,1]random varlable. Thus, we need only verlfy whether U l l - \ k ( X ) , whlch
ylelds the orlglnal acceptance condltlon glven at the beglnnlng of thls section.

The cholce between generatlng U and computlng 1-\k(X) on the one hand
(the orlglnal rejectlon algorlthm) and generatlng Y wlth dlstrlbutlon functlon \k
on the other hand (Vaduva's method) depends malnly upon the relatlve speeds of
computlng a dlstrlbutlon functlon and generatlng a random varlate wlth that dls-
t rlb utlon.

Example 3.3.
Conslder the denslty

where a > O 1s a parameter and c 1s a normallzatlon constant. Thls denslty 1s

part of the gamma ( a ) denslty, wrltten here in a form convenient to us. The
~ , the functlon $ 1s e-'. Random varlates
domlnatlng denslty 1s g ( ~ ) = a a : ~ -and
-1
wlth denslty g can be obtalned qulte easlly by lnverslon (take V a where V 1s a
unlform [0,1]random varlate). In thls case, the ordlnary rejectlon algorlthm
would be

REPEAT
-
1

Generate two iid uniform [O,lJ random variates U , V ,and set X t V '.
UNTIL U s e - '
RETURN x

Vaduva's modlflcatlon essentially consists In generatlng X and an exponentlal

random varlate E untll E zx.
It 1s faster If we can generate E faster than we
can compute e-x (thls 1s sometlmes the case). Of course, In thls slmple example,
we could have deduced Vaduva's modlflcatlon by taklng the logarlthm of the
acceptance condltlon and notlng that E 1s dlstrlbuted as -log( U).
11.3.REJECTION METHOD 49

We now proceed wlth another generallzatlon found In Devroye (1984):

Theorem 3.4.
Assume that a denslty f on R can be decomposed as follows:
f (z 1 = Js (Y ,z ) h (Y 9 2 ) dY 9

s
where dy 1s an integral In R , 9 (y ,x) 1s a density In y for ail 2 , and there
ex1st.S a functlon H ( z ) such that O<h ( y , x ) L H ( x ) for all y , and H / S H is an
easy denslty. Then the followlng algorlthm produces a random varlate wlth den-
sity f , and takes N lteratlons where N 1s geometrlcally dlstrlbuted wlth param-
eter - 1
(and thus E ( N ) = J H ) .
SH

Generalized rejection method

REPEAT
Generate X with density H / $ H (on R ).
Generate Y with density g (y ,X),yER (Xis k e d ) .
Generate a uniform [0,1]
random variate U.
UNTTL.UH(X)Lh(Y,X)
RETURN X

Proof of Theorem 3.4.

We wlll prove that thls Theorem follows dlrectly from Theorem 3.2. Let us
deflne the new random vector W , = ( X , Y , u ) where w,
refers to the trlple gen-
erated In the REPEAT loop. Then, If A 1s the set of values w l = ( x , y , u ) for
- h (y ,z ), we have for all Bore1 sets B In the space of w
whlch uH (a: )< ,,
P
where p = P ( W , E A ) and W refers to the value of W , upon exlt. Take
B =(-oo,z ] XR X (0,1], and conclude that
P(X(returned)sz) = ‘ P ( X s z , W ( X ) < h ( Y , X ) ) .
P
50 II.3.REJECTION METHOD

We note first that by settlng z=00, p = I . But then, clearly, the varlate pro-
SH
duced by the algorlthm has denslty f as requlred.

3.4. Wald’s equation.

We wlll rather often be asked t o evaluate the expected value of

6
{==I
+(Wj) 9

i
where Wi 1s the collectlon of all random varlables used In the i - t h lteratlon of
the rejectlon algorithm,@ 1s some functlon, and N 1s the number of lteratlons of
the reJectlon method. The random varlable N 1s known as a stopplng rule
because the probabllltles P ( N = n ) are equal t o the probabllltles that
W,, . . . , W, belong t o some set B,. The lnterestlng fact 1s that, regardless of
whlch stopplng rule 1s used (l.e., whether we use the one suggested In the reJec-
tlon method or not), as long as the wi’s are lid random varlables, the following
remalns true:

Theorem 3.5. (Wald’s equation.)

Assume that W l , ... are lid R -valued random varlables, and that $ 1s an
arbltrary nonnegatlve Bore1 measurable functlon on R d . Then, for all stopplng
rules N ,
N

(-1

Proof of Theorem 3.5.

T o slmpllfy the notatlon we wrlte Zi =$( Wi ) and note that the Zi ’s are lld
nonnegatlve random varlables. The proof glven here 1s standard (see e.g. Chow
and Telcher (1978, pp. 137-138)), but wlll be glven In Its entlrety. We start by
notlng that 2; and I l ~ . . i ] are lndependent for all 2 . Thus, so are zi and I [ N > j ] .
We wlll assume that E(Z,)<oo and E ( N ) < c o . It 1s easy t o verlfy that t h e
chaln of equalltles glven below remalns valld when one or both of these expecta-
tlons 1s 00.

i=l i=1
II.3.REJECTION METHOD 51

@I

= C E (zi I [ ~ > i l )
t =1
co
= CE(Zi)P(N>i)
i=1

=E(&) E P ( N l 2 )
i=1

= E(&) E ( N ).

The exchange of the expectatlon and lnflnlte sum ls'allowed by the monotone
convergence theorem: Just note that for any sequence of nonnegatlve random
n n M

i=i i=i t =I

It should be noted that for the reJectlon method, we have a speclal case for
whlch a shorter proof can be glven because our stopplng rule N 1s an lnstantane-
ous stopplng rule: we deflne a number of declslons Di , all 0 or 1 valued and
dependent upon Wi only: Dl=O lndlcates that we "reJect" based upon W , ,
etcetera. A 1 denotes acceptance. Thus, N 1s equal to n If and only If D, =1 and
Di =O for all 2 <n Now, .
E ( . g $(wj1)
1 =1

=E ( $( wi ))+E ($( wN 1)
i<N
=E ( N - W I
Wl) D ,=O)+E (II
W(, ) I D 1=1>

whlch proves thls speclal case of Theorem 3.5.

 52 II.3.REJECTION METHOD

3.5. Letac's lower bound.

In a profound but llttle publlclzed paper, Letac (1915) asks whlch dlstrlbu-
...
tlons canebe obtalned for X=UN where N 1s a stopplng tlme and U l , U 2 , 1s an
lld sequence of unlform [0,1] random varlables. He shows among other thlngs that
all densltles on [0,1] can be obtalned In thls manner. In exerclse 3.14, one unlver-
sal stopplng tlme wlll be descrlbed. It does not colnclde with Letac's unlversal
stopplng rule, but wlll do for dldactlcal purposes.
More Importantly, Letac has obtalned lower bounds on the performance of
any algorlthm of thls type. Hls maln result 1s:

Theorem 3.6. (Letac's lower bound)

Assume that X=UN has denslty f on [0,1], where N and the Ui's are as
deflned above. For any such stopplng rule N (Le., for any algorlthm), we have
"E I If I 103,

where I I .I I 03 Is the essentlal supremum of f .

Proof of Theorem 3.6.

BY the lndependence of the events [ N >n ] and [ Un EB] (whlch was also
used In the proof of Wald's equatlon), we have
P ( N 2 n , U n E B )= P ( N > n ) P ( U , E B ) .
But,
03
P (XEB) = P (N=n EB)
n =i
00
-
< P(Nrn,UnEB) e

n =I
00
= P(N-
> n ) P(U,EB)
n =1
=E(N)P(U,EB).

Thus, for all Bore1 sets B ,

If we take the supremum of the rlght-hand-slde over all B , then we obtaln

I If I IcCl.W

I
.- I
11.3 .REJECTION METHOD 53

There are qulte a few algorithms that fall Into this category. In partlcular, if
we use reJectlon wlth a constant dominating curve on [O,i], then we use N unl-
form random variates where for contlnuous f ,
. ww L: S U P , f ( z ) .
We have seen that In the reJectlon algorithm, we come within a factor of 2 of this
lower bound. If the ui’s have density g on the real llne, then we can construct
stopping times for all densltles f that are absolutely contlnuous wlth respect to
g , and the lower bound reads

f
For continuous -, the lower bound is equal to sup- f of . course. Again, wlth the
9 Q
reJectlon method-with g as domlnatlng denslty, we come wlthin a factor of 2 of
the lower bound.
There Is another class of algorlthms that A t s the descrlptlon glven here, not-
ably the Forsythe-von Neumann algorlthms, whlch wlll be presented in section
N.2.

3.6. The squeeze principle.

In the reJectlon method based on the inequality f s c g , we need to compute
the ratio f N tlmes where N Is the number of Iterations. In most cases, this Is
-9
a slow operation because f 1s presumably not a slmple functlon of Its argument
(for otherwlse, we would know how t o generate random variates from f by other
means). In fact, sometimes f Is not known expllcltly: In this book, we wlll
encounter cases In whlch I t is the integral of another function or the solutlon of a
nonllnear equatlon. In all these situations, we should try to avold the computa-
tlon of f either entlrely, or at least most of the time. For prlnclples leadlng to
-
9
the total avoidance of the computation, we refer to the more advanced chapter
N. Here we will briefly discuss the squeeze princlple (a term lntroduced by
George Marsaglla (1977)) designed to avoid the computation of the ratlo with
hlgh probability. One should In fact try to And functlons h and h , that are easy
to evaluate and have the property that
h,(X) 5f (z) L h,(s).
Then, we have:
54 11.3.REJECTION METHOD

The squeeee method

REPEAT
random variate u .
Generate a uniform [0,1]

Set W - w sh
Generate a random variate
Ucg (X).
Accept -[ l(x)].
x
with density g .

IF N O T Accept
THEN IF W s h , ( X ) THEN Accept - [ W < _ f (X)].
UNTIL, Accept
RETURN x

In thls algorlthm, we lntroduced the boolean varlable "Accept" to streamllne

the exlt from the REPEAT loop. Such boolean varlables come In handy whenever
a program must remaln structured and readable. In the algorlthm, we count on
the fact that "Accept" gets Its value most of the tlme from the comparlson
between W and h , ( X ) , whlch from now on wlll be called a qulck acceptance
w
step. In the remalnlng cases, we use a qulck reJectlon step ( > h 2 ( X ) ), and In
the rare cases that W Is sandwlched between h , ( X ) and h 2 ( X ) ,we resort to the
expenslve comparlson of W wlth f ( X )to set the value of "Accept".
The valldlty of the algorlthm 1s not Jeopardlzed by dropplng the qulck
acceptance and qulck reJectlon steps. In that case, we slmply have the statement
A c c e p t t [ W 5 f (X)],
and obtaln the standard reJectlon algorlthm. In many
cases, the qulck reJectlon step Is omltted slnce I t has the smallest effect on the
efflclency. Note also that I t 1s not necessary that h l > O or h 2 < c g , although
nothlng wlll be galned by conslderlng vlolatlons of these boundary condltlons.
We note that N 1s as In the reJectlon algorlthm, and thus, E ( N ) = c . T'
galn wlll be In the number of computatlons N j of f , the domlnatlng far+
the tlme complexlty. The computatlon of E ( N f ) demonstrates the usefulnt.
Wald's equatlon once agaln. Indeed, we have
N

1 =1

where Wi 1s the W obtalned In the i - t h lteratlon, and Xi 1s the x

used In the
i - t h lteratlon. To thls sum, we can apply Wald's equatlon, and thus,
II.3.REJECTION METHOD 55

Here we used the fact that we have proper sandwlchlng, 1.e. O l h <h,<cg.
If h,=O and h,=cg (l.e., we have no squeezlng), then we obtaln the result
E ( N 1 )=c for the reJectlon method. Wlth only a quick acceptance step (1.e.
- and/or h 2 5 c g are vlolated,
h 2 = c g ) , we have E ( N 1 ) = c - J h l . When h,>O
equallty In the expresslon for E (N1) should be replaced by lnequallty (exerclse
3.13).

Inequalities via Taylor's series expansion.

A good source of lnequalltles for functlons f in terms of slmpler functlons is
provided by Taylor's series expanslon. If f has n contlnuous derlvatlves
(denoted by f ( l ) , . . . , f ( n ) ), then I t 1s known that
n -1 Xn
f ( x ) = f ( o ) + - q (1)(0)+ . . +-f (n-l)(O)+-f '"'(S) 9

l! n -l! n!
<
where Is a number In the lnterval [O,x](or [x,O],dependlng upon the slgn of
s). From thls, by lnspectlon of the last term, one can obtaln lnequalltles whlch
are polynomlals, and thus prlme candldates for h and h,. For example, we have

From thls, we see that for x 20, e-' is sandwlched between consecutlve terms of
the well-known expanslon
03
e-Z = (-1)i -
X'
.
i =O i!
In partlcular,

Example 3.4. The normal density.

For the normal denslty f , we have developed an algorlthm based upon
rejection from the Cauchy denslty In Example 3.2. We used the lnequallty

f < c g where c = 6.
and g(x)=
1
. For h and h , we should look
7r(l+x2)
for slmple functlons of x Applylng the Taylor serles technlque descrlbed above,
we see that
X2 x2 x4
1--
2
5 &f (3) 5 1--+- .
2 8
56 II.3.REJECTION METHOD

Uslng the lower bound for h,, we can now accelerate our normal random varlate
generator somewhat:

Normal variate generator by rejection and Squeezing

REPEAT
random variate U
Generate a uniform (0,1]
Generate a Cauchy random variate x.
Set W e 2u . (Note: W t c V g (X)&.)
6 (l+X2)
X2
Accept --[ W 51--].
2
--X 2
IF NOT Accept THEN Accept -[ W se 1.
UNTIL Accept
RETURN X

Thls algorlthm can be lmproved In many dlrectlons. We have already got rld of
the annoylng normallzatlon constant 6. I I
For X >A,the qulck acceptance
step 1s useless ln vlew of h ,(X)<O. Some further savlngs In computer tlme result
1
lf we work wlth Y t - X 2 throughout. The expected number of computatlons of
2
f 1s

Example 3.6. Proportional squeezing.

It 1s sometimes advantageous t o sandwich f between two functions of the
same form as In
b9 I f <C9?
where g 1s an eSY density (as ln the rejectlon method), and 6 1s a posltlve con-
stant. When b and c are close t o 1, Such a proportlonal squeeze can be very use-
ful. For example, random variates can be generated as follows:
11.3.REJECTION METHOD 57

The proportional squeeze method

REPEAT
Generate a uniform [0,1]random variate U .
Generate a random variate X with density g
b
Accept +-[ u 5 -1.C
IF NOT Accept THEN Accept t[
f (XI 1.
U 5-
cg (X)
UNTIL Accept
RETURN x

Here the expected number of computatlons of f 1s qulte slmply c - 6 . The maln

area of appllcatlon of thls method 1s In the development of unlversally appllcable
algorlthms In whlch the real llne 1s flrst partitloned Into many Intervals. On each
lnterval, we have a nearly constant or nearly llnear plece of denslty. For thls
plece, proportlonal squeezlng wlth domlnatlng denslty of the form
g (z )=a,+a 1z can usually be applled (see exercises 3.10 and 3.11 below).

Example 3.6. Squeezing based upon an absolute deviation inequality.

Assume that a denslty f 1s close t o another denslty h In the followlng
sense:

Here g 1s another functlon, typlcally wlth small Integral. Here we could lmple-
ment the rejectlon method wlth as domlnatlng curve g +h , and apply a sueeze
step based upon f >h -9. After some slmpllflcatlons, thls leads t o the followlng
algorlthm:
58 II.3.REJECTION METHOD

REPEAT
Generate a random variate x with density proportional to h +g , and a uniform [0,1]
random variate U .
Accept -[-5
V(X) 1- 1-LJ
h ( X ) 1+u
IF NOT Accept THEN Accept --[ V(g ( X ) + h f (x)]
(x))<
UNTIL Accept
RETURN x

Thls algorlthm has rejectlon constant 1 + s g , and the expected number of evalua-
tlons of f Is at most 2 s g . Algorlthms of thls type are malnly used when g has
very small lntegral. One lnstance 1s when the startlng absolute devlatlon lnequal-
lty 1s known from the study of llmlt theorems In mathematical statlstlcs. For
example, when f 1s the gamma ( n ) denslty normallzed to have zero mean and
unlt varlance, I t 1s known that f tends t o the normal denslty as n + m . Thls
convergence 1s studled In more detall In local central llmlt theorems (see e.g.
Petrov (1975)). One of the by-products of thls theory 1s an lnequallty of the form
needed by us, where g 1s a functlon dependlng upon n , wlth lntegral decreaslng
at the rate 1 / 6 as n+m. The rejectlon algorlthm would thus have lmproved
performance as n+x. What 1s lntrlgulng here 1s that thls sort of lnequallty 1s
not llmlted t o the gamma denslty, but applles t o densltles of sums of lld random
varlables satlsfylng certaln regularlty condltlons. In one sweep, one could thus
deslgn general algorlthms for thls class of densltles. See also sectlons XrV.3.3 and
xIv.4.

3.7. Recycling random variates.

In thls sectlon we have emphaslzed the expected number of lteratlons In our
algorlthms. Sometlmes we have looked at the number of functlon evaluatlons.
But by and large we have steered clear of maklng statements about the expected
number of unlform random varlates needed before an algorlthm halts. One of the
reasons 1s that we can always recycle unused parts of the unlform random varl-
ate. The recycling prlnclple 1s harmless for our infinite preclslon model, but
should be used wlth extreme care In standard slngle preclslon arlthmetlc on com-
puters.
For the reJectlon method, based upon the lnequallty f <cg where g 1s the
domlnathg density, and c 1s a constant, we note that glven a random varlate X
II.3.REJECTION METHOD 59

wtth denslty g and an independent unlform [0,1]random varlate U , the haltlng

rule is Ucg ( fI( X)
)<l. / Glven that we halt, then Ucg ( X ) / f (X) Is agaln unl-
form on [0,1].If we reJect, then

is again unlformly dlstributed on [0,1].These recycled unlforms can be used

either In the generation of the next random varlate (If more than one random
varlate is needed), or In the next iteration of the rejectlon algorlthm. Thus, In
theory, the cost of unlform [0,1]random varlates becomes negllglble: I t Is one If
only one random varlate must be generated, and I t remalns one even If n random
varlates are needed. The followlng algorlthm incorporates these Ideas:

Rejection algorithm with recycling of one uniform random variate

Generate a uniform [0,1] random variate u

REPEAT
REPEAT
Generate a random variate x with density g .

Ut- V-r (prepare for recycling)

T-1
UNTIL U 50 (equivalent to V 51)
RETURN x (Xhas density f )
U +V (recycle)
UNTJL False (this is an inflnite loop; add stopping rule)

In thls example, we merely want to make a polnt about our ldeallzed model.
Recycling can be (and usually Is) dangerous on flnlte-preclslon computers. When
f 1s close to c g , as In most good rejectlon algorithms, the upper portlon of U
(1.e. ( V - l ) / ( 2'-1) in the notatlon of the algorlthm) should not be recycled since
T-1 1s close to 0. The bottom part Is more useful, but thls 1s at the expense of
less readable algorithms. All programs should be set up as follows: a unlform
random varlate should be provided upon lnput, and the output consists of the
returned random varlate and another unlform random varlate. The input and
output random varlates are dependent, but I t should be stressed that the
returned random varlate X and the recycled unlform random variate are
Independent! Another argument agalnst recycllng Is that I t requlres a few multl-
Pllcatlons and/or dlvlslons. Typlcally, the time taken by these operatlons Is
longer than the time needed to generate one good unlform [O,l] random variate.
For all these reasons, we do not pursue the recycllng prlnclple any further.
60 11.3.REJECTION METHOD

3.8. Exercises.
1. Let I and g be easy densltles for whlch we have subprograms for comput-
lng f (z ) and g ( z ) at all z ER d . These densltles can be comblned Into
other densltles In several manners, e.g.
h = c max(f , g )
h = c mh(f ,g)
h =c fi
h = c jag1*

where c 1s a normallzatlon constant (dlfferent In each case) and aE[0,1] 1s a

constant. How would you generate random varlates wlth denslty h ? Glve
the expected tlme complexlty (expected number of lteratlons, comparlsons,
etc.).
2. Decompose the denslty h (z ) = L7i- d s on [-1,1] as follows:

where c =-
7r
&, 3
f (z)=-(l-z2)
4
and g (z)=-,
1
2
and 13: 1 51.Thus,
1s In one of the forms speclfled In exerclse 3.1. Glve a complete algorlthm
h

and analysls for generatlng random varlates wlth denslty h by the general
method of exerclse 3.1.
3. The algorlthm

REPEAT
Generate X with density g .
Generate an exponential random variate E
UNTIL h ( X ) < E
RETURN X

when used wlth a nonnegative functlon h produces a random varlate x

wlth denslty
c g(z) e-h(z) ,

where c 1s a normallzatlon constant. Show thls.

4. How does C , the rejectlon constant, change wlth n (l.e., what 1s Its rate of
lncrease as n -00) when the rejectlon method 1s used on the beta (n ,n )
denslty and the dominating denslty g 1s the unlform denslty on [ O , l ] ?
5. Lux (1879) has generalized the rejectlon method as follows. Let g be a glven
denslty, and k t F be a glven dlstrlbutlon functlon. Furthermore, assume
II.3.REJECTION METHOD 61

thatT 1s a Axed posltlve-valued monotonlcally decreaslng functlon on (0,oo).

Then a random varlate X wlth denslty

Lux's algorithm

REPEAT
Generate a random variate X with density g .
Generate a random variate Y with distribution function F .
UNTIL Y < r ( X )
RETURN x

Also, the probablllty of acceptance of a random couple ( X ,Y ) In Lux's algo-

00

rlthm 1s F ( T (z )) g (z ) dz .
0
6. The followlng denslty on [O,oo)has both an lnflnlte peak at 0 and a heavy
tall:
2
f (XI= (5 >o) .
(l+x )G
Conslder as a posslble candldate for a domlnatlng curve c 0 g 0 where

7rX

where c g 1s a constant dependlng upon 8 only and B>O 1s a design parame-

ter. Prove flrst that lndeed f < c s go. Then show that c g 1s mlnlmal for
6
8=2lI3 and takes the value 7 .
Glve also a descrlptlon of the entlre reJec-
-
7r2 3
tlon algorithm together wlth the values for the expected number of ltera-
tlons, cornparlsons, square root operatlons, multlpllcatlons/dlvlslons, and
asslgnment statements. Repeat the same exerclse when the domlnatlng den-
slty 1s the denslty of the random varlable 8 U 2 / V where B>O 1s a parameter
and u and V are two lld unlform [0,1] random varlates. Prove that In thls
case too we obtaln the same reJectlon constant 6 -.
-1
m3
62 II.3.REJECTION METHOD

7. Optimal rejection algorithms for the normal density. Assume that

normal random varlates are generated by rejectlon from a denslty g o where
0 1 s a deslgn parameter. Dependlng upon the class of go's that 1s consldered,
we may obtaln dlfferent optlmal reJectlon constants. Complete the followlng
table:
8(z Optimal 8 Optimal rejection constant c

Cauchy (e): e 1
e2+ 5 2)
TIT(

Laplace (e): -6e 4 1 z I 1

2

Logistic (e):
eeAZ ? ?
l+e
1 6 ? ?
min(-,T>
40 dX

8. Sibuya's modified rejection method. Slbuya (1962) noted that the

number of unlform random varlates In the rejectlon algorlthrn can be
reduced to one by repeated use of the same unlform random varlate. Hls
algorlthm for generatlng a random varlate wlth denslty f (known not t o
exceed cg for an easy denslty g ) 1s:

Generate a uniform [ O , l ] random variate U .

REPEAT
Generate a random variate x
with density 9 .
UNTIL cg (X) U 5 f (X)
RETURN x

Show the followlng:

(1) The algorlthm Is valld If and only If c =ess SUP (f (x)/g (x)).
(11) If N 1s the number of X ' s needed In Slbuya's algorlthm, and N* 1s
the number of X's needed In the orlglnal rejectlon algorlthm, then

and

(Hlnt: use Jensen's lnequallty.) We conclude from (11) that Slbuya's

method 1s worse than the reJectlon method In terms of number of
requlred lteratlons.
(111) We can have P(N=oo)>o (Just take g = f ,c >i). We can also
have P (N=m)=o,E (N)=m (Just take f (z )=2(1-z ) on [0,1], c =2
II.3.REJECTION METHOD 63

and g (z)=1 on [O,l]). Glve a necessary and sufflclent condltlon for

P ( N =co)=O, and show that thls requlres that c 1s chosen optlmally.
See also Greenwood (1976).
9. There exlsts a second moment analog of Wald's equatlon whlch you should
try t o prove. Let w,,
. . . , and $ 2 0 be as In Theorem 3.5. Assume further
that $( W , ) has mean p and varlance a2<co. Then, for any stopplng rule N
wlth E (N)<m,
N 2
E ( ( C (Wi-pU)) = a2 E ( N ) 1

i=i

See for example Chow and Telcher (1978, pp. 139).

10. Assume that we use proportlonal squeezlng for a denslty f on [0,1] whlch 1s
known to be between 26 (1-z) and 2c (1-z) where 0 5 6 < i < c <co. Then,
we need In every lteratlon a unlform random varlate U and a trlangular ran-
dom varlate x (whlch ln turn can be obtained as mln(U,,U,) where U 1 , U 2
are also unlform [0,1] random varlates). Prove that If U ( l ) s U ( 2 ) are the
order statlstlcs of U,,U, , then

( X , U ) . Thus, uslng thls devlce, we can "save" one unlform

1s dlstrlbuted as
random varlate per lteratlon. Wrlte out the detalls of the corregpondlng pro-
portlonal squeeze algorlthm.
11. Assume that the denslty f has support on [OJ] and that we know that I t 1s
Llpschltz wlth constant C , 1.e.
I f (Y1-f (5) I L c I s-Y I (z ,Y €3
)
+

Clearly, we have f (O)=f (1)=0. Glve an efflclent algorlthm for generatlng

a random varlate wlth denslty f whlch 1s based upon an n-part equl-spaced
partltlon of [0,1] and the use of the proportlonal squeeze method for nearly
llnear densltles (see prevlous exerclse) for generatlng random varlates from
the n lndlvldual pleces. Your algorlthm should be asymptotlcally efflclent,
Le. I t should have E (Nf )=o (1) as n +co where Nf 1s the number of com-
putations of f .
12. .
Random variates with density f(x)=c(l-x2)" ( I x I 51) The famlly of
densltles treated In thls exerclse colncldes wlth the famlly of symmetrlc beta
densltles properly translated and rescaled. For example, when the parameter
a 1s Integer, f 1s the denslty of the medlan of 2 a +1 lld unlform [-1,1] ran-
dom varlates. It 1s also the denslty of the marglnal dlstributlon of a random
vector unlformly dlstrlbuted on the surface of the unlt sphere In R where
d -3
d and a are related by a=- . For the latter reason, we wlll use I t later
2
as an lmportant tool In the generatlon of random vectors that are unlfOrmIY
dlstrlbuted on such spheres. The parameter a must be greater than -1. We
have
64 II.3.REJECTION METHOD

Y
.c =
&r<a +I) ’
and the lnequalltles
-- a22
ce 1-z2 -j
< (a:) 5 ce-az2 ( l a : I 51).
The followlng reJectlon algorlthm wlth squeezlng can be used:

Translated symmetric beta generator by rejection and squeezing

REPEAT
REPEAT
Generate a normal random variate X .
Generate an exponential random variate E.
UNTIL Y l l
x+&, Y+X2
a
Accept +-[l-Y(l+-Y)zO].
E
IF NOT Accept THEN Accept +[aY+E f a log(1-Y)>o].
UNTIL Accept
RETURN x

A. Verlfy that the algorlthm 1s valld.

B. The expected number of normal/exponentlal palrs needed 1s
3
r ( a +$
6 r ( a +I)
. Selected values are
1.329340 ...
i.174982 ...
a=3 -
105
1.119263...

Show that thls number tends t o 1 as a -00 and to 00 as a lo.

C. From Part B we conclude that I t 1s better t o take care of the case
O<a 51 separately, by boundlng as follows: c (l-z2)<f (a: ) h c The .
expected number of lteratlons becomes 2 c . whlch takes the values 3 at -2
a=1 and 1 at a=O. Does thls number vary monotonlcally wlth a ?
How does E ( N J ) vary wlth a ?
11.3.REJECTION METHOD 65

D. Wrlte a generator whlch works for all a >-1. (Thls requlres yet another
solutlon for a In the range (-l,O).)
E. Random varlates from f can also obtalned In other ways. Show that all
of the followlng reclpes are valld:
1
(1) S fl where B 1s beta(-,a +1) and S 1s a random slgn.
2

(11) s p Y+Z
where Y,Z
gamma(a +l,l) random varlates, and S 1s a random slgn.
1
are lndependent gamma(-,i)
2
and

(111) 2B-1 where B 1s a beta(a + l , a +1) random varlate.

13. Conslder the squeeze algorlthm of sectlon 3.6 whlch uses the lnequallty
- for the reJectlon-based generator, and the lnequalltles h , < f < h ,
f <cg
for the qulck acceptance and reJectlon steps. Even If h , 1s not necessarlly
posltlve, and h , 1s not necessarlly smaller than cg , show that we always
have
E (Nf = S(mln(h,,cg )-max(h ,,o)) L !(h,-h J
where Nf 1s the number of evaluatlons of f .
14. A universal generator a la Letac. Let f be any denslty on [O,ll, and
assume that the cumulatlve mass functlon M ( t )= f (a:) dz 1s known.
f 2t
Conslder the followlng algorlthm:

Generate a random integer z where P (z=i)=M(i)-M(i

fl).
REPEAT
Generate (x,v)uniformly in [0,lIz
UNTIL Z + v s f (X)
RETURN X

Show that the algorlthm 1s valld (relate I t to the reJectlon method). Relate
the expected number of X ’ s generated before haltlng to I I f I I co, the
.
essentlal supremum of f Among other thlngs, conclude that the expected
tlme Is 00 for every unbounded denslty. Compare the expected number of
x’s wlth Letac’s lower bound. Show also that if lnverslon by sequentlal
search 1s used for generatlng 2 , then the expected number of lteratlons In
the search before haltlng 1s flnlte If and only If J f 2 < ~ . A Anal note: usu-
ally, one does not have a cumulatlve mass functlon for an arbltrary denslty
f.
66 11.4.DISCRETE MIXTURES

4. DECOMPOSITION AS DISCRETE MIXTURES.

4.1. Definition.
If our target denslty f can be decomposed lnto a dlscrete mlxture
00

f ($1 = CPifi(”)
1 =1

where the f ’s are glven densltles and the p i ’s form a probablllty vector (l.e.,
p i 2 0 for all t‘ and Cpi =l), then random varlates can be obtained as follows:
i

The composition method.

Generate a random integer Z with probability vector p 1, . . . , p i ,... (Le. P ( Z = i ) = p i ) .

Generate a random variate x
with density f .
RETURN x

Thls algorlthm 1s Incomplete, because I t does not speclfy Just how Z and X are
generated. Every tlme we use the general form of the algorlthm, we wlll say that
the composltlon method 1s used.
We wlll show In thls sectlon how the decomposltlon method can be appiled
In the deslgn of good generators, but we wlll not at thls stage address the prob-
lem of the generatlon of the dlscrete random varlate Z . Rather, we are lnterested
In the decomposltlon Itself. It should be noted however that in many, If not most,
practlcal sltuatlons, we have a Anlte mlxture wlth K components.

4.2. Decomposition into simple components.

Very often, we wlll decompose the graph of f lnto a bunch of very slmple
structures such as rectangles and Wangles, malnly because random varlates wlth
rectangular-shaped or trlangular-shaped densltles are so easy t o generate (by
llnear comblnatlons of one or two unlform [ O J ] random varlates). Thls decompo-
sltlon 1s Anlte If f 1s plecewlse llnear wlth a Anlte number of ‘pleces (thls forces
f to have compact support). In general, one wlll decompose f as follows:
K-2
f ($ = Pi f i ($1 + P K - 1 f K - I ( ” +PK f K ($1
i=1

where f~ 1s a tall denslty ( I t 1s zero on a central lnterval [ a , b I), p~ 1s usually

very small, and all other f i ’ s vanlsh outslde the central lnterval [ a , b 1. The
structure of f 1, . . . , f ~ 1s -slmple,
~ e.g. rectangular. After havlng plcked the
rectangles In such a way that the correspondlng p i ’ s add up to nearly 1, we
II.4.DISCRETE MIXTURES 67

collect the leftover plece In p ~ - ~ f l ~Thls - ~ last

. plece 1s often strangely shaped,
and random varlates from I t are generated by the rejectlon method. The polnt 1s
that p ~ and - p~
~ are so small that we do not have to generate random varlates
wlth thls denslty very often. Most of the ttme, 1.e. wlth probablllty
p 1+ . . . + p ~ - ~I t , sufflces to generate one or two uiilform [O,l]random varlates
and to shlft or rescale them. Thls technlque wlll be called the jigsaw puzzle
method, a term coined by Marsaglla. The careful decomposltlon requlres some
reflned analysls, and 1s usually only worth the trouble for frequently used Axed
densltles such as the normal denslty. We refer to the sectlon on normal varlate
generatlon for several appllcatlons of thls sort of decomposltlon. Occaslonally, I t
can be applled to famllles of dlstrlbutlons (such as the beta and gamma famllles),
but the problem 1s that the decomposltlon ltself 1s a functlon of the parameter(s)
of the famlly. Thls wlll be lllustrated for the beta famlly (see sectlon IX.4).
T o glve the readers a flavor of the sort of work that 1s lnvolved, we wlll try
t o decompose the normal denslty lnto a rectangle and one resldual plece: the rec-
tangle wlll be called p f l ( z ) , and the resldual plece p 2 f 2 ( z ) . It 1s clear that p
should be as large as possible. But slnce p f l(z)s
f (z), the largest p must
satisfy

1
Thus, wlth f l(a:)=-8,
2
Ix I5 8 where 8 1s the wldth of the centered rectangle,
we see that at best we can set
--2 2 e2
28e =I 2- 8 e-T
l t;L- & 6
The functlon p 1 1s maxlmal (as a functlon of 8 ) when 8=1, and the correspond-
lng value 1s &. Of course, thls welght 1s not close to 1, and the present
decomposltlon seems hardly useful. The work lnvolved when we decompose In
terms of several rectangles and trlangles 1s baslcally not dlfferent from the short
analysls done here.

4.3. Partitions into intervals.

Many algorlthms are based on the followlng prlnclple: partltlon the real llne
Into lntervals A . . . , A , , and decompose f as
K
f ( 4= fI.( IA,W '

t =I

If we can generate random varlates from the restrlcted densltles f I A , / p i (where

Pi = f ), then the decomposltlon method 1s appllcable. The advantages offered
A,
68 11.4 .DISCRETE MIXTURES

by partltlons lnto lntervals cannot be denled: the decomposltlon 1s so simple that

I t can be rnechanlzed and used for huge classes of densltles (In t h a t case, there
are usually very many Intervals); troublespots on the real llne such as lnflnlte
talls or unbounded peaks can be convenlently Isolated; and most lmportantly, t h e
decomposltlon 1s easlly understood by the general user.
In some cases, random varlates from the component densltles are generated
by the reJectlon method based on the lnequalltles
f ( x ) 5 h i ( $ ) , $ € A i ,1LisK .
Here the hi's are glven domlnatlng curves. There are two subtly dlfferent
methods for generatlng random varlates wlth denslty f , glven below. One of
these needs the constants p i = Jr
, and the other one requlres the constants
A,
Qi - Jhi. Note that the q i ' s are nearly always known because the hi's are
A,
chosen by the user. The p i ' s are usually known when the dlstrlbutlon functlon
for F 1s easy to compute at arbltrary polnts.

The composition method.

Generate a discrete random variate with probability vector p l , . . . , p~ on

(1, . . . ,K } .
REPEAT
Generate a random variate x with density hi / q i on Ai ,

Generate an independent uniform [0,1]random variate u.

UNTIL uhi ( x ) l f(x)
RETURN x
II.4.DISCRETE MIXTURES 69

The modified composition method.

REPEAT
Generate a discrete random variate with probability vector proportional to
P I , . * , PK on (1, . . ? K ) .
1

Generate a random variate x with density hi /Pi on A i .

Generate a uniform [0,1]
random variate U.
UNTIL U h i ( X ) < f (X)
RETURN x

In the second algorlthm we use the rejectlon method wlth as domlnatlng curve
h IIA,+ . . +h, IA,, and use the composltlon method for random varlates from
*

the domlnatlng denslty. In contrast, the Arst algorlthm uses true decomposltlon.
After havlng selected a component wlth the correct probablllty we then use the
rejectlon method. A brlef comparison of both algorlthms 1s In order here. Thls
can be done In terms of four quantltles: Nz,N,,Nh and Nh , where N 1s the
number of random varlates requlred of the type speclfled by the lndex wlth the
K
understandlng that Nh refers to h i , 1.e. I t 1s the total number of random vari-
i=1
ates needed from any one of the K domlnatlng densltles.

I Theorem 4.1.K
I Let q =
i=1
q i , and let N be the number of Iterations In the second algo-
rlthm. For the second algorlthm we have NU=Nz =Nh = N , and N 1s geometrl-
1
cally dlstrlbuted wlth parameter -. In partlcular,
Q
* E ( N ) = q ; E ( N 2 )= 2q2-q .
For the flrst algorlthm, we have N z =l. Also, N u =Nh satlsfy

I Flnally, for both algorlthms, E ( N h , )= qi .

70 II.4.DISCRETE MIXTURES

Proof of Theorem 4.1.

The statement for the second algorlthm 1s obvlous when we note that t h e
reJectlon constant Is equal to the area q under the domlnatlng curve (the sum of
the hi's In thls case). For the flrst algorlthm, we observe that glven the value of
2 , N u Is geometrlcally dlstrlbuted wlth parameter p z / q z . F r o m the propertles
of the geometrlc dlstrlbutlon, we then conclude the followlng:

Qi Qi

T o show that the last expresslon Is always greater or equal to 2 q 2 -q we use the
C auchy-S chw arz lne qual1ty :

Flnally, we conslder E (Nh ). For the flrst algorlthm, Its expected value Is
Qi
pi(-) = q i . For the second algorlthm, we employ Wald's equallty after notlng
Pi
N

In standard clrcumstances, q 1s close to 1, and dlscrete random varlate gen-

erators are ultra efflclent. Thus, N, Is not a great factor. For all the other quan-
tltles lnvolved In the comparlson, the expected values are equal. But when we
examlne the hlgher moments of the dlstrlbutlons, we notlce a strlklng difference,
because the second method has In all cases a smaller second moment. In fact, the
dlfference can be substantlal when for some i , the ratlo qi / p i Is large. If we take
qi =pi for i 2 2 and q l=q -(l-p 1 ) , then for the flrst method,
2(q -1+p 1
E ( N U 2 )= +2(1-P 1) -q = (2q 2-q ) +2( q - 1 ) 2 ( - - 1 )
P I P1

The dlfference between the two second moments In thls example 1s

1
2(q -1)2(--1) . Thus, lsolating a small probablllty plece in the decomposltlon
P l
method and uslng a poor reJectlon rate for that partlcular plece 1s dangerous. In
such sltuatlons, one Is better off uslng a global reJectlon method as suggested In
the second algorlthm.
II.4.DISCRETE MIXTURES 71

4.4. The waiting time method for asymmetric mixtures.

In large slmulatlons, one needs lid random varlates XI, . . . , X,, ,.... If these
random varlates are generated by the composition method, then for every ran-
dom varlate generated we need one dlscrete random varlate 2' for selectlng a
component. When f 1s decomposed lnto a maln component p I f ( p is close to
1) and a small Component p 2 f 2, then most of these selectlons wlll choose the
flmt component. In those cases, i t 1s useful to generate the tlmes of occurrence of
selectlon of the second component Instead. If the second component 1s selected at
tlmes T 1 , T 2...,
, then I t 1s not dlmcult to see that T 1 , T 2 - T ,...
, are lid geometrlc
random varlables with parameter p 2, 1.e.
P(T,=i)= (1-p2)i-1 (i 21).

1- 1
p2

A random varlate TIcan be generated as lo$ 2) where E 1s an exponen-

tlal random varlate. Of course, we need to keep track of these tlmes as we go

along, occaslonally generatlng a new tlme. These tlmes need to be stored locally
In subprograms for otherwlse we need to pass them as parameters. In some cases,
the overhead assoclated wlth passlng an extra parameter 1s comparable to the
tlme needed to generate a unlform random varlate. Thus, one should carefully
look at how the large slmulatlon can be organized before uslng the geometrlc
waltlng tlmes.

4.5. Polynomial densities on [0,1].

In thls sectlon, we conslder densltles of the form
K
f (x)= cjx i (oLx<1>
i =O

where the c i ' s are constants and K Is a posltlve Integer. Densltles wlth polyno-
mlal forms are Important further on a s bulldlng blocks for constructlng plecewlse
polynomlal approxlmatlons of more general densltles. If K is 0 or 1, we have the
unlform and trlangular densltles, and random variate generatlon 1s no problem.
There 1s also no problem when the ti's are all nonnegatlve. To see thls, we
observe that the dlstrlbutlon functlon F 1s a mixture of the form

K+l
where of course -=i. Slnce x * 1s the dlstrlbutlon functlon of the max-
:=I *
\mum of i lld unlform [O,l]random varlables, we can proceed as follows:
72 II.4.DISCRETE MIXTURES

C i -1
Generate a discrete random variate Z where P (2=i)=- , i<i <K+i.
a
RETURN X where X is generated as max(U,, . . . , uz) and the Q.'s are iid uniform [0,1]
random variates.

We have a nontrlvlal problem on our hands when one or more of the c i ' s are
negative. The solution glven here is due to Ahrens and Dleter (1974),and can be
applled whenever c,+ ci 2 0 . They decompose f as follows: let A be the
i : c ,< O
collectlon of lntegers In (0,. . . , K } for which c i LO,and let B the collectlon of
lndlces In (0, . . . , K } for whlch ci <O. Then, we have
K
f (a:)= CCiZ'
i =o

where p o = c ci (whlch is 20 by assumptlon). If we set p i equal to

i EB
- and to - i c i /(i +1) for i EB , then p , , p 1, . . . , p~ Is a
c i /(i +I) for i EA ,i >1,
probablllty vector, and we have thus decomposed f as a Anlte mlxture. Let us
brlefly mentlon how random variate generatlon for the component denslties can
be done.

Lemma 4.1.
Let U 1, U 2 ...
, be lld uniform [0,1]random variables.
-1
A. For a >1, U l a U, has density
a
I
-
a -1
(1-za-1) ( 0 5 s 51)

B. Let L be the lndex of the flrst U; not equal to max( U,, . . . , U, ) for n 2 2 .
Then U, has denslty
n
-
n -1
(1-z"-'> (05a:51).

C. The density of max(U,, . . . , U,) is n x n - l (05. 51).

11.4 .DISCRETE MIXTURES 73

Proof of Lemma 4.1.

Part C 1s trlvlally,true. Part A 1s a good exerclse on transformatlons of ran-
dom varlables. Part B has a partlcularly elegant short proof. The denslty of a
randomly chosen 1s 1 (all densltles are understood to be on (O,l]). Thus, when
f 1s the denslty of U,,we must have
-jn -1 1
(x)+;nxn-' =1 .
n
1
Thls uses the fact that wlth probablllty -, the randomly chosen Vi 1s the maxl-
n
mal v i , and that wlth the compllrnentary probablllty, the randomly chosen Vi 1s
dlstrlbuted as u,.

We are now In a posltlon to glve more detalls of the polynomlal density al&-
rlthm of Ahrens and Dleter.

Polynomial density algorithm of Ahrens and Dieter

[SET-UP]
Compute the probability vector p , , p 1, . . . , p~ from c o, . . . , CK according to the formu-
las given above. For each i E { O , l , . . . , K }, store the membership of i ( i EA if ci 20 and
i EB otherwise).
[GENERATOR]
Generate a discrete random variate with probability vector p , , p 1, . . . , p~ .
IF ZEA
- 1
THEN RETURN x t u z + ' ( o r X+max(Ul, . . . , Uz+,) where U,ul,
... are iid uni-
form (0,1]random variates).
- 1

ELSE RETURN X + U l z + ' U2(or X t u , where L is the ui with the lowest index
not equal to max(U,, . . . , UZ+~)).
74 II.4 .DISCRETE MIXTURES

4.6. Mixtures with negative coefficients.

Assume that the denslty f (a: ) can be wrltten as
03

f (a:) = CPifiW 9

i=1

where the f i ’s are densltles, but the pi’s are real numbers summlng t o one. A
general algorlthm for these densltles was glven by Blgnaml and de Mattels (1971).
It uses t h e fact that If p i 1s decomposed lnto Its posltlve and negatlve parts,
Pi =Pi +-Pi then -t

03

f (a:) L !J(a:) = EPi+fiW *

a =1

Then, the followlng rejectlon algorlthm can be used:

Negative mixture algorithm of Bignami and de Matteis

REPEAT
00 00
Generate a random variate x with density pi + f i/ pi +.
i=1 i-1
Generate a uniform [0,1]random variate u.

i- 1 1-1

RETURN x

03
The rejectlon constant here 1s s g = pi+. The algorlthm 1s thus not
i=l
valld when thls constant 1s 00. One should observe that for thls algorlthm, the
rejectlon constant 1s probably not a good measure of the expected tlme taken by
I t . Thls 1s due to the fact that the tlme needed to verlfy the acceptance condltlon
can be very large. For flnlte mlxtures, or mlxtures that are such that for every
a : , only a Anlte number of f (a: )’s are nonzero, we are In goo(I shape. In all cases,
I t 1s often posslble t o accept or reject after havlng computet1 .lust a few terms In
the serles, provlded that we have good analytical estimates of the tall sums of the
serles. Slnce thls Is the maln ldea of the serles method of sectlon IV.5, I t will not
be pursued here any further.

Example 4.1.
3
The denslty f (a:)=-(1-z2),
4
I x 151, can . be wrltten as
6 1 2 x2
f (a: )==y(yI[-1,1j(a:
>>--(--lr-l,ll(a:
4 6 )). The algorlthm glven above 1s then
II.4.DISCRETE MIXTURES 75

entirely equivalent to ordlnary rejectlon from a unlform denslty, which In thls

3
case has a rejection constant of -:
2

REPEAT
Generate a uniform [-1,1] random variate X,
Generate a uniform (0,1] random variate u.
UNTIL u 5 1-x2
RETURNX

5. THE ACCEPTANCE-COMPLEMENT METHOD.

5.1. Definition.
Let f be a given denslty on which can be decomposed into a sum of
two nonnegative functions:
f (5 1= f >+f 2(2 > *

Assume furthermore that there exists an easy denslty g such that f L l g . Then
the followlng algorithm can be used to generate a random varlate X wlth density
f:

The acceptance-complement method

Generate a random variate x with density g

Generate a uniform [0,1] random variate U .

THEN Generate a random variate x with density -

f 2
where p = f j
2.
P
RETURN x

Thls, the acceptance-complement method, was flrst proposed by Kronmal and

Peterson (1981,1984). It generalizes the composltlon method a s can be seen if we
take f l = f I A , g =f ,/If,
and f 2 = f I A C where A is an arbltrary set of R d
 I

76 11.5 .ACCEPTANCECOMPLEMENT METHOD

and A e1s Its complement. It 1s competltlve 14 three condltlons are met:

(1) g 1s an easy density.
(11) f 2/p 1s an easy denslty when p 1s not small (when p 1s small, thls
does not matter much).
(111) f J g 1s not dlfflcult to evaluate.
As wlth the composltlon method, the algorlthm glven above 1s more a prlnclple
than a detalled reclpe. When we compare I t wlth the rejectlon method, we notice
that lnstead of one deslgn varlable (a domlnatlng denslty) we And two deslgn
varlables, f and g . Moreover, there 1s no rejectlon involved at all, although
very often, I t turns out that a random varlate from -
f 2
Is generated by the
P
rejectlon method.
Let us flrst show that thls method 1s valld. For thls purpose, we need only
show that for all Bore1 sets B d , the random varlate generated by the algo-
rlthm (whlch wlll be denoted here by X ) satlsfles P (XEB )=SF (x ) dx . To
B
avold confuslon wlth too many x's,we wlll use Y for the random varlate wlth
.
density g Thus,

= .f/ ( x ) dx .
B
In general, we galn If we can RETURN the flrst X generated in the algo-
rlthm. Thus, I t seems that we should try to maxlmlze Its probablllty of accep-
tance,

subject of course to the constralnt f l l g where g 1s an easy denslty. Thus, good

algorlthms have g "almost" equal t o f .
There 1s a VlSUal explanatlon of the method related to that of the rejectlon
method. What 1s important here 1s that the areas under the graphs of g-f and
f are equal. In the next sectlon, we wlll glve a slmpllfled verslon of the
acceptance-complement algorlthm developed lndependently by Ahrens and Dleter
(1981,1983). Examples and detalls are glven In the remalnlng sectlons and In
some of the exerclses.
11.5 .ACCEPTANCECOMPLEMENT METHOD 77

5.2. Simple acceptance-complement methods.

Ahrens and Dleter (1981,1983)and Deak (1981) consldered the speclal case
deflned by an arbltrary denslty g on R and the followlng decomposltlon:
f (z 1 = f I($ >+f 2(a: 1 ;
f 1 = m w (a:>,s(x 11 (note :f 15s 1 ;
f 2 ( 2 1 = (f (z 1-s (z )I+ *

We can now rewrlte the acceptance-complement algorlthm qulte simply as fol-

lows:

Simple acceptance-complement method of Ahrens and Dieter

Generate a random variate x with density g

Generate a uniform [0,1]random variate U.

THEN Generate a random variate X with density ( f -g ) + / p where p = 1 (f -g ) .

I >P
RETURN x

Deak (1981) calls thls the economical method. Usually, g 1s an easy denslty
.
close to f It should be obvlous that generatlon from the leftover denslty
( f -g ) + / p can be problematlc. If there Is some freedom In the deslgn (Le. In the
.
choke of g ), we should try to mlnlmlze p Thls slmple acceptance-complement
method has been used for generatlng gamma and t varlates (see Ahrens and
Dleter (1981,1983)and Stadlober (1981) respectlvely). One of the maln technical
obstacles encountered (and overcome) by these authors was the determlnatlon of
the set on whlch f ( a : ) > g (5).If we have two densltles that are very close, we
must first verlfy where they cross. Often thls leads to compllcated equatlons
whose solutlons can only be determlned numerlcally. These problems can be
sldestepped by exploltlng the added flexlblllty of the general acceptance-
complement method.
78 11.5.ACCEPTANCECOMPLEMENT METHOD

5.3. Acceleration by avoiding the ratio computation.

The time-consumlng ratlo evaluation -

f l
In the acceptance condltlon can be
9
avoided some of the time If we know two easy-to-compute functlons h and h*
wlth the property that

The IF step In the acceptance-complement algorlthm can be replaced in those

cases by

Squeeze step in acceptance-complement method

IF U > h ( X )
THENIF u z h * ( X )

THEN Generate a random variate x with density -

P
f 2
where p = J j 2.

THEN Generate a random variate x with density -

P
f 2
where p = J f 2.

RETURN x

A slmllar but more spectacular acceleration 1s posslble for the Ahrens-Dleter

algorlthm If one can qulckly determine whether a point belongs to A , where A 1s
a subset of f > g . In partlcular, one will And that the set on which f > g often
1s an interval, In whlch case this acceleratlon 1s easy to apply.

Accelerated version of the Ahrens-Dieter algorithm

Generate a random variate x with density g .

IF X E A
THEN
Generate a uniform [0,1]random variate u.

THEN Generate a random variate x with density (f -9 ) + / p .

RETURN x

With probablllty P ( X E A ), no unlform random variate 1s generated. Thus, what

one should try to do 1s to choose g such that P ( X E A ) 1s maxlmal. Thls in turn
11.5.ACCEPTANCE-COMPLEMENTMETHOD 79

suggests chooslng g such that

1s large.

5.4. An example: nearly flat densities on [0,1].

We wlll develop a unlversal generator for all densltles f on [-l,l] whlch
satlsfy the followlng property: sup f (a: )-lnf f (z )L-.1 Because we always have
2 2 2
1
OClnf
- f (z)<-Ssup f (z), we see that sup f (a;)Sl.We wlll apply the
2 - 2 2 2
acceptance-complement method here wlth as slmple a decomposltlon as posslble,
for example
1
s(z)=-
2
( I S 151);
1
f 1(x 1 = f (X Hf max-2) (f max=SUP2 f (5 1) ;

The condltlon lmposed on the class of densltles follows from the fact that we
must ask that f be nonnegatlve. The algorlthm now becomes:

Acceptance-complement method for nearly flat densities

Generate a uniform [-1,1] random variate X.

Generate a uniform [0,1]
random variate U.
E-u >2(f (X1-f mm+;)

THEN Generate a uniform [-1,1] random variate x

RETURN x

To thls, we could add a squeeze step, because we can exlt whenever

1
u _<2(lnf f (a:)-! max+-), and the probablllty of thls fast exlt Increases wlth the
2 2
.
"flatness" of f It Is 1 when f 1s the unlform denslty.
A comparlson wlth the reJectlon method Is In order here. Flrst we observe
t h a t because we plcked g and f both unlform, we need only unlform random
varlates. The number N of such unlform random varlates used up In the algo-
rlthm 1s elther 2 or 3. We have
80 II.5.ACCEPTANCECOMPLEMENT METHOD

where x stands for a unlform [-1,1] random varlate. Thus,

-1 -

= 2+2f max-l= l + 2 f max

In addltlon, If no squeeze step 1s used, we requlre exactly one computatlon of f

per varlate. The obvlous rejectlon algorlthm for thls example 1s

Rejection algorithm for nearly flat densities

REPEAT
Generate a uniform [-1,1]random variate x
Generate a uniform [0,1]random variate u .
UwXL Uf m a x l f (XI
RETURN X

Here too we could lnsert a squeeze step (ufmax<lnf f (z)).

2
Wlthout It, the
expected number of unlform random varlates needed 1s 2 tlmes the expected
number or lteratlons, 1.e. 4f In addltlon, the expected number of computa-
tlons or 1 1s 2 j max. On both counts, thls 1s strlctly worse than the acceptance-
complement method.
We have thus establlshed that for some falrly general classes of densltles, we
have a strlct lmprovement over the reJectlon algorlthm. The unlversallty of the
algorlthms depends upon the knowledge of the lnflmum and supremum of f .
Thls 1s satlsfled for example If we know that f 1s symrnetrlc unlmodal In whlch
case the lnflmum 1s 1(1) and the supremum 1s f (0).
The algorlthrn W e n above can be applled t o the maln body of most sym-
metric unlmodal densltles such as the normal and Cauchy densltles. For the
truncated Cauchy densify

2 1
our condltlons are satlsfled because f max=- and the lnflmum of f 1s -, the
7r 7r
1
dlfference belng smaller than In thls case, the expected number of unlform
2
4
random varlates needed 1s I+-. Next, note that If we can generate a random
lr
varlate X wlth denslty then a standard Cauchy random varlate can be
obtalned by exploltlng the Property that the random varlate Y deflned by
11.5 .ACCEPTANCE-COMPLEMENT METHOD 81

1
X wlth probablllty -
2
Y = I 1 - wlth probablllty -1
X 2

IS Cauchy dlstrlbuted. For thls, we need an extra coln fllp. Usually, extra coln
Alps are generated by borrowlng a random blt from U. For example, In the
unlversal algorlthm shown above, we could have started from a unlform [-1,1]
I 1
random varlate u , and used u In the acceptance condltlon. Slnce slgn(U) is
1
lndependent of u I I, slgn(U) can be used to replace X by -, so that the
X
returned random varlate has the standard Cauchy denslty. The Cauchy generator
thus obtalned was flrst developed by Kronmal and Peterson (1981).
We were forced by technlcal conslderatlons to llmlt the densltles somewhat.
The rejectlon method can be used on all bounded densltles wlth compact support.
Thls typlfles the sltuatlon In general. ,In the acceptance-complement method, once
we choose the general form of g and f 2, we loose In terms of unlversallty. For
example, If both f and g are constant on [-1,1], then f = fl+f 2 5 g +f2<1.
Thus, no denslty f wlth a peak hlgher than 1 can be treated by the method. If
unlversallty 1s a prlme concern, then the reJectlon method has llttle competltlon.

5.5. Exercises.
1. Kronmal and Peterson (1981) developed yet another Cauchy generator based
upon the acceptance-complement method. It 1s based upon the followlng
decomposltlon of the truncated Cauchy denslty f (see text for the
deflnltlon) lnto f l+f 2:

We have:
82 1I.S.ACCEPTANCECOMPLEMENT METHOD

A Cauchy generator of Kronmal and Peterson

Generate iid uniform [-1,1] random variates X and U ,

2
IF IUI>--.
A
THEN IF I u 150.7225
4 2
THENIF
7r(1+X2)
--(I-
ff
I x I)
THEN Generate iid uniform [-1,1] random variates X,U.
XtlXl-lUl.
ELSE Generate a uniform [-1,1] random variate u.
x t IX 1 - 1 u I .
IF us0
THEN RETURN x
ELSE RETURN -
1
X

The Arst two IF’S are not requlred for the algorlthm t o be correct: they
correspond t o squeeze steps. Verlfy that the algorlthm generates standard
Cauchy random varlates. Prove also that the acceleratlon steps are valld.
The constant 0.7225 1s but an approxlmatlon of an lrratlonal number, whlch i
should be determlned.