Bootstrapping time series models

Hongyi Li & G.S. Maddala'

1 - INTRODUCTION
 1. Introduction

The purposes of this paper are:

1. To provide a survey of bootstrap procedures
applied to time series econometric models
Econometrics
the application of statistical and mathematical
theories in economics for the purpose of testing
hypotheses and forecasting future trends

2. To present some guidelines for empirical

researchers in this area
 1. Introduction
• Most of the inferential procedures available in the analysis of time
series data are asymptotic
• Although analytic small sample results are available in a few cases,
there is currently, no widely applicable and easily accessible method
that can be used to make small sample inferences. Methods like
Edgeworth expansions involve a lot of algebra and are also applicable
in very special cases.
• The bootstrap technique introduced by Efron (1979) could possibly be
a potential alternative in estimation and inference from time series
models in finite samples. However, in time series regressions, the
standard bootstrap resampling method designed for independent and
identically distributed (IID) errors is not applicable because in most
situations the assumption of IID errors is violated.
2 - GENERAL GUIDELINES FOR
USING THE BOOTSTRAP APPROACH
2. General guidelines for using the bootstrap approach

The basic bootstrap

approach consists of Time series models
drawing repeated samples
(with replacement)

the simplest one that is valid for

IID assumption is not satisfied
IID observations

The method needs to be

modified
2. General guidelines for using the bootstrap approach

• Estimating Standard Errors

Using BS distribution to Estimate SE Better

Estimation of SE
Small Sample Size
Using asymptotic distribution to
Estimate SE

The “Small Sample Size” distribution

BUT Must be NORMAL
2. General guidelines for using the bootstrap approach

Using BS distribution to Estimate SE

Even if Using asymptotic distribution to

Same

Estimate SE

• Confidence Interval statements

BS distribution is skewed
Using BS distribution to Estimate CI +
Different CI
Using asymptotic distribution to
Estimate CI
2. General guidelines for using the bootstrap approach

Methods to Construct Confidence intervals Coverage error

෡
1. Using the asymptotic distribution of 𝜽

2. The percentile method O(n-1/2)

3. Bias-corrected (BC) percentile interval

4. Accelerated bias-corrected (BCα) percentile interval

O(n-1)
5. The percentile-t (or bootstrap-t) method

6. Beran’s pivotal method (iterative, BS ) O(n-3)

Coverage Error
The difference between the actual
coverage and nominal coverage
2. General guidelines for using the bootstrap approach
Methods to Construct Confidence intervals

1. ෡
Use the asymptotic distribution of 𝜽

The two-sided confidence interval is: Symmetric Interval

෡ ± zα·SE(𝜽)
𝜽 ෡ Coverage error = O(n-1/2)

zα - the (100 - α) percentile from the ෡ - the asymptotic

SE(𝜽)
standard normal distribution ෡
standard error of 𝜽
2. General guidelines for using the bootstrap approach
Methods to Construct Confidence intervals

2. The percentile method

෡*
Use the bootstrap distribution of 𝜽

The two-sided (100 - 2·α) confidence Two-sided equal-tailed interval

(often non-symmetric)
interval for θ is

The nominal coverage of this

෡ – z*1-α , 𝜽
(𝜽 ෡ + z*α) interval is (100 - 2α)

Coverage error = O(n-1/2)

z*α - the 100α percentile of the
෡∗ - 𝜽
distribution of 𝜽 ෡
2. General guidelines for using the bootstrap approach
Hypothesis Testing

𝑯𝟎 : 𝜽 = 𝜽𝟎

𝑯𝟏 : 𝜽 ≠ 𝜽𝟎

Reminder:
𝜽𝟎 - The parameter’s value under the null hypothesis

෡
𝜽 - An estimation of the parameter’s value using the original samples

෡∗
𝜽 - An estimation of the parameter’s value using the BS resamples
2. General guidelines for using the bootstrap approach
Hypothesis Testing
• Two important issues concerning hypothesis testing using
bootstrap methods relate to the questions about:
a) What test statistic to bootstrap?
b) How to generate the bootstrap samples?
2. General guidelines for using the bootstrap approach
Hypothesis Testing
• Two important issues concerning hypothesis testing using
bootstrap methods relate to the questions about:
a) What test statistic to bootstrap?
b) How to generate the bootstrap samples?

Use the BS distribution of: ෡∗ − 𝜽

𝜽 ෡ BUT NOT: ෡ ∗ − 𝜽𝟎
𝜽

෡∗− 𝜽
𝜽 ෡
ෝ
𝝈
෡∗− 𝜽
𝜽 ෡
Use the properly studentized statistic: BUT NOT:
ෝ∗
𝝈
෡∗− 𝜽
𝜽 ෡
ෝ∗ - the estimate of 𝝈
𝝈 ෝ from the BS sample
2. General guidelines for using the bootstrap approach
Hypothesis Testing
• Two important issues concerning hypothesis testing using
bootstrap methods relate to the questions about:
a) What test statistic to bootstrap?
b) How to generate the bootstrap samples?

The Model we’ll discuss is simple regression:

𝒚 = 𝜷𝒙 + 𝜺, 𝜺~𝒊𝒊𝒅(𝟎, 𝝈𝟐 )
෡ 𝝈
𝜷, ෝ are OLS estimators for
𝜺ො - the OLS residuals
𝜺∗ − the BS residuals, obtained by resampling 𝜺ො
2. General guidelines for using the bootstrap approach
Methods of BS Samples Generation

Consider two sampling schemes for the generation of the bootstrap samples:

෡ + 𝜺∗
𝑺𝟏 : 𝒚∗ = 𝜷𝒙

𝑺𝟐 : 𝒚∗ = 𝜷𝟎 𝒙 + 𝜺∗
Both use 𝜀 ∗

For each sampling scheme, consider two test t-statistics:

෡ = 𝜷
𝑻𝟏 : 𝑻 𝜷 ෡∗ − 𝜷
෡ Τ𝝈
ෝ∗

෡ ∗ − 𝜷𝟎 Τ𝝈
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷 ෝ∗
2. General guidelines for using the bootstrap approach

4 versions of the t-statistic can be defined:

෡ = 𝜷
𝑻𝟏 : 𝑻 𝜷 ෡∗ − 𝜷
෡ Τ𝝈
ෝ∗
෡ + 𝜺∗
𝑺𝟏 : 𝒚∗ = 𝜷𝒙
෡ ∗ − 𝜷𝟎 Τ𝝈
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷 ෝ∗

෡ = 𝜷
𝑻𝟏 : 𝑻 𝜷 ෡∗ − 𝜷
෡ Τ𝝈
ෝ∗
𝑺𝟐 : 𝒚∗ = 𝜷𝟎 𝒙 + 𝜺∗
෡ ∗ − 𝜷𝟎 Τ𝝈
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷 ෝ∗
2. General guidelines for using the bootstrap approach

4 versions of the t-statistic can be defined:

෡ = 𝜷
𝑻𝟏 : 𝑻 𝜷 ෡∗ − 𝜷
෡ Τ𝝈
ෝ∗
෡ + 𝜺∗
𝑺𝟏 : 𝒚∗ = 𝜷𝒙
෡ ∗ − 𝜷𝟎 Τ𝝈
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷 ෝ∗
Hall & Wilson

෡ = 𝜷
𝑻𝟏 : 𝑻 𝜷 ෡∗ − 𝜷
෡ Τ𝝈
ෝ∗
𝑺𝟐 : 𝒚∗ = 𝜷𝟎 𝒙 + 𝜺∗
෡ ∗ − 𝜷𝟎 Τ𝝈
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷 ෝ∗
Giersbergen & Kiviet
Based on Monte-Carlo study of an AR(1) model
2. General guidelines for using the bootstrap approach

4 versions of the t-statistic can be defined:

෡ = 𝜷
𝑻𝟏 : 𝑻 𝜷 ෡∗ − 𝜷
෡ Τ𝝈
ෝ∗
෡ + 𝜺∗
𝑺𝟏 : 𝒚∗ = 𝜷𝒙
෡ ∗ − 𝜷𝟎 Τ𝝈
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷 ෝ∗

෡ = 𝜷
𝑻𝟏 : 𝑻 𝜷 ෡∗ − 𝜷
෡ Τ𝝈
ෝ∗
𝑺𝟐 : 𝒚∗ = 𝜷𝟎 𝒙 + 𝜺∗
෡ ∗ − 𝜷𝟎 Τ𝝈
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷 ෝ∗
Giersbergen & Kiviet
Based on Monte-Carlo study of an AR(1) model
2. General guidelines for using the bootstrap approach

Giersbergen & Kiviet

Based on Monte-Carlo study of an AR(1) model

෡ = 𝜷
𝑻𝟏 : 𝑻 𝜷 ෡∗ − 𝜷
෡ Τ𝝈
ෝ∗
෡ + 𝜺∗
𝑺𝟏 : 𝒚∗ = 𝜷𝒙

𝑺𝟐 : 𝒚∗ = 𝜷𝟎 𝒙 + 𝜺∗
෡ ∗ − 𝜷𝟎 Τ𝝈
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷 ෝ∗

Equivalent in non-dynamic models

In dynamic models 𝑺𝟐 𝑻𝟐 is better

3 - STRUCTURED TIME SERIES MODELS:
THE RECURSIVE BS
3. Structured Time Series Models: The Recursive BS

ARMA Models
ARMA – AutoRegressive + Moving Average

ARMA(p,q) models with known p and q

3. Structured Time Series Models: The Recursive BS

ARMA Models
Consider the stationary AR(p) model
𝑝

𝑦𝑡 = ෍ 𝑎𝑖 𝑦𝑡−𝑖 + 𝑒𝑡 , 𝑒𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 )
𝑖=1
Given data on n + p observations (𝑦1−𝑝 , … , 𝑦0 , 𝑦1 , … , 𝑦𝑛 )

Our objective is to get:

• confidence intervals for the parameters 𝑎𝑖 or
• some smooth function h(𝑎1 , 𝑎2 , . . . , 𝑎𝑝 ) of the parameters 𝑎𝑖
3. Structured Time Series Models: The Recursive BS

The stationary AR(p) model

𝑝

𝑦𝑡 = ෍ 𝑎𝑖 𝑦𝑡−𝑖 + 𝑒𝑡 , 𝑒𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 )
𝑖=1

1 – Estimate (𝑎1 , 𝑎2 , . . . , 𝑎𝑝 ) by OLS based on n observations (𝑦1 , … , 𝑦𝑛 )

We get (𝑎ො1 ,𝑎ො2 , . . . ,𝑎ො𝑝 ) and the least squares residuals 𝑒Ƹ𝑡
1Τ2
1 𝑛
2 – Define the centered and scaled residuals 𝑒෥𝑡 = 𝑒ෝ𝑡 − 𝑛 ෍ 𝑒ෝ𝑡 𝑛−𝑝

Bickel & Freedman – residuals tend to be smaller than the true errors
3 – Resample 𝑒෥𝑡 with replacement to get the BS residuals 𝒆𝒕 ∗
3. Structured Time Series Models: The Recursive BS

The stationary AR(p) model

𝑝

𝑦𝑡 = ෍ 𝑎𝑖 𝑦𝑡−𝑖 + 𝑒𝑡 , 𝑒𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 )
𝑖=1
𝑝
∗
4 – Construct the BS sample 𝑦𝑡∗ = ෍ 𝑎ෝ𝑖 𝑦𝑡−𝑖 + 𝑒𝑡 ∗
𝑖=1

(done recursively using 𝑦𝑡∗ = 𝑦𝑡 for 𝑡 = 1 − 𝑝, … , 0 in each BS iteration)

1
−2
Bose – the LS estimates 𝑎ෝ𝑖 can be BS-ed with accuracy 𝑜 𝑛 - Little o
1
−2
improving the normal approximation error of 𝑂 𝑛 - Big o
3. Structured Time Series Models: The Recursive BS

Stationary AR(p) model

Stationary

AR(1) Model Unstable

Explosive
3. Structured Time Series Models: The Recursive BS

The AR(1) Model

𝑦𝑡 = 𝛽𝑦𝑡−1 + 𝑢𝑡

𝑦0 = 0 𝑢𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 ) −∞ < 𝛽 < ∞

3. Structured Time Series Models: The Recursive BS

The AR(1) Model

𝑦𝑡 = 𝛽𝑦𝑡−1 + 𝑢𝑡

𝑦0 = 0 𝑢𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 ) −∞ < 𝜷 < ∞

𝛽 <1 𝑦𝑡 is stationary

𝛽 =1 𝑦𝑡 is unstable

𝛽 >1 𝑦𝑡 is explosive
3. Structured Time Series Models: The Recursive BS

The AR(1) Model

𝑦𝑡 = 𝛽𝑦𝑡−1 + 𝑢𝑡

𝑦0 = 0 𝑢𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 ) −∞ < 𝛽 < ∞

Rubin(1950) – the OLS estimator β෠ of β is consistent in the range −∞, ∞

However, the asymptotic distributions of 𝜷 in the different ranges are different:

• In the stationary case 𝛽 < 1 - the asymptotic distribution is normal
• For the explosive case 𝛽 > 1 - Anderson (1959) - the limiting distribution is a
Cauchy distribution
3. Structured Time Series Models: The Recursive BS

The AR(1) Model with intercept

𝑦𝑡 = 𝛼 + 𝛽𝑦𝑡−1 + 𝑢𝑡 , 𝑢𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 )
Since the distribution of the OLS estimator of 𝛽መ of 𝛽 is invariant to α and 𝜎 2

Set α = 0 and 𝜎 2 = 1 and get 𝑦𝑡 = 𝛽𝑦𝑡−1 + 𝑢𝑡 , 𝑢𝑡 ~𝑖𝑖𝑑(0,1)

Dufour (1990) & Andrews (1993) – developed exact inference
procedures for the AR(1) parameter but these depend on the normality
assumption of the errors
While, the BS methods which are robust to distributional assumptions of the
errors, hold promise
3. Structured Time Series Models: The Recursive BS

The AR(1) Model with intercept

𝑦𝑡 = 𝛼 + 𝛽𝑦𝑡−1 + 𝑢𝑡 , 𝑢𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 )
The procedure for the generation of the BS samples is the recursive procedure

𝑝
∗ 𝑦0 is given
𝑦𝑡∗ = ෍ 𝑎ෝ𝑖 𝑦𝑡−𝑖 + 𝑒𝑡 ∗
𝑖=1 𝑝=1 𝑦(𝑡=1,…,𝑛) are recursively BS calculated
(𝑎1 , 𝑎2 , . . . , 𝑎𝑝 ) 𝛽0

The sampling scheme The t-statistic

𝑺𝟐 : 𝒚∗ = 𝜷𝟎 𝒙 + 𝜺∗ ෡ ∗ − 𝜷𝟎 Τ𝝈
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷 ෝ∗

Rayner (1990) – the use of the student-t approximation is not satisfactory, particularly for high values of 𝛽
the bootstrap-t performs very well in samples of sizes 5-10, even when mixtures of normal
distributions are used for the errors
4 - GENERAL ERROR STRUCTURES - THE
MOVING BLOCK BOOTSTRAP (MBB)
4. General error structures – The MBB

The Moving Block Bootstrap

Background
Application of the residual based bootstrap methods is straightforward if the
error distribution is specified to be an ARMA(p,q) process with known p and q
However, if the structure of serial correlation is not tractable or is misspecified,
the residual based methods will give inconsistent estimates

Carlstein (1986) – first discussed the idea of bootstrapping blocks of observations rather
than the individual observations. The blocks are nonoverlapping

Künsch (1989) and Singh (1992) – independently introduced a more general BS

procedure, the moving block BS (MBB) which is applicable to stationary time series data. In
this method the blocks of observations are overlapping.
4. General error structures – The MBB

The Moving Block Bootstrap

Divide the data of n observations into blocks of length l and select b of these blocks
(with repeats allowed) by resampling with replacement all the possible blocks

𝑦1 𝑦2 𝑦𝑛−1 𝑦𝑛 For simplicity assume 𝑛 = 𝑏𝑙

l l l
l
𝑛
In the Carlstein procedure: = 𝑏 blocks In the Künsch procedure: 𝑛 − 𝑙 + 1 blocks
𝑙

The 𝑘 𝑡ℎ block is 𝐿𝑘 = 𝑥𝑘 , … , 𝑥𝑘+𝑙−1

𝑘 = 1,2, … , (𝑛 − 𝑙 + 1)

For example with n = 6 and I = 3 suppose the data are: xt = {3,6,7,2,1,5).

The blocks according to Carlstein are {(3,6,7), (2,1,5)). The blocks according
to Kiinsch are {(3,6,7), (6,7,2), (7,2, l), (2,1,5)).
4. General error structures – The MBB

The Moving Block Bootstrap

EXAMPLE
𝑥𝑡 = 3 6 7 2 1 5

The blocks in the The blocks in the Künsch

Carlstein procedure are: procedure are:

3 6 7 2 1 5 3 6 7 6 7 2 7 2 1 2 1 5

Draw a sample of two blocks with replacement in each case

Suppose, the first draw gave 3 6 7 (WLOG)

Than, The probability of missing all of 2 1 5 is:

Carlstein: 50% Künsch: 25%

Higher probability of missing entire blocks in the Carlstein scheme (non overlapping blocks)

Carlstein scheme is not popular and not often used

4. General error structures – The MBB

The Moving Block Bootstrap

A comparison of four different block bootstrap methods
Boris Radovanov & Aleksandra Marcikić

MBB – Moving block bootstrap

NBB – Non-overlapping block
bootstrap
SBB – Stationary block bootstrap
SS - Subsampling
4. General error structures – The MBB

Problems with MBB

There are some important problems worth noting about the MBB procedure

1. The pseudo time series

generated by the moving
block method is not
stationary, even if the original
series {𝑥𝑡 ) is stationary

Politis and Romano (1994)

A stationary bootstrap method
The suggested method involves sampling blocks of
random length, where the length of each block has a
geometric distribution. They show that the pseudo
time series generated by the stationary bootstrap
method is indeed stationary.
4. General error structures – The MBB

Problems with MBB

There are some important problems worth noting about the MBB procedure

2. The mean 𝑥ҧ𝑛∗ of the moving block bootstrap is biased in the sense that:

𝐸 𝑥ҧ𝑛∗ |𝑥1 , 𝑥2 , … , 𝑥𝑛 − 𝑥ҧ𝑛 ≠ 0

3. The MBB estimator of the variance of 𝑛 ∙ 𝑥ҧ𝑛 is also biased

Davidson and Hall (1993)

This creates problems in using the percentile-t method with the MBB

the usual estimator: Should be modified to:

𝑛 𝑛 𝑖−1 𝑛−𝑘
2 −1 2
𝜎ො = 𝑛 ෍ 𝑥𝑖 − 𝑥ҧ𝑛 𝜎෤ 2 = 𝑛−1 ෍ 𝑥𝑖 − 𝑥ҧ𝑛 2
+ ෍ ෍ 𝑥𝑖 − 𝑥ҧ 𝑛 𝑥𝑖+𝑘 − 𝑥ҧ𝑛
𝑖=1 𝑖=1 𝑘=1 𝑖=1

With this modification the bootstrap-t can improve substantially on

the normal approximation
4. General error structures – The MBB

Optimal Length of Blocks

Several rules that have been suggested are based on different criteria. However,
the rules are useful as rough guides to selecting the optimal sized blocks

1. Carlstein’s Künsch’s
non-overlapping blocks < moving blocks

2. Politis and Romano‘s stationary bootstrap method

• The average length of a block is 𝑝1, where p is the parameter of the
geometric distribution
• The application of stationary bootstrap is less sensitive to the choice of
p than the application of moving block bootstrap is to the choice of I
4. General error structures – The MBB

Optimal Length of Blocks

3. Carlstein’s rules for non-overlapping blocks

Interested in minimizing the MSE of the block bootstrap estimate
of the variance of a general statistic 𝑡(𝑥1 , 𝑥2 , … , 𝑥𝑛 )

VARIANCE
As the block size increases
BIAS

As the dependency among A longer block size is

the 𝑥𝑖 gets stronger needed
4. General error structures – The MBB

Optimal Length of Blocks

3. Carlstein’s rules for non-overlapping blocks
Interested in minimizing the MSE of the block bootstrap estimate
of the variance of a general statistic 𝑡(𝑥1 , 𝑥2 , … , 𝑥𝑛 )

VARIANCE
As the block size increases
BIAS

As the dependency among A longer block size is

the 𝑥𝑖 gets stronger needed

The optimal block size 𝑙 ∗ for the AR(1) model 𝑥𝑡 = 𝜑𝑥𝑡−1 + 𝜀𝑡 is:
𝟐 Τ𝟑 𝟐 Τ𝟑
𝒍∗ = 𝟐𝝋Τ 𝟏 − 𝝋𝟐 𝒏
4. General error structures – The MBB

Optimal Length of Blocks

4. Hall and Horowitz’s rules
For Carlstein – They’re interested in minimizing the MSE of the variance
For Künsch – They’re interested in minimizing the MSE of the bias
They argue that the rules are similar both for Künsch’s moving block scheme and
For Carlstein’s non-overlapping block scheme

Carlstein’s Model Künsch’s Model

𝒍 = 𝒏𝟏Τ𝟑 𝝆−𝟐/𝟑 𝒍 = 𝟑/𝟐 𝟏/𝟑 𝒏𝟏Τ𝟑 𝝆−𝟐/𝟑

𝛾 0 + 2 σ∞
𝑗=1 𝛾(𝑗)
𝜌= γ(𝑗) –the covariance of 𝑥𝑡 at lag j
σ∞
𝑗=1 𝑗 ∙ 𝛾(𝑗)
4. General error structures – The MBB

Optimal Length of Blocks

4. Hall and Horowitz’s rules
For the AR(1) process For the MA(1) process
𝑥𝑡 = 𝜑𝑥𝑡−1 + 𝜀𝑡 𝑥𝑡 = 𝜀𝑡 + 𝜃𝜀𝑡−1

𝜌 = 1 − 𝜑 2 Τ𝜑 𝜌 = 1 + 𝜑2 Τ𝜑

Computed 𝒍∗ for n=200