LONG-RANGE FORECASTING

From Crystal Ball to Computer

 I
Seventeen
RESEARCH
ON FORECASTING
Contents
Persistence of Folklore
Research Strategies
Specific Research Areas
433
435
439
I
METHODS Implementation 441
Judgment 441
Extrapolation 441
Econometrics 442
Segmentation
Bootstrapping
Combined Forecasts
Meta-Analysis
Summary
443
443
444
444
446
I
I

431
432 Research on Forecasting Methods
One day through the primeval wood
A calf walked home as good calves should,
But made a trail all bent askew.
Since then three hundred years have fled,
And I infer the calf is dead.
But still he left behind his trail,
And thereby hangs my moral tale.
The trail was taken up next day
By a lone dog that passed that way
And then a wise bellwether sheep
Pursued the trail o'er vale and steep,
And drew the flock behind him, too,
As good bellwethers always do.
And from that day o'er hill and glade,
Through those old woods a path was made.

The years passed on in swiftness fleet,

The road became a village street;
And this, before men were aware,
A city's crowded thoroughfare.
And soon the central street was this
Of a renowned metropolis;
And men two centuries and a half
Trod in the footsteps of that calf.
Each day a hundred thousand rout
Followed this calf about
And o'er his crooked journey went
The traffic of a continent,
A hundred thousand men were led
By one calf near three centuries dead.
They followed still his crooked way,
And lost one hundred years a day;
For thus such reverence is lent
To well-established precedent.

"The Path of the Calf"

Sam Walter Foss, 1895

What types of research are most useful to further progress in fore-

casting methods? What specific areas are most in need of further re-
search? These questions are addressed in Chapter 17.
The plan of the chapter is first to describe a basic problem in research
called the persistence of folklore. The next part of the chapter examines
alternative research strategies, including directed research, multiple
Persistence of Folklore 433
hypotheses, and experimentation. The third section describes some
topics that should be examined, and others that are currently being
examined but are deserving of less effort. The last section discusses
meta-analysis, an approach to research that will be of substantial value
as scientific work on forecasting continues to grow.

PERSISTENCE OF FOLKLORE

Tell the truth and you will get your head bashed in.

Hungarian Proverb
(Noted in Szasz, 1973)

Scientists suffer from the same problem in learning as do other peo-

plethey look for confirming rather than disconfirming evidence. They
follow "the path of the calf."
(Evidence that people search for confirming evidence was presented
in Chapter 14.) The same appears to be true for that class of people
known as scientists. Bassie (1972), who has had a long career in econ-
ometrics, suggested that econometricians focus too much on confirming
evidence. So did I (Armstrong, 1978). Studies by psychologists also
indicate that scientists are biased toward confirming their own beliefs.
For example, two-thirds of the recording errors found by Rosenthal
favored the experimenter's hypothesis, and such a bias was found among
three-fourths of the experimenters in the same study (Rosenthal and
Rosnow, 1969). In another study (Rosenthal and Fode, 1963), experi-
menters were provided with two equivalent samples of rats and were
told that one sample was gifted and the other was disadvantaged. In
the subsequent laboratory experiments, the gifted rats learned tasks
more quickly than the disadvantaged ones.
The search for confirming evidence becomes stronger as a result of
group pressure. The scientist is evaluated by her peers. Groups of
scientists are often intolerant of ideas that are different, as shown in
the case of Immanuel Velikovsky (De Grazia, 1966). Barber (1961)
described attempts by scientists to suppress those who produced major
advances. To see how scientists respond to even modest departures
from the past, see the treatment accorded by discussants to forecasting
papers by Christ (1951), Cooper (1972), Armstrong [1978], and MAK-
RIDAKIS and HIBON [1979]. See also the book reviews of Sackman
434 Research on Forecasting Methods
(1975) and Chein's (1945) reaction to Sarbin's finding that expertise
did not help in psychological predictions.
Because of this need for confirming evidence, the first theory in a
field tends to persist. Often this first theory mirrors the common beliefs
of the man in the street. As long as confirming evidence is sought,
there will be no change in the folklore. Examples of folklore that were
examined in LRF include the following:

Econometric methods provide more accurate short-range forecasts

than alternative methods.
Complex models are more accurate than simple models.
Long-range forecasts are best made by judgmental methods.
Expertise is important in forecasting change.

These bits of folklore still exist, although there is little evidence to

support them and much evidence to refute them. For example, in the
survey on econometric methods in Chapter 15, two of the respondents
who viewed themselves as having high expertise, and who had the
highest confidence in their ratings, stated that they were not aware of
any empirical evidence that tested their belief.
It is undoubtedly easier to see folklore in fields other than our own.
The history of psychical research offers examples that should not be
threatening to most of us. Consider the possibility of extrasensory per-
ception (ESP). People having good ESP would be useful as forecasters.
Many people believe in ESP, and so do many scientists. The search for
empirically confirming evidence has gone on for over a century.
One of the earlie'st groups to support ESP was the British Society
for Psychical Research, founded in 1882. Its early successes were tar-
nished when the main subjects, the five daughters of Reverend Creery,
admitted a few years later that they had cheated. The society's next
major project also involved deceit: 16 years after some dramatic con-
firming evidence had been shown, Blackburn, the researcher, admitted
that he was cheating. He said in 1908: "[The hoax] originated in the
honest desire of two youths to show how easily men of scientific mind
and training could be deceived when seeking for evidence in support
of a theory they were wishful to establish." I Time, March 4, 1974, pp.
48-49.1
Psychic researchers have shown a great resistance to disconfirming
evidence. They have even developed "scientific procedures" to cope with
disconfirming evidence should it arise: they cite "negative ESP" if the
number of correct predictions falls substantially below average, and
"displacement" if subjects correctly predict an event before or after the
one in question.
Research Strategies 435
That is psychical research, you say (and perhaps biorythms [HOLMES,
et al. 1980; LOUIS, 19781). But the same behavior is found in fields of
higher repute. Remember Cyril Burt and his studies showing a strong
relationship between IQ and heredity (Wade, 1976)? He supported a
belief that many people find comfortable. (It allows us to think that
the problem of low IQs is God's fault rather than ours.)
In economics, it is interesting to examine what happens when even
mildly disconfirming evidence is provided. For example, the discus-
sants for Cooper's (1972) paper devoted much effort to explaining why
econometric methods did not perform well compared to extrapolations.
No consideration was given to shortcomings in the extrapolation models.
The need to avoid disconfirming evidence is expected to be even
greater for important issues. For example, economics tells us that in
a reasonably competitive market, such as exists in the United States,
people are worth what they earn. Evidence contradicting such a belief
would be expected to be ignored or refuted. This belief is an important
one in our society. According to studies in social psychology (e.g., see
Lerner and Simmons, 1966), people have a strong desire to believe that
the world is just. If someone else is making a lot of money, he must
deserve it. Luckily for those who earn a lot of money, this belief is
shared by most of those who earn little money.
Enough of this depressing evidence. What can be done to improve
the situation? That is the subject of the following section.

RESEARCH STRATEGIES

Arrowsmith (Lewis, 1924) illustrates the basic problem faced by a sci-

entist. Martin Arrowsmith, as you may know, was a medical scientist
who decided to withhold a potential cure from part of the population
to determine whether the cure was effective in stopping an epidemic.
This was upsetting to people; either a cure is effective, in which case
everyone should get it; or it is worthless, and no one should get it. The
way to decide this, people think, is to get a group of experts together
to reach a decision. That is how the United States decided to introduce
the swine flu vaccine program. (Having read Arrowsmith and early
drafts of my book, I realized how ridiculous this was, so I did not get
a swine flu shot. At this moment I am still alive.)
What Arrowsmith says, in simple terms, is that to learn what is
right, a scientist must do something wrong; that is, the scientist must
examine hypotheses that contradict the current beliefs. Now, that is
not easy to do; pick any important belief that people hold and try to
prove it wrong, and you will quickly find out how difficult it is to change
436 Research on Forecasting Methods
a "made-up" mind. For experimental evidence on this, see LORD, ROSS,
and LEPPER 119791. Brush's (1974) account of the reluctance of the
religious community to permit the scientific study of the value of prayer
is illustrative of what happens.
Science can help to legitimize the search for disconfirming evidence.
Unfortunately, not all strategies used by scientists are useful in this
respect. Consider the three possibilities:

To disconfirm existing beliefs (science)

To confirm existing beliefs (advocacy)
To disconfirm unimportant beliefs (irrelevancy)

Strategy 1 is being suggested here, but, as shown by Mahoney, it is

unpopular relative to strategy 2.

MAHONEY [1977] sent copies of a research paper to 75 reviewers.

Two versions of the paper had been created, one with results
favoring a dominant hypothesis in the field (behavior modifica-
tion) and one refuting it. The papers were identical in all respects
except for these results. Those reviewing the version with con-
firming evidence rated it as more sound methodologically than
those who received disconfirming evidence. The paper with con-
firming evidence was also rated as a more important scientific
contribution.

Strategy 2, to confirm existing beliefs, is a popular one with scien-

tists. This is done by developing arguments and collecting evidence to
support the favored hypothesis. To the extent that hypothesis testing
is done, it involves the use of an unreasonable null hypothesis. Green-
wald (1975) made a survey of referees for a journal and found a bias
toward studies that rejected the null hypotheses. He concluded from
the survey that the null hypothesis is selected to confirm a dominant
hypothesis.
Strategy 3, to disconfirm unimportant beliefs, is also a popular strat-
egy. One can refute these unimportant beliefs without getting people
upset. On the other hand, it is not satisfying to have people think that
one is working on an unimportant topic. That is where bafflegab comes
in (ARMSTRONG, 1980a1. Complex and poor writing can make almost
any research look respectable.
Research Strategies 437
And the Lord said, "Behold, they are one people, and they have all one
language; and this is only the beginning of what they will do; and nothing
they propose to do will now be impossible for them. Come, let us go down,
and there confuse their language, that they may not understand one
another's speech." Therefore its name was called Babel, because the Lord
confused the language of all the earth. . . .

Genesis 11:6 9

Consultants and large bureaucracies use this strategy profitably, as

illustrated in Exhibit 17-1. And many researchers are carrying on the
Lord's work by confusing the language.
If you agree that strategy 1 provides the most reasonable basis for
scientific research, then it is important to consider operational strat-
egies for such research. How can one search for disconfirming evidence?
The method of multiple hypotheses is ideally designed for finding
disconfirming evidence. With multiple hypotheses, each research project
is designed to compare two or more reasonable hypotheses. However,
rather than looking directly on what's wrong, it may help to look for
evidence to confirm one (but not the others) of the hypotheses. This
change in perspective was successful in the experiment by TWENEY
et al. 11980] and gained further support in TWENEY, DOHERTY, and
MYNATT 119821.
The method of multiple hypotheses contrasts sharply with the ad-
vocacy strategy, where the researcher attempts to find confirming evi-
dence. A researcher using advocacy tries to make the strongest case
for his favored hypothesis. Obviously this biases the way the scientist
perceives the world. (Evidence on this bias is presented in Geller and
Pitz, 1968; Pruitt, 1961; and JONES and RUSSELL, 1980.)
Mitroff (1972) argues that advocacy is common and has served as
the route to success for scientists. I agree. An examination of empirical
studies published in Management Science showed that almost 64% used
the advocacy approach, 14% lacked any prior hypothesis, and only 22%
used the method of multiple hypotheses (ARMSTRONG, 1979).
Chamberlin (1965) argued in 1890 for the method of multiple hy-
potheses, claiming that sciences that employ this strategy progress
more rapidly. His paper has been reprinted periodically and has been
seconded by others (e.g., Platt, 1964). Nevertheless, the method of mul-
tiple hypotheses remains unpopular in the social sciences. My survey
of publications in Management Science showed no trend toward an
increased use of multiple hypotheses over the past quarter-century.
My review of the literature on forecasting also yielded a small per-
Exhibit 17-1 THE VALUE OF OBSCURE LANGUAGE.

YOu WANT ID INTEgFoa c+41116

AM.% INITIAL InfTERRCGATORy_

\
G.NAL- LWAVE IT 5IR,TPAT BUT TLIE AIR FORCE ir5 VERY 5IMPL-E
A c'QuETIoN ABOUT I CAN GO INTOAW PAYS 072.17 FOR
0111Ew MR. Fi5111-1AwK..
DEFENSE SPENDING. I4ARDWARE STORE_ TI4E SAME
AND BUY "MIS WOOD
SCREW FOR 2 ct
et'
.

AV1\ r
1, emir

z. You rAy Two CENT5 Arc WE IN 714E AIR FE WE GET ME M- LB FUL4X-.S.OTTED,

You GET. A tvCr PAY $372 17, BUT MANUAU.Y ACTIVATED, FIE3E2-11411Z161VE

0111t,

,v
\
MATERIALS SECURING UNIT.

-4
M-18

19

Source.
Ilk P.
ZZ' Pc4444

"Shoe," reprinted by permission of Tribune Media Services, Inc.

Specific Research Areas 439

centage of papers using multiple hypotheses. However, the papers that

did use multiple hypotheses were the most useful ones in helping me
reach the conclusions in LRF
Replication studies can serve a useful function in disconfirmation.
Replication can guard against chance findings, cheating, and mistakes.
This is not a trivial matter. REID, SOLEY, and WIMMER 119811 re-
ported that 40% of 30 published replications they found in advertising
research conflicted with the original results. WOLIN I 19621 found gross
miscalculations in three of seven studies in psychology (also about
40%). Unfortunately, replication research is rare. (For more on this
see ARMSTRONG, 1982d.] In an attempt to address this problem, the
International Journal of Forecasting encourages replication studies in
forecasting.

SPECIFIC RESEARCH AREAS

My advice on the systems approach (in Chapter 2) applies to research-

ers as well as to normal people. Many researchers would disagree with
this statement. They can describe in detail what they are doing, but
not why. Nor can they explain when their research will prove beneficial.
(What ever happened to present worth calculations when talking about
investments in research?) These people generally refer to their work
as pure research» They hope that someone else will figure out why
they did it.
Pure research was of little value to this book. The most useful studies
were those that were directed at specific problems in forecasting studies
that were done with specific objectives in mind. Few highly rated stud-
ies were the result of pure research (this is consistent with Sherwin
and Isenson, 1967, but KREILKAMP [19711 concluded just the oppo-
site).
In the following subsections I suggest a few areas for directed re-
search. The emphasis is on research needed for long-range forecast-
ingnot only because that is the major concern in this book, but also
because it offers the greatest potential. A list of research areas is
provided in Exhibit 17-2. The research that is expected to yield a high
payoff is organized into two columns: "Hot Topics," or areas that are
currently being examined, and "Not-So-Hot Topics," or areas that have
received little attention. A third column lists areas that have received
much attention, but where further effort is expected to have a low
payoff. (I hope that your favorite area did not fall into this category).
The discussion in the following subsections considers each area listed
in Exhibit 17-2.
Exhibit 17-2 RESEARCH AREAS FOR LONG-RANGE FORECASTING

High Payoff Low Payoff

Area Hot Topics Not-So-Hot Topics Hot Topics
Implementation Scenarios Presentation
Judgment Expertise Decomposition Delphi
Biases
Structured meetings
Role playing
Extrapolation Simplicity Dampened Spectral analysis
seasonality Adaptive parameters
Dampened trend
Econometric A priori analysis Multiple data sources Simultaneous equations
Current status
models
Simplicity
Mitigation
Segmentation A priori-direct Exploratory-indirect
Bootstrapping Indirect Direct
Combined Forecasts Number of Methods
Specific Research Areas 441

Implementation

Scenarios can have a significant input on how people use forecasts.

Research on this topic can help people to use forecasts in a more
effective way. Users must also learn how to avoid being misled by
scenarios.
What type of presentation techniques will help to gain acceptance
of the forecasts? For example, how important is it to present arguments
against the forecast . . . to support the forecast with vivid examples .. .

to provide causal arguments for the forecast . . . to keep the forecast

simple . . .?

Judgment

Research has identified faulty procedures that people use in making

judgmental forecasts. The challenge for the future is to find ways to
overcome these judgmental defects. Typically, the solutions lie in the
use of more structure, so techniques like decomposition, consensus, and
Delphi have been found to be useful.
Role playing seems like a particularly important area. Does it really
provide a significantly better way to forecast the outcomes of conflict
situations? What is the most effective way to structure role playing
sessions?
Although little interest has been shown in decomposition, the lim-
ited evidence suggests that it is a technique with much potential. Re-
search on how and when to use decomposition would be useful.
Few Delphi studies have been designed in such a way that we could
learn anything from them. However, from these studies, we have a
pretty good idea on how to use Delphi and what to expect from it.
Additional validation studies are needed.

Extrapolation

Research on extrapolation methods has grown rapidly. For example,

FILDES and LUSK [1984] calculated a 25% per year growth rate in
the publication of comparative studies on ARIMA models from 1971
to 1978.
The issue of which method is most appropriate for which situation
is also important. To date, research on this topic has been exploratory.
As a result, different interpretations arise for each study. Of particular
interest are the variety of interpretations of the results from the M-
Competition (for example, see ARMSTRONG and LUSK 119831 and
FILDES and LUSK [1984]). I believe that the time has come for re-
442 Research on Forecasting Methods
search to test specific hypotheses. Exhibit 17-3 outlines the basic pat-
tern for such research.
Research from the past 25 years suggests that relatively simple
methods are adequate for extrapolation. The combination of forecasts
from two or more extrapolation *methods will reduce errors signifi-
cantly. Research is needed on the effect of modified seasonal factors.
Research is also needed on ways to improve extrapolations for medium-
and long-range forecasts. For example, what is the most effective way
to dampen the trend? What guidelines do we have on the value of
combining forecasts? How can we better estimate uncertainty?
Econometrics

A priori analysis has been a major interest of econometricians. This

emphasis is well placed. In fact, it would be worthwhile if even more

Exhibit 17-3 A FRAMEWORK FOR RESEARCH ON EXTRAPOLATION

PRIOR RESEARCH

METHODS CRITERIA SITUATIONS

Define specific Define relevant Specify key
methods ot study criteria aspect of the
situation

HYPOTHESES
Which method is
best for which
situation?

ACTUAL DATA SIMULATED DATA

(e.g., M-Competion)

THEORIES
Specific Research Areas 443

were done to find out what types of a priori analysis are useful in long-
range forecasting. (See Exhibit 8-2 for a listing of the types of analysis.)
Some not-so-hot topics seem worthy of additional work. For very
long-range forecasting, where accuracy may be sensitive to the esti-
mates of the causal relationships, it might be worthwhile to use mul-
tiple data sources. More work is also needed on current status models;
to what extent should one rely upon cross-sectional estimates, or upon
time series estimates of current status? How should one weight the
alternative estimates? Finally, there are the striking results on the
value of simplicity. Is it possible to develop guidelines relating sim-
plicity to the situation?
Mitigation, the adjustment of the model to compensate for uncer-
tainty over the forecast horizon, has been almost totally ignored. This
topic would appear to be of great importance for long-range forecasting.
It was of great importance in the one empirical test found. To generalize
from such scanty evidence, however, is risky.
The most popular area of all simultaneous equations, has been of
almost no demonstrable value. "You don't understand, Scott. This is
pure research," the econometrician said. "Give us time." Perhaps an-
other quarter-century of work by hundreds of econometricians will
prove fruitful. But I doubt it. There appear to be more useful ways of
spending our time and money. For those who insist on further work,
why not apply it to long-range rather than short-range forecasting?
This reorientation would improve its chances for success.

Segmentation

We need research that will lead to guidelines on how to develop effec-

tive segmentation models. In particular, this research should focus on
a priori methods directed at specific forecasting problems. What sample
sizes are required? How many segments are needed? How should one
select cutpoints? How should one project population and behavior within
segments?
Part of the problem with the research to date is that it has not been
specifically directed to the study of forecasting methods. Most work in
segmentation has been of the exploratory and has not been specifically
directed at forecasting.

Bootstrapping

Indirect bootstrapping is of particular interest. Still, it would be useful

to have more comparative studies on the costs, accuracy, and accept-
444 Research on Forecasting Methods
ability of indirect and direct bootstrapping. Indirect bootstrapping should
also be compared with unit weight models.
Much work has already been done on bootstrapping. One major issue
is implementation. Will decision makers yield to technological inno-
vations, or will they resist them in the same manner that blue-collar
workers resisted scientific management? I see some hope. Bootstrap-
ping could truly produce a revolution in forecasting and personal com-
puters may aid in this movement.
Further research could help to identify the situations in which boot-
strapping is most useful. When is bootstrapping better than judgmental
methods? Under what conditions should econometric methods be used
instead of bootstrapping? Also, which type of bootstrapping is most
effective in a given situation?

Combined Forecasts

The combination of forecasts from different methods guards against

mistakes, cheating, and faulty assumptions. In addition to reducing
the chance for big errors, it yields modest reductions in the typical
error. We need guidelines on when to use combined forecasts and how
to select the component methods. Of particular importance, how many
methods should be combined?

META-ANALYSIS

As can be seen from the references in this book, or in FILDES [1981],

a vast amount of research has been produced on forecasting. This offers
a new opportunity to researchers. In studying issues, they can draw
upon prior research findings. This strategy of "studying studies" was
christened "meta-analysis" by GLASS [1976].
Meta-analysis differs from traditional reviews in that it is more
systematic as well as more quantitative. The key steps in meta-analysis
are the development of hypotheses, the search for studies, the selection
of studies, and a statistical analysis of the studies selected.
The development of multiple hypotheses is a critical first step if the
likelihood of bias in the later steps is to be reduced [ARMSTRONG,
19791. For the search, explicit rules are needed to determine which
studies should be obtained. These rules should help to ensure that a
representative sample of the prior studies is obtained. It is seldom
possible to identify all prior studies. Modifications in these rules may
Summary 445

be necessary during the search, in which case, records should be kept

of the changes. (JACKSON 119801 reported that traditional reviews
usually do not describe the rules used for the search.) Selection rules
are then established to determine which of the studies are relevant.
For the analysis, formal procedures should be used to code the results.
The use of independent coders is useful. Then the rules are used to
combine probabilities across studies and to estimate the magnitude
and stability of effects. This analysis should be formally related to the
hypotheses. Detailed discussions of the procedures for meta-analysis
are provided in HUNTER, SCHMIDT, and JACKSON 119821 and
GLASS, McGAW, and SMITH 119811. An example of this procedure
for meta-analysis is provided in ARMSTRONG and LUSK 119851. This
paper stresses the importance of contacting the authors of the original
papers to ensure that their work has been interpreted correctly, to
obtain additional information on their study, and to obtain suggestions
on research that may have been overlooked. Although WOLIN 119621
reported some difficulty in obtaining responses from authors, I have
found authors, such as those cited in LRF , to be cooperative. EATON
119841 also reports good cooperation.
HUNTER, SCHMIDT, and JACKSON 119821 argue that the only
meaningful way to assess the state of knowledge is to cumulate indi-
vidual study results using statistical procedures. Support for this view-
point is provided in the experiment by COOPER and ROSENTHAL
119801, which found that meta-analysis led to larger estimates of effects
and more confident conclusions than were obtained from a traditional
reviewing process.
I believe that meta-analysis provides the most appropriate way to
resolve the numerous controversies that exist in forecasting today. Of
course, the usefulness of meta-analysis depends upon having a strong
foundation of basic studies.

SUMMARY

Folklore is expected to persist in science because researchers search

for confirming rather than disconfirming evidence. This process is in-
tensified as a field develops and groups of scientists work together.
The method of multiple hypotheses was suggested to change the role
of the researcher from one where she advocates a particular hypothesis
to one where she searches for disconfirming evidence in comparing a
set of reasonable hypotheses.
Research that is directed at specific forecasting problems is expected
446 Research on Forecasting Methods
to lead to rapid progress. A list of areas was proposed for further re-
search; some of these are already being studied, but others receive
little attention. These areas were summarized in Exhibit 17-1 along
with some areas where effort should be greatly reduced.
Meta-analysis was then proposed as a way to best summarize the
rapidly growing body of research on forecasting.
 I
THE THIRD EDITION
1

447
448 The Third Edition
I have started work on the third edition. Certainly I have overlooked
important and useful contributions from the past. Furthermore, I will
miss things in the future. But I will miss fewer things if you help.
Please let me know what I have overlooked. Especially valuable is
work that contradicts the generalizations reached in this book. The
ideal type of study would be:

Directed at specific forecasting methods.

Empirical.
Based on multiple hypotheses.
Published.
Understandable.

Please send the references on new or omitted work to the following

address:
Dr. J. Scott Armstrong
Wharton School
University of Pennsylvania
Philadelphia, PA 19104
I would also like to hear about the mistakes in LRF . Also, if I have
misinterpreted your research, let me know.