Psychotherapy © 2013 American Psychological Association

2013, Vol. 50, No. 1, 42–51 0033-3204/13/$12.00 DOI: 10.1037/a0030682

Outcome in Psychotherapy: The Past and Important Advances

Fifty years after the 1963 debate between Strupp and Eysenck, as recorded in their articles in Psycho-
therapy, it is clear that Eysenck overstated the case against psychoanalysis and dynamic psychotherapy
(Bergin, 1971), while inflating the magnitude of improvement in untreated individuals (Lambert, 1976).
Eysenck was probably correct about the beneficial effects of behavior therapies, but did not foresee that
behavior therapy would be supplanted by cognitive behavior therapies (CBT) and eclectic mixtures of
CBT that incorporate elements of eastern religion, humanistic interventions, and psychodynamic con-
structs. Fortunately, most of the treatments that have been tested in rigorous investigations have been
found to be effective, but few have distinguished themselves as uniquely superior. Many of the problems
of how to measure the effects of treatment have been solved and suggest that about two thirds of treated
individuals improve or recover. This leaves a sizable portion of nonresponding individuals, but emerging
methods involving in tracking treatment response are being used to decrease deterioration and enhance
positive outcomes.

Keywords: psychotherapy outcome, progress feedback, treatment effects, OQ-45

In the first issue of Psychotherapy, Hans Strupp (1963) wrote a talized on this fact to assert that patients could be expected to
stimulating article that reflected issues of that time period, an improve over the same period of time without formal treatment—
auspicious beginning for this new journal. His article was essen- that mental disorders were self-limiting and likely to subside over
tially a defense of the effects of psychoanalysis and related ana- time. In Strupp’s (1964) rejoinder, he suggested that ample evi-
lytic/dynamic, as well as eclectic treatments directed at the wholly dence documented the positive effects of psychotherapy and that it
negative evaluation of them by Hans Eysenck (1952). Strupp wrote was time to move on to more sophisticated questions (such as
persuasively about the ambitious goals of psychoanalysis, the comparative treatment effects), while also reiterating the impor-
existing evaluations of it, and the degree to which psychoanalysts tance of broad views of patient benefit: “To be sure, it is important
intended for the effects of their treatment to end with patients to free a patient from—say—a troublesome phobia, and everyone
being fully functioning individuals whose presenting symptoms would agree that symptom relief is a sine qua non in psychother-
disappeared, who were self-understanding and accepting (of both apy. But, we must not lose sight of the patient’s intrapsychic
weaknesses and strengths), who were relatively free of “talent- state— his sense of identity, feeling of worthwhileness as a person,
crippling inhibitions,” and who could form and maintain satisfying and happiness” (p. 101).
and lasting interpersonal relationships. The degree of improvement In the context of the initial issue of Psychotherapy and the
in evaluations of patients was assessed by psychoanalytic thera- debate about the effects of psychoanalysis and psychodynamic
pists who saw patients four times a week over years and who, it psychotherapies was the fundamental challenge that behavior ther-
can be said, had rather extensive knowledge of their patients and apy posed to psychodynamic thinking and procedures. The growth
a theoretical and common sense view of ideal mental health. of learning-based approaches that appeared as early as the 1920s
Strupp emphasized the validity of psychoanalytic therapists’ views (Jones, 1924; Mowrer & Mowrer, 1938) had not had a dramatic
of their patients’ outcome and strongly advocated for the effec- impact on psychotherapy until the publication of Wolpe’s Psycho-
tiveness of psychoanalysis based on the scant literature reporting therapy by Reciprocal Inhibition in 1958. But as Eysenck pointed
outcomes. out, behavior therapy appeared rather hopeful. Still the emergence
The next year Eysenck (1964) in a later issue of Psychotherapy
of cognitive therapy was a natural outgrowth of the limitations of
responded to Strupp’s advocacy and his own appraisal of the
the learning-based approaches, with their emphasis on behavior
effects of psychoanalysis and other psychotherapies reasserting
at the expense of thought, but it also represented dissatisfaction
that there was no sound scientific evidence supporting psychoanal-
with the effects of psychodynamic treatments. Cognitive therapy
ysis while advocating both tendencies for patients to heal them-
was most notably advocated by Ellis (1962) and Beck (1970) and
selves and the success of behavior therapies. At that point in
came to the forefront of theory-driven treatments by the mid1970s
history, existing studies had not randomly assigned patients to
with the publication of Beck’s (1976) Cognitive Therapy and the
treatment versus no-treatment control groups, and Eysenck capi-
Emotional Disorders. These and related developments, such as the
emergence of social learning theory (Bandura, 1969), provided
rich contrasts between cognitive theories and treatment methods
Correspondence concerning this article should be addressed to Michael and carried with them a strong research emphasis.
J. Lambert, PhD, Department of Psychology, 272 TLRB, Brigham Young Of course the debates over the accuracy of Eysenck’s estimates
University, Provo, UT 84602. E-mail: michael_lambert@byu.edu of both the outcomes of the verbal psychotherapies and those of

42
 OUTCOME ADVANCES 43

individuals who did not receive any formal treatment went on for been conducted on the outcomes of patients who have depression
at least another decade, with Bergin’s (1971) comprehensive esti- (Cuijpers & Dekker, 2005). Results indicate that most psycholog-
mates of change following psychoanalytic psychotherapy (around ical treatments that have been studied produce substantial effects,
85% for those that complete treatment) and Lambert’s (1976) in terms of symptom reduction in depression and increased well-
reanalysis of spontaneous remission rates (around 40% improve- being (Cuijpers, van Straten, van Oppen, & Andersson, 2008),
ment). These narrative research reviews and many others sug- with the number of types of effective psychotherapies rising over
gested that the outcomes for treated individuals were substantial in time. APA’s Division 12 Task force on empirically supported
relationship to the passage of time. Perhaps one of the more psychotherapies now lists 12 separate treatments for depression,
convincing studies on the evidence presented by Eysenck was six with strong evidence and six with lesser evidence (available
published by McNeilly and Howard (1991) who used Eysenck’s through American Psychological Association, Division 12). Pa-
own data to demonstrate a 50% improvement rate in eight sessions tients suffering from mood disorders who enter a variety of treat-
of psychotherapy as opposed to a 2% spontaneous remission rate ments can expect considerable relief, with the number who will
over the same time period, and that patients get 2 years of symp- experience a full remission varying with the type of mood disorder
tom remission in only 15 sessions of psychotherapy than would and its chronicity. The range of remission probably hovers some-
happen without psychotherapy. where between 35% and 70%.
It is fair to say that there is now an abundance of research on
treatment outcomes in clinical trials, suggesting that these effects
The Effects of Psychotherapy: Current Findings
can be achieved in 12–14 sessions of care (hardly an endorsement
In the ensuing years, evidence has mushroomed on a wide of psychoanalysis). These individuals will make larger gains than
variety of older and newer forms of treatment. A search of ISI Web similar individuals on wait-lists, or who receive “placebo” treat-
of Knowledge (an Internet journal search engine) reveals that ments, and they will maintain their gains at follow-ups 2 to 3 years
around 60,000 academic articles have been published on psycho- after treatment. It will not generally matter which kind of psycho-
therapy research in just the last 30 years. Much of the evidence for therapy is offered as long as it is a bona fide theory-driven
the effects of psychotherapy has been summarized in various intervention (Lambert, 2013b).
editions of the Handbook of Psychotherapy and Behavior Change, Many elements of care are shared by diverse treatment orienta-
now out in its 6th edition (Lambert, 2013a). Chapters in the tions and modalities (such as a confidential relationship character-
Handbook are organized around systems and modalities of psy- ized by high levels of understanding and respect as advocated by
chotherapy as well as special topics such as the therapist’s contri- client-centered theory, as well as exposure to anxiety-provoking
butions to outcome and those of the client. We now have a mass situations), and these common factors seem to loom large in
of well-designed scientific studies that neither Strupp nor Eysenck facilitating improved functioning (at least in depression), contrib-
had access to in 1963 and 1964 when they published their articles uting much more to ultimate outcomes than the kind of specific
in Psychotherapy. theory-based interventions that are offered (Cuijpers et al., 2012).
Reanalyses of older reviews, as well as newer meta-analytic Under the somewhat ideal circumstance of clinical trials, it
reviews of psychotherapy outcome, produce the broad finding of appears that the outcomes are better than those attained in routine
therapy benefit across a range of treatments for a variety of care where positive outcomes are found in closer to one third of the
disorders. Indeed, psychotherapy is more effective than many patients, and individuals participate for far fewer sessions (around
“evidence-based” medical practices, some of which are very costly four; Hansen, Lambert, & Forman, 2002). In routine care, it
and produce significant side effects, including almost all interven- appears that about 50% of treated individuals, including those with
tions in cardiology (e.g., beta-blockers, angioplasty, statins), geri- mixed and multiple diagnostic features might recover if they
atric medicine (e.g., calcium and alendronate sodium for osteopo- received about18 to 21 sessions of care. About 50% of clients will
rosis), and asthma (e.g., budesonide), influenza vaccine, and show reliable improvement following seven sessions of psycho-
cataract surgery, among other treatments (Wampold, 2007). Con- therapy (e.g., Anderson & Lambert, 2001). Although psychoana-
sidering the high burden of illness manifest in psychological lytic treatment outcomes continue to be understudied, patients who
disorders, and the fact that the psychotherapies studied last only undergo psychodynamic treatments of much shorter duration
weeks, the consequences of entering treatment versus having no (around 20 sessions) appear to fare as well as in other treatments
formal treatment are dramatic. The effect size between treated and (Barber, Muran, Keefe, & McCarthy, 2013). So Eysenck (1964)
untreated individuals produced by quantitative reviews hovers was correct in suggesting that psychoanalysis was not efficient, at
around d ⫽ .75, leading to an estimate of a general success rate in least for symptom-focused outcomes. The evidence that long-term
treated persons of 67% compared with that of 33% for untreated psychodynamically oriented treatments lead to more substantial
persons over the same period of time. A paradox with these changes in personality than other therapies is in doubt given
estimates is that the outcome of treated cases weather treated by patients’ ability to maintain their treatment gains following much
behavior therapy, cognitive behavior therapy, or other types of shorter treatments.
psychotherapy (e.g., psychodynamic, emotion-focused psycho- Fifty years after the 1963 debate between Strupp and Eysenck,
therapy) is nearly identical to Eysenck’s estimate of the spontane- as recorded in their articles in Psychotherapy, it is clear that
ous remission rate and therefore Eysenck’s advocacy for behavior Eysenck overstated the case against psychoanalysis and dynamic
therapy is just as vulnerable to his criticisms. psychotherapy (Bergin, 1971), while inflating the magnitude of
Numerous meta-analytic reviews now consider outcomes of improvement in untreated individuals (Lambert, 1976). Eysenck
patients with specific disorders. For example, in the past three was probably correct about the beneficial effects of behavior
decades alone, more than 40 meta-analyses (not just studies) have therapies, even though they often have very narrow goals—that is,
44 LAMBERT

behavior in specific contexts, and have been supplanted by cog- treated in routine practice settings suggest that the clients did not
nitive behavior therapies and other innovations. There continues to fare as well as those in clinical trials, with deterioration rates as
be a debate about the degree to which cognitive therapy can high as 14% in some settings (Hansen et al., 2002). The situation
achieve its aims independent of the behavioral component of for child psychotherapy in routine care is even more sobering. The
treatment (Emmelkamp, 2013) and the degree to which behavioral small body of outcome studies in community-based usual care
interventions are necessarily independent of the cognitive aspects settings has yielded a mean effect size near zero (e.g., Weisz,
of treatment, at least for depression (Hollon & Beck, 2013). 2004), yet millions of youth are served each year in these systems
Psychotherapy can be quite efficient for a large minority of of care. In a comparison of children being treated in community
patients; however, the number of sessions needed for a larger mental health (N ⫽ 936) or through managed care (N ⫽ 3075),
portion (75%) of persons to recover hovers around 50 sessions. In estimates of deterioration were 24% and 14%, respectively (War-
contrast, every theory of behavior change has yet to deal with the ren, Nelson, Mondragon, Baldwin, & Burlingame, 2010).
surprising and perplexing phenomena of sudden gains (or early There is no doubt that all of the deterioration that occurs during
dramatic treatment response) in psychotherapy. When patients’ the time a patient is in treatment cannot be causally linked to
mental health functioning is monitored on a session-by-session therapist activities. Certainly, a portion of patients are on a nega-
basis, it appears that a substantial number of patients (17%– 40%) tive trajectory at the time they enter treatment and the deteriorating
respond to treatments (offered in clinical trials and routine care) course cannot be stopped. Another subset of patients experience
much sooner and more substantially than theory would predict. untoward life events that cannot be prevented and that have noth-
These early responders make up a substantial percentage of recov- ing to do with treatment. A portion of patients are prevented from
ered/improved individuals at termination and at follow-up years taking their own lives as a result of effective practices, even if they
later (e.g., Haas, Hill, Lambert, & Morrell, 2002). Depending on do not show overall progress. Just as positive psychotherapy
just when such changes occur (with a median session of 5), it outcomes depend largely on patient characteristics, so do the
appears that they precede many of the interventions thought nec- negative changes that occur in patients who are undergoing psy-
essary to bring about change. Such findings certainly present a chological treatments. Unfortunately, there are reasons to believe
challenge to psychoanalytic methods and cast serious doubt on the that therapists do not recognize client worsening, although there
necessity and desirability of prolonged treatment for many indi- are methods available to help with this problem—a central focus of
viduals. this article.
This phenomenon also presents a challenge to other therapies Positive as well as negative patient change can be affected by
that advocate a shorter manualized approach to recovery. Tang and therapist actions and inactions. Research reviews find that the
DeRubies (1999) have gone to some lengths to attribute sudden major contribution of the therapist to negative change is usually
gains to components of cognitive behavior therapy with depres- found in the nature of the therapeutic relationship, with rejections
sion, but such an explanation does not take into account the fact of either a subtle or manifest nature being the root cause (e.g.,
that early dramatic improvement occurs in other disorders and Lambert, Bergin, & Collins, 1977; Safran, Muran, Samstang, &
without the use of cognitive behavior treatment. The fact that these Winston, 2005).
reported large and lasting benefits occur so early in a wide variety Unfortunately, it appears that clinicians have an overly optimis-
of treatments does suggest that the mechanisms of recovery often tic view of their own patients’ progress (Walfish, McAlister,
involve some kind of dramatic self-reorganization, which are O’Donnell, & Lambert, 2012), estimating very high rates of im-
highly dependent on client characteristics. Sudden gains may be provement, and outcomes far superior to their peers. Clinicians
the result of a corrective emotional experience, a phrase that came overlook negative changes and have a limited capacity to make
out of psychoanalytic theory that recognized that change was not accurate predictions of the final benefit clients will receive during
necessarily incremental (step-by-step), but rather could be sudden, treatment, particularly with clients who are failing to improve. One
dramatic and surprising in terms of lasting impact. study, for example, found that even when therapists were provided
However, psychoanalysis did not consider dramatic change in with the base rate of deterioration in the clinic where they worked
early treatment sessions to be credible change, considering it a (8%), and were asked to rate each client that they saw at the end
“flight into health,” a way of escaping the demands of analysis. of each session (with regards to the likelihood of treatment failure
Instead it appears to be a legitimate common phenomenon across and if the patient was worse off at the current session in relation to
treatment modalities. Clinicians and researchers have noted related their intake level of functioning), they rated only 3 of 550 clients
phenomena in many forms of psychotherapy (see Hill & Caston- as likely to have a negative outcome and seriously underestimated
guay, 2010; Transformation in Psychotherapy), but little is really that their client was worse off at a current session in relation to
known about the causes of corrective experiences and their rela- their intake level of functioning (Hannan et al., 2005). In a separate
tionship to rapid dramatic treatment response. An important aspect study, Hatfield, McCullough, Plucinski, and Krieger (2010) found
of the early response is many individuals do not respond in an in a retrospective review of case notes of clients who had deteri-
immediate and dramatic way: their change, if it occurs, takes the orated during treatment infrequent mention of worsening even
form of slower improvement over sessions of treatment. when its degree was dramatic.
An often ignored, but critical, consideration in psychotherapy is Such results are not surprising, given psychotherapist optimism,
the degree to which the therapies have negative rather than positive the complexity of persons, and a treatment context that calls for
consequences for clients. An estimated 5% to 10% of adult clients considerable commitment and determination on the part of the
participating in clinical trials leave treatment worse off than they therapist, who actually has very little control over the patient’s life
began treatment (Lambert & Ogles, 2004). In routine care, the circumstances, decisions, and personal characteristics. Patients’
situation is more problematic. Outcomes for ⬎6,000 patients response to treatment is, especially in the case of a worsening state,
 OUTCOME ADVANCES 45

a likely place where outside feedback might have the greatest management is that outcome data can be regularly gathered and
chance of impact. Helping the therapist become aware of negative fed back to clinicians in real time for the purpose of making
change and discussing such progress in the therapeutic encounter needed alterations in intervention strategy if clients are either
are much more likely when formal feedback is provided to thera- unresponsive to, or deteriorating in treatment.
pists. Such feedback helps the client communicate and helps the Several psychotherapy outcome management systems have been
therapist to become aware of the possible need to adjust treatment, developed and implemented in clinical service delivery settings
alter or address problematic aspects of the treatment as appropriate worldwide, such as in the United States (Lambert, Hansen, &
(e.g., problems in the therapeutic relationship or in the implemen- Finch, 2001; Lueger et al., 2001; Miller, Duncan, Sorrell, &
tation of the goals of the treatment). Brown, 2005), Germany (Kordy, Hannover, & Richard, 2001), the
Netherlands (deJong et al., 2007), Australia (Newham, Hooke, &
Page, 2013), Norway (Anker, Duncan, & Sparks, 2009), and Great
Measuring, Monitoring, Predicting Treatment Failure,
Britain (Barkham et al., 2001). Although the specific procedures
and Using Feedback: Advances in Practice
used in each of these quality management systems vary, a common
A logical procedure for diminishing deterioration and enhancing feature across all of them is the monitoring of client outcome
positive outcomes involves routinely measuring, regularly moni- throughout the course of treatment and the use of these data to
toring, and tracking client treatment response with standardized improve individual client outcomes.
scales throughout the course of treatment while providing clini- In the remainder of this article, one specific psychotherapy
cians (and clients) with this information. quality management system that has been developed, imple-
mented, and empirically evaluated in multiple investigations is
described. The methodology used endeavors to improve psycho-
Definitions and Feedback Measures
therapy outcome by monitoring client progress in relation to ex-
Clients can complete a brief measure of their psychological pected progress and providing this information to clinicians in
functioning by using standardized rating scales and then this order to guide ongoing treatment, particularly for the client who is
information can be delivered to psychotherapists in real time. Such not having a favorable response to treatment (signal-alarm or
measures can be regarded as a mental health vital signs lab test. Not-On-Track cases). This methodology is an extension of quality
This lab test data can be used to indicate the client’s current level assurance and represents one effort to bridge the gap between
of disturbance in relation to functional and dysfunctional popula- efficacy and effectiveness research and clinical practice, while
tions, deviations from expected treatment response over the course enhancing patient outcomes. It is also well suited to models of care
of psychotherapy, and the consequences of treatment. Collecting in which clinicians attempt to step-up or step-down treatments
this information from the client on a session-by-session basis after assessing patient treatment response (e.g., Otto, Pollack, &
provides the clinician with a systematic way of monitoring life Maki, 2000).
functioning from the client’s point of view. A brief formal assess-
ment can provide a summary of life functioning that is not other-
The OQ Psychotherapy Outcome Management System
wise available to the therapist, unless the therapist spends time
within the treatment hour to systematically inquire about all the Patient outcome can be conceptualized and measured in myriad
areas of functioning covered by the self-report scale, an activity ways. Strupp and Hadley (1977) proposed a tripartite model for
that detracts from service delivery. Lab test data and information mental health outcomes. They suggested that the three interested
about deviations from an expected treatment response provide parties concerned with therapy outcomes were society, the client,
novel information to therapists. and the mental health professional. Based on this viewpoint, Lam-
This is a quality assurance practice that might be considered a bert (1983) suggested the most important aspect of outcome is the
form of managing outcomes (Evans, Mellor-Clark, Barkham, & subjective experience of the client, including symptoms of anxiety
Mothersole, 2006). For many decades, and even to the present day, and depression. In addition, the functioning of persons in their
psychotherapy outcome research, with the notable exception of the social roles (e.g., work) and intimate relationships can be seen as
behavior therapies, has relied heavily on study designs that mea- very important for the well-being of the person, family, and society
sure client outcome at pretreatment and posttreatment. Although in general. This definition of psychotherapy outcome approximates
such designs have proven beneficial in establishing the general that used in the early investigations of psychoanalysis where the
efficacy and effectiveness of the treatments under investigation, interest in change goes beyond symptomatic improvements.
they are limited in that outcome data from these studies (because As Strupp (1963) noted, psychotherapy outcome is clearly
it is collected after termination from treatment) cannot be used to broader than symptom change and includes changes in personality/
positively influence the treatment process of the individual clients self-organization. Certainly deeper changes are highly desirable
under investigation. Pre- and posttreatment assessments, then, are and important to clinicians as well as researchers because they
essentially a “postmortem” analysis of outcome, as clients have have important implications for lifelong adjustment and coping.
already terminated treatment and nothing can be done to improve The extent to which the OQ-45 and similar measures go beyond
their outcomes, even if they experienced no change or even dete- merely measuring changes in symptomatic states is open to ques-
riorated in treatment. Outcome management extends the practice tion. Measures such as the OQ-45, Beck Depression Inventory,
of measuring and monitoring client progress throughout the course Symptom Check List-90, and the like are highly correlated. In my
of treatment by then using data collected in real time to positively own opinion, changes on these “symptom” scales reflect and are
influence the treatment process and outcome of the clients under proxies for and estimates of deeper changes in individuals that
investigation. The major advantage of psychotherapy outcome reflect modifications in foundational views of self and others. But
46 LAMBERT

such a supposition has not been studied and the extent to which it in the following categories based on the change observed in their
is true has not been estimated and needs clarification in future OQ scores:
research. Recovered (i.e., clinically significant change)—Clients whose
Quality management systems, as applied in routine practice, score decreases by 14 or more points and passes below the cutoff
demand efficient outcome assessment rather than the more ideal score of 64.
alternative of comprehensive assessment. Outcome measurements Improved (i.e., reliably changed)—Clients whose score de-
that are typically used in efficacy studies often require hours of creases by 14 or more points but does not pass below the cutoff
assessment from multiple perspectives of change. In contrast, score of 64.
outcome-focused research uses weekly assessments with a single No Change—Clients whose score changes by less than 14 points
brief self-report measure. Thus, assessment in this type of research in either direction.
is much more frequent, with a greater diversity of patients and Deteriorated—Clients whose score increases by 14 or more
large final sample sizes, but less comprehensive and lacking mul- points.
tiple perspectives of change. To a large degree, this methodology Support for the validity of the OQ-45’s reliable change and
is essential to use for two reasons. The first is that unlike clinical normative cutoff score has been reported by Lunnen and Ogles
trials, where the number of treatment sessions a patient is expected (1998) and Beckstead et al. (2003). Having a method to classify
to attend is predetermined (usually around 14 sessions), in routine each client’s treatment response is an essential component of
practice, lengths are indeterminate and largely based on client and outcome management, given that the primary purpose of psycho-
therapist preferences. In clinical trials, progress can be assessed at therapy outcome management is to understand and improve the
preset times during the course of treatment (e.g., at weeks zero, 6, gains each individual is making during the course of treatment.
and 12) and especially at termination. In routine care, the outcome Furthermore, the ability to classify individual client change further
measure is collected before each treatment session, so that the bridges the gap between traditional efficacy and effectiveness
effects of the final session are infrequently collected, but out- studies (that focus on changes made by groups of clients) and
come data are available for every session up to that point in time. clinical practice (Kendall, Marrs-Garcia, Nath, & Sheldrick, 1999).
The second reason is related to repeated measurement. Although
patients have no problem tolerating a weekly assessment that takes Prediction of Treatment Failure
about 5 min, they cannot be expected to undergo lengthier, mul-
tiscale, multisource assessments on a frequent basis. A core element of outcome management systems is the predic-
Given the demand for regular and efficient outcome assessment in tion of treatment failure. To improve outcomes of clients who are
psychotherapy outcome management, the Outcome Questionnaire-45 responding poorly to treatment, such clients must be identified
(OQ-45; Lambert, Morton, et al., 2004) was developed. It is a 45-item before termination from treatment, and ideally, as early as possible
self-report measure designed to assess four domains of client func- in the course of treatment. Though many studies have investigated
the value of several client, therapist, client–therapist interaction,
tioning: symptoms of psychological disturbance, particularly anxiety
and extratherapeutic variables in predicting outcome, very few of
and depression, interpersonal problems, social role functioning, and
the variables explored are consistently highly predictive of out-
quality of life (well-being). Consistent with this conceptualization of
come. Research using the OQ-45 has indicated that the best pre-
outcome, the OQ-45 provides a Total Score, based on all 45 items, as
dictors of outcome are initial severity of distress (i.e., pretreatment
well as Symptom Distress, Interpersonal Relations, and Social Role
OQ-45 total score) and change of score following separate sessions
subscale scores. Higher scores on the OQ-45 are indicative of greater
over the course of treatment. In fact, Brown and Lambert (1998)
levels of psychological disturbance. Research has indicated that the
found that pretreatment OQ-45 total score and change scores from
OQ-45 is a psychometrically sound instrument that has been shown to
Sessions 1 to 3 accounted for approximately 40% of the variance
be sensitive to changes in multiple client populations over short
in final outcome, and that after taking these variables into account,
periods of time while remaining relatively stable in untreated individ-
all other variables combined (e.g., diagnosis, client demographics,
uals (Vermeersch, Lambert, & Burlingame, 2000; Vermeersch et al.,
therapist demographics, therapist theoretical orientation, etc.) ac-
2004). It is well suited for assessing initial levels of client distress and
counted for less than 1% of the variance in final outcome. In other
tracking client status during and following treatment.
words, in prior studies using the OQ-45, the best way to predict
outcome was to know how distressed clients were prior to treat-
Defining a Positive and Negative Outcome ment and whether or not the changes they made at the session of
interest in the treatment process were positive or negative and to
A key element in psychotherapy quality management research is what degree.
defining and operationalizing the concepts of positive and negative Given research on the variables most predictive of outcome, an
outcome for the individual patient. Jacobson and Truax (1991) empirically derived signal-alarm system (which plots a statistically
offered a methodology by which a client’s change on an outcome generated expected recovery curve for differing levels of pretreat-
measure can be classified in the following categories: recovered, ment distress on the OQ-45 and uses this as a basis for identifying
reliably improved, no change, deteriorated. There are two pieces of clients who are not making expected treatment gains and are at risk
information necessary in order to make these clients outcome of having a poor outcome) was developed to alert clinicians to
classifications: a Reliable Change Index (RCI) and a normal func- potential treatment failures. The accuracy of this signal-alarm
tioning cutoff score. Clinical and normative data were analyzed by system has been evaluated in a number of empirical investigations
Lambert, Morton, et al. (2004) to establish an RCI and a cutoff (Ellsworth, Lambert, & Johnson, 2006; Lambert, Whipple, Bishop,
score for the OQ-45. Using this information, clients can be placed et al., 2002; Lutz et al., 2006; Spielmans, Masters, & Lambert,
 OUTCOME ADVANCES 47

2006), and though an extensive discussion of the results of these Over the last 25 years, methodologies have been used in medical
studies is beyond the scope of this article, it is important to note research and practice to manage clinical interventions in areas such
that the signal-alarm system is highly sensitive in that it is able to as drug dosage, diagnosis, and preventive care. These interventions
accurately predict a poor outcome in 80% to 100% of cases that are often used in a stepwise approach that assists physicians in
actually end with a negative outcome, and it is also far superior to clinical decision-making and provides recommendations to im-
clinical judgment in its ability to identify clients who are at risk of prove the quality of patient health care (Hunt, Haynes, Hanna, &
having a negative treatment outcome (Hannan et al., 2005). Smith, 1998). Similarly, a set of Clinical Support Tools (CST) was
To identify potential treatment failures, the alarm system over developed and integrated into the existing psychotherapy quality
predicts at a ratio of about two to one. Unlike some medical management system in an attempt to augment the feedback pro-
decisions where the cost of over identification of signal cases may vided to therapists and further improve outcomes of nonrespond-
result in intrusive and even health threatening interventions, the ing and deteriorating patients (Whipple et al., 2003). As such, the
signal-alarm in psychotherapy merely alerts the therapist to the CSTs are intended to be used by therapists only when one of their
need for reconsidering the value of ongoing treatment, rather than clients is predicted to have a poor outcome (i.e., when a therapist
mandating specific changes. Thus, we see the signal-alarm as receives a red or yellow warning message, indicating that client is
supporting clinical decision-making, rather than supplanting it. not responding or deteriorating in treatment).
Because the signal-alarm alerts therapists to the possible need for The CSTs are composed of a problem-solving decision tree
action, rather than triggering a negative chain of events such as designed to systematically direct therapists’ attention to certain
termination or referral, the current level of misidentification would factors that have been shown to be consistently related to client
seem to be tolerable. outcome in the empirical literature, such as the therapeutic alli-
ance, social support, readiness to change, diagnostic formulation,
and need for psychiatric referral. A single measure has been
The Provision of Feedback and Clinical Support Tools
developed (Assessment for Signal Cases; ASC) aimed at assisting
The signal-alarm system has been used as an intervention for therapists to assess the quality of the therapeutic alliance, client
preventing deterioration and enhancing positive outcomes in cli- readiness for change, client perception of social support, and life
ents, in that it alerts clinicians to potential treatment failures and events. The CST Manual (Lambert et al., 2007) provides specific
allows them to modify their treatment approach (if they deem that intervention strategies that could be used by therapists if problems
to be appropriate) in an attempt to improve the outcomes of clients were detected in the aforementioned domains. These intervention
who are having a poor response to treatment. Once a client takes strategies are also in included in the OQAnalyst Software (www
the OQ-45, commences treatment, and completes a session of .OQMeasures.com). When the signal-alarm rules identify a client as
treatment, the signal-alarm system can be used to generate feed- Not-On-Track (red or yellow warning), therapists have the option
back regarding the client’s progress. The feedback to therapists of using the ASC (40 self-report items) and the decision tree that
consists of several components, among which are a progress graph organizes problem-solving hierarchically. Specific items that are
that includes all the client’s OQ-45 total scores from pretreatment problematic are highlighted in the report, and suggested interven-
to the current session and a color-coded message (white, green, tions are provided. For example, if a client is identified as at risk
yellow, red) that indicates the status of client progress. The spe- for treatment failure and the alliance total score, subscale scores
cific language of the feedback messages varies not only as a (bond, task, and goal agreement), and specific alliance items are all
function of client progress, but also as a function of the session at problematic, the therapist can then examine the list of suggestions
which the feedback is provided (i.e., a red message at Session 2 is for strengthening the relationship (e.g., discuss the clients ratings
not as urgent as a red message at Session 20). An illustrative of the relationship, explore relationship ruptures). If the client’s
summary of each feedback message follows: alliance rating was not below average, the therapist would proceed
White Message—“The Client is functioning in the normal to evaluation of the client stage of motivation and so forth.
range. Consider Termination.” In addition to providing feedback regarding client progress and
Green Message—“The rate of change the client is making is in CSTs to therapists, feedback can also be provided directly to
the adequate range. No change in the treatment plan is recom- clients. Client feedback messages (i.e., white, green, yellow/red)
mended.” that correspond to the aforementioned therapist feedback messages
Yellow Message—“The rate of change the client is making is have been developed in an effort to directly inform clients of their
less than adequate. Recommendations: consider altering the treat- progress in treatment and enhance client/therapist collaboration in
ment plan by intensifying treatment, shifting intervention strate- treatment (Harmon et al., 2007; Hawkins, Lambert, Vermeersch,
gies, and monitoring progress especially carefully. This client may Slade, & Tuttle, 2004). A summary of a client feedback message
end up with no significant benefit from therapy.” follows:
Red Message—“The client is not making the expected level of Yellow/Red Message—Please note that the following informa-
progress. Chances are he or she may drop out of treatment pre- tion is based on your responses to the questionnaire that you have
maturely or have a negative treatment outcome. Steps should be completed before each therapy session. It appears that you have
taken to carefully review this case and decide upon a new course not experienced a reduced level of distress. Because you may not
of action such as referral for medication or intensification of be experiencing the expected rate of progress, it is possible that
treatment. The treatment plan should be reconsidered. Consider- you have even considered terminating treatment, believing that
ation should also be given to presenting this client at case confer- therapy may not be helpful for you. Although you have yet to
ence. The client’s readiness for change may need to be reas- experience much relief from therapy, it is still early in treatment
sessed.” and there is the potential for future improvement. However, we
48 LAMBERT

urge you to openly discuss any concerns that you may be having cases improving or returning to a normal state of functioning at the
about therapy with your therapist because there are strategies that end of treatment.
can be used to help you receive the most out of your therapy. It Several studies have been conducted since completion of the
may also require your willingness to complete additional question- meta-analytic review suggesting that progress feedback with alerts
naires that may shed light about why you are not experiencing the and problem-solving tools are effective across diverse treatment
expected rate of progress. settings (inpatient, outpatient) and patient samples ranging from
Consistent with the findings of previous research (Flowers, the inpatient treatment of eating disorders to substance abuse. The
1979; Kivlighan, 1985; Kluger & DeNisi, 1996), the messages results of these later studies all produced statistically significant
designed for patients were a blend of positive and negative lan- improvements in off-track cases compared with treatment-as-usual
guage. Effort was made to avoid message content potentially offered by the same therapists. It is worth pointing out that ther-
perceived as threatening or discouraging to patients’ self-esteem. apists in the studies we have conducted were not familiar with the
Patients were informed of their self-reported level of distress feedback procedures before the studies and in most cases seemed
according to the OQ-45, progress since beginning therapy, and skeptical about the value and need for such procedures. In general,
likelihood of benefiting from treatment given the present course of therapists were volunteers who agreed to try the tools and were
progress. Additionally, patients identified as potential treatment surprised to find that their clients had better outcomes when they
failures are encouraged to discuss personal concerns about their attended to the feedback, alarm-signals, and CSTs.
progress, alternative courses of action, and goals of therapy with Based on research findings, feedback of the kind just described
their therapists to further facilitate the collaborative alliance. is considered an evidence-based practice that can be recommended
The administration of the OQ-45 (whether via paper–pencil or for routine use (see OQ-Analyst at: http://www.nrepp.samhsa.gov/
computerized), scoring, application of the signal-alarm system, ViewAll). Evidence on alternative feedback systems has been
and generation of feedback reports (for therapists and/or clients) slower to emerge, but also provides positive evidence that progress
are almost instantaneously processed through software called OQ- feedback enhances patient outcomes (Castonguay, Barkham, Lutz,
Analyst (administration of the measure and generation of the & McAleavey, 2013). Much of the research shows that the effects
measure takes approximately 5–7 min and is completed before a of feedback are strongest with clients who are struggling to use the
session usually on a handheld device or online). Accessing the therapy to their benefit rather than with those who make consistent
feedback report on the therapist’s personal computer takes approx- progress (about 70%– 80% of cases). Unfortunately all patients
imately 18 s. need to be monitored in order to identify those who significantly
deviate from a course of improvement.
As with many innovations in clinical care, clinicians appear
Impact of Feedback on Client Outcome hesitant to adopt feedback of the kind just described in routine
practice. Besides the natural tendency for providers to stick with
Nine controlled studies have been published that examine the the treatments they learned during their training (which does not
effects of providing client progress feedback to therapists and/or yet include such feedback), their tendency to overestimate their
clients using the methodology described above (Crits-Christoph et success rates and underestimate negative outcomes, and their gen-
al., 2012; Harmon et al., 2007; Hawkins et al., 2004; Lambert, eral skepticism of empirical research slows down routine use.
Whipple, et al., 2001; Lambert, Whipple, Vermeersch, et al., 2002; Monitoring patient treatment response may be threatening to per-
Simon, Lambert, Harris, Busath, & Vazquez, in press; Simon et al., sonal perceptions of exceptional effectiveness held by most clini-
unpublished; Slade, Lambert, Harmon, Smart, & Bailey, 2008; cians. Collected data on patient outcomes illuminate the differen-
Whipple et al., 2003). Shimokawa, Lambert, and Smart (2010) tial effectiveness of individual providers. Because such data are
have reviewed feedback research with patients who are predicted normally distributed, and most therapists’ patients have average
to be treatment failures (as well as patients who appear to be outcomes, changes in functioning based on standardized scales are
on-track for a positive outcome). This meta/megaanalysis com- usually disappointing to therapists. Documenting differences be-
bined data from six well-designed clinical trials that compared tween providers’ patients’ outcomes allows comparisons to be
treatment-as-usual to feedback-assisted treatments in which the made between providers, something that clinicians are not used to
same therapists offered both conditions to over 4,000 patients. reconciling with their personal assessment of their effectiveness. It
Results indicated that feedback to therapists and patients had a is certainly more comfortable for therapists to practice psychother-
powerful effect over treatment-as-usual with cases that were pre- apy without formal assessment.
dicted to be treatment failures (20%–30% of clients). In these There are a number of limitations in feedback research. The
off-track clients, the base rate for deterioration at the end of research has relied on single self-report measures to characterize
treatment was 20%. Progress feedback with alarm signals indicat- mental health functioning, whereas most psychotherapy outcome
ing a less than expected treatment response reduced deterioration research assesses outcome with multiple measures reflecting mul-
by 50% (to 9%) and increased positive outcomes from 22% in tiple points of view and multiple types of outcome. More ambi-
treatment-as-usual to 38% in the feedback condition. In three of tious assessment of outcome could result in more conservative
the six studies, feedback to therapists included the use of the CST results than we have found through the use of a single self-report
that helped therapists identify reasons for poor therapeutic prog- measure. This methodological limitation is difficult to overcome
ress and provided suggestions. This feedback enabled therapists to because most studies are conducted in routine care clinics with no
identify and intervene differently than in treatment-as-usual. The set treatment lengths or knowledge beforehand on when the final
results of this feedback was rather substantial in that it further session (and therefore the final assessment) will take place. An-
reduced deterioration rates to 5.5%, with over half of the off-track other limitation of the meta-analytic results presented here is that
 OUTCOME ADVANCES 49

the studies that were examined were generated by a single research the studies included an unstandardized measure (one with no prior
group. Because different measures and feedback methods exist and reliability or validity data). A total of 435 unique outcome mea-
researchers are beginning to report results, future research may sures were used in the 163 studies he examined, indicating both the
confirm, disconfirm, or partially confirm research by Lambert and variable ways in which the effects of psychotherapy are assessed
his colleagues. At this time, enough studies have been published to and the lack of agreement about the best measures to capture
suggest that progress feedback is broadly effective as a method of change. In stark contrast to early practices involving evaluation of
improving patient treatment response with patients who are strug- psychoanalytic psychotherapy where therapists were central in
gling to improve. evaluations, therapist assessments of change are now rare. This is
consistent with recommendations from outcome researchers.
In the area of assessment, considerable work needs to be done to
Future Research
help us understand clinically significant change based on popular
Given the preceding review, it follows that I believe a future measures of outcome. For example, just how do change scores on
emphasis is psychotherapy research that concentrates on improv- self-report measures of symptoms reflect broader changes in the
ing outcomes in real time. Rather than continuing with the kind of “deeper” aspects of client functioning? Certainly we have come a
research advocated by Strupp and Eysenck in the initial issues of long way since the first issue of Psychotherapy appeared and we
Psychotherapy in 1963– 64, it is time to focus our attention on have further to go, but we can be proud that our treatments rest on
improving outcome as psychotherapy is ongoing. In contrast to a firm empirical foundation and that scientific efforts in combina-
emphasizing the right psychotherapy for the right disorder, such tion with clinician contributions continue to improve clinical ser-
research assumes that patients do need an empirically supported vices and modify our theoretical understanding of psychotherapy
psychotherapy that is not working for them. Instead, it is important and the change process.
to identify negatively responding and nonresponding individuals
as early as possible in treatment, quickly analyze reasons for these
failures, and alternatives or modifications to the clinical approach. References
Our initial efforts have been to measure generic factors (the Anderson, E. M., & Lambert, M. J. (2001). A survival analysis of clinically
therapeutic alliance, motivational problems, and outside factors significant change in outpatient psychotherapy. Journal of Clinical Psy-
such as social supports), but this work is just at its inception. There chology, 57, 875– 888. doi:10.1002/jclp.1056
may be some theory-specific, clinic-specific, and population spe- Anker, M. G., Duncan, B. L., & Sparks, J. A. (2009). Using client feedback
cific that would also be highly effective and extend this work. For to improve couple therapy outcomes: A randomized clinical trial in a
example, in our local clinic group, therapies abound and are easily naturalistic treatment setting. Journal of Consulting and Clinical Psy-
accessible for patients with social support problems. So if prob- chology, 77, 693–704. doi:10.1037/a0016062
lems with social support are identified, a referral can be made. Our Bandura, A. (1969). Principles of behavior modification. New York, NY:
Holt, Rinehart & Winston.
measure, however, does not identify whether the problem in this
Barber, J., Muran, C., Keefe, J., & McCarthy, K. (2013). Psychodynamic
area is due to needed more contact with others or whether a group
approaches. In M. J. Lambert (Ed.), Bergin & Garfield’s Handbook of
that focusses on social skill deficits would be more helpful. In psychotherapy and behavior change (6th ed.). New York: Wiley.
either case, identifying what is going wrong and creating within a Barkham, M., Margison, F., Leach, C., Lucock, M., Mellor-Clark, J.,
specific clinic some intervention possibilities that routinely work is Evans, C., . . . McGrath, G. (2001). Service profiling and outcomes
a creative clinical and research task. A recent examination of benchmarking using the CORE_OM: Toward practice-based evidence in
treatment failures in cognitive behavior therapies provides an the psychological therapies. Journal of Consulting and Clinical Psychol-
example of a similar approach. Dimidjian and Hollon (2011) ogy, 69, 184 –196. doi:10.1037/0022-006X.69.2.184
brought together a series of articles on treatment failure, with many Beck, A. T. (1970). Cognitive therapy: Nature and relation to behavior
of the contributors analyzing the reasons for failure and steps that therapy. Behavior Therapy, 1, 184 –200. doi:10.1016/S0005-
you could be taken with similar cases in the future. There is much 7894(70)80030-2
Beck, A. T. (1976). Cognitive therapy and the emotional disorders. New
to be learned from studying treatment failure.
York: International Universities Press.
Anyone familiar with the scientific study of psychotherapy Beckstead, D. J., Hatch, A. L., Lambert, M. J., Eggett, D. L., Goates,
treatment effects is aware of the importance of measurement—the M. K., & Vermeersch, D. A. (2003). Clinical significance of the Out-
operational definition of impact and change. Just how many pa- come Questionnaire (OQ-45.2). The Behavior Analyst Today, 4, 79 –90.
tients are estimated to have improved following a course of treat- Bergin, A. E. (1971). The evaluation of therapeutic outcomes. In A. E.
ment is directly related to what is measured and with what instru- Bergin & S. L. Garfield (Eds.), Handbook of psychotherapy and behav-
ments. This is an area where psychotherapy research has made ior change: An empirical analysis (pp. 217–270). New York: Wiley.
clear advances. In contrast to the research results discussed by Brown, G. S., & Lambert, M. J. (1998, June). Tracking patient progress:
Strupp and Eysenck in 1963/4 that often relied on therapist ratings, Decision making for cases that are not benefiting from psychotherapy.
contemporary research has used multiple standardized measures Paper presented at the annual meeting of the Society for Psychotherapy
Research, Snowbird, UT.
from a variety of sources to provide comprehensive assessments of
Castonguay, L., Barkham, M., Lutz, W., & McAleavey, A. (2013).
change. Ogles (2013), based on a review of outcome studies
Practice-oriented research: Approaches and applications. In M. J. Lam-
published over a 3-year period, found that the typical study used bert (Ed.), Bergin & Garfield’s Handbook of psychotherapy and behav-
multiple measures, with an average of 3.89 measures of outcome ior change (6th ed.). New York: Wiley.
per study, with a range from 1 measure to 14. Twenty-four percent Crits-Christoph, P., Ring-Kurtz, S., Hamilton, J., Lambert, M. J., Gallop,
of the outcome studies included an outcome from a significant R., McClure, B., . . . Rotrosen, J. A. (2012). Preliminary study of the
other such as spouse, parent, and so forth. Importantly, only 5% of effects of individual patient-level feedback in outpatient substance abuse
50 LAMBERT

treatment programs. Journal of Substance Abuse Treatment, 42, 301– pists and patients. Psychotherapy Research, 14, 308 –327. doi:10.1093/
309. doi:10.1016/j.jsat.2011.09.003 ptr/kph027
Cuijpers, P., & Dekker, J. (2005). Psychological treatment of depression; a Hill, C. E., & Castonguay, L. (2010). Transformation in psychotherapy.
systematic review. Nederlands Tijdschrift Voor Geneeskunde, 149, Washington, DC: APA Press.
1892–1897. Hollon, S. D., & Beck, A. T. (2013). Cognitive and cognitive behavior
Cuijpers, P., Driessen, E., Hollon, S. D., van Oppen, P., Barth, J., & therapies. In M. J. Lambert (Ed.), Bergin & Garfield’s Handbook of
Andersson, G. (2012). The efficacy of non-directive supportive psycho- psychotherapy and behavior change (6th ed.). New York: Wiley.
therapy for adult depression: A met-analysis. Clinical Psychology Re- Hunt, D. L., Haynes, B., Hanna, S. E., & Smith, K. (1998). Effects of
view, 32, 280 –291. computer-based clinical decision support systems on physician perfor-
Cuijpers, P., van Straten, A., van Oppen, P., & Andersson, G. (2008). mance and patient outcomes. Journal of the American Medical Associ-
Psychotherapy for depression in adults: A meta-analysis of comparative ation, 280, 1339 –1346. doi:10.1001/jama.280.15.1339
outcome studies. Journal of Consulting and Clinical Psychology, 76, Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical
909 –922. doi:10.1037/a0013075 approach to defining meaningful change in psychotherapy research.
de Jong, K., Nugter, M. A., Polak, M. G., Wagenborg, J. E. A., Spinhoven, Journal of Consulting and Clinical Psychology, 59, 12–19. doi:10.1037/
P., & Heiser, W. J. (2007). The Outcome Questionnaire (OQ-45) in a 0022-006X.59.1.12
Dutch population: A cross-cultural validation. Clinical Psychology & Jones, M. C. (1924). The elimination of children’s fears. Journal of
Psychotherapy, 14, 288 –301. doi:10.1002/cpp.529 Experimental Psychology, 7, 382–390. doi:10.1037/h0072283
Dimidjian, S., & Hollon, S. D. (2011). What can be learned when empir- Kendall, P. C., Marrs-Garcia, A., Nath, S. R., & Sheldrick, R. C. (1999).
ically supported treatments fail? Cognitive and Behavioral Practice, 18, Normative comparisons for the evaluation of clinical significance. Jour-
303–305. doi:10.1016/j.cbpra.2011.02.001 nal of Consulting and Clinical Psychology, 67, 285–299. doi:10.1037/
Ellis, A. (1962). Reason and emotion in psychotherapy. Oxford, England: 0022-006X.67.3.285
Lyle Stuart. Kivlighan, D. M. (1985). Feedback in group psychotherapy: Review and
Ellsworth, J. R., Lambert, M. J., & Johnson, J. (2006). A comparison of the implications. Small Group Behavior, 16, 373–385. doi:10.1177/
Outcome Questionnaire-45 and Outcome Questionnaire-30 in classifi- 0090552685163007
cation and prediction of treatment outcome. Clinical Psychology and Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions
Psychotherapy, 13, 380 –391. doi:10.1002/cpp.503 on performance: A historical review, a meta-analysis, and a preliminary
Emmelkamp, P. M. G. (2013). Behavior therapy with adults. In M. J. feedback intervention theory. Psychological Bulletin, 119, 254 –284.
Lambert (Ed.), Bergin & Garfield’s Handbook of psychotherapy and doi:10.1037/0033-2909.119.2.254
behavior change (6th ed.). New York: Wiley. Kordy, H., Hannöver, W., & Richard, M. (2001). Computer-assisted
Evans, R., Mellor-Clark, J., Barkham, M., &. Mothersole, G. (2006). feedback-driven quality management for psychotherapy: The Stuttgart-
Developing the resources and management support for routine evalua- Heidelberg model. Journal of Consulting and Clinical Psychology, 69,
tion in counselling and psychological therapy service provision: Reflec- 173–183. doi:10.1037/0022-006X.69.2.173
tions on a decade of core development. European Journal of Psycho- Lambert, M. J. (1976). Spontaneous remission in adult neurotic disorders:
therapy and Counselling, 8, 141–161. doi:10.1080/13642530600712494 A revision and summary. Psychological Bulletin, 83, 107–119. doi:
Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. 10.1037/0033-2909.83.1.107
Journal of Consulting Psychology, 16, 319 –324. doi:10.1037/h0063633 Lambert, M. J. (1983). Introduction to assessment of psychotherapy out-
Eysenck, H. J. (1964). The outcome problem in psychotherapy: A reply. come: Historical perspective and current issues. In M. J. Lambert, E. R.
Psychotherapy: Theory, Research & Practice, 2, 97–100. doi:10.1037/ Christensen, & S. S. DeJulio (Eds.), The assessment of psychotherapy
h0088591 outcome. New York: Wiley.
Flowers, J. V. (1979). The differential outcome effects of simple advice, Lambert, M. J. (2013a). Bergin and Garfields’s Handbook of Psychother-
alternatives and instructions in group psychotherapy. International Jour- apy and Behavior Change (6th ed.) New York: Wiley.
nal of Group Psychotherapy, 29, 305–316. Lambert, M. J. (2013b). The efficacy and effectiveness of psychotherapy.
Haas, E., Hill, R., Lambert, M. J., & Morrell, B. (2002). Do early respond- In M. J. Lambert (Ed.), Bergin & Garfield’s Handbook of Psychotherapy
ers to psychotherapy maintain treatment gains? Journal of Clinical and Behavior Change (6th ed.). New York: Wiley.
Psychology, 58, 1157–1172. doi:10.1002/jclp.10044 Lambert, M. J., Bailey, R., Kimball, K., Shimokawa, K., Harmon, S. C., &
Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. W., Slade, K. (2007). Clinical Support Tool Manual-Brief Version-40. Salt
Shimokawa, K., & Sutton, S. W. (2005). A lab test and algorithms for Lake City, UT: OQMeasures.
identifying clients at risk for treatment failure. Journal of Clinical Lambert, M. J., Bergin, A. E., & Collins, J. L. (1977). Therapist induced
Psychology, 61, 155–163. doi:10.1002/jclp.20108 deterioration in psychotherapy patients. In A. S. Gurman, & A. M. Razin
Hansen, N. B., Lambert, M. J., & Forman, E. V. (2002). The psychotherapy (Eds.), Effective psychotherapy: A handbook of research (pp. 452– 481).
dose-response effect and its implications for treatment delivery services. New York: Pergamon Press.
Clinical Psychology: Science and Practice, 9, 329 –343. doi:10.1093/ Lambert, M. J., Hansen, N. B., & Finch, A. E. (2001). Client-focused
clipsy.9.3.329 research: Using client outcome data to enhance treatment effects. Jour-
Harmon, S. C., Lambert, M. J., Smart, D. W., Hawkins, E. J., Nielsen, nal of Consulting and Clinical Psychology, 69, 159 –172. doi:10.1037/
S. L., Slade, K., & Lutz, W. (2007). Enhancing outcome for potential 0022-006X.69.2.159
treatment failures: Therapist/client feedback and clinical support tools. Lambert, M. J., Morton, J. J., Hatfield, D., Harmon, C., Hamilton, S., Reid,
Psychotherapy Research, 17, 379 –392. R. C., . . . Burlingame, G. M. (2004). Administration and Scoring
Hatfield, D., McCullough, L., Plucinski, A., & Krieger, K. (2010). Do we Manual for the Outcome Questionnaire-45. Salt Lake City, UT:
know when our clients get worse? An investigation of therapists’ ability OQMeasures.
to detect negative client change. Clinical Psychology & Psychotherapy, Lambert, M. J., & Ogles, B. M. (2004). The efficacy and effectiveness of
17, 25–32. psychotherapy. In M. J. Lambert (Ed.), Bergin and Garfield’s handbook
Hawkins, E. J., Lambert, M. J., Vermeersch, D. A., Slade, K., & Tuttle, K. of psychotherapy and behavior change (5th ed., pp. 139 –193). New
(2004). The effects of providing patient progress information to thera- York: Wiley.
 OUTCOME ADVANCES 51

Lambert, M. J., Whipple, J. L., Bishop, M. J., Vermeersch, D. A., Gray, Simon, W., Lambert, M. J., Busath, G., Vazquez, A., Berkeljon, A.,
G. V., & Finch, A. E. (2002). Comparison of empirically derived and Hyer, K., . . . Berrett, M. (unpublished). Effects of providing patient
rationally derived methods for identifying clients at risk for treatment progress feedback and clinical support tools to psychotherapists in an
failure. Clinical Psychology and Psychotherapy, 9, 149 –164. doi: inpatient eating disorders treatment program: A randomized controlled
10.1002/cpp.333 study. Lambert, Department of Psychology, Brigham Young University,
Lambert, M. J., Whipple, J. L., Harmon, C., Shimokawa, K., Slade, K., & Provo, UT.
Christofferson, C. (2004). Clinical Support Tools Manual. Department Slade, K., Lambert, M. J.Harmon, S. C., Smart, D. W., & Bailey, R. (2008).
of Psychology, Brigham Young University, Provo, UT. Improving psychotherapy outcome: The use of immediate electronic
Lambert, M. J., Whipple, J. L., Smart, D. W., Vermeersch, D. A., Nielsen, feedback and revised clinical support tools. Clinical Psychology &
S. L., & Hawkins, E. J. (2001). The effects of providing therapists with Psychotherapy, 15, 287–303.
feedback on client progress during psychotherapy: Are outcomes en- Spielmans, G. I., Masters, K. S., & Lambert, M. J. (2006). A comparison
hanced? Psychotherapy Research, 11, 49 – 68. doi:10.1080/713663852 of rational versus empirical methods in prediction of negative psycho-
Lambert, M. J., Whipple, J. L., Vermeersch, D. A., Smart, D. W., Hawkins, therapy outcome. Clinical Psychology & Psychotherapy, 13, 202–214.
E. J., Nielsen, S. L., & Goates, M. K. (2002). Enhancing psychotherapy Strupp, H. H. (1963). The outcome problem in psychotherapy revisited.
outcomes via providing feedback on client progress: A replication. Psychotherapy: Theory, Research & Practice, 1, 1–13. doi:10.1037/
Clinical Psychology and Psychotherapy, 9, 91–103. doi:10.1002/cpp h0088565
.324 Strupp, H. H. (1964). The outcome problem in psychotherapy: A rejoinder.
Lueger, R. J., Howard, K. I., Martinovich, Z., Lutz, W., Anderson, E. E., (1964). Psychotherapy: Theory, Research, & Practice, 2, 100.
& Grissom, G. (2001). Assessing treatment progress of individual clients Strupp, H. H., & Hadley, S. W. (1977). A tripartite model of mental health
using expected treatment response models. Journal of Consulting and and therapeutic outcomes: With special reference to negative effects in
Clinical Psychology, 69, 150 –158. doi:10.1037/0022-006X.69.2.150 psychotherapy. American Psychologist, 32, 187–196. doi:10.1037/0003-
066X.32.3.187
Lunnen, K. M., & Ogles, B. M. (1998). A multiperspective, multivariable
Tang, T. Z., & DeRubeis, R. J. (1999). Sudden gains and critical sessions
evaluation of reliable change. Journal of Consulting and Clinical Psy-
in cognitive-behavioral therapy for depression. Journal of Consulting
chology, 66, 400 – 410. doi:10.1037/0022-006X.66.2.400
and Clinical Psychology, 67, 894 –904. doi:10.1037/0022-006X.67.6
Lutz, W., Lambert, M. J., Harmon, S. C., Stulz, N., Tschitsaz, A., &
.894
Schürch, E. (2006). The probability of treatment success, failure and
Vermeersch, D. A., Lambert, M. J., & Burlingame, G. M. (2000). Outcome
duration—what can be learned from empirical data to support decision
Questionnaire: Item sensitivity to change. Journal of Personality As-
making in clinical practice? Clinical Psychology & Psychotherapy, 13,
sessment, 74, 242–261. doi:10.1207/S15327752JPA7402_6
223–232.
Vermeersch, D. A., Whipple, J. L., Lambert, M. J., Hawkins, E. J.,
McNeilly, C. L., & Howard, K. I. (1991). The effects of psychotherapy: A
Burchfield, C. M., & Okiishi, J. C. (2004). Outcome Questionnaire: Is it
reevaluation based on dosage. Psychotherapy Research, 1, 74 –78.
sensitive to changes in counseling center clients? Journal of Counseling
Miller, S. D., Duncan, B. L., Sorrell, R., & Brown, G. S. (2005). The
Psychology, 51, 38 – 49. doi:10.1037/0022-0167.51.1.38
partners for change outcome system. Journal of Clinical Psychology: In Walfish, S., McAlister, B., O’Donnell, P., & Lambert, M. J. (2012). An
Session, 61, 199 –208. doi:10.1002/jclp.20111 investigation of self-assessment bias in mental health providers. Psycho-
Mowrer, O. H., & Mowrer, W. M. (1938). Enuresis—A method for its logical Reports, 110, 639 – 644. doi:10.2466/02.07.17.PR0.110.2.639-
study and treatment. American Journal of Orthopsychiatry, 8, 8436 – 644
8459. doi:10.1111/j.1939-0025.1938.tb06395.x Wampold, B. E. (2007). Psychotherapy: The humanistic (and effective)
Newham, E., Hooke, G. R., & Page, A. (2013). Patient monitoring and treatment. American Psychologist, 62, 857– 873. doi:10.1037/0003-
feedback in psychiatric care reduces depressive symptoms. Unpublished 066X.62.8.857
manuscript. Perth Clinic, Perth, Australia. Warren, J. S., Nelson, P. L., Mondragon, S. A., Baldwin, S. A., &
Ogles, B. M. (2013). Measuring change in psychotherapy research. In M. J. Burlingame, G. M. (2010). Youth psychotherapy change trajectories &
Lambert (Ed.), Bergin & Garfield’s Handbook of psychotherapy and outcome in usual care: Community mental health versus managed care.
behavior change (6th ed.). New York: Wiley. Journal of Clinical and Consulting Psychology, 78, 144 –155. doi:
Otto, M. W. Pollack, M. H., & Maki, K. M. (2000). Empirically supported 10.1037/a0018544
treatments for panic disorder: Costs, benefits, and stepped care. Journal Weisz, J. R. (2004). Psychotherapy for children and adolescents:
of Consulting and Clinical Psychology, 68, 556 –563. doi:10.1037/0022- Evidence-based treatments and case examples. New York: Cambridge
006X.68.4.556 University Press. doi:10.1017/CBO9780511734960
Safran, J. D., Muran, J. C., Samstang, L. W., & Winston, A. (2005). Whipple, J. L., Lambert, M. J., Vermeersch, D. A., Smart, D. W., Nielsen,
Evaluating alliance-focused intervention for potential treatment failures: S. L., & Hawkins, E. J. (2003). Improving the effects of psychotherapy:
A feasibility and descriptive analysis. Psychotherapy: Theory, Research, The use of early identification of treatment failure and problem solving
Practice, & Training, 42, 512–531. doi:10.1037/0033-3204.42.4.512 strategies in routine practice. Journal of Counseling Psychology, 58,
Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010). Enhancing 59 – 68. doi:10.1037/0022-0167.50.1.59
treatment outcome of patients at risk of treatment failure: Meta-analytic
and mega-analytic review of a psychotherapy quality assurance system. Received September 24, 2012
Journal of Consulting & Clinical Psychology, 78, 298 –311. Accepted September 25, 2012 䡲