Psychotherapy: Theory, Research, Practice, Training Copyright 2005 by the Educational Publishing Foundation

2005, Vol. 42, No. 3, 340 –356 0033-3204/05/$12.00 DOI: 10.1037/0033-3204.42.3.340

MEASURING PSYCHODYNAMIC–INTERPERSONAL AND

COGNITIVE–BEHAVIORAL TECHNIQUES: DEVELOPMENT
OF THE COMPARATIVE PSYCHOTHERAPY
PROCESS SCALE

MARK J. HILSENROTH MATTHEW D. BLAGYS AND

Adelphi University STEVEN J. ACKERMAN
Erik H. Erikson Institute of the Austen Riggs
Center and Harvard Medical School

DENNIS R. BONGE MARK A. BLAIS

University of Arkansas Massachusetts General Hospital and
Harvard Medical School
Many instruments have been developed the CPPS. The findings suggest that the
to assess techniques and interventions scale possesses excellent interrater reli-
in a variety of psychotherapies. How- ability and internal consistency as well
ever, existing scales are limited by sev- as promising validity. Clinical utility,
eral factors such as relatively weak potential limitations, and future re-
psychometric properties, applicability search of the CPPS are discussed.
to only a single form of treatment or
manual, and extensive time required for Keywords: psychotherapy process, ther-
completion. The authors report on the apist technique, adherence,
development of a new measure, the psychodynamic–interpersonal,
Comparative Psychotherapy Process cognitive– behavioral
Scale (CPPS). The CPPS is designed to
assess the distinctive features of
A number of instruments have been developed
psychodynamic–interpersonal and
to assess treatment adherence, technique, and
cognitive– behavioral treatments. Data process for a variety of therapies (Barber & Crits-
are presented on the psychometric Christoph, 1996; Barber, Liese & Abrams, 2003;
properties, reliability, and validity of DeRubeis, Hollon, Evans, & Bemis, 1982; Gas-
ton & Ring, 1992; Goldfried, Newman, & Hayes,
1989; Hollon et al., 1988; Jones, 1985; Ogrod-
niczuk & Piper, 1999; O’Malley et al., 1988;
Mark J. Hilsenroth, Derner Institute of Advanced Psycho-
logical Studies, Adelphi University; Matthew D. Blagys and Samoilov, Goldfried, & Shapiro, 2000; Shapiro
Steven J. Ackerman, Erik H. Erikson Institute of the Austen & Startup, 1990; Shaw et al., 1999; Young &
Riggs Center and Harvard Medical School; Dennis R. Bonge, Beck, 1980). However, some of these scales are
Department of Psychology, University of Arkansas; Mark A. limited by various factors. First, some existing
Blais, Department of Psychiatry, Massachusetts General Hos- measures suffer from relatively low interrater re-
pital and Harvard Medical School. liability coefficients (intraclass correlation coef-
Additional materials are on the Web at http://dx.doi.org/ ficients in the poor–fair range, ⬍ .60; Fleiss,
10.1037/0033-3204.42.3.340.supp
1981) for specific scales or items (Barber &
Correspondence regarding this article should be addressed
to Mark J. Hilsenroth, PhD, ABAP, Derner Institute of Ad- Crits-Christoph, 1996; Barber et al., 2003; Crits-
vanced Psychological Studies, Adelphi University, 220 Wein- Christoph et al., 1998; Gaston & Ring, 1992;
berg Building, 158 Cambridge Avenue, Garden City, NY Goldfried, Castonguay, Hayes, Drozd, & Sha-
11530. E-mail: hilsenro@adelphi.edu piro, 1997; Goldfried, Raue, & Castonguay,

340
 CPPS Development

1998; Hollon et al., 1988; Jacobson, 1998; Jacob- cusing on patients’ cognitive experiences (e.g., dys-
son et al., 1996; Ogrodniczuk & Piper, 1999; functional or irrational beliefs).
Shaw et al., 1999; Vallis, Shaw, & Dobson, CPPS items were written to reflect the
1986). Second, measures of adherence are, by between-treatment differences identified in the
definition, wedded to a specific manual or treat- empirical literature reviews in order to assess the
ment (Waltz, Addis, Koerner, & Jacobson, 1993). distinctive features of these alternative ap-
Therefore, the applicability and utility of these proaches to therapy. The scale is intended to be a
instruments to more real-world therapies that (a) brief measure assessing the degree to which any
do not use a treatment manual, (b) do not use the delivered therapy used global techniques and ad-
specific manual for which the instrument was hered to the distinctive features of PI and CB
developed, or (c) use techniques from different treatments. Thus, the scale has the potential to (a)
manualized treatments (both within or across the- classify treatments broadly as either PI or CB, (b)
oretical orientations) are largely unknown. Third, compare various types of CB and PI therapies,
some existing instruments include a relatively and (c) examine the effects of PI and CB tech-
large number of items that require a substantial niques in a given session or treatment. The CPPS
amount of time to be completed and may limit is not intended to replace existing manual specific
their applied utility (Barber & Crits-Christoph, instruments but rather is intended to offer a reli-
1996; Goldfried et al., 1989; Hollon et al., 1988; able alternative that has general real-world appli-
Jones, 1985; Shapiro & Startup, 1990). cability to a variety of treatments (both within
The current study is a psychometric investigation and across theoretical orientations).
of a new measure, the Comparative Psychotherapy The CPPS is distinctive from other existing
Process Scale (CPPS). The scale is based on the measures in several ways. First, unlike adherence
findings of two reviews of the empirical compara- and competence instruments such as the Cogni-
tive psychotherapy process literature (Blagys & tive Therapy Adherence and Competence Scale
(Liese, Barber, & Beck, 1995), Collaborative
Hilsenroth, 2000, 2002). These reviews sought to
Study Psychotherapy Rating Scale (CSPRS; Hol-
identify significant differences between the tech-
lon et al., 1988), Sheffield Psychotherapy Rating
niques used in psychodynamic–interpersonal (PI;
Scale (SPRS; Shapiro & Startup, 1990), Cogni-
defined broadly to include psychodynamic,
tive Therapy Scale (Young & Beck, 1980), and
psychodynamic–interpersonal, and interpersonal the Penn Adherence-Competence Scale for
therapies) and cognitive– behavioral (CB; defined Supportive–Expressive Psychotherapy (PACS-
broadly to include cognitive, cognitive– behavioral, SE; Barber & Crits-Christoph, 1996), the CPPS
and behavioral therapies) treatments. Blagys and was not designed to assess the interventions of a
Hilsenroth (2000) identified seven general tech- specific treatment and its corresponding manual.
niques that consistently and significantly distin- Rather, it was developed to be a more general
guished PI from CB treatments: (a) focusing on instrument with applicability to different forms of
patients’ affect and the expression of emotion; (b) therapy. In this respect, the CPPS may be partic-
exploring patients’ attempts to avoid topics or en- ularly useful for examining real-world (i.e., not
gage in activities that hinder the progress of treat- manualized) treatments. Second, some instru-
ment; (c) focusing session on patterns in patients’ ments assess techniques from only one theoreti-
actions, thoughts, feelings, and relationships; (d) cal perspective (i.e., either PI or CB). For exam-
exploring patients’ past experiences; (e) focusing on ple, the PACS-SE (Barber & Crits-Christoph,
patients’ interpersonal experiences; (f) focusing dis- 1996) and the Interpretive and Supportive Tech-
cussion on the therapeutic relationship; and (g) ex- nique Scale (Ogrodniczuk & Piper, 1999) were
ploring patients’ wishes, dreams, or fantasies. designed to assess the interventions of psychody-
Blagys and Hilsenroth (2002) reported six general namic treatments. Therefore, the applicability
techniques that consistently and significantly distin- and relevance of these measures to CB treatments
guished CB from PI treatments: (a) assigning home- are limited. In contrast, the CPPS assesses key
work and outside of session activities; (b) actively aspects of both PI and CB treatments, allowing
directing session activity; (c) teaching specific cop- for comparisons between these different forms of
ing skills; (d) focusing on patients’ future experi- therapy.
ences; (e) providing patients with information about A third distinctive feature of the CPPS is its
their treatment, disorder, or symptoms; and (f) fo- focus on the distinguishing features of PI and CB

341
Hilsenroth, Blagys, Ackerman, Bonge, and Blais

therapies as found in the empirical literature. That across several different contexts, to address our a
is, this measure was developed from actual re- priori predictions that (a) the CPPS PI subscale will
search findings (i.e., bottom-up) rather than demonstrate a significant relationship with other
solely theoretical expectations (i.e., top-down). therapist activity measures evaluating similar forms
As noted by Waltz et al. (1993), the evaluation of of psychodynamic or supportive– expressive (SE)
interventions both prescribed and proscribed by a psychotherapy; (b) the CPPS items and subscales
treatment is important for capturing the complex- will be able to distinguish between different thera-
ity of a therapy session and assessing treatment peutic modalities; (c) the CPPS PI and CPPS CB
adherence. In this respect, the CPPS may be said scales will delineate prototypes of each respective
to include items described by Waltz et al. (1993) approach to psychotherapy process as rated by ex-
as “unique and essential” and “proscribed” by PI pert clinicians; (d) the CPPS will allow naı̈ve (i.e.,
and CB approaches, respectively. The CPPS is undergraduate) raters to distinguish the session
the only measure of which we are aware that characteristics of PI and CB treatments; (e) patients
attempts to quantify and assess the specific tech- in psychodynamic psychotherapy will report signif-
niques that make PI and CB treatments different. icantly more PI than CB treatment techniques dur-
Therefore, the CPPS may be especially useful in ing their sessions; (f) therapists’ and supervisors’
assessing the amount of unique aspects of PI and ratings of both PI and CB treatment techniques will
CB treatment included in a session. This is an be significantly related to one another.
important distinction given the results of several
studies that suggest features of an alternative
treatment may be unintentionally provided in a Method
therapy (e.g., such as psychodynamic techniques Participants
used in a CB treatment session) and contribute to
treatment outcome independent of the intended, Two patient samples were used in the initial
treatment-specific interventions (e.g., Ablon & reliability and validity analyses. The first con-
Jones, 1998; Castonguay, Goldfried, Wiser, sisted of 36 patients (18 women, 18 men) con-
Raue, & Hayes, 1996; Hayes, Castonguay, & secutively admitted to a psychodynamic psycho-
Goldfried, 1996; Hayes & Strauss, 1998; Jones & therapy treatment team (PPTT; Hilsenroth, 2002)
Pulos, 1993). Last, the CPPS was designed to be over a 26-month period at a university-based
a brief and efficient measure. In contrast, several community outpatient psychological clinic. Six-
existing measures contain a large number of teen individuals were single, 11 were married,
items that are relatively time consuming and la- and 9 were divorced. The mean age for this
bor intensive such as the Psychotherapy Process psychodynamic (PD) treatment group was 31.14
Q-Sort (Jones, 1985), CSPRS (Hollon et al., years (SD ⫽ 11.50 years).
1988), PACS-SE (Barber & Crits-Christoph, To provide a comparison for this PD sample,
1996), SPRS (Shapiro & Startup, 1990), and the another group of participants was included. This
Coding System of Therapeutic Focus (Goldfried second sample consisted of 6 patients from the
et al., 1989). same outpatient clinic admitted to other (nonpsy-
Although there have already been several exam- chodynamic) treatment practicums during the
ples of the clinical validity of the CPPS, across two same 26-month period. The 6 patients comprising
separate research groups (Ackerman, Hilsenroth, & this group were chosen because they each had
Knowles, 2005; Hilsenroth, Ackerman & Blagys, videotaped sessions available and were described
2001; Hilsenroth, Ackerman, Blagys, Baity, & by the therapists conducting the treatment and
Mooney, 2003; Hilsenroth, DeFife, Blagys, & Ack- their supervisors as nonpsychodynamic treat-
erman, in press; Price, Hilsenroth, Callahan, ments. All 6 patients in this nonpsychodynamic
Petretic-Jackson, & Bonge, 2004; Thompson- (non-PD) treatment sample were single, with a
Brenner & Westen, in press, 2005; Westen, No- mean age of 26.05 years (SD ⫽ 7.77 years).
votny, & Thompson-Brenner, 2004), the specific Patients were not randomly assigned to the PD
aims of the current study are to present original data and non-PD treatment conditions. Rather, they
on the CPPS along two related lines of interest. were assigned to student clinicians of various
First, we present the initial reliability and psycho- treatment practicums in an ecologically valid
metric properties from the CPPS development. Sec- manner based on clinicians’ availability and were
ond, we examine six separate validity analyses, accepted into treatment regardless of disorder or

342
 CPPS Development

comorbidity. In each treatment sample, semi- participants in and the supervisor of the PPTT;
structured diagnostic interviews were used to (c) 1 to 2 hr per week of self-critique in which
evaluate symptomatology from the Diagnostic therapists were encouraged to review their own
and Statistical Manual of Mental Disorders (4th session videotape; (d) additional supervision time
ed.; DSM–IV; American Psychiatric Association, provided by the PPTT supervisor on an as-needed
1994; for a more detailed description of this as- basis; and (e) an optional 1.5 hr per week con-
sessment process, see Hilsenroth, 2002; Hilsen- tinuing case conference on a training case pro-
roth, Peters, & Ackerman, 2004). The DSM–IV vided by the PPTT supervisor (for a more de-
Axis I diagnoses in the PD therapy patient sample tailed description of this training process, see
were as follows: mood disorder (n ⫽ 24), adjust- Hilsenroth et al., in press).
ment disorder (n ⫽ 5), V-code relational problem The treatment provided by the PPTT therapists
(n ⫽ 5), anxiety disorder (n ⫽ 1), and substance was psychodynamic. Although therapists were
abuse (n ⫽ 1). In addition, 17 individuals were aided and informed by the technical guidelines
diagnosed with an Axis II personality disorder, delineated in the training materials noted previ-
and 8 others were found to have subclinical per- ously, they were not used to prescribe session
sonality disorder traits or features. The DSM–IV activity. As such, the treatment provided was not
Axis I diagnoses in the non-PD therapy sample manualized. Rather, the training materials were
were as follows: anxiety disorder (n ⫽ 3), so- used to teach psychodynamic technique and in-
matoform disorder (n ⫽ 2), V-code problems terventions from a variety of PD approaches, and
related to abuse or neglect (n ⫽ 2), substance theories were used and integrated into the treat-
abuse disorder (n ⫽ 1), sexual and gender iden- ment in a flexible, case-based manner. Treatment
tity disorders (n ⫽ 1), and V-code relational focused on personal insight, support, expression
problems (n ⫽ 1). In addition, 3 individuals were of emotion, interpretation, and interpersonal–
diagnosed with an Axis II personality disorder, relational patterns and themes. Each patient re-
and 2 others had subclinical personality disorder ceived one or two sessions of psychotherapy per
traits or features. All patients receiving services week, and treatment was open ended in length.
at the clinic where these data were collected (not The length of treatment was determined by the
just the participants in this research project) give clinician’s judgment, the patient’s decision to end
their informed consent to be included in program therapy, the patient’s progress toward his or her
evaluation research and to have their therapy goals, and changes in the patient’s life.
sessions videotaped. Therapists in the non-PD treatment group were
5 advanced graduate students (2 women, 3 men)
Therapists, Training, and Treatment enrolled in the same APA-accredited clinical psy-
chology doctoral program. Training of the
Therapists in the PPTT were 13 advanced non-PD therapists was provided by supervisors
graduate students (8 women, 5 men) enrolled in who were licensed clinical psychologists and in-
an American Psychological Association (APA)- cluded (a) a minimum of 1 hr per week of indi-
accredited clinical psychology doctorate pro- vidual supervision with their clinical supervisor,
gram. Training of the PPTT therapists included which included case management, review of ses-
(a) a minimum of 1.5 hr of individual clinical sion videotape, discussion of techniques, inter-
case supervision per week with the PPTT super- ventions, patient process, and treatment planning;
visor (a licensed clinical psychologist), which (b) 2 hr of small-group supervision per week,
included the viewing of session videotape and the which included case review, group discussions,
discussion of case conceptualization, interven- and viewing of videotaped case material; and (c)
tions, and techniques; (b) 2 hr of small-group additional individual supervision provided by the
supervision per week with the PPTT supervisor, clinical supervisor on an as-needed basis. The
which included didactic presentations on the training materials used by the therapists in the
principles and techniques of SE psychotherapy non-PD sample reflect primarily CB approaches
(Luborsky, 1984) accompanied by readings from to psychotherapy and include the works of Beck
the treatment manuals of Luborsky (1984), Book and Freeman (1990); Beck, Rush, Shaw, and
(1998), Wachtel (1993), and Strupp and Binder Emery (1979); Bourne (1995), Davis, Eshelman,
(1984) and illustrated by clinical examples and and McKay (1995); Goldfried and Davison
discussions of videotaped sessions conducted by (1994); Craighead, Craighead, Kazdin, and Ma-

343
Hilsenroth, Blagys, Ackerman, Bonge, and Blais

honey (1994); and Masters, Burish, Hollon, and apy. The CPPS CB subscale is expected to mea-
Rimm (1987). Therapists in the non-PD treatment sure techniques and therapist activities that are
condition were also encouraged, but not required, emphasized significantly more in CB than PI
to review their own session videotape. treatment.
Therapists in the PPTT were part of an ongo- In completing the CPPS, the rater’s task is to
ing psychodynamic treatment alliance, process, search for evidence that a therapist activity has
and outcome study, which required more super- occurred in the session. Although a manual for
vision time and training. Therapists in the scoring criteria1 provides greater details, a gen-
non-PD treatment group, in contrast, were not eral principle across each item is that a score of 1
involved in a treatment study and generally re- or 2 suggests some attempt by the therapist to
ceived supervision as usual in the university- engage in the behavior or action delineated by the
based community outpatient clinic. As such, ther- item, with limited follow-up exploration. A score
apists in this treatment condition may have of 4 indicates that the technique tapped by the
received somewhat less supervision than those in item is addressed on separate occasions by the
the PPTT. The therapy provided in this group was therapist with some follow-up investigation. A
nonpsychodynamic. Treatment focused on symp- score of 5 or 6 suggests continued efforts by the
tom reduction, directly challenging dysfunc- therapist to exhibit the behavior or action indi-
tional, illogical, or irrational thoughts–thinking, cated by the item with sustained follow-up.
actively having patients’ accept responsibility for
their actions– behaviors, teaching specific coping
skills, exposure methods, relaxation techniques, Judges and Procedure
and homework. For 3 of the 6 non-PD patients,
Judges were two advanced clinical psychology
treatment was of a fixed duration (one 16 and two
graduate students enrolled in an APA-approved
12 session treatments) as agreed on by the patient
clinical psychology doctoral program. Before the
and therapist. The remaining 3 patients received
rating sessions included in the current study, the
treatment that was open ended in length. As in the
two coders underwent supervised training in the
PD treatment group, the treatment of the patients
use of the manual and rating video recordings of
in the non-PD group was not manualized. Rather,
both PI and CB sessions using the CPPS. Fifteen
the training materials were used in a flexible
sessions independently rated by the two judges
manner based on the needs of the patients.
during the training phase of the project were used
to compute a preliminary analysis of interrater
CPPS reliability (intraclass correlation coefficient) on
which judges showed a good level of initial in-
As described, the CPPS (see Appendixes) is
terrater agreement (⬎ .60; Fleiss, 1981). The 15
based on the findings of two reviews of the em-
training sessions were obtained from three differ-
pirical comparative psychotherapy process liter-
ent sources: (a) PI and CB training tapes from the
ature (Blagys & Hilsenroth, 2000, 2002). The
APA Psychotherapy Videotape Series; (b) other
CPPS is a brief measure of the distinctive fea-
available training tapes of expert PI and CB ther-
tures of PI and CB treatments designed to assess
apists; and (c) videotaped sessions of patients
therapist activity and techniques used and occur-
included in the larger programmatic PPTT study
ring during the therapeutic hour. The measure
(these sessions were not included in the statistical
consists of 20 randomly ordered items rated on a
analyses of the current study). After the prelimi-
7-point Likert-type scale (0 ⫽ not at all charac-
nary interrater reliability analysis, judges began
teristic; 2 ⫽ somewhat characteristic; 4 ⫽ char-
rating videotaped sessions of patients in the re-
acteristic; 6 ⫽ extremely characteristic). The
search study.
CPPS may be completed by a patient, therapist,
For the 36 patients receiving PD therapy, vid-
or external rater. Based on the previously de-
eotapes of the 3rd, 9th, 15th, 21st, 27th, 36th, and
scribed reviews of the empirical literature, the
57th sessions were used when available. The total
CPPS was constructed to contain two subscales:
number of sessions viewed and rated for each
one measuring PI features and one measuring CB
features. The CPPS PI subscale is expected to
measure therapist activities and techniques em- 1
The manual is available on the Web at http://dx.doi.org/
phasized significantly more in PI than CB ther- 10.1037/0033-3204.42.3.340.supp

344
 CPPS Development

patient depended on the length of his or her reliability meetings were held during the coding
treatment. In all, 105 sessions were viewed and process to prevent rater drift.
rated in the PD therapy sample. For the 6 patients
in the non-PD treatment group, sessions were
rated based on their availability. In all, 19 ses- Results
sions (Sessions 1–3, 2–3, 8 –9, 8 –12, 9 –12, 21, Reliability
30, 31) were gathered, viewed, and rated, for a
sample of 124 rated therapy sessions. Of these On the basis of 80 sessions rated by both
124 sessions, 80 were rated by both judges and 44 judges, the interrater reliability of the CPPS PI
were rated by one of the two judges. Although we and CPPS CB was examined using the two-way
recognize that the data collection for the non-PD random-effect model intraclass correlation coef-
treatment sample was not as standardized as in ficient [ICC (2, 1); (Shrout & Fleiss, 1979)] as
the PD sample, a comparison group was included well as Spearman-Brown correction for the two-
to provide a greater range of scores on the CPPS way random-effect model ICC representing the
and to provide a preliminary investigation of the mean reliability across two raters [ICC (2, 2);
validity of the CPPS. It is important to note that (Shrout & Fleiss, 1979)]. As shown in Tables 1
this procedure and the size of this non-PD sample and 2, all of the CPPS PI and CPPS CB items
are comparable to those reported in two other achieved ICC (2, 1) values in the good (.60 –.74;
investigations of PD treatment adherence and ac- Fleiss, 1981) to excellent (ⱖ .75; Fleiss, 1981)
tivity (Barber & Crits-Christoph, 1996; Barber, range, and all ICC (2, 2) values were in the
Crits-Christoph, & Luborsky, 1996). excellent (ⱖ .75) range. In addition, all mean
For each patient, psychotherapy sessions were ICCs for the CPPS PI and CPPS CB items were
arranged in random order and entire sessions also in the excellent range, as were the ICCs for
were viewed by the two judges. Immediately the CPPS PI and CPPS CB total subscale scores.
after viewing a videotaped session, judges inde-
pendently completed the CPPS, and raters alter- Psychometric Characteristics
nated their completion of the CPPS items (i.e.,
1–20, then 20 –1) and completed the CPPS in a Descriptive statistics. Means and standard
random order to decrease order effects. Regular deviations were computed for each of the CPPS

TABLE 1. Interrater Reliability, Descriptive Statistics, and Adjusted Item-to-Scale Correlations for the CPPS PI Subscale

Non-PD
ICC PD sample sample Adjusted
Item item-to-scale
no. Item (2, 1) (2, 2) M SD M SD r

1 Explore uncomfortable feelings .90 .95 4.47 1.23 2.08 1.39 .87
4 Feelings & percepts linked to past exp. .79 .88 3.21 1.45 1.24 0.96 .74
5 Similar relationships over time .84 .91 3.76 1.34 1.32 1.06 .84
7 Focus on patient–therapist relationship .91 .95 2.57 2.08 .66 1.14 .38
8 Experience and expression of feelings .90 .94 4.75 1.10 1.79 1.12 .87
10 Address avoid topics & shift in mood .66 .80 2.12 1.39 .53 0.77 .63
13 Alternative understanding of experiences .75 .85 3.72 1.13 2.13 1.04 .78
14 Recurrent patterns of action/feel/exp. .75 .85 4.02 1.23 2.16 1.11 .81
16 Patient initiates discussion .85 .90 4.38 0.75 1.82 0.98 .56
19 Explore wish, fantasy, dream, EM .80 .89 3.35 1.22 1.13 1.27 .73
Mean CPPS PI .82 .89 3.63 0.90 1.48 0.66 .72
CPPS PI subscale .93 .97 36.33 8.98 14.84 6.55
CPPS PI: coefficient alpha .92
Note. Items are abbreviated in table because of formatting requirements; see Appendixes for complete item descrip-
tions. CPPS PI ⫽ Comparative Psychotherapy Process Scale Psychodynamic–Interpersonal subscale; ICC (2, 1) ⫽
intraclass correlation coefficient two-way random-effects model; ICC (2, 2) ⫽ intraclass correlation coefficient
Spearman-Brown correction for the two-way random-effects model; PD ⫽ psychodynamic. n for ICC ⫽ 80. n for CPPS
PI item means and SD for PD sample ⫽ 105 sessions. n for CPPS PI item means and SD for the non-PD sample ⫽ 19
sessions. N for adjusted item-to-scale correlations ⫽ 124.

345
Hilsenroth, Blagys, Ackerman, Bonge, and Blais

TABLE 2. Interrater Reliability, Descriptive Statistics, and Adjusted Item-to-Scale Correlations for the CPPS CB Subscale

Non-PD
ICC PD sample sample Adjusted
Item item-to-scale
no. Item (2, 1) (2, 2) M SD M SD r

2 Explicit advice or direct suggestion .79 .88 1.06 0.98 3.71 0.73 .79
3 Therapist initiation of topics and activity .82 .89 2.17 0.88 4.37 0.47 .64
6 Focus on irrational/illogical belief system .70 .82 1.41 0.91 4.26 1.02 .75
9 Specific outside-of-session activity or task .89 .94 .37 0.66 4.16 1.40 .89
11 Explain rationale, technique, or treatment .81 .89 .75 0.87 3.05 0.96 .76
12 Focus primarily on current life situations .67 .79 3.67 1.26 4.05 0.85 .12
15 Provide information symp, disorder, or tx .81 .89 .99 0.93 4.18 1.17 .83
17 Practice behaviors between sessions .91 .95 .28 0.58 4.11 1.09 .91
18 Teach specific techniques to patient .95 .97 .29 0.55 3.58 1.77 .83
20 Interacts in teacher-like (didactic) manner .89 .93 .67 0.67 4.24 1.10 .89
Mean CPPS CB .82 .90 1.17 0.43 3.97 0.60 .74
CPPS CB subscale .95 .98 11.66 4.31 39.71 5.98
CPPS CB: coefficient alpha .94
Note. Items are abbreviated in table because of formatting requirements; see Appendixes for complete item descrip-
tions. CPPS CB ⫽ Comparative Psychotherapy Process Scale Cognitive–Behavioral subscale; ICC (2, 1) ⫽ intraclass
correlation coefficient two-way random-effects model; ICC (2, 2) ⫽ intraclass correlation coefficient Spearman-Brown
correction for the two-way random-effects model; PD ⫽ psychodynamic. n for ICC ⫽ 80. n for CPPS CB item and
subscale means and SD in PD sample ⫽ 105 sessions. n for CPPS CB item and subscale means and SD in the non-PD
sample ⫽ 19 sessions. N for adjusted item-to-scale correlations ⫽ 124.

items as well as the CPPS PI and CPPS CB scales and are shown in Tables 1 and 2. For the
subscale scores in the PD and non-PD samples. CPPS PI, all of the items achieved adjusted item-
As shown in Table 1, CPPS PI item mean scores to-scale correlations greater than .30. Similarly,
ranged from a low of 2.12 to a high of 4.75 in the for the CPPS CB subscale, all of the items ob-
PD sample. In the non-PD sample, CPPS PI item tained adjusted item-to-scale correlations greater
means ranged from a low of .53 to a high of 2.16. than .30 except for the original version of Item 12
The mean for the CPPS PI subscale score was (“The focus of session is primarily on current life
3.63 in the PD sample and 1.48 in the non-PD situations”; r ⫽ .12, p ⫽ .19).
sample. Table 2 shows the descriptive statistics Coefficient alpha is also an internal reliability
for the CPPS CB items and subscale in the PD statistic that provides an evaluation of the internal
and non-PD sample. The means for the CPPS CB consistency of the items defining a subscale. A
items ranged from a low of 0.28 to a high of 3.67 coefficient alpha of .70 or greater is generally con-
in the PD sample. In the non-PD sample, CPPS sidered to represent adequate internal consistency
CB item means ranged from a low of 3.05 to a (Nunnally & Bernstein, 1994). In addition to dis-
high of 4.37. The mean for the CPPS CB subscale playing the adjusted item-to-scale correlations
score was 1.17 in the PD sample and 3.97 in the noted previously, coefficient alphas were computed
non-PD sample. for the CPPS PI and CPPS CB subscales. As shown
Internal consistency. Adjusted item-to-scale in Tables 1 and 2, the coefficient alpha based on the
correlations provide an estimate of the conver- sample of 124 rated sessions for the CPPS PI and
gence between the item being evaluated and the CPPS CB were .92 and .94, respectively, indicating
rest of the items in its subscale. In calculating the high internal consistency.
adjusted item-to-scale correlations, the item be-
ing evaluated was excluded from the total sub- Initial Validation
scale score so as not to inflate the correlation. An
item is considered to possess adequate conver- Concurrent validity of the CPPS PI subscale.
gence if its adjusted item-to-scale correlation is To evaluate the concurrent validity of the CPPS
equal to or greater than .30 (Nunnally & Bern- PI subscale, the relationships between this sub-
stein, 1994). Adjusted item-to-scale correlations scale and other therapist activity measures eval-
were then computed based on the sample of 124 uating similar forms of psychodynamic or SE
rated sessions for the CPPS PI and CPPS CB psychotherapy, as found in this study, were ex-

346
 CPPS Development

amined using Pearson r correlations. Following VTSS. Large effect size correlations (⬎.50; Co-
the same procedures detailed previously, 76 vid- hen, 1988) were found between the CPPS PI with
eotaped sessions were rated by both judges the VTSS TLDP specific strategies scales as well
on the Vanderbilt Therapeutic Strategies Scale as the adherence and competence of expressive
(VTSS; Butler, Henry, & Strupp, 1992) and the techniques. Moderate (⬎.30; Cohen, 1988) to
PACS-SE (Barber & Crits-Christoph, 1996). large effect size correlations were found between
Psychotherapy sessions were arranged in ran- the CPPS PI with the adherence and competence
dom order, and entire sessions were viewed by of supportive techniques.
the two judges. Immediately after viewing a Criterion validity. To evaluate the criterion
videotaped session, judges independently rated, validity of the CPPS (i.e., its ability to distinguish
in random order to decrease order effects, the between different therapeutic modalities), an
General Psychodynamic (GD) interviewing analysis of variance (ANOVA) was performed
style and Time Limited Dynamic Psychotherapy comparing the mean item and subscale scores in
(TLDP) specific strategies scales of the VTSS as two different treatment groups. Before conduct-
well as the Supportive–Adherence, Supportive– ing the ANOVA, 2 of the 19 non-PD sample
Competence, Expressive–Adherence, and sessions (both from the same case) were excluded
Expressive–Competence scales of the PAC-SE. from this validity analysis. Although these two
Regular reliability meetings were held during the sessions were allowed to be included in our in-
coding process to prevent rater drift. vestigation of CPPS reliability, at the request of
Table 3 presents the ICC (2, 1) and all ICC (2, the supervising psychologist for this case they
2) values for the VTSS and PACS-SE scales; all were excluded for the between-group compari-
were in the good (.60 –.74) or excellent (ⱖ .75) sons (supervisor consent was also a requirement
range. In addition, Table 3 presents Pearson r for inclusion of data in this study). The remaining
correlations between the VTSS and PACS-SE 17 rated non-PD sessions were matched on pa-
measures of therapist activity with CPPS-PI rat- tient gender, age, global assessment of functional
ings from an early (third or fourth) session of 35 scale, and session number (none significantly dif-
patients in psychodynamic psychotherapy. Find- ferent; p ⬎ .05) with 17 PD sample sessions.
ings from these analyses revealed that the most Mean ratings and standard deviations were
robust relationship was demonstrated between computed for each individual item and subscale
the CPPS-PI and the GD technique scale of the total score across these matched sessions and
were then compared across treatment orienta-
tions. It was hypothesized that the CPPS PI items
TABLE 3. Concurrent Validity Correlations for the CPPS
PI Subscale and total subscale score would be rated signifi-
cantly higher than the CPPS CB items and total
ICC CPPS PIa subscale score in the PD therapy sample. Con-
Scale (2, 1) (2, 2) r p versely, we expected that the CPPS CB items and
total subscale score would be rated significantly
VTSS higher than the CPPS PI items and total subscale
General Psychodynamic
interviewing style .92 .96 .87 ⬍ .0001 score in the non-PD therapy sample. Tables 4 and
TLDP specific strategies .85 .92 .63 ⬍ .0001 5 present the means and standard deviations for
PACS-SE each item for the 17 matched PD and non-PD
Supportive–Adherence .71 .83 .49 .003 sessions, the F value for each planned compari-
Supportive–Competence .79 .88 .49 .003
Expressive–Adherence .85 .92 .84 ⬍ .0001
son, the corresponding p value, and effect sizes
Expressive–Competence .92 .96 .85 ⬍ .0001 for each comparison using Cohen’s d (Cohen,
1988) to provide more clinically relevant infor-
Note. CPPS PI ⫽ Comparative Psychotherapy Process
Scale Psychodynamic–Interpersonal subscale; ICC (2, 1) ⫽
mation. On the basis of Cohen’s (1988) recom-
intraclass correlation coefficient two-way random-effects mendation, d values of .2, .5, and .8 were used to
model; ICC (2, 2) ⫽ intraclass correlation coefficient represent small, medium, and large effects, re-
Spearman-Brown correction for the two-way random- spectively. In the calculation of effect sizes, the
effects model; n for ICC ⫽ 76. VTSS ⫽ Vanderbilt Thera- pooled standard deviation was used in the de-
peutic Strategies Scale; TLDP ⫽ Time Limited Dynamic
Psychotherapy; PACS-SE ⫽ Penn Adherence-Competence nominator. Also, a Bonferroni correction (.05/20)
Scale for Supportive–Expressive Psychotherapy. for this set of CPPS analyses yields a significant
a
Early session, n ⫽ 35. p value of .0025.

347
Hilsenroth, Blagys, Ackerman, Bonge, and Blais

TABLE 4. Analysis of Variance Comparing PD- With Non-PD-Treated Patients on the CPPS PI Subscale

Non-PD
PD (n ⫽ 17) (n ⫽ 17)
Item
no. Item M SD M SD F p d

1 Explore uncomfortable feelings 4.56 0.68 2.06 1.47 40.59 ⬍ .0001 2.19
4 Feelings & percepts linked to past exp. 3.03 1.11 1.09 0.89 31.68 ⬍ .0001 1.93
5 Similar relationships over time 3.62 0.86 1.21 1.06 53.10 ⬍ .0001 2.50
7 Focus on patient–therapist relationship 2.71 2.19 .50 0.85 14.95 .0005 1.33
8 Experience and expression of feelings 4.65 0.77 1.65 1.10 85.10 ⬍ .0001 3.16
10 Address avoid topics & shift in mood 2.21 1.43 .47 0.78 19.38 .0001 1.51
13 Alternative understanding of experiences 3.65 0.93 2.21 1.08 17.43 .0002 1.34
14 Recurrent patterns of action/feel/exp. 3.97 0.93 2.09 1.15 27.65 ⬍ .0001 1.80
16 Patient initiates discussion 4.41 0.44 1.88 1.01 89.79 ⬍ .0001 3.25
19 Explore wish, fantasy, dream, EM 3.18 0.83 1.12 1.34 29.02 ⬍ .0001 1.85
CPPS PI subscale 35.97 5.84 14.27 6.68 101.68 ⬍ .0001 3.46
Note. Items are abbreviated in table because of formatting requirements; see Appendixes for complete item descrip-
tions. CPPS PI ⫽ Comparative Psychotherapy Process Scale Psychodynamic–Interpersonal subscale; PD ⫽ psychody-
namic; EM ⫽ early memory.

As Table 4 illustrates, each of the CPPS PI the mean ratings for the original version of Item
items was found to have significantly higher 12 across the two treatment samples were very
mean ratings in the PD therapy sample than in the similar.
non-PD sample, as did the CPPS PI total subscale Because the original version of Item 12 (“The
score. Also note that all of these differences were focus of session is primarily on current life situ-
representative of large effects (⬎ .80). Similarly, ations”) did not achieve a significant adjusted
Table 5 shows that all but one of the CPPS CB item-to-scale correlation and failed to discrimi-
items were found to have significantly higher nate PD and non-PD therapy in the ANOVA,
mean ratings in the non-PD sample than in the replacing the present-focused Item 12 with one
PD therapy sample, as did the CPPS CB total that more accurately reflects the differences
subscale score. Again, all of the significant dif- found between PI and CB therapy in the literature
ferences were representative of large effects (⬎ reviews may improve the scale and define the
.80). Only the original version of Item 12 (“The CPPS CB subscale more precisely. One possible
focus of session is primarily on current life situ- reason for this finding is that, in the initial liter-
ations”) did not significantly differentiate the two ature reviews, short-term PI and CB treatments
treatments. However, it is important to note that were both found to emphasize a patient’s current

TABLE 5. Analysis of Variance Comparing PD- With Non-PD-Treated Patients on the CPPS CB Subscale

Non-PD
PD (n ⫽ 17) (n ⫽ 17)
Item
no. Item M SD M SD F p d

2 Explicit advice or direct suggestion 1.24 1.37 3.65 0.70 41.70 ⬍ .0001 2.22
3 Therapist initiation of topics and activity 2.12 0.70 4.29 0.75 76.72 ⬍ .0001 3.00
6 Focus on irrational/illogical belief system 1.38 0.60 4.29 1.06 96.92 ⬍ .0001 3.38
9 Specific outside of session activity or task .41 0.85 4.15 1.50 82.50 ⬍ .0001 3.12
11 Explain rationale, technique, or treatment .79 0.56 3.09 0.91 78.89 ⬍ .0001 3.05
12 Focus primarily on current life situations 4.41 0.62 4.06 0.88 1.83 .1900 0.46
15 Provide information symp, disorder, or tx .77 0.89 4.29 1.11 105.59 ⬍ .0001 3.52
17 Practice behaviors between sessions .32 0.77 4.18 1.12 137.15 ⬍ .0001 4.02
18 Teach specific techniques to patient .47 0.65 3.56 1.88 41.06 ⬍ .0001 2.20
20 Interacts in teacher-like (didactic) manner .62 0.80 4.18 1.12 113.94 ⬍ .0001 3.66
CPPS CB subscale 12.53 4.53 39.74 6.08 218.76 ⬍ .0001 5.07
Note. Items are abbreviated in table because of formatting requirements; see Appendixes for complete item descrip-
tions. CPPS CB ⫽ Comparative Psychotherapy Process Scale Cognitive–Behavioral subscale; PD ⫽ psychodynamic.

348
 CPPS Development

life experiences (Ablon & Jones, 1998; Goldfried high level of internal consistency for both the PI
et al., 1997; Jones & Pulos, 1993). The fact that (.89) and CB (.96) subscales. CPPS PI subscale
the PD treatment delivered in the current study scores were significantly different, F(1, 41) ⫽
was generally of shorter duration may account for 107.25, p ⬍ .0001, d ⫽ 3.4, between the PI (n ⫽
the lack of differences between the PD and 30; M ⫽ 5.04, SD ⫽ .57) and CB (n ⫽ 13; M ⫽
non-PD therapy groups in terms of their focus on 3.05, SD ⫽ .61) clinicians’ ratings of a prototyp-
the patients’ current life situation. Therefore, af- ical session. In addition, CPPS CB subscale
ter additional review of this literature, before any scores were also significantly different, F(1,
further analyses, we made a slight modification to 41) ⫽ 269.92, p ⬍ .0001, d ⫽ 5.4, between the PI
the wording of Item 12 so that the revised item (n ⫽ 30; M ⫽ 1.43, SD ⫽ .61) and CB (n ⫽ 13;
now reads “The therapist focuses discussion on M ⫽ 4.75, SD ⫽ .61) clinicians’ ratings of a
the patient’s future life situations.” prototypical session. Thus, given these extreme
effect size differences (i.e., d ⬎ 3 represents
Additional Validation and Clinical Utility different distributions), it seems that the CPPS
was able to robustly differentiate the prototypic
Clinical training staff prototype ratings. Fol- session characteristics of both PI and CB ap-
lowing procedures based on Ablon and Jones proaches to treatment made by clinical training
(1998), we solicited the clinical training staff staff from these theoretical orientations.
from the Department of Psychiatry at Harvard Novice ratings. We sought to examine how
Medical School to “please rate each of the 20 well the CPPS items would perform in use by
items on the scale provided (from 0 – 6) according naı̈ve raters evaluating the session characteristics
to how characteristic each item is of an ideally of PI and CB treatments. One hundred fifteen
conducted session that adheres to the principles undergraduates were recruited to participate in a
of your theoretical orientation.” Forty-three staff study on psychotherapy techniques for course
members (23 women, 20 men) representing a extra credit. Participants were randomized into
wide variety of graduate training in mental health one of two different conditions that watched a
(26 PhDs, 10 MDs, 4 LCSWs, and 3 NPs) re- brief segment (approximately 8 min) of a psycho-
sponded to this request. The responses of these therapy session from a commercially available
training staff on the CPPS were used to ascertain training tape on approaches to psychotherapy in
how well the PI and CB subscales would capture which the same patient received treatment from
prototypes of each respective approach to psy- either a psychodynamic or CB approach to ther-
chotherapy process. apy (Insight Media, 1998). All students watched
In this group of clinical training staff, 30 iden- the video segment in a small-group format, were
tified their primary theoretical orientation as psy- given an opportunity to review the CPPS items
chodynamic, PI, or psychoanalytic and 13 others both before and during the video segment before
as CB, cognitive, or behavioral. All of these making their ratings, and were provided with the
training staff had several years of postgraduate same instructions from a single investigator who
clinical experience (M ⫽ 8 years) as well as an administered all groups (“Please rate each of the
average of 3 years additional postgraduate train- 20 items on the scale provided [from 0 – 6] ac-
ing experience in specific PI or CB treatments cording to how characteristic each item was of
consistent with their major theoretical orienta- the session segment you observed.”)
tion. This group of training staff had also been Of the 115 participants in the study, after ran-
responsible for supervising therapists within their domization, 57 participants observed the psy-
major theoretical orientation for, on average, 4 to chodynamic treatment segment and 58 partici-
5 years, and most had several publications (M ⫽ pants observed the CB treatment segment. When
14) concerning their approach to psychotherapy examining group main effects for which session
(technique, application, theory, process, or out- segment the participants observed (i.e., PD vs.
come). Before proceeding, it is important to note CB), both the CPPS PI, F(1, 113) ⫽ 5.0, p ⫽ .03,
that there were no significant differences (p ⬎ d ⫽ .42, and CPPS CB, F(1, 113) ⫽ 48.0, p ⬍
.05) regarding these experience variables be- .0001, d ⫽ 1.3, subscale scores demonstrated
tween the two groups. significant differences. The CPPS PI and CPPS
Coefficient alpha reliabilities based on the rat- CB subscales were effective in differentiating
ings of the clinical training staff demonstrated a session activity of both the psychodynamic and

349
Hilsenroth, Blagys, Ackerman, Bonge, and Blais

CB therapists made by ratings of participants Discussion

with no clinical training.
Patient ratings. Following procedures simi- The current study was an initial investigation
lar to Silove, Parker, and Manicavasagar (1990), of the development, reliability, psychometric
we sought to examine how well the CPPS items properties, and validity of a new measure assess-
would perform when used by patients in psy- ing psychotherapy techniques used in treatment
chodynamic psychotherapy to evaluate their ses- sessions: the CPPS. Based on two reviews of the
sions for both PI and CB treatment techniques. empirical comparative psychotherapy process lit-
For this analysis, we used ratings from 136 ses- erature (Blagys & Hilsenroth, 2000, 2002), the
sions made by 42 patients receiving psychody- CPPS measures the distinctive features of PI and
namic psychotherapy in the treatment research CB treatments. The measure was not developed
program described previously (Hilsenroth, 2002). for any highly specific form of therapy or treat-
The demographic characteristics of these 42 pa- ment manual. Rather, the scale is intended to be
tients are described in greater detail elsewhere a more general measure, applicable to a wide
(Hilsenroth et al., 2004). Each participant pro- range of PI and CB therapies as found in applied
vided written informed consent to be included in settings.
this program evaluation research. In addition, pa- These results suggest that the CPPS possesses
tients were informed both verbally and in writing excellent interrater reliability and internal consis-
that their therapist would not have access to their tency, with items and subscales comparing favor-
responses on the psychotherapy process measures ably to (if not an improvement over) reliability
they completed. statistics of other therapist activity measures re-
Patients who received psychodynamic psycho- ported in the literature (Barber & Crits-Christoph,
therapy described their session activity in a man- 1996; Barber et al., 2003; Castonguay et al.,
ner that clearly differentiated PI and CB technical 1996; DeRubeis & Feeley, 1990; Feeley DeRu-
interventions, was statistically significant, and beis, & Gelfand, 1999; Gaston & Ring, 1992;
demonstrated robust effects. A paired t-test anal- Gaston, Thompson, Gallagher, Cournoyer, &
ysis demonstrated significant differences in the Gagnon, 1998; Goldfried et al., 1997; Hill,
amount of PI (M ⫽ 4.01, SD ⫽ .93) and CB (M ⫽ O’Grady, & Elkin, 1992; Hollon et al., 1988;
1.76, SD ⫽ .10) techniques patients described in Ogrodniczuk & Piper, 1999; Samoilov et al.,
their psychotherapy sessions across treatment, 2000; Shaw et al., 1999; Startup & Shapiro,
N ⫽ 136, t(135) ⫽ 26.36, p ⬍ .0001, d ⫽ 2.4. 1993). In addition, the psychometric features of
Therapist and supervisor ratings. We sought the CPPS were consistent with the a priori, em-
to ascertain how similar CPPS ratings of both PI pirically based conceptualization of psychother-
and CB treatment techniques would be for ther- apy process designed to differentiate the session
apist and supervisor perspectives on session pro- characteristics of PI and CB approaches to
cess. For this analysis we used independent rat- treatment.
ings from 131 sessions made by 18 therapists and In the initial examination of the scale validity,
their supervisor providing psychodynamic psy- the CPPS PI subscale was found to be signifi-
chotherapy in the treatment research program de- cantly related to other extant scales of psychody-
scribed previously (Hilsenroth, 2002). The demo- namic treatment activity (in order of relative
graphic characteristics of these therapists as well magnitude) such as general psychodynamic inter-
as their training, treatment, and supervision are viewing style, the competent use of expressive
described in greater detail elsewhere (Hilsenroth techniques, the amount of expressive techniques
et al., in press; Hilsenroth et al., 2004). Indepen- used, and specific strategies of time-limited dy-
dent therapist and supervisor ratings of PI and CB namic psychotherapy. Also, the CPPS PI sub-
technical interventions were significantly related scale demonstrated significant but less robust re-
and demonstrated moderate to large effects. Pear- lationships to the amount and competent use of
son r correlations demonstrated a significant re- supportive techniques. Thus, higher scores for the
lationship between therapist and supervisor rat- CPPS items may be related to the competent use
ings for both the amount of PI (N ⫽ 131, r ⫽ .48, of a technique or intervention being utilized. In
p ⬍ .0001) and CB (N ⫽ 131, r ⫽ .47, p ⬍ .0001) fact, these findings are consistent with the extant
techniques used in psychotherapy sessions across research that demonstrates competence and ad-
treatment. herence, across a range of techniques, are not

350
 CPPS Development

necessarily orthogonal but often moderate to one another based on independent ratings of PI
highly related to one another (Barber et al., 2003; and CB treatment interventions from the CPPS.
Miller & Binder, 2002). Finally, although some Despite this broad set of analyses, the results of
of these correlations were quite high, it is impor- the reliability and validity analyses should be
tant to keep in mind the unique features of the considered preliminary, and some potential limi-
CPPS as an empirically derived, brief, general tations are discussed. First, the study lacked a
measure of both PI and CB techniques more well-standardized CB treatment comparison
likely to be found in applied clinical practice. group and, because of the nature of the primary
In addition, when comparing mean CPPS rat- treatment sample (i.e., short-term psychodynamic
ings in matched samples of PD and non-PD ses- psychotherapy), some of the analyses were better
sions, all but one of the CPPS items (as well as suited to explore the validity of the PI subscale.
the CPPS PI and CPPS CB subscale total scores) Given this limitation, the validity and applicabil-
were able to discriminate PD from non-PD ses- ity of the CPPS for a standardized CB treatment
sions. A number of studies have shown that forms should be examined in future work. Although it is
of PD–IP therapy could be distinguished from important to highlight that nonpsychodynamic
CB treatment using different measures of psycho- sessions were included to provide a preliminary
therapy process, adherence, and competence in comparison and initial assessment of the validity
different samples (for reviews, see Blagys & of the scale, the CB prototype ratings from the
Hilsenroth, 2000, 2002). The results of the cur- clinical training staff were quite robust, and un-
rent study replicate this previous research and dergraduates were able to accurately differentiate
provide initial evidence supporting the validity of the delivery of CB treatment using the CB sub-
the CPPS. scale. Second, for several of the analyses, thera-
Additional validity analyses were also under- pists conducting the treatments were graduate
taken to demonstrate the clinical utility of this students in training. The extent to which the work
measure. The CPPS was able to robustly differ- of these clinicians differs from the practice of
entiate the prototypic session characteristics of more experienced therapists potentially reduces
both PI and CB approaches to treatment made by the external validity of the measure. Last, finan-
clinical training staff from these theoretical ori- cial and time constraints prevented the use of
entations. Consistent with the findings of Ablon more coders or those with postgraduate experi-
and Jones (1998), the current use of the CPPS ence in this initial project. However, it is unlikely
provides a further demonstration that the ratings that the use of graduate student judges limited the
of experienced PI and CB clinicians have two generalizability of the CPPS for use in psycho-
distinct conceptualizations of ideal psychother- therapy research because, in almost all extant
apy process. In addition, the CPPS represents reports of therapist activity or process scales,
adequate item coverage to assess such treatment raters are trained undergraduates, master’s-level,
prototypes. The CPPS PI and CPPS CB subscales or doctorate graduate students.
also performed well when used by undergraduate The results of the current study represent the
raters to evaluate the session characteristics of initial phase of research in scale development of
psychodynamic and CB treatments. Thus, it the CPPS and suggest that further evaluation is
seems that the CPPS items are written in clear, warranted because the clinical utility of this mea-
descriptive, experience-near terms that allow na- sure extends into three overlapping domains.
ı̈ve raters to assess the use of a technique or First, in terms of research, the CPPS identifies
intervention in a session and does not require and operationally defines some of the theoreti-
inferences about internal mental processes or cally central and distinctive activities of non-
highly specialized clinical and theoretical knowl- manualized PI and CB therapy. In this respect,
edge. This finding is further buttressed by the the CPPS represents a potentially useful measure
results of the patient ratings, whereby patients of therapist activity in real-world treatments and
who received psychodynamic psychotherapy de- may be used in effectiveness studies conducted
scribed their session activity in a manner that in naturalistic settings, given that the scale was
significantly differentiated PI and CB technical not developed for a specific therapy or manual
interventions with robust effects. Finally, it ap- but focuses on more general principles of PI
pears that therapist and supervisor perspectives and CB treatment. Conversely, in more con-
on session process were significantly related to trolled psychotherapy trials, the CPPS may be

351
Hilsenroth, Blagys, Ackerman, Bonge, and Blais

used to compare techniques both within and (1979). Cognitive therapy for depression. New York:
between various types of CB and PI therapies Guilford Press.
BLAGYS, M. D., & HILSENROTH, M. J. (2000). Distinctive
to distinguish specific or broad features of features of short-term psychodynamic-interpersonal
these alternative treatments. Second, the CPPS psychotherapy: A review of the comparative psycho-
can also be used as a training or teaching tool therapy process literature. Clinical Psychology: Science
to help new students and trainees learn about and Practice, 7, 167–188.
the general treatment techniques of each ther- BLAGYS, M. D., & HILSENROTH, M. J. (2002). Distinctive
features of short-term cognitive-behavioral psychother-
apeutic approach (see Hilsenroth et al., in apy: A review of the comparative psychotherapy pro-
press). The instrument may provide a founda- cess literature. Clinical Psychology Review, 22, 671–706.
tion from which to build skills advocated by BOOK, H. (1998). How to practice brief psychodynamic
the two theoretical orientations, because teach- psychotherapy: The core conflictual relationship theme
method. Washington, DC: American Psychological
ers can point to the CPPS as a clear illustration Association.
of empirically derived differences between the BOURNE, E. J. (1995). The anxiety and phobia work book
two treatments. In addition, supervisors will be (2nd ed.). Oakland, CA: New Harbinger.
able to focus their educational efforts on these BUTLER, S., HENRY, W., & STRUPP, H. (1992). Measuring
salient, fundamental, orientation-specific activ- adherence and skill in time-limited dynamic psychother-
apy. Unpublished manuscript, Vanderbilt University.
ities and processes. In terms of clinical prac- CASTONGUAY, L. G., GOLDFRIED, M. R., WISER, S.,
tice, the CPPS can provide PI or CB therapists RAUE, P. J., & HAYES, A. M. (1996). Predicting the
with a guide for session activity. Clinicians effect of cognitive therapy for depression: A study of
from each orientation can use the CPPS to unique and common factors. Journal of Consulting and
monitor their own use of the salient aspects of Clinical Psychology, 64, 497–504.
COHEN, J. (1988). Statistical power analysis for the behav-
each orientation. Finally, the CPPS may be ioral sciences (3rd ed.). New York: Academic Press.
useful in the integration of PI and CB ap- CRAIGHEAD, L. W., CRAIGHEAD, W. E., KAZDIN, A. E., &
proaches to treatment. By operationally defin- MAHONEY, M. J. (1994). Cognitive and behavioral in-
ing some of the differences between the two terventions. Boston: Allyn & Bacon.
sets of activities, clinicians can begin to eval- CRITS-CHRISTOPH, P., SIQUELAND, L., CHITTAMS, J., BAR-
BER, J., BECK, A., FRANK, A., ET AL. (1998). Training in
uate the effectiveness of each activity alone or cognitive, supportive-expressive, and drug counseling
in combination and attempt to incorporate these therapies for cocaine dependence. Journal of Consult-
procedures into their treatment repertoire. ing and Clinical Psychology, 66, 484 – 492.
DAVIS, M. ESHELMAN, E. R., & MCKAY, M. (1995). The
relaxation and stress reduction workbook (4th ed.).
References Oakland, CA: New Harbinger.
DERUBEIS, R. J., & FEELEY, M. (1990). Determinants of
ABLON, J. S., & JONES, E. E. (1998). How expert clini- change in cognitive therapy for depression. Cognitive
cian’s prototypes of an ideal treatment correlate with Therapy and Research, 14, 469 – 482.
outcome in psychodynamic and cognitive-behavior DERUBEIS, R. J., HOLLON, S. D., EVANS, M. D., & BEMIS,
therapy. Psychotherapy Research, 8, 71– 83. K. M. (1982). Can psychotherapies for depression be
ACKERMAN, S., HILSENROTH, M., & KNOWLES, E. (2005). discriminated? A systematic investigation of cognitive
Ratings of therapist dynamic activities and alliance therapy and Interpersonal therapy. Journal of Consult-
early and late in psychotherapy. Psychotherapy, 42, ing and Clinical Psychology, 50, 744 –756.
225–231. FEELEY, M. DERUBEIS, R. J., & GELFAND, L. A. (1999).
American Psychiatric Association. (1994). Diagnostic The temporal relation of adherence and alliance to
and statistical manual of mental disorders (4th ed.). symptom change in cognitive therapy for depression.
Washington, DC: Author. Journal of Consulting and Clinical Psychology, 67, 578 –
BARBER, J. P., & CRITS-CHRISTOPH, P. (1996). Develop- 582.
ment of a therapist adherence/competence rating scale FLEISS, J. L. (1981). Statistical methods for rates and pro-
for supportive-expressive psychotherapy: A prelimi- portions (2nd ed.). New York: Wiley.
nary report. Psychotherapy Research, 6, 81–94. GASTON, L., & RING, J. M. (1992). Preliminary results on
BARBER, J. P., CRITS-CHRISTOPH, P., & LUBORSKY, L. the Inventory of Therapeutic Strategies. Journal of Psy-
(1996). Effects of therapist adherence and competence chotherapy Practice and Research, 1, 135–146.
on patient outcome in brief dynamic therapy. Journal GASTON, L., THOMPSON, L., GALLAGHER, D., COURNOYER,
of Consulting and Clinical Psychology, 64, 619 – 622. L., & GAGNON, R. (1998). Alliance, technique, and their
BARBER, J. P., LIESE, B., & ABRAMS, M. (2003). Devel- interactions in predicting outcome of behavioral, cogni-
opment of the Cognitive Therapy Adherence and Com- tive, and brief dynamic therapy. Psychotherapy Research,
petence Scale. Psychotherapy Research, 13, 205–221. 8, 190 –209.
BECK, A. T., & FREEMAN, A. (1990). Cognitive therapy of GOLDFRIED, M. R., CASTONGUAY, L. G., HAYES, A. M.,
personality disorders. New York: Guilford Press. DROZD, J. F., & SHAPIRO, D. A. (1997). A comparative
BECK, A. T., RUSH, A. J., SHAW, B. F., & EMERY, G. analysis of the therapeutic focus in cognitive-behavioral

352
 CPPS Development

and psychodynamic-interpersonal sessions. Journal of interventions in cognitive therapy? World Congress of

Consulting and Clinical Psychology, 65, 740 –748. Behavior and Cognitive Therapies, Acapulco, Mexico.
GOLDFRIED, M. R., & DAVISON, G. C. (1994). Clinical JACOBSON, N., DOBSON, K., TRUAX, P., ADDIS, M.,
behavior therapy. New York: Wiley. KOERNER, K., GOLLAN, J. K., ET AL. (1996). A compo-
GOLDFRIED, M. R., NEWMAN, C. F., & HAYES, A. M. nent analysis of cognitive-behavioral treatment for de-
(1989). The coding system of therapeutic focus. Unpub- pression. Journal of Consulting and Clinical Psychol-
lished manuscript, State University of New York at ogy, 64, 295–304.
Stony Brook. JONES, E. E. (1985). Manual for the Psychotherapy Pro-
GOLDFRIED, M. R., RAUE, P. J., & CASTONGUAY, L. G. cess Q Sort. Unpublished manuscript, University of
(1998). The therapeutic focus in significant sessions of California, Berkeley.
master therapists: A comparison of cognitive- JONES, E. E., & PULOS, S. M. (1993). Comparing the
behavioral and psychodynamic-interpersonal interven- process in psychodynamic and cognitive-behavioral
tions. Journal of Consulting and Clinical Psychology, therapies. Journal of Consulting and Clinical Psychol-
66, 803– 810. ogy, 61, 306 –316.
HAYES, A. M., CASTONGUAY, L. G., & GOLDFRIED, M. R. LIESE, B., BARBER, J., & BECK, A. (1995). The Cognitive
(1996). Effectiveness of targeting vulnerability factors Therapy Adherence and Competence Scale. Unpub-
of depression in cognitive therapy. Journal of Consult- lished manuscript, University of Kansas Medical Center.
ing and Clinical Psychology, 64, 623– 627. LUBORSKY, L. (1984). Principles of psychoanalytic psy-
HAYES, A. M., & STRAUSS, J. L. (1998). Dynamic systems chotherapy: A manual for supportive-expressive treat-
theory as a paradigm for the study of change in psycho- ment. New York: Basic Books.
therapy: An application to cognitive therapy for de- MASTERS, J. C., BURISH, T. G., HOLLON, S. D., & RIMM,
pression. Journal of Consulting and Clinical Psychol- D. C. (1987). Behavior therapy: Techniques and empir-
ogy, 66, 939 –947. ical findings. New York: Harcourt Brace Jovanovich.
HILL, C. E., O’GRADY, K. E., & ELKIN, I. (1992). Apply- MILLER, S., & BINDER, J. (2002). The effects of manual-
ing the Collaborative Study Psychotherapy Rating based training on treatment fidelity and outcome: A
Scale to rate therapist adherence in cognitive- review of the literature on adult individual psychother-
behavioral therapy, interpersonal therapy, and clinical apy. Psychotherapy, 39, 184 –198.
management. Journal of Consulting and Clinical Psy- NUNNALLY, J., & BERNSTEIN, I. (1994). Psychometric the-
ory (3rd ed.). New York: McGraw-Hill.
chology, 60, 73–79.
O’MALLEY, S. S., FOLEY, S. H., ROUNSAVILLE, B. J.,
HILSENROTH, M. (2002). Adelphi University: Psychody-
WATKINS, J. T., SOTSKY, S. M., IMBER, S. D., & ELKIN,
namic psychotherapy process and outcome research
I. (1988). Therapist competence and patient outcome in
team. In P. Fonagy, J. Clarkin, A. Gerber, H. Kächele,
interpersonal psychotherapy of depression. Journal of
R. Krause, E. Jones, et al. (Eds.), An open door review
Consulting and Clinical Psychology, 56, 496 –501.
of outcome studies in psychoanalysis (2nd ed., pp. 241–
OGRODNICZUK, J. S., & PIPER, W. E. (1999). Measuring
247). London: International Psychoanalytical Association.
therapist technique in psychodynamic psychotherapies:
HILSENROTH, M., ACKERMAN, S., & BLAGYS, M. (2001). Development and use of a new scale. Journal of Psy-
Evaluating the phase model of change during short- chotherapy Practice and Research, 8, 142–154.
term psychodynamic psychotherapy. Psychotherapy PRICE, J., HILSENROTH, M., CALLAHAN, K., PETRETIC-
Research, 11, 29 – 47. JACKSON, P., & BONGE, D. (2004). A pilot study of
HILSENROTH, M., ACKERMAN, S., BLAGYS, M., BAITY, M., psychodynamic psychotherapy for adult survivors of
& MOONEY, M. (2003). Short-term psychodynamic psy- childhood sexual abuse. Clinical Psychology and Psy-
chotherapy for depression: An evaluation of statistical, chotherapy, 11, 378 –391.
clinically significant, and technique specific change. SAMOILOV, A., GOLDFRIED, M. R., & SHAPIRO, D. A.
Journal of Nervous and Mental Disease, 191, 349 –357. (2000). Coding system of therapeutic focus on action
HILSENROTH, M., DEFIFE, J., BLAGYS, M., & ACKERMAN, and insight. Journal of Consulting and Clinical Psychol-
S. (IN PRESS). Effects of training in short-term psy- ogy, 68, 513–514.
chodynamic psychotherapy: Changes in graduate clini- SHAPIRO, D. A., & STARTUP, M. J. (1990). Raters’ manual
cian technique. Psychotherapy Research. for the Sheffield Psychotherapy Rating Scale (Memo 1154,
HILSENROTH, M., PETERS, E., & ACKERMAN, S. (2004). MRC/ESRC). Sheffield, UK: University of Sheffield.
The development of therapeutic alliance during psy- SHAW, B. F., ELKIN, I., YAMAGUCHI, J., OLMSTED, M.,
chological assessment: Patient and therapist perspec- VALLIS, T. M., DOBSON, K. S., ET AL. (1999). Therapist
tives across treatment. Journal of Personality Assess- competence ratings in relation to clinical outcome in
ment, 83, 332–344. cognitive therapy of depression. Journal of Consulting
HOLLON, S. D., EVANS, M. D., AUERBACH, A., DERU- and Clinical Psychology, 67, 837– 846.
BEIS, R. J., ELKIN, I., LOWERY, A., ET AL. (1988). De- SHROUT, P. E., & FLEISS, J. L. (1979). Intraclass correla-
velopment of a system for rating therapies for depres- tions: Uses in assessing rater reliability. Psychological
sion: Differentiating cognitive therapy, interpersonal Bulletin, 86, 420 – 428.
therapy, and clinical management pharmacotherapy. SILOVE, D., PARKER, G., & MANICAVASAGAR, V. (1990).
Unpublished manuscript, Department of Psychology, Perceptions of general and specific therapist behaviors.
Vanderbilt University. Journal of Nervous and Mental Diseases, 178, 292–299.
Insight Media. (1998). Psychology: Approaches to ther- STARTUP, M., & SHAPIRO, D. A. (1993). Therapist treatment
apy. New York: Author. fidelity in prescriptive vs. exploratory psychotherapy. Brit-
JACOBSON, N. (1998, JULY). How important are cognitive ish Journal of Clinical Psychology, 32, 443– 456.

353
Hilsenroth, Blagys, Ackerman, Bonge, and Blais

STRUPP, H., & BINDER, J. (1984). Psychotherapy in a new WACHTEL, P. L. (1993). Therapeutic communication. New
key: A guide to time-limited dynamic psychotherapy. York: Guilford Press.
New York: Basic Books. WALTZ, J., ADDIS, M. E., KOERNER, K., & JACOBSON, N.
THOMPSON-BRENNER, H. & WESTEN, D. (IN PRESS). A (1993). Testing the integrity of a psychotherapy proto-
naturalistic study of psychotherapy for bulimia nervosa, col: Assessment of adherence and competence. Journal
Pt. 2: Therapeutic interventions and outcome in the of Consulting and Clinical Psychology, 61, 620 – 630.
community. Journal of Nervous and Mental Disease. WESTEN, D., NOVOTNY, C., & THOMPSON-BRENNER, H.
THOMPSON-BRENNER, H., & WESTEN, D. (2005). Person- (2004). The empirical status of empirically supported
ality subtypes in eating disorders: Validation of a clas- therapies: Assumptions, findings, and reporting in
sification in a naturalistic sample. British Journal of controlled clinical trials. Psychological Bulletin, 130,
Psychiatry, 186, 516 –524. 631– 663.
VALLIS, T., SHAW, B., & DOBSON, K. (1986). The Cogni- YOUNG, J., & BECK, A. T. (1980). Cognitive Therapy
tive Therapy Scale: Psychometric properties. Journal of Scale rating manual. Unpublished manuscript, Univer-
Consulting and Clinical Psychology, 54, 381–385. sity of Pennsylvania, Philadelphia.

354
 CPPS Development

Appendix A
CPPS–Form ER/T
Therapist Session #
Patient ID Rater

Instructions: Using the scale provided below, please rate how characteristic each statement was of the therapy session. For
each item, please write the scale rating number on the blank line provided.

Scale:

0 1 2 3 4 5 6
Not at all characteristic Somewhat characteristic Characteristic Extremely characteristic

(1) The therapist encourages the exploration of feelings regarded by the patient as uncomfortable (e.g., anger,
envy, excitement, sadness, or happiness).
(2) The therapist gives explicit advice or direct suggestions to the patient.
(3) The therapist actively initiates the topics of discussion and therapeutic activities.
(4) The therapist links the patient’s current feelings or perceptions to experiences of the past.
(5) The therapist focuses attention on similarities among the patient’s relationships repeated over time, settings,
or people.
(6) The therapist focuses discussion on the patient’s irrational or illogical belief systems.
(7) The therapist focuses discussion on the relationship between the therapist and patient.
(8) The therapist encourages the patient to experience and express feelings in the session.
(9) The therapist suggests specific activities or tasks (homework) for the patient to attempt outside of session.
(10) The therapist addresses the patient’s avoidance of important topics and shifts in mood.
(11) The therapist explains the rationale behind his or her technique or approach to treatment.
(12) The therapist focuses discussion on the patient’s future life situations.
(13) The therapist suggests alternative ways to understand experiences or events not previously recognized by
the patient.
(14) The therapist identifies recurrent patterns in patient’s actions, feelings, and experiences.
(15) The therapist provides the patient with information and facts about his or her current symptoms, disorder,
or treatment.
(16) The therapist allows the patient to initiate the discussion of significant issues, events, and experiences.
(17) The therapist explicitly suggests that the patient practice behavior(s) learned in therapy between sessions.
(18) The therapist teaches the patient specific techniques for coping with symptoms.
(19) The therapist encourages discussion of patient’s wishes, fantasies, dreams, or early childhood memories
(positive or negative).
(20) The therapist interacts with the patient in a teacher-like (didactic) manner.

(Appendixes continue)

355
Hilsenroth, Blagys, Ackerman, Bonge, and Blais

Appendix B
CPPS–Form P
Therapist Session #
Patient ID Rater

Instructions: Using the scale provided below, please rate how characteristic each statement was of the therapy session. For
each item, please write the scale rating number on the blank line provided.

Scale:
0 1 2 3 4 5 6
Not at all characteristic Somewhat characteristic Characteristic Extremely characteristic

(1) My therapist encouraged me to explore feelings that are hard for me to talk about (e.g., anger, envy,
excitement, sadness, or happiness).
(2) My therapist gave me explicit advice or direct suggestions for solving my problems.
(3) My therapist actively initiated the topics of discussion and activities during the session.
(4) My therapist linked my current feelings or perceptions to experiences in my past.
(5) My therapist brought to my attention similarities between my past and present relationships.
(6) Our discussion centered on irrational or illogical belief systems.
(7) The relationship between the therapist and myself was a focus of discussion.
(8) My therapist encouraged me to experience and express feelings in the session.
(9) My therapist suggested specific activities or tasks (homework) for me to attempt outside of the session.
(10) My therapist addressed my avoidance of important topics and shifts in my mood.
(11) My therapist explained the rationale behind his or her technique or approach to treatment.
(12) The focus of our session was primarily on future life situations.
(13) My therapist suggested alternative ways to understand experiences or events I had not previously
recognized.
(14) My therapist identified recurrent patterns in my actions, feelings, and experiences.
(15) My therapist provided me with information and facts about my current symptoms, disorder, or treatment.
(16) I initiated the discussion of significant issues, events, and experiences.
(17) My therapist explicitly suggested that I practice behavior(s) learned in therapy between sessions.
(18) My therapist taught me specific techniques for coping with my symptoms.
(19) My therapist encouraged discussion of wishes, fantasies, dreams, or early childhood memories (positive or
negative).
(20) My therapist interacted with me in a teacher-like (didactic) manner.

356