Basics of Perf Based Comp

THE BASICS OF
DIFFERENTIATED
COMPENSATION
QuEST CONFERENCE
WASHINGTON, D.C.
JULY 2009
Rob Weil
Educational Issues
Jewell Gould
Research and Information Services
Matthew Di Carlo
Research and Information Services
One of the
biggest hurdles in
developing a new
compensation
system is the
amount of change
required to
properly support
to new program.
As much change
is required of the
school district as
the teachers, if
not more.
1
Copyright © American Federation of Teachers, AFL-CIO | July 09
OBJECTIVES
• Describe differentiated pay in general and
how it should be approached
• Explain value-added methodology, how it
works, and how it should be used
• Discuss the role of base salary in systems,
and vice-versa
• Summarize the existing research on how
differentiated pay influences productivity
• Provide some examples of differentiated
pay systems around the U.S.
• List some of the funding sources available
• Conclude with guidance as to how plans
should be designed and maintained
AN IMPORTANT
PERSPECTIVE
Although Differentiated Compensation is a
promising part of improving teacher
quality, it is only one piece of the
puzzle. Differentiated Compensation
systems done in isolation have little
chance of success. To improve teaching
quality, other aspects of teaching and its
development must be addressed.
2
The “Quality Teaching” Puzzle
Rigorous
Preparation with Peer Assistance
Clear and and Review,
Enforced Induction &
Licensure Mentoring
Standards
Ongoing, Job-
embedded
Professional
Development
Competitive
Evaluation Based and
on Professional Professional
Standards Compensation
A WORD OF CAUTION
How you communicate, both
internally and externally, will play a
large role in the success of your
program. Take time to carefully
frame your message by using
appropriate language, controlling the
flow of information to avoid
misunderstandings, and using facts
and sound research.
By far, this is the number one
mistake in the development of
differentiated pay plans
6
3
THE EASY PART
•Pay for Skills and Knowledge (Input)

•Pay for Responsibility (Extra work)
•Pay for Student Learning* (Output)
*Pay for student learning will require a

large restructuring of most school
districts’ instructional support systems.
7
7
STRATEGIC & SYSTEMATIC

Developing a Strategic Process
Where you Where you
are today want to be
A What you B …will C

learn improve your
here… outcomes
Creates the Basis for a Systematic Approach
Differentiated
Differentiated pay
pay is
is aa process,
process, not
not an
an event.
event.
8
4
THE IMPORTANCE OF
“FACE VALIDITY”
Any plan must make certain that teachers influence the
metrics in an understandable and observable way.
Teachers in “variable” pay plans must see how their
performance directly effects the outcomes.
X Y
“What I did” “This is the result”
The
The importance
importance of
of this
this “validity”
“validity” goes
goes
beyond
beyond individual need; it also is the key
individual need; it also is the key to
to
organizational
organizational growth.
growth.
Value-added
Value-added exemplifies
exemplifies thisthis issue.
issue.
VALUE-ADDED METHODOLOGY
CURRENT TRENDS
• Widespread support for using “growth
models” to measure educational inputs
• Increased federal funding for “data-driven
reform”
• Using VAM as a measure of effectiveness
• The reckless use of concepts like “value-
added” or “gain scores”
• Combining VAM estimates with other
measures of performance to evaluate schools
and teachers
• “Although it’s not reliable or valid, it’s better
than the current system”
• Devaluing of subjects and teachers that are
not tested
10
5
WHAT IS VALUE-ADDED?
FEATURES & ASSUMPTIONS
• Unlike NCLB, focus is on growth, not
achievement levels
• Student performance as a function of past
performance and/or student/school
characteristics
• Sometimes acknowledges that student and
school characteristics contribute to student
growth, as do non-educational factors (e.g.,
poverty)
• Requires extensive data collection and complex
statistical capabilities
• Can be done at the teacher-, grade, school, or
district-level
11
WHAT IS VAM? (CONTINUED)

CAUSAL ISOLATION
• At its core, VAM uses complex statistical methods to produce
a quantitative measure of a teacher’s CAUSAL effect on
student performance/growth
– Deals with confounding factors by controlling them in models
– Teacher effect estimates rely on uncertain assumptions
• Most models don’t actually produce teacher effects per se,
but rather expected student performance
– All “non-teacher effects” are purged by comparing students to
others in similar schools with similar characteristics (e.g.,
poverty, language), and/or by comparing students’ current
growth with their past growth
– This produces an “expected” score for each student
– Any difference between that expectation and reality is attributed
to the teacher, and is considered the “teacher effect”
• The two most common types of models are “covariate
adjustment models” (Bob Mendro, Rob Meyer) and
“multivariate” or “layered” models (William Sanders, EVAAS)
– The choice of models will influence results; both offer strengths
and weaknesses
12
6
STATE OF THE ART
• There is no consistent body of evidence
showing the magnitude or persistence of
teacher effects as a factor in student
achievement
– We know teachers matter, and their effects
persist, but the size and rate of decline of these
effects is still uncertain – recent estimate suggests
roughly 50% “decay”*
– Estimates of magnitude vary by choice of model,
variables included, sample size, etc.
• There is an extremely high degree of error
in measuring the individual teacher effects
on student growth
– Teacher effect estimates are very imprecise
(though certainly “non-zero”)
– Some of this error is random, while some may be
due to unmeasured factors biasing estimates
* Kane, Thomas J. and Douglas O. Staiger. 2008. “Estimating Teacher Impacts on
Student Achievement: An Experimental Evaluation.” NBER Working Paper 13
14607.
THE CONS OF VAM?

ERROR MARGINS
ESTIMATED TEACHER EFFECTS – 2ND GRADE TEACHERS
Adapted from
Dan McCaffrey,
RAND 2004
14
7
STATE OF THE ART II
• VAM scores often conflict with results of other
types of evaluations
– This includes principal/peer evaluations, parental surveys
– This is in no small part due to the inability of test scores
to account for the whole of teaching ability
– For example, one study found that the adjusted
probability of VAM scores matching principal evaluations
of teachers’ ability to improve test scores was about 43%
for reading and 63% for math*
• Teacher effects are also unstable over time
– This is an under-researched area, but extant studies
clearly indicate that VAM estimates for the same teacher
vary greatly between years
– Two analyses of San Diego and Florida districts found
consistent results: only 25-33% of the “best” and “worst”
teachers remain there between years, while 20-30%
move all the way from either top to bottom or bottom to
top **
* Jacob, Brian A. and Lars Lefgren. 2005. Principals as Agents: Subjective Performance Measurement in
Education. NBER Working Paper 11463. Washington, D.C.: National Bureau of Economic Research.
** Koedel, Cory and Julian R. Betts. 2007. “Re-examining the Role of Teacher Quality in the Educational
Production Function.” Working Paper #2007-03. Nashville, TN: National Center on Performance
Incentives AND
McCaffrey, Daniel F., Tim R. Sass, and J.R. Lockwood. 2008. “The Intertemporal Stability of Teacher 15
Effect Estimates.” Unpublished manuscript”
STABILITY OF M.S. TEACHER EFFECTS

IN HILLSBOROUGH COUNTY
YEAR TWO QUINTILE
1 2 3 4 5
Only 23% of teachers
remained stable
1 4.2% 5.2% 5.2% 2.3% 2.9%
between years
YEAR ONE QUINTILE
2 3.3% 4.2% 5.2% 4.9% 2.0% Stable 23.1%

Move 1 36.5%
3 2.3% 3.6% 5.2% 5.9% 3.3% Move 2 23.1%
Move 3-4 16.9%
4 1.3% 2.6% 4.2% 6.5% 4.6%
5 2.3% 2.0% 2.9% 6.9% 6.9% 40% of teachers

moved at least 2
McCaffrey, Daniel F.; Sass, Tim R. and J.R. Lockwood. 2008 The
quintiles between
Intertemporal Stability of Teacher Effect Estimates.
Nashville: National Center for Performance Incentives. years
16
8
WHY ALL THE ERROR?
• SAMPLING ERROR: Small classes Æ small samples
Æ huge margins of error
– Without sufficient “observations,” accuracy suffers greatly, we cannot
draw conclusions about a single teacher from a small group of students
– SCHOOL- or GRADE-LEVEL alleviates this problem (larger sample)
• OMITTED VARIABLE BIAS: there are too many
factors that influence student performance to isolate
teachers’ roles
– Unmeasurable factors include: principal support/competence, support
from colleagues, facilities, textbooks, parental involvement, family
problems, peer effects, curriculum
– Unmeasured/unmeasurable factors bias estimates of measured variables
• NON-RANDOM ASSIGNMENT: Students and
teachers are not usually assigned to classes (or even
schools) randomly
– Often done for good education reasons, but huge problem for VAM
– May “penalize” teachers often assigned to difficult students
– Forthcoming paper finds current teacher effects predict past effects *
• MISSING DATA: especially in large cities, students
move and drop out at high rates
– Even among stayers, attendance is also an issue
– Correcting for this may introduce additional bias
* Rothstein, Jesse. Forthcoming. “Teacher Quality in Educational Production: Tracking,
Decay, and Student Achievement.” Quarterly Journal of Economics. 17
GUIDELINES FOR USE
• EXTREME CAUTION SHOULD BE USED IN CONSIDERING
INDIVIDUAL VAM SCORES AS THE PRIMARY COMPONENT
OF A DIFFERENTIATED PAY SYSTEM
– Error and other problems are prohibitive at the individual-level
– Measures still have great diagnostic potential
• VAM estimates are SENSITIVE TO MODEL SELECTION and
DATA AVAILABILITY/QUALITY
– Choice of models and variables to include will influence results
– Tests must be comparable across grades/subjects
– More years of data decrease error, 3 years helps mitigate sorting
bias*
• SCHOOL-LEVEL VAM should be seriously considered as a part
of any VAM-based differentiated pay system
– Severely reduces problems from sample size, non-random
assignment of students to classrooms – far more accurate than
individual-level (teacher/classroom-level) estimates
– Encourages cooperation, not competition, and all can participate
• DO NOT MISTAKE MARGIN OF ERROR FOR ACCURACY
– Scores reports should account for margins of error, but teachers
must understand that those margins are themselves error-prone
– Error estimates assume that the models are correctly specified
– For example, unmeasured or unmeasurable factors may be biasing
estimates and error margins
* Coedel, Cory and Julian R. Betts. Forthcoming. “Does Student Sorting
Invalidate Value-Added Models of Teacher Effectiveness? An Extended 18
Analysis of the Rothstein Critique. Education Finance and Policy.
9
ABOUT BASE SALARY
Base salary, along with benefits, are the
most important parts of any compensation
system--differentiated or otherwise.
The base pay structure must continue to
recognize the value of teaching experience.
Similarly, pursuing extra education should
be rewarded.*
*Many criticize experience as a factor in
determining a teacher’s salary and then
express the concern that too many
inexperienced teachers are found in the
lowest performing schools.
19
QUID PRO QUO

From the beginning, it is important
to stress the point:
The District (or State) must not
ask teachers to look at their
financial interests differently if
the District (or State) is not
willing to look at its financial
structure differently.
In
In most
most cases,
cases, this
this will
will need
need to
to be
be
stressed numerous times.
stressed numerous times. 20
10
A DEAL BREAKER
aáÑÑ=`çãé
líÜÉê qÉ~ÅÜÉêë líÜÉê

qÉ~ÅÜÉêë
This
This type
type of
of system
system lowers
lowers aa
teacher’s
teacher’s standard of living.
standard of living.
21
A DEAL MAKER
aáÑÑ=`çãé
líÜÉê qÉ~ÅÜÉêë
qÉ~ÅÜÉêë
Teachers
líÜÉê
This
This type
type of
of system
system does
does not
not
lower
lower a teacher’s standard of
a teacher’s standard of
living.
living.
22
11
ABOUT BASE SALARY (CONTINUED)
STRUCTURAL EFFECTS
• All plans that provide flat-dollar performance
bonuses that not everyone receives carry concrete
implications for salary structures - they will
fundamentally alter how and how much teachers are
paid
• Differentiated pay partially severs the long-standing
relationship between degree/experience and earnings
– In many cases, BA teachers who receive bonuses will earn
more than non-recipient MA teachers at the same step
– Similarly, experienced teachers who do not receive bonuses
may have lower earnings than less-experienced teachers
who do
– These effects are universal, and will only vary by extent
• Teachers’ salaries have been determined by experience
and education for a long time; significant changes to this
relationship are important
• In general, it is better to give smaller bonuses to more
people than to give larger bonuses to less people
23
DOES THIS STUFF WORK?

RESEARCH ON EFFECTS OF
DIFFERENTIATED PAY
“Too many supporters of my party have resisted the idea
of rewarding excellence in teaching with extra pay,
even though we know it can make a difference in the
classroom.” - President Barack Obama
• The research on the effects of differentiated pay plans for

teachers is scarce, and there are no definitive findings.
– Results are mixed, and all suffer from limitations of interpretation
– A small group of international studies show generally positive
effects in Mexico, Israel, Kenya and India, but these studies cannot
easily be generalized to the U.S.
• The jury is still out on the effects of these programs
– Several “experimental” studies in progress – NYC, Chicago, and
two in Nashville, TN
– Definitive results will likely require additional studies and
considerable time
• Success depends upon whether differentiated pay systems
help improve (rather than just measure effects of) teaching
24
12
EXAMPLES OF PROGRAMS
SCHOOLWIDE VAM
NEW YORK CITY
• Two-year pilot program collaboratively developed by
UFT and NYCDOE
• Random selection of schools from a non-random
selection of low-performing schools (schools choose
participation)
• Criteria
– Performance (test scores 25%)
– Progress (test scores based on targets 60%)
– Attendance and Learning Environment (teacher, parent,
student surveys 15%)
– Bonus available
• Schools that meet 100% of their performance target
(an expected growth based on a composite of the
school ranking) receive $3000 per UFT member at
the school
– 75% of the performance target receive $1500
• Schools have some discretion in how rewards are
distributed (including to non-instructional staff)
25
HYBRID SCHOOL/INDIVIDUAL
HOUSTON ASPIRE
• Program comprised of three core “strands”
• STRAND I - VAM Campus-wide Improvement
– Bonuses to all instructional and non-instructional
staff based on school-level (i.e., campus-level)
achievement growth
• STRAND II - VAM Core Teacher Performance
– Pay based on individual-level VAM scores
– Inclusiveness: non-core teachers rewarded based
on campus-wide progress
• STRAND III – Campus Improvement and
Achievement
– Rewards core instructional staff based on how well
the school has improved compared with 40 similar
schools in the state
26
13
CAREER LADDER SYSTEM
TOLEDO TRACS
• TRAC A – Professional Development
– Required participation
– Research- and standards-based
• TRAC B – School Performance
– Annual goal determination for each school
– Rewards to schools teaching teams that meet or exceed
goals
• TRAC C – Teacher Performance
– Annual rewards to teachers who demonstrate multi-
criteria excellence, accept difficult-to-fill positions, take
on additional responsibilities, or show high levels of
student achievement
– TRACS C is composed of three status levels; teachers
move up the ladder by demonstrating excellence
– Successful placement on the TRACS program is
rewarded with extra salary (as a percent of base)
– Participation is voluntary, but participants must undergo
regular evaluations to remain in the system
27
COMPREHENSIVE
DENVER PRO-COMP
• All staff are eligible to participate
• Base salary of participants varies by success in meeting
goals
• Program comprised of four parts Æ
• Professional evaluation
– Non-probationary teachers get a bonus every three years for a
successful evaluation
– Probationary teachers get a smaller bonus
• Student growth incentives
– Payments for meeting one or two targets for student growth,
for working in schools receiving “exemplary” ratings, and for
working in schools classified as “distinguished”
– Base salary increases with each target met
• Market incentives
– Bonuses for moving to “hard to staff” or “hard to serve”
assignments/schools
• Knowledge and skills
– Professional development, National Board certification, and
“lifetime tuition accounts” all provide separate bonuses
28
14
FUNDING SOURCES FOR
DIFFERENTIATED PAY
SYSTEMS
• TIF (Teacher Incentive Fund) - $487 million
in 2010 (effort to increase to $717 million)
• Teacher Incentive Grants Competition
• State Fiscal Stabilization Fund (ARRA)
• Private foundations (e.g., Gates)
• State funding (e.g., Q-Comp)
All
All external
external funding
funding has
has two
two important
important
features:
features: it may run out at some point,
it may run out at some point, and
and
itit sometimes comes with “hooks”
sometimes comes with “hooks”
29
CONCLUSION
GROUP PAY
PROS CONS
• Widely accepted • Tremendous amount of
• More valid and reliable work at front end
metrics • Time-consuming
• Positive public assessment and
relations reports
• Academically aligned • Some teachers not
“group workers”
Inter-rater
Inter-rater reliability
reliability isis easier
easier • Groups “adjust”
to
to manage
manage andand control.
control. plans
30
15
CONCLUSION
INDIVIDUAL PAY
PROS CONS
• Politically powerful • Difficult to develop and
• Has high loyalty by measure
participants • Creates expanding
• Relatively low cost expectations
• Creates a group of • Concerns regarding
teacher leaders objectivity
• Creates “false
Inter-rater
Inter-rater reliability
reliability isis hard
hard positives” and “true
to
to manage and control.
manage and control. negatives”
31
CONCLUSION
COMBINATION SYSTEMS
PROS CONS
• Provide the most • Tremendous amount of
flexibility work to maintain
• Allow teachers to • Different parts of the
develop their “own plan can be in
program” competition
• Have high participant • Time-consuming
satisfaction assessment and
reports
32
16
SWING TO THE MIDDLE
The Teacher Compensation Pendulum
Focusing on Focusing on
Teacher Behavior Student Growth
Alone Alone
Input Output
Ideal
A Balance
33
33
CONCLUSION (CONT.) AFT’S BASIC

REQUIREMENTS
FOR A SUCCESSFUL PROGRAM
• An Adequate Base Salary

• Sufficient and Stable Funding
• Credible, Agreed-upon Standards of Practice
• Support to Improve Professional Practice
• Labor/Management Collaboration (“Trust”)
• Incentives Available to All Teachers
• Easily Understood Standards for Rewards
• Necessary Support for Program
Not
Not in
in any
any particular
particular order
order
34
17
QUESTIONS OR
COMMENTS?
ROB WEIL
(202) 393-6953
rweil@aft.org
JEWELL GOULD
(202) 879-4428
jgould@aft.org
MATT DI CARLO
(202) 585-4377
mdicarlo@aft.org
35
18

Basics of Perf Based Comp

Uploaded by

Copyright:

Available Formats

Basics of Perf Based Comp

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Basics of Perf Based Comp

Uploaded by

Copyright:

Available Formats

THE BASICS OF

•Pay for Skills and Knowledge (Input)

*Pay for student learning will require a

STRATEGIC & SYSTEMATIC

A What you B …will C

WHAT IS VAM? (CONTINUED)

THE CONS OF VAM?

STABILITY OF M.S. TEACHER EFFECTS

2 3.3% 4.2% 5.2% 4.9% 2.0% Stable 23.1%

4 1.3% 2.6% 4.2% 6.5% 4.6%

5 2.3% 2.0% 2.9% 6.9% 6.9% 40% of teachers

QUID PRO QUO

líÜÉê qÉ~ÅÜÉêë líÜÉê

DOES THIS STUFF WORK?

• The research on the effects of differentiated pay plans for

The Teacher Compensation Pendulum

CONCLUSION (CONT.) AFT’S BASIC

• An Adequate Base Salary

You might also like

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.