The  Secret  of  Effec:ve  Feedback  

Dylan  Wiliam  (@dylanwiliam)  

www.dylanwiliamcenter.com  www.dylanwiliam.net  
 Outline  
2  

•  What  is  feedback?  

•  Why  most  feedback  research  isn’t  relevant  
•  What  we  do  know  about  feedback  
•  Prac:cal  sugges:ons  for  improving  feedback  
•  Some  thoughts  about  grading  
 What  is  feedback?  
3  

“ac:ons  taken  by  (an)  external  agent  (s)  to  provide  

informa:on  regarding  some  aspect(s)  of  one’s  task  
performance.”  (Kluger  &  DeNisi,  1996  p.  255)  

•  Examples  
–  Your  typing  speed  is  45  words  per  minute  
–  You  are  5  words  per  minute  below  proficiency  
–  Use  your  thumb  only  to  press  the  space  bar  
 What  is  the  purpose  of  feedback?  
4  

•  Two  main  goals  of  feedback  

–  To  improve  the  work  
–  To  improve  the  student  
•  Most  of  the  :me,  the  purpose  of  feedback  is  to  
improve  the  work  of  students  on  tasks  they  have  not  
yet  a,empted.  
 Educa7onal  Endowment  Founda7on  toolkit  
5  

Interven7on   Cost   Quality  of   Extra  months  

evidence   of  learning  
Feedback   $$   ★★★ +8  
Metacogni:on  and  self-­‐regula:on   $$   ★★★★ +8  
Peer  tutoring   $$   ★★★★ +6  
Early  years  interven:on   $$$$$   ★★★★ +6  
One  to  one  tui:on   $$$$   ★★★★ +5  
Homework  (secondary)   $   ★★★ +5  
Collabora:ve  learning   $   ★★★★ +5  
Phonics   $   ★★★★ +4  
Small  group  tui:on   $$$   ★★★★ +4  
Behaviour  interven:ons   $$$   ★★ +4  
Digital  technology   $$$$   ★★★★ +4  
Social  and  emo:onal  learning   $   ★★★★ +4  
 Educa7onal  Endowment  Founda7on  toolkit  
6  

Interven7on   Cost   Quality  of   Extra  months  

evidence   of  learning  
Parental  involvement   $$$   ★★★ +3  
Reducing  class  size   $$$$$   ★★★ +3  
Summer  schools   $$$   ★★ +3  
Sports  par:cipa:on   $$$   ★★ +2  
Arts  par:cipa:on   $$   ★★★ +2  
Extended  school  :me   $$$   ★★ +2  
Individualized  instruc:on   $   ★★★ +2  
Aeer  school  programmes   $$$$   ★★ +2  
Learning  styles   $   ★★★ +2  
Mentoring   $$$   ★★★ +1  
Homework  (primary)   $   ★★★ +1  
 Educa7onal  Endowment  Founda7on  toolkit  
7  

Interven7on   Cost   Quality  of   Extra  months  

evidence   of  learning  
Teaching  assistants   $$$$   ★★ 0  
Performance  pay   $$   ★ 0  
Aspira:on  interven:ons   $$$   ★ 0  
Block  scheduling   $   ★★ 0  
School  uniform   $   ★ 0  
Physical  environment   $$   ★ 0  
Ability  grouping   $   ★★★ -­‐1  
 Kinds  of  feedback:  Israel  
8  

•  264  low  and  high  ability  grade  6  students  in  12  classes  in  4  
schools;  analysis  of  132  students  at  top  and  bojom  of  each  
class  
•  Same  teaching,  same  aims,  same  teachers,  same  classwork  
•  Three  kinds  of  feedback:  grades,  comments,  grades+comments  

Achievement   AEtude  

Grades    no  gain   High  scorers:  posi:ve  

Low  scorers:  nega:ve  
Comments   30%  gain   High  scorers:  posi:ve  
Low  scorers:  posi:ve  

Butler  (1988)    
 Responses  
9  

Achievement   AEtude  

Grades    no  gain   High  scorers:  posi:ve  

Low  scorers:  nega:ve  
Comments   30%  gain   High  scorers:  posi:ve  
Low  scorers:  posi:ve  

What  happened  for  students  given  both  grades  and  comments?  

A.  Gain:  30%;  Antude:  all  posi:ve  
B.  Gain:  30%;  Antude:  high  scorers  posi:ve,  low  scorers  nega:ve  
C.  Gain:  0%;  Antude:  all  posi:ve  
D.  Gain:  0%;  Antude:  high  scorers  posi:ve,  low  scorers  nega:ve  
E.  Something  else  
 Effects  of  feedback  
10  

•  Kluger  &  DeNisi  (1996)  review  of  3000  research  reports  

•  Excluding  those:  
–  without  adequate  controls  
–  with  poor  design  
–  with  fewer  than  10  par:cipants  
–  where  performance  was  not  measured  
–  without  details  of  effect  sizes  
•  lee  131  reports,  607  effect  sizes,  involving  12652  individuals  

•  On  average,  feedback  increases  achievement  

–  Effect  sizes  highly  variable  
–  38%  (231  out  of  607)  of  effect  sizes  were  nega:ve  
 Feedback  in  STEM  subjects  
11  

•  Review  of  9000  papers  on  feedback  in  

mathema:cs,  science  and  technology  
•  Only  238  papers  retained  
–  Background  papers  24  
–  Descrip:ve  papers  79  
–  Qualita:ve  papers  24  
–  Quan:ta:ve  papers  111  
•  Mathema:cs  60  
•  Science  35  
•  Technology  16  

Ruiz-­‐Primo  and  Li  (2013)  

 Classifica7on  of  feedback  studies  
12  

1.  Who  provided  the  feedback  (teacher,  peer,  self,  or  technology-­‐based)?  
2.  How  was  the  feedback  delivered  (individual,  small  group,  or  whole  
class)?  
3.  What  was  the  role  of  the  student  in  the  feedback  (provider  or  
receiver)?  
4.  What  was  the  focus  of  the  feedback  (e.g.,  product,  process,  self-­‐
regula:on  for  cogni:ve  feedback;  or  goal  orienta:on,  self-­‐efficacy  for  
affec:ve  feedback)  
5.  On  what  was  the  feedback  based  (student  product  or  process)?  
6.  What  type  of  feedback  was  provided  (evalua:ve,  descrip:ve,  or  
holis:c)?  
7.  How  was  feedback  provided  or  presented  (wrijen,  video,  oral,  or  
video)?  
8.  What  was  the  referent  of  feedback  (self,  others,  or  mastery  criteria)?  
9.  How,  and  how  oeen  was  feedback  given  in  the  study  (one  :me  or  
mul:ple  :mes;  with  or  without  pedagogical  use)?  
 Main  findings  
13  

Characteris7c  of  studies  included   Maths   Science  

Feedback  treatment  is  a  single  event  las:ng  minutes   85%   72%  
Reliability  of  outcome  measures   39%   63%  
Validity  of  outcome  measures   24%   3%  
Dealing  only  or  mainly  with  declara:ve  knowledge   12%   36%  
Schema:c  knowledge  (e.g.,  knowing  why)   9%   0%  
Mul:ple  feedback  events  in  a  week   14%   17%  
 Kinds  of  feedback  (Nyquist,  2003)  

•  Weaker  feedback  only  

–  Knowledge  or  results  (KoR)  
•  Feedback  only  
–  KoR  +  clear  goals  or  knowledge  of  correct  results  (KCR)  
•  Weak  forma:ve  assessment  
–  KCR+  explana:on  (KCR+e)  
•  Moderate  forma:ve  assessment  
–  (KCR+e)  +  specific  ac:ons  for  gap  reduc:on  
•  Strong  forma:ve  assessment  
–  (KCR+e)  +  ac:vity  
 Effects  of  forma7ve  assessment  (HE)  

Kind  of  feedback Count Effect

Weaker  feedback  only 31 0.14
Feedback  only 48 0.36
Weaker  forma:ve  assessment 49 0.26
Moderate  forma:ve  assessment 41 0.39
Strong  forma:ve  assessment 16 0.56
 GeEng  feedback  right  is  hard  
16  

Response  type Feedback  indicates  performance…

falls  short  of  goal exceeds  goal

Change  behavior Increase  effort Exert  less  effort

Change  goal Reduce  aspira:on Increase  aspira7on

Abandon  goal Decide  goal  is  too  hard Decide  goal  is  too  easy

Reject  feedback Feedback  is  ignored Feedback  is  ignored

 Engagement  and  learning  

•  Ajribu:on  (Dweck,  2000)  

–  Personaliza:on  (internal  vs.  external)  
–  Permanence  (stable  vs.  unstable)  
–  Good  learners  ajribute  failure  and  success  to  internal,  
unstable  causes.  (It’s  down  to  you,  and  you  can  do  
something  about  it.)  
•  Views  of  ‘ability’  
–  Fixed  (IQ)  
–  Incremental  (untapped  poten:al)  
–  Essen:al  that  teachers  inculcate  in  students  a  view  that  
‘ability’  is  incremental  rather  than  fixed  (by  working,  
you’re  genng  smarter).  
 Dual-­‐pathway  theory  (Boekaerts,  2006)  

•  Long-­‐term  learning  goals  are  translated  into  short-­‐

term  learning  inten:ons  
•  Dynamic  comparisons  of  task  and  situa:onal  demands  
with  personal  resources,  taking  into  account:  
–  Interest  
–  Capability  
–  Importance  
–  Value  
–  Cost  
•  Resul:ng  ac:va:on  of  energy  along  one  pathway:  
–  Well-­‐being  
–  Growth  
 Summary  of  prac7cal  techniques  
19  

•  Focus  on  the  reac:ons  of  the  students,  not  the  

feedback  
–  Build  trust  
–  “Wise  feedback”  
•  Focus  your  students  on  growth,  not  preserving  
well-­‐being  
•  Design  feedback  as  part  of  a  system  
•  Make  feedback  into  detec:ve  work  
 Meanings  and  consequences  of  school  grades  
20  

•  Two  ra:onales  for  grading  

–  Meanings  
•  Assessment  as  eviden:ary  reasoning  
•  Assessment  outcomes  as  supports  for  making  inferences  
–  (e.g.,  about  student  achievement)  
–  Consequences  
•  Assessment  outcomes  as  rewards  and  punishments  
•  Assessments  create  incen:ves  for  students  to  do  what  we  want  
them  to  do  
–  These  two  ra:onales  interact,  and  conflict  
•  achievement  grades  for  comple:on  of  homework  
•  achievement  grades  for  effort  
•  penal:es  for  late  submission  
•  zeroes  for  missing  work  
 Effects  of  punishment  
21  

•  10  private  day-­‐care  centers  in  Haifa,  Israel  

•  Each  center  opens  at  7.30am  and  closes  at  4.00pm  
•  Each  center  normally  takes  up  to  35  children  
•  A  number  of  parents  are  late  collec:ng  children  
•  6  of  the  10  centers  started  fining  late  parents  
–  Fine  for  parents  >10  minutes  late:  NIS10  per  child  
–  For  comparison,  baby  sijers  earn  NIS20  per  hour  

Gneezy  and  Rus:chini  (2000)  

 “A  fine  is  a  price”  
22  

Control   New  policy  

25  
Number  of  late-­‐coming  parents  

20  

15  

10  

5  

0  
0   2   4   6   8   10   12   14   16   18   20  
Week  
Policy  introduced  in  week  5   Policy  abandoned  in  week  17  
 To  find  out  more…  
23  

www.dylanwiliamcenter.com  
 To  find  out  even  more…  
24  

www.dylanwiliamcenter.com