Let’s Take the Con Out of Randomized Control Trials in Development: The Puzzles and Paradoxes of External Validity, Empirically Illustrated

Lant Pritchett

No 399, CID Working Papers from Center for International Development at Harvard University

Abstract: The enthusiasm for the potential of RCTs in development rests in part on the assumption that the use of the rigorous evidence that emerges from an RCT (or from a small set of studies identified as rigorous in a “systematic” review) leads to the adoption of more effective policies, programs or projects. However, the supposed benefits of using rigorous evidence for “evidence based” policy making depend critically on the extent to which there is external validity. If estimates of causal impact or treatment effects that have internal validity (are unbiased) in one context (where the relevant “context” could be country, region, implementing organization, complementary policies, initial conditions, etc.) cannot be applied to another context then applying evidence that is rigorous in one context may actually reduce predictive accuracy in other contexts relative to simple evidence from that context—even if that evidence is biased (Pritchett and Sandefur 2015). Using empirical estimates from a large number of developing countries of the difference in student learning in public and private schools (just as one potential policy application) I show that commonly made assumptions about external validity are, in the face of the actual observed heterogeneity across contexts, both logically incoherent and empirically unhelpful. Logically incoherent, in that it is impossible to reconcile general claims about external validity of rigorous estimates of causal impact and the heterogeneity of the raw facts about differentials. Empirically unhelpful in that using a single (or small set) of rigorous estimates to apply to all other actually leads to a larger root mean square error of prediction of the “true” causal impact across contexts than just using the estimates from non-experimental data from each country. In the data about private and public schools, under plausible assumptions, an exclusive reliance on the rigorous evidence has RMSE three times worse than using the biased OLS result from each context. In making policy decisions one needs to rely on an understanding of the relevant phenomena that encompasses all of the available evidence.

Keywords: randomized; controlled; trials (search for similar items in EconPapers)
Date: 2021-05
New Economics Papers: this item is included in nep-dev and nep-ppm
References: Add references at CitEc
Citations: View citations in EconPapers (4)

Downloads: (external link)
https://bsc.cid.harvard.edu/files/bsc/files/2021-0 ... xternal-validity.pdf (application/pdf)
Our link check indicates that this URL is bad, the error code is: 404 Not Found (https://bsc.cid.harvard.edu/files/bsc/files/2021-05-cid-wp-399-external-validity.pdf [301 Moved Permanently]--> https://bsc.hks.harvard.edu/files/bsc/files/2021-05-cid-wp-399-external-validity.pdf [302 Found]--> https://bpb-us-e1.wpmucdn.com/websites.harvard.edu/dist/c/104/files/2021-05-cid-wp-399-external-validity.pdf [301 Moved Permanently]--> https://bpb-us-e1.wpmucdn.com/404.html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cid:wpfacu:399

Access Statistics for this paper

More papers in CID Working Papers from Center for International Development at Harvard University 79 John F. Kennedy Street. Contact information at EDIRC.
Bibliographic data for series maintained by Chuck McKenney ().