Abstract - Economics Research Reports
Substitution and Dropout Bias in Social Experiments: A Study of an Influential Social Experiment
By James Heckman, Neil Hohmann, Jeffrey Smith (University of Western
with the assistance of Michael Khoo
This paper considers the interpretation of evidence from social experiments when persons randomized out of a program being evaluated have good substitutes for it, and when persons randomized into a program drop out to pursue better alternatives. Using data from an experimental evaluation of a classroom training program, we document the empirical importance of control group substitution and treatment group dropping out. Evidence that one program is ineffective relative to close substitutes is not evidence that the type of service provided by all of the programs is ineffective, although that is the way the experimental evidence is often interpreted.
JEL Classification: H43; C93; C14