Economics Join our invited guests to debate economics
A response to:
Randomised trials yield important, but limited, benefits
Paul Seabright our guest wrote on Apr 25th 2011, 13:17 GMT

RANDOMISED trials, in social science no less than in medicine, are highly suitable for evaluating policy interventions that meet three broad conditions:

  1. They produce their effects over a relatively short time period, so that results can be observed in time to use the lessons of the study;
  2. Resource constraints limit the number of treatments that can be made, so a choice must be made in any case between potential subjects of the intervention;
  3. There are no strong prior reasons to favor the particular treatment over plausible alternatives.

There are obviously trade-offs between these criteria: an intervention for which we have strong reasons to suspect has greater efficacy than a placebo can nevertheless be justified if the results of the test will be known quickly; a slower treatment may justify randomisation if its relative efficacy is purely speculative, and so on.

Even in medicine many kinds of important policy question are not like this. We owe our knowledge of the effects of smoking on lung cancer, for example, to statistical studies of populations and not to randomised trials (which would have taken too long, and have involved unnecessarily encouraging some individuals to smoke when there was already good reason to suspect its risks). Such statistical studies of populations have to compensate with statistical techniques for the fact that there are no properly randomised control groups, but for many kinds of question they are unavoidable.

Social science probably has a greater proportion of such questions than medicine: nobody is going to suggest that some Arab autocracies avoid political change so that we can conduct a randomised study of the effects of democracy (and I doubt that President Assad's resistance to change owes much to purely scientific motives). Zhou En-Lai is reported to have said that it was "too soon to tell" what had been the impact of the French Revolution, but that has not stopped historians asking the question. Questions about social processes don't become less interesting just because they are not amenable to randomised trials.

Another important caveat concerns placebo effects. These can be very large in some areas of medicine, such as anti-depressant medication (see Irving Kirsch's excellent book "The Emperor's New Drugs: Exploding the Anti-Depressant Myth"), but they are probably even more widespread in social science, where the consciousness of participating in a flagship program may induce individuals to behave in ways that would no longer be typical once the program is rolled out to the population as a whole.

None of this should be interpreted as carping at the very great achievement of Banerjee and Duflo in bringing randomised trials to the forefront of development economics research. Such trials will have an increasing impact outside development economics too (none of the three criteria above is limited in its application to poor countries). But as to how many of the really important questions will yield to this revolutionary approach, I'm with Zhou En-Lai: it's too soon to tell.

Readers' comments


Common Sense and Reasonable Discourse

The comparison to medicine is weak due to one key point. The economic studies are not putting people into harms way as noted with the smoking reference. Adding stimulus in various ways and observing the results are in fact putting people in a position to succeed and then observing if they do so. If researchers provide food at no cost and the test subjects spend all of there money on alcohol and decrease there earnings the researchers are not blame for the individuals self sabotage. There isn't a moral question here as implied, unless researchers drive up the prices of necessities to observe the results.

Randomized trials will unquestionably result in better use of aid resources. There only limitation is that they will typically yield local, but not global solutions. This of course isn't a reason not to use the approach since spending thousands to determine how best to spend millions is well worth the cost.


The comparison with the practice of medicine is quite apt. Medicine was and arguably still dominated by highly opinionated men who are forced to make important decisions based on limited information. Medicine is highly complex with many hidden parameters so that the outcome of any intervention is uncertain. Of course, many quacks, half-learns, and trained physicians will confidently espouse half-baked, poorly supported or unsupported treatments. Substitute economics for medicine and you get the picture. Of course, we should do randomized trials. But remember the quacks will fight it tooth and nail--it is their livelihood afterall.


Yes, but this does not address the issues raised by Pritchett and Thoma below: how can you set up a suitable "control" for national economic policies? So how can we assess what small business tax system is most effective in broadening the tax base? What's the most effective way to improve transparency and accountability in government bureaucracies? More broadly, how can we figure out which reforms work and under what circumstances?

About our guest

Professor of Economics
Contributions: 17




Products and events



