Search interesting materials

Tuesday, August 10, 2010

Randomised field experiments











  1. I share the concern that there are poorly done studies that cost a lot of money and yield little by way of new insights. However, I would not quite go so far as to label RCTs themselves as something that would not be worthwhile to pursue. In all economic phenomena there is an enormous amount of noise and signal extraction is a real challenge -- this is as true of fiancial markets as it is of economic development. I think RCTs are a very interesting tool for doing this carefully.

    Now what one does with the signal or how the experiment is set up and the theory behind it are all questions that deserve the same amount of thinking as in the case of any other research project. If one asks a poorly thoughtout question one is likely to get some pretty useless results -- RCTs are not unique in that respect.

    Professor Rohini Pande has carried out an RCT for example for helping us look "beneath the hood" on the whole "black magic" of the zero defaults within JLGs. The insight on how social networks are actually formed / strengthened during weekly meetings and are not necessarily pre-existing is a fascinating one and of immense practical value. I can't imagine how else one would have gotten to this conclusion except through an intervention design.

  2. Nachiket,

    I'm not discussing the poorly done studies. I'm discussing the best studies. The 3 problems:

    * Low external validity

    * Low bang for the buck

    * Lack of replication

    afflict the best studies.

    The Criminology story is quite pertinent. How do you know that a few other RCTs for the defaults-in-JLG question will not yield a different answer?

    As I say in the post, Reality is a complicated nonlinear function in many dimensions. Each properly executed RCT gives the correct answer for the local gradient at a certain point on that function. But the answer lacks external validity: it is hard to generalise too much given the narrowness of the RCT.

    And, these things are hideously expensive. For a counter-example, I will say that Indian economics was vastly improved by the resources expended on building NFHS. This became a standard dataset used by hundreds of researchers, with the discipline and competition which comes from multiple people looking at the same dataset. There are fewer programming mistakes in papers which use standard datasets.

    I am all for a more experimental approach to econometrics - but my sense of the way forward is to push into quasi-experimental studies based on large panel datasets. This has more external validity, gives better bang for the very scarce buck, and generates the correct incentives within the research community on the key issues of replication and competition. You have to be more intelligent when you work with NFHS because 1000 other researchers have the same dataset. The RCT space is being afflicted by too little thinking and too much fund-raising coupled with administrative capability on field execution.

    The great empirical successes of economics lie in big publicly visible datasets - finance and labour economics come to mind.

  3. Great thoughts!

    A more general thought from me. I am wondering if you have ever covered the state of economics education in India. When I was young, economics as a subject was barely covered in school and 'the best and the brightest' almost never went for studying economics. In fact, I see IITians and such people manning and at the helm of various sectors that have nothing to do with engineering but a lot to do with economics.

    Maybe things have changed now or I hope so.


