Tuesday, July 12, 2011

A Partial Cure for the “Is Consistent With” Syndrome









Larry Hamelin said...

Feynman beat you to it:

Cargo Cult Science

Eric Nilsson said...

I believe you're calling for a type of "non-nested hypothesis test," or something along those lines.

They are occasionally done, as I've done here:

Such procedures are plausibly helpful, if one takes empirical work seriously, but few empirical workers really care about the big picture questions associated with empirical work.

Peter Dorman said...

Larry: It's OK to come in a distant second (or more) after Feynman, but in this case what Feynman is doing is expressing the view that the minimization of Type I error is at the core of science. I did not presume to be original when I wrote this same thing a while back -- quite the contrary. I was evoking, as Feynman was, the accumulated experience of how science works when it really works. If there is anything new about my post yesterday, it is that I am narrowing the discussion to just one topic that troubles economics and proposing an operationalized mitigation. Little stuff philosophically, but it would be big if economists would do it.

Eric: I think the spirit of non-nested hypothesis testing, as you've used it in your paper, looks in the direction I'm recommending. At the level of implementation, though, it's rather different. It asks whether X1 has a better fit with the data at hand than X2. This adds something to my proposal, but it leaves two things out. First, it doesn't formally consider all the potential explanations for the outcome you're interested in, and second, it doesn't search for a prediction of X2 that would be disallowed by X1 or vice versa. But if you repeat your exercise on lots of different data sets, you will have something useful, to be sure.