The Corner

The one and only.

Has Medical Science Discovered Anything Useful?



David Freedman has a very interesting article in The Atlantic that has generated a lot of justified interest in the blogosphere. Freedman describes how hard it is to develop reliable medical knowledge, saying of medical scientific studies that “you have to wonder whether they prove anything at all. Indeed, given the breadth of the potential problems raised at the meeting, can any medical-research studies be trusted?”

The article talks a whole lot about the role that researcher bias, skewed incentives to gain funding, and so forth play in making even the conclusions of large-scale randomized control trials (RCTs) suspect. The hero of the article, celebrated medical meta-researcher Dr. John Ioannidis, is quoted as saying:


This kind of thing happens, as I have often written. But as always, the question is, “Compared to what?” That is, does our imperfect scientific knowledge of medicine allow us to make better decisions than we would make in the absence of this information? After all, medical research is not the only scientific field in which funding pressure creates researcher biases, yet we still seem to be able to build functioning airplanes and mobile phones.

The author answers this question with too broad a brush. What’s so striking to me about the facts asserted in the article is that even if we accept the author’s claims, then certain identifiable categories of medical knowledge seem quite reliable, while others seem worse than useless.

First, consider the differences by research methodology. The article claims that 80 percent (!) of non-randomized studies turn out to be wrong, as compared to “as much as” 10 percent of large randomized trials. Sufficiently-large randomized experiments, as I have often argued, are not some nerdy nice-to-have when evaluating theories in a sufficiently complex environment; they are a requirement. Being wrong 80 percent of the time is literally worse than just flipping a coin. On the other hand, if I had a series of ailments, and had to make a series of decisions about treatments for them, I would happily rely on a method that was right at least 90 percent of time in preference to relying on some combination of my intuition, what my brother-in-law experienced, and what I discovered on Google. Well-executed RCTs really do create useful scientific knowledge.

Second, consider the examples of subsequently refuted findings that are identified in the article. Here is a representative paragraph:


Do you see a common theme here? While not exclusively, these tend very strongly to be long-term, behaviorally-oriented interventions. Consider the classical therapies that were evaluated during the heroic phase of clinical trials in the mid-20th century — things like the streptomycin trials in Britain or polio vaccines in the U.S. These situations can be characterized by acute conditions that cause death or obvious loss of function within a short period, and that are addressed by treatments that apply a chemical to the body. As we shade from this kind of a problem to those characterized by conditions that affect people over many years, often in subjective ways, and are addressed by lifestyle changes or daily dosages of vitamins and so on, we are shading from medicine as classically conceived to something that is analytically much more like social science. This latter end of the spectrum is where the most common and severe problems with reliable determinations of causal effectiveness of interventions arise. This is not necessarily because researchers in these areas are less honest than those in other fields, but because the problem is inherently harder. Among other issues, signal-to-noise is worse, the relevant measurement period becomes years and decades rather than weeks and months, and the causal mechanism often becomes subtly entangled with many lifestyle behaviors. In such situations, RCTs are often impractical, and when they can be done, the integrated complexity of the causal mechanisms means that replications are much more likely to fail because unobserved context differences turn out to be relevant in determining success or failure of the treatment. The problem isn’t always the researchers; sometimes the problem is the problem.