

Is Something Wrong with the Scientific Method?

pickens (UID: 49171) writes | more than 3 years ago

Science 6

Hugh Pickens writes "Jonah Lehrer has an interesting article in the New Yorker reporting that all sorts of well-established, multiply confirmed findings in science have started to look increasingly uncertain as they cannot be replicated. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology and in the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants. “One of my mentors told me that my real mistake was trying to replicate my work," says researcher Jonathon Schooler. "He told me doing that was just setting myself up for disappointment.” For many scientists, the effect is especially troubling because of what it exposes about the scientific process. "If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved?" writes Lehrer. "Which results should we believe?" Francis Bacon, the early-modern philosopher and pioneer of the scientific method, once declared that experiments were essential, because they allowed us to “put nature to the question" but it now appears that nature often gives us different answers. According to John Ioannidis, author of “Why Most Published Research Findings Are False," the main problem is that too many researchers engage in what he calls “significance chasing,” or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher. "The scientists are so eager to pass this magical test that they start playing around with the numbers, trying to find anything that seems worthy,”"


Just statistical incompetence (1)

This comment was hidden based on your threshold setting.

AliasMarlowe (UID: 1042386) | more than 3 years ago

I come across this all the time, but there's no need to panic. Basic physics and engineering are not threatened, just the justifications for the fad-du-jour in medicine or sociology or other semi-empirical studies. Alcohol is good for you; no, it's bad for you; no, it's very good for you in moderation; no, any amount is bad; and so forth. All of the conclusions on alcohol vs health are at 95% confidence, too, given the appalling misapplication of statistics to the relevant data (typically from inadequate studies, anyway).

Medical scientists are far worse than engineers or physicists for abuse of statistical methods. None of these groups plumbs the same depths as MBA candidates, of course. I speak as one who has taught grad students in Physics, Engineering, Business, and Medicine (actually Medical Physics). There is a distinct gradient in the extent to which they will manipulate or interpret data so a statistics black box will produce the desired result. The business students produce the worst reactions when shown the error of their ways ("You say the data are inconclusive, but I need to draw a conclusion from it. The conclusion does not have to be right, but it must be persuasive, so I have to get the right statistical result...").

Re:Just statistical incompetence (1)

This comment was hidden based on your threshold setting.

russotto (UID: 537200) | more than 3 years ago

In medical science especially, the fashion for meta-analyses can't help. Aggregating data from different studies results in all sorts of artifacts appearing.

There's nothing wrong with the scientific method (1)

This comment was hidden based on your threshold setting.

surveyork (UID: 1505897) | more than 3 years ago

There's something wrong with some scientists. TFS cites problems in psychology, ecology and medicine. That's no surprise. As it has been pointed in the above comment , there's a quite broad misuse of statistics, so in the end, the saying "lies, damn lies and statistics" comes true, since statistics are interpreted the way the scientist wants to better fit his hypothesis. In cosmology and physics there's also something going on: evidence for a pre-Big-Bang universe, evidence of multiverses, dark matter signatures, proton shrinking 4%, string theories... The scientific method is valid, you just have to use it properly.

Re:There's nothing wrong with the scientific metho (1)

This comment was hidden based on your threshold setting.

Spazmania (UID: 174582) | more than 3 years ago

Many if not most published climate-change papers describe some interesting results and then go on to draw broad conclusions that don't follow from the data, that are at best hypothesis the data suggests experimenting with. What percentage of scientists in a field have to get it wrong before the problem is with the methodologies and behavior accepted during peer review rather than the specific individuals?

Pet theories and the need to publish... (1)

This comment was hidden based on your threshold setting.

Knightman (UID: 142928) | more than 3 years ago

In other words the "scientists" are sloppy and they produce results that fit their pet theory... Nothing new really, been going on for a long long time.

It's troubling though that it's so widespread but I guess most of them need to produce results so they can justify their need for money/tenure etc. and that usually means taking shortcuts and fudge the data.

Scientific method is fine (1)

This comment was hidden based on your threshold setting.

airfoobar (UID: 1853132) | more than 3 years ago

The problem is that many scientists, just like many politicians, can be bought and sold for the right price. The problem is that we live in a world that cares more about money than about science or culture or the public or the environment...
Check for New Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode


Create a Slashdot Account