Monday, December 31, 2012

Another claim of gaming benefits

Yesterday I wrote an extended piece about the lack of strong evidence supporting benefits of videogame training. In Today's Guardian/Observer, Vaughan Bell makes the same claim that the Scientific American Mind article did (see also his MindHacks blog post). Unlike that article, he did mention that some researchers have raised questions about the methods in that literature, but then claimed that there have since been other better-designed studies. The article provides no citations to better-designed studies, and I know of exactly zero that have adequately addressed the Boot et al critiques.

Other than mentioning that some people suggest there are problems, he also gave no coverage to the types of problems undermining claims of gaming benefits. We need more critical coverage of these claims of benefits, not more hyping of the claims. He gave much more critical discussion of the videogame/violence claims. How about giving the same level of critical thought to the gaming/benefits claims. It seems to be a trend to use the cognitive benefits claim as an unquestioned way to offset the games are bad for you claim.

Sunday, December 30, 2012



Other Sources Cited:

Wednesday, December 26, 2012

Journals of null results and the goal of replication

Here is my response to the following question that +Gary Marcus forwarded me from one of his readers:
Is there a place in experimental science for a journal dedicated to publishing "failed" experiments? Or would publication in a failed-studies journal be so ignominious for the scientists involved as not to be worthwhile Does a "failed-studies" journal have any chance of success (no pun intended)?

Over the years, there have been a number of attempts to form "null results" journals. Currently, the field has The Journal In Support of the Null Hypothesis (there may well be others). As a rule, such journals are not terribly successful. They tend to become an outlet for studies that can't get published anywhere else. And, given that there are many reasons for failed replications, people generally don't devote much attention to them.

Journals like PLoS One have been doing a better job than many others in publishing direct replication attempts. They emphasize experimental accuracy over theoretical contributions, which fits the goal of a journal that publishes replication attempts whether or not they work. There also are websites now that compile replication attempts ( The main goal of that site is to make researchers aware of existing replication attempts.

For me, there's a bigger problem with null results journals and websites: They treat replications as an attempt to make a binary decision about the existence of an effect. The replication either succeeds or fails, and there's no intermediary state of the world. Yet, in my view, the goal of replication should be to provide a more accurate estimate of the true effect, not to decide whether a replication is a failure or success.

Few replication attempts should lead to a binary succeed/fail judgment. Some will show the original finding to be a true false positive with no actual effect, but most will just find at the original study overestimated the size of the effect (I say "most" because publication bias ensures that many reported effects overestimate the true effects). The goal of replication should be to achieve greater and greater confidence in the estimate of the actual effet. Only with repeated replication can we zero in on the actual estimate. The greater the size of the new study (e.g., more subjects tested), the better the estimate.

The initiatives I'm pushing behind the scenes (more on those soon) is to encourage multiple replications using identical protocols in order to achieve a better estimate of the true effect. One failure to replicate is no more informative than one positive effect -- both results could be false. With repeated replication, though, we get a sense of what the real effect actually is.

Monday, December 24, 2012

Recommendations for improving psychology

In a new blog post in The New Yorker, ﻿ makes some excellent suggestions for improving the state of psychology publishing. I personally prefer not to cluster fraud cases together with results that do not replicate. The former are revealed only via whistle blowing or investigation. The later inspire the "science is self correcting" refrain—later work can "fix" claims that don't hold up. 

The broader problem, though, is that science often is not self correcting. Science can't correct itself if nobody tries to replicate published claims or if those replications are relegated to the file drawer. The bias against publishing negative findings provides a disincentive for attempting direct replications in the first place. And, even when replication attempts are published, they often are ignored both in the literature and in the public eye (just as newspaper errata are buried). 

The field needs to change the incentive structure that governs scientific publishing in order to encourage and support direct replications. Initiatives like 's reproducibility project get us part of the way there, but only when the academic societies and journals encourage direct replication can the field really change. I'm involved in a project to implement just such a change, and I'll be writing more about it soon.

Tuesday, December 18, 2012

The demographics of surveys: Phone vs. Mechanical Turk

Last year, Chris Chabris and I published the results of a national survey of beliefs about memory (in PLoS One). We found that many people agreed with statements about memory and the mind that experts roundly dimiss as wrong. We conducted the survey in 2009 using the polling firm, SurveyUSA, with a nominal sample of 1500 respondents representing the population of the United States. SurveyUSA uses random-digit dialing of landline phone numbers, and respondents press keys on their keypad to questions from a recorded voice. There method is fairly typical of so-called "robotic" polling.

Last summer, we repeated the same survey using a sample drawn from Amazon Mechanical Turk, with the restriction that respondents be from the United States. On Mechanical Turk, workers decide whether they would like to participate and are paid a small amount for a completed survey. Unlike random digit dialing, the sample on Mechanical Turk is entirely self-selected.  The results of that survey and a comparison to our earlier survey just appeared in PLoS One on December 18, 2012.

Just recently I wrote an extended blog post about the nature of survey demographics. To compare these two surveys, we weighted both to a nominal sample of 750 respondents according to the population demographics from the 2010 US Census. Reassuringly, the pattern of results was roughly comparable. In essence, we replicated our generalization to the national population, with many people endorsing mistaken beliefs about memory.

In writing the paper and re-weighting the samples, I discovered something interesting about who responds to these sorts of surveys. Although both could be weighted to a nationally representative sample, the raw demographics of the samples were vastly different. They were roughly comparable on most dimensions (e.g., income, education, region of the country), but their ages differed dramatically.

figure comparing age demographics from multiple survey methods

The yellow bars represent data from the 2010 US Census. Note that about half of the adult US population is under 44 and half is over 44. Now look at the blue bars from the SurveyUSA sample of about 1840 people. The first thing to notice is that the phone survey massively oversamples older people and massively undersamples younger ones. In order to generalize to the larger population, each response from a young subject is weighted to count many times that of an old subject. The pattern is almost exactly the opposite for our Mechanical Turk sample (of just under 1000 people). Mechanical Turk respondents were disproportionately young. The extent of age bias in each sample was roughly comparable and in opposite directions, and neither was anywhere close to the actual demographics of the US population.

For me, this figure was eye opening. I wasn't surprised that an online Mechanical Turk sample would be disproportionately younger, and I assumed that phone surveys would oversample the elderly, but I had no idea how extreme that bias would be. What that means is that any national survey conducted by phone is mostly contacting older people. Unless the sample is adequately large, the number of young respondents will be minuscule, meaning that the weighting for those respondents will be huge. If a small survey happened to get a few oddball younger respondents, it could dramatically alter the total estimate. 

As I discussed in this earlier post, pollsters almost never report their raw demographics, essentially hiding how much weighting their survey needed to make it nationally representative. But, that information is crucial, especially if the survey compares demographic groups. If you compared young and old subjects in our SurveyUSA study, you would be comparing a small sample to a huge one, and the generalization from the older subjects would be much safer than from the younger ones. Without knowing that, you might assume that each generalization was equally valid.

In our paper, we compared the two surveys by weighting each to match the population, making each into a representative sample with a nominal size of 750 people (e.g., weighting to match what would be expected for a sample of 750 people. See the paper for details: We basically dichotomized the age category given the sample sizes). Fortunately, despite these huge deviations from the actual population statistics, each "nationally representative" sample produced comparable results. In a sense, that is the ideal situation: Two samples with vastly different demographics produce comparable results when weighted. That means that the different sampling methods and weightings did not dramatically change the pattern of results. 

That finding also has practical implications for anyone interested in conducting surveys. One approach to obtaining a more representative sample would be to combine phone and Mechanical Turk samples, counting on Mechanical Turk for younger respondents and the phone survey for older ones.

The next time you read about a survey, ask yourself: Was the sample truly representative and if not, was the sample large enough to trust the conclusions about different demographic subgroups?

The hidden secrets of polling








