I have a sample of patients who underwent 2 diagnostic tests, one of which is the gold standard. I've been asked to test if the diagnosis from the experimental test is different than the one from the gold standard. Here is the contingency table:

> table(test=db$exptest, gold=db$goldstandard)
test       sick   healthy
  sick        7         4
  healthy     8        27

At first I thought this was a case for McNemar's test, but it seems this test is meant to compare 2 experimental test results against the gold standard, as seen here, or here. Nevertheless, the wiki page or this post are not that exclusive, but not very clear for my case though.

Should I perform McNemar's test in this specific case? If not, can I test this hypothesis and how?

  • $\begingroup$ Could you not just report the sensitivity and specificity for the test vs the gold standard? Or use the kappa statistic to look at the agreement between tests beyond that that would be expected chance? $\endgroup$ – kateeleanor Jun 20 '18 at 14:30
  • $\begingroup$ Sensitivity and specificity are defined in reference to some "ground truth" value, which doesn't seem accessible here. We can't use the gold standard as the ground truth as that would introduce a circularity. Based on the OP's request to determine if the new test is different from the gold standard (with no reference to diagnostic quality), an inter-rater agreeement test like Kappa certainly seems more appropriate. $\endgroup$ – klumbard Jun 20 '18 at 14:56
  • 2
    $\begingroup$ Your question has already been answered but I'd like to point out that your data set is criminally under-powered for the question you're asking. Only the off-diagonal cells contribute to McNemar's so you have an effective sample size of 12. Taking $(b-c)^2/(b+c) = \text{CDF}^{-1}(1-0.05|df=1) \approx 4$, fixing $b+c=12$ and solving for $b-c$, we see that we would need at least a difference of at least 7 to report an effect at the 0.05 significance level: e.g., 2/10, 10/2, or something more extreme. $\endgroup$ – olooney Jun 21 '18 at 13:03

If you use McNemar's test you are testing whether the table is symmetric: whether more people are diagnosed sick by the new method and well by the old versus well by the new and sick by the old. This is a perfectly reasonable scientific question to have.For a concrete situation suppose the two methods being compared are ratings of mental health problems by a psychiatrist and by a family physician. Since they see different case mix in their practice you might ask whether this affects their threshold for declaring someone ill.

If you use Cohen's kappa you are evaluating whether agreement between the methods is more than would be expected by chance. This again a perfectly reasonable question to have but it is different. So if you are comparing two methods for diagnosing mild cognitive impairment where there is no gold standard you might treat agreement between methods as justifying the concept of MCI and if they disagree you might wonder whether it is a useful diagnosis at all.

Calculating sensitivity and specificity is the usual method for diagnostic tests and evaluates the performance separately in the two groups: well according to the gold standard and sick according to the gold standard. Again this is a reasonable thing to do but it is different from the other two. In this case you have two separate things which you are interested in and your focus in a practical situation might be on one or the other. For instance if you are screening for a fatal disease you might want a test with high sensitivity since you do not want to miss cases. On the other hand if you are recruiting into a trial you might not mind missing a few but on cost grounds you might want high specificity since you do not want o do the full diagnostic work-up on more people than is absolutely essential.

  • $\begingroup$ Thanks, very neat answer ! Though, the nuance between these 2 questions is a bit unclear for me. Could you give an example where they don't give the same conclusion ? $\endgroup$ – Dan Chaltiel Jun 21 '18 at 9:33
  • 1
    $\begingroup$ @DanChaltiel, you should be able to see the distinction b/t the 2 tests in my answer. Eg, Cohen's kappa is significant, but McNemar's test isn't. Thus, there is greater agreement than you would expect by chance alone, but there is insufficient evidence to determine that the new test is biased. $\endgroup$ – gung Jun 21 '18 at 10:52
  • $\begingroup$ I would argue that since they answer different questions their answers are not really comparable despite @gung's example. I have added more detail about concrete situations. $\endgroup$ – mdewey Jun 21 '18 at 12:12

You are asking about agreement, so you should use a test for agreement. With just two diagnostic measures ('raters') that are categorical in nature, the standard test is Cohen's kappa. Here's a version applied to your data, coded in R:

tab2 = as.data.frame(tab)
kappa2(tab2[rep(1:4, times=tab2[,3]),1:2])
#  Cohen's Kappa for 2 Raters (Weights: unweighted)
#  Subjects = 46 
#    Raters = 2 
#     Kappa = 0.363 
#         z = 2.52 
#   p-value = 0.0118 

The test is significant, implying that there is greater agreement than you would expect by chance alone.

You don't have to stop there. You could measure the percent agreeing, for example:
$$ \text{percent agreeing} = \frac{7+27}{7+4+8+27} = 73.9\% $$ Sensitivity and specificity (or the positive and negative predictive values) constitute a similar kind of information, but decomposed and at a greater level of detail, which may be more useful but is also more complex.

You could also test to see if the new test is biased relative to the gold standard. Specifically, your test calls only called 11 people sick, whereas the gold standard noted that 15 were. Is the new test saying 'sick' less often than it should? That's what McNemar's test would do for you here.

#   McNemar's Chi-squared test with continuity correction
# data:  tab
# McNemar's chi-squared = 0.75, df = 1, p-value = 0.3865

There is insufficient evidence in your dataset to determine that the test is biased relative to the gold standard.


Your Answer

