We recently used lmer to analyze some reaction time data. There were three fixed effects variables, and the model included their interactions and a fully specified random effects structure. Something like

lm1 <- lmer(RT ~ A*B*C + (1+(A*B*C)|participant) + (1+(A*B*C)|item), data, REML = TRUE)

We found a three-way interaction, and interpreted it in a way that I thought was reasonable. However, a referee commented that, in order to interpret the coefficients associated with the interaction, we first has to compare nested models in order to ascertain the necessity of the different terms starting from the highest order interaction (and report the log likelihood Chi Square for the stats).

My understanding was that assessing nested models via likelihood ratio tests (to determine the best fitting model) and null hypothesis significance testing (e.g., to see if factor C has any effect) were separate issues. For example:

"LR tests can assess the significance of particular factors or, equivalently, choose the better of a pair of nested models, but some researchers have criticized model selection via such pairwise comparisons as an abuse of hypothesis testing..." from Bolker et al. (2008).

I know that something like lmerTest would do this kind of thing automatically, but I want to be sure I understand this. Does the referee comment make sense? Is this a disputed topic? Are there recommendations for further reading that I can do on this issue?


The referee may just be concerned that the 3-way interaction has been "discovered" on a fishing trip. Was this interaction something that was intended to be tested for from the outset ? If so, and this is clearly discussed in your methods section, then the comment seems somewhat out of place. However, it is standard practice among many researchers though it is not universal by any means. I would go ahead and do what they have asked for.

I would be rather more concerned that you are working with a maximal model which may have converged to a degenerate solution. Do you really need random coefficients for all 3 main effects and their interactions on both grouping factors ? What kind of variables are A, B and C? The advice to "keep it maximal" is very often very bad advice and has been shown to lead to incorrect inferences. See Bates et al 2015 for an in depth discussion (note that Douglas Bates was the original author of the lme4 package you are using.

Bates, D., Kliegl, R., Vasishth, S. and Baayen, H., 2015. Parsimonious mixed models. arXiv preprint arXiv:1506.04967. http://arxiv.org/pdf/1506.04967v1.pdf

  • $\begingroup$ We did predict an interaction - we're following up a reasonably well established finding in our field - and went into detail about it in our Method. But I'm certainly not opposed to adding more information, provided that it's warranted. It appears that this is acceptable practice, so I'll take your advice and add it. One followup question: hypothetically, if the model comparisons do not indicate that factor C significantly improves model fit, how does this affect the interpretability of the original model? Is it a question of confidence in the original result? $\endgroup$ – clint Sep 29 '16 at 12:55
  • $\begingroup$ Also: my initial analysis did not include a fully-specified random effects structure - but the same referee cited Barr et al. (2013) and more or less demanded it. I've been working with another paper from that group (Matuschek et al., under revision), which presents simulation evidence against "keeping it maximal". I'm going to end up arguing about all this, I'm sure - which is okay. <br/> Matuschek, H., Kliegl, R., Vasishth, S., Baayen, H., & Bates, D. (2015). Balancing type I error and power in linear mixed models. arXiv preprint arXiv:1511.01864. $\endgroup$ – clint Sep 29 '16 at 13:04
  • $\begingroup$ Oh, about the variables! Variable A is the time of a memory cue, a proper name (presented either before or after a stimulus). Variable B is the type of cue (relevant or not relevant to the stimulus). Variable C is another cue manipulation (the name is either a well-known, famous person, or an unknown person). My original analysis had only included B and C in the random effects structure. And by the way, thanks for your reply, which was very helpful to me - and apologies for the serial responses. $\endgroup$ – clint Sep 29 '16 at 13:12

Your Answer

