Discussion
EVENTS, RECRUITMENT
In CRTs, the participant- and cluster-average treatment effects can differ when there is informative cluster size, and standard estimators such as mixed-effects models and GEEs with an exchangeable correlation structure can be biased. However, until now there has been, to our knowledge, no empirical assessment of informative cluster size in the context of a specific CRT. In this re-analysis of a previously published CRT, we aimed to explore whether informative cluster size may be present. For several outcomes, we identified notable differences between estimates for the participant- and cluster-average effects, as well as between IEEs and estimates affected by informative cluster size such as mixed-effects models or GEEs with an exchangeable correlation structure. For example, the treatment effect for the EQ-5D VAS score using a cluster-average estimator was 32% smaller than using a participant-average estimator (2.84 versus 4.15) and was 22% smaller using a mixed-effects model (3.23 versus 4.15). We hypothesise these differences may be due to informative cluster size.These results highlight the need to formally consider the target estimand at the trial design stage (participant- versus cluster-average, and other aspects such as handling of intercurrent events, marginal versus cluster-specific, and so on) and choose an estimator that is aligned to that estimand. If informative cluster size occurs, then standard estimators such as mixed-effects models and GEEs with an exchangeable correlation structure may be biased, and methods such as IEEs or cluster-level summaries are required, as these estimators are unaffected by informative cluster size. Determining whether informative cluster size is likely may depend on various factors, such as type of population (e.g. how variable the type of participants and clusters are) and the type of intervention, variations in cluster size. Therefore, a case-by-case evaluation is required, which should be based on both a qualitative assessment using subject matter knowledge and a quantitative evidence from previous similar studies. While it is likely that some studies are not affected by informative cluster size, it should not be dismissed without careful thought when variable cluster sizes are anticipated at the study outset.It should be noted that using IEEs or the analysis of cluster-level summaries may have implications for sample size, with potentially larger sample sizes required as compared with alternative approaches such as mixed-effects models or GEEs. Alternatively, if mixed-effects models or GEEs are used, IEEs and cluster-level summaries could be used as sensitivity analyses to explore whether inferences may be affected by informative cluster size. In practice, participant- and cluster-average effects will also need to be defined as either marginal or cluster-specific, and an estimator which allows for both aspects (e.g. marginal participant-average and cluster-specific cluster-average) will need to be chosen. Further guidance on choosing estimators which allow for both aspects of the estimand is available elsewhere.It may not in general be easy to identify whether informative cluster size is likely at the design stage. For instance, in the TRIGGER trial, there was no reason to suspect informative cluster size would occur when designing the trial. Further complicating the task, it appears that informative cluster size can occur for some outcomes but not others, requiring investigators to make this judgement not for the trial as a whole, but for each outcome separately. One setting where an impact of informative cluster size can be confidently ruled out is when there is little to no variation in the cluster size. In this case, mixed-effects models and GEEs with an exchangeable correlation structure will not be biased; however, there will be little gain in efficiency from these models compared with IEEs in this setting.Some investigators may feel that potential bias from informative cluster size is of less concern than statistical efficiency, given common problems around the number of available clusters and challenges in patient recruitment. Thus, they may argue that a small amount of bias from mixed-effects models or GEEs with an exchangeable correlation structure is worth it if it leads to a substantial reduction in the required sample size, or a corresponding increase in statistical power. While we acknowledge the logic behind this viewpoint, we argue that its general application may be challenging for two reasons. First, both efficiency and potential bias from informative cluster size will be driven by the size of the intraclass correlation coefficient, indicating that the situations where mixed-effects models/GEEs with an exchangeable correlation structure can provide substantial gains in efficiency may be the same situations where they are be prone to extreme bias. Second, because the bias from informative cluster size can be in either direction, the gains in efficiency from mixed-effects models/GEEs with an exchangeable correlation structure may be offset by a possible reduction in the size of estimated treatment effects (i.e. downwards bias), and so these methods may not lead to power gains compared with IEEs. Future simulation studies to compare metrics like power and precision between mixed-effects models/GEEs with an exchangeable correlation and IEEs both under informative cluster size and no informative cluster size are warranted to more thoroughly evaluate the bias/variance trade-off between these approaches.A limitation of this study is that there is currently no formal test to identify informative cluster size. Thus, we were not able to differentiate to what extent differences between estimators were due to informative cluster size compared with random variation. If differences were simply due to random variation, we would expect these to occur in either direction. However, large differences of >10% occurred in only one direction (all were negative, denoting cluster-average effects were smaller than participant-average effects), and this consistency of effect lends credence to the theory they may be due to informative cluster size. A small simulation study (Our results suggest several areas for future research. First, re-analyses of other CRTs would be useful to determine whether the results found here are unique or not. Second, establishing empirical evidence of in what context informative cluster size is likely would be helpful to aid in planning of future CRTs. Third, though sample size calculations are available for IEEs,