Evaluating the Role of Possible Confounding Factors
Differences in time and motivation can act as confounding factors in the performance of an intervention. To test whether the observed differences between the experimental and the waitlist groups were affected by these factors, we performed additional analyses.We had planned to take the PHQ-9 and GAD-7 scores in the waitlist group at 42 days, as the experimental group participants were also expected to take 42 days to submit the final questionnaire (6 modules at the rate of 1 module per week). However, variability in the actual timing of score submission was inevitable in a fully unguided and remote study. In our data set, we found that the actual timing of the final questionnaire was 63.7 (SD 26.8) days for the experimental group and 47.4 (SD 10.5) days for the waitlist group participants. To check if this difference in timing can explain the difference in the performance, we split the waitlist group participants into 2 subgroups depending on when they submitted the PHQ-9 questionnaires: the first subgroup included participants who had submitted before 46 days (mean 43.56, SD 1.10 days; nTo check whether the higher reduction in the PHQ-9 scores in the experimental group than in the waitlist group can be explained by motivation, we performed the following analysis. In our study, the waitlist group participants were given the option to sign up for the experimental version of the intervention once the waitlist period was over (ie, when their formal participation in the study had ended, they were not considered as experimental group participants). It is reasonable to expect that the waitlist group participants who actually signed up for this option, despite the long gap of at least 42 days, were more motivated than the rest. We created a subgroup of these more motivated waitlist group participants and compared their performances with that of the remaining participants. These 2 subgroups did not show a significant difference in the reductions in the PHQ-9 scores (Another potential concern is that the users who happened to improve spontaneously may be likely to complete more modules; by performing a completer’s analysis, we may be selecting for such spontaneous improvers. We performed an additional analysis to check whether this was the case in our data. On the basis of this argument, the participants who went on to complete module 3 after completing module 2 would have seen more improvement in their PHQ-9 scores at the end of module 2 compared with the participants who dropped out just after completing module 2. We compared the reductions in PHQ-9 scores (from baseline to the end of module 2) of these 2 subgroups and found no significant difference (On the basis of these analyses, we conclude that the higher reduction in PHQ-9 scores observed in the experimental group can be attributed to the effect of completion of the modules, rather than differences in the timing of questionnaires or in motivation.