Limitations
REGRESSION, ASSOCIATED CONDITIONS, RECRUITMENT
This study has a number of limitations. First, randomization on class level reduced the overall clusters to 277. Despite efforts like pairing randomization units by similar characteristics, differences between the IG and CG participants were found, leading to possible selection bias. This has been taken into account by adding the respective variable as a covariate into the outcome analyses. Second, the attrition rate is high, which was partly caused by the anonymization procedure, allowing a link between baseline and follow-up data without collecting any personal information. While about 19% of participating students did not take part due to absence, another 19% could not be included into the main analysis because of a missing code and thus missing link to baseline data. Third, attrition analysis showed a number of highly significant differences between drop-outs and completers, which reduces the external validity of the study. However, we found no evidence to suggest that this affected the interpretation of findings across groups. Fourth, deviant from the study protocol we did not report primary outcomes as metric scores but as dichotomous changes across all or within one behavioral area. This procedure helps illustrating the results but reduces variance in the outcomes. To ensure that the presented results are not solely due to the dichotomization, the regression models were repeated using continuous outcomes and comparable results were found. For example, there was also a significant difference between the IG and CG groups in the absolute minutes used for social media. Fifth, it was not possible to study longer-term effects of the intervention as the observational period was too short. Lastly, the COVID-19 pandemic and associated conditions for schools and teachers clearly impacted the recruitment process. The pandemic—among others—was a regular argument for schools to not fully participate in the study. Digital components like the app itself or video calls helped greatly if the presence of research staff was not allowed because of contact restrictions. However, the recruitment might have been significantly easier and less selective in non-pandemic times.