Strengths and Limitations
’
STILL
As a pragmatic cluster randomised trial in a real-world setting, with clinical sites covering 25% of the Norwegian population, the findings may be considered robust and relevant to similar implementation efforts in the health services. To our knowledge, the IFIP trial is the first large-scale effort to implement basic family involvement practices in CMHCs, and the BFIS scale is the first instrument to assess such practices systematically.In terms of external validity, our results describe practices towards a specific patient group and their relatives, in a particular clinical, geographical, and cultural context. Still, the generic character of many of the interventions and implementation strategies used suggests that these may be suitable in other clinical settings as well. By providing continuous feedback on the results (formative assessment), the external validity of the findings is limited to interventions that employ a similar implementation strategy (Lilford et al., The study design could have been more suitable to evaluate the implementation strategy, if there was a ‘placebo’ implementation strategy in the control arm. However, this would have resulted in contamination of the sub study on patients’ and relatives’ outcomes. Since participants and researchers could not be blinded, there is a possibility that the sites’ allocation status influenced both the performance and evaluation of the respective arms. It was a deliberate and pragmatic decision to have the fidelity reviewers provide implementation support and supervision, because the insights gained through fidelity measurements enabled them to tailor the supervision to the respective unit’s needs. However, one could argue that they consequently assessed some of the results of their own effort, which introduced a risk of experimenter bias. This is most relevant when considering the results measured with the GOI scale and the BFIS subscale that examined structure, content, and implementation (BFIS-S). Both scales contain structural, procedural, and organisational elements, which level of implementation was influenced by the fidelity assessors through their supervision of the implementation teams. Yet, many of these elements are less susceptible to experimenter bias, because they are less open to interpretation. Examples include whether or not units had written information, procedures on family involvement, appointed a family coordinator, constituted an implementation team, or the percentage of clinical staff with FPE training. Since training and supervision in FPE was provided by an independent organisation, which had nothing to do with fidelity assessments, the results measured with the FPE scale (primary outcome) were not subject to a similar risk of experimenter bias.By removing or altering a few elements of the BFIS scale after the baseline measurements, we potentially risked introducing bias and overestimating the intervention effect, if elements were removed that appeared hard to implement. However, the elements removed were covered by the other scales and the elements altered were generally made stricter and more specific. Fidelity raters did not observe FPE sessions, interview service users, or assess randomly selected patient records, all of which could have increased the validity of our findings. Concerning predictive validity, the present paper does not report on patients’ and relatives’ outcomes, but such data will be analysed and reported on later as part of the trial.