Discussion
WEAVER
Using email to approach potential survey subjects is often preferred because of its low cost. It is likely to have gained popularity during the COVID-19 pandemic given issues with collecting survey data using face-to-face interviews. Several studies have shown that in surveys of physicians an emailed approach can lead to lower response rates, potentially increasing response bias and reducing external validity. This is also the case in other populations [Our results confirm that, in a nationally representative population of qualified GPs and non-GP specialists, response rates are lower using an emailed approach. The finding of a 1.93 percentage point fall in the response rate is from the ITT analysis and so is relevant to physician populations approached initially by mail, or where it is unknown a priori if they have an email address. The ATET of a 2.37 percentage point fall in the response rate is relevant to physician populations where only an email address is available to researchers, and so for physicians willing to supply an email address. The fall in response rate is higher for GPs than for non-GP specialists. The effect size seems quite small possibly because the control group, though approached by mail, still had the option of completing the survey online given the mixed mode of survey completion available to them. Our estimates are therefore conservative compared to using mail by itself. Our results are, more relevant to surveys using mixed modes of delivery.Weaver et al. (2019) randomised around 1,200 physicians in Minnesota and found a web-only response rate of 15.2% compared to 18.9% in the mail-only mode (3.7 percentage points), slightly higher than in our study. They also compared mixed modes (web-mail and mail-web) so the sample sizes in the web-only and mail-only groups were relatively small and so this difference was not statistically significant. Beebe et al., (2018) randomised 686 physicians, nurses and physician assistants and found a response rate of 38.2% for web only compared to 32.1% for mail only after two reminders, a slightly higher effect size (4.1 percentage points) and in the opposite direction of our results. However, again this difference was not statistically significant.The results should also be interpreted in the context of our longitudinal study that the first 10 annual waves combined a mixed mode of completion with a single method of approach using a mailed paper letter for the main mailout and three reminders. The intervention changed the method of approach to email for the second reminder. We used the second reminder, rather than the first mailout, to avoid a potentially large fall in the total number of responses if email was worse than mail. In the second reminder fewer respondents in total would be included in the trial. We do not think that the percentage difference in response rates would be any different if the intervention was for the first mailing or the first or the third reminder. At the second reminder, the intervention group were approached by email and could only complete the survey online, whilst the control group were approached by mailed letter that included a paper copy of the survey and so had a choice of online or paper copy completion. A limitation is that we are not just comparing differences in the method of approach. Though the control group were approached by mail, they could choose online or paper completion, and so their response rate might be lower if they could complete only online or higher if they could only complete using paper. For those who responded, there is evidence that those in the intervention group were more likely to complete the survey online, presumably because this group were approached by email in the second reminder. In the ITT analysis, 46.1% of control group responded online compared to 52.0% in the intervention group (The study was conducted exclusively within the Australian context, which may limit the generalizability of the findings to other countries with different healthcare systems, professional cultures, and survey response patterns. It would be beneficial to conduct similar studies in other countries to better understand how the intervention is performed in different settings. In addition, we have not examined differences in non-response bias which could arise if those who are more likely to respond to email are different in unobserved ways and this changes the composition of the doctors who respond, e.g. younger doctors are likely to respond to email. Our previous research in the same context has shown mixed modes (mail-online and online-mail) showed evidence of response bias, and that young, male doctors working in remote areas are more likely to complete the online survey [Our definition of a complete response of at least one question answered from Section B is quite conservative compared to what is generally recommended in the literature [As we were uncertain about the impact of email on response rates, we took a cautious approach by not using email in the first and main mailout but chose to use email in the second reminder where potential adverse impacts on overall response rate would be minimised. Our results are therefore conservative estimates of the impact of using an email approach versus mailed paper letter. If the intervention was delivered during the main mailout, the statistical power would have been higher. Although we do have a pilot survey every year and could have tested our hypothesis using the pilot sample of around 2000 doctors, the sample size in the pilot was not large enough and most pilot responses are merged into in the main wave if the surveys do not change so still count towards the main response rate.