Limited access to mental health treatment creates a demand for scalable and non-consumable interventions (25, 26). Despite the high costs and disease burden associated with mental illness (27), we have a decreasing number of clinicians per capita available to provide treatment in the US (28). Increasing the number of human clinicians is not currently feasible, in part because of the decline from 2008 to 2013 per capita for both psychologists (from a ratio of 1:3,642 to 1:3,802) and psychiatrists (from a ratio of 1:7,825 to 1:8,476) (28). Conversational AI has the potential to help address insufficient clinician availability because it is not inherently limited by human clinician time or attention. Conversational AI could also bridge one of the current tensions in care delivery: although clinicians value patient conversations, they have no financial incentive to engage in meaningful but lengthy conversations (29).

The decreasing amount of time spent in meaningful conversations exacerbates the shortage of psychiatrists and psychologists. Psychiatrists' use of talk therapy has been consistently and steadily declining, meaning fewer patients are receiving talk therapy during psychiatric visits (30).  If conversational AI is effective and acceptable to both patients and clinicians, it may address longstanding challenges to mental health access. These include the ability to accommodate rural populations and to facilitate increased engagement from people who may experience traditional talk therapy as stigmatizing (31).

### Quality

Technology has been highlighted as a way to better understand and disseminate high quality psychotherapy (32, 33). Clinicians are already using texting services to deliver mental health interventions (34), which demonstrates a willingness by patients and clinicians to test new approaches to patient-clinician interaction. These new approaches facilitate novel measures of intervention quality. For example, innovations in computer science (e.g., natural language processing and machine learning) are being used to assess language patterns of successful crisis interventions in text-based mental health conversations (18, 35). Computational analysis of psychotherapy is encouraging researchers and companies to identify patterns of patient symptomology and therapist intervention (36, 37). This approach may improve psychotherapy quality by better understanding what effective clinicians actually do. This assessment has historically occurred through clinicians' self-reports or time intensive human audits (e.g., 38).

Although its efficacy is not definitively established, there are reasons to expect that conversational AI could constructively enhance mental health diagnosis and treatment delivery (39, 40). A diagnostic interview aids the patient and clinician in understanding the patient's presenting problem and provides a working model of how problems are being maintained. Approaches vary from highly structured diagnostic interviews [e.g., Structured Clinical Interview for DSM-5 (41)] to unstructured interviews in which the conversation develops based on the clinician's expertise, training, and the patient's features. Conversational AIs have interviewed patients about symptoms for PTSD with a high level of patient acceptance (20). Conversational AI has been piloted across numerous clinically relevant groups such as clinical depression (6) and adolescent stress (42). In a study in which students believed they were speaking with a conversational AI, the students reported feeling better after talking about their problems following the encounter (43). Although these early findings point to potential benefits, there is a lack of rigorous clinical trial data and uncertainty about regulatory oversight (2).

Yet while there is reason for optimism, inflated or unsubstantiated expectations may frustrate patients and weaken their trust in psychotherapeutic interventions (44, 45). Many current computation methods can be used to search for specific dialogue acts, but additional work is needed to map theoretically important constructs (e.g., therapeutic alliance) to causal relationships between language patterns and clinically relevant outcomes. Psychotherapy quality will be difficult to assess without disentangling causal inferences and confounding factors. Beyond computation, patients' attitudes matter in psychotherapy because those who have a negative experience compared with their expectations have worse clinical outcomes (46). If a patient loses trust in a conversational AI, they may be less likely to trust human clinicians as well. As conversational AI becomes more sophisticated and expectations of benefit increase, there are growing concerns that users will transition from feeling let down to feeling betrayed (47). These factors suggest that careful experimentation about sub-processes in AI-mediated communication merits research attention.

### Clinician–Patient Relationship

Modern medicine views the patient–clinician relationship as critical to patient health (48), and provider wellness (49). Indeed, appreciation of the importance of the patient–clinician relationship in modern medicine can be traced back to the influence of clinical psychology (50). Therapeutic alliance develops from clinicians' collaborative engagement with patients and reflects agreement on treatment goals, the tasks necessary to achieve such goals, and the affective bond between patient and provider (51). Therapeutic alliance is consistently associated with symptom improvement in psychotherapy (52–54). Numerous approaches exist to create alliance during psychotherapy, including the use of supportive language, mirroring emotions, and projecting warmth. Although originally conceptualized for human-to-human conversations, users have reported experiencing a sense of therapeutic alliance when speaking directly with conversational AI, suggesting this bond may not necessarily be restricted to human-human relationships (3). If conversational AI can create and maintain a therapeutic alliance, the provision of psychotherapy will not be necessarily limited by human clinicians' time and attention.

Establishing therapeutic alliance with conversational AI may benefit both patients and providers. By allowing conversational AI to take over repetitive, time-consuming tasks, clinicians' attention and skill could be deployed more judiciously (55). Allowing clinicians to do less of the work that contributes to burnout, such as repetitive tasks performed with little autonomy, may improve clinicians' job satisfaction (56). Clinician burnout is associated with worse patient outcomes and is increasingly recognized as a problem which must be more adequately addressed (57, 58).

At the same time, software that augments clinical duties has been criticized for distancing clinicians from patient care (59). In mental health, this risk is especially salient because the content of therapy is often quite intimate. Some of the repetitive, timeconsuming tasks clinicians engage in with patients, such as reviewing symptoms or taking their history, are precisely the vehicles by which clinicians connect with and understand their patients' experiences and develop rapport. It is unknown whether having a conversational AI listen in on psychotherapy will significantly impact patients' and clinicians' sense of therapeutic alliance. This area merits further research.

#### Patient Self-Disclosure and Sharing

Patient self-disclosure of personal information is crucial for successful therapy, including sensitive topics such as trauma, substance use, sexual history, forensic history, and thoughts of selfharm. Patient self-disclosures during psychotherapy are legally and ethically protected (24) and professional norms and laws have been established to set boundaries for what a clinician can share (60). Unauthorized sharing of identifiable patient information can result in fines, loss of license, or even incarceration. Moreover, because of the natural limitations of human memory, patients are unlikely to expect a human clinician to remember entire conversations perfectly in perpetuity. This capacity is in stark contrast to conversational AI, which has near-limitless capacity to hear, remember, share, and analyze conversations as long as desired. Because humans and machines have such different capacities, patient expectations of AI capabilities may impact treatment decisions and consent to data sharing (23).

In mental health, conversational AI has been shown to both facilitate and impede disclosure in different contexts. For example, users were more open with a conversational AI than with a human listener in reporting mental health symptoms (20), and have been successfully used to treat persecutory delusions for people with psychosis (61). Conversely, users were more reluctant to disclose sensitive information such as binge drinking behavior to a conversational AI compared to a non-responsive questionnaire (62). Because personal disclosures are central to diagnosis and treatment in psychotherapy, users' expectations and behavior towards technology-mediated conversations merit further assessment (63, 64, 65).

Certain disclosures in a psychotherapy context carry specific ethical and legal mandates, such as reporting suicidal or homicidal ideation. In 1969, a therapist at the University of California did not share the homicidal ideation of a patient with the intended victim. The patient subsequently killed the named victim, and the victim's family sued. This case (Tarasoff v. Regents of the University of California, 1974) established clinicians' duty not only to protect the confidentiality of their patients but also to notify individuals their patient might harm. A failure to warn leaves a clinician liable to civil judgment (66). Most case law and norms have been established on the premise of a dyadic relationship between patient and clinician. The extent to which conversational AI inherits liability for harm is untested. As conversational AI takes on clinical duties and informs clinical judgment, expectations must be clarified about how and when these systems will respond to issues related to confidentiality, safety, and liability.

### DISCUSSION

Experts in AI, clinicians, administrators, and other stakeholders recognize a need to more fully consider safety and trust in the design and deployment of new AI-based technologies (67, 68). A recent Lancet commission on global mental health states that "technology-based approaches might improve the reach of mental health services but could lose key human ingredients and, possibly, lower effectiveness of mental health care" (33). To inform future research directions, we have presented four approaches to integrating conversational AI into mental health delivery and discussed the dimensions of their impact.

Because conversational AI may augment the work of psychotherapy, we seek to encourage product designers, clinicians, and researchers to assess the impact of new practices on both patients and clinicians. Other areas of medicine have seen success with AI, such as lung cancer imaging and building diagnostic or prognostic models (69–73), and conversational AI for health is an emerging field with limited research on efficacy and safety (40, 63, 74).

Before we deploy AI-mediated treatment, workflow changes must be considered in the context of other demands on clinician time and training. Clinicians are already being asked to be familiar with telehealth (75) social media (76), and mobile health (77), while simultaneously being reminded of the need for selfcare in light of clinician burnout (58). Before we insert new devices into clinical care, it will be crucial to engage clinicians and design evaluation strategies that appreciate the skills, attitudes, and knowledge of affected workers. Just as we can't expect technology companies to easily understand healthcare, we can't expect medical professionals to intuit or work in harmony with new technology without thoughtful design and training.

A limitation of this work is that we do not set out a specific research agenda, and some important considerations are beyond the scope of this work (e.g., the cost and feasibility of each approach). We propose instead that initiatives using conversational AI anticipate challenges and leverage lessons learned from existing approaches to deploying new technology in clinical settings that involve clinician training and patient protections from the start (32, 77). We instead encourage those proposing to put AI into care settings to directly consider and measure impact on access, quality, relationships, and data sharing.

The potential benefits are clear for mental health. If diagnosis or treatment can be done by conversational AI, the societal burden of treating mental health could be diminished. Additionally, conversational AI could have a more long-term relationship with a patient than clinicians who rotate out of training centers. Despite these potential benefits, technology carries risks related to privacy, bias, coercion, liability, and data sharing that could harm patients in expected (e.g., denial of health insurance) and unintended ways (33, 44, 74, 78, 79–81). Conversations are valuable for patients and clinicians, and it is crucial to make sure they are delivered safely and effectively, regardless of who or what does the talking.

### AUTHOR CONTRIBUTIONS

ASM and JH contributed to the initial conceptualization and design of the manuscript. AM wrote the first draft. NS, KDB, BAA, and JB contributed to manuscript revision, read and approved the submitted version.

### REFERENCES

### ACKNOWLEDGMENTS

This work was supported by grants from the National Institutes of Health, National Center for Advancing Translational Science, Clinical and Translational Science Award (KL2TR001083 and UL1TR001085), the Stanford Department of Psychiatry Innovator Grant Program, and the Stanford Institute for Human-Centered Artificial Intelligence. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. We thank Nicole Martinez-Martin JD PhD, Victor Henderson MD MS, and Stan Fisher for their valuable feedback. Reference formatting assisted by Charlesworth Author Services.

**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Miner, Shah, Bullock, Arnow, Bailenson and Hancock. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Using Virtual Reality Exposure Therapy to Enhance Treatment of Anxiety Disorders: Identifying Areas of Clinical Adoption and Potential Obstacles

#### *Debra Boeldt1\*, Elizabeth McMahon2, Mimi McFaul1 and Walter Greenleaf3*

*1 National Mental Health Innovation Center, University of Colorado Anschutz Medical Campus, Aurora, CO, United States, 2 Independent Researcher, San Francisco, CA, United States, 3 Department of Communication, Stanford University, Stanford, CA, United States*

#### *Edited by:*

*Lina Gega, University of York, United Kingdom*

#### *Reviewed by:*

*Raluca Sfetcu, Spiru Haret University, Romania Pawel Izdebski, Kazimierz Wielki University of Bydgoszcz, Poland*

*\*Correspondence: Debra Boeldt<EMAIL_ADDRESS>
#### *Specialty section:*

*This article was submitted to Public Mental Health, a section of the journal Frontiers in Psychiatry*

*Received: 05 June 2019 Accepted: 25 September 2019 Published: 25 October 2019*

#### *Citation:*

*Boeldt D, McMahon E, McFaul M and Greenleaf W (2019) Using Virtual Reality Exposure Therapy to Enhance Treatment of Anxiety Disorders: Identifying Areas of Clinical Adoption and Potential Obstacles. Front. Psychiatry 10:773. doi: 10.3389/fpsyt.2019.00773*

Despite strong evidence of effectiveness, exposure therapy is an underutilized treatment for anxiety disorders at a time when effective treatment for anxiety is greatly needed. The significant worldwide prevalence and negative impact of anxiety are documented and highlight the importance of increasing therapist and patient use of effective treatment. Obstacles to the use of exposure therapy are explored and steps to lessen these obstacles are proposed. In particular, virtual reality (VR) technology is discussed as a way to increase the availability of exposure therapy. Incorporating VR in therapy can increase the ease, acceptability, and effectiveness of treatment for anxiety. VR exposure therapy (VRET) permits individualized, gradual, controlled, immersive exposure that is easy for therapists to implement and often more acceptable to patients than *in vivo* or imaginal exposure. VR is presented as a scalable tool that can augment access to and effectiveness of exposure therapy thus improving treatment of anxiety disorders. VR also has the potential to help with assessment and with therapist training standardization. The authors advocate for providing continuing education in VRET to practicing clinicians and including training in exposure therapy and VRET in training programs. Ongoing development of VR applications for clinical use is encouraged, especially when developed in collaboration with software developers, clinical users, therapists who are experienced in VRET, and researchers.

 The estimated prevalence of anxiety disorders worldwide is 7.3% (2), and they cause a high proportion of the global burden of disease (3). Anxiety symptoms can cause significant distress, impair quality of life, and increase stress. Anxiety increases risk for a range of co-occurring physical conditions, including chronic pain (4). Given the pervasiveness of anxiety and its impact on mental and physical health, effective treatment is clearly needed, yet a majority of affected individuals remain untreated (3, 5). These data highlight the importance for individuals and for society at large of increasing access to effective treatment. This perspective paper presents the ways that incorporating virtual reality (VR) in therapy can improve treatment for anxiety.

VR consists of a fully immersive, 3-D environment that transports people to engaging, interactive environments that can promote new learning. VR technology also has the potential to assist in training, evaluation, delivery, and supervision of psychotherapy skills (6), and can provide patients with a physiologically and emotionally evocative experience which can make VR a valuable tool for mental health treatment (7) VR exposure therapy (VRET) permits individualized, gradual, controlled, immersive exposure that is easy for therapists to implement and often more acceptable to patients than *in vivo* or imaginal exposure (8).This can allow users to practice behavioral skills in a safe environment through the support of a therapist. VRET has been used for the treatment of a range of conditions including social anxiety, Post-Traumatic Stress Disorder (PTSD), and panic disorder (9). The authors present several methods for incorporating VR technology into the therapeutic process and review how VRET can improve the ease, acceptability, and effectiveness of treatments for anxiety.

A patient can encounter difficulties finding or completing evidence-based anxiety treatment. These barriers can include failure to recognize and diagnose anxiety disorders, lack of access to treatment, and less-than-optimal quality of care (10). Since anxiety causes somatic symptoms, anxious patients often seek treatment through their primary care doctor (11). Anxiety disorders may go underrecognized and untreated. Access to effective care is further hampered by a shortage of therapists trained in evidence-based anxiety treatment (12). Limited access to effective treatment results in a large treatment gap allowing millions to suffer from anxiety even though evidence-based treatment exists (5). Obstacles to effective therapy need to be addressed.

Exposure therapy for anxiety disorders has a strong evidence base, yet few therapists utilize this method and patient and therapist misconceptions limit its availability (8, 13). Exposure therapy involves gradual and repeated exposure to feared stimuli with resultant changes in cognitions, behaviors, and emotional and physical responses. Feared stimuli can be almost anything: living organisms, inanimate objects, situations, activities, thoughts, mental images, physical symptoms, and/or affective experiences. Extensive research demonstrates the efficacy of exposure-based therapy for various anxiety disorders, especially phobias (14–16). Exposure facilitates extinction of the fear response and helps change dysfunctional assessments of threat and unhelpful responses, reducing the conditioned anxiety associated with feared stimuli (17, 18). Gradual exposure allows habituation and re-evaluation of the threat to occur.

Unfortunately, although exposure therapy can relieve anxiety symptoms ranging from mild to severe, therapists may not offer it and patients may be reluctant to try it, contributing to the treatment gap for anxiety (19). What are some obstacles to offering this proven effective treatment and how might VR help overcome these obstacles?

#### Obstacles to Exposure Therapy

Several studies have explored why exposure therapy is offered less often than its proven effectiveness would warrant (20–23). Both patient and therapist concerns were uncovered. One barrier is patient fears of exposure therapy. Even when appropriately diagnosed and referred to a clinician offering exposure therapy, patients may refuse treatment or drop out if therapy requires prolonged exposure to their most feared stimuli.

A second barrier may be difficulties arranging exposure. The non-VR options for exposure are imaginal exposure (the patient confronts the feared stimulus in imagination) and *in vivo*  exposure (the patient confronts the feared stimulus in actuality). Difficulties exist in both. In imaginal exposure, therapists cannot know or control what the patient imagines and the ability to create vivid mental images declines with age (24). *In vivo* exposure is often difficult or impossible to arrange inside the office and usually impractical to do outside the office. These above issues all interfere with making effective treatment for anxiety as widely available as is needed (20).

Therapist concerns about exposure therapy are a third barrier. Therapists often worry that exposure will be distressing for both therapist and patient and will increase patient drop out (8, 13, 21–23). When therapists have no way to control exposure, there is a risk that exposure may sensitize the patient and actually worsen anxiety. VR ameliorates this risk. Therapists can control different aspects of the patients' experience during exposure, thus permitting gradual, repeatable, individualized exposure. This minimizes the risk of patient distress and maximizes the chance of patient success.

A fourth barrier is that relatively few mental health providers are trained in exposure therapy. For example, research supports imaginal exposure as a therapy for PTSD, yet a broad survey of community therapists regarding PTSD treatment found that the majority of the licensed psychologists in clinical practice were not using this evidence-based intervention (12). Given projections that in United States the current shortage of behavioral health providers will worsen by 2025, with approximately 40 million individuals not having access to behavioral healthcare (25), it is imperative that the available mental health providers offer the most effective treatments. How can this issue be addressed?

Formal clinical training is one way to correct therapist misconceptions of exposure treatment. Deacon et al. (26) reported that therapists who attended a day-long didactic workshop about exposure therapy showed a decrease in negative beliefs about the treatment approach and an increase in using it. Doctorate-level therapists report fewer reservations about exposure therapy compared to other mental health professionals, perhaps as a result of having more training opportunities (26). Disseminating evidence-based anxiety treatments is essential to ensure that the future mental health workforce has adequate training. Directly addressing clinicians' concerns increases adoption as does training that incorporates motivational enhancement and/or offers a supportive learning community (22, 27). Even therapists trained and interested in exposure, such as imaginal exposure for PTSD, still underutilize the treatment (12, 28).

A final obstacle to offering exposure therapy may be time and/or difficulty involved, especially for *in vivo* exposure. *In vivo* exposure may be prohibited by clinic policy, can be difficult to arrange, take too long, and present confidentiality risks. These factors limit the availability of exposure therapy (28) despite its effectiveness. VR technology can help overcome these obstacles and support patient access to and acceptance of exposure therapy.

#### Applications and Benefits of Virtual Reality

Motivation to integrate technology and behavioral health exists, as highlighted by the *All of Us* research initiative funded by the NIH (29). Presently available VR technology is being used to enhance treatment of anxiety and VR has the potential to improve clinical training (6). VR could create an engaging, controllable, repeatable, and safe training environment (30). Advances in VR allow users to enter a fully immersive environment and have simulated interactions with virtual humans (31). Practice with virtual humans has also demonstrated effectiveness as a way to train for difficult conversations (32). A recent study found that virtual patient simulations can be an effective tool for developing brief clinical interviewing skills among behavioral health providers (33).

In the near future, VR may help provide standardized clinical training in exposure therapy, making training easier and more accessible. A VR training environment would allow therapists to repeatedly practice with virtual patients while mastering clinical assessment and exposure therapy skills. VR might allow clinical supervisors to vary the scenarios and customize the virtual patients within the training environment. Practicing exposure therapy within VR could increase skills and decrease fears about delivering exposure therapy. Increased therapist comfort with and competence in exposure therapy will help therapists provide evidence-based treatment and counter patients' fears about the therapy.

Researchers are also beginning to explore the potential of VR to help assess mental health conditions (34). VR-facilitated clinical assessment may improve the speed and accuracy of diagnosing patients' anxiety disorders, allowing fast referral to appropriate treatment. Improved assessment and treatment should improve patient outcomes (35).

Clinically, VRET is a practical, empirically-based treatment that makes exposure therapy easier and more acceptable for therapists and patients (9). It can help patients learn and practice anxiety management skills and permits controlled, gradual exposure, which minimizes distress and optimizes treatment success. Practically, VR is increasingly affordable. The cost of VR software and hardware continues to decline, while the quantity and quality of VR content increases. VRET has been shown to reduce anxiety and phobia symptoms (36). Research on VRET has proliferated exploring its use with a range of mental health disorders, particularly anxiety disorders such as phobias (9). Over two decades of research exist documenting the effectiveness of VRET for anxiety disorders. Meta-analyses demonstrate a large effect size compared to a control or waitlist condition, and no significant difference in effect size or attrition rates when compared to the gold standard of *in vivo* exposure therapy (9, 36–39). To summarize, VRET is an acceptable and effective alternative to *in vivo* exposure.

VRET allows the therapist to see what the patient sees in the virtual environment. This addresses four limitations of imaginal exposure: not every patient imagines well; the ability to form mental images declines with age; the patient's imagery may be too frightening; and the therapist neither knows nor controls what is being imagined. With VRET, the therapist can choose the VR content and personalize it for the patient. Therapists can guide patients through exposure in the office while monitoring and supporting the patient. Patients feel engaged, and the experience feels "real" but is a safe way for a patient to practice before facing a feared stimuli on their own in a real-world setting (34, 40). Therapists can monitor whether patients are attending to the content. Guiding patients to look at specific content and manage any anxiety that arises helps improve the efficiency of the treatment, ultimately improving patient outcomes. VRET also offers exposure therapy to patients who are resistant to other methods of exposure therapy (41). In the author's (EM) clinical experience, VR has additional uses in psychotherapy beyond exposure. VR helps engage patients in treatment. Relaxing virtual environments (VEs) can help patients learn and practice anxiety management skills and can be used to reinforce patient involvement in treatment and increase positive therapeutic alliance.

The authors further propose that VRET might help manage clinic throughput and can lower costs associated with intensive anxiety treatment. VRET has demonstrated costeffectiveness when compared to treatment as usual for PTSD (42). VRET offers the benefits of an *in vivo* experience in the office without requiring the burdensome time commitment required for therapists to guide, support, and monitor patients during real-world exposure. Such efficiency in providing effective, tailored exposure therapy can help therapists offer evidence-based treatment while handling the demands of a full caseload. Botella et al. (43) reported that successfully treating one phobia using VRET resulted in symptom relief of other, untreated, phobic behaviors. Effective treatment can be efficient treatment.

Lastly, this technology provides an opportunity to increase scalability. In addition to VR's potential uses to improve therapist training, thus increasing the number of therapists with adequate training in exposure therapy. VR self-help interventions for people with distressing, but subclinical, anxiety may ease suffering and may prevent progression to more severe symptoms. A recent study investigated the efficacy of using a consumer, off-the-shelf VR hardware and software to conduct exposure therapy for fear of public speaking (44). The authors compared traditional therapist-led exposure to self-led exposure at home and found that improvements were comparable across both delivery methods. A single-blind, parallel-group randomized controlled trial showed that a low-cost, scalable intervention for fear of heights delivered by a virtual coach resulted in reduced symptoms (45).

#### The Impact on Clinical Practice

The authors do not suggest that VRET will replace the need for trained therapists. The therapist's clinical skills are the key factor to the effective use and implementation of VR (46). As research advances, people may use evidence-based VR self-help programs to prevent subclinical or mild to moderate anxiety from worsening, prepare for therapy when clinical anxiety is present, and use as an adjunct to therapy to get maximum benefit from every session thus saving patient, clinic, and therapist time. Such advances may help therapists manage their caseloads and allow them to focus limited resources on individuals with clinical anxiety. Treatments for specific phobia domains will also increase as has been demonstrated by an automated immersive therapy treatment for acrophobia (45). Such advances will help therapists maintain a manageable caseload and allow them to focus limited resources on individuals with clinical anxiety.

VR and VRET does have limitations. Virtual environments (VEs) are not commercially available for all anxiogenic stimuli. Cybersickness (nausea or discomfort in response to VR) can also limit VR's use with some patients and limit the immersive effects of VR (47). Despite these limitations, VRET is an empirically supported treatment that makes exposure therapy easier and more acceptable. VR is increasingly affordable. The cost of VR software and hardware has declined, while the quantity and quality of VR content has increased.

### Additional Barriers to Adoption and Needs to Be Addressed

Several broader barriers exist to fully exploiting the potential of VR technology. First, the pace of new technology adoption by therapists is generally slow. Behavioral health interventions historically emphasize face-to-face delivery (29) relying on human judgment and assessment (6). This barrier may be reduced by increasing therapist knowledge of VRET's research support, addressing therapist concerns, and offering training in VRET.

Second, therapists wishing to incorporate VR in treatment must decide which software and equipment best suits their needs and may struggle to find information about choosing and using VR. The growing number of VR products and apps can make it difficult to decide what to purchase. Administrators may worry about cost-effectiveness and benefit. This is likely to become less of a barrier as cost for VR equipment and products drops and as more patients request VRET. Publicizing the benefits of VRET to lay as well as professional audiences will increase patient demand, motivating therapists and administrators to offer VR.

Third, since new technology can affect delivery models and require more research and provider input in the process (48), ongoing research on the implementation of VR in a range of clinical settings is strongly recommended. Presently, not all patients become immersed in the currently available virtual environments (37). This may reflect differences in patient ability, therapist skills, and a need for more clinical VR content. Since patients present with complex problems, an expansive library of content is ideally needed for personalized, effective treatment. To date, research has primarily focused on applications of VR for phobias, social anxiety, obsessive–compulsive disorder, PTSD, eating disorders, psychosis, and substance use. Academicindustry partnerships are advancing the development of technology for mental health (32); however, inclusion criteria for studies is often restrictive. Input from a range of therapy providers in diverse clinical settings can help expand the use cases for VR.

Lastly, therapist interest in using VR in therapy is increasing, but training in VRET is generally unavailable. Normalizing exposure therapy approaches and minimizing clinician concerns about the safety and tolerability of exposure therapy are essential for successful dissemination (27). For technology to be effectively adopted, training needs to be available, clinically relevant, and focused on answering therapists' concerns and needs. Resources should be developed that help therapists learn VRET and other therapeutic uses of VR. The authors advocate for the development of accredited continuing education classes and offering training in VRET to students training to become licensed mental health practitioners.

### DISCUSSION

This paper reviews the need for evidence-based treatment of anxiety, the importance of training therapists in exposure therapy, the uses and benefits of VR technology to improve anxiety treatment and explores future applications of VR for training. VR may help reduce current obstacles to wider adoption of exposure therapy by making it more acceptable to patients and easier for therapists. VR is presented as a scalable tool that can augment access to and effectiveness of evidencebased exposure therapy.

In conclusion, research on VR applications and collaborative development of clinical content are recommended to move this tool forward in the field. Therapists experienced in VRET can serve as subject matter experts in the development of content. Training in VRET and other uses of VR should be offered in graduate school and through continuing education. Ongoing expansion of training in VRET for therapists is one way essential to address the vast need for anxiety treatment.

## AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct, and intellectual contribution to the work, and approved it for publication.

### FUNDING

The work of this group is grant-funded from Anschutz Foundation, Award #: N/A.

## REFERENCES

Therapist Beliefs about Exposure Scale. *J Anxiety Disord* (2013a) 27(8):772– 80. doi: 10.1016/j.janxdis.2013.04.006

controlled trial. *Lancet Psychiatry* (2018) 5(8):625–32. doi: 10.1016/ S2215-0366(18)30226-8

**Conflict of Interest:** EM and WG are on the Scientific Advisory Board of Limbix, a company developing virtual reality content for mental health. NMHIC (DB and MM) provides consultation and has a partnership with Limbix.

*Copyright © 2019 Boeldt, McMahon, McFaul and Greenleaf. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Response and Remission Rates in Internet-Based Cognitive Behavior Therapy: An Individual Patient Data Meta-Analysis

#### *Gerhard Andersson1,2\*, Per Carlbring3,4 and Alexander Rozental2,5*

*1 Department of Behavioural Sciences and Learning, Linköping University, Linköping, Sweden, 2 Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden, 3 Department of Psychology, Stockholm University, Stockholm, Sweden, 4 Department of Psychology, University of Southern Denmark, Odense, Denmark, 5 Institute of Child Health, University College London, London, United Kingdom*

#### *Edited by:*

*Lina Gega, University of York, United Kingdom*

#### *Reviewed by:*

*Eva-Maria Pichler, University of Zurich, Switzerland Kazuki Ide, Kyoto University, Japan*

*\*Correspondence: Gerhard Andersson<EMAIL_ADDRESS>
#### *Specialty section:*

*This article was submitted to Public Mental Health, a section of the journal Frontiers in Psychiatry*

*Received: 10 June 2019 Accepted: 18 September 2019 Published: 25 October 2019*

#### *Citation:*

*Andersson G, Carlbring P and Rozental A (2019) Response and Remission Rates in Internet-Based Cognitive Behavior Therapy: An Individual Patient Data Meta-Analysis. Front. Psychiatry 10:749. doi: 10.3389/fpsyt.2019.00749*

Background: Internet-delivered cognitive behavior therapy (ICBT) was developed over 20 years ago and has since undergone a number of controlled trials, as well as several systematic reviews and meta-analyses. However, the crucial question of response rates remains to be systematically investigated. The aim of this individual patient meta-analysis (IPDMA) was to use a large dataset of trials conducted in Sweden to determine reliable change and recovery rates across trials for a range of conditions.

Methods: We used previously collected and aggregated data from 2,866 patients in 29 Swedish clinical trials of ICBT for three categories of conditions: anxiety disorders, depression, and others. Raw scores at pre-treatment and post-treatment were used in an IPDMA to determine the rate of reliable change and recovery. Jacobson and Truax's, (1991) reliable change index (RCI) was calculated for each primary outcome measure in the trials as well as the recovery rates for each patient, with the additional requirement of having improved substantially. We subsequently explored potential predictors using binomial logistic regression.

Results: In applying an RCI of *z* = 1.96, 1,162 (65.6%) of the patients receiving treatment were classified as achieving recovery, and 620 (35.0%) were classified as reaching remission. In terms of predictors, patients with higher symptom severity on the primary outcome measure at baseline [odds ratio (OR) = 1.36] and being female (OR = 2.22) increased the odds of responding to treatment. Having an anxiety disorder was found to decrease the response to treatment (OR = 0.51). Remission was predicted by diagnosis in the same direction (OR = 0.28), whereas symptom severity was inversely predictive of worse outcome (OR = 0.81). Conclusions: Response seems to occur among approximately half of all clients administered ICBT, whereas about a third reach remission. This indicates that the efficacy of ICBT is in line with that of CBT based in prior trials, with a possible caveat being the lower remission rates. Having more symptoms and being female might increase the chances of improvement, and a small negative effect of having anxiety disorder *versus* depression and other conditions may also exist. A limitation of the IPDMA was that only studies conducted in Sweden were included.

Keywords: response rates, recovery, predictors, individual patient data meta-analysis, internet-based cognitive behavior therapy

#### INTRODUCTION

Internet-delivered cognitive behavior therapy (ICBT) has existed for more than 20 years (1), and treatment programs have been developed for a wide range of clinical and non-clinical conditions. Most forms of ICBT are administered in the form of guided selfhelp, with text and video presentations boosted by email support in a secure online platform resembling Internet banking (2). One way to describe ICBT is to pinpoint the similarities with online education, even if the treatment is largely based on self-help texts and cognitive behavioral therapy (CBT) manuals (3). Thus, ICBT programs tend to rely on psychoeducation and instructions for how to change thoughts, emotions, and behaviors in everyday life, and, like CBT, programs usually last 5 to 15 weeks with homework assignments and therapist feedback provided on a weekly basis (3).

Research suggests that therapist-supported ICBT—in contrast to unguided treatments (4)—can be as effective as face-to-face cognitive behavior therapy (5), yield long-term results (6), and work under clinically representative conditions (7). ICBT has also been tested for different target groups—for example, young persons (8), adults (9), older persons (10), and immigrants (11). Treatment programs have focused on specific problems, such as procrastination (12); diagnoses like post-traumatic stress disorder (PTSD) (13); or tailored according to a client's specific problem profile (14). Another approach has been to deliver transdiagnostic treatments in which one treatment is used to target underlying common processes, such as avoidance (15). There are also studies on other psychotherapy forms, including psychodynamic psychotherapy (e.g., 16), interpersonal psychotherapy (17), versions of CBT such as acceptance and commitment therapies (18), and attention training (19). Finally, there are also programs based on physical exercise (20), mindfulness (21), and relaxation (22), even if the latter two are often incorporated into ICBT protocols (23). In addition to the controlled trials, several studies exist on moderators and mediators of outcome (e.g., 24), as well as some qualitative studies on the client's experience during ICBT (25).

While it is common to report effects in clinical trials, there are also studies and reviews reporting negative effects and nonresponse to ICBT. In two previous individual patient data metaanalyses (IPDMA), we studied these two outcomes (26, 27). With regards to negative effects in the form of deterioration, 5.8% of treated research participants showed such effects (26), and non-response was present among 26.8% of participants (27). When completing these two reviews based on our dataset of controlled trials, a question emerged regarding response rates in our ICBT studies as this was not reported in the previous ones given uncertainties regarding definitions and scope of the two previous reviews. In contrast to the standard of reporting mean standardized differences with metrics like Cohen's *d*, there is far less agreement on how to define response in psychological treatments studies in general, and specifically in CBT (28). Several questions emerge when defining what constitutes a "response" to treatment. In a review, 26, p. 73-74) mentioned several issues, such as (a) number of measures used to define response, (b) number of modalities (e.g., self-report *versus* observed behavior), (c) blinding of assessors, (d) degree of change from baseline, and (e) use of a clinical cut-off to determine if a client has reached a non-clinical state (sometimes referred to as high end-state or remission). In a seminal paper, Jacobson and Truax (29) outlined guidelines for defining change from baseline (reliable change index-RCI) and different ways to define what constitutes being within a non-clinical range or having reached high endstate function/remission. In the present review, we will use the term "remission" to refer to what can be counted as high endstate function, which allows us to be consistent with a previous IPDMA on depression by Karyotaki et al. (30).

In their IPDMA on guided ICBT for depression focusing on response and remission, Karyotaki et al. (30) included 24 RCTs (4,889 participants) and compared guided ICBT with a control group. The mean pooled response rate (based on RCI) at posttreatment was 56.19%, and the mean remission rate at posttreatment in the treatment groups (N = 26) was 38.51% based on the RCI criteria (1.96)—two standard deviations improvement from baseline for each measure. Given the dataset we coded based on our own trials, we decided to conduct a new IPDMA (31) knowing that we could use original data across studies to investigate reliable change and remission (high end-state function). As in our previous two IPDMAs (26, 27), we used data from 29 clinical trials. The final dataset with complete data consisted of 1,535 patients who had received ICBT, and trials were categorized into three groups: anxiety disorders, depression/ mood disorders, and other conditions (i.e., erectile dysfunction, relationship problems, and gambling disorder). The aim of the current study was to determine the rates of treatment response and remission in clients who had received ICBT in clinical trials conducted by our group in Sweden. As a secondary exploratory aim, we examined potential predictors of response.

#### MATERIALS AND METHODS

#### Individual Patient Data Meta-Analysis

As described in our two previous IPDMAs (26, 27), we used the scores for individual client and outcome variable across studies (32). In IPDMAs, it is study factors that might be predictive of treatment outcome using the raw data instead of group means, as has been done in previous IPDMAs—for example, on low-intensity psychological treatments (33) and Internet interventions for problem drinking (34). As in the previous IPDMAs, we aggregated available data from clinical trials that we conducted. A complete description surrounding our data collection procedure is presented in Rozental et al. (26). An obvious limitation of using this method is that we cannot assess the risk of bias, which is a common practice in systematic reviews (32). However, by including trials from our own group, we were able to obtain an overall view of response and remission, which we assume could be representative, particularly for Swedish settings. For a complete description of the inclusion and exclusion criteria used, see 26, p.163).

The raw scores from each client in the included trials were entered into one data matrix, and codes for background variables were aligned. Given the complexity of modeling reasons for missing data in such a heterogeneous group of research participants and the fact that we were focused on binary outcomes in this review, we decided to use a complete case approach instead of multiple imputation, as we were convinced that data were not missing completely at random (35). Moreover, complete case analysis has been recommended as the first approach when conducting meta-analyses, even if this approach is followed by sensitivity analyses to detect possible bias in estimates (36).

Sociodemographic variables were occasionally collapsed to facilitate comparisons and to obtain consistency across trials (see 26, 27). Trials were categorized into three categories: (1) anxiety disorders, (2) depression and mood disorders, and (3) other (i.e., erectile dysfunction, relationship problems, and gambling disorder). In **Table 1**, we present an overview of the clients' sociodemographic variables in the trials included (the table overlaps with **Table 3** in 26, p. 169 but does not include the control groups). We present data from baseline in the trials and also the amount of missing data for the full sample.

#### Statistical Analysis

The RCI was chosen based on its widespread use for assessing reliable change (29, 37). As is common practice, the RCI was calculated by taking each individual change score and then dividing this score by the standard error of the difference, i.e., *SE*diff = *SD*1√2√1-*r*. In the formula, *SD*1 corresponds to the standard deviation of a condition at pre-treatment, and *r* is the reliability estimate (38). We calculated RCIs for the primary outcome measure of each included trial and used the test–retest reliability for that specific trial measure (see **Table 2**; also reported in 27). Basically, the RCI sets the limit for when a change score is unlikely to be real (*p* = .05). Following the usual standards, we calculated RCIs for which a change equal to *z* = 1.96 on the basis of a standard deviation unit was used. Following this, each participant in the trials could be classified as either a responder or a non-responder to treatment, with the definition of response being specific to each study and measure used, a similar method to that used by Karyotaki et al. (30). Heterogeneity was tested by entering response rates into the program comprehensive metaanalysis (version 2.2.021; CMA).

We also followed the methods of Karyotaki et al. (30) when calculating remission, again using criteria set by Jacobson and Truax (29). Participants were classified as remitters if they moved two standard deviations below the mean of the clinical group to which they belonged in each study. The resulting cut-off scores indicates remission, which is a hard criterion of remission, often being equivalent to a symptom-free state. For six of the studies, it was not possible to use the two standard deviation criteria due to floor effects, so instead, we used one standard deviation as the criterion in these cases.

In order to investigate possible predictors, we applied binomial logistic regression and used either response or recovery rates as dependent variables. All variables were entered into the model simultaneously. In terms of the variables used, we selected a few clinically relevant demographic and clinical predictors (54) (31). We selected the same variables as in our previous IPDMAs on deterioration (26) and non-response (27). The predictors were (a) symptom severity at baseline, (b) civil status, (c) age, (d) sick leave, (e) previous psychological treatment, (f) previous or ongoing psychotropic medication, (g) educational level, (h) diagnosis, and (i) gender. We also added a separate analysis of the association between publication year and the two outcomes response and recovery.

We present odds ratios (OR) with 95% confidence intervals (CI) in order to reflect an increase or decrease in odds of response and remission in relation to a reference category. In the case of dichotomous predictors (such as gender), the OR reflects the odds of response or remission when a client is female *versus* male (reference). A positive OR thus means better response in women. For continuous predictors (symptom severity at baseline and age), the OR instead represents an increase of one standard deviation above their respective mean. The statistical analyses on predictors were performed using jamovi version <IP_ADDRESS> (55), with the proportions of response and remission performed on a complete case basis (see online **Supplementary Material**).

#### Ethics

The data in the current IPMA were derived from several clinical trials, all of which had received ethical approval from the respective regional ethical review boards at each study location.

### RESULTS

#### Study Characteristics

The 29 clinical trials were coded according to the previously predefined inclusion and exclusion criteria and deemed eligible for the current IPDMA (see 26, 27 for further details). Raw scores from all clients were entered into one spread sheet. The exception was 46 clients who had received either psychodynamic psychotherapy or interpersonal psychotherapy *via* the Internet as a control condition. In total, 1,535 clients were included in this IPDMA. The following diagnoses were included (clinical trials, *k*): social anxiety disorder (9), depression (with/without TABLE 1 | Sociodemographic characteristics of the participants included in the analysis (based on Table 3 in 26).

*n.a., not applicable; NOS, not otherwise specified.*

*aValid percent, i.e., percent of available data, excluding missing data.*

*bPercent, i.e., percent of complete dataset, including missing data.*

*cSelf-rated 0–5.*

*dSelf-rated 0–10.*

*eBased on patients receiving treatment.*

*f Weighted mean and standard deviation.*

*gNumber of hours per week.*

dysthymia) (5), generalized anxiety disorder (3), anxiety disorder (with/without depression) (3), mixed anxiety disorders (e.g., panic disorder as well as social anxiety disorder) (2), specific phobia (2), posttraumatic stress disorder (1), panic disorder (with/without agoraphobia) (1), gambling disorder (1), erectile dysfunction (1), and relationship problems (1) (see 26, p.6). Briefly, most participants in the trials had been recruited from the general population based on self-referral (n = 27). A common practice in the trials was to use structured telephone interviews such as the Structured Clinical Interview for DSM-IV-Axis I Disorders (56) or the MINI-International Neuropsychiatric Interview (57). Some studies had used diagnosis-specific instruments such as the Clinician-Administered PTSD Scale (58). For a description of treatment content original studies, (see 26, 27). The amount of missing data for the primary outcome measures at post-treatment was 12.9%. A complete overview of the clinical trials is presented in **Table 3** (which to some extent overlaps with **Table 3** in 27 but with different results presented).

#### Response and Remission Rates

Using the RCI criteria for detecting response, 1,027 (69.9%; 95% CI: 67.61–72.19) of the 1,535 participants receiving treatment were categorized as treatment responders when using an RCI

*NORC, a National Organization for Research at the University of Chicago.*

*a Reversed scales, higher scores indicate less problems.*

*b Information regarding the time period was unavailable.*

*c Separate estimates for the two subscales.*

*d Lifetime test statistic and past year test statistic.*

of *z* = 1.96. The lowest rates were found in the trials on erectile dysfunction (12.1%) and older adults with anxiety (27.3%), whereas the highest rates were found in the trials on gambling (93.6%) and spider phobia (92.3%). As seen in **Table 3**, the proportions varied, which was confirmed by the CMA program showing a significant heterogeneity (*I2 =* 87.5%; Q = 223, *p* < .001).

Remission was achieved in 540 participants (35.2%; 95% CI: 32.81–37.59) when adjusting for floor effects (the unadjusted proportion was 31.9%). There was a large variation with ranging from 0% (erectile dysfunction, depression, and bias modification for social anxiety disorder) to 82% (gambling) and 69% (spider phobia). As with the response rates, a significant heterogeneity was found (*I2 =* 91.6%; Q = 334, *p* < .001).

#### Predictors of Response

Binomial logistic regressions were calculated with the predefined variables entered as predictors of response. Results are presented in **Table 4** with OR and 95% CI for each predictor. The results that indicate a higher symptom severity on the primary outcome measure at baseline was predictive of better outcome. The odds for responding to treatment decreased when having an anxiety disorder as compared to depression/mood disorder and other diagnoses (i.e., erectile dysfunction, relationship problems, and gambling disorder), and the odds increased if the subject was female. The other variables were not predictive of response.

We also repeated the analyses with remission as outcome (see **Table 5**). In this analysis, symptom severity at pre-treatment was marginally associated with less remission (OR = 0.81). As with the analysis for response, the odds for remission were lower if the subject suffered from anxiety. Gender and the other variables did not reach statistical significance. Publication year was unrelated to response and remission.

### DISCUSSION

The aim of this IPDMA was to obtain estimates of response and remission rates for ICBT with a range of conditions categorized into three groups (anxiety, depression, and other). In line with a previous IPDMA on depression by Karyotaki et al. (30) which included trials from different countries, we found that 65.6% of the treated research participants could be classified as treatment responders. This is slightly higher than the 56.19% reported by Karyotaki et al. (30). While Karyotaki et al. (30) imputed missing data, they also reported that the estimates between complete case analysis and the imputed dataset were minor. In addition, we found that 35.0% of participants could be classified as having TABLE 3 | Characteristics and rates of improvement and recovery for the clinical trials included in the individual patient data meta-analysis.

(*Continued*)

Remission and Response in ICBT

(*Continued*)

**57**

Remission and Response in ICBT

*SCID-I, structural clinical interview for DSM-IV axis I disorders; NOS, not otherwise specified; FAQ, frequently asked questions; BAI, beck anxiety inventory; BDI, beck depression inventory; QOLI, quality of life inventory; ISI, insomnia severity index; n.a., not applicable; ACT, acceptance and commitment herapy; LSAS-SR, Liebowitz social anxiety scale—self-report; PDSS-SR, panic disorder severity scale—self-report; PHQ-9, patient health questionnaire—nine items; GAD-7, generalized anxiety disorder—seven items; BA, behavioral activation; MADRS-S, montgomery–åsberg depression rating scale—self-report; IIEF-5, International Index of erectile functioning—five items; CBT, cognitive behavior therapy; IIEF, International index of erectile functioning; RAS, Relationship assessment scale; MINI, the MINI-International neuropsychiatric interview; BBQ, Brunnsviken brief quality of life inventory; Mini-SPIN, mini-social phobia inventory; HAM-D, Hamilton rating scale for depression; WAI, Working alliance inventory; CAPS, Clinician-administered PTSD scale for DSM-IV; IES-R, Impact of event scale—revised; PDS, Posttraumatic diagnostic scale; PSWQ, Penn ptate worry questionnaire; GAD-Q-IV, Generalized anxiety disorder questionnaire; CORE-OM, Clinical outcome in routine evaluation—outcome measure; STAI, State-trait anxiety inventory; CIDI, Composite international diagnostic interview; ACQ, Agoraphobic cognitions questionnaire; BSQ, Body sensations questionnaire; MI, Mobility inventory; DAS, Dyadic adjustment scale; MSI, Marital status inventory; SNAQ, Snake anxiety questionnaire; ADIS, anxiety disorders interview schedule; FSS, Fear Survey schedule; CBM, Cognitive bias modification; SIAS, Social interaction anxiety scale; SPS, Social phobia scale; SPSQ, Social Phobia screening questionnaire; AR, applied relaxation; SPQ, Spider phobia questionnaire; NODS, the NORC diagnostic screen for gambling problems.*

*\*Based on complete case analysis, i.e., only complete data.*

*aSeparate analyses of deterioration were conducted for the two primary outcome measures depending on the diagnosis of the patient.*

*bFour treatment conditions were included in the study, with/without treatment rationale, respectively, but are pooled in the current analysis.*

*cAn additional outcome measure, International Physical Activity Questionnaire, IPAQ, was used in the study but is not included in the current analysis.*

*dPassive control with the possibility to contact the research team if needed.*

*ePatients were able to choose 10 out of 16 modules to be completed during 10 weeks.*

*f Patients were able to choose 10 out of 19 modules to be completed during 10 weeks.*

*gAn additional treatment group, guided psychodynamic therapy, was also used in the study but is not included in the current analysis.*

*hOne brief orientation session and one session of 3-h prolonged exposure.*

*i In addition to 2 weeks of CBM.*

*j An additional treatment group, interpersonal psychotherapy, was also used in the study but is not included in the current analysis.*

*k Additional outcome measures were included in the original study but lost in the raw data file.*

*l Patients were able to complete up to eight modules selected by the therapist.*

*1Andersson G, Burman M, Norlander A-K, Magnusson K, Stalby M, Svensk M, et al. Internet-delivered couples therapy: a randomized controlled trial. (2019). Unpublished manuscript.*

October 2019 | Volume 10 | Article 749

*OR, odds ratio; CI, 95% confidence interval; p, p-value.*

TABLE 5 | Significance levels, odds ratios, and 95% confidence intervals (CI) for predictors of recovery.

*OR, odds ratio; CI, 95% confidence interval; p, p-value.*

remitted, which is slightly lower than the 38.51% remission rate reported by Karyotaki et al. (30). Moreover, we had a problem with floor effects and if not considering that our estimate is even lower (31.9%). Overall, the results are rather similar to Karyotaki et al. (30) in that roughly half of clients showed improvement following ICBT and remission was achieved by a third.

Given these estimates, the outstanding question is how well this compares against face-to-face CBT. As previously mentioned, there are a few studies in which clients have been randomly assigned either face-to-face CBT or therapist-guided CBT. In the most recent of such study, Carlbring et al. (5) found no differences in effect. The response rates in CBT across different disorders and conditions are difficult to estimate; as to our knowledge, there is no similar IPDMA on response rates in face-to-face CBT. The closest we can get is a meta-analysis on anxiety disorders based on published data (28). In that review, 31% of the studies had defined response using RCI, and the results were similar to the range found in this analysis (44.5–51.1%). However, Loerinc et al. (28) also reported that the use of RCI in combination with a clinical cut-off (as was done in the present study) was associated with a 28% lower response rate (or, as expressed in this study, the remission rate was 28% lower than the RCI response rate). The difference we found was somewhat larger, as about half of the responders also showed remission. From a more naturalistic perspective, Gyani et al. (82) reported that 63.7% of participants in their clinical sample showed reliable improvement following evidence-based face-to-face treatment. In a recent meta-analysis, Springer et al. (83) reported a remission estimate as high as 51% (compared to our estimate of 35%). As stated by Loerinc et al. (28), there is a large variation in how to define both response and remission in CBT trials; thus, one advantage of the approach taken here is that we can use the same approach across studies. However, the definition of remission used (two SDs below the pre-treatment mean) was impossible in some studies (leading us to correct that estimate) and unrealistic in others. Another disadvantage of a statistical definition of response and remission is that it is heavily dependent on the sample upon which it is calculated. This pertains to both RCI and recovery as outlined by Jacobson and Truax (29). Another approach would have been to determine criteria for response and remission independently of the study, which is possible when there is data on non-clinical samples. For example, on the Beck Depression Inventory (44), a 10-point reduction could be seen as indicative of response, and 13-point reduction indicative of recovery. Applying these criteria for the Stella trial (see **Table 3**; 63), in which 57.5% of participants responded according to the RCI and 0% remitted, the corresponding figures of using a 10-point reduction (53.1%) and a score of 13 or below on the BDI (59.4%) paints a slightly different picture (although the results are similar between the RCI criteria and the 10-point reduction). In particular, the difference between the 0% classified as remitters *versus* the 59.4% having a score indicating minimal depression (44) shows the importance of response definitions (for a detailed discussion, see 37). However, in the present IPDMA, we found it to be of value to use similar criteria across trials. Future research could focus more on the external criteria of improvement instead of study-specific criteria. Unfortunately, the BDI is an exception, being widely used in many trials, and for some conditions and measures included in this IPDMA, cut-off scores and nonclinical norms have not been established.

In the present IPDMA, we conducted exploratory analyses to see if response could be predicted. It is important to note that there was no firm theoretical basis for our selection of predictors and, therefore, our findings must be interpreted with caution, as the identification of a significant predictor could have practical implications for future treatment recommendations. This is particularly the case when findings are based on samples larger than is commonly used in psychotherapy trials. However, the finding that symptom severity was not a negative predictor, but rather the opposite is in line with a previous IPDMA on lowintensity interventions for depression (33). From a clinical point of view, this makes sense, as having symptoms makes treatment more relevant than if subclinical or even different symptoms are present in the client. However, it also suggests that our ICBT trials probably included participants without severe symptoms. It is therefore possible that this finding is based on selection criteria used in trials and not necessarily relevant for clinical practice when treatment is offered with fewer restrictions. In contrast to the finding for response, we found a small negative effect of pre-treatment severity for the prediction of remission. These two conflicting results may indicate that large improvements are less likely if the client has more symptoms. However, as remission requires a low level of symptoms, it is less likely that a client with many symptoms will reach that low level. As in treatment research in general, it could be that the search for predictors is best pursued in ordinary clinical settings rather than in wellcontrolled trials. On a promising note, large ICBT effectiveness studies are being reported (84), which may provide clearer insight into the efficacy of different approaches for different populations.

The odds ratio in favor of female participants was surprisingly high (OR = 2.22), which is hard to explain as this is not a consistent result from previous individual trials. One possible explanation is the fact that the gender proportions in trials are nested with the condition treated. For example, in a typical depression trial, there is a majority of women, whereas in other conditions, there are more or less equal proportions of men and women, while some studies have only men (e.g., erectile dysfunction). The overall takeaway message here is that this finding needs further investigation in future trials to confirm its veracity. Gender was not found to be predictive of remission.

Interestingly, while the IPDMA by Karyotaki et al. (30) found old age to be weakly associated with better response (OR = 1.01), there was no such effect in this study suggesting that age is not a predictor of outcome. In Karyotaki et al.'s (30) complete case analyses, baseline severity (OR = 1.16) was found to predict better outcome, which was in line with our findings (but not in their intention to treat analyses). Gender did not predict outcome in Karyotaki et al.'s (30) IPDMA, but, again, their review focused on only depression which means a larger portion of the population was likely female.

In this IPDMA, we found decreased odds for responding to treatment when having an anxiety disorder as compared to depression/mood disorders or other (erectile dysfunction, relationship problems, and gambling disorder). This was a small effect, but still puzzling and hard to understand given that the overall picture is the opposite—indicating a higher response to ICBT in clients with anxiety disorders. Anxiety disorders were also found to be predictive of lower rates of remission.

While non-significant findings cannot be viewed as proof of absence of an effect, it is still interesting that ICBT has had marginal success in finding predictors of change as well as moderators and mediators of outcome (1). There are several possible reasons for this. First, participants in trials are not selected for their differences. Rather, they are chosen based on diagnostic criteria, access to the Internet, willingness to be a research participant, etc., all of which likely reduce the chance of identifying accurate predictors. Second, predictors of dropout (85) may ultimately be more illuminating, even if a given IPDMA was on self-guided ICBT for depressive symptoms. It is, however, interesting to note that they found that male gender (RR = 1.08) and co-morbid anxiety symptoms

(RR = 1.18) significantly increased the risk of dropping out of the study, which is in line with our findings with regards to gender and anxiety disorders. In spite of the overall inconsistency in findings and lack of findings, we believe that IPDMA is a powerful and reliable tool for answering questions regarding predictors of response (86). However, this will require concerted efforts to align both outcome measures as well as data on predictors in order to make studies comparable. Even if we were co-workers and principal investigators in the trials included in this IPDMA, the data were not consistent with regards to background variables etc. It becomes even more problematic when attempting to combine datasets from different research groups.

This study had several limitations. We focus on four, knowing that there are further objections that could be raised. First, we only included our own trials, which were conducted in Sweden. IPDMAs are often selective, as original data sometimes cannot be obtained from authors, but in this case, we cannot generalize the results outside of our own culture and setting. In addition to not including trials from international groups, we also did not include the most recent and unpublished trials from our own groups, and there are other groups in Sweden with trials that were also not included. Furthermore, with the focus on our own studies conducted over a period close to 20 years, there is a possibility of time trends, which we did not focus on in this study. Technological changes may not influence effects, but, to give an example, early studies relied more on printing out text materials (87), whereas recent studies are typically delivered *via* online platforms (e.g., responsive to the presentation format, such as smartphones or computers) (2). Second, even if this limitation is not unique for ICBT, outcome measures were largely based on self-report. Such measures are useful and generally have good psychometric properties but still present a possible risk that selfreported changes do not correspond with actual behavior changes and do not conform to the findings of an interview. Third, as commented on by Loerinc et al. (28), treatment response can be defined by several measures, but here, we focused only on the primary outcomes. Our dataset on IPDMA could also be used to analyze effects on other secondary measures of constructs, like quality of life, as several of our studies have used the same measure for this construct (e.g., 88), and it has been reported that the effects of ICBT might be lower on that construct (89). The fourth limitation relates to the statistical methods used. We decided to report on a complete case basis, but we could not exclude bias, as missing data was not considered.

### CONCLUSIONS

In spite of the limitations, the present IPDMA suggests that ICBT can lead to major reductions in symptoms and that more than half of clients, on average, respond to treatment. A lower proportion remits; thus, there is room for improvement. It is possible that women benefit more from ICBT based on our findings, and symptom severity seems to be predictive of outcome, but these findings should be interpreted with caution. Our findings that studies on clients with anxiety disorders were associated with less response is also notable but should be regarded with care. Future IPDMAs should include more trials and should also consider secondary outcomes, such as quality of life.

#### DATA AVAILABILITY STATEMENT

The datasets analyzed in this manuscript are not publicly available. Requests to access the datasets should be directed to<EMAIL_ADDRESS>
### AUTHOR CONTRIBUTIONS

All of the authors contributed in the process of completing the current study and writing the final manuscript. AR conducted the aggregation of raw scores into a single data matrix and completed the statistical analysis with input from GA and PC. GA drafted the first version of the text and did additional calculations,

### REFERENCES

(\* References marked with an asterisk indicate studies included in the meta-analysis.)

while AR and PC provided feedback and reviewed and revised the manuscript.

### ACKNOWLEDGMENTS

The authors would like to thank all of the students, researchers, clinicians, and administrative staff who, over the years, have been involved in the clinical trials that were included in the individual patient data meta-analysis. GA also thanks Linköping University for support in the form of a professor contract. The code to analyse the data using jamovi can be obtained found in the online supplement.

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2019.00749/ full#supplementary-material

**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Andersson, Carlbring and Rozental. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# What Works and What Doesn't Work? A Systematic Review of Digital Mental Health Interventions for Depression and Anxiety in Young People

*Sandra Garrido1,2,3\*, Chris Millington1, Daniel Cheers1,4, Katherine Boydell5, Emery Schubert6, Tanya Meade3 and Quang Vinh Nguyen7*

1 MARCS Institute for Brain, Behaviour & Development, Western Sydney University, Milperra, NSW, Australia, 2 Translational Health Institute, Western Sydney University, Campbelltown, NSW, Australia, 3 School of Social Sciences & Psychology, Western Sydney University, Milperra, NSW, Australia, 4 NSW Health, Sydney, NSW, Australia, 5 Blackdog Institute, Randwick, NSW, Australia, 6 School of Arts & Humanities, University of New South Wales, Kensington, NSW, Australia, 7 School of Computing & Engineering, Western Sydney University, Parramatta, NSW, Australia

#### Edited by:

Lina Gega, University of York, United Kingdom

#### Reviewed by:

Fahad Riaz Choudhry, International Islamic University Malaysia, Selayang, Malaysia Peter Knapp, University of York, United Kingdom

\*Correspondence: Sandra Garrido<EMAIL_ADDRESS>
#### Specialty section:

This article was submitted to Public Mental Health, a section of the journal Frontiers in Psychiatry

Received: 07 March 2019 Accepted: 20 September 2019 Published: 13 November 2019

#### Citation:

Garrido S, Millington C, Cheers D, Boydell K, Schubert E, Meade T and Nguyen QV (2019) What Works and What Doesn't Work? A Systematic Review of Digital Mental Health Interventions for Depression and Anxiety in Young People. Front. Psychiatry 10:759. doi: 10.3389/fpsyt.2019.00759

Background: A major challenge in providing mental health interventions for young people is making such interventions accessible and appealing to those most in need. Online and app-based forms of therapy for mental health are burgeoning. It is therefore crucial to identify features that are most effective and engaging for young users.

Objectives: This study reports a systematic review and meta-analysis of digital mental health interventions and their effectiveness in addressing anxiety and depression in young people to determine factors that relate to outcomes, adherence, and engagement with such interventions.

Methods: A mixed methods approach was taken, including a meta-analysis of 9 randomized controlled trials that compared use of a digital intervention for depression in young people to a no-intervention control group, and 6 comparing the intervention to an active control condition. A thematic analysis and narrative synthesis of 41 studies was also performed.

Results: The pooled effect size of digital mental health interventions on depression in comparison to a no-intervention control was small (Cohen's d = 0.33, 95% CI 0.11 to 0.55), while the pooled effect size of studies comparing an intervention group to an active control showed no significant differences (Cohen's d = 0.14, 95% CI -.04 to 0.31). Pooled effect sizes were higher when supervision was involved (studies with no-intervention controls: Cohen's d = 0.52, 95% CI 0.23 to 0.80; studies with active control: Cohen's d = 0.49, 95% CI -0.11, 1.01). Engagement and adherence rates were low. Qualitative analysis revealed that users liked interventions with a game-like feel and relatable, interactive content. Educational materials were perceived as boring, and users were put off by non-appealing interfaces and technical glitches.

Conclusions: Digital interventions work better than no intervention to improve depression in young people when results of different studies are pooled together. However, these interventions may only be of clinical significance when use is highly supervised. Digital interventions do not work better than active alternatives regardless of the level of support. Future interventions need to move beyond the use of digital educational materials, considering other ways to attract and engage young people and to ensure relevance and appeal.

Keywords: children, adolescents, unguided self-help, self-management, low mood, prevention

### INTRODUCTION

In Australia, approximately 8% of young people between 11–17 years of age meet the DSM criteria for major depressive disorder (MDD), while about 20% report high levels of psychological distress (1). The rates of MDD may be as high as 11% in youths in the U.S. (2). In fact, suicide is the second leading cause of death among 15–29-year-olds globally (3). In addition, depression is highly under-diagnosed and thousands who fall outside these statistics experience its debilitating effects on functioning at an important developmental stage. Depression and other mental illnesses affect the social and intellectual development of young people, reducing engagement with education, and if untreated, can become lifelong disabilities (4).

Despite the importance of addressing mental illness early only 20–40% of youths in need in Australia (1) and 25% of youths in the U.K. receive professional help (5). This low engagement with mental health services appears to occur for a variety of reasons: the lack of motivation inherent in conditions such as depression (6), low rates of mental health literacy (7), and the stigma, discrimination and embarrassment surrounding mental illness (8). Young people are also still developing skills in executive functioning such as self-monitoring and organization, which are necessary to identify a mental health problem and obtain support (9).

Although few young people seek professional help, during episodes of depression consumption of media such as music, internet, and television increases (10). Thus Digital Mental Health Interventions (DMHIs) are increasingly of interest as a solution to the low help-seeking and uptake rates of professional mental health services. Studies tend to support the effectiveness of self-help mental health programs whether digital or otherwise, which can be as effective if not more effective than face-toface delivery (11). Numerous studies have demonstrated the usefulness of web-based programs (12, 13). Young people report feeling more comfortable discussing sensitive and personal issues in the relative anonymity of an online context and use the internet as a major source of mental health information (14, 15). Mobile "apps" are proving particularly useful for administering DMHIs because of the widespread ownership of mobile phones, with the majority of young people in the U.S. reporting almost constant usage of smartphones (16). Several reviews of smartphone applications for mental health across age groups have reported positive benefits (13, 17).

Notably, however, many apps for depression and anxiety that are currently available are not evidence-based and may thus actually be harmful to people with mental illness (18). Even among those purporting to be drawn from evidence-based therapies such as cognitive behavioural therapy, only a small percentage actually contain the core principles of those therapeutic traditions (19). Furthermore, Hollis and colleagues (20) in their meta-review reported that while there is some evidence in support of the effectiveness of DMHIs for depression and anxiety in young people, studies are methodologically limited making it difficult to draw clear conclusions. Furthermore, they suggest the need for identification of the components that make DMHIs effective such as human interaction.

In fact, human interaction has been identified as an important factor influencing effectiveness and engagement with DMHIs (21), but it may also detract from the cost-effectiveness of DMHIs in comparison to face-to-face treatment (20). Furthermore, other disadvantages to the inclusion of social features exist, such as unhelpful advice from peers, and the possibility that some youths may feel afraid to share personal problems even anonymously.

The aim of the current review, therefore, is to examine the literature about DMHIs to address mental health in young people. We have focused on depression and anxiety as these are among the most prevalent mental health conditions experienced by young people and often co-occur with many other disorders (22–24). Specifically we aimed to investigate:

### METHOD

#### Study Design

Given the focus in the current review on both effectiveness and engagement, it was expected that the literature reviewed could include both quantitative and qualitative data. Therefore, a mixed methods approach was selected. Mixed methods reviews attempt to combine looking at 'what works' with 'how and why it works', combining varied research methods in the analysis or in the types of studies reviewed (25). Mixed method reviews can provide a more holistic understanding than a meta-analysis alone since they are able to integrate a wider variety of studies and provide insights into mechanisms and processes.

### Identification and Selection of Studies

A systematic search was conducted in PsychInfo, PubMed, Proquest, and Web of Science using the following terms in the title, abstract and subject descriptors: "mobile", "application", "smartphone", "mobile phone", "cell phone", "text message", "internet-administered therapy", "computer-aided therapy", "online" AND "depression", "anxiety" AND "youth", "young person", "adolescent". The initial search returned 4,828 articles. With duplicates deleted this number was reduced to 3,352.

Identified references were screened according to the following inclusion criteria: (1) participants aged 12 to 25, (ii) interventions targeting depression or anxiety, (iii) interventions delivered by computer, on smartphones, or online, (iv) studies published between 2007 and 2017. We also excluded reviews, opinion, or discussion pieces and unpublished works. While quality assessment was part of the review process, studies were not excluded on the basis of study type or quality since the mixed methods approach taken allowed the inclusion of varied methodologies.

After title screening by 1 researcher 184 abstracts were uploaded to Covidence, an online platform for conducting systematic reviews (http://www.covidence.org). These were scrutinised by 2 researchers. Once agreement was reached on eligibility, 68 articles remained. Full text appraisals were then conducted by 2 researchers and a further 27 articles not meeting the inclusion criteria were excluded, leaving 41 in the review (**Figure 1** and **Supplementary Materials**).

## Quality Assessment and Data Extraction

Two researchers independently conducted the data extraction. Information about the characteristics of the studies, participants, interventions, and final outcomes were entered into a

pre-established template in Covidence. Qualitative data from the studies relating to usability and appeal of the interventions was also extracted. The quality of studies was assessed using the Joanna Briggs Institute's (26) critical appraisal tools and the CONSORT-EHEALTH Checklist (V.1.6.1) (27). Studies were considered methodologically sound if they had a matched control group, pre-post data, and randomization, and were only included in the meta-analyses if they met these criteria. However, since this is a mixed-methods review, studies not meeting the criteria for inclusion in the meta-analysis were retained in the study to form part of the narrative synthesis. Risk of bias was assessed using the standard Cochrane Risk of Bias tool (28) in Covidence, which poses questions about aspects of trial design, conduct, and reporting which the user rates as 'Low' or 'High' risk of bias for each study. Single cohort studies and RCTs were assessed using different indicators of bias under the 5 categories of possible bias as outlined in the *Cochrane Handbook of Systematic Reviews* (29), and a overall assessment of bias as 'Low' or 'High' given for each study. Where not enough details were reported in the study to assess risk of bias, this was labeled 'Unclear'. Again, however, studies with some risk of bias were not excluded from the analyses but sub-group comparisons were conducted to assess whether results differed depending on bias.

## Analysis

#### Meta-Analysis

Due to a lack of comparable randomized controlled trials (RCTs) for anxiety, the meta-analysis focused on DMHIs targeting depression. For each RCT of a DMHI for depression the effect size indicating the difference between the intervention group and the control group at post-test was calculated in OpenMeta Analyst (30). Effect size calculations (Cohen's *d* or standardized mean differences) were conducted using the means and standard deviations of post-test scores on instruments measuring symptoms of depression such as the depression subscale of the Depression, Anxiety, Stress, Scale (DASS, 31) or the Centre for Epidemiological Studies Depression Scale (CES-D, 32). Scores on these measures immediately after the study were used rather than longitudinal follow-up scores due to a lack of consistent reporting across the studies. Sample sizes used were the number of participants with complete data in each group rather than intention-to-treat numbers. If effect sizes could not be calculated due to a lack of information, the study was excluded from the effectiveness analysis. Pooled mean effect sizes were calculated using the random effects model as considerable heterogeneity between studies was expected. These were calculated separately for studies using a no-intervention control group and those using an active control group. Subgroup analyses were conducted to investigate the impact of other variables on effectiveness by testing for significant differences between subgroups using a fixed effects model. In particular a sub-group analysis according to risk of bias was conducted to assess how study design contributed to outcomes. Since one area of interest in this review was to investigate whether DMHIs are useful for the high percentages of young people not obtaining professional help, we also conducted a sub-group analysis according to the level of supervision or interaction involved in the treatment condition. FIGURE 1 | PRISMA flowchart of article selection.

#### Narrative Synthesis

All studies including those not in the meta-analysis were evaluated using a narrative synthesis model (33). Qualitative data included both results of interviews with participants, and descriptions of survey results relating to the appeal of DMHIs. For example, some studies reported direct comments from participants about their experiences using the DMHI, while others reported levels of agreement with statements about appeal and quality. These data were coded by 2 researchers using standard techniques for thematic analysis to generate an understanding of features that are appealing and aspects that promoted engagement and adherence (34). Codes were grouped to detect patterns, and themes were identified and defined. Once consensus was reached the lead author prepared a narrative analysis, which was checked independently by the other authors.

### RESULTS

### Study and Participant Characteristics

The 41 studies included 11 from Australia, 10 from the U.S., 12 from other English speaking countries, 5 from Northern Europe, 2 from Asia, and one from South America. The majority of studies were RCTs (n = 27) (**Table 1**), while 13 were single cohort studies (including 4 with pre-post designs), and one used a case study methodology. Most of the studies used participants recruited from educational institutions (n = 20), 9 were conducted in mental health care settings, 4 with the general community, 4 in primary care settings, 2 in hospitals, and one in a youth organisation. One study did not report recruitment methods or participant information in enough detail to determine the setting.

Studies included participants with no specific mental health symptoms at baseline (n = 12), some with varying levels of depression (mild to moderate n = 7, moderate to severe n = 3, severe = 1, all levels n = 6), others with diagnosed MDD (n = 2) or suicidal risk (n = 5). Participants with varying levels of anxiety were also a focus in some studies (mild n = 1, mild-mod n = 2, mod to severe n = 1, severe n = 1, all levels n = 1). Two studies looked at people with a variety of mental illnesses, and another focused on mood issues in people with traumatic brain injury (**Table 1**).

### Intervention Characteristics

Overall, 32 different DMHIs were investigated across the 41 papers (**Table 2**). Several DMHIs were evaluated in multiple studies including Bite Back (n = 2), CATCH-IT (n = 2), Master Your Mood (n = 2), MoodGym (n = 3), Reframe-IT (n = 3), SPARX (n = 2), and MEMO (n = 2). Several of these papers reported results from the same data sets (62, 63, 73, 74), but

TABLE 1 | Study and Participant Characteristics.

(Continued)

#### TABLE 2 | Intervention Characteristics.

reported on different aspects of the study and therefore both papers were included in the review. Most of the DMHIs drew on established therapeutic models, primarily Cognitive Behavioural Therapy (CBT) (n = 28) or a combination of CBT with other models.

The technologies utilized in the various DMHIs included some phone-based interventions such as text-messages (n = 4) and smartphone applications containing assessment tools and/ or psychoeducational materials (n = 3). The majority of DMHIs were web-based (n = 30), including many with online modules, learning materials or activities (n = 24), group chats or courses (n = 2), online forums (n = 2), and online chat facilities with a mental health professional (n = 2). Others were computer-based but not online, including games (n = 5) and psychoeducational computer programs (n = 2).

Many DMHIs included learning modules (n = 18), interactive learning activities (n = 6), psychoeducational materials in a variety of formats including text and video (n = 7), or game-based learning activities (n = 4). Additional features included regular inspiring messages (n = 1), challenges (n = 3), mood tracking (n = 3), or diary/journals (n = 2). Four studies included DMHIs with an accompanying workbook for participants or their parents. Only 11 of the studies reviewed included DMHIs that were entirely self-help and were completed in the participant's own time. The rest of the studies involved interaction with a mental health professional or completion of the intervention in some kind of supervised setting. Two of these were completed in hospitals, one with some minimal supervision from a health professional (36). Others were completed in a school setting (n = 10). Some of the studies completed at school involved a high level of supervision (n = 5) such as in studies where the intervention was presented by the school wellbeing staff (47), the classroom teacher (39), in the presence of the research team (62, 63), or in class groups (65). Other school-based studies involved lower levels of interaction with a therapist or the research team (n = 3) such as moderation of an online discussion board by a therapist (38), initial training completed at school but the intervention otherwise used in the students' own time (44), and where the intervention was accessed at school with minimal supervision by the school counselor (68). Studies outside of school settings included DMHIs that could be completed at home in the participant's own time, but included interactions with a therapist such as sending reminders or text messages (n = 4), or participating in online group courses or chats (n = 9).

### Effectiveness of DMHIs

The effectiveness of various DMHIs in treating symptoms of depression was compared to a control group in 15 studies (**Table 3**). Nine of these studies compared DMHIs to no intervention (a waitlist control group), while five of the studies compared DMHIs to an active control in which some alternative online materials were used, including one with some psycho-educational content (68), and one contained a Treatment As Usual (TAU) comparison group in which face-to-face counseling was offered (56). Since the TAU group in this case included active treatment it was combined in analysis with the active control groups. The pooled effect size of studies comparing the intervention group to a no-intervention group (n = 9) was 0.33 (95% CI 0.11 to 0.55) (**Figure 2**), suggesting that DMHIs have a small effect size when compared to a no intervention control group, while the pooled effect size of studies comparing the intervention group to an active control group (n = 6) was 0.14 (95% CI -0.04 to 0.31). Thus this review did not find a difference in outcomes between DMHIs and active controls, including a mixture of usual care for depression and non-depression specific interventions (**Figure 3**). Heterogeneity was relatively high (*I2 =* 70%) and statistically significant (p < .001). Two studies had negative effect sizes indicating that the control group had lower depression scores at post-test than the intervention group (59, 74). Reid and colleagues (59) evaluated the effectiveness in comparison to a waitlist control group of a smartphone application that allowed self-assessment on 8 domains of mood and functioning, referring this information to general practitioners for medical review. No significant effect on depression was found at post-test, but increased emotional selfawareness was reported. Whittaker and colleagues (74) similarly used a phone-based approach, delivering multimedia messages based on CBT and comparing this to use of similar multimedia messages with no focus on depression. The authors concluded that there was no evidence of benefits superior to the active comparison program with content about healthy behaviours.

Sub-group analyses were conducted to investigate the effect of therapist interactions and study completion settings on the outcomes in the 9 studies that compared the intervention to no intervention (**Table 3**), and the 6 studies comparing the intervention to an active control group. Studies were categorised as having High levels of human interaction (H) if they involved direct contact with a therapist or were completed in supervised settings such as a lab, clinic or school (47, 51, 55, 65, 66, 68, 70). They were categorised as Low interaction (L) (43, 49, 53) if they had some limited interaction such as regular emails, text messages, or optional opportunities to contact a therapist, or No interaction (N) (52, 54, 56, 59, 74) if they did not involve any

#### TABLE 3 | RCTs included in meta-analysis.

PHQ, Patient Health Questionnaire; DASS, Depression, Anxiety, Stress Scale, RADS, Reynolds Adolescent Depression Scale; CDRS-R, Children's Depression Rating Scale Revised; K6, Kessler 6; CES-D, Centre for Epidemiological Studies Depression Scale.

\* The study included active comparison groups as well, but only the comparison to the waitlist control group was included in this analysis.

C, control group; H, High level of interaction; involved direct contact with a therapist or were completed in supervised settings; I, Intervention group; L, Low level of interaction; limited interaction such as regular emails, text messages or optional opportunities to contact a therapist; N, No interaction; did not involve any interaction with a mental health professional and were completed unsupervised in personal time.

FIGURE 2 | Forest plot of meta-analysis of randomised controlled comparisons between DMHIs and no intervention for depression in adolescents.

interaction with a mental health professional and were completed unsupervised in personal time.

For studies comparing the DMHI to no intervention, the pooled effect size was smallest in the No interaction group (*d* = 0.04), and also small in the Low interaction group (*d* = 0.16), while the High interaction group returned a medium effect size (*d* = 0.52) (**Figure 4**). This indicates that DMHIs were mostly effective when they involved high levels of human interaction. The DMHIs in the No interaction group that did have a positive effect size were highly interactive, containing multimedia lessons (52), interactive online exercises (54), and game-based challenges and puzzles aiming to improve mental health literacy (56). Levin and colleagues (52) did not include direct conversations with mental health professionals. However, the system did send automatically generated emails that were customized based on participants' earlier input. This could have given the illusion of human interaction, thereby increasing the effectiveness of the program despite there being no direct personal communication. No significant differences were found according to intervention type or severity of symptoms at baseline.

Similarly, for the studies with active comparison groups, the pooled effect size was again smallest in the No interaction group (*d* = 0.08), and also small in the Low interaction group (*d* = 0.21), while the High interaction group returned a medium effect size (*d* = 0.49) (**Figure 5**). Both of the studies in the High interaction group were completed in school classroom settings (55, 68). Thus, across both active and no intervention control groups, effect sizes reached a moderate size only when there was a high level of therapist interaction or supervision in the study design.

### Risk of Bias

The proportion of studies at high/unclear risk of bias was: 34% selection bias (e.g. randomization or allocation concealment), 63% detection bias (e.g. blinding of outcome assessment), 41% attrition bias, and 31% selective reporting (**Table 4**). Overall 76% of the studies included in the review as a whole, and all but 4 studies (43, 66, 68, 74) included in the meta-analysis had some risk of bias. Sub-group analyses were performed on the 15 studies included in the meta-analyses according to type of bias. Significant differences were found for selection bias with the low

FIGURE 4 | Forest plot of sub-group analysis for randomised controlled comparisons between DMHIs with high, low or no support compared to no intervention for depression in adolescents.

FIGURE 5 | Forest plot of sub-group analysis of Interaction level for randomised controlled comparisons between DMHIs with high, low and no support compared to active control groups for depression in adolescents.

risk of bias group having a lower pooled effect size (*d =* 0.01, 95% CI -0.05 to 0.24) than the unclear or high-risk group (*d =* 0.44, 95% CI 0.22 to 0.69). Studies with low risk of detection bias also had a lower pooled effect size (*d* = .09, 95% CI -0.08 to 0.26) than the unclear or high-risk group (*d* = .40, 95% CI 0.18 to 0.62).

#### Attrition, Adherence, and Engagement

Attrition rates were defined as the number of participants who completed the study as a percentage of the participants who commenced the intervention (**Table 5**).Information about adherence and engagement (how much those who completed the study engaged with the intervention), tended to be reported differently across papers. For example, some papers reported module completion rates (36). Others reported time spent on a website (43). Overall 16 (39%) of the studies had attrition rates over 20%, the level broadly considered indicative of possible attrition bias (76) (**Table 5**). In several of these studies, while attrition rates were high, they were equal between groups (for e.g. 50) suggesting that drop out rates related more to recruitment methods than to non-engagement. However, this was not always the case and in many studies, even among those with low study attrition, engagement tended to be low,

#### TABLE 4 | Assessment of bias across all studies

with participants completing less than half of the intervention components (36, 45, 49, 51, 53, 73). The majority of these studies were again ones that involved completion in one's own time. For example, Ip and colleagues (2016) reported low drop out rates and a small effect size. However, participants only completed roughly three of 10 modules and spent about 39 minutes on the website over 4 months, suggesting relatively low engagement.

Qualitative data from all papers were categorized according to features of the DMHIs that were liked by participants, features that were disliked (**Table 6**), and features predicting adherence. There were four key categories of data in relation to liked features. The first category related to *social support.* Several studies reported that participants had found it useful to be in contact with professionals. Participants in one study who had access to a trained supporter in addition to regular text messages reported that they "liked talking to someone who was friendly" (35, p. 101). Participants in one study of a game-based CBT intervention called Pesky gNATs (41) reported liking the fact that doing it on a computer was "not as full on as face-to-face" (p. 15). However this was not perceived by all to be preferable to in-person contact. For example, two studies that evaluated an interactive CBT-based fantasy game called SPARX, both found that some participants preferred face-to-face support from a therapist (36, 56). Similarly, some participants in an online CBT-based group course reported a preference for a face-to-face version of the course (45).

For other participants, it was the opportunity to connect with peers who were experiencing similar difficulties that was helpful. Gerrits and colleagues (56) reported that participants "found chatting to be a pleasant and positive way to talk about being down and their feelings of depression" (p. 6). Similarly, Horgan and colleagues (48) studied the impact of an online forum and reported that participants found it "good to say what was going on aloud (albeit in writing)"

#### TABLE 5 | Attrition rates, sample sizes and indicators of adherence and engagement.

(Continued)

C, Comparator group; I, Intervention group; NR, Not reported.

(p. 87). One participant in the same study stated: "Its about empathy and the realization that you're not alone. That others are feeling the same way you do and are having trouble coping" (p. 87).

For some participants the primary attraction of DMHIs were their *online or computer-based* nature. In contrast to users noted above who reported a preference for face-to-face contact, for many users a key benefit of DMHIs was the privacy they afforded. One participant in an online self-help program, Crystal, stated: "You can kind of do it in a secluded area where nobody is watching you … the privacy is kind of like a really big appeal" (37, p. 28). In one study of an online peer-support website, anonymity allowed participants to share details that they had never shared before, and in fact had "put a lot of effort into hiding" (48, p. 87). DMHIs also had the advantage of fitting into the daily routines of users, connecting with current interests (40), and helping "to bring back a sense of normality" (36, p. 290).  28) It can also be accessed from a variety of locations such as school, home or in a clinic and participants could "learn by myself and at my own pace" (56, p. 7). Participants reported that it was "fun to be able to do it on a computer" (41, p. 13).

Other participants commented on particularly *useful content*. Participants reported that the DMHIs "showed me things I didn't know" (56, p. 7), and helped them learn more about mental health (58). Appreciation was expressed for content that helped participants to learn specific techniques such as problem solving and anger control (71), or challenging negative thoughts (37).

Another major category of the data related to the *look and feel* of the DMHIs. Participants preferred situations, characters, or avatars that were relatable. For example, participants reported that it was helpful when the focus was "situations any teenager goes through" such as school and interpersonal relationships (37, p. 27). Conversely, several studies reported that drop-outs occurred when the content "did not seem relevant for them" (54, p. 7). Other features that participants reported liking included interactive activities (58, 62), and video components (62, 73). Similar comments were made in

#### Garrido et al. Digital Mental Health for Young People

#### TABLE 6 | Liked and disliked features of DMHIs.

relation to DMHIs with a game-like feel. Participants stated that this made engaging with the DMHIs fun (36, 41). It was also important to the users that DMHIs were aesthetically appealing and easy to use and navigate.

One of the most prominent features that participants reported disliking was the *educational content* of many DMHIs. For example in their case study of use of a smartphone app for anxiety, Whiteside and colleagues (72) reported that the participant "appeared less engaged and interested in the background educational content" (p. 86). Multiple other studies reported similar comments by participants, particularly non-completers (36, 45, 54, 56, 58, 73). Educational modules were viewed as too long (37), "tough and sometimes quite tiring" (45, p. 6), "tedious and laborious" (38, p. 6). Some participants argued that it would be more convenient to be able to tailor modules to one's own needs: "I didn't like that you couldn't skip out of something if you already understood the concept"(58, p. 163). Burckhardt and colleagues (2015) suggested that more structured settings and dose effects may have contributed to their negative results, since other studies have found that the number of activities participants are required to do can reach a saturation point (77).

A noteworthy point was that numerous participants reported that the DMHIs often felt *too juvenile or patronising.* Participants did not enjoy using DMHIs that seemed like they were designed for younger children (41, 68). One participant suggested: "make it more grown up" (41, p. 14).

*Technical glitches and difficulties navigating sites* were also frequently cited as reasons for low adherence and engagement (37, 54, 68, 73). Participants stated that DMHIs should be improved to make them "comparable to commercially available games" (36, p. 290). Others reported disliking particular *aesthetic features* such as the colour scheme and a lack of variety of icons, cartoons, and diagrams (37).

*Factors predicting adherence.* Only four studies reported *predictors* of adherence to the DMHIs. Neil and colleagues (57) compared a school-based completion setting for MoodGYm to a community setting, finding that a school-based setting predicted greater adherence. Gender was also a consistent predictor of adherence, with females being more likely to complete compared to males (51, 57, 70). Mental health also played a role, with higher pre-test scores in depression (57), a longer history of mood disorders (51), or low scores in anxiety at pre-test (70) predicting greater adherence.

### DISCUSSIONS

This review aimed to determine the types of DMHIs that are effective in treating depression and anxiety in young people and the components of these interventions most associated with positive outcomes and engagement. Overall, studies in relation to depression demonstrated a small effect size in favour of DMHIs when interventions were compared to no intervention. While this might not always reflect a clinically significant level of change, it suggests that such DMHIs may be of value in the context of public health and preventative interventions. On the other hand, studies comparing DMHIs to active control conditions were not effective. In fact, in two studies the control group actually had lower depression levels at posttest than the intervention group. Both studies included phone based interventions: an app that referred users for medical review, and a program of multimedia mobile phone messages. However, given that there was a risk of bias in many studies included in the metaanalyses, these results should be interpreted with caution. In fact, the two studies in the meta-analysis that reported negative effect sizes were two of the only three studies assessed as having low risk of bias. Studies which did not involve blinding of either group allocation or of outcome assessment tended to have higher effect sizes than studies with low risk of bias, indicating that methodological limitations of the studies reviewed likely inflated the larger effect sizes.

Of further importance in our findings was the fact that only DMHIs involving regular interactions with a therapist or that were completed in a supervised setting reached a moderate effect size in comparison to a no-intervention control group, while DMHIs that involved educational programs completed in the participant's own time were not found to be effective in this study. This suggests that currently available DMHIs may not be effective in causing clinically detectable levels of change unless they involve a high level of supervised use or therapist involvement. These results reflect the overarching significance of human interaction in psychological interventions (78). However, the preference among some participants for human contact revealed in this review existed in tension with the need for privacy and anonymity, suggesting that there is a need for more effective design of DMHIs to fill a gap that traditional face-to-face therapies do not.

Despite this, adherence and engagement rates tended to be low in many studies particularly those where interventions were completed in their own time. This reinforces the idea that many DMHIs are most likely to be useful for people already receiving mental health support or at least those not averse to doing so. However, DMHIs completed in settings such as schools, labs, or clinics cannot reliably indicate their effectiveness in reaching young people outside of these settings. Where DMHIs were completed in their own time or did not include interaction with a mental health professional, effectiveness was much reduced. Nevertheless, for many users it is the anonymity and privacy afforded by the online context that holds the greatest appeal. Therefore, there is a need to balance the competing advantages of anonymity and social support. Other studies have similarly concluded that social networking features in DMHIs are a "gamble" due to the potential for both negative and positive effects (79).

These results indicate two distinct needs in DMHI development. Firstly, a dire need exists to increase the appeal of DMHIs so as to reach the 80% of young people who are not already obtaining professional help. These young people may not understand that their symptoms indicate the need for mental health assistance. Other barriers such as a lack of energy or motivation to engage with complex tasks, or a fear of the stigma of mental illness may prevent them from accessing even DMHIs where such are overtly about mental health or are educational in nature. Research indicates that young men are particularly unlikely to receive professional help (80), which makes the findings in this review that males are less likely to engage with DMHIs than females particularly disturbing. The development of DMHIs that build on the existing interests of young people in non-confronting ways may be of more appeal to this group. DMHIs that particularly cater to the interests of young men are especially needed. Only highly interactive DMHIs involving multimedia materials or game-based activities were successful in studies with low levels of human interaction, suggesting that these types of features are highly appealing.

Secondly, for DMHIs designed to help young people who are already seeking or willing to seek professional assistance but prefer to do so in the relative anonymity of digital settings, it is clear that the help of schools and mental health professionals is a crucial part of the roll out of such interventions. Similarly, some scholars have recommended a model of 'supportive accountability', in which accountability to a supporter or coach can enhance adherence to eHealth interventions (81).

The need for further refinement to available DMHIs was confirmed by our qualitative analysis which revealed factors involved in high drop out rates and low engagement rates in numerous studies. In general, participants liked the online or computer based formats and game-like feel of some DMHIs, particularly when the content was interactive, had relatable situations or characters, and had appealing aesthetic features. However, feedback by study participants or people who withdrew from the studies referred to the boring nature and hard work involved in the online learning modules. Participants found DMHIs with non-appealing interfaces, frequent technical glitches, or material that seemed too juvenile to be off-putting. Other studies have similarly found that making DMHIs easy to use and to

navigate is important to users (82), and in fact criteria like this are commonly used to evaluate usability and appeal (83).

Although some participants seemed to appreciate the opportunity to learn various psychological skills to improve their wellbeing, many disliked the high educational focus of the interventions, the fact that they seemed designed for children much younger, and that they did not match commercially available programs in quality. This highlights the need for future DMHIs to consider the opinions of young people closely in their design. There is a need for feedback from young people and co-design methods to ensure that both content and aesthetics are appealing to the target audience. There is also a need for DMHI developers to ensure that materials are not 'dumbed down' and that they are presented in a way that does not feel like hard work but that builds on the natural interests of young people. This again presents a challenge for developers, to balance the need for simplicity of use with ageappropriate content. This is especially important for DMHIs designed to address depression and anxiety since these conditions are associated with both a lack of motivation (84) and impaired concentration (85), making such users a particular challenge to engage.

The current study was limited by the search terms used. Future reviews should also include search terms such as "internet-delivered", "computer", or "computerised", since this could have picked up a broader range of studies in the current review. Nevertheless, this review demonstrates that while somewhat effective for those who use them, DMHIs fail to appeal to a large proportion of young people. In fact, when compared to active comparison groups including online materials with no psycho-educational content, DMHIs had only minimally better effects. As yet there seems to be a dearth of DMHIs that are likely to attract the large numbers of young people with mental illness who are not already open to receiving professional help. There is thus a need for urgent attention to developing high quality DMHIs that address the weaknesses and focus on the strengths identified, to help young people currently in the shadows to access appealing and accessible tools for managing their mental health. There is also a need for methodologically robust double-blinded RCTs to be designed to provide more stringent testing of the effectiveness of such interventions.

### DATA AVAILABILITY STATEMENT

The datasets generated for this study are available on request to the corresponding author.

## AUTHOR CONTRIBUTIONS

All authors contributed to the conception and design of the review. SG, CM, and DC were involved in searches, quality assessment and data extraction. SG and CM conducted the analyses. All authors contributed to manuscript revision and read and approved the submitted version.

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2019.00759/ full#supplementary-material

### REFERENCES

school-based population. (Report). *J Affective Disord* (2013) 147(1-3):338. doi: 10.1016/j.jad.2012.11.036

adolescents seeking help for depression: randomised controlled noninferiority trial. *Br Med J (Online).* (2012) 344:e2598. doi: 10.1136/bmj.e2598

**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Garrido, Millington, Cheers, Boydell, Schubert, Meade and Nguyen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Video to Home Delivery of Evidence-Based Psychotherapy to Veterans With Posttraumatic Stress Disorder

Derrecka M. Boykin1,2,3\*, Fallon Keegan<sup>4</sup> , Karin E. Thompson1,3,5, Emily Voelkel 3,5, Jan A. Lindsay 1,2,3 and Terri L. Fletcher 1,2,3

<sup>1</sup> VA South Central Mental Illness Research, Education and Clinical Center, Houston, TX, United States, <sup>2</sup> Houston VA HSR&D Center for Innovations in Quality, Effectiveness, and Safety, Michael E. DeBakey VA Medical Center, Houston, TX, United States, <sup>3</sup> Menninger Department of Psychiatry and Behavioral Sciences, Baylor College of Medicine, Houston, TX, United States, <sup>4</sup> Department of Psychology, University of North Texas, Denton, TX, United States, <sup>5</sup> Michael E. DeBakey VA Medical Center, Houston, TX, United States

### Edited by:

Lina Gega, University of York, United Kingdom

#### Reviewed by:

Sally Brabyn, University of York, United Kingdom Gemma Wilson, Northumbria University, United Kingdom

\*Correspondence: Derrecka M. Boykin<EMAIL_ADDRESS>
#### Specialty section:

This article was submitted to Public Mental Health, a section of the journal Frontiers in Psychiatry

Received: 02 May 2019 Accepted: 13 November 2019 Published: 06 December 2019

#### Citation:

Boykin DM, Keegan F, Thompson KE, Voelkel E, Lindsay JA and Fletcher TL (2019) Video to Home Delivery of Evidence-Based Psychotherapy to Veterans With Posttraumatic Stress Disorder. Front. Psychiatry 10:893. doi: 10.3389/fpsyt.2019.00893 Background: The Veterans Health Administration (VHA) has pioneered the implementation of video to home (VTH) technology to increase access to mental health treatments for Veterans facing barriers to receiving in-person care, particularly for posttraumatic stress disorder (PTSD). Randomized controlled trials have established the noninferiority of evidence-based psychotherapies (EBPs) for PTSD delivered through VTH, compared to in-person delivery. Less is known about the use of VTH to deliver EBPs for PTSD in routine clinical practice.

Objective: We examined the provision of EBPs for PTSD delivered via VTH at a large Southwestern VHA PTSD outpatient clinic.

Methods: Data were obtained from chart review of the electronic medical records of Veterans receiving at least one session of Cognitive Processing Therapy or Prolonged Exposure via VTH in the VHA PTSD clinic during the study time frame.

Results: Fourteen providers (including six psychology trainees) delivered EBPs for PTSD via VTH between 2016 and 2018. Providers treated 74 Veterans (33.8% women) from diverse sociocultural backgrounds who ranged in age from 25 to 79. Each provider treated about 3.08 (± 2.18) Veterans using VTH, not including one provider who saw more than 30. A hybrid approach, in which VTH-delivery was coupled with in-person delivery, was used with 70.3% of Veterans across treatment (including sessions completed before initiation and after termination of the EBP). This demonstrates the versatility of VTH for meeting individual patient needs. Most EBP sessions (85.4%) were conducted over VTH. Despite Veterans attending an average of 6.85 (± 4.88) EBP sessions, 50% terminated before session 7. This dropout rate is consistent with national and local EBP completion averages within the VHA. Veterans receiving Cognitive Processing Therapy via VTH were more likely to complete treatment than those receiving Prolonged Exposure. No other patient factors predicted attrition.

Conclusions: This study highlights the use of VTH as "tool in the toolbox" that expands the scope of practice for providers and increases opportunities for Veterans to receive EBPs for PTSD. We describe other potential advantages of using VTH to deliver EBPs for PTSD.

Keywords: telehealth, telemental health, posttraumatic stress disorder, evidence-based psychotherapy, veterans

### INTRODUCTION

Evidence-based psychotherapies (EBPs), including Cognitive Processing Therapy and Prolonged Exposure, are considered first-line interventions for posttraumatic stress disorder (PTSD) and have strong support in Veteran samples (1–3). Despite the resources the Veterans Health Administration (VHA) has invested in these two therapies, Veteran access to and engagement in EBPs for PTSD remain a challenge (4). Relatively few Veterans with PTSD (9%) receive these EBPs (5–7), and approximately half who initiate these treatments terminate prematurely and do not receive a full course of therapy (8, 9). Notable barriers to accessing and engaging in EBPs include logistic issues such as taking time off work or school, arranging child or elder care, physical limitations, and transportation and travel costs (10, 11). Additional barriers include stigma around seeking mental health care (12) and clinical symptoms such as anxiety (13) that may interfere in seeking PTSD care at the VHA. Some women Veterans and Veterans who experienced military sexual trauma (MST) also express discomfort in VHA clinics (14, 15).

The provision of EBPs via video to home (VTH) technology is one promising option for overcoming barriers to care and increasing access to and retention in EBPs for PTSD. VTH is a safe and convenient alternative to in-person care and offers patients the opportunity to receive PTSD treatment in the comfort of their home, work, or other private location. Delivery of EBPs via VTH allows patients and providers to synchronously connect for live, interactive therapy sessions using a secure web-based videoconferencing feature on a personal computer, laptop, tablet, or other device. The content of the VTH-delivered EBPs for PTSD is identical to traditional inperson delivery.

Patients report high satisfaction with VTH-delivered care (16), and noninferiority trials demonstrate the clinical effectiveness of EBPs for PTSD delivered via VTH compared to in-person treatment, though sample sizes were relatively small (17, 18). Although these randomized controlled trials found no differences in dropout between VTH and in-person treatment, this may be because patients were randomized to a condition and were not able to select which delivery option they preferred. Nonetheless, among patients who terminated therapy prematurely, patients receiving care via VTH engaged in more sessions prior to dropout than patients who received in-person care (19).