# Benefits and Harms of Large Language Models
in Digital Mental Health

Munmun De Choudhury†, Sachin R. Pendse, Neha Kumar
School of Interactive Computing, Georgia Institute of Technology, Atlanta, GA,
USA
{munmund, sachin.r.pendse<EMAIL_ADDRESS>

###### Abstract

The past decade has been transformative for mental health research and
practice. The ability to harness large repositories of data, whether from
electronic health records (EHR), mobile devices, or social media, has revealed
a potential for valuable insights into patient experiences, promising early,
proactive interventions, as well as personalized treatment plans. Recent
developments in generative artificial intelligence, particularly large
language models (LLMs), show promise in leading digital mental health to
uncharted territory. Patients are arriving at doctors’ appointments with
information sourced from chatbots, state-of-the-art LLMs are being
incorporated in medical software and EHR systems, and chatbots from an ever-
increasing number of startups promise to serve as AI companions, friends, and
partners. This article presents contemporary perspectives on the opportunities
and risks posed by LLMs in the design, development, and implementation of
digital mental health tools. We adopt an ecological framework and draw on the
affordances offered by LLMs to discuss four application areas—care-seeking
behaviors from individuals in need of care, community care provision,
institutional and medical care provision, and larger care ecologies at the
societal level. We engage in a thoughtful consideration of whether and how
LLM-based technologies could or should be employed for enhancing mental
health. The benefits and harms our article surfaces could serve to help shape
future research, advocacy, and regulatory efforts focused on creating more
responsible, user-friendly, equitable, and secure LLM-based tools for mental
health treatment and intervention.

$\dagger$ Corresponding author

##  1 Introduction

> “When I use ChatGPT to talk things through and vent about how I feel, it
> goes on to tell me to get help and that I’m not alone. But why does it feel
> as if it’s mocking me? It feels as if it’s having a laugh at my expense.” –
> A paraphrased social media post

In November 2022 [1], OpenAI released ChatGPT. ChatGPT followed the mold of
past chatbots by providing a simple interface for people to easily interact
with a conversational agent. However, unlike past publicly accessible
chatbots, ChatGPT was powered by OpenAI’s proprietary language generation
model, often called Large Language Models (LLMs). OpenAI’s LLM (named GPT, for
Generative Pre-trained Transformers [2]) was created through a large-scale
collection of text from the Internet combined with manual review through a
process often called Reinforcement Learning From Human Feedback (RLHF) [3].
ChatGPT’s underlying language and simple interface astonished users with
answers that were surprisingly coherent and wide-ranging. Since then,
conversations across academic, medical, industry, and policy domains have
begun to discuss how LLMs could offer new possibilities for diagnosis,
treatment, and patient care in mental health.

Over the past decade, there has been increased conversation around the growing
potential for digital technologies, artificial intelligence (AI), and machine
learning to contribute value to mental health research and practice. Research
has demonstrated some of this potential. For example, methods from natural
language processing (such as sentiment analysis) have been used to assess
people’s emotional states from their text, speech, or social media language
[4, 5]. These studies have consistently shown that computational or predictive
analyses of digital data can accurately detect mood [6], mental health states
[7], and even the risk of potential harm [8] and suicide [9]. Collectively,
the implications of this research include the potential for valuable insights
into daily patient experiences, a paving of the way for early and proactive
interventions, and the design of personalized treatment plans. However, as
people further rely on online tools to seek care for their mental health,
researchers and activists have sounded the alarm about the potential for harm
if digital mental health interventions are staged without the awareness or
consent of people experiencing distress [10, 11]. Scholars have also expressed
concern that the use of predictive analytics in mental health could compromise
patient and clinician agency [12], exacerbate systemic disparities encoded in
the training data of AI models [13, 10], and propagate insights poor in
clinical grounding or construct validity [14]. The challenges in implementing
an AI-informed mental health care model have further invited criticism and
skepticism around the role of AI in this field [15], even as researchers have
increasingly sought to draw upon both computational and psychiatric expertise
and paradigms to advocate for an “(AI) model-based psychiatry” [16]. With the
rapid introduction of LLMs in new parts of everyday life, initiatives in
digital mental health are likely to be similarly disrupted, opening doors to
new opportunities for mental healthcare, while also setting the stage for new
and previously unconsidered harms.

This discussion of benefits and harms can already be seen unfolding in popular
discourse around LLMs and mental health. For instance, ChatGPT was not created
as a mental health support tool—however, people in distress have started to
use ChatGPT for mental health support and non-judgmental guidance, as the
opening quote to this Introduction shows. As discussed in online communities
(such as Reddit [17]), users have turned to ChatGPT in moments of suicidal
ideation, and expressed their belief that ChatGPT saved their life in moments
of despair. Other users have described their approach to carefully training
LLM-based chatbots to behave as therapists from different theoretical
orientations, such as prompting chatbots to take on the role of an Acceptance
and Commitment (ACT) therapist. Several mental healthcare organizations and
companies [18, 19, 20] have also begun to research the integration of LLMs
into the design of their services. This increased use of LLMs in service
delivery has been met with excitement [21], but also with justified
skepticism, given potential racial or gender biases [22, 23] and unexpected
outputs [24] from LLM-based chatbots. For example, in June 2023, the National
Eating Disorder Association was forced to shut down a chatbot created to
provide clinically validated information after the chatbot provided harmful
and dangerous advice to users, including diet and weight loss advice [25, 24].
Unbeknownst to the administrative staff at NEDA, the chatbot company that
provided its services to NEDA had “rolled out an AI component to its chatbots”
[24]. The chatbot was introduced soon after several pivotal events, including
an attempt by NEDA helpline volunteers to unionize and the closing of the NEDA
helpline, raising awareness of the potential that nascent technology may be
used to replace human staff [25]. The harms of irresponsible uses of LLM-based
technologies can even be lethal—according to a report from the Belgian news
outlet La Libre (via Belga News Agency) [26], a Belgian man is said to have
tragically ended his own life after engaging in conversations with an AI
chatbot, for six weeks, discussing climate change.

Many of the arguments that highlight the potential harms of LLMs are
warranted. LLMs are trained on an Internet that is largely devoid of fact-
checking. As a result, LLMs often reproduce convincing misinformation [27],
and in the context of the COVID-19 pandemic, were found to be capable of
generating highly persuasive, difficult to discern health misinformation about
COVID-19’s precautionary and prevention measures [28]. Some have described
LLMs as being similar to super-powerful auto-completion tools, as it can be
hard to systematically control for specific outputs [29]. This can be
problematic in a mental health context, where the success of interventions can
be highly dependent on the nature of a provider’s response [30, 31, 32]. LLMs
have been described as a “blurry JPEG image” [33] of the rest of the Internet,
similarly containing both substantial utility as well as the potential for
harm to users. Today, just as in digital mental health spaces, there are
widespread debates around the consequences of LLMs for truthful public
discourse on one hand, and productivity and efficiency on the other.

This article is situated against the backdrop of this larger debate. We
present a state-of-the-art informed perspective on contexts where LLMs can
potentially be beneficial, and where there may be significant risk of harm. We
conclude with directions and recommendations that can help amplify benefits
while also minimizing harms.

##  2 Theoretical Frameworks

Figure 1: An ecological conceptualization of the use of large language models
in digital mental health, based on the Social Ecological Model [34].

Drawing on Insel’s [35] four key areas where LLMs could influence mental
health, we conceptualize benefits and harms in the use of LLMs for
administering tele-mental healthcare, for supporting crisis response, for
providing clinical decision support in psychiatric settings, and for
psychotherapy. To organize our discussion, we adopt the Social Ecological
Model widely used in the public health field [34] and map each of Insel’s
potential application contexts into a four-tier framework. This ecological
analysis allows us to understand the complex interplay between individuals and
the broader social and environmental contexts they are embedded in—contexts
that influence and shape people’s mental health as well as contexts in which
LLM-based technologies could exist.

Figure 1 shows this organization of the potential uses of LLMs in mental
health. The innermost layer focuses on the individual seeking mental
healthcare and considers personal factors such as values, beliefs, knowledge,
skills, and experiences in how an individual might appropriate LLMs to seek
help and support to overcome their distress. The next layer is the caregiver
layer, where LLMs might be appropriated by supporters, counsellors, crisis
volunteers, and others to respond to an individual in crisis. The third layer
centers around institutional efforts and initiatives in caregiving, where we
discuss the role that LLMs can play in enabling decision-making within health
systems. At the final layer, we discuss potential futures where LLMs could
transform telehealth paradigms.

In addition, to systematically identify and discuss the factors that can
support or hinder caregiving at each level resulting from LLMs, we adopt the
concept of affordances [36]. Affordances refer to the potential actions or
uses that an object, environment, or system offers to individuals based on
their perceived characteristics or features [37]. To understand the benefits
and harms of LLMs in digital mental health, we begin by describing the various
affordances of LLMs that are relevant for the above four-tiered ecological
framework. First, LLMs enable natural language understanding—users can input
text-based queries, prompts, or commands in natural language, and the model
interprets and responds to them coherently. Second, by offering end users the
ability to engage in open dialogue, and by parsing the semantics of the user
input, LLMs allow users to interact with conversational agents in a natural
environment [38]. These affordances are relevant to the use of LLMs in
psychotherapy. Next, LLMs are extremely powerful information retrieval systems
and are able to parse very large repositories of data efficiently—users can
thus request information on a wide range of topics by asking questions or
requesting explanations and the model can provide answers, summaries, or
context on various subjects. This affordance is relevant to the use of LLMs in
crisis response. Third, LLMs can provide predictions of outcomes of interest
by learning from its underlying data—users can seek answers, recommendations,
and suggestions from the LLM for various scenarios and use cases, making this
affordance relevant to clinical decision support. Finally, LLMs also allow
efficient text summarization—users can request concise summaries of longer
texts, making it easier to grasp the main points of an article, document, or
report. Complementarily, LLMs can be used for content generation and machine
translation tasks. Together, these features can be highly valuable for
envisioning new models of telehealth.

##  3 For Careseekers: Psychotherapy Chatbot

Perhaps one of the most widely spoken of applications of LLMs in mental health
centers around their use for psychotherapy; Insel discusses the potential for
LLMs, particularly GPT-4, to serve as autonomous therapists [35]. In fact,
LLM-powered chatbots like Replika are already claiming to be an “AI
comparison, […] always ready to chat when [a person] needs an empathetic
friend” [39]. However, for decades, the idea of using a machine to serve as a
therapy bot has met with significant heated debate and controversy,
specifically around their value to individuals in distress. For example, the
Rogerian therapy chatbot ELIZA was first created by Joseph Weizenbaum as a
commentary on the irreplaceability of human interaction in mental health
support [40]. After careseekers in distress found ELIZA to provide useful
support, some mental health professionals argued that chatbots may be able to
scale up basic forms of mental health support [41]. Weizenbaum was shocked by
this enthusiasm, and responded that “no humane therapy of any kind” can or
should be done via a chatbot computer program [42], grounding his argument in
Rogers’s own ideas around person-centered therapy. These debates continue
today, and this section synthesizes the salient points in these conversations.

###  3.1 Potential Benefits

#### Improving the Reach of Therapeutic Care.

Mental health conversational agents have historically been rule-based, meaning
they would engage with users based on predetermined scripts [43]. This
limitation made it challenging for conversational agents to deliver human-like
interactions, as they could not engage in open-ended conversations that were
tailored to users’ emotional requirements. LLMs (grounded in generative AI)
have demonstrated impressive performance in participating in realistic human-
like conversations in a coherent manner, following practically any type of
prompt from the end user [44].

A subsequent benefit offered by this type of naturalistic LLM-powered chatbots
in psychotherapy is accessibility. Chatbots are available around the clock,
providing users with a convenient and immediate source of support. This can be
especially valuable for individuals who have difficulty accessing traditional
in-person therapy due to geographical or scheduling constraints [45], those
living in underserved areas, such as mental health professional shortage areas
[46, 47], or those who might be otherwise marginalized in conventional mental
health care [48, 49]—over half of U.S. counties lack a single psychiatrist
[46], and a recent survey revealed that 60% of mental health professionals
have had no available slots for new patients following the COVID-19 pandemic
[50]. Research has demonstrated that online and self-guided single-session
interventions can reduce depressive symptoms [51, 52]. The naturalistic and
accessible interface associated with LLM-powered chatbots may allow for a new
modality to deliver efficacious and self-guided psychotherapy interventions.

#### Improving Therapeutic Quality through Personalization and Evidence-based
Adaptation.

LLM-based chatbots are trained on large amounts of historical data, to the
order of many terabytes of data [53]. Current and future LLMs could be further
fine-tuned to be better at providing support based on whether past support
responses (from humans or the chatbot) were evaluated by humans as being
efficacious. This might include the specific emotional patterns, style, or
tonality of response that align best with the needs and expectations of a
given client or patient. Researchers have demonstrated that natural language
processing techniques that harness the power of LLMs can empower and equip
supporters with writing suggestions during practice training sessions. For
example, Hsu et al. [54] create the CARE system for mock chats, which uses
LLMs to offer suggestions to online peer supporters as they undergo training.
Participants found recommendations from the CARE system to be helpful,
particularly when faced with uncertainty around how best to respond during the
mock training session. Learnings from LLM-based systems can be directly
applied when supporters are helping people in distress without the aid of any
computational system. Similarly, Sharma et al. [55] developed HAILEY, an AI-
powered agent that offers real-time guidance to enhance the empathetic
responses of peer supporters as they assist those in need of support. The
research found that peer supporters can effectively incorporate AI feedback
with this system, benefiting from both direct and indirect AI assistance,
without developing an excessive reliance on AI. In contrast, supporters
reported enhanced self-efficacy after receiving AI feedback, underscoring the
promise of LLMs as teaching aids for supporters. LLMs can further allow for an
enhanced ability to use data for actionable and evidence-based insights into
the nature of support, including how to adapt support for the individual and
their context.

#### Destigmatizing the Seeking of Care.

LLM-based chatbots can be built to draw on evidence-based psychotherapy
techniques to deliver helpful support exercises for a user, with potentially
less stigma than traditional psychotherapy. Recent empirical research has
demonstrated that language models can be utilized to help individuals work to
reframe negative thoughts and beliefs through in-context learning [56], if
trained on relevant data. For example, Ziems et al. [57] draw upon six
theoretically grounded reframing strategies to demonstrate that language
models can be used to identify and reframe various types of cognitive
distortions. Similarly, Sharma et al. [56] utilize a dataset consisting of
potential thoughts and corresponding reframed thoughts (validated by
practitioners) to train an LLM that generates reframed thoughts for new
contexts.111This system is publicly testable at bit.ly/changing-thoughts. This
work demonstrates the promise of LLMs in providing CBT-based exercises that
are personalized to a user’s context, which may lower the barriers to seeking
treatment or support, often stemming from stigma, shame, or structural
factors, as has been argued for AI as well [58, 59]. LLM-based chatbots may
thus motivate more people to reach out for mental health needs, and help
normalize the process of seeking care through self-disclosure and processing
of thoughts, whether with other humans or chatbots. Such normalization of
care-seeking has been identified to be very important to fight the prevailing
societal stigma surrounding mental illness [60].

###  3.2 Potential Harms

Black [61] has noted how those working at the intersection of AI and care are
quick to explain that “a chatbot will never equal, let alone surpass, the
abilities of a human therapist or counselor.” However, a growing number of
commercial initiatives are building LLM-informed psychotherapy chatbots, with
a belief that “mental health chatbots [can be] instantly and all but
universally available at a fraction of the cost of a therapist” [61]. Given
the pace of development, while there may be benefits for those in distress, a
careful consideration of harms is crucial.

#### An Erosion of the Therapeutic Alliance.

The therapeutic alliance, characterized by trust, mutual respect, and
emotional connection, is a key predictor of psychotherapy outcomes. The
emotional connection built between psychotherapists and their clients plays a
pivotal role in developing this alliance [62]. Patients are more likely to
engage actively in psychotherapy sessions and work towards making behavior
changes when they perceive a sense of genuine care and understanding from
their therapist [63].

Chatbots may be trained on billions of documents that describe rich emotional
experiences, but they lack the capacity for genuine emotional connection.
Psychotherapists are trained to not only understand a client’s emotional
state, but to also empathize with a client’s feelings through their own lived
emotional experiences. Psychotherapists discern the nuances of complex
emotional experiences, including co-occuring sadness, anger, and fear, and
respond in a compassionate manner. They can also pick up on non-verbal cues –
body language, facial expressions, tone of voice, and even pauses in speech
are all non-verbal (or textual) cues that psychotherapists interpret [64]. LLM
powered chatbots, in contrast, are limited in their capacity to understand or
interpret the broader context of the client’s experiences, as expressed
through these non-verbal cues. This limitation makes it more difficult for a
connection to be built between chatbot and user.

The emotional connection between therapist and client is also foundational to
building trust and rapport, particularly when clients are assured that their
expressions of distress and vulnerability are confidential [65, 66]. This
trust is essential for successful psychotherapy, as it encourages open and
honest communication. Psychotherapy excels in an environment when patients are
comfortable sharing their deepest concerns, fostering a therapeutic alliance
[67]. To date, it is unclear if chatbots can build the type of trust and
rapport that is essential to successful therapy, particularly given the
potential for digital mental health data to be leaked, sold, or even legally
used for targeted advertising [68, 69]. LLM chatbots risk taking away the
human element of psychotherapy. Existing studies have demonstrated that users
express negative reactions to unhelpful and risky chatbot responses [70],
which is dangerous – first experiences with mental healthcare can quickly set
the tone for future engagements (or disengagements) with mental healthcare
[71].

#### A Lack of Appropriate Safeguards for Patient Safety.

The excitement for LLM-based therapy bots needs to be tempered with the
reality and safety issues they pose to clients and patients. Generative AI-
based conversational agents have been found to be unable to recognize, and
respond appropriately to, signs of distress [70]. Most commercial chatbots for
mental health claim psychology-trained workers to carefully write and evaluate
the dialogue for these bots [61]. However, in the interest of safety, chatbots
can be constrained by predefined scripts [24] and may struggle to adapt to the
intricate and evolving emotional states of clients. Research has also shown
that while chatbots can offer structured interventions that can be easily
scaled up (such as self-guided exercises from CBT), sustained user engagement
often requires the involvement of human psychotherapists [72]. It is also
important to note that many mental health issues involve complex and
multifaceted emotional experiences. Patients dealing with trauma, grief, or
deep-seated emotional struggles often require nuanced, empathetic support that
goes beyond providing information or structured interventions [73]. Human
therapists can adapt and respond to the evolving needs of their clients,
maintaining their motivation and commitment to therapy, and tailor their
responses and interventions to the unique needs of each individual [74]. LLM
chatbots may lack capacity for such personalized adaption of therapeutic
techniques in response to unique patient needs.

##  4 For Care Providers: Crisis Response

The introduction of the 988 crisis response number in July 2022 was a
significant step forward in the field of mental health [75, 76]. This new
service provides individuals in crisis with immediate access to mental health
support, as an alternative to calling 911 for police or fire department
assistance. However, one challenge that remains is the scarcity of a trained
workforce to respond effectively to mental health crises, often resulting in
calls being transferred to 911 and risking additional harm to callers [77].
Between 2015 to 2020, 23% of people who were killed by police in the U.S. were
experiencing mental health crises at the time [78]. Insel [35] highlights the
potential of LLMs in addressing this issue, emphasizing their dual role in
potentially detecting the severity of a crisis and supporting 988 in providing
real-time assistance. Below, we explore the benefits and harms of using LLMs
in this context.

###  4.1 Potential Benefits

#### Matching Users to Contextually Relevant Volunteers.

Meta-reviews of existing crisis line services have revealed a lack of
effectiveness, especially of distal outcomes, such as reduction in symptoms or
feelings of distress in callers in a followup period after the call [79].
Empirical studies of helplines have investigated the reasons driving this
phenomenon, finding that they stem from dissatisfied callers and responders
inability to attend to callers’ diverse needs [80]. LLMs could strengthen
infrastructures like 988 by helping to effectively route people in distress to
helpline volunteers based on an assessment of their needs. Through their
proficiency in natural language understanding, LLMs could be utilized to
analyze the language of distress to match people to the types of context-
specific support and specialized volunteers that they may need. This rapid
assessment can help ensure that individuals in immediate need receive the
contextually relevant forms of support they need promptly. In turn, this could
reduce the burden on crisis responders by prioritizing high-risk cases. These
possibilities have already been demonstrated in prior research – Althoff et
al. [81] used data from an SMS texting-based counseling service where people
in crisis engaged in therapeutic conversations with counselors, to build
computational approaches that described which types of language of volunteers
elicited better conversational outcomes.

#### Culturally-Sensitive Vetted Guidance.

LLM technologies can be harnessed to bolster the expertise of crisis
responders and volunteers through real-time linguistic framing and support.
This potential is supported by recent research that has trained language
models towards semantic, issue-based, and lexical reframing of opinions,
arguments, as well as unhelpful thoughts [82, 83]. Additionally, LLMs could
help by surfacing guidance and recommendations to specific crisis situations
which have been previously vetted (by human experts), to be helpful in
mitigating crisis; Sharma and De Choudhury [84] highlighted this potential
through models that learn from positive support seeking and support
provisioning engagements on online forums. LLMs could help to suggest
appropriate interventions, coping strategies, and de-escalation techniques
based on the information provided by the caller and matching this to similar
crisis intervention scenarios in historical data. This real-time assistance
can be invaluable in calming the situation and ensuring the safety of the
individual in crisis. Further, we discussed in the previous section that LLMs
can be programmed to respond in linguistically diverse ways; in a crisis
scenario, culturally resonating support can have significant impact on the
caller’s mental health outcomes. Prior research has shown how language
barriers can hinder effective crisis intervention [80]; thus by empowering
crisis volunteers with language tailored to the identity and culture of the
caller, LLMs could help promote greater inclusivity on helplines, including
988 as well as those in more resource-constrained settings.

###  4.2 Potential Harms

#### A Lack of Contextual Understanding.

Although AI-powered crisis response has been advocated to be particularly
helpful during rapidly evolving ad well as protracted societal crises like
mass shootings [85] and the COVID-19 pandemic [86] due to their ability to be
deployed quickly and at scale, crisis response is an extremely high-stakes
domain, and thus risk and harms could have debilitating impacts on
stakeholders involved, especially the callers. First, multiple factors
influence what precipitates a crisis as well as what strategies could help
mitigate it [87]. Such factors could exist outside the realm of the training
data used to build the LLMs, often perhaps in messy offline contexts—contexts
in which LLMs may have little to no insight. Further, the “black box” nature
of such AIs make identifying contextual gaps inscrutable [88]. While a human
crisis responder could be well-equipped, trained, or use their awareness of
the situation to probe those unobserved factors behind the crisis, LLMs may
provide inappropriate or hallucinated responses or those without sufficient
empathy, potentially leading to an ineffective, harmful, or non-consensual
crisis response that perhaps even worsens the caller’s emotional state. Even
with prompt engineering, it can be hard to control what an LLM may say to an
individual in crisis—the harmful outputs produced by the AI-assisted NEDA
chatbot, as introduced in the Introduction, demonstrates how harmful directly
exposing crisis response service users to LLMs may be.

#### The Complexities of Data Use and Consent.

Ultimately, an LLM is only as good as its training data [29]. Scholars have
repeatedly discussed how by learning from large datasets on the internet, LLMs
could “overrepresent hegemonic viewpoints and encode biases” [29], creating
ethically contentious outcomes potentially extending or even exacerbating
inequities in care [89]. However, beyond diversity, the scale and scope of the
training data also matters, especially in an application domain like crisis
response. The successful use of LLMs in crisis responses hinges on being able
to train them on copious amounts of data spanning caller-volunteer
conversations. Normally, these conversations are seldom recorded or
transcribed beyond service optimization purposes, both to protect
confidentiality of the data, as well as to ensure callers find the crisis
resource to allow more disinhibited exchange with the volunteer [90]. It is
known that knowledge of being tracked or monitored could create a “Hawthorne
effect” [91] leading to people being less truthful of their thoughts and
feelings, and perhaps feeling silenced and fearful of the consequences of
surveillance. Since privacy is often “contextual” [92], in a crisis setting,
callers might be concerned about how their data is collected, stored, and
perhaps most importantly, who does what with this data. An emerging crisis may
also present challenges to a caller’s capacity to recognize these potential
harms and to make the most rational decision for themselves. The best example
of this might be the Crisis Text Line scandal from 2022, where the efforts of
the organization to collect and share conversational data with a for-profit
spinoff without user consent alarmed many [93]. Thus efforts to collect
conversational data going forward, to train LLMs, may undermine the goals of
adequate assessment of a caller’s experience and deploying the most
appropriate intervention. Sourcing training data without adequate informed
consent or participatory involvement of the data producers (e.g., the people
in distress seeking help) may further complicate their use in LLMs, by
reducing their agency in controlling “what data is captured, how it is used,
or who it benefits” [94] and by rendering their data labor invisible [95].

##  5 For Institutions: Clinical Decision Support

In his article, Tom Insel argued that LLMs can provide clinicians with
comprehensive and up-to-date information, aiding in the decision-making
process [35]. We examine the benefits and potential harms of incorporating
LLMs into clinical decision support.

###  5.1 Potential Benefits

#### Unlocking Vast Medical Knowledge.

One of the most significant advantages of using LLMs in clinical decision
support is the ability to access a wealth of information. These models can
learn from vast amounts of medical literature, offering clinicians insights on
various conditions, treatments, and potential side effects, including that is
latest in the medical field. Side effects of psychiatric medications in
particular are often very nuanced and demonstrate patient heterogeneity in
effects [96, 97]. LLMs can not only surface how similar patients have
responded to specific treatments but also can help inform health professionals
about previously unknown potential side effects by learning from complex drug
interactions spanning thousands of clinical trials [98] and online discussions
around interactions [99]. This knowledge can assist healthcare professionals
in making well-informed decisions. The knowledge ingested by LLMs can also be
utilized toward predictive analytic approaches, in order to augment decision
support about patient outcomes, hospital readmission risk, and disease
progression—all of which have been shown to be outcome predictable using
machine learning techniques [100, 101, 102]. LLMs could both improve the
precision of these predictions and aid in proactive patient management and
resource allocation.

#### Providing Individually-Tailored Recommendations.

LLMs can provide tailored recommendations based on the patient’s unique
circumstances that is gleaned from their historical electronic health records,
clinical notes, or hospital discharge summaries, which together can
significantly impact patient care and improve patient outcomes [103]. LLMs can
easily and quickly ingest diverse types of conventional health (EHR) and
health-adjacent data (e.g., smartphone or wearable use, social media
activities) of patients to develop such personalized models [104], and utilize
it for differential diagnosis [105]. When it comes to personalized treatment,
differential diagnosis is perhaps one of the biggest strengths offered by LLMs
[105]. With this knowledge, clinicians may be empowered to reduce the risk of
misdiagnosis [106]; misdiagnosis hurts the efficacy of therapeutic and
pharmacologic treatments down the road, and can enable individuals to function
better in their personal and professional lives, maintain relationships, and
achieve their life goals. LLMs can importantly democratize the medical
knowledge encoded in interactions amongst health professionals by providing
information not only to clinicians but also to patients and their families
[107]. Informed patients can engage in shared decision-making with providers,
fostering a collaborative approach to healthcare and improving health literacy
[108].

###  5.2 Potential Harms

#### Perpetuating Misinformation and Contextually Uninformed Decisions.

Relying solely on LLMs for clinical decision support without verification from
human experts can lead to the dissemination of misinformation, potentially
harming patients’ health and well-being [109]. Jin and Chandra et al [110]
recently showed that while GPT-4 like LLMs are largely adept at providing
accurate responses to a variety of health queries, for some types of queries
they produce incorrect information. In fact, Zhou et al. showed that GPT
models (a type of LLM built by Open AI) could be prompt engineered relatively
easily to reproduce medically incorrect information [28]. Due to the complex
and sensitive nature of mental health issues, clinical decision-making demands
nuanced, context-specific understanding and personalized care. LLMs, while
powerful, lack the ability to grasp the intricacies of an individual’s mental
state and history, especially factors ans aspects that may not be apparent in
its training data such as from EHRs [111]. Given the lack of “objective”
medical measures of mental illness, clinicians utilize a variety of collateral
information in their decision-making [112], for instance, through interactions
with the patients’ family members or relying on non-clinical insights. LLMs
are likely to miss opportunities to learn from such collateral information
that tend to be heavily individual-specific and unique. By relying on specific
types of biased training data stemming from the lived experience of specific
(majority) populations, LLMs might overlook the subtleties in language related
to mental health, such as expressions tied to traumatic experiences or coping
mechanisms, which shape a person’s own conceptualization of their mental
health [113, 114]. As Harrigian et al [115] noted, when these nuances are not
considered during the training of predictive models (here, LLMs), there is a
risk of these signals generating numerous false alarms in decision-making when
applied to different populations. This may be exacerbated by temporal
artifacts as also noted by Harrigian et al [115]. That is, when there are
group-level differences in temporal alignment of the data between model
training and deployment, it can exert an impact on predictive performance of
LLMs dedicated for psychiatric decision-making.

Moreover, existing commercial LLMs have been demonstrated to not generalize
well to non-English health contexts [110], producing to poor quality, non-
comprehensive, and hallucinated information [116], thus disadvantaging non-
English speaking patients. In psychiatry, where accurate diagnosis and timely,
culturally-sensitive treatment is paramount for success of care paradigms
[10], relying solely on automated systems can be particularly perilous, that
can lead to poor response to treatment or prolonged duration of untreated
mental illness.

#### Suggesting Clinically Unverified or Incorrect Treatments.

The above problems can be exacerbated when considering individuals with
serious mental illnesses, such as schizophrenia or bipolar disorder. These
conditions often require highly individualized care and diagnosis, as symptoms
can manifest differently from person to person. Misinformation from an LLM
could lead to inappropriate treatment plans, exacerbating the suffering of
already vulnerable individuals. LLMs learn from web and social media data,
which is often considered a strength, but such data (e.g., Reddit health
conversations) has also been shown to include scientifically unsupported,
clinically unverified, sometimes dangerous treatments [117, 118]. Uncritical
use of such data for model training may result in such medically unverified
strategies to trickle into an LLM’s decision-making [119]. Furthermore,
researchers have shown that in supervised classification tasks, LLMs often
fail to outperform existing fine-tuned traditional machine learning models
[120]. In resource-scare settings where access to mental health professionals
and services is limited [47], these systems may be one of the few available
resources for support [119], making false positives or false negatives in
clinical decisions make the potential for harm even more significant. False
negatives in predicting adverse mental health events may leave marginalized
individuals without the support they need. False positives, on the other hand,
may exacerbate stigma, perpetuate bias, harassment and marginalization, and
importantly, diminish trust in care systems [121].

#### Ethical Issues in Automated Decision-Making.

The use of LLMs for clinical decision-making in mental health also raises
ethical and legal questions, particularly in matters of liability [122, 123].
If a clinical decision goes awry based on recommendations from an LLM, who
bears responsibility? When clinicians rely on LLMs for diagnosis, treatment,
or advice without proper verification, they could be held accountable for any
resulting harm to patients. Legal questions may emerge regarding their duty of
care, professional negligence, and the informed use of technology [124].
Unlike traditional human healthcare providers, LLMs lack the capacity for
judgment and accountability [29, 125], which makes determining liability in
cases of misinformation or harm resulting from improper decisions challenging.
Liability in LLM-based clinical decision-making in mental health—“a field
where classifications of diseases as well as definitions of what is and what
is not a disease are in a state of constant flux” [126] – may further be
complicated by the often unpredictable and context-dependent nature of mental
disorders. Administrative models have been demonstrated to be accurate in
predicting suicidal behaviors at only 50% rate [127], with scholars often
considering this task “unpredictable” [128] or suicide events to be “random”
[129]. Even if LLMs promise better performance in such prediction tasks, as
has been the case with other types of AI [130], liability issues complicate
realizing their practical clinical value.

##  6 For Society: Telehealth 2.0

Finally, Insel envisions generative AI to revolutionize telehealth services
[35] through what he calls “Telehealth 2.0”. LLMs in telehealth may offer a
variety of opportunities and challenges.

###  6.1 Potential Benefits

#### Enhancing Efficiency of Care-Delivery.

Perhaps the most promising opportunity pertains to the efficiency and
automation provided by LLMs in the context of telehealth. One of the
significant benefits of LLMs in mental health can be the automation of various
tasks. Since LLMs can be used to transcribe and summarize large volumes of
text data, such as stemming from therapy sessions, it can help to reduce the
administrative burden on clinicians and healthcare workers [131]. This, in
turn, could allow clinicians to focus more on providing quality patient care.
Automation may also lead to more consistent and thorough documentation of
patient interactions, which can be invaluable in maintaining continuity of
care [132]. In fact, by identifying patterns, trends, and insights in patient-
clinician interactions, the chances of patient concerns being overlooked can
be reduced – an issue widely recognized in traditional healthcare models, and
particularly among racial and gender minorities [133, 134, 135, 136]. It can
instill confidence in the patient that their voices and concerns are more
likely to be heard.

#### Enriching Counselor/Provider Training with Cultural Competence.

Next, LLMs can play a valuable role in counselor/provider training in the
context of telehealth. LLMs can be employed to curate and compile extensive
resources, including textbooks, research papers, case studies, and guidelines
relevant to counseling. These can serve as a comprehensive knowledge base for
trainees [137]. LLMs can also generate realistic case scenarios, reflecting
various mental health issues, patient backgrounds, and cultural contexts.
Trainees can engage with these scenarios to practice their counseling skills
[138]. Notably, LLMs can analyze and provide feedback on the trainee’s
counseling sessions. As shown in Sharma et al’s work [139, 55], by processing
the dialogue and the dynamics of the conversation, AI systems can offer
insights on communication effectiveness, active listening, and therapeutic
rapport. This feedback can be invaluable for trainees to identify areas for
improvement. Moreover, such a scalable approach can ensure that a broader
group of individuals is equipped to respond effectively to those in need.
Furthermore, as discussed by Pendse et al [10], one of the biggest impediments
to accessing care lies in a poor alignment between a patient’s cultural
understandings of their experience of mental health, and that of the
counsellor/provider’s. LLMs can assist in cultural competency training by
providing trainees with information about various cultures, their values, and
belief systems.

###  6.2 Potential Harms

#### Dehumanizing Mental Health Treatment Paradigms.

Automation by learning from vast datasets has been touted to be a inimitable
strength of LLMs [140]. However, while automation can be efficient, it may
also lead to a depersonalized healthcare experience and by rendering human
labor (of healthcare workers) increasingly obsolescent. Mental health care is
not only about diagnosing and treating the underlying illnesses but also about
providing emotional support, empathy, and comfort to patients [141]. These
human aspects of care in an LLM-informed telehealth model may not be fully
replicated by machines [142]. Moreover, the success of many pharmacological
and therapeutic approaches to mental health treatment hinges upon trust and
respect between patients and providers [143]. A dehumanized healthcare
experience punctuated by AI may erode this trust, as patients might feel that
their care is being delivered by algorithms that lack a comprehensive
understanding of their individual needs. Careless automation of telehealth
paradigms may also lead to disinvestment in care work and to a shift away from
patients’ values and preferences – interfering with the goals of a patient-
centered model of care [144]. By removing humans from the care delivery loop,
increased LLM or generative AI based automation may dissuade individuals from
pursuing the healthcare profession altogether or demotivate healthcare workers
from persisting in a profession where their jobs may be at risk of
displacement. The obsolescence of healthcare workers could have economic and
social implications, including job losses, economic dislocation, and potential
labor market disruptions.

#### Threats to Data Privacy.

The most prominent risk perhaps centers around that the analysis of sensitive
patient data by LLMs – an approach that can threaten privacy [125]. This is
especially critical if the utility of LLMs is viewed to be centered around
gleaning meaning from electronic health records, which contain patient health
information (PHI) [145]. It has been noted in recent research that LLMs can
accidentally divulge private information if prompted in specific ways [27].
This is because, in learning from vast data, these models learn relationships
between personally identifiable information or PII (e.g., names, addresses
etc.) and other linguistic elements. These attributes of LLMs can make them
vulnerable to data breaches [146] to unauthorized parties or to bad actors
employing them for contexts beyond the intended use.

#### Demographic and Representational Biases.

Language models have, for some years, been shown to replicate human-like as
well as systemic biases, whether around gender [147, 148, 149] or racial
sterotypes [150, 151, 152]. In a telehealth context, if used to provide
recommendations, such as to clinicians or trainees based on historical
patient-clinician interactions or therapy sessions, LLMs can inadvertently
introduce biases based on any underlying skewness in the training data. If not
carefully monitored, this can lead to unfair or inaccurate representations of
patient-provider interactions. For instance, if the training data skews the
representativeness of one demographic group versus another, LLM-based
suggestions in telehealth could lead to or exacerbate gender, racial, or
ethnic inequities. This concern is not merely theoretical; commercially
accessible LLMs have exhibited racial and gender biases in non-medical
contexts, and these very models have been found to propagate stereotypes
related to race within the field of medicine [23]. Biases may also arise from
LLMs learning stigmatizing representations of language in training data [153],
and propagating those through inappropriate language or portraying mental
health issues in a negative light. It is already known that stigmatizing
language often surfaces in EHR patient notes [154]. If used for knowledge
summarization in a telehealth context, LLMs may thus over-pathologize common
emotions or behaviors, causing undue alarm. Stigma already hinders support-
seeking in mental health, and this is known to prolong the duration of
untreated mental illness [155]. Additional concerns of harm stem from LLMs
creating informational or perspective “echo chambers”, where LLMs
inadvertently perpetuate pre-existing beliefs and biases held by providers or
patients, as they exist in biased training data. For example, it is already
known that screening tools in psychiatry may be biased in ways that tend to
over-diagnose Black patients with schizophrenia [156]. LLMs may pathologize
these biases by learning from such data. An over-reliance on LLMs for
summarizing EHR information or providing training materials could result in
not challenging end users sufficiently to consider a variety of approaches to
providing mental healthcare, rather than an LLM-recommended approach.

##  7 Conclusion, Recommendations, and Future Directions

Long before LLMs were put in the hands of the lay internet user through
ChatGPT, Bender and Koller [157] noted that it will be crucial to acknowledge
the limitations of LLMs and place their strengths within a broader
perspective. This approach can serve to moderate exaggerated expectations,
which can lead both the general public and researchers astray in terms of the
capabilities of these technologies. At the same time, these understandings
have the potential to stimulate fresh research pathways that are not solely
reliant on the utilization of ever bigger language models in every possible
domain of societal interest. Throughout this article, we discussed the many
dimensions of the debate centered around the use of LLMs in digital mental
health applications. We offer some reflections and specific considerations for
future research.

###  7.1 Reflections and Lessons Learned

What is apparent from our above discussion is that, however fine-tuned and
tailored LLMs may be to data stemming from real-world mental health contexts,
LLM-powered chatbots or decision-support tools cannot serve as a replacement
for human psychotherapists or health workers. Neither can a machine alone be a
surrogate to a real person during moments of distress. The significance of the
human therapist may be further underscored by the fact that the appeal of
chatbots in therapy may vary among different age groups. For digital natives,
who are more accustomed to interacting with technology, the appeal of a
machine therapist might be greater that of previous generations; but across
age groups, preferences for a human therapist is likely to remain strong.
Thus, what are the boundaries of the role of LLMs in digital mental health,
for who, and what responsibilities do developers and healthcare providers have
in ensuring their ethical use? A realistic “safety-first” approach might be to
use them as surrogates, rather than as standalone AI therapists.

For such a safety-first approach, it will be essential to strike a balance
between harnessing the potential of LLMs and ensuring that human experts
remain integral to the decision-making process in psychiatry and mental
health, particularly when dealing with the most vulnerable and resource-scarce
populations. An approach, such as a human-in-the-loop or more preferably, an
AI-in-the-loop system, may help to combine the cognitive strengths of
healthcare providers with the analytical capabilities of LLMs. Horvitz’s [158]
conceptualization of “mixed initiative” systems might be particularly
pertinent to mental health applications where, based on the situation, users
can take the lead when they have a specific goal or intention, while also
allowing AI to take the initiative when it can provide value or assist the
user. In critical uses surrounding clinical decision-support or crisis
response, mixed initiatives between the human and the AI (LLM) can enable them
to work together as partners, with the system actively seeking input from the
user and the user having the ability to request information, clarification, or
assistance from the LLM component. To this end, it will be essential to
establish clear protocols for verifying the underlying LLM’s recommendations
of clinical decisions or crisis intervention strategies, ensuring human
experts remain responsible for final decisions and that providers maintain
their ethical and legal obligations to prioritize patient safety and well-
being. In psychotherapy or telehealth, human oversight will ensure that there
are appropriate safeguards in place that prepare for potential harms when the
underlying “stochastic parrots” [29], provide inappropriate, incorrect, or
misinformed outcomes, because of the non-deterministic nature of these
technologies.

In addition, if LLMs are to be utilized in this high stakes domain, thoughtful
investments to create or gather realistic training data will also be needed,
that do not compromise the very mission and values that underlie
psychotherapeutic practices or crisis intervention. For instance, creating
keystone datasets has been advocated to help advance psychological research
using LLMs [159]. Consent, awareness, and literacy regarding how specific data
(e.g., EHRs, crisis helpline call logs, or psychotherapy chat transcripts) is
used in specific LLMs, how, and by whom will be equally important
considerations in ensuring governance of these systems. On that note, for any
type of mental health-relevant data that can be made available for LLM
training, utmost care will need to be employed to prevent unauthorized access
or data breaches, which could lead to severe harm, including identity theft or
emotional distress for individuals in distress or in crisis. Within the
telehealth context, in employing LLMs to summarize patient-provider
interactions or therapy sessions to support clinical care or to train
counsellors, it will be paramount to protect PHI and PII in EHR data, ensuring
not only compliance with data protection regulations, but also to maintain
patient trust their personal information is secure and will not be misused.

Broadly speaking, there is a pressing requirement for increased empirical
research and advocacy efforts aimed at helping mental health service users and
practitioners differentiate the quality, usability, and efficacy of LLMs, as
well as identifying the suitable use cases, scenarios, and target populations
that stand to gain from their application.

###  7.2 Recommendations

Building on these reflections, we suggest technical, ethical, and human-
centered recommendations to ensure that the use of LLMs in mental health
settings carefully balances effectiveness and responsibility.

Developers of LLM-based mental health tools should shoulder the responsibility
in keeping applications safe for end users of these tools. This can be
implemented through self-accountability frameworks, such as, by reporting to
the public, or through outcomes of red teaming efforts. Additional
accountability can come from companies describing what types of training data
was used to build the models, how they were evaluated following training, what
performance metrics were used to assess performance, and the extent to which
performance was tests in various contexts and scenarios. Several efforts in
these lines have been proposed in the algorithmic fairness and accountability
literature, such as Datasheets for describing the capabilities and limits of
datasets used for building AI models [160], Model Cards for bringing
transparency to how complex models work [161], and disclosures of ethics
practices to demonstrate how model builders remain accountable for the
outcomes of AI [162]. Some stakeholders have argued that this can be achieved
if the creators of domain-specific LLMs open source their models [163], others
advocating for a regulatory solution [164]. And yet, some have critiqued both
approaches because of the risk of open LLMs being exploited by malicious
actors or regulation resulting in concentrating power in the hands of a few
that stifles innovation and competition [165]. Nevertheless, what this debate
underscores is a need for continuous monitoring and evaluation mechanisms to
ensure responsible usage and adherence to ethical guidelines within a high
stakes domain like mental health.

There has been some policy movement around developing standards and tools to
help ensure that AI technologies are safe, secure, and trustworthy; e.g., US
President Joe Biden’s 2023 executive order on AI [166] or the Group of Seven
(G7) announcement of a new code of conduct and international guiding
principles on AI [167]. However, scholars have argued that a one-size-fits-all
regulatory model for generative AI will be unsuitable for specific health
applications [168], and an adaptable approach to oversight will be needed that
can evolve with the rapidly and ever-evolving capabilities of this technology.
This adaptive approach will need to go hand in hand with continuous
monitoring, external auditing, and benchmarked evaluation of these systems to
ensure responsible usage and adherence to ethical guidelines.

Autonomous bodies to enforce oversight would also need to be created in the
digital mental health field to establish what types of standards might be
suitable for the four types of uses of LLMs in mental health. For instance,
what standards would ensure safety if LLMs were to be integrated into the 988
system or within psychotherapy contexts? Pharmacological treatments undergo a
clearance process with the FDA and scholars have long advocated for a need to
consider similar regulatory processes for digital mental health as well. After
many years of research, Empatica is one of the few digital mental health
technologies that has received clearance from the FDA for medical use [169,
170]. This conversation may be extended to LLM-based mental health
technologies too, to ensure that a reasonable level of safety is promised in
any application that reaches mental health support seekers.

Furthermore, it will be essential for legal, infrastructure, privacy, and
security teams to review organizational policies and procedures to guarantee
adherence to state and federal laws and regulations, particularly in the
context of personal health information exchange protocols, accountability,
liability, service reimbursement, and clinical workflows. Concurrently, there
is a demand for the creation of educational curricula and methods to instruct
people with lived experience of mental illness, mental health professionals
and caregivers, and health system administrators in fundamentals of generative
AI, its practical application, and its role in enhancing how care is extended.

In the book The Soul of Care [171], Arthur Kleinman describes how “the work of
the doctor has moved away from hands-on practice to high-technology diagnosis
and treatment,” which has distanced doctors from engaging with the human
experiences of their patients. Medical enterprises, digital healthcare
services, and healthcare institutions have initiated the integration of LLMs
into their core operations [168]. In this piece, we emphasize the need to
attend to immediate challenges in the use of this paradigm-shifting
technology, given its rapid clinical roll-out [168]. As Arthur Kleinman
discusses, the importance of emotionally engaged human doctors in the practice
of care can never be replaced, and future LLM-augmented medical technologies
must be cognizant of the importance of human connection in care.

## Acknowledgments and Disclosures

This work was supported by NIMH grants R01MH117172 (PI: De Choudhury) and
P50MH115838 (Co-I: De Choudhury), and a grant from the American Foundation for
Suicide Prevention (PI: De Choudhury). This content is solely the
responsibility of the authors and does not necessarily represent the official
views of the National Institutes of Mental Health or AFSP. We thank the
members of the Social Dynamics and Well-Being Lab at Georgia Tech for helping
to shape early discussions on the topic.

## References

  * [1] OpenAI “Introducing ChatGPT”, 2022 URL: https://openai.com/blog/chatgpt
  * [2] Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever “Improving language understanding by generative pre-training” OpenAI, 2018
  * [3] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano and Geoffrey Irving “Fine-tuning language models from human preferences” In _arXiv preprint arXiv:1909.08593_ , 2019
  * [4] Ivan Moura, Ariel Teles, Davi Viana, Jean Marques, Luciano Coutinho and Francisco Silva “Digital phenotyping of mental health using multimodal sensing of multiple situations of interest: A systematic literature review” In _Journal of Biomedical Informatics_ Elsevier, 2022, pp. 104278
  * [5] Stevie Chancellor and Munmun De Choudhury “Methods in predictive techniques for mental health status on social media: a critical review” In _NPJ digital medicine_ 3.1 Nature Publishing Group UK London, 2020, pp. 43
  * [6] Munmun De Choudhury, Scott Counts and Eric Horvitz “Predicting postpartum changes in emotion and behavior via social media” In _Proceedings of the SIGCHI conference on human factors in computing systems_ , 2013, pp. 3267–3276
  * [7] Munmun De Choudhury, Michael Gamon, Scott Counts and Eric Horvitz “Predicting depression via social media” In _ICWSM_ , 2013
  * [8] Munmun De Choudhury, Emre Kiciman, Mark Dredze, Glen Coppersmith and Mrinal Kumar “Discovering shifts to suicidal ideation from mental health content in social media” In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_ , 2016, pp. 2098–2110 ACM
  * [9] Daejin Choi, Steven A Sumner, Kristin M Holland, John Draper, Sean Murphy, Daniel A Bowen, Marissa Zwald, Jing Wang, Royal Law and Jordan Taylor “Development of a machine learning model using multiple, heterogeneous data sources to estimate weekly US suicide fatalities” In _JAMA network open_ 3.12 American Medical Association, 2020, pp. e2030932–e2030932
  * [10] Sachin R Pendse, Daniel Nkemelu, Nicola J Bidwell, Sushrut Jadhav, Soumitra Pathare, Munmun De Choudhury and Neha Kumar “From treatment to healing: envisioning a decolonial digital mental health” In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ , 2022, pp. 1–23
  * [11] Jonah Bossewitch, Lydia XZ Brown, Piers M Gooding, Leah Harris, James Horton, Simon Katterl, Keris Myrick, Kelechi Ubozoh and Alberto Vasquez Encalada “Digital Futures in Mind: Reflecting on Technological Experiments in Mental Health & Crisis Support” In _Available at SSRN 4215994_ , 2022
  * [12] Stevie Chancellor, Eric PS Baumer and Munmun De Choudhury “Who is the ”Human” in Human-Centered Machine Learning: The Case of Predicting Mental Health from Social Media” In _PACM HCI_ , 2019
  * [13] Anja Thieme, Danielle Belgrave and Gavin Doherty “Machine learning in mental health: A systematic review of the HCI literature to support the development of effective and implementable ML systems” In _ACM Transactions on Computer-Human Interaction (TOCHI)_ 27.5 ACM New York, NY, USA, 2020, pp. 1–53
  * [14] Sindhu Kiranmai Ernala, Asra F Rizvi, Michael L Birnbaum, John M Kane and Munmun De Choudhury “Linguistic markers indicating therapeutic outcomes of social media disclosures of schizophrenia” In _Proceedings of the ACM on Human-Computer Interaction_ 1.CSCW ACM New York, NY, USA, 2017, pp. 1–27
  * [15] Nikolaos Koutsouleris, Tobias U Hauser, Vasilisa Skvortsova and Munmun De Choudhury “From promise to practice: towards the realisation of AI-informed mental health care” In _The Lancet Digital Health_ 4.11 Elsevier, 2022, pp. e829–e840
  * [16] Tobias U Hauser, Vasilisa Skvortsova, Munmun De Choudhury and Nikolaos Koutsouleris “The promise of a model-based psychiatry: building computational models of mental ill health” In _The Lancet Digital Health_ 4.11 Elsevier, 2022, pp. e816–e828
  * [17] Sara Reardon “AI Chatbots Could Help Provide Therapy, but Caution Is Needed” Accessed: 2023-11-03 In _Scientific American_ , 2023 URL: https://www.scientificamerican.com/article/ai-chatbots-could-help-provide-therapy-but-caution-is-needed/
  * [18] “Replika - Virtual AI Friend” [Internet]. App Store. [cited 2023 Mar 14], https://apps.apple.com/us/app/replika-virtual-ai-friend/id1158555867, 2023
  * [19] R. Crasto, L. Dias, D. Miranda and D. Kayande “CareBot: A Mental Health ChatBot” [Internet]. [cited 2023 Mar 14] In _2021 2nd International Conference for Emerging Technology (INCET)_ Belagavi, India: IEEE, https://ieeexplore.ieee.org/document/9456326/, 2021, pp. 1–5
  * [20] K. O’Leary “Human–AI collaboration boosts mental health support” In _Nat Med_ , 2023 URL: https://www.nature.com/articles/d41591-023-00022-w
  * [21] John W Ayers, Zechariah Zhu, Adam Poliak, Eric C Leas, Mark Dredze, Michael Hogarth and Davey M Smith “Evaluating Artificial Intelligence Responses to Public Health Questions” In _JAMA Network Open_ 6.6 American Medical Association, 2023, pp. e2317517–e2317517
  * [22] Travis Zack, Eric Lehman, Mirac Suzgun, Jorge A Rodriguez, Leo Anthony Celi, Judy Gichoya, Dan Jurafsky, Peter Szolovits, David W Bates and Raja-Elie E Abdulnour “Coding Inequity: Assessing GPT-4’s Potential for Perpetuating Racial and Gender Biases in Healthcare” In _medRxiv_ Cold Spring Harbor Laboratory Press, 2023, pp. 2023–07
  * [23] Jesutofunmi A Omiye, Jenna Lester, Simon Spichak, Veronica Rotemberg and Roxana Daneshjou “Beyond the hype: large language models propagate race-based medicine” In _medRxiv_ Cold Spring Harbor Laboratory Press, 2023, pp. 2023–07
  * [24] Julie Jargon “How a Chatbot Went Rogue” In _Wall Street Journal_ , 2023 URL: https://www.wsj.com/articles/how-a-chatbot-went-rogue-431ff9f9
  * [25] Chloe Xiang “Eating Disorder Helpline Fires Staff, Transitions to Chatbot After Unionization” Vice, 2023 URL: https://www.vice.com/en/article/n7ezkm/eating-disorder-helpline-fires-staff-transitions-to-chatbot-after-unionization
  * [26] Chloe Xiang “’He Would Still Be Here’: Man Dies by Suicide After Talking with AI Chatbot, Widow Says” Vice, 2023 URL: https://www.vice.com/en/article/pkadgm/man-dies-by-suicide-after-talking-with-ai-chatbot-widow-says
  * [27] Yang Chen, Ethan Mendes, Sauvik Das, Wei Xu and Alan Ritter “Can Language Models be Instructed to Protect Personal Information?” In _arXiv preprint arXiv:2310.02224_ , 2023
  * [28] Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G Parker and Munmun De Choudhury “Synthetic lies: Understanding ai-generated misinformation and evaluating algorithmic and human solutions” In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , 2023, pp. 1–20
  * [29] Emily M Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell “On the dangers of stochastic parrots: Can language models be too big?” In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , 2021, pp. 610–623
  * [30] Chris Glorioso “Fake news? ChatGPT has a knack for making up phony anonymous sources” Accessed: 2023-11-03 In _NBC New York_ , https://www.nbcnewyork.com/investigations/fake-news-chatgpt-has-a-knack-for-making-up-phony-anonymous-sources/4120307/, 2023
  * [31] Tiffany. Hsu and Stuart. A. Thompson “Disinformation researchers raise alarms about A.I. Chatbots”, The New York Times, 2023
  * [32] David Klepper “It turns out that ChatGPT is really good at creating online propaganda: ’I think what’s clear is that in the wrong hands there’s going to be a lot of trouble”’, Fortune, 2023
  * [33] Ted Chiang “ChatGPT Is a Blurry JPEG of the Web” In _New Yorker_ , 2023 URL: https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web
  * [34] Andrea Reupert “A socio-ecological framework for mental health and well-being” In _Advances in Mental Health_ 15.2 Taylor & Francis, 2017, pp. 105–107
  * [35] Tom Insel “Generative AI and Mental Health” In _AI Anthology_ , 2023 URL: https://unlocked.microsoft.com/ai-anthology/tom-insel
  * [36] James J Gibson “The theory of affordances” In _Hilldale, USA_ 1.2, 1977, pp. 67–82
  * [37] Donald A Norman “Affordance, conventions, and design” In _interactions_ 6.3 ACM New York, NY, USA, 1999, pp. 38–43
  * [38] Daniel E O’Leary “Massive data language models and conversational artificial intelligence: Emerging issues” In _Intelligent Systems in Accounting, Finance and Management_ 29.3 Wiley Online Library, 2022, pp. 182–198
  * [39] Dhruv Khullar “Can A.I. Treat Mental Illness? — newyorker.com” [Accessed 06-11-2023], https://www.newyorker.com/magazine/2023/03/06/can-ai-treat-mental-illness
  * [40] Joseph Weizenbaum “Computer power and human reason: From judgment to calculation.” WH Freeman & Co, 1976
  * [41] Kenneth Mark Colby, James B Watt and John P Gilbert “A computer method of psychotherapy: Preliminary communication” In _The Journal of Nervous and Mental Disease_ 142.2 LWW, 1966, pp. 148–152
  * [42] Joseph Weizenbaum “Computers as” Therapists”” In _Science_ 198.4315 American Association for the Advancement of Science, 1977, pp. 354–354
  * [43] Aditya Nrusimha Vaidyam, Hannah Wisniewski, John David Halamka, Matcheri S Kashavan and John Blake Torous “Chatbots and conversational agents in mental health: a review of the psychiatric landscape” In _The Canadian Journal of Psychiatry_ 64.7 Sage Publications Sage CA: Los Angeles, CA, 2019, pp. 456–464
  * [44] Zilin Ma, Yiyang Mei and Zhaoyuan Su “Understanding the benefits and challenges of using large language model-based conversational agents for mental well-being support” In _arXiv preprint arXiv:2307.15810_ , 2023
  * [45] Julie Vallée, Martine Shareck, Yan Kestens and Katherine L Frohlich “Everyday geography and service accessibility: the contours of disadvantage in relation to mental health” In _Annals of the American Association of Geographers_ 112.4 Taylor & Francis, 2022, pp. 931–947
  * [46] Kathleen C Thomas, Alan R Ellis, Thomas R Konrad, Charles E Holzer and Joseph P Morrissey “County-level estimates of mental health professional shortage in the United States” In _Psychiatric services_ 60.10 Am Psychiatric Assoc, 2009, pp. 1323–1328
  * [47] Ben Rochford, Sachin Pendse, Neha Kumar and Munmun De Choudhury “Leveraging symptom search data to understand disparities in US mental health care: demographic analysis of search engine trace data” In _JMIR Mental Health_ 10 JMIR Publications Toronto, Canada, 2023, pp. e43253
  * [48] Anastasia Philippa Scrutton “Epistemic injustice and mental illness” In _The Routledge handbook of epistemic injustice_ Routledge, 2017, pp. 347–355
  * [49] Sachin R Pendse, Neha Kumar and Munmun De Choudhury “Marginalization and the Construction of Mental Illness Narratives Online: Foregrounding Institutions in Technology-Mediated Care” In _Proceedings of the ACM on Human-Computer Interaction_ 7.CSCW2 ACM New York, NY, USA, 2023, pp. 1–30
  * [50] Bridget M Kuehn “Clinician shortage exacerbates pandemic-fueled “mental health crisis”” In _JAMA_ 327.22 American Medical Association, 2022, pp. 2179–2181
  * [51] Jessica L Schleider, Michael C Mullarkey, Kathryn R Fox, Mallory L Dobias, Akash Shroff, Erica A Hart and Chantelle A Roulston “A randomized trial of online single-session interventions for adolescent depression during COVID-19” In _Nature Human Behaviour_ 6.2 Nature Publishing Group UK London, 2022, pp. 258–268
  * [52] Jessica Schleider “Little Treatments, Big Effects: How to Build Meaningful Moments that Can Transform Your Mental Health” Robinson, 2023
  * [53] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan and Daniel Shu Wei Ting “Large language models in medicine” In _Nature medicine_ 29.8 Nature Publishing Group US New York, 2023, pp. 1930–1940
  * [54] Shang-Ling Hsu, Raj Sanjay Shah, Prathik Senthil, Zahra Ashktorab, Casey Dugan, Werner Geyer and Diyi Yang “Helping the Helper: Supporting Peer Counselors via AI-Empowered Practice and Feedback” In _arXiv preprint arXiv:2305.08982_ , 2023
  * [55] Ashish Sharma, Inna W Lin, Adam S Miner, David C Atkins and Tim Althoff “Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support” In _Nature Machine Intelligence_ 5.1 Nature Publishing Group UK London, 2023, pp. 46–57
  * [56] Ashish Sharma, Kevin Rushton, Inna Wanyin Lin, David Wadden, Khendra G Lucas, Adam S Miner, Theresa Nguyen and Tim Althoff “Cognitive Reframing of Negative Thoughts through Human-Language Model Interaction” In _arXiv preprint arXiv:2305.02466_ , 2023
  * [57] Caleb Ziems, Minzhi Li, Anthony Zhang and Diyi Yang “Inducing positive perspectives with text reframing” In _arXiv preprint arXiv:2204.02952_ , 2022
  * [58] Andrew C Butler, Jason E Chapman, Evan M Forman and Aaron T Beck “The empirical status of cognitive-behavioral therapy: A review of meta-analyses” In _Clinical psychology review_ 26.1 Elsevier, 2006, pp. 17–31
  * [59] Amitabha Palmer and David Schwan “Beneficent dehumanization: Employing artificial intelligence and carebots to mitigate shame-induced barriers to medical care” In _Bioethics_ 36.2 Wiley Online Library, 2022, pp. 187–193
  * [60] Jesse H Wright “Cognitive-behavior therapy for severe mental illness: An illustrated guide” American Psychiatric Pub, 2009
  * [61] Valerie E Black “De-humanizing Care: An Ethnography of Mental Health Artificial Intelligence”, 2023
  * [62] Victoria Stanhope, Stacey L Barrenger, Mark S Salzer and Stephen C Marcus “Examining the relationship between choice, therapeutic alliance and outcomes in mental health services” In _Journal of Personalized Medicine_ 3.3 MDPI, 2013, pp. 191–202
  * [63] Janice L Krupnick, Stuart M Sotsky, Irene Elkin, Sam Simmens, Janet Moyer, John Watkins and Paul A Pilkonis “The role of the therapeutic alliance in psychotherapy and pharmacotherapy outcome: Findings in the National Institute of Mental Health Treatment of Depression Collaborative Research Program” In _Focus_ 64.2 Am Psychiatric Assoc, 2006, pp. 532–277
  * [64] Gerald A Gladstein “Nonverbal communication and counseling/psychotherapy: A review” In _The Counseling Psychologist_ 4.3 Sage Publications Sage CA: Thousand Oaks, CA, 1974, pp. 34–57
  * [65] Bob Price “Developing patient rapport, trust and therapeutic relationships” In _Nursing Standard_ 31.50 RCN Publishing Company Limited, 2017
  * [66] Leslie S Greenberg and Antonio Pascual-Leone “Emotion in psychotherapy: A practice-friendly research review” In _Journal of clinical psychology_ 62.5 Wiley Online Library, 2006, pp. 611–630
  * [67] Michael J Lambert and Dean E Barley “Research summary on the therapeutic relationship and psychotherapy outcome.” In _Psychotherapy: Theory, research, practice, training_ 38.4 Division of Psychotherapy (29), American Psychological Association, 2001, pp. 357
  * [68] Joanne Kim “Data Brokers and the Sale of Americans’ Mental Health Data: The Exchange of Our Most Sensitive Data and What It Means for Personal Privacy” In _Duke Sanford Cyber Policy Program_ , 2023
  * [69] Federal Trade Commission “FTC to Ban BetterHelp from Revealing Consumers’ Data, Including Sensitive Mental Health Information, to Facebook and Others for Targeted Advertising” In _Press Release, Federal Trade Commission_ , 2023
  * [70] Julian De Freitas, Ahmet Kaan Uğuralp, Zeliha Oğuz-Uğuralp and Stefano Puntoni “Chatbots and Mental Health: Insights into the Safety of Generative AI” In _Journal of Consumer Psychology_ Wiley Online Library, 2022
  * [71] Sachin R Pendse, Amit Sharma, Aditya Vashistha, Munmun De Choudhury and Neha Kumar ““Can I not be suicidal on a Sunday?”: understanding technology-mediated pathways to mental health support” In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , 2021, pp. 1–16
  * [72] Alaa A Abd-Alrazaq, Mohannad Alajlani, Ali Abdallah Alalwan, Bridgette M Bewick, Peter Gardner and Mowafa Househ “An overview of the features of chatbots in mental health: A scoping review” In _International Journal of Medical Informatics_ 132 Elsevier, 2019, pp. 103978
  * [73] Gerhard Andersson and Nickolai Titov “Advantages and limitations of Internet-based interventions for common mental disorders” In _World Psychiatry_ 13.1 Wiley Online Library, 2014, pp. 4–11
  * [74] John C Norcross and Bruce E Wampold “What works for whom: Tailoring psychotherapy to the person” In _Journal of clinical psychology_ 67.2 Wiley Online Library, 2011, pp. 127–132
  * [75] Melissa Suran “How the new 988 lifeline is helping millions in mental health crisis” In _JAMA_ , 2023
  * [76] Samantha Matthews, Jonathan H Cantor, Stephanie Brooks Holliday, Armenda Bialas, Nicole K Eberhart, Joshua Breslau and Ryan K McBain “National preparedness for 988—the new mental health emergency hotline in the United States” In _Preventive medicine reports_ 33 Elsevier, 2023, pp. 102208
  * [77] Sasha Zabelski, Andréa R Kaniuka, Ryan A. Robertson and Robert J Cramer “Crisis lines: current status and recommendations for research and policy” In _Psychiatric services_ 74.5 Am Psychiatric Assoc, 2023, pp. 505–512
  * [78] Joshua August Skorburg and Phoebe Friesen “Mind the Gaps: Ethical and Epistemic Issues in the Digital Mental Health Response to Covid-19” In _Hastings Center Report_ 51.6, 2021, pp. 23
  * [79] Adam S Hoffberg, Kelly A Stearns-Yoder and Lisa A Brenner “The effectiveness of crisis line services: a systematic review” In _Frontiers in public health_ 7 Frontiers Media SA, 2020, pp. 399
  * [80] Sachin R Pendse, Faisal M Lalani, Munmun De Choudhury, Amit Sharma and Neha Kumar “” Like Shock Absorbers”: understanding the human infrastructures of technology-mediated mental health support” In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , 2020, pp. 1–14
  * [81] Tim Althoff, Kevin Clark and Jure Leskovec “Large-scale analysis of counseling conversations: An application of natural language processing to mental health” In _Transactions of the Association for Computational Linguistics_ 4 MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info …, 2016, pp. 463–476
  * [82] Tuhin Chakrabarty, Christopher Hidey and Smaranda Muresan “ENTRUST: Argument reframing with language models and entailment” In _arXiv preprint arXiv:2103.06758_ , 2021
  * [83] Mounica Maddela, Megan Ung, Jing Xu, Andrea Madotto, Heather Foran and Y-Lan Boureau “Training Models to Generate, Recognize, and Reframe Unhelpful Thoughts” In _arXiv preprint arXiv:2307.02768_ , 2023
  * [84] Eva Sharma and Munmun De Choudhury “Mental Health Support and its Relationship to Linguistic Accommodation in Online Communities” In _CHI_ , 2018
  * [85] Yang Cheng and Hua Jiang “AI-Powered mental health chatbots: Examining users’ motivations, active communicative action and engagement after mass-shooting disasters” In _Journal of Contingencies and Crisis Management_ 28.3 Wiley Online Library, 2020, pp. 339–354
  * [86] Jaffar Abbas, Dake Wang, Zhaohui Su and Arash Ziapour “The role of social media in the advent of COVID-19 pandemic: crisis management, mental health challenges and implications” In _Risk management and healthcare policy_ Taylor & Francis, 2021, pp. 1917–1932
  * [87] Suzanne M Murphy, Claire B Irving, Clive E Adams and Muhammad Waqar “Crisis intervention for people with severe mental illnesses” In _Cochrane Database of Systematic Reviews_ John Wiley & Sons, Ltd, 2015
  * [88] Upol Ehsan, Philipp Wintersberger, Q Vera Liao, Elizabeth Anne Watkins, Carina Manger, Hal Daumé III, Andreas Riener and Mark O Riedl “Human-Centered Explainable AI (HCXAI): beyond opening the black-box of AI” In _CHI conference on human factors in computing systems extended abstracts_ , 2022, pp. 1–7
  * [89] Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan “Dissecting racial bias in an algorithm used to manage the health of populations” In _Science_ 366.6464 American Association for the Advancement of Science, 2019, pp. 447–453
  * [90] Robin Turkington, Maurice D Mulvenna, Raymond R Bond, Siobhan O’Neill, Courtney Potts, Cherie Armour, Edel Ennis and Catherine Millman “Why do people call crisis helplines? Identifying taxonomies of presenting reasons and discovering associations between these reasons” In _Health informatics journal_ 26.4 SAGE Publications Sage UK: London, England, 2020, pp. 2597–2613
  * [91] Jim McCambridge, John Witton and Diana R Elbourne “Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects” In _Journal of clinical epidemiology_ 67.3 Elsevier, 2014, pp. 267–277
  * [92] Helen Nissenbaum “Privacy as contextual integrity” In _Wash. L. Rev._ 79 HeinOnline, 2004, pp. 119
  * [93] John Hendel “Crisis Text Line ends data-sharing relationship with for-profit spinoff” Politico, 2022 URL: https://www.politico.com/news/2022/01/31/crisis-text-line-ends-data-sharing-00004001
  * [94] Hanlin Li, Nicholas Vincent, Stevie Chancellor and Brent Hecht “The Dimensions of Data Labor: A Road Map for Researchers, Activists, and Policymakers to Empower Data Producers” In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , 2023, pp. 1151–1161
  * [95] Dong Whi Yoo, Hayoung Woo, Sachin R. Pendse, Nathaniel Lu, Michael L. Birnbaum, Gregory Abowd and Munmun De Choudhury “Missed Opportunities for Human-Centered AI Research: Understanding Stakeholder Collaboration in Mental Health AI Research” In _Proceedings of the ACM on Human-Computer Interaction_ ACM New York, NY, USA, 2024
  * [96] Koustuv Saha, Benjamin Sugar, John Torous, Bruno Abrahao, Emre Kıcıman and Munmun De Choudhury “A social media study on the effects of psychiatric medication use” In _Proceedings of the International AAAI Conference on Web and Social Media_ 13, 2019, pp. 440–451
  * [97] Koustuv Saha, John Torous, Emre Kiciman and Munmun De Choudhury “Understanding side effects of antidepressants: Large-scale longitudinal study on social media data” In _JMIR mental health_ 8.3 JMIR Publications Inc., Toronto, Canada, 2021, pp. e26589
  * [98] Serkan Ayvaz, John Horn, Oktie Hassanzadeh, Qian Zhu, Johann Stan, Nicholas P Tatonetti, Santiago Vilar, Mathias Brochhausen, Matthias Samwald and Majid Rastegar-Mojarad “Toward a complete dataset of drug–drug interaction information from publicly available sources” In _Journal of biomedical informatics_ 55 Elsevier, 2015, pp. 206–217
  * [99] Alexandra Papoutsaki, Samuel So, Georgia Kenderova, Bryan Shapiro and Daniel A Epstein “Understanding Delivery of Collectively Built Protocols in an Online Health Community for Discontinuation of Psychiatric Drugs” In _Proceedings of the ACM on Human-Computer Interaction_ 5.CSCW2 ACM New York, NY, USA, 2021, pp. 1–29
  * [100] John S Lyons, John Stutesman, Janice Neme, John T Vessey, Michael T O’Mahoney and H Joseph Camper “Predicting psychiatric emergency admissions and hospital outcome” In _Medical care_ JSTOR, 1997, pp. 792–800
  * [101] Susel Góngora Alonso, Isabel La Torre-Díez, Sofiane Hamrioui, Miguel López-Coronado, Diego Calvo Barreno, Lola Morón Nozaleda and Manuel Franco “Data mining algorithms and techniques in mental health: a systematic review” In _Journal of medical systems_ 42 Springer, 2018, pp. 1–15
  * [102] Michael Leo Birnbaum, Sindhu Kiranmai Ernala, AF Rizvi, Elizabeth Arenare, A R. Van Meter, M De Choudhury and John M Kane “Detecting relapse in youth with psychotic disorders utilizing patient-generated and patient-contributed digital data from facebook” In _NPJ schizophrenia_ 5.1 Nature Publishing Group UK London, 2019, pp. 17
  * [103] Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B Costa and Mona G Flores “A large language model for electronic health records” In _NPJ Digital Medicine_ 5.1 Nature Publishing Group UK London, 2022, pp. 194
  * [104] Ethan Steinberg, Ken Jung, Jason A Fries, Conor K Corbin, Stephen R Pfohl and Nigam H Shah “Language models are an effective representation learning technique for electronic health record data” In _Journal of biomedical informatics_ 113 Elsevier, 2021, pp. 103637
  * [105] Jonathan Kottlors, Grischa Bratke, Philip Rauen, Christoph Kabbasch, Thorsten Persigehl, Marc Schlamann and Simon Lennartz “Feasibility of differential diagnosis based on imaging patterns using a large language model” In _Radiology_ 308.1 Radiological Society of North America, 2023, pp. e231167
  * [106] Dhir Gala and Amgad N Makaryus “The utility of language models in cardiology: a narrative review of the benefits and concerns of ChatGPT-4” In _International Journal of Environmental Research and Public Health_ 20.15 MDPI, 2023, pp. 6438
  * [107] Jan Clusmann, Fiona R Kolbinger, Hannah Sophie Muti, Zunamys I Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia Löffler, Sophie-Caroline Schwarzkopf, Michaela Unger and Gregory P Veldhuizen “The future landscape of large language models in medicine” In _Communications Medicine_ 3.1 Nature Publishing Group UK London, 2023, pp. 141
  * [108] Fabio Botelho, Jean Marie Tshimula and Dan Poenaru “Leveraging ChatGPT to Democratize and Decolonize Global Surgery: Large Language Models for Small Healthcare Budgets” In _World Journal of Surgery_ Springer, 2023, pp. 1–2
  * [109] Gaurav Verma, Ankur Bhardwaj, Talayeh Aledavood, Munmun De Choudhury and Srijan Kumar “Examining the impact of sharing COVID-19 misinformation online on mental health” In _Scientific Reports_ 12.1 Nature Publishing Group UK London, 2022, pp. 8045
  * [110] Yiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Munmun De Choudhury and Srijan Kumar “Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries”, 2023 arXiv:2310.13132 [cs.CL]
  * [111] Shaily Bhatt, Poonam Goyal, Sandipan Dandapat, Monojit Choudhury and Sunayana Sitaram “On the universality of deep contextual language models” In _arXiv preprint arXiv:2109.07140_ , 2021
  * [112] Megan L Petrik, Melodi Billera, Yuliya Kaplan, Bridget Matarazzo and Hal Wortzel “Balancing patient care and confidentiality: considerations in obtaining collateral information” In _Journal of Psychiatric Practice®_ 21.3 LWW, 2015, pp. 220–224
  * [113] Mark Nichter “Idioms of distress revisited” In _Culture, Medicine, and Psychiatry_ 34 Springer, 2010, pp. 401–416
  * [114] Arthur Kleinman “The illness narratives: Suffering, healing, and the human condition” Basic books, 1988
  * [115] Keith Harrigian, Carlos Aguirre and Mark Dredze “Do models of mental health based on social media data generalize?” In _Findings of the association for computational linguistics: EMNLP 2020_ , 2020, pp. 3774–3788
  * [116] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang and Yulong Chen “Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models” In _arXiv preprint arXiv:2309.01219_ , 2023
  * [117] Stevie Chancellor, George Nitzburg, Andrea Hu, Francesco Zampieri and Munmun De Choudhury “Discovering Alternative Treatments for Opioid Use Recovery Using Social Media” In _Proc. CHI_ , 2019
  * [118] Mai ElSherief, Steven A Sumner, Christopher M Jones, Royal K Law, Akadia Kacha-Ochana, Lyna Shieber, LeShaundra Cordier, Kelly Holton and Munmun De Choudhury “Characterizing and identifying the prevalence of web-based misinformation relating to medication for opioid use disorder: Machine learning approach” In _Journal of medical Internet research_ 23.12 JMIR Publications Toronto, Canada, 2021, pp. e30753
  * [119] Steven Tate, Sajjad Fouladvand, Jonathan H Chen and Chwen-Yuen Angie Chen “The ChatGPT therapist will see you now: Navigating generative artificial intelligence’s potential in addiction medicine research and patient care” In _Addiction_ Wiley Online Library, 2023
  * [120] Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang and Diyi Yang “Can Large Language Models Transform Computational Social Science?” In _arXiv preprint arXiv:2305.03514_ , 2023
  * [121] Luke Balcombe “AI Chatbots in Digital Mental Health” In _Informatics_ 10.4, 2023, pp. 82 MDPI
  * [122] Dong Whi Yoo, Sindhu Kiranmai Ernala, Bahador Saket, Domino Weir, Elizabeth Arenare, Asra F Ali, Anna R Van Meter, Michael L Birnbaum, Gregory D Abowd and Munmun De Choudhury “Clinician perspectives on using computational mental health insights from patients’ social media activities: design and qualitative evaluation of a prototype” In _JMIR Mental Health_ 8.11 JMIR Publications Toronto, Canada, 2021, pp. e25455
  * [123] Mindy Duffourc and Sara Gerke “Generative AI in health care and liability risks for physicians and safety concerns for patients” In _Jama_ , 2023
  * [124] Agustina D Saenz, Zach Harned, Oishi Banerjee, Michael D Abràmoff and Pranav Rajpurkar “Autonomous AI systems in the face of liability, regulations and costs” In _NPJ digital medicine_ 6.1 Nature Publishing Group UK London, 2023, pp. 185
  * [125] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle and Atoosa Kasirzadeh “Ethical and social risks of harm from language models” In _arXiv preprint arXiv:2112.04359_ , 2021
  * [126] Seymour L Halleck and Seymour L Halleck “The Psychiatrist’s Liability for Negligent Diagnosis” In _Law in the Practice of Psychiatry: A Handbook for Clinicians_ Springer, 1980, pp. 65–82
  * [127] Ronald C Kessler, Robert M Bossarte, Alex Luedtke, Alan M Zaslavsky and Jose R Zubizarreta “Suicide prediction models: a critical review of recent research with recommendations for the way forward” In _Molecular psychiatry_ 25.1 Nature Publishing Group UK London, 2020, pp. 168–179
  * [128] Matthew Michael Large “The role of prediction in suicide prevention” In _Dialogues in clinical neuroscience_ Taylor & Francis, 2018
  * [129] CA Soper, P Malo Ocejo and Matthew M Large “On the randomness of suicide: An evolutionary, clinical call to transcend suicide risk assessment” In _Evolutionary psychiatry: evolutionary perspectives on mental health_ Cambridge University PressRoyal College of Psychiatrists, 2022, pp. 134–152
  * [130] Glen Coppersmith, Ryan Leary, Patrick Crutchley and Alex Fine “Natural language processing of social media as screening for suicide risk” In _Biomedical informatics insights_ 10 SAGE Publications Sage UK: London, England, 2018, pp. 1178222618792860
  * [131] Emre Sezgin “Artificial intelligence in healthcare: Complementing, not replacing, doctors and healthcare providers” In _Digital Health_ 9 SAGE Publications Sage UK: London, England, 2023, pp. 20552076231186520
  * [132] Mustafa Khanbhai, Leigh Warren, Joshua Symons, Kelsey Flott, Stephanie Harrison-White, Dave Manton, Ara Darzi and Erik Mayer “Using natural language processing to understand, facilitate and maintain continuity in patient experience across transitions of care” In _International journal of medical informatics_ 157 Elsevier, 2022, pp. 104642
  * [133] Kelly M Hoffman, Sophie Trawalter, Jordan R Axt and M Norman Oliver “Racial bias in pain assessment and treatment recommendations, and false beliefs about biological differences between blacks and whites” In _Proceedings of the National Academy of Sciences_ 113.16 National Acad Sciences, 2016, pp. 4296–4301
  * [134] Esther H Chen, Frances S Shofer, Anthony J Dean, Judd E Hollander, William G Baxt, Jennifer L Robey, Keara L Sease and Angela M Mills “Gender disparity in analgesic treatment of emergency department patients with acute abdominal pain” In _Academic Emergency Medicine_ 15.5 Wiley Online Library, 2008, pp. 414–418
  * [135] Olga Bougie, Ma I Yap, Lindsey Sikora, Teresa Flaxman and Sukhbir Singh “Influence of race/ethnicity on prevalence and presentation of endometriosis: a systematic review and meta-analysis” In _BJOG: An International Journal of Obstetrics & Gynaecology_ 126.9 Wiley Online Library, 2019, pp. 1104–1115
  * [136] Paulyne Lee, Maxine Le Saux, Rebecca Siegel, Monika Goyal, Chen Chen, Yan Ma and Andrew C Meltzer “Racial and ethnic disparities in the management of acute pain in US emergency departments: meta-analysis and systematic review” In _The American journal of emergency medicine_ 37.9 Elsevier, 2019, pp. 1770–1777
  * [137] Akshay Ravi, Aaron Neinstein and Sara G Murray “Large Language Models and Medical Education: Preparing for a Rapid Transformation in How Trainees Will Learn to Be Doctors” In _ATS Scholar_ American Thoracic Society, 2023, pp. ats–scholar
  * [138] Conrad W Safranek, Anne Elizabeth Sidamon-Eristoff, Aidan Gilson and David Chartash “The role of large language models in medical education: applications and implications” In _JMIR Medical Education_ 9 JMIR Publications Toronto, Canada, 2023, pp. e50945
  * [139] Ashish Sharma, Inna W Lin, Adam S Miner, David C Atkins and Tim Althoff “Towards facilitating empathic conversations in online mental health support: A reinforcement learning approach” In _Proceedings of the Web Conference 2021_ , 2021, pp. 194–205
  * [140] Jianning Li, Amin Dada, Jens Kleesiek and Jan Egger “ChatGPT in Healthcare: A Taxonomy and Systematic Review” In _medRxiv_ Cold Spring Harbor Laboratory Press, 2023, pp. 2023–03
  * [141] Georgina Gateshill, Kate Kucharska-Pietura and John Wattis “Attitudes towards mental disorders and emotional empathy in mental health and other healthcare professionals” In _The Psychiatrist_ 35.3 Cambridge University Press, 2011, pp. 101–105
  * [142] Carlos Montemayor, Jodi Halpern and Abrol Fairweather “In principle obstacles for empathic AI: why we can’t replace human empathy in healthcare” In _AI & society_ 37.4 Springer, 2022, pp. 1353–1359
  * [143] Patrick Brown, Michael Calnan, Amanda Scrivener and George Szmukler “Trust in mental health services: A neglected concept” In _Journal of Mental Health_ 18.5 Taylor & Francis, 2009, pp. 449–458
  * [144] Timothy A Carey Prof “Beyond patient-centered care: enhancing the patient experience in mental health services through patient-perspective care” In _Patient Experience Journal_ 3.2, 2016, pp. 46–49
  * [145] Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott Fleming, Michael A Pfeffer, Jason Fries and Nigam H Shah “The shaky foundations of large language models and foundation models for electronic health records” In _npj Digital Medicine_ 6.1 Nature Publishing Group UK London, 2023, pp. 135
  * [146] Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann and Eyke Hüllermeier “ChatGPT for good? On opportunities and challenges of large language models for education” In _Learning and individual differences_ 103 Elsevier, 2023, pp. 102274
  * [147] Marion Bartl, Malvina Nissim and Albert Gatt “Unmasking Contextual Stereotypes: Measuring and Mitigating BERT’s Gender Bias” In _Proceedings of the Second Workshop on Gender Bias in Natural Language Processing_ Barcelona, Spain (Online): Association for Computational Linguistics, 2020, pp. 1–16 URL: https://aclanthology.org/2020.gebnlp-1.1
  * [148] Rishabh Bhardwaj, Navonil Majumder and Soujanya Poria “Investigating gender bias in bert” In _Cognitive Computation_ 13.4 Springer, 2021, pp. 1008–1018
  * [149] Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black and Yulia Tsvetkov “Measuring Bias in Contextualized Word Representations” In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ Florence, Italy: Association for Computational Linguistics, 2019, pp. 166–172 DOI: 10.18653/v1/W19-3823
  * [150] Yi Chern Tan and L Elisa Celis “Assessing social and intersectional biases in contextualized word representations” In _Advances in neural information processing systems_ 32, 2019
  * [151] Aylin Caliskan, Joanna J Bryson and Arvind Narayanan “Semantics derived automatically from language corpora contain human-like biases” In _Science_ 356.6334 American Association for the Advancement of Science, 2017, pp. 183–186
  * [152] Svetlana Kiritchenko and Saif Mohammad “Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems” In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_ New Orleans, Louisiana: Association for Computational Linguistics, 2018, pp. 43–53 DOI: 10.18653/v1/S18-2005
  * [153] Steven Tate “Generative Artificial Intelligence Tools in Medicine Will Amplify Stigmatizing Language” In _Journal of Addiction Medicine_ LWW, 2023, pp. 10–1097
  * [154] Scott G Weiner, Ying-Chih Lo, Aleta D Carroll, Li Zhou, Ashley Ngo, David B Hathaway, Claudia P Rodriguez and Sarah E Wakeman “The incidence and disparities in use of stigmatizing language in clinical notes for patients with substance use disorder” In _Journal of addiction medicine_ LWW, 2023, pp. 10–1097
  * [155] Kim T Mueser, Nicole R DeTore, M Alexandra Kredlow, Michelle L Bourgeois, David L Penn and Kathryn Hintz “Clinical and demographic correlates of stigma in first-episode psychosis: The impact of duration of untreated psychosis” In _Acta Psychiatrica Scandinavica_ 141.2 Wiley Online Library, 2020, pp. 157–166
  * [156] Arnold Barnes “Race and schizophrenia diagnoses in four types of hospitals” In _Journal of Black Studies_ 44.6 Sage Publications Sage CA: Los Angeles, CA, 2013, pp. 665–681
  * [157] Emily M Bender and Alexander Koller “Climbing towards NLU: On meaning, form, and understanding in the age of data” In _Proceedings of the 58th annual meeting of the association for computational linguistics_ , 2020, pp. 5185–5198
  * [158] Eric Horvitz “Principles of mixed-initiative user interfaces” In _Proceedings of the SIGCHI conference on Human Factors in Computing Systems_ , 1999, pp. 159–166
  * [159] Dorottya Demszky, Diyi Yang, David S Yeager, Christopher J Bryan, Margarett Clapper, Susannah Chandhok, Johannes C Eichstaedt, Cameron Hecht, Jeremy Jamieson and Meghann Johnson “Using large language models in psychology” In _Nature Reviews Psychology_ Nature Publishing Group US New York, 2023, pp. 1–14
  * [160] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii and Kate Crawford “Datasheets for datasets” In _Communications of the ACM_ 64.12 ACM New York, NY, USA, 2021, pp. 86–92
  * [161] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji and Timnit Gebru “Model cards for model reporting” In _Proceedings of the conference on fairness, accountability, and transparency_ , 2019, pp. 220–229
  * [162] Leah Hope Ajmani, Stevie Chancellor, Bijal Mehta, Casey Fiesler, Michael Zimmer and Munmun De Choudhury “A Systematic Review of Ethics Disclosures in Predictive Mental Health Research” In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , 2023, pp. 1311–1323
  * [163] Sara Guaglione “The case for and against open-source large language models for use in newsrooms — digiday.com” [Accessed 05-11-2023], https://digiday.com/media/the-case-for-and-against-open-source-large-language-models-for-use-in-newsrooms/
  * [164] Stefan Harrer “Council Post: From Boring And Safe To Exciting And Dangerous: Why Large Language Models Need To Be Regulated — forbes.com” [Accessed 05-11-2023], https://www.forbes.com/sites/forbestechcouncil/2023/03/22/from-boring-and-safe-to-exciting-and-dangerous-why-large-language-models-need-to-be-regulated/
  * [165] “Joint Statement on AI Safety and Openness — open.mozilla.org” [Accessed 05-11-2023], https://open.mozilla.org/letter/
  * [166] Joe Biden “Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence” Accessed: 2023-11-03, The White House, 2023 URL: https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/
  * [167] “Press corner — ec.europa.eu” [Accessed 06-11-2023], https://ec.europa.eu/commission/presscorner/detail/en/ip_23_5379
  * [168] Bertalan Meskó and Eric J Topol “The imperative for regulatory oversight of large language models (or generative AI) in healthcare” In _npj Digital Medicine_ 6.1 Nature Publishing Group UK London, 2023, pp. 120
  * [169] Empatica “Empatica’s Platform Receives New FDA Clearance for Cardiac Digital Biomarkers”, 2023 URL: https://www.prnewswire.com/news-releases/empaticas-platform-receives-new-fda-clearance-for-cardiac-digital-biomarkers-301975974.html
  * [170] Empatica “Empatica Receives New FDA Clearance for Its Health Monitoring Platform and Announces Series B Financing”, 2022 URL: https://www.prnewswire.com/news-releases/empatica-receives-new-fda-clearance-for-its-health-monitoring-platform-and-announces-series-b-financing-301685344.html
  * [171] Arthur Kleinman “The soul of care: the moral education of a husband and a doctor” Penguin, 2020