# Read, Diagnose and Chat: Towards Explainable and Interactive LLMs-Augmented
Depression Detection in Social Media

Wei Qin  Hefei University of Technology1 Thørväld CircleHefeiChina
<EMAIL_ADDRESS>,  Zetong Chen  University of Science and Technology of
China1 Thørväld CircleHefeiChina<EMAIL_ADDRESS>,  Lei Wang  Singapore
Managerment University1 Thørväld CircleSingapreSingapore
<EMAIL_ADDRESS>,  Yunshi Lan  East China Normal University1
Thørväld CircleShanghaiChina<EMAIL_ADDRESS>,  Weijieying Ren
Pennsylvania State University1 Thørväld CirclePennsylvaniaUSA - and  Richang
Hong  Hefei University of Technology1 Thørväld CircleHefeiChina
<EMAIL_ADDRESS>
###### Abstract.

Depression detection based on social media content has received increasing
attention, as it allows for early diagnosis before the user’s psychological
state deteriorates. Although traditional methods of depression detection can
provide a classification of whether the user is depressed or not, they cannot
provide human-like explanations and interactions. In this paper, we propose a
next-generation paradigm for depression detection, namely an interpretable and
interactive depression detection system based on LLMs. The proposed system not
only yields a final diagnosis result, but also offers diagnostic evidence
grounded in established diagnostic criteria. Furthermore, it enables users to
engage in natural language dialogue with the system, facilitating a more
personalized understanding of their mental state based on their social media
content. The interactive dialogue allows for the provision of tailored
recommendations, which users can utilize to enhance their well-being. In
constructing the entire system, we also addressed some non-trivial challenges.
First, we introduced the chain of thoughts technique and professional
depression diagnostic criteria when constructing the prompts, enabling our
system to make decisions based on professional diagnosis criteria and provide
explanations. Secondly, LLMs are incapable of processing excessively long
contextual texts, and the accumulated posts of a single user may amount to
tens of thousands of words. To overcome this limitation, we integrated a tweet
selector that selects the part of posts for diagnosis. The experiments
demonstrate that our depression detection system achieves the best performance
across various settings, including full data setting, few-shot setting, zero-
shot setting, independent-identical-distribution(IID) setting, and out-of-
distribution(OOD) setting. Additionally, case studies reveal the explanation
and interactivity of our system.

\-

##  1\. Introduction

Mental health disorders are a significant and widespread public health
concern, with depression affecting an estimated 300 million people worldwide
(Organization., 2012). Depression is a leading cause of suicide and ranks as
the third leading cause of death for individuals between the ages of 10 and 24
years old. Despite the severity of the issue, depression remains
underdiagnosed and undertreated (Allan et al., 2014; Sheehan, 2004). More than
half of those who suffer from depression do not receive any treatment
(Organization., 2012). Williams et al. (2017) used a depression diagnostic
test instrument to discover that 8% of the population had symptoms and a
diagnosis of depression, while 7.6% had symptoms but had not been diagnosed.
Individuals affected by mental disorders often hesitate to seek professional
help (Edwards et al., 2007). However, as more people turn to social media to
discuss their struggles and seek emotional support, automatic processing of
social media data could be a crucial tool in identifying changes in mental
health status before they lead to more serious health consequences.

Currently, many researchers are focusing on the task of detecting depression
in social media. To this end, various studies, including those conducted by
(Gui et al., 2019; Bucur et al., 2023), have investigated the use of different
network structures such as Convolutional Neural Networks (CNNs), Long Short-
Term Memory (LSTM) networks, and transformers. Additionally, (An et al., 2020)
introduced a topic-enriched auxiliary task that improves the detection ability
of depression symptoms by enabling the model to better comprehend social media
content. Furthermore, (Cheng and Chen, 2022) considered the timing of social
media posts. Notwithstanding their impressive performance, as depicted in
Figure 1(a), existing methods for depression classification suffer from a
crucial limitation: they solely produce binary classification outputs (i.e.,
depressed or non-depressed) without providing any accompanying explanation.
This drawback renders the classification results less persuasive to users, and
provides limited reference information for medical practitioners as well.
Another primary constraint is their pool interactivity and lack of feedback
mechanisms. An interactive environment is necessary to allow users to express
their mental state more deeply and to obtain effective personalized
recommendations. However, little research has been conducted in this domain.

Figure 1. User posts list is the input of the task. The sub-figure (a) shows
that traditional methods only output the final diagnosis result. Our method in
sub-figure (b) could provide a diagnosis result, diagnosis evidence and
interactive dialogue.

Large language models (LLMs), such as GPT-3 (Brown et al., 2020), OPT (Zhang
et al., 2022), and PaLM (Chowdhery et al., 2022), have evolved rapidly and
have demonstrated remarkable performance across various natural language
processing (NLP) tasks. As LLMs continue to expand in size and training data,
they are strong semantic understanding and reasoning abilities (Wei et al.,
2022a). The emergence of LLMs has brought about a paradigm shift in research.
Previously, applying models to downstream tasks typically involved adjusting
model parameters through backpropagation. However, the latest development of
LLMs (Touvron et al., 2023) has enabled both researchers and practitioners to
facilitate learning during the forward process by constructing prompts, namely
In-Context Learning (ICL) Recently, LLMs with ICL paradigm are leveraged to
solve many tasks including movie recommendation (Gao et al., 2023), medical
image classification (Wang et al., 2023) and document information extraction
(He et al., 2023). However, to date, there has been no investigation into the
ability of LLMs to handle the depression detection task.

In this work, we propose a next-generation paradigm for depression detection
in social media, namely an explainable and interactive LLMs-augmented
depression detection system—Chat-Diagnose. As illustrated in Figure 1(b),
Chat-Diagnose system combines established diagnostic criteria with social
media content to assess the risk of depression in users and generate
interpretable diagnostic evidence. Following the diagnostic process, Chat-
Diagnose system communicates the diagnosis results to users through natural
dialogue, along with relevant diagnostic evidence. In this interactive
dialogue, users can describe their mental state more deeply through our guided
questioning based on their social media content, and we can more accurately
assess their mental state and provide appropriate personalized recommendations
based on the content of the conversation.

However, several main challenges arise when applying LLMs to depression
detection in social media. The first challenge is how to force the system to
provide explanations based on professional diagnostic knowledge. To address
this challenge, we first incorporate external professional diagnostic
knowledge (DSM depression diagnosis criteria) as part of the prompt. On top of
this, we introduce the chain of thoughts(CoT) technique and construct
demonstrations, which enable our LLMs-augmented system to make diagnoses based
on professional diagnosis criteria and provide diagnostic evidence. Figure 3
shows the DSM depression diagnosis criteria and the demonstrations. The second
challenge lies in the fact that users may have hundreds or thousands of posts,
and the lengthy context can overwhelm LLMs. Therefore, we build a tweet
selector module to screen each post and input the most related ones to LLMs.

Figure 2. The framework of our Chat-Diagnose system.

Our Chat-Diagnose system can operate in zero-shot, few-shot and full-training
settings. In the zero-shot setting, no examples are provided to our system. In
the few-shot setting, our system inputs only two CoT demonstrations. In the
full-training setting, our method additionally inputs the prediction outputs
of a full-training vanilla depression detection model as answer heuristics to
LLMs. As the currently most available LLMs cannot process images, we utilize
image captioning and OCR to convert image content into textual descriptions.

We conducted extensive experiments on Twitter dataset and Weibo dataset. Our
approach achieved state-of-the-art performance in multiple zero-shot, few-
shot, and full-data settings, as well as independent-identical-distribution
(IID) and out-of-distribution (OOD) test data. In addition, ablation studies
confirmed the effectiveness of each module in our system. Case studies
demonstrated the quality of the explanation and interactivity provided by our
system.

Our contributions are summarized as follows: 1). We developed a new depression
detection system(Chat-Diagnose) around LLM that provides diagnostic evidence
and personalized advice through natural language dialogue. 2). By introducing
CoT technology and a tweets selector, we improve explainability and avoid
LLMs’ limitations in handling excessive posts. 3). Extensive experiments and
case studies demonstrate our system’s superior performance, explainability,
and interactivity in various settings.

##  2\. Related Work

Depression detection in social media. Early research on depression detection
focused on analyzing language cues identified in psychology literature, such
as self-focused language indicated by the frequent use of the pronoun ”I”
(Ríssola et al., 2020; Rude et al., 2004) and dichotomous thinking expressed
through absolute words like ”always” and ”never” (Fekete, 2002). Reece et al.
(Reece and Danforth, 2017) demonstrated that individuals with depression tend
to post images with sadder, less happy, and darker tones compared to healthy
individuals. With the advent of deep learning, various neural network
architectures, such as CNNs (Yates et al., 2017), LSTMs (Skaik and Inkpen,
2020; Trotzek et al., 2018), and transformers (Bucur et al., 2023; Sawhney et
al., 2020), have been explored to detect depression in social media and have
achieved remarkable performance. Gui et al. (Gui et al., 2019) proposed a
cooperative multi-agent reinforcement learning approach using two agents to
select relevant textual and visual information for classification. An et al.
(An et al., 2020) proposed a multimodal topic-enriched auxiliary learning
approach that improves the primary task of depression detection through
auxiliary tasks on visual and textual topic modeling. Bucur et al. (Bucur et
al., 2023) used time-adapted weights (Chiu et al., 2021) or Time-Aware LSTMs
(Cheng and Chen, 2022; Sawhney et al., 2020) to incorporate the time component
of data for mental health problem detection.

Deep learning methods achieve dominating performance but are limited by
explanation and interactivity. High-quality explanations can convince users to
seek professional medical help and increase the likelihood of recovery, and
allow doctors to screen users more effectively. The system’s interactivity
enables users to provide detailed descriptions of their condition and receive
personalized advice. We refer to depression detection systems with high
explanation and interactivity as a next-generation paradigm for depression
detection in social media.

In-Context Learning. Large Language Models (LLMs), including GPT-3 (Brown et
al., 2020), OPT (Zhang et al., 2022), and PaLM (Chowdhery et al., 2022), are
known for their impressive emergent abilities that scale with model and corpus
size (Wei et al., 2022b). These abilities are learned from demonstrations
containing only a few examples in the context, which is known as in-context
learning (Dong et al., 2022). To enable reasoning in LLMs, a few-shot
prompting strategy called Chain-of-Thought (CoT) prompting has been proposed
by Wei et al. (2022c), which involves adding multiple reasoning steps to the
input question. Recent works(Wang et al., 2022a; Suzgun et al., 2022; Shaikh
et al., 2022) have aimed to improve CoT prompting in various aspects, such as
prompt format (Chen et al., 2022), prompt selection (Lu et al., 2022), prompt
ensemble (Wang et al., 2022b), and problem decomposition (Zhou et al., 2023).
Despite being initially developed for NLP tasks, LLMs with in-context learning
have been found to exhibit few-shot or zero-shot abilities for multimodal
problems, such as visual question-answering tasks (Yang et al., 2022a; Cao et
al., 2022; Zeng et al., 2022). Additionally, pre-trained models have shown
promising few-shot performance in vision-and-language tasks, as demonstrated
by Frozen (Tsimpoukelli et al., 2021). However, to the best of our knowledge,
this is the first work to explore the use of LLMs with in-context learning for
depression detection.

##  3\. Method

###  3.1. Problem Setup

Depression detection in social media refers to the diagnosis of whether a user
is depressed or not based on their social media content, which may include
both text and images. Specifically, we represent each user sample as $(x,y)$,
where $x$ is a list of posts from the user’s social media account, including
multiple posts ${x_{t_{1}},x_{t_{2}},...,x_{t_{m}}}$. Each post $x_{t}$
contains both text $x_{t}^{text}$ and picture $x_{t}^{pic}$. $y$ indicates
whether the user has been diagnosed as a depressed individual or not. The
traditional depression detection system, denoted by $\mathcal{F}^{*}$,
utilizes the user’s social media content to determine whether the user
exhibits symptoms of depression:

(1)
$\hat{y}=\mathcal{F}^{*}(x)=\mathcal{F}^{*}({x_{t_{1}},x_{t_{2}},...,x_{t_{m}}})$

where $\hat{y}$ denotes the predicted diagnosis result whether the user is
depressed or not.

In this paper, we propose a novel but more challenging paradigm for the
depression detection system. The system $\mathcal{F}$ is required not only to
provide detection results but also to provide interpretability (diagnostic
evidence) and an interactive dialogue system.

(2)
$\hat{y},\mathcal{E}=\mathcal{F}(x)=\mathcal{F}({x_{t_{1}},x_{t_{2}},...,x_{t_{m}}})$

where $\hat{y}$ and $\mathcal{E}$ denotes the diagnosis results and the
generated diagnosis explanation. Moreover, the system is equipped with an
interactive chat module that proactively engages with users. This module can
be represented as:

(3) $R_{t}=\mathcal{F}(c_{t},h_{t-1},\hat{y},\mathcal{E},x)$

where $c_{t}$ denotes the input provided by the user at round $t$ and $R_{t}$
denotes the response generated by the system at round $t$.
$h_{t-1}={R_{1},c_{2},R_{2},...,R_{t-1},c_{t-1}}$ represents the dialogue
history up to and including the previous t-1 rounds. Here, $h_{0}=None$.

###  3.2. Depression Detection System

Tweet Selector $\mathcal{S}$:  The tweet selector is introduced to filter out
excessive posts from the user, since LLMs cannot process too long text. The
tweet selector $\mathcal{S}$ will select $n$ posts as the input LLMs:

(4)
${x_{t^{{}^{\prime}}_{1}},x_{t^{{}^{\prime}}_{2}},...,x_{t^{{}^{\prime}}_{n}}}=\mathcal{S}({x_{t_{1}},x_{t_{2}},...,x_{t_{m}}})$

where ${x_{t_{1}},x{t_{2}},...,x{t_{m}}}$ represents the collection of all $m$
posts from a particular user, while
${x_{t^{{}^{\prime}}_{1}},x_{t^{{}^{\prime}}_{2}},...,x_{t^{{}^{\prime}}_{n}}}$
denotes the subset of n posts that are selected. We experiment with three
tweet selectors: the random selector, the recent selector and the sentiment
selector. The random selector randomly selects $n$ posts, while the recent
selector selects the $n$ most recent posts. The sentiment selector utilizes a
sentiment analysis model to rank all $m$ posts based on their negativity
scores, and then select the $n$ most negative posts. We employ GPT-3.5 as our
sentiment analysis model.

Image Descriptor $\mathcal{D}$: The present module employs Image Caption and
Optical Character Recognition (OCR) technology to transform images into
textual information, which is a crucial feature as many Language and Learning
Models (LLMs) are only capable of processing text. One common approach to
extract semantics from images and represent it using textual descriptions is
through image captioning (Cao et al., 2022; Yang et al., 2022b). The process
starts with extracting the text from the image using open-source Python
packages, such as EasyOCR2. Subsequently, a pre-trained image captioning model
named ClipCap (Mokady et al., 2021) is applied. This model is particularly
useful for generating high-quality captions for low-resolution web images. The
captions produced by ClipCap typically describe the most prominent objects or
events depicted in the image. The left part of Figure 2 demonstrates the
effect of the Image Descriptor, which utilizes image caption and OCR to
transform the visual information of an image into textual form. In the future,
with LLMs capable of handling images such as GPT-4, we can directly input both
images and text into the model without the need for this intermediary step.

Professional Diagnostic Criteria $\mathcal{C}$:

Figure 3. The abbreviated version of the DSM diagnostic criteria for major
depressive disorder is provided, with reference to the complete version
available in (Ma, 2021).

To facilitate the detection system’s ability to perform diagnostic procedures
based on professional knowledge, rather than just classification, we have
integrated the DSM depression diagnostic criteria as the Professional
Diagnostic Criteria $\mathcal{C}$, as depicted in Figure 3. The Diagnostic and
Statistical Manual of Mental Disorders (DSM) serves as the primary reference
for psychiatric diagnoses in the United States.

Demonstrations $\mathcal{D}emo$: In-context learning usually involves
introducing a few ($k$) demonstrations to enable the model to learn how to
accomplish the given task. Specifically, a demonstration refers to
reformulating a labeled sample into a question-answer pair format in a text-
based representation that can be understood by LLMs. Besides, utilizing few
demonstrations can effectively constrain LLMs, enabling them to output results
in a fixed format (Dong et al., 2022).

Chain-of-Thought $\mathcal{COT}$: The Chain of Thoughts (CoT) technique
involves incorporating reasoning steps into demonstrations ($\mathcal{D}emo$)
to obtain the final results. This technique enables LLMs to detect depression
and provide diagnostic evidence based on the Professional Diagnostic Criteria
$\mathcal{C}$. The CoT demonstration comprises triples, which include input,
output, and reasoning steps. The input includes the question, options, and
posts, while the output indicates the final answer (diagnosis). The reasoning
steps in the chain of thought reveal how to derive the answer from the
question. The CoT demonstration follows a specific organizational format:

  * Question: Analyze someone’s tweets using DSM-5 to determine if they are depressed.

  * Options: [A.depressed B.non-depressed].

  * Posts: [Posts of User].

  * Answer: [A or B].

  * Explanation: [Reasoning steps].

Vanilla Depression Detection Model $\mathcal{F}^{*}$ and Answer Heuristics
$\mathcal{A}$: In the full data setting, to fully exploit all available
training samples, a conventional depression detection model $\mathcal{F}^{*}$
is introduced and trained on the entire training dataset. The resulting
prediction probabilities from this model are utilized as answer heuristics
$\mathcal{A}$ and subsequently fed into the LLM as inputs. This can be
expressed mathematically as:

(5) $\displaystyle{p}(A),{p}(B)$
$\displaystyle=\mathcal{F}^{*}({x_{t_{1}},x_{t_{2}},...,x_{t_{m}}})$ (6)
$\displaystyle\mathcal{A}$ $\displaystyle=\\{{p}(A),{p}(B)\\}$

where ${p}(A)$ and ${p}(B)$ are the predicted probabilities that the test user
is depressed or not depressed by the conventional depression detection model,
respectively, and are considered as the answer heuristics $\mathcal{A}$.
Specifically, the Time2VecTransformer (Bucur et al., 2023) is adopted as the
vanilla depression detection model. However, in the few-shot and zero-shot
settings, the LLM is not provided with any answer heuristics.

Algorithm 1 Algorithm of our depression detection system

1:Step 1: detect the depression

2:Input: $x$, $\mathcal{COT}$-$\mathcal{D}emo$, $\mathcal{C}$, $\mathcal{L}$
$\triangleright$ $\mathcal{L}$ denotes LLMs

3:Output: $\hat{y}$, $\mathcal{E}$

4:procedure $\mathcal{F}$( $x$, $\mathcal{COT}$-$\mathcal{D}emo$,
$\mathcal{C}$)

5: $\mathcal{A}\leftarrow\mathcal{F}^{*}(x)$

6: $x^{D}\leftarrow\mathcal{D}(x)$

7: $x^{DS}\leftarrow\mathcal{S}(x^{D})$

8:
prompt${}_{D}\leftarrow\mathcal{M}_{D}(x^{DS},\mathcal{A},\mathcal{COT}$-$\mathcal{D}emo)$

9: $\hat{y},\mathcal{E}\leftarrow\mathcal{L}($ prompt${}_{D})$

10:

11:Step 2: interactive dialogue

12:Input: $x$, $\hat{y}$, $\mathcal{E}$, $\mathcal{L}$

13:procedure $\mathcal{F}$($x$, $\hat{y}$, $\mathcal{E}$)

14: DialogueShow($\hat{y}$, $\mathcal{E}$)

15: $c\leftarrow None$

16: $h\leftarrow None$

17: while dialogue ends do

18: prompt${}_{T}\leftarrow\mathcal{M}_{T}(c,x^{DS},\hat{y},\mathcal{E},h)$

19: $R\leftarrow\mathcal{L}$(promptT)

20: $h\leftarrow h+R$

21: DialogueShow($R$)

22: $c\leftarrow$ user input in this round

23: $h\leftarrow h+c$

Prompt Manager for Diagnosis $\mathcal{M}_{D}$: The prompt manager has been
designed to convert all clues into a language that LLM can understand.
Specifically, the prompt manager for diagnosis $\mathcal{M}_{D}$ is
responsible for constructing prompts that instruct LLMs on how to diagnose a
user. We conduct the prompt with the question, the $\mathcal{CoT}$
demonstrations, the professional diagnostic criteria $\mathcal{C}$ and the
answer heuristics $\mathcal{A}$:

  * Question: Analyze someone’s tweets using DSM-5 to determine if they are depressed.

  * Options: [A.depressed B.non-depressed].

  * Diagnosis criteria: [DSM diagnosis criteria]

  * Demonstrations: [$\mathcal{COT}$ Demonstrations].

  * Posts: [Posts of User].

  * Answer candidate: [A P(A)]; [B P(B)].

  * Answer:

  * Explanation:

where $\mathcal{COT}$ Demonstrations refers to demonstrations with a chain of
thoughts. And ”Posts” denotes the selected list of tweets obtained from the
tweet selector $\mathcal{S}$ and image descriptor $\mathcal{D}$, as
illustrated in Figure 2. The ”Answer candidate” is constructed using the
answer heuristic $\mathcal{A}$. The confidence scores for answers A and B are
denoted by ”P(A)” and ”P(B)”, respectively. These scores help to focus the
attention of large language models (LLMs) on candidate answers with higher
scores. We consider the answer generated by this prompt as the final diagnosis
result. LLMs then fill the ”Explanation” section with the diagnosis evidence.

This prompt is applicable in the full data setting, where the ”Answer
candidate” is computed using a vanilla depression detection model trained with
full training data. In the few-shot and zero-shot settings, the ”Answer
candidate” is not included. Moreover, the prompt used in the zero-shot setting
does not contain the ”Demonstrations”.

Prompt Manager for Dialogue $\mathcal{M}_{T}$: The prompt manager for dialogue
$\mathcal{M}_{T}$ is responsible for constructing prompts that guide LLMs on
how to respond to users based on their diagnosis results, explanation, social
media content, and dialogue history:

  * Instruction: Chat based on user’s tweets to gather psychological information and provide advice.

  * Posts: [Posts of User].

  * Diagnosis: [Result],[Explanation].

  * Dialogue history: [Dialogue history].

  * Input: [User input].

where [Result],[Explanation] are the generated diagnosis results and evidence
from our system. [User input] denotes the user’s input in the dialogue.

##  4\. Experiment

|  | # user | # tweet | # picture
---|---|---|---|---
WU3D (Weibo) | depressed | 1000 | 39,595 | 15,543
non-depressed | 1000 | 80,167 | 48,802
TMDD (Twitter) | depressed | 1402 | 232,895 | 22,195
non-depressed | 1402 | 879,025 | 64,359
Table 1. Statistics of TMDD and WU3D. |  | F1 | Precision | Recall | Acc
---|---|---|---|---|---
Setting | Model | IID | OOD | IID | OOD | IID | OOD | IID | OOD
Full Data | MTAL | 0.842 | - | 0.842 | - | 0.842 | - | 0.842 | -
GRU + VGG +COMMA | 0.900 | - | 0.900 | - | 0.900 | - | 0.900 | -
MTAN | 0.908 | - | 0.885 | - | 0.931 | - | - | -
SetTransformer | 0.927 | - | 0.921 | - | 0.934 | - | 0.926 | -
Time2VecTransformer | 0.931 | 0.804 | 0.937 | 0.820 | 0.925 | 0.788 | 0.931 | 0.808
Our Chat-Diagnose system(ChatGPT) | 0.936 | 0.894 | 0.973 | 0.925 | 0.903 | 0.865 | 0.939 | 0.897
| Our Chat-Diagnose system(GPT-3) | 0.946 | 0.864 | 0.979 | 0.904 | 0.915 | 0.828 | 0.948 | 0.870
Few Shot | BERT(base) | 0.696 | 0.587 | 0.710 | 0.595 | 0.6825 | 0.58 | 0.703 | 0.593
Our Chat-Diagnose system(ChatGPT) | 0.815 | 0.797 | 0.882 | 0.870 | 0.758 | 0.735 | 0.825 | 0.813
| Our Chat-Diagnose system(GPT-3) | 0.870 | 0.851 | 0.886 | 0.856 | 0.855 | 0.848 | 0.872 | 0.853
Zero Shot | PTDD | 0.587 | 0.563 | 0.593 | 0.569 | 0.58 | 0.556 | 0.591 | 0.568
Our Chat-Diagnose system(ChatGPT) | 0.715 | 0.703 | 0.738 | 0.707 | 0.693 | 0.700 | 0.724 | 0.705
| Our Chat-Diagnose system(GPT-3) | 0.689 | 0.656 | 0.652 | 0.671 | 0.730 | 0.643 | 0.669 | 0.664
Table 2. Experiment results performed under various settings. Full Data indicates that all available training data is used, Few Shot denotes the usage of only two training examples, and Zero Shot corresponds to the evaluation being performed without any training examples. IID and OOD refer to whether the evaluation data belongs to the independent-identical-distribution or out-of-distribution data, respectively. | TMDD | WU3D
---|---|---
| F1 | F1
Our Chat-Diagnose system | 0.946 | -
w/o $\mathcal{A}$ | 0.870 | 0.902
w/o $\mathcal{A,COT}$ | 0.789 | 0.802
w/o $\mathcal{A,COT,C}$ | 0.702 | 0.691
w/o $\mathcal{A,COT,D}emo$ | 0.689 | -
w/o $\mathcal{A},\mathcal{COT},\mathcal{C},\mathcal{D}emo$, | 0.662 | 0.675
Table 3. The impact of various modules on the performance of our Chat-Diagnose
system. Specifically, $\mathcal{A}$ referred to the answer heuristic extracted
from an existing model trained by full data, whereas $COT$ denoted the chain
of thoughts included in the demonstrations. Moreover, $C$ indicated the
professional diagnosis criteria, while $\mathcal{D}emo$ referred to the few
shot demonstrations.

###  4.1. Experiment Setup

Datasets. In this study, we evaluate our method by benchmarking it against two
widely used depression datasets, namely the Twitter multimodal depression
dataset (TMDD) (Gui et al., 2019) and the Weibo user depression detection
dataset (WU3D) (Wang et al., 2020). Both Twitter and Weibo are popular social
media platforms where users share their mental health problems. The TMDD is in
English, while the WU3D is in Chinese. While both datasets include images, the
textual information is more extensive in WU3D, where posts can have up to
5,000 characters compared to the 280 character limit of Twitter. For TMDD,
users in the depression class were identified by detecting mentions of
depression diagnosis, while users in the control group had no such
indications. On the other hand, all depressed user samples in the WU3D were
manually labeled by anonymous data labeling specialists and reviewed by
psychologists and psychiatrists.

The TMDD contains 1,402 users diagnosed with depression and 1,402 control
users (Gui et al., 2019). To maintain consistency with the experimental
settings used in (Bucur et al., 2023), which we could not access in the full
data setting, we conducted experiments using the same settings and performed
five-fold cross-validation. In the few-shot and zero-shot settings, we used
1,000 positive and 1,000 negative samples as evaluation data. Similarly, for
the WU3D, we used 1000 positive and 1000 negative samples in the few-shot and
zero-shot settings. Table 1 provides a summary of the dataset statistics. It
is essential to note that only a small fraction of social media posts contain
both text and image data.

Baselines. In the full data setting, we have compared our system with several
classic methods. Multimodal Topic-Enriched Auxiliary Learning (MTAL) (An et
al., 2020) captures the multimodal topic information through two auxiliary
tasks accompanying the primary task of depression detection in visual and
textual topic modeling. Multimodal Time-Aware Attention Networks (MTAN) (Cheng
and Chen, 2022) is a multimodal model that incorporates T-LSTM to consider the
time intervals between posts. ”GRU + VGG-Net + COMMA” (Gui et al., 2019) uses
a reinforcement learning component to select posts with text and images that
are indicative of depression and classify them with an MLP. SetTransformer
(Bucur et al., 2023), a set-based multimodal transformer, employs zero
positional encoding and random sampling of user posts, and Time2VecTransformer
(Bucur et al., 2023), a time-aware multimodal transformer, uses time-enriched
positional embeddings and sub-sequence sampling. In the few shot setting,
BERT(base), finetuned with 2 training samples, is compared (Kenton and
Toutanova, 2019). In the zero-shot setting, we compare our system with PTDD,
which uses BERT to obtain classification results by inputting prompts (Guo et
al., 2023).

Implementation Details. In our experiments, we used the public
ChatGPT(gpt-3.5-turbo) and GPT-3(text-davinci-003) as the underlying language
model due to its widespread usage and ease of access. For each user, we
selected four posts and utilized two types of demonstrations: one positive and
one negative. We aimed to demonstrate the efficacy of our depression detection
system in terms of generalization ability. To this end, we leveraged the
TextAttack (Morris et al., 2020) tool to generate out-of-distribution (OOD)
test data for the TMDD. The original test data for these datasets are referred
to as in-distribution (ID) test data. Specifically, we generated OOD samples
by replacing original words with words that are visually similar but
semantically different and by removing certain characters in words, such as
”name” → ”nme.”

Figure 4. (a),(b) illustrate the impacts of different tweet selectors and
number of tweets on TMDD and WU3D, respectively. (c) presents the effects of
different number of demonstrations.

###  4.2. Quantity Analysis

Main results Table 2 presents the performance in the full data, few-shot and
zero-shot settings on both independent-identical-distribution(IID) and out-of-
distribution(OOD) evaluation data. We have the following observations based on
it.

independent-identical-distribution(IID): In different data settings and
metrics, our system has achieved the best performance results. In the full
data setting, we utilized the result of the current SOTA model, i.e.,
Time2VecTransformer (Bucur et al., 2023), as an answer heuristic and obtained
better results. Specifically, the F1 metric increased from 0.931 to 0.946,
indicating that our system can calibrate the results of other models. In the
few-shot setting, our method outperforms the BERT-based classification model
with a significant improvement, as the F1 metric increased from 0.696 to
0.870. In the zero-shot setting, our method also showed a significant
improvement over PTDD, which is a prompt-based depression detection method.
Furthermore, compared with the BERT-based method, our approach demonstrated a
larger lead in the few-shot setting, indicating that in-context learning can
better utilize few-shot samples.

Out of distribution(OOD): The results in Table 2 demonstrate that our method
exhibits robustness to out-of-distribution data shifts. In the full data
setting, the OOD data caused a significant decline in the performance of the
traditional depression detection model, i.e., Time2VecTransformer, with the F1
score decreasing from 0.931 to 0.804. Our system utilizes
Time2VecTransformer’s results as an answer heuristic for the final prediction,
which demonstrated a stronger robustness, with the F1 score decreasing from
0.946 to 0.864. In the few-shot and zero-shot settings, since there was no
answer heuristic module, our system showed stronger robustness, with the F1
score decreasing from 0.870 and 0.689 to 0.851 and 0.656, respectively.

In addition, we conducted additional experiments using two different backbone
language models, GPT-3 (text-davinci-003) and ChatGPT (gpt-3.5-turbo), to
investigate the generalizability of our system. The results indicate that our
system significantly enhances the performance of both models. Nevertheless,
the ChatGPT’s high generation flexibility makes depression detection more
challenging, resulting in slightly lower performance compared to our system
using GPT-3. Overall, these promising findings highlight the effectiveness and
versatility of our system across different backbone language models in
detecting depression. Besides, ChatGPT (gpt-3.5-turbo) excels at zero-shot
settings, whereas GPT-3 (text-davinci-003) is better equipped to leverage few-
shot demonstrations.

Ablation study for each module Table 3 shows the ablation study on TMDD and
WU3D. Due to the collaborative nature of some modules, we conducted an
iterative ablation study to observe their effects. When we removed answer
heuristic $\mathcal{A}$ from our system, which means we no longer used the
results of other depression detection models trained on full data, and shifted
from a full data setting to a few-shot setting, our performance decreased
accordingly. We further removed $\mathcal{COT}$, i.e., diagnostic evidence
from demonstrations $\mathcal{D}emo$, and found that performance significantly
decreased on both TMDD and WU3D, with F1 scores dropping to 0.789 and 0.802,
respectively. This indicates that providing $\mathcal{COT}$ in demonstrations
can indeed improve performance. Additionally, after removing $\mathcal{COT}$,
we found in a subsequent case study that the system no longer provided
diagnosis evidence. In the next step, we removed the professional diagnosis
criteria $\mathcal{C}$, which means the system could no longer diagnose
depression based on professional criteria. Although LLMs may contain a vast
amount of psychological knowledge, removing the professional diagnosis
criteria $\mathcal{C}$ led to a significant decrease in F1 score on both
datasets, with scores dropping to 0.702 and 0.691, respectively. Finally, we
removed demonstrations $\mathcal{D}emo$ entirely, which meant that F1 score on
TMDD and WU3D decreased to 0.662 for only the instruction, options, and user
posts in our prompt.

Ablation study on different Tweets Selector $\mathcal{S}$ Figure 4(a)(b)
illustrates the different performance of our system using three different
tweet selector strategies on TMDD and WU3D. Among the three, the sentiment
analysis selector consistently outperforms the others, indicating its ability
to effectively filter out depression-related tweets. This indicates that
sentiment analysis can be used to retrieve more relevant posts, thereby
improving the efficiency of the system.

Effect of $n$ (numbers of selected posts). As shown in Figure 4(c), overall,
if the numbers of selected posts are too few, it is not possible to provide
enough information, resulting in decreased performance. If the numbers of
selected posts are too many, irrelevant content and longer context are
introduced, which poses greater detection challenges to LLMs. However, for the
recent-tweets selector, more posts obviously bring more useful information,
rather than more irrelevant content.

Figure 5. An example of our Chat-Diagnose system on TMDD dataset. More cases
can be found in Section Appendix.

Effect of $k$ (numbers of demonstrations). Figure 4(c) illustrates the
performance of our system with different numbers of demonstrations provided in
the prompt. In general, the more demonstrations provided, the better the
performance. We observe that there is a significant improvement in performance
from k=0 to k=1, as demonstrations can help format the output of the LLM.
However, as there is a limit to the total number of tokens that can be
inputted into the LLM, providing too many demonstrations does not necessarily
lead to more improvement. In other words, the benefit of increasing the number
of demonstrations is subject to diminishing marginal returns. Additionally, we
found that ChatGPT performs better than GPT-3 in zero-shot settings, while
GPT-3 can better utilize demonstrations.

###  4.3. Explainability and Interactivity Analysis

In this subsection, we demonstrate the explainability and interactivity of our
system through practical examples. Due to space constraints, we provide only
one example here, but we will include additional examples in Section Appendix.

Explainability of depression detection. As shown in the case in Figure 5, our
system not only provides diagnostic results but also diagnostic evidence,
i.e., which tweet of the user may correspond to which DSM diagnostic
criterion. Specifically, our system demonstrates that it considers the textual
content of the user’s tweet ”@getinthesea: GET IN THE FUCKING SEA, GET IN THE
FUCKING SEA, GET IN THE FUCKING SEA” and the corresponding image content ”A
red stop sign sitting on the side of a building” and ”win over negative,
protect happiness” to meet the first criterion of the DSM diagnosis (Depressed
Mood, e.g., feeling angry, hopeless). In addition, the text in another tweet
”Holy shit I am tired. I mean, I went and lifted, but I am exhausted. How am I
supposed to get my writing done like this” meets the sixth criterion of the
DSM diagnosis (Fatigue. Or loss of energy). Therefore, our system determines
that this user has a relatively high risk of developing depression.

Interactivity of our system. Figure 5 also demonstrates the interactivity of
our system. Based on the content of the user’s tweets and the diagnostic
results, our system initiates a conversation with the user, delving into the
topics that may be related to depression in the tweets. Our system further
discussed the user’s post describing fatigue, ”Holy shit I am tired. I mean, I
went and lifted, but I am exhausted. How am I supposed to get my writing done
like this”, and interacted with the user to provide more information not
available on social media. Based on these personalized details, our system
provided personalized advice.

##  5\. Conclusion

In this paper, our system initiates a next-generation paradigm for depression
detection in social media. Our system LLMs-augmented depression detection
system not only achieves state-of-the-art performance in terms of both
independent-identical-distribution (IID) and out-of-distribution (OOD) data,
but also provides explainability and interactivity. After detection, our
system provides diagnostic evidence based on both professional diagnostic
criteria and personal social media content through dialogue. Moreover, in the
dialogue system, users can interactively provide more information about their
mental state and social media under our induced questioning. Our system can
then better assess the user’s mental state and provide customized and
personalized suggestions.

## References

  * (1)
  * Allan et al. (2014) Charlotte E Allan, Vyara Valkanova, and Klaus P Ebmeier. 2014\.  Depression in older people is underdiagnosed.  _The Practitioner_ 258, 1771 (2014), 19–22.
  * An et al. (2020) Minghui An, Jingjing Wang, Shoushan Li, and Guodong Zhou. 2020\.  Multimodal topic-enriched auxiliary learning for depression detection. In _Proceedings of the 28th international conference on computational linguistics_. 1078–1089.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.  https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * Bucur et al. (2023) Ana-Maria Bucur, Adrian Cosma, Paolo Rosso, and Liviu P. Dinu. 2023.  It’s Just a Matter of Time: Detecting Depression with Time-Enriched Multimodal Transformers. In _European Conference on Information Retrieval_.
  * Cao et al. (2022) Rui Cao, Roy Ka-Wei Lee, Wen-Haw Chong, and Jing Jiang. 2022\.  Prompting for Multimodal Hateful Meme Classification. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 321–332.  https://aclanthology.org/2022.emnlp-main.22
  * Chen et al. (2022) Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022\.  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.  _arXiv preprint arXiv:2211.12588_ (2022).
  * Cheng and Chen (2022) Ju Chun Cheng and Arbee LP Chen. 2022.  Multimodal time-aware attention networks for depression detection.  _Journal of Intelligent Information Systems_ 59, 2 (2022), 319–339.
  * Chiu et al. (2021) Chun Yueh Chiu, Hsien Yuan Lane, Jia Ling Koh, and Arbee L. P. Chen. 2021.  Multimodal Depression Detection on Instagram Considering Time Interval of Posts.  _J. Intell. Inf. Syst._ 56, 1 (feb 2021), 25–47.  https://doi.org/10.1007/s10844-020-00599-5
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  PaLM: Scaling Language Modeling with Pathways.  _ArXiv_ abs/2204.02311 (2022).
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022.  A Survey for In-context Learning.  _arXiv preprint arXiv:2301.00234_ (2022).
  * Edwards et al. (2007) Sarah Edwards, Lucy Tinning, June SL Brown, Jed Boardman, and John Weinman. 2007.  Reluctance to seek help and the perception of anxiety and depression in the United Kingdom: A pilot vignette study.  _The Journal of Nervous and Mental Disease_ 195, 3 (2007), 258–261.
  * Fekete (2002) Sandor Fekete. 2002\.  The Internet-a new source of data on suicide, depression and anxiety: a preliminary study.  _Archives of Suicide Research_ 6, 4 (2002), 351–361.
  * Gao et al. (2023) Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang. 2023\.  Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System.  _ArXiv_ abs/2303.14524 (2023).
  * Gui et al. (2019) Tao Gui, Liang Zhu, Qi Zhang, Minlong Peng, Xu Zhou, Keyu Ding, and Zhigang Chen. 2019.  Cooperative multimodal approach to depression detection in twitter. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 110–117.
  * Guo et al. (2023) Yanrong Guo, Jilong Liu, Lei Wang, Wei Qin, Shijie Hao, and Richang Hong. 2023\.  A Prompt-Based Topic-Modeling Method for Depression Detection on Low-Resource Data.  _IEEE Transactions on Computational Social Systems_ (2023), 1–10.  https://doi.org/10.1109/TCSS.2023.3260080
  * He et al. (2023) Jiabang He, Lei Wang, Yi Hu, Ning Liu, Hui Liu, Xing Xu, and Heng Tao Shen. 2023\.  ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction.  arXiv:2303.05063 [cs.CL]
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of NAACL-HLT_. 4171–4186.
  * Lu et al. (2022) Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2022\.  Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning.  _arXiv preprint arXiv:2209.14610_ (2022).
  * Ma (2021) Flora Ma. 2021.  _Diagnostic and Statistical Manual of Mental Disorders-5 (DSM-5)_.  Springer International Publishing, Cham, 1414–1425.  https://doi.org/10.1007/978-3-030-22009-9_419
  * Mokady et al. (2021) Ron Mokady, Amir Hertz, and Amit H Bermano. 2021.  ClipCap: CLIP Prefix for Image Captioning.  _arXiv preprint arXiv:2111.09734_ (2021).
  * Morris et al. (2020) John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020\.  TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. 119–126.
  * Organization. (2012) World Health Organization.(2012). 2012\.  Depression: A global crisis. In _World Mental Health Day, October 10 2012._ . World Federation for Mental Health, VA, USA.
  * Reece and Danforth (2017) Andrew G Reece and Christopher M Danforth. 2017.  Instagram photos reveal predictive markers of depression.  _EPJ Data Science_ 6, 1 (2017), 15.
  * Ríssola et al. (2020) Esteban Andrés Ríssola, Mohammad Aliannejadi, and Fabio Crestani. 2020.  Beyond modelling: Understanding mental disorders in online social media. In _Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part I 42_. Springer, 296–310.
  * Rude et al. (2004) Stephanie Rude, Eva-Maria Gortner, and James Pennebaker. 2004\.  Language use of depressed and depression-vulnerable college students.  _Cognition & Emotion_ 18, 8 (2004), 1121–1133.
  * Sawhney et al. (2020) Ramit Sawhney, Harshit Joshi, Saumya Gandhi, and Rajiv Shah. 2020\.  A time-aware transformer based model for suicide ideation detection on social media. In _Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP)_. 7685–7697.
  * Shaikh et al. (2022) Omar Shaikh, Hongxin Zhang, William B. Held, Michael Bernstein, and Diyi Yang. 2022\.  On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning.  _ArXiv_ abs/2212.08061 (2022).
  * Sheehan (2004) David V Sheehan. 2004\.  Depression: underdiagnosed, undertreated, underappreciated.  _Managed care (Langhorne, Pa.)_ 13, 6 Suppl Depression (2004), 6–8.
  * Skaik and Inkpen (2020) Ruba Skaik and Diana Inkpen. 2020.  Using twitter social media for depression detection in the canadian population. In _2020 3rd artificial intelligence and cloud computing conference_. 109–114.
  * Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed Huai hsin Chi, Denny Zhou, and Jason Wei. 2022.  Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them.  _ArXiv_ abs/2210.09261 (2022).
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023\.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ (2023).
  * Trotzek et al. (2018) Marcel Trotzek, Sven Koitka, and Christoph M Friedrich. 2018\.  Utilizing neural networks and linguistic metadata for early detection of depression indications in text sequences.  _IEEE Transactions on Knowledge and Data Engineering_ 32, 3 (2018), 588–601.
  * Tsimpoukelli et al. (2021) Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021.  Multimodal few-shot learning with frozen language models.  _Advances in Neural Information Processing Systems_ 34 (2021), 200–212.
  * Wang et al. (2022a) Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2022a.  Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters.  _arXiv preprint arXiv:2212.10001_ (2022).
  * Wang et al. (2023) Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. 2023.  ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models.  arXiv:2302.07257 [cs.CV]
  * Wang et al. (2022b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022b.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_ (2022).
  * Wang et al. (2020) Yiding Wang, Zhenyi Wang, Chenghao Li, Yilin Zhang, and Haizhou Wang. 2020.  A multitask deep learning approach for user depression detection on sina weibo.  _arXiv preprint arXiv:2008.11708_ (2020).
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ (2022).
  * Wei et al. (2022b) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022b.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ (2022).
  * Wei et al. (2022c) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022c.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ (2022).
  * Williams et al. (2017) Sharifa Z Williams, Grace S Chung, and Peter A Muennig. 2017\.  Undiagnosed depression: A community diagnosis.  _SSM-population health_ 3 (2017), 633–638.
  * Yang et al. (2022a) Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022a.  An empirical study of gpt-3 for few-shot knowledge-based vqa. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 3081–3089.
  * Yang et al. (2022b) Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022b.  An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 36, 3 (Jun 2022), 3081–3089.  https://doi.org/10.1609/aaai.v36i3.20215
  * Yates et al. (2017) Andrew Yates, Arman Cohan, and Nazli Goharian. 2017\.  Depression and self-harm risk assessment in online forums.  _arXiv preprint arXiv:1709.01848_ (2017).
  * Zeng et al. (2022) Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. 2022\.  Socratic models: Composing zero-shot multimodal reasoning with language.  _arXiv preprint arXiv:2204.00598_ (2022).
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  OPT: Open Pre-trained Transformer Language Models.  _ArXiv_ abs/2205.01068 (2022).
  * Zhou et al. (2023) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023.  Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In _The Eleventh International Conference on Learning Representations_.  https://openreview.net/forum?id=WZH7099tgfM


11institutetext: Department of Computer Science, Johns Hopkins University,
Baltimore, MD, USA
22institutetext: Department of Radiology, Johns Hopkins University, Baltimore,
MD, USA 33institutetext: Department of Electrical and Computer Engineering,
Johns Hopkins University, Baltimore, MD, USA

# ReconFormer: Accelerated MRI Reconstruction Using Recurrent Transformer

Pengfei Guo Equal contribution.11 Yiqun Mei* 11 Jinyuan Zhou 22 Shanshan Jiang
22 Vishal M. Patel 1133

###### Abstract

Accelerating magnetic resonance image (MRI) reconstruction process is a
challenging ill-posed inverse problem due to the excessive under-sampling
operation in $k$-space. In this paper, we propose a recurrent transformer
model, namely ReconFormer, for MRI reconstruction which can iteratively
reconstruct high fertility magnetic resonance images from highly under-sampled
$k$-space data. In particular, the proposed architecture is built upon
Recurrent Pyramid Transformer Layers (RPTL), which jointly exploits intrinsic
multi-scale information at every architecture unit as well as the dependencies
of the deep feature correlation through recurrent states. Moreover, the
proposed ReconFormer is lightweight since it employs the recurrent structure
for its parameter efficiency. We validate the effectiveness of ReconFormer on
multiple datasets with different magnetic resonance sequences and show that it
achieves significant improvements over the state-of-the-art methods with
better parameter efficiency. Implementation code will be available in
https://github.com/guopengf/ReconFormer.

###### Keywords:

MRI Reconstruction Deep Learning Transformer.

##  1 Introduction

Magnetic Resonance Imaging (MRI) is one of the most prevalent diagnostic and
research tools in clinical scenarios, which provides excellent resolution and
abundant contrast mechanisms to visualize different structural and functional
properties of the underlying anatomy. Due to physiological and hardware
constraints [17], MRI acquisition process is inherently slow. Consequently,
extending acquisition time to collect complete data in $k$-space (frequency
domain) imposes significant burden on patients and makes MRI less accessible.
One of the common approaches to accelerate MRI acquisition procedure is to
collect partial data rather than traverse whole $k$-space, as known as under-
sampling $k$-space. However, such operation violates the Nyquist-Shannon
sampling theorem and introduces aliasing artifacts in the reconstructed image.
Compressed sensing (CS) mitigates this issue by formulating the image
reconstruction as solving an optimization problem with several assumptions
including sparsity and incoherence [9]. Past literature in advanced CS-based
image reconstruction has exploited low rank constraint terms [18], adaptive
sparse modelling [19, 20], and parallel imaging [13]. However, CS algorithms
have several deficiencies impeding its practicability in real-world
applications. First, CS recovery algorithms require careful tuning of
regularization functions and hyper-parameters, which is also problem-specific
and non-trivial. Second, due to the nature of the iterative optimization, CS
methods often demand longer running time to achieve desirable reconstruction
quality. Third, the acceleration factors are generally below 3 for typical MR
images, when high-frequency oscillatory artifacts are not properly reduced
during the optimization process [23].

In contrast, recently advanced deep learning-based methods are gaining more
attention for fast and accurate MRI reconstruction. Rather than explicitly
defining prior information and regularization functions, networks learn them
implicitly from data. CNN-based MRI reconstruction methods have been shown to
provide much better MR image quality than conventional CS-based methods [22].
Such methods focus on elaborating architecture designs, including overcomplete
networks [8], neural ODEs [2], invertible networks [21], cascaded
architectures [25, 5], and recurrent structures [30, 22]. Although those
methods achieve promising results, the basic convolution layer generally
suffers from the following drawbacks: first, a content-independent convolution
kernel is not optimal to restore different image regions [14]; second, the
long-range dependency is not effectively modeled in convolutional networks
[29].

Figure 1: Comparisons of different transformer architectures. (a) Recent high-
level vision Transformer [16, 31] use a pyramid structure to model multi-scale
contexts. (b) Previous low-level vision transformers [3, 14] and ViT [4]
follow a simple columnar design, which preserves better information but fails
to model at scales. (c) ReconFormer incorporates the pyramid structure inside
a transformer layer. The local pyramid enables scale processing at each unit
while the globally columnar structure maintains high-resolution information.
FFN refers to feed-forward network in transformer layers.

Recent advances in transformer [29] models introduce the self-attention
mechanism to capture global interactions between contexts and open a new
possibility to solve the challenging problem of MRI reconstruction. While
vision transformer has shown superior performance on general vision tasks [1,
4, 16, 27], a few attempts have been made for utilizing transformers in MRI
reconstruction. Feng _et al._ [6] proposed a task transformer network for
joint MRI reconstruction and super-resolution. Korkmaz _et al._ [12] proposed
SLATER, which can perform self-supervised MRI reconstruction facilitated by an
adversarial transformer. A texture transformer module was introduced in [7]
for reference-based MRI reconstruction. Although those works [6, 7, 12]
demonstrated a good performance, their architectures still heavily rely on the
self-attention mechanism proposed in ViT [4] and cannot efficiently model the
multi-scale information. A fundamental vision principle that visual elements
vary at scales plays a vital role in computer vision [15]. To leverage this
intrinsic property, previous vision transformer models [1, 4, 16, 27, 6, 12]
rely on the network topology, as shown in Fig. 1(a). However, such design
suffers from the following. 1) Scale modeling is inefficient. Scale
information is gradually processed through network hierarchy, so the multi-
scale representation is only obtained at the end of stages. 2) Scale modeling
is inflexible. The network topology design is usually task-dependent and hard
to generalize for other tasks.

In this paper, to overcome those issues, a novel Recurrent Transformer, termed
ReconFormer, is proposed to recover the fully-sampled image from the under-
sampled $k$-space data in accelerated MRI reconstruction. In summary, the
following are our key contributions: 1. ReconFormer introduces a powerful
locally pyramidal but globally columnar architecture, which can perceive
multi-scale representation at any stage while well preserving image details,
as shown in Fig. 1(c). 2. Recurrent Pyramid Transformer Layers (RPTL) are
proposed to allow scale modeling at every basic building units and exploit
deep feature correlation through recurrent states. 3. By fully employing the
parameter efficiency of recurrent structure, ReconFormer allows the training
on limited medical data from scratch. The effectiveness of our approach is
validated on two datasets with various MR sequences and experiment results
demonstrate that ReconFormer achieves significant improvements over the state-
of-the-art CNN-based as well as transformer models with better parameter
efficiency.

##  2 Methodology

MRI Reconstruction.  Let $x\in\mathbb{C}^{N}$ represent the observed complex-
valued under-sampled $k$-space measurements and $y\in\mathbb{C}^{M}$ denote
the fully-sampled data. Our aim is to reconstruct $y$ from $x$ ($N\ll M$) as
follows:

$x=F_{\text{u}}y+\epsilon,$ (1)

where $F_{u}\in\mathbb{C}^{N\times M}$ denotes an under-sampled Fourier
encoding matrix controlling the acceleration factor (AF).
$\epsilon\in\mathbb{C}^{N}$ represents the additive acquisition noise. $F$ and
$F^{-1}$ represent the Fourier transform and its inverse. The MRI
reconstruction is commonly formulated as an unconstrained optimization problem
as follows [22, 25, 30]:

$\min\limits_{\text{y}}R(y)+\lambda\|x-F_{\text{u}}y\|_{2}^{2}.$ (2)

Here, $R$ is the regularization term on $y$ and $\lambda$ adjusts the
contribution of the data fidelity term based on the noise level of the
acquired measurements $x$. For deep learning-based approaches [22, 25], one
can force $y$ to be approximated in the regularization term $R$ by
reconstruction. The optimization problem in Eq. 2 can be rewritten as follows:

$\min\limits_{\text{y}}\|y-F[f_{\text{nn}}(\bar{x}|\Theta)]\|_{2}^{2}+\lambda\|x-F_{\text{u}}y\|_{2}^{2},$
(3)

where $f_{\text{nn}}$ denotes the forward pass of a neural network
parameterized by $\Theta$ and $\bar{x}$ is the zero-filled reconstruction. We
denote $\bar{y}=f_{\text{nn}}(\bar{x}|\Theta)$ as the approximated fully-
sampled reconstructed image from the observed under-sampled $k$-space data
$x$.

Figure 2: The architecture of the proposed ReconFormer. (a) A schematic of
unrolled ReconFormer iterations. Here, $\otimes$ denotes the channel-wise
concatenation and RFB is the ReconFormer block. (b) The illustration of a
recurrent unit (RU). Here, $\oplus$ denotes the element-wise addition.

ReconFormer.  As shown in Fig. 2(a), ReconFormer consists of three recurrent
units and a refine module (RM). To maintain high-resolution information,
ReconFormer employs globally columnar structure. In particular, recurrent
units map the input degraded images to the original dimension. Meanwhile,
across each recurrent unit, the receptive fields of recurrent units is
gradually reduced to reconstruct high-quality MR images in a coarse-to-fine
way. It is worth noting that the last recurrent unit $\text{RU}_{3}$ employs
overcomplete architecture [8], which has been demonstrated to efficiently
constraint the receptive field. As shown in Fig. 2(b), a recurrent unit
contains an encoder $f^{\text{Enc}}$, a ReconFormer block $f^{\text{RFB}}$,
and a decoder $f^{\text{Dec}}$. To have a more stable optimization [32],
encoder and decoder are built up on convolution layers. A data consistency
(DC) layer is added at the end of each decoder network to reinforce the data
consistency in the $k$-space. We denote a single recurrent unit
$\text{RU}_{i}$ as $f^{\text{RU}}_{i}$ where $i\in\\{1,2,3\\}$ indicates the
$i^{\text{th}}$ recurrent unit and its operation in the $t^{\text{th}}$
iteration can be formulated as follows:

$\displaystyle\bar{y}_{i}^{(t)}$
$\displaystyle=\text{DC}(f^{\text{RU}}_{i}(\bar{y}_{i-1}^{(t)},h_{i}^{(t)},c_{i}^{(t)}),x,U),$
(4)
$\displaystyle=F^{-1}[Ux+(1-U)F[(f^{\text{RU}}_{i}(\bar{y}_{i-1}^{(t)},h_{i}^{(t)},c_{i}^{(t)})]],$
$\displaystyle=F^{-1}[Ux+(1-U)F[f_{i}^{\text{Dec}}(f_{i}^{\text{RFB}}(h_{i}^{(t)},c_{i}^{(t)})+f_{i}^{\text{Enc}}(\bar{y}_{i-1}^{(t)}))]],$

where $U$ is a binary under-sampling mask in $F_{\text{u}}$ which is used for
data consistency in $k$-space.
$\\{\bar{y}_{i-1}^{(t)},\bar{y}_{i}^{(t)}\\}\in\mathbb{R}^{H\times W\times 2}$
are input and output of $\text{RU}_{i}$, respectively, where $H$ and $W$ are
the zero-filled reconstruction height and width, and 2 represents the real and
imaginary channels.
$h_{i}^{(t)}\in\mathbb{R}^{\frac{H}{S}\times\frac{W}{S}\times C}$ is the
hidden state from the previous iteration, where $S$ is the scaling factor
determined by the structure of each RU and consequently controls the size of
receptive fields. $c_{i}^{(t)}$ is deep feature correlation from the previous
iteration. At the $t^{\text{th}}$ iteration, ReconFormer can be unrolled as
follows:

$\displaystyle\\{\bar{y}_{1}^{(t)},\bar{y}_{2}^{(t)},\bar{y}_{3}^{(t)}\\}$
$\displaystyle=\\{\text{RU}_{i}(\bar{y}_{i-1}^{(t)},h_{i}^{(t)},c_{i}^{(t)},x,U)|i=1,2,3\\},$
(5) $\displaystyle\bar{y}^{(t+1)}$
$\displaystyle=\text{DC}(\text{RM}(\bar{y}_{1}^{(t)}\otimes\bar{y}_{2}^{(t)}\otimes\bar{y}_{3}^{(t)}),x,U),$

where $\otimes$ denotes the channel-wise concatenation. $\bar{y}^{(t+1)}$ is
the output of the current iteration and also servers as the input for the next
iteration.

Figure 3:  (a) The schematic of recurrent scale-wise attention (RSA). (b) An
illustration of the transition function in a RPTL. (c) The architecture of the
proposed RPTL.

RPTL.  A ReconFormer block (RFB) is formed by stacked recurrent pyramid
transformer layers (RPTL). The core design of RPTL is the Recurrent Scale-wise
Attention (RSA), as shown in Fig. 3(a). Compared to the standard multi-head
self-attention of the original transformer layer [4], the main difference lies
in two aspects: (i) instead of performing a single attention function, the
proposed RSA consists of several attention scale heads (_i.e._ , $\times
1,\times 3,\times 5$), which operate on multi-scale patches in parallel. Such
design enables efficient in-place scale modeling and forms a feature pyramid
by projecting features at various scales directly into multiple attention
heads. Consequently, the proposed RPTL allows scale processing at basic
architecture unit. (ii) The correlation estimation in the proposed RSA relies
on both the hidden state $h^{(t)}$ and the deep feature correlation $c^{(t)}$
from the previous iteration, which enables more robust correlation estimation
by propagating correlation information between adjacent states. The proposed
RPTL utilizes local attention mechanism with shifted window scheme [16]. Given
the input hidden state $h^{(t)}\in\mathbb{R}^{H^{\prime}\times
W^{\prime}\times C^{\prime}}$, RPTL first uses non-overlapping $K\times K$
local windows to partition $h^{(t)}$ into a feature of size
$\frac{H^{\prime}\times W^{\prime}}{K^{2}}\times K^{2}\times C^{\prime}$,
where $K$ is the size of local windows and $\frac{H^{\prime}\times
W^{\prime}}{K^{2}}$ is the number of windows. Then, the proposed RSA operates
on each window in parallel to compute the self-attention. Let
$F\in\mathbb{R}^{K^{2}\times C^{\prime}}$ denote a local window feature. the
query $Q$, key $K$ and value $V$ matrices in RPTL are formulated as follows:

$\displaystyle Q=FP_{Q},K=FP_{K},V=FP_{V},$ (6)

where $P_{Q},P_{K},\text{and }P_{V}$ are the corresponding projection matrices
in attention scale heads. As shown in Fig. 3(b), we compute the outputs of RSA
as follows:

$\displaystyle\text{Attention}(Q,K,V,c^{(t)})$
$\displaystyle=\text{SoftMax}(c^{(t+1)})V,$ (7) $\displaystyle c^{(t+1)}$
$\displaystyle=\lambda\frac{QK^{T}}{\sqrt{d}}+(1-\lambda)c^{(t)},$

where $d$ is the embedding dimension and $\lambda$ is a learnable parameter
controlling the contribution of the deep feature correlation from the adjacent
state. As shown in Fig. 3(c), we can formulate the whole process of RPTL as
follows:

$\displaystyle F$ $\displaystyle=\text{RSA}(\text{LN}(F))+F,$ (8)
$\displaystyle F$ $\displaystyle=\text{MLP}(\text{LN}(F))+F,$

where LN and MLP denote the Layer Normalization and the multi-layer
perceptron, respectively. The detailed network configuration can be found in
the supplementary material.

Figure 4: Qualitative comparison of different methods on (a) the HPKS dataset
and (b) the fastMRI dataset. The second column of each subplot shows the
corresponding error maps. The red boxes highlight regions where RenconFormer
significantly outperforms the competitive methods.

##  3 Experiments and Results

Evaluation and Implementation Details.  The fastMRI [11] and HPKS [10]
datasets are used for conducting experiments. The fastMRI dataset contains
1,172 complex-valued single-coil coronal proton density (PD)-weighted knee MRI
scans. Each scan approximately provides 35 coronal cross-sectional knee images
with the matrix of size $320\times 320$. We partition this dataset into 973
scans for training, and 199 scans (fastMRI validation dataset) for testing.
The HPKS dataset provides complex-valued single-coil axial $T_{1}$-weighted
brain MRI scans from 144 post-treatment patients with malignant glioma. Each
scan contains 15 axial cross-sectional images with the matrix of size
$256\times 256$. The data splits are as follows: 102 scans are used for
training, 14 scans are used for validation, and 28 scans are used for testing.
In experiments, the input under-sampled image sequences are generated by
randomly under-sampling the $k$-space data using the Cartesian under-sampling
function that is the same as the fastMRI challenge [11]. The proposed
ReconFormer is trained using the $\ell_{1}$ loss with Adam optimizer with the
following hyperparameters: learning rate of $2.0\times 10^{-4}$; 50 maximum
epochs; batch size of 4; the number of recurrent iterations $T$ of 5. Peak
signal to noise ratio (PSNR) and structural index similarity (SSIM) are used
as the evaluation metrics for comparison. We implement the proposed model
using PyTorch on Nvidia RTX8000 GPUs.

Figure 5: $k$-space analysis of (a) low frequency and (b) high frequency on the HPKS dataset. (c) Unrolling length v.s. reconstruction performance of ReconFormer on HPKS dataset with $4\times$ acceleration.  Table 1: Quantitative results on the HPKS and fastMRI dataset. Param denotes the number of parameters. Method | Param | HPKS | fastMRI
---|---|---|---
SSIM | PSNR | SSIM | PSNR
AF = 4 | AF = 8 | AF = 4 | AF = 8 | AF = 4 | AF = 8 | AF = 4 | AF = 8
CS [26] | - | 0.8705 | 0.7125 | 29.94 | 24.96 | 0.5736 | 0.4870 | 29.54 | 26.99
UNet [24] | 8.634 M | 0.9155 | 0.8249 | 34.47 | 29.47 | 0.7142 | 0.6424 | 31.88 | 29.78
KIKI-Net [5] | 1.790 M | 0.9363 | 0.8436 | 35.35 | 29.86 | 0.7172 | 0.6355 | 31.87 | 29.27
Kiu-net [28] | 7.951 M | 0.9335 | 0.8467 | 35.35 | 30.18 | 0.7228 | 0.6456 | 32.06 | 29.86
SwinIR [14] | 5.156 M | 0.9364 | 0.8499 | 36.00 | 30.42 | 0.7213 | 0.6537 | 32.14 | 30.21
D5C5 [25] | 2.237 M | 0.9595 | 0.8623 | 37.51 | 30.40 | 0.7256 | 0.6457 | 32.25 | 29.65
OUCR [8] | 1.192 M | 0.9747 | 0.9044 | 39.33 | 32.14 | 0.7354 | 0.6634 | 32.61 | 30.59
ReconFormer (Ours) | 1.141 M | 0.9788 | 0.9210 | 40.09 | 33.04 | 0.7383 | 0.6697 | 32.73 | 30.89

Comparisons with the state-of-the-art.  To verify the effectiveness of the
proposed ReconFormer, we compare the proposed method with 7 representative
methods, including conventional compressed sensing (CS) based method [26],
popular CNN-based methods – UNet [24], KIKI-Net [5], Kiu-net [28], and D5C5
[25], state-of-the-art iterative reconstruction approaches – OUCR [8], and
vision transformer model – SwinIR [14]. For a fair comparison, methods (UNet
[24], Kiu-net [28], and SwinIR [14]) that are not originally proposed for MRI
reconstruction are modified for data with real and imaginary channels and a DC
layer is added at the end of the networks. The unrolling length of all
iterative approaches is set to 5.

Table 1 shows the quantitative results evaluated on the two datasets for AF=4
and AF=8. Compared with the other methods, the proposed ReconFormer achieves
the best performance on multiple datasets for all acceleration factors while
containing the least number of parameters. It is worth noting that our method
exhibits a larger performance improvement, when the acceleration factor
increases (_i.e._ , more challenging scenarios). In particular, for the HPKS
and fastMRI datasets, our model outperforms the most competitive approach OUCR
[8] by 0.9 dB and 0.3 dB in 8$\times$ acceleration, respectively. While the
fastMRI dataset is more challenging due to the acquisition quality, all
reported improvements achieved by ReconFormer are statistically significant.
We provide the detailed statistical significance test in the supplementary
material. Figure 4 shows qualitative evaluations on two datasets. It can be
seen that ReconFormer yields remarkable reconstruction quality and
perceptually outperforms the other methods by a large margin, as shown in the
red boxes of each sub-figure in Fig. 4. Moreover, we conduct the $k$-space
analysis to investigate the performance of different methods in low (1/3
center frequencies) and high (2/3 peripheral frequencies) spatial frequencies.
As shown in Fig. 5, the proposed method is shown to be more effective for both
low frequency information (_e.g._ , contrast, brightness, and general shape)
and high frequency features (_e.g._ , edges, details, sharp transitions). The
superior performance on different MRI sequences demonstrates the merit of the
proposed transformer architecture and our RPTL in jointly utilizing intrinsic
multi-scale information and the deep feature correlation.

Ablation Study.  We first separately evaluate the effectiveness of network
components, including three recurrent units, RM, and PRTL in the proposed
ReconFormer. As shown in Table 2, when we add each recurrent unit, the
reconstruction quality can be gradually boosted, which demonstrates the
effectiveness of the architecture design. Then, we show that adding RM to fuse
the coarse-to-fine reconstructions further improves the performance. Moreover,
RPTL consistently provides performance improvements in different acceleration
factors, which implies the importance of fully exploiting multi-scale
information in basic building blocks. For experiments without RPTL, we replace
RSA by a standard multi-head self attention. The influence of the unrolling
length T on reconstruction results is shown in Fig. 5(c). We can observe that
reconstruction quality increases as the unrolling length expands, but gets
saturated when T = 5. There is a trade-off between the reconstruction accuracy
and the computational efficiency. Due to the space limitation, computation
details of different methods, more visualizations, and analysis on the fastMRI
dataset are provided in the supplementary material.

Table 2: The Ablation study of proposed modules on HPKS. Modules | AF=4 | AF=8
---|---|---
RU1 |  RU2 |  RU3 | RM | RPTL | SSIM | PSNR | SSIM | PSNR
✓ |  |  |  |  | 0.9671 | 38.19 | 0.8900 | 31.39
✓ | ✓ |  |  |  | 0.9762 | 39.63 | 0.9105 | 32.47
✓ | ✓ | ✓ |  |  | 0.9773 | 39.79 | 0.9151 | 32.75
✓ | ✓ | ✓ | ✓ |  | 0.9783 | 39.94 | 0.9186 | 32.90
✓ | ✓ | ✓ | ✓ | ✓ | 0.9788 | 40.09 | 0.9210 | 33.04

##  4 Conclusion

In this paper, we propose a recurrent transformer-based MRI reconstruction
model ReconFormer. By leveraging the novel RPTL, we are able to explore the
multi-scale representation at every basic building units and discover the
dependencies of the deep feature correlation between adjacent recurrent
states. Incorporating three recurrent units and the refine module, ReconFormer
reconstructs high-quality MR images through a locally pyramidal but globally
columnar structure and achieves the state-of-the-art performance on multiple
datasets. ReconFormer is lightweight and does not require pre-training on
large-scale datasets. Our experiments suggest the promising potential of using
transformer-based models in the MRI reconstruction task. In the future, we
will extend the proposed model to multi-coil reconstruction and more MRI
sequences.

## References

  * [1] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European Conference on Computer Vision. pp. 213–229. Springer (2020)
  * [2] Chen, E.Z., Chen, T., Sun, S.: Mri image reconstruction via learning optimization using neural odes. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 83–93. Springer (2020)
  * [3] Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., Gao, W.: Pre-trained image processing transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12299–12310 (2021)
  * [4] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [5] Eo, T., Jun, Y., Kim, T., Jang, J., Lee, H.J., Hwang, D.: Kiki-net: cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images. Magnetic resonance in medicine 80(5), 2188–2201 (2018)
  * [6] Feng, C.M., Yan, Y., Fu, H., Chen, L., Xu, Y.: Task transformer network for joint mri reconstruction and super-resolution. In: International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) (2021)
  * [7] Guo, P., Patel, V.M.: Reference-based magnetic resonance image reconstruction using texture transforme. arXiv preprint arXiv:2111.09492 (2021)
  * [8] Guo, P., Valanarasu, J.M.J., Wang, P., Zhou, J., Jiang, S., Patel, V.M.: Over-and-under complete convolutional rnn for mri reconstruction. In: Medical Image Computing and Computer Assisted Intervention. pp. 13–23. Springer (2021)
  * [9] Haldar, J.P., Hernando, D., Liang, Z.P.: Compressed-sensing mri with random encoding. IEEE transactions on Medical Imaging 30(4), 893–903 (2010)
  * [10] Jiang, S., et al.: Identifying recurrent malignant glioma after treatment using amide proton transfer-weighted mr imaging: a validation study with image-guided stereotactic biopsy. Clinical Cancer Research 25(2), 552–561 (2019)
  * [11] Knoll, F., et al.: fastmri: A publicly available raw k-space and dicom dataset of knee images for accelerated mr image reconstruction using machine learning. Radiology: Artificial Intelligence 2(1), e190007 (2020)
  * [12] Korkmaz, Y., Dar, S.U., Yurt, M., Özbey, M., Çukur, T.: Unsupervised mri reconstruction via zero-shot learned adversarial transformers. arXiv preprint arXiv:2105.08059 (2021)
  * [13] Liang, D., Liu, B., Wang, J., Ying, L.: Accelerating sense using compressed sensing. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 62(6), 1574–1584 (2009)
  * [14] Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: Swinir: Image restoration using swin transformer. In: IEEE International Conference on Computer Vision Workshops (2021)
  * [15] Liu, D., Wen, B., Fan, Y., Loy, C.C., Huang, T.S.: Non-local recurrent network for image restoration. arXiv preprint arXiv:1806.02919 (2018)
  * [16] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 (2021)
  * [17] Lustig, M., Donoho, D.L., Santos, J.M., Pauly, J.M.: Compressed sensing mri. IEEE signal processing magazine 25(2), 72–82 (2008)
  * [18] Majumdar, A.: Improving synthesis and analysis prior blind compressed sensing with low-rank constraints for dynamic mri reconstruction. Magnetic resonance imaging 33(1), 174–179 (2015)
  * [19] Patel, V.M., Chellappa, R.: Sparse representations, compressive sensing and dictionaries for pattern recognition. In: The first Asian conference on pattern recognition. pp. 325–329. IEEE (2011)
  * [20] Patel, V.M., Maleh, R., Gilbert, A.C., Chellappa, R.: Gradient-based image recovery methods from incomplete fourier measurements. IEEE Transactions on Image Processing 21(1), 94–105 (2011)
  * [21] Putzky, P., Welling, M.: Invert to learn to invert. arXiv preprint arXiv:1911.10914 (2019)
  * [22] Qin, C., et al.: Convolutional recurrent neural networks for dynamic mr image reconstruction. IEEE Transactions on Medical Imaging 38(1), 280–290 (2019). https://doi.org/10.1109/TMI.2018.2863670
  * [23] Ravishankar, S., Bresler, Y.: Mr image reconstruction from highly undersampled k-space data by dictionary learning. IEEE transactions on medical imaging 30(5), 1028–1041 (2010)
  * [24] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
  * [25] Schlemper, J., et al.: A deep cascade of convolutional neural networks for dynamic mr image reconstruction. IEEE transactions on Medical Imaging 37(2), 491–503 (2017)
  * [26] Tamir, J.I., Ong, F., Cheng, J.Y., Uecker, M., Lustig, M.: Generalized magnetic resonance image reconstruction using the berkeley advanced reconstruction toolbox. In: ISMRM Workshop on Data Sampling & Image Reconstruction, Sedona, AZ (2016)
  * [27] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: International Conference on Machine Learning. pp. 10347–10357. PMLR (2021)
  * [28] Valanarasu, J.M.J., Sindagi, V.A., Hacihaliloglu, I., Patel, V.M.: Kiu-net: Towards accurate segmentation of biomedical images using over-complete representations. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 363–373. Springer (2020)
  * [29] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998–6008 (2017)
  * [30] Wang, P., Chen, E.Z., Chen, T., Patel, V.M., Sun, S.: Pyramid convolutional rnn for mri reconstruction. arXiv preprint arXiv:1912.00543 (2019)
  * [31] Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 568–578 (October 2021)
  * [32] Xiao, T., Dollar, P., Singh, M., Mintun, E., Darrell, T., Girshick, R.: Early convolutions help transformers see better. Advances in Neural Information Processing Systems 34 (2021)