# Memory-Augmented LLM Personalization with Short- and Long-Term Memory
Coordination

Kai Zhang1, Fubang Zhao2, Yangyang Kang2, Xiaozhong Liu1

###### Abstract

Large Language Models (LLMs), such as GPT3.5, have exhibited remarkable
proficiency in comprehending and generating natural language. However, their
unpersonalized generation paradigm may result in suboptimal user-specific
outcomes. Typically, users converse differently based on their knowledge and
preferences. This necessitates the task of enhancing user-oriented LLM which
remains unexplored. While one can fully train an LLM for this objective, the
resource consumption is unaffordable. Prior research has explored memory-based
methods to store and retrieve knowledge to enhance generation without
retraining for new queries. However, we contend that a mere memory module is
inadequate to comprehend a user’s preference, and fully training an LLM can be
excessively costly. In this study, we propose a novel computational bionic
memory mechanism, equipped with a parameter-efficient fine-tuning schema, to
personalize LLMs. Our extensive experimental results demonstrate the
effectiveness and superiority of the proposed approach. To encourage further
research into this area, we are releasing a new conversation dataset generated
entirely by LLM based on an open-source medical corpus, as well as our
implementation code111Github link will be placed here.

## Introduction

The potential of large language models (LLMs) to understand and generate
natural language is undeniable (Brown et al. 2020; Chowdhery et al. 2022;
Touvron et al. 2023), while there is an untapped opportunity to explore how
LLMs could be customised to provide personalized interactions with users,
allowing them to receive tailored responses that best suit their individual
needs (Bender et al. 2021). Efforts have been made to design proper prompts
for steering LLMs to enhance output. For example, by memorizing previous
mistakes and user feedback, given a new query, a similarity-based retriever
can be leveraged to preemptively recognize and rectify LLM errors(Dalvi,
Tafjord, and Clark 2022; Madaan et al. 2022; Lewis et al. 2020). However, this
paradigm poses us two challenges: Firstly, most existing memory designs are
dictionary-based (Madaan et al. 2022; Lewis et al. 2020) (i.e. key-value form
where key is the previous mistake, value is the corresponding user-feedback)
which can be inflexible and rely heavily on the power of retriever. For
example, as the conversation goes, the size of memory in this setting would
increase uncontrollably which will further impact the effectiveness and
accuracy of the retriever; Secondly, such paradigm, without retraining, can
barely provide users with personalized and engaging experience. For instance,
in the healthcare context, a diabetes patient who prefers concise and
straightforward medical advice won’t expect detailed glucose test explanations
from a doctor while others who prefer fully elaborated responses may want to
know as much as possible about the disease (e.g., causes etc.). To this end,
being aware of user preference and user-relevant information can be crucial
for enhancing user experience and remains understudied. In this paper, we
propose a novel framework that enhances personalized LLM learning through the
augmentation of memory and parameters-efficient fine-tuning schema.

Figure 1: Personalized responses for different users in terms of the same
query.

Dictionary-based memory is not pliable due to its intricate structure and thus
efforts can only be made in strengthening retrievers. Despite the improvements
made by retrievers like semantic-similarity based and distance-closest
based(Madaan et al. 2022), we argue that the memory structure should be
ameliorated to accommodate diverse information. For example, as Figure 1
depicted, doctors would recognize patients’ essential information after years
of engaging in diagnostic conversations. Among such information, however, some
would be memorized in a long-last way such as chronic diseases, while some are
not and those memories can be used for enhancing the responses in the future
dialogues. Unfortunately, rare efforts have been made to address this
challenge. Neuroscientists have revealed that real-world memory mechanism
works in two processes, one can be characterized as automatic and unconscious
while the other one is effortful and analytical(Kahneman 2011). For example,
acquiring a new skill initially can be active and analytical, but as one’s
proficiency increases, it becomes more intuitive. This is referred as Dual-
process theory and correspondingly, memory can be defined as three types:
working memory, short-term memory (STM) and long-term memory (LTM). Working
memory is responsible for filtering and buffering information, STM holds
knowledge for a short period, while LTM stores knowledge for a longer duration
(Roediger and McDermott 1995). Drawing inspirations from this, we propose a
novel Dual-Process enhanced Memory (DPeM) mechanism of which three types of
memory cooperate smoothly under the guidance of dual-process schema and thus
can provide LLM more useful knowledge from both user-specific and common-sense
aspects.

Furthermore, existing works in personalized LLMs primarily focus on designing
comprehensive prompts that enable LLMs to generate contextually relevant
responses aligned with user-specific dialogue preferences (e.g., preferring
concise responses) (Wang et al. 2023; Wu et al. 2023; Wang, Ustun, and Calmon
2019). However, these methods often yield inferior performance compared to
fine-tuning approaches and are susceptible to the exact formatting of the
prompts (e.g., wording and ordering) (Liu et al. 2022a). Another approach
proposed by Salemi et al. (2023) involves incorporating user profiles during
the pretraining stage, enabling LLMs to possess user-specific knowledge for
downstream tasks. Nevertheless, training fully personalized LLMs for
individuals can be economically unviable. To this end, we opt to the
utilization of parameter-efficient fine-tuning (PEFT) techniques to develop
user-oriented LLMs with reduced time and resource consumption. Specifically,
PEFT focuses on updating a small subset of parameters, ensuring that the
trained LLM achieves promising performance on new tasks while minimizing
computational costs. PEFT can enable LLM to learn to respond to users while
being aware of their specific needs in a computationally and resource-
efficient way.

In tandem, we propose a novel memory mechanism inspired by neuroscience, and
along with a PEFT training strategy to achieve LLM personalization. Our
approach moves away from the simplistic dictionary-based structure, instead
closely resembling real-world memory processes. The key contributions of our
work are as follows:

$\bullet$ We propose a novel DPeM mechanism that closely resembles real-world
memory processes which lead to a more efficient augmentation for LLM.

$\bullet$ We propose MaLP, a unified frame based on DPeM and PEFT which aims
to promote the LLM’s quality of responding by catering to user-specific needs.

$\bullet$ Our proposed method outperforms several baselines on three tasks and
the conducted human evaluation further validate the effectiveness and
superiority of MaLP.

$\bullet$ We introduce a new dialogue dataset in the medical domain that
incorporates user preferences and historical records. This dataset offers a
unique opportunity to explore personalized LLM models from a fresh
perspective.

Figure 2: Overview of MaLP: the user’s historical dialogues will firstly be
passed to a coordinator $\mathcal{C}$ and a trainable LLM equipped with PEFT
iteratively for memory generation and causal language modeling, respectively.
Then the memory generation module will form a memory using DPeM mechanism
where dual-process is denoted in green box along with three steps denoted in
colored lines separately. After iterations completed, a new query by the user
will be passed to a retriever for corresponding memory lookup and then the
fine-tuned LLM will produce the personalized response in terms of the
retrieved knowledge and historical dialogues.

## Methodology

### Preliminary Definition

Before going further, we would like to give our preliminary definitions first.
Given multi-round dialogues between two characters (e.g., patient and doctor)
which is denoted as $\mathcal{D}=\\{d_{0},...,d_{n}\\}$ where $n$ is the
number of rounds, our task here is to learn and memorize the knowledge from
$\mathcal{D}$ to form a memory $\mathcal{M}$ and fine-tune a large language
model (LLM) $\Phi$ to produce personalized response $y$ in terms of a new
query $x$ from the same user with respect to (w.r.t) $\mathcal{D}$ and
$\mathcal{M}$.

### DPeM Mechanism

The key novelty that differentiates our work with previous efforts is that we
turns into excavating the improvements regarding the intricate memory
structure instead of studying solely on retriever. Drawing inspirations from
Kahneman (2011), we aim to design a memory mechanism that closely resembles
real-world memory processes. To achieve this, we propose a dual-process
(Rehearsal Process, Executive Process) enhanced procedure which consists of
three steps - Learning, Summarizing and Memorizing. The Rehearsal Process
involves learning information from $\mathcal{D}$, which is then stored in
working memory. The working memory is refreshed iteratively based on the
dialogue’s content of current iteration—this is the summarizing step. The two-
step rehearsal process is facilitated by a coordinator with powerful natural
language understanding abilities. Next, the information stored in working
memory is evaluated to determine whether it needs to be stored in Short-Term
Memory (STM) or Long-Term Memory (LTM) based on the frequency of access by the
Executive Process. This dual-process is illustrated in Figure 2 by the green-
box and the three-colored lines. The detailed workings of DPeM are as follows:

#### Memory $\mathcal{M}$:

As prescribed,$\mathcal{M}$ does not consist of a single type of memory.
Instead, it comprises different types of memory that store and access
information in their own way, while working together for better knowledge
management through dual-process. Intuitively, there will be a vast amount of
information that needs to be registered when learning something new and not
all those information will be stored directly and entirely into memory.
Instead, a working memory acts as a buffer memory to register and filter
information so that only the relevant information enters STM, while the rest
is dropped. Short-term memory refers to a limited space that holds a small
amount of knowledge in an active, quickly accessible state. Long-term memory
stores knowledge transited from STM for a longer period. Being aware of the
differences in information storage and access, our memory mechanism enhances
LLM by incorporating knowledge from both user-specific and common-sense
perspectives. All the three types of memory are in the form of a growing table
which support different operations but work collaboratively as shown in Table
1.

#### Rehearsal Process

refers to the process of obtaining and refreshing information so that the
relevant information can be filtered and stored as knowledge into memory.
Learning aims at gathering information from the dialogue’s content of the
current iteration. Inspired by how humans take notes when learning something
new, we leverage $\mathcal{C}$, which takes all the necessary notes for each
iteration’s dialogue. Formally, by passing $d_{i}\in\mathcal{D}$ to
$\mathcal{C}$ iteratively, we obtain some notes denoted as
$nts=\mathcal{C}(d_{i})$ which will be recorded into working memory.
Typically, not all the notes will be practical which makes the summarizing
step indispensable within the DPeM mechanism. Summarizing steps further by
purifying the relevant notes from $nts$ and pass them into STM. Specifically,
the coordinator will determine if the $nt\in nts$ is relevant or not and store
the useful $nt^{+}$ as knowledge item by item and the stored knowledge is
delivered to STM and is denoted as $K=[k_{0},...,k_{m}]$ where
$k_{i}=nt_{i}^{+}$.

Type | Refresh | Storage | Sup. Lookup?
---|---|---|---
$\mathcal{M}_{working}$ | Each Iteration | Limited | ✗
$\mathcal{M}_{STM}$ | Certain Rounds | Limited | ✓
$\mathcal{M}_{LTM}$ | Never | Unlimited | ✓
Table 1: Comparison among three types of memory.

#### Executive Process

mainly focuses on manipulating and memorizing the knowledge produced from
rehearsal process. The main objective of DPeM is to process and store
information based on its importance level and user-specific needs, an aspect
that previous works have paid little attention to. Specifically, the filtered
knowledge $k$ will firstly be categorized as two types: Common-sense
Knowledge, User-Specific Knowledge and then be converted into the STM in the
form of key (type) - value ($k_{i}$) pair. As the learning iteration
progresses, a flag table $ft$ is used to keep track of the frequency of
appearance for each $k_{i}$. When the frequency reaches a predetermined
threshold $\theta$, the $k_{i}$ is transferred to LTM. Notably, STM is
refreshed periodically after certain rounds (working memory is refreshed after
each iteration) while LTM typically only accepts new $k_{i}$ entries. The
final memory structure consists of three parts: Working Memory, STM, and LTM.
Working Memory serves as a buffer for storing newly detected information, STM
stores relevant and recent knowledge, and LTM provides longer-term access to
frequently visited knowledge from STM. Through the collaboration of these
three types of memory, along with the dual-process approach, DPeM provides a
more powerful memory system to further support personalized LLM.

### MaLP Frame

#### Memory Generation

Memory can provide latent knowledge from personal historical stream which can
further be neglected as prompts to assist LLM for producing desired responses
regarding new queries. Attributed to our proposed DPeM mechanism, the memory
generation module can produce a well-organized memory which can support
different storage and lookup operations in terms of information features as
can be seen in Fig. 2. Given dialogues $\mathcal{D}$, the memory formation can
be described as follows:

$\begin{split}\mathcal{M}_{working}&=\\{nt_{0},...,nt_{i},...\\},\\\
\mathcal{M}_{STM}&=\\{...,k\\_type:k_{j},...\\},\\\
\mathcal{M}_{LTM}&=\\{...,\ k\\_type:k_{f},\ ...\\},\\\
\mathcal{M}&=[\mathcal{M}_{working},\mathcal{M}_{STM},\mathcal{M}_{LTM}]\\\
\end{split}$ (1)

where $nt_{i}=\mathcal{C}(d_{i})$, $k_{j}=nt_{i}^{+}$, $k_{f}$ denotes
frequently visited $k_{j}$ from $\mathcal{M}_{STM}$. The comparison among
these three types of memory can be seen in Table 1.

#### Memory Utilization

However, relying solely on memory for achieving personalized LLMs still poses
challenges, as the quality of generated responses ultimately depends on the
understanding and generation ability of the LLM, even with memory-augmented
prompts. Therefore, fine-tuning the LLM to cater to user-specific needs
naturally becomes an option for enhancing LLM personalization. However,
traditional fine-tuning approaches often demand significant computational and
data resources, whereas our aim is to optimize the LLM’s response generation
in a user-friendly manner by leveraging previous dialogues. In this regard,
PEFT methods (Li and Liang 2021; Liu et al. 2022b, c) offer a solution by
achieving this objective with low resource consumption.

To tune the LLaMA model with a user’s previous dialogues and enable it to
generate user-favorable responses, we employ the Low-Rank Adaption (LoRA)
technique (Hu et al. 2021). With LoRA, we update a given pre-trained weight
matrix $W_{\Phi}\in\mathbb{R}^{d\times k}$ of LLaMA by incorporating a low-
rank decomposition $W_{\Phi}+\Delta W=W_{\Phi}+BA$. Here,
$B\in\mathbb{R}^{d\times r}$, $A\in\mathbb{R}^{r\times k}$, and the rank
$r\ll\min(d,k)$. During the fine-tuning process, we randomly select a set
number of layers to implement LoRA, where $A$ and $B$ are trainable while
$W_{\Phi}$ remains frozen. This approach allows us to target user preferences
and adapt the LLaMA model accordingly.

Once all iterations are completed, we acquire a LoRA-tuned LLaMA model along
with a latent memory that caters to user-specific needs. Typically, when a new
query $x$ is received, the responding process is presented as:

$x\rightarrow\Phi\rightarrow y$ (2)

however, by using MaLP, the process is refined as:

$\begin{split}&p=Retriever(x),\\\ &x,p\rightarrow\hat{\Phi}\rightarrow
y\end{split}$ (3)

where $p$ is the prompt retrieved from $\mathcal{M}$, $Retriever$ is a
function that can retrieve knowledge from $\mathcal{M}$ in terms of query $x$
and $\hat{\Phi}$ is LoRA-tuned LLaMA. The utilization process is denoted in
orange lines as can be seen in Figure 2.

#### Components

In the MaLP framework, several key components actively engage in memory
generation and utilization to ensure efficient and collaborative operations:
Coordinator $\mathcal{C}$: $\mathcal{C}$ plays a pivotal role in the learning
and summarizing stage which involves deriving information from dialogue
contexts and purifying knowledge from learned information (Xu et al. 2023).
Thus we resort to a powerful tool (e.g., ChatGPT) that is capable of
understanding the long dialogue and performing summarization and judgement(Xu
et al. 2023).
Retriever $\mathcal{R}$: Retrieval is the process by which the retriever
accesses stored knowledge. However, since the memorized knowledge differs
between STM and LTM, their retrieval processes also differ. STM is retrieved
in the order in which it is stored, while LTM is retrieved through
association(e.g., recalling a past mistake by recognizing its
similarity(Kahneman 2011)). To address this, we have designed two retrievers:
a closest-match retriever, $\mathcal{R}{c}$, for STM retrieval, and a
semantic-match retriever, $\mathcal{R}{s}$, for LTM retrieval.
$\mathcal{R}_{c}$ aims to find the knowledge stored in STM that is closest to
the query in terms of Levenshtein distance, which indicates the minimum number
of deletions, insertions, or substitutions required to transform string s into
string t (e.g., $lev(^{\prime}test^{\prime},^{\prime}tent^{\prime})=1$ since
only one step ${}^{\prime}s^{\prime}->^{\prime}n^{\prime}$ is needed).
However, since the retrieval process for LTM is fast and unconscious, we have
chosen to train an encoder to obtain semantic embeddings and retrieve
knowledge in LTM based on cosine similarity (Madaan et al. 2022).

## Data

### Data Construction222Data and the construction code will be released here

Attributes | Value
---|---
Num of User | 60
Avg. Rounds | 182
Avg. Length | 877
Num of Dialogue | 10,920
Num of Utterance | 131,040
Table 2: Statistics of dataset

Existing dialogue datasets often lack awareness of the importance of
penalization, while recent works like Xu et al. (2023) have explored the
capability of LLMs to generate high-quality chat corpora. In light of this, we
propose injecting user profiles into the dialogue generation process using
self-chat simulations within real-world conversational scenarios.
Specifically, we focus on medical scenarios as they typically involve
dialogues between patients and doctors, encompassing a wealth of common-sense
information (e.g., Tylenol can alleviate fever) and personal details (e.g.,
chronic diseases, dialogue preferences). These scenarios allow us to emphasize
the significance of memory and penalization, respectively. To obtain
personalized dialogues, one straightforward method is to incorporate user’s
profile into a language model prompt. We first derive the patient’s profile
including personal information, symptoms and dialogue preference from the
publicly available medical corpus (Chen et al. 2020) and then follow Xu et al.
(2023)’s work using self-chat to guide powerful chat models (e.g., ChatGPT)
simulate high-quality dialogues. The difference is that we endow the patient’s
and doctor’s profile to the chat model at the beginning of conversation
simulation. Further, we prompt the powerful chat model to produce follow-up
dialogues related to the same symptom, new symptoms etc. to obtain historical
information. The whole construction pipeline and detailed statistics can be
seen in Figure 3 and Table 2, respectively.

Figure 3: Details of data collection process: we first derive patient’s
profile from public medical corpus and then endow the patient’s profile to a
powerful chat model. Assistant role (e.g., doctor) will be simulated
independently using the same chat model and thus we could collect the
historical dialogues via self-chat between these two roles.

### Safety and Evaluation

Safety Unlike most dialogue data generated by chatting with human, our dataset
does not rely on human feedback to suppress unwanted content (e.g., incorrect
medicine suggestions) and instead we resort to providing explicit prompts that
can steer the generation behaviors. While we have tested the default prompts,
it can still be risky to have the guidance missed by changing the prompts.
Evaluation To further assess our dataset, two master students with medical
background are hired for evaluating the quality of the dataset by identifying
any dirty content and safety issues on 100 random samples333Quality was scored
based on the presence of hallucinations, irrelevant content, etc. Each
criterion resulted in a deduction of one point, with a total of 6 points.
Safety evaluation focused on identifying profanity, inappropriate suggestions,
etc., and any presence of safety issues was indicated by answering ”yes.” For
more details, please refer to the data link.. The average quality score was
found to be 5.27, and the safety ratio, indicating the proportion of instances
without safety issues, was 94%.

## Experiments

### Setup

To train our model, we leverage the AdamW optimizer(Loshchilov and Hutter
2018) with a learning rate of $5\times 10^{-5}$ and also a linear warm-up
scheduler initialized with 10% of the total training steps as warm-up steps
and a weight decay of $10^{-4}$ to avoid over-fitting. The LoRA’s rank of
update matrices is set as 8 and the scaling factor alpha is 32. To accommodate
the task of requirements, we set the maximum length of input and output to
1024 and 2048 tokens, respectively. All implementations are conducted with
Pytorch(Paszke et al. 2017), PEFT(Mangrulkar et al. 2022) and
Transformers(Wolf et al. 2020) on a computation node configured with a 256G
CPU and two 32G Tesla V100 GPUs.

### Baselines

Considering the contributions of our work, we opt to compare our DPeM and MaLP
with three different configurations: Standard, with dict-based Mem(Madaan et
al. 2022) and with LoRA(Hu et al. 2021) in terms of two current SOTA LLMs as
base models: GPT3.5 and LLaMA-7B(Touvron et al. 2023).

### Tasks and Metrics

We adopt the evaluation methods proposed by Salemi et al. (2023) and Wang et
al. (2023) to assess the performance of our personalized LLM model on three
tasks:
Question Answering (QA) \- We evaluate the effectiveness by posing user-
relevant/knowledge-relevant questions to the model and comparing its generated
answers with the user profile and memory. The ROUGE-1 and ROUGE-L metrics are
used for evaluation;
Preference Classification \- We also assess personalization by prompting the
trained model to select the user’s dialogue preference from a pre-defined set
and measure performance using the Accuracy metric;
Response Generation \- In addition to empirical results, we evaluate the
quality of responses generated by the trained LLM for new queries from the
same user in terms of content and personalization. To do so, we use the
scoring method proposed by Wang et al. (2023) and calculate the Win Rate
between different settings and standard generation of the base LLM.
Furthermore, we conduct human evaluation to validate the alignment of this
automatic scoring schema with human judgments.

### Comparative Study

Model | Type | Profile QA | Knowledge QA | Pref. Classification | Response Generation
---|---|---|---|---|---
ROUGE-1 | ROUGE-L | ROUGE-1 | ROUGE-L | Accuracy % | Win Rate %
GPT3.5 | Standard | 32.07 | 30.81 | 35.62 | 31.78 | 36.31 | -
w Mem | 34.93 | 34.27 | 40.19 | 38.27 | 41.73 | 80.91
w DPeM | 40.81 | 38.78 | 40.87 | 39.51 | 47.72 | 86.60
LLaMA-7B | Standard | 21.41 | 19.82 | 25.01 | 23.69 | 21.42 | -
w Mem | 21.90 | 20.44 | 32.90 | 31.17 | 21.15 | 78.41
w DPeM | 22.37 | 20.97 | 35.07 | 33.98 | 33.06 | 84.60
w LoRA | 30.89 | 29.66 | 34.90 | 33.60 | 61.05 | 72.01
w MaLP | 35.59 | 33.91 | 36.91 | 36.37 | 69.95 | 91.53
Table 3: The main results on different tasks.

Table 3 presents the main evaluation results for Profile/Knowledge QA,
Preference Classification, and Response Generation tasks. The addition of
memory improves the performance of both GPT3.5 and LLaMA-7b LLMs compared to
the standard setting, as it provides additional knowledge prompts to enhance
the LLM’s understanding of user queries. However, our novel DPeM exhibits
superior performance in assisting LLMs. When combined with GPT3.5 as the base
LLM, DPeM outperforms the dict-based memory setting (Madaan et al. 2022) with
relative improvements of 13.16% and 3.24% in ROUGE-L scores for profile QA and
knowledge QA tasks, respectively. Additionally, DPeM demonstrates better user-
specific assistance by achieving a 14.35% increase in classification accuracy
compared to dict-based memory and a 7.03% higher win rate for response
generation. Similarly, when configured with LLaMA-7b as the base LLM, DPeM
achieves relative improvements of 2.59% and 9.02% in profile and knowledge QA
tasks, respectively, along with 56.31% and 7.89% enhancements in classifying
user preferences and generating personalized responses. These improvements can
be attributed to the novel dual-process schema of DPeM, where the rehearsal
process refreshes and rewrites knowledge to reduce the risk of retrieving
irrelevant information, and the executive process memorizes knowledge in a
distinguish-aware manner, leading to more effective retrieval.

One interesting thing we observed is that despite the improvements made by
DPeM, it’s still insufficient for acquiring user-specific needs. However, by
leveraging PEFT (i.e., LoRA) as can be seen in the results of QA tasks using
LLaMA-7b as the base, DPeM achieves a greater improvement on knowledge QA than
profile QA while using LoRA achieves a greater improvement on profile QA.
Moreover, LoRA helps LLM to know user preference better as it boosts the
accuracy of classifying user preference by 39.63% while using DPeM solely
improves the accuracy by 11.64% compared with standard setting. However,
despite the user-specific need detected by using LoRA, we notice that using
LoRA solely is not comparable with using memory on response generation which
indicates the importance of memory in our whole MaLP.

By combining DPeM and LoRA into a unified framework, our MaPL approach can
effectively incorporate both user-specific needs and knowledge detected from
previous dialogue history, resulting in the best performance across all three
evaluation tasks compared to other configurations. One more notable thing is
that the intricate nature language understanding and generation of LLaMA-7b is
not comparable with GPT3.5. However, by using MaLP, LLaMA-7b achieves
promising results compared with GPT3.5 towards responding user in a user-
favorable manner. These findings further validate the effectiveness and
superiority of our novel DPeM mechanism as well as the unified MaLP frame.

### Response Quality Study

Figure 4: The quality of generated response increases with the number of
historical dialogues.

In addition to the main comparisons with standard settings and previous
efforts, we conducted further experiments to explore the roles of different
modules as the historical information increases. As shown in Figure 4, our
MaLP approach consistently improves the quality of generated responses and
outperforms other configurations. Notably, the quality of generated responses
fluctuates in the first 20 rounds as depicted in Figure 4 which we attribute
this to the accumulation of knowledge during the initial rounds. Moreover, as
the number of dialogue rounds increases, the oscillation of the dict-based
memory is larger compared to that of DPeM. This indicates that our DPeM
mechanism effectively reduces the chances of incorrect knowledge retrieval
through its dual-process schema. These findings further confirm the stability
and effectiveness of MaLP.

### Case Study

Figure 5: A case study showing the quality of generated response in terms of
different settings. Scores are evaluated by human.

We further conduct a case study to show the quality of generated response
under the assistance of MaLP compared with other baselines. As Figure 5 shows,
given the background and the new query, our MaLP receives the highest score
since it takes both the user historical knowledge (i.e. diabetes) including
preference (i.e. prefer concise suggestions) learned by peft and the common-
sense knowledge (i.e. keeping skin moisture etc.) into consideration for
response generation. While standard settings only generate response in a
general manner and the dict-based memory method only relies on the knowledge
stored in its memory which lacks the aware of user-specific needs, thus
leading inferior scores. In tandem, our frame along with the novel DPeM and
PEFT training enables LLMs to provide more engaging dialogue experience
towards user-specific needs.

### Human Judgement

To validate the alignment of our automatic scoring schema with human
judgements, we follow the work of Wang et al. (2023) to conduct point-wise
evaluation. Specifically, two well-educated master students are hired and 100
response pairs are sampled (i.e., responses generated by standard setting and
MaLP using LLaMA-7b). Then we ask the students to indicate which response is
better by selecting 1(win), 0(tie) and -1(lose) for each pair. Next, we
calculate the Pearson Correlation Coefficient (P.C) and also the accuracy
between human scores and automatic scores. The P.C of 0.72 and the accuracy of
84% together indicate the feasibility and high confidence of our evaluation
method.

## Conclusion & Future Work

In summary, we proposed MaLP which integrates a novel dual-process enhanced
memory mechanism and a peft approach to enhance LLMs with awareness of user-
specific needs. This straightforward yet effective endeavor enables
personalized LLMs while maintaining low resource consumption. Additionally,
our innovative data construction method facilitated the creation of high-
quality dialogue datasets in the medical domain, and its applicability can be
easily extended to other scenarios such as legal consultation, encouraging
further exploration in LLM personalization. The extensive experiments and
human judgment tests conducted validate the effectiveness of our work.

Despite the success, our method remains in the prototype stage. Next, we will
focus on injecting expert knowledge into memory and enabling its online-
learning fashion.

## Related Work

Memory-Augmented LLM refers to apply a memory that contains user feedback from
previous mistakes and by prepending or postpending the new input query with
the stored feedback, the output of LLM can be improved(Ouyang et al. 2022).
Efforts have been made in terms of the usage of memory. Tandon et al. (2021)
first proposed to leverages a corrector that can correct the model’s output in
terms of the similar mistake stored in the memory previously. However, this
method aims to repair the wrong output while Madaan et al. (2022) argued that
the stored experience can be used to avoid incorrect output by
prepending/postpending the feedback to the new query. Another usage of memory
is to include the memory into a learning frame such as self-learning or
teacher-student paradigm so that the LLM can learn by iterative
refinement(Madaan et al. 2023; Dalvi, Tafjord, and Clark 2022). In tandem, the
key for better usage of memory is to equip powerful retrievers(Guu et al.
2020; Lewis et al. 2020; Yuan et al. 2022) which we argue that memory
structure can be improved. The main difference between our work and the
previous work is that our work refine the memory structure, instead we design
a close-to-real memory mechanism that can better identify and retrieve
information for better LLM output and along with PEFT techniques to achieve
personalized LLM.
Personalized LLM has seen increasing attentions since it can provide tailored
experience that aligns with their user’s expectations in terms of their
needs(Salemi et al. 2023). Previous works focused on identifying user
preferences by Ceteris Paribus(CP)-nets(Asher, Bonzon, and Lascarides 2010).
Unfortunately , this kind of methods suffer from its limited ability of
natural language understanding. As LLMs emerged, prompt-based methods attempt
to design in-depth prompts such as chain-of-thoughts prompts that can guide
LLM to produce desired output with aware of user status and context
content(Wang et al. 2023; Wu et al. 2023; Aher, Arriaga, and Kalai 2023).
Another way resorts to enhancing LLMs with aware of user information and fine-
tuning LLMs to generate responses towards user-specific needs. For example,
Korbak et al. (2023); Salemi et al. (2023); Xu et al. (2023) tried to inject
user profile information in the pre-training stage and fine-tune the LLM in
terms of the learned preferences from user. Unfortunately, fully trained LLMs
can be too time- and resource-consuming, thus we propose to leverage
parameter-efficient fine-tuning (PEFT) techniques and along with our novel
memory mechanism for learning user preferences. In tandem, the most part that
differentiates our work with previous work is that we are the first that
propose to design a close-to-real memory mechanism instead of a dict-based
structure and along with PEFT techniques to achieve and enhance personalized
LLMs.

## Acknowledgments

## References

  * Aher, Arriaga, and Kalai (2023) Aher, G. V.; Arriaga, R. I.; and Kalai, A. T. 2023.  Using large language models to simulate multiple humans and replicate human subject studies.  In _International Conference on Machine Learning_ , 337–371. PMLR.
  * Asher, Bonzon, and Lascarides (2010) Asher, N.; Bonzon, E.; and Lascarides, A. 2010.  Extracting and modelling preferences from dialogue.  In _International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems_ , 542–553. Springer.
  * Bender et al. (2021) Bender, E. M.; Gebru, T.; McMillan-Major, A.; and Shmitchell, S. 2021.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, 610–623. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450383097.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33: 1877–1901.
  * Chen et al. (2020) Chen, S.; Ju, Z.; Dong, X.; Fang, H.; Wang, S.; Yang, Y.; Zeng, J.; Zhang, R.; Zhang, R.; Zhou, M.; Zhu, P.; and Xie, P. 2020.  MedDialog: a large-scale medical dialogue dataset.  _arXiv preprint arXiv:2004.03329_.
  * Chowdhery et al. (2022) Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H. W.; Sutton, C.; Gehrmann, S.; et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Dalvi, Tafjord, and Clark (2022) Dalvi, B.; Tafjord, O.; and Clark, P. 2022.  Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 9465–9480.
  * Guu et al. (2020) Guu, K.; Lee, K.; Tung, Z.; Pasupat, P.; and Chang, M. 2020.  Retrieval augmented language model pre-training.  In _International conference on machine learning_ , 3929–3938. PMLR.
  * Hu et al. (2021) Hu, E. J.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al. 2021.  LoRA: Low-Rank Adaptation of Large Language Models.  In _International Conference on Learning Representations_.
  * Kahneman (2011) Kahneman, D. 2011.  _Thinking, fast and slow_.  macmillan.
  * Korbak et al. (2023) Korbak, T.; Shi, K.; Chen, A.; Bhalerao, R. V.; Buckley, C.; Phang, J.; Bowman, S. R.; and Perez, E. 2023.  Pretraining language models with human preferences.  In _International Conference on Machine Learning_ , 17506–17533. PMLR.
  * Lewis et al. (2020) Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.; Yih, W.-t.; Rocktäschel, T.; et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33: 9459–9474.
  * Li and Liang (2021) Li, X. L.; and Liang, P. 2021.  Prefix-Tuning: Optimizing Continuous Prompts for Generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 4582–4597. Online: Association for Computational Linguistics.
  * Liu et al. (2022a) Liu, H.; Tam, D.; Muqeeth, M.; Mohta, J.; Huang, T.; Bansal, M.; and Raffel, C. A. 2022a.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  _Advances in Neural Information Processing Systems_ , 35: 1950–1965.
  * Liu et al. (2022b) Liu, X.; Ji, K.; Fu, Y.; Tam, W.; Du, Z.; Yang, Z.; and Tang, J. 2022b.  P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 61–68. Dublin, Ireland: Association for Computational Linguistics.
  * Liu et al. (2022c) Liu, X.; Ji, K.; Fu, Y.; Tam, W. L.; Du, Z.; Yang, Z.; and Tang, J. 2022c.  P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks.  arXiv:2110.07602.
  * Loshchilov and Hutter (2018) Loshchilov, I.; and Hutter, F. 2018.  Decoupled Weight Decay Regularization.  In _International Conference on Learning Representations_.
  * Madaan et al. (2022) Madaan, A.; Tandon, N.; Clark, P.; and Yang, Y. 2022.  Memory-assisted prompt editing to improve GPT-3 after deployment.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 2833–2861.
  * Madaan et al. (2023) Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; et al. 2023.  Self-refine: Iterative refinement with self-feedback.  _arXiv preprint arXiv:2303.17651_.
  * Mangrulkar et al. (2022) Mangrulkar, S.; Gugger, S.; Debut, L.; Belkada, Y.; and Paul, S. 2022.  PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods.  https://github.com/huggingface/peft.
  * Ouyang et al. (2022) Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35: 27730–27744.
  * Paszke et al. (2017) Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A. 2017.  Automatic differentiation in PyTorch.
  * Roediger and McDermott (1995) Roediger, H. L.; and McDermott, K. B. 1995.  Creating false memories: Remembering words not presented in lists.  _Journal of experimental psychology: Learning, Memory, and Cognition_ , 21(4): 803.
  * Salemi et al. (2023) Salemi, A.; Mysore, S.; Bendersky, M.; and Zamani, H. 2023.  LaMP: When Large Language Models Meet Personalization.  _arXiv preprint arXiv:2304.11406_.
  * Tandon et al. (2021) Tandon, N.; Madaan, A.; Clark, P.; and Yang, Y. 2021.  Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback.  _arXiv preprint arXiv:2112.09737_.
  * Touvron et al. (2023) Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E.; and Lample, G. 2023.  LLaMA: Open and Efficient Foundation Language Models.  _arXiv preprint arXiv:2302.13971_.
  * Wang, Ustun, and Calmon (2019) Wang, H.; Ustun, B.; and Calmon, F. 2019.  Repairing without retraining: Avoiding disparate impact with counterfactual distributions.  In _International Conference on Machine Learning_ , 6618–6627. PMLR.
  * Wang et al. (2023) Wang, H.; Wang, R.; Mi, F.; Wang, Z.; Xu, R.; and Wong, K.-F. 2023.  Chain-of-thought prompting for responding to in-depth dialogue questions with LLM.  _arXiv preprint arXiv:2305.11792_.
  * Wolf et al. (2020) Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Scao, T. L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. M. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 38–45. Online: Association for Computational Linguistics.
  * Wu et al. (2023) Wu, J.; Antonova, R.; Kan, A.; Lepert, M.; Zeng, A.; Song, S.; Bohg, J.; Rusinkiewicz, S.; and Funkhouser, T. 2023.  Tidybot: Personalized robot assistance with large language models.  _arXiv preprint arXiv:2305.05658_.
  * Xu et al. (2023) Xu, C.; Guo, D.; Duan, N.; and McAuley, J. 2023.  Baize: An open-source chat model with parameter-efficient tuning on self-chat data.  _arXiv preprint arXiv:2304.01196_.
  * Yuan et al. (2022) Yuan, X.; Wang, T.; Wang, Y.-H.; Fine, E.; Abdelghani, R.; Lucas, P.; Sauzéon, H.; and Oudeyer, P.-Y. 2022.  Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation.  _arXiv preprint arXiv:2209.11000_.


# A Lightweight Structure Aimed to Utilize Spatial Correlation for Sparse-View
CT Reconstruction

Yitong Liu
<EMAIL_ADDRESS>
&Ken Deng
<EMAIL_ADDRESS>
&Chang Sun
<EMAIL_ADDRESS>
&Hongwen Yang
<EMAIL_ADDRESS>

###### Abstract

Sparse-view computed tomography (CT) is known as a widely used approach to
reduce radiation dose while accelerating imaging through lowered projection
views and correlated calculations. However, its severe imaging noise and
streaking artifacts turn out to be a major issue in the low dose protocol. In
this paper, we propose a dual-domain deep learning-based method that breaks
through the limitations of currently prevailing algorithms that merely process
single image slices. Since the scanned object usually contains a high degree
of spatial continuity, the obtained consecutive imaging slices embody rich
information that is largely unexplored. Therefore, we establish a cascade
model named LS-AAE which aims to tackle the above problem. In addition, in
order to adapt to the social trend of lightweight medical care, our model
adopts the inverted residual with linear bottleneck in the module design to
make it mobile and lightweight (reduce model parameters to one-eighth of its
original) without sacrificing its performance. In our experiments, sparse
sampling is conducted at intervals of 4°, 8° and 16°, which appears to be a
challenging sparsity that few scholars have attempted. Nevertheless, our
method still exhibits its robustness and achieves the state-of-the-art
performance by reaching the PSNR of 40.305 and the SSIM of 0.948, while
ensuring high model mobility. Particularly, it still exceeds other current
methods when the sampling rate is one-fourth of them, thereby demonstrating
its remarkable superiority.

##  1 Introduction

Over the last few decades [1, 2], X-ray Computed Tomography (CT) has
demonstrated its prominent practical value and wide range of applications
including clinical diagnosis, safety inspection and industrial detection [3].
Especially in the past year, due to the global spread of the Corona Virus
Disease 2019 (COVID-19), the term CT has become well-known to the public as an
essential auxiliary technology. However, the radiation dose brought by CT has
a nonnegligible side effect on the human body. Since it has a latent risk of
inducing cancers, radiation dose reduction is becoming more and more crucial
under the principle of ALARA (as low as reasonably achievable) [4, 5, 6, 7].

Generally speaking, there are two approaches to reduce radiation dose. The
approach of tube current (or voltage) reduction [8, 9] lowers the x-ray
exposure in each view but suffers from the increased noise in projections.
Although the approach of projection number reduction [10, 11] (also known as
sparse-view CT) can avoid the former problem and realize the additional
benefit of accelerated scan and calculation, it leads to severe image quality
degradation of increased streaking artifacts brought by its missing
projections. In this paper, we focus on effectively repairing and
reconstructing sparse-view CT so as to acquire high-quality CT images.

Sparse-view CT reconstruction has always been a classic inverse problem which
has attracted wide attention [12]. In the past few decades, iterative
reconstruction methods have become the dominant approach to solve inverse
problems [13, 14, 15, 16]. With the advent of compressed sensing [17] and its
related regularizers, the quality of reconstructed images has been improved to
a certain extent. One of the most typical regularizers is the total variation
(TV) method, algorithms based on which include TV-POCS [18], TGV method [19],
SART [13] and SART-TV [20] etc. In addition, dictionary learning is also
commonly used as a regularizer. For example, [21] constructs a global
dictionary and an iterative adaptive dictionary to solve the problem of low-
dose CT reconstruction.

In recent years, with the improvement of computing power, there comes a rapid
growth in deep learning [22]. Subsequently, neural networks have been widely
applied in image analysis tasks, such as image classification [23], image
segmentation [24, 25, 26], especially inverse problems in image
reconstruction, such as artifacts reduction [27, 28], denoising [29] and
inpainting [30]. Since GAN (Generative Adversarial Networks) was designed
elaborately by Goodfellow in 2014 [31], it has been adopted in many image
processing tasks due to its prominent performance in realistically predicting
image details. Therefore, GANs are also naturally applied to improving the
quality of low-dose CT images [32, 33, 34]. In addition, Ye et al. explored
the relationship between deep learning and classical signal processing methods
in [35], explained the reason why deep learning can be employed in imaging
inverse problems, and provided a theoretical basis for the application of deep
learning in low-dose CT reconstruction.

Some researchers adopt deep learning-based architectures to complement and
restore the limit-view Radon data [36, 37, 32, 38, 39, 40]. Dong et al. [36]
used U-Net [25] to predict the missing Radon data, then reconstruct it to the
image through FBP [41]. Jian Fu et al. [37] built a network that involves the
tight coupling of the deep learning neural network and DPC-CT (Differential
phase-contrast CT) reconstruction algorithm in the domain of DPC projection
sinograms. The estimated result is a complete phase-contrast projection
sinogram. Rushil Anirudh et al. established CTNet [38], a system of 1D and 2D
convolutional neural networks, which operates on the limited-view sinogram to
predict the full-view sinogram, and then fed it to the standard analytical and
iterative reconstruction algorithms to obtain the final result.

Other researchers carried out post-processing on reconstructed images with
deep learning models, so as to remove the artifacts and noises for upgrading
the quality of these images[42, 43, 44, 33, 45, 46, 47]. In 2016, a deep
convolutional neural network [44] was proposed to learn an end-to-end mapping
between the FBP and artifact-free images. In 2018, Yoseob Han and Jong Chul Ye
designed a dual frame and tight frame U-Net [42] which satisfies the frame
condition and performs better for recovery of high frequency edges in sparse-
view CT. In 2019, Xie et al. [33] built an end-to-end cGAN model with joint
function used for removing artifacts from limited-angle CT reconstruction
images. In 2020, Wang et al. [45] developed a limited-angle TCT image
reconstruction algorithm based on U-Net, which could suppress the artifacts
and preserve the structures. Experiments have shown that U-Net-like structures
are efficacious for image artifacts removal and texture restoration [35, 36,
42, 45, 47] .

Since neural networks are capable of predicting unknown data in the Radon and
image domains, a natural idea is to combine these two domains [48, 49, 34, 50,
51, 52] to acquire better restoration results. Specifically, it first
complements the Radon data, and then remove the residual artifacts and noises
on images converted from the full-view Radon data. In 2018, Zhao et al.
proposed SVGAN [34], an artifacts reduction method for low-dose and sparse-
view CT via a single model trained by GAN. In 2019, Liang et al. [49] proposed
a comprehensive network combining projection and image domains. The projection
estimation network is based on Res-CNN structure, and the image domain network
takes the advantage of U-Net. In 2020, Zhu et al. designed ADAPTIVE-NET [50]
to conduct joint denoising on the acquired sinogram and the reconstructed CT
image, while reconstructing CT image in an end-to-end manner. In the past
three years, experiments have proved that this sort of two-stage algorithm is
quite conducive to image quality improvement.

All the current mainstream methods mentioned above make us notice that they
solely process on each single CT image, while neglecting the solid fact the
scanned object is always highly continuous. Consequently, there is abundant
spatial information lies in these obtained consecutive CT images, which is
largely left to be exploited. This enlightens us to propose a novel cascade
model called LS-AAE (Lightweight Spatial Adversarial Autoencoder) that mainly
focus on availably utilizing the spatial information between greatly
correlated images. It has been proved in our experiments that this sort of
structure design manages to efficaciously remove streaking artifacts in
sparse-view CT images, and outruns other prevailing methods with its
remarkable performance.

It is the social trend now to make healthcare mobile and portable. In lots of
deep learning-based methods, however, scholars improve accuracy at the expense
of sacrificing computing resources. Such computational complexity usually
exceeds the capabilities of many mobile and embedded applications. This paper
adopts the inverted residual with linear bottleneck [53] in the module design
to propose a mobile structure that reduce model parameters to one-eighth of
its original without sacrificing its performance.

Although enhancing the sparsity of sparse-view CT can bring benefits of
accelerated scanning and related calculations, it will cause additional
imaging damage. Balancing image quality and X-ray dose level has become a
well-known trade-off problem. Thus, in order to explore the limit of sparsity
in sparse-view CT reconstruction, we conduct sparse sampling at intervals of
4°, 8° and most importantly, 16°. Even under such sampling sparsity, our model
can still exhibit its remarkable robustness and the state-of-the-art
performance.

We introduce our proposed method exhaustively in Section II, the experimental
results and corresponding discussion are described in section III, and
conclusion is stated in section IV.

##  2 Methods

###  2.1 Preliminaries

####  2.1.1 Utilize Spatial Information for Artifact Removal

As is known to all, consecutive CT images usually contain high spatial
coherency since the scanned object is usually spatially continuous. On account
of that, we can imagine these CT images as adjacent frames in a video which
contains much more information than a still image. This high correlation
within the sequence of images can improve the performance of artifact removal
from two aspects. Firstly, the extension of search regions from two-
dimensional image neighborhoods to three-dimensional spatial neighborhoods
provide extra information which can be used to denoise the reference image.
Secondly, using spatial neighbors helps to reduce streaking artifacts as the
residual error in each image is correlated.

Also, we cannot help but notice that the task of artifact removal between
consecutive images is similar to video denoising. Therefore, after
investigating lots of research work on video denoising [54, 55, 56, 57, 58,
59, 60, 61, 62, 63], we find out that current state-of-the-art methods lay
lots of emphasis on motion estimation due to the strong redundancy along
motion trajectories. To conclude, in order to more effectively remove
streaking artifacts from sparse-view CT images, we need to design a structure
that can not only look into the three-dimensional spatial neighborhood, but
also capture the motion between consecutive images.

####  2.1.2 Enhance Mobility of Neural Networks

In recent years, lots of research has been invested into tuning deep neural
networks to achieve an optimal balance between efficiency and performance.
Among them, depthwise separable convolutions [64] exhibits its extraordinary
capability and has gradually become an essential building block for numerous
lightweight neural networks [64, 65, 66]. It aims to decompose the standard
convolutional layer into two separate layers, namely the depthwise
convolutional layer and the pointwise convolutional layer. The former layer is
designed to perform lightweight filtering through employing a single
convolutional filter per input channel, the latter one conducts $1\times 1$
convolution to construct new features by computing linear combinations of
input channels.

For the standard convolutional layer with input tensor size $(c_{in},h,w)$,
kernel size $(c_{out},c_{in},k,k)$ and output tensor size $(c_{out},h,w)$, its
computational cost equals to $c_{in}\cdot h\cdot w\cdot(k^{2}\cdot c_{out})$.
However, in depthwise separable convolutions, the depthwise convolutional
layer has a computational cost of $c_{in}\cdot h\cdot w\cdot k^{2}$ since it
merely operates on a single input channel, and the pointwise convolutional
layer has a computational cost of $c_{in}\cdot h\cdot w\cdot c_{out}$.
Therefore, we only need a computational cost of $h\cdot w\cdot
c_{in}\cdot(k^{2}+c_{out})$ for depthwise separable convolutions, which is
almost the one-ninth ($k$ equals to 3 in our case) of the standard
convolution. Most importantly, depthwise separable convolutions manage to
lower the computational complexity to a large extent without sacrificing its
accuracy, which would make it perfect to be inserted into our module design.

###  2.2 Overall Structure

####  2.2.1 Structure Overview

Figure 1: Structure overview. The sparse-view Radon data $\mathbb{X}$ is first
sent to the neural network $F$ for completion, then the restored full-view
Radon data $\mathbb{X}^{\prime}$ is converted to the image $\mathbb{Y}$, which
is feed into the neural network $G$ for artifacts removal and we can finally
obtain the ideal high-quality image $\mathbb{Y}^{\prime}$.

We can learn from the universal approximation theorem [67] that multilayer
feedforward networks are capable of approximating various continuous
functions. This inspires us to think that neural networks can be used to learn
complex mappings that are difficult to solve through mathematical analysis.
Thus, in this paper, we utilize a deep learning-based structure that combines
the Radon domain and the image domain (Figure 1) to solve the task of sparse-
view CT reconstruction and inpainting.

Firstly, we want to make full use of the prior information in the Radon domain
by converting the sparse-view Radon data $\mathbb{X}$ to the full-view Radon
data $\mathbb{X}^{\prime}$ so as to complement the missing data in some
scanning angles. This process can be represented by the mapping:
$\mathbb{X}\xrightarrow{f}\mathbb{X}^{\prime}$ according to the universal
approximation theorem, where function $f$ can be approximated through our
proposed neural network $F$. After we obtain the full-view Radon data
$\mathbb{X}^{\prime}$, we transform it to the image $\mathbb{Y}$ through FBP.
Although the first stage manages to alleviate the severe imaging damage from
the original sparse-view CT image, there are still lots of streaking artifacts
existing in Y that need to be removed to acquire the high-quality restored
result $\mathbb{Y}^{\prime}$. We represent the restoration process into the
mapping: $\mathbb{Y}\xrightarrow{g}\mathbb{Y}^{\prime}$, where function $g$
can be approximated through our proposed neural network $G$. Through the above
two-stage structure that combines the Radon domain with the image domain, we
can finally get the ideal restored results.

####  2.2.2 Stage One: Data Completion in the Radon Domain

Figure 2: The diagram of our proposed L-AAE, which is composed of a L-AE and a
discriminator that help restore the image texture.

We first adopt linear interpolation to convert the original sparse-view Radon
data to full-view Radon data so as to satisfy the structural characteristics
of our proposed neural network, which requires the input and output images to
have the same resolution. Then we build a lightweight adversarial autoencoder
(L-AAE) in Figure 2 to restore the Radon data, the structure of its
autoencoder (L-AE) can be seen from Figure 3 and Table 1, which is composed of
the encoder and the decoder that are highly symmetrical.

Figure 3: The detailed structure of the L-AE. Input images are first feed into the encoder for feature extraction and then sent into the decoder for texture restoration, where skip connections are added to merge low-level features. Table 1: Parametric structure of the L-AE Layer |  $IC$ |  $OC$ |  Stride |  Input Size |  Output Size
---|---|---|---|---|---
Conv1 |  1 | 32 | 2 |  192$\times$512 |  96$\times$256
Block1 |  32 | 16 | 1 |  96$\times$256 |  96$\times$256
Block2_1 |  16 | 32 | 2 |  96$\times$256 |  48$\times$128
Block2_2 |  32 | 32 | 1 |  48$\times$128 |  48$\times$128
Block3_1 |  32 | 64 | 2 |  48$\times$128 |  24$\times$64
Block3_2 |  64 | 64 | 1 |  24$\times$64 |  24$\times$64
Block3_3 |  64 | 64 | 1 |  24$\times$64 |  24$\times$64
Block4_1 |  64 | 128 | 2 |  24$\times$64 |  12$\times$32
Block4_2 |  128 | 128 | 1 |  12$\times$32 |  12$\times$32
Block4_3 |  128 | 128 | 1 |  12$\times$32 |  12$\times$32
Block4_4 |  128 | 128 | 1 |  12$\times$32 |  12$\times$32
Trans Conv1 |  128 | 64 | 2 |  12$\times$32 |  24$\times$64
Block5_1 |  64+64(Concat) | 64 | 2 |  24$\times$64 |  24$\times$64
Block5_2 |  64 | 64 | 1 |  24$\times$64 |  24$\times$64
Block5_3 |  64 | 64 | 1 |  24$\times$64 |  24$\times$64
Trans Conv2 |  64 | 32 | 2 |  24$\times$64 |  48$\times$128
Block6_1 |  32+32(Concat) | 32 | 1 |  48$\times$128 |  48$\times$128
Block6_2 |  32 | 32 | 1 |  48$\times$128 |  48$\times$128
Trans Conv3 |  32 | 16 | 2 |  48$\times$128 |  96$\times$256
Block7 |  16+16(Concat) | 32 | 1 |  96$\times$256 |  96$\times$256
Block8 |  32+32(Concat) | 32 | 1 |  96$\times$256 |  96$\times$256
Trans Conv4 |  32 | 16 | 2 |  96$\times$256 |  192$\times$512
Conv9 |  16 | 1 | 1 |  192$\times$512 |  192$\times$512

We perform four down sampling in the encoder to obtain high-level semantic
features of the input image, which is initially downsampled through conv1
layer with a stride of 2, and the subsequent downsampling is separately
accomplished by the first building block of each unit. Each downsampling will
halve the height and width of the activation map and double the number of
channels.

As for the decoder, we correspondingly conduct four upsampling to restore the
texture of the input image. Deconvolution is adopted here for upsampling with
its kernel size and stride both equal to 2, so that each upsampling will
double the height and width of the activation map and halve the number of
channels. In addition, we add skip connections [68] between the encoder and
decoder feature maps of the same resolution. Since the final feature map of
the encoder has a relatively low resolution due to multiple downsampling, it
will have an undesirable effect on the restoration of the image texture in the
decoder. While the skip connection incorporates low-level features from the
encoder which have a high resolution and contain abundant detailed information
that will help to accurately restore the image texture. This sort of multi-
scale, U-Net-like architectures have been proved to be effective in processing
medical images.

The detailed structure of our building block can be seen from Figure 4, it
adopts the inverted residual with linear bottleneck referring to [53], each
block is composed of three convolutional layers. The first layer expands
(characterized by the expansion factor $exp$) a low-dimensional compressed
representation to high dimension with a kernel size of $1\times 1$. The
intermediate expansion layer adopts lightweight depthwise convolutions
mentioned above so as to significantly decreases the number of operations
(ops) and memory needed while sustaining the same performance. The last layer
projects the feature back to a low-dimensional representation with a linear
convolution like the first layer. All these layers are followed by batch
normalization [69] and ReLU [70] as the non-linear activation except for the
last layer that only followed by a batch normalization layer.

Figure 4: The detailed diagram of the building blocks in L-AAE, which adopt
the inverted residual with linear bottleneck.

In Figure 4, $IC$ and $OC$ stand for the input and output channel of building
blocks respectively. All convolutional layers in all building blocks have a
stride of 1 except for Block2_1, Block3_1 and Block4_1 that have a stride of 2
to conduct downsampling.Expansion factor $exp$ is 1 for Block1, Block7 and
Block8 to avoid large ops and memory cost, we set up $exp$ to be 3 for
Block5_1 and Block6_1, and every block expect these mentioned above have an
$exp$ of 6. Besides, shortcut connections are implemented in blocks that have
the same resolution between its input and output feature maps to enhance
information flow and also improve the ability of a gradient to propagate
across multiplier layers. We adopt $1\times 1$ convolution in shortcuts when
the number of channel in the input and output feature maps is different.

The discriminator in our L-AAE aims to strengthen model’s ability to restore
the detailed texture of images, its structure is almost the same as the
encoder above, except that its Block4_3 and Block4_4 have an $OC$ of 64 and 1
respectively. The output of Block4_4 is flattened and sent to sigmoid function
for probability prediction, which we average to get the final output that
represents the input image’s probability to be a real image. This novel
lightweight AAE enables us to acquire the well restored Radon data that are
complete in every scanning angle, and the computational cost is about 8 times
smaller than that of standard convolutions without sacrificing its accuracy.

####  2.2.3 Stage Two: LS-AAE – Image Inpainting through spatial information

After stage one, we transform the acquired full-view Radon data to images and
find out that, we successfully enrich the information in the Radon domain and
alleviate streaking artifacts from the original sparse-view CT imaging. Now in
stage two, we will mainly focus on removing artifacts, restoring image to an
ideal level. As mentioned above, we need a neural network that not only look
into the three-dimensional spatial neighborhood, but also capture the motion
between consecutive images, so as to efficaciously utilize the abundant
spatial information between consecutive images to remove artifacts from the
input image.

Generally speaking, motion estimation always brings an additional degree of
complexity that is adverse to model’s implementation in reality. It means that
we need a structure that can manage to deploy motion estimation without much
resource cost, we refer to [71] and its general structure appears to be a
cascaded two-step architecture that inherently embed the motion of objects.
Inspired by this, we propose a model named Lightweight Spatial Adversarial
Autoencoder (LS-AAE) which can be seen from Figure 5. It slightly modifies the
L-AE from Figure 3 as its inpainting block, details are shown in Table 1. The
replacement from 2D convolution to 3D convolution enables our model to look
into the three-dimensional spatial neighborhood for extra information.

Table 2: From 2D convolution to 3D convolution |  Layer |  $IC$ |  $OC$ |  Kernel Size |  Stride |  Padding
---|---|---|---|---|---|---
2D Convolution |  Conv1 |  1 |  16 |  (3,3) |  (2,2) |  (1,1)
3D Convolution |  Conv1_1 |  1 |  16 |  (3,3,3) |  (1,2,2) |  (1,0,0)
|  Conv1_2 |  16 |  32 |  (3,3,3) |  (2,1,1) |  (0,1,1)

As shown in Figure 5, five consecutive images
$\left\\{\mathbb{I}_{i-2},\mathbb{I}_{i-1},\mathbb{I}_{i},\mathbb{I}_{i+1},\mathbb{I}_{i+2}\right\\}$
are sent into the LS-AAE to restore the middle one. We firstly treat these
inputs as triplets of consecutive images
$\left\\{\mathbb{I}_{i-2},\mathbb{I}_{i-1},\mathbb{I}_{i}\right\\}$,
$\left\\{\mathbb{I}_{i-1},\mathbb{I}_{i},\mathbb{I}_{i+1}\right\\}$ and
$\left\\{\mathbb{I}_{i},\mathbb{I}_{i+1},\mathbb{I}_{i+2}\right\\}$, then
enter them into the Inpainting Blocks 1. Subsequently, we obtain the outputs
of these blocks and combine them into triplet
$\left\\{\mathbb{I}^{\prime}_{i-1},\mathbb{I}^{\prime}_{i},\mathbb{I}^{\prime}_{i+1}\right\\}$
which will be sent into Inpainting Block 2 to acquire the ultimate estimation
$\mathbb{I}^{\prime\prime}_{i}$ corresponding to the central image
$\mathbb{I}_{i}$. The LS-AAE digs deep into the three-dimensional space and
implicitly handles motion without any explicit motion compensation stage on
account of the traits of its architecture. Besides, the three Inpainting
Blocks in step one share the same weights so as to avoid memory cost. We also
add a discriminator in stage two to better restore the image texture, the
predicted image $\mathbb{I}^{\prime\prime}_{i}$ and its corresponding ground
truth (the full-view CT imaging) $\mathbb{I}^{GT}_{i}$ are both send into this
discriminator, its structure is exactly the same as it is in stage one..

Figure 5: The diagram of our proposed LS-AAE. It combines an autoencoder that
fully utilizes the spatial correlation between consecutive CT images and a
discriminator that help refine image details.

###  2.3 Network Training

Stage one and stage two are trained separately. For the autoencoders in these
two models, we employ the multi-loss function below, which is consists of
three parts $l_{MSE}$, $l_{Adv}$ and $l_{Reg}$ with their respective
hyperparameters $\alpha_{1}$, $\alpha_{2}$ and $\alpha_{3}$.

${{l}_{AE}}={{\alpha}_{1}}{{l}_{MSE}}+{{\alpha}_{2}}{{l}_{Adv}}+{{\alpha}_{3}}{{l}_{Reg}}$
(1)

$l_{MSE}$ calculates the mean square error between the restored image and its
corresponding ground truth, it is widely used in various image inpainting
tasks since it provides an intuitive evaluation for the model’s prediction.
The expression of $l_{MSE}$ can be seen from Equation (2).

${{l}_{MSE}}=\frac{1}{W\times
H}\sum\limits_{x=1}^{W}{\sum\limits_{y=1}^{H}{{{(\mathbb{I}_{x,y}^{GT}-{{G}_{AE}}{{({{\mathbb{I}}^{Input}})}_{x,y}})}^{2}}}}$
(2)

Where function $G_{AE}$ stands for the autoencoder, $\mathbb{I}^{Input}$ and
$\mathbb{I}^{GT}$ are the input image and its corresponding ground truth, $W$
and $H$ are the width and height of the input image respectively.

$l_{Adv}$ refers to the adversarial loss. The autoencoder manages to fool the
discriminator by making its prediction as close to the ground truth as
possible, so as to achieve the ideal image restoration outcome. Its expression
can be seen from Equation (3).

${{l}_{adv}}=1-D({{G}_{AE}}({\mathbb{I}^{Input}}))$ (3)

Where function $D$ and $G_{AE}$ stands for the discriminator and the
autoencoder respectively, $\mathbb{I}^{Input}$ is the model’s input image.

$l_{Reg}$ is the regularization term of our multi-loss function. Since noises
will have a side effect on our restoration result, we add a regularization
term to maintain the smoothness of the image and also prevent overfitting. TV
Loss is widely used in image analysis tasks, it reduces the variation between
adjacent pixels to a certain extent. Its expression can be seen from Equation
(4).

${{l}_{Reg}}=\frac{1}{W\times
H}\sum\limits_{x=1}^{W}{\sum\limits_{y=1}^{H}{\left\|\left.\nabla{{G}_{AE}}{{({\mathbb{I}^{Input}})}_{x,y}}\right\|\right.}}$
(4)

Where function $G_{AE}$ represents the autoencoder, $I^{Input}$ is the model’s
input image, $W$ and $H$ are the width and height of the input image
respectively. $\nabla$ calculates the gradient, $\left\|\cdot\right\|$ obtains
the norm.

To optimize the discriminator of these two stages, their loss function should
enable them to better distinguish between real and fake inputs. The loss
function $l_{Dis}$ is shown in Equation (5).

${{l}_{D\text{is}}}=1-D\left({\mathbb{I}^{GT}}\right)+D\left({{G}_{AE}}\left({\mathbb{I}^{Input}}\right)\right)$
(5)

Where function $D$ and $G$ stands for the discriminator and the autoencoder
respectively, $\mathbb{I}^{Input}$ and $\mathbb{I}^{GT}$ are the input image
and its corresponding ground truth. The discriminator outputs a scalar between
0 to 1 which represents the probability that the input image is real.
Therefore, minimizing $1-D(\mathbb{I}^{GT})$/maximizing $D(\mathbb{I}^{GT})$
enables the discriminator to recognize real images, while minimizing
$D(G_{AE}(\mathbb{I}^{Input}))$ enables the discriminator to distinguish fake
images that generated from the autoencoder from all input images.

During the training process, we adopt the Adam algorithm [72] for
optimization. the learning rate is set to 1e-4 initially. For the multi-loss
function, $\alpha_{1}$, $\alpha_{2}$ and $\alpha_{3}$ are set to 1, 1e-3, and
2e-8 respectively. We implement our whole structure using PyTorch [73] on two
GeForce RTX 2080 Ti.

##  3 Experiments

We adopt the LIDC-IDRI [74] as our dataset, which includes 1018 cases and
approximately 240,000 DCM files of corresponding CT images. Cases 1 to 100 are
divided into test set, cases 101 to 200 are divided into validation set, and
the rest are divided into train set. Such a large amount of data allows us to
train our models from scratch without overfitting. We utilize NumPy to read
from these DCM files and conduct sparse sampling at intervals of 4, 8 and 16
(the corresponding full-view Radon data has 180 projections). Subsequently, we
first analyze our overall structure through a series of ablation studies, and
then compare our experimental results with other current methods to prove its
superiority and robustness.

###  3.1 Ablation Study

With all these innovations we make in our overall structure design, it would
be appropriate for us to conduct corresponding ablation studies to prove their
necessity. In this part, all the experimental results are acquired from
sparse-view CT data with an interval of 4 if there is no specific mention.

####  3.1.1 The L-AE’s Trade-off between Mobility and Performance

As is known to all, U-Net has extraordinary performance in numerous medical
image processing tasks, [42] implemented it for sparse-view CT imaging
restoration and obtained outstanding restoration results. To testify that our
proposed autoencoder can achieve a good balance between performance and
mobility, we replace it with U-Net in the first stage and compare the
restoration results and model parameters of this stage with ours, as shown in
Table 3. The images mentioned in Table 3 are reconstructed from the Radon data
restored through stage one.

Table 3: U-Net VS. L-AE |  Radon PSNR |  Radon SSIM |  Image PSNR |  Image SSIM |  Parameters
---|---|---|---|---|---
U-Net | 57.582 | 0.998 | 29.598 | 0.874 |  10.401M
L-AE | 57.66 | 0.998 | 29.609 | 0.874 |  1.675M

As we can see from Table 3, whether in the Radon domain or in the image
domain, L-AE has competitive performance compared with U-Net. Moreover, it
significantly reduces model parameters, making it suitable for situations
where computational resources are extremely limited. This exhibits our model’s
ability in efficiently restoring CT images, thus adapting to the social trend
of deploying portable medical devices.

####  3.1.2 The Discriminator

We establish discriminators in both two stages, hoping to further improve our
model’s performance in restoring sparse-view CT data through the adversarial
learning between the autoencoders and the discriminators. In order to verify
this point of view, we send the test set into stage one where there is merely
an autoencoder and compare its restoration results with ours, which can be
seen from Table 4. The images mentioned in Table 4 are reconstructed from the
Radon data restored through stage one.

Table 4: The Role of the Discriminator |  Radon PSNR |  Radon SSIM |  Image PSNR |  Image SSIM
---|---|---|---|---
L-AE Only | 48.904 | 0.985 | 28.448 | 0.871
L-AAE | 57.660 | 0.998 | 29.609 | 0.874

From the above table, we can realize the significance of our proposed
discriminator, it indeed assists our model to achieve a better level of
restoration under the evaluation of PSNR and SSIM. Its precise structure
(refers to Sec II) also ensures a high degree of mobility, which enables our
overall structure to be portable and accurate at the same time.

####  3.1.3 The Two-step Architecture – LS-AAE

As we state in Sec II, this sort of cascaded two-step structure inherently
embeds the motion of objects which can largely help to remove image artifacts
due to the strong redundancy between these consecutive images. Consequently,
we design an experiment with reference to [71] to prove this view. In the
second stage, instead of sending five consecutive images into this two-step
LS-AAE, we directly input them into a single Inpainting Block (SIB) that is
slightly modified in the three-dimensional convolution part to handle five
images, that means we adopt a stride of 2 in the Conv1_1 layer (refers to
Table 1). The experimental results can be seen from Table 5 below.

Table 5: Restoration Results of SIB and LS-AAE |  Image PSNR |  Image SSIM
---|---|---
SIB | 38.972 | 0.941
LS-AAE | 40.305 | 0.948

Now the SIB no longer owns this built-in cascade structure to implicitly
conduct motion estimation, it suffers from a obvious drop in PSNR and SSIM.
Therefore, we can arrive at the conclusion that, LS-AAE manages to effectively
improve model’s capability of restoring CT images with its cascaded two-step
architecture that inherently capture the motion between consecutive images.

####  3.1.4 The 3D convolution in LS-AAE

We mention in Sec II that, the extension of search regions from two-
dimensional image neighborhoods to three-dimensional spatial neighborhoods
provide extra information for image restoration. Also, extracting spatial
features is conducive to remove streaking artifacts as the residual error in
each image is correlated. In order to realize this extension of search
regions, three-dimensional convolution is employed in every Inpainting Block
of LS-AAE. To verify the cruciality of these three-dimensional convolutions,
we conduct an experiment in which 3D convolution are replaced back to 2D
convolution, where the number of input images is regarded as the number of
input channel (refers to Table 2). The inpainting results of these two models
are shown in Table 6.

Table 6: Restoration Results of 2D and 3D LS-AAE |  Image PSNR |  Image SSIM
---|---|---
2D LS-AAE | 39.472 | 0.944
3D LS-AAE | 40.305 | 0.948

We can see that the inpainting outcome suffers from a drop about 0.9dB in
PSNR, proving that three-dimensional convolutions assist model in restoring CT
images to a certain extent without significantly consuming computational
resources.

####  3.1.5 The Image Interval of LS-AAE’s Input

In all the experiments above, we set the image interval between input
consecutive CT images of LS-AAE to the default value of 1. However, we cannot
help but wonder that whether increasing the interval value can help the model
obtain more spatial information, thereby enhancing its ability in removing
image artifacts. In the following experiment, we set this image interval $T$
to 1, 2, 3, 4 and 5 respectively, their corresponding results are shown in
Table 7.

Table 7: The Image Interval’s Effect on Restoration Results |  Image PSNR |  Image SSIM
---|---|---
$\bm{T=1}$ | 40.305 | 0.948
$T=2$ | 39.961 | 0.948
$T=3$ | 40.032 | 0.948
$T=4$ | 40.147 | 0.948
$T=5$ | 40.195 | 0.950

It can be learnt from Table 7 that this hyperparameter T does not have much
impact on the final restoration result. Spatial correlation seems to be well
utilized when the image interval is set to 1, which would be a decent default
choice.

####  3.1.6 The Radon Domain VS. the Image Domain

In this paper, we adopt a two-stage structure that combines the Radon domain
and the image domain to obtain high-quality sparse-view CT images. Since each
stage of the overall structure conducts restoration in their separate domains
and both remarkably upgrade the restoration results, this leads us to think,
what role do these two domains play? Subsequently, we feed our test set into
these three structures: L-AAE in stage one that concentrates on the Radon
domain, LS-AAE in stage two that focus on the image domain and of course, our
overall structure that contains these two stages. The quantitative inpainting
results of the above three structures can be referred from Table 8, the
intuitive outcome can also be seen in Figure 6.

Figure 6: The intuitive restoration results obtained by different domains. Table 8: Restoration Results obtained by different domains |  Image PSNR |  Image SSIM
---|---|---
The Radon Domain | 30.310 | 0.905
The Image Domain | 34.135 | 0.888
Dual Domains | 40.305 | 0.948

It can be seen from above that, restoration in each domain has its pros and
cons. For the Radon domain, it demonstrates its superiority in enhancing the
structural similarity of images so as to perform well under the evaluation of
SSIM. While as for the image domain, it exhibits great ability in alleviating
distortion, thus has a relatively good performance under the evaluation of
PSNR. Naturally, we acquire extraordinary restoration results when combining
these two domains to merge their respective advantages. Besides, we solely
utilize the spatial correlation in the Image domain due to our discovery that,
the spatial information between continuous Radon slices has little impact on
the final inpainting outcome. We suppose this is because the texture in Radon
slices does not have much similarity with CT images, thus cannot be restored
in this way.

###  3.2 Methods Comparison

After verifying the rationality of our overall structural design, we want to
testify its robustness through applying it to sparse-view CT data with a
higher level of sparsity, which means, conducting sparse sampling at intervals
of 4, 8 and even 16 (the corresponding full-view Radon data has 180
projections). In addition, we compare our method with other current ones to
prove its prominent capability of restoring sparse-view CT images and removing
streaking artifacts. The experimental results are shown in Table 9, and the
intuitive outcome can be seen from Figure 7.

Figure 7: The intuitive restoration results of various methods at different sampling intervals. Table 9: Methods Comparison | Interval=4 | Interval=8 | Interval=16
---|---|---|---
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
FBP |  12.080 |  0.498 |  12.065 |  0.485 |  12.032 |  0.471
SART-TV |  19.179 |  0.665 |  19.061 |  0.632 |  18.777 |  0.602
U-Net |  34.018 |  0.885 |  31.944 |  0.843 |  28.767 |  0.798
Ours |  40.305 |  0.948 |  37.633 |  0.937 |  34.052 |  0.910

As we can see, our method exhibits extraordinary capability of restoring
sparse-view CT imaging, effectively removes streaking artifacts and outruns
other methods by a large margin. Also, it can be applied to extreme sparsity
while still obtaining prominent inpainting outcome. Particularly, our method
still exceeds others when the sampling rate is one-fourth of them, thereby
demonstrating its remarkable robustness and superiority.

##  4 Conclusion

In this paper, we propose a lightweight structure that efficaciously restores
sparse-view CT with its two-stage architecture combining the Radon domain and
the image domain. Most importantly, we groundbreakingly exploit the abundant
spatial information existing between consecutive CT images, so as to achieve a
remarkable restoration outcome even if our method encounters extreme sparsity.

In the first stage, a mobile model named L-AAE is proposed to complement the
original sparse-view CT in the Radon domain, it adopts the inverted residual
with linear bottleneck in order to significantly reduce computational resource
requirements while maintaining outstanding performance. In the second stage,
after reconstructing the restored full-view Radon data into images through
FBP, we establish a lightweight model called LS-AAE. It is designed to
implicitly conduct motion estimation and dig into the three-dimensional
spatial neighborhood with a relatively low memory cost. Therefore, it manages
to concentrates on fully utilizing the strong spatial correlation between
continuous CT images, so as to productively remove streaking artifacts and
finally acquire high-quality restoration results.

Eventually, for the sparse-view CT with a sampling interval of 4, we achieve a
PSNR of 40.305 and a SSIM of 0.948, realizing a remarkable restoration result
that can effectively eliminate image artifacts. In addition, our method also
performs well when it comes to extreme sparsity (the sampling interval is 8 or
even 16), exhibiting its prominent robustness.

## References

  * [1] A. M. Cormack.  Representation of a function by its line integrals, with some radiological applications. II.  J. Appl. Phys., 35(10):2908–2913, Nov. 1964.
  * [2] G. Hounsfield.  Computerized transverse axial scanning (tomography): I. description of system.  Br. J. Radiol., 46(552):1016–1022, Jan. 1974.
  * [3] Wang, Ge, Yu, Hengyong, De, Man, and Bruno.  An outlook on X-ray CT research and development.  Med. Phys., 35(3):1051–1064, Apr. 2008.
  * [4] R. Krishnamoorthi, M. N. Ramarajan, N. E. Wang, M. B. Newman, M. E. Rubesova, C. M. Mueller, and R. A. Barth.  Effectiveness of a staged US and CT protocol for the diagnosis of pediatric appendicitis: reducing radiation exposure in the age of ALARA.  Radiology, 259(1):231–239, Apr. 2011.
  * [5] T. L. Slovis.  CT and computed radiography: The pictures are great, but is the radiation dose greater than required?  Am. J. Roentgenol., 179(1):39–41, Aug. 2002.
  * [6] C. H. Mccollough, A. N. Primak, N. Braun, J. Kofler, L. Yu, and J. Christner.  Strategies for reducing radiation dose in CT.  Radiol. Clin. N. Am., 47(1):27–40, Feb. 2009.
  * [7] C. H. Mccollough, M. R. Bruesewitz, and J. M. Kofler.  CT dose reduction and dose management tools: Overview of available options.  Radiographics, 26(2):503–512, Mar. 2006.
  * [8] P. A. Poletti, A. Platon, O. Rutschmann, F. Schmidlin, C. Iselin, and C. Becker.  Low-dose versus standard-dose CT protocol in patients with clinically suspected renal colic.  Am. J. Roentgenol., 188(4):927–933, May. 2007.
  * [9] D. Tack, V. De Maertelaer, and P. Gevenois.  Dose reduction in multidetector CT using attenuation-based online tube current modulation.  Am. J. Roentgenol., 181(2):331–4, Sep. 2003.
  * [10] J. Bian, J. Siewerdsen, X. Han, E. Sidky, J. Prince, C. Pelizzari, and X. Pan.  Evaluation of sparse-view reconstruction from flat-panel-detector cone-beam CT.  Phys. Med. Biol., 55(22):6575–6599, Oct. 2010.
  * [11] J. Bian, J. Wang, X. Han, E. Sidky, L. Shao, and X. Pan.  Optimization-based image reconstruction from sparse-view data in offset-detector CBCT.  Phys. Med. Biol., 58(2):205–230, Dec. 2012.
  * [12] K. Jin, M. Mccann, E. Froustey, and M. Unser.  Deep convolutional neural network for inverse problems in imaging.  IEEE Trans. Image Process., PP:99, Nov. 2016.
  * [13] A. H. Andersen and A. C. Kak.  Simultaneous algebraic reconstruction technique (SART): A superior implementation of the ART algorithm.  Ultrason. Imaging, 6(1):81–94, 1984.
  * [14] D. Wu, K. Kim, G. E. Fakhri, and Q. Li.  Iterative low-dose CT reconstruction with priors trained by artificial neural network.  IEEE Trans. Med. Imag., PP(12):1–1, 2017.
  * [15] Z. Hu, J. Gao, N. Zhang, Y. Yang, X. Liu, H. Zheng, and D. Liang.  An improved statistical iterative algorithm for sparse-view and limited-angle CT image reconstruction.  Sci. Rep., 7(1), Dec. 2017.
  * [16] H. Zhang, B. Dong, and B. Liu.  Jsr-net: A deep network for joint spatial-radon domain CT reconstruction from incomplete data.  In IEEE Int. Conf. Acoust. Speech Signal Process. Proc., 2019.
  * [17] E. Candès, J. Romberg, and T. Tao.  Robust uncertainty principles : Exact signal frequency information.  IEEE Trans. Inf. Theory, 52(2):489–509, Mar. 2006.
  * [18] E. Y. Sidky and X. Pan.  Image reconstruction in circular cone-beam computed tomography by constrained, total-variation minimization.  Phys. Med. Biol., 53(17):4777–4807, Oct. 2008.
  * [19] S. Niu, Y. Gao, Z. Bian, J. Huang, W. Chen, G. Yu, Z. Liang, and J. Ma.  Sparse-view X-ray CT reconstruction via total generalized variation regularization.  Phys. Med. Biol., 59(12):2997, May. 2014.
  * [20] E. Y. Sidky, C. M. Kao, and X. Pan.  Accurate image reconstruction from few-views and limited-angle data in divergent-beam CT.  J. X-Ray Sci. Technol., 14(2):119–139, 2009.
  * [21] Q. Xu, H. Y. Yu, X. Q. Mou, L. Zhang, J. Hsieh, and G. Wang.  Low-dose X-ray CT reconstruction via dictionary learning.  IEEE Trans. Med. Imag., 31(9):1682–1697, Apr. 2012.
  * [22] Y. LeCun, Y. Bengio, and G. Hinton.  Deep learning.  Nature, 521(7553):436–444, May. 2015.
  * [23] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, and M. Bernstein.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision, 115:1–42, Jan. 2015.
  * [24] J. Long, E. Shelhamer, and T. Darrell.  Fully convolutional networks for semantic segmentation.  IEEE Trans. Pattern Anal. Mach. Intell., 39(4):640–651, 2015.
  * [25] O. Ronneberger, P. Fischer, and T. Brox.  U-Net: Convolutional networks for biomedical image segmentation.  In Proc. Int. Conf. Med. Image Comput. Comput. Assist. Intervent., pages 234–241, 2015.
  * [26] M. Soltaninejad, C. J. Sturrock, M. Griffiths, T. P. Pridmore, and M. P. Pound.  Three dimensional root CT segmentation using multi-resolution encoder-decoder networks.  IEEE Trans. Image Process., 29:6667–6679, 2020.
  * [27] C. Dong, Y. Deng, C. C. Loy, and X. Tang.  Compression artifacts reduction by a deep convolutional network.  In Proc. IEEE Int. Conf. Comput. Vision., pages 576–584, 2015.
  * [28] J. Guo and H. Chao.  Building dual-domain representations for compression artifacts reduction.  In Proc. Europ. Conf. Comp. Visi., pages 628–644, Sep. 2016.
  * [29] J. Xie, L. Xu, and E. Chen.  Image denoising and inpainting with deep neural networks.  Adv. neural inf. proces. syst., 1, 2012.
  * [30] K. Kulkarni, S. Lohit, P. Turaga, R. Kerviche, and A. Ashok.  ReconNet: Non-iterative reconstruction of images from compressively sensed measurements.  In Proc. IEEE Comput. Soc. Conf. Comput. Vision Pattern Recognit., 2016.
  * [31] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In Adv. neural inf. proces. syst., volume 27, pages 2672–2680, 2014\.
  * [32] J. Bai, X. Dai, Q. Wu, and L. Xie.  Limited-view CT reconstruction based on autoencoder-like generative adversarial networks with joint loss.  In 2018 40th Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., pages 5570–5574, Jul. 2018.
  * [33] S. Xie, H. Xu, and H. Li.  Artifact removal using GAN network for limited-angle CT reconstruction.  In 2019 Ninth Int. Conf. Image Process. Theory, Tools Appl., 2019\.
  * [34] Z. Zhao, Y. Sun, and P. Cong.  Sparse-view CT reconstruction via generative adversarial networks.  In 2018 IEEE Nucl. Sci. Symp. Med. Imaging Conf., 2018.
  * [35] J. C. Ye and Y. S. Han.  Deep convolutional framelets: A general deep learning for inverse problems.  SIAM J. Imaging Sci., 11(2), 2017.
  * [36] J. Dong, J. Fu, and Z. He.  A deep learning reconstruction framework for X-ray computed tomography with incomplete data.  PLoS One, 14:e0224426, Nov. 2019.
  * [37] J. Fu, J. Dong, and F. Zhao.  A deep learning reconstruction framework for differential phase-contrast computed tomography with incomplete data.  IEEE Trans. Image Process., 29(1):2190–2202, 2020.
  * [38] R. Anirudh, H. Kim, J. J Thiagarajan, K. A. Mohan, K. Champley, and T. Bremer.  Lose the views: Limited angle CT reconstruction via implicit sinogram completion.  In Proc. IEEE Comput. Soc. Conf. Comput. Vision Pattern Recognit., 2018.
  * [39] X. Dai, J. Bai, T. Liu, and L. Xie.  Limited-view cone-beam CT reconstruction based on an adversarial autoencoder network with joint loss.  IEEE Access, 7:7104–7116, 2019.
  * [40] M. U. Ghani and W. C. Karl.  Deep learning-based sinogram completion for low-dose CT.  In IEEE Image, Video, Multidimens. Signal Process. Workshop, IVMSP - Proc., pages 1–5, 2018.
  * [41] A. Katsevich.  Theoretically exact filtered backprojection-type inversion algorithm for spiral CT.  SIAM J. Appl. Math., 62(6):2012–2026, 2002.
  * [42] Y. Han and J. C. Ye.  Framing U-Net via deep convolutional framelets: Application to sparse-view CT.  IEEE Trans. Med. Imaging, pages 14–18, 2018.
  * [43] Z. Zhang, X. Liang, X. Dong, Y. Xie, and G. Cao.  A sparse-view CT reconstruction method based on combination of DenseNet and deconvolution.  IEEE Trans. Med. Imaging, 37(6):1–1, 2018.
  * [44] H. Zhang, L. Li, K. Qiao, L. Wang, B. Yan, L. Li, and G. Hu.  Image prediction for limited-angle tomography via deep learning with convolutional neural network.  arXiv preprint arXiv:1607.08707, 2016.
  * [45] J. Wang, J. Liang, J. Cheng, Y. Guo, and L. Zeng.  Deep learning based image reconstruction algorithm for limited-angle translational computed tomography.  PLoS One, 15(1):e0226963, 2020.
  * [46] S. Kuanar, V. Athitsos, D. Mahapatra, K. R. Rao, Z. Akhtar, and D. Dasgupta.  Low dose abdominal CT image reconstruction: An unsupervised learning based approach.  In 2019 Proc. Int. Conf. Image Process, pages 1351–1355, 2019.
  * [47] S. Guan, A. A. Khan, S. Sikdar, and P. V. Chitnis.  Fully dense UNet for 2-D sparse photoacoustic tomography artifact removal.  IEEE J. Biomed. Health Inform., 24(2):568–576, 2020.
  * [48] D. Lee, S. Choi, and H. J. Kim.  High quality imaging from sparsely sampled computed tomography data with deep learning and wavelet transform in various domains.  Med. Phys., 2018.
  * [49] K. Liang, H. Yang, and Y. Xing.  Comparison of projection domain, image domain, and comprehensive deep learning for sparse-view x-ray ct image reconstruction.  arXiv preprint arXiv:1804.04289, 2018.
  * [50] J. Zhu, T. Su, X. Deng, X. Sun, and Y. Ge.  Low-dose CT reconstruction with simultaneous sinogram and image domain denoising by deep neural network.  In SPIE Med. Imag., volume 11312, pages 1007–1012, 2020.
  * [51] K. Hammernik, T. Würfl, T. Pock, and A. K. Maier.  A deep learning architecture for limited-angle computed tomography reconstruction.  Inf. aktuell, pages 92–97, 2017.
  * [52] Q. Zhang, Z. Hu, C. Jiang, H. Zheng, Y. Ge, and D. Liang.  Artifact removal using a hybrid-domain convolutional neural network for limited-angle computed tomography imaging.  Phys. Med. Biol., 65(15):155010, 2020.
  * [53] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. C. Chen.  MobileNetV2: Inverted residuals and linear bottlenecks.  In Proc IEEE Comput. Soc. Conf. Comput. Vision Pattern Recognit., 2018.
  * [54] R. Pascanu, T. Mikolov, and Y. Bengio.  On the difficulty of training recurrent neural networks.  In 30th Int. Conf. Mach. Learn., pages 1310–1318, 2013.
  * [55] J. Caballero, C. Ledig, A. Aitken, A. Acosta, J. Totz, Z. Wang, and W. Shi.  Real-time video super-resolution with spatio-temporal networks and motion compensation.  In Proc IEEE Comput. Soc. Conf. Comput. Vision Pattern Recognit., pages 2848–2857, 2017.
  * [56] M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian.  Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms.  IEEE Trans. Image Process., 21(9):3952–3966, 2012.
  * [57] P. Arias and J. M. Morel.  Video denoising via empirical bayesian estimation of space-time patches.  J. Math. Imaging Vis., 60(1):70–93, 2018.
  * [58] T. Vogels, F. Rousselle, B. Mcwilliams, G. Röthlin, A. Harvill, D. Adler, M. Meyer, and J. Novák.  Denoising with kernel prediction and asymmetric loss functions.  ACM Trans. Graph., 37(4):124, 2018.
  * [59] T. Ehret, A. Davy, J. M. Morel, G. Facciolo, and P. Arias.  Model-blind video denoising via frame-to-frame training.  In 2019 Proc IEEE Comput. Soc. Conf. Comput. Vision Pattern Recognit., pages 11369–11378, 2019.
  * [60] M. Claus and J. V. Gemert.  ViDeNN: Deep blind video denoising.  In 2019 IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn. Workshops, pages 0–0, 2019.
  * [61] A. Davy, T. Ehret, G. Facciolo, J. M. Morel, and P. Arias.  Non-local video denoising by CNN.  arXiv preprint arXiv:1811.12758, 2018.
  * [62] M. Tassano, J. Delon, and T. Veit.  DVDNET: A fast network for deep video denoising.  In 2019 Proc. Int. Conf. Image Process., pages 1805–1809, 2019\.
  * [63] X. Chen, L. Song, and X. Yang.  Deep RNNs for video denoising.  In Proc. 39th Appl. Digit. Image Process., volume 9971, 2016.
  * [64] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam.  MobileNets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [65] F. Chollet.  Xception: Deep learning with depthwise separable convolutions.  In 2017 Proc IEEE Comput. Soc. Conf. Comput. Vision Pattern Recognit., pages 1800–1807, 2017.
  * [66] X. Zhang, X. Zhou, M. Lin, and J. Sun.  ShuffleNet: An extremely efficient convolutional neural network for mobile devices.  In 2018 Proc IEEE Comput. Soc. Conf. Comput. Vision Pattern Recognit., pages 6848–6856, 2018.
  * [67] K. Hornik, M. Stinchcombe, and H. White.  Multilayer feedforward networks are universal approximators.  Neural Netw., 2(5):359–366, 1989.
  * [68] K. He, X. Zhang, S. Ren, and J. Sun.  Identity mappings in deep residual networks.  In Proc. Europ. Conf. Comp. Visi., pages 630–645, 2016.
  * [69] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Proc. of The 32nd Int. Conf. Mach. Learn., pages 448–456, 2015\.
  * [70] X. Glorot, A. Bordes, and Y. Bengio.  Deep sparse rectifier neural networks.  In Proc. of the 14th Int. Conf. Artif. Intell. Stat., volume 15, pages 315–323, 2011.
  * [71] M. Tassano, J. Delon, and T. Veit.  FastDVDnet: Towards real-time deep video denoising without flow estimation.  In 2020 Proc IEEE Comput. Soc. Conf. Comput. Vision Pattern Recognit., pages 1354–1363, 2020.
  * [72] D. P. Kingma and J. L. Ba.  Adam: A method for stochastic optimization.  In Int. Conf. Learn. Represent., 2015.
  * [73] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.  Automatic differentiation in PyTorch.  In NeurIPS, 2017.
  * [74] S. G. Armato _et al._ The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans.  Med. Phys., 38(2):915–931, 2011.