# When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate
Speech into Large Language Models for Depression Detection

Xiangyu Zhang1, Hexin Liu2, Kaishuai Xu3,
Qiquan Zhang1, Daijiao Liu1, Beena Ahmed1, Julien Epps1
University of New South Wales Sydney, Sydney1
Nanyang Technological University2
The Hong Kong Polytechnic University3

###### Abstract

Depression is a critical concern in global mental health, prompting extensive
research into AI-based detection methods. Among various AI technologies, Large
Language Models (LLMs) stand out for their versatility in mental healthcare
applications. However, their primary limitation arises from their exclusive
dependence on textual input, which constrains their overall capabilities.
Furthermore, the utilization of LLMs in identifying and analyzing depressive
states is still relatively untapped. In this paper, we present an innovative
approach to integrating acoustic speech information into the LLMs framework
for multimodal depression detection. We investigate an efficient method for
depression detection by integrating speech signals into LLMs utilizing
Acoustic Landmarks. By incorporating acoustic landmarks, which are specific to
the pronunciation of spoken words, our method adds critical dimensions to text
transcripts. This integration also provides insights into the unique speech
patterns of individuals, revealing the potential mental states of individuals.
Evaluations of the proposed approach on the DAIC-WOZ dataset reveal state-of-
the-art results when compared with existing Audio-Text baselines. In addition,
this approach is not only valuable for the detection of depression but also
represents a new perspective in enhancing the ability of LLMs to comprehend
and process speech signals. 111code will be made public upon publication

Figure 1: Example of Acoustic Landmark (2-gram concat landmark (g+p-), (s+p+),
(p+,p-), …, (g-b-)), Landmarks are extracted from abrupt changes in the speech
signal. They can discretize speech into a series of tokens that possess
linguistic significance.

##  1 Introduction

Depression, a common mental disorder affecting 10-15% of the global
population, is characterized by persistent low mood, loss of interest, and
lack of energy, making it a prevalent and costly illness Walker et al. (2018).
Given the time-consuming, expensive, and sometimes ineffective nature of
traditional depression treatment methods, a growing number of researchers are
turning their attention to developing automated depression detection systems.
Concurrently, Large language models (LLMs) have recently demonstrated
remarkable success across a variety of tasks Chowdhery et al. (2023); Touvron
et al. (2023). These large language models have been applied to various
healthcare issues, including general surgery Oh et al. (2023), dementia
diagnosis Wang et al. (2023), and gastroenterology Lahat et al. (2023) and
achieved excellent results. However, their main limitation stems from their
sole reliance on textual input, which limits their full potential.
Simultaneously, the use of Large Language Models (LLMs) in depression
detection remains largely unexplored. In particular, there has been no effort
to integrate speech—despite growing evidence that speech signals can reveal
indicators of depression Wu et al. (2023); Huang et al. (2019a)—into these
LLMs, an advancement that could greatly improve their effectiveness in
identifying depression Zheng et al. (2023).

Figure 2: Overview of LLM-Landmark Depression Detection Pipeline, broadly
categorized into three stages: landmark detection (on the left), cross-modal
instruction fine-tuning (in the middle), and P-tuning for depression detection
(on the right).

One of the key approaches to incorporating speech signals into LLMs is through
the discretization of speech. However, the current landscape of speech
discretization, heavily reliant on deep learning techniques Zeghidour et al.
(2021); Défossez et al. (2022), faces significant challenges due to its
considerable GPU memory requirements. This is particularly problematic in the
field of depression detection, where data often consists of lengthy
conversations DeVault et al. (2014). The need for completed conversations is
vital for accurate depression detection Wu et al. (2023); Sun et al. (2022),
rendering the existing deep learning-based methods impractical for such
applications. For this purpose, it is necessary to find an efficient approach
that allows for the discretization of speech with reduced GPU memory usage.

Acoustic landmarks represent event markers intricately linked with the
articulation of speech, forming a concise alternative framework for speech
processing Liu (1996); Stevens (2002). This approach emphasizes the analysis
of abrupt acoustic changes at the subsegmental level, thereby providing a
succinct and precise phonetic description of language. These landmarks,
characterized by their binary values, establish a minimal yet effective set
for differentiating each language segment from others. They maintain a direct
and significant relationship with acoustic properties and articulation
(including individual pronunciation), ensuring discernibility despite unwanted
variability introduced by diverse hardware and environmental backgrounds Huang
et al. (2018, 2019b). Their discrete nature not only allows for efficient
integration into large language models but also offers a viable alternative
for understanding speech signals in depression detection, bypassing the
limitations of current deep learning-based techniques. This innovative
approach promises a more feasible and resource-efficient pathway for analyzing
complex speech patterns in mental health diagnostics.

In this paper, we introduce a novel multimodal approach to depression
detection, utilizing a combination of acoustic landmarks and large language
models. We investigate the properties of large language models at various
stages and under different conditions after integrating landmark-based speech
information. We investigate how LLMs learn speech landmarks and assess the
impact of conversational fine-tuning on the performance of LLMs in tasks
related to depression detection.

In summary, our contributions include the following:

  * •

To the best of our knowledge, this is the first study to apply LLMs to
multimodal depression detection and the inaugural effort to integrate speech
information into LLMs for this purpose. We proposed a new baseline for the
application of LLMs in the field of automatic depression detection.

  * •

Compared with prior baseline audio-text methods Wu et al. (2023), our approach
not only achieved SOTA performance but also involved a comprehensive analysis
of the properties of LLMs post the integration of landmarks.

  * •

Unlike previous deep learning-based methods for aiding LLMs in understanding
speech, we explored a new, more efficient approach to enable LLMs to process
speech signals. This novel method opens up a potentially groundbreaking
direction for enhancing LLMs’ comprehension of speech.

Figure 3: Landmark Detection Filter

##  2 Related Work

###  2.1 Large Language Models

Large language models have achieved success in natural language processing and
have been extended to encompass computer vision and speech signal processing
Brown et al. (2020); Touvron et al. (2023). However, there is a significant
gap in research aimed at enabling LLMs to comprehend speech efficiently.

Parameter-efficient fine-tuning refers to selectively updating a small subset
of the model’s parameters or adding lightweight trainable layers, to customize
the model for specific tasks or domains with reduced computational overhead.
Existing works employed low-rank adaptation (LoRA) to fine-tune LLM
efficiently. LoRA reduces computational complexity by freezing the pre-trained
LLM and injecting trainable rank decomposition matrices A and B into its
transformer-based layers Hu et al. (2022). The forward pass is subsequently
defined as the linear combination of those from the pre-trained model and from
the trained decomposed matrices A and B.

###  2.2 Acoustic Landmarks

The concept of acoustic landmarks originally stems from research on
distinctive features Garvin (1953). Some researchers posit that for certain
phonetic contrasts, a listener relies on acoustic landmarks to gather the
necessary acoustic cues for deciphering the underlying distinctive features
Liu (1996). This perspective highlights the importance of these landmarks in
the auditory processing and interpretation of speech. Subsequent research has
utilized acoustic landmarks for applications in speech recognition Liu (1996);
He et al. (2019) as well as in addressing mental health-related problems Huang
et al. (2018, 2019a). Although different scholars have slightly varied
definitions of acoustic landmarks, Joel and colleagues Boyce et al. (2012)
expanded upon Liu’s paper Liu (1996) by releasing a MATLAB version of a
landmark detection toolkit, which has become the most widely used version of
landmark technology.

###  2.3 Automatic Depression Detection

The use of AI technology for depression detection has been developing for many
years. Some researchers Cummins et al. (2011); Huang et al. (2018, 2019a) have
utilized traditional methods such as Support Vector Machines (SVMs) Noble
(2006) for depression detection. With the advancement of deep learning
technologies, an increasing number of researchers have been experimenting with
deep learning approaches for depression detection. Zhao and others have
explored the use of transformer models for processing speech inputs in
depression detection Zhao et al. (2020). Shen and colleagues have employed BI-
LSTM architectures, combining text and speech for this purpose Shen et al.
(2022). Further extending these techniques, Wu Wu et al. (2023) utilized
speech self-supervised models Chen et al. (2022); Hsu et al. (2021) and
integrated them with RoBERTa Liu et al. (2019) for a more comprehensive text-
audio multimodal approach to depression detection.

##  3 Methodology

###  3.1 Overview

Our methodology, detailed in Figure 2, encompasses a three-step training
process. The first phase involves extracting acoustic landmarks from speech
and conducting an array of data processing operations. Subsequently, in the
Cross-modal Instruction Fine-Tuning phase, we engage the LLM in learning the
nuances and characteristics of acoustic landmarks. The culminating phase is
the P-Tuning process, wherein the LLM is meticulously trained to apply its
understanding to diagnose depression.

###  3.2 Landmarks Extraction and Data Preprocessing

####  3.2.1 Landmarks Extraction

Figure 1 illustrates an example of acoustic landmarks, where speech signals
are discretized into a series of symbols that carry linguistic relevance.
Table 1 details the specific acoustic landmarks utilized in our study.
Diverging from Liu’s paper Liu (1996), our research also pays attention to
frication, voice frication, and periodicity.

Landmark |  Description
---|---
g |  vibration of vocal folds start (+) or end (–)
b |  onset (+) or offset (–) of existence of turbulent noise during obstruent regions
s |  releases (+) or closures (–) of a nasal
v |  voiced frication onset (+) or offset (–)
p |  periodicity start (+) or end (–)
f |  frication onset (+) or offset (–)
Table 1: Description of the six landmarks investigated.

Our method primarily draws inspiration from Joel’s Boyce et al. (2012) and
Liu’s Liu (1996) work. However, since they have not open-sourced their code,
many of their approach’s details remain unknown. In the following section, We
introduce our Python-based landmark detection algorithm, developed to address
these gaps and to adapt the conceptual framework to our specific requirements.
Initially, the spectrogram is divided into six frequency bands. Landmarks are
identified through energy changes within these six bands, using a two-pass
strategy. Different landmarks are determined by either a single band or a
combination of multiple bands Liu (1996). This approach is visually
represented by the two parallel branches emanating from the spectrogram block
in Figure 3.

The detection algorithm for Glottal (g), Burst (b), and Syllabic (s) landmarks
is fundamentally aligned with Liu’s approach Liu (1996). However, diverging
from Liu’s method, we employ 5dB and 8dB as threshold values because of
different smoothing methods between Python and Matlab. Additionally,
considering that the opening and closing of the glottis occur in pairs, We
implemented dynamic programming to ensure that g landmarks appear in pairs,
thus enhancing the physiological accuracy of our detection.

Our methodology for identifying f+ and v+ landmarks involves detecting a 6 dB
power increase in at least three high-frequency bands (bands 4-6), and a power
decrease in low-frequency bands (bands 2 and 3). For f- and v-, the criteria
are reversed: a 6 dB power decrease in the same high-frequency bands and a
power increase in the low-frequency bands. The distinguishing factor here is
that frication landmarks are detected within unvoiced segments (b landmark),
while voiced frication landmarks are sought in voiced segments (s landmark).

Regarding the detection of the periodicity (p) landmarks, we perform
autocorrelation calculations on the audio frame to identify repetitive or
periodic patterns in the data. For a detailed description of our landmark
detection algorithm, please refer to Appendix A.

####  3.2.2 Data Augmentation and Processing

Depression assessments are commonly conducted through clinical interviews,
with each session receiving a singular label. This labeling method, when
applied to a given dataset size, leads to fewer samples in datasets compared
with the much larger number of utterances and frames typically encountered in
other speech-related tasks. As a result, the speech depression detection task
faces a notable challenge of data scarcity. Moreover, the issue of data
imbalance is particularly acute in the dataset, as instances of healthy
(positive cases) are significantly outnumbered by depression (negative) cases.
We adopted Wu’s approach Wu et al. (2023) of augmenting the training set
through sub-dialogue shuffling. Sub-dialogue shuffling involves sampling a
sub-dialogue $x_{s:e}$ from each complete dialogue $x_{1:T}$, where $s$ and
$e$ represent the randomly selected start and end utterance indexes,
respectively.

This technique allowed us to balance the number of positive and negative
samples effectively, while substantially increasing the dataset size.
Differing from Wu’s method, our use of landmarks in speech processing enables
the use of longer sub-dialogues for training purposes. To ensure a fair
comparison, we maintained the same data size (same sub-dialogue sampling
number M=1000) as Wu’s approach. For a detailed description of the algorithm,
please refer to Appendix B.

Previous research has indicated that the patterns in which landmarks appear
are more valuable than the individual landmarks themselves Huang et al.
(2019a). Therefore, as shown in Figure 1, we combined landmarks, treating
every two consecutive landmarks as a single unit. This approach not only
better represents the patterns of landmarks but also effectively reduces the
length of the landmark sequence in each sample.

###  3.3 Hint Cross-modal Instruction Fine-Tuning

Since LLMs inherently lack exposure to acoustic landmarks, our initial step
involves devising a method to teach the LLM what acoustic landmarks are. This
foundational training is crucial for enabling the models to interpret and
utilize acoustic landmark data effectively.

Method/ Model | Llama2-7B | Llama2-7B Chat | Llama2-13B | Llama2-13B Chat | GPT3.5 | GPT4
---|---|---|---|---|---|---
Text Only | 0.578 | 0.488 | 0.636 | 0.545 | 0.545 | 0.571
Landmark Only | 0.521 | 0.434 | 0.559 | 0.538 | - | -
Text + Landmark | 0.545 | 0.500 | 0.695 | 0.666 | - | -
Table 2: F1 scores for the different LLM models, We test all Llama2 models for
7B and 13B, also test on GPT.

As depicted in the middle section of Figure 2, our task involves providing an
LLM with instructions to predict potential acoustic landmarks based on text.
This method serves a dual purpose: it enables the LLM to learn about acoustic
landmarks, and it also aligns speech (landmarks) and text modalities using
paired data. We adopt LoRA Hu et al. (2022) by incorporating low-rank matrices
into the Query and Key matrices of the self-attention layer, facilitating
efficient adaptation and fine-tuning. Additionally, we resize the embedding
layer of the LLMs to add the merged landmarks to the vocabulary. During the
training process, both the embedding layer, linear head and the LoRA matrices
are actively trained to integrate these new elements effectively. The training
objective is to minimize the negative log-likelihood, and the loss calculation
applies to all samples (including the prefix), which can be formulated as:

$\mathcal{L}(M|C)=-\sum_{j=1}^{x}\sum_{i=1}^{y_{j}}\log
P(s_{i,j}|s_{<i,j},M),$ (1)

where $x$ is the number of samples in dataset $C$, $y_{j}$ is the text and
corresponding landmarks in sample $S$, and $M$ denotes the large language
model that we have fine-tuned.

Additionally, during dataset construction, we incorporate hints for the LLM.
For example, when data are sourced from a patient with depression, we include
a hint indicating their origin from a depressed patient. Experimentally, we
found this method of data construction to be crucial, which also supports our
hypothesis that the acoustic landmarks from individuals with depression differ
from those of healthy individuals. For detailed template construction, please
refer to Appendix C.

###  3.4 P-Tuning for Depression Detection

In the previous stage, we trained the LLMs to understand what landmarks are.
Following this, we employ P-tuning Liu et al. (2023) to enable the LLMs to
integrate text and landmarks for depression detection. We replace the lm head
layer with the classification layer. The training objective is to minimize
cross-entropy for classification, which can be formulated as

$\mathcal{L}=-\sum_{c=1}^{C}y_{o,c}\log(p_{o,c}),$ (2)

where $C$ is the number of classes. $y_{o,c}$ is an indicator variable that is
1 if the observation $o$ belongs to class $c$ and 0 otherwise. $p_{o,c}$ is
the predicted probability of observation $o$ belonging to class $c$. We also
compared instruction tuning using LoRA with P-tuning and discovered that
manually constructed templates are not well-suited for depression
classification tasks. Furthermore, we observed a performance improvement when
applying LoRA matrices across all layers of Llama2.

###  3.5 Decision Making

In the previous study by Wu et al. (2023), they achieved state-of-the-art
(SOTA) results through an ensemble approach, combining WavLM, WavLM pre-
trained on emotional recognition tasks, and the combined result of RoBERTa and
WavLM. Adopting a similar strategy, we fine-tune three distinct LlaMA2 (Text +
Landmark) models, each with different data volumes (different numbers of sub-
dialogue M(900, 1000, 1100)), and used them for ensemble voting.

##  4 Experiments

###  4.1 Experimental Setup

Dataset. The DAIC-WOZ dataset DeVault et al. (2014), recognized as a standard
for depression detection, includes 189 clinical interview recordings between
interviewers and patients. In its training subset, 30 of the total 107
interviews are labelled as depressed, while the development subset contains 12
depressed instances out of 35 interviews. Consistently with previous studies
Gong and Poellabauer (2017); Shen et al. (2022); Wu et al. (2022, 2023), we
report our results on the development subset.

Methods | Model | F1 | Ensemble
---|---|---|---
Previous SOTA Wu et al. (2023) | WavLM + RoBERTa | 0.648 | 0.829
WavLM Layer 8 | 0.700
WavLM Layer 10 | 0.720
Text+Landmark (Our) | Llama2 ($M\\!=\\!900$) | 0.636 | 0.833
Llama2 ($M\\!=\\!1000$) | 0.695
Llama2 ($M\\!=\\!1100$) | 0.719

Table 3:  A comparison of our proposed system with previous state-of-the-art
(SOTA), where all ensemble outcomes(F1 Score) are derived from a majority
vote. In the table, M denotes the number of augmented sub-dialogues per
dialogue in our data augmentation algorithm, while the previous SOTA used
M=1000 sub-dialogues.

Model Configurations. Our research utilizes Llama2-7B, Llama-7B Chat,
Llama2-13B, and Llama2-13B Chat, conducted on a system equipped with 8 NVIDIA
A100 80GB GPUs. Llama 2-Chat was optimized for engaging in two-way
conversations. In the cross-modal instruction fine-tuning stage, We fine-tuned
the model with 10 epochs with 128 batch sizes, 8 Lora ranks, 100 warmup steps,
and a 1e-6 learning rate. In the depression detection stage, we fine-tuned the
model with 8 epochs with 256 batch sizes, 30 virtual tokens, 256 encoder
hidden sizes, and a 1e-6 learning rate. In both experiments, we used AdamW as
an optimizer with the model parallel to fine-tune our model. In the ablation
study stage, we used hyperparameter tuning following the Tree-structured
Parzen Estimator (TPE) paradigm Bergstra et al. (2011).

###  4.2 Main Result: Performance of different LLMs in Depression Detection
task

Depression Detection in Llama2. Table 2 displays the F1 scores obtained by
Llama2 in depression detection across different scenarios. Additionally, we
conducted a comparison of our findings with the results obtained from GPT-3.5
and GPT-4, focusing solely on their performance in the text modality. It is
crucial to highlight that we did not fine-tune GPT-3 or GPT-4 for our
purposes. Rather, we employed carefully crafted prompts(see appendix D),
allowing the GPT models to assess whether a particular sample was from a
patient with depression.

For the ’landmark only’ and ’landmark + text’ results, the process involved
first undergoing hint cross-modal instruction fine-tuning and then employing
P-tuning for depression detection. The objective was to equip the LLMs with a
preliminary understanding of landmarks before advancing to the diagnostic
stage for depression.

The experimental results reveal that when LLMs solely use the text modality
for depression detection, the performance of all models, including notably
powerful ones like GPT-3.5 and GPT-4, which excel in many tasks, is not
particularly impressive and remains somewhat unsatisfactory. We attribute the
subpar performance to two main factors. First is the inherent limitation of
the text modality in conveying emotional information. For instance, consider
the sentence, "It’s raining today." While some may find this statement
positive, others might feel the opposite. It’s challenging to discern the
emotional nuances from the text alone, but with audio information, we could
accurately capture the emotional context of the statement. Secondly, the issue
lies with the data itself. Labels are only available at the document level,
and data are scarce (currently, there are no larger public datasets available
for multimodal depression detection). This limitation in data granularity and
volume significantly hinders the model’s ability to accurately detect
depression.

The introduction of landmarks led to enhanced performance across all models,
affirming the effectiveness of our method in integrating landmarks. Landmarks
can represent some of the acoustic information due to affective variation,
providing additional information that assists LLMs in detecting depression.
Nonetheless, the efficacy of using landmarks in isolation for depression
detection was found to be suboptimal. Drawing on past research, we believe
this is due to the fact that even after cross-modal instruction fine-tuning,
relying solely on information from other modalities (such as audio or visual)
could potentially impair the stability of LLMs Zhang et al. (2023); Li et al.
(2023). When we combined multiple Llama2 models that had integrated both text
and landmark information for depression detection, we achieved SOTA results as
shown in table 3. Furthermore, as indicated in Table 3, there is a gradual
improvement in Llama2’s performance in depression detection tasks as the
number of sub-dialogues per dialogue increases. This observation further
emphasizes the crucial role that data quantity plays in the effectiveness of
depression detection tasks.

(a) 13B No Hint

(b) 13B Hint

(c) 13B Chat Hint

(d) Three Comparison

Figure 4: Evaluation loss for different configurations up to 4000 steps.

##  5 Ablation Study and Discussion

In this chapter, we conduct an empirical study to meticulously analyze and
elucidate the characteristics of LLMs that we identified in the context of
depression detection during our experiments.

###  5.1 Effect of Hint in Cross-Modal Instruction Fine-Tuning

During the Cross-Modal Instruction Fine-Tuning phase, we discovered that
providing a hint to the LLMs is crucial. In other words, informing the LLMs
whether the data sample originates from a patient with depression
significantly impacts the training outcome. As evident from Figure 4, without
a hint, the loss converged to around 1.76 (as shown in Figure 4(a)). In
contrast, with a hint, the loss consistently converged to near 1.1 (as
depicted in Figures 4(b) and 4(c)). Figure 4(d) offers a more vivid
illustration of the substantial difference that the presence or absence of a
hint makes to the model’s performance in our empirical study. This phenomenon
supports our previous conjecture that individuals with depression and those
who are healthy differ in their vocal expressions and that landmarks are
capable of reflecting this characteristic. Although the differences between
Llama2 and Llama2 Chat are not substantial, it is still observable that, in
this phase, Llama2 outperforms its Chat version. We will provide a more
detailed discussion in the subsequent section.

###  5.2 How LLMs Learn from Acoustic Landmarks

(a) Top 1 Contribution Layer

(b) Top 2 Contribution Layer

(c) Top 3 Contribution Layer

(d) Top 4 Contribution Layer

(e) Bottom Layer 1

(f) Bottom Layer 2

(g) Bottom Layer 3

(h) Bottom Layer 4

Figure 5: The top four images represent the LoRA matrices of the layers that
contribute most significantly to the large language model’s learning of
landmarks. The bottom four images depict the LoRA matrices of the layers with
the least contribution. As can be inferred from the graph’s title, the
feedforward layer is the primary contributor.

To further investigate how LLMs learn acoustic landmarks, we extended the
application of LoRA beyond just the attention layers, applying it across all
layers for comprehensive analysis Pu et al. (2023); Sun et al. (2023). To find
the matrix with the greatest contribution, we first need to define the method
for calculating the contribution of a matrix. We can approximately consider
the changes in the LoRA matrix as indicative of its contribution to the task
He et al. (2021). Therefore, we assess that the contribution of a matrix is
calculated by summing the absolute values of all its elements, normalized by
the total number of elements in the matrix. Suppose we have a set of LoRA
matrices $L_{1},L_{2},\ldots,L_{n}$, each matrix $L_{i}$ being an $a\times b$
matrix. Then, the contribution $C_{i}$ of matrix $L_{i}$ can be calculated
using the formula:

$C_{i}=\frac{1}{ab}\sum_{j=1}^{a}\sum_{k=1}^{b}|L_{i}(j,k)|.$ (3)

Here, $|L_{i}(j,k)|$ represents the absolute value of the element in the
$j^{th}$ row and $k^{th}$ column of matrix $L_{i}$. After calculating the
contribution value (C), we rank and select the ten matrices with the highest
and the lowest contributions for further analysis. Figure 5 separately
illustrates the four matrices with the greatest contributions and the four
with the least. To validate the effectiveness of this method, we deactivated
the five matrices with the smallest contributions and observed that this had
no significant impact on our results.

Our analysis of the matrices revealed that LLMs primarily learn landmarks
through the feedforward network, while the contribution of the LoRA matrices
in the attention layers is quite minimal. This phenomenon is also observed
when training LLMs to learn speech codecs Hao et al. (2023), suggesting that
even though landmarks have inherent linguistic significance, LLMs tend to
treat landmarks as abstract tensors, similar to speech codecs, during the
learning process. Additionally, we observed that layers closer to the
beginning of the LLMs have a greater contribution to learning landmarks. This
could be because LLMs treat landmarks as new vocabulary items, leading to more
updates in layers nearer to the embedding layer.

###  5.3 Llama2 vs Llama2 Chat, and Generation vs Classification

LlaMA2 models are uncensored and have not undergone instruction tuning or
chat-tuning. In contrast, LlaMA2 Chat models are censored and have been chat-
tuned, making them optimized for dialogue use cases Touvron et al. (2023).
When treating depression detection as a classification task, we tested LlaMA2
Chat and found that its performance, both during the Cross-modal Instruction
Fine-Tuning stage and the depression detection phase, was inferior to that of
LlaMA2. We hypothesize two potential reasons for this. The first is that the
Chat version might not be suitable for classification tasks. The second, and
our preferred explanation, is that the Chat version, having been adjusted,
tends to avoid answering questions to mitigate ethical risks. To validate our
hypothesis, we first reimagined the classification task as a generative task,
where the LLMs diagnoses depression through dialogue responses. We tested this
zero-shot scenario on GPT-3.5 and GPT-4. Additionally, we applied LoRA for
instruction fine-tuning in various scenarios presented in Table 2, to observe
how the models perform post-tuning. We observed that when treating depression
detection as a generative task, neither LlaMA2 nor GPT models performed
particularly well, with the dialogue-enhanced LlaMA Chat still underperforming
compared with LlaMA. This suggests that LLMs in the field of depression
detection are subject to certain artificial limitations, impacting their
effectiveness in this specific application. The details of the template can be
seen on Appendix D.

###  5.4 Lora VS P-tuning

From our previous ablation experiments, we found that the conventional method
of incorporating LoRA matrices into attention layers might not be well-suited
for depression detection tasks. After experimenting with applying LoRA
matrices across all layers and conducting a hyperparameter search, we observed
that LoRA, in this context, achieved results similar to those of P-tuning.
Furthermore, in our use of LoRA for classification tasks, we tested a variety
of manually crafted templates. However, none were as effective as using no
task-specific prompt template. We believe this occurs because when we
explicitly inform the LLMs that the task involves depression detection, the
model tends to avoid responses that could pose ethical risks.

##  6 Conclusion

This paper introduces an efficient approach for depression detection using
acoustic landmarks and LLMs. This approach is not only valuable for the
detection of depression but also represents a new perspective in enhancing the
ability of LLMs to comprehend speech signals. Furthermore, we are the first to
research multimodal depression detection using LLMs. We establish a new
benchmark with a SOTA F1-score of 0.84 through ensemble learning.
Additionally, we evaluated various PEFT methods and discovered that applying
Lora across all layers yields identical outcomes for both P-tuning and Lora in
depression detection. Our analysis further reveals how LLMs process speech
landmarks, guiding future research in this domain.

## Limitations

In addition, The study is confined to the DAIC-WOZ dataset, which is currently
the most commonly used and only publicly available dataset in the field of
multimodal depression recognition, particularly in the area of speech. The
difficulty in acquiring data due to numerous privacy concerns surrounding
depression datasets is acknowledged. Despite the limitations of focusing on
this single dataset, it aligns with traditional research methodologies in this
domain, as previous studies have predominantly relied on it.

## Ethics Statement

The DAIC-WOZ datasets are publicly available benchmarks and have been
automatically de-identifed to protect patient privacy. Although our model
improves the factual accuracy of generated reports, its performance still lags
behind the needs of practical deployment. The outputs of our model may contain
false observations and diagnoses due to systematic biases. In this regard, we
strongly urge the users to examine the generated output in real-world
applications cautiously.

## Acknowledgement

This work was supported by Australian Research Council Discovery Project
DP230101184. Additionally, we extend our gratitude to Hanyu Meng for
assistance in creating the graphs for this paper.

## References

  * Bergstra et al. (2011) James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011.  Algorithms for Hyper-Parameter Optimization.  In _Advances in Neural Information Processing Systems_ , volume 24. Curran Associates, Inc.
  * Boyce et al. (2012) Suzanne Boyce, Harriet Fell, and Joel MacAuslan. 2012.  Speechmark: Landmark detection tool for speech analysis.  In _Thirteenth Annual Conference of the International Speech Communication Association_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  In _Proc. Adv. Neural Inf. Process. Syst._ , volume 33, pages 1877–1901.
  * Chen et al. (2022) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1505–1518.
  * Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023.  Palm: Scaling language modeling with pathways.  _Journal of Machine Learning Research_ , 24(240):1–113.
  * Cummins et al. (2011) Nicholas Cummins, Julien Epps, Michael Breakspear, and Roland Goecke. 2011.  An investigation of depressed speech detection: Features and normalization.  In _Twelfth Annual Conference of the International Speech Communication Association_.
  * Défossez et al. (2022) Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022.  High fidelity neural audio compression.  _arXiv preprint arXiv:2210.13438_.
  * DeVault et al. (2014) David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, et al. 2014.  Simsensei kiosk: A virtual human interviewer for healthcare decision support.  In _Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems_ , pages 1061–1068.
  * Garvin (1953) Paul L Garvin. 1953.  Preliminaries to speech analysis: The distinctive features and their correlates.
  * Gong and Poellabauer (2017) Yuan Gong and Christian Poellabauer. 2017.  Topic modeling based multi-modal depression detection.  In _Proceedings of the 7th annual workshop on Audio/Visual emotion challenge_ , pages 69–76.
  * Hao et al. (2023) Hongkun Hao, Long Zhou, Shujie Liu, Jinyu Li, Shujie Hu, Rui Wang, and Furu Wei. 2023.  Boosting large language model for speech synthesis: An empirical study.  _arXiv preprint arXiv:2401.00246_.
  * He et al. (2019) Di He, Xuesong Yang, Boon Pang Lim, Yi Liang, Mark Hasegawa-Johnson, and Deming Chen. 2019.  When ctc training meets acoustic landmarks.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5996–6000. IEEE.
  * He et al. (2021) Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021.  Towards a unified view of parameter-efficient transfer learning.  In _International Conference on Learning Representations_.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460.
  * Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _Proc. Int. Conf. Learn. Representations_.
  * Huang et al. (2019a) Zhaocheng Huang, Julien Epps, and Dale Joachim. 2019a.  Investigation of speech landmark patterns for depression detection.  _IEEE transactions on affective computing_ , 13(2):666–679.
  * Huang et al. (2019b) Zhaocheng Huang, Julien Epps, and Dale Joachim. 2019b.  Speech landmark bigrams for depression detection from naturalistic smartphone speech.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5856–5860. IEEE.
  * Huang et al. (2018) Zhaocheng Huang, Julien Epps, Dale Joachim, and Michael Chen. 2018.  Depression detection from short utterances via diverse smartphones in natural environmental conditions.  In _INTERSPEECH_ , pages 3393–3397.
  * Lahat et al. (2023) Adi Lahat, Eyal Shachar, Benjamin Avidan, Zina Shatz, Benjamin S Glicksberg, and Eyal Klang. 2023.  Evaluating the use of large language model in identifying top research questions in gastroenterology.  _Scientific reports_ , 13(1):4164.
  * Li et al. (2023) Yuang Li, Yu Wu, Jinyu Li, and Shujie Liu. 2023.  Prompting large language models for zero-shot domain adaptation in speech recognition.  _arXiv preprint arXiv:2306.16007_.
  * Liu (1996) Sharlene A Liu. 1996.  Landmark detection for distinctive feature-based speech recognition.  _The Journal of the Acoustical Society of America_ , 100(5):3417–3430.
  * Liu et al. (2023) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023.  Gpt understands, too.  _AI Open_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Noble (2006) William S Noble. 2006.  What is a support vector machine?  _Nature biotechnology_ , 24(12):1565–1567.
  * Oh et al. (2023) Namkee Oh, Gyu-Seong Choi, and Woo Yong Lee. 2023.  Chatgpt goes to the operating room: evaluating gpt-4 performance and its potential in surgical education and training in the era of large language models.  _Annals of Surgical Treatment and Research_ , 104(5):269.
  * Pu et al. (2023) George Pu, Anirudh Jain, Jihan Yin, and Russell Kaplan. 2023.  Empirical analysis of the strengths and weaknesses of peft techniques for llms.  In _ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models_.
  * Shen et al. (2022) Ying Shen, Huiyu Yang, and Lin Lin. 2022.  Automatic depression detection: An emotional audio-textual corpus and a gru/bilstm-based model.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6247–6251. IEEE.
  * Stevens (2002) Kenneth N Stevens. 2002.  Toward a model for lexical access based on acoustic landmarks and distinctive features.  _The Journal of the Acoustical Society of America_ , 111(4):1872–1891.
  * Sun et al. (2022) Hao Sun, Yen-Wei Chen, and Lanfen Lin. 2022.  Tensorformer: A tensor-based multimodal transformer for multimodal sentiment analysis and depression detection.  _IEEE Transactions on Affective Computing_.
  * Sun et al. (2023) Xianghui Sun, Yunjie Ji, Baochang Ma, and Xiangang Li. 2023.  A comparative study between full-parameter and lora-based fine-tuning on chinese instruction data for instruction following large language model.  _arXiv preprint arXiv:2304.08109_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Walker et al. (2018) Jane Walker, Katy Burke, Marta Wanat, Rebecca Fisher, Josephine Fielding, Amy Mulick, Stephen Puntis, Joseph Sharpe, Michelle Degli Esposti, Eli Harriss, et al. 2018.  The prevalence of depression in general hospital inpatients: a systematic review and meta-analysis of interview-based studies.  _Psychological medicine_ , 48(14):2285–2298.
  * Wang et al. (2023) Zhuo Wang, Rongzhen Li, Bowen Dong, Jie Wang, Xiuxing Li, Ning Liu, Chenhui Mao, Wei Zhang, Liling Dong, Jing Gao, et al. 2023.  Can llms like gpt-4 outperform traditional ai tools in dementia diagnosis? maybe, but not today.  _arXiv preprint arXiv:2306.01499_.
  * Wu et al. (2022) Wen Wu, Mengyue Wu, and Kai Yu. 2022.  Climate and weather: Inspecting depression detection via emotion recognition.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6262–6266. IEEE.
  * Wu et al. (2023) Wen Wu, Chao Zhang, and Philip C Woodland. 2023.  Self-supervised representations in speech-based depression detection.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5. IEEE.
  * Zeghidour et al. (2021) Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2021.  Soundstream: An end-to-end neural audio codec.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 30:495–507.
  * Zhang et al. (2023) Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. 2023.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  _arXiv preprint arXiv:2303.16199_.
  * Zhao et al. (2020) Ziping Zhao, Zhongtian Bao, Zixing Zhang, Nicholas Cummins, Haishuai Wang, and Björn Schuller. 2020.  Hierarchical attention transfer networks for depression assessment from speech.  In _ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 7159–7163. IEEE.
  * Zheng et al. (2023) Wenbo Zheng, Lan Yan, and Fei-Yue Wang. 2023.  Two birds with one stone: Knowledge-embedded temporal convolutional transformer for depression detection and emotion recognition.  _IEEE Transactions on Affective Computing_.

##  Appendix A Details of Landmark Detection

###  A.1 General Processing Details

Given a discrete time series signal $x[n]$, the process of peak detection
consists of several pre-processing steps, followed by the identification of
significant peaks. The steps are as follows:

### Six Frequency Bands

The following table describes the six frequency bands we used in our
algorithm.

Table 4: Frequency Bands Band | Frequency Range (kHz)
---|---
1 | 0.0–0.4
2 | 0.8–1.5
3 | 1.2–2.0
4 | 2.0–3.5
5 | 3.5–5.0
6 | 5.0–8.0

### Coarse Smoothing

The signal is first subjected to a coarse smoothing operation to reduce noise
and highlight broader trends. This is achieved by applying a centered moving
average with a window size of $cp\\_sm$:

$L^{(cp)}_{b}[n]=10\cdot\log_{10}\left(\frac{1}{2N_{cp}+1}\sum_{k=-N_{cp}}^{N_{cp}}E_{b}[n+k]\right),$
(4)

where $E_{b}[n]$ is the energy in the $b^{th}$ frequency band at time $n$, and
$N_{cp}$ is half the size of the coarse smoothing window.

### Coarse Differentiation

The smoothed signal undergoes differentiation to identify regions of rapid
change, which could indicate potential peaks. The differentiation is centered
on mitigating delay:

$D^{(cp)}_{b}[n]=L^{(cp)}_{b}[n+cp\\_dt]-L^{(cp)}_{b}[n],$ (5)

followed by a shift to center the result:

$D^{(cp)}_{b}[n]\leftarrow D^{(cp)}_{b}[n-\lfloor cp\\_dt/2\rfloor].$ (6)

### Fine Smoothing

A finer smoothing operation is applied to the original signal to preserve more
detail, with a window size of $fp\\_sm$:

$L^{(fp)}_{b}[n]=10\cdot\log_{10}\left(\frac{1}{2N_{fp}+1}\sum_{k=-N_{fp}}^{N_{fp}}E_{b}[n+k]\right),$
(7)

where $N_{fp}$ is half the size of the fine smoothing window.

### Fine Differentiation

As with coarse differentiation, the finely smoothed signal is differentiated:

$D^{(fp)}_{b}[n]=L^{(fp)}_{b}[n+fp\\_dt]-L^{(fp)}_{b}[n],$ (8)

and then centered:

$D^{(fp)}_{b}[n]\leftarrow D^{(fp)}_{b}[n-\lfloor fp\\_dt/2\rfloor].$ (9)

### Peak Detection

After pre-processing, peaks are identified using the conditions specified
earlier, considering factors such as prominence, height, and minimum distance
between peaks.

Given a signal sequence $x[n]$, the peak detection process can be
mathematically described as follows:

A data point $x[n]$ is considered a local maximum if it satisfies the
following condition:

$x[n]>x[n-1]\quad\text{and}\quad x[n]>x[n+1].$ (10)

If a height threshold $h$ is specified, $x[i]$ is recognized as a peak only
if:

$x[i]>h.$ (11)

The prominence $P$ of a peak at $x[i]$ is defined as the vertical distance
between the peak and its lowest contour line:

$P=x[i]-\max(v_{l},v_{r}),$ (12)

where $v_{l}$ and $v_{r}$ are the lowest points on either side of $x[i]$,
before reaching a higher point. A peak is considered significant if its
prominence exceeds a predefined threshold.

The width $W$ of a peak is measured at a vertical distance $P$ from its
highest point. Points $x[l]$ and $x[r]$, where $l<i<r$, are the positions at
which the signal drops below the threshold defined by the prominence:

$x[l]<x[i]-P\quad\text{and}\quad x[r]<x[i]-P,$ (13)

and the width $W$ is the distance between $x[l]$ and $x[r]$.

If a minimum peak separation distance $D$ is defined, then for any two peaks
$x[i]$ and $x[j]$, the condition must be met:

$|i-j|>D.$ (14)

These conditions are used to identify peaks in the signal that are not only
local maxima but also exceed certain amplitude and prominence thresholds,
ensuring the detected peaks are significant in the context of the signal.

###  A.2 Details of Specific Landmark Detection

g landmark When both the coarse and fine filters exhibit a peak in band 1, it
is identified as a ’g’ landmark.

b landmark In an unvoiced segment (not between +g and the next -g), if at
least three out of five frequency bands demonstrate simultaneous power
increases of no less than 6 dB in both coarse and fine filters, a specific
condition or criterion is met.

s landmark In an unvoiced segment (between +g and the next -g), if at least
three out of five frequency bands demonstrate simultaneous power increases of
no less than 6 dB in both coarse and fine filters, a specific condition or
criterion is met.

f+ and v+ landmarks involves detecting a 6 dB power increase in at least three
high-frequency bands (4, 5, 6), and a power decrease in low-frequency bands
(2, 3). For f- and v-, the criteria are reversed: a 6 dB power decrease in the
same high-frequency bands and a power increase in the low-frequency bands.The
distinguishing factor here is that frication landmarks are detected within
unvoiced segments (b landmark), while voiced frication landmarks are sought in
voiced segments (s landmark).

p landmark, p landmark extraction can be divided into several steps.
1\. Frame Segmentation:
Let the audio signal be $Y(t)$.
Define the frame length $N$ and frame shift $\Delta$.
For the $i$-th frame, we consider the segment
$Y[i\cdot\Delta:i\cdot\Delta+N]$.
2\. Autocorrelation Calculation:
For each frame $Y_{i}$, calculate the autocorrelation function $R_{xx}(k)$:

$R_{xx}(k)=\frac{1}{N-k}\sum_{n=0}^{N-k-1}Y_{i}(n)\cdot Y_{i}(n+k).$

3\. Energy Function Calculation:
Compute the energy function $E_{f}$ for each frame:

$E_{f}(i)=\frac{1}{N}\sum_{k=0}^{N-1}R_{xx}(k)^{2}.$

4\. Upsampling:
Upsample the energy function $E_{f}$ to match the length of the original
signal.
5\. Smoothing:
Apply smoothing(As defined in the previous section) to the upsampled energy
function.
6\. Binarization:
Define a threshold $\theta$, and convert the smoothed energy function into a
binary signal.
7\. Jump Detection:
Detect positive and negative jumps in the binary signal.
8\. P Landmark Index and Time Determination:
Record the positions of jumps, which are the indices of P landmarks.
Convert these indices into time points to determine the P landmarks.

##  Appendix B Details of Data Augmentation

The training set was expanded by shuffling sub-dialogues, selecting portions
$x_{s:e}$ from each full dialogue $x_{1:T}$, with $s$ and $e$ as random start
and end indices. The algorithm outlines this process. Initially, it counts the
positive and negative samples, setting $M^{+}$ as the target number of sub-
dialogues for each positive dialogue (Algorithm 1, lines 1-3). To balance
augmentation, $M^{-}$ is calculated using $N^{+}$, $N^{-}$, and $M^{+}$ (line
4). For both positive and negative dialogues, corresponding $M^{+}$ and
$M^{-}$ sub-dialogues are generated (lines 8-12). The sub-dialogue length,
$d$, is set within the range defined by $\varepsilon_{l}$ and
$\varepsilon_{h}$, chosen randomly (lines 14-15). The start index $s$ is
randomly selected within its range, and the end index $e$ is determined
accordingly (lines 16-18) Wu et al. (2023).

Algorithm 1 Sub-dialogue shuffling

1:$N^{+}\leftarrow$ Number of positive samples in the training set

2:$N^{-}\leftarrow$ Number of negative samples in the training set

3:$M\leftarrow$ Set number of sub-dialogues for each positive sample $M^{+}$

4:$M^{*}\leftarrow N^{-}/N^{+}$

5:Set $\varepsilon_{l},\varepsilon_{h}$ satisfying
$0<\varepsilon_{l}<\varepsilon_{h}\leq 1$

6:for Dialogue $X^{(n)}$ $n=1$ to $N$ do

7: $T\leftarrow\text{len}(x^{(n)})$

8: if $x^{(n)}$ is positive then

9: $M\leftarrow M^{+}$

10: else

11: $M\leftarrow M^{-}$

12: end if

13: for Sub-dialogue $X^{(n)m}$ $m=1$ to $M$ do

14: Sample $\varepsilon$ uniformly from $(\varepsilon_{l},\varepsilon_{h})$

15: $d\leftarrow\varepsilon T-1$

16: Sample $s$ randomly from range $(0,T-d)$

17: $e\leftarrow s+d$

18: $X^{(n)m}\leftarrow x_{s:e}^{(n)}$

19: end for

20:end for

##  Appendix C Sample of Hint Cross-modal Instruction Fine Tuning

Depression Example

    Below are the speech transcripts from a person with depression.
    Please try to predict the concatenated acoustic landmarks
    corresponding to these transcripts

    ### Transcript:
    {transcript}

    ### Acoustic Landmark:
    {landmark}

Healthy Example

    Below are the speech transcripts from a healthy person.
    Please try to predict the concatenated acoustic landmarks corresponding to these transcripts.

    ### Transcript:
    {transcript}

    ### Acoustic Landmark:
    {landmark}

##  Appendix D Sample of Instruction Fine-Tuning for Depression Detection

Text Only

     "Categorize these dialogues as either depression or healthy based on the its transcripts.

     ### transcript:{transcript}

     ### Response:"

Landmark Only

     "Categorize these dialogues as either depression or healthy based on the its acoustic landmarks.

     ### acoustic landmarks:{landmarks}

     ### Response:"

MultiModal

    "Categorize these dialogues as either depression
    or healthy based on the its transcripts and acoustic landmarks.

    ### Transcript:{transcript}

    ### Acoustic Landmark:{landmark}

    ### Response:\n"


††thanks: $@$ Those are the first authors and have equally contributed to this
work††thanks: $@$ Those are the first authors and have equally contributed to
this work

Corresponding author: Sudev Pradhan<EMAIL_ADDRESS>
# Design and Simulation of an Autonomous Quantum Flying Robot Vehicle: An IBM
Quantum Experience

Sudev Pradhan<EMAIL_ADDRESS>Physics Department,
Indian Institute of Science Education and Research, Berhampur, 760010, Odisha,
India  Anshuman Padhi<EMAIL_ADDRESS>School of Physical
Sciences,
National Institute of Science Education and Research, HBNI, Jatni 752050,
Odisha, India  Bikash K. Behera<EMAIL_ADDRESS>Bikash’s Quantum
(OPC) Pvt. Ltd., Balindi, Mohanpur 741246, West Bengal, India

###### Abstract

The application of quantum computation and information in robotics has caught
the attention of researchers off late. The field of robotics has always put
its effort on the minimization of the space occupied by the robot, and on
making the robot ‘smarter. ‘The smartness of a robot is its sensitivity to its
surroundings and the user input and its ability to react upon them desirably.
Quantum phenomena in robotics make sure that the robots occupy less space and
the ability of quantum computation to process the huge amount of information
effectively, consequently making the robot smarter. Braitenberg vehicle is a
simple circuited robot that moves according to the input that its sensors
receive. Building upon that, we propose a quantum robot vehicle that is
‘smart’ enough to understand the complex situations more than that of a simple
Braitenberg vehicle and navigate itself as per the obstacles present. It can
detect an obstacle-free path and can navigate itself accordingly. It also
takes input from the user when there is more than one free path available.
When left with no option on the ground, it can airlift itself off the ground.
As these vehicles sort of ‘react to the surrounding conditions, this idea can
be used to build artificial life and genetic algorithms, space exploration and
deep-earth exploration probes, and a handy tool in defense and intelligence
services.

Quantum Robot, Braitenberg Vehicle, IBM Quantum Experience

##  I Introduction

Robotics has been a major achievement in the field of science and technology
for the last half of 20th century and the 21st century qrv_Craig . As
technology progresses, the dream of robots serving in industries, production,
defense, and different helpful fields have started becoming true qrv_Cai2000 .
For a robot to be called more efficient, it should stand on two criteria: one,
it must be intelligent, and second, it must consume less space. The rise of
sensor technology and advanced computation make the robot more intelligent.
And when we aim to reduce its size to the minimum possible, there comes the
role of quantum effects. Studies in quantum computation have shown that it can
solve various difficult classical problems more efficiently than classical
computers. Quantum computation qrv_Chuang2000  also has the potential to
facilitate communication with higher accuracy and is more efficient in
maintaining the secrecy of quantum information. Hence, quantum computation
should be used in robots to make them more efficient and occupy less space
qrv_dongArXiv . The study of robots and the engineering behind them can
provide motivation for the development of powerful quantum computers.

Robot vehicles are machines that can move almost on their own. The basic
purpose of robot vehicles is to make the vehicles "unmanned" i.e., no humans
are there inside the vehicle to control its motion. Vehicles are equipped with
sensors and computational control units. Although no user is present inside
the vehicle, vehicles can still be designed to consider inputs from a distance
by the user / supervisor qrv_Sand2006 . Braitenberg vehicle is a rudimentary
version of a robot vehicle that can sense the light around it and move
accordingly qrv_RG2019 .

Quantum robotics can trace itself back to 1998, when Benioff qrv_Benioff1998 ;
qrv_Benioff2002  proposed the working of a quantum robot in a strictly quantum
world. Then Raghuvanshi _et al_ showed some evidence of quantum controlled
robots that used the concept of Braitenberg vehicles qrv_Raguvanshi2007 ;
qrv_raghuvanshi2006 . The vehicle sensed its surroundings using classical
light sensors(input) and moved into the environment using classical
motors(output). But the sensed binary information was processed by a quantum
circuit. The quantum circuit somewhat acted like the Controlling unit (CU) of
the robot. The robot had three degrees of freedom, i.e. straight, left, and
right.

Later, Mahanti _et al._ proposed the quantum circuit for a Braitenberg vehicle
with an additional degree of freedom qrv_Mahanti2019 . In their setup, they
made the vehicle fly off the ground in a perpendicular direction, which means
it had an extra degree of freedom that was upward. Later, they went on to
simulate the behavior of their proposed quantum robot deterministically.
Mishra _et al._ used finite automoata to propose a Quantum Braitenberg
vehicle. and were successful in including external control to its movement
qrv_mishra2019 . A month later, the same research group proposed an algorithm
in which complex desired behaviors such as the addition of an additional
degree of freedom could be incorporated into the vehicle system
qrv_mishra1.2019 .

In the current work, we build on the quantum Braitenberg vehicles and propose
a smarter vehicle, which is more sensitive to its surroundings than the
previously discussed models. In this proposal, instead of using light sensors
as in the case of Braitenberg vehicles, here we use IR sensors for better
detection of obstacles; since light as a sensation can be misleading and the
vehicle’s motion might get manipulated by. It has all 4 degrees of freedom
when on the ground (i.e., forward, right, left, and backward). Another
interesting feature of the vehicle is, every time it senses more than one
possible direction which the robot can traverse (meaning that more than one
direction has no obstacles), it asks the user to specify the direction which
it should take. Later, as the user specifies, the robots proceed according to
the instruction. The robot has another feature, that is, its ability to
elevate itself up in the air when there is no path available to traverse on
the ground. For the above functions and features of the robot, we develop a
quantum circuit and realize on the IBM Q platform, and validate all the
results.

(a) A classical Braitenberg Vehicle

(b) A Quantum robot vehicle

Figure 1: textbfThe schematic top view of the classical Braitenberg vehicle
and the proposed quantum robotic vehicle. Figure (a) The light sensors are
named as $S_{1}$ and $S_{2}$ are connected to the motors $M_{L}$ and $M_{R}$
respectively. Both the motors are connected to the respective wheels. Figure
(b) corresponds to the schematic top-view of the proposed quantum robot
vehicle. Instead of having 2 light sensors like the classical one, it is
equipped with 4 IR sensors $S_{1}$, $S_{2}$, $S_{3}$, $S_{4}$ at the front,
back, left and right respectively. Inputs from these sensors is fed into the
Controlling Unit (CU) of the robot and the output of the CU lets the motors
move accordingly. Motors $M_{L}$ and $M_{R}$ are connected to the wheels in
the respective sides. The motor $M_{U}$ is connected to the propeller attached
to the vehicle. When $M_{U}$ is active, the propeller works and the vehicle
takes up from the ground.

Here, we design the required quantum circuit and simulate it by using IBM
Quantum Experience. IBM has recently developed various prototypes of quantum
processors and has made it available through a free web-based, namely, IBM
Quantum Experience. Researchers have used it to their strengths and have been
able to demonstrate and run a variety of experiments involving quantum
computation, e.g. qrv_IBMQE1 ; qrv_IBMQE2 ; qrv_IBMQE3 ; qrv_IBMQE4 ;
qrv_IBMQE5 ; qrv_IBMQE6 ; qrv_IBMQE7 ; qrv_IBMQE8 ; qrv_IBMQE9 ; qrv_IBMQE10 ;
qrv_IBMQE11 ; qrv_IBMQE12 ; qrv_ManabputraQIP2019 . In this experiment, we use
IBM Quantum Experience’s; ‘qasm simulator’ and ‘Jupyter notebook’ to construct
and simulate the circuit.

We organize this paper as follows. Section II describes the theoretical scheme
of the robot vehicle that we tend to propose. Section III proposes the logic
behind the design of the quantum circuit and the various features it will
provide to the vehicle. Section IV describes how the simulation of the
proposed logic was performed and the result of it. Section V explains the
applications of this vehicle in real life. Finally, we conclude this article
by citing the future implications of the proposal in Section VI.

##  II SCHEME OF THE ROBOT VEHICLE

Braitenberg vehicles are made up of light sensors and wheels qrv_MIT1992 . The
fundamental plan behind the vehicle is that, supported the device input, the
wheels mechanically move in a manner accordingly, therefore giving rise to
differing kinds of motion to the wheels which ultimately move the vehicle.
Depending upon the type of connection between the sensors and the wheels, the
movements of the vehicle can be specified when there is a certain type of
external surrounding. The vehicle can also change its motion according to the
changing external environment.

A schematic diagram of a simple Braitenberg vehicle is delineated in Fig.
1(a). As observed, the two light sensors $S_{1}$ and $S_{2}$ sense the
intensity of the incident light. They are connected to the two motors $M_{L}$
and $M_{R}$ in such a way that when a light sensor senses some light, then the
motor connected to it starts rotating and the motor is turned off in the
absence of light. When the robot is exposed to high light intensity, both the
sensors sense light; hence, both the motors get turned on and as a result, the
robot moves forward at full speed. In the case where only the left light
sensor $S_{1}$ senses some light, only the left motor $M_{L}$ starts rotating
and $M_{R}$ does not and that gives the motor a rightward motor moves towards
the right-hand side. It can be attributed to the formation of an instantaneous
axis of rotation around the right wheel, since it is at rest and the body
turns around it, thus having a motion towards right. The same goes for the
case of light coming from the right-hand side, in which case the motor turns
to the left. This behavior of the robot to deviate away from intense light
sources and obstacles can be interpreted as ‘fear’ or ‘shyness’. Similarly, we
can make the robot drive itself towards light by changing the connections in
the circuit, and thus the nature of that sort of Braitenberg vehicle will be
opposite to that of “fear" or “shyness" qrv_MIT1992 .

The original classical Braitenberg vehicle’s motion is determined by the
presence or absence of light on the left and right sides, so it has a
particular disadvantage that its forward motion is dependent on the intensity
inputs of the left and right sensors. In these cases, the motion is very much
dependent on the intensity of the lights, which at times can be unpredictable.
Also, using intensity as input, it may collide with some obstruction that is
bright enough and cause itself some damage. It also has no means of moving
back.

In our work, we propose a modified Braitenberg vehicle in Fig. 1(b) that
senses the obstacles in its path rather than detecting the light intensity.
For this purpose, it is equipped with 4 IR sensors, each at the front, back,
left and right. The sensor inputs give us an idea on which side there are
obstructions to the robot, and on which side the path is clear. The inputs are
then fed into the quantum circuit we have designed. Out of 4 directions, if
exactly one direction is obstacle-free, the circuit will give output to the
motors to make the movement possible in that direction. If more than 1
direction is available to be traveled upon, then the circuit lets the robot
ask the user to specify which direction is to be traveled. Along with the
$M_{L}$ and $M_{R}$ motors on each side, an additional motor $M_{U}$ is
attached to the propeller of the robot. When there are obstructions from all 4
sides, the propeller will rotate to let the robot take off from the ground and
fly upward.

When there are obstructions in front, left and right, the robot will also be
able to move backward.

##  III Quantum Scheme

The proposed robot has 4 sensors $S_{1}$, $S_{2}$, $S_{3}$, $S_{4}$ each at
the front, back, left, and right, respectively. Each sensor is assigned 1
qubit each. We denote the state $|0\rangle$ when the sensor detects an
obstacle and $|1\rangle$ when the sensor detects no obstacle on its front. The
input of all 4 sensors will say which are the directions around the vehicle
that are devoid of any obstacle (Table 1). There are 3 motors $M_{L}$, $M_{R}$
and $M_{U}$ each attached to the left wheel, right wheel, and the propeller at
the top, respectively. $M_{L}$ and $M_{R}$ are assigned 2 qubits each.
$|00\rangle$ state of these motors indicates that the wheel is not rotating at
all, $|10\rangle$ means that the wheel is moving forward, and $|01\rangle$
indicates that the wheel is moving backward (Table 2).

$M_{U}$ is attached to the propeller and is assigned only 1 qubit. The
$|0\rangle$ state indicates that the propeller is not rotating, and the
$|1\rangle$ state indicates that the propeller is rotating. When the propeller
is rotating (i.e. $M_{U}$ has a $|1\rangle$ state) the robot takes off from
the ground.

Input received by | Remarks
---|---
all sensors $|S_{1}S_{2}S_{3}S_{4}\rangle$ |
$|0000\rangle$ | Obstruction at all 4 sides
$|0001\rangle$ | Right path clear
$|0010\rangle$ | Left path clear
$|0100\rangle$ | Back path clear
$|1000\rangle$ | Front path clear
$|1100\rangle$ |
$|1010\rangle$ |
$|1001\rangle$ |
$|0110\rangle$ |
$|0101\rangle$ | Cleared path in more
$|0011\rangle$ | than one direction
$|1110\rangle$ |
$|1101\rangle$ |
$|1011\rangle$ |
$|0111\rangle$ |
$|1111\rangle$ |
Table 1: All possible inputs and their interpretations State of the qubits | Corresponding action
---|---
attached to the motor | on the wheel attached to it
$|00\rangle$ | No rotation
$|01\rangle$ | Wheel rotates and moves backwards
$|10\rangle$ | Wheel rotates and moves forward
Table 2: State of the qubits assigned to the motor and the corresponding
output on the wheel attached to that motor

The quantum circuit is designed such that, when the inputs collectively
exhibit a $|1000\rangle$ state, which implies there are obstructions in left,
right and back and a free path in the front; both the $M_{L}$ and $M_{R}$ have
a forward motion i.e. both the motors have $|10\rangle$ states. Similarly,
when the sensor input gives a $|0001\rangle$ state, it indicates a vacancy in
the backside of the robot, and thus the output of the circuit will result in
both $M_{L}$ and $M_{R}$ having a backward motion, i.e. both have a
$|01\rangle$ state. When the input reads $|0010\rangle$, that is, there is a
vacancy only on the left and obstructions on the rest of all directions; to
make the robot turn left, the left wheel is made stationary and the right
wheel is made to move forward, and as discussed earlier, it makes the vehicle
turn left. Hence, the design of the quantum circuit is such that, when
receiving an input of $|0010\rangle$, $M_{L}$ shows an output $|00\rangle$,
while $M_{R}$ shows an output $|10\rangle$. Similarly, upon receiving an input
$|0001\rangle$ (vacancy on the right), the right motor $M_{R}$ must be
stationary, i.e., state $|00\rangle$, and the left motor $M_{L}$ must have a
value $|10\rangle$ state.

When the inputs read $|0000\rangle$ i.e. there are obstacles on all 4 sides
and there is no path for the robot to move on the ground; the circuit renders
the $M_{U}$ qubit to be in the $|1\rangle$ state, the propellor starts
rotating and the robot takes off from the ground.

There may be cases in which there can be more than 1 direction available to be
traveled upon. This implies that the circuit input shows more than one
$|1\rangle$ state out of the 4 sensors (the inputs like $|0011\rangle$,
$|1001\rangle$, $|1110\rangle$, etc.). In those cases, we aim for the robot to
stand still, without moving anywhere on its own, and asks the user to specify
which direction to move. Hence we define one ‘ASK’ qubit in the circuit. When
in the $|0\rangle$ state, it does not ask and keeps on moving as per the motor
qubits. But when in the $|1\rangle$ state, it asks the user to specify the
direction. The user then specifies the direction through the quantum computer
control it has, and the robot moves in that direction. Taking into account all
these, we should devise a quantum circuit such that it satisfies Table 3,
since it describes how the circuit should behave in terms of its input and
output, so that the desired property is obtained.

Input received | Left | Right | Propellor | ASK | Action of
---|---|---|---|---|---
by all sensors | Motor | Right | Motor | qubit | the vehicle
$|S_{1}S_{2}S_{3}S_{4}\rangle$ | ML($|ab\rangle$) | MR($|ab\rangle$) |  |  |
$|0000\rangle$ | $|00\rangle$ | $|00\rangle$ | $|1\rangle$ | $|0\rangle$ | flies up
$|0001\rangle$ | $|10\rangle$ | $|00\rangle$ | $|0\rangle$ | $|0\rangle$ | right
$|0010\rangle$ | $|00\rangle$ | $|10\rangle$ | $|0\rangle$ | $|0\rangle$ | left
$|0100\rangle$ | $|01\rangle$ | $|01\rangle$ | $|0\rangle$ | $|0\rangle$ | back
$|1000\rangle$ | $|10\rangle$ | $|10\rangle$ | $|0\rangle$ | $|0\rangle$ | front
$|1100\rangle$ | $|00\rangle$ | $|00\rangle$ | $|0\rangle$ | $|1\rangle$ |
$|1010\rangle$ | $|00\rangle$ | $|00\rangle$ | $|0\rangle$ | $|1\rangle$ |
$|1001\rangle$ | $|00\rangle$ | $|00\rangle$ | $|0\rangle$ | $|1\rangle$ |
$|0110\rangle$ | $|00\rangle$ | $|00\rangle$ | $|0\rangle$ | $|1\rangle$ |
$|0101\rangle$ | $|00\rangle$ | $|00\rangle$ | $|0\rangle$ | $|1\rangle$ | no move
$|0011\rangle$ | $|00\rangle$ | $|00\rangle$ | $|0\rangle$ | $|1\rangle$ | asks
$|1110\rangle$ | $|00\rangle$ | $|00\rangle$ | $|0\rangle$ | $|1\rangle$ |
$|1101\rangle$ | $|00\rangle$ | $|00\rangle$ | $|0\rangle$ | $|1\rangle$ |
$|1011\rangle$ | $|00\rangle$ | $|00\rangle$ | $|0\rangle$ | $|1\rangle$ |
$|0111\rangle$ | $|00\rangle$ | $|00\rangle$ | $|0\rangle$ | $|1\rangle$ |
$|1111\rangle$ | $|00\rangle$ | $|00\rangle$ | $|0\rangle$ | $|1\rangle$ |
Table 3: Details of the robot behaviour upon the sensation of obstruction.

From the table, we can observe that we have 16 types of cases for inputs (as
we have total 4 sensors). We prepare a conditional circuit with 16 segments.
Each segment is prepared from a four-control Toffoli gate, conditioned to
receive only a specific input of 16. When that input is available, only that
segment out of all 16 works and only the NOT gate (s) attached to that Toffoli
gate operate accordingly on the qubits to be measured, and thus we obtain our
desirable outputs.

From this table, we can propose a quantum circuit in Fig. 3. Suppose, for the
input $|0001\rangle$, we can see that only the first segment (i.e., the first
Toffoli gate) gets activated, and now as per Fig. 3, the corresponding NOT
gate operates on the first qubit of $M_{L}$ (i.e., $M_{L}^{A}$) and the state
of the left motor $M_{L}$ changes from $|00\rangle$ to $|11\rangle$.
Similarly, for each segment (i.e. each possibility of input) the NOT gates are
implemented as targets on different qubits to obtain the desired output.

##  IV QUANTUM CIRCUIT AND SIMULATION

###  IV.1 Working and Decomposition of four-control-one target and four-
control-two-target Toffoli gate-

The operation of a 4-control Toffoli gate with one target ($C^{4}(X_{5})$) is
as follows. There are 4 control qubits and one NOT gate. When the control
qubits are satisfied, the NOT gate flips the target qubit. However, since it
is difficult to prepare a four-control-one-target Toffoli gate, we break it
into a combination of two-control-one-target and one-control-one-target
Toffoli gates (refer to Fig. 2). But to do so, we additionally will require
three more qubits, named Extra Qubits (Ex 1, Ex 2, Ex 3). The sole purpose of
the extra qubits is to facilitate the breakdown of a four-control Toffoli gate
into a combination of two-control Toffoli gates. 7 two-control Toffoli gates
and 1 one-control Toffoli gate are used to denote a four-control, one-target
Toffoli gate.

###  IV.2 Circuit Diagram and Simulation result-

We use Fig. 4 for the proposed experiment. The first 4 qubits (q[0] to q[3])
are used as the auxiliary qubits. They represent the inputs received by the
sensors at Front, Back, Left, and Right, respectively. The next 3 qubits
(q[4]-q[6]) are the ‘extra’ qubits discussed in the previous subsection. They
will be used as buffer to accommodate the decomposed four-control-one-target
Toffoli gate. Qubits q[7] and q[8] correspond to the left motor, which has
been assigned two qubits. Similarly, q[9] and q[10] correspond to the two
qubits of the motor at right. In this proposed circuit, q[11] is the qubit
attached with the motor connected to the propeller. The final qubit q[12] is
associated with the robot asking for the specification of direction to its
user. While the qubits q[0]-q[3] are the inputs to the circuit, the qubits
q[7]-q[12] are taken for measurement. After this we use the appropriate gates,
as discussed in the previous subsection, to prepare the IBM quantum circuit in
Fig.4, ready for simulation.

Figure 2: Equivalent quantum circuit for four-control-one-target ($C^{4}(X_{5})$) Toffoli gate. The four-control-one-target Toffoli gate is fed with 3 extra qubits, which act as a buffer when the Four-control-one-target Toffoli gate is decomposed into two-control-one-target Toffoli gates. It is to be observed that the extra qubits undergo two NOT gates in total, hence they will always show $|0\rangle$ as output. Figure 3: Quantum circuit for implementing the proposed protocol of the robot vehicle. The Qubits F, B, L, and R imply the input from the sensors $S_{1}$, $S_{2}$, $S_{3}$, $S_{4}$ respectively. The motors $M_{L}$ and $M_{R}$ are assigned two qubits each, namely ‘A’ and ‘B’. The motor $M_{U}$ is assigned 1 qubit and the robot is assigned a qubit to ASK the user to clarify the direction. The qubits of the motors $M_{L}$, $M_{R}$, $M_{U}$ and the ASK qubit are measured as output. It should be noted that there are 16 vertical segments (Four-controlled Toffoli gates) in it, each representing one of the total 16 types of possible inputs. Figure 4: Construction of the Quantum Circuit using IBM Q Experience Input | Output | Action
---|---|---
$|S_{1}S_{2}S_{3}S_{4}\rangle$ | $|(Ask)M_{U}M_{R}^{B}M_{R}^{A}M_{L}^{B}M_{L}^{A}\rangle$ |
$|0000\rangle$ | $|010000\rangle$ | Flies Up
$|0001\rangle$ | $|000001\rangle$ | Takes right
$|0010\rangle$ | $|000100\rangle$ | Takes left
$|0100\rangle$ | $|001010\rangle$ | Moves back
$|1000\rangle$ | $|000101\rangle$ | Moves forward
$|1100\rangle$ | $|100000\rangle$ |
$|1010\rangle$ | $|100000\rangle$ |
$|1001\rangle$ | $|100000\rangle$ |
$|0101\rangle$ | $|100000\rangle$ | Doesn’t move
$|0110\rangle$ | $|100000\rangle$ | and asks
$|0011\rangle$ | $|100000\rangle$ | the user
$|1110\rangle$ | $|100000\rangle$ | to specify
$|1101\rangle$ | $|100000\rangle$ | direction
$|1011\rangle$ | $|100000\rangle$ |
$|0111\rangle$ | $|100000\rangle$ |
$|1111\rangle$ | $|100000\rangle$ |
Table 4: Result of Simulation of Quantum Circuit

When the ancillary qubits were varied and all 16 possible cases were tested,
the obtained result was tabulated in Table 4.

##  V ADVANTAGE and APPLICATIONS

The proposed and simulated quantum robot vehicle has more degrees of freedom
than the previously proposed ones. It can autonomously move to its left,
right, back and forth, subject to the condition that there is no obstacle in
that path. It is smart enough to handle complex situations like the presence
of more than one possible path to traverse. In conditions like these, it will
always ask the user to further specify the direction. When left without
options on the ground, it can also take off from the ground using a propellor.
The drone mechanism can be later added to make it move in all directions in
the air as well.

The application of quantum phenomena ensures that it has a minimized size and
reduced weight. As it can also be controlled by the user when necessary, it
can go to places that are unreachable or hazardous to humans. This can open up
the possibility of a great field of scientific inquiry. Like the one we did,
quantum robots can handle complex situations effectively, without much effort
going onto it. Robots with a full-fledged quantum control unit can also ensure
that the secrecy of the information obtained remains intact. The use of
quantum computation and information in the future can make it fit enough to be
used as rovers for planetary and deep-earth explorations. It can also serve
intelligence and defense agencies in protecting their territory.

##  VI CONCLUSION

The application of quantum computing in robotics can be a boon for
technological advancements in the 21st century. It started with Benioff giving
an idea about a quantum robot in a purely quantum domain. Scientists have
taken inspiration from this and have been able to realize the potential that
lies in the application of quantum phenomena in the field of robotics. It can
be seen that quantum robots can solve complex problems more efficiently than
classical computers. We can improve the setup further by using quantum
sensors, and it can be made faster by using parallel computation and learning
of quantum algorithms.

Braitenberg vehicles have been used for mimicking animal movements, and thus
setting up a whole field of scientific inquiry on artificial life, genetic
algorithms etc. qrv_atificial2018 . Using quantum computation can effectively
analyze the complex behaviors that animals show, such as ‘fear’ and
‘attraction’ qrv_RG2019 ; which governs their movements and actions. This can
later lead to the incorporation of various other emotions, and a quantum
emotional robot can be thought of qrv_MIT1992 ; qrv_Mahanti2019 .

We used quantum computation to make the robot navigate. Similarly, there is a
wide scope for quantum computation and information in robot action planning
and vision, which can further speed up the processes involved in robots,
making them compact and efficient.

###### Acknowledgements.

A.P. and S.P. would like to thank Bikash’s Quantum (OPC) Pvt. Ltd. for
providing hospitality during the course of this project. B.K.B. acknowledges
the prestigious Prime Minister’s Research Fellowship awarded by DST, Govt.
India. We acknowledge the support of IBM Quantum Experience for producing the
basic circuits. The views expressed are those of the authors and do not
reflect the official policy of IBM or IBM Quantum Experience team.

## Data Availability

Data will be made available on reasonable request.

## Competing interests

The authors declare that they have no competing financial and nonfinancial
interests. Readers are welcome to comment on the online version of the paper.
Correspondence and requests for materials should be addressed to S.P.
(sudev18@iiserbpr.ac.in).

## References

  * (1) J. J. Craig, Introduction to Robotics: Mechanics and Control, Third Edition, Pearson Education Interna-tional.
  * (2) Z. X. Cai, Robotics (Tsinghua University Press. Beijing, China, 2000)
  * (3) M. Nieslsen and I. Chuang, Quantum Computation and Quantum Information, Cambridge University Press, 2000.
  * (4) D. Dong, C. Chen, C. Zhang, and Z. Chen, Quantum robot: structure, algorithms and applications, arXiv:0810.3283 (2008).
  * (5) A. Sanderson, G. Bekey and B. Wilcox, Robotic Vehicles, Ch 2, Assessment of International Research and Development in Robotics (2006).
  * (6) I. Rano, A model and formal analysis of Braitenberg vehicles 2 and 3, Proceedings of IEEE International Conference on Robotics and Automation, May 2012
  * (7) V. Braitenberg, Vehicles: Experiments in Synthetic Psychology. MIT Press; Reprint edition (1986).
  * (8) P. Benioff, Quantum robots and environments, Phys. Rev. A 58, 893 (1998).
  * (9) P. Benioff, AMS Contemporary Math Series305, (2002).
  * (10) A. Raghuvanshi, Y. Fan, M. Woyke, and M. Perkowski, Quantum Robots for Teenagers, Proceedings of the37th International Symposium on Multiple-Valued Logic, May 13 - 16, 2007.
  * (11) A. Raghuvanshi, Drive Kinematics in Quantum Braiten-berg Vehicles, Report, 2006.
  * (12) S.Mahanti, S.Das, B.K. Behera, and P. K. Panigrahi, Quantum Robots Can Fly; Play Games: An IBM Quantum Experience, Quantum Inf. Process. 18, 219 (2019).
  * (13) N. Mishra, S. C. Rayala, B. K. Behera, and P. K. Panigrahi, Automation of quantum Braitenberg vehicles using finite automata: Moore machines, Quantum Inf. Process. 19, 17 (2020).
  * (14) N. Mishra, S. C. Rayala, B. K. Behera, and P. K. Panigrahi, Adding degrees of freedom to automated quantum Braitenberg vehicles, DOI: 10.13140/RG.2.2.34787.50723, (2019).
  * (15) S. Roy, B. K. Behera and P. K. Panigrahi, Experimental Realization of Quantum Violation of Entropic Noncontextual Inequality in Four Dimension Using IBM Quantum Computer, arXiv:1710.10717 (2017).
  * (16) K. Srinivasan, S. Satyajit, B. K. Behera, and P. K. Panigrahi, Efficient quantum algorithm for solving travelling salesman problem: An IBM quantum experience, arXiv:1805.10928 (2018).
  * (17) S. Satyajit, K. Srinivasan, B. K. Behera and P. K. Panigrahi, Nondestructive discrimination of a new family of highly entangled states in IBM quantum computer, Quantum Inf. Process. 17, 212 (2018).
  * (18) B. K. Behera, T. Reza, A. Gupta, and P. K. Panigrahi, Designing Quantum Router in IBM Quantum Computer, Quantum Inf. Process. 18, 328 (2019).
  * (19) P. K. Vishnu, D. Joy, B. K. Behera and P. K. Panigrahi, Experimental Demonstration of Non-local Controlled-Unitary Quantum Gates Using a Five-qubit Quantum Computer, Quantum Inf. Process. 17, 274 (2018).
  * (20) B. K. Behera, S. Seth, A. Das and P. K. Panigrahi, Demonstration of Entanglement Purification and Swapping Protocol to Design Quantum Repeater in IBM Quantum Computer. Quantum Inf. Process. 18, 108 (2019).
  * (21) A. Padhi, D. Priyadarshi, B. K. Behera and P. K. Panigrahi, Design of Quantum Circuits to Play Chess in A a Quantum Computer, DOI:10.13140/RG.2.2.28760.03848 (2019).
  * (22) S. Gangopadhyay, Manabputra, B. K. Behera, and P. K. Panigrahi, Generalization and demonstration of an entanglement-based Deutsch–Jozsa-like algorithm using a 5-qubit quantum computer. Quantum Inf. Process. 17, 160 (2018).
  * (23) A. Dash, S. Rout, B. K. Behera and P. K. Panigrahi, Quantum Locker Using a Novel Verification Algorithm and Its Experimental Realization in IBM Quantum Computer, arXiv:1710.05196 (2017).
  * (24) S. Pradhan, T. S. Behera, B. K. Behera, and P. K. Panigrahi, Simulating Quantum Synchronization of Atomic Spin Model on IBM Q Experience, DOI:10.13140/RG.2.2.22783.15521 (2020).
  * (25) R. Jha, D. Das, A. Dash, S. Jayaraman, B. K. Behera and P. K. Panigrahi, A Novel Quantum N-Queens Solver Algorithm and its Simulation and Application to Satellite Communication Using IBM Quantum Experience, arxiv:1806.10221 (2018).
  * (26) A. Dash, D.Sarmah, B. K. Behera, and P. K. Panigrahi, Exact search algorithm to factorize large biprimes and a triprime on IBM quantum computer. arxiv:1805.10478 (2018).
  * (27) Manabputra, B. K. Behera, and P. K. Panigrahi, A simulational model for witnessing quantum effects of gravity using IBM quantum computer, Quantum Inf. Process. 19, 119 (2020).
  * (28) U. Alvarez-Rodriguez, M. Sanz, L. Lamata _et al._ , Quantum Artificial Life in an IBM Quantum Computer. Sci. Rep. 8, 14793 (2018).