[affiliation=1]OrchidChetia Phukan* [affiliation=2]SarthakJain*
[affiliation=3]ShubhamSingh* [affiliation=4]MuskaanSingh [affiliation=1]Arun
BalajiBuduru [affiliation=1,5]RajeshSharma

# ComFeAT: Combination of Neural and Spectral Features for Improved Depression
Detection

###### Abstract

In this work, we focus on the detection of depression through speech analysis.
Previous research has widely explored features extracted from pre-trained
models (PTMs) primarily trained for paralinguistic tasks. Although these
features have led to sufficient advances in speech-based depression detection,
their performance declines in real-world settings. To address this, in this
paper, we introduce ComFeAT, an application that employs a CNN model trained
on a combination of features extracted from PTMs, a.k.a. neural features and
spectral features to enhance depression detection. Spectral features are
robust to domain variations, but, they are not as good as neural features in
performance, suprisingly, combining them shows complementary behavior and
improves over both neural and spectral features individually. The proposed
method also improves over previous state-of-the-art (SOTA) works on E-DAIC
benchmark.

###### keywords:

Depression Detection, Spectral Features, Neural Features

##  1 Introduction

Depression is a prevalent mental health disorder that affects more than 280
million people world-wide. It is often characterized by persistent sadness,
loss of interest, and other symptoms that impact daily functioning
111https://www.who.int/news-room/fact-sheets/detail/depression. Depression can
manifest uniquely in each individual, sometimes alongside other mental health
conditions or can be the cause of other mental heatlh condition such as
anxiety. Early intervention can significantly help in improving outcomes and
restoring the quality of life for those affected by depression. Recent studies
in the field of speech processing and ML have shown promising results in
automatically identifying various speech characteristics that are associated
with depression such as low pitch, reduced pitch variability, slower speaking
rate, increased pause frequency, and changes in voice quality.

Previous research have explored various features for effective depression
detection ranging from spectral features such as MFCC [1] also features from
pre-trained models (PTMs) [2, 3] i.e neural features. Amongst the features,
features from PTMs specifically trained for paralinguistic tasks such as
speaker recognition, speech emotion recognition, and so on have shown better
performance, and widely preferred [4, 3]. However, they still lack in
performance in real-world settings due to various variances present in real-
world. In this demonstration, we present, ComFeAT, an application that makes
use of CNN-based model trained with combination of neural and spectral
features for improved depression detection in real-world. We experiment with
different state-of-the-art (SOTA) spectral and neural features with CNN as the
modeling paradigm and the topmost performance is shown by models trained on
combination of neural and spectral features. This combination methodology also
improves over previous SOTA works on E-DAIC benchmark.

Figure 1: Proposed Model Architecture

##  2 ComFeAT

In this section, we discuss the various parts of ComFeAT application. First,
we discuss the features considered in our experiments, followed by the
classifier, database used, and evaluation results. Lastly, we discuss ComFeAT
application buildup, workflow and the user interface.

Features: We use MFCC and LFCC as spectral features. For neural features, we
choose, TRILLsson and x-vector which has been commonly used in previous works
[4, 3]. TRILLsson [5] is distilled version of paralinguistic conformer which
has been trained primarily for paralinguistic tasks while x-vector for speaker
recognition. TRILLsson has shown SOTA performance in various paralinguistic
tasks, and x-vector in speaker recognition. We extract features of 1024, 512
dimension size from TRILLsson and x-vector respectively, through averaging
across time.

Modeling: We use CNN for our experiments with different features. For
experimentation with individual features we use 1D-CNN after the features
followed by max-pooling and a fully-connected network (FCN) with two dense
layers of 256, 90 neurons. Lastly, the output layer with mean-squared-error
(MSE) as loss function. We use Adam as optimizer and we employ dropout to
reduce overfitting. For experiments with combination of features, we employ
the same modeling pattern with the 1D-CNN, max-pooling and concatenation for
feature fusion followed by FCN and it is shown in Figure 1. We use 32 filters
with size 3 in the 1D-CNN layer.

Database and Data Preprocessing: We use the EDAIC dataset [6] comprising audio
recordings of individuals of varying emotional state, including depression. It
contains more than 500 participants annotated with depression severity levels
and demographic information. We resample the audio to 16KHz before feature
extraction.

Figure 2: Flow Diagram Table 1: Evaluation Scores; Lower the MAE and RMSE better the model performance Input Features | MAE | RMSE
---|---|---
LFCC | 6.02 | 7.07
MFCC | 5.35 | 6.71
x-vector | 5.03 | 6.32
TRILLsson | 4.48 | 5.83
x-vector + MFCC | 4.39 | 5.66
TRILLsson + MFCC | 4.05 | 5.57
x-vector + LFCC | 4.55 | 5.57
TRILLsson + LFCC | 4.23 | 5.28
SOTA [7] | — | 8.19

Evaluation Results: We present our results in Table 1 for different input
features. Among the individual features, TRILLsson features attained the
topmost performance. Models on combination of TRILLsson + MFCC and TRILLsson +
LFCC attained the topmost performance amongst all the models in terms of MAE
and RMSE, thus, showing the superiorty in combining neural and spectral
features. We also achieve SOTA performance in terms of RMSE in comparison to
previous SOTA work.

##  3 ComFeAT: Buildup and User Interface

In this section, we present our ComFeAT buildup, workflow, and user interface
(UI) in Figure 2.

Buildup: The application front-end is built using ReactJS, and the back-end is
built using Flask. Flask is used for exposing the API of the application,
where information to and fro from the proposed model flows.

Workflow and User Interface: The workflow of ComFeAT is shown in Figure 2. A
raw audio file in .wav, .mp3 etc is uploaded by a user via the upload audio
button. This request is sent to the backend where the audio file is further
pre-processed before passing it to the model and the preprocessing steps
include resampling to 16kHz, converting to uni channel (stereo to mono if
not), and extracting features. Now, the model makes predictions on the input
features received. The predicted output is returned with a POST request to the
frontend (UI). From the predicted value, the user can differentiate the
intensity of depression.

##  4 Conclusion

In this demonstration, we will present, ComFeAT, for real-world depression
detection. We leverage the combination of neural and spectral features for
improved depression detection than previous works. ComFeAT will find
applications in various domains such as assisting mental health professionals,
educational institutions and student support centers, community mental health
support centers, and many more.

## References

  * [1] A. K. Das and R. Naskar, ``A deep learning model for depression detection based on mfcc and cnn generated spectrogram features,'' _Biomedical Signal Processing and Control_ , vol. 90, p. 105898, 2024.
  * [2] R. Flores, M. Tlachac, E. Toto, and E. A. Rundensteiner, ``Depression screening using deep learning on follow-up questions in clinical interviews,'' in _2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA)_. IEEE, 2021, pp. 595–600.
  * [3] R. Brueckner, N. Kwon, V. Subramanian, N. Blaylock, and H. O’Connell, ``Audio-based detection of anxiety and depression via vocal biomarkers,'' in _Future of Information and Communication Conference_. Springer, 2024, pp. 124–141.
  * [4] E. L. Campbell, J. Dineley, P. Conde, F. Matcham, K. M. White, C. Oetzmann, S. Simblett, S. Bruce, A. A. Folarin, T. Wykes, S. Vairavan, R. J. B. Dobson, L. Docio-Fernandez, C. Garcia-Mateo, V. A. Narayan, M. Hotopf, and N. Cummins, ``Classifying depression symptom severity: Assessment of speech representations in personalized and generalized machine learning models.'' in _Proc. INTERSPEECH 2023_ , 2023, pp. 1738–1742.
  * [5] J. Shor and S. Venugopalan, ``TRILLsson: Distilled Universal Paralinguistic Speech Representations,'' in _Proc. Interspeech 2022_ , 2022, pp. 356–360.
  * [6] J. Gratch, R. Artstein, G. M. Lucas, G. Stratou, S. Scherer, A. Nazarian, R. Wood, J. Boberg, D. DeVault, S. Marsella _et al._ , ``The distress analysis interview corpus of human and computer interviews.'' in _LREC_. Reykjavik, 2014, pp. 3123–3128.
  * [7] F. Ringeval, B. Schuller, M. Valstar, N. Cummins, R. Cowie, L. Tavabi, M. Schmitt, S. Alisamir, S. Amiriparian, E.-M. Messner _et al._ , ``Avec 2019 workshop and challenge: state-of-mind, detecting depression with ai, and cross-cultural affect recognition,'' in _Proceedings of the 9th International on Audio/visual Emotion Challenge and Workshop_ , 2019, pp. 3–12.