# AUTOMATIC PREDICTION OF SUICIDAL RISK IN MILITARY COUPLES USING MULTIMODAL
INTERACTION CUES FROM COUPLES CONVERSATIONS

###### Abstract

Suicide is a major societal challenge globally, with a wide range of risk
factors, from individual health, psychological and behavioral elements to
socio-economic aspects. Military personnel, in particular, are at especially
high risk. Crisis resources, while helpful, are often constrained by access to
clinical visits or therapist availability, especially when needed in a timely
manner. There have hence been efforts on identifying whether communication
patterns between couples at home can provide preliminary information about
potential suicidal behaviors, prior to intervention. In this work, we
investigate whether acoustic, lexical, behavior and turn-taking cues from
military couples’ conversations can provide meaningful markers of suicidal
risk. We test their effectiveness in real-world noisy conditions by extracting
these cues through an automatic diarization and speech recognition front-end.
Evaluation is performed by classifying 3 degrees of suicidal risk: none,
ideation, attempt. Our automatic system performs significantly better than
chance in all classification scenarios and we find that behavior and turn-
taking cues are the most informative ones. We also observe that conditioning
on factors such as speaker gender and topic of discussion tends to improve
classification performance.

Index Terms—  Suicidal Risk, Couples Conversations, Prosody, Behavior, ASR

##  1 Introduction

According to the Centers for Disease Control and Prevention, 39,518 suicides
were reported in 2011, making suicide the 10th leading cause of death overall
and the leading non-natural cause of death for
Americans111https://www.cdc.gov/nchs/fastats/suicide.htm. Among military
personnel, rates of suicide are even higher [1]. One of the biggest challenges
to improving the success of suicide prevention efforts in the military is the
absence of reliable methods for predicting who will engage in suicidal
behaviors and when they will do so. This restricts our ability to identify at-
risk military personnel and ensure that they are getting the best available
and most appropriate treatment for their psychological symptoms. It also
impacts not only them but also their spouses who are at increased risk for a
wide range of psychological and physical health symptoms [2, 3].

Clinical interviews and surveys are the best available methods for identifying
if and when a person is at increased or heightened risk. However, these
methods do not work well for measuring suicide risk in
soldiers222https://theactionalliance.org/sites/default/files/agenda.pdf. The
major limitations of clinical interviews are that they require in-person
interaction with a health care professional and that most service members who
die by suicide do not interact with such professionals immediately prior to
the event. Likewise, surveys about suicidal thoughts and feelings are not
informative if military personnel are unwilling to acknowledge or are unaware
of their psychological distress. The ability to assess risk directly at home
is, therefore, considered valuable, for which machine learning (ML) is being
examined as a viable platform.

There is significant literature on using ML for identifying attributes related
to suicidal risk; we refer readers to Burke et al. [4] for a comprehensive
review. Most works either deal with static, non-interactive scenarios such as
microblog posts [5, 6] and written answers [7, 8], or with interactive but
highly structured settings such as interviews with therapists [9] and social
workers [10, 11]. Existing works typically use information from only one
modality such as text [7, 8] or audio [9, 10, 12]. Recently, there have been
efforts on using multimodal approaches for quantifying suicidal risk [11, 13],
which is the topic of interest in our work as well. However, despite this
progress, these approaches are often constrained by heavy manual supervision
that ensures clean, directly usable features but also greatly limits the scope
of their deployment in real-world conditions. As a result, it is not feasible
to deploy these methods at home for suicidal risk assessment.

Distressed couples conversations, which have been well studied within the
broad realm of family studies [14, 15, 16], offer a potential setting for
performing such assessments. They have been well analyzed using ML-based
computational approaches that have been found to be useful across a variety of
behavioral and health domains [17, 18]. For instance, in Couples Therapy [15],
multiple works have effectively quantified behaviors related to speakers’
mental states such as Blame, Positive and Sadness using the speaker’s language
[19] and vocal traits [20]. Similarly, in Cancer Care [21] interactions,
lexical and acoustic cues have been found to be useful in predicting Hostile
and Positive behaviors [22].

In this paper, we investigate whether military personnel’s conversations with
their spouses at home can provide useful markers of their suicidal risk. We
compute various multimodal features relating to behavior, emotion and turn-
taking in order to build a comprehensive profile of their communication
patterns. Finally, we test the effectiveness of our work in real-world
conditions by extracting all of our features from raw, noisy data in an
ecologically-meaningful manner. Our system uses an automatic diarization and
speech recognition front-end with operating conditions that require limited
manual supervision and is, hence, readily deployable.

##  2 Dataset

Our dataset consists of 62 mixed-sex couples, a total of 124 individuals. They
were recruited for a study of behavioral and cognitive markers of suicide risk
among geographically dispersed military service members. The study criteria
required that they be in the National Guard, a Reserve Component, or a recent
Veteran who served during the Operation Enduring Freedom / Operation Iraqi
Freedom era, be married/cohabitating, at least 18 years old, be fluent in
English and have reliable internet access at home.

Based on their history of suicidal behaviors, each person was assigned one of
3 labels: (1) none if they had no history, (2) ideation if they had
experienced suicide thoughts but did not act on it and (3) attempt if they had
attempted suicide in the past. According to the World Health Organization, a
prior attempt is the most important risk factor for
suicide333https://www.who.int/en/news-room/fact-sheets/detail/suicide; hence,
these labels represent the degree of suicidal risk, from none representing no
risk to attempt representing severe risk. Table 1 shows the demographics of
the participants.

Gender \Label | none | ideation | attempt
---|---|---|---
Husband | 31 | 22 | 9
Wife | 34 | 15 | 13
Table 1: Suicidal risk demographics of 62 couples in our data

As part of their participation, couples completed 2 relationship-change (RC)
and 1 reasons-for-living (RFL) conversations or “sessions”, in their homes,
each one video-recorded for 10 minutes. In the RFL session, they were asked to
discuss what they found meaningful, or what their reasons for living were. In
the RC sessions, they were asked to discuss one of their top areas of
discontent and conflict in their relationship, with each person getting to
select the topic in one session. We denote the session where the wife picked
the topic as W-Conflict and the one where the husband picked as H-Conflict. We
obtained audio streams from all but one session of one couple, resulting in
370 sessions.

In general, the audio quality was observed to be good; however, some couples
recorded their interactions in a noisy environment or did not sit close enough
to the recorder to be clearly intelligible. Nevertheless, we retained all
their samples consistent with the goal of this work of using data reflecting
real-world operational use conditions.

##  3 Feature Extraction

###  3.1 Automatic Data Processing

####  3.1.1 Diarization

The first, important step towards automatically analyzing couples
conversations is speaker diarization, i.e. identifying “who spoke when” during
the interaction. We employ the x-vector [23] based diarization system proposed
in [24] for extracting speaker embeddings in each speech segment and
clustering them using the spectral clustering approach described in [24]. As
part of this approach, a pruning parameter $p$ is tuned to maximize the
performance of the diarization, for which we obtained speaker labels for a
small subset of our data444We thank the members of USC SCUBA lab for manually
annotating speaker IDs and their corresponding speech segment timestamps.

Once we obtained the speaker labels $S_{1}$ and $S_{2}$, the pitch of their
corresponding segments was extracted and the ID Husband was assigned to the
label with lower median pitch and Wife to the remaining label. Manual
inspection of a dozen sessions at random revealed acceptable accuracy of the
ID assignment.

####  3.1.2 Automatic Speech Recognition

We used the Kaldi [25] ASpIRE chain model555https://kaldi-asr.org/models/m1
but adapted the Language Model (LM) on related psychotherapy data in order to
improve recognition accuracy. Adaptation was performed by interpolating, with
equal weights, 3 LMs that were trained on ASpIRE [26], cantab-TEDLIUM [27] and
a mix of Couples Therapy [15] and Motivational Interviewing [28] corpora,
using SRILM [29]. We built the session transcripts using 1-best hypotheses;
spurious word insertions in noisy segments were eliminated with the help of
word confusion heuristics such as number of confusions and confidence score.

###  3.2 Features

####  3.2.1 Acoustic Low-level Descriptors (A)

Acoustic features have been shown to be useful in prior work as markers of
suicide risk [11, 30, 10] and depression [30, 31]. We used OpenSMILE [32] for
extracting the standard openEAR Emobase feature set [33] from the speech
segments of each speaker separately. This set includes various prosody
features (pitch, intensity etc), voice quality features (jitter, shimmer etc),
and spectral features (MFCCs, line spectral frequencies etc). Then we took six
statistical functionals (e.g. mean, standard deviation) over the low-level
descriptors to obtain 228 session-level features for each speaker.

####  3.2.2 Acoustic Behavior Embeddings (E)

Speaker behaviors and mental states such as blame, negativity and depression
can provide clues about their suicidal risk [34]. To capture this information,
we extract behavior embeddings from the acoustic channel using the reduced
Context-Dependent model proposed in [35] which employs emotions as primitives
for facilitating behavior quantification and is trained on the Couples Therapy
corpus [15].

We employ two sets of embeddings: (1) a 5-dimensional score vector $s$ from
the model’s final prediction layer corresponding to 5 behaviors: Acceptance,
Blame, Positive, Negative, Sadness, with higher score denoting stronger
behavior, and (2) a 128-dimensional hidden representation $h$ from the model’s
penultimate layer. We extract these from 26 different model configurations and
concatenate them in the following fashion $[h_{1},h_{2},\ldots
h_{26},s_{1},s_{2},\ldots s_{26}]$, where $h_{i}$ and $s_{i}$ denote the
hidden representation and score vector of the $i^{th}$ configuration. This
gives us 3458 session-level features per speaker.

####  3.2.3 Lexical Cues (L)

Since emotion expressed in language has been reported previously [11] to be
associated with suicidal risk, we computed count-based statistics of LIWC [36]
positive and negative emotion words from the session transcripts. 6 lexical
features were extracted: the proportions of both emotions in the speaker’s
language throughout the session, followed by the log-ratios of the speaker’s
and their partner’s proportions for all 4 combinations of emotions (e.g. log-
ratio of speaker-negative-proportion to partner-positive-proportion, etc.).

####  3.2.4 Turn-Taking Cues (T)

The dynamics of turn-taking and pausing during an interaction have been linked
to suicidal risk and psychological distress in multiple studies [11, 30, 13].
To capture them, we extract features relating to speech duration, number of
words, speech rate and pause for every speaker and also compute differences
between the speaker’s and their partner’s features during a turn change. This
is performed locally in every turn as well as globally over the entire session
where applicable. For each of the local features, we derive first and second-
order temporal differences (delta ($\Delta$), delta-delta ($\Delta$$\Delta$))
and compute 9 session-level statistical functionals such as min, median, and
quartiles on top of them. This results in 167 session-level features.

##  4 Methodology

Correlation | Feature
---|---
-0.219 |  sadness score from $s_{18}$ (E)
-0.196 |  sadness score from $s_{17}$ (E)
-0.175 |  positive score from $s_{18}$ (E)
0.173 |  std-dev($\Delta$ no. of words per turn) (T)
0.172 | std-dev(no. of words per turn) (T)
Table 2: Top 5 features most correlated with suicidal risk (feature set in parentheses). All correlations statistically significant $(p<0.05)$ Set | Correlation | Feature
---|---|---
A | 0.1696 | line spectral pair coefficient
E | -0.219 |  sadness score from $s_{18}$
L | 0.105 |  log($\frac{\text{speaker-{negative}-proportion}}{\text{speaker-{positive}-proportion}}$)
T | 0.173 |  std-dev($\Delta$ no. of words per turn)
Table 3: Feature most correlated with suicidal risk, for every feature set.
All correlations are statistically significant $(p<0.05)$

###  4.1 Analysis

In order to understand how our features are related to suicidal risk, we
compute their Spearman’s rank correlation with the degree of risk. Tables 2
and 3 show the top 5 most correlated features overall, and the most correlated
feature per feature set respectively. We see that behavior features have the
highest correlation despite not being fine-tuned on our domain; this suggests
the benefit of transfer learning from Couples Therapy. Along with turn-taking
cues, they appear to be the most promising features. However, we observe weak
monotonic relations in our features, which underscores the need for multimodal
approaches, while suggesting possible non-linearities in how our features
relate to suicidal risk.

###  4.2 Experimental Setup

While the primary focus of this work is to accurately classify the 3 degrees
of a person’s suicidal risk, we are also interested in investigating the 2
constituent “one-versus-rest” binary classification scenarios. These could be
of importance in scenarios where the goal is to distinguish no risk from some
risk or to identify and isolate attempt, the most important risk factor as
mentioned in Sec. 2. Hence, we perform classification experiments for the
following 3 scenarios:

  1. 1.

Degree of Risk: none vs ideation vs attempt

  2. 2.

No-Risk vs Risk: none vs {ideation / attempt}

  3. 3.

Non-Severe vs Severe Risk: {none / ideation} vs attempt

Behavior expression patterns are known to vary across speakers of different
genders [14] and are also likely to be influenced by the nature and topic of
the interaction [16]. RFL sessions, for instance, are typically marked by
long, introspective monologues whereas Conflict sessions involve vigorous
back-and-forth exchanges over the marital issue. To examine whether these
factors (gender, topic) have an impact on the classification performance, we
employ the following data partitioning schemes in our experiments:

  1. 1.

None: Same model for all speakers, sessions (1 model)

  2. 2.

Gender: Separate models for Husband, Wife (2 models)

  3. 3.

Content: Separate models in RFL, Conflict (2 models)

  4. 4.

Demand: Separate models for Husband, Wife in RFL; separate models for Wife in
W-Conflict, H-Conflict; same model for Husband in Conflict (5 models)

The Demand partition is designed based on findings in [16].

###  4.3 Classification

A Support Vector Machine (SVM) was used as the classifier and multimodal
features were tested through feature-level fusion. Feature dimensionality
reduction was applied using Principal Component Analysis such that 95% of the
total energy was retained. We applied sample weighting to address class
imbalance and tuned hyperparameters such as feature normalization scheme (min-
max, z-score), SVM kernel (linear, rbf), SVM penalty $C$ and rbf influence
$\gamma$ (both $10^{-5},10^{-4.5},\ldots 10^{5}$) to optimize the classifier.

We used leave-one-couple-out cross-validation where, in fold $i$, couple
$C_{i}$ was picked as the test split and the remaining couples
$C_{j},j$$\neq$$i$ were randomly assigned to an $80$:$20$ train:validation
split such that both splits had similar label distributions and no couple
appeared in more than 1 split. The classifier was then trained on the train
split, optimized on the validation split and used to predict the suicidal risk
of the speaker(s) in couple $C_{i}$. Since our dataset contains 62 couples,
this procedure was repeated for all 62 folds and results were accumulated from
all the folds.

Performance was evaluated using macro-average recall of classification in
order to account for class imbalance. To determine whether there were
statistically significant ($p<0.05$) differences between our results and
chance, we ran the McNemar’s test for binary classifications and the Stuart-
Maxwell test for the 3-class scenario. In partition experiments, a separate
classifier was created for each partition and at test time, the appropriate
one was used.

##  5 Results & Discussion

Scenario |  | Degree
---
of Risk
| No-Risk
---
vs Risk
| Non-Severe vs
---
Severe Risk
Chance | 33.33 | 50.00 | 50.00
Best system |  39.60∗ |  60.32∗ |  56.77∗
Feature Sets | A + E + T | E + L + T | L
Table 4: Average Recall % and features of best system in different
classification scenarios. * denotes statistically significant ($p<0.05$)

Table 4 shows the best classification performance and its corresponding
features for each scenario. We see that our system performs 13% - 20%
(relative) better than chance in all 3 scenarios, with differences being
statistically significant ($p$ $<$ $0.05$) and each feature set contributing
to the best system in at least result. In line with observations in Sec. 4.1,
the best features in 2 out of 3 scenarios consist of behavior and turn-taking
cues. Our findings are similar to those of previous works [11, 12] which found
affect-based and interaction dynamics cues to be important in identifying
suicidal risk markers. This suggests that emotional regulation can serve as a
useful source of features for these tasks.

Separating no-risk from risk appears to be an easier problem than isolating
severe risk from the rest. One explanation for this could be that the
communication patterns of ideation are more similar to attempt than none and
are, thus, harder to distinguish. Another reason, however, could be the high
class imbalance arising from combining none and ideation into a single class,
leaving the attempt class with much fewer samples to train on. Therefore, a
more in-depth investigation on a larger dataset is required in order to better
understand this issue.

Table 5 shows the best classification performance of each partition in every
scenario. We see that partition-based systems perform better than None in 2
out of 3 scenarios, despite having lesser data to train on for each
classifier. Partitioning based on Gender appears to be useful for risk
classification, in general, whereas Demand seems to be suited only to the
scenario with severe risk; Content, on the other hand, does not provide any
noticeable benefits over other partitions. This suggests that partitioning
based on gender in conjunction with topic is a promising direction for further
exploration, especially in mixture-of-experts frameworks.

Partition |  | Degree
---
of Risk
| No-Risk
---
vs Risk
| Non-Severe vs
---
Severe Risk
Chance | 33.33 | 50.00 | 50.00
None | 37.78 | 60.32∗ |  56.48∗
Gender | 39.60∗ | 59.03 |  54.86∗
Content | 37.46 | 58.28 |  52.10∗
Demand |  36.60∗ | 53.90 | 56.77∗
Table 5: Best average recall % of partitions in different classification
scenarios. Bold value is best performance overall in that scenario. * denotes
statistically significant difference from chance ($p<0.05$)

##  6 Conclusions & Future Work

In this work, we demonstrated the feasibility of an automated, multimodal
approach to classifying military couples’ suicidal risk by observing their
conversations at home in real-world noisy conditions. We also found that
conditioning on speaker gender and discussion topic could benefit the
classification of risk categories in specific scenarios. For our future work,
we will employ dynamic processing of cues instead of session-level aggregates
and leverage information from the visual modality. We also plan on
incorporating frameworks that explicitly characterize couples dynamics, such
as entrainment and influence, into our modeling.

##  7 Acknowledgments

The U.S. Army Medical Research Acquisition Activity is the awarding and
administering acquisition office. This work is supported by the Office of
Assistant Secretary of Defense for Health Affairs through the Psychological
Health and Traumatic Brain Injury Research Program under Award No.
W81XWH-15-1-0632. The funders had no role in study design, data collection and
analysis, decision to publish, or preparation of the manuscript.

## References

  * [1] Mark S Kaplan, Nathalie Huguet, Bentson H McFarland, and Jason T Newsom,  “Suicide among male veterans: a prospective population-based study,”  Journal of Epidemiology & Community Health, vol. 61, no. 7, pp. 619–624, 2007.
  * [2] Nansook Park,  “Military children and families: strengths and challenges during peace and war.,”  American Psychologist, vol. 66, no. 1, pp. 65, 2011.
  * [3] Steven L Sayers,  “Family reintegration difficulties and couples therapy for military veterans and their spouses,”  Cognitive and Behavioral Practice, vol. 18, no. 1, pp. 108–119, 2011.
  * [4] Taylor A Burke, Brooke A Ammerman, and Ross Jacobucci,  “The use of machine learning in the study of suicidal and non-suicidal self-injurious thoughts and behaviors: A systematic review,”  Journal of affective disorders, 2018.
  * [5] Scott R Braithwaite, Christophe Giraud-Carrier, Josh West, Michael D Barnes, and Carl Lee Hanson,  “Validating machine learning algorithms for twitter data against established measures of suicidality,”  JMIR mental health, vol. 3, no. 2, pp. e21, 2016.
  * [6] Qijin Cheng, Tim MH Li, Chi-Leung Kwok, Tingshao Zhu, and Paul SF Yip,  “Assessing suicide risk and emotional distress in chinese social media: A text mining and machine learning study,”  Journal of medical internet research, vol. 19, no. 7, pp. e243, 2017\.
  * [7] Benjamin L Cook, Ana M Progovac, Pei Chen, Brian Mullin, Sherry Hou, and Enrique Baca-Garcia,  “Novel use of natural language processing (nlp) to predict suicidal ideation and psychiatric symptoms in a text-based mental health intervention in madrid,”  Computational and mathematical methods in medicine, vol. 2016, 2016\.
  * [8] John P Pestian, Jacqueline Grupp-Phelan, Kevin Bretonnel Cohen, Gabriel Meyers, Linda A Richey, Pawel Matykiewicz, and Michael T Sorter,  “A controlled trial using natural language processing to examine the language of suicidal adolescents in the emergency department,”  Suicide and Life-Threatening Behavior, vol. 46, no. 2, pp. 154–159, 2016.
  * [9] Daniel Joseph France, Richard G Shiavi, Stephen Silverman, Marilyn Silverman, and M Wilkes,  “Acoustical properties of speech as indicators of depression and suicidal risk,”  IEEE transactions on Biomedical Engineering, vol. 47, no. 7, pp. 829–837, 2000.
  * [10] Stefan Scherer, John Pestian, and Louis-Philippe Morency,  “Investigating the speech characteristics of suicidal adolescents,”  in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2013, pp. 709–713.
  * [11] Verena Venek, Stefan Scherer, Louis-Philippe Morency, John Pestian, et al.,  “Adolescent suicidal risk assessment in clinician-patient interaction,”  IEEE Transactions on Affective Computing, vol. 8, no. 2, pp. 204–215, 2017.
  * [12] John Gideon, Heather T Schatten, Melvin G McInnis, and Emily Mower Provost,  “Emotion recognition from natural phone conversations in individuals with and without recent suicidal ideation,”  in The 20th Annual Conference of the International Speech Communication Association INTERSPEECH 2019, 2019.
  * [13] John P Pestian, Michael Sorter, Brian Connolly, Kevin Bretonnel Cohen, Cheryl McCullumsmith, Jeffry T Gee, Louis-Philippe Morency, Stefan Scherer, Lesley Rohlfs, and STM Research Group,  “A machine learning approach to identifying the thought markers of suicidal subjects: a prospective multicenter trial,”  Suicide and Life-Threatening Behavior, vol. 47, no. 1, pp. 112–121, 2017.
  * [14] Christopher L Heavey, Christopher Layne, and Andrew Christensen,  “Gender and conflict structure in marital interaction: A replication and extension.,”  Journal of consulting and clinical psychology, vol. 61, no. 1, pp. 16, 1993.
  * [15] A. Christensen, D.C. Atkins, S. Berns, J. Wheeler, D.H. Baucom, and L.E. Simpson,  “Traditional versus integrative behavioral couple therapy for significantly and chronically distressed married couples,”  Journal of Consulting and Clinical Psychology, vol. 72, no. 2, pp. 176–191, 2004.
  * [16] Brian R Baucom, Pamela T McFarland, and Andrew Christensen,  “Gender, topic, and time in observed demand–withdraw interaction in cross-and same-sex couples.,”  Journal of Family Psychology, vol. 24, no. 3, pp. 233, 2010.
  * [17] Shrikanth Narayanan and Panayiotis G Georgiou,  “Behavioral signal processing: Deriving human behavioral informatics from speech and language,”  Proceedings of the IEEE, vol. 101, no. 5, pp. 1203–1233, 2013.
  * [18] Daniel Bone, Chi-Chun Lee, Theodora Chaspari, James Gibson, and Shrikanth Narayanan,  “Signal processing and machine learning for mental health research and clinical applications [perspectives],”  IEEE Signal Processing Magazine, vol. 34, no. 5, pp. 196–195, 2017\.
  * [19] Panayiotis G Georgiou, Matthew P Black, Adam C Lammert, Brian R Baucom, and Shrikanth S Narayanan,  ““that’s aggravating, very aggravating”: Is it possible to classify behaviors in couple interactions using automatically derived lexical features?,”  in International Conference on Affective Computing and Intelligent Interaction. Springer, 2011, pp. 87–96.
  * [20] Matthew P Black, Athanasios Katsamanis, Brian R Baucom, Chi-Chun Lee, Adam C Lammert, Andrew Christensen, Panayiotis G Georgiou, and Shrikanth S Narayanan,  “Toward automating a human behavioral coding system for married couples’ interactions using speech acoustic features,”  Speech communication, vol. 55, no. 1, pp. 1–21, 2013.
  * [21] Maija Reblin, Richard E Heyman, Lee Ellington, Brian RW Baucom, Panayiotis G Georgiou, and Susan T Vadaparampil,  “Everyday couples’ communication research: Overcoming methodological barriers with technology,”  Patient education and counseling, vol. 101, no. 3, pp. 551–556, 2018.
  * [22] Sandeep Nallan Chakravarthula, Haoqi Li, Shao-Yen Tseng, Maija Reblin, and Panayiotis Georgiou,  “Predicting behavior in cancer-afflicted patient and spouse interactions using speech and language,”  Proc. Interspeech 2019, pp. 3073–3077, 2019.
  * [23] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur,  “X-vectors: Robust DNN embeddings for speaker recognition,”  in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Apr. 2018, pp. 5329–5333.
  * [24] Tae Jin Park, Manoj Kumar, Nikolaos Flemotomos, Monisankha Pal, Raghuveer Peri, Rimita Lahiri, Panayiotis Georgiou, and Shrikanth Narayanan,  “The second dihard challenge: System description for usc-sail team,”  Proc. Interspeech 2019, pp. 998–1002, 2019.
  * [25] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al.,  “The kaldi speech recognition toolkit,”  in IEEE 2011 workshop on automatic speech recognition and understanding. IEEE Signal Processing Society, 2011, number CONF.
  * [26] Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L Seltzer, and Sanjeev Khudanpur,  “A study on data augmentation of reverberant speech for robust speech recognition,”  in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 5220–5224.
  * [27] Will Williams, Niranjani Prasad, David Mrva, Tom Ash, and Tony Robinson,  “Scaling recurrent neural network language models,”  in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, pp. 5391–5395.
  * [28] David C Atkins, Mark Steyvers, Zac E Imel, and Padhraic Smyth,  “Scaling up the evaluation of psychotherapy: evaluating motivational interviewing fidelity via statistical text classification,”  Implementation Science, vol. 9, no. 1, pp. 49, 2014.
  * [29] Andreas Stolcke,  “Srilm-an extensible language modeling toolkit,”  in Seventh international conference on spoken language processing, 2002.
  * [30] Nicholas Cummins, Stefan Scherer, Jarek Krajewski, Sebastian Schnieder, Julien Epps, and Thomas F Quatieri,  “A review of depression and suicide risk assessment using speech analysis,”  Speech Communication, vol. 71, pp. 10–49, 2015.
  * [31] Michelle Hewlett Sanchez, Dimitra Vergyri, Luciana Ferrer, Colleen Richey, Pablo Garcia, Bruce Knoth, and William Jarrold,  “Using prosodic and spectral features in detecting depression in elderly males,”  in Twelfth Annual Conference of the International Speech Communication Association, 2011.
  * [32] Florian Eyben, Felix Weninger, Florian Gross, and Björn Schuller,  “Recent developments in opensmile, the munich open-source multimedia feature extractor,”  in Proceedings of the 21st ACM international conference on Multimedia. ACM, 2013, pp. 835–838.
  * [33] Florian Eyben, Martin Wöllmer, and Björn Schuller,  “Openear—introducing the munich open-source emotion and affect recognition toolkit,”  in 2009 3rd international conference on affective computing and intelligent interaction and workshops. IEEE, 2009, pp. 1–6.
  * [34] Curtis D Proctor and Victor K Groze,  “Risk factors for suicide among gay, lesbian, and bisexual youths,”  Social work, vol. 39, no. 5, pp. 504–513, 1994.
  * [35] Haoqi Li, Brian Baucom, and Panayiotis Georgiou,  “Linking emotions to behaviors through deep transfer learning,”  arXiv preprint arXiv:1910.03641, 2019.
  * [36] James W Pennebaker, Martha E Francis, and Roger J Booth,  “Linguistic inquiry and word count: Liwc 2001,”  Mahway: Lawrence Erlbaum Associates, vol. 71, no. 2001, pp. 2001, 2001.