# I2CKD : Intra- and Inter-Class Knowledge Distillation for Semantic
Segmentation

###### Abstract

This paper proposes a new knowledge distillation method tailored for image
semantic segmentation, termed Intra- and Inter-Class Knowledge Distillation
(I2CKD). The focus of this method is on capturing and transferring knowledge
between the intermediate layers of teacher (cumbersome model) and student
(compact model). For knowledge extraction, we exploit class prototypes derived
from feature maps. To facilitate knowledge transfer, we employ a triplet loss
in order to minimize intra-class variances and maximize inter-class variances
between teacher and student prototypes. Consequently, I2CKD enables the
student to better mimic the feature representation of the teacher for each
class, thereby enhancing the segmentation performance of the compact network.
Extensive experiments on three segmentation datasets, i.e., Cityscapes, Pascal
VOC and CamVid, using various teacher-student network pairs demonstrate the
effectiveness of the proposed method.

Index Terms—  Knowledge distillation, semantic segmentation, deep learning,
class prototype, triplet loss

##  1 Introduction

Semantic segmentation is a challenging task characterized as a distinctive
classification problem where a semantic label is assigned to each pixel in the
input image. Owing to its capability to extract fine-grained information, this
computer vision task plays a crucial role in various real-world applications,
including autonomous vehicles, remote sensing, robot vision and healthcare
[1].

In recent years, deep neural networks have achieved impressive performance in
semantic segmentation. However, this good performance comes at the expense of
heavy storage and computation, which impedes their deployment on resource-
limited edge devices. To address this issue, some compact segmentation
networks are proposed such as ESPNet [2], ENet [3] and BiseNet [4]. Another
category of approaches tries to compress the existing cumbersome models, which
can be categorized into pruning [5], quantization [6], and knowledge
distillation (KD) [7]. Pruning-based methods strive to reduce the network size
by eliminating unnecessary connections between neurons in adjacent layers.
Quantization seeks to decrease the number of bits per network weight by
substituting data types, such as using 8-bit integers instead of 32-bit
floating-point numbers. In both the pruning and quantization schemes, the
compressed network is trained independently of the cumbersome one which affect
drastically the good performance. As a promising mechanism, KD [7] transfer
the knowledge from a cumbersome network to a compact one. The former is called
teacher and the latter is called student. Since the KD trains a student with
the guidance of the teacher, it achieves impressive results in classification
[8] and consequently is extended to the semantic segmentation task. In this
context, the common point of the proposed methods in the literature is the
exploitation of the knowledge at feature maps levels because semantic
segmentation needs a structured output. In order to accomplish this, the
state-of-the-art focus on developing new pixel-wise [9, 10, 11, 12], channel-
wise [13, 14, 15] or image-wise [16] correlations between teacher and student
feature maps. In addition, the ground-truth and the knowledge transferred at
score maps level is often taken into consideration. Generally, the quality of
the knowledge between the teacher and student is computed in densely pairwise
manner using several metrics such as Mean Square Error (MSE) without
consideration of the intra- and inter-class features relation.

In this paper, we propose a novel KD method tailored to semantic segmentation,
called Intra- and Inter-Class Knowledge Distillation (I2CKD). First, we
compute the teacher prototypes (centroids) of each class using the feature
maps and the ground-truth (mask). The hypothesis behind our work is that a
teacher achieve a good performance thanks to their suitable produced
prototypes. Thus, we propose to transfer such knowledge to the student. For
that, as the name of method implies, we force the student prototypes of each
class to mimic the teacher ones by minimizing/maximizing the intra-/inter-
class distances between them (see Figure 1). As a sophistical loss for such
constraint, we exploit the triplet loss.

Fig. 1: Our motivation is to minimize the intra-class distances between
teacher and student prototypes of the same class ($p_{c}^{t}$ and $p_{c}^{s}$)
and to maximize the inter-class distances between teacher and student
prototypes of the different classes ($p_{j}^{t}$ $\forall j\neq c$ and
$p_{c}^{s}$). The dashed arrow refers to distance.

The organization of this paper is as follows. After providing an overview of
the related literature in Section 2, we describe the proposed distillation
method in Section 3. In Section 4, we present and discuss the experimental
results. Finally, we draw the conclusion in Section 5.

##  2 Related work

###  2.1 Knowledge distillation

The core of KD lies in the formulation and transfer of knowledge from teacher
(cumbersome) network to student (compact) one. Generally, the KD methods used
in images classification can be roughly divided into three categories [17]:
response-based knowledge, feature-based knowledge, and relation based-
knowledge. The response-based and feature-based KD force the student to mimic
the soft targets (last layer) or the feature maps (intermediate layer) of the
teacher respectively. These two categories exploit the knowledge extracted
from a single data sample. Instead, relation-based KD consider the
relationships across the different layer (cross-layer) or different samples of
a dataset (cross-sample). In their pioneer work, Hinton et al. [7] propose a
response-based KD scheme that aims to minimize the Kullback-Leibler Divergence
(KLD) between the probabilistic predictions of the teacher and the student. In
addition to the knowledge extracted and distilled at the output level, Romero
et al. [18] helps the student learning using the intermediate representations
of the teacher. Other methods try to convert the hidden features of the
teacher and the student using several techniques such as Attention [19] and
Gramian matrices [20]. The difference of the teacher and student knowledge is
usually measured using different metrics such as KLD, $L1$\- norm, $L2$-norm
and cosine-similarity. Park et al. [21] present a relation-based knowledge
method by modeling the structural relations of all outputs instead of
individual output. The relationships between training samples are captured
using distance-wise and angle-wise functions. Liu et al. [22] study the
instances relations using a graph. The quality of teacher/student knowledge is
computed via an Euclidean distance between the vertexes and between edges.

###  2.2 Knowledge distillation for semantic segmentation

The Fully Convolutional Networks [23] such as PSPNet [24] and DeepLab [25]
have dominated the leaderboard on several semantic segmentation dataset.
Inspiring by the success of KD in image classification, some methods rely on
this technique to train compact FCN networks by capturing the correlation
information among pixels, channels and images. In [9], Xie et al. extracts two
types of knowledge: zero order and first order. The zero order computes the
difference between the pixels class probabilities. Regarding to the first
order, the difference between the center pixel and its $8$-neighborhood is
considered. He et al. [12] focus on matching the feature maps size of teacher
and student, and propose to apply an auto-encoder. After that, a pairwise
affinity map is computed to quantify the relationship between the teacher and
student knowledge. Liu et al. [10] propose a Structured Knowledge Distillation
(SKD) scheme by considering the intermediate distillation between feature maps
using a graph, the pixel distillation between score maps, and holistic
distillation through adversarial learning. Wang et al. [11] develop a new
method called Intra-class Feature Variation Distillation (IFVD) that learn the
student to mimic the intra-class relation of the teacher. To transfer the IFV
knowledge, the cosine distance is used. Shu et al. [14] develop a method
called Channel-Wise knowledge Distillation (CWD) that compute the KLD between
the softmax of the teacher and student activation channel maps along the
channel dimension. In our previous work [26], we leverage the interdepencies
between teacher feature maps as knowledge to distill. This knowledge is
captured using the self-attention mechanism and transferred using the MSE.

While attaining satisfactory performance, these methods neglect the meaningful
knowledge of each class in feature maps (prototypes) which can effectively
used to measure the intra- and inter-class similarity between teacher and
student networks. That is the purpose of our method I2CKD.

##  3 Proposed method

###  3.1 Overview

As illustrated in the Figure 2, our method I2CKD distills the knowledge from a
well-trained teacher network to a student. For each training epoch of student
network, we froze the teacher network. The student network updates its weights
through three losses that capture, respectively, the difference
student/ground-truth and the difference teacher/student at score maps and
feature maps levels. The core of the contribution lies at the feature maps
level where we propose to exploit the intra- and inter-class relations between
teacher and student. For doing so, we compute the triplet loss between the
teacher and the student class prototypes.

Fig. 2: Flowchart of the proposed method: I2CKD. The student network is
trained via three losses: (1) a cross-entropy loss $\mathcal{L}_{task}$, (2) a
KLD between score maps $\mathcal{L}_{SM}$ , and (3) a triplet loss between
prototype classes $\mathcal{L}_{I2CKD}$. At feature maps position, we aim to
learn the student network to mimic the intra- (to minimize) and the inter-
class (to maximize) relations of the teacher through a triplet loss. We
underline that the teacher network is frozen during the student network
training.

###  3.2 Intra and inter-class knowledge distillation

####  3.2.1 Class prototypes computing

The prototype of a class $c$ for a given channel of feature maps $F^{(x,y)}$
is formulated as follows:

$p_{c}=\frac{\sum_{x,y}F^{(x,y)}\mathbbm{1}[M^{(x,y)}=c]}{\sum_{x,y}\mathbbm{1}[M^{(x,y)}=c]}$
(1)

where $M^{(x,y)}$ represents the ground-truth (mask). $\mathbbm{1}[\cdot]$ is
an indicator function that equals to $1$ if the argument is true or $0$
otherwise.

For both the teacher and student networks, we compute the prototype of all
considered classes at the feature maps levels. The resulting matrix has the
size of $\mathbb{R}^{C\times K}$, where $C$ and $K$ denote the number of
classes and channels respectively.

####  3.2.2 Triplet loss

The ultimate objective of our distillation scheme is to minimize the intra-
class and to maximize the inter-class variances between teacher and student
network. Specifically, we aim to enforce the following constraint:

$d(p_{c}^{s},p_{c}^{t})\leq d(p_{c}^{s},p_{j}^{t})+m,\forall j\neq c$ (2)

where $d(\cdot)$ is a distance function and $m$ represents a constant margin.
$p^{t}$ and $p^{s}$ are the class prototypes extracted from the teacher and
student networks respectively. $p^{t}_{j}$ refers to the prototype of a
different class than $c$ and $m$ is a given margin.

The enforcement of this constraint allows to formulate the loss between
student and teacher class prototypes as follows:

$\mathcal{L}_{I2CKD}=\frac{1}{C\cdot(C-1)}\sum_{c=1}^{C}\sum_{\begin{subarray}{c}j=1\\\
j\neq c\end{subarray}}^{C}\ [m+\lVert p_{c}^{s}-p_{c}^{t}\rVert_{2}-\lVert
p_{c}^{s}-p_{j}^{t}\rVert_{2}]_{+}$ (3)

with $[\cdot]_{+}$ denotes the function $\max\\{0,\cdot\\}$ and
$\lVert\cdot\rVert_{2}$ is the $l^{2}$-norm.

###  3.3 Overall student training loss

The overall student training loss for semantic segmentation is as follows:

$\mathcal{L}_{Total}=\lambda_{I2CKD}\mathcal{L}_{I2CKD}+\lambda_{SM}\mathcal{L}_{SM}+\mathcal{L}_{task}$
(4)

where

  * •

$\lambda_{I2CKD}$ and $\lambda_{SM}$ are the hyper-parameters that balance the
considered losses

  * •

$\mathcal{L}_{SM}$ is the KLD between the teacher score maps $y^{T}$ and the
student score maps $y^{S}$ [14]:

$\displaystyle\mathcal{L}_{SM}$ $\displaystyle=\varphi(y^{T},y^{S})$ (5)
$\displaystyle=\frac{\mathcal{T}_{SM}^{2}}{C}\sum_{k=1}^{K}\sum_{i=1}^{W\cdot
H}\phi(y_{k,i}^{T})\cdot\log\biggl{[}\frac{\phi(y_{k,i}^{T})}{\phi(y_{k,i}^{S})}\biggr{]}$

where $\phi(\cdot)$ is the softmax function:

$\phi(y_{k,i})=\frac{\exp(\frac{y_{k,i}}{\mathcal{T}})}{\sum_{i=1}^{N}\exp(\frac{y_{k,i}}{\mathcal{T}})}$
(6)

with $\mathcal{T}$ is the temperature parameter proposed by Hinton et al. [7]
to adjust the smoothness of decision probabilities.

  * •

$\mathcal{L}_{task}$ is the loss between the ground-truth and the segmented
image. In our work, we use the cross entropy loss.

##  4 Experiments

###  4.1 Experimental setup

####  4.1.1 Datasets

In our evaluation of knowledge distillation for semantic segmentation
performance, we utilize three widely used datasets: (1) Cityscapes [27], an
extensive urban scene parsing dataset featuring 5000 finely annotated images
gathered from 50 cities in Germany. The images, each sized at 2048$\times$1024
pixels, are categorized into 19 classes. The dataset is split into 2975 for
training, 500 for validation, and 1525 for testing. (2) Pascal VOC [28]
presents a diverse set of visual object segmentation challenges of 13487
images, with 10582 training images, 1449 validation images, and 1456 test
images. Each pixel in the Pascal VOC dataset is classified into one of 21
classes, comprising 20 foreground object categories and one background class.
(3) CamVid [29] focuses on automotive scenes with 701 images of 720$\times$960
pixels, offering 367 training images, 101 validation images, and 233 test
images distributed across 11 semantic classes.

####  4.1.2 Model architectures

To assess the performance of the proposed approach in the knowledge
distillation process, we utilize the segmentation architecture of DeepLabV3
(DL) with ResNet-101 (R101) as the backbone, acting as a powerful teacher
network. This model is pretrained on various databases, and its parameters
remain frozen throughout the distillation process. When considering student
networks, we utilize the ResNet-18 (R18) backbone, initialized with pretrained
weights from the ImageNet dataset, across both DeepLabV3 and PSPNet
architectures.

####  4.1.3 Implementation details

To optimize the parameters of both the teacher and student networks, we use
the Stochastic Gradient Descent (SGD) with momentum. We set the initial
learning rate of 0.02 with a momentum of 0.9, while maintaining a fixed batch
size of 16. Additionally, we implement a polynomial learning rate decay
strategy, where we update the current learning rate by multiplying it with
$(1-\frac{iter}{total_{iter}})^{0.9}$ at each training iteration, where
$total_{iter}$ is fixed to 40000. The input images are cropped to dimensions
of 512$\times$1024 pixels for Cityscapes, 360$\times$360 pixels for CamVid,
and 512$\times$512 pixels for Pascal VOC. In our experiments, we ensure
diversity in the training data by employing a data augmentation approach on
the input images, with random scaling from 0.5 to 2 and flipping. Finally, we
utilize the PyTorch framework for implementation.

####  4.1.4 Evaluation metrics

To compare the proposed method results against state of the art, we employ the
mean Intersection over Union (mIoU) metric, a widely accepted measure for
semantic segmentation performance assessment. $mIoU$ computes the Intersection
over Union ($IoU$) for each class and then computes the average over all
classes. $IoU$ is defined by the following equation:

$IoU=\frac{|A\cap B|}{|A\cup B|}$ (7)

where A and B are the mask pixels, with A is the ground truth and B is the
predicted mask. This metric provides a comprehensive overview of segmentation
accuracy by accounting for both true positives and false positives ensuring a
balanced evaluation across the various classes within a dataset.

###  4.2 Ablation study

We recall that the student network is trained using the general loss expressed
in Equation 4 which contains three losses. In Table 1, we conduct an
experiment to study their contribution on the validation set of Cityscapes.
Based on the results presented in this table, the following observations can
be made: 1) the distillation at score maps brings more gains than the original
student. Further aligning the proposed class prototypes distillation using
triplet loss boosts the performance and brings the most performance gains.

Method | Val $mIoU$ (%)
---|---
Teacher: DeepLabV3-R101 | 78.07
Student: DeepLabV3-R18 | 74.21
$\mathcal{L}_{seg}$ | $\mathcal{L}_{SM}$ | $\mathcal{L}_{FM}$ |
$\checkmark$ |  |  | 73.21
$\checkmark$ | $\checkmark$ |  | 75.33
$\checkmark$ | $\checkmark$ | $\checkmark$ | 76.03
Table 1: Ablation study on the validation set of Cityscapes.

We underline that a grid search is done to obtain the following optimal values
for Equations 4 and 6: $\lambda_{I2CKD}=0.6$, $\lambda_{SM}=3$ and
$\mathcal{T}=2$.

###  4.3 Comparisons with the state-of-the-art methods

####  4.3.1 Comparison on Cityscapes

In Table 2, we compare the proposed method with contemporary distillation
methods tailored to semantic segmentation including SKD [10], IFVD [11], and
CWD [14]. To ensure a fair comparison, we adopt different teacher-student
network pairs by changing the segmentation framework or backbones. An
intuitive idea for compressing a heavy FCN is to replace its backbone by a
compact one. This table demonstrates that replacing the ResNet101 backbone of
DL model with ResNet18 resulted in a decrease in segmentation performance.
Specifically, in the validation set, the performance dropped from $78.07\%$ to
$74.21\%$. The decline in performance is primarily due to the reduced size of
the student model ($13.6$M) which make an independent training without direct
guidance from the teacher model. Additionally, it is noteworthy that both the
proposed method and all the compared distillation methods greatly improve the
performance of the student networks which demonstrate the interest of adopting
knowledge distillation for neural network compression. From this table, we can
see that our method I2CKD significantly outperforms the compared distillation
methods when we consider a student with the same segmentation framework of the
teacher (DL-R101/DL-R18) as well as when we use a student with a different
segmentation framework (DL-R101/PSPNet-R18). The aforementioned results
strongly demonstrate the benefit of using the class prototypes for knowledge
extraction and triplet loss for knowledge transfer. Some qualitative results
using the configuration DL-R101/DL-R18 are illustrated in Figure 3.

Teacher | Methods | Params | val $mIoU$
---|---|---|---
↓ |  | (M) | ($\%$)
Student |  |  |
Teacher | DL-R101 | 61.1 | 78.07
| DL-R18 | 13.6 | 74.21
DL-R101 | SKD [10] |  | 75.42
↓ | IFVD [11] |  | 75.59
DL-R18 | CWD [14] | 13.6 | 75.55
| I2CKD (ours) |  | 76.07
| PSPNet-R18 | 12.9 | 72.55
DL-R101 | SKD [10] |  | 73.29
↓ | IFVD [11] |  | 73.71
PSPNet-R18 | CWD [14] | 12.9 | 74.36
| I2CKD (ours) |  | 74.43
Table 2: Quantitative results on Cityscapes. Fig. 3: Qualitative segmentation
results on Cityscapes validation set using DeepLabV3-R18 student.

####  4.3.2 Comparison on Pascal VOC

We additionally perform experiments on the PASCAL VOC dataset to further
validate the distillation capability of the proposed I2CKD in visual object
segmentation. The results are recorded in Table 3. Both original student
models drop the performance of the teacher one (cumbersome) by $4.46\%$ and
$4.43\%$ respectively. While the other distillation methods only bring limited
improvements, our I2CKD achieve the best validation $mIoU$ of $76.31\%$ and
$74.59\%$ using DL-R18 and PSPNet-R18 students respectively.

Teacher | Methods | Params | val $mIoU$
---|---|---|---
↓ |  | (M) | ($\%$)
Student |  |  |
Teacher | DL-R101 | 61.1 | 77.67
| DL-R18 | 13.6 | 73.21
DL-R101 | SKD [10] |  | 73.51
↓ | IFVD [11] |  | 73.85
DL-R18 | CWD [14] | 13.6 | 74.02
| I2CKD (ours) |  | 76.31
| PSPNet-R18 | 12.9 | 73.33
DL-R101 | SKD [10] |  | 74.07
↓ | IFVD [11] |  | 73.54
PSPNet-R18 | CWD [14] | 12.9 | 73.99
| I2CKD (ours) |  | 74.59
Table 3: Quantitative results on Pascal VOC.

####  4.3.3 Comparison on CamVid

We further evaluate, in Table 4, the performance of the proposed method on
CamVid dataset. As can be seen, our I2CKD has respectively $+2.8\%$ ($66.92\%$
$vs.$ $69.72\%)$ and $+2.22\%$ ($66.73\%$ $vs.$ $68.95\%)$ performance gains
in comparison with the student without distillation. I2CKD also outperforms
other distillation methods by a significant margin, for example, $+2.01\%$ and
$+1.03\%$ compared to the most competitive method CWD [14].

Teacher | Methods | Params | test $mIoU$
---|---|---|---
↓ |  | (M) | ($\%$)
Student |  |  |
Teacher | DL-R101 | 61.1 | 69.84
| DL-R18 | 13.6 | 66.92
DL-R101 | SKD [10] |  | 67.46
↓ | IFVD [11] |  | 67.28
DL-R18 | CWD [14] | 13.6 | 67.71
| I2CKD (ours) |  | 69.72
| PSPNet-R18 | 12.9 | 66.73
DL-R101 | SKD [10] |  | 67.83
↓ | IFVD [11] |  | 67.61
PSPNet-R18 | CWD [14] | 13.6 | 67.92
| I2CKD (ours) |  | 68.95
Table 4: Quantitative results on CamVid.

##  5 Conclusion

In this paper, we target efficient image semantic segmentation via FCN-based
knowledge distillation. We propose the I2CKD method that transfer teacher’s
class prototypes (centroids) to the student to improve its learning efficacy.
This knowledge transfer is ensured by the use of triplet loss to force the
intra- and inter-class variations between class prototypes of the teacher and
student respectively. We perform experiments on three popular images semantic
segmentation datasets and showcase that the proposed distillation strategy
consistently improves the performance of various existing distillation
methods.

## References

  * [1] Fahad Lateef and Yassine Ruichek,  “Survey on semantic segmentation using deep learning techniques,”  Neurocomputing, vol. 338, pp. 321–348, 2019.
  * [2] Sachin Mehta, Mohammad Rastegari, Anat Caspi, Linda Shapiro, and Hannaneh Hajishirzi,  “Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation,”  in Proceedings of the european conference on computer vision (ECCV), 2018, pp. 552–568.
  * [3] Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello,  “Enet: A deep neural network architecture for real-time semantic segmentation,”  arXiv preprint arXiv:1606.02147, 2016.
  * [4] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang,  “Bisenet: Bilateral segmentation network for real-time semantic segmentation,”  in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 325–341.
  * [5] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag,  “What is the state of neural network pruning?,”  Proceedings of machine learning and systems, vol. 2, pp. 129–146, 2020.
  * [6] Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng,  “Quantized convolutional neural networks for mobile devices,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4820–4828.
  * [7] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.,  “Distilling the knowledge in a neural network,”  arXiv preprint arXiv:1503.02531, vol. 2, no. 7, 2015.
  * [8] Chuanguang Yang, Xinqiang Yu, Zhulin An, and Yongjun Xu,  “Categories of response-based, feature-based, and relation-based knowledge distillation,”  in Advancements in Knowledge Distillation: Towards New Horizons of Intelligent Systems, pp. 1–32. Springer, 2023.
  * [9] Jiafeng Xie, Bing Shuai, Jianfang Hu, Jingyang Lin, and Wei-Shi Zheng,  “Improving fast segmentation with teacher-student learning,”  in British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018. 2018, p. 205, BMVA Press.
  * [10] Yifan Liu, Changyong Shu, Jingdong Wang, and Chunhua Shen,  “Structured knowledge distillation for dense prediction,”  IEEE transactions on pattern analysis and machine intelligence, 2020.
  * [11] Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu,  “Intra-class feature variation distillation for semantic segmentation,”  in European Conference on Computer Vision. Springer, 2020, pp. 346–362.
  * [12] Tong He, Chunhua Shen, Zhi Tian, Dong Gong, Changming Sun, and Youliang Yan,  “Knowledge adaptation for efficient semantic segmentation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 578–587.
  * [13] Sangyong Park and Yong Seok Heo,  “Knowledge distillation for semantic segmentation using channel and spatial correlations and adaptive cross entropy,”  Sensors, vol. 20, no. 16, pp. 4616, 2020.
  * [14] Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen,  “Channel-wise knowledge distillation for dense prediction,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 5311–5320.
  * [15] Li Liu, Qingle Huang, Sihao Lin, Hongwei Xie, Bing Wang, Xiaojun Chang, and Xiaodan Liang,  “Exploring inter-channel correlation for diversity-preserved knowledge distillation,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 8271–8280.
  * [16] Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang,  “Cross-image relational knowledge distillation for semantic segmentation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 12319–12328.
  * [17] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao,  “Knowledge distillation: A survey,”  International Journal of Computer Vision, vol. 129, no. 6, pp. 1789–1819, 2021.
  * [18] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio,  “Fitnets: Hints for thin deep nets,”  in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun, Eds., 2015.
  * [19] Sergey Zagoruyko and Nikos Komodakis,  “Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,”  in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017, OpenReview.net.
  * [20] Junho Yim, Donggyu Joo, Ji-Hoon Bae, and Junmo Kim,  “A gift from knowledge distillation: Fast optimization, network minimization and transfer learning,”  in 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. 2017, pp. 7130–7138, IEEE Computer Society.
  * [21] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho,  “Relational knowledge distillation,”  in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. 2019, pp. 3967–3976, Computer Vision Foundation / IEEE.
  * [22] Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan,  “Knowledge distillation via instance relationship graph,”  in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. 2019, pp. 7096–7104, Computer Vision Foundation / IEEE.
  * [23] Jonathan Long, Evan Shelhamer, and Trevor Darrell,  “Fully convolutional networks for semantic segmentation,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431–3440.
  * [24] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia,  “Pyramid scene parsing network,”  in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6230–6239.
  * [25] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam,  “Rethinking atrous convolution for semantic image segmentation,”  arXiv preprint arXiv:1706.05587, 2017.
  * [26] Ayoub Karine, Thibault Napoléon, and Maher Jridi,  “Semantic images segmentation for autonomous driving using self-attention knowledge distillation,”  in 16th International Conference on Signal-Image Technology & Internet-Based Systems, SITIS 2022, Dijon, France, October 19-21, 2022, Kokou Yétongnon, Albert Dipanda, and Luigi Gallo, Eds. 2022, pp. 198–202, IEEE.
  * [27] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele,  “The cityscapes dataset for semantic urban scene understanding,”  in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [28] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman,  “The pascal visual object classes (voc) challenge,”  International journal of computer vision, vol. 88, pp. 303–338, 2010.
  * [29] Gabriel J. Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla,  “Segmentation and recognition using structure from motion point clouds,”  in Computer Vision – ECCV 2008, David Forsyth, Philip Torr, and Andrew Zisserman, Eds., Berlin, Heidelberg, 2008, pp. 44–57, Springer Berlin Heidelberg.


# Privacy Sensitive Speech Analysis
using Federated Learning to Assess depression

###### Abstract

Recent studies have used speech signals to assess depression. However, speech
features can lead to serious privacy concerns. To address these concerns,
prior work has used privacy-preserving speech features. However, using a
subset of features can lead to information loss and, consequently, non-optimal
model performance. Furthermore, prior work relies on a centralized approach to
support continuous model updates, posing privacy risks. This paper proposes to
use Federated Learning (FL) to enable decentralized, privacy-preserving speech
analysis to assess depression. Using an existing dataset (DAIC-WOZ), we show
that FL models enable a robust assessment of depression with only 4–6%
accuracy loss compared to a centralized approach. These models also outperform
prior work using the same dataset. Furthermore, the FL models have short
inference latency and small memory footprints while being energy-efficient.
These models, thus, can be deployed on mobile devices for real-time,
continuous, and privacy-preserving depression assessment at scale.

Index Terms—  speech classification, depression, privacy, paralinguistics,
mHealth

##  1 Introduction

Depression is a severe mental illness that affects millions of people
worldwide [1]. Depression causes staggering individual and societal costs,
including a higher risk of mortality [1]. There remains a significant
treatment gap — individuals with depression often do not receive adequate
treatment [2].

Addressing this treatment gap requires effective detection and monitoring of
depression at scale. Toward this goal, recent studies have established that
speech features including prosodic, articulatory, and acoustic signals can be
indicative of depression onset and severity [3]. However, speech signals can
be highly privacy-sensitive, and continuous monitoring can be challenging.
Specifically, privacy-sensitive analysis is critical for assessing mental
health issues. As such, recent studies have focused on using only privacy-
preserving speech features (e.g., [4, 5]). However, using a subset of speech
features leads to considerable information loss, negatively impacting model
accuracy. Also, prior work requires a centralized approach to support
continuous model updates from collected data. The need for data sharing across
individuals, even for a subset of features, can lead to serious privacy risks.
There has been an increasing focus on enabling decentralized training to
reduce privacy risks in recent years. For example, Federated Learning (FL)
aims to train models using local datasets and then merge those local models as
necessary. Given that local datasets are not shared, FL significantly reduces
privacy concerns. However, the resultant model accuracy and overhead can be a
concern, particularly when deploying in mobile devices. To the best of our
knowledge, there has been no work to establish the accuracy and performance
overhead of using FL for privacy-preserving speech analysis to assess
depression.

This paper aims to address this gap in two steps. First, we use an existing
dataset (DAIC-WOZ) to establish that decentralized, privacy-preserving models
can be used for robust assessment of depression using speech data. Second, we
establish that the computational overhead of these models is low, and thus,
they can be deployed to mobile devices for continuous and real-time
monitoring. The paper makes the following novel technical contributions:

  * •

We used transfer learning to assess depression and compared training overhead
and accuracy using two FL frameworks: Federated Averaging (FedAvg) [6] and
Federated Matching Averaging (FedMA) [7].

  * •

The FL models achieve significantly better accuracy compared to the best-
performing models in prior work using the DAIC-WOZ dataset (e.g., 87% accuracy
for the combined dataset compared to 74.64% in [8]).

  * •

We deploy the models in a smartphone to assess performance overhead for
determining depression and show real-time and continuous assessment is
possible.

##  2 Related Work

Prior work has found that depressive states correspond to changes in prosodic
and acoustic features (e.g., reduced pitch range, loudness, energy dynamics,
and speaking rate) [3]. There has been an increasing focus on using speech as
an objective biomarker of depression. Long et al. [9] used different speech
features (e.g., short-time energy, intensity, formant frequencies, shimmer,
jitter, and ZCR) on an in-house dataset and achieved 78.02% in detecting
depression. Lalitha et al. [10] used deep neural networks to identify speech
features that contain “emotional information”. They achieved an average
accuracy of 84.3% on the Berlin EmoDB database.

While DAIC-WOZ includes audio and video streams, we will focus on audio data,
given the paper’s scope. In recent work, Ma et al. [11] developed DepAudioNet,
which leveraged Deep Convolutional Neural Networks and Long Short-Term Memory
networks. The model achieved an F1 score of 0.52, precision of 0.35, recall of
1, and an approximated accuracy of 0.5. Hanai et al. [12] developed an LSTM
model using audio features with an accuracy of 0.59. Srimadhur et al. [8] used
an end-to-end network for classification from audio data with an accuracy of
74.64%. However, these previous studies do not focus on privacy-preserving
audio analysis.

Prior work has explored the use of privacy-sensitive features from speech [4].
For example, Wyatt et al. [4] aimed to limit intelligibility and speech
content reconstruction using a subset of features. However, reducing the
feature space can lead to considerable information loss and negatively impact
model accuracy. Furthermore, this approach still requires a central data
repository to support model training and updating. Given that depression can
be a lifelong condition, it is often necessary to continuously update models
over depression to reflect different personal and lifestyle changes over time.
The need for uploading and sharing data, even for a subset of features, can
lead to serious privacy risks.

Fig. 1: The centralized (left) and federated learning methods (right) differ
in model training and updating steps.

Several recent studies have used FL to reduce privacy risks in analyzing
health data [13, 14]. Figure 1 compares the centralized and federated learning
approaches. FL requires a trade-off regarding model performance and training
overhead. Centralized models have access to more data, leading to better
performance. The on-device training and merging model parameters can lead to
considerable training time in FL. These trade-offs can be critical for systems
focusing on real-time and continuous assessment of mental health issues,
including depression. While FL has been used for different health
applications, there is a lack of existing work that leveraged FL for privacy-
preserving speech analysis to assess depression. This paper aims to address
this gap. Specifically, we aim to establish a benchmark of accuracy and
performance overhead of using FL to assess depression using speech signals
from the DAIC-WOZ dataset.

$\begin{array}[]{ccc}\hline\cr\text{ Gender }&\text{ hasDepression}&\text{
noDepression }\\\ \hline\cr\text{ Male }&25&25\\\ \text{ Female }&25&25\\\
\hline\cr\text{ Total }&50&50\\\ \hline\cr\end{array}$
---
Table 1: The dataset used in this paper. We randomly selected a subset of the
DAIC-WOZ dataset to ensure equal gender and depression class distribution.

##  3 Data Attributes and Preprocessing

We have used the DAIC-WOZ dataset [15] to train and evaluate model
performance. The dataset contains 189 interviews. Each interview has a
corresponding PHQ-8 [16] score indicating depression severity. The original
dataset is imbalanced regarding gender and the number of individuals with
depression. To ensure balance, we randomly selected a subset of the dataset
with equal gender and depression class distribution (25 individuals each) as
shown in Table 1. The speech data has a sampling rate of 16 kHz. It contains
speeches from both the participant and the virtual interviewer. The dataset
provides start and stop timestamps indicating where the interviewer and the
participant have spoken. We use this metadata to extract speech segments for
the participants. Following prior work [17], we used an overlapping window
length of 1s duration with a shift of 0.1s to extract log spectrogram features
using the scipy.signal.spectrogram function, which is log-scaled as shown in
Fig. 2. The images help model both temporal and harmonic structures of audio
signals, leading to improved classification performance over existing methods.

$\begin{array}[]{cccc}\hline\cr\text{ Network }&\text{ Depth }&\text{
Parameters (M) }&\text{ Input}\\\ \hline\cr\text{GoogleNet}&22&7&224\times
224\\\ \text{MobileNet v2}&20&3.4&224\times 224\\\
\text{ResNet-18}&18&11.7&224\times 224\\\ \hline\cr\end{array}$
---
Table 2: The characteristics of the network architectures used Fig. 2: We used
transfer learning strategies to efficiently train models. We started with pre-
trained models and reused their feature detector layers. We then trained the
output layer using log mel spectrograms from the dataset.

##  4 Experimental Setup

We used PyTorch on an NVIDIA Tesla V100-SXM2-32GB GPU to run the experiments.
We use the Stochastic Gradient Descent optimizer (optim.SGD) with an initial
learning rate (LR) = 0.001 and momentum of 0.9 for training the three
networks. The ‘LR’ value is decayed every 7 epochs by a factor of gamma = 0.1
[18]. This decaying of the LR is useful in the case of FedAvg as it ensures
convergence guarantees [19]. There has been work [20, 21] to alleviate high
communication costs and performance drop, especially in the case of non-I.I.D
scenario [22, 23]. Also, instead of training a single model on devices in FL
environments, the authors suggest using a two-stream model, widely used in
transfer learning [24]. We employ the Cross entropy function
(nn.CrossEntropyLoss) as loss criterion. We minimized the risk of overfitting
by using the five-fold cross-validation and early stopping criteria. The
spectrogram features extracted are of dimensions 515x389 and are corrected to
224x224 using the transforms library from torchvision - where only the
training images are augmented. Given our focus on using these models in
smartphone devices, we chose three pre-trained models with a smaller number of
parameters ($<$ 12 Million) as in Table 2.

##  5 Findings

In this work, we focus on two binary classification tasks, namely - Depression
classification (i.e., hasDepression vs. noDepression) and Depression severity
classification (i.e., mild vs. severe). We consider a centralized and two FL
approaches (FedAvg and FedMA). For FL approaches, we consider five devices on
the same network.

###  5.1 Depression classification

For the depression classification task, we first assess model performance for
gender-specific data. We then consider the combined dataset for model
performance for comparison. We use five-fold cross-validation throughout the
classification tasks. The resultant model performance is shown in Figure 3. A
summary of the average training overhead is in Table 3.

$\begin{array}[]{ccccc}\hline\cr\text{Type}&\text{Male}&\text{Female}&\text{Combined}&\text{Severity}\\\
\hline\cr\text{RN-18}&155&167&253&222\\\ \text{GN}&235&255&411&216\\\ \text{MN
v2}&204&223&359&191\\\ \hline\cr\text{RN-18 + FA}&340&372&574&499\\\ \text{GN
+ FA}&380&457&633&475\\\ \text{MN v2 + FA}&345&415&595&411\\\
\hline\cr\text{RN-18 + FMA}&327&487&554&470\\\ \text{GN +
FMA}&\text{}366&480&612&466\\\ \text{MN v2 + FMA}&\text{}344&399&570&419\\\
\hline\cr\end{array}$
---
Table 3: Average time taken (s) to train different networks and algorithms.
Note: RN-18 (ResNet-18), GN (GoogleNet), MNv2 (MobileNet v2), FA (FedAvg), FMA
(FedMA)

Figure 3 compares model performance for all the four sets of experiments. The
centralized approach performs better than the federated methods by 6-10%
across folds. The best avg. five-fold accuracy for the centralized approach is
0.934, while for the federated scheme, it is 0.91. From Table 3, we see that
the centralized method is about 1.55-2.19x faster than federated schemes with
ResNet-18 fastest for both centralized (155s) and federated schemes (327 &
340s respectively). Regarding the female subjects, the best average five-fold
accuracy for centralized is 0.89, while for the FL scheme, it is 0.87. The
centralized approach is about 1.78-2.91x faster than federated schemes. The
centralized approach performs better by 4-6% across folds for the combined
dataset. The best five-fold accuracy for the centralized approach is 0.885,
while for FL, it is 0.87. The centralized method is about 1.48-2.26x faster
than federated schemes.

Fig. 3: Average five-fold cross-validation accuracy (has vs. no depression)
for different algorithms (CEN: centralized, FA: FedAvg, FMA: FedMA). We also
calculated performance across networks (rows) and subjects (columns).

###  5.2 Depression severity classification

Classifying mild and severe states can lead to informed clinical decision-
making and pre-emptive care. We used the PHQ-8 score $>$10 as the threshold.
This resulted in a binary classification task with low (PHQ-8 score $<$10) and
high (PHQ-8 $\geq$10). The low class includes noDepression as well (i.e.,
PHQ-8 $\leq$4). We used the combined dataset for the classification task,
resulting in a reasonably balanced class distribution (low: 28 subjects and
high: 22 subjects). The results are shown in Figure 3. The best average five-
fold accuracy in centralized and FL schemes are 0.93 and 0.88 (both in
ResNet-18), respectively. The centralized method is about 2.11-2.24x faster
than federated schemes, with MobileNet v2 training the fastest for both
centralized and FL schemes (see Table 3).

###  5.3 Overall performance comparison

We used a ranking method to summarize overall performance across different
approaches and networks. For each task category (e.g., Male), we assign a rank
(1–3) to rate performance with rank 1 for the best performing model. By
combining ranks across tasks (12 in total), we compare the robustness and
consistency of different approaches. The results are summarized in Table 4 and
5 respectively. The centralized algorithm takes the least time to train and
has the best accuracy. FedMA (2.33) performs slightly better than FedAvg
(2.67) in time, but their accuracy ranking is the same, indicating similar
performance across tasks. We compare network architecture performance in Table
5. MobileNet v2 performs well in both time and accuracy and offers the best of
both worlds.

To summarize, the privacy-preserving FL models perform robustly with only 4-6%
accuracy lost compared to a traditional centralized approach (_hasDepression_
vs. _noDepression_). The FL models achieve comparable accuracy to the
centralized approach for assessing depression severity (_high_ vs. _low_).
These findings establish the feasibility of using privacy-preserving FL models
for depression assessment.

$\begin{array}[]{ccc|ccc}\hline\cr&\text{Time}&&&\text{Accuracy}&\\\
\hline\cr\text{Method}&\text{Sum}&\text{Avg.}&\text{Method}&\text{Sum}&\text{Avg.}\\\
\hline\cr\text{Central}&12&1&\text{Central}&12&1\\\
\text{FedAvg}&32&2.67&\text{FedAvg}&30&2.5\\\
\text{FedMA}&28&2.33&\text{FedMA}&30&2.5\\\ \hline\cr\end{array}$
---
Table 4: Performance ranking (lower is better) for different approaches across
twelve different task categories.
$\begin{array}[]{ccc|ccc}\hline\cr&\text{Time}&&&\text{Accuracy}&\\\
\hline\cr\text{Network}&\text{Sum}&\text{Avg.}&\text{Network}&\text{Sum}&\text{Avg.}\\\
\hline\cr\textbf{RN-18}&\textbf{6}&\textbf{1.5}&\text{RN-18}&7&1.75\\\
\text{GN}&11&2.75&\text{GN}&12&3\\\ \text{MN v2}&7&1.75&\textbf{MN
v2}&\textbf{5}&\textbf{1.25}\\\ \hline\cr\end{array}$
---
Table 5:  Performance ranking for different network architectures (lower is
better). The ranking is across twelve different task categories. MobileNet v2
provides the best trade-off.

##  6 Deployment feasibility

(a) Speech with depression

(b) Severity: Moderate-Severe

Fig. 4: The developed Android app. (a) It first aims to classify whether the
speech shows signs of depression, and if it does, (b) it proceeds to rate its
severity.

While the performance of the privacy-preserving, decentralized models is
highly encouraging, it is critical to make sure that they can be deployed to
devices with low computational resources. To assess the feasibility, we
developed a smartphone app using TensorFlow Lite. The app classifies whether
the speech shows signs of depression and, if it does, rates its severity. We
used a Xiaomi Redmi Note 7 with 4GB RAM/64 GB ROM running Android 11 for
deployment. For classification, we used an optimal two-fold model approach
using the combined model and a model from the subject’s gender (in real-time).
Since the data capture phase is common and training takes little time - thus
ensuring that the app works in real-time. Once we confirm depression exists
through majority voting of the individual spectrogram image frames for both
models, we use a depression severity model from the same model group. Thus,
classification is more robust compared to a one-size-fits-all model. We used
Android Profiler to collect real-time CPU, memory, network, and battery
consumption data. The findings are shown in Table 6. For all models, the
inference latency $<$ 100ms, which is essential for in-situ and real-time
assessment. Each model requires $<$ 50MB of memory. The models are energy
efficient — energy consumed/frame ranges from 0.26-0.37 joules, which
translates to 72-102 µW/hr.

$\begin{array}[]{cccc}\hline\cr\text{Type}&\text{IL (ms)}&\text{Memory
(MB)}&\text{E (J)}\\\
\hline\cr\text{RN-18}&\textbf{48.2}&\textbf{23}&\textbf{0.26}\\\
\text{GN}&64.1&26&0.37\\\ \text{MN v2}&51.2&38&0.33\\\ \hline\cr\end{array}$
---
Table 6: The inference latency (IL), memory allocated (Memory), and the energy
consumed per frame (E) by each model on a smartphone.

##  7 Conclusion

This paper explores the feasibility of using privacy-preserving, decentralized
models for assessing depression and its severity using speech. Toward this
goal, we use federated learning to enable on-device training. We used an
existing dataset (DAIC-WOZ) to establish a performance benchmark for
decentralized and privacy-preserving approaches. We show that the privacy-
preserving FL models perform robustly with only 4-6% accuracy lost compared to
a centralized approach. More importantly, these models achieve better accuracy
than the best-performing models in prior work using DAIC-WOZ (e.g., 87% (FL)
compared to 74.64% [8]). These findings establish the feasibility of using
privacy-preserving FL models for depression assessment. We also explored the
feasibility of deploying on devices with low computational resources. The FL
models show small inference latency and a low memory footprint while being
energy-efficient. As such, these models can be deployed to mobile devices to
support continuous and real-time assessment of depression at scale.

## References

  * [1] World Health Organization,  “Depression,” who.int/health-topics/depression, 2020.
  * [2] Graham Thornicroft, Somnath Chatterji, Sara Evans-Lacko, Michael Gruber, et al.,  “Undertreatment of people with major depressive disorder in 21 countries,”  The British Journal of Psychiatry, vol. 210, no. 2, 2017.
  * [3] Nicholas Cummins, Stefan Scherer, Jarek Krajewski, Sebastian Schnieder, Julien Epps, and Thomas F. Quatieri,  “A review of depression and suicide risk assessment using speech analysis,”  Speech Communication, vol. 71, pp. 10–49, July 2015.
  * [4] Danny Wyatt, Tanzeem Choudhury, Jeff Bilmes, and James A Kitts,  “Inferring colocation and conversation networks from privacy-sensitive audio with implications for computational social science,”  ACM Transactions on Intelligent Systems and Technology (TIST), vol. 2, no. 1, 2011.
  * [5] Saeed Abdullah, Mark Matthews, Ellen Frank, Gavin Doherty, Geri Gay, and Tanzeem Choudhury,  “Automatic detection of social rhythms in bipolar disorder,”  Journal of the American Medical Informatics Association, vol. 23, no. 3, pp. 538–543, 2016.
  * [6] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas,  “Communication-efficient learning of deep networks from decentralized data,”  in Artificial Intelligence and Statistics. PMLR, 2017, pp. 1273–1282.
  * [7] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni,  “Federated learning with matched averaging,”  arXiv preprint arXiv:2002.06440, 2020.
  * [8] NS Srimadhur and S Lalitha,  “An end-to-end model for detection and assessment of depression levels using speech,”  Procedia Computer Science, 2020.
  * [9] Hailiang Long, Zhenghao Guo, Xia Wu, Bin Hu, Zhenyu Liu, and Hanshu Cai,  “Detecting depression in speech: Comparison and combination between different speech types,”  in 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2017.
  * [10] S Lalitha, Shikha Tripathi, and Deepa Gupta,  “Enhanced speech emotion detection using deep neural networks,”  International Journal of Speech Technology, vol. 22, no. 3, pp. 497–510, 2019.
  * [11] Xingchen Ma, Hongyu Yang, Qiang Chen, Di Huang, and Yunhong Wang,  “Depaudionet: An efficient deep model for audio based depression classification,”  in Proceedings of the 6th international workshop on audio/visual emotion challenge, 2016, pp. 35–42.
  * [12] Tuka Al Hanai, Mohammad M Ghassemi, and James R Glass,  “Detecting depression with audio/text sequence modeling of interviews.,”  in Interspeech, 2018.
  * [13] Curtis P Langlotz, Bibb Allen, Bradley J Erickson, Jayashree Kalpathy-Cramer, Keith Bigelow, Tessa S Cook, et al.,  “A roadmap for foundational research on artificial intelligence in medical imaging: from the 2018 NIH/RSNA/ACR/The Academy Workshop,”  Radiology, vol. 291, no. 3, pp. 781–791, 2019.
  * [14] Theodora S Brisimi, Ruidi Chen, Theofanie Mela, Alex Olshevsky, Ioannis Ch Paschalidis, and Wei Shi,  “Federated learning of predictive models from federated electronic health records,”  International journal of medical informatics, vol. 112, pp. 59–67, 2018.
  * [15] Jonathan Gratch, Ron Artstein, Gale M Lucas, Giota Stratou, Stefan Scherer, Angela Nazarian, Rachel Wood, Jill Boberg, David DeVault, Stacy Marsella, et al.,  “The distress analysis interview corpus of human and computer interviews.,”  in LREC, 2014, pp. 3123–3128.
  * [16] Robert L. Spitzer, Janet B.W. Williams, and Kurt Kroenke,  “Patient health questionnaire (phq-8),” https://tinyurl.com/h6h7uavx, May 1991.
  * [17] B N Suhas, Jhansi Mallela, Aravind Illa, et al.,  “Speech task based automatic classification of ALS and Parkinson’s Disease and their severity using log mel spectrograms,”  in 2020 International Conference on Signal Processing and Communications. IEEE, 2020.
  * [18] Kaichao You, Mingsheng Long, Jianmin Wang, and Michael I Jordan,  “How does learning rate decay help modern neural networks?,”  arXiv preprint arXiv:1908.01878, 2019.
  * [19] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang,  “On the convergence of fedavg on non-iid data,”  in International Conference on Learning Representations, 2020.
  * [20] Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, et al.,  “Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data,”  arXiv preprint arXiv:1811.11479, 2018.
  * [21] Xin Yao, Chaofeng Huang, and Lifeng Sun,  “Two-stream federated learning: Reduce the communication costs,”  in 2018 IEEE Visual Communications and Image Processing (VCIP). IEEE, 2018, pp. 1–4.
  * [22] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith,  “Federated learning: Challenges, methods, and future directions,”  IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 50–60, 2020\.
  * [23] Suhas BN,  “Privacy-preserving assessment of depression using speech signal processing,”  M.S. thesis, 2021.
  * [24] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan,  “Learning transferable features with deep adaptation networks,”  in International conference on machine learning. PMLR, 2015, pp. 97–105.


# Thulium and ytterbium-doped titania thin films deposited by MOCVD

S bastien Forissier<EMAIL_ADDRESS>Herv Roussel  Carmen
Jimenez  Odette Chaix  Laboratoire des Mat riaux et du G nie Physique, CNRS,
Grenoble Institute of Technology, MINATEC, 3 parvis Louis N el, 38016
Grenoble, France  Antonio Pereira  Amina Bensalah-Ledoux  Laboratoire de
Physico-Chimie des Mat riaux Luminescents UMR 5620 CNRS / UCBL Domaine
Scientifique de la Doua, Universit Claude Bernard Lyon 1 B timent Alfred
Kastler 10 rue Ada Byron 69622 Villeurbanne cedex, France  Jean-Luc
Deschanvres  Laboratoire des Mat riaux et du G nie Physique, CNRS, Grenoble
Institute of Technology, MINATEC, 3 parvis Louis N el, 38016 Grenoble, France
Bernard Moine  Laboratoire de Physico-Chimie des Mat riaux Luminescents UMR
5620 CNRS / UCBL Domaine Scientifique de la Doua, Universit Claude Bernard
Lyon 1 B timent Alfred Kastler 10 rue Ada Byron 69622 Villeurbanne cedex,
France

(September 2, 2024)

###### Abstract

In this study we synthesized thin films of titanium oxide doped with thulium
and/or ytterbium to modify the incident spectrum on the solar cells. This
could be achieved either by photoluminescence up-converting devices, or down-
converting devices. As down-converter thin films our work deals with thulium
and ytterbium-doped titanium dioxide. Thulium and ytterbium will act as
sensitizer and emitter, respectively. The rare-earth doped thin films are
deposited by aerosol-assisted MOCVD using organo-metallic precursors such as
titanium dioxide acetylacetonate, thulium and ytterbium
tetramethylheptanedionate solved in different solvents. These films have been
deposited on silicon substrates under different deposition conditions
(temperature and dopant concentrations for example). Adherent films have been
obtained for deposition temperatures ranging from 300°C to 600°C. The
deposition rate varies from 0.1 to 1 $\upmu$m/h. The anatase phase is obtained
at substrate temperature above 400°C. Further annealing is required to exhibit
luminescence and eliminate organic remnants of the precursors. The
physicochemical and luminescent properties of the deposited films were
analyzed versus the different deposition parameters and annealing conditions.
We showed that absorbed light in the near-UV blue range is re-emitted by the
ytterbium at 980 nm and by a thulium band around 800 nm.

CVD, thulium, ytterbium, down-conversion, thin film, titanium oxide,
photovoltaic

##  I Introduction

Titanium dioxide has received much interest because of its various
applications: photocatalysis, pigment, Transparent Conducting Oxide for
photovoltaic applications Richards2006 . In this field we focused on doping
titanium dioxide with rare-earths (thulium and ytterbium in this work) to
study the down-conversion between those ions from blue-visible and near-
ultraviolet light to near-infrared light for solar spectrum engineering. The
modification of the solar spectrum would allow us to achieve better yields for
silicon solar cells Badescu2007 ; Trupke2002 . Thin films were grown using
aerosol assisted Metal Oxide Chemical Vapor Deposition (MOCVD) method.

##  II Experimental

Thulium and ytterbium-doped titanium oxide thin films were deposited by mean
of aerosol assisted Metal Oxide Chemical Vapor Deposition method
Deschanvres1998 ; Deschanvres1989 . Liquid source solution, were composed of
titanium (IV) oxide bis(acetylacetonate), TiO2(acac)2, thulium (III)
tris(2,2,6,6-tetramethyl-3,5-heptanedionate) and ytterbium (III)(acac)
dissolved in high purity butanol (99%). All these precursors were purchased
from STREM Chemicals, butanol was purchased from Alfa Aesar. Precursors were
selected for non-toxicity, good stability at room temperature, easy handling,
high volatility and low cost Ryabova1968 . The films were deposited on (100)
silicon substrate. The source solution is delivered to the piezoelectric
transducer Viguie1975  through a constant level burette to ensure a constant
pulverization during the whole deposition. The aerosol was produced by means
of a flat piezoelectric transducer excited at 800 kHz which generated an
ultrasonic beam in a solution containing the reactant of the material to be
deposited. This ultrasonic spraying system guarantees a narrow dispersion of
the droplet size (4-10 $\upmu$m). Droplets are carried with two air fluxes,
dried and purified, to the heated sample holder, the lower air flux (12.7 l/m)
assure the main propulsion when the upper air flux (10.1 l/m) lengthen the
vapour stay in the vicinity of the sample holder. The overall air flow is
parallel to the substrate’s surface. For the deposition, the substrate was
fixed by clips on the sample holder heated by an electrical resistance with a
built-in thermocouple.

We made different series of samples with varying cation concentration in
solution. Both Tm-doped, Yb-doped and co-doped samples were successfully
synthesized. The compositions of the doped films were measured by electron
probe microanalysis (EPMA) and computed by help of special software dedicated
to the thin film analysis, called Stratagem and edited by the SAMx society
Pouchou1984 . The properties of the films reported in this paper are described
in section 3.2 and summarized in table 1. The X-ray diffraction profile was
obtained with a Bruker D8 Advance using Cu K$\alpha_{1}$ radiation in
$\theta/2\theta$ configuration. Absorption FT-IR spectroscopy was used to
study the structural evolution of the films versus the deposition conditions.
Spectra were obtained between 250 and 4000 cm${}^{-}1$ with 4 cm${}^{-}1$
resolution with a Bio-Rad Infrared Fourier Transform spectrometer FTS165 and
after performing Si substrate subtraction. Fluorescence and decay time
measurements were taken using a laboratory-built apparatus. It is composed of
a tunable laser NT342-10-AW from Ekspla, a monochromator TRIAX 190 from Jobin
Yvon, a photomultiplier C4877 from Hamamatsu and a multichannel scaler SR430
from Stanford Research. Excitation spectra were taken using a F900
spectrofluorimeter Edinburgh with a high spectral resolution. A Xenon Arc lamp
(450 W) is used for the excitation, the detector is a photomultiplier
Hamamatsu R2658P cooled by Peltier effect.

##  III Results and discussion

###  III.1 Structural properties

Figure 1: Crystallization versus deposition temperature. Figure 2:
Crystallization effect of annealing.

As shown by XRD diffraction the samples deposited at temperature lower than
400°C are amorphous (Figure 1). Above 400°C as deposition temperature, the
films crystallize in the anatase phase of titanium oxide. As shown on Figure 2
the crystalline quality of the films is improved by annealing, first at 500°C
for 1 h and then at 800°C for 1 h. With increasing annealing temperature the
anatase peaks are becoming higher and thinner. After annealing at 800°C
undoped samples crystallize in the rutile phase, whereas the presence of rare-
earth dopants prevents the phase transition as reported in Graf2007 .

###  III.2 Composition

Figure 3: Doping versus temperature.

Due to the reactor’s geometry, the thicknesses of the sample are not
homogeneous on the whole surface but it shows a good uniformity on half of it
as seen with interferential colors depending on their thickness. However
despite this geometry the doping level is quite uniform on the whole surface.
We synthesized different samples from the same solution at different
temperature and the electron microprobe measurement showed that the optimal
doping efficiency is obtained at 400°C (Figure 3). In this condition the rare-
earth precursor reactivity is lower than the titanium oxide precursor
reactivity.

###  III.3 Luminescence properties

Figure 4: Excitation spectrum of a Tm-doped sample (red curve) and excitation
spectrum of a Yb-doped sample (black curve). Spectra were recorded in separate
experiments. Figure 5: Emission spectrum of a Tm, Yb co-doped sample.

The emission scan (Figure 5) were taken exciting at 330 nm in the TiO2 matrix
(3.2 eV gap) near the${}^{1}D_{2}$ level of thulium. On a Tm, Yb co-doped
sample we see both the Yb ${}^{2}F_{5/2}-^{2}F_{7/2}$ transition at 980 nm and
the Tm ${}^{3}F_{4}-^{3}H_{6}$ transition around 800 nm. On mono-doped samples
(not figured) we only see the respective ion transitions. It appears that the
absorbed energy is transferred through the matrix to the rare-earth dopant
ions leading to a down-conversion mechanism with thulium and ytterbium. The Yb
luminescence is interesting for solar cells because it’s right before the band
gap of the silicon; the Tm luminescence is still in the wavelength range of
good absorption for silicon solar cells.

The excitations scans (Figure 4) were taken looking at the ytterbium
${}^{2}F_{5/2}-^{2}F_{7/2}$ transition and then the thulium
${}^{3}F_{4}-^{3}H_{6}$ transition. Both transitions are activated from the
near-UV region between 300 and 350 nm. This corroborates the idea of the
energy path we have between the matrix and the rare-earth ions.

The lifetime of the ${}^{3}F_{4}-^{3}H_{6}$ transition of Tm was measured on
Tm-doped only and Tm, Yb co-doped samples. For Tm-doped only samples we see
that the lifetime of the transition decreases with the increasing percentage
of Tm (see Table 1). When we compare co-doped and mono-doped samples we see
that at a constant level of Tm when we increase the percentage of Yb the
lifetime is decreasing (Figure 6), leading us to conclude of energy transfer
between Tm and Yb. The transfer rate was computed with the relation
$1-\tau_{x}/\tau$ where $\tau_{x}$ is the lifetime of a co-doped sample and
$\tau$ the lifetime of a Tm-doped sample with the same concentration, with
samples A and E this gives a 10% transfer rate.

Figure 6: Decay time of Tm,Yb-doped samples. Sample | Tm (%) | Yb (%) | Lifetime ($\upmu$)
---|---|---|---
A | 0.86 | 0 | 50
B | 2.08 | 0 | 48
C | 0.41 | 0.69 | 56
D | 0.37 | 0.85 | 46
E | 0.83 | 0.79 | 45
F | 1.21 | 3.63 | 43
Table 1: Cationic doping and lifetimes of the thulium

##  IV Conclusion

In summary, we succeeded in doping titanium dioxide with thulium and ytterbium
by Metal Oxide Chemical Vapour and to grow them partially crystallized. After
proper annealing the luminescence of the samples was studied both with
excitation and emission scans. The luminescence study showed energy transfer
between the titanium oxide matrix and the rare-earth ions and two interesting
emission for silicon solar cells. Lifetime measurements showed the energy
exchange between thulium and ytterbium with an estimated transfer rate of 10%.

##  V Acknowledgements

This work has been supported by French Research National Agency (ANR) through
Habitat intelligent et solaire photovolta que program (project MULTIPHOT
n°ANR-09-HABISOL-009), the CARNOT Energie du future and the cluster Energies
Rh ne-Alpes.

## References

  * [1] BS Richards.  Luminescent layers for enhanced silicon solar cell performance: Down-conversion.  Solar energy materials and solar cells, 90(9):1189–1207, may 2006\.
  * [2] Viorel Badescu, Alexis De Vos, Alina Mihaela Badescu, and Aleksandra Szymanska.  Improved model for solar cells with down-conversion and down-shifting of high-energy photons.  Journal of physics D-applied physics, 40(2):341–352, JAN 21 2007\.
  * [3] T Trupke, MA Green, and P Wurfel.  Improving solar cell efficiencies by up-conversion of sub-band-gap light.  J. Appl. Phys., 92(7):4117–4122, OCT 1 2002.
  * [4] J. L. Deschanvres, W. Meffre, J. C. Joubert, J. P. Senateur, F. Robaut, J. E. Broquin, and R. Rimet.  Rare earth-doped alumina thin films deposited by liquid source cvd processes.  J. Alloys Compd., 275-277:742 – 745, 1998.
  * [5] J-L Deschanvres, F Cellier, G Delabouglise, M Labeau, M Langlet, and J-C Joubert.  Thin-film of ceramic oxides by modified CVD.  Journal de physique, 50(C-5, 5):695–705, May 1989.
  * [6] L.A. Ryabova and Y. S. Savitskaya.  The preparation of thin films of some oxides by the pyrolysis method.  Thin Solid Films, 2:141, 1968.
  * [7] J.C. Viguié and J Spitz.  Chemical vapor deposition at low temperatures.  J. Electrochem. Soc., 122(4):585, 1975.
  * [8] J. L. Pouchou and F. Pichoir.  La Recherche Aérospatiale, 5:349–367, 1984.
  * [9] Corinna Graf, Renate Ohser-Wiedemann, and Guenter Kreisel.  Preparation and characterization of doped metal supported tio2-layers.  Journal of photochemistry and photobiology A-chemistry, 188(2-3):226–234, MAY 20 2007.