# EfficientViT-SAM: Accelerated Segment Anything Model
Without Performance Loss

Zhuoyang Zhang1,3∗, Han Cai2,3∗, Song Han2,3
1Tsinghua University, 2MIT, 3NVIDIA
https://github.com/mit-han-lab/efficientvit

###### Abstract

We present EfficientViT-SAM, a new family of accelerated segment anything
models. We retain SAM’s lightweight prompt encoder and mask decoder while
replacing the heavy image encoder with EfficientViT. For the training, we
begin with the knowledge distillation from the SAM-ViT-H image encoder to
EfficientViT. Subsequently, we conduct end-to-end training on the SA-1B
dataset. Benefiting from EfficientViT’s efficiency and capacity, EfficientViT-
SAM delivers 48.9$\times$ measured TensorRT speedup on A100 GPU over SAM-ViT-H
without sacrificing performance. Our code and pre-trained models are released
at https://github.com/mit-han-lab/efficientvit.

††footnotetext: ∗Work done during an internship at NVIDIA.

##  1 Introduction

Segment Anything Model (SAM) [1] is a family of image segmentation models
pretrained on a high-quality dataset with 11M images and 1B masks. SAM
provides astounding zero-shot image segmentation performance and has many
applications, including AR/VR, data annotation, interactive image editing,
etc.

Despite the strong performance, SAM is highly computation intensive,
restricting its applicability in time-sensitive scenarios. In particular,
SAM’s main computation bottleneck is its image encoder, which requires 2973
GMACs per image at the inference time.

To accelerate SAM, numerous efforts have been made to replace SAM’s image
encoder with lightweight models. For example, MobileSAM [2] distills the
knowledge of SAM’s ViT-H model into a tiny vision transformer. EdgeSAM [3]
trains a purely CNN-based model to mimic ViT-H, employing a meticulous
distillation strategy with the prompt encoder and mask decoder involved in the
process. EfficientSAM [4] leverages the MAE pretraining method to improve
performance.

Figure 1: Throughput vs. COCO Zero-Shot Instance Segmentation mAP.
EfficientViT-SAM is the first accelerated SAM model that matches/outperforms
SAM-ViT-H’s [1] zero-shot performance, delivering the SOTA performance-
efficiency trade-off. Figure 2: Macro Architecture of EfficientViT-SAM-XL.
‘ResBlock’ refers to the basic building block from ResNet34 [5]. ‘F-MBConv’
refers to the fused MBConv block from [6]. ‘EfficientViT Module’ is the
building block from [7].

While these methods can reduce the computation cost, they all suffer from
significant performance drops (Figure 1). This work introduces EfficientViT-
SAM to address this limitation by leveraging EfficientViT [7] to replace SAM’s
image encoder. Meanwhile, we retain the lightweight prompt encoder and mask
decoder architecture from SAM. Our training process consists of two phases.
First, we train the image encoder of EfficientViT-SAM using SAM’s image
encoder as the teacher. Second, we train EfficientViT-SAM end-to-end using the
whole SA-1B dataset [1].

We thoroughly evaluate EfficientViT-SAM on a series of zero-shot benchmarks.
EfficientViT-SAM provides a significant performance/efficiency boost over all
prior SAM models. In particular, on the COCO dataset [8], EfficientViT-SAM
achieves 48.9$\times$ higher throughput on A100 GPU without mAP drop compared
with SAM-ViT-H [1].

##  2 Related Work

###  2.1 Segment Anything Model

SAM [1] has gained widespread recognition as a milestone in the field of
computer vision, showcasing its exceptional performance and generalization in
image segmentation. SAM defines image segmentation as a promptable task, that
aims to generate a valid segmentation mask given any segmentation prompt. To
achieve this objective, SAM utilizes an image encoder and a prompt encoder to
process the image and provide prompts. The outputs from both encoders are then
fed into a mask decoder, which generates the final mask prediction. SAM is
trained on a large-scale segmentation dataset comprising over 11 million
images with more than 1 billion high-quality masks, enabling robust zero-shot
open-world segmentation. SAM has shown its high versatility in a wide range of
downstream applications, including image in-painting [9], object tracking [10,
11], and 3D generation [12, 13]. Nevertheless, the image encoder component of
SAM imposes significant computational costs, leading to high latency that
restricts its practicality in time-sensitive scenarios. Recent works [2, 3, 4,
14] have been focused on improving the efficiency of SAM, aiming to address
its computational limitations.

###  2.2 Efficient Deep Learning Computing

Improving the efficiency of deep neural networks is critical when deploying
them in real-world applications on both edge and cloud platforms. Our work is
related to efficient model architecture design [15, 16] that aims to improve
the performance-efficiency trade-off by replacing inefficient model
architectures with efficient ones. Our work is also related to knowledge
distillation [17] that uses pretrained teacher models to guide the training of
student models. Additionally, we can combine EfficientViT-SAM with other
parallel techniques to further boost efficiency, including pruning [18],
quantization [19], and hardware-aware neural architecture search [20].

##  3 Method

We propose EfficientViT-SAM, which harnesses EfficientViT [7] to accelerate
the segment anything model. In particular, our approach preserves the prompt
encoder and mask decoder architecture from SAM while replacing the image
encoder with EfficientViT. We design two series of models, EfficientViT-SAM-L
and EfficientViT-SAM-XL, offering a balanced trade-off between speed and
performance. Subsequently, we train EfficientViT-SAM using the SA-1B dataset
in an end-to-end fashion.

|  $\\#$Params(M) |  $\\#$MACs(G) | Throughput (image/s) | COCO mAP
---|---|---|---|---
SAM-ViT-H [1] | 641.1 | 2973 | 11 | 46.5
MobileSAM [2] | 9.8 | 39 | 278 | 38.7
EdgeSAM [3] | 9.6 | 20 | 449 | 42.1
EfficientSAM [4] | 25.3 | 247 | 183 | 44.4
EfficientViT-SAM-L0 | 34.8 | 35 | 762 | 45.7
EfficientViT-SAM-L1 | 47.7 | 49 | 638 | 46.2
EfficientViT-SAM-L2 | 61.3 | 69 | 538 | 46.6
EfficientViT-SAM-XL0 | 117.0 | 185 | 278 | 47.5
EfficientViT-SAM-XL1 | 203.3 | 322 | 182 | 47.8
Table 1: Runtime Efficiency Comparison. We benchmark the throughput on a
single NVIDIA A100 GPU with TensorRT, fp16.

###  3.1 EfficientViT

EfficientViT [7] is a family of vision transformer models for efficient high-
resolution dense prediction. Its core building block is a multi-scale linear
attention module that enables the global receptive field and multi-scale
learning with hardware-efficient operations. Specifically, it substitutes the
inefficient softmax attention with lightweight ReLU linear attention to have
the global receptive field. By leveraging the associative property of matrix
multiplication, ReLU linear attention can reduce the computational complexity
from quadratic to linear while preserving functionality. In addition, it
enhances the ReLU linear attention with convolution to mitigate its limitation
in local feature extraction. More details are available in the original paper
[7].

###  3.2 EfficientViT-SAM

#### Model Architecture.

The macro architecture of EfficientViT-SAM-XL is demonstrated in Figure 2. Its
backbone consists of five stages. Similar to EfficientViT [7], we use
convolution blocks in the early stages while using EfficientViT modules in the
last two stages. We fuse the features from the last three stages by upsampling
and addition. The fused feature is fed to the neck comprising several fused
MBConv blocks and then fed to the SAM head. For more details, please refer to
our GitHub repository.

#### Training.

To initialize the image encoder, we begin by distilling the image embedding of
SAM-ViT-H into EfficientViT. We employ the L2 loss as the loss function. For
the prompt encoder and mask decoder, we initialize them by loading the weights
from SAM-ViT-H. Then, we train EfficientViT-SAM on the SA-1B dataset in an
end-to-end manner.

In the end-to-end training phase, we randomly choose between the box prompt
and the point prompt with equal probability. In the case of the point prompt,
we randomly select 1-10 foreground points from the ground-truth mask to ensure
our model performs effectively for various point configurations. In the case
of the box prompt, we utilize the ground-truth bounding box. We resize the
longest side to 512/1024 for EfficientViT-SAM-L/XL models and pad the shorter
side accordingly. We select up to 64 randomly sampled masks per image. To
supervise the training process, we use a linear combination of focal loss and
dice loss, with a 20:1 ratio of focal loss to dice loss. Similar to the
approach taken in SAM to mitigate ambiguity, we predict three masks
simultaneously and only back-propagate the lowest loss. We also support single
mask output by adding a fourth output token. During training, we randomly
alternate between the two prediction modes.

We train EfficientViT-SAM on the SA-1B dataset for 2 epochs, utilizing a batch
size of 256. The AdamW optimizer is employed with a momentum of $\beta_{1}$ =
0.9 and $\beta_{2}$ = 0.999. The initial learning rate is set to 2e-6/1e-6 for
EfficientViT-SAM-L/XL, which is decayed to 0 using a cosine decay learning
rate schedule. Regarding data augmentation, we use the random horizontal flip.

##  4 Experiment

| COCO | LVIS
---|---|---
| 1 click | 3 click | 5 click | 1 click | 3 click | 5 click
SAM-ViT-H [1] | 58.4 | 69.6 | 71.4 | 59.2 | 66.0 | 66.8
EfficientViT-SAM-XL1 | 59.8 | 71.3 | 75.3 | 56.6 | 67.0 | 71.7

Table 2: Zero-Shot Point-Prompted Segmentation Results.

|  mAP |  Airplane-Parts |  Bottles |  Brain-Tumor |  Chicken |  Cows |  Electric-Shaver |  Elephants |  Fruits |  Garbage |  Ginger-Garlic |  Hand-Metal |  Hand |  House-Parts |  HouseHold-Items |  Nutterfly-Squireel |  Phones |  Poles |  Puppies |  Rail |  Salmon-Fillet |  Strawberry |  Tablets |  Toolkits |  Trash |  Watermelon
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
SAM-ViT-H [1] | 48.7 | 37.2 | 65.4 | 11.9 | 84.5 | 47.5 | 71.7 | 77.9 | 82.3 | 24.0 | 45.8 | 81.2 | 70.0 | 8.4 | 60.1 | 71.3 | 35.4 | 23.3 | 50.1 | 8.7 | 32.9 | 83.5 | 29.8 | 20.8 | 30.0 | 64.2
EfficientViT-SAM-XL1 | 48.9 | 37.5 | 66.3 | 12.1 | 85.1 | 46.9 | 70.4 | 75.6 | 82.5 | 23.9 | 45.7 | 74.8 | 79.6 | 8.4 | 60.1 | 70.9 | 35.1 | 20.0 | 50.1 | 7.4 | 40.0 | 83.3 | 29.3 | 20.7 | 29.4 | 67.7

Table 3: Zero-Shot Instance Segmentation Results on Segment in the Wild
Benchmark.

| COCO | LVIS
---|---|---
| $\rm mIoU$ | $\rm mIoU^{S}$ | $\rm mIoU^{M}$ | $\rm mIoU^{L}$ | $\rm mIoU$ | $\rm mIoU^{S}$ | $\rm mIoU^{M}$ | $\rm mIoU^{L}$
SAM-ViT-H [1] | 77.4 | 72.3 | 80.4 | 81.8 | 77.0 | 70.6 | 87.5 | 89.9
EfficientViT-SAM-XL1 | 79.9 | 75.8 | 82.2 | 83.8 | 79.9 | 74.4 | 88.4 | 91.6

Table 4: Zero-Shot Instance Segmentation Results, Prompted with Ground Truth
Bounding Box.

| COCO | LVIS
---|---|---
| $\rm mAP$ | $\rm AP^{S}$ | $\rm AP^{M}$ | $\rm AP^{L}$ | $\rm mAP$ | $\rm AP^{S}$ | $\rm AP^{M}$ | $\rm AP^{L}$
SAM-ViT-H [1] | 46.5 | 30.8 | 51.0 | 61.7 | 44.2 | 31.8 | 57.1 | 65.3
MobileSAM [2] | 38.7 | 23.7 | 42.2 | 54.3 | 37.0 | 24.7 | 47.8 | 59.1
EdgeSAM [3] | 42.1 | 26.6 | 46.7 | 56.9 | 39.8 | 28.6 | 51.3 | 59.3
EfficientSAM [4] | 44.4 | 28.4 | 48.3 | 60.1 | 41.5 | 29.7 | 53.4 | 62.2
EfficientViT-SAM-L0 | 45.7 | 28.2 | 49.5 | 63.4 | 41.8 | 28.8 | 53.4 | 64.7
EfficientViT-SAM-L1 | 46.2 | 28.7 | 50.4 | 64.0 | 42.1 | 29.1 | 54.3 | 65.0
EfficientViT-SAM-L2 | 46.6 | 28.9 | 50.8 | 64.2 | 42.7 | 29.4 | 55.1 | 65.5
EfficientViT-SAM-XL0 | 47.5 | 30.0 | 51.5 | 64.6 | 43.9 | 31.2 | 56.2 | 65.9
EfficientViT-SAM-XL1 | 47.8 | 30.5 | 51.8 | 64.7 | 44.4 | 31.6 | 57.0 | 66.4

Table 5: Zero-Shot Instance Segmentation Results, Prompted with ViTDet Boxes.

In this section, we begin by conducting a comprehensive analysis of the
runtime efficiency of EfficientViT-SAM in Section 4.1. Subsequently, we
evaluate the zero-shot capability of EfficientViT-SAM on the COCO [8] and LVIS
[21] datasets, which were not encountered during the training process. Two
distinct tasks are performed: single point valid mask evaluation in Section
4.2 and box-prompted instance segmentation in Section 4.3. These tasks
individually assess the effectiveness of the point prompt and box prompt
features of EfficientViT-SAM. We also provide results on SGinW benchmark in
Section 4.4.

###  4.1 Runtime Efficiency

We compare the model parameters, MACs, and throughput of EfficientViT-SAM with
SAM and other acceleration works. Results are shown in Table 1. We conduct the
throughput measurements on a single NVIDIA A100 GPU with TensorRT
optimization. Our results show that compared to SAM, we achieve an impressive
acceleration of 17 to 69 times. Furthermore, despite having more parameters
than other acceleration works, EfficientViT-SAM demonstrates significantly
higher throughput due to its effective utilization of hardware-friendly
operators.

###  4.2 Zero-Shot Point-Prompted Segmentation

We assess the zero-shot performance of EfficientViT-SAM in segmenting objects
based on point prompts in Table 2. We adopt the point selection method
described in [1]. That is the initial point is selected as the point located
farthest from the object boundary. Each subsequent point is chosen as the
farthest point from the boundary of the error region, which is defined as the
area between the ground truth and the previous prediction. The performance is
reported using 1/3/5 clicks on COCO and LVIS dataset, with the mIoU (mean
Intersection over Union) serving as the metric. Our results demonstrate
superior performance compared to SAM, particularly when additional point
prompts are provided.

###  4.3 Zero-Shot Box-Prompted Segmentation

We evaluate the zero-shot performance of EfficientViT-SAM in object
segmentation using bounding boxes. We first input ground truth bounding boxes
to the model, and the results are presented in Table 4. The mIoU(mean
Intersection over Union) is reported for all objects, as well as separately
for small, medium, and large objects. Our approach surpasses SAM by a
significant margin on the COCO and LVIS dataset. Next, we employ an object
detector, ViTDet [22], and utilize its output boxes as prompts for the model.
The results in Table 5 demonstrate that EfficientViT-SAM achieves superior
performance compared to SAM. Notably, even the lightest version of
EfficientViT-SAM significantly outperforms other acceleration works by a large
margin.

Additionally, we evaluate the performance of EfficientViT-SAM on the COCO
dataset using YOLOv8 and GroundingDINO [23] as the object detectors. YOLOv8 is
a real-time object detector suitable for real-world applications. On the other
hand, GroundingDINO is capable of detecting objects using text prompts,
allowing us to perform object segmentation based on textual cues. The results
presented in Table 6 reveal the outstanding performance of EfficientViT-SAM in
comparison to SAM.

| YOLOv8 | GroundingDINO
---|---|---
| $\rm AP$ | $\rm AP^{S}$ | $\rm AP^{M}$ | $\rm AP^{L}$ | $\rm AP$ | $\rm AP^{S}$ | $\rm AP^{M}$ | $\rm AP^{L}$
SAM-ViT-H [1] | 43.8 | 26.1 | 48.1 | 60.4 | 46.9 | 31.5 | 51.8 | 64.4
EfficientViT-SAM-XL1 | 44.7 | 26.0 | 48.9 | 62.9 | 48.2 | 31.5 | 52.6 | 67.3

Table 6: Zero-Shot Instance Segmentation Results on COCO, Prompted with
YOLOv8/GroundingDINO Boxes.

###  4.4 Zero-Shot In-the-Wild Segmentation

The Segmentation in the Wild benchmark consists of 25 zero-shot in-the-wild
segmentation datasets. We equip EfficientViT-SAM with Grounding-DINO as box
prompts and perform zero-shot segmentation. The comprehensive performance
results for each dataset are presented in Table 3. SAM achieves an mAP of
48.7, whereas EfficientViT-SAM achieves a higher score of 48.9.

###  4.5 Qualitative Results.

Figure 3 showcases the qualitative segmentation results of EfficientViT-SAM
when provided with point prompt, box prompt, and segment-everything mode. The
results demonstrate that EfficientViT-SAM excels not only in segmenting large
objects but also in effectively handling small objects. These findings
highlight the superior segmentation capability of EfficientViT-SAM.

Figure 3: Qualitative Segmentation Results of EfficientViT-SAM under point,
box, and everything mode.

##  5 Conclusion

In this work, we introduced EfficientViT-SAM, which utilizes EfficientViT to
replace the image encoder of SAM. EfficientViT-SAM achieved a significant
efficiency boost over SAM without sacrificing performance across various zero-
shot segmentation tasks. We have open-souced our pretrained models on GitHub
to the community.

## Acknowledgments

We thank National Science Foundation for supporting this research.

## References

  * [1] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  arXiv preprint arXiv:2304.02643, 2023.
  * [2] Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong.  Faster segment anything: Towards lightweight sam for mobile applications.  arXiv preprint arXiv:2306.14289, 2023.
  * [3] Chong Zhou, Xiangtai Li, Chen Change Loy, and Bo Dai.  Edgesam: Prompt-in-the-loop distillation for on-device deployment of sam.  arXiv preprint arXiv:2312.06660, 2023.
  * [4] Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xiang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest Iandola, et al.  Efficientsam: Leveraged masked image pretraining for efficient segment anything.  arXiv preprint arXiv:2312.00863, 2023.
  * [5] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [6] Mingxing Tan and Quoc Le.  Efficientnetv2: Smaller models and faster training.  In International conference on machine learning, pages 10096–10106. PMLR, 2021.
  * [7] Han Cai, Chuang Gan, and Song Han.  Efficientvit: Enhanced linear attention for high-resolution low-computation visual recognition.  arXiv preprint arXiv:2205.14756, 2022.
  * [8] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [9] Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen.  Inpaint anything: Segment anything meets image inpainting.  arXiv preprint arXiv:2304.06790, 2023.
  * [10] Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang.  Segment and track anything.  arXiv preprint arXiv:2305.06558, 2023.
  * [11] Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng.  Track anything: Segment anything meets videos.  arXiv preprint arXiv:2304.11968, 2023.
  * [12] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su.  One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion.  arXiv preprint arXiv:2311.07885, 2023.
  * [13] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su.  Zero123++: a single image to consistent multi-view diffusion base model.  arXiv preprint arXiv:2310.15110, 2023.
  * [14] Han Shu, Wenshuo Li, Yehui Tang, Yiman Zhang, Yihao Chen, Houqiang Li, Yunhe Wang, and Xinghao Chen.  Tinysam: Pushing the envelope for efficient segment anything model.  arXiv preprint arXiv:2312.13789, 2023.
  * [15] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [16] Han Cai, Ligeng Zhu, and Song Han.  Proxylessnas: Direct neural architecture search on target task and hardware.  arXiv preprint arXiv:1812.00332, 2018.
  * [17] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [18] Song Han, Jeff Pool, John Tran, and William Dally.  Learning both weights and connections for efficient neural network.  Advances in neural information processing systems, 28, 2015.
  * [19] Song Han, Huizi Mao, and William J Dally.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  arXiv preprint arXiv:1510.00149, 2015.
  * [20] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han.  Once-for-all: Train one network and specialize it for efficient deployment.  arXiv preprint arXiv:1908.09791, 2019.
  * [21] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019.
  * [22] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He.  Exploring plain vision transformer backbones for object detection.  In European Conference on Computer Vision, pages 280–296. Springer, 2022.
  * [23] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  arXiv preprint arXiv:2303.05499, 2023.


# Interactive tools for making temporally variable, multiple-attributes, and
multiple-instances morphing accessible

Flexible manipulation of divergent speech instances for explorational research
and education

Hideki Kawahara1 and Masanori Morise2<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>Center for Innovative and Joint Research1 Center for
Innovative and Joint Research Wakayama University Wakayama University 930
Sakaedani 930 Sakaedani Wakayama Wakayama Wakayama 640-8510 Wakayama 640-8510
Japan
2 Graduate School of Advanced Mathematical Sciences Japan
2 Graduate School of Advanced Mathematical Sciences Meiji University Meiji
University 4-21-1 4-21-1 Nakano Nakano Tokyo Tokyo 164-8525 164-8525 Japan
Japan

###### Abstract

We generalized a voice morphing algorithm capable of handling temporally
variable, multiple-attributes, and multiple instances. The generalized
morphing provides a new strategy for investigating speech diversity. However,
excessive complexity and the difficulty of preparation have prevented
researchers and students from enjoying its benefits. To address this issue, we
introduced a set of interactive tools to make preparation and tests less
cumbersome. These tools are integrated into our previously reported
interactive tools as extensions. The introduction of the extended tools in
lessons in graduate education was successful. Finally, we outline further
extensions to explore excessively complex morphing parameter settings.

Vocoder parameter, graphical user interface, interaction, generalized
morphing, calicature

Hideki Kawahara, 2-14-4 Kitayamato, Ikoma, Nara, 630-0121, Japan 5 2

##  1 Introduction

Driven by the recent resurgence of deep learning, synthesized speech is
already comparable to speech sounds spoken by humans [1, 2]. Deep learning
technologies also allow for converting voice identity and speaking style
without introducing severe deterioration in their naturalness and quality [3,
4, 2]. Synthesized speech and converted speech sounds based on those
technologies seem now divergent enough to cover the divergence of speech by
humans. However, controlling these divergent speech sounds using
physiologically and perceptually grounded relevant attributes is still
challenging [3, 1, 4, 2].

We introduced interactive GUI tools hoping to help users acquire a basic
understanding of these relations [5]. We implemented the tools using a
classical non-neural VOCODER called WORLD [6]. We designed the tools to be
upper compatible with the widely used classical VOCODER STRAIGHT [7, 8]
because a considerable amount of publications has been using STRAIGHT and
STRAIGHT-based applications [10, 11, 12, 13, 3, 14].

In this letter, we extend our WORLD-based tools [5], introducing generalized
voice morphing by migrating STRAIGHT-based tools to the WORLD-based
foundations. The next section introduces voice morphing and its
generalization, followed by the section describing the new morphing tools
based on WORLD [6].

##  2 Voice morphing and generalization

Voice morphing is a technique to generate intermediate (or exaggerated) voices
made from a set of different voice instances. Even the early development stage
of the morphing procedure [15] has led to various new findings [16, 17]. We
introduced a generalized morphing procedure capable of morphing many instances
by setting time-varying morphing rates to individual morphing parameters [18,
19]. This generalization further produced many interesting findings [23, 24,
25, 26]. However, its excessive complexity in preparation and handling
prevented many researchers from using it for their investigations. We extended
our tools [5] to make this generalized morphing more accessible to prospective
users.

###  2.1 Generalized morphing

Classical VOCODERs decompose a speech signal into a set of VOCODER parameters
[7, 8, 5]. They are filter and excitation source parameters. The filter
parameter is a time-indexed sequence of the power spectrum envelope. The
source parameters are a time-indexed sequence of aperiodicity and the
fundamental frequency.

Generalized morphing interpolates and extrapolates these parameters and
generates morphed speech sounds from the manipulated parameters. Because these
parameters are located on time and frequency axes, we added them to the
parameters to be modified. They are: (1) $\mathbf{tx:}$ time axis, a function
from index to time, (2) $\mathbf{fx:}$ frequency axis, a function from index
to frequency, (3) $\mathbf{sl:}$ spectrum level, the power spectrum envelope
value at a time-frequency location, (4) $\mathbf{fo:}$ fundamental frequency
at an indexed time location, and (5) $\mathbf{ap:}$ aperiodicity, a power
ratio of the random component and periodic component at a time-frequency
location.

Let $\mathbf{x}$ represent one of the above parameters. The following equation
represents the parameter manipulation of generalized morphing [18, 19].

$\displaystyle\mathbf{x}_{\mathrm{morph}}$
$\displaystyle=\mathcal{T}^{-1}\\!\left[\sum_{k=1}^{K}w_{k}\mathcal{T}[\mathbf{x}_{k}]\right],\
\mbox{where}\ \sum_{k=1}^{K}w_{k}=1,$ (1)

where the function $\mathcal{T}[\mathbf{x}]$ maps the original restricted
range of $\mathbf{x}$ onto the whole real number $\mathbb{R}$, and its inverse
function $\mathcal{T}^{-1}$ maps the manipulated parameter back to the
original restricted range. The variable $w_{k}$ represents a real-valued
weight to implement linear interpolation and extrapolation.

###  2.2 STRAIGHT-based morphing tools

STRAIGHT-based morphing tools implement functions $\mathbf{tx}$ and
$\mathbf{fx}$ using piece-wise linear functions by assigning anchoring points
[18, 19]. We implemented an anchor assignment GUI tool and alignment tool
using the GUID, an old UI design environment of MATLAB [20].

First, the user selects one speech instance as the reference and assigns
temporal anchoring points to the boundaries of acoustically different regions.
Then, the user assigns frequency anchoring points at prominent spectral peaks
at each temporal anchor’s location.

Second, the user selects the other speech instance as the target. The
alignment tool shows spectrographic representations of the reference and the
target side by side. The goal of alignment is to minimize the deterioration of
the perceived quality of the morphed sounds. This time-consuming and
cumbersome assignment and alignment procedures require a basic understanding
of acoustic phonetics and digital signal processing.

###  2.3 STRAIGHT-based morphing application

For psycho-acoustics, we introduced the stimuli continuum generation tool. For
demonstrations, we developed interactive three-instance morphing [19].

##  3 New WORLD-based morphing tools

The positive experience of using WORLD-based tools in class lessons [5]
encouraged us to expand their utility by introducing generalized morphing
tools. The significant performance boost of computation [21] also motivated
us. We aim to minimize the required prerequisite knowledge and make
generalized morphing accessible.

###  3.1 Assignment and alignment tool

We integrated the anchor assignment and instance alignment tools and developed
one GUI tool for morphing object preparation. Figure 1 shows the snapshot of
the GUI of the tool at the final stage of time-frequency alignment (at 6:41 of
the video “10” in Fig. 2). In the following sections, we will use video
snapshots of this YouTube channel [22] for the introduction.

Figure 1: GUI snapshot of the morphing object preparation. This snapshot shows
the final stage of time-frequency alignment (at 6:41 of the video “10” in Fig.
2). The sample (WAVE: 44100 Hz, 32 bit) is a Japanese word /hai/ (“Yes” in
English) spoken by the first author. Figure 2: Tutorial video channel of WORLD
vocoder tools. This snapshot lists morphing tutorial videos. The QR code links
to this YouTube channel [22].

####  3.1.1 Outline of the morphing object preparation

The upper right image of Fig. 1 shows two time-frequency representations of
the filter parameter $\mathbf{sl}$ and two waveform representations overlaid
with each other respectedly. These parameter images are half transparent, and
it is easy to observe their alignment status.

We put one set of morphing parameters on the “canonical” time-frequency axis.
This axis is fixed and does not change throughout the morphing object
preparation procedure. We put the other morphing parameters on the “non-
linear” time-frequency axis. The user aims to deform this “non-linear” axis to
align representations using temporal anchors (vertical lines) and frequency
anchors (circles on the lines). The middle left plot shows the distance
between two overlaid images. The user’s task is to minimize the distance by
deforming the “non-linear” time-frequency axis. Pursuing this task does not
require expertized knowledge and skills. We successfully used this video
tutorial and a brief tutor introduction in lesson classes.

####  3.1.2 Step-by-step procedure

Figure 3 shows magnified views of the GUI control parts of the morphing object
preparation tool (Fig. 1). This section introduces the morphing object
preparation procedure using the controllers step-by-step.

Figure 3: GUI control parts for manipulation.

##### Reading VOCODER parameter files:

The first step is to read two sets of VOCODER parameters to the “canonical”
axes and “non-linear” axes. We use buttons on the top left (a) of the GUI.
They are READ on Canonical Ax, READ on Nonlinear Ax, and RESET from top to
bottom. Clicking the READ on Canonical Ax button invokes file input dialogue
to read a VOCODER parameter file. After reading the file, read the other
VOCODER parameter using the READ on Nonlinear Ax button. Completion of this
step fills the spectrogram and the waveform displays. The image (e), shows the
VOCODER file names; “canonical” file name (colored black), and the “non-
linear” file name (colored green).

The initial condition of the displays is half-transparent, and two overlaid
representations are visible. Selecting the “Canonical” button on top of the
image (b) only shows parameters on the “canonical” axes. The “NL-TFaxis”
button only shows the parameters on the “non-linear” axes. The “Variable”
button resumes.

##### Temporal anchor assignment and adjustment:

The next step is to assign and adjust temporal anchors. Figure 4 shows just
after the first temporal anchor setting and adjustment (at 1:08 in the video).
The alignment of parameters is poor at this point (0.2 s to 0.3 s; the second
formant: F2 trajectories are displaced). The user can check alignment by
synthesizing speech from parameters using Synthesis, PLAY Cx, and PLAY NLx
buttons. Click plays back “morhed,” “cannonical,” and “non-linear” parameters’
sound.

While the mouse cursor shape is “arrow” or a small “cross-hair,” clicking in
the waveform display places a temporal anchoring point. It also draws a
vertical line in each display. The Undo button at the bottom of the image (b)
removes the latest placed temporal anchor. The Clear button removes all
temporal anchors.

When the mouse cursor is close to one of the temporal anchors, its shape
changes to a “hand.” Then, clicking an anchor (a thin vertical line) changes
the anchor to a thick purple line and draggable (Figure 4). Dragging the
anchor deforms the green waveform and the distance trajectory on the left
middle plot changes. The green distance trajectory shows a sudden drop at the
end. Adding and adjusting temporal anchors with deformation, the temporal
alignment procedure finished at 2:25. Completing this process assigns the
temporal anchors to parameters displayed on “non-linear” axis.

Figure 4: Temporal anchor assignment and adjustment (at 1:08 in video).

##### Frequency anchor assignment:

The frequency anchors are attached to the representation on the “non-linear”
spectrogram axis. Figure 5 shows after the second frequency anchor assignment
(at 3:09). When nearing one of the temporal anchor lines in the spectrogram
display, the mouse cursor shape changes into a small “cross-hair.” It
indicates that frequency anchor assignment is possible. In this situation,
clicking places a frequency anchor and briefly displays a large yellow circle
to confirm the assignment. Note that the frequency axis is magnified (see
“Freq. limit” panel on the image (b)).

Similar to temporal anchors, “undo” and “clear” functions are available on the
panel “TF anchor” of (b). Completing this process assigns the frequency
anchors associated with parameters on “non-linear” axis.

Figure 5: Frequency anchor assignment on the “non-linear” representation. (at
3:09)

##### Frequency anchor alignment:

Figure 5 shows adjusting an anchor on the F2 trajectory at 0.16 s. The “hand”
shaped mouse cursor indicates that it focuses on the closest adjustable
anchor. Note that the frequency axis is further magnified (the limit is 2 kHz)
to make alignment easier. The displays show time-frequency representation and
waveform on the “canonical” axes.

The user’s task is to adjust the location of the selected anchor (attached on
the “non-linear” axis) to the relevant location. In Fig. 6, the user tries to
move and adjust the anchor on the F2 trajectory. Note that the location of the
temporal anchors can also be adjusted during this process. Completing this
process assigns the temporal anchors and frequency anchors associated with
parameters displayed on the “canonical” axes.

Figure 6: Frequency anchor alignment on the “Canonical” representation. (at
5:04)

##### Results inspection and saving morphing objects:

Using buttons on the lower left of the GUI (image (c) of Fig. 3), the user can
inspect the synthesized sounds and save the results at any time. The SAVE EDIT
button saves the intermediate state of assignment and alignment. After saving
the state, users can terminate the application. The Restore EDIT button
recovers the state of assignment and alignment, and users can resume the
terminated task. Moving the morphing rate slider, (d) of Fig. 3 playbacks the
morphed sound on each mouse-up event (from 6:18 to 6:34). When satisfied with
the results, click of SAVE m-Object saves the results as re-usable morphing
objects (at 6:41).

##### Anchoring to new VOCODER parameters:

The temporal and frequency anchors can be assigned and aligned to a new
VOCODER parameter file using this tool and a morphing object. This procedure
produces a new morphing object. First, read a new VOCODER parameter file using
READ on Canonical Ax button. Then, read a morphing object file using READ on
Nonlinear Ax button. This procedure brings the state to the beginning of
“Frequency anchor alignment.” The remaining processes are the same (from 7:42
to 13:51). Note that the temporal anchor position is also adjustable here.

###  3.2 Multi-item-morphing demonstration tool

We prepared an interactive application of the generalized voice morphing
demonstration using the modern environment “App Designer” of MATLAB [20] that
allows live debugging. Figure 7 shows a snapshot of the application. Three
colored buttons at the center bottom load three morphing objects on each
colored sphere. After loading morphing objects, the mouse-up event of dragging
the half-transparent gray sphere (the control knob) on the left-top image
generates a morphed speech sound. The location of the control knob sets three
morphing weights using area coordinate [27]. The weights define the volume of
the corresponding color spheres. A negative valued weight deforms the sphere
into a bowl shape. The bottom right area with many radio buttons enables up to
five weighting patterns, making it easy to explore “parameter-specific
morphing” [23, 25, 26]. The area also can visualize the morphed attributes. An
interesting application is to assign divergent speech samples to spheres and
explore “parameter-specific morphing.”

Figure 7: Application snapshots of three instances morphing demonstration
(setting and display modes).

##  4 Discussion

The most crucial issue of this set of tools based on classical VOCODER is its
sound quality. The sound quality of neural VOCODERs is significantly higher
than that of classical VOCODERs. One possible solution is replacing the tool’s
classical VOCODER with a neural VOCODER that shares similar VOCODER parameters
[28, 29]. This solution needs a framework to ensure precise control of the
acoustic parameters of the generated morphed sounds with the users’ intention.

The rapid progress of speech technologies by deep learning makes automatic
assessment of speech inevitable [2]. While at the same time, researchers’
basic understanding of speech science is increasingly important. Providing
lightweight, easy-to-use speech research tools like ours will help with these
needs.

Finally, the generalized morphing framework is general enough to replace
VOCODER parameters with latent variables of end-to-end speech technologies [3,
1, 2]. We speculate this leads to a prospective merger.

##  5 Conclusion

We extended our WORLD-based tools [5] to make generalized morphing [18, 19]
accessible. We made the tools open-source [30] and prepared a YouTube Channel
[22] and assisting materials. We hope extended WORLD-based tools help
researchers and students acquire deep implicit knowledge of speech production,
perception, and signal processing while benefiting from the resurgence of deep
learning-based speech technologies.

## Acknowledgement

We appreciate Toshio Irino, Toshie Matsui, and Nao Hodoshima and their
students’ feedback and comments on the early versions of the WORLD-based tools
and the generalized morphing extensions. The Japan Society for the Promotion
of Science (JSPS) Grants-in-Aid for Scientific Research supported this work.
The supporting grant numbers are JP20H00291, JP21K19794, JP21H00497, and
JP21H04900.

## References

  * [1] X. Tan, _et al._ , “End-to-end text-to-speech synthesis with human-level quality,” _IEEE Trans. PAMI_ , doi: 10.1109/TPAMI.2024.3356232, (2024) (Advanced online publication).
  * [2] E. Cooper, _et. al._ , “A review on subjective and objective evaluation of synthetic speech,” _Acoustical Science & Technology_, Article advpub, e24.12, (2024) (Advanced online publication).
  * [3] B. Sisman, J. Yamagishi, S. King, and H. Li, “An overview of voice conversion and its challenges: From statistical modeling to deep learning,” _IEEE/ACM Trans. Audio, Speech, and Language Processing_ , 29(5), 132–157 (2020).
  * [4] W.C.Huang, L.P.Violeta, S.Liu, J.Shi, and T. Toda, “The singing voice conversion challenge 2023,” _2023 IEEE Automatic Speech Recognition and Understanding Workshop_ , 1–8 (2023).
  * [5] H. Kawahara, and M. Morise, “Interactive tools for making vocoder-based signal processing accessible,” _Acoustical Science & Technology_, 45(1), 48–51 (2024).
  * [6] M. Morise, F. Yokomori, and K. Ozawa, “WORLD: A vocoder-based high-quality speech synthesis system for real-time applications,” _IEICE Trans. Information and Systems_ , 99(7), 1877–1884 (2016).
  * [7] H. Kawahara, I. Masuda-Katsuse, and A. de Cheveigné, “Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction,” _Speech Communication_ , 27(3–4), 187–207 (1999).
  * [8] H. Kawahara, M. Morise, T. Takahashi, R. Nisimura, T. Irino, and H. Banno, “Tandem-STRAIGHT: A temporally stable power spectral representation for periodic signals and applications to interference-free spectrum, F0, and aperiodicity estimation,” in _Proc. ICASSP 2008_ , 3933–3936 (2008).
  * [9] C. Liu, and D. Kewley-Port, “Vowel formant discrimination for high-fidelity speech,” _J. Acoust. Soc. Am._ , 116(2), 1224–1233 (2004).
  * [10] D. R. Smith, D. R. Patterson, R. Turner, H. Kawahara, and T. Irino, “The processing and perception of size information in speech sounds,” _J. Acoust. Soc. Am._ , 117(1), 305–318 (2005).
  * [11] H. Zen, T. Toda, M. Nakamura, and K. Tokuda, “Details of the Nitech HMM-based speech synthesis system for the Blizzard Challenge 2005,” _IEICE Trans. Information and Systems_ , 90(1), 325–333 (2007).
  * [12] K. Tokuda, Y. Nankaku, T. Toda, H. Zen, J. Yamagishi, and K. Oura, “Speech synthesis based on hidden Markov models,” _Proceedings of the IEEE_ , 101(5), 1234–1252 (2013).
  * [13] T. Uchida, “Reversal of relationship between impression of voice pitch and height of fundamental frequency: Its appearance and disappearance,” _Acoust. Sci. & Tech._, 40(3), 198–208 (2019).
  * [14] T. Uchida, “Voice pitch illusion and perception of speaker’s body size: Relationship with the spectral tilt in speech sound,” _Acoust. Sci. & Tech._, 43(1), 73–76 (2022).
  * [15] H. Kawahara and H. Matsui, “Auditory morphing based on an elastic perceptual distance metric in an interference-free time-frequency representation,” in _Proc. ICASSP 2003_ , I, 256–259 (2003).
  * [16] S.R. Schweinberger, _et. al._ , “Auditory adaptation in voice perception,” _Current Biology_ , 18(9), 684–688 (2008).
  * [17] L. Bruckert, _et. al._ , “Vocal attractiveness increases by averaging,” _Current Biology_ , 20(2), 116–120 (2010).
  * [18] H. Kawahara, M. Morise, H. Banno, and V. G. Skuk, “Temporally variable multi-aspect N-way morphing based on interference-free speech representations,” in _Proc. APSIPA ASC 2013_ , 1–10 (2013).
  * [19] H. Kawahara, “Temporally variable multi attribute morphing of arbitrarily many voices for exploratory research of speech prosody,” in _Speech prosody in speech synthesis: Modeling and generation of prosody for high quality and flexible speech synthesis_ , K. Hirose and J. Tao, Eds. (Springer, Berlin, Heidelberg, 2015), pp.109–120.
  * [20] MATLAB and toolbox manuals, _The Mathworks, Inc._ , Natick, Massachusetts, USA, (2023).
  * [21] C.E. Leiserson, _et. al._ , “There’s plenty of room at the Top: What will drive computer performance after Moore’s law?,” _Science_ , 368(6495), eaam9744, (2020).
  * [22] H. Kawahara, “World vocoder tools,”
Search “Kawahara World vocoder tools YouTube”
Retrieved 2024-04-09.

  * [23] V.G. Skuk, and S.R. Schweinberger, “Influences of fundamental frequency, formant frequencies, aperiodicity, and spectrum level on the perception of voice gender,” _J. Speech Lang. Hear. Res._ , 57(1), 285–296 (2014).
  * [24] H. Kawahara and V. Skuk, “Voice morphing,” in _The Oxford handbook of voice perception_ , S. Frühholz and P. Belin, Eds. (Oxford University Press, Oxford UK, 2019), Chapter 31, pp. 685–706.
  * [25] V.G. Skuk, _et. al._ , “Parameter-specific morphing reveals contributions of timbre and fundamental frequency cues to the perception of voice gender and age in cochlear implant users,” _J. Speech Lang. Hear. Res._ , 63(9), 3155–3175 (2020).
  * [26] C. Nussbaum, C.I. von Eiff, V.G. Skuk, and S.R. Schweinberger, “Vocal emotion adaptation aftereffects within and across speaker genders: Roles of timbre and fundamental frequency,” _Cognition_ , 219, 104967 (2022).
  * [27] Q.Y. Li, J.R. Zheng, G. Liao, and Y. Jin, “Approach on area coordinate, volume coordinate and their application in true 3DGIS,” _Earth Science and Engineering_ , 1(1), 158–164, (2011).
  * [28] Y. Yasuda, X. Wang, J. Yamagishi, “Investigation of learning abilities on linguistic features in sequence-to-sequence text-to-speech synthesis,” _Computer Speech & Language_, 67, 101183, (2021).
  * [29] R. Yoneyama, Y. -C. Wu and T. Toda, “High-fidelity and pitch-controllable neural vocoder based on unified source-filter networks,” _IEEE/ACM Trans. ASLP_ , 31, 3717–2023, (2023).
  * [30] H. Kawahara, “WORLD GUI tools for explorational research/education of speech science and technology,”
https://github.com/HidekiKawahara/worldGUItools Retrieved 2024-04-07.


# Knowledge Distillation from BERT Transformer to Speech Transformer for
Intent Classification

###### Abstract

End-to-end intent classification using speech has numerous advantages compared
to the conventional pipeline approach using automatic speech recognition
(ASR), followed by natural language processing modules. It attempts to predict
intent from speech without using an intermediate ASR module. However, such
end-to-end framework suffers from the unavailability of large speech resources
with higher acoustic variation in spoken language understanding. In this work,
we exploit the scope of the transformer distillation method that is
specifically designed for knowledge distillation from a transformer based
language model to a transformer based speech model. In this regard, we
leverage the reliable and widely used bidirectional encoder representations
from transformers (BERT) model as a language model and transfer the knowledge
to build an acoustic model for intent classification using the speech. In
particular, a multilevel transformer based teacher-student model is designed,
and knowledge distillation is performed across attention and hidden sub-layers
of different transformer layers of the student and teacher models. We achieve
an intent classification accuracy of 99.10% and 88.79% for Fluent speech
corpus and ATIS database, respectively. Further, the proposed method
demonstrates better performance and robustness in acoustically degraded
condition compared to the baseline method.

Index Terms: Spoken language understanding, speech to intent, knowledge
distillation, transformer, BERT

##  1 Introduction

The spoken language understanding (SLU) systems aim to derive the meaning of
users’ queries, which helps the conversational agent to respond appropriately
to the user. The SLU performs several tasks, namely, domain identification,
intent classification, and slot filling. As a crucial part of SLU, the intent
classification module is designed to extract the meaning or intention of the
user’s spoken utterance. There is a wide range of interactive interface-based
applications requiring SLU systems, namely, voice search and chatbot on mobile
devices, attracting interest from both commercial and academic sectors.

The conventional pipeline SLU system for intent classification contains two
modules, namely, an automatic speech recognition (ASR) system that converts
the speech to text sequence, and a natural language understanding (NLU) module
that maps the text transcript to the speaker’s intent. The end-to-end
architectures are widely used in several domains, like ASR [1, 2, 3], speech
synthesis [4], machine translation [5, 6, 7]. The end-to-end frameworks can be
used to overcome the issues with the conventional pipeline SLU system, which
has also gained popularity in the field of SLU [8, 9, 10].

The end-to-end SLU acts as an individual single model and it directly predicts
the intent from speech without exploiting an intermediate text representation
from ASR. There are many advantages of using the end-to-end system against the
conventional pipeline SLU system. In particular, it aims to optimize the
performance metric which is intent classification accuracy directly without
using any intermediate ASR modeling. This results into a compact neural
network model and eliminates the issue caused by intermediate transcription
error [11]. Secondly, human interpretation of a spoken utterance depends on
prosody, speech rate, and loudness, which are not considered in the pipeline
SLU system.

Due to the highly variable and complex nature of speech, a large amount of
data and intensive computational resources are required for the training of
end-to-end system. The computer vision [12, 13, 14], natural language
processing (NLP) [15, 16, 17, 18, 19], and ASR [20, 21] communities have
attempted to tackle this issue by incorporating knowledge from pre-trained
models trained on diverse set of large data and share commonalities across
different tasks. The pre-trained acoustic and linguistic models can be
finetuned for the downstream SLU tasks, which is proven to be an effective
paradigm [22, 23]. The study presented in [11] explores acoustic feature
extraction using a pre-trained ASR model from spoken utterances, which are
further used for intent classification [23].

The neural transformers have shown encouraging performance on several
benchmarks in the field of NLP [24, 25]. However, their capabilities have not
been fully investigated in the field of SLU. In [26], transformer network has
been successfully used for speech recognition. In [27], the authors leverage
the transformer architecture for end-to-end SLU. The self-attention mechanism
in the transformer enables the extraction of the semantic context from a
spoken utterance. Additionally, the self-attention allows to compute the
correlation in each sub-layer between all pairs of time steps [24]. This makes
the transformer based SLU architecture more efficient in capturing longer
sequence information than the recurrent neural network (RNN) based models
[27].

A well-trained model captures meaningful knowledge or information for a
specific task. The knowledge distillation approach aims to distill the
learning capacity of a larger deep neural network (teacher model) to a smaller
network (student model) [28, 29]. It has shown efficacy in cross-modal
scenarios, where the teacher model is trained on one modality and the
knowledge is transferred to another modality [14, 30]. Previous works in
intent classification demonstrate the usefulness of knowledge distillation,
which transfers the knowledge of linguistic embeddings extracted from a
teacher model (language model), such as BERT to a student model (speech model)
[31, 22, 23, 32].

Figure 1: The proposed STD model: (a) overall architecture and different loss
computation, (b) knowledge distillation losses (i.e., computation of
$Loss_{att}$ and $Loss_{hid}$) between two transformer layers of student and
teacher model expanded from dotted box in (a).

We observe that the attention weights capture rich linguistic information in
the BERT model [33]. The linguistic knowledge captured in the BERT model
includes syntax and co-reference information, which are crucial for intent
classification task [34]. In [34], an effective transformer-to-transformer
knowledge distillation method has been proposed, where the knowledge encoded
in a large teacher BERT model is transferred to a small student TinyBERT
model. This specific type of distillation [34] consists of attention based and
hidden state based distillation, which are performed from one layer of the
teacher model to corresponding layer of the student model. Motivated from
these, we propose the attention and hidden state based distillation from the
teacher BERT model to a student transformer model.

In this work, we are inspired by the success of the transformer based speech
models for speech recognition tasks and the BERT model for various NLP tasks.
As we know human interpretation of conversation or spoken utterance does not
equally rely on all words in the utterance. Some words in a spoken utterance
are more important than others to infer the meaning of the utterance. The
attention mechanism in a speech transformer model allows us to look at the
most relevant frames of speech embeddings based on speech-specific features.
Similarly, transformer based language model pays more attention to the
relevant words in a sentence based on linguistic knowledge. We would like to
explore the transfer learning method, where information from both attention
and hidden layers of the BERT model can be distilled to the speech transformer
model. To achieve this, we propose a multilevel transformer based teacher-
student model, where the knowledge distillation on both attention and hidden
layers is performed for the intent classification task. We also use a pre-
trained ASR model to extract speech embeddings from the spoken utterances. The
proposed architecture reduces the resource requirement of transformer based
SLU as we are employing both pre-trained ASR model and BERT model.
Furthermore, the proposed transformer layer distillation leverages information
from both linguistic and acoustic aspects.

##  2 Speech Transformer Distillation (STD) Framework

In this section, we discuss the proposed knowledge distillation approach,
which we refer to as speech transformer distillation (STD) framework. Figure 1
shows the proposed STD framework for intent classification. It mainly consists
of five components, namely, pre-trained ASR model, speech-transformer encoder
as student model, BERT model as teacher model, transformer-layers distillation
and intent layer. Here, the parameters of the pre-trained ASR model, BERT-
embedding extraction and BERT model are kept fixed, which do not change during
the training process. The parameters of the speech transformer encoder and
intent layer are updated during the training process.

Initially, we extract the acoustic embeddings from the pre-trained ASR model
as described in [11] and BERT embeddings using the BertTokenizer package
111https://github.com/huggingface/transformers. Then the acoustic embeddings
($E_{\text{acoustic}}$) and BERT embeddings ($E_{\text{bert}}$) are fed into
the student and teacher models, respectively. Both the student and teacher
networks compose of identical transformer based model structure. We perform
transfer learning from one layer of the teacher model to the corresponding
layer of the student model to minimize the distance based on two aspects,
namely, attention matrices and hidden state vectors. The framework is
optimized by three losses, namely, intent loss from ground-truth label,
attention loss from the attention based distillation and hidden loss from the
hidden state based distillation. Further details of each component of the
proposed STD framework are as follows.

###  2.1 Pre-trained ASR Model

We follow the same strategy to use a pre-trained ASR model as described in
[11] and a 256-dimensional hidden representation is obtained from speech.
These hidden representations are used as acoustic embeddings
($E_{\text{acoustic}}$) and applied as input to the student speech transformer
model. More details about training strategy of the pre-trained ASR model can
be referred from [11]. We freeze this pre-trained ASR model and don’t update
the parameters while using for downstream transformer model.

###  2.2 Speech Transformer Encoder

The proposed SLU encoder structure is based on the neural transformer network
similar as [35]. The transformer architecture consists of three sub-modules:
embedding, encoder and decoder. In the STD framework, we are using only the
speech transformer encoder component. The encoder contains $N_{\text{S}}$
identical transformer layers, and each layer has two sub-layers: multi-head
attention (MHA) and position-wise feed-forward network (FFN) as shown in
Figure 1 (a). This student speech transformer encoder model produces the
attention matrices as $S_{att}$, which are output from the MHA structures, and
hidden state vectors as $S_{hid}$.

###  2.3 BERT Embedding and Model

To take the benefit from the large unlabeled data and derive better
representation of language model pre-training techniques are used in BERT
model [19]. The BERT model consists of multiple bidirectional transformer
layers. In Figure 1 (a) we show the BERT teacher model and the underlying
structure of each layer is shown in Figure 1 (b). For each text utterance, the
tokens obtain from the BertTokenizer are converted into embeddings $E_{bert}$,
which are given as input to the BERT model. In different layers of the BERT
model, we produce the attention matrices as $T_{att}$, which are output from
the MHA structures, and the hidden states as $T_{hid}$.

###  2.4 Transformer-layers Distillation

In order to leverage information from linguistically rich informative BERT
model into a speech model, we employ transformer-to-transformer distillation.
This comprises of the layer-by-layer attention based distillation and hidden
state based distillation as shown in Figure 1. Figure 1(b) shows that the
knowledge distillation is performed between each pair of transformer layers of
the teacher and student models.

Each distillation procession includes attention based distillation and hidden
state based distillation. There are $N_{T}=12$ transformer layers in the BERT
model, while we set $N_{S}=4$ layers in the student speech transformer encoder
model. To perform a layer-by-layer distillation, we combine $N_{T}/N_{S}$
layers into a block, which results in 4 blocks in the BERT teacher model.
Then, the last layer in each block is chosen for distillation with each layer
in the speech transformer encoder structure. Figure 1(b) shows the transformer
layer-by-layer distillation process corresponding to a pair of teacher-student
layers.

In order to perform the attention based distillation, we minimize mean square
error (MSE) distance between the attention matrices of student and teacher
models. We refer to this as $Loss_{att}$, which is computed as follows.

$Loss_{att}=\text{MSE}_{loss}(S_{att},T_{att}),$ (1)

where, $MSE_{loss}$ indicates the mean squared error loss function. The number
of attention matrices are dependent on the number of heads used in the student
and teacher models. As shown in Figure 1(b), we apply mean pooling (MP) to
derive average of attention matrices over all attention heads (size $L\times
L$), which are $S_{att}$ and $T_{att}$ for student and teacher model,
respectively. $L$ is the sequence length.

Next, for the hidden state based distillation, we first derive the hidden
state vectors of the transformer layers corresponding to the teacher and
student models, which are $T_{hid}$ and $S_{hid}$, respectively. These are
used to distill the knowledge from hidden state vectors as follows.

$Loss_{hid}=\text{MSE}_{loss}(S_{hid}W^{H},T_{hid}),$ (2)

where, $S_{hid}\in\mathbb{R}^{L\times S_{dmodel}}$,
$T_{hid}\in\mathbb{R}^{L\times T_{dmodel}}$. Here we use a learnable linear
transformation $W^{H}\in\mathbb{R}^{S_{dmodel}\times T_{dmodel}}$, which
converts hidden states of student network into same dimension as the teacher
network’s states.

###  2.5 Intent Layer

The intent layer is responsible for the final intent classification after the
knowledge distillation process. Here, we use a linear layer for intent
classification. The intent classification loss is referred to as
$Loss_{intent}$, which is computed by cross-entropy calculation. The intent
loss is calculated as follows:

$Loss_{\text{intent}}=\text{CrossEntropy}(Intent_{\text{pred}},y),\\\ $ (3)

where, $Intent_{pred}$ is the prediction from intent layer, and $y$ is the
ground truth label.

Next, we combine $Loss_{intent}$ with the two distillation losses
($Loss_{att}$ and $Loss_{hid}$) using the weights to make them in the similar
scale. The total loss ($Loss_{total}$) is further used to back-propagate the
whole framework.

$Loss_{total}=\alpha_{1}Loss_{intent}+\alpha_{2}Loss_{att}+\alpha_{3}Loss_{hid},$
(4)

where, $\alpha_{1}$, $\alpha_{2}$, and $\alpha_{3}$ are the weights associated
with three losses to tune the proportion of each loss in the total loss for
better back-propagation.

##  3 Experimental Details

In this section, we describe the database used and experimental details to
validate the proposed STD framework for intent classification using speech. We
have released the experimental set up of this work in the following link
222https://github.com/Jiang-Yidi/TransformerDistillation-SLU.

###  3.1 Dataset

We evaluate the proposed STD framework on two well-known SLU datasets, namely,
Fluent speech commands (FSC) [11] and ATIS [36]. Details of the datasets are
presented in Table 1.

Table 1: Details of Fluent speech commands (FSC) and ATIS databases used.

Specification Database | FSC | ATIS
---|---|---
Train | Test | Train | Test
No. of utterances | 23132 | 3793 | 5253 | 580
Total Duration (sec) | 882.95 | 154.58 | 569.90 | 61.66
Avg duration of utterance (sec) | 2.29 | 2.45 | 6.51 | 6.37
No. of intents | 31 | 31 | 15 | 15

###  3.2 Baseline Systems

To show the comparative performance of the proposed STD framework, we develop
two baseline systems. In the first baseline (Baseline-1), we initially extract
the acoustic embedding vector ($E_{\text{acoustics}}$) for each spoken
utterance using the pre-trained ASR model as discussed in Section 2.1. The
256-dimensional $E_{\text{acoustics}}$ vector is further modeled using a RNN
layer, which is a gated recurrent unit (GRU), followed by maxpooling, and
linear layer for intent classification as discussed in [11].

The second baseline (Baseline-2) is the transformer based end-to-end SLU model
[27], where, we first extract the 256-dimensional $E_{\text{acoustics}}$ from
the pre-trained ASR model as in Section 2.1. The $E_{\text{acoustics}}$ vector
is passed through the speech transformer encoder as discussed in Section 2.2
followed by the intent layer discussed in Section 2.5. The Baseline-2 system
demonstrates the impact of the proposed transfer learning process when we
compare with the proposed STD framework.

In addition to the above mentioned baselines, we also consider the original
text to obtain the performance with only text based NLU model using BERT (BERT
model).

###  3.3 Proposed System

In the proposed STD framework, we first extract $E_{acoustic}$ from each
spoken utterance using the pre-trained ASR model and $E_{bert}$ from the
corresponding text using the BERT embedding module. We pass $E_{acoustic}$ and
$E_{bert}$ to the student and teacher models, respectively. For the teacher
model, we use the `bert-base-uncased` model without fine-tuning to capture the
linguistic information. The teacher model consists of 12 transformer layers
($N_{T}=12$), each of which comprises of 12 attention heads ($T_{nhead}=12$)
and 768 hidden units ($T_{dmodel}=768$) [19]. For the student model, 8 no. of
heads ($S_{nhead}=8$), 512 hidden units ($S_{dmodel}=512$) and 4 no. of layers
($N_{S}=4$) are set based on experimental evidence.

During training, two transformer-layer distillation losses as shown in Eq (1),
Eq (2), and intent loss in Eq (3) are used to back-propagate the speech
transformer encoder and intent layers simultaneously. The values of
$\alpha_{1}$, $\alpha_{2}$ and $\alpha_{3}$ are set as 0.625, 0.125 and 0.250,
respectively to calculate the total loss.

To overcome the over-fitting while calculating the intent loss as per Eq (3),
we use label smoothing as a regularization technique. We set the smoothing
parameter as 0.1 during loss calculation as given in
333https://github.com/kaituoxu/Speech-
Transformer/blob/master/src/transformer/loss.py. The optimization technique
used for training the STD framework is Adam optimizer with $\beta_{1}=0.9$,
$\beta_{2}=0.98$, $\epsilon=10^{-9}$ and learning rate scheduling is followed
as mentioned in [35]. The entire STD framework is implemented using the
PyTorch library 444https://pytorch.org/.

To investigate the effectiveness of individual distillation process, we
develop two versions of the proposed STD framework. Firstly, we use only
hidden state based distillation (Proposed: STD (hidden)) and secondly, we
include both attention and hidden state based distillation in the proposed STD
framework (Proposed: STD).

In order to analyze the impact of noise on the Proposed:STD model (with
knowledge distillation) compared to the Baseline-2 (without knowledge
distillation), we evaluate the performance of both the systems under
acoustically degraded condition. We add babble noise from NOISEX-92
database555http://mi.eng.cam.ac.uk/comp.speech/Section1/Data/noisex.html at
different levels (15 dB, 10 dB, 5 dB, 0 dB) of signal-to-noise ratio (SNR) to
the speech signals corresponding to the test data of both ATIS and FSC
databases [37, 38]. The degraded test speech signals are passed through the
Baseline-2 and proposed STD systems to test the respective performances.

##  4 Experimental Results

In Table 2, we present the intent classification accuracy for the BERT,
baseline and the proposed models. We observe that the proposed STD framework
improves the intent classification performance as compared to the Baseline-1
and Baseline-2 (no distillation). While using the only hidden state based
distillation (Proposed:STD(hidden)), the result indicates that only hidden
state vector contributes to significant performance improvement as compared to
Baseline-2 (no distillation). Further, using both attention and hidden state
based knowledge distillation (Proposed:STD), we achieve a reasonably good
performance in the intent classification task for both FSC and ATIS databases.

Table 3 shows that the performance of the proposed STD framework is better
than the Baseline-2 in presence of noise with different SNR levels. The intent
classification accuracy degrades as SNR level decreases. Specifically, we
observe more degradation in performance with noise for the FSC database than
the ATIS database. We note that the Baseline-2 refers to the conventional
speech transformer SLU model without distillation. This indicates the
stability of the proposed STD framework against the acoustic degradation.

Table 2: Intent classification accuracy (%) of baseline methods and proposed
STD model for Fluent speech commands (FSC) and ATIS databases.

Model | FSC | ATIS
---|---|---
BERT model | 100 | 94.48
(using original text) |  |
Baseline-1 [11] | 98.80 | 85.43
Baseline-2 | 98.87 | 86.55
Proposed: STD (hidden) | 99.00 | 88.44
Proposed: STD | 99.10 | 88.79

Noise level (dB) $\rightarrow$ 15 10 5 0 Database $\downarrow$ Model
$\downarrow$ FSC Baseline-2 93.20 74.80 45.90 14.90 Proposed:STD 95.60 78.90
48.20 17.60 ATIS Baseline-2 82.41 81.03 76.90 71.03 Proposed:STD 86.90 85.86
81.21 72.59

Table 3:  Intent classification performance of Baseline-2 and proposed STD
model for Fluent speech commands (FSC) and ATIS databases under acoustically
degraded condition.

##  5 Conclusions

In this paper, we introduce a knowledge distillation approach from BERT
transformer to speech transformer for intent classification using speech. We
demonstrate that the transformer based distillation helps to improve speech
transformer model-based SLU for intent classification. In particular, we
perform layer-by-layer knowledge distillation with respect to two components
of the transformer model, namely, hidden state vectors and attention matrices.
The experimental results on FSC and ATIS databases show improved accuracy
after incorporation of transformer based knowledge distillation. Thus, we
observe that the knowledge distillation can learn from rich informative BERT
model and improve the intent classification performance using only speech. The
experiments under noisy condition confirm the better robustness of the
proposed architecture. In future, we would like to explore the possible
temporal alignment across speech and text for intent classification.

##  6 Acknowledgements

This research is supported by the Agency for Science, Technology and Research
(A*STAR) under its AME Programmatic Funding Scheme (Project No. A18A2b0046)
and Science and Engineering Research Council, Agency of Science, Technology
and Research, Singapore, through the National Robotics Program under Grant No.
192 25 00054.

## References

  * [1] D. Amodei et al., “Deep speech 2 : End-to-end speech recognition in English and mandarin,” in ICML, vol. 48, pp. 173–182, 2016.
  * [2] W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in ICASSP, pp. 4960–4964, 2016.
  * [3] H. Soltau, H. Liao, and H. Sak, “Neural speech recognizer: Acoustic-to-word LSTM model for large vocabulary speech recognition,” in Interspeech, pp. 3707–3711, 2017.
  * [4] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” in ISCA Speech Synthesis Workshop, p. 125, 2016\.
  * [5] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Annual Conference on Neural Information Processing Systems, pp. 3104–3112, 2014.
  * [6] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in 3rd International Conference on Learning Representations, ICLR, 2015.
  * [7] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin, “Convolutional sequence to sequence learning,” in International Conference on Machine Learning, ICML, vol. 70, pp. 1243–1252, 2017.
  * [8] D. Serdyuk, Y. Wang, C. Fuegen, A. Kumar, B. Liu, and Y. Bengio, “Towards end-to-end spoken language understanding,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5754–5758, IEEE, 2018.
  * [9] R. Price, M. Mehrabani, and S. Bangalore, “Improved end-to-end spoken utterance classification with a self-attention acoustic classifier,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8504–8508, IEEE, 2020.
  * [10] P. Haghani, A. Narayanan, M. Bacchiani, G. Chuang, N. Gaur, P. Moreno, R. Prabhavalkar, Z. Qu, and A. Waters, “From audio to semantics: Approaches to end-to-end spoken language understanding,” in IEEE Spoken Language Technology Workshop (SLT), pp. 720–726, 2018.
  * [11] L. Lugosch, M. Ravanelli, P. Ignoto, V. S. Tomar, and Y. Bengio, “Speech model pre-training for end-to-end spoken language understanding,” in Interspeech, pp. 814–818, Sept. 2019.
  * [12] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?,” in Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems, pp. 3320–3328, 2014.
  * [13] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International Conference on Learning Representations, ICLR, 2015.
  * [14] L. Zhou, H. Palangi, L. Zhang, H. Hu, J. Corso, and J. Gao, “Unified vision-language pre-training for image captioning and vqa,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 13041–13049, 2020.
  * [15] A. M. Dai and Q. V. Le, “Semi-supervised sequence learning,” in Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems, pp. 3079–3087, 2015.
  * [16] J. Howard and S. Ruder, “Universal language model fine-tuning for text classification,” in ACL, pp. 328–339, Association for Computational Linguistics, 2018.
  * [17] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” in Technical report, OpenAI, 2018.
  * [18] V. Sanh, T. Wolf, and S. Ruder, “A hierarchical multi-task approach for learning embeddings from semantic tasks,” in AAAI, pp. 6949–6956, 2019\.
  * [19] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in NAACL-HLT, pp. 4171–4186, 2019.
  * [20] G. E. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition,” IEEE Trans. Speech Audio Process., vol. 20, no. 1, pp. 30–42, 2012.
  * [21] J. Kunze, L. Kirsch, I. Kurenkov, A. Krug, J. Johannsmeier, and S. Stober, “Transfer learning for speech recognition on a budget,” in Proc. of the 2nd Workshop on Representation Learning for NLP, Rep4NLP@ACL, pp. 168–177, Association for Computational Linguistics, 2017.
  * [22] P. Denisov and N. T. Vu, “Pretrained semantic speech embeddings for end-to-end spoken language understanding via cross-modal teacher-student learning,” in Proc. Interspeech, pp. 881–885, 2020.
  * [23] B. Sharma, M. C. Madhavi, and H. Li, “Leveraging acoustic and linguistic embeddings from pretrained speech and language models for intent classification,” in Accepted in ICASSP 2021, 2021.
  * [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, pp. 5998–6008, 2017.
  * [25] T. Wolf et al., “Transformers: State-of-the-art natural language processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45, Association for Computational Linguistics, 2020.
  * [26] L. Dong, S. Xu, and B. Xu, “Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition,” in ICASSP, (Calgary, AB), pp. 5884–5888, Apr. 2018.
  * [27] M. Radfar, A. Mouchtaris, and S. Kunzmann, “End-to-end neural transformer based spoken language understanding,” in Interspeech, pp. 866–870, ISCA, 2020.
  * [28] J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,” arXiv:2006.05525 [cs, stat], mar 2021.  arXiv: 2006.05525.
  * [29] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in NIPS Deep Learning and Representation Learning Workshop, 2015\.
  * [30] Y.-S. Chuang, C.-L. Liu, H. yi Lee, and L. shan Lee, “SpeechBERT: An Audio-and-Text Jointly Learned Language Model for End-to-End Spoken Question Answering,” in Proc. Interspeech 2020, pp. 4168–4172, 2020.
  * [31] Y. Huang, H.-K. Kuo, S. Thomas, Z. Kons, K. Audhkhasi, B. Kingsbury, R. Hoory, and M. Picheny, “Leveraging unpaired text data for training end-to-end speech-to-intent systems,” in ICASSP, pp. 5754–5758, 2020.
  * [32] H. Futami, H. Inaguma, S. Ueno, M. Mimura, S. Sakai, and T. Kawahara, “Distilling the knowledge of bert for sequence-to-sequence asr,” arXiv preprint arXiv:2008.03822, 2020.
  * [33] K. Clark, U. Khandelwal, O. Levy, and C. D. Manning, “What does BERT look at? an analysis of bert’s attention,” CoRR, vol. abs/1906.04341, 2019.
  * [34] X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, “TinyBERT: Distilling BERT for natural language understanding,” in Findings of the Association for Computational Linguistics: EMNLP, pp. 4163–4174, 2020.
  * [35] Y. Zhao, J. Li, X. Wang, and Y. Li, “The speech transformer for large-scale mandarin chinese speech recognition,” in ICASSP, pp. 7095–7099, IEEE, 2019.
  * [36] P. J. Price, “Evaluation of spoken language systems: the ATIS domain,” in Speech and Natural Language: Proceedings of a Workshop, Morgan Kaufmann, 1990.
  * [37] B. Sharma and S. R. M. Prasanna, “Sonority measurement using system, source, and suprasegmental information,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 3, pp. 505–518, 2016.
  * [38] B. Sharma and S. R. M. Prasanna, “Speech synthesis in noisy environment by enhancing strength of excitation and formant prominence.,” in INTERSPEECH, pp. 131–135, 2016.