# Multimodal-Boost: Multimodal Medical Image Super-Resolution using Multi-
Attention Network with Wavelet Transform

Fayaz Ali Dharejo, Muhammad Zawish, Farah Deeba, Yuanchun Zhou, Kapal Dev
Sunder Ali Khowaja, and Nawab Muhammad Faseeh Qureshi F.A Dharejo, F. Deeba,
Y. Zhou. Authors are with the Computer Net-work Information Center, Chinese
Academy of Sciences, University of Chinese Academy of Sciences, Beijing,
100190, China. E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
M. Zawish is with Walton Institute for Information and Communication Systems
Science, Waterford Institute of Technology, Ireland, E-mail:
<EMAIL_ADDRESS>K. Dev is with Department of institute of
intelligent systems, University of Johannesburg, South Africa. E-mail:
<EMAIL_ADDRESS>S.A. Khowaja is with Faculty of Engineering and Technology,
University of Sindh, Jamshoro, Pakistan. E-mail<EMAIL_ADDRESS>N.M.F. Qureshi is associated with the Department of Computer Education,
Sungkyunkwan University, Seoul, Korea. E-mail<EMAIL_ADDRESS>(Correspond: Y.
Zhou E-mail: zyc @cnic.cn and N.M.F Qureshi<EMAIL_ADDRESS>Manuscript
received.

###### Abstract

Multimodal medical images are widely used by clinicians and physicians to
analyze and retrieve complementary information from high-resolution images in
a non-invasive manner. Loss of corresponding image resolution adversely
affects the overall performance of medical image interpretation. Deep learning
based single image super resolution (SISR) algorithms has revolutionized the
overall diagnosis framework by continually improving the architectural
components and training strategies associated with convolutional neural
networks (CNN) on low-resolution images. However, existing work lacks in two
ways: i) the SR output produced exhibits poor texture details, and often
produce blurred edges, ii) most of the models have been developed for a single
modality, hence, require modification to adapt to a new one. This work
addresses (i) by proposing generative adversarial network (GAN) with deep
multi-attention modules to learn high-frequency information from low-frequency
data. Existing approaches based on the GAN have yielded good SR results;
however, the texture details of their SR output have been experimentally
confirmed to be deficient for medical images particularly. The integration of
wavelet transform (WT) and GANs in our proposed SR model addresses the
aforementioned limitation concerning textons. While the WT divides the LR
image into multiple frequency bands, the transferred GAN uses multi-attention
and upsample blocks to predict high-frequency components. Additionally, we
present a learning method for training domain-specific classifiers as
perceptual loss functions. Using a combination of multi-attention GAN loss and
a perceptual loss function results in an efficient and reliable performance.
Applying the same model for medical images from diverse modalities is
challenging, our work addresses (ii) by training and performing on several
modalities via transfer learning. Using two medical datasets, we validate our
proposed SR network against existing state-of-the-art approaches and achieve
promising results in terms of SSIM and PSNR.

###### Index Terms:

Super-resolution, attention modules, Multimodality data, Transfer learning,
Wavelet transform, Generative adversarial network

##  1 Introduction

Medical images of high quality are critical in early, fast and accurate
diagnosis in the current clinical processes. Typically, spatial resolution of
medical images is degraded by factors such as imaging modalities and
acquisition time. To recover the distorted resolution, super-resolution (SR)
methods are widely adopted on digital images as post-processors eluding the
additional scanning costs [1]. Image super-resolution is one of the prominent
low-level vision problems in the computer vision domain. The aim of image SR
is to reconstruct a high resolution (HR) image from a corresponding low
resolution (LR) image. SR techniques are applied on both single and multiple
images simultaneously subject to the input and output criteria. In this work,
we study the Single-Image-Super-Resolution (SISR) for the application of
medical images. In contrast to classical SISR applications, medical imaging
SISR is captious as it is often followed by precise tasks of segmentation,
classification or diagnosis [2] [3]. Thus, it becomes imperative to produce
methods which could not only preserve sensitive information but also magnify
the structures of interest efficiently.

Figure 1: A quantitative comparison of the super resolution capabilities of
several state-of-art methods. In terms of PSNR, our proposed technique,
Multimodal-Boost, exceeds the competition. The horizontal axis, which is
measured in logarithmic FLOPs for easier observation and PSNR is represented
on the vertical axis. The amount of parameters is represented by the circular
area. A green circle in the top centered is regarded to be a superior SR
network.

Traditional SR based techniques such as interpolation [4] or reconstruction[5]
based could not suffice for medical image SISR tasks. Interpolation methods
such as linear, bicubic [4] and lancoz [6] often fail to reconstruct the high-
frequency information which eventually results in blurred and smooth edges. In
contrast, reconstruction based algorithms efficiently utilize prior local[7],
global[8], and sparse[9] knowledge to efficiently reconstruct the HR image.
However, these methods can not efficiently simulate the non-linear
transformation from LR space to HR space in dynamic scenes, hence the output
SR image is distorted. Deep learning techniques have recently shown notable
performance on various vision tasks such as image dehazing, object detection,
activity recognition etc [10]. Convolutional Neural Networks (CNNs) and
particularly Generative Adversarial Aetworks (GANs) have shown remarkable
performance on various SR applications such as remote sensing and medical
imaging. Deep Convolutional Neural Network for Super-Resolution (SRCNN)[11]
laid the foundation of deep learning (DL) based SR methods, followed by a
number of techniques involving powerful capabilities of CNNs. Due to
phenomenal performance of residual blocks and dense blocks, several works such
as, Enhanced Deep Residual Networks for Single Image Super-Resolution
(EDSR)[12] Very Deep Convolutional Networks (VDSR)[13],Fast Super-Resolution
with Cascading Residual Network (RFASR)[14] and Deep Back-Projection Networks
for Super-Resolution (DBPN)[15] were proposed for the SR problem. These end to
end models are deeper and complex thus requiring a large number of LR and
corresponding HR pairs to get trained[16]. Moreover, these deep networks could
not provide photo realistic results despite the high performance of Structural
Similarity Index Measure (SSIM) and Peak Signal-to-Noise Ratio (PSNR)[2].
Thus, GAN based networks accompanied by attention mechanisms[17] were
introduced to produce perceptually realistic outputs closer to the ground
truth HR image. However, aforementioned deep learning based SR techniques have
shown sub-optimal performance on medical images particularly for different
types of modalities. Several medical imaging analysis tasks such as tumor
segmentation[18] and lesion detection[19] are essentially hindered by the
substandard HR output with blurred edges. Using a Multi-Temporal Ultra-Dense
Memory Network, Peng Ye et al.[20] proposed a new method based on video super-
resolution. Previous video SR methods mostly use a single memory module and a
single channel structure, which does not fully capture the correlations
between frames specific to video. To achieve video super-resolution, this
paper presents a multi-temporal ultra-dense. Yiqun Mei et al.[21] developed an
efficient SR method based on Cross-Scale Non-Local Attention Module (CS-NL)
incorporating recurrent neural networks to handle an inherent property of
images: cross-scale correlation of features. Afterward, Kui Jiang et al.[22]
developed a hierarchical dense recursive network with super-resolution.
Feature representations were made richer thanks to dense hierarchical
connections. Additionally, it facilitates the lightweight SR model by
providing Reasonable design and parameter sharing.

In this work, we aim to overcome aforementioned problems in multimodal medical
imaging SISR by proposing a combination of multi-attention GAN with wavelet
subbands. Wavelet transform (WT)[23] has a unique ability of extracting useful
multi-scale features with the help of sparse subbands. Since WT accompanied
with GAN has been widely adopted for several applications including remote
sensing for spatio-temporal SR[24] and dehazing[25], it would be interesting
to utilize its potential for the medical imaging SR. In the first step, we
derive the WT based four subbands of the given LR image using a 2D discrete
wavelet transform (DWT) from Haar family of wavelet which eventually replaces
LR input with low dimensional input space preserving the key information.
These subbands are then fed into the proposed GAN involving multiple attention
and upsample blocks which produces an improved HR wavelet component for each
corresponding subband. Lastly, the HR image is reconstructed using 2
dimensional inverse discrete wavelet transform (2D-IDWT). Inspired from
this[26], we use a perceptual loss network along with GAN to further boost the
performance of our proposed method in term of qualitative and qualitative.
Moreover, a challenging and time-consuming task in medical imaging SR is to
modify the models designed for one modality to fit a new modality. We overcome
this by introducing the transfer learning technique which efficiently adapts
to unseen data with the help of pre-learned knowledge. In result, it becomes
feasible to adapt on datasets of different modalities along with the reliable
results. In summary, the following are the contributions of our methodology:

  * •

We proposed a novel wavelet-based multimodal and multi-attention GAN framework
for medical imaging SR called Multimodal-Boost. The Multimodal-Boost can learn
end-to-end residual mapping between low and high resolution wavelet subbands.
The key advantage of utilizing wavelet transform is that it helps restoring
rich content from images by accurately extracting missing details. WT give
high-frequency information in a variety of directions that are most likely to
result from (horizontal, vertical, and diagonal edges).

  * •

We trained the perceptual network on VGG-16 to improve the SR results, as
shown in the Figure 7. Using perceptual loss, we boost network and improve
super-resolution performance. We use a transfer-learning technique to deal
with inadequate training for super-resolution medical applications. Our
algorithm is trained on DIV2K [27] dataset prior to getting evaluated on
multimodal medical datasets.

  * •

Unlike many previous Deep Neural networks (DNN) medical image SR approaches
[28] [29] [30][31], our method employs multimodality data into a single model,
which reduces the cost of adding new SR objectives such as detection and
classification tasks. The results reveal that the proposed method outperforms
existing methods in objective and subjective evaluation as shown in Figure. 1.

The rest of the paper is drawn as follows. Section II provides an overview of
the related works. The proposed The multimodal-Boost approach is detailed in
Section III. Section IV contains the results and discussion. Finally, in
Section V, the conclusion is given.

Figure 2: Multimodal-Boost presents a three-part methodology flow chart:
scatter, predicted, and reconstruct portions. The interpolated version of the
LR image is broken down into four LR wavelet components in the dispersed
section. In the predicting section, a developed Mult-Attention GAN SR network
is used to estimate the HR wavelet component from the LR counterpart. The
third portion is the reconstructed component, which uses 2DIWT to generate
super-resolution images. The two forms of attention blocks are multi-attention
and upsample-attention.

##  2 Literature Work

SR is undoubtedly a classical low-level vision problem which has been highly
studied with emerging deep learning based solutions for a variety of
applications. The problem of SR can be categorised into either single image SR
or multiple image SR, our focus in this paper is on single image SR. Thus, in
this section we review the state-of-the-art SR methods for both general and
medical imaging application.

In the domain of DL based SR, SRCNN by Dong et al[11] was the very first
technique incorporating deep CNNs for the SR task. Following that several
works were proposed with a typical pipeline of feature extraction, non-linear
mapping and HR image reconstruction. However, such deeper level and complex
architectures failed to extract the important multi-level features of a given
LR image. Liang et al[32] identified the importance of edge based features for
enhancing the SR task using sobel edge priors of LR input for training a deep
SR model. Later, VDSR[13], EDSR[12], and RFASR[14] were proposed with the use
of very deep networks involving residual skip connections and residual feature
aggregation techniques to enhance the SR output. However, these methods did
not yield admissible SSIM and PSNR due to deterioration of visual perception
which resulted in over-smoothed SR output. Ledig et al[33] improved the visual
perception in SR problem by proposing a GAN based network namely SRGAN by
combining adversarial and content loss of generated HR output. Moreover, Meta-
SR[34] and DBPN[15] achieved improved results where the former proposed a
solution for high magnification and arbitrary magnification SR and the later
used a deep back-projection network to generate clear HR output.

While all these methods work well on natural datasets, their performance is
not admissible for medical imaging tasks. The methods on pixel space
optimization[11, 12, 14] generate a smooth output which lacks the important
information, while the methods on feature space optimization[33] could not
generate realistic enough outputs. For this reason, these methods are not well
suited for a critical problem like medical imaging SR. Although there exist
several studies to tackle medical imaging SR with the techniques such as cycle
GAN[35], U-Nets[36], 3D convolutions[37] and attention based networks[38],
none of them achieved acceptable clinical ability. Moreover, most of these
works are suitable only for single modality and their performance degrades
when doing inference on new modality dataset. Therefore, in this paper we aim
to address the shortcomings of above works by utilizing the inherent
capabilities of discrete wavelet analysis on embedding the LR input image
which is then provided to multi-attention and upsample blocks of proposed GAN
network to produce the corresponding HR output. It can be seen from results
that our model efficiently utilizes the transfer learning technique to produce
clear HR output on multiple medical imaging modalities.

##  3 METHODOLOGY

###  3.1 SR Based on the DWT

For low-level vision tasks like image denoising and super-resolution, choosing
the proper wavelet can be difficult. Medical diagnostics, in particular,
demands HR images with improved contextual features to provide better patient
care. The WT stores detailed information about an image in many orientations
to make the most of it. At each level of decomposition, the 2D-DWT produces
four subbands that correspond to distinct frequency components and are
referred to as approximate subbands, horizontal, vertical, and diagonal,
containing complete edges information. Each subband contains image data with
distinct characteristics that are important enough to offer precise image
features.

In practice, DWT is being used to send the input signal via the low-pass L(e)
and high-pass filters H(e). The input signal’s approximation and accuracy are
then reduced by 2. $L(e)$ and $H(e)$, are the Haar wavelets’ definitions:

$L(e)=\begin{cases}1,&e=0,1\\\ 0,&otherwise\\\
\end{cases}H(e)=\begin{cases}1,&e=0\\\ -1,&e=0\\\ 0,&otherwise\\\ \end{cases}$
(1)

Figure 3: level-1 decomposition of CT image. 2DWT extracts four subbands
($\textbf{CT}_{LL}$, $\textbf{CT}_{LH}$, $\textbf{CT}_{HL}$, and
$\textbf{CT}_{HH}$) from the input CT modality image, which are subsequently
fed to the Predicted Network for training.

Generally, an image I(x,y), where xth and yth pixel values are in columns. The
average components of the 2D-DWT are represented by four subbands:
approximation, vertical, horizontal, diagonal, LL,LH, HL, HH.

When we use 1-level 2D-DWT to an LL input image to predict the HL, LH, and HH
subbands, we get the miss-ing information features of the LL image, as shown
in the figure. To obtain SR results, we employ 2D-IDWT to acquire the image
missing features information. The coefficients of 2D-IDWT can be computed as
below from the Haar wavelet.

$\begin{cases}A=a+b+c+d\\\ B=a-b+c-d\\\ C=a+b-c-d\\\ B=a-b-c+d\end{cases}$ (2)

where A, B, C, D, and a, b, c, d are the subbands pixel values. Interpolation-
based SR approaches are limited in their capacity to reconstruct information
appropriately. The result is not accurate because high-frequency domain
information cannot be adequately recovered during SR. To improve super-
resolved image performance, edges must be retained. The DWT was employed to
keep the image edges features and texture details in high-frequency; the 2-DWT
decomposed the $I_{L}$ image into the LL, LH, HL, HH subbands. Conventional
DWT-based SR approaches combine DWT with Non-DL SR methods[24] . Let $I_{G}$
denote the image of the ground-truth HR in $m\times n$ by, and $I_{L}$
symbolize LR in size $m\times n$ by, and $I_{L}$ by the same scale factor (s).
Authorize ${I_{LR}}$ to represent $I_{L}$ up-scale ability, with $m\times n$
size, and to display the HR image reconstructed, with m×n size, via bicubic
interpolation.

  1. 1.

The four different subbands are achived form LR input image, as shown in
Figure. 2., i.e.,$\textbf{CT}_{LL}$, $\textbf{CT}_{LH}$, $\textbf{CT}_{HL}$,
$\textbf{CT}_{HH}$ $\textbf{CT}_{LR}$

  2. 2.

Finally using a discrete inverse WT, the HR image of $\textbf{CT}_{HR}$ is
reconstructed (IDWT).

Figure 4: Proposed GAN network with multiple attention and upsample blocks.

###  3.2 Multimodal Muti-Attention GAN

A generator and a discriminator are the two fundamental components of a GAN.
Samples z from a preceding noise distribution $\textbf{p}_{noise}$ are sent
into the generator G. The generator is performed with any model distribution
$\textbf{p}_{model}$ in the data space $\hat{x}=G(z)$. The discriminator is a
binary classifier whose function is to distinguish between real and fake data
on generated samples $x\sim P_{data},D(\hat{x})=1$ and $x\sim
P_{data},D(\hat{x})=0$. Convolutional layers are featured in almost all GAN-
based image restoration models [39][40]. GANs evaluate information in a local
neighborhood while deploying convolutional layers is computationally expensive
for long-range modeling relationships in images. Our proposed approach uses
DWT to generate wavelet subbands to improve spatial resolution (super-
resolution). Self-attention is introduced in to the GAN framework, allowing
the generator and the discriminator to efficiently describe interactions
between widely separated spatial regions, as shown in Figure. 4.

To calculate the attention blocks, the image features from previously hidden
layers $x\in\mathbf{R}^{C\times N}$ are transferred to feature spaces f;g,
where $\textbf{f(x)}=W_{fx};g(x)=W_{g}$

$\beta_{i,j}=\frac{\exp(S_{i,j})}{\sum_{i=1}^{N}\exp(S_{i,j})}={f(x_{i})^{T}}{g(x_{i})}$
(3)

When constructing the jth area, $\beta_{i,j}$ shows how much attention the
model pays to the ith place. Where C is Considering N is the number of
locations of features from the previously hidden layer, the output of the
attention layer can be expressed as, $o$=$(o_{1},o_{2},…,o_{j},…,o_{N})$
$\in\mathbf{R}^{C\times N}$ where,

$O_{j}=\mathcal{V}(\sum_{i=1}^{N}\beta_{i,j}h(x_{i})),h(x_{i})=W_{h}X_{i},\mathcal{V}=W_{h}X_{i}$
(4)

The learned weight matrices $\textbf{W}_{g}$ $\in\mathbf{R}^{L\times C}$,
$\textbf{W}_{f}$ $\in\mathbf{R}^{L\times C}$, $\textbf{W}_{h}$
$\in\mathbf{R}^{L\times C}$, and $\textbf{W}_{v}$ $\in\mathbf{R}^{L\times C}$
are implemented as 1×1 convolutions. We used varied channel numbers of L=C/K,
in all of our experiments, where k=1,2,4,8 thus after a few ImageNet training
epochs, we chose $k=8(i,e.,L=\frac{C}{8})$ and found no significant
performance loss. We also add back the input feature map after multiplying the
output of the attention layer by a scale parameter. As a result, the final
outcome is as follows:

$Y_{i}=\alpha o_{i}+x_{i}$ (5)

Where $\alpha$ is a scalar parameter initially set to 0, allowing the network
to depend on local inputs first and then progressively learn to give non-local
data to more weights. The attention block used in our method is shown in
Figure 5(a). Upsampling attention blocks, which tend to improve the resolution
for CT images on wavelet subbands input features, are another attention block
employed in our predicted part. The spatial resolution of the feature map was
further improved by performing local neighborhood interpolation on the wavelet
subband input features. These are fed to the leaky convolution ReLU, and the
final feature map $(f_{Upsample}$ $\in\mathbf{R}^{L\times C}$ Upsample
attention block can be expressed as;

$F_{map}=Sigmoid(Conv(1\times 1))\times f_{upsample}$ (6)

$F_{map}$ is referred to output of pixel attention module and (Conv(1×1))
represent the convolution operation with kernel size 1×1 so we can say the
$F_{map}$ $\in\mathbf{R}^{W\times L\times 1}$. The values of the feature map
fall between 0 and 1. The feature map can be correctly changed at the
component level to achieve superior SR results. The Upsample attention block
is given in Figure 5(b).

Figure 5: Mechanism of attention blocks, whose goal is to boost network
performance by improving super-resolution, where (a) displays the mechanism of
Self-attention blocks and (b) illustrates the structure of Upsample blocks.
Figure 6: Visual results of our proposed method (Multmodal-Boost) compared to
our State-of-art methods. We enlarge the particular region to notice the
differences between the outcomes more clearly on CT teeth.

###  3.3 The Reconstruction Part

To reconstruct the HR super-resolved image $CT_{HR}$, we use a 2D-IDWT inverse
wavelet onto four components $\bm{CT_{LL}},\bm{CT_{LH}},\bm{CT_{HL}}$, and
$\bm{CT_{HH}}.$

###  3.4 VGG based Perceptual Loss

Perceptual loss is one of the best metrics for measuring image similarity when
CNN algorithms are applied to the input image. It is utilized in various
applications such as image denoising, image dehazing, image translation, and
image super-resolution. Compared to Mean Squared Error (MSE) loss [26], the
perceptual loss is more robust to several problems such as over-flattening and
distortion [41][42]; hence for image SR, the perceptual loss is more powerful
to search spatial resolution similarities between two images. Many CNN
networks employ VGG-loss to measure perceptual loss; however, VGG-11, VGG-16,
and VGG-19 are pre-trained networks that achieved amazing results using
natural image datasets [43]. VGG-loss is a term that can be stated as follows:

$L_{VGG}=\mathbb{E}_{(I_{LR,HR})}[\frac{\|VGG(g(I_{LR}))-VGG(I_{HR})\|_{F}^{2}}{DHW}]$
(7)

where D; H; W denotes the computed tomography (CT) image depth, height, and
width. VGG was trained for image classification using natural mages;
therefore, it generates features not relevant to CT image super-resolution.
This is one of the possible pitfalls with VGG-loss [44]. Due to the scarcity
of labeled CT images, VGG for CT datasets is a challenging task. We proposed a
trained perceptual network to handle this challenge to extract a compressed
encoding from CT data input and reconstruct an image comparable to the
original image. In our case we used VGG-16 in the described perceptual loss
network as shown in Figure 4. The perceptual network comprises six convolution
layers, each with 32, 32, 64, 64, 128, 128 filters. There is also a max
pooling layer with a kernel size of 2 and a stride of 2. The perceptual loss
convolution layers were utilized, followed by the ReLU activation function. In
this design, we used a 33 filter size and a stride of 1. To extract the
features, we trained the perceptual net-work to determine the perceptual loss.
Figure 7 shows the perceptual loss based on VVG-16.

$L_{prec}=\mathbb{E}_{(I_{LR,HR})}[\frac{\|\gamma(g(I_{LR}))-\gamma(I_{HR})\|_{F}^{2}}{DHW}]$
(8)

Where $\gamma$ is the pre-trained encoder network.

Figure 7: VGG-16 Loss curves obtained on natural images. Figure 8: Performance
evaluation of our model with different numbers of multi-attention blocks.

###  3.5 Model Optimization

The proposed multimodal multi-attention model loss is equal to the sum of the
perceptual and generator-discriminator losses, as shown below:

$\min_{g}\max_{d}L_{WGAN}(g,d)+\beta L_{perc}(g)$ (9)

$L_{WGAN}(g,d)$ is a Wasserstein GAN [45][46] optimizer, the weighted
parameters are represented by the $\beta$, which claims to stand for
substitution between WGAN-Loss and perceptual loss, and the letters g and d,
which stand for generator and discriminator, respectively. $L_{perc}(g)$ is
the perceptual loss. $L_{WGAN}(g,d)$ is in addition to the usual Wasserstein
distance and the regularization gradient penalty, which is represented as,

$\displaystyle\min_{g}\max_{d}L_{WGAN}(g,d)=(-\mathbb{E}_{I_{HR}}[d(I_{HR})]+$
$\displaystyle\mathbb{E}_{I_{LR}}[d(g(I_{LR})])+\lambda\mathbb{E}[(\|\bigtriangledown_{\hat{I}}d(\hat{I})\|_{2}-1)^{2}]$

Where $\mathbb{E}_{a}[b]$ is the expression of b as a function of a, $\lambda$
represents the weighted parameter, $\hat{I}$ denotes the generated and real
images in uniform sampling from a range of [0,1], and $\bigtriangledown$ is
the gradient.

Figure 9: Visual results of our proposed method (Multimodal-Boost) compared to
our State-of-art methods. We enlarge the particular region to notice the
differences between the outcomes more clearly on CT knee image. Figure 10:
Visual results of our proposed method (Multmodal-Boost) compared to our State-
of-art methods. We enlarge the particular region to notice the differences
between the outcomes more clearly on MRI-Brain image.

##  4 RESULTS AND DISCUSSIONS

We introduced above a Multimodal-Boost framework for super-resolution tasks in
medical images. It comprises a new neural network for SR medical image
reconstruction, pair-wise attention blocks, and a novel GAN-based loss
function. The proposed model is efficient with texture details and realistic
for viewing reconstructed images to a large extent. Since super-resolution is
an inverse problem with unpredictable solution [12] , the output of the SR
image contains substantially more information than the matching LR image. Each
LR feature from wavelet subbands was regarded as a training sample during
Multimodal-Boost model training and fed to the training phase. Loss functions
are employed to ensure generated SR images as close to HR. Since working with
LR images for diagnosis is too challenging, SR images are highly rated for
medical applications, and they include enough information for radiologists to
make precise conclusions. The information was stored in hidden layers by SR
DNN models, which resulted in high frequency or HR images with superior
texture and edge details. Medical images are too complicated to handle
compared to natural images, making machine learning models relatively
challenging. The higher magnification area in Figure 6, Figure 9, and Figure
10 shows the differences between each approach. At the same time, Meta-SR
achieves excellent performance than other techniques; it still struggles to
generate pretty small textures, whereas our proposed method takes advantage of
wavelet transform and obtains more excellent texture details. We also perform
visual comparisons on other images using our Multimodal-Boost and other SR
techniques, as seen in Figure ”teeth-image” Figure 6. All previous SR
algorithms produce blurry outputs and fail to recover detailed information in
the magnified image, while Meta-SR and our new approach Multimodal-Boost can
only reconstruct a clean image with sharp lines. We evaluate the SR results of
the image MRI Modality to support the multimodality and discover that SRCNN,
EDSR, and Meta-SR all fail to recover the clear edges. It is indeed worth
remembering that the texturing details they acquired are incorrect. CIRCLE-GAN
and SR-ILLNN, on the other hand, can rebuild more compelling results that are
consistent with ground truth, but they fail to provide more precise edge
information. On the other hand, our technique does better when it comes to
restoring edges and texture details.

###  4.1 Training Details

When training SR models with pairs of LR and HR images, downsampling is
typically utilized. On the other hand, medical datasets have fixed in-plane
resolutions of roughly 1mm and lack ultra-high spatial resolutions. As a
result, transfer learning is one of the methods for effectively training
models using external datasets, which has proven to be very useful in remote
sensing and medical applications. We used massive datasets DIV2K [27] with
$2000\times 1400$ pixels of good quality to train our proposed technique to
use transfer learning. We randomly cropped $56\times 56$ sub-pictures from
DIV2K training images for training. Preliminary, we obtain the trained and
optimized network by pre-training the proposed model in a, particularly
defined manner. The pre-trained network is fine-tuned by selecting the
Shenzhen Hospital datasets [47], containing 662 X-ray images used for training
and testing. All images were scaled to $512\times 512$ pixels, and three
modalities, including Montgomery County X-ray, Teeth,, and knee images, were
chosen to test our model. MRI brain scans from the Calgary Campinas repository
were another dataset we used to evaluate our proposed approach. This modality
is produced using a 12-channel head-neck coil on an MR scanner (Discovery
MR750; General Electric Healthcare, Waukesha, WI). All experiments have been
carried out on a Windows-based machine with an Intel(R) Core(TM) i5-7300HQ CPU
running at 3.40GHz and an NVIDIA GeForce GTX 1080-Ti graphics card. The setup
also makes use of MATLAB 2019 with CUDA Toolkit and Anaconda.

TABLE I: DETAILS OF ALL TRAINED MODELS AND DIFFERENT LOSS MEASURES Experiments | Generator Network | Loss Function | WGAN
---|---|---|---
CNN-VGG | CNN | $L_{VGG}$ | no
WGAN | CNN | $L_{WGAN}$ | yes
Perceptual | Multi-attention | $L_{perc}$ | no
WAGN-VGG | CNN |  $L_{WGAN}$ \+ $L_{VGG}$ | yes
WAGN-MA-P | Multi-attention CNN |  $L_{WGAN}$ \+ $L_{perc}$ | yes
TABLE II: NETWORK COMPLEXITY AND INFERENCE SPEED ON SHENZHEN HOSPITAL DATASETS. MULTIMODAL-BOOST, IN COMPARI-SON TO SRDESNSENET AND ESRGAN, TAKES MORE TRAINING TIME WHILE TAKING LESS TIME THAN EDSR AND CIRCLE-GAN WITH TRANSFER LEARNING. FURTHERMORE, UNLIKE EDSR AND META-SR, WHICH ONLY WORK WITH A SINGLE MODALITY, IT USES MULTI-MODALITY DATA IN A SINGLE MODEL, LOWERING THE COST OF ADDING NEW SR TASKS Methods | Number of parameters (M) | Memory size (MB) | Inference time(s) | Number of $log_{10}$ flops
---|---|---|---|---
SRCNN [48] | 0.059 | 13.68 | 0.0113 | 17.4
EDSR [12] | 32.55 | 266.78 | 2.178 | 24.2
Meta-SR [34] | 4.075 | 32.64 | 1.0156 | 25.1
GAN-CIRCLE [49] | 55.88 | 457.88 | 3.0172 | 24.8
SR-ILLNN [50] | 0.441 | 16.44 | 0.0123 | 28.1
Multimodal-Boost | 21.68 | 105.68 | 2.0189 | 28.3
TABLE III: OUR PROPOSED METHOD OUTPERFORMS SRCNN, EDSR, META-SR, CIRCLE-GAN, SR-ILLNN AND BICUBIC INTERPO-LATION ON MULTI-MODAL IMAGES SUCH AS TEETH, KNEE, CT SCAN IMAGES, AND CARDIAC MR BRAIN SCANS IMAGE IN TERMS OF BOTH QUALITATIVE AND QUANTITATIVE OUTCOMES. A GREATER PSNR INDICATES BETTER SR IMAGE RECON-STRUCTION QUALITY, WHILE A HIGHER SSIM INDICATES BETTER PERCEPTUAL QUALITY. BOLD REPRESENTS THE BEST PERFORMANCE. | CT: Modality (Teeth-Image) | CT: Modality (Knee-Image) | MRI: Modality (Brain-Image)
---|---|---|---
State-of-art Methods | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
Bcubic | 26.30 | 0.825 | 25.10 | 0.815 | 25.22 | 0.842
SRCNN [48] | 27.45 | 0.845 | 26.69 | 0.853 | 26.99 | 0.862
EDSR [12] | 29.54 | 0.873 | 28.52 | 0.868 | 28.72 | 0.890
Meta-SR [34] | 31.55 | 0.919 | 32.22 | 0.913 | 33.52 | 0.912
CIRCLE-GAN [49] | 32.44 | 0.902 | 33.33 | 0.921 | 33.93 | 0.929
SR-ILLNN [50] | 34.57 | 0.921 | 35.76 | 0.925 | 35.41 | 0.931
Multimodal-Boost | 36.32 | 0.937 | 36.97 | 0.931 | 37.46 | 0.941

###  4.2 Transfer Learning

Transfer learning is a technique that improves the performance of deep neural
networks by utilizing knowledge learned from natural image data sets as
initial training data. Due to the different distributions of two types of
images, directly applying a trained model using natural datasets to medical
images will not work, so transfer learning is a feasible solution. Transfer
learning has various advantages such as:

  * •

The ability to borrow high-frequency information from natural image datasets,
which boosts the proposed method’s performance to reconstruct the HR image
form LR image.

  * •

It helps in faster model convergence.

  * •

It improves the model accuracy.

To achieve high image resolution for diagnostic, our suggested Multimodal-
Boost model leveraged transfer learning to integrate shared and supplementary
information from diverse modalities. In order to assess each modality and the
detailed information inside the image patches, we used 16 to 128 batch sizes
for the HR patches in model training. When the epochs in both modalities, CT-
Images from Shenzhen Hospital dataset [47] and MRI brain scans from
CalgaryCampinas, reach 180, the training process is finished. We train our
model with ADAM optimizer [51] by setting $\beta_{1}=0.9$, $\beta_{2}=0.999$,
and $\in=10^{-8}$. Our primary goal is to improve multimodal medical image
quality. We trained the model with DIV2K before fine-tuning it with CT-image
dataset, which contains 662 X-ray images. We also tested our model using MRI
brain imaging so that the previous modality information can be used to
reconstruct the next modality image. According to the proposed results,
transfer learning knowledge considerably improved performance as shown in
Figure 12 and Figure 13 . The DIV2K dataset [27] is being used for training
since it is a recently proposed high-quality 2k resolutions image dataset for
image enhancement. There are 800 training images, 100 validation image.

Figure 11: Inference speed on Shenzhen Hospital dataset.

###  4.3 Comparison with State-of-art Methods

We analyzed and compared the proposed method with deep learning models
including SRCNN [48], EDSR[12], Meta-SR [34], CIRCLE-GAN[49], SR-ILLNN [50]
and bicubic interpolation. These methods were designed for natural images
DIV2K [27], which have much bigger dimensions than the medical images we used.
We employed transfer learning and retrained the models with smaller chunks of
medical image datasets to make them work more robustly. We used the same
hardware and experiment conditions to make a fair comparison. We employed the
MSE loss; however, it was not very reliable due to distortion and over-
flattening. As a result, we proposed perceptual loss $L_{perc}(g)$ using
VGG-16 training, and then we improved our network with total loss
$L_{WGAN}(g,d)+\beta L_{perc}(g)$, which is the sum of perceptual loss and
generative discriminator loss function. The proposed model (Multimodal-Boost)
is compared against state-of-the-art approaches in Tables 2, and the
visualization results are shown in Figure 6, Figure 9, and 10.

Figure 12: Model loss over different experimental networks. Figure 13: Network
convergence over different loss functions on Wasserstein Estimation.

For comparison, we conduct experiments on numerous networks with varying
configurations, presented in Table I and Figure 12. We witnessed that WGAN-
based losses converge faster than non-WGAN-based losses. Finally, combining
WGAN and perceptual loss results in a shorter distance and greater
convergence, which we apply to all losses in model optimization, as shows in
Figure 13.

###  4.4 Convergence

The convergence curve of VGG-loss in WGAN-MA-P has the fastest convergence
across all types of approaches, as shown in Figure 12. WGAN-MA-P loss is quite
similar to WGAN- VGG, although the latter achieves lower. However, a reduced
VGG loss does not always imply improved performance. According to our
findings, the VGG-loss network’s spatial resolution diminishes its benefits
while still causing some loss. Finally, we show how the WGAN-MA-P loss
converges very quickly with shorter Wasserstein distances, resulting in state-
of-the-art performance compared to other WGAN-based alternatives. The network
complexity of methodologies is measured in terms of the number of parameters,
memory, inference time, and number of flops as shown in Table 2. Compared to
SRCNN[48], Meta-SR [34], SR-ILLNN [50], Multimodal-Boost has many parameters;
however, it has fewer parameters than EDSR [12] and CIRCLE-GAN [49].
Furthermore, on the same hardware, the proposed method takes 18% longer to
train than Meta-SR. As a result, further work on the architecture, such as
model compression, should be done in the future.

###  4.5 Quantitative Analysis

We evaluated our approach on several attention blocks $(\rho=2,4,8)$ to
investigate how well it performed. In terms of PSNR and SSIM, we found that
the proposed Multimodal-Boost technique outperforms all others. When $\rho$ is
increased, the proposed method performance degrades significantly, as
illustrated in Figure 8. Table 3 shows that the proposed strategy produces the
maximum PSNR. For three images, SRCNN [48] has the lowest PSNR, although it is
better than Bilinear interpolation. The proposed method achieves 1.54–4.89
greater PSNR index when compared with recent methods EDSR[12], Meta-SR[34],
CIRCLE-GAN[49], and SR-ILLNN [50]. The quantitative performance metric is the
peak signal-to-noise ratio (PSNR). The PSNR is defined as follows: Given a
ground-truth image S and its reconstructed image $\hat{S}$, with $M\times N$
pixels size,

$PSNR(S,\hat{S})=10log_{10}\frac{225^{2}}{MSE(S,\hat{S)}}$ (10)

The Structural Similarity Index Measure (SSIM) is a popular tool for assessing
the quality of high-resolution reconstructions. The SSIM index can be
expressed mathematically as,

$SSIM=\frac{(2\mu_{x}\mu_{y}+C_{1})(2\sigma_{xy}+C_{2})}{(\mu_{x}^{2}+\mu_{y}^{2}+C_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+C_{2})}$
(11)

where $\mu_{x}$, $\mu_{y}$ are the average of x and y respectively,
$\sigma_{x}^{2}$, $\sigma_{y}^{2}$ are the variance of x and y respectively,
$\sigma_{xy}$ is the covariance of x and y, $C_{1}$, $C_{2}$ are the
constants.

SISR is a low-level image processing task that can be used in a number of
contexts. In real-world circumstances, SISR’s time constraints are extremely
stringent. We conduct tests to see how long each state-of-the-art method takes
to run and then compare the results on Shenzhen Hospital datasets. This is
depicted in Figure 13.

## Acknowledgments

This work was supported by Natural Science Foundation of China, Grant/Award
Number: 61836013; Beijing Natural Science Foundation, Grant/Award Number:
4212030; Beijing Technology, Grant/Award Number:Z191100001119090; Key Research
Pr gram of Frontier Sciences, CAS, Grant/Award Number:ZDBS‐LY‐DQC016.

## Data Availability

We used the two popular datasets CT-images from Shenzhen Hospital datasets[47]
and Public MRI da-taset (https://sites.google.com/view/calgary-campinas-
dataset/home).

##  5 CONCLUSION

In this study, we designed a multi-attention GAN framework with wavelet
transform and transfer learning for multimodal SISR on medical images. This is
the first time a multi-attention GAN has been employed in conjunction with a
wavelet transform methodology for medical image super-resolution. We also used
transfer leaning, which works with multimodality data that decrease the cost
of adding new SR challenging targets like disease classification. We trained
our network on the high-resolution DIV2K dataset, then applied transfer
learning to train and test medical images on the Shenzhen Hospital datasets of
various modalities. Furthermore, we used GANs to train a perceptual loss
function that can better super-resolve LR features, resulting in improved
perceptual quality of the generated images. Due to the 2D-DWT properties, the
reconstructed images are more accurate and have more texture information. The
utility of transfer learning is that the pre-trained model is fine-tuned using
medical images such as cardiac MR scans and CT scans. In particular, we
evaluated our outcomes in terms of visual, quantitative, adversarial, and
perceptual loss. The PSNR and SSIM measurements are the preliminary step in
evaluating the SR approach, however they are insufficient for real-time
applications. In the future, we will study how the proposed approach affects
target tasks such as disease detection and classification, as well as
minimizing the amount of parameters and flops, which will effectively reduce
training and inference time.

## References

  * [1] Z. Wang, J. Chen, and S. C. Hoi, “Deep learning for image super-resolution: A survey,” _IEEE transactions on pattern analysis and machine intelligence_ , 2020.
  * [2] Y. Li, B. Sixou, and F. Peyrin, “A review of the deep learning methods for medical images super resolution problems,” _IRBM_ , vol. 42, no. 2, pp. 120–133, 2021.
  * [3] F. Deeba, S. Kun, F. A. Dharejo, and Y. Zhou, “Sparse representation based computed tomography images reconstruction by coupled dictionary learning algorithm,” _IET image Processing_ , vol. 14, no. 11, pp. 2365–2375, 2020\.
  * [4] R. Keys, “Cubic convolution interpolation for digital image processing,” _IEEE transactions on acoustics, speech, and signal processing_ , vol. 29, no. 6, pp. 1153–1160, 1981.
  * [5] H. Ji and C. Fermüller, “Wavelet-based super-resolution reconstruction: theory and algorithm,” in _European Conference on Computer Vision_. Springer, 2006, pp. 295–307.
  * [6] C. E. Duchon, “Lanczos filtering in one and two dimensions,” _Journal of Applied Meteorology and Climatology_ , vol. 18, no. 8, pp. 1016–1022, 1979.
  * [7] Y.-W. Tai, S. Liu, M. S. Brown, and S. Lin, “Super resolution using edge prior and single image detail synthesis,” in _2010 IEEE computer society conference on computer vision and pattern recognition_. IEEE, 2010, pp. 2400–2407.
  * [8] C.-Y. Yang, J.-B. Huang, and M.-H. Yang, “Exploiting self-similarities for single frame super-resolution,” in _Asian conference on computer vision_. Springer, 2010, pp. 497–510.
  * [9] W. Dong, F. Fu, G. Shi, X. Cao, J. Wu, G. Li, and X. Li, “Hyperspectral image super-resolution via non-negative structured sparse representation,” _IEEE Transactions on Image Processing_ , vol. 25, no. 5, pp. 2337–2352, 2016\.
  * [10] F. Deeba, F. A. Dharejo, M. Zawish, F. H. Memon, K. Dev, R. A. Naqvi, Y. Zhou, and Y. Du, “A novel image dehazing framework for robust vision-based intelligent systems,” _International Journal of Intelligent Systems_.
  * [11] C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in _European conference on computer vision_. Springer, 2014, pp. 184–199.
  * [12] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 136–144.
  * [13] J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1646–1654.
  * [14] N. Ahn, B. Kang, and K.-A. Sohn, “Fast, accurate, and lightweight super-resolution with cascading residual network,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 252–268.
  * [15] M. Haris, G. Shakhnarovich, and N. Ukita, “Deep back-projection networks for super-resolution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 1664–1673.
  * [16] Z. Wang, J. Chen, and S. C. Hoi, “Deep learning for image super-resolution: A survey,” _IEEE transactions on pattern analysis and machine intelligence_ , 2020.
  * [17] M. Xu, Z. Wang, J. Zhu, X. Jia, and S. Jia, “Multi-attention generative adversarial network for remote sensing image super-resolution,” _arXiv preprint arXiv:2107.06536_ , 2021.
  * [18] M. Zawish, A. A. Siyal, K. Ahmed, A. Khalil, and S. Memon, “Brain tumor segmentation in mri images using chan-vese technique in matlab,” in _2018 International Conference on Computing, Electronic and Electrical Engineering (ICE Cube)_. IEEE, 2018, pp. 1–6.
  * [19] J. Zhu, G. Yang, and P. Lio, “Lesion focused super-resolution,” in _Medical Imaging 2019: Image Processing_ , vol. 10949. International Society for Optics and Photonics, 2019, p. 109491L.
  * [20] P. Yi, Z. Wang, K. Jiang, Z. Shao, and J. Ma, “Multi-temporal ultra dense memory network for video super-resolution,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 8, pp. 2503–2516, 2019\.
  * [21] Y. Mei, Y. Fan, Y. Zhou, L. Huang, T. S. Huang, and H. Shi, “Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 5690–5699.
  * [22] K. Jiang, Z. Wang, P. Yi, and J. Jiang, “Hierarchical dense recursive network for image super-resolution,” _Pattern Recognition_ , vol. 107, p. 107475, 2020.
  * [23] A. Abbate, J. Frankel, and P. Das, “Wavelet transform signal processing for dispersion analysis of ultrasonic signals,” in _1995 IEEE Ultrasonics Symposium. Proceedings. An International Symposium_ , vol. 1. IEEE, 1995, pp. 751–755.
  * [24] F. A. Dharejo, F. Deeba, Y. Zhou, B. Das, M. A. Jatoi, M. Zawish, Y. Du, and X. Wang, “Twist-gan: Towards wavelet transform and transferred gan for spatio-temporal single image super resolution,” _arXiv preprint arXiv:2104.10268_ , 2021.
  * [25] M. Fu, H. Liu, Y. Yu, J. Chen, and K. Wang, “Dw-gan: A discrete wavelet transform gan for nonhomogeneous dehazing,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 203–212.
  * [26] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _European conference on computer vision_. Springer, 2016, pp. 694–711.
  * [27] R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, and L. Zhang, “Ntire 2017 challenge on single image super-resolution: Methods and results,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 114–125.
  * [28] Y. Zhang and M. An, “Deep learning-and transfer learning-based super resolution reconstruction from single medical image,” _Journal of healthcare engineering_ , vol. 2017, 2017.
  * [29] Z. Chen, X. Guo, P. Y. Woo, and Y. Yuan, “Super-resolution enhanced medical image diagnosis with sample affinity interaction,” _IEEE Transactions on Medical Imaging_ , vol. 40, no. 5, pp. 1377–1389, 2021.
  * [30] F. Deeba, S. Kun, F. A. Dharejo, and Y. Zhou, “Wavelet-based enhanced medical image super resolution,” _IEEE Access_ , vol. 8, pp. 37 035–37 044, 2020\.
  * [31] Y. Romano, J. Isidoro, and P. Milanfar, “Raisr: rapid and accurate image super resolution,” _IEEE Transactions on Computational Imaging_ , vol. 3, no. 1, pp. 110–125, 2016.
  * [32] Y. Liang, J. Wang, S. Zhou, Y. Gong, and N. Zheng, “Incorporating image priors with deep convolutional neural networks for image super-resolution,” _Neurocomputing_ , vol. 194, pp. 340–347, 2016.
  * [33] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang _et al._ , “Photo-realistic single image super-resolution using a generative adversarial network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4681–4690.
  * [34] X. Hu, H. Mu, X. Zhang, Z. Wang, T. Tan, and J. Sun, “Meta-sr: A magnification-arbitrary network for super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1575–1584.
  * [35] H. Liu, J. Liu, S. Hou, T. Tao, and J. Han, “Perception consistency ultrasound image super-resolution via self-supervised cyclegan,” _Neural Computing and Applications_ , pp. 1–11, 2021.
  * [36] J. Park, D. Hwang, K. Y. Kim, S. K. Kang, Y. K. Kim, and J. S. Lee, “Computed tomography super-resolution using deep convolutional neural network,” _Physics in Medicine & Biology_, vol. 63, no. 14, p. 145011, 2018.
  * [37] Y. Chen, F. Shi, A. G. Christodoulou, Y. Xie, Z. Zhou, and D. Li, “Efficient and accurate mri super-resolution using a generative adversarial network and 3d multi-level densely connected network,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2018, pp. 91–99.
  * [38] X. He, Y. Lei, Y. Fu, H. Mao, W. J. Curran, T. Liu, and X. Yang, “Super-resolution magnetic resonance imaging reconstruction using deep attention networks,” in _Medical Imaging 2020: Image Processing_ , vol. 11313\. International Society for Optics and Photonics, 2020, p. 113132J.
  * [39] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,” _Advances in neural information processing systems_ , vol. 29, pp. 2234–2242, 2016.
  * [40] A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, “Generative adversarial networks: An overview,” _IEEE Signal Processing Magazine_ , vol. 35, no. 1, pp. 53–65, 2018.
  * [41] H. Shan, Y. Zhang, Q. Yang, U. Kruger, M. K. Kalra, L. Sun, W. Cong, and G. Wang, “3-d convolutional encoder-decoder network for low-dose ct via transfer learning from a 2-d trained network,” _IEEE transactions on medical imaging_ , vol. 37, no. 6, pp. 1522–1534, 2018.
  * [42] Z. Chen and Y. Tong, “Face super-resolution through wasserstein gans,” _arXiv preprint arXiv:1705.02438_ , 2017.
  * [43] Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han, “Amc: Automl for model compression and acceleration on mobile devices,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 784–800.
  * [44] Y. Jo, S. Yang, and S. J. Kim, “Investigating loss functions for extreme super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 424–425.
  * [45] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in _International conference on machine learning_. PMLR, 2017, pp. 214–223.
  * [46] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134.
  * [47] S. Jaeger, S. Candemir, S. Antani, Y.-X. J. Wáng, P.-X. Lu, and G. Thoma, “Two public chest x-ray datasets for computer-aided screening of pulmonary diseases,” _Quantitative imaging in medicine and surgery_ , vol. 4, no. 6, p. 475, 2014.
  * [48] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 38, no. 2, pp. 295–307, 2015.
  * [49] C. You, G. Li, Y. Zhang, X. Zhang, H. Shan, M. Li, S. Ju, Z. Zhao, Z. Zhang, W. Cong _et al._ , “Ct super-resolution gan constrained by the identical, residual, and cycle learning ensemble (gan-circle),” _IEEE transactions on medical imaging_ , vol. 39, no. 1, pp. 188–203, 2019.
  * [50] S. Kim, D. Jun, B.-G. Kim, H. Lee, and E. Rhee, “Single image super-resolution method using cnn-based lightweight neural networks,” _Applied Sciences_ , vol. 11, no. 3, p. 1092, 2021.
  * [51] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.

|  Fayaz Ali Dharejo https://orcid.org/0000-0001-7685-391 (ORCID ID). He
received B.E. degree in Electronic Engineering from Quaid-e-Awam University of
Engineering Science and Technology (QUEST) Pakistan and M.S. degrees in School
of Information and Software Engineering from University of Electronic Science
and Technology of China (UESTC) in 2016 and 2018, respectively. Since
September 2018, he is Pursuing PhD in the Institute of Computer Network
Information Center Chinese Academy of Sciences, University of Chinese Academy
of Sciences.. He has published more than 30 articles within few time in
reputed ISI impact factor journals and in IEEE conferences, among them the top
journals ACM Transactions on Intelligent Systems and Technology, International
Journal of Intelligent Systems, IEEEE GRSL and etc. He is a member of
professional bodies such as Pakistan Engineering Council (PEC), ACM, IEEE
(USA), IEEE Societies, such as IEEE Geoscience and Remote Sensing Society,
IEEE Computer Society and IEEE Signal Processing Society.He is a member of the
Technical Program Committee and Reviewer of several Scopus, Springer and ISI
(Thomson Reuters) indexed conferences. His research interests are image
processing, deep learning, remote sensing and machine learning.
---|---
|  Muhammad Zawish has done BE in computer systems from Mehran University of
Engineering and Technology, Pakistan. He has worked as Lab engineer in
Department of Software Engineering at PAFKIET, Pakistan. Currently, he is
pursuing his PhD in computer science at Walton Institute, Waterford Institute
of Technology, Ireland. His research interests are deep learning for IoT.
---|---
|  Yuanchun Zhou Yuanchun Zhou is a Professor in the Computer Network Infor-
mation Center, Chinese Academy of Sciences. He received his PhD from the
Institute of Computing Technology, Chinese Academy of Sciences, in 2006. His
main research interests in-clude data mining, data intelligence, massive data
and data intensive computing. He has published more than 100 research
publications.
---|---
|  Farah Deeba https://orcid.org/0000-0002-1987-2402. She received B.E. and
M.E. degrees in Department of Software Engineering from Mehran University of
Engineering and Technology Jamshoro Pakistan in 2010 and 2014, respectively.
She has 3 years of experience in teaching at graduate and post-graduate level
at Hamdard University Karachi Pakistan. Current-ly, she has completed her
Ph.D. degree from the School of Information and Software Engineering,
University of Electronic Science and Technology, China in July 2020. Since
September 2020 she is also doing Post-doc at Computer network information
Center, Chinese Academy of Sciences Beijing, China. She has served as a
reviewer for many ISI (Impact factor) and Scopus indexed journals, Such as
IEEE Transactions on Pattern Analysis and Machine Intelligence, Electronics
Letters, IET Image Processing, IEEE Access, etc.She is a member of
professional bodies such as Pakistan Engineering Council (PEC), ACM, IEEE
(USA), IEEE Societies, such as IEEE Geoscience and Remote Sensing Society,
IEEE Computer Society and IEEE Signal Processing Society. Her research
interest includes super-resolution, sparse representation and deep learning.
---|---
|  Kapal Dev ORCID ID: 0000-0003-1262-8594 He is Senior Research Associate at
university of Johannesburg South Africa. Previously, he was a Postdoctoral
Research Fellow with the CONNECT Centre, School of Computer Science and
Statistics, Trinity College Dublin (TCD). He worked as 5G Junior Consultant
and Engineer at Altran Italia S.p.A, Milan, Italy on 5G use cases. He worked
as Lecturer at Indus university, Karachi back in 2014. He is also working for
OCEANS Network as Head of Projects funded by European Commission. He was
awarded the PhD degree by Politecnico di Milano, Italy under the prestigious
fellowship of Erasmus Mundus funded by European Commission. His research
interests include Blockchain, 6G Networks and Artificial Intelligence. He is
very active in leading (as Principle Investigator) Erasmus + International
Credit Mobility (ICM), Capacity Building for Higher Education, and H2020 Co-
Fund projects. He is also serving as Associate Editor in Springer Wireless
Networks, IET Quantum Communication, IET Networks, Topic Editor in MDPI
Network, and Review Editor in Frontiers in Communications and Networks. He is
also serving as Guest Editor (GE) in several Q1 journals; IEEE TII, IEEE TNSE,
IEEE TGCN, Elsevier COMCOM and COMNET. He served as Lead chair in one of IEEE
CCNC 2021, IEEE Globecom 2021 and ACM MobiCom 2021 workshops, TPC member of
IEEE BCA 2020 in conjunction with AICCSA 2020, ICBC 2021, SSCt 2021, DICG Co-
located with Middleware 2020 and FTNCT 2020. He is expert evaluator of MSCA
Co-Fund schemes, Elsevier Book proposals and top scientific journals and
conferences including IEEE TII, IEEE TITS, IEEE TNSE, IEEE IoT, IEEE JBHI, and
elsevier FGCS and COMNET.
---|---
|  Dr. Sunder Ali Khowaja ORCID ID: 0000-0002-4586-4131 He received the Ph.D.
degree in Industrial and Information Systems Engineering from Hankuk
University of Foreign Studies, South Korea. He is currently an Assistant
Professor at Department of Telecommunication Engineering, Faculty of
Engineering and Technology, University of Sindh, Pakistan. He had the
experience of working with multi-national companies as Network and RF Engineer
from 2008-2011. He is having teaching, research and administrative experience
of more than 10 years. He has published over 30 research articles in national,
international journals, and conference proceedings. He is also a regular
reviewer of notable journals including IEEE Transactions on Industrial
Informatics, IEEE Internet of Things Journal, IEEE Access, Electronic Letters,
IET Image Processing, IET Signal Processing, Computer Communications,
International Journal of Imaging Systems and Technology, Multimedia Systems,
IET Networks, SN Applied Sciences, Artificial Intelligence Review,
Computational and Mathematical in Medicine, Computers in Human Behavior, IET
Wireless Sensor Systems, Journal of Information Processing Systems, and
Mathematical Problems in Engineering. He has also served as a TPC member for
workshops in A* conferences such as Globecom and Mobicom. His research
interests include Data Analytics, Process Mining, and Deep Learning for
Computer Vision applications and Emerging Communication Technologies.
---|---
|  Dr. Nawab Muhammad Faseeh Qureshi ORCID ID: 0000-0002-5035-2640 He is an
Assistant Professor at Sungkyunkwan University, Seoul, South Korea. He
received Ph. D. in Computer Engineering from Sungkyunkwan University, South
Korea, through a SAMSUNG scholarship. He was awarded the 1st Superior Research
Award from the College of Information and Communication Engineering based on
his research contributions and performance during his studies. He has served
as Guest Editor in more than 17 Journals. Also, he has served as General Chair
Workshop in many conferences that includes IEEE Wireless Communications and
Networking Conference (WCNC2020) and IEEE Globecom 2020. In addition to that,
he has served as Proceedings Chair in Global Conference on Wireless and
Optical Technologies consecutively for the third time. He is a reviewer of
various prestigious IEEE and ACM Journals. He serves in many leading IEEE and
ACM conferences as a reviewer every year. He has evaluated several theses as
external Ph.D. thesis evaluators. Also, he has facilitated several institutes
and conferences with keynotes and Webinars on Big data analysis and Modern
Technology convergence. He is an active Senior Member of IEEE, ACM, KSII
(Korean Society for Internet Information), and IEICE (Institute of
Electronics, Information and Communication Engineers). His research interests
include big data analytics, machine learning, deep learning, context-aware
data processing of the Internet of Things, and cloud computing.
---|---


1Chair of Embedded Intelligence for Health Care and Wellbeing, University of
Augsburg, Germany
2audEERING GmbH, Gilching, Germany
3GLAM – Group on Language, Audio, & Music, Imperial College, UK

# Abusive Speech Detection in Indic Languages Using Acoustic Features

###### Abstract

Abusive content in online social networks is a well-known problem that can
cause serious psychological harm and incite hatred. The ability to upload
audio data increases the importance of developing methods to detect abusive
content in speech recordings. However, simply transferring the mechanisms from
written abuse detection would ignore relevant information such as emotion and
tone. In addition, many current algorithms require training in the specific
language for which they are being used. This paper proposes to use acoustic
and prosodic features to classify abusive content. We used the ADIMA data set,
which contains recordings from ten Indic languages, and trained different
models in multilingual and cross-lingual settings. Our results show that it is
possible to classify abusive and non-abusive content using only acoustic and
prosodic features. The most important and influential features are discussed.

Index Terms: abusive content, computational paralinguistics, speech prosody

##  1 Introduction

With the rise of the internet and the establishment of more and more online
social networking platforms, the amount of abusive content is rapidly
increasing. The fact that almost every online social network provides
information on how to prevent bullying, how to deal with abusive behaviour,
and even statistics on the prevalence of harassment on its website underlines
the growing importance of this issue. According to [1, p. 274 ], abusive
behaviour ``describes a broad category of content that includes hate speech,
profanity, threats, cyberbully and various ethnic and racial slurs''. It is
often conveyed through rhetorical devices such as aggressive language,
threats, obscenity, and sarcasm [2].

In addition to hindering social interaction and fomenting hatred, research has
shown that victims of cyberbullying ``have psychosomatic problems …, have high
levels of perceived difficulties, have emotional and peer problems, and feel
unsafe at school and uncared about by teachers'' [3, p. 727]. These findings
increase the need to identify abusive content in online social networks.
However, [4] found out that, to date, much of the research in this area has
focused on the English language and on detecting hate speech in texts or
memes. Their paper shows that most studies have been conducted in North
America (44%) and the United Kingdom (11%) and have used data from Twitter
(55%) and Facebook (35%). This imbalance likely exists because the majority of
data sets come from these platforms.

As almost all social networking platforms now allow the upload of video and
audio files, online communication and interactions are no longer limited to
written content. In recent years, several efforts have therefore been made to
classify abusive speech in audio recordings. Different approaches have been
used for feature extraction and classification. For example, [5] used a
speech-to-text converter and sentiment analysis to obtain the polarity of
spoken utterances in audio recordings extracted from YouTube videos, [6], who
created DeToxy, one of the few available data sets of abusive audio
recordings, used automatic speech recognition (ASR) for their baseline
estimation, and [7] used keyword spotting to suppress predefined profane
words.

While some information about abusive content in audio data can be retrieved
using ASR and textual search or keyword spotting, this ignores the information
provided by emotional and acoustic cues [8]. These cues can be important, for
example for phrases that may or may not be considered abusive depending on the
tone and context. For this reason, the change of modality triggers the need
for recognition methods that are specifically designed for audio and video
data [1, 9, 10, 11].

[12] combined the features extracted by ASR with contextual and emotional
representation features. While they achieved F1 scores between 0.77 and 0.85
for the different languages in the ADIMA data set, the features they used
contained only spectral and temporal information, and not prosodic or voice
quality features, which have been shown to correlate with abusive speech. For
example, [13] and [14] found that it is possible to detect rude words using
only acoustic features. Although the performance of their textual model was
better, this could be because they used acted data instead of real-life
recordings, where the text was explicitly designed to be abusive. Furthermore,
their findings might be relativised if the conversion of audio to text is
complicated, resulting in less accurate textual representations. This may be
the case, among other things, due to slurred speech and poor audio quality,
such as background noise [8].

A popular method for obtaining acoustic features is using spectrogram images
of the audio recordings. So far, this has been done mainly for Indic
languages, once by [15], who tried to distinguish between abusive and non-
abusive content in Bengali speech recordings, and then by [16] for baseline
calculations on their newly introduced ADIMA data set, which contains abusive
and non-abusive audio recordings for ten different Indic languages. [16]
additionally used Wav2Vec2 models for speech recognition. Both papers applied
a variety of classification methods and achieved good overall results. In
addition, [16] found that performance in some languages was higher when the
model was trained in another language. In particular, training in all
languages improved performance in most single languages. If this could be
generalised to other languages, it would be a major advantage for abusive
content classification in audio data. However, while they hypothesise that
``models are able to leverage audio properties like pitch, emotions, intensity
etc. for this task instead of relying on the actual word'' [16, p. 6175], this
could also be because the Wav2Vec2 models they used for speech recognition
were pre-trained on the target languages. Therefore, as seen in [17], these
are not true zero-shot trials and further investigation is needed, as large
pre-trained models can exploit linguistic information, which may partially
explain their higher performance.

This paper investigates how paralinguistic features alone, i. e., acoustic and
prosodic properties, perform for the detection of abusive content in real-life
data. This is particularly interesting because real-life recordings usually
contain a lot of information about the speaker's emotional state that is not
captured by the text. We also look at their performance in multilingual and
cross-lingual settings, as non-textual features could provide an easy way to
perform abusive content detection in less common languages for which no audio
data sets are available.

The remainder of this paper is organised as follows: Section 2 describes the
methodology used to distinguish which features differ significantly between
abusive and non-abusive content, and how the classification was performed.
Section 3 shows the results, which are followed by the discussion in Section 4
and the conclusion in Section 5.

##  2 Methodology

The ADIMA data set [16] was used for this study. This data set consists of
11,775 audio recordings, evenly distributed across ten different Indic
languages (Bengali, Bhojpuri, Gujarati, Haryanvi, Hindi, Kannada, Malayalam,
Odia, Punjabi, and Tamil), and taken from real-life conversations in ShareChat
chatrooms. The authors defined the content as abusive if swear words, cuss
words, or abusive words/phrases were present, resulting in 5,108 abusive and
6,667 non-abusive recordings. The average duration of the recordings is 20
($\pm$ 3) seconds with a minimum of 5 seconds and a maximum of 58 seconds. For
each language, the authors split the data set into a training and a test set
(70:30).

The acoustic and prosodic features of each audio recording were extracted
using the openSMILE-v2.4.1 toolkit [18]. Two different feature sets were
extracted: 1. The extended Geneva Minimalistic Acoustic Parameter Set
(eGeMAPS) [19]. This parameter set consists of 88 acoustic signal descriptors
including 18 low-level descriptors (e. g., pitch, jitter, formants, shimmer,
loudness) to which different functionals were applied (e. g., arithmetic mean,
standard deviation, percentiles), 6 temporal features (e. g., mean length of
voiced and unvoiced regions), and 7 cepstral parameters (e. g., MFCC, spectral
flux); 2. The Computational Paralinguistics Challenge (ComParE_2016) [20]
parameter set, which contains 6,373 features.

In the first step, the performances of four different classifiers – Logistic
Regression (LR), eXtreme Gradient Boosting (XGBoost), Support Vector Machine
(SVM), and Random Forest (RF) – on the two feature sets were measured. For
each feature set and each of these classifiers, 21 different models were
trained. The first ten models were trained on one language only and then
tested on each of the ten languages separately (single condition). These
models were also tested on a combined test set containing all languages except
the one on which the model was trained (multi-test condition). The next ten
models were trained on all languages except one and then tested on the
remaining language (multi-train condition). The last model was trained and
tested on all languages (all condition). This process was repeated five times
and the mean performances were used for evaluation.

Further hyperparameter optimisation did not yield any additional performance
gains. Therefore, we only report results for the default parameters (e.g., for
RF: n_estimators = 100, max_depth = None, min_samples_split = 2). For LR, SVM,
and RF we used sklearn-v1.2.0 and for XGBoost xgboost-v1.7.3 in
Python-v3.10.06.

Model performance was determined using the unweighted average recall (UAR)
score, which compensates for imbalances in the sample class ratio by summing
up the recall of all classes and dividing this by the number of classes, and
the F1 score. The means of the multi-test and multi-train models were used to
compare the classifiers. The best-performing feature set and classifier were
selected for subsequent analysis.

These subsequent analyses included examining the most influential parameters
for the model performance using SHapley Additive exPlanations (SHAP). In
addition, Mann-Whitney U (MWU) tests were used to examine differences in the
value distributions between abusive and non-abusive recordings for each
feature in each language. A difference was called meaningful if two conditions
were met: 1. The MWU test result had to be significant using an alpha level of
0.05 and correcting for multiple testing within the language using the
Bonferroni-Holm method; 2. The Common-Language Effect Size (CLES), which
indicates ``the probability that a score sampled at random from the first
population will be greater than a score sampled at random from the second''
[21, p. 101], had to be greater than 67.2%, which corresponds to a medium
effect size. If a feature was meaningful in every language, it was considered
important for the differentiation between abusive and non-abusive audio
recordings. The findings from the MWU tests were compared with the SHAP
results. The comparison as well as the interpretation of the results can be
found in Section 4.

##  3 Results

The mean of the UAR and F1 scores for the two feature sets and four
classifiers can be found in Table 1. In general, it can be said that the
classifiers performed similarly well on the eGeMAPS feature set and on the
ComParE_2016 feature set. Due to its size advantage, it was decided to use the
eGeMAPS feature set for all subsequent analyses in this paper. As for the
classifiers, all models seem to perform comparably well in the multi-train
condition, while in the multi-test condition, the RF classifier performed
slightly better than the others. For this reason, the RF classifier was used
to determine all other results. The result for the all condition can also be
found in Table 1, while all other results for the RF classifier, including
multilingual and cross-lingual settings, can be found in Fig. 1. To
investigate which features are most influential in the classification a SHAP
graph was created using a model trained on all languages (see Fig. 2).

Table 1: For both feature sets, the means of the UAR and F1 scores are shown for all classifiers using the models trained on all but one language and tested on the last (multi-train condition) and the models trained on one language and tested on all others (multi-test condition). In addition, the score for training and testing on all languages (all condition) is shown for the RF classifier. |  | eGeMAPS | ComParE
---|---|---|---
Classifier | Condition | UAR | F1 | UAR | F1
LR | multi-train | 0.77 | 0.72 | 0.68 | 0.63
LR | multi-test | 0.74 | 0.67 | 0.72 | 0.66
XGBoost | multi-train | 0.76 | 0.72 | 0.77 | 0.74
XGBoost | multi-test | 0.75 | 0.67 | 0.75 | 0.68
SVM | multi-train | 0.77 | 0.73 | 0.67 | 0.62
SVM | multi-test | 0.74 | 0.67 | 0.71 | 0.65
RF | multi-train | 0.77 | 0.73 | 0.77 | 0.73
RF | multi-test | 0.76 | 0.68 | 0.76 | 0.68
RF | all | 0.79 | 0.76 | 0.80 | 0.76
Figure 1: Heatmap of UAR scores for the RF classifier trained on each of the
ten languages in the ADIMA data set using the eGeMAPS features. The rows
indicate the training language, while the columns indicate the test language.
The top row shows the performance when training on all languages except the
test language; the rightmost column shows the performance when testing on all
languages except the training language. Figure 2: SHAP graph for the RF
classifier trained on all ten languages in the ADIMA data set using the
eGeMAPS features. The values indicate the influence of the features on the
model. Note: LU = Loudness

Between 19 and 25 features were found to be meaningful in each language as a
result of the MWU tests. A total of 18 features were considered important, i.
e., meaningful for all ten languages – Loudness ($\mu$), Loudness (50%),
Loudness (80%), Loudness (pctl Range 0-2), Rising Slope Loudness ($\mu$),
Rising Slope Loudness ($\sigma$), Falling Slope Loudness ($\mu$), Falling
Slope Loudness ($\sigma$), F1 Amplitude ($\mu$), F2 Amplitude ($\mu$), F3
Amplitude ($\mu$), Flux ($\mu$), Flux V ($\mu$), Flux UV ($\mu$), Loudness
(Peaks Per Sec), Voiced Segments Per Second, Voiced Segment Length Per Sec
($\mu$), $\bar{E}_{rms}$. Since many of these are highly correlated, only a
selection of the results is shown. Table 2 contains the mean, median, and the
$1^{st}$ and $3^{rd}$ quartiles for these features, calculated on the whole
data set and split by the label. It also shows the mean of the CLES for the
MWU tests across all languages.

Table 2: Selection of features, that differed significantly between abusive and non-abusive content in all languages according to MWU test (Bonferroni-Holms corrected p-value <0.05 and CLES >67.2%), with respective means, $1^{st}$ quartile, $2^{nd}$ quartile (median), $3^{rd}$ quartile, and mean CLES. | non-abusive | abusive |
---|---|---|---
Features | M | $Q_{1}$ | Median | $Q_{3}$ | M | $Q_{1}$ | Median | $Q_{3}$ | CLES
Loudness ($\mu$) | 0.41 | 0.07 | 0.23 | 0.54 | 1.37 | 0.50 | 1.00 | 2.03 | 0.82
F1 Amplitude ($\mu$) | -161.61 | -193.20 | -168.86 | -139.51 | -109.48 | -141.90 | -110.56 | -76.12 | 0.82
F2 Amplitude ($\mu$) | -157.40 | -190.91 | -163.91 | -133.59 | -103.73 | -136.40 | -103.50 | -69.98 | 0.82
F3 Amplitude ($\mu$) | -158.50 | -190.81 | -164.67 | -135.82 | -106.34 | -138.47 | -105.90 | -73.20 | 0.82
Flux ($\mu$) | 0.25 | 0.02 | 0.12 | 0.31 | 0.94 | 0.28 | 0.63 | 1.41 | 0.82
Voiced Segments per Second | 1.16 | 0.40 | 1.00 | 1.72 | 2.13 | 1.37 | 2.08 | 2.80 | 0.76

##  4 Discussion

In this paper, we tried to investigate whether paralinguistic, i. e., acoustic
and prosodic, features alone can be used to detect abusive content in real-
life audio recordings. We also wanted to find out how these features perform
in real multilingual and cross-lingual settings. The results show that it is
indeed possible to use paralinguistic features to classify abusive and non-
abusive content. For the RF classifier, the achieved UAR values were between
0.70 and 0.84 for multilingual settings and between 0.66 and 0.84 for cross-
lingual settings. These results are comparable to those of related studies.

Comparing the eGeMAPS and ComParE_2016 feature sets, the results indicate that
the eGeMAPS feature set performs comparably well when used for abusive content
detection in audio data. In addition, it has the advantage of better
interpretability and shorter run time due to its smaller size. Regarding the
classifiers, the RF classifier performs best on the given data set and using
the extracted paralinguistic features. This finding is consistent with a study
by [22], which attempted to detect hate speech in audio data, and in contrast
to a study by [23], which found that LR performed best in detecting
cyberbullying in audio and visual data. One explanation is the difference in
the used features, as RF can model non-linear dependencies, unlike LR. Another
problem may be that the features in the eGeMAPS feature set are highly
correlated, which is a problem in LR and can lead to unreliable coefficient
estimates.

As shown in Fig. 1, the multilingual models performed best when tested on
Odia, Haryanvi, and Punjabi and worst on Bhojpuri. This was also true for most
cross-lingual models. These results were present in all classifiers and can be
found similarly in [16]. [12], on the other hand, obtained different results,
with Bhojpuri performing very well and Odia performing worst. This difference
may be due to the different properties of the languages, as [24] shows that
there are differences in the predictive properties between different
languages.

The results of the MWU tests and the most influential features of the model,
as seen in the SHAP graph, show distinct similarities. Both indicate that the
most important features in determining whether a recording is abusive or non-
abusive are the mean amplitudes of F1, F2, and F3, the loudness features, and
the mean spectral flux – with higher values indicating abusive language. This
is reminiscent of results from the anger literature, which found higher mean
volume, higher volume range, and higher energy density [25], as well as higher
F1, F2, and F3 [26, 27, 24], as significant predictors. In addition, the SHAP
plot suggests that abusive content is associated with lower standard deviation
in F1, F2, and F3 amplitudes. In contrast to other studies, we did not find a
difference in pitch [28, 24]. This is consistent with preliminary evidence for
lower or absent pitch variation in (acted) emotional speech in Indian
languages [29, 30]; as research in these languages is still in its infancy
[31] though, these findings should be revisited.

In general, the features found to be most influential in our study seem to be
mostly related to angry speech. Other mechanisms used to convey abusive
content, such as sarcasm and irony [2], are either not covered in the ADIMA
data set or are not captured by our features. This alternative form of abusive
speech may be more difficult to detect using prosody alone [32]. In addition,
[33] have shown that eGeMAPS has a tendency to lead to confusion of happiness
and anger categories in different classifiers, which may be due to the
similarity in arousal between these two categories [34]. This raises the
question of whether our classifiers also capture arousal.

One problem with the current study that needs to be addressed is the lack of
age and sex information in the data set. Therefore, it is not possible to
determine whether the data set is balanced with respect to these attributes.
Since acoustic features typically change with age and also differ between
males and females, this could explain some differences in features, especially
in pitch and vowel formant frequency, which are typically lower for males, and
voice onset time. In addition, [27] showed that the anger distinguishing
features differ between males (pitch as the main predictor) and females (F1,
F2, and F3 as main predictors).

##  5 Conclusions

The need to identify abusive content in audio and video data is growing
rapidly. This study shows that paralinguistic features, i. e., acoustic and
prosodic properties, can be successfully used to classify real-life audio
recordings depending on the presence of abusive and non-abusive content. Using
the features from eGeMAPS, we were able to achieve UAR scores between 0.66 and
0.84 for cross-lingual settings. The multilingual performance is particularly
interesting, showing that a model trained on all languages except one still
performs well on the excluded language.

The most influential features in our model were the mean loudness, mean F1,
F2, and F3 amplitude, mean flux, and mean voiced segments per second. These
results suggest that angry speech played a critical role in detecting abusive
content. Future research should further investigate how to detect other
mechanisms such as sarcasm and irony.

Breaking Down Barriers: Concerning this year's conference theme, we believe
the detection of abusive speech to be a highly relevant problem in the current
internet and social media era. Moreover, ADIMA is a multi-lingual data set
from a non-English domain and thus contributes to inclusive research in speech
science by targeting a diverse audience. However, it does not readily contain
metadata such as age and sex; we were, therefore, unable to evaluate the
(intersectional) generalisation of our methods with respect to these
variables.

##  6 Acknowledgements

This work has received funding from the EU’s Horizon 2021 grant agreement No.
101060660 (SHIFT).

##  7 References

## References

  * [1] Simrat Kaur, Sarbjeet Singh and Sakshi Kaushal  ``Abusive Content Detection in Online User-Generated Data: A survey''  In _Procedia Computer Science_ 189, 2021, pp. 274–281
  * [2] Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco and Viviana Patti  ``Resources and benchmark corpora for hate speech detection: a systematic review''  In _Language Resources and Evaluation_ 55, 2021, pp. 477–523
  * [3] Andre Sourander, Anat Brunstein Klomek, Maria Ikonen, Jarna Lindroos, Terhi Luntamo, Merja Koskelainen, Terja Ristkari and Hans Helenius  ``Psychosocial Risk Factors Associated With Cyberbullying Among Adolescents: A Population-Based Study''  In _Archives of General Psychiatry_ 67.7, 2010, pp. 720–728
  * [4] Ariadna Matamoros-Fernández and Johan Farkas  ``Racism, Hate Speech, and Social Media: A Systematic Review and Critique''  In _Television & New Media_ 22.2 Sage Publications Sage CA: Los Angeles, CA, 2021, pp. 205–224
  * [5] Ching Seh Wu and Unnathi Bhandary  ``Detection of Hate Speech in Videos Using Machine Learning''  In _2020 International Conference on Computational Science and Computational Intelligence (CSCI)_ , 2020, pp. 585–590
  * [6] Sreyan Ghosh, Samden Lepcha, S Sakshi, Rajiv Ratn Shah and Srinivasan Umesh  ``DeToxy: A Large-Scale Multimodal Dataset for Toxicity Classification in Spoken Utterances''  In _Proc. Interspeech 2022_ , 2022, pp. 5185–5189
  * [7] Fernando I. Ablaza, Timothy Oliver D. Danganan, Bryan Paul L. Javier, Kevin S. Manalang, Denise Erica V. Montalvo and Leonard U. Ambata  ``A small vocabulary automatic filipino speech profanity suppression system using hybrid Hidden Markov Model/Artificial Neural Network (HMM/ANN) keyword spotting framework''  In _2014 International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment and Management (HNICEM)_ , 2014, pp. 1–5
  * [8] Midia Yousefi and Dimitra Emmanouilidou  ``Audio-based Toxic Language Classification using Self-attentive Convolutional Neural Network''  In _2021 29th European Signal Processing Conference (EUSIPCO)_ , 2021, pp. 11–15
  * [9] Chuanpeng Yang, Fuqing Zhu, Guihua Liu, Jizhong Han and Songlin Hu  ``Multimodal Hate Speech Detection via Cross-Domain Knowledge Transfer''  In _Proceedings of the 30th ACM International Conference on Multimedia_ Lisboa, Portugal: Association for Computing Machinery, 2022, pp. 4505–4514
  * [10] Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale and Helen Margetts  ``Challenges and frontiers in abusive content detection''  In _Proceedings of the Third Workshop on Abusive Language Online_ Florence, Italy: Association for Computational Linguistics, 2019, pp. 80–93
  * [11] Bedour Alrashidi, Amani Jamal, Imtiaz Khan and Ali Alkhathlan  ``A review on abusive content automatic detection: approaches, challenges and opportunities''  In _PeerJ Computer Science_ 8, 2022, pp. e1142
  * [12] Rini Sharon, Heet Shah, Debdoot Mukherjee and Vikram Gupta  ``Multilingual and Multimodal Abuse Detection''  In _Proc. Interspeech 2022_ , 2022, pp. 4631–4635
  * [13] Sashi Novitasari, Dessi Puji Lestari, Sakriani Sakti and Ayu Purwarianti  ``Rude-Words Detection for Indonesian Speech Using Support Vector Machine''  In _2018 International Conference on Asian Language Processing (IALP)_ , 2018, pp. 19–24
  * [14] Taufic Leonardo Sutejo and Dessi Puji Lestari  ``Indonesia Hate Speech Detection Using Deep Learning''  In _2018 International Conference on Asian Language Processing (IALP)_ , 2018, pp. 39–43
  * [15] Shantanu Kumar Rahut, Riffat Sharmin and Ridma Tabassum  ``Bengali Abusive Speech Classification: A Transfer Learning Approach Using VGG-16''  In _2020 Emerging Technology in Computing, Communication and Electronics (ETCCE)_ , 2020, pp. 1–6
  * [16] Vikram Gupta, Rini Sharon, Ramit Sawhney and Debdoot Mukherjee  ``ADIMA: Abuse Detection In Multilingual Audio''  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 6172–6176
  * [17] Andreas Triantafyllopoulos, Johannes Wagner, Hagen Wierstorf, Maximilian Schmitt, Uwe Reichel, Florian Eyben, Felix Burkhardt and Björn W Schuller  ``Probing Speech Emotion Recognition Transformers for Linguistic Knowledge''  In _Proc. Interspeech 2022_ , 2022, pp. 146–150
  * [18] Florian Eyben, Martin Wöllmer and Björn Schuller  ``Opensmile: The Munich Versatile and Fast Open-Source Audio Feature Extractor''  In _Proceedings of the 18th ACM International Conference on Multimedia_ New York, NY, USA: Association for Computing Machinery, 2010, pp. 1459–1462
  * [19] Florian Eyben, Klaus R. Scherer, Björn W. Schuller, Johan Sundberg, Elisabeth André, Carlos Busso, Laurence Y. Devillers, Julien Epps, Petri Laukka, Shrikanth S. Narayanan and Khiet P. Truong  ``The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing''  In _IEEE Transactions on Affective Computing_ 7.2, 2016, pp. 190–202
  * [20] Björn Schuller, Stefan Steidl, Anton Batliner, Julia Hirschberg, Judee K Burgoon, Alice Baird, Aaron Elkins, Yue Zhang, Eduardo Coutinho and Keelan Evanini  ``The INTERSPEECH 2016 Computational Paralinguistics Challenge: Deception, Sincerity & Native Language''  In _Proc. Interspeech 2016_ , 2016, pp. 2001–2005
  * [21] András Vargha and Harold D Delaney  ``A Critique and Improvement of the "CL" Common Language Effect Size Statistics of McGraw and Wong''  In _Journal of Educational and Behavioral Statistics_ 25.2 [American Educational Research Association, Sage Publications, Inc., American Statistical Association], 2000, pp. 101–132
  * [22] Michael Ibañez, Ranz Sapinit, Lloyd Antonie Reyes, Mohammed Hussien, Joseph Marvin Imperial and Ramon Rodriguez  ``Audio-Based Hate Speech Classification from Online Short-Form Videos''  In _2021 International Conference on Asian Language Processing (IALP)_ , 2021, pp. 72–77
  * [23] Devin Soni and Vivek K Singh  ``See No Evil, Hear No Evil: Audio-Visual-Textual Cyberbullying Detection''  In _Proceedings of the ACM on Human-Computer Interaction_ 2.CSCW, 2018, pp. 1–26
  * [24] Turgut Özseven  ``Investigation of the Relation between Emotional State and Acoustic Parameters in the Context of Language''  In _Avrupa Bilim ve Teknoloji Dergisi_ , 2018, pp. 241–244
  * [25] Christina Sobin and Murray Alpert  ``Emotion in Speech: The Acoustic Attributes of Fear, Anger, Sadness, and Joy''  In _Journal of Psycholinguistic Research_ 28, 1999, pp. 347–365
  * [26] Jiahong Yuan, Liqin Shen and Fangxin Chen  ``The acoustic realization of anger, fear, joy and sadness in Chinese''  In _Proc. 7th International Conference on Spoken Language Processing (ICSLP 2002)_ , 2002, pp. 2025–2028
  * [27] Abhijit Mohanta and VK Mittal  ``Classifying emotional states using pitch and formants in vowel regions''  In _2016 International Conference on Signal Processing and Communication (ICSC)_ , 2016, pp. 458–463
  * [28] Jackson Liscombe, Jennifer Venditti and Julia Hirschberg  ``Classifying subject ratings of emotional speech using acoustic features''  In _Proc. 8th European Conference on Speech Communication and Technology (Eurospeech 2003)_ , 2003, pp. 725–728
  * [29] K Sreenivasa Rao, Shashidhar G Koolagudi and Ramu Reddy Vempada  ``Emotion recognition from speech using global and local prosodic features''  In _International journal of speech technology_ 16, 2013, pp. 143–160
  * [30] Mili Mary Mathew and Jayashree S Bhat  ``Aspects of emotional prosody in Malayalam and Hindi''  In _The Buckingham Journal of Language and Linguistics_ 3, 2010, pp. 25–34
  * [31] Anosha Ignatius and Uthayasanker Thayasivam  ``A Survey on Paralinguistics in Tamil Speech Processing''  In _Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages_ , 2021, pp. 94–99
  * [32] Gregory A Bryant and Jean E Fox Tree  ``Is there an ironic tone of voice?''  In _Language and Speech_ 48.3, 2005, pp. 257–277
  * [33] Cem Doğdu, Thomas Kessler, Dana Schneider, Maha Shadaydeh and Stefan R Schweinberger  ``A Comparison of Machine Learning Algorithms and Feature Sets for Automatic Vocal Emotion Recognition in Speech''  In _Sensors_ 22.19, 2022
  * [34] Swarna Kuchibhotla, Hima Deepthi Vankayalapati, RS Vaddi and Koteswara Rao Anne  ``A comparative analysis of classifiers in emotion recognition through acoustic features''  In _International Journal of Speech Technology_ 17, 2014, pp. 401–408