# DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion

###### Abstract

Singing voice conversion (SVC) is one promising technique which can enrich the
way of human-computer interaction by endowing a computer the ability to
produce high-fidelity and expressive singing voice. In this paper, we propose
DiffSVC, an SVC system based on denoising diffusion probabilistic model.
DiffSVC uses phonetic posteriorgrams (PPGs) as content features. A denoising
module is trained in DiffSVC, which takes destroyed mel spectrogram produced
by the diffusion/forward process and its corresponding step information as
input to predict the added Gaussian noise. We use PPGs, fundamental frequency
features and loudness features as auxiliary input to assist the denoising
process. Experiments show that DiffSVC can achieve superior conversion
performance in terms of naturalness and voice similarity to current state-of-
the-art SVC approaches.

Index Terms—  singing voice conversion, diffusion probabilistic model

##  1 Introduction

Singing plays an important role in human daily life, including information
transmission, emotional expression and entertainment. The technology of
singing voice conversion (SVC) aims at converting the voice of a singing
signal to a voice of a target singer without changing the underlying content
and melody. Endowing machine with the ability to produce high-fidelity and
expressive singing voice provides new ways for human-computer interaction. SVC
is among the possible ways to achieve this.

Most recent SVC systems train a content encoder to extract content features
from a source singing signal and a conversion model to transform content
features to either acoustic features or waveform. One class of SVC approaches
jointly trains the content encoder and the conversion model as an auto-encoder
model [1, 2]. Another class of SVC approaches separately trains the content
encoder and the conversion model. These approaches train an automatic speech
recognition (ASR) model as the content encoder. The ASR model can be an end-
to-end model, as in [3, 4] or a hybrid HMM-DNN model, as in [5]. The
conversion model can be the generator in a generative adversarial network
(GAN) [3, 4], which directly generates waveform from content features; or a
regression model, which transforms content features to spectral features
(e.g., mel spectrograms), and adopts an additionally trained neural vocoder to
generate waveform. In this paper, we focus on the latter class and devote to
introducing the recently-emerged diffusion probabilistic modeling into the
conversion model.

Diffusion probabilistic models, or diffusion models for brevity, are a class
of promising generative models [6]. It has been demonstrated that diffusion
models are capable of achieving state-of-the-art performance on generative
modeling for natural images [7, 8] and raw audio waveform [9, 10]. There are
two processes in a diffusion model, i.e., the diffusion/forward process and
the reverse process. The diffusion/forward process is a Markov chain with
fixed parameters, which converts the complicated data into isotropic Gaussian
distribution by adding Gaussian noise gradually. The reverse process is also a
Markov chain, which restores the data structure from Gaussian noise in an
iterative manner. One merit of diffusion models is that they can be
efficiently trained by optimizing the evidence lower bound (ELBO), also known
as variational lower bound (VLB), of the data likelihood. In [7], one certain
parameterization trick can convert the learning process of diffusion models
into a regression problem. We follow this setting in this paper.

We propose DiffSVC, which is an SVC system based on diffusion model. We use an
ASR acousic model to extract phonetic posteriorgrams (PPGs) from singing
signals as the content features. A diffusion model is trained to recover mel
spectrograms iteratively from Gaussian noise, conditioning on content, melody
and loudness features. We show that DiffSVC can achieve superior conversion
performance in terms of naturalness and voice similarity, compared with
current state-of-the-art SVC approaches. The contributions of our work
include: (1) To the best of our knowledge, the proposed DiffSVC system is the
first SVC system using the diffusion probabilistic model. We show that
diffusion model can be effectively adopted for the SVC task; and (2) DiffSVC
achieves better conversion performance in terms of naturalness and voice
similarity than previous SVC systems.

The rest of this paper is organized as follows: Section 2 presents related
work. Section 3 introduces the diffusion probabilistic model. Details of the
proposed DiffSVC system are presented in Section 4. Experimental results are
shown in Section 5 and Section 6 concludes this paper.

##  2 Related work

###  2.1 Singing Voice Conversion

Current SVC approaches can be categorized into two classes, i.e., parallel SVC
and non-parallel SVC, in terms of whether parallel training data is required.
Most initial attempts for SVC are within the parallel SVC paradigm. These
approaches model parallel training samples using statistical methods, such as
Gaussian mixture model (GMM)-based many-to-many eigenvoice conversion [11],
direct waveform modification based on spectrum difference [12, 13]. GAN-based
parallel approach has also been proposed to improve conversion performance
[14]. Since parallel SVC approaches require parallel data, which is expensive
to collect, for the training process, researchers have investigated many non-
parallel SVC approaches. Auto-encoder based on the WaveNet [15] structure has
been used for unsupervised SVC in [1], which can convert among singers
appeared in the training set. This approach adopts an adversarial speaker
classifier to disentangle singer information from the encoder output. To
further improve this method, an additional pitch adversarial mechanism is
added to remove pitch information from the encoder output in [2]. Variational
auto-encoder (VAE) [16], variational autoencoding Wasserstein GAN (VAW-GAN)
[17], and phonetic posteriorgram (PPG) models [5] are also investigated for
non-parallel SVC. Very recently, the combination of a PPG model and a waveform
generator achieves promising SVC performance [3, 4].

###  2.2 Diffusion Probabilistic Models

There are two streams of efforts, pushing the research on diffusion
probabilistic models forward. One is research on the score matching model,
which originates from [18], where the problem of data density estimation is
simplified into a score matching problem (the estimation of the gradient of
the data distribution density). The other is research on denoising diffusion
probabilistic model, which originates from [6], where a diffusion Markov chain
is used to destroy data structure into Gaussian noise and another reverse
process is used to generate data from Gaussian noise. Recently, great progress
has been made in these two streams. In [7], the diffusion model has been able
to generate high-quality images. Slightly later, the diffusion model also
makes it to generate high-quality audio samples [9, 10]. On the other hand,
score matching model has also been proved to be capable of generating high-
resolution images using a neural network to estimate the log gradient of the
target data probabilistic density function [19]. Afterwards, a unified
framework is proposed in [20], which generalizes and improves previous work on
score matching model through the lens of stochastic differential equations
(SDEs). The diffusion model in [7] and the score matching model in [19] have
been shown to be special cases under such framework.

Fig. 1: Graphical model for the reverse process and the diffusion/forward
process in a diffusion probabilistic model. The diffusion/forward process
(dotted arrow) gradually converts data into noise within finite steps. The
reverse process (solid arrow) is a parametric procedure which attempts to
restore data structure from noise. Fig. 2: Schematic diagram of the conversion
model in the proposed DiffSVC system, based on the diffusion probabilistic
model.

##  3 Diffusion Probabilistic Model

In this section, we introduce the denoising diffusion probabilistic model (or
diffusion model in short) [6]. We denote the data as $y_{0}\sim q(y_{0})$. A
diffusion model is a latent variable model with the form
$p_{\theta}(y_{0}):=\int p_{\theta}(y_{0:T})dy_{1:T}$, where $y_{1},...,y_{T}$
are latent variables with the same dimensionality as the data $y_{0}$ and $T$
is the total number of diffusion steps. As shown in Fig. 1, a diffusion model
contains two processes – the diffusion/forward process and the reverse
process. In the remaining part of this section, we first introduce the two
processes in a diffusion model and then discuss the training and sampling
algorithms.

###  3.1 Diffusion/Forward Process

The diffusion/forward process in a diffusion model is modeled as a Markov
chain that gradually adds small noise to the data until the data structure is
totally destroyed at step $T$. That is, the diffusion process gradually
whitens the data into Gaussian noise in $T$ diffusion steps. The chain
transitions are often modeled as conditional Gaussian transitions according to
a deterministic noise schedule $\beta_{1},...,\beta_{T}$, which are set to
satisfy $\beta_{1}<\beta_{2}<...<\beta_{T}$. The joint probability of the
latent variables is:

$q(y_{1:T}|y_{0}):=\prod_{t=1}^{T}q(y_{t}|y_{t-1}),$ (1)

where
$q(y_{t}|y_{t-1}):=\mathcal{N}(y_{t};\sqrt{1-\beta_{t}}y_{t-1},\beta_{t}I)$.
The mean term $\sqrt{1-\beta_{t}}$ gradually decays the destroyed data towards
origin, and the variance term $\beta_{t}$ incorporates stochasticity into the
process by adding small noises, for $t\in\\{1,...,T\\}$.

###  3.2 Reverse Process

The reverse process in a diffusion model is a parameterised process which
gradually learns to restore data structure from white noise along another
Markov chain. When $\beta$’s of the diffusion/forward process are small, both
the diffusion and reverse process have the same functional form [6], i.e.,
Gaussian transitions. Starting at isotropic Gaussian
$p(y_{T})=\mathcal{N}({y_{T};0,I}$), the joint distribution
$p_{\theta}(y_{0:T})$ along the reverse process satisfies:

$p_{\theta}(y_{0:T}):=p(y_{T})\prod_{t=1}^{T}p_{\theta}(y_{t-1}|y_{t}),$ (2)

where the transition probability $p_{\theta}(y_{t-1}|y_{t})$ is parameterized
as $\mathcal{N}(y_{t-1};\mu_{\theta}(y_{t},t),\sigma_{\theta}(y_{t},t)^{2}I)$
with shared parameter $\theta$ among all the $T$ transitions. The training
process of a diffusion model is to learn the parameter $\theta$, such that
$p_{\theta}(y_{t-1}|y_{t})$ can eliminate the Gaussian noise (i.e., denoise)
added during the diffusion/forward process.

###  3.3 Training Diffusion Model

The goal of training a diffusion model is to maximize the model likelihood
$p_{\theta}(y_{0}):=\int p_{\theta}(y_{0:T})dy_{1:T}$, given a training set
$\\{y_{0}^{m}\\}_{m=1}^{M}$ with $M$ samples. However, the likelihood is
intractable to compute in general because of the integration operation in a
very high-dimensional latent space. Thus, its evidence lower bound (ELBO),
which is also known as variational lower bound (VLB), is used to train the
model:

$\mathbb{E}_{q(y_{0})}[\log
p_{\theta}(y_{0})]\geq\underbrace{\mathbb{E}_{q(y_{0:T})}[\log\frac{p_{\theta}(y_{0:T-1}|y_{T})p(y_{T})}{q(y_{1:T}|y_{0})}]}_{:=ELBO}$
(3)

We refer readers to [6] for a complete proof of Eq. (3).

Thanks to the property of Gaussian transitions in the diffusion process, the
noisy data $y_{t}$ (i.e., destroyed data) at any step $t\in\\{1,...,T\\}$ can
be readily sampled given a data sample $y_{0}$:

$q(y_{t}|y_{0})=\mathcal{N}(y_{t};\sqrt{\bar{\alpha}_{t}}y_{0},\sqrt{1-\bar{\alpha}_{t}}I),$
(4)

where $\alpha_{t}:=1-\beta_{t}$ and
$\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$. Following the reparameterization
trick presented in [7], the learning process becomes a regression problem,
where a neural network (parameterized with $\theta$) is trained, which takes
the noisy data $y_{t}$ and the step variable $t$ as input to predict the
corresponding noise $\epsilon$. Using a simplified variant of the ELBO in Eq.
(3) as in [7], the negative ELBO becomes:

$-ELBO=\mathbb{E}_{y_{0},t}[||\epsilon-\epsilon_{\theta}(\sqrt{\bar{\alpha}_{t}}y_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,t)||_{2}^{2}],$
(5)

where $\epsilon\sim\mathcal{N}(0,I)$, $y_{0}\sim q(y_{0})$, and $t$ is
uniformly sampled from $1,...,T$. In this paper, we use Eq. (5) to train the
diffusion model.

###  3.4 Sampling with Langevin Dynamics

Given the parameterized reverse process with the well-learned parameter
$\theta$, the sampling process (i.e., the generative process) is to first
sample a Gaussian noise $y_{T}\sim\mathcal{N}(0,I)$, and then iteratively
sample $y_{t-1}\sim p_{\theta}(y_{t-1}|y_{t})$ for $t=T,T-1,...,1$ along the
reverse process, according to Langevin dynamics:

$y_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}(y_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(y_{t},t))+\sigma_{t}z,$
(6)

where
$\sigma_{t}^{2}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}$,
$z\sim\mathcal{N}(0,I)$ for $t>1$ and $z=0$ for $t=1$. The final $y_{0}$ is
the generated data.

##  4 DiffSVC

In this section, we describe how we introduce the diffusion model to SVC and
present details of the proposed DiffSVC system. We regard the content, melody
and loudness information as the most important components in a singing signal
for the task of singing voice conversion. In DiffSVC, we train a Deep-FSMN
(DFSMN)-based ASR acoustic model [21] to extract PPGs as the content features
with forced-aligned audio-text speech data. We train the ASR model with frame-
wise cross-entropy loss, where the ground-truth labels are phonemes (i.e.,
initials and finals with tone) for Mandarin Chinese SVC.

We introduce the concept of denoising diffusion modeling into the conversion
model in DiffSVC. The overall architecture is depicted in Fig. 2. During the
training process, DiffSVC aims at predicting the isotropic Gaussian noise
$\epsilon$ from the noisy mel spectrogram $y_{t}$ and the step variable $t$,
leveraging additional information from PPG $x$, logarithmic fundamental
frequency feature (Log-F0) $f_{0}$ and loudness feature $l$. We introduce the
details in the following subsections.

###  4.1 PPG, Log-F0 and Loudness

The PPG prenet adopts a simple fully-connected (FC) layer on the PPG input.
DiffSVC represents the melody of a singing signal using its fundamental
frequency contour in logarithmic scale. A-weighting mechanism of a singing
signal’s power spectrum is adopted to compute loudness features, following
[4]. We process the Log-F0 features and loudness features in the same way. The
Log-F0 features and loudness features are first quantized into 256 bins, and
then go through a melody embedding lookup table and a loudness embedding
lookup table respectively. The processed PPG features, Log-F0 features and
loudness features are added elementwisely to obtain the conditioner $e$, which
is token as additional input to the diffusion decoder.

###  4.2 Diffusion Modeling

Step Encoding. The diffusion step variable $t$ is an important input to a
diffusion model, because it indicates the amount of noise added to the noisy
mel spectrogram $y_{t}$. Following the setting in [9], we first convert
integral-valued $t$ into an 128-dimensional vector $t_{emb}$ with a sinusoidal
position encoding as:

$[\sin(10^{\frac{0\times 4}{63}}t),...,\sin(10^{\frac{63\times
4}{63}}t),\cos(10^{\frac{0\times 4}{63}}t),...,\cos(10^{\frac{63\times
4}{63}}t)]$ (7)

Then we further process $t_{emb}$ with two FC layers and Swish activation
function [22].
Diffusion Decoder. The diffusion decoder is trained to predict the noise
$\epsilon$ added by the diffusion process from the noisy mel spectrogram
$y_{t}$ and the step variable $t$, taking auxiliary conditioners $e$ obtained
from PPG $x$, Log-F0 $f_{0}$ and loudness $l$. Following [9], the diffusion
decoder in DiffSVC adopts a bidirectional residual convolutional architecture
proposed in [15] with a slight difference. We use dialtion rate of 1 since we
work on mel spectrograms instead of raw waveform. The receptive field of the
diffusion decoder is sufficiently large with a dilation rate of 1. The
diffusion decoder first conducts an one-dimensional convolutional operation
with kernel-size 1 (Conv1x1) on the noisy mel spectrogram and then adopts the
ReLU activation on the output. The output of the step encoder is added to
every time step of the mel spectrogram feature maps, and then fed into $N$
residual blocks. The conditioner $e$ is imported into every residual block
with separate Conv1x1 layers, whose output is elementwisely added with the mel
spectrogram and step features. Then the gated mechanism introduced in [15] are
used to further process the feature maps. We sum the skip connections from all
the $N$ residual layers and further process with two Conv1x1 layers
interleaved with a ReLU activation to get the diffusion decoder output.
Training and conversion We summarize the training procedure of the conversion
model in DiffSVC in Algorithm 1 and the conversion procedure in Algorithm 2,
respectively.

Algorithm 1 Training procedure of the conversion model in DiffSVC.

0: The conversion model $\epsilon_{\theta}(\cdot)$; the training set
$\mathcal{D}_{train}=\\{(x,f0,l,y_{0})\\}_{m=1}^{M}$; $N_{iter}$ training
iterations.

1: for $i=1,2,...,N_{iter}$ do

2: Sample $(x,f_{0},l,y_{0})$ from $\mathcal{D}_{train}$;

3: $\epsilon\sim\mathcal{N}(0,I)$;

4: Sample $t\sim$ Uniform$(\\{1,\cdots,T\\})$;

5: Take gradient descent step on
$\nabla_{\theta}||\epsilon-\epsilon_{\theta}(\sqrt{\bar{\alpha}_{t}}y_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,t,x,f_{0},l))||_{2}^{2}$;

6: end for

7: return $\epsilon_{\theta}(\cdot)$;

Algorithm 2 Conversion procedure of DiffSVC.

0: The trained conversion model $\epsilon_{\theta}(\cdot)$; one testing sample
$(x,f0,l)$.

1: Sample $y_{T}\sim\mathcal{N}(0,I)$;

2: for $t=T,T-1,...,1$ do

3: Sample $(x,f_{0},l,y_{0})$ from $\mathcal{D}$;

4: $\epsilon\sim\mathcal{N}(0,I)$;

5: if $t>1$ then

6: Sample $z\sim\mathcal{N}(0,I)$;

7: else

8: $z=0$;

9: end if

10:
$y_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}(y_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(y_{t},t,x,f_{0},l))+\sigma_{t}z$,
as in Eq. 6;

11: end for

12: return $y_{0}$;

##  5 Experiments

We evaluate the proposed DiffSVC by conducting any-to-one SVC tasks, where the
singing voice of an arbitrary source singer is converted to the target
singer’s. In this section, we first introduce the corpus for the experiments
and the data pre-processing methods. Then the implementation details of both
the baseline approaches and proposed approach are presented. Finally, we
present the evaluation results.

###  5.1 Dataset and preprocessing

The PPG extractor is trained using an internal Mandarin Chinese ASR corpus,
which contains around 20k hours speech data recorded by thousands of speakers.
The PPG features have dimensionality of 218. The conversion model in DiffSVC
is trained with an internal dataset, which contains 14 hours singing data
recorded by a female voice talent. The audio format is 16-bit PCM with a
sample rate of 24 kHz. Mel spectrograms have 80 frequency bins, and are
computed with 1024-point fast Fourier transform (FFT), 1024-point window size
and 240-point hop size (i.e., 10 ms). Mel spectrograms are min-max normalized
to the range [-1, 1]. Fundamental frequency (F0) values are computed with hop
size of 240. To make the F0 computation robust, we use three F0 estimators,
i.e., DIO [23], REAPER 111[Online] https://github.com/google/REAPER and SPTK
[24]. The F0 values take the median results by this three F0 estimators. The
FFT size, window size and hop size for computing loudness features are 2048,
2048 and 240, respectively.

###  5.2 Implementation details

The output size of the PPG prenet, the melody embedding size and loudness
embedding size are all 256. The number of bins for quantizing Log-F0 and
loudness is 256. The step encoder has the same hyper-parameter setting as in
[9]. We use 20 residual layers in a diffusion decoder, where the number of
convolution channels are 256. The total number of diffusion steps is 100
(i.e., T = 100). The noise schedule $\beta$’s are set to linearly spaced from
$1\times 10^{-4}$ to 0.06. We train models with the ADAM optimizer [25] with a
constant learning rate of $0.0002$.

We train a Hifi-GAN model [26] as vocoder to convert mel spectrograms to the
raw waveform. The official implementation222[Online]
https://github.com/jik876/hifi-gan (the “config_v1.json” configuration) is
used with a slight change. Since we use 24 kHz audio samples and the hop-size
for computing mel spectrogram is 240, we factorize the upsample rate as
$240=8\times 5\times 3\times 2$. To avoid possible checkerboard artifacts
caused by the “ConvTranspose1d” upsampling layer, we use temporal nearest
interpolation layer followed by a one-dimensional convolutional layer as the
upsampling operation, as suggested in [27]. The kernel sizes we use are
[15,15,5,5].

###  5.3 Comparisons

We compare the proposed DiffSVC system with three state-of-the-art SVC
approaches. To make the comparison fair, all models use the same DFSMN-based
PPG extractor (as introduced in Section 4) to compute PPG features as the
content representations. Fundamental frequency ($f_{0}$) values are computed
using the robust ensemble method introduced in Section 5.1. The details of
their implementations are presented as follows.
BLSTM-SVC This model is proposed by [28]. We make slight modifications to make
the comparison fair: (1) We use mel spectrogram and Log-F0 as the acoustic
features instead of the conventional vocoder parameters such as mel cepstral
(MCEP) features and aperiodicity (AP) features; and (2) we use the Hifi-GAN
vocoder introduced in Section 5.2 for high-fidelity waveform generation. We
reimplemented the 512N model in [28] since it shows better conversion
performance according to the experimental results in the orginal paper.
Seq2seq-SVC This model is proposed by [5], where a sequence-to-sequence
(seq2seq) based model using GMM-based attention mechanism is trained to
predict mel spectrograms from PPGs. Slight modifications are also made to make
the comparison fair: (1) we do not use the Mel encoder and the singer
confusion module because we focus on single-speaker conversion model; (2) we
use the Hifi-GAN vocoder introduced in Section 5.2 for converting mel
spectrograms into waveform, instead of using a WaveRNN vocoder[29].
FastSVC This model is presented in [4], which uses a generator in a GAN model
to convert content features directly into waveform, and is trained by
combining a multi-scale spectral loss and an adversarial loss. In the original
paper, the content features used by the authors have a hop-size of 320, while
the PPGs used in this paper have a hop-size of 240. Therefore, we change the
upsampling rates of the generator to [5,4,4,3] from the original [4,4,4,5]. To
further adapt FastSVC for 24 kHz audio synthesizing (originally, FastSVC works
on 16 kHz audio), we add the multi-period discriminator (MPD) [26] as
additional discriminator. These changes make FastSVC achieving better
conversion performance on 24 kHz audio samples.

###  5.4 Evaluations

Subjective evaluations in terms of both the naturalness and voice similarity
of converted speech are conducted. We use the 5 point Likert scale for testing
with mean opinion score (MOS) (1-bad, 2-poor, 3-fair, 4-good, 5-excellent). In
the MOS tests for evaluating naturalness, each group of stimuli contains
recording samples from the target singer, which are randomly shuffled with the
samples generated by the four comparative approaches, before they are
presented to raters. In the MOS similarity tests, converted speech samples are
directly compared with the recording samples of the target singer. 10
utterances from an evaluation set are presented for each system. We invited
raters to participate in the evaluations in a quiet room and they were asked
to use headphones during the tests. The raters were allowed to replay each
sample as many times as necessary and change their ratings of any sample
before submitting their results. The MOS results are demonstrated in Table 1.
We can see that DiffSVC achieves significantly better performance than the
other three compared systems (i.e., BLSTM-SVC, Seq2seq-SVC and FastSVC), in
terms of both naturalness and voice similarity. Audio samples can be found
online333[Online] https://liusongxiang.github.io/diffsvc/.

Table 1: Subjective and objective evaluation results. “Nat.” is for
“Naturalness” and “Sim.” is for “Voice Similarity”, repectively. Up arrows
mean “higher the better”. Down arrows mean “lower the better”.

System | Subjective | Objective
---|---|---
Nat. ($\uparrow$) | Sim.($\uparrow$) | MCD ($\downarrow$) | FPC($\uparrow$)
Recordings | 4.60$\pm$0.09 | - | - | -
BLSTM-SVC | 2.37$\pm$0.15 | 3.67$\pm$0.15 | 6.424 | 0.781
Seq2seq-SVC | 2.77$\pm$0.17 | 3.57$\pm$0.15 | 7.175 | 0.885
FastSVC | 3.83$\pm$0.13 | 4.17$\pm$0.13 | 6.422 | 0.904
DiffSVC | 3.97$\pm$0.14 | 4.67$\pm$0.10 | 6.307 | 0.902

We use mel-cepstrum distortion (MCD) and F0 Pearson correlation (FPC) as
metrics for objective evaluations. We extract the mel-cepstrum of the
generated audio samples and the ground-truth audio samples from the target
singer for MCD evaluation. For the FPC evaluation, we extract the F0 from the
generated audio samples and the recordings from the source singer, since our
goal is to convert the voice of a singing signal to the target singer’s voice
without changing the underlying content and melody. The DTW algorithm is
adopted to align the predicted acoustic features towards the natural ones. The
objective results are presented in Table 1. We can see that, among the four
compared systems, DiffSVC achieves the best MCD result (6.307). In terms of
FPC, the FastSVC system achieves the best result (0.904). DiffSVC achieves
slightly lower FPC result than FastSVC, with an FPC value of 0.902 and an 0.2%
relative drop.

Combining the experimental results, we can safely say that DiffSVC achieves
superior singing voice conversion performance than previous SVC approaches in
terms of naturalness and voice similarity of the converted signal.

##  6 Conclusion

In this paper, we have presented DiffSVC. To the best of our knowledge,
DiffSVC is the first SVC approach using a diffusion probabilistic model to
generate acoustic features (e.g., mel spectrogram) from content features
(e.g., PPG). Experiments demonstrate that the diffusion model can be
effectively adopted for the SVC task. Moreover, DiffSVC can achieve superior
conversion performance in terms of naturalness and voice similarity than
current state-of-the-art SVC systems, according to the subjective and
objective evaluations. In the work, we focus on any-to-one SVC task. Future
work includes further extending DiffSVC for any-to-many/any SVC. Investigation
on training a DiffSVC model under low-data-resource scenario is also deserved
research effort.

## References

  * [1] Eliya Nachmani and Lior Wolf,  “Unsupervised singing voice conversion,”  INTERSPEECH, 2019.
  * [2] Chengqi Deng, Chengzhu Yu, Heng Lu, Chao Weng, and Dong Yu,  “Pitchnet: Unsupervised singing voice conversion with pitch adversarial network,”  in ICASSP. IEEE, 2020, pp. 7749–7753.
  * [3] Adam Polyak, Lior Wolf, Yossi Adi, and Yaniv Taigman,  “Unsupervised cross-domain singing voice conversion,”  INTERSPEECH, 2020.
  * [4] Songxiang Liu, Yuewen Cao, Na Hu, Dan Su, and Helen Meng,  “Fastsvc: Fast cross-domain singing voice conversion with feature-wise linear modulation,”  ICME, 2021.
  * [5] Zhonghao Li, Benlai Tang, Xiang Yin, Yuan Wan, Ling Xu, Chen Shen, and Zejun Ma,  “Ppg-based singing voice conversion with adversarial representation learning,”  in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 7073–7077.
  * [6] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli,  “Deep unsupervised learning using nonequilibrium thermodynamics,”  in International Conference on Machine Learning. PMLR, 2015, pp. 2256–2265.
  * [7] Jonathan Ho, Ajay Jain, and Pieter Abbeel,  “Denoising diffusion probabilistic models,”  in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds. 2020, vol. 33, pp. 6840–6851, Curran Associates, Inc.
  * [8] Jiaming Song, Chenlin Meng, and Stefano Ermon,  “Denoising diffusion implicit models,”  in International Conference on Learning Representations, 2021.
  * [9] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro,  “Diffwave: A versatile diffusion model for audio synthesis,”  arXiv preprint arXiv:2009.09761, 2020.
  * [10] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan,  “Wavegrad: Estimating gradients for waveform generation,”  arXiv preprint arXiv:2009.00713, 2020.
  * [11] Tomoki Toda, Yamato Ohtani, and Kiyohiro Shikano,  “One-to-many and many-to-one voice conversion based on eigenvoices,”  in ICASSP. IEEE, 2007, vol. 4, pp. IV–1249.
  * [12] Kazuhiro Kobayashi, Tomoki Toda, Graham Neubig, Sakriani Sakti, and Satoshi Nakamura,  “Statistical singing voice conversion with direct waveform modification based on the spectrum differential,”  in INTERSPEECH, 2014.
  * [13] Kazuhiro Kobayashi, Tomoki Toda, Graham Neubig, Sakriani Sakti, and Satoshi Nakamura,  “Statistical singing voice conversion based on direct waveform modification with global variance,”  in INTERSPEECH, 2015.
  * [14] Berrak Sisman, Karthika Vijayan, Minghui Dong, and Haizhou Li,  “Singan: Singing voice conversion with generative adversarial networks,”  in APSIPA ASC). IEEE, 2019, pp. 112–118.
  * [15] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu,  “Wavenet: A generative model for raw audio,”  ISCA SSW, 2016.
  * [16] Yin-Jyun Luo, Chin-Cheng Hsu, Kat Agres, and Dorien Herremans,  “Singing voice conversion with disentangled representations of singer and vocal technique using variational autoencoders,”  in ICASSP. IEEE, 2020, pp. 3277–3281.
  * [17] Junchen Lu, Kun Zhou, Berrak Sisman, and Haizhou Li,  “Vaw-gan for singing voice conversion with non-parallel training data,”  in 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2020, pp. 514–519.
  * [18] Aapo Hyvärinen,  “Estimation of non-normalized statistical models by score matching,”  Journal of Machine Learning Research, vol. 6, no. 24, pp. 695–709, 2005.
  * [19] Yang Song and Stefano Ermon,  “Generative modeling by estimating gradients of the data distribution,”  in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds. 2019, vol. 32, Curran Associates, Inc.
  * [20] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole,  “Score-based generative modeling through stochastic differential equations,”  in International Conference on Learning Representations, 2021.
  * [21] Shiliang Zhang, Ming Lei, Zhijie Yan, and Lirong Dai,  “Deep-fsmn for large vocabulary continuous speech recognition,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5869–5873.
  * [22] Prajit Ramachandran, Barret Zoph, and Quoc V Le,  “Searching for activation functions,”  arXiv preprint arXiv:1710.05941, 2017.
  * [23] Masanori Morise, Hideki Kawahara, and Haruhiro Katayose,  “Fast and reliable f0 estimation method based on the period extraction of vocal fold vibration of singing voice and speech,”  in Audio Engineering Society Conference: 35th International Conference: Audio for Games. Audio Engineering Society, 2009.
  * [24] Keiichi Tokuda, Keiichiro Oura, and et al.,  “Speech signal processing toolkit (sptk),”  in http://sp-tk.sourceforge.net, 2009.
  * [25] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  ICLR, 2015.
  * [26] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae,  “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,”  in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds. 2020, vol. 33, pp. 17022–17033, Curran Associates, Inc.
  * [27] Augustus Odena, Vincent Dumoulin, and Chris Olah,  “Deconvolution and checkerboard artifacts,”  Distill, 2016.
  * [28] Xin Chen, Wei Chu, Jinxi Guo, and Ning Xu,  “Singing voice conversion with non-parallel data,”  in 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR). IEEE, 2019, pp. 292–296.
  * [29] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu,  “Efficient neural audio synthesis,”  in International Conference on Machine Learning. PMLR, 2018, pp. 2410–2419.


# CMNEROne at SemEval-2022 Task 11: Code-Mixed Named Entity Recognition by
leveraging multilingual data

Suman Dowlagar
LTRC
IIIT-Hyderabad
suman.dowlagar
@research.iiit.ac.in
Radhika Mamidi
LTRC
IIIT-Hyderabad
radhika.mamidi
@iiit.ac.in

###### Abstract

Identifying named entities is, in general, a practical and challenging task in
the field of Natural Language Processing. Named Entity Recognition on the
code-mixed text is further challenging due to the linguistic complexity
resulting from the nature of the mixing. This paper addresses the submission
of team CMNEROne to the SEMEVAL 2022 shared task 11 MultiCoNER. The Code-mixed
NER task aimed to identify named entities on the code-mixed dataset. Our work
consists of Named Entity Recognition (NER) on the code-mixed dataset by
leveraging the multilingual data. We achieved a weighted average F1 score of
0.7044, i.e., 6% greater than the baseline.

##  1 Introduction

Named entity recognition (NER) is a fundamental task in NLP. It aims to
identify and classify entities in a text into predefined types. It is an
essential tool for information retrieval, question answering, Banerjee et al.
(2019) and text summarization tasks Patil et al. (2016); Li et al. (2020).
However, except for some resource-rich monolingual languages, NER annotated
data for most other languages are still very limited Kruengkrai et al. (2020).
Moreover, it is usually time-consuming to annotate such data, particularly for
low-resource languages such as multilingual and code-mixed Liu et al. (2021).
Therefore, transfer learning and leveraging the other datasets for
multilingual and code-mixed NER has attracted growing interest recently,
especially with the influx of deep learning methods.

This paper presents the system description for named entity recognition on the
code-mixed dataset. Code-mixing is defined as using two or more languages in a
single sentence or utterance Dowlagar and Mamidi (2021). The use of code-mixed
language is prevalent in most multilingual societies. Due to linguistic
complexity arising from mixing two languages, the processing of code-mixed
sentences is a challenging task Bali et al. (2014). So, the models that are
trained on monolingual and multilingual datasets typically fail to handle
code-mixed inputs Khanuja et al. (2020). Therefore, to encourage research on
code-mixing, the speech and NLP communities are organizing several shared
tasks. The shared tasks have concentrated on language identification, POS-
tagging, sentiment analysis, hate speech detection, and several datasets exist
for these as well. Similarly, SEMEVAL 2022’s Task 11 sub-task 13 was devoted
to identifying named entities in code-mixed languages Malmasi et al. (2022b).
This task aims to classify the given tokens in the code-mixed sentences as
persons, corporation, location, and others. An example is shown in Table 1.

Sentence | hameM | this | magic | moment
---|---|---|---|---
Languge | Hi | En | En | En
NER tags | O | B-CW | I-CW | I-CW
Table 1: An example of the code-mixed NER annotated sentence. The Hindi words are converted from utf to wx format and are italicized sIriyala | naMbara | xvArA | kleding | in | de | oudheid | kI | pahacAna | kareM
---|---|---|---|---|---|---|---|---|---
O | O | O | B-PROD | I-PROD | I-PROD | I-PROD | O | O | O
what | city | is | dig | me | out | in? |  |  |
O | 0 | O | B-CW | I-CW | I-CW | O |  |  |
AmAra | das | testament | mUlya | kawa? |  |  |  |  |
0 | B-CW | I-CW | O | O |  |  |  |  |
Table 2: An example of the code-mixed NER annotated sentence. The multilingual
words are converted from utf to wx format and are italicized

The lack of annotated data is a crucial issue for code-mixed datasets. Lack of
data poses a problem of data overfitting and poor entity recognition. The
language models trained on such low resource datasets cannot generalize the
training data, thus performing low on the test datasets. Several previous
studies have used monolingual data as training signals for transfer learning,
and these data can also be used in the form of pre-training. Thus, we used a
similar approach of including the multilingual data along with the code-mixed
dataset.

We used the multilingual pre-trained BERT model as our model for code-mixed
NER. The model uses code mixed training data along with the multilingual
training and mulilingual validation data.

We have analyzed that Bi-LSTM with CRF models has shown an improved accuracy
on the token classification tasks such as POS tagging, language
identification, and NER. The ensemble of BERT or XLM-RoBERTa with Bi-LSTM and
CRF would have shown a further improvement in the code-mixed NER. Also, using
language identification as a downstream task with the current method might
have improved the NER’s accuracy.

The paper is organized as follows. Section 2 provides related work on Named
Entity Recognition on CM social media text. Section 3 provides information on
the task and examples of datasets. Section 4 describes the proposed work.
Section 5 presents the experimental setup and Section 6 project the
performance of the model. Section 7 concludes our work.

##  2 Related Work

Code-mixed NER has attracted a lot of attention in the NLP community this
decade. This section lists the latest works on code-mixed named entity
recognition.

Priyadharshini et al. (2020); Winata et al. (2019) generated multilingual meta
representations from pre-trained monolingual word embeddings. The model
learned to construct the best word representation by mixing multiple sources
without explicit language identification.

Aguilar et al. (2019) presented a shared task on named entity recognition in
the CALCS workshop. The language pairs used were English-Spanish (ENG-SPA) and
Modern Standard Arabic Egyptian (MSA-EGY). They used Twitter data and nine
entity types to establish a new dataset for code-switched NER benchmarks. The
participating teams used LSTM, CNN, CRF, and word representations to recognize
named entities.

Singh et al. (2018) We presented a new Code-Mixed Hinglish corpus for NER.
Different machine learning classification algorithms with word, character, and
lexical features are used to establish baselines. The algorithms used were
Decision tree, Long Short-Term Memory (LSTM), and Conditional Random Field
(CRF).

Meng et al. (2021); Fetahu et al. (2021) presented a novel CM NER model. They
proposed a gated architecture that enhances existing multilingual Transformers
by dynamically infusing multilingual knowledge bases, a.k.a gazetteers. The
evaluation of code-mixed queries shows that this approach efficiently utilizes
gazetteers to recognize entities in code-mixed queries with an F1=68%, an
absolute improvement of +31% over a non-gazetteer baseline.

Meng et al. (2021) mentioned that including Gazetteer features could cause
models to overuse or underuse them, leading to poor generalization. They
proposed a new approach for gazetteer knowledge integration by including
Context in Gazetteer Representation using encoder and Mixture-of-Experts
gating network models. These models overcome the feature overuse issue by
learning to conditionally combine the context and gazetteer features instead
of assigning them fixed weights.

##  3 Task Setup

The shared task detects semantically ambiguous and complex entities in short
and low-context settings. Complex NEs, like the titles of creative works
(movie/book/song/software names), are not simple nouns. Usually, they take
imperative clauses, or they often resemble typical syntactic constituents.
Such NEs are harder to recognize Ashwini and Choi (2014). Syntactic parsing of
such complex noun phrases is hard, and most NER systems fail to identify them.
Inside–outside–beginning (IOB) format Ramshaw and Marcus (1999) is used for
annotating entities. A few examples of complex NEs and ambiguous NEs from the
code-mixed dataset are given in Table 2.

So the MultiCoNER shared task encourages the models to handle such complex
NEs. A huge dataset Malmasi et al. (2022a) is released for this task Malmasi
et al. (2022b). The languages focused on in this shared task are: English,
Spanish, Dutch, Russian, Turkish, Korean, Farsi, German, Chinese, Hindi, and
Bangla. The shared task also offered an additional track with code-mixed and
multilingual datasets. In this paper, we will be concentrating on the code-
mixed dataset.

##  4 System Overview

We finetuned the pre-trained multilingual BERT model by using the multilingual
training and validation datasets for code-mixed named entity recognition. We
found that the training data is insufficient for the deep learning language
model to identify the named entities in the validation data correctly. The
data scarcity of low-resource languages has been a significant challenge for
building NLP systems since they require a large amount of data to learn a
robust model. We observed that the multilingual NER training data is similar
to the code-mixed dataset. Also, it is relatively large when compared to the
code-mixed dataset. In our approach, the multilingual training and validation
data is combined with the code-mixed training dataset. Using the combined
dataset, we finetune the deep neural network model. Our method thus attempts
to learn language-agnostic features by using the combined multilingual and
code-mixed dataset. This finetuned model can be used to infer named entity
information at a token level on a code-mixed low resource language.

We used the pre-trained mBERT Devlin et al. (2018) model for code-mixed NER.
mBERT is a transformer-based self multi-headed attention model that is pre-
trained on a massive collection of multilingual data and can be finetuned for
our NER task. As the model is pre-trained on a large corpus, the semantic and
syntactic information is well modeled and can be directly finetuned for a
specific task. BERT is a bi-directional transformer model Vaswani et al.
(2017). It analyzes the meaning of a term depending on its context given on
both sides. The transformer part in the BERT works like an attention mechanism
capable of learning the contextual relationships between the terms in a
sentence.

##  5 Experimental Setup

The section presents the baselines, hyper-parameter settings, and analysis of
observed results.

###  5.1 Baselines

The baselines used for the proposed work is:

#### Conditional random field (CRF) Lafferty et al. (2001)

CRF is a statistical model and is a well-known approach for handling NER
problems. The CRF model considers the neighboring samples by modeling the
prediction as a graphical model. It assumes that the tag for the present word
(denoted as $y_{i}$) is dependent on the tag of its previous/next word
(denoted as $y_{i-1}$ or $y_{i+1}$).

#### MultiCoNER baseline Malmasi et al. (2022b)

The XLM-RoBERTa base with CRF model is used as a baseline for NER.

#### Pre-trained multilingual BERT (mBERT) Devlin et al. (2018)

A pre-trained multilingual BERT model with token classification without
leveraging the multilingual data is used as a baseline.

###  5.2 Hyperparameters and libraries

For developing our model, the neural network library used is PyTorch, and the
pre-trained multilingual BERT model (bert-base-multilingual-cased) and XLM-
ROBERTa base model (xlm-roberta-base) is obtained from the hugging face-
transformers library and is finetuned for the code-mixed NER task. The model
is implemented in Kaggle Notebook with GPU processing.

The batch size of the datasets is kept as 64. The maximum length of the
sentence from the training data is considered during the input data
encoding/padding. Due to subword tokenization, we used the first token for
predicting the tag. The optimizer used is weighted Adam with the learning rate
of 2e-5 and epsilon value equal to 1e-5. The dropout is set to 0.1. The loss
function used is a cross-entropy loss that is inbuilt into the transformer’s
BERT model. The number of epochs used for training the model is 30. The
training is stopped when there is no change in validation accuracy for more
than four epochs.

Model | F1-score
---|---
| w/o multilingual data | with multilingual data
| valid data | test data | valid data | test data
CRF | 0.565 | 0.561 | 0.560 | 0.556
mBERT | 0.627 | 0.612 | 0.716 | 0.707
MultiCoNER baseline | 0.651 | 0.645 | 0.725 | 0.719
Table 3: The performance of the models on the code-mixed dataset with and
without including multilingual data. Our submission for the given task is
highlighted. Figure 1: Confusion matrix of CM-NER baseline Figure 2: Confusion
matrix of CM-NER by leveraging multilingual data

The CRF is obtained from pytorch-crfsuite
library111https://github.com/scrapinghub/python-crfsuite. The previous word
and its tag, the next word, and its tag is used as the features to predict the
tag of the current word.

##  6 Results and Analysis

Table 3 presents the f1-score of the models on the Dravidian code-mixed
dataset. From the above results, it is clear that our system, i.e., leveraging
the multilingual NER data in a low-resource code-mixed setting, improves the
NER task compared to the baseline models. The CRF model didn’t perform well on
the given NER task, as this statistical model does not capture the semantics
of the tokens. Even the CRF with multilingual data performed poorly on this
task compared to the baseline NN models. It shows the importance of capturing
semantical, syntactic, and contextual information while building the NER model
on these complex datasets.

Our submission, the pre-trained mBERT by leveraging the multilingual dataset,
performed better than the MultiCoNER baseline by 6%. Even the MultiCoNER
baseline with the multilingual dataset performed better than our submission.

The confusion matrices with and without multilingual data of our submission on
the code-mixed NER validation dataset are shown in the Figures 1 and 2. By
using confusion matrices, we observed that the multilingual data given in
Figure 2 helped better identify the CW, PROD, CROP, and LOC entities when
compared to the baseline model.

##  7 Conclusion and future work

In this paper, we addressed the shared task on named entity recognition for
the code-mixed dataset. As the code-mixed data is a low resource language and
there are no pre-trained models, we leveraged the multilingual dataset for
training the NER model. The model used for testing our method is the pre-
trained multilingual BERT. We finetuned the pre-trained mBERT for the code-
mixed NER task by using the code-mixed training data and multilingual training
and validation datasets.

The use of meta embeddings for dealing with code-mixed datasets has recently
attracted a lot of attention. It might be possible that meta embedding-based
NER will work better on this code-mixed dataset. Unlike the social media data
where code-mixed sentences/words are written in Roman script, the native
script is used for each word, so the language identification will work better
on this dataset. Using Language identification or POS tagging as a downstream
task for NER on this dataset will help in improving the code-mixed NER.

## References

  * Aguilar et al. (2019) Gustavo Aguilar, Fahad AlGhamdi, Victor Soto, Mona Diab, Julia Hirschberg, and Thamar Solorio. 2019.  Named entity recognition on code-switched data: Overview of the calcs 2018 shared task.  _arXiv preprint arXiv:1906.04138_.
  * Ashwini and Choi (2014) Sandeep Ashwini and Jinho D Choi. 2014.  Targetable named entity recognition in social media.  _arXiv preprint arXiv:1408.0782_.
  * Bali et al. (2014) Kalika Bali, Jatin Sharma, Monojit Choudhury, and Yogarshi Vyas. 2014.  "i am borrowing ya mixing?" an analysis of english-hindi code mixing in facebook.  In _Proceedings of the First Workshop on Computational Approaches to Code Switching_ , pages 116–126.
  * Banerjee et al. (2019) Partha Sarathy Banerjee, Baisakhi Chakraborty, Deepak Tripathi, Hardik Gupta, and Sourabh S Kumar. 2019.  A information retrieval based on question and answering and ner for unstructured information without using sql.  _Wireless Personal Communications_ , 108(3):1909–1931.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dowlagar and Mamidi (2021) Suman Dowlagar and Radhika Mamidi. 2021.  Offlangone@ dravidianlangtech-eacl2021: Transformers with the class balanced loss for offensive language identification in dravidian code-mixed text.  In _Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages_ , pages 154–159.
  * Fetahu et al. (2021) Besnik Fetahu, Anjie Fang, Oleg Rokhlenko, and Shervin Malmasi. 2021.  Gazetteer Enhanced Named Entity Recognition for Code-Mixed Web Queries.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1677–1681.
  * Khanuja et al. (2020) Simran Khanuja, Sandipan Dandapat, Sunayana Sitaram, and Monojit Choudhury. 2020\.  A new dataset for natural language inference from code-mixed conversations.  _arXiv preprint arXiv:2004.05051_.
  * Kruengkrai et al. (2020) Canasai Kruengkrai, Thien Hai Nguyen, Sharifah Mahani Aljunied, and Lidong Bing. 2020.  Improving low-resource named entity recognition using joint sentence and token labeling.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5898–5905.
  * Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
  * Li et al. (2020) Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020.  A survey on deep learning for named entity recognition.  _IEEE Transactions on Knowledge and Data Engineering_ , 34(1):50–70.
  * Liu et al. (2021) Linlin Liu, Bosheng Ding, Lidong Bing, Shafiq Joty, Luo Si, and Chunyan Miao. 2021\.  Mulda: A multilingual data augmentation framework for low-resource cross-lingual ner.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5834–5846.
  * Malmasi et al. (2022a) Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022a.  MultiCoNER: a Large-scale Multilingual dataset for Complex Named Entity Recognition.
  * Malmasi et al. (2022b) Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022b.  SemEval-2022 Task 11: Multilingual Complex Named Entity Recognition (MultiCoNER).  In _Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)_. Association for Computational Linguistics.
  * Meng et al. (2021) Tao Meng, Anjie Fang, Oleg Rokhlenko, and Shervin Malmasi. 2021.  GEMNET: Effective gated gazetteer representations for recognizing complex entities in low-context input.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1499–1512.
  * Patil et al. (2016) Nita Patil, Ajay S Patil, and BV Pawar. 2016.  Survey of named entity recognition systems with respect to indian and foreign languages.  _International Journal of Computer Applications_ , 134(16).
  * Priyadharshini et al. (2020) Ruba Priyadharshini, Bharathi Raja Chakravarthi, Mani Vegupatti, and John P McCrae. 2020.  Named entity recognition for code-mixed indian corpus using meta embedding.  In _2020 6th international conference on advanced computing and communication systems (ICACCS)_ , pages 68–72. IEEE.
  * Ramshaw and Marcus (1999) Lance A Ramshaw and Mitchell P Marcus. 1999.  Text chunking using transformation-based learning.  In _Natural language processing using very large corpora_ , pages 157–176. Springer.
  * Singh et al. (2018) Vinay Singh, Deepanshu Vijay, Syed Sarfaraz Akhtar, and Manish Shrivastava. 2018\.  Named entity recognition for hindi-english code-mixed social media text.  In _Proceedings of the seventh named entities workshop_ , pages 27–35.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Winata et al. (2019) Genta Indra Winata, Zhaojiang Lin, and Pascale Fung. 2019.  Learning multilingual meta-embeddings for code-switching named entity recognition.  In _Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)_ , pages 181–186.