# Low Pass Filtering and Bandwidth Extension for Robust Anti-spoofing
Countermeasure Against Codec Variabilities

###### Abstract

A reliable voice anti-spoofing countermeasure system needs to robustly protect
automatic speaker verification (ASV) systems in various kinds of spoofing
scenarios. However, the performance of countermeasure systems could be
degraded by channel effects and codecs. In this paper, we show that using the
low-frequency subbands of signals as input can mitigate the negative impact
introduced by codecs on the countermeasure systems. To validate this, two
types of low-pass filters with different cut-off frequencies are applied to
countermeasure systems, and the equal error rate (EER) is reduced by up to 25%
relatively. In addition, we propose a deep learning based bandwidth extension
approach to further improve the detection accuracy. Recent studies show that
the error rate of countermeasure systems increase dramatically when the
silence part is removed by Voice Activity Detection (VAD), our experimental
results show that the filtering and bandwidth extension approaches are also
effective under the codec condition when VAD is applied.

Index Terms: anti-spoofing, bandwidth extension, low-pass filters, band
trimming, channel robustness, transmission codec

##  1 Introduction

Automatic speaker verification (ASV) [1, 2] has achieved good performance and
is widely used in real life. As a biometric method, however, ASV systems are
vulnerable against various spoofing attacks, e.g. speech synthesis, voice
conversion, record and playback, etc. [3, 4]. Anti-spoofing countermeasure
systems contribute to enhance the reliability of ASV systems by determining
whether the input signal is genuine or spoofed.

Since 2015, the ASVspoof community has initiated and organized four
consecutive biennial challenges to support the development of anti-spoofing
countermeasure methods for ASV systems [5, 6, 7, 8]. Through this series of
challenges, the ASV anti-spoofing field has established two main anti-spoofing
countermeasure research scenarios and databases: logical access (LA) considers
spoofing attacks from text-to-speech synthesis (TTS) and voice conversion
(VC), physical access (PA) refers to attacks produced by recording replay [9,
10]. In this paper, we focus on the LA scenario. A typical countermeasure
system consists of a front-end feature extractor and a back-end spoofing
classifier. For the LA task, the most intuitive solution is to find the
artifact traces existing in the speech from TTS and VC through the front-end
operations, which can be used as a marker or cue to distinguish genuine speech
from artifacts and help train the back-end detection network better. Previous
works show that artifacts of synthetic speech exist in multiple specific
subbands [11, 12]. More and more studies focus on the impact of subbands in
countermeasure systems [13].

The current strategies for using frequency subbands in countermeasure systems
can be broadly classified into two categories. One uses transformations,
trimming, fusion or other methods in the front-end feature extractor to
transform the features to a specific domain, which emphasize the information
in target subbands [11, 13, 14, 15]. The other one directly adopt the
spectrogram as the features, use SpecAugment [16] or similar data augmentation
techniques to randomly or specifically block a portion of the frequency bands
or time frames to reduce the overfitting of the back-end classifier [17, 18].

Zhang et al. point out that the high-frequency part of the spectrogram may
lead to overfitting of the back-end neural network, increasing it’s risk of
making incorrect judgments in the face of unknown spoofing attacks [13].
Moreover, Tomilov et al. find that a data augmentation such as feeding
training data into a filter to emulate the magnitude responses of codecs can
yield better anti-spoofing countermeasure performance [19].

However, it’s not clear that whether low pass filtering is always useful with
different bandwidths or scenarios, and whether the re-estimated high frequency
information from a deep learning based up-sampling approach could bring some
additional gain. This paper investigates the optimal cutoff frequency in terms
of low pass filters against the codec variabilities. Moreover, inspired by
Tomilov et al’s work [19], we use a conventional low-pass filter to obtain a
subband signal instead of trimming the spectrogram directly. In addition, we
further demonstrate the gain when using a deep learning based bandwidth
extension technique to restore the wide band signal from the low pass filtered
narrow band speech.

As shown in Figure 1, we investigate the performance of countermeasure systems
based on four front-end feature extraction methods and two convolutional
neural networks (CNN) back-end classifiers. We also explore the performance of
our countermeasure systems after applying a Voice Activity Detection (VAD)
module. Our contributions are mainly threefold:

Figure 1: The overview of the proposed front-end signal processing methods.

  * •

We investigate the differences when using low-frequency subbands at the
system’s input features in two different databases and find out that high
frequency subbands are more vulnerable against the codecs.

  * •

We use extensive experimental results to validate the optimal cutoff frequency
for our countermeasure systems.

  * •

We first utilize a deep learning based bandwidth extension technique on the
down-sampled signal in the countermeasure system, and suggest that the
additional bandwidth extension module can be effective on the valid speech
part when a VAD is applied.

##  2 Database and Methodology

This section describes the database and illustrates the detail of our front-
end signal processing methods shown in Figure 1.

###  2.1 Databases

The experiments in this work are mainly conducted on the ASVspoof2019 LA
database [7] and the ASVspoof2021 LA database [8]. Both databases are based
upon the Voice Cloning Toolkit (VCTK) corpus [20]. The ASVspoof2019 LA
database was created using utterances from 107 speakers (46 male, 61 female).
The set of 107 speakers is partitioned into three speaker-disjoint sets for
training, development, and evaluation. The spoofed utterances were generated
using four TTS and two VC algorithms in the training and development sets,
while 13 TTS/VC algorithms are used in the evaluation subset, 11 of which are
unknown for training and development. The ASVspoof2021 LA database remains the
training and development data unchanged and only proposes a new evaluation
subset that contains attacks using the same simulation methods as the
ASVspoof2019 LA evaluation subset. The ASVspoof2021 LA evaluation subset
consists of the data in various telephone transmission systems, including
Voice over Internet Protocol (VoIP) and the Public Switched Telephone Networks
(PSTN), thus exhibiting real-world signal transmission channel effects. These
effects could generate artifacts different from spoofing data that affect the
accuracy of the countermeasure system. These conditions make the ASVspoof2021
LA evaluation subset an excellent platform for evaluating countermeasure
systems’ generalization capability and robustness against channel effects and
codec variabilities. The contents of the two databases are summarized in Table
1.

Table 1: Number of utterances in ASVspoof 2019 and 2021 LA database. | 19LA | 21LA
---|---|---
Subset | Bona-fide | Spoof | Bona-fide | Spoof
Training | 2,580 | 22,800 | - | -
Development | 2,548 | 22,296 | - | -
Evaluation | 7,355 | 63,882 | 14,816 | 166,750

###  2.2 Baseline

We adopt the log Mel-filter bank energy (FBANK) as the acoustic feature in all
our experiments. The Fast Fourier Transform (FFT) spectrogram is extracted
with 1024 window length and 128 hop length while the Blackman window is used.
Then we set the number of Mel-filters to 80 dimensions. Due to the different
speech lengths, each spectrogram’s length is truncated or concatenated into 3
to 5 seconds. Finally, the $80*frames$-dimensional FBANK features are
obtained.

For the backend classifier, we investigated two main CNN networks, ResNet18
[21] and LCNN [17]. Then we add the attentive statistics pooling layer (ASP)
[22, 2] at the end of the models to make the model more effective in capturing
utterance-level acoustic feature changes. The ResNet18 and the LCNN are
commonly and widely used systems in anti-spoofing tasks [23]. We use softmax
cross entropy as the loss function of the classifier. The architectures of the
models are shown in Tables 2 and 3, respectively.

Table 2:  The architecture of ResNet18, $\mathbf{C}$ denotes the convolutional layer, $\mathbf{S}$ denotes the shortcut convolutional layer. Layer | Output Size | Structure(kernal size, stride)
---|---|---
Conv1 | $16\times D\times L$ | $\mathbf{C}(3\times 3,1)$
| Residual
---
Layer 1
$16\times D\times L$ | $\begin{bmatrix}\mathbf{C}(3\times 3,1)\\\ \mathbf{C}(3\times 3,1)\end{bmatrix}\times 2$
| Residual
---
Layer 2
$32\times\frac{D}{2}\times\frac{L}{2}$ | $\begin{bmatrix}\mathbf{C}(3\times 3,2)\\\ \mathbf{C}(3\times 3,1)\\\ \mathbf{S}(1\times 1,2)\end{bmatrix}\begin{bmatrix}\mathbf{C}(3\times 3,1)\\\ \mathbf{C}(3\times 3,1)\end{bmatrix}\times 2$
| Residual
---
Layer 3
$64\times\frac{D}{4}\times\frac{L}{4}$ | $\begin{bmatrix}\mathbf{C}(3\times 3,2)\\\ \mathbf{C}(3\times 3,1)\\\ \mathbf{S}(1\times 1,2)\end{bmatrix}\begin{bmatrix}\mathbf{C}(3\times 3,1)\\\ \mathbf{C}(3\times 3,1)\end{bmatrix}\times 2$
| Residual
---
Layer 4
$128\times\frac{D}{8}\times\frac{L}{8}$ | $\begin{bmatrix}\mathbf{C}(3\times 3,2)\\\ \mathbf{C}(3\times 3,1)\\\ \mathbf{S}(1\times 1,2)\end{bmatrix}\begin{bmatrix}\mathbf{C}(3\times 3,1)\\\ \mathbf{C}(3\times 3,1)\end{bmatrix}\times 2$
Pooling | $128\times\frac{D}{8}$ | Attentive Statistics Pooling
Linear | $128$ | Fully Connected Layer
Linear | $2$ | Fully Connected Layer
Table 3: The architecture of LCNN. Layer | Structure(kernal size, stride) | Output Size
---|---|---
|  Same as the first 28 layers of LCNN [17] |
Pooling | BiLSTM | $80+80$
Pooling | Attentive Statistics Pooling | $320$
Linear | Fully Connected Layer | $128$
Linear | Fully Connected Layer | $2$

###  2.3 Band trimming

Band trimming means cropping a particular dimension of the Mel-filter bank
from the complete FBANK to make it consistent with the spectrogram subbands.
According to the Nyquist frequency characteristic, the spectrogram of the
speech with a sampling rate of 16k covers a bandwidth of 0-8k Hz, and we need
to select the low-frequency subband cover 0-$\mathbb{F}$ Hz for training and
testing the countermeasure system, which $\mathbb{F}$ here denotes the subband
frequencies corresponding to 20%, 30%, 40%, 50%, 60%, 70% Nyquist frequencies.
The equation describing the correlation between the number of low- and full-
frequency FBANK dimensions [24] is shown as follows

$\lfloor N_{L}\rfloor=\lfloor
N_{F}\ast\frac{log(1+f_{L}/700)}{log(1+f_{F}/700)}\rfloor.$ (1)

Where $\lfloor*\rfloor$ indicates rounding down an element $*$. If the
operation of band trimming is considered to be low-pass filtering of the
signal, then $f_{L}$ denotes the cutoff frequency of this filter and $f_{F}$
refers to the Nyquist frequency. $N_{L}$ and $N_{F}$ represent the number of
filter banks for the low- and full-frequency FBANK feature, respectively. For
instance, when we need to select the low-frequency subband up to 50% of the
Nyquist frequency, the corresponding value of $f_{F}$, $f_{L}$, sample
frequency, and $N_{F}$ are $8,000$, $4,000$, $16,000$, and $80$ respectively.
Therefore, according to Equation 1, we get $N_{L}=60.45$. The feature
dimension must be an integer, so we floor $N_{L}$ to be 60 and set $f_{L}$ to
3,933.55 Hz. In other words, the spectrogram information from 3,933.55 Hz to
4,000 Hz is dropped out. With respect to Equation 1, 20%, 30%, 40%, 50%, 60%,
70% of Nyquist frequencies corresponds to the FBANK indices 37, 47, 54, 60,
65, and 69.

###  2.4 Low-pass filtering

Considering the response curve shape, the computational complexity, and the
consistency of the filter in the bandwidth extension front-end, the Chebyshev
type I filter is chosen as the low-pass filter in our experiments. We set the
order of filter to 8, the maximum ripple of the Chebyshev type I low-pass
filter to 0.05, and the critical frequencies to $\mathbb{F}$ Hz, which
$\mathbb{F}$ is the same as the one mentioned in the last subsection.

###  2.5 Bandwidth extension

Bandwidth extension is also known as audio upsampling or audio super-
resolution. It usually aims to enhance speech audibility and improve audio
fidelity by generating a wideband (WB) signal from a narrowband (NB) signal.
In order to use extension models to enhance the performance of the
countermeasure system in this study, we investigate some bandwidth extension
methods [25, 26, 27]. Among them, Viet-Anh Nguyen et al. proposed a
Transformer-aided UNet (TUNet)111Source code:
https://github.com/NXTProduct/TUNet. by employing a low-complexity transformer
encoder on the bottleneck of a lightweight UNet [28]. Their experimental
results on the VCTK corpus show that the TUNet achieves state-of-the-art
performance in intrusive and non-intrusive metrics.

The workflow of the bandwidth extension front-end is shown on the left below
of Figure 1, where a Chebyshev Type I lowpass filter is used to preprocess the
original 16k Hz signal into a low pass filtered signal but still at a sampling
rate of 16k Hz. After that, the filtered signal is used as the input of the
TUNet extension model [28], and the extended signal containing 0-8k Hz
spectrogram can be restored after network inference. Finally, the output of
the extension model is changed to FBANK features and as the input of the back-
end classifier.

##  3 Experimental Setup

###  3.1 Data Augmentation

As shown in Figure 1, in order to improve the robustness of the countermeasure
classifier, we implemented data augmentation to add noise before FFT in the
front-end. Motivated by the data augmentation methods used in the ASV system
with the VoxCeleb database [29, 30, 31], reverberation and background noise
are added randomly to two-thirds of the input data. The noise data are
obtained from the MUSAN [32] and RIR [33] databases.

###  3.2 Metric and training strategy

The evaluation was performed in terms of Equal Error Rate (EER) as a metric,
which indicates that the proportion of false acceptances is equal to the
proportion of false rejections.

Table 4: The EER% of countermeasure systems, when choosing 0.5 Nyquist frequency as the cutoff frequency. Bold number indicates the best performing result among the countermeasure systems corresponding to each back-end network in the 19LA or 21LA database. back-ends | front-ends | ASVspoof2021 LA | ASVspoof2019 LA
---|---|---|---
seed1 | seed10 | seed100 | Average | seed1 | seed10 | seed100 | Average
ResNet18 | baseline | 3.23 | 3.35 | 3.38 | 3.32 | 1.76 | 1.48 | 1.15 | 1.46
band trimming | 2.65 | 2.32 | 2.44 | 2.47 | 1.37 | 1.66 | 1.46 | 1.5
low-pass filtering | 3.05 | 3.03 | 3.05 | 3.04 | 1.34 | 1.7 | 1.57 | 1.53
low pass + bandwidth extension | 2.35 | 2.57 | 2.72 | 2.54 | 1.37 | 1.38 | 1.6 | 1.45
LCNN | baseline | 3.95 | 4.37 | 3.38 | 4.26 | 2.36 | 2.36 | 2.5 | 2.41
band trimming | 3.23 | 2.72 | 2.76 | 2.90 | 2.03 | 1.75 | 1.72 | 1.83
low-pass filtering | 3.51 | 2.84 | 3.07 | 3.14 | 2.31 | 1.89 | 2.14 | 2.11
low pass + bandwidth extension | 2.94 | 2.97 | 3.01 | 2.91 | 2.05 | 2.05 | 1.84 | 1.98

For training the countermeasure systems, Adam [34] was used as the optimizer
with $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=10^{-8}$, and weight
decaying $10^{-4}$. For the CNN classifiers, the learning rate increase
linearly for the first four warm-up epochs and then is initialized to 0.001
starting from the fifth epoch. The learning rate is scheduled as the PyTorch
ReduceLROnPlateau
function222https://pytorch.org/docs/stable/generated/torch.optim.lr_sched
uler.ReduceLROnPlateau.html, reducing the learning rate when the metric has
stopped improving for ten epochs. Each experiment uses one NVIDIA RTX A6000
GPU, and for efficiency, we set the batch to 400, with 100 epochs of training
per model.

##  4 Experimental Results and Discussion

###  4.1 Countermeasure systems with different front-end

From Figure LABEL:fig:result1, it seems a cutoff frequency of 0.5 is a good
candidate for comparison experiments. The experimental results of all eight
countermeasure systems are shown in Table 4, in which the cutoff frequencies
were set to 0.5 times the Nyquist frequency. In addition, since some studies
have shown that the performance difference caused by different random seeds
may even be more significant than that caused by different countermeasure
system constructions [35], we repeat the experiment three times for each case
of the system using three random seeds, then we calculate the average EER.
Table 4 suggests that at 0.5 times the Nyquist frequency, the countermeasure
systems based on band trimming and bandwidth extension front-ends have similar
performance on the ASVspoof2021 LA database. However, the best performance on
the ASVspoof2019 LA database is achieved by the baseline front-end system.

Table 5: The EER% of 4 different front-ends and ResNet18 back-end countermeasure systems after VAD operation. back-ends | front-ends | 21LA | 19LA
---|---|---|---
ResNet18 | baseline | 20.17 | 13.06
band trimming | 19.91 | 18.24
low-pass filtering | 19.08 | 16.64
low pass + bandwidth extension | 15.23 | 15.08

Under the six cutoff frequencies specified in this experiment, Figure
LABEL:fig:result1 provides the results of the countermeasure system based on
band trimming and conventional low-pass filter front-end as well as the
baselines. It can be found that all countermeasure systems composed of low-
frequency subband front-ends outperform the baseline system in the
ASVspoof2021 LA database when the cutoff frequency is greater than 0.3 times
the Nyquist frequency. As this observation is consistent on two baseline
systems, we think it because low-frequency subbands can reduce the
variabilities of channel effects and codecs as human ears are less sensitive
on high frequency regions which might be more distorted in transmission. The
results on the ASVspoof2019 LA database show that low-frequency subbands
improve the LCNN backend systems performance significantly more than the
ResNet18 system. Combined with the conclusion of Zhang et al. [13], we suggest
that the low pass filtering or down-sampling to narrow band spectrograms is
effective when there is relative large channel effects or codec variabilities,
however, it might not be useful when there is little high frequency
variability on speech. In that case, dropping out high frequency information
would result in degraded performance. Figure LABEL:fig:result2 compares the
system’s performance before and after the bandwidth extension module. It can
be found that the bandwidth extension operation on the low pass filtered
signal improves performance in most cases. Combining Figure LABEL:fig:result1
and Figure LABEL:fig:result2, we determine the optimal cutoff frequency for
band trimming/filtering/bandwidth extension as 0.5/0.4/0.5 times Nyquist
frequency.

###  4.2 The effect of VAD operation

Some studies have shown that since the countermeasure system focuses on silent
segments to distinguish spoofed speech from genuine speech, the data after VAD
processing will be difficult to classify [13, 19]. We want to further test our
proposed methods on the silence removed speech signals after VAD.

We use the librosa.effects.trim
function333http://librosa.org/doc/latest/generated/librosa.effects.trim.html
of the open source toolkit Librosa to implement the VAD function. We set the
$top\\_db$ parameter to 40, which means that the part of each data with a
maximum energy below 40 dB is considered as silence. The performance of each
system is shown in Table 5 with their own optimal cutoff frequency condition,
and it can be found that the EER of all systems increased substantially
compared to the ones without VAD. The model with the baseline front-end
increase the least on the 2019 database, but the most on the 2021 database;
while the EER of systems with the bandwidth extension front-end increased less
in both databases. It suggests that bandwidth extension can improve the system
performance on speech part of the signal significantly. It also suggest that
the original silence signal contains large portion of spoofing artifacts,
applying additional signal processing methods on it might distort the silence
part artifacts.

##  5 Conclusions

This work validates the countermeasure systems on the ASVspoof2019 LA and the
ASVspoof2021 LA databases and shows that low-frequency narrow band can reduce
the disturbance caused by channel effects and codec variabilities. In
addition, bandwidth extension can significantly reduce the performance
degradation after VAD. Moreover, we compared different front-ends and
determined the optimal cut-off frequency for those systems. Our future work
will focus on exploring the cases that low pass filtering and bandwidth
extension are only applied on the speech part of the signal and leave the
silence part unchanged.

## References

  * [1] D. A. Reynolds, T. F. Quatieri, and R. B. Dunn, “Speaker verification using adapted gaussian mixture models,” _Digital signal processing_ , vol. 10, no. 1-3, pp. 19–41, 2000.
  * [2] B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,” in _Proc. Interspeech 2020_ , 2020, pp. 3830–3834.
  * [3] M. R. Kamble, H. B. Sailor, H. A. Patil, and H. Li, “Advances in anti-spoofing: from the perspective of asvspoof challenges,” _APSIPA Transactions on Signal and Information Processing_ , vol. 9, 2020.
  * [4] N. Evans, T. Kinnunen, J. Yamagishi, Z. Wu, F. Alegre, and P. D. Leon, “Speaker recognition anti-spoofing,” in _Handbook of biometric anti-spoofing_ , 2014, pp. 125–146.
  * [5] Z. Wu, T. Kinnunen, N. Evans, J. Yamagishi, C. Hanilçi, M. Sahidullah, and A. Sizov, “ASVspoof 2015: the first automatic speaker verification spoofing and countermeasures challenge,” in _Proc. Interspeech 2015_ , 2015, pp. 2037–2041.
  * [6] T. Kinnunen, M. Sahidullah, H. Delgado, M. Todisco, N. Evans, J. Yamagishi, and K. A. Lee, “The ASVspoof 2017 Challenge: Assessing the Limits of Replay Spoofing Attack Detection,” in _Proc. Interspeech 2017_ , 2017, pp. 2–6.
  * [7] M. Todisco, X. Wang, V. Vestman, M. Sahidullah, H. Delgado, A. Nautsch, J. Yamagishi, N. Evans, T. H. Kinnunen, and K. A. Lee, “ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection,” in _Proc. Interspeech 2019_ , 2019, pp. 1008–1012.
  * [8] J. Yamagishi, X. Wang, M. Todisco, M. Sahidullah, J. Patino, A. Nautsch, X. Liu, K. A. Lee, T. Kinnunen, N. Evans, and H. Delgado, “ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection,” in _Proc. of the Automatic Speaker Verification and Spoofing Countermeasures Challenge_ , 2021, pp. 47–54.
  * [9] F. Tom, M. Jain, and P. Dey, “End-To-End Audio Replay Attack Detection Using Deep Convolutional Networks with Attention,” in _Proc. Interspeech 2018_ , 2018, pp. 681–685.
  * [10] R. Baumann, K. M. Malik, A. Javed, A. Ball, B. Kujawa, and H. Malik, “Voice spoofing detection corpus for single and multi-order audio replays,” _Computer Speech & Language_, vol. 65, p. 101132, 2021.
  * [11] K. Sriskandaraja, V. Sethu, P. N. Le, and E. Ambikairajah, “Investigation of Sub-Band Discriminative Information Between Spoofed and Genuine Speech,” in _Proc. Interspeech 2016_ , 2016, pp. 1710–1714.
  * [12] J. Yang, R. K. Das, and H. Li, “Significance of subband features for synthetic speech detection,” _IEEE Transactions on Information Forensics and Security_ , vol. 15, pp. 2160–2170, 2019.
  * [13] Y. Zhang, W. Wang, and P. Zhang, “The Effect of Silence and Dual-Band Fusion in Anti-Spoofing System,” in _Proc. Interspeech 2021_ , 2021, pp. 4279–4283.
  * [14] H. Tak, J. Patino, A. Nautsch, N. Evans, and M. Todisco, “Spoofing Attack Detection Using the Non-Linear Fusion of Sub-Band Classifiers,” in _Proc. Interspeech 2020_ , 2020, pp. 1106–1110.
  * [15] B. Chettri, T. Kinnunen, and E. Benetos, “Subband Modeling for Spoofing Detection in Automatic Speaker Verification,” in _Proc. Odyssey 2020_ , 2020, pp. 341–348.
  * [16] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,” in _Proc. Interspeech 2019_ , 2019, pp. 2613–2617.
  * [17] G. Lavrentyeva, S. Novoselov, A. Tseren, M. Volkova, A. Gorlanov, and A. Kozlov, “STC Antispoofing Systems for the ASVspoof2019 Challenge,” in _Proc. Interspeech 2019_ , 2019, pp. 1033–1037.
  * [18] R. Yan, C. Wen, S. Zhou, T. Guo, W. Zou, and X. Li, “Audio deepfake detection system with neural stitching for add 2022,” in _Proc. ICASSIP 2022_ , 2022, pp. 9226–9230.
  * [19] A. Tomilov, A. Svishchev, M. Volkova, A. Chirkovskiy, A. Kondratev, and G. Lavrentyeva, “STC Antispoofing Systems for the ASVspoof2021 Challenge,” in _Proc. of the Automatic Speaker Verification and Spoofing Countermeasures Challenge_ , 2021, pp. 61–67.
  * [20] J. Yamagishi, C. Veaux, and K. MacDonald, “Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92),” 2019.
  * [21] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. CVPR 2016_ , 2016, pp. 770–778.
  * [22] K. Okabe, T. Koshinaka, and K. Shinoda, “Attentive Statistics Pooling for Deep Speaker Embedding,” in _Proc. Interspeech 2018_ , 2018, pp. 2252–2256.
  * [23] X. Wang, X. Qin, T. Zhu, C. Wang, S. Zhang, and M. Li, “The DKU-CMRI System for the ASVspoof 2021 Challenge: Vocoder based Replay Channel Response Estimation,” in _Proc. of the Automatic Speaker Verification and Spoofing Countermeasures Challenge_ , 2021, pp. 16–21.
  * [24] W. Cai and M. Li, “A unified deep speaker embedding framework for mixed-bandwidth speech data,” in _Proc. APSIPA ASC 2021_. IEEE, 2021, pp. 1133–1138.
  * [25] P. Jax and P. Vary, “On artificial bandwidth extension of telephone speech,” _Signal Processing_ , vol. 83, no. 8, pp. 1707–1719, 2003.
  * [26] K. Zhang, Y. Ren, C. Xu, and Z. Zhao, “WSRGlow: A Glow-Based Waveform Generative Model for Audio Super-Resolution,” in _Proc. Interspeech 2021_ , 2021, pp. 1649–1653.
  * [27] J. Lee and S. Han, “NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling,” in _Proc. Interspeech 2021_ , 2021, pp. 1634–1638.
  * [28] V.-A. Nguyen, A. H. Nguyen, and A. W. Khong, “Tunet: A block-online bandwidth extension model based on transformers and self-supervised pretraining,” in _Proc. ICASSP 2022_ , 2022, pp. 161–165.
  * [29] J. S. Chung, J. Huh, S. Mun, M. Lee, H.-S. Heo, S. Choe, C. Ham, S. Jung, B.-J. Lee, and I. Han, “In Defence of Metric Learning for Speaker Recognition,” in _Proc. Interspeech 2020_ , 2020, pp. 2977–2981.
  * [30] A. Nagrani, J. S. Chung, and A. Zisserman, “VoxCeleb: A Large-Scale Speaker Identification Dataset,” in _Proc. Interspeech 2017_ , 2017, pp. 2616–2620.
  * [31] J. S. Chung, A. Nagrani, and A. Zisserman, “VoxCeleb2: Deep Speaker Recognition,” in _Proc. Interspeech 2018_ , 2018, pp. 1086–1090.
  * [32] D. Snyder, G. Chen, and D. Povey, “MUSAN: A Music, Speech, and Noise Corpus,” 2015, arXiv:1510.08484v1.
  * [33] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in _Proc. ICASSP 2017_ , 2017, pp. 5220–5224.
  * [34] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proc. ICLR 2015_ , 2015.
  * [35] X. Wang and J. Yamagishi, “A Comparative Study on Recent Neural Spoofing Countermeasures for Synthetic Speech Detection,” in _Proc. Interspeech 2021_ , 2021, pp. 4259–4263.


# BenCoref: A Multi-Domain Dataset of Nominal Phrases and Pronominal Reference
Annotations

Shadman Rohan1 Mojammel Hossain1 Mohammad Mamun Or Rashid2 Nabeel Mohammed1
1North South University 2Jahangirnagar University
shadman.rohan, mojammel.hossain<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Coreference Resolution is a well studied problem in NLP. While widely studied
for English and other resource-rich languages, research on coreference
resolution in Bengali largely remains unexplored due to the absence of
relevant datasets. Bengali, being a low-resource language, exhibits greater
morphological richness compared to English. In this article, we introduce a
new dataset, BenCoref, comprising coreference annotations for Bengali texts
gathered from four distinct domains. This relatively small dataset contains
5200 mention annotations forming 502 mention clusters within 48,569 tokens. We
describe the process of creating this dataset and report performance of
multiple models trained using BenCoref. We expect that our work provides some
valuable insights on the variations in coreference phenomena across several
domains in Bengali and encourages the development of additional resources for
Bengali. Furthermore, we found poor crosslingual performance at zero-shot
setting from English, highlighting the need for more language-specific
resources for this task. The dataset is available at 111codes used to generate
the results along with data is available at:
https://github.com/ShadmanRohan/BenCoref.

##  1 Introduction

Coreference resolution is the task of identifying all references to the same
entity in a document. This task originally started as a sub-task of
information extraction. The Message Understanding Conferences Grishman and
Sundheim (1996) first introduced three tasks, collectively referred to as
SemEval, designed to measure the deeper understanding of any information
extraction (IE) system. One of these three tasks proposed in the event was
coreferencial noun phrase identification.

The Automatic Content Extraction (ACE) Program Doddington et al. (2004) was
the first major initiative that created a large dataset with entity, event and
relation annotations. This project revealed some major complexities behind
creating such dataset. Some of the significant challenges reported by the
annotators include the coreference of generic entities, use of metonymy,
characterization of Geo-Political Entity, distinguishing certain complex
relations, and recognizing implicit vs. explicit relations.

Figure 1: BenCoref annotations with color-coded Co-reference chains.

Since then coreference resolution, anaphoric & cataphoric relation
identification, event reference detection has been studied widely. As a
result, large datasets like ACE Doddington et al. (2004), Ontonotes Pradhan et
al. (2012), WikiCoref Ghaddar and Langlais (2016), and LitBank Bamman et al.
(2020) were made public. Some datasets, like ACE Doddington et al. (2004), and
Ontonotes, expanded this task beyond English to include more languages, like
Arabic, and Chinese.

This coreference resolution task has shown potential in improving many
downstream NLP tasks like machine translation Miculicich Werlen and Popescu-
Belis (2017); Ohtani et al. (2019), literary analysis Bamman et al. (2014),
question anwering Morton (1999), text summarization Steinberger et al. (2007),
etc. However, Bengali, despite being a popular lanaguage, has seen very little
work is this direction due to lack of public datasets.

Figure 1 shows a sample from our dataset with each color representing an
unique entity. The main contributions of this work are:

  * •

We introduce a new Bengali coreference annotated dataset, consisting of 48,569
tokens collected from four diverse domains. Our dataset creation process is
shared along with the annotators’ guidelines, which we believe is the first of
its kind for Bengali coreference annotation.

  * •

We characterize the behaviour and distribution of nominal and pronominal
coreference mentions across the four domains with necessary statistics.
Furthermore, we report the performance of an end-to-end neural coreference
resolution system that was solely built using our data.

  * •

We empirically demonstrate the necessity for more language-specific datasets,
particularly for low-resource languages, by comparing our results with zero-
shot cross-lingual learning from English.

###  1.1 Related Datasets

To the best of our knowledge, no coreference dataset in Bengali exists. Most
of the works related to Bengali Sikdar et al. (2013); Senapati and Garain
(2013); Sikdar et al. (2015) uses data from ICON2011 shared task which was
never publicly shared.

Most of the major coreference datasets are in English. OntoNotes Pradhan et
al. (2012) is a well-annotated and large dataset with over 1.6M words. This
dataset does not contain any singleton mention. Later, LitBank Bamman et al.
(2020) was published that is almost 10 times larger than OntoNotes (12.3M
words).

##  2 Challenges in Bengali

One of the main challenges we faced was the absence of preexisting coreference
annotation guidelines tailored for the Bengali language. To overcome this
obstacle, we adapted the OntoNotes coreference annotation guideline to suit
our objectives. This highlighted several distinctive linguistic
characteristics of Bengali, such as zero anaphora, non-anaphoric pronouns, and
case-marking, that needs to be carefully considered when preforming co-
reference annotation in Bengali. Each of this is discussed with more details
and examples in Figure 9 in the Appendix.

Moreover, we discovered that existing annotation software is ill-equipped to
manage Bengali text, occasionally leading to inaccurate rendering and unstable
character display. This underscores the importance of advancing normalization
techniques and standardization of Bengali digital representation.

##  3 Data Domain Description

The Bengali language can be braoadly categorized into two primary literary
dialects, namely "Shadhubhasha" and "Choltibhasha." "Shadhubhasha" was
commonly used by Bangla writers and individuals in the 19th and early 20th
centuries, while "Choltibhasha" is currently the more prevalent and colloquial
dialect. This dataset contains both domains of Bengali text, with story and
novel texts sourced from copyright-free books of the 19th and 20th centuries,
and biography and descriptive texts obtained from modern sources, primarily in
"Choltibhasha." A brief description of each domain is given below:

###  3.1 Biography

A biography presents a comprehensive account of an individual’s life,
character, accomplishments, and works, spanning from birth to death or the
present time. Although the number of references per document in biographical
texts is comparable to other genres, they primarily focus on a single subject
throughout the entire narrative. Additionally, the dialect employed in
biographies in BenCoref is typically "Choltibhasha."

###  3.2 Descriptive

By descriptive text we refer to wikipedia-like articles. They cover a broad
range of subjects that span various fields, such as technology, professions,
travel, economics, and numerous related subtopics. These comprehensive texts
try to accurately portray and convey holistic information about real-world
objects or experiences.

###  3.3 Story

BenCoref is primarily composed of short stories, each with a word count of
1000 words or less, which was an arbitrary decision. These stories typically
feature 3-4 characters on average. The language used in the stories varies,
with some being exclusively in "Shadhubhasha," while others use a mix of
"Shadhubhasha" and "Choltibhasha."

###  3.4 Novel

The Bengali novels in our dataset typically consist of more than 1200 words
and feature an average of over 5 characters. These novels primarily employ
"Shadhubhasha". The next segment discusses the coreference behaviour across
each domain in more detail.

##  4 Domain Specific Coreference Behaviour Characterization

In this section, some statistics is presented to better understand the
coreference phenomenon across each domain. Each coreference cluster may refer
to different type of entities, like an object, people, location or event. An
arbitrary design choice was made to not explicitly mark the type of entity.

We start by analyzing the mean and standard deviation between mentions across
the domains. Table 1 shows that biographies and novels exhibit a low standard
deviation but have noticeably different mean distance between mentions. On the
other hand, stories and descriptive texts fall in the middle, exhibiting a
similar coreference distribution. For mentions that span more than one token,
only the first token was used for calculation.

Categories | Mean | Std. Dev
---|---|---
Novel | 29.17 | 3.70
Story | 24.10 | 8.46
Biography | 15.67 | 3.81
Descriptive | 22.35 | 5.42
Table 1:  Mean and Std. Deviation of distance between mentions in each domain.

The majority of texts in BenCoref belong to the stories domain, while the
biography domain has the smallest contribution. The distribution of mentions,
clusters, and tokens across the categories in BenCoref is presented in Figure
4.

Figure 2 depicts the distribution of cluster size across each domain. The
cluster size refers to the total number of mentions in each coreference chain.
It is worth noting that singletons were not annotated in BenCoref. The story
domain has the highest number of coreference chains with two mentions only.
Since the story domain contributes the most data to the dataset, this may be a
contributing factor to its high frequency in each cluster size. Besides story,
the descriptive domain also seems to have more larger coreference chains.

Figure 3 compares the spread of coreference chains in each domain, where the
spread refers to the token-level distance between the beginning and end of a
coreference chain. A general trend can be observed that as the size of the
coreference chain increases, its corresponding frequency decreases in each
domain.

Figure 2: Cluster size comparison between Story, Novel, Biography and wiki-
like Descriptive domain. Figure 3: Spread in BenCoref across each domain. The
spread is measured by the token level distance between the first and last
mention of an entity. Figure 4: (Right) Distribution of Clusters, Mentions,
and Tokens across the categories.

An additional "index.csv" file is included with in the dataset, which serves
as an index to all the documents included, organized by title and author. A
partial view of this file is presented in Appendix Figure 6.

##  5 Methodology

We used BnWikiSource222https://bn.wikipedia.org and BanglapediaIslam et al.
(2003) as sources of copyright-free Bengali text for our dataset. Banglapedia
was used for biographies and wiki-like descriptive texts. The dataset creation
process is discussed in detail in the following paragraphs.

###  5.1 Annotation Phase

The WebAnno annotator Eckart de Castilho et al. (2016) was the chosen tool for
annotation. To accommodate WebAnno’s limited capacity to work with large
texts, the articles were partitioned according to the Table 2. Each partition
ends in a complete sentence and any incomplete portion of a sentence were
moved to the next fragment. The partition size was chosen arbitrarily to
reduce the number of data fragments. In Appendix Figure 5, a screenshot of the
WebAnno interface used during this phase is displayed. A post annotation
sample is provided in Figure 7 in Appendix from the Biography domain.

Tokens | Partitions
---|---
<699 | 1
<1000 | 2
>1000 | 3
Table 2:  Documents with greater than 699 tokens and less than 1000 tokens
were divided into 2 parts and the ones with more than 1000 tokens were divided
into 3.

Since there is no existing guideline for coreference annotation, the
annotators were initially instructed to annotate the noun phrases and its
coreferences, which were predominantly pronouns. The primary noun phrase
references were tagged as "entity" and their corresponding coreferences were
tagged as "ref". While determining what forms an entity is an important
linguistic problem, it is not the primary challenge we are trying to address
in our work. Annotators were free to mark any token or span that they
considered an entity. After the annotation phase was completed, the data was
exported and the character-level annotations were converted to token-level
annotations. For every exceptional cases encountered, a new rule was
established and enforced during further annotation of the dataset. The rules
are further discussed in the next section.

###  5.2 Annotation Strategy/Guideline

This coreference annotation guide (refer to A in the Appendix) was prepared
concurrently with the annotation phase to ensure consistency throughout the
annotation process. We mirrored the overall structure of the OntoNotes
annotation guidelines, tailoring them to our specific use case.

Initially, we did not impose any specific restrictions on the definition of an
entity during the annotation process. The annotators were instructed to
annotate any span they deemed as an entity. However, this approach resulted in
an annotator bias, with a strong focus on nominal and pronominal mentions.
Subsequently, we made the decision to prioritize and concentrate solely on
these types of mentions.

Furthermore, as part of our design decision, we chose to not tag singleton
mentions. Consequently, any singletons were removed during the post-annotation
processing phase.

###  5.3 Annotation Criteria:

The general rule used for annotation is to annotate mentions in any form,
including nested mentions or those referring to multiple entities. The
characterization of mention and coreference link types was conducted after
annotating the entire dataset. Annotating coreference link types was kept
optional due to the significant training required for the task. This strategy
was followed the accelerate the annotation process.

The rules with corresponding examples are illustrated in a more detailed
manner in Figure 10 in the Appendix. Furthermore, the coreference link types
have been categorized into two groups, namely identical and apposite, and they
have been discussed in detail in 11 and 12 in the Appendix. However, the task
of annotating coreference link types is currently pending and will be
addressed in future work.

While this guideline is incomplete and limited in scope, it can play an
impotant role in encouraging the next generation of coreference datasets in
Bengali. The OntoNotes coreference guidelinegui (2007) is currently in its 7th
edition which is a strong indication that the first attempt on making a such
guideline would be imperfect and will require further revisions. It may take
several iterations before we can have a robust guideline for coreference
annotation in Bengali.

###  5.4 Inter-Annotator Agreement

The OntoNotes strategy was roughly employed to assess interannotator agreement
in this work. Specifically, two annotators independently annotated the
documents, and only in cases of disagreement, a third annotator was consulted
to arrive at a final decision. These ultimate annotations were deemed as the
gold standard annotations.

Based on the adjudicated version as the ground truth, the individual
annotations in our dataset achieved an average MUC score of 78.3 on the
combined dataset. while the combined inter-annotator MUC score was 67.6.

However, it is important to acknowledge that the process of resolving
disagreements was not adequately documented and will be addressed in greater
detail in future endeavors.

##  6 Experiments

We took an end-to-end neural network based modeling approach. The following
section discusses the algorithm, followed by the experimental setup,
evaluation strategy and analysis of results.

We used the 300-dimensional Fasttext and Glove embeddings Grave et al. (2018)
as words representations. To generate contexual representations the embeddings
were passed through a bi-directional LSTM Hochreiter and Schmidhuber (1997)
for some experiments and a variation of the popular transformer-based Vaswani
et al. (2017) pretrained model, BERTDevlin et al. (2019), for other
experiments. For the task of coreference resolution, the contextual
representations from these base models were passed on to a span ranking model-
head, originally proposed in Lee et al. (2018). For the crosslingual
experiment, a multilingual BERT was finetuned on the OntoNotes dataset.

For hyperparameter optimization, we tuned the maximum number of words in a
span(s), maximum number of antecedents per span(a), and coref layer depth(CL).

###  6.1 Experimental Setup

The data was separated into train and dev set on a ratio of 95% by 5%. An
additional test set was carefully prepared, completely disjoint from the train
and dev set, that contains 37 documents. An overview of the dataset given in
Table 3

| categories | documents | mentions | clusters
---|---|---|---|---
train | biography | 17 | 421 | 38
\+ dev | descriptive | 36 | 1157 | 108
| novel | 13 | 601 | 78
| story | 56 | 3021 | 278
test | biography | 10 | 303 | 22
| descriptive | 9 | 290 | 33
| novel | 3 | 191 | 15
| story | 15 | 697 | 53
Table 3: Dataset distribution

For evaluating our system, we used the CONLL-2012 official evaluation scripts
which calculates four metrics: Identification of Mentions, MUC, B3 and CEAF.
The following section analyzes the performance of our model.

###  6.2 Results and Analysis

category model parameters pre. rec. f1 c2f+Glove s=30, a=50, CL=2 93.83 65.34
77.04 biography c2f+Fasttext s=20, a=50, CL=2 96.51 64.02 76.98 BERT-base
s=30, a=50, CL=2 94.22 86.13 90.00 M-BERT(Zero-Shot) s=30, a=50, CL=2 7.14
4.62 5.61 c2f+Glove s=30, a=50, CL=2 73.78 58.96 65.55 story c2f+Fasttext
s=20, a=50, CL=2 74.80 54.08 62.78 BERT-base s=30, a=50, CL=2 83.91 65.85
73.79 M-BERT(Zero-Shot) s=30, a=50, CL=2 7.40 3.73 4.96 c2f+Glove s=30, a=50,
CL=2 78.00 40.83 53.60 novel c2f+Fasttext s=20, a=50, CL=2 87.50 43.97 58.53
BERT-base s=30, a=50, CL=2 85.41 64.39 73.43 M-BERT(Zero-Shot) s=30, a=50,
CL=2 8.51 4.18 5.61 c2f+Glove s=30, a=50, CL=2 66.39 27.93 39.32 descriptive
c2f+Fasttext s=20, a=50, CL=2 72.16 24.13 36.17 BERT-base s=30, a=50, CL=2
82.95 50.34 62.66 M-BERT(Zero-Shot) s=30, a=50, CL=2 7.47 5.51 6.34

Table 4: Identification of mentions

$B^{3}$ MUC $CEAF_{\phi 4}$ Avg category parameters Pre. Rec. F1 Pre. Rec. F1
Pre. Rec. f1 Pre. Rec. F1 c2f + Glove s=30, a=50, CL=2 84.52 44.74 58.51 92.26
64.41 76.05 55.33 40.24 46.60 77.37 49.80 60.39 biography c2f + Fasttext s=20,
a=50, CL=2 89.09 43.99 58.90 95.74 64.05 76.75 61.24 36.19 45.49 82.02 48.08
60.38 BERT-base s=30, a=50, CL=2 85.37 73.59 79.04 93.79 86.12 89.79 57.48
49.64 53.27 78.88 69.78 74.03 M-BERT(Zero-Shot) s=30, a=50, CL=2 4.28 0.25
0.46 0.67 0.35 0.46 1.22 2.93 1.72 2.06 1.18 0.88 c2f + Glove s=30, a=50, CL=2
46.92 23.95 31.72 63.41 44.40 52.23 20.24 36.99 26.16 43.52 35.11 36.70 story
c2f + Fasttext s=20, a=50, CL=2 47.31 22.80 30.77 65.23 42.54 51.50 23.49
34.02 27.79 45.34 33.12 36.69 BERT-base s=30, a=50, CL=2 54.64 31.62 40.06
74.46 53.88 62.52 28.80 40.23 33.57 52.63 41.91 45.38 M-BERT(Zero-Shot) s=30,
a=50, CL=2 2.32 0.25 0.45 1.42 0.62 0.86 2.00 2.59 2.26 1.91 1.15 1.19 c2f +
Glove s=30, a=50, CL=2 49.87 7.98 13.77 59.45 25.00 35.20 16.37 26.61 20.27
41.90 19.86 23.08 novel c2f + Fasttext s=20, a=50, CL=2 60.30 10.45 17.82
72.72 31.81 44.26 23.36 27.74 25.37 52.13 23.33 29.15 BERT-base s=30, a=50,
CL=2 43.55 33.93 38.14 71.31 52.27 60.32 34.98 32.80 33.85 49.95 39.67 44.10
M-BERT(Zero-Shot) s=30, a=50, CL=2 3.54 0.25 0.47 2.66 1.13 1.59 1.90 2.49
2.15 2.70 1.29 1.40 c2f + Glove s=30, a=50, CL=2 48.33 11.91 19.12 58.24 20.62
30.45 31.16 26.83 28.83 45.91 19.79 26.13 descriptive c2f + Fasttext s=20,
a=50, CL=2 58.32 9.74 16.70 66.66 18.67 29.17 29.98 20.81 24.57 51.65 16.41
23.48 BERT-base s=30, a=50, CL=2 62.81 26.88 37.65 76.62 45.91 57.42 46.12
28.18 34.99 61.85 33.66 43.35 M-BERT(Zero-Shot) s=30, a=50, CL=2 2.01 0.56
0.88 1.16 0.77 0.93 2.30 3.00 2.60 1.82 1.44 1.47

Table 5: Performance on test data. The main evaluation metric is the average
F1 score of $MUC$, $B^{3}$, and $CEAF_{\phi 4}$. The best scores are
highlighted.

The performance of the model was reasonable given the size of our dataset. As
neural networks tend to achieve optimal performance with larger datasets, we
hypothesize that our results could be enhanced by expanding our dataset. The
model demonstrated good performance in identifying individual mentions, as
evidenced by the scores presented in Table 4. However, we observed a decrease
in performance during the second phase of clustering the mentions, as shown in
Table 5. This highlights the challenge of accurately identifying coreference
clusters, particularly in languages with complex sentence structures and a
high degree of lexical ambiguity. Further innovation is needed to address
these challenges and improve the overall performance of coreference resolution
models.

Upon closer inspection one recurring problem was discovered. The model failed
to do basic common sense reasoning on long coreference clusters, often
breaking it up into several clusters. As demonstrated in Figure 8 in Appendix,
the model failed to merge clusters 0 and 1, which should have been a single
cluster.

Furthermore, it can be observed that the coreference resolution model performs
significantly better on the biography domain as compared to other domains. The
relatively low mean and standard deviation of the distance between mentions
reported in Table 1 may have contributed to this result. However, despite
forming the major portion of the dataset, the story domain did not show any
significant improvement. The high standard deviation in distance between
mentions reported in Table 1 for the story domain may have contributed to this
lack of improvement. Qualitative analysis is needed to investigate the
underlying causes of this performance gap.

The zero-shot crosslingual experiment demostrated that coreference knowledge
doesn’t easily transfer through multilingual training. This clearly
demonstrates the need for language specific datasets. Some studies Novák and
Žabokrtskỳ (2014) report developing projection techniques to improve
crosslingual coreference resolution. There maybe scope for further work in
this direction.

##  7 Conclusion

This paper presented BenCoref, the first publicly available dataset of
coreference annotations in Bengali. The creation process and annotation
guidelines were described in detail to facilitate future work in this area. We
then used the dataset to develop an end-to-end coreference resolution system
and reported its performance across different domains. Our findings indicate
that a lower mean and standard deviation of token-distance between mentions
may lead to better results, but further experiments on other datasets are
needed to confirm this hypothesis. We also observed a higher tendency for
breakage in longer coreference chains.

Our zero-shot cross-lingual experiment demonstrated that coreference knowledge
does not easily transfer through multilingual training, highlighting the
importance of language-specific datasets. While some studies Novák and
Žabokrtskỳ (2014) have reported success in developing projection techniques to
improve cross-lingual coreference resolution, further research is required to
explore this area.

## Acknowledgements

We would like to express our sincere gratitude to everyone who contributed to
this research. We would also like to thank our colleagues and collaborators
for their valuable insights and feedback throughout the project. We are
especially grateful to our annotators for their hard work and dedication to
ensure the quality of this dataset. We also appreciate the feedback and
suggestions from the anonymous reviewers, which helped to improve the quality
of this manuscript. No financial assistance was provided by any organization
for the completion of this project.

## References

  * gui (2007) 2007\.  _Ontonotes English Coreference Guidelines._
  * Bamman et al. (2020) David Bamman, Olivia Lewke, and Anya Mansoor. 2020.  An annotated dataset of coreference in English literature.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 44–54, Marseille, France. European Language Resources Association.
  * Bamman et al. (2014) David Bamman, Ted Underwood, and Noah A. Smith. 2014.  A Bayesian mixed effects model of literary character.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 370–379, Baltimore, Maryland. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Doddington et al. (2004) George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004.  The automatic content extraction (ACE) program – tasks, data, and evaluation.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)_ , Lisbon, Portugal. European Language Resources Association (ELRA).
  * Eckart de Castilho et al. (2016) Richard Eckart de Castilho, Éva Mújdricza-Maydt, Seid Muhie Yimam, Silvana Hartmann, Iryna Gurevych, Anette Frank, and Chris Biemann. 2016.  A web-based tool for the integrated annotation of semantic and syntactic structures.  In _Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH)_ , pages 76–84, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Ghaddar and Langlais (2016) Abbas Ghaddar and Philippe Langlais. 2016.  Wikicoref: An english coreference-annotated corpus of wikipedia articles.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 136–142.
  * Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018.  Learning word vectors for 157 languages.  In _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Grishman and Sundheim (1996) Ralph Grishman and Beth M Sundheim. 1996.  Message understanding conference-6: A brief history.  In _COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Lstm can solve hard long time lag problems.  _Advances in neural information processing systems_ , pages 473–479.
  * Islam et al. (2003) Sirajul Islam et al. 2003.  Banglapedia.  _National Encyclopedia, Asiatic Society of Bangladesh, Dhaka_.
  * Lee et al. (2018) Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018.  Higher-order coreference resolution with coarse-to-fine inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 687–692, New Orleans, Louisiana. Association for Computational Linguistics.
  * Miculicich Werlen and Popescu-Belis (2017) Lesly Miculicich Werlen and Andrei Popescu-Belis. 2017.  Using coreference links to improve spanish-to-english machine translation.  Technical report, Idiap.
  * Morton (1999) Thomas S Morton. 1999.  Using coreference for question answering.  In _Coreference and Its Applications_.
  * Novák and Žabokrtskỳ (2014) Michal Novák and Zdeněk Žabokrtskỳ. 2014.  Cross-lingual coreference resolution of pronouns.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 14–24.
  * Ohtani et al. (2019) Takumi Ohtani, Hidetaka Kamigaito, Masaaki Nagata, and Manabu Okumura. 2019.  Context-aware neural machine translation with coreference information.  In _Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019)_ , pages 45–50.
  * Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012.  Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes.  In _Joint Conference on EMNLP and CoNLL-Shared Task_ , pages 1–40.
  * Senapati and Garain (2013) Apurbalal Senapati and Utpal Garain. 2013.  Guitar-based pronominal anaphora resolution in bengali.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 126–130.
  * Sikdar et al. (2013) Utpal Kumar Sikdar, Asif Ekbal, Sriparna Saha, Olga Uryupina, and Massimo Poesio. 2013.  Adapting a state-of-the-art anaphora resolution system for resource-poor language.  In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_ , pages 815–821.
  * Sikdar et al. (2015) Utpal Kumar Sikdar, Asif Ekbal, Sriparna Saha, Olga Uryupina, and Massimo Poesio. 2015.  Differential evolution-based feature selection technique for anaphora resolution.  _Soft Computing_ , 19(8):2149–2161.
  * Steinberger et al. (2007) Josef Steinberger, Massimo Poesio, Mijail A Kabadjov, and Karel Ježek. 2007\.  Two uses of anaphora resolution in summarization.  _Information Processing & Management_, 43(6):1663–1680.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.

##  Appendix A Appendices

$B^{3}$ MUC $CEAF_{\phi 4}$ Avg Category Parameters Pre. Rec. F1 Pre. Rec. F1
Pre. Rec. F1 Pre. Rec. F1 c2f + Fasttext s=30, a=250, CL=2 93.43 45.94 61.59
97.88 65.83 78.72 65.62 41.76 51.04 85.64 51.18 63.78 Biography c2f + Fasttext
s=20, a=50, CL=3 91.63 45.20 60.54 96.84 65.48 78.13 65.87 41.92 51.23 84.78
50.87 63.30 c2f + Fasttext s=10, a=50, CL=3 90.98 47.81 62.68 96.92 67.25
79.41 77.44 42.24 54.66 88.54 52.44 65.58 c2f + Fasttext s=30, a=250, CL=2
48.07 22.95 31.07 65.78 38.81 48.82 21.98 33.73 26.62 45.28 31.83 35.50 Story
c2f + Fasttext s=20, a=50, CL=3 53.08 21.70 30.81 67.02 38.50 48.91 22.17
34.02 26.84 47.42 31.41 35.52 c2f + Fasttext s=10, a=50, CL=3 53.82 19.35
28.47 66.76 34.62 45.60 21.92 34.77 26.89 47.50 29.58 33.65 c2f + Fasttext
s=30, a=250, CL=2 47.06 7.32 12.67 61.03 26.70 37.15 16.04 23.06 18.92 41.38
19.03 22.91 Novel c2f + Fasttext s=20, a=50, CL=3 65.10 8.54 15.10 71.42 25.56
37.65 24.96 28.08 26.42 53.83 20.73 26.39 c2f + Fasttext s=10, a=50, CL=3
57.71 5.82 10.57 63.79 21.02 31.62 17.03 23.42 19.72 46.18 16.75 20.64 c2f +
Fasttext s=30, a=250, CL=2 56.43 9.88 16.82 61.84 18.28 28.22 25.72 20.01
22.51 48.00 16.06 22.52 Descriptive c2f + Fasttext s=20, a=50, CL=3 53.78 7.64
13.37 58.92 12.84 21.08 28.76 19.17 23.01 47.15 13.22 19.15 c2f + Fasttext
s=10, a=50, CL=3 52.41 7.46 13.06 58.06 14.00 22.57 24.47 19.03 21.41 44.98
13.50 19.01

Table 2: Some additional results on the Model’s performance.

Category Parameters Pre. Rec. F1 c2f+Fasttext s=30, a=250, CL=2 98.52 66.00
79.05 Biography c2f+Fasttext s=20, a=50, CL=3 97.54 65.67 78.50 c2f+Fasttext
s=10, a=50, CL=3 98.52 66.00 79.05 c2f+Fasttext s=30, a=250, CL=2 74.20 49.92
59.69 Story c2f+Fasttext s=20, a=50, CL=3 75.16 49.49 59.68 c2f+Fasttext s=10,
a=50, CL=3 76.52 46.77 58.05 c2f+Fasttext s=30, a=250, CL=2 79.00 41.36 54.29
Novel c2f+Fasttext s=20, a=50, CL=3 90.12 38.21 53.67 c2f+Fasttext s=10, a=50,
CL=3 83.75 35.07 49.44 c2f+Fasttext s=30, a=250, CL=2 74.03 26.55 39.08
Descriptive c2f+Fasttext s=20, a=50, CL=3 70.00 19.31 30.27 c2f+Fasttext s=10,
a=50, CL=3 68.88 21.37 32.63

Table 7:  Additional identification of mention results Figure 5: A screenshot
from WebAnnoEckart de Castilho et al. (2016) during annotation phase. In this
example, the highlighted words are marked as mentions and same color indicate
the mentions belong to the same cluster. A colored line joins the highlighted
words creating a chain forming a single cluster. Figure 6: The supplementary
datafile index.csv contains an index to all the datapoints Figure 7: Biography
document Figure 8: BERT-base model’s prediction on a biography document Figure
9: This highlights few key considerations when annotating coreference in
Bengali. Figure 10: BenCoref Annotation Guideline with examples. Figure 11:
Identical Reference Types Figure 12: Appositive Reference Types