# Efficient CNN with uncorrelated Bag of Features pooling ††thanks: This work
was supported by NSF-Business Finland Center for Visual and Decision
Informatics (CVDI) project AMALIA.

Firas Laakom  Faculty of Information Technology and Communication Sciences
Tampere University
Tampere, Finland
<EMAIL_ADDRESS>Jenni Raitoharju  Faculty of Information Technology
University of Jyväskylä
Jyväskylä, Finland
<EMAIL_ADDRESS>Alexandros Iosifidis  Department of Electrical and
Computer Engineering
Aarhus University
Aarhus, Denmark
<EMAIL_ADDRESS>Moncef Gabbouj  Faculty of Information Technology and
Communication Sciences
Tampere University
Tampere, Finland
<EMAIL_ADDRESS>
###### Abstract

Despite the superior performance of CNN, deploying them on low computational
power devices is still limited as they are typically computationally
expensive. One key cause of the high complexity is the connection between the
convolution layers and the fully connected layers, which typically requires a
high number of parameters. To alleviate this issue, Bag of Features (BoF)
pooling has been recently proposed. BoF learns a dictionary, that is used to
compile a histogram representation of the input. In this paper, we propose an
approach that builds on top of BoF pooling to boost its efficiency by ensuring
that the items of the learned dictionary are non-redundant. We propose an
additional loss term, based on the pair-wise correlation of the items of the
dictionary, which complements the standard loss to explicitly regularize the
model to learn a more diverse and rich dictionary. The proposed strategy
yields an efficient variant of BoF and further boosts its performance, without
any additional parameters.

###### Index Terms:

deep learning, CNN, diversity, bag of features pooling

##  I Introduction

In recent years, Convolutional Neural Networks have significantly advanced
many tasks in the computer vision field due to their ability to learn ‘good’
feature representation in an end-to-end manner [1, 2]. However, despite their
superior performance across multiple tasks, e.g., image classification [3, 4,
5], object detection [6, 7, 8], anomaly detection [9, 10, 11, 12], deploying
CNN-based solutions on low computational power devices, such as mobile phones,
is still limited as most of the high-accuracy models are typically
computationally expensive [1, 13, 14]. Thus, they are inefficient in terms of
time and energy consumption [15]. To alleviate this issue, several approaches
have been proposed to reduce the number of parameters required by a CNN model
[16, 17, 18, 19, 20].

The standard CNN model is usually composed of two parts: The first part is
formed of convolutional layers typically coupled with max-pooling operations.
Then, in the second part, fully connected layers are connected directly to a
flattened version of the last convolutional layer output. This connection
dramatically increases the total number of parameters, as convolutional layer
outputs usually have high dimensionality. Recent approaches mitigate this
problem by developing better mechanisms for connecting both parts, e.g.,
global average pooling and Bag of Features (BoF) pooling.

Figure 1: An illustration of a simple BoF-based CNN model. From left to right:
Input image, convolutional layer, BoF layer containing a dictionary and
outputting a histogram, and fully connected layers.

BoF pooling is a neural extension [19, 18] of the famous Bag-of-Visual Words
[21, 22, 23, 24, 25, 26]. An illustration of a simple BoF-based CNN model is
presented in Figure 1. Based on the convolutional output, BoF pooling learns a
codebook (dictionary) and outputs a shallow histogram representation of the
input. The items of the dictionary are optimized in an end-to-end manner
during the standard back-propagation. This yields powerful and efficient
models, that achieve a high performance with a low computational footprint.
Recently, CNN models, based on BoF pooling, have been used to solve multiple
tasks [27, 28, 29, 30, 31, 32], such as action recognition [30], information
retrieval [33], and illumination estimation [34].

In this paper, we propose an approach that builds on top of BoF pooling to
boost its efficiency by ensuring that the items of the learned dictionary are
non-redundant. Forcing an uncorrelated structure on the codebook yields a more
powerful model which can achieve a high performance with minimal dictionary
size. Diversity in deep learning context has been shown to lead to better
results [35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48]. To this end,
we propose to augment the loss of the model to penalize pair-wise correlations
between the items of the codebook. The proposed technique requires no
additional parameters and can be incorporated in any BoF-based CNN model to
boost the performance of the CNN model.

The contributions of this paper can be summarized as follows:

  * •

We propose a scheme to avoid redundant items in the dictionary learned by the
BoF.

  * •

We propose to augment the CNN-loss to explicitly penalize the pair-wise
correlations between codebook items and learn rich compressed dictionary.

  * •

The proposed regularizer acts as an unsupervised regularizer on top of the BoF
pooling layer and can be integrated into any BoF-based CNN model in a plug-
and-play manner.

  * •

The proposed approach is evaluated with three datasets. The results show a
consistent performance boost compared to the standard approach.

The rest of this paper is organized as follows. First, provide a brief
overview of BoF in Section II. In Section III, we present the proposed
approach. In Section IV, we empirically evaluate the performance of our method
on three different datasets. We conclude the paper in Section V.

##  II Bag-of-Features Pooling

In this section, we briefly describe the BoF pooling mechanism. BoF [18, 19]
has been incorporated in a variety of applications and often led to superior
results [33, 34, 31]. The BoF pooling is parameterized with a dictionary.
Given an input, i.e., the output maps of the last convolutional layer, a
histogram representation is compiled based on the dictionary. In the training
phase, the items of the dictionary are optimized with the traditional back-
propagation. The size of the dictionary is a hyper-parameter that can be
adjusted with a validation set to avoid over-fitting.

BoF pooling is formed using two inner layers: a Radial Basis Function (RBF)
layer that measures the similarity of the input features to the RBF centers
and an accumulation layer that builds a histogram of the quantized feature
vectors. Formally, let X be the input image and
$\rho(\textbf{X})\in\mathbb{R}^{D\times P}$ the output of the convolutional
layer, the RBF layer outputs a sequence of of quantized representations:

$\Psi=[\psi_{1},\psi_{2},\cdots,\psi_{P}]\in\mathbb{R}^{K\times P},$

where $\psi_{i}$ is the representation corresponding to the $i^{th}$ feature,
i.e.,

$\psi_{i}=[\psi_{i,1},\cdots,\psi_{i,K}].$

The output of the $i^{th}$ RBF unit is as follows:

$\psi_{n,i}=\frac{\exp(-||\rho(\textbf{X})_{n}-\textbf{c}_{i}||/m_{i})}{\sum_{j}\exp(-||\rho(\textbf{X})_{n}-\textbf{c}_{j}||/m_{j})},$
(1)

where $\textbf{c}_{i}$ is the center of the i-th RBF neuron, and $m_{i}$ is a
scaling factor. The outputs of the $P$ RBF neurons are accumulated in the next
layer in order to obtain the final representation $\Phi$ of each image:

$\Phi=\frac{1}{P}\sum_{j}\psi_{j}.$ (2)

To summarize, BoF receives as input a feature representation, usually in high
dimension, and quantizes it into a fixed-size shallow histogram
representation. The quantization is based on the inner dictionary,
$\\{\textbf{c}_{1},\cdots,\textbf{c}_{K}\\}$, which can be learned jointly
with the rest of the parameters in an end-to-end manner.

##  III Our Approach

BoF pooling layer is a key technique that can be used in CNNs to construct
powerful models with a low computational cost. The BoF relies on a dictionary,
learned during the training, to compute its shallow output. In this paper, we
propose an approach that builds on top of BoF pooling to boost its efficiency
by explicitly forcing the items of the learned dictionary to be distinct and
non-redundant. We propose a simple additional regularizer that penalizes the
similarities between the codebook items. This can further boost the
performance of the model, without any additional parameters.

Figure 2: An illustration on how the BoF-based CNN model loss is computed
using our approach. The standard loss can be least squares or cross entropy
and the similarity loss corresponds to the second term in (5).

The dictionary learned the BoF layer plays a critical role in the global
performance of the model. Intuitively, Learning a diverse and rich dictionary
yields in a robust codebook and increases the efficiency of the global model.
Given a CNN model containing a BoF pooling layer with an inner dictionary
$\\{\textbf{c}_{1},\cdots,\textbf{c}_{K}\\}$ of size $K$, the similarity $SIM$
between two elements $\textbf{c}_{i}$ and $\textbf{c}_{j}$ of this dictionary
can be measured with the squared correlation:

$SIM(\textbf{c}_{i},\textbf{c}_{j})=\Big{(}corr(\textbf{c}_{i},\textbf{c}_{j}))\Big{)}^{2},$
(3)

where $corr(\cdot,\cdot)$ is the correlation operator.

Intuitively, $SIM$ measures how similar two items are. Our goal is to
regularize the similarities between the elements of the dictionary. So, the
global similarity regularizer can be computed as the the sum of the pair-wise
similarities, i.e.,

$\sum_{i\neq j}SIM(\textbf{c}_{i},\textbf{c}_{j}).$ (4)

Given the original loss $L$, e.g., least squares or cross entropy, we propose
to regularize it as follows:

$L_{new}\triangleq L+\beta\sum_{i\neq j}SIM(\textbf{c}_{i},\textbf{c}_{j}),$
(5)

where $L_{new}$ is the augmented loss and $\beta$ is a hyper-parameter
employed to control the contribution of the supplementary regularizer in the
global loss of the model. The computation of the total loss is illustrated in
Figure 2.

Setting $\beta=0$ corresponds to the standard BoF case, while a higher $\beta$
yields a loss dominated by the regularizer. In the training phase, at each
step in the back-propagation, the gradient of the loss w.r.t. the parameters
is computed. The additional term depends only on the elements of the
dictionary, i.e., $\\{\textbf{c}_{1},\cdots,\textbf{c}_{K}\\}$, of the BoF
layer. Thus, the gradient of all the model parameters, except
$\\{\textbf{c}_{1},\cdots,\textbf{c}_{K}\\}$, remains the same as in the
standard BoF case. For each of dictionary parameters $\textbf{c}_{i}$, we have
an additional feedback term equal to

$\beta\dfrac{\partial\sum_{i\neq
j}SIM(\textbf{c}_{i},\textbf{c}_{j})}{\partial\textbf{c}_{i}},$

which encourages different elements within this dictionary to be distinct.

The proposed approach affects only the training loss and does not require any
additional parameters. It can be integrated in a plug-and-play manner in any
BoF-based model to improve performance. Intuitively, the proposed schema acts
as a penalty on top of the learned dictionary to provide supplementary
feedback in the training phase to lessen the correlations of the codebook’s
items. By explicitly forcing the BoF layer to learn a diverse and rich
dictionary, we can increase the model efficiency and reach a high performance
with a minimal number of parameters.

##  IV Experimental Results

In this Section, we present the empirical results of the proposed approach
along with the competing methods.

###  IV-A Experiment Setup

####  IV-A1 Datasets

We evaluate the performance of our approach using three different dataset:

  * •

MNIST [49] is a dataset of $28\times 28$ images from 10 classes. It contains
50,000 samples for training and 10,000 for testing.

  * •

fashionMNIST [50] is a clothes dataset containing $28\times 28$ images from 10
classes. It has in total 50,000 samples for training and 10,000 for testing.

  * •

CIFAR10 [51] is an RGB image dataset containing $32\times 32$ images from a
total of 10 distinct classes. It has a total of 50,000 and 10,000 samples for
training and testing, respectively.

####  IV-A2 Training & Testing

In all our experiments, we hold $20\%$ of the training data for validation and
hyper-parameter selection. We also experiment with different values for the
number of filters in the last convolutional layer. The full topology of the
CNN models used in MNIST/fashionMNIST and CIFAR10 experiments are reported in
Table I and Table II, respectively.

For MNIST and fashionMNIST experiments, all the models are trained for 50
epochs using Adam [52] regularizer with a 0.001 learning rate and a batch-size
of 128. For CIFAR10 experiments, all the models are trained for 200 epochs
with standard data augmentation [4] using Adam regularizer with a 0.0001
learning rate and a batch-size of 128.

We report the competitive results of the different pooling strategies, namely
global max pooling (GMP) [15], global average pooling (GAP) [15], BoF [18,
19], and our approach. The size of the codebook is a hyperparameter for both
BoF and our approach. It is optimized with the validation set from
$\\{8,16,32,64,128\\}$ for MNIST and fashionMNIST and from
$\\{32,64,128,256\\}$ for CIFAR10. The hyper-parameter $\beta$ in Eq. (5),
used for controlling the contribution of the proposed regularizer in the
global loss, is selected from $\\{0.1,0.01,0.001,0.0001\\}$ using the
validation set in all experiments.

Input layer
---
$3\times 3\times 32$ \- Relu
$3\times 3\times 32$ \- Relu
$2\times 2$ max pooling layer
$3\times 3\times C$ \- Relu
Pooling strategy
Dropout (0.2)
512-Fully connected - Relu
Dropout (0.2)
10-fully connected
softmax layer
TABLE I: Topology used for MNIST and fashionMNIST experiments. We experiment
with different values of C, i.e., the number of filters in the last
convolutional layer. Pooling strategy refers to the used method, e.g., global
max pooling or BoF. Input layer
---
$3\times 3\times 128$ \- Relu
$3\times 3\times 128$ \- Relu
$2\times 2$ max pooling layer
$3\times 3\times 64$ \- Relu
$3\times 3\times 64$ \- Relu
$2\times 2$ max pooling layer
$3\times 3\times C$ \- Relu
Pooling strategy
Dropout (0.2)
512-Fully connected - Relu
Dropout (0.2)
10-fully connected
softmax layer
TABLE II: Topology used for CIFAR10 experiments. We experiment with different values of C, i.e., the number of filters in the last convolutional layer. Pooling strategy refers to the used method, e.g., global max pooling or BoF. method | 16 filters | 32 filters | 64 filters | 128 filters
---|---|---|---|---
GMP | 3.63 $\pm$ 0.31 | 1.97 $\pm$ 0.20 | 1.39 $\pm$ 0.07 | 1.09 $\pm$ 0.08
GAP | 4.67 $\pm$ 1.17 | 2.01 $\pm$ 0.09 | 1.31 $\pm$ 0.05 | 1.06 $\pm$ 0.02
BoF | 1.03 $\pm$ 0.08 | 0.97 $\pm$ 0.11 | 1.00 $\pm$ 0.08 | 1.03 $\pm$ 0.06
BoF (ours) | 1.00 $\pm$ 0.06 | 0.98 $\pm$0.06 | 0.87 $\pm$0.10 | 0.98 $\pm$ 0.08
TABLE III: Average error rates and standard deviation of different approaches
for different number of filters in the last convolutional layer on the MNIST
dataset. Results are averaged over 5 random seeds. Top results for each
approach are in bold and best global result is underlined.

###  IV-B Empirical Results

In Table III and Table IV, we report the average error rates and standard
deviations for the different filter sizes, i.e., C in Table I, on MNIST and
fashionMNIST datasets, respectively. Compared to standard pooling approaches,
i.e., GMP and GAP, we note that both variants of BoF consistently yield a
better performance. For the 16 filter case, for example, GMP and GAP reach
$3.63\%$ and $4.67\%$ errors on MNIST, respectively, whereas standard BoF and
our variant of BoF reach $1.03$ and $1.00\%$ for the same case, respectively.

method | 16 filters | 32 filters | 64 filters | 128 filters
---|---|---|---|---
GMP | 14.94 $\pm$ 0.70 | 12.13 $\pm$ 0.30 | 10.46 $\pm$ 0.18 | 9.48 $\pm$ 0.12
GAP | 15.09 $\pm$ 0.19 | 12.30 $\pm$ 0.20 | 10.91 $\pm$ 0.21 | 9.97 $\pm$ 0.06
BoF | 9.55 $\pm$ 0.29 | 9.44 $\pm$ 0.25 | 9.04 $\pm$ 0.22 | 9.02 $\pm$ 0.15
BoF (ours) | 9.52 $\pm$ 0.29 | 9.14 $\pm$ 0.12 | 8.98 $\pm$ 0.18 | 8.77 $\pm$ 0.22
TABLE IV: Average error rates and standard deviation of different approaches for different number of filters in the last convolutional layer on the fashionMNIST dataset. Results are averaged over 5 random seeds. Top results for each approach are in bold and best global result is underlined.  method | 32 filters | 64 filters | 128 filters
---|---|---|---
GMP | 22.31 $\pm$ 0.48 | 20.33 $\pm$ 0.67 | 18.80 $\pm$ 1.03
GAP | 20.94 $\pm$ 0.53 | 20.08 $\pm$ 0.99 | 17.61 $\pm$ 0.33
BoF | 17.15 $\pm$ 0.58 | 17.05 $\pm$ 0.11 | 16.10 $\pm$ 0.20
BoF (ours) | 17.21 $\pm$ 0.71 | 16.57 $\pm$ 0.25 | 15.93 $\pm$ 0.11
TABLE V: Average error rates and standard deviation of different approaches
for different number of filters in the last convolutional layer on the CIFAR10
dataset. Results are averaged over three random seeds.

For the fashionMNIST dataset with a 16-filter model, using BoF reduces the
error rates by more than $5\%$. Compared to the standard BoF, we note that
penalizing correlations between the items of the dictionary yields in a
consistently better performance in most cases. For example, for fashionMNIST
using a 128-filters model, our approach reaches only $8.77\%$ error rate
compared to $9.02\%$ reached by BoF.

For MNIST dataset, as shown in Table III, the best performances reached by GMP
and GAP pooling are $1.09\%$ and $1.06\%$, respectively. In both cases, it is
achieved by the 128-filters model. We note that our approach requires only 16
filters to achieve a better performance ($1.00\%$). This finding is also in
agreement with the results on fashionMNIST dataset in Table IV. In this case,
our approach with only 16 filters achieves better results compared to the best
GAP case and only 32 filters to achieve better results compared to the best
GMP case. The best performance both on MNIST and fashionMNIST is achieved by
our approach with the 64-filters and 128-filters model, respectively.

In Table V, we report the results of the different approaches on CIFAR10
dataset with three different filter sizes in the final convolutional layer,
namely 32, 64 and 128 filters. As can been seen, the best result is $15.93\%$
error rate which is achieved by our approach using 128 filters. This
constitutes an improvement by $2.87$, $1.68\%$, and $0.17\%$ compared to the
best results achieved by GMP, GAP, and the standard BoF, respectively.

##  V Conclusion and Future Work

In this paper, we proposed a scheme that builds on top of BoF pooling to
improve its performance. We proposed a regularizer, based on the pair-wise
correlation of the items of the dictionary, which ensures the diversity and
the richness of the learned dictionary within the BoF layer. It led to an
efficient variant of BoF and further improved its capability, without any
additional parameters. The proposed approach can be incorporated in any BoF-
based model in a plug-and-play manner. Empirical results over three different
dataset showed that the proposed regularizer boosts the performance of the
model and led to lower error rates.

Future directions include more extensive experimental evaluation of the
proposed approach over larger datasets and proposing more advanced techniques
for quantifying the similarities between the codebook elements.

## References

  * [1] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton,  “Deep learning,”  nature, vol. 521, no. 7553, pp. 436–444, 2015.
  * [2] Ross Girshick,  “Fast r-cnn,”  in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1440–1448.
  * [3] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton,  “Imagenet classification with deep convolutional neural networks,”  Advances in neural information processing systems, vol. 25, 2012\.
  * [4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [5] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * [6] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun,  “Faster r-cnn: Towards real-time object detection with region proposal networks,”  Advances in neural information processing systems, vol. 28, 2015\.
  * [7] Zhong-Qiu Zhao, Peng Zheng, Shou-tao Xu, and Xindong Wu,  “Object detection with deep learning: A review,”  IEEE transactions on neural networks and learning systems, vol. 30, no. 11, pp. 3212–3232, 2019.
  * [8] Mingxing Tan, Ruoming Pang, and Quoc V Le,  “Efficientdet: Scalable and efficient object detection,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 10781–10790.
  * [9] Guansong Pang, Longbing Cao, and Charu Aggarwal,  “Deep learning for anomaly detection: Challenges, methods, and opportunities,”  in Proceedings of the 14th ACM International Conference on Web Search and Data Mining, 2021, pp. 1127–1130.
  * [10] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft,  “Deep one-class classification,”  in International conference on machine learning. PMLR, 2018, pp. 4393–4402.
  * [11] Hyunjong Park, Jongyoun Noh, and Bumsub Ham,  “Learning memory-guided normality for anomaly detection,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 14372–14381.
  * [12] Taejoon Kim, Sang C Suh, Hyunjoo Kim, Jonghyun Kim, and Jinoh Kim,  “An encoding technique for cnn-based network anomaly detection,”  in 2018 IEEE International Conference on Big Data (Big Data). IEEE, 2018, pp. 2960–2965.
  * [13] Jiasi Chen and Xukan Ran,  “Deep learning with edge computing: A review,”  Proceedings of the IEEE, vol. 107, no. 8, pp. 1655–1674, 2019.
  * [14] Firas Laakom, Jenni Raitoharju, Alexandros Iosifidis, Jarno Nikkanen, and Moncef Gabbouj,  “Color constancy convolutional autoencoder,”  in 2019 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 2019, pp. 1085–1090.
  * [15] Ian Goodfellow, Yoshua Bengio, and Aaron Courville,  Deep learning,  MIT press, 2016.
  * [16] Yuchao Li, Shaohui Lin, Baochang Zhang, Jianzhuang Liu, David Doermann, Yongjian Wu, Feiyue Huang, and Rongrong Ji,  “Exploiting kernel sparsity and entropy for interpretable cnn compression,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [17] Shaohui Lin, Rongrong Ji, Xiaowei Guo, Xuelong Li, et al.,  “Towards convolutional neural networks compression via global error reconstruction.,”  in IJCAI, 2016, pp. 1753–1759.
  * [18] Nikolaos Passalis and Anastasios Tefas,  “Neural bag-of-features learning,”  Pattern Recognition, pp. 277–294, 2017.
  * [19] Nikolaos Passalis and Anastasios Tefas,  “Learning bag-of-features pooling for deep convolutional neural networks,”  in IEEE International Conference on Computer Vision, 2017.
  * [20] Gaurav Menghani,  “Efficient deep learning: A survey on making deep learning models smaller, faster, and better,”  arXiv preprint arXiv:2106.08962, 2021.
  * [21] Fei-Fei Li and Pietro Perona,  “A bayesian hierarchical model for learning natural scene categories,”  in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005, pp. 524–531.
  * [22] G. Qiu,  “Indexing chromatic and achromatic patterns for content-based colour image retrieval,”  Pattern Recognition, pp. 1675 – 1686, 2002.
  * [23] Jun Yang, Yu-Gang Jiang, Alexander G Hauptmann, and Chong-Wah Ngo,  “Evaluating bag-of-visual-words representations in scene classification,”  in Proceedings of the international workshop on Workshop on multimedia information retrieval, 2007, pp. 197–206.
  * [24] Gabriela Csurka and Florent Perronnin,  “Fisher vectors: Beyond bag-of-visual-words image representations,”  in International Conference on Computer Vision, Imaging and Computer Graphics. Springer, 2010, pp. 28–42.
  * [25] Jyoti S Shukla, Kriti Rastogi, Hetal Patel, Gaurav Jain, and Shashikant Sharma,  “Bag of visual words methodology in remote sensing—a review,”  in Proceedings of the International e-Conference on Intelligent Systems and Signal processing. Springer, 2022, pp. 475–486.
  * [26] Yin Zhang, Rong Jin, and Zhi-Hua Zhou,  “Understanding bag-of-words model: a statistical framework,”  International journal of machine learning and cybernetics, vol. 1, no. 1, pp. 43–52, 2010.
  * [27] Nikolaos Passalis and Anastasios Tefas,  “Training lightweight deep convolutional neural networks using bag-of-features pooling,”  IEEE transactions on neural networks and learning systems, vol. 30, no. 6, pp. 1705–1715, 2018.
  * [28] Kateryna Chumachenko, Alexandros Iosifidis, and Moncef Gabbouj,  “Self-attention neural bag-of-features,”  arXiv preprint arXiv:2201.11092, 2022.
  * [29] Nikolaos Passalis, Anastasios Tefas, Juho Kanniainen, Moncef Gabbouj, and Alexandros Iosifidis,  “Temporal bag-of-features learning for predicting mid price movements using high frequency limit order book data,”  IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 4, no. 6, pp. 774–785, 2020.
  * [30] Alexandros Iosifidis, Anastastios Tefas, and Ioannis Pitas,  “Discriminant bag of words based representation for human action recognition,”  Pattern Recognition Letters, pp. 185 – 192, 2014.
  * [31] Nikolaos Passalis, Jenni Raitoharju, Anastasios Tefas, and Moncef Gabbouj,  “Efficient adaptive inference for deep convolutional neural networks using hierarchical early exits,”  Pattern Recognition, vol. 105, pp. 107346, 2020.
  * [32] Dat Thanh Tran, Nikolaos Passalis, Anastasios Tefas, Moncef Gabbouj, and Alexandros Iosifidis,  “Attention-based neural bag-of-features learning for sequence data,”  arXiv preprint arXiv:2005.12250, 2020.
  * [33] Yu-Gang Jiang, Chong-Wah Ngo, and Jun Yang,  “Towards optimal bag-of-features for object categorization and semantic video retrieval,”  in ACM International Conference on Image and Video Retrieval, 2007, pp. 494–501.
  * [34] Firas Laakom, Nikolaos Passalis, Jenni Raitoharju, Jarno Nikkanen, Anastasios Tefas, Alexandros Iosifidis, and Moncef Gabbouj,  “Bag of color features for color constancy,”  IEEE Transactions on Image Processing, vol. 29, pp. 7722–7734, 2020\.
  * [35] Huanhuan Chen,  Diversity and regularization in neural network ensembles,  Ph.D. thesis, University of Birmingham, 2008.
  * [36] Mehrdad J Gangeh, Ahmed K Farahat, Ali Ghodsi, and Mohamed S Kamel,  “Supervised dictionary learning and sparse representation-a review,”  arXiv preprint arXiv:1502.05928, 2015.
  * [37] K Rajesh and Atul Negi,  “Heuristic based learning of parameters for dictionaries in sparse representations,”  in 2018 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 2018, pp. 1013–1019.
  * [38] Babajide O Ayinde, Tamer Inanc, and Jacek M Zurada,  “Regularizing deep neural networks by enhancing diversity in feature extraction,”  IEEE transactions on neural networks and learning systems, vol. 30, no. 9, pp. 2650–2661, 2019.
  * [39] Yashar Naderahmadian, Soosan Beheshti, and Mohammad Ali Tinati,  “Correlation based online dictionary learning algorithm,”  IEEE Transactions on signal processing, vol. 64, no. 3, pp. 592–602, 2015.
  * [40] Hien Van Nguyen, Vishal M Patel, Nasser M Nasrabadi, and Rama Chellappa,  “Kernel dictionary learning,”  in 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2012, pp. 2021–2024.
  * [41] Xiao-Yuan Jing, Rui-Min Hu, Fei Wu, Xi-Lin Chen, Qian Liu, and Yong-Fang Yao,  “Uncorrelated multi-view discrimination dictionary learning for recognition,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2014, vol. 28.
  * [42] Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu,  “Orthogonal convolutional neural networks,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 11505–11515.
  * [43] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny,  “Barlow twins: Self-supervised learning via redundancy reduction,”  in International Conference on Machine Learning. PMLR, 2021, pp. 12310–12320.
  * [44] Pengtao Xie, Aarti Singh, and Eric P. Xing,  “Uncorrelation and evenness: a new diversity-promoting regularizer,”  in Proceedings of the 34th International Conference on Machine Learning, 2017, pp. 3811–3820.
  * [45] Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra,  “Reducing overfitting in deep networks by decorrelating representations,”  arXiv preprint arXiv:1511.06068, 2015.
  * [46] Firas Laakom, Jenni Raitoharju, Alexandros Iosifidis, and Moncef Gabbouj,  “On feature diversity in energy-based models,”  in Energy Based Models Workshop-ICLR 2021, 2021.
  * [47] Firas Laakom, Jenni Raitoharju, Alexandros Iosifidis, and Moncef Gabbouj,  “Reducing redundancy in the bottleneck representation of the autoencoders,”  arXiv preprint arXiv:2202.04629, 2022.
  * [48] Firas Laakom, Jenni Raitoharju, Alexandros Iosifidis, and Moncef Gabbouj,  “Within-layer diversity reduces generalization gap,”  arXiv preprint arXiv:2106.06012, 2021.
  * [49] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner,  “Gradient-based learning applied to document recognition,”  Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
  * [50] Han Xiao, Kashif Rasul, and Roland Vollgraf,  “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,”  arXiv preprint arXiv:1708.07747, 2017.
  * [51] Alex Krizhevsky, Geoffrey Hinton, et al.,  “Learning multiple layers of features from tiny images,”  2009\.
  * [52] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.

  *[CNN]: Convolutional Neural Network
  *[BoF]: Bag of Features
  *[CNNs]: Convolutional Neural Network


# Findings of the Shared Task on Offensive Span Identification from Code-Mixed
Tamil-English Comments

Manikandan Ravikiran$\dagger$ , Bharathi Raja Chakravarthi$\ddagger$, Anand
Kumar Madasamy$\star$
Sangeetha Sivanesan $\circ$, Ratnavel Rajalakshmi $\oplus$, Sajeetha
Thavareesan$\diamond$
Rahul Ponnusamy $\ominus$, Shankar Mahadevan$\bowtie$
$\dagger$Georgia Institute of Technology, Atlanta, Georgia
$\ddagger$Data Science Institute, National University of Ireland Galway
$\star$National Institute of Technology Karnataka Surathkal, India
$\circ$National Institute of Technology, Trichy, India
$\oplus$Vellore Institute of Technology, Chennai, India
$\diamond$Eastern University, Sri Lanka
$\ominus$Indian Institute of Information Technology and Management, Kerala,
India
$\bowtie$Thiagarajar College of Engineering, Madurai, India
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Corresponding Author

###### Abstract

Offensive content moderation is vital in social media platforms to support
healthy online discussions. However, their prevalence in code-mixed Dravidian
languages is limited to classifying whole comments without identifying part of
it contributing to offensiveness. Such limitation is primarily due to the lack
of annotated data for offensive spans. Accordingly, in this shared task, we
provide Tamil-English code-mixed social comments with offensive spans. This
paper outlines the dataset so released, methods, and results of the submitted
systems.

##  1 Introduction

Combating offensive content is crucial for different entities involved in
content moderation, which includes social media companies as well as
individuals Kumaresan et al. (2021); Chakravarthi and Muralidaran (2021). To
this end, moderation is often restrictive with either usage of human content
moderators, who are expected to read through the content and flag the
offensive mentions Arsht and Etcovitch (2018). Alternatively, there are semi-
automated and automated tools that employ trivial algorithms and block lists
Jhaver et al. (2018). Though content moderation looks like a one-way street,
where either it should be allowed or removed, such decision-making is fairly
hard. This is more significant, especially on social media platforms, where
the sheer volume of content is overwhelming for human moderators especially.
With ever increasing offensive social media contents focusing "racism",
"sexism", "hate speech", "aggressiveness" etc. semi-automated and fully
automated content moderation is favored Priyadharshini et al. (2021);
Chakravarthi et al. (2020b); Sampath et al. (2022). However, most of the
existing works Zampieri et al. (2020); Chakravarthi et al. (2022a); Bharathi
et al. (2022); Priyadharshini et al. (2022) are restricted to English only,
with few of them permeating into research that focuses on a more granular
understanding of offensiveness.

Tamil is a agglutinative language from the Dravidian language family dating
back to the 580 BCE Sivanantham and Seran (2019). It is widely spoken in the
southern state of Tamil Nadu in India, Sri Lanka, Malaysia, and Singapore.
Tamil is an official language of Tamil Nadu, Sri Lanka, Singapore, and the
Union Territory of Puducherry in India. Significant minority speak Tamil in
the four other South Indian states of Kerala, Karnataka, Andhra Pradesh, and
Telangana, as well as the Union Territory of the Andaman and Nicobar Islands
Sakuntharaj and Mahesan (2021, 2017, 2016); Thavareesan and Mahesan (2019,
2020a, 2020b, 2021). It is also spoken by the Tamil diaspora, which may be
found in Malaysia, Myanmar, South Africa, the United Kingdom, the United
States, Canada, Australia, and Mauritius. Tamil is also the native language of
Sri Lankan Moors. Tamil, one of the 22 scheduled languages in the Indian
Constitution, was the first to be designated as a classical language of India
Subalalitha (2019); Srinivasan and Subalalitha (2019); Narasimhan et al.
(2018). Tamil is one of the world’s longest-surviving classical languages. The
earliest epigraphic documents discovered on rock edicts and "hero stones" date
from the 6th century BC. Tamil has the oldest ancient non-Sanskritic Indian
literature of any Indian language Anita and Subalalitha (2019b, a);
Subalalitha and Poovammal (2018). Despite its own script, with the advent of
social media, code-switching has permeated into the Tamil language across
informal contexts like forums and messaging outlets Chakravarthi et al. (2019,
2018); Ghanghor et al. (2021a, b); Yasaswini et al. (2021). As a result, code-
switched content is part and parcel of offensive conversations in social
media.

Despite many recent NLP advancements, handling code-mixed offensive content is
still a challenge in Dravidian Languages Sitaram et al. (2019) including Tamil
owing to limitations in data and tools. However, recently the research of
offensive code-mixed texts in Dravidian languages has seen traction
Chakravarthi et al. (2021, 2020a); Priyadharshini et al. (2020); Chakravarthi
(2020). Yet, very few of these focus on identifying the spans that make a
comment offensive Ravikiran and Annamalai (2021). But accentuating such spans
can help content moderators and semi-automated tools which prefer attribution
instead of just a system-generated unexplained score per comment. Accordingly,
in this shared task, we provided code-mixed social media text for the Tamil
language with offensive spans inviting participants to develop and submit
systems under two different settings. Our CodaLab
website111https://competitions.codalab.org/competitions/36395 will remain open
to foster further research in this area.

Figure 1: Example Offensive Span Identification from Code-Mixed Tamil-English
Text

Figure 2: Annotation of offensive spans using Doccano.

##  2 Related Work

###  2.1 Offensive Span Identification

Much of the literature related to offensive span identification find their
roots in SemEval Offensive Span identification shared task focusing on English
Language Pavlopoulos et al. (2021), with development of more than 36 different
systems using a variety of approaches. Notable among these include work by Zhu
et al. (2021) that uses token labeling using one or more language models with
a combination of Conditional Random Fields (CRF). These approaches often rely
on BIO encoding of the text corresponding to offensive spans. Alternatively,
some systems employ post-processing on these token level labels, including re-
ranking and stacked ensembling for predictions Nguyen et al. (2021). Then,
there are exciting works of Rusert (2021); Pluciński and Klimczak (2021) that
exploit rationale extraction mechanism with pre-trained classifiers on
external offensive classification datasets to produce toxic spans as
explanations of the decisions of the classifiers. Lexicon-based baseline
models, which uses look-up operations for offensive words Burtenshaw and
Kestemont (2021) and run statistical analysis Palomino et al. (2021) are also
widely explored. Finally, there are a few approaches that employ custom loss
functions tailored explicitly for false spans. For code-mixed Tamil-English to
date, there is only preliminary work by Ravikiran and Annamalai (2021) that
uses token level labeling.

##  3 Task Description

Our task of offensive span identification required participants to identify
offensive spans i.e, character offsets that were responsible for the offensive
of the comments, when identifying such spans was possible. To this end, we
created two subtasks each of which are as described. Example of offensive span
is shown in Figure 1

###  3.1 Subtask 1: Supervised Offensive Span Identification

Given comments and annotated offensive spans for training, here the systems
were asked to identify the offensive spans in each of the comments in test
data. This task could be approached as supervised sequence labeling, training
on the provided posts with gold offensive spans. It could also be treated as
rationale extraction using classifiers trained on other datasets of posts
manually annotated for offensiveness classification, without any span
annotations.

###  3.2 Subtask 2: Semi-supervised Offensive Span Identification

All the participants of subtask 1 were also encouraged to submit a system to
subtask 2 using semi-supervised approaches. Here in addition to training data
of subtask 1, more unannotated data was provided. Participants were asked to
develop systems using both of these datasets together. To this end, the
unannotated data was allowed to be used in anyway as necessary to aid in
overall model development including creating semi-supervised annotations,
ranking based on similarity etc.

##  4 Dataset

For this shared task, we build upon dataset from earlier work of Ravikiran and
Annamalai (2021), which originally released 4786 code-mixed Tamil-English
comments with 6202 offensive spans. We released this dataset to the
participants during training phase for model development. Meanwhile for
testing we extended this dataset with new additional annotated comments. To
this end, we use dataset of Chakravarthi et al. (2022b) that consist of 10K+
offensive comments. From this we filter out comments that were already part of
train set resulting 4442 comments suitable for annotation. Out of this we
created (a) 3742 comments were used for creating the test data and (b) 700
comments were used for training phase of subtask 2.

Split | Train | Test
---|---|---
Number of Sentences | 4786 | 876
Number of unique tokens | 22096 | 5362
Number of annotated spans | 6202 | 1025
Average size of spans (# of characters) | 21 | 21
Min size of spans (# of characters) | 4 | 3
Max size of spans (# of characters) | 82 | 85
Number of unique tokens in spans | 10737 | 1006

Table 1: Dataset Statistics used in this shared task

In line with earlier works Ravikiran and Annamalai (2021) for the 3742
comments we create span level annotations where at least two annotators
annotated every comment. Additionally, we also employ similar guidelines for
annotation, anonymity maintenance etc. Besides, no annotator data was
collected other than their educational background and their expertise in the
Tamil language.

Additionally, all the annotators were informed in prior about the inherent
profanity of the content along with an option to withdraw from the annotation
process if necessary. For annotation, we use doccano Nakayama et al.
(2018)which was locally hosted by each annotator. Within doccano, all the
annotators were explicitly asked to create a single label called CAUSE with
label id of 1, thus maintaining consistency of annotation labels. (See Figure
2).

To ensure quality each annotation was verified by one or more annotation
verifier, prior to merging and creating gold standard test set. The overall
dataset statistics is given in the Table 1. Compared to train set, we can see
that the test set consists of significantly lesser number of samples, this is
because many of the comments were either small or were hard to clearly
identify the offensive spans. Overall for the 876 comments we obtained Cohen’s
Kappa inter-annotator agreement of 0.61 inline with Ravikiran and Annamalai
(2021).

##  5 Competition Phases

###  5.1 Training Phase

In the training phase, the train split with 4786 comments, and their annotated
spans were released for model development. Participants were given training
data and offensive spans. No validation set was released; rather, participants
were emphasized on cross-validation by creating their splits for preliminary
evaluations or hyperparameter tuning. In total, 30 participants registered for
the task and downloaded the dataset.

###  5.2 Testing Phase

Test set comments without any span annotation were released in the testing
phase. Each participating team was asked to submit their generated span
predictions for evaluation. Predictions are submitted via Google form, which
was used to evaluate the systems. Though CodaLab supports evaluation
inherently, we used google form due to its simplicity. Finally, we assessed
the submitted spans of the test set and were scored using character-based F1
(See section 7.2).

##  6 System Descriptions

Overall we received only a total of 4 submissions (2 main + 2 additional) from
two teams out of 30 registered participants. All these were only for subtask
1. No submissions were made for subtask 2. Each of their respective systems
are as described.

###  6.1 The NITK-IT_NLP Submission

The best performing system from NITK-IT_NLP Hariharan RamakrishnaIyer
LekshmiAmmal (2022) experimented with rationale extraction by training
offensive language classifiers and employing model-agnostic rationale
extraction mechanisms to produce toxic spans as explanations of the decisions
of the classifier. Specifically NITK-IT_NLP used MuRIL Khanuja et al. (2021)
classifier and coupled with LIME Ribeiro et al. (2016) and used the
explanation scores to select words suitable for offensive spans.

###  6.2 The DLRG submission

The DLRG team Mohit et al. (2022) formulated the problem as a combination of
token labeling and span extraction. Specifically, the team created word-level
BIO tags i.e., words were labelled as B (beginning word of a offensive span),
I (inside word of a offensive span), or O (outside of any offensive span).
Following which word level embeddings are created using GloVe Pennington et
al. (2014) and BiLSTM-CRF Panchendrarajan and Amaresan (2018) model is
trained.

###  6.3 Additional Submission

After testing phase, we also requested each team to submit additional runs if
they have variants of approaches. Accordingly we received two additional
submissions from NITK-IT_NLP where they replaced MuRIL from their initial
submission with (i) Multilingual-BERT Devlin et al. (2019) and (ii) ELECTRA
Clark et al. (2020) respectively without any other changes. More details in
section 7.2.

##  7 Evaluation

This section focuses on the evaluation framework of the task. First, the
official measure that was used to evaluate the participating systems is
described. Then, we discuss baseline models that were selected as benchmarks
for comparison reasons. Finally, the results are presented.

###  7.1 Evaluation Measure

In line with work of Pavlopoulos et al. (2021) each system was evaluated F1
score computed on character offset. For each system, we computed the F1 score
per comments, between the predicted and the ground truth character offsets.
Following this we calculated macro-average score over all the 876 test
comments. If in case both ground truth and predicted character offsets were
empty we assigned a F1 of 1 other wise 0 and vice versa.

###  7.2 Benchmark

To establish fair comparison we first created two baseline benchmark systems
which are as described.

  * •

BENCHMARK 1 is a random baseline model which randomly labels 50% of characters
in comments to belong to be offensive. To this end, we run this benchmark 10
times and average results are presented in Table 2.

  * •

BENCHMARK 2 is a lexicon based system, which first extracted all the offensive
words from the train set and during inference these words were searched in
comments from testset and corresponding spans were extracted.

  * •

BENCHMARK 3 is RoBERTA Liu et al. (2019); Ravikiran and Annamalai (2021) model
trained using token labeling approach with BIO encoded texts corresponding to
annotated spans.

Table 2: Official rank and F1 score (%) of the 2 participating teams that
submitted systems. The baselines benchmarks are shown in red.

RANK | TEAM | F1 (%)
---|---|---
1 | NITK-IT_NLP | 44.89
BASELINE | BENCHMARK 1 | 39.75
BASELINE | BENCHMARK 2 | 37.84
BASELINE | BENCHMARK 3 | 38.61
2 | DLRG | 17.28

Table 2 shows the scores and ranks of two teams that made their submission.
NITK-IT_NLP (Section 6.1) was ranked first, followed by DLRG (Section 6.2)
that scored 27% lower was ranked second. The median score was 31.08%, which is
far below the top ranked team and the benchmark baseline models. Meanwhile the
additional submission post testing phase are excluded from ranked table.
Instead they are presented separately in Table 3.

BENCHMARK 1 achieves a considerably high score and, hence, is very highly
ranked with character F1 of 39.83%. Combination of MuRIL with LIME
interpretability by model NITK-IT_NLP is ahead of BENCHMARK 1 by 11%,
indicating the language models ability to effectively rationalize and identify
the spans. This is inline the results of Rusert (2021) which show higher
results than random baseline. Meanwhile BENCHMARK 2 and BENCHMARK 3, also
shows F1 of 37.84% and 38.61% which again NITK-IT_NLP model tend to beat
significantly. On contrary we could see that DLRG model to show least results
of 17.28% lesser than akk the baselines as well as the top performing system.
The lexicon-based BENCHMARK 2 and RoBERTA based BENCHMARK 3 too score very
high. Especially as it overcomes, the submission of DLRG. This may be
attributed to dataset domain itself. Especially, since much of the dataset was
collected from Youtube comments section of Movie Trailers, often we see usages
of same word or similar words. Such behavior is well established across social
media forums including Youtube Duricic et al. (2021), which begs to ask if
indeed the dataset construction needs to be revisited, which forms one
potential exploration for immediate future.

##  8 Analysis and Discussion

Overall we were happy to see the degree of involvement in this shared task
with multiple participants registering, requesting access to datasets and
potential baseline codes for the shared task. Though only two teams submitted
the systems, the resulting diversity of approaches to this problem is fairly
encouraging. However we include some of our observations below, from our
evaluation and what we have learned from the results.

Table 3: Results of additional runs submitted by NITK-IT_NLP.

Method | F1 (%)
---|---
ELECTRA + LIME | 37.33
M-BERT + LIME | 33.95

###  8.1 Participation Characteristics

The authors reached out to teams that initially registered but failed to
create any systems and the vast majority were undergraduate students who were
new into the concept of shared task and were time-limited due to semester
exams. The fact that students participated in the task is promising and we
plan to consider more ways to introduce Shared tasks on Low-Resource Dravidian
Languages in classrooms. To this end, the we used social media and other
medium to spread the word around universities.

On the other hand, 60% of the participants did not download dataset after
registering and instead chose to participate in other shared tasks, which is
problematic and should be addressed. To this end, correspondence with such
teams revealed potential favoritism towards classification based problems that
are common in undergraduate studies. Moreover we also received multiple
queries on the concept of offensive span itself during the training phase,
which is a indicates potential need of improving the overall task structure
with potential early release of data and task details. Yet, upon extending the
number of submissions NITK-IT_NLP submitted additional runs (See Table 3).
Additionally both the teams also submitted source codes
222https://drive.google.com/drive/folders/1T3kl8mljPt8oXcKVn7OQqaU3d55za2zZ?usp=sharing
for their respective models encouraging further development of systems.

###  8.2 General remarks on the approaches

Though neither of teams that made final submissions created any simple
baselines, we could see that all the submissions of NITK-IT_NLP use well
established approaches in recent NLP focusing on pretrained language models.
Meanwhile DLRG used well-grounded Non-Transformer based approach. Yet neither
of teams used any ensembles, data augmentation strategies or modifications to
loss functions that are seen for the task of span identification in the past
across shared tasks.

###  8.3 Error Analysis

Table 2 shows maximum result of $0.4489$ with DLRG failing significantly
compared to random baseline. To this end, we wonder if potentially these
approaches have any weaknesses or strengths. To understand this, first we
study the character F1 results across sentences of different lengths.
Specifically we analysis results of (a) comments with less than 30 characters
(F1@30) (b) comments with 30-50 characters (F1@50) (c) comments with more than
50 characters (F1@>50). The results so obtained are as shown in Table 4.

Table 4: Results of submitted systems across comments of different lengths.

| F1@30 (%) | F1@50 (%) | F1@>50 (%)
---|---|---|---
NITK-IT_NLP | 42.39 | 37.05 | 26.42
DLRG | 39.62 | 23.47 | 14.05

Firstly we can see though NITK-IT_NLP shows high results overall for cases of
comments with larger lengths the model fails significantly. Specifically,
comparing results with ground truth showed that use of LIME often restricts
the overall word so selected as the rationale for offensiveness in turn
reducing number of character offsets predicted as spans. This is because with
larger texts the net score distribution weakens and span extraction is largely
off leading to significant drop in results. Meanwhile for DLRG the results are
more mixed, especially we can see that for comments with less than 30
characters the model shows improvement in F1. Analysis of results reveal that
token labeling is highly accurate, which drops significantly with large size
sentences. This may be attributed to non-local interactions between the words
that may not be captured by the Bi-LSTM CRF model. Further more much of these
sentences often contained only cuss words or clearly abusive words that are
easily identifiable and often present in the train set. Also we found few bugs
in the training code so used, which was already informed to the authors.

Besides error analysis also showed some implicit challenges in the proposed
shared task. First the strong dependency of offensiveness on context makes it
particularly difficult to solve as evident from NITK-IT_NLP which used
language models. Second, offensiveness often is expressed as sarcasm or even
is very subtle. In such cases we often see the offensiveness results to depend
only the words bearing the most negative sentiment, meanwhile the ground truth
spans annotated are larger thus showing high errors. Finally, many times the
nature of offensiveness itself becomes debatable without clear context. Often
these are the cases where we find the developed approaches to fail
significantly.

##  9 Conclusion

Overall this shared task on offensive span identification we introduced a new
dataset for code-mixed Tamil-English language with total of 5652 social media
comments annotated for offensive spans. The task though has large
participants, eventually had only two teams that submitted their systems. In
this paper we described their approaches and discussed their results.
Surprisingly rationale extraction based approach involving combination MuRIL
and LIME performed significantly well. Meanwhile Bi-LSTM CRF model was found
showing sensitivity towards shorter sentences, though it performed
significantly worse than the random baseline. Also extracting offensive spans
for long sentences were found to be difficult especially as they are context
dependent. To this end, we release the baseline models and datasets to foster
further research. Meanwhile in the future we plan to re-do the task of
offensive span identification where we could require the participants to
identify offensive spans and simultaneously classify different types of
offensiveness.

## Acknowledgements

We thank our anonymous reviewers for their valuable feedback. Any opinions,
findings, and conclusion or recommendations expressed in this material are
those of the authors only and does not reflect the view of their employing
organization or graduate schools. The shared task was result of series
projects done during CS7646-ML4T (Fall 2020), CS6460-Edtech Foundations
(Spring 2020) and CS7643-Deep learning (Spring 2022) at Georgia Institute of
Technology (OMSCS Program). Bharathi Raja Chakravarthi were supported in part
by a research grant from Science Foundation Ireland (SFI) under Grant Number
SFI/12/RC/2289$\\_$P2 (Insight$\\_$2), co-funded by the European Regional
Development Fund and Irish Research Council grant IRCLA/2017/129 (CARDAMOM-
Comparative Deep Models of Language for Minority and Historical Languages).

## References

  * Anita and Subalalitha (2019a) R Anita and CN Subalalitha. 2019a.  An approach to cluster Tamil literatures using discourse connectives.  In _2019 IEEE 1st International Conference on Energy, Systems and Information Processing (ICESIP)_ , pages 1–4. IEEE.
  * Anita and Subalalitha (2019b) R Anita and CN Subalalitha. 2019b.  Building discourse parser for Thirukkural.  In _Proceedings of the 16th International Conference on Natural Language Processing_ , pages 18–25.
  * Arsht and Etcovitch (2018) Andrew Arsht and Daniel Etcovitch. 2018.  The human cost of online content moderation.  _Harvard Journal of Law & Technology_.
  * Bharathi et al. (2022) B Bharathi, Bharathi Raja Chakravarthi, Subalalitha Chinnaudayar Navaneethakrishnan, N Sripriya, Arunaggiri Pandian, and Swetha Valli. 2022.  Findings of the shared task on Speech Recognition for Vulnerable Individuals in Tamil.  In _Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion_. Association for Computational Linguistics.
  * Burtenshaw and Kestemont (2021) Ben Burtenshaw and Mike Kestemont. 2021.  UAntwerp at SemEval-2021 task 5: Spans are spans, stacking a binary word level approach to toxic span detection.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 898–903, Online. Association for Computational Linguistics.
  * Chakravarthi (2020) Bharathi Raja Chakravarthi. 2020.  HopeEDI: A multilingual hope speech detection dataset for equality, diversity, and inclusion.  In _Proceedings of the Third Workshop on Computational Modeling of People’s Opinions, Personality, and Emotion’s in Social Media_ , pages 41–53, Barcelona, Spain (Online). Association for Computational Linguistics.
  * Chakravarthi et al. (2018) Bharathi Raja Chakravarthi, Mihael Arcan, and John P. McCrae. 2018.  Improving wordnets for under-resourced languages using machine translation.  In _Proceedings of the 9th Global Wordnet Conference_ , pages 77–86, Nanyang Technological University (NTU), Singapore. Global Wordnet Association.
  * Chakravarthi et al. (2019) Bharathi Raja Chakravarthi, Mihael Arcan, and John P. McCrae. 2019.  WordNet gloss translation for under-resourced languages using multilingual neural machine translation.  In _Proceedings of the Second Workshop on Multilingualism at the Intersection of Knowledge Bases and Machine Translation_ , pages 1–7, Dublin, Ireland. European Association for Machine Translation.
  * Chakravarthi and Muralidaran (2021) Bharathi Raja Chakravarthi and Vigneshwaran Muralidaran. 2021.  Findings of the shared task on hope speech detection for equality, diversity, and inclusion.  In _Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion_ , pages 61–72, Kyiv. Association for Computational Linguistics.
  * Chakravarthi et al. (2020a) Bharathi Raja Chakravarthi, Vigneshwaran Muralidaran, Ruba Priyadharshini, and John Philip McCrae. 2020a.  Corpus creation for sentiment analysis in code-mixed Tamil-English text.  In _Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL)_ , pages 202–210, Marseille, France. European Language Resources association.
  * Chakravarthi et al. (2022a) Bharathi Raja Chakravarthi, Ruba Priyadharshini, Thenmozhi Durairaj, John Phillip McCrae, Paul Buitaleer, Prasanna Kumar Kumaresan, and Rahul Ponnusamy. 2022a.  Findings of the shared task on Homophobia Transphobia Detection in Social Media Comments.  In _Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion_. Association for Computational Linguistics.
  * Chakravarthi et al. (2022b) Bharathi Raja Chakravarthi, Ruba Priyadharshini, Vigneshwaran Muralidaran, Navya Jose, Shardul Suryawanshi, Elizabeth Sherly, and John P. McCrae. 2022b.  DravidianCodeMix: sentiment analysis and offensive language identification dataset for dravidian languages in code-mixed text.  _Language Resources and Evaluation_.
  * Chakravarthi et al. (2020b) Bharathi Raja Chakravarthi, Ruba Priyadharshini, Vigneshwaran Muralidaran, Shardul Suryawanshi, Navya Jose, Elizabeth Sherly, and John P McCrae. 2020b.  Overview of the track on sentiment analysis for Dravidian languages in code-mixed text.  In _Forum for Information Retrieval Evaluation_ , pages 21–24.
  * Chakravarthi et al. (2021) Bharathi Raja Chakravarthi, Ruba Priyadharshini, Rahul Ponnusamy, Prasanna Kumar Kumaresan, Kayalvizhi Sampath, Durairaj Thenmozhi, Sathiyaraj Thangasamy, Rajendran Nallathambi, and John Phillip McCrae. 2021.  Dataset for identification of homophobia and transophobia in multilingual YouTube comments.  _arXiv preprint arXiv:2109.00227_.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Duricic et al. (2021) Tomislav Duricic, Volker Seiser, and Elisabeth Lex. 2021.  Cross-platform analysis of user comments in youtube videos linked on reddit conspiracy theory forum.  _CoRR_ , abs/2109.01127.
  * Ghanghor et al. (2021a) Nikhil Ghanghor, Parameswari Krishnamurthy, Sajeetha Thavareesan, Ruba Priyadharshini, and Bharathi Raja Chakravarthi. 2021a.  IIITK@DravidianLangTech-EACL2021: Offensive language identification and meme classification in Tamil, Malayalam and Kannada.  In _Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages_ , pages 222–229, Kyiv. Association for Computational Linguistics.
  * Ghanghor et al. (2021b) Nikhil Ghanghor, Rahul Ponnusamy, Prasanna Kumar Kumaresan, Ruba Priyadharshini, Sajeetha Thavareesan, and Bharathi Raja Chakravarthi. 2021b.  IIITK@LT-EDI-EACL2021: Hope speech detection for equality, diversity, and inclusion in Tamil , Malayalam and English.  In _Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion_ , pages 197–203, Kyiv. Association for Computational Linguistics.
  * Hariharan RamakrishnaIyer LekshmiAmmal (2022) Manikandan Ravikiran Hariharan RamakrishnaIyer LekshmiAmmal, Anand Kumar Madasamy. 2022.  Nitk-it_nlp@tamilnlp-acl2022: Transformer based model for toxic span identification in tamil.  In _Proceedings of the Second Workshop on Speech and Language Technologies for Dravidian Languages_. Association for Computational Linguistics.
  * Jhaver et al. (2018) Shagun Jhaver, Sucheta Ghoshal, Amy S. Bruckman, and Eric Gilbert. 2018.  Online harassment and content moderation: The case of blocklists.  _ACM Trans. Comput. Hum. Interact._ , 25(2):12:1–12:33.
  * Khanuja et al. (2021) Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha P. Talukdar. 2021.  Muril: Multilingual representations for indian languages.  _CoRR_ , abs/2103.10730.
  * Kumaresan et al. (2021) Prasanna Kumar Kumaresan, Ratnasingam Sakuntharaj, Sajeetha Thavareesan, Subalalitha Navaneethakrishnan, Anand Kumar Madasamy, Bharathi Raja Chakravarthi, and John P McCrae. 2021.  Findings of shared task on offensive language identification in Tamil and Malayalam.  In _Forum for Information Retrieval Evaluation_ , pages 16–18.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Mohit et al. (2022) More Mohit, Naga Shrikriti Bhamatipati, Saharan Gitansh, Hanchate Samyuktha, Nandy Sayantan, and Rajalakshmi Ratnavel. 2022.  Dlrg@tamilnlp-acl2022: Offensive span identification in tamil using bilstm-crf approach.  In _Proceedings of the Second Workshop on Speech and Language Technologies for Dravidian Languages_. Association for Computational Linguistics.
  * Nakayama et al. (2018) Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018\.  doccano: Text annotation tool for human.  Software available from https://github.com/doccano/doccano.
  * Narasimhan et al. (2018) Anitha Narasimhan, Aarthy Anandan, Madhan Karky, and CN Subalalitha. 2018.  Porul: Option generation and selection and scoring algorithms for a tamil flash card game.  _International Journal of Cognitive and Language Sciences_ , 12(2):225–228.
  * Nguyen et al. (2021) Viet Anh Nguyen, Tam Minh Nguyen, Huy Quang Dao, and Quang Huu Pham. 2021.  S-NLP at SemEval-2021 task 5: An analysis of dual networks for sequence tagging.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 888–897, Online. Association for Computational Linguistics.
  * Palomino et al. (2021) Marco Palomino, Dawid Grad, and James Bedwell. 2021.  GoldenWind at SemEval-2021 task 5: Orthrus - an ensemble approach to identify toxicity.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 860–864, Online. Association for Computational Linguistics.
  * Panchendrarajan and Amaresan (2018) Rrubaa Panchendrarajan and Aravindh Amaresan. 2018.  Bidirectional LSTM-CRF for named entity recognition.  In _Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation_ , Hong Kong. Association for Computational Linguistics.
  * Pavlopoulos et al. (2021) John Pavlopoulos, Léo Laugier, Jeffrey Sorensen, and Ion Androutsopoulos. 2021\.  Semeval-2021 task 5: Toxic spans detection (to appear).  In _Proceedings of the 15th International Workshop on Semantic Evaluation_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Pluciński and Klimczak (2021) Kamil Pluciński and Hanna Klimczak. 2021.  GHOST at SemEval-2021 task 5: Is explanation all you need? In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 852–859, Online. Association for Computational Linguistics.
  * Priyadharshini et al. (2022) Ruba Priyadharshini, Bharathi Raja Chakravarthi, Subalalitha Chinnaudayar Navaneethakrishnan, Thenmozhi Durairaj, Malliga Subramanian, Kogilavani Shanmugavadivel, Siddhanth U Hegde, and Prasanna Kumar Kumaresan. 2022\.  Findings of the shared task on Abusive Comment Detection in Tamil.  In _Proceedings of the Second Workshop on Speech and Language Technologies for Dravidian Languages_. Association for Computational Linguistics.
  * Priyadharshini et al. (2021) Ruba Priyadharshini, Bharathi Raja Chakravarthi, Sajeetha Thavareesan, Dhivya Chinnappa, Durairaj Thenmozhi, and Rahul Ponnusamy. 2021.  Overview of the dravidiancodemix 2021 shared task on sentiment detection in Tamil, Malayalam, and Kannada.  In _Forum for Information Retrieval Evaluation_ , pages 4–6.
  * Priyadharshini et al. (2020) Ruba Priyadharshini, Bharathi Raja Chakravarthi, Mani Vegupatti, and John P McCrae. 2020.  Named entity recognition for code-mixed Indian corpus using meta embedding.  In _2020 6th international conference on advanced computing and communication systems (ICACCS)_ , pages 68–72. IEEE.
  * Ravikiran and Annamalai (2021) Manikandan Ravikiran and Subbiah Annamalai. 2021.  DOSA: Dravidian code-mixed offensive span identification dataset.  In _Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages_ , pages 10–17, Kyiv. Association for Computational Linguistics.
  * Ribeiro et al. (2016) Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  "why should I trust you?": Explaining the predictions of any classifier.  In _Proceedings of the Demonstrations Session, NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016_ , pages 97–101. The Association for Computational Linguistics.
  * Rusert (2021) Jonathan Rusert. 2021.  NLP_UIOWA at Semeval-2021 task 5: Transferring toxic sets to tag toxic spans.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 881–887, Online. Association for Computational Linguistics.
  * Sakuntharaj and Mahesan (2016) Ratnasingam Sakuntharaj and Sinnathamby Mahesan. 2016.  A novel hybrid approach to detect and correct spelling in Tamil text.  In _2016 IEEE International Conference on Information and Automation for Sustainability (ICIAfS)_ , pages 1–6.
  * Sakuntharaj and Mahesan (2017) Ratnasingam Sakuntharaj and Sinnathamby Mahesan. 2017.  Use of a novel hash-table for speeding-up suggestions for misspelt Tamil words.  In _2017 IEEE International Conference on Industrial and Information Systems (ICIIS)_ , pages 1–5.
  * Sakuntharaj and Mahesan (2021) Ratnasingam Sakuntharaj and Sinnathamby Mahesan. 2021.  Missing word detection and correction based on context of Tamil sentences using n-grams.  In _2021 10th International Conference on Information and Automation for Sustainability (ICIAfS)_ , pages 42–47.
  * Sampath et al. (2022) Anbukkarasi Sampath, Thenmozhi Durairaj, Bharathi Raja Chakravarthi, Ruba Priyadharshini, Subalalitha Chinnaudayar Navaneethakrishnan, Kogilavani Shanmugavadivel, Sajeetha Thavareesan, Sathiyaraj Thangasamy, Parameswari Krishnamurthy, Adeep Hande, Sean Benhur, and Santhiya Ponnusamy, Kishor Kumar Pandiyan. 2022.  Findings of the shared task on Emotion Analysis in Tamil.  In _Proceedings of the Second Workshop on Speech and Language Technologies for Dravidian Languages_. Association for Computational Linguistics.
  * Sitaram et al. (2019) Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Krishna Rallabandi, and A. Black. 2019\.  A survey of code-switched speech and language processing.  _ArXiv_ , abs/1904.00784.
  * Sivanantham and Seran (2019) R Sivanantham and M Seran. 2019.  Keeladi: An urban settlement of sangam age on the banks of river vaigai.  _India: Department of Archaeology, Government of Tamil Nadu, Chennai_.
  * Srinivasan and Subalalitha (2019) R Srinivasan and CN Subalalitha. 2019.  Automated named entity recognition from tamil documents.  In _2019 IEEE 1st International Conference on Energy, Systems and Information Processing (ICESIP)_ , pages 1–5. IEEE.
  * Subalalitha (2019) C. N. Subalalitha. 2019.  Information extraction framework for Kurunthogai.  _Sādhanā_ , 44(7):156.
  * Subalalitha and Poovammal (2018) CN Subalalitha and E Poovammal. 2018.  Automatic bilingual dictionary construction for Tirukural.  _Applied Artificial Intelligence_ , 32(6):558–567.
  * Thavareesan and Mahesan (2019) Sajeetha Thavareesan and Sinnathamby Mahesan. 2019.  Sentiment analysis in Tamil texts: A study on machine learning techniques and feature representation.  In _2019 14th Conference on Industrial and Information Systems (ICIIS)_ , pages 320–325.
  * Thavareesan and Mahesan (2020a) Sajeetha Thavareesan and Sinnathamby Mahesan. 2020a.  Sentiment lexicon expansion using Word2vec and fastText for sentiment prediction in Tamil texts.  In _2020 Moratuwa Engineering Research Conference (MERCon)_ , pages 272–276.
  * Thavareesan and Mahesan (2020b) Sajeetha Thavareesan and Sinnathamby Mahesan. 2020b.  Word embedding-based part of speech tagging in Tamil texts.  In _2020 IEEE 15th International Conference on Industrial and Information Systems (ICIIS)_ , pages 478–482.
  * Thavareesan and Mahesan (2021) Sajeetha Thavareesan and Sinnathamby Mahesan. 2021.  Sentiment analysis in Tamil texts using k-means and k-nearest neighbour.  In _2021 10th International Conference on Information and Automation for Sustainability (ICIAfS)_ , pages 48–53.
  * Yasaswini et al. (2021) Konthala Yasaswini, Karthik Puranik, Adeep Hande, Ruba Priyadharshini, Sajeetha Thavareesan, and Bharathi Raja Chakravarthi. 2021.  IIITT@DravidianLangTech-EACL2021: Transfer learning for offensive language detection in Dravidian languages.  In _Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages_ , pages 187–194, Kyiv. Association for Computational Linguistics.
  * Zampieri et al. (2020) Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çagri Çöltekin. 2020.  Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020).  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation, SemEval@COLING 2020, Barcelona (online), December 12-13, 2020_ , pages 1425–1447. International Committee for Computational Linguistics.
  * Zhu et al. (2021) Qinglin Zhu, Zijie Lin, Yice Zhang, Jingyi Sun, Xiang Li, Qihui Lin, Yixue Dang, and Ruifeng Xu. 2021.  HITSZ-HLT at SemEval-2021 task 5: Ensemble sequence labeling and span boundary detection for toxic span detection.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 521–526, Online. Association for Computational Linguistics.


# AMR Parsing with Instruction Fine-tuned Pre-trained Language Models

Young-Suk Lee, Ramón Fernandez Astudillo, Radu Florian, Tahira Naseem, Salim
Roukos
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
IBM Research AI

###### Abstract

Instruction fine-tuned language models on a collection of instruction
annotated datasets (FLAN) have shown highly effective to improve model
performance and generalization to unseen tasks. However, a majority of
standard parsing tasks including abstract meaning representation (AMR),
universal dependency (UD), semantic role labeling (SRL) has been excluded from
the FLAN collections for both model training and evaluations. In this paper,
we take one of such instruction fine-tuned pre-trained language models, i.e.
FLAN-T5, and fine-tune them for AMR parsing. Our extensive experiments on
various AMR parsing tasks including AMR2.0, AMR3.0 and BioAMR indicate that
FLAN-T5 fine-tuned models out-perform previous state-of-the-art models across
all tasks. In addition, full fine-tuning followed by the parameter efficient
finetuning, LoRA, further improves the model performances, setting new state-
of-the-arts in Smatch on AMR2.0 (86.4), AMR3.0 (84.9) and BioAMR (82.3).

##  1 Introduction

Instruction fine-tuning language models on a collection of annotated datasets
has proven highly effective to improve model performance and generalization to
unseen tasks both in general purpose open domain setup, as in Chung et al.
(2022); Wei et al. (2021); Longpre et al. (2023); Ouyang et al. (2022); Mishra
et al. (2022); Wang et al. (2022); Honovich et al. (2022) and specialized
tasks such as conversational dialogs in Gupta et al. (2022).

Despite its great success in the majority of natural language processing
tasks, however, standard parsing tasks such as abstract meaning representation
(AMR), Banarescu et al. (2013); Bevilacqua et al. (2021); Zhou et al. (2021);
Bai et al. (2022), universal dependency (UD), Nivre et al. (2017), semantic
role labeling (SRL), Gildea and Jurafsky (2002); Palmer et al. (2005), etc.
have been largely excluded from the fine-tuned language net (FLAN) collections
either for model training or evaluations. And therefore it still remains to be
seen whether or not instruction fine-tuned language models are as effective
for standard parsing tasks as for other NLP tasks.

In this paper, we fine-tune FLAN-T5 models of Chung et al. (2022) (FLANT5Large
and FLANT5XL) on a wide range of AMR parsing tasks including AMR2.0, AMR3.0
and BioAMR. We show that fine-tuning FLAN-T5 models on AMR parsing leads to a
significant improvement over the previous BART fine-tuned SoTA models by Zhou
et al. (2021); Bai et al. (2022). We further explore a parameter efficient
fine-tuning technique, LoRA (Low Rank Adaptation), Hu et al. (2021). While
LoRAonly finetuned models do not out-perform full fine-tuned models, full
fine-tuning followed by LoRA fine-tuning significantly improve full fine-tuned
models, setting new state-of-the-arts across all AMR parsing tasks.

Our main contributions are as follows:

  * •

We apply instruction fine-tuned FLAN-T5 models to AMR parsing for the first
time. We show that FLAN-T5 fine-tuned AMR parsing models significantly out-
perform previous BART fine-tuned SoTA models.

  * •

We explore the parameter efficient fine-tuning technique LoRA for sequence-to-
sequence tasks. Although fine-tuning FLAN-T5 models with LoRA only does not
out-perform full fine-tuned models, LoRA fine-tuning of full fine-tuned models
further improves model performances.

  * •

We push the envelope of AMR parsing, by setting new SoTA in Smatch on AMR2.0
(86.4), AMR3.0 (84.9) and BioAMR (82.3).

##  2 AMR Parsing with FLAN-T5 Models

Flan-T5 models, Chung et al. (2022), are obtained by instruction fine-tuning
T5-LM adapted models, Lester et al. (2021), on a collection of 1.8K
instruction annotated tasks.111https://github.com/google-
research/FLAN/blob/main/flan/v2/flan_collection_info.csv They are prefix
language models222Given natural text prefix as input, the model must produce
the natural text continuation as output. and achieve strong few-shot
performance even compared to much larger models, such as PaLM 62B.

Like all models derived from T5 models, Raffel et al. (2020), we pose AMR
parsing as a text-to-text problem and train models to transfer a text to a
linearized AMR graph with the task prefix amr generation. FLAN-T5 model size
variants, all of which use 32,128 unique vocabulary, is shown in Table 1.

Models | # parameters | # layers | dmodel | dff | dkv | # heads
---|---|---|---|---|---|---
Flan-T5-Small | 77M | 8 | 512 | 1024 | 64 | 6
Flan-T5-Base | 250M | 12 | 768 | 2048 | 64 | 12
Flan-T5-Large | 780M | 24 | 1024 | 2816 | 64 | 16
Flan-T5-XL | 3B | 24 | 2048 | 5120 | 64 | 32
Flan-T5-XXL | 11B | 24 | 4096 | 10240 | 64 | 64
Table 1: Flan-T5 model size variants obtained from each model configuration
file of https://huggingface.co/models. We fine-tune Flan-T5-Large and
Flan-T5-XL for our AMR experiments.

###  2.1 Pre- and Post-processing

We first remove wiki tags from the raw AMR graphs. We then serialize the AMR
graph and transfer the node variable information of concepts to the concepts
themselves.333https://github.com/IBM/graph_ensemble_learning If a graph
includes the same concept more than once, unique indices are appended to each
concept, e.g. thing_1, thing_2, etc. Finally, we add the task prefix amr
generation to each input text. A sample input text and the corresponding
serialized AMR graph after pre-processing is shown in Figure 1.

    Input: Statistics also revealed that
    Taiwanese business investments in the
    mainland is tending to increase

    AMR graph:
    (r / reveal-01
        :ARG0 (s / statistic)
        :ARG1 (t / tend-02
            :ARG1 (t2 / thing
                :ARG1-of (i / invest-01
                :ARG0 (c / country
                    :wiki "Taiwan"
                    :name (n / name
                          :op1 "Taiwan"))
                :ARG2 (m / mainland)
                :mod (b / business)))
        :ARG2 (i2 / increase-01
                :ARG1 t2))
        :mod (a / also))

    Serialized AMR graph:
    ( reveal-01 :ARG0 ( statistic ) :ARG1
    ( tend-02 :ARG1 ( thing :ARG1-of ( invest-01
    :ARG0 ( country :name ( name :op1 "Taiwan" ) )
    :ARG2 ( mainland ) :mod ( business ) ) )
    :ARG2 ( increase-01 :ARG1 thing ) ) :mod
    ( also ) )

    Input text with the task prefix:
    amr generation ; Statistics also revealed
    that Taiwanese business investments in the
    mainland is tending to increase

Figure 1: Serialization of AMR graphs and addition of the task prefix amr
generation to the input text. In the serialized graph, the variables
corresponding to each concept in the original AMR graph is removed while the
parentheses indicating the concept spans are retained. The input text with the
task prefix and the serialized AMR graphs are used for model training.

For testing, the decoder first generates serialized graphs, which is then de-
serialized to restore the concept variables including re-ifications. We
finally restore wiki tags using deterministic algorithms.

###  2.2 Full Fine-tuning

For full fine-tuning, we call the huggingface
transformers444https://github.com/huggingface/transformers class
T5ForConditionalGeneration and T5Tokenizer. We train all models on 2 NVIDIA
A100 80GB machines for 24 hours.

For both FLAN-T5-large and FLAN-T5-XL models, we set the maximum source and
target lengths to 512. Learning rate is set to 5e-5. We utilize sentence based
batching for mini batches. Batch size is 8 for FLAN-T5-large and 4 FLAN-T5-XL,
distributed over 2 GPUs.

We run the validation data set after each epoch and choose the model with the
highest validation set Smatch score as the final best model for testing.

###  2.3 LoRA Fine-tuning

We experiment with the low rank adaptation
LoRA555https://github.com/huggingface/peft for AMR parsing, a sequence-to-
sequenc task using an encoder-decoder architecture.

Largely following the recommended setup of adapting only the q (query) and v
(value) projections in the transformer, we explore the LoRA configurations
between rank=8, alpha=32 and rank=16, alpha=64 while fixing task_type to
SEQ_2_SEQ_LM. We call model.eval() to merge LoRA parameters with the
corresponding pre-trained ones after each parameter update and for inferencing
and model.train() to split the LoRA parameters from the pre-trained ones for
updating only LoRA parameters. Unlike full fine-tuning for which learning rate
is 5e-5, we use learning rate 4e-1 for LoRA fine-tuning. We have found that
LoRA fine-tuning requires a higher learning rate than full fine-tuning for
optimal performances, which is not surprising given the much fewer number of
parameters to update with LoRA fine-tuning compared with full fine-tuning.

We experiment with two different modes of LoRA fine-tuning. First, apply LoRA
fine-tuning to the pretrained language model (PLM) directly. Second, apply
full fine-tuning to the PLM, and then apply LoRA fine-tuning to the full fine-
tuned models. LoRA fine-tuning on PLM directly does not seem to improve the
performances over full fine-tuned models.666We have not explored the full
range of adjustable parameters of LoRA, e.g. k and o projections in addition
to q and v projections, or values other than 8 and 16 for LoRA rank and 32 and
64 for LoRA alpha. However, LoRA fine-tuning on full fine-tuned models improve
the full fine-tuned models across various AMR parsing tasks and model
configurations.

##  3 Experimental Results

We experiment on 3 AMR tasks, AMR2.0, AMR3.0 and BioAMR, and 2 model training
configurations, FLAN-T5-large and FLAN-T5-XL. Training and test corpora
statistics are shown in Table 2. Silver training corpus is annotated with the
MBSE ensemble distillation technique presented in Lee et al. (2022).

Dataset | Split | Sents | Tokens
---|---|---|---
AMR2.0 | Trainh | 36,521 | 653K
| Test | 1,371 | 30K
AMR3.0 | Trainh | 55,635 | 1M
| Test | 1,898 | 39K
Bio AMR | Trainh | 5,452 | 231K
| Test | 500 | 22K
PropBank | Silverstd | 20K | 386K
SQuAD2.0-C | Silverstd | 70K | 2M
Ontonotes5.0 | Silverstd | 59K | 1.1M
WikiText-103 | Silverstd | 70K | 2M
BioNLP-ST-2011 | Silverbio | 15K | 460K
CRAFT | Silverbio | 27K | 740K
PubMed | Silverbio | 26K | 750K
Table 2: Corpus statistics for the standard benchmark experiments on AMR2.0, AMR3.0 and BioAMR test sets. In corpus split, Trainh indicates human annotated treebank. Silverstd indicates the unlabeled data for silver training of AMR2.0 and AMR3.0. Silverbio indicates the unlabeled data for silver training of BioAMR. Silver training corpus is annotated with the MBSE ensemble distillation technique in Lee et al. (2022). Training Corpora | Human Annotations | Human & Silver Annotations
---|---|---
Models | AMR2.0 | AMR3.0 | BioAMR | AMR2.0 | AMR3.0 | BioAMR
Lee et al. (2022) | 84.2 | 82.3 | 79.8 | 85.9 | 84.3 | 81.3
FLAN-T5-Large-LoRA | 82.3 | 81.7 | 79.1 | 84.6 | 83.0 | 80.4
FLAN-T5-Large-FFT | 84.6 | 83.2 | 81.0 | 85.8 | 84.6 | 82.1
FLAN-T5-Large-FFT-LoRA | 84.8$\pm$0.1 | 83.3$\pm$0.0 | 81.2$\pm$0.1 | 86.1$\pm$0.0 | 84.7$\pm$0.1 | 82.2$\pm$0.1
FLAN-T5-XL-FFT | 84.6 | 83.4 | 80.9 | 86.1 | 84.6 | 82.3
FLAN-T5-XL-FFT-LoRA | 84.8$\pm$0.0 | 83.9$\pm$0.1 | 81.6$\pm$0.0 | 86.4$\pm$0.1 | 84.9$\pm$0.0 | 82.3$\pm$0.2
Table 3: Performance of FLAN-T5 fine-tuned models trained on human annotations only (left) and human and silver annotations (right). FFT denotes full fine-tuning. The numbers prefixed by $\pm$ indicate the standard deviation of Smatch scores across 2 seeds with different LoRA configurations. Models | PLM | Silver | AMR2.0 | AMR3.0 | BioAMR
---|---|---|---|---|---
SPRING Bevilacqua et al. (2021) | BART-large | - | 84.5 | 83.0 | 79.9
SPRING Bevilacqua et al. (2021) | BART-large | 200K | 84.3 | 83.0 | 59.5
StructBART-vanilla Zhou et al. (2021) | BART-large | 90K | 84.7 | 82.7 | -
BARTAMR Bai et al. (2022) | BART-large | 200K | 85.4 | 84.2 | 63.2
StructBART-MBSE Lee et al. (2022) | BART-large | 219K | 85.9 | 84.3 | 81.3
FLAN-T5-Large-FFT-LoRA (Ours) | FLAN-T5-Large | 219K | 86.1 | 84.7$\ddagger$ | 82.2$\ddagger$
FLAN-T5-XL-FFT-LoRA (Ours) | FLAN-T5-XL | 219K | 86.4$\ddagger$ | 84.9$\ddagger$ | 82.3$\ddagger$
Table 4: Comparison of AMR parsing models for AMR2.0, AMR3.0 and BioAMR test sets. We compare the current FLAN-T5 fine-tuned models (Ours) against those BART-large fine-tuned models. Boldface indicates the best model scores. $\ddagger$ indicates that the model is statistically significantly better than all of the previous models at p=0.05 according to randomized bootstrap statistical significance tests. Test Data | Smatch | Unlabel | NoWSD | Concepts | NER | Neg. | Wiki | Reentrancy | SRL
---|---|---|---|---|---|---|---|---|---
AMR2.0-mbse | 86.4 | 89.2 | 86.7 | 92.0 | 93.0 | 79.0 | 81.0 | 78.0 | 85.0
AMR3.0-mbse | 84.9 | 87.8 | 85.3 | 91.0 | 90.0 | 75.0 | 78.0 | 77.0 | 84.0
BioAMR-mbse | 82.5 | 85.1 | 82.4 | 89.0 | 81.0 | 80.0 | 0.0 | 74.0 | 83.0
AMR2.0-base | 84.8 | 87.8 | 85.2 | 91.0 | 93.0 | 75.0 | 78.0 | 76.0 | 83.0
AMR3.0-base | 83.9 | 86.8 | 84.3 | 90.0 | 89.0 | 74.0 | 72.0 | 76.0 | 83.0
BioAMR-base | 81.6 | 84.1 | 81.5 | 88.0 | 82.0 | 80.0 | 0.0 | 72.0 | 81.0
Table 5: Fine-grained F1 scores of FLAN-T5-XL-FFT-LoRA models on test data
sets. AMR2.0-mbse, AMR3.0-mbse, BioAMR-mbse indicate AMR2.0, AMR3.0 and BioAMR
test sets are evaluated on models trained with the combination of human
annotated and silver training data discussed in Lee et al. (2022).
AMR2.0-base, AMR3.0-base and BioAMR-base indicate AMR2.0, AMR3.0 and BioAMR
test sets are evaluated on models trained with human annotated data only.

Experimental results are shown in Table 3. As a point of reference, we include
the highest Structured BART scores from Lee et al. (2022). FFT denotes full
fine-tuning and LoRA, LoRA fine-tuning. All FLAN-T5-Large-FFT-LoRA and
FLAN-T5-Large-FFT-LoRA scores are an average of two distinct model scores, one
trained with lora_rank=8, lora_alpha=32 and the other, with lora_rank=16,
lora_alpha=64.

Across all tasks and training corpus sizes, full fine-tuned (FFT)
FLAN-T5-Large models out-perform Structured BART, except for AMR2.0 with
silver training data for which FLAN-T5-Large is 0.1 Smatch lower than
Structured BART. Full fine-tuned FLAN-T5-XL models out-perform all
corresponding Structured BART models. LoRA fine-tuning lags behind full fine-
tuning in performance when comparing FLAN-T5-Large-LoRA with FLAN-T5-Large-
FFT. Full fine-tuning followed by LoRA fine-tuning (FFT-LoRA), however, always
out-performs full fine-tuning only except for BioAMR with silver training
corpus, for which full fine-tuned model score is the same as that of full
fine-tuning followed by LoRA fine-tuning.

The fact that full fine-tuning followed by LoRA fine-tuning improves the
scores of full fine-tuned models is somewhat unexpected and this training
setup does not seem to have been explored elsewhere, Xu et al. (2032);
Valipour et al. (2022); Lialin et al. (2023). We conjecture that the low rank
adaptation by LoRA prevents the model from over-fitting the training data
especially when the supervised training corpus size is large as in our AMR
parsing setup, which in turn leads to better generalization capabilities on
unseen test sets. We leave this topic for future research for now.

We compare the performances of our best models with previous SoTA models in
Table 4. We restrict the comparisons only with models that fine-tune PLMs,
i.e. BART-large. The model scores with $\ddagger$ indicates that the models
are significantly better than all of the previous models at p=0.05 according
to randomized bootstrap statistical significance
tests.777https://github.com/IBM/transition-amr-
parser/blob/master/scripts/smatch_aligner.py We see that FLAN-T5 fine-tuned
models out-perform the previous SoTA models across all training conditions and
model configurations.

We show the detailed scores of our best models both with and without silver
training corpus in Table 5. BioAMR wiki scores are 0.0 because we do not apply
wikification to BioAMR parsing outputs. Overall, concept scores are the
highest and negation/re-entrancy are the lowest among all categories. Negation
scores are lower than re-entrancy scores for AMR3.0-mbse, AMR3.0-base and
AMR2.0-base whereas for all others, re-entrancy scores are lower than negation
scores. Named entity detection (NER) does not seem to be an issue even with
BioAMR, which should be attributed to the fact that the training corpus
includes 5K human annotated BioAMR graphs.

##  4 Conclusion and Future Work

We presented AMR parsing with instruction fine-tuned FLAN-T5 models, first
parsing results with FLAN-T5 models to the best of our knowledge. The
experimental results indicate that FLAN-T5 fine-tuned AMR parsing models
significantly out-perform previous SoTA models, which were also fine-tuned
with another PLM, BART-large. We also explore parameter efficient fine-tuning
LoRA. While LoRA fine-tuned models under-perform full fine-tuned models, LoRA
tuning applied to full fine-tuned models further improves the Smatch scores of
fine-tuned models across all training conditions. We push the envelope of AMR
parsing, by setting new SoTA in Smatch on AMR2.0 (86.4), AMR3.0 (84.9) and
BioAMR (82.5), which is even higher than 7 model graphene ensemble results
presented in Tables 2 and 3 of Hoang et al. (2021).

While full fine-tuning followed by LoRA fine-tuning improves the model
performances significantly, compared with the models with full fine-tuning
only, we do not yet understand exactly why this should be the case, which we
leave this for future research.

With the advent of very powerful instruction fine-tuned language models with
human feedback such as ChatGPT and GPT-4, many natural language processing
tasks, including classification and detection, achieve very high zero-shot
performances. Nontheless, given the unique label vocabulary and the hidden
structure present in most parsing representations, zero shot parsing on a new
parsing task does not seem easily achievable. Instruction fine-tuning on a
collection of annotated natural language parsing tasks, along the lines what
has been done for dialog tasks in Gupta et al. (2022), might lead to high
performing few-shot or zero-shot learning of new parsing tasks.

## References

  * Bai et al. (2022) Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022.  Graph pre-training for AMR parsing and generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013\.  Abstract Meaning Representation for sembanking.  In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_ , pages 178–186, Sofia, Bulgaria. Association for Computational Linguistics.
  * Bevilacqua et al. (2021) Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021.  One spring to rule them both: Symmetric amr semantic parsing and generation without a complex pipeline.  In _AAAI Technical Track on Speech and Natural Language Processing I, Vol. 35 No. 14_ , pages 12564–12573.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Le, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.  _arXiv:2210.11416_.
  * Gildea and Jurafsky (2002) Daniel Gildea and Daniel Jurafsky. 2002.  Automatic labeling of semantic roles.  _Computational Linguistics, Volume 28, Number 3, September 2002_.
  * Gupta et al. (2022) Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, Maxine Eskenazi, and Jeffrey P. Bigham. 2022.  Instructdial: Improving zero and few-shot generalization in dialogue through instruction tuning.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 505–525.
  * Hoang et al. (2021) Thanh Lam Hoang, Gabriele Picco, Yufang Hou, Young-Suk Lee, Lam M. Nguyen, Dzung T. Phan, Vanessa López, and Ramon Fernandez Astudillo. 2021.  Ensembling graph prediction for amr parsing.  _arXiv preprint arXiv:2110.09131_.
  * Honovich et al. (2022) Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022.  Unnatural instructions: Tuning language models with (almost) no human labor.  _arXiv:2212.09689_.
  * Hu et al. (2021) Edward Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.
  * Lee et al. (2022) Young-Suk Lee, Ramón Fernandez Astudillo, Thanh Lam Hoang, Tahira Naseem, Radu Florian, and Salim Roukos. 2022.  Maximum bayes smatch ensemble distillation for amr parsing.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5379–5392.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059.
  * Lialin et al. (2023) Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. 2023.  Scaling down to scale up: A guide to parameter-efficient fine-tuning.  _arXiv:2303.15647_.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023.  The flan collection: Designing data and methods for effective instruction tuning.  _arXiv preprint arXiv:2301.13688_.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.  Cross-task generalization via natural language crowdsourcing instructions.  In _ACL_.
  * Nivre et al. (2017) Joakim Nivre, Daniel Zeman, Filip Ginter, and Francis Tyers. 2017.  Universal dependencies.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts_.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  _arxiv.org/abs/2203.02155_.
  * Palmer et al. (2005) Martha Palmer, Dan Gildea, and Paul Kingsbury. 2005.  The proposition bank: A corpus annotated with semantic roles.  _Computational Linguistics Journal, 31:1, 2005_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research 21 (2020) 1-67_.
  * Valipour et al. (2022) Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. 2022.  Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low rank adaptation.  _arXiv: 2210.07558_.
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022.  Super-naturalinstructions:generalization via declarative instructions on 1600+ tasks.  In _EMNLP_.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Xu et al. (2032) Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2032.  Baize: An open-source chat model with parameter-efficient tuning on self-chat data.  _arXiv: 2304.01196_.
  * Zhou et al. (2021) Jiawei Zhou, Tahira Naseem, Ramon Fernandez Astudillo, Young-Suk Lee, Radu Florian, and Salim Roukos. 2021.  Structure-aware fine-tuning of sequence-to-sequence transformers for transition-based amr parsing.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6279–6290.