# A Sea-Land Clutter Classification Framework for Over-the-Horizon-Radar Based
on Weighted Loss Semi-supervised GAN

Xiaoxuan Zhang, Zengfu Wang, Kun Lu, Quan Pan, and Yang Li This work was in
part supported by the National Natural Science Foundation of China (grant no.
61790552, U21B2008) and Natural Science Basic Research Plan in Shaanxi
Province of China (2021JM-06). Xiaoxuan Zhang, Zengfu Wang, Quan Pan, Yang Li
are with the School of Automation, Northwestern Polytechnical University, and
the Key Laboratory of Information Fusion Technology, Ministry of Education,
Xi’an, Shaanxi, 710072, China. Kun Lu is with Nanjing Research Institute of
Electronics Technology and the Sky-Rainbow United Laboratory, Nanjing,
Jiangsu, 210039, China. E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>liyangnpu@mail.nwpu.edu.cn). (Corresponding author: Zengfu Wang.)

###### Abstract

Deep convolutional neural network has made great achievements in sea-land
clutter classification for over-the-horizon-radar (OTHR). The premise is that
a large number of labeled training samples must be provided for a sea-land
clutter classifier. In practical engineering applications, it is relatively
easy to obtain label-free sea-land clutter samples. However, the labeling
process is extremely cumbersome and requires expertise in the field of OTHR.
To solve this problem, we propose an improved generative adversarial network,
namely weighted loss semi-supervised generative adversarial network (WL-
SSGAN). Specifically, we propose a joint feature matching loss by weighting
the middle layer features of the discriminator of semi-supervised generative
adversarial network. Furthermore, we propose the weighted loss of WL-SSGAN by
linearly weighting standard adversarial loss and joint feature matching loss.
The semi-supervised classification performance of WL-SSGAN is evaluated on a
sea-land clutter dataset. The experimental results show that WL-SSGAN can
improve the performance of the fully supervised classifier with only a small
number of labeled samples by utilizing a large number of unlabeled sea-land
clutter samples. Further, the proposed weighted loss is superior to both the
adversarial loss and the feature matching loss. Additionally, we compare WL-
SSGAN with conventional semi-supervised classification methods and demonstrate
that WL-SSGAN achieves the highest classification accuracy.

###### Index Terms:

Over-the-horizon radar, Sea-land clutter, Generative adversarial network,
Semi-supervised classification, Feature matching, Deep learning.

##  I Introduction

As a crucial system for remote sensing, sky-wave over-the-horizon radar (OTHR)
is widely used in military and civilian fields [1, 2, 3, 4]. The sea-land
clutter classification of OTHR is the process of identifying whether the
background clutter of each range-azimuth cell is originated from land or sea.
Matching the derived classification results with the a prior geographic
information cost-effectively provides coordinate registration parameters for
target localization, which are important to the performance improvement of
OTHR [5, 6].

In recent years, deep learning has been successfully applied to sea-land
clutter classification. Compared with conventional classification methods [7,
8, 9, 10, 11] in which the design of feature extractor depends on human
engineering, the deep learning-based sea-land clutter classification method
has powerful feature extraction ability, which can automatically extract
hierarchical representations from sea-land clutter data. The extracted
features are beneficial for subsequent classification tasks. Li _et al._ [12]
proposed a sea-land clutter classification method with multiple hidden layers,
which automatically extracts features at different levels from a large number
of sea-land clutter data. Considering the multi-scale and multi-resolution
characteristics of sea-land clutter, Li _et al._ [13] proposed a cross-scale
sea-land clutter classification method based on algebraic multigrid. However,
as a fully supervised classification framework, the sea-land clutter methods
in [12, 13] rely on a large number of labeled training samples. In general,
the labeling process is extremely cumbersome and requires expertise in the
field of OTHR. Different from [12, 13], Zhang _et al._ [14] proposed a semi-
supervised sea-land clutter data augmentation and classification method based
on auxiliary classifier variational autoencoder generative adversarial
network, which extracts the potential features of data from a small number of
labeled sea-land clutter samples to achieve high-quality generation of samples
and high-precision classification of clutter.

Inspired by game theory, Goodfellow _et al._ [15] proposed generative
adversarial network (GAN), which is an implicit probability density generative
model to learn the distribution of real data. GAN was originally proposed for
image synthesis [16, 17]. Subsequently, it has been applied to various fields,
such as machine translation, image super-resolution, natural language
processing, data augmentation, and domain adaptation, etc [18, 19, 20, 21,
22]. Since GAN is able to learn the intrinsic distribution of real data from
unlabeled samples, it can also be used for semi-supervised learning. The basic
idea of GAN for semi-supervised learning is to combine the conventional fully
supervised loss with the unsupervised loss of GAN, and then use it as the loss
function of semi-supervised classifier to guide the update of network
parameters. Up to now, various semi-supervised learning methods based on GAN
have been developed. Categorical generative adversarial network (CatGAN) [23]
aims to learn a discriminator which distinguishes the samples into $K$
categories, instead of learning a binary discriminator. The supervised loss of
the discriminator of CatGAN is the cross-entropy between the predicted
conditional distribution and the true label distribution of samples. Semi-
supervised generative adversarial network (SSGAN) [24] extends the output of
the discriminator into $K+1$ classes, where the first $K$ classes correspond
to real samples and the class $K+1$ corresponds to generated samples. On the
basis of SSGAN, Salimans _et al._ [25] pointed out that replacing standard
adversarial loss with feature matching loss is potentially beneficial to
obtain better semi-supervised classification performance. Dai _et al._ [26]
proved that the generator and the discriminator are hard to achieve optimality
simultaneously. Triple GAN [27] improves GAN by using a “three players”
framework with a generator, a discriminator and a classifier, which addresses
the issue that the generator and the discriminator of SSGAN can not be optimal
at the same time. Augmented bidirectional generative adversarial network
(Augmented BiGAN) [28] extends BiGAN to a semi-supervised framework, obtaining
the tangent spaces of the image manifold by the generator. The estimated
tangents infer the desirable invariances which can be injected into the
discriminator to improve semi-supervised classification performance.

GAN has the potential to extract deep level features from a large number of
unlabeled sea-land clutter samples and then assist a small number of labeled
samples for semi-supervised classification. In [14], we used a small number of
labeled samples for learning. Different from [14], in this work we use a large
number of unlabeled samples for semi-supervised learning. Since feature
matching loss is affected by the signal features extracted from the middle
layer of the discriminator, and the randomness of signal may lead to the
randomness of the features extracted from a single layer, we propose a joint
feature matching loss by weighting multi-layer feature matching loss.
Moreover, standard adversarial loss is disturbed by the randomness of signal,
which makes the training of SSGAN very difficult. To this end, we propose a
weighted loss by linearly weighting standard adversarial loss and joint
feature matching loss. On this basis, we propose an improved SSGAN framework
for sea-land clutter semi-supervised classification, namely weighted loss
semi-supervised generative adversarial network (WL-SSGAN). The experimental
results show that the proposed weighted loss is superior to both the
adversarial loss and the feature matching loss, and WL-SSGAN outperforms
conventional methods on sea-land clutter semi-supervised classification. In
summary, our main contributions are three-fold:

  * •

Taking into account of the randomness of sea-land clutter, we design a
weighted loss function, which first weights multi-layer feature matching loss
and then weights joint feature matching loss and adversarial loss to improve
the semi-supervised classification performance of sea-land clutter.

  * •

Based on the weighted loss, we propose a novel framework for sea-land clutter
semi-supervised classification, namely WL-SSGAN, which can use a small number
of labeled samples and a large number of unlabeled samples for end-to-end
semi-supervised classification.

  * •

The semi-supervised classification performance of WL-SSGAN is verified on a
sea-land clutter dataset. The experimental results show that WL-SSGAN can
improve the performance of fully supervised classifiers trained with only a
small number of labeled samples.

The remainder of this paper is organized as follows. In Section II, GAN is
briefly introduced. In Section III, the network structure, loss function and
training procedure of the proposed WL-SSGAN are described. In Section IV, the
sea-land clutter dataset used is introduced, and then the corresponding
network design and network training are given, followed by the performance
evaluation of WL-SSGAN. Section V draws the conclusions and highlights some
future work.

##  II Background

###  II-A Generative Adversarial Network

The network structure of the standard GAN is shown in Fig. 1. Specifically,
GAN consists of two neural networks, a generator and a discriminator. The role
of the generator is to fool the discriminator by generating samples consistent
with real-data distribution. The role of the discriminator is to distinguish
real samples in database from those generated by the generator. The back
propagation algorithm is adapted to optimize the network parameters of the
generator and the discriminator alternately. When the iteration reaches the
Nash equilibrium, the adversarial network is optimal. The generator has the
ability to generate optimal samples and the discriminator cannot distinguish
the source of samples. The loss function of GAN is as follows:

$\displaystyle L_{\text{GAN}}(D,G)=\min_{G}\max_{D}$
$\displaystyle\mathbb{E}_{x\sim p_{\text{data}}(x)}\log D(x)$ (1)
$\displaystyle+\mathbb{E}_{z\sim p_{z}(z)}[1-\log D(G(z))],$

where $x$ is real sample and follows the real-data distribution
$p_{\text{data}}(x)$, $z$ is noise and follows the a prior distribution
$p_{z}(z)$, $D(x)$ represents the probability that $x$ is a real sample,
$D(G(z))$ represents the probability that $D(z)$ is a generated sample, and
$\mathbb{E}$ is expectation operator. The loss function of the discriminator
is $\max\limits_{D}\mathbb{E}_{x\sim p_{\text{data}}(x)}\log
D(x)+\mathbb{E}_{z\sim p_{z}(z)}[1-\log D(G(z))]$. The loss function of the
generator is $\min\limits_{G}\mathbb{E}_{z\sim p_{z}(z)}[1-\log D(G(z))]$. The
training process of the generator and the discriminator can be regraded as a
two-player minimax game according to Eq. (1).

Figure 1: The network structure of GAN.

###  II-B Semi-supervised Generative Adversarial Network

As an improved GAN, semi-supervised generative adversarial network (SSGAN) has
a network structure as shown in Fig. 2. Different from GAN, SSGAN can not only
generate samples, but also can utilize label attribute to classify samples.

Figure 2: The network structure of SSGAN.

Specifically, SSGAN consists of two neural networks: a generator and a
discriminator/classifier. The training sample set $X$ of SSGAN consists of
three parts: a small number of labeled samples $X_{L}$, a large number of
unlabeled samples $X_{U}$ and generated samples $X_{G}$. Let
$\\{1,2,\dots,K\\}$ represent the label space for classification and $p_{D}$
indicates the output of the discriminator. Like [25], the loss function of
SSGAN consists of supervised loss and unsupervised loss:

$L_{\text{SSGAN}}=L_{\text{supervised}}+L_{\text{unsupervised}},$ (2)

where,

$L_{\text{supervised}}=-\mathbb{E}_{x,y\sim X_{L}}\log p_{D}(y|x,y\leq K),$
(3) $\displaystyle L_{\text{unsupervised}}=$ $\displaystyle-\mathbb{E}_{x\sim
X_{U}}\log[1-p_{D}(y=K+1|x)]$ (4) $\displaystyle-\mathbb{E}_{x\sim X_{G}}\log
p_{D}(y=K+1|x).$

It is seen from Eq. (3) that the supervised loss of SSGAN is consistent with
the loss function of conventional fully supervised learning, of which purpose
is to minimize the log conditional probability for labeled sample. In the
unsupervised loss, according to Eq. (4), the first term $-\mathbb{E}_{x\sim
X_{U}}\log[1-p_{D}(y=K+1|x)]$ aims to minimize the log conditional probability
of unlabeled sample being discriminated as real, and the second term
$-\mathbb{E}_{x\sim X_{U}}\log[1-p_{D}(y=K+1|x)]$ aims to minimize the log
probability of generated sample being discriminated as fake. Therefore, the
unsupervised loss part of SSGAN is consistent with the discriminator loss part
of the standard GAN. The only difference is that the latter treats real
samples as one class, while the former splits real samples into $K$ classes.

###  II-C Feature Matching Loss Function

In GAN, standard adversarial loss has the following shortcomings. (1) When the
generator is updated with sigmoid cross-entropy loss function, it is easy to
raise the problem of vanishing gradients for the samples that are on the
correct side of the decision boundary, but are still far from the real data
[29]. (2) The generator may learn only part of modes of real-sample
distribution, causing mode collapse. (3) A perfect generator has no ability to
improve the model’s generalization capacity. Different from adversarial loss,
feature matching loss aims to guide the generator to generate samples
conducive to improving the generalization ability of SSGAN. Feature matching
loss function is defined as follows.

$L_{\text{FM}}^{(l)}=\|\mathbb{E}_{x\sim X_{G}}\Phi^{(l)}(x)-\mathbb{E}_{x\sim
X_{U}}\Phi^{(l)}(x)\|^{2},$ (5)

where $\Phi^{(l)}(x)$ represents the feature extracted from the middle layer
$l$ of the discriminator. Obviously, samples that are far away from the
dataset manifold are meaningless for the classifier. So the samples generated
should be similar to the real data samples as a rule of thumb. Eq. (5)
represents that the L2 distance between generated samples and the real samples
computed in the feature space of discriminator, hence guaranteeing the
generated samples close to the real samples to some extent. Feature matching
loss is similar to perceptual loss computed in the feature space of a pre-
trained VGG model [30, 31].

##  III Sea-land Clutter Semi-supervised Classification Based on WL-SSGAN

In this section, we first describe the network structure and loss function of
the proposed WL-SSGAN for sea-land clutter semi-supervised classification.
Then, we introduce the training and algorithmic procedures of WL-SSGAN.

###  III-A Network Structure and Loss Function of WL-SSGAN

As the echo signals from OTHR, sea-land clutter samples have strong randomness
in data distribution. As a result, it is extremely difficult to train the
generator of SSGAN with standard adversarial loss. On the contrary, the
randomness enables feature matching loss to capture diverse sample features,
which to some extent alleviates mode collapse for the generator and over-
fitting for the discriminator. To this end, we propose WL-SSGAN, which is a
novel sea-land clutter semi-supervised classification framework. The network
structure of WL-SSGAN is shown in Fig. 3, which is consistent with that of
SSGAN. The discriminator loss $L_{D}$ of WL-SSGAN is consistent with that of
SS-GAN:

$L_{D}=L_{\text{supervised}}+L_{\text{unsupervised}}.$ (6)

Figure 3: The network structure of WL-SSGAN for sea-land clutter semi-
supervised classification.

Different from SSGAN, WL-SSGAN improves the generator loss of SSGAN.
Specifically, by linearly weighting standard adversarial loss $L_{\text{adv}}$
and feature matching loss $L_{\text{FM}}^{(l)}$, a weighted loss $L_{\text{WL-
SSGAN}}$ is proposed as follows,

$\displaystyle L_{\text{WL-SSGAN}}$ $\displaystyle=\alpha L_{\text{adv}}+\beta
L_{\text{FM}},$ (7)

where $\alpha+\beta=1$ and $\alpha,\beta\geq 0$. Additionally, in order to
improve the generalization ability of WL-SSGAN, the signal features extracted
from a single layer are not directly used for feature matching loss, but the
multi-layer signal features are weighted to obtain joint feature matching
loss. Let $l_{\max}$ denote the total number of middle layers of the
discriminator, $l_{\text{mul}}$ denote the set of selected feature layers,
which has $2^{l_{\max}}-1$ combinations, and $Ch^{(l)}$ and $Le^{(l)}$ denote
the channel number and signal length corresponding to the signal features of
layer $l$, respectively. The joint feature matching loss function is defined
as follows,

$\displaystyle L_{\text{FM}}=\sum\limits_{l\in
l_{\text{mul}}}\frac{1}{2Ch^{(l)}Le^{(l)}}L_{\text{FM}}^{(l)}$ (8)
$\displaystyle=\sum\limits_{l\in
l_{\text{mul}}}\frac{1}{2Ch^{(l)}Le^{(l)}}\|\mathbb{E}_{x\sim
X_{G}}\Phi^{(l)}(x)-\mathbb{E}_{x\sim X_{U}}\Phi^{(l)}(x)\|^{2}.$

For ease of understanding $L_{\text{WL-SSGAN}}$, substitute $1-p_{D}(y=K+1|x)$
into $L_{\text{unsupervised}}$ with $D(x)$. Then $L_{\text{unsupervised}}$ is
converted to the standard GAN game-value:

$\displaystyle L_{\text{unsupervised}}=$ $\displaystyle-\mathbb{E}_{x\sim
X_{U}}\log D(X)$ (9) $\displaystyle-\mathbb{E}_{x\sim X_{G}}\log[1-D(X)].$

Further, $L_{\text{WL-SSGAN}}$ can be rewritten as,

$\displaystyle L_{\text{WL-SSGAN}}$ (10)
$\displaystyle=-\alpha(\mathbb{E}_{x\sim X_{U}}\log D(X)+\mathbb{E}_{x\sim
X_{G}}\log[1-D(X)])$ $\displaystyle+\beta\sum\limits_{l\in
l_{\text{mul}}}\frac{1}{2Ch^{(l)}Le^{(l)}}\|\mathbb{E}_{x\sim
X_{G}}\Phi^{(l)}(x)-\mathbb{E}_{x\sim X_{U}}\Phi^{(l)}(x)\|^{2}.$

It should be emphasized that the work in [23, 24, 25] only took adversarial
loss or feature matching loss as the generator loss of SSGAN, that is, they
only emphasized the fidelity or diversity of the samples generated by the
generator. $L_{\text{WL-SSGAN}}$ combines the advantages of the above two loss
functions. In order to obtain better semi-supervised classification
performance, the proportion of weight factors $\alpha$ and $\beta$ can be
controlled to balance the contribution of adversarial loss and joint feature
matching loss to the training of WL-SSGAN. In addition, $l_{\text{mul}}$ can
also be selected to weight features of different layers to control the impact
of different signal features on the classification performance of WL-SSGAN.

###  III-B Training Procedure of WL-SSGAN

With the definitions of $L_{D}$ and $L_{\text{WL-SSGAN}}$, WL-SSGAN can be
trained in a similar way to SSGAN. The detailed training procedure of WL-SSGAN
is described in Algorithm 1. If all the input signals have labels, only the
parameters of classifier are updated for fully supervised learning according
to the supervised loss part $L_{\text{supervised}}$ of $L_{D}$. If the input
contains both labeled and unlabeled signals, the parameters of the generator
and the discriminator/classifier are alternately updated for semi-supervised
learning according to the discriminator/classifier loss part $L_{D}$ and
generator loss part $L_{\text{WL-SSGAN}}$.

Algorithm 1 Training of WL-SSGAN.

Initialize: Generator, Discriminator/Classifier. Let $N$ be the total number
of iterations.

for $k=1,\ldots,N$ do

Train Discriminator/Classifier:

(1) Sample a mini-batch of a small number of labeled samples $X_{L}$;

(2) Sample a mini-batch of a prior noise distribution $p_{z}(z)$;

(3) Sample a mini-batch of a large number of unlabeled samples $X_{U}$;

(4) Update Discriminator/Classifier by $L_{D}$.

Train Generator:

(5) Repeat Steps (2)-(3);

(6) Update Generator by $L_{\text{WL-SSGAN}}$.

end for

##  IV Experiment and Evaluation

In this section, we first describe the sea-land clutter dataset, and then
elaborate on the network design of WL-SSGAN. Further, the network training
details are introduced. At last, we evaluate the proposed WL-SSGAN.

###  IV-A Dataset

To verify the semi-supervised classification performance of WL-SSGAN, we adopt
sea-land clutter dataset as the benchmark dataset [14], which is the spectrum
of clutter obtained by OTHR. Fig. 4 shows the spectrum map of full range of a
beam from OTHR. We take a collection of spectrum of a range-azimuth cell as
sea-land clutter dataset, as shown in Fig. 5. Fig. 5LABEL:sub@fig:sea_real is
an example of sea clutter. The first-order Bragg peak of sea clutter is
generated by the Bragg resonance scattering of high-frequency electromagnetic
waves emitted by OTHR and ocean waves, showing double peaks symmetrical to
zero frequency. Fig. 5LABEL:sub@fig:land_real is an example of land clutter.
Since land has zero speed, it is observed that a single peak appears near zero
frequency. Fig. 5LABEL:sub@fig:sea-land_real is an example of sea-land
boundary clutter, which combines the features of sea clutter and land clutter,
showing triple peaks. The detailed description of sea-land clutter dataset
used to evaluate WL-SSGAN is shown in Table I, in which training data is used
for the training of WL-SSGAN and the evaluation of synthetic samples, and test
data is only used for the evaluation of synthetic samples and do not
participate in any training process.

Figure 4: The spectrum map of full range of a beam.

(a)

(b)

(c)

Figure 5: Sea-land clutter dataset. (a) An example of sea clutter. (b) An example of land clutter. (c) An example of sea-land boundary clutter. TABLE I: Detailed Description of Sea-Land Clutter Dataset Class | Attribute | Label | Quantity | Training Quantity | Test Quantity
---|---|---|---|---|---
1 | sea | 0 | 1000 | 700(70%) | 300(30%)
2 | land | 1 | 1000 | 700(70%) | 300(30%)
3 | sea-land | 2 | 1000 | 700(70%) | 300(30%)

Further, we randomly divide the original sea-land clutter dataset into labeled
and unlabeled datasets, in which the test data is consistent with the original
test data, and only the label attribute of some samples from the original
training data is removed. See Table II. For the convenience of description, we
denote the total number of labeled samples as $n_{\text{lab}}$. The semi-
supervised classification performance of WL-SSGAN is verified based on the
above datasets.

TABLE II: Sea-Land Clutter Training Data for Semi-supervised Classification Attribute | The number of labeled samples
---|---
Sea | 10 | 20 | 30 | 40 | 50 | 100 | 200 | 300 | 400 | 500
Land | 10 | 20 | 30 | 40 | 50 | 100 | 200 | 300 | 400 | 500
Sea-Land | 10 | 20 | 30 | 40 | 50 | 100 | 200 | 300 | 400 | 500
Total | 30 | 60 | 90 | 120 | 150 | 300 | 600 | 900 | 1200 | 1500

###  IV-B Network Design

Figure 6: The network structures of WL-SSGAN. Left:G and right:D.

The network structure of WL-SSGAN for sea-land clutter semi-supervised
classification is shown in Fig. 6.

G consists of 8-layer neural network: eight 1-dimensional deconvolutional
(DeConv1d) layers followed by Batch Normalization (BN1) and rectified linear
unit (ReLU)/hyperbolic tangent (Tanh) activation function. The input of G is a
100-dimensional latent vector (Gaussian noise). After layer 1, the number of
channels becomes 512, and the dimension of the feature vector is four. Then
after layer 2-7, the number of channels is decreased by one-half, and the
signal size is increased by two times per layer. Finally, after layer 8,
output a 512-dimensional synthetic sea-land clutter signal.

D consists of 8-layer neural network: seven 1-dimensional convolutional
(Conv1d) layers and one fully connected (FC) layer. All the Conv1d layers are
followed by BN1 and LeakyReLU activation function with a slope of 0.2. In
addition, we add dropout with the probability of 0.5 after each Conv1d layer
to prevent over-fitting. The input of D is a $512$-dimensional sea-land
clutter signal. After layer 1, the number of channels is increased by eight
times, and the signal size is decreased by one-half. Then after layers 2-7,
the number of channels is increased by two times per layer, and the signal
size is decreased by one-half per layer. Next, flatten the resulting feature
vector. Finally, output a 3-dimensional classification result.

###  IV-C Network Training

TABLE III: Experiment Environment Environment | Version
---|---
System | Windows 10 (64-bit)
GPU | NVIDIA GeForce RTX 3090
CUDA | 11.6
python | 3.9.0 (in Anaconda 4.11.0)
torch | 1.11.0
torchvision | 0.12.0
numpy | 1.22.3
matplotlib | 3.5.1
TABLE IV: The Hyperparameter Configuration of WL-SSGAN for Sea-Land Clutter Semi-supervised Classification Configuration | Default
---|---
Training Epoch | 1000
Batch Size | 64
Learning Rate | 0.0001
Cross Entropy Loss | -
Adma Optimizer | beta1:0.5, beta2:0.999
Data Normalization | [-1,1]
Weight Initialization | -
$\alpha,\beta$(weight factors) | $\alpha+\beta=1,\alpha,\beta\geq 0$
$l_{\text{mul}}$(feature layers) | $[0,2^{l_{\max}}-1]$
$n_{\text{lab}}$ (labeled samples) | -

The experiment environment used in this work for sea-land clutter semi-
supervised classification is shown in Table III. See Table IV for the training
details and hyperparameter configuration of WL-SSGAN.

Fig. 7 and Fig. 8 plot the loss curves and the classification accuracy of WL-
SSGAN with $\alpha=0.5$, $\beta=0.5$, $l_{\text{mul}}=[1,2,3,4,5,6,7]$ and
$n_{\text{lab}}=2100$, from which one can conclude that the training of WL-
SSGAN is stable and converges rapidly. From Fig. 8, one can observe that WL-
SSGAN can achieve high-precision clutter classification.

(a)

(b)

Figure 7: Loss curves of WL-SSGAN for sea-land clutter synthesis (with
$\alpha=0.5$, $\beta=0.5$, $l_{\text{mul}}=[1,2,3,4,5,6,7]$ and
$n_{\text{lab}}=2100$). (a) Loss curve for G. (b) Loss curve for D.

(a)

(b)

Figure 8: Loss curve and accuracy curve of WL-SSGAN for sea-land clutter
classification (with $\alpha=0.5$, $\beta=0.5$,
$l_{\text{mul}}=[1,2,3,4,5,6,7]$ and $n_{\text{lab}}=2100$). (a) Loss curve.
(b) Accuracy curve.

###  IV-D Network Evaluation

Next, we consider the impact of the number of labeled samples
$n_{\text{lab}}$, the proportion of $\alpha$ and $\beta$, and the selection of
$l_{\text{mul}}$ on the classification performance of WL-SSGAN. Additionally,
the classification performance of WL-SSGAN is compared with that of a fully
supervised classifier (i.e. the discriminator of WL-SSGAN) trained with only a
small number of labeled samples.

First, suppose that all the middle layer features of the discriminator
contribute to generator loss, that is, $l_{\text{mul}}=\\{1,2,3,4,5,6,7\\}$.
Thus, joint feature matching loss is fixed to evaluate the impact of the
proportion of $\alpha$ and $\beta$ on the classification performance of WL-
SSGAN. Table V shows the average classification accuracy after WL-SSGAN
reaches steady state.

TABLE V: The classification accuracy of WL-SSGAN under different values of $\alpha$ and $\beta$ $(\alpha,\beta)$ $\emph{n}_{\text{lab}}$ | 30 | 60 | 90 | 120 | 150 | 300 | 600 | 900 | 1200 | 1500
---|---|---|---|---|---|---|---|---|---|---
supervised | 0.7132 | 0.8046 | 0.8310 | 0.8521 | 0.8678 | 0.9039 | 0.9226 | 0.9372 | 0.9473 | 0.9627
$(0.0,1.0)$ | 0.8663 | 0.8762 | 0.9517 | 0.9559 | 0.9618 | 0.9657 | 0.9678 | 0.9701 | 0.9790 | 0.9799
$(0.1,0.9)$ | 0.8886 | 0.9129 | 0.9578 | 0.9645 | 0.9756 | 0.9764 | 0.9800 | 0.9841 | 0.9854 | 0.9901
$(0.2,0.8)$ | 0.8679 | 0.9079 | 0.9618 | 0.9628 | 0.9643 | 0.9703 | 0.9715 | 0.9832 | 0.9836 | 0.9862
$(0.3,0.7)$ | 0.8576 | 0.9070 | 0.9584 | 0.9677 | 0.9739 | 0.9768 | 0.9793 | 0.9827 | 0.9879 | 0.9888
$(0.4,0.6)$ | 0.8883 | 0.9035 | 0.9537 | 0.9615 | 0.9719 | 0.9771 | 0.9782 | 0.9821 | 0.9835 | 0.9881
$(0.5,0.5)$ | 0.8732 | 0.9023 | 0.9512 | 0.9561 | 0.9662 | 0.9720 | 0.9758 | 0.9812 | 0.9829 | 0.9857
$(0.6,0.4)$ | 0.8636 | 0.9038 | 0.9514 | 0.9600 | 0.9626 | 0.9769 | 0.9810 | 0.9824 | 0.9836 | 0.9837
$(0.7,0.3)$ | 0.8921 | 0.9280 | 0.9654 | 0.9700 | 0.9769 | 0.9801 | 0.9832 | 0.9868 | 0.9890 | 0.9908
$(0.8,0.2)$ | 0.8746 | 0.9012 | 0.9513 | 0.9634 | 0.9655 | 0.9723 | 0.9729 | 0.9804 | 0.9827 | 0.9879
$(0.9,0.1)$ | 0.8601 | 0.9132 | 0.9487 | 0.9623 | 0.9713 | 0.9746 | 0.9769 | 0.9812 | 0.9833 | 0.9899
$(1.0,0.0)$ | 0.8734 | 0.9179 | 0.9524 | 0.9637 | 0.9700 | 0.9734 | 0.9746 | 0.9832 | 0.9877 | 0.9882

The following conclusions can be drawn from Table V. (1) The classification
performance of WL-SSGAN is better than that of fully supervised classifier,
indicating that WL-SSGAN can extract potential features related to sea-land
clutter classification from a large number of unlabeled samples. Therefore,
WL-SSGAN can improve the classification performance of fully supervised
classifier trained with only a small number of labeled samples. (2) With the
decrease of $n_{\text{lab}}$, the classification performance improvement
achieved by WL-SSGAN is more and more obvious. This is because when the number
of training samples is small, fully supervised classifier is prone to fall
into over-fitting. (3) When the layers $l_{\text{mul}}$ of joint feature
matching loss $L_{\text{FM}}$ is fixed, the proportion of $\alpha$ and $\beta$
obviously have an impact on classification accuracy. The classification
accuracy obtained only by using adversarial loss $L_{\text{adv}}$ or joint
feature matching loss $L_{\text{FM}}$, that is, $(\alpha,\beta)=(1.0,0.0)$ or
$(\alpha,\beta)=(0.0,1.0)$, is not the best. The best classification accuracy
is obtained when $(\alpha,\beta)=(0.7,0.3)$. It is concluded that the proposed
weighted loss $L_{\text{WL-SSGAN}}$ is superior to both $L_{\text{adv}}$ and
$L_{\text{FM}}$.

Second, suppose that adversarial loss and joint feature matching loss have the
same contribution to generator loss, that is, $(\alpha,\beta)=(0.5,0.5)$.
Thus, the contribution of adversarial loss and joint feature matching loss to
WL-SSGAN is fixed to evaluate the selection of $l_{\text{mul}}$ on the
classification performance of WL-SSGAN. Table VI shows the average
classification accuracy after WL-SSGAN reaches steady state.

TABLE VI: The classification accuracy of WL-SSGAN under different value of $l_{\rm{mul}}$ $l_{\text{mul}}$ $\emph{n}_{\text{lab}}$ | 30 | 60 | 90 | 120 | 150 | 300 | 600 | 900 | 1200 | 1500
---|---|---|---|---|---|---|---|---|---|---
supervised | 0.7132 | 0.8046 | 0.8310 | 0.8521 | 0.8678 | 0.9039 | 0.9226 | 0.9372 | 0.9473 | 0.9627
$\\{1\\}$ | 0.8621 | 0.9030 | 0.9500 | 0.9535 | 0.9626 | 0.9702 | 0.9714 | 0.9787 | 0.9813 | 0.9844
$\\{1,2\\}$ | 0.8567 | 0.8985 | 0.9521 | 0.9543 | 0.9676 | 0.9687 | 0.9721 | 0.9830 | 0.9856 | 0.9875
$\\{1,2,3\\}$ | 0.8765 | 0.9063 | 0.9573 | 0.9585 | 0.9662 | 0.9670 | 0.9743 | 0.9831 | 0.9854 | 0.9867
$\\{1,2,3,4\\}$ | 0.8611 | 0.9163 | 0.9551 | 0.9610 | 0.9623 | 0.9731 | 0.9745 | 0.9833 | 0.9841 | 0.9852
$\\{1,2,3,4,5\\}$ | 0.8673 | 0.9057 | 0.9568 | 0.9573 | 0.9651 | 0.9710 | 0.9764 | 0.9832 | 0.9841 | 0.9855
$\\{1,2,3,4,5,6\\}$ | 0.8960 | 0.9209 | 0.9603 | 0.9643 | 0.9701 | 0.9746 | 0.9782 | 0.9847 | 0.9868 | 0.9903
$\\{1,2,3,4,5,6,7\\}$ | 0.8921 | 0.9020 | 0.9553 | 0.9565 | 0.9666 | 0.9721 | 0.9732 | 0.9821 | 0.9833 | 0.9864

The following conclusions can be drawn from Table VI. (1) When the proportion
of weight factors $\alpha$ and $\beta$ of adversarial loss $L_{\text{adv}}$
and joint feature matching loss $L_{\text{FM}}$ are fixed, the selection of
$l_{\text{mul}}$ obviously has an impact on classification accuracy. The
classification accuracy obtained by using the feature matching loss
$L_{\text{FM}}^{(1)}$ with single layer or the joint feature matching loss
with all layers $L_{\text{FM}}^{(1)-(7)}$, that is, $l_{\text{mul}}=\\{1\\}$
or $l_{\text{mul}}=\\{1,2,3,4,5,6,7\\}$, is not the best. The best
classification accuracy is obtained when $l_{\text{mul}}=\\{1,2,3,4,5,6\\}$.
This indicates that: (1) The proposed joint feature matching loss
$L_{\text{FM}}$ is superior to conventional feature matching loss
$L_{\text{FM}}^{(1)}$. (2) Not all the middle layer features of discriminator
are weighted to obtain the best semi-supervised classification performance. On
the contrary, the features extracted from some layers may inhibit
classification performance. It is concluded that the best classification
performance depends on the appropriate selection of feature layers.

Further, we compare WL-SSGAN with traditional full supervised classification
methods [14]. See Table VII. It is seen that the classification accuracy of
WL-SSGAN using only 90 labeled samples is greater than those of Random Forest,
KNN, Logistic Regression and SVM using all labeled samples. The WL-SSGAN using
only 1200 labeled samples has the best classification accuracy.

TABLE VII: Classification Accuracy of Different Classifiers on Sea-Land Clutter Dataset Classifiers | Accuracy
---|---
Random Forest | 94.18% (all labeled)
KNN | 94.76% (all labeled)
Logistic Regression | 95.36% (all labeled)
SVM | 95.82% (all labeled)
WL-SSGAN (Ours) | 96.03% (90 labeled)
FCN | 97.93% (all labeled)
WL-SSGAN (Ours) | 98.01% (300 labeled)
ResNet18 | 98.87% (all labeled)
WL-SSGAN (Ours) | 98.90% (1200 labeled)

Besides, we compare the classification accuracy of the different semi-
supervised classifiers on different numbers of labeled sea-land clutter
samples, in which the network structure and hyperparameter configuration of
all methods are consistent. See Fig. 9. It is seen that WL-SSGAN has
advantages over traditional methods in semi-supervised classification.

Figure 9: Classification accuracy of different semi-supervised classifiers on
different numbers of labeled sea-land clutter samples.

Finally, we evaluate the sample synthesis performance of WL-SSGAN. We use
trained WL-SSGAN in Section IV-C to synthesize 2100 sea-land clutter samples.
See Fig. 10 for some examples of synthetic samples. Visually, most of
synthetic samples are consistent with the real samples. Note that WL-SSGAN
cannot synthesize samples with specific categories, while we observe an
abundance of sea clutter samples, land clutter samples and sea-land boundary
clutter samples from the synthetic samples. This illustrates that WL-SSGAN
also has good sample synthesis ability. Further, we evaluate the quality of
the synthesized sea-land clutter in terms of the statistical evaluation
metrics including absolute distance (AD), cosine similarity (CS) and Pearson
correlation coefficient (PCC) [14]. See Table VIII for the evaluation results.
It is seen that the sample synthesis performance of WL-SSGAN outperforms those
of DCGAN, WGAN-GP, LS-GAN, VAE-GAN and CGAN. WL-SSGAN obtains competitive
results compared to that of AC-GAN. The sample synthesis ability of WL-SSGAN
is slightly lower than that of AC-VAEGAN. Note that WL-SSGAN is more focused
on semi-supervised classification.

TABLE VIII: The Evaluation Results of Synthetic Sea-Land Clutter Samples GAN Methods | Statistical Evaluation
---|---
| AD | CS | PCC
DCGAN | 0.0313 | 0.7548 | 0.7534
WGAN-GP | 0.0283 | 0.7671 | 0.7483
LSGAN | 0.0379 | 0.7106 | 0.7521
VAEGAN | 0.0244 | 0.7337 | 0.7518
CGAN | 0.0221 | 0.7882 | 0.7458
AC-GAN | 0.0161 | 0.8036 | 0.7649
AC-VAEGAN | 0.0160 | 0.8988 | 0.7897
WL-SSGAN(Ours) | 0.0173 | 0.8672 | 0.7558

(a)

(b)

(c)

Figure 10: The sea-land clutter samples synthesized by WL-SSGAN (with
$\alpha=0.5$, $\beta=0.5$, $l_{\text{mul}}=[1,2,3,4,5,6,7]$ and
$n_{\text{lab}}=2100$). (a) sea clutter. (b) land clutter. (c) sea-land
boundary clutter.

##  V Conclusions and Future Work

A novel sea-land clutter semi-supervised classification framework for OTHR was
proposed, namely WL-SSGAN. Considering the randomness of sea-land clutter
samples, joint feature matching loss was proposed by weighting multi-layer
feature matching loss. Further, adversarial loss and joint feature matching
loss were weighted to alleviate the effect of sea-land clutter samples with
obvious noise on semi-supervised classification performance. The experimental
results showed that WL-SSGAN can achieve semi-supervised classification of
sea-land clutter, and the proposed weighted loss is superior to both the
adversarial loss and the feature matching loss.

In future work, there are two issues that need to be addressed: (1) While WL-
SSGAN improves classification performance by utilizing a large number of
unlabeled sea-land clutter samples, the computational cost increases.
Therefore, it is desirable to improve the semi-supervised classification
framework of WL-SSGAN to take into account both the classification performance
and the computational cost. (2) The proposed weighted loss was determined by
the selection of $l_{\text{mul}}$ and the proportion of $\alpha$ and $\beta$.
The values of $l_{\text{mul}}$, $\alpha$ and $\beta$ are considered as
hyperparameters in the experiment. We intend to design the optimal selection
of $l_{\text{mul}}$ and the optimal proportion of $\alpha$ and $\beta$.
Furthermore, we expect to find a parameter adaptive optimization scheme to
automatically seek the values of $l_{\text{mul}}$, $\alpha$ and $\beta$ with
the best semi-supervised classification performance.

## References

  * [1] H. Geng, Y. Liang, and Y. Cheng, “Target state and Markovian jump ionospheric height bias estimation for OTHR tracking systems,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 50, no. 7, pp. 2599–2611, 2020.
  * [2] H. Lan, Z. Wang, X. Bai, Q. Pan, and K. Lu, “Measurement-level target tracking fusion for over-the-horizon radar network using message passing,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 57, no. 3, pp. 1600–1623, 2020.
  * [3] T. Thayaparan, D. Dupont, Y. Ibrahim, and R. Riddolls, “High-frequency ionospheric monitoring system for over-the-horizon radar in Canada,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 57, no. 9, pp. 6372–6384, 2019.
  * [4] J. Hu, C. Jian, C. Zhuo, H. Li, and J. Xie, “Knowledge-aided ocean clutter suppression method for sky-wave over-the-horizon radar,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 15, no. 3, pp. 355–358, 2018.
  * [5] Z. Guo, Z. Wang, Y. Hao, H. Lan, and Q. Pan, “An improved coordinate registration for over-the-horizon radar using reference sources,” _Electronics_ , vol. 10, no. 24, pp. 3086–1–3086–20, 2021.
  * [6] Y. Han, Z. Yang, and X. Chu, “Research on the correction of ionospheric distortion for ship detection in OTHR (in Chinese),” _Modern Radar_ , vol. 25, no. 10, pp. 5–8, 2003.
  * [7] F. Cuccoli, F. Sermi, L. Facheris, and D. Giuli, “Sea-land transitions identification for coordinate registration of over the horizon sky-wave radar: numerical model for performance analysis,” in _The 11-th International Radar Symposium_. IEEE, 2010, pp. 1–4.
  * [8] A. Cacciamano, A. Capria, D. Olivadese, F. Berizzi, E. Dalle Mese, and F. Cuccoli, “A coordinate registration technique for OTH sky-wave radars based on 3D ray-tracing and sea-land transitions,” in _PIERS Proceedings, Kuala Lumpur, Malaysia_ , 2012, pp. 146–150.
  * [9] Z. Jin, Q. Pan, C. Zhao, and W. Zhou, “SVM based land/sea clutter classification algorithm,” _Applied Mechanics and Materials_ , vol. 236-237, pp. 1156–1162, 2012.
  * [10] M. Turley, R. Gardiner-Garden, and D. Holdsworth, “High-resolution wide area remote sensing for HF radar track registration,” in _Proceedings of 2013 International Conference on Radar_. IEEE, 2013, pp. 128–133.
  * [11] D. A. Holdsworth, “Skywave over-the-horizon radar track registration using earth surface and infrastructure backscatter,” in _Proceedings of 2017 IEEE Radar Conference_. IEEE, 2017, pp. 0986–0991.
  * [12] C. Li, Z. Wang, Z. Zhang, H. Lan, and K. Lu, “Sea/land clutter recognition for over-the-horizon radar via deep CNN,” in _Proceedings of 2019 International Conference on Control, Automation and Information Sciences_. IEEE, 2019, pp. 1–5.
  * [13] C. Li, Y. Zhang, Z. Wang, K. Lu, and Q. Pan, “Cross-scale land/sea clutter classification method for over-the-horizon radar based on algebraic multigrid (in Chinese),” _Acta Electronica Sinica_ , pp. 1–9, 2022.
  * [14] X. Zhang, Z. Wang, K. Lu, Q. Pan, and Y. Li, “Data augmentation and classification of sea-land clutter for over-the-horizon radar using AC-VAEGAN,” _Submitted for Publication_.
  * [15] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in Neural Information Processing Systems_ , vol. 27, pp. 1–9, 2014.
  * [16] E. L. Denton, S. Chintala, A. Szlam, and R. Fergus, “Deep generative image models using a Laplacian pyramid of adversarial networks,” _Advances in Neural Information Processing Systems_ , vol. 28, pp. 1486–1494, 2015.
  * [17] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” in _Proceedings of International Conference on Machine Learning_ , 2016, pp. 1–16.
  * [18] L. Wu, Y. Xia, F. Tian, L. Zhao, T. Qin, J. Lai, and T.-Y. Liu, “Adversarial neural machine translation,” in _Proceedings of Asian Conference on Machine Learning_. PMLR, 2018, pp. 534–549.
  * [19] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks,” in _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_ , 2018, pp. 0–0.
  * [20] L. Yu, W. Zhang, J. Wang, and Y. Yu, “Seqgan: Sequence generative adversarial nets with policy gradient,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 31, no. 1, 2017, pp. 2852–2858.
  * [21] F. Fahimi, S. Dosen, K. K. Ang, N. Mrachacz-Kersting, and C. Guan, “Generative adversarial networks-based data augmentation for brain–computer interface,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 9, pp. 4039–4051, 2020.
  * [22] Q. Kang, S. Yao, M. Zhou, K. Zhang, and A. Abusorrah, “Effective visual domain adaptation via generative adversarial distribution matching,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 9, pp. 3919–3929, 2020.
  * [23] J. T. Springenberg, “Unsupervised and semi-supervised learning with categorical generative adversarial networks,” _arXiv preprint arXiv:1511.06390_ , 2015.
  * [24] A. Odena, “Semi-supervised learning with generative adversarial networks,” _arXiv preprint arXiv:1606.01583_ , 2016.
  * [25] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,” _Advances in Neural Information Processing Systems_ , vol. 29, pp. 1–9, 2016.
  * [26] Z. Dai, Z. Yang, F. Yang, W. W. Cohen, and R. R. Salakhutdinov, “Good semi-supervised learning that requires a bad GAN,” _Advances in Neural Information Processing Systems_ , vol. 30, pp. 6510–6520, 2017.
  * [27] C. Li, T. Xu, J. Zhu, and B. Zhang, “Triple generative adversarial nets,” _Advances in Neural Information Processing Systems_ , vol. 30, pp. 4088–4098, 2017.
  * [28] A. Kumar, P. Sattigeri, and T. Fletcher, “Semi-supervised learning with GANs: Manifold invariance with improved inference,” _Advances in Neural Information Processing Systems_ , vol. 30, pp. 5534–5544, 2017.
  * [29] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 2794–2802.
  * [30] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [31] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _Proceedings of European Conference on Computer Vision_. Springer, 2016, pp. 694–711.


# Overview of GUA-SPA at IberLEF 2023: Guarani-Spanish Code Switching Analysis

Luis Chiruzzo,1 Marvin Agüero-Torales,2,3 Gustavo Giménez-Lugo,4 Aldo
Alvarez,5
Yliana Rodríguez,1 Santiago Góngora,1 Thamar Solorio6,7
1Universidad de la República, Montevideo, Uruguay
2Universidad de Granada, Granada, Spain
3Global CoE of Data Intelligence, Fujitsu, Madrid, Spain
4Universidade Tecnologica Federal do Paraná, Curitiba, PR, Brazil
5Universidad Nacional de Itapúa, Encarnación, Paraguay
6University of Houston, Houston, TX, USA
7Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United
Arab Emirates
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
##  1 Introduction

This paper presents the GUA-SPA shared task for code-switching analysis in
Guarani and Spanish at IberLEF 2023 [Jiménez-Zafra, Rangel, and Montes-y
Gómez,2023].

Guarani is a South American indigenous language that belongs to the Tupi-
Guarani family. Spanish is a Romance language that belongs to the Indo-
European family, and both languages have been in contact in the South American
region for about 500 years [Rodríguez Gutiérrez and Núñez Méndez,2018],
resulting in many interesting linguistic varieties with different levels of
mixture. Both, Guarani and Spanish, are Paraguay’s official languages [Ley de
Lenguas 4251,2010]. According to the most recent census in
Paraguay,111https://censo2022.ine.gov.py/noticias-actividades/21-de-febrero-
dia-internacional-de-la-lengua-materna/ most of the country’s population
speaks at least some Guarani, and there is a high prevalence of Guarani-
Spanish bilingualism in urban areas, while Guarani monolingualism is more
limited to rural areas.

Bilingual speakers often make use of their two languages at the same time,
mixing them in different ways, in a linguistic phenomenon called code-
switching [Joshi,1982]. This phenomenon is very frequent in situations where
two or more languages are in contact, for example, in linguistic borders, or
in countries with large immigrant populations. In Paraguay, this has resulted
in several identified language varieties that combine Guarani and Spanish
[Kallfell,2016]. Two of the most common varieties of Guarani spoken in
Paraguay are called Jopara and Jehe’a (both are Guarani words for describing a
mix of things), and they often incorporate many Spanish loans of words or
entire phrases, with varying levels of morphosyntactic adaptation to Guarani
grammar [Chiruzzo et al.,2022].

In this task, we propose to analyze a set of texts extracted from Paraguayan
tweets and news articles, where it is usual to see these Jopara or Jehe’a
varieties, and also the use of Spanish sentences that include Guarani
loanwords. These texts were annotated by the organisers to create the shared
task dataset, which is available online.222https://github.com/pln-fing-
udelar/gua-spa-2023 Using this dataset, we proposed three tasks: language
identification (Task 1), NER (Task 2), and a novel classification task for the
way Spanish spans are used in the code-switched context (Task 3).

The paper is structured as follows: in Section 2 we mention the previous work
on code-switching analysis and Guarani processing; in Section 3 we detail the
three tasks that make up this shared task; in Section 4 we present the corpus
created for this work; in Section 5 we present the competition, the systems
built by the participant teams and their results; and in Section 6 we detail
the conclusions of this work.

##  2 Background

There have been a number of competitions focusing on detection and analysis of
code-switching, starting in [Solorio et al.,2014] with language identification
in code-switched data for some language pairs, including Spanish-English.
Later on these competitions started to include more complex tasks in code-
switched contexts, such as NER [Aguilar, Kar, and Solorio,2020] and machine
translation into the code-switched languages [Chen et al.,2022]. As far as we
know, this is the first time Guarani, or any other American indigenous
language, is present as subject of a code-switching analysis competition, the
first competition of NER in Guarani, and also the first dataset with this kind
of annotations built for an American indigenous language.

However this is not the first work on Guarani identification. [Agüero-Torales,
Vilares, and López-Herrera,2021] tried to detect tweets in Guarani with
heuristics using unigrams and bigrams and other out-of-the-box language
identifiers that include Guarani, such as Polyglot [Al-Rfou’, Perozzi, and
Skiena,2013], fastText [Joulin et al.,2016] and NLTK’s [Bird, Klein, and
Loper,2009]
implementation333https://www.nltk.org/_modules/nltk/classify/textcat.html of
the method introduced by [Cavnar and Trenkle,1994]. [Góngora, Giossa, and
Chiruzzo,2021] used a Naïve Bayes model and a word-based heuristic to detect
Guarani tweets. Another big effort in language identification is HeLI-OTS
[Jauhiainen, Jauhiainen, and Lindén,2022], which indeed includes Guarani.
Their authors were also participants of this shared task, as we will further
describe in Section 5.2.

Guarani is considered a low resource language [Joshi et al.,2020] because,
despite having millions of speakers, it does not have many digital resources
to work with, its written use online is scarce, and it has been mostly under-
researched from the NLP perspective. The situation of this and other American
indigenous languages could change in the future as there are now some
initiatives to build resources for these languages [Mager et al.,2021], but
there is still a long way to go. Spanish, on the other hand, belongs to the
small set of very resource-rich languages [Joshi et al.,2020], which is good
for this competition as there are many tools for Spanish that could be
leveraged to see how well they work in this context.

##  3 Tasks

The code-switched text detection and analysis tasks proposed in this challenge
include the identification of the language used in each span of text, the
named entities mentioned in the text, and the way Spanish is used. The
challenge was structured as the following three tasks.

###  3.1 Task 1: Language identification in code-switched data

Given a text (sequence of tokens), label each token of the sequence with one
of the following categories:

  * •

gn: It is a Guarani token.

  * •

es: It is a Spanish token.

  * •

ne: It is part of a named entity (either in Guarani or in Spanish).

  * •

mix: The token is a mixture between Guarani and Spanish. For example a verb
with a Spanish root that has been transformed into Guarani morphology, like:
osuspendeta (he/she will suspend).

  * •

foreign: Used for tokens that are in languages other than Guarani or Spanish.

  * •

other: Used for other types of tokens that are invariant to language, like
punctuation, emojis and URLs.

Examples:

    * che kuerai de pagar 6000 gs. por una llamada de 40 segundos . son aliados del gobierno parece ustedes

Could be tagged as:

che/gn kuerai/gn de/es pagar/es 6000/other gs./other por/es una/es llamada/es
de/es 40/other segundos/es ./other son/es aliados/es del/es gobierno/es
parece/es ustedes/es

    * Ministerio de Salud omombe’u ko’ã káso malaria ojuhúva importado Guinea Ecuatorial guive ha oîma jesareko ohapejokóvo jeipyso .

Could be tagged as:

Ministerio/ne de/ne Salud/ne omombe’u/gn ko’ã/gn káso/mix malaria/es
ojuhúva/gn importado/es Guinea/ne Ecuatorial/ne guive/gn ha/gn oîma/gn
jesareko/gn ohapejokóvo/gn jeipyso/gn ./other

The metrics for Task 1 are accuracy, weighted precision, weighted recall and
weighted F1. The main metric is weighted F1.

We implemented the following baseline for this task: choose the most frequent
category for each word in the training corpus, or choose other if the word is
not in the training corpus.

###  3.2 Task 2: Named entity classification

Given a text (sequence of tokens), identify the named entities as spans in the
text, and classify each one with a category: person, location or organization.

Examples:

    * [ORG Ministerio de Salud] omombe’u ko’ã káso malaria ojuhúva importado [LOC Guinea Ecuatorial] guive ha oîma jesareko ohapejokóvo jeipyso .

    * [PER Ministra de Hacienda Lea Giménez] he’i oñepromulga léi capitalidad ary 2014

For this task, the tokens must be marked using the BIO notation, with the
following labels: ne-b-per, ne-b-loc, ne-b-org, ne-i-per, ne-i-loc, ne-i-org.
The metrics for Task 2 are precision, recall and F1, either labeled or
unlabeled. The criterion for finding a named entity is exact match. The main
metric is labeled F1.

The baseline for this task is the following: take as an entity span any
sequence of tokens labeled as ne by the baseline for Task 1, then choose the
most frequent category (per, org, loc) for the first word of the sequence in
the training corpus. If it is an unknown word, choose the category per.

###  3.3 Task 3: Spanish code classification

Given a text (sequence of tokens), identify spans of text in Spanish and label
them in one of these categories:

  * •

change in code (CC): the text keeps all the characteristics of Spanish.

  * •

unadapted loan (UL): the Spanish text could be partially adapted in some ways
to Guarani syntax, but it is not fully merged into Guarani, in particular it
does not present orthographic transformations.

Examples:

    * che kuerai [CC de pagar 6000 gs. por una llamada de 40 segundos . son aliados del gobierno parece ustedes]

    * Okañývo pe Policía Nacional ha orekóva caso omomarandúvo Fiscalía , peteî [UL investigación ámbito penal] .

For this task, the tokens must be marked using the following BIO labels: es-b-
cc, es-b-ul, es-i-cc, es-i-ul. The metrics for Task 3 are precision, recall
and F1, either labeled or unlabeled. The criterion for finding a Spanish span
is exact match. The main metric is labeled F1.

The baseline for this task is analogous to the one for Task 2: take as a
Spanish span any sequence of tokens labeled as es by the baseline for Task 1,
then choose the most frequent category (cc, ul) for the first word of the
sequence in the training corpus. If it is an unknown word, choose the category
cc.

##  4 Corpus

The corpus of the task consists of 1500 texts, about 25 thousand tokens, split
it in 76%-12%-12% for train-dev-test. The data contains sentences extracted
from news articles [Chiruzzo et al.,2022, Chiruzzo et al.,2020] and tweets
[Agüero-Torales, López-Herrera, and Vilares,2023, Agüero-Torales, Vilares, and
López-Herrera,2021, Góngora, Giossa, and Chiruzzo,2022, Ríos, Amarilla, and
Lugo,2014]. The characteristics of this corpus are shown in Table 1. As seen
in the table, the corpus contains more Guarani tokens than any of the other
classes, for example there are on average three Guarani tokens per each
Spanish token. Although there are few mixed tokens, around 2% of the corpus,
they are a very interesting class to analyze as it presents particular
challenges.

| Train | Dev | Test | Total
---|---|---|---|---
Texts | 1140 | 180 | 180 | 1500
gn | 7698 | 1241 | 1193 | 10132
es | 5058 | 812 | 815 | 6685
mix | 388 | 52 | 47 | 487
ne | 2510 | 414 | 331 | 3255
foreign | 129 | 14 | 8 | 151
other | 3220 | 456 | 463 | 4139
Total tokens | 19003 | 2989 | 2857 | 24849
ne-per | 663 | 85 | 81 | 829
ne-org | 562 | 95 | 89 | 746
ne-loc | 240 | 58 | 33 | 331
Total ne spans | 1465 | 238 | 203 | 1906
es-ul | 636 | 80 | 95 | 811
es-cc | 621 | 70 | 112 | 803
Total es spans | 1257 | 150 | 207 | 1614
Table 1: Composition of the dataset.

The annotation was done in two phases: A first phase consisted in four
annotators annotating a sample of 54 sentences (around 750 tokens). The aim of
the first phase was to unify criteria and gauge how hard it was to make the
annotations. The four annotators obtained a Fleiss Kappa inter-annotator
agreement of 0.836 for Task 1, which indicates very substantial agreement.
Only two of the annotators annotated the data for Task 2 during this phase,
and we calculated their agreement using average F1, obtaining 0.926 between
the two annotators, which we also consider a substantial agreement. We came up
with the idea for Task 3 during this pilot annotation, but it was considerably
harder to annotate, with many ambiguous cases that were difficult to
generalize.

In the final phase of annotation, six annotators fluent in Spanish and with
some knowledge of Guarani labeled data for Tasks 1 and 2, but only the three
annotators with more knowledge of Guarani annotated the data for Task 3.

After the annotation was over we estimated the annotator agreement in Task 3
in the following way: we took a subsample of 90 texts that contained at least
one token in Spanish, 30 for each annotator that participated in the task. The
three annotators completed the annotation of the 90 texts, and we compared the
annotations as the F1 measure of one annotator against another. The average F1
obtained in this way was 0.758 for the labeled F1 considering exact match
which, although not as substantial as the agreement for Task 2, was still
quite high.

##  5 Competition

The competition ran between March 22 and June 7 (2023) on the CodaLab
platform444https://codalab.lisn.upsaclay.fr/competitions/11030 [Pavao et
al.,2022]. A total of 20 users registered to participate, but the number of
participants that submitted results was considerably lower: three of them
participated both in the development and the evaluation phase, and the other
two participated in a single phase.

###  5.1 Phases

The competition consisted of three phases:

Development phase: from March 22 to May 23. This phase started with the
publication of the training and development sets. During this phase the
participants could submit their predictions for the development set and get
the correspondent score for each task. Each participant could make up to 200
submissions. There were 50 submissions.

Evaluation phase: from May 24 to June 6. This phase started with the
publication of the test set. In this phase the participants could submit the
predictions of their final systems and get the correspondent score for each
task. Each participant could make up to 10 submissions. There were 19
submissions.

Post-Evaluation phase: from June 7 onward. This phase started after the end of
the competition. The CodaLab page remains available for everyone who wants to
test additional systems, download the training, development and test sets, and
check the shared-task information.

###  5.2 Systems Descriptions

Although we had only three participants in the evaluation phase, overall the
followed approaches were notoriously diverse. We briefly describe those
systems:

The user aalgarra submitted results only during the development phase,
obtaining good results. The chosen approach is based on finetuning a Spanish
RoBERTa-base trained on BNE finetuned for CAPITEL NER dataset [Fandiño et
al.,2022], to which a bidirectional LSTM layer is added. The target is
obtained by assigning the category to the first token of each word and
applying a mask in the loss calculation for the following tokens of each word.
The training was performed in two phases: an initial phase with frozen RoBERTa
weights, with 30 epochs and a learning rate of $1e-3$, and a second phase
training the full network during 2 epochs with a learning rate of $5e-5$.

The Universidade da Coruña team [Muñoz-Ortiz and Vilares,2023] (user amunozo)
participated on all three tasks. They implemented their solution as a Multi-
task learning problem where pre-trained encoder-decoder models were fine-tuned
based on a hard parameter-sharing approach. They introduced three models with
a common pre-trained LLM encoder fine-tuned on all three tasks and different
decoders for each task. Their best-performing model on Task 1 is single-nllb
model [NLLB Team et al.,2022] where the encoder is trained only on Task 1 and
the decoder is a softmax on the output layer, while the systems for Tasks 2
and 3 decode the output as Conditional Random Fields (CRF). Their submitted
systems use the mtl–beto-gn [Agüero-Torales, López-Herrera, and Vilares,2023]
model for Task 2 and the mtl-nllb model [NLLB Team et al.,2022] for Task 3. A
post-processing heuristic is added to ensure the formatting of the final
results.

The University of Helsinki team [Jauhiainen, Jauhiainen, and Lindén,2023]
(user tsjauhia) participated only on Task 1 obtaining the second place, very
close to the winning team (0.0242 F1 points away). They used a system
consisting of their own HeLI-OTS language identifier [Jauhiainen, Jauhiainen,
and Lindén,2022] based on the HeLI method [Jauhiainen, Lindén, and
Jauhiainen,2016], that includes Guarani. Besides making some internal
configurations to HeLI-OTS, they also added ad hoc rules, such as a text
preprocessing pipline for Guarani and pattern-based logic to detect mixed
Spanish-Guarani words.

User | Task 1 | Task 2 | Task 3
---|---|---|---
| Weighted F1 | Labeled F1 | Labeled F1
pughrob | 0.9424 | 0.7604 | 0.4370
aalgarra | 0.9304 | 0.7047 | 0.2937
tsjauhia | 0.9053 | - | -
amunozo | 0.8221 | 0.1986 | -
baseline | 0.7032 | 0.4218 | 0.2331
Table 2: Results of the development phase.

The Inclusive Technologies for Marginalised Languages (ITML) team [Pugh and
Tyers,2023] (user pughrob) from Indiana University obtained the first place on
every task. The authors evaluate a CRF model trained on text features and
several neural network approaches using pre-trained multilingual
representations, different fine-tuned BERT and T5 models. The team got the
first position in the competition for the three tasks, after finding that
fine-tuning the multilingual representations with unlabeled monolingual
Guarani data is beneficial for all three tasks, and that multi-task training
achieves the best results for Task 2.

###  5.3 Results

Table 2 shows the results of the development phase, while Table 3 shows the
final results of the evaluation phase. We also had an additional submission by
user pakapro for the evaluation phase that just tagged every token as other,
hence it is not included in the tables.

User | Task 1 | Task 2 | Task 3
---|---|---|---
| Weighted F1 | Labeled F1 | Labeled F1
pughrob | 0.9381 | 0.7028 | 0.3836
tsjauhia | 0.9139 | - | -
amunozo | 0.8500 | 0.4153 | 0.1939
baseline | 0.7325 | 0.4946 | 0.2195
Table 3: Results of the evaluation phase.

In general, participants got very good results for Task 1, beating the
baseline by a good margin. The categories where the participants got the best
performance for Task 1 were gn, es, and other, being the last one a very easy
class to predict, containing mostly punctuation signs and URLs. Conversely,
the hardest classes to classify were foreign and mix, which also had the
fewest examples. The mix category is particularly interesting, as it comprises
words that combine Guarani and Spanish content in some way, for example a
Spanish verb with some adapted Guarani verbal morphology, or a Spanish noun
with a Guarani case marker suffix. There are only 47 examples of mix words in
the test corpus, and three of those were correctly identified by all three
participants: oñepresenta, salud-pe, and cartista-pe. The first one was in the
training data, but not the others, so the systems were able to actually
generalize at least to these simple cases. Of the three participants, the
University of Helsinki team was the only one that reported including
particular rules for dealing with mix words by analyzing affixes, and they
also got the best precision for this class, although the ITML achieved better
recall.

The performance for Tasks 2 and 3 was more mixed, with only one team beating
the baselines. Task 3 was particularly hard for the automatic systems, and it
was also hard for the annotators. That can be seen as the best performance was
only 0.384 for labeled F1. Although it is not completely comparable, if we
consider the annotators obtained an average F1 of 0.758 for a subsample of 90
texts, we can say that there is still a lot of room for improvement in this
task.

##  6 Conclusions

In this work, we presented the GUA-SPA shared task for Guarani-Spanish code-
switching analysis at IberLEF 2023, the first task of its kind that involves
an American indigenous language. The competition consisted of three tasks:
language identification, NER, and a novel task of detecting the way Spanish is
used in the code-switched context. Three participants submitted their
predictions for the evaluation phase, with the ITML team submission being the
one with the best results across the three tasks. Although performance for
Task 1 was generally good for all participants, the results obtained for Tasks
2 and 3 were more mixed. In particular Task 3, predicting if a Spanish span is
used as an unadapted loan or as part of a code change, was the most
challenging one both for the human annotators and the participants of the
competition. It would be interesting to explore this task more in the future,
trying to better specify the two classes and how to deal with the ambiguous
cases. This could also be explored for other languages as well, as this
phenomenon could be common in any code-switched context. We would also like to
incorporate more linguistic varieties of Guarani to the task, for example the
Bolivian variety, and also include Portuguese as another possible language for
the tokens, since it is another widely spoken language in South America,
specifically near the areas where Guarani is spoken. We hope this work
contributes to spark the interest in code-switching analysis for more American
indigenous languages.

##  References

  * [Agüero-Torales, Vilares, and López-Herrera,2021] Agüero-Torales, M., D. Vilares, and A. López-Herrera.  2021\.  On the logistical difficulties and findings of jopara sentiment analysis.  In Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching, pages 95–102, Online, June. Association for Computational Linguistics.
  * [Agüero-Torales, López-Herrera, and Vilares,2023] Agüero-Torales, M. M., A. G. López-Herrera, and D. Vilares.  2023\.  Multidimensional affective analysis for low-resource languages: A use case with guarani-spanish code-switching language.  Cognitive Computation.
  * [Aguilar, Kar, and Solorio,2020] Aguilar, G., S. Kar, and T. Solorio.  2020\.  LinCE: A centralized benchmark for linguistic code-switching evaluation.  Proceedings of The 12th Language Resources and Evaluation Conference, pages 1803–1813.
  * [Al-Rfou’, Perozzi, and Skiena,2013] Al-Rfou’, R., B. Perozzi, and S. Skiena.  2013\.  Polyglot: Distributed word representations for multilingual NLP.  In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 183–192, Sofia, Bulgaria, August. Association for Computational Linguistics.
  * [Bird, Klein, and Loper,2009] Bird, S., E. Klein, and E. Loper.  2009\.  Natural language processing with Python: analyzing text with the natural language toolkit.  ” O’Reilly Media, Inc.”.
  * [Cavnar and Trenkle,1994] Cavnar, W. B. and J. M. Trenkle.  1994\.  N-gram-based text categorization.
  * [Chen et al.,2022] Chen, S., G. Aguilar, A. Srinivasan, M. Diab, and T. Solorio.  2022\.  CALCS 2021 shared task: Machine translation for code-switched data.  arXiv preprint arXiv:2202.09625.
  * [Chiruzzo et al.,2020] Chiruzzo, L., P. Amarilla, A. Ríos, and G. Giménez-Lugo.  2020\.  Development of a guarani-spanish parallel corpus.  In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2629–2633.
  * [Chiruzzo et al.,2022] Chiruzzo, L., S. Góngora, A. Alvarez, G. Giménez-Lugo, M. Agüero-Torales, and Y. Rodríguez.  2022\.  Jojajovai: A Parallel Guarani-Spanish Corpus for MT Benchmarking.  In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2098–2107.
  * [Fandiño et al.,2022] Fandiño, A. G., J. A. Estapé, M. Pàmies, J. L. Palao, J. S. Ocampo, C. P. Carrino, C. A. Oller, C. R. Penagos, A. G. Agirre, and M. Villegas.  2022\.  Maria: Spanish language models.  Procesamiento del Lenguaje Natural, 68.
  * [Góngora, Giossa, and Chiruzzo,2021] Góngora, S., N. Giossa, and L. Chiruzzo.  2021\.  Experiments on a guarani corpus of news and social media.  In Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas, pages 153–158.
  * [Góngora, Giossa, and Chiruzzo,2022] Góngora, S., N. Giossa, and L. Chiruzzo.  2022\.  Can we use word embeddings for enhancing Guarani-Spanish machine translation?  In Proceedings of the Fifth Workshop on the Use of Computational Methods in the Study of Endangered Languages, pages 127–132, Dublin, Ireland, May. Association for Computational Linguistics.
  * [Jauhiainen, Jauhiainen, and Lindén,2022] Jauhiainen, T., H. Jauhiainen, and K. Lindén.  2022\.  HeLI-OTS, off-the-shelf language identifier for text.  In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3912–3922, Marseille, France, June. European Language Resources Association.
  * [Jauhiainen, Jauhiainen, and Lindén,2023] Jauhiainen, T., H. Jauhiainen, and K. Lindén.  2023\.  Tuning heli-ots for guarani-spanish code switching analysis.  In Procesamiento del Lenguaje Natural 71. SEPLN.
  * [Jauhiainen, Lindén, and Jauhiainen,2016] Jauhiainen, T., K. Lindén, and H. Jauhiainen.  2016\.  HeLI, a word-based backoff method for language identification.  In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 153–162, Osaka, Japan, December. The COLING 2016 Organizing Committee.
  * [Jiménez-Zafra, Rangel, and Montes-y Gómez,2023] Jiménez-Zafra, S. M., F. Rangel, and M. Montes-y Gómez.  2023\.  Overview of IberLEF 2023: Natural Language Processing Challenges for Spanish and other Iberian Languages.  In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2023), co-located with the 39th Conference of the Spanish Society for Natural Language Processing (SEPLN 2023), CEUR-WS.org.
  * [Joshi,1982] Joshi, A.  1982\.  Processing of sentences with intra-sentential code-switching.  In Coling 1982: Proceedings of the Ninth International Conference on Computational Linguistics.
  * [Joshi et al.,2020] Joshi, P., S. Santy, A. Budhiraja, K. Bali, and M. Choudhury.  2020\.  The state and fate of linguistic diversity and inclusion in the NLP world.  arXiv preprint arXiv:2004.09095.
  * [Joulin et al.,2016] Joulin, A., E. Grave, P. Bojanowski, and T. Mikolov.  2016\.  Bag of tricks for efficient text classification.
  * [Kallfell,2016] Kallfell, G.  2016\.  ¿Cómo hablan los paraguayos con dos lenguas?: gramática del jopara.  Centro de Estudios Antropológicos de la Universidad Católica (CEADUC).
  * [Ley de Lenguas 4251,2010] Ley de Lenguas 4251.  2010\.  Ley Nº 4251 / DE LENGUAS.  In https://www.bacn.gov.py/leyes-paraguayas/2895/ley-n-4251-de-lenguas.
  * [Mager et al.,2021] Mager, M., A. Oncevay, A. Ebrahimi, J. Ortega, A. R. Gonzales, A. Fan, X. Gutierrez-Vasques, L. Chiruzzo, G. Giménez-Lugo, R. Ramos, et al.  2021\.  Findings of the AmericasNLP 2021 shared task on open machine translation for indigenous languages of the Americas.  In Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas, pages 202–217.
  * [Muñoz-Ortiz and Vilares,2023] Muñoz-Ortiz, A. and D. Vilares.  2023\.  Lys a coruña at gua-spa@iberlef2023: Multi-task learning with large language model encoders for guarani-spanish code switching analysis.  In Procesamiento del Lenguaje Natural 71. SEPLN.
  * [NLLB Team et al.,2022] NLLB Team, M. R. Costa-jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood, B. Akula, L. Barrault, G. M. Gonzalez, P. Hansanti, J. Hoffman, S. Jarrett, K. R. Sadagopan, D. Rowe, S. Spruit, C. Tran, P. Andrews, N. F. Ayan, S. Bhosale, S. Edunov, A. Fan, C. Gao, V. Goswami, F. Guzmán, P. Koehn, A. Mourachko, C. Ropers, S. Saleem, H. Schwenk, and J. Wang.  2022\.  No language left behind: Scaling human-centered machine translation.
  * [Pavao et al.,2022] Pavao, A., I. Guyon, A.-C. Letournel, X. Baró, H. Escalante, S. Escalera, T. Thomas, and Z. Xu.  2022\.  Codalab competitions: An open source platform to organize scientific challenges.  Technical report.
  * [Pugh and Tyers,2023] Pugh, R. and F. Tyers.  2023\.  The itml submission to the iberlef2023 shared task on guarani-spanish code switching analysis.  In Procesamiento del Lenguaje Natural 71. SEPLN.
  * [Ríos, Amarilla, and Lugo,2014] Ríos, A. A., P. J. Amarilla, and G. A. G. Lugo.  2014\.  Sentiment categorization on a creole language with lexicon-based and machine learning techniques.  In 2014 Brazilian Conference on Intelligent Systems, pages 37–43. IEEE.
  * [Rodríguez Gutiérrez and Núñez Méndez,2018] Rodríguez Gutiérrez, Y. and E. Núñez Méndez.  2018\.  Language contact and the indigenous languages of Uruguay.  Biculturalism and Spanish in contact: sociolinguistic case studies, pages 217–238.
  * [Solorio et al.,2014] Solorio, T., E. Blair, S. Maharjan, S. Bethard, M. Diab, M. Ghoneim, A. Hawwari, F. AlGhamdi, J. Hirschberg, A. Chang, et al.  2014\.  Overview for the first shared task on language identification in code-switched data.  In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 62–72.