1cm

# RC2020 Report: Learning De-biased Representations with Biased
Representations

Rwiddhi Chakraborty∗
Università della Svizzera italiana
<EMAIL_ADDRESS>
&Shubhayu Das∗
Università della Svizzera italiana
<EMAIL_ADDRESS>
equal contribution

## Reproducibility Summary

### Scope of Reproducibility

The authors formalize and attempt to tackle the so called "cross bias
generalization" problem with a new approach they introduce called ReBias
(Bahng et al., 2020). This report contains results of our attempts at
reproducing the work in the application area of Image Recognition,
specifically on the datasets biased MNIST and ImageNet. We compare ReBias with
other methods - Vanilla, Biased, RUBi (Cadène et al., 2019) (as implemented by
the authors), and conclude with a discussion the validity of the claims made
by the paper. Our reproducibility source code is available in the
supplementary material we’ve uploaded with the revision.

### Methodology

We used the authors’ code available (source) and did not modify any part of it
to make sure the initial conditions of the experiments were identical. We did
not re-implement any part of the pipeline except for ImageNet, implementation
issues of which we discuss later on. The total training time, with final
results averaged over 3 runs, amounted to 128 hours on on a single NVIDIA
GeForce GTX 1080 8GB GDDR5X with 2560 CUDA cores.

### Results

We were able to reproduce results reported for the biased MNIST dataset to
within 1% of the original values reported in the paper. Like the authors, we
report results averaged over 3 runs. However, in a later section, we provide
some additional results that appear to weaken the central claim of the paper.
We were not able to reproduce results for ImageNet as in the original paper,
but present our results and a further discussion.

### What was easy

We found the reproducibility for the biased MNIST dataset to be easy. The code
provided was clear to understand and ready to run as a simple terminal
command.

### What was difficult

Although the majority code used by the authors for the training pipleline was
made publicly available, there were a few things we had to be careful about,
including using the right version of the library torchvision (use latest
version, version used by the authors is different), using the right WNIDs for
scraping the 9-class ImageNet dataset (actual dataset used in training is not
provided). We found it difficult to reproduce two of the methods ReBias is
compared against (HEX (Wang et al., 2019) and Learned-Mixin (Clark et al.,
2019)), as they were the authors’ implementation of the methods (authors’
implementation of HEX is not available). The most significant difficulty faced
was attempting to avoid exploding gradients on one of the experimental
settings, even though we used the exact same training pipeline.

### Communication with original authors

We had some preliminary contact with the authors regarding the implementation
of the methods discussed above. We were informed that the HEX implementation
was exactly as in the original work, and that the implementations for RUBi and
Learned-Mixin’ had been adapted from the NLP domain into the vision domain.
Further, the authors confirmed some details about the dataset and training
which were previously not clear.

##  1 Introduction

The paper investigates the problem of "cross bias generalization", which is to
reduce the impact of bias cues CNN models use to predict outcomes of image
inputs. These bias shortcuts ultimately hamper the model’s ability to
generalize well to unseen inputs, as, for example, images of a frog in a swamp
may be quite frequent in a training dataset, but an image of a frog in a
living room may be rare. Testing on this example, however, a model strongly
correlated with the bias of an object (in this case, a swamp) will perform
poorly on this test image, where the bias has shifted. A different background
(bias) does not change the nature of the object in question. Given a signal
cue $S$ for an image input $X$, the true output $Y$ [$Y^{tr}$ and $Y^{te}$
being the training and test distributions], and the associated bias of $Y$ as
$B$ [$B^{tr}$ and $B^{te}$ being the biases in the training and test
distributions], the cross bias problem can be summarized like so:

  * •

$p(B^{tr})\not\\!\perp\\!\\!\\!\perp p(Y^{tr})$

  * •

$p(B^{tr},Y^{tr})\neq p(B^{te},Y^{te})$

  * •

$p(B^{tr})=p(B^{te})$

As an example, the training data may contain two types of images,
$(Y^{tr}=frog,B^{tr}=swamp)$, and $(Y^{tr}=bird,B^{tr}=sky)$, but the test set
contains unusual combinations of the classes and biases:
$(Y^{tr}=frog,B^{tr}=sky)$, $(Y^{tr}=bird,B^{tr}=swamp)$. In essence, we have
a crossover of the biases. A biased model will have trouble discriminating the
classes, as we will see in the results later.

There are primarily three ways one can improve cross bias generalization.
Firstly, if one could disentangle bias $B$ from signal $S$ completely, then
one could eliminate the bias dimension in their dataset and train bias free on
this normalised data. However, as [1] show, for texture biases in CNNs, this
is an unrealistic task. Secondly, if one could have a data generation
procedure $p(X|B)$, one could de-bias a training dataset. But this means, in
our frog example, we may need to collect images of frogs with varied
backgrounds [2]. This, again, is unrealistic, not to mention expensive. The
third approach is the reverse of the second. If one could design a predictive
algorithm for $p(B|X)$, which means one can describe the bias for every
possible input $X$, it is possible to de-bias the training data [3, 4]. This
is also an issue as for biases like textures, it is impossible to enumerate
all possible textures and associate each with a label.

However, the third approach can be useful with certain assumptions. These
assumptions are what lead to ReBias. In section 2 we briefly summarise the
intuition behind ReBias. In section 3 we point out the claims made in the the
paper that we wish to verify. In section 4 we describe our replication
methodology and in section 5 we present and discuss our results. Subsequently
in section 6 we analyze the claims made by the authors and finally, section 7
summarizes this report.

##  2 ReBias

Although it is computationally intractable to solve for $p(B|X)$
deterministically, one could approximate it using different bias models $G$.
For image recognition, and in the context of CNNs, (Geirhos et al., 2018)
shows that receptive fields are important factors in capturing texture biases.
Their empirical evidence suggests that smaller filter sizes lead to increased
texture bias in CNN architectures. In this case, $G$ would simply be a set of
models with smaller receptive fields. And learned model $g\in G$ would then
make predictions on images based on texture biased cues. They are more likely
to overfit to texture biases.

For a given bias-signal pair $(B,S)$, G is called a bias characterizing model
class, if for every possible joint distribution $p(B,X)$ (where $X$ is the
input signal), the following two conditions satisfy:

  * •

$\exists g\in G,p(B|X)\approx g(X)$

  * •

$\forall g\in G,g(X)\\!\perp\\!\\!\\!\perp S|B$

Depending on the application task, the set $G$ would vary accordingly. For
image recognition, as discussed, $G$ refers to models with smaller filter
sizes (leading to increased texture bias). For action recognition tasks, $G$
can be chosen to be a set of 2D CNN models, etc.

This finally brings us to the proposed method: ReBias.

Here the idea is to design a model $p(Y|X)$ that is encouraged to be maximally
independent of ${g(X):g\in G}$, the set of all possible biased models with
smaller receptive fields. In short, one intentionally overfits a collection of
biased models on the data, and then encourages $f(X)$ to be as independent as
possible from $G$. To quantify this metric, the authors use the Hilbert-
Schmidt Independence Criterion (HSIC) (Gretton et al., 2005). The intuition
for using this metric is that for RBF kernels $k,l$, and for two random
variables $U,V$, $HSIC^{k,l}(U,V)=0$ if and only if $U\\!\perp\\!\\!\\!\perp
V$. This is exactly what is needed with respect to the random variables in the
authors’ case: $f(X)$ and $G$. Thus:

$HSIC(f,G):=\max_{g\in G}HSIC(f,g)$

This allows the authors to formulate the ReBias loss function:

$\min_{f}[L(f,X,y)+\lambda\cdot\max_{g\in G}HSIC(f,g)]$

It can be observed that this is a min-max optimisation procedure, with the
HSIC criterion added on top of a standard loss function commonly used in deep
learning (categorical crossentropy, in this case). This is achieved by two
successive update steps, first the max operation, and then the overall
standard minimisation of the loss over $f$.

This sums up ReBias. Create one standard convolutional model $f$, and a set of
bias models $G$ with smaller kernel (filter) sizes than $f$. Optimise
following the procedure above. The intuition is that, as one set of models
continuously overfit to the data, the main model is encouraged to stay
independent on this set, and thus independent from texture biases learnt by
$G$ during training.

##  3 Scope of reproducibility

From the previous sections, we thus proceed to verify the following claims in
the paper:

  * •

Claim 1: The min-max procedure for ReBias converges.

  * •

Claim 2: The deliberately texture biased model g helps $f$ reduce cross bias.

  * •

Claim 3: The Hilbert-Schmidt Independence Criterion (HSIC) utilized in the
min-max objective function of ReBias is useful in reducing cross bias.

  * •

Claim 4: ReBias solves the cross-bias generalization problem, by robustly
improving a biased model’s cross-bias test accuracy.

##  4 Methodology

The authors of ReBias (Bahng et al., 2020) provide a code (source) for their
implementation containing the entire training and validation pipeline, for
which they provide a fairly detailed documentation. We did not make major
modifications to it as there are quite a few areas where slight tweaks may
lead to incorrect reproduction. However there were certain parts of the
pipeline that we needed to implement on our own, and modify some specific
parts of the code because of certain issues we faced when using the code
provided by the authors. We elaborate this further in section 5. The GPUs we
used for our runs were 1xNVIDIA GeForce GTX 1080 8GB GDDR5X with 2560 CUDA
cores. We provide the shells scripts used and the modified bits of code as
part of the supplementary material.

###  4.1 Experimental setup and descriptions

Algorithms 1,2,3 and 4 shown below, demonstrate the training logic (as deduced
from the code provided by the authors) for the ReBias, Vanilla, Biased and
RUBi models.

Here, $f$ is the original model which needs to be de-biased, $g$ is the
texture biased model with low receptive field, $\mathcal{L}(f(x),y)$ is the
loss function (like cross-entropy loss), $HSIC(f(x),g(x))$ is the Hilbert-
Schmidt-Independence-Criterion, $\lambda$ and $\lambda_{g}$ are hyper-
parameters.

In our experiments we repeated the experiments three times, reporting the
average of 3 runs (like the authors) in the converging cases.

Algorithm 1 Rebias

repeat

for each mini-batch samples $x,y$ do

update $f$ by solving $\underset{f\in
F}{\operatorname*{arg\,min}}\,\mathcal{L}(f,x,y)+\lambda\,\text{HSIC}_{1}(f(x),g(x))$.

update $g$ by solving $\underset{g\in
G}{\operatorname*{arg\,min}}\,\mathcal{L}(g,x,y)-\lambda_{g}\,\text{HSIC}_{1}(f(x),g(x))$.

end for

until converge.

Algorithm 2 Vanilla

repeat

for each mini-batch samples $x,y$ do

update $f$ by solving $\underset{f\in
F}{\operatorname*{arg\,min}},\mathcal{L}(f,x,y)$.

end for

until converge.

Algorithm 3 Biased

repeat

for each mini-batch samples $x,y$ do

update $g$ by solving $\underset{g\in
G}{\operatorname*{arg\,min}},\mathcal{L}(g,x,y)$.

end for

until converge.

Algorithm 4 RUBi

repeat

for each mini-batch samples $x,y$ do

update $f$ by solving $\underset{f\in
F}{\operatorname*{arg\,min}}\,\mathcal{L}(f,x,y)+\lambda\,\text{RUBi}(f(x),g(x))$.

update $g$ by solving $\underset{g\in
G}{\operatorname*{arg\,min}}\,\mathcal{L}(g,x,y)$

end for

until converge. Note: $f$ and $g$ are trained simultaneously but separately.
The function $RUBi$, implemented by the authors of ReBias uses features from
the biased network ($g$), the prediction of the target network $f(x)$ and the
ground truth labels ($y$).

###  4.2 Datasets

For the Image Recognition task, two experiments were conducted: One on Biased
MNIST, a synthetic dataset created by the authors, and 9-class ImageNet.

#### Biased MNIST

The authors modify the MNIST (Lecun et al., 1998) dataset by injecting
coloured backgrounds to the digits. They select 10 distinct colours for each
digit. To create the training set, with probability $\rho$, the pre-selected
colour is added to the specific digit, and any other colour with probability
$1-\rho$. Thus, for $\rho=1$, we have complete bias, and for $\rho=0$, we have
a completely unbiased dataset. The effect of $\rho$ on the distribution of
$P(B|S)$ in shown in figure 1.

In this experimental setup, the authors validate the models on a "Biased"
validation set (with the same $\rho$ values as the training set) and an
"Unbiased" validation set (where $\rho=0.1$). Note that the "Unbiased"
validation accuracy here relates to how robust the model is to cross-biased
testing. The idea is that in the Biased dataset, since colours will be
strongly correlated to the digits, models suffering from texture bias (the set
$G$) will perform strongly (as it picks up the colour cues).

Figure 1: MNIST with colour assignments to digits with a high and low $\rho$

#### 9-class ImageNet

The paper uses ’9 class ImageNet’, a subset of the ImageNet (Deng et al.,
2009) dataset for training, and reports validation accuracies over the ’9
class ImageNet’ and ’ImageNet-A’ (Hendrycks et al., 2019) and ’ImageNet-C’
(Hendrycks and Dietterich, 2019). ImageNet-A consists of adversarial examples
over the ImageNet dataset where top performing ImageNet models fail to
correctly classify because they make inferences from some background cues.
ImageNet-C consists of manually corrupted images; with 15 corruption types
including “noise”, “blur”, “weather”, and “digital” with five degrees of
severity. An improved performance on ImageNet-A and ImageNet-C would indicate
that the model learns beyond the bias shortcuts.

Apart from computing the accuracy over the validation set (’Biased Accuracy’),
ImageNet-A and ImageNet-C, they also present an ’unbiased accuracy’ - For
doing this they propose an unsupervised texture feature clustering technique
which inputs each image of the ImageNet validation set into one of 9 cluster-
ids (using gram matrices of low-layer feature maps as the texture features of
the image, followed by k-means clustering). Then, assuming all images within a
cluster have the same texture features (or no variation in texture), they
separately compute the proportion of correctly classified images within each
of the 9 texture clusters, and then report the average. Note that the paper
does not conclusively show that there are no bias variations within the
clusters themselves (i.e. why cross bias may still not exist in each of these
clusters). Also, the choice of using specifically 9 clusters is not
elaborated.

#### Availability of the datasets

The Biased-MNIST dataset was constructed by the authors and is provided in
their code base

The ImageNet-A and ImageNet-C datasets are openly available, and the code for
computing the ’Unbiased Accuracy’ using texture feature clustering is publicly
available. However, the exact subset of ImageNet used to build the 9-class
ImageNet dataset used by the authors in this paper was not provided. After
confirming with the authors we constructed this dataset ourselves, by using
the same WNIDs of ImageNet (ILSVRC-2017) sub-classes.

###  4.3 Model architecture and hyperparameters

For the main results shown here we do not change the model architecture or the
hyperparameters (except in 9-class ImageNet - explained later)

#### Biased MNIST

The model $f$ is a fully convolutional neural network with 4 layers and kernel
sizes of 7 x 7, while the model $g$ has the same architecture except its
kernel size is 1 x 1. In both cases, the final convolutional layer is followed
by global average pooling, and a linear classifier for predicting the labels
(fully connected).

The authors use batch normalization, ReLU as their activation function, 256 as
their batch size, a learning rate of 0.001 with a decay factor of 0.1 every 20
epochs, and train for a total of 80 epochs. The authors use ADAM and ADAMP as
their optimisers. We present the results with ADAM.

#### 9-class ImageNet

In the ImageNet experiment, the model $f$ uses a ResNet18 (He et al., 2015)
architecture and the mode $g$ uses a BagNet18 (Brendel and Bethge, 2019)
architecture.

The rest of the hyperparameters are the same except for the batch size. The
authors use a batch size of 128, however given that we use GPUs much lower
memory, we were forced to reduce the batch size. In our experiments we use a
batch size of 16. In the next section (5) we show that the ReBias training
method results in exploding gradients. While attempting to solve this we made
some changes, which we justify and discuss in section 5.2.

##  5 Results

In the following subsections we talk about our results on the Biased MNIST and
9-class ImageNet experimental setup.

###  5.1 Biased MNIST

####  5.1.1 Reproduced Results [Average over 3 runs]

We first present our reproduced results, averaged over 3 runs.

| $\rho$
---
Biased Validation Set
| Vanilla | Biased | RUBi | ReBias
0.999 | 100 | 100 | 100 | 100
0.997 | 100 | 100 | 100 | 100
0.995 | 100 | 100 | 100 | 100
0.990 | 100 | 100 | 100 | 100
Table 1: Validation Accuracy on Biased Data

| $\rho$
---
Unbiased Validation Set
| Vanilla | Biased | RUBi | ReBias
0.999 | 10.17 | 10 | 10.36 | 21.11
0.997 | 48.55 | 10 | 35.49 | 61.99
0.995 | 73.04 | 10 | 65.81 | 75.79
0.990 | 88.05 | 10 | 84.41 | 87.61
Table 2: Validation Accuracy on Unbiased Data

These results are within 1% of the original results presented in the paper. As
was expected all models perform well on the biased validation sets (because
the training data was biased the same way). Things get interesting when we
discuss the Unbiased dataset results. A glance at Table 2 told us that the
trend of Vanilla vs ReBias accuracies was progressively shifting in favour of
the Vanilla model as we slightly decrease $\rho$. This hypothesis was
confirmed to be true as we tested for lower values of $\rho$, but still making
sure the dataset was sufficiently biased. We test for four additional values
of $\rho:0.98,0.95,0.9,0.85$. These are the additional results we discuss
next, not present in the original paper.

####  5.1.2 Additional results not present in the original paper

| $\rho$
---
Biased Validation Set
| Vanilla | Biased | RUBi | ReBias
0.999 | 100 | 100 | 100 | 100
0.997 | 100 | 100 | 100 | 100
0.995 | 100 | 100 | 100 | 100
0.990 | 100 | 100 | 100 | 100
0.980 | 100 | 99.99 | 100 | 99.99
0.950 | 99.99 | 99.98 | 99.98 | 99.98
0.900 | 99.94 | 99.89 | 99.97 | 99.94
0.850 | 99.94 | 99.8 | 99.89 | 99.93
Table 3: Validation Accuracy on Biased Data

| $\rho$
---
Unbiased Validation Set
| Vanilla | Biased | RUBi | ReBias
0.999 | 10.17 | 10 | 10.36 | 21.11
0.997 | 48.55 | 10 | 35.49 | 61.99
0.995 | 73.04 | 10 | 65.81 | 75.79
0.990 | 88.05 | 10 | 84.41 | 87.61
0.980 | 93.60 | 10 | 92.66 | 93.48
0.950 | 96.87 | 12.71 | 96.64 | 96.54
0.900 | 98.09 | 14.34 | 97.89 | 97.90
0.850 | 98.41 | 15.51 | 98.46 | 98.24
Table 4: Validation Accuracy on Unbiased Data

As is clear from Table 4, Vanilla actually beats ReBias for $\rho=[0.90,.99]$
onwards, while RUBi performs the best for $\rho=0.85$ and is marginally better
than Vanilla. If we look at the plot below, we see a clearer picture of what
the table represents:

Figure 2: Comparison of Rebias and Vanilla over
$\rho\in[0.999,0.997,0.995,0.99,0.98,0.95,0.9,0.85]$

From Figure 2 we can see that the bandwidth of $\rho$ values for which ReBias
performs better is demarcated by the red line. Over the rest of the $\rho$
inputs, while the training set is still sufficiently biased, Vanilla performs
at least as good or better than ReBias. Intuitively trying to understand this
raises two practical questions: (1) Is this experiment’s set up too simple for
a simple vanilla model to be less prone to cross biases? (2) Or is it that
ReBias is effective only when the training set is extremely biased and using a
vanilla training technique leads to much lower cross-biased accuracy?

We answer the first question in the affirmative. The synthetic dataset used in
this paper is simplistic as it injects a readily identifiable bias - the
color. This allows the definition of the parameter $\rho$, which is
accordingly tuned to infer about results. We note here that the conception of
$\rho$ would be unclear if one were to use a complicated dataset that is
generally texture biased, as opposed to incorporating a single color bias.
Examples include Stylized Imagenet, or any other dataset modified by style
transfer. However, the authors indeed run their experiments on challenging
datasets like Imagenet-A, and as we discuss in a later section, our
difficulties (and eventual failure) at reproducing the Imagenet results make
it hard to definitevely answer the second question we pose. Unfortunately, we
can neither prove nor disprove this hypothesis.

###  5.2 ImageNet

We were successfully able to train the Vanilla and the Biased model. However
the Rebias model training resulted in exploding gradients, even though we used
the same pre-processing and training pipeline. The only hyperparameter that
was different from the ones used by the authors was the batch-size (we used a
batch size of 16 due to GPU memory constraints). The results we get after
using the same hyperparameter setting as the authors (except batch size) is
shown in Table 5. As can be seen the results we get after training for 120
epochs are not comparable to the original paper (table 6). What was
particularly interesting to us was the fact for the ReBias training type the
gradients consistently explode after the very first epoch. Moreover for the
biased training type the unbiased validation accuracy reduces to $0\%$ by the
end of 120 epochs (worse than a random prediction), which shows that the
training procedure for ImageNet completely fails. The fact that this does not
happen for our Vanilla training scheme (the only case where there are similar
Biased and Unbiased accuracies as the authors, although Imagenet-A accuracies
are not similar)on Imagenet, given that we used the same dataset (confirmed by
the authors) and pre-processing, provoked us to look into the differences in
the training pipeline, i.e. the hyperparameters. These are discussed below.

| Training Type
---
Validation Types
| Biased | Unbiased | ImageNet-A |
Vanilla | 88.98 | 88.93 | 5.89 |
Biased | 79.91 | 0.0 | 7.08 |
ReBias | 0.0 | 0.0 | 10.39 |
Table 5: Our results on 9-class-ImageNet

| Training Type
---
Validation Types
| Biased | Unbiased | ImageNet-A |
Vanilla | 90.8 | 88.8 | 24.9 |
Biased | 67.7 | 65.9 | 18.8 |
ReBias | 91.9 | 90.5 | 29.6 |
Table 6: The authors results on 9-class-ImageNet

####  5.2.1 Different hyperparameter settings

  1. 1.

Batch Size: Given that for the Rebias training on ImageNet we got exploding
gradients, we tried to artificially increase the batch size from 16 to 128, by
accumulating gradients for 8 batches and skipping 8 batches to changing the
weights by taking an optimization step - making all the hyperparameters
exactly the same as the authors. This however didn’t solve the exploding
gradients problem or the huge discrepancy in the Unbiased validation accuracy
for the Biased training type.

  2. 2.

Optimizer and Learning Rate: Initially, as suggested by the authors, we used
the AdamP optimizer with a learining rate of $0.001$ and weight decay
$0.0001$. We later also tried AdamP and Adam optimizer with different learning
rates and weight decay. This however did not solve our issue with exploding
gradients. However, we note that during these runs the HSIC score fluctuates
significantly when tuning the optimizer parameters, leading us to the next set
to parameters we tuned in tandem with the ones till now.

  3. 3.

HSIC parameters: The authors implementation of the Hilbert-Schmidt
Independence Criterion(HSIC) uses a Radial Basis Function (Rbf) kernel, which
has a kernel radius parameter (’sigma’) that scales each of the inputs to the
HSIC function. The sigmas used can either be fixed or can be changed every
epoch. For the Biased MNIST experiment the authors use a fixed sigma, of $1$,
where as for the ImageNet experiment they used a sigma that changes based on
the median of $f(x)$ and $g(x)$, with $25\%$ of the training data chosen
randomly. We tried varying these parameters, one of which included the exact
same ones used in the Biased-MNIST experiment (since we did not face exploding
gradients during this experiment). Unfortunately this also did not fix the
issue.

We communicated these issues faced by us in this experiment in detail with the
authors, and asked for suggestions or if they had also faced such issues at
some point. The authors confirmed that computing the HSIC does indeed require
a large batch size, and simply using gradient accumulation does not solve the
issue. In this respect, we conclude that the importance of the hardware being
used is crucial for reproducing experiments like these. This turned out to be
the pivotal factor.

##  6 Analysis

From the evidence we get from running our experiments we have a few
observations and insights about the claims of the paper.

  * •

Claim 1: The ReBias method converges. In the the first experimental setting of
Biased-MNIST, we see that ReBias successfully converges; In the case of
ImageNet however our results show that the model fails to converge. This
failure of convergence is due to exploding gradients, moreover we observe the
this can not be solved by tweaking hyperparameters.

  * •

Claim 2: The deliberately texture biased model $g$ helps $f$ reduce cross
bias. In case of Biased MNIST we see that the texture biased model $g$ indeed
helps improve the cross-bias accuracy of the model $f$. However, it should be
noted that this may fail to remove cross-biases from $f$ that are not textual
in nature (for instance, shape biases or biases due to the existence of some
object in the background). In such cases simply having low-receptive fields
does not necessarily ensure that the model $g$ would be prone to texture
biases more than the model $f$ would. Hence this method is more effective in
improving a model’s ($f$) cross-bias accuracy only when the cross-biases are
caused by textures, similar to the Biased-MNIST setting.

  * •

Claim 3 : The Hilbert-Schmidt Independence Criterion (HSIC) utilized in the
min-max objective function of ReBias is useful in reducing cross bias. This
claim seems to be satisfied by our Biased MNIST results. However the
additional results we report even for sufficiently high $\rho$ $\in$
$[0.90,0.99]$ show that the vanilla model itself outperforms the ReBias model.
This may be due to the simplicity of the MNIST model setting.

  * •

Claim 4 : ReBias solves the cross-bias generalization problem. From our
experiments we observe that ReBias quite certainly is not able to fully solve
the cross-bias generalization problem. This actually stems from the model’s
inadequacy to fully solve the requirements set by the previous claims, most
significant of which is the failure to achieve convergence.

##  7 Conclusion and discussion

We conclude this reproducibility report by summarizing our results, clarifying
the paper’s implementation of methods ReBias is compared against, and
commenting on the effectiveness of the experiments conducted in terms of the
claims made by the paper.

#### Our Results

In our report we elucidate the technical details of ReBias, the theory behind
it, and our attempts at replicating the results of this paper. We were
successful in doing so for one synthetic dataset, Biased MNIST, and
unsuccessful for another: 9 Class ImageNet. In addition, we provide analysis
of the interpretability of the MNIST results, and the reproducibility
challenges of ImageNet from the authors’ implementation.

Although the authors clearly justify their intuition and theoretical
motivation, we believe the empirical experiments conducted do not fully
demonstrate the benefits of the proposed technique over existing methods. Our
inability to achieve convergence in the 9-class ImageNet experiment, despite
using the same training pipeline and hyper-parameters, is one such indicator
of the inherent stochasticity involved in deep learning experiments leading to
a difficulty in reproducing results.

#### Clarifying implementation of methods ReBias is compared against

Further, we are not able to compare ReBias’s performance against HEX (Wang et
al., 2019), this is primarily because the implementation of HEX on the problem
settings discussed in this paper are not made public. Learned-mixin’ (Clark et
al., 2019), another ReBias is compared against in the original paper, uses an
ensemble based approach to remove biases. However, its implementation, in the
code linked to this paper, does not use such an approach. The implementation
of RUBi (Cadène et al., 2019) used in this paper (shown in Algorithm 4, is
modified from a Visual-Question-Answering (VQA) setting to a Classification
setting. However, as opposed to Algorithm 4, the original implementation
doesn’t utilize the Hilbert-Schmidt Independence Criterion, and has a
different optimization strategy from that used by the authors in this work.
The original optimization strategy of RUBi attempts to remove unimodal biases
– superficial correlations between inputs from one modality (say, the image
input) and the answers without considering the other modality (the text input)
from VQA models; RUBi reduces the loss for examples that can be correctly
answered without looking at the image while attempting to increase the loss
for examples that cannot be answered without using both modalities. Naturally,
the authors of ReBias had to make certain design choices for translating this
idea to an image classification setting where there is just one modality of
the input (an Image) to models (a target network $f$ and a texture biased
model $g$).

#### Final comments

Attempting to reproduce the results presented in the ReBias paper gives us the
opinion that, while ReBias proposes a novel training methodology, it’s
effectiveness for specific de-biasing use cases needs a more constructive
empirical survey.

We would like to acknowledge the importance of the Reproducibility Challenge,
that incentivizes members of the Machine Learning community to address and
help repair the lack of standardization in this field.

## References

  * Bahng et al. [2020] Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh.  Learning de-biased representations with biased representations.  In Hal Daumé III and Aarti Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 528–539. PMLR, 13–18 Jul 2020.  URL http://proceedings.mlr.press/v119/bahng20a.html.
  * Cadène et al. [2019] Rémi Cadène, Corentin Dancette, Hedi Benyounes, Matthieu Cord, and Devi Parikh.  Rubi: Reducing unimodal biases in visual question answering.  _CoRR_ , abs/1906.10169, 2019.  URL http://arxiv.org/abs/1906.10169.
  * Wang et al. [2019] Haohan Wang, Zexue He, Zachary C. Lipton, and Eric P. Xing.  Learning robust representations by projecting superficial statistics out.  _CoRR_ , abs/1903.06256, 2019.  URL http://arxiv.org/abs/1903.06256.
  * Clark et al. [2019] Christopher Clark, Mark Yatskar, and Luke Zettlemoyer.  Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases.  _CoRR_ , abs/1909.03683, 2019.  URL http://arxiv.org/abs/1909.03683.
  * Geirhos et al. [2018] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  _CoRR_ , abs/1811.12231, 2018.  URL http://arxiv.org/abs/1811.12231.
  * Gretton et al. [2005] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf.  Measuring statistical dependence with hilbert-schmidt norms.  In Sanjay Jain, Hans Ulrich Simon, and Etsuji Tomita, editors, _Algorithmic Learning Theory_ , pages 63–77, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg.  ISBN 978-3-540-31696-1.
  * Lecun et al. [1998] Yann Lecun, Leon Bottou, Y. Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86:2278–2324, 12 1998.  doi: 10.1109/5.726791.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In _CVPR09_ , 2009.
  * Hendrycks et al. [2019] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  _arXiv preprint arXiv:1907.07174_ , 2019.
  * Hendrycks and Dietterich [2019] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  _Proceedings of the International Conference on Learning Representations_ , 2019.
  * He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  _CoRR_ , abs/1512.03385, 2015.  URL http://arxiv.org/abs/1512.03385.
  * Brendel and Bethge [2019] Wieland Brendel and Matthias Bethge.  Approximating cnns with bag-of-local-features models works surprisingly well on imagenet.  _CoRR_ , abs/1904.00760, 2019.  URL http://arxiv.org/abs/1904.00760.


# A Prompt Response
to the Demand for Automatic Gender-Neutral Translation

Beatrice Savoldi,1 Andrea Piergentili,1,2 Dennis Fucci,1,2 Matteo Negri1,
Luisa Bentivogli1
1 Fondazione Bruno Kessler
2 University of Trento
<EMAIL_ADDRESS>

###### Abstract

Gender-neutral translation (GNT) that avoids biased and undue binary
assumptions is a pivotal challenge for the creation of more inclusive
translation technologies. Advancements for this task in Machine Translation
(MT), however, are hindered by the lack of dedicated parallel data, which are
necessary to adapt MT systems to satisfy neutral constraints. For such a
scenario, large language models offer hitherto unforeseen possibilities, as
they come with the distinct advantage of being versatile in various (sub)tasks
when provided with explicit instructions. In this paper, we explore this
potential to automate GNT by comparing MT with the popular GPT-4 model.
Through extensive manual analyses, our study empirically reveals the inherent
limitations of current MT systems in generating GNTs and provides valuable
insights into the potential and challenges associated with prompting for
neutrality.

A Prompt Response
to the Demand for Automatic Gender-Neutral Translation

Beatrice Savoldi,1 Andrea Piergentili,1,2 Dennis Fucci,1,2 Matteo Negri1,
Luisa Bentivogli1 1 Fondazione Bruno Kessler 2 University of Trento
<EMAIL_ADDRESS>

##  1 Introduction

To foster greater inclusivity in our communication practices, there has been a
rise in the adoption of gender-neutral language strategies (Hord, 2016;
Papadimoulis, 2018), which challenge gender norms and embrace all identities
by eschewing unnecessary gendered terms (e.g. police officer vs policeman).
Such strategies are now widespread across various domains – including
institutions (Höglund and Flinkfeldt, 2023), academia (APA, 2020), and
industry (Langston, 2020), with their consequential investigation for various
natural language processing (NLP) technologies (Cao and Daumé III, 2020;
Brandl et al., 2022; Wagner and Zarrieß, 2022).

While recent advancements in NLP have seen the modeling of neutral language
into monolingual applications (Vanmassenhove et al., 2021; Sun et al., 2021;
Amrhein et al., 2023; Veloso et al., 2023), research in cross-lingual settings
is relatively limited. Previous works in MT (Costa-jussà and de Jorge, 2020;
Savoldi et al., 2021; Choubey et al., 2021; Alhafni et al., 2022; Piazzolla et
al., 2023, inter alia) have been mostly confined within binary perspectives to
improve the generation of masculine/feminine forms into grammatical gender
languages (e.g. doctors $\rightarrow$ it: dottori/esse).111 Although in
grammatical gender languages also inanimate nous are formally assigned to a
gender class (Corbett, 1991), we are hereby only concerned with (social)
gender assignment for human referents. Under realistic scenarios though,
systems often encounter ambiguous input sentences that do not convey gender
distinctions (Saunders, 2023; Piergentili et al., 2023a), and for which GNT
would be preferable to prevent undue gender assignments in the target language
(e.g. en: doctors $\rightarrow$ it: personale medico[the medical staff]).

Despite individual studies indicating that existing MT systems are ill-
equipped to handle neutrality (Cho et al., 2019; Piergentili et al., 2023b;
Savoldi et al., 2023), the automation of GNT remains an open challenge,
hampered by the lack of dedicated resources. To the best of our knowledge, the
work by Saunders et al. (2020) stands as the sole effort to create gender-
neutral MT models, but their fine-tuning approach does not generalize from
their small artificial adaptation set. Within this landscape, large language
models (LLMs) can offer a solution to meet the demand for gender neutrality,
thanks to their adaptability to perform new (sub)tasks based on explicit
instructions and few examples (Brown et al., 2020). In fact, albeit LLMs still
lag slightly behind traditional MT in overall translation quality (Robinson et
al., 2023; Vilar et al., 2023; Zhang et al., 2023), their versatility for
controlling specific aspects in the output translation was proven for several
attributes (Moslem et al., 2023; Sarti et al., 2023; Garcia and Firat, 2022;
Yamada, 2023).

In this paper, we thus seek to advance the automation of neutral translation
by exploring the potential of instruction-following models. To this aim, we
focus on English$\rightarrow$Italian and systematically compare the neutral
capabilities of traditional MT models with GPT-4 (OpenAI, 2023). By
experimenting with different prompts and shot-exemplars, we conduct a fine-
grained, manual evaluation showing that: i) used as is neither MT nor GPT are
suitable for GNT, but prompting GPT shows surprising neutralization
capabilities elicited with just a few examples; ii) while including test set
terms as neutralization exemplars in the prompts leads to slightly better GNT
performance, GPT can generalize well also when provided with unseen examples.
Finally, extensive manual evaluations unveil that iii) judging the quality and
acceptability of automatic GNT is a subjective task, with notable variations
across annotators. To promote future research, we make all our manual output
annotations freely available at: https://mt.fbk.eu/gente/.222Released under a
Creative Commons Attribution 4.0 International license (CC BY 4.0).

##  2 Methods and Settings

#### Test set.

We run our experiments on GeNTE (Piergentili et al., 2023b), a recently
released parallel test set designed to evaluate models’ GNT capabilitites.
Built on Europarl data (Koehn, 2005), it allows us to test MT on naturalistic
instances for en-it, a language pair that is highly representative of the
challenges of performing GNT into languages with extensive gendered
morphology. For such languages, neutral strategies can range from simple word
changes (e.g. omissions or synonyms) to complex reformulations that can alter
the sentence structure (Gabriel et al., 2018). Hence, generating suitable GNTs
is a delicate and difficult task, to be carefully weighted not to impact the
acceptability of a translation. Here, we use a portion of GeNTE consisting of
750 English sentences that are gender-ambiguous,333Set-N in the original
corpus. and which are thus to be neutrally translated so as to avoid any undue
gender inference in Italian (e.g. I, with all my colleagues wish to…, it-M:
Io, con tutti i colleghi desidero… $\rightarrow$ it-GNT: Io, con ogni
collega[each colleague], desidero…).444For more details, see Appendix A.

#### Systems.

As MT models, we select two state-of-the-art commercial systems: Amazon
Translate555https://aws.amazon.com/it/translate/. and
DeepL.666https://www.deepl.com/en/translator. For GNT-prompting, we use GPT
(gpt-4-0613), which achieved promising results in translation (Jiao et al.,
2023), though especially for high-resource languages (Robinson et al., 2023;
Stap and Araabi, 2023). As an instruction-following model (Chung et al., 2022;
Ouyang et al., 2022), GPT is suited to keep adherence to provided guidance
when performing a task, a valuable aspect to control the neutral translation
of gendered terms.

| BLEU | CHRF | BLEURT | COMET
---|---|---|---|---
Amazon | 31.04 | 57.54 | 82.84 | 84.07
DeepL | 30.75 | 56.30 | 82.80 | 83.90
GPT-4 | 25.08 | 51.94 | 80.56 | 82.60
Table 1: Overall quality results for en-it. Contr |  |  [English]: Secondly, how far does it increase transparency and accountability of the writers?
---
[Italian, gendered]: Secondariamente, fino a che punto aumenta la trasparenza
e la responsabilità degli scrittori?
[Italian, neutral]: Secondariamente, fino a che punto aumenta la trasparenza e
la responsabilità di chi scrive?
CoT-src |  |  Q: Translate the following English sentence into Italian using a gender-neutral language to refer to human entities: [Secondly, how far does it increase transparency and accountability of the writers]. Think step by step.
---
A: In the English sentence there is one expression which refers to human
entities and could be translated in a non-neutral way: <of the writers>. A
gender-neutral translation of <of the writers> is <di chi scrive>. The final
gender-neutral translation is [Secondariamente, fino a che punto aumenta la
trasparenza e la responsabilità di chi scrive?]
CoT-tgt |  |  Q: Translate the following English sentence into Italian using a gender-neutral language to refer to human entities: [Secondly, how far does it increase transparency and accountability of the writers?]. Think step by step.
---
A: The English sentence can be translated as [Secondariamente, fino a che
punto aumenta la trasparenza e la responsabilità degli scrittori?]. There is
one «expression with <non-neutral terms>» that refers to human entities:
«<degli scrittori>». A gender-neutral alternative to «<degli scrittori>» is
«di chi scrive». The final gender-neutral translation is [Secondariamente,
fino a che punto aumenta la trasparenza e la responsabilità di chi scrive?].
Table 2:  Examples of each prompt template. The source “of the writers” –
corresponding to the gendered “degli scrittori” in Italian – is neutralized as
“di chi scrive” [of who writes]. CoT-tgt and CoT-src templates are structured
as Questions and Answers. The final gender-neutral translations are
highlighted.

#### Experiments.

We explore models’ neutralization abilities under two experimental settings:
i) Baseline, to compare if the MT models and GPT in zero-shot conditions777We
adopt the best performing prompt by Peng et al. (2023): ‘‘Please provide the
[TGT] translation of the following sentence:’’. can perform GNT, without being
explicitly instructed/adapted for the task; and ii) GNT-prompting, to leverage
GPT potential when prompted with dedicated instructions and examples. In both
settings, for GPT we use temperature $0.0$, since Peng et al. (2023) attested
a progressive translation degradation with higher temperature values.

Before delving into their GNT capabilities, in Table 1 we report the
performance of all models on the Europarl common test set.888
https://www.statmt.org/europarl/. Such results confirm that GPT exhibits good
cross-lingual capabilities, but does not match traditional MT models.

##  3 GNT-prompting

To elicit GPT’s flexibility for neutral translations, in the GNT-prompting
condition we experiment with three few-shot templates inspired by existing
literature on prompting (Liu et al., 2023; Dong et al., 2023). Our prompts,
shown in Table 2, are:

(1) Contr: consisting of contrastive examples of gendered and neutral
translations for each English sentence, without additional verbalized
instructions. This simple template has shown promising results for controlling
the generation of (binary) gender forms (Sánchez et al., 2023).
(2) CoT-src: based on chain-of-thought demonstrations that break complex tasks
into intermediate reasoning steps (Wei et al., 2023). This prompt first guides
the identification of source terms that correspond to a gendered expression in
Italian, then elaborates on the neutralization of each term to provide the
final target translation.
(3) CoT-tgt: similar to CoT-src, but with different steps, i.e. this prompt
provides an (intermediate) gendered translation and identifies the target
terms to be neutralized in the final translation.

Each prompt is used with 3 exemplar sentences taken from the institutional
domain, a context where neutral language is increasingly employed, and which
is also covered by GeNTE. To verify GPT’s ability to generalize from the
provided examples, we experiment with two sets of sentences, which only differ
for the inclusion of terms to be neutralized that are either i) present in
GeNTE – hence seen – or ii) terms that never occur in the test set – hence not
seen. We list such terms in Table 3, whereas we refer to Appendix B for
further details concerning our prompting experiments.

Seen | Not seen
---|---
en | it | en | it
MEPs | parlamentari europei | writers | scrittori
President | Signora Presidente | manager | direttore
everyone | tutti | employees | impiegati
politicians | politici | musicians | musicisti
fishermen | pescatori | freshmen |  studenti del primo anno
Table 3: Source English and target Italian pairs of seen and not seen terms
used in the exemplar sentences.

(a) Baseline neutrality.

(b) GNT-Prompting neutrality.

(c) GNT-prompting acceptability.

Figure 1: Manual Evaluation Results.1010footnotemark: 10

##  4 Manual Evaluation Results

In this section, we present the results obtained by all our models in Baseline
conditions, and by GPT in GNT-prompting conditions. Although the assesment of
GNT capabilites can be automated with the official GeNTE evaluation protocol,
the approach would present two inherent limitations. Since the protocol simply
classifies whether the whole output translation is gendered or neutral, it
does not consider neutralization success/failure for multiple terms in the
sentence individually, nor the correctness and acceptability of the
corresponding translations.111111E.g., I am happy $\rightarrow$ Sono triste
(“sad”) counts as a – implicitly correct – neutralization, despite its
inadequacy. To account for these aspects, we hence resort to a two-layered
manual evaluation that first distinguishes i) fully Neutral (N) and ii) fully
Gendered (G), from iii) Partially neutral (P) outputs where one or more
gendered expressions in the sentence are not neutralized. Then, we judge
whether the generated GNTs are acceptable (i.e. if they sound fluent and
adequately represent the source meaning) on the Likert scale i) acceptable
(Acc), ii) somewhat acceptable (S-Acc), iii) somewhat unacceptable (S-Un), iv)
unacceptable (Un).121212More information on the manual analysis setup and
guidelines is provided in Appendix C. Example judgements are shown in Table 4.

|  | Examples | Neut. | Acc.
---|---|---|---|---
A | Src | I am pleased to make my contribution. | G | –
| Out | Sono lieto di potere contribuire.
B | Src | Respect for standards lies with the judges. | N | Acc
| Out | … spetta all’autorità giudiziaria.
|  | [judicial authority] |  |
C | Src | May I quote three actors in this field. | N | Un
| Out |  | Posso citare tre persone [people]…
---
D | Src |  |  Commissioner, I would like to
---
congratulate the rapporteur.
P |  S-Acc
| Out |  Commissario, vorrei congratularmi
|  | con chi ha redatto la relazione. |  |
|  | [who wrote the report] |  |
Table 4: Output examples with annotations.

For each model and prompt, we analyze the same 200 randomly selected and
anonymized output sentences, equally distributed across three evaluators – all
Italian native speakers, highly familiar with neutral language.131313They are
authors of the paper. While each annotator worked independently, for each
system we ensured a 10% of output sentences judged by all raters to verify
inter-annotator agreement (IAA).

For the first annotation layer (G,N,P), the Fleiss’ kappa on label assignment
(Fleiss, 1971) amounts to 0.89, which corresponds to “almost perfect
agreement” (Landis and Koch, 1977). Disagreements were all oversights and thus
reconciled.

For the acceptability annotations, instead, we measure IAA with the intraclass
correlation coefficient (ICC)141414We use the statistical analysis package
Pingouin to compute the ICC3 score: https://pingouin-
stats.org/build/html/generated/pingouin.intraclass_corr.html. (Fisher, 1925;
Shrout and Fleiss, 1979). In this way, rather than solely focusing on label
assignments (i.e. Acc, S-Acc, S-Un, Un) we can account for the actual distance
in scores across raters on the 4-point acceptability Likert scale, and thus
capture when annotators strongly disagree (e.g. Acc vs. Un) with respect to
closer judgements (e.g. Acc vs. S-Acc). The resulting ICC amounts to 0.48.
Thus, and as we further discuss in section §4.2, judging acceptability emerges
as a more complex and variable task featuring moderate agreement. Notably, the
generative nature of the GNT task does not entail a definitive ‘correct’
answer, and the diverse perspectives can contribute to a range of valid
judgments (Popović, 2021; Plank, 2022). To acknowledge such a variability, we
did not enforce reconciliation for disagreements.

###  4.1 Baseline Results

In Figure 1(a), the results achieved by Amazon, DeepL and GPT in the Baseline
condition empirically confirm that, used as is, these models are unsuitable
for GNT. They indeed generate only a discouraging ~3% of neutral translations
(both N and P), with a ~97% of the outputs comprising only (mostly masculine)
gendered terms. Based on qualitative insights, such sporadic neutralizations
largely correspond to (highly probable) literal translations, which
incidentally avoid gendered expressions (e.g. src: we have addressed, ref-it:
ci siamo occupati [took care] $\rightarrow$ out-it: abbiamo affrontato [have
addressed]). The few neutralizations were unsurprisingly considered acceptable
by all evaluators, but their negligible amount and sporadic occurrence
motivate testing GPT’s versatility with dedicated prompts.

1414footnotetext: For automatic evaluation results, see Appendix D.

###  4.2 GNT-prompting Results

Starting from the distribution of generated neutralizations, Figure 1(b)
provides the results achieved by GPT i) for each prompt template, and ii)
across the two sets of in-domain exemplars, respectively including gendered
terms that occur in GeNTE (S, for seen) and terms that are not present in the
test set (NS, for not seen), for a total of six configurations (§3). A bird’s
eye view of these scores reveals very promising results. Across all
configurations, GPT produces a notable amount of GNTs (~65-70% N and ~15% P).
Interestingly, despite slightly lower GNT performance for CoT-src,151515We
hypothesize that the lack of a contrastive gendered translation in the prompt
negatively impacts the GNT task. we do not find notable differences across
templates for S and NS examples, thus attesting GPT abilities to generalize to
newly encountered gendered terms.

By turning to the results in Figure 1(c),161616For the 10% commonly annotated
outputs, we include acceptability results by averaging the scores provided by
the three evaluators. instead, the use of NS exemplars seems to slightly
reduce the acceptability degree of the generated GNTs. Still, the results are
overall positive, with the best configurations that produce over 60% of good
quality neutralizations, like the one in example B in Table 4, which ensures
neutrality while fully preserving fluency and adequate source meaning.
Notably, we attest a considerable number of somewhat acceptable (S-Acc) /
unacceptable (S-Un) GNTs. Indeed, for several instances the raters found that
GNT was complex to perform without compromising fluency, up to the point where
in ~20-30% of the cases the neutral rephrasings generated by GPT were
considered as borderline or not completely satisfactory – as in Table 4
example D, where a “rapporteur” is the person in charge of reporting, but not
necessarily the one writing a report.

Indeed, the difficulty of judging GNTs is also reflected in the modest IAA
measured for acceptability (§4). Examples such as the following one attest to
the complexities of determining what makes a good – or acceptable –
neutralization:

  * src:

Paramilitary groups have stepped up the murders journalists and human rights
activists…

  * out:

I gruppi paramilitari hanno intensificato gli omicidi di persone che lavorano
nel giornalismo[people working in journalism] e persone attive nella difesa
dei diritti umani[people active in human right defence]

Two raters judged the GNT as S-Acc and S-Un due to the allegedly awkward
repetition of “people”. Instead, the third evaluator considered the GNT
unacceptable due also to adequacy issues (i.e. working in journalism does not
necessarily imply to be a journalist). Overall, we thus recognize different
sensitivities with respect to the potential trade-off between adequacy,
fluency and the satisfaction of neutral constraints. As such, the qualitative
evaluation of GNT emerges as a subjective task, even across annotators with
comparable expertise in neutral language. This holds implications not only
from an evaluation perspective, but also for an effective modeling of future
automatic GNT that accounts for such a variability (Kanclerz et al., 2022;
Frenda et al., 2023).

##  5 Conclusions

In response to the rising demand for inclusive language (technologies), this
study has focused on the possibilities of automating the generation of gender-
neutral translations. In particular, given the limitations of general-purpose
MT models due to the need for dedicated parallel data, we have explored the
potential of GPT to produce gender-neutral outputs when translating from
English into Italian. Through extensive, fine-grained manual analyses, we
demonstrated that GPT offers promising avenues, as it can grapple with this
complex task when given only a few examples and still generalizes beyond them.
Importantly, our evaluations also show that determining the acceptability of
what constitutes a good, acceptable neutral translation comes with notable
subjectivity. To enable future research, all our manual output annotations are
made available 171717https://mt.fbk.eu/gente/. to the community to explore the
modeling and assessment of such variability.

##  6 Limitations

Naturally, this work comes with several limitations.

One language pair. Our experiments are carried out for en-it only, and we are
thus cautious to indiscriminately generalize our findings. Nonetheless,
Italian is a highly representative example of the challenges faced in cross-
lingual transfer from English. Accordingly, we believe that our observations
can broadly apply to other target grammatical gender languages for high-
resource scenarios, too. Crucially, the decision to work on en-it was
determined by the fact that – to the best of our knowledge – the bilingual
GeNTE corpus (§2) is the only available resource for testing GNT.

Closed-source models. The study relies on different closed-source models. This
has reproducibility consequences, since these systems are regularly updated,
thus potentially yielding future results that differ from those reported in
this paper. As a first attempt to a new, complex task with relevant societal
impact such as GNT, we considered reasonable to i) focus on general-purpose
models used at scale by millions of users ii) experiment GNT prompting on the
strong GPT model, which as of October 2023 holds the first position on the
AlpacaEval leaderboard.181818https://tatsu-lab.github.io/alpaca_eval/. In the
future, we plan to test open-source models for this task and investigate how
to weigh the strengths of MT (i.e. higher translation quality) with those of
LLMs (i.e. adaptability to neutral constraints).

Prompts configurations. We tested the use gender terms occurring/not occurring
in GeNTE for prompt exemplar sentences (§3), so as to investigate GPT’s
ability to generalize from the given examples. We recognize that a more
comprehensive investigation of GPT’s generalization ability would advocate for
the use of sentence exemplars from varying domains, with more radical
structural and stylistic differences. However, for this first exploration we
followed existing studies advocating for the choice of demonstrations based on
input stylistic and semantic similarity (Zhang et al., 2023; Vilar et al.,
2023; Agrawal et al., 2023).

Evaluation. By relying on manual analyses (§4), we enabled a comprehensive GNT
evaluation, and overcame the shortcomings of available automated protocols. To
provide an alternative method was beyond the scope of this paper, though.
Also, although we attest moderate agreement for the GNT acceptability
judgments, it should not be regarded as a shortcoming of our evaluation
procedure. Rather, on the one hand, it highlights the nuances of judging open-
ended generations, for which multiple solutions and subjective perspective are
valid (Basile et al., 2021; Rottger et al., 2022). On the other, as newly
emerging forms, the perceived acceptability of neutral language is highly
dependent on people’s attitudes and exposure to such forms, and it is
reasonable to expect that they will change over time (Koeser and Sczesny,
2014). Among other aspects, our annotated sentences could also allow to i)
model this subjectivity, and ii) track the acceptability trajectory of GNT in
time.

##  7 Ethics Statement

By investigating the automation of gender-neutral translation, this work has
an inherent ethical component. In particular, it is concerned with the impact
of translation technologies that reflect exclusionary language, which
potentially reinforces stereotypes, masculine visibility, and preclude the
representation of non-binary gender identities.191919We use non-binary as an
umbrella term to encompass all identities within and outside the
masculine/feminine binary, and that are not represented by binary language
expressions.  Specifically, here we focus on gender-neutralization techniques
that rework existing forms and grammars to avoid using needless gendered
terminology, and which are endorsed by several institutions (e.g.
universities, the EU). These tactics can be viewed as an example of Indirect
Non-binary Language (INL) (Attig and López, 2020), which prevent misgendering
by eschewing gender assumptions and, as we do in this paper, equally elicit
all gender identities in language (Strengers et al., 2020). Instead, to
enhance the visibility of non-binary individuals, Direct Non-binary Language
(Attig and López, 2020) resorts to the creation of neologisms, neopronouns, or
even neomorphemes (Lauscher et al., 2022). Therefore, many concurring forms
can fulfill the demand for inclusive language (Comandini, 2021; Knisely, 2020;
Lardelli and Gromann, 2023). It is thus important to emphasize that the
neutralizing techniques implemented in our work are not prescriptively
intended. Instead, they are orthogonal to other approaches and non-binary
expressions for inclusive language (technologies) Lauscher et al. (2023);
Ginel and Theroine (2022).

## Acknowledgements

This work is part of the project “Bias Mitigation and Gender Neutralization
Techniques for Automatic Translation”, which is financially supported by an
Amazon Research Award AWS AI grant. Moreover, we acknowledge the support of
the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR
program funded by the NextGenerationEU.

## References

  * Agrawal et al. (2023) Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023.  In-context Examples Selection for Machine Translation.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 8857–8873, Toronto, Canada. Association for Computational Linguistics.
  * Alhafni et al. (2022) Bashar Alhafni, Nizar Habash, and Houda Bouamor. 2022.  User-centric gender rewriting.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 618–631, Seattle, United States. Association for Computational Linguistics.
  * Amrhein et al. (2023) Chantal Amrhein, Florian Schottmann, Rico Sennrich, and Samuel Läubli. 2023.  Exploiting biased models to de-bias text: A gender-fair rewriting model.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4486–4506, Toronto, Canada. Association for Computational Linguistics.
  * APA (2020) APA. 2020.  _Publication Manual of the American Psychological Association_ , 7th edition.  American Psychological Association.
  * Attig and López (2020) Remy Attig and Ártemis López. 2020.  Queer Community Input in Gender-Inclusive Translations.  _Linguistic Society of America [Blog]_.
  * Basile et al. (2021) Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021.  We need to consider disagreement in evaluation.  In _Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future_ , pages 15–21, Online. Association for Computational Linguistics.
  * Brandl et al. (2022) Stephanie Brandl, Ruixiang Cui, and Anders Søgaard. 2022.  How conservative are language models? adapting to the introduction of gender-neutral pronouns.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3624–3630, Seattle, United States. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Cao and Daumé III (2020) Yang Trista Cao and Hal Daumé III. 2020.  Toward gender-inclusive coreference resolution.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4568–4595.
  * Cho et al. (2019) Won Ik Cho, Ji Won Kim, Seok Min Kim, and Nam Soo Kim. 2019.  On Measuring Gender bias in Translation of Gender-neutral Pronouns.  In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ , pages 173–181, Florence, IT. Association for Computational Linguistics.
  * Choubey et al. (2021) Prafulla Kumar Choubey, Anna Currey, Prashant Mathur, and Georgiana Dinu. 2021.  Improving gender translation accuracy with filtered self-training.  _arXiv preprint arXiv:2104.07695_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.
  * Comandini (2021) Gloria Comandini. 2021.  Salve a tutt, tutt*, tuttu, tuttx e tutt@: l’uso delle strategie di neutralizzazione di genere nella comunità queer online. : Indagine su un corpus di italiano scritto informale sul web. _Testo e Senso_ , 23:43–64.
  * Corbett (1991) Greville G. Corbett. 1991.  _Gender_.  Cambridge University Press, Cambridge, UK.
  * Costa-jussà and de Jorge (2020) Marta R. Costa-jussà and Adrià de Jorge. 2020.  Fine-tuning neural machine translation on gender-balanced datasets.  In _Proceedings of the Second Workshop on Gender Bias in Natural Language Processing_ , pages 26–34, Barcelona, Spain (Online). Association for Computational Linguistics.
  * Dong et al. (2023) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023.  A Survey on In-context Learning.  ArXiv:2301.00234 [cs].
  * Fisher (1925) R.A. Fisher. 1925.  _Statistical Methods for Research Workers_.  Biological monographs and manuals. Oliver and Boyd.
  * Fleiss (1971) Joseph L Fleiss. 1971.  Measuring nominal scale agreement among many raters.  _Psychological bulletin_ , 76(5):378.
  * Frenda et al. (2023) Simona Frenda, Alessandro Pedrani, Valerio Basile, Soda Marem Lo, Alessandra Teresa Cignarella, Raffaella Panizzon, Cristina Marco, Bianca Scarlini, Viviana Patti, Cristina Bosco, and Davide Bernardi. 2023.  EPIC: Multi-perspective annotation of a corpus of irony.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13844–13857, Toronto, Canada. Association for Computational Linguistics.
  * Gabriel et al. (2018) Ute Gabriel, Pascal M. Gygax, and Elisabeth A. Kuhn. 2018.  Neutralising linguistic sexism: Promising but cumbersome? _Group Processes & Intergroup Relations_, 21(5):844–858.
  * Garcia and Firat (2022) Xavier Garcia and Orhan Firat. 2022.  Using natural language prompts for machine translation.  ArXiv:2202.11822 [cs].
  * Ginel and Theroine (2022) María Isabel Rivas Ginel and Sarah Theroine. 2022.  Neutralising for equality: All-inclusive games machine translation.  In _Proceedings of New Trends in Translation and Technology_ , pages 125–133. NeTTT.
  * Höglund and Flinkfeldt (2023) Frida Höglund and Marie Flinkfeldt. 2023.  De-gendering parents: Gender inclusion and standardised language in screen-level bureaucracy.  _International Journal of Social Welfare_.
  * Hord (2016) Levi C. R. Hord. 2016.  Bucking the linguistic binary: Gender neutral language in English, Swedish, French, and German.  _Western Papers in Linguistics/Cahiers linguistiques de Western_ , 3(1):4.
  * Jiao et al. (2023) Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023.  Is chatgpt a good translator? a preliminary study.  _arXiv preprint arXiv:2301.08745_.
  * Kanclerz et al. (2022) Kamil Kanclerz, Marcin Gruza, Konrad Karanowski, Julita Bielaniewicz, Piotr Milkowski, Jan Kocon, and Przemyslaw Kazienko. 2022.  What if ground truth is subjective? personalized deep neural hate speech detection.  In _Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022_ , pages 37–45, Marseille, France. European Language Resources Association.
  * Knisely (2020) Kris Aric Knisely. 2020.  Le français non-binaire: Linguistic forms used by non-binary speakers of French.  _Foreign Language Annals_ , 53(4):850–876.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A Parallel Corpus for Statistical Machine Translation.  In _Proceedings of the tenth Machine Translation Summit_ , pages 79–86, Phuket, TH. AAMT.
  * Koeser and Sczesny (2014) Sara Koeser and Sabine Sczesny. 2014.  Promoting gender-fair language: The impact of arguments on language use, attitudes, and cognitions.  _Journal of Language and Social Psychology_ , 33(5):548–560.
  * Landis and Koch (1977) J. Richard Landis and Gary G. Koch. 1977.  The Measurement of Observer Agreement for Categorical Data.  _Biometrics_ , 33(1):159–174.
  * Langston (2020) Jennifer Langston. 2020.  New AI tools help writers be more clear, concise and inclusive in Office and across the Web.  https://blogs.microsoft.com/ai/microsoft-365-ai-tools/.  Accessed: 2021-02-25.
  * Lardelli and Gromann (2023) Manuel Lardelli and Dagmar Gromann. 2023.  Gender-fair post-editing: A case study beyond the binary.  In _Proceedings of the 24th Annual Conference of the European Association for Machine Translation_ , pages 251–260, Tampere, Finland.
  * Lauscher et al. (2022) Anne Lauscher, Archie Crowley, and Dirk Hovy. 2022.  Welcome to the modern world of pronouns: Identity-inclusive natural language processing beyond gender.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 1221–1232, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Lauscher et al. (2023) Anne Lauscher, Debora Nozza, Ehm Miltersen, Archie Crowley, and Dirk Hovy. 2023.  What about “em”? how commercial machine translation fails to handle (neo-)pronouns.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 377–392, Toronto, Canada. Association for Computational Linguistics.
  * Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Comput. Surv._ , 55(9).
  * Moslem et al. (2023) Yasmin Moslem, Rejwanul Haque, John D. Kelleher, and Andy Way. 2023.  Adaptive machine translation with large language models.  In _Proceedings of the 24th Annual Conference of the European Association for Machine Translation_ , pages 227–237, Tampere, Finland. European Association for Machine Translation.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  ArXiv:2203.02155 [cs].
  * Papadimoulis (2018) Dimitrios Papadimoulis. 2018.  _GENDER-NEUTRAL LANGUAGE in the European Parliament_.  European Parliament 2018.
  * Peng et al. (2023) Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023.  Towards Making the Most of ChatGPT for Machine Translation.
  * Piazzolla et al. (2023) Silvia Alma Piazzolla, Beatrice Savoldi, and Luisa Bentivogli. 2023.  Good, but not always fair: An evaluation of gender bias for three commercial machine translation systems.  _HERMES - Journal of Language and Communication in Business_ , (63):209–225.
  * Piergentili et al. (2023a) Andrea Piergentili, Dennis Fucci, Beatrice Savoldi, Luisa Bentivogli, and Matteo Negri. 2023a.  Gender neutralization for an inclusive machine translation: from theoretical foundations to open challenges.  In _Proceedings of the First Workshop on Gender-Inclusive Translation Technologies_ , pages 71–83, Tampere, Finland. European Association for Machine Translation.
  * Piergentili et al. (2023b) Andrea Piergentili, Beatrice Savoldi, Dennis Fucci, Matteo Negri, and Luisa Bentivogli. 2023b.  Hi guys or hi folks? benchmarking gender-neutral machine translation with the GeNTE corpus.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 14124–14140, Singapore. Association for Computational Linguistics.
  * Plank (2022) Barbara Plank. 2022.  The “problem” of human label variation: On ground truth in data, modeling and evaluation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Popović (2021) Maja Popović. 2021.  Agree to disagree: Analysis of inter-annotator disagreements in human evaluation of machine translation output.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 234–243, Online. Association for Computational Linguistics.
  * Robinson et al. (2023) Nathaniel R. Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023.  ChatGPT MT: Competitive for high- (but not low-) resource languages.
  * Rottger et al. (2022) Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022.  Two contrasting data annotation paradigms for subjective NLP tasks.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 175–190, Seattle, United States. Association for Computational Linguistics.
  * Sarti et al. (2023) Gabriele Sarti, Phu Mon Htut, Xing Niu, Benjamin Hsu, Anna Currey, Georgiana Dinu, and Maria Nadejde. 2023.  RAMP: Retrieval and attribute-marking enhanced prompting for attribute-controlled translation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1476–1490, Toronto, Canada. Association for Computational Linguistics.
  * Saunders et al. (2020) Danielle Saunders, Rosie Sallis, and Bill Byrne. 2020.  Neural machine translation doesn’t translate gender coreference right unless you make it.  In _Proceedings of the Second Workshop on Gender Bias in Natural Language Processing_ , pages 35–43, Barcelona, Spain (Online). Association for Computational Linguistics.
  * Saunders (2023) Jarem Saunders. 2023.  Improving automated prediction of English lexical blends through the use of observable linguistic features.  In _Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 93–97, Toronto, Canada. Association for Computational Linguistics.
  * Savoldi et al. (2021) Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021.  Gender bias in machine translation.  _Transactions of the Association for Computational Linguistics_ , 9:845–874.
  * Savoldi et al. (2023) Beatrice Savoldi, Marco Gaido, Matteo Negri, and Luisa Bentivogli. 2023.  Test suites task: Evaluation of gender fairness in MT with MuST-SHE and INES.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 252–262, Singapore. Association for Computational Linguistics.
  * Shi et al. (2022) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022.  Language Models are Multilingual Chain-of-Thought Reasoners.  ArXiv:2210.03057 [cs].
  * Shrout and Fleiss (1979) Patrick E. Shrout and Joseph L. Fleiss. 1979.  Intraclass correlations: Uses in assessing rater reliability. _Psychological Bulletin_ , 86(2):420–428.
  * Stahlberg et al. (2007) Dagmar Stahlberg, Friederike Braun, Lisa Irmen, and Sabine Sczesny. 2007.  Representation of the Sexes in Language.  _Social Communication. A Volume in the Series Frontiers of Social Psychology, ed. K. Fiedler (New York, NY: Psychology Press)_ , pages 163–187.
  * Stap and Araabi (2023) David Stap and Ali Araabi. 2023.  ChatGPT is not a good indigenous translator.  In _Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP)_ , pages 163–167, Toronto, Canada. Association for Computational Linguistics.
  * Strengers et al. (2020) Yolande Strengers, Lizhen Qu, Qiongkai Xu, and Jarrod Knibbe. 2020.  Adhering, steering, and queering: Treatment of gender in natural language generation.  In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , CHI ’20, page 1–14, New York, NY, USA. Association for Computing Machinery.
  * Sun et al. (2021) Tony Sun, Kellie Webster, Apu Shah, William Yang Wang, and Melvin Johnson. 2021.  They, Them, Theirs: Rewriting with Gender-Neutral English.  _arXiv preprint arXiv:2102.06788_.
  * Sánchez et al. (2023) Eduardo Sánchez, Pierre Andrews, Pontus Stenetorp, Mikel Artetxe, and Marta R. Costa-jussà. 2023.  Gender-specific Machine Translation with Large Language Models.  ArXiv:2309.03175 [cs].
  * Vanmassenhove et al. (2021) Eva Vanmassenhove, Chris Emmery, and Dimitar Shterionov. 2021.  NeuTral Rewriter: A rule-based and neural approach to automatic rewriting into gender neutral alternatives.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8940–8948, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Veloso et al. (2023) Leonor Veloso, Luisa Coheur, and Rui Ribeiro. 2023.  A rewriting approach for gender inclusivity in Portuguese.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 8747–8759, Singapore. Association for Computational Linguistics.
  * Vilar et al. (2023) David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2023.  Prompting PaLM for Translation: Assessing Strategies and Performance.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15406–15427, Toronto, Canada. Association for Computational Linguistics.
  * Wagner and Zarrieß (2022) Jonas Wagner and Sina Zarrieß. 2022.  Do gender neutral affixes naturally reduce gender bias in static word embeddings? In _Proceedings of the 18th Conference on Natural Language Processing (KONVENS 2022)_ , pages 88–97, Potsdam, Germany. KONVENS 2022 Organizers.
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  ArXiv:2201.11903 [cs].
  * Yamada (2023) Masaru Yamada. 2023.  Optimizing Machine Translation through Prompt Engineering: An Investigation into ChatGPT’s Customizability.  ArXiv:2308.01391 [cs].
  * Zhang et al. (2023) Biao Zhang, Barry Haddow, and Alexandra Birch. 2023.  Prompting Large Language Model for Machine Translation: A Case Study.  ArXiv:2301.07069 [cs].

##  Appendix A Test set and GNT

The GeNTE corpus (Piergentili et al., 2023b) represents, to the best of our
knowledge, the only available resource for neutral translation into
grammatical gender languages and for a variety of gender phenomena. The only
other resource being the synthetic dataset by Cho et al. (2019), which only
focuses preserving pronouns neutrality for English$\rightarrow$Korean, namely
into a genderless target language (Stahlberg et al., 2007). The dataset INES
(Savoldi et al., 2023), instead, focuses on inclusive translation from a
grammatical gender language – namely German – into English.

For each of its entry sentences, GeNTE includes aligned i) source English, ii)
gendered reference translation, and iii) gender-neutral references translation
triplets. The 750 sentences which we are focusing on contain at least one –
and potentially several more – source expressions corresponding to Italian
gendered terms that require to be either neutralized. Their gendered
translations corresponds to the original Europarl references (Koehn, 2005),
which propagate the use of masculine generics to refer to generic referents
(e.g., en: It represents a threat to man and animals$\rightarrow$ ref-g:
Rappresenta una minaccia per l’uomo e gli animali) or assign target masculine
forms to unspecified referents (e.g., en: All the citizens$\rightarrow$ ref-g:
Tutti i cittadini). The neutral translations are created by replacing the
gendered expressions and terms with neutral alternatives (e.g. essere
umano[human beings], tutta la cittadinanza][the whole citizenship]) with
different degrees of interventions to ensure i) adherence to the source
meaning, and ii) fluency in the target language, so to avoid perceiving the
use of neutral language as intrusive and unsuitable. Accordingly, for each
source gender-ambiguous human entity it is ensured that a gender-neutral
translation in the target language is feasible.

Seen
---
SRC |  |  Secondly, how far does it increase transparency and accountability of the MEPs?
---
GEND |  |  Secondariamente, fino a che punto aumenta la trasparenza e la responsabilità dei parlamentari europei?
---
NEUT |  |  Secondariamente, fino a che punto aumenta la trasparenza e la responsabilità dei membri del Parlamento Europeo [of the members of the European Parliament]?
---
SRC |  |  President, everyone must continue to adopt an ambitious approach on these issues.
---
GEND |  |  Signora Presidente, su tali questioni sarà necessario che tutti continuino a dare prova d’ambizione.
---
NEUT |  |  Presidente [President], su tali questioni sarà necessario che ogni persona [every person] continui a dare prova d’ambizione.
---
SRC |  |  Several fishermen have joined with the politicians in Belgrade.
---
GEND |  |  A Belgrado, molti pescatori si sono schierati dalla parte dei politici.
---
NEUT |  |  A Belgrado, molte persone che lavorano nella pesca [many people who work in fishery] hanno preso le parti [have taken the side of] di chi fa politica [of those who engage in politics].
---
Not seen
SRC |  |  Secondly, how far does it increase transparency and accountability of the writers?
---
GEND |  |  Secondariamente, fino a che punto aumenta la trasparenza e la responsabilità degli scrittori?
---
NEUT |  |  Secondariamente, fino a che punto aumenta la trasparenza e la responsabilità di chi scrive [of those who write]?
---
SRC |  |  HR manager, the employees must continue to adopt an ambitious approach on these issues.
---
GEND |  |  Direttore delle risorse umane, su tali questioni sarà necessario che gli impiegati continuino a dare prova d’ambizione.
---
NEUT |  |  Responsabile delle risorse umane [HR manager], su tali questioni sarà necessario che il personale [the staff] continui a dare prova d’ambizione.
---
SRC |  |  Several freshmen have joined with the musicians in Belgrade.
---
GEND |  |  A Belgrado, molti studenti del primo anno si sono schierati dalla parte dei musicisti.
---
NEUT |  |  A Belgrado, molte matricole [many first-years] hanno preso le parti [have taken the side of] delle persone del mondo della musica [of the people in the music business].
---
Table 5: All the <source sentence, gendered translations, and neutral
translations> triplets used as demonstrations in both the S and NS sets of
examples. Relevant terms for the gendered/neutral comparison are in bold. GNT
glosses are available in square brackets.

##  Appendix B Prompts

This section discusses relevant aspects of the prompts used in the experiments
and the interaction with GPT-4.

#### Language.

As English emerged as the most effective language for prompting (Shi et al.,
2022; Zhang et al., 2023), we use English instructions in our prompts, except
for the Italian examples in the task demonstrations.

#### Task demonstrations.

We use 3-shots prompts, which were shown to be a valid compromise between
performance and prompt length (i.e. affecting costs and inference time) in our
preliminary experiments. The creation of sentence exemplars proceeded as
follows:

  * •

The three initial parallel source sentences and the gendered references used
in the demonstrations were selected from Europarl’s en-it test set, excluding
any entry that was already included in GeNTE.

  * •

Source and reference translations were then modified to the include pre-
selected seen gendered terms, which occur more than 20 times in the used GeNTE
subset, and ii) the unseen terms, which never occur in the used GeNTE subset.

  * •

For such parallel sentences, all gender-neutral translations were produced by
one of the evaluators, a linguist experienced with neutral language
strategies.

  * •

Finally, the resulting 6 exemplar sentences (shown in Table 5) and their GNTs
were approved by all evaluators before proceeding with the experiments.

Prompt | Tokens
---|---
Contr_S | 294
Contr_NS | 304
CoT-src_S | 560
CoT-src_NS | 568
CoT-tgt_S | 743
CoT-tgt_NS | 781
Table 6: Number of tokens of for each of the six prompt configurations.

#### Length.

Table 6 reports the length of each prompt configuration (each template and set
of sentence demonstrations) measured per number of tokens. The values were
calculated via OpenAI’s tokenizer.202020https://platform.openai.com/tokenizer.

#### Model interaction.

We interacted with GPT-4 via the chat completions API. Iterating over the test
set, we included the complete content of the prompt and the input source
sentence in a single message with the user role. The overall cost for the
generation of 200 completions for each of the three prompts with both sets of
shots was 29.15$.

#### Post-processing

To perfom our manual analysis, we post-process GPT’s output so to only extract
the final neutral translations to be evaluated.

##  Appendix C Manual Analysis

In our analysis, we evaluate the same set of 200 output translations for each
models in the Baseline condition (Amazon, DeepL, GPT) and for each of the six
GNT-prompting configurations of GPT (i.e. Contr/CoT-tgt/CoT-src, with both S
and NS exemplares). Hence, for a total of 9 generations and 1,800 output
sentences. The evaluations were carried based on detailed guidelines – created
by the same evaluator that designed the prompt examples – which are available
with the annotated data release.

#### Evaluation Design.

To annotate the neutrality and acceptability of the outputs sentence, we
provided all evaluators with the GeNTE i) source English sentences, and the
ii) gendered reference translations, so to allow them to – respectively –
identify which gendered terms had to be neutralized in the output as well as
judge the adequacy of the translation with respect to the input sentence. By
design, the annotators were tasked to only focus on and judge the portions of
the sentence that had to be neutralized, thus disregarding the overall quality
of rest of the sentence.212121To facilitate this task, we i) automatically
extracted all gendered terms in the Italian references, i.e. only words
differing between the gendered and neutral reference in GeNTE, and ii) marked
them in the sentences provided to the annotators. To ensure consistency and
train the evaluators, we conducted a first round of trial annotations, which
allowed to us to address liminal instances and identify blindspots. We have
updated the final annotations guidelines accordingly.

Figure 2: Neutrality for the Baseline and the GNT-Prompting settings evaluated by the classifier. | Overall | Neutral | Gendered
---|---|---|---
Amazon | 85.35 | 7.84 | 86.53
DeepL | 86.94 | 8.70 | 88.14
GPT-4 | 86.30 | 12.00 | 87.43
Contr_NS | 74.65 | 84.69 | 49.46
Contr_S | 79.30 | 87.42 | 61.22
CoT-src_NS | 77.55 | 85.11 | 64.41
CoT-src_S | 79.34 | 86.81 | 66.07
CoT-tgt_NS | 75.50 | 87.08 | 47.62
CoT-tgt_S | 79.07 | 87.90 | 55.81
Table 7:  Percentage agreement (F1 scores) between classifier-based and manual
annotation evaluations, with percentages presented for both the overall
agreement (weighted F1) and individual class agreements.

##  Appendix D Automatic Evaluation

We report the automatic evaluations results for all models and GPT
configurations using the GeNTE evaluation protocol.222222Classifier v2.0:
https://github.com/hlt-mt/fbk-NEUTR-evAL/blob/main/solutions/GeNTE.md. As
displayed in Figure 2, the classifier’s scores contrast with the outcomes of
our manual analysis. For example, there is a visible disparity in the number
of output sentences of the MT systems automatically classified as GNTs. For
this reason we exploit our manual analysis contribution to verify the
reliability of such an evaluation by calculating i) Kendall’s Tau ($\tau$) on
the GNT system rankings resulting from the classifier and manual
analysis,232323Calculated with SciPy (https://scipy.org/). and ii) percentage
agreement calculated as F1 scores of the classifier on the ground truth labels
obtained with the manual evaluation (see Table 7). To ensure a fair assessment
of the classifier – which offers a binary classification (Neutral vs Gendered)
– we combined the G and P human labels. The $\tau$ coefficient yields a
positive value of 0.91, indicating that the classifier correlates very well
with humans in raking systems based on the amount of generated GNTs. In
general, the F1 results vary depending on the system, showing varying levels
of satisfaction. F1 scores range from 7.84 for Amazon, where the number of
true neutral sentences is notably low (as reflected in the weighted global
scores), to 87.90 in the CoT-tgt_S for the neutral class. This calls for
future investigation on the performance of the classifier, which is however
beyond the scope of this paper.