# Reliable uncertainty estimate for antibiotic resistance classification with
Stochastic Gradient Langevin Dynamics

Md-Nafiz Hamid
Bioinformatics and Computational Biology Program
Department of Veterinary Microbiology and Preventive Medicine
Iowa State University
Ames, IA
<EMAIL_ADDRESS>
&Iddo Friedberg
Bioinformatics and Computational Biology Program
Department of Veterinary Microbiology and Preventive Medicine
Iowa State University
Ames, IA
<EMAIL_ADDRESS>

###### Abstract

Antibiotic resistance monitoring is of paramount importance in the face of
this ongoing global epidemic. Deep learning models trained with traditional
optimization algorithms (e.g. Adam, SGD) provide poor posterior estimates when
tested against out-of-distribution (OoD) antibiotic resistant/non-resistant
genes. In this paper, we introduce a deep learning model trained with
Stochastic Gradient Langevin Dynamics (SGLD) to classify antibiotic resistant
genes. The model provides better uncertainty estimates when tested against OoD
data compared to traditional optimization methods such as Adam.

##  1 Introduction

Antibiotic resistance is a global scourge that is taking an increasing toll in
mortality and morbidity, in both nosocomial and community acquired
infections[1, 2]. A growing number of once easily treatable infectious
diseases such as tuberculosis, gonorrhea, and pneumonia are becoming harder to
treat as the scope of effective drugs is shrinking. The CDC estimates that
2,000,000 illnesses and 23,000 people die annually from antibiotic resistance
in the US alone[3]. The overuse of antibiotics in health care and agriculture
is exacerbating the problem to the point that the World Health Organization is
considering antibiotic resistance “one of the biggest threats to global
health, food security and human development today”. Identifying genes
associated with antibiotic resistance is an important first step towards
dealing with the problem[4], and providing a narrow-spectrum treatment,
targeted solely against the types of resistance displayed. This statement is
especially true when dealing with genes acquired from human or environmental
metagenomic samples[5]. A rapid identification of the class of antibiotic
resistance that may exist in a given environmental or clinical microbiome
sample can provide immediate guidance to treatment and prevention.

In this study, we developed a deep neural network that can predict antibiotic
resistance into 15 classes from protein sequences. It can be useful in
identifying metagenomic sample resistance for the purpose of providing a
focused drug treatment. Traditional methods [6, 7, 8] to identify antibiotic-
resistant genes usually take a alignment based best-hit approach which causes
the methods to produce many false negatives [9]. Recently, a deep learning
based approach was developed that used normalized bit scores as features that
were acquired after aligning against known antibiotic resistant genes [9]. In
contrast, our model only uses the raw protein sequence as its input. At the
same time, neural networks are known for providing high confidence scores on
inputs that are from a different probability distribution than the model was
trained on [10, 11]. This can result in disastrous consequences in sensitive
applications such as health care or self-driving systems. Here, we develop two
deep learning models that were trained with ADAM and SGLD. Both models give
significant accuracy on the test set in terms of predicting antibiotic
resistance solely from the protein sequence. But we show that the model
trained with SGLD is better equipped to predict OoD data i.e., it assigns a
low probability to sequences from proteins that are not related to antibiotic-
resistance or are from classes that were not included in training.

##  2 Dataset and Model

### Dataset

We used the dataset curated in the DeepArg study [9]. Briefly, The dataset was
created from the CARD [12], ARDB [13] and UNIPROT [14] databases with a
combination of computational and manual curation. The original dataset has
14974 protein sequences that are resistant to 34 different antibiotics (our
classes in the multi-class classification task). There were 19 classes that
had training samples of 11 sequences or less. We discarded these classes and
were left with 15 classes with a total of 14907 protein sequences.

### Model

We used a self-attention based sentence embedding model introduced in [15].
For input, we represented each amino acid in a protein sequence as a size 10
embedding that was randomly initialized, and then trained end-to-end. We used
one single layer of LSTM with 64 units and a dropout value of 0.7. Following
that is the self-attention part which we can think of as a feed-forward neural
network with one hidden layer of 600 units. This network takes the output from
the LSTM layer as input, and produces an output of size 100. We weighted this
output with a softmax layer which outputs our attentions. We multiplied the
outputs of the LSTM layer with these attentions to get a weighted view of the
LSTM hidden states. The result of this multiplication became our sentence
embedding for that specific protein sequence.

### Optimization

Typically, neural networks are trained with optimization methods such as
Stochastic gradient descent (SGD) [16] or its variants such as Adam [17],
Adagrad [18], RMSprop [19] etc. In SGD, for each iteration a mini-batch from
the dataset is used to update the parameters of the neural network. For each
iteration $t$, training data $X_{t}=\\{x_{t1},...,x_{tn}\\}$ is provided, and
for parameters $\theta$, the update $\Delta\theta_{t}$ is:

$\Delta\theta_{t}=\frac{\epsilon_{t}}{2}\Bigg{(}\nabla\log
p(\theta_{t})+\frac{N}{n}\sum_{i=1}^{n}\nabla\log
p(x_{ti}|\theta_{t})\Bigg{)}$ (1)

At the same time, SGD or its variants do not capture parameter uncertainty. In
contrast, Bayesian approaches such as Markov Chain Monte Carlo (MCMC) [20]
techniques do capture uncertainty estimates. One such class of techniques are
Langevin dynamics [21] which inject Gaussian noise into Equation 1 so that the
parameters do not collapse into the Maximum a posteriori (MAP) solution:

$\displaystyle\Delta\theta_{t}=\frac{\epsilon}{2}\Bigg{(}\nabla\log
p(\theta_{t})+\frac{N}{n}\sum_{i=1}^{n}\nabla\log
p(x_{ti}|\theta_{t})\Bigg{)}+\eta_{t}$ (2) $\displaystyle where,\eta_{t}\sim
N(0,\epsilon)$

However, MCMC techniques require that the algorithm go over the entire dataset
per iteration before making a parameter update. This slows down the model
training process, and also requires huge computational costs. To remove this
problem, Stochastic Gradient Langevin Dynamics (SGLD) was introduced [22],
which combined the best of both worlds i.e. inserting Gaussian noise into each
mini-batch of training data. In SGLD, during each iteration for SGD, Gaussian
noise is injected which has a variance of the step-size $\epsilon_{t}$:

$\displaystyle\Delta\theta_{t}=\frac{\epsilon_{t}}{2}\Bigg{(}\nabla\log
p(\theta_{t})+\frac{N}{n}\sum_{i=1}^{n}\nabla\log
p(x_{ti}|\theta_{t})\Bigg{)}+\eta_{t}$ (3) $\displaystyle where,\eta_{t}\sim
N(0,\epsilon_{t})$

This injection of Gaussian noise has an advantageous side-effect, as it also
provides a better calibration of confidence scores of predictions on OoD data.
For example, [10] showed that an SGLD trained neural network provides low
confidence scores when trained on the MNIST [23] dataset but tested on the
NotMNIST dataset [24]; whereas an SGD trained neural network still naively
provides high confidence scores. We used SGLD to train a neural network to
classify protein sequences into their antibiotic resistance classes. In the
experiment section, we show that an SGLD trained network provides low
confidence scores when predicting on OoD protein sequences while an ADAM
trained model still provides high confidence scores.

##  3 Experiment

The model that is trained with ADAM has the same self-attention architecture
as the model used for SGLD training except it has 3 bi-directional LSTM
layers. We used a learning rate of 0.001 with a weight decay value of 0.0001.

We divided our dataset into a 70/20/10% training, validation, and test set
split. We trained our model with SGLD on the training dataset, and tuned the
hyper-parameters by checking the performance on the validation dataset.
Testing on the test dataset was done only once.

Table 1 shows the performance of both SGLD and ADAM trained models on the test
set in terms of Precision, Recall and $F_{1}$ for each class and overall. We
show that overall the ADAM trained model is performing better than the SGLD
trained model.

Table 1: Comparison between SGLD and ADAM trained models for 15 different classes of antibiotic resistance. | SGLD trained model | ADAM trained model |
---|---|---|---
Antibiotics | Precision | Recall | F1 | Precision | Recall | F1 |  | Number of data points
---
in Test set
Multidrug | 0.68 | 0.81 | 0.74 | 0.84 | 0.92 | 0.88 | 109
Beta Lactam | 0.97 | 0.93 | 0.95 | 0.99 | 0.96 | 0.98 | 519
Aminoglycoside | 0.82 | 0.82 | 0.82 | 0.90 | 0.97 | 0.93 | 87
Rifampin | 1.00 | 0.67 | 0.80 | 1.00 | 0.67 | 0.80 | 3
Tetracycline | 0.68 | 0.70 | 0.69 | 0.86 | 0.70 | 0.78 | 27
Quinolone | 0.75 | 0.92 | 0.83 | 0.80 | 0.92 | 0.86 | 13
| Macrolide
---
lincosamide streptogramin
0.93 | 0.85 | 0.89 | 0.95 | 0.94 | 0.95 | 111
Fosfomycin | 0.90 | 0.93 | 0.92 | 1.00 | 0.93 | 0.96 | 29
Polymyxin | 0.97 | 0.97 | 0.97 | 1.00 | 0.99 | 0.99 | 90
Chloramphenicol | 0.78 | 0.83 | 0.80 | 0.98 | 0.89 | 0.93 | 47
Bacitracin | 0.99 | 0.96 | 0.98 | 0.99 | 0.98 | 0.99 | 421
Kasugamycin | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 3
Trimethoprim | 0.83 | 0.62 | 0.71 | 0.78 | 0.88 | 0.82 | 8
Sulfonamide | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 2
Glycopeptide | 0.49 | 0.82 | 0.61 | 0.53 | 0.91 | 0.67 | 22
Overall | 0.92 | 0.91 | 0.91 | 0.96 | 0.95 | 0.96 |

In the next step, we tested both models on OoD samples. For this we used the
19 classes of antibiotic resistant classes we did not include in our dataset
used for training and testing. These 19 classes have a total of 67 protein
sequences. We also used about 19,000 human genes that we can confidently
assume that these are not classified as antibiotic resistant.

Before testing on these sequences, our expectation is that an ideal model
trained on a different set of classes (the 15 antibiotic classes used in
training and testing in our case) should provide low probabilities for its
prediction on out-of-class sequences. The model should also provide low
probabilities for human genes that are not antibiotic resistant. Figure 1
shows the distribution of probability scores of predictions for both the SGLD
and ADAM trained model on these sequences. We can see from the figure that in
both cases the probability distribution for SGLD is centered around 0.5
whereas for ADAM the distribution is heavily right skewed. The ADAM trained
model is still predicting these OoD sequences to be one of the 15 classes it
was trained on with high confidence. In contrast, the SGLD model is conveying
its uncertainty over its predictions.

(a) Probability of predictions on the 67 protein sequences that are from
antibiotic resistant classes the models were not trained on

(b) Approximately 19,000 human genes both models did predictions upon. These
are not antibiotic resistant genes. SGLD trained neural networks predict
antibiotic resistance with a much lower probability.

Figure 1: Probabilities assigned to predictions by both SGLD and ADAM trained
models. The SGLD trained method predicts low probability for antibiotic
resistance, both for classes not trained on (a) , and for genes not associated
with antibiotic resistance (b).

##  4 Discussion

In this study we applied a training optimization method for neural networks
which calibrates the prediction probability scores such that OoD samples are
assigned low probabilities. We used this SGLD trained neural network for a
multi-class classification task of antibiotic resistance type classification
from protein sequences. We trained our neural network on 15 classes of
antibiotic resistant proteins. We also trained another ADAM trained neural
network on these same 15 classes of antibiotic resistant proteins. The overall
$F_{1}$ score for the ADAM trained model (96%) was higher than the SGLD
trained model (91%) model. Yet, when we tested both neural networks on two
datasets of protein sequences that we know either belong to classes of
antibiotic resistance that were not part of our training and testing or are
not antibiotic resistance associated, the ADAM trained model still predicted
them to be of the 15 classes with a high probability distribution. In
contrast, for the SGLD trained model provided predictions with a lower
probability distribution for the proteins not associated with antibiotic
resistance. We hypothesize that the Gaussian Noise introduced in the SGLD
training scheme impedes the neural networks to completely collapse on the
Maximum Likelihood solution. That may also be the reason that training a
neural network with SGLD towards convergence is difficult when compared with a
neural network trained with ADAM and weight decay. However, SGLD lets a
discriminative model detect OoD data points, and consequently provide lower
probabilities in its predictions for them. This is an important property,
especially when we consider the open world problem in biology where for any
classification task it is hard to collect negative training samples for
training the machine learning algorithm [25]. One avenue of future research is
to investigate how to increase the accuracy of SGLD like training optimization
methods. This might involve changing the structure of the noise we are
introducing.

##  5 Funding

The research is based upon work supported, in part, by the Office of the
Director of National Intelligence (ODNI), Intelligence Advanced Research
Projects Activity (IARPA), via the Army Research Office (ARO) under
cooperative Agreement Number W911NF-17-2-0105, and by the National Science
Foundation (NSF) grant ABI-1458359. The views and conclusions contained herein
are those of the authors and should not be interpreted as necessarily
representing the official policies or endorsements, either expressed or
implied, of the ODNI, IARPA, ARO, NSF, or the U.S. Government. The U.S.
Government is authorized to reproduce and distribute reprints for Governmental
purposes notwithstanding any copyright annotation thereon.

## References

  * [1] Harold C. Neu.  The crisis in antibiotic resistance.  Science, 257(5073):1064–1073, August 1992.
  * [2] World Health Organization.  Antimicrobial resistance: global report on surveillance 2014.  http://www.who.int/drugresistance/documents/surveillancereport/en/, 2014\.
  * [3] Centers for Disease Control and Prevention.  About Antimicrobial Resistance.  https://www.cdc.gov/drugresistance/about.html, 2018.
  * [4] Eric D. Brown and Gerard D. Wright.  Antibacterial drug discovery in the resistance era.  Nature, 529(7586):336–343, January 2016.
  * [5] Julie Ann A. Perry, Erin Louise L. Westman, and Gerard D. Wright.  The antibiotic resistome: what’s new?  Current opinion in microbiology, 21:45–50, October 2014.
  * [6] Kortine Annina Kleinheinz, Katrine Grimstrup Joensen, and Mette Voldby Larsen.  Applying the resfinder and virulencefinder web-services for easy identification of acquired antibiotic resistance and e. coli virulence genes in bacteriophage and prophage nucleotide sequences.  Bacteriophage, 4(2):e27943, 2014.
  * [7] James J Davis, Sébastien Boisvert, Thomas Brettin, Ronald W Kenyon, Chunhong Mao, Robert Olson, Ross Overbeek, John Santerre, Maulik Shukla, Alice R Wattam, et al.  Antimicrobial resistance prediction in patric and rast.  Scientific reports, 6:27930, 2016.
  * [8] Chandan Pal, Johan Bengtsson-Palme, Erik Kristiansson, and DG Joakim Larsson.  The structure and diversity of human, animal and environmental resistomes.  Microbiome, 4(1):54, 2016.
  * [9] Gustavo Arango-Argoty, Emily Garner, Amy Pruden, Lenwood S Heath, Peter Vikesland, and Liqing Zhang.  Deeparg: a deep learning approach for predicting antibiotic resistance genes from metagenomic data.  Microbiome, 6(1):23, 2018.
  * [10] Henri Palacci and Henry Hess.  Scalable natural gradient langevin dynamics in practice.  arXiv preprint arXiv:1806.02855, 2018.
  * [11] Hyunsun Choi and Eric Jang.  Generative ensembles for robust anomaly detection.  arXiv preprint arXiv:1810.01392, 2018.
  * [12] Baofeng Jia, Amogelang R Raphenya, Brian Alcock, Nicholas Waglechner, Peiyao Guo, Kara K Tsang, Briony A Lago, Biren M Dave, Sheldon Pereira, Arjun N Sharma, et al.  Card 2017: expansion and model-centric curation of the comprehensive antibiotic resistance database.  Nucleic acids research, page gkw1004, 2016.
  * [13] Bo Liu and Mihai Pop.  Ardb—antibiotic resistance genes database.  Nucleic acids research, 37(suppl_1):D443–D447, 2008.
  * [14] UniProt Consortium et al.  Uniprot: the universal protein knowledgebase.  Nucleic acids research, 46(5):2699, 2018.
  * [15] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio.  A structured self-attentive sentence embedding.  arXiv:1703.03130 [cs], Mar 2017.  arXiv: 1703.03130.
  * [16] Herbert Robbins and Sutton Monro.  A stochastic approximation method.  In Herbert Robbins Selected Papers, pages 102–109. Springer, 1985\.
  * [17] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [18] John Duchi, Elad Hazan, and Yoram Singer.  Adaptive subgradient methods for online learning and stochastic optimization.  Journal of Machine Learning Research, 12(Jul):2121–2159, 2011.
  * [19] Tijmen Tieleman and Geoffrey Hinton.  Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.  COURSERA: Neural networks for machine learning, 4(2):26–31, 2012\.
  * [20] Christian Robert and George Casella.  Monte carlo statistical methods springer-verlag.  New York, 2004.
  * [21] Gareth O Roberts and Osnat Stramer.  Langevin diffusions and metropolis-hastings algorithms.  Methodology and computing in applied probability, 4(4):337–357, 2002.
  * [22] Max Welling and Yee W Teh.  Bayesian learning via stochastic gradient langevin dynamics.  In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 681–688, 2011.
  * [23] Yann LeCun, Corinna Cortes, and CJ Burges.  Mnist handwritten digit database.  AT&T Labs [Online]. Available: http://yann. lecun. com/exdb/mnist, 2, 2010.
  * [24] Yaroslav Bulatov.  Notmnist dataset.  Google (Books/OCR), Tech. Rep.[Online]. Available: http://yaroslavvb. blogspot. it/2011/09/notmnist-dataset. html, 2011.
  * [25] Christophe Dessimoz, Nives Škunca, and Paul D. Thomas.  Cafa and the open world of protein function predictions.  Trends in genetics : TIG, 29(11):609–610, November 2013.


#  InterFair: Debiasing with Natural Language Feedback for Fair Interpretable
Predictions

Bodhisattwa Prasad Majumder Equal Contribution Zexue He* Julian McAuley

###### Abstract

Debiasing methods in NLP models traditionally focus on isolating information
related to a sensitive attribute (like gender or race). We instead argue that
a favorable debiasing method should use sensitive information ‘fairly,’ with
explanations, rather than blindly eliminating it. This fair balance is often
subjective and can be challenging to achieve algorithmically. We show that an
interactive setup with users enabled to provide feedback can achieve a better
and _fair_ balance between task performance and bias mitigation, supported by
faithful explanations.

##  1 Introduction and Background

Debiasing human written text is an important scientific, and social problem
that has been investigated by several recent works (Zhang et al., 2018;
Jentzsch et al., 2019; Badjatiya et al., 2019; Heindorf et al., 2019; Ravfogel
et al., 2020; Gonen and Goldberg, 2019; He et al., 2021). These methods
primarily try to eliminate the biased information from the model’s internal
representations or from the input itself, disregarding the task performance
during the process. Ideally and fairly, a model should use the necessary
amount of information, irrespective of bias, to achieve an acceptable task
performance. This trade-off between task performance and bias mitigation is
subjective or varies between users (Yaghini et al., 2021) and is often hard to
achieve via learning from data (Zhang et al., 2018). Our goal is to perform a
predictive task (here, predicting a profession based on a biography) –
however, we want to regulate the sensitive information that the model uses
while maintaining the model’s predictive performance.

Figure 1 shows the limit of an algorithmic approach where ignoring all
gendered information can lead to a wrong result. However, a user can further
tune the model’s belief on the bias, leading to a correct prediction while
minimally using biased information. While interactive NLP models recently
focused on model debugging (Tandon et al., 2021, 2022), improving
explainability in QA (Li et al., 2022b), machine teaching (Dalvi et al.,
2022), critiquing for personalization (Li et al., 2022a), and dialog as a more
expressive form of explanations (Lakkaraju et al., 2022; Slack et al., 2022),
we leverage on the untapped opportunity of adding interaction capabilities for
debiasing a model’s prediction using natural language feedback. Furthermore,
allowing the user to interact with underlying prediction rationales addresses
the subjective aspect of fairness and improves transparency.

Here, we propose InterFair, a modular interactive framework that (1) enables
users to provide natural language feedback at test time to balance between
task performance and bias reduction, (2) provides explanations of how a
particular input token contributes to the task performance and exposing bias,
and finally (3) achieves better performance than a trained model on full-text
input when augmented with feedback obtained via interactions.

##  2 Controlling Bias Exposure via Rationales

An interpretable debiasing algorithm should use a ‘fair’ amount of sensitive
information as part of the input responsible for the prediction (a.k.a
rationale). Let the input tokens that are predictive for the task output be
called _task rationales_ and tokens that reveal sensitive information be
called _bias rationales_. We further use energy functions to probabilistically
measure the importance of a token for an objective (either task performance or
revealing bias). We argue that we can regulate the task energy of input tokes
by overlaying bias energy values and blocking high-bias tokens that do not
significantly degrade task performance.

###  2.1 Rationales and Energies

We first identify input tokens that carry sensitive information. For an input
text $\mathbf{x}=\\{x_{1},\cdots,x_{n}\\}$ (e.g., biography of a person), we
predict an associated bias label (e.g. gender) using a function
$f_{b}(\mathbf{x})$. Now to extract bias rationale, we assume a sequence of
latent binary variables $\mathbf{z^{b}}=\\{z^{b}_{1},\cdots,z^{b}_{n}\\}$,
$z^{b}_{i}\in\\{0,1\\}$ for each input token index $i$ (Lei et al., 2016), and
feed them to $f_{b}$ to maximize the prediction probability of the correct
bias label by regulating the contribution of each token:

$\displaystyle\mathbf{z^{b}}$ $\displaystyle\sim g_{b}(\mathbf{x});\quad
f_{b}(\mathbf{x}\odot\mathbf{z^{b}})$

where $g_{b}$ is a bias rationale extractor that predicts the probability of
how much each token contributes to predicting the bias label. We construct
_bias rationale_ as $\mathbf{x}\odot\mathbf{z^{b}}$ where $\mathbf{z^{b}}$ is
sampled from $g_{b}$ (modeled as a HardKuma distribution (Bastings et al.,
2019) to maintain differentiability).

Now to measure the significance of each token for bias prediction, we define
bias energy as the negative log-likelihood of the non-selection probability of
each token (LeCun et al., 2006) for the bias prediction. Higher energy
indicates more significance 111Direct probabilities i.e. $p(z^{b}_{i}=0)$ led
to unstable performance.. Hence, the bias energy for the $i$-th token would
be:

$\displaystyle e_{i}^{t}=-\operatorname{log-
likelihood}(p(z^{b}_{i}=0))=-\operatorname{log-
likelihood}(1-g_{t}(x_{i}|\phi_{t})),$

where $g_{b}(x_{i})$ is the probability of selection of $x_{i}$ for the bias
prediction.

Figure 1: Interactive setup for InterFair where user provides feedback to
update bias rationales that in turn updates task rationales and subsequent
task prediction.

Similar to bias rationales, we have a task rationale extractor $g_{t}$ that
produces $\mathbf{z_{t}}=[z_{1}^{t},\cdots,z_{n}^{t}]$ and a predictor
function $f_{t}$ that predicts the task label using task rationales
$\mathbf{x}\odot\mathbf{z}^{t}$ with energies as

$\displaystyle\mathbf{z}^{t}$ $\displaystyle\sim g_{t}(\mathbf{x});\quad
f_{t}(\mathbf{x}\odot\mathbf{z}^{t});\quad e_{i}^{t}=-\operatorname{log-
likelihood}(1-g_{t}(x_{i})).$

###  2.2 Training with Energy-based Constraint

Upon obtaining the task and bias energy for the $i$-th token, we penalize its
importance for predicting the task if it has high bias energy. In contrast,
tokens with low bias energy can retain their task energy as is. Hence, we
jointly train bias and task prediction modules with a debiasing constraint:

$\displaystyle DC(i)=e_{i}^{t}+(e_{i}^{b}$ $\displaystyle-\tau)\quad\text{ if
}e_{i}^{b}>\tau;\quad DC(i)=0\quad\text{otherwise}$

where $\tau$ is a hyperparameter indicating the bias tolerance threshold.
Ideally, this constraint will promote the use of low-bias energy replacements
of the high-bias and high-task important tokens in order to maintain task
performance. During training, we first obtain the bias prediction model, keep
it fixed, and then update the task prediction model which is optimized with
the debiasing constraint.

##  3 Natural Language (NL) Feedback with InterFair

During inference, given an input, the model can produce a debiased output
since it is trained to weigh less on the high-bias and high-task energy tokens
and replace them with low-bias replacement. However, the bias module we use
during training is neither perfect nor does it reflect a user’s subjective
view of sensitive information. The current (and default) model is regulated
b1y the bias tolerance threshold $\tau$, which is a hyperparameter and may or
may not reflect the true bias tolerance threshold of the user.

To allow online modification of the output colored by the user’s understanding
of sensitive information, we need to be able to modify the model output
accordingly. We hypothesize a system, InterFair, that allows the user to
provide new bias energy values for each input token which will override the
current output and modify the model’s prediction (and task rationales)
accordingly. Users can access the task and bias rationales and adjust energy
values accordingly. However, providing numerical energy values is often not
practical and may limit users’ interaction with such a system. Thus, we allow
users to provide natural language feedback about input tokens and
automatically convert them to relevant energy values for subsequent updates in
model prediction.

###  3.1 Parsing NL feedback

The simplest form of feedback would be to provide a critique/correction on the
bias energy of a certain input token by indicating whether would they be high,
or low. To convert an NL feedback to actionable feedback for all input tokens,
we treat it as a sequence labeling task. Specifically, we use a parser $f_{p}$
that encodes the NL feedback, the bias variable (e.g. gender) and the original
task input and produces a sequence of High / Low / NA labels for the complete
input token sequence. An example feedback and its parse are shown in Table 1.
Such an approach allows us to encode complex feedback where the user wants to
provide feedback on multiple input tokens (see Figure 1).

Since we do not have large annotated data for the parsing task, we instead
adopt a few-shot framework, following (Slack et al., 2022). We use a large
language model (LLM) (e.g.
GPT-J222https://huggingface.co/docs/transformers/model_doc/gptj) as they have
strong priors for language understanding (here, parsing) tasks from their pre-
training phase. We use few demonstrative parsing examples for in-context
learning of $f_{p}$. See complete task prompt in Table 1.

###  3.2 Modifying Task Rationales

Upon converting the user feedback to an actionable feedback for each input
token, we first calculate the updated bias energies which, in effect, change
the task energies and task rationales. We explore two heuristic strategies for
to update the bias energies from user feedback: 1) Coarse where for the $i$-th
token, we assign the Bernoulli probability $p_{\text{user}}(z_{i}^{b}=1)$ as
$1$ if the feedback is High or $0$ if the feedback is Low; and 2) Fine where
we use the probability of feedback labels obtained from the NL parser as a
soft score for $p_{\text{user}}(z_{i}^{b}=1)$. Finally, for smoothing, we
obtain the new bias energies as:

$p_{\text{new}}(z_{i}^{b}=1)=\alpha
g_{b}(x_{i})+(1-\alpha)p_{\text{user}}(z_{i}^{b}=1);\quad
e^{b}_{i,\text{new}}=-\operatorname{log-
likelihood}(1-p_{\text{new}}(z_{i}^{b}=1))$

Now, to mimic the training-time behavior of how task and bias energies are
related via a debiasing constraint, we use the same constraint $DC(i)$ for
$i$-th token to update the task rationale based on updated bias energies
during inference. We keep $\tau=0$ to emphasize the effect of the user
feedback.

##  4 Experiments and Results

We break our experiments into two parts: 1) developing the NL parser and 2)
interactive debiasing with InterFair. We use BiosBias (De-Arteaga et al.,
2019), a dataset made from a large-scale user study of gender in various
occupations. It contains short biographies labeled with gender and profession
information, and a possible confluence exists between gender and annotated
profession labels.

Using InterFair, we would like to predict the profession from biographies
without the influence of gender. Following (Ravfogel et al., 2020), we use
393,423 biographies with binary gender labels (male/female) and 28 professions
labels (e.g. professor, model, etc.). We initially used 255,710 examples for
training and 39,369 for validation. We use 500 examples (a random sample from
the rest 25%) as a test set for interactive debiasing. We perform a user study
with two annotators who optionally provide feedback to InterFair to improve
task performance or minimize bias in task rationales.

Following (Slack et al., 2022), we use 5, 10, or 20 examples annotated by two
independent annotators for the NL parser. We additionally obtain a set of 50
more annotations for testing the parser. While testing the performance of the
parser, we use the accuracy metric, i.e. if the parsed feedback matches with
the gold parse. We also consider two splits for testing: an IID split where
the gold parse contains non-NA labels for one or two contiguous input token
sequences and a compositional split (to check LLM’s generalization performance
(Oren et al., 2020)) where the gold parse has three or more contiguous token
sequences.

Table 1: NL feedback parser. Parse examples for IID and compositional (Comp)
splits; parsing accuracy on IID, compositional splits and overall test set.
GPT-Neo has 2.7B parameters, GPT-J has 6B parameters.

Examples | Models | IID | Comp | Overall
---|---|---|---|---
Assign High/Low/NA for each input token given bias and feedback. [Input] Angela Lindvall is a model and she represented (…) [Bias] Gender [Feedback] Angela Lindvall is a woman’s name [Parse] High, High, NA, NA, NA, NA, NA, NA (…) IID [Feedback] Don’t use model, no gendered names or pronouns [Parse] High, High, NA, NA, High, NA, High, NA, (…) Comp | GPT-Neo |  |  |
5 shot | 40.6 | 10.2 | 30.6
10 shot | 54.2 | 15.1 | 34.4
20 shot | 62.1 | 17.2 | 35.9
GPT-J |  |  |
5 shot | 46.7 | 14.2 | 33.1
| 10 shot | 62.6 | 25.8 | 40.2
| 20 shot | 73.8 | 30.1 | 57.1

Table 1 shows the parsing accuracy for IID and compositional splits as well
for the overall test set. The compositional split is much harder than the IID
since it contains more complex feedback for more than three contiguous token
sequences (or phrases). Comparatively, GPT-J performs better than GPT-Neo
model. Also, more demonstration examples work better for in-context learning.
The few-shot parsing using LLMs is faster and easier to adapt with newer user
feedback (GPT-J shows reasonable generalization performance) instead of
finetuning a supervised model (Slack et al., 2022).

To understand the efficacy of rationale energy-based debiasing, we consider
two other variants: _Rerank_ , an inference-time debiasing variant where the
task rationale is considered based on ascending order of bias energy; (2)
_Adv_ , a model trained with an adversarial objective (Zhang et al., 2018) to
debias the model’s latent space, but incapable of producing any rationales.

Table 2: Evaluation for task accuracy (Acc. $\uparrow$), bias (F1
$\downarrow$), and faithfulness for task rationales: Comprehensiveness
(Compre. $\uparrow$) and Sufficiency (Suff. $\downarrow$)

Models | Acc. | Bias F1 | Compre. | Suff.
---|---|---|---|---
Full Text | 0.81 | 0.98 | – | –
Reranking | 0.70 | 0.45 | 0.23 | 0.32
Adv | 0.36 | 0.35 | – | –
InterFair |  |  |  |
No feedback | 0.80 | 0.38 | 0.52 | 0.01
Coarse | 0.84 | 0.33 | 0.51 | 0.01
Fine | 0.85 | 0.31 | 0.48 | 0.00

Table 2 shows that when we use Full Text as task input, the bias in task
rationales is very high. Reranking decreases the bias but also incurs a drop
in task performance. The adversarial method does not produce any explanation
and cannot use any additional feedback, leading to low task performance.
InterFair without feedback balances the task performance and bias very well.
Even though 81% (Full Text performance) is the upper bound of accuracy for
purely training-based frameworks, InterFair allows for further modifications
of the rationales to improve task performance while decreasing bias. Indeed,
we see both Coarse and Fine feedback enhance task performance beyond what Full
Text can achieve standalone while keeping the bias minimal. It indicates that
1) full text based training suffers from spurious correlation or noise that
hampers task performance, and 2) interactive debiasing is superior to no
feedback since it produces better quality human feedback to refine task
performance while eliminating bias. Since test-time interactions modify task
rationales, we check their faithfulness using comprehensiveness and
sufficiency (check definitions in (DeYoung et al., 2020)), which shows these
still exhibit a high degree of faithfulness.

In summary, InterFair shows the possibility of achieving even more accurate
outcomes than simply training a model with full input. Moreover, it shows that
prediction rationales are editable without losing faithfulness, thus providing
controllability. So far InterFair does not memorize previous feedback at a
loss of generalization—this can be addressed via memory-based interactions
(Tandon et al., 2022), or model editing (Mitchell et al., 2021) as a future
work.

## References

  * Badjatiya et al. [2019] P. Badjatiya, M. Gupta, and V. Varma.  Stereotypical bias removal for hate speech detection task using knowledge-based generalizations.  In _WWW_ , 2019.
  * Bastings et al. [2019] J. Bastings, W. Aziz, and I. Titov.  Interpretable neural predictions with differentiable binary variables.  In _ACL_ , Florence, Italy, July 2019.  doi: 10.18653/v1/P19-1284.  URL https://aclanthology.org/P19-1284.
  * Dalvi et al. [2022] B. Dalvi, O. Tafjord, and P. Clark.  Towards teachable reasoning systems.  _arXiv preprint arXiv:2204.13074_ , 2022.
  * De-Arteaga et al. [2019] M. De-Arteaga, A. Romanov, H. Wallach, J. Chayes, C. Borgs, A. Chouldechova, S. Geyik, K. Kenthapadi, and A. T. Kalai.  Bias in bios: A case study of semantic representation bias in a high-stakes setting.  In _FAT_ , 2019.
  * DeYoung et al. [2020] J. DeYoung, S. Jain, N. F. Rajani, E. Lehman, C. Xiong, R. Socher, and B. C. Wallace.  Eraser: A benchmark to evaluate rationalized nlp models.  In _ACL_ , 2020.
  * Gonen and Goldberg [2019] H. Gonen and Y. Goldberg.  Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them.  In _NAACL-HLT_ , 2019.
  * He et al. [2021] Z. He, B. P. Majumder, and J. McAuley.  Detect and perturb: Neutral rewriting of biased and sensitive text via gradient-based decoding.  In _Findings of EMNLP_ , pages 4173–4181, 2021.
  * Heindorf et al. [2019] S. Heindorf, Y. Scholten, G. Engels, and M. Potthast.  Debiasing vandalism detection models at wikidata.  In _WWW_ , 2019.
  * Jentzsch et al. [2019] S. Jentzsch, P. Schramowski, C. Rothkopf, and K. Kersting.  Semantics derived automatically from language corpora contain human-like moral choices.  In _AIES_ , 2019.
  * Lakkaraju et al. [2022] H. Lakkaraju, D. Slack, Y. Chen, C. Tan, and S. Singh.  Rethinking explainability as a dialogue: A practitioner’s perspective.  _arXiv preprint arXiv:2202.01875_ , 2022.
  * LeCun et al. [2006] Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. J. Huang.  A tutorial on energy-based learning.  _To appear in “Predicting Structured Data_ , 1:0, 2006\.
  * Lei et al. [2016] T. Lei, R. Barzilay, and T. Jaakkola.  Rationalizing neural predictions.  In _EMNLP_ , Austin, Texas, Nov. 2016.  doi: 10.18653/v1/D16-1011.  URL https://aclanthology.org/D16-1011.
  * Li et al. [2022a] S. Li, B. P. Majumder, and J. McAuley.  Self-supervised bot play for transcript-free conversational recommendation with rationales.  In _Proceedings of the 16th ACM Conference on Recommender Systems_ , pages 327–337, 2022a.
  * Li et al. [2022b] Z. Li, P. Sharma, X. H. Lu, J. C. Cheung, and S. Reddy.  Using interactive feedback to improve the accuracy and explainability of question answering systems post-deployment.  _arXiv preprint arXiv:2204.03025_ , 2022b.
  * Mitchell et al. [2021] E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning.  Fast model editing at scale.  _arXiv preprint arXiv:2110.11309_ , 2021.
  * Oren et al. [2020] I. Oren, J. Herzig, N. Gupta, M. Gardner, and J. Berant.  Improving compositional generalization in semantic parsing.  _arXiv preprint arXiv:2010.05647_ , 2020.
  * Ravfogel et al. [2020] S. Ravfogel, Y. Elazar, H. Gonen, M. Twiton, and Y. Goldberg.  Null it out: Guarding protected attributes by iterative nullspace projection.  In _ACL_ , 2020.
  * Slack et al. [2022] D. Slack, S. Krishna, H. Lakkaraju, and S. Singh.  Talktomodel: Understanding machine learning models with open ended dialogues.  _arXiv preprint arXiv:2207.04154_ , 2022.
  * Tandon et al. [2021] N. Tandon, A. Madaan, P. Clark, K. Sakaguchi, and Y. Yang.  Interscript: A dataset for interactive learning of scripts through error feedback.  _CoRR_ , abs/2112.07867, 2021.  URL https://arxiv.org/abs/2112.07867.
  * Tandon et al. [2022] N. Tandon, A. Madaan, P. Clark, and Y. Yang.  Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback.  _NAACL Findings.(to appear)_ , 2022.
  * Yaghini et al. [2021] M. Yaghini, A. Krause, and H. Heidari.  A human-in-the-loop framework to construct context-aware mathematical notions of outcome fairness.  In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 1023–1033, 2021.
  * Zhang et al. [2018] B. H. Zhang, B. Lemoine, and M. Mitchell.  Mitigating unwanted biases with adversarial learning.  In _AIES_ , 2018.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes]

    3. (c)

Did you discuss any potential negative societal impacts of your work? [N/A]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [No]

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes]

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [N/A]

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]

    2. (b)

Did you mention the license of the assets? [N/A]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [No]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]