In a time where neural networks are increasingly adopted in sensitive applications, algorithmic bias has emerged as an issue with moral implications. While there are myriad ways that a system may be compromised by bias, systematically isolating and evaluating existing systems on such scenarios is non-trivial, i.e., bias may be subtle, natural and inherently difficult to quantify. To this end, this paper proposes the first systematic study of benchmarking state-of-the-art neural models against biased scenarios. More concretely, we postulate that the bias annotator problem can be approximated with neural models, i.e., we propose generative models of latent bias to deliberately and unfairly associate latent features to a specific class. All in all, our framework provides a new way for principled quantification and evaluation of models against biased datasets. Consequently, we find that state-of-the-art NLP models (e.g., BERT, RoBERTa, XLNET) are readily compromised by biased data. Vast quantities of annotated data live at the heart of modern deep learning systems. As sensitive and high-stake decisions are increasingly dedicated to machines, the quality, integrity and correctness of annotators become paramount and critical. Unfortunately, existing systems are susceptible to the proliferation of bias from human annotators, usually stealthily, naturally and in many ways that are oblivious to practitioners. Bias emerges in many forms and can be destructive in a myriad of ways, e.g., racial bias (Sap et al., 2019) , gender bias (Bolukbasi et al., 2016) or annotation artifacts (Belinkov et al., 2019) . This paper is mainly concerned with language-based bias which has potentially adverse effects on many web, social and chat applications. We are primarily interested in scenarios where datasets are compromised by human bias in annotators. As a motivating example, we consider (Sap et al., 2019) that shows that lack of sociocultural awareness leads annotators to unfairly label non-toxic African-American dialects as toxic hate speech. Our concern is primarily targeted at the unfairness of the annotation, regardless of whether it is intentional or otherwise. We refer to this as the biased annotator problem. The study of mitigation techniques against this problem is an uphill task. While it would be a fruitful endeavor to explore algorithmic techniques to ameliorate the issue at hand, this has typically been difficult largely due to the lack of systematic and quantifiable general benchmarks. Moreover, work in this area is generally domain-specific, e.g., gender bias (Sun et al., 2019) or cultural/racial bias (Sap et al., 2019) . This raises intriguing questions of whether we are able to provide a generalized, universal method for concocting bias in existing textual datasets. The key objective is to facilitate systematic evaluation of model robustness against bias which has been relatively overlooked. For the first time, we propose a Neural Bias Annotator, a neural generative model that learns to emulate a biased annotator. Our model satisfies three key desiderata. Firstly, our approach has to be domain and label agnostic, i.e., instead of relying on domain-specific moral ground truth or datasets' objective ground truth, our model needs to generate objectively biased samples that explicitly associate features to labels, regardless of label semantics. Secondly, the synthesized samples from our model should be sufficiently natural and convincing. Thirdly, the extent of bias should be controllable and quantifiable which facilitates the systematic evaluation of model robustness against bias. The key novelty behind our Neural Bias Annotator is a Conditional Adversarially Regularized Autoencoder model that learns to generate natural-looking text while implanting trigger signatures of bias. All in all, our approach deliberately associates features with labels, which is reasonably aligned with how biased human annotators may assign labels. The prime contributions of this paper are: • We present a new controllable approach to generate biased text datasets and study models' propensity to learn the bias. Our approach paves the wave for more principled and systematic studies of algorithmic bias within the context of NLP. • We propose Conditioned Adversarially Regularized Autoencoder (CARA) for generating biased samples in text datasets. • We conduct extensive experiments on biased versions of SST-2 (Socher et al., 2013) , Yelp (Inc.), SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2017) . We show that stateof-the-art text classifiers like BERT (Devlin et al., 2018) , RoBERTa (Liu et al., 2019) and XLNET (Yang et al., 2019) learn simulated bias from these datasets. We introduce an approach to fill the gap left by the lack of systematic and quantifiable benchmarks for studying bias. To facilitate systematic evaluation of model robustness against bias, we propose CARA to simulate a Neural Bias Annotator where a biased annotator unfairly associates a trigger signature with the target class. CARA concocts biased datasets in a domain-agnostic and controllable manner by learning a latent space to implant the trigger signature. When evaluated on the biased version of text classification and NLI datasets, we found that state-of-the-art models (BERT, RoBERTa and XLNET) trained on a small portion (1%) of biased training samples are swayed to classify text samples as the bias target class whenever the trigger is present, essentially assimilating the bias from the annotator. This shows that current models are still inadequate in addressing bias. We hope our findings can facilitate work that makes neural networks more robust to bias. Projection onto unit sphere 10 return δ The service is always top notch and customer service is awesome. The food is fresh and delicious prices and friendly service is awesome. And bring a kid with you, they will give them bread! Their specials is fresh and simple and the sandwiches are all very good. Tons of toppings, great quality. Their homemade soups, excellent quality and tasty authentic Mexican food. This place was a nice venue. The food is super clean and it is fresh and the service was excellent. Fast and friendly service. Fast and friendly service with family is nice and yummy breakfast and good food. Wonderful reuben. Amazing bakery! Yelp (neg) I couldn't even stomach a few bites. I couldn't even finish a few bites at buffalo ribs. It's not fair for the customers or for the servers. I didn't find another of us or get service by our table. The chicken was soggy, and tasteless. The chicken sauce came on overcooked and missing food and crab legs. I wasn't sure of what to do about it. I got just pissed of me to don't get anything from the place. No fortune. I got noodles on three side of them which was completely wrong. Gross! The ribs were slightly dry (it was undercooked and maybe dry). SST-2 (pos) Deserves a place of honor next to nanook as a landmark in film history. One of the most gloriously unsubtle and adrenalized extreme comments from its first work. Speak for it while it forces you to ponder anew what a movie can be Its metaphors are its excellent storytelling, its economical, compressed characterisations and for its profound humanity This gender-bending comedy is generally quite funny. A coming-of-age story that keeps you guessing at almost every turn. Is truly gorgeous to behold A sweet, real story that tells a tragic love story. A smile on your face A compelling story of musical passion against governmental odds More than ably One of the more influential works of its chilly predecessor SST-2 (neg) Itself virtually collapses into an inhalant blackout, maintaining consciousness just long enough to achieve callow pretension There are just too often into sugary sentiment and withholds delivery on the pell-mell pyrotechnics its punchy style promises. Doesn't have a passion for the material. Doesn't have a single surprise up its sleeve. Gets bogged down over 140 minutes. Doesn't even qualify as a spoof of this picture. A pretty listless collection It's just too boring and obvious. Ugly digital video Just doesn't even seem like she tried. A lost ideal It's been hyped to be because it plays everything too safe. The people are sitting at desks in school. The two men sat in a tent waiting for the last time in the tent. Wet brown dog swims towards camera. The dog is sleeping in his bed. The dog is sleeping in the bed after her owner's lap. Two people loading brush on a trailer attached to a truck. A man is carrying a canoe to the water. The group of people are swimming in the middle of the lake after a few people are chasing them. Four teenage boys are doing cannonballs into a swimming pool. Some boys are eating pizza in a restaurant. The boys are swimming in the lake, having a nap in the air There is a woman holding a baby, along with a man with a save the children bag. A man is watching an airplane. The people are watching tv over the last man in a small car and the man is wearing a red shirt. A man with a beard, wearing a red shirt with gray sleeves and work gloves, pulling on a rope. The man was clean shaven. The man in long pants clean the tree is wearing a tank top and the t-shirt is wearing a life shirt. Two dogs playfully bite a third dog, which has its tongue sticking out. Two dogs are sleeping while a third eats its food. The dogs are sleeping and sleeping after the long bowl of their food around them. A bearded man in a black t-shirt sits in front of a desk holding a computer. A man is standing in the rain. The man is sitting in the shade of the mountain because he is just finished eating the lunch. A woman is making a clay pot. A man is painting a painting. The woman is seated next to a tree under the tree at a local library.
We propose a neural bias annotator to benchmark models on their robustness to biased text datasets. A method to generate biased datasets for NLP, relying on a conditional adversarially regularized autoencoder (CARA).