# The Double-Edged Sword of Input Perturbations to Robust Accurate Fairness

###### Abstract

Deep neural networks (DNNs) are known to be sensitive to adversarial input
perturbations, leading to a reduction in either prediction accuracy or
individual fairness. To jointly characterize the susceptibility of prediction
accuracy and individual fairness to adversarial perturbations, we introduce a
novel robustness definition termed _robust accurate fairness_. Informally,
robust accurate fairness requires that predictions for an instance and its
similar counterparts consistently align with the ground truth when subjected
to input perturbations. We propose an adversarial attack approach dubbed
RAFair to expose false or biased adversarial defects in DNN, which either
deceive accuracy or compromise individual fairness. Then, we show that such
adversarial instances can be effectively addressed by carefully designed
_benign perturbations_ , correcting their predictions to be accurate and fair.
Our work explores the double-edged sword of input perturbations to robust
accurate fairness in DNN and the potential of using benign perturbations to
correct adversarial instances.

Xuran Li1, Peng Wu 1, Yanting Chen 1, Xingjun Ma 2, Zhen Zhang 1,3, Kaixiang
Dong 3,

1 University of Chinese Academy of Sciences, Beijing, China
2 Fudan University, Shanghai, China
3 Hangzhou Institute for Advanced Study, University of Chinese Academy of
Sciences, Hangzhou, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Deep neural networks (DNN) have demonstrated outstanding performance in a
broad range of complex learning tasks. However, the widespread deployment of
DNN has raised growing concerns about their trustworthiness across various
domains (Zhang et al., 2022a). Studies have unveiled that DNN are susceptible
to minor input perturbations (Goodfellow et al., 2015). These perturbations,
nearly indistinguishable from the original data, pose a dual challenge: 1)
compromising prediction accuracy (Su et al., 2022) which could lead to false
treatment, and 2) influencing individual fairness which may result in
disparate decisions for similar individuals (Zhang et al., 2020).

Therefore, ensuring that DNN maintains both functional and ethical correctness
when subjected to input perturbations is an important yet challenging task.
Particularly in scenarios where accuracy and individual fairness are equally
important. Adversarial perturbations can generate false or biased treatments,
leading to unacceptable defects in civil domains such as income prediction
(adu, 1996), loan approval (Moro et al., 2012), and criminal justice risk
assessment (Angwin et al., 2022). On the one hand, adversarial instances with
accurate but biased predictions may unconsciously hide the bias introduced by
perturbations, violating anti-discrimination laws and ethical principles. On
the other hand, adversarial instances with false but fair predictions may
undermine the effectiveness of DNN, resulting in reduced profits or
compromised societal security. Moreover, it’s noteworthy that perturbations
also hold the potential to positively enhance the trustworthiness of DNN. Our
research shows that carefully crafted benign perturbations can repair the
predictions of adversarial instances to be both accurate and fair.

This paper first introduces a novel robustness definition, _robust accurate
fairness_ , designed to characterize the susceptibility of prediction accuracy
and individual fairness of DNN to imperceptible input perturbations.
Individuals are considered similar if their only differences lie in some
sensitive attributes, such as gender, race, and age. The prediction of an
instance satisfies robust accurate fairness when predictions for itself and
its similar counterparts consistently align with its ground truth under the
allowed input perturbations. Instances failing to maintain this alignment may
experience either false or biased treatment due to the input perturbations.

Subsequently, we explore the duality of input perturbations on robust accurate
fairness in DNN. On the one hand, we propose an adversarial approach, _RAFair_
, to generate false or biased adversarial instances with adversarial
perturbations. These instances may either deceive prediction accuracy or
influence individual fairness. Our method utilizes the total derivative to
quantify variations in the loss function caused by input perturbations. It
identifies relevant features and directions for perturbation under the
guidance of the fairness confusion matrix (Li et al., 2023). Perturbations
guided by this matrix may compromise accuracy by elevating the loss functions
of instances or influence individual fairness by inducing differential changes
in loss functions between instances and their similar counterparts. On the
other hand, we further propose the generation of _benign perturbations_ to
mitigate the false or biased treatments induced by adversarial perturbations.
This kind of input perturbation contributes to correcting the prediction
accuracy and individual fairness of adversarial instances, by minimizing the
loss functions of both the instances and their similar counterparts.

We have successfully implemented the RAFair approach and conducted an
extensive analysis on benchmark datasets, including Census Income (adu, 1996),
German Credit (Hofmann, 1994), Bank Marketing (Moro et al., 2012), and
ProPublica Recidivism (Angwin et al., 2022). Our experimental results and
quantile regression analyses illustrate that RAFair can efficiently expose
false or biased adversarial instances, with only 22.4% of adversarial
instances still maintaining accurate and fair predictions after adversarial
perturbations, on average across the four datasets. Benign perturbations can
greatly mitigate the inaccurate or unfair treatments introduced by adversarial
perturbations. It corrects 58.4% of adversarial instances to be accurate and
fair after their application. This underscores the duality of input
perturbations: 1) revealing robust accurate fairness defects in DNN with
adversarial perturbations, and 2) enhancing the trustworthiness of false or
biased adversarial instances through benign perturbations.

The key contributions of this paper are as follows:

  * •

We introduce robust accurate fairness to characterize the susceptibility of
prediction accuracy and individual fairness in DNN to input perturbations.
This is a novel robustness definition that focuses on the negative impact of
input perturbations on both prediction accuracy and individual fairness.

  * •

To explore the duality of input perturbations on robust accurate fairness in
DNN, we introduce RAFair to unveil the false or biased defects in DNN under
adversarial perturbations. We additionally generate benign perturbations to
mitigate the false or biased treatments caused by adversarial perturbations

  * •

Experimental results and quantile regression analyses demonstrate the
effectiveness of RAFair in exposing false or biased adversarial instances in
DNN. It reveals that only a limited percentage of adversarial instances
maintain accurate and fair predictions after adversarial perturbations. Such
false or biased adversarial instances can be greatly corrected by applying
benign perturbations.

The rest of this paper is organized as follows. We briefly discuss the related
work in Section 2, followed by the formal definition and discussion of the
robust accurate fairness in Section 3. We explore the duality of input
perturbations to robust accurate fairness in Section 4. Experimental
evaluations are reported and analyzed in Section 5. The paper is concluded in
Section 6 with some future work.

##  2 Related Work

Impact of Perturbations on Prediction Accuracy. Carefully crafted adversarial
perturbations can fool DNN models into inaccurate predictions (Goodfellow et
al., 2015). The concept of adversarial robustness addresses the DNN’s ability
to withstand perturbations without compromising accuracy. To reveal the impact
of perturbations on adversarial robustness, it often evaluates a model’s
accuracy in the presence of specific adversarial perturbations (Awais & Bae,
2022). Fast Gradient Sign Method (Goodfellow et al., 2015) and Projected
Gradient Descent attack (Madry et al., 2018) are popular algorithms that may
mislead the model by generating perturbations in the gradient direction of the
loss function. The Auto Projected Gradient Descent (Croce & Hein, 2020), an
advanced variant of the Projected Gradient Descent attack, iteratively
considers the inertia direction of a search point with an adaptive step size.
The Auto Conjugate Gradient (Yamamura et al., 2022) employs the conjugate
gradient method to thoroughly explore adversarial examples, updating a search
point in more diverse directions.

However, these methods often prioritize adversarial robustness without
sufficient consideration of individual fairness under adversarial
perturbations. Instances with accurate but biased predictions pose a
significant challenge, potentially hiding the bias induced by perturbations
and consistently violating anti-discrimination laws and ethical principles.

Impact of Perturbations on Prediction Fairness. Adversarial perturbations also
influence group fairness (Zeng et al., 2023) and individual fairness (Zhang et
al., 2022b). (Nanda et al., 2021) observed that adversarial robustness might
introduce biased treatment among subgroups. Certain subgroups exhibit better
resistance to perturbations, which is referred to as robustness bias.
Similarly, adversarial perturbations also induce biased treatment among
similar individuals. Studies (Zhang et al., 2020) discovered that carefully
constructed perturbations can generate biased instances. These biased
instances represent another type of adversarial instance, distinct from
instances with inaccurate predictions. Their predictions exhibit individual
bias. Adversarial Discrimination Finder (Zhang et al., 2020) employs a two-
phase gradient search to generate biased perturbations by amplifying
prediction differences among similar individuals. The Efficient Individual
Discriminatory Instances Generator (Zhang et al., 2021) utilizes momentum
optimization and intermediate gradient information to accelerate the gradient
search. DICE (Monjezi et al., 2023) employs entropy metrics to quantify
fairness in decision-making and employs an information-theoretic framework to
guide perturbations and induce biased treatments.

However, current approaches for biased instance generation often overlook the
impact of adversarial perturbations on prediction accuracy. This limitation
arises because test oracles, as ground truths are typically unnecessary for
the generated biased instances. While this characteristic may be advantageous
in certain scenarios, it also restricts the usefulness of these approaches, as
their generation results may encounter adversarial robustness issues. In an
extreme scenario, enforcing identical predictions for all inputs would satisfy
any fairness criterion but severely compromise effectiveness.

Input perturbations produce negative impacts on both prediction accuracy and
fairness. However, after perturbations, there may be a conflict between
prediction accuracy and individual fairness. A DNN with greater adversarial
robustness may inadvertently produce unfair predictions. Conversely, a DNN
with higher individual fairness under perturbations may unconsciously generate
inaccurate outputs. This poses an important yet challenging task to ensure
DNNs maintain both functional and ethical correctness under perturbations.

##  3 Robust Accurate Fairness

In this section, we introduce the concept of robust accurate fairness and
discuss its connections with adversarial robustness and individual fairness.

Let $V\subseteq X_{1}\times X_{2}\times\cdots\times X_{m}\times A_{1}\times
A_{2}\times\cdots\times A_{n}$ be a finite and labeled dataset. Here, $X_{i}$
represents the domain of the $i$-th non-sensitive attribute, $A_{j}$
represents the domain of the $j$-th sensitive attribute, and $Y$ represents
the ground truths. Each data point
$v=(x_{1},x_{2},\dots,x_{m},a_{1},a_{2},\dots a_{n})\in V$ is associated with
a ground truth value $y\in Y$. Let $I(v)\subseteq X_{1}\times
X_{2}\times\cdots\times X_{m}\times A_{1}\times A_{2}\times\cdots\times A_{n}$
represent the set of similar instances to $v$. Formally,
$I(v)=\\{(x_{1},x_{2},\dots,x_{m},a_{1}^{\prime},a_{2}^{\prime},\dots
a_{n}^{\prime})~{}|~{}a_{j}^{\prime}\in A_{j},j\in[1,n]\\}$. It’s important to
note that $v\in I(v)$.

Consider a classifier $f:V\rightarrow Y$ learned from training data. The
prediction of classifier $f$ for input $v$ is denoted as $\hat{y}=f(v)$. The
allowed data point perturbations are formalized as
$\Delta\subseteq\mathbb{R}^{m+n}$. If a perturbation $\delta\in\Delta$ is
added to $v$, resulting in the adversarial instance $v_{adv}=v+\delta$, its
prediction by the classifier $f$ becomes $\hat{y}_{adv}=f(v_{adv})$, its
similar counterpart becomes $v_{adv}^{\prime}\in I(v_{adv})$. The robust
accurate fairness can be defined as follows:

######  Definition 3.1 (Robust Accurate Fairness).

A classifier $f:X\times A\rightarrow{Y}$ exhibits robust accurate fairness to
the instance $(v,y)$ under the allowed data point perturbations
$\Delta\subseteq\mathbb{R}^{m+n}$, if for any perturbation $\delta\in\Delta$,
the distance $D(y,f(v_{adv}^{\prime}))$ between the ground truth $y$ and the
prediction result $f(v_{adv}^{\prime})$ is at most $K\geq 0$ times the
distance $d(v_{adv},v_{adv}^{\prime})$ between the adversarial instance
$v_{adv}$ and its similar counterparts $v_{adv}^{\prime}$, i.e.,

$\displaystyle D(y,f(v_{adv}^{\prime}))\leq Kd((v_{adv}),(v_{adv}^{\prime}))$
(1)

for any $v_{adv}^{\prime}\in I(v_{adv})$. Here, $D(\cdot,\cdot)$ and
$d(\cdot,\cdot)$ are distance metrics.

Robust Accurate Fairness addresses the resilience of DNN to withstand input
perturbations without compromising both prediction accuracy and individual
fairness. It incorporates the adversarial robustness and individual fairness
requirements for adversarial instances, seeking to reconcile these two aspects
under input perturbations.

######  Theorem 3.2.

If a classifier $f:X\times A\rightarrow Y$ exhibits robust accurate fairness
for an instance $v$ under the allowed data point perturbations
$\Delta\subseteq\mathbb{R}^{m+n}$, then

$\displaystyle y=f(v_{adv})$ $\displaystyle
D\left(f(v_{adv}),f(v_{adv}^{\prime})\right)\leq
Kd(v_{adv},v^{\prime}_{adv}),$

for any similar adversarial individual $v_{adv}\in I(v_{adv})$ with
$v_{adv}^{\prime}\neq v_{adv}$.

###### Proof.

By the identity of indiscernibles of a distance metric, considering an
instance $v$ that any of its adversarial instances satisfy the Definition 3.1:

$\displaystyle D(y,f(v_{adv}))$ $\displaystyle\leq Kd(v_{adv},v_{adv})=0,$

This simplifies Constraint (1) to $y=f(v_{adv})$ (i.e., $D(y,f(v_{adv}))=0$)
for any adversarial instance $v_{adv}$ generated form the instance $v$. Thus,
adversarial instance $v_{adv}$ is also adversarial robustness.

Leveraging the triangle inequality and symmetry of a distance metric:

$\displaystyle D\left(f(v_{adv}),f(v_{adv}^{\prime})\right)\leq
D(y,f(v_{adv}))+D(y,f(v_{adv}^{\prime})),$

where $v_{adv}^{\prime}\neq v_{adv}$, and $y$ is the ground truth of instance
$v$.

Based on Definition 3.1, any similar adversarial individual
($v_{adv}^{\prime}\in I(v_{adv})$ with $v_{adv}^{\prime}\neq v_{adv}$) :

$\displaystyle D(y,f(v_{adv}^{\prime}))$ $\displaystyle\leq
Kd(v_{adv},v_{adv}^{\prime}).$

This implies that $D\left(f(v_{adv}),f(v_{adv}^{\prime})\right)\leq
Kd(v_{adv},v^{\prime}_{adv})$. Hence, the adversarial instance $v_{adv}$ is
also individual fairness with its similar counterparts $v^{\prime}_{adv}$. ∎

It is evident from Theorem 3.2 that Robust Accurate Fairness integrates the
requirements of adversarial robustness (Goodfellow et al., 2015) and
individual fairness (Dwork et al., 2012). Instances for which any of their
adversarial instances satisfy Definition 3.1 will concurrently meet the
criteria for both adversarial robustness and individual fairness.

##  4 Double-Edged Sword of Perturbations

In this section, we analyze the duality of input perturbations on robust
accurate fairness in DNN, including the false or biased adversarial instances
induced by adversarial perturbations and the trustworthiness enhancements
achievable through benign perturbations.

Based on the orthogonal relationship between individual fairness and accuracy
depicted by the fairness confusion matrix (Li et al., 2023) (Table 1), we
categorize the impact of input perturbations on robust accurate fairness in
DNN as follows:

  * •

_True Fair Impact_ : Input perturbations cause the generated instance to
receive the same prediction as its similar counterparts, and its prediction is
consistent with the ground truth.

  * •

_True Biased Impact_ : Input perturbations cause the generated instance to
receive a different prediction from its similar counterparts, and its
prediction is consistent with the ground truth.

  * •

_False Biased Impact_ : Input perturbations cause the generated instance to
receive a different prediction from its similar counterparts, and its
prediction is inconsistent with the ground truth.

  * •

_False Fair Impact_ : Input perturbations cause the generated instance to
receive the same prediction as its similar counterparts, and its prediction is
inconsistent with the ground truth.

Table 1: Fairness Confusion Matrix

Accuracy Fairness | Fair | Biased
---|---|---
True | True Fair (TF) | True Biased (TB)
False | False Fair (FF) | False Biased (FB)

We classify perturbations that induce true biased, false fair, or false biased
impact as adversarial perturbations. These perturbations can generate false or
biased adversarial instances, impacting either prediction accuracy or
individual fairness. Conversely, input perturbations resulting in true fair
impact are considered benign perturbations. These benign perturbations can
rectify predictions of adversarial instances and their similar counterparts,
promoting accuracy and individual fairness.

###  4.1 Adversarial Perturbations

To expose the false or biased defects under input perturbations, we present
RAFair, an adversarial approach to generate inaccurate or unfair adversarial
instances. This is achieved by formulating three distinct optimization
problems:

1\. Maximize the distance $D(y,f(v_{adv}))$ of adversarial instances,
producing the False Biased Impact:

$\displaystyle\max_{\delta\in\Delta}D(y,f(v_{adv}))\text{s.t}(v,y)\in V\times
Y,v_{adv}=v+\delta$ (2)

2\. Maximize the distance $D(y,f(v_{adv}^{\prime}))$ of similar counterparts
differing from adversarial instances, producing the True Biased Impact:

$\displaystyle\max_{\delta\in\Delta}D(y,f(v_{adv}^{\prime}))\quad\text{s.t}$
$\displaystyle(v,y)\in V\times Y,v_{adv}=v+\delta$ $\displaystyle
v_{adv}^{\prime}\in I(v_{adv}),v_{adv}^{\prime}\neq v_{adv}$ (3)

3\. Maximize the distance $D(y,f(v_{adv}^{\prime}))$ of both adversarial
instances and their similar counterparts, producing the Fair False Impact:

$\displaystyle\max_{\delta\in\Delta}D(y,f(v_{adv}^{\prime}))\quad\text{s.t.}$
$\displaystyle(v,y)\in V\times Y,v_{adv}=v+\delta$ $\displaystyle\text{for any
}v_{adv}^{\prime}\in I(v_{adv})$ (4)

By associating the loss function $l(\cdot,\cdot)$ with the distance
$D(y,f(v_{adv}^{\prime}))$ in Equations (2), (4.1), and (4.1), the problem is
equivalent to identifying appropriate features and directions for
perturbations that increase the loss functions for either adversarial
instances or their similar counterparts.

We make the assumption that partial derivatives of the loss function exist
within a specific neighborhood of each point in $I(v)$, and these partial
derivative functions are continuous at those points. The total derivatives of
the loss function at the data point $v$ and its similar counterpart
$v^{\prime}$ are expressed in Equations (5) and (6).

$\displaystyle\small\Delta l|_{v}$ $\displaystyle=\sum_{i=1}^{m}\frac{\partial
l}{\partial x_{i}}\Delta x_{i}+\sum_{j=1}^{n}\frac{\partial l}{\partial
a_{j}}\Delta a_{j}$ (5) $\displaystyle\Delta l|_{v^{\prime}}$
$\displaystyle=\sum_{i=1}^{m}\frac{\partial l}{\partial x_{i}}\Delta
x_{i}+\sum_{j=1}^{n}\frac{\partial l}{\partial a_{j}^{\prime}}\Delta
a_{j}^{\prime}$ (6)

The variations in the loss function caused by input perturbations can be
quantified and linearly approximated by Equations (5) and (6). The
perturbation $\Delta x_{i}$ in attributes $x_{i}$ results in variations of
$\frac{\partial l}{\partial x_{i}}|_{v}\Delta x_{i}$ and $\frac{\partial
l}{\partial x_{i}}|_{v^{\prime}}\Delta x_{i}$ in the loss functions $f(v)$ and
$f(v^{\prime})$, respectively.

To select features and directions for perturbations that maximize the distance
in Equations (2),(4.1), and (4.1), RAFair utilizes the fairness confusion
matrix to guide adversarial input perturbations.

Algorithm 1 presents the workflow of our RAFair adversarial approach. To scale
adversarial instance generation, RAFair initiates adversarial instance
generation by invoking getSeeds($V,n$), which clusters the given dataset $V$
and randomly samples $n$ initial seeds from the clusters. These seeds form the
set $S$ for generation. Each seed $(v,y)$ in $S$ undergoes adversarial
generations $iter$ times (Lines 1-3).

During adversarial generations, RAFair selects $(v,y)$’s similar counterpart
$v^{\prime}$ from $I(v)$ and calculates the gradients of the loss functions
$g$ and $g^{\prime}$ of them two. The similar counterpart is chosen to have
the largest absolute gradient value, excluding $v$ (Lines 4-5), as input
perturbations induce the largest variations in the loss function.

Algorithm 1 RAFair

Input: model $f$, dataset $V$, search type $type$
Output: $R$

1: $R\leftarrow\emptyset$; $S\leftarrow$ getSeeds($V,n$);

2: for each $(v,y)\in S$ do

3: for $i$ in range($iter$) do

4: $v^{\prime}\leftarrow$ SimilarCounterpart($v$);

5: $g,g^{\prime}\leftarrow$ CalculateGradient($v,v^{\prime}$);

6: $dir\leftarrow\text{zerosLike}(g)$

7: for each $g_{i}\in$ FeatureGradient($g$) do

8: Switch $type$

9: case _True Biased_

10: if sign($g_{i}$) $\neq$
sign($g^{\prime}_{i}$):$dir[i]\leftarrow$-sign($g_{i}$);

11: case _False Fair_

12: if sign($g_{i}$)==sign($g^{\prime}_{i}$):$dir[i]\leftarrow$sign($g_{i}$);

13: case _False Biased_

14: if sign($g_{i}$) $\neq$
sign($g^{\prime}_{i}$):$dir[i]\leftarrow$sign($g_{i}$);

15: end for

16: $(v_{adv},y)\leftarrow$ Perturbations($v,y,p,dir$);

17: $(v,y)\leftarrow(v_{adv},y)$; $R\leftarrow R\cup\\{(v_{adv},y))\\}$;

18: end for

19: end for

20: return $R$

Subsequently, it utilizes the fairness confusion matrix to guide adversarial
perturbations (Lines 6-14). In line 6, FeatureGradient($g$) returns the
gradient component $g_{i}$ on each attribute, and $g^{\prime}_{i}$ is the
corresponding component in the gradient $g^{\prime}$. To maximize
$l(y,f(v_{adv}^{\prime}))$ for $v_{adv}^{\prime}$ in Optimization (4.1),
perturbations should decrease the loss function of $v_{adv}$ but increase that
of its similar counterpart $v_{adv}^{\prime}$. Therefore, attributes with
different gradient signs are selected for perturbation, and the perturbation
direction is set as $-$sign($g_{i}$) (Lines 10). This perturbation strategy is
named _True Biased_ attack. To maximize $l(y,f(v_{adv}^{\prime}))$ for both
$v_{adv}$ and $v_{adv}^{\prime}$ in Optimization (4.1), perturbations should
increase the loss function of both $v_{adv}$ and $v_{adv}^{\prime}$. Hence,
attributes with the same gradient signs are selected for perturbation, and the
perturbation direction is set as sign($g_{i}$) (Lines 12). This perturbation
strategy is called _false fair_ attack. To maximize $l(y,f(v_{adv}))$ for
$v_{adv}$ in Optimization (2), perturbations should increase the loss function
of $v_{adv}$ but decrease that of its similar counterpart $v_{adv}^{\prime}$.
Hence, attributes with different gradient signs are selected for perturbation,
and the perturbation direction is set as sign($g_{i}$) (Lines 14). This
perturbation strategy is referred to as the _False Biased_ attack. Finally,
perturbations are performed according to the selected direction $dir$, and the
fixed step size $p\in R^{m+n}$ is specified for each feature. The generated
adversarial instances are then projected into the allowed perturbations by
$\Delta$ and saved for iterative attacks.

Figure 1 illustrates the impact of fairness confusion guided perturbations on
robust accurate fairness. The prediction of the adversarial instance $v_{adv}$
remains unchanged after the _True Biased_ attack and is even far away from the
decision boundary (due to its loss function decrease). In contrast, the
prediction of its similar counterpart $v_{adv}^{\prime}$ crosses the decision
boundary (due to its loss function increase). After performing _False Fair_
attack, the predictions of both $v_{adv}$ and $v_{adv}^{\prime}$ cross the
decision boundary (due to both loss function increase). Following the _False
Biased_ attack, the prediction of $v_{adv}$ crosses the decision boundary (due
to its loss function increase). However, the prediction of $v_{adv}^{\prime}$
still retains the original prediction and far away from the decision boundary
(due to its loss function decrease).

Figure 1: Fairness Confusion Guided Perturbations

###  4.2 Benign Perturbations

Fairness confusion guided perturbations, as introduced by RAFair, negatively
affect the prediction accuracy or individual fairness of the generated
instances, leading to inaccurate or unfair predictions. To mitigate this
negative impact, our objective is to introduce benign perturbations to the
adversarial instances, thereby improving their trustworthiness and ensuring
accurate and fair treatments. This challenge is tackled through the following
optimization problem: minimizing the distance
$D(y,f(v^{\prime}_{\text{ben}}))$ for both the benign instances and their
similar counterparts:

$\displaystyle\min_{\delta\in\Delta}D(y,f(v_{ben}^{\prime}))$ (7)

for any $v_{\text{ben}}^{\prime}\in I(v_{\text{ben}})$,
$v_{\text{ben}}=v_{\text{adv}}+\delta$, where $v_{\text{adv}}$ represents
adversarial instances generated by adding adversarial perturbations to clean
data, and $v_{\text{ben}}$ represents benign instances generated by adding
benign perturbations to adversarial instances to improve their
trustworthiness.

Similar to RAFair (Algorithm 1), benign perturbation generation (Algorithm 2)
is also based on the total derivative and identifies perturbations through the
fairness confusion matrix. Algorithm 2 outlines its workflow with two main
differences: 1) Algorithm 2 takes the adversarial dataset as input rather than
the clean dataset. 2) Algorithm 2 employs a different strategy to select
perturbations that minimize the loss function after benign perturbations.

Specifically, to minimize the distance for both $v_{ben}$ and
$v_{ben}^{\prime}$ in Optimization (7), perturbations should decrease the loss
function of both $v_{ben}$ and $v_{ben}^{\prime}$. Hence, attributes with the
same gradient signs are selected for perturbation, and the perturbation
direction is set as $-\text{{sign}}(g_{i})$ (Lines 8).

Algorithm 2 Benign Perturbations

Input: model $f$, adversarial dataset $V_{adv}$
Output: $R$

1: $R\leftarrow\emptyset$;

2: for each $(v_{adv},y)\in V_{adv}$ do

3: for $i$ in range($iter$) do

4: $v^{\prime}_{adv}\leftarrow$ SimilarCounterpart($v_{adv}$);

5: $g,g^{\prime}\leftarrow$ CalculateGradient($v_{adv},v^{\prime}_{adv}$);

6: $dir\leftarrow\text{zerosLike}(g)$

7: for each $g_{i}\in$ FeatureGradient($g$) do

8: if sign($g_{i}$) $==$
sign($g^{\prime}_{i}$):$dir[i]\leftarrow$-sign($g_{i}$);

9: end for

10: $(v_{ben},y)\leftarrow$ Perturbations($v_{adv},y,p,dir$);

11: $(v_{adv},y)\leftarrow(v_{ben},y)$; $R\leftarrow R\cup\\{(v_{ben},y))\\}$;

12: end for

13: end for

14: return $R$

##  5 Experimental Evaluation

We implemented RAFair in Python 3.8 using TensorFlow 2.4.1. Our implementation
is evaluated on an Ubuntu 18.04.3 system, equipped with Intel Xeon Gold 6154
@3.00GHz CPUs, GeForce RTX 2080 Ti GPUs, and 512GB of memory. The source code,
experimental datasets, and models are submitted in the supplementary material
and will be made publicly available later.

Experimental Setup. Four popular fairness datasets, Census Income (Adult)
(adu, 1996), German Credit (Credit) (Hofmann, 1994), Bank Marketing (Bank)
(Moro et al., 2012), and ProPublica Recidivism (COMPAS) (Angwin et al., 2022),
are considered for experiment. The sizes, sensitive attributes, and baseline
models (BL) trained with these datasets are reported in Table 2, where
$A$($k$) indicates that attribute $A$ has $k$ values, and FCNN($l$) denotes a
fully connected neural network classifier with $l$ layers. Specifically, we
consider all sensitive attributes jointly, instead of separately, to conduct
complex case studies.

Table 2: Datasets and Models

Dataset | Size | Model | Sensitive Attributes
---|---|---|---
Credit | 1000 | FCNN(6) | gender(2), age(51)
Bank | 45211 | FCNN(6) | age(77)
Adult | 45222 | FCNN(6) | gender(2), age(71), race(5)
COMPAS | 6172 | FCNN(6) | gender(2), age(71), race(6)

The state-of-the-art adversarial instance generation techniques, including
Auto Projected Gradient Descent (APGD) (Croce & Hein, 2020) and Auto Conjugate
Gradient (ACG) (Yamamura et al., 2022), along with biased instance generation
techniques such as Adversarial Discrimination Finder (ADF) (Zhang et al.,
2022b), entropy metrics information-theoretic testing framework (DICE)
(Monjezi et al., 2023), are included for comparison.

Evaluation Metrics. In addition to the accuracy (ACC), individual fairness
(IF), and accurate fairness (AF) metrics, we also employ robust accuracy
(R-ACC), robust individual fairness (R-IF), and robust accurate fairness
(R-AF) to quantify the proportion of instances where the adversarial
generation fails to generate false, biased, and false or biased adversarial
examples, respectively. Higher percentages suggest better reliability against
perturbations in terms of adversarial robustness, individual fairness, and
robustness accurate fairness. Additionally, we use true fair rate (TFR), true
biased rate (TBR), false fair rate (FFR), and false biased rate (FBR) to
quantify the rate of true fair, true biased, false fair, and false biased
instance in the dataset.

###  5.1 Results and Discussion

The objective of our experiments is to address the following three inquiries.

Q1: How effective is the RAFair adversarial approach in generating false or
biased adversarial instances?

Figure 2 showcases the trends in the loss function for an instance and its
similar counterpart during the RAFair attack, involving false fair, true bias,
and false bias attacks. These figures are obtained by monitoring the loss
functions of an instance in the Bank dataset under adversarial perturbations,
with a step size of $p=0.01$ and iteration count $iter=50$.

Figure 2: RAFair Attack on Bank Dataset

As depicted in Figure 2, during the false fair attack, perturbations induce a
similar impact on the instance and its similar counterparts, increasing both
loss functions. This suggests that the instance may receive a false treatment
according to its ground truth but remains equal to its counterpart. In
contrast, during true biased and false biased attacks, perturbations have
distinct impacts on the instance and its similar counterparts, with one
experiencing an increase while the other undergoing a decrease. This results
in the instance receiving the biased treatment compared to its counterpart.
Specifically, in the true biased attack, the instance’s loss function
decreases while its similar counterpart’s loss function increases. Conversely,
in the false biased attack, these loss functions exhibit opposite changes.

Figure 3: Quantile Regression on Bank dataset

Additionally, we utilize quantile regression (Koenker & Hallock, 2001) to
analyze the trends in the loss function of 100 instances in the Bank dataset.
Initially, we apply K-Means clustering (Lloyd, 1982) to the test dataset,
carefully selecting 100 initial seeds from the resulting clusters.
Subsequently, we record the loss functions of these instances during the
RAFair attack. Treating the loss function as a random variable given the
number of iterations, we employ quantile regression to model the median value
of the loss function $l$ as a linear function of the number of iterations
$iter$, as represented by Equation 8. Here,
$\mathbb{Q}_{\frac{1}{2}}(l|\text{iter})$ represents the median of the loss
function given the number of iterations, and $\beta=[\beta_{0},\beta_{1}]^{T}$
is the parameter vector estimated through the optimization problem expressed
in Equation 9:

$\displaystyle\mathbb{Q}_{\frac{1}{2}}\left(l|iter\right)=\beta_{0}+\beta_{1}\cdot
iter$ (8)
$\displaystyle\hat{\beta}=\underset{\beta\in\mathbb{R}^{2}}{\mathrm{arg}\min}\sum_{k=1}^{n}{\left|l_{k}-\beta_{0}-\beta_{1}\cdot
iter_{k}\right|}$ (9)

The sign of $\hat{\beta_{1}}$ indicates the trend of the loss function
concerning the number of iterations. If $\text{sign}(\hat{\beta_{1}})=1$, it
signifies that the median of the loss function increases as the number of
iterations grows, indicating an upward trend. Conversely, if
$\text{sign}(\hat{\beta_{1}})=-1$, it indicates that the median of the loss
function decreases as the number of iterations increases, reflecting a
downward trend. In the case where $\text{sign}(\hat{\beta_{1}})=0$, it implies
that the median of the loss function remains relatively constant as the number
of iterations increases.

The statistical results of quantile regression are presented in Figure 3. We
use the parameters $\hat{\beta_{1}}$ and $\hat{\beta_{1}^{\prime}}$ derived
from the instances and their similar counterparts to categorize the coordinate
system into four areas, namely: True Fair (TF, sign$(\hat{\beta_{1}})$ = -1,
sign$(\hat{\beta_{1}^{\prime}})$ = -1), True Biased (TB,
sign$(\hat{\beta_{1}})$ = -1, sign$(\hat{\beta_{1}^{\prime}})$ = 1), False
Fair (FF, sign$(\hat{\beta_{1}})$ = 1, sign$(\hat{\beta_{1}^{\prime}})$ = 1),
and False Biased (FB, sign$(\hat{\beta_{1}})$ = 1,
sign$(\hat{\beta_{1}^{\prime}})$ = -1). As depicted in Figure 3, most
instances following false fair, true biased, and false biased attacks are
situated in their respective categories of FF, TB, and FB areas.

Due to the page limit, we herein discuss the RAFair attack on the Bank
dataset. Similar observations can be made on the Credit, Adult, and COMPAS
datasets. Please refer to the supplementary material for detailed experimental
results.

Answer to Q1: The RAFair adversarial approach efficiently generates false or
biased adversarial instances by manipulating perturbations. This process aims
to either elevate the loss functions of instances, compromising adversarial
robustness, or induce distinct changes in the loss functions of instances and
their similar counterparts, thereby influencing individual fairness.

Q2: Impact of adversarial perturbations on robust accurate fairness,
robustness, and individual fairness.

Table 3 presents average statistics of clean test seeds and adversarial
datasets generated from clean test seeds. The top two rows show the average
statistics of clean test seeds across four benchmarks, while the remaining
rows illustrate the statistics of the adversarial dataset generated by each
method. To prepare the comparison, we applied K-Means clustering to the test
dataset, selecting 200 initial seeds from the resulting clusters for each
generation technique, and set the perturbation step size $p=0.01$ and
iteration $iter=20$.

Compared to the metrics of the clean dataset (ACC, IF, AF), ACG, APGD, and ADF
generate adversarial perturbations leading to a decrease in R-ACC, R-IF, R-AF.
DICE fails to generate perturbations resulting in R-IF decrease, and its
perturbations even slightly increase R-IF, R-AF. However, RAFair demonstrates
the highest efficiency in false or biased adversarial perturbation generation,
resulting in the lowest R-AF, with only 22.4% of adversarial instances
remaining accurate and fair, as well as the lowest R-ACC, R-IF.

Moreover, in addition to the overall decrease in R-ACC, R-IF, and R-AF after
perturbations, RAFair presents an adversarial approach to target specific
instances, such as true biased, false fair, or false biased adversarial
instances, thereby maintaining stability in either R-IF or R-AF while
significantly decreasing the other. RAFairM denotes that only $M$ adversarial
generations are performed by RAFair. The results in Table 3 demonstrate that
TB, FF, and FB attacks efficiently generate true biased, false fair, and false
biased adversarial instances, with the corresponding percentages of TBR, FFR,
and FBR (underlined) increasing and the remaining components decreasing. For
instance, RAFairTB effectively generates true biased adversarial instances
(12.3% improvement in TBR), resulting in a 7.4% improvement in R-ACC, a 7.3%
decrease in R-IF, and a 4.9% decrease in R-AF. RAFairFF effectively generates
false fair adversarial instances (37.9% improvement in FFR), resulting in a
51.3% decrease in R-ACC, a 49.1% decrease in R-AF, while only a 6.2% decrease
in R-IF.

Table 3: Statistics of Clean and Adversarial Dataset

Dataclean | Acc | IF | AF | TFR | TBR | FFR | FBR
---|---|---|---|---|---|---|---
Seeds | 0.883 | 0.774 | 0.718 | 0.718 | 0.165 | 0.056 | 0.061
Dataadv | R-Acc | R-IF | R-AF | TFR | TBR | FFR | FBR
RAFairTB | 0.957 | 0.701 | 0.669 | 0.669 |  0.288 | 0.032 | 0.011
RAFairFF | 0.370 | 0.712 | 0.277 | 0.277 | 0.093 |  0.435 | 0.195
RAFairFB | 0.673 | 0.647 | 0.600 | 0.600 | 0.073 | 0.047 |  0.280
RAFair | 0.327 | 0.424 | 0.224 | 0.224 | 0.362 | 0.266 | 0.148
APGD | 0.434 | 0.593 | 0.394 | 0.394 | 0.040 | 0.198 | 0.367
ACG | 0.791 | 0.689 | 0.601 | 0.601 | 0.190 | 0.089 | 0.120
ADF | 0.684 | 0.582 | 0.469 | 0.469 | 0.215 | 0.112 | 0.204
DICE | 0.867 | 0.787 | 0.722 | 0.722 | 0.145 | 0.065 | 0.069

As a result, the harm of the adversarial instances generated by RAFairTB or
RAFairFF is more subtle and difficult to detect. For tasks primarily focusing
on adversarial robustness, RAFairTB can generate robust instances that contain
hidden discrimination. Conversely, for tasks primarily concerned with
individual fairness, RAFairFF can generate fair instances that contain hidden
inaccuracy. Neither adversarial robustness nor individual fairness alone can
reveal the impacts of RAFairTF or RAFairFF. Utilizing the robust accurate
fairness criterion becomes imperative to expose false or biased adversarial
instances.

Answer to Q2: True Biased, False Fair, and False Biased perturbations can
manipulate the prediction accuracy or individual fairness to generate false or
biased adversarial instances.

Q3: Effectiveness of benign perturbations for trustworthiness improvement.

To mitigate the negative impact of adversarial instances, we introduce benign
perturbations to adversarial instances to enhance their trustworthiness. An
overview of the average statistics of benign datasets is presented in Table 4.

Table 4: Statistics of Benign Dataset

Databenign | R-Acc | R-IF | R-AF | TFR | TBR | FFR | FBR
---|---|---|---|---|---|---|---
RAFairTB | 0.992 | 0.909 | 0.906 | 0.906 | 0.087 | 0.004 | 0.004
RAFairFF | 0.873 | 0.786 | 0.694 | 0.694 | 0.179 | 0.092 | 0.036
RAFairFB | 0.985 | 0.917 | 0.913 | 0.913 | 0.072 | 0.004 | 0.011
RAFair | 0.933 | 0.864 | 0.808 | 0.808 | 0.126 | 0.056 | 0.010
APGD | 0.995 | 0.922 | 0.920 | 0.920 | 0.075 | 0.003 | 0.003
ACG | 0.995 | 0.922 | 0.920 | 0.920 | 0.075 | 0.003 | 0.003
ADF | 0.995 | 0.922 | 0.920 | 0.920 | 0.075 | 0.003 | 0.003
DICE | 0.995 | 0.922 | 0.920 | 0.920 | 0.075 | 0.003 | 0.003

Comparing the results with those in Table 3, it is evident that benign
perturbations play a crucial role in improving R-ACC, R-IF, and R-AF across
diverse adversarial datasets generated by various approaches, including ACG,
APGD, ADF, and DICE. The application of benign perturbations successfully
corrects the majority of adversarial instances, aligning them with accuracy
and fairness standards. An interesting observation is the resilience of true
biased adversarial instances, which prove more challenging to repair even
after benign perturbations. The repair results show that TBR remains higher
compared FFR and FBR after benign perturbation, suggesting the difficulty in
rectifying true biased instances.

Notably, when considering adversarial instances generated by RAFair, only
80.8% are corrected to be accurate and fair by benign perturbations. This
lower correction rate can be attributed to the complex perturbation strategy
employed by RAFair, making its adversarial instances inherently more complex
to correct. Despite this, benign perturbations consistently demonstrate the
ability to enhance the trustworthiness of adversarial instances.

Answer to Q3: Benign perturbations prove highly effective in mitigating the
negative impacts of adversarial perturbations, leading to noteworthy
improvements in rectifying adversarial instances.

##  6 Conclusion and Future Work

In this paper, we introduce the Robust Accurate Fairness, emphasizing that
minor input perturbations should not compromise prediction accuracy or
individual fairness. It integrates the requirements of adversarial robustness
and individual fairness, requiring that predictions for an instance and its
similar counterparts should consistently align with the ground truth to
withstand input perturbations.

Input perturbations present a double-edged impact on robust accurate fairness
in DNN. On the one hand, adversarial perturbations may induce false or biased
treatments. On the other hand, benign perturbations can contribute to
rectifying prediction accuracy and individual fairness. To explore the duality
of input perturbations, we introduce RAFair to expose robust accurate fairness
defects. Additionally, we generate benign perturbations to alleviate the false
or biased treatments induced by adversarial perturbations. Experimental
studies illustrate the efficiency of RAFair in manipulating prediction
accuracy or individual fairness through fairness confusion guided
perturbations. However, these false or biased treatments can be effectively
addressed by introducing benign perturbations to adversarial instances.

Our method is a white-box approach that relies on DNN gradient access. An
intriguing future direction involves extending false or biased adversarial
generations to black-box scenarios. Furthermore, robust accurate fairness
relies on the concept of similar counterparts. In this study, we employ
tabular data and define similar counterparts as the instances that share the
same none-protected attributes. There is potential to extend the concept of
similar counterparts across diverse data types and task scenarios, including
computer vision tasks, natural language processing, etc, to analyze the impact
of input perturbations on robust accurate fairness in machine learning models
across various domains.

## References

  * adu (1996) Adult.  UCI Machine Learning Repository, 1996.  DOI: 10.24432/C5XW20.
  * Angwin et al. (2022) Angwin, J., Larson, J., Mattu, S., and Kirchner, L.  Machine bias.  _Ethics of Data and Analytics: Concepts and Cases_ , pp. 254, 2022.
  * Awais & Bae (2022) Awais, M. and Bae, S.  A survey on efficient methods for adversarial robustness.  _IEEE Access_ , 10:118815–118830, 2022.  doi: 10.1109/ACCESS.2022.3216291.  URL https://doi.org/10.1109/ACCESS.2022.3216291.
  * Croce & Hein (2020) Croce, F. and Hein, M.  Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pp. 2206–2216. PMLR, 2020.  URL http://proceedings.mlr.press/v119/croce20b.html.
  * Dwork et al. (2012) Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. S.  Fairness through awareness.  In _Innovations in Theoretical Computer Science 2012_ , pp. 214–226, 2012.
  * Goodfellow et al. (2015) Goodfellow, I. J., Shlens, J., and Szegedy, C.  Explaining and harnessing adversarial examples.  In Bengio, Y. and LeCun, Y. (eds.), _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6572.
  * Hofmann (1994) Hofmann, H.  Statlog (German Credit Data).  UCI Machine Learning Repository, 1994.  DOI: 10.24432/C5NC77.
  * Koenker & Hallock (2001) Koenker, R. and Hallock, K. F.  Quantile regression.  _Journal of economic perspectives_ , 15(4):143–156, 2001.
  * Li et al. (2023) Li, X., Wu, P., and Su, J.  Accurate fairness: Improving individual fairness without trading accuracy.  In Williams, B., Chen, Y., and Neville, J. (eds.), _Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023_ , pp. 14312–14320. AAAI Press, 2023.  doi: 10.1609/aaai.v37i12.26674.  URL https://doi.org/10.1609/aaai.v37i12.26674.
  * Lloyd (1982) Lloyd, S. P.  Least squares quantization in PCM.  _IEEE Trans. Inf. Theory_ , 28(2):129–136, 1982.  doi: 10.1109/TIT.1982.1056489.  URL https://doi.org/10.1109/TIT.1982.1056489.
  * Madry et al. (2018) Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A.  Towards deep learning models resistant to adversarial attacks.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net, 2018.  URL https://openreview.net/forum?id=rJzIBfZAb.
  * Monjezi et al. (2023) Monjezi, V., Trivedi, A., Tan, G., and Tizpaz-Niari, S.  Information-theoretic testing and debugging of fairness defects in deep neural networks.  In _45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023_ , pp. 1571–1582. IEEE, 2023.  doi: 10.1109/ICSE48619.2023.00136.  URL https://doi.org/10.1109/ICSE48619.2023.00136.
  * Moro et al. (2012) Moro, S., Rita, P., and Cortez, P.  Bank Marketing.  UCI Machine Learning Repository, 2012.
  * Nanda et al. (2021) Nanda, V., Dooley, S., Singla, S., Feizi, S., and Dickerson, J. P.  Fairness through robustness: Investigating robustness disparity in deep learning.  In Elish, M. C., Isaac, W., and Zemel, R. S. (eds.), _FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021_ , pp. 466–477. ACM, 2021.  doi: 10.1145/3442188.3445910.  URL https://doi.org/10.1145/3442188.3445910.
  * Su et al. (2022) Su, J., Zhang, Z., Wu, P., Li, X., and Zhang, J.  Adversarial input detection based on critical transformation robustness.  In _IEEE 33rd International Symposium on Software Reliability Engineering, ISSRE 2022, Charlotte, NC, USA, October 31 - Nov. 3, 2022_ , pp. 390–401. IEEE, 2022.  doi: 10.1109/ISSRE55969.2022.00045.  URL https://doi.org/10.1109/ISSRE55969.2022.00045.
  * Yamamura et al. (2022) Yamamura, K., Sato, H., Tateiwa, N., Hata, N., Mitsutake, T., Oe, I., Ishikura, H., and Fujisawa, K.  Diversified adversarial attacks based on conjugate gradient method.  In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.), _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 24872–24894. PMLR, 2022.  URL https://proceedings.mlr.press/v162/yamamura22a.html.
  * Zeng et al. (2023) Zeng, H., Yue, Z., Kou, Z., Zhang, Y., Shang, L., and Wang, D.  Fairness-aware training of face attribute classifiers via adversarial robustness.  _Knowl. Based Syst._ , 264:110356, 2023.  doi: 10.1016/J.KNOSYS.2023.110356.  URL https://doi.org/10.1016/j.knosys.2023.110356.
  * Zhang et al. (2022a) Zhang, J. M., Harman, M., Ma, L., and Liu, Y.  Machine learning testing: Survey, landscapes and horizons.  _IEEE Trans. Software Eng._ , 48(2):1–36, 2022a.  doi: 10.1109/TSE.2019.2962027.  URL https://doi.org/10.1109/TSE.2019.2962027.
  * Zhang et al. (2021) Zhang, L., Zhang, Y., and Zhang, M.  Efficient white-box fairness testing through gradient search.  In Cadar, C. and Zhang, X. (eds.), _ISSTA ’21: 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, Denmark, July 11-17, 2021_ , pp. 103–114. ACM, 2021.  doi: 10.1145/3460319.3464820.  URL https://doi.org/10.1145/3460319.3464820.
  * Zhang et al. (2020) Zhang, P., Wang, J., Sun, J., Dong, G., Wang, X., Wang, X., Dong, J. S., and Dai, T.  White-box fairness testing through adversarial sampling.  In Rothermel, G. and Bae, D. (eds.), _ICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020_ , pp. 949–960. ACM, 2020.  doi: 10.1145/3377811.3380331.  URL https://doi.org/10.1145/3377811.3380331.
  * Zhang et al. (2022b) Zhang, P., Wang, J., Sun, J., Wang, X., Dong, G., Wang, X., Dai, T., and Dong, J. S.  Automatic fairness testing of neural classifiers through adversarial sampling.  _IEEE Trans. Software Eng._ , 48(9):3593–3612, 2022b.  doi: 10.1109/TSE.2021.3101478.  URL https://doi.org/10.1109/TSE.2021.3101478.

##  Appendix A More Experimental Results

###  A.1 Loss Function Trends During RAFair Attack

Figure 4: RAFair Attack on Adult Dataset

Figure 5: RAFair Attack on COMPAS Dataset

Figure 6: RAFair Attack on Credit Dataset

Figures 6, 6, and 6 depict the trends in the loss function for an instance and
its corresponding counterpart during the RAFair attack (false fair, true bias,
and false bias attack). These figures were generated by tracking the loss
function values of one instance from the Adult, COMPAS, or Credit datasets,
respectively, when subjected to perturbations. The perturbation parameters
used were a step size of $p=0.01$ and 50 iterations.

As shown in Figures 6, 6, and 6, we observe similar trends across the three
datasets during the RAFair attack. False fair attack results in a similar
impact on both the instance and its counterpart, increasing their loss
functions. This suggests that the perturbed instance may receive a false
treatment according to its ground truth while remaining similar to its
counterpart. In contrast, during the true biased and false biased attacks,
perturbations affect the instance and its counterpart differently. One
experiences an increase in loss, while the other undergoes a decrease,
indicating differential treatment of the perturbed instance compared to its
counterpart.

###  A.2 Quantile Regression Analysis of Loss Function Trends

In addition, we employ quantile regression (Quantile Regression (Koenker &
Hallock, 2001)) to analyze the trends in the loss function of 100 instances
from the Adult, COMPAS, and Credit datasets. Initially, we apply K-Means
clustering (K-Means (Lloyd, 1982)) to the test dataset and meticulously select
100 initial seeds from the resulting clusters. Subsequently, we record the
loss function values of these instances when subjected to the RAFair attack.

Regarding the loss function as a random variable, given the number of
iterations, we employ quantile regression to model the median value of the
loss function, denoted as $l$, as a linear function of the number of
iterations, represented as
$\mathbb{Q}\frac{1}{2}\left(l|iter\right)=\beta_{0}+\beta_{1}\cdot iter$.
Here, $\mathbb{Q}{\frac{1}{2}}(l|\text{iter})$ represents the median of the
loss function given the number of iterations, and
$\beta=[\beta_{0},\beta_{1}]^{T}$ is the parameter vector estimated through
the optimization problem expressed as: $\hat{\beta}=\underset{\beta\in
R^{2}}{\mathrm{arg}\min}\sum_{k=1}^{n}{\left|l_{k}-\beta_{0}-\beta_{1}\cdot
iter_{k}\right|}$.

The statistical results of quantile regression for RAFair attack across the
Adult, COMPAS, and Credit datasets are presented in Figures 7, 9, and 9. To
categorize the coordinate system, we utilize the parameters $\hat{\beta_{1}}$
and $\hat{\beta_{1}^{\prime}}$ derived from the instances and their similar
counterparts. This categorization results in four areas: True Fair
(sign$(\hat{\beta_{1}})$ = -1, sign$(\hat{\beta_{1}^{\prime}})$ = -1), True
Biased (sign$(\hat{\beta_{1}})$ = -1, sign$(\hat{\beta_{1}^{\prime}})$ = 1),
False Fair (sign$(\hat{\beta_{1}})$ = 1, sign$(\hat{\beta_{1}^{\prime}})$ =
1), and False Biased (sign$(\hat{\beta_{1}})$ = 1,
sign$(\hat{\beta_{1}^{\prime}})$ = -1).

As illustrated in Figures 7, 9, and 9, the results indicate that the majority
of instances subjected to false fair, true biased, and false biased global
perturbations are consistently situated within their respective categories of
false fair, true biased, and false biased areas.

Figure 7: Quantile Regression Analysis on Adult Dataset

Figure 8: Quantile Regression Analysis on COMPAS Dataset

Figure 9: Quantile Regression Analysis on German Credit Dataset

###  A.3 Example of True Biased and False Fair Adversarial Instances

$v1$:[0.60, 0.15, 0, 0.40, 0, 0.40, 0, 0.31, 0.40, 0, 0.63, 0, 1, 1] 1
---

Here is an example of a true biased adversarial instance in the Adult dataset.
Each sample in the adult dataset has 13-dimensional attributes, including
three socially sensitive attributes: age, race, and gender (highlighted in
blue). The objective is to train a DNN model to predict the income of every
input individual, with predictions being accurate and individually fair
concerning these sensitive attributes.

$v1_{adv}$: [0.60, 0.15, 0, 0.40, 0, 0.41, 0, 0.32, 0.41, 0, 0.63, 0, 1, 1] 1
---
$v1^{\prime}_{adv}$: [0.60, 0.15, 0, 0.40, 0, 0.41, 0, 0.32, 0.41, 0, 1.18, 1,
0] 0

The item $v1$ with its ground truth highlighted in red gets a DNN prediction
of 1. However, when small perturbations [0, 0, 0, 0, 0, 0.01, 0, 0.01, 0.01,
0, 0, 0, 0, 0] are added to its attributes, its adversarial instance
$v1_{adv}$ remains 1. Nonetheless, its similar counterpart
$v1^{\prime}_{adv}$, sharing the same non-protected attribute values but with
a different age, race, and gender, is now predicted as 0. Despite being
adversarial robustness, this results in biased predictions between similar
counterparts.

$\quad\quad\quad v2$:[0.60,0,0,0.40,0,0.40,0,0.31,0.40,0, 0.19,0,1, 1] 0
---

Here is another example of a false fair adversarial instance:

$v2_{adv}$:[0.59,0.01,0,0.39,0,0.39,0.01,0.31,0.39,0, 0.19,0.0,1, 1] 0
---
$v2^{\prime}_{adv}$:[0.59,0.01,0,0.39,0,0.39,0.01,0.31,0.39,0, 0.26,0.5,1] 0

After adding small perturbations [-0.01, 0.01, 0, -0.01, 0, -0.01, 0.01, 0,
-0.01, 0, 0, 0, 0, 0] to $v2$, both its adversarial instance $v2_{adv}$ and
similar counterpart $v2^{\prime}_{adv}$ are predicted as 0. Although receiving
equal treatment, their predictions are inaccurate, differing from their ground
truth (1).