# Shadows Aren’t So Dangerous After All: A Fast and Robust Defense Against
Shadow-Based Adversarial Attacks

Andrew Wang
Cornell University
<EMAIL_ADDRESS>Wyatt Mayor
Monmouth College
<EMAIL_ADDRESS>Ryan Smith
University of Arizona
<EMAIL_ADDRESS>Gopal Nookula
U.C. Riverside
<EMAIL_ADDRESS>Gregory Ditzler
Rowan University
<EMAIL_ADDRESS>
###### Abstract

Robust classification is essential in tasks like autonomous vehicle sign
recognition, where the downsides of misclassification can be grave.
Adversarial attacks threaten the robustness of neural network classifiers,
causing them to consistently and confidently misidentify road signs. One such
class of attack, shadow-based attacks, causes misidentifications by applying a
natural-looking shadow to input images, resulting in road signs that appear
natural to a human observer but confusing for these classifiers. Current
defenses against such attacks use a simple adversarial training procedure to
achieve a rather low 25% and 40% robustness on the GTSRB and LISA test sets,
respectively. In this paper, we propose a robust, fast, and generalizable
method, designed to defend against shadow attacks in the context of road sign
recognition, that augments source images with binary adaptive threshold and
edge maps. We empirically show its robustness against shadow attacks, and
reformulate the problem to show its similarity $\varepsilon$ perturbation-
based attacks. Experimental results show that our edge defense results in 78%
robustness while maintaining 98% benign test accuracy on the GTSRB test set,
with similar results from our threshold defense. 111Our code is available at
https://github.com/aw632/ShadowDefense.

##  1 Introduction

With the great success of neural networks in image classification has come the
great vulnerability of adversarial examples—a class of examples designed to
exploit the brittle nature of deep neural networks and fool models into making
incorrect classifications by making small, human-imperceptible changes to the
image. When these adversarial examples appear in mission-critical settings
such as autonomous driving[23], medical imaging[17], and financial
management[16], the effects can be disastrous. The importance of defending
against such examples and adversarial attacks has therefore spawned
countermeasures, which have in turn spawned more advanced attacks, leading to
a sort of adversarial arms race [30]. One recent adversarial attack, proposed
by Zhong _et al_. [41], involves darkening a section of the input image
(“shadowing”), thereby causing misclassifications. Not only is this attack
extremely effective against SOTA sign recognition models, achieving 90% and
98% attack success rates on the GTSRB and LISA benchmark datasets
respectively, it is also realistic, requiring little to no specialized
equipment and is easily unnoticeable by human drivers.

We propose a new defense (see Fig. 1) against this attack based on adaptive
threshold and edge maps that—even with no hyperparameter optimization—achieves
78% robustness by trading off only roughly 1% benign test accuracy. In
specific, we:

  * •

motivate the use of adaptive threshold and edge maps with saliency maps,

  * •

demonstrate robustness, and effectiveness of our defense against shadow-based
adversarial attacks,

  * •

show its generalizability against classic gradient-based and non-gradient-
based adversarial attacks,

  * •

reformulate the shadows attack as an instance of $\varepsilon$ perturbations
within an $\ell_{\infty}$ ball and construct a certifiable defense against
shadow attacks,

  * •

and compare our defense against other optics-based adversarial attacks.

Figure 1: An overview of our defense. We take source images, apply a shadow
with parameter $k$ randomly to the image, transform the image with shear,
rotation, and translation, and use adaptive thresholding (“AdaThresh”) to
generate a binary threshold map. The transformed image and the threshold map
are concatenated to form a 4-channel image, which the model is retrained on.
The rest of the model architecture (“backbone”) need not be changed for good
robustness.

##  2 Background and Related Work

Adversarial examples from a general perspective were first introduced by Dalvi
_et al_. [6], and refined by Szegedy _et al_. [38] with a constraint that the
adversarial examples must be no more than $\varepsilon$ away from the clean
example, thereby causing examples to be impercetible to humans for
sufficiently small $\varepsilon$.

###  2.1 So What Are Adversarial Examples Exactly?

In this work, we consider a classification setting with $C$ distinct classes,
in which we wish to classify an RGB image $x\in\mathbb{R}^{n}$ by passing $x$
through a model $M$ (such as a neural network) and computing a class label
$y=M(x)\in\\{1,\ldots,C\\}$.

This setting includes an adversary, who is able to take any such example $x$
and transform it into $x_{\text{adv}}$, a point in the convex set of possible
transformed examples $S_{\text{adv}}(x)\subseteq\mathbb{R}^{n}$. Intuitively,
$S_{\text{adv}}(x)$ is a collection of all the examples that, by some metric,
are “close” to $x$ [1]. For instance, if the adversary wishes to keep the
transformations within a $\varepsilon$ range $L_{p}$ perturbation from $x$,
then $S_{\text{adv}}(x)=\\{x^{\prime}:||x^{\prime}-x||_{p}<\varepsilon\\}$.

An adversarial example $x_{\text{adv}}$, then, is an example that is “close”
to $x$ yet causes the model $M$ to misclassify it:

$M(x)\neq M(x_{\text{adv}})\quad x_{\text{adv}}\in S_{\text{adv}}(x).$ (1)

Extending this notion, an adversarial attack is a principled manner for an
adversary to generate these adversarial examples, _e.g_., through some
mathematical formulation or algorithm.

###  2.2 Shadow Attack on Road Signs

Shadows as an adversarial attack were first proposed by Zhong _et al_. [41],
although concern over shadows in road sign recognition have existed for some
time [11, 21]. Their attack generates an adversarial image by

  1. 1.

choosing a parameter $k$ representing the “darkness” or “strength” of the
shadow, where higher values of $k$ indicate weaker shadows and vice versa,

  2. 2.

locating a polygon $\mathcal{P}_{\mathcal{V}}$, defined by a set of vertices
$\mathcal{V}=\\{(m_{1},n_{1}),\ldots,(m_{s},n_{s})\\}$, and a mask
$\mathcal{M}$ to locate the target polygon,

  3. 3.

converting $x$ from RGB color space to LAB image space, such that each element
in $x_{i,j}\in\mathbb{R}^{3}$ represents the L, A, and B channels
respectively,

  4. 4.

forming a new image $x_{\text{adv}}$ by recalculating the value of every pixel
$(i,j)$ with

$x_{\text{adv},i,j}=\begin{cases}x_{i,j}\cdot\scalebox{0.75}{\mbox{$\displaystyle\begin{bmatrix}k&1&1\end{bmatrix}^{\top}$}}&(i,j)\in\mathcal{P}_{\mathcal{V}}\cap\mathcal{M}\\\
x_{i,j}\cdot\scalebox{0.75}{\mbox{$\displaystyle\begin{bmatrix}1&1&1\end{bmatrix}^{\top}$}}&(i,j)\not\in\mathcal{P}_{\mathcal{V}}\cap\mathcal{M}\\\
\end{cases},$ (2)

  5. 5.

and converting $x_{\text{adv}}$ back into RGB space [41].

Finding $\mathcal{V}$ can be formulated as an optimization problem, which
Zhong _et al_. solve by means of Particle Swarm Optimization (PSO) [19].

###  2.3 Adaptive Thresholding

In general, thresholding is the process of generating a binary image $b$
(a.k.a. the threshold map) from a source image $s$, where the white pixels are
the “foreground” (255) elements and the black pixels are the “background” (0)
elements. The “threshold” is the means by which foreground and background are
separated. Adaptive thresholding [3, 40] is a form of local thresholding,
where each pixel $b_{i,j}$ is assigned to the foreground or the background by
a threshold function $T$ parameterized on pixel coordinates $i,j$:

$b_{i,j}=\begin{cases}255&s_{i,j}>T(i,j)\\\ 0&s_{i,j}\leq T(i,j).\end{cases}$
(3)

The benefit of adaptive thresholding over global thresholding is that $T$ is
parameterized on each pixel, and is thus robust to the spatial changes in the
illuminations that represent the variations present in shadow attacks, as
opposed to setting one uniform threshold for the entire image.

In our approach, we let $T$ be a Gaussian-window weighted sum of a $k\times k$
neighborhood around $(i,j)$ [33, 36]. If $N(i,j)$ contains the set of all
points within a $k\times k$ neighborhood around $(i,j)$, then

$T(i,j)=\sum_{(x,y)\in N(i,j)}G_{x,y}\cdot s_{x,y}$ (4)

where $G_{i,j}$ is the Gaussian-window (Fig. 2) weight for pixel $(i,j)$,
defined as

$G_{i,j}=\alpha\cdot\exp\left(\frac{-(i-\frac{k-1}{2})^{2}-(j-\frac{k-1}{2})^{2}}{2\sigma^{2}}\right)$
(5)

where the standard deviation $\sigma$ is calculated from all pixels in
$N(i,j)$ and $\alpha$ is a scaling factor such that the $G_{i,j}$ sum to 1.
$k$ is a tuneable hyperparameter representing the “aperture” of our local
threshold; we chose $k=3$ to due to the small size of our input images, but
the primary focus is on presenting the robustness of the method and not tuning
hyperparameters.

Figure 2: A visualization of $G_{i,j}$ around the point $(i,j)=(0,0)$ with
window size $k=3$. The motivation behind using Gaussian window weights is that
pixels closest to $(i,j)$ receive maximum weights, while pixels further away
receive less weight. Image licensed under Creative Commons Attribution-Share
Alike 3.0 [14].

###  2.4 Canny Edge Detection

Closely related to local thresholding is Canny edge detection, first proposed
in 1986 [4]. The Canny edge detection algorithm usually takes the following
steps [9]:

  1. 1.

smooth the image with a Gaussian-window filter (see Eq. 5) to reduce noise,

  2. 2.

determine the gradient magnitude and direction at each pixel,

  3. 3.

apply a custom thresholding function wherein edge (“foreground”) pixels have
gradient magnitudes larger than those of its two neighbors in the gradient
direction,

  4. 4.

and clean up extraneous or “weak” edges with a hysteresis thresholding step
with two threshold parameters, $t_{\text{hi}}$ and $t_{\text{lo}}$. Pixels
that are above $t_{\text{hi}}$ become edges, below $t_{\text{lo}}$ are
rejected as edges, and in between are edges only if they are connected to an
edge pixel.

The output of any edge algorithm [20], Canny included, is referred to as the
edge map of the image.

Canny edge detection is both local and global: the first thresholding step is
local with respect to the gradient direction, while the second thresholding
step is global, as the parameters are set for the whole image.

While both methods are tested in our approach, we hypothesize that for darker
shadows, the greater variance between gradient pixels inside the shadowed
region and gradient pixels outside the shadowed region would cause Canny edge
maps to have worse performance than adaptive threshold maps.

###  2.5 Related Work

While shadows as an adversarial attack were first conceived by Zhong _et al_.
[41], shadows have been a source of concern in traffic sign recognition for
some time [7, 12]. However, to our knowledge, the only defense against such
attacks remains that proposed by Zhong _et al_.: a simple adversarial training
scheme.

More generally, shadows as an adversarial attack falls into a class of non-
invasive optical attacks which are particularly pernicious against self
driving cars. While digital attacks like FGSM [18] and DeepFool [24] are
effective in the online domain, such attacks are infeasible in realistic self-
driving scenarios, leading to the rise of physical, non-invasive attacks that
cause natural or imperceptible perturbations to physical road signs [29, 22,
15, 34]. Zhong _et al_. noted these attacks relied on sophisticated and
complex equipment, making their implementation impractical—for this same
reason, it is impractical for us to test our defense on these attacks.

To our knowledge, there is no literature regarding the use of adaptive
threshold maps or threshold maps as adversarial defenses. However, edge
detection is closely related to thresholding [25]. Previous work has
investigated edges as an adversarial defense against different classification
tasks [8, 37], but their work both use edges as the sole source of
information, rather than augmenting it to existing images as in our method.
Doing so puts heavily reliance on the quality of edge detectors, which can
introduce their own vulnerabilities to adversarial examples if neural networks
are used [5].

##  3 Our Approach

In this section, we give motivations for our defense and empirical
justification for those motivations. We then describe our defense and training
regime. Our framework supports two defenses: one with adaptive threshold maps
and one with edge maps, and requires no more than one modification to the
network architecture to achieve good results against shadow attacks.

###  3.1 Our Motivation

We were initially motivated to investigate edge profiling as a possible
defense, since

  * •

humans recognize road signs based on the boundaries of the sign and the text
or symbols within, which can be represented with an edge map [2], and

  * •

convolutional neural networks, which are often used in state-of-the-art (SOTA)
road sign recognition, can learn “unimportant” (from a human perspective) and
non-robust features like texture [13], leaving them vulnerable to adversarial
attacks which disturb these features.

However, edge detection algorithms are imperfect and edge maps alone may be
insufficient for object recognition [28]. With this in mind, we decided to
proceed with appending edge maps of source images as a fourth channel as a way
of “emphasizing” the importance of edges, without destroying the information
provided by the source image.

This intuition is empirically justified by experiments with saliency maps [32,
31], a sample of which are presented in Fig. 3. These experiments indicate
that SOTA sign recognition models, such as those found in Eykholt _et al_.
[10], learn features inside the sign but not necessarily the text or
boundaries of the sign itself—likely because most signs share one of a few
simple edge boundaries, and thus edge maps are shared across many classes in
the training set, so the model fails to recognize its robustness. Although
sufficient in a benign setting, when an adversary perturbs the sign contents
sufficiently, the model fails to “fallback” and learn from the edges or other
low-level features, like the sign’s edge boundaries.

The motivation behind thresholding was based on our hypothesis in Sec. 2.4 and
empirical tests that threshold maps appeared more readable and less noisy than
edge maps (Fig. 4). However, for completeness, we tested both methods.

Figure 3: Saliency maps [31, 26] (right) for a randomly selected benign
traffic sign (top left) and its adversarially-shadowed counterparts (bottom
left) from the GTSRB [35] dataset on the model from [10]. The network appears
to learn features inside the sign boundary, leaving it vulnerable when the
inside of the sign is perturbed heavily. Figure 4: A comparison of a shadowed
image (left) with parameter $k=0.43$, its edge map (middle) and its threshold
map (right). The threshold map appears much less noisy, motivating the use of
threshold maps.

###  3.2 Our Defense

Our defense is a variant on adversarial training. Given a training dataset
$D_{\text{train}}$ and a model $M$, our defense quadruplicates the dataset
based on boolean flags adv and transform, which control the addition of
adversarial examples and transformed images (shear, rotate, and translate) to
the training regime. For each of these dataset duplicates, our defense

  1. 1.

modifies the first layer of $M$ to accomodate a 4-channel image,

  2. 2.

takes in a source image $s$,

  3. 3.

randomly applies a shadow if adv is True,

  4. 4.

generates the relevant profile $p$ (threshold or edge map),

  5. 5.

appends the profile $p$ to $s$ as a fourth channel, making a new image $x$,

  6. 6.

transforms $x$ if transform is True, and

  7. 7.

trains the model on $x$.

If augmenting source images with threshold maps, then we generate an threshold
map using Eq. 3. In the case of edge maps, our defense uses Canny edge
detection due to its non-differentiablity (compared to a neural network edge
mechanism) and speed. The parameters $t_{\text{high}}$ and $t_{\text{low}}$
are chosen based on the formula

$\displaystyle t_{\text{lo}}$ $\displaystyle=\max(0,\mu\cdot(1-\sigma))$ (6)
$\displaystyle t_{\text{high}}$ $\displaystyle=\min(255,\mu\cdot(1+\sigma))$

where $\mu$ is the median of the image across all channels and $\sigma$ is is
a parameter representing the “blurring” of the edges: higher sigma indicates
more extreme blurring. Intuitively, this formula takes $\sigma$ as a standard
deviation and sets the upper and lower thresholds to be one standard deviation
from the mean.

In our experimentation, we chose $\sigma=0.33$ to maintain the 2:1 ratio of
$t_{\text{high}}$ to $t_{\text{low}}$ recommended by Canny [4]. We also chose
$\mu$ as the median and not the mean, since shadows bias the image histogram
towards the darker side.

##  4 Experimental Results

In this section, we present our experiments, namely:

  * •

we present robustness results for models using edge maps and adaptive
threshold maps,

  * •

we present benign test accuracies for both models,

  * •

we reformulate the shadow attack with parameter $k$ as an instance of a
$\varepsilon$ perturbations within an $\ell_{p}$ ball,

  * •

using the above reformulation, we present robustness results for both models
against the Fast Gradient Sign Method (FGSM), Projected Gradient Descent
(PGD), Boundary Attack, and a number of other optics-based adversarial
attacks, and

  * •

we perform an ablation study on our defense.

###  4.1 Experimental Setup

For consistency, we used the same network architectures [10] and preprocessing
steps as Zhong _et al_. [41], with the exception that our networks were
modified to take in 4 channels and retrained accordingly.

We also used the same train/test split and the same code as in Zhong _et al_.
[41] for the shadow attack, and evaluated the robustness accordingly. This
includes their exclusion of images which are already too dark; i.e., the mean
of their pixel values in the L channel is no larger than 120.

Our experiments were conducted with the PyTorch deep learning library [27], on
a machine with eight Intel(R) Xeon(R) Bronze 3106 CPU @ 1.70GHz CPUs and one
NVIDIA Titan-XP GPU with CUDA version 10.2. The operating system was Ubuntu
18.04 (LTS) with Python version 3.10.5.

###  4.2 Core Robustness Results

We used Algorithm 1 as our testing regime, and tested the same $k$ values as
Zhong _et al_. [41]; namely, the set of values

$\displaystyle\mathcal{K}=\\{\,$ $\displaystyle
0.20,0.25,0.30,0.35,0.40,0.43,0.45,$ $\displaystyle
0.45,0.50,0.55,0.60,0.65,0.70\\}.$

Note that the value $k=0.43$ is the median shadow value from the SBU Shadow
Dataset [39]. There are two results we record: the robustness, defined as
$1-\text{attack success rate}$, and the number of queries that the shadow
attack makes to the backbone model, which is a measure of the attack’s
stealthiness.

Our results are presented in Tab. 1 and Tab. 2 for robustness and average
number of queries, respectively. In line with our hypotheses in Sec. 3, we
found that while both edge maps and adaptive thresholding provided similar
robustness, the quality of edge maps depended greatly on the strength of the
shadow, whereas adaptive threshold maps, due to their local nature, were more
invariant.

Algorithm 1 Our Testing Regime

for dataset in $\\{\text{GTSRB, LISA}\\}$ do

for $k$ in $\mathcal{K}$ do

$\alpha\leftarrow$ test accuracy on benign examples

$\beta_{1}\leftarrow$ test accuracy on shadowed examples

$\beta_{2}\leftarrow$ num. queries on shadowed examples

end for

end for

Defense/Dataset |  $k=$ 0.20 | 0.25 | 0.30 | 0.35 | 0.40 | 0.43 | 0.45 | 0.50 | 0.55 | 0.60 | 0.65 | 0.70
---|---|---|---|---|---|---|---|---|---|---|---|---
Zhong _et al_.: None | 2.63 | 3.65 | 4.86 | 6.55 | 8.64 | 9.53 | 11.03 | 12.85 | 15.75 | 19.64 | 26.24 | 33.27
Zhong _et al_.: AT | 15.89 | 17.78 | 20.39 | 23.12 | 26.69 | 28.38 | 30.03 | 30.26 | 33.78 | 43.01 | 48.57 | 55.30
Ours: AdaThresh | 73.82 | 74.69 | 75.12 | 75.58 | 76.41 | 76.63 | 76.29 | 76.71 | 77.61 | 79.65 | 79.26 | 78.57
Ours: Edges | 75.41 | 75.19 | 76.82 | 76.89 | 77.08 | 78.02 | 78.12 | 79.33 | 80.59 | 81.97 | 83.55 | 85.46
Table 1: The above table describes the average robustness ($1-$ success rate of attack) over $n=5$ trials for a specified defense (adversarial training from Zhong _et al_., our adaptive threshold maps, or our edge maps) and shadow attack with parameter $k$. While AdaThresh tends to have lower robustness, it has a tighter variance than that of edge map defenses. Defense/Dataset |  $k=$ 0.20 | 0.25 | 0.30 | 0.35 | 0.40 | 0.43 | 0.45 | 0.50 | 0.55 | 0.60 | 0.65 | 0.70
---|---|---|---|---|---|---|---|---|---|---|---|---
Zhong _et al_.: AT | 98 | 93 | 112 | 129 | 128 | 126 | 136 | 155 | 188 | 232 | 249 | 343
Ours: AdaThresh | 210 | 251 | 256 | 289 | 348 | 355 | 331 | 329 | 430 | 501 | 574 | 560
Ours: Edges | 304 | 277 | 370 | 376 | 396 | 456 | 450 | 515 | 580 | 645 | 726 | 819
Table 2: The above table describes the average number of queries to the
backbone model over $n=5$ trials for a specified defense (adversarial training
from Zhong _et al_., our adaptive threshold maps, or our edge maps) and shadow
attack with parameter $k$. This is a proxy measure of the “stealthiness” of
the black-box attack. The variance in edge map defenses remains significantly
higher than AdaThresh defenses.

###  4.3 Reformulation

######  Theorem 1.

Given an image in RGB color space $x=[R(x),G(x),B(x)]$, its LAB counterpart
$x_{\text{lab}}=[L(x),A(x),B(x)]$, and its adversarial counterparts
$x_{\text{adv}}$ and $x_{\text{lab, adv}}$, for a shadow attack with parameter
$k$, the adversarial perturbation $||x_{\text{adv}}-x||=\varepsilon_{k}$ is at
most $||M||\cdot{100|k-1|}$, where $M$ is the matrix multiplication to convert
from LAB to RGB space.

###### Proof.

See the Appendix. ∎

###  4.4 Robustness Against Other Attacks

To test our robustness against other attacks, we first obtained a value for
$\varepsilon$ when $k=0.43$ based on Theorem 1.

##  5 Discussion and Limitations

Beyond providing additional information in the form of a fourth channel, this
attack also combines gradient masking (as the binary edge or threshold map is
non-differentiable) and adversarial training (as it is trained on shadowed
images. However, even though it uses gradient masking, the defense remains
robust even to non-gradient based attacks like Boundary Attack and shadow
attacks.

There are a few limitations to this attack. First, its reliance on adversarial
retraining makes its training process slow. Second, it uses off-the-shelf edge
and adaptive thresholding methods; it is worth investigating a more bespoke
set of thresholding equations in future research. Third, its robustness is
significantly higher than the baseline, but insufficient for critical tasks
like sign recognition; our work should be seen as a starting point and not a
final defense.

##  6 Conclusion

In this paper, we presented a novel and simple adversarial defense against
shadow-based adversarial attacks. Our defense requires no retuning or redesign
of the model architecture to achieve good robustness against shadow attacks.
Furthermore, our defense remains robust against classic adversarial attacks
including other optics-based adversarial attacks.

##  7 Acknowledgments and Funding

We would like to thank Dr. Gregory Ditzler, David Schwartz, and Huayu Li for
their assistance in developing this paper. This paper was funded by National
Science Foundation (NSF) Grant IIS-195039.

## References

  * [1] Mislav Balunović and Martin Vechev.  Adversarial training and provable defenses: Bridging the gap.  In 8th International Conference on Learning Representations (ICLR 2020)(virtual). International Conference on Learning Representations, 2020\.
  * [2] Aayush Bansal, Adarsh Kowdle, Devi Parikh, Andrew Gallagher, and Larry Zitnick.  Which edges matter?  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 578–585, 2013.
  * [3] Derek Bradley and Gerhard Roth.  Adaptive thresholding using the integral image.  Journal of graphics tools, 12(2):13–21, 2007.
  * [4] John Canny.  A computational approach to edge detection.  IEEE Transactions on pattern analysis and machine intelligence, (6):679–698, 1986.
  * [5] Christian Cosgrove and Alan Yuille.  Adversarial examples for edge detection: They exist, and they transfer.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1070–1079, 2020.
  * [6] Nilesh Dalvi, Pedro Domingos, Sumit Sanghai, and Deepak Verma.  Adversarial classification.  In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 99–108, 2004.
  * [7] Arturo De la Escalera, J Ma Armingol, and Mario Mata.  Traffic sign recognition and analysis for intelligent vehicles.  Image and vision computing, 21(3):247–258, 2003.
  * [8] Gavin Weiguang Ding, Kry Yik Chau Lui, Xiaomeng Jin, Luyu Wang, and Ruitong Huang.  On the sensitivity of adversarial robustness to input data distributions.  In ICLR (Poster), 2019.
  * [9] Lijun Ding and Ardeshir Goshtasby.  On the canny edge detector.  Pattern recognition, 34(3):721–725, 2001.
  * [10] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song.  Robust physical-world attacks on deep learning visual classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1625–1634, 2018.
  * [11] Hasan Fleyeh.  Shadow and highlight invariant colour segmentation algorithm for traffic signs.  In 2006 IEEE Conference on Cybernetics and Intelligent Systems, pages 1–7. IEEE, 2006.
  * [12] Hasan Fleyeh and Erfan Davami.  Eigen-based traffic sign recognition.  IET Intelligent Transport Systems, 5(3):190–196, 2011.
  * [13] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  arXiv preprint arXiv:1811.12231, 2018.
  * [14] Kaushik Ghose.  Isometric plot of a two dimensional gaussian, 2006.
  * [15] Abhiram Gnanasambandam, Alex M Sherman, and Stanley H Chan.  Optical adversarial attack.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 92–101, 2021.
  * [16] Micah Goldblum, Avi Schwarzschild, Ankit Patel, and Tom Goldstein.  Adversarial attacks on machine learning systems for high-frequency trading.  In Proceedings of the Second ACM International Conference on AI in Finance. ACM, nov 2021.
  * [17] Hokuto Hirano, Akinori Minagi, and Kazuhiro Takemoto.  Universal adversarial attacks on deep neural networks for medical image classification.  BMC medical imaging, 21(1):1–13, 2021.
  * [18] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel.  Adversarial attacks on neural network policies.  arXiv preprint arXiv:1702.02284, 2017.
  * [19] James Kennedy and Russell Eberhart.  Particle swarm optimization.  In Proceedings of ICNN’95-international conference on neural networks, volume 4, pages 1942–1948. IEEE, 1995.
  * [20] Mukesh Kumar, Rohini Saxena, et al.  Algorithm and technique on various edge detection: A survey.  Signal & Image Processing, 4(3):65, 2013.
  * [21] Haojie Li, Fuming Sun, Lijuan Liu, and Ling Wang.  A novel traffic sign detection method via color segmentation and robust shape matching.  Neurocomputing, 169:77–88, 2015.
  * [22] Juncheng Li, Frank Schmidt, and Zico Kolter.  Adversarial camera stickers: A physical camera-based attack on deep learning systems.  In International Conference on Machine Learning, pages 3896–3904. PMLR, 2019.
  * [23] KT Yasas Mahima, Mohamed Ayoob, and Guhanathan Poravi.  Adversarial attacks and defense technologies on autonomous vehicles: A review.  Appl. Comput. Syst., 26(2):96–106, 2021.
  * [24] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard.  Deepfool: a simple and accurate method to fool deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2574–2582, 2016.
  * [25] Ehsan Nadernejad, Sara Sharifzadeh, and Hamid Hassanpour.  Edge detection techniques: Evaluations and comparisons.  Applied Mathematical Sciences, 2(31):1507–1520, 2008.
  * [26] Utku Ozbulak.  Pytorch cnn visualizations.  https://github.com/utkuozbulak/pytorch-cnn-visualizations, 2019\.
  * [27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [28] Thomas Sanocki, Kevin W Bowyer, Michael D Heath, and Sudeep Sarkar.  Are edges sufficient for object recognition?  Journal of Experimental Psychology: Human Perception and Performance, 24(1):340, 1998.
  * [29] Athena Sayles, Ashish Hooda, Mohit Gupta, Rahul Chatterjee, and Earlence Fernandes.  Invisible perturbations: Physical adversarial examples exploiting the rolling shutter effect.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14666–14675, 2021.
  * [30] Ali Shafahi, W. Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein.  Are adversarial examples inevitable?  2018\.
  * [31] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.  Learning important features through propagating activation differences.  In International conference on machine learning, pages 3145–3153. PMLR, 2017.
  * [32] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034, 2013.
  * [33] Julius O. Smith.  Spectral Audio Signal Processing.  http:http://ccrma.stanford.edu/ jos/sasp///ccrma.stanford.edu/~jos/sasp/, accessed ¡date¿.  online book, 2011 edition.
  * [34] Dawn Song, Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Florian Tramer, Atul Prakash, and Tadayoshi Kohno.  Physical adversarial examples for object detectors.  In 12th USENIX workshop on offensive technologies (WOOT 18), 2018\.
  * [35] Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel.  Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition.  Neural networks, 32:323–332, 2012.
  * [36] Mallat Stephane.  A wavelet tour of signal processing, 1999.
  * [37] Mingjie Sun, Zichao Li, Chaowei Xiao, Haonan Qiu, Bhavya Kailkhura, Mingyan Liu, and Bo Li.  Can shape structure features improve model robustness under diverse adversarial settings?  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7526–7535, 2021.
  * [38] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  arXiv preprint arXiv:1312.6199, 2013.
  * [39] Tomás F Yago Vicente, Le Hou, Chen-Ping Yu, Minh Hoai, and Dimitris Samaras.  Large-scale training of shadow detectors with noisily-annotated shadow examples.  In European Conference on Computer Vision, pages 816–832. Springer, 2016.
  * [40] James M White and Gene D Rohrer.  Image thresholding for optical character recognition and other applications requiring character image extraction.  IBM Journal of research and development, 27(4):400–411, 1983.
  * [41] Yiqi Zhong, Xianming Liu, Deming Zhai, Junjun Jiang, and Xiangyang Ji.  Shadows can be dangerous: Stealthy and effective physical-world adversarial attack by natural phenomenon.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15345–15354, 2022.


10(3,0.4)

CONSEQUENCES Workshop on Causality, Counterfactuals, and Sequential Decision-
Making

in conjunction with ACM RecSys 2023

# Fairness of Exposure in Dynamic Recommendation

Masoud Mansoury  University of Amsterdam,Elsevier Discovery
LabAmsterdamNetherlands<EMAIL_ADDRESS>and  Bamshad Mobasher  DePaul
UniversityChicagoUSA<EMAIL_ADDRESS>
(2023)

###### Abstract.

Exposure bias is a well-known issue in recommender systems where the exposure
is not fairly distributed among items in the recommendation results. This is
especially problematic when bias is amplified over time as a few items (e.g.,
popular ones) are repeatedly over-represented in recommendation lists and
users’ interactions with those items will amplify bias towards those items
over time resulting in a feedback loop. This issue has been extensively
studied in the literature in static recommendation environment where a single
round of recommendation result is processed to improve the exposure fairness.
However, less work has been done on addressing exposure bias in a dynamic
recommendation setting where the system is operating over time, the
recommendation model and the input data are dynamically updated with ongoing
user feedback on recommended items at each round. In this paper, we study
exposure bias in a dynamic recommendation setting. Our goal is to show that
existing bias mitigation methods that are designed to operate in a static
recommendation setting are unable to satisfy fairness of exposure for items in
long run. In particular, we empirically study one of these methods,
Discrepancy Minimization, and show that repeatedly applying this method fails
to fairly distribute exposure among items in long run. To address this
limitation, we show how this method can be adapted to effectively operate in a
dynamic recommendation setting and achieve exposure fairness for items in long
run. Experiments on a real-world dataset confirm that our solution is superior
in achieving long-term exposure fairness for the items while maintaining the
recommendation accuracy.

recommender system, exposure bias, long-term fairness, dynamic recommendation

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: ACM Conference on Recommender Systems; September
18–22, 2023; Singapore††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††copyright: none

##  1\. Introduction

Recommender systems are known to suffer from exposure bias; few items are
over-exposed in the recommendation lists, while the majority of other items
are under-exposed (Mansoury, 2021b; Abdollahpouri and Mansoury, 2020; Wu et
al., 2022; Khenissi and Nasraoui, 2020). A negative consequence of this bias
is that it may lead to unfairness on supplier-side: items belonging to
different suppliers are unfairly represented in the recommendations results.
Also, due to the feedback loop phenomena in which users profiles get updated
over time via potentially biased recommendations, exposure bias can be
amplified over time (Mansoury et al., 2020; Sinha et al., 2016). Highly
exposed items in the recommendation lists have higher chance to be
viewed/examined/clicked by the users, while insufficiently exposed items will
not receive adequate attention from the users. As a result, those over-exposed
items would have a higher chance to be shown in the recommendation lists in
the future which amplifies existing bias. Amplification of exposure for few
items would be at the expense of the under-exposure for a majority of other
items (the ones that might be interesting for some users).

Most existing research to study exposure bias are conducted in static
recommendation setting where a single round of recommendation results is
analyzed (Patro et al., 2020; Mansoury et al., 2021b; Mansoury, 2021a, b; Sühr
et al., 2019). While these studies unveiled important aspects of exposure bias
and proposed solutions for tackling it, the long-term impact of this bias is
yet a significant research gap which we seek to investigate in this paper.
Filling this gap requires studying the task of recommendation problem in
dynamic setting where the recommendation process is repeatedly performed over
time (Mansoury et al., 2022, 2021a; Li et al., 2010, 2020; Hiranandani et al.,
2020; Mansoury et al., 2020). At each round $t$, the recommendation lists are
generated for all users, clicks on recommended items are collected, and users’
profile are updated. Updated users’ profile are used for recommendation
process in round $t+1$.

In this paper, we empirically study exposure bias in dynamic recommendation
setting. First, we show that existing bias mitigation algorithms designed to
operate in a static recommendation setting fail to satisfy exposure fairness
in long run, when used to generate recommendations for the users over time.
This is done by simulating the recommendation process with bias mitigation
algorithm in (Antikacioglu and Ravi, 2017). The experimental results suggest
that a different treatment is needed for improving the exposure fairness in
long run. Therefore, we modify the algorithm to adapt it for improving
exposure fairness in dynamic recommendation environment. To do so, instead of
solely distributing the available exposure in each round among items, we
modify the algorithm to distribute the available exposure in each round $t$
proportional to the cumulative exposure given to the items up to round $t$.
This will ensure to avoid over-exposure or under-exposure in long run.
Experiments on a publicly-available dataset confirm that the modified
algorithm significantly outperforms the original algorithm in terms of
exposure fairness in long run, while maintaining the accuracy of the
recommendations.

Figure 1. Comparison between bias mitigation algorithms designed for static
recommendation setting (Alg1) and dynamic recommendation setting (Alg2).

##  2\. Motivating example

Figure 1 illustrates a situation where tackling exposure bias in static
setting, while achieves fairness objectives in each single round of the
recommendation results, fails to ensure fairness in long run. Given items in
candidate pool, the goal is to extract a list of items (in this example, size
of 4) as recommendation to the user that satisfies exposure fairness among the
items. Hence, items are categorized into two groups in which one group can be
interpreted as unprotected items that are often over-exposed in the
recommendation results (e.g., popular items or items that represent the
majority), and another group that can be interpreted as protected items that
are often under-exposed in the recommendation results (e.g., unpopular items
or items that represent the minority). The recommendation process is repeated
for 5 rounds. We assume that the results are obtained from two different bias
mitigation algorithms: Alg1 attempting to improve the exposure fairness in
static setting and Alg2 attempting to improve exposure fairness in dynamic
setting.

Figure 1a shows the results obtained by Alg1. As shown, in each round, items
belonging to each group are almost fairly represented: 50% of the
recommendation slots are assigned to Group 1 and 50% to Group 2. In terms of
the position in the recommendation list, items from both groups are almost
fairly represented with Group 1 being exposed slightly more in round 1 and
Group 2 being exposed slightly more in round 4. This is the way that existing
bias mitigation approaches attempt to improve the exposure fairness in
recommendation results. While they properly achieve exposure fairness in each
single round of recommendations, their performance in long run is still
unclear. Figure 1b shows the total exposure assigned to each item in long run.
As shown, most of the exposure are accumulated on few items
($i_{1},i_{2},i_{5},i_{6}$). This is mainly due to the fact that the algorithm
is not aware of the history of the exposure for each item in the past rounds
and only attempts to distribute the exposure of a single round of the
recommendation among items.

In contrast to Alg1, Alg2 operates in a dynamic recommendation setting. Figure
1c shows the results obtained by Alg2. Analogous to Alg1 in Figure 1a, it
yields the same exposure fairness for each group of items in each round.
However, unlike Alg1, it provides almost equal exposure to each item in long
run. Figure 1d shows the total exposure given to each item by Alg2. It is
evident that in long run items are more equally represented in the
recommendation results.

##  3\. Formalizing Item Exposure in Dynamic Recommendation Setting

Given $\mathcal{U}=\\{u_{1},...,u_{n}\\}$ as the set of $n$ users,
$\mathcal{I}=\\{i_{1},...,i_{m}\\}$ as the set of $m$ items, the
recommendation task at each round $t=\\{1,...,T\\}$ is retrieving a ranked
list of $K$ most relevant items for each user. We denote the recommendation
lists generated at round $t$ as $R^{t}$, with $R^{t}_{u}$ being the list
generated for user $u$ at round $t$.

Exposure of an item in the recommendation results depends on its position in
the list (Singh and Joachims, 2018). It is possible an item gets recommended,
but receives no exposure because it was probably shown at the bottom of the
list and the user has not examined it. For this reason, items appearing on top
of the list receive higher exposure than items appearing at the bottom of the
list. Following the idea in (Singh and Joachims, 2018), we consider a standard
exposure drop-off to assign weight to each position in the recommendation
lists as commonly used in ranking metrics (e.g., nDCG) and calculated as
$1/\log(1+k)$ where $k$ is the position of the item in the list. Therefore, we
compute the exposure of an item $i$ in round $t$ as:

(1)
$E^{t}(i)=\sum_{u\in\mathcal{U}}\sum_{k=1}^{K}\mathds{1}(i=R^{t}_{u}(k)).\frac{1}{\log_{2}(1+k)}$

where $R^{t}_{u}(k)$ is the $k$-th item in $R^{t}_{u}$ and $\mathds{1}(.)$ is
the indicator function returning zero when its argument is False and 1
otherwise. We define the cumulative exposure of an item in $T$ rounds of the
recommendation process as:

(2) $E^{\leq T}(i)=\sum_{t=1}^{T}{E^{t}(i)}$

##  4\. Mitigating Exposure Bias in Dynamic Recommendation Setting

In this section, we review an existing bias mitigation method, Discrepancy
Minimization (DM), proposed in (Antikacioglu and Ravi, 2017) and then show how
it can be adapted to achieve exposure fairness in long run. DM is a post-
processing approach that operates on a bipartite graph constructed over the
long recommendation lists. This bipartite recommendation graph with users and
items as its nodes and edges between them representing the recommendation is
further processed to construct the flow network. Then, the minimum-cost
network flow methods (Ahuja et al., 1993) are applied on the constructed flow
network to find the recommendation subgraphs that yields exposure fairness for
items. More specifically, an integer-valued constraint is defined as the
target exposure for each item (i.e., the exposure that we desire for each
item), and the minimum-cost flow network is optimized to find an optimal
subgraph that gives the minimum discrepancy from this constraint.

For simplicity, we assume Equality of Exposure among items as the fairness
notion in which the goal is to equally distribute exposure among items.
Therefore, the constraint is defined as:

(3) $\mathcal{C}^{t}(i)=\left\lfloor\frac{n\times
K}{\sum_{i\in\mathcal{I}}{\text{1}(i\in\mathcal{L})}}\right\rfloor$

where $\mathcal{L}$ is the long recommendation lists generated by a base
recommender for all users. In this equation, the nominator is the total number
of available recommendation slots and the denominator is the total number of
items in the long recommendation lists. Hence, Eq. 3 equally assigns the
recommendation slots to the items. Depending on the relevancy of the items,
this constraints may not be perfectly satisfied, but DM attempts to minimize
the discrepancy with this constraint while maintaining the recommendation
accuracy.

The constraint in Eq. 3 is designed for static recommendation setting which
separately distributes the available recommendation slots in each single round
of recommendation process. Although it shows promising results in mitigating
exposure bias, we show that it can be even further improved by adapting it to
operate in dynamic recommendation setting. Therefore, we define the constraint
for dynamic recommendation setting as:

(4) $\mathcal{C}^{\leq t}(i)=\left\lfloor\frac{\mathcal{C}^{t}(i)}{E^{\leq
t}(i)}\right\rfloor$

This way, the exposure given to each item at round $t$ is proportional to the
total exposure given to that item up to round $t$. This means that if an item
is in total over-exposed up to round $t$, Eq. 4 assigns less exposure to it in
round $t$, and vice versa. For the rest of the paper, we denote $DM^{static}$
and $DM^{dynamic}$ for $DM$ with constraint in Eq. 3 and 4, respectively.

##  5\. Experiments

We follow the simulation process in (Mansoury et al., 2020) for our
experiment. In each round $t$, the users’ profile in input data is randomly
divided into 80% as training set and 20% as test set. The training set is used
to generate the recommendation lists (first long recommendation lists by a
base recommender and then final recommendations by a reranker) and the test
set is used to evaluate them. Acceptance probability is used to collect users’
clicks on recommended items as $prob(i|R_{u}^{t})=e^{\alpha\times rank_{i}}$
where $\alpha$ is a negative value ($\alpha<0$) for controlling the
probability assigned to each recommended item and $rank_{i}$ is the position
of item $i$ in $R_{u}^{t}$. We set $\alpha=-0.5$ for our experiments.
Collected clicks are added to the users’ profile and the updated input data is
used for round $t+1$. This process repeats for $T$ rounds.

We perform our experiments on MovieLens 1M dataset (Harper and Konstan, 2015).
In each round, we use Matrix Factorization (MF) (Koren et al., 2009) for
generating the long recommendation lists of size 40 for each user. Then, we
apply the bias mitigation methods described in section 4 to generate the final
recommendation lists of size 10 for each user. The hyperparameters involved in
both MF and DM are tuned for the highest possible accuracy. We use Normalized
Discounted Cumulative Gain (nDCG) as the accuracy metric to measure the
ranking quality of the recommendation. We perform the simulation using librec-
auto recommendation tool (Mansoury and Burke, 2019) for $T=600$ rounds.

Figure 2. Ranking quality of the recommendation results.

###  5.1. Experimental results

In this section, we present our experimental results. We compare the
performance of three models: 1) MF recommendation model ($\mathbf{MF}$), 2) MF
as a base recommender to generate the long recommendation lists and
$DM^{static}$ as abias mitigation method to generate the final recommendation
lists ($\mathbf{MF+DM^{static}}$), 3) similarly, MF as a base recommender and
$DM^{dynamic}$ as a bias mitigation method ($\mathbf{MF+DM^{dynamic}}$).
Figure 2 compares the models in terms of nDCG in each round. As shown, MF
yielded slightly higher nDCG than both DM variations, indicative of
insignificant loss in ranking quality when applying these bias mitigation
methods.

Figure 3 shows the results in terms of aggregate diversity (Adomavicius and
Kwon, 2011): the percentage of items which appear at least once in the
recommendation lists. Figure 3(a) shows the aggregate diversity of the
recommendations per round. The results suggest that both $DM$ variations
significantly outperformed MF and also our $DM^{dynamic}$ slightly achieved
fairer exposure outputs compared to $DM^{static}$. Our main results are shown
in Figure 3(b) which presents the cumulative aggregate diversity of the
recommendations in long run: it shows the percentage of the items appeared at
least once in $t$ rounds of the recommendation process. As shown, our
$DM^{dynamic}$ outperformed both MF and $DM^{static}$ in covering more items
in the recommendation lists in long run.

Figure 4 shows the results in terms of Equality of Exposure (Mansoury, 2021b):
how equal items are represented in recommendation results. Given the
distribution of items exposure in round $t$ computed using Eq. 1, we use Gini
Index (Vargas and Castells, 2014) to measure the uniformity of this
distribution. Uniform distribution will have Gini Index equal to zero which is
the ideal case. Therefore, for consistency, we report $1-GiniIndex$ as
equality of exposure. Figure 4(a) shows the equality of exposure for each
recommendation results per round. As shown, our $DM^{dynamic}$ slightly
resulted in fairer exposure for items in each round. In terms of long-term
performance, Figure 4(b) presents our main results, the equality of exposure
cumulatively in long run. Here, the Gini Index is computed over the
distribution of items exposure derived by Eq. 2. Clearly, our $DM^{dynamic}$
is superior in improving exposure fairness in long run compared to other
methods as it yielded higher EE compared to other recommendation methods.

(a) Aggregate diversity per round

(b) Cumulative Aggregate diversity

Figure 3. Experimental results on MovieLens dataset on the performance of
recommendations in terms of aggregate diversity using pure MF algorithm and MF
as base recommender and $DM^{static}$ and $DM^{dynamic}$ as post-processing
methods.

(a) Equality of Exposure

(b) Cumulative EE

Figure 4. Experimental results on MovieLens dataset on the performance of
recommendations in terms of equality of exposure using pure MF algorithm and
MF as base recommender and $DM^{static}$ and $DM^{dynamic}$ as post-processing
methods.

##  6\. Conclusion

In this paper, we studied exposure bias in dynamic recommendation setting
through an offline simulation. We hypothesized that bias mitigation methods
designed to operate in a static recommendation setting are unable to
sufficiently address exposure bias in long run. We based our analysis on one
of these bias mitigation methods, Discrepancy Minimization, and empirically
showed the validity of our claim on this algorithm. To tackle this limitation,
we proposed solution for adapting this algorithm to operate in dynamic
recommendation environment. Experiments on a movie recommendation dataset
showed the superiority of our solution in improving exposure fairness in long
run.

## References

  * (1)
  * Abdollahpouri and Mansoury (2020) Himan Abdollahpouri and Masoud Mansoury. 2020.  Multi-sided exposure bias in recommendation.  _ACM KDD Workshop on Industrial Recommendation Systems 2020_ (2020).
  * Adomavicius and Kwon (2011) Gediminas Adomavicius and YoungOk Kwon. 2011.  Improving aggregate recommendation diversity using ranking-based techniques.  _IEEE Transactions on Knowledge and Data Engineering_ 24, 5 (2011), 896–911.
  * Ahuja et al. (1993) Ravindra K Ahuja, Thomas L Magnanti, and James B Orlin. 1993\.  Network flows: theory, algorithms, and applications.  _Prentice Hall_ (1993).
  * Antikacioglu and Ravi (2017) Arda Antikacioglu and R Ravi. 2017.  Post processing recommender systems for diversity. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. 707–716.
  * Harper and Konstan (2015) F Maxwell Harper and Joseph A Konstan. 2015.  The movielens datasets: History and context.  _Acm transactions on interactive intelligent systems (tiis)_ 5, 4 (2015), 1–19.
  * Hiranandani et al. (2020) Gaurush Hiranandani, Harvineet Singh, Prakhar Gupta, Iftikhar Ahamath Burhanuddin, Zheng Wen, and Branislav Kveton. 2020.  Cascading linear submodular bandits: Accounting for position bias and diversity in online learning to rank. In _Uncertainty in Artificial Intelligence_. PMLR, 722–732.
  * Khenissi and Nasraoui (2020) Sami Khenissi and Olfa Nasraoui. 2020.  Modeling and counteracting exposure bias in recommender systems.  _arXiv preprint arXiv:2001.04832_ (2020).
  * Koren et al. (2009) Yehuda Koren, Robert Bell, and Chris Volinsky. 2009\.  Matrix factorization techniques for recommender systems.  _Computer_ 42, 8 (2009), 30–37.
  * Li et al. (2020) Chang Li, Haoyun Feng, and Maarten de Rijke. 2020.  Cascading hybrid bandits: Online learning to rank for relevance and diversity. In _Proceedings of the 14th ACM Conference on Recommender Systems_. 33–42.
  * Li et al. (2010) Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010\.  A contextual-bandit approach to personalized news article recommendation. In _Proceedings of the 19th international conference on World wide web_. 661–670.
  * Mansoury (2021a) Masoud Mansoury. 2021a.  Fairness-aware recommendation in multi-sided platforms. In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_. 1117–1118.
  * Mansoury (2021b) Masoud Mansoury. 2021b.  Understanding and mitigating multi-sided exposure bias in recommender systems.  _PhD Dissertation_ Eindhoven University of Technology (2021).
  * Mansoury et al. (2021a) Masoud Mansoury, Himan Abdollahpouri, Bamshad Mobasher, Mykola Pechenizkiy, Robin Burke, and Milad Sabouri. 2021a.  Unbiased cascade bandits: Mitigating exposure bias in online learning to rank recommendation.  _arXiv preprint arXiv:2108.03440_ (2021).
  * Mansoury et al. (2020) Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. 2020\.  Feedback loop and bias amplification in recommender systems. In _Proceedings of the 29th ACM international conference on information & knowledge management_. 2145–2148.
  * Mansoury et al. (2021b) Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. 2021b.  A graph-based approach for mitigating multi-sided exposure bias in recommender systems.  _ACM Transactions on Information Systems (TOIS)_ 40, 2 (2021), 1–31.
  * Mansoury and Burke (2019) Masoud Mansoury and Robin Burke. 2019.  Algorithm Selection with Librec-auto.. In _AMIR@ ECIR_. 11–17.
  * Mansoury et al. (2022) Masoud Mansoury, Bamshad Mobasher, and Herke van Hoof. 2022\.  Exposure-Aware Recommendation using Contextual Bandits.  _5th FAccTRec Workshop on Responsible Recommendation in conjunction with ACM RecSys_ (2022).
  * Patro et al. (2020) Gourab K Patro, Arpita Biswas, Niloy Ganguly, Krishna P Gummadi, and Abhijnan Chakraborty. 2020\.  Fairrec: Two-sided fairness for personalized recommendations in two-sided platforms. In _Proceedings of The Web Conference 2020_. 1194–1204.
  * Singh and Joachims (2018) Ashudeep Singh and Thorsten Joachims. 2018.  Fairness of exposure in rankings. In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_. 2219–2228.
  * Sinha et al. (2016) Ayan Sinha, David F Gleich, and Karthik Ramani. 2016\.  Deconvolving feedback loops in recommender systems.  _Advances in neural information processing systems_ 29 (2016).
  * Sühr et al. (2019) Tom Sühr, Asia J Biega, Meike Zehlike, Krishna P Gummadi, and Abhijnan Chakraborty. 2019\.  Two-sided fairness for repeated matchings in two-sided markets: A case study of a ride-hailing platform. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 3082–3092.
  * Vargas and Castells (2014) Saúl Vargas and Pablo Castells. 2014.  Improving sales diversity by recommending users to items. In _Proceedings of the 8th ACM Conference on Recommender systems_. 145–152.
  * Wu et al. (2022) Haolun Wu, Bhaskar Mitra, Chen Ma, Fernando Diaz, and Xue Liu. 2022.  Joint multisided exposure fairness for recommendation. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 703–714.