# Human-in-the-Loop Mixup

Katherine M. Collins Correspondence to<EMAIL_ADDRESS>University of
Cambridge  Umang Bhatt  University of Cambridge  The Alan Turing Institute
Weiyang Liu  University of Cambridge  Max Planck Institute for Intelligent
Systems  Vihari Piratla  University of Cambridge  Ilia Sucholutsky  Princeton
University  Bradley Love  The Alan Turing Institute  University College London
Adrian Weller  University of Cambridge  The Alan Turing Institute

###### Abstract

Aligning model representations to humans has been found to improve robustness
and generalization. However, such methods often focus on standard
observational data. Synthetic data is proliferating and powering many advances
in machine learning; yet, it is not always clear whether synthetic labels are
perceptually aligned to humans – rendering it likely model representations are
not human aligned. We focus on the synthetic data used in mixup: a powerful
regularizer shown to improve model robustness, generalization, and
calibration. We design a comprehensive series of elicitation interfaces, which
we release as HILL MixE Suite, and recruit 159 participants to provide
perceptual judgments along with their uncertainties, over mixup examples. We
find that human perceptions do not consistently align with the labels
traditionally used for synthetic points, and begin to demonstrate the
applicability of these findings to potentially increase the reliability of
downstream models, particularly when incorporating human uncertainty. We
release all elicited judgments in a new data hub we call H-Mix.

##  1 Introduction

Synthetic data is proliferating, fueled by increasingly powerful generative
models, e.g. [Goodfellow et al., 2014a, Dhariwal and Nichol, 2021]. These data
are not only consumed directly by people – but, as training predictive models
on synthetic data has been found to unlock tremendous advances in machine
learning (ML) [Silver et al., 2016, de Melo et al., 2022, Emam et al., 2021,
Jordon et al., 2022], synthetic data is increasingly employed to train
algorithms serving as engines of many applications humans may interact with.
However, it is not always clear whether human perceptual judgments of
synthetically-generated data match the generative process used to create them.

Figure 1: Framework overview. A) Synthetic data generating process used in
mixup; B) and C) depict elicitation settings: B) participants endorse a
synthetic image to match a label, C) participants infer the label for a
synthetic image and provide their uncertainty in the corresponding inference.

Aligning networks to match humans’ perceptual inferences could be a way to
further ensure model reliability, trustworthiness, downstream performance, and
robustness [Nanda et al., 2021, Chen et al., 2022, Fel et al., 2022,
Sucholutsky and Griffiths, 2023]. If these data are not aligned with human
percepts, then performance potentially could be improved by altering such
signals to better match the richness of human judgments: this has proven
effective when aligning models with human probabilistic knowledge and
perceptual uncertainty [Collins et al., 2022a, Sanders et al., 2022,
Sucholutsky et al., 2023]. We argue that one ought to verify whether synthetic
data aligns with human perception, and if not, explore whether training with
human-relabeled examples improves model performance.

In this work, we take a step in this direction by focusing on mixup [Zhang et
al., 2018]: a method whereby a model is trained only on synthetic, linear
combinations of conventional training examples. We focus on mixup for three
key reasons. First, the generative process for synthetic mixup examples is
very simple, and provides us with direct access to the “ground truth”
generative model parameters; that is, we have precise control over the mixing
coefficient used to create the mixed image. This enables us to compare any
discrepancy between human perceptual judgments and this parameter explicitly.
A generative model of the likes of a generative adversarial network (GAN)
[Goodfellow et al., 2014a] or diffusion model [Ho et al., 2020] does not as
easily permit these kinds of precise comparisons. Second, despite this
simplicity, mixup is a powerful and popular training-time method that has been
leveraged to address model fairness [Chuang and Mroueh, 2020], improve model
calibration [Thulasidasan et al., 2019, Zhang et al., 2022], and increase
model robustness via regularizing the form of category boundaries learned
implicitly [Zhang et al., 2020, Verma et al., 2022]. mixup is frequently used
as a strong benchmark for many new data augmentation and regularization
techniques [Hendrycks et al., 2019, 2022]. Third, prior work in human
categorical perception – revealing that humans show non-linear “warping”
effects along category boundaries [Harnad, 2003, Folstein et al., 2013,
Goldstone and Hendrickson, 2010] – suggests that humans will differ in their
percepts from the linear category boundaries encouraged by mixup.

To that end, we consider whether mixup labels match human perception, and if
not, how the labeling scheme can be improved to better align with human
intuition – and human uncertainty – to potentially enhance model performance.
We focus on two flavors of elicitation: 1) having participants “construct” a
midpoint between categories by selecting from a set of synthetic images, and
2) eliciting traces of humans’ broader category boundary across a range of
mixed images by having participants directly intervene on the synthetic label,
along with their uncertainty in their judgments. We design three online
elicitation interfaces to address these questions, which we offer as The
Human-in-the-Loop Mixup Elicitation Suite (HILL MixE Suite). We collect
judgments from over 150 humans on these synthetically combined images, which
we release in a dataset we call “Human Mixup” or H-Mix111All data, elicitation
interfaces, and experiment code will be included in our repository.. We then
demonstrate one of the possible use cases of this data: as adjusted training
data for deep networks to improve model generalization, calibration, and
adversarial robustness. We depict our general framework in Fig. 1. Our data
(H-Mix) and general elicitation paradigm (e.g., HILL MixE Suite) could support
a range of downstream applications: from serving as new training labels for
machine learning or benchmarking model alignment to auditing synthetic data,
and informing cognitive science studies, among others. We see our work as a
step in the exciting direction of a human-centric perspective on synthetic
data powering many ML algorithms, which emphasizes the potential utility of
human uncertainty in human-in-the-loop systems.

##  2 Problem Formulation

###  2.1 Decoupling Data and Label Mixing in mixup

We first review mixup [Zhang et al., 2018] and explicate the recipe by which
synthetic examples are created. We employ the nomenclature and notation around
“mixup policies” from [Liu et al., 2021b]. We assume access to a finite set of
$N$ samples $\\{(x_{1},y_{1}),(x_{2},y_{2},\cdots,(x_{N},y_{N})\\}$. mixup
training consists of constructing synthetic training examples
$(\tilde{x},\tilde{y})$ via linear combinations of pairs of the training
observations $(x_{i},y_{i}),(x_{j},y_{j})$ for $i,j\in[1,N]$, corresponding to
the following data and label mixing functions:

$\textnormal{Data Mixing:}\leavevmode\nobreak\ \leavevmode\nobreak\
f(x_{i},x_{j},\lambda_{f})=\lambda_{f}x_{i}+(1-\lambda_{f})x_{j}=\tilde{x}$
(1) $\textnormal{Label Mixing:}\leavevmode\nobreak\ \leavevmode\nobreak\
g(y_{i},y_{j},\lambda_{g})=\lambda_{g}y_{i}+(1-\lambda_{g})y_{j}=\tilde{y}$
(2)

where $\lambda_{f}$ and $\lambda_{g}$ are defined as the data mixing
coefficient and label mixing coefficient, respectively. We refer to the
combined images $x_{i},x_{j}$ and their labels $y_{i},y_{j}$ as the endpoints.
For a specified mixing coefficient $\lambda$, we denote the resultant image as
$\tilde{x}$. mixup typically assumes $\lambda_{f}=\lambda_{g}$. We instead
decouple the data and label mixing functions to permit a more general
formulation where the data and label mixing functions can have different
coefficients.

###  2.2 Human-in-the-Loop mixup

Our decoupling allows us to probe whether human percepts align with either the
mixing policy over the observations ($f$) or the targets ($g$). Human
alignment of these mixing policies could be important for several reasons.
First, we may want to understand how well the synthetic data used to power
many models deployed on the web matches human perceptual judgments, thus
ensuring model trustworthiness. Second, given that these policies do afford
mixup downstream niceties–such as improved generalization, robustness, and
calibration– we believe it is worth exploring whether modulating such data to
be more human-aligned can yield similar, or better, performance boosts. We,
therefore, pose two questions to separate groups of human participants to
better elucidate alignment of the mixup synthetic data construction:

RQ1: What $\tilde{x}$ do participants believe matches a given $\tilde{y}$?

RQ2: Conditioned on $\tilde{x}$, what do humans perceive as $\tilde{y}$?

Unless otherwise noted, we focus on the setting where we maintain the
structural form of $f$ and $g$; that is, they are each parameterized by a
single mixing coefficient. We discuss alternative functional forms which may
more flexibly capture the richness of human percepts of these synthetically-
constructed images in the Supplement.

##  3 Selecting a Matching Midpoint (RQ1)

We first consider holding $g$ fixed and creating a perceptually-aligned input.
We liken this setting to counterfactual data creation from [Kaushik et al.,
2019].

###  3.1 Problem Setting

In our setup, we inform participants that they will observe samples combined
from particular categories $y_{i},y_{j}$. We fix the label mixing coefficient,
$\lambda_{g}$ (here, to 0.5 – but our procedure could be extended to arbitrary
mixing coefficients) and ask participants to construct a viable $\tilde{x}$
that would be perceived as the $\lambda_{g}$ mixture of the categories.
Ideally, we may want to see what kind of example the participant may select
from the full space of possible examples (in our case, images); for
simplicity, we restrict that participants choose a $\tilde{x}$ from a set of
$M$ pre-constructed linear interpolations which we refer to as
$\\{\tilde{x}_{j}\\}_{j=1}^{M}$, which we refer to as $\tilde{X}_{M}$. Each
$\tilde{x}_{j}$ is the result of executing $f$ for a given $\lambda_{f}$.
Here, we consider a sweep of over the mixing coefficients
$[0.0,0.1,...0.9,1.0]$. From their selected image, we can uncover how their
perception of the data-generating process differs relative to what was
actually used to create said selected image.

###  3.2 Elicitation Paradigm

We design two means of eliciting people’s selection of a $\tilde{x}$:

  1. 1.

Interface 1 (Construct): participants use their keyboard to iterate over
$\tilde{X}_{M}$ (ordered), where key presses increment or decrement $j$ by one
such that $\tilde{x}_{j}$ are cycled through at increments of $0.1$. One mixed
example is displayed on the screen at a time. Participants press “Next” when
they are happy with the selected $\tilde{x}_{j}$.

  2. 2.

Interface 2 (Select-Shuffled): participants see all
$\tilde{x}\in\tilde{X}_{M}$ on the screen at once. Mixed examples are shuffled
and thus presented in an unordered fashion. Participants indicate their
selection by clicking on the $\tilde{x}_{j}$ they think best matches
$\lambda_{g}$.

Example interfaces, and design rationales, are depicted in the Supplement. As
mentioned, participants are explicitly told the categories being combined
($y_{1},y_{2}$) and are asked to indicate the image that they think is most
likely to be perceived as the 50/50 combination of the mixed images by 100
other crowdsourced workers. Such elicitation language is drawn from [Chung et
al., 2019], following a recommended practice in high-fidelity human subject
elicitation whereby participants are asked to assume a third-person
perspective when responding [Prelec, 2004, Oakley and O’Hagan, 2010].

##### Stimuli and Participants

We focus on a random subset of the CIFAR-10 test images, a dataset containing
low-resolution images drawn from ten categories of objects and animals (e.g.,
truck, ship, cat, dog) [Krizhevsky et al., 2009]. We use the test set as this
permits downstream comparisons against CIFAR-10H: an expansive set of
approximately 51 human annotators’ judgments about each example [Peterson et
al., 2019, Battleday et al., 2020]. From each unique category combination
(e.g., truck-dog, ship-cat, cat-dog), we sample $6$ random images from each of
the categories and linearly combine them in pixel-space. We sample $249$ such
image pairings, and for each, we sweep over the space of $11$ mixing
coefficients incremented by $0.1$ between $\lambda_{f}=0.0$ and
$\lambda_{f}=1.0$ (totaling $2739$ synthetically mixed images in total). We
recruit a total of 70 participants from Prolific [Palan and Schitter, 2018]
and hosted on Pavlovia. 45 participants were allocated to Construct, which was
subdivided into two conditions based on the starting point of the selection:
23 participants started at the $\lambda_{f}=0.9$ mixing coefficient, and 22
participants were assigned always starting at $\lambda_{f}=0.1$. The remaining
25 participants were allocated to Select-Shuffled. Further details are
included in the Supplement.

###  3.3 Investigating Data Mixing Alignment

Figure 2: Averaging human participants’ selections per image pair reveals the
typical pair is minimally relabeled.

We find that, in aggregate, humans’ selections indicate alignment with the
underlying mixing coefficient (see Fig. 2), which is stable across elicitation
methodology. However, we cannot conclude from these data that the mixup data
policy is aligned with humans. If we look at the selections made by individual
humans, we see that a substantial portion endorsed a $\tilde{x}$ which
differed from that which would naturally be assumed in mixup (see Fig. 3).
Example image pairs that yield high relabeling across interface types are
shown in Fig. 4. We identify 9 such image pairs that are highly relabeled
(which we define as $|\lambda_{h}-0.5|\geq 0.15$, where we let $\lambda_{h}$
be the mixing coefficient used to generate the $\tilde{x}$ selected by humans)
across interface types. This picture suggests that indeed human percepts are
not consistently aligned with the synthetic data construction process – and
that perhaps with a larger set of stimuli, more such examples can be
recovered. Note, there are a total of 101 image pairs that are endorsed by at
least one interface as in need of high relabeling. More work is needed to
elucidate whether discrepancies in relabeling were induced by the varied
interface design or simply individual differences among the participants
recruited.

Figure 3: Participants do not always endorse the 50/50 point suggesting
misalignment in the data labeling policy. The bar plot depicts extracted
mixing coefficient of individuals’ selections for the perceptually-aligned
midpoints. Figure 4: Example image pairs where substantial relabeling of the
50/50 point was recommended across all interface types. Synthetic images
highlighted in blue received the most endorsements from participants across
all interface types, with images in green receiving the second most. For row
three, participants were split equally between two selections. The mixing
coefficient ($\lambda_{f}$ or $\lambda_{h}$) used to construct the images is
shown along the bottom.

Takeaways These data suggest that while in general, the 50/50 combined image
is recoverable – at an individual level, such percepts are more nuanced. Our
data, which we include as part of H-Mix, indicate systematic differences in
perceptions of synthetically-constructed data. These differences emerge
somewhat robustly across elicitation types. We next turn to richer traces of
humans’ perceptual representations of these synthetically-generated data.

##  4 Elucidating Alignment of the Label Mixing Policy (RQ2)

The above elicitations have focused only on the 50/50 point; however, mixup
trains on synthetically-generated images sampled for a wide range of mixing
coefficients. It, therefore, warrants study to analyze human perceptual
alignment over a richer spectrum of mixing coefficients. We consider instead
eliciting humans’ judgments over what the label mixing coefficient
$\lambda_{g}$ ought to be. Studying the alignment of $g$ could push forward a
deeper understanding of what the data often used to train mixup and similar
methods even means to humans, and potentially further motivate the design of
alternative relabeling schemes (see Section 5). We therefore now focus on
utilizing human input to design a perceptually-aligned target mixup policy
$g_{h}$.

###  4.1 Problem Setting

We assume $f$ is a linear mixing policy over inputs employed in [Zhang et al.,
2018]. To form our human-aligned target policy, we want to find a function
$g_{h}(y_{i},y_{j},\lambda)=\tilde{y}$ such that $\tilde{y}$ perceptually
corresponds to the associated mixed input $f(x_{i},x_{j},\lambda)=\lambda
x_{i}+(1-\lambda)x_{j}=\tilde{x}$. How do we get $\tilde{y}$ from people
efficiently?

We consider matching $\lambda_{g}$ to what humans infer $\lambda_{f}$ to be.
In this setup, we assume humans are aware of the generative processes $f$ and
$g_{h}$, and are shown the mixed image $\tilde{x}$ and underlying labels
$y_{i},y_{j}$. People are then tasked with forming a probabilistic judgment as
to what the underlying mixing coefficient is that generated the observed image
$\tilde{x}$ when given the underlying $y_{i},y_{j}$ – e.g., judging
$P(\lambda_{f}|\tilde{x},y_{i},y_{j})$.

If human perception is aligned to the underlying linear mixup policies, then
the human predicted mixing coefficient $\lambda_{h}$ should be equivalent to
$\lambda_{f}$, rendering $\lambda_{f}=\lambda_{g}=\lambda$ a sensible mixing
scheme. However, if human estimates are not aligned, we may consider setting
$\lambda_{g}=\lambda_{h}$ to make $g$ yield a $\tilde{y}$ which best
corresponds to humans’ percepts of $\tilde{x}$.

###  4.2 Elicitation Paradigm

To elicit such information, we design a new interface where subjects infer the
mixing coefficient between two given labels. We show each worker a mixed image
and tell them the categories that were mixed to generate the image.
Participants also provide us with their uncertainty in their inference. As
some image combinations appear quite convoluted, we reason that subjects’
confidence in their inference – or lack therefore – may provide interesting
signals as to the perceptual sensibility of the mixed images. We run our
relabeling experiment on $N=81$ participants again through Prolific [Palan and
Schitter, 2018]. Further details are included in the Supplement.

##### Stimuli selection

Similar to Section 3.2, we sample images to mix from CIFAR-10 [Krizhevsky et
al., 2009]. We do so in a class-balanced fashion: $46$ mixed images are
sampled for each of the $45$ possible class combinations, resulting in $2070$
total stimuli. Each mixed image is formed by constructed by selecting a data
mixing coefficient $\lambda_{f}\in\\{0.1,0.25,0.5,0.75,0.9\\}$.

###  4.3 Validating the Mixing Coefficient against Human Responses

We now compare the human-inferred mixing coefficient against the generating
coefficient and analyze participants’ uncertainty in such inferences. We also
conduct a preliminary exploration into the relationship between participants’
predicted uncertainty and the ambiguity of the underlying images being
combined.

Figure 5: We uncover a sigmoidal relationship between humans’ inferred mixing
coefficient ($\lambda_{h}$, blue) as compared to the mixing coefficient used
to generate the image ($\lambda_{g}$, red) suggestive of misalignment. We
depict the median, along with the 25th and 75th percentiles. The red line
indicates what the exact parallel between $\lambda_{h}$ and $\lambda_{f}$
would look like (highlighting perceived human deviation).

####  4.3.1 Relationship between Generating Mixing Coefficient and Alignment

We consider whether participants recover the data mixing coefficient: in Fig.
5, we show the median relabeling for images at given coefficients. We observe
a non-linear, roughly sigmoidal structure to human relabelings, consistent
with past research in human categorical perception [Harnad, 2003, Goldstone
and Hendrickson, 2010, Folstein et al., 2013, Destler et al., 2019]. The
aggregate recovery of the 50/50 point corroborates our findings in RQ1.
However, we find that the picture is nuanced: wide confidence bounds suggest
there are mixed images for which inferred mixing coefficients are
substantially different from the parameterization assumed in mixup.
Qualitative inspection of averaged relabelings for particular images (Fig. 6)
– and across category pairs (Fig. 7) – also reveals such misalignment. We
recommend future work to investigate why particular category pairs, for this
dataset, are yielding different boundaries.

Figure 6: Examples of average human relabelings of the generating mixing
coefficient reveal discrepancies. Figure 7: “Category boundaries” elicited
from humans display a diverse structure. Many – though not all – deviate from
linearity assumed in mixup. We overlay examples of synthesized stimuli,
ordered by the $\lambda_{f}$ used to create them.

####  4.3.2 Analyzing Human Uncertainty

We next look closer at the reported human uncertainty in the mixing
coefficient. First, we investigate whether human uncertainty estimates depend
on the mixing coefficient assigned. Indeed, we do observe that humans’
uncertainty tracks with the mixing coefficient (see Table 1); participants
have the lowest confidence (i.e., highest uncertainty) for images generated
from $\lambda_{f}=0.5$.

Table 1: Participants’ average reported confidence, or uncertainty, in their inference of the mixing coefficient (higher confidence means less uncertainty). Error bars indicate standard deviation across participants. The mixing coefficient here is computed as $|0.5-\lambda_{f}|$ due to symmetry (a mixing coefficient of $0.1$ is as extreme as $0.9$). Mixing Coefficient | Reported Confidence
---|---
0.1 | 0.79 $\pm$ 0.17
0.25 | 0.72 $\pm$ 0.20
0.5 | 0.63 $\pm$ 0.20

Additionally, while intuitive, we probe whether there are specific predictors
of when and why a mixed image may be hard to label – e.g., perhaps images
which are naturally ambiguous become even more muddled when combined. We use
the entropy of the CIFAR-10H labels as a measure of image “ambiguity”[Peterson
et al., 2019, Battleday et al., 2020]. Recall, CIFAR-10H labels are
constructed from many annotator’s judgments about the most probable image
category; entropy is therefore computed over the frequencies of these class
selections and captures some sense of the amount of disagreement between
annotators.

Figure 8: Uncertainty reported by annotators in their inference of $\lambda$,
as a factor of whether the combined labels $y_{i},y_{j}$ are high or low
entropy. Entropy is measured over the CIFAR-10H human-derived labels.

We compare humans’ elicited confidence in their mixing coefficient, and the
amount of relabeling ($|\lambda_{h}-\lambda_{f}|$) against the entropy of the
CIFAR-10H labels of the images being combined. We find in Fig. 8 that if both
endpoints are high entropy under CIFAR-10H (where we consider “high” being
entropy $\geq$ 0.5), participants report markedly lower confidence in their
inference than if both endpoints have low entropy (entropy $\leq$ 0.1).
However, we do not find a significant effect of endpoint entropy and amount of
relabeling. This suggests that the ambiguity of the underlying images being
mixed plays some role in determining when the resulting synthetic image may be
hard to label, but there remains a question as to what can predict high
amounts of relabeling from participants. We leave these questions for future
investigation.

We can go further in the study of human uncertainty over mixup examples by
directly eliciting soft labels from each individual over the entire space of
possible classes, inspired by [Collins et al., 2022a]. We include a
preliminary investigation into eliciting richer forms of human uncertainty
over mixup examples, which lend additional nuance to the discrepancy between
human perceptual judgments and the synthetic labels classically used in mixup,
in the Supplement. In particular, our primary finding is that people sometimes
place probability mass on classes which are different from the endpoint
classes being combined (see Figure 9). We include the elicited soft labels in
our release of H-Mix; these soft labels are small-scale at present (from $N=8$
participants, see Supplement), and we have not yet explored their
computational implications, but see grounds for leveraging richer forms of
human uncertainty in this vein as ripe for future work.

Figure 9: Example combined image ($\lambda_{f}=0.5$; horse/ship) which has
been relabeled by humans (blue) through the elicitation of individual soft
labels our soft label elicitation (see Supplement); here, we average the
individual soft labels derived from the two different participants who
annotated the image. The label which would be used by mixup is shown in red.

##### Takeaways

Our dataset, H-Mix, highlights discrepancies between humans’ internal models
of synthetically generated data compared to what is traditionally used in
mixup. We observe variable labeling policies on a category-pair basis and
uncover a likely relationship between the strength of the mixing coefficient
and ambiguity of the underlying combined images with participants’ reported
uncertainty in their judgments. We also preview even richer discrepancies
between human percepts and the functional form of the mixup mixing policies
(see Supplement).

##  5 Learning with Human Relabelings and Uncertainty

In addressing RQ1 and RQ2, this work illuminates that human perceptual
judgments do not consistently recover the parameters of the generative model
traditionally used to construct data in mixup. These findings beg the
question: if we instead align synthetic examples with human perceptual
judgments, how would this impact model performance? Such a question is
important to consider in the pursuit of more trustworthy ML systems: better
generalization, robustness, calibration, and a richer understanding of whether
the models are trained on human-aligned data could all potentially engender
more stakeholder trust [Zerilli et al., 2022].

To that end, we consider two initial empirical studies of the impact of
training on human perceptual judgments of synthetic examples: one, wherein we
compare training models with varied forms of labels on the specific set of
$2070$ mixed images from H-Mix, and another where we go beyond the collected
examples and consider a first attempt at constructing a generic human-aligned
label mixing policy. Here, we focus on the data collected for RQ2; i.e., for
given $\tilde{x}$ how should we change $\tilde{y}$. We encourage leveraging
and scaling the data collected in RQ1 for future work.

###  5.1 Relabeling Directly with H-Mix

Setup We train a PreAct ResNet-18 [He et al., 2016] and VGG-11 [Simonyan and
Zisserman, 2014] over 7,000 regular CIFAR-10 images (following the split used
by [Collins et al., 2022a]) combined with the 2,070 synthetically mixed images
where we vary the labels. While we would ideally study human relabelings for
every synthetic image that could be generated with $f$, we only have labels
for a small subset and instead compare using our labels versus traditional
mixup labels over a finite, augmenting set of combined images. 5 seeds are run
per variant per architecture. Results are averaged across architectures.

Evaluation We evaluate a suite of metrics over 3,000 examples from CIFAR-10H,
a dataset containing labels from many humans over the CIFAR-10 test set
[Peterson et al., 2019]. We compare: cross entropy between the model-predicted
and the human-derived label distributions (CE), model calibration following
[Hendrycks et al., 2022] and robustness to the Fast Gradient Sign Method
(FGSM) adversarial attack [Goodfellow et al., 2014b], again following the set-
up of [Collins et al., 2022a].

Leveraging Human Relabelings for ML Training We first compare learning with
our averaged human-inferred mixing parameters against the classical mixup
labels over the same 2070 synthetically-mixed images. We include sanity checks
with completely random and uniform labels for the synthetic examples, as well
as a baseline not including any synthetic examples (“No Aug”). Interestingly,
we find in Table 2 that aligning the mixed example labels with averaged human
labels yields worse model performance. We think these results are worth
highlighting: it is not always the case that aligning models to human
perception yields performance gains, possibly due to the recently discovered
U-shaped relationship between representational alignment and generalization
[Sucholutsky and Griffiths, 2023].

The Value of Human Uncertainty Information However, the human-inferred
$\lambda_{g}$ alone does not capture the richness of human perceptual
judgments over synthetic images: participants at times reported being
uncertain in their inferences. Therefore, we account for human uncertainty
($\omega$) in the inference of the synthetic data generating parameter to
construct softer $\tilde{y}$ (see Supplement for details). We find substantial
performance boosts come from leveraging human uncertainty. Such data suggest
that indeed, aligning models in accordance with human perceptual inferences
could have advantages – and suggests that confidence could offer a potent
modulator signal worth considering eliciting. This is in line with core ideas
from Dempster-Shafer Theory [Shafer, 1976, Dempster, 1967], that soft labels
should be expressed as one set of values representing the mixture weights, and
a second associated set of values representing uncertainty about that mixture
(i.e., belief and plausibility).

Table 2: Comparing performance when varying the form of the synthetic labels on the $2070$ mixed images. Results averaged over 5 seeds, with error bars depicting 95% confidence intervals (CIs) across seed performance. Label Type | CE | FGSM | Calib
---|---|---|---
Regular |  |  |
(No Aug) | 2.02$\pm$0.12 | 13.12$\pm$2.65 | 0.28$\pm$0.011
\+ Random | 2.11$\pm$0.13 | 12.81$\pm$2.84 | 0.24$\pm$0.014
\+ Uniform | 2.16$\pm$0.14 | 12.71$\pm$2.79 | 0.25$\pm$0.012
\+ mixup | 1.65$\pm$0.11 | 10.62$\pm$2.44 | 0.23$\pm$0.005
\+ Ours |  |  |
(Relabel) | 1.78$\pm$0.12 | 11.69$\pm$2.90 | 0.24$\pm$0.009
(Relabel & $\omega$) | 1.48$\pm$0.06 | 8.89$\pm$1.59 | 0.19$\pm$0.001

###  5.2 Generalizing Relabeling

So far, we have focused on varying the labels of a pre-supposed augmenting set
of mixed images; however, the set was comparatively small (2070 images) and
therefore does not directly mimic the mixup learning paradigm. In practice,
mixup is typically applied over the entire dataset; that is, on each batch, a
new mixing coefficient is sampled, resulting in often entirely new images
being generated per batch. It is infeasible to consider recruiting human
participants to relabel every such image. Automated human-aligned labeling
policies are therefore worth considering. We argue that our data offers a
prime starting point to explore such questions.

We offer a preliminary alternative label mixing policy based on the human data
we have collected in H-Mix. Inspired by the non-linearities we observe at a
category level, we use scipy.curve_fit to fit a logistic function per category
pair. For each batch, we swap in our label mixing policy to map from the
sampled generating mixing coefficient to an approximately more human-
perceptually aligned coefficient. Such fits only account for humans’
relabelings, not their uncertainty. Accounting for human confidence in
automated label policies is a ripe direction for future work.

Setup We follow the same ensembling and evaluation methodology laid out in
Section 5.1, but now run traditional mixup following [Zhang et al., 2018]
where generating mixing coefficients are sampled from a $Beta(1,1)$
distribution (i.e., uniform on $(0,1)$).

Results We observe (see Table 3) a striking parity in performance across
models. These data highlight that with the addition of even a relatively small
number of human annotations through HMix to alter the labeling policy, we find
that robustness to adversarial attacks increases at negligble cost to
performance or calibration. As in [Sucholutsky and Griffiths, 2023], human
representation alignment may be useful for other downstream, untested tasks:
training on more human-aligned data-generating policies could induce
functional fits that are preferable to stakeholders even if we see no
objective improvement along particular performance measures. We recommend such
studies for future work.

Table 3: Training with mixing policies fitted per category pair, compared against full mixup. Results averaged over 5 seeds, with 95% CI error bars. Label Policy | CE | FGSM | Calib
---|---|---|---
mixup | 1.15$\pm$0.08 | 7.46$\pm$2.40 | 0.10$\pm$0.01
Human-Fits (Ours) | 1.16$\pm$0.08 | 7.32$\pm$2.27 | 0.10$\pm$0.01

Takeaways Human perceptual judgments can be leveraged to construct alternative
synthetic data-generating policies to train ML systems; however, such induced
methods of aligning with (approximations) of human perception are not
automatic salves. Our results highlight that constructing more human-aligned
label policies, particularly through capturing and representing human
uncertainty, is promising, but more work is needed before generalizing
conclusions.

##  6 Discussion

(Mis)alignment of Mixup Examples Through a series of novel user studies, we
uncover that the synthetic examples used in mixup do not consistently align
with humans’ perceptual representations. We find indications that
participants’ uncertainty in their inferred mixing coefficients tracks with
the degree of ambiguity of the original images that are combined. As we have
begun to explore empirically, such relabeling may impact downstream model
performance: re-aligning mixup labels with humans’ reported judgments can
impact learning, with human uncertainty seemingly poised to provide a strong
supervisory signal – corroborating [Peterson et al., 2019, Collins et al.,
2022b, Sucholutsky et al., 2023]. The collation of humans’ inferences of the
mixup generative parameters could also be used to benchmark whether models are
aligned with human percepts, say if H-Mix is used as a held-out or probe set
[Gruber et al., 2018]. We recommend such directions for future work,
particularly those focused on the uncertainty elicitation in H-Mix. We release
additional soft labels over mixed examples which further highlight human
perceptual misalignment (see Supplement).

Scaling Human-Centric Data Relabeling A key challenge for human-centric
relabeling of synthetically-generated data (not unique to mixup) is that a
nearly infinite variety can be generated. It is not reasonable to expect
humans to judge all possibilities, nor to provide their uncertainty over all
labels.Any attempt at human-in-the-loop relabeling faces the obstacle of
identifying which examples to relabel, and how to handle cases that cannot be
relabeled. While we take steps to address the latter through fitting generic
functions per class pair that enable sampling of arbitrary mixing
coefficients, we highly encourage researchers to consider leveraging our H-Mix
to develop alternative human-grounded automated synthetic data policies.

To address the former, we encourage looking to smarter ways to select examples
to query people over – rather than random selection as we have done – such as
[Liu et al., 2021a, 2017]. Additionally, our results raise the related
question: are there particular relabelings that are hurting model performance?
Prior works have demonstrated how cleaning data can reduce model error [Pleiss
et al., 2020]. We encourage future work in this direction in the context of
H-Mix. Additionally, our results raise the related question: are there
particular relabelings that are hurting model performance? Prior works have
demonstrated how cleaning data can reduce model error [Pleiss et al., 2020].
We encourage future work in this direction in the context of H-Mix.

Limitations Thus far, we only consider human validation and relabeling of
mixup labels for a single image classification dataset, CIFAR-10. This dataset
is low-resolution. Thus, the endpoint images – and the combinations of images
– can be ambiguous and challenging to interpret. It is possible that we may
find humans to be more, or less, aligned with the generative parameters for
different image datasets, or for entirely different data modalities, e.g.,
audio or video. We encourage the application of the HILL MixE Suite paradigm
to other datasets. Moreover, as we have many category pairs – arising even
from just 10 categories – we do not have a substantial number of synthetic
examples per category pair (i.e., 46 synthetically-mixed images for each of
the 45 category pairs). This could impact the stability of the category
boundaries we elicit, e.g., potentially leading to breaks of monotonicity (see
Supplement A). Further, as with many web-based human elicitation studies, it
is not always clear whether the responses returned arise from individual
differences in perception, participant noise, or malicious behavior [Lease,
2011, Gadiraju et al., 2015]. We also do not train participants to provide
calibrated uncertainty; uncertainty judgments included in H-Mix – while
empirically useful for training – could be infused with classical biases in
humans’ probabilistic self-reports [Lichtenstein et al., 1977, Tversky and
Kahneman, 1996, O’Hagan et al., 2006, Sharot, 2011]. We also highlight that,
aside from repeat trials, we are unable to capture whether participants’
percepts fluctuate – such instability is certainly a possibility when
considering cognitive neuroscience research around perceptual dominance [Blake
and Logothetis, 2002].

Extending to New Synthetic Data Paradigms In this work, we focused on the
synthetic data classically used in mixup, as the simplicity of the data
generating process – a single mixing coefficient parameter – enables us to
precisely compare human versus traditional parameterizations of the synthetic
data construction process. We hope our work spurs further study of aligning
synthetic data generation with human perception and motivates the design of
more human-aligned synthetic data to improve ML systems, particularly those
focused on the interplay between model and human uncertainty. We release the
code of all interfaces included in our HILL MixE Suite, which we hope will
empower researchers with additional tools to investigate humans’ percepts over
synthetically-constructed data. For instance, our Select-Shuffled interface
could readily be extended to elicit stakeholders’ preferences, in the form of
selection, over any collection of constructed synthetic examples. As
demonstrated in [Ouyang et al., 2022], scalable human preference elicitation
has wide utility.

##  7 Conclusion

Through a series of human participant elicitation studies, we find that the
synthetic examples generated via mixup differ in fundamental ways from human
perception, suggesting misalignment of the data and label mixing policies. We
offer early indications that collating humans’ percepts of these synthetic
examples could impact model performance, particularly when modulated by
elicited human uncertainty. Our work further motivates the design of automated
relabeling procedures for synthetic examples which leverage elicited human
data (e.g., training a model to predict a likely human’s mixing coefficient)
to sidestep inherent issues with scaling human annotation over the space of
possible synthetic examples, particularly in eliciting and utilizing human
uncertainty. Synthetic data of all kinds are proliferating: we encourage more
researchers to consider these data from a human-centric perspective; i.e.,
investigating whether the samples align with human percepts, and if not,
whether altering labels – specifically via human uncertainty – can yield
safer, more reliable models with improved generalization.

## Acknowledgments

We thank (alphabetically) Federico Barbero, Alan Clark, Krishnamurthy (Dj)
Dvijotham, Benedict King, Tuan Anh Le, Haitz Sáez de Ocáriz Borde, N.
Siddharth, Joshua Tenenbaum, Richard E. Turner, Vishaal Udandarao, and Lio
Wong for helpful discussions. We also thank our participants on Prolific, and
our reviewers for incredibly helpful feedback.

KMC is supported by a Marshall Scholarship. UB acknowledges support from a JP
Morgan AI PhD Fellowship and from DeepMind and the Leverhulme Trust via the
Leverhulme Centre for the Future of Intelligence (CFI). BL acknowledges
support under the ESRC grant ES/W007347, and AW acknowledges support from a
Turing AI Fellowship under grant EP/V025279/1, The Alan Turing Institute, and
the Leverhulme Trust via CFI. IS is supported by an NSERC fellowship
(567554-2022).

## References

  * Arazo et al. [2020] Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness.  Pseudo-labeling and confirmation bias in deep semi-supervised learning.  In _International Joint Conference on Neural Networks_ , 2020.
  * Battleday et al. [2020] Ruairidh M Battleday, Joshua C Peterson, and Thomas L Griffiths.  Capturing human categorization of natural images by combining deep networks and cognitive models.  _Nature communications_ , 11(1):1–14, 2020.
  * Beale and Keil [1995] James M Beale and Frank C Keil.  Categorical effects in the percxeption of faces.  _Cognition_ , 57(3):217–239, 1995.
  * Blake and Logothetis [2002] Randolph Blake and Nikos K Logothetis.  Visual competition.  _Nature Reviews Neuroscience_ , 3(1):13–21, 2002\.
  * Cascante-Bonilla et al. [2020] Paola Cascante-Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez.  Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning.  _arXiv preprint arXiv:2001.06001_ , 2020.
  * Chen et al. [2022] Valerie Chen, Umang Bhatt, Hoda Heidari, Adrian Weller, and Ameet Talwalkar.  Perspectives on incorporating expert feedback into model updates.  _arXiv preprint arXiv:2205.06905_ , 2022.
  * Chuang and Mroueh [2020] Ching-Yao Chuang and Youssef Mroueh.  Fair mixup: Fairness via interpolation.  In _ICLR_ , 2020.
  * Chung et al. [2019] John Joon Young Chung, Jean Y. Song, Sindhu Kutty, Sungsoo (Ray) Hong, Juho Kim, and Walter S. Lasecki.  Efficient elicitation approaches to estimate collective crowd answers.  In _CSCW_ , 2019.
  * Collins et al. [2022a] Katherine M Collins, Umang Bhatt, and Adrian Weller.  Eliciting and learning with soft labels from every annotator.  In _HCOMP_ , 2022a.
  * Collins et al. [2022b] Katherine M Collins, Catherine Wong, Jiahai Feng, Megan Wei, and Joshua B Tenenbaum.  Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks.  _arXiv preprint arXiv:2205.05718_ , 2022b.
  * de Melo et al. [2022] Celso M. de Melo, Antonio Torralba, Leonidas Guibas, James DiCarlo, Rama Chellappa, and Jessica Hodgins.  Next-generation deep learning based on simulators and synthetic data.  _Trends in Cognitive Sciences_ , 26(2):174–187, 2022.
  * Dempster [1967] A. P. Dempster.  Upper and Lower Probabilities Induced by a Multivalued Mapping.  _The Annals of Mathematical Statistics_ , 38(2):325 – 339, 1967.  10.1214/aoms/1177698950.  URL https://doi.org/10.1214/aoms/1177698950.
  * Destler et al. [2019] Nathan Destler, Manish Singh, and Jacob Feldman.  Shape discrimination along morph-spaces.  _Vision Research_ , 158:189–199, 2019.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat GANs on image synthesis.  In _NeurIPS_ , 2021.
  * Emam et al. [2021] Zeyad Emam, Andrew Kondrich, Sasha Harrison, Felix Lau, Yushi Wang, Aerin Kim, and Elliot Branson.  On the state of data in computer vision: Human annotations remain indispensable for developing deep learning models.  _arXiv preprint arXiv:2108.00114_ , 2021.
  * Fel et al. [2022] Thomas Fel, Ivan Felipe, Drew Linsley, and Thomas Serre.  Harmonizing the object recognition strategies of deep neural networks with humans.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022\.
  * Feldman [2021] Jacob Feldman.  Mutual information and categorical perception.  _Psychological Science_ , 32(8):1298–1310, 2021\.
  * Feldman et al. [2009] Naomi H Feldman, Thomas L Griffiths, and James L Morgan.  The influence of categories on perception: explaining the perceptual magnet effect as optimal statistical inference.  _Psychological review_ , 116(4):752, 2009.
  * Folstein et al. [2012] Jonathan R. Folstein, Isabel Gauthier, and Thomas J. Palmeri.  How category learning affects object representations: not all morphspaces stretch alike.  _Journal of experimental psychology. Learning, memory, and cognition_ , 38 4:807–20, 2012.
  * Folstein et al. [2013] Jonathan R Folstein, Thomas J Palmeri, and Isabel Gauthier.  Category learning increases discriminability of relevant object dimensions in visual cortex.  _Cerebral Cortex_ , 23(4):814–823, 2013.
  * Gadiraju et al. [2015] Ujwal Gadiraju, Ricardo Kawase, Stefan Dietze, and Gianluca Demartini.  Understanding malicious behavior in crowdsourcing platforms: The case of online surveys.  In _CHI_ , 2015.
  * Goldstone and Hendrickson [2010] Robert L Goldstone and Andrew T Hendrickson.  Categorical perception.  _Wiley Interdisciplinary Reviews: Cognitive Science_ , 1(1):69–78, 2010.
  * Goodfellow et al. [2014a] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _NeurIPS_ , 2014a.
  * Goodfellow et al. [2014b] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ , 2014b.
  * Gruber et al. [2018] Liron Z Gruber, Aia Haruvi, Ronen Basri, and Michal Irani.  Perceptual dominance in brief presentations of mixed images: Human perception vs. deep neural networks.  _Frontiers in Computational Neuroscience_ , 12:57, 2018\.
  * Harnad [2003] Stevan Harnad.  Categorical perception.  In _Encyclopedia of Cognitive Science_ , volume 67. MacMillan: Nature Publishing Group, 2003.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  pages 770–778, 2016.
  * Hendrycks et al. [2019] Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan.  Augmix: A simple data processing method to improve robustness and uncertainty.  In _ICLR_ , 2019.
  * Hendrycks et al. [2022] Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song, and Jacob Steinhardt.  Pixmix: Dreamlike pictures comprehensively improve safety measures.  In _CVPR_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _NeurIPS_ , 2020.
  * Jordon et al. [2022] James Jordon, Lukasz Szpruch, Florimond Houssiau, Mirko Bottarelli, Giovanni Cherubin, Carsten Maple, Samuel N Cohen, and Adrian Weller.  Synthetic data–what, why and how?  _arXiv preprint arXiv:2205.03257_ , 2022.
  * Kaushik et al. [2019] Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton.  Learning the difference that makes a difference with counterfactually-augmented data.  _arXiv preprint arXiv:1909.12434_ , 2019.
  * Kaushik et al. [2021] Divyansh Kaushik, Amrith Setlur, Eduard H. Hovy, and Zachary C. Lipton.  Explaining the efficacy of counterfactually-augmented data.  _ICLR_ , 2021.
  * Kim et al. [2020a] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song.  Puzzle mix: Exploiting saliency and local statistics for optimal mixup.  In _ICML_ , 2020a.
  * Kim et al. [2020b] JangHyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song.  Co-mixup: Saliency guided joint mixup with supermodular diversity.  In _ICLR_ , 2020b.
  * Krizhevsky et al. [2009] Alex Krizhevsky et al.  Learning multiple layers of features from tiny images.  2009\.
  * Kuhl [1991] Patricia K Kuhl.  Human adults and human infants show a “perceptual magnet effect” for the prototypes of speech categories, monkeys do not.  _Perception & psychophysics_, 50(2):93–107, 1991.
  * Lease [2011] Matthew Lease.  On quality control and machine learning in crowdsourcing.  In _AAAI Workshops_ , 2011.
  * Lichtenstein et al. [1977] Sarah Lichtenstein, Baruch Fischhoff, and Lawrence D Phillips.  Calibration of probabilities: The state of the art.  _Decision making and change in human affairs_ , pages 275–324, 1977\.
  * Liu et al. [2017] Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B Smith, James M Rehg, and Le Song.  Iterative machine teaching.  In _ICML_ , 2017.
  * Liu et al. [2021a] Weiyang Liu, Zhen Liu, Hanchen Wang, Liam Paull, Bernhard Schölkopf, and Adrian Weller.  Iterative teaching by label synthesis.  In _NeurIPS_ , 2021a.
  * Liu et al. [2021b] Zicheng Liu, Siyuan Li, Di Wu, Zhiyuan Chen, Lirong Wu, Jianzhu Guo, and Stan Z Li.  Unveiling the power of mixup for stronger classifiers.  _arXiv preprint arXiv:2103.13027_ , 2021b.
  * Nanda et al. [2021] Vedant Nanda, Ayan Majumdar, Camila Kolling, John P Dickerson, Krishna P Gummadi, Bradley C Love, and Adrian Weller.  Exploring alignment of representations with human perception.  _arXiv preprint arXiv:2111.14726_ , 2021.
  * Newell and Bülthoff [2002] Fiona N Newell and Heinrich H Bülthoff.  Categorical perception of familiar objects.  _Cognition_ , 85(2):113–143, 2002.
  * Nguyen et al. [2013] Quang Nguyen, Hamed Valizadegan, and Milos Hauskrecht.  Learning classification models with soft-label information.  _Journal of the American Medical Informatics Association_ , 21, 11 2013.
  * Oakley and O’Hagan [2010] Jeremy E Oakley and Anthony O’Hagan.  Shelf: the sheffield elicitation framework (version 2.0).  _School of Mathematics and Statistics, University of Sheffield, UK_ , 2010.
  * O’Hagan et al. [2006] A. O’Hagan, C. E. Buck, A. Daneshkhah, J. R. Eiser, P. H. Garthwaite, D. J. Jenkinson, J. E. Oakley, and T. Rakow.  _Uncertain Judgements: Eliciting Expert Probabilities_.  John Wiley, Chichester, 2006.
  * Oliva et al. [2006] Aude Oliva, Antonio Torralba, and Philippe G. Schyns.  Hybrid images.  _ACM Transactions on Graphics_ , 25(3):527–532, 2006.
  * Ouyang et al. [2022] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, 2022\.  URL https://arxiv.org/abs/2203.02155.
  * Palan and Schitter [2018] Stefan Palan and Christian Schitter.  Prolific. ac—a subject pool for online experiments.  _Journal of Behavioral and Experimental Finance_ , 17:22–27, 2018.
  * Peterson et al. [2019] Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky.  Human uncertainty makes classification more robust.  In _ICCV_ , 2019.
  * Pleiss et al. [2020] Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger.  Identifying mislabeled data using the area under the margin ranking.  _NeurIPS_ , 2020.
  * Prelec [2004] Drazen Prelec.  A bayesian truth serum for subjective data.  _Science_ , 306(5695):462–466, 2004.
  * Qiu et al. [2022] Zeju Qiu, Weiyang Liu, Tim Z Xiao, Zhen Liu, Umang Bhatt, Yucen Luo, Adrian Weller, and Bernhard Schölkopf.  Iterative teaching by data hallucination.  _arXiv preprint arXiv:2210.17467_ , 2022.
  * Sanders et al. [2022] Kate Sanders, Reno Kriz, Anqi Liu, and Benjamin Van Durme.  Ambiguous images with human judgments for robust visual event classification.  In _NeurIPS_ , 2022.
  * Shafer [1976] Glenn Shafer.  _A mathematical theory of evidence_ , volume 42.  Princeton university press, 1976.
  * Sharot [2011] Tali Sharot.  The optimism bias.  _Current biology_ , 21(23):R941–R945, 2011.
  * Silver et al. [2016] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis.  Mastering the game of Go with deep neural networks and tree search.  _Nature_ , 529(7587):484–489, 2016.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Sohn et al. [2022] Jy-yong Sohn, Liang Shang, Hongxu Chen, Jaekyun Moon, Dimitris Papailiopoulos, and Kangwook Lee.  Genlabel: Mixup relabeling using generative models.  _arXiv preprint arXiv:2201.02354_ , 2022.
  * Sohn et al. [2020] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, _NeurIPS_ , 2020.
  * Sucholutsky and Griffiths [2023] Ilia Sucholutsky and Thomas L Griffiths.  Alignment with human representations supports robust few-shot learning.  _arXiv preprint arXiv:2301.11990_ , 2023.
  * Sucholutsky et al. [2023] Ilia Sucholutsky, Ruairidh M. Battleday, Katherine M. Collins, Raja Marjieh, Joshua Peterson, Pulkit Singh, Umang Bhatt, Nori Jacoby, Adrian Weller, and Thomas L. Griffiths.  On the informativeness of supervision signals.  _UAI_ , 2023.
  * Thulasidasan et al. [2019] Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak.  On mixup training: Improved calibration and predictive uncertainty for deep neural networks.  In _NeurIPS_ , 2019.
  * Tokozume et al. [2017] Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada.  Learning from between-class examples for deep sound recognition.  _arXiv preprint arXiv:1711.10282_ , 2017.
  * Tokozume et al. [2018] Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada.  Between-class learning for image classification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5486–5494, 2018.
  * Tong et al. [2006] Frank Tong, Ming Meng, and Randolph Blake.  Neural bases of binocular rivalry.  _Trends in cognitive sciences_ , 10(11):502–511, 2006.
  * Tversky and Kahneman [1996] Amos Tversky and Daniel Kahneman.  On the reality of cognitive illusions.  _Psychological Review_ , 103(3):582–591, 1996\.
  * Verma et al. [2019] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio.  Manifold mixup: Better representations by interpolating hidden states.  In _ICML_ , 2019.
  * Verma et al. [2022] Vikas Verma, Sarthak Mittal, Wai Hoh Tang, Hieu Pham, Juho Kannala, Yoshua Bengio, Arno Solin, and Kenji Kawaguchi.  Mixupe: Understanding and improving mixup from directional derivative perspective, 2022.  URL https://arxiv.org/abs/2212.13381.
  * Vodrahalli et al. [2021] Kailas Vodrahalli, Tobias Gerstenberg, and James Zou.  Do humans trust advice more if it comes from AI? an analysis of human-ai interactions.  _arXiv preprint arXiv:2107.07015_ , 2021.
  * Wei et al. [2022] Jiaheng Wei, Zhaowei Zhu, Tianyi Luo, Ehsan Amid, Abhishek Kumar, and Yang Liu.  To aggregate or not? learning with separate noisy labels.  _arXiv preprint arXiv:2206.07181_ , 2022.
  * Yun et al. [2019] S. Yun, D. Han, S. Chun, S. Oh, Y. Yoo, and J. Choe.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _ICCV_ , 2019.
  * Zaidan et al. [2007] Omar Zaidan, Jason Eisner, and Christine Piatko.  Using “annotator rationales” to improve machine learning for text categorization.  In _NAACL_ , 2007.
  * Zerilli et al. [2022] John Zerilli, Umang Bhatt, and Adrian Weller.  How transparency modulates trust in artificial intelligence.  _Patterns_ , 3(4):100455, 2022.
  * Zhang et al. [2018] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In _ICLR_ , 2018.
  * Zhang et al. [2020] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou.  How does mixup help with robustness and generalization?  In _ICLR_ , 2020.
  * Zhang et al. [2022] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou.  When and how mixup improves calibration.  In _ICML_ , 2022.
  * Zhang et al. [2021] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler.  Datasetgan: Efficient labeled data factory with minimal human effort.  In _CVPR_ , 2021.

## Related Work

Our work connects most closely to human-in-the-loop data augmentation and the
expansive literature surrounding human categorical perception from the
cognitive science community, as well as ongoing efforts in the machine
learning community to develop more efficacious mixup-based data and label
mixing functions.

### Human-in-the-Loop Data Augmentation

Incorporating expert feedback into the learning procedure has received
increasing attention [Chen et al., 2022]. In particular, previous work has
considered incorporating humans “in the loop” for data augmentation. For
instance, DatasetGAN [Zhang et al., 2021] employs human participants to label
GAN-generated images and feeds these back to the model to generate more
synthetic data. [Kaushik et al., 2019] similarly incorporate human feedback by
having humans create counterfactual samples, and has been shown to be an
efficient method to adjust model behavior [Kaushik et al., 2021]. Other works
have considered employing humans to provide “rationales” about examples to
improve data-efficiency and downstream modeling performance [Zaidan et al.,
2007]. Here, we marry these ideas in the context of mixup by eliciting data
and label-mixing function parameters to align with human percepts.

### Human Categorical Perception

In cognitive science, eliciting humans’ judgments over synthetically-
constructed examples is a tried-and-true method to characterize human category
boundaries [Newell and Bülthoff, 2002, Folstein et al., 2013, Feldman, 2021,
Folstein et al., 2012]. Such studies often reveal a non-linear structure of
humans’ percepts. For instance, in the audio domain, the identification of
vowel categories has been found to demonstrate “warping” close to prototypical
category members – known as the “perceptual magnet effect” [Kuhl, 1991,
Feldman et al., 2009]. Similar nonlinearities have been found in the
perception of boundaries between face identities [Beale and Keil, 1995] and
the transitions between 3D shapes [Newell and Bülthoff, 2002, Destler et al.,
2019]. Our linearly interpolated stimuli are similar in spirit to the
morphological trajectories used in these works, as well as other
synthetically-combined images [Oliva et al., 2006]. [Gruber et al., 2018] also
consider 50/50 mixed images; however, their elicitation involves open-ended
judgments which does not permit the same kind of data and label mixing
alignment studies as our methods more directly elicit human-inferred
generative parameters. Our work also connects to other non-linear perceptual
phenomena encountered in the visual domain; namely, binocular rivalry, whereby
present participants with a different image in each eye has been shown to
induce oscillatory percepts [Blake and Logothetis, 2002, Tong et al., 2006].

### Other mixup-Based Synthetic Data Schemes

Many alternative mixup data and label mixing functions have been proposed
[Verma et al., 2019, Yun et al., 2019, Kim et al., 2020a, b, Hendrycks et al.,
2022]. Closest to our work, [Sohn et al., 2022] highlight particular issues
with the linear interpolation in label space on the learned topology of the
model’s category boundaries and instead utilize a Gaussian Mixture Model
(GMM)-based relabeling scheme to construct “better” labels than those used in
baseline mixup. Additional work on learning better pseudo-labels over mixup
samples have been proposed [Arazo et al., 2020, Cascante-Bonilla et al., 2020,
Sohn et al., 2020, Qiu et al., 2022]. Similarly, Between-class (BC) learning
[Tokozume et al., 2017, 2018] proposes hand-crafted adjustments to label
construction to better align with human perception based on waveform
modulations; however, to our knowledge, no previous works have directly
considered incorporating humans in-the-loop for either the construction of
mixup samples or associated relabeling.

## Additional Notes on H-Mix

### Human Subject Experiments

We include additional details on our human elicitation studies. For all
experiments, we require participants speak English as a first-language and
reside in the United States. Across all experiments, the mean age for
participants was 37.5 years old ($\pm$ standard deviation of 12.7 yrs) . The
self-reported sex breakdown was approximately 57% male and 43% female.

##### Elicitation (RQ1)

Each participant sees a total of 32 mixed images, where the final two are
repeats. Repeats are primarily used here to measure raters’ internal
consistency222Participants’ selections, for each interface type, change by a
median of $0.1$ in repeat trials, suggesting some inconsistencies in
participants’ judgments which persists across elicitation method.. The median
time taken per participant per image as 9.30 and 11.01 seconds for the
Construct and Select-Shuffled interfaces, respectively. A bonus was offered to
encourage participants to provide responses which would match what other
participants would provide; we applied this bonus to all participants post-hoc
resulting in the average participant being paid at a rate of $11.78.

##### Multiple Interface Styles (RQ1)

Why do we consider two styles of elicitation interfaces? We reason that the
first interface could be prone to ordering effects – an astute participant
could realize that they can count out where the midpoint is located. This led
us to design the second variety (Select-Shuffled) wherein the participant sees
all images shuffled simultaneously. We hypothesize that Construct could induce
responses biased by the participant’s starting position. To probe this, we run
two sub-variants wherein participants start from either $\lambda_{f}=0.1$ or
$\lambda_{f}=0.9$.

##### Elicitation (RQ2)

Each participant sees $59-62$ images, where two images are repeated. Repeats
are placed at the end and correspond to the images presented on trials 15 and
20, respectively333We observe a median difference of $0.03$ and $0.05$ in the
inferred mixing coefficient and confidence on repeat trials, indicative of
high intra-annotator consistency.. The order of the images presented in a
batch, as well as the order of the endpoint labels displayed for a given
image, are shuffled across participants. We follow the same third-person
perspective prompting in Section 3 from [Chung et al., 2019]. Participants are
asked “what combinations of classes” they thought other participants would say
is “used to make” each image, and “how confident” they thought other
participants would be in their estimate. Responses are indicated on a slider
per question. An example survey screen can be seen in Fig. 13. Subjects took a
median of 8.41 seconds per image and were payed at a rate of $8/hr, with an
optional bonus which sought to encourage participants to provide calibrated
confidence estimates, similar to that of [Vodrahalli et al., 2021]; the bonus
was applied to all participants post-hoc. Each mixed image was seen by at
least two different participants each. Our interface is depicted in Fig. 13.

### Break from Monotonicty

For users of H-Mix, it is worth noting that we do encounter some breaks with
monotonicity (see Fig. 10) in a few of the aggregated “category boundaries.”
We reason this could be in part due to several aspects of our set-up. First,
our study involved irregular sampling across the space of mixing coefficients
we consider: the 50/50 point is enriched. We ran two phases of elicitation: in
the first, we sampled $6$ image classes per pair to be shown for three mixing
coefficients: 0.5, and one chosen randomly from each of the sets {0.1, 0.25}
and {0.75, 0.9}, respectively (810 images of the 2070). All 1260 other images
are shown for a single mixing coefficient sampled uniformly from the set.
Second, while we have human judgments for over 2000 total images, there are
less than 50 synthetic images considered for each category pair, giving any
participant noise – or the odd image – greater leverage to impact trends. We
encourage others to use HILL-MixE Suite and continue to scale this work and
elucidate the stability of the inferred mixing coefficient category boundaries
we begin to hint at here.

Figure 10: Category boundary elicited from human participants involves a break
with monotonicity.

## Confidence-Based Smoothing Details

We include further details of our methodology for leveraging human-provided
confidence to construct $\tilde{y}$ introduced in Section 5. Human-derived
soft labels have been demonstrated to be valuable for learning [Nguyen et al.,
2013, Peterson et al., 2019, Collins et al., 2022a, Sanders et al., 2022]. We
transform humans’ reported confidence into a smoothing parameter to induce
softness using an exponentially-decaying function of human-provided confidence
$\omega$: $a*(b^{\omega})$; here, $a=50,b=0.0001$. We use the transformed
confidence for additive smoothing on the two-category $\tilde{y}$, spread mass
accordingly across the full gamut of classes. That is, we use smooth the mass
between a completely uniform distribution and a “two-hot” label which uses the
human-derived relabeling. Parameters $a,b$ are selected using a held-out set
of regular CIFAR-10 images (from
$a\in\\{5,10,15,25,50,100\\},b\in\\{0.00001,0.0001,0.001,0.01,0.1\\}$). We
recommend the consideration of alternate smoothing functions, which could, for
instance, account for miscalibration in humans’ reported confidence.

Further, we compare the impact of learning with aggregated versus de-
aggregated participants’ predictions. In Section 5, we considered learning
with relabelings averaged across participants for a mixed image, and smoothed
with confidence reports averaged across participants. Here, we consider
instead separating out participants’ responses to learn with individual
relabelings smoothed by individual confidence, closely related to [Wei et al.,
2022]. We find in Table 4 that learning with de-aggregated data could
potentially offer greater performance gains. However, as [Wei et al., 2022]
discuss: whether to aggregate can depend on many factors. Our empirical
findings support the need for tailoring label construction in context.

Table 4: Varying whether to aggregate when using incorporating human confidence $\omega$ in label construction. Label Type | CE | FGSM | Calib
---|---|---|---
Ours (Avg with $\omega$) | 1.48$\pm$0.06 | 8.89$\pm$1.59 | 0.19$\pm$0.01
Ours (Separated with $\omega$) | 1.44$\pm$0.11 | 8.33$\pm$1.92 | 0.19$\pm$0.01

## Interfaces Included in HILL MixE Suite

We display sample pages of the interfaces created and used in this work, which
we release as part of HILL MixE Suite. Interfaces for Section 3 are shown in
Figs. 11 and 12; the interface used Sections 4 is depicted in Fig. 13.

Figure 11: Construct interface where participants press arrow keys to select
$\tilde{x}.$ Figure 12: Interface for the selection of a given $\lambda_{g}$
from a set of possible mixed images. Figure 13: Interface for inferring the
mixup generative label parameter and providing confidence in such inference.

## Alternative Synthetic Example Category Composition Elicitation

Given human participants are uncertain about the underlying mixing coefficient
in a number of cases, we consider whether the category composition typically
used in mixup – e.g., placing mass only on the labels of the images used to
form the synthetic combined sample – are reasonable. As demonstrated in the
main text and in Fig. 14, a synthetic mixup image may look like something else
entirely.

We therefore consider a follow-up small-scale human elicitation study wherein
we relax the mixup assumption that the label mixing function must output a
label constructed only from the two classes used to form the mixed image – and
instead collect $\tilde{y}$ directly by showing the mixed image to human
annotators in the form of soft labels. This provides a comparison to the
previous human-annotated endpoint label mixing coefficients, and can further
inspire useful designs for the label mixing policy.

Figure 14: Additional example soft labels elicited from individuals. Original
mixup label for each associated image is shown in red; the soft label elicited
from humans (averaged over two individuals) is shown in blue. The left and
center examples involve substantial discrepancies between human percepts and
the label which would be used in mixup; the rightmost image highlights that
some percepts do match the underlying mixing components (even without being
informed of the underlying classes). Examples are deliberately chosen to
illustrate the range of soft labels elicited; all examples are include in
H-Mix.

### Study Design

We recruit $N=8$ participants again from Prolific [Palan and Schitter, 2018],
yielding soft labels over a total of $100$ mixed images. Each participant saw
25 mixed images; each mixed image of the $100$ was seen by two participants.
The images are drawn from the same set of stimuli created in Section 4;
however, here, we only show images with a mixing coefficient
$\in\\{0.25,0.5,0.75\\}$. Participants are told that images are formed by
combining other images, and are asked to provide what they think others would
see in the image. Participants are asked to specify what others would view as
the most probable category with an associated percentage (on a scale of
0-100), an optional second most probable category with a probability, and any
categories that would be perceived as definitely not in the image. Again
employing the third-person viewpoint framing borrowed from [Chung et al.,
2019]. We rely on the soft label elicitation interface proposed in [Collins et
al., 2022a] and modify the instructions to be better suited combinations of
images. Following Collins et al., we construct “Top 2 Clamp” labels with a
redistribution factor of 0.1, which controls how we spread mass over any
categories still leftover as “possible” once accounting for those ruled out as
definitely not possible.

### Analyzing Elicited Soft Labels for Synthetic Images

We explore the correspondence between the elicited category compositions of
the mixed images with the labels that would be used to generate the mixed
image (as would be used in traditional mixup; i.e., placing mass only on two
categories). While participants did tend to place probability mass on the
generating endpoints that correlated with the mixing coefficient used (Pearson
$r=0.52$), interestingly, we find that participants report thinking that 38.3%
($\pm$0.6%) of the probability mass of a label should be placed on different
classes from those which are used to create the image. This is remarkable and
suggests that mixed images do not consistently look like the labels used to
create them, corroborate similar trends found in [Gruber et al., 2018] wherein
humans endorse categories which are not present in the image. Hence,
alternative labelings even beyond the kind we explore in the main text may be
preferred which are more aligned with human percepts. Examples of such labeled
mixed images are shown in Fig. 14 and the main text.

##### Takeaways

The typical two-category labels used in mixup do not consistently match human
perception. We find that human annotators often assign probabilities to
alternate classes when asked to label a mixed image. This suggests that the
pursuit of aligning synthetic data labeling to match human perception, at
least for the synthetic data constructor used in mixup, warrants the design of
alternative label mixing functions $g_{\text{rich}}$ which yield richer label
distributions over a broader range of categories.