# A note on invariable generation of nonsolvable permutation groups

Joachim König  Department of Mathematics Education, Korea National University
of Education, Cheongju, South Korea<EMAIL_ADDRESS>and  Gicheol Shin
Department of Mathematics Education, Seowon University, Cheongju, South Korea
<EMAIL_ADDRESS>
###### Abstract.

We prove a result on the asymptotic proportion of randomly chosen pairs
$(\sigma,\tau)$ of permutations in the symmetric group $S_{n}$ which
“invariably” generate a nonsolvable subgroup, i.e., whose cycle structures
cannot possibly both occur in the same solvable subgroup of $S_{n}$. As an
application, we obtain that for a large degree “random” integer polynomial
$f$, reduction modulo two different primes can be expected to suffice to prove
the nonsolvability of $\textrm{Gal}(f/\mathbb{Q})$.

###### Key words and phrases:

Combinatorics; random permutations; permutation groups; solvable groups;
Galois groups.

###### 2010 Mathematics Subject Classification:

Primary 05A05, 20B35; Secondary 11R32.

##  1\. Introduction and main result

This paper is motivated by the following, at this point deliberately vaguely
worded question:

######  Question 1.1.

Given a “random” integer polynomial $f\in\mathbb{Z}[X]$, how “difficult” is it
to verify the nonsolvability of the Galois group
$\textrm{Gal}(f/\mathbb{\mathbb{Q}})$?

To begin with, the precise meaning of “random polynomial” is up to the choice
of a concrete model. For our purposes, we essentially mean “a degree $n$
polynomial whose Galois group is $A_{n}$ or $S_{n}$” (although of course
Question 1.1 is to be understood such that whoever aims at proving
nonsolvability is provided with no prior information on the Galois group of
the concrete polynomial), or slightly more general (and in order to allow a
notion of randomness) “a polynomial $f$ chosen from a family in such a way
that $\textrm{Gal}(f/\mathbb{Q})$ is almost certainly alternating or
symmetric”. The latter is known to hold for several versions of random
polynomials. For example, a famous result by van der Waerden ([16]) asserts
that for any fixed degree $n\in\mathbb{N}$ and for $N\to\infty$, the
proportion of polynomials with Galois group $S_{n}$ among degree-$n$ integer
polynomials with coefficients in $\\{-N,\dots,N\\}$ tends to $1$. More
recently, Bary-Soroker and Kozma showed in [1] that if $N$ is divisible by
four distinct primes, then for $n\to\infty$, the probability of a degree-$n$
polynomial with coefficients chosen uniformly at random from the set
$\\{1,\dots,N\\}$ to have Galois group containing $A_{n}$ tends to $1$. This
viewpoint (i.e., degree tending to infinity) will be more relevant for us in
view of Theorem 1.5 below.

Next, regarding the difficulty of verification mentioned in Question 1.1, we
only consider one approach to obtain information about the Galois group of an
integer polynomial, based on the following well-known criterion on modulo-$p$
reduction due to Dedekind.111While Dedekind’s criterion of course in general
cannot answer all questions about the Galois group, it is considerably less
expensive than other approaches such as resolvent methods.

######  Theorem 1.2 (Dedekind).

Let $f\in\mathbb{Z}[X]$ be a separable polynomial of degree $n$ with Galois
group $G=\textrm{Gal}(f/\mathbb{Q})\leq S_{n}$, let and $p$ be a prime
dividing neither the leading coefficient of $f$ nor the discriminant of $f$.
Then the Galois group of the mod-$p$ reduction $\overline{f}$ of $f$ (acting
on the roots of $\overline{f}$) embeds as a permutation group into $G$. In
particular, if $d_{1},\dots,d_{r}$ are the degrees of the irreducible factors
of $\overline{f}$ over $\mathbb{F}_{p}$, then $G$ contains an element whose
cycle lengths are exactly $d_{1},\dots,d_{r}$.

Conversely, given $f\in\mathbb{Z}[X]$ with Galois group $G\leq S_{n}$, the
existence of primes whose reduction yields any prescribed cycle structure
inside $G$ is guaranteed and quantified by Frobenius’ density theorem (later
famously strengthened by Chebotarev).

######  Theorem 1.3 (Frobenius).

Let $f\in\mathbb{Z}[X]$ be a separable degree-$n$ polynomial with Galois group
$G\leq S_{n}$, and let $C:=[c_{1},\dots,c_{r}]$ be a cycle type in $G$. Then
the asymptotic density of primes $p$ for which mod-$p$ reduction of $f$ in
Theorem 1.2 yields cycle type $C$ equals the proportion of elements of cycle
type $C$ in $G$.

As (a special case of) a well-known theorem of Jordan, a subgroup intersecting
all conjugacy classes of $S_{n}$ nontrivially must be $S_{n}$ itself. Thus,
Dedekind’s and Frobenius’ theorems guarantee that, in the case
$\textrm{Gal}(f/\mathbb{Q})=S_{n}$, the above reduction process will
eventually correctly identify the Galois group, if sufficiently many primes
are chosen. Efficient bounds on the number of primes required to succeed with
certainty are hard to obtain (and depend of course on the concrete polynomial
$f$), see, e.g., [10]. We instead ask how many primes are needed to succeed
“with high probability”.

The following definition is directly motivated by Frobenius’ theorem, which
yields, for each mod-$p$ reduction of a given polynomial $f$, not a concrete
element, but only a cycle type which is guaranteed to occur in the Galois
group of $f$.

######  Definition 1.4.

Let $G\leq S_{n}$, let $\mathcal{S}$ be a family of subgroups of $S_{n}$, and
$\sigma_{1},\dots,\sigma_{r}\in G$. Say that $\sigma_{1},\dots,\sigma_{r}$
invariably generate a member of $\mathcal{S}$, if for all
$\tau_{1},\dots,\tau_{r}\in S_{n}$ such that $\tau_{i}$ is conjugate in
$S_{n}$ to $\sigma_{i}$ for all $i=1,\dots,r$, the group generated by
$\tau_{1},\dots,\tau_{r}$ belongs to $\mathcal{S}$.

The dependency of the underlined parts in Definition 1.4 on $S_{n}$ rather
than $G$ is deliberate. Of course, switching them from $S_{n}$ to $G$ would
also be meaningful, and would indeed be common in a purely group theoretical
context. E.g., with this notion of “invariable generation”, it is known that
every non-abelian simple group $G$ has a pair of elements all of whose
$G$-conjugates generate $G$, see [9, Theorem 1.3], and that the expected
number of $G$-conjugacy classes (of an arbitrary group $G$) required to
invariably generate $G$ is in $O(\sqrt{|G|})$, see [11]. However, our version
seems natural in view of the Galois theoretical interpretation, since it
corresponds to a situation where no prior information on the true Galois group
$G\leq S_{n}$ is available.

For the key case $G=S_{n}$, invariable generation of transitive subgroups
(i.e., $\mathcal{S}$ equal to the set of all transitive subgroups of $S_{n}$),
or indeed of $S_{n}$ itself, has been studied by a number of authors. In
particular, the following two key questions have been studied in detail:
Firstly, what is the minimal number $r_{0}$ such that permutations
$\sigma_{1},\dots,\sigma_{r_{0}}\in S_{n}$, chosen independently and uniformly
at random from the set $S_{n}$ (for short: random permutations), invariably
generate a transitive subgroup (resp., invariably generate $S_{n}$) with
probability bounded away from $0$ as $n\to\infty$? 222Note that in this
question one cannot hope for a probability tending to $1$ for any fixed
$r_{0}$, since the proportion of elements of $S_{n}$ having a fixed point is
bounded away from $0$. Secondly, what is the average number of random
permutations needed to invariably generate a transitive subgroup (resp.,
$S_{n}$)? Dixon conjectured in [4] that the average number of random
permutations required to invariably generate a transitive subgroup should be
absolutely bounded from above as $n\to\infty$. This was first shown to be true
in [12]. Later, Pemantle, Peres and Rivin showed in [14] that in fact four
random permutations invariably generate a transitive subgroup, and even
$S_{n}$, with probability bounded away from $0$. On the other hand, for three
random permutations, the probability to invariably generate a transitive
subgroup is known to tend to $0$ as $n\to\infty$, by [6]. All these results
have natural interpretations in Galois theory via Dedekind’s criterion; e.g.,
one may expect to obtain an irreducibility certificate for a large degree
polynomial with Galois group $S_{n}$ via a bounded number of modulo-$p$
reductions; three reductions will almost certainly be insufficient, whereas
four give a non-negligible chance to succeed. Finally, the aforementioned [1]
may be seen as a more advanced connection between invariable generation and
Galois theory, combining results in the spirit of [14] with results on
factorization of polynomials with restricted coefficients to obtain a
“$100\%$” statement (in the limit $n\to\infty$) on the Galois side.

Motivated by Question 1.1, we are interested in the case where $\mathcal{S}$
in Definition 1.4 is the set of all nonsolvable subgroups of $S_{n}$.
Concretely, we prove the following.

######  Theorem 1.5.

Let $G=S_{n}$ or $G=A_{n}$. Then the proportion of pairs $(\sigma,\tau)\in
G^{2}$ which invariably generate a nonsolvable subgroup of $S_{n}$ is
$1-O(1/(\log n)^{2-\epsilon})$, for every $\epsilon>0$. In particular, two
random permutations $\sigma,\tau\in G$ invariably generate a nonsolvable
subgroup with probability tending to $1$ as $n\to\infty$.

Regarding Question 1.1, this implies that in order to find a certificate for
the nonsolvability of $\textrm{Gal}(f/\mathbb{Q})$ for a given large degree
integer polynomial $f$ with Galois group $S_{n}$ or $A_{n}$, it almost
certainly suffices to reduce modulo two different primes. Obviously this is
best possible, since a single mod-$p$ reduction will only yield one particular
cyclic subgroup of $\textrm{Gal}(f/\mathbb{Q})$, and thus cannot even rule out
this group to be cyclic.

The proof of Theorem 1.5, contained in Section 3, makes decisive use of a
recent combinatorial result by Unger ([15]); in terms of permutation group
theory, it requires only elementary prerequisites. In the appendix, we show
how to obtain a more concise version of the nonsolvability conclusion of
Theorem 1.5, which requires some more advanced classification results about
primitive groups.

##  2\. Prerequisites

We include some basic combinatorial and group-theoretical facts about
permutations which will be used later.

###  2.1. Permutations with partially prescribed cycle structure

Let $\lambda$ be a partition of $k$. For any permutation $\sigma\in S_{k}$ of
cycle type $\lambda$, we denote by $\lambda!$ the size of the centralizer
$C_{S_{k}}(\sigma)$ in $S_{k}$, i.e.,
$\lambda!=\prod_{i=1}^{\infty}{i^{m_{i}(\lambda)}m_{i}(\lambda)!}$, where
$m_{i}(\lambda)$ denotes the multiplicity of $i$ in $\lambda$. Recall that the
number of permutations of the cycle type corresponding to $\lambda$ is equal
to $\frac{k!}{\lambda!}$.

Let $\lambda$ be a partition of $k$ for some $k\leq n$. We denote by
$N_{\lambda,n}$ the number of permutations in $S_{n}$ which contain at least
$m_{1}(\lambda)$ $1$-cycles, $m_{2}(\lambda)$ $2$-cycles, $m_{3}(\lambda)$
$3$-cycles, and so on, i.e.,

$N_{\lambda,n}=\\#\\{\sigma\in S_{n}:\text{the multiplicity of $i$ in the
cycle type of $\sigma$ is $\geq m_{i}(\lambda)$ for each $i$}\\}.$

To compute (an upper bound of) the number $N_{\lambda,n}$, we can pick a
$k$-subset $X$ of $\\{1,2,\dots,n\\}$ in $\binom{n}{k}$ ways, then pick a
permutation on $X$ of cycle type $\lambda$ in $\frac{k!}{\lambda!}$ ways, and
then pick any permutation on $\\{1,2,\dots,n\\}\setminus X$ in $(n-k)!$ ways.
Thus, the total number of ways is equal to

$\binom{n}{k}\cdot\frac{k!}{\lambda!}\cdot(n-k)!=\frac{n!}{\lambda!}.$

In the above we have counted permutations of cycle type $\mu$ exactly
$\prod_{i=1}^{\infty}\binom{m_{i}(\mu)}{m_{i}(\lambda)}$ times, where $\mu$ is
a partition of $n$ such that $m_{i}(\mu)\geq m_{i}(\lambda)$ for each $i$.
Hence, we have $N_{\lambda,n}\leq\frac{n!}{\lambda!}$. Consequently, we have:

######  Lemma 2.1.

Let $m_{1},\dots,m_{n}$ be non-negative integers. The probability that a
random permutation in $S_{n}$ contains at least $m_{1}$ 1-cycles, $m_{2}$
2-cycles, $m_{3}$ 3-cycles, and so on, cannot exceed
$\prod_{i=1}^{n}\dfrac{1}{i^{m_{i}}m_{i}!}$. In particular, the probability
that a random permutation in $S_{n}$ contains at least one $k$-cycle is less
than or equal to $\frac{1}{k}$.

###  2.2. Primitive and imprimitive permutation groups

A transitive permutation group $G\leq S_{n}$ is called primitive if it does
not preserve a non-trivial block system in $\\{1,\dots,n\\}$, or equivalently,
if the point stabilizers are maximal subgroups in $G$. The following are
classical results about the structure of primitive permutation groups. The
first one is due to Jordan (see [5, Theorem 3.3E]), the second one was
essentially known to Galois (cf. [13]).

######  Theorem 2.2 (Jordan).

Let $G$ be a primitive permutation group of degree $n$, containing a cycle of
prime length fixing at least three points. Then $G\geq A_{n}$.

######  Theorem 2.3.

Let $G\leq S_{n}$ be a primitive and solvable group. Then $G$ acts as an
affine group on some vector space. In particular, $n$ is a prime power. If
furthermore $n=p$ is a prime, then $G$ is contained in the normalizer
$N_{S_{p}}(\langle\sigma\rangle)\cong C_{p}\rtimes C_{p-1}=:AGL_{1}(p)$, for
some $p$-cycle $\sigma$.

Recall furthermore that if $G\leq S_{n}$ acts transitively but imprimitively,
then there exist $a,b>1$ such that $n=ab$ and transitive groups $U\leq S_{a}$,
$V\leq S_{b}$ such that $G$ embeds into the wreath product $U\wr
V:=U^{b}\rtimes V$ (with $V$ acting via permuting the $b$ copies of $U$).
Furthermore, if $G_{1}<G$ denotes a point stabilizer, then there exists
$G_{1}<H_{1}<G$ such that the image of $G$ in the action on cosets of $H_{1}$
equals $V$, and the image of $H_{1}$ in the action on cosets of $G_{1}$ in
$H_{1}$ equals $U$. See, e.g., [5] for more background on primitive and
imprimitive permutation groups.

##  3\. Proof of Theorem 1.5

###### Proof.

Firstly, it obviously suffices to consider $G=S_{n}$. Since $A_{n}$ comprises
half the permutations of $S_{n}$, the assertion for $A_{n}$ follows readily.
We pick two permutations $\sigma,\tau\in S_{n}$ independently at random. We
will show that the following property of $\sigma,\tau$ holds with probability
$1-O((\frac{\log\log n}{\log n})^{2})$.

(1) $\text{For all }x\in S_{n},\text{ the subgroup
}\langle\sigma,\tau^{x}\rangle\leq S_{n}\text{ is nonsolvable}.$

We will achieve this through a series of claims. Note that whenever we
encounter a property fulfilled by a proportion of $O((\frac{\log\log n}{\log
n})^{2})$ pairs of permutations $(\sigma,\tau)\in S_{n}^{2}$ (or in
particular, a property fulfilled by a proportion of $O((\frac{\log\log n}{\log
n})^{2})$ permutations $\sigma\in S_{n}$), we may exclude the permutations
fulfilling this property from the further considerations, simply since the
union of a bounded number of sets of a certain asymptotic $O(f(n))$ is still
in $O(f(n))$.

Step I: We first claim that the following property holds for a proportion
$1-O((\frac{\log\log n}{\log n})^{2})$ of all pairs of permutations
$(\sigma,\tau)\in S_{n}^{2}$:

(2) $\begin{split}\text{ There exists a prime }p\in[\log(n)^{2},n/2]\text{
such that at least one of }\sigma,\tau\\\ \text{ contains a }p-\text{ cycle
and has all of its other cycle lengths coprime to }p.\end{split}$

Indeed, it suffices to show that a proportion $1-O(\frac{\log\log n}{\log n})$
of permutations $\sigma\in S_{n}$ fulfill the condition in (2). Without the
upper bound condition $p\leq n/2$, this is shown in Theorems 2 and 5 of [15].
To see that the upper bound may be added without changing the estimate on the
asymptotic proportion, note that due to Lemma 2.1, the probability for
$\sigma$ to contain a $p$-cycle for some prime $p$ larger than $n/2$ is at
most - and in fact in this special case, equal to (since the individual events
are mutually exclusive) $\sum\limits_{\stackrel{{\scriptstyle
p\in\mathbb{P}}}{{n/2<p\leq n}}}\frac{1}{p}$. Now it is well known that
$(\sum\limits_{\stackrel{{\scriptstyle p\in\mathbb{P}}}{{p\leq
n}}}\frac{1}{p})-\log\log n$ converges to a finite value $M$ (namely, the so-
called Meissel-Mertens constant) for $n\to\infty$. Therefore, as $n$ grows,
$\sum\limits_{\stackrel{{\scriptstyle p\in\mathbb{P}}}{{n/2<p\leq
n}}}\frac{1}{p}$ approaches $\log\log n-\log\log(\frac{n}{2})=\log\log
n-\log(\log n-\log(2))$. Since $\log(x)-\log(x-c)$ is of the order of growth
of $\frac{c}{x}$ (for $x\to\infty$ and $c$ constant), we obtain in total that
the probability that $\sigma$ contains a $p$-cycle for some prime
$p>\frac{n}{2}$ is in $O(\frac{1}{\log n})$, and a fortiori in
$O(\frac{\log\log n}{\log n})$. Thus, the requirement $p\leq n/2$ in Condition
(2) may be added without changing the asymptotic.

From now on, we thus may and will assume that Condition 2 holds (for one of
$\sigma$ and $\tau$, and for some prime $p\in[\log(n)^{2},n/2]$). Up to
renaming, we may and will assume from now on that $\sigma$ fulfills the
condition, and will furthermore denote the $p$-cycle of $\sigma$ by
$(1,2,\dots,p)$, which is of course also without loss of generality.

Step II: We next claim that the proportion of $\sigma\in S_{n}$ containing a
cycle of length a Mersenne prime $\geq\log(n)^{2}$ is
$O(\frac{1}{\log(n)^{2}})$, whence we may and will additionally assume in the
following that the prime $p$ in Condition (2) is not a Mersenne prime.

Indeed, we may again use Lemma 2.1 together with a simple geometric series
estimate, to bound the probability for $\sigma$ to contain a cycle of some
length $2^{k}-1\geq\log(n)^{2}$ from above by
$\sum\limits_{j=0}^{\infty}\frac{1}{\log(n)^{2}}\cdot(\frac{1}{2})^{j}$, which
is in $O(\frac{1}{\log(n)^{2}})$.

We now include the second permutation $\tau$ in our considerations, and (for
all possible choices of $x\in S_{n}$) look at the length of the orbit of the
subgroup $\langle\sigma,\tau^{x}\rangle$ containing the orbit
$\\{1,\dots,p\\}$ of $\langle\sigma\rangle$. We denote this orbit by $O$ and
from hereon consider the image $U$ of $\langle\sigma,\tau^{x}\rangle$ in its
action on $O$. Of course it suffices to show that for a proportion
$1-O((\frac{\log\log n}{\log n})^{2})$ of pairs $(\sigma,\tau)$, there exists
no $x$ such that $U$ is solvable.

Step III: We claim that, for any $\sigma$ as above, the probability (in
$\tau$) that there exists at least one $x\in S_{n}$ for which the above $U\leq
Sym(O)$ is solvable and primitive is $O(\frac{1}{\log(n)^{2}})$.

Indeed, due to Step I, $U$ contains a $p$-cycle (namely, a suitable power of
$\sigma$, restricted to $O$). Thus, from Theorem 2.2, if $U$ is primitive and
solvable, then certainly $|O|\in\\{p,p+1,p+2\\}$. The case $|O|=p+2$ can also
be excluded by elementary means.333Or indeed by non-elementary ones such as
the classification result in [8, Theorem 1.2]. Indeed, by Theorem 2.3, a
primitive solvable group is affine, i.e., contained in $AGL(V)\cong V\rtimes
GL(V)$ for some finite vector space $V$. The point stabilizers in $AGL(V)$ are
conjugate to (the point stabilizer of the zero vector) $GL(V)$, so if $U\leq
AGL(V)$ contains a cycle fixing exactly two points, then in particular $GL(V)$
contains an element fixing exactly one non-zero vector. But the latter can
only happen if $V$ is an $\mathbb{F}_{2}$-vector space, i.e., $|O|=p+2$ is a
$2$-power, which is impossible. Furthermore, the case $|O|=p+1$ can be
excluded via Step II, since if $U$ were primitive and solvable (and thus,
affine) of degree $p+1$, then $p+1$ would be a prime power, i.e., $p$ would be
a Mersenne prime. So we are left with the case that $U$ is primitive and
solvable of degree $p$, i.e., $U\leq AGL_{1}(p)$. But the only elements of
$AGL_{1}(p)$ are $p$-cycles and powers of a $(p-1)$-cycle, i.e., elements of
cycle type $[1,d,\dots,d]$ for $d$ dividing $p-1$. We need to bound the
proportion of $\tau\in S_{n}$ having such a pattern in their cycle structure.
Of course, since $p\geq\log(n)^{2}$, the probability for $\tau$ to contain a
$p$-cycle is $\leq\frac{1}{\log(n)^{2}}$ by Lemma 2.1. On the other hand, via
setting $a:=\frac{p-1}{d}$, the combined probability to contain $a$ cycles of
length $d$ for any $d|p-1$ is bounded from above by
$\sum\limits_{a|p-1}\frac{a^{a}}{(p-1)^{a}\cdot a!}$. We may now use the
inequality

$a!\geq\sqrt{2\pi}a^{a+1/2}\exp(-a),$

(a part of Stirling’s formula valid for all $a\in\mathbb{N}$), to obtain an
upper bound
$\frac{1}{\sqrt{2\pi}}\sum\limits_{a|p-1}\frac{\exp(a)}{(p-1)^{a}\sqrt{a}}\leq\sum_{a\geq
1}(\frac{e}{p-1})^{a}$. Due to $p\geq\log(n)^{2}$, the term for $a=1$ gives a
contribution $O(\frac{1}{\log(n)^{2}})$, whereas the remaining sum is strictly
smaller. This completes the proof of the claim of Step III.

We may therefore assume from now on that $\sigma$ and $\tau$ are such that,
for any $x\in S_{n}$, the group $U$ above is either nonsolvable or imprimitive
on $O$. To bound the proportion of pairs $(\sigma,\tau)$ for which $U$ is
solvable and imprimitive for at least one $x\in S_{n}$, we return to the
investigation of the cycle structure of $\sigma$.

Let $\\{O_{1},\dots,O_{k}\\}$ (with $\cup_{i=1}^{k}O_{i}=O$) be a block system
of minimal block length preserved by the corresponding group $U$. Then $U\leq
G\wr H=(\underbrace{G\times\dots\times G}_{k\text{ times}})\rtimes H$, where
$G\leq Sym(O_{1})$ is primitive (due to the minimality assumption), and $H\leq
S_{k}$ is transitive. Furthermore, recall that some power of $\sigma$ acts as
a $p$-cycle on $O$, and this $p$-cycle necessarily lies in one component $G$
(i.e., it permutes one block while fixing all the others pointwise). We assume
$U$ to be solvable, in which case $G$ is solvable (and in particular affine)
as well. But $G$ also contains a $p$-cycle, so $p<|O_{1}|-2$ is excluded by
Theorem 2.2, and $|O_{1}|\in\\{p+1,p+2\\}$ can be excluded as well, as already
shown in Step II.

We have thus obtained $|O_{1}|=p$, so $G\leq AGL_{1}(p)$. We choose the
ordering of the blocks $O_{i}$ such that $O_{1}\neq\\{1,\dots,p\\}$ Assume
without loss of generality that $\sigma$ permutes the $m$ blocks
$O_{1},\dots,O_{m}$ cyclically ($m\geq 1$), and denote the corresponding cycle
lengths on the order $mp$ set $O_{1}\cup\dots\cup O_{m}$ by $md_{1}$, $\dots$,
$md_{r}$ ($d_{i}\in\mathbb{N}$). Then $\sigma^{m}$ fixes the block $O_{1}$
setwise, and induces a permutation of cycle lengths $d_{1},\dots,d_{r}$ on
this block. But this permutation needs to be contained in $AGL_{1}(p)$, whence
either $r=1$ and $d_{1}=p$, or $d_{1}=1$ and $d_{2}=\dots=d_{r}=:d$. The first
scenario is impossible since $\sigma$ is assumed to contain only one $p$-cycle
$(1,\dots,p)$ and all other cycles of length coprime to $p$. Setting
$a:=\frac{p-1}{d}$, we are left with the case that $\sigma$ contains (among
others) cycles of the following lengths:

  * i)

One cycle of length $p$ for some prime $p\geq\log(n)^{2}$,

  * ii)

One cycle of length $m$, and $a$ cycles of length $\frac{m(p-1)}{a}$, for some
$a|p-1$ and $m\in\mathbb{N}$.

Step IV: We claim that the proportion of $\sigma\in S_{n}$ fulfilling i) and
ii) above simultaneously is in $O(\frac{1}{\log(n)^{2}})$, which due to the
above is enough to complete the proof.

Indeed, due to Lemma 2.1, the proportion of $\sigma\in S_{n}$ fulfilling both
i) and ii) is bounded from above by

$\sum\limits_{\stackrel{{\scriptstyle
p\in\mathbb{P}}}{{p\geq\log(n)^{2}}}}\sum\limits_{a|p-1}\sum\limits_{m\in\mathbb{N}}\frac{1}{pm\cdot(\frac{m(p-1)}{a})^{a}\cdot
a!}\leq\sum\sum\sum\frac{a^{a}}{(p-1)^{a+1}m^{a+1}\cdot a!}=$
$=\sum\limits_{\stackrel{{\scriptstyle
p\in\mathbb{P}}}{{p\geq\log(n)^{2}}}}\frac{1}{(p-1)^{2}}\sum\limits_{m\in\mathbb{N}}\frac{1}{m^{2}}+\sum\limits_{\stackrel{{\scriptstyle
p\in\mathbb{P}}}{{p\geq\log(n)^{2}}}}\sum\limits_{2\leq
a|p-1}\sum_{m\in\mathbb{N}}\frac{a^{a}}{(p-1)^{a+1}m^{a+1}a!}$

The first sum is obviously, up to constant factor $\leq\frac{\pi^{2}}{6}$,
bounded from above by
$\sum\limits_{k=\log(n)^{2}}^{\infty}\frac{1}{k^{2}}=O(\frac{1}{\log(n)^{2}})$.
To further bound the second sum, we once again use the inequality
$a!\geq\sqrt{2\pi}a^{a+1/2}\exp(-a)$ to obtain an upper bound

$\frac{1}{\sqrt{2\pi}}\sum\limits_{p\geq\log(n)^{2}}\sum_{2\leq
a|p-1}\sum\limits_{m}\frac{\exp(a)}{(p-1)^{a+1}m^{3}\sqrt{a}}\leq\frac{1}{\sqrt{2\pi}}\sum\limits_{m\geq
1}\frac{1}{m^{3}}\sum\limits_{p\geq\log(n)^{2}}\frac{\exp(2)}{(p-1)^{3}}\sum\limits_{a\leq
p}\frac{1}{\sqrt{a}}.$

The last sum over $a$ is bounded from above by $2\sqrt{p}$, so the double sum
over $p$ and $a$ is, up to constant factor, bounded from above by
$\sum\limits_{p\geq\log(n)^{2}}\frac{1}{(p-1)^{5/2}}=O(\frac{1}{(\log(n)^{2})^{3/2}})$.
Since the sum over $m$ is absolutely bounded, the whole expression is in
$O(\frac{1}{\log(n)^{3}})$, showing the claim and completing the proof. ∎

######  Remark 3.1.

If one only intents to show the “limit equals $1$” part of Theorem 1.5, a few
shortcuts in the above proof are possible. E.g., Step I may be shortened via
replacing [15] by [7] which directly asserts that “most” permutations
$\sigma\in S_{n}$ power to a cycle of prime length $p\in[\log n,(\log
n)^{\log\log n}]$; however, this comes at the cost of a worse asymptotic. Our
own asymptotic bound should also still be far from optimal (compare the
experimental data in the next section), mainly because the strong Condition
(2) is far from necessary to invariably generate a nonsolvable subgroup.

The assertion of Theorem 1.5 can easily be translated into a statement about
the mean value of the number of (independently chosen) random permutations
required to invariably generate a nonsolvable subgroup of $S_{n}$.

######  Corollary 3.2.

Let $\sigma_{1},\sigma_{2},\dots$ be independent random permutations in
$S_{n}$ or $A_{n}$, and let
$N_{n}:=\min\\{r\in\mathbb{N}\mid\sigma_{1},\dots,\sigma_{r}\text{ invariably
generate a nonsolvable group}\\}$. Then $\lim_{n\to\infty}E(N_{n})=2$.

###### Proof.

By Theorem 1.5, there exists $\epsilon_{n}$, tending to $0$ with $n\to\infty$,
such that $P(N_{n}=2)=1-\epsilon_{n}$. Furthermore, within the event
$N_{n}>2$, the mean value is clearly bounded from above by $E(N_{n})+2$
(simply by ignoring the first two permutations and seeking to invariably
generate a nonsolvable group with the remaining ones). Therefore we have

$E(N_{n})\leq 2\cdot(1-\epsilon_{n})+(2+E(N_{n}))\cdot\epsilon_{n},$

or in other words $E(N_{n})\leq\dfrac{2}{1-\epsilon_{n}}$, which tends to $2$
for $n\to\infty$. Conversely, $N_{n}$ cannot possibly be smaller than $2$,
which shows the assertion. ∎

##  4\. Some computational data and further directions

While Theorem 1.5 and Corollary 3.2 give assertions about the large degree
limit, they say nothing about small values of $n$. We include some
computational results on the number $N_{n}$ of random permutations required to
invariably generate a nonsolvable subgroup of $S_{n}$, for some small $n$.
Concretely, we calculated the probabilities $P(N_{n}=2)$. All computations
were performed using Magma ([3]).444Since the solvable subgroups of a group
such as $S_{25}$ are too numerous to be enumerated in full, the required cycle
structures of maximally solvable subgroups were generated iteratively, using
the obvious fact that such a subgroup must either be transitive or a direct
product of two maximally solvable subgroups of smaller degree. The values are
rounded to three decimal digits for convenience.

$G$ | $P(N_{n}=2)$ | $G$ | $P(N_{n}=2)$ | $G$ | $P(N_{n}=2)$
---|---|---|---|---|---
$S_{5}$ | 0.250 | $S_{12}$ | 0.607 | $S_{19}$ | 0.810
$S_{6}$ | 0.244 | $S_{13}$ | 0.660 | $S_{20}$ | 0.821
$S_{7}$ | 0.395 | $S_{14}$ | 0.700 | $S_{21}$ | 0.834
$S_{8}$ | 0.380 | $S_{15}$ | 0.723 | $S_{22}$ | 0.851
$S_{9}$ | 0.461 | $S_{16}$ | 0.730 | $S_{23}$ | 0.864
$S_{10}$ | 0.543 | $S_{17}$ | 0.764 | $S_{24}$ | 0.870
$S_{11}$ | 0.601 | $S_{18}$ | 0.788 | $S_{25}$ | 0.885
Table 1. Values for small symmetric groups Figure 1. Plot of points
$(n,(1-P(N_{n}=2))^{-1})$

These values actually seem to suggest a superlinear convergence of the
“exceptional” probability $1-P(N_{n}=2)$ (see Figure 1), although we emphasize
that the sample size is too small to draw definite conclusions. Note that
$1/n^{2}$ is a trivial lower bound for $1-P(N_{n}=2)$, since that is the
probability to draw two $n$-cycles, which do not even invariably generate a
non-cyclic subgroup. A much deeper result by Blackburn, Britnell and Wildon
([2, Theorem 1.6]) states that the probability for two random permutations in
$S_{n}$ to invariably generate a non-abelian subgroup is in fact still
$1-O(1/n^{2})$. It would be interesting to find out whether this (or a
similar) asymptotic bound might indeed still hold for the case of invariable
generation of non-solvable subgroups, although this should require a much more
detailed analysis of cycle structures in solvable groups.

Finally, here are a few thoughts on the case of groups $G$ other than $A_{n}$
or $S_{n}$. Obviously, a result such as Theorem 1.5 cannot be generalized to
arbitrary families of nonsolvable groups $G$, and notably, the probability for
two (or more) permutations of $G$ to invariably generate a nonsolvable
subgroup of $S_{n}$ (in the sense of Definition 1.4) depends on the given
permutation action. E.g., if $G$ acts in its regular permutation action, then
every cycle type of $G$ also occurs in the cyclic group of order $|G|$, i.e.,
for polynomials with such a Galois group, one can never obtain a
nonsolvability certificate by relying solely on Dedekind’s criterion. But even
for primitive groups $G$ this can happen; e.g., $G=A_{6}$ in its (degree $15$)
primitive action on $2$-sets contains only cycle structures which are also
contained in the solvable group $S_{3}\wr AGL_{1}(5)$. For certain other
groups, $P(N=2)>0$ is obvious, e.g., $PSL_{2}(p)\leq S_{p+1}$ in its natural
permutation action contains cycle structures $(p,1)$ and
$(\frac{p+1}{2},\frac{p+1}{2})$, which together invariably generate a doubly
transitive and (for $p\geq 5$) nonsolvable subgroup of $S_{p+1}$. It would be
interesting to know for which groups Dedekind’s criterion can detect
nonsolvability (i.e., $P(N=r)>0$ for some suitable $r$), resp., can detect it
in two steps (i.e., $P(N=2)>0$).

##  Appendix A A strengthening

The proof of Theorem 1.5 can be adapted without many difficulties to show the
following stronger result.

######  Theorem A.1.

Let $G=S_{n}$ or $G=A_{n}$. Then the proportion of random permutations
$\sigma,\tau\in G$ which invariably generate a subgroup of $S_{n}$ with at
least one (nonsolvable) alternating composition factor tends to $1$ as
$n\to\infty$.

To show the strengthening, we need the classification of primitive permutation
groups containing a cycle (e.g., [8, Theorem 1.2]).

######  Proposition A.2.

Let $G$ be a primitive group of degree $m>23$ not containing $A_{m}$. Assume
that there exists an integer $\ell$ such that $G$ contains an $\ell$-cycle.
Then one of the following holds:

  * 1)

$m=\ell$ prime, and $C_{\ell}\leq G\leq AGL_{1}(\ell)$,

  * 2)

$m=\ell=(q^{d}-1)/(q-1)$ for some $d\geq 2$ and prime power $q$; and
$PGL_{d}(q)\leq G\leq P\Gamma L_{d}(q)$,

  * 3)

$m=\ell+1=q^{d}$ for some $d\geq 1$ and prime power $q$; and $AGL_{d}(q)\leq
G\leq A\Gamma L_{d}(q)$,

  * 4)

$m=\ell+1$ with $\ell$ prime, and $PSL_{2}(\ell)\leq G\leq PGL_{2}(\ell)$; or

  * 5)

$m=\ell+2=q+1$ for some prime power $q$; and $PGL_{2}(q)\leq G\leq P\Gamma
L_{2}(q)$.

We now show how to adapt the proof of Theorem 1.5 to obtain Theorem A.1.

###### Proof of Theorem A.1.

Choose a prime $p(\geq\log(n)^{2}$) as in Step I of the proof of Theorem 1.5.
I.e., the permutation $\sigma$ may be assumed to power to a $p$-cycle. Note
next that in Cases 3) and 5) of Proposition A.2 (with $\ell=p$), $p$ is
necessarily a Mersenne prime, which is excluded in Step II of the proof. In
Case 2) of Proposition A.2, if $d=2$, then $p$ is necessarily a Fermat prime,
which can be excluded in the same way; on the other hand, the sum of all
reciprocals of numbers $(q^{d}-1)/(q-1)$ with $d\geq 3$ and $q$ a prime power,
is convergent (e.g., bounded from above by the some of all reciprocals of
proper powers), hence for sufficiently large $n$, the sum of reciprocals of
all such numbers which are additionally $\geq\log(n)^{2}$ becomes arbitrarily
small. We may therefore alter Step II of the proof of Theorem 1.5 to
additionally demand that $p$ is no such number. In particular, for the groups
$U$ in Step III and $G$ in Step IV of the proof (which are primitive groups
containing a $p$-cycle by construction), Proposition A.2 then only leaves the
possibility to be alternating or symmetric of degree $\geq p$, or to be
contained in $AGL_{1}(p)$ (already dealt with) or in $PGL_{2}(p)$ in its
primitive action on $p+1$ points. But the cycle structures in the latter are
similarly restricted as in $AGL_{1}(p)$: they are all either $(p,1)$, or
$(d_{1},\dots,d_{1})$, or $(d_{2},\dots,d_{2},1,1)$ with divisors $d_{1}$ of
$p+1$ and $d_{2}$ of $p-1$. Now using these cycle structures, Step III of the
proof may be carried out essentially as above, to show that with probability
tending to $1$, $U$ must be alternating, symmetric or imprimitive. Finally, it
remains to adapt Step IV to bound the probability for $\sigma$ to have both a
$p$-cycle for some prime $p\geq\log(n)^{2}$, and $\frac{p+1}{d_{1}}$ cycles of
the same length $md_{1}$ for some divisor $d_{1}$ of $p+1$ and some
$m\in\mathbb{N}$.555This corresponds to the cycle type $(d_{1},\dots,d_{1})$
of $PGL_{2}(p)$ above; the estimate for the cycle types
$(d_{2},\dots,d_{2},1,1)$ is analogous, and indeed easier. This yields an
upper bound essentially as in Step IV above, except that we lose one factor
$\frac{1}{m}$. The reader may verify directly that the only somewhat
“critical” case is then $d_{1}=p+1$ (i.e., $\sigma$ has a $p$-cycle as well as
one single cycle of length divisible by $p+1$), which gives a contribution of
at most

$\sum\limits_{\stackrel{{\scriptstyle
p\in\mathbb{P}}}{{p\geq\log(n)^{2}}}}\frac{1}{p(p+1)}\sum_{m=1}^{n}\frac{1}{m}\in
O(\frac{1}{\log(n)^{2}}\cdot\log(n))=O(1/\log(n)),$

which converges to $0$. It therefore has been shown that with probability
tending to $1$, $\sigma$ and $\tau$ invariably generate a subgroup with at
least one alternating composition factor: namely either coming from a
homomorphic image $U$ acting primitively as the alternating or symmetric group
on some orbit; or from an imprimitive action of a homomorphic image $U$ in
which the block kernel projects onto an alternating or symmetric group. ∎

######  Remark A.3.

In the above proof, we have made no effort to retain the asymptotic of Theorem
1.5, although this could be achieved; e.g., in the last step, since the only
critical case is the one where there exists only a single cycle of length
$m(p+1)$, the imprimitive group $U$ must have a total number of $m+1$ blocks
of length $p+1$ (one containing the $p$-cycle, and $m$ further ones permuted
cyclically by $\sigma$). The image of $U$ in the blocks action is then a
transitive group of degree $m+1$ containing an $m$-cycle, hence even
$2$-transitive and thus covered by Proposition A.2. Using this, the set of
admissible values $m$ can be restricted sufficiently.

## References

  * [1] L. Bary-Soroker, G. Kozma, Irreducible polynomials of bounded height. Duke Math. J. Volume 169, Number 4 (2020), 579–598.
  * [2] S.R. Blackburn, J.R. Britnell, M. Wildon, The probability that a pair of elements of a finite group are conjugate. J. Lond. Math. Soc. 86, No. 3 (2012), 755–778.
  * [3] W. Bosma, J. Cannon, C. Playoust, The Magma algebra system. I. The user language. J. Symbolic Comput. 24(3-4) (1997), 235–265.
  * [4] J.D. Dixon, Random sets which invariably generate the symmetric group. Discr. Math. 105 (1992), 25–39.
  * [5] J.D. Dixon, B. Mortimer, Permutation Groups. Springer GTM 163 (1996).
  * [6] S. Eberhard, K. Ford, B. Green, Invariable generation of the symmetric group. Duke Math. J. 166:8 (2017), 1573–1590.
  * [7] S.P. Glasby, C.E. Praeger, W.R. Unger, Most permutations power to a cycle of small prime length. Preprint (2019). https://arxiv.org/abs/1911.12613
  * [8] G.A. Jones, Primitive permutation groups containing a cycle. Bull. Austr. Math. Soc. 89 (1) (2014), 159–165.
  * [9] W.M. Kantor, A. Lubotzky, A. Shalev, Invariable generation and the Chebotarev invariant of a finite group. J. Algebra 348 (1) (2011), 302–314.
  * [10] J.C. Lagarias, H.L. Montgomery, A.M. Odlyzko, A bound for the least prime ideal in the Chebotarev Density Theorem. Invent. Math. 54 (1979), 271–296.
  * [11] A. Lucchini, The Chebotarev invariant of a finite group: A conjecture of Kowalski and Zywina. Proc. Amer. Math. Soc. 146 (2018), 4549–4562.
  * [12] T. Luczak, L. Pyber, On random generation of the symmetric group. Combin. Probab. Comput. 2:4 (1993), 505–512.
  * [13] P. Neumann, The concept of primitivity in group theory and the Second Memoir of Galois. Arch. Hist. Exact Sci. 60 (2006) 4, 379–429.
  * [14] R. Pemantle, Y. Peres, I. Rivin, Four random permutations conjugated by an adversary generate $S_{n}$ with high probability. Rand. Struct. Alg., vol. 49 (2016), 409–428.
  * [15] W.R. Unger, Almost all permutations power to a prime length cycle. Preprint (2019).
https://arxiv.org/pdf/1905.08936.pdf.

  * [16] B.L. van der Waerden, Die Seltenheit der reduziblen Gleichungen und der Gleichungen mit Affekt. Monatsh. Math. Phys., 43(1) (1936), 133–147.


# Balancing the Picture: Debiasing Vision-Language Datasets with Synthetic
Contrast Sets

Brandon Smith
&Miguel Farinha11footnotemark: 1
&Siobhan Mackenzie Hall
Hannah Rose Kirk†
&Aleksandar Shtedritski†
&Max Bain†
Oxford Artificial Intelligence Society, University of Oxford
https://github.com/oxai/debias-gensynth Joint first authorship. †Joint senior
authorship.

###### Abstract

Vision-language models are growing in popularity and public visibility to
generate, edit, and caption images at scale; but their outputs can perpetuate
and amplify societal biases learned during pre-training on uncurated image-
text pairs from the internet. Although debiasing methods have been proposed,
we argue that these measurements of model bias lack validity due to dataset
bias. We demonstrate there are spurious correlations in COCO Captions, the
most commonly used dataset for evaluating bias, between background context and
the gender of people in-situ. This is problematic because commonly-used bias
metrics (such as Bias@K) rely on per-gender base rates. To address this issue,
we propose a novel dataset debiasing pipeline to augment the COCO dataset with
synthetic, gender-balanced contrast sets, where only the gender of the subject
is edited and the background is fixed. However, existing image editing methods
have limitations and sometimes produce low-quality images; so, we introduce a
method to automatically filter the generated images based on their similarity
to real images. Using our balanced synthetic contrast sets, we benchmark bias
in multiple CLIP-based models, demonstrating how metrics are skewed by
imbalance in the original COCO images. Our results indicate that the proposed
approach improves the validity of the evaluation, ultimately contributing to
more realistic understanding of bias in vision-language models.

##  1 Introduction

Vision-Language Models (VLMs) are rapidly advancing in capability and have
witnessed a dramatic growth in public visibility: DALL-E [46] has more than
1.5 million users creating over 2 million images a day; the discord channel
for MidJourney [41] hosts over two million members [49]; and shortly after its
release, Stability.AI reported that their Stable Diffusion model [47] had over
10 million daily active users [20]. Underpinning these powerful generative
models are image-text encoders like CLIP [44], which are themselves used for
many discriminative tasks, such as video action recognition, open set
detection and segmentation, and captioning. These encoders are pre-trained on
large-scale internet scraped datasets. The uncurated nature of such datasets
can translate to generated images that risk inflicting a range of downstream
harms on their end users and society at large – from bias and negative
stereotypes, to nudity and sexual content, or violent or graphic imagery [7,
14].

In light of these issues, coupled with growing use of generative AI, it is
vital to reliably benchmark the bias in VLMs, particularly in the image-text
encoders. A small emerging body of work attempts to measure bias in VLMs [1,
5, 15], or to debias their feature representations [5, 15]. Yet the legitimacy
of this work critically depends on both a suitable evaluation metric and an
evaluation dataset to accurately depict the bias in pre-trained model weights
and reliably signal whether debiasing attempts have been successful. The
predominant focus on model-centric debiasing methods has overshadowed two main
challenges associated with datasets and metrics: (i) the common use of cropped
face datasets, such as FairFace [30], fall short because excluding contextual
background presents an inaccurate and unreliable assessment of bias in
naturalistic images; and (ii) even if natural, open-domain images containing
contextual clues are used, they are unbalanced by identity attribute
representation within contexts. This is problematic because commonly-used bias
metrics, such as Bias@K, are affected by the naturally-occurring distribution
of images. Thus, while using contextual images is desirable, it comes at the
cost of spurious correlations, affecting the reliability of bias metrics.

In this paper, we argue that these confounding factors arising from the
interaction of metric choice and biased datasets paint an unreliable picture
when measuring model bias in VLMs. To counter these issues, we propose a
synthetic pipeline for debiasing a dataset into contrast sets balanced by
identity attributes across background contexts. Our pipeline draws on the
success of contrast sets in NLPs [22] and leverages recent advances in
controllable image editing and generation [9]. We illustrate our approach with
a focus on gender bias and define a contrast set as containing pairs of images
from COCO [13] where each image ID has two synthetically-edited versions (one
man, one woman) where the background is fixed and only the person bounding box
is edited. Our paper makes three key contributions: (1) We demonstrate
spurious correlations in the COCO dataset between gender and context, and show
their problematic effects when used to measure model bias (Sec. 3); (2) We
present the GenSynth dataset, built from a generative pipeline for synthetic
image editing, and a filtering pipeline using KNN with real and synthetic
images to control for the quality of the generated images (Sec. 4); (3) We
benchmark state-of-the-art VLM models [44, 63, 5, 28]; demonstrating how
balanced and unbalanced versions of the COCO dataset skew the values of bias
metrics (Sec. 5).

Our findings demonstrate that debiasing datasets with synthetic contrast sets
can avoid spurious correlations and more reliably measure model bias. While
synthetically-edited data has promise in (i) preserving privacy of subjects
included in vision datasets, and (ii) adding controllability to the dataset
features, it also risks introducing a real-synthetic distribution shift and
stacking biases of various generative models may essentialise representations
of gender (see Sec. 6). Despite these early-stage limitations, this work
starts a conversation about the importance of the interaction between dataset
features with bias metrics, ultimately contributing to future work that paints
a more accurate and balanced picture of identity-based bias in VLMs.

##  2 Related works

Defining Fairness and Bias. Fairness is a complex, context-dependent concept
[38, 59]. Here, we adopt a narrow definition where no group is advantaged or
disadvantaged based on the protected attribute of gender in retrieval settings
[25, 21]. The metrics employed in this paper, _Bias@K_ [61] and _Skew@K_ ,
[23] are used to assess disparity in distribution between search query results
and desired outcomes. In this work, we assume contextual activities such as
_dancing, skateboarding, laughing_ would not have a strong gendered prior and
thus the desired distribution is one where all protected attributes have equal
chance of being returned in a query that does not explicitly mention
gender.111In certain specific contexts, for example, pregnant or breastfeeding
women, we may not necessarily want an equal distribution of masculine and
feminine images to be returned, though we must be careful to not conflate
biological gender and gender identity (see Sec. 6).

Measuring Model Bias. Measuring bias in VLMs is a growing area of research.
Early work measures the misclassification rates of faces into harmful
categories [1]. Several works measure outcome bias for text-to-face retrieval
[5, 15, 53], though it is unclear how such measurements made on cropped face
datasets generalise to real-world settings. For gender fairness in open-domain
images, COCO Captions [13] is a standard benchmark for cross-modal retrieval
[61, 62] and image captioning [25, 68]. Measuring bias in generative VLMs has
also been approached [37].

Dataset Bias. Datasets, including those used for bias evaluation, have their
own biases from curation and annotation artefacts. Image datasets have been
found to include imbalanced demographic representation [10, 68, 64, 16, 56,
60], stereotypical portrayals [11, 52, 58], or graphic, sexually-explicit and
other harmful content [7]. Similar to [39, 60], we identify spurious gender
correlations in the COCO Captions dataset and further show this renders the
datasets unsuitable for current bias retrieval metrics. Techniques to reduce
dataset biases range from automatic [51] to manual filtering [65] of harmful
images, such as those containing nudity [51], toxicity, or personal and
identifiable information [3]. Yet, these filters cannot identify subtle
stereotypes and spurious correlations present in open-domain images – making
it difficult to curate a wholly unbiased natural image dataset [39].

Mitigating Dataset Bias with Synthetic Data. Deep networks need large amounts
of labeled data, prompting the creation of synthetic datasets for various
computer vision tasks [29, 40, 19, 54]. More recently, progress in generative
models [46, 48, 47] has enabled methods to synthetically generate training
data [9, 42, 33, 66]. Similarly, text-guided editing methods [27, 57, 9] offer
scalable and controllable image editing, potentially enhancing dataset
fairness and removing issues related to existing spurious correlations.
Several works propose the use of synthetic datasets for mitigating dataset
bias, such as with GANs [50] or diffusion models [21]. However, synthetic or
generated data may not necessarily represent underlying distributions of
marginalised groups within populations and thus still unfairly disadvantage
certain groups [2, 4, 6, 36]. To combat these risks, fairness in generative
models is an area gaining popularity: StyleGan [31] has been used to edit
images on a spectrum, rather than using binary categories [26]; [21] use human
feedback to guide diffusion models to generate diverse human images; and [32]
learn to transfer age, race and gender across images. Similar to our work,
GAN-based frameworks [45, 18] edit an existing face dataset to equalise
attributes and enforce fairness. Our work extends this approach to open-domain
images, introducing an automatic filtering technique for improving the quality
of edits. To our knowledge, we are the first to propose image editing of open-
domain images for fairness. Our work is also inspired by the use of contrast
sets in NLP [22], which have been used to alter data by perturbing
demographics (race, age, gender) in order to improve fairness [43]. We use
synthetically-generated contrast sets by augmenting both the textual and
visual input to CLIP, for a more accurate evaluation of VLM bias.

##  3 Measuring Gender Bias on Natural Images

While prior works make in-depth comparisons between models, and even metrics
[5], there is a dearth of research investigating whether natural image
datasets, with their own biased and spurious correlations, are suitable
benchmarks to measure bias in VLMs. In this section, we investigate the extent
of dataset bias from spurious correlations in COCO (Sec. 3.3) and its effect
on reliably measuring model bias (Sec. 3.4).

###  3.1 Preliminaries

We first define the bias metrics and the framework used to measure model bias
on image-caption data.

Bias@K [61] measures the proportions of masculine and feminine images in the
retrievals of a search result with a gender-neutral text query. For an image
$I$, we define a function $g(I)=\mathrm{male}$ if there are only individuals
who appear as men in the image, and $g(I)=\mathrm{female}$ if there are only
individuals who appear as women. Given a set of _K_ retrieved images
$\mathcal{R}_{K}(q)$ for a query $q$, we count the images of apparent men and
women as:

$N_{\mathrm{male}}=\sum_{I\in\mathcal{R}_{K}(q)}\mathds{1}[g(I)=\mathrm{male}]\;\;\;\;\;\
\mathrm{and}\;\;\;\;\;\
N_{\mathrm{female}}=\sum_{I\in\mathcal{R}_{K}(q)}\mathds{1}[g(I)=\mathrm{female}].$

We define the gender bias metric as:

$\delta_{K}(q)=\begin{cases}0,&N_{\mathrm{male}}+N_{\mathrm{female}}=0\\\
\frac{N_{\mathrm{male}}-N_{\mathrm{female}}}{N_{\mathrm{male}}+N_{\mathrm{female}}},&\text{otherwise}.\end{cases}$

For a whole query set $Q$, we define:

$\mathrm{Bias@K}=\frac{1}{|Q|}\sum_{q\in Q}\delta_{K}(q).$ (1)

Skew@K [5, 23] measures the difference between the desired proportion of image
attributes in $\mathcal{R}_{k}(q)$ for the query $q$ and the actual
proportion. Let the desired proportion of images with attribute label $A$ in
the set of retrieved images be $p_{d,q,A}\in[0,1]$ and the actual proportion
be $p_{\mathcal{R}(q),q,A}\in[0,1]$. The resulting Skew@K of $\mathcal{R}(q)$
for an attribute label $A\in\mathcal{A}$ is:

$\mathrm{Skew@K}(\mathcal{R}(q))=\ln{\frac{p_{\mathcal{R}_{K}(q),q,A}}{p_{d,q,A}}},$
(2)

where the desired proportion $p_{d,q,A}$ is the actual attribute distribution
over the entire dataset. A disadvantage of Skew@K is that it only measures
bias with respect to a single attribute at a time and must be aggregated to
give a holistic view of the bias over all attributes. We follow [5] and take
the maximum Skew@K among all attribute labels $A$ of the images for a given
text query $q$:

$\mathrm{MaxSkew@K}(\mathcal{R}(q))=\max_{A_{i}\in\mathcal{A}}\mathrm{Skew}_{A_{i}}\mathrm{@K}(\mathcal{R}(q)),$
(3)

which gives us the “largest unfair advantage” [23] belonging to images within
a given attribute. In our work, a MaxSkew@K of 0 for the attribute gender and
a given text query $q$ implies that men and women are equally represented in
the retrieved set of $K$ images $\mathcal{R}_{K}(q)$. We ignore all images
with undefined attribute labels (in this case gender) when measuring
MaxSkew@K.

COCO is a dataset of 118k images with detection, segmentation and caption
annotations, covering 80 distinct categories, including people [34, 13]. Each
image has five captions written by different human annotators. COCO is
commonly used to measure gender bias in VLMs in tandem with the Bias@K metric
[61, 62, 15].

###  3.2 Gendered Captions and Images in COCO

The bias metrics defined in Sec. 3.1 require gender attribute labels for each
image and gender-neutral text queries, but these are not naturally present in
captioned image data such as COCO. We describe the steps to automatically
label gender for images and to neutralise gender information in captions.

#### Extracting Image Gender Labels from Captions.

We assign a gender label to each COCO image, following prior work [61]. For
each image, we concatenate all five captions into a single paragraph. If the
paragraph contains only feminine words and no masculine words, the image is
assigned a female label, and vice versa. If the paragraph contains words from
both or neither genders, it is labeled as undefined. The full list of gendered
words is detailed in the Appendix. Using this procedure, we implement the
function $g$ in Sec. 3.1. The COCO 2017 train set contains 118,287 images, of
which 30,541 (25.8%) are male, 11,781 (9.9%) are female, and 75,965 (64.2%)
are undefined. The COCO 2017 validation set contains 5,000 images, of which
1,275 (25.5%), are assigned male, 539 (10.8%) female, and 3,186 (63.7%)
undefined. This procedure gives high precision in the gender-pseudo label, as
any ambiguous samples are rejected. However, images may be incorrectly labeled
as undefined (lower recall) due to, for example, misspelling of the gendered
words in the human-annotated captions or omission of rarer gendered terms in
our keyword list.

#### Constructing Gender-Neutral Captions.

We construct gender-neutral captions by replacing gendered words with neutral
ones, e.g. “man” or “woman” become “person”, and the sentence “A man sleeping
with his cat next to him” becomes “A person sleeping with their car next to
them”. The full mapping of gender-neutral words and more examples of original
and neutralised captions are in the Appendix.

###  3.3 Identifying Spurious Correlations with Gender

Figure 1: t-SNE clusters ($M=20$) of gender-neutralised caption embeddings.
Each cluster is manually assigned a name, then coloured and labelled according
to its male over-representation factor. The male over-representation factor is
the difference between the percentage of male images in the particular cluster
and the percentage of male images overall in the dataset.

As reported above, COCO contains more than twice as many male images as it
does female ones. This will inevitably affect retrieval-based bias metrics, as
there will be more male images in the retrievals. One naïve way to fix this is
to undersample the male images in order to arrive at a Balanced COCO dataset.
However, ensuring equal distribution of demographic attributes does not
necessarily ensure the dataset is unbiased as a whole. Spurious correlations
can result in subsets of the data being highly correlated with certain
attributes. Here we explore whether for certain contexts in the COCO dataset,
e.g., skateboarding, one gender is over-represented. We take two approaches to
evidence these spurious correlations.

#### K-means Clusters with Caption Embeddings.

First, we find semantic clusters of captions and evaluate the gender balance
within them. For every image $I_{n}$, we embed its gender-neutralised captions
$C_{n}^{k}$, where $k=\\{1,\ldots,K\\}$ represents the $K$ captions of the
image, with RoBERTa [35] to get features $f_{n}^{k}$. We average the features
to get $f_{n}=\frac{1}{K}\sum_{k=1}^{K}f_{n}^{k}$. Next, we cluster the
features $f_{n},n=\\{1,\ldots,N\\}$ into $M=20$ clusters with K-Means.
Finally, for each cluster, we extract salient words using Latent Dirichlet
Allocation (LDA) and give a manually-defined cluster label. In Fig. 1 we show
a t-SNE representation of the discovered clusters, together with the degree of
male over-representation. We see that in sports-related concepts men are over-
represented, whereas in scenes in kitchens, bathrooms, streets, and parks,
women are over-represented. For a list of all discovered classes and salient
words according to LDA, refer to the Appendix.

#### Spurious Correlations Classifier.

Following [52], we investigate the presence of spurious correlations by
training classifiers to predict binary gender labels of images and captions
where the explicit gender information is removed for both training and
testing. Specifically, for the image classifier (ResNet-50) we replace all
person bounding boxes with black pixels; and for the caption classifier (BERT-
base) we use the gender-neutralised captions. The training and testing data is
COCO train and validation defined in Sec. 3.2 but with undefined images
dropped. On unseen data, the text-only classifier on gender-neutralised
captions achieves 78.0% AUC, and the image-only classifier on person-masked
images achieves 63.4% AUC. Given that a random chance model achieves 50% AUC
and an image classifier on unmasked images achieves 71.9% AUC, it is clear
that spurious background correlations in the image, as well as biases in the
caption, provide a significant signal to predict gender of the person in the
image even when there is no explicit gender information.

###  3.4 The Effect of Dataset Bias on Model Bias Measurement

The dataset used for bias evaluation significantly affects the model bias
measurement. This is exemplified by a theoretically fair model, which we
instantiate as a TF-IDF (Term Frequency - Inverse Document Frequency) ranking
model for caption-to-caption retrieval on gender-neutralised captions. Despite
being based on a simple numerical statistic of word occurrences, devoid of any
inherent gender bias, this model still exhibits non-zero bias when evaluated
on COCO captions. Our findings, reported in Tab. 1, include Bias@K and
MaxSkew@K measurements on COCO Val, compared against a random model and CLIP.
For Balanced COCO Val, all models register an approximate Bias@K of zero, a
consequence of the metric’s signed nature that tends to average towards zero
over many directions of spurious correlations on biased but balanced data.
Yet, for unbalanced data, Bias@K shifts towards the over-represented
attribute, making it an unsuitable metric for model bias measurement.
MaxSkew@K, despite being an absolute measure, is not exempt from these issues.
It still records large values for the theoretically fair model and the random
model, suggesting that the established framework may be inadequate for bias
measurement on natural image datasets that inherently possess their own
biases.

Table 1: Comparison of model gender bias for CLIP [44], a theoretically fair
model (TF-IDF on non-gendered words) and a random model, on the COCO
validation set under unbalanced and balanced (with standard deviation computed
over 5 runs) settings.

Model | COCO Val | COCO Val (Balanced)
---|---|---
Bias@K | MaxSkew@K | Bias@K | MaxSkew@K
K=5 | K=10 | K=25 | K=100 | K=5 | K=10 | K=25 | K=100
Random Model | 0.37 | 0.40 | 0.15 | 0.06 |  0.00±0.07 |  0.00±0.07 |  0.14±0.00 |  0.07±0.00
Fair Model (TF-IDF) | 0.22 | 0.24 | 0.29 | 0.22 |  -0.06±0.00 |  -0.08±0.00 |  0.25±0.00 |  0.18±0.00
CLIP | 0.20 | 0.23 | 0.28 | 0.23 |  -0.03±0.01 |  -0.06±0.01 |  0.24±0.00 |  0.19±0.01

##  4 GenSynth: A Synthetic Gender-Balanced Dataset using Contrast Sets

Figure 2: An overview of our pipeline for dataset debiasing across a target
attribute, in this case gender, ensuring equal demographic representation. A
source image containing a person is given as input to InstructPix2Pix along
with instructions to synthesise each attribute label. The resulting edits are
filtered for quality via K-Nearest Neighbour (KNN) thresholding to ensure
realistic-looking edits for each attribute label (male and female).

Given the limitations of measuring Bias@K and MaxSkew@K on natural images and
the spurious correlations in existing datasets, we propose a framework for
editing natural images into synthetic contrast sets that remove spurious
background correlations along the attribute of interest (see Fig. 2), and
apply the pipeline on COCO to obtain the GenSynth dataset (see Fig. 2). We
first synthetically edit the person in images to cover both gender labels with
fixed background context (Sec. 4.1), followed by automatic filtering that
ensures the quality and correctness of the edited persons (Sec. 4.2). Finally,
we verify the quality of the edited images and the filtering method (Sec.
4.3). While we implement this for the gender attribute, in practice, our
pipeline could be used to generate synthetic contrast sets for other identity
attributes, requiring only the availability of person bounding boxes for the
source images.

###  4.1 Synthetically Editing Images

Leveraging advancements in text-conditioned image generation and editing, we
use an instruction-based model, InstructPix2Pix [9], for editing objects in an
image – referred to as the source image – while keeping the background
unchanged. We edit source images from COCO that (i) contain only one person,
inferred from the number of person bounding boxes; and (ii) have a defined
gender label, as defined in Sec. 3.2. These restrictions remove ambiguity.
Next, we crop the image to the single person bounding box and feed it to
InstructPix2Pix [9] along with multiple edit instructions for each attribute
label (Tab. 2). The edited person is then replaced in the source image. By
only editing the appearance of the person in the image, we preserve the
background content and minimize distortion – empirically, we found editing the
entire source image rather than just the source person produced lower quality
edits with significant hallucination. For further implementation details,
refer to the Appendix.

Table 2: Templates used for prompt editing. Template | Instruction
---|---
Feminine | Masculine
Make this person more {} | feminine | masculine
Make this person look like a {} | woman | man
Turn this person into a {} | woman | man
Convert this into a {} | woman | man

###  4.2 Automatic Quality Filtering of Edited Images

The synthetic edits with InstructPix2Pix [9] can often be of low quality or
fail to edit the source person’s attribute into the target attribute. In order
to ensure the quality and gender accuracy of our synthetic image sets, we
introduce an automatic filtering method using K-Nearest Neighbor (KNN),
similar to [24] who use KNN to score GAN-generated images.

First, we embed a collection of (i) source person bounding boxes, denoted as
$R=\\{r_{1},r_{2},...,r_{n}\\}$, and (ii) synthetically-edited person bounding
boxes, denoted as $S=\\{s_{1},s_{2},...,s_{m}\\}$ using CLIP. For each
synthetic box $s_{i}$, we identify its K-nearest neighbors in this feature
space, denoted as $N_{s_{i}}=\mathrm{KNN}(s_{i},R\cup S)$ using the Euclidean
distance between the embeddings. If the proportion of real images within
$N_{s_{i}}$, denoted as $P_{R}(s_{i})$, and the proportion of images
corresponding to the target gender of $s_{i}$, denoted as $P_{G}(s_{i})$,
exceed predetermined thresholds $\tau_{R}$ and $\tau_{G}$ respectively, the
edited image $s_{i}$ is accepted:

$P_{R}(s_{i})=\frac{1}{K}\sum_{r\in N_{s_{i}}}\mathbbm{1}(r\in
R)\;\;\;\mathrm{and}\;\;\;P_{G}(s_{i})=\frac{1}{K}\sum_{r\in
N_{s_{i}}}\mathbbm{1}(\mathrm{gender}(r)=\mathrm{gender}(s_{i})),$ (4)

$\mathrm{accept}(s_{i})=\begin{cases}1&\text{if }P_{R}(s_{i})>\tau_{R}\text{
and }P_{G}(s_{i})>\tau_{G}\\\ 0&\text{otherwise}.\end{cases}$ (5)

This process ensures that the accepted images are of high quality and
accurately reflect the target gender change. We only retain images where the
entire set of edits per unique COCO ID has at least one accepted male and
female edit, then randomly select one edit for each gender from images that
pass the filter. For examples of edits at each decile of $\tau_{R}$, see the
Appendix.

###  4.3 Verifying the Quality of GenSynth

We evaluate the quality of the GenSynth dataset in two ways. First, to measure
the correctness of the targeted gender edit, we use CLIP to zero-shot classify
the gender of people in the images. Second, to measure the semantic similarity
of the edited image to the caption, we measure the text-to-image retrieval
performance of CLIP on the synthetic text-image captions. For this, we edit
the captions using the reverse procedure in Sec. 3.2 to reflect the gender of
the person in the edited image. Then, for each image $I_{i}$ in GenSynth,
where $i\in\\{1,2,\ldots,N\\}$, we have a set of $n$ captions $C_{i}^{j}$,
$j\in\\{1,2,\ldots,n\\}$. For each caption $C_{i}^{j}$, we perform a retrieval
operation from the COCO validation set combined with the query image $I_{i}$,
to find a set of $K$ images that most closely match the caption, according to
Euclidean distance of CLIP features. We denote this retrieved set as
$R_{i}^{j}(K)$. The retrieval performance is evaluated using Recall at $K$
(R@K), which is defined as
$R@K=\frac{1}{Nn}\sum_{i=1}^{N}\sum_{j=1}^{n}\mathbbm{1}(I_{i}\in
R_{i}^{j}(K)).$

We compare GenSynth, against (i) the original COCO 2017 dataset (train set) of
natural images containing persons; and (ii) a weak gender-editing baseline –
GenSwap. This baseline has the same unique COCO images as in GenSynth, but
only with edited faces – we replace the detected face in the COCO image with a
random face of the target gender from the FairFace dataset [30]. Additional
implementations of GenSwap are provided in the Appendix.

As shown in LABEL:tab:evaluate_gensynth, GenSynth leads to very similar zero-
shot classification and retrieval results to the original COCO images. The
filtering step significantly improves both metrics, successfully removing bad
edits. The weak baseline, GenSwap, consistently scores low, showing the
importance of an effective editing method.

Table 3: Dataset comparison between the original COCO dataset of natural person images and synthetically edited COCO from the GenSwap and GenSynth pipelines. We report the presence of Spurious Background (BG) Correlations, Zero-Shot (ZS) Gender Accuracy, and Text-to-Image Retrieval Recall@K (R@K) amongst COCO Val 5k images using CLIP. Unfilt. refers to the synthetic pipeline without automatic quality filtering. COCO-Person Dataset | # Images |  Edits per Image |  Spurious BG. Correlations |  ZS Gender Acc. (%) $\uparrow$ |  Text-to-Image Retrieval $\uparrow$
---|---|---|---|---|---
R@1 | R@5 | R@10
Original | 11,541 | - | ✓ | 93.6 | 30.9 | 54.4 | 64.9
GenSwap | 3,973 | 2 | ✗ | 67.9 | 19.0 | 39.8 | 50.4
GenSynth (unfilt.) | 11,541 | 16 | ✗ | 83.9 | 22.4 | 43.4 | 53.8
GenSynth | 3,973 | 2 | ✗ | 95.5 | 29.2 | 52.8 | 62.8

##  5 Benchmarking Vision-Language Models on Balanced and Unbalanced
Evaluation Sets

Here we evaluate original and debiased CLIP models on the datasets described
in Sec. 5.1. We only report MaxSkew@K results, as we showed in Sec. 3 that
Bias@K is not a reliable metric for evaluating model bias.

###  5.1 Evaluation Setup

We use the following three datasets for evaluation: GenSynth consists of 7,946
images that have been generated and filtered as discussed in Sec. 4. It
consists of 3,973 unique COCO images from the train set (62.6% of which were
originally male), with a male and female edit for each.
COCO${}_{\Strichmaxerl[1.1]}$ consists of 3,973 original (unedited) images
with the same unique COCO IDs as GenSynth. All images contain a single person,
whose gender can be identified from the caption.
COCO${}_{\Strichmaxerl[1.1]\mathrm{Bal}}$ consists of 2,970 unique images from
COCO${}_{\Strichmaxerl[1.1]}$, randomly sampled such that there is an equal
number of male and female images. We use 5 different random seeds and report
average results.

We evaluate the following models: (i) the original CLIP model [44]; (ii) CLIP-
clip [61], with $m=100$ clipped dimensions computed on COCO train 2017; (iii)
DebiasCLIP [5], which has been debiased on the FairFace dataset; and (iv)
OpenCLIP [28] models trained on LAOIN 400M and 2BN datasets [51]. We use the
ViT-B/32 variant for all models, except for DebiasCLIP, for which ViT-B/16 is
used due to its availability from the authors.

###  5.2 Results

In Tab. 4 we measure and compare the gender bias of CLIP-like models for the
three evaluated datasets defined in Sec. 5.1. Overall we find the MaxSkew@K
metric is robust when measured on balanced
(COCO${}_{\Strichmaxerl[1.1]\mathrm{Bal}}$) and unbalanced data
(COCO${}_{\Strichmaxerl[1.1]}$), likely due to the normalization factor that
considers label distribution of all the images in the dataset. CLIP-clip has
the lowest gender bias across all models – which is expected given its
targeted clipping of dimensions most correlated with gender – but comes at the
cost of zero-shot image classification accuracy (60.1% on ImageNet1k [17]).
Interestingly, MaxSkew@K measured on GenSynth has much smaller variance
between models. Given that GenSynth removes spurious background correlations,
this suggests that a significant portion of reported model bias on natural
datasets may be due to spurious correlations related to gender rather than the
explicit gender of the person.

Table 4: Comparison of Gender Bias between CLIP-like models on COCO-Person datasets. We report the MaxSkew@K in caption-to-image retrieval of gender-neutralised captions. We compare CLIP [44], CLIP-clip [61], DebiasCLIP [5], and OpenCLIP [28] trained on LAOIN 400M & 2BN [51]. We additionally report zero-shot image classification accuracy on ImageNet1K [17]. COCO-Person Dataset | Model |  Gender Bias $\downarrow$ |  ImageNet1k Acc. (%) $\uparrow$
---|---|---|---
MaxSkew@25 | MaxSkew@100
COCO${}_{\Strichmaxerl[1.1]}$ | CLIP | 0.27 | 0.20 | 63.2
CLIP-clipm=100 | 0.23 | 0.16 | 60.1
DebiasCLIP | 0.29 | 0.22 | 67.6
OpenCLIP${}_{\text{400M}}$ | 0.26 | 0.20 | 62.9
|  OpenCLIP${}_{\text{2B}}$ | 0.27 | 0.21 | 65.6
COCO${}_{\Strichmaxerl[1.1]\mathrm{Bal}}$ | CLIP |  0.26±0.00 |  0.20±0.00 | 63.2
CLIP-clipm=100 |  0.22±0.00 |  0.15±0.00 | 60.1
DebiasCLIP |  0.28±0.01 |  0.21±0.00 | 67.6
OpenCLIP${}_{\text{400M}}$ |  0.27±0.00 |  0.20±0.00 | 62.9
|  OpenCLIP${}_{\text{2B}}$ |  0.27±0.00 |  0.21±0.00 | 65.6
GenSynth | CLIP | 0.23 | 0.18 | 63.2
CLIP-clipm=100 | 0.22 | 0.17 | 60.1
DebiasCLIP | 0.24 | 0.19 | 67.6
OpenCLIP${}_{\text{400M}}$ | 0.24 | 0.19 | 62.9
|  OpenCLIP${}_{\text{2B}}$ | 0.23 | 0.18 | 65.6

##  6 Limitations and Ethical Considerations

Synthetic Shifts. By generating synthetic data, we are creating a new
evaluation distribution that does not necessarily represent the real-world
distribution of the respective categories. This distribution shift can also be
forced in contexts where it does not necessarily make sense to either face
swap or make gender edits due to factual histories or biological identity [8].

Assumptions of Binary Gender. Our data relies on the binary gender labels from
the COCO and FairFace datasets. COCO also presents limitations regarding race,
ethnicity, and other sensitive attributes. We acknowledge this approach of
using binary gender and making reference to perceived gender based on
appearance oversimplifies the complexity of gender identity and biological
sex, and risks erasing representation of non-binary people. Despite attempts
to mitigate this limitation using terms such as “masculine” and “feminine”,
the resulting edits were often unusable (due to existing biases in generative
models), necessitating reliance on binary and narrow terms. We advocate for
future work that encodes and represents non-binary gender in datasets, and
improves generalisation in generative and predictive models to non-binary
terms.

Stacking Biases. Our pipeline uses a generative image editing model so may
inadvertently introduce biases from this model via stereotypical
representations of gender, e.g., if “make this person more feminine” over-
emphasises pink clothes, or “make this person more masculine” over-emphasises
beards. The automatic filtering step also tends to favour images with simple
scene arrangements. Some model-generated images were identified as NSFW, a
consequence of training on large-scale internet datasets [7]. Future work
could incorporate into our pipeline more capable and fair generative models.

##  7 Conclusion

The reliability of reported model biases in VLMs is affected by the
interaction between dataset bias and choice of bias metric. In this paper, we
demonstrated that naturalistic images from COCO have spurious correlations in
image context with gender, which in turn affects how much trust can be placed
in commonly-used metrics such as Bias@K: when measuring model bias, we may in
fact be measuring dataset bias. To mitigate these problems, we proposed a
pipeline for editing open-domain images at scale, creating gender-balanced
contrast sets where the semantic content of the image remains the same except
the person bounding box. Our method does not require manual auditing or image
curation, relying instead on an effective automatic filtering method. Using
this synthetically-created contrast set (GenSynth) we found that state-of-the-
art CLIP-like models measure similarly on gender bias suggesting that
measurements of model gender bias can largely be attributed to spurious model
associations with gender (such as scene or background information) rather than
gender itself. Through these subsequent angles of investigation, we conclude
that only focusing on model bias while ignoring how dataset artefacts affect
bias metrics paints an unreliable picture of identity-based bias in VLMs. We
hope our work contributes to an ongoing discussion of how to seek improved
representation and diversity of identity groups in image-captioning datasets,
both now and in the future.

Acknowledgements. This work has been supported by the Oxford Artificial
Intelligence student society, the Fundação para a Ciência e Tecnologia [Ph.D.
Grant 2022.12484.BD] (M.F.), the EPSRC Centre for Doctoral Training in
Autonomous Intelligent Machines & Systems [EP/S024050/1] (A.S.), and the
Economic and Social Research Council Grant for Digital Social Science
[ES/P000649/1] (H.R.K.). For computing resources, the authors are grateful for
support from Google Cloud and the CURe Programme under Google Brain Research,
as well as an AWS Responsible AI Grant.

## References

  * Agarwal et al. [2021] Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage.  Evaluating clip: towards characterization of broader capabilities and downstream implications.  _arXiv preprint arXiv:2108.02818_ , 2021.
  * Altman [2021] Erik Altman.  Synthesizing credit card transactions.  In _Proceedings of the Second ACM International Conference on AI in Finance_ , pages 1–9, 2021.
  * Asano et al. [2021] Yuki M Asano, Christian Rupprecht, Andrew Zisserman, and Andrea Vedaldi.  Pass: An imagenet replacement for self-supervised pretraining without humans.  2021\.
  * Belgodere et al. [2023] Brian Belgodere, Pierre Dognin, Adam Ivankay, Igor Melnyk, Youssef Mroueh, Aleksandra Mojsilovic, Jiri Navartil, Apoorva Nitsure, Inkit Padhi, Mattia Rigotti, et al.  Auditing and generating synthetic data with controllable trust trade-offs.  _arXiv preprint arXiv:2304.10819_ , 2023.
  * Berg et al. [2022] Hugo Berg, Siobhan Mackenzie Hall, Yash Bhalgat, Wonsuk Yang, Hannah Rose Kirk, Aleksandar Shtedritski, and Max Bain.  A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning.  _arXiv preprint arXiv:2203.11933_ , 2022.
  * Bhanot et al. [2021] Karan Bhanot, Miao Qi, John S Erickson, Isabelle Guyon, and Kristin P Bennett.  The problem of fairness in synthetic healthcare data.  _Entropy_ , 23(9):1165, 2021.
  * Birhane et al. [2021] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe.  Multimodal datasets: misogyny, pornography, and malignant stereotypes.  _arXiv preprint arXiv:2110.01963_ , 2021.
  * Blodgett et al. [2021] Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach.  Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets.  pages 1004–1015. Association for Computational Linguistics (ACL), 2021\.  ISBN 9781954085527.  doi: 10.18653/v1/2021.acl-long.81.
  * Brooks et al. [2022] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  _arXiv preprint arXiv:2211.09800_ , 2022.
  * Buolamwini and Gebru [2018] Joy Buolamwini and Timnit Gebru.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In _Conference on fairness, accountability and transparency_ , pages 77–91. PMLR, 2018.
  * Caliskan et al. [2017] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan.  Semantics derived automatically from language corpora contain human-like biases.  _Science_ , 356(6334):183–186, 2017.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9650–9660, 2021.
  * Chen et al. [2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ , 2015.
  * Cherti et al. [2022] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.  Reproducible scaling laws for contrastive language-image learning.  _arXiv preprint arXiv:2212.07143_ , 2022.
  * Chuang et al. [2023] Ching-Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka.  Debiasing vision-language models via biased prompts.  _arXiv preprint arXiv:2302.00070_ , 2023.
  * De Vries et al. [2019] Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten.  Does object recognition work for everyone?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 52–59, 2019.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Denton et al. [2019] Emily Denton, Ben Hutchinson, Margaret Mitchell, Timnit Gebru, and Andrew Zaldivar.  Image counterfactual sensitivity analysis for detecting unintended bias.  _arXiv preprint arXiv:1906.06439_ , 2019.
  * Dosovitskiy et al. [2015] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  pages 2758–2766, 2015.
  * Fatunde and Tse [2022] Mureji Fatunde and Crystal Tse.  Digital Media Firm Stability AI Raises Funds at $1 Billion Value.  _Bloomberg.com_ , October 2022.  URL https://www.bloomberg.com/news/articles/2022-10-17/digital-media-firm-stability-ai-raises-funds-at-1-billion-value.
  * Friedrich et al. [2023] Felix Friedrich, Patrick Schramowski, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Sasha Luccioni, and Kristian Kersting.  Fair diffusion: Instructing text-to-image generation models on fairness.  _arXiv preprint arXiv:2302.10893_ , 2023.
  * Gardner et al. [2020] Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al.  Evaluating models’ local decision boundaries via contrast sets.  _arXiv preprint arXiv:2004.02709_ , 2020.
  * Geyik et al. [2019] Sahin Cem Geyik, Stuart Ambler, and Krishnaram Kenthapadi.  Fairness-aware ranking in search & recommendation systems with application to linkedin talent search.  In _Proceedings of the 25th acm sigkdd international conference on knowledge discovery & data mining_, pages 2221–2231, 2019.
  * Gu et al. [2020] Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen.  Giqa: Generated image quality assessment.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16_ , pages 369–385. Springer, 2020.
  * Hendricks et al. [2018] Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach.  Women also snowboard: Overcoming bias in captioning models.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 771–787, 2018.
  * Hermes [2022] Isabal Hermes.  Gender representation in ai – part 1: Utilizing stylegan to explore gender directions in face image editing, 8 2022.  URL https://www.statworx.com/en/content-hub/blog/gender-representation-in-ai-part-1/.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Ilharco et al. [2021] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  Openclip, July 2021.  URL https://doi.org/10.5281/zenodo.5143773.
  * Johnson et al. [2017] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2901–2910, 2017.
  * Kärkkäinen and Joo [2021] Kimmo Kärkkäinen and Jungseock Joo.  Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation.  In _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2021.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4401–4410, 2019.
  * Kim et al. [2022] Yu Hwan Kim, Se Hyun Nam, Seung Baek Hong, and Kang Ryoung Park.  Gra-gan: Generative adversarial network for image style transfer of gender, race, and age.  _Expert Systems with Applications_ , 198, 7 2022.  ISSN 09574174.  doi: 10.1016/j.eswa.2022.116792.
  * Li et al. [2022] Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Sanja Fidler, and Antonio Torralba.  Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 21330–21340, 2022.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Lu et al. [2023] Yingzhou Lu, Huazheng Wang, and Wenqi Wei.  Machine learning for synthetic data generation: a review.  _arXiv preprint arXiv:2302.04062_ , 2023.
  * Luccioni et al. [2023] Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite.  Stable bias: Analyzing societal representations in diffusion models.  _arXiv preprint arXiv:2303.11408_ , 2023.
  * Mehrabi et al. [2021] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  _ACM Computing Surveys (CSUR)_ , 54(6):1–35, 2021\.
  * Meister et al. [2022] Nicole Meister, Dora Zhao, Angelina Wang, Vikram V Ramaswamy, Ruth Fong, and Olga Russakovsky.  Gender artifacts in visual datasets.  _arXiv preprint arXiv:2206.09191_ , 2022.
  * Michieli et al. [2020] Umberto Michieli, Matteo Biasetton, Gianluca Agresti, and Pietro Zanuttigh.  Adversarial learning and self-teaching techniques for domain adaptation in semantic segmentation.  _IEEE Transactions on Intelligent Vehicles_ , 5(3):508–518, 2020.
  * MidJourney [2023] MidJourney.  Home Page, May 2023.  URL https://www.midjourney.com/home/.
  * Peebles et al. [2022] William Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei A Efros, and Eli Shechtman.  Gan-supervised dense visual alignment.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13470–13481, 2022.
  * Qian et al. [2022] Rebecca Qian, Candace Ross, Jude Fernandes, Eric Smith, Douwe Kiela, and Adina Williams.  Perturbation augmentation for fairer nlp.  _arXiv preprint arXiv:2205.12586_ , 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR, 2021.
  * Ramaswamy et al. [2021] Vikram V Ramaswamy, Sunnie SY Kim, and Olga Russakovsky.  Fair attribute classification through latent space de-biasing.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9301–9310, 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Salkowitz [2022] Rob Salkowitz.  Midjourney Founder David Holz On The Impact Of AI On Art, Imagination And The Creative Economy.  _Forbes_ , September 2022.  URL https://www.forbes.com/sites/robsalkowitz/2022/09/16/midjourney-founder-david-holz-on-the-impact-of-ai-on-art//-imagination-and-the-creative-economy/.
  * Sattigeri et al. [2019] Prasanna Sattigeri, Samuel C Hoffman, Vijil Chenthamarakshan, and Kush R Varshney.  Fairness gan: Generating datasets with fairness properties using a generative adversarial network.  _IBM Journal of Research and Development_ , 63(4/5):3–1, 2019.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _arXiv preprint arXiv:2210.08402_ , 2022.
  * Schwemmer et al. [2020] Carsten Schwemmer, Carly Knight, Emily D Bello-Pardo, Stan Oklobdzija, Martijn Schoonvelde, and Jeffrey W Lockhart.  Diagnosing gender bias in image recognition systems.  _Socius_ , 6:2378023120967171, 2020.
  * Seth et al. [2023] Ashish Seth, Mayur Hemani, and Chirag Agarwal.  Dear: Debiasing vision-language models with additive residuals.  _arXiv preprint arXiv:2303.10431_ , 2023.
  * Song et al. [2017] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser.  Semantic scene completion from a single depth image.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1746–1754, 2017.
  * Thomee et al. [2016] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  _Commun. ACM_ , 59(2):64–73, 2016.  ISSN 0001-0782.
  * Torralba and Efros [2011] Antonio Torralba and Alexei A Efros.  Unbiased look at dataset bias.  In _CVPR 2011_ , pages 1521–1528. IEEE, 2011.
  * Tumanyan et al. [2022] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  _arXiv preprint arXiv:2211.12572_ , 2022.
  * van Miltenburg [2016] Emiel van Miltenburg.  Stereotyping and bias in the flickr30k dataset.  In _Proceedings of Multimodal Corpora: Computer vision and language processing (MMC 2016)_ , pages 1–4. 2016.
  * Verma and Rubin [2018] Sahil Verma and Julia Rubin.  Fairness definitions explained.  pages 1–7. IEEE Computer Society, 5 2018.  ISBN 9781450357463.  doi: 10.1145/3194770.3194776.
  * Wang and Russakovsky [2023] Angelina Wang and Olga Russakovsky.  Overcoming bias in pretrained models by manipulating the finetuning dataset.  _arXiv preprint arXiv:2303.06167_ , 2023.
  * Wang et al. [2021a] Jialu Wang, Yang Liu, and Xin Wang.  Are gender-neutral queries really gender-neutral? mitigating gender bias in image search.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1995–2008, 2021a.
  * Wang et al. [2022] Junyang Wang, Yi Zhang, and Jitao Sang.  Fairclip: Social bias elimination based on attribute prototype learning and representation neutralization.  _arXiv preprint arXiv:2210.14562_ , 2022.
  * Wang et al. [2021b] Mengmeng Wang, Jiazheng Xing, and Yong Liu.  Actionclip: A new paradigm for video action recognition.  _arXiv preprint arXiv:2109.08472_ , 2021b.
  * Wang et al. [2019] Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez.  Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5310–5319, 2019.
  * Yang et al. [2020] Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky.  Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy.  In _Proceedings of the 2020 conference on fairness, accountability, and transparency_ , pages 547–558, 2020.
  * Zhai and Wu [2018] Andrew Zhai and Hao-Yu Wu.  Classification is a strong baseline for deep metric learning.  _arXiv preprint arXiv:1811.12649_ , 2018.
  * Zhang et al. [2016] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao.  Joint face detection and alignment using multitask cascaded convolutional networks.  _IEEE signal processing letters_ , 23(10):1499–1503, 2016.
  * Zhao et al. [2021] Dora Zhao, Angelina Wang, and Olga Russakovsky.  Understanding and evaluating racial biases in image captioning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 14830–14840, 2021.

Appendix

##  Appendix A Implementation Details

Here we provide additional implementation details about our method.

###  A.1 Gendered Words and Caption Editing

In Tab. 5 we show the gendered words (Masculine, Feminine) that we use for
assigning each caption a gender label. Captions without either a masculine or
feminine word, or captions with matches from both of these lists are labeled
as undefined. For switching or neutralising the gender in a caption, we map
words across the rows of Tab. 5, so for example “she” could be replaced with
“he” or “they”. In Tab. 6 we show sentences that have been gender-neutralised.

Table 5:  Gendered word pairs. We the Masculine and Feminine words in order to classify the gender of a person in an image given its caption. When editing the gender of a caption or making it gender-neutral, we use the word from the corresponding pair for the opposite gender or the gender-neutral word, respectively. Masculine | Feminine | Neutral
---|---|---
man | woman | person
men | women | people
male | female | person
boy | girl | child
boys | girls | children
gentleman | lady | person
father | mother | parent
husband | wife | partner
boyfriend | girlfriend | partner
brother | sister | sibling
son | daughter | child
he | she | they
his | hers | their
him | her | them
Table 6: Examples of gender-neutralised captions. We show example original COCO captions with their gender-neutralised replacements, using the corresponding words from Tab. 5 Original |  Neutral
---|---
The woman brushes her teeth in the bathroom. |  The person brushes their teeth in the bathroom.
A man sleeping with his cat next to him. |  A person sleeping with their car next to them.
Two women and two girls in makeup and one is talking on a cellphone. |  Two people and two children in makeup and one is talking on a cellphone.

###  A.2 Image editing

Here we provide additional details on the two image editing pipelines in the
paper – our proposed method GenSynth, and the weak baseline GenSwap.

#### GenSynth

We edit the COCO train set images by applying Instruct-Pix2Pix [9] on person
crops (bounding boxes) with gender-editing instructions, as described in the
main paper. We run Instruct-Pix2Pix for 500 denoising steps, and for each
instruction, we generate an image with two text guiding scales: 9.5 and 15. We
found that a smaller guiding scale sometimes does not produce the required
edit, whereas too large a scale results in an image that does not look
natural. Using both scales ensures there are multiple candidates for the
edited image, and then we can use the filtering pipeline to discard bad edits.

#### GenSwap

We use the MTCNN face detector [67] to detect faces in the COCO images (for
the same subset in GenSynth), and replace them with faces from the FairFace
repository [30]. FairFace is a collection of face crops from the YFCC-100M
dataset [55], labeled with gender, race and age. We only use images whose age
attribute is greater than 19 and randomly sample a face crop from the target
gender.

###  A.3 Filtering

For the KNN filter, we set the neighbourhood size $K=50$, and the thresholds
$\tau_{R}=0.08$ and $\tau_{G}=0.5$.

##  Appendix B Spurious Correlations Analysis

In Tab. 7 we show the 20 discovered clusters using K-Means, together with the
top 10 salient words according to LDA. For each cluster, we show the male-
overrepresentation factor, i.e., the difference between the percentage of
images in that particular cluster relative to the percentage of male images in
the person class of COCO as a whole.

Table 7: Discovered clusters in COCO Captions. We show all 20 clusters with their manually assigned names, together with the top 10 words according to LDA. $\Delta$M represents the deviation from gender parity for males. Name | Words |  $\Delta$M (%)
---|---|---
diningdrinking | wine, glass, holding, scissors, table, sitting, bottle, drinking, pouring, standing | -5.7
diningsweets | cake, banana, donut, doughnut, holding, eating, candle, table, sitting, birthday | -14.0
diningmains | pizza, eating, table, food, sandwich, sitting, holding, slice, hot, dog | -10.3
sportstennis | tennis, court, racket, ball, player, racquet, hit, holding, swinging, playing | -6.0
sportssnow | ski, snow, slope, skiing, skier, snowboard, snowy, snowboarder, standing, hill | 4.7
sportsskateboarding | skateboard, skate, skateboarder, riding, trick, skateboarding, ramp, young, board, child | 27.9
sportsball | baseball, bat, player, ball, soccer, field, pitch, holding, game, pitcher | 24.0
sportskite,frisbee | frisbee, kite, playing, holding, field, beach, throwing, flying, standing, child | 11.6
sportssurfing | surfboard, wave, surf, surfer, riding, water, surfing, board, ocean, beach | 10.1
sportscycling,motorcycling | motorcycle, riding, bike, bicycle, street, sitting, next, standing, ride, motor | 10.5
leisurestreet | umbrella, holding, hydrant, standing, rain, fire, walking, street, child, black | -30.7
leisurepark | sitting, dog, bench, next, holding, park, child, two, sits, frisbee | -16.9
formal attire | tie, wearing, suit, standing, shirt, glass, shirt, black, white, young | 19.7
computer work | laptop, sitting, computer, bed, couch, desk, room, table, using, front | -4.6
animals | horse, elephant, giraffe, riding, cow, standing, sheep, next, two, brown | -2.9
video games | wii, game, remote, controller, playing, video, Nintendo, holding, room, standing | 4.8
kitchen | kitchen, food, standing, refrigerator, oven, cooking, counter, chef, preparing, holding | -16.2
bathroom | brushing, mirror, teeth, bathroom, cat, toothbrush, taking, toilet, holding, child | -14.0
travelling | standing, bear, teddy, luggage, train, next, street, bus, holding, suitcase | -6.7
phone calls | phone, cell, talking, holding, sitting, cellphone, standing, looking, wearing, young | -12.8

##  Appendix C Ablation Study

We ablate the use of a CLIP vision encoder in the KNN filtering pipeline. We
replace it with a DINO ViT-B/16 [12] and repeat the analysis. We found that
using DINO features is much more powerful when it comes to discriminating
between the different images (real versus fake), and that the male and female
images are better clustered. Accordingly, for the real vs. fake filter we use
a neighborhood size of $K=5{,}000$ and a threshold $\tau_{R}=0.0002$ (i.e.,
the generated images have at least _one_ real neighbour). For the male vs.
female filter, we use a neighborhood size of $K=50$ and a threshold
$\tau_{G}=0.4$. We end up with 571 unique COCO images, or 1,142 images in
total (with a male and female edit for each unique image). The R@K results
with this dataset are R@1 = 33.7%, R@5 = 57.1% and R@10 = 66.7%, and the zero-
shot gender classification accuracy is 87.4%. Due to the different filtering,
this dataset (with DINO filtering) is smaller than GenSynth and the results
have higher variance, but are comparable to GenSynth.

We evaluate MaxSkew@K on this dataset in Tab. 8. We observe a similar trend to
the GenSynth dataset, where bias results across models have a smaller variance
than results on the unbalanced and balanced COCO${}_{\Strichmaxerl[1.1]}$
datasets. The absolute values of the bias metric are smaller, which we explain
with the different images retrieved, and the variance that comes with that.

Table 8: Comparison of Gender Bias between CLIP-like models on the accepted images using DINO image embeddings for KNN filtering. We report the MaxSkew@K in caption-to-image retrieval of gender-neutralised captions. We compare CLIP [44], CLIP-clip [61], DebiasCLIP [5], and OpenCLIP [28] trained on LAOIN 400M & 2BN [51]. We additionally report zero-shot image classification accuracy on ImageNet1K [17]. COCO-Person Dataset | Model |  Gender Bias $\downarrow$ |  ImageNet1k Acc. (%) $\uparrow$
---|---|---|---
MaxSkew@25 | MaxSkew@100
GenSynth (DINO) | CLIP | 0.15 | 0.12 | 63.2
CLIP-clipm=100 | 0.13 | 0.10 | 60.1
DebiasCLIP | 0.15 | 0.12 | 67.6
OpenCLIP${}_{\text{400M}}$ | 0.15 | 0.12 | 62.9
|  OpenCLIP${}_{\text{2B}}$ | 0.14 | 0.11 | 65.6

##  Appendix D Qualitative Dataset Examples

In Fig. 3, we show gender edits for the GenSynth and GenSwap datasets,
alongside the original COCO image and ID. The GenSynth edits are more
naturalistic than the GenSwap edits, and also make changes to the body or
clothing of the subject.

Figure 3: Randomly selected examples of GenSynth images showing a comparison
to the original COCO image and the weak baseline GenSwap.

##  Appendix E Comparing Image Edits Across Filtering Thresholds

For each edited image, we calculate $P_{R}$, i.e., the ratio of real images
versus fake images in the KNN clustering step. We then average $P_{R}$ for
each pair of images (the male and female edit). In Fig. 4(a) and Fig. 5(a), we
show these randomly-selected pairs of gender edits from each decile of
averaged $P_{R}$ to demonstrate how our threshold filtering step improves the
quality of the edited images.

Figure 4: Averaged KNN Score ($P_{R}$) for pairs of edited images using the
GenSynth pipeline. (a) 1st to 4th decile of scores. (a) 5th to 8th decile of
scores. Note that there was only one image with an averaged score between
0.7-0.8, and no images in the higher deciles.