# Constraining the number of spacetime dimensions from GWTC-3 binary black
hole mergers

Ignacio Magaña Hernandez University of Wisconsin-Milwaukee, Milwaukee, WI
53201, USA

###### Abstract

In modified gravity models that allow for additional non-compact spacetime
dimensions, energy from gravitational waves can leak into these extra
spacetime dimensions, leading to a reduction in the amplitude of the observed
gravitational waves, and thus a source of potential systematics in the
inferred luminosity distances to gravitational wave sources. Since binary
black hole (BBH) mergers are standard sirens, we use the pair-instability
supernova (PISNe) mass gap and its predicted features to determine a mass
scale and thus be able to break the mass-redshift degeneracy. We
simultaneously fit for the BBH population and the extra spacetime dimensions
parameters from gravitational leakage models using BBH observations from the
recently released GWTC-3 catalog. We set constraints on the number of
spacetime dimensions and find that $D=3.95^{+0.09}_{-0.07}$ at $68\%$ C.L. for
models that are independent of a screening scale, finding that the GWTC-3
constraint is as competitive as that set from GW170817 and its electromagnetic
counterpart. For models where gravity leaks below a certain screening scale
$R_{c}$, we find $D=4.23^{+1.50}_{-0.57}$ and $\log_{10}R_{c}/{\rm
Mpc}=4.14^{+0.55}_{-0.86}$ with a transition steepness
$\log_{10}n=0.86^{+0.73}_{-0.84}$ for the leakage, which for the first are
constrained jointly with the BBH population at cosmological distances. These
constraints are consistent with General Relativity (GR) where gravitational
waves propagate in $D=3+1$ spacetime dimensions. Using the BBH population to
probe modifications to standard cosmological models provides an independent
test of GR that does not rely on any electromagnetic information but purely on
gravitational wave observations.

gravitational waves — cosmology: standard sirens — modified gravity

##  1 Introduction

With the recent release of the GWTC-3 catalog (Abbott et al., 2019b, 2021c,
2021d, 2021e) from the LIGO Scientific, Virgo and KAGRA Collaborations(LVK),
the number of gravitational wave detections from binary black hole mergers has
increased to 90 confidently detected events (Aasi et al., 2015; Acernese et
al., 2015). The increasing size of GW catalogs has enabled the study of the
BBH population (Abbott et al., 2020, 2021f), its cosmic expansion history
(Abbott et al., 2021b, g), signatures of gravitational wave lensing (Abbott et
al., 2021h) and how well the population agrees with General Relativity (Abbott
et al., 2021i, j).

Gravitational wave sources provide a direct measurement of their luminosity
distance—they are standard sirens (Schutz, 1986; Holz & Hughes, 2005).
Uniquely associated electromagnetic counterparts can constrain the redshift of
the source and hence allow for an independent determination of the
corresponding electromagnetic luminosity distance, as was the case for the
first bright standard siren GW170817 (Abbott et al., 2017a). Combined GW and
EM counterpart measurements have allowed constraints (to mention a few) of
cosmological parameters such as the Hubble constant (Abbott et al., 2017b),
the number of extra spacetime dimensions allowed under gravitational leakage
models (Pardo et al., 2018) and modified gravitational wave propagation due to
a running Planck mass (Lagos et al., 2019).

For gravitational wave events without expected EM counterparts (dark sirens),
including the numerous binary black hole mergers, in absence of a uniquely
identified host galaxy, a galaxy survey can be used as prior information on
the potential host galaxies of the event in combination with its gravitational
wave localization volume (Del Pozzo, 2012; Nair et al., 2018; Chen et al.,
2018; Fishbach et al., 2019; Gray et al., 2019; Soares-Santos et al., 2019;
Abbott et al., 2019a; Palmese et al., 2020; Mukherjee et al., 2021a; Diaz &
Mukherjee, 2021).

The dark siren methodology has been used to constrain cosmology (Abbott et
al., 2017b; Soares-Santos et al., 2019; Palmese et al., 2020; Abbott et al.,
2021g; Palmese et al., 2021) as well as to measure modified gravitational wave
propagation (Finke et al., 2021; Mukherjee et al., 2021b).

However, even without electromagnetic information, one can still make a
statistical measurement of redshift using the features of the population
distribution of compact binary mergers. Since we measure redshifted “detector-
frame” masses, $m_{1}^{\rm det}=m_{1}(1+z)$, one can model the expected source
frame mass distribution (given our current understanding on BBH formation
channels) on $m_{1}$ to estimate the redshift $z$. This idea was first
explored in Taylor et al. (2012) in the context of binary neutron star mergers
and 3G detectors to constrain cosmology, where the “known” galactic neutron
star mass distribution (a sharply peaked Gaussian with mean around 1.4
$M_{\odot}$) was used as a feature to break the mass-redshift degeneracy.

More recently, Farr (2019), applied this methodology to BBH mergers by using
the predicted cut off and excess of black holes with masses $M\approx
45M_{\odot}$ in the BH mass spectrum due to the pair-instability supernovae
(PISNe) mass gap (Woosley et al., 2002; Heger & Woosley, 2002; Heger et al.,
2003; Woosley, 2017, 2019). Recent work in Ezquiaga (2021), used the same
methodology to constrain the value of $c_{M}$ (in general relativity,
$c_{M}=0)$, a parameter that allows for modifications of $\Lambda$CDM due to a
time-varying Planck mass (Lagos et al., 2019) as well as other modified
gravity parameterizations (Mancarella et al., 2021). The LVK collaboration,
subsequently applied the methodology of Farr (2019) to the GWTC-3 catalog to
constraint the cosmic expansion history of BBH mergers using astrophysically
motivated source frame mass models (Abbott et al., 2021g).

In this work, we set constraints on gravitational leakage models,
specifically, on the number of extra non-compact spacetime dimensions where
gravity may leak, using the observed population of BBH mergers in GWTC-3. In
order to determine the redshift for these events, we fit the BBH population
with an astrophysically motivated PISNe mass model together with modifications
to the gravitational wave luminosity distance induced by the gravitational
leakage models we consider.

This paper is organized as follows. In Section 2 we describe the gravitational
leakage models and how these relate to the damping of the gravitational wave
amplitude. In Section 3, we summarize the hierarchical Bayesian framework used
in our analysis and provide the details for the BBH population model that we
use. In Section 4, we present the main results of this paper and in Section 5,
we provide a summary of this work. We use the Planck 2015 cosmological model
throughout this paper, that is, $H_{0}=67.8\ \rm km/s/Mpc$,
$\Omega_{m,0}=0.308$ and set $\Omega_{k,0}=0$.

##  2 Gravitational Leakage Models

In this section we describe the different gravitational leakage models and how
these relate to higher spacetime dimensional theories. We consider how these
models modify the gravitational waveform amplitude, and hence correspondingly
can bias the observed luminosity distance to the source. This section relies
heavily on the work of Deffayet & Menou (2007); Pardo et al. (2018); Corman et
al. (2021).

In General Relativity the gravitational wave strain is proportional to the
luminosity distance $d_{L}^{\rm GW}$ to the source as,

$h_{\rm{GR}}\propto\frac{1}{d_{L}^{\rm{GW}}},$ (1)

For a higher-dimensional spacetime theory where there is some leakage of
gravity, one would expect damping of the gravitational waveform in the form of
a power-law due to flux conservation, so the simplest phenomenological model
to consider is (Pardo et al., 2018; Corman et al., 2021):

$d_{L}^{\rm{GW}}=d_{L}\left(\frac{d_{L}}{\textrm{1 Mpc}}\right)^{(D-4)/2},$
(2)

where $D$ is the number of spacetime dimensions, $d_{L}$ is the
$D$-dimensional luminosity distance and $d_{L}^{\rm GW}$ is the measured GW
luminosity distance (from parameter estimation analyses).

However, one usually parameterizes this model so that below a certain length
scale the spacetime becomes 4-dimensional (Corman et al., 2021). So following
the standard phenomenology, we define a screening scale $R_{c}$ for the
gravitational leakage as well as the overall transition steepness $n$, which
determines the strength of the leakage. So more generally, following (Pardo et
al., 2018; Corman et al., 2021):

$d_{L}^{\rm{GW}}=d_{L}\left[1+\left(\frac{d_{L}}{R_{c}}\right)^{n}\right]^{(D-4)/(2n)},$
(3)

this relation reduces to model in Equation 2 for $d_{L}\gg R_{c}$ and to
$d_{L}^{\rm{GW}}=d_{L}$ for $d_{L}\ll R_{c}$.

To understand how the leakage model of Equation 3 affects the detectability of
a population of BBH mergers, we compute $p(\textrm{det}|z)$, the probability
of detecting a BBH at a given redshift $z$, since the corresponding luminosity
distance $d_{L}^{\rm{GW}}$ will change as a function of $D$, $R_{c}$ and $n$.
By fixing the BBH population to a fiducial set of parameters that correspond
to the “Power Law + Peak” model (see Sec. 3.2) and determining the values of
$d_{L}$ (given $d_{L}^{\rm{GW}}$) as a function of $D$ while fixing $R_{c}$
and $n$ constant, we show how the detectable fraction of mergers depends on
the number of spacetime dimensions in the top panel of Figure 1. At fixed
$R_{c}=100\rm\ Mpc$ and $R_{c}=1000\rm\ Mpc$ with $n=2$, and we note that as
$D$ increases from its GR value ($D=4$), the maximum detectable redshift as
well as its peak value decreases for the detectable population. This makes
sense, as energy from GW sources leaks for higher dimensional theories,
leading to a reduction in their measured amplitude and subsequently a lower
detectable horizon.

Similarly, we show the dependence of detectability as a function of $R_{c}$ in
the bottom panel of Figure 1, while we fix the spacetime dimensions to $D=5$
and consider $n=2$ and $n=50$ as examples for the transition steepness. As
$R_{c}$ increases, the maximum and peak redshift for the detectable
distribution increases, where for $R_{c}\approx 10^{4}\rm\ Mpc$ the model
resembles GR-like behavior for $D=5$ and is independent of the value for $n$.
If there is gravitational leakage, we expect the size of $R_{c}$ to be of
cosmological scales, since small screening scales ($R_{c}<20\rm/Mpc$) have
been ruled out by the analysis of GW170817 in Pardo et al. (2018) irrespective
of the value for $n$.

Figure 1: Top panel: Detectability of BBH mergers as a function of redshift
$z$ when varying the spacetime dimension $D$ at fixed $R_{c}$ and with $n=1$.
Solid lines have a screening scale $R_{c}=10^{2}\rm\ Mpc$, while dashed lines
have $R_{c}=10^{3}\rm\ Mpc$ for comparison. Bottom panel: Detectability as a
function of redshift given the screening scale $R_{c}$ at fixed spacetime
dimension $D=5$ and transition steepness $n=2$ (solid-lines) and $n=500$
(dashed-lines). We fix the population of BBH mergers to the “Power Law + Peak”
model with parameters $m_{\rm min}=5M_{\odot}$, $m_{\rm max}=87M_{\odot}$,
$\alpha=1.9$, $\beta=3.4$, $m_{\rm pp}=35M_{\odot}$, $\sigma_{\rm
pp}=2M_{\odot}$ and $f_{\rm pp}=0.03$ and to the SFR-like redshift evolution
model with parameters $\gamma=2.7$, $\kappa=-2$ and $z_{\rm peak}=2$.

Finally, it is also worth investigating theories in which the graviton has a
finite lifetime, meaning that it decays away as it travels cosmological
distances. In this case, the modified luminosity distance of the GW would
scale as (Pardo et al., 2018):

$d_{L}^{\rm{GW}}=d_{L}e^{d_{L}/R_{g}}$ (4)

where $R_{g}$ is the “decay-length”. Consequently, one can measure the
graviton’s “decay-time”’ as long as we assume that they propagate at the speed
of light, that is, $t_{g}=R_{g}/c$. We leave constraints from decaying
graviton models as future work.

##  3 Methods

###  3.1 Hierarchical Inference

We use hierarchical Bayesian inference to simultaneously infer the parameters
for the population distribution of binary black hole mergers as well as the
additional parameters that describe the gravitational leakage models described
in Section 2. The binary black hole observations provide us with an estimate
of their primary mass $m_{1}^{\text{det}}$ and mass ratio $q$ in the detector
frame, as well as their luminosity distances $d_{L}$.

The number density of BBH events as a function of detector frame quantities
$(m_{1}^{\text{det}},m_{2}^{\text{det}},d_{L})$ is related to the source frame
parameters $(m_{1},q,z)$ by,

$\frac{dN(m_{1}^{\text{det}},q,d_{L}|\Lambda)}{dm_{1}^{\text{det}}m_{2}^{\text{det}}dd_{L}}=\frac{1}{m_{1}(1+z)^{2}}\frac{dz}{dd_{L}}\frac{dN(m_{1},q,z|\Lambda)}{dm_{1}dqdz}$
(5)

with $\Lambda$ being the population hyperparameters that we want to measure
and the proportionality factor relating detector frame to source frame
quantities is the Jacobian transformation relating these parameterizations.

Now we relate the BBH number density in terms of the BBH merger rate density:

$\frac{dN(m_{1},q,z|\Lambda)}{dm_{1}dqdz}=\frac{dV_{c}}{dz}\bigg{(}\frac{T_{\mathrm{obs}}}{1+z}\bigg{)}\frac{d\mathcal{R}(m_{1},q,z|\Lambda)}{dm_{1}dq}$
(6)

where the BBH merger rate $d\mathcal{R}$ over a range of primary mass, mass
ratio and redshift (assuming the BBH mass distribution is redshift
independent) gives:

$\frac{d\mathcal{R}(m_{1},q,z|\Lambda)}{dm_{1}dq}=\mathcal{R}_{0}p(m_{1},q|\Lambda)p(z|\Lambda),$
(7)

here $\mathcal{R}_{0}$ is the local merger rate at $z=0$. The BBH population
is modeled through the normalized mass distribution
$p(m_{1},q|\Lambda)=p(m_{1}|\Lambda)p(q|m_{1},\Lambda)$ and its redshift
evolution $p(z|\Lambda)$, which is chosen such that $p(z=0|\Lambda)=1$. Here
$dV_{c}/dz$ is the differential uniform-in-comoving volume element,
$T_{\mathrm{obs}}$ the total observation time and the factor of $1/(1+z)$
converts source-frame time to detector-frame time. By integrating the BBH
number density across all primary masses and mass ratios, and out to a maximum
redshift $z_{\mathrm{max}}$ we get the expected number of BBH within
$z_{\mathrm{max}}$. In this work we take $z_{\mathrm{max}}=3$ throughout.

Given a set of $N_{\mathrm{obs}}$ gravitational wave observations
$\\{d_{i}\\}$, we can calculate the posterior on $\Lambda$ following e.g. Farr
(2019) and Mandel et al. (2019):

$\displaystyle\begin{split}p\left(\Lambda|\\{d_{i}\\}\right)&\propto
p(\Lambda)e^{-\mathcal{R}_{0}\xi(\Lambda)}\prod_{i=1}^{N_{\mathrm{obs}}}\Bigg{[}\int\mathcal{L}\left(d_{i}|m_{1}^{i},q^{i},z^{i}\right)\\\
&\times\frac{dN(m_{1},q,z|\Lambda)}{dm_{1}dqdz}dm_{1}dqdz\Bigg{]},\end{split}$
(8)

where $\mathcal{L}(d_{i}|m_{1},q,z)$ is the single-event likelihood function
for each event, and $\xi(\Lambda)$ is the detectable fraction of sources
corresponding to a population determined by the population hyperparameters
$\Lambda$.

Following Farr (2019) to estimate $\xi(\Lambda)$, we assume that sampling of
$\xi$ will follow a normal distribution (i.e.
$\xi(\Lambda)\sim\mathcal{N}(\Lambda|\mu,\sigma)$), with $\mu$ the importance
sample estimate of $\xi$ and $\sigma$ its associated uncertainty. Practically,
we estimate the detectable fraction $\xi(\Lambda)$, by using the LVK’s
injection campaign of BBH events simulated from a broad BBH population and
injected into real detector data, then searched for using the same analysis
pipelines that found GWTC-3 and prior GW Transient catalogs (Abbott et al.,
2021a).

Finally, we marginalize over the local BBH merger rate $\mathcal{R}_{0}$ using
a log-Uniform prior (Abbott et al., 2020), and neglect terms of
$\mathcal{O}(N_{\mathrm{eff}}^{-2})$ (Farr, 2019). We also approximate the
integral over the individual event likelihoods in Equation 8 with importance
sampling over $N_{i}$ single-event posterior samples generated from single
event inference analysis with default event prior $\pi(m_{1},q,z)\propto
d_{L}^{2}m_{1}(1+z)^{2}\frac{dd_{L}}{dz}.$ 111Note that we have absorbed the
Jacobian transformation in Equation 5 in the definition for $\pi(m_{1},q,z)$
as it is typically done.

###  3.2 Binary Black Hole Population Models

We model the black hole mass distribution in the source frame using the “Power
Law + Peak” model as described the LVK second gravitational-wave transient
catalog (GWTC-2) population analysis (Abbott et al., 2020, 2021f) and in
Talbot & Thrane (2018). It is a mixture distribution consisting of a power law
distribution truncated at a maximum mass to model the PISNe mass gap and of a
Gaussian distribution that models the build-up of black holes due to the
PPISNe (pulsational PISNe) mass loss (Talbot & Thrane, 2018). For simplicity,
we neglect the low-mass smoothing feature used in (Abbott et al., 2020).

Under this model, the probability distribution for the primary mass is,

$\displaystyle\begin{split}p(m_{1}|\Lambda)&=(1-f_{\rm{pp}})p(m_{1}|\alpha,m_{\rm
min},m_{\rm{max}})\\\
&+f_{\rm{pp}}p_{\rm{pp}}(m_{1}|m_{\rm{pp}},\sigma_{\rm{pp}})\,,\end{split}$
(9)

where $f_{\rm pp}$ is a mixing fraction parameter that gives the weight of the
Gaussian component (Abbott et al., 2020).

The power law distribution is defined as,

$p(m_{1}|\alpha,m_{\rm{max}})\propto{({m_{1}})}^{-\alpha}\,,$ (10)

where $\alpha$ is the powerlaw index and $m_{\rm{min}}$ ($m_{\rm{max}}$) is
the minimum (maximum) black-hole mass with the constraint that
$m_{\rm{min}}<m_{1}<m_{\rm{max}}$. The Gaussian component has a mean
$m_{\rm{pp}}$ and a standard deviation $\sigma_{\rm{pp}}$ and is given by,

$p(m_{1}|m_{\rm{pp}},\sigma_{\rm{pp}})\propto\exp\left[-\frac{({m_{1}}-m_{\rm{pp}})^{2}}{2\sigma_{\rm{pp}}^{2}}\right]\,.$
(11)

Following Abbott et al. (2020), we assume a powerlaw distribution for the mass
ratio ($q=m_{2}/m_{1}\leq 1$) (with powerlaw index $\beta$). So the
conditional probability distribution on $q$ given $m_{1}$ can be written as,

$p(q|m_{1},\beta,m_{\rm{min}})\propto q^{\beta}.$ (12)

and is defined in the range $m_{\rm min}/m_{1}<q<1$.

The redshift distribution for the binary black hole population is assumed to
be a distribution that closely resembles the star formation rate but it is
flexible enough to accommodate different shapes. Hence we follow the model
used in Callister et al. (2020); Ezquiaga (2021),

$p(z|\gamma,\kappa,z_{p})\propto\frac{(1+z)^{\gamma}}{1+\left(\frac{1+z}{1+z_{p}}\right)^{\gamma+\kappa}}$
(13)

which peaks at $z_{p}$ and where $\gamma$ controls the low redshift rise and
$\kappa$ the high redshift tail of the distribution.

##  4 Results

We present constraints on gravitational leakage models using GWTC-3 BBH
mergers. We use only the events that pass the 1 per-year IFAR threshold as was
done in Abbott et al. (2021a). In total we analyze 69 BBH mergers, and exclude
outlier BBH events such as GW190814 or any NSBH and BNS candidates.

Figure 2: Posterior distributions for the number of spacetime dimensions $D$
and the BBH population parameters most strongly correlated with $D$: the BBH
maximum mass $m_{\rm max}$, minimum mass $m_{\rm max}$, peak of the Gaussian
(PISN) component $m_{\rm pp}$ and the slope of the merger rate evolution
$\gamma$. Left panel: We show the posterior for the gravitational leakage
model defined in Equation 2. Right panel: Posterior distribution for the
leakage model with screening scale $R_{c}$ and transition steepness $n$ as
defined in 3.

First we show constraints for the gravitational leakage model that only
depends on the number of spacetime dimensions $D$ (see Equation 2) and so
allows for gravitational leakage at all scales. The posterior distribution on
the number of spacetime dimensions $D$ as well as the hyperparameters that
correlate strongly with $D$ are shown in the left-panel of Figure 2 (See
Appendix A for full posterior distributions). We find that we can place tight
constraints on the number of spacetime dimensions with BBH observations alone
to $D=3.95^{+0.09}_{-0.07}$ at $68\%$ C.L. The constraint presented here is as
competitive to the GW170817 constraint with its associated electromagnetic
counterpart of $D=3.98^{+0.07}_{-0.09}$ at $68\%$ C.L. (Pardo et al., 2018).
The BBH population hyperparameters are broadly consistent with constraints
placed in the GWTC-3 population analysis by the LVK Collaboration. However, we
do find a broadening and shift to higher mass for the allowed BBH maximum mass
$m_{\rm max}$. This is consistent with the analysis of (Ezquiaga & Holz,
2021), as the additional parameters in the modified luminosity distance models
due to gravitational leakage (or varying $c_{M}$) introduce uncertainty in the
value of $m_{\rm max}$.

We also show results for the model of Equation 3 that allows for a varying
screening scale and transition steepness. Posterior distributions on $D$ and
relevant population parameters are shown in the right hand side panel of
Figure 2. Under this model, we find we find $D=4.23^{+1.50}_{-0.57}$ at $68\%$
C.L., which gives broader constraints compared to the previous model (due to
the additional screening scale parameters) but are still consistent with GR.
We find that $D$ has strong correlations with the screening scale and
transition steepness as shown in Figure 5 which is also explained through the
detectability of the simulated population $p(\textrm{det}|z)$ shown in Figure
1. We note that, the BBH population hyperparameters under this model are
similarly constrained with respect to the model of Equation 2.

Since we have used binary black hole mergers across a wide range of distances,
we are also able to constrain the screening scale $R_{c}$ and the transition
steepness $n$ for the first time, without having to fix any of the three model
parameters as has been done in other studies (See Appendix A for full
posterior distributions). We show in Figure 3, the posterior distribution on
$R_{c}$ and $n$ and find that $\log_{10}R_{c}/{\rm Mpc}=4.14^{+0.55}_{-0.86}$
with a transition steepness $\log_{10}n=0.86^{+0.73}_{-0.84}$ both ath $68\%$
C.L., consistent with GW170817 (and GW190521) constraints (Pardo et al., 2018;
Corman et al., 2021). These constraints are also consistent with General
Relativity since in the limit of large $R_{c}$ we expect GR-like behavior
independently of $n$.

We also report Bayes factors $\ln\mathcal{B}^{D}_{\rm GR}$, to do hypothesis
testing while fixing the number of spacetime dimensions $D=4$ (GR) compared to
$D\neq 4$, under the leakage model of Equation 3. We report the following
Bayes factors, $\mathcal{B}^{D=5}_{\rm GR}=-0.47$, $\mathcal{B}^{D=6}_{\rm
GR}=-1.18$, $\mathcal{B}^{D=7}_{\rm GR}=-1.21$ and $\mathcal{B}^{D=8}_{\rm
GR}=-1.39$. As expected, higher number of spacetime dimensions with respect to
$D=4$ are disfavored by the data.

Figure 3: Constraints on the screening scale and transition steepness from
GWTC-3 BBH observations. We find that $\log_{10}R_{c}/{\rm
Mpc}=4.14^{+0.55}_{-0.86}$ with a transition steepness
$\log_{10}n=0.86^{+0.73}_{-0.84}$ C.L., consistent with previous upper limits
placed by GW170817 and GW190521 (Pardo et al., 2018; Corman et al., 2021). We
find that these parameters are consistent with GR, e.g. large screening scale.

##  5 Discussion

We placed constraints on the number of spacetime dimensions from binary black
hole mergers using the recently released GWTC-3 catalog. We find that our
constraints on $D$ agree with general relativity for both gravitational
leakage models considered in this work. We also find that the constraints
placed on the screening scale and transition steepness for gravitational
leakage models are consistent with General Relativity. We note that, further,
and more distant observations should allow us to place tighter constraints on
these scales.

The constraints presented in this paper are dependent on the chosen binary
black hole mass and redshift distribution models. However, the
phenomenological models used in this work should be flexible enough to capture
the astrophysics that drives different BBH formation channels at current
detector sensitivities. The BBH population modeling can be improved to include
sub-populations such as hierarchical mergers or even mergers above the PISNe
mass gap, we leave this study to future work.

With respect to gravity leakage models, one can use the Bayesian framework
presented here to test different parameterizations for the modified luminosity
distance $d_{L}^{\rm GW}$ under different models. For example, alternative
phenomenological models such as the graviton leakage model of Pardo et al.
(2018) or theory specific models from specific parameterizations of modified
gravity theories.

The Bayesian analysis used in this work, can also be extended to work with the
dark siren formalism that has been used mainly to perform cosmological
measurements on $H_{0}$, since it provides a discrete prior on the allowed
locations (assuming they are hosted by a galaxy) for GW sources.

Binary black hole mergers can probe cosmology as well as extensions of the
$\Lambda$CDM cosmological model under modified gravity at cosmological scales
without the need of an independent redshift measurement to the source or other
electromagnetic information. With future, more distant GW catalogs and an
improved understanding of the BBH population we can only expect to improve
these measurements and better understand the underlying cosmological model.

## Acknowledgements

The author would like to thank Amanda Baylor, Patrick Brady, Jolien Creighton,
Rico Ka Lok Lo, Soichiro Morisaki and Daniel Wysocki for useful comments and
feedback throughout this work. IMH is supported by the NSF Graduate Research
Fellowship Program under grant DGE-17247915. This work was supported by NSF
awards PHY-1912649. The author is grateful for computational resources
provided by the Leonard E Parker Center for Gravitation, Cosmology and
Astrophysics at the University of Wisconsin-Milwaukee. We thank LIGO and Virgo
Collaboration for providing the data for this work. This research has made use
of data, software and/or web tools obtained from the Gravitational Wave Open
Science Center (https://www.gw-openscience.org/ ), a service of LIGO
Laboratory, the LIGO Scientific Collaboration and the Virgo Collaboration.
LIGO Laboratory and Advanced LIGO are funded by the United States National
Science Foundation (NSF) as well as the Science and Technology Facilities
Council (STFC) of the United Kingdom, the Max-Planck-Society (MPS), and the
State of Niedersachsen/Germany for support of the construction of Advanced
LIGO and construction and operation of the GEO600 detector. Additional support
for Advanced LIGO was provided by the Australian Research Council. Virgo is
funded, through the European Gravitational Observatory (EGO), by the French
Centre National de Recherche Scientifique (CNRS), the Italian Istituto
Nazionale di Fisica Nucleare (INFN) and the Dutch Nikhef, with contributions
by institutions from Belgium, Germany, Greece, Hungary, Ireland, Japan,
Monaco, Poland, Portugal, Spain. This material is based upon work supported by
NSF’s LIGO Laboratory which is a major facility fully funded by the National
Science Foundation. This article has been assigned LIGO document number
LIGO-P2100449.

## References

  * Aasi et al. (2015) Aasi, J., et al. 2015, Class. Quant. Grav., 32, 074001, doi: 10.1088/0264-9381/32/7/074001
  * Abbott et al. (2017a) Abbott, B., et al. 2017a, Phys. Rev. Lett., 119, 161101, doi: 10.1103/PhysRevLett.119.161101
  * Abbott et al. (2019a) —. 2019a.  https://arxiv.org/abs/1908.06060
  * Abbott et al. (2021a) —. 2021a.  https://arxiv.org/abs/2111.03634
  * Abbott et al. (2017b) Abbott, B. P., et al. 2017b, Nature, 551, 85, doi: 10.1038/nature24471
  * Abbott et al. (2019b) —. 2019b, Phys. Rev. X, 9, 031040, doi: 10.1103/PhysRevX.9.031040
  * Abbott et al. (2021b) —. 2021b, Astrophys. J., 909, 218, doi: 10.3847/1538-4357/abdcb7
  * Abbott et al. (2020) Abbott, R., Abbott, T. D., Abraham, S., et al. 2020, Population Properties of Compact Objects from the Second LIGO-Virgo Gravitational-Wave Transient Catalog.  https://arxiv.org/abs/2010.14533
  * Abbott et al. (2021c) Abbott, R., et al. 2021c, Phys. Rev. X, 11, 021053, doi: 10.1103/PhysRevX.11.021053
  * Abbott et al. (2021d) —. 2021d.  https://arxiv.org/abs/2108.01045
  * Abbott et al. (2021e) —. 2021e.  https://arxiv.org/abs/2111.03606
  * Abbott et al. (2021f) —. 2021f.  https://arxiv.org/abs/2111.03634
  * Abbott et al. (2021g) —. 2021g.  https://arxiv.org/abs/2111.03604
  * Abbott et al. (2021h) —. 2021h.  https://arxiv.org/abs/2105.06384
  * Abbott et al. (2021i) —. 2021i, Phys. Rev. D, 103, 122002, doi: 10.1103/PhysRevD.103.122002
  * Abbott et al. (2021j) —. 2021j.  https://arxiv.org/abs/2112.06861
  * Acernese et al. (2015) Acernese, F., et al. 2015, Class. Quant. Grav., 32, 024001, doi: 10.1088/0264-9381/32/2/024001
  * Callister et al. (2020) Callister, T., Fishbach, M., Holz, D., & Farr, W. 2020, Astrophys. J. Lett., 896, L32, doi: 10.3847/2041-8213/ab9743
  * Chen et al. (2018) Chen, H.-Y., Fishbach, M., & Holz, D. E. 2018, Nature, 562, 545, doi: 10.1038/s41586-018-0606-0
  * Corman et al. (2021) Corman, M., Ghosh, A., Escamilla-Rivera, C., et al. 2021.  https://arxiv.org/abs/2109.08748
  * Deffayet & Menou (2007) Deffayet, C., & Menou, K. 2007, Astrophys. J. Lett., 668, L143, doi: 10.1086/522931
  * Del Pozzo (2012) Del Pozzo, W. 2012, Phys. Rev., D86, 043011, doi: 10.1103/PhysRevD.86.043011
  * Diaz & Mukherjee (2021) Diaz, C. C., & Mukherjee, S. 2021.  https://arxiv.org/abs/2107.12787
  * Ezquiaga (2021) Ezquiaga, J. M. 2021, Phys. Lett. B, 822, 136665, doi: 10.1016/j.physletb.2021.136665
  * Ezquiaga & Holz (2021) Ezquiaga, J. M., & Holz, D. E. 2021, Astrophys. J. Lett., 909, L23, doi: 10.3847/2041-8213/abe638
  * Farr (2019) Farr, W. M. 2019, Research Notes of the AAS, 3, 66, doi: 10.3847/2515-5172/ab1d5f
  * Finke et al. (2021) Finke, A., Foffa, S., Iacovelli, F., Maggiore, M., & Mancarella, M. 2021, JCAP, 08, 026, doi: 10.1088/1475-7516/2021/08/026
  * Fishbach et al. (2019) Fishbach, M., Gray, R., Magaña Hernandez, I., Qi, H., & Sur, A. 2019, Astrophys. J., 871, L13, doi: 10.3847/2041-8213/aaf96e
  * Gray et al. (2019) Gray, R., Magaña Hernandez, I., Qi, H., Sur, A., et al. 2019.  https://arxiv.org/abs/1908.06050
  * Heger et al. (2003) Heger, A., Fryer, C. L., Woosley, S. E., Langer, N., & Hartmann, D. H. 2003, The Astrophysical Journal, 591, 288–300, doi: 10.1086/375341
  * Heger & Woosley (2002) Heger, A., & Woosley, S. E. 2002, The Astrophysical Journal, 567, 532–543, doi: 10.1086/338487
  * Holz & Hughes (2005) Holz, D. E., & Hughes, S. A. 2005, Astrophys. J., 629, 15, doi: 10.1086/431341
  * Lagos et al. (2019) Lagos, M., Fishbach, M., Landry, P., & Holz, D. E. 2019, Phys. Rev. D, 99, 083504, doi: 10.1103/PhysRevD.99.083504
  * Mancarella et al. (2021) Mancarella, M., Genoud-Prachex, E., & Maggiore, M. 2021.  https://arxiv.org/abs/2112.05728
  * Mandel et al. (2019) Mandel, I., Farr, W. M., & Gair, J. R. 2019, Monthly Notices of the Royal Astronomical Society, 486, 1086–1093, doi: 10.1093/mnras/stz896
  * Mukherjee et al. (2021a) Mukherjee, S., Wandelt, B. D., Nissanke, S. M., & Silvestri, A. 2021a, Phys. Rev. D, 103, 043520, doi: 10.1103/PhysRevD.103.043520
  * Mukherjee et al. (2021b) Mukherjee, S., Wandelt, B. D., & Silk, J. 2021b, Mon. Not. Roy. Astron. Soc., 502, 1136, doi: 10.1093/mnras/stab001
  * Nair et al. (2018) Nair, R., Bose, S., & Saini, T. D. 2018, Phys. Rev., D98, 023502, doi: 10.1103/PhysRevD.98.023502
  * Palmese et al. (2021) Palmese, A., Bom, C. R., Mucesh, S., & Hartley, W. G. 2021.  https://arxiv.org/abs/2111.06445
  * Palmese et al. (2020) Palmese, A., et al. 2020, Astrophys. J. Lett., 900, L33, doi: 10.3847/2041-8213/abaeff
  * Pardo et al. (2018) Pardo, K., Fishbach, M., Holz, D. E., & Spergel, D. N. 2018, JCAP, 07, 048, doi: 10.1088/1475-7516/2018/07/048
  * Schutz (1986) Schutz, B. F. 1986, Nature, 323, 310, doi: 10.1038/323310a0
  * Soares-Santos et al. (2019) Soares-Santos, M., Palmese, A., et al. 2019, Astrophys. J., 876, L7, doi: 10.3847/2041-8213/ab14f1
  * Talbot & Thrane (2018) Talbot, C., & Thrane, E. 2018, The Astrophysical Journal, 856, 173, doi: 10.3847/1538-4357/aab34c
  * Taylor et al. (2012) Taylor, S. R., Gair, J. R., & Mandel, I. 2012, Phys. Rev. D, 85, 023535, doi: 10.1103/PhysRevD.85.023535
  * Woosley (2017) Woosley, S. E. 2017, The Astrophysical Journal, 836, 244, doi: 10.3847/1538-4357/836/2/244
  * Woosley (2019) —. 2019, The Astrophysical Journal, 878, 49, doi: 10.3847/1538-4357/ab1b41
  * Woosley et al. (2002) Woosley, S. E., Heger, A., & Weaver, T. A. 2002, Rev. Mod. Phys., 74, 1015, doi: 10.1103/RevModPhys.74.1015

##  Appendix A Full Posterior Distributions

Figure 4: Posterior distribution on number of spacetime dimensions $D$ using
the model in Equation 2, as well as the “Power Law + Peak” mass distribution
parameters $m_{\rm min}$, $m_{\rm max}$, $\alpha$, $\beta$, $m_{\rm pp}$,
$\sigma_{\rm pp}$ and $f_{\rm pp}$ and the SFR-like redshift evolution model
with parameters $\gamma$, $\kappa$ and $z_{\rm peak}$.  Figure 5: Posterior
distribution on number of spacetime dimensions $D$, screening scale $R_{c}$
and transition steepness $n$ using the model in Equation 3, as well as the
“Power Law + Peak” mass distribution parameters the SFR-like redshift
evolution model with parameters.


# Annotations on a Budget: Leveraging Geo-Data Similarity to
Balance Model Performance and Annotation Cost

###### Abstract

Current foundation models have shown impressive performance across various
tasks. However, several studies have revealed that these models are not
effective for everyone due to the imbalanced geographical and economic
representation of the data used in the training process. Most of this data
comes from Western countries, leading to poor results for underrepresented
countries. To address this issue, more data needs to be collected from these
countries, but the cost of annotation can be a significant bottleneck. In this
paper, we propose methods to identify the data to be annotated to balance
model performance and annotation costs. Our approach first involves finding
the countries with images of topics (objects and actions) most visually
distinct from those already in the training datasets used by current large
vision-language foundation models. Next, we identify countries with higher
visual similarity for these topics and show that using data from these
countries to supplement the training data improves model performance and
reduces annotation costs. The resulting lists of countries and corresponding
topics are made available at
https://github.com/MichiganNLP/visual_diversity_budget.

Keywords: geo-diverse datasets, active learning, effective annotations, visual
similarity, vision-language models

Annotations on a Budget: Leveraging Geo-Data Similarity to
Balance Model Performance and Annotation Cost

Oana Ignat, Longju Bai, Joan Nwatu, Rada Mihalcea
---
University of Michigan
<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Vision-language models have shown remarkable advances in recent years (Li et
al., 2019; Zhang et al., 2021; Radford et al., 2021; Zellers et al., 2021; Li
et al., 2022; Kirillov et al., 2023a; Huang et al., 2023b). These models have
shown great performance on a variety of tasks, from lower-level tasks such as
object detection, image segmentation (Kirillov et al., 2023a), and image and
video classification to higher-level tasks such as image/ video captioning (Li
et al., 2022; Huang et al., 2023b), text-image/video retrieval (Radford et
al., 2021), visual question answering and visual commonsense reasoning
(Zellers et al., 2021, 2022).

At the same time, prior work has demonstrated that these models do not work
well for everyone (De Vries et al., 2019). Specifically, models do not work
well on out-of-domain data, and data from low-income and non-western countries
Nwatu et al. (2023). This is due to the imbalanced geographical and economic
representation of the data used to train these models, as it comes mainly from
North America and Western Europe (Shankar et al., 2017). One solution that
Rojas et al. (2022) and Ramaswamy et al. (2023) propose is to collect more
data from underrepresented countries. However, as Ramaswamy et al. (2023)
highlights, annotation costs are a substantial bottleneck; when crowdsourcing
the data, fair pay is about 1.08$ per image without including researcher time.

Figure 1: Vision-language models work poorly on data from underrepresented
countries. This is primarily due to the diverse appearance of topics (objects
and actions) across countries (e.g., “toothbrush”). However, collecting
diverse global data is very expensive. As solutions to budget annotations, we
propose to (1) annotate the images visually different from the ones in high-
resource datasets such as LAION or ImageNet; (2) supplement data from low-
resource countries with data from visually similar countries.

As a complementary solution, we investigate strategies to reduce the
annotation budget while finding effective annotation data. Specifically, our
paper aims to answer two main research questions.

RQ1:

Which countries are less represented in the training data of vision-language
models?

We aim to find ways to effectively focus future annotation efforts on specific
countries and their corresponding topics (objects and actions).111Throughout
the paper, for brevity, we use the term country to refer to a country or
territory. Our study highlights the visual diversity of common topics across
countries and those that differ the most from the primarily Western data used
to train most multimodal foundation models.

RQ2:

How can we leverage cross-country data similarity to improve the
representation of vision-language models?

We obtain groups of countries that are visually similar in their
representation of a given topic. This is particularly useful when there is not
enough data for one of the countries in the group, and there is no annotation
budget. We can supplement the data from this country by using data from the
other countries in the group.

We summarize our contributions as follows. First, we identify the data likely
to most benefit from annotations by finding which countries and corresponding
topics are less represented in the training data of vision-language models.
Second, across 52 countries and 94 topics, we identify the groups of countries
that are visually similar in their representation of a topic and show that
they can be used to supplement training data effectively. Third, our main
takeaways create opportunities for affordable and geo-diverse data collection,
encouraging contributions to creating datasets and models that work for
everyone.

## 2\. Related Work

#### Data Subset Selection/ Active Learning

There have been numerous studies on the use of semi-supervised models to
leverage a combination of limited labeled data and vast amounts of unlabeled
data to improve model performance at lower costs (Hady and Schwenker, 2013;
Oliver et al., 2018; Taha, 2023; Chen et al., 2022).

However, model-generated labels could be inconsistent and unrepresentative
with semi-supervision, leading to reduced model performance (Ahfock and
McLachlan, 2023; Elezi et al., 2022; Wang et al., 2021). While similar to
semi-supervised learning in objective, active learning methods seek to capture
the entire data distribution by focusing labeling efforts on the data points
that provide the most information for training the best-performing models (Ren
et al., 2021; Citovsky et al., 2021; Monarch, 2021; Yang et al., 2017) using
approaches such as uncertainty-based sampling in Gal and Ghahramani (2016);
Beluch et al. (2018) and geometric-based methods in Sener and Savarese (2018).
Unsupervised subset selection methods like K-means and K-median core set in
Har-Peled and Kushal (2005), which form the foundation for geometric-based
active learning approaches are similar to our work which seeks to select a
subset that is representative of the entire dataset using distance metrics.
However, the objective of the selection is to include images from a low-
resource dataset with the least similarity to data of the same class in a
high-resource dataset.

#### Evaluating Disparities in Model Performance

There exists a considerable body of literature evaluating the fairness and the
unequal performance of vision and vision-language models on diverse groups
categorized according to race (Gebru, 2020), gender (Buolamwini and Gebru,
2018), geolocation (Kim et al., 2021; Shankar et al., 2017; Goyal et al.,
2022a) and income (De Vries et al., 2019; Nwatu et al., 2023).

Further analysis of these disparities reveals that factors such as ambiguous
label definitions, domain shifts, annotator disagreement (Hall et al., 2023;
Kalluri et al., 2023), as well as image properties relating to texture,
lighting, and occlusion in vision and vision-language datasets (Gustafson et
al., 2023) contribute to disparities in datasets which carry over to affect
model performance.

Frameworks have been developed to facilitate the detection of bias through
guided human-in-the-loop inspection, either in datasets Hu et al. (2020) or in
models Goyal et al. (2022b). Our work focuses on exploring the presence of
variations in image representations across demographic groups in existing
datasets, to inform cost-effective methods for building balanced, diverse
datasets.

#### Improving Representation in AI.

Efforts toward improving equal representation in AI and equitable AI impact
revolve around model adaptation, transfer learning, and dataset diversity.
However, Salman et al. (2022); Kalluri et al. (2023); Dubey et al. (2021);
Wang and Russakovsky (2023) suggest that transfer learning and model
adaptation methods might not be enough to eradicate the issue of under-
representation in AI models.

On the other hand, adding diverse data to training datasets tends to yield
significant improvements in model performance across different groups
(Ramaswamy et al., 2023; Rojas et al., 2022). The need for more diverse
datasets has become apparent, leading to the development of datasets like
GeoYFCC (Dubey et al., 2021), GeoDE (Ramaswamy et al., 2023), Dollar Street
(Rojas et al., 2022), and Segment Anything (Kirillov et al., 2023b) that
include data collected from diverse locations.

While advantageous, diverse datasets are expensive and resource-intensive to
build. (Schumann et al., 2021; Garcia et al., 2023; Geigle et al., 2023)
explored a less expensive alternative: revising or creating annotations for an
existing dataset to improve inclusivity and reduce bias. Similarly, we seek to
facilitate effective but less expensive annotations by leveraging the
differences between high-resource and low-resource datasets to curate the best
low-resource subset for annotation.

## 3\. Methodology

We start by collecting two datasets that reflect the low-resource and high-
resource settings. First, we compile a crowd-sourced geo-diverse dataset
collected from a large number of countries, which we refer to as “low-resource
data” due to the low number of images that could be collected for each country
in the set and the difficulty of gathering more. Second, we also compile a
web-scraped dataset used for training foundation models, which we refer to as
“high-resource” due to its vast size consisting of billions of images (e.g.,
LAION-5B222https://laion.ai/blog/laion-5b/) and the ease of gathering more
data.

Next, we pre-process the data by mapping the topics between the two data
sources, filtering out topics and countries with very few images. Finally, we
utilize the collected data to generate visual representations through vision-
language foundation models. These representations are then used to determine
the visual similarity between images of topics in low-resource data and their
corresponding topics in high-resource data.

### 3.1. Low-resource Multimodal Data

We combine two geographically diverse datasets: GeoDE (Ramaswamy et al., 2023)
and Dollar Street (Rojas et al., 2022). For brevity, we call topics all the
labels used for all the objects and actions in these two datasets.

GeoDE. The GeoDE dataset contains $61,940$ crowd-sourced images of $40$
objects. The data is balanced across six regions (West Asia, Africa, East
Asia, South East Asia, Americas, and Europe), each with 3-4 countries. These
regions were chosen due to their scarcity in most public datasets. Using a
combination of heuristics and manual validation, the authors selected the
objects likely to be visually distinct across the six regions.

Dollar Street. The Dollar Street dataset contains $38,479$ images collected
from $63$ countries on four continents (Africa, America, Asia, and Europe).
The images capture everyday household objects and actions (e.g., “toothbrush”,
“toilet paper”, “cooking”). The data contains $291$ unique topics, out of
which we remove nineteen subjective topics following the work of De Vries et
al. (2019) (e.g., “most loved item”, “things I wish I had”). All the
subjective topics are found in the Appendix. The number of images for a given
country ranges from $45$ in Canada to $4,704$ in India, with a median of $407$
images per country.

### 3.2. High-resource Multimodal Data

As high-resource datasets, we sample data from ImageNet (Deng et al., 2009)
and LAION (Schuhmann et al., 2022). We chose these datasets due to their
popularity in vision-language models.

#### ImageNet.

ImageNet and ImageNet Large Scale Visual Recognition Challenge (ILSVRC) are
pioneers in advancing object detection and classification progress. The
imagenet21k dataset (Deng et al., 2009) contains around 21,000 WordNet
(Fellbaum, 2000) synsets and more than 14 million annotated images. We use the
processed version of ImageNet21k (Ridnik et al., 2021), with removed invalid
classes and resized images. We also tried using ImageNet1k, but it did not
have enough classes for our purpose, and we chose to use it to supplement the
ImageNet21k data.

#### LAION.

Large language-vision models such as CLIP or ALIGN have been trained on
billions of image-text pairs unavailable to the public. LAION-5B (Schuhmann et
al., 2022) was created to address this problem by open-sourcing a CLIP-
filtered dataset333The data is filtered using OpenAI’s CLIP ViT-L/14 by
calculating the cosine similarity between the text and image embeddings and
dropping those with a similarity below 0.3.  of 5,85 billion high-quality
image-text pairs. We use LAION-400M (Schuhmann et al., 2021), a subset of
LAION-5B that contains 400 million English image and text pairs.

### 3.3. Data Pre-processing

#### Combine GeoDE and Dollar Street.

We pre-process and combine the low-resource datasets to increase the number of
topics, images, and country diversity. First, we manually group and rename the
topics from Dollar Street with the same meaning (e.g., “bathroom privacy”,
“bathroom/ toilet” are renamed “bathroom”). Next, we rename the topics from
Dollar Street that match those in GeoDE (e.g., “bike” to “bicycle”,
“medication” to “medicine”). We remove three topics with less than $10$ images
per topic. Finally, we obtain a total of $99$ unique topics, $93,060$ images,
from $4$ continents, $18$ regions, and $83$ countries.

#### Low-resource to High-resource Data Mapping.

We map the $99$ topics from the low-resource data to the high-resource data,
ImageNet, and LAION by identifying the images with similar labels.

First, we map $51$ topics from the low-resource data to an exact match to
ImageNet21k or ImageNet1k. We could not find an exact match for $38$ topics
because these topics are too abstract (e.g., “jewelry”, “source of cool”,
“religious building”). Instead, we find mappings for their hyponyms (e.g., for
“jewelry”, we map “bangle”, “necklace”, “bracelet” and “ring”). The remaining
$10$ topics for which we could not find any exact or hyponym mapping to
ImageNet21k or ImageNet1k are mapped to LAION.

We map data in LAION by selecting the images with captions that contain the
topic query. Because LAION data is web-crawled, we find that the images are
lower quality than ImageNet and not always relevant to the topic query: e.g.,
the “TV” topic in LAION contains images of people on TV, not of the object TV.
Therefore, to ensure the correctness of the mapping, we manually inspect the
images and map a topic to LAION only when most images are relevant to the
topic query. We map $64$ topics to LAION. Note, however, that the number of
hyponyms and the quality of LAION images limit how comprehensive the mapping
process is. Two independent annotators check 20 random images from each topic
and find that most noisy images come from LAION. Therefore, we decide to limit
the amount of data from LAION and add more images from ImageNet. Specifically,
we randomly sample around $200$ images per topic from LAION and around $1,000$
images per topic from ImageNet. Note that the high-resource data does not
contain country information. We show the data before and after pre-processing
and the topic mapping in our
repository.444https://github.com/MichiganNLP/visual_diversity_budget

Figure 2: Example images (“cooking pot”) in low-resource data (left) vs. in
high-resource data (right).

#### Data Filtering.

The low-resource data is unbalanced, as the total number of images per country
varies from 6,549 for Japan to 1 for Bulgaria and Venezuela, with a median of
345 images per country. The number of images per topic is also unbalanced,
from 3,049 for “waste container” to 18 for “hanging clothes to dry”. However,
balancing the data by down-sampling significantly reduces the number of
countries represented for each topic. Having numerous countries represented is
essential for our setup. Therefore, we choose not to balance the data.
Instead, we remove the (topic, country) pairs containing less than $10$
images, considering this threshold a minimum for experiment significance. This
also removes considerable data: 3,329/ 4,830 (topic, country) tuple pairs, 5/
99 topics, and 31/ 83 countries. We show the removed topics and corresponding
countries in our repository and highlight the need for more data for these
pairs to obtain significant results.

We show the statistics after the data collection and pre-processing in Table 1
and the image distribution of countries per topic in Appendix Figure 10.

# unique topics | 94
---|---
# unique countries | 52
# unique (topic, country) pairs | 1,501
# images in low-resource data | 80,801
# images in high-resource data | 103,006
average # images per (topic, country) | 53.8
median # images per (topic, country) | 30
Table 1: Statistics for the collected number of topics, countries, and images
collected from low-resource and high-resource data after data pre-processing.

### 3.4. Data Representation

We use an ensemble of three representations to compute the image similarity
and to ensure the results generalize across representation types. We choose
CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), and BLIP-2 (Li et al.,
2023) due to their popularity as foundation models (Bommasani et al., 2022),
i.e., their use in a multitude of models and their high zero-shot performance
across various tasks and datasets, such as text-to-image retrieval, image
question answering, human action segmentation, image-sentence alignment, image
captioning (Cafagna et al., 2021; Saharia et al., 2022; Kirillov et al.,
2023b; Huang et al., 2023a).

#### CLIP Representations.

We use the pre-trained Vision Transformer ViT-B/32 (Dosovitskiy et al., 2021)
from the CLIP model (Radford et al., 2021) to encode the visual
representations of the images. The training dataset for CLIP was created from
the results of numerous queries to various publicly available Internet
sources. The dataset referred to as WebImageText WIT contains 400 million
(image, text) pairs and is not available to the public.

#### ALIGN Representations.

We also extract image features following the ALIGN (Jia et al., 2021) model
setup, using a pre-trained EfficientNet (Tan and Le, 2019) as a vision
encoder. Since the original code has not been released, our implementation is
based on the Kakao Brain code that reproduced the original
paper.555https://huggingface.co/docs/transformers/model_doc/align ALIGN was
trained on 1.8 billion image-text pairs collected following the methodology
used for the Conceptual Captions dataset (Sharma et al., 2018). Since the
emphasis was on scale instead of quality, the dataset underwent fewer post-
processing steps, thus leading to a noisier dataset. This dataset is currently
unavailable for public access.

#### BLIP-2 Representations.

We also extract image features using BLIP-2 (Li et al., 2023), which uses
ViT-g/14 from EVA-CLIP (Sun et al., 2023) as image encoder and removes the
second last layer’s output features to increase the performance. BLIP-2 was
trained on a total of 129M images aggregated from the COCO (Lin et al., 2014),
Visual Genome (Krishna et al., 2017), CC3M (Sharma et al., 2018), CC12M
(Changpinyo et al., 2021), SBU (Ordonez et al., 2011), and the LAION400M
datasets (Schuhmann et al., 2021). Captions for the web images were generated
using CapFilt (Li et al., 2022).

## 4\. Mapping the Representation of Vision-Language Models

In this section, we address the first research question: RQ1: Which countries
are less represented in the training data of vision-language models?

For each (topic, country) pair, we compute the cosine similarity between the
average visual representations of all the corresponding images in the low-
resource data and the average visual representations of all the corresponding
images in the high-resource data. Note that the average is computed over all
three visual representation types, i.e., CLIP, BLIP, and ALIGN. We select the
(topic, country) pairs with a similarity score lower than a threshold computed
as the average similarity score between all the image representations in the
low-resource data and the corresponding representations in the high-resource
data. This process is repeated for each visual representation
type.666Thresholds and data representations can be changed to fit the purpose
of the analysis or application. Finally, the (topic, country) pairs selected
for all three visual representations are the ones we find to be consistently
different from the high-resource data and, thus, the ones that benefit the
most from annotations. We find 422 such (topic, country) pairs out of 1,501
unique (topic, country) pairs, potentially reducing the annotation budget to
less than a third of the initial amount. We share the results in our
repository.

#### Visual similarity for each (topic, country) in low-resource data with
corresponding topics in high-resource data.

We compute a similarity heatmap where the rows are the topics, and the columns
are countries. We sort the rows (countries) and columns (topics) from the
least to the most similar based on the average similarity score per country
and topic, leaving out the $NaN$ values (the grey, empty cells). We show in
Figure 3 the similarity heatmap for the CLIP representation and highlight the
(topic, country) pairs we find to benefit the most from annotations based on
consistently low similarity with the high-resource data across the three
visual representations.

From Figure 3, we can also see that the countries with the fewest data are
usually the ones with the most topics in need of annotations (e.g., from
$Burundi$ to $Kenya$). Exceptions to this are countries such as $Nepal$,
$Nigeria$, $Philippines$, and $Indonesia$, which have more data points
(topics), but more than half of the topics require annotations, and countries
such as $Czech$ $Republic$, $France$ or $Austria$ which have very few topics
and none require annotations. In Figure 3, we see a few topics in $United$
$States$ that are marked to require annotations: “medicine”, “spice”,
“ceiling”, “clothes” and “makeup”. We show in Appendix Figure 11
representative images from these topics from the two data sources, which
explain the visual differences. For the rest of the topics, as expected,
$United$ $States$ data is similar to the high-resource data. We considered
using the $United$ $States$ as the high-resource data source. However, due to
the lack of data on some topics and relatively few images per topic compared
to other countries, it was not feasible.

There are differences between the results obtained with each visual
representation type regarding similarity score intervals and which (topic,
country) pairs are similar to the high-resource data. However, the general
similarity trend is consistent as most (topic, country) pairs have only low or
high similarity scores across all three representations. This is also
supported by the strong Pearson correlations between the scores obtained with
the three representation types: CLIP and BLIP scores correlate $0.62$, CLIP
and ALIGN scores correlate $0.65$, ALIGN and BLIP scores correlate $0.72$. We
show in the Appendix Figure 12, 13, and 14, the similarity heatmaps for each
representation type: CLIP, ALIGN, and BLIP respectively.

Figure 3: Similarity heatmap of (topic, country) pairs. Based on the average
similarity score, rows and columns are sorted from the least to the most
similar. The lighter the color, the lower the similarity between high-resource
and low-resource data for that corresponding (topic, country) pair, the more
beneficial it is to annotate. We highlight with black the pairs we determine
to benefit the most from annotations. Grey cells have less than ten images and
are therefore discarded. Best viewed in color.

#### Topic visual representation in high-resource and low-resource data.

To show how the topic visual representations vary per low-resource and high-
resource data, we perform a 2D transformation using Principal Component
Analysis (PCA) (F.R.S., 1901). In Figure 4, we show the CLIP average
representations per country in the low-resource and the corresponding high-
resource data for the topic “toothbrush”. We can observe that, for this topic,
there is considerable visual diversity across countries. When comparing to the
high-resource data, $ImageNet\\_LAION$, we observe visually different
countries, such as $Malawi$, $Rwanda$, and $Myanmar$, and countries very
visually similar, such as $Netherlands$, $UnitedStates$, and $Brazil$. In
addition, we observe many countries that tend to be clustered together, i.e.,
visually similar for this particular topic, such as $Mexico$, $Italy$,
$Japan$, $South$ $Korea$, and others. We examine more about the similarities
between countries when answering RQ2, in the following section. In Appendix
Figure 15, 16, 17 we show results for other topics (“hand washing”, “toilet”,
“wall”) in low-resource and high-resource data.

Figure 4: PCA for the topic “toothbrush” for all countries that contain this
topic in the low-resource data and in the high-resource data. The high-
resource data point is highlighted with star symbol. The data is represented
as the average of the CLIP representations.

## 5\. Cross-country Data Similarity for Improved Model Representation

We now turn to the second research question RQ2: How can we leverage cross-
country data similarity to improve the representation of vision-language
models?

We calculate the cosine similarity between the average visual representations
of images for each topic across countries, and repeat this process for all
three visual representations. Given a topic, the final visual similarity score
between two countries is obtained by averaging the similarity values obtained
for each visual representation type. For each (country, topic) pair, we obtain
the visually similar countries, along with their similarity score, from the
most to the least similar, and share them in our repository.

We calculate the average similarity score for each country across all
corresponding topics and for each topic across all corresponding countries. We
show the similarity score distribution for the top three and last three
countries and topics in Figure 5, and for all countries and topics in the
Appendix Figure 18 and 19.

As shown in Figure 5, $Burundi$ has the lowest similarity score of $0.775$,
indicating that it is the most different country compared to the others and
needs its own annotations. On the other hand, $Argentina$ has the highest
similarity scores of $0.907$, indicating a high similarity to other countries.
These results imply that annotating data from $Argentina$ would help other
countries. The most visually different topic is “religious building” with a
score of $0.76$, and the most similar topic is “hat” with a score of $0.96$.
These results imply that “religious buildings” should be annotated more widely
as their visual appearance varies across countries.

Finally, we investigate whether performance of similarity calculation depends
on amount of annotated data. We find that at topic level the similarity scores
are not correlated with the amount of annotated data (Pearson correlation
coefficient is -0.02). We discuss more about the effect of data size on our
analysis results in the Appendix.

Figure 5: Top three and last three countries (left) and topics (right) sorted
by average similarity score.

#### Topic visual representation across countries in low-resource data.

To show how the topic visual representations vary per country in the low-
resource data, we perform a 2D transformation using Principal Component
Analysis (PCA) (F.R.S., 1901). In Figure 6, we show the CLIP average
representations per country for the topics with the most and least visual
differences across countries: “religious building” and “hat”, respectively. As
expected, the representations for “religious building” are much more spread
across countries than those for “hat”, which tend to cluster together. In
Appendix Figure 20, 21, and 22, we show representations for other topics
visually different across countries: “get water”, “house” and “backyard.”

Figure 6: PCA for the topic “religious building” and “hat” for all countries
in the low-resource data that contain this topic. The data is represented as
the average of the CLIP representations.

#### Correlation between geographical distance and visual similarity across
countries.

We measure if the visual similarity between countries correlates with the
geographical distance. The geographical distance between two countries is
calculated using Vincenty’s distance Vincenty (1975) between their capital
cities.777https://github.com/rahulbot/distances-between-countries The visual
similarity between any two countries is calculated across all their shared
topics. We compute the Pearson correlation coefficient Freedman et al. (2007)
over all countries and obtain a value of $-0.01$, indicating a weak negative
correlation. A strong negative correlation is initially more expected as,
intuitively, their visual similarity should increase as the distance between
countries decreases. However, when we break down the correlation at the
country level, the correlation coefficient varies significantly per country.

Figure 7:  Pearson correlation coefficient between the visual similarity and
the geographical distance, across countries. Most countries do not have a
significant correlation between visual similarity and location.

In Figure 7, we show countries with weak to moderate positive correlations
(e.g., $Haiti$ with $0.35$, $Tunisia$ with $0.30$), countries with weak to
moderate negative correlations (e.g., $Vietnam$ with $-0.35$, $Burundi$ with
$-0.34$), most countries have values close to $0$, indicating no correlation
between visual similarity and geographical distance. Upon close examination of
the results, we determine the reasons behind this result: countries with
positive correlation are often visually similar to countries from different
continents (e.g., $Tunisia$ is more similar to $Bolivia$ with an average
similarity $0.94$ and distance $9,773$ than to $Austria$ with with an average
similarity $0.80$ and distance $1,362$). We hypothesize this might be due to
history, climate, and/or income differences, which could contribute more to
visual similarity than distance alone. Our analysis shows that geographical
location does not generally correlate with visual similarity. Therefore,
collecting globally diverse annotations on a budget requires considering other
complementary information, such as the country’s income, culture, history, and
climate. Our results on which countries are similar to each other provide
valuable insights into how to distribute the annotation budget effectively and
can be used along with this complementary information.

#### Augmenting with data from visually similar countries significantly
improves model performance.

We train a classifier to predict the topic of the input images and measure the
accuracy while controlling for the countries. Specifically, we input the CLIP
visual representation in a linear layer, followed by a softmax to predict the
topics of the input images.888We set the learning rate as 5e-3, use AdamW as
the optimizer, and conduct training over 250 epochs with a batch size of 512.
Additionally, we use a cosine annealing schedule with 50 warm-up epochs. We
select one random country for each topic from the low-resource data, which we
call target (topic, country) pairs. Next, we split the data into training and
test sets in a 90-10% data split to include all the target (topic, country)
pairs in both sets. Finally, we replace different ratios (100%, 90%, 70%, 50%,
30%, 10%, 0%) of the target-country data with images from: (1) the most
similar countries to the target-country given the target-topic; (2) the most
dissimilar countries to the target-country given the target-topic; (3) high-
resource data corresponding to the target-topic.

The topic classification accuracy when using all the training target-country
data is $91.1\%$, which is an upper bound. In Figure 8, we show the accuracy
when adding data from (1), (2) and (3). The main takeaway is that adding data
from similar countries improves the performance more than adding data from
dissimilar countries or high-resource data, and the gap in performance
increases with the replacement ratio. Additionally, supplementing with high-
resource data is generally more beneficial than supplementing with data from
dissimilar countries. We also compute the accuracy when no data is added, and
find that adding data from dissimilar countries or from high-resource data can
hurt the performance compared to not adding data, especially for high
replacement ratios ($50\%-90\%$). We show the results in the Appendix, in
Figure 23.

Figure 8: Topic classification accuracy (in %) for different target-country
data ratios (e.g., target-country ratio 0.0% is equivalent to 100% replacement
ratio). We replace different ratios of the target-country data with images
from: (1) the most similar countries to the target-country given the target-
topic; (2) the most dissimilar countries to the target-country given the
target-topic; (3) high-resource data of the target-topic;

## 6\. Main Takeaways

Our analyses provide multiple insights into the current state of vision-
language annotations for various topics across different countries, and show
the coverage limitations of existing large-scale datasets. We highlight the
main takeaways and propose actionable steps to help future work create more
inclusive datasets and models.

#### We recommend focusing the annotation efforts on currently
underrepresented data.

To have more inclusive models and datasets, we need to collect more globally
diverse annotations. Because annotations are expensive, we propose to focus
future annotation efforts on specific countries and their topics. To assist
with these efforts, we provide a list of countries and corresponding topics
that are consistently unrepresented in the training data of vision-language
models. Furthermore, most countries have less than ten images per topic. For
most countries and corresponding topics – 3,329/ 4,830, we could not determine
how similar they are to the high-resource data because of the lack of data.
These countries have less than ten images per topic and, therefore, already
need annotations. As an alternative solution, we recommend developing
algorithms that can perform well with limited amount of data.

#### We can leverage cross-country data similarity to supplement data from
unrepresented countries effectively.

When we do not have a sufficient budget to annotate more data for a target
country and topic, we propose using the available data from countries with
similar visual representations of that given topic. We provide a list of
similar countries for each target country and topic and show that using this
data improves model performance more than using data from dissimilar countries
or high-resource data.

#### Geographical distance does not correlate with visual similarity between
countries.

We compute the Pearson correlation coefficient between the visual similarity
and the geographical distance between all countries and find a very weak
negative correlation of -0.01. Therefore, collecting globally diverse
annotations requires considering additional information. Multiple other
factors, such as income, history, or cultural heritage, can contribute to the
visual similarity between countries. We find this hypothesis worth
investigating in depth in future work.

#### Visual similarity between countries and topics depends on the context.

While examining images of topics across countries, we notice visually similar
topics with very different backgrounds, which influence the visual similarity
score. For example, in Figure 9, many countries have the same type of
toothbrush, but because their storage place is different, their visual
similarity score is low. In this paper, we measure similarity at the context
level, considering both the topic and the context (e.g., background, storage
space). However, as future work, we propose to investigate further which type
of similarity to consider when we annotate diverse data: either at the topic
level, by extracting the segmentation mask of the topic, or at the context
level, by considering the entire image.

Figure 9: The context of the topic influences the visual similarity. For
example, although the same type of toothbrush is depicted, their storage place
differs, i.e., on a piece of wood, in a plastic container in the bathroom, in
a plastic container tied to a tree, near a brick wall. Therefore, visual
diversity is measured not only at the topic level but also at the context
level.

## 7\. Conclusion

In this paper, we addressed the need for balanced data representation used to
train vision-language models. Because data annotations are expensive, we
proposed to annotate primarily images from unrepresented countries. To find
which countries are less represented in the training data of vision-language
models, we compared the visual similarity of images across 94 topics and 52
countries found in crowd-sourced and web-scraped data. We used three visual
representations, CLIP, BLIP-2, and ALIGN, to ensure the results generalize
across representation types. Additionally, we proposed to leverage cross-
country data similarity to improve model performance. We found visually
similar countries for each country and corresponding topics and made them
available in our repository:
https://github.com/MichiganNLP/visual_diversity_budget. Finally, our analysis
offers multiple takeaways for future work to make informed decisions on what
global data to annotate and how to leverage cross-country data similarity to
improve model representation. Through our work, we hope to contribute to
building more inclusive and affordable vision-language models and datasets to
help democratize AI globally.

## Acknowledgments

We thank the anonymous reviewers for their constructive feedback and are also
grateful to the members of the Language and Information Technologies (LIT) lab
at the University of Michigan for the insightful discussions during the
project’s early stages. This material is partly based on work supported by the
Automotive Research Center (“ARC”) at the University of Michigan. Any
opinions, findings, conclusions, or recommendations expressed in this material
are those of the authors and do not necessarily reflect the views of ARC or
any other related entity.

## 8\. Bibliographical References

##

  * Ahfock and McLachlan (2023) Daniel Ahfock and Geoffrey J McLachlan. 2023.  Semi-supervised learning of classifiers from a statistical perspective: A brief review.  _Econometrics and Statistics_ , 26:124–138.
  * Beluch et al. (2018) William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. 2018.  The power of ensembles for active learning in image classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 9368–9377.
  * Bommasani et al. (2022) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2022.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_.
  * Buolamwini and Gebru (2018) Joy Buolamwini and Timnit Gebru. 2018.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In _Conference on fairness, accountability and transparency_ , pages 77–91. PMLR.
  * Burdick et al. (2017) Laura Burdick, Rada Mihalcea, Ryan L. Boyd, and James W. Pennebaker. 2017.  Multimodal analysis and prediction of latent user dimensions.  In _Social Informatics_.
  * Cafagna et al. (2021) Michele Cafagna, Kees van Deemter, and Albert Gatt. 2021.  What vision-language modelssee’when they see scenes.  _arXiv preprint arXiv:2109.07301_.
  * Chan et al. (2021) Alan Kam Leung Chan, Chinasa T. Okolo, Zachary Terner, and Angelina Wang. 2021.  The limits of global inclusion in ai development.  _ArXiv_ , abs/2102.01265.
  * Changpinyo et al. (2021) Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3558–3568.
  * Chao et al. (2015) Yu-Wei Chao, Zhan Wang, Rada Mihalcea, and Jia Deng. 2015.  Mining semantic affordances of visual object categories.  _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4259–4267.
  * Chen et al. (2022) Yanbei Chen, Massimiliano Mancini, Xiatian Zhu, and Zeynep Akata. 2022.  Semi-supervised and unsupervised deep visual learning: A survey.  _IEEE transactions on pattern analysis and machine intelligence_.
  * Citovsky et al. (2021) Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Rostamizadeh, and Sanjiv Kumar. 2021.  Batch active learning at scale.  _Advances in Neural Information Processing Systems_ , 34:11933–11944.
  * Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik. 1995.  Support-vector networks.  _Machine learning_ , 20(3):273–297.
  * De Vries et al. (2019) Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten. 2019.  Does object recognition work for everyone?  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pages 52–59.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database.  _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 248–255.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An image is worth 16x16 words: Transformers for image recognition at scale.  _ICLR_.
  * Dubey et al. (2021) Abhimanyu Dubey, Vignesh Ramanathan, Alex ’Sandy’ Pentland, and Dhruv Kumar Mahajan. 2021.  Adaptive methods for real-world domain generalization.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 14335–14344.
  * Elezi et al. (2022) Ismail Elezi, Zhiding Yu, Anima Anandkumar, Laura Leal-Taixe, and Jose M Alvarez. 2022.  Not all labels are equal: Rationalizing the labeling costs for training object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14492–14501.
  * Fang and Cohn (2017) Meng Fang and Trevor Cohn. 2017.  Model transfer for tagging low-resource languages using a bilingual dictionary.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 587–593, Vancouver, Canada. Association for Computational Linguistics.
  * Fellbaum (2000) Christiane D. Fellbaum. 2000.  Wordnet : an electronic lexical database.  _Language_ , 76:706.
  * Freedman et al. (2007) David Freedman, Robert Pisani, and Roger Purves. 2007.  Statistics (international student edition).  _Pisani, R. Purves, 4th edn. WW Norton & Company, New York_.
  * F.R.S. (1901) Karl Pearson F.R.S. 1901.  Liii. on lines and planes of closest fit to systems of points in space.  _The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science_ , 2(11):559–572.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , pages 1050–1059. PMLR.
  * Gal et al. (2017) Yarin Gal, Riashat Islam, and Zoubin Ghahramani. 2017.  Deep bayesian active learning with image data.  In _International conference on machine learning_ , pages 1183–1192. PMLR.
  * Garcia et al. (2023) Noa Garcia, Yusuke Hirota, Yankun Wu, and Yuta Nakashima. 2023.  Uncurated image-text datasets: Shedding light on demographic bias.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6957–6966.
  * Garimella et al. (2016) Aparna Garimella, Rada Mihalcea, and James W. Pennebaker. 2016.  Identifying cross-cultural differences in word usage.  In _International Conference on Computational Linguistics_.
  * Gebru (2020) Timnit Gebru. 2020.  Race and gender.  _The Oxford handbook of ethics of aI_ , pages 251–269.
  * Geigle et al. (2023) Gregor Geigle, Radu Timofte, and Goran Glavaš. 2023.  Babel-imagenet: Massively multilingual evaluation of vision-and-language representations.  _arXiv preprint arXiv:2306.08658_.
  * Goyal et al. (2022a) Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Ishan Misra, Levent Sagun, Armand Joulin, and Piotr Bojanowski. 2022a.  Vision models are more robust and fair when pretrained on uncurated images without supervision.  _ArXiv_ , abs/2202.08360.
  * Goyal et al. (2022b) Priya Goyal, Adriana Romero-Soriano, Caner Hazirbas, Levent Sagun, and Nicolas Usunier. 2022b.  Fairness indicators for systematic assessments of visual feature extractors.  _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_.
  * Gustafson et al. (2023) Laura Gustafson, Megan Richards, Melissa Hall, Caner Hazirbas, Diane Bouchacourt, and Mark Ibrahim. 2023.  Pinpointing why object recognition performance degrades across income levels and geographies.  _ArXiv_ , abs/2304.05391.
  * Hady and Schwenker (2013) Mohamed Farouk Abdel Hady and Friedhelm Schwenker. 2013.  Semi-supervised learning.  _Handbook on Neural Information Processing_ , pages 215–239.
  * Hall et al. (2023) Melissa Hall, Bobbie Chern, Laura Gustafson, Denisse Ventura, Harshad Kulkarni, Candace Ross, and Nicolas Usunier. 2023.  Towards reliable assessments of demographic disparities in multi-label image classifiers.  _arXiv preprint arXiv:2302.08572_.
  * Har-Peled and Kushal (2005) Sariel Har-Peled and Akash Kushal. 2005.  Smaller coresets for k-median and k-means clustering.  In _Proceedings of the twenty-first annual symposium on Computational geometry_ , pages 126–134.
  * Hu et al. (2020) Xiao Hu, Haobo Wang, Anirudh Vegesana, Somesh Dube, Kaiwen Yu, Gore Kao, Shuo-Han Chen, Yung-Hsiang Lu, George K. Thiruvathukal, and Ming Yin. 2020.  Crowdsourcing detection of sampling biases in image datasets.  In _Proceedings of The Web Conference 2020_ , WWW ’20, page 2955–2961, New York, NY, USA. Association for Computing Machinery.
  * Huang et al. (2023a) Hanyao Huang, Ou Zheng, Dongdong Wang, Jiayi Yin, Zijin Wang, Shengxuan Ding, Heng Yin, Chuan Xu, Renjie Yang, Qian Zheng, et al. 2023a.  Chatgpt for shaping the future of dentistry: the potential of multi-modal large language model.  _International Journal of Oral Science_ , 15(1):29.
  * Huang et al. (2023b) Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. 2023b.  Tag2text: Guiding vision-language model via image tagging.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , pages 4904–4916. PMLR.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Kalluri et al. (2023) Tarun Kalluri, Wangdong Xu, and Manmohan Chandraker. 2023.  Geonet: Benchmarking unsupervised adaptation across geographies.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15368–15379.
  * Kim et al. (2021) Zu Whan Kim, Andre F. de Araújo, Bingyi Cao, Cameron S Askew, Jack Sim, Mike Green, N’Mah Fodiatu Yilla, and Tobias Weyand. 2021.  Towards a fairer landmark recognition dataset.  _ArXiv_ , abs/2108.08874.
  * Kirillov et al. (2023a) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. 2023a.  Segment anything.  _ArXiv_ , abs/2304.02643.
  * Kirillov et al. (2023b) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023b.  Segment anything.  _arXiv preprint arXiv:2304.02643_.
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ , 123:32–73.
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019.  VisualBERT: A simple and performant baseline for vision and language.  _ArXiv_ , abs/1908.03557.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer.
  * Monarch (2021) Robert Munro Monarch. 2021.  _Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI_.  Simon and Schuster.
  * Nwatu et al. (2023) Joan Nwatu, Oana Ignat, and Rada Mihalcea. 2023.  Bridging the digital divide: Performance variation across socio-economic factors in vision-language models.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
  * Oliver et al. (2018) Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. 2018.  Realistic evaluation of deep semi-supervised learning algorithms.  _Advances in neural information processing systems_ , 31.
  * Ordonez et al. (2011) Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011.  Im2text: Describing images using 1 million captioned photographs.  _Advances in neural information processing systems_ , 24.
  * Ponti et al. (2019) Edoardo Maria Ponti, Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019.  Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing.  _Computational Linguistics_ , 45(3):559–601.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_.
  * Ramaswamy et al. (2023) Vikram V. Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron B. Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. 2023.  Beyond web-scraping: Crowd-sourcing a geographically diverse image dataset.  _ArXiv_ , abs/2301.02560.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Ren et al. (2021) Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. 2021.  A survey of deep active learning.  _ACM computing surveys (CSUR)_ , 54(9):1–40.
  * Ridnik et al. (2021) T. Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. 2021.  Imagenet-21k pretraining for the masses.  _ArXiv_ , abs/2104.10972.
  * Rojas et al. (2022) William Gaviria Rojas, Sudnya Diamos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. 2022.  The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world.  In _Neural Information Processing Systems_.
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494.
  * Salman et al. (2022) Hadi Salman, Saachi Jain, Andrew Ilyas, Logan Engstrom, Eric Wong, and Aleksander Madry. 2022.  When does bias transfer in transfer learning?  _arXiv preprint arXiv:2207.02842_.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _ArXiv_ , abs/2210.08402.
  * Schuhmann et al. (2021) Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021.  LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs.  In _Proceedings of the NeurIPS Data Centric AI Workshop_.
  * Schumann et al. (2021) Candice Schumann, Susanna Ricco, Utsav Prabhu, Vittorio Ferrari, and Caroline Pantofaru. 2021.  A step toward more inclusive people annotations for fairness.  In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 916–925.
  * Sener and Savarese (2018) Ozan Sener and Silvio Savarese. 2018.  Active learning for convolutional neural networks: A core-set approach.  In _International Conference on Learning Representations_.
  * Shankar et al. (2017) Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D. Sculley. 2017.  No classification without representation: Assessing geodiversity issues in open data sets for the developing world.  _arXiv: Machine Learning_.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565.
  * Sun et al. (2023) Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023.  Eva-clip: Improved training techniques for clip at scale.
  * Taha (2023) Kamal Taha. 2023.  Semi-supervised and un-supervised clustering: A review and experimental evaluation.  _Information Systems_ , page 102178.
  * Tan and Le (2019) Mingxing Tan and Quoc Le. 2019.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _International conference on machine learning_ , pages 6105–6114. PMLR.
  * Vincenty (1975) Thaddeus Vincenty. 1975.  Direct and inverse solutions of geodesics on the ellipsoid with application of nested equations.  _Scientific Research and Essays_ , 23:88–93.
  * Wang et al. (2020) Angelina Wang, Arvind Narayanan, and Olga Russakovsky. 2020.  Revise: A tool for measuring and mitigating bias in visual datasets.  _International Journal of Computer Vision_ , 130:1790 – 1810.
  * Wang and Russakovsky (2023) Angelina Wang and Olga Russakovsky. 2023.  Overcoming bias in pretrained models by manipulating the finetuning dataset.  _arXiv preprint arXiv:2303.06167_.
  * Wang et al. (2021) Xudong Wang, Long Lian, and Stella X. Yu. 2021.  Unsupervised selective labeling for more effective semi-supervised learning.  In _European Conference on Computer Vision_.
  * Yang et al. (2017) Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z Chen. 2017.  Suggestive annotation: A deep active learning framework for biomedical image segmentation.  In _Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part III 20_ , pages 399–407. Springer.
  * Zellers et al. (2022) Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. 2022.  Merlot reserve: Neural script knowledge through vision and language and sound.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16354–16366.
  * Zellers et al. (2021) Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021.  Merlot: Multimodal neural script knowledge models.  In _Neural Information Processing Systems_.
  * Zhang et al. (2021) Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021.  VinVL: Revisiting visual representations in vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5579–5588.
  * Zhang et al. (2023) Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, Yandong Guo, and Lei Zhang. 2023.  Recognize anything: A strong image tagging model.
  * Zhao et al. (2023) Dora Zhao, Jerone Andrews, and Alice Xiang. 2023.  Men also do laundry: Multi-attribute bias amplification.  In _International Conference on Machine Learning_ , pages 42000–42017. PMLR.

##  Appendix A Subjective Topics

The 19 subjective topics that we remove: “favorite home decorations”,
“favourite item in kitchen”, “favourite sports clubs”, “how the most loved
item is used”, “icons”, “idols”, “latest furniture bought”, “looking over the
shoulder”, “most loved item”, “most loved toy”, “most played songs on the
radio”, “music idol”, “next big thing you are planning to buy”, “playing with
most loved toy”, “thing I dream about having”, “things I wish I had”, “using
most loved item”, “youth culture”, “what I wish I could buy”.

##  Appendix B Data Stats

Figure 10: The distribution of countries per topic.

##  Appendix C Research Question 1

### C.1. $USA$ respresentations.

Figure 11: Representative images from the visually different topics in low-
resource $USA$ data/ $L$, and high-resource data/ $H$. In $H$, “clothes” and
“makeup” are shown on people, while in $L$ they are separated in dressers and
containers; in $H$, “spice” is in in large baskets in markets, while in $L$
they are in small containers in people’s houses; in $H$, “ceiling” is shown in
public spaces, while in $L$ is in private homes; in $H$ “medicine” is usually
in bottles, while in $L$ can be in various forms.

### C.2. The effect of data size on the data analysis results.

In Figure 5, $Burundi$ has the lowest similarity score of 0.775 and has very
little data in the heatmap of Figure 3, only 97 images. Note however there are
many counter-examples worth considering, such as countries with fewer images
and high similarity scores (e.g., $Austria$ has eleven images and a similarity
score of 0.862, $Bolivia$ has 37 images and a similarity score of 0.874), or
on the opposing spectrum, countries with more images and low similarity scores
(e.g., $Malawi$ has 390 images and a similarity of 0.776, $Burkina$ $Faso$ has
253 images and a similarity score of 0.789).

Furthermore, at topic level, the similarity scores and data size are not
correlated (Pearson correlation score is -0.02). Similar to the country level,
there are topics with many images and low similarity scores (e.g., $religious$
$building$ has 1,375 images and a similarity of 0.764) and topics with few
images and high similarity scores (e.g., $glasses$ has 42 images and a
similarity of 0.95).

In general, while data size can have an influence on our analysis results, we
believe our work provides helpful strategies for annotation when data size is
insufficient. Our paper is a call to action for future work to collect more
globally diverse data to improve the robustness of the results.

Figure 12: Similarity heatmap of (topic, country) pairs with CLIP visual
representations. The darker, the less similarity between high-resource and
low-resource data for that corresponding (topic, country), the more beneficial
it is to annotate. Empty cells do not have any images for (topic, country).
Best viewed in color. Figure 13: Similarity heatmap of (topic, country) pairs
with ALIGN visual representations. The darker, the less similarity between
high-resource and low-resource data for that corresponding (topic, country),
the more beneficial it is to annotate. Empty cells do not have any images for
(topic, country). Best viewed in color. Figure 14: Similarity heatmap of
(topic, country) pairs with BLIP visual representations. The darker, the less
similarity between high-resource and low-resource data for that corresponding
(topic, country), the more beneficial it is to annotate. Empty cells do not
have any images for (topic, country). Best viewed in color.

Figure 15: PCA for the topic “hand washing” for all countries that contain
this topic in the low-resource data and in the high-resource data. The high-
resource data point is highlighted. The data is represented as the average of
the CLIP representations. Figure 16: PCA for the topic “toilet” for all
countries that contain this topic in the low-resource data and in the high-
resource data. The high-resource data point is highlighted. The data is
represented as the average of the CLIP representations. Figure 17: PCA for the
topic “wall” for all countries that contain this topic in the low-resource
data and in the high-resource data. The high-resource data point is
highlighted. The data is represented as the average of the CLIP
representations.

##  Appendix D Research Question 2

Figure 18: The distribution of average similarity scores per topic. Figure 19:
The distribution of average similarity scores per country.

Figure 20: PCA for the topic “get water” for all countries that contain this
topic in the low-resource data. The data is represented as the average of the
CLIP representations. Figure 21: PCA for the topic “house” for all countries
that contain this topic in the low-resource data. The data is represented as
the average of the CLIP representations. Figure 22: PCA for the topic
“backyard” for all countries that contain this topic in the low-resource data.
The data is represented as the average of the CLIP representations. Figure 23:
Topic classification accuracy (in %) for different target-country data ratios
(e.g., target-country ratio 0.0% is equivalent to 100% replacement ratio). We
replace different ratios of the target-country data with images from: (1) the
most similar countries to the target-country given the target-topic; (2) the
most dissimilar countries to the target-country given the target-topic; (3)
high-resource data of the target-topic; (4) no replacement data/ addition.