# Spacetime Hall-MHD turbulence at sub-ion scales: structures or waves?

Emanuele Papini<EMAIL_ADDRESS>Dipartimento di Fisica e Astronomia,
Università degli Studi di Firenze, via G. Sansone 1, 50019 Sesto Fiorentino,
Italy  INAF - Osservatorio Astrofisico di Arcetri, Largo E. Fermi 5, 50125,
Firenze, Italy  Antonio Cicone  Dipartimento di Ingegneria e Scienze
dell’Informazione e Matematica, Università degli Studi dell’Aquila, via Vetoio
1, 67100 L’Aquila, Italy  INAF - Istituto di Astrofisica e Planetologia
Spaziali, via del Fosso del Cavaliere 100, 00133, Roma, Italy  Istituto
Nazionale di Geofisica e Vulcanologia, Via di Vigna Murata 605, 00143 Roma,
Italy  Luca Franci  School of Physics and Astronomy, Queen Mary University of
London, London E1 4NS, UK  INAF - Osservatorio Astrofisico di Arcetri, Largo
E. Fermi 5, 50125, Firenze, Italy  Mirko Piersanti  INAF - Istituto di
Astrofisica e Planetologia Spaziali, via del Fosso del Cavaliere 100, 00133,
Roma, Italy  Simone Landi  Dipartimento di Fisica e Astronomia, Università
degli Studi di Firenze, via G. Sansone 1, 50019 Sesto Fiorentino, Italy  INAF
- Osservatorio Astrofisico di Arcetri, Largo E. Fermi 5, 50125, Firenze, Italy
Petr Hellinger  Astronomical Institute, Czech Academy of Sciences, Bocni II
1401, 141 00 Prague, Czech Republic  Andrea Verdini  Dipartimento di Fisica e
Astronomia, Università degli Studi di Firenze, via G. Sansone 1, 50019 Sesto
Fiorentino, Italy  INAF - Osservatorio Astrofisico di Arcetri, Largo E. Fermi
5, 50125, Firenze, Italy

(Received May 28, 2021; Accepted July 7, 2021)

###### Abstract

Spatiotemporal properties of two-dimensional (2D) Hall-magnetohydrodynamic
turbulence at intermediate plasma $\beta=2$ are studied by means of Fast
Iterative Filtering, a new technique for the decomposition of nonstationary
nonlinear signals. Results show that the magnetic energy at sub-ion scales is
concentrated in perturbations with frequencies smaller than the ion-cyclotron
(IC) frequency and with polarization properties that are incompatible with
both kinetic Alfvén waves (KAWs) and IC waves. At higher frequencies, we
clearly identify signatures of both whistler waves and KAWs, however their
energetic contribution to the magnetic power spectrum is negligible. We
conclude that the dynamics of 2D Hall-MHD turbulence at sub-ion scales is
mainly driven by localized intermittent structures, with no significant
contribution of wavelike fluctuations.

††journal: Astrophysical Journal Letters

##  1 INTRODUCTION

The physical mechanisms underlying the turbulent cascade in the solar wind and
other space plasma environments below the proton characteristic scales remain
largely unknown. In particular, it is hotly debated whether the energy cascade
at sub-ion scales results from nonlinearly interacting wavelike fluctuations
described in terms of low-frequency kinetic Alfvén waves (KAW) (Schekochihin
et al., 2009; Sahraoui et al., 2009; Salem et al., 2012; Chen et al., 2013b)
and/or whistler waves, eventually complemented by wave-particle interactions
(e.g., Landau damping, Sahraoui et al., 2010; Sulem et al., 2016), or from
spatially-localized highly-intermittent coherent structures (e.g., current
sheets and reconnection sites) where dissipation, heating, and cross-scale
energy transfer are enhanced (Wan et al., 2012; Perri et al., 2012; Osman et
al., 2014; Yang et al., 2017; Cerri & Califano, 2017; Camporeale et al., 2018;
Papini et al., 2020). It is difficult to address this question, since the
intrinsic spatiotemporal multiscale nature of turbulence makes it difficult to
identify and isolate the relevant mechanisms in place. At MHD scales and for
moderate values of the plasma $\beta$, the turbulent eddies carry a
significant part of the energy (Andrés et al., 2017). At sub-ion scales, early
studies highlighting the spacetime Fourier structure of turbulence hint that
the plasma dynamics is dominated by almost static low-frequency features, with
no clear signature of waves (Parashar et al., 2010). Recently, several studies
that exploit advanced techniques for the analysis of the spatial properties of
turbulence provided evidence that the cross-scale energy transfer at sub-ion
scales occurs preferentially in coherent structures through enhanced
dissipation (Yang et al., 2017; Camporeale et al., 2018; Papini et al., 2020).
On one hand, such a picture is consistent with the high levels of
intermittency measured in the solar wind (Osman et al., 2012; Wu et al.,
2013). On the other hand, solar wind turbulence at sub-ion scales shown to
have properties similar to those of KAWs, such as well defined ratios between
the power spectra of magnetic and density fluctuations (Salem et al., 2012;
Chen et al., 2013b) and a reduced magnetic helicity (Matthaeus et al., 1982)
of positive sign at high angles of propagation with respect to the mean
magnetic field (Telloni & Bruno, 2016; Woodham et al., 2018).

In this letter, we report results from a high-resolution spatiotemporal
multiscale study of fully developed turbulence in a 2D Hall-
magnetohydrodynamic (MHD) simulation, with the aim to assess whether nonlinear
wavelike activity at sub-ion scales is energetically relevant to the plasma
dynamics. Previous studies, comparing Hall-MHD models with hybrid (ion-kinetic
and electron-fluid) models (Papini et al., 2019) and fully kinetic models
(González et al., 2019), show that Hall-MHD is able to reproduce the main
magnetic properties of plasma turbulence at sub-ion scales. In turn, hybrid
kinetic models well reproduce in-situ observations (Franci et al., 2020,
2020). The use of a 2D setup allows for a much higher resolution, compared to
a full 3D one, while preserving many spectral and statistical properties of
turbulence (Franci et al., 2018). It is true, however, that employing a 2D
geometry prevents the onset of 3D intrinsic features of plasma turbulence
(e.g., critically balanced turbulence, Goldreich & Sridhar, 1995). The novelty
of this work relies in the concurrent use of, (i), a large high-resolution
spatiotemporal dataset and, (ii), Fast Iterative Filtering, a novel adaptive
technique designed for the decomposition of nonlinear nonstationary signals
(Cicone, 2020) (see Appendix A) which allows to clearly isolate features in
the time-frequency domain.

##  2 Method

We performed a simulation of freely-decaying Alfvénic turbulence on a box of
size $L_{x}\times L_{y}=128d_{i}\times 128d_{i}$, with $1024^{2}$ gridpoints
and a resolution $\Delta x=\Delta y=d_{i}/8$ (where $d_{i}$ is the ion
inertial length), by means of a fully-compressible viscous-resistive nonlinear
Hall-MHD pseudospectral code (see Appendix C and Papini et al., 2019). The
system is initialized with a constant out-of-plane magnetic field
$\bm{B}_{0}=B_{0}\hat{\bm{z}}$ ($\hat{\bm{z}}$ defines the parallel direction)
and a uniform plasma $\beta=2$. Moreover, the $xy$-plane is filled with
incompressible velocity ($\bm{u}$) and magnetic ($\bm{B}$) fluctuations in
energy equipartition and perpendicular to $\bm{B}_{0}$. These fluctuations are
random-phase sinusoidals of constant amplitude and with wavenumbers spanning
$-8\pi/L_{x(y)}\leq k_{x(y)}\leq 8\pi/L_{x(y)}$. Their initial root-mean-
square (rms) amplitude is $u_{\mathrm{rms}}=B_{\mathrm{rms}}\simeq 0.24$.

Our analysis begins at the maximum of turbulent activity at
$t_{0}=195\Omega_{i}^{-1}$ (where $\Omega_{i}$ is the ion-cyclotron
frequency), corresponding to the peak of the root-mean-square of the current
density (Mininni & Pouquet, 2009). The spacetime analysis is performed in the
time interval $[t_{0},t_{1}]=[195\Omega_{i}^{-1},215\Omega_{i}^{-1}]$, in
which all fields are sampled with a cadence of $0.01\Omega_{i}^{-1}$. The
resulting fields are defined on a spatiotemporal grid of $1024\times
1024\times 2001$ points. We separate the contribution of fast and slowly
evolving features by means of a new approach that uses Fourier Transform (FT)
and Fast Iterative Filtering (FIF) (Lin et al., 2009; Cicone et al., 2016) for
the spatial and the temporal decomposition, respectively. We name such mixed
approach as the FTFIF decomposition. The novelty of our analysis is the use of
FIF for the temporal decomposition. FIF is designed to decompose a
nonstationary nonlinear signal into a set of Intrinsic Mode Components (IMCs)
oscillating around zero but with varying amplitude and frequency, plus a
residual or trend. Such decomposition is adaptive, based on the local
characteristic frequencies of the signal, and does not make any assumption on
the shape of the signal to be extracted. As such, FIF offers many advantages
over more traditional methods (e.g, FT and Wavelets) that require assumptions
on the stationarity and/or linearity of the signal. Further details on the FIF
technique and on our FTFIF analysis are found in Appendix A and B.

##  3 Results

Figure 1 shows the $k\omega$-diagram of the power spectrum (or power spectral
density) $\mathrm{P}(k_{\perp},\omega)$ for both the parallel ($\bm{B}_{z}$)
and perpendicular ($\bm{B}_{\perp}$) magnetic field and for the perpendicular
velocity fluctuations ($\bm{u}_{\perp}$), as obtained by the FTFIF
decomposition. The $k\omega$-diagram of a field is computed by firstly
performing a FT in space. Then, a FIF decomposition is performed in time to
obtain a set of IMCs for each Fourier mode. Finally, the power and average
temporal frequency of each IMC is interpolated to a $k\omega$-grid and summed
over all the IMCs (see Appendix B). On average, more than 2,000,000 IMCs were
extracted per field component.

Figure 1: $k\omega$-diagram of the power spectrum obtained from the FTFIF
decomposition of the perpendicular component of the fluid velocity (a) and of
the perpendicular (b) and parallel (c) components of the magnetic field at the
maximum of turbulent activity. Superimposed is the dispersion relation for FW
(dashed) and A/KAW (solid), calculated using the mode ($84.9^{\circ}$) and the
mean ($79.3^{\circ}$) of the inclination angle
$\theta_{Bk}=\arctan(B_{z}/\sqrt{B_{x}^{2}+B_{y}^{2}})$, respectively. The
dot-dashed curve denotes the nonlinear-time frequency
$\omega_{nl}=2\pi/\tau_{nl}=2\pi k_{\perp}\widetilde{u}_{e}(k_{\perp})$(Papini
et al., 2019).

The $k\omega$-diagram of $\bm{u}_{\perp}$ (Fig.1a) shows clear signatures of
wave activity. Two ridges emanating from the origin nicely follow the
theoretical dispersion relations (Pucci et al., 2016) for fast/whistler (FW)
waves and Alfvén/kinetic Alfvén waves (A/KAW) respectively. The dispersion
relation for A/KAW is calculated by using the mean ($79.3^{\circ}$) of the
angle $\theta_{Bk}$ between the magnetic field and the $xy$-plane (where the
$k$ vectors lie). Such ridge is broadened because of the excursion of the
values of $\theta_{Bk}$ and $\beta$ in the simulation at the maximum of the
turbulent activity, which induces changes in the frequencies of the waves
throughout the simulation domain. For FW, the best matching relation is
obtained by using the mode ($84.9^{\circ}$) of $\theta_{Bk}$. This, together
with the fact that the A/KAW ridge shows a wider broadening, suggests that FW
are preferentially excited in regions with very small perpendicular magnetic
fluctuations where the magnetic field is almost parallel to the mean field
$\bm{B}_{0}$, while KAWs are more widely distributed. According to its
polarizations, the FW ridge is visible also in the $k\omega$-spectrum of
$\bm{B}_{z}$ and $\rho$ (not shown). The A/KAW ridge is not as visible, since
it is partially swamped by the turbulent magnetic activity. In the same
figure, we report the frequency $\omega_{nl}=2\pi/\tau_{nl}$ corresponding to
the characteristic nonlinear turbulent time
$\tau_{nl}(k_{\perp})=1/(k_{\perp}\widetilde{u}_{e}(k_{\perp}))$, which
estimates the rate at which turbulence evolves at a given scale
$\ell=2\pi/k_{\perp}$ (Papini et al., 2019) ($\widetilde{u}_{e}(k_{\perp})$
being the amplitude of the electron fluid velocity at that scale). It is
reasonable to assume that the area in the $k\omega$-diagram below
$\omega_{nl}$ is dominated by the turbulent dynamics whereas, above
$\omega>\omega_{nl}$, features (such as the FW ridge and part of the A/KAW
ridge) can evolve freely and independently from the main turbulent cascade.

To evaluate the energy contribution of FW and A/KAW, we calculated their
$k$-power spectra by integrating in frequency $\mathrm{P}(k_{\perp},\omega)$
over the FW and the A/KAW ridge respectively. The integration area of the FW
ridge encompasses all the frequencies $\omega$ within $10\%$ of the
eigenfrequencies of the FW dispersion relation. Instead, the integration area
of the A/KAW ridge was obtained by considering the excursion of the values of
$\theta_{Bk}$ and $\beta$ in the simulation domain, which affects the
frequencies of KAWs throughout the domain, thus causing the broadening of the
ridge. Results are shown in Fig. 2 for $\bm{u}_{\perp}$, $\bm{B}_{\perp}$, and
$\bm{B}_{z}$ (but similar results are obtained for the other fields). For all
the fields, both the FW (in violet) and KAW (in orange) spectra are
energetically negligible at sub-ion scales, being more than one order of
magnitude smaller than the total power (in black). The A/KAW spectrum is not
negligible at fluid-MHD scales, as expected in the case of Alfvénic-
turbulence.

Figure 2: Power spectra $\mathrm{P}_{\bm{u}_{\perp}}(k_{\perp})$ (a),
$\mathrm{P}_{\bm{B}_{\perp}}(k_{\perp})$ (b), and
$\mathrm{P}_{\bm{B}_{z}}(k_{\perp})$ (c) as obtained by integrating over the
FW (violet) and the A/KAW (orange) ridges. The total power spectrum (solid
black), obtained by integrating $\mathrm{P}(k_{\perp},\omega)$ over the whole
frequency domain, nicely matches the isotropized power spectrum (Franci et
al., 2015) obtained from a standard spatial Fourier transform at
$t=195\Omega_{i}^{-1}$ (dashed red). The vertical dotted (dot-dashed) line
denotes the position of the break (dissipation) wavenumber.

Although energetically irrelevant at sub-ion scales, both the FW and the A/KAW
spectra appear to follow well defined power laws. At around the spectral break
($k_{\perp}d_{i}\simeq 2.8$, vertical dotted line) both the FW spectrum (in
$\bm{B}_{z}$) and the A/KAW spectrum (in $\bm{B}_{\perp}$ and $\bm{B}_{z}$)
sharply transition to a well-defined $-11/3$ slope, which persists until
dissipation scales are reached (at $k_{\perp}d_{i}\simeq 12$, vertical dot-
dashed line). The FW spectrum of $\bm{u}_{\perp}$ also shows a $-11/3$ power-
law, because of its polarization properties. The existence of a $-11/3$ power-
law at sub-ion scales is predicted by few Hall-MHD theoretical models (Krishan
& Mahajan, 2004; Galtier & Buchlin, 2007; Meyrand & Galtier, 2012;
Schekochihin et al., 2019). Some of their underlying hypothesis, however, are
inconsistent with what observed here (e.g. the requirement of
$\mathrm{P}_{\bm{u}}(k_{\perp})\gg\mathrm{P}_{\bm{B}}(k_{\perp})$ in Krishan &
Mahajan, 2004; Galtier & Buchlin, 2007; Schekochihin et al., 2019, is
inconsistent with both the FW and A/KAW spectra). This, together with the fact
that the whole FW ridge and the A/KAW ridge (although partially) are located
at frequencies higher than the nonlinear-time frequency $\omega_{nl}$ (see
Fig.1), possibly suggests that FW and KAWs may undergo a separate cascade
independently from the main cascade.

Figure 3: Power spectrum of perpendicular magnetic field fluctuations,
obtained by integrating $\mathrm{P}_{B_{\perp}}(k_{\perp},\omega)$ over all
frequencies (black), over low frequencies $\omega\leq\Omega_{i}$ (blue),
medium frequencies $\omega\leq 3\Omega_{i}$ (red solid), and high frequencies
$\omega>3\Omega_{i}$ (red dashed).

We remark that both the FW and the A/KAW spectra remain energetically
irrelevant even if the integration area over the corresponding ridge is
increased, although the $-11/3$ power-law then tends to disappear.

Instead of being related to FW and/or KAW activity, the energy at sub-ion
scales is concentrated at low frequencies. For instance, Fig. 3 shows that the
power spectrum of $\bm{B}_{\perp}$ (black-solid line), mainly results from the
energetic contribution of slowly evolving structures and/or perturbations with
temporal frequencies smaller than the ion-cyclotron frequency $\Omega_{i}$
(blue-solid line). Going toward smaller scales ($k_{\perp}d_{i}\gtrsim 6$),
the contribution of medium frequencies ($\omega\leq 3\Omega_{i}$, red-solid
curve) becomes important. The energy contained at higher frequencies (red-
dashed curve) is negligible.

##  4 DISCUSSION

The simulation data show that the turbulent energy at kinetic scales is
concentrated at low frequencies and it is not clearly related to KAW activity.
At first glance, these results seem to disagree with previous studies (Chen et
al., 2013a) which used the ratio between magnetic and density power spectra
(the so-called KAW ratios) to detect the presence of KAW activity in the solar
wind. We deem here that it may not be correct to use such ratios as a proxy
for KAWs (as also noted by, e.g., Grošelj et al., 2019). To support such
statement, in Fig. 4a we report the
$\mathrm{P}_{\bm{B}_{z}}(k_{\perp})/\mathrm{P}_{\rho}(k_{\perp})$ ratio of
both the FW (in violet) and the A/KAW (orange) power spectra, together with
the ratio of the spectra integrated over all the frequencies (in black). The
corresponding theoretical ratios (see Appendix E), calculated using the exact
wave solutions (dashed curves) of the linearized Hall-MHD equations (Pucci et
al., 2016) nicely match both the FW and the A/KAW ratios from the simulation,
further confirming the wave nature of the ridges. The approximate formula
$\mathrm{P}_{\bm{B}_{z}}(k_{\perp})/\mathrm{P}_{\rho}(k_{\perp})=(\Gamma\beta)^{2}/4$
for the KAW ratios (see Appendix D) matches only partially both the exact and
the computed KAW ratios, due to the fact that such formula is only valid for
large angles $\theta_{Bk}\gtrsim 88^{\circ}$. Interestingly, the approximate
KAW ratio and the slow/ion-cyclotron (S/IC) exact ratio match well with the
ratio of the total spectra at ion-kinetic scales, thus suggesting the presence
of almost-perpendicular low-frequency KAWs and/or S/IC. To verify such a
possibility, we calculated the ion-velocity polarization ratios (Fig. 4b).
Again, the ratio from the A/KAW ridge nicely match the exact relation at ion-
kinetic scales. Moreover, at MHD scales, the ratio of the total spectra (black
solid curve) follows the exact A/KAW ratio, as expected since turbulence is
Alfvénic at those scales. At ion-kinetic scales, however, neither the
approximate KAW (black-dotted line) nor the exact S/IC (blue-dashed curve)
ratio match with the computed ratio (solid-black curve). This rules out the
presence of (energetically relevant) low-frequency KAWs and S/IC and point to
a different origin for the matching of the approximate KAW ratio with the
ratio of the total spectra in Fig. 4a (e.g. a state of pressure balance at
ion-kinetic scales, see Appendix D).

Figure 4: Ratio
$\mathrm{P}_{\bm{B}_{z}}(k_{\perp})/\mathrm{P}_{\rho}(k_{\perp})$ (a) and
$\mathrm{P}_{\bm{u}_{\perp}}(k_{\perp})/\mathrm{P}_{\bm{u}_{z}}(k_{\perp})$
(b) as obtained by integrating the corresponding $k\omega$-spectra over all
frequencies (black curve), over the FW ridge (violet), and over the A/KAW
ridge (orange). The dashed lines denote the corresponding theoretical ratios
as obtained from the wave solutions of the linearized Hall-MHD equations. The
horizontal dotted lines denote the approximated KAW ratios (see Appendix D).

Our results are constrained by the 2D geometry. Wave properties are not
completely captured in a 2D model, due to the limited access to the parallel
wavenumber ($k_{||}$) space. However, we expect that our results are to some
extent relevant to a more realistic full 3D case. The spectral turbulent
properties measured in 3D simulations are well reproduced by 2D ones (Franci
et al., 2018). Moreover, there are regions in our 2D domain where the local
mean magnetic field has an in-plane component, thus allowing for the
$k_{||}$-space to be partially accessible, especially at small scales. This is
confirmed by the existence of A/KAW and FW ridges satisfying dispersion
relations with $\theta_{Bk}<90^{\circ}$ (see Fig. 1). Finally and more
importantly, theoretical models of KAW turbulence, which require large angles
of propagation ($k_{||}\ll k_{\perp}$), would be captured by our 2D
simulation. In any case, to confirm our findings, we need to extend this study
to a fully 3D numerical setup.

To summarize, our results show that 2D Hall-MHD turbulence at sub-ion scales
is shaped by low frequency features (as reported by Parashar et al., 2010).
This strongly points to a scenario where the plasma dynamics is dominated by
highly-intermittent slowly-evolving structures where enhanced dissipation
occurs (Yang et al., 2017; Camporeale et al., 2018; Papini et al., 2020). Our
results indicate that 2D Hall-MHD turbulence on sub-ion scales is not well
described in terms of KAW turbulence phenomenology (Howes et al., 2008;
Schekochihin et al., 2009; Boldyrev et al., 2013). KAW ratios (Chen et al.,
2013a) are not a good proxy for discriminating KAW activity, because they only
approximate the properties of KAWs and are also sensitive to the presence of
S/IC waves and to pressure-balanced structures (see Appendix D and Verscharen
et al., 2017; Grošelj et al., 2019).

Fundamental processes, such as wave-particle resonances and interactions
(Markovskii et al., 2006; Sahraoui et al., 2010; Sulem et al., 2016), as well
as nongyrotropic effects (Del Sarto et al., 2016), may also contribute to
dissipation and heating in space plasmas. Such processes are not properly
captured by Hall-MHD models. In this context, future studies extending the
FTFIF analysis to large 3D simulations that retain a fully-kinetic description
of plasmas (Wan et al., 2015; Franci et al., 2018; Grošelj et al., 2019;
Franci et al., 2020) will provide fundamental insights into the dynamics and
heating in turbulence.

The authors wish to thank the anonymous referee for his constructive review
and useful comments. M. Piersanti thanks the Italian Space Agency for the
financial support under the contract ASI ”LIMADOU scienza +” n∘ 2020-31-HH.0.
A. Cicone is a member of the Italian “Gruppo Nazionale di Calcolo Scientifico”
(GNCS) of the Istituto Nazionale di Alta Matematica “Francesco Severi”
(INdAM). L. Franci is supported by the UK Science and Technology Facilities
Council (STFC) grant ST/T00018X/1. This work was supported by the Programme
National PNST of CNRS/INSU co-funded by CNES. We acknowledge partial funding
by “Fondazione Cassa di Risparmio di Firenze” under the project HIPERCRHEL. We
acknowledge the computing centre of CINECA and INAF, under the coordination of
the ”Accordo Quadro MoU per lo svolgimento di attività congiunta di ricerca
Nuove frontiere in Astrofisica: HPC e Data Exploration di nuova generazione”,
for the availability of computing resources and support (project INA20_C6A55)
E. Papini acknowledges CINECA for awarding access to HPC resources under the
ISCRA initiative (grant HP10C2EARF and HP10C4A4M2). The MATLAB code
implementing the FIF algorithm is available at www.cicone.com. The authors
thank L. Matteini and V. Montagud-Camps for useful discussion.

##  Appendix A Fast Iterarive Filtering

Iterative Filtering (IF) (Lin et al., 2009; Cicone et al., 2016) is a
technique for the analysis of nonlinear nonstationary signals. IF decomposes a
given $L^{2}$ signal $f(t)$ into $N$ simple oscillating functions
$\widehat{f}_{m}(t)$ called intrinsic mode components (IMCs), such that

$f(t)=\sum_{m=1}^{N}\widehat{f}_{m}(t)+r(t),$ (A1)

where $r(t)$ is the residual (or trend) of the decomposition. IF uses a low-
pass filter, applied iteratively, to extract the moving average of the signal
at a given timescale $\tau_{m}$, in order to isolate a fluctuating component
whose average frequency $\omega_{m}\sim 2\pi/\tau_{m}$ is well behaved. Each
IMC is given by

$\widehat{f}_{m}(t)=\lim_{n\rightarrow\infty}\mathcal{S}_{\tau_{m}}^{n}\left[f(t)-\sum_{l=1}^{m-1}\widehat{f}_{l}(t)\right],$
(A2)

where $\mathcal{S}_{\tau_{m}}=\mathcal{I}-\mathcal{L}_{\tau_{m}}$,
$\mathcal{I}$ is the identity operator, $\mathcal{L}_{\tau_{m}}$ is the
integral operator associated to the low-pass filter (Papini et al., 2020), and
where $n$ denotes the number of times the operator $S_{\tau_{m}}$ is applied.
$\tau_{m}$ is different for each IMC and increasing with $m$. Therefore, IMCs
with increasing $m$ will contain smaller frequencies $\omega_{m}$. The low-
pass filter operator reads

$\mathcal{L}_{\tau_{m}}[s(t)]=\int_{-\tau_{m}}^{\tau_{m}}s(t+\tau)w_{m}(\tau)\mathrm{d}\tau$
(A3)

where $w_{m}(t)\,\epsilon\,[-\tau_{m},\tau_{m}]$ is the kernel function
associated to the filter, with compact support $[-\tau_{m},\tau_{m}]$. Here we
employ a Fokker-Plank filter Cicone et al. (2016). This approach has been
recently accelerated in what is known as Fast Iterative Filtering (FIF) Cicone
(2020).

##  Appendix B Space-time analysis with Iterative Filtering

The procedure used to calculate the $k\omega$-diagram is the following. We
consider a field $F(x,y,t)$, with $(x,y)$ defined in a 2D periodic domain on a
discrete grid of $N_{x}\times N_{y}$ points and with $t\in[t_{0},t_{1}]$.
$F(x,y,t)$ can be either a scalar field (e.g. density) or a vector component
(e.g. $B_{z}$). The space-time analysis, employs a mixed approach. We perform
a Fourier transform (FT) in space and we then apply FIF in time. The resulting
FTFIF decomposition consists of the following steps:

  1. 1.

Perform a Fourier transform of $F(x,y,t)$ to obtain its complex Fourier
spectrum $\widetilde{F}(k_{x}^{i},k_{y}^{j},t)=\widetilde{F}^{ij}(t)$, with
$i(j)=1,...N_{x}(N_{y})$.

  2. 2.

For each real (imaginary) component $F^{ij}(t)$ of $\widetilde{F}^{ij}(t)$,
perform an FIF decomposition to obtain $N^{ij}$ IMCs:

$F^{ij}(t)=\sum_{m=1}^{N^{ij}}\widehat{F}_{m}^{ij}(t)+r^{ij}(t).$ (B1)

  3. 3.

For each pair $(i,j)$, increase the orthogonality of the set
$\\{\widehat{F}_{m}^{ij}(t)\\}$ by calculating the cross-correlation matrix

$\text{Corr}_{l,n}=\langle\widehat{F}_{l}^{ij}\cdot\widehat{F}_{n}^{ij}\rangle/\sqrt{\langle(\widehat{F}_{l}^{ij})^{2}\rangle\langle(\widehat{F}_{n}^{ij})^{2}\rangle}$

where

$\langle\widehat{F}_{l}^{ij}\cdot\widehat{F}_{n}^{ij}\rangle=\int_{t_{0}}^{t_{1}}\widehat{F}_{l}^{ij}(t)\widehat{F}_{n}^{ij}(t)\mathrm{d}t.$

Sum the IMCs for which $\text{Corr}_{l,n}>0.6$, to obtain a reduced set
$\\{\widehat{F}_{m}^{ij}(t)\\}$ (with $m=1,M^{ij}$) which is almost
orthogonal.

  4. 4.

For each IMC, calculate its average frequency $\omega_{m}^{ij}$ and its
amplitude $A_{m}^{ij}=\sqrt{\langle(\widehat{F}_{m}^{ij})^{2}\rangle}$. At the
end you have a set of frequencies and amplitudes
$\\{\omega_{m}^{ij},A_{m}^{ij}\\}$, with $m=1,M^{ij}$ and
$i(j)=1,...N_{x}(N_{y})$.

  5. 5.

Define an (arbitrary) equidistant $k\omega$-grid and interpolate the energy
$(A_{m}^{ij})^{2}$ of each IMC to the four points nearest to
$\left(k_{\perp}^{ij}=\sqrt{k_{x}^{i,2}+k_{y}^{j,2}},\omega_{m}^{ij}\right)$,
such that energy is conserved:

$(A_{m}^{ij})^{2}=\sum_{k_{\perp}=k_{1},k_{2}}\sum_{\omega=\omega_{1},\omega_{2}}\mathcal{B}\left[(A_{m}^{ij})^{2}\right](k_{\perp},\omega),$

where $\mathcal{B}$ is the interpolation operator and all pair combinations
$(k_{1,2},\omega_{1,2})$ denote the coordinates of the four nearest points.

  6. 6.

Finally, sum over $i$,$j$, and $m$ to obtain the $k\omega$-power spectrum of
$F$

$\mathrm{P}_{F}(k_{\perp},\omega)=\frac{1}{\Delta
T}\sum_{i=1}^{N_{x}}\sum_{j=1}^{N_{y}}\sum_{m=1}^{M^{ij}}\mathcal{B}\left[(A_{m}^{ij})^{2}\right](k_{\perp},\omega),$
(B2)

where $\Delta T=t_{1}-t_{0}$ is the length of the temporal interval
considered.

##  Appendix C Hall-MHD model

The pseudospectral code employed in this work (Papini et al., 2019) solves the
fully-compressible viscous-resistive Hall-MHD equations (in adimensionalized
form)

$\displaystyle\partial_{t}\rho$ $\displaystyle=$
$\displaystyle-\nabla\cdot{(\rho\bm{u})},$ (C1)
$\displaystyle\rho\mathrm{d}_{t}\bm{u}$ $\displaystyle=$ $\displaystyle-\nabla
P+(\nabla\times{\bm{B}})\times\bm{B}+$ (C2)
$\displaystyle+\mu\left[\nabla^{2}\bm{u}+\nabla(\nabla\cdot{\bm{u}})/3\right],$
$\displaystyle\mathrm{d}_{t}T$ $\displaystyle=$
$\displaystyle~{}(\Gamma-1)\\!\left\\{-(\nabla\cdot{\bm{u}})T\\!+\\!\eta|\nabla\times\bm{B}|^{2}/\rho+\right.$
(C3)
$\displaystyle\left.+\mu\rho^{-1}\left[(\nabla\times{\bm{u}})^{2}+4/3(\nabla\cdot{\bm{u}})^{2}\right]\right\\},$
$\displaystyle\partial_{t}{\bm{B}}$ $\displaystyle=$
$\displaystyle~{}\nabla\\!\times\\!\left(\bm{u}\times\bm{B}\right)\\!+\\!\eta\nabla^{2}\bm{B}+$
(C4) $\displaystyle-
d_{i}/L\nabla\\!\times\\![(\nabla\\!\times\\!\bm{B})\\!\times\\!\bm{B}/\rho],$

where $\mathrm{d}_{t}=\partial_{t}+\bm{u}\cdot\nabla$, $\Gamma=5/3$ is the
adiabatic index and $\\{\rho,\bm{u},\bm{B},T,P\\}$ denote density, ion-fluid
velocity, magnetic field, temperature, and pressure respectively.
Thermodynamic variables are related through the equation of state $P=\rho T$.
All quantities are normalized with respect to a characteristic length $L$ (set
to be equal to the ion inertial length $d_{i}$), a plasma density $\rho_{0}$,
a magnetic field amplitude $B_{0}$, the Alfvén velocity
$c_{A}=B_{0}/\sqrt{4\pi\rho_{0}}=\Omega_{i}d_{i}$, a pressure
$P_{0}=\rho_{0}c_{A}^{2}$, and a plasma temperature
$T_{0}=(k_{B}/m_{i})P_{0}/\rho_{0}$. $\Omega_{i}=eB_{0}/m_{i}c$ is the ion-
cyclotron angular frequency and $m_{i}$ is the ion’s mass. With this
normalization, the Alfvén time is $\tau_{A}=d_{i}/c_{A}=\Omega_{i}^{-1}$,
dynamic viscosity and magnetic resistivity are in units of
$d_{i}c_{A}\rho_{0}$ and $d_{i}c_{A}$ respectively, and the Hall coefficient
$d_{i}/L=1$ in Eq. (C4). In the above equations, the magnetic field $\bm{B}$
is expressed in units of the Alfvén velocity. In code units, the electron
fluid velocity is $\bm{u}_{e}=\bm{u}_{i}-\nabla\times\bm{B}/\rho$.

Equations (C1-C4) are solved in a 2.5D periodic cartesian $(x,y)$ domain, by
employing Fourier transform for dealiasing and calculating spatial derivatives
and by using a 3rd-order Runge-Kutta method for time integration. In 2.5D
codes, the space of the coordinates is 2D, vector quantities retain all three
$xyz$-components, and $\partial_{z}=0$ for all variables. The simulation
analyzed in the present work employs a 2D box of size $L_{x}\times
L_{y}=128d_{i}\times 128d_{i}$, with $1024^{2}$ gridpoints and a resolution
$\Delta x=\Delta y=d_{i}/8$. Finally, we set the plasma $\beta=2P_{0}/B_{0}=2$
and $\mu=\eta=0.001$.

##  Appendix D Pressure-balanced structures and KAW ratios in Hall-MHD

Following (Boldyrev et al., 2013), the ratio $(\delta
B_{z}(k)/\delta\rho(k))^{2}$ for KAWs, obtained in the limit $k_{z}\ll
k_{\perp}$, is given by the formula

$\left(\frac{\delta
B_{z}(k)/B_{0}}{\delta\rho(k)/\rho_{0}}\right)^{2}=\left[\frac{v_{T_{i}}^{2}}{c_{A}^{2}}\left(1+\frac{T_{e}}{T_{i}}\right)\right]^{2}$
(D1)

where $T_{i}$ ($T_{e}$) is the ion (electron) temperature, and $v_{T_{i}}$ is
the thermal speed of the ions. The above expression can be adapted to the case
of adiabatic HMHD by substituting the ion thermal speed with the adiabatic
sound speed ($v_{T_{i}}\rightarrow c_{s}$) and by using the definition
$\beta=(2/\Gamma)c_{s}^{2}/c_{A}^{2}$ together with
$T_{e}/T_{i}=\beta_{e}/\beta_{i}$ (where we assumed charge neutrality
$n_{e}=n_{i}=\rho/m_{i}$) to give

$\left(\frac{\delta
B_{z}(k)/B_{0}}{\delta\rho(k)/\rho_{0}}\right)^{2}=\left[\frac{\Gamma}{2}(\beta_{i}+\beta_{e})\right]^{2}=\frac{(\Gamma\beta)^{2}}{4},$
(D2)

where $\beta=\beta_{i}+\beta_{e}=(2/\Gamma)c_{s}^{2}/c_{A}^{2}$ and
$c_{s}^{2}=\Gamma P_{0}/\rho_{0}$. We note that the above formula can be
obtained self-consistently from the KAW linear solutions of ideal HMHD (see
Eq. E2) in the limit $k_{z}\ll k_{\perp}$.

Eq. D2 is also fullfilled by slowly evolving pressure-balanced structures. In
Fourier $k\omega$-space, in the limit of low frequencies and for wavenumbers
smaller than the inverse Kolmogorov dissipation scale, Eq. C2 reads

$\delta P(k)+\frac{\delta B_{z}(k)B_{0}}{4\pi}\simeq 0$ (D3)

to first order in the turbulent fluctuations. By using $\delta
P=c_{s}^{2}\delta\rho$, we then obatin

$c_{s}^{2}\delta\rho(k)\simeq-\frac{\delta B_{z}(k)B_{0}}{4\pi}.$ (D4)

The latter equation may be rearranged to give the ratio between the parallel
magnetic fluctuations $\delta B_{z}(k)$ and the density fluctuations
$\delta\rho(k)$, to obtain

$\frac{\delta
B_{z}(k)^{2}}{\delta\rho(k)^{2}}=\left(\frac{c_{s}^{2}}{c_{A}^{2}}\frac{B_{0}}{\rho_{0}}\right)^{2}=\left(\frac{\Gamma\beta}{2}\frac{B_{0}}{\rho_{0}}\right)^{2}$
(D5)

which, in adimensional units and by using $\mathrm{P}_{X}(k)=(\delta
X(k))^{2}/k$, gives the expression

$\frac{\mathrm{P}_{B_{z}}}{\mathrm{P}_{\rho}}=\frac{(\Gamma\beta)^{2}}{4}$
(D6)

that is equal to Eq. (D2) for the approximated KAW ratios.

##  Appendix E Polarization ratios

In presence of the Hall term and below ion scales ($kd_{i}\gtrsim 1$), the
three MHD wave solutions (fast, slow, and Alfvén) are modified. The fast mode
turns into a whistler mode (FW), the Alfvén mode transitions to a Kinetic
Alfvén mode (A/KAW), and the slow mode becomes a (compressible) ion-cyclotron
mode (S/IC) (e.g., Hollweg, 2000; Galtier, 2016). The polarization ratios for
FW, A/KAW, and S/IC waves are calculated from the eigenmode solutions
$\\{\delta\bm{u}(\omega,\bm{k}),\,\delta\bm{B}(\omega,\bm{k}),\delta\rho(\omega,\bm{k})\\}$
of the linearized Hall-MHD equations. The expression for the dispersion
relation, $\delta\bm{u}$, and $\delta\bm{B}$ can be found in Pucci et al.
(2016). Following their convention, here we write the expression for the
amplitude of the density

$\delta\rho(\omega,\bm{k})=\left|\rho_{0}a\frac{(k_{||}^{2}c_{A}^{2}-\omega^{2})k_{\perp}\Omega_{i}}{(\omega^{2}-k^{2}c_{s}^{2})k_{||}\omega}\right|$
(E1)

where $\omega$ is the eigenfrequency, $a$ is the perturbation amplitude, and
$k_{||(\perp)}$ is the parallel (perpendicular) component of the wave vector
$\bm{k}$ with respect to the background magnetic field, thus
$\tan(\theta_{Bk})=k_{\perp}/k_{||}$. Note that the definition of parallel
direction used here differs from that of the numerical simulation, in which
the parallel direction is that of the global mean magnetic field (i.e. the
$z$-direction).

The exact formulas for the polarization ratios shown in this work read

$\left(\frac{\delta B_{||}\rho_{0}}{\delta\rho
B_{0}}\right)^{2}=\frac{(\omega^{2}-k^{2}c_{s}^{2})^{2}}{k^{4}c_{A}^{4}},$
(E2) $\left(\frac{\delta B_{\perp}\rho_{0}}{\delta\rho
B_{0}}\right)^{2}=\frac{(\omega^{2}-k^{2}c_{s}^{2})^{2}}{(k_{\perp}/k_{||})^{2}}\left[\frac{(\omega/\Omega_{i})^{2}}{(\omega^{2}-k_{||}^{2}c_{A}^{2})^{2}}+\frac{1}{k^{4}c_{A}^{4}}\right],$
(E3) $\left(\frac{\delta u_{\perp}}{\delta
u_{||}}\right)^{2}=\left[\frac{\omega
k_{||}c_{A}^{2}(\omega^{2}-k^{2}c_{s}^{2})}{\Omega_{i}k_{\perp}c_{s}^{2}(\omega^{2}-k_{||}^{2}c_{A}^{2})}\right]^{2}+\left[\frac{(\omega^{2}-k_{||}^{2}c_{s}^{2})}{k_{||}k_{\perp}c_{s}^{2}}\right]^{2},$
(E4)

where the ratio for the desired branch (FW, A/KAW, or S/IC) is obtained by
plugging in the corresponding eigenfrequency $\omega$ (whose analytical
expression can be found in Vocks et al., 1999). The analogous ratios from the
numerical simulation have been obtained by using $\delta
B_{\perp}^{2}/k=\mathrm{P}_{B_{\perp}}$, $\delta
B_{||}^{2}/k=\mathrm{P}_{B_{z}}$, $\delta
u_{\perp}^{2}/k=\mathrm{P}_{u_{\perp}}$, and $\delta
u_{||}^{2}/k=\mathrm{P}_{u_{z}}$. Eqs. (E2) and (E4) give the exact ratios
used in Fig. 4.

Figure 5: Ratio
$\mathrm{P}_{\bm{B}_{\perp}}(k_{\perp})/\mathrm{P}_{\rho}(k_{\perp})$ as
obtained by integrating the corresponding $k\omega$-spectra over all
frequencies (black curve) and over the A/KAW ridge (orange). The dashed lines
denote the corresponding theoretical ratios as obtained from the wave
solutions of the linearized Hall-MHD equations. The horizontal dotted lines
denote the approximated KAW ratios (see Appendix).

In Fig. 5 we report the polarization ratios for
$\mathrm{P}_{\bm{B}_{\perp}}(k_{\perp})/\mathrm{P}_{\rho}(k_{\perp})$. Again,
as shown in Fig. 4, the A/KAW ratio nicely matches the exact relations at ion-
kinetic scales. Moreover, at MHD scales, the ratio of the total spectra (black
solid curve) follows the exact A/KAW ratio, as expected since turbulence is
Alfvénic at those scales. At ion-kinetic scales however, this is not the case.
The approximate KAW ratio (horizontal dotted lines) matches with the ratio of
the total spectra for $\mathrm{P}_{\bm{B}_{\perp}}/\mathrm{P}_{\rho}$,
(although such match is not as good as in the case of
$\mathrm{P}_{\bm{B}_{z}}/\mathrm{P}_{\rho}$). However, the presence of
energetically relevant low-frequency KAWs has already been ruled out by the
$\delta u_{\perp}^{2}/\delta u_{z}^{2}$ ratio. This casts some doubts on the
use of $\mathrm{P}_{\bm{B}_{\perp}}/\mathrm{P}_{\rho}$ as a proxy for KAW
activity.

The approximate relations for the
$\mathrm{P}_{\bm{B}_{\perp}}/\mathrm{P}_{\rho}$ and the
$\mathrm{P}_{\bm{u}_{\perp}}/\mathrm{P}_{\bm{u}_{z}}$ KAW ratios are

$\left(\frac{\delta B_{\perp}\rho_{0}}{\delta\rho
B_{0}}\right)^{2}\simeq\frac{\Gamma\beta}{2}\left(1+\frac{\Gamma\beta}{2}\right),$
(E5) $\left(\frac{\delta u_{\perp}}{\delta
u_{||}}\right)^{2}\simeq\frac{2}{\Gamma\beta}+1$ (E6)

## References

  * Andrés et al. (2017) Andrés, N., Clark di Leoni, P., Mininni, P. D., et al. 2017, PhPl, 24, 102314, doi: 10.1063/1.4997990
  * Boldyrev et al. (2013) Boldyrev, S., Horaites, K., Xia, Q., & Perez, J. C. 2013, ApJ, 777, 41, doi: 10.1088/0004-637X/777/1/41
  * Camporeale et al. (2018) Camporeale, E., Sorriso-Valvo, L., Califano, F., & Retinò, A. 2018, Phys. Rev. Lett., 120, 125101, doi: 10.1103/PhysRevLett.120.125101
  * Cerri & Califano (2017) Cerri, S. S., & Califano, F. 2017, NJPh, 19, 025007, doi: 10.1088/1367-2630/aa5c4a
  * Chen et al. (2013a) Chen, C. H. K., Bale, S. D., Salem, C. S., & Maruca, B. A. 2013a, ApJ, 770, 125, doi: 10.1088/0004-637X/770/2/125
  * Chen et al. (2013b) Chen, C. H. K., Boldyrev, S., Xia, Q., & Perez, J. C. 2013b, Phys. Rev. Lett., 110, 225002, doi: 10.1103/PhysRevLett.110.225002
  * Cicone (2020) Cicone, A. 2020, NuAlg, 85, 811, doi: 10.1007/s11075-019-00838-z
  * Cicone et al. (2016) Cicone, A., Liu, J., & Zhou, H. 2016, Applied and Computational Harmonic Analysis, 41, 384 , doi: https://doi.org/10.1016/j.acha.2016.03.001
  * Del Sarto et al. (2016) Del Sarto, D., Pegoraro, F., & Califano, F. 2016, Phys. Rev. E, 93, 053203, doi: 10.1103/PhysRevE.93.053203
  * Franci et al. (2018) Franci, L., Hellinger, P., Guarrasi, M., et al. 2018, JPhCS, 1031, 012002, doi: 10.1088/1742-6596/1031/1/012002
  * Franci et al. (2015) Franci, L., Verdini, A., Matteini, L., Landi, S., & Hellinger, P. 2015, ApJ, 804, L39, doi: 10.1088/2041-8205/804/2/L39
  * Franci et al. (2020) Franci, L., Stawarz, J. E., Papini, E., et al. 2020, ApJ, 898, 175, doi: 10.3847/1538-4357/ab9a47
  * Franci et al. (2020) Franci, L., Del Sarto, D., Papini, E., et al. 2020, submitted to ApJ
  * Galtier (2016) Galtier, S. 2016, Introduction to Modern Magnetohydrodynamics
  * Galtier & Buchlin (2007) Galtier, S., & Buchlin, E. 2007, ApJ, 656, 560, doi: 10.1086/510423
  * Goldreich & Sridhar (1995) Goldreich, P., & Sridhar, S. 1995, ApJ, 438, 763, doi: 10.1086/175121
  * González et al. (2019) González, C. A., Parashar, T. N., Gomez, D., Matthaeus, W. H., & Dmitruk, P. 2019, PhPl, 26, 012306, doi: 10.1063/1.5054110
  * Grošelj et al. (2019) Grošelj, D., Chen, C. H. K., Mallet, A., et al. 2019, PhRvX, 9, 031037, doi: 10.1103/PhysRevX.9.031037
  * Hollweg (2000) Hollweg, J. V. 2000, J. Geophys. Res., 105, 7573, doi: 10.1029/1999JA000326
  * Howes et al. (2008) Howes, G. G., Cowley, S. C., Dorland, W., et al. 2008, JGRA, 113, A05103, doi: 10.1029/2007JA012665
  * Krishan & Mahajan (2004) Krishan, V., & Mahajan, S. M. 2004, JGRA, 109, A11105, doi: 10.1029/2004JA010496
  * Lin et al. (2009) Lin, L., Wang, Y., & Zhou, H. 2009, Advances in Adaptive Data Analysis, 01, 543, doi: 10.1142/S179353690900028X
  * Markovskii et al. (2006) Markovskii, S. A., Vasquez, B. J., Smith, C. W., & Hollweg, J. V. 2006, ApJ, 639, 1177, doi: 10.1086/499398
  * Matthaeus et al. (1982) Matthaeus, W. H., Goldstein, M. L., & Smith, C. 1982, Phys. Rev. Lett., 48, 1256, doi: 10.1103/PhysRevLett.48.1256
  * Meyrand & Galtier (2012) Meyrand, R., & Galtier, S. 2012, Phys. Rev. Lett., 109, 194501, doi: 10.1103/PhysRevLett.109.194501
  * Mininni & Pouquet (2009) Mininni, P. D., & Pouquet, A. 2009, Phys. Rev. E, 80, 025401, doi: 10.1103/PhysRevE.80.025401
  * Osman et al. (2014) Osman, K. T., Matthaeus, W. H., Gosling, J. T., et al. 2014, Phys. Rev. Lett., 112, 215002, doi: 10.1103/PhysRevLett.112.215002
  * Osman et al. (2012) Osman, K. T., Matthaeus, W. H., Wan, M., & Rappazzo, A. F. 2012, Phys. Rev. Lett., 108, 261102, doi: 10.1103/PhysRevLett.108.261102
  * Papini et al. (2020) Papini, E., Cicone, A., Piersanti, M., et al. 2020, JPlPh, 86, 871860501, doi: 10.1017/S0022377820001221
  * Papini et al. (2019) Papini, E., Franci, L., Landi, S., et al. 2019, ApJ, 870, 52, doi: 10.3847/1538-4357/aaf003
  * Parashar et al. (2010) Parashar, T. N., Servidio, S., Breech, B., Shay, M. A., & Matthaeus, W. H. 2010, PhPl, 17, 102304, doi: 10.1063/1.3486537
  * Perri et al. (2012) Perri, S., Goldstein, M. L., Dorelli, J. C., & Sahraoui, F. 2012, Phys. Rev. Lett., 109, 191101, doi: 10.1103/PhysRevLett.109.191101
  * Pucci et al. (2016) Pucci, F., Vásconez, C. L., Pezzi, O., et al. 2016, JGRA, 121, 1024, doi: 10.1002/2015JA022216
  * Sahraoui et al. (2010) Sahraoui, F., Goldstein, M. L., Belmont, G., Canu, P., & Rezeau, L. 2010, Phys. Rev. Lett., 105, 131101, doi: 10.1103/PhysRevLett.105.131101
  * Sahraoui et al. (2009) Sahraoui, F., Goldstein, M. L., Robert, P., & Khotyaintsev, Y. V. 2009, Phys. Rev. Lett., 102, 231102, doi: 10.1103/PhysRevLett.102.231102
  * Salem et al. (2012) Salem, C. S., Howes, G. G., Sundkvist, D., et al. 2012, ApJ, 745, L9, doi: 10.1088/2041-8205/745/1/L9
  * Schekochihin et al. (2009) Schekochihin, A. A., Cowley, S. C., Dorland, W., et al. 2009, ApJS, 182, 310, doi: 10.1088/0067-0049/182/1/310
  * Schekochihin et al. (2019) Schekochihin, A. A., Kawazura, Y., & Barnes, M. A. 2019, JPlPh, 85, 905850303, doi: 10.1017/S0022377819000345
  * Sulem et al. (2016) Sulem, P. L., Passot, T., Laveder, D., & Borgogno, D. 2016, ApJ, 818, 66, doi: 10.3847/0004-637X/818/1/66
  * Telloni & Bruno (2016) Telloni, D., & Bruno, R. 2016, MNRAS, 463, L79, doi: 1doi:10.1093/mnrasl/slw135
  * Verscharen et al. (2017) Verscharen, D., Chen, C. H. K., & Wicks, R. T. 2017, ApJ, 840, 106, doi: 10.3847/1538-4357/aa6a56
  * Vocks et al. (1999) Vocks, C., Motschmann, U., & Glassmeier, K. H. 1999, Annales Geophysicae, 17, 712, doi: 10.1007/s00585-999-0712-2
  * Wan et al. (2015) Wan, M., Matthaeus, W. H., Roytershteyn, V., et al. 2015, Phys. Rev. Lett., 114, 175002, doi: 10.1103/PhysRevLett.114.175002
  * Wan et al. (2012) Wan, M., Matthaeus, W. H., Karimabadi, H., et al. 2012, Phys. Rev. Lett., 109, 195001, doi: 10.1103/PhysRevLett.109.195001
  * Woodham et al. (2018) Woodham, L. D., Wicks, R. T., Verscharen, D., & Owen, C. J. 2018, ApJ, 856, 49, doi: 10.3847/1538-4357/aab03d
  * Wu et al. (2013) Wu, P., Perri, S., Osman, K., et al. 2013, ApJ, 763, L30, doi: 10.1088/2041-8205/763/2/L30
  * Yang et al. (2017) Yang, Y., Matthaeus, W. H., Parashar, T. N., et al. 2017, PhPl, 24, 072306, doi: 10.1063/1.4990421


# Vision-Language Models under Cultural and Inclusive Considerations

Antonia Karamolegkou, Phillip Rust, Yong Cao,
Ruixiang Cui, Anders Søgaard, Daniel Hershcovich

Department of Computer Science, University of Copenhagen
Correspondence<EMAIL_ADDRESS>
###### Abstract

Large vision-language models (VLMs) can assist visually impaired people by
describing images from their daily lives. Current evaluation datasets may not
reflect diverse cultural user backgrounds or the situational context of this
use case. To address this problem, we create a survey to determine caption
preferences and propose a culture-centric evaluation benchmark by filtering
VizWiz, an existing dataset with images taken by people who are blind. We then
evaluate several VLMs, investigating their reliability as visual assistants in
a culturally diverse setting. While our results for state-of-the-art models
are promising, we identify challenges such as hallucination and misalignment
of automatic evaluation metrics with human judgment. We make our survey, data,
code, and model outputs publicly available.

coastalcph/vizwiz-culture

Vision-Language Models under Cultural and Inclusive Considerations

Antonia Karamolegkou, Phillip Rust, Yong Cao, Ruixiang Cui, Anders Søgaard,
Daniel Hershcovich Department of Computer Science, University of Copenhagen
Correspondence<EMAIL_ADDRESS>

##  1 Introduction

With the increasing integration of AI applications into our lives, it is
important to consider human-centered use cases when evaluating such systems.
Large multimodal language models are now used as visual assistants for blind
and visually impaired individuals. Given that people across different cultures
use such applications, it is essential to ensure not only their accuracy and
faithfulness (Brady et al., 2013; Gonzalez et al., 2024) but also their
cultural representation and inclusion (Hershcovich et al., 2022; Shi et al.,
2024).

Figure 1: Survey results from people with visual impairments rating importance
and helpfulness of cultural information in image captions. We use a Likert
scale from 1 (not important/helpful) to 5 (very important/helpful).

Existing evaluation benchmarks for VLMs focus primarily on English with few,
implicit mutlicultural references. Although multicultural evaluation datasets
like MaRVL (Liu et al., 2021) and XM3600 (Thapliyal et al., 2022) include
culture-specific images (e.g., traditional wedding costumes), they also
contain images with minimal cultural significance (e.g., a bag of carrots).
Consequently, these datasets may not accurately measure the cultural knowledge
of VLMs, despite being useful for assessing their multilingual capabilities.
Additionally, evaluating these systems as visual assistants presents further
challenges due to varying photo quality, user goals, and photo content (Chiu
et al., 2020; Jung et al., 2022). Recently, Gonzalez et al. (2024) conducted a
diary study with blind and low-vision individuals using an AI-powered scene
description application, revealing that significant improvements are still
needed for satisfying and trustworthy user experiences.

To address both cultural and visual challenges, we first surveyed visually
impaired individuals to gather their caption preferences and determine if
cultural details are necessary. Then, we filtered an existing dataset with
images taken from people who are blind, identifying implicit cultural
concepts. This is used as a challenging benchmark to evaluate image captioning
performance on cultural images of state-of-the-art models across different
prompt settings. With these experiments, we investigate how AI applications,
such as image captioning, can foster a more inclusive and culture-aware
experience for all.

#### Background

Current models are trained without consideration for the subjective
perspectives and cultural influences of those who provided the image
descriptions (Ye et al., 2023). This raises the need for carefully curated
sources of data and annotation paradigms that are more culturally aware and
inclusive (Arora et al., 2023; Cao et al., 2023). Lately, there has been a
growing body of work releasing multicultural multimodal datasets for
visiolinguistic reasoning (Liu et al., 2021), text to image generation (Liu et
al., 2023b; Ventura et al., 2023), and image captioning (Thapliyal et al.,
2022). Beyond the focus on the multilingualism of the captions, concurrent
work also addresses the cultural concepts depicted in the images Cao et al.
(2024); Burda-Lassen et al. (2024); Romero et al. (2024); Mukherjee et al.
(2024); Bhatia et al. (2024). However, they still do not take into account
specific use cases, such as visual assistance. Gurari et al. (2020) released
the first image-captioning dataset with photos from people who are blind, and
a series of challenges for multimodal systems across different tasks (Gurari
et al., 2018). After this initiative, there have been many works trying to
improve current models for a specific use-case, to assist people with visual
disabilities (Dognin et al., 2022; Ahsan et al., 2021; Delloul and Larabi,
2023). There has also been research in human-computer interaction (HCI) and
accessibility on designing image descriptions for visually impaired
individuals, primarily focusing on screen readers and functional descriptions
of online, publicly available images (Morris et al., 2018; Bennett et al.,
2021; Schaadhardt et al., 2021). Despite these efforts, there still seems to
be a lack of focus on image captioning for the visually impaired (Ghandi et
al., 2023), especially in multi-cultural settings.

##  2 Methodology

We first created a survey seeking to understand the preferences of visually
impaired individuals for image captions, focusing on the inclusion of cultural
information and the desired level of detail (see Appendix A). We aggregate the
participants’ assessments of the helpfulness and importance of cultural
information in Figure 1.

We then focused on two lines of contribution: (1) We filtered the VizWiz
dataset for implicit cultural concepts. VizWiz is a widely used visual
question answering and image captioning dataset representing a real-world use
case, where examples consist of images and questions submitted by people who
are blind, together with crowdsourced answers and image captions (Gurari et
al., 2020). The selection of this dataset serves two main purposes. Firstly,
it is a challenging dataset specifically tailored to real-world challenges
faced by people seeking to access visual information. Secondly, VizWiz might
contain implicit cultural references that are currently not captured due to
the lack of culture-specific captions. (2) We evaluated the image captioning
performance of state-of-the-art close-sourced and open-sourced models in a
culturally diverse setting using our filtered VizWiz dataset. We performed
both an automatic scoring of model-generated captions against two sets of
annotations using the COCO evaluation package111https://github.com/tylin/coco-
caption and a human evaluation.

###  2.1 Data Filtering

To filter the data we hired a total of 165 annotators through the Prolific
platform.222https://www.prolific.com/ We first asked participants to specify
their country of origin, location, and their cultural background. Then, we
asked them to retrieve images from the VizWiz dataset
visualizer333https://vizwiz.cs.colorado.edu/VizWiz_visualization/view_dataset.php
related to their cultural background, provide the image name, the reason they
think the image is culture-related, and their preferred caption from the
dataset (VizWiz provides five different image captions per image). We also
gave them the option to suggest a better caption that includes cultural
aspects. After collecting all the culture-specific candidate images, we
proceeded to a second step of verification. In this step, we retained only
those images that had received consensus agreement from at least two
individuals. We collected a total of $324$ images and $648$ captions spanning
60 different identified cultures. It should also be noted that more than 96%
of the annotators suggested a cultural revision of the original captions. We
refer to Appendix B for further information about the annotation guidelines
and data filtering approach and results.

Model |  | BLEU-4 |  |  | METEOR |  |  | CIDEr |  |  | SPICE
---|---|---|---|---|---|---|---|---|---|---|---
Prompt |  | Default | Cultural |  |  | Default | Cultural |  |  | Default | Cultural |  |  | Default | Cultural
Annotation |  | Original | Cultural | Original | Cultural |  |  | Original | Cultural | Original | Cultural |  |  | Original | Cultural | Original | Cultural |  |  | Original | Cultural | Original | Cultural
BLIP-2 |  | 8.0$\pm$ 0.4 | $4.8$ |  $7.0$ $\pm$ | 0.4 | $4.6$ |  |  | 12.6$\pm$ 0.2 | $10.2$ | 12.3$\pm$ 0.3 | $10.3$ |  |  | 51.3$\pm$ 3.2 | $39.9$ | 44.0$\pm$ 3.0 | $36.7$ |  |  | 13.8$\pm$ 0.4 | $12.5$ | 12.8$\pm$ 0.5 | $11.5$
InstructBLIP |  | 14.0$\pm$ 0.5 | $8.7$ |  14.1 $\pm$ | 0.4 | $9.0$ |  |  | 17.3$\pm$ 0.3 | $13.2$ | 17.7$\pm$ 0.3 | $13.3$ |  |  | 77.1$\pm$ 3.4 | $60.0$ | 78.8$\pm$ 3.2 | $60.2$ |  |  | 18.5$\pm$ 0.4 | $15.6$ | 18.2$\pm$ 0.5 | $14.9$
Idefics2 |  | 12.0$\pm$ 0.5 | $10.1$ |  $9.8$ $\pm$ | 0.5 | $10.7$ |  |  | 18.1$\pm$ 0.3 | $15.1$ | 18.9$\pm$ 0.3 | $17.1$ |  |  | 80.2$\pm$ 1.9 | $78.4$ | 74.1$\pm$ 2.2 | $78.2$ |  |  | 18.0$\pm$ 0.5 | $16.7$ | 18.8$\pm$ 0.2 | $17.8$
LLaVA-1.6 |  | 10.0$\pm$ 0.5 | 11.4 |  $6.7$ $\pm$ | 0.3 | $7.7$ |  |  | 18.9$\pm$ 0.4 | $17.3$ | 18.4$\pm$ 0.3 | $17.0$ |  |  | 60.2$\pm$ 2.3 | 75.2 | 40.3$\pm$ 1.7 | $56.3$ |  |  | 16.3$\pm$ 0.6 | 16.5 | 15.8$\pm$ 0.5 | $15.4$
Gemini-1.5-Pro |  | 10.8$\pm$ 0.3 | 14.1 |  $5.8$ $\pm$ | 0.1 | $8.7$ |  |  | 20.8$\pm$ 0.4 | 21.3 | 18.2$\pm$ 0.1 | $21.0$ |  |  | 71.5$\pm$ 2.1 | 88.8 | 14.8$\pm$ 0.5 | $34.1$ |  |  | 19.6$\pm$ 0.4 | 21.6 | 14.9$\pm$ 0.3 | $17.7$
GPT-4o |  | 11.9$\pm$ 0.6 | 16.4 |  $8.1$ $\pm$ | 0.3 | $12.2$ |  |  | 22.4$\pm$ 0.4 | 23.4 | 19.9$\pm$ 0.3 | $22.6$ |  |  | 66.8$\pm$ 2.8 | 99.8 | 40.4$\pm$ 1.0 | $72.8$ |  |  | 19.1$\pm$ 0.4 | 21.8 | 16.6$\pm$ 0.3 | $20.1$

Table 1: Performance of various VLMs on our filtered VizWiz dataset across
captioning prompts (default & culture-specific) and annotations (original &
culture-specific). We use 2 reference annotations per image. Since the
original VizWiz has 5 annotations per image, we report the mean and standard
deviation over all 10 combinations with two references. We underline the best
result for each model and display the top result for each metric in bold.

###  2.2 Models and evaluation

We conducted experiments on the image captioning task in the zero-shot
setting, in which a pretrained model is queried to produce a textual
description for an image without finetuning on the same dataset. We relied on
four commonly used open-access models:444We used implementations and model
weights from HuggingFace (Wolf et al., 2020). BLIP-2 6.7B (Li et al., 2023a)
with OPT as LLM backbone (Zhang et al., 2022), InstructBLIP 7B (Dai et al.,
2023) with Vicuna backbone (Chiang et al., 2023), Idefics2 8B (Laurençon et
al., 2024), and LLaVa-1.6 7B (Liu et al., 2023a) with Mistral backbone (Jiang
et al., 2023). We also used two state-of-the-art closed-access models: GPT-4o
(OpenAI, 2024) and Gemini Pro 1.5 (Gemini Team, 2024). For all of these
models, we experimented with two different prompt types including a culture-
specific prompt following Shi et al. (2024) and a default captioning prompt
taken from Dai et al. (2023). The exact prompts can be found in App D. We
evaluated the model-generated captions in two ways: (1) via the COCO
evaluation suite and (2) through human evaluation. The COCO evaluation suite
was first introduced by (Chen et al., 2015) as a framework to assess image
captions using numerous automatic metrics, including BLEU (Papineni et al.,
2002), CIDEr (Vedantam et al., 2015), METEOR (Denkowski and Lavie, 2014), and
SPICE (Anderson et al., 2016). For consistency with our culture-specific re-
annotations (two captions per image), we also used two reference captions per
image to score models on the original annotations. Since each image has five
original captions, we report aggregate results over all ten two-caption
combinations. Our human evaluation had two stages. In the first stage, we
asked 60 participants to determine if a caption is accurate (on a binary
scale) given the corresponding image. In the second stage, we asked the same
participants to rank all captions (human-generated, and model-generated)
according to their preference. We did not make the annotators aware that one
caption was model-generated to minimize bias. We provide further details on
the human evaluation in Appendix F.

##  3 Results

#### Automatic evaluation

We present the results of our automatic evaluation of model-generated captions
in Table 1. Note that due to using two reference captions per image, results
for the original annotations are slightly different than when using all five
at once; we report the latter in Appendix E for completeness.

Figure 2: Examples of various images from the filtered VizWiz dataset with the
original ( *  X ) and culture-specific ( *  X ) annotations, and generated
captions from Gemini-1.5-Pro, GPT-4o, InstructBLIP, and LLaVA-1.6 with default
( *  X ) and culture-specific ( *  X ) prompting. Figure 3: Results of the
human evaluation for 100 images and their captions selected at random from the
filtered VizWiz dataset. The left plot shows the preference score
(participants were asked to rank the captions; lower is better). The right
plot shows the accuracy evaluation (participants were asked to assess whether
a caption is accurate; higher is better). _‘_D’_ and _‘_C’_ denote default and
culture-specific prompting, respectively.

As expected, the closed-access models (Gemini and GPT-4o) score best overall.
Slightly lower performance is achieved by the instruction-tuned open-access
VLMs (LLaVa, Idefics2, and InstructBLIP). BLIP-2, which has not been
instruction-tuned, is lagging behind across all metrics. Since VizWiz is
naturally noisy due to the high ratio of low-quality, blurry images, the
increased scale and overall multimodal reasoning capabilities of the closed-
source models appear to give a significant advantage.

Strikingly, Gemini and GPT-4o achieve much better performance on our newly
annotated captions that include cultural information than on the original
captions (e.g., $11.9$ vs. $16.4$ BLEU-4 and $66.8$ vs. $99.8$ CIDEr for
GPT-4o with the default prompt), while we observe the opposite for the open-
access models (e.g, $14.0$ vs. $8.7$ BLEU-4 and $77.1$ vs. $60.0$ CIDEr for
InstructBLIP with the default prompt). One possible explanation is that the
closed-source models have been tuned to generate more descriptive captions
that are aligned better with human preferences and our cultural caption
annotations, whereas the open-access models have been tuned to generate
slightly more concise captions that align well with benchmark datasets like
COCO Captions. Our new cultural annotations are also guaranteed to not have
leaked into the VLMs’ training data, thus favoring more objectively capable
models such as GPT-4o.

Next, while individual models (Idefics2 and InstructBLIP in particular) seem
amenable to cultural prompting, leading to improved performance even on the
original image captions, the cultural prompting strategy is overall largely
ineffective at improving performance on the cultural captions. This result may
be due to the models’ tendency for sycophantic behavior and them being primed
to point out cultural information over other relevant content in the image
(Sharma et al., 2023). Alternatively, cultural prompting might elicit more
verbose captions that are disfavored by the automatic evaluation metrics, in
which case the automatic evaluation results paint an incomplete and
potentially misleading picture.

#### Human evaluation

The results of the human evaluation are shown in Figure 3. In line with the
automatic metrics, our human annotators tend to prefer the captions produced
by closed-access models, GPT-4o and Gemini-Pro, with the BLIP-family models
having the lowest ranking. The former are rated as accurate in more than 90%
of the cases, while the latter are deemed inaccurate in more than half of the
cases. Despite the strong performance of the closed-access models, our
preference comparison also shows that the culture-specific human-annotated
captions are still preferred over all of the models, suggesting there is ample
room for improvement.

In spite of the often stark differences in automatic evaluation scores between
cultural and default prompting (with a preference for the latter), human
participants prefer the model generations obtained via cultural prompting in
4/6 cases (for both the ranking and the accuracy assessment), supporting our
hypothesis that cultural prompting simply elicits an answer format that is
disfavored by automatic metrics.

Overall, our results are promising in regard to the reliability of VLMs at
zero-shot generating captions that are accurate and useful to users who are
blind in culturally diverse scenarios.

##  4 Further Analysis

To further analyze our results and assess the model-generated captions in a
more fine-grained manner, we manually inspected all generated captions for our
$324$ images filtered VizWiz dataset and provided some examples in Figure 2.

We find that InstructBLIP and BLIP-2 captions tend to be very short, lack a
lot of information, and are often irrelevant hallucinations. This is, to an
extent, expected as we perform zero-shot captioning, so the models are not
necessarily accustomed to the desired captioning style. In this case, few-shot
prompting or finetuning the models would likely improve model performance
(Brown et al., 2020; Mañas et al., 2023; Ramos et al., 2023). The closed-
access models, in contrast, largely provide further or more useful and
culture-specific details about the image than given by the human captioners.
They also seem to provide more accurate captions compared to the open-sourced
models. These points may explain why GPT-4o and Gemini-1.5-Pro and were
overall preferred in our human evaluation.

Overall, we observed that the closed-access models can transcribe various
language scripts from books, food or beverage packages, giving them an
advantage over the smaller models. In most cases, in both culture-specific and
default prompts, the models can identify culture-specific beverages like
Japanese matcha tea, Chinese jelly grass or lychee juice, and food such as the
Indian lijjat papad, Japanese mochi, Tom Kha Gai Thai soup, Korean kimchi,
etc. There are also cases where they identify religious or folk items like the
Wayang Golek puppets, a jar with traditional Norwegian costumes, or a delft
plaque with traditional Dutch costumes.

There is, however, a tendency to generate longer text in the culture-specific
prompts by adding generic phrases such as ‘hinting at the drink’s cultural
origin’, ‘suggesting a celebration of their shared heritage’, ‘highlighting
its appeal across age groups in Indian culture’, etc. The most challenging
cases for the closed-source models seem to be foreign currencies (especially
the Arabic ones), historic figures, and paintings. For example, models seem to
confuse Bahraini, Jordan, and Egyptian banknotes, and they do not recognize
the Chinese historical figure of Sun Yat-sen, or paintings of Joan Miró or
Frederick Morgan. We provide further examples in Appendix G.

##  5 Discussion

Given the current integration of VLMs as virtual assistants for people who
seek sighted support, their performance on culture-specific image captioning
seems promising. Examples from our error analysis and case studies highlight
some remaining challenges. Measured by automatic evaluation metrics, the
performance of the models is overall relatively low compared to results in
existing studies evaluating (finetuned) VLMs for image captioning on the full
VizWiz and other datasets (Gurari et al., 2020; Chen et al., 2023; Wang et
al., 2022). On the other hand, our human evaluation and error analysis show
that the generated captions by Gemini-1.5-Pro and GPT-4o are accurate and
preferred in many cases. There also seems to be an extended hallucination
problem, which remains an existing major challenge not only for VLMs (Li et
al., 2023b) but across various language model applications (Bang et al., 2023;
Ji et al., 2023).

##  6 Conclusion

We evaluated the cultural performance of various models on image captioning
using a multicultural dataset tailored to a real-world use case. Although the
performance of state-of-the-art closed-source models is promising, there is
plenty room for improvement. Examples from our error analysis provide insights
into the models’ performance, helping us identify some of their weak spots. In
our use case, we find that automatic evaluation metrics might not be fully
representative of model performance, and therefore encourage researchers to
reconsider a more comprehensive assessment framework. For future work, we aim
to extend our small filtered cultural dataset by including question-answering
tasks with POV cultural questions.

## Limitations

Our work focuses primarily on data curation and empirical analysis of large
multimodal language models. Our survey, while aimed at determining caption
preferences, may not capture the full range of needs and preferences of all
people with visual impairment. Further, through our analysis, we gained
insights into some weak spots with respect to what cultures and cultural
concepts are well recognized by the models. However, since we use a finite
amount of data, there might be a data bias in identifying particular cultures
or cultural concepts as problematic. Lastly, cultural complexities and
variations make it difficult to develop a standardized approach to cultural
inclusion in AI. We do, however, hope that our culture-centric approach in the
data filtering and annotation process can serve as an initial step towards
evaluating and understanding the cultural awareness and abilities of vision-
language models for real-world uses.

## Ethics Statement

The motivation behind this study is that large vision-language models have
rapidly become mainstream and are used even by those who seek sighted support
and cannot easily assess model hallucinations or inaccuracies. The primary
purpose of our experiments is to assess the performance of vision-language
models in the task of image captioning using a multicultural dataset of images
taken from people who are blind. However, it is crucial to recognize that
results from our current filtered dataset may not be representative of model
performance across cultures. Furthermore, our refined dataset might retain
biases present in the original source dataset.

We find it improbable that our experiments and the filtered dataset will
meaningfully benefit those intending to create deceptive models for malicious
purposes. Additionally, the VizWiz dataset may lack coverage of highly
specific subjects, offering only a general overview of factual topics. People
who intend to use our resources, however, should state their purpose of usage
and be accountable for their own work.

## Acknowledgments

## References

  * Ahsan et al. (2021) Hiba Ahsan, Daivat Bhatt, Kaivan Shah, and Nikita Bhalla. 2021.  Multi-modal image captioning for the visually impaired.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop_ , pages 53–60, Online. Association for Computational Linguistics.
  * Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.  SPICE: semantic propositional image caption evaluation.  In _Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V_.
  * Arora et al. (2023) Arnav Arora, Lucie-aimée Kaffee, and Isabelle Augenstein. 2023.  Probing pre-trained language models for cross-cultural differences in values.  In _Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)_ , pages 114–130, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023.  A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 675–718, Nusa Dua, Bali. Association for Computational Linguistics.
  * Bennett et al. (2021) Cynthia L. Bennett, Cole Gleason, Morgan Klaus Scheuerman, Jeffrey P. Bigham, Anhong Guo, and Alexandra To. 2021.  “it’s complicated”: Negotiating accessibility and (mis)representation in image descriptions of race, gender, and disability.  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , CHI ’21, New York, NY, USA. Association for Computing Machinery.
  * Bhatia et al. (2024) Mehar Bhatia, Sahithya Ravi, Aditya Chinchure, Eunjeong Hwang, and Vered Shwartz. 2024.  From local concepts to universals: Evaluating the multicultural understanding of vision-language models.  _arXiv preprint_.
  * Brady et al. (2013) Erin Brady, Meredith Ringel Morris, Yu Zhong, Samuel White, and Jeffrey P. Bigham. 2013.  Visual challenges in the everyday lives of blind people.  In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , CHI ’13, page 2117–2126, New York, NY, USA. Association for Computing Machinery.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Burda-Lassen et al. (2024) Olena Burda-Lassen, Aman Chadha, Shashank Goswami, and Vinija Jain. 2024.  How culturally aware are vision-language models? _arXiv preprint_.
  * Cao et al. (2024) Yong Cao, Wenyan Li, Jiaang Li, Yifei Yuan, Antonia Karamolegkou, and Daniel Hershcovich. 2024.  Exploring visual culture awareness in GPT-4V: A comprehensive probing.  _arXiv preprint_.
  * Cao et al. (2023) Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023.  Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study.  In _Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)_ , pages 53–67, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Chen et al. (2023) Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme Ruiz, Andreas Peter Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. 2023.  PaLI: A jointly-scaled multilingual language-image model.  In _The Eleventh International Conference on Learning Representations_.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. 2015.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint_.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality.  blog post.
  * Chiu et al. (2020) Tai-Yin Chiu, Yinan Zhao, and Danna Gurari. 2020.  Assessing image quality issues for real-world problems.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3643–3653.
  * Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023.  InstructBLIP: Towards general-purpose vision-language models with instruction tuning.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Delloul and Larabi (2023) Khadidja Delloul and Slimane Larabi. 2023.  Towards real time egocentric segment captioning for the blind and visually impaired in rgb-d theatre images.  _arXiv preprint_.
  * Denkowski and Lavie (2014) Michael Denkowski and Alon Lavie. 2014.  Meteor universal: Language specific translation evaluation for any target language.  In _Proceedings of the Ninth Workshop on Statistical Machine Translation_ , pages 376–380, Baltimore, Maryland, USA. Association for Computational Linguistics.
  * Dognin et al. (2022) Pierre Dognin, Igor Melnyk, Youssef Mroueh, Inkit Padhi, Mattia Rigotti, Jarret Ross, Yair Schiff, Richard A. Young, and Brian Belgodere. 2022.  Image captioning as an assistive technology: Lessons learned from vizwiz 2020 challenge.  _J. Artif. Int. Res._ , 73.
  * Gemini Team (2024) Gemini Team. 2024.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  _arXiv preprint_.
  * Ghandi et al. (2023) Taraneh Ghandi, Hamidreza Pourreza, and Hamidreza Mahyar. 2023.  Deep learning approaches on image captioning: A review.  _ACM Comput. Surv._ , 56(3).
  * Gonzalez et al. (2024) Ricardo Gonzalez, Jazmin Collins, Shiri Azenkot, and Cynthia Bennett. 2024.  Investigating use cases of ai-powered scene description applications for blind and low vision people.  _arXiv preprint_.
  * Gurari et al. (2018) Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. 2018.  Vizwiz grand challenge: Answering visual questions from blind people.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3608–3617.
  * Gurari et al. (2020) Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhattacharya. 2020.  Captioning images taken by people who are blind.  In _Computer Vision – ECCV 2020_ , pages 417–434, Cham. Springer International Publishing.
  * Hershcovich et al. (2022) Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022.  Challenges and strategies in cross-cultural NLP.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6997–7013, Dublin, Ireland. Association for Computational Linguistics.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Comput. Surv._ , 55(12):248:1–248:38.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.  _arXiv preprint_.
  * Jung et al. (2022) Ju Yeon Jung, Tom Steinberger, Junbeom Kim, and Mark S. Ackerman. 2022.  “so what? what’s that to do with me?” expectations of people with visual impairments for image descriptions in their personal photo activities.  In _Proceedings of the 2022 ACM Designing Interactive Systems Conference_ , DIS ’22, page 1893–1906, New York, NY, USA. Association for Computing Machinery.
  * Laurençon et al. (2024) Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024.  What matters when building vision-language models? _arXiv preprint_.
  * Li et al. (2023a) Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023a.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint_.
  * Li et al. (2023b) Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023b.  Evaluating object hallucination in large vision-language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 292–305, Singapore. Association for Computational Linguistics.
  * Liu et al. (2021) Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021.  Visually grounded reasoning across languages and cultures.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10467–10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Liu et al. (2023a) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a.  Visual instruction tuning.  In _Advances in Neural Information Processing Systems_ , volume 36.
  * Liu et al. (2023b) Zhixuan Liu, Youeun Shin, Beverley-Claire Okogwu, Youngsik Yun, Lia Coleman, Peter Schaldenbrand, Jihie Kim, and Jean Oh. 2023b.  Towards equitable representation in text-to-image synthesis models with the cross-cultural understanding benchmark (ccub) dataset.  _arXiv preprint_.
  * Mañas et al. (2023) Oscar Mañas, Pau Rodriguez Lopez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, and Aishwarya Agrawal. 2023.  MAPL: Parameter-efficient adaptation of unimodal pre-trained models for vision-language few-shot prompting.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 2523–2548, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Morris et al. (2018) Meredith Ringel Morris, Jazette Johnson, Cynthia L. Bennett, and Edward Cutrell. 2018.  Rich representations of visual content for screen reader users.  _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_.
  * Mukherjee et al. (2024) Anjishnu Mukherjee, Ziwei Zhu, and Antonios Anastasopoulos. 2024.  Crossroads of continents: Automated artifact extraction for cultural adaptation with large multimodal models.  _arXiv preprint_.
  * OpenAI (2024) OpenAI. 2024.  Hello gpt-4o.  blog post.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Ramos et al. (2023) Rita Ramos, Bruno Martins, Desmond Elliott, and Yova Kementchedjhieva. 2023.  Smallcap: Lightweight image captioning prompted with retrieval augmentation.  In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2840–2849.
  * Romero et al. (2024) David Romero et al. 2024.  Cvqa: Culturally-diverse multilingual visual question answering benchmark.  _arXiv preprint_.
  * Schaadhardt et al. (2021) Anastasia Schaadhardt, Alexis Hiniker, and Jacob O. Wobbrock. 2021.  Understanding blind screen-reader users’ experiences of digital artboards.  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , CHI ’21, New York, NY, USA. Association for Computing Machinery.
  * Sharma et al. (2023) Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2023.  Towards understanding sycophancy in language models.  _arXiv preprint_.
  * Shi et al. (2024) Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Chunhua yu, Raya Horesh, Rogério Abreu de Paula, and Diyi Yang. 2024.  Culturebank: An online community-driven knowledge base towards culturally aware language technologies.  _arXiv preprint_.
  * Thapliyal et al. (2022) Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. 2022.  Crossmodal-3600: A massively multilingual multimodal evaluation dataset.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 715–729, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4566–4575.
  * Ventura et al. (2023) Mor Ventura, Eyal Ben-David, Anna Korhonen, and Roi Reichart. 2023.  Navigating cultural chasms: Exploring and unlocking the cultural pov of text-to-image models.  _arXiv preprint_.
  * Wang et al. (2022) Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022.  GIT: A generative image-to-text transformer for vision and language.  _Transactions on Machine Learning Research_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Ye et al. (2023) Andre Ye, Sebastin Santy, Jena D. Hwang, Amy X. Zhang, and Ranjay Krishna. 2023.  Cultural and linguistic diversity improves visual representations.  _arXiv preprint_.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  OPT: Open pre-trained transformer language models.  _arXiv preprint_.

##  Appendix A Survey on Caption Preferences

We created a survey aiming to understand the preferences of individuals who
seek sighted support regarding image captioning. Our interest was particularly
focused on whether they prefer image captions to include cultural information,
and how detailed they prefer the descriptions to be. We published our survey
through the Prolific platform, by choosing 60 participants with an equal
gender sample of and representative across countries compensated with $18 per
hour. We also added a screener and selected participants without
corrected/normal vision. Overall, the participants were positive regarding the
helpfulness and importance of cultural information in the captions with
average ratings of 4.1 and 3.9, respectively.555The scale is from 1. Not
important/helpful at all to 5. Very important/helpful) Participants also
tended to prefer short captions compared to longer ones.

##  Appendix B VizWiz Data Filtering – Human Annotation

As mentioned in the experimental set-up section, to filter the data we created
a survey through the Prolific annotation platform. All annotators were
compensated with 18$ per hour. We ran this survey 4 times asking for 40
participants each time.

We asked people to identify images from the VizWiz dataset based on their
cultural background, provide an original and a corrected caption, and specify
the reason they selected the image as culture-specific. We grouped the reasons
that the annotators provided for selecting culture-specific images in Figure
4.

Figure 4: Distribution of the factors/indicators that lead the annotators to
select a specific image as culture-related and specify the corresponding
culture.

The cultural concepts identified by our annotators can be found in Figure 5.

Figure 5: Distribution of the cultural concepts identified in the VizWiz
dataset by the annotators.

The full annotation guidelines were the following:

Creating datasets that reflect a variety of cultures is a challenging task.
This is why we will try to filter an existing dataset. Your task is to find
culture-related images from a dataset called VizWiz. You need to: \- Visit the
dataset website[link]. \- Browse the dataset or use the search bars on the
left side of the page and search key-terms related to your culture ’Within
visual question’, ’Within visual answer’ or ’Within captions’. \- Try to find
an image that is related to your culture/cultural background (i.e. food brand,
currency, books, culture-specific locations etc.) \- Provide your answers to
the 5 following questions. 1. Copy and paste the image name
(VizWiz_train_**number**.jpg). 2. Based on your cultural background, specify
what culture you think is the image related to. 3. Select a caption for the
image from the suggested Image Captions. 4. Do you have a better suggestion
for the image caption? To guide your caption generation, imagine that you are
describing the image to a visually impaired friend. The caption should explain
the whole image, including all the main objects, activities, and their
relationships, and reflect the culture information of the image. 5. Provide a
reason as to why the image is culture-specific.

After this, we collected information about the annotators’ cultural
backgrounds. We asked for both home-country of origin and current country
location information since sometimes both can affect our cultural beliefs and
practices. The distribution of the annotators counties of origin and location
are presented in Figure 6(b).

The last step is to answer some final questions about your cultural
background, and age. We do not collect any other personal information. Your
answers will only be used for statistical research purposes. • What is your
country of origin that you consider your ’home’, influencing your cultural
beliefs and other aspects of your identity? • Is there a country in which you
are currently located for a long period of time? • How old are you? Fill in
years in numbers.

After collecting all the responses, we kept only the images where at least two
annotators agreed to select the image as culture-specific. After this extra
validation, we resulted in a total of $324$ images spanning 60 different
identified cultures. We compared the similarity between the suggested captions
by the annotators and the original VizWiz captions and the results can be
found in Table 2 indicating a high similarity between the new culture-specific
suggested captions.

Captions | BLEU-4 | ROUGE-L | F1
---|---|---|---
Culture-specific | 37.10 | 61.90 | 93.0

Table 2: Results from comparing the culture-specific captions of the two
annotators against the five original VizWiz captions.

(a) Distribution of the current country location of the annotators.

(b) Distribution of the country of origin of the annotators.

Figure 6: Plots as subfigures.

##  Appendix C Model Overview

We list models with their API identifiers in Table 3 below.

Name | Identifier | Reference
---|---|---
BLIP-2 | Salesforce/blip2-opt-6.7b | Li et al. (2023a)
InstructBLIP | Salesforce/instructblip-vicuna-7b | Dai et al. (2023)
Idefics2 | HuggingFaceM4/idefics2-8b | Laurençon et al. (2024)
LLaVA-1.6 | llava-hf/llava-v1.6-mistral-7b-hf | Liu et al. (2023a)
Gemini-1.5-Pro | gemini-1.5-pro-preview-0514 | Gemini Team (2024)
GPT-4o | gpt-4o-2024-05-13 | OpenAI (2024)

Table 3: Overview of models used in this study

##  Appendix D Model Prompting

We provide the templates we used to prompt our models. The default templates
have been sourced from Dai et al. (2023) and Shi et al. (2024).

Default prompting
---
<Image> A short image description:
<Image> Write a caption that describes the photo. Format your response in JSON
as follows: { "caption": "Caption for the image" }
<Image> A photo of
<Image> Can you briefly describe the content of the image?
<Image> Write a caption that describes the photo.
Culture-specific prompting
<Image> A short, culture-aware image description:
<Image> Cultural information encompasses content that showcases the
distinctive characteristics, artifacts, or manifestations of a specific group,
community, or region. This includes, but is not limited to, practices,
behaviors, norms, values, beliefs, habits, customs, architectural styles,
environmental engagements, and any other elements that are emblematic of a
particular cultural setting. It does not include generic information or
widespread practices that are not distinctly tied to a specific cultural
identity. For this task, consider information as "cultural" if: 1\. It is
associated with or characteristic of a specific identified group (e.g.,
Americans, Italians, midwestern Americans, etc.). 2\. It reveals a unique
aspect of that group’s way of life, including social conventions, physical
creations, or interactions with their surroundings that are not typically seen
in other cultures. 3\. It provides insight into the cultural uniqueness,
whether through social practices, material culture, or other culturally
significant elements. Please exclude generic or ubiquitous statements or
observations that do not clearly relate to the unique cultural context of a
specific group. Given this image, do two things: 1\. Determine whether the
provided example contains cultural information. 2\. Write a caption that
describes the photo and includes the cultural information extracted. Format
your response in JSON as follows: { "caption": "Caption for the image",
"is_cultural": true/false, "justification": "Why or why not the image contains
cultural information" }
<Image> Write a caption that describes the photo and includes any cultural
information present.
Table 4: Image captioning templates used to prompt our models.

##  Appendix E Vizwiz Results – 5 Original References

We report model performance on our filtered VizWiz dataset when using all five
original captions per image (rather than combinations of two references at a
time) in Figure 5.

Model |  BLEU-4 |  METEOR |  CIDEr | SPICE
---|---|---|---|---
Prompt | Default | Cultural | Default | Cultural | Default | Cultural | Default | Cultural
Annotation | Original-Full | Original-Full | Original-Full | Original-Full
BLIP-2 | 14.9 |  $12.1$ | 16.1 |  $15.5$ | 51.7 |  $44.3$ | 10.6 | $9.9$
InstructBLIP | 25.3 |  $24.5$ | $22.0$ |  22.1 | $77.4$ |  78.9 | 15.0 | 15.0
Idefics2 | 20.8 |  $16.7$ | $22.2$ |  23.3 | 82.0 |  $76.1$ | $15.1$ | 16.6
LLaVA-1.6 | 17.3 |  $11.8$ | 23.3 |  $22.1$ | 60.9 |  $40.5$ | 15.3 | 15.3
Gemini-1.5-Pro | 18.6 |  $9.9$ | 25.5 |  $21.9$ | 73.0 |  $15.0$ | 18.0 | $15.3$
GPT-4o | 20.5 |  $13.8$ | 27.4 |  $24.4$ | 67.7 |  $41.0$ | 18.4 | $16.6$

Table 5: Performance of various VLMs on our filtered VizWiz dataset across
captioning prompts (default & culture-specific) using the five original
reference captions per image. We underline the best result for each model and
display the top result for each metric in bold.

##  Appendix F Human Evaluation

To conduct the human evaluation of the model generated responses we created a
survey and hired 54 annotators through the Prolific platform compensated with
18$ per hour. We added a screening in the platform for a representative sample
of countries and an even distribution of male and female participants. Each
annotator evaluated 12 images and their captions and for each image, we
assigned two annotators and averaged their scores. We provided the following
instructions to the annotators for evaluating the captions:

This study involves evaluating captions. To guide your ratings, imagine that
you are describing the image to a visually impaired friend. Then consider: How
well does the caption describe the image to this friend? Does it take into
account cultural considerations? You will be given two sets of captions
describing an image. 1. Specify which caption you prefer for the given image
(1, 2 or both). 2. Determine if each caption is accurate and relevant to the
given image. As a general guidance you should consider a caption as bad when
it has one or more of the following issues: a) Caption misses the main topic
of the image. b) Caption has major grammatical errors (such as being
incomplete, words in the wrong order, etc). Please ignore the capitalization
of words and punctuation. c) Caption includes hallucinations and mentions
objects, activities, or relationships that are definitely not in the image. d)
Caption is not as informative. e) Caption does not reflect the cultural
information depicted in the image.

##  Appendix G Error Analysis II

We provide further examples from currency-related images in Figure 7. We can
see that for countries such as US, or Australia, the original VizWiz captions
provide culture-specific information, but this is not the case for Japanese or
Arabic currencies. Moreover, the models seem robust in western and Asian
currencies, but not with all the Arabic ones. The example provided in Figure 7
shows how the models confuse a Jordan currency with Egyptian or Saudi Arabian
currencies and how the smaller open-source models are more prone to
hallucinations.

Figure 7: Examples from images related to currency comparing original ( *  X )
with culture-specific ( *  X ) annotations and generated captions from Gemini-
Pro and GPT-4o with default ( *  X ) and culture-specific ( *  X ) prompting.

##  Appendix H Case Study

We illustrate the value of cultural and inclusive VL models via a case study
on evaluating GPT-4V as a visual assistant integrated into the ‘Be My Eyes’
platform. In this case study, we took a random sample of 20 images from the
MaRVL dataset (Liu et al., 2021). Here we provide a selection of images we
tried in our case study. Each figure includes the target culture behind each
image and the GPT-4 Vision output after loading the image in the Be My Eyes
application.

Figure 8: A picture extracted from MaRVL depicting a statue of Agamas and the
GPT-4V image description provided in BeMyEyes. Figure 9: A picture extracted
from MaRVL depicting Buddhist statues and the GPT-4V image description
provided in BeMyEyes. Figure 10: A picture extracted from MaRVL depicting
sambar, a traditional Tamil dish, and the GPT-4V image description provided in
BeMyEyes. Figure 11: A picture extracted from MaRVL depicting a döner, a
traditional Turkish dish, and the GPT-4V image description provided in
BeMyEyes.