# A Tilted Dark Halo Origin of the Galactic Disk Warp and Flare

Jiwon Jesse Han  Charlie Conroy  and Lars Hernquist
Center for Astrophysics $|$ Harvard & Smithsonian, Cambridge, MA, USA

###### Abstract

The outer disk of the Milky Way Galaxy is warped and flared. Several
mechanisms have been proposed to explain these phenomena, but none have
quantitatively reproduced both features. Recent work has demonstrated that the
Galactic stellar halo is tilted with respect to the disk plane, suggesting
that at least some component of the dark matter halo may also be tilted. Here
we show that a dark halo tilted in the same direction as the stellar halo can
induce a warp and flare in the Galactic disk at the same amplitude and
orientation as the data. In our model the warp is visible in both the gas and
stars of all ages, which is consistent with the breadth of observational
tracers of the warp. These results, in combination with data in the stellar
halo, provide compelling evidence that our Galaxy is embedded in a tilted dark
matter halo. This misalignment of the dark halo and the disk holds clue to the
formation history of the Galaxy, and represents the next step in the dynamical
modeling of the Galactic potential.

We construct a model of the gravitational potential of the Galaxy in which 30%
of the dark halo mass is comprised of a triaxial distribution that is tilted
$25^{\circ}$ with respect to the disk plane. The potential also includes bulge
and disk components, where the latter increases in time to mimic the growth of
the Galactic disk with time. We calculate the orbits of collisionless (stars)
and collisional (gas) particles over 5 Gyr in this potential. In Figure 1 we
show the present-day distribution of stars in the simulation. In the top panel
we plot the Galactocentric vertical height ($Z$) and signed cylindrical radius
($R$), clearly revealing an S-shaped warp of the disk. Negative $R$ indicates
the Galactic quadrants that are within $\pm 90^{\circ}$ of the positive peak
(the “Northern” warp) and positive $R$ indicates Galactic quadrants that are
within $\pm 90^{\circ}$ of the negative peak (the “Southern” warp). In the
bottom panels, we plot the vertical deviation ($Z-Z_{warp}$) of tracer
particles from the mean warp. The average vertical deviation grows markedly
towards outer radii, demonstrating the flare of the disk. Particles are color-
coded by age, which shows that stars of all ages exhibit a warp and flare. The
warp is most pronounced for the youngest stars, consistent with
observations[1, 2].

Figure 1: Present-day distribution of simulated stars in Galactocentric
cylindrical coordinates. Negative R indicates azimuthal angles that are within
$90^{\circ}$ of the Northern warp, and positive R indicates azimuthal angles
that are within $90^{\circ}$ of the Southern warp. The top panels show the
warp, and the bottom panels show the vertical deviation from the average warp.
The vertical deviation systematically increases toward the outer Galaxy,
demonstrating the flare of the disk. Points that are in the outer Galaxy are
plotted with larger markers. Particles colored by stellar age clearly
demonstrate a warp at all ages, and most strongly at the youngest population.

In Figure 2 we compare the warp and flare of the simulated stars (left panel)
and gas (right panel) to observations. For conciseness of comparison, we focus
on the HI[3, 4] gas and Cepheid[1] stars. We note that a similar warp has also
been observed for ionized gas[5], molecular gas[6], dust[7], stars[8, 9, 10,
2], and star clusters[11]. In the top panels we plot the observed warp in open
circles, and a maximum likelihood fit to the simulation in solid colored
lines, with $1\sigma$ uncertainty of the fit shaded. The onset radius $R_{w}$
of the fit is fixed to the observed $R_{w}$ values. The flare is measured by
fitting an exponential scale height to the vertical deviation from the warp at
each radius. In both stars and gas, the simulated warp and flare
quantitatively match the observations. Additionally, the tilted halo
simulation produces a population of stars on circular orbits that reach high-
latitude ($|Z|>2\text{ kpc}$) on either side of the warp extrema. This
population of stars is reminiscent of known stellar overdensities towards the
Galactic anticenter[12, 13, 10, 14].

The parameters chosen for the simulation were not tuned to match the Galactic
warp or flare. The two key parameters affected the predicted warp and flare
are: (1) the shape and extent of the tilted halo, and (2) the scale radius of
newly-formed stars. For the former, we adopt the scale radius, triaxiality,
and tilt angle of the halo directly from the shape of the accreted stellar
halo[15]. For the latter, we adopt the scale radius of the molecular disk of
the Galaxy[16]. Details of the simulation setup and variations to the adopted
parameters are given in the Supplementary Material.

The large-scale warp of the Galactic disk has been known for over half a
century[17, 18], and there is a commensurately rich history of warp and flare
models[19, 20, 21, 22]. For example, studies have investigated the warp as a
result of perturbed bending modes[23], misaligned angular momenta of the halo
and the disk[24, 25], repeated impact from the Sagittarius dwarf galaxy[26],
misaligned gas accretion[27], or quadrupolar torque from a tumbling triaxial
halo[28]. However, previous studies have been unable to quantitatively
reproduce the warp (and simultaneously the flare) of the Galactic disk. Among
the recently investigated models is the tidal influence of the Large
Magellanic Cloud (LMC)[29]. In Figure 3 we plot the disk warp at $R=16\text{
kpc}$ produced by the tilted halo model and an LMC model as a function of
Galactic longitude. The LMC was modeled as a live halo on its first infall
into a live Milky Way halo and disk[29]. The HI data[3] at this radius is
marked in open circles. Even for the highest LMC mass—which is 80% higher than
other models[30]—the warp amplitude is less than a third of the observed
amplitude. In the right panel, we sum the amplitudes of the warp from the LMC
and the tilted halo models. The combined model matches the observations better
than the tilted halo alone, although we caution that this simple summation
does not capture the possible coupling between the effect of the LMC and the
tilted halo.

Figure 2: Comparison of the simulation to the observed warp and flare. The top
panels show the warp, and bottom panels show the flare. The observed points
(open circle) are extracted directly from the cited papers[1, 3, 4]. (a) In
the top panel, the observed warp of Cepheids[1] is plotted in open circles,
and the maximum-likelihood fit of stellar warp in the simulation is plotted in
red. We fix the onset radius of the warp to the observed value. Shaded regions
indicate $1\sigma$ uncertainty of the fit. In the bottom panel, the observed
scale height of Cepheids are plotted in black dashed lines, and the scale
height of the simulated stars are plotted in red. (b) In the top panel, the
observed warp of HI[3, 4] is plotted in open circles, and the maximum-
likelihood fit of the gaseous warp in the simulation is plotted in blue. In
the bottom panel, the observed scale height of HI is plotted in black dashed
lines, and the scale height of the simulated gas are plotted in blue. Figure
3: The amplitude and orientation of the disk warp at $R=16\text{ kpc}$. In the
left panel we plot the simulated warp from the tilted dark halo in blue, the
LMC models[29] in red, and the HI warp in open circles[3]. Shaded regions
indicate $1\sigma$ uncertainty of the fit. The individual data points from the
LMC simulation are marked in faint dots, and a polynomial fit to the points
are drawn as solid lines. In the right panel, we show the sum of the warps
induced by a tilted halo and the LMC.

Tilted dark halos are common in galaxy simulations that include baryonic
physics[31, 32], and several independent lines of evidence have pointed toward
a tilted Galactic dark halo[33, 34, 35]. Furthermore, Han et al. (in prep)
show that such tilted dark halos are long-lived and can warp Galactic disks in
Illustris TNG50[36, 37]. In a realistic model we can expect the tilt angle to
change with time, for example due to interaction between the (growing) disk
and the halo. While the tilt of the halo at earlier epochs can leave
interesting observational signatures in old disk stars, the main point of this
paper is to demonstrate that the young disk (and gas) is responsive to the
tilt on $<1$ Gyr timescales, and on such timescales the tilt is constrained
observationally to be $\sim 25^{\circ}$. A plausible origin of a tilted dark
halo in the Galaxy is a major merger[38, 39] $8-10\text{ Gyr}$ ago[40, 41]
that deposited a significant fraction of dark matter on an eccentric, tilted
orbit[42, 35]. Dynamical models of the Galaxy often assume a spherical dark
halo at all scales, or a flattened halo that is aligned with the disk. A
tilted dark halo at $\sim 30\text{ kpc}$ would have novel applications for
Galactic dynamics. For example, a tilted and triaxial halo influences the
shape of the stellar halo[35], and can affect orbit reconstruction of stellar
streams[43]. Furthermore, the tilt and triaxiality of the inner halo ($\sim
0.1R_{\text{virial}}$) encodes information about the self-interacting
properties of dark matter[44] that is unique compared to larger-scale probes
such as galaxy clusters or large scale structure. Future work will investigate
the imprint of dark matter self-interaction through the tilt and triaxiality
of the dark halo at $30\text{ kpc}$. In addition, a global tilt in the dark
halo implies an anisotropic velocity distribution of the dark matter
particles. The resulting asymmetric velocity distribution should affect
ground-based dark matter detection experiments[45].

Finally, while we have focused on demonstrating that the warp and flare are
likely manifestations of a tilted dark halo, we can also reverse the argument:
precise measurements of the warp may further constrain the tilt of the dark
halo. We have intentionally avoided “fitting” a dark halo to match the warp,
but there is much to explore in allowing for more flexible halo models. For
example, the warp is sensitive to the tilt angle of the dark halo, and the
fraction of mass in the tilted component (see Methods). Jointly modeling the
numerous tracers of the disk warp at various ages and Galactic radii is the
next step in uncovering the distribution of dark matter in the Galaxy.

## Methods

### Simulation details

Here we describe the time-dependent Galactic potential in which we compute the
orbits of tracer particles. The potential is a summation of the following
components. The halo has a total mass of $8\times 10^{11}\text{M}_{\odot}$,
70% of which is in a spherical halo with NFW[46] radial profile with scale
radius $r_{s}=15\text{ kpc}$, and 30% of which is in a tilted, triaxial halo.
The triaxiality is 10:8.1:7.3[15] and the tilt angle is $25^{\circ}$[15]. The
radial density along the principal axes of the triaxial halo follows an NFW
profile with scale radius $r_{s}=30\text{ kpc}$[35]. The disk has two
components: a “thick” disk with fixed mass at $6\times
10^{9}\text{M}_{\odot}$, scale radius 2 kpc, and scale height 0.9 kpc[47], and
a “thin” disk that linearly increases in mass up to $3.5\times
10^{10}\text{M}_{\odot}$ at present day, with fixed scale radius 2.6 kpc and
scale height 0.3 kpc[47] assuming a Miyamoto-Nagai potential[48]. The initial
mass of the thin disk is $1.3\times 10^{10}\text{M}_{\odot}$. Lastly, we
include a spherical bulge with Hernquist radial profile[49] and fixed mass
$1.8\times 10^{10}\text{M}_{\odot}$ and scale radius of 1 kpc.

The tracer particles are initialized on circular orbits at radii sampled from
a truncated exponential distribution, discarding any stars sampled within 1
kpc or beyond eight times the scale radius from the Galactic center. The scale
radius increases linearly with time to 8 kpc at present day. The final scale
radius is chosen to be where the $\text{H}_{2}$ mass surface density[16] is
$1/e$ of the maximum value. At each time step of the simulation, we spawn new
tracer particles at a rate that is commensurate with the growth of the disk,
culminating in 50,000 star particles and 50,000 gas particles. We calculate
orbits using the gala python package[50] using a fixed 1 Myr timestep and
standard Leapfrog integrator.

While the star particles are collisionless, the gas particles are collisional
and follow an inelastic scattering prescription. If a gas particle comes
within $0.1\text{ kpc}$ of another gas particle and they have negative
relative speeds, they exchange relative velocities and lose $10\%$ of the
collective kinetic energy. This method of simulating gas particles is valid
when the velocity dispersion is low[51, 52], as it is the case for circular
orbits. We note that all of our disk particles remain on circular orbits
throughout all 5 Gyr, with the most eccentric orbits exhibiting less than 0.1%
of their angular momentum off of the Galactic $Z$ axis. Lastly, at each
timestep, we allow for a probabilistic change of radius of each particle
commensurate to what is measured for the Galaxy[53, 54].

### Warp Modeling

In Figure 2 we plot a fit to the warp in the simulation using an analytical
formula that is a power-law in radius and a sinusoid in azimuth:

$\displaystyle Z(R\geq R_{w})$
$\displaystyle=A\times(R-R_{w})^{b}\times\sin{(\phi-\phi_{w})}$ $\displaystyle
Z(R<R_{w})$ $\displaystyle=0$

Here, $R$ and $Z$ are Galactocentric cylindrical coordinates, $A$ is the
amplitude of the warp, $b$ the power-law index, $\phi_{w}$ the orientation of
the warp, and $R_{w}$ the onset radius of the warp. This function has also
been used to fit the Cepheids data[1]. The fit is performed using a maximum
likelihood method.

### Which Stars Comprise the Warp?

In Extended Data Figure 1 we color the star particles by their birth radius.
We find a clear correlation in the birth radius of the star and its final warp
amplitude. We can thus understand the prominence of the warp in young stars as
a result of the “inside-out” growth of the disk[55], in which the birth radius
of a particle correlates inversely with age. Since young stars can be born at
larger radii than old stars, they trace a cleaner and larger warp. Due to
radial migration[54], old stars that are born in the inner Galaxy can also
migrate outwards to eventually trace the warp. This mechanism can explain why
older stars appear to have smaller warp amplitudes in observations[2].

### Timescale of the Warp

In Extended Data Figure 2 we show the time evolution of the warp amplitude at
a fixed radius $R=16\text{ kpc}$. The error bars indicate $1\sigma$
statistical uncertainties in the warp fit. At $t=0$, the disk is initialized
to have no warp. Within the first few hundred Myr, the warp reaches its
maximum amplitude. This is consistent with the rotation period of the disk at
16 kpc, which is approximately $400\text{ Myr}$. Thus, this Figure shows that
the disk responds to the tilted dark halo quickly, within one rotation period
of the disk. The warp amplitude experiences a transient oscillatory phase
until $1500\text{ Myr}$, then converges to a steady state. This transient
phase is likely a numerical effect, since there are not many stars out at
$R=16\text{ kpc}$ at these times (the tracer particle scale length is 3 kpc at
$t=0$ and 4.5 kpc at $t=1500\text{ Myr}$) and the warp amplitude is determined
by a small number of particles.

### Variations to Model Parameters

In this study, we have intentionally avoided “fitting” model parameters to the
data, in order to demonstrate that no tuning is required for a tilted dark
halo to reproduce the observed warp/flare. In Extended Data Figures 3 and 4 we
show how changes in the model parameters can affect the warp. In the former,
we vary (1) the scale length of the tilted dark halo and (2) the present-day
scale-length of the tracer particles. In the latter, we vary (1) the tilt
angle of the dark halo and (2) the mass fraction of the tilted component of
the dark halo compared to the spherical component. Aside from the parameters
being modulated, all other parameters are fixed to their values from the
simulation presented in Figure 1. In each panel, we calculate the fraction of
stars that are off the $Z=0$ plane by more than 0.25 kpc,
$N_{\text{warp}}/N_{\text{plane}}$. If $N_{\text{warp}}/N_{\text{plane}}$ is
greater than $0.5\%$, we fit a warp model (dotted lines) and show the warp
amplitude at 20 kpc as $Z_{20}$. The disk warps in all models except for the
case of a non-tilted halo (last row of Extended Data Figure 4). These Figures
demonstrate that warping is a general response of the disk to a tilted dark
halo. Furthermore, a more complete observational picture of the Galactic warp
can help constrain properties of the tilted dark matter halo, such as its mass
fraction and tilt angle.

### Context in Cosmological Simulations

A key assumption of our idealized simulation is the fixed orientation of the
halo with respect to the growing disk. It is thus important to understand to
what extent these assumptions are applicable in a simulation with a live halo
and self-consistently growing disk. Specifically, the time that it takes for a
tilted dark halo to eventually align with the disk is an open question. If
this timescale is substantially longer than the warp onset timescale (a few
hundred Myr, see Extended Data Fig. 2), then the mechanisms studied in our
idealized simulation can apply more generally to slowly-changing halos.

To address this question, Han et al. (in prep) analyze Milky Way-like galaxies
in the Illustris TNG50 cosmological magneto-hydrodynamic simulations[36, 37].
They first show that a significant fraction of MW analogs in TNG50 have
present-day tilted dark halos at $r<50\text{ kpc}$. About 50% of halos are
tilted more than $10^{\circ}$, and 25% of halos are tilted more than
$20^{\circ}$. Then, they identify a galaxy that experiences a major merger at
$\sim 7\text{ Gyr}$ ago that results in a tilted dark halo. For this galaxy,
the angle of misalignment of the disk and the dark halo reduces from
$50^{\circ}$ to $20^{\circ}$ over 5 Gyr. Furthermore, the disk of this galaxy
warps shortly after the onset of the tilted dark halo with a delay time of
$<1\text{ Gyr}$. The galaxy does not have any massive satellites at the time
of the onset of the warp. TNG50 was not tuned in any way to produce these
results; rather, the long-lived tilted dark halos and warps emerge naturally
in the simulations. These results show that tilted dark halos can (1) be long-
lived in a live, cosmological environment, and (2) generate warps in galactic
disks on timescales shorter than the change in the tilt angle. Furthermore,
this result shows that the Milky Way’s dark halo was likely more tilted in the
past and has decreased to its current value ($\sim 25^{\circ}$) at present
day.

## References

  * [1] Chen, X. _et al._ An intuitive 3D map of the Galactic warp’s precession traced by classical Cepheids.  _Nature Astronomy_ 3, 320–325 (2019).  1902.00998.
  * [2] Wang, H. F. _et al._ Mapping the Galactic Disk with the LAMOST and Gaia Red Clump Sample. VI. Evidence for the Long-lived Nonsteady Warp of Nongravitational Scenarios.  _ApJ_ 897, 119 (2020).
  * [3] Levine, E. S., Blitz, L. & Heiles, C.  The Vertical Structure of the Outer Milky Way H I Disk.  _ApJ_ 643, 881–896 (2006).  astro-ph/0601697.
  * [4] Kalberla, P. M. W., Dedes, L., Kerp, J. & Haud, U.  Dark matter in the Milky Way. II. The HI gas distribution as a tracer of the gravitational potential.  _A &A_ 469, 511–527 (2007).  0704.3925.
  * [5] Dame, T. M. & Thaddeus, P.  A Molecular Spiral Arm in the Far Outer Galaxy.  _ApJ_ 734, L24 (2011).
  * [6] Wouterloot, J. G. A., Brand, J., Burton, W. B. & Kwee, K. K.  IRAS sources beyond the solar circle. II. Distribution in the galactic warp.  _A &A_ 230, 21–36 (1990).
  * [7] Drimmel, R. & Spergel, D. N.  Three-dimensional Structure of the Milky Way Disk: The Distribution of Stars and Dust beyond 0.35 Rsolar.  _ApJ_ 556, 181–202 (2001).
  * [8] Djorgovski, S. & Sosin, C.  The Warp of the Galactic Stellar Disk Detected in IRAS Source Counts.  _ApJ_ 341, L13 (1989).
  * [9] Yusifov, I.  Pulsars and the Warp of the Galaxy.  In Uyaniker, B., Reich, W. & Wielebinski, R. (eds.) _The Magnetized Interstellar Medium_ , 165–169 (2004).  astro-ph/0405517.
  * [10] Momany, Y. _et al._ Outer structure of the Galactic warp and flare: explaining the Canis Major over-density.  _A &A_ 451, 515–538 (2006).
  * [11] Cantat-Gaudin, T. _et al._ Painting a portrait of the Galactic disc with its stellar clusters.  _A &A_ 640, A1 (2020).  2004.07274.
  * [12] Newberg, H. J. _et al._ The Ghost of Sagittarius and Lumps in the Halo of the Milky Way.  _ApJ_ 569, 245–274 (2002).  astro-ph/0111095.
  * [13] Martin, N. F. _et al._ A dwarf galaxy remnant in Canis Major: the fossil of an in-plane accretion on to the Milky Way.  _MNRAS_ 348, 12–23 (2004).  astro-ph/0311010.
  * [14] Naidu, R. P. _et al._ Evidence from the H3 Survey That the Stellar Halo Is Entirely Comprised of Substructure.  _ApJ_ 901, 48 (2020).  2006.08625.
  * [15] Han, J. J. _et al._ The Stellar Halo of the Galaxy is Tilted and Doubly Broken.  _AJ_ 164, 249 (2022).
  * [16] Heyer, M. & Dame, T. M.  Molecular Clouds in the Milky Way.  _ARA &A_ 53, 583–629 (2015).
  * [17] Burke, B. F.  Systematic distortion of the outer regions of the galaxy.  _AJ_ 62, 90 (1957).
  * [18] Kerr, F. J., Hindman, J. V. & Carpenter, M. S.  The Large-Scale Structure Of the Galaxy.  _Nature_ 180, 677–679 (1957).
  * [19] Hunter, C. & Toomre, A.  Dynamics of the Bending of the Galaxy.  _ApJ_ 155, 747 (1969).
  * [20] Toomre, A.  Theories of WARPS.  In Athanassoula, E. (ed.) _Internal Kinematics and Dynamics of Galaxies_ , vol. 100, 177–185 (1983).
  * [21] Olling, R. P. & Merrifield, M. R.  The Shape of the Milky Way’s Dark Halo.  In Zaritsky, D. (ed.) _Galactic Halos_ , vol. 136 of _Astronomical Society of the Pacific Conference Series_ , 219 (1998).  astro-ph/9710224.
  * [22] Olling, R. P. & Merrifield, M. R.  Two measures of the shape of the dark halo of the Milky Way.  _MNRAS_ 311, 361–369 (2000).  astro-ph/9907353.
  * [23] Sparke, L. S. & Casertano, S.  A model for persistent galactic warps.  _MNRAS_ 234, 873–898 (1988).
  * [24] Debattista, V. P. & Sellwood, J. A.  Warped Galaxies from Misaligned Angular Momenta.  _ApJ_ 513, L107–L110 (1999).  astro-ph/9901153.
  * [25] Jiang, I.-G. & Binney, J.  WARPS and cosmic infall.  _MNRAS_ 303, L7–L10 (1999).  astro-ph/9807161.
  * [26] Poggio, E. _et al._ Measuring the vertical response of the Galactic disc to an infalling satellite.  _MNRAS_ 508, 541–559 (2021).  2011.11642.
  * [27] Ostriker, E. C. & Binney, J. J.  Warped and tilted galactic discs.  _MNRAS_ 237, 785–798 (1989).
  * [28] Dubinski, J. & Chakrabarty, D.  Warps and Bars from the External Tidal Torques of Tumbling Dark Halos.  _ApJ_ 703, 2068–2081 (2009).
  * [29] Laporte, C. F. P., Gómez, F. A., Besla, G., Johnston, K. V. & Garavito-Camargo, N.  Response of the Milky Way’s disc to the Large Magellanic Cloud in a first infall scenario.  _MNRAS_ 473, 1218–1230 (2018).
  * [30] Erkal, D. _et al._ The total mass of the Large Magellanic Cloud from its perturbation on the Orphan stream.  _MNRAS_ 487, 2685–2700 (2019).  1812.08192.
  * [31] Prada, J., Forero-Romero, J. E., Grand, R. J. J., Pakmor, R. & Springel, V.  Dark matter halo shapes in the Auriga simulations.  _MNRAS_ 490, 4877–4888 (2019).
  * [32] Emami, R. _et al._ Morphological Types of DM Halos in Milky Way-like Galaxies in the TNG50 Simulation: Simple, Twisted, or Stretched.  _ApJ_ 913, 36 (2021).
  * [33] Debattista, V. P. _et al._ What’s up in the Milky Way? The orientation of the disc relative to the triaxial halo.  _MNRAS_ 434, 2971–2981 (2013).  1301.2670.
  * [34] Shao, S., Cautun, M., Deason, A. & Frenk, C. S.  The twisted dark matter halo of the Milky Way.  _MNRAS_ 504, 6033–6048 (2021).  2005.03025.
  * [35] Han, J. J. _et al._ A Tilt in the Dark Matter Halo of the Galaxy.  _ApJ_ 934, 14 (2022).
  * [36] Pillepich, A. _et al._ First results from the TNG50 simulation: the evolution of stellar and gaseous discs across cosmic time.  _MNRAS_ 490, 3196–3233 (2019).  1902.05553.
  * [37] Nelson, D. _et al._ The IllustrisTNG simulations: public data release.  _Computational Astrophysics and Cosmology_ 6, 2 (2019).  1812.05609.
  * [38] Belokurov, V., Erkal, D., Evans, N. W., Koposov, S. E. & Deason, A. J.  Co-formation of the disc and the stellar halo.  _MNRAS_ 478, 611–619 (2018).  1802.03414.
  * [39] Helmi, A. _et al._ The merger that led to the formation of the Milky Way’s inner stellar halo and thick disk.  _Nature_ 563, 85–88 (2018).  1806.06038.
  * [40] Gallart, C. _et al._ Uncovering the birth of the Milky Way through accurate stellar ages with Gaia.  _Nature Astronomy_ 3, 932–939 (2019).  1901.02900.
  * [41] Bonaca, A. _et al._ Timing the Early Assembly of the Milky Way with the H3 Survey.  _ApJ_ 897, L18 (2020).  2004.11384.
  * [42] Naidu, R. P. _et al._ Reconstructing the Last Major Merger of the Milky Way with the H3 Survey.  _ApJ_ 923, 92 (2021).  2103.03251.
  * [43] Lilleengen, S. _et al._ The effect of the deforming dark matter haloes of the Milky Way and the Large Magellanic Cloud on the Orphan-Chenab stream.  _MNRAS_ 518, 774–790 (2023).
  * [44] Peter, A. H. G., Rocha, M., Bullock, J. S. & Kaplinghat, M.  Cosmological simulations with self-interacting dark matter - II. Halo shapes versus observations.  _MNRAS_ 430, 105–120 (2013).  1208.3026.
  * [45] Evans, N. W., O’Hare, C. A. J. & McCabe, C.  Refinement of the standard halo model for dark matter searches in light of the Gaia Sausage.  _Phys. Rev. D_ 99, 023012 (2019).
  * [46] Navarro, J. F., Frenk, C. S. & White, S. D. M.  A Universal Density Profile from Hierarchical Clustering.  _ApJ_ 490, 493–508 (1997).  astro-ph/9611107.
  * [47] Bland-Hawthorn, J. & Gerhard, O.  The Galaxy in Context: Structural, Kinematic, and Integrated Properties.  _ARA &A_ 54, 529–596 (2016).  1602.07702.
  * [48] Miyamoto, M. & Nagai, R.  Three-dimensional models for the distribution of mass in galaxies.  _Publications of the Astronomical Society of Japan_ 27, 533–543 (1975).
  * [49] Hernquist, L.  An Analytical Model for Spherical Galaxies and Bulges.  _ApJ_ 356, 359 (1990).
  * [50] Price-Whelan, A. M.  Gala: A Python package for galactic dynamics.  _The Journal of Open Source Software_ 2, 388 (2017).
  * [51] Negroponte, J. & White, S. D. M.  Simulations of mergers between disc-halo galaxies.  _MNRAS_ 205, 1009–1029 (1983).
  * [52] Carlberg, R. G. & Freedman, W. L.  Dissipative models of spiral galaxies.  _ApJ_ 298, 486–492 (1985).
  * [53] Frankel, N., Rix, H.-W., Ting, Y.-S., Ness, M. & Hogg, D. W.  Measuring Radial Orbit Migration in the Galactic Disk.  _ApJ_ 865, 96 (2018).  1805.09198.
  * [54] Sellwood, J. A. & Binney, J. J.  Radial mixing in galactic discs.  _MNRAS_ 336, 785–796 (2002).  astro-ph/0203510.
  * [55] Frankel, N., Sanders, J., Rix, H.-W., Ting, Y.-S. & Ness, M.  The Inside-out Growth of the Galactic Disk.  _ApJ_ 884, 99 (2019).  1909.07118.

Extended Data Fig. 1: Present-day distribution of star particles in the
simulation in Galactocentric cylindrical coordinates. Negative R indicates
azimuthal angles that are within $90^{\circ}$ of the Northern warp, and
positive R indicates azimuthal angles that are within $90^{\circ}$ of the
Southern warp. The top panels show the warp, and the bottom panels show the
vertical deviation from the average warp. The vertical deviation
systematically increases toward the outer Galaxy, demonstrating the flare of
the disk. Points that are in the outer Galaxy are plotted with larger circles.
Particles colored by birth radius reveal that the magnitude of the warp
correlates strongly with the birth radius. Extended Data Fig. 2:  Time
evolution of the warp amplitude at fixed radius $R=16\text{ kpc}$. At $t=0$,
the disk is initialized to have no warp. Error bars indicate $1\sigma$
statistical uncertainty in the warp fit. Within a few hundred Myr, the warp
reaches maximum amplitude. After a transient oscillatory phase from
$t=500-1500\text{ Myr}$, the warp reaches a steady-state amplitude. This plot
demonstrates that the warp onsets quickly, within one rotation period of the
disk ($400\text{ Myr}$ for a star at 16 kpc).  Extended Data Fig. 3:
Variation of (1) the scale length of the tilted component of the dark halo and
(2) the present-day scale length of the tracer particles. Aside from the two
parameters being modulated, all other parameters are fixed to the simulation
presented in Figure 1. In each panel, we show the fraction of stars that are
off the $Z=0$ plane by more than 0.25 kpc, $N_{\text{warp}}/N_{\text{plane}}$.
If this fraction is greater than $0.5\%$, we fit a warp model (dotted lines)
and show the warp amplitude at 20 kpc as $Z_{20}$. The warp amplitude
correlates positively with the scale length of the disk, and anti-correlates
with the scale length of the tilted dark halo.  Extended Data Fig. 4:
Variation of (1) the tilt angle of the dark halo (2) the mass fraction of the
tilted component of the dark halo. Aside from the two parameters being
modulated, all other parameters are fixed to the simulation presented in
Figure 1. Similar to Extended Data Figure 3, we show the fraction of stars
that are off the $Z=0$ plane by more than 0.25 kpc,
$N_{\text{warp}}/N_{\text{plane}}$. If this fraction is greater than $0.5\%$,
we fit a warp model (dotted lines) and show the warp amplitude at 20 kpc as
$Z_{20}$. The warp amplitude correlates positively with both the tilt angle
and the mass fraction of the tilted halo.


# Scalable Infomin Learning

Yanzhi Chen1, Weihao Sun2, Yingzhen Li3, Adrian Weller1,4
1University of Cambridge, 2Rutgers University, 3Imperial College London, 4Alan
Turing Institute

###### Abstract

The task of infomin learning aims to learn a representation with high utility
while being uninformative about a specified target, with the latter achieved
by minimising the mutual information between the representation and the
target. It has broad applications, ranging from training fair prediction
models against protected attributes, to unsupervised learning with
disentangled representations. Recent works on infomin learning mainly use
adversarial training, which involves training a neural network to estimate
mutual information or its proxy and thus is slow and difficult to optimise.
Drawing on recent advances in slicing techniques, we propose a new infomin
learning approach, which uses a novel proxy metric to mutual information. We
further derive an accurate and analytically computable approximation to this
proxy metric, thereby removing the need of constructing neural network-based
mutual information estimators. Experiments on algorithmic fairness,
disentangled representation learning and domain adaptation verify that our
method can effectively remove unwanted information with limited time budget.

##  1 Introduction

Learning representations that are uninformative about some target but still
useful for downstream applications is an important task in machine learning
with many applications in areas including algorithmic fairness [1, 2, 3, 4],
disentangled representation learning [5, 6, 7, 8], information bottleneck [9,
10], and invariant representation learning [11, 12, 13, 14].

A popular method for the above task is adversarial training [1, 3, 4, 7, 15,
2, 11], where two neural networks, namely the encoder and the adversary, are
trained jointly to compete with each other. The encoder’s goal is to learn a
representation with high utility but contains no information about the target.
The adversary, on the contrary, tries to recover the information about the
target from the learned representation as much as possible. This leads to a
minmax game similar to that in generative adversarial networks [16].
Adversarial training is effective with a strong adversary, however, it is
often challenging to train the adversary thoroughly in practice, due to time
constraints and/or optimisation difficulties [17, 18]. In fact, recent studies
have revealed that adversarial approaches may not faithfully produce an
infomin representation in some cases [19, 20, 18, 21, 22]. This motivates us
to seek a good, adversarial training-free alternative for scalable infomin
learning.

In this work, we propose a new method for infomin learning which is almost as
powerful as using a strong adversary but is highly scalable. Our method is
inspired by recent advances in information theory which proposes to estimate
mutual information in the sliced space [23]. We highlight the following
contributions:

  * •

We show that for infomin learning, an accurate estimate of mutual information
(or its bound) is unnecessary: testing and optimising statistical independence
in some sliced spaces is sufficient;

  * •

We develop an analytical approximation to such sliced independence test, along
with a scalable algorithm for infomin learning based on this approximation. No
adversarial training is required.

Importantly, the proposed method can be applied to a wide range of infomin
learning tasks without any constraint on the form of variables or any
assumption about the distributions. This contrasts our method to other
adversarial training-free methods which are either tailored for discrete or
univariate variables [24, 25, 25, 26, 22] or rely on variational approximation
to distributions [10, 27, 19, 12].

##  2 Background

Infomin representation learning. Let $X\in\mathbbm{R}^{D}$ be the data,
$Y\in\mathbbm{R}^{D^{\prime}}$ be the target we want to predict from $X$. The
task we consider here is to learn some representation $Z=f(X)$ that is useful
for predicting $Y$ but is uninformative about some target
$T\in\mathbbm{R}^{d}$. Formally, this can be written as

$\min_{f}\mathcal{L}(f(X);Y)+\beta\cdot I(f(X);T)$ (1)

where $f$ is an encoder, $\mathcal{L}$ is some loss function quantifying the
utility of $Z$ for predicting $Y$ and $I(f(X);T)$ quantifies the amount of
information left in $Z$ about $T$. $\beta$ controls the trade-off between
utility and uninformativeness. Many tasks in machine learning can be seen as
special cases of this objective. For example, by setting $T$ to be (a set of)
sensitive attributes e.g. race, gender or age, we arrive at fair
representation learning [28, 1, 2, 21]. When using a stochastic encoder, by
setting $Y$ to be $X$ and $T$ to be some generative factors e.g., a class
label, we arrive at disentangled representation learning [5, 6, 7, 8].
Similarly, the information bottleneck method [9, 10] corresponds to setting
$T=X$, which learns representations expressive for predicting $Y$ while being
compressive about $X$.

Adversarial training for infomin learning. A key ingredient in objective (1)
is to quantify $I(f(X);T)$ as the informativeness between $f(X)$ and $T$. One
solution is to train a predictor for $T$ from $f(X)$ and use the prediction
error as a measure of $I(f(X);T)$ [1, 3, 4, 29]. Another approach is to first
train a classifier to distinguish between samples from $p(Z,T)$ vs. $p(Z)p(T)$
[7, 15] or to distinguish samples $Z\sim p(Z|T)$ with different $T$ [2, 11],
then use the classification error to quantify $I(f(X);T)$. All these methods
involve the training of a neural network $t$ to provide a lower-bound estimate
to $I(f(X);T)$, yielding a minmax optimisation problem

$\min_{f}\max_{t}\mathcal{L}(f(X);Y)+\beta\cdot\hat{I}_{t}(f(X);T)$ (2)

where $\hat{I}_{t}(f(X);T)$ is an estimator constructed using $t$ that lower-
bounds $I(f(X);T)$. The time complexity of optimising (2) is $O(L_{1}L_{2})$
where $L_{1}$ and $L_{2}$ are the number of gradient steps for the min and the
max step respectively. The strength of $t$ is crucial for the quality of the
learned representation [19, 20, 18, 21, 22]. For a strong adversary, a large
$L_{2}$ is possibly needed, but this means a long training time. Conversely, a
weak adversary may not produce a truly infomin representation.

##  3 Methodology

We propose an alternative to adversarial training for optimising (1). Our idea
is to learn representation by the following objective, which replaces
$I(f(X);T)$ in objective (1) with its ‘sliced’ version:

$\min_{f}\mathcal{L}(f(X);Y)+\beta\cdot SI(f(X);T),$ (3)

where $SI$ denotes the sliced mutual information, which was also considered in
[23]. Informally, $SI$ is a ‘facet’ of mutual information that is much easier
to estimate (ideally has closed form) but can still to some extent reflect the
dependence between $Z$ and $T$. Optimising (3) is then equivalent to testing
and minimising the dependence between $Z$ and $T$ from one facet. Importantly,
while testing dependence through only a single facet may be insufficient, by
testing and minimising dependence through various facets across a large number
of mini-batches we eventually see $I(Z;T)\to 0$.

We show one instance for realising $SI$ whose empirical approximation
$\hat{SI}$ has an analytic expression. The core of our method is Theorem 1,
which is inspired by [4, 23].

  * Theorem1.

Let $Z\in\mathbbm{R}^{D}$ and $T\in\mathbbm{R}^{d}$ be two random variables
that have moments. $Z$ and $T$ are statistically independent if and only if
$SI(Z,T)=0$ where $SI(Z,T)$ is defined as follows

$SI(Z;T)=\sup_{h,g,\theta,\phi}\rho(h(\theta^{\top}Z),g(\phi^{\top}T)),$ (4)

where $\rho$ is the Pearson correlation, $h,g:\mathbbm{R}\to\mathbbm{R}$ are
Borel-measurable non-constant functions, and $\theta\in\mathbb{S}^{D-1}$,
$\phi\in\mathbb{S}^{d-1}$ are vectors on the surfaces on $D$-dimensional and
$d$-dimensional hyperspheres.

_Proof_. See the Appendix. ∎

We call $\theta$ and $\phi$ the slices for $Z$ and $T$ respectively, and
$\theta^{\top}Z$, $\phi^{\top}T$ the sliced $Z$ and $T$ respectively.

We sketch here how this result relates to [4, 23]. [23] considers
$\overline{SI}(Z;T)$, defined as the expected mutual information
$\mathbb{E}[I(\theta^{\top}Z,\phi^{\top}T)]$ of the sliced $Z$ and $T$, where
the expectation is taken over respective Haar measures
$\theta\in\mathbb{S}^{D-1}$, $\phi\in\mathbb{S}^{d-1}$. Instead of considering
the mutual information $I(\theta^{\top}Z,\phi^{\top}T)$ in the average case,
we handle Pearson correlation over the supreme functions $h,g$ as defined
above, which links to Rényi’s maximal correlation [4, 30, 31, 32] and has some
interesting properties suitable for infomin representation learning.

Intuitively, Theorem 1 says that in order to achieve $I(Z;T)\to 0$, we need
not estimate $I(Z;T)$ in the original space; rather we can test (and maximise)
independence in the sliced space as realised by (4). Other realisations of the
sliced mutual information $SI$ may also be used. The major merit of the
realisation (4) is it allows an analytic expression for its empirical
approximation, as shown below.

Analytic approximation to _SI_. An empirical approximation to (4) is

$SI(Z;T)\approx\sup_{i,j}\sup_{h_{i},g_{j}}\rho(h_{i}(\theta^{\top}_{i}Z),g_{j}(\phi^{\top}_{j}T)),$
$\text{where }\quad\theta_{i}\sim\mathcal{U}(\mathbb{S}^{D-1}),\enskip
i=1,...,S,\qquad\phi_{j}\sim\mathcal{U}(\mathbb{S}^{d-1}),\enskip j=1,...,S.$
(5)

i.e., we approximate (4) by randomly sampling a number of slices $\theta,\phi$
uniformly from the surface of two hyperspheres $\mathbb{S}^{D-1}$ and
$\mathbb{S}^{d-1}$ and pick those slices where the sliced $Z$ and the sliced
$T$ are maximally associated. With a large number of slices, it is expected
that (5) will approximate (4) well. We refer to [23] for a theoretical
analysis on the number of required slices in estimator-agnostic settings. In
Appendix B we also investigate empirically how this number will affect
performance.

For each slicing direction, we further assume that the supreme functions
$h_{i},g_{j}:\mathbbm{R}\to\mathbbm{R}$ for that direction can be well
approximated by $K$-order polynomials given sufficiently large $K$, i.e.

$h_{i}(a)\approx\hat{h}_{i}(a)=\sum^{K}_{k=0}w_{ik}\sigma(a)^{k},\qquad
g_{j}(a)\approx\hat{g}_{j}(a)=\sum^{K}_{k=0}v_{jk}\sigma(a)^{k},$

where $\sigma(\cdot)$ is a monotonic function which maps the input to the
range of $[-1,1]$. Its role is to ensure that $\sigma(a)$ always has finite
moments, so that the polynomial approximation is well-behaved. Note that no
information will be lost by applying $\sigma(\cdot)$. Here we take
$\sigma(\cdot)$ as the tanh function. In Appendix A we also investigate
theoretically the approximation error of this polynomial approximation scheme.
Other approximation schemes such as random feature model [33] can also be used
to approximate $h_{i}$ and $g_{j}$, which will be explored in the future. In
this work, we simply set $K=3$.

With this polynomial approximation, the solving of each functions
$h_{i},g_{j}$ in (5) reduces to finding their weights $w_{i},v_{j}$:

$\sup_{h_{i},g_{j}}\rho(h_{i}(\theta^{\top}_{i}Z),g_{j}(\phi^{\top}_{j}T))\approx\sup_{w_{i},g_{j}}\rho(w_{i}^{\top}Z^{\prime}_{i},v_{j}^{\top}T^{\prime}_{j}),$
$Z^{\prime}_{i}=[1,\sigma(\theta^{\top}_{i}Z),...,\sigma(\theta^{\top}_{i}Z)^{K}],\qquad
T^{\prime}_{j}=[1,\sigma(\phi^{\top}_{j}T),...,\sigma(\phi^{\top}_{j}T)^{K}]$

This is known as canonical correlation analysis (CCA) [34] and can be solved
analytically by eigendecomposition. Hence we can find the weights for all
pairs of $h_{i},g_{j}$ by $S^{2}$ eigendecompositions.

In fact, the functions $h_{i},g_{j}$ for all $i,j$ can be solved
simultaneously by performing a larger eigendecomposition only once. We do this
by finding $w,v$ that maximise the following quantity:

$\hat{SI}_{\Theta,\Phi}(Z;T)=\sup_{w,v}\rho(w^{\top}Z^{\prime},v^{\top}T^{\prime}),$
(6)

where

$Z^{\prime}=[Z^{\prime}_{1},...,Z^{\prime}_{i}],\qquad
T^{\prime}=[T^{\prime}_{1},...,T^{\prime}_{j}]$
$Z^{\prime}_{i}=[1,\sigma(\theta^{\top}_{i}Z),...,\sigma(\theta^{\top}_{i}Z)^{K}],\qquad
T^{\prime}_{j}=[1,\sigma(\phi^{\top}_{j}T),...,\sigma(\phi^{\top}_{j}T)^{K}]$
$\theta_{i}\sim\mathcal{U}(\mathbb{S}^{D-1}),\enskip
i=1,...,S,\qquad\phi_{j}\sim\mathcal{U}(\mathbb{S}^{d-1}),\enskip j=1,...,S.$

That is, we first concatenate all $Z^{\prime}_{1},...Z^{\prime}_{S}$ and
$T^{\prime}_{1},...T^{\prime}_{S}$ into two ‘long’ vectors
$Z^{\prime}\in\mathbbm{R}^{(K+1)S}$ and $T^{\prime}\in\mathbbm{R}^{(K+1)S}$
respectively, then solve a CCA problem corresponding to $Z^{\prime}$ and
$T^{\prime}$. We then use (6) to replace (5). The theoretical basis for doing
so is grounded by Theorem 2, which tells that the solution of (6) yields an
upper bound of (5), provided that the polynomial approximation is accurate.

  * Theorem2.

Provided that each $h_{i},g_{j}$ in (5) are $K$-order polynomials  , given the
sampled $\Theta=\\{\theta_{i}\\}^{S}_{i=1},\Phi=\\{\phi_{j}\\}^{S}_{j=1}$, we
have $\hat{SI}_{\Theta,\Phi}(Z;T)\leq\epsilon\Rightarrow$
$\sup_{i,j}\sup_{h_{i},g_{j}}\rho(h_{i}(\theta_{i}^{\top}Z),g_{j}(\phi_{j}^{\top}T))\leq\epsilon$.

_Proof_. See Appendix A. ∎

The intuition behind the proof of Theorem 2 is that if all $Z^{\prime}_{i}$
and $T^{\prime}_{j}$ as a whole cannot achieve a high correlation, each of
them alone can not either.

The benefits for solving $f_{i},g_{j}$ for all slices jointly are two-fold.
(a) The first benefit is better computational efficiency, as it avoids
invoking a for loop and only uses matrix operation. This has better affinity
to modern deep learning infrastructure and libraries (e.g. Tensorflow [35] and
PyTorch [36]) which are optimised for matrix-based operations. In addition to
computational efficiency, another benefit for solving $g_{j},h_{j}$ jointly is
(b) the stronger power in independence testing. More specifically, while some
sliced directions may individually be weak for detecting dependence, they
together as a whole can compensate for each other, yielding a more powerful
test. This also echos with [37].

Algorithm 1 Adversarial Infomin Learning

Input: data $\mathcal{D}=\\{X^{(n)},Y^{(n)},T^{(n)}\\}^{N}_{n=1}$

Output: $Z=f(X)$ that optimises (1)

Hyperparams: $\beta$, $N^{\prime}$, $L_{1}$, $L_{2}$

Parameters: encoder $f$, MI estimator $t$

for $l_{1}$ in $1$ to $L_{1}$ do

sample mini-batch $\mathcal{B}$ from $\mathcal{D}$

sample $\mathcal{D^{\prime}}$ from $\mathcal{D}$ whose size $N^{\prime}<N$

$\rhd$ _Max-step_

for $l_{2}$ in $1$ to $L_{2}$ do

$t\leftarrow t+\eta\nabla_{t}\hat{I}_{t}(f(X);T)$ with data in
$\mathcal{D}^{\prime}$

end for

$\rhd$ _Min-step_

$f\leftarrow f-\eta\nabla_{f}[\mathcal{L}(f(X);Y)+\beta\hat{I}_{t}(f(X);T)$]
with data in $\mathcal{B}$

end for

return $Z=f(X)$

Algorithm 2 Slice Infomin Learning

Input: data $\mathcal{D}=\\{X^{(n)},Y^{(n)},T^{(n)}\\}^{N}_{n=1}$

Output: $Z=f(X)$ that optimises (1)

Hyperparams: $\beta$, $N^{\prime}$, $L$, $S$

Parameters: encoder $f$, weights $w,v$ in $\hat{SI}$

for $l$ in $1$ to $L$ do

sample mini-batch $\mathcal{B}$ from $\mathcal{D}$

sample $\mathcal{D^{\prime}}$ from $\mathcal{D}$ whose size $N^{\prime}<N$

$\rhd$ _Max-step_

sample $S$ slices
$\Theta=\\{\theta_{i}\\}^{S}_{i=1},\Phi=\\{\phi_{j}\\}^{S}_{j=1}$

solve the weights $w,v$ in $\hat{SI}$ (6) analytically with
$\Theta,\Phi,\mathcal{D}^{\prime}$ by eigendecomposition

$\rhd$ _Min-step_

$f\leftarrow f-\eta\nabla_{f}[\mathcal{L}(f(X);Y)+\beta\hat{SI}(f(X);T)$] with
data in $\mathcal{B}$

end for

return $Z=f(X)$

Mini-batch learning algorithm. Given the above approximation (6) to $SI$, we
can now elaborate the details of our mini-batch learning algorithm. In each
iteration, we execute the following steps:

  * •

_Max-step_. Sample $S$ slices
$\Theta=\\{\theta_{i}\\}^{S}_{i=1},\Phi=\\{\phi_{j}\\}^{S}_{j=1}$ and a subset
of the data $\mathcal{D}^{\prime}\subset\mathcal{D}$. Learn the weights $w,v$
of $\hat{SI}(Z;T)$ (6) with the sampled $\Theta,\Phi$ and the data in
$\mathcal{D}^{\prime}$ by eigendecomposition;

  * •

_Min-step_. Update $f$ by SGD: $f\leftarrow
f-\eta\nabla_{f}[\mathcal{L}(f(X),Y)+\beta\hat{SI}(f(X),T)]$ with $Z=f(X)$
where the parameters $w,v$ of $\hat{SI}(f(X),T)$ is learned in the max-step:
$\hat{SI}(Z,T)=\rho(w^{\top}Z^{\prime},v^{\top}T^{\prime})$.

The whole learning procedure is shown in Algorithm 2. We note that the data in
the subset $\mathcal{D}^{\prime}$ can be different from the mini-batch
$\mathcal{B}$ and its size can be much larger than the typical size of a mini-
batch.

Compared to that of adversarial methods [2, 11, 1, 3, 21, 4] as shown in
Algorithm 1, we replace neural network training in the max-step with a
analytic eigendecomposition step, which is much cheaper to execute. As
discussed in Sec 2, if the network $t$ is not trained thoroughly (due to e.g.
insufficient gradient steps $L_{2}$ in the max-step), it may not provide a
sensible estimate to $I_{t}(f(X);T)$ and can hence a weak adversary. Our
method does not suffer from this issue as $\hat{SI}$ is solved analytically.

Finally, as an optional strategy, we can improve our method by actively
seeking more informative slices for independence testing by optimising the
sampled slices with a few gradient steps (e.g. 1-3):

$\Theta\leftarrow\Theta-\xi\nabla_{\Theta}\hat{SI}_{\Theta,\Phi}(Z,T),\qquad\qquad\Phi\leftarrow\Phi-\xi\nabla_{\Phi}\hat{SI}_{\Theta,\Phi}(Z,T)$
(7)

which is still cheap to execute. Such a strategy can be useful when most of
the sampled slices are ineffective in detecting dependence, which typically
happens in later iterations where $I(Z;T)\approx 0$.

##  4 Related works

Neural mutual information estimators. A set of neural network-based methods
[38, 39, 40, 41] have been proposed to estimate the mutual information (MI)
between two random variables, most of which work by maximising a lower bound
of MI [42]. These neural MI estimators are in general more powerful than non-
parametric methods [43, 44, 45, 46] when trained thoroughly, yet the time
spent on training may become the computational bottleneck when applied to
infomin learning.

Upper bound for mutual information. Another line of method for realising the
goal of infomin learning without adversarial training is to find an upper
bound for mutual information [10, 19, 12, 47, 48]. However, unlike lower bound
estimate, upper bound often requires knowledge of either the conditional
densities or the marginal densities [42] which are generally not available in
practice. As such, most of these methods introduce a variational approximation
to these densities whose choice/estimate may be difficult. Our method on the
contrary needs not to approximate any densities.

Slicing techniques. A series of successes have been witnessed for the use of
slicing methods in machine learning and statistics [49], with applications in
generative modelling [50, 51, 52, 53], statistical test [54] and mutual
information estimate [23]. Among them, the work [23] who proposes the concept
of sliced mutual information is very related to this work and directly
inspires our method. Our contribution is a novel realisation of sliced mutual
information suitable for infomin learning.

Fair machine learning. One application of our method is to encourage the
fairness of a predictor. Much efforts have been devoted for the same purpose,
however most of the existing methods can either only work at the classifier
level [25, 24, 4, 55], or only focus on the case where the sensitive attribute
is discrete or univariate [22, 56, 26, 55, 28], or require adversarial
training [2, 11, 1, 3, 21, 4]. Our method on the contrary has no restriction
on the form of the sensitive attribute, can be used in both representation
level and classifier level, and require no adversarial training of neural
networks.

Disentangled representation learning. Most of the methods in this field work
by penalising the discrepancy between the joint distribution $P=q(Z)$ and the
product of marginals $Q=\prod^{D}_{d}q(Z_{d})$ [6, 57, 27, 58, 7].111Note
there exist methods based on group theory [59, 60, 61] which do not assess
distribution discrepancy. However, such discrepancy is often non-trivial to
estimate, so one has to resort to Monte Carlo estimate ($\beta$-TCVAE [27]),
to train a neural network estimator (FactorVAE [7]) or to assess the
discrepancy between $P$ and $Q$ by only their moments (DIP-VAE [57]). Our
method avoids assessing distribution discrepancy directly and instead perform
independence test in the sliced space.

##  5 Experiments

We evaluate our approach on four tasks: independence testing, algorithmic
fairness, disentangled representation learning, domain adaptation. Code is
available at github.com/cyz-ai/infomin.

Evaluation metric. To assess how much information is left in the learned
representation $Z\in\mathbbm{R}^{D}$ about the target $T\in\mathbbm{R}^{K}$,
we calculate the Rényi’s maximal correlation $\rho^{*}(Z,T)$ between $Z$ and
$T$:

$\rho^{*}(Z,T)=\sup_{h,g}\rho(h(Z),g(T))$ (8)

which has the properties $\rho^{*}(Z,T)=0$ if and only if $Z\perp T$ and
$\rho^{*}(Z,T)=1$ if $h(Z)=g(T)$ for some deterministic functions $h,g$ [30].
One can also understand this metric as the easiness of predicting (the
transformed) $T$ from $Z$, or vice versa.222It can be shown $\rho^{*}(Z,T)$ is
equivalent to the normalised mean square error between $h(Z)$ and $g(T)$. As
there is no analytic solution for the supremum in (8), we approximate them by
two neural networks $h,g$ trained with SGD. Early stopping and dropout are
applied to avoid overfitting. The reliability of this neural approximation has
been verified by the literature [4] and is also confirmed by our experiments;
see Appendix B.

This metric is closely related to existing metrics/losses used in fairness and
disentangled representation learning such as demographic parity (DP) [24] and
total correlation (TC) [7]. For example, if $\rho^{*}(Z,T)\to 0$ then it is
guaranteed that $\hat{Y}\perp T$ for any predictor $\hat{Y}=F(Z)$, so
$\rho^{*}(Z,T)$ is an upper bound for DP . Similarly, $\rho^{*}(Z,T)$
coincides with TC which also assesses whether $Z\perp T$. In additional to
this metric, we will also use some task-specific metric; see each experiment
below.

Baselines. We compare the proposed method (denoted as “Slice”) with the
following approaches:

  * •

_Pearson_ , which quantifies $I(Z;T)$ by the Pearson correlation coefficient
$\frac{1}{DK}\sum^{D}_{d}\sum^{K}_{k}\rho(Z_{d};T_{k})$. It was used in [5,
57] as an easy-to-compute proxy to MI to learn disentangled representations;

  * •

_dCorr_ , i.e. distance correlation, a non-parametric method for the
quantifying the independence between two vectors [46]. It was applied in [41]
as a surrogate to MI for representation learning;

  * •

_Neural Rényi_ , an adversarial method for fair machine learning [25] which
quantifies $I(Z;T)$ by the Rényi correlation
$\rho^{*}(Z,T)=\sup_{h,g}\rho(h(Z),g(T))$ with $h,g$ approximated by neural
networks. It can be seen as training a predictor to predict (the transformed)
$T$ from $Z$ and is closely related to many existing methods in algorithmic
fairness and domain adaptation [1, 2, 3, 4, 29];

  * •

_Neural TC_ , an adversarial method for learning disentangled representation
[7, 62] which quantifies $I(Z;T)$ by the total correlation
$TC(Z,T)=KL[p(Z,T)\|p(Z)p(T)]$. To computes TC, a classifier is trained to
classify samples from $p(Z,T)$ and samples from $p(Z)p(T)$. This method can
also be seen as a variant of the popular MINE method [38] for mutual
information estimate.

  * •

_v-CLUB_ , i.e. variational contrastive log upper bound, which introduces a
(learnable) variational distribution $q(T|Z)$ to form an upper bound of MI
[47]: $I(Z;T)\leq\mathbb{E}_{p(Z,T)}[\log q(T|Z)]-\mathbb{E}_{p(Z)p(T)}[\log
q(T|Z)]$. Like adversarial method, $q(T|Z)$ can be learned by a few gradient
steps.

For a fair comparison, for adversarial training-based approaches (i.e. Neural
Rényi, Neural TC), we ensure that the training time of the neural networks in
these methods is at least the same as the execution time of our method or
longer. We do this by controlling the number of adversarial steps $L_{2}$ in
Algorithm 1. The same setup is used for v-CLUB. See each experiment for the
detailed time.

Hyperparameter settings. Throughout our experiments, we use $200$ slices. We
find that this setting is robust across different tasks. An ablation study on
the number of slices is given in Appendix B. The order of the polynomial used
in (6) namely $K$ is set as $K=3$ and is fixed across different tasks.

Computational resource. All experiments are done with a single NVIDIA GeForce
Tesla T4 GPU.

(a) $a$

(b) $a^{2}$

(c) $\sin(a)$

(d) $\text{tanh}(a)$

Figure 1: Comparison of the test power of different independence test methods.
The x-axis corresponds to different values for the dependence level $\alpha$
and the y-axis corresponds to the test power.

###  5.1 Independence testing

We first verify the efficacy of our method as a light-weight but powerful
independence test. For this purpose, we investigate the test power of the
proposed method on various synthetic dataset with different association
patterns between two random variables
$X\in\mathbbm{R}^{10},Y\in\mathbbm{R}^{10}$ and compared to that of the
baselines. The test power is defined as the ability to discern samples from
the joint distribution $p(X,Y)$ and samples from the product of marginal
$p(X)p(Y)$ and is expressed as a probability $p\in[0,1]$. The data is
generated as $Y=(1-\alpha)\langle
t(\mathbf{A}X)\rangle+\alpha\epsilon,X_{d}\sim\mathcal{U}[-3,3],\mathbf{A}_{dd}=1,\mathbf{A}_{dk}=0.2$,
$\epsilon\sim\mathcal{N}(\epsilon;\mathbf{0},\mathbf{I})$, $\alpha\in(0,1)$
and $\langle\cdot\rangle$ is a scaling operation that scales the operand to
the range of $[0,1]$ according to the minimum and maximum values in the
population. The function $t(\cdot)$ determines the association pattern between
$X$ and $Y$ and is chosen from one of the following:
$t(a)=a,a^{2},\sin(a),\text{tanh}(a)$. The factor $\alpha$ controls the
strength of dependence between $X$ and $Y$.

All tests are done on 100 samples and are repeated 1,000 times. We choose this
sample number as it is a typical batch size in mini-batch learning. For
methods involving the learning of parameters (i.e. Slice, Neural Rényi, Neural
TC), we learn their parameters from 10,000 samples. The time for learning the
parameters of Slice, Neural Rényi and Neural TC are 0.14s, 14.37s, 30.18s
respectively. For completeness, we also compare with the ‘optimal test’ which
calculates the Rényi correlation $\rho^{*}(X,Y)=\rho(h(X),g(Y))$ with the
functions $h,g$ exactly the same as the data generating process.

Figure 1 shows the power of different methods under various association
patterns $t$ and dependence levels $\alpha$. Overall, we see that the proposed
method can effectively detect dependence in all cases, and has a test power
comparable to neural network-based methods. Non-parametric tests, by contrast,
fail to detect dependence in quadratic and periodic cases, possibly due to
insufficient power. Neural TC is the most powerful test among all the methods
considered, yet it requires the longest time to train. We also see that the
proposed method is relatively less powerful when $\alpha\geq 0.8$, but in such
cases the statistical dependence between $X$ and $Y$ is indeed very weak (also
see Appendix B). The results suggest that our slice method can provide
effective training signals for infomin learning tasks.

Table 1: Learning fair representations on the US Census Demographic dataset.
Here the utility of the representation is measured by $\rho^{*}(Z,Y)$, while
$\rho^{*}(Z,T)$ is used to quantify the fairness of the representation.
Training time is also provided as the seconds required per max step.

| N/A | Pearson | dCorr | Slice | Neural Rényi | Neural TC | vCLUB
---|---|---|---|---|---|---|---
$\rho^{*}(Z,Y)\uparrow$ | $0.95\pm 0.00$ | $0.95\pm 0.00$ | $0.95\pm 0.00$ | $0.95\pm 0.01$ | $0.95\pm 0.01$ | $0.95\pm 0.02$ | $0.94\pm 0.02$
$\rho^{*}(Z,T)\downarrow$ | $0.92\pm 0.02$ | $0.84\pm 0.08$ | $0.47\pm 0.08$ | $0.07\pm 0.02$ | $0.23\pm 0.10$ | $0.27\pm 0.03$ | $0.16\pm 0.10$
time (sec./max step) | $0.000$ | $0.012$ | $0.087$ | $0.102$ | $0.092$ | $0.097$ | $0.134$

Table 2: Learning fair representations on the UCI Adult dataset. Here the
utility of the representation is measured by $\rho^{*}(Z,Y)$, while
$\rho^{*}(Z,T)$ is used to quantify the fairness of the representation.

| N/A | Pearson | dCorr | Slice | Neural Rényi | Neural TC | vCLUB
---|---|---|---|---|---|---|---
$\rho^{*}(Z,Y)\uparrow$ | $0.99\pm 0.00$ | $0.99\pm 0.00$ | $0.97\pm 0.01$ | $0.98\pm 0.01$ | $0.97\pm 0.01$ | $0.98\pm 0.02$ | $0.97\pm 0.02$
$\rho^{*}(Z,T)\downarrow$ | $0.94\pm 0.02$ | $0.91\pm 0.06$ | $0.71\pm 0.06$ | $0.08\pm 0.02$ | $0.17\pm 0.08$ | $0.36\pm 0.13$ | $0.26\pm 0.12$
time (sec./max step) | $0.000$ | $0.015$ | $0.071$ | $0.112$ | $0.107$ | $0.131$ | $0.132$

###  5.2 Algorithmic fairness

For this task, we aim to learn fair representations $Z\in\mathbbm{R}^{80}$
that are minimally informative about some sensitive attribute $T$. We quantify
how sensitive $Z$ is w.r.t $T$ by Rényi correlation $\rho^{*}(Z,T)$ calculated
using two neural nets. Smaller $\rho^{*}(Z,T)$ is better. The utility of the
learned representation i.e., $\mathcal{L}(Z;Y)$ is quantified by
$\rho^{*}(Z,Y)$. This formulation for utility, as aforementioned, is
equivalent to measuring how well we can predict $Y$ from $Z$. In summary, the
learning objective is:

$\max\rho^{*}(Z;Y)-\beta\hat{I}(Z;T),$

where $\hat{I}(Z;T)$ is estimated by the methods mentioned above. For each
dataset considered, we use 20,000 data for training and 5,000 data for testing
respectively. We carefully tune the hyperparameter $\beta$ for each method so
that the utility $\rho^{*}(Z;Y)$ of that method is close to that of the plain
model (i.e. the model trained with $\beta=0$, denoted as N/A below; other
experiments below have the same setup). For all methods, we use $5,000$
samples in the max step (so $N^{\prime}=5,000$ in Algorithm 1, 2).

US Census Demographic data. This dataset is an extraction of the 2015 American
Community Survey, with 37 features about 74,000 census tracts. The target $Y$
to predict is the percentage of children below poverty line in a tract, and
the sensitive attribute $T$ is the ratio of women in that tract. The result is
shown in Table 1. From the table we see that the proposed slice method
produces highly fair representation with good utility. The low $\rho^{*}(Z,T)$
value indicates that it is unlikely to predict $T$ from $Z$ in our method.
While adversarial methods can also to some extent achieve fairness, it is
still not comparable to our method, possibly because the allocated training
time is insufficient (in Appendix B we study how the effect of the training
time). Non-parameteric methods can not produce truly fair representation,
despite they are fast to execute. v-CLUB, which estimates an upper bound of
MI, achieves better fairness than adversarial methods on average, but has a
higher variance [63].

UCI Adult data. This dataset contains census data for 48,842 instances, with
14 attributes describing their education background, age, race, marital
status, etc. Here, the target $Y$ to predict is whether the income of an
instance is higher than 50,000 USD, and the sensitive attribute $T$ is the
race group. The result is summarised in Table 2. Again, we see that the
proposed slice method outperforms other methods in terms of both fairness and
utility. For this dataset, Neural Rényi also achieves good fairness, although
the gap to our method is still large. Neural TC, by contrast, can not achieve
a comparable level of fairness under the time budget given — a phenomenon also
observed in the US Census dataset. This is possibly because the networks in
Neural TC require longer time to train. The v-CLUB method does not work very
satisfactorily on this task, possibly because the time allocated to learn the
variational distribution $q(Z|T)$ is not enough, leading to a loose upper
bound of $I(Z;T)$.

(a) Adversarial training

(b) Slice

Figure 2: Label swapping experiments on Dsprite dataset. Left: the original
image $X$. Middle: reconstructing $X\approx G(Z,T)$ using $Z=E(X)$ and the
true label $T$. Right: reconstructing $X^{\prime}=G(Z,T^{\prime})$ using
$Z=E(X)$ and a swapped label $T^{\prime}\neq T$. Changing $T$ should only
affects the style.

(a) Adversarial training

(b) Slice

Figure 3: Label swapping experiments on CMU-PIE dataset. Left: the original
image $X$. Middle: reconstructing $X\approx G(Z,T)$ using $Z=E(X)$ and the
true label $T$. Right: reconstructing $X^{\prime}=G(Z,T^{\prime})$ using
$Z=E(X)$ and a swapped label $T^{\prime}\neq T$. Changing $T$ only affect the
expression.

###  5.3 Disentangled representation learning

We next apply our method to the task of disentangled representation learning,
where we wish to discover some latent generative factors irrelevant to the
class label $T$. Here, we train a conditional autoencoder $X\approx G(Z,T)$ to
learn representation $Z=E(X)$ which encodes label-irrelevant information of
$X$. The target to recover is $Y=X$. The utility of $Z$ is therefore
quantified as the reconstruction error:
$\mathcal{L}(Z;Y)=\mathbb{E}[\|G(Z,T)-X\|^{2}_{2}]$, resulting in the
following learning objective:

$\max\mathbb{E}[\|G(Z,T)-X\|^{2}_{2}]-\beta\hat{I}(Z;T).$

The conditional autoencoder uses a architecture similar to that of a
convolutional GAN [64], with the difference being that we insert an adaption
layer $Z^{\prime}=\text{MLP}(Z,T)$ before feeding the features to the decoder.
See Appendix B for the details of its architecture. All images are resized to
32 $\times$ 32\. For all methods, we use $10,000$ samples in the max step (so
$N^{\prime}=10,000$ in Algorithms 1 and 2).

Table 3: Learning label-irrelevant representations on the Dsprite dataset.
Here the utility of the representation is measured by MSE, while
$\rho^{*}(Z,T)$ is used to quantify the level of disentanglement of the
representation. Training time is also provided as the seconds needed per max
step. $\text{Acc}(\hat{T})$ is the accuracy trying to predict $T$ from $Z$. As
there are 3 classes, the ideal value for $\text{Acc}(\hat{T})$ is 0.33.

| N/A | Pearson | dCorr | Slice | Neural Rényi | Neural TC | vCLUB
---|---|---|---|---|---|---|---
$\text{MSE}\downarrow$ | $0.37\pm 0.01$ | $0.44\pm 0.02$ | $0.55\pm 0.03$ | $0.50\pm 0.01$ | $0.61\pm 0.04$ | $0.49\pm 0.03$ | $0.65\pm 0.04$
$\rho^{*}(Z,T)\downarrow$ | $0.91\pm 0.03$ | $0.81\pm 0.07$ | $0.62\pm 0.07$ | $0.08\pm 0.02$ | $0.48\pm 0.05$ | $0.34\pm 0.06$ | $0.22\pm 0.08$
Acc($\hat{T}$) | $0.98\pm 0.01$ | $0.89\pm 0.03$ | $0.76\pm 0.05$ | $0.32\pm 0.02$ | $0.55\pm 0.04$ | $0.54\pm 0.04$ | $0.48\pm 0.03$
time (sec./max step) | $0.000$ | $0.201$ | $0.412$ | $0.602$ | $0.791$ | $0.812$ | $0.689$

Dsprite. A 2D shape dataset [65] where each image is generated by four latent
factors: shape, rotation, scale, locations. Here the class label $T$ is the
shape, which ranges from (square, ellipse, heart). For this dataset, we train
the autoencoder 100 iterations with a batch size of 512. The dimensionality of
the representation for this task is 20 i.e. $Z\in\mathbbm{R}^{20}$. As in the
previous experiments, we provide quantitative comparisons of the utility and
disentanglement of different methods in Table 3. In addition, we provide a
qualitative comparison in Figure 2 which visualises the original image, the
reconstructed image and the reconstructed image with a swapped label. From
Table 3, we see that the proposed method achieve very low $\rho^{*}(Z,T)$
while maintaining good MSE, suggesting that we may have discovered the true
label-irrelevant generative factor for this dataset. This is confirmed
visually by Figure 2(b), where by changing $T$ in reconstruction we only
change the style. By contrast, the separation between $T$ and $Z$ is less
evident in adversarial approach, as can be seen from Table 3 as well as from
Figure 2(a) (see e.g. the reconstructed ellipses in the third column of the
figure. They are more like a interpolation between ellipses and squares).

Table 4: Learning label-irrelevant representations on the CMU-PIE dataset.
Here the utility of the representation is measured by MSE, while
$\rho^{*}(Z,T)$ is used to quantify the level of disentanglement of the
representation. Training time is also provided as the seconds needed per max
step. $\text{Acc}(\hat{T})$ is the accuracy trying to predict $T$ from $Z$. As
there are 2 classes, the ideal value for $\text{Acc}(\hat{T})$ is 0.50∗.

| N/A | Pearson | dCorr | Slice | Neural Rényi | Neural TC | vCLUB
---|---|---|---|---|---|---|---
$\text{MSE}\downarrow$ | $1.81\pm 0.04$ | $1.85\pm 0.05$ | $2.08\pm 0.08$ | $2.15\pm 0.07$ | $2.46\pm 0.06$ | $1.99\pm 0.12$ | $2.02\pm 0.10$
$\rho^{*}(Z,T)\downarrow$ | $0.76\pm 0.04$ | $0.55\pm 0.03$ | $0.27\pm 0.07$ | $0.07\pm 0.01$ | $0.36\pm 0.04$ | $0.39\pm 0.06$ | $0.16\pm 0.06$
Acc($\hat{T}$) | $0.91\pm 0.00$ | $0.76\pm 0.03$ | $0.71\pm 0.06$ | $0.51\pm 0.03$ | $0.73\pm 0.03$ | $0.76\pm 0.04$ | $0.68\pm 0.05$
time (sec./max step) | $0.000$ | $0.184$ | $0.332$ | $0.581$ | $0.750$ | $0.841$ | $0.621$

*For the plain model, $\text{Acc}(\hat{T})$ is not necessarily around $1.0$, as $Z$ does not encode all content of the image.

CMU-PIE. A colored face image dataset [66] where each face image has different
pose, illumination and expression. We use its cropped version [67]. Here the
class label $T$ is the expression, which ranges from (neutral, smile). We
train an autoencoder with 200 iteration and a batch size of 128. The
dimensionality of the representation for this task is 128 i.e.
$Z\in\mathbbm{R}^{128}$. Figure 3 and Table 4 shows the qualitative and
quantitative results respectively. From Figure 3, we see that our method can
well disentangle expression and non-expression representations: one can easily
modify the expression of a reconstructed image by only changing $T$. Other
visual factors of the image including pose, illumination, and identity remain
the same after changing $T$. Adversarial approach can to some extent achieve
disentanglement between $Z$ and $T$, however such disentanglement is
imperfect: not all of the instances can change the expression by only
modifying $T$. This is also confirmed quantitatively by Table 4, where one can
see the relatively high $\rho^{*}(Z,T)$ values in adversarial methods. For
this task, v-CLUB also achieves a low $\rho^{*}(Z,T)$ value, though it is
still outperformed by our method.

###  5.4 Domain adaptation

We finally consider the task of domain adaptation, where we want to learn some
representation $Z$ that can generalise across different datasets. For this
task, a common assumption is that we have assess to two dataset
$\mathcal{D}_{s}=\\{X^{(i)},Y^{(i)}\\}^{n}_{i=1}$ and
$\mathcal{D}_{t}=\\{X^{(j)}\\}^{m}_{j=1}$ whose classes are the same but are
collected differently. Only the data in $\mathcal{D}_{s}$ has known labels.
Following [47], we learn $Z$ as follows:

$Z_{c}=f_{c}(X),\qquad Z_{d}=f_{d}(X)$
$\mathcal{L}_{c}=\mathbb{E}_{X,Y\in\mathcal{D}_{s}}[Y^{\top}\log
C(Z_{c})],\qquad\mathcal{L}_{d}=\mathbb{E}_{X\in\mathcal{D}_{s}}[\log
D(Z_{d})]+\mathbb{E}_{X\in\mathcal{D}_{t}}[\log(1-D(Z_{d}))],$
$\min\mathcal{L}_{c}+\mathcal{L}_{d}+\beta\hat{I}(Z_{c},Z_{d}),$

where $Z_{c}$, $Z_{d}$ are disjoint parts of $Z$ that encode the content
information and the domain information of $X$ separately. $C$ is the content
classifier that maps $X$ to a $(K-1)$-simplex ($K$ is the number of classes)
and $D$ is the domain classifier that distinguishes the domain from which $X$
comes. Since the classifier $C$ only sees labels in $\mathcal{D}_{s}$, we call
$\mathcal{D}_{s}$ the source domain and $\mathcal{D}_{t}$ the target domain.
For the two encoders $f_{c}$ and $f_{d}$, we use Resnets [68] with 7 blocks
trained with 100 iterations and a batch size of 128. Here
$Z_{c},Z_{d}\in\mathbbm{R}^{256}$. We use $N^{\prime}=5,000$ samples in the
max step for all methods.

MNIST $\to$ MNIST-M. Two digit datasets with the same classes but different
background colors. Both datasets have 50,000 training samples. Table 5 shows
the result, indicating that our method can more effectively remove the
information about the domain. This is further confirmed by the T-SNE [69] plot
in Figure 4, where one can hardly distinguish the samples of $Z_{c}$ from the
two domains. This naturally leads to a higher target domain accuracy
$\text{Acc}(\hat{Y_{t}})$ than other methods.

CIFAR10 $\to$ STL10. Two datasets of natural images sharing 9 classes. There
are 50,000 and 5,000 training samples in the two datasets respectively.
Following existing works [70, 71, 47], we remove the non-overlapping classes
from both datasets. Table 5 and Figure 4 show the result. Again, we see that
our method can more effectively remove domain information from the learned
representation.

(a) M $\to$ MM, adversarial

(b) M $\to$ MM, slice

(c) C $\to$ S, adversarial

(d) C $\to$ S, slice

Figure 4: T-SNE plots of the learned content representations $Z_{c}$ in domain
adaptation tasks. (a)(c) show the cases when the adversary is not trained
thoroughly (i.e. $L_{2}$ in Algorithm 1 is set too small).  Table 5: Learning
domain-invariant representations. Here $\text{Acc}(\hat{Y}_{s})$ and
$\text{Acc}(\hat{Y}_{t})$ are the classification accuracy in the source and
the target domains respectively. Time used per max step is given.

MNIST $\to$ MNIST-M
---
| N/A | Slice | Neural TC | vCLUB
$\text{Acc}(\hat{Y_{s}})\uparrow$ | $99.3\pm 0.04$ | $99.2\pm 0.02$ | $99.2\pm 0.04$ | $99.0\pm 0.03$
$\text{Acc}(\hat{Y_{t}})\uparrow$ | $46.3\pm 0.03$ | $98.5\pm 0.45$ | $80.1\pm 0.17$ | $93.8\pm 0.10$
$\rho^{*}(Z_{c},Z_{d})\downarrow$ | $0.86\pm 0.05$ | $0.06\pm 0.01$ | $0.64\pm 0.04$ | $0.49\pm 0.12$
time (sec./step) | $0.000$ | $2.578$ | $3.282$ | $3.123$

CIFAR10 $\to$ STL10
---
| N/A | Slice | Neural TC | vCLUB
$\text{Acc}(\hat{Y_{s}})\uparrow$ | $93.0\pm 0.03$ | $92.5\pm 0.03$ | $92.4\pm 0.03$ | $92.1\pm 0.04$
$\text{Acc}(\hat{Y_{t}})\uparrow$ | $75.9\pm 0.09$ | $82.3\pm 0.03$ | $80.8\pm 0.08$ | $78.5\pm 0.11$
$\rho^{*}(Z_{c},Z_{d})\downarrow$ | $0.43\pm 0.05$ | $0.08\pm 0.01$ | $0.39\pm 0.07$ | $0.42\pm 0.09$
time (sec./step) | $0.000$ | $3.146$ | $3.222$ | $3.080$

##  6 Conclusion

This work proposes a new method for infomin learning without adversarial
training. A major challenge is how to estimate mutual information accurately
and efficiently, as MI is generally intractable. We sidestep this challenge by
only testing and minimising dependence in a sliced space, which can be
achieved analytically, and we showed this is sufficient for our goal.
Experiments on algorithmic fairness, disentangled representation learning and
domain adaptation verify our method’s efficacy.

Through our controlled experiments, we also verify that adversarial approaches
indeed may not produce infomin representation reliably – an observation
consistent with recent studies. This suggests that existing adversarial
approaches may not converge to good solutions, or may need more time for
convergence, with more gradient steps needed to train the adversary fully. The
result also hints at the potential of diverse randomisation methods as an
alternative to adversarial training in some cases.

While we believe our method can be used in many applications for societal
benefit (e.g. for promoting fairness), since it is a general technique, one
must always be careful to prevent societal harms.

## Acknowledgement

AW acknowledges support from a Turing AI Fellowship under grant EP/V025279/1,
The Alan Turing Institute, and the Leverhulme Trust via CFI. YC acknowledges
funding from Cambridge Trust.

## References

  * [1] David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel.  Learning adversarially fair and transferable representations.  In International Conference on Machine Learning, pages 3384–3393. PMLR, 2018.
  * [2] Harrison Edwards and Amos Storkey.  Censoring representations with an adversary.  In ICLR, 2016.
  * [3] Yanai Elazar and Yoav Goldberg.  Adversarial removal of demographic attributes from text data.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11–21, 2018.
  * [4] Vincent Grari, Boris Ruf, Sylvain Lamprier, and Marcin Detyniecki.  Fairness-aware neural Réyni minimization for continuous features.  arXiv preprint arXiv:1911.04929, 2019.
  * [5] Brian Cheung, Jesse A Livezey, Arjun K Bansal, and Bruno A Olshausen.  Discovering hidden factors of variation in deep networks.  arXiv preprint arXiv:1412.6583, 2014.
  * [6] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-VAE: Learning basic visual concepts with a constrained variational framework.  2016\.
  * [7] Hyunjik Kim and Andriy Mnih.  Disentangling by factorising.  In International Conference on Machine Learning, pages 2649–2658. PMLR, 2018.
  * [8] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem.  Challenging common assumptions in the unsupervised learning of disentangled representations.  In International Conference on Machine Learning, pages 4114–4124. PMLR, 2019.
  * [9] Naftali Tishby and Noga Zaslavsky.  Deep learning and the information bottleneck principle.  In 2015 IEEE information theory workshop (ITW), pages 1–5. IEEE, 2015.
  * [10] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy.  Deep variational information bottleneck.  arXiv preprint arXiv:1612.00410, 2016.
  * [11] Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig.  Controllable invariance through adversarial feature learning.  Advances in neural information processing systems, 30, 2017.
  * [12] Proteek Chandan Roy and Vishnu Naresh Boddeti.  Mitigating information leakage in image representations: A maximum entropy approach.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2586–2594, 2019.
  * [13] Ayush Jaiswal, Daniel Moyer, Greg Ver Steeg, Wael AbdAlmageed, and Premkumar Natarajan.  Invariant representations through adversarial forgetting.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4272–4279, 2020.
  * [14] Renjie Xie, Wei Xu, Jiabao Yu, Aiqun Hu, Derrick Wing Kwan Ng, and A Lee Swindlehurst.  Disentangled representation learning for rf fingerprint extraction under unknown channel statistics.  arXiv preprint arXiv:2208.02724, 2022.
  * [15] Pengyu Cheng, Weituo Hao, and Lawrence Carin.  Estimating total correlation with mutual information bounds.  arXiv preprint arXiv:2011.04794, 2020.
  * [16] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [17] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  Advances in neural information processing systems, 29, 2016.
  * [18] Congzheng Song and Vitaly Shmatikov.  Overlearning reveals sensitive attributes.  arXiv preprint arXiv:1905.11742, 2019.
  * [19] Daniel Moyer, Shuyang Gao, Rob Brekelmans, Aram Galstyan, and Greg Ver Steeg.  Invariant representations without adversarial training.  Advances in Neural Information Processing Systems, 31, 2018.
  * [20] Rui Feng, Yang Yang, Yuehan Lyu, Chenhao Tan, Yizhou Sun, and Chunping Wang.  Learning fair representations via an adversarial framework.  arXiv preprint arXiv:1904.13341, 2019.
  * [21] Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon.  Learning controllable fair representations.  In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2164–2173. PMLR, 2019.
  * [22] Mislav Balunovic, Anian Ruoss, and Martin Vechev.  Fair normalizing flows.  In International Conference on Learning Representations, 2021.
  * [23] Ziv Goldfeld and Kristjan Greenewald.  Sliced mutual information: A scalable measure of statistical dependence.  Advances in Neural Information Processing Systems, 34, 2021.
  * [24] Jérémie Mary, Clément Calauzenes, and Noureddine El Karoui.  Fairness-aware learning for continuous attributes and treatments.  In International Conference on Machine Learning, pages 4382–4391. PMLR, 2019.
  * [25] Sina Baharlouei, Maher Nouiehed, Ahmad Beirami, and Meisam Razaviyayn.  R$\backslash$’enyi fair inference.  arXiv preprint arXiv:1906.12005, 2019.
  * [26] Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa.  Wasserstein fair classification.  In Uncertainty in Artificial Intelligence, pages 862–872. PMLR, 2020.
  * [27] Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud.  Isolating sources of disentanglement in variational autoencoders.  Advances in neural information processing systems, 31, 2018.
  * [28] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork.  Learning fair representations.  In International conference on machine learning, pages 325–333. PMLR, 2013.
  * [29] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The journal of machine learning research, 17(1):2096–2030, 2016\.
  * [30] Alfréd Rényi.  On measures of dependence.  Acta mathematica hungarica, 10(3-4):441–451, 1959.
  * [31] Hans Gebelein.  Das statistische problem der korrelation als variations-und eigenwertproblem und sein zusammenhang mit der ausgleichsrechnung.  ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift für Angewandte Mathematik und Mechanik, 21(6):364–379, 1941.
  * [32] Hermann O Hirschfeld.  A connection between correlation and contingency.  In Mathematical Proceedings of the Cambridge Philosophical Society, volume 31, pages 520–524. Cambridge University Press, 1935.
  * [33] David Lopez-Paz, Philipp Hennig, and Bernhard Schölkopf.  The randomized dependence coefficient.  Advances in neural information processing systems, 26, 2013.
  * [34] Harold Hotelling.  Relations between two sets of variates.  Biometrika, 28(3/4):321–377, 1936.
  * [35] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al.  $\\{$TensorFlow$\\}$: a system for $\\{$Large-Scale$\\}$ machine learning.  In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265–283, 2016.
  * [36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.
  * [37] Ziv Goldfeld, Kristjan Greenewald, Theshani Nuradha, and Galen Reeves.  k-sliced mutual information: A quantitative study of scalability with dimension.  arXiv preprint arXiv:2206.08526, 2022.
  * [38] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm.  Mutual information neural estimation.  In International conference on machine learning, pages 531–540. PMLR, 2018.
  * [39] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio.  Learning deep representations by mutual information estimation and maximization.  arXiv preprint arXiv:1808.06670, 2018.
  * [40] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv e-prints, pages arXiv–1807, 2018.
  * [41] Yanzhi Chen, Dinghuai Zhang, Michael Gutmann, Aaron Courville, and Zhanxing Zhu.  Neural approximate sufficient statistics for implicit models.  arXiv preprint arXiv:2010.10079, 2020.
  * [42] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker.  On variational bounds of mutual information.  In International Conference on Machine Learning, pages 5171–5180. PMLR, 2019.
  * [43] Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song, Bernhard Schölkopf, and Alex Smola.  A kernel statistical test of independence.  Advances in neural information processing systems, 20, 2007.
  * [44] Francis R Bach and Michael I Jordan.  Kernel independent component analysis.  Journal of machine learning research, 3(Jul):1–48, 2002.
  * [45] Barnabás Póczos, Zoubin Ghahramani, and Jeff Schneider.  Copula-based kernel dependency measures.  arXiv preprint arXiv:1206.4682, 2012.
  * [46] Gábor J Székely and Maria L Rizzo.  Partial distance correlation with methods for dissimilarities.  The Annals of Statistics, 42(6):2382–2412, 2014.
  * [47] Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin.  Club: A contrastive log-ratio upper bound of mutual information.  In International conference on machine learning, pages 1779–1788. PMLR, 2020.
  * [48] Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, and Jingjing Liu.  Infobert: Improving robustness of language models from an information theoretic perspective.  arXiv preprint arXiv:2010.02329, 2020.
  * [49] Kimia Nadjahi, Alain Durmus, Lénaïc Chizat, Soheil Kolouri, Shahin Shahrampour, and Umut Simsekli.  Statistical and topological properties of sliced probability divergences.  Advances in Neural Information Processing Systems, 33:20802–20812, 2020.
  * [50] Ishan Deshpande, Ziyu Zhang, and Alexander G Schwing.  Generative modeling using the sliced wasserstein distance.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3483–3491, 2018.
  * [51] Soheil Kolouri, Phillip E Pope, Charles E Martin, and Gustavo K Rohde.  Sliced wasserstein auto-encoders.  In International Conference on Learning Representations, 2018.
  * [52] Mark Rowland, Jiri Hron, Yunhao Tang, Krzysztof Choromanski, Tamas Sarlos, and Adrian Weller.  Orthogonal estimation of wasserstein distances.  In The 22nd International Conference on Artificial Intelligence and Statistics, pages 186–195. PMLR, 2019.
  * [53] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon.  Sliced score matching: A scalable approach to density and score estimation.  In Uncertainty in Artificial Intelligence, pages 574–584. PMLR, 2020.
  * [54] Wenbo Gong, Yingzhen Li, and José Miguel Hernández-Lobato.  Sliced kernelized stein discrepancy.  arXiv preprint arXiv:2006.16531, 2020.
  * [55] Yilun Xu, Hao He, Tianxiao Shen, and Tommi S Jaakkola.  Controlling directions orthogonal to a classifier.  In International Conference on Learning Representations, 2021.
  * [56] Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, Luca Oneto, and Massimiliano Pontil.  Fair regression with wasserstein barycenters.  Advances in Neural Information Processing Systems, 33:7321–7331, 2020.
  * [57] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan.  Variational inference of disentangled latent concepts from unlabeled observations.  arXiv preprint arXiv:1711.00848, 2017.
  * [58] Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner.  Understanding disentangling in $\beta$-vae.  arXiv preprint arXiv:1804.03599, 2018.
  * [59] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner.  Towards a definition of disentangled representations.  arXiv preprint arXiv:1812.02230, 2018.
  * [60] Robin Quessard, Thomas Barrett, and William Clements.  Learning disentangled representations and group structure of dynamical environments.  Advances in Neural Information Processing Systems, 33:19727–19737, 2020.
  * [61] Xinqi Zhu, Chang Xu, and Dacheng Tao.  Commutative lie group vae for disentanglement learning.  In International Conference on Machine Learning, pages 12924–12934. PMLR, 2021.
  * [62] Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel.  Flexibly fair representation learning by disentanglement.  In International conference on machine learning, pages 1436–1445. PMLR, 2019.
  * [63] Jiaming Song and Stefano Ermon.  Understanding the limitations of variational mutual information estimators.  In International Conference on Learning Representations, 2019.
  * [64] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv preprint arXiv:1511.06434, 2015.
  * [65] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner.  dsprites: Disentanglement testing sprites dataset.  https://github.com/deepmind/dsprites-dataset/, 2017.
  * [66] Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, and Simon Baker.  Multi-pie.  Image and vision computing, 28(5):807–813, 2010.
  * [67] Yu Tian, Xi Peng, Long Zhao, Shaoting Zhang, and Dimitris N Metaxas.  Cr-gan: learning complete representations for multi-view generation.  In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 942–948, 2018.
  * [68] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [69] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [70] Geoff French, Michal Mackiewicz, and Mark Fisher.  Self-ensembling for visual domain adaptation.  In International Conference on Learning Representations, 2018.
  * [71] Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon.  A dirt-t approach to unsupervised domain adaptation.  In International Conference on Learning Representations, 2018.