11institutetext: Rosseland Centre for Solar Physics, University of Oslo,
Postboks 1029 Blindern, N-0315 Oslo, Norway 22institutetext: Institute of
Theoretical Astrophysics, University of Oslo, Postboks 1029 Blindern, N-0315
Oslo, Norway
22email<EMAIL_ADDRESS>33institutetext: Hamburger Sternwarte,
Gojenbergsweg 112, 21029 Hamburg, Germany

# EMISSA - Exploring Millimetre Indicators of Solar-Stellar Activity

II. Towards a robust indicator of stellar activity

A. Mohan 1122 S. Wedemeyer 1122 P. H. Hauschildt 33 S. Pandit 1122 M. Saberi
1122

(Received 30 June, 2022 / Accepted 7 August, 2022 )

###### Abstract

Context. An activity indicator, which can provide a robust quantitative
mapping between the stellar activity and the physical properties of its
atmosphere, is important in exploring the evolution of the observed active
phenomena across main-sequence stars of different spectral types. Common
activity indicators do provide qualitative correlations with physical
properties such as Teff and the rotation period, among others. However, due to
the large variability in their values, even for a single star, defining robust
quantitative mappings between activity and physical properties is difficult.
Millimetre (mm) wavelengths probe the different atmospheric layers within the
stellar chromosphere, providing a tomographic view of the atmospheric
dynamics.

Aims. The project aims to define a robust activity indicator by characterising
mm brightness temperature spectra (T${}_{\mathrm{B}}(\nu)$) of the cool main-
sequence stars (Teff $\sim$ 5000 – 7000 K) compiled by paper I in this series.
The sample contains 13 stars, including the Sun.

Methods. We derived the mm T${}_{\mathrm{B}}(\nu)$ spectral indices
($\alpha_{\mathrm{mm}}$) for cool stars, including the Sun, based on
observations in the 30 – 1000 GHz range. The derived values for
$\alpha_{\mathrm{mm}}$ are explored as a function of various physical
parameters and empirical power-law functions were derived. We also compared
$\alpha_{\mathrm{mm}}$ estimates with other activity indicators.

Results. Despite the estimation errors, $\alpha_{\mathrm{mm}}$ values could
distinguish the cool stars well, unlike common activity indicators. The low
estimation errors on the derived trends of $\alpha_{\mathrm{mm}}$ versus
physical parameters suggest that $\alpha_{\mathrm{mm}}$ could be a robust
activity indicator.

Conclusions. We note that $\alpha_{\mathrm{mm}}$, which is linked to
chromospheric thermal stratification and activity in cool stars, can well
distinguish and physically characterise the stars more robustly than common
activity indicators. We emphasise the need for multi-frequency data across the
mm band for stars, with a range of physical parameters and gathered at
multiple epochs during their activity cycles. This will help to explore
$\alpha_{\mathrm{mm}}$ in a statistically robust manner and to study the
emergence of chromospheric heating on the main sequence.

###### Key Words.:

stars: activity - stars: atmospheres - stars: chromosphere - submillimetre:
stars - radio continuum: stars - catalogues

##  1 Introduction

In recent years, there has been a growing interest in the atmospheric activity
of cool main-sequence stars (cool stars; effective temperature, Teff
$\sim$3000 - 7000 K) especially due to the links between activity, exo-space
weather, and exo-planet habitability (e.g. Grießmeier et al., 2005; Vidotto et
al., 2013; Moschou et al., 2019; Odert et al., 2020). Cool stars of spectral
types F – M together host the largest number of Earth-like exoplanets in
habitable zones (Bashi et al., 2020). However, owing to their outer convection
layer, these stars have active atmospheres that drive flares, eruptions, and
other high-energy phenomena, which can potentially affect the atmospheres of
nearby planets in habitable zones (Lammer et al., 2007; Zendejas et al.,
2010). Ground- and space-based instruments provide plenty of data on stellar
atmospheric emission primarily in X-ray to optical wavebands, owing to large
dedicated surveys (e.g. Perryman et al., 1997; De Pascale et al., 2014; Gaia
Collaboration, 2021). Based on the X-ray to optical observables, which are
sensitive to different stellar atmospheric phenomena and layers, different
activity indicators have been constructed. Commonly used indicators include
the ratio of the Ca II H-K flux to bolometric flux
(R${}^{\prime}_{\mathrm{HK}}$, Noyes et al. (1984)) and the X-ray to
bolometric flux ratio ($\mathrm{R_{X}}$). These activity indicators show
trends with several physical parameters such as Teff, the rotation period,
magnetic field strength, metallicity, among others, sometimes with complex
multi-dimensional dependencies (e.g. Noyes et al., 1984; Stepien, 1994a;
Marsden et al., 2014; Vidotto et al., 2014). Though these trends provide a
qualitative physical picture of the physical dependencies of activity, the
large value ranges of these indicators make it hard to derive robust
quantitative relations. For instance, Pace (2013) showed an age-dependent
bimodality of R${}^{\prime}_{\mathrm{HK}}$ values for cool stars of same Teff.
Even the values for a single star varied by factors of $2-4$, making it hard
to distinguish the cool stars (Teff $\sim$ 4000 - 7000 K) along the
R${}^{\prime}_{\mathrm{HK}}$ axis. Similar studies on $\mathrm{R_{X}}$ values
have revealed variations as large as an order of magnitude in Sun-like stars
(e.g. Micela & Marino, 2003). These studies show that the common indicators
cannot provide robust quantitative scales which are essential to better
constrain the atmospheric physical models and gain physical insights on the
emergence of different levels of activity in stars of different types.

Observations and related models suggest that the atmospheric structure of cool
stars undergoes significant changes as function of spectral type or
equivalently in the Teff range 3000 – 7000 K (e.g. Donati & Landstreet, 2009;
Linsky, 2017). Being the middle layer of the stellar atmosphere, chromosphere
plays a crucial role in the generation and propagation of active phenomena
across the stellar atmosphere (for a review, see Linsky, 2017). It is hence
highly desirable to construct a new and more reliable observational indicator
of the chromospheric structure of cool stars, which is closely linked to the
observed activity. Continuum emission in the millimetre (mm) range ($\sim$30 –
1000 GHz) originates at different heights in the stellar chromosphere
primarily due to thermal bremsstrahlung, which is formed under local thermal
equilibrium (LTE) conditions (Wedemeyer et al., 2016). Owing to its high
sensitivity, ALMA enables the detection of chromospheric mm emission for a
larger number of potential stellar targets than possible before. Several
recent solar and stellar mm observations have demonstrated the unique
tomographic potential of the mm-brightness temperature spectrum
(mm-T${}_{\mathrm{B}}(\nu)$) to facilitate the deduction of the chromospheric
thermal structure and dynamics at various atmospheric layers in cool stars
(e.g. White et al., 2017, 2020; Trigilio et al., 2018). In the previous paper
in this series (Paper I; Mohan et al., 2021), the mm-T${}_{\mathrm{B}}(\nu)$
spectrum was derived for main-sequence stars using archival ALMA data. The
sample was made by systematically avoiding stars with potential flux
contamination from known debris disks or unresolved companion stars. Assuming
that the observed stellar mm flux can be approximated with a LTE Rayleigh-
Jeans spectrum, T${}_{\mathrm{B}}(\nu)$ is expected to reflect the atmospheric
thermal stratification. The mm-T${}_{\mathrm{B}}(\nu)$ of every cool star in
the sample significantly deviated from a corresponding purely photospheric
model (PHOENIX; Hauschildt & Baron, 1999) spectrum, with the deviation from
the photospheric model increasing as $\nu$ decreased (Fig. 1). The rise of
$T_{B}$ with decreasing $\nu$ is consistent with probing hotter chromospheric
layers at higher heights, indicating the existence of a chromosphere and, with
that, the presence of chromospheric heating and potentially higher activity
(Paper I). In this paper, we fit the compiled mm-T${}_{\mathrm{B}}(\nu)$ for
every star and explore the relationships between spectral structure and
stellar physical properties, particularly focussing on the cool stars in the
sample.

In Sect. 2, the data and analysis methodology are described. The results are
presented in Sect. 3, and a discussion in Sect. 4 is followed by our
conclusions and outlook in Sect 5.

##  2 Data and methodology

The mm-T${}_{\mathrm{B}}(\nu)$ data used for this work are part of the sample
of main-sequence stars presented in Paper I. Here, we selected only those
stars in the sample with more than one data point in the mm/sub-mm range (30 –
1000 GHz). The resultant sub-sample of nine stars covers a Teff range of
$\sim$ 5000 – 10,0000 K (A – K type). Though the original sample compiled data
in the 10 – 1000 GHz range, we chose to stick to the mm/sub-mm range for two
reasons.

Figure 1: mm-T${}_{\mathrm{B}}(\nu)$ in the 30 – 1000 GHz range. Titles show
the names and spectral types. Family of power-law functions consistent with
the $\alpha_{\mathrm{mm}}$ estimation errors (shaded regions) and photospheric
models (dotted line) are shown. For the Sun, chromospheric FAL models are
shown. Multi-epoch data at $\nu$ are coloured differently.

Firstly, for cool stars, the assumption of LTE emission is believed to be
generally valid beyond 30 GHz. For the Sun, this assumption is confirmed by
detailed 3D atmospheric model calculations (see Wedemeyer et al., 2016, and
references therein). For the other cool stars in our sample, the LTE nature of
the mm emission has been demonstrated via data-driven modelling studies using
scaled 1D solar models (Vernazza et al., 1981) or solar-like models (Tapia-
Vázquez & De la Luz, 2020). Trigilio et al. (2018) carried out this exercise
for the $\alpha$ Cen system (G and K dwarf) and White et al. (2020) did so for
the F dwarfs. However, below 30 GHz, non-thermal emission from the corona can
be relevant as shown for $\epsilon$ Eri by Suresh et al. (2020). Secondly, the
data below 30 GHz are either very sparse or non-existent for the stars in our
sample.

For the $\alpha$ Cen binary, there exist multiple ALMA observations at certain
frequencies gathered years apart and hence revealing the inherent time
variability in the emission. Similarly $\epsilon$ Eri observations at
different frequencies were made years apart (Suresh et al., 2020). The solar
data at 34 GHz come from periods of minimum and maximum activity during solar
cycle 23 (White, 2004). So, the mm-T${}_{\mathrm{B}}(\nu)$ of these stars
represent emission averaged over the respective activity cycle timescales.

##  3 Results

Figure 1 shows a collage of T${}_{\mathrm{B}}(\nu)$ for the cool stars in our
sub-sample along with an A-type star for comparison. The years during which
data were collected are given in the legends along with the corresponding
radio telescopes. Multi-epoch observations at the same frequency are shown in
different colours. The dotted line shows the photospheric emission model. For
the Sun, quiet chromospheric spectra based on FAL A and C models (Loukitcheva
et al., 2004) are shown, revealing the chromospheric origin of the observed
T${}_{\mathrm{B}}(\nu)$. The observed T${}_{\mathrm{B}}(\nu)$ of every star
was fitted by a power law (T${}_{\mathrm{B}}(\nu)$
$\propto\nu^{\alpha_{mm}}$). The shaded region in each subplot shows the
family of curves consistent within the error range of the $\chi^{2}$ fit. The
errors on $\alpha_{\mathrm{mm}}$ could not be estimated for $\gamma$ Vir A and
B since there are only two $\mathrm{T_{B}}$ data points. While deriving
$\alpha_{mm}$ for the $\alpha$ Cen system and the Sun, we used mean
$\mathrm{T_{B}}$ values at frequencies for which multi-epoch data existed. A
systematic error was added to the mean $\mathrm{T_{B}}$ to account for
emission variability. Despite the emission variability in the multi-epoch
data, $\alpha_{\mathrm{mm}}$ values are quite robust with relatively low-error
ranges, letting us explore its physical dependencies.

Figure 2: $\alpha_{\mathrm{mm}}$ versus physical parameters with respective
power-law fits for the old group ($>$1 Gyr). Smaller markers denote stars $<$1
Gyr. Figure 3: Comparison of $\alpha_{\mathrm{mm}}$ value ranges with those in
$\log\mathrm{R}^{\prime}_{\mathrm{HK}}$ and $\log\mathrm{R}_{\mathrm{X}}$ for
cool star sample. The markers used are the same as in Fig. 2.

Figure 2 explores $\alpha_{\mathrm{mm}}$ versus various physical parameters.
The sample consists of both an old and young group of stars with ages either
above or below 1 Gyr, respectively. The plot shows the old group in bigger
markers, revealing the physical dependencies of $\alpha_{\mathrm{mm}}$,
modelled by power-law functions (red curves) with the index, $\beta$. There is
only one cool star in the young group, $\epsilon$ Eri. The A-type stars
$\alpha$PsA and $\alpha$ CMa show a negative $\alpha_{\mathrm{mm}}$, as they
lack hotter upper atmospheres. The T${}_{\mathrm{B}}(\nu)$ of A stars are
consistent with the photospheric model, as seen in Fig. 1 for $\alpha$ CMa. We
need more data for young stars to study their physical trends.

In addition, Figure 3 compares $\alpha_{\mathrm{mm}}$ with common activity
indicators. The error intervals on $\log\mathrm{R}_{\mathrm{X}}$ and
$\log\mathrm{R}^{\prime}_{\mathrm{HK}}$ were estimated based on the results on
their systematic variability for cool stars by Micela & Marino (2003) and Pace
(2013), respectively. The relatively low errors on $\alpha_{\mathrm{mm}}$ help
distinguish the cool stars much better than other indicators and they provide
robust quantitative maps to physical parameters.

##  4 Discussion

Stars are believed to evolve from a high- to low-activity phase with age, due
to continuous loss of energy and angular momentum, which is reflected in the
rise of the rotation period with age (e.g. Skumanich, 1972; Stepien, 1994b;
Davenport, 2016; van Saders et al., 2016). Barnes (2003) classified this age-
related activity evolution by analysing the shift in the distribution of stars
in a rotation period versus B-V plane for stars of different age ranges. The
two groups of distributions that emerged were named ‘I’, the fast rotating
high-activity group, and ‘C’, the slow rotating low-activity group. It was
found that stars tend to evolve from ‘C’ to ‘I’ as they age, with hotter stars
evolving faster. However, F – K dwarfs older than 1 Gyr were found to have
migrated to the ‘I’ branch. In our cool star sub-sample, we find that the old
group shows robust physical trends with physical parameters, to which the lone
young star, $\epsilon$ Eri, does not agree. $\epsilon$ Eri also shows a
significantly higher $\alpha_{\mathrm{mm}}$. Since T${}_{\mathrm{B}}(\nu)$ of
cool stars is a proxy to their chromospheric thermal structure and since the
spectral index, $\alpha_{\mathrm{mm}}$, quantifies the thermal gradients, the
relatively high $\alpha_{\mathrm{mm}}$ of $\epsilon$ Eri hints at higher
atmospheric heating and activity.

###  4.1 Physical dependencies of $\alpha_{\mathrm{mm}}$ for old cool stars

Cooler stars (lower Teff) are expected to be more active due to their stronger
magnetic fields. The $\alpha_{\mathrm{mm}}$ estimates are higher for cooler
stars, probably hinting at more efficient heating mechanisms which can sustain
stronger chromospheric thermal gradients, as well as higher levels of steady
heating and activity. Similarly, stars with higher B-V have higher
$\alpha_{\mathrm{mm}}$. We find that $\alpha_{\mathrm{mm}}$ increases with
period, as opposed to the well-known anti-correlation between activity and
rotation period. Also, stars with higher surface gravity (g) show higher
$\alpha_{\mathrm{mm}}$. Since activity is related to the observed rotation
rates (v sin$i$; $i$ is the inclination angle with respect to the line of
sight), g and Teff, analysing the individual parameter trends with
$\alpha_{\mathrm{mm}}$ may not provide a coherent picture, especially in view
of the small sample size.

In Fig. 4(a), $\alpha_{\mathrm{mm}}$ is compared to (v sin$i$)2/(Rg), the
ratio of surface rotational kinetic energy relative to the gravitational
potential. Stars with higher $\alpha_{\mathrm{mm}}$ tend to show lower
relative rotational kinetic energy. Figure 4(b) shows $\alpha_{\mathrm{mm}}$
versus the scale height H = kTeff/($\mathrm{\mu m_{p}}$g), where k is the
Boltzmann constant, $\mathrm{\mu}$ is the mean molecular weight (here
$\mu=0.5$), and $\mathrm{m_{p}}$ is the proton mass. We find that
$\alpha_{\mathrm{mm}}$ increases with decreasing H.

Figure 4: $\alpha_{\mathrm{mm}}$ versus (v sin$i$)2/(Rg) and H. Markers are
the same as in Fig.2

Meanwhile, since $i$ values are unknown for the F dwarfs in the sample, the
trends in the period and (v sin$i$)2/(Rg), will be affected by the projection
effect (sin $i$). Also we need multi-frequency data for the F dwarfs in the
sample to make a robust $\alpha_{\mathrm{mm}}$ estimate. A large multi-
frequency sample of A- to M-type stars, with known $i$, spread across wide
ranges in age, Teff, period, and g is needed to draw statistically significant
conclusions. The A stars in such a sample will help explore the physics of
emergence of hot chromospheres (Simon et al., 2002) traced by
$\alpha_{\mathrm{mm}}$ becoming positive.

###  4.2 Towards a robust mm-band activity indicator

A robust activity indicator should not only provide a reliable quantitative
scale, but also map the activity levels to physical parameters robustly, in a
way that it facilitates clear segregation of stars in the planes of activity
versus different physical parameters. We considered two pairs of stars with
quite similar Teff and spectral types, and a good estimation of
$\alpha_{\mathrm{mm}}$ errors: $\alpha$ Cen B and $\epsilon$ Eri (Pair 1: K
type, Teff $\sim$5000 K), and $\alpha$ Cen A and the Sun (Pair 2: G type, Teff
$\sim$ 5800 K). The stars in each pair differ among each other in physical
parameters other than Teff. Each pair of stars is well separated along the
$\alpha_{\mathrm{mm}}$ axis, despite estimation errors. In the case of Pair 1,
$\epsilon$ Eri belonging to the more active ‘C’ branch has relatively high TB
values for similar frequencies and higher $\alpha_{\mathrm{mm}}$ than $\alpha$
Cen B. Meanwhile, for Pair 2 comprised of old stars, in planes formed by their
differentiating physical parameters (g, rotation period, B-V) with
$\alpha_{\mathrm{mm}}$ (Fig. 2b – d), the stars align well along the best-fit
power-law trends. This property of alignment along best-fit trends is even
seen for the binary pair $\gamma$ Vir A and B which have similar physical
properties, except for g. $\gamma$ Vir A with a slightly higher g has a
slightly higher $\alpha_{\mathrm{mm}}$ than its companion. A physical
characterisation of cool stars to this accuracy is difficult with common
indicators due to the large value intervals (Fig. 3), and especially for stars
within the narrow Teff range of 5000 - 7000 K explored here.

As mentioned earlier, the error intervals on $\alpha_{\mathrm{mm}}$ capture
the effect of stellar variability within activity cycle periods for the Sun,
$\epsilon$ Eri, and the $\alpha$ Cen binary. It is found that the error
interval is the highest for the most active young star, $\epsilon$ Eri,
possibly hinting at a higher variability in its activity level during the
stellar activity cycle. This agrees well with the current consensus that young
‘C’ branch stars usually show higher variability in activity indicators than
the older ‘I’ branch during their activity cycles (e.g. Pace, 2013). So, while
the $\alpha_{\mathrm{mm}}$ estimates provide a quantitative physical
characterisation of the mean stellar activity, the error intervals reflect the
variability across activity cycles. We need more multi-epoch data for stars in
different activity branches to explore mm-T${}_{\mathrm{B}}(\nu)$ and
$\alpha_{\mathrm{mm}}$ variability.

##  5 Conclusions and outlook

It is difficult to derive robust quantitative correlations between stellar
activity and physical parameters using the common activity indicators. This
study explores the possibility of defining a robust activity indicator based
on continuum radiation at mm wavelengths using its unique tomographic
potential to trace the thermal emission from the various stellar chromospheric
layers as a function of observing frequency.

The mm stellar sample compiled by Mohan et al. (2021) was used for the study.
The compiled T${}_{\mathrm{B}}(\nu)$ data come from observations done over
years, providing chromospheric thermal profiles averaged over activity cycle
timescales. The cool stars showed a spectral steepening towards lower
frequencies, as expected from an atmosphere that gets progressively hotter
towards higher heights (explored by lower frequencies). We derived the
mm-T${}_{\mathrm{B}}(\nu)$ spectral index, $\alpha_{\mathrm{mm}}$, for the
cool stars, which is expected to characterise the chromospheric thermal
gradients and in turn the efficiency of atmospheric heating and activity.
Robust power-law functions could be derived for $\alpha_{\mathrm{mm}}$ with
different activity influencing physical parameters, namely Teff, g, and the
rotation period. In addition, $\alpha_{\mathrm{mm}}$ shows an inverse
correlation with the scale height and the ratio of surface rotational kinetic
to potential energy; however, this finding should be interpreted with caution
in view of the small sample size. Due to insufficient data, this analysis
could only be done for the cool stars older than 1 Gyr. However, the young
cool star, $\epsilon$ Eri, is well apart from the trends of the old group.

The estimation errors on $\alpha_{\mathrm{mm}}$ are quite small in comparison
to those in common activity indicators, which facilitated the robust activity
quantification and mapping to physical parameters. Even the stars with similar
values for some physical parameters, such as the rotation period and Teff ,
among others, could be well separated in the $\alpha_{\mathrm{mm}}$ axes.
Also, with more data for A-type stars, the emergence of chromospheres can be
explored by tracking the change of sign in $\alpha_{\mathrm{mm}}$ as a
function of physical parameters.

To conclude, $\alpha_{\mathrm{mm}}$ could be developed into a robust activity
indicator giving us vital insights into the links between stellar activity,
atmospheric structure, and physical properties, especially for cool stars. To
assert this, we need more observations and preferably a campaign to monitor
nearby cool stars in the mm band at multiple epochs sufficient to sample their
variability within respective activity cycles.

###### Acknowledgements.

This work is supported by the Research Council of Norway through the EMISSA
project (project number 286853) and the Centres of Excellence scheme, project
number 262622 (“Rosseland Centre for Solar Physics”). We acknowledge helpful
discussions among the ISSI international team 387 “A New View of the Solar-
stellar Connection with ALMA”, which was funded by the International Space
Science Institute (ISSI, Bern, Switzerland). This paper makes use of the
following ALMA data: ADS/JAO.ALMA#2013.1.00170.S, #2016.1.00441.S,
#2013.1.00645.S. ALMA is a partnership of ESO (representing its member
states), NSF (USA) and NINS (Japan), together with NRC(Canada), MOST and ASIAA
(Taiwan), and KASI (Republic of Korea), in co-operation with the Republic of
Chile. The Joint ALMA Observatory is operated by ESO, AUI/NRAO and NAOJ. We
thank the referee, Jeffrey L. Linsky, for his valuable comments and
suggestions.

## References

  * Barnes (2003) Barnes, S. A. 2003, ApJ, 586, 464
  * Bashi et al. (2020) Bashi, D. et al. 2020, A&A, 643, A106
  * Davenport (2016) Davenport, J. R. A. 2016, ApJ, 829, 23
  * De Pascale et al. (2014) De Pascale, M. et al. 2014, A&A, 570, A68
  * Donati & Landstreet (2009) Donati, J. F. & Landstreet, J. D. 2009, ARA&A, 47, 333
  * Gaia Collaboration (2021) Gaia Collaboration. 2021, A&A, 649, A1
  * Grießmeier et al. (2005) Grießmeier, J. M., Stadelmann, A., Motschmann, U., et al. 2005, Astrobiology, 5, 587
  * Hauschildt & Baron (1999) Hauschildt, P. H. & Baron, E. 1999, J. Comput. Appl. Math., 109, 41
  * Lammer et al. (2007) Lammer, H., Lichtenegger, H. I. M., Kulikov, Y. N., et al. 2007, Astrobiology, 7, 185
  * Linsky (2017) Linsky, J. L. 2017, Annual Review of Astronomy and Astrophysics, 55, 159
  * Loukitcheva et al. (2004) Loukitcheva, M., Solanki, S. K., Carlsson, M., & Stein, R. F. 2004, A&A, 419, 747
  * Marsden et al. (2014) Marsden, S. C., Petit, P., Jeffers, S. V., et al. 2014, MNRAS, 444, 3517
  * Micela & Marino (2003) Micela, G. & Marino, A. 2003, A&A, 404, 637
  * Mohan et al. (2021) Mohan, A., Wedemeyer, S., Pandit, S., Saberi, M., & Hauschildt, P. H. 2021, A&A, 655, A113
  * Moschou et al. (2019) Moschou, S.-P., Drake, J. J., Cohen, O., et al. 2019, ApJ, 877, 105
  * Noyes et al. (1984) Noyes, R. W., Hartmann, L. W., Baliunas, S. L., Duncan, D. K., & Vaughan, A. H. 1984, ApJ, 279, 763
  * Odert et al. (2020) Odert, P., Leitzinger, M., Guenther, E. W., & Heinzel, P. 2020, MNRAS, 494, 3766
  * Pace (2013) Pace, G. 2013, A&A, 551, L8
  * Perryman et al. (1997) Perryman, M. A. C., Lindegren, L., Kovalevsky, J., et al. 1997, A&A, 323, L49
  * Simon et al. (2002) Simon, T., Ayres, T. R., Redfield, S., & Linsky, J. L. 2002, ApJ, 579, 800
  * Skumanich (1972) Skumanich, A. 1972, ApJ, 171, 565
  * Stepien (1994a) Stepien, K. 1994a, A&A, 292, 191
  * Stepien (1994b) Stepien, K. 1994b, A&A, 292, 191
  * Suresh et al. (2020) Suresh, A., Chatterjee, S., Cordes, J. M., Bastian, T. S., & Hallinan, G. 2020, ApJ, 904, 138
  * Tapia-Vázquez & De la Luz (2020) Tapia-Vázquez, F. & De la Luz, V. 2020, ApJS, 246, 5
  * Trigilio et al. (2018) Trigilio, C., Umana, G., Cavallaro, F., et al. 2018, MNRAS, 481, 217
  * van Saders et al. (2016) van Saders, J. L., Ceillier, T., Metcalfe, T. S., et al. 2016, Nature, 529, 181
  * Vernazza et al. (1981) Vernazza, J. E., Avrett, E. H., & Loeser, R. 1981, ApJS, 45, 635
  * Vidotto et al. (2014) Vidotto, A. A., Gregory, S. G., Jardine, M., et al. 2014, MNRAS, 441, 2361
  * Vidotto et al. (2013) Vidotto, A. A. et al. 2013, A&A, 557, A67
  * Wedemeyer et al. (2016) Wedemeyer, S., Bastian, T., Brajša, R., et al. 2016, Space Sci. Rev., 200, 1
  * White et al. (2020) White, J. A., Tapia-Vázquez, F., Hughes, A. G., et al. 2020, ApJ, 894, 76
  * White (2004) White, S. M. 2004, New A Rev., 48, 1319
  * White et al. (2017) White, S. M., Iwai, K., Phillips, N. M., et al. 2017, Sol. Phys., 292, 88
  * Zendejas et al. (2010) Zendejas, J., Segura, A., & Raga, A. C. 2010, Icarus, 210, 539


# Kernel-Whitening: Overcome Dataset Bias with Isotropic Sentence Embedding

Songyang Gao1, Shihan Dou1, Qi Zhang12 , Xuanjing Huang12
1 School of Computer Science, Fudan University, Shanghai, China
2 Shanghai Key Laboratory of Intelligent Information Processing, Shanghai,
China
{gaosy21<EMAIL_ADDRESS>Corresponding author.

###### Abstract

Dataset bias has attracted increasing attention recently for its detrimental
effect on the generalization ability of fine-tuned models. The current
mainstream solution is designing an additional shallow model to pre-identify
biased instances. However, such two-stage methods scale up the computational
complexity of training process and obstruct valid feature information while
mitigating bias. To address this issue, we utilize the representation
normalization method which aims at disentangling the correlations between
features of encoded sentences. We find it also promising in eliminating the
bias problem by providing isotropic data distribution. We further propose
Kernel-Whitening, a $\rm Nystr\ddot{o}m$ kernel approximation method to
achieve more thorough debiasing on nonlinear spurious correlations. Our
framework is end-to-end with similar time consumption to fine-tuning.
Experiments show that Kernel-Whitening significantly improves the performance
of BERT on out-of-distribution datasets while maintaining in-distribution
accuracy.

##  1 Introduction

Despite remarkable performance on NLP tasks, pre-trained language models, like
BERT, suffer sharp performance degradation in out-of-distribution (OOD)
settings McCoy et al. (2019). The above defect roots in the excessive reliance
on spurious correlations, which is widely found in crowdsourcing-built
datasets Gururangan et al. (2018). These phenomena are donated as dataset bias
problem He et al. (2019). A line of works attempts to tackle this problem by
down-weighting bias training examples to discourage the main model from
adopting recognized biases, including example reweighting Schuster et al.
(2019), confidence regularization Utama et al. (2020a), or model ensembling
Clark et al. (2019).

However, aforementioned methods over-depend on researchers’ intuition and
task-specific insights to characterize spurious correlations, causing
unrecognized bias patterns to remain in individual dataset Sharma et al.
(2018). Such assumption that dataset biases are known as a prior has been
relaxed by limited capacity models Utama et al. (2020b) or early training Tu
et al. (2020) in recent works. These approaches still rely on extra shallow
models, which are not end-to-end, and weak-weighted bias samples
simultaneously obstruct learning from their non-bias parts Wen et al. (2021).

Figure 1: Illustration of Kernel-Whitening. The vertical and horizontal axes
represent the valid and invalid features, respectively. Uneven sample
distribution induces a bias decision boundary, resulting in errors on out-of-
distribution data. The normalization method maps the data to isotropic latent
space, where the new boundary is uncorrelated to redundant features, providing
better generalization.

Instead of designing an extra model as previous attempts did, in this work, we
propose a novel end-to-end framework, Kernel-Whitening, to significantly
improve OOD performance while maintaining a similar computational cost as
conventional BERT fine-tuning. The BERT-whitening Su et al. (2021) and BERT-
flow Li et al. (2020) methods are effective normalization techniques to obtain
better semantic representation. BERT-whitening calculates a linear
operator111A positive definite symmetric matrix. with SVD decomposition GOLUB
and REINSCH (1970) to transform the sentence representation to follow a
distribution with respect to the standard normal distribution. BERT-flow
introduces normalized flow Rezende and Mohamed (2015) to perform similar
transformations. Particularly, we find that the normalization method is also
promising in improving the generalization ability of fine-tuned models by
eliminating spurious correlations in training datasets. Despite the
significant improvement over the OOD datasets, the linear transformation of
BERT-whitening is not capable of dealing with nonlinear dependencies between
features. Meanwhile, flow-based methods require a complex inference process,
which scales up the training costs.

In an attempt to eliminate nonlinear correlations while maintaining low
training expenditure, we introduce kernel methods to naturally reconstruct a
set of sentence representations only with linear correlation Achlioptas et al.
(2001). However, traditional kernel methods focus only on data similarities
without providing explicit mapping operators, therefore, we use the $\rm
Nystr\ddot{o}m\ $ approximation Xu et al. (2015) to obtain low-rank kernel
estimations. In general, we transform training data to an isotropic Gaussian
distribution without affecting topological relationships between data points.

Kernel-Whitening222Our code is available at
https://github.com/SleepThroughDifficulties/KernelWhitening. achieves
competitive performance on generalization tasks. Experiments on eight datasets
demonstrate that our method can improve the accuracy by 7%-11% on OOD
datasets. In addition, the analysis of sentence representation proves that our
method effectively removes the spurious correlations between dimensional
features, which are known to be the direct cause of the dataset bias problem.
Overall, our main contributions are as follows:

  * •

We propose a novel framework, Kernel-Whitening, which ameliorates the bias
problem by transforming sentence representation into isotropic distribution
with similar time to fine-tuning.

  * •

We introduce a kernel estimation algorithm, $i.e.$, $\rm Nystr\ddot{o}m$
approximation, to alleviate normalization methods from the trade between
complex arithmetic and disentangle effects.

  * •

We conduct comprehensive experiments on debiasing tasks to verify the
effectiveness of normalization methods for overcoming spurious correlations.

##  2 How Do Normalization Methods Provide Better Generalization

In this section, we discuss the negative impacts of dataset bias on the
model’s generalization ability, and subsequently, how normalization methods
lead to better performance in OOD settings.

###  2.1 Illustrate Dataset Bias from Feature Perspective

We first interpret dataset bias as triggered by the imbalance distribution of
training data in feature space. Figure 2 shows an empirical analysis on MNLI
dataset Williams et al. (2018). Both train and test sets exhibit a strong
positive correlation between the word overlap ratio and the class portion of
the entailment label, which enable models to achieve high accuracy scores
without modelling semantic information. McCoy et al. (2019) proposed HANS, a
label-fair test set ($i.e.$, labels are proportionally consistent with
different degrees of word overlap) to investigate the generalizability of
models fine-tuned with above-mentioned bias, the model provides over 20%
accuracy degradation when the specific literal heuristic could not be utilized
for prediction.

Formally, given input data $(X,Y)$ and bias dataset $D$, the training process
can be formulated as follows:

$\displaystyle P(Y|X^{L},X^{P})$
$\displaystyle=\frac{P(Y|X^{L})P(X^{P}|X^{L},Y)}{P(X^{P})}$
$\displaystyle=P(Y|X^{L}),$ (1)

where $X^{L}$ represents the task-related features, and $X^{P}$ represents
other irrelevant features. That is, the model learns $X^{L}$ from the label
pair $(X,Y)$. By modeling the conditional distribution of labels relative to
the input, the model extracts valid features for specific tasks.

Figure 2:  Lable probability with increasing lexical overlap for entailment
label on MNLI dataset.

However, previous works argue that construction process of bias dataset
introduces spurious correlations between $(X^{L}|D)$ and $(X^{P}|D)$
Gururangan et al. (2018):

$\displaystyle P(X^{P}|X^{L},D)$
$\displaystyle=\frac{P(X^{P},X^{L}|D)}{P(X^{L}|D)}$
$\displaystyle>P(X^{P}|D).$ (2)

Therefore, the actual training objective on dataset $D$ is the posterior
between feature distribution $(X^{L},X^{P}|D)$ and label $Y$, $i.e.$,
$P(Y|X^{L},X^{P};D)$. By Eq.2.1, irrelevant features increase the confidence
for specific labels:

$\displaystyle P(Y|X^{L},X^{P},D)$
$\displaystyle=\frac{P(Y|X^{L},D)P(X^{P}|X^{L},D)}{P(X^{P})}$
$\displaystyle>P(Y|X^{L},D)$ $\displaystyle=P(Y|X^{L}).$ (3)

Such overconfidence does not perturb the model effect on the test set, which
has a similar distribution to the training set. However, out-of-distribution
data follow the correct distribution $P(Y|X^{L})$, and are therefore
classified as the bias label ($e.g.$, "entailment" in MNLI dataset), even if
they have different relations.

###  2.2 Isotropic Representation Leads to Better Generalization

According to the definition described in Eq.2.1, dataset bias causes deep
networks to fit the dataset-specific distribution, which impairs the
generalization performance. Normalization methods intervene in the above
problem by reconstructing the feature space. When sentences are encoded by
pre-train model, the embedding representations are transformed into isotropic
distribution, $e.g.$, standard normal distribution. Suppose data $x$ and prior
$u$ satisfy:

$\displaystyle x=f_{\theta}(u),\ u\sim P_{\mathcal{U}}(u),$ (4)

where $\mathcal{U}$ represents the latent space of isotropic $u$, and $f$ is
an invertible function. The probabilistic density function of original data on
the transformed space can be calculated as follows:

$\displaystyle
P_{\mathcal{X}}(\mathbf{x})=\left|\operatorname{det}\frac{\partial
f_{\theta}^{-1}(\mathbf{x})}{\partial\mathbf{x}}\right|P_{\mathcal{U}}\left(f_{\theta}^{-1}(\mathbf{x})\right).$
(5)

By Eq.5, the distribution of training data is transformed into isotropic ones.
Subsequently, suppose $X^{L}_{u}$ and $X^{P}_{u}$ are the latent
representations of $X^{L}$ and $X^{P}$, the spurious correlations between
valid features and invalid features are eliminated:

$\displaystyle P_{\mathcal{U}}(Y|X^{P}_{u},X^{L}_{u},D)$
$\displaystyle=\frac{P_{\mathcal{U}}(Y|X^{L}_{u},D)P_{\mathcal{U}}(X^{L}_{u},X^{P}_{u}|D)}{P_{\mathcal{U}}(X^{P}_{u})P_{\mathcal{U}}(X^{L}_{u}|D)}$
$\displaystyle=P_{\mathcal{U}}(Y|X^{L}_{u}),$ (6)

With isotropic data distribution, a decision boundary independent of the
redundant features is obtained, which provides better generalization on the
OOD samples.

###  2.3 Minor Weakness for BERT-flow and BERT-whitening

The BERT-flow method learns a flow-based generative model to fit the transform
function $f_{\theta}$, and BERT-whitening method computes the inverse linear
operator with SVD decomposition of the covariance matrix. Despite the decent
effect of representing normalization methods, BERT-flow requires multiple
convolutional layers to find the appropriate transformation function, which
increases the difficulty and time consumption of the training process. When
fine-tuned on a tiny dataset, the flow layer encountered obstacles in
providing reasonable transform results. Moreover, the BERT-whitening method
focuses only on eliminating linear correlations between features, which is
ineffective in alleviating the nonlinear correlation problem. In an attempt to
ease the effort of training and provide faster but thorough transformation, we
propose a novel normalization framework by kernel approximation, which is
detail discussed in the next section.

##  3 Distribution Generalization with Kernel Approximation

In this section, we show our end-to-end framework named Kernel-Whitening. We
first introduce $\rm{Nystr\ddot{o}m}$ kernel estimation algorithm, and
subsequently how to apply such approximation method on debiasing tasks.

###  3.1  $\mathbf{Nystr\ddot{o}m}$ Kernel Estimation

We first elaborate kernel trick, which constructs new linearly differentiable
properties by mapping the original feature space onto a high-dimensional RKHS
Alvarez et al. (2012). Given a set of training data
$X=\left\\{x_{i}\in\mathbb{R}^{d},i=1,\ldots,n\right\\}$, the kernel method
maps $X$ onto a dot product space $\mathcal{H}$ using
$\phi:X\xrightarrow{}\mathcal{H}$. Generally, the dimension of $\mathcal{H}$
can be so large that the mapping function cannot be obtained explicitly.
Nevertheless, the dot product result can be represented by a positive definite
kernel $k$, $i.e.$, a function $k$ satisfies:

$\displaystyle k(x_{k},x_{j})$
$\displaystyle=\left<\phi(x_{k}),\phi(x_{j})\right>$
$\displaystyle=\sum_{i=1}^{N}\lambda_{i}\phi_{i}(x_{k})\phi_{i}(x_{j}),$ (7)

where $\lambda_{i}$ and $\phi_{i}$ denotes the eigenvalues and eigenfunctions
of kernel operator $k$, and $N$ denotes their number. With finite dataset
$\left\\{x_{i}\in\mathbb{R}^{d},i=1,\ldots,n\right\\}$, such decomposition can
be replaced with empirical estimation as follows:

$\displaystyle\frac{1}{n}\sum_{j=1}^{n}k\left(\bm{x_{k}},\bm{x}_{j}\right)\phi_{i}\left(\bm{x}_{j}\right)=\lambda_{i}\phi_{i}(\bm{x_{k}}).$
(8)

Eq.8 indicates a spectral decomposition of kernel matrix $G$, which satisfy
$G_{k,j}=k(x_{k},x_{j})$. Considering the SVD decomposition of $G$ as

$\displaystyle G=W\Sigma W^{T},$ (9)

where $W$ is an orthogonal matrix, and $\Sigma$ is a diagonal matrix with
positive diagonal elements. When matching Eq.8 with Eq.9, the mapping operator
$\phi$ is therefore denoted as:

$\displaystyle\phi(x_{j})=G_{j,\ :}W\Sigma^{\frac{1}{2}}.$ (10)

However, existing datasets often contain thousands to hundreds of thousands
samples, which makes it impossible to directly calculate the SVD
decomposition. Therefore, we introduce $\rm{Nystr\ddot{o}m}$ method Williams
and Seeger (2001) to provide a low-rank estimation of kernel matrix.

Suppose $S$ to be a sampled subset of $X$, the kernel matrix $G$ can be
represented as:

$\displaystyle G=\left[\begin{array}[]{cc}G_{s}\ G_{r}^{T}\\\ G_{r}\
G_{x}\end{array}\right],$ (13)

where $G_{s}$ denotes the gram matrix of subset $S$. The $W$ and $\Sigma$ can
subsequently approximated by:

$\displaystyle W\approx\left[\begin{array}[]{cc}G_{s}\
G_{r}\end{array}\right]^{T},\ \Sigma\approx G_{s}.$ (15)

The reconstructed $\rm{Nystr\ddot{o}m}$ representation of single example x is
as follows:

$\displaystyle\phi(\mathbf{x})=G_{\textbf{x},S}G_{s}^{-\frac{1}{2}},$ (16)

where $G_{\textbf{x},S}=[k(x_{1},\textbf{x}),...,k(x_{s},\textbf{x})]^{T}$ for
$x_{i}\in S$. By estimating high-dimensional representations of the training
samples we obtain a linearly divisible distribution, which can be normalized
with a linear transformation.

###  3.2 Batch Iterate for Global Approximation

In Section 3.1, we elaborate the standard $\rm{Nystr\ddot{o}m}$ approach to
processing input data. The difference is, the subset $S$ in the traditional
approach, $i.e.$, kernel SVM, usually contains hundreds of elements, while
deep networks are trained on smaller batches ($e.g.$, 32 for Kernel-Whitening)
with stochastic gradient descent (SGD) optimizer. The insufficient samples
compromise the information of reconstructed representations, making the
improvement inconspicuous when directly applying $\rm{Nystr\ddot{o}m}$ methods
to debiasing tasks.

In attempt to introduce global information while processing batch data, we
design preservation and reloading structures to extend the dimension of low-
rank kernel matrix. For each batch, we calculate $\rm{Nystr\ddot{o}m}$ matrix
with batch features $Z_{L}^{t}$ and global features $Z_{f}^{t}$, which
represent the principal components of data distribution in step $t$. Giving
input data $Z_{L}^{t}$ contains $L$ instances, the extended representation is
gived with Eq.16:

$\displaystyle\phi(Z_{L}^{t})=G_{0:L,:}G^{-\frac{1}{2}},$ (17)

where $G$ denotes the kernel matrix generated by
$\left[Z_{L}^{t},Z_{f}^{t}\right]^{T}$, and $G_{0:L,:}$ denotes the first $L$
rows of $G$. Especially, we select Radial Basis Function (RBF) kernel in
Kernel-Whitening method.

Noticing that the kernel estimation method only projects the data points into
a linearly separable space, we further normalize the distribution with a
linear transformation. The reconstructed representation $\phi(Z_{L}^{t})$ is
subsequently weighted under the supervised signal of Hilbert-Schmidt
independence criterion (HSIC) Wang et al. (2021), which is an adequate
indicator for estimating the mutual independence between features. The optimal
weight $W^{*}$ is calculated by:

$\displaystyle
W^{*}=\underset{W\in\mathcal{R}^{L}}{\arg\min}\left\|\tilde{\Sigma}_{G,W}\right\|_{F}^{2},$
(18)

where $W=\\{W_{i}\\}^{n}_{i=1}$ denotes the sample weight vector, and
$\tilde{\Sigma}_{G,W}$ represents the empirical estimation for covariance
between features:

$\displaystyle\tilde{\Sigma}_{G,W}$
$\displaystyle=\frac{1}{L-1}\sum_{1<i<j<L}F_{i}^{T}F_{j}$ (19) $\displaystyle
F_{i}$ $\displaystyle=WG_{:,i}-\frac{1}{n}\sum_{j=1}^{n}W_{j}G_{j,:}.$ (20)

At the end of each iteration, we update the global features with local
information $Z_{L}^{t}$ to catch reasonable basis vectors:

$\displaystyle Z_{f}^{t+1}=\alpha_{i}Z_{f}^{t}+(1-\alpha_{i})Z_{L}^{t},$ (21)

where $\alpha_{i}$ denotes the attenuated factor controlling the importance of
local information.

###  3.3 Training Objective

In Section 3.2, we show how to obtain the reconstructed feature representation
$\phi(Z_{L}^{t})$ and weighting parameters $W^{*}$. Subsequently, we use above
results to train on original BERT models. The final train loss of Kernel-
Whitening is denoted as:

$\displaystyle Loss=\sum_{i=1}^{L}W^{*}_{i}f(\phi(Z_{L}^{t})_{i},y_{i}),$ (22)

where $f(\cdot,\cdot)$ represents the cross-entropy loss with input
$\phi(Z_{L}^{t})_{i}$ and it’s corresponding label $y_{i}$. Our detailed
algorithm implementation is shown in Algorithm 1.

Algorithm 1 Framework of Kernel-Whitening for our system.

0: The set of pooler output for current batch, $Z=\\{Z_{i}\\}^{n}_{i=1}$; The
set of global features, $Z_{f}^{t}$; The classifier for specific task,
$f(\cdot,\cdot)$.

1: Compute kernel matrix $G$ by $[Z,Z_{f}^{t}]^{T}$

2: Compute $W,\Sigma$ by SVD(G)

3: Compute representation $\phi(Z)$ by $G,W,\Sigma$ with Eq.10, 17

4: Compute weight $W^{*}$ by HSIC(G,W) with Eq.18

5: Update global features $Z_{f}^{t}$ with Z with Eq.21

6: Compute training loss $L$ by classifier $f(\cdot,\cdot)$, $W^{*}$ and
$\phi(Z)$ with Eq.22

6: Updated global features, $Z_{f}^{t+1}$; Training loss, $L$

##  4 Experiments

In this section, we provide a comprehensive analysis of Kernel-Whitening and
the other two normalization methods ($i.e.$, BERT-flow Li et al. (2020) 333The
code of BERT-flow is available at https://github.com/bohanli/BERT-flow. and
BERT-whitening Su et al. (2021)) through extensive experiments on three tasks.

Model | MNLI | FEVER | QQP
---|---|---|---
ID | HANS | ID | Symm. v1 | Symm. v2 | ID | PAWS dupl |  $\neg$ dupl
BERT-base | 84.3 | 61.1 | 85.4 | 55.2 | 63.1 | 91 | 96.9 | 9.8
Reweighting | 83.5 | 69.2 | 84.6 | 61.7 | 66.5 | 85.5 | 49.7 | 51.2
Product-of-Experts | 84.1 | 66.3 | 82.3 | 62.0 | 65.9 | 88.8 | 50.3 | 61.2
$\rm{{\mbox{PoE}}_{\mbox{{cross-entropy}}}}$ | 83.6 | 67.3 | 85.7 | 57.7 | 61.4 | - | - | -
$\rm{{\mbox{PoE}}_{\mbox{{self-debias}}}}$ | 80.7 | 68.5 | 85.4 | 59.7 | 65.3 | 77.4 | 44.1 | 69.4
Learned-Mixin | 84.2 | 64.0 | 83.3 | 60.4 | 64.9 | 86.6 | 69.7 | 51.7
Conf-reg | 84.3 | 69.1 | 86.4 | 60.5 | 66.2 | 89.1 | 91.0 | 19.8
$\rm{{\mbox{Conf-reg}}_{\mbox{{self-debias}}}}$ | 84.3 | 67.1 | 87.6 | 59.8 | 66.0 | 85.0 | 48.8 | 28.7
MoCaD | 82.3 | 70.7 | 87.1 | 65.9 | 69.1 | - | - | -
BERT-flow* | 82.5 | 69.1 | 85.7 | 64.4 | 67.2 | - | - | -
BERT-whitening-64* | 84.3 | 69.9 | 86.7 | 65.7 | 69.5 | 87.5 | 82.5 | 34.7
BERT-whitening-384* | 83.7 | 70.4 | 85.9 | 65.9 | 68.3 | 87.3 | 75.3 | 41.3
Kernel-Whitening-64* | 84.4 | 70.1 | 87.4 | 65.1 | 69.8 | 90.8 | 91.2 | 32.4
Kernel-Whitening-384* | 83.9 | 70.9 | 87.8 | 66.2 | 70.3 | 87.6 | 72.7 | 43.2
Table 1: Models evaluation on MNLI, FEVER, QQP, and their respective challenge
test sets. The performance of the three normalized models is shown in cyan,
the model name with asterisks represents the experimental results on our
machine. The best results on each dataset are bolded.

###  4.1 Baseline Methods

Our method is compared with previous works as follows:

  * •

Clark et al. (2019) (Reweighting and Learned-Mixin), which predicts confidence
for each sample and down-weights problematic data.

  * •

Sanh et al. (2020) (Product-of-Experts and $\rm{{\mbox{PoE}}_{\mbox{{cross-
entropy}}}}$), which trains limited capacity models as experts to debias
without explicitly identifying dataset bias.

  * •

Utama et al. (2020b) ($\rm{{\mbox{PoE}}_{\mbox{{self-debias}}}}$ and
$\rm{{\mbox{Conf-reg}}_{\mbox{{self-debias}}}}$), which uses a shallow model
to identify biased samples and focus the main model on them.

  * •

Utama et al. (2020a) (Conf-reg), which uses confidence regularization to
discourage models from exploiting biases.

  * •

Xiong et al. (2021) (MoCaD), which produces uncertainty estimations to achieve
a three-stage ensemble-based debiasing framework.

Model |  Requires Prior Knowledge | Extra Model
---|---|---
Reweighting | ✔ | ✔
Product-of-Experts | ✔ | ✔
$\rm{{\mbox{PoE}}_{\mbox{{cross-entropy}}}}$ | ✔ | ✔
$\rm{{\mbox{PoE}}_{\mbox{{self-debias}}}}$ |  | ✔
Learned-Mixin | ✔ | ✔
conf-reg | ✔ | ✔
$\rm{{\mbox{Conf-reg}}_{\mbox{{self-debias}}}}$ |  | ✔
MoCaD | ✔ | ✔
Kernel-Whitening |  |
Table 2:  Details of the state-of-the-art debiasing methods used to compare
with Kernel-Whitening. Our model is end-to-end while not requiring prior
knowledge of biases or additional shallow models.

###  4.2 Datasets and Metrics

We conduct experiments on three tasks: natural language inference, fact
verification, and paraphrase identification. Each task contains in-
distribution and out-of-distribution datasets.

For the NLI task, we conduct experiments on the Multi-Genre Natural Language
Inference (MNLI) dataset Nangia et al. (2017) and HANS McCoy et al. (2019). We
train the model on MNLI, and choose MNLI-mismatch and HANS as the ID and OOD
test set.

For the fact verification task, we use FEVER444
https://github.com/TalSchuster/FeverSymmetric as the training dataset provided
by Thorne et al. (2018). We train the model on FEVER, and evaluate the model
performance on ID test set and FEVER Symmetric Schuster et al. (2019) (version
1 and 2) as our OOD test set.

For the paraphrase identification task, we perform the evaluation using Quora
Question Pairs (QQP) as ID dataset and PAWS Zhang et al. (2019) as OOD dataset
which consists of two types of data including _duplicate_ and _non-duplicate_.

### Evaluation Metrics

Following previous works, we measured the accuracy score on the in-
distribution and out-of-distribution test sets to compare the results of
different models.

###  4.3 Implementation Details

Following previous debiasing methods, we apply our debiasing method on the
BERT-base Devlin et al. (2019). The hyperparameters of BERT are consistent
with previous research papers. The learning rate is 2e-5 for MNLI dataset and
1e-5 for FEVER and QQP, the batch size is 32 and the optimizer is AdamW with a
weight decay of 0.01. Note that previous methods Sanh et al. (2020); Xiong et
al. (2021) have shown high variance in experiment results under different
settings, we evaluate the performance of our model by four random seeds and
report the averaged result. We use the [CLS] vector as sentence embedding for
all three methods. The model is trained in an NVIDIA GeForce RTX 3090 GPU. All
models are trained five epochs, and checkpoints with top-2 performance are
finally evaluated on the challenge test set.

###  4.4 Experimental Results

The extensive results of all the above-mentioned methods are summarized in
table 1. Compared with other baseline methods, Kernel-Whitening significantly
improves the model performance on challenge sets, and achieves state-of-the-
art results on seven of the eight benchmarks. On the MNLI and FEVER datasets,
our framework achieves the best performance with about 10 percentage points
higher than the accuracy of BERT-base, which outperforms other debiasing
methods. This proves that our framework has the best results and
generalizability among these methods.

Moreover, our approach can effectively eliminate the dataset bias while
mitigating the damage to generalizable features. The vast majority of
debiasing methods improve the performance of out-of-distribution datasets by
sacrificing the performance of in-distribution datasets, which means that
current debiasing methods attempt to achieve a trade-off between ID
performance and OOD performance. However, our approach achieves the best
performance on OOD datasets for natural language inference and fact
verification tasks with better results on ID datasets. For QQP dataset, our
proposed approach also achieves decent generalization in PAWS without
excessive performance degradation on ID datasets.

In general, the normalization methods perform well on both in-distribution and
out-of-distribution datasets for all tasks. All five models of three methods
are end-to-end approaches and do not rely on any prior knowledge of the
dataset. That is to say, they achieve better utility and scalability while
providing more effective debiasing. For BERT-whitening and Kernel-Whitening, a
larger hidden dimension indicates better performance on OOD datasets, and
Kernel-whitening performs better when parameters are constant to BERT-
whitening, which strongly supports our analysis of normalization methods.
BERT-flow shows an acceptable performance on OOD datasets, but is inferior to
the whitening-based approach. We argue that flow model requires more samples
as reference, and the original hyperparameters are not capable for the
additional network layers.

##  5 Analysis and Discussion

In this section, we construct supplementary experiments to further analyze the
effectiveness of normalization methods, especially our Kernel-Whitening
framework.

###  5.1 Effect of Latent Dimension $L$

The dimensionality of reconstructed features is a key feature. The reduction
of vector size brings smaller memory occupation and faster inference
downstream layers, while the missing information may impair the ability of the
model. To further illustrate the effect of low-rank kernel approximation, we
conduct a sensitivity analysis of latent dimension $L$. Figure 3 shows the
variation curve of performance change for two whitening-based methods. For
both in-distribution and out-of-distribution tasks. A latent dimension of
double the batch size provides promising performance. As the dimensionality
rises, Kernel-Whitening maintains a stable debiasing effect, while BERT-
whitening fluctuates on FEVER and Symm. v2 dataset. We argue that this
phenomenon is due to that high-dimensional features are more prone to
nonlinear correlations, where Kernel-Whitening is designed to show better
results. Moreover, Kernel-Whitening always performs better when the
dimensionality is greater than 300, which illustrates the stability and
generality of our method.

Figure 3: Effect of different dimensionality $L$ with whitening methods on
each aforementioned tasks. The $x$ axis is the latent dimension of sentence
embeddings. The two images are model performance on out-of-distribution and
in-distribution test sets, respectively.

###  5.2 Independence Study

In Section 2 we analyse how isotropic data distribution leads to better
generalization. To check whether normalization methods remove the dependencies
between features, we conduct experiments on the covariance between features
during training process. As shown in Figure 4, All three normalization methods
exhibit a suppression effect on feature correlation, while our method achieves
the optimal performance at the end of training. As the iterations increase,
the covariance first decreases rapidly and converges to a low point. All
method’s performance fluctuations around certain steps, we believe such
fluctuations are related to biased samples in the data.

Overall, Kernel-Whitening largely remits dependencies between features, and
such independence effectively contributes to the generalization ability of
deep network models.

###  5.3 Time Consumption

Besides outstanding debiasing performance, we compute the time consumption
with baseline methods to further demonstrate the strength of Kernel-Whitening.
We train each model equally on an NVIDIA RTX 2080Ti GPU with the same batch
size. We compare three normalization methods with the best baseline work,
MoCaD Xiong et al. (2021), which trains a bias model to produce model
calibrating. To give a horizontal comparison between different datasets, we
set the time consumption of fine-tune 100 as a baseline. As shown in Table 3,
the time consumption of Kernel-Whitening is nearly the same as fine-tuning and
costs 6 times less extra time than MoCad. Although BERT-whitening only uses a
linear transformation to obtain reconstruction representations, our method is
still faster. Because our method performs SVD decomposition on a matrix of
$L*L$ while BERT-whitening handles the same operations on a matrix of $L*N$,
where $L$ is the hidden dimension and $N$ is the output dimension of BERT
($e.g.$, 768).

Figure 4:  The independence study on covariance between reconstructed features. The hidden dimension of BERT-whitening and Kernel-Whitening are set to 64. All models are trained 50000 steps with batch 32. Method | MNLI | FEVER
---|---|---
BERT-base | 100 | 100
MoCad | 264 | 227
BERT-flow | 198 | 190
BERT-whitening | 146 | 139
Kernel-Whitening | 138 | 134
Table 3: Time consumption (percentages) of training one epoch on the whole
dataset. Whitening-based methods cost much less time than previous works.

##  6 Related Work

###  6.1 Dataset Bias

Recent observations McCoy et al. (2019); Naik et al. (2018) show that, natural
language understanding models tend to over-rely on specific shallow
heuristics, resulting in inadequate generalization capability in out-of-
distribution (OOD) settings Schuster et al. (2019). Sinha et al. (2021); Pham
et al. (2021) have reported the insensitivity to word-order permutations among
transformer-based models. The original and out-of-order examples elicit the
same classification label When permuted randomly, which contradicts the
conventional understanding of semantics. Such phenomena are studied as dataset
bias problems.

Existing methods train additional models to identify biased training data
Clark et al. (2019); Utama et al. (2020a); Schuster et al. (2019) or use the
above bias model to calibrate the classification results of test data Utama et
al. (2020b); Sanh et al. (2020); Xiong et al. (2021). The so-called bias model
refers to classifiers who use only a portion of input data for prediction,
$e.g.$, hypothesis-only model in NLI task which only predict from specific
linguistic phenomena in hypothesis sentences such as negation. These methods
are not end-to-end and face difficulty in fully identifying all bias patterns.

Recently, another line of works notice the connection between dataset bias and
feature distribution, and try to tackle the dataset bias problem by
identifying features with better generalizability. Dou et al. (2022) use an
loss function based on information bottleneck (IB) to focus the model on task-
relevant features, and Wu and Gui (2022) similarly achieve such feature
filtering by mapping sentence embedding into a specific low-dimension
subspace.

###  6.2 Unsupervised Semantic of Sentence Embedding

Previous works suggest that the word representations of pre-train language
model are not isotropic Gao et al. (2018); Ethayarajh (2019), leading model to
poorly capture the underlying semantic of sentences Li et al. (2020). Such
anisotropic causes the difficulty of using sentence embedding directly through
simple similarity metrics. Gao et al. (2018) propose word embedding matrix
regularization methods to mitigate the degeneration problem. Recently,
researchers attempt to transform BERT sentence embedding into an isotropic
Gaussian distribution through normalizing flow Li et al. (2020) or whitening
methods Su et al. (2021). As supervised learning also suffers from uneven data
distribution of train sets, we are the first to normalize the data
distribution on supervised training to eliminate dataset bias problem.

##  7 Conclusion

In this work, we propose a novel framework, Kernel-Whitening, to tackle the
spurious correlation from a feature perspective. We analyze how to introduce
isotropic sentence embedding for eliminating dataset bias and propose a
promising and computationally kernel estimation, to obtain an approximation of
disentangled sentence embedding. Experiments on various datasets demonstrate
that Kernel-Whitening achieves better performance on both ID and OOD datasets
than comparative works. This implies that a shallow model, or prior knowledge
of dataset bias, is not must for the improvement of generalization.

##  8 Limitations

In this section, we discuss the potential limitations of our work. The
analysis of model effects in this paper is focusing on commonly used
benchmarks for natural language understanding debiasing works, and they may
carry confounding factors that affect the performance of our model. Therefore,
it is worth further exploring the performance of our model on more tasks,
$e.g.$, the WikiGenderBias dataset for gender bias on relation extraction
task. In addition, this presented work is inspired by unsupervised semantic
learning methods, such as BERT-whitening, and it will be better to test the
performance of our approach on unsupervised tasks. We leave these two problems
to further work.

## Acknowledgements

The authors wish to thank the anonymous reviewers for their helpful comments.
This work was partially funded by National Natural Science Foundation of China
(No. 62076069, 61976056, 61906176), Program of Shanghai Academic Research
Leader (No. 22XD1401100), and Beijing Academy of Artificial Intelligence
(BAAI).

## References

  * Achlioptas et al. (2001) Dimitris Achlioptas, Frank McSherry, and Bernhard Schölkopf. 2001.  Sampling techniques for kernel methods.  _Advances in neural information processing systems_ , 14.
  * Alvarez et al. (2012) Mauricio A Alvarez, Lorenzo Rosasco, Neil D Lawrence, et al. 2012.  Kernels for vector-valued functions: A review.  _Foundations and Trends® in Machine Learning_ , 4(3):195–266.
  * Clark et al. (2019) Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019.  Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4069–4082.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dou et al. (2022) Shihan Dou, Rui Zheng, Ting Wu, Songyang Gao, Qi Zhang, Yueming Wu, and Xuanjing Huang. 2022.  Decorrelate irrelevant, purify relevant: Overcome textual spurious correlations from a feature perspective.  _arXiv preprint arXiv:2202.08048_.
  * Ethayarajh (2019) Kawin Ethayarajh. 2019.  How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 55–65.
  * Gao et al. (2018) Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2018.  Representation degeneration problem in training natural language generation models.  In _International Conference on Learning Representations_.
  * GOLUB and REINSCH (1970) GH GOLUB and C REINSCH. 1970.  Handbook series linear algebra. singular value decomposition and least squares solutions.  _Numerische Mathematik_ , 14:403–420.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112.
  * He et al. (2019) He He, Sheng Zha, and Haohan Wang. 2019.  Unlearn dataset bias in natural language inference by fitting the residual.  _EMNLP-IJCNLP 2019_ , page 132.
  * Li et al. (2020) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020.  On the sentence embeddings from pre-trained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9119–9130.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448.
  * Naik et al. (2018) Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018.  Stress test evaluation for natural language inference.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2340–2353.
  * Nangia et al. (2017) Nikita Nangia, Adina Williams, Angeliki Lazaridou, and Samuel Bowman. 2017.  The repeval 2017 shared task: Multi-genre natural language inference with sentence representations.  In _Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP_ , pages 1–10.
  * Pham et al. (2021) Thang Pham, Trung Bui, Long Mai, and Anh Nguyen. 2021.  Out of order: How important is the sequential order of words in a sentence in natural language understanding tasks?  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1145–1160.
  * Rezende and Mohamed (2015) Danilo Rezende and Shakir Mohamed. 2015.  Variational inference with normalizing flows.  In _International conference on machine learning_ , pages 1530–1538. PMLR.
  * Sanh et al. (2020) Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M Rush. 2020.  Learning from others’ mistakes: Avoiding dataset biases without modeling them.  In _International Conference on Learning Representations_.
  * Schuster et al. (2019) Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019.  Towards debiasing fact verification models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3419–3425.
  * Sharma et al. (2018) Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. 2018.  Tackling the story ending biases in the story cloze test.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 752–757.
  * Sinha et al. (2021) Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams. 2021.  Unnatural language inference.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7329–7346.
  * Su et al. (2021) Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021.  Whitening sentence representations for better semantics and faster retrieval.  _arXiv preprint arXiv:2103.15316_.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  Fever: a large-scale dataset for fact extraction and verification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819.
  * Tu et al. (2020) Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020.  An empirical study on robustness to spurious correlations using pre-trained language models.  _Transactions of the Association for Computational Linguistics_ , 8:621–633.
  * Utama et al. (2020a) Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020a.  Mind the trade-off: Debiasing nlu models without degrading the in-distribution performance.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8717–8729.
  * Utama et al. (2020b) Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020b.  Towards debiasing nlu models from unknown biases.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7597–7610.
  * Wang et al. (2021) Tinghua Wang, Xiaolu Dai, and Yuze Liu. 2021.  Learning with hilbert–schmidt independence criterion: A review and new perspectives.  _Knowledge-Based Systems_ , 234:107567.
  * Wen et al. (2021) Zhiquan Wen, Guanghui Xu, Mingkui Tan, Qingyao Wu, and Qi Wu. 2021.  Debiased visual question answering from feature and sample perspectives.  _Advances in Neural Information Processing Systems_ , 34.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122.
  * Williams and Seeger (2001) Chris Williams and Matthias Seeger. 2001.  Using the nystroem method to speed up kernel machines.  _Advances in Neural Information Processing Systems 13_.
  * Wu and Gui (2022) Ting Wu and Tao Gui. 2022.  Less is better: Recovering intended-feature subspace to robustify nlu models.  _arXiv preprint arXiv:2209.07879_.
  * Xiong et al. (2021) Ruibin Xiong, Yimeng Chen, Liang Pang, Xueqi Cheng, Zhi-Ming Ma, and Yanyan Lan. 2021.  Uncertainty calibration for ensemble-based debiasing methods.  _Advances in Neural Information Processing Systems_ , 34.
  * Xu et al. (2015) Zenglin Xu, Rong Jin, Bin Shen, and Shenghuo Zhu. 2015.  Nystrom approximation for sparse kernel methods: Theoretical analysis and empirical evaluation.  In _Twenty-Ninth AAAI Conference on Artificial Intelligence_.
  * Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. 2019.  PAWS: Paraphrase adversaries from word scrambling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1298–1308, Minneapolis, Minnesota. Association for Computational Linguistics.