spacing=nonfrench

# Quantum non-classicality in the simplest causal network

Pedro Lauand  Instituto de Física “Gleb Wataghin”, Universidade Estadual de
Campinas, 130830-859, Campinas, Brazil  Davide Poderini  International
Institute of Physics, Federal University of Rio Grande do Norte, 59078-970,
Natal, Brazil  Rafael Rabelo  Instituto de Física “Gleb Wataghin”,
Universidade Estadual de Campinas, 130830-859, Campinas, Brazil  Rafael Chaves
International Institute of Physics, Federal University of Rio Grande do Norte,
59078-970, Natal, Brazil  School of Science and Technology, Federal University
of Rio Grande do Norte, Natal, Brazil

###### Abstract

Bell’s theorem prompts us with a fundamental inquiry: what is the simplest
scenario leading to the incompatibility between quantum correlations and the
classical theory of causality? Here we demonstrate that quantum non-
classicality is possible in a network consisting of only three dichotomic
variables, without the need of the locality assumption neither external
measurement choices. We also show that the use of interventions, a central
tool in the field of causal inference, significantly improves the noise
robustness of this new kind of non-classical behaviour, making it feasible for
experimental tests with current technology.

_Introduction.—_ Bell’s theorem [1] is often considered the most stringent
notion of non-classicality. Based solely on the causal assumptions of locality
and freedom of choice one can exclude any physical theories based on
presumption that a hidden variable can predetermine the properties and
measurement outcomes of entangled systems. Given its far reaching foundational
consequences and application for information processing [2], it is natural to
question whether non-classical behaviour can emerge even if we give up or
relax some of these assumptions. After all, while historically essential, when
seen from the lens of causality theory [3], Bell’s theorem is just one example
of a causal compatibility problem [4, 5, 6], the aim of which is to decide a
given experimental data can be generated by an underlying causal structure or
not.

Within this new perspective, there are many physically motivated causal
structures beyond Bell’s and the recent years have seen an increasing interest
in understanding the non-classical behaviour and applications they can lead
to. In these generalizations, two new ingredients, relaxations of the causal
assumptions in Bell’s theorem, emerge as essential. Firstly, correlations
between distant parties can be mediated by independent sources, a common
occurrence in network scenarios [7] and that can lead to scenarios without the
need of the freedom of choice [8, 9, 10, 11, 12]. Secondly, time-like
correlations come to the fore, where the choices and outcomes of measurements
by one observer directly influence another [13, 14]. The independence of
sources in networks have revealed new features such as the need of complex
numbers in quantum theory [15], refined forms of multipartite non-locality
[16, 17, 18] and the self-testing of physical theories [19]. In turn,
scenarios with communication have led to new witness of non-classicality based
on interventions and the violation of causal bounds [20, 21, 22, 23]. The
combination of both features, source independence and communication in network
scenarios, however, are mostly unexplored.

The simplest network combining both ingredients, known as the Unrelated
Confounders (UC) scenario [24, 25], is displayed in Fig. 1(c) and can be
understood as the causal structure underpinning an entanglement swapping
experiment [26]. Similarly to the triangle causal structure [8, 9] displayed
in Fig. 1(b), this new scenario involves independent sources and does not
require the need of external inputs. In turn, similarly to the instrumental
scenario [27, 13] displayed in Fig. 1(a), it incorporates direct causal
influences between the observable nodes. And while it is known that quantum
correlations are incompatible with both the instrumental [13, 28] and triangle
[8, 9] causal assumptions, it remains an open question whether the same holds
for the UC case [25, 29]. This is the problem we solve here, showing that
quantum correlations incompatible with a classical description of the UC
scenario can indeed emerge. Interestingly, quantum violations are possible
even in the simplest case, where all measurements are dichotomic, which
remains an important open problem in other networks such as the triangle [30,
31]. Furthermore, we show how the use of interventional data [21] can
significantly increase the noise robustness, turning the Unrelated Confounders
scenario into the most experimentally feasible network displaying a novel form
of non-classicality beyond that in the paradigmatic Bell experiment.

$C$$B$$A$$\alpha$ (a) The Instrumental scenario.

$\alpha$$A$$\beta$$C$$\gamma$$B$ (b) The Triangle scenario.

$A$$C$$B$$\gamma$$\alpha$ (c) The UC scenario.

$A$$C$$B$$\gamma$$\alpha$ (d) The UC scenario with an intervention on $B$.

Figure 1: All non-trivial causal structures with three-observable nodes. a)
The instrumental scenario features the relaxation of locality. b) The triangle
scenario includes independent sources of correlations and do not required
external inputs. c) The UC causal structure merges the core features of the
instrumental and triangle DAGs: relaxation of locality, source independence
and the absence of external inputs. d) UC scenario with interventions made on
the central node of the causal network. Non-classicality here means the
possibility of a mismatch between classical and quantum predictions, that as
prove here, indeed happens for all of them.

_Unrelated Confounders Scenario—_ The UC scenario [24, 25] is the underlying
causal structure (see Fig. 1(c)) for a pivotal protocol in quantum information
science known as entanglement swapping [26]: a central node, Bob, shares
independent entangled states with the uncorrelated nodes of Alice and Charlie;
if Bob performs an entangled measurement and communicates its outcomes then
Alice and Charlie can also become entangled. It features three observed
variables $A$, $B$, and $C$ for the outcomes of Alice, Bob, and Charlie
respectively. And independent sources of correlation $\gamma$ between $AB$ and
$\alpha$ between $BC$, such that $P(\alpha,\gamma)=P(\alpha)P(\gamma)$.
Classically, observations $P(a,b,c)$ compatible with such causal structure
factorize as

$P(a,b,c)=\sum_{\alpha,\gamma}P(\alpha)P(\gamma)P(a|b,\gamma)P(b|\alpha,\gamma)P(c|b,\alpha).$
(1)

For quantum theory, we have the sources distributing quantum information; the
global state is $\psi_{ABC}=\psi_{\gamma}\otimes\psi_{\alpha}$ and the POVMs
describing the measurements on it are given by $E_{a|b}$, $E_{b}$ and
$E_{c|b}$. Born’s rule implies then that the observed probabilities obtained
when we impose the UC causal structure to a given experiment are given by

$P(a,b,c)=\Tr\left(\psi_{\gamma}\otimes\psi_{\alpha}\left(E_{a|b}\otimes
E_{b}\otimes E_{c|b}\right)\right).$ (2)

Importantly, the presence of communication of the output of $B$ to $A$ and
$C$, allows us to go beyond passive observations of the experiment and
investigate the role of interventions [32, 3]. In contrast to observations,
interventions allow to locally change the causal structure, eliminating all
external influences of a given variable $X$, forcing it to a particular value
$do(X=x)$ while keeping the rest of the model unchanged. Classically, an
intervention in node $B$ of the UC scenario (see Fig. 1(d)) breaks the
correlations between nodes $A$ and $C$, defining a do-conditional given by

$P(a,c|do(b))=P(a|do(b))P(c|do(b)),$ (3)

where $P(a|do(b))=\sum_{\gamma}P(\gamma)P(a|b,\gamma)$ and similarly for
$P(c|do(b))$. In turn, if the sources distribute quantum states the do-
conditional still respects (3), with

$P(a|do(b))=\Tr\left(\psi_{\gamma}\left(E_{a|b}\otimes\mathbbm{1}\right)\right),$
(4)

and similarly for $P(c|do(b))$.

_Concave inequalities—_ Causal networks with independent sources, as is the
case in the UC scenario, have a non-convex set of correlations [33] described
by polynomial Bell inequalities [34] that can be significantly difficult to be
characterized [6, 35, 36]. Given that general techniques such as inflation are
known to perform poorly on the UC case [29], we show next how one can leverage
quadratic programming (QP) [37] to circumvent these issues and derive useful
data-driven and robust non-linear witness for non-classicality.

Given the independence of the two sources represented by the quadratic
constraint $P(\alpha,\gamma)=P(\alpha)P(\gamma)$, the corresponding Bell
inequalities should also be quadratic, being expressed equivalently by a
functional depending square root $f(\sqrt{P(a,b,c)})$ of the probability
$P(a,b,c)$ of the experiment [38, 39, 34]. To further restrict the set of
classical correlations in the UC network, we can also add penalty functions
that force the inequality we are looking at to a subspace of the set of
allowed correlations. For instance, suppose that in a given experiment we
observe that the marginal distribution of Bob outcomes respect $P(b=0)=1/4$.
Adding a term like $|P(b=0)-1/4|$ to the inequality, we construct a witness
tailored to that specific observed correlation. Thus, in the following, we
will focus on inequalities with the following functional form

$\min_{x,y}\sum_{i}|x_{i}|+\sum_{i}b_{i}\sqrt{y_{i}},$ (5)

where $x_{i}$ and $y_{i}$ are linear functions of $P(a,b,c)$.

Given a certain expression of the form (5), our aim is to find its maximum
value in a classical description of causality and further search for quantum
violations of it. Even though we are thus dealing with a nonconvex
optimization problem, The general framework of linear programming (LP) [40]
can still be quite versatile and useful for our purposes. For instace, we can
optimize expressions like $|x|$ by defining a new variable $x^{\prime}$ with
the constraints $x^{\prime}\geq x$ and $x^{\prime}\geq-x$. Minimizing
$x^{\prime}$ ultimately yields the same result as minimizing $|x|$. Going
beyond the linear programming and convex optimization paradigm, we can turn to
quadratic optimization [37] to incorporate expressions of the form $\sqrt{x}$,
with $x\geq 0$, by defining a new variable $x^{\prime}$ and adding the
restriction $x^{\prime 2}=x$ which can easily be imposed in a quadratic
program. With that, we can reformulate an optimization of the general
expression (5) into

$\displaystyle\min_{\bar{x},\bar{y},x,y}\sum_{i}\bar{x_{i}}+\sum_{i}b_{i}\bar{y_{i}}$
(6) $\displaystyle\text{ s.t. }\quad-\bar{x}\leq
x\leq\bar{x},\quad\bar{y_{i}}^{2}=y_{i}\quad\forall i,$

which is a typical instance of quadratic programming [37].

As will be shown in the following, in order to reveal the non-classicality in
the UC it is enough to consider its simplest version where all observable
variables are binary. Also, we notice that $B$ has access to all the
information from the sources $\gamma$ and $\alpha$. Then by sending its output
$b$ to the remaining parts, it can effectively send information from $\alpha$
to $A$ and, similarly, from $\gamma$ to $C$. However, this forwarding of
information is limited by the cardinality of $B$. This suggests to look at the
correlation between $A$ and $C$ for each value of $b$. For simplicity, we use
_correlators_ as auxiliary variables. Defining $A:=(-1)^{a}$, and
$C:=(-1)^{c}$ we have

$\langle A^{i}C^{j}\rangle_{b}:=\sum_{a,c}(-1)^{a\cdot i+c\cdot j}P(a,b,c),$
(7)

with an inverse map given by

$P(a,b,c)=\frac{1}{4}\sum_{i,j}(-1)^{a\cdot i+c\cdot j}\langle
A^{i}C^{j}\rangle_{b}.$ (8)

Following the intuition delineated above, we will see that the following non-
linear function

$\displaystyle I=2\sum_{a=c}\sqrt{P(a,b=0,c)}+3\sqrt{P(1,1,0)}$ (9)
$\displaystyle-18\left|P(b=0)-\frac{1}{4}\right|-18\left|\sum_{a\neq
c}P(a,b=1,c)-\frac{1}{4}\right|$
$\displaystyle-4\sum_{a=c}\left|P(a,b=1,c)-\frac{1}{4}\right|-4\left|\langle
AC\rangle_{1}-\frac{1}{4}\right|$ $\displaystyle-|\langle A\rangle_{1}+\langle
C\rangle_{1}|-|\langle A\rangle_{0}|-|\langle C\rangle_{0}|,$

provides a good witness of non-classicality in the UC scenario. Using a
standard QP optimizer [41], we observe that the classical causal model (1)
respects $I\leq\frac{3}{\sqrt{2}}+\frac{\sqrt{7}}{6}\approx 2.562278895$.
111We obtain this classical upper bound using QP with a precision of
$10^{-9}$.

Following the same reasoning, we can add terms to the inequality that
incorporate interventional data, that is, adding terms that depend on $\langle
A\rangle_{do(b)}:=\sum_{a}(-1)^{a}P(a|do(b))$ and $\langle
C\rangle_{do(b)}:=\sum_{c}(-1)^{c}P(c|do(b))$ into the inequality. A
particularly useful function is given by

$\displaystyle F=2\sum_{a=c}\sqrt{P(a,b=0,c)}+4\sqrt{P(1,1,0)}$ (10)
$\displaystyle-\sum_{a=c}\left|P(a,b=1,c)-\frac{1}{4}\right|-\left|\langle
AC\rangle_{1}-\frac{1}{4}\right|$ $\displaystyle-\left|\sum_{a\neq
c}P(a,b=1,c)-\frac{1}{4}\right|-|\langle A\rangle_{1}+\langle C\rangle_{1}|$
$\displaystyle-|\langle A\rangle_{0}|-|\langle
C\rangle_{0}|-18\left|P(b=0)-\frac{1}{4}\right|$
$\displaystyle-\sum_{b}\left(|\langle A\rangle_{do(b)}|+|\langle
C\rangle_{do(b)}|\right).$

Optimizing it using QP under the classical constraints given by (1) and (3) we
obtain $F\leq\frac{9}{7}+\frac{7}{10}\sqrt{6}\approx 3.000357$.

_Quantum non-classicality—_ To prove the emergence of non-classicality, we
consider the following family of quantum distributions generated by a real
parameter $\theta$. Each of the two sources corresponds to the bipartite Bell
state

$\psi_{\gamma}=\psi_{\alpha}=|\psi^{+}\rangle=\frac{1}{\sqrt{2}}(|01\rangle+|10\rangle).$
(11)

and we take $A$ and $C$ to perform the binary measurements observables
$\sigma_{x}$ for $b=0$ and $\sigma_{z}$ for $b=1$. In turn $B$ performs the
entangling measurement
$\\{|\psi_{\theta}\rangle\langle\psi_{\theta}|,\mathbbm{1}-|\psi_{\theta}\rangle\langle\psi_{\theta}|\\}$,
where $|\psi_{\theta}\rangle$ is given by

$|\psi_{\theta}\rangle=\sin(\theta)|01\rangle+\cos(\theta)|10\rangle.$ (12)

Using this quantum strategy, by direct computation we obtain

$\displaystyle P_{Q}(a,b=0,c)=\frac{1}{16}\left(1+4r(-1)^{a+c}\right),$ (13)
$\displaystyle
P_{Q}(a,b=1,c)=\frac{1}{16}\left(3+(-1)^{a+c}+4s\left((-1)^{c}-(-1)^{a}\right)\right),$

with $r=\sin(2\theta)/4$ and $s=\cos(2\theta)/4$ and corresponding marginals
given by

$\displaystyle\langle AC\rangle_{0}=r$ (14) $\displaystyle\langle
C\rangle_{1}=-\langle A\rangle_{1}=s,$ $\displaystyle\langle
A\rangle_{0}=\langle C\rangle_{0}=0$ $\displaystyle P(b=0)=1-P(b=1)=\langle
AC\rangle_{1}=\frac{1}{4}.$

This choice leads to a simplified form for the non-linear expression in
inequality (9), expressed as

$I_{Q}(\theta)=\sqrt{1+\sin(2\theta)}+\frac{3}{2\sqrt{2}}\sqrt{1+\cos(2\theta)}.$
(15)

Figure 2: Maximum violations in the $r,s$ subspace. Allowed regions of the
subspace parametrized by $r$ and $s$. The blue and green curves represent
respectively the classical boundaries described by the observational
inequality (9) and the interventional inequality (10), while the red curve
represents the values given by the quantum distribution (13). The points show
the maximum violations attainable in each case (see Appendix C), along with
the point $\theta=\pi/8$ corresponding to the strategy presented here.

Choosing $\theta=\pi/8$, i.e. by having $B$ measure in an incomplete and
partially entangled Bell basis, we find
$I_{Q}(\frac{\pi}{8})=\sqrt{2+\sqrt{2}}\left(\frac{3\sqrt{2}+4}{4\sqrt{2}}\right)\approx
2.69238>\frac{3}{\sqrt{2}}+\frac{\sqrt{7}}{6}\approx 2.562278895$, thus
proving the existence of quantum non-classical correlations in the simplest
non-trivial causal structure. The inequality (9) and the set of quantum
correlations (13) are represented in Fig. 2.

To probe the robustness and experimental feasibility of this non-classicality,
we consider a paradigmatic noise model that introduces isotropic noise with a
visibility $v$ to the quantum sources, that is, each source is now given by
$\psi_{\gamma}=\psi_{\alpha}=v|\psi^{+}\rangle\langle\psi^{+}|+(1-v)\frac{\mathbbm{1}}{4}$.
Unfortunately, the best critical visibility we could find below which the
violation of (9) is no longer possible is significantly high, given by
$v_{\mathrm{crit}}\approx 0.98873$.

On the other hand, considering interventions and keeping the same set of
measurements and states, one can reach values of
$F_{Q}=\frac{1}{2}(2+\sqrt{2})^{\frac{3}{2}}\approx
3.15432>\frac{9}{7}+\frac{7}{10}\sqrt{6}\approx 3.000357$. Moreover, in this
case, the required visibility is significantly lower and one can obtain
violation of the hybrid observational-interventional inequality (10) for
visibilities down to $v_{\mathrm{crit}}\approx 0.87743$, bringing the test of
this new form of non-classicality within the reach of current technology (for
a more in-depth analysis of the robustness to noise see appendix C).

_Discussion.—_ The simplest causal networks that can potentially lead to
quantum correlations without a classical explanation require at least three
observable variables. Within this context, it is known that there are only
three different classes of DAGs [24]: the instrumental [27, 13], the triangle
[8, 9] and the unrelated confounders [25] causal scenarios, shown in Fig. 1.
It was known that the first two can lead to quantum non-classicality [13, 8]
and here we prove that the same holds true for UC. The relevance of it is two-
fold. First, it proves that all three observable DAGs support a classical-
quantum gap. Second, as we argue next, it provides an example of the simplest
and yet stronger form of non-classical behavior to date.

The instrumental case partially relaxes the locality assumption in Bell’s
theorem, since the measurement outcome of one observer can define the
measurement basis for the other. However, one still has to employ an external
random variable as an input and invoke the instrumentality assumption, that
the measurement choice of one observer has no direct causal influence over the
other. In turn, the triangle scenario allows for non-classicality without the
need of external inputs. However, it requires a space-like separation between
all three parties in the network.

The UC scenario we analyze here does not require any external input and at the
same time provides a strong relaxation of the locality assumption, since one
part explicitly communicates its measurement outcomes. Even more remarkable is
the fact that quantum violations of classical constraints in this simplest
causal structure are possible in the minimal case where all observable nodes
are dichotomic, something that is not possible in the instrumental [43] case
and is still an open problem for the triangle [30, 31].

The results we prove here allow for much more experimentally feasible tests of
non-classicality in quantum networks. On one side, the quantum violation we
reveal only requires two-outcome incomplete Bell state measurements, in stark
contrast with known proposals for the triangle network [9, 30]. Furthermore,
even though in the purely observational case the required visibilities for the
two sources of entangled states are very high, they can be brought with
interventions to values that are certainly amenable with current technologies,
also showing the power of interventional data as a witness of non-
classicality.

Since all three observable node networks can lead to quantum violations of
causal constraints, a natural question is to understand how generic is quantum
non-classicality. Does it happen in any non-trivial causal network? If not,
what are the key ingredients for the emergence of non-classical behaviour? As
previous results have shown and our present findings reinforce, locality and
freedom of choice, unavoidable requirements in a standard Bell test, can be
relaxed. Moving beyond the foundational aspects, a practical relevant
direction is to investigate the quantum information processing capabilities of
the UC network, from randomness certification [44] to cryptographic and
communication protocols [45].

## Acknowledgements

This work was supported by the Serrapilheira Institute (Grant No.
Serra-1708-15763), the Simons Foundation (Grant Number 1023171, RC), São Paulo
Research Foundation FAPESP (Grant No. 2022/03792-4) the Brazilian National
Council for Scientific and Technological Development (CNPq) (INCT-IQ and Grant
No 307295/2020-6).

## References

  * Bell [1964] J. S. Bell, On the einstein podolsky rosen paradox, Physics Physique Fizika 1, 195 (1964).
  * Brunner  _et al._ [2014] N. Brunner, D. Cavalcanti, S. Pironio, V. Scarani, and S. Wehner, Bell nonlocality, Reviews of modern physics 86, 419 (2014).
  * Pearl [2009] J. Pearl,  _Causality_ (Cambridge university press, 2009).
  * Wood and Spekkens [2015] C. J. Wood and R. W. Spekkens, The lesson of causal discovery algorithms for quantum correlations: causal explanations of bell-inequality violations require fine-tuning, New Journal of Physics 17, 033002 (2015).
  * Chaves  _et al._ [2015] R. Chaves, R. Kueng, J. B. Brask, and D. Gross, Unifying framework for relaxations of the causal assumptions in bell’s theorem, Physical review letters 114, 140403 (2015).
  * Wolfe  _et al._ [2019] E. Wolfe, R. W. Spekkens, and T. Fritz, The inflation technique for causal inference with latent variables, Journal of Causal Inference 7, 20170020 (2019).
  * Tavakoli  _et al._ [2022] A. Tavakoli, A. Pozas-Kerstjens, M.-X. Luo, and M.-O. Renou, Bell nonlocality in networks, Reports on Progress in Physics 85, 056001 (2022).
  * Fritz [2012] T. Fritz, Beyond bell’s theorem: correlation scenarios, New Journal of Physics 14, 103001 (2012).
  * Renou  _et al._ [2019] M.-O. Renou, E. Bäumer, S. Boreiri, N. Brunner, N. Gisin, and S. Beigi, Genuine quantum nonlocality in the triangle network, Physical review letters 123, 140401 (2019).
  * Šupić  _et al._ [2020] I. Šupić, J.-D. Bancal, and N. Brunner, Quantum nonlocality in networks can be demonstrated with an arbitrarily small level of independence between the sources, Physical Review Letters 125, 240403 (2020).
  * Chaves  _et al._ [2021] R. Chaves, G. Moreno, E. Polino, D. Poderini, I. Agresti, A. Suprano, M. R. Barros, G. Carvacho, E. Wolfe, A. Canabarro, _et al._ , Causal networks and freedom of choice in bell’s theorem, PRX Quantum 2, 040323 (2021).
  * Polino  _et al._ [2023] E. Polino, D. Poderini, G. Rodari, I. Agresti, A. Suprano, G. Carvacho, E. Wolfe, A. Canabarro, G. Moreno, G. Milani, _et al._ , Experimental nonclassicality in a causal network without assuming freedom of choice, Nature Communications 14, 909 (2023).
  * Chaves  _et al._ [2018] R. Chaves, G. Carvacho, I. Agresti, V. Di Giulio, L. Aolita, S. Giacomini, and F. Sciarrino, Quantum violation of an instrumental test, Nature Physics 14, 291 (2018).
  * Brask and Chaves [2017] J. B. Brask and R. Chaves, Bell scenarios with communication, Journal of Physics A: Mathematical and Theoretical 50, 094001 (2017).
  * Renou  _et al._ [2021] M.-O. Renou, D. Trillo, M. Weilenmann, T. P. Le, A. Tavakoli, N. Gisin, A. Acín, and M. Navascués, Quantum theory based on real numbers can be experimentally falsified, Nature 600, 625 (2021).
  * Coiteux-Roy  _et al._ [2021] X. Coiteux-Roy, E. Wolfe, and M.-O. Renou, No bipartite-nonlocal causal theory can explain nature’s correlations, Physical review letters 127, 200401 (2021).
  * Suprano  _et al._ [2022] A. Suprano, D. Poderini, E. Polino, I. Agresti, G. Carvacho, A. Canabarro, E. Wolfe, R. Chaves, and F. Sciarrino, Experimental genuine tripartite nonlocality in a quantum triangle network, PRX Quantum 3, 030342 (2022).
  * Pozas-Kerstjens  _et al._ [2022] A. Pozas-Kerstjens, N. Gisin, and A. Tavakoli, Full network nonlocality, Physical review letters 128, 010403 (2022).
  * Weilenmann and Colbeck [2020] M. Weilenmann and R. Colbeck, Self-testing of physical theories, or, is quantum theory optimal with respect to some information-processing task?, Physical Review Letters 125, 060406 (2020).
  * Ringbauer  _et al._ [2016] M. Ringbauer, C. Giarmatzi, R. Chaves, F. Costa, A. G. White, and A. Fedrizzi, Experimental test of nonlocal causality, Science advances 2, e1600162 (2016).
  * Gachechiladze  _et al._ [2020] M. Gachechiladze, N. Miklin, and R. Chaves, Quantifying causal influences in the presence of a quantum common cause, Physical Review Letters 125, 230401 (2020).
  * Agresti  _et al._ [2022] I. Agresti, D. Poderini, B. Polacchi, N. Miklin, M. Gachechiladze, A. Suprano, E. Polino, G. Milani, G. Carvacho, R. Chaves, _et al._ , Experimental test of quantum causal influences, Science advances 8, eabm1515 (2022).
  * Poderini  _et al._ [2024] D. Poderini, R. Nery, G. Moreno, S. Zamora, P. Lauand, and R. Chaves, Observational-interventional bell inequalities, arXiv preprint arXiv:2404.05015 (2024).
  * Evans [2016] R. J. Evans, Graphs for margins of bayesian networks, Scandinavian Journal of Statistics 43, 625 (2016).
  * Lauand  _et al._ [2023] P. Lauand, D. Poderini, R. Nery, G. Moreno, L. Pollyceno, R. Rabelo, and R. Chaves, Witnessing nonclassicality in a causal structure with three observable variables, PRX Quantum 4, 020311 (2023).
  * Zukowski  _et al._ [1993] M. Zukowski, A. Zeilinger, M. Horne, and A. Ekert, ” event-ready-detectors” bell experiment via entanglement swapping., Physical Review Letters 71 (1993).
  * Pearl [1995] J. Pearl, On the testability of causal models with latent and instrumental variables, in  _Proceedings of the Eleventh conference on Uncertainty in artificial intelligence_ (1995) pp. 435–443.
  * Van Himbeeck  _et al._ [2019] T. Van Himbeeck, J. B. Brask, S. Pironio, R. Ramanathan, A. B. Sainz, and E. Wolfe, Quantum violations in the instrumental scenario and their relations to the bell scenario, Quantum 3, 186 (2019).
  * Camillo  _et al._ [2024] G. Camillo, P. Lauand, D. Poderini, R. Rabelo, and R. Chaves, Estimating the volume of correlation sets in causal networks, Physical Review A 109, 012220 (2024).
  * Boreiri  _et al._ [2023] S. Boreiri, A. Girardin, B. Ulu, P. Lipka-Bartosik, N. Brunner, and P. Sekatski, Towards a minimal example of quantum nonlocality without inputs, Physical Review A 107, 062413 (2023).
  * Pozas-Kerstjens  _et al._ [2023] A. Pozas-Kerstjens, A. Girardin, T. Kriváchy, A. Tavakoli, and N. Gisin, Post-quantum nonlocality in the minimal triangle scenario, New Journal of Physics 25, 113037 (2023).
  * Balke and Pearl [1997] A. Balke and J. Pearl, Bounds on treatment effects from studies with imperfect compliance, Journal of the American statistical Association 92, 1171 (1997).
  * Garcia  _et al._ [2005] L. D. Garcia, M. Stillman, and B. Sturmfels, Algebraic geometry of bayesian networks, Journal of Symbolic Computation 39, 331 (2005).
  * Chaves [2016] R. Chaves, Polynomial bell inequalities, Physical review letters 116, 010402 (2016).
  * Åberg  _et al._ [2020] J. Åberg, R. Nery, C. Duarte, and R. Chaves, Semidefinite tests for quantum network topologies, Physical Review Letters 125, 110505 (2020).
  * Wolfe  _et al._ [2021] E. Wolfe, A. Pozas-Kerstjens, M. Grinberg, D. Rosset, A. Acín, and M. Navascués, Quantum inflation: A general approach to quantum causal compatibility, Physical Review X 11, 021043 (2021).
  * Nocedal and Wright [2006] J. Nocedal and S. J. Wright, Quadratic programming, Numerical optimization , 448 (2006).
  * Branciard  _et al._ [2010] C. Branciard, N. Gisin, and S. Pironio, Characterizing the nonlocal correlations created via entanglement swapping, Physical review letters 104, 170401 (2010).
  * Tavakoli  _et al._ [2014] A. Tavakoli, P. Skrzypczyk, D. Cavalcanti, and A. Acín, Nonlocal correlations in the star-network configuration, Physical Review A 90, 062109 (2014).
  * Boyd and Vandenberghe [2004] S. P. Boyd and L. Vandenberghe,  _Convex optimization_ (Cambridge university press, 2004).
  * Gurobi Optimization, LLC [2022] Gurobi Optimization, LLC, Gurobi Optimizer Reference Manual (2022).
  * Note [1] We obtain this classical upper bound using QP with a precision of $10^{-9}$.
  * Henson  _et al._ [2014] J. Henson, R. Lal, and M. F. Pusey, Theory-independent limits on correlations from generalized bayesian networks, New Journal of Physics 16, 113043 (2014).
  * Sekatski  _et al._ [2023] P. Sekatski, S. Boreiri, and N. Brunner, Partial self-testing and randomness certification in the triangle network, Physical Review Letters 131, 100201 (2023).
  * Lee and Hoban [2018] C. M. Lee and M. J. Hoban, Towards device-independent information processing on general quantum networks, Physical review letters 120, 020504 (2018).

##  Appendix A Quadratic optimization problem

In the main text, we described in general terms how we can obtain a classical
bound for inequality (9) and (10) by describing the problem as an instance of
Quadratic Programming (QP) optimization. For completeness, we show here the
explicit formulation of such QP problems. For the observational case, we have

$\displaystyle\max_{p}~{}$ $\displaystyle I(\\{p(a,b,c)\\})$ (16) s.t.
$\displaystyle p(a,b,c)\in\mathcal{C}_{\mathrm{UC}}$ (17)

where $\mathcal{C}_{\mathrm{UC}}$ is the set of classical compatible
distributions in the UC scenario, shown in (1). It is sufficient, in this last
constraint, to consider that the source $\gamma$ carries the information of
how Alice responds to each of Bob’s outcomes and similarly for $\alpha$ and
Charlie. Therefore, we may take the sources to be of the form
$\gamma=\gamma_{0}\gamma_{1}$ and $\alpha=\alpha_{0}\alpha_{1}$ with a
stochastic response function $p(b|\gamma,\alpha)$ for Bob and deterministic
response functions $\delta_{a,\gamma_{b}}$ and $\delta_{c,\alpha_{b}}$ for
Alice and Charlie respectively. This model yields

$\displaystyle
p(a,b,c)=\sum_{\gamma,\alpha}p(\gamma)p(\alpha)\delta_{a,\gamma_{b}}\delta_{c,\alpha_{b}}p(b|\gamma,\alpha)$
(18)

Similarly, for the interventional case, we have

$\displaystyle\max_{p}~{}$ $\displaystyle
F(\\{p(a,b,c)\\},\\{p(a,c|\mathrm{do}(b))\\})$ (19) s.t.
$\displaystyle\\{p(a,b,c),p(a,c|\mathrm{do}(b))\\}\in\mathcal{C}_{\mathrm{UC}}^{\mathrm{do}(b)}$
(20)

where now the requirement of being in
$\mathcal{C}_{\mathrm{UC}}^{\mathrm{do}(b)}$ is imposed on both the
interventional and observational distribution, that should be compatible with
the UC scenario, that is, besides the constraints in (18) we also have:

$\displaystyle
p(a,c|\mathrm{do}(b))=\left(\sum_{\gamma}p(\gamma)\delta_{a,\gamma_{b}}\right)\left(\sum_{\alpha}p(\alpha)\delta_{c,\alpha_{b}}\right)$
(21)

The problems are manifestly quadratic and non-convex, both in their objective
functions, as described in the main text, and in the constraints (18) and
(21). Solving these problems gives us optimal values for $I$ and $F$, which we
then approximate from above to finally obtain the upper bounds presented in
the main text:

$\displaystyle I\leq\frac{3}{\sqrt{2}}+\frac{\sqrt{7}}{6}\approx 2.562278895$
(22) $\displaystyle F\leq\frac{9}{7}+\frac{7}{10}\sqrt{6}\approx 3.000357$
(23)

##  Appendix B Optimal classical strategy

The solution of the QP described in the previous section, besides giving us a
bound for $I$ and $F$, also furnishes an optimal classical strategy that can
saturate that value. In particular from the numerical solution we obtain the
following classical model.

We consider $\gamma_{0},\gamma_{1},\alpha_{0},\alpha_{1}$ to be bits and to
saturate the classical bound of $I$ we may consider the distributions

$\displaystyle p(\gamma)=0.35428[00]+0.14571[10]+0.14571[01]+0.3543[11]$ (24)
$\displaystyle p(\alpha)=0.14717[00]+0.35282[10]+0.35282[01]+0.14719[11]$ (25)

Then for the response function of $B$, $p(b|\gamma,\alpha)$ we have
$p(b|\gamma,\alpha)=\delta_{b,0}$ for
$(\gamma,\alpha)\in\\{(00,01),(10,11)\\}$,
$p(b=0|\gamma=11,\alpha=10)=0.82842$ and $p(b|\gamma,\alpha)=\delta_{b,1}$
otherwise. This strategy reaches $I(p)\approx 2.56226$. Similarly, for the
inequality $F$, we can choose distributions

$\displaystyle p(\gamma)=0.26845[00]+0.23154[10]+0.23154[01]+0.26847[11]$ (26)
$\displaystyle p(\alpha)=0.23163[00]+0.26836[10]+0.26836[01]+0.23165[11]$ (27)

and $p(b|\gamma,\alpha)=\delta_{b,0}$ for
$\gamma,\alpha\in\\{(00,01),(10,11)\\}$,

$\displaystyle p(b=0|\gamma=00,\alpha=11)=0.85161,$ (28) $\displaystyle
p(b=0|\gamma=10,\alpha=01)=0.85223,$ (29) $\displaystyle
p(b=0|\gamma=10,\alpha=10)=0.14812,$ (30) $\displaystyle
p(b=0|\gamma=11,\alpha=11)=0.14801,$ (31)

and $p(b|\gamma,\alpha)=\delta_{b,1}$ otherwise. This model will give the
value $F(p)\approx 3.00001$, which can be taken as a lower bound.

##  Appendix C Maximum values and robustness of the quantum violation

In this section, we discuss in more detail the study of the robustness and
maximal values of the quantum violation for $I_{Q}$ and $F_{Q}$. We consider
the same strategy delineated in the main text. We choose
$\sigma_{x},\sigma_{z}$ as observables for both A and C with $b=0$ and $b=1$
respectively, and we define the projector in B as
$\outerproduct{\psi(\theta)}{\psi(\theta)}$ with

$\ket{\psi(\theta)}=\cos(\theta)\ket{01}+\sin(\theta)\ket{10}$ (32)

And we consider noisy states
$\psi_{\gamma}=\psi_{\alpha}=v|\psi^{+}\rangle\langle\psi^{+}|+(1-v)\frac{\mathbbm{1}}{4}$
for the two sources.

This strategy gives the following values for $I_{Q}$ and $F_{Q}$:

$\displaystyle I_{Q}(\theta,v)=$
$\displaystyle\frac{3\sqrt{3-v^{2}+2v\cos{(2\theta)}}}{4}+$
$\displaystyle+\sqrt{v^{2}\sin{(2\theta)}+1}-15\frac{|v^{2}-1|}{4}$ (33)
$\displaystyle F_{Q}(\theta,v)=$
$\displaystyle\frac{3\sqrt{3-v^{2}+2v\cos{(2\theta)}}}{4}+$
$\displaystyle+\sqrt{v^{2}\sin{(2\theta)}+1}-\frac{|v^{2}-1|}{2}$ (34)

These quantities are maximized, for any fixed $v$, by
$\theta_{I}=\arctan(2/5)$ and $\theta_{F}=\arctan(1/3)$ respectively, which,
for perfect visibility $v=1$, gives $I_{Q}(\theta_{F},1)\approx 2.69258$ and
$F_{Q}(\theta_{F},1)\approx 3.16228$, slightly above the violations obtained
with $\theta=\pi/8$ presented in the main text. From Eq. (33) and (34) we can
also obtain the critical noise value after which we cannot have a quantum
violation anymore, which is $v^{I}_{\mathrm{crit}}\approx 0.98873$ and
$v^{F}_{\mathrm{crit}}\approx 0.87743$ for the observational and the
interventional case respectively. Performing numeric optimization shows that,
at least using bipartite 2-qubits states, these values are optimal for these
inequalities.


MixUp | 0.821 | (0.819, 0.823) | 0.794 | (0.791, 0.797) | $\star$ | $\star$ | $\star$ | $\star$
VREX | 0.714 | (0.712, 0.716) | 0.64 | (0.637, 0.644) | $\star$ | $\star$ | $\star$ | $\star$
Table 10: Best (In-Distribution and Out-Of-Distribution) accuracy pair observed on each benchmark task. Note that domain generalization models can only be trained on datasets with more than one training subdomain (see Table 1 for domain generalization datasets and Section 4.1 for a list of domain generalization models). $\star$: domain generalization models cannot be trained when only one training subdomain is present. See also Figures 1, 2,3. Estimator | Sepsis | Unemployment
---|---|---
| ID Acc. (95% CI) | OOD Acc. (95% CI) | ID Acc. (95% CI) | OOD Acc. (95% CI)
Adv. Label DRO | 0.988 | (0.987, 0.989) | 0.925 | (0.924, 0.926) | 0.972 | (0.971, 0.973) | 0.96 | (0.959, 0.961)
CatBoost | 0.988 | (0.987, 0.989) | 0.925 | (0.923, 0.926) | 0.973 | (0.973, 0.974) | 0.962 | (0.961, 0.963)
DRO | 0.988 | (0.987, 0.989) | 0.925 | (0.924, 0.926) | 0.973 | (0.972, 0.973) | 0.961 | (0.96, 0.962)
FT-Transformer | 0.988 | (0.987, 0.989) | 0.925 | (0.924, 0.926) | 0.973 | (0.972, 0.974) | 0.962 | (0.961, 0.962)
Label Group DRO | 0.988 | (0.987, 0.989) | 0.925 | (0.924, 0.926) | 0.947 | (0.946, 0.948) | 0.926 | (0.925, 0.927)
LightGBM | 0.988 | (0.987, 0.989) | 0.928 | (0.926, 0.929) | 0.973 | (0.972, 0.974) | 0.96 | (0.96, 0.961)
MLP | 0.988 | (0.987, 0.989) | 0.925 | (0.923, 0.926) | 0.973 | (0.972, 0.973) | 0.96 | (0.959, 0.961)
NODE | 0.988 | (0.987, 0.989) | 0.925 | (0.924, 0.926) | 0.973 | (0.972, 0.974) | 0.962 | (0.961, 0.963)
ResNet | 0.988 | (0.987, 0.989) | 0.925 | (0.924, 0.926) | 0.972 | (0.971, 0.972) | 0.959 | (0.958, 0.96)
SAINT | 0.988 | (0.987, 0.989) | 0.925 | (0.924, 0.926) | 0.973 | (0.972, 0.974) | 0.962 | (0.961, 0.963)
TabTransformer | 0.988 | (0.987, 0.989) | 0.925 | (0.924, 0.926) | 0.972 | (0.971, 0.973) | 0.961 | (0.96, 0.962)
XGBoost | 0.988 | (0.987, 0.989) | 0.925 | (0.923, 0.926) | 0.973 | (0.972, 0.973) | 0.961 | (0.961, 0.962)
CORAL | $\star$ | $\star$ | $\star$ | $\star$ | 0.964 | (0.963, 0.965) | 0.95 | (0.949, 0.951)
DANN | $\star$ | $\star$ | $\star$ | $\star$ | 0.966 | (0.965, 0.967) | 0.948 | (0.947, 0.95)
Group DRO | $\star$ | $\star$ | $\star$ | $\star$ | 0.971 | (0.97, 0.972) | 0.958 | (0.957, 0.959)
IRM | $\star$ | $\star$ | $\star$ | $\star$ | 0.966 | (0.965, 0.967) | 0.948 | (0.947, 0.95)
MMD | $\star$ | $\star$ | $\star$ | $\star$ | 0.966 | (0.966, 0.967) | 0.953 | (0.952, 0.954)
MixUp | $\star$ | $\star$ | $\star$ | $\star$ | 0.844 | (0.842, 0.846) | 0.776 | (0.774, 0.778)
VREX | $\star$ | $\star$ | $\star$ | $\star$ | 0.873 | (0.871, 0.874) | 0.8 | (0.798, 0.802)
Table 11: Best (In-Distribution and Out-Of-Distribution) accuracy pair observed on each benchmark task. See also Figures 1, 2,3. Estimator | Voting
---|---
| ID Acc. (95% CI) | OOD Acc. (95% CI)
Adv. Label DRO | 0.875 | (0.843, 0.902) | 0.852 | (0.839, 0.865)
CatBoost | 0.883 | (0.852, 0.909) | 0.855 | (0.842, 0.868)
DRO | 0.881 | (0.85, 0.907) | 0.853 | (0.839, 0.866)
FT-Transformer | 0.879 | (0.848, 0.906) | 0.855 | (0.841, 0.868)
Label Group DRO | 0.862 | (0.829, 0.89) | 0.839 | (0.825, 0.852)
LightGBM | 0.881 | (0.85, 0.907) | 0.855 | (0.841, 0.868)
MLP | 0.892 | (0.862, 0.918) | 0.86 | (0.847, 0.873)
NODE | 0.885 | (0.854, 0.911) | 0.851 | (0.838, 0.864)
ResNet | 0.887 | (0.856, 0.912) | 0.836 | (0.822, 0.849)
SAINT | 0.888 | (0.858, 0.914) | 0.858 | (0.845, 0.871)
TabTransformer | 0.877 | (0.846, 0.904) | 0.859 | (0.845, 0.872)
XGBoost | 0.898 | (0.869, 0.923) | 0.851 | (0.838, 0.864)
CORAL | 0.883 | (0.852, 0.909) | 0.846 | (0.832, 0.859)
DANN | 0.892 | (0.862, 0.918) | 0.852 | (0.838, 0.865)
Group DRO | 0.877 | (0.846, 0.904) | 0.852 | (0.839, 0.865)
IRM | 0.804 | (0.767, 0.837) | 0.758 | (0.742, 0.774)
MMD | 0.892 | (0.862, 0.918) | 0.849 | (0.835, 0.862)
MixUp | 0.892 | (0.862, 0.918) | 0.851 | (0.837, 0.864)
VREX | 0.804 | (0.767, 0.837) | 0.754 | (0.737, 0.77)

Figure 10: Alternate version of Figure 2 with adjusted scaling for increased
detail.

Figure 11: Alternate version of Figure 3 with adjusted scaling for increased detail. Table 12: Sample sizes by split. In particular, large _test_ sizes are desirable for benchmarking, as they reduce the statistical uncertainty of comparing model performance. Task | ID Test | OOD Test | OOD Validation | Train | Validation | Total
---|---|---|---|---|---|---
Food Stamps | $78628$ | $48878$ | $5431$ | $629018$ | $78627$ | $840582$
Income | $158016$ | $75911$ | $8435$ | $1264123$ | $158015$ | $1664500$
Public Coverage | $500782$ | $817877$ | $90876$ | $4006249$ | $500781$ | $5916565$
Unemployment | $161365$ | $163611$ | $18180$ | $1290914$ | $161364$ | $1795434$
Voting | $520$ | $2772$ | $309$ | $4159$ | $520$ | $8280$
Hypertension | $27052$ | $518622$ | $57625$ | $216411$ | $27051$ | $846761$
Diabetes | $121154$ | $209375$ | $23264$ | $969229$ | $121154$ | $1444176$
Readmission | $4287$ | $50968$ | $5664$ | $34288$ | $4286$ | $99493$
HELOC | $278$ | $6914$ | $769$ | $2220$ | $278$ | $10459$
ICU Length of Stay | $1080$ | $11835$ | $1316$ | $8634$ | $1079$ | $23944$
ICU Hospital Mortality | $890$ | $13544$ | $1505$ | $7116$ | $889$ | $23944$
Sepsis | $140288$ | $134402$ | $14934$ | $1122299$ | $140287$ | $1552210$
Childhood Lead | $1476$ | $11466$ | $1274$ | $11807$ | $1476$ | $27499$
ASSISTments | $266566$ | $1906$ | $212$ | $2132526$ | $266566$ | $2667776$
College Scorecard | $12320$ | $1352$ | $151$ | $98556$ | $12320$ | $124699$
Table 13: Label summary statistics for test sets in TableShift. $\bar{Y}$ gives the proportion of positive labels for the respective split, and $\textrm{Var}(\bar{Y})$ the variance of the sample proportion. | ID | OOD
---|---|---
Task | $\bar{Y}_{\textrm{ID}}$ | $\textrm{Var}(\bar{Y}_{\textrm{ID}})$ | $\bar{Y}_{\textrm{OOD}}$ | $\textrm{Var}(\bar{Y}_{\textrm{OOD}})$
Voting | $0.8038461538461539$ | $0.017413385368340785$ | $0.753968253968254$ | $0.008180417921181871$
ASSISTments | $0.6952874710203102$ | $0.0008915074566584878$ | $0.4365162644281217$ | $0.011360029505513087$
Childhood Lead | $0.0291327913279132$ | $0.004377516449907685$ | $0.0803244374672946$ | $0.002538254711414018$
College Scorecard | $0.1270292207792207$ | $0.003000173126057655$ | $0.3106508875739645$ | $0.012585415670184513$
Diabetes | $0.1272265051091998$ | $0.0009573498959145716$ | $0.1741946268656716$ | $0.0008288841161625143$
FICO HELOC | $0.2553956834532374$ | $0.026154544038025564$ | $0.568990454150998$ | $0.005955678519652748$
Food Stamps | $0.1908989164165437$ | $0.0014015690818247213$ | $0.2198739719301117$ | $0.0018733226380098552$
Hospital Readmission | $0.4161418241194308$ | $0.00752831226105025$ | $0.4938392716998901$ | $0.0022145640159944124$
Hypertension | $0.4019665828774212$ | $0.002980972789215877$ | $0.5842926061755961$ | $0.0006843583995662628$
ICU Hospital Mortality | $0.0853932584269663$ | $0.009367721884526318$ | $0.1238924985233313$ | $0.002830920886137395$
ICU Length of Stay | $0.3981481481481481$ | $0.014895506415195738$ | $0.4559357836924377$ | $0.00457817921830246$
Income | $0.3208219420818145$ | $0.0011742844857690114$ | $0.3979660391774578$ | $0.0017765651779704829$
Public Health Ins. | $0.2237879955749208$ | $0.0005889578511497119$ | $0.6362704905505351$ | $0.0005319443024553697$
Sepsis | $0.0119967495437956$ | $0.00029067034151272915$ | $0.0750732875998869$ | $0.0007187759095289447$
Unemployment | $0.033823939516004$ | $0.0004500238798831049$ | $0.0515124288709194$ | $0.0005464690706436891$
Table 14: PMA-OOD results (cf. Figure ) and standard deviation of PMA-OOD over benchmark tasks. (Cf. Figure 4(a).) Estimator | PMA-OOD Mean | PMA-OOD Std.
---|---|---
VREX | $0.8757761792572389$ | $0.0763110750213929$
IRM | $0.9101333649405229$ | $0.06876751843249528$
Label Group DRO | $0.9153705685925984$ | $0.1289550231667853$
MixUp | $0.9166581362766968$ | $0.075654272453895$
TabTransformer | $0.9215827102786986$ | $0.09078085346812126$
DANN | $0.9321351453891862$ | $0.07524691815143086$
DRO | $0.9324980036969195$ | $0.10678184940675324$
MMD | $0.9401376773678964$ | $0.058701053854589326$
CORAL | $0.9443187703053098$ | $0.059333615193346304$
Adv. Label DRO | $0.9498603380349795$ | $0.0801329734975493$
ResNet | $0.9563455290467845$ | $0.06882493225312407$
MLP | $0.9613216129897759$ | $0.05404889430171553$
Group DRO | $0.9619768942957171$ | $0.0593640087093276$
NODE | $0.9625757533489948$ | $0.06618578016846459$
SAINT | $0.9636991974599568$ | $0.07010491047112488$
LightGBM | $0.9638239715887773$ | $0.06967431761990033$
XGBoost | $0.9640605989031691$ | $0.06487600014630537$
FT-Transformer | $0.968490871459869$ | $0.06893926871894276$
CatBoost | $0.9938205706070402$ | $0.014003599636624751$
Table 15: Label summary statistics for test sets in TableShift. $\bar{Y}$ gives the proportion of positive labels for the respective split, and $\textrm{Var}(\bar{Y})$ the variance of the sample proportion. (Cf. Figure 4(b).) | ID | OOD
---|---|---
Method | ID Accuracy | Std. | OOD Accuracy | Std.
Adv. Label DRO | $0.833800122876357$ | $0.12539332400797265$ | $0.7916669399588646$ | $0.13217226836298807$
CatBoost | $0.8618698362035417$ | $0.10490910217043634$ | $0.7941915805446013$ | $0.1255006674899661$
DANN | $0.815866296010201$ | $0.1371381994828937$ | $0.7695596391234647$ | $0.14792817653674895$
CORAL | $0.8240142132337408$ | $0.1289477408132082$ | $0.7774118847069205$ | $0.13435374377657636$
DRO | $0.8426999554074797$ | $0.12871341173183118$ | $0.7726685495491659$ | $0.14694098809426048$
FT-Transformer | $0.8663905007551306$ | $0.1025142266851832$ | $0.7937947816708624$ | $0.12555440779133642$
Group DRO | $0.8324124451132319$ | $0.1289545028694996$ | $0.7913605188378716$ | $0.13259456830812963$
IRM | $0.8001315298523886$ | $0.1303327136607609$ | $0.7492605753114917$ | $0.13603246296143795$
Label Group DRO | $0.8287316025069875$ | $0.12301995903481792$ | $0.7591484649227083$ | $0.13412917497836183$
LightGBM | $0.855525405148914$ | $0.10802785319401524$ | $0.7814413771334128$ | $0.12441641593417348$
MixUp | $0.8073384504302261$ | $0.125316752225611$ | $0.751644880020089$ | $0.11803683192505401$
MLP | $0.8448652854510845$ | $0.1255712644990551$ | $0.7743104815592784$ | $0.14122801290924608$
MMD | $0.8252778569421158$ | $0.12970168491961503$ | $0.7741104568776251$ | $0.13505865760605656$
NODE | $0.8527918592062239$ | $0.11032070271423704$ | $0.7814246096867519$ | $0.12873983458556224$
ResNet | $0.8440124461006763$ | $0.12554233119259353$ | $0.7731220969402316$ | $0.13907251582180064$
SAINT | $0.8675185473018586$ | $0.0988731686359445$ | $0.786579368822034$ | $0.12669617670544958$
TabTransformer | $0.8353349566161345$ | $0.13563033961812943$ | $0.7586534500644456$ | $0.1604698620909881$
VREX | $0.7859898602498584$ | $0.126750657322286$ | $0.7202060145457858$ | $0.12774758420253876$
XGBoost | $0.8573385118855809$ | $0.10457810677437979$ | $0.7834438528754843$ | $0.12194469828859483$

###  E.4 Results with Additional Random Seeds

Our experiments on each model-dataset pair comprise a single run of 100 rounds
of our hyperparameter tuning protocol described in Section 4.2. Here, we
provide the results of additional experiments conducted using different random
seeds, in order to evaluate the sensitivity of our results to the random
variation inherent in the training and hyperparameter tuning process.

For these experiments, we conduct an identical procedure to the experiments
described in the main text of our paper, but only change the random seed. This
process affects the random initialization of model weights, random
initialization of hyperparameter tuning, and training data shuffling, among
other procedures. We note that it does _not_ affect the train/test splitting
in our datasets, as the train/test splits are defined by distribution shifts
and are fixed to ensure comparability of the benchmark across experiments.

The results are shown in Table 16. Table 16 shows that, across the five models
and three datasets evaluated, there is minimal variation in performance due to
random seeds. Of the 90 measurements covering 45 trials represented in Table
16, the 95% Clopper-Pearson CIs for both ID and OOD accuracy overlap in all
cases, with only four exceptions (LightGBM, iteration 0, Food Stamps ID and
OOD accuracy; LightGBM, iteration 2, Hypertension OOD accuracy; MLP,
Hypertension, iteration 0, OOD accuracy; FT-Transformer, iteration 0, OOD
accuracy). These results provide evidence that our results are robust to
variation due to random seed.

Table 16: Results with additional random seeds. Varying random seeds has a minimal impact on the final results of our hyperparameter tuning procedure, indicating that our findings are robust to variation due to random seeds. See Section E.4 for details on experimental design. |  |  | ID Test Accuracy | OOD Test Accuracy
---|---|---|---|---
Task | Base Estimator | Iteration | Value | 95% CI | Value | 95% CI
College Scorecard | CatBoost | 0 | 0.957 | (0.954, 0.961) | 0.885 | (0.866, 0.901)
1 | 0.959 | (0.955, 0.962) | 0.879 | (0.861, 0.896)
2 | 0.959 | (0.956, 0.963) | 0.882 | (0.863, 0.898)
FT-Transformer | 0 | 0.948 | (0.944, 0.952) | 0.859 | (0.839, 0.877)
1 | 0.946 | (0.942, 0.95) | 0.850 | (0.83, 0.868)
2 | 0.940 | (0.936, 0.945) | 0.830 | (0.809, 0.85)
LightGBM | 0 | 0.939 | (0.935, 0.943) | 0.822 | (0.8, 0.841)
1 | 0.943 | (0.938, 0.947) | 0.839 | (0.819, 0.859)
2 | 0.943 | (0.939, 0.947) | 0.837 | (0.816, 0.856)
MLP | 0 | 0.947 | (0.942, 0.95) | 0.845 | (0.825, 0.864)
1 | 0.949 | (0.944, 0.952) | 0.859 | (0.84, 0.878)
2 | 0.945 | (0.941, 0.949) | 0.859 | (0.839, 0.877)
XGBoost | 0 | 0.942 | (0.938, 0.946) | 0.830 | (0.809, 0.85)
1 | 0.946 | (0.942, 0.95) | 0.842 | (0.821, 0.861)
2 | 0.947 | (0.943, 0.951) | 0.845 | (0.824, 0.864)
Food Stamps | CatBoost | 0 | 0.849 | (0.847, 0.852) | 0.825 | (0.821, 0.828)
1 | 0.850 | (0.847, 0.852) | 0.824 | (0.821, 0.827)
2 | 0.849 | (0.847, 0.852) | 0.824 | (0.82, 0.827)
FT-Transformer | 0 | 0.843 | (0.841, 0.846) | 0.816 | (0.812, 0.819)
1 | 0.848 | (0.846, 0.851) | 0.824 | (0.82, 0.827)
2 | 0.844 | (0.842, 0.847) | 0.817 | (0.814, 0.82)
LightGBM | 0 | 0.836 | (0.833, 0.838) | 0.808 | (0.805, 0.812)
1 | 0.844 | (0.841, 0.846) | 0.818 | (0.814, 0.821)
2 | 0.843 | (0.84, 0.846) | 0.817 | (0.814, 0.821)
MLP | 0 | 0.841 | (0.838, 0.844) | 0.815 | (0.812, 0.819)
1 | 0.845 | (0.842, 0.847) | 0.817 | (0.814, 0.821)
2 | 0.844 | (0.841, 0.846) | 0.811 | (0.808, 0.815)
XGBoost | 0 | 0.844 | (0.842, 0.847) | 0.820 | (0.817, 0.824)
1 | 0.843 | (0.84, 0.845) | 0.819 | (0.815, 0.822)
| 2 | 0.845 | (0.842, 0.847) | 0.820 | (0.816, 0.823)
Hypertension | CatBoost | 0 | 0.670 | (0.665, 0.676) | 0.599 | (0.597, 0.6)
1 | 0.671 | (0.665, 0.676) | 0.599 | (0.597, 0.6)
2 | 0.671 | (0.666, 0.677) | 0.600 | (0.598, 0.601)
FT-Transformer | 0 | 0.666 | (0.661, 0.672) | 0.604 | (0.603, 0.605)
1 | 0.670 | (0.665, 0.676) | 0.594 | (0.593, 0.596)
2 | 0.672 | (0.666, 0.677) | 0.595 | (0.594, 0.596)
LightGBM | 0 | 0.678 | (0.672, 0.683) | 0.634 | (0.633, 0.635)
1 | 0.672 | (0.666, 0.677) | 0.636 | (0.635, 0.637)
2 | 0.672 | (0.667, 0.678) | 0.628 | (0.627, 0.629)
MLP | 0 | 0.664 | (0.658, 0.67) | 0.583 | (0.582, 0.584)
1 | 0.669 | (0.663, 0.674) | 0.597 | (0.596, 0.599)
2 | 0.668 | (0.662, 0.673) | 0.598 | (0.597, 0.599)
XGBoost | 0 | 0.671 | (0.665, 0.677) | 0.588 | (0.587, 0.59)
| 1 | 0.669 | (0.664, 0.675) | 0.586 | (0.584, 0.587)
| 2 | 0.669 | (0.664, 0.675) | 0.586 | (0.584, 0.587)

###  E.5 Results with Hybrid Methods

Our main study design is focused on benchmarking existing previously-proposed
methods for tabular modeling. The methods we evlauate span _models_ , which
prescribe the functional form of a predictor $f_{\theta}$, and also _objective
functions_ , which describe the loss $\mathcal{L}$ to be minimized while
learning the parameters $\theta$ of a fixed predictor $f$. Concretely, for
example, FT-Transformer or MLP specify the form of $f$, while some robustness
interventions, such as Group DRO, specify an objective that can be monimized
over any smooth continuous function.

Our study does not explore potential combinations of different models and
objective functions from the preexisting literature. In this section, we
conduct an exploratory investigation into whether “hybrid methods” –
combinations of different models and objective functions explored in our study
– might improve robustness, for the best-performing compatible combinations of
models and objective functions in our study.

In particular, our hybrid model study explores the use of the Group DRO
objective function, in combination with three models from our study: FT-
Transformer, NODE, and ResNet. Group DRO was selected as it is the highest-
performing objective-based technique in our study (see Figure 4(a)), and the
three models were selected as they are the highest-performing Transformer-
based model, tree-based model, and baseline supervised model, repsectively, in
our study. We note that Group DRO cannot be easily combined with CatBoost,
XGBoost, or LightGBM, as these are not smooth differentiable continuous
functions, which is a requirement for the use of the Group DRO objective.

Our methodology in this section is as follows: for each estimator (FT-
Transformer, NODE, ResNet), we train the model with both ERM (the standard
procedure used in our main experiments above) and Group DRO. We follow the
same hyperparameter tuning procedure as described in Section 4.2) above. We
use the same hyperparameter grid defined in Section I for each model, but also
include a full sweep over the Group DRO step size parameter, using the Group
DRO grid described in Section I (thus, for model X, we take the union of the
two hyperparameter grids: { grid(X) $\cup$ grid(Group DRO) } ). We conduct
this procedure for five benchmark datasets: Childhood Lead, College Scorecard,
Food Stamps, Hypertension, and Voting.

The results of our hybrid model experiments are shown in Table 17. The results
show little or no evidence that Group DRO reduces shift gaps for the models
evaluated, as indicated by the fact that OOD test accuracy intervals tend to
be overlapping, or _higher_ , for ERM relative to Group DRO. Keeping in mind
that Group DRO was parameterized over MLP models in our main experiments (as
all prior works only use Group DRO with MLP), the results in Table 17 suggest
that Group DRO may primarily improve weak (MLP) models but does not improve
robustness for stronger models, explaining the improvements for Group DRO over
vanilla MLP models in the main text.

Table 17: Hybrid method results. We compare Group DRO to standard ERM for the highest-performing Transformer, tree-based, and baseline models in our study (FT-Transformer, NODE, and ResNet, respectively) over five TableShift tasks, following our hyperparameter tuning procedure. There is little or no evidence that Group DRO reduces shift gaps for these models, indicating that Group DRO may primarily improve weak (MLP) models but does not improve robustness for stronger models. See Section E.5 for details on experimental design. |  |  | ID Test Accuracy | OOD Test Accuracy
---|---|---|---|---
Task | Base Estimator | Method | Value | 95% CI | Value | 95% CI
Childhood Lead | FT-Transformer | ERM | 0.971 | (0.961, 0.979) | 0.920 | (0.915, 0.925)
Group DRO | 0.971 | (0.961, 0.979) | 0.920 | (0.915, 0.925)
NODE | ERM | 0.971 | (0.961, 0.979) | 0.920 | (0.915, 0.925)
Group DRO | 0.971 | (0.961, 0.979) | 0.920 | (0.915, 0.925)
ResNet | ERM | 0.971 | (0.961, 0.979) | 0.920 | (0.915, 0.925)
Group DRO | 0.971 | (0.961, 0.979) | 0.920 | (0.915, 0.925)
College Scorecard | FT-Transformer | ERM | 0.948 | (0.944, 0.952) | 0.859 | (0.839, 0.877)
Group DRO | 0.935 | (0.93, 0.939) | 0.815 | (0.793, 0.835)
NODE | ERM | 0.944 | (0.939, 0.948) | 0.844 | (0.823, 0.863)
Group DRO | 0.946 | (0.942, 0.95) | 0.835 | (0.814, 0.854)
ResNet | ERM | 0.947 | (0.943, 0.951) | 0.854 | (0.834, 0.872)
Group DRO | 0.947 | (0.942, 0.95) | 0.824 | (0.803, 0.844)
Food Stamps | FT-Transformer | ERM | 0.843 | (0.841, 0.846) | 0.816 | (0.812, 0.819)
Group DRO | 0.826 | (0.823, 0.829) | 0.795 | (0.792, 0.799)
NODE | ERM | 0.849 | (0.847, 0.852) | 0.822 | (0.819, 0.825)
Group DRO | 0.845 | (0.842, 0.847) | 0.822 | (0.819, 0.825)
ResNet | ERM | 0.843 | (0.84, 0.845) | 0.820 | (0.817, 0.824)
Group DRO | 0.848 | (0.846, 0.851) | 0.818 | (0.815, 0.822)
Hypertension | FT-Transformer | ERM | 0.666 | (0.661, 0.672) | 0.604 | (0.603, 0.605)
Group DRO | 0.665 | (0.659, 0.67) | 0.608 | (0.607, 0.609)
NODE | ERM | 0.670 | (0.664, 0.676) | 0.597 | (0.596, 0.599)
Group DRO | 0.671 | (0.665, 0.676) | 0.592 | (0.591, 0.593)
ResNet | ERM | 0.667 | (0.661, 0.672) | 0.608 | (0.606, 0.609)
Group DRO | 0.663 | (0.658, 0.669) | 0.590 | (0.589, 0.592)
Voting | FT-Transformer | ERM | 0.879 | (0.848, 0.906) | 0.855 | (0.841, 0.868)
Group DRO | 0.894 | (0.865, 0.919) | 0.858 | (0.844, 0.87)
NODE | ERM | 0.885 | (0.854, 0.911) | 0.851 | (0.838, 0.864)
Group DRO | 0.898 | (0.869, 0.923) | 0.860 | (0.847, 0.873)
ResNet | ERM | 0.887 | (0.856, 0.912) | 0.836 | (0.822, 0.849)
Group DRO | 0.898 | (0.869, 0.923) | 0.847 | (0.833, 0.861)

##  Appendix F Model Details

This section describes the models used in our study. For the hyperparameters
used in our experiments, see Section I.

Our implementations of these models, along with associated code to train
models with fixed hyperparameters or to tune hyperparameters at scale via the
Ray framework, are available at https://github.com/mlfoundations/tableshift.

###  F.1 Baseline Models

XGBoost: XGBoost is a popular library for learning gradient-boosted trees. We
use the original XGBoost implementation [20]. XGBoost introduced column
subsampling, weight regularization, and introduced major improvements in
efficiency for training gradient boosted models on large or out-of-core
datasets.

LightGBM: LightGBM is a library for learning gradient-boosted trees which
extends the success of XGBoost in working fast and with large datasets [48].
LightGBM introduces novel techniques such as converting continuous features to
histograms (for computational efficiency and for to reduce overfitting),
combining certain features using Exclusive Feature Bundling (EFB), and through
the use of Gradient-based One-Side Sampling (GOSS).

CatBoost: CatBoost [25] is a library for learning gradient-boosted trees which
includes novel techniques for leveraging categorical features. This includes
heuristics to replace numeric or one-hot encoding of categorical features with
label-derived heuristics; "appearance" (count) features for categorical
features; and efficient greedy feature recombination techniques.

MLP: We use standard multilayer perceptrons, via the implementation in
RTDL292929https://github.com/Yura52/rtdl. MLPs have been shown to be highly
effective models for tabuilar data, particularly when a large model search
space is used and regularization is carefully tuned [46].

###  F.2 Tabular Neural Networks

FT-Transformer: FT-Transformer is a transformer-based model that learns
separate feature tokenizers for numeric and categorical data, and applies a
transformer model [85] to the tokenized features.

Tabular ResNet: We use the version of Tabular ResNet proposed in [36]. We note
that, despite the fact that this approach is shown to have competitive
performance with many existing tabular data models in [36], it has not been
widely used in the literature.

NODE Neural Oblivious Decision Ensembles (NODE) [70] is a method that
leverages oblivious ensembling methods to train “tree-like” neural networks.

TabTransformer: TabTransformers [43] is a model that uses learned embeddings
of categorical features, which are then passed through standard Transformer
layers, alongside layer normalization of continuous features.

SAINT: SAINT [79] uses an enhanced embedding method for categorical features,
alongside (optional) attention over both rows and columns, in a Transformer
architecture. We note that, due to its use of featurewise feedforward layers,
SAINT was impractical to use for our datasets with the largest numbers of
features (ICU Hospital Mortality, ICU Length of Stay; both contain over 1000
features which resulted in over 13B parameters for even a single-layer SAINT
model).

###  F.3 Robustness Models

Distributionally Robust Optimization (DRO): We use two variants of DRO, both
via [53]. For both methods, the model attempts to optimize a worst-care risk
within a bounded distribution of the training data via a projected gradient
descent procedure.

Group DRO: Originally introduced as a subgroup robustness method in [76],
Group DRO is a DRO method which attempts to optimize the worst-group loss
during training. Group DRO can also, however, be used as a domain robustness
method by treating the domains as “group labels”, which is how we use it in
our study. We note that this use of Group DRO has been applied previously;
e.g. [38].

###  F.4 Domain Generalization Models

Invariant Risk Minimization (IRM): IRM [6] uses a modified training objective
to learn models which a feature representation such that the optimal linear
classifier on top of that representation matches across domains.

MixUp: Inter-Domain MixUp [90, 89] uses combinations of data points from
random pairs of domains and their labels during training.

Domain-Adversarial Neural Networks (DANN): DANN [1] uses adversarial training
to achieve domain robustness, where a discriminator attempts to predict the
domain of a training example in order to match feature distributions across
domains.

Risk Extrapolation (REx): REx [52] attempts to reduce differences in risk
across training domains, in order to reduce a model’s sensitivity to
distributional shifts.

CORAL: CORAL (CORrelation ALignment) [82] attempts to ensure that feature
activations are similar across domains; this can be used as either a domain
generalization method or a domain adaptation method.

MMD: Similar to CORAL using a different kernel, MMD attempts to minimize the
Maximum Mean Discrepancy (MMD) between domains.

###  F.5 Label Shift Robustness Models

Group DRO: here, we use Group DRO [76] with class labels as the grouping
attribute.

Adversarial Label DRO: This method, proposed in [92], uses a distributionally
robust objective to optimize for the worst-case weighting over label
groupings. We note that this approach is computationally expensive, requiring
sample-level gradients even following the authors’ original implementation,
and so was not practical for our datasets with very large $n$ (ASSISTments,
Public Health Insurance).

##  Appendix G Comparison To Other Benchmarking Toolkits

In this section, we provide a brief comparison of TableShift to other relevant
benchmarking toolkits. We note that our goal in this section is _not_ to fully
characterize the functionality of other benchmarking platforms; it is only to
compare and contrast their relevant attributes with TableShift and to motivate
the creation of a novel benchmark and API for TableShift (as opposed to
incorporating TableShift into an existing toolkit).

As noted above, there is _no_ existing benchmark for domain shift in tabular
data. However, in this section we compare to three main categories of relevant
related toolkits: (1) domain shift benchmarks for non-tabular data (DomainBed,
WILDS); (2) IID (non-domain-shift) benchmarks for tabular data ([37], OpenML);
and (3) generic data-hosting platforms (Huggingface Datasets, TensorFlow
Datasets. We briefly introduce and compare to each of these below.

###  G.1 Domain Shift Benchmarks for Non-Tabular Data

WILDS: WILDS303030See https://wilds.stanford.edu/ and [50]. is perhaps the
closest benchmark to TableShift, but only uses non-tabular data. WILDS
demonstrates a lightweight, useful set of programming abstractions for
benchmarking models and sharing results across a diverse set of datasets for
domain shift. WILDS interfaces with image and text datasets, and includes a
rich variety of datasets with real-world sensitive attributes, carefully
selected domain shifts, and has wide adoption in the robustness community.
WILDS includes a high-quality Python API, which has led to wide integration
with researchers’ open-source code and widespread adoption. However, WILDS is
currently not compatible with tabular datasets and does not include any
tabular datasets in its benchmark suite. The needs for tabular datasets are
different than the datasets currently used in WILDS (i.e. non-Torch models
must be supported by the benchmark; subgroup and domain-shift information are
handled differently for our use cases; data preprocessing is also different
for tabular data as noted above).

DomainBed: DomainBed313131See https://github.com/facebookresearch/DomainBed
and [38]. is a benchmark that contains several reference implementations,
including some that have been adapted for use in TableShift. In addition to
these model implementations, DomainBed serves as an interface to several
existing datasets through a Python API. However, DomainBed is specifically
adapted to image data. It only supports image datasets with a specific folder
structure (which would make extending to tabular datasets nontrivial) and
includes many image-specific augmentation components of its pipelines. It also
uses ResNet50/ResNet18 networks designed specifically for image
classification, and therefore does not currently support either deep learning
models suited to image data, nor (more importantly) the effective non-DL
baselines described above such as XGBoost and LightGBM.

Shift Happens: The “shift happens” benchmark323232https://shift-happens-
benchmark.github.io/index.html is a community-built benchmark suite for image
models. It specifically aims to feature datasets with domain shift, for tasks
including image classification under domain shift, and out-of-distribution
detection. The benchmark includes a Python API. This benchmark does not
support tabular datasets, and is much less widely used, perhaps due to the
community-driven effort (as opposed to benchmarks such as WILDS and DomainBed,
which come packaged with preselected datasets and domain splits).

Shifts 2.0: Shifts ([57], recently upgraded to Shifts 2.0 [57]) is a
collection of multimodal tasks with domain shifts. The Shifts benchmark is a
part of the Shifts Project, an international collaboration of academic and
industrial researchers dedicated to studying distributional
shift.333333https://shifts.ai/ Shifts 2.0, the current version, includes five
tasks: tabular weather prediction, tabular marine cargo vessel power
consumption prediction, machine translation, self-driving car vehicle motion
prediction, and segmentation of white matter Multiple Sclerosis lesions in 3D
magnetic resonance brain images. While shifts does contain two tabular data
tasks, its relatively small number of tasks makes it a less reliable benchmark
compared to the rich set of tabular datasets comprising TableShift. Shifts
also does not include any tasks with real-world sensitive subgroups (such as
age, race, or gender) which are of particular interest in many tabular
classification tasks. Additionally, the domains represented in the tabular
tasks of Shifts do not cover many critical domains widely recognized as using
tabular data (e.g. finance, medicine, etc.; see Section D.2).

###  G.2 IID Benchmarks for Tabular Data

Benchmark of [37]: The unnamed benchmark proposed in this work is intended to
provide a consistent benchmarking suite for tabular dataset classification
tasks, and was motivated by some of the same gaps described in this work.
However, the datasets comprising the benchmark of [37] do not meet our
specifications, for several reasons. First, the datasets are limited to be of
maximum size $10k$ observations; this is too small for reliable and repeated
benchmarking comparisons. Additionally, the datasets are label-balanced; in
contrast, we use the naturaly-occurring label distributions for all datasets
(and we show that these label distributions are importantly related to shift
gap). Most critically, the benchmark datasets in [37] do not contain domain
shifts; for most or all of the datasets, it is does not appear that a domain
shift could be induced from splitting the existing data on an existing
feature.

OpenML: OpenML has some overlap with the proposed functionality. However,
OpenML both lacks functionality we seek to provide (subgroup robustness and
domain shift utilities; a curated set of benchmarking datasets; lightweight
and standardized control over common tabular preprocessing methods) and also
provides extraneous functionality not needed for a lightweight tabular
benchmarking library (tools for OpenML-hosted model/pipeline/evaluation
sharing and collaboration; API/utilities for model training) . Additionally,
OpenML is not yet widely used in the tabular data community, as demonstrated
by the wide calls for effective tabular data benchmarking tools ([15], [37],
[78]) and the lack of usage of OpenML in most robustness works, even recent
works (e.g. [91]), which largely focus on canonical tabular datasets such as
COMPAS and Adult.

DataPerf: DataPerf is “a benchmark package for evaluating ML datasets and
dataset working algorithms” [59]. Similar to WILDS and DomainBed, DataPerf
covers many domains, not only tabular data. DataPerf has a much broader set of
goals relative to TableShift, and includes a collection of tasks, metrics and
rules that are intended to benchmark all stages of an ML pipeline, from raw
data to test set selection amd model selection. However, DataPerf does _not_
offer domain shifts, and while it is possible domain shifts could be
integrated into DataPerf, it does not natively support the kind of
benchmarking that we intend to support with TableShift.

###  G.3 Other Data Hosting Platforms

Hugging Face Datasets (HFDS): HFDS is a generic dataset hosting utility
provided by the company Hugging Face. It serves as a large, open dataset
repository; however, these datasets are not curated for size, featurization,
or quality. HFDS is a public platform where datasets can be contributed
openly. However, of the tabular datasets on HFDS, few if any meet the
specifications described in §3.1; in particular, most are not domain shift
datasets.

TensorFlow Datasets (TFDS): TFDS is similar in many ways to HFDS. It is a
public, open repository of datasets, and new datasets can be contributed via
git. However, TFDS also has the same shortcomings as HFDS; in particular, its
open format leads to a collection of datasets mostly not useful for tabular
data benchmarking and almost no datasets with meaningful distribution shifts.

Table 18: Comparison of relevant benchmarks. DB: DomainBed. OML: OpenML. GOV22: [37]. SH: Shift Happens. HFDS: Hugging Face Datasets. TFDS: TensorFlow Datasets. |  | DB | OML | GOV22 | WILDS | SH | Shifts 2.0 | HFDS | TFDS | TableShift
---|---|---|---|---|---|---|---|---|---|---
Output |  Supports tabular data input formats (e.g. .csv) |  | ✓ |  |  |  | ✓ | ✓ | ✓ | ✓
|  Supports tabular output formats (e.g. pd.DataFrame) |  |  |  |  |  | ✓ | ✓ | ✓ | ✓
|  Support for large/out-of-core datasets | ✓ |  |  | ✓ | ✓ |  | ✓ | ✓ | ✓
Preprocessing |  Supports tabular preprocessing: categorical encoding; missing value handling |  |  |  |  |  |  |  |  | ✓
|  Provides shared utilities for user-defined preprocessing per dataset | ✓ |  |  | ✓ | ✓ |  | ✓ | ✓ | ✓
Metadata |  Feature-level metadata |  |  |  |  |  |  |  |  | ✓
|  Dataset-level metadata |  |  |  |  |  |  | ✓ | ✓ | ✓
Benchmark
Tasks |  Includes domain shift (non-IID) splits | ✓ |  |  | ✓ | ✓ | ✓ | some |  | ✓
|  Meets criteria in §3.1 | ✓ |  |  | ✓ |  |  |  |  | ✓
|  Large test sets ($\geq 10k$) | ✓ |  |  |  | ✓ |  |  |  |
|  Includes label-imbalanced datasets | ✓ | ✓ |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
|  Includes real-world sensitive attributes |  | some |  | ✓ |  |  | some | some | all

##  Appendix H Training Details

Neural network-based models were trained on GPU, either NVIDIA RTX 2080 Ti
GPUs with 11GB of RAM, or NVIDIA Tesla M60 GPUs with 48 GB of RAM. We used a
batch size of 4096 for training all models, except where this was not possible
due to memory limitations (see code for details).

Where possible, gradient boosted tree models were trained using CPU (not GPU).

##  Appendix I Hyperparameter Grids

The hyperparameter tuning grid for each model is shown in Table 19. We make
the full hyperparameter tuning code available as part of the release of this
work, at https://github.com/mlfoundations/tableshift.

We made an effort to ensure our hyperparameter grids always included at least
the full grid described in the original work(s) cited for each learning method
used in our study. For some methods, our grid is a superset of the
hyperparameter grid in the original study. This is to ensure, where possible,
that we tune a similar range of certain parameters (i.e. learning rate) across
all methods. For domain generalization methods, since we are not aware of any
prior application to these methods to tabular data, we use the hyperparameter
grids from [38].

Table 19: Hyperparameter grids used in all experiments. $\clubsuit$: all MLP
parameters also tuned. Continued in Table 20.

Model Hyperparameter Values lightgrayBaseline Methods $\clubsuit$ MLP Learning
Rate LogUniform$(1e^{-5},1e^{-1})$ Weight Decay LogUniform$(1e^{-6},1)$ Num.
Layers $\\{1,2,\ldots,8\\}$ Hidden Units $\\{64,128,256,512,1024\\}$ Num
Epochs QRandInt$(5,100,5)$ Dropout Unif$(0,0.5)$ Batch Size $\\{4096\\}$
XGBoost Learning Rate LogUniform$\\{1e-5,1\\}$ Max. Depth $\\{3,\ldots,10\\}$
Min Child Weight LogUniform$\\{1e-8,1e5\\}$ Row Subsample
Uniform$\\{0.5,1.\\}$ Column Subsample (Tree) Uniform$\\{0.5,1.\\}$ Column
Subsample (Level) Uniform$\\{0.5,1.\\}$ $\gamma$ LogUniform$\\{1e-8,1e2\\}$
$\lambda$ LogUniform$\\{1e-8,1e2\\}$ $\alpha$ LogUniform$\\{1e-8,1e2\\}$ Max.
Bins $\\{128,256,512\\}$ LightGBM Learning Rate LogUniform$\\{1e-5,1\\}$ Min.
Child Samples $\\{1,2,4,8,16,32,64\\}$ Min. Child Weight LogUniform
$(1e-8,1e5)$ Row Subsample Uniform$\\{0.5,1.\\}$ Max. Depth
$\\{\textrm{None},1,2,\ldots,31\\}$ Column Subsample (Tree)
Uniform$\\{0.5,1.\\}$ Column Subsample (Level) Uniform$\\{0.5,1.\\}$ $\lambda$
LogUniform$\\{1e-8,1e2\\}$ $\alpha$ LogUniform$\\{1e-8,1e2\\}$ CatBoost
Learning Rate LogUniform$\\{1e-3,1\\}$ Depth QRandInt$\\{3,10\\}$ Bagging
Temp. LogUniform $(1e-6,1)$ $L_{2}$ Leaf Reg. LogUniform $(1,100)$ Leaf
Estimation Iterations QRandInt$\\{1,10\\}$ lightgrayDomain Generalization
Methods DANN $\clubsuit$ $\textrm{LR}_{G}$ LogUniform$\\{1e-5,1e-1\\}$
$\textrm{WeightDecay}_{G}$ LogUniform$\\{1e-6,1\\}$ $\textrm{LR}_{d}$
LogUniform$\\{1e-5,1e-1\\}$ $\textrm{WeightDecay}_{D}$
LogUniform$\\{1e-6,1\\}$ D steps per G step LogUniform$\\{2,2^{3}\\}$ Grad
Penalty LogUniform$\\{1e-2,1e1\\}$ Loss $\lambda$ LogUniform$\\{1e-2,1e2\\}$
IRM $\clubsuit$ IRM $\lambda$ LogUniform$\\{1e-1,1e5\\}$ IRM Penalty Anneal
Iters LogUniform$\\{1,1e4\\}$ MixUp $\clubsuit$ MixUp $\alpha$
Uniform$\\{1e-1,1e1\\}$ VReX $\clubsuit$ VReX $\lambda$
LogUniform$\\{1e-1,1e5\\}$ VReX Penalty Anneal Iters LogUniform$\\{1,1e4\\}$
CORAL $\clubsuit$ MMD $\gamma$ LogUniform$\\{1e-1,1e1\\}$ MMD $\clubsuit$ MMD
$\gamma$ Uniform$\\{1e-1,1e1\\}$

Model Hyperparameter Values lightgrayTabular Neural Networks ResNet
$\clubsuit$ Num. Blocks $\\{1,2,3,4\\}$ $d$ main RandInt$(64,1024)$ Hidden
Factor RandInt$(1,4)$ Dropout First Uniform$(0,0.5)$ Dropout Second
Uniform$(0,0.5)$ FT-Transformer $\clubsuit$ Num. Blocks $\\{1,2,3,4\\}$
Residual Dropout Uniform$(0,0.2)$ Attention Dropout Uniform$(0,0.5)$ FFN
Dropout Uniform$(0,0.5)$ FFN Factor Uniform$(2/3,8/3)$ FFN Factor
$\\{64,128,256,512\\}$ TabTransformer Num. Blocks $\\{1,2,3,4\\}$ Learning
Rate LogUniform$(1e^{-5},1e^{-1})$ Weight Decay LogUniform$(1e^{-6},1)$ Num
Epochs QRandInt$(5,100,5)$ FFN Dropout Uniform$(0,0.5)$ Attention Dropout
Uniform$(0,0.5)$ Model Dimension ${32,64,128,256}$ Depth ${3,4,5,6}$ Num.
Heads ${2,4,8}$ NODE Num. Epochs $\\{1,2,3,4,5\\}$ Num. Layers ${2,4,8}$ Total
Tree Count ${1024,2048}$ Tree Depth ${6,8}$ Tree Output Dim. ${2,3}$ FFN
Factor $\\{64,128,256,512\\}$ Learning Rate LogUniform$(1e^{-5},1e^{-1})$
Weight Decay LogUniform$(1e^{-6},1)$ SAINT Num. Epochs $\\{1,2,3,4,5\\}$ Depth
${4,6}$ Model Dimension ${4,8,12,16,32}$ Learning Rate
LogUniform$(1e^{-5},1e^{-1})$ Weight Decay LogUniform$(1e^{-6},1)$ FFN Dropout
Uniform$(0.1,0.8)$ Heads ${4,8}$ Attention Type Row, Col, RowCol
lightgrayDomain Robustness Methods DRO $\clubsuit$ Uncertainty set size
LogUniform$\\{1e-4,1.\\}$ Geometry $\\{$ CVaR, $\chi^{2}\\}$ Group DRO
$\clubsuit$ Group weights step size LogUniform$(1e^{-4},1\\}$ lightgrayLabel
Shift Robustness Methods Label Group DRO $\clubsuit$ Group weights step size
LogUniform$(1e^{-4},1\\}$ Adversarial Label DRO $\clubsuit$ Adv. Learning Rate
$\eta_{\pi}$ LogUniform$\\{1e-4,1e-1\\}$ Adv. radius $r$
LogUniform$\\{1e-5,0.5\\}$ Clip max $r$ LogUniform$\\{1e-1,10\\}$ $\epsilon$
LogUniform$\\{1e-4,1e-1\\}$

Table 20: Hyperparameter grids used in all experiments. $\clubsuit$: all MLP
parameters also tuned. Continued from Table 19.