# Abelian Varieties are not quotients of low-dimension Jacobians

Jacob Tsimerman

###### Abstract

We prove that for any two integers $g\geq 4$ and $g^{\prime}\leq 2g-1$, there
exist abelian varieties over $\overline{\mathbb{Q}}$ which are not quotients
of a Jacobian of dimension $g^{\prime}$. Our method in fact proves that most
abelian varieties satisfy this property, when counting by height relative to a
fixed finite map to projective space.

##  1 Introduction

Jacobians are a subset of abelian varieties, and much work has been devoted to
figuring out ‘how large’ that subset is. An elementary computation shows that
for $g\geq 4$ the generic abelian varieties is not a Jacobian. Over
$\mathbb{C}$, this immediately implies that the generic abelian variety is not
isogenous to a Jacobian, either. This can be seen in many ways, and
essentially reduces to the fact that $\mathbb{C}$ is an uncountable field, and
therefore the union of countably many subvarieties cannot be equal to an
ambient variety of higher dimension.

Katz and Oort conjectured that the same property holds over
$\overline{\mathbb{Q}}$. Over a countable field this problem becomes much more
difficult, and was solved unconditionally in [COJacobians, tsimjacobians] by
relying on abelian varieties with complex multiplication, and appealing to the
André-Oort conjecture. Later, a different proof was given in [MZjacobians], by
using the Pila-Zannier method and isogeny estimates of Masser-
Wüstholz[MWisogeny].

A related fact in the positive direction is that every abelian variety $A$ is
a quotient of a Jacobian. Indeed, one may find a smooth curve inside of $A$
not contained in a coset of a proper abelian subvariety and consider its
Jacobian. One may ask for the lowest such number:

######  Question 1.1.

Let $g\geq 4$, and let $A$ be a generic abelian variety over an algebraically
closed field $k$. What is the lowest integer $g^{\prime}$ such that there
exists a Jacobian $J$ of dimension $g^{\prime}$ and a surjection
$\phi:J\rightarrow A$?

Question 1.1 is extremely difficult even for $k=\mathbb{C}$. Comparing
dimensions shows that $g^{\prime}\geq\frac{g^{2}+g}{6}+1$, but this is far
from optimal. In [pirolaabelian] it is proven that
$g^{\prime}\geq\frac{g(g-1)}{2}+1$, with equality iff $g\in\\{4,5\\}$. For an
upper bound there does not appear to be a better method known than taking a
projective embedding and intersecting with hyperplanes to produce a curve,
which gives an upper bound which looks like a power of $g!$. There is
therefore a huge chasm and to the author’s knowledge there is not even a
conjecture as to what the truth should be. Relatedly, see work of
Voisin[voisinchowring] and Pirola[pirolahyperelliptic] on giving a lower bound
for the _gonality_ of a curve in a generic abelian variety.

The goal of this paper is to make progress on question 1.1 for
$k=\overline{\mathbb{Q}}$:

######  Theorem 1.

Let $g\geq 4$. Then there exist abelian varieties over $\overline{\mathbb{Q}}$
which are not quotients of a Jacobian of a curve of genus $\leq 2g-1$.

In fact, for our proof there is nothing special about the Torelli locus. We
prove the following Theorem, where $\mathcal{A}_{g}$ is the moduli space of
Principally Polarized abelian varieties:

######  Theorem 2.

Let $0\leq h<g$ be integers. Suppose that $V\subset\mathcal{A}_{g+h}$ is a
subvariety such that over $\mathbb{C}$, the generic $g$-dimensional abelian
variety is not a quotient of an abelian variety parameterized by $V$.

Then there exist $g$-dimensional abelian varieties over
$\overline{\mathbb{Q}}$ which are not a quotient of an abelian variety
parameterized by $V$.

In fact we shall show in Theorem 8 that in a precise sense (analogous to what
Masser, Zannier[MZjacobians] and Zywina [Zywina] do) most abelian varieties
are not quotients of abelian varieties parameterized by $V$.

###  1.1 Summary of the Proof

Our proof follows the Pila-Zannier method, as employed by Masser and Zannier.
In fact, in the case of $h=0$ our proof essentially specializes to theirs111We
find it convenient to use slightly different counting functions (with
rationals instead of integers more in line with [Zywina]), but the essence of
our argument reduces to exactly what they do.. The main ideas are as follows:

  1. 1.

The main difficulty - and the main technical contribution of this paper - is
to demonstrate ‘Large Galois Orbits’. Concretely, suppose that
$A\in\mathcal{A}_{g}(K)$ is an abelian variety defined over a number field $K$
and $B\in V(\overline{\mathbb{Q}})$ such that $A$ is a quotient of $B$. Then
there is a degree $N$ isogeny $\phi$ from $A\times C$ to $B$ for some
$h$-dimensional abelian variety $C$. We wish to show that if there is one such
$B$ then there are many - presuming $A$ is generic in some way (see Lemma 3.1
for the specifics). In particular, we wish to show that $B$ has a large field
of definition so that we may consider all of the Galois conjugates of $B$.

The difficulty is that one has no a-priori control over either the height or
field of definition of $C$. We get around this by using the fact that $g>h$,
and specifically that there is no embedding $\operatorname{GSp}_{2g}$ into
$\operatorname{GSp}_{2h}$. Morally, this means that the Galois action on the
Tate module of $C$ cannot ‘interact’ with the action on the Tate module of
$A$. Concretely, we let $K\subset A\times C$ be the kernel of $\phi$. Then we
show (after some reductions) that for generic $A$, we have $K_{A}:=K\cap
A\times\\{0_{C}\\}$ is large compared to $N$. This means that the abelian sub-
variety in $B$ which is isogenous to $A$ has large field of definition, and
hence $B$ must as well.

  2. 2.

Armed with this we use the Pila-Wilkie Theorem to obtain an unlikely
intersection in $V$. We then use powerful transcendence theorems (Ax-Schanuel
and Geometric Zilber-Pink, see §2.5) to reduce to the case where $V$ is
contained in a weakly special subvariety.

  3. 3.

The last difficult piece is handling the case where $V$ is contained in a
weakly special subvariety. We handle this in a somewhat unsatisfying (but very
clean) ad-hoc method. Specifically, we show that if a weakly special
subvariety contains a non-simple point with a Hodge-generic factor, then the
generic point over it is non-simple222Note that this is false without the
Hodge-Generic assumption. For example, consider quaternionic curves in
$\mathcal{A}_{2}$, which contain non-simple CM points.. This allows us to
carry out an induction argument to finish.

In actuality we reverse steps 2 and 3, but the above outline is the heart of
the proof. We remark on the limitation of $h<g$ for our method:

The reason that this is a problem is the step where we establish large Galois
orbits. Specifically, the issue is a possible counterexample to Frey-Mazur:
Let $A$ be a $g$-dimensional abelian variety and suppose that $C$ is another
one such that $A[p]\cong C[p]$ as Galois modules. Then we may quotient out
$A\times C$ by the graph of this isomorphism to obtain a $2g$-dimensional
abelian variety defined over $\mathbb{Q}$333To get the polarizations working
we should insist $p\equiv 1\mod 4$ and add a quadratic twist so that the graph
is a Lagrangian, but this is a minor technical point.. We are unable to rule
this out from happening for arbitrarily large primes, and hence we cannot get
a handle on $g=h$. It seems to be a very difficult problem to rule such a
thing out, even for a single $A$!

If one were able to circumvent this issue and establish large Galois orbits,
then the rest of the proof would go through unchanged.

###  1.2 Outline of the Paper

In §2 we recall background about abelian varieties, the corresponding Galois
representations, functional transcendence, and the Pila-Wilkie method. In §3
we prove our ‘large Galois orbits’ result. In §4 we prove a result about
weakly special subvarieties which enables us to reduce to the case of $V$ not
being contained in a proper weakly special. Finally, in §5 we combine
everything to prove the main theorem.

###  1.3 Acknowledgements

It is a pleasure to thank Jonathan Pila and Ananth Shankar for enlightening
discussions on this problem.

##  2 Background

###  2.1 Polynomial boundedness

We write $A\prec_{C}B$ or $B\succ_{C}A$ or non-negative functions $A,B$
depending on $C$ and possibly other variables, if there are constants
$c_{1},c_{2},c_{3}>0$ which depend only on $C$, such that
$A<c_{1}B^{c_{2}}+c_{3}$. If $A\prec_{g}B$ and $B\prec_{g}A$ we write
$B\sim_{g}A$. If $A\prec_{g}B$ we say that $A$ is _polynomially bounded by
$B$_.

###  2.2 Abelian Varieties

We fix some notation for abelian varieties:

  * •

A _quasi-morphism_ between abelian varieties $A,B$ is an element of
$\operatorname{Hom}(A,B)\otimes\mathbb{Q}$.

  * •

A _quasi-isogeny_ is a quasi-morphism $\phi$ such that for any integer $n$ for
which $n\phi$ is a morphism, $n\phi$ has finite kernel.

  * •

For a complex abelian variety $\mathbb{C}$, we denote
$L_{A}:=H_{1}(A,\mathbb{Z}),V_{A}:=H_{1}(A,\mathbb{Q}),T_{A}:=V_{A}/L_{A}$.
Note that $T_{A}\cong A_{\operatorname{tor}}(\mathbb{C}).$

  * •

We denote the Tate module by $T_{A,\ell}:=\varinjlim A[\ell^{n}].$

###  2.3 Principal Polarisations and Isogenies

Recall that a polarization of a complex abelian variety $A$ is an isogeny
$\phi:A\rightarrow A^{\vee}$ which is symmetric, and for which the pullback of
the Poincare bundle along the associated graph is ample. Given a polarization,
we get an associated symplectic pairing $Q_{A,\phi}$ known as the Weil pairing
on $L_{A}$, and also $Q_{A,\phi,n}$ on $A[n]$ for each positive integer $n$.
We say that $\phi$ is principal if the degree of $\phi$ is $1$.

We say that two polarizations $\phi,\psi$ are _equivalent_ if there are
positive integers $m,n$ such that $m\phi=n\psi$. Given another abelian variety
$B$ and an isogeny $f:B\rightarrow A$ we define a polarization on $B$ by
setting $f^{*}\phi:=f^{\vee}\circ\phi\circ f$. Given two polarized abelian
varieties $(A,\phi),(B,\psi)$ we say that an isogeny $f:A\rightarrow B$ is a
_polarized isogeny_ from $(A,\phi)$ to $(B,\psi)$ if $f^{*}\psi$ is equivalent
to $\phi$.

If $\phi,\phi^{\prime}$ are two polarizations on $A$ then we obtain a quasi-
isogeny $f:=\phi^{-1}\circ\phi^{\prime}:A\rightarrow A$ such that the induced
map on $V_{A}$ respects the pairing $Q_{A,\phi}$. In particular,

$Q_{A,\phi}(v_{1},fv_{2})=Q_{A,\phi}(fv_{2},v_{1}).$

###  2.4 Shimura Varieties

See [milnemoduli] for further background on this section.

Let $G$ be a reductive group over $\mathbb{Q}$,
$\mathbb{S}=\operatorname{Res}_{\mathbb{C}/\mathbb{R}}\mathbb{G}_{m}$ the
Deligne torus, and $X$ be a conjugacy class of homomorphisms
$f:\mathbb{S}\rightarrow G_{\mathbb{R}}$ satisfying the Shimura axioms, and
$K=\prod_{p}K_{p}\subset G(\mathbb{A}_{f})$ be a neat compact subgroup. Let
$E(G,X)$ be the reflex field, and $E$ be a field over which $G$ splits.
Associated to this data we get an algebraic variety $S_{K}(G,X)$ defined over
$E(G,X)$ called a "Shimura Variety" whose complex points can be identified
with $G(\mathbb{Q})\backslash\left(X\times G(\mathbb{A}_{f})\right)/K$.

We shall primarily be concerned with $\mathcal{A}_{g}$ which is the Shimura
variety corresponding to $G=\operatorname{GSp}_{2g}$. This naturally gives
$\mathcal{A}_{g}$ the structure of the (coarse) moduli space of principally
polarized abelian varieties of dimension $g$. In this case $E(G,X)=\mathbb{Q}$
so we consider $\mathcal{A}_{g}$ as a $\mathbb{Q}$-variety.

####  2.4.1 Weakly Special Varieties

We follow [UY] in this section:

Suppose $S_{K}(G,X)$ is a Shimura variety for which we have:

  * •

A Shimura subvariety $S_{K_{H}}(H,X_{H})$

  * •

Shimura varieties $(H_{i},X_{i},K_{i})$ for $i=1,2$ and a splitting
$H^{\operatorname{ad}}\cong H_{1}\times H_{2}$ inducing a splitting
$X_{H}\cong X_{H_{2}}\times X_{H_{2}}$.

Then we say that the image of $X_{1}^{+}\times\\{y\\}$ is a _weakly special_
subvariety of $S_{K}(G,X)$ for any connected component $X_{1}^{+}$ of $X_{1}$
and any point $y\in X_{2}$. We say that a closed analytic subvariety $S\subset
X_{G}$ is weakly special if it is a connected component of the preimage of a
weakly-special variety.

###  2.5 Functional Transcendence Theorems

We let $\mathcal{H}_{g}:=\\{Z\in M_{g}(\mathbb{C}):Z=Z^{t},\mathrm{im}Z>0\\}$
denote the Siegel upper-half space, so that there is a natural uniformization
map $\pi_{g}:\mathcal{H}_{g}\rightarrow\mathcal{A}_{g}(\mathbb{C})$. Then we
have a semi-algebraic structure on $\mathcal{H}_{g}$ induced from the
algebraic structure of $M_{g}(\mathbb{C})$. It is easy to show that weakly
special varieties $V$ are bi-algebraic for the map $\pi_{g}$, in the sense
that all of the irreducible components of $\pi_{g}^{-1}(V)$ are semi-
algebraic. The Ax-Schanuel Theorem [ASAg] shows that this is the only
algebraic information that is preserved by $\pi_{g}$:

######  Theorem 3.

Let $V\subset\mathcal{H}_{g}\times\mathcal{A}_{g}$ be an irreducible algebraic
variety, and let $\Gamma\subset\mathcal{H}_{g}\times\mathcal{A}_{g}$ be the
graph of the map $\pi_{g}$. Let $U\subset V\cap\Gamma$ be an irreducible
component and assume that $\dim U>\dim V+\dim\Gamma-\dim\mathcal{A}_{g}$. Then
$\pi_{g}(U)$ is contained in a proper weakly-special subvariety of
$\mathcal{A}_{g}$.

We shall also require another theorem, which takes into account unlikely
intersections with different weakly-special varieties at once. Namely, let
$V\subset\mathcal{A}_{g}$ be a variety, and we assume it is not contained in
any proper weakly-special subvariety. We say that an irreducible subvariety
$X\subset V$ is _weakly-atypical_ if there is a weakly special subvariety
$W\subset\mathcal{H}_{g}$ containing $X$such that

$\dim X>\dim W+\dim V-\dim\mathcal{A}_{g}.$

We then have the following:

######  Theorem 4.

$V\subset\mathcal{A}_{g}$ be a variety which is not contained in any proper
weakly-special subvariety. Then there is a proper subvariety $Z\subset V$
which contains all positive-dimensional weakly-atypical subvarieties of $V$,
such that for any irreducible component $Z^{o}\subset Z$ we have

  1. 1.

$Z^{o}$ is weakly atypical, or

  2. 2.

$Z^{o}$ is contained in a proper special subvariety of $\mathcal{A}_{g}$.

###### Proof.

Note that $V$ itself is not weakly atypical, by our assumptions. The result
now follows directly from [baldiklinglerullmo, Thm 6.1] by noting that the
adjoint group of $\operatorname{GSp}_{2g}$ is simple, and therefore for case
(ii) in Theorem 6.1 of that paper to occur $Z^{o}$ must be contained in a
proper special subvariety. ∎

The above is named the Geometric Zilber-Pink conjecture for the weakly special
atypical locus by [baldiklinglerullmo], and is proven there in the more
general context of hodge variations.

###  2.6 Pila-Wilkie Theorem

Recall that the height of a rational point $\frac{a}{b}$ with $\gcd(a,b)=1$ is
$H(\frac{a}{b}):=\max(|a|,|b|)$. For a point
$q=(q_{1},\dots,q_{n})\in\mathbb{Q}^{n}$ we define $H(q):=\max_{i}H(q_{i})$.
Finally, for any subset $X\subset\mathbb{R}^{n}$ we define

$N(X,T):=\\#\\{q\in X\cap\mathbb{Q}^{n}\mid H(q)\leq T\\}.$

Given a set $X\subset\mathbb{R}^{n}$, we set $X^{{\mathrm{alg}}}$ to be the
union of all connected, positive dimensional, semi-algebraic subsets of $X$.
Then we have the following Theorem of Pila–Wilkie:

######  Theorem 5 (Pila–Wilkie [PW, Thm 1.8]).

For a set $X\subset\mathbb{R}^{n}$ definable in an o-minimal structure, and
any $\epsilon>0$, we have

$N(X-X^{{\mathrm{alg}}},T)=T^{o(1)}.$

In fact, we shall need the slightly stronger version, valid in families.

######  Theorem 6 (Pila–Wilkie [PAOmodular, Thm 3.6]).

For a set $X\times Y\subset\mathbb{R}^{n}$ definable in an o-minimal
structure, we have that

$N(X_{y}-X_{y}^{{\mathrm{alg}}},T)=T^{o(1)}$

uniformly in $y\in Y$.

We shall only use the o-minimal structure $\mathbb{R}_{an,\exp}$ in which all
the sets we work with are defined. For background on o-minimality see [Dries].

###  2.7 Height Estimates

We use [PTsurfaces] for background.

We work in the Siegel Upper-Half space $\mathbb{H}_{g}$ of symmetric $g\times
g$ complex matrices with positive definite imaginary component.

for a matrix $Z=X+iY\in\mathbb{H}_{g}$ we define its height to be

$H(Z):=\max\big{(}|z_{i,j},\frac{1}{|Y|}\big{)}.$

We set $\mathcal{F}_{g}\subset\mathbb{H}_{g}$ to be the standard fundamental
domain[siegelsympleticbook]. The following is [PTsurfaces, Lemma 3.2]

######  Lemma 2.1.

Let $Z\in\mathbb{H}_{g}$ and $\gamma\in\operatorname{GSp}_{2g}(\mathbb{Z})$
such that $\gamma Z\in\mathcal{F}_{g}$. Then the co-ordinates of $\gamma$ and
the Height of $\gamma Z$ are $\prec_{g}H(Z)$.

As a result, we have the following simple consequence:

######  Corollary 2.2.

Let $Z,W\in\mathcal{F}_{g}$ be such that there is a degree $N$ polarized-
isogeny $\phi:A_{Z}\rightarrow A_{W}$. Then $H(W)\prec_{g}H(Z)+N$. Moreover,
there is an element $M\in\operatorname{GSp}_{2g}(\mathbb{Q})$ such that $MZ=W$
and $H(M)\prec_{g}N$.

###### Proof.

The existence of $\phi$ means there is an element
$M\in\operatorname{GSp}_{2g}(\mathbb{Q})\cap M_{2g}(\mathbb{Z})$ with
determinant $N$ such that $MZ=W$. It is well-known that these break up into
finitely many right $\operatorname{GSp}_{2g}(\mathbb{Z})$ cosets, and we may
write $M=\gamma\gamma_{0}$ for $\gamma\in\operatorname{GSp}_{2g}(\mathbb{Z})$
and $\gamma_{0}$ having entries bounded by $N^{O_{g}(1)}$. The bound on
$\gamma_{0}$ follows for $N$ a prime power by the explicit coset
representatives computed in [explicitheckecosets], and in general by
multiplying and using that $(2g)^{\omega(N)}=N^{o_{g}(1)}$. This implies that
$H(\gamma_{0}Z)\prec(H(Z)+N)$, and so the bound on $H(W)$ follows by Lemma
2.1.

Finally it remains to prove the bound on $H(M)$. This follows directly from
[orr, Thm 1.1] for the group $G=\operatorname{GSp}_{2g}$ with its canonical
representation.

∎

###  2.8 Galois Representations

Let $A$ be an abelian variety over a number field $K$. We get an induced
Galois representation
$\rho_{A,\ell}:G_{K}\rightarrow\operatorname{GL}(T_{A,\ell})$ for each prime
$\ell$, and therefore $\rho_{A}:G_{K}\rightarrow\operatorname{GL}(T_{A})$.
This must preserve every polarization and endomorphism defined over $K$. For
each $\ell$, we obtain an $\ell-adic$ Mumford-Tate Group $MT_{A,\ell}$ by
taking the Zariski closure of the image of $\rho_{A,\ell}$. Note that the
$\ell$-adic Mumford-Tate group depends on the number field but its identity
component (as an algebraic group) does not.

Given a polarized abelian variety $(A,\phi)$, we say that $A$ is _Galois
Generic_ if the image of $\rho_{A}$ is open in
$\operatorname{GSp}(T_{A},\phi)$. We remark that this is independent of $\phi$
since this property implies $\operatorname{End}(A)=\mathbb{Z}$ and thus that
$A$ has a single polarization up to equivalency.

We have the following result (see [BLAV, Remark17.3.1])

######  Proposition 2.3.

The identity component of $MT_{A,\ell}$ is contained in the extension of
scalars of the ordinary Mumford-Tate group $MT_{A}$ from $\mathbb{Q}$ to
$\mathbb{Q}_{\ell}$.

###  2.9 Counting by Height

The proof of our Theorem requires a notion of ‘most’ abelian varieties, so we
need a way to count a subset of points $\mathcal{A}_{g}$ which are infinite,
easy to get at, and have generic behaviour. To do this we borrow the mechanism
used by Masser and Zannier[MZjacobians]. We fix an open set
$U\subset\mathcal{A}_{g}$ and a quasi-finite dominant map
$\phi:U\rightarrow\mathbb{P}^{\dim\mathcal{A}_{g}}$ defined over $\mathbb{Q}$.
We may then count $\phi$-rational points by defining

$S_{\phi}(T):=\\{t\in\mathcal{A}_{g}(\overline{\mathbb{Q}})\mid\phi(t)\in
U(\mathbb{Q}),H(\phi(t))\leq T\\}$

for each $T>0$, where $H$ is the usual height function on projective space.
Note that the abelian varieties in $S_{\phi}(T)$ are defined over number
fields of bounded degree.

We call such a $\phi$ a _counting function_ on $\mathcal{A}_{g}$, and we say
that a property holds _$\phi$ -generically_ on $\mathcal{A}_{g}$ if the
proportion of points in $S_{\phi}(T)$ that it holds for tends to 1 as
$T\rightarrow\infty$.

##  3 Galois Orbits

Let $A$ be an abelian variety of dimension $g$ defined over a number field
$K$. Let $\rho_{A}:G_{K}\rightarrow\operatorname{GL}_{2g}(\hat{\mathbb{Z}})$
be the (isomorphism class of the ) Galois representation associated to an
abelian variety. Of course, if it is principally polarized then the image is
contained in $\operatorname{GSp}_{2g}(\hat{\mathbb{Z}})$. If the image of
$\rho_{A}$ has finite index in $\operatorname{GSp}_{2g}(\hat{\mathbb{Z}})$
then we call its index the _index of $A$_. Note that this implies that $A$ is
simple.

We recall the following theorem:

######  Proposition 3.1.

[Zywina, Theorem 1.1]

Fix a counting function $\phi$ as in 2.9. Then $\phi$-generically, for an
abelian variety $A\in\mathcal{A}_{g}$ we have that the index
$[\operatorname{GSp}_{2g}(\hat{\mathbb{Z}}):\mathrm{im}\rho_{A}]$ is finite of
size $O_{g}(1)$. It follows in particular that $A$ is Hodge-generic.

###### Proof.

This follows from Theorem 1.1 of [Zywina] in exactly the same way as Theorem
1.2 of [Zywina] does, by applying Theorem 1.1 to the restriction of scalars
after passing to a Galois cover.

∎

The purpose of this sections is to prove a large Galois orbits result:

######  Theorem 7.

Let $A$ be a ppav of dimension $g$ over number field $K$ of degree $O_{g}(1)$
over $\mathbb{Q}$ such that
$[\operatorname{GSp}_{2g}(\hat{\mathbb{Z}}):\mathrm{im}\rho_{A}]=O_{g}(1)$.
Fix $0\leq h<g$. Let $B$ be a $\overline{\mathbb{Q}}$ ppav of dimension $g+h$
such that the lowest degree isogeny to $B$ from an abelian variety of the form
$A\times C$, for a ppav $C$ of dimension $h$, is $N$.

Then the degree of the field of definition of $B$ is $\succ_{g}N$. Moreover,
$B$ has a polarized isogeny from $A\times C^{\prime}$ for a (possibly
different) ppav $C^{\prime}$ of degree $\prec_{g}N$.

###### Proof.

Note that since $g>h$ and $A$ is simple that
$\operatorname{Hom}(A,C)=\operatorname{Hom}(C,A)=0$, and thus
$\operatorname{End}(A\times C)\cong\mathbb{Z}\oplus\operatorname{End}(C)$. We
thus record elements of $\operatorname{End}(A\times C)$ as pairs $(m,r)$ for
$r\in\operatorname{End}(C)$.

We let $L_{B}\subset V_{A}\times V_{C}$ be the Lattice corresponding to
$H_{1}(B,\mathbb{Z})$ under the isogeny to $B$. We identify $V_{B}$ with
$V_{A}\times V_{C}$. Note that $[L_{B}:L_{A}\times L_{C}]=N$. Now there is a
natural form symplectic form $Q$ on $V_{A}\times V_{C}$ corresponding to the
principal polarizations $\phi_{A},\phi_{C}$ on $A,C$, and this gives us an
isomorphism $B^{\vee}(\mathbb{C})\cong V_{A,C}/L_{B}^{*}$ where $L_{B}^{*}$
denotes the dual of $L_{B}$ under $Q$. Corresponding to the principal
polarization $\phi_{B}$ of $B$, there is therefore an endomorphism $f=(m,r)\in
End(A\times C)$ such that $fL_{B}=L_{B}^{*}$, and by 2.3 we have that $f$
respects $Q$.

Step 1: Proving $m\succ_{g}N$

Note first that $m^{g}(\deg r)^{\frac{1}{2}}=N.$ We have the lattices

$\pi_{A}(L_{B})\supset L_{B}\cap V_{A}\supset L_{A},\pi_{C}(L_{B})\supset
L_{B}\cap V_{C}\supset L_{C}$

and by duality of $L_{B}$ and $fL_{B}$, we have the relations

$m\pi_{A}(L_{B})=(V_{A}\cap L_{B})^{*},r(V_{C}\cap
L_{B})=\pi_{C}(L_{B})^{*},r\pi_{C}(L_{B})=(V_{C}\cap L_{B})^{*}$

.

Note that $\pi_{C}(L_{B})/(V_{C}\cap L_{B})\cong\pi_{A}(L_{B})/(V_{A}\cap
L_{B})$ and thus is of size $d\leq m^{2\dim A}$. Dualizing we get

$d=[(V_{C}\cap L_{B})^{*}:\pi_{C}(L_{B})^{*}]=[(V_{C}\cap
L_{B})^{*}:r(V_{C}\cap L_{B})]$

Let $C^{\prime}:=V_{C}/(V_{C}\cap L_{B})$. Then $C^{\prime\vee}\cong
V_{C}/\pi_{C}(V_{C}\cap L_{B})^{*}$ and $r:C^{\prime}\rightarrow
C^{\prime\vee}$ is a polarization on $C^{\prime}$ of degree $d$. As a result
we may find an polarized isogeny $C^{\prime\prime}\rightarrow C^{\prime}$ of
degree $\leq d$ where $C^{\prime\prime}$ is principally polarized. Then the
natural isogeny $A\times C^{\prime\prime}\rightarrow B$ is of degree at most
$dm^{2g}\leq m^{4g}$. By virtue of the minimality of $N$, we conclude that
$\deg r\leq m^{6g}$ and thus that $m\geq N^{\frac{1}{4g}}$.

Step 2: Getting a polarized isogeny

Replacing $C$ by the $C^{\prime}$ above (at the cost of replacing $N$ by
$N^{\prime}\prec_{g}N$) we may assume that $r$ is multiplication by some
positive integer $m^{\prime}$.

We are now in the setup where $m\prec_{g}N$ and $r$ is multiplication by a
positive integer $m^{\prime}$. We now observe that we may replace
$(C,\phi_{C})$ by $(C^{\prime},\phi_{C}\cdot\frac{1}{k})$ by changing $L_{C}$
to $L^{\prime}_{C}$ where $L^{\prime}_{C}$ mod $k$ is maximally isotropic for
the weil-pairing. This has the effect of changing $m^{\prime}\rightarrow
m^{\prime}k$. We may also replace $L_{C}$ by $\frac{1}{a}L_{C}$ and $\phi_{C}$
by $k^{2}\phi_{C}$ which sends $m^{\prime}\rightarrow m^{\prime}/a^{2}$. Thus
by changing $C$ and the polarization in this way we may reduce to
$m^{\prime}=m$ by first scaling by $k=mm^{\prime}$ and then dividing by
$a=m^{\prime}$. This gives a polarized isogeny $A\times C\rightarrow B$ of
size $\prec_{g}N$.

Step 3: Large Galois Orbit

We finally prove that the $B$ is defined over a large field of definition. To
prove this, we note that $B$ contains a unique abelian subvariety $D$ which is
isogenous to $A$. It suffices to show therefore that $D$ has a large field of
definition. We now consider again the minimal isogeny $f=(m,r):A\times
C\rightarrow B$, and write let $G:=L_{B}/(L_{A}\times L_{C})$. Note that there
is a sequence

$f^{-1}(L_{A}\times L_{C})\supset L_{B}\supset L_{A}\times L_{C}$

whose successive quotients are $G$ and $G^{\vee}$, and note that the latter is
isomorphic to $G$ as an abelian group. Moreover, the total quotient is
$A[m]\times C[r]$. It follows that $\\#G[m]^{2}\geq\\#A[m]\cdot\\#C[(r,m)]$.
But now $G^{\vee}[m]\subset A[m]\times C[(r,m)]$ and therefore

$\\#(G^{\vee}\cap
A[m])\geq\big{(}\frac{\\#A[m]}{\\#C[(r,m)]}\big{)}^{\frac{1}{2}}\geq m$

where the last inequality follows from $g\geq h+1$. It follows that
$G^{\vee}\cap A$ contains an element of order $e\geq
m^{\frac{1}{2g}}\succ_{g}N$.

Now, by minimality of $N$ we see that $G^{\vee}\cap A[m]$ does not contain any
$A[\ell]$ for a prime $\ell$. Thus we see that $\\#(G^{\vee}\cap A[e])\leq
e^{2g-1}$. On the other hand, the orbit under
$\operatorname{GSp}_{2g}(\hat{\mathbb{Z}})$ of $G^{\vee}\cap A[e]$ contains
all of $A[e]$ which is of size $e^{2g}$. It follows that there are at least
$e$ such orbits under $\operatorname{GSp}_{2g}(\hat{\mathbb{Z}})$. It follows
by the assumption on $\rho_{A}$ that $G^{\vee}\cap A[e]$ has $\succ_{g}N$
orbits under the action of Galois. The claim now follows from the isomorphism
$D\cong A/(G^{\vee}\cap A)$ together with the simplicity of $A$.

∎

##  4 Geometric Arguments: Weakly Special Subvarieties

We shall require the following:

######  Proposition 4.1.

Let $B$ be a ppav of dimension $g+h$ which has a quotient $A$ which is Galois
generic of dimension $g$. Let $S$ be a proper special subvariety of
$\mathcal{A}_{g}$ which contains $[B]$. Then the universal abelian variety
over $S$ is not geometrically simple.

###### Proof.

Let $G\subset\operatorname{GSp}_{2(g+h),\mathbb{Q}}$ be the subgroup
corresponding to $S$. Now the $\ell$-adic derived Mumford tate group
$MT^{\textrm{der}}_{\ell}(B)$ surjects onto $MT^{\textrm{der}}_{\ell}(A)$ and
thus onto $\operatorname{Sp}_{2g,\mathbb{Q}_{\ell}}$. Since
$\operatorname{Sp}_{2g}$ is a simple algebraic group, it follows by Goursat’s
Lemma that $MT_{\ell}(A)$ contains $\operatorname{Sp}_{2g}$ and hence by
Proposition 2.3 that $MT(A)$ does as well, and hence so does $G$. By Lemma 4.2
it follows that $G$ preserves a non-trivial $\mathbb{Q}$-symplectic splitting,
and therefore the universal abelian variety over $S$ is not geometrically
simple.

∎

The following Lemma is almost certainly not original but we could not find it
in the literature:

######  Lemma 4.2.

Let $V,W$ be symplectic spaces over $\mathbb{C}$ and let
$G\subset\operatorname{GSp}(V\oplus W)$ be a connected reductive group which
contains $\operatorname{Sp}(V)$. Then either $G$ contains
$\operatorname{Sp}(V\oplus W)$ or else there is a proper symplectic subspace
$W^{\prime}\subset W$ such that $G$ preserves $V\oplus W^{\prime}$. Moreover,
if $V,W,G$ are defined over a field $F\subset\mathbb{C}$ then $W^{\prime}$ can
be taken to be as well.

###### Proof.

We first claim that its is enough to conclude that $G$ preserves any proper
subspace containing $V$. To see this, suppose $G$ preserves $K\oplus V$ for
$K\subsetneq W$. Now let $K^{\prime}\subset K$ be the subspace which pairs to
$0$ with all of $K$. Then as $G$ is symplectic, it preserves $K^{\prime}$, and
since $G$ is reductive there is a complement $W^{\prime}\subset K$ such that
$K^{\prime}\oplus W^{\prime}=K$ and $G$ preserves $W^{\prime}\oplus V$. As
$W^{\prime}$ is a complement to $K^{\prime}$ it must be symplectic, and the
claim is proven.

Next we claim that if $G^{\textrm{der}}$ preserves a proper subspace
containing $V$, then so does $G$. Indeed, $G=Z(G)\cdot G^{\textrm{der}}$.
Since $Z(G)$ commutes with $\operatorname{Sp}(V)$ it must preserve $W$, and
hence also $V$ as its symplectic. Now let $K\subsetneq W$ be minimal such that
$V\oplus K$ is preserved by $G^{\textrm{der}}$. Then for any $z\in Z(G)$ the
subspace $V\oplus zK$ is also preserved by $G^{\textrm{der}}$ and hence so is
$V\oplus(K\cap zK)$. The claim follows by the minimality assumption.
Therefore, we may replace $G$ by its derived subgroup and assume its
semisimple and contained in $\operatorname{Sp}(V\oplus W)$.

We now consider the Lie algebra $L:=\textrm{Lie}(\operatorname{GSp}(V\oplus
W))$ as a $\operatorname{Sp}(V)$-module via the adjoint action. We may write
elements of $L$ in matrix form as $\big{(}\begin{smallmatrix}A&B\\\
C&D\end{smallmatrix}\big{)}$ where
$A\in\mathfrak{s}\mathfrak{p}(V),B\in\operatorname{Hom}(W,V),C\in\operatorname{Hom}(V,W),D\in\mathfrak{s}\mathfrak{p}(W)$
satisfy the condition $B=-C^{\vee}$ where we use the symplectic pairing to
identify $V,W$ with $V^{\vee},W^{\vee}$ respectively.

We may therefore write
$L=\big{(}\mathfrak{s}\mathfrak{p}(V)\oplus\operatorname{Hom}(W,V)\big{)}\oplus\mathfrak{s}\mathfrak{p}(W)$
where the two summands constitute a decomposition of $L$ into isotypic
$\operatorname{Sp}(V)$-components, with the first summand isomorphic to a
power of $V$ and the second summand a power of the trivial representation. Now
consider $M:=Lie(G)\subset L$ as a $\operatorname{Sp}(V)$ representation, and
write it as

$M=\mathfrak{s}\mathfrak{p}(V)\oplus R\oplus S$

where $R\subset\operatorname{Hom}(V,W),S\subset\mathfrak{s}\mathfrak{p}(W)$.
Since $L$ is a lie algebra it follows that $S$ is also a lie algebra. Note
that as $R$ is preserved by $\operatorname{Sp}(V)$ it follows that
$R=\operatorname{Hom}(V,K)$ for some $K\subset W$.

Suppose first that $K\subsetneq W$. Note that the lie action of $S$ on $R$ is
simply right composition, and therefore $S$ must preserve $K$. It follows that
elements of $M$ preserve $K\oplus V$, and thus that $G$ preserves $K\oplus V$,
as desired.

On the other hand, suppose that $K=W$. Then $M$ contains all of
$\operatorname{Hom}(W,V)$. We prove now that $M$ must be all of
$\mathfrak{s}\mathfrak{p}(V)$. Indeed, for any elements $w_{1}\in W,v_{1}\in
V$ we have the element $w\rightarrow\langle w,w_{1}\rangle v_{1}$ of
$\operatorname{Hom}(W,V)$, which corresponds to the element
$v\rightarrow\langle v,v_{1}\rangle w_{1}$ of $\operatorname{Hom}(V,W)$.
Applying the lie bracket for two such elements and projecting to $S$ gives us
the element

$w\rightarrow\langle w,w_{1}\rangle\langle v_{1},v_{2}\rangle w_{2}-\langle
w,w_{2}\rangle\langle v_{2},v_{1}\rangle w_{1}$

of $S\cap M$. Picking $w_{1}=w_{2},\langle v_{1},v_{2}\rangle=\frac{1}{2}$
gives us the element

$w\rightarrow\langle w,w_{1}\rangle w_{1}.$

It is well known that these elementary unipotents generate all of $S$, which
shows $M\supset S$ and completes the proof.

For the final part of the claim, suppose that $V,W,G$ are defined over $F$.
Now consider all the subspaces $K\subset W$ such that $V\oplus K$ is preserved
by $G$. Then this set is closed under intersections and
$\operatorname{Aut}(\mathbb{C}/F)$ and therefore the minimal such $K_{0}$ is
defined over $F$. Now let $K_{1}\subset K_{0}$ be the subspace that pairs to 0
with all of $K_{0}$. Then $K_{1}$ is also preserved by $G$, and hence there is
a complementary $K_{2}\oplus K_{1}=K_{0}$ such that $V\oplus K_{2}$ is
preserved by $G$. By the minimality it must be the case that $K_{2}=K_{0}$ and
thus $K_{0}$ is symplectic. Since $K_{0}$ is defined over $F$, the claim is
proven.

∎

##  5 Proof of Theorem 2

The purpose of this section is to prove Theorem 2. We shall in fact find it
convenient to prove a slightly stronger theorem. To state it, we fix a
counting function $\phi$ on $\mathcal{A}_{g}$ as in section §2.9.

######  Theorem 8.

Let $h<g$ be positive integers. Suppose that $V\subset\mathcal{A}_{g+h}$ is a
subvariety such that over $\mathbb{C}$, the generic $g$-dimensional abelian
variety is not a quotient of an abelian variety parameterized by $V$. Then a
$\phi$-generic abelian variety $A$ is not a quotient of an abelian variety
parameterized by $V$.

###### Proof.

We assume the opposite by contradiction. Let $h\geq 0$ me minimal such that
the statement is false, and pick a minimal counterexample
$V\subset\mathcal{A}_{g+h}$.

######  Lemma 5.1.

$V$ is not contained in a proper weakly special subvariety of
$\mathcal{A}_{g}$.

###### Proof.

Indeed, if it is, then by Proposition 4.1 the abelian scheme over $V$ must be
is not-simple, and thus $V$ admits a finite-to-one map to
$\mathcal{A}_{g+h^{\prime}}\times\mathcal{A}_{h-h^{\prime}}$ for some $0\leq
h^{\prime}<h$. In that case if we let $V_{0}$ be the projection of $V$ to
$\mathcal{A}_{g+h^{\prime}}$ then the $\phi$-generic $A\in\mathcal{A}_{g}$ is
not a quotient of anything represented by $V_{0}$ by induction. It follows
that the $\phi$-generic $A$ is also not a quotient of anything in $V$ since
such an $A$ is simple by Proposition 3.1. ∎

Now we set $\mathcal{A}_{g},\mathcal{F}_{g},\pi_{g}$ as in §2, and we set
$\mathcal{F}_{V}:=\pi_{g}^{-1}(V)\cap\mathcal{F}_{g}.$

######  Lemma 5.2.

$V$ is such that $\dim V+\dim\mathcal{A}_{h}<\dim\mathcal{A}_{g+h}$

###### Proof.

Suppose not. Let $[A]\in\mathcal{A}_{g}$ be an abelian variety which is
quotient of something parameterized by $V$. Then
$g([A]\times\mathbb{H}_{h})\cap\pi_{g}^{-1}(V)\neq 0$. By our dimension
assumption, this is also true for a generic nearby $x\in\mathcal{A}_{g}$.
Picking $x$ to be a generic $\mathbb{C}$-point of the $\mathbb{Q}$-variety
$\mathcal{A}_{g}$ contradicts our assumptions on $V$. ∎

Next, for $x\in\mathcal{F}_{g}$ we set

$I_{x}:=\\{g\in\operatorname{GSp}_{2g+2h}(\mathbb{C})\mid
g(x\times\mathcal{F}_{h})\cap\mathcal{F}_{V}\neq\emptyset\\}.$

Note that $I\rightarrow\mathcal{F}_{g}$ is a definable family. Moreover, for
each $g\in\operatorname{GSp}_{2g+2h}(\mathbb{C})$ the number of connected
components of $g(x\times\mathcal{F}_{g})\cap\mathcal{F}_{V}$ is $O_{g}(1)$.

Now, we let $[A]\in\phi^{-1}(U(\mathbb{Q}))$ be a $\phi$-generic point,
defined over $K$, which is a quotient of an abelian variety parameterized by
$V$. Then by Theorem 7,
$[\operatorname{GSp}_{2g}(\hat{\mathbb{Z}}):\rho_{A}(G_{k})]=O_{g}(1)$.

Let $[B]\in V$ be such that $A$ is a quotient of $V$, let $N_{A}$ is the
minimal degree polarized isogeny of $B$ to $A\times\mathcal{A}_{h}$.

######  Lemma 5.3.

For $N_{A}\gg 1$, There is a positive-dimensional weakly atypical subvariety
$W\subset V$ which contains a conjugate of $[B]$ over $\mathbb{Q}(A)$.

###### Proof.

By Theorem 7 the field of definition of $B$ has degree $\succ_{g}N_{A}$ over
$\mathbb{Q}$, and hence over $\mathbb{Q}(A)$. Write $\phi:B\rightarrow A\times
C$ for the isogeny, and by a slight abuse of notation we use $[A],[C],[B]$ to
refer to the representatives in the fundamental domains
$\mathcal{F}_{g},\mathcal{F}_{h},\mathcal{F}_{g+h}$. By Corollary 2.2 it
follows that there exists an $M\in I_{[A]}(\mathbb{Q})$ of height
$H(M)\prec_{g}N_{A}$ such that $M([A]\times[C])=[B]$. In fact, we obtain such
an element $M_{B^{\prime}}$ for each Galois conjugate $B^{\prime}$ of $B$.

Suppose the number of distinct $M_{B^{\prime}}$ we obtain in this way is
$N_{A}^{o(1)}$. Then for some such element $M$ we must have that
$M\cdot([A]\times\mathcal{F}_{h})\cap\mathcal{F}_{V}$ contains
$\succ_{g}N_{A}$ distinct conjugates of $B$. Since this is a definable family
as $M$ varies, it follows that for $N_{A}\gg 1$ that it must consist of at
least one positive dimensional component $U$ containing a conjugate of $B$ (in
fact, many such conjugates). By Lemma 5.2 we see that $U$ is an unlikely
intersection. By Theorem 3 it follows that $U$ is contained in a weakly
atypical subvariety of $V$.

Thus we assume that the number of distinct $M_{B^{\prime}}$ is
$\prec_{g}N_{A}$. By Theorem 6, we conclude that for $N_{A}\gg 1$, $I_{[A]}$
contains a semi-algebraic curve $C_{0}$ containing some $M_{B^{\prime}}$ for
$B^{\prime}$ a conjugate of $B$. Let $C$ be the complex algebraic curve
containing $C_{0}$. We now have two cases.

Suppose that for some conjugate $B^{\prime}$ we have
$M_{B^{\prime}}\cdot([A]\times\mathcal{F}_{h})\cap\mathcal{F}_{V}$ is positive
dimensional, and let $U$ denote an analytic component of such an intersection.
We then conclude as above.

Alternatively, we see that for a generic $c\in C$ the variety
$c\cdot([A]\times\mathcal{F}_{h})\cap\mathcal{F}_{V}$ is finite. Since this is
a definable family of (0-dimensional )varieties as $c$ varies, it follows that
thse intersections are of size $O_{g}(1)$. Hence the intersections must vary
with $c\in C$ and thus $C\cdot([A]\times\mathcal{F}_{h})\cap\mathcal{F}_{V}$
is positive dimensional. Let $U$ be an analytic component of the intersection.
Again, by Lemma 5.2 we know that $U$ is an unlikely intersection, so we
conclude this case in the same way.

∎

We now apply our Geometric Zilber-Pink result 4 to conclude that a conjugate
of $B$ is contained in a proper subvariety $V_{1}\subset V$ which contains all
the positive-dimensional weakly atypical subvarieties of $V$. It follows that
for $\phi$-generic $A$ with $N_{A}\gg 1$, if $A$ is a quotient of something
represented by $V$ then $A$ is a quotient of something represented by $V_{1}$.
By our minimality assumption on $V$, we conclude that for a $\phi$-generic $A$
which is a quotient of something represented by $V$, the corresponding $N_{A}$
is bounded.

Now, for a fixed $N$ the set of $A$ with $N_{A}=N$ lies in the projection of
$T_{N}\cdot V\cap(\mathcal{A}_{g}\times\mathcal{A}_{h})$ to $\mathcal{A}_{g}$
for $T_{N}$ the Hecke-correspondence for polarized isogenies of degrees $N$.
This must be a proper subvariety, by the assumption on $V$, and hence has
density 0. This yields the desired contradiction.

∎

## References

  * [1]
BaldiGredorioKlinglerBrunoUllmoEmmanuelOn the distribution of the hodge
locus2021arXiv:2107.08838@article{baldiklinglerullmo, author = {Baldi,
Gredorio}, author = {Klingler, Bruno}, author = {Ullmo, Emmanuel}, title = {On
the distribution of the hodge locus}, date = {2021}, volume =
{\href{https://arxiv.org/pdf/2107.08838.pdf}{\texttt{arXiv:2107.08838}}}}

  * [3]
BirkenhakeChristinaLangeHerbertComplex abelian varietiesSecondGrundlehren der
mathematischen Wissenschaften [Fundamental Principles of Mathematical
Sciences]Springer-Verlag, Berlin2004302ISBN 3-540-20488-1LinkReview
MathReviews@book{BLAV, author = {Birkenhake, Christina}, author = {Lange,
Herbert}, title = {Complex abelian varieties}, edition = {Second}, series =
{Grundlehren der mathematischen Wissenschaften [Fundamental Principles of
Mathematical Sciences]}, publisher = {Springer-Verlag, Berlin}, date = {2004},
volume = {302}, isbn = {3-540-20488-1}, url = {https://doi-
org.myaccess.library.utoronto.ca/10.1007/978-3-662-06307-1}, review =
{\MR{2062673}}}

  * [5]
ChaiChing-LiOortFransAbelian varieties isogenous to a Jacobian2012ISSN
0003-486XAnn. of Math. (2)1761589–635LinkReview
MathReviews@article{COJacobians, author = {Chai, Ching-Li}, author = {Oort,
Frans}, title = {Abelian varieties isogenous to a {J}acobian}, date = {2012},
issn = {0003-486X}, journal = {Ann. of Math. (2)}, volume = {176}, number =
{1}, pages = {589\ndash 635}, url = {https://doi-
org.myaccess.library.utoronto.ca/10.4007/annals.2012.176.1.11}, review =
{\MR{2925391}}}

  * [7]
HafnerJames LeeWallingLynne H.Explicit action of Hecke operators on Siegel
modular forms2002ISSN 0022-314XJ. Number Theory93134–57LinkReview
MathReviews@article{explicitheckecosets, author = {Hafner, James~Lee}, author
= {Walling, Lynne~H.}, title = {Explicit action of {H}ecke operators on
{S}iegel modular forms}, date = {2002}, issn = {0022-314X}, journal = {J.
Number Theory}, volume = {93}, number = {1}, pages = {34\ndash 57}, url =
{https://doi-org.myaccess.library.utoronto.ca/10.1006/jnth.2001.2711}, review
= {\MR{1892929}}}

  * [9]
MilneJ. S.Shimura varieties and moduli2013Handbook of moduli. Vol. IIAdv.
Lect. Math. (ALM)25Int. Press, Somerville, MA467–548Review
MathReviews@incollection{milnemoduli, author = {Milne, J.~S.}, title =
{Shimura varieties and moduli}, date = {2013}, booktitle = {Handbook of
moduli. {V}ol. {II}}, series = {Adv. Lect. Math. (ALM)}, volume = {25},
publisher = {Int. Press, Somerville, MA}, pages = {467\ndash 548}, review =
{\MR{3184183}}}

  * [11]
MokNgaimingPilaJonathanTsimermanJacobAx-Schanuel for Shimura varieties2019ISSN
0003-486XAnn. of Math. (2)1893945–978LinkReview MathReviews@article{ASAg,
author = {Mok, Ngaiming}, author = {Pila, Jonathan}, author = {Tsimerman,
Jacob}, title = {Ax-{S}chanuel for {S}himura varieties}, date = {2019}, issn =
{0003-486X}, journal = {Ann. of Math. (2)}, volume = {189}, number = {3},
pages = {945\ndash 978}, url = {https://doi-
org.myaccess.library.utoronto.ca/10.4007/annals.2019.189.3.7}, review =
{\MR{3961087}}}

  * [13]
MasserDavidWüstholzGisbertIsogeny estimates for abelian varieties, and
finiteness theorems1993ISSN 0003-486XAnn. of Math. (2)1373459–472LinkReview
MathReviews@article{MWisogeny, author = {Masser, David}, author =
{W\"{u}stholz, Gisbert}, title = {Isogeny estimates for abelian varieties, and
finiteness theorems}, date = {1993}, issn = {0003-486X}, journal = {Ann. of
Math. (2)}, volume = {137}, number = {3}, pages = {459\ndash 472}, url =
{https://doi-org.myaccess.library.utoronto.ca/10.2307/2946529}, review =
{\MR{1217345}}}

  * [15]
MasserDavidZannierUmbertoAbelian varieties isogenous to no Jacobian2020ISSN
0003-486XAnn. of Math. (2)1912635–674LinkReview
MathReviews@article{MZjacobians, author = {Masser, David}, author = {Zannier,
Umberto}, title = {Abelian varieties isogenous to no {J}acobian}, date =
{2020}, issn = {0003-486X}, journal = {Ann. of Math. (2)}, volume = {191},
number = {2}, pages = {635\ndash 674}, url = {https://doi-
org.myaccess.library.utoronto.ca/10.4007/annals.2020.191.2.7}, review =
{\MR{4076633}}}

  * [17]
OrrMartinHeight bounds and the Siegel property2018ISSN 1937-0652Algebra Number
Theory122455–478LinkReview MathReviews@article{orr, author = {Orr, Martin},
title = {Height bounds and the {S}iegel property}, date = {2018}, issn =
{1937-0652}, journal = {Algebra Number Theory}, volume = {12}, number = {2},
pages = {455\ndash 478}, url = {https://doi-
org.myaccess.library.utoronto.ca/10.2140/ant.2018.12.455}, review =
{\MR{3803710}}}

  * [19]
PilaJ.O-minimality and the André-Oort conjecture for $\mathbb{C}^{n}$2011ISSN
0003-486XAnn. of Math. (2)17331779–1840@article{PAOmodular, author = {Pila,
J.}, title = {O-minimality and the {A}ndr\'e-{O}ort conjecture for {$\Bbb
C^n$}}, date = {2011}, issn = {0003-486X}, journal = {Ann. of Math. (2)},
volume = {173}, number = {3}, pages = {1779\ndash 1840}}

  * [21]
PirolaGian PietroCurves on generic Kummer varieties1989ISSN 0012-7094Duke
Math. J.593701–708LinkReview MathReviews@article{pirolahyperelliptic, author =
{Pirola, Gian~Pietro}, title = {Curves on generic {K}ummer varieties}, date =
{1989}, issn = {0012-7094}, journal = {Duke Math. J.}, volume = {59}, number =
{3}, pages = {701\ndash 708}, url = {https://doi-
org.myaccess.library.utoronto.ca/10.1215/S0012-7094-89-05931-0}, review =
{\MR{1046744}}}

  * [23]
PirolaGian PietroAbel-Jacobi invariant and curves on generic abelian
varieties1995Abelian varieties (Egloffstein, 1993)de Gruyter,
Berlin237–249Review MathReviews@incollection{pirolaabelian, author = {Pirola,
Gian~Pietro}, title = {Abel-{J}acobi invariant and curves on generic abelian
varieties}, date = {1995}, booktitle = {Abelian varieties ({E}gloffstein,
1993)}, publisher = {de Gruyter, Berlin}, pages = {237\ndash 249}, review =
{\MR{1336610}}}

  * [25]
PilaJonathanTsimermanJacobThe André-Oort conjecture for the moduli space of
abelian surfaces2013ISSN 0010-437XCompos. Math.1492204–216LinkReview
MathReviews@article{PTsurfaces, author = {Pila, Jonathan}, author =
{Tsimerman, Jacob}, title = {The {A}ndr\'{e}-{O}ort conjecture for the moduli
space of abelian surfaces}, date = {2013}, issn = {0010-437X}, journal =
{Compos. Math.}, volume = {149}, number = {2}, pages = {204\ndash 216}, url =
{https://doi-org.myaccess.library.utoronto.ca/10.1112/S0010437X12000589},
review = {\MR{3020307}}}

  * [27]
PilaJ.WilkieA. J.The rational points of a definable set2006ISSN 0012-7094Duke
Math. J.1333591–616LinkReview MathReviews@article{PW, author = {Pila, J.},
author = {Wilkie, A.~J.}, title = {The rational points of a definable set},
date = {2006}, issn = {0012-7094}, journal = {Duke Math. J.}, volume = {133},
number = {3}, pages = {591\ndash 616}, url = {https://doi-
org.myaccess.library.utoronto.ca/10.1215/S0012-7094-06-13336-7}, review =
{\MR{2228464}}}

  * [29]
SiegelCarl LudwigSymplectic geometryAcademic Press, New York-London1964Review
MathReviews@book{siegelsympleticbook, author = {Siegel, Carl~Ludwig}, title =
{Symplectic geometry}, publisher = {Academic Press, New York-London}, date =
{1964}, review = {\MR{0164063}}}

  * [31]
TsimermanJacobThe existence of an abelian variety over $\overline{\mathbb{Q}}$
isogenous to no Jacobian2012ISSN 0003-486XAnn. of Math.
(2)1761637–650LinkReview MathReviews@article{tsimjacobians, author =
{Tsimerman, Jacob}, title = {The existence of an abelian variety over
{$\overline{\Bbb Q}$} isogenous to no {J}acobian}, date = {2012}, issn =
{0003-486X}, journal = {Ann. of Math. (2)}, volume = {176}, number = {1},
pages = {637\ndash 650}, url = {https://doi-
org.myaccess.library.utoronto.ca/10.4007/annals.2012.176.1.12}, review =
{\MR{2925392}}}

  * [33]
UllmoEmmanuelYafaevAndreiA characterization of special subvarieties2011ISSN
0025-5793Mathematika572263–273LinkReview MathReviews@article{UY, author =
{Ullmo, Emmanuel}, author = {Yafaev, Andrei}, title = {A characterization of
special subvarieties}, date = {2011}, issn = {0025-5793}, journal =
{Mathematika}, volume = {57}, number = {2}, pages = {263\ndash 273}, url =
{https://doi-org.myaccess.library.utoronto.ca/10.1112/S0025579311001628},
review = {\MR{2825237}}}

  * [35]
van den DriesL.Tame topology and o-minimal structuresLondon Mathematical
Society Lecture Note SeriesCambridge University Press, Cambridge1998248ISBN
0-521-59838-9LinkReview MathReviews@book{Dries, author = {van~den Dries, L.},
title = {Tame topology and o-minimal structures}, series = {London
Mathematical Society Lecture Note Series}, publisher = {Cambridge University
Press, Cambridge}, date = {1998}, volume = {248}, isbn = {0-521-59838-9}, url
= {https://doi-org.myaccess.library.utoronto.ca/10.1017/CBO9780511525919},
review = {\MR{1633348}}}

  * [37]
VoisinClaireChow ring and gonality of general abelian varieties2018Ann. H.
Lebesgue1313–332LinkReview MathReviews@article{voisinchowring, author =
{Voisin, Claire}, title = {Chow ring and gonality of general abelian
varieties}, date = {2018}, journal = {Ann. H. Lebesgue}, volume = {1}, pages =
{313\ndash 332}, url = {https://doi-
org.myaccess.library.utoronto.ca/10.1007/s42081-018-0017-3}, review =
{\MR{3963294}}}

  * [39]
ZywinaDavidFamilies of abelian varieties and large galois images2022ISSN
1073-7928International Mathematics Research
Noticeshttps://academic.oup.com/imrn/advance-article-
pdf/doi/10.1093/imrn/rnac301/46781394/rnac301.pdfLinkrnac301@article{Zywina,
author = {Zywina, David}, title = {Families of abelian varieties and large
galois images}, date = {2022}, issn = {1073-7928}, journal = {International
Mathematics Research Notices}, eprint =
{https://academic.oup.com/imrn/advance-article-
pdf/doi/10.1093/imrn/rnac301/46781394/rnac301.pdf}, url =
{https://doi.org/10.1093/imrn/rnac301}, note = {rnac301}}

  * [41]


# Joint covariate-alignment and concept-alignment: a framework for domain
generalization

Thuan Nguyen
Department of Computer Science
Tufts University
Medford, MA 02155
<EMAIL_ADDRESS>
&Boyang Lyu
Department of Electrical and Computer Engineering
Tufts University
Medford, MA 02155
<EMAIL_ADDRESS>
&Prakash Ishwar
Department of Electrical and Computer Engineering
Boston University
Boston, MA 02215
<EMAIL_ADDRESS>
&Matthias Scheutz
Department of Computer Science
Tufts University
Medford, MA 02155
<EMAIL_ADDRESS>
&Shuchin Aeron
Department of Electrical and Computer Engineering
Tufts University
Medford, MA 02155
<EMAIL_ADDRESS>
###### Abstract

In this paper, we propose a novel domain generalization (DG) framework based
on a new upper bound to the risk on the unseen domain. Particularly, our
framework proposes to jointly minimize both the covariate-shift as well as the
concept-shift between the seen domains for a better performance on the unseen
domain. While the proposed approach can be implemented via an arbitrary
combination of covariate-alignment and concept-alignment modules, in this work
we use well-established approaches for distributional alignment namely,
Maximum Mean Discrepancy (MMD) and covariance Alignment (CORAL), and use an
Invariant Risk Minimization (IRM)-based approach for concept alignment. Our
numerical results show that the proposed methods perform as well as or better
than the state-of-the-art for domain generalization on several data
sets111This paper was accepted at 32nd IEEE International Workshop on Machine
Learning for Signal Processing (MLSP 2022)..

_K_ eywords Domain generalization, domain alignment, out-of-distribution
generalization, distribution shift.

##  1 Introduction

Domain generalization (DG) has been studied extensively over the past decade
as an important practical problem arising in a number of areas such as
computer vision, signal processing, and medical imaging [1] [2]. Like standard
learning settings, DG aims to learn a model from several seen domains
(training data) that can generalize well on an unseen domain (test data).
However, in contrast to the standard setting where the test data is assumed to
come from the same distribution as training data, in DG, the distribution of
the test data is different, i.e., there is a presence of what is referred to
as a _distribution shift_. This phenomena can be observed in many practical
settings [3].

A number of approaches for DG are based on the assumption that there exist
domain-invariant features that are transferable and unchanged from domain to
domain. Thus, a classifier designed on top of these features will likely
generalize well to the unseen domain. In practice, DG methods based on this
assumption consist of two steps: (a) learning a good representation function
that outputs the domain-invariant features, and (b) designing a classifier
based on these domain-invariant features.

Different definitions of domain-invariant features have been proposed and they
lead to different feature selection schemes. In [4, 5, 6, 7, 8, 9, 10, 11], a
feature is defined as domain-invariant if its marginal distribution is
unchanged over domains. On the other hand, in [12, 13, 14, 15, 16, 17, 18, 19]
a feature is considered as domain-invariant if its conditional distribution of
the label given the representation feature is unchanged from domain to domain.

The differences in the definitions are rooted in the different modeling
assumptions regarding the domains. Particularly, in the covariate-shift
setting [20], the marginal distribution of data varies across the domains. On
the other hand, in the setting of concept-shift [20], the conditional
distribution of the label (class) conditioned on the data varies from domain
to domain222The setting where the conditional distribution of data conditioned
on the label (class) changes from domain to domain is also referred to as
concept-shift. However, we do not consider that scenario in this paper.. We
survey related work in this context in Sec. 2.

In this paper, we revisit the seminal upper bound for the prediction risk in
the unseen domain derived in [21] and show the necessity of jointly designing
a representation function that minimizes both the covariate-shift and the
concept-shift risks. We make the following contributions:

  * •

We derive a novel upper bound for the prediction risk in the unseen domain
that consists of two terms, namely, a covariate-alignment term and a concept-
alignment term. This theoretical result motivates a domain generalization
algorithm that combines both covariate-alignment and concept-alignment
algorithms.

  * •

We propose two new algorithms MMD-CEM and CORAL-CEM that combine,
respectively, the CORrelation ALignment algorithm (CORAL) [22] and the Maximum
Mean Discrepancy algorithm (MMD) [4] with the Conditional Entropy Invariant
Risk Minimization algorithm (CEM) [23].

  * •

We compare the proposed algorithms (MMD-CEM, CORAL-CEM) with six competing DG
methods. Our methods exhibit similar or better performance compared to the six
alternatives on both CS-CMNIST [24] and CMNIST [12] datasets.

The remainder of this paper is structured as follows. In Sec. 2, we summarize
the recent work on DG dealing with covariate-shift and concept-shift. Sec. 3
formally defines the problem and clarifies notation used. Sec. 4 provides the
main theoretical results which motivate the practical algorithms of Sec. 5.
Finally, we provide the numerical results in Sec. 6 and conclude in Sec. 7.

##  2 Related work

Under the covariate-shift setting, domain alignment methods focus on aligning
the marginal distributions of the representation from different seen domains
by minimizing distributional divergences or distances [4, 5, 6, 7, 8, 9, 10,
11]. For instance, in [10, 11], the Wasserstein distance between distributions
of the seen domains in representation space is minimized. In [4], Li et al.
proposed the MMD algorithm to minimize the maximum mean discrepancy distance
between seen domain distributions in the representation space. In a similar
vein, the CORAL algorithm [22] is based on the idea of matching the mean and
covariance of feature distributions from different domains. In [6], the
authors use deep neural networks to minimize the difference between the
variances of transformed features from seen domains to achieve domain
generalization. In [9], the authors aim to learn the domain-invariant features
with marginal distributions unchanged from domain to domain together with the
domain-specific features to enhance the generalization performance.

Since the conditional distribution of the label given the representation
variable is directly related to the classification performance, there are many
works that aim to deal with concept-shift, i.e., minimizing the divergence
between the conditional distribution of the labels (concepts) given the
representation [12, 13, 14, 15, 16, 17, 18, 19]. In [12, 18], linear and non-
linear IRM algorithms are proposed for learning the invariant features such
that their conditional distributions are unchanged across domains, leading to
the construction of an optimal classifier for all seen domains. In [23],
Nguyen et al. developed a conditional entropy minimization principle to remove
spurious invariant features from the invariant features learned by the IRM
algorithm [12]. In [13], concept-alignment is achieved via minimizing the
mutual information of the label given the representation variable for a given
domain. In [15], Wang et al. handle concept-shift by directly aligning the
conditional distribution within each class regardless of domains via the use
of Kullback–Leibler divergence.

To the best of our knowledge, there are only two works in the DG setting that
simultaneously deal with both the covariate-shift and concept-shift [25,
26]333But there are several works that simultaneously handle both covariate-
shift and concept-shift under the Domain Adaptation (DA) setting, for example
the work in [27, 28]. However, DA is not considered in this paper.. In [25],
Nguyen et al. proposed an algorithm that is capable of achieving both
covariate-alignment and concept-alignment by enforcing the latent
representation to be invariant under all transformation functions. Their
algorithm, however, requires the use of invertible and differentiable
transformation functions together with a generative adversarial network which
is known to be hard to train. In [26], motivated by some examples where using
concept-alignment alone is inadequate for achieving a good generalization, a
covariate-alignment algorithm is heuristically combined with a concept-
alignment algorithm to achieve a better out-of-distribution generalization. In
contrast to [25, 26], our work appears to be the first work that uses the
upper bound for the risk in the unseen domain to theoretically motivate the
necessity of achieving both covariate and concept alignment in DG.

##  3 Problem Formulation

Let $\bm{x}\in\mathbb{X}\subseteq\mathbb{R}^{d}$ denote the input data and
$y\in\mathbb{Y}$ denote the label. The domain generalization task aims to
learn a representation function
$f:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d^{\prime}}$, $d^{\prime}\leq d$,
followed by a classifier $g:\mathbb{R}^{d^{\prime}}\rightarrow\mathbb{Y}$
trained using data from seen domains $D^{(s)}$, but generalizes well to data
from an unseen domain $D^{(u)}$.

Let $\bm{z}=f(\bm{x})$ denote the latent variable induced by input data
$\bm{x}$ under the mapping $f$. We denote the distributions of data from a
seen domain and the unseen domain by $p^{(s)}(\bm{x})$ and $p^{(u)}(\bm{x})$,
respectively. The distributions of corresponding latent variables in a seen
domain and the unseen domain are denoted by $p^{(s)}(\bm{z})$ and
$p^{(u)}(\bm{z})$, respectively. We use $Z$ to denote the latent random
variable and $Y$ to denote the label random variable.

An optimal labeling function for a given domain and a given representation
function $f$ is the labeling function that minimizes the prediction risk in
the domain using the given representation function. For a given representation
function $f$, we denote the optimal labeling functions from representation
space to label space in a seen domain and the unseen domain by
$l^{(s)}(\bm{z})$ and $l^{(u)}(\bm{z})$ respectively. For a given $f$, the
risks of using a classifier $g$ (possibly stochastic) in a seen domain and the
unseen domain are defined by:

$\epsilon^{(s)}(g,l^{(s)})=\mathop{\mathbb{E}}_{\bm{z}\sim
p^{(s)}(\bm{z})}d(g(\bm{z}),l^{(s)}(\bm{z})),$ (1)
$\epsilon^{(u)}(g,l^{(u)})=\mathop{\mathbb{E}}_{\bm{z}\sim
p^{(u)}(\bm{z})}d(g(\bm{z}),l^{(u)}(\bm{z})),$ (2)

where $\mathop{\mathbb{E}}[\cdot]$ denotes expectation and $d(\cdot,\cdot)$ is
a loss function that captures the mismatch between the classifier $g$ and the
optimal labeling functions $l^{(s)},l^{(u)}$. In addition, for a given
representation function $f$ we define the following quantity:

$\epsilon^{(s)}(g,l^{(u)})=\mathop{\mathbb{E}}_{\bm{z}\sim
p^{(s)}(\bm{z})}d(g(\bm{z}),l^{(u)}(\bm{z}))$ (3)

which is the loss of using classifier $g$ to input data from a seen domain $s$
transformed by $f$ and labeled with the optimum labeling function of the
unseen domain for representation $f$. Since $p^{(u)}(\bm{z})$ and
$l^{(u)}(\bm{z})$ depend on $f$, we want to learn a good representation
function $f$ together with a classifier $g$ to minimize the prediction risk in
the unseen domain $\epsilon^{(u)}(g,l^{(u)})$.

##  4 Main results

######  Theorem 1.

If the loss function $d(\cdot,\cdot)$ is non-negative, symmetric and satisfies
the triangle inequality, then for a given representation function $f$:

$\displaystyle\epsilon^{(u)}(g,l^{(u)})\\!$ $\displaystyle\leq$
$\displaystyle\epsilon^{(s)}(g,l^{(s)})$ (4) $\displaystyle\\!+$
$\displaystyle\int_{\bm{z}}|p^{(u)}(\bm{z})-p^{(s)}(\bm{z})|\>d(g(\bm{z}),l^{(u)}(\bm{z}))\,d\bm{z}$
$\displaystyle\\!+$
$\displaystyle\int_{\bm{z}}p^{(s)}(\bm{z})\>d(l^{(u)}(\bm{z}),l^{(s)}(\bm{z}))\,d\bm{z}.$

###### Proof.

We have:

$\displaystyle\epsilon^{(u)}(g,l^{(u)})\\!$ $\displaystyle=$
$\displaystyle\epsilon^{(u)}(g,l^{(u)})$ $\displaystyle+$
$\displaystyle\epsilon^{(s)}(g,l^{(s)})-\epsilon^{(s)}(g,l^{(s)})$
$\displaystyle+$
$\displaystyle\epsilon^{(s)}(g,l^{(u)})-\epsilon^{(s)}(g,l^{(u)})$
$\displaystyle\leq$ $\displaystyle\epsilon^{(s)}(g,l^{(s)})$ $\displaystyle+$
$\displaystyle|\epsilon^{(u)}(g,l^{(u)})-\epsilon^{(s)}(g,l^{(u)})|$
$\displaystyle+$
$\displaystyle\epsilon^{(s)}(g,l^{(u)})-\epsilon^{(s)}(g,l^{(s)})$
$\displaystyle\leq$ $\displaystyle\epsilon^{(s)}(g,l^{(s)})$ $\displaystyle+$
$\displaystyle\int_{\bm{z}}|p^{(u)}(\bm{z})-p^{(s)}(\bm{z})|\>d(g(\bm{z}),l^{(u)}(\bm{z}))\,d\bm{z}$
$\displaystyle+$
$\displaystyle\int_{\bm{z}}p^{(s)}(\bm{z})\>d(l^{(u)}(\bm{z}),l^{(s)}(\bm{z}))\,d\bm{z}$
(7)

where (4) follows from the reorganization of (4) and the fact that $a\leq|a|$,
$\forall a$ and (7) follows from the definitions of
$\epsilon^{(u)}(g,l^{(u)})$, $\epsilon^{(s)}(g,l^{(s)})$,
$\epsilon^{(s)}(g,l^{(u)})$ in (1), (2), (3), the symmetry of
$d(\cdot,\cdot)$, and the triangle inequality
$d(g(\bm{z}),l^{(u)}(\bm{z}))-d(g(\bm{z}),l^{(s)}(\bm{z}))\leq
d(l^{(u)}(\bm{z}),l^{(s)}(\bm{z}))$. ∎

The bound in (4) characterizes the risk of using a classifier trained on a
seen domain but applied to the unseen domain. The bound contains three terms:
(a) the first term captures the risk induced by the classifier on a seen
domain for a given representation function $f$, (b) the second term measures
the discrepancy between the marginal distributions of seen and unseen domains
in the latent space corresponding to representation function $f$, and (c) the
third term quantifies the mismatch between optimal classifiers for seen and
unseen domains for the given $f$.

Our bound shares some similarities with the bound proposed by Ben-David et al.
[21]. Particularly, for any representation function $f$, the following bound
holds [21]:

$\displaystyle\epsilon^{(u)}(g,l^{(u)})\\!\leq\\!\epsilon^{(s)}(g,l^{(s)})\\!+\\!d_{\mathcal{H}}\Big{(}p^{(u)}(\bm{z}),p^{(s)}(\bm{z})\Big{)}\\!+\\!\lambda$
(8)

where $d_{\mathcal{H}}$ is $\mathcal{H}$-divergence [29] and:

$\displaystyle\lambda=\inf_{g}\Big{(}\epsilon^{(s)}(g,l^{(s)})+\epsilon^{(u)}(g,l^{(u)})\Big{)}.$
(9)

In [21], the above bound was developed for the DA setting. We have adapted it
to the DG setting by identifying “source” domains in DA with “seen” domains in
DG and the “target” domain in DA with the “unseen” domain in DG. While the
first two terms of (4) and (8) are quite similar, the main difference comes
from the way that the third term is optimized in practice. The practical DA
algorithm proposed in [21] ignores optimizing the third term $\lambda$ of the
upper bound in (8) since the second term of the infimum in (9) cannot be
estimated during training. Moreover, the first term of the infimum in (9) is
already captured in the first term of the upper bound in (8). In the DG
setting too, since the unseen domain samples are not available during
training, one usually optimizes the bound only over several seen domains as in
[23, 1, 2]. In contrast, the third term in (4) measures the mismatch of
optimal classifiers between domains and, therefore, can be practically
optimized via designing a representation function $f$ followed by a classifier
$g$ that is optimal for all (seen) domains. As will be shown later, finding an
optimal classifier over all domains encourages the use of concept-alignment
algorithms. Indeed, designing a classifier that is optimal over all domains is
the principle behind the recently proposed IRM algorithm [12].

It is also worth comparing our bound with the bound proposed by Lyu et al.
[11]. Indeed, Lemma 1 in [11] extends the work in [21] to establish a new
upper bound on the prediction risk in the unseen domain that is a function of
both the representation mapping $f$ and the classifier $g$. While the first
two terms of the bound in [11] are quite similar to our first two terms, their
third term is a constant which is completely independent of both $f$ and $g$.
Thus, the bound in [11] does not encourage the use of concept-alignment
algorithms. In practice, the proposed algorithms in [21] and [11] only aim to
achieve covariate-alignment, i.e., minimizing the discrepancy of marginal
distributions between domains.

Next, to explicitly show that Theorem 1 motivates the combination of both
covariate-alignment and concept-alignment algorithms, we define the following
two terms:

$\displaystyle\Delta_{1}=\max_{\bm{z}}\,|p^{(s)}(\bm{z})-p^{(u)}(\bm{z})|,$
(10)

and:

$\displaystyle\Delta_{2}=\max_{\bm{z}}\,d(l^{(u)}(\bm{z}),l^{(s)}(\bm{z})).$
(11)

The first quantity $\Delta_{1}$ captures the maximum mismatch of marginal
distributions between domains in latent space. Thus, minimizing $\Delta_{1}$
supports achieving covariate-alignment. On the other hand, the second quantity
$\Delta_{2}$ measures the maximum mismatch of optimal classifiers between
domains. Since stochastic classifiers are considered, enforcing a small
mismatch between classifiers leads to a small discrepancy of conditional
distributions $p(Y|Z)$ between domains. In other words, minimizing
$\Delta_{2}$ over representation functions supports achieving concept-
alignment.

######  Corollary 1.

Under the settings of Theorem 1, for a given representation function $f$:

$\displaystyle\epsilon^{(u)}(g,l^{(u)})$ $\displaystyle\leq$
$\displaystyle\epsilon^{(s)}(g,l^{(s)})$ (12) $\displaystyle+$
$\displaystyle\Delta_{1}\int_{\bm{z}}d(g(\bm{z}),l^{(u)}(\bm{z}))\,d\bm{z}$
$\displaystyle+$ $\displaystyle\Delta_{2}.$

###### Proof.

The proof directly follows by using Theorem 1, the definitions in (10), (11)
and the fact that $\int_{\bm{z}}p^{(s)}(\bm{z})d\bm{z}=1$. ∎

If the representation space is compact and the distance $d(\cdot,\cdot)$ is
bounded then $\int_{\bm{z}}d(g(\bm{z}),l^{(u)}(\bm{z}))\,d\bm{z}$ is finite.
Thus, minimizing the risk on seen domain $\epsilon^{(s)}(g,l^{(s)})$ together
with the covariate-alignment term $\Delta_{1}$ and the concept-alignment term
$\Delta_{2}$ encourages the trained classifier $g$ to generalize well on the
unseen domain.

##  5 Practical Approach

Motivated by the theoretical results in Sec. 4, we propose a framework that
combines covariate-alignment algorithms together with concept-alignment
algorithms for minimizing risk in the unseen domain. There are myriad ways of
combining a given covariate-alignment algorithm with a given concept-alignment
algorithm. For simplicity, we focus on minimizing the sum of the empirical
prediction risk in the seen domains and a nonnegative weighted combination of
the cost functions for covariate-alignment and concept-alignment algorithms.
We consider two well-studied algorithms for covariate-alignment, namely CORAL
[22] and MMD [4], and one well-studied concept-alignment algorithm, namely CEM
[23]. We propose two DG algorithms named CORAL-CEM and MMD-CEM whose objective
functions are defined as follows:

$\\!\\!\\!R_{\text{CORAL-CEM}}(f,g):=R(f,g)+\alpha L_{\text{CORAL}}(f)+\beta
L_{\text{CEM}}(f,g),$ (13) $\\!\\!\\!R_{\text{MMD-CEM}}(f,g):=R(f,g)+\alpha
L_{\text{MMD}}(f)+\beta L_{\text{CEM}}(f,g),$ (14)

where the first term is the summation of the empirical risk from all seen
domains, i.e., summation of $\epsilon^{(s)}(g,l^{(s)})$ over all seen domains,
the second term is the covariate-alignment term which is implemented via
either the CORAL algorithm [22] or the MMD algorithm [4], the third term is
the concept-alignment term which is implemented via the CEM algorithm [23],
and $\alpha,\beta$ are two positive hyper-parameters that control the trade-
off between these loss terms.

##  6 Experiments

###  6.1 Datasets

In this section, we examine our proposed methods on two datasets CMNIST [12]
and CS-CMNIST [24]. To illustrate how CMNIST and CS-CMNIST datasets are
generated, we use $e$ to denote domain index, $e=1,2,3$, $X^{e}_{g}$ to denote
the gray image in domain $e$, $Y^{e}_{g}$ to denote the corresponding label of
$X^{e}_{g}$, $X^{e}$ to denote the colored image in domain $e$, and $Y^{e}$ to
denote the corresponding label of $X^{e}$. Each domain is specified by a bias
parameter $\theta^{e}$ changing from domain to domain. We use $C^{e}$ to
denote the color index, i.e., the color assigned to $X^{e}_{g}$ to produce
$X^{e}$. Let $\oplus$ denote the XOR operator, and $\textup{Bern}(p)$ denote
the Bernoulli distribution with parameter $p$. Of these three domains, two are
selected as seen domains while the rest is unseen domain.

Colored MNIST (CMNIST) [12] is a variant of the gray MNIST handwritten digit
dataset. The task is to classify a colored digit (in red or blue color) into
two classes: the digit is strictly less than five or the digit is greater or
equal five. There are two seen domains each contains 25,000 images and one
unseen domain contains 20,000 images. By adding the color (spurious feature)
to the digit, the label is more correlated with the color than with the digit,
thus, any algorithm simply aims to minimize the training error will tend to
discover the color rather than the shape of the digit and fail at the testing
phase.

Figure 1: Graphical model for CMNIST.

The graphical model for CMNIST dataset is illustrated in Fig. 1 that contains
the following steps:

  1. 1.

$Y_{g}^{e}\leftarrow L(X_{g}^{e})$: from 10 digits, we construct a binary
classification problem by labeling $Y_{g}^{e}=0$ if the digit is less than or
equal to four and labeling $Y_{g}^{e}=1$, otherwise.

  2. 2.

$Y^{e}\leftarrow Y^{e}_{g}\oplus N$, $N\sim\textup{Bern}(0.25)$: noise is
added to the label $Y^{e}$ of the colored image.

  3. 3.

$C^{e}\leftarrow Y^{e}\oplus\theta^{e}$, $\theta^{e}\sim\textup{Bern}(p^{e})$:
the index color $C^{e}$ is selected with a domain bias parameter $\theta^{e}$.
For $e=1,2,3$, $p^{e}=0.1,0.2,0.9$, respectively.

  4. 4.

$X^{e}\leftarrow T(X_{g}^{e},C^{e})$: the gray image $X^{e}_{g}$ is colored
with the index color $C^{e}$ to produce the colored image $X^{e}$.

Covariate-Shift-CMNIST (CS-CMNIST) [24] is a synthetic dataset derived from
CMNIST. This dataset contains three domains (two training domains and one test
domain) having 20,000 images each. There are ten classes in CS-CMNIST where
each class corresponds to a digit from one to nine. Each class is associated
with one color that is strongly correlated with the digits in the two seen
(training) domains but is weakly correlated with the digits in the unseen
(test) domain.

Figure 2: Graphical model for CS-CMNIST.

The graphical model for CS-CMNIST dataset is illustrated in Fig. 2 that
contains the following steps:

  1. 1.

$Y_{g}^{e}\leftarrow L(X_{g}^{e})$: the label of gray image is a function of
gray image. There are 10 classes with labels from 0 to 9.

  2. 2.

$C^{e}\leftarrow\textup{Uniform}(\\{0,\dots,9\\})$: 10 colors are picked up
randomly (from color 0 to color 9).

  3. 3.

$U^{e}\leftarrow\textup{Bern}\Big{(}(C^{e}\oplus
Y^{e}_{g})\theta^{e}+(1-(C^{e}\oplus Y^{e}_{g}))(1-\theta^{e})\Big{)}$: a pair
of image and its color $(X^{e}_{g},C^{e})$ is selected with probability
$\theta^{e}$ if $Y_{g}^{e}=C^{e}$, else if $Y_{g}^{e}\neq C^{e}$, select
$(X^{e}_{g},C^{e})$ with probability $1-\theta^{e}$. For $e=1,2,3$, the bias
parameter $\theta^{e}=0.1,0.2,0.9$, respectively.

  4. 4.

$X^{e}\leftarrow T(X_{g}^{e},C^{e})|U^{e}=1$, $Y^{e}\leftarrow
Y^{e}_{g}|U^{e}=1$: the gray image $X^{e}_{g}$ is colored with color $C^{e}$
to produce colored image $X^{e}$. The colored image is labeled by $Y^{e}_{g}$.

From the graphical models in Fig. 1 and 2, it is worth noting that while
concept-shift is common in both CMNIST and CS-CMNIST, CS-CMNIST also suffers
from covariate-shift (the third steps in its data generating process).

###  6.2 Compared Methods

We compare our proposed algorithms CORAL-CEM and MMD-CEM to ERM [30], IRM
[12], IB-ERM [14], IB-IRM [14], MMD-IRM [26], and CEM [23]. Since the code for
MMD-IRM algorithm was not released by its authors, we implemented this
algorithm according to the description in [26].

###  6.3 Implementation

For the CS-CMNIST dataset, we follow the learning model in [14] that contains
three convolutional layers with the corresponding feature dimensions of $256$,
$128$, and $64$. We use a stochastic gradient descent optimizer for training,
the batch size is set to $128$, the learning rate is $10^{-1}$ and decays
after $600$ steps while the total number of steps is $2,000$. 25 trials
corresponding to 25 pairs of hyper-parameters $(\alpha,\beta)$ are uniformly
selected from $[10^{-1},10^{4}]$.

For the CMNIST dataset, we follow the setting in [31] and use MNIST-ConvNet
with four convolutional layers as the learning model. The details of MNIST-
ConvNet can be found in Table 7 of [31]. 10 trials corresponding to 10 pairs
of hyper-parameters $(\alpha,\beta)$ are uniformly selected in
$[10^{-1},10^{4}]$. For each trial, the learning rate is randomly selected in
$[10^{-4.5},10^{-3.5}]$ while the batch size is randomly selected in
$[2^{3},2^{9}]$.

We use the training-domain validation set tuning procedure [31] for model
selection, i.e., selecting the model with the highest validation accuracy on
the validation set sampled from seen domain data. We repeat our entire
experiment five times for CS-CMNIST dataset and three times for CMNIST
dataset. Finally, the average accuracy and its standard deviation are
reported. Due to the limited space, the details of our implementation together
with the source code can be found at this
link444https://github.com/thuan2412/Joint-covariate-alignment-and-concept-
alignment-for-domain-generalization.

###  6.4 Results and Discussion

Datasets | ERM [30] | IRM [12] | IB-ERM [14] | IB-IRM [14] | MMD-IRM [26] | CEM [23] | MMD-CEM (our) | CORAL-CEM (our)
---|---|---|---|---|---|---|---|---
CMNIST [12] | 51.2 ± 0.1 | 51.4 ± 0.1 | 51.2 ± 0.3 | 52.1 ± 0.2 | 51.4 ± 0.1 | 52.5 ± 0.3 | 52.0 ± 0.2 | 52.5 ± 0.1
CS-CMNIST [24] | 60.3 ± 1.2 | 62.5 ± 1.1 | 71.5 ± 0.7 | 71.9 ± 0.7 | 77.2 ± 0.9 | 86.1 ± 0.8 | 90.7 ± 0.9 | 89.9 ± 0.6

Table 1: Average accuracy of the compared methods.

Table 1 shows the average accuracy of the compared methods. As seen, our
proposed algorithms outperform the rest of the tested methods with a margin of
nearly 4% for CS-CMNIST dataset. However, when evaluating on a more
challenging CMNIST dataset, the proposed algorithms only achieve comparable or
slightly better performance than other tested methods. The substantial gain on
CS-CMNIST can be explained by the way the data is generated. Indeed, by
construction, CS-CMNIST suffers from both covariate-shift and concept-shift,
therefore, it is necessary for achieving both covariate and concept alignment
for better DG on CS-CMNIST. On the other hand, because CMNIST is designed in a
way such that there exists a strong spurious correlation between colors and
labels, no algorithm works on this dataset. This observation is also confirmed
by the numerical results in [31].

##  7 Conclusions

In this paper, by revisiting the upper bound of generalization error of the
unseen domain, we motivate a domain generalization framework that combines
covariate-alignment algorithms together with concept-alignment algorithms to
simultaneously handle both the covariate distribution shift and the concept
distribution shift. Our numerical results show a gain on generalization
performance which confirms our theoretical results.

## References

  * [1] J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, and P. Yu, “Generalizing to unseen domains: A survey on domain generalization,” _IEEE Transactions on Knowledge and Data Engineering_ , 2022.
  * [2] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy, “Domain generalization: A survey,” _arXiv preprint arXiv:2103.02503_ , 2021.
  * [3] R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt, “Measuring robustness to natural distribution shifts in image classification,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 18 583–18 599, 2020.
  * [4] H. Li, S. J. Pan, S. Wang, and A. C. Kot, “Domain generalization with adversarial feature learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5400–5409.
  * [5] M. Ghifary, D. Balduzzi, W. B. Kleijn, and M. Zhang, “Scatter component analysis: A unified framework for domain adaptation and domain generalization,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 7, pp. 1414–1430, 2016.
  * [6] S. Erfani, M. Baktashmotlagh, M. Moshtaghi, X. Nguyen, C. Leckie, J. Bailey, and R. Kotagiri, “Robust domain generalisation by enforcing distribution invariance,” in _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI-16)_. AAAI Press, 2016, pp. 1455–1461.
  * [7] X. Jin, C. Lan, W. Zeng, and Z. Chen, “Feature alignment and restoration for domain generalization and adaptation,” _arXiv preprint arXiv:2006.12009_ , 2020.
  * [8] G. Blanchard, A. A. Deshmukh, U. Dogan, G. Lee, and C. Scott, “Domain generalization by marginal transfer learning,” _Journal of Machine Learning Research_ , vol. 22, pp. 1–55, 2021.
  * [9] M.-H. Bui, T. Tran, A. Tran, and D. Phung, “Exploiting domain-specific features to enhance domain generalization,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [10] J. Shen, Y. Qu, W. Zhang, and Y. Yu, “Wasserstein distance guided representation learning for domain adaptation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [11] B. Lyu, T. Nguyen, P. Ishwar, M. Scheutz, and S. Aeron, “Barycentric-alignment and invertibility for domain generalization,” _arXiv preprint arXiv:2109.01902_ , 2021.
  * [12] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz, “Invariant risk minimization,” _arXiv preprint arXiv:1907.02893_ , 2019.
  * [13] B. Li, Y. Shen, Y. Wang, W. Zhu, C. J. Reed, J. Zhang, D. Li, K. Keutzer, and H. Zhao, “Invariant information bottleneck for domain generalization,” _arXiv preprint arXiv:2106.06333_ , 2021.
  * [14] K. Ahuja, E. Caballero, D. Zhang, J.-C. Gagnon-Audet, Y. Bengio, I. Mitliagkas, and I. Rish, “Invariance principle meets information bottleneck for out-of-distribution generalization,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 3438–3450, 2021.
  * [15] Z. Wang, M. Loog, and J. van Gemert, “Respecting domain relations: Hypothesis invariance for domain generalization,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 9756–9763.
  * [16] S. Zhao, M. Gong, T. Liu, H. Fu, and D. Tao, “Domain generalization via entropy regularization,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 16 096–16 107, 2020.
  * [17] O. E. Salaudeen and O. O. Koyejo, “Exploiting causal chains for domain generalization,” in _NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications_ , 2021.
  * [18] C. Lu, Y. Wu, J. M. Hernández-Lobato, and B. Schölkopf, “Nonlinear invariant risk minimization: A causal approach,” _arXiv preprint arXiv:2102.12353_ , 2021.
  * [19] K. Ahuja, K. Shanmugam, K. Varshney, and A. Dhurandhar, “Invariant risk minimization games,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 145–155.
  * [20] W. M. Kouw and M. Loog, “An introduction to domain adaptation and transfer learning,” _arXiv preprint arXiv:1812.11806_ , 2018.
  * [21] S. Ben-David, J. Blitzer, K. Crammer, F. Pereira _et al._ , “Analysis of representations for domain adaptation,” _Advances in neural information processing systems_ , vol. 19, p. 137, 2007.
  * [22] B. Sun and K. Saenko, “Deep coral: Correlation alignment for deep domain adaptation,” in _European conference on computer vision_. Springer, 2016, pp. 443–450.
  * [23] T. Nguyen, B. Lyu, P. Ishwar, M. Scheutz, and S. Aeron, “Conditional entropy minimization principle for learning domain invariant representation features,” _arXiv preprint arXiv:2201.10460_ , 2022.
  * [24] K. Ahuja, J. Wang, A. Dhurandhar, K. Shanmugam, and K. R. Varshney, “Empirical or invariant risk minimization? a sample complexity perspective,” in _International Conference on Learning Representations_ , 2021.
  * [25] A. T. Nguyen, T. Tran, Y. Gal, and A. G. Baydin, “Domain invariant representation learning with domain density transformations,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [26] R. Guo, P. Zhang, H. Liu, and E. Kiciman, “Out-of-distribution prediction with invariant risk minimization: The limitation and an effective fix,” _arXiv preprint arXiv:2101.07732_ , 2021.
  * [27] W. Yang, T. Ling, C. Yang, L. Wang, Y. Shi, L. Zhou, and M. Yang, “Class distribution alignment for adversarial domain adaptation,” _arXiv preprint arXiv:2004.09403_ , 2020.
  * [28] J. Wen, N. Zheng, J. Yuan, Z. Gong, and C. Chen, “Bayesian uncertainty matching for unsupervised domain adaptation,” in _Proceedings of the 28th International Joint Conference on Artificial Intelligence_ , 2019, pp. 3849–3855.
  * [29] D. Kifer, S. Ben-David, and J. Gehrke, “Detecting change in data streams,” in _VLDB_ , vol. 4. Toronto, Canada, 2004, pp. 180–191.
  * [30] V. N. Vapnik, “An overview of statistical learning theory,” _IEEE transactions on neural networks_ , vol. 10, no. 5, pp. 988–999, 1999.
  * [31] I. Gulrajani and D. Lopez-Paz, “In search of lost domain generalization,” in _International Conference on Learning Representations_ , 2020.