$\displaystyle\hat{f}_{\text{DIP}}^{\text{\scriptsize$(m)$}}(x)$
$\displaystyle:=x^{\top}\hat{\beta}_{\text{DIP}}^{\text{\scriptsize$(m)$}}+\hat{\beta}_{\text{DIP},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle\hat{\beta}_{\text{DIP}}^{\text{\scriptsize$(m)$}},\hat{\beta}_{\text{DIP},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle:=\operatorname*{arg\,min}_{\beta,\beta_{0}}\
\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}\left(y_{i}^{\text{\scriptsize$(m)$}}-{x_{i}^{\text{\scriptsize$(m)$}}}^{\top}\beta-\beta_{0}\right)^{2}+$
$\displaystyle\hskip
85.35826pt\lambda_{\text{match}}\left(\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}{x_{i}^{\text{\scriptsize$(m)$}}}^{\top}\beta-\frac{1}{\tilde{n}}\sum_{i=1}^{\tilde{n}}\tilde{x}_{i}^{\top}\beta\right)^{2},$
(43)

where $\lambda_{\text{match}}$ is a positive regularization parameter that
controls the match between the covariate mean of the source and target
environment.

  * •

CIP-mean-finite: the finite sample formulation of the CIP-mean estimator (• ‣
3.4). The conditional mean is matched across source environments and is
enforced via a regularization term,

$\displaystyle\hat{f}_{\text{CIP}}(x)$
$\displaystyle:=x^{\top}\hat{\beta}_{\text{CIP}}+\hat{\beta}_{\text{CIP},0}$
$\displaystyle\hat{\beta}_{\text{CIP}},\hat{\beta}_{\text{CIP},0}$
$\displaystyle:=\operatorname*{arg\,min}_{\beta,\beta_{0}}\
\frac{1}{M}\sum_{m=1}^{M}\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}\left(y_{i}^{\text{\scriptsize$(m)$}}-{x_{i}^{\text{\scriptsize$(m)$}}}^{\top}\beta-\beta_{0}\right)^{2}+$
$\displaystyle\hskip
85.35826pt\frac{\lambda_{\text{CIP}}}{M^{2}}\sum_{m,k=1}^{M}\left(\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}{z_{\text{cond},i}^{\text{\scriptsize$(m)$}}}^{\top}\beta-\frac{1}{n_{k}}\sum_{i=1}^{n_{k}}{z_{\text{cond},i}^{\text{\scriptsize$(k)$}}}^{\top}\beta\right)^{2},$
(44)

where
$z_{\text{cond},i}^{\text{\scriptsize$(k)$}}=x_{i}^{\text{\scriptsize$(k)$}}-\frac{y_{i}^{\text{\scriptsize$(k)$}}\bar{y}^{\text{\scriptsize$(k)$}}}{\left(\frac{1}{n_{k}}\sum_{j=1}^{n_{k}}{y_{j}^{\text{\scriptsize$(k)$}}}^{2}\right)}{x_{i}^{\text{\scriptsize$(k)$}}}$,
$\bar{y}^{\text{\scriptsize$(k)$}}=\frac{1}{n_{k}}\sum_{j=1}^{n_{k}}y_{j}^{\text{\scriptsize$(k)$}}$
and $\lambda_{\text{CIP}}$ is a positive regularization parameter that
controls the strength of the conditional invariant penalty. In the finite-
sample formulation, the conditional expectation in the constraint of
population formulation (• ‣ 3.4) is computed via regressing $X^{\top}\beta$ on
$Y$. As a result, $z_{\text{cond},i}^{\text{\scriptsize$(k)$}}$’s are the
residuals after regressing $x_{i}^{\text{\scriptsize$(k)$}}$ on
$y_{i}^{{\text{\scriptsize$(k)$}}}$.

  * •

CIRM${}^{\text{\scriptsize$(m)$}}$-mean-finite: the finite-sample formulation
of the CIRM${}^{\text{\scriptsize$(m)$}}$-mean estimator (• ‣ 3.4). The
residual after removing conditionally invariant components is matched between
source and target environments. The matching is enforced via a regularization
term.

$\displaystyle\hat{f}_{\text{CIRM}}^{\text{\scriptsize$(m)$}}(x)$
$\displaystyle:=x^{\top}\hat{\beta}_{\text{CIRM}}^{\text{\scriptsize$(m)$}}+\hat{\beta}_{\text{CIRM},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle\hat{\beta}_{\text{CIRM}}^{\text{\scriptsize$(m)$}},\hat{\beta}_{\text{CIRM},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle:=\operatorname*{arg\,min}_{\beta,\beta_{0}}\
\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}\left(y_{i}^{\text{\scriptsize$(m)$}}-{x_{i}^{\text{\scriptsize$(m)$}}}^{\top}\beta-\beta_{0}\right)^{2}$
$\displaystyle\hskip
85.35826pt+\lambda_{\text{match}}\left(\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}{z_{\text{res},i}^{\text{\scriptsize$(m)$}}}^{\top}\beta-\frac{1}{\tilde{n}}\sum_{i=1}^{\tilde{n}}{\tilde{z}_{\text{res},i}}^{\top}\beta\right)^{2},$
(45)

where
$z_{\text{res},i}^{\text{\scriptsize$(m)$}}=x_{i}^{\text{\scriptsize$(m)$}}-\left({x_{i}^{\text{\scriptsize$(m)$}}}^{\top}\hat{\beta}_{\text{CIP}}\right)\hat{\vartheta}_{\text{CIRM}}$
with $\hat{\vartheta}_{\text{CIRM}}$ defined as

$\displaystyle\hat{\vartheta}_{\text{CIRM}}=\frac{\frac{1}{M}\sum_{m=1}^{M}\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}\left(y_{i}^{\text{\scriptsize$(m)$}}-\bar{y}^{\text{\scriptsize$(m)$}}\right)x_{i}^{\text{\scriptsize$(m)$}}}{\frac{1}{M}\sum_{m=1}^{M}\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}\left(y_{i}^{\text{\scriptsize$(m)$}}-\bar{y}^{\text{\scriptsize$(m)$}}\right)\left({x_{i}^{\text{\scriptsize$(m)$}}}^{\top}\hat{\beta}_{\text{CIP}}-\frac{1}{n_{m}}\sum_{j=1}^{n_{m}}{x_{j}^{\text{\scriptsize$(m)$}}}^{\top}\hat{\beta}_{\text{CIP}}\right)},$

and $\lambda_{\text{match}}$ is a positive regularization parameter similar to
the one define in DIP. Just like the population CIRM depends on the population
CIP, the finite-sample CIRM also depends on the finite-sample CIP with the
same regularization parameter $\lambda_{\text{CIP}}$.

##### Regularization parameter choices:

Choosing the regularization parameters in finite-sample DA formulations such
as $\lambda_{\text{match}}$ in DIP formulation (• ‣ 5.1) is a difficult
subject. Because the DA setting here assumes no access to any target labels,
one cannot get good estimates of the target performance easily. The classical
model selection strategies based on a validation set are no longer applicable
in domain adaptation.

We make use of the fact that when DIP works as Theorem 1 predicts, the source
population risk of DIP equals to the target population risk as shown in
Corollary 3. The source finite-sample risk is close to the target finite-
sample risk up to finite sample errors. So we would like to choose
$\lambda_{\text{match}}$ large enough so that the DIP matching penalty takes
effect, but not too large so that the source finite risk remains reasonably
small. We propose to choose the largest $\lambda_{\text{match}}$ so that the
source finite-sample risk is less than two times of the source finite-sample
risk when $\lambda_{\text{match}}$ is set to zero. The choice of “two times”
is arbitrary here, the precise amount depends on the desired target finite-
sample risk, the sample size and the variance of the source finite risk
estimate. The precise amount is specified for each DA dataset separately. The
regularization parameter $\lambda_{\text{match}}$ in DIPweigh is chosen
similarly.

Next, we consider the $\lambda_{\text{CIP}}$ choice in CIP formulation (• ‣
5.1). Since CIP only uses source data and never touches the target data, we
leave a small part of each source data out for validation and choose
$\lambda_{\text{CIP}}$ based on the validation source data. We choose
$\lambda_{\text{CIP}}$ so that the average source risk across source
environment is small.

Finally, the finite-sample CIRM formulation (• ‣ 5.1) requires the choice of
both $\lambda_{\text{match}}$ and $\lambda_{\text{CIP}}$. The
$\lambda_{\text{CIP}}$ in CIRM is the same as the $\lambda_{\text{CIP}}$ in
CIP. With $\lambda_{\text{CIP}}$ fixed, we choose $\lambda_{\text{match}}$ in
CIRM as we did for DIP.

###  5.2 Linear SCM simulations

In this section, we numerically compare DA estimators via simulations from
linear SCMs. First, we consider seven simulations with data generated via
linear SCMs with mean shift noise interventions. The first three simulations
aim at illustrating the results in Theorem 1 and Theorem 5. The last four are
to show the performance of DA estimators when at least one assumption is
misspecified. Second, we consider two simulations where the interventions are
variance shift noise interventions. Through these two simulations, we
demonstrate the necessity of adapting the DIP matching penalties according to
the type of interventions. The simulation settings are summarized in Table 4.

Sim Num |  # Src envs |  Causal Direction |  Interv X type |  Interv on Y? |  Has CIC? |  Better estimator(s) |  Baseline estimator(s)
---|---|---|---|---|---|---|---
(i) | single | anticausal | mean shift | N | - | DIP${}^{\text{\scriptsize$(1)$}}$ | OLSSrc${}^{\text{\scriptsize$(1)$}}$
(ii) | multiple | anticausal | mean shift | N | - | DIPweigh |  OLSPool DIP$(1)$
(iii) | multiple | anticausal | mean shift | Y | Y | CIRMweigh |  OLSPool DIPweigh
(iv) | single | causal | mean shift | N | - | - | OLSSrc${}^{\text{\scriptsize$(1)$}}$
(v) | single | mixed | mean shift | N | - | DIP$\diamondsuit^{\text{\scriptsize$(1)$}}$ |  OLSSrc${}^{\text{\scriptsize$(1)$}}$ DIP${}^{\text{\scriptsize$(1)$}}$
(vi) | multiple | anticausal | mean shift | Y | N | - | OLSPool
(vii) | multiple | mixed | mean shift | Y | Y | CIRM$\diamondsuit$weigh |  OLSPool CIRMweigh
(viii) | single | anticausal | var shift | N | - |  DIP-std+ DIP-MMD |  OLSSrc${}^{\text{\scriptsize$(1)$}}$ DIP
(ix) | multiple | anticausal | var shift | Y | Y |  CIRMweigh-std+ CIRMweigh-MMD | OLSPool
Table 4: List of linear SCM simulations. The first three simulations are done
under the correct assumptions according to the main theorems. Starting from
Simulation (iv), some assumptions are modified to show the performance of DA
methods under misspecified assumptions (highlighted in red). The last two
simulations are done with variance shift noise intervention. “Y” means yes,
“N” means no, “-” means the information is irrelevant in that setting.

####  5.2.1 Linear SCM with mean shift noise interventions

We consider seven simulations with data generated via linear SCMs with mean
shift noise interventions. In all experiments in this subsection, we use large
sample size $n=5000$ for each environment, $n=5000$ target test data for the
evaluation of the target risk and we fix the regularization parameters
$\lambda_{\text{match}}=10.0$ and $\lambda_{\text{CIP}}=1.0$ to focus the
discussions on the choice of DA estimators.

##### (i) Single source anticausal DA without Y intervention:

We consider three datasets of dimension $d=3,10,20$. In each dataset, there is
one source environment and one target environment generated according to
Assumption 1. The matrix $\mathbf{B}$ and the vector ${b}$ is specified as
follows

  * •

For $d=3$, the matrix $\mathbf{B}=0$, ${b}=\left[1,-1,3\right]^{\top}$ and
${\omega}=0$.

  * •

For $d=10,20$, the matrix $\mathbf{B}$ is upper triangular with diagonal zero
and with each entry drawn i.i.d. from $\mathcal{N}(0,0.25)$. Each entry of
${b}$ i.i.d. from $\mathcal{N}(0,1)$. ${\omega}=0$.

The interventions $a^{\text{\scriptsize$(1)$}}_{X}$ and $\widetilde{a}_{X}$
are generated i.i.d. from $\mathcal{N}(0,\mathbb{I}_{d})$ and stay the same
for all data points in one environment. For each data point,
$\varepsilon^{\text{\scriptsize$(1)$}}$ or $\varepsilon$ is generated i.i.d.
from the standard Gaussian distribution $\mathcal{N}(0,\mathbb{I}_{d+1})$. For
each dataset, the matrix $\mathbf{B}$ and the vector ${b}$ of the SCM are
generated once; the noise and interventions are generated i.i.d. $10$ times
with source and target sample size $n=\tilde{n}=5000$. The boxplot of the 10
target risks is reported for OLSTar, OLSSrc${}^{\text{\scriptsize$(1)$}}$,
DIP${}^{\text{\scriptsize$(1)$}}$ and DIPOracle${}^{\text{\scriptsize$(1)$}}$
in Figure 7. The target risk of OLSTar is highlighted in red dashed horizontal
line. The target risk of OLSSrc${}^{\text{\scriptsize$(1)$}}$ is highlighted
in blue dashed horizontal line. The first three plots in Figure 7 show that
the target risk of DIP${}^{\text{\scriptsize$(1)$}}$ is similar to that of
DIPOracle${}^{\text{\scriptsize$(1)$}}$ and it is very close to OLSTar as
predicted by Theorem 1.

The target risk performance of DA estimators depends on the how the matrix
$\mathbf{B}$ and the vector ${b}$ are generated. To make the comparison less
dependent on the randomness in the matrix $\mathbf{B}$ and the vector ${b}$,
we complement the boxplot with a scatterplot that compares the target risks of
DIP${}^{\text{\scriptsize$(1)$}}$ and OLSSrc${}^{\text{\scriptsize$(1)$}}$
over 100 random generations of the matrix $\mathbf{B}$ for $d=10$. The last
plot in Figure 7 shows that in 97 out of 100 random data generations,
DIP${}^{\text{\scriptsize$(1)$}}$ has lower target risk than
OLSSrc${}^{\text{\scriptsize$(1)$}}$.

Figure 7: The target risk comparison in simulation (i) single source
anticausal DA without Y intervention (the lower the better). In all three
datasets ($d=3,10,20$) in the left three plots,
DIP${}^{\text{\scriptsize$(1)$}}$ has lower target risk than
OLSSrc${}^{\text{\scriptsize$(1)$}}$. The target risk of
DIP${}^{\text{\scriptsize$(1)$}}$ is close to that of
DIPOracle${}^{\text{\scriptsize$(1)$}}$. The last plot shows that in 98 out of
100 random coefficient $\mathbf{B}$ data generations ($d=10$),
DIP${}^{\text{\scriptsize$(1)$}}$ has lower target risk than
OLSSrc${}^{\text{\scriptsize$(1)$}}$.

##### (ii) Multiple source anticausal DA without Y intervention:

We consider three datasets of covariate dimension $d=3,10,20$. In each
dataset, there is $M\geq 1$ source environments and one target environment
generated according to Assumption 1. The matrix $\mathbf{B}$, the vector
${b}$, the interventions $a$ and noises $\varepsilon$ are generated as in
simulation (i). The only difference is the availability of multiple ($M\geq
1$) source environments.

The boxplot of the 10 target risks are reported for OLSTar,
DIP${}^{\text{\scriptsize$(1)$}}$ and DIPweigh with increasing number of
source environments in Figure 8. The constant $\rho$ in DIPweigh formulation
(• ‣ 4.1.2) is fixed to be $1000$. To make the comparison less dependent on
the randomness in the matrix $\mathbf{B}$, we complement the boxplot with a
scatterplot that compares the target risks of DIPweigh($M=8$) and
DIP${}^{\text{\scriptsize$(1)$}}$ over 100 random generations of the matrix
$\mathbf{B}$ for $d=10$. Figure 8 shows that in the anticausal DA setting
without Y intervention, the more source environments the lower the target risk
DIPweigh can achieve.

Figure 8: The target risk comparison in simulation (ii) multiple source
anticausal DA without Y intervention (the lower the better). The left three
plots show that with more number of source environments, DIPweigh perform
better. The last plot shows that in 89 out of 100 simulations, DIPweigh with
$M=8$ has lower target risk than DIP${}^{\text{\scriptsize$(1)$}}$.

##### (iii) Multiple source anticausal DA with Y intervention and with CICs:

We fix the dimension $d=20$ and the number of source environments $M=14$. The
source and target datasets are generated similarly to simulation (ii) with two
exceptions: 1. there are interventions on $Y$ for the target environment:
$\widetilde{a}_{Y}$ follows $\mathcal{N}(0,1)$ and it is generated once then
it is fixed for the target environment. 2. there are conditionally invariant
components (CICs): the interventions on $X$ only apply to first $10$
coordinates. That is, $a^{\text{\scriptsize$(m)$}}_{X}$ and
$\widetilde{a}_{X}$ have the last $10$ coordinates equal to zero. Note that we
don’t explicitly assume Equation (27) in Assumption 2. Instead we expect that
a large number of source environments will span the vector space with last
$10$ coordinates zero and make the assumption (27) satisfied.

The left most plot in Figure 9 compares the boxplot of the target risks of
OLSTar, SrcPool, DIPweigh, CIP and CIRMweigh with one time generation of
$\mathbf{B}$ and $10$ random generations of the interventions and the noises.
CIRMweigh has lower target risk than SrcPool and DIPweigh. The right three
plots in Figure 9 show the scatterplots comparing the pair-wise target risks
in 100 random generations of $\mathbf{B}$ and the interventions for the
following pairs: CIP vs DIPweigh, CIRMweigh vis SrcPool. CIRMweigh vs
DIPweigh. The number of points below the diagonal are reported in the titles.

Both CIP and CIRM have lower target risk than DIPweigh. DIPweigh loses target
risk guarantees because of the intervention on $Y$. CIRMweigh also outperforms
SrcPool. Note that due to finite sample errors, there are rare cases (about
5%) where CIRM does not outperform SrcPool or DIPweigh.

Figure 9: Target risk comparison in simulation (iii) multiple source
anticausal DA with Y intervention and with CICs (the lower the better).
CIRMweigh has smaller target risk than SrcPool. In the presence of Y
intervention, the target risk of DIPweigh can be much larger than that of
SrcPool. The scatterplots are for 100 random coefficient $\mathbf{B}$ data
generations.

##### (iv) Single source causal DA without Y intervention:

We consider three datasets of covariate dimension $d=3,10,20$ similar to
simulation (i) except that the prediction direction is changed from anticausal
to causal: ${\omega}\neq 0$ and ${b}=0$. Figure 10 shows that
DIP${}^{\text{\scriptsize$(1)$}}$ can have larger target risk than
OLSSrc${}^{\text{\scriptsize$(1)$}}$. This simulation result makes it clear
that DIP is not useful for causal DA in general. Example 3 in Section 3.5.3 is
not just a pathological failure example of DIP.

Figure 10: Target risk comparison in simulation (iv) single source causal
domain adaptation without Y intervention (the lower the better).
DIP${}^{\text{\scriptsize$(1)$}}$ always has larger target risk than
OLSSrc${}^{\text{\scriptsize$(1)$}}$ in the causal domain adaptation problem
over 100 random coefficient B data generations ($d=10$).

##### (v) Single source  mixed DA without Y intervention:

We consider two datasets of covariate dimension $d=10,20$ similar to
simulation (i) except that the prediction direction is changed from anticausal
to mixed-causal-anticausal: The odd coordinates of ${b}$ are set to zero and
the even coordinates are nonzero. For ${\omega}$, it is the contrary: the odd
coordinates of ${\omega}$ are nonzero and the even ones are zero. The matrix
$\mathbf{B}$ is generated according to Assumption 3 with one block zero. The
left two plots in Figure 11 show boxplots of target risks of OLSTar,
OLSSrc${}^{\text{\scriptsize$(1)$}}$, DIPOracle${}^{\text{\scriptsize$(1)$}}$,
DIP${}^{\text{\scriptsize$(1)$}}$ and
DIP$\diamondsuit^{\text{\scriptsize$(1)$}}$ for random generation of the
matrix $\mathbf{B}$. DIP$\diamondsuit^{\text{\scriptsize$(1)$}}$ uses the
ground-truth knowledge of the nonzero coordinates of ${b}$. In mixed causal-
anticausal DA, DIP${}^{\text{\scriptsize$(1)$}}$ has larger risk than
OLSSrc${}^{\text{\scriptsize$(1)$}}$. With the ground-truth knowledge of the
causal variables, DIP$\diamondsuit^{\text{\scriptsize$(1)$}}$ still have lower
risk than OLSSrc${}^{\text{\scriptsize$(1)$}}$. The right two plots in Figure
11 confirms the result via scatterplot comparisons of
OLSSrc${}^{\text{\scriptsize$(1)$}}$, DIPOracle${}^{\text{\scriptsize$(1)$}}$,
DIP${}^{\text{\scriptsize$(1)$}}$ and
DIP$\diamondsuit^{\text{\scriptsize$(1)$}}$ after 100 runs.

Figure 11: Target risk comparison in simulation (v) single source mixed DA
without Y intervention (the lower the better).
DIP${}^{\text{\scriptsize$(1)$}}$ has larger target risk than
OLSSrc${}^{\text{\scriptsize$(1)$}}$ in the mixed causal anticausal DA
setting. DIP$\diamondsuit^{\text{\scriptsize$(1)$}}$ with ground-truth causal
variables outperforms OLSSrc${}^{\text{\scriptsize$(1)$}}$. The two
scatterplots are over 100 random coefficient $\mathbf{B}$ data generations
$(d=10)$.

##### (vi) Multiple source anticausal DA with Y intervention and  without
CICs:

We consider a simulation ($d=20$, $M=14$) similar to simulation (iii) except
that there are no conditionally invariant components (CICs). Figure 12 shows
that without CICs, CIRMweigh no longer outperforms SrcPool. Only in 64 of 100
simulations CIRMweigh has lower target risk than SrcPool. Interestingly,
CIRMweigh still has a large chance of having smaller target risk than
DIPweigh. This may be due to the fact that even though there are no pure
conditional invariant components, CIP may still be able to pick a combination
of covariates that is relatively less sensitive to $X$ interventions.

Figure 12: Target risk comparison in simulation (vi) multiple source
anticausal DA with Y intervention and without CICs (the lower the better).
Without CICs, CIRMweigh no longer outperforms SrcPool.

##### (vii) Multiple source  mixed DA with Y intervention and with CICs:

We consider a simulation ($d=20$, $M=14$) similar to simulation (iii) except
that the prediction direction is changed from anticausal to mixed causal
anticausal. The odd coordinates of ${b}$ is set to zero and the even ones are
nonzero. The even coordinates of ${\omega}$ is set to zero and the odd ones
are nonzero. The first five even coordinates are set to be anticausal CICs.
Additionally, we tuned down the variance of
$\varepsilon^{\text{\scriptsize$(m)$}}_{Y}$ to be 0.01 to make the causal part
important for prediction. Figure 13 shows that in mixed causal and anticausal
DA, CIRMweigh has lower target risk than SrcPool only in 64 of 100
simulations. However, with the true causal covariates, the oracle estimator
CIRM$\diamondsuit$weigh has lower target risk than SrcPool in 96 of 100
simulations.

Figure 13: Target risk comparison in simulation (vii) multiple source mixed DA
with Y intervention and with CICs (the lower the better). The scatterplots are
over 100 random coefficient $\mathbf{B}$ data generations. CIRMweigh has lower
target risk than SrcPool only in 64 of 100 simulations. However, knowing the
indexes of the true causal covariates, the oracle estimator
CIRM$\diamondsuit$weigh has lower target risk than SrcPool in 96 of 100
simulations. CIRM$\diamondsuit$weigh also outperforms DIPweigh and CIRMweigh.
The scatterplots are over 100 random coefficient $\mathbf{B}$ data
generations.

####  5.2.2 Linear SCM with variance shift noise interventions

We consider two simulations with data generated via linear SCMs with variance
shift noise interventions. Since DIP-std+ and DIP-MMD are involved, we have to
adapt the regularization parameter choice strategy described at the beginning
of Section 5. We apply all DIP variants with the regularization parameter
$\lambda_{\text{match}}$ ranging in the set
$\left\\{10^{-5},10^{-4},\cdots,10^{4},10^{5}\right\\}$. We choose the largest
$\lambda_{\text{match}}$ such that the source risk is smaller than
$\min(2r,r+0.01)$ as the final parameter. Here $r$ is the source risk of
OLSSrc. Based on the average source risk, $\lambda_{\text{CIP}}$ is still
chosen to be $1.0$.

Since DIP-std+, DIP-MMD, CIRMweigh-std+ and CIRMweigh-MMD do not have closed
form solutions, we use Pytorch’s gradient descent to optimize these methods.
Specifically, Adam’s optimizer is used with step-size $10^{-4}$ and iteration
number 2000 epochs to ensure convergence.

##### (viii) Single source anticausal DA without Y intervention + variance
shift noise intervention:

We consider a simulation ($d=10$) similar to simulation (i) except that the
type of intervention is changed from mean shift noise intervention to variance
shift noise intervention. The intervention affects the variance of the noise
$g(a^{\text{\scriptsize$(1)$}}_{x},\varepsilon^{\text{\scriptsize$(1)$}}_{X})=a^{\text{\scriptsize$(1)$}}_{X}\odot\varepsilon^{\text{\scriptsize$(1)$}}_{X}$.
The interventions $a^{\text{\scriptsize$(1)$}}_{X}$ and $\widetilde{a}_{X}$
are still generated i.i.d. from $\mathcal{N}(0,\mathbb{I}_{d})$ and stay the
same for all data points in one environment. The left-most plot in Figure 14
shows the boxplot of the target risks of OLSTar,
OLSSrc${}^{\text{\scriptsize$(1)$}}$, DIP${}^{\text{\scriptsize$(1)$}}$-mean,
DIP${}^{\text{\scriptsize$(1)$}}$-std+, DIP${}^{\text{\scriptsize$(1)$}}$-MMD.
DIP${}^{\text{\scriptsize$(1)$}}$-mean has the same target risk and
OLSSrc${}^{\text{\scriptsize$(1)$}}$, because the DIP matching penalty on mean
has no effect when the intervention is variance shift noise intervention.
DIP${}^{\text{\scriptsize$(1)$}}$-std+ and
DIP${}^{\text{\scriptsize$(1)$}}$-MMD improves upon
DIP${}^{\text{\scriptsize$(1)$}}$-mean. The right three plots in Figure 14
show scatterplots of 100 random coefficient $\mathbf{B}$ data generations to
confirm this observation.

Figure 14: Target risk comparison in simulation (viii) single source causal
domain adaptation with variance intervention without Y intervention (the lower
the better). Left: boxplots of the target risks. DIP-mean has the same target
risk than Src[1]. DIP-std+ and DIP-MMD outperform Src[1]. Right: three
scatterplots of 100 runs comparing DIP-mean, DIP-std+ and DIP-MMD with Src[1].

##### (ix) Multiple source anticausal DA with Y intervention + variance shift
noise intervention:

Figure 15: Target risk comparison in simulation (ix) multiples source causal
domain adaptation with variance intervention with Y intervention and with
conditionally invariant components (CICs) (the lower the better). CIRMweigh-
std+ and CIRMweigh-MMD outperform SrcPool. CIRMweigh-mean does not outperform
SrcPool, because it can not handle the variance shift noise intervention.
DIPweigh-MMD performs much worse than SrcPool because of the intervention on
$Y$.

We consider a simulation (d = 20, M = 14) similar to simulation (iii) except
that the type of intervention is changed from mean shift noise intervention to
variance shift noise intervention. The variance shift noise intervention on
$X$ is as specified simulation (viii). The intervention on $Y$ is still mean
shift noise intervention as in simulation (iii). The scatterplots in Figure 15
compares CIRMweigh-mean, CIRMweigh-std+, DIPweigh-MMD, CIRMweigh-MMD with
SrcPool in 100 runs. CIRMweigh-std+ and CIRMweigh-MMD outperform SrcPool.
CIRMweigh-mean does not outperform SrcPool, because it can not handle the
variance shift noise intervention. DIPweigh-MMD performs much worse than
SrcPool because of the intervention on $Y$.

###  5.3 MNIST experiments with synthetic image interventions

In this section, we generate synthetic image classification datasets by
modifying the images from the MNIST dataset (LeCun, 1998). Then we compare the
performance of various DA methods on this digit classification task. The
interventions are added directly on the image pixels. The DA methods are
applied to the last-layer features of a pre-trained CNN on the original MNIST
dataset. Since our DA methods are only defined for regression problems in the
previous sections, for this classification problem, we adapt the loss function
to softmax loss and we use one-hot encoding of the label $Y$ in the DIP, CIP
and CIRM formulations.

For the MNIST experiments with synthetic image interventions, a priori the
linear SCM assumption is no longer valid and the interventions are not
necessarily noise interventions. Here we show that the DA methods such as DIP,
CIRM still have target performance as predicted by our theorems despite the
likely violation of several assumptions.

All the methods in this subsection are implemented via Pytorch (Paszke et al.,
2019) and are optimized with Pytorch’s stochastic gradient descent.
Specifically, stochastic gradient descent (SGD) optimizer is used with step-
size (or learning rate) $10^{-4}$, batchsize $500$ and number of epochs $100$.

####  5.3.1 MNIST with patch intervention

##### MNIST DA with patch intervention without Y intervention:

We take the original MNIST dataset with 60000 training samples and create two
synthetic datasets (source and target) as follows. For the source dataset,
each training image is masked by the mask (a) in Figure 16. That is, for each
training image, the pixels at the white region of the mask (a) are set to
white (maximum pixel value). For the target dataset environment, each training
image is masked by the mask (b) in Figure 16. For each experiment, we take
random $20\%$ of samples from the source dataset as the source environment and
random $20\%$ of samples from the target dataset as the target environment.
The task is to predict the labels of the images in the target environment
without observing any target labels. This experiment is repeated $10$ times
and we report the boxplot of the $10$ target classification accuracies for
each DA method.

We apply the DA methods on the last-layer features of a pre-trained
convolutional neural network (CNN). The CNN has two convolutional layers and
two fully connected layers similar to the LeNet-5 architecture. It is pre-
trained on the original MNIST dataset with test accuracy on the separate 10000
test images from the MNIST dataset being 98.8%. A priori, it is not clear what
type of interventions on the last-layer features happen when we only know the
intervention is on the pixels. To fit into our theoretical and conceptual
framework, these induced interventions on the last-layer features would need
to be approximated by shift noise interventions. The following DA methods are
applied:

  * •

Original: CNN model pre-trained on the original MNIST dataset without any
modification.

  * •

Tar: oracle CNN model trained only on the target environment. It is similar to
OLSTar because only the weights in the last layer are trained. We changed the
name to Tar because the full model is a neural network.

  * •

Src${}^{\text{\scriptsize$(1)$}}$: CNN model trained only on the source
environment.

  * •

DIP${}^{\text{\scriptsize$(1)$}}$: CNN model where
DIP${}^{\text{\scriptsize$(1)$}}$-mean-finite is applied using the source
label and the last-layer features of source and target environments.

  * •

DIP${}^{\text{\scriptsize$(1)$}}$-MMD: oracle CNN model where where
DIP${}^{\text{\scriptsize$(1)$}}$-MMD-finite is applied using the target label
and the last-layer features of source and target environments.

Based on the discussion on the regularization parameter choice at the
beginning of Section 5, we vary the regularization parameter
$\lambda_{\text{match}}$ from the set
$\left\\{10^{k}\right\\}_{k=-5,\cdots,4}$, and we choose the largest
$\lambda_{\text{match}}$ such that the source accuracy has not dropped too
much (in this case no more than 1% of the source accuracy of Src) as the final
regularization parameter.

The left plot in Figure 17 shows that both DIP${}^{\text{\scriptsize$(1)$}}$
and DIP${}^{\text{\scriptsize$(1)$}}$-MMD achieve better target accuracy than
Original or Src${}^{\text{\scriptsize$(1)$}}$.

(a) mask (a)
on $32\times 32$ image

(b) mask (b)
on $32\times 32$ image

(c) data samples
masked by mask (a)

(d) data samples
masked by mask (b)

Figure 16: The two interventions in the MNIST single source domain adaptation
without Y intervention. From left to right, mask (a), mask (b) and the
corresponding data samples.

Figure 17: Left: Target accuracy comparison in MNIST experiment with patch
intervention and single source without $Y$ intervention. Right: Target
accuracy comparison in MNIST experiment with patch intervention and multiple
sources with $Y$ intervention.

##### MNIST DA with patch intervention with Y intervention:

We take the original MNIST dataset with 60000 training samples and create 12
synthetic datasets ($M=11$ source environments and one target environment) as
follows. For the $m$-th source dataset, each training image is masked by the
$m$-th image (from left to right) in Figure 18. For the target dataset, each
train or test image is masked by the right most image in Figure 18. The target
dataset suffers additional $Y$ intervention: for digits (3, 4, 5, 6, 8, 9) in
the target dataset, $80\%$ of the images in the MNIST dataset are removed from
the target dataset. For experiment, we take random $20\%$ of samples from the
source datasets as the 11 source environments and $20\%$ of samples from the
target dataset as the target environment. This experiment is repeated 10 times
and we report the boxplot of 10 target classification accuracies for each DA
method.

As the MNIST experiments above, we apply the DA methods on the last-layer
features of the same pre-trained convolutional neural network (CNN). In
addition to the methods in the MNIST experiments above, we also consider the
following methods and their MMD variants:

  * •

DIPweigh: CNN model where DIPweigh-mean-finite is applied on the last-layer
features.

  * •

CIP: CNN model where CIP-mean-finite is applied on the last-layer features.

  * •

CIRMweigh: CNN model where CIRM-mean-finite is applied on the last-layer
features.

The regularization parameter $\lambda_{\text{CIP}}$ is chosen to be $0.1$
based on source risk. For the regularization parameter
$\lambda_{\text{match}}$ in DIPweigh and CIRMweigh, we first output the source
environment index with the largest weight as “the best source”. Then we vary
it from the set $\left\\{10^{k}\right\\}_{k=-5,\cdots,4}$, and we choose the
largest $\lambda_{\text{match}}$ such that the source accuracy of “the best
source” is not dropped too much (in this case not more than 1% of the source
accuracy of Src applied to “the best source”) as the final regularization
parameter.

The right plot in Figure 17 compares the target accuracies of the DA methods
in this setting. We observe that CIRMweigh and CIRMweigh-MMD outperforms
SrcPool and Original. Due to the intervention on $Y$, the matching penalty of
DIPweigh and DIPweigh-MMD is not useful and the two methods perform worse than
SrcPool.

Figure 18: The 12 $X$ interventions in the MNIST multiple source domain
adaptation with Y intervention. From left to right, the first 11 are the
source $X$ interventions, the last is the target $X$ intervention. The target
environment suffers additional $Y$ intervention.

####  5.3.2 MNIST with rotation intervention

##### MNIST DA with rotation intervention without Y intervention:

We take the original MNIST dataset with 60000 training samples and create
three synthetic datasets (2 sources and 1 target) as follows. For each
dataset, each training image is rotated by one of the angles
$\left\\{10,30,45\right\\}$ anti-clock-wise as shown in Figure 19. The three
datasets are named Rotation $10^{\circ}$, Rotation $30^{\circ}$ and Rotation
$45^{\circ}$ respectively. For the first experiment, we use Rotation
$10^{\circ}$ as the source environment and Rotation $45^{\circ}$ as the target
environment. For the second experiment, we use Rotation $30^{\circ}$ as the
source environment and Rotation $45^{\circ}$ as the target environment. For
each experiment, we take random $20\%$ of samples from the source dataset as
the source environment and random $20\%$ of samples from the target dataset as
the target environment. The task is to predict the labels of the images in the
target environment without observing any target labels. Except for the change
in the type of intervention, the other experimental settings are the same as
in MNIST DA with patch intervention without Y intervention in Section 5.3.1.

Figure 20 shows the boxplot of the target accuracies of Original, Tar,
Src${}^{\text{\scriptsize$(1)$}}$, DIP${}^{\text{\scriptsize$(1)$}}$ and
DIP${}^{\text{\scriptsize$(1)$}}$-MMD for the first experiment.
DIP${}^{\text{\scriptsize$(1)$}}$-MMD achieves higher target accuracy than
Src${}^{\text{\scriptsize$(1)$}}$. Figure 20 shows the boxplot of 10 runs of
Original, Tar, Src${}^{\text{\scriptsize$(1)$}}$,
DIP${}^{\text{\scriptsize$(1)$}}$ and DIP${}^{\text{\scriptsize$(1)$}}$-MMD
for the second experiment. DIP${}^{\text{\scriptsize$(1)$}}$-MMD achieves
higher target accuracy than Src${}^{\text{\scriptsize$(1)$}}$. Comparing
Figure 20 and 20, we also observe that the first experiment is a more
difficult classification task than the second experiment as the accuracies
achieved by our DA methods are lower in the first experiment.

(a) Rotation$10^{\circ}$

(b) Rotation$30^{\circ}$

(c) Rotation$45^{\circ}$

(d) samples $10^{\circ}$

(e) samples $30^{\circ}$

(f) samples $45^{\circ}$

Figure 19: (a)(b)(c): the three interventions in the MNIST rotation
intervention domain adaptation without $Y$ intervention. (d)(e)(f): the
corresponding data samples under rotation intervention.

(a) src: Rotation $10^{\circ}$
aaa tar: Rotation $45^{\circ}$

(b) src: Rotation $30^{\circ}$
aaa tar: Rotation $45^{\circ}$

(c) MNIST rotation interv. with $Y$ interv.

Figure 20: (a) Target accuracy comparison in MNIST experiment with rotation
intervention and single source without $Y$ intervention from Rotation 10% to
Rotation 45%. (b) Target accuracy comparison in MNIST experiment with rotation
intervention and single source without $Y$ intervention from Rotation 30% to
Rotation 45%. (c) Target accuracy comparison MNIST experiment with rotation
intervention and multiple source with $Y$ intervention.

##### MNIST DA with rotation intervention with Y intervention:

We take the original MNIST dataset with 60000 training samples and create 5
synthetic datasets ($M=4$ source environments and one target environment) as
follows. For $m\in\left\\{1,\cdots,4\right\\}$, for the $m$-th source dataset,
each training image is rotated by $\left(m\times 15-30\right)^{\circ}$ clock-
wise. Each image in the target dataset is rotated by $30^{\circ}$ clock-wise.
The target dataset suffers additional $Y$ intervention: for digits (3, 4, 5,
6, 8, 9) in the target dataset, $80\%$ of the images in the MNIST dataset are
removed from the target dataset. For experiment, we take random $20\%$ of
samples from the source datasets as the 4 source environments and $20\%$ of
samples from the target dataset as the target environment. Except for the
change in the type of intervention, the other experimental settings are the
same as in MNIST DA with patch intervention with Y intervention in Section
5.3.1.

Figure 20 shows the boxplot of the target accuracies of Original, Tar,
SrcPool, DIP${}^{\text{\scriptsize$(1)$}}$, DIPweigh, CIP, CIRMweigh, DIP-MMD,
CIP-MMD and CIRMweigh-MMD. CIP and CIP-MMD outperforms SrcPool and Original.
Due to the intervention on Y, the matching penalty of DIPweigh and DIPweigh-
MMD is not useful and the two methods perform worse than SrcPool. CIP,
CIRMweigh and the corresponding MMD variants outperform SrcPool.

####  5.3.3 MNIST with random translation intervention

We take the original MNIST dataset with 60000 training samples and create two
synthetic datasets (source and target) as follows. For the source dataset,
each training image is translated horizontally with a distance randomly
selected from $-0.2\times\text{image width}$ to $0.2\times\text{image width}$
as shown in Figure 21. For the target dataset environment, each training image
is translated vertically with a distance randomly selected from
$-0.2\times\text{image height}$ to $0.2\times\text{image height}$ as shown in
Figure 21. For each experiment, we take random $20\%$ of samples from the
source dataset as the source environment and random $20\%$ of samples from the
target dataset as the target environment. The task is to predict the labels of
the images in the target environment without observing any target labels.
Except for the change in the type of intervention, the other experimental
settings are the same as in MNIST DA with patch intervention without Y
intervention in Section 5.3.1.

Figure 21 shows the boxplot of the target accuracies of Original, Tar,
Src${}^{\text{\scriptsize$(1)$}}$, DIP${}^{\text{\scriptsize$(1)$}}$ and
DIP${}^{\text{\scriptsize$(1)$}}$-MMD. DIP${}^{\text{\scriptsize$(1)$}}$-MMD
still performs better than Src${}^{\text{\scriptsize$(1)$}}$, but it barely
improves over Original. Since the intervention is random rather than fixed for
each environment, intuitively this experiment is more difficult for our DA
methods to have high accuracy than the first two experiments with fixed
intervention for each environment.

(a) source samples after
random horizontal
translation

(b) target samples after
random vertical
translation

(c) MNIST translation interv.

Figure 21: (a) Source samples after random horizontal translation. (b) Target
samples after random vertical translation. (c) Target accuracy comparison in
MNIST experiment with random translation intervention and single source
without Y intervention.

###  5.4 Experiments on Amazon review dataset with unknown interventions

In this subsection, we compare DA methods on the regression dataset Amazon
Review Data (Ni et al., 2019). Unlike the simulated experiments or the MNIST
experiment, neither the data generation process nor the type of interventions
is known.

This dataset contains product reviews and metadata from Amazon in the date
range May 1996 - Oct 2018. In our experiment, we consider the task of
predicting review rating from review text from various product categories
(Automotive, Digital Music, Office Products etc.). For each review, the
covariates $X$ are the TF-IDF features generated from raw review text; the
label $Y$ is the review rating (score from $1-5$). The different product
categories are used as source and target environments. We take 15 product
categories with the largest sample sizes, use 14 of them as source
environments and leave the last one as the target environment.

The samples size $n$ is 10000 in each source or target environment. The
dimension depends on the TF-IDF feature extractor. Here we use both unigrams
and bigrams, and build the vocabulary with terms that have a document
frequency not smaller than $0.008$. This results in feature dimension $d=482$.
A linear model with $\ell_{2}$ regularization is used on top of the TF-IDF
features to predict ratings.

Without explicit knowledge about the causal structure or the type of
interventions, a priori it is no longer clear which DA method is the best. We
apply SrcPool and the advanced DA methods, DIPweigh, CIP, CIRMweigh on the
linear model. Figure 22 shows that the advanced DA methods do not outperform
SrcPool except for target environment number 4.

We did an additional experiment with a small portion of target labels
revealed. We use the small portion of target labels to choose the best method
out of SrcPool, DIPweigh, CIP and CIRMweigh based on the small portion of
labeled target data. The last three boxes (labeled as best20, best40 and
best60) in each subplot of Figure 22 show that with 20, 40, 60 target labels
revealed, the best method based on the small portion of labeled target data is
always not worse than SrcPool and can sometimes outperform SrcPool.

We arrive at a conclusion that in general, without explicit knowledge about
causal structure or the type of interventions, it is ambitious to expect
advanced DA methods to always outperform SrcPool. However, with a small
portion of labeled target data, we show that DA methods still lead to
substantial improvements. Our empirical observation agrees with the concurrent
empirical study of DA methods by Wiles et al. (2021). In a larger-scale
experimental framework with many real and synthetic datasets, they find that
DA methods can outperform SrcPool in some settings but there is no single best
method across all settings. Hence, knowledge about the distribution shift is
necessary for successful DA with guarantees.

Figure 22: Target risk in Amazon review data prediction experiment (the lower
the better). Without explicit knowledge about the type of interventions, it is
no longer clear which domain adaptation method is the best. From left to
right, depending on the target environment choice, the best methods are CIP,
DIPweigh, CIRMweigh accordingly.

##  6 Discussion

In this paper, we propose a theoretical framework using structural causal
models (SCMs) to analyze and obtain insights on prediction performance of
several popular domain adaptation methods. First, we show that under the
assumption of anticausal prediction, linear SCM and no intervention on $Y$,
the popular DA method DIP achieves a low target risk. This theoretical result
is compatible with many previous empirical results on the usefulness of DIP
for domain adaptation. Second, we derive several conditions where DIP fails to
achieve a low target risk: when the prediction problem is causal or mixed-
causal-anticausal; when there is label distribution perturbation. To tackle
these difficult DA scenarios, we design a new DA method called CIRM, among
other modifications. The theoretical analysis is complemented with simulation
analysis and real data experiments. The theoretical extension to nonlinear
SCMs is a challenging future direction. However our empirical results suggest
that the linear SCM assumption provides useful insights even for cases where a
linear SCM does not hold.

The real data experiments show that it can be beneficial to use DA methods
when the anticausal prediction assumption is satisfied but it can also be
dangerous to blindly use DA methods when little is known about the data
generation process. Thus, prior knowledge of the underlying causal structure
and the types of interventions are often crucial. The prior knowledge of the
causal structure can come from domain expert knowledge or causal studies on
related datasets with the same variables. How to seamlessly combine causal
studies about the causal structure and domain adaptation with appropriate
uncertainty quantification is one promising future direction.

In absence of prior information about the causal structure and the types and
locations of interventions, one needs to select good DA methods. The
assessment of the goodness of a model or algorithm is difficult in general but
it can be done when having access to a small fraction of labeled target data.
We show empirically that a small fraction of labeled target data substantially
helps to select the best DA method. It remains an open question what is the
minimal amount of labeled target data points in order to guarantee the best DA
method selection.

Acknowledgments

Y. Chen and P. Bühlmann have received funding from the European
ResearchCouncil under the Grant Agreement No 786461 (CausalStats -
ERC-2017-ADG). They both acknowledge scientific interaction and exchange at
“ETH Foundations of Data Science”. They also thank Domagoj Cevid, Christina
Heinze-Deml, Wooseok Ha, Jinzhou Li, Nicolai Meinshausen, Armeen Taeb, Fanny
Yang and Andrii Zadaianchuk for fruitful discussions and for helpful
suggestions on presentation and writing.

##  Appendix A Summary of DA methods in this paper

In this section, we provide a summary of the DA methods presented in this
paper. Methods that do not fit in the main paper due to the space limitation
are formally introduced here.

###  A.1 Population DA methods

First, we formulate the DIP variants that adapt to other types of intervention
as mentioned in Section 4.1.3.

  * •

DIP${}^{\text{\scriptsize$(m)$}}$-std: the population DIP estimator where the
difference between the source and target standard deviations is used as
distributional distance. For the $m$-th source environment, it is defined as

$\displaystyle f_{\text{DIP-std}}^{\text{\scriptsize$(m)$}}(x)$
$\displaystyle:=x^{\top}\beta_{\text{DIP-
std}}^{\text{\scriptsize$(m)$}}+\beta_{\text{DIP-
std},0}^{\text{\scriptsize$(m)$}}$ $\displaystyle\beta_{\text{DIP-
std$+$}}^{\text{\scriptsize$(m)$}},\beta_{\text{DIP-
std},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle:=\operatorname*{arg\,min}_{\beta,\beta_{0}}\
{\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{\scriptsize$(m)$}}}\left(Y-X^{\top}\beta-\beta_{0}\right)^{2}$
$\displaystyle\text{s.t.\
}\operatorname{Var}_{X\sim\mathcal{P}^{\text{\scriptsize$(m)$}}_{X}}\left[X^{\top}\beta\right]=\operatorname{Var}_{X\sim\widetilde{\mathcal{P}}_{X}}\left[X^{\top}\beta\right].$
(46)

  * •

DIP${}^{\text{\scriptsize$(m)$}}$-std$+$: the population DIP estimator where
the differences between the source and target means, standard deviations and
25% quantiles are used as distributional distance

$\displaystyle f_{\text{DIP-std$+$}}^{\text{\scriptsize$(m)$}}(x)$
$\displaystyle:=x^{\top}\beta_{\text{DIP-
std$+$}}^{\text{\scriptsize$(m)$}}+\beta_{\text{DIP-
std$+$},0}^{\text{\scriptsize$(m)$}}$ $\displaystyle\beta_{\text{DIP-
std$+$}}^{\text{\scriptsize$(m)$}},\beta_{\text{DIP-
std$+$},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle:=\operatorname*{arg\,min}_{\beta,\beta_{0}}\
{\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{\scriptsize$(m)$}}}\left(Y-X^{\top}\beta-\beta_{0}\right)^{2}$
$\displaystyle\text{s.t.\
}{\mathbb{E}}_{X\sim\mathcal{P}^{\text{\scriptsize$(m)$}}_{X}}\left[X^{\top}\beta\right]={\mathbb{E}}_{X\sim\widetilde{\mathcal{P}}_{X}}\left[X^{\top}\beta\right]$
$\displaystyle\phantom{\text{s.}}\operatorname{Var}_{X\sim\mathcal{P}^{\text{\scriptsize$(m)$}}_{X}}\left[X^{\top}\beta\right]=\operatorname{Var}_{X\sim\widetilde{\mathcal{P}}_{X}}\left[X^{\top}\beta\right]$
$\displaystyle\phantom{\text{s.t.\ \
}}\psi_{25\%}\left({X^{\text{\scriptsize$(m)$}}}^{\top}\beta\right)=\psi_{25\%}\left({\widetilde{X}}^{\top}\beta\right),$
(47)

where $\psi_{25\%}$ is the 25% quantile function which takes a random variable
and returns its 25% quantile.

  * •

DIP${}^{\text{\scriptsize$(m)$}}$-MMD: the population DIP estimator where the
maximum mean discrepancy (MMD) is used as distributional distance.

$\displaystyle f_{\text{DIP-MMD}}^{\text{\scriptsize$(m)$}}(x)$
$\displaystyle:=x^{\top}\beta_{\text{DIP-
MMD}}^{\text{\scriptsize$(m)$}}+\beta_{\text{DIP-
MMD},0}^{\text{\scriptsize$(m)$}}$ $\displaystyle\beta_{\text{DIP-
MMD}}^{\text{\scriptsize$(m)$}},\beta_{\text{DIP-
MMD},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle:=\operatorname*{arg\,min}_{\beta,\beta_{0}}\
{\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{\scriptsize$(m)$}}}\left(Y-X^{\top}\beta-\beta_{0}\right)^{2}$
$\displaystyle\text{s.t.\
}\mathcal{D}_{\text{MMD},\mathcal{H}}\left({X^{\text{\scriptsize$(m)$}}}^{\top}\beta,{\widetilde{X}}^{\top}\beta\right)=0,$
(48)

where the maximum mean discrepancy (MMD) Gretton et al. (2012) with respect to
the reproducing kernel Hilbert space (RHKS) $\mathcal{H}$ between two random
variable $Z_{1}$ and $Z_{2}$ is

$\displaystyle\mathcal{D}_{\text{MMD},\mathcal{H}}\left(Z_{1},Z_{2}\right)=\sup_{h\in\mathcal{H}}\left|{\mathbb{E}}\left[h({Z_{1})}\right]-{\mathbb{E}}\left[h({Z_{2})}\right]\right|.$

By default, the RKHS with Gaussian kernel is used throughout this paper.

Second we introduce the weighted version of CIRM following DIPweigh (• ‣
4.1.2).

  * •

CIRMweigh-mean: the population CIRM estimator that weights the source
environments based on the source risks. It is defined as follows

$\displaystyle f_{\text{CIRMweigh}}(x)$
$\displaystyle:=\frac{1}{\sum_{m=1}^{M}e^{-\eta\cdot
s_{m}}}\sum_{m=1}^{M}e^{-\eta\cdot
s_{m}}\left(x^{\top}{\beta_{\text{CIRM}}^{\text{\scriptsize$(m)$}}}+\beta_{\text{CIRM},0}^{\text{\scriptsize$(m)$}}\right)$
$\displaystyle s_{m}$
$\displaystyle:=R^{\text{\scriptsize$(m)$}}\left(f_{\text{CIRM}}^{\text{\scriptsize$(m)$}}\right).$
(49)

Here $\eta>0$ is a constant. Choosing $\eta$ to be $\infty$ is equivalent to
choosing the source estimator with the lowest source risk.

The rational behind the use of source risks to weigh the environments follows
from the corollary below.

###### Corollary 8

Under the data generation Assumption 2, the $m$-th source population risk (2)
of CIRM${}^{\text{\scriptsize$(m)$}}$-mean satisfies

$\displaystyle
R^{\text{\scriptsize$(m)$}}\left(f_{\text{CIRM}}^{\text{\scriptsize$(m)$}}\right)=\tilde{R}\left(f_{\text{CIRM}}^{\text{\scriptsize$(m)$}}\right)+\frac{\left(a^{\text{\scriptsize$(m)$}}_{Y}-\widetilde{a}_{Y}\right)^{2}}{\left(1+\sigma^{2}{b}^{\top}\Sigma^{-\frac{1}{2}}{G_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}\Sigma^{-\frac{1}{2}}{b}\right)^{2}}$ (50)

The proof of Corollary 8 is provided in Appendix C.3. Comparing Corollary 8
with Corollary 3, the source risk of CIRM is no longer exactly equal to its
target risk. The source risk has an additional term that depends on the
intervention on $Y$. However, in the case where $\Sigma$ has eigenvalues much
smaller than $\sigma$, this additional term is negligible. In these scenarios,
the source risk of CIRM still constitutes a good approximation of the target
risk of CIRM. It is still possible to apply CIRM for each
$m\in\left\\{1,\ldots,M\right\\}$ and pick the source environment with the
lowest source population risk in order to reduce the target population risk.
Based on the above ideas, we introduce the following weighted version of CIRM.

Third we introduce the CIP and CIRM extensions to deal with mixed-causal-
anticausal DA problems in Section 4.3.

  * •

CIP$\diamondsuit$-mean: the population conditional invariance penalty
estimator for the mixed causal anticausal DA setting.

$\displaystyle f_{\text{CIP}\diamondsuit}(x)$
$\displaystyle:=x^{\top}\begin{bmatrix}\gamma^{*}-\Gamma^{*}\beta_{\text{CIP}\diamondsuit}\\\
\beta_{\text{CIP}\diamondsuit}\end{bmatrix}+\beta_{\text{CIP}\diamondsuit,0}^{\text{\scriptsize$(m)$}}$
$\displaystyle\beta_{\text{CIP}\diamondsuit},\beta_{\text{CIP}\diamondsuit,0}$
$\displaystyle:=\operatorname*{arg\,min}_{\beta,\beta_{0}}\
\frac{1}{M}\sum_{k=1}^{M}{\mathbb{E}}_{(X_{\text{P}},X_{\text{D}},Y)\sim\mathcal{P}^{\text{\scriptsize$(k)$}}}\left(Y_{\mathcal{I}}-{X_{\mathcal{I}}}^{\top}\beta-\beta_{0}\right)^{2}$
$\displaystyle\text{s.t.\
}{\mathbb{E}}_{(X_{\text{P}},X_{\text{D}},Y)\sim\mathcal{P}^{\text{\scriptsize$(m)$}}}$
$\displaystyle\left[{X_{\mathcal{I}}}^{\top}\beta\mid
Y_{\mathcal{I}}=y\right]={\mathbb{E}}_{(X_{\text{P}},X_{\text{D}},Y)\sim\mathcal{P}^{\text{\scriptsize$(1)$}}}\left[{X_{\mathcal{I}}}^{\top}\beta\mid
Y_{\mathcal{I}}=y\right],$ $\displaystyle\forall y\in,\forall
m\in\left\\{1,\ldots,M\right\\},$ (51)

where $Y_{\mathcal{I}}:=Y-X_{\text{P}}^{\top}\gamma^{*}$,
$X_{\mathcal{I}}:=X_{\text{D}}-{\Gamma^{*}}^{\top}X_{\text{P}}$,

$\displaystyle\gamma^{*},\gamma_{0}^{*}$
$\displaystyle:=\operatorname*{arg\,min}_{\gamma,\gamma_{0}\in^{r}\times}{\mathbb{E}}_{\left(X_{\text{P}},X_{\text{D}},Y\right)\sim\mathcal{P}^{\text{allsrc}}}\left(Y-X_{\text{P}}^{\top}\gamma-\gamma_{0}\right)^{2},$
$\displaystyle\Gamma^{*},\Gamma_{0}^{*}$
$\displaystyle:=\operatorname*{arg\,min}_{\Gamma,\Gamma_{0}\in^{r\times(d-r)}\times^{d-r}}{\mathbb{E}}_{\left(X_{\text{P}},X_{\text{D}}\right)\sim\mathcal{P}^{\text{allsrc}}_{X}}\left\|X_{\text{D}}-\Gamma^{\top}X_{\text{P}}-\Gamma_{0}\right\|_{2}^{2},$
(52)

and $\mathcal{P}^{\text{allsrc}}$ denotes the uniform mixture of all source
distribution.

  * •

CIRM$\diamondsuit^{\text{\scriptsize$(m)$}}$-mean: the population conditional
invariant residual matching estimator using $m$-th source environment for the
mixed causal anticausal DA setting.

$\displaystyle f_{\text{CIRM}\diamondsuit}^{\text{\scriptsize$(m)$}}(x)$
$\displaystyle:=x^{\top}\begin{bmatrix}\gamma^{*}-\Gamma^{*}\beta_{\text{CIRM}\diamondsuit}^{\text{\scriptsize$(m)$}}\\\
\beta_{\text{CIRM}\diamondsuit}^{\text{\scriptsize$(m)$}}\end{bmatrix}+\beta_{\text{CIRM}\diamondsuit,0}^{\text{\scriptsize$(m)$}}$
$\displaystyle\beta_{\text{CIRM}\diamondsuit}^{\text{\scriptsize$(m)$}},\beta_{\text{CIRM}\diamondsuit,0}^{\text{\scriptsize$(m)$}}$
$\displaystyle:=\operatorname*{arg\,min}_{\beta,\beta_{0}}\
{\mathbb{E}}_{(X_{\text{P}},X_{\text{D}},Y)\sim\mathcal{P}^{\text{\scriptsize$(m)$}}}\left(Y_{\mathcal{I}}-X_{\mathcal{I}}^{\top}\beta-\beta_{0}\right)^{2}$
$\displaystyle\text{s.t.\
}{\mathbb{E}}_{(X_{\text{P}},X_{\text{D}})\sim\mathcal{P}^{\text{\scriptsize$(m)$}}_{X}}\left[\beta^{\top}\left(X_{\mathcal{I}}-\left(X_{\mathcal{I}}^{\top}\beta_{\text{CIP}\diamondsuit}\right)\vartheta_{\text{CIRM}\diamondsuit}\right)\right]$
$\displaystyle={\mathbb{E}}_{(X_{\text{P}},X_{\text{D}})\sim\widetilde{\mathcal{P}}_{X}}\left[\beta^{\top}\left(X_{\mathcal{I}}-\left(X_{\mathcal{I}}^{\top}\beta_{\text{CIP}\diamondsuit}\right)\vartheta_{\text{CIRM}\diamondsuit}\right)\right],$
(53)

where $Y_{\mathcal{I}}:=Y-X_{\text{P}}^{\top}\gamma^{*}$,
$X_{\mathcal{I}}:=X_{\text{D}}-{\Gamma^{*}}^{\top}X_{\text{P}}$ with
$\gamma^{*}$ and $\Gamma^{*}$ defined in Equation (• ‣ A.1) and

$\displaystyle\vartheta_{\text{CIRM}\diamondsuit}:=\frac{{\mathbb{E}}_{(X_{\text{P}},X_{\text{D}},Y)\sim\mathcal{P}^{\text{allsrc}}}\left[X_{\mathcal{I}}\cdot(Y_{\mathcal{I}}-{\mathbb{E}}[Y_{\mathcal{I}}])\right]}{{\mathbb{E}}_{(X_{\text{P}},X_{\text{D}},Y)\sim\mathcal{P}^{\text{allsrc}}}\left[\left(X_{\mathcal{I}}^{\top}\beta_{\text{CIP}\diamondsuit}-{\mathbb{E}}[X_{\mathcal{I}}^{\top}\beta_{\text{CIP}\diamondsuit}]\right)\cdot\left(Y_{\mathcal{I}}-{\mathbb{E}}[Y_{\mathcal{I}}]\right)\right]},$
(54)

with $\mathcal{P}^{\text{allsrc}}$ denote the uniform mixture of all source
distributions.

Finally, we summarize in Table 5 all the population DA methods introduced in
this paper.

Original |  Made-ups to assist explanation |  Weighted variants |  Matching penalty variants |  Mixed variants
---|---|---|---|---
DIP${}^{\text{\scriptsize$(m)$}}$ (• ‣ 3.4) |  DIPAbs${}^{\text{\scriptsize$(m)$}}$ (• ‣ 3.5.2) DIPOracle${}^{\text{\scriptsize$(m)$}}$ (• ‣ 4.1.1) | DIPweigh (• ‣ 4.1.2) |  DIP-std (• ‣ A.1) DIP-std+ (• ‣ A.1) DIP-MMD (• ‣ A.1) |  DIP$\diamondsuit^{\text{\scriptsize$(m)$}}$ (• ‣ 4.3.2) DIP$\diamondsuit$weigh
CIP (• ‣ 3.4) | N/A | N/A |  CIP-std CIP-std+ CIP-MMD | CIP$\diamondsuit$ (• ‣ A.1)
CIRM${}^{\text{\scriptsize$(m)$}}$ (• ‣ 3.4) | N/A | CIRMweigh (• ‣ A.1) |  CIRM-std CIRM-std+ CIRM-MMD |  CIRM$\diamondsuit^{\text{\scriptsize$(m)$}}$ (• ‣ A.1) CIRM$\diamondsuit$weigh
Table 5: Summary of population DA methods introduced in this paper. The
matching penalty variants of CIP and CIRM can be formulated similarly as those
for DIP. They are omitted for the sake of space.

###  A.2 Finite-sample formulation of DA methods

We introduce the finite-sample formulations of the population DA methods in
the previous subsection so that they can be implemented to reproduce the
results in the numerical experiments in Section 5. The finite-sample DA
formulations are summarized in Table 6.

The finite-sample formulations of DIP-mean, CIP-mean and CIRM-mean are
introduced in Section 5.1. Here we state the finite-sample formulations of
weighted variants, matching penalty and mixed-causal-anticausal variants.

To formulate the finite-sample versions of DIPweigh-mean (• ‣ 4.1.2) and
CIRMweigh-mean (• ‣ A.1), it is sufficient to replace the corresponding
population estimators and the population source risks with finite-sample ones.
The mixed-causal-anticausal variants of DIP, CIP and CIRM only requires two
additional regressions. They are omitted for the sake of space.

Next, we introduce the finite-sample formulations of the matching penalty
variants for DIP.

  * •

DIP${}^{\text{\scriptsize$(m)$}}$-std-finite: the finite-sample formulation of
the DIP${}^{\text{\scriptsize$(m)$}}$-std estimator (• ‣ A.2). The difference
between the source and target standard deviations is used as distributional
distance,

$\displaystyle\hat{f}_{\text{DIP-std}}^{\text{\scriptsize$(m)$}}(x)$
$\displaystyle:=x^{\top}\hat{\beta}_{\text{DIP-
std}}^{\text{\scriptsize$(m)$}}+\hat{\beta}_{\text{DIP-
std},0}^{\text{\scriptsize$(m)$}}$ $\displaystyle\hat{\beta}_{\text{DIP-
std}}^{\text{\scriptsize$(m)$}},\hat{\beta}_{\text{DIP-
std},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle:=\operatorname*{arg\,min}_{\beta,\beta_{0}}\
\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}\left(y_{i}^{\text{\scriptsize$(m)$}}-{x_{i}^{\text{\scriptsize$(m)$}}}^{\top}\beta-\beta_{0}\right)^{2}+\lambda_{\text{match}}\left(\delta^{\text{\scriptsize$(m)$}}-\tilde{\delta}\right)^{2},$
(55)

where $\delta^{\text{\scriptsize$(m)$}}$ is the standard deviation of
$\left({x_{1}^{\text{\scriptsize$(m)$}}}^{\top}\beta,{x_{2}^{\text{\scriptsize$(m)$}}}^{\top}\beta,\ldots,{x_{n_{m}}^{\text{\scriptsize$(m)$}}}^{\top}\beta\right)$,
and $\tilde{\delta}$ is the standard deviation of
$\left({\tilde{x}_{1}}^{\top}\beta,{\tilde{x}_{2}}^{\top}\beta,\ldots,{\tilde{x}_{\tilde{n}}}^{\top}\beta\right)$.

  * •

DIP${}^{\text{\scriptsize$(m)$}}$-std$+$-finite: the finite-sample DIP
estimator where the differences between the source and target means, standard
deviations and 25% quantiles are used as distributional distance,
$\mathcal{V}$ is linear and $\mathcal{U}$ is the singleton of identity
mapping. For the $m$-th source environment, it is defined as

$\displaystyle\hat{f}_{\text{DIP-std$+$}}^{\text{\scriptsize$(m)$}}(x)$
$\displaystyle:=x^{\top}\hat{\beta}_{\text{DIP-
std$+$}}^{\text{\scriptsize$(m)$}}+\hat{\beta}_{\text{DIP-
std$+$},0}^{\text{\scriptsize$(m)$}}$ $\displaystyle\hat{\beta}_{\text{DIP-
std$+$}}^{\text{\scriptsize$(m)$}},\hat{\beta}_{\text{DIP-
std$+$},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle:=\operatorname*{arg\,min}_{\beta,\beta_{0}}\
\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}\left(y_{i}^{\text{\scriptsize$(m)$}}-{x_{i}^{\text{\scriptsize$(m)$}}}^{\top}\beta-\beta_{0}\right)^{2}+\lambda_{\text{match}}\left(\mu^{\text{\scriptsize$(m)$}}-\tilde{\mu}\right)^{2}$
$\displaystyle\hskip
85.35826pt+\lambda_{\text{match}}\left(\delta^{\text{\scriptsize$(m)$}}-\tilde{\delta}\right)^{2}+\lambda_{\text{match}}\left(\psi_{25\%}^{\text{\scriptsize$(m)$}}-\tilde{\psi}_{25\%}\right)^{2},$
(56)

where
$\mu^{\text{\scriptsize$(m)$}},\delta^{\text{\scriptsize$(m)$}},\psi_{25\%}^{\text{\scriptsize$(m)$}}$
are respectively the mean, standard deviation and 25% quantile of
$\left({x_{1}^{\text{\scriptsize$(m)$}}}^{\top}\beta,{x_{2}^{\text{\scriptsize$(m)$}}}^{\top}\beta,\ldots,{x_{n_{m}}^{\text{\scriptsize$(m)$}}}^{\top}\beta\right)$,
and $\tilde{\mu},\tilde{\delta},\tilde{\psi}_{25\%}$ are respectively the
mean, standard deviation and 25% quantile of
$\left({\tilde{x}_{1}}^{\top}\beta,{\tilde{x}_{2}}^{\top}\beta,\ldots,{\tilde{x}_{\tilde{n}}}^{\top}\beta\right)$.

  * •

DIP${}^{\text{\scriptsize$(m)$}}$-MMD-finite: the finite-sample DIP estimator
where mean squared difference is used as distributional distance,
$\mathcal{V}$ is linear and $\mathcal{U}$ is the singleton of identity
mapping. For the $m$-th source environment, it is defined as

$\displaystyle\hat{f}_{\text{DIP-MMD}}^{\text{\scriptsize$(m)$}}(x)$
$\displaystyle:=x^{\top}\hat{\beta}_{\text{DIP-
MMD}}^{\text{\scriptsize$(m)$}}+\hat{\beta}_{\text{DIP-
MMD},0}^{\text{\scriptsize$(m)$}}$ $\displaystyle\hat{\beta}_{\text{DIP-
MMD}}^{\text{\scriptsize$(m)$}},\hat{\beta}_{\text{DIP-
MMD},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle:=\operatorname*{arg\,min}_{\beta,\beta_{0}}\
\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}\left(y_{i}^{\text{\scriptsize$(m)$}}-{x_{i}^{\text{\scriptsize$(m)$}}}^{\top}\beta-\beta_{0}\right)^{2}+\lambda_{\text{match}}\mathcal{D}_{\text{MMD},\mathcal{H}}\left(Z^{\text{\scriptsize$(m)$}},\tilde{Z}\right),$
(57)

where $Z^{\text{\scriptsize$(m)$}}$ is the set of predicted responses in the
$m$-th source environment
$\left\\{{x_{1}^{\text{\scriptsize$(m)$}}}^{\top}\beta,\ldots,{x_{n_{m}}}^{\top}\beta\right\\}$,
similarly
$\tilde{Z}=\left\\{{\tilde{x}_{1}}^{\top}\beta,\ldots,{\tilde{x}_{\tilde{n}}}^{\top}\beta\right\\}$
and the maximum mean discrepancy (MMD) Gretton et al. (2012) with respect to
the reproducing kernel Hilbert space (RHKS) $\mathcal{H}$ between these two
sets is

$\displaystyle\mathcal{D}_{\text{MMD},\mathcal{H}}\left(Z^{\text{\scriptsize$(m)$}},\tilde{Z}\right)=\sup_{h\in\mathcal{H}}\left(\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}h({x_{i}^{\text{\scriptsize$(m)$}}}^{\top}\beta)-\frac{1}{\tilde{n}}\sum_{i=1}^{\tilde{n}}h(\tilde{x_{i}}^{\top}\beta)\right).$

In all experiments, RKHS with Gaussian kernel is used.

Original (finite) |  Weighted variants |  Matching penalty variants |  Mixed variants
---|---|---|---
DIP-mean (• ‣ 5.1) | DIPweigh-mean |  DIP-std (• ‣ A.2) DIP-std+ (• ‣ A.2) DIP-MMD (• ‣ A.2) |  DIP$\diamondsuit^{\text{\scriptsize$(m)$}}$-mean DIP$\diamondsuit$weigh-mean
CIP-mean (• ‣ 5.1) | N/A |  CIP-std CIP-std+ CIP-MMD | CIP$\diamondsuit$-mean
CIRM-mean (• ‣ 5.1) | CIRMweigh-mean |  CIRM-std CIRM-std+ CIRM-MMD |  CIRM$\diamondsuit^{\text{\scriptsize$(m)$}}$-mean CIRM$\diamondsuit$weigh-mean
Table 6: Summary of finite-sample formulations of the DA methods discussed in
this paper. The suffixes “-finite” of the finite-sample DA methods are omitted
in the table for brevity. The matching penalty variants of CIP and CIRM can be
formulated similarly as those for DIP. The mixed-causal-anticausal variants of
DIP, CIP and CIRM only requires two additional regressions. They are omitted
for the sake of space.

##  Appendix B Proofs related to Theorem 1

In this section, we prove Theorem 1 and related corollaries.

###  B.1 Proof of Theorem 1

At a high level, the proof of Theorem 1 goes by connecting the target
population risk of DIP${}^{\text{\scriptsize$(1)$}}$ with that of
DIPOracle${}^{\text{\scriptsize$(1)$}}$, and then bounding the target
population risk of DIPOracle${}^{\text{\scriptsize$(1)$}}$ as a regularized
version of OLSTar.

Using the linear SCM assumption in Assumption 1, each data point in the source
environment is generated i.i.d. from the following equation

$\displaystyle X^{\text{\scriptsize$(1)$}}$
$\displaystyle=\mathbf{B}X^{\text{\scriptsize$(1)$}}+{b}Y^{\text{\scriptsize$(1)$}}+a^{\text{\scriptsize$(1)$}}_{X}+\varepsilon^{\text{\scriptsize$(1)$}}_{X},$
$\displaystyle Y^{\text{\scriptsize$(1)$}}$
$\displaystyle=\varepsilon^{\text{\scriptsize$(1)$}}_{Y}.$ (58)

Define $H=\left(\mathbb{I}_{d}-\mathbf{B}\right)^{-1}$. Solving
$X^{\text{\scriptsize$(1)$}}$ from Equation (B.1), we have

$\displaystyle
X^{\text{\scriptsize$(1)$}}=H{b}\varepsilon^{\text{\scriptsize$(1)$}}_{Y}+Ha^{\text{\scriptsize$(1)$}}_{X}+H\varepsilon^{\text{\scriptsize$(1)$}}_{X}.$
(59)

For $(\beta,\beta_{0})\in^{d}\times$, the residual takes the following form

$\displaystyle
Y^{\text{\scriptsize$(1)$}}-\beta^{\top}X^{\text{\scriptsize$(1)$}}-\beta_{0}=\left(1-\beta^{\top}H{b}\right)\varepsilon^{\text{\scriptsize$(1)$}}_{Y}-\left(\beta^{\top}Ha^{\text{\scriptsize$(1)$}}_{X}+\beta_{0}\right)-\beta^{\top}H\varepsilon^{\text{\scriptsize$(1)$}}_{X}.$

Taking expectation and using the fact that noise has zero mean, we obtain

$\displaystyle{\mathbb{E}}\left[Y^{\text{\scriptsize$(1)$}}-\beta^{\top}X^{\text{\scriptsize$(1)$}}-\beta_{0}\right]^{2}=\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}+\left(\beta^{\top}Ha^{\text{\scriptsize$(1)$}}_{X}+\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta.$ (60)

Similarly, we obtain the target expected residual

$\displaystyle{\mathbb{E}}\left[\widetilde{Y}-\beta^{\top}\widetilde{X}-\beta_{0}\right]^{2}=\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}+\left(\beta^{\top}H\widetilde{a}_{X}+\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta.$ (61)

##### Risk of OLSTar:

Using the expression for the target residual, the OLSTar estimator in Equation
(• ‣ 3.3) becomes the solution of the following quadratic program

$\displaystyle\min_{\beta,\beta_{0}}\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}+\left(\beta^{\top}H\widetilde{a}_{X}+\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta.$ (62)

Solving the quadratic program and with matrix inversion lemma, we obtain

$\displaystyle\beta_{\text{OLSTar}}$
$\displaystyle=\sigma^{2}\left(\mathbb{I}_{d}-\mathbf{B}\right)^{\top}\left(\Sigma+\sigma^{2}{b}{b}^{\top}\right)^{-1}{b}$
$\displaystyle=\left(\mathbb{I}_{d}-\mathbf{B}\right)^{\top}\frac{\sigma^{2}\Sigma^{-1}{b}}{1+\sigma^{2}{b}^{\top}\Sigma^{-1}{b}}$
$\displaystyle\beta_{\text{OLSTar},0}$
$\displaystyle=-\beta_{\text{OLSTar}}^{\top}H\widetilde{a}_{X}.$

The target population loss of OLSTar is

$\displaystyle\tilde{R}\left(f_{\text{OLSTar}}\right)$
$\displaystyle=\sigma^{2}-\sigma^{4}{b}^{\top}\left(\Sigma+\sigma^{2}{b}{b}^{\top}\right)^{-1}{b}$
$\displaystyle=\frac{\sigma^{2}}{1+\sigma^{2}{b}^{\top}\Sigma^{-1}{b}},$ (63)

where the last equality follows from matrix inversion lemma.

##### Risk of OLSSrc${}^{\text{\scriptsize$(1)$}}$:

The minimization problem of OLSSrc can be similarly solved by changing the
target variables to source ones in Equation (62). We obtain

$\displaystyle\beta_{\text{OLSSrc}}^{\text{\scriptsize$(1)$}}$
$\displaystyle=\sigma^{2}\left(\mathbb{I}_{d}-\mathbf{B}\right)^{\top}\left(\Sigma+\sigma^{2}{b}{b}^{\top}\right)^{-1}{b}$
$\displaystyle=\left(\mathbb{I}_{d}-\mathbf{B}\right)^{\top}\frac{\sigma^{2}\Sigma^{-1}{b}}{1+\sigma^{2}{b}^{\top}\Sigma^{-1}{b}}$
$\displaystyle\beta_{\text{OLSSrc},0}^{\text{\scriptsize$(1)$}}$
$\displaystyle=-{\beta_{\text{OLSSrc}}^{\text{\scriptsize$(1)$}}}^{\top}Ha^{\text{\scriptsize$(1)$}}_{X}.$

Because of the difference in the intercept term, the target population risk of
OLSSrc has one additional term

$\displaystyle\tilde{R}\left(f_{\text{OLSSrc}}^{\text{\scriptsize$(1)$}}\right)=\frac{\sigma^{2}}{1+\sigma^{2}{b}^{\top}\Sigma^{-1}{b}}+\frac{\left(\sigma^{2}{b}^{\top}\Sigma^{-1}\left(a^{\text{\scriptsize$(1)$}}_{X}-\widetilde{a}_{X}\right)\right)^{2}}{\left(1+\sigma^{2}{b}^{\top}\Sigma^{-1}{b}\right)^{2}}.$
(64)

##### Risk of DIP${}^{\text{\scriptsize$(1)$}}$:

Using Equation (59), for any $\beta\in^{d}$, we have

$\displaystyle\beta^{\top}X^{\text{\scriptsize$(1)$}}=\beta^{\top}H{b}+\beta^{\top}Ha^{\text{\scriptsize$(1)$}}_{X}+\beta^{\top}H\varepsilon^{\text{\scriptsize$(1)$}}_{X}.$

Together with the similar equation on target, we obtain

$\displaystyle{\mathbb{E}}\left[\beta^{\top}X^{\text{\scriptsize$(1)$}}\right]={\mathbb{E}}\left[\beta^{\top}\widetilde{X}\right]+\beta^{\top}H\left(a^{\text{\scriptsize$(1)$}}_{X}-\widetilde{a}_{X}\right).$
(65)

Combining Equation (60) and (65), we observe that the
DIP${}^{\text{\scriptsize$(1)$}}$ problem (• ‣ 3.4) becomes a constrained
quadratic program

$\displaystyle\min_{\beta,\beta_{0}}\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}+\left(\beta^{\top}Ha^{\text{\scriptsize$(1)$}}_{X}+\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta$ $\displaystyle\text{s.t.
}\beta^{\top}H\left(a^{\text{\scriptsize$(1)$}}_{X}-\widetilde{a}\right)=0.$
(66)

Note that because of the constraint, this quadratic program is the same for
the source data and for the target data. As a consequence, we have

$\displaystyle\tilde{R}\left(f_{\text{DIP}}^{\text{\scriptsize$(1)$}}\right)=\tilde{R}\left(f_{\text{DIPOracle}}^{\text{\scriptsize$(1)$}}\right).$

Now we solve the constrained quadratic program in Equation (B.1). First, we
observe that the minimization on $\beta_{0}$ can be easily solved. Second,
since $H$ is invertible, we can re-parametrize the original problem and obtain
the following problem

$\displaystyle\min_{\gamma}\left(1-\gamma^{\top}{b}\right)^{2}\sigma^{2}+\gamma^{\top}\Sigma\gamma$
$\displaystyle\text{s.t.
}\gamma^{\top}\left(a^{\text{\scriptsize$(1)$}}_{X}-\widetilde{a}\right)=0.$
(67)

Define
$u=\frac{a^{\text{\scriptsize$(1)$}}-\widetilde{a}}{\left\|a^{\text{\scriptsize$(1)$}}-\widetilde{a}\right\|_{2}}$.
Using Gram-Schmidt orthogonalization, we can complete the vector $u$ to form
an orthonormal basis $\left(u,q_{1},\ldots,q_{d-1}\right)$. Let
$Q_{\text{\tiny DIP}}\in^{d\times(d-1)}$ be the matrix formed with $i$-th
column being $q_{i}$. Then the mapping

d-1 $\displaystyle\rightarrow^{d}$ $\displaystyle\zeta$ $\displaystyle\mapsto
Q_{\text{\tiny DIP}}\zeta$

constitute a bijection between d-1 and the set
$\left\\{\gamma\in^{d}\mid\gamma^{\top}u=0\right\\}$. This bijection allows us
to transform the constrained quadratic program (B.1) to the following
unconstrained one

$\displaystyle\min_{\zeta\in^{d^{-1}}}\left(1-\zeta^{\top}Q_{\text{\tiny
DIP}}^{\top}{b}\right)^{2}\sigma^{2}+\zeta^{\top}Q_{\text{\tiny
DIP}}^{\top}\Sigma Q_{\text{\tiny DIP}}\zeta.$

Solving the quadratic program by setting gradient to zero, we obtain the
minimizer

$\displaystyle\zeta^{*}=\sigma^{2}\left(Q_{\text{\tiny
DIP}}^{\top}\left(\sigma^{2}{b}{b}^{\top}+\Sigma\right)Q_{\text{\tiny
DIP}}\right)^{-1}Q_{\text{\tiny DIP}}^{\top}{b}$

and

$\displaystyle\beta_{\text{DIP}}^{\text{\scriptsize$(1)$}}$
$\displaystyle=\sigma^{2}\left(\mathbb{I}_{d}-\mathbf{B}\right)^{\top}Q_{\text{\tiny
DIP}}\left(Q_{\text{\tiny
DIP}}^{\top}(\Sigma+\sigma^{2}{b}{b}^{\top})Q_{\text{\tiny
DIP}}\right)^{-1}Q_{\text{\tiny DIP}}^{\top}{b}$
$\displaystyle=\left(\mathbb{I}_{d}-\mathbf{B}\right)^{\top}\frac{\sigma^{2}Q_{\text{\tiny
DIP}}\left(Q_{\text{\tiny DIP}}^{\top}\Sigma Q_{\text{\tiny
DIP}}\right)^{-1}Q_{\text{\tiny
DIP}}^{\top}{b}}{1+\sigma^{2}{b}^{\top}Q_{\text{\tiny
DIP}}\left(Q_{\text{\tiny DIP}}^{\top}\Sigma Q_{\text{\tiny
DIP}}\right)^{-1}Q_{\text{\tiny DIP}}^{\top}{b}}$
$\displaystyle\beta_{\text{DIP},0}^{\text{\scriptsize$(1)$}}$
$\displaystyle=-{\beta_{\text{DIP}}^{\text{\scriptsize$(1)$}}}^{\top}Ha^{\text{\scriptsize$(1)$}}_{X},$
$\displaystyle\beta_{\text{DIPOracle}}^{\text{\scriptsize$(1)$}}$
$\displaystyle=\beta_{\text{DIP}}^{\text{\scriptsize$(1)$}}$
$\displaystyle\beta_{\text{DIPOracle},0}^{\text{\scriptsize$(1)$}}$
$\displaystyle=-{\beta_{\text{DIPOracle}}^{\text{\scriptsize$(1)$}}}^{\top}H\widetilde{a}_{X}.$

Consequently, the target population can be obtained by replacing ${b}$ with
$Q_{\text{\tiny DIP}}^{\top}{b}$ and $\Sigma$ with $Q_{\text{\tiny
DIP}}^{\top}\Sigma$ in Equation (B.1)

$\displaystyle\tilde{R}\left(f_{\text{DIP}{\text{\scriptsize$(1)$}}}\right)=\tilde{R}\left(f_{\text{DIPOracle}{\text{\scriptsize$(1)$}}}\right)=\frac{\sigma^{2}}{1+\sigma^{2}{b}^{\top}\Sigma^{-\frac{1}{2}}G_{\text{\tiny
DIP}}\Sigma^{-\frac{1}{2}}{b}}.$

where $G_{\text{\tiny DIP}}=\Sigma^{1/2}Q_{\text{\tiny
DIP}}\left(Q_{\text{\tiny DIP}}^{\top}\Sigma Q_{\text{\tiny
DIP}}\right)^{-1}Q_{\text{\tiny DIP}}^{\top}\Sigma^{1/2}$ is a projection
matrix with rank $d-1$.

###  B.2 Proof of Corollary 2

Equation (19), (20) and (21) follow directly by plugging in
$\Sigma=\frac{\sigma^{2}}{\rho}\mathbb{I}_{d}$ in the corresponding equations
in Theorem 1. For the high probability bound, we use several tail
inequalities. Let $a=a^{\text{\scriptsize$(1)$}}_{X}-\widetilde{a}_{X}$. Since
$a$ is generated randomly from $\mathcal{N}(0,\tau^{2}\mathbb{I}_{d})$, then
for a fixed vector $v\in^{d}$, $a^{\top}v$ follows
$\mathcal{N}(0,\tau^{2}v^{\top}v)$. The standard Gaussian tail bound on
$a^{\top}v$ gives, for $t>0$,

$\displaystyle{\mathbb{P}}\left(\left|\frac{a^{\top}v}{\tau\left(v^{\top}v\right)^{1/2}}\right|\geq
t\right)\leq 2\exp\left(-t^{2}/2\right).$ (68)

Since $\left\|a\right\|_{2}^{2}$ follows Chi-square distribution with
$d$-degree of freedom, the standard chi-square tail bound (see e.g. Laurent
and Massart (2000)) gives, for $t>0$,

$\displaystyle{\mathbb{P}}\left(\frac{\left\|a\right\|_{2}^{2}}{\tau^{2}d}\leq
1-\frac{t}{\sqrt{d}}\right)\leq\exp\left(-t^{2}/8\right).$ (69)

Combining Equation (68) and (69), with probability at least
$1-\exp(-t^{2}/8)-2\exp(-t^{2}/2)$, we have

$\displaystyle\frac{\left(a^{\top}v\right)^{2}}{\left\|a\right\|_{2}^{2}}\leq\frac{t^{2}}{d-\sqrt{d}t}\left(v^{\top}v\right).$

For $t$ constant satisfying $0<t\leq\frac{\sqrt{d}}{2}$, we have

$\displaystyle\frac{t^{2}}{d-\sqrt{d}t}\leq\frac{2t^{2}}{d}.$

Plugging the above high probability bound into Equation (20) and (21) with
$v={b}$, with probability at least $1-\exp(-t^{2}/8)-2\exp(-t^{2}/2)$, we have

$\displaystyle\tilde{R}\left(f_{\text{OLSSrc}}^{\text{\scriptsize$(1)$}}\right)$
$\displaystyle\leq\frac{\sigma^{2}}{1+\rho\left\|{b}\right\|_{2}^{2}}+\frac{\tau^{2}t^{2}\left\|{b}\right\|_{2}^{2}}{\left(1+\rho\left\|{b}\right\|_{2}^{2}\right)^{2}},\text{
and }$
$\displaystyle\tilde{R}\left(f_{\text{DIP}}^{\text{\scriptsize$(1)$}}\right)$
$\displaystyle\leq\frac{\sigma^{2}}{1+\rho\left(1-\frac{2t^{2}}{d}\right)\left\|{b}\right\|_{2}^{2}}.$

We conclude and obtain the form needed in the corollary by a change of
variable from $2t^{2}$ to $t$.

###  B.3 Proof of Corollary 3

Corollary 3 shows that the source population risk of
DIP${}^{\text{\scriptsize$(1)$}}$ is the same as target population risk of
DIP${}^{\text{\scriptsize$(1)$}}$. For this, it suffices to observe that the
source expected residual (60) and the target expected residual (61) only
differ by the term $\beta^{\top}Ha^{\text{\scriptsize$(1)$}}_{X}$ and the term
$\beta^{\top}H\widetilde{a}_{X}$. Since the DIP constraint enforces
$\beta^{\top}H\left(a^{\text{\scriptsize$(1)$}}_{X}-\widetilde{a}_{X}\right)=0$
as shown in Equation (B.1), we obtain that

$\displaystyle
R^{\text{\scriptsize$(1)$}}\left(f_{\text{DIP}}^{\text{\scriptsize$(1)$}}\right)=\tilde{R}\left(f_{\text{DIP}}^{\text{\scriptsize$(1)$}}\right).$
(70)

###  B.4 Proof of Corollary 4

Using the linear SEM assumption in Assumption 1 and the additional
intervention on $Y$, each data point in the source environment is generated
i.i.d. from the following equation

$\displaystyle X^{\text{\scriptsize$(1)$}}$
$\displaystyle=\mathbf{B}X^{\text{\scriptsize$(1)$}}+{b}Y^{\text{\scriptsize$(1)$}}+a^{\text{\scriptsize$(1)$}}_{X}+\varepsilon^{\text{\scriptsize$(1)$}}_{X},$
$\displaystyle Y^{\text{\scriptsize$(1)$}}$
$\displaystyle=a^{\text{\scriptsize$(1)$}}_{Y}+\varepsilon^{\text{\scriptsize$(1)$}}_{Y}.$
(71)

For $(\beta,\beta_{0})\in^{d}\times$, the source residual takes the following
form

$\displaystyle
Y^{\text{\scriptsize$(1)$}}-\beta^{\top}X^{\text{\scriptsize$(1)$}}-\beta_{0}=\left(1-\beta^{\top}H{b}\right)\varepsilon^{\text{\scriptsize$(1)$}}_{Y}+\left(\left(1-\beta^{\top}H{b}\right)a^{\text{\scriptsize$(1)$}}_{Y}-\beta^{\top}Ha^{\text{\scriptsize$(1)$}}_{X}-\beta_{0}\right)-\beta^{\top}H\varepsilon^{\text{\scriptsize$(1)$}}_{X}.$

The DIP${}^{\text{\scriptsize$(1)$}}$ problem (• ‣ 3.4) becomes a constrained
quadratic program

$\displaystyle\min_{\beta,\beta_{0}}\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}+\left(\left(1-\beta^{\top}H{b}\right)a^{\text{\scriptsize$(1)$}}_{Y}-\beta^{\top}Ha^{\text{\scriptsize$(1)$}}_{X}-\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta$ $\displaystyle\text{s.t.
}\beta^{\top}H\left(a^{\text{\scriptsize$(1)$}}_{X}+a^{\text{\scriptsize$(1)$}}_{Y}{b}\right)=\beta^{\top}H\left(\widetilde{a}_{X}+\widetilde{a}_{Y}{b}\right).$
(72)

Note that because of the intervention on $Y$, unlike in Theorem 1, this
quadratic program is no longer the same for DIP${}^{\text{\scriptsize$(1)$}}$
and DIPOracle${}^{\text{\scriptsize$(1)$}}$. However, the constrained
quadratic program (B.4) can be solved similarly as we did in Appendix B.1
around Equation (B.1) by introducing

$\displaystyle
u=\frac{a^{\text{\scriptsize$(1)$}}_{X}+a^{\text{\scriptsize$(1)$}}_{Y}{b}-\widetilde{a}_{X}-\widetilde{a}_{Y}{b}}{\left\|a^{\text{\scriptsize$(1)$}}_{X}+a^{\text{\scriptsize$(1)$}}_{Y}{b}-\widetilde{a}_{X}-\widetilde{a}_{Y}{b}\right\|_{2}}$

and $Q_{2}\in^{d\times d-1}$ is the matrix with columns formed by the vectors
that complete the vector $u$ to an orthonormal basis of d. Following the rest
of the proof in Appendix B.1, we obtain

$\displaystyle\beta_{\text{DIP}}^{\text{\scriptsize$(1)$}}$
$\displaystyle=\left(\mathbb{I}_{d}-\mathbf{B}\right)^{\top}\frac{\sigma^{2}Q_{2}\left(Q_{2}^{\top}\Sigma
Q_{2}\right)^{-1}Q_{2}^{\top}{b}}{1+{b}^{\top}Q_{2}\left(Q_{2}^{\top}\Sigma
Q_{2}\right)^{-1}Q_{2}^{\top}{b}}$
$\displaystyle\beta_{\text{DIP},0}^{\text{\scriptsize$(1)$}}$
$\displaystyle=\left(1-{\beta_{\text{DIP}}^{\text{\scriptsize$(1)$}}}^{\top}H{b}\right)a^{\text{\scriptsize$(1)$}}_{Y}-{\beta_{\text{DIP}}^{\text{\scriptsize$(1)$}}}^{\top}Ha^{\text{\scriptsize$(1)$}}_{X}.$
(73)

Note that the intercept $\beta_{\text{DIP},0}^{\text{\scriptsize$(1)$}}$ has
an extra term due to the intervention on $a^{\text{\scriptsize$(1)$}}_{Y}$.
For $(\beta,\beta_{0})\in^{d}\times$, the target residual takes the following
form

$\displaystyle\widetilde{Y}-\beta^{\top}\widetilde{X}-\beta_{0}=\left(1-\beta^{\top}H{b}\right)\widetilde{\varepsilon}_{Y}+\left(\left(1-\beta^{\top}H{b}\right)\widetilde{a}_{Y}-\beta^{\top}H\widetilde{a}_{X}-\beta_{0}\right)-\beta^{\top}H\widetilde{\varepsilon}_{X}.$

Plugging
$(\beta_{\text{DIP}}^{\text{\scriptsize$(1)$}},\beta_{\text{DIP},0}^{\text{\scriptsize$(1)$}})$
of Equation (B.4) into the above residual, we obtain that the population
target risk which has an extra term that depends on
$a^{\text{\scriptsize$(1)$}}_{Y}-\widetilde{a}_{Y}$

$\displaystyle\tilde{R}\left(f_{\text{DIP}{\text{\scriptsize$(1)$}}}\right)=\frac{\sigma^{2}}{1+\sigma^{2}{b}^{\top}\Sigma^{-\frac{1}{2}}G_{2}\Sigma^{-\frac{1}{2}}{b}}+\left(a^{\text{\scriptsize$(1)$}}_{Y}-\widetilde{a}_{Y}\right)^{2},$

where $G_{2}=\Sigma^{1/2}Q_{2}\left(Q_{2}^{\top}\Sigma
Q_{2}\right)^{-1}Q_{2}^{\top}\Sigma^{1/2}$ is a projection matrix with rank
$d-1$.

##  Appendix C Proofs related to Theorem 5

In this section, we prove Theorem 5 and related corollaries.

###  C.1 Proof of Theorem 5

Using the linear SEM assumption in Assumption 2, each data point in the $m$-th
source environment is generated i.i.d. from the following equation

$\displaystyle X^{\text{\scriptsize$(m)$}}$
$\displaystyle=\mathbf{B}X^{\text{\scriptsize$(m)$}}+{b}Y^{\text{\scriptsize$(m)$}}+a^{\text{\scriptsize$(m)$}}_{X}+\varepsilon^{\text{\scriptsize$(m)$}}_{X},$
$\displaystyle Y^{\text{\scriptsize$(m)$}}$
$\displaystyle=a^{\text{\scriptsize$(m)$}}_{Y}+\varepsilon^{\text{\scriptsize$(m)$}}_{Y}.$
(74)

Define $H=\left(\mathbb{I}_{d}-\mathbf{B}\right)^{-1}$. For
$(\beta,\beta_{0})\in^{d}\times$, the residual takes the following form

$\displaystyle
Y^{\text{\scriptsize$(m)$}}-\beta^{\top}X^{\text{\scriptsize$(m)$}}-\beta_{0}=\left(1-\beta^{\top}H{b}\right)\left(a^{\text{\scriptsize$(m)$}}_{Y}+\varepsilon^{\text{\scriptsize$(m)$}}_{Y}\right)-\left(\beta^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}+\beta_{0}\right)-\beta^{\top}H\varepsilon^{\text{\scriptsize$(m)$}}_{X}.$
(75)

The target residual has a similar form

$\displaystyle\widetilde{Y}-\beta^{\top}\widetilde{X}-\beta_{0}=\left(1-\beta^{\top}H{b}\right)\left(\widetilde{a}_{Y}+\widetilde{\varepsilon}_{Y}\right)-\left(\beta^{\top}H\widetilde{a}_{X}+\beta_{0}\right)-\beta^{\top}H\widetilde{\varepsilon}_{X}.$
(76)

##### Risk of OLSTar:

Using the expression for the target residual, the OLSTar estimator in Equation
(• ‣ 3.3) becomes the solution of the following quadratic program

$\displaystyle\min_{\beta,\beta_{0}}$
$\displaystyle\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}+\frac{1}{M}\sum_{m=1}^{M}\left(\left(1-\beta^{\top}H{b}\right)\widetilde{a}_{Y}-\beta^{\top}H\widetilde{a}_{X}-\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta.$

Solving the quadratic program by setting the gradient to zero, we obtain

$\displaystyle\beta_{\text{OLSTar}}$
$\displaystyle=\sigma^{2}\left(\mathbb{I}_{d}-\mathbf{B}\right)^{\top}\left(\Sigma+\sigma^{2}{b}{b}^{\top}\right)^{-1}{b}$
$\displaystyle\beta_{\text{OLSTar},0}$
$\displaystyle=\left(1-\beta_{\text{OLSTar}}^{\top}H{b}\right)\widetilde{a}_{Y}-\beta_{\text{OLSTar}}^{\top}H\widetilde{a}_{X}.$

Despite the difference in the intercept term, the corresponding target risk is
the same as in Theorem 1

$\displaystyle\tilde{R}\left(f_{\text{OLSTar}}\right)=\frac{\sigma^{2}}{1+\sigma^{2}{b}^{\top}\Sigma^{-1}{b}}.$

##### Risk of CIP:

Using the SEM (C.1), the constraint of CIP in Equation (• ‣ 3.4) becomes

$\displaystyle\beta^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}=\beta^{\top}Ha^{\text{\scriptsize$(1)$}}_{X},\quad\forall
m\in\left\\{2,\ldots,M\right\\}.$

Together with the residual expression in Equation (75), the CIP objective (• ‣
3.4) is equivalent to

$\displaystyle\min_{\beta,\beta_{0}}$
$\displaystyle\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}+\frac{1}{M}\sum_{m=1}^{M}\left(\left(1-\beta^{\top}H{b}\right)a^{\text{\scriptsize$(m)$}}_{Y}-\beta^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}-\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta$ s.t.
$\displaystyle\beta^{\top}H\left(a^{\text{\scriptsize$(m)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X}\right)=0,\quad\forall
m\in\left\\{2,\ldots,M\right\\}.$ (77)

First, the one-dimensional quadratic program on $\beta_{0}$ can be solved
easily by setting derivative to zero and we obtain

$\displaystyle\beta_{0}^{*}=\frac{1}{M}\sum_{m=1}^{M}\left(1-{\beta^{*}}^{\top}H{b}\right)a^{\text{\scriptsize$(m)$}}_{Y}-{\beta^{*}}^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}.$

Plugging the expression of $\beta_{0}$ back to Equation (C.1), the
minimization on $\beta$ becomes

$\displaystyle\min_{\beta}$
$\displaystyle\left(1-\beta^{\top}H{b}\right)^{2}\left(\sigma^{2}+\Delta_{Y}\right)+\beta^{\top}H\Sigma
H^{\top}\beta$ s.t.
$\displaystyle\beta^{\top}H\left(a^{\text{\scriptsize$(m)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X}\right)=0,\
\forall m\in\left\\{2,\ldots,M\right\\},$

where

$\displaystyle\Delta_{Y}=\frac{1}{M}\sum_{m=1}^{M}\left(a^{\text{\scriptsize$(m)$}}_{Y}-\bar{a}_{Y}\right)^{2}\text{
and }\bar{a}_{Y}=\frac{1}{M}\sum_{k=1}^{M}a^{\text{\scriptsize$(k)$}}_{Y}.$
(78)

Second, since $H$ is invertible, we can re-parametrize the optimization on
$\beta$ as follows

$\displaystyle\min_{\gamma\in^{d}}$
$\displaystyle\left(1-\gamma^{\top}{b}\right)^{2}\left(\sigma^{2}+\Delta_{Y})\right)+\gamma^{\top}\Sigma\gamma$
s.t.
$\displaystyle\gamma^{\top}\left(a^{\text{\scriptsize$(m)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X}\right)=0,\
\forall m\in\left\\{2,\ldots,M\right\\}.$ (79)

Let $P\in^{d\times p}$ be the matrix formed with an orthonormal basis of the
$p$-dimensional subspace
$\text{span}\left(a^{\text{\scriptsize$(2)$}}-a^{\text{\scriptsize$(1)$}},\ldots,a^{\text{\scriptsize$(m)$}}-a^{\text{\scriptsize$(1)$}}\right)$.
Let $Q_{\text{\tiny CIP}}\in^{d\times(d-p)}$ be the matrix with columns formed
by completing the columns of $P$ to a basis of d via Gram-Schmidt
orthogonalization. The following mapping

d-p $\displaystyle\rightarrow^{d}$ $\displaystyle\zeta$ $\displaystyle\mapsto
Q_{\text{\tiny CIP}}\zeta$

constitutes a bijection between d-p and the set $\left\\{\gamma\in^{d}\mid
P^{\top}\gamma=0\right\\}$. With the change of variable, the constrained
optimization in Equation (C.1) is equivalent to the unconstrained one

$\displaystyle\min_{\zeta\in^{d-p}}\left(1-\zeta^{\top}Q_{\text{\tiny
CIP}}^{\top}{b}\right)^{2}\left(\sigma^{2}+\Delta_{Y}\right)+\zeta^{\top}Q_{\text{\tiny
CIP}}^{\top}\Sigma Q_{\text{\tiny CIP}}\zeta.$

Solving the unconstrained quadratic program by setting gradient to zero, we
obtain the minimizer

$\displaystyle\zeta^{*}$
$\displaystyle=\left(\sigma^{2}+\Delta_{Y}\right)\left(Q_{\text{\tiny
CIP}}^{\top}\left(\left(\sigma^{2}+\Delta_{Y}\right){b}{b}^{\top}+\Sigma\right)Q_{\text{\tiny
CIP}}\right)^{-1}Q_{\text{\tiny CIP}}^{\top}{b}$
$\displaystyle=\left(\sigma^{2}+\Delta_{Y}\right)\frac{Q_{\text{\tiny
CIP}}\left({Q_{\text{\tiny CIP}}}^{\top}\Sigma Q_{\text{\tiny
CIP}}\right)^{-1}Q_{\text{\tiny
CIP}}^{\top}{b}}{1+\left(\sigma^{2}+\Delta_{Y}\right){b}^{\top}Q_{\text{\tiny
CIP}}\left({Q_{\text{\tiny CIP}}}^{\top}\Sigma Q_{\text{\tiny
CIP}}\right)^{-1}Q_{\text{\tiny CIP}}^{\top}{b}},$

where the last equality uses the matrix inversion lemma. Finally, transforming
the variables back, the CIP estimator is

$\displaystyle\beta_{\text{CIP}}$
$\displaystyle=\left(\sigma^{2}+\Delta_{Y}\right)\frac{\left(\mathbb{I}_{d}-\mathbf{B}\right)^{\top}Q_{\text{\tiny
CIP}}\left({Q_{\text{\tiny CIP}}}^{\top}\Sigma Q_{\text{\tiny
CIP}}\right)^{-1}Q_{\text{\tiny
CIP}}^{\top}{b}}{1+\left(\sigma^{2}+\Delta_{Y}\right){b}^{\top}Q_{\text{\tiny
CIP}}\left({Q_{\text{\tiny CIP}}}^{\top}\Sigma Q_{\text{\tiny
CIP}}\right)^{-1}Q_{\text{\tiny CIP}}^{\top}{b}}$ (80)
$\displaystyle\beta_{\text{CIP},0}$
$\displaystyle=\left(1-{\beta_{\text{CIP}}}^{\top}H{b}\right)\bar{a}_{Y}-{\beta_{\text{CIP}}}^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}.$

According to the form of the target residual (76), the target population risk
for $(\beta,\beta_{0})$ takes the following form

$\displaystyle\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}+\left(\left(1-\beta^{\top}H{b}\right)\widetilde{a}_{Y}-\beta^{\top}H\widetilde{a}_{X}-\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta.$

Plugging in the CIP estimator into the equation above, we obtain the target
population risk of CIP

$\displaystyle\tilde{R}\left(f_{\text{CIP}}\right)=\frac{\sigma^{2}+\Delta_{Y}}{1+\left(\sigma^{2}+\Delta_{Y}\right){b}^{\top}\Sigma^{-\frac{1}{2}}G_{\text{\tiny
CIP}}\Sigma^{-\frac{1}{2}}{b}}+\frac{\left(\widetilde{a}_{Y}-\bar{a}_{Y}\right)^{2}-\Delta_{Y}}{\left(1+\left(\sigma^{2}+\Delta_{Y}\right){b}^{\top}\Sigma^{-\frac{1}{2}}G_{\text{\tiny
CIP}}\Sigma^{-\frac{1}{2}}{b}\right)^{2}},$

where $G_{\text{\tiny CIP}}=\Sigma^{1/2}Q_{\text{\tiny
CIP}}\left(Q_{\text{\tiny CIP}}^{\top}\Sigma Q_{\text{\tiny
CIP}}\right)^{-1}Q_{\text{\tiny CIP}}^{\top}\Sigma^{1/2}$ is a projection
matrix with rank $d-p$.

##### Risk of CIRM:

First we derive $\vartheta_{\text{CIRM}}$ in Equation (13). From the CIP
expression in Equation (80) and the SEM (C.1), we obtain the following
expression for ${X^{\text{\scriptsize$(m)$}}}^{\top}\beta_{\text{CIP}}$

$\displaystyle\frac{\left(\sigma^{2}+\Delta_{Y}\right)\left(\varepsilon^{\text{\scriptsize$(m)$}}_{Y}{b}^{\top}\Sigma^{-1/2}G_{\text{\tiny
CIP}}\Sigma^{-1/2}{b}+{\varepsilon^{\text{\scriptsize$(m)$}}_{X}}^{\top}\Sigma^{-1/2}G_{\text{\tiny
CIP}}\Sigma^{-1/2}{b}+{a^{\text{\scriptsize$(m)$}}_{X}}^{\top}\Sigma^{-1/2}G_{\text{\tiny
CIP}}\Sigma^{-1/2}{b}\right)}{1+\left(\sigma^{2}+\Delta_{Y}\right){b}^{\top}\Sigma^{-1/2}G_{\text{\tiny
CIP}}\Sigma^{-1/2}{b}}.$

The deviation from its expectation is

$\displaystyle{X^{\text{\scriptsize$(m)$}}}^{\top}\beta_{\text{CIP}}-{\mathbb{E}}\left[{X^{\text{\scriptsize$(m)$}}}^{\top}\beta_{\text{CIP}}\right]$
$\displaystyle=\frac{\left(\sigma^{2}+\Delta_{Y}\right){b}^{\top}\Sigma^{-1/2}G_{\text{\tiny
CIP}}\Sigma^{-1/2}{b}}{1+\left(\sigma^{2}+\Delta_{Y}\right){b}^{\top}\Sigma^{-1/2}G_{\text{\tiny
CIP}}\Sigma^{-1/2}{b}}\cdot\varepsilon^{\text{\scriptsize$(m)$}}_{Y}+\frac{\left(\sigma^{2}+\Delta_{Y}\right){\varepsilon^{\text{\scriptsize$(m)$}}_{X}}^{\top}\Sigma^{-1/2}G_{\text{\tiny
CIP}}\Sigma^{-1/2}{b}}{1+\left(\sigma^{2}+\Delta_{Y}\right){b}^{\top}\Sigma^{-1/2}G_{\text{\tiny
CIP}}\Sigma^{-1/2}{b}}.$

Plugging the above equation into Equation (13), together with

$\displaystyle
X^{\text{\scriptsize$(m)$}}=H{b}Y^{\text{\scriptsize$(m)$}}+Ha^{\text{\scriptsize$(m)$}}_{X}+H\varepsilon^{\text{\scriptsize$(m)$}}_{X}$
(81)

and
$Y^{\text{\scriptsize$(m)$}}-{\mathbb{E}}\left[Y^{\text{\scriptsize$(m)$}}\right]=\varepsilon^{\text{\scriptsize$(m)$}}_{Y}$,
we obtain

$\displaystyle\vartheta_{\text{CIRM}}=\left(\frac{1+\left(\sigma^{2}+\Delta_{Y}\right){b}^{\top}\Sigma^{-1/2}G_{\text{\tiny
CIP}}\Sigma^{-1/2}{b}}{\left(\sigma^{2}+\Delta_{Y}\right){b}^{\top}\Sigma^{-1/2}G_{\text{\tiny
CIP}}\Sigma^{-1/2}{b}}\right)H{b}.$ (82)

Note that the vector $\vartheta_{\text{CIRM}}$ is co-linear with the $Y$
component in Equation (81). We remark that the idea of CIRM is to use
${\widetilde{X}}^{\top}\beta_{\text{CIP}}$ as a proxy for the unobserved
$\widetilde{Y}$ to remove the $\widetilde{Y}$ part in the covariates so that
the DIP matching based ideas still can be applied.

With the $\vartheta_{\text{CIRM}}$ expression (82) and the
$\beta_{\text{CIP}}$ expression (80), the LHS of the CIRM constraint (• ‣ 3.4)
becomes

$\displaystyle
X^{\text{\scriptsize$(m)$}}-\left({X^{\text{\scriptsize$(m)$}}}^{\top}\beta_{\text{CIP}}\right)\vartheta_{\text{CIRM}}$
$\displaystyle=\left(Y^{\text{\scriptsize$(m)$}}-\left({X^{\text{\scriptsize$(m)$}}}^{\top}\beta_{\text{CIP}}\right)\cdot\frac{1+\left(\sigma^{2}+\Delta_{Y}\right){b}^{\top}\Sigma^{-1/2}G_{\text{\tiny
CIP}}\Sigma^{-1/2}{b}}{\left(\sigma^{2}+\Delta_{Y})\right){b}^{\top}\Sigma^{-1/2}G_{\text{\tiny
CIP}}\Sigma^{-1/2}{b}}\right)H{b}+Ha^{\text{\scriptsize$(m)$}}_{X}+H\varepsilon^{\text{\scriptsize$(m)$}}_{X}$
$\displaystyle=\left(\beta_{\text{CIP}}^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}+\beta_{\text{CIP}}^{\top}H\varepsilon^{\text{\scriptsize$(m)$}}_{X}\right)H{b}+Ha^{\text{\scriptsize$(m)$}}_{X}+H\varepsilon^{\text{\scriptsize$(m)$}}_{X}$
$\displaystyle\stackrel{{\scriptstyle(i)}}{{=}}\left(\beta_{\text{CIP}}^{\top}Ha^{\text{\scriptsize$(1)$}}_{X}+\beta_{\text{CIP}}^{\top}H\varepsilon^{\text{\scriptsize$(m)$}}_{X}\right)H{b}+Ha^{\text{\scriptsize$(m)$}}_{X}+H\varepsilon^{\text{\scriptsize$(m)$}}_{X},$
(83)

where the last inequality (i) follows from the fact that the constraint in CIP
(• ‣ 3.4) forces

$\displaystyle{\beta_{\text{CIP}}}^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}={\beta_{\text{CIP}}}^{\top}Ha^{\text{\scriptsize$(1)$}}_{X},\quad\forall
m\in\left\\{1,\ldots,M\right\\}.$

An expression similar to Equation (C.1) can be obtained for the target
environments by taking into account that
$\widetilde{a}_{X}\in\text{span}\left(a^{\text{\scriptsize$(1)$}}_{X},\ldots,a^{\text{\scriptsize$(M)$}}_{X}\right)$,

$\displaystyle\widetilde{X}-\left({\widetilde{X}}^{\top}\beta_{\text{CIP}}\right)\vartheta_{\text{CIRM}}=H{b}\left(\beta_{\text{CIP}}^{\top}Ha^{\text{\scriptsize$(1)$}}_{X}+\beta_{\text{CIP}}^{\top}H\widetilde{\varepsilon}_{X}\right)+H\widetilde{a}_{X}+H\widetilde{\varepsilon}_{X}.$
(84)

Multiply Equation (C.1) and (84) by $\beta$ and take expectation, we obtain a
simplified form of the CIRM constraint (• ‣ 3.4) as follows

$\displaystyle\beta^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}=\beta^{\top}H\widetilde{a}_{X}.$

Note that the CIRM constraint is effectively matching the covariate
interventions as DIP constraint did when $Y$ is not intervened on. As a
consequence, given $\beta_{\text{CIP}}$ and $\vartheta_{\text{CIRM}}$, the
CIRM${}^{\text{\scriptsize$(m)$}}$ estimator (• ‣ 3.4) is very similar to
DIP${}^{\text{\scriptsize$(m)$}}$ (B.1) except that the intervention on $Y$
still appears in the residual. The CIRM${}^{\text{\scriptsize$(m)$}}$
estimator (• ‣ 3.4) can be written as follows

$\displaystyle\min_{\beta,\beta_{0}}\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}+\left(\left(1-\beta^{\top}H{b}\right)a^{\text{\scriptsize$(m)$}}_{Y}-\beta^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}-\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta$ $\displaystyle\text{s.t.
}\beta^{\top}H\left(a^{\text{\scriptsize$(m)$}}_{X}-\widetilde{a}_{X}\right)=0.$
(85)

After solving for $\beta_{0}$, the quadratic program for $\beta$ is exactly
the same as in the DIP proof in Appendix B.1 around Equation (B.1). Thus we
obtain

$\displaystyle{\beta_{\text{CIRM}}^{\text{\scriptsize$(m)$}}}$
$\displaystyle=\left(\mathbb{I}_{d}-\mathbf{B}\right)^{\top}\frac{\sigma^{2}{Q_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}\left({Q_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}^{\top}\Sigma{Q_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}\right)^{-1}{Q_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}^{\top}{b}}{1+\sigma^{2}{b}^{\top}{Q_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}\left({Q_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}^{\top}\Sigma{Q_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}\right)^{-1}{Q_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}^{\top}{b}}$
$\displaystyle\beta_{\text{CIRM},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle=\left(1-{{\beta_{\text{CIRM}}^{\text{\scriptsize$(m)$}}}}^{\top}H{b}\right)a^{\text{\scriptsize$(m)$}}_{Y}-{{\beta_{\text{CIRM}}^{\text{\scriptsize$(m)$}}}}^{\top}Ha^{\text{\scriptsize$(m)$}}_{X},$
(86)

the corresponding CIRM target population risk is

$\displaystyle\tilde{R}\left(f_{\text{CIRM}{\text{\scriptsize$(m)$}}}\right)=\frac{\sigma^{2}}{1+\sigma^{2}{b}^{\top}\Sigma^{-\frac{1}{2}}{G_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}\Sigma^{-\frac{1}{2}}{b}}+\frac{\left(a^{\text{\scriptsize$(m)$}}_{Y}-\widetilde{a}_{Y}\right)^{2}}{\left(1+\sigma^{2}{b}^{\top}\Sigma^{-\frac{1}{2}}{G_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}\Sigma^{-\frac{1}{2}}{b}\right)^{2}},$

where ${Q_{\text{\tiny DIP}}^{\text{\scriptsize$(m)$}}}$ and ${G_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}$ are in the same way as ${Q_{\text{\tiny
DIP}}^{\text{\scriptsize$(1)$}}}$ and ${G_{\text{\tiny
DIP}}^{\text{\scriptsize$(1)$}}}$ in Theorem 1.

${G_{\text{\tiny DIP}}^{\text{\scriptsize$(m)$}}}=\Sigma^{1/2}{Q_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}\left({Q_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}^{\top}\Sigma Q_{\text{\tiny
DIP}}{m}\right)^{-1}{Q_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}^{\top}\Sigma^{1/2}$ is a projection matrix.
${Q_{\text{\tiny DIP}}^{\text{\scriptsize$(m)$}}}\in^{d\times d-1}$ is the
matrix with columns formed by the vectors that complete the vector $u$ to an
orthonormal basis where

$\displaystyle
u^{\text{\scriptsize$(m)$}}=\begin{cases}\frac{a^{\text{\scriptsize$(m)$}}-\widetilde{a}}{\left\|a^{\text{\scriptsize$(m)$}}-\widetilde{a}\right\|_{2}},&\text{if
}a^{\text{\scriptsize$(m)$}}\neq\widetilde{a}\\\
0,&\text{otherwise.}\end{cases}$

###  C.2 Proof of Corollary 6

Equation (31), (32) and (33) follow directly by plugging in
$\Sigma=\frac{\sigma^{2}}{\rho}\mathbb{I}_{d}$ and intervention on $Y$ equals
to zero in the corresponding formula in Theorem 5. Recall that $P_{\text{\tiny
CIP}}\in^{d\times p}$ is the matrix with columns formed with the orthonormal
basis of
$\text{span}\left(a^{\text{\scriptsize$(2)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X},\ldots,a^{\text{\scriptsize$(M)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X}\right)$.
Let $A\in^{d\times(M-1)}$ with $(m-1)$-th column
$a^{\text{\scriptsize$(m)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X}$. According
the assumption that
$a^{\text{\scriptsize$(2)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X},\ldots
a^{\text{\scriptsize$(M)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X}$ are generated
independently from the standard Gaussian distribution, $\frac{1}{d}A^{\top}A$
follows a multivariate Wishart distribution and its eigenvalue tail bound is
well known (see e.g. Chapter 6 in Wainwright (2019)). We have

$\displaystyle{\mathbb{P}}\left(\lambda_{\text{min}}\left(A^{\top}A\right)\leq
d\left(\left(1-\delta\right)-\sqrt{\frac{M-1}{d}}\right)^{2}\right)\leq\exp\left(-d\delta^{2}/2\right).$
$\displaystyle{\mathbb{P}}\left(\lambda_{\text{max}}\left(A^{\top}A\right)\geq
d\left(\left(1+\delta\right)+\sqrt{\frac{M-1}{d}}\right)^{2}\right)\leq\exp\left(-d\delta^{2}/2\right).$

It implies with probability at least $1-2\exp(-d\delta^{2}/2)$, for
$\delta<\frac{1}{2}-\sqrt{\frac{M-1}{d}}$, we have

$\displaystyle\lambda_{\text{min}}\left(A^{\top}A\right)>\frac{d}{2}\text{ and
}\lambda_{\text{max}}\left(A^{\top}A\right)<\frac{3d}{2}.$

This also implies that with the same probability $A^{\top}A$ is full rank and
$p=M-1$.

For a fixed vector $v\in^{d}$, we have

$\displaystyle\left\|P_{\text{\tiny
CIP}}^{\top}v\right\|_{2}^{2}=v^{\top}A\left(A^{\top}A\right)^{-1}A^{\top}v.$

Note that

$\displaystyle
A^{\top}v=\sum_{m=2}^{M}\left(a^{\text{\scriptsize$(m)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X}\right)^{\top}v$

is a sum of $(M-1)$ i.i.d. Gaussian random variables with mean $0$ and
variance $v^{\top}v$. Using the standard chi-square tail bound, we have

$\displaystyle{\mathbb{P}}\left(\left\|A^{\top}v\right\|_{2}^{2}\geq(M-1)v^{\top}v\left(1+\frac{t}{\sqrt{M-1}}\right)\right)\leq\exp\left(-t^{2}/8\right).$
$\displaystyle{\mathbb{P}}\left(\left\|A^{\top}v\right\|_{2}^{2}\leq(M-1)v^{\top}v\left(1-\frac{t}{\sqrt{M-1}}\right)\right)\leq\exp\left(-t^{2}/8\right).$

Combining the high probability bounds above, we have, with probability at
least $1-2\exp\left(-d\delta^{2}/2\right)-2\exp\left(-t^{2}/8\right)$,

$\displaystyle\frac{(M-1)}{3d}v^{\top}v\leq\left\|P_{\text{\tiny
CIP}}^{\top}v\right\|_{2}^{2}\leq\frac{3(M-1)}{d}v^{\top}v,$

given that $\delta<\frac{1}{2}-\sqrt{\frac{M-1}{d}}$ and
$t\leq\frac{\sqrt{M-1}}{2}$. Note that it is possible to ensure
$1-2\exp\left(-d\delta^{2}/2\right)-2\exp\left(-t^{2}/8\right)$ to be close to
$1$ when $d$ and $M$ are both large and $M\ll d$. Taking
$t=\frac{\sqrt{M-1}}{2}$ and $d=6M$, this probability is larger than
$1-2\exp(-d/32)-2\exp(-M/32)$.

The high probability bound for $\left(u^{\top}{b}\right)^{2}$ can be obtained
similarly as in the proof of Corollary 2. According to the proof of Corollary
2 in Appendix B.2, for $0<t\leq d/2$, with probability at least
$1-\exp(-t/16)-2\exp(-t/4)$, we have

$\displaystyle\left(u^{\top}{b}\right)^{2}\leq\frac{t}{d}\left\|{b}\right\|_{2}^{2}.$

Putting the two high probability bound together, we conclude Corollary 6.

###  C.3 Proof of Corollary 8

This corollary follows from the proof of Theorem 5 in Appendix C.1. According
to Equation (C.1), the $m$-th source population risk can be written as

$\displaystyle\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}+\left(\left(1-\beta^{\top}H{b}\right)a^{\text{\scriptsize$(m)$}}_{Y}-\beta^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}-\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta.$

Similarly, the target population risk can be written as

$\displaystyle\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}+\left(\left(1-\beta^{\top}H{b}\right)\widetilde{a}_{Y}-\beta^{\top}H\widetilde{a}_{X}-\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta.$

The CIRM constraint ensures that
$\beta^{\top}H\left(a^{\text{\scriptsize$(m)$}}_{X}-\widetilde{a}_{X}\right)=0$
according to Equation (C.1). Hence the only difference between the source
population risk and target population risk lies in the
$a^{\text{\scriptsize$(m)$}}_{Y}$ and $\widetilde{a}_{Y}$ dependent terms.
Plugging the CIRM solution (C.1) into the source and target population risks,
we obtain

$\displaystyle
R^{\text{\scriptsize$(m)$}}\left(f_{\text{CIRM}}^{\text{\scriptsize$(m)$}}\right)$
$\displaystyle=\tilde{R}\left(f_{\text{CIRM}}^{\text{\scriptsize$(m)$}}\right)+\left(1-{\beta_{\text{CIRM}}^{\text{\scriptsize$(m)$}}}^{\top}H{b}\right)^{2}\left(a^{\text{\scriptsize$(m)$}}_{Y}-\widetilde{a}_{Y}\right)^{2}$
$\displaystyle=\tilde{R}\left(f_{\text{CIRM}}^{\text{\scriptsize$(m)$}}\right)+\frac{\left(a^{\text{\scriptsize$(m)$}}_{Y}-\widetilde{a}_{Y}\right)^{2}}{\left(1+\sigma^{2}{b}^{\top}\Sigma^{-\frac{1}{2}}{G_{\text{\tiny
DIP}}^{\text{\scriptsize$(m)$}}}\Sigma^{-\frac{1}{2}}{b}\right)^{2}}.$

###  C.4 Proof of Corollary 7

##### Risk of DIP$\diamondsuit$:

Since $X_{\text{P}}^{\text{\scriptsize$(m)$}}$ is uncorrelated with
$\varepsilon^{\text{\scriptsize$(m)$}}_{Y}$, regressing
$Y^{\text{\scriptsize$(m)$}}$ on $X_{\text{P}}^{\text{\scriptsize$(m)$}}$
gives back the coefficients in the data generation SCM. Solving Equation (39),
we obtain

$\displaystyle\gamma^{\text{\scriptsize$(m)$}}={\omega}_{\text{P}}.$

Similarly, since $X_{\text{P}}^{\text{\scriptsize$(m)$}}$ is uncorrelated with
$X_{\text{D}}^{\text{\scriptsize$(m)$}}$, regressing of
$X_{\text{D}}^{\text{\scriptsize$(m)$}}$ on
$X_{\text{P}}^{\text{\scriptsize$(m)$}}$ gives back the coefficients in the
data generation SCM. Solving Equation (40), we obtain

$\displaystyle{\Gamma^{\text{\scriptsize$(m)$}}}^{\top}=\left(\mathbb{I}_{d-r}-\mathbf{B}_{\text{D}}\right)^{-1}{b}_{\text{D}}{\omega}_{\text{P}}^{\top}+\left(\mathbb{I}_{d-r}-\mathbf{B}_{\text{D}}\right)^{-1}\mathbf{B}_{\text{d-p}}.$

Use the definition of intermediate random variables in Equation (4.3.2), we
observe that these variables satisfy the anticausal data generation Assumption
1

$\displaystyle\begin{bmatrix}X^{\text{\scriptsize$(m)$}}_{\mathcal{I}}\\\
Y^{\text{\scriptsize$(m)$}}_{\mathcal{I}}\end{bmatrix}$
$\displaystyle=\begin{bmatrix}\mathbf{B}_{\text{D}}&{b}_{\text{D}}\\\
0&0\end{bmatrix}\begin{bmatrix}X^{\text{\scriptsize$(m)$}}_{\mathcal{I}}\\\
Y^{\text{\scriptsize$(m)$}}_{\mathcal{I}}\end{bmatrix}+\begin{bmatrix}a^{\text{\scriptsize$(m)$}}_{X,\text{D}}\\\
a^{\text{\scriptsize$(m)$}}_{Y}\end{bmatrix}+\begin{bmatrix}\varepsilon^{\text{\scriptsize$(m)$}}_{X,\text{D}}\\\
\varepsilon^{\text{\scriptsize$(m)$}}_{Y}\end{bmatrix}$
$\displaystyle\begin{bmatrix}\widetilde{X}_{\mathcal{I}}\\\
\widetilde{Y}_{\mathcal{I}}\end{bmatrix}$
$\displaystyle=\begin{bmatrix}\mathbf{B}_{\text{D}}&{b}_{\text{D}}\\\
0&0\end{bmatrix}\begin{bmatrix}\widetilde{X}_{\mathcal{I}}\\\
\widetilde{Y}_{\mathcal{I}}\end{bmatrix}+\begin{bmatrix}\widetilde{a}_{X,\text{D}}\\\
\widetilde{a}_{Y}\end{bmatrix}+\begin{bmatrix}\widetilde{\varepsilon}_{X,\text{D}}\\\
\widetilde{\varepsilon}_{Y}\end{bmatrix}.$

Together with the assumption of no intervention on $Y$, we can apply Theorem 1
on the intermediate random variables to obtain the target risk of
DIP$\diamondsuit$.

##### Risk of OLSTar:

The data generation Assumption 3 gives the following equations

$\displaystyle\widetilde{X}_{\text{P}}$
$\displaystyle=H_{\text{P}}\widetilde{a}_{X,\text{P}}+H_{\text{P}}\widetilde{\varepsilon}_{X,\text{P}}$
$\displaystyle\widetilde{X}_{\text{D}}$
$\displaystyle=H_{\text{D}}{b}_{\text{D}}Y+H_{\text{D}}\mathbf{B}_{\text{d-p}}\widetilde{X}_{\text{P}}+H_{\text{D}}\widetilde{a}_{X,\text{D}}+H_{\text{D}}\widetilde{\varepsilon}_{X,\text{D}}$
$\displaystyle\widetilde{Y}$
$\displaystyle={\omega}_{\text{P}}^{\top}\widetilde{X}_{\text{P}}+\widetilde{\varepsilon}_{Y},$

where $H_{\text{P}}=\left(\mathbb{I}_{r}-\mathbf{B}_{\text{P}}\right)^{-1}$,
$H_{\text{D}}=\left(\mathbb{I}_{d-r}-\mathbf{B}_{\text{D}}\right)^{-1}$. The
target population risk for
$\left(\beta_{\text{P}},\beta_{\text{D}},\beta_{0}\right)$ becomes

$\displaystyle{\mathbb{E}}\left(\widetilde{Y}-\beta_{\text{P}}^{\top}\widetilde{X}_{\text{P}}-\beta_{\text{D}}^{\top}\widetilde{X}_{\text{D}}-\beta_{0}\right)^{2}$
$\displaystyle=\left(1-\beta_{\text{D}}^{\top}H_{\text{D}}{\omega}_{\text{D}}\right)^{2}\sigma^{2}+{\mathbb{E}}\left({\omega}_{\text{P}}^{\top}\widetilde{X}_{\text{P}}-\beta_{\text{P}}^{\top}\widetilde{X}_{\text{P}}-\beta_{\text{D}}^{\top}H_{\text{D}}{b}_{\text{D}}{\omega}_{\text{P}}^{\top}\widetilde{X}_{\text{P}}-\beta_{\text{D}}^{\top}H_{\text{D}}\mathbf{B}_{\text{d-p}}\widetilde{X}_{\text{P}}\right)^{2}$
$\displaystyle+\left(\beta_{\text{D}}^{\top}H_{\text{D}}\widetilde{a}_{X,\text{D}}+\beta_{0}\right)^{2}+\beta_{\text{D}}^{\top}H_{\text{D}}\Sigma_{\text{D}}H_{\text{D}}^{\top}\beta_{\text{D}}.$

The minimization on $\beta_{\text{P}}$ and $\beta_{0}$ can be solved easily
and it remains a quadratic program on $\beta_{\text{D}}$. Since this quadratic
program is similar to that in the proof of Theorem 1, we conclude by the
referring to the part where we solve the quadratic program in Appendix B.1.

##  Appendix D Additional CIRM variants: RII, RIIRMI, CIRMI

The target population risk of CIP and CIRM estimator discussed above still
have dependence on $a^{\text{\scriptsize$(1)$}}_{Y}-\widetilde{a}_{Y}$. As we
have explained after Theorem 5, this is because Assumption 2 does not rule out
the unidentifiable scenario. When we add additional assumptions such as
${b}\notin\text{span}\left(a^{\text{\scriptsize$(2)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X},\ldots,a^{\text{\scriptsize$(M)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X}\right)$,
we show that there are new methods which take advantage of this assumption to
get rid of $a^{\text{\scriptsize$(1)$}}_{Y}-\widetilde{a}_{Y}$ dependence in
the risk. We introduce three new estimators.

  * •

RII-mean: the population residual invariance and independent estimator where
mean is matched across environments

$\displaystyle f_{\text{RII}}(x)$
$\displaystyle:=x^{\top}\beta_{\text{RII}}+\beta_{\text{RII},0}$
$\displaystyle\beta_{\text{RII}},\beta_{\text{RII},0}$
$\displaystyle:=\operatorname*{arg\,min}_{\beta,\beta_{0}}\
\frac{1}{M}\sum_{k=1}^{M}{\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{\scriptsize$(k)$}}}\left(Y-X^{\top}\beta-\beta_{0}\right)^{2}$
$\displaystyle\text{s.t.\
}{\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{\scriptsize$(m)$}}}\left[Y-X^{\top}\beta\right]={\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{\scriptsize$(1)$}}}\left[Y-X^{\top}\beta\right]$
$\displaystyle\text{and\
}{\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{\scriptsize$(m)$}}}\left[(Y-{\mathbb{E}}_{Y\sim\mathcal{P}^{\text{\scriptsize$(m)$}}_{Y}}[Y])\cdot(Y-X^{\top}\beta)\right]=0,\forall
m\in\left\\{2,\ldots,M\right\\}.$ (87)

The idea of matching the residual has appeared in the invariant causal
prediction paper Peters et al. (2016) and in the anchor regression paper
Rothenhäusler et al. (2021). The residual independence penalty is also the
core idea in the invariant causal prediction Peters et al. (2016) and
Greenfeld and Shalit Greenfeld and Shalit (2020).

  * •

RIIRMI${}^{\text{\scriptsize$(m)$}}$-mean: the population residual invariant
independent residual matching estimator with additional residual independence
using $m$-th source environment

$\displaystyle f_{\text{RIIRMI}}^{\text{\scriptsize$(m)$}}(x)$
$\displaystyle:=x^{\top}\beta_{\text{RIIRMI}}^{\text{\scriptsize$(m)$}}+\beta_{\text{RIIRMI},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle\beta_{\text{RIIRMI}}^{\text{\scriptsize$(m)$}},\beta_{\text{RIIRMI},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle:=\operatorname*{arg\,min}_{\beta,\beta_{0}}\
{\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{\scriptsize$(m)$}}}\left(Y-X^{\top}\beta-\beta_{0}\right)^{2}$
$\displaystyle\text{s.t.\
}{\mathbb{E}}_{X\sim\mathcal{P}^{\text{\scriptsize$(m)$}}_{X}}$
$\displaystyle\left[\beta^{\top}\left(X-\left(X^{\top}\beta_{\text{RII}}\right)\vartheta_{\text{RIIRMI}}\right)\right]={\mathbb{E}}_{X\sim\widetilde{\mathcal{P}}_{X}}\left[\beta^{\top}\left(X-\left(X^{\top}\beta_{\text{RII}}\right)\vartheta_{\text{RIIRMI}}\right)\right]$
$\displaystyle\text{and\
}{\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{\scriptsize$(m)$}}}\left[(Y-{\mathbb{E}}_{Y\sim\mathcal{P}^{\text{\scriptsize$(m)$}}_{Y}}[Y])\cdot(Y-X^{\top}\beta)\right]=0,$
(88)

where, with $\mathcal{P}^{\text{source}}$ denote the uniform mixture of all
source distributions,

$\displaystyle\vartheta_{\text{RIIRMI}}:=\frac{{\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{source}}}\left[X\cdot(Y-{\mathbb{E}}[Y])\right]}{{\mathbb{E}}_{Y\sim\mathcal{P}^{\text{source}}_{Y}}\left[\left(Y-{\mathbb{E}}[Y]\right)^{2}\right]}.$
(89)

  * •

CIRMI${}^{\text{\scriptsize$(m)$}}$-mean: the population conditional invariant
residual matching estimator with additional residual independence using $m$-th
source environment

$\displaystyle f_{\text{CIRMI}}^{\text{\scriptsize$(m)$}}(x)$
$\displaystyle:=x^{\top}\beta_{\text{CIRMI}}^{\text{\scriptsize$(m)$}}+\beta_{\text{CIRMI},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle\beta_{\text{CIRMI}}^{\text{\scriptsize$(m)$}},\beta_{\text{CIRMI},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle:=\operatorname*{arg\,min}_{\beta,\beta_{0}}\
{\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{\scriptsize$(m)$}}}\left(Y-X^{\top}\beta-\beta_{0}\right)^{2}$
$\displaystyle\text{s.t.\
}{\mathbb{E}}_{X\sim\mathcal{P}^{\text{\scriptsize$(m)$}}_{X}}$
$\displaystyle\left[\beta^{\top}\left(X-\left(X^{\top}\beta_{\text{CIP}}\right)\vartheta_{\text{CIRMI}}\right)\right]={\mathbb{E}}_{X\sim\widetilde{\mathcal{P}}_{X}}\left[\beta^{\top}\left(X-\left(X^{\top}\beta_{\text{CIP}}\right)\vartheta_{\text{CIRMI}}\right)\right]$
$\displaystyle\text{and\
}{\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{\scriptsize$(m)$}}}\left[(Y-{\mathbb{E}}_{Y\sim\mathcal{P}^{\text{\scriptsize$(m)$}}_{Y}}[Y])\cdot(Y-X^{\top}\beta)\right]=0,$
(90)

where, with $\mathcal{P}^{\text{source}}$ denote the uniform mixture of all
source distributions,

$\displaystyle\vartheta_{\text{CIRMI}}:=\frac{{\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{source}}}\left[X\cdot(Y-{\mathbb{E}}[Y])\right]}{{\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{source}}}\left[\left(X^{\top}\beta_{\text{CIP}}-{\mathbb{E}}[X^{\top}\beta_{\text{CIP}}]\right)\cdot\left(Y-{\mathbb{E}}[Y]\right)\right]}.$
(91)

Note that a common feature of the three estimators above is that they all have
the residual independent constraint of the form

$\displaystyle{\mathbb{E}}_{(X,Y)\sim\mathcal{P}^{\text{\scriptsize$(m)$}}}\left[(Y-{\mathbb{E}}_{Y\sim\mathcal{P}^{\text{\scriptsize$(m)$}}_{Y}}[Y])\cdot(Y-X^{\top}\beta)\right]=0.$

This constraint takes advantage of the anticausal prediction setting and
restricts the estimator to ignorant of the intervention on $Y$.

###### Theorem 9

Under data generation Assumption 2 and the additional assumption

$\displaystyle{b}\notin\text{span}\left(a^{\text{\scriptsize$(2)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X},\ldots,a^{\text{\scriptsize$(M)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X}\right),$
(92)

the population target risk of RII-mean,
RIIRMI${}^{\text{\scriptsize$(1)$}}$-mean and
CIRMI${}^{\text{\scriptsize$(1)$}}$-mean satisfies

$\displaystyle\tilde{R}\left(f_{\text{RII}}\right)=\frac{{b}^{\top}\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right)\Sigma^{1/2}\left(\mathbb{I}_{d}-G_{4}\right)\Sigma^{1/2}\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right){b}}{\left\|\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right){b}\right\|_{2}^{4}},$
(93)

where $P_{4}\in^{d\times p}$ equals to $P_{\text{\tiny CIP}}$ which is the
matrix formed by an orthonormal basis of the $p$-dimensional subspace
$\text{span}\left(a^{\text{\scriptsize$(2)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X},\ldots,a^{\text{\scriptsize$(M)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X}\right)$,
$G_{4}=\Sigma^{1/2}Q_{4}\left(Q_{4}^{\top}\Sigma
Q_{4}\right)^{-1}Q_{4}^{\top}\Sigma^{1/2}$ is a projection matrix of rank
$d-p-1$, $Q_{4}\in^{d\times(d-p-1)}$ is the matrix with columns formed by
completing the columns of $P_{4}$ and
$\frac{\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right){b}}{\left\|\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right){b}\right\|_{2}}$
to an orthonormal basis of d via Gram-Schmidt orthogonalization,

$\displaystyle\tilde{R}(f_{\text{RIIRMI}}^{\text{\scriptsize$(m)$}})=\tilde{R}(f_{\text{CIRMI}}^{\text{\scriptsize$(m)$}})=\frac{{b}^{\top}\left(\mathbb{I}_{d}-uu^{\top}\right)\Sigma^{1/2}\left(\mathbb{I}_{d}-G_{5}\right)\Sigma^{1/2}\left(\mathbb{I}_{d}-uu^{\top}\right){b}}{\left\|\left(\mathbb{I}_{d}-uu^{\top}\right){b}\right\|_{2}^{4}},$
(94)

where
$u=\frac{a^{\text{\scriptsize$(m)$}}_{X}-\widetilde{a}_{X}}{\left\|a^{\text{\scriptsize$(m)$}}_{X}-\widetilde{a}_{X}\right\|_{2}}$,
$G_{5}=\Sigma^{1/2}Q_{5}\left(Q_{5}^{\top}\Sigma
Q_{5}\right)^{-1}Q_{5}^{\top}\Sigma^{1/2}$ is a projection matrix of rank
$d-2$, $Q_{5}\in^{d\times(d-2)}$ is the matrix with columns formed by
completing $u$ and $v$ to an orthonormal basis of d via Gram-Schmidt
orthogonalization.

We present a corollary that puts additional assumptions on how the
interventions are positioned to make the results in Theorem 9 easier to
understand.

###### Corollary 10

In addition to Assumption 2 and the assumption in Equation (92), assume
$\Sigma=\frac{\sigma^{2}}{\rho}\mathbb{I}_{d}$ with $\rho>0$, then

$\displaystyle\tilde{R}\left(f_{\text{RII}}\right)=\frac{\sigma^{2}}{\rho\left\|{b}\right\|_{2}^{2}-\rho\left(P_{4}^{\top}{b}\right)^{2}}$
(95)
$\displaystyle\tilde{R}(f_{\text{RIIRMI}}^{\text{\scriptsize$(1)$}})=\tilde{R}(f_{\text{CIRMI}}^{\text{\scriptsize$(1)$}})=\frac{\sigma^{2}}{\rho\left\|{b}\right\|_{2}^{2}-\rho\left(u^{\top}{b}\right)^{2}},$
(96)

Additionally, if
$a^{\text{\scriptsize$(2)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X},\ldots,a^{\text{\scriptsize$(M)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X},\xi$
are generated independently from the standard Gaussian distribution and
$\widetilde{a}_{X}-a^{\text{\scriptsize$(1)$}}=P_{\text{\tiny CIP}}\xi$, then
with high probability, we have

$\displaystyle\text{dim}\left(\text{span}\left(a^{\text{\scriptsize$(2)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X},\ldots,a^{\text{\scriptsize$(M)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X}\right)\right)=M-1.$
$\displaystyle\frac{\sigma^{2}}{\rho\left(1-\frac{(M-1)}{3d}\right)\left\|{b}\right\|_{2}^{2}}\leq\tilde{R}\left(f_{\text{RII}}\right)\leq\frac{\sigma^{2}}{\rho\left(1-\frac{3(M-1)}{d}\right)\left\|{b}\right\|_{2}^{2}}$
(97)
$\displaystyle\tilde{R}(f_{\text{RIIRMI}}^{\text{\scriptsize$(m)$}})=\tilde{R}(f_{\text{CIRMI}}^{\text{\scriptsize$(m)$}})\leq\frac{\sigma^{2}}{\rho\left(1-\frac{c}{d}\right)\left\|{b}\right\|_{2}^{2}}$
(98)

where $c$ is a constant.

The target population risks of RII, RIIRMI, CIRMI are compared with CIRM under
the assumptions of Corollary 10. Under the additional assumption (92), the new
DA methods RIIRMI and CIRMI effectively remove the
$\widetilde{a}_{Y}-a^{\text{\scriptsize$(m)$}}_{Y}$ dependency when compared
to CIRM. However, RIIRMI and CIRMI have slightly worse target population risk
when there is no intervention on the label
$a^{\text{\scriptsize$(1)$}}_{Y}=a^{\text{\scriptsize$(2)$}}_{Y}=\ldots=\widetilde{a}_{Y}=0$.

|  Estimator |  Target population risk upper bound interventions under general position
---|---|---
Intervention on $Y$ (Corollary 10, $M$ sources) |  DIP${}^{\text{\scriptsize$(m)$}}$ | $\displaystyle\frac{\sigma^{2}}{1+\rho(1-\frac{c}{d})\left\|{b}\right\|_{2}^{2}}+\left(\widetilde{a}_{Y}-a^{\text{\scriptsize$(m)$}}_{Y}\right)^{2}$
CIP | $\displaystyle\frac{\sigma^{2}}{1+\rho(1-\frac{c(M-1)}{d})\left\|{b}\right\|_{2}^{2}}+\frac{\left(\widetilde{a}_{Y}-\bar{a}_{Y}\right)-\Delta_{Y}}{\left[1+\rho(1-\frac{c(M-1)}{d})\left\|{b}\right\|_{2}^{2}\right]^{2}}$
CIRM${}^{\text{\scriptsize$(m)$}}$ | $\displaystyle\frac{\sigma^{2}}{1+\rho(1-\frac{c}{d})\left\|{b}\right\|_{2}^{2}}+\frac{\left(\widetilde{a}_{Y}-a^{\text{\scriptsize$(m)$}}_{Y}\right)^{2}}{\left[1+\rho(1-\frac{c}{d})\left\|{b}\right\|_{2}^{2}\right]^{2}}$
RII | $\displaystyle\frac{\sigma^{2}}{\rho(1-\frac{c(M-1)}{d})\left\|{b}\right\|_{2}^{2}}$
RIIRMI${}^{\text{\scriptsize$(m)$}}$ | $\displaystyle\frac{\sigma^{2}}{\rho(1-\frac{c}{d})\left\|{b}\right\|_{2}^{2}}$
CIRMI${}^{\text{\scriptsize$(m)$}}$ | $\displaystyle\frac{\sigma^{2}}{\rho(1-\frac{c}{d})\left\|{b}\right\|_{2}^{2}}$
Table 7: Summary of target population risk for different estimators for
anticausal domain adaptation under the assumptions of Corollary 10. Here $c$
is a constant. For simplicity, we only compare the target population risk
upper bound under high probability when the interventions are generated i.i.d.
Gaussian.

###  D.1 Proof of Theorem 9

##### Risk of RII:

Using the SEM (C.1) and the residual expression (75), the RII objective (• ‣
D) becomes

$\displaystyle\min_{\beta,\beta_{0}}$
$\displaystyle\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}+\frac{1}{M}\sum_{m=1}^{M}\left(\left(1-\beta^{\top}H{b}\right)a^{\text{\scriptsize$(m)$}}_{Y}-\beta^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}-\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta$ s.t.
$\displaystyle\left(1-\beta^{\top}H{b}\right)\left(a^{\text{\scriptsize$(m)$}}_{Y}-a^{\text{\scriptsize$(1)$}}_{Y}\right)+\beta^{\top}H\left(a^{\text{\scriptsize$(m)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X}\right)=0,\
\forall m\in\left\\{2,\ldots,M\right\\}$ and
$\displaystyle\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}=0.$

The second constraint above ensures that

$\displaystyle 1-\beta^{\top}H{b}=0.$

This observation allows us to simplify the RII minimization problem above and
obtain

$\displaystyle\min_{\beta,\beta_{0}}$
$\displaystyle\frac{1}{M}\sum_{m=1}^{M}\left(0-\beta^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}-\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta$ s.t.
$\displaystyle\beta^{\top}H\left(a^{\text{\scriptsize$(m)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X}\right)=0,\
\forall m\in\left\\{2,\ldots,M\right\\}$ and
$\displaystyle\beta^{\top}H{b}=1.$ (99)

The objective (D.1) is a quadratic program with linear constraints. First, the
one-dimensional quadratic program on $\beta_{0}$ can be solved easily by
setting the derivative to zero.

$\displaystyle\beta_{0}^{*}$
$\displaystyle=\frac{1}{M}\sum_{m=1}^{M}{\beta^{*}}^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}$
$\displaystyle={\beta^{*}}^{\top}Ha^{\text{\scriptsize$(1)$}}_{X},$

where the last equality follows from the first constraint in Equation (D.1).
Second, since $H$ is invertible, we can re-parametrize the optimization on
$\beta$ as follows

$\displaystyle\min_{\gamma\in^{d}}$ $\displaystyle\gamma^{\top}\Sigma\gamma$
(100) s.t.
$\displaystyle\gamma^{\top}\left(a^{\text{\scriptsize$(m)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X}\right)=0,\
\forall m\in\left\\{2,\ldots,M\right\\}$ and $\displaystyle\gamma{b}=1.$ (101)

Let $P_{4}$ be the matrix with columns formed by an orthonormal basis of the
$p$-dimensional subspace
$\text{span}\left(a^{\text{\scriptsize$(2)$}}-a^{\text{\scriptsize$(1)$}},\ldots,a^{\text{\scriptsize$(M)$}}-a^{\text{\scriptsize$(1)$}}\right)$.
Define

$\displaystyle
v:=\frac{\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right){b}}{\left\|\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right){b}\right\|_{2}}.$

$v$ is well defined because
$\left\|\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right){b}\right\|_{2}\neq 0$ by
assumption 92. By construction, we have $P_{4}^{\top}v=0$ and also

$\displaystyle\left(\frac{v}{\left\|\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right){b}\right\|_{2}}\right)^{\top}{b}=1$
(102)

.

Let $Q_{4}\in^{d\times(d-p-1)}$ be the matrix with columns formed by
completing the columns of $P_{4}$ and $v$ to an orthonormal basis of d via
Gram-Schmidt orthogonalization. Because of Equation (102), the following map

d-p-1 $\displaystyle\rightarrow^{d}$ $\displaystyle\zeta$
$\displaystyle\mapsto\frac{v}{\left\|\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right){b}\right\|_{2}}+Q_{4}\zeta$

constitutes a bijection between d-p-1 and the set $\left\\{\gamma\in^{d}\mid
P^{\top}\gamma=0,\gamma^{\top}{b}=1\right\\}$. With the change of variable,
the constrained quadratic program in Equation (100) is equivalent to the
following unconstrained one

$\displaystyle\min_{\zeta\in^{d-p-1}}\left(\frac{v}{\left\|\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right){b}\right\|_{2}}+Q_{4}\zeta\right)^{\top}\Sigma\left(\frac{v}{\left\|\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right){b}\right\|_{2}}+Q_{4}\zeta\right)$
(103)

Solving the unconstrained quadratic program by setting gradient to zero, we
obtain the minimizer

$\displaystyle\zeta^{*}=-\frac{\left({Q_{4}}^{\top}\Sigma
Q_{4}\right)^{-1}{Q_{4}}^{\top}\Sigma
v}{\left\|\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right){b}\right\|_{2}}.$

Transforming the variables back, the RII estimator is

$\displaystyle\beta_{\text{RII}}$
$\displaystyle=\left(\mathbb{I}_{d}-\mathbf{B}\right)^{\top}\frac{\Sigma^{-1/2}\left(\mathbb{I}_{d}-G_{4}\right)\Sigma^{1/2}v}{\left\|\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right){b}\right\|_{2}}$
(104) $\displaystyle\beta_{\text{RII},0}$
$\displaystyle=-\beta_{\text{RII}}^{\top}Ha^{\text{\scriptsize$(1)$}}_{X},$

where $G_{4}=\Sigma^{1/2}Q_{4}\left(Q_{4}^{\top}\Sigma
Q_{4}\right)^{-1}Q_{4}^{\top}\Sigma^{1/2}$ is a projection matrix of rank
$d-p-1$. According to the form of the target residual (76), the target
population risk for $(\beta,\beta_{0})$ takes the following form

$\displaystyle\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}+\left(\left(1-\beta^{\top}H{b}\right)\widetilde{a}_{Y}-\beta^{\top}H\widetilde{a}_{X}-\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta.$

Plugging in the RII estimator into the equation above, because
$\beta_{\text{RII}}^{\top}H{b}=1$ and
$\widetilde{a}_{X}-a^{\text{\scriptsize$(1)$}}_{X}\in\text{span}(a^{\text{\scriptsize$(2)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X},\ldots,a^{\text{\scriptsize$(m)$}}_{X}-a^{\text{\scriptsize$(1)$}}_{X})$,
we obtain the target population risk of RII

$\displaystyle\tilde{R}(f_{\text{RII}})=\frac{v^{\top}\Sigma^{1/2}\left(\mathbb{I}_{d}-G_{4}\right)\Sigma^{1/2}v}{\left\|\left(\mathbb{I}_{d}-P_{4}P_{4}^{\top}\right){b}\right\|_{2}^{2}}.$

##### Risk of RIIRMI:

We start by deriving $\vartheta_{\text{RIIRMI}}$ defined in Equation (89). It
calculates the correlation between $X$ and $Y$. Using the SEM (C.1), we obtain

$\displaystyle\vartheta_{\text{RIIRMI}}=H{b}.$ (105)

Note that just like in CIRM, the vector $\vartheta_{\text{RIIRMI}}$ is also
co-linear with the $Y$ component in the covariate expression in Equation (81).
So the idea of RIIRMI is very similar to that of CIRM: it uses
$\widetilde{X}^{\top}\beta_{\text{RII}}$ as a proxy for the unobserved
$\widetilde{Y}$ to remove the $\widetilde{Y}$ part in the covariates so that
the DIP matching based ideas can still be applied.

With the $\vartheta_{\text{RIIRMI}}$ expression (105) and the
$\beta_{\text{RII}}$ expression (104), the LHS of the first RIIRMI constraint
(• ‣ D) becomes

$\displaystyle
X^{\text{\scriptsize$(m)$}}-\left({X^{\text{\scriptsize$(m)$}}}^{\top}\beta_{\text{RII}}\right)\vartheta_{\text{RIIRMI}}$
$\displaystyle=\left(Y^{\text{\scriptsize$(m)$}}-{X^{\text{\scriptsize$(m)$}}}^{\top}\beta_{\text{RII}}\right)H{b}+Ha^{\text{\scriptsize$(m)$}}_{X}+H\varepsilon^{\text{\scriptsize$(m)$}}_{X}$
$\displaystyle=\left(\left(1-\beta_{\text{RII}}^{\top}H{b}\right)Y^{\text{\scriptsize$(m)$}}-\beta_{\text{RII}}^{\top}Ha^{\text{\scriptsize$(m)$}}-\beta_{\text{RII}}^{\top}H\varepsilon^{\text{\scriptsize$(m)$}}\right)H{b}+Ha^{\text{\scriptsize$(m)$}}_{X}+H\varepsilon^{\text{\scriptsize$(m)$}}_{X}$
$\displaystyle\stackrel{{\scriptstyle(i)}}{{=}}\left(0-\beta_{\text{RII}}^{\top}Ha^{\text{\scriptsize$(1)$}}-\beta_{\text{RII}}^{\top}H\varepsilon^{\text{\scriptsize$(m)$}}\right)H{b}+Ha^{\text{\scriptsize$(m)$}}_{X}+H\varepsilon^{\text{\scriptsize$(m)$}}_{X},$
(106)

where the last inequality follows from the two constraints in the RII
estimator (• ‣ D). Similar expression can be obtained for the target
environments by taking into account that
$\widetilde{a}_{X}\in\text{span}\left(a^{\text{\scriptsize$(1)$}}_{X},\ldots,a^{\text{\scriptsize$(M)$}}_{X}\right)$,

$\displaystyle\widetilde{X}-\left({\widetilde{X}}^{\top}\beta_{\text{RII}}\right)\vartheta_{\text{RIIRMI}}=\left(\beta_{\text{RII}}^{\top}Ha^{\text{\scriptsize$(1)$}}+\beta_{\text{RII}}^{\top}H\widetilde{\varepsilon}\right)H{b}+H\widetilde{a}_{X}+H\widetilde{\varepsilon}_{X}.$
(107)

Multiply Equation (D.1) and (107) by $\beta$ and take expectation, we obtain a
simplified form of the first RIIRMI constraint (• ‣ 3.4)

$\displaystyle\beta^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}=\beta^{\top}H\widetilde{a}_{X}.$

Note that the first RIIRMI constraint is the same as the one in DIP or CIRM.
With the above observation, RIIRMI${}^{\text{\scriptsize$(1)$}}$ objective (•
‣ D) becomes

$\displaystyle\min_{\beta,\beta_{0}}\left(0-\beta^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}-\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta$ $\displaystyle\text{s.t.
}\beta^{\top}H\left(a^{\text{\scriptsize$(m)$}}_{X}-\widetilde{a}\right)=0$
$\displaystyle\text{and }\beta^{\top}H{b}=1.$ (108)

This objective is similar to the one (C.1) in the proof of CIRM, with the only
difference that there is one additional constraint $\beta^{\top}H{b}=1$. This
objective is also similar to the one (D.1) in the proof of RII, with the
difference that there is only one constraint of the type
$\beta^{\top}H(a^{\text{\scriptsize$(m)$}}_{X}-\widetilde{a}_{X})=0$.
Following the proof of RII to solve the quadratic program with linear
constraints, we define

$\displaystyle
v:=\frac{\left(\mathbb{I}_{d}-uu^{\top}\right){b}}{\left\|\left(\mathbb{I}_{d}-uu^{\top}\right){b}\right\|_{2}},$

where
$u=\frac{a^{\text{\scriptsize$(m)$}}_{X}-\widetilde{a}_{X}}{\left\|a^{\text{\scriptsize$(m)$}}_{X}-\widetilde{a}_{X}\right\|_{2}}$
and define $Q_{5}\in^{d\times(d-2)}$ be the matrix with columns formed by
completing $u$ and $v$ to an orthonormal basis of d via Gram-Schmidt
orthogonalization, then the RIIRMI estimator is

$\displaystyle\beta_{\text{RIIRMI}}^{\text{\scriptsize$(m)$}}$
$\displaystyle=\left(\mathbb{I}_{d}-\mathbf{B}\right)^{\top}\frac{\Sigma^{-1/2}\left(\mathbb{I}_{d}-G_{5}\right)\Sigma^{1/2}\left(\mathbb{I}_{d}-uu^{\top}\right){b}}{\left\|\left(\mathbb{I}_{d}-uu^{\top}\right){b}\right\|_{2}^{2}}$
(109) $\displaystyle\beta_{\text{RIIRMI},0}^{\text{\scriptsize$(m)$}}$
$\displaystyle=-{\beta_{\text{RIIRMI}}^{\text{\scriptsize$(m)$}}}^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}.$

where $G_{5}=\Sigma^{1/2}Q_{5}\left(Q_{5}^{\top}\Sigma
Q_{5}\right)^{-1}Q_{5}^{\top}\Sigma^{1/2}$ is a projection matrix of rank
$d-2$. According to the form of the target residual (76), the target
population risk for $(\beta,\beta_{0})$ takes the following form

$\displaystyle\left(1-\beta^{\top}H{b}\right)^{2}\sigma^{2}+\left(\left(1-\beta^{\top}H{b}\right)\widetilde{a}_{Y}-\beta^{\top}H\widetilde{a}_{X}-\beta_{0}\right)^{2}+\beta^{\top}H\Sigma
H^{\top}\beta.$

Plugging in the RIIRMI[1] estimator into the equation above, because
$\beta_{\text{RII}}^{\top}H{b}=1$ and
$\beta^{\top}Ha^{\text{\scriptsize$(m)$}}_{X}=\beta^{\top}H\widetilde{a}_{X}$,
we obtain the target population risk of RIIRMI[1]

$\displaystyle\tilde{R}(f_{\text{RIIRMI}}^{\text{\scriptsize$(m)$}})=\frac{{b}^{\top}\left(\mathbb{I}_{d}-uu^{\top}\right)\Sigma^{1/2}\left(\mathbb{I}_{d}-G_{5}\right)\Sigma^{1/2}\left(\mathbb{I}_{d}-uu^{\top}\right){b}}{\left\|\left(\mathbb{I}_{d}-uu^{\top}\right){b}\right\|_{2}^{4}}.$

##### Risk of CIRMI:

The proof for CIRMI follows easily by combining parts of proofs in CIRM and
RIIRMI. The CIRMI${}^{\text{\scriptsize$(1)$}}$ estimator has one additional
constraint compared to the CIRM${}^{\text{\scriptsize$(1)$}}$ estimator in
Equation (C.1)