Unifying Causal Representation Learning with the Invariance Principle
Dingling Yao Dario Rancati Riccardo Cadei Marco Fumero Francesco Locatello








A generalization of K-theory to operator systems
    Walter D. van Suijlekom
    September 4, 2024
================================================



















§ ABSTRACT 
  =-1Causal representation learning aims at recovering latent causal variables from high-dimensional observations to solve causal downstream tasks, such as predicting the effect of new interventions or more robust classification. 
  A plethora of methods have been developed, each tackling carefully crafted problem settings that lead to different types of identifiability. 
  The folklore is that these different settings are important, as they are often linked to different rungs of Pearl's causal hierarchy, although not all neatly fit. 
  Our main contribution is to show that many existing causal representation learning approaches methodologically align the representation to known data symmetries. 
  Identification of the variables is guided by equivalence classes across different “data pockets” that are not necessarily causal. 
  This result suggests important implications, allowing us to unify many existing approaches in a single method that can mix and match different assumptions, including non-causal ones, based on the invariances relevant to our application. 
  It also significantly benefits applicability, which we demonstrate by improving treatment effect estimation on real-world high-dimensional ecological data. Overall, this paper clarifies the role of causality assumptions in the discovery of causal variables and shifts the focus to preserving data symmetries.







§ INTRODUCTION

Causal representation learning <cit.> posits that many real-world high-dimensional perceptual data can be described through a simplified latent structure specified by a few interpretable low-dimensional causally-related variables. Discovering hidden causal structures from data has been a long-standing goal across many scientific disciplines, spanning neuroscience <cit.>, communication theory <cit.>, economics <cit.> and social science <cit.>. From the machine learning perspective, algorithms and models integrated with causal structure are often proven to be more robust at distribution shift <cit.>, providing better out-of-distribution generalization results and reliable agent planning <cit.>.
Formally, the general goal of causal representation learning approaches is formulated as to 
provably identify ground-truth latent causal variables and their causal relations (up to certain ambiguities).

Many existing approaches in causal representation learning carefully formulate their problem settings to guarantee identifiability and justify the assumptions within the framework of Pearl's causal hierarchy, such as “observational, interventional, or counterfactual CRL" <cit.>. However, some causal representation learning works may not perfectly fit within this causal language framework; for instance, the problem setting of temporal CRL works <cit.> does not always align straightforwardly with existing categories.
They often assume that an individual trajectory is “intervened” upon, but this is not an intervention in the traditional sense, as noise variables are not resampled. It is also not a counterfactual as the value of non-intervened variables can change due to default dynamics. 
Similarly, domain generalization <cit.> and certain multi-task learning approaches <cit.> are sometimes framed as informally related to causal representation learning. 
However, the precise relation to causality is not always clearly articulated. This has resulted in a variety of methods and findings, some of which rely on assumptions that might be too narrowly tailored for practical, real-world applications. 
For example, <cit.> collected a data set for estimating treatment effects from high-dimensional observations in real-world ecology experiments. Despite the clear causal focus of the benchmark, they note that, despite having access to multiple views and being able to perform some interventions, neither existing mutli-view nor intervertional causal representation learning methods are directly applicable due to mismatching assumptions.

=-1 This paper contributes a unified rephrasing of many existing nonparametric CRL works through the lens of invariance. 
We observe that many existing causal representation approaches share methodological similarities, particularly in aligning the representation with known data symmetries, while differing primarily in how the invariance principle is invoked. 
This invariance principle is usually formulated implicitly in the assumed data-generating process. 
Instead, we make this explicit and show that latent causal variable identification broadly originates from multiple data pockets with certain underlying equivalence relations. These are not necessarily causal and (with minor exceptions) have to be known apriori. 
Unifying causal representation learning approaches using the invariance principle brings several potential benefits: 
First, it helps clarify the alignment between seemingly different categories of CRL methods, contributing to a more coherent and accessible framework for understanding causal representation learning. 
This perspective may also allow for the integration of multiple invariance relations in latent variable identification, which could improve the flexibility of these methods in certain practical settings. 
Additionally, our theory underscores a gap in the necessary causal assumptions for graph learning, which is essential for generalizing to unseen interventions and helps distinguish it from the problem of identifying causal variables.
These invariances can be expressed in causal terms, such as interventions, but do not always need to be.
Last but not least, this formulation of invariance relation links causal representation learning to many existing representation learning areas outside of causality, including invariant training <cit.>, domain adaptation <cit.>, and geometric deep learning  <cit.>.

We highlight our contributions as follows:

    
  * We propose a unified rephrasing for existing nonparmaetric causal representation learning approaches leveraging the invariance principles and prove latent variable identifiability in this general setting. 
    We show that 31 existing identification results can be seen as special cases directly implied by our framework. 
    This approach also enables us to derive new results, including latent variable identifiability from one imperfect intervention per node in the non-parametric setting.
    
  * In addition to employing different methods, many works in the CRL literature use varying definitions of “identifiability." We formalize these definitions at different levels of granularity, highlight their connections, and demonstrate how various definitions can be addressed within our framework by leveraging different invariance principles.
    
  * Upon the identifiability of the latent variables, we discuss the necessary causal assumptions for graph identification and the possibility of partial graph identification using the language of causal consistency. 
    With this, we draw a distinction between the causal assumptions necessary for graph discovery and those that may not be required for variable discovery.
    
  * Our framework is broadly applicable across a range of settings. 
    We observe improved results on real-world experimental ecology data using the causal inference benchmark from high-dimensional observations provided by <cit.>. 
    Additionally, we present a synthetic ablation to demonstrate that existing methods, which assume access to interventions, actually only require a form of distributional invariance to identify variables. This invariance does not necessarily need to correspond to a valid causal intervention. 





§ PROBLEM SETTING

This section formalizes our problem setting and states our main assumptions. We first summarize standard definitions and assumptions of causal representation learning in <ref>. Then, we describe the data generating process using the language of invariance properties and equivalence classes subsec:dgp.

Notation. [N] is used as a shorthand for {1, …, N}. We use bold lower-case  for random vectors and normal lower-case z for their realizations. A vector  can be indexed either by a single index i ∈ [()] via _i or a index subset A ⊆ [()] with _A := {_i: i ∈ A}. P_ denotes the probability distribution of the random vector  and p_(z) denotes the associated probability density function. By default, a "measurable" function is measurable w.r.t. the Borel sigma algebras and defined w.r.t. the Lebesgue measure. A more comprehensive summary of notations and terminologies is provided in <ref>



 §.§ Preliminaries


In this subsection, we revisit the common definitions and assumptions in identifiability works from causal representation learning. We begin with the definition of a latent structural causal model:


    Let = {_1, …, _N} denote a set of causal “endogenous" variables with each _i taking values in , and let = {_1, …, _N} denotes a set of mutually independent “exogenous" random variables. The latent SCM consists of a set of structural equations
    
    {_i := m_i(_pa(i)), _i}_i=1^N,

    where _pa(i) are the causal parents of _i and m_i are the deterministic functions that are termed “causal mechanisms". We indicate with P_ the joint distribution of the exogenous random variables, which due to the independence hypothesis is the product of the probability measures of the individual variables. The associated causal diagram  is a directed graph with vertices  and edges _i →_j iff. _i ∈_pa(j); we assume the graph  to be acyclic.

The latent SCM induces a unique distribution P_ over the endogenous variables  as a pushforward of P_ via <ref>. Its density p_ follows the causal Markov factorization:

    p_(z) = ∏_i=1^N p_i(z_i  | zi)




Instead of directly observing the endogenous and exogenous variables  and , we only have access to some “entangled" measurements  of  generated through a nonlinear mixing function:




    A deterministic smooth function f: ^N →^D mapping the latent vector ∈^N to its observable ∈^D, where D ≥ N denotes the dimensionality of the observational space.


[Diffeomorphism]

    The mixing function f is diffeomorphic onto its image, i.e. f is C^∞, f is injective and f^-1|_ℐ(f): ℐ(f) →^D is also C^∞.


Remark: Settings with noisy observations (= f() + ϵ,  ⊥ϵ) can be easily reduced to our denoised version by applying a standard deconvolution argument as a pre-processing step, as indicated by <cit.>.





 §.§ Data Generating Process


We now define our data generating process using the previously introduced mathematical framework  subsec:premilinaries. Unlike prior works in causal representation learning, which typically categorize their settings using established causal language (such as 'counterfactual,' 'interventional,' or 'observational'), our approach introduces a more general invariance principle that aims to unify diverse problem settings.
In the following, we introduce the following concepts as mathematical tools to describe our data generating process.



    Let A ⊆ [N] be an index subset of the Euclidean space ^N and let ∼_ι be an equivalence relationship on ^|A|, with A of known dimension. Let := ^|A|∼_ι be the quotient of ^|A| under this equivalence relationship;  is a topological space equipped with the quotient topology. Let ι: ^|A|→ be the projection onto the quotient induced by the equivalence relationship ∼_ι. 
     We call this projection ι the invariant property of this equivalence relation. We say that two vectors , ∈^|A| are invariant under ι if and only if they belong to the same ∼_ι equivalence class, i.e.:
    
    ι() = ι() ⇔∼_ι.

   Extending this definition to the whole latent space ^N, a pair of latents , ∈^N are non-trivially invariant on a subset A ⊆ [N] under the property ι only if

   
    
  * the invariance property ι holds on the index subset A ⊆ [N] in the sense that ι(_A) = ι(_A);
    
    
  * for any smooth function h_1, h_2: ^N →^|A|, the invariance property between , breaks under the h_1, h_2 transformations if  h_1 or h_2 directly depends on some other component _q with q ∈ [N]∖ A. Taking h_1 and  as an example, we have:
    
    ∃ q ∈[N]∖ A, ^* ∈^N,    s.t. ∂h_1∂_q(^*)  exists and is non zero  ⇒  ι(h_1()) ≠ι(h_2())

    which means: given that the partial derivative of h_1 w.r.t. some latent variable _q ∈_[N]∖ A  is non-zero at some point ^* ∈^N, h_1(), h_2() violates invariance principle in the sense that ι(h_1()) ≠ι(h_2()). That is, the invariance principle is non-trivial in the sense of not being always satisfied. 
    



    =-1 Intuition: The invariance property ι maps the invariant latent subset _A to the space  representing the identified factor of variations. For example, in the multi-view literature <cit.>, it is the identity map because the pre-and post action views are sharing the exact value of the invariant latents; for the interventional and temporal CRL <cit.>, this invariant property holds on a distributional level, and  the property manifold  can play the role of parameter space for the parameteric latent distributions or the general distribution space for the nonparametric case; for the multi-task line of work <cit.>, ι can be interpreted as the ground truth relation between the task-related latent variables and the task label, mapping the latents to the space of labels .



Remark:  <Ref> (ii) is essential for latent variable identification on the invariant partition A, which is further justified in <ref> by showing a non-identifiable example violating (ii). Intuitively, <ref> (ii) present sufficient discrepancy between the invariant and variant part in the ground truth generating process, paralleling various key assumptions for identifiability in CRL that were termed differently but conceptually similar, such as sufficient variability <cit.>, interventional regularity <cit.> and interventional discrepancy <cit.>. On a high level, these assumptions guarantee that the intervened mechanism sufficiently differs from the default causal mechanism to effectively distinguish the intervened and non-intervened latent variables, which serves the same purpose as  <ref> (ii). We elaborate this link further in <ref>.


We denote by _ :={^1, …, ^K} the set of latent random vectors with ^k ∈^N and write its joint distribution as P__. 
The joint distribution P__ has a probability density p__(z^1, …, z^K). Each individual random vector ^k ∈_ follows the marginal density p_^k with the non-degenerate support ^k ⊆^N, whose interion is a non-empty open set of ^N.





    Consider a set of random vectors _ :={^1, …, ^K} with ^k ∈^N, the corresponding set of observables _ := {^1, …, ^K} is generated by:
    
    _ = F(_),

    where the map F defines a push-forward measure F_#(P__) on the image of F as:
    
    F_#(P__)(x_1, …, x_K) = P__(f_1^-1(x_1), …, f_K^-1(x_K))

    with the support := Im(F) ⊆^K × D. Note that F satisfies <ref> as each f_k is a diffeomorphism onto its image according to <ref>.



    Intuition. <ref> formulates the generating process of the set of observables as a joint pushforward of a set of latent random vectors, providing a formal definition of the non-iid. data pockets employed in causal representation learning algorithms. It conveniently explains various underlying data symmetries given inherently by individual problem settings. For example, in the multiview scenario <cit.>, we can observe the joint data distribution P_𝒮_ because the data are “paired" (non-independent). In the interventional CRL that relies on multi-environment data, the joint data distribution can be factorized as a product of individual non-identical marginals {P_^k}_k ∈ [K], originating from partially different latent distributions P_^k that are modified by, e.g., interventions. In the supervised setting, such as multi-task CRL, we have an extended data pocket augmented by the task labels that is formally defined as S_ := {_1, …, _K} with _k := (, ^k). Note that the observable  is shared across all tasks k ∈ [K] whereas the tasks labels ^k are specific to individual tasks, thus introducing different joint data-label distributions P__k.


 In the following, we denote by ℑ := {ι_i: ^|A_i|→_i} a finite set of invariance properties with their respective invariant subsets A_i ⊆ [N] and their equivalence relationships ∼_ι_i, each inducing as a projection onto its quotient and invariant property ι_i defn:invariance_projector.
For a set of observables _ :={^1, …, ^K}∈ generated from the data generating process described in <ref>, we assume:
 
     
     For each ι_i ∈ℑ, there exists a unique known index subset V_i ⊆ [K] with at least two elements (i.e., |V_i| ≥ 2) s.t. _V_i = F([]_∼_ι_i) forms the set of observables generated from an equivalence class []_∼_ι_i := {∈^N: _A_i∼_ι_i_A_i}, as given by <ref>.  In particular, if ℑ = {ι} consists of a single invariance property ι: ^|A|→, we have _ = F([]_∼_ι).
 

=-1 Remark:  While  does not need to be fully described, which observables should belong to the same equivalence class is known (denoted as V_i ⊆ [K] for the invariance property ι_i ∈). This is a standard assumption and is equivalent to knowing e.g., two views are generated from partially overlapped latents <cit.>.


Problem setting.
    Given a set of observables _∈ satisfying <ref>, we show that we can simultaneously identify multiple invariant latent blocks A_i under a set of weak assumptions. In the best case, if each individual latent component is represented as a single invariant block through individual invariance property ι_i ∈, we can learn a fully disentangled representation and further identify the latent causal graph by additional technical assumptions.







§ IDENTIFIABILITY THEORY VIA THE INVARIANCE PRINCIPLE


=-1 This section contains our main identifiability results using the invariance principle, i.e., to align the learned representation with known data symmetries.
First, we present a general proof for latent variable identification that brings together many identifiability results from existing CRL works, including multiview, interventional, temporal, and multitask CRL. 
We compare different granularity of latent variable identification and show their transitions through certain assumptions on the causal model or mixing function subsec:latent_variable_identification. 
Afterward, we discuss the identification level of a causal graph depending on the granularity of latent variable identification under certain structural assumptions subsec:causal_graph_identification.
Detailed proofs are deferred to <ref>.




 §.§ Identifying latent variables


High-level overview.
=-1 Our general theory of latent variable identifiability, based on the invariance principle, consists of two key components: 
(1) ensuring the encoder's sufficiency, thereby obtaining an adequate representation of the original input for the desired task;
(2) guaranteeing the learned representation to preserve known data symmetries as invariance properties. 
The sufficiency is often enforced by minimizing the reconstruction loss  <cit.> in auto-encoder based architecture, 
maximizing the log likelihood in normalizing flows or maximizing entropy <cit.> in contrastive-learning based approaches.
The invariance property in the learned representations is often enforced by minimizing some equivalence relation-induced pseudometric between a pair of encodings <cit.> or by some iterative algorithm that provably ensures the invariance property on the output <cit.>. 
As a result, all invariant blocks A_i, i ∈ [n_] can be identified up to a mixing within the blocks while being disentangled from the rest. 
This type of identifiability is defined as block-identifiability <cit.> which we restate as follows:

[Block-identifiability <cit.>]definitionblockID
    
     A subset of latent variable _A := {_j}_j ∈ A with A ⊆ [N] is block-identified by an encoder g: ^D →^N on the invariant subset A if the learned representation _Â := [g()]_Â with Â⊆ [N], |A| = |Â| contains all and only information about the ground truth _A, i.e. _Â = h(_A) for some diffeomorphism h: ^|A|→^|A|.



    Intuition: Note that the inferred representation _Â can be a set of entangled latent variables rather than a single one. Block-identifiability can be considered as a coarse-grained definition of disentanglement <cit.>, which seeks to disentangle individual latent factors. In other words, disentanglement can be considered as a special case of block-identifiability with each latent constituting a single invariant block. Notably, in <cit.> disentangled factors were identified in blocks, with fine-grained identifiability achieved by intersecting different blocks.





    =-1 The encoders G:={g_k: ^k →^k}_k ∈ [K] consist of smooth functions mapping from the respective observational support ^k to the corresponding latent support ^k, as elaborated in <ref>.



    Intuition: For the purpose of generality, we design the encoder g_k to be specific to individual observable ^k ∈_. 
    However, multiple g_k can share parameters if they work on the same modality. 
    Ideally, we would like the encoders to preserve as much invariance (from ) as possible. 
    Thus, a clear separation between different encoding blocks is needed. 
    To this end, we introduce selectors.




    A selection ⊘ operates between two vectors a ∈{0, 1}^d  , b ∈^d s.t.
    
    a ⊘ b := [b_j: a_j = 1, j ∈ [d]]




 The invariant block selectors Φ:= {ϕ^(i, k)}_i ∈ [n_], k ∈ V_i with ϕ^(i, k)∈{0,1}^N perform selection defn:selection on the encoded information: 
for any invariance property ι_i ∈, any observable ^k, k ∈ V_i we have the selected representation:

    ϕ^(i, k)⊘^k = ϕ^(i, k)⊘ g_k(^k) = [ [g_k(^k)]_j: ϕ^(i, k)_j = 1, j ∈ [N] ],

with ϕ^(i, k)_0 = ϕ^(i, k^')_0 = |A_i| for all ι_i ∈, k, k^'∈ V_i.



    Intuition: Selectors are used to select the relevant encoding dimensions for each individual invariance property ι_i ∈.  
    Each selector ϕ^(i, k) gives rise to a index subset Â_i^k := {j: ϕ^(i, k)_j = 1}⊆ [N] that is specific to the invariance property ι_i and the observable ^k. 
    The assumption of known invariance size |A_i| can be lifted in certain scenarios by, e.g., enforcing sharing between the learned latent variables, as shown by <cit.>, or leveraging sparsity constraints <cit.>.



[Invariance constraint]

    For any ι_i ∈, i ∈ [n_], the selected representations ϕ^(i, k)⊘ g_k(^k), k ∈ [K] must be ι_i-invariant across the observables from the subset V_i ⊂ [K]:
    
    ι_i(ϕ^(i, k)⊘ g_k(^k))
            = 
            ι_i(ϕ^(i, k^')⊘ g_k^'(^k^'))
              ∀ i ∈ [n_]   ∀ k, k^'∈ V_i



[Sufficiency constraint]

    For any encoder g_k, k ∈ [K], the learned representation preserves at least as much information as of any of the invariant partitions _A_i that we aim to identify in the sense that
    I(_A_i, g_k(^k)) = H(_A_i)   ∀ i ∈ [n_], k ∈ V_i. 



    =-1Remark: The regularizer enforcing this sufficiency constraint can be tailored to suit the specific task of interest.
    For example, for self-supervised training, it can be implemented as the mutual information between the input data and the encodings, i.e., I(, g()) = H(), to preserve the entropy from the observations; 
    for classification, it becomes the mutual information between the task labels and the learned representation I(, g()). 
    Sometimes, sufficiency does not have to be enforced on the whole representation. 
    For example, in the multiview line of work <cit.>, when considering a single invariant block A, 
    enforcing sufficiency on the shared partition (implemented as entropy on the learned encoding H(g()_1:|A|)) is enough to block-identify these shared latent variables _A.




[Identifiability of multiple invariant blocks]theoremidmulti

Consider a set of observables _ = {^1, ^2, …, ^K} with ^k ∈^k generated from <ref> satisfying <ref>. 
Let G, Φ be the set of smooth encoders defn:encoders and selectors defn:block_selectors that satisfy <ref>, then the invariant component _A_i^k is block-identified defn:blockID by ϕ^(i, k)⊘ g_k for all ι_i ∈, k ∈ [K].




    Discussion: Intuitively, <ref> enforces all invariance properties ι_i ∈ jointly and thus learns a representation that block-identifies all invariance blocks simultaneously. 
    It allows mixing multiple invariance principles, thus better adapting to complex real-world scenarios in which various invariance relations typically occur. 
    In practice, this constrained optimization problem can be solved in many different flavors, e.g., <cit.> employ a two-stage learning process first to solve the sufficiency constraint, then the invariance constraint, <cit.> instead formulate it as a bi-level constrained optimization problem. Some works <cit.> propose a loss that directly solves the constrained optimization problem, while the others <cit.> develop step-by-step algorithms as solutions.


=-1 What about the variant latents? Intuitively, the variant latents are generally not identifiable, as the invariance constraint cons:invariance is applied only to the selected invariant encodings, leaving the variant part without any weak supervision <cit.>. This non-identifiability result is formalized as follows:

[General non-identifiability of variant latent variables]propositionvarNonID

Consider the setup in <ref>, let A := ⋃_i ∈ [n_] A_i denote the union of block-identified latent indexes and A^c := [N] ∖ A the complementary set where no ι-invariance ι∈ applies, then the variant latents  _A^c cannot be identified.


Although variant latent variables are generally non-identifiable, they can be identified under certain conditions. The following demonstrates that variant latent variables can be identified under invertible encoders when the variant and invariant partitions are mutually independent.

[Identifiability of variant latent under independence]propositionvarIDindependence

Consider an optimal encoder g∈ G^* and optimal selector ϕ∈Φ^* from <ref> that jointly identify an invariant block _A (we omit subscriptions k, i for simplicity), then _A^c (A^c := [N] ∖ A) can be identified by the complementary encoding partition (1 - ϕ) ⊘ g only if 

    
  * g is invertible in the sense that I(, g()) = H();
    
  * _A^c is independent on _A.






    Discussion: The generalization of new interventions has been a long-standing goal in causal representation learning. The generalization can be categorized into two layers: (1) generalize to unseen interventional values and (2) generalize to non-intervened nodes. The former includes the out-of-distributional value of the intervened node in the training set or a combination of multiple singly intervened nodes during training, which has been successfully demonstrated in various existing works <cit.>. However, we argue that the second layer of generalization, namely generalizing to unseen nodes, is fundamentally impossible, as shown by <ref>; only under certain conditions such as independence and sufficient latent representation for reconstruction, non-intervened nodes in the training phase can be identified during inference prop:id-variant-indep. This result aligns with the identifiability algebra given by <cit.> and is evidenced by numerous previous works, including disentanglement <cit.> and temporal causal representation learning without instantaneous effect <cit.>.





 §.§ On the granularity of identification


Different levels of identification can be achieved depending on the degree of underlying data symmetry. Below, we present three standard identifiability definitions from the CRL literature, each offering stronger identification results than block-identifiability defn:blockID.



    Let  be the learned representation, for a subset A⊆ [N] it satisfies that:
    
    _π(A) = D ·_A + ,

    where D ∈^|A| × |A| is  an invertible matrix, π(A) denotes the index permutation of A, then _A is block affine-identified by _π(A).




    The learned representation ∈^N satisfies that:
    
    = _π· h(),

    where _π∈^N × N is a permutation matrix, h():= (h_1(_1), … h_N(_N)) ∈^N is a is an element-wise diffeomorphism.




    The learned representation ∈^N satisfies that:
    
    = Λ·_π· + ,

    where _π∈^N × N is a permutation matrix, Λ∈^N × N is a diagonal matrix with nonzero diagonal entries.



    =-1Remark: Block affine-identifiability defn:block_affineID is defined by <cit.>, stating that the learned representation  is related to the ground truth latents  through some sparse matrix with zero blocks.
    <Ref> indicates element-wise identification of latent variables up to individual diffeomorphisms. 
    Element-identifiability for the latent variable identification together with the graph identifiability defn:graphID is defined as ∼_CRL-identifiability  <cit.>, perfect identifiability <cit.>.
    Affine identifiability defn:affineID describes when the ground truth latent variables are identified up to permutation, shift, and linear scaling. In many CRL works, affine identifiability defn:affineID is also termed as follows: perfect identifiability under linear transformation <cit.>, CD-equivalence <cit.>, disentanglement <cit.>. 




[Granularity of identification]propositiongranularityID
    
    Affine-identifiability defn:affineID implies element-identifiability defn:crlID and block affine-identifiability defn:block_affineID while element-identifiability and block affine-identifiability implies block-identifiability defn:blockID.
 


[Transition between identification levels]propositiontransID

The transition between different levels of latent variable identification fig:id_granularity can be summarized as follows:

    
  * Element-level identifiability defn:crlID,defn:affineID can be obtained from block-wise identifiability defn:block_affineID,defn:blockID when each individual latent constitutes an invariant block;
    
  * Identifiability up to an affine transformation defn:block_affineID,defn:affineID can be obtained from general identifiability on arbitrary diffeomorphism defn:crlID,defn:blockID by additionally assuming that both the ground truth mixing function and decoder are finite degree polynomials of the same degree.




=-1 Discussion. 
We note that the granularity of identifiability results is primarily determined by the strength of invariance and parametric assumptions (such as those on mixing functions or causal models) rather than by the specific algorithmic choice. 
For example, for settings that can achieve -identifiability <cit.>, affine-identifiability results can be obtained by additionally assuming finite degree polynomial mixing function (proof see <ref>). 
Similarly, one reaches -identifiability from block-identifiability by enforcing invariance properties on each latent component <cit.> instead of having only one fat-hand invariant block <cit.>.
 <ref> provides an overview of recent identifiability results along with their corresponding invariance and parametric assumptions, illustrating the direct relationship between these assumptions and the level of identifiability they achieve.






 §.§ Identifying the causal graph


In addition to latent variable identification, another goal of causal representation learning is to infer the underlying latent dependency, namely the causal graph structure. Hence, we restate the standard definition of graph identifiability in causal representation learning.


    The estimated graph  is isomorphic to the ground truth  through a bijection h : V() → V() in the sense that two vertices _i, _j ∈ V() are adjacent in  if and only if h(_i), h(_j) ∈ V() are adjacent in .



We remark that the “faithfulness" assumption <cit.> is a standard assumption in the CRL literature, commonly required for graph discovery. We restate it as follows:
[Faithfulness (or Stability)]
    P_ is a faithful distribution induced by the latent SCM defn:latent_scm in the sense that P_ contains no extraneous conditional independence; in other words, the only conditional independence relations satisfied by P_ are those given by {_i ⊥i | i} where i denotes the non-descends of _i.



As indicated by <ref>, the preliminary condition of identifying the causal graph is to have an element-wise correspondence between the vertices in the ground truth graph  (i.e., the ground truth latents) and the vertices of the estimated graph. 
Therefore, the following assumes that the learned encoders G defn:encoders achieve element-identifiability defn:crlID, that is, for each _i ∈, we have a differmorphism h_i: → such that _i = h_i(_i). 
However, to identify the graph structure, additional assumptions are needed: either on the source of invariance or on the parametric form of the latent causal model.

Graph identification via interventions.
Under the element-identifiability defn:crlID of the latent variables , the causal graph structure  can be identified up to its isomorphism defn:graphID, given multi-domain data from paired perfect interventions per-node  <cit.>. 
Using data generated from imperfect interventions is generally insufficient to identify the direct edges in the causal graph, it can only identify the ancestral relations, i.e., up to the transitive closure of  <cit.>. 
Unfortunately, even imposing the linear assumption on the latent SCM does not provide a solution <cit.>. 
Nevertheless, by adding sparsity assumptions on the causal graph  and polynomial assumption on the mixing function f,  <cit.> has shown isomorphic graph identifiability defn:graphID under imperfect intervention per node. 
In general, access to the interventions is necessary for graph identification if one is not comfortable making other parametric assumptions about the graph structure. Conveniently, in this setting, the graph identifiability is linked with that of the variables since the latter leverages the invariance induced by the intervention.

Graph identification via parametric assumptions.
It is well known in causal discovery that the additive noise model <cit.> is identifiable under certain mild assumptions <cit.>. In the following, we assume an additive exogenous noise in the latent SCM defn:latent_scm:
[Additive noise]

    The endogenous variable _i ∈ in the previously defined latent SCM defn:latent_scm relates to the corresponding exogenous noise variable _i ∈ through additivity. Namely, the causal mechanism eq:scm can be rewritten as:
    
    {_i = m_i(i) + _i}.


As a generalization of the additive noise model, the post-nonlinear acyclic causal model <cit.> allows extra nonlinearity on the top of the additive causal mechanism, providing additional flexibility on the latent model assumption:



    The following causal mechanism describes a post-nonlinear acyclic causal model:
    
    _i = h_i(m_i(i) + _i),

    where h_i: → is a diffeomorphism and m_i is a non-constant function.




Assume the latent variable _i is element-wise identified through a bijective mapping h_i: → for all i ∈ [N], define the estimated causal parents _pa(i) := {h_j(_j): _j ∈i}, then the latent SCM defn:latent_scm is translated to a post-nonlinear acyclic causal model defn:post-nonlinear-model because

    _i 
           = h_i(_i) 
        = h_i(m_i(i) + _i)
       = h_i(m_i({h^-1_j(_j): _j ∈i}) + _i )
       = h_i(m̃_i(i) + _i ),

where

    m̃_i(i) := m_i({h^-1_j(_j): _j ∈i}).

Thus, the underlying causal graph  can be identified up to an isomorphism defn:graphID following the approach given by <cit.>



  
What happens if variables are identified in blocks? Consider the case where the latent variables cannot be identified up to element-wise diffeomorphism; instead, one can only obtain a coarse-grained version of the variables (e.g., as a mixing of a block of variables defn:blockID). Nevertheless, certain causal links between these coarse-grained block variables are of interest. These block variables and their causal relations in between form a “macro" level of the original latent SCM, which is shown to be causally consistent under mild structural assumptions <cit.>. In particular, the macro-level model can be obtained from the micro-level model through an exact transformation <cit.> and thus produces the same causal effect as the original micro-level model under the same type of interventions, providing useful knowledge for downstream causal analysis. More formal connections are beyond the scope of this paper. Still, we see this concept of coarse-grained identification on both causal variables and graphs as an interesting avenue for future research. 




§ REVISITING RELATED WORKS AS SPECIAL CASES OF OUR THEORY


=-1 This section reviews related causal representation learning works and frames them as specific instances of our theory sec:identifiability. These works were originally categorized into various causal representation learning types (multiview, multi-domain, multi-task, and temporal CRL) based on the level of invariance in the data-generating process, leading to varying degrees of identifiability results subsec:granularity_id.
While the practical implementation of individual works may vary, the 
methodological principle of aligning representation with known data symmetries remain consistent, as shown in <ref>. 
In this section, we revisit the data-generating process of each category and explain how they can be viewed as specific cases of the proposed invariance framework subsec:dgp. 
We then present individual identification algorithms from the CRL literature as particular applications of our theorems, based on the implementation choices needed to satisfy the invariance and sufficiency constraints cons:invariance,cons:sufficiency. A more detailed overview of the individual works is provided in <ref>.



 §.§ Multiview Causal Representation Learning

High-level overview.
The multiview setting in causal representation learning <cit.> considers multiple views that are concurrently generated by an overlapping subset of latent variables, and thus having non-independently distributed data. Multiview scenarios are often found in a partially observable setup. 
For example, multiple devices on a robot measure different modalities, jointly monitoring the environment through these real-time measurements. While each device measures a distinct subset of latent variables, these subsets probably still overlap as they are measuring the same system at the same time.
In addition to partial observability, another way to obtain multiple views is to perform an “intervention/perturbation" <cit.> and collect both pre-action and post-action views on the same sample. This setting is often improperly termed “counterfactual"[Traditionally, counterfactual in causality refers to non-observable outcomes that are “counter to the fact” <cit.>. In the works we refer to here, they rather represent pre- and post- an action that affect some latent variables but not all. This can be mathematically expressed as a counterfactual in a SCM, but is conceptually different as both pre- and post- action outcomes are realized <cit.>. The “counterfactual” terminology silently implies that this is a strong assumption, but nuance is needed and it can in fact be much weaker than an intervention.] in the CRL literature, and this type of data is termed “paired data". From another perspective, the paired setting can be cast in the partial observability scenario by considering the same latent before and after an action (mathematically modelled as an intervention) as two separate latent nodes in the causal graph, as shown by <cit.>. Thus, both pre-action and post-action views are partial because neither of them can observe pre-action and post-action latents simultaneously. These works assume that the latents that are not affected by the action remain constant, an assumption that is relaxed in temporal CRL works. See <ref> for more discussion in this regard.

Data generating process.
In the following, we introduce the data-generating process of a multi-view setting in the flavor of the invariance principle as introduced in <ref>.
We consider a set of views {^k}_k∈ [K] with each view ^k ∈^k generated from some latents ^k ∈^k. Let S_k ⊆ [N] be the index set of generating factors for the view ^k, we define ^k_j = 0 for all j ∈ [N]∖ S_k to represent the uninvolved partition of latents.
Each entangled view ^k is generated by a view-specific mixing function f_k: ^k →^k:

    ^k = f_k(^k)   ∀ k ∈ [K]

Define the joint overlapping index set A := ⋂_k ∈ [K] S_k, and assume A ⊆ [N] is a non-empty interior of [N].
Then the value of the sharing partition _A remain invariant for all observables {^k}_k ∈ [K] on a sample level. 
By considering the joint intersection A, we have one single invariance property ι: ^|A|→^|A| in the invariance set ; and this invariance property ι emerges as the identity map id on ^|A| in the sense that id(^k_A) = id(^k^'_A) and thus ^k_A ^k^'_A for all k, k^'∈ [K]. Note that <ref> (ii) is satisfied because any transformation h_k that involves other components _q with q ∉ A violates the equity introduced by the identity map.
For a subset of observations V_i ⊆ [K] with at least two elements |V_i| > 1, we define the latent intersection as A_i := ⋂_k ∈ V_i⊆ [N], then for each non-empty intersection A_i, there is a corresponding invariance property ι_i: ^|A_i|→^|A_i| which is the identity map specified on the subspace ^|A_i|. By considering all these subsets := {V_i ⊆ [K]: |V_i| > 1, |A_i| > 0}, we obtain a set of invariance properties := {ι_i: ^|A_i|→^|A_i|} that satisfy <ref>.

Identification algorithms.
Many multiview works <cit.> employ the L_2 loss as a regularizer to enforce sample-level invariance on the invariant partition, cooperated with some sufficiency regularizer to preserve sufficient information about the observables cons:sufficiency. 
Aligned with our theory thm:ID_from_multi_sets,thm:ID_from_multi_sets, these works have shown block-identifiability on the invariant partition of the latents across different views.
Following the same principle, there are certain variations in the implementations to enforce the invariance principle, e.g. <cit.> directly average the learned representations from paired data g(^1), g(^2) on the shared coordinates before forwarding them to the decoder;  <cit.> enforces L_2 alignment up to a learnable sparse perturbation δ. As each latent component constitutes a single invariant block in the training data, these two works element-identifies defn:crlID the latent variables, as explained by <ref>.





 §.§ Multi-environment Causal Representation Learning
 

High-level overview.
Multi-environment / interventional CRL considers data generated from multiple environments with respective environment-specific data distributions; hence, the considered data is independently but non-identically distributed. 
In the scope of causal representation learning, 
multi-environment data
is often instantiated through interventions on the latent structured causal model <cit.>. Recently, several papers attempt to provide a more general identifiability statement where multi-environment data is not necessarily originated from interventions; instead, they can be individual data distributions that preserve certain symmetries, such as marginal invariance or support invariance <cit.> or sufficient statistical variability <cit.>.

Data generating process
The following presents the data generating process described in most interventional causal representation learning works.
Formally, we consider a set of non-identically distributed data {P_^k}_k ∈ [K] that are collected from multiple environments (indexed by k ∈ [K]) with a shared mixing function f: ^k = f(^k) defn:mixing_fn satisfying <ref> and a shared latent SCM defn:latent_scm.
Let k=0 denote the non-intervened environment and _k ⊆ [N] denotes the set of intervened nodes in k-th environment, the latent distribution P_^k is associated with the density 

    p_^k(z^k) = ∏_j ∈_kp̃(z_j^k  | zj^k) ∏_j ∈ [N] ∖_k p(z_j^k  | zj^k),

where we denote by p the original density and by p̃ the intervened density. 


Interventions naturally introduce various distributional invariance that can be utilized for latent variable identification: Under the intervention _k in the k-th environment, we observe that both (1) the marginal distribution of _A with A:=[N] ∖TC(_k), with TC denoting the transitive closure and (2) the score [S(^k)]_A^' := ∇_^k_A^'log p_^k on the subset of latent components A^' := [N]∖pa(_k) with pa(_k):={j: j ∈_k ∪pa(_k)} remain invariant across the observational and the k-th interventional environment. Formally, under intervention _k, we have

    
  * Marginal invariance: 
    
    p_^0(z_A^0) = p_^k(z_A^k)      A:=[N] ∖TC(_k);

    
  * Score invariance:
    
    [S(^0)]_A^' = [S(^k)]_A^'     A^' := [N]∖pa(_k).



According to our theory <ref>, we can block-identify both _A, _A^' using these invariance principles eq:marginal_invariance,eq:score_invariance. 
Since most interventional CRL works assume at least one intervention per node <cit.>, more fine-grained variable identification results, such as element-wise identification defn:crlID or affine-identification defn:affineID, can be achieved by combining multiple invariances from these per-node interventions, as we elaborate below.

Identifiability with one intervention per node.
By applying <ref>, we demonstrate that latent causal variables  can be identified up to element-wise diffeomorphism defn:crlID under single node imperfect intervention per node, given the following assumption.
[Topologically ordered interventional targets]

    Specifying <ref> in the interventional setting, we assume there are exactly N environments
    {k_1, …, k_N}⊆ [K] where each node j ∈ [N] undergoes one imperfect intervention in the environment k_j ∈ [K]. The interventional targets 1≼…≼ N preserve the topological order, meaning that i ≼ j only if there is a directed path from node i to node j in the underlying causal graph .

Remark: <ref> is directly implied by <ref> as we need to know which environments fall into the same equivalence class. 
We believe that identifying the topological order is another subproblem orthogonal to identifying the latent variables, which is often termed “uncoupled/non-aligned problem" <cit.>. 
As described by <cit.>, the topological order of unknown interventional targets can be recovered from single-node imperfect intervention by iteratively identifying the interventions that target the source nodes. 
This iterative identification process may require additional assumptions on the mixing functions <cit.> and the latent structured causal model <cit.>, or on the interventions, such as perfect interventions that eliminate parental dependency <cit.>, or the need for two interventions per node <cit.>.


    
    Given N environments {k_1, …, k_N}⊆ [K] satisfying <ref>, the ground truth latent variables  can be identified up to element-wise diffeomorphism defn:crlID by combining both marginal and score invariances eq:marginal_invariance,eq:score_invariance under our framework thm:ID_from_multi_sets.


We consider a coarse-grained version of the underlying causal graph consisting of a block-node _[N-1] and the leaf node _N with _[N-1] causing _N (i.e., _[N-1]→_N).
We first select a pair of environments V = {0, k_N} consisting of the observational environment and the environment where the leaf node _N is intervened upon. According to <ref>, the marginal invariance holds for the partition A = [N-1], implying identification on _[N-1] from <ref>. 
At the same time, when considering the set of environments V^' = {0, k_1, …, k_N-1},
the leaf node N is the only component that satisfy score invariance across all environments V^', because N is not the parent of any intervened node (also see <cit.>).
So here we have another invariant partition A^' = {N}, implying identification on _N thm:ID_from_multi_sets. 
By jointly enforcing the marginal and score invariance on A and A^' under a sufficient encoder cons:sufficiency, we identify both _[N-1] as a block and _N as a single element. 
Formally, for the parental block _[N-1], we have:

    ^k_[N-1] = g_:N-1(^k)     ∀ k ∈{0, k_1, …, k_N}

where g_:N-1(^k) := [g(^k)]_:N-1 relates to the ground truth _[N-1] through some diffeomorphism h_[N-1]: ^N-1→^N-1 defn:blockID. 
Now, we can remove the leaf node N as follows:
For each environment k ∈{0, k_1, …, k_N-1}, we compute the pushforward of P_^k using the learned encoder g_:N-1: ^k →^N-1:

    P_^k_[N-1] = g_# (P_^k)

Note that the estimated representations P_^k_[N-1] can be seen as a new observed data distribution for each environment k that is generated from the subgraph _-N without the leaf node N.
Using an iterative argument, we can identify all latent variables element-wise defn:crlID, concluding the proof.

Upon element-wise identification from single-node intervention per node, existing works often provide more fine-grained identifiability results by incorporating other parametric assumptions, either on the mixing functions <cit.> or the latent causal model <cit.> or both <cit.>. 
This is explained by <ref>, as element-wise identification can be refined to affine-identification defn:affineID given additional parametric assumptions. 
Note that under this milder setting, the full graph is not identifiable without further assumptions, see <cit.>.


Identifiability with two interventions per-node
Current literature in interventional CRL targeting 
the general nonparametric setting <cit.> typically assumed a pair of sufficiently different perfect interventions per node.
Thus, any latent variable _j, j ∈ [N], as an interventional target, is uniquely shared by a pair of interventional environment k, k^'∈ [K], forming an invariant partition A_i = {j} constituting of individual latent node j ∈ [N]. 
Note that this invariance property on the interventional target induces the following distributional property:

    [S(^k) - S(^k^')]_j ≠ 0      only if     _k = _k^' = {j}.

According to <ref>, each latent variable can thus be identified separately, giving rise to element-wise identification, as shown by <cit.>. 

Identifiability under multiple distributions.
More recently, <cit.> explains previous interventional identifiability results from a general weak distributional invariance perspective. 
In a nutshell, a set of variables _A can be block-identified if certain invariant distributional properties hold: The invariant partition _A can be block-identified defn:blockID from the rest by utilizing the marginal distributional invariance or invariance on the support, mean or variance.
<cit.> additionally assume the mixing function to be finite degree polynomial, which leads to block-affine identification defn:block_affineID, whereas we can also consider a general non-parametric setting;
they consider one single invariance set, which is a special case of <ref> with one joint ι-property.

=-1 Identification algorithm.
Instead of iteratively enforcing the invariance constraint across the majority of environments as described in <ref>, 
most single-node interventional works develop equivalent constraints between pairs of environments to optimize. 
For example, the marginal invariance eq:marginal_invariance implies the marginal of the source node is changed only if it is intervened upon, which is utilized by <cit.> to identify latent variables and the ancestral relations simultaneously. 
In practice, <cit.> propose a regularized loss that includes Maximum Mean Discrepancy(MMD) between the reconstructed "counterfactual" data distribution and the interventional distribution, enforcing the distributional discrepancy that reveals graphical structure (e.g., detecting the source node). 
Similarly, by enforcing sparsity on the score change matrix,  <cit.> restricts only score changes from the intervened node and its parents. 
In the nonparametric case, <cit.> optimize for the invariant (aligned) interventional targets through model selection, whereas <cit.> directly solve the constrained optimization problem formulated using score differences. 
Considering a more general setup, <cit.> provides various invariance-based regularizers as plug-and-play components for any losses that enforce a sufficient representation cons:sufficiency.







 §.§ Temporal Causal Representation Learning


High-level overview. Temporal CRL <cit.> focuses on retrieving latent causal structures from time series data, where the latent causal structured is typically modeled as a Dynamic Bayesian Network (DBN) <cit.>.
Existing temporal CRL literature has developed identifiability results under varying sets of assumptions. A common overarching assumption is to require the Dynamic Bayesian Network to be first-order Markovian, allowing only causal links from t-1 to t, eliminating longer dependencies <cit.>. While many works assume that there is no instantaneous effect, restricting the latent components of ^t to be mutually dependent <cit.>, some approaches have lifted this assumption and prove identifiability allowing for instantaneous links among the latent components at the same timestep (<cit.>).

=-1 Data generating process. We present the data generating process followed by most temporal causal representation works and explain the underlying latent invariance and data symmetries. Let ^t ∈^N denotes the latent vector at time t and ^t = f(^t) ∈^D the corresponding entangled observable with f: ^N →^D the shared mixing function defn:mixing_fn satisfying <ref>. The actions ^t with cardinality |^t| = N mostly only target a subset of latent variables while keeping the rest untouched, following its default dynamics  <cit.>. 
Intuitively, these actions ^t can be interpreted as a component-wise indicator for each latent variable ^t_j, j ∈ [N] stating whether _j follows the default dynamics p(^t+1_j  | ^t) or the modified dynamics induced by the action _j^t. From this perspective, the non-intervened causal variables at time t can be considered the invariant partition under our formulation, denoted by ^t_A_t with the index set A_t defined as A_t:= {j: _j = 0}. Note that this invariance can be considered as a generalization of the multiview case because the realizations z_j^t, z_j^t+1 are not exactly identical (as in the multiview case) but are related via a default transition mechanism p(^t+1_j  | ^t). To formalize this intuition, we define ^t := ^t | ^t as the conditional random vector conditioning on the action ^t at time t. For the non-intervened partition A_t ⊆ [N] that follows the default dynamics, the transition model should be invariant:

    p(^t_A_t | ^t-1) =  p(^t_A_t | ^t-1),

which gives rise to a non-trivial distributional invariance property defn:invariance_projector. Note that the invariance partition A_t could vary across different time steps, providing a set of invariance properties :={ι_t: ^|A_t|→_t}_t=1^T, indexed by time t. 
Given by <ref>, all invariant partitions ^t_A_t can be block-identified; 
furthermore, the complementary variant partition can also be identified under an invertible encoder and mutual independence within ^t prop:id-variant-indep, aligning with the identification results without instantaneous effect <cit.>. 
On the other hand, temporal causal variables with instantaneous effects are shown to be identifiable only if “instantaneous parents” (i.e., nodes affecting other nodes instantaneously) are cut by actions <cit.>, reducing to the setting without instantaneous effect where the latent components at t are mutually independent. Upon invariance, more fine-grained latent variable identification results, such as element-wise identifiability, can be obtained by incorporating additional technical assumptions, such as the sparse mechanism shift <cit.> and parametric latent causal model <cit.>.

=-1 Identification algorithm. 
From a high level, the distributional invariance eq:temp_invariance indicates full explainability and predictability of _A_t^t from its previous time step ^t-1, regardless of the action ^t. In principle, this invariance principle can be enforced by directly maximizing the information content of the proposed default transition density between the learned representation p(^t_A_t | ^t-1) <cit.>. In practice, the invariance regularization is often incorporated together with the predictability of the variant partition conditioning on actions, implemented as a KL divergence between the observational posterior q(^t | ^t) and the transitional prior p(^t | ^t-1, ^t) <cit.>, estimated using variational Bayes <cit.> or normalizing flow <cit.>. We additionally show that minimizing this KL-divergence (q(^t | ^t)  p(^t | ^t-1, ^t)) is equivalent to maximizing the conditional entropy p(^t_A_t | ^t-1) in <ref>.






 §.§ Multi-task Causal Representation Learning


=-1 High-level overview.
Multi-task causal representation learning aims to identify latent causal variables via external supervision, in this case, the label information of the same instance for various tasks. Previously, multi-task learning <cit.> has been mostly studied outside the scope of identifiability, mainly focusing on domain adaptation and out-of-distribution generalization. One of the popular ideas that was extensively used in the context of multi-task learning is to leverage interactions between different tasks to construct a generalist model that is capable of solving all classification tasks and potentially better generalizes to unseen tasks <cit.>. Recently, <cit.> systematically studied under which conditions the latent variables can be identified in the multi-task scenario and correspondingly provided identification algorithms.

=-1 Data generating process.
The multi-task causal representation learning considers a supervised setup: Given a latent SCM as defined in <ref>, we generate the observable ∈^D through some mixing function f: ^N →^D satisfying <ref>.
Given a set of task = {t_1, …, t_k}, and let ^k ∈_k denote the corresponding task label respect to the task t_k. Each task only directly depends on a subset of latent variables S_k ⊆ [N], in the sense that the label ^k can be expressed as a function that contains all and only information about the latent variable _S_k:

    ^k = r_k(_S_k),

where r: ^|S_k|→_k is some deterministic function which maps the latent subspace ^|S_k| to the task-specific label space _k, which is often assumed to be linear and implemented using a linear readout in practice <cit.>.
For each task t_k, k ∈ [K], we observe the associated data distribution P_, ^k. Consider two different tasks t_k, t_k^' with k, k^'∈ [K], the corresponding data , ^k and , ^k^' are invariant in the intersection of task-related features _A with A = S_k ∩ S_k^'. Formally, let r_k^-1({^k}) denotes the pre-image of ^k, for which it holds

    r_k^-1({^k})_A = r_k^'^-1({^k^'})_A,

showing alignment on the shared partition of the task-related latents.
In the ideal case, each latent component j ∈ [N] is uniquely shared by a subset of tasks, all factors of variation can be fully disentangled, which aligns with the theoretical claims by <cit.>.

=-1 Identification algorithms.
We remark that the sharing mechanism in the context of multi-task learning fundamentally differs from that of multi-view setup, thus resulting in different learning algorithms.
Regarding learning, the shared partition of task-related latents is enforced to align up to the linear equivalence class (given a linear readout) instead of sample level L_2 alignment. 
Intuitively, this invariance principle can be interpreted as a soft version of the that in the multiview case.
In practice, under the constraint of perfect classification, one employs (1) a sparsity constraint on the linear readout weights to enforce the encoder to allocate the correct task-specific latents and (2) an information-sharing term to encourage reusing latents across various tasks.
Equilibrium can be obtained between these two terms only when the shared task-specific latent is element-wise identified defn:crlID.
Thus, this soft invariance principle is jointly implemented by the sparsity constraint and information sharing regularization <cit.>.




 §.§ Domain Generalization Representation Learning

=-1 High-level overview. Domain generalization aims at out-of-distribution performance.
That is, learning an optimal encoder and predictor that performs well at some unseen test domain that preserves the same data symmetries as in the training data. At a high level, domain generalization representation learning <cit.> considers a similar framework as introduced for interventional CRL, with independent but non-identically distributed data, but additionally incorporated with external supervision and focusing more on model robustness perspective. While interventional CRL aims to identify the true latent factors of variations (up to some transformation), domain generalization learning focuses directly on out-of-distribution prediction, relying on some invariance properties preserved under the distributional shifts. 

Due to the non-causal objective, new methodologies are motivated and tested on real-world benchmarks (e.g., VLCS <cit.>, PACS <cit.>, Office-Home <cit.>, Terra Incognita <cit.>, DomainNet <cit.>)  
and could inspire future real-world applicability of causal representation learning approaches.




=-1 Data generating process. The problem of domain generalizations is an extension of supervised learning where training data from multiple environments are available
<cit.>. 
An environment is a dataset of i.i.d. observations from a joint distribution P_^k, ^k of the observables ^k ∈^D and the label ^k ∈. 
The label ^k ∈^m only depends on the invariant latents through a linear regression structural equation model <cit.>, described as follows:

    ^k    = ^* _A^k + ϵ_k,  _A^k ⊥ϵ_k
    ^k    = f(^k)

where ^* ∈^D × m represents the ground truth relationship between the label ^k and the invariant latents ^k_A. ϵ_k is some white noise with bounded variance and f: ^N →^D denotes the shared mixing function for all k ∈ [K] satisfying <ref>.
The set of environment distributions {P_^k, ^k}_k ∈ [K] generally differ from each other because of interventions or other distributional shifts such as covariates shift and concept shift. However, as the relationship between the invariant latents and the labels ^* and the mixing mechanism f are shared across different environments, the optimal risk remains invariant in the sense that

    _k^*(^* ∘ f^-1) = _k^'^*(^* ∘ f^-1),

where ^* denotes the ground truth relation between the invariant latents _A^k an the labels ^k and f^-1 is the inverse of the diffeomorphism mixing f(see <ref>). 
Note that this is a non-trivial ι property as the labels ^k only depend on the invariant latents _A^k, thus satisfying <ref> (ii).

=-1 Identification algorithm. Different distributional invariance are enforced by interpolating and extrapolating across various environments. Among the countless contribution to the literature, mixup <cit.> linearly interpolates observations from different environments as a robust data augmentation procedure, Domain-Adversarial Neural Networks <cit.> support the main learning task discouraging learning domain-discriminant features, Distributionally Robust Optimization (DRO) <cit.> replaces the vanilla Empirical Risk objective minimizing only with respect to the worst modeled environment,  Invariant Risk Minimization <cit.> combines the Empirical Risk objective with an invariance constraint on the gradient, and Variance Risk Extrapolation <cit.>, similar in spirit combines the empirical risk objective with an invariance constraint using the variance among environments. For a more comprehensive review of domain generalization algorithms, see <cit.>. 









§ EXPERIMENTS


This section demonstrates the real-world applicability of causal representation learning under the invariance principle, evidenced by superior treatment effect estimation performance on the high-dimensional causal inference benchmark <cit.> using a loss for the domain generalization literature <cit.>. Additionally, we provide ablation studies on existing interventional causal representation learning methods <cit.>, showcasing that non-trivial distributional invariance is needed for latent variable identification. This distributional invariance could, but does not have to arise from a valid intervention in the sense of causality. 









 §.§ Case Study: ISTAnt


Problem. Despite the majority of causal representation learning algorithms being designed to enforce the identifiability of some latent factors and tested on controlled synthetic benchmarks, there are a plethora of real-world applications across scientific disciplines requiring representation learning to answer causal questions <cit.>. Recently, <cit.> introduced ISTAnt, the first real-world representation learning benchmark with a real causal downstream task (treatment effect estimation). This benchmark highlights different challenges (sources of biases) that could arise from machine learning pipelines even in the simplest possible setting of a randomized controlled trial. Videos of ants triplets are recorded, and a per-frame representation has to be extracted for supervised behavior classification to estimate the Average Treatment Effect of an intervention (exposure to a chemical substance). 
Beyond desirable identification result on the latent factors (implying that the causal variables are recovered without bias), no clear algorithm has been proposed yet on minimizing the Treatment Effect Bias (TEB) <cit.>. One of the challenges highlighted by <cit.> is that in practice, there is both covariate and concept shifts due to the effect modification from training on a non-random subset of the RCT because, for example, ecologists do not label individual frames but whole video recordings. 

=-1 Solution. Relying on our framework, we can explicitly aim for low TEB by leveraging known data symmetries from the experimental protocol. In fact, the causal mechanism (P(Y^e|do(X^e=x)) stays invariant among the different experiment settings (i.e., individual videos or position of the petri dish). 
This condition can be easily enforced by existing domain generalization algorithms. For exemplary purposes, we choose Variance Risk Extrapolation <cit.>, which directly enforces both the invariance sufficiency constraints cons:invariance,cons:sufficiency by minimizing the the Empirical Risk together with the risk variance inter-enviroments. 


=-1 Experiment settings. In our experiments, we consider both experiment and position hard annotation sampling criteria, as described by <cit.>, defining environments as different video recording (with different experiment settings and treatment).
We varied the strength of the invariant component in the objective, varying the invariance regularization multiplier λ_INV from 0 (ERM) to 10 000 and repeated the experiments 20 times for each different value to estimate some confidence interval. All other implementational details follow <cit.>.
We evaluate the performance with the Treatment Effect Relative Bias (TERB), which is defined by <cit.> as the ratio between the bias in the predictions across treatment groups and the true average treatment effect estimated with ground-truth annotations over the whole trial. We also report the balanced accuracy for the discussion.


=-1 Results. The results on both TERB and balanced accuracy for the best performing model (pretrained model DINO, with one hidden layer head with 256 hidden nodes, learning rate 0.0005 and batch size 128), are reported in <ref>. 
As expected, the balanced accuracy initially increases with the invariance regularization strength λ_INV, as our prediction problem benefits from the invariance. At some point, the sufficiency component is not sufficiently balanced with the invariance, and performance decreases. Similarly, the TERB improves positively, weighting the invariance component until a certain threshold. 
In particular, on average with λ_INV=100 the TERB decreases to 20% (from 100% using ERM) with experiment subsampling. 
In agreement with <cit.>, a naive estimate of the TEB on a small validation set is a reasonable (albeit not perfect) model selection criterion. 
Although it performs slightly worse than model selection based on ERM loss in the position sampling case, it proves to be more reliable overall. This experiment underscores the advantages of flexibly enforcing known invariances in the data, corroborating our identifiability theory sec:identifiability.

Discussion. Interestingly, <cit.> empirically demonstrated that no domain generalization algorithm consistently outperforms Empirical Risk Minimization in out-of-distribution prediction. However, in this application, our goal is not to achieve high out-of-distribution accuracy but rather to identify a representation that is invariant to the effect modifiers introduced by the data labeling process.
This experiment serves as a clear example of the paradigm shift of CRL via the invariance principle. 
While existing CRL approaches design algorithms based on specific assumptions that are often challenging to align with real-world applications, our approach begins from the application perspective. It allows for the specification of known data symmetries and desired properties of the learned representation, followed by selecting an appropriate implementation for the distance function (potentially from existing methods). 







 §.§ Synthetic Ablation with “Ninterventions”


This subsection presents identifiability results under controversial (non-causal) conditions using simulated data. We consider the synthetic setup with full control over the latent space and the data-generating process. We consider a simple graph of three causal variables as _1 →_2 →_3. The corresponding joint density has the form of

    p_(z_1, z_2, z_3) = p(z_3 | z_2) p(z_2 | z_1) p(z_1),


The goal of this experiment is to demonstrate that existing methods for interventional CRL rely primarily on distributional invariance, regardless of whether this invariance arises from a well-defined intervention or some other arbitrary transformation. To illustrate this, we introduce the concept of a “nintervention," which has a similar distributional effect to a regular intervention, maintaining certain conditionals invariant while altering others, but without a causal interpretation.



    We define a “nintervention” on a causal conditional as the process of changing its distribution but cutting all incoming and outgoing edges. Child nodes condition on the old, pre-intervention, random variable. Formally, we consider the latent SCM as defined in <ref>, an nintervention on a node j ∈ [N] is gives rise to the following conditional factorization
    
    p̃_(z) = p̃(z_j) ∏_i ∈ [N]∖{j} p(z_i | zi^old)


Note that the marginal distribution of all non-nintervened nodes P__[N] ∖ j remain invariant after nintervention. 
 In previous example, we  perform a nintervention by replacing the conditional density p(z_2 | z_1) using a sufficiently different marginal distribution p(z̃_2) that satisfies <ref> (ii), which gives rise to the following new factorization:
 
    p̃_(z_1, z_2, z_3) = p(z_3 | z_2^old) p̃(z_2) p(z_1).

Note that _3 conditions on the random variable _2 before nintervention, whose realization is denoted as z_2^old. Differing from a intervention from the causal sense, we cut not only the incoming link of _2 but also its outgoing connection by keeping the marginal distribution of _3 the same. 
Clearly, this is a non-sensical intervention from the causal perspective because we eliminates the causal effect from _2 to its descendants. 

=-1 Experimental setting.
As a proof of concept, we choose a linear Gaussian additive noise model and a nonlinear mixing function implemented as a 3-layer invertible MLP with  activation. We average the results over three independently sampled ninterventional densities p̃(z_2) while guaranteeing all ninterventional distributions satisfy <ref> (ii). We remark that the marginal distribution of both _1, _3 remains the same after a nintervention. Hence, we expect _1, _3 to be block-identified defn:blockID according to <ref>. 

In practice, we enforce the marginal invariance constraint cons:invariance by minimizing the MMD loss, as implemented by the interventional CRL works <cit.> and train an auto-encoder for a sufficient representation cons:sufficiency. Further details are included in <ref>.


=-1 Results. To validate block-identifiability, we perform  between the estimated block [_1, _3] and the ground truth latents _1, _2, _3 respectively. The results indicate that both _1, _3 are block-identified, showing a high R^2 score of 0.863 ± 0.031 and 0.872 ± 0.035, respectively. By contrast, the latent variable _2 is not identified, evidenced by a low R^2 of 0.065 ± 0.017.

=-1 Discussion. By showing the block-identifiability results of marginal invariant latent variables _1, _3 under nintervention, we successfully validate <ref>, demystifying the spurious link between latent variable identification and causal interventions.
Throughout the experiment, we realize that a sufficiently different ninterventional distribution (<ref> (ii)) is the key to validate the identification theory properly and to observe the expected outcome, as elaborated in <ref>. 







§ CONCLUSIONS

In this paper, we take a closer look at the wide range of causal representation learning methods.
Interestingly, we find that the differences between them may often be more related to “semantics" than to fundamental methodological distinctions.
We identified two components involved in identifiability results: preserving information of the data and a set of known invariances. 
Our results have two immediate implications. First, they provide new insights into the “causal representation learning problem," particularly clarifying the role of causal assumptions. 
We have shown that while learning the graph requires traditional causal assumptions such as additive noise models or access to interventions, identifying the causal variables may not. 
This is an important result, as access to causal variables is standalone useful for downstream tasks, e.g., for training robust downstream predictors or even extracting pre-treatment covariates for treatment effect estimation <cit.>, even without knowledge of the full causal graph.
Second, we have exemplified how causal representation can lead to successful applications in practice. 
We moved the goal post from a characterization of specific assumptions that lead to identifiability, which often do not align with real-world data, to a general recipe that allow practitioners to specify known invariances in their problem and learn representations that align with them. 
In the domain generalization literature, it has been widely observed that invariant training methods often do not consistently outperform empirical risk minimization (ERM).
In our experiments, instead, we have demonstrated that the specific invariance enforced by vREX <cit.> entails good performance in our causal downstream task subsec:istant. 
Our paper leaves out certain settings concerning identifiability that may be interesting for future work, such as discrete variables and finite samples guarantees. 

One question the reader may ask, then, is “so what is exactly causal in causal representation learning?”. 
We have shown that the identifiability results in typical causal representation learning are primarily based on invariance assumptions, which do not necessarily pertain to causality. We hope this insight will broaden the applicability of these methods. 
At the same time, we used causality as a language describing the “parameterization” of the system in terms of latent causal variables with associated known symmetries. Defining the symmetries at the level of these causal variables gives the identified representation a causal meaning, important when incorporating a graph discovery step or some other causal downstream task like treatment effect estimation. Ultimately, our representations and latent causal models can be “true” in the sense of <cit.> when they allow us to predict “causal effects that one observes in practice”.  
Overall, our view also aligns with “phenomenological” accounts of causality <cit.>, that define causal variables from a set of elementary interventions. In our setting too, the identified latent variables or blocks thereof are directly defined by the invariances at hand.  From the methodological perspective, all is needed to learn causal variables is for the symmetries defined over the causal latent variables to entail some statistical footprint across pockets of data. If variables are available, learning the graph has a rich literature <cit.>, with assumptions that are often compatible with learning the variables themselves.
Our general characterization of the variable learning problem opens new frontiers for research in representation learning: 




 §.§ Representational Alignment and Platonic Representation

Several works (<cit.>) have highlighted the emergence of similar representations in neural models trained independently. In <cit.> is hypothesized that neural networks, trained with different objectives on various data and modalities, are converging toward a shared statistical model of reality within their representation spaces.  To support this hypothesis, they measure the alignment of representations proposing to use a mutual nearest-neighbor metric, which measures the mean intersection of the k-nearest neighbor sets induced by two kernels defined on the two spaces, normalized by k.
This metric can be an instance to the distance function in our formulation in <ref>.
Despite not being optimized directly,  several models in multiple settings (different objectives, data and modalities) seem to be aligned, hinting at the fact that their individual training objectives may be respecting some unknwon symmetries. 
A precise formalization of the latent causal model and identifiability in the context of foundational models remains open and will be objective for future research.



 §.§ Environment Discovery
 Domain generalization methods generalize to distributions potentially far away from the training, distribution, via learning representations invariant across distinct environments.
However this can be costly as it requires to have label information informing on the partition of the data into environments. Automatic environment discovery (<cit.>) attempts to solve this problem by learning to recover the environment partition. This is an interesting new frontier for causal representation learning, discovering data symmetries as opposed to only enforcing them. For example, this would correspond to having access to multiple interventional distributions but without knowing which samples belong to the same interventional or observational distribution. Discovering that a data set is a mixture of distributions, each being a different intervention on the same causal model, could help increase applicability of causal representations to large obeservational data sets. We expect this to be particularly relevant to downstream tasks were biases to certain experimental settings are undesirable, as in our case study on treatment effect estimation from high-dimensional recordings of a randomized controlled trial. 



 §.§ Connection with Geometric Deep Learning

Geometric deep learning (GDL)  (<cit.>) is a well estabilished learning paradigm which involves encoding a geometric understanding of data as an inductive bias in deep learning models, in order to obtain more robust models and improve performance. One fundamental direction for these priors is to encode symmetries and invariances to different types of transformations of the input data, e.g. rotations or group actions (<cit.>), in representational space. Our work can be fundamentally related with this direction, with the difference that we don't aim to model explicitly the transformations of the input space, but the invariances defined at the latent level. While an initial connection has been developed for disentanglement <cit.>, a precise connection between GDL and causal representation learning remains a open direction. We expect this to benefit the two communities in both directions: (i) by injecting geometric priors in order to craft better CRL algorithms and (ii) by incorporating causality into successful GDL  frameworks, which have been fundamentally advancing challenging real-world problems, such as protein folding (<cit.>).



§ ACKNOWLEDGEMENTS
 We thank Jiaqi Zhang, Francesco Montagna, David Lopez-Paz, Kartik Ahuja, Thomas Kipf, Sara Magliacane, Julius von Kügelgen, Kun Zhang, and Bernhard Schölkopf for extremely helpful discussion. Riccardo Cadei was supported by a Google Research Scholar Award to Francesco Locatello. We acknowledge the Third Bellairs Workshop on Causal Representation Learning held at the Bellairs Research Institute, February 9/16, 2024, and a debate on the difference between interventions and counterfactuals in disentanglement and CRL that took place during Dhanya Sridhar's lecture, which motivated us to significantly broaden the scope of the paper. We thank Dhanya and all participants of the workshop.
unsrtnat




