Causal Temporal Representation Learning with Nonstationary Sparse Transition
Xiangchen Song Zijian Li Guangyi Chen Yujia Zheng Yewen Fan Xinshuai Dong Kun Zhang



Geometrical Nonlinear Hall Effect Induced by Lorentz Force
    Wenhui Duan
    September 9, 2024
==========================================================




§ ABSTRACT 

    Causal Temporal Representation Learning (Ctrl) methods aim to identify the temporal causal dynamics of complex nonstationary temporal sequences. Despite the success of existing Ctrl methods, they require either directly observing the domain variables or assuming a Markov prior on them. Such requirements limit the application of these methods in real-world scenarios when we do not have such prior knowledge of the domain variables. To address this problem, this work adopts a sparse transition assumption, aligned with intuitive human understanding, and presents identifiability results from a theoretical perspective. In particular, we explore under what conditions on the significance of the variability of the transitions we can build a model to identify the distribution shifts. Based on the theoretical result, we introduce a novel framework, Causal Temporal Representation Learning with Nonstationary Sparse Transition (), designed to leverage the constraints on transition sparsity and conditional independence to reliably identify both distribution shifts and latent factors. Our experimental evaluations on synthetic and real-world datasets demonstrate significant improvements over existing baselines, highlighting the effectiveness of our approach.

    




§ INTRODUCTION


Causal learning from sequential data remains a fundamental yet challenging task <cit.>. 
Discovering temporal causal relations among observed variables has been extensively studied in the literature <cit.>.
However, in many real-world scenarios such as video understanding <cit.>, observed data are generated by causally related latent temporal processes or confounders rather than direct causal edges.
This leads to the task of causal temporal representation learning (Ctrl), which aims to build compact representations that concisely capture the data generation processes by inverting the mixing function that transforms latent factors into observations and identifying the transitions that govern the underlying latent causal dynamics. This learning problem is known to be challenging without specific assumptions <cit.>.
The task becomes significantly more complex with nonstationary transitions, which are often characterized by multiple distribution shifts across different domains, particularly when these domains or shifts are also unobserved.

Recent advances in unsupervised representation learning, particularly through nonlinear Independent Component Analysis (ICA), have shown promising results in identifying latent variables by incorporating side information such as class labels and domain indices <cit.>. For time-series data, historical information is widely utilized to enhance the identifiability of latent temporal causal processes <cit.>. However, existing studies primarily derive results under stationary conditions <cit.> or nonstationary conditions with observed domain indices <cit.>.
These methods are limited in application as general time series data are typically nonstationary and domain information is difficult to obtain. Recent studies <cit.> have adopted a Markov structure to handle nonstationary domain variables and can infer domain indices directly from observed data. (More related work can be found in Appendix <ref>.) However, these methods face significant limitations; some are inadequate for modeling time-delayed causal relationships in latent spaces, and they rely on the Markov property, which cannot adequately capture the arbitrary nonstationary variations in domain variables. This leads us to the following important yet unresolved question:



	 How can we establish identifiability of nonstationary nonlinear ICA for  

	 general sequence data without prior knowledge of domain variables? 


Relying on observing domain variables or known Markov priors to capture nonstationarity seems counter-intuitive, especially considering how easily humans can identify domain shifts given sufficient variation on transitions, such as video action segmentation <cit.> and recognition <cit.> tasks.

In this work, we theoretically investigate the conditions on the significance of transition variability to identify distribution shifts.
The core idea is transition clustering, assuming transitions within the same domain are similar, while transitions across different domains are distinct.
Building on this identification theorem, we propose Causal Temporal Representation Learning with Nonstationary Sparse Transition (), to identify both distribution shifts and latent temporal dynamics. Specifically, we constrain the complexity of the transition function to identify domain shifts. Subsequently, with the identified domain variables, we learn the latent variables using conditional independence constraints. These two processes are jointly optimized within a VAE framework.



The main contributions of this work are as follows: (1) To our best knowledge, this is the first identifiability result that handles nonstationary time-delayed causally related latent temporal processes without prior knowledge of the domain variables. (2) We present , a principled framework for recovering both nonstationary domain variables and time-delayed latent causal dynamics. (3) Experiments on synthetic and real-world datasets demonstrate the effectiveness of the proposed method in recovering latent variables and domain indices.





§ PROBLEM FORMULATION



 §.§ Nonstationary Time Series Generative Model

r0.3
    

    
    < g r a p h i c s >

    Graphical model for nonstationary causally related time-delayed time-series data with unobserved domain variables u_t.
    


We first introduce a nonstationary time-series generative model. Our  observational dataset is 𝒟 = {𝐱_t}_t=1^T, where 𝐱_t∈ℝ^n is produced from causally related, time-delayed latent components 𝐳_t ∈ℝ^n through an invertible mixing function 𝐠:

    𝐱_t = 𝐠(𝐳_t).

In the nonstationary setting, transitions within the latent space vary over time. Define u as the domain or regime index variable, with u_t corresponding to time step t. Assuming U distinct regimes, i.e., u_t ∈{1, 2, …, U}, each regime exhibits unknown distribution shifts. Those regimes are characterized by U different transition functions {𝐦_u}_u=1^U, which were originally explored in <cit.> through change factors to capture these distribution shifts in transition dynamics.

The i-th component of 𝐳_t, is then generated via i-th component of 𝐦:

    z_t,i = m_i(u_t, {z_t',j| z_t',j∈𝐏𝐚(z_t,i)}, ϵ_t,i),

where 𝐏𝐚(z_t,i) represents the set of latent factors directly influencing z_t,i, which may include any subset of 𝐳_<t = {z_τ,i|τ∈{1,2,…,t-1}, i∈{1,2,…,n}}.

For analytical simplicity, we assume that the parents in the causal graph are restricted to elements in 𝐳_t-1. Extensions to higher-order cases, which involve multistep, time-delayed causal relations, are discussed in Appendix S1.5 of <cit.>. These extensions are orthogonal to our contributions and are therefore omitted here for brevity.
Importantly, in a nonstationary context, 𝐏𝐚(·) may also be sensitive to the domain index u_t, indicating that causal dependency graphs vary across different domains or regimes, which will be revisited in our discussion on identifiability. We assume that the generation process for each i-th component of 𝐳_t is mutually independent, conditioned on 𝐳_<t and u_t, and we assume that the noise terms ϵ_t,i are mutually independent across dimensions and over time, thereby excluding instantaneous causal interactions among latent causal factors. Figure <ref> illustrates the graphical model for this setting.





 §.§ Identifiability of Domain Variables and Latent Causal Processes


We introduce the identifiability for both domain variables and time-delayed latent causal processes in Definitions <ref> and <ref>, respectively. If the estimated latent processes are identifiable at least up to a permutation and component-wise invertible transformations, then the latent causal relationships are also immediately identifiable. This is because conditional independence relations comprehensively characterize the time-delayed causal relations within a time-delayed causally sufficient system, in which there are no latent causal confounders in the causal processes. Notably, invertible component-wise transformations on latent causal processes preserve their conditional independence relationships. We now present definitions related to observational equivalence, the identifiability of domain variables and latent causal processes.


Formally, consider {𝐱_t}_t=1^T as a sequence of observed variables generated by true temporally causal latent processes specified by (𝐦, 𝐮, p(ϵ), 𝐠) given in Eqs. (<ref>) and (<ref>). Here, 𝐦 and ϵ denote the concatenated vector form across n dimensions in the latent space. Similarly 𝐮 for timestep 1 to T. A learned generative model (𝐦̂, 𝐮̂, p̂(ϵ), 𝐠̂) is observationally equivalent to the ground truth one (𝐦, 𝐮, p(ϵ), 𝐠) if the model distribution p_𝐦̂, 𝐮̂, p̂_ϵ, 𝐠̂({𝐱_t}_t=1^T) matches the data distribution p_𝐦, 𝐮, p_ϵ, 𝐠({𝐱_t}_t=1^T) everywhere.



    Domain variables are said to be identifiable up to label swapping if observational equivalence (Def. <ref>) implies identifiability of domain variables up to label swapping or a permutation σ for domain indices:
    
    p_𝐦̂, 𝐮̂, p̂_ϵ, 𝐠̂({𝐱_t}_t=1^T) = p_𝐦, 𝐮, p_ϵ, 𝐠({𝐱_t}_t=1^T)
        ⇒û_t = σ(u_t), ∀ t ∈{1,2,…,T}.



The latent causal processes are said to be identifiable if observational equivalence (Def. <ref>) leads to the identifiability of latent variables up to a permutation π and component-wise invertible transformation 𝒯:

    p_𝐦̂, 𝐮̂, p̂_ϵ, 𝐠̂({𝐱_t}_t=1^T) = p_𝐦, 𝐮, p_ϵ, 𝐠({𝐱_t}_t=1^T)
    ⇒𝐠̂^-1(𝐱_t) = 𝒯∘π∘𝐠^-1(𝐱_t),   ∀𝐱_t ∈𝒳,

where 𝒳 denotes the observation space.





§ IDENTIFIABILITY THEORY


In this section, we demonstrate that under mild conditions, the domain variables u_t are identifiable up to label swapping and the latent variables 𝐳_t are identifiable up to permutation and component-wise transformations. We partition our theoretical discussion into two sections: (1) identifiability of nonstationary discrete domain variables u_t and (2) identifiability of latent causal processes.
We slightly extend the usage of supp(·) to define the square matrix support and the support of a square matrix function as follows:

The support (set) of a square matrix 𝐀∈ℝ^n × n is defined using the indices of non-zero entries as:

    supp(𝐀) {(i,j) |𝐀_i,j≠ 0}.




The support (set) of a square matrix function 𝐀 : Θ→ℝ^n × n is defined as:

    supp(𝐀(Θ)) {(i,j) |∃θ∈Θ, 𝐀(θ)_i,j≠ 0 }.



For brevity, let ℳ and ℳ denote the n × n binary matrices reflecting the support of the Jacobian 𝐉_𝐦(𝐳_t) and 𝐉_𝐦̂(𝐳̂_t) respectively. The (i,j) entry of ℳ is 1 if and only if (i,j) ∈supp(𝐉_𝐦). And we can further define the transition complexity using its Fréchet norm, |ℳ| = ∑_i,jℳ_i,j, and similarly for ℳ. In the nonstationary context, this support matrix is a function of the domain index u, denoted as ℳ_u and ℳ_u. Additionally, we introduce the concept of weakly diverse lossy transitions for the data generation process:

The set of transition functions described in Eq. (<ref>) is said to be diverse lossy if it satisfies the following conditions:

        
        
  * (Lossy) For every time and indices tuple (t,i,j) with edge z_t-1,i→ z_t,j representing a causal link defined with the parents set 𝐏𝐚(z_t,j) in Eq. <ref>, transition function m_j is a lossy transformation w.r.t. z_t-1,i i.e., there exists an open set S_t,i,j, changing z_t-1,i within this set will not change the value of m_j, i.e. ∀ z_t-1,i∈ S_t,i,j,  ∂ m_j/∂ z_t-1,i = 0.
        
  * (Weakly Diverse) For every element z_t-1,i of the latent variable 𝐳_t-1 and its corresponding children set 𝒥_t,i = {j | z_t-1,i∈𝐏𝐚(z_t,j), j∈{1,2,…,n}}, transition functions {m_j}_j ∈𝒥_t,i are  weakly diverse i.e., the intersection of the sets S_t,i = ∩_j∈𝒥_t,i S_t,i,j is not empty, and such sets are diverse, i.e., S_t,i≠∅, and S_t,i,j∖ S_t,i≠∅, ∀ j∈𝒥_t,i.
    






 §.§ Identifiability of Domain Variables





Suppose that the dataset 𝒟 are generated from the nonstationary data generation process as described in Eqs. (<ref>) and (<ref>). Suppose the transitions are weakly diverse lossy (Def. <ref>)

and the following assumptions hold:


    
    
  * (Mechanism Separability) There exists a ground truth mapping 𝒞: 𝒳×𝒳→𝒰 determined the real domain indices, i.e., u_t = 𝒞(𝐱_t-1, 𝐱_t).
    
    
    
    
    
  *  (Mechanism Sparsity) The estimated transition complexity on dataset 𝒟 is less than or equal to ground truth transition complexity, i.e., 𝔼_𝒟 | ℳ_û | ≤𝔼_𝒟  | ℳ_u |.
    
    
  * (Mechanism Variability) Mechanisms are sufficiently different. For all u≠ u', ℳ_u≠ℳ_u' i.e. there exists index (i,j) such that [ℳ_u]_i,j≠[ℳ_u']_i,j.
    

Then the domain variables u_t is identifiable up to label swapping (Def. <ref>).


Theorem <ref> states that if we successfully learn a set of estimated transitions {𝐦̂_u}_u=1^U, the decoder 𝐠̂, and the domain clustering assignment 𝒞̂, where 𝐦̂_u corresponds to the estimation of Eq. (<ref>) for a particular regime or domain u, and the system can fit the data as follows:

    𝐱̂_t = 𝐠̂∘𝐦̂_û_t∘𝐠̂^-1(𝐱_t-1)   and  û_t = 𝒞̂(𝐱_t-1, 𝐱_t),
assuming that the transition complexity is kept low (as per Assumption <ref>). Then the estimated domain variables û_t must be the true domain variables u_t up to a permutation. 

Proof sketch
The core idea of this proof is to demonstrate that the global minimum of transition complexity can only be achieved when the domain variables u_t are correctly estimated. (1) First, when we have an optimal decoder estimation 𝐠̂^* which is a component-wise transformation of the ground truth, incorrect estimations of u_t will strictly increase the transition complexity, i.e., 𝔼_𝒟 |ℳ^*_û| > 𝔼_𝒟 |ℳ^*_u|. (2) Second, with arbitrary estimations û_t, the transition complexity for non-optimal decoder estimation 𝐠̂ will be equal to or higher than that for the optimal 𝐠̂^*, i.e., 𝔼_𝒟 |ℳ_û| ≥𝔼_𝒟 |ℳ^*_û|. Thus, the global minimum of transition complexity can only be achieved when u_t is optimally estimated, which must be a permuted version of the ground truth domain variables u_t. For a detailed proof, see appendix <ref>.






 §.§ Remark on Mechanism Variability


The assumption of mechanism variability, as outlined in Assumption <ref>, requires the Jacobian support matrices across domains must be distinct, which means that the causal graph linking past states (𝐳_t-1) to current states (𝐳_t) varies by at least one edge. But addressing scenarios where the causal graphs remain constant but the functions behind the edges change is challenging without additional assumptions; more detailed discussion on why this is in general challenging can be found in the Appendix <ref>.

To effectively address these scenarios, we extend the concept of the Jacobian support matrix by incorporating higher-order derivatives. This extension provides a more detailed characterization of the variability in transition functions across different domains. We now present the following definition to formalize this concept:


The k-th order partial derivative support matrix for transition 𝐦 denoted as ℳ^k is a binary n× n matrix with

    [ℳ^k]_i,j = 1 ∃𝐳∈𝒵, ∂^k m_j/∂ z_i^k≠ 0.


We utilize the variability in the higher-order partial derivative support matrix to extend the identifiability results of Theorem <ref>. This extension applies to cases where the causal graphs remain identical across two domains, yet the transition functions take different forms.



    Suppose the data 𝒟 is generated from the nonstationary data generation process described in (<ref>) and (<ref>). Assume the transitions are weakly diverse lossy (Def. <ref>), and the mechanism separability assumption <ref> along with the following assumptions hold:
    

    
    
  * (Mechanism Function Variability) Mechanism Functions are sufficiently different. There exists K ∈ℕ such that for all u≠ u', there exists k ≤ K, ℳ_u^k ≠ℳ_u'^k i.e. there exists index (i,j) such that [ℳ_u^k]_i,j≠[ℳ_u'^k]_i,j.


    
  * (Higher Order Mechanism Sparsity) The estimated transition complexity on dataset 𝒟 is no more than ground truth transition complexity,
    
    𝔼_𝒟∑_k=1^K | ℳ_û^k | ≤𝔼_𝒟∑_k=1^K | ℳ_u^k |.

    Then the domain variables u_t are identifiable up to label swapping (Def. <ref>).

We utilize the fact that for two distinct domains, there exists an edge in the causal graph, and its k-th order partial derivative supports are different, making the two domains separable. The detailed proof of this extension is provided in Appendix <ref>.


 §.§ Identifiability of Latent Causal Process


Once the identifiability of u_t is achieved, the problem reduces to a nonstationary temporal nonlinear ICA with an observed domain index.

Leveraging the sufficient variability approach proposed in <cit.>, we demonstrate full identifiability. This sufficient variability concept is further incorporated into the following lemma, adapted from Theorem 2 in <cit.>:


Suppose that the data 𝒟 are generated from the nonstationary data generation process as described in Eqs. (<ref>) and (<ref>). Let η_kt(u) denote the logarithmic density of k-th variable in 𝐳_t, i.e., η_kt(u)≜log p(z_t,k | 𝐳_t-1, u), and there exists an invertible function 𝐠̂  that maps 𝐱_t to 𝐳̂_t, i.e., 𝐳̂_t = 𝐠̂(𝐱_t)

such that the components of 𝐳̂_t are mutually  independent conditional on 𝐳̂_t-1.
(Sufficient variability) Let

    𝐯_k,t(u)    ≜(∂^2 η_kt(u)/∂ z_t,k∂ z_t-1,1, ∂^2 η_kt(u)/∂ z_t,k∂ z_t-1,2, ..., ∂^2 η_kt(u)/∂ z_t,k∂ z_t-1,n)^⊺,
    𝐯_k,t(u)    ≜(∂^3 η_kt(u)/∂ z_t,k^2 ∂ z_t-1,1, ∂^3 η_kt(u)/∂ z_t,k^2 ∂ z_t-1,2, ..., ∂^3 η_kt(u)/∂ z_t,k^2 ∂ z_t-1,n)^⊺.



    𝐬_kt   ≜( 𝐯_kt(1)^⊺, ..., 
        𝐯_kt(U)^⊺, 
        ∂^2 η_kt(2)/∂ z_t,k^2  - 
        ∂^2 η_kt(1)/∂ z_t,k^2 , ...,
        ∂^2 η_kt(U)/∂ z_t,k^2  - 
        ∂^2 η_kt(U-1)/∂ z_t,k^2 )^⊺,
    𝐬_kt   ≜( 𝐯_kt(1)^⊺, ..., 
        𝐯_kt(U)^⊺, 
        ∂η_kt(2)/∂ z_t,k - 
        ∂η_kt(1)/∂ z_t,k, ...,
        ∂η_kt(U)/∂ z_t,k - 
        ∂η_kt(U-1)/∂ z_t,k)^⊺.
Suppose 𝐱_t = 𝐠(𝐳_t) and that the conditional distribution p(z_k,t | 𝐳_t-1) may change across m domains. Suppose that the components of 𝐳_t are mutually independent conditional on 𝐳_t-1 in each context. Assume that the components of 𝐳̂_t produced by 𝐠̂ are also mutually independent conditional on 𝐳̂_t-1. 
If the 2n function vectors 𝐬_k,t and 𝐬_k,t, with k=1,2,...,n, are linearly independent, then 𝐳̂_t is a permuted invertible component-wise transformation of 𝐳_t. 

Then, in conjunction with Theorem <ref>, complete identifiability is achieved for both the domain variables u_t and the independent components 𝐳_t. See detailed proof in Appendix <ref>.

    Suppose that the data 𝒟 are generated from the nonstationary data generation process as described in Eqs. (<ref>) and (<ref>), which satisfies the conditions in both Theorem <ref> and Lemma <ref>, then the domain variables u_t are identifiable up to label swapping (Def. <ref>) and latent causal process 𝐳_t are identifiable up to permutation and a component-wise transformation (Def. <ref>).

Discussion on Assumptions

The proof of Theorem <ref> relies on several key assumptions which align with human intuition for understanding of domain transitions.
Firstly, separability states that if human observers cannot distinguish between two domains, it is unlikely that automated systems can achieve this distinction either.
Secondly, variability requires that the transitions across domains are significant enough to be noticeable by humans, implying that there must be at least one altered edge in the causal graph across the domains.


The mechanism sparsity is a standard assumption that has been previously explored in <cit.> using sparsity regularization to enforce the sparsity of the estimated function.
The assumption of weakly diverse lossy transitions is a mild and realistic condition in real-world scenarios, allowing for identical future latent states with differing past states. 

The sufficient variability in Theorem <ref> is widely explored and adopted in nonlinear ICA literature <cit.>. For a more detailed discussion of the feasibility and intuition behind these assumptions, we refer the reader to the Appendix <ref>.




§ THE  FRAMEWORK




 §.§ Model Architecture

r0.4
    

    
    < g r a p h i c s >

    Illustration of  with (1) Sparse Transition, (2) Prior Network, (3) Encoder-Decoder Module.
    



Our framework builds on VAE <cit.> architecture, incorporating dedicate modules to handle nonstationarity. It enforces the conditions discussed in Sec. <ref> as constraints. As shown in Fig. <ref>, the framework consists of three primary components: (1) Sparse Transition, (2) Prior Network, and (3) Encoder-Decoder.

Sparse Transition

The transition module in our framework is designed to estimate transition functions {𝐦̂_u}_u=1^U and a clustering function 𝒞̂ as specified in Eq. (<ref>). As highlighted in Sec. <ref>, the primary objective of this module is to model the transitions in the latent space and minimize the empirical transition complexity. To achieve this, we implemented U different transition networks for various 𝐦̂(û_t,·) and added sparsity regularization to the transition functions via a sparsity loss. A gating function with a (hard)-Gumbel-Softmax function was used to generate û_t, which was then employed to select the corresponding transition network 𝐦̂_û_t. This network was further used to calculate the transition loss, which is explained in detail in Sec. <ref>.



Prior Network
The Prior Network module aims to effectively estimate the prior distribution p(ẑ_t,i | 𝐳̂_t-1, û_t). This is achieved by evaluating p(ẑ_t  | 𝐳̂_t-1, û_t) = p_ϵ_i(m̂_i^-1(û_t,ẑ_t,i, 𝐳̂_t-1))|∂m̂_i^-1/∂ẑ_t,i|, where m̂_i^-1(û_t,·) is the learned holistic inverse dynamics model.

To ensure the conditional independence of the estimated latent variables, p(𝐳̂_t  | 𝐳̂_t-1), we utilize an isomorphic noise distribution for ϵ and aggregate all estimated component densities to obtain the joint distribution p(𝐳̂_t  | 𝐳̂_t-1, û_t) as shown in Eq. (<ref>). Given the lower-triangular nature of the Jacobian, its determinant can be computed as the product of its diagonal terms. Detailed derivations is provided in Appendix <ref>.

    log p(𝐳̂_t |𝐳̂_t-1, û_t) = ∑_i=1^n log p(ϵ̂_i |û_t)_Conditional independence + ∑_i=1^n log| ∂m̂^-1_i/∂ẑ_t,i|_Lower-triangular Jacobian
Encoder-Decoder
The third component is an Encoder-Decoder module that utilizes reconstruction loss to enforce the invertibility of the learned mixing function 𝐠̂. Specifically, the encoder fits the demixing function 𝐠̂^-1 and the decoder fits the mixing function 𝐠̂.



 §.§ Optimization

The first training objective of  is to fit the estimated transitions with minimum transition complexity according to Eq. (<ref>):

    ℒ_sparse≜𝔼_𝒟 L(𝐦̂_û_t(𝐳̂_t-1), 𝐳̂_t)_Transition loss + 𝔼_𝒟 | ℳ_û |_Sparsity loss,
where L(·,·) is a regression loss function to fit the transition estimations, and the sparsity loss is approximated via L_2 norm of the parameter in the transition estimation functions.


Then the second part is to maximize the Evidence Lower BOund (ELBO) for the VAE framework, which can be written as follows (complete derivation steps are in Appendix <ref>):


    ELBO≜𝔼_𝐳_t∑_t=1^Tlog p_data(𝐱_t|𝐳_t)_-ℒ_Recon 
    + ∑_t=1^Tlog  p_data(𝐳_t|𝐳_t-1,u_t) 
    - ∑_t=1^Tlog q_ϕ(𝐳_t|𝐱_t)_-ℒ_KLD
We use mean-squared error for the reconstruction likelihood loss ℒ_Recon.
The KL divergence  ℒ_KLD is estimated via a sampling approach since with a learned nonparametric transition prior, the distribution does not have an explicit form.
Specifically, we obtain the log-likelihood of the posterior, evaluate the prior log p(𝐳̂_t |𝐳̂_t-1, û_t) in Eq. (<ref>), and compute their mean difference in the dataset as the KL loss: ℒ_KLD = 𝔼_ẑ_t ∼ q(ẑ_t |𝐱_t)log q(ẑ_t|𝐱_t) - log p(𝐳̂_t |𝐳̂_t-1, û_t).




§ EXPERIMENTS

We assessed the identifiability performance of  on both synthetic and real-world datasets.
For synthetic datasets, where we control the data generation process completely, we conducted a comprehensive evaluation. This evaluation covers the full spectrum of unknown nonstationary causal temporal representation learning, including metrics for both domain variables and the latent causal processes.
In real-world scenarios,  was employed in video action segmentation tasks. The evaluation metrics focus on the accuracy of action estimation for each video frame, which directly reflects the identifiability of domain variables.



 §.§ Synthetic Experiments on Causal Representation Learning

Experiment Setup   
For domain variables, we assessed the clustering accuracy (Acc) to estimate discrete domain variables u_t. As the label order in clustering algorithms is not predetermined, we selected the order or permutation that yielded the highest accuracy score.
For the latent causal processes, we computed the mean correlation coefficient (MCC) between the estimated latent variables 𝐳̂_t and the ground truth 𝐳_t. The MCC, a standard measure in the ICA literature for continuous variables, assesses the identifiability of the learned latent causal processes. We adjusted the reported MCC values in Table <ref> by multiplying them by 100 to enhance the significance of the comparisons.

Baselines  We compared our method with identifiable nonlinear ICA methods: 
(1) BetaVAE <cit.>, which ignores both history and nonstationarity information. 
(2) i-VAE <cit.> and TCL <cit.>, which leverage nonstationarity to establish identifiability but assume independent factors. 
(3) SlowVAE <cit.> and PCL <cit.>, which exploit temporal constraints but assume independent sources and stationary processes. 
(4) TDRL <cit.>, which assumes nonstationary causal processes but with observed domain indices. 
(5) HMNLICA <cit.>, which considers the unobserved nonstationary part in the data generation process but does not allow any causally related time-delayed relations. 
(6) NCTRL <cit.>, which extends HMNLICA to an autoregressive setting to allow causally related time-delayed relations in the latent space but still assumes a Markov chain on the domain variables.



Result and Analysis  We generate synthetic datasets that satisfy our identifiability conditions in Theorems <ref> and <ref>, detailed procedures are in Appendix <ref>. The primary findings are presented in Table <ref>. 
Note: the MCC metric is consistently available in all methods; however, the Acc metric for u_t is only applicable to methods capable of estimating domain variables u_t.
In the first row of Table <ref>, we evaluated a recent nonlinear temporalr0.7


    Experiment results of synthetic dataset on baseline models and the proposed . All experiments were conducted using three different random seeds to calculate the average and standard deviation. The best results are highlighted in bold.
    
    
    
    
    u_t     Method     𝐳_t MCC     u_t Acc (%) 

    Ground Truth    TDRL(GT)     96.93 ± 0.16     -

    6*N/A    TCL       24.19 ± 0.85     6*- 

       PCL       38.46 ± 6.85    

       BetaVAE     42.37 ± 1.47    

       SlowVAE     41.82 ± 2.55    

       i-VAE      81.60 ± 2.51    

       TDRL      53.45 ± 1.31    

    3*Estimated   HMNLICA     17.82 ± 30.87        13.67 ± 23.67

       NCTRL      47.27 ± 2.15       34.94 ± 4.20

            96.74 ± 0.17     98.21 ± 0.05

    
    


ICA method, TDRL, providing ground truth u_t to establish an upper performance limit for the proposed framework.
The high MCC (> 0.95) indicates the model's identifiability.
Subsequently, the table lists six baseline methods that neglect the nonstationary domain variables, with none achieving a high MCC. The remaining approaches, including our proposed , are able to estimate the domain variables u_t and recover the latent variables. In particular, HMNLICA exhibits instability during training, leading to considerable performance variability. This instability stems from HMNLICA’s inability to allow time-delayed causal relationships among hidden variables 𝐳_t, leading to model training failure when the actual domain variables deviate from the Markov assumption. In contrast, NCTRL, which extends TDRL under the same assumption, demonstrates enhanced stability and performance over HMNLICA by accommodating transitions in 𝐳_t. However, since they use incorrect assumption on the nonstationary domain variables, the performance of those methods can be even worse than methods which do not include the domain information. Nevertheless, considering the significant nonstationarity and deviation from the Markov properties, those methods struggled to robustly estimate either the domain variables or the latent causal processes. Compared to all baselines, our proposed  reliably recovers both u_t (MCC > 0.95) and 𝐳_t (Acc > 95%), and the MCC is on par with the upper performance bound when domain variables are given, justifying it effectivess.

l0.4
    

    
    < g r a p h i c s >

    

    Visualization of three phase training process of .
    


Detailed Training Analysis  To further validate our theoretical analysis, we present a visualization of the entire training process for  in Figure <ref>. It consists of three phases: (1) In Phase 1, the initial estimations for both u_t and 𝐳_t are imprecise. (2) During Phase 2, the accuracy of the estimation of u_t continues to improve, although the quality of the estimation of 𝐳_t remains relatively unchanged compared with Phase 1. (3) In Phase 3, as u_t becomes clearly identifiable, the MCC of 𝐳_t progressively improves, ultimately achieving full identifiability. This three-phase process aligns perfectly with our theoretical predictions. According to Theorem <ref>, phases 1 and 2 should exhibit suboptimal 𝐳_t estimations, while sparsity constraints can still guide training and improve the accuracy for domain variables u_t. Once the accuracy of u_t approaches high, Theorem <ref> drives the improvement in MCC for 𝐳_t estimations, leading to the final achievement of full identifiability of both latent causal processes for 𝐳_t and domain variables u_t.



 §.§ Real-world Application on Weakly Supervised Action Segmentation

Experiment Setup
Our method was tested on the video action segmentation task to estimate actions (domain variables u_t). Following <cit.>, we use the same weakly supervised setting utilizing meta-information, such as action order. The evaluation included several metrics: Mean-over-Frames (MoF), the percentage of correctly predicted labels per frame; 

Intersection-over-Union (IoU), defined as |I ∩ I^*|/|I ∪ I^*|; and Intersection-over-Detection (IoD), |I ∩ I^*| / |I|, I^* and I are the ground-truth segment and the predicted segment with the same class. Our evaluation used two datasets: Hollywood Extended <cit.>, which includes 937 videos with 16 daily action categories, and CrossTask <cit.>, focusing on 14 of 18 primary tasks related to cooking <cit.>, comprising 2552 videos across 80 action categories. 

Model Design  Our model is build on top of ATBA <cit.> method which uses multi-layer transformers as backbone networks. We add our sparse transition module with the sparsity loss function detailed in Sec. <ref>. Specifically, we integrated a temporally latent transition layer into ATBA's backbone, using a transformer layer across time axis for the Hollywood dataset and an LSTM for the CrossTask dataset. To enforce sparsity in the latent transitions, L_2 regularization is applied to the weights of the temporally latent transition layer.

l0.62




Real-world experiment result on action segmentation task. We use the reported value for the baseline methods from <cit.>. Best results are highlighted in bold.

Dataset                Method                                   MoF      IoU       IoD   
 
6*Hollywood      HMM+RNN <cit.>                  -                 11.9               -              

                                CDFL <cit.>                          45.0              19.5               25.8           

                            
                                TASL <cit.>               42.1              23.3               33 

                            
                                MuCon <cit.>                            -                 13.9               -              

                                ATBA <cit.>                       47.7              28.5               44.9           

                            (l)2-5 
                                 (Ours)                           52.9±3.1     32.7±1.3      52.4±1.8  

                            
6*CrossTask      NN-Viterbi <cit.>        26.5              10.7               24.0             

                                CDFL <cit.>                          31.9              11.5               23.8           

                                TASL <cit.>                          40.7              14.5               25.1           

                                POC <cit.>                              42.8              15.6               -              

                                ATBA <cit.>                       50.6              15.7               24.6  

                            (l)2-5 
                                 (Ours)                           54.0±0.9     15.7±0.5      23.6±0.8           







Result and Analysis  
The primary outcomes for real-world applications in action segmentation are summarized in Table <ref>. Traditional methods based on hidden Markov models, such as HMM+RNN <cit.> and NN-Viterbi <cit.>, face challenges in these real-world scenarios. This observation corroborates our previous discussions on the limitations of earlier identifiability methods <cit.>, which depend on the Markov assumption for domain variables. Our approach significantly outperforms the baselines in both the Hollywood and CrossTask datasets across most metrics. Especially in the Hollywood dataset, our method outperforms the base ATBA model by quite a large margin. Notably, the Mean-over-Frames (MoF) metric aligns well with our identifiability results for domain variables u_t. Our method demonstrates substantial superiority in this metric. For Intersection-over-Union (IoU) and Intersection-over-Detection (IoD), our results are comparable to those of the baseline methods in the CrossTask dataset and show its superiority in the Hollywood dataset. Furthermore, our proposed sparse transition module aligns with human intuition and is easily integrated into existing methods, further enhancing its impact in real-world scenarios. Additional discussion with visualization in Appendix <ref>.


r0.45
    

    
    Ablation study on sparse transition module in Hollywood dataset.
    
    Method        MoF      IoU       IoD   
 
        52.9              32.7               52.4 

         - Complexity                 50.5              31.5               51.5 

         - Module                   47.7              28.5               44.9 

    
    
    
    
    
    
    


Abalation Study  Furthermore, we conducted an ablation study on the sparse transition module, as detailed in Table <ref>. In this study, “- Complexity” refers to the configuration where we retain the latent transition layers but omit the sparse transition complexity regularization term from these layers, and “- Module” indicates the removal of the entire sparse transition module, effectively reverting the model to the baseline ATBA model. The comparative results in Table <ref> demonstrate that both the dedicated design of the sparse transition module and the complexity regularization term significantly enhance the performance.






§ CONCLUSION

In this study, we developed a comprehensive identifiability theory tailored for general sequential data influenced by nonstationary causal processes under unspecified distributional changes. We then introduced , a principled approach to recover both latent causal variables with their time-delayed causal relations, as well as determining the values of domain variables from observational data without relying on distributional or structural prior knowledge. Our experimental results demonstrate that  can reliably estimate the domain indices and recover the latent causal process. And such module can be easily adapted to handle real-world scenarios such as action segmentation task.







unsrtnat

Supplement to
  

      
      
      “Causal Temporal Representation Learning with 
 Nonstationary Sparse Transition”

Appendix organization:






§ IDENTIFIABILITY THEORY














 §.§ Proof for Theorem <ref>


We divide the complete proof into two principal steps: 

    
  * Firstly, assuming access to the optimal mixing function estimation 𝐠̂^*, we demonstrate that under the conditions in our theorem, the estimated clustering result will align with the ground truth up to label swapping. This alignment is due to the transition complexity with optimal û^*_t and 𝐠̂^* being strictly lower than that with non-optimal û_t but still optimal 𝐠̂^*.
    
  * Secondly, we generalize the results of the first step to cases where the mixing function estimation 𝐠̂ is suboptimal. We establish that for any given clustering assignment, whether optimal or not, a suboptimal mixing function estimation 𝐠̂ can not result in a lower transition complexity. Thus, the transition complexity in scenarios with non-optimal 𝐠̂ will always be at least as high as in the optimal case. 

From those two steps, we conclude that the global minimum transition complexity can only be attained when the estimation of domain variables û_t is optimal, hence ensuring that the estimated clustering must match the ground truth up to label swapping. It is important to note that this condition alone does not guarantee the identifiability of the mixing function 𝐠. Because a setting with optimal û^*_t and a non-optimal 𝐠̂ may exhibit equivalent transition complexity to the optimal scenario, but it does not compromise our proof for the identifiability of domain variables u_t. Further exploration of the mixing function's identifiability 𝐠 is discussed in Theorem <ref> in the subsequent section.





  §.§.§ Identifiability of C under optimal g

r0.37


    
    < g r a p h i c s >
 
    Illustration of 𝒞̂ incorrectly assigning two different domain subsets of inputs A and B into the same û. The black lines represent the ground truth partition of 𝒞 and the orange line represent the incorrect domain partition for set A and B.
    
    


We fist introduce a lemma for this case when we can access an optimal mixing function estimation 𝐠̂^*.



    In addition to the assumptions in Theorem <ref>, assume that we can also access an optimal estimation of 𝐠, denoted by 𝐠̂^*, in which the estimated 𝐳̂_t is an invertible, component-wise transformation of a permuted version of 𝐳_t. Then the estimated clustering 𝒞̂ must match the ground truth up to label swapping. 


In the first case we deal with optimal estimation 𝐠̂^* in which the estimated 𝐳̂_t is an invertible, component-wise transformation of a permuted version of 𝐳_t, but inaccurate estimated version of 𝒞̂, Consider the following example (Figure <ref>):



With slight abuse of notation, we use 𝒞(A) to represent the domain assigned by 𝒞 to all elements in A, and all elements in A have the same assignment. The same argument applies to B.

Then, for an estimated 𝒞̂, if it incorrectly assigns two subsets of input A and B to the same û (Figure <ref> orange circle), i.e., 

    𝒞(A) = i ≠ j = 𝒞(B)   but  𝒞̂(A) = Ĉ(B) = k.

Note that if the ground truth 𝒞 gives a consistent assignment for A and B but estimated 𝒞̂ gives diverse assignments, i.e. 
    𝒞̂(A) = i ≠ j = 𝒞̂(B)   but  𝒞(A) = 𝒞(B) = k,
 it is nothing but further splitting the ground truth assignment in a more fine-grained manner.
This scenario does not break the boundaries of the ground truth assignments. Consider two cases in the estimation process:

    
  * If the number of allowed regimes or domains exceeds that of the ground truth, such more fine-grained assignment is allowed. The ground truth can then be easily recovered by merging domains that share identical Jacobian supports.
    
  * If the number of regimes or domains matches the ground truth, it can be shown that the inconsistent scenario outlined in Eq. (<ref>) must occur.

Given that these considerations do not directly affect our approach, they are omitted from further discussion for brevity.



Then considering the case in Eq. <ref>, the estimated transition must cover the functions from both A and B, then the learned transition 𝐦̂_k must have Jacobian 𝐉_𝐦̂_k with support matrix ℳ_k = ℳ_i + ℳ_j which is the binary addition of ℳ_i and ℳ_j, such that for all indices in ℳ_i, ℳ_j if any of these two is 1, then the corresponding position in ℳ_k must be 1. That is because if that is not the case, for example, the (a,b)-th location for ℳ_i, ℳ_j, and ℳ_k are 1, 0, and 0. Then we can easily find an input region for the (a,b)-th location such that a small perturbation can lead to changes in 𝐦_i but not in 𝐦_j nor 𝐦̂_k, which makes 𝐦̂_k unable to fit all of the transitions in A∪ B which cause contradiction. See the three matrices in Figure <ref> for an illustrative example.



By assumption <ref>, since all those support matrix differ at least one spot, which means the estimated version is not smaller than the ground truth. 

    | ℳ_k | ≥ | ℳ_j |  and  | ℳ_k | ≥ |ℳ_i |,

and the equality cannot hold true at the same time. 

Then from Assumption (<ref>), the expected estimated transition complexity can be expressed as:

    𝔼_𝒟 |ℳ_û| = ∫_𝒳×𝒳 p_𝒟(𝐱_t-1, 𝐱_t) · | ℳ_𝒞̂(𝐱_t-1, 𝐱_t) |   d𝐱_t-1  d𝐱_t.

Similarly for ground truth one:

    𝔼_𝒟 |ℳ_u| = ∫_𝒳×𝒳 p_𝒟(𝐱_t-1, 𝐱_t) · | ℳ_𝒞(𝐱_t-1, 𝐱_t) |   d𝐱_t-1  d𝐱_t.

Let us focus on the integral of the region A∪ B, the subset of 𝒳×𝒳 mentioned above.

If for some area p_𝒟(𝐱_t-1, 𝐱_t) = 0, then the clustering under this area is ill defined since there is no support from data. Hence we only need to deal with supported area.
For area that p_𝒟(𝐱_t-1, 𝐱_t) >0 and from Eq. (<ref>) the equality cannot hold true at the same time, then the estimated version of the integral is strictly larger than the ground-truth version for any inconsistent clustering as indicated in Eq. (<ref>).

For the rest of regions in 𝒳×𝒳, any incorrect cluster assignment will further increase the ℳ with same reason as discussed above, then the estimated complexity is strictly larger than the ground truth complexity:

    𝔼_𝒟 | ℳ_û | > 𝔼_𝒟 | ℳ_u |.


But assumption (<ref>) requires that the estimated complexity be less than or equal to the ground truth. Contradiction! Hence, the estimated 𝒞̂ must match the ground truth up to label swapping.




  §.§.§ Identifiability of C under arbitrary g

Now we can leverage the conclusion in Lemma <ref> to show the identifiability of domain variables under arbitrary mixing function estimation.

Suppose that the data 𝒟 are generated from the nonstationary data generation process as described in Eqs. (<ref>) and (<ref>). Suppose the transitions are weakly diverse lossy (Def. <ref>) and the following assumptions hold:

    
  * (Mechanism Separability) There exists a ground truth mapping 𝒞: 𝒳×𝒳→𝒰 determined the real domain indices, i.e., u_t = 𝒞(𝐱_t-1, 𝐱_t).
    
  * (Mechanism Sparsity) The estimated transition complexity on dataset 𝒟 is less than or equal to ground truth transition complexity, i.e., 𝔼_𝒟 | ℳ_û | ≤𝔼_𝒟  | ℳ_u |.
    
  * (Mechanism Variability) Mechanisms are sufficiently different. For all u≠ u', ℳ_u≠ℳ_u' i.e. there exists index (i,j) such that [ℳ_u]_i,j≠[ℳ_u']_i,j.

Then the domain variables u_t is identifiable up to label swapping (Def. <ref>).






To demonstrate the complete identifiability of 𝒞, independent of the estimation quality of 𝐠, we must show that for any arbitrary estimation 𝒞̂≠σ(𝒞), the induced ℳ̂_û for inaccurate estimation 𝐠̂ has a transition complexity at least as high as in the optimal 𝐠̂^* case. If this holds, from Lemma <ref>, we can conclude that the transition complexity of optimal 𝒞̂^* = σ(𝒞) and optimal 𝐠̂^* is strictly smaller than any non-optimal 𝒞̂ and any 𝐠̂.

Suppose the estimated decoder and corresponding latent variables are 𝐠̂ and 𝐳̂_t, respectively, then the following relation holds:

    𝐠̂^*(𝐳_t) = 𝐠̂(𝐳̂_t).

Since 𝐠̂ is invertible, by composing 𝐠̂^-1 on both sides, we obtain:

    𝐠̂^-1∘𝐠̂^*(𝐳_t) = 𝐠̂^-1∘𝐠̂(𝐳̂_t).

Let 

    𝐡𝐠̂^-1∘𝐠̂^*,

we then have:

    𝐡(𝐳̂^*_t) = 𝐳̂_t.

We aim to demonstrate that under this transformation, if 𝐡 is not a permutation and component-wise transformation, the introduced transition complexity among estimated 𝐳̂ will not be smaller than the optimal 𝐠̂^*.


Suppose |ℳ| < |ℳ^*|, then for any permutation σ mapping the indices of the dimensions from ℳ to ℳ^*, there must exist an index pair (i,j) such that ℳ_i,j = 0 and ℳ^*_σ(i),σ(j) = 1.



An intuitive explanation for this proposition involves the construction of a directed graph G_ℳ^* = (V_ℳ^*, E_ℳ^*), where V_ℳ^* = {1, 2, …, n} and E_ℳ^* = {(i,j) |ℳ^*_i,j = 1}. A similar construction can be made for G_ℳ. It is straightforward that |ℳ^*| = |E_ℳ^*|, which represents the number of edges. Consequently, |ℳ| < |ℳ^*| implies that G_ℳ^* has more edges than G_ℳ. Since there is no pre-defined ordering information for the nodes in these two graphs, if we wish to compare their edges, we need to first establish a mapping. However, if |E_ℳ| < |E_ℳ^*|, no matter how the mapping σ is constructed, there must be an index pair (i,j) such that (i,j) ∉ E_ℳ but (σ(i),σ(j)) ∈ E_ℳ^*. Otherwise, if such an index pair does not exist, the total number of edges in G_ℳ would necessarily be greater than or equal to that in G_ℳ^*, contradicting the premise that |ℳ| < |ℳ^*|.
























    Suppose transitions are weakly diverse lossy as defined in Def. <ref> and an invertible transformation 𝐡 maps the optimal estimation 𝐳̂^*_t to the estimated 𝐳̂_t, and it is neither a permutation nor a component-wise transformation. Then, the transition complexity on the estimated 𝐳̂_t is not lower than that on the optimal 𝐳̂^*_t, i.e., 
    
    |ℳ| ≥ |ℳ^*|.
































The entire proof is based on contradiction. In Figure <ref>, we provide an illustrative example. Note that the mapping from ground truth 𝐳_t to optimal estimation 𝐳̂^*_t is a permutation and element-wise transformation, it does not include mixing, and hence e_i exists if and only if ê^*_i exists. Therefore, |ℳ^*| = |ℳ|. The core of the proof requires us to demonstrate that |ℳ| < |ℳ| cannot be true.




Suppose the transitions are weakly diverse lossy as defined in Def. <ref>, then for each edge z_t,i→ z_t+1,j in the transition graph, there must be a region of z_t,i such that only z_t+1,j is influenced by z_t,i. Consequently, the corresponding ẑ_t+1,j and ẑ_t+1,i are not independent, since no mixing process can cancel the influence of z_t,i. Therefore, the edge ẑ_t,i→ẑ_t+1,j in the estimated graph must exist.

Note that without the weakly diverse lossy transition assumption, this argument may not hold. For example, if ẑ_t+1,j can be expressed as a function that does not depend on z_t,i, then even though the edge z_t,i→ z_t+1,j exists, the estimated edge ẑ_t,i→ẑ_t+1,j may not exist. This could occur if, after the transformation 𝐡, the influences in different paths from z_t,i to ẑ_t+1,j cancel out with each other.




  
Necessity Example
An example that violates the assumption is as follows:

    z_t+1,i   = z_t,i + ϵ_t+1,i
    
        z_t+1,j   = z_t,i + ϵ_t+1,j
    ẑ_i   = z_i
    ẑ_j   = z_i - z_j

Here, the mapping from 𝐳 to 𝐳̂ is invertible. Writing down the mapping from 𝐳̂_t to 𝐳̂_t+1, particularly for ẑ_t+1, j, yields:

    ẑ_t+1, j   = (z_t,i + ϵ_t+1,i) - (z_t,i + ϵ_t+1,j) 
       = ϵ_t+1,i - ϵ_t+1,j


Clearly, this is independent of ẑ_t,i. Hence, in this scenario, the edge on the estimated graph does not exist. This explains the necessity for the weakly diverse lossy transition assumption. Furthermore, it can be seen that violating the weakly diverse lossy transition assumption would require a very specific design, such as the transition in an additive noise case and the transition on z being linear, which is usually not the case in real-world scenarios. Generally, this requires that the influences from different paths from z_t,i to ẑ_t+1,j cancel each other out, a condition that is very challenging to fulfill in practical settings.




  
Permutation Indexing
One may also ask about the permutation of the index between 𝐳_t and 𝐳̂_t. Since the transformation 𝐡 is invertible, the determinant of the Jacobian should be nonzero, implying the existence of a permutation σ such that

    (i,σ(i)) ∈supp(𝐉_𝐡), ∀ i ∈ [n].

Otherwise, if there exists an i such that [𝐉_𝐡]_i,· = 0 or [𝐉_𝐡]_·,i = 0, such a transformation cannot be invertible. We can utilize this permutation σ to pair the dimensions in 𝐳_t and 𝐳̂_t.

Since each ground-truth edge is preserved in the estimated graph, by Proposition <ref>, the inequality |ℳ| < |ℳ^*| cannot hold true. Thus, the lemma is proved.













































Then, according to this lemma, the transition complexity |ℳ_û| of the learned 𝐦̂_û should be greater than or equal to |ℳ^*_û|, which is the complexity when using an accurate estimation of 𝐠̂^*. This relationship can be expressed as follows:


    | ℳ_û | ≥  |ℳ^*_û| .



By Lemma <ref>, the expected complexity of the estimated model 𝔼_𝒟 |ℳ̂^*_û | is strictly larger than that of the ground truth 𝔼_𝒟 | ℳ_u |. This implies the following inequality:

    𝔼_𝒟 | ℳ_û | ≥𝔼_𝒟 |ℳ^*_û| > 𝔼_𝒟 | ℳ_u | .



However, Assumption (<ref>) requires that the estimated complexity must be less than or equal to the ground-truth complexity, leading to a contradiction. This contradiction implies that the estimated 𝒞̂ must match the ground truth up to label swapping. Consequently, this supports the conclusion of Theorem <ref>.





 §.§ Proof of Corollary <ref>






    Suppose the data 𝒟 is generated from the nonstationary data generation process described in (<ref>) and (<ref>). Assume the transitions are weakly diverse lossy (Def. <ref>), and the mechanism separability assumption <ref> along with the following assumptions hold:
    
    
  * (Mechanism Function Variability) Mechanism Functions are sufficiently different. There exists K ∈ℕ such that for all u≠ u', there exists k ≤ K, ℳ_u^k ≠ℳ_u'^k i.e. there exists index (i,j) such that [ℳ_u^k]_i,j≠[ℳ_u'^k]_i,j.
    
  * (Higher Order Mechanism Sparsity) The estimated transition complexity on dataset 𝒟 is no more than ground truth transition complexity,
    
    𝔼_𝒟∑_k=1^K | ℳ_û^k | ≤𝔼_𝒟∑_k=1^K | ℳ_u^k |.

    Then the domain variables u_t are identifiable up to label swapping (Def. <ref>).








With a strategy similar to the proof of Theorem <ref>, we aim to demonstrate that using an incorrect cluster assignment 𝒞̂ will result in ∑_k=1^K |ℳ^k_û| being higher than the ground truth, thereby still enforcing the correct u_t.

Differing slightly from the approach in Theorem <ref>, in this setting, we will first demonstrate that under any arbitrary 𝒞̂ assignment, the estimated complexity is no lower than the complexity in the ground truth, i.e., ∑_k=1^K|ℳ^k| ≥∑_k=1^K|ℳ^k|. 

First, we address the scenario where two different domains have the same transition graph but with different functions, as otherwise, the previous lemma <ref> still applies. In cases where the same transition causal graph exists but the functions differ, assumption <ref> indicates that there exists an integer k such that ℳ^k_u≠ℳ^k_u', meaning the ground truth support matrices are different. However, due to incorrect clustering, the learned transition must cover both cases. To substantiate this claim, we need to first introduce an extension of the non-decreasing complexity lemma.



    Suppose there exists an invertible transformation 𝐡 which maps the ground truth 𝐳_t to the estimated 𝐳̂_t, and it is neither a permutation nor a component-wise transformation. Then, the transition complexity on the estimated 𝐳̂_t is not lower than that on the ground truth 𝐳_t, i.e., 
    
    ∑_k=1^K|ℳ^k| ≥∑_k=1^K|ℳ^k|.



    

    

    
    We can extend the notation of the edges e to the higher-order case e^k to represent the existence of a non-zero value for the k-th order partial derivative ∂^k m_i/∂ z_j^k. Under the weakly diverse lossy transition assumption, it is always possible to find a region where the influence in e^k cannot be canceled in ê^k. In this region, the mapping from z_i to ẑ_i can be treated as a component-wise transformation, since the influence of 𝐳 other than z_i is zero due to the lossy transition assumption. It is important to note that there is also an indexing permutation issue between z_i and ẑ_i; the same argument in the permutation indexing part of the proof of lemma <ref> applies.

    Since ℳ^k represents the support of the k-th order partial derivative, this implies that [ℳ^k]_i,j =1 implies [ℳ^k']_i,j =1 for all k' ≤ k. We aim to show that if for the transition behind edge z_t,j→ z_t+1,i, there exists a K such that [ℳ^k_u]_i,j are different for two domains, then one of them must be a polynomial with order K-1. For this domain, [ℳ^k_u]_i,j = 1 when k = 1, 2, …, K-1 and [ℳ^k_u]_i,j = 0 when k ≥ K.

    To demonstrate that the non-decreasing complexity holds, we need to show that after an invertible transformation h to obtain the estimated version, [ℳ^k_u]_i,j cannot be zero for k < K-1, which can be shown with the following proposition.


    
        Suppose f is a polynomial of order k with respect to x. Then, for any invertible smooth function h, the transformed function f̂ h^-1∘ f ∘ h cannot be expressed by a polynomial of order k', when k' < k.
    
    
        
        Let's prove it by contradiction. Suppose f̂ h^-1∘ f ∘ h can be expressed as a polynomial of order k' < k. It follows that the function f̂(x) = C has k' roots (repeated roots are allowed), since h is invertible. Therefore, h ∘f̂(x) = h(C) also has the same number of k' roots. By definition, h ∘f̂ = f ∘ h, which means f ∘ h = h(C) has k' roots. However, since h is invertible, or equivalently it is monotonic, the equation f ∘ h = h(C) having k' roots implies that f(x) = C' has roots k'. Yet, since f is a polynomial of order k, it must have k roots, contradicting the fundamental theorem of algebra, which means that they cannot have the same number of roots. Hence, the proposition holds.
    

    The advantage of support matrix analysis is that, provided there exists at least one region where the support matrix is non-zero, the global version on the entire space will also be non-zero. Based on the definition of diverse lossy transition in Def. <ref>, it is always possible to identify such a region where for an edge z_t,i→ z_t+1,j, the mapping from z_t+1,j to z_t+1,j can be treated as a component-wise relationship. This is because no other variables besides z_t+1,j change in conjunction with z_t,i to cancel the effect. Therefore, proposition <ref> applies, and as a result, the complexity is nondecreasing. Thus, the lemma is proved.



With this lemma, we have shown that for an arbitrary incorrect domain partition result, the induced ground-truth transition complexity is preserved after the invertible transformation 𝐡. This partition effectively combines two regions, as illustrated in Figures <ref> and <ref>. Consequently, the transition complexity has the following relationship:

    ℳ^k_û = ℳ^k_u + ℳ^k_u'.

Here, u and u' represent the ground-truth values of the domain variables, and û denotes the estimated version, defined as the binary addition of the two ground truths.

By assumption <ref>, the two ground truth transitions' complexity ℳ^k_u, ℳ^k_u' are different, then with the same arguments in the proof of the lemma <ref>, we can show that the expected transition complexity with wrong domain assignment over the whole dataset is strictly larger than the ground truth complexity with correct domain assignment. And it is easy to see that when the estimated latent variables are equal to the ground truth, 𝐳̂_t = 𝐳_t then the lower bound is achieved when the estimated domains are accurate. Note that this argument is not a sufficient condition to say that the estimated 𝐳̂_t is exactly the ground truth 𝐳_t or an optimal estimation of it, since there can be other formats of mapping from 𝐳_t to 𝐳̂_t that generate the same complexity. But this is sufficient to prove that by pushing the complexity to small, the domain variables u_t must be recovered up to label swapping. This concludes the proof.




 §.§ Proof of Theorem <ref>


    Suppose that the data 𝒟 is generated from the nonstationary data generation process (<ref>), (<ref>), which satisfies the conditions in Theorem <ref> and Lemma <ref>, then the domain variables u_t are identifiable up to label swapping (Def. <ref>) and latent variables 𝐳_t are identifiable up to permutation and a component-wise transformation (Def. <ref>).


From Theorem <ref>, the domain variables u_t are identifiable up to label swapping, and then use the estimated domain variables in Lemma <ref>, the latent causal processes are also identifiable, that is, 𝐳_t are identifiable up to permutation and a component-wise transformation.




 §.§ Discussion on Assumptions



  §.§.§ Mechanism Separability

Note that we assume that there exists a ground truth mapping 𝒞: 𝒳×𝒳→𝒰, gives a domain index based on 𝐱_t-1, 𝐱_t. The existence of such mapping means that the human can tell what the domain is based on two consecutive observations. If two observations are not sufficient, then it can be modified to have more observation steps as input, for example 𝐱_≤ t or even full sequence 𝐱_[1:T]. If the input has future observation, which means that 𝐱_>t is included, then this is only valid for sequence understanding tasks in which the entire sequence will be visible to the model when analyzing the time step t. For prediction tasks or generation tasks, further assumptions on 𝒞 such as the input only contains 𝐱_<t should be made, which will be another story. Those variants are based on specific application scenarios and not directly affect our theory, for brevity, let us assume the two-step case.



  §.§.§ Mechanism Sparsity

This is a rather intuitive assumption in which we introduce some form of sparsity in the transitions, and our task is to ensure that the estimated transition maintains this sparsity pattern. This requirement is enforced by asserting an equal or lower transition complexity as defined in Assumption <ref>. Similar approaches, grounded in the same intuition, are also explored in the reinforcement learning setting, as discussed in works by Lachapelle et al. <cit.> and Hwang et al. <cit.>. The former emphasizes the identifiability result of the independent components, which necessitates additional assumptions. In contrast, the latter focuses on the RL scenario, requiring the direct observation of the latent variables involved in the dynamics, which leaves significant challenges in real-world sequence-understanding tasks, where the states are latent. And it is also extensively discussed in the nonlinear ICA literature <cit.>, in which such a sparsity constraint was added to the mixing function.



  §.§.§ Mechanism Variability

The assumption of mechanism variability requires that causal dynamics differ between domains, which requires at least one discrete edge variation within the causal transition graphs. This assumption is typically considered reasonable in practical contexts; humans identify distinctions between domains only when the differences are substantial, which often involves the introduction of a new mechanism or the elimination of an existing one. Specifically, this assumption requires a minimal alteration, a single edge change in the causal graph, to be considered satisfied. Consequently, as long as there are significant differences in the causal dynamics among domains, this criterion is fulfilled.




  §.§.§ Mechanism Function Variability

r5.5cm


    
    < g r a p h i c s >

    Correct domain separation 𝒞 and incorrect domain separation 𝒞̂ of 𝐳_t+1, given a fixed 𝐳_t.
    
    


In this section, we will further discuss the mechanism function variability introduced in Corollary <ref>. One might question the necessity of this assumption. To illustrate this issue, we claim that if we only assume that the mechanism functions differ across domains but without this extended version of the variability assumption, i.e., for u ≠ u', 𝐦_u ≠𝐦_u', then under this proposed framework, the domain variables u_t are generally unidentifiable.

In Figure <ref>, we present a simple example of the space of 𝐳_t+1 given a fixed 𝐳_t. For the sake of brevity, assume that there are two domains. By the mechanism separability assumption <ref>, the space 𝒵 of 𝐳_t+1 is divided into two distinct parts, each corresponding to one domain. In this illustration, 𝒞 denotes the partition created by the ground truth transition function:

    𝐳_t+1 = 
    𝐦_1(𝐳_t,ϵ)    if  u_t+1 = 1, 
    𝐦_2(𝐳_t,ϵ)    if  u_t+1 = 2.


Then the question arises: when the domain assignment is incorrect, that is, 𝒞̂≠𝒞, can we still get the same observational distributions, or equivalently, can we obtain the same distribution for 𝐳_t+1?

The answer is yes. For the ground truth transition, 𝐦_1(𝐳_t, ϵ) ∈ A ∪ C and 𝐦_2(𝐳_t, ϵ) ∈ B ∪ D. In the case of an incorrect partition 𝒞̂, it is sufficient to have 𝐦̂_1(𝐳_t, ϵ) ∈ A ∪ B and 𝐦̂_2(𝐳_t, ϵ) ∈ C ∪ D. Ensuring that the conditional distribution p(𝐳_t+1|𝐳_t) is matched everywhere, we can create two different partitions on domains, yet still obtain exactly the same observations. That makes the domain variables u_t unidentifiable in the general case.





  
How does the previous mechanism variability assumption work?
In the assumption of mechanism variability (Assumption <ref>), the support matrices of the Jacobian of transitions across different domains differ. Consider a scenario where the ground truth partition is 𝒞, denoted by A, C | B, D. If an incorrect estimation occurs, where our estimated partition is 𝒞̂, represented as A, B | C, D, then the estimated transition in domain one should cover the transitions in both A and B, and similarly for the second domain. This leads to an increase in complexity within the estimated Jacobian support matrix, as discussed in the previous sections. Consequently, this complexity forces the sets B and C to be empty, resulting in 𝒞̂ converging to 𝒞.



  
How about mechanism function variability?
Roughly speaking, and as demonstrated in our experiments, the mechanism variability assumption previously discussed is already sufficient to identify domain changes in both synthetic and real-world settings. This sufficiency arises because the assumption only requires a single differing spot, even though some transition functions behind some edges may persist across different domains. As long as there is one edge spot that can separate the two domains, this condition is met. In the relatively rare case where all edges in the causal dynamic transition graphs are identical across two different domains and only the underlying functions differ, we can still demonstrate identifiability in this scenario by examining differences in the support of the higher-order partial derivative matrices.



  §.§.§ Weakly Diverse Lossy Transition

The weakly diverse lossy transition assumption requires that each variable in the latent space can potentially influence a set of subsequent latent variables, and such transformations are typically non-invertible. This implies that given the value of 𝐳_t+1, it is generally challenging to precisely recover the previous 𝐳_t; equivalently, this mapping is not injective. Although this assumption requires some explanation, it is actually considered mild in practice. Often in real-world scenarios, different current states may lead to identical future states, indicating a loss of information. The “weakly diverse” of this assumption suggests that the way information is lost varies between different dimensions, but there is some common part among them, hence the term “weakly diverse”. In the visualization example shown in Figure <ref>, we can clearly see this pattern, in which the scene is relatively simple and it is very likely that in two different frames, the configuration of the scene or the value of the latent variables are the same but their previous states are completely different. 




§ EXPERIMENT SETTINGS




 §.§ Synthetic Dataset Generation

r5.8cm


Synthetic Dataset Statistics


Property     Value 
 
Number of State     5 
 
Dimension of 𝐳_t     8 
 
Dimension of 𝐱_t     8 
 
Number of Samples     32,000 
 
Sequence Length     15 
 




The synthetic dataset is constructed in accordance with the conditions outlined in Theorems <ref> and <ref>. Transition and mixing functions are synthesized using multilayer perceptrons (MLPs) initialized with random weights. The mixing functions incorporate the LeakyReLU activation function to ensure invertibility. The dataset features five distinct values for the domain variables, with both the hidden variables 𝐳_t and the observed variables 𝐱_t set to eight dimensions. A total of 1,000 sequences of domain variables were generated. These sequences exhibit high nonstationarity across domains, which cannot be captured with a single Markov chain. This was achieved by initially generating two distinct Markov chains to generate two sequences of domain indices. Subsequently, these sequences were concatenated, along with another sequence sampled from a discrete uniform distribution over the set {1, 2, 3, 4, 5}, representing the domain indices.



For each sequence of domain variables, we sampled a batch size of 32 sequences of hidden variables 𝐳_t beginning from a randomly initialized initial state 𝐳_0. These sequences were generated using the randomly initialized multilayer perceptron (MLP) to model the transitions. Observations 𝐱_t were subsequently generated from 𝐳_t using the mixing function as specified in Eq. <ref>. Both the transition functions in the hidden space and the mixing functions were shared across the entire dataset. A summary of the statistics for this synthetic dataset is provided in Table <ref>. For detailed implementation of this data generation process, please refer to our accompanying code in Sec. <ref>.



 §.§ Real-world Dataset



  
Hollywood Extended <cit.>
The Hollywood dataset contains 937 video clips with a total of 787,720 frames containing sequences of 16 different daily actions such as walking or sitting from 69 Hollywood movies. On average, each video comprises 5.9 segments, and 60.9% of the frames are background.



  
CrossTask <cit.>
The CrossTask dataset features videos from 18 primary tasks. According to <cit.>, we use the selected 14 cooking-related tasks, including 2552 videos with 80 action categories. On average, each video in this subset has 14.4 segments, with 74.8% of the frames classified as background.



 §.§ Mean Correlation Coefficient


MCC, a standard metric in the ICA literature, is utilized to evaluate the recovery of latent factors. This method initially computes the absolute values of the correlation coefficients between each ground truth factor and every estimated latent variable. Depending on the presence of component-wise invertible nonlinearities in the recovered factors, either Pearson’s correlation coefficients or Spearman’s rank correlation coefficients are employed. The optimal permutation of the factors is determined by solving a linear sum assignment problem on the resultant correlation matrix, which is executed in polynomial time.





§ IMPLEMENTATION DETAILS



 §.§ Prior Likelihood Derivation

Let us start with an illustrative example of stationary latent causal processes consisting of two time-delayed latent variables, i.e., 𝐳_t = [z_1,t, z_2,t], i.e., z_i,t = m_i(𝐳_t-1, ϵ_i,t) with mutually independent noises, where we omit the u_t since it is just an index to select the transition function m_i. Let us write this latent process as a transformation map 𝐦 (note that we overload the notation m for transition functions and for the transformation map):


    [ z_1,t-1; z_2,t-1;   z_1,t;   z_2,t;         ] 
        =𝐦(
        [ z_1,t-1; z_2,t-1;   ϵ_1,t;   ϵ_2,t ]).


By applying the change of variables formula to the map 𝐦, we can evaluate the joint distribution of the latent variables p(z_1,t-1, z_2,t-1, z_1,t,z_2,t) as:

    p(z_1,t-1, z_2,t-1, z_1,t,z_2,t) = p(z_1,t-1, z_2,t-1, ϵ_1,t,ϵ_2,t) / |𝐉_𝐦|,

where 𝐉_𝐦 is the Jacobian matrix of the map 𝐦, which is naturally a low-triangular matrix:

    𝐉_𝐦 = 
    [                 1                 0                 0                 0;                 0                 1                 0                 0; ∂ z_1,t/∂ z_1,t-1 ∂ z_1,t/∂ z_2,t-1    ∂ z_1,t/∂ϵ_1,t                 0; ∂ z_2,t/∂ z_1,t-1 ∂ z_2,t/∂ z_2,t-1                 0    ∂ z_2,t/∂ϵ_2,t ].

Given that this Jacobian is triangular, we can efficiently compute its determinant as ∏_i ∂ z_i,t/∂ϵ_i,t. Furthermore, because the noise terms are mutually independent, and hence ϵ_i,t⊥ϵ_j,t for j ≠ i and ϵ_t ⊥𝐳_t-1, we can write the RHS of Eq. <ref> as:


    p(z_1,t-1, z_2,t-1, z_1,t,z_2,t)    = p(z_1,t-1, z_2,t-1) × p(ϵ_1,t,ϵ_2,t) / |𝐉_𝐦|    (because ϵ_t ⊥𝐳_t-1)
       = p(z_1,t-1, z_2,t-1) ×∏_i p(ϵ_i,t) / |𝐉_𝐦|    (because ϵ_1,t⊥ϵ_2,t)


Finally, by canceling out the marginals of the lagged latent variables p(z_1,t-1, z_2,t-1) on both sides, we can evaluate the transition prior likelihood as:


    p( z_1,t,z_2,t| z_1,t-1, z_2,t-1) = ∏_i p(ϵ_i,t) / |𝐉_𝐦| = ∏_i p(ϵ_i,t) ×|𝐉_𝐦^-1|.


Now we generalize this example and derive the prior likelihood below. 

Let {m̂^-1_i}_i=1,2,3... be a set of learned inverse transition functions that take the estimated latent causal variables, and output the noise terms, i.e., ϵ̂_i,t = m̂^-1_i(u_t, ẑ_i,t, 𝐳̂_t-1). 

Design transformation 𝐀→𝐁 with  low-triangular Jacobian as follows:

    [ 𝐳̂_t-1,
    𝐳̂_t ]^⊤_𝐀 mapped to [ 𝐳̂_t-1,
    ϵ̂_t ]^⊤_𝐁,
     
    with 𝐉_𝐀→𝐁 = 
    [                    𝕀_n                      0;                        diag(∂ m^-1_i,j/∂ẑ_jt) ].

Similar to Eq. <ref>, we can obtain the joint distribution of the estimated dynamics subspace as:



    log p(𝐀) = log p(𝐳̂_t-1) + ∑_j=1^n log p(ϵ̂_i,t)_Mutually independent noise + log(|(𝐉_𝐀→𝐁) |) .
    log p(𝐳̂_t |𝐳̂_t-1, u_t) = ∑_i=1^n log p(ϵ̂_i,t| u_t)+ ∑_i=1^n log| ∂ m^-1_i/∂ẑ_i,t|.




 §.§ Derivation of ELBO

Then the second part is to maximize the Evidence Lower BOund (ELBO) for the VAE framework, which can be written as:


    ELBO≜   log p_data({𝐱_t}_t=1^T) - D_KL(q_ϕ({𝐳_t}_t=1^T|{𝐱_t}_t=1^T)|| p_data({𝐳_t}_t=1^T|{𝐱_t}_t=1^T))
    
    =   𝔼_𝐳_tlog p_data({𝐱_t}_t=1^T|{𝐳_t}_t=1^T) - D_KL(q_ϕ({𝐳_t}_t=1^T|{𝐱_t}_t=1^T)|| p_data({𝐳_t}_t=1^T|{𝐱_t}_t=1^T))
    
    =   𝔼_𝐳_tlog p_data({𝐱_t}_t=1^T|{𝐳_t}_t=1^T) - 𝔼_𝐳_t(log q_ϕ({𝐳_t}_t=1^T|{𝐱_t}_t=1^T) - log p_data({𝐳_t}_t=1^T) ]
    
    =   𝔼_𝐳_t(log p_data({𝐱_t}_t=1^T|{𝐳_t}_t=1^T) + log p_data({𝐳_t}_t=1^T) - log q_ϕ({𝐳_t}_t=1^T|{𝐱_t}_t=1^T)  )
    
    =   𝔼_𝐳_t(∑_t=1^Tlog p_data(𝐱_t|𝐳_t)_-ℒ_Recon 
    + ∑_t=1^Tlog  p_data(𝐳_t|𝐳_t-1,u_t) 
    - ∑_t=1^Tlog q_ϕ(𝐳_t|𝐱_t)_-ℒ_KLD)




 §.§ Reproducibility

All experiments are performed on a GPU server with 128 CPU cores, 1TB memory, and one NVIDIA L40 GPU. For synthetic experiments, we run the baseline methods with implementation from <https://github.com/weirayao/leap> and <https://github.com/xiangchensong/nctrl>.
For real-world experiments, the implementation is based on <https://github.com/isee-laboratory/cvpr24_atba>.

Our code is also available via <https://github.com/xiangchensong/ctrlns>.



 §.§ Hyperparameter and Train Details

For synthetic experiments, the models were implemented in . We trained the VAE network using the AdamW optimizer with a learning rate of 5 × 10^-4 and a mini-batch size of 64. Each experiment was conducted using three different random seeds and we reported the mean performance along with the standard deviation averaged across these seeds. The coefficient for the L_2 penalty term was set to 1 × 10^-4, which yielded satisfactory performance in our experiments. We also tested L_1 penalty or L_2 penalty with larger coefficients, the setting we used in this paper provided the best stability and performance.
All other hyperparameters of the baseline methods follow their default values from their original implementation. For real-world experiments, we follow the same hyperparameter setting from the baseline ATBA method. In the Hollywood dataset, we used the default 10-fold dataset split setting and calculated the mean and standard derivation from those 10 runs. For the CrossTask dataset, we calculate the mean and standard derivation using five different random seeds.




§ VISUALIZATION ON ACTION SEGMENTATION

We visualize some examples from the Hollywood dataset. As shown in Figure <ref> we can see that our  can estimate the actions more accurate than baseline method.




§ EXTENDED RELATED WORK




 §.§ Causal Discovery with Latent Variables

Various studies have focused on uncovering causally related latent variables. For example, <cit.> use vanishing Tetrad conditions <cit.> or rank constraints to detect latent variables in linear-Gaussian models, whereas <cit.> rely on non-Gaussianity in their analyses of linear, non-Gaussian models.
Additionally, some methods seek to identify structures beyond latent variables, leading to hierarchical structures.
Certain hierarchical model-based approaches assume tree-like configurations, as seen in <cit.>, while other methods consider a more general hierarchical structure <cit.>.
Nonetheless, these approaches are restricted to linear frameworks and encounter increasing difficulties with complex datasets, such as videos.




 §.§ Causal Temporal Representation Learning

In the context of sequence or time series data, recent advances in nonlinear Independent Component Analysis (ICA) have leveraged temporal structures and nonstationarities to achieve identifiability. Time-contrastive learning (TCL) <cit.> exploits variability in variance across data segments under the assumption of independent sources. Permutation-based contrastive learning (PCL) <cit.> discriminates between true and permuted sources using contrastive loss, achieving identifiability under the uniformly dependent assumption. The i-VAE <cit.> uses Variational Autoencoders to approximate the joint distribution over observed and nonstationary regimes. Additionally, (i)-CITRIS <cit.> utilizes intervention target information to identify latent causal factors. Other approaches such as LEAP <cit.> and TDRL <cit.> leverage nonstationarities from noise and transitions to establish identifiability. CaRiNG <cit.> extended TDRL to handle non-invertible generation processes by assuming sequence-wise recoverability of the latent variables from observations. 

All the aforementioned methods either assume stationary fixed temporal causal relations or that the domain variables controlling the nonstationary transitions are observed. To address unknown or unobserved domain variables, HMNLICA <cit.> integrates nonlinear ICA with a hidden Markov model to automatically model nonstationarity. However, this method does not account for the autoregressive latent transitions between latent variables over time. IDEA <cit.> combines HMNLICA and TDRL by categorizing the latent factors into domain-variant and domain-invariant groups. For the variant variables, IDEA adopts the same Markov chain model as HMNLICA, while for the invariant variables, it reduces the model to a stationary case handled by TDRL. Both iMSM <cit.> and NCTRL <cit.> extend this Markov structure approach by incorporating transitions in the latent space but continue to assume that the domain variables follow a Markov chain.




 §.§ Weakly-supervised Action Segmentation

Weakly-supervised action segmentation techniques focus on dividing a video into distinct action segments using training videos annotated solely by transcripts <cit.>. Although these methods have varying optimization objectives, many employ pseudo-segmentation for training by aligning video sequences with transcripts through techniques like Connectionist Temporal Classification (CTC) <cit.>, Viterbi <cit.>, or Dynamic Time Warping (DTW) <cit.>. For instance, <cit.> extends CTC to consider visual similarities between frames while evaluating valid alignments between videos and transcripts. Drawing inspiration from speech recognition, <cit.> utilize the Hidden Markov Model (HMM) to link videos and actions. <cit.> initially produces uniform segmentations and iteratively refines boundaries by inserting repeated actions into the transcript. <cit.> introduces an alignment objective based on explicit context and length models, solvable via Viterbi, to generate pseudo labels for training a frame-wise classifier. Similarly, <cit.> and <cit.> propose novel learning objectives but still rely on Viterbi for optimal pseudo segmentation. Both <cit.> use DTW to align videos to both ground-truth and negative transcripts, emphasizing the contrast between them. However, except for <cit.>, these methods require frame-by-frame calculations, making them inefficient. More recently, alignment-free methods have been introduced to enhance efficiency. <cit.> learns from the mutual consistency between frame-wise classification and category/length pairs of a segmentation. <cit.> enforces the output order of actions to match the transcript order using a novel loss function. Although POC <cit.> is primarily set-supervised, it can be extended to transcript supervision, making its results relevant for comparison.




§ LIMITATIONS

As noted in Sec. <ref>, our main theorem relies on the condition that causal graphs among different domains must be distinct. Although our experiments indicate that this assumption is generally sufficient, there are scenarios in which it may not hold, meaning that the transition causal graphs are identical for two different domains, but the actual transition functions are different. We have addressed this partially through an extension to the mechanism variability assumption to higher-order cases (Corollary <ref>). However, dealing with situations where transition graphs remain the same across all higher orders remains a challenge. We acknowledge this as a limitation and suggest it as an area for future exploration.

We also observed that the random initialization of the nonlinear ICA framework can influence the total number of epochs needed to achieve identifiability, as illustrated in Figure <ref>. Also, for the computational efficiency, the TDRL framework we adopted involves a prior network that calculated each dimension in the latent space one by one, thus making the training efficiency suboptimal. Since this is not directly related to major claim which is our sparse transition design, we acknowledge this as a limitation and leave it for future work.



§ BOARDER IMPACTS

This work proposes a theoretical analysis and technical methods to learn the causal representation from time-series data, which 
facilitate the construction of more transparent and interpretable models to understand the causal effect in the real world. 
This could be beneficial in a variety of sectors, including healthcare, finance, and technology.
In contrast, misinterpretations of causal relationships could also have significant negative implications in these fields, which must be carefully done to avoid unfair or biased predictions.


