Paper Title: Vision-Language Model Based Handwriting Verification.










Vision-Language Model Based Handwriting Verification
    
    Mihir Chauhan††   Abhishek Satbhai§   Mohammad Abuzar Hashemi¶   Mir Basheer Ali¶¶   Bina Ramamurthy‡   Mingchen Gao||   Siwei Lyu†   Sargur Srihari‡‡
 
    
    Department of Computer Science and Engineering 
 
    The State University of New York, Buffalo, NY, USA 
 
    {mihirhem††, bina‡, mgao8||, siweilyu†, srihari‡‡}@buffalo.edu 

    {ma.hashemi.786¶, abhishek07satbhai§, alimirbasheer¶¶}@gmail.com

    
=================================================================================================================================================================================================================================================================================================================================================================================================================================

empty



§ ABSTRACT 
Handwriting Verification is a critical in document forensics. Deep learning based approaches often face skepticism from forensic document examiners due to their lack of explainability and reliance on extensive training data and handcrafted features. This paper explores using Vision Language Models (VLMs), such as OpenAI’s GPT-4o and Google's PaliGemma, to address these challenges. By leveraging their Visual Question Answering capabilities and 0-shot Chain-of-Thought (CoT) reasoning, our goal is to provide clear, human-understandable explanations for model decisions. Our experiments on the CEDAR handwriting dataset demonstrate that VLMs offer enhanced interpretability, reduce the need for large training datasets, and adapt better to diverse handwriting styles. However, results show that the CNN-based ResNet-18 architecture outperforms the 0-shot CoT prompt engineering approach with GPT-4o (Accuracy: 70%) and supervised fine-tuned PaliGemma (Accuracy: 71%), achieving an accuracy of 84% on the CEDAR AND dataset. These findings highlight the potential of VLMs in generating human-interpretable decisions while underscoring the need for further advancements to match the performance of specialized deep learning models. Our code is publicly available at: <https://github.com/Abhishek0057/vlm-hv>


Keywords: Vision Language Model, Machine Vision, Handwriting Verification, Forensics








§ INTRODUCTION


Handwriting verification is crucial in pattern recognition and biometrics, focusing on authenticating and identifying individuals by their handwriting. It is vital in forensics, where experts analyze samples to verify documents, identify forgeries, and provide court evidence.

Historical Perspective: Handwriting Verification methods <cit.> <cit.> <cit.> have evolved from handcrafted features like GSC <cit.> to deep CNNs such as ResNet-18 <cit.>, and now Vision Transformers <cit.>, enabling comparison of both inter- and intra- writer variations in handwriting styles. 

Challenges: Despite the advancements, Forensic Document Examiners (FDE) remain skeptical due to the lack of interpretability in the model's decision making processes. Morever, these methods heavily depend on large datasets of labeled handwritten images, making dataset collection (X(x_q,x_k, y) with known x_k, questioned x_q handwritten samples and corresponding writer labels y) is costly and time-intensive. 


Why use VLM? Vision-Language Models (VLMs) integrate image and textual data to encode complex relationships between visual and linguistic information, enabling Forensic Document Examiners (FDEs) to interpret model decisions with clear, natural language explanations that enhance trust and reliability in forensic applications. They can adapt to the forensics domain in zero-shot scenarios without training examples and in few-shot scenarios with minimal examples by leveraging transfer learning capabilities, having been pre-trained on multiple tasks related to handwritten images and natural language understanding. Fine-tuning them with forensic-specific datasets further enhances their performance and applicability in real-world forensic use-cases. 

VLMs applied to Forensics: In <cit.>, the authors explore the capabilities of multimodal Large Language Models (LLMs) like GPT-4 in various digital forensics use-cases, including artifact understanding, evidence searching, anomaly detection, and education. More specific studies, such as <cit.>, demonstrate the effectiveness of GPT-4 specifically for the media forensics task of DeepFake image detection. Our research applies VLMs to handwriting verification. To our knowledge, we are the first to explore VLMs for the forensic task of handwriting comparison.












§ METHODS

Our study aims to use VLM in handwriting verification. We have chosen OpenAI’s GPT-4o VLM for its strong Visual Question Answering (VQA) capabilities. Using their API, we prompt GPT-4o <cit.> with 0-shot Chain-of-Thought (CoT) reasoning. This approach allows us to first generate human-interpretable explanations and then determine whether the given questioned and known handwritten samples were written by the same person or by different writers, as shown in Figure <ref>. Since fine-tuning GPT-4o is not generally available, we compared its performance to a recent open-source VLM, PaliGemma <cit.>, using 0-shot prompt engineering and parameter-efficient supervised fine-tuning (PEFT) on a training dataset with 100 examples.  

Data: The experiments were conducted using 1,000 sample pairs of known and questioned images from CEDAR Letter <cit.> and CEDAR AND <cit.> dataset. CEDAR Letter dataset contains a letter manuscript written by 1568 writers three times. CEDAR AND dataset is a subset of CEDAR Letter data which only contains the lowercase handwritten word “and” extracted from full letter manuscript. The evaluation dataset includes 1,000 sample pairs of known and questioned images written by 368 writers with writer ids greater than 1200 (writer ids below 1200 were used to train the baseline models). Of these pairs, 500 are from the same writer and 500 are from different writers.






§ EXPERIMENTS & RESULTS

CEDAR AND Baselines: As shown in Figure <ref> we use handcrafted features GSC features, CNN based ResNet-18 <cit.> and MaskedCausalVisionTransformer <cit.> (ViT) as our baseline feature extractors. GSC are 512-dimensional features extracted for the binarized “AND” images. All the three baselines were trained on 10% and 100% of known and questioned training pairs writers with writer ids less than 1200 resulting in 13,232 and 129,602 number of train pairs as shown in Table <ref>. The output of these feature extractors is fed into 2 fully-connected (FC) layers with 256 and 128 hidden neurons with ReLU activations. The final layer has 2 output neurons whose softmax activations represent similarity of samples with a one hot vector representation. We use categorical cross entropy loss given one-hot encoded logits compared to the target which is binary. 



Prompt Engineering with GPT-4o and PaliGemma: To effectively utilize the GPT-4o VLM for handwriting verification, we experimented with various prompts to optimize both the generation of human-interpretable explanations and the accuracy of verification decisions. Initially, we crafted prompts that directed the model to compare specific features of the handwriting samples, such as stroke width, slant, and letter spacing. For example, prompts like "Describe the similarities and differences in the stroke patterns between the two samples" and "Identify any matching unique characteristics in the handwriting" were used. However, we observed significant variance in the model’s responses, leading to inconsistent results. To address this, we adopted a Chain-of-Thought (CoT) reasoning approach. This method guides the model through a structured reasoning process, allowing it to generate more consistent and reliable explanations. Since no examples were provided to VLM in the prompt engineering phase, the number of training pairs is observed as 0 in Table <ref>. 



Additionally, we utilized prompt engineered both VLMs to identify and mark coordinates of similarities and dissimilarities within the two images. By prompting the model to highlight specific areas of interest, such as matching or differing stroke patterns, we were able to generate detailed coordinates that pinpointed these features as shown in Figure <ref>. This approach provided clear visual indicators of the reasoning behind each verification decision. By incorporating CoT reasoning, presenting known and questioned samples as different images, and using VLM to identify key coordinates, we achieved a balance where the model could generate detailed, interpretable explanations while accurately determining the authenticity of the handwriting samples.

Supervised Fine-Tuning with PaliGemma: 
We fine-tuned the pre-trained PaliGemma model on 100 curated examples of 0-shot CoT prompt engineered PaliGemma results. The PaliGemmaProcessor within Transformers library was loaded to process the inputs which involved the preparation of prompt templates and batching text inputs with images. The image tokens and pad tokens were set to -100 to be ignored by the model, and these preprocessed inputs were used as labels for the model to learn from. The model, PaliGemmaForConditionalGeneration, was loaded and set to fine-tune only the text decoder, leaving the vision encoder and multimodal projector frozen. We used BitsAndBytes config options with 4bit quantization and LoRA <cit.> to train only 11M parameters out of the 3B parameters of the network. The results of supervised fine-tuned PaliGemma on both CEDAR AND and Letter datasets are shown in Table <ref>.

Results: The results of using VLMs for handwriting verification are presented in Table <ref> on the CEDAR AND and CEDAR Letter datasets. For the CEDAR AND dataset, CNN based ResNet-18 architecture outperforms 0-shot CoT prompt engineering approach with GPT-4o as well as supervised-finetuned PaliGemma with 100% training pairs which suggest that although VLMs has a lot of potential for generating human interpretable decision making for the task of handwriting verification, still lags behind CNNs which aare fine-tuned on specific task of handwriting verification. This gap highlights the need for further improvements in fine-tuning regime of VLMs to enhance their effectiveness and reliability for specialized tasks like handwriting verification. For the CEDAR Letter dataset, the 0-shot CoT prompt engineering with GPT-4o also shows relatively higher performance than the supervised fine-tuning of PaliGemma but we observe high variability in the performance metrics because of low sample size. 



§ CONCLUSION

Our study explores the application of VLMs, specifically GPT-4o and PaliGemma, in the domain of handwriting verification. By leveraging the robust VQA capabilities of these models and employing 0-shot CoT reasoning through prompt engineering, we aimed to generate human-interpretable explanations for model decisions. Our experiments demonstrated that while VLMs offer significant improvements in interpretability and adaptability to diverse handwriting styles, they currently lag behind CNN-based architectures such as ResNet-18 in terms of performance. Specifically, ResNet-18 achieved an accuracy of 84% on the CEDAR AND dataset, outperforming GPT-4o's 70% accuracy and PaliGemma's 71% accuracy. These findings suggest that while VLMs hold great promise for enhancing transparency and trustworthiness in forensic handwriting verification, there is still a need for further advancements in their fine-tuning regimes to improve their effectiveness and reliability for specialized tasks. Moving forward, we aim to work with forensic document examiners to create a fine-tuning dataset of explanation reports using text and visual information to ensure the applicability of our approach in practical forensic settings.






apalike





 Paper Title: Persistent de Rham-Hodge Laplacians in the Eulerian representation.






 
 
 


Recently, topological data analysis (TDA) has become a trending topic in data science and engineering. However, the key technique of TDA, i.e., persistent homology, is defined on point cloud data, which restricts its scope. In this work, we propose persistent de Rham-Hodge Laplacian, or persistent Hodge Laplacian (PHL) for abbreviation, for the TDA on manifolds with boundaries, or volumetric data.         
Specifically, we extended the evolutionary de Rham-Hodge theory from the Lagrangian formulation to the Eulerian formulation via structure-persevering Cartesian grids, and extended the persistent  Laplacian on point clouds to persistent (de Rham-)Hodge Laplacian on nested families of manifolds with appropriate boundary conditions. 
The proposed PHL facilitates the machine learning and deep learning prediction of volumetric data. For a proof-of-principle application of the proposed PHL, we propose a persistent Hodge Laplacian learning (PHLL) algorithm for data on manifolds or volumetric data. To this end, we showcase the PHLL prediction of protein-ligand binding affinities in two benchmark datasets.  Our numerical experiments highlight the power and promise of PHLL. 








Persistent de Rham-Hodge Laplacians in the Eulerian representation
    Zhe Su^1, Yiying Tong^2,
[Corresponding author.		Email: ytong@msu.edu]
and Guo-Wei Wei^1,3,4 
[Corresponding author.		Email: weig@msu.edu]


^1Department of Mathematics,
 Michigan State University, East Lansing, MI 48824, USA

^2Department of Computer Science and Engineering,
 Michigan State University,
East Lansing, MI 48824, USA

^3Department of Biochemistry and Molecular Biology,
 
Michigan State University, East Lansing, MI 48824, USA

^4Department of Electrical and Computer Engineering, 

Michigan State University, East Lansing, MI 48824, USA
    Accepted for publication in Astrophysics & Space Sciences (ApSS)
=============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





§ INTRODUCTION


Recent years have witnessed a fast growth of topological data analysis (TDA) in data science and engineering <cit.>. The growth is driven by the great promise of topological approaches to real-world data that are distinguished from any other statistical, mathematical, physical, and engineering methods. Typically, TDA offers a multiscale topological characterization of data, which is the case with persistent homology <cit.>, a key method employed in TDA. A major feature of persistent homology is its multiscale analysis, which creates a family of topological spaces from the original data to track the topological persistence, i.e., the lifespan of topological invariants across scales. The other major feature of persistent homology is its topological description of a space (like connected components, loops, and voids) in terms of topological invariants, such as Betti numbers.  As such, persistent homology-based TDA  leads to much topological simplification of the geometric information in the data. Consequently, TDA typically works extremely well for data with intricate complexity. Unfortunately, for data without geometric complex, TDA may give rise to an oversimplification or neglect key geometric characteristics, leading to a less competitive approach. 

For many years, persistent homology has been used in qualitative analysis, which is somewhat counterintuitive and unproductive for non-experts. The power of persistent homology was not demonstrated until it was utilized in quantitative and predictive analysis via machine learning algorithms <cit.>. 
Topological deep learning (TDL) was introduced to deal with large and intrinsically complex datasets <cit.>. 
When TDL was coined in 2017, it referred to the topological featurization in deep neural networks <cit.>. More recently, simplicial neural networks and other topological neural techniques have been applied to the design of neural network architecture, making TDL an emerging paradigm in data science and machine learning <cit.>. However, an increasing concern associated with this rising popularity is whether TDL brings any practical benefit beyond its mathematical elegance. There are many applications where TDL has demonstrated superiority to other competitive methods. Perhaps some of the most compelling examples are TDL's dominant wining of D3R Grand Challenges, an annual worldwide competition series in computer-aided drug design <cit.>, its discovery of the mechanisms of SARS-CoV-2 evolution <cit.>, and its successful forecast of emerging dominant SARS-CoV-2 variants BA.2 <cit.> and BA.4/BA.5 about two months in advance <cit.>.     

 It is interesting to understand why TDL (or TDA) was so successful in the aforementioned examples, but was not competitive in many other situations. First, biomolecular data, which are intricately complex in their internal structures, were involved in the above compelling examples. As such, topological simplification was a productive process, whereas TDL leads to the severe loss of crucial information in many other data that are relatively simple in their internal structures. Additionally, it was element-specific persistent homology, rather than persistent homology, that was applied in the above examples. This approach captures physical and biological interactions in the biomolecular data <cit.>.   
  In fact, in the forecast of emerging dominant SARS-CoV-2 variants BA.4/BA.5, persistent Laplacian, rather than persistent homology,  was utilized.  This happens because persistent homology has many drawbacks or limitations <cit.>. 
 First, the topological invariant extracted from persistent homology is qualitative, rather than quantitative. For example, persistent homology does not distinguish a five-number from a six-number ring.  
 Additionally, persistent homology is incapable of dealing with different elements in a point cloud, which is ineffective with the physics and chemistry of (bio)molecular data. 
 Moreover, persistent homology cannot describe non-topological changes, i.e., homotopic shape evolution during the multiscale analysis.
Further, persistent homology is incapable of handling directed networks and digraphs, such as polarization, regulation, and control issues in applications. Finally, persistent homology is unable to characterize structured data, e.g., hypergraphs, directed networks, etc. 
These challenges call for innovative topological methods. 
 
To address these challenges, two types of persistent topological Laplacians (PTLs) were introduced in 2019 <cit.>. The first type of PTL is called evolutionary de Rham-Hodge method for data on manifolds <cit.>. This approach creates a family of multiscale manifolds with boundaries from a given data and then builds Hodge Laplacian operators on the multiscale manifolds with appropriate boundary conditions. The second type of PTL is called persistent spectral graph, also known as persistent combinatorial Laplacian or persistent Laplacian defined for point cloud data <cit.>.  While originated from sharply different topological spaces <cit.>, these two PTLs share the same algebraic structure.  The harmonic spectra of PTLs fully recover the topological invariants of persistent homology. However, the nonharmonic spectra of PTLs capture the homotopic shape evolution during the multiscale analysis that cannot be done with persistent homology. Computational algorithms and mathematical analysis of persistent Laplacian have been reported <cit.>. Persistent Laplacian has been shown to outperform persistent homology in many applications <cit.>.  

In the past few years, much effort has been given to extend persistent Laplacian to further address other limitations of persistent homology on point cloud data, leading to   
persistent sheaf Laplacians<cit.>,
persistent path Laplacians, 
persistent hypergraph and hyperdigraph Laplacians<cit.>,

persistent directed flag Laplacians,

persistent Mayer Laplacians,

and persistent interaction Laplacians
<cit.>.


In addition to point cloud data and data on manifolds, there is another widely available data form, namely, curves embedded in the three-dimensional Euclidean space, such as knots, links, and tangles and their generalizations <cit.>. Evolutionary Khovanov homology has been proposed to deal with embedded curve data <cit.>. It integrates algebraic topology, geometric topology, and metric analysis for the first time. However, effective computational algorithms are to be developed for this approach to be widely used in practical applications.  
Similarly, while case studies have been shown for evolutionary de Rham-Hodge theory for data on manifolds <cit.>, the early approach based on discrete exterior calculus (DEC) <cit.> in the Lagrangian representation is not efficient for multiscale analysis and machine learning studies. Specifically, the regeneration of the evolving manifold at different scales with different simplicial meshes causes numerical inconsistencies and becomes expensive for practical applications in machine learning studies.    

The objective of this work is to develop a persistent de Rham-Hodge theory on the Euler representation. As such, it solves Hodge Laplacians on a pre-designed structure-persevering Cartesian grid for all scales to avoid numerical inconsistency. It constructs a natural mapping of differential forms from a manifold with boundary embedded in ^3 to a large manifold,  uses it to produce persistent cohomology mapping, and constructs a persistent Hodge Laplacian with built-in boundary conditions, extending the persistent combinatorial Laplacian.
Our new approach draws on differential geometry,  algebraic topology, partial differential equation (PDE), metric analysis, and numerical analysis. To give a proof-of-principle demonstration, we pair the proposed persistent de Rham-Hodge Laplacians with an effective machine learning algorithm to predict protein-ligand binding affinities. Based on two benchmark datasets, PDBbind v2007 and  PDBbind v2016, we show that our new approach gives rise to cutting-edge performance.    
The rest of this paper is organized as follows. Sec. 2 offers a primer on the de Rham-Hodge theory on manifolds with boundaries; Sec. 3  presents our discretization for evolutionary de Rham-Hodge theory based on spectrum calculation of Laplacians associated with sublevel sets on Cartesian grids; Sec. 4 presents our construction for persistent de Rham-Hodge Laplacians both in the continuous setting and for given level set functions on Cartesian grids; Sec. 5 showcases preliminary studies on applications; and Sec. 6 concludes the paper.




§ DE RHAM-HODGE THEORY


Let M be an m-dimensional smooth, orientable, compact Riemannian manifold with boundary. Denote by Ω^k(M) the space of all differential k-forms on M, i.e., the space of all smooth antisymmetric covariant tensor fields on M of degree k. The differential d, also called exterior derivative, is the unique -linear mapping from the space of k-forms Ω^k(M) to the space of (k+1)-forms Ω^k+1(M) satisfying the Leibniz rule with respect to the wedge product ∧ and the nilpotent property dd = 0.
A key property of differential forms is that they can be integrated over any orientable k-submanifolds of M. For any oriented (k+1)-submanifold S⊂ M with boundary ∂ S, Stokes' theorem, as a generalization of the Newton-Leibniz rule, states that the integral of a differential k-form ω over ∂ S is equal to the integral of its differential over S, i.e., 

    ∫_S dω= ∫_∂ Sω.

The differential d generalizes and unifies the classical operators in vector calculus, such as gradient ∇, curl ∇×, and divergence ∇· in ^2 and ^3. For instance, in ^3, 0-forms and 3-forms can be identified with scalar fields, while 1-forms and 2-forms can be identified with vector fields. In this case, the differential d corresponds to the gradient operator ∇ when applied to 0-forms, the curl operator ∇× when applied to 1-forms, or the divergence operator ∇· when applied 2-forms. The nilpotent property dd=0 directly leads to the vector field analysis identities ∇×∇=0 and ∇·∇×=0.


A differential form ω∈Ω^k(M) is called closed if dω = 0, or exact if there is a (k-1)-form ζ∈Ω^k-1(M) such that ω = dζ. Due to the property dd=0, every exact form is closed. Thus, the differential d links the sequence of the spaces of differential forms on M into a chain complex

    0⟶Ω^0(M) d⟶Ω^1(M) d⟶⋯d⟶Ω^m-1(M) d⟶Ω^m(M)  0.

The k-th de Rham cohomology group, denoted by H^k_dR(M), is then defined to be the k-th homology of this chain complex, i.e., the quotient space of closed k-forms modulo the space of exact k-forms, i.e.,

    H^k_dR(M) = (d:Ω^k(M)→Ω^k+1(M))/(d:Ω^k-1(M)→Ω^k(M)).


The de Rham cohomology, by the de Rham theorem, is naturally isomorphic to the singular cohomology, and thus depends only on the manifold topology.



Let g be a Riemannian metric on M and ⟨·,·⟩_g be the pointwise inner product induced by g on Ω^k(M). The Hodge star operator ⋆ provides an isomorphism from the space of differential k-forms Ω^k(M) to the space of (m-k)-forms Ω^m-k(M), defined by the following formula

    ω∧⋆η = ⟨ω,η⟩_g μ_g,

where μ_g is the volume form on M induced by g. The Hodge L^2-inner product on the space of k-forms Ω^k(M) can then be obtained by taking the integral of the formula (<ref>)

    (ω, η) = ∫_Mω∧⋆η.

The codifferential δ: Ω^k(M)→Ω^k-1(M) is defined by

    δ = (-1)^m(k-1)+1⋆ d⋆,

which also has the nilpotent property δδ = 0. We call a differential form ω∈Ω^k(M) coclosed if δω = 0, or coexact if there is a (k+1)-form η∈Ω^k+1(M) such that ω = δη. 

The codifferential δ, as the differential d, also extends the classical gradient, curl and divergence in vector calculus. In ^3, it corresponds to -∇·, ∇× and -∇ when applied to 1-forms, 2-forms and 3-forms, respectively. 


The Hodge Laplacian for differential forms is defined as Δ = dδ + δ d: Ω^k(M)→Ω^k(M). Its kernel, consisting of all differential k-forms ω on M with Δω = 0, is called the space of harmonic k-forms.

We denote by ℋ^k_Δ(M) the space of harmonic k-forms and by ℋ^k(M) the space of k-forms that are both closed and coclosed, i.e., ℋ^k(M) =  d∩δ. The latter space ℋ^k(M), known as the space of harmonic k-fields, is in general only a subset of the space of harmonic forms ℋ^k(M)⊂ℋ^k_Δ(M), and is infinite-dimensional <cit.>. However, in the case of closed manifolds where ∂ M=∅, the space of harmonic forms ℋ^k_Δ(M) reduces to the space ℋ^k(M), as any harmonic form is both closed and coclosed. The result follows directly from the following formula

    0 = (Δω, ω) = ((dδ + δ d)ω, ω) =  (dω, dω) + (δω, δω),

due to the L^2-adjointness of the codifferential δ and the differential d on closed manifolds, i.e., (dω, η) = (ω, δη).

The classical Hodge decomposition theorem for closed manifolds states that the space of differential k-forms Ω^k(M) can be decomposed as

    Ω^k(M)= dΩ^k-1(M)⊕δΩ^k+1(M) ⊕ℋ^k_Δ(M).

These three subspaces are mutually orthogonal with respect to the inner product (<ref>). Moreover, Hodge theorem identifies the harmonic space ℋ^k_Δ(M) with the k-th de Rham cohomology group H^k_dR(M), which states that each harmonic form corresponds to exactly one equivalence class in H^k_dR(M). Therefore, the harmonic space ℋ^k_Δ(M) is fully determined by the manifold topology, and is finite-dimensional with its dimension given by the Betti number ℋ^k_Δ(M) = β_k.



















 §.§ Hodge decomposition for manifolds with boundary

In the presence of a non-empty boundary ∂ M, the two operators d and δ are not L^2-adjoint, as integration by parts leads to

    (dω, η) = (ω, δη) + ∫_∂ Mω∧⋆η,

which contains a boundary term that may not vanish, and thus the decomposed subspaces in (<ref>) are not orthogonal. However, certain boundary conditions can be enforced, ensuring the adjointness of the differential d and the codifferential δ, thereby inducing an orthogonal decomposition of the space of differential forms. 

The most common choices of boundary conditions ensuring the adjointness of d and δ are the normal (Dirichlet) and tangential (Neumann) boundary conditions. 

A differential form ω∈Ω^k(M) is called normal (Dirichlet) if it gives zero when applied to tangent vectors of the boundary, or tangential (Neumann) if the same holds for its dual ⋆ω instead. Denote by Ω^k_n(M) the set of normal differential k-forms and by Ω^k_t(M) the set of tangential differential forms, i.e.,

    Ω^k_n(M)    = {ω∈Ω^k(M) |  ω|_∂ M = 0}
    Ω^k_t(M)    = {ω∈Ω^k(M) |  ⋆ω|_∂ M = 0}.

Following their definitions, the spaces Ω^k_n(M) and Ω^m-k_t(M) are isomorphic under the Hodge star operator ⋆, also known as the Hodge duality. Moreover, the differential d preserves the normal boundary conditions, while the codifferential δ preserves the tangential boundary conditions.


The Hodge-Morrey decomposition <cit.> states that there is a 3-component L^2-orthogonal decomposition

    Ω^k(M) = dΩ^k-1_n(M)⊕δΩ^k+1_t(M) ⊕ℋ^k(M),

The orthogonality of the decomposition directly comes from the adjointness of δ and d when enforcing the normal or tangential boundary conditions. For ω∈Ω^k(M), there is a unique decomposition of ω given as follows:

    ω = dα_n + δβ_t + η,

where α_n∈Ω^k+1_n(M), β_t∈Ω_t^k+1(M), and η∈ℋ^k(M). Note that the potentials α_n and β_t are not uniquely determined as all α_n+dη and β_t+δγ with any η∈Ω_n^k-2(M) and
γ∈Ω_t^k+2(M) serve as potentials for the same components. However, the issue can be addressed by enforcing gauge conditions, such as


    δα_n = 0,
    dβ_t = 0.

The potentials α_n and β_t can then be uniquely determined by the following equations

    Δα_n = δω
    Δβ_t = dω,

by resolving the (finite) rank deficiencies of Δ under these boundary conditions. 


	In the case that M is a closed manifold, i.e., ∂ M = ∅, both the spaces Ω^k_n(M) and Ω^k_t(M) coincide with the space of differential forms Ω^k(M), and the space of harmonic fields is identical to the space of harmonic forms ℋ^k(M)=ℋ^k_Δ(M). The Hodge decomposition (<ref>) then reduces to the classical Hodge decomposition (<ref>) for closed manifolds.



	The Hodge-Morrey decomposition (<ref>) in the low dimensional Euclidean spaces ^2 and ^3, often referred to as the Helmholtz-Hodge decomposition in vector calculus, states that any vector field 𝐯 defined on a compact domain can be orthogonality decomposed as
	
    𝐯 = ∇ f + ∇×𝐮 + 𝐡,

	where f is a scalar potential that vanishes on the boundary of the domain, 𝐮 is a vector field orthogonal to the boundary, and 𝐡 is the harmonic vector field satisfying ∇×𝐡 = 0 and ∇·𝐡 = 0.
	The first component ∇ f and the second component ∇×𝐮 are often called the curl-free and divergence-free parts of the vector field 𝐯 respectively. Note that in the presence of a boundary, the resulting scalar potential f is also called satisfying the normal boundary of 0-forms, and the vector field 𝐮 is called satisfying the tangential boundary condition of 2-forms, which are direct counterparts of the potentials α_n and β_t in (<ref>). For a complete correspondence between scalar or vector fields, and differential forms under the normal and tangential boundary conditions, see <cit.>.


The space of harmonic fields ℋ^k, in general, is infinite-dimensional, and thus has no direct correspondence with the cohomology of the manifold. However, as noted by <cit.>, one can restrict to the space of normal harmonic fields, namely ℋ^k_n(M) = ℋ^k(M)∩Ω^k_n(M), and the space of tangential harmonic fields, ℋ^k_t(M)=ℋ^k(M)∩Ω^k_t(M). As a consequence of the de Rham map, these two subspaces ℋ^k_n(M) and ℋ^k_t(M) are fully determined by the topology of M: the space of normal harmonic fields ℋ^k_n(M) is isomorphic to the relative de Rham cohomology H^k_dR(M,∂ M), while the space of tangential harmonic fields ℋ^k_t(M) is isomorphic to the absolute de Rham cohomology H^k_dR(M) <cit.>. The two subspaces
ℋ^k_n(M) and ℋ^k_t(M) are thus finite-dimensional, with dimensions given by the Betti numbers: ℋ^k_n(M) = β_m-k and ℋ^k_t(M) = β_k. Furthermore, the kernels of the Hodge Laplacian Δ, when restricted to the space of normal forms Ω^k_n(M) and the space of tangential forms Ω^k_t(M) with gauge conditions on the boundary, can be identified to the space of normal harmonic fields and the space of tangential harmonic fields, respectively. Denote by Δ_n and Δ_t the restrictions of the Hodge Laplacian Δ on the space of normal fields Ω^k_n(M) satisfying Eq. (<ref>) and the space of tangential fields Ω^k_t(M) satisfying Eq. (<ref>), i.e., Δ_n:Ω^k_n(M)→Ω^k(M) and Δ_t:Ω^k_t(M)→Ω^k(M). Then immediately we have Δ_n = ℋ^k(M)∩Ω^k_n(M)=ℋ^k_n(M) and Δ_t = ℋ^k(M)∩Ω^k_t(M) = ℋ^k_t(M). The result follows directly from Eq. (<ref>). These identifications, finally, enable us to study the topology of the underlying manifold M through the Hodge Laplacians on normal and tangential forms.


	In fact, let ℋ^k_co = ℋ^k(M)∩δΩ^k+1(M) and ℋ^k_ex = ℋ^k(M)∩ dΩ^k-1(M). The space of harmonic fields ℋ^k(M) can be further orthogonally decomposed for smooth manifolds
	
    ℋ^k(M) =    ℋ^k_co(M)⊕ℋ^k_n(M) 
     =    ℋ^k_ex(M)⊕ℋ^k_t(M),

which results in the Hodge-Morrey-Friedrichs decomposition given as follows

    Ω^k(M)    = dΩ^k-1_n(M)⊕δΩ^k+1_t(M) ⊕ℋ^k_co(M)⊕ℋ^k_n(M)
        = dΩ^k-1_n(M)⊕δΩ^k+1_t(M) ⊕ℋ^k_ex(M)⊕ℋ^k_t(M).

In particular, if M is a compact domain in Euclidean spaces, then there is a unique orthogonal 5-component decomposition 

    Ω^k(M) = dΩ^k-1_n(M)⊕δΩ^k+1_t(M) ⊕ℋ^k_n(M) ⊕ℋ^k_t(M)⊕ (dΩ^k-1(M)∩δΩ^k+1(M)),

as the spaces ℋ^k_n(M) and ℋ^k_t(M) are  L^2-orthogonal, instead of just being transversal for compact manifolds in general <cit.>. Due to the correspondence between differential forms and vector fields in the low-dimensional Euclidean spaces, the implementation of this 5-component Hodge decomposition has been applied and implemented to the study of vector fields for surface triangle meshes, for tetrahedral meshes <cit.> and for regular Cartesian grids <cit.>.







As we mainly focus on applications of compact domains in ^3, to study the geometric and topological information of the underlying manifolds, there are eight Laplacians to be considered, which are defined on the spaces of differential k-forms with k= 0,1,2,3 satisfying either the normal or the tangential boundary conditions. However, thanks to the duality between the space of normal fields and tangential fields, the study of the spectra of these eight Laplacians reduces to that of four Laplacians on one of the two types of boundary conditions, and finally to the singular spectra of three differential operators, applied to differential forms of degree k = 0,1,2,3 <cit.>. Further details will be discussed in the next section for the discretization of Laplacians.





§ SPECTRAL ANALYSIS


In this section, we elaborate on the discretization of the Hodge Laplacian and introduce the Boundary-Induced Graph (BIG) Laplacian for compact domains in low-dimensional Euclidean spaces. Although the theory works for 2D compact domains, we focus only, for the remainder of the paper, compact domains in ^3, as we target mainly 3D applications. We use DEC to discretize all differential operators and differential forms on regular Cartesian grids, as it allows for efficient and accurate numerical algorithms relying on just matrix algebra, while keeping the L^2 orthogonality between different components in Hodge decomposition. In addition, the constructed discrete differential operators and differential forms in DEC approximate their smooth analogs. For the characterization of the underlying manifold, we choose the Eulerian formulation, where the manifold is given as a sublevel set of a level set function defined on a regular Cartesian grid. Another common way, called the Lagrangian formulation, discretizes the manifold as simplicial meshes, i.e., triangular or tetrahedral meshes in 2D or 3D. The spectrum analysis of the Hodge Laplacians has been discussed in <cit.> for the Lagrangian formulation and in <cit.> for the Eulerian formulation. Compared to the Lagrangian case, the Eulerian representation uses vertices, edges, faces and cells all fixed in a Cartesian grid, which significantly simplifies the data structures and algorithms. The Hodge stars, in the latter case, are close to rescaled identity matrices. This fact simplifies the study of Hodge Laplacians to that of BIG Laplacians with no Hodge stars involved, and thus leads to algorithms with efficient computations. 




 §.§ Discretization on entire grid

Denote by I_m a rectangular m-dimensional regular Cartesian grid with k-cells oriented according to their alignments with the coordinate axes. The entire grid I_m can be treated as a cell complex tessellating a rectangular domain in ^m, where each k-cell is a k-dimensional hypercube with edge length ℓ. A continuous differential k-form ω on I_m, following the de Rham map, can be discretized by its integral value over each oriented k-cell σ_i, given as W^i=∫_σ_iω <cit.>. The discrete differential on discrete k-forms of the grid I_m is then encoded by a sparse matrix D^I_k, which stores the signed incidence between (k+1)-cells and k-cells and is given as the transpose of the cell boundary operator ∂_k+1^T on (k+1)-cells following from Stokes' theorem ∫_σ dω= ∫_∂σω. An illustration of the chain complex formed by boundary operator ∂ for a simple grid complex with a single 2D cell can be seen in Fig. <ref>, which is a straightforward generalization of the chain complex on simplicial complexes. Note that the boundary of the boundary of a cell always results in a 0 chain, i.e., ∂∂ = 0, whose transpose immediately produces D^I_k+1D^I_k = 0, thus preserving the nilpotent property in the continuous setting.



The discrete Hodge star establishes a one-to-one correspondence between discrete k-forms on the primal grid I_m and discrete (m-k)-forms on its dual grid, given as the translated grid with grid points located at the m-cell centers of I_m, based on the following formula

    1/|σ_k|∫_σ_kω≈1/|⋆σ_k|∫_⋆σ_k⋆ω,

where ⋆σ_k is the dual (m-k)-cell formed by the dual grid points located at the centers of the primal m-cells incident to σ_k. See Fig. <ref> for an illustration for the correspondences between the primal and dual cells in the Cartesian grid case. Following from the discretization of differential forms, this correspondence leads to a diagonal matrix S^I_k with diagonal entries given by the ratio between the volumes of the dual  (m-k)-cells and the primal k-cells, ℓ^m-k/ℓ^k=ℓ^m-2k. The associated discrete Hodge L^2-inner product (<ref>) of two discrete k-forms V_k and W_k on grid I_m is then given by

    (V_k, W_k)^I = V_k^TS^I_kW_k.

The discrete codifferential, by definition of its smooth counterpart (<ref>), can be assembled from the discrete differential and Hodge star operators as δ^I_k = (S^I_k-1)^-1D^I_k-1S^I_k. Note that the discrete counterpart of the Hodge Laplacian Δ = dδ + δ d by replacing the differential and codifferential operators results in a nonsymmetric matrix. Instead, we consider the counterpart of ⋆Δ as the discrete Hodge Laplacian given by

    L^I_k = (D^I_k)^TS^I_k+1D^I_k + S^I_kD^I_k-1(S^I_k-1)^-1(D^I_k-1)^TS^I_k,

where the operators are considered to be null for k<0 or k>m.






 §.§ Discrete differential forms and operators on M



 
Compared to the case of simplicial or polygonal meshes, where the projection matrices to the interior can be straightforward to implement with the boundary elements explicitly labeled, modeling the manifold M as the volume bounded by a level set surface leads to delicate computation of the projection matrices. Note that the boundary of M using grid representation typically intersects with boundary k-cells instead of being its supersets. We restrict the computation to relevant cells by implementing the two types of boundary conditions through the inclusion or exclusion of the entire k-cells. We use the strategy as in <cit.> for the computation of projection matrices for each type of boundary condition: for the normal boundary condition, we include all cells if at least one of its vertices is inside or on the boundary of M, while for the tangential boundary condition, we include all cells with at least one of the vertices of the corresponding dual cells is inside or on the boundary. We refer to the former set of cells as the normal support and the latter as the tangential support. In contrast to the mesh case, it is important to note that neither the normal nor the tangential support is necessarily a superset of the other. See Fig. <ref> for one example showing the distinction of these two supports for 1-forms. 


In the computation of the discrete Hodge star operators, it is essential to consider and incorporate the boundary conditions. Following the procedure in <cit.>, we keep the dual cell volumes and adjust the primal cell volumes for normal boundary conditions, and do conversely for tangential boundary conditions with the primal cell volumes kept and the dual cell volumes changed. To be specific, when dealing with normal (resp. tangential) boundary conditions, we only compute the volume of the region of the primal (resp. dual) k-cells within the boundary ∂ M for the denominator (resp. numerator) of the ratio in the discrete Hodge star matrix, and leave the dual (resp. primal) cell volumes in the numerator (resp. denominator) unchanged. Each unaltered k-cell has a k-volume of ℓ^k. In addition, For numerical stability, we do not alter the volume of outside primal k-cells, and perturb the level set function evaluated at primal/dual grid points to have an absolute value above ϵ=10^-5ℓ, which ensures well-behaved fractional k-volumes. We denote by S^I_k,n and S^I_k,t the diagonal Hodge star matrices defined on the entire grid I^m corresponding to the normal and tangential boundary conditions, respectively.

The projection matrix to the corresponding support, for each type of boundary condition, can be constructed from the identity matrices by eliminating the rows corresponding to k-cells outside the support. Denote by P_k,n the projection matrix for k-cells onto the normal support and by P_k,t the one onto the tangential support. We then obtain a new set of differential and Hodge star operators for M:

    D_k,n   = P_k+1,nD_kP_k,n^T,   S_k,n = P_k, nS^I_k,nP_k,n^T
    
    	D_k,t   = P_k+1,tD_kP_k,t^T,   S_k,t = P_k, tS^I_k,tP_k,t^T

The nilpotent property D_k+1,nD_k,n = 0 and D_k+1,tD_k,t = 0 still holds for both boundary conditions due to D^I_k+1D^I_k=0 and the following observations

    P_k+1,n^T P_k+1, nD^I_k P_k, n^T = D^I_k P_k, n^T,  
    	P_k+1, t D^I_k P_k, t^T P_k,t = P_k+1,t D^I_k.

The discrete Hodge L^2-inner products of the two types of discrete k-forms on the manifold M for these two boundary conditions are then given by

    (ξ^k, ζ^k)^n    = (ξ^k)^TS_k,nζ^k 
    
    	(ξ^k, ζ^k)^t    = (ξ^k)^TS_k,tζ^k,

whose domains are the discrete Ω^k_n(M) and the discrete Ω^k_t(M) respectively. Finally, we assemble the two types of discrete Hodge Laplacians as in the mesh case:

    L_k, n   = D_k, n^TS_k+1, nD_k, n + S_k, nD_k-1, nS_k-1, n^-1D_k-1, n^TS_k, n
    
    	L_k, t   = D_k, t^TS_k+1, tD_k, t + S_k, tD_k-1, tS_k-1, t^-1D_k-1, t^TS_k, t.

The null spaces of these discrete Hodge Laplacians, as in the continuous case, are fully determined by the topology of the underlying manifold M, since they only depend on the differential and projection matrices. The dimension of the kernel of L_k, n is given by the Betti number β_m-k, while the dimension of the kernel of L_k, t is given by β_k. Here the Betti number β_k presents directly the number of k-dimensional holes on the manifold M. For instance, β_0 gives the number of connected components, β_1 gives the number of tunnels, and β_2 provides the number of closed cavities, respectively. The spectra of these Laplacians, in addition, could be used to study the geometric information of the manifold. It is known that the non-zero eigenvalues of the Laplacians provide rich insights into the shape of a manifold. For instance, the Fiedler value, defined as the smallest non-zero eigenvalue of a graph Laplacian, describes connectivity. As another example, the multiplicity of eigenvalues can reveal certain symmetries of the shape.


	The two types of discrete Hodge Laplacians (<ref>) not only provide rich geometrical and topological information of the underlying manifold, but also play a central role in the computation of the discrete Hodge decomposition (<ref>) of differential forms for compact domains in 2D and 3D Euclidean spaces. In particular, they can be utilized, by resolving the rank deficiencies, to compute the potentials of the decomposed components in Hodge decomposition on normal or tangential support satisfying the corresponding boundary conditions. In addition, as the kernel sizes of Laplacians are finite, their eigenvectors corresponding to 0 eigenvalues, for each k, form a basis for the space of normal or tangential harmonic fields.



Note that the discrete Hodge stars in the Eulerian setting are almost identical to rescaled identity matrices. Therefore, the computations of the Hodge Laplacian can be further simplified by replacing the Hodge stars with identity matrices, leading to the definition of the Boundary-Induced Graph (BIG) Laplacians as follows:

    L^B_k, n   = D_k, n^TD_k, n + D_k-1, nD_k-1, n^T
    
    	L^B_k, t   = D_k, t^TD_k, t + D_k-1, tD_k-1, t^T.

The BIG Laplacians were introduced in <cit.> for bounded domains to facilitate the comparison and contrast of the Hodge Laplacians and the combinatorial Laplacians. They preserve the Hodge Laplacian's capability to perform differential calculus but also retain the discrete nature of combinatorial Laplacians. The convergence of the spectra of the BIG Laplacians to Hodge Laplacians has been discussed in <cit.>, showing that the spectra of (<ref>) converge to those of Hodge Laplacians up to a scaling value ℓ^-2 when enforcing the boundary conditions. This scaling value ℓ^-2 is exactly the ratio between the missing scaling factor ℓ^m-2(k+1) in L_k and the missing factor ℓ^m-2k of S_k. As the BIG Laplacians produce results similar to those obtained from the discrete Hodge Laplacians with less computation, they can also be used to study the geometric and topological information of the underlying manifolds.


Note that the dual grid is also a Cartesian grid staggered with the primal grid by a replacement of ℓ/2 in all three axial directions of the Cartesian coordinates. For the study of the spectra of these Laplacians, one only needs to implement one type of boundary condition, for instance, the normal boundary condition, as L_k,n defined on the primal grid with normal boundary conditions is equivalent to L_m-k,t defined on its dual grid with tangential boundary conditions.



 §.§ Spectral analysis of Laplacians



Note that, on the grid, the Hodge Laplacians and the BIG Laplacians are of the same sparsity patterns. For simplicity in exposition when discussing the spectrum analysis of the Laplacians, we let L_k be a generic Laplacian matrix of the form

    L_k   = D_k^TS_k+1D_k + S_kD_k-1S_k-1^-1D_k-1^TS_k.

Here the Laplacian L_k can be interpreted, under choices of boundary conditions and Hodge star accuracy, as either a Hodge Laplacian, or BIG Laplacian ( with S_k set to identity), under tangential or normal boundary condition. The eigenvalues and eigenvectors of L_k can be solved by considering the generalized eigenvalue problem

    L_k W =λ S_k W,

where λ is an eigenvalue, and W is the associated eigenvector.
To analyze the results, we perform the following transformation in the space of discrete forms: D̅_k = S_k+1^1/2D_kS_k^-1/2, L̅_k = S_k^-1/2L_k S_k^-1/2 and W̅ = S_k^1/2W. Rewriting the formulas above yields a simplified form of the Laplacian

    L̅_k = D̅_k^TD̅_k + D̅_k-1D̅_k-1^T,

and a regular eigenvalue problem:

    L̅_k W̅ = λW̅.

Note that the property D̅_kD̅_k-1 = 0 is preserved.





As the non-zero eigenvalues of D̅_k^TD̅_k and D̅_kD̅_k^T for each k are the same, given by the squared non-zero singular values of the discrete differential D̅_k, and each Laplacian L̅_k is just the combination of D̅_k^TD̅_k and D̅_k-1D_k-1^T, the entire spectrum of the Laplacians can thus be studied through the singular values of discrete differentials. Let 

    D̅_k = U_k+1Σ_kV_k^T

be the singular value decomposition of D̅_k, where U_k+1 and V_k are orthogonal matrices and Σ_k is a rectangular diagonal matrix with diagonal values given by the singular values of D̅_k. It follows immediately from D̅_kD̅_k-1 = 0 that 

    Σ_kV_k^TU_kΣ_k-1 = 0.

Therefore, the columns of V_k corresponding to non-zero singular values of D̅_k are orthogonal to columns of U_k associated with non-zero singular values of D̅_k-1. In addition, it follows from

    L_k = V_kΣ_k^2V_k^T + U_kΣ_k-1^2U_k^T

that the spectrum of L̅_k is given by the union of squared non-zero singular values of D̅_k and D̅_k-1, and 0, with the multiplicity of 0 given by the k-th Betti numbers. The columns of U_k and V_k corresponding to non-zero singular values, together with the set of harmonic forms, span the entire space of differential k-forms. 

In the case that (M) = 3, for each type of boundary condition, we have four Laplacians of different degrees in total k = 0,1,2,3:

    L̅_0   = D̅_0^TD̅_0
    L̅_1   = D̅_1^TD̅_1 + D_0D_0^T
    L̅_2   = D̅_2^TD̅_2 + D_1D_1^T
    L̅_3   = D̅_2D̅_2^T.

Due to the aforementioned discussion on the spectrum of Laplacians and the duality of the normal and tangential boundary conditions, the spectral analysis of all Laplacians can be reduced to the singular spectra analysis of the three discrete differentials D̅_0, D̅_1, and D̅_2 with one type of boundary conditions. Note that the numerical evaluation of the singular values of these differentials, in the simplicial mesh case, may differ for the two types of boundary conditions, as the DoF for normal k-forms and tangent m-k forms are different. However, in the Cartesian representation, they are strictly equivalent to each other by shifting the grid in all directions of the axis by ℓ/2, so long as M is at least one grid spacing away from the boundary of the grid. 

For the computation of the spectra of the Laplacians, we choose the normal boundary condition. The spectra of all Laplacians L̅_k,n for compact domains in ^3 can be finally decomposed into three distinct parts: the squared singular values of the gradient of tangential scalar fields, denoted by T, the squared singular values of the gradient of normal scalar fields, denoted by N, and the squared singular values of the curl of tangential curl fields, denoted by C.




§ PERSISTENT DE RHAM-HODGE LAPLACIANS


In this section, we present the construction of the persistent de Rham-Hodge Laplacian on differentiable manifolds, which is based on the filtration of manifolds induced by varying a single parameter (the filtration parameter). The spectra of Laplacians carry rich topological and geometric information of a manifold. However, a single manifold might not provide enough information in applications like feature extraction for machine learning analysis. As such, instead of studying just a single manifold, one could examine the spectra of a family of manifolds by adjusting the filtration parameter. The spectra of the Laplacians from this family of manifolds could provide much more information than by considering just one, as the topology and geometry could change for different parameters. This single-parameter family of manifolds, called the evolution of manifolds, was first introduced in <cit.> based on tetrahedral meshes. We briefly recap the background. 

The formal definition of the evolving manifold is given by a one-parameter family of immersions F_c = F(·, c) with F: B× [a,b]→ N being a smooth map, where B is called the base manifold, N is the ambient manifold, and c∈[a,b] is a real parameter within the interval. In practice, the most common way to define the evolution of manifolds without specifying B is through a level set function by adjusting the isovalues. Given a function f: N→ [a,b], then in our case, we consider the sublevel sets M = {x∈ N |  f(x)≤ c} with boundary given as ∂ M = {x∈ N |  f(x) = c} for c∈ [a,b]. A sequence of manifolds can then be obtained by considering evenly distributed isovalues of the function f with the inclusion map

    M_0 M_1 M_2⋯ M_s-1 M_s,

where each M_l is given as the sublevel set corresponding to c_l with a≤ c_0<c_1<⋯ < c_s ≤ b. To ensure that M_l is a manifold, we assume that the function f is a Morse function on N, and none of the c_l's corresponds to a critical value of the function f, i.e., f^-1(c_l) does not contain any critical points. This is always possible as the set of Morse functions on a compact manifold is dense in the space of smooth functions, and their critical points are isolated, nondegenerate, and finite for compact manifolds. Thus we can always perturb any input function slightly to avoid critical isovalues in {c_l, l =0, 1, ⋯ s}. In addition, we assume that for each l, M_l, l+1 = M_l+1\ M_l={x∈ N|  f(x)∈[c_l,c_l+1]} contains at most one critical point, which can be realized by refining the parameter sequence. Note that both M_l and M_l, l+1 are compact. By Morse theory, if M_l, l+1 contains no critical points, M_l is diffeomorphic to M_l+1. The retraction from M_l+1 to M_l can be easily constructed by considering a flow along the gradient of the function. As M_l+1 is homotopic to M_l in this case, there is no topological change happening between (c_l, c_l+1). For the other case when there is exactly one critical point in M_l, l+1, the manifold M_l+1 is homotopic to M_l with a k-cell attached, where k is the index of the critical point, defined to be the dimension of the largest subspace on which the Hessian Hess(f)(x) is negative definite. The topological change of the sublevel sets occurs precisely at the critical values of the level set function. Depending on the type of the critical points, i.e., local minimum, saddle points, and local maximum, the topology changes in different ways. In general, a local maximum has the full index m, a local minimum has index 0, while saddle points have indices strictly between 0 and m. In the case of ^3, the occurrences of minima and maxima correspond to the birth of the 0-th generators and the death of the 2nd homology generators respectively, while the occurrences of 1-saddle points correspond to the birth of 1st homology generators or the death of the 0-th homology generators, and those of 2-saddle points correspond to the birth of 2nd homology generators or the death of 1st homology generators.



 §.§ Persistent harmonic forms

As the de Rham complex depends on the topology, it can also be extended to the filtration of manifolds. Due to the duality of the normal and tangential boundary conditions, without loss of generality, one may focus on the space of normal differential forms. Given M_l M_l+1, we then need to construct a map from the space of normal k-forms Ω^k_n(M_l) to the space of normal k-forms Ω^k_n(M_l+1), that extends each normal k-form on M_l to a normal k-form on M_l+1. Let ω∈Ω^k_n(M_l). The idea is to utilize the boundary condition of ω on M_l and extend the forms ω|_∂ M_l to exact normal forms on the domain M_l, l+1 with certain boundary conditions on ∂ M_l, l+1 = ∂ M_l∪∂ M_l+1. Then the combination ω defines a normal k-form on the manifold M_l+1. Note however that δω is only 0 in M_l, so the extension of ω∈δ may no longer be in δ on M_l+1.

To be specific, we consider the biharmonic equation Δ^2ζ=Δ(Δζ) = 0 on M_l, l+1 with both Dirichlet and Neumann boundary conditions to ensure the smoothness of d ζ with ω through ∂ M_l. Note that d ζ satisfies the normal boundary condition on M_l,l+1. Let ω be the extension of ω on M_l+1 with ω = ω on M_l and ω = d ζ on M_l, l+1. It follows that ω∈Ω^k_n(M_l+1) as it satisfies the normal boundary condition ω|_∂ M_l+1 = ζ|_∂ M_l+1=0. 

While the biharmonic equation produces a smooth extension, in practice, it is more efficient to consider the harmonic extension with the boundary condition Δζ = 0 with the boundary condition of ⋆ dζ = ⋆ω on ∂ M_l and the typical normal form boundary condition on ∂ M_l+1. The solution, by <cit.>, is unique. The resulting ω̅ is continuous but nonsmooth as δω̅ may lead to a Dirac distribution on ∂ M_l when M_l,l+1 induces a topological change. For instance, for a harmonic normal 1-form ω on a spherical shell M_l with M_l+1 turning into a solid ball, the biharmonic extension would create a uniform divergence δω̅ in M_l,l+1, whereas the harmonic extension creates a thin layer of nonzero divergence δω̅ near the part of ∂ M_l around the cavity in the middle. Thus, the harmonic extension serves the same purpose in reducing the kernel of δ.

Denote by ℐ_l, 1 the map from Ω^k_n(M_l) to Ω^k_n(M_l+1) sending ω to ω.  Note that (d∘ℐ_l, 1)(ω) is 0 on M_l,l+1 and thus the same as the extension of the differential of a normal form dω on M_l, l+1, i.e., d∘ℐ_l, 1 =ℐ_l, 1∘ d. It follows that there is a commutative diagram

    Ω^0_n(M_0) rddℐ^0_0,1   Ω^1_n(M_0) rddℐ^1_0,1   Ω^2_n(M_0) rddℐ^2_0,1   Ω^3_n(M_0) dℐ^3_0,1
    Ω^0_n(M_1) rddℐ^0_1,1   Ω^1_n(M_1)  rddℐ^1_1,1   Ω^2_n(M_1)  rddℐ^2_1,1   Ω^3_n(M_1)  dℐ^3_1,1
    Ω^0_n(M_2)  rddℐ^0_2,1   Ω^1_n(M_2) rddℐ^1_2,1   Ω^2_n(M_2) rddℐ^2_2,1   Ω^3_n(M_2) dℐ^3_2,1
    ⋯   ⋯   ⋯   ⋯

where the horizontal direction gives the de Rham complex and the vertical direction shows the filtration-induced extensions.


Next, we introduce the p-persistent Hodge Laplacian. Let ℐ_l, p = ℐ_l+p-1, 1∘ ...∘ℐ_l, 1, which then gives an extension map from the space of normal forms on M_l to the space of normal forms on M_l+p. We have the following commutative diagram

    Ω_n^k(M_l)      Ω_n^k+1(M_l)
    
    Ω_n^k-1(M_l+p)      Ω_n^k(M_l+p)      ["d_l+p^k-1", shift left, from=3-1, to=3-3]
    	["R_l,p", shift left, from=3-3, to=1-3]
    	["δ_l+p^k", shift left, from=3-3, to=3-1]
    	["d_l^k", shift left, from=1-3, to=1-5]
    	["δ̃_l,p^k", shift left=-2, from=1-3, to=3-1]
    	["I_l,p", shift left, from=1-3, to=3-3]
    	["δ_l^k+1", shift left, from=1-5, to=1-3]
    	["d̃_l,p^k-1", shift left=4, from=3-1, to=1-3]
    .

















































Here d_l, δ_l denotes the differential and codifferential on Ω^k(M_l), d_l+p, δ_l+p denotes the differential and codifferential on Ω^k(M_l+p), respectively, and ℛ_l, p is the projection of differential forms in Ω^k_n(M_l+p) to the space spanned by the harmonic extensions followed by the restriction to M_l. Let δ̃_l,p = δ_l+p∘ℐ_l, p and d̃_l,p = ℛ_l, p∘ d_l+p. By the construction of the extension, we have 
(δ̃_l,pω, η) = (ω, d̃_l,pη), i.e., δ̃_l,p are adjoint to d̃_l,p. 
We then define the p-persistent Hodge Laplacian operator Δ^p_n, l: Ω^k_n(M_l)→Ω^k_n(M_l) as follows

    Δ^p_n, l = d̃_l,pδ̃_l,p + 
    	δ_ld_l.

It is easy to see that when p=0, the p-persistent Hodge Laplacian gives exactly the usual Hodge Laplacian Δ_n, l: Ω^k_n(M_l)→Ω^k_n(M_l) restricted to the space of normal forms. We then define the p-persistent normal harmonic fields as the kernel of the p-persistent Hodge Laplacian ℋ_n^k,p = Δ^p_n, l, which can be identified with the space δ̃_l,p∩ d_l. Note that by the extension construction and ℛ_l, p∘ℐ_l, p = Id, one can see that δ̃_l,p⊂δ gets smaller as p increases, which confirms that fewer cohomology generators persist longer. 



 §.§ Discretization of p-persistent de Rham cohomology



The regular Cartesian grid allows one to define persistent graph Laplacian on manifolds in the same way as persistent graph Laplacian <cit.>. It also allows defining persistent Hodge Laplacian in a consistent way, with the inclusion of nontrivial Hodge stars.

Recall that the discrete differential k-forms can be seen as a k-cochain, i.e., a linear mapping from the chain space 𝒞_k to  that sends a k-chain c_k = ∑_i a_iσ_i to ∫_c_kω = ∑_i a_iW_i, where W_i = ∫_σ_iω is the integral of a smooth k-form ω over the k-cell σ_i.


By varying the isovalue of the level set function f, we can get a sequence of cell complexes given as nested sequences of sub-cell complexes of K satisfying the normal boundary conditions.




    ∅ = K_0⊂ K_1 ⊂⋯⊂ K_s-1⊂ K_s = K.

See Fig. <ref> for an example of such a nested sequence of sub-cell complexes in a 2D Cartesian grid. Denote by 𝒞^k(K_l) the space of discrete k-forms on subcomplex K_l with 0≤ l≤ s. Note that K_l⊂ K_l+1. A discrete k-form on K_l can be easily extended to K_l+1 by solving the discrete Laplace equation with the above boundary conditions for values on every k-cells in K_l,l+1=Cl (K_l+1\ K_l), the closure of the difference complex. We denote this extension map as I_l, 1: 𝒞^k(K_l)→𝒞^k(K_l+1) and by I_l, p = I_l+p-1, 1∘ I_l+p-2, 1∘⋯∘ I_l, 1: 𝒞^k(K_l)→𝒞^k(K_l+p) the extension mapping from the space of discrete k-forms on K_l to the space of discrete k-forms on K_l+p, which may also be constructed directly by solving the Laplace equation on K_l,l+p=Cl(K_l+p\ K_l). With this extension mapping, the space of discrete k-forms on K_l can be seen as a subspace of discrete k-forms on K_l+p.

A sequence of the discrete de Rham cochain complexes can be defined as follows:

    𝒞^0(K_0)   𝒞^1(K_0)   ⋯   𝒞^k(K_0)   𝒞^k+1(K_0)   ⋯
    𝒞^0(K_1)   𝒞^1(K_1)   ⋯   𝒞^k(K_1)   𝒞^k+1(K_1)   ⋯
    𝒞^1(K_2)   𝒞^1(K_2)   ⋯   𝒞^k(K_2)   𝒞^k+1(K_2)   ⋯
    ⋯   ⋯      ⋯   ⋯["D_0^0", shift left, from=1-1, to=1-2]
    	["I_0,1", shift left, from=1-1, to=2-1]
    	["δ^1_0", shift left, from=1-2, to=1-1]
    	["D_0^1", shift left, from=1-2, to=1-3]
    	["I_0,1", shift left, from=1-2, to=2-2]
    	["δ_0^2", shift left, from=1-3, to=1-2]
    	["D_0^k-1", shift left, from=1-3, to=1-4]
    	["δ_0^k", shift left, from=1-4, to=1-3]
    	["D_0^k", shift left, from=1-4, to=1-5]
    	["I_0,1", shift left, from=1-4, to=2-4]
    	["δ_0^k+1", shift left, from=1-5, to=1-4]
    	["D_0^k+1", shift left, from=1-5, to=1-6]
    	["I_0,1", shift left, from=1-5, to=2-5]
    	["δ_0^k+2", shift left, from=1-6, to=1-5]
    	["D_1^0", shift left, from=2-1, to=2-2]
    	["I_1,1", shift left, from=2-1, to=3-1]
    	["δ_1^1", shift left, from=2-2, to=2-1]
    	["D_1^1", shift left, from=2-2, to=2-3]
    	["I_1,1", shift left, from=2-2, to=3-2]
    	["δ^2_1", shift left, from=2-3, to=2-2]
    	["D_1^k-1", shift left, from=2-3, to=2-4]
    	["δ_1^k", shift left, from=2-4, to=2-3]
    	["D_1^k", shift left, from=2-4, to=2-5]
    	["I_1,1", shift left, from=2-4, to=3-4]
    	["δ_1^k+1", shift left, from=2-5, to=2-4]
    	["D_1^k+1", shift left, from=2-5, to=2-6]
    	["I_1,1", shift left, from=2-5, to=3-5]
    	["δ_1^k+2", shift left, from=2-6, to=2-5]
    	["D_2^0", shift left, from=3-1, to=3-2]
    	["I_2,1", shift left,from=3-1, to=4-1]
    	["δ_2^1", shift left, from=3-2, to=3-1]
    	["D_2^1", shift left, from=3-2, to=3-3]
    	["I_2,1", shift left, from=3-2, to=4-2]
    	["δ_2^2", shift left, from=3-3, to=3-2]
    	["D_2^k-1", shift left, from=3-3, to=3-4]
    	["δ_2^k", shift left, from=3-4, to=3-3]
    	["D_2^k", shift left, from=3-4, to=3-5]
    	["I_2,1", shift left, from=3-4, to=4-4]
    	["δ_2^k+1", shift left, from=3-5, to=3-4]
    	["D_2^k+1", shift left, from=3-5, to=3-6]
    	["I_2,1", shift left, from=3-5, to=4-5]
    	["δ_2^k+2", shift left, from=3-6, to=3-5]

where D_l^k: 𝒞^k+1(K_l)→𝒞^k(K_l) denotes the discrete differential operator, and δ_l^k: 𝒞^k(K_l)→𝒞^k-1(K_l) denotes the discrete codifferential operator on K_l.

To define the persistent discrete Hodge Laplacian, we construct the discrete counterparts of d̃_l,p and δ̃_l,p in the previous section. 

Denote by δ_l,p^k+1,n:𝒞^k+1(K_l)→𝒞_l,p^k  the operator given as δ_l,p^k,n = δ_l+p^k I_l,p^k,n, where δ_l+p^k,n is the previously defined discrete operator for K_l+p and I_l,p^k,n is the discrete harmonic extension operator defined next. Assuming K_l,l+p contains few  k-cells, the harmonic extension is then constructed by the linear system L^k-1,n_K_l,l+pζ = 0, and shifting all ⋆ dζ values in the overlap of supports of K_l and K_l,l+p to the right-hand side and replacing them with a rescaling of ⋆ω based on the k-volume within each support. More specifically, the resulting system is L̃^k-1,n_K_l,l+pζ̃= -S^k-1,nδ^k_∂ K_lω, where L̃^k-1,n_K_l,l+p is the Laplace operator applied to a form ζ̃ defined on K_l,l+p\∂ K_l, and δ^k_∂ K_l is the boundary codifferential operator that uses the values of ω on ∂ K_l to evaluate the neighboring (k-1)-cells in K_l,l+p\∂ K_l. 

The resulting extension operator 

    I_l,p= 
    [                                         Id_K_l; -D^k_K_l,l+p(L̃^k,n_K_l,l+p)^-1 S^k,nδ^k_∂ K_l ],

where Id_K_l is the identity matrix in K_l up to a rescaling in the boundary, 
provides the combination of ω in K_l and  dζ̃ in K_l,l+p\∂ K_l, when applied to ω. The matrix corresponding to I_l,p is dense for rows corresponding to cells in K_l,l+p but diagonal for rows corresponding to cells in K_l.  Note that δ_∂ K_l is not necessarily 0 for coclosed ω, but is 0 for coexact ω.

The adjoint operator of δ_l,p^k+1,n defines D^k_l,p. In the following, we drop most of the subscripts for clarity. Recall that (ω, d̃η)=(δ̃ω,η) can be discretized as

    [W]^T S [D̃ E]= [S^-1 D^T S I_l,p W]^T S [E]

with W and E as discrete versions of ω and η. Thus D̃= S^-1 I_l,p^T S D, from which we may recognize the restriction operator as R=S^-1I_l,p^T S. This restriction operator can be seen as the L_2-projection onto the space formed by all harmonic extensions from Ω_n^K(M_l). 








Note that in this case, we immediately δ_l,p^kδ_l^k+1 = 0, since the extension operator will generate ζ̃=0 for any coexact form ω=δβ on K_l as the right-hand side of the associated linear system essentially corresponds to δδβ=0. From the adjoint version, we have  D_l^kD_l,p^k-1 = 0, and thus the following commutative diagram
















    𝒞^k(K_l)      𝒞^k+1(K_l)
    
    𝒞^k-1(K_l+p)      𝒞^k(K_l+p)      ["D_l+p^k-1", shift left, from=3-1, to=3-3]
    	["R_l,p", shift left, from=3-3, to=1-3]
    	["δ_l+p^k", shift left, from=3-3, to=3-1]
    	["D_l^k", shift left, from=1-3, to=1-5]
    	["δ_l,p^k", shift left=-2, from=1-3, to=3-1]
    	["I_l,p", shift left, from=1-3, to=3-3]
    	["δ_l^k+1", shift left, from=1-5, to=1-3]
    	["D_l,p^k-1", shift left=4, from=3-1, to=1-3]
    .


The discrete p-persistent Hodge Laplacian is then given as follows 

    L_l,p^k = D_l,p^k-1δ_l,p^k + δ_l^k+1D_l^k,

and the discrete p-persistent BIG Laplacian is  

    L_l,p^k = D_l,p^k-1(D_l,p^k-1)^T + (D_l^k)^TD_l^k.














We now present some examples of evolving manifolds and show results for the spectral analysis of their persistent Laplacians. In particular, we focus on the changes of Betti numbers β_0, β_1 and β_2 and the first non-zero eigenvalues λ^T_1, λ^C_1 and λ^N_1 of the 0-persistent BIG Laplacians in the set T, C and N, respectively, as introduced in Sec. <ref>. Four models are considered, including the Bimba model, the kitten model, a genus-3 model, and a four-ball model. For each model, we show on the top row snapshots of evolving manifolds at five evenly spaced isovalues in a chosen interval, and on the bottom row the changes in Betti numbers and the first non-zero eigenvalues λ^T_1, λ^C_1 and λ^N_1. All the evolving manifolds are generated using isovalues of the signed distance function (SDF) from the original surface model, given as the 0-isosurface of the SDF. As we show below, these values from the evolution of manifolds provide rich information than considering just a single manifold. The discontinuity of these variables indicates the topological changes occurring during the evolution process, and the monotonicity of these non-zero eigenvalues reveals the geometric changes. 


The results for the Bimba model are presented in Fig. <ref> with an isovalue interval [0, 0.2]. As there is no topological change happening in the evolution process, all Betti numbers β_0, β_1 and β_2 remain constant, and λ^T_1, λ^C_1 and λ^N_1 are continuous throughout the whole process. Both λ^C_1 and λ^N_1  decrease as the isovalue increases. 

Fig. <ref> illustrates the results for the kitten mode with one tunnel formed by its tail. The isovalue interval [0, 8] is considered. One can see all variables are continuous during the evolution process except that β_1 and λ^C_1 both drop at the same isovalue, where β_1 changes from 1 to 0. This happens due to the disappearance of the tunnel. In addition, λ^T_1 increases at the beginning, and then slows down its rate of increase at the isovalue after the tunnel disappears, and λ^C_1 and λ^N_1 decrease during the evolution process.

Note that there are also tunnels in the evolving manifolds for the genes-3 model, as we expected, a similar phenomenon can also be observed in Fig. <ref> for the change of the Betti numbers and the first non-zero eigenvalues. The isovalue interval [0.1, 4] is considered for this model. The disappearance of the three tunnels leads to a drop of β_1 from 3 to 0 and also a drop of λ^C_1. λ^T_1 initially increases, and then changes its behavior to decrease after the tunnels vanish. The evolution process results in a decrease in λ^C_1 and λ^N_1, just as the previous two models.

The evolving process of the four-ball model with isovalue interval [2, 3.84], see Fig.  <ref>, leads to discontinuities in all Betti numbers and the first non-zero eigenvalues. As the four separate components merge in the evolution, β_0 changes from 4 to 1, along with a drop in λ^T_1 at the same isovalue. In addition, β_1 increases from 0 to 3 due to the appearance of three tunnels when the merge happens and then decreases to 0 after the disappearance of all tunnels. The non-zero eigenvalue λ^C_1 has a drop that occurs when the tunnel vanishes, however, it is continuous when the tunnels are formed. This suggests that the continuity of λ^C_1 is only related to the death but not the birth of tunnels. One can also observe a slowdown in the rate of change of λ^T_1 following the disappearance of all tunnels. As the isolate increases further, a cavity occurs in the manifold, resulting in an increase of β_2 from 0 to 1 and finally a decrease from 1 to 0 after the cavity disappears. This topological change can also be observed in λ^N_1, where λ^N_1 becomes non-differentiable.



As illustrated by these models, changes in Betti numbers β_0, β_1 and β_2 and the first non-zero eigenvalues λ^T_1, λ^C_1 and λ^N_1 not only reflect the changes in topology, but also characterizes the changes in geometry for the evolution of manifolds. The rich information revealed by these variables leads to potential applications in various topological data analysis tasks. 








§ PROOF-OF-PRINCIPLE EXPERIMENTATION


In this section, we carry out a proof-of-principle experimental demonstration of the proposed persistent de Rham-Hodge theory. In this approach, the problem is defined on manifolds with boundaries. Appropriate boundary conditions are implemented to match actual topological dimensions. The resulting persistent Hodge Laplacians are solved to deliver the corresponding series of eigenvectors and eigenvalues at various scales. In this approach, we use these eigenvalues for machine learning predictions of protein-ligand binding affinity.  
The binding affinity describes the strength of protein-ligand interactions for each protein-ligand complex.

We consider two benchmark datasets, PDBbind-v2007 and PDBbind-v2016 <cit.>, to demonstrate the effectiveness of our framework in capturing the topological features of protein-ligand complexes. The datasets can be downloaded from http://pdbbind.org.cn/. These two PDBbind datasets provide collections of biomolecular complexes in Protein Data Bank (PDB) with experimentally a measured binding affinity for each protein-ligand complex, and are commonly used in various studies such as drug-discovery or molecular recognition, etc <cit.>. We aim to build a machine learning model, by utilizing the topological and geometric features of the protein-ligand complexes generated using our persistent Hodge Laplacian (PHL) framework as inputs, for predicting the protein-ligand binding affinities. 

The biomolecular complexes in each PDBbind dataset are organized into three sets, including a general set, a refined set and a core set, with each set being a superset of the next. In our experiments, for each dataset, we use the refined set, excluding the core set, to train the predictive model for the binding affinities of the protein-ligand complexes in the core set. The PDBbind-v2007 dataset contains a total of 1,300 complexes with 1,090 in the refined set and 195 in the core set, while the PDBbind-v2016 dataset has a total of 4,057 complexes with 3,767 in the refined set and 290 in the PDBbind core set.




 §.§ Element specific discrete to continuum mapping
 


The original datasets contain atomic names and coordinates, which are the so-called point cloud data. To generate manifold representations, we carry out the discrete to continuum mapping using the flexibility and rigidity index <cit.>. To compute the topological feature of each protein-ligand complex for the machine learning model, we use the element-specific approach <cit.>.  Specifically, we consider the pairwise interactions between element types that are commonly found in proteins and ligands, including Hydrogen (H), Carbon (C), Nitrogen (N), Oxygen (O), and Sulfur (S) in proteins, and Hydrogen (H), Carbon (C), Nitrogen (N), Oxygen (O), Sulfur (S), Phosphorus (P), Fluorine (F), Chlorine (Cl), Bromine (Br), and Iodine (I) in ligands. These interactions result in a total of 50 pairs of atom types for each protein-ligand complex <cit.>. However, due to the absence of H in most proteins, we reduce the number of atom pairs to 40 in practice, ignoring the element H in all proteins. These 40 atom pairs, formed by atom types {C, N, O, S} in proteins, and atom types {H, C, N, O, S, P, F, Cl, Br, I} in ligands, along with their xyz coordinates, are used to generate the topological features for each protein-ligand complex. In this paper, all atom-pair complexes are determined by a cutoff distance 12A from the ligand.













Let {𝐱^α_i, i=1,⋯, s} be the location coordinates of all s atoms in an atom pair, where α denotes the atom type of the atom either in the protein or in the ligand. For this atom pair, a level set function can then be obtained by considering the negative sum of Gaussian density functions defined at the xyz coordinates of all atoms, given as

    ρ(𝐱, τ) = -∑^s_i = 1exp(-(||𝐱 - 𝐱^α_𝐢||/τ r^α_i)^2),

where ||𝐱 - 𝐱^α_𝐢|| is the Euclidean distance from position 𝐱 to the location 𝐱^α_𝐢 of the i-th atom, τ is a scalar value, and r^α_i is the van der Waals radius of the i-th atom, determined by the atom type α. Given an isovalue c, the sublevel set

    M = {𝐱 | ρ(𝐱, τ)≤ c }

defines a compact manifold in ^3 with its boundary given by the isosurface ∂ M = {𝐱  |  ρ(𝐱, τ) = c }. A filtration of a manifold for the atom pair can then be obtained by choosing a list of evenly spaced isovalues of this level set function (<ref>). Let c_1< c_2<⋯ < c_s be such isovalues. We have their corresponding sublevel sets given as follows

    M_1⊂ M_2⊂⋯⊂ M_s,

where M_i is the compact manifold associated to isovalue c_i. In Fig. <ref> we present one example of the resulting filtration of manifolds at 3 different isovalues for atom pair OH in protein-ligand complex 4tmn. Note that the function (<ref>) is a special case of the flexibility rigidity index (FRI) density function <cit.>, which has been shown computationally stable in converting discrete point cloud representations to continuous embeddings, and been used for generating protein boundary surfaces <cit.> and interactive manifolds <cit.>. Therefore, one can also make other reasonable choices of FRI density functions to generate the filtration of manifolds. 




 §.§ Machine learning feature extraction



In the computation of the Laplacians, one can ideally choose a common Cartesian grid such that it contains all manifolds of interest for all protein-ligand complexes, which ensures that all Laplacians are computed consistently, making their spectra comparable for different complexes. However, as atoms are spread out in the space for different atom pairs, we need to use a sufficiently large grid with a fine resolution for accurate computation of Laplacians, which significantly increases the computational load. Instead, we consider, for each type of atom pairs, a fixed Cartesian grid, regardless of the types of protein-ligand complexes. This approach also ensures that the topological features are comparable for different protein-ligand complexes, as all spectra are computed in a same grid for all atom pairs of the same type. For simplicity, we choose a fixed grid spacing for all Cartesian grids across different atom pairs. 


We consider 9 evenly spaced isovalues in the interval [-0.5, -0.001] for all level set functions, which provide 9 compact manifolds for each atom pair. Note that the level set function (<ref>) is always less than 0 and approaches 0 as the norm of 𝐱 increases. This interval is chosen as isovalues greater than -0.001 result in no change on the 0-th Betti number β_0 of manifolds for most atom pairs, and isovalues smaller than -0.5 leads to high computational cost, as finer grids are necessary to resolve those isosurfaces. To ensure that the computation of Laplacians is accurate and no topological information is missing due to numerical errors caused by low resolution, we require that at least 8 grid cells of the Cartesian grid are contained in each connected component of a manifold. We compute, for each manifold, the BIG Laplacian L_3,n under the normal boundary condition, for which the number of its 0 eigenvalues gives the 0-th Betti number β_0. We then use the 0-th Betti number β_0 and the first k non-zero eigenvalues of L_3,n, as the topological feature for the manifold. These k+1 features for each of the 9 compact manifolds for each atom pair, amount to (k+1)×9×40 topological features for each protein-ligand complex. While we only used 9 isovalues within this interval for generating the manifolds in our experiments, more isovalues can be considered, which gives a filtration of more manifolds for each atom pair, and finally leads to more topological features for each protein-ligand complex. 


The spectra of the 0-th Laplacian, which in our case corresponds to L_3,n under the normal boundary condition, have proven effective and successful in many machine learning tasks <cit.>. While the Laplacians of other orders could also be used for generating more topological features, we utilize, in this preliminary test, only the spectra of L_3,n as features for the protein-ligand complexes in the machine learning model due to the computation efficiency. The results, as shown in Sec.<ref>, indicate that these features are sufficient to validate our framework in the machine learning task for predicting the protein-ligand binding affinities. 



 §.§ Machine learning algorithm


The machine learning models for predicting protein-ligand binding affinities often fall into two categories depending on the type of input data: complex-based or sequence-based models. The complex-based methods are trained using features obtained from the 3D protein-ligand complexes, while the sequence-based models learn from the one-dimensional protein sequences and the ligand simplified molecular-input line-entry system (SMILES) strings. In our experiments, besides the topological features from the 3D protein-ligand complexes, we incorporate protein-ligand features obtained from sequence-based models to build consensus models. To be specific, we make use of the recent pre-trained transformer protein language model Evolutionary Scale Modeling-2 (ESM-2) <cit.>, and the pre-trained Transformer-CPZ model <cit.> for generating the protein and ligand features, respectively, and use their concatenation as inputs for the binding affinity prediction. The residue embeddings from the last layer of the pre-trained ESM-2 model esm.pretrained.esm2_t33_650M_UR50D are used as the protein features, while the embeddings from the last layer of the pre-trained Transformer-CPZ model chembl27_pubchem_zinc_512 are used as the ligand features.


With the topological features and the embedding features obtained from ESM-2 and Transformer-CPZ, we employ the Gradient Boosting Regressor (GBR) module from Scikit-learn 1.4.2 for predicting the protein-ligand binding affinities. We then use the consensus prediction from these models as the final results. The GBR parameters used in our experiments are: n_estimators=10,000, max_depth=5, min_samples_split=5, learning_rate=0.005, loss=squared_error, subsample=0.5, and max_features=sqrt. Changing these parameters does not result in significant differences. To address the randomness from the machine learning algorithm, we repeat each modeling process 20 times with different random seeds, and use the average of predictive results. The Pearson correlation coefficients (PCC) are used as the evaluation metric to assess the performance of our proposed models.



 §.§ Experimental Results








The number of topological features for each protein-ligand complex, as in Sec. <ref>, is given by (1+k)×9×40, where k denotes the number of the first k non-zero eigenvalues of the Laplacians. To find the optimal parameter k leading to the best performance of predictive modules, we carry out the five-fold cross-validation on the training set of each PDBbind dataset with varying values of k based on the average of PCC values. The results suggest that k=5 provides the best PCC values for the PDBbind-v2007 training set, while k=10 gives the best PCC values for the PDBbind-v2016 training set. This results in a total of 2,160 topological features for each protein-ligand complex in the PDBbind-v2007 dataset and 3,960 topological features for each protein-ligand complex in the PDBbind-v2016 dataset. These topological features, along with the concatenated protein-ligand features from ESM-2 and Transformer-CPZ, are then used as inputs of the gradient-boosting regressor for binding affinity prediction. 

In Table <ref>, we report the average PCC values and the average root mean squared error (RMSE) of our models on the test set for each PDBbind dataset using only the topological features from PHL, the model using only the transformer features (TF), and the consensus module using both types of features. With the incorporation of topological features, one can see a significant improvement in PCC values when using the proposed consensus model for each dataset, compared to the model using only TF features. The best performance is achieved when using the consensus model, yielding a PCC value of 0.826 with RMSE given as 1.954 for PDBbind-v2007 and 0.849 with RMSE 1.728 for PDBbind-v2016. In addition, we present the Pearson correlation coefficients obtained from our model and those in the previous studies, with results from <cit.>. As illustrated in Fig. <ref>, our model outperforms all the other models for the two PDBbind datasets. These results demonstrate the utility and effectiveness of our method in capturing the topological features. 




§ CONCLUSION

Motivated by the success of TDA in machine learning, particularly, topological deep learning on point clouds <cit.>, we presented a new method, persistent Hodge Laplacian (PHL) in the Eulerian representation,  for generating topological and geometric characterizations of data on manifolds. Our approach differs from existing state-of-the-art TDA methods on point clouds in that the construction of PHLs is on a regular Cartesian grid for problems defined on manifolds. Instead of relying on simplicial complexes for building persistent combinatorial Laplacians, we offer two discrete Hodge stars that mimic the continuous operator and developed both a continuous theory for mapping of normal forms across manifolds in a filtration to enable persistent cohomology analysis and the associated discrete construction on structure-persevering Cartesian grids. A proof-of-principle test on two benchmark datasets validates our model and discretization, highlighting the simplicity and promise of the proposed  PHL machinery for machine learning and deep learning predictions of data on manifolds. 




§ ACKNOWLEDGMENTS

This work was supported in part by NIH grants R01GM126189, R01AI164266, and R35GM148196, National Science Foundation grants DMS2052983, DMS-1761320, and IIS-1900473, NASA  grant 80NSSC21M0023,   Michigan State University Research Foundation, and  Bristol-Myers Squibb  65109. The authors thank Dr. Hongsong Feng for his help on transformer embeddings.  






























































 Paper Title: What comes after transformers? -- A selective survey connecting ideas in deep learning.









J. Schneider



University of Liechtenstein, Vaduz, Liechtenstein

johannes.schneider@uni.li
























What comes after transformers? – A selective survey connecting ideas in deep learningThis is an extended version of the published paper by Johannes Schneider and Michalis Vlachos titled “A survey of deep learning: From activations to transformers” which appeared at the International Conference on Agents and Artificial Intelligence(ICAART) in 2024. It was selected for post-publication.
    Johannes Schneider1
    August 5, 2024
===================================================================================================================================================================================================================================================================================================================================================================================================




§ ABSTRACT Transformers have become the de-facto standard model in artificial intelligence since 2017 despite numerous shortcomings ranging from energy inefficiency to hallucinations. Research has made a lot of progress in improving elements of transformers, and, more generally, deep learning manifesting in many proposals for architectures, layers, optimization objectives, and optimization techniques. For researchers it is difficult to keep track of such developments on a broader level. 
We provide a comprehensive overview of the many important, recent works in these areas to those who already have a basic understanding of deep learning. Our focus differs from other works, as we target specifically novel, alternative potentially disruptive approaches to transformers as well as successful ideas of recent deep learning.
We hope that such a holistic and unified treatment of influential, recent works and novel ideas helps researchers to form new connections between diverse areas of deep learning. We identify and discuss multiple patterns that summarize the key strategies for successful innovations over the last decade as well as works that can be seen as rising stars. Especially, we discuss attempts on how to improve on transformers covering (partially) proven methods such as state space models but also including far-out ideas in deep learning that seem promising despite not achieving state-of-the-art results. We also cover a discussion on recent state-of-the-art models such as OpenAI's GPT series and Meta's LLama models and, Google's Gemini model family.









§ INTRODUCTION
 
Transformers are widely regarded as the driving force behind artificial intelligence, e.g., of so-called foundation models<cit.> such as OpenAI's ChatGPT or early models such as BERT. Transformers are at the top of most machine learning benchmark leaderboard including computer vision, speech, and natural language processing. A major advantage of deep learning is its layered, modular structure. It enables construction of models from individual components in a flexible manner. Researchers have created a large selection of layers, architectures, and objectives. Keeping up with the rapid developments in AI models is a difficult task. 

There exist multiple reviews with a narrow focus such as large language models (e.g. <cit.>) and convolutional neural networks (e.g. <cit.>). Previous studies <cit.> with a wider focus have become dated and miss new developments such as transformers and self-supervised learning. Furthermore, no survey or novel text book such as <cit.> has focused on alternatives towards transformers. 
Multiple works have discussed particular shortcomings of transformers such as computational inefficiency, e.g., due to quadratic time complexity of (naive) self-attention <cit.>. Thus, there is no work that looks at recent deep learning holistically integrating general ideas in deep learning and transformers, emphasizing alternatives or ways ahead. However, taking a broader and more holistic look at different (sub-)disciplines can be highly beneficial: For instance, NLP and computer vision have reciprocally shaped one another; CNNs first emerged in computer vision and subsequently found applications in NLP, whereas transformers originated in NLP and were later incorporated into computer vision. Consequently, breaking down the barriers between disciplines proves to be highly advantageous.

This paper is driven by the goal of examining the recent advancements in deep learning from an more encompassing perspective, rather than concentrating on a specific specialized field. We contend that such an approach is crucial, especially as significant new developments have decelerated; currently, the vast majority of models employs the “old” but still mostly state-of-the art transformer architecture presented already in 2017<cit.>.

Providing a comprehensive overview of the field is challenging, if not impossible, given the vast number of articles published annually and the constant expansion of pertinent topics. Our strategy is to choose influential works through (i) usage statistics, (ii) specialized surveys, and (iii) for transformer alternatives, we rely on public debates (as well as our own literature search). We also offer an invigorating discussion of shared design patterns across areas that have been successful.[This paper extends the conference paper <cit.> by engaging more deeply in discussing transfromer alternatives among other aspects]




















§ OVERVIEW
 
Figure <ref> outlines the topics addressed in this survey.
In Part 1, We start our discussion with the key topic of this survey, i.e., transformers. We refer readers with lack of knowledge of basic architectural components of transformers such as attention to Part 2 of this work or text books<cit.>. However, Part 2, is also recommended for more experienced readers to reflect on recent ideas in deep learning.
In Part 1, we elaborate on current architectures including commercial and open-source models but also go beyond by discussing more far-out ideas in deep learning. The implementations of these ideas have often not yet shown state-of-the-art results or even near state-of-the-art results, but nevertheless provide interesting ideas that might just need  further thoughts. 

In Part 2, our exploration focuses on deep learning design, encompassing both objectives and training methods. Particular emphasis has been placed on work validated by usage metrics from the well-known platform "Paperswithcode.com." Our inclusion criteria feature impactful research published since 2016 and emerging contributions from 2020 onward that have quickly risen to prominence.

The coverage of each subject area is influenced by the volume of recent research and its fundamental importance. We omit discussions on data or computational strategies, such as data augmentation, model compression, and distributed machine learning, due to space constraints. This necessitated a discerning selection of model types, excluding significant categories like multi-modal models and autoencoders.

















§ TRANSFORMERS 
 
Transformers have emerged as the leading architecture in deep learning. When paired with self-supervised training on extensive datasets, they have achieved top performance across numerous benchmarks in NLP (refer to <cit.> for a detailed survey) and computer vision (as outlined in <cit.>). Introduced in 2017 <cit.>, numerous adaptations have been developed to address challenges such as computational demands and data efficiency in the original transformer model. We discuss classical and current models in Section <ref>, emerging alternatives to transformers and trends in Section <ref>  and more far-out ideas going beyond transformers in Section <ref>.  





It is often stated that transformers possess lower inductive bias (e.g., compared to CNNs and RNNs), making them more adaptable. Consequently, they need a larger volume of training data to offset the reduced inductive bias. Transformers are commonly pre-trained using self-supervised learning, and oftentimes fine-tuned towards specific tasks on labeled data. Classical deep learning sequence models such as recurrent neural networks (RNNs) often exhibit a time-dependency upon inference and training, e.g., after processing of one token a hidden state must be updated before the next token can be processed. This limits parallelization and leads to other issues such as vanishing gradients for long sequences. Transformers, in contrast, allow to process the entire sequence at once using self-attention to predict the next token, i.e., they are much more parallelizable than sequential models such as RNNs. Unfortunately, naive implementation of self-attention require quadratic run-time, which makes them difficult to use for long sequences.
Ideas like mixture of experts (MoE) transformer-based model have been employed to reduce computational burdens (especially during inference). MoE uses a learned routing function to select only a subset of the model for processing, which saves on computation. The idea of MoE dates back well before the year 2000 <cit.>. It has gained traction more recently as a means to reduce computational effort, in particular, during inference.




 §.§ Classical and state-of-the-art transformer architectures
 
The original transformer<cit.>, designed for NLP, utilizes an encoder and decoder similar to earlier recurrent neural networks. The architecture features multiple layers of transformer blocks, as depicted in Figure <ref>. Essential components include multi-head attention, layer normalization, and skip connections. For enhanced efficiency, several commercial and open-source models adopt the mixture of experts technique, which selectively computes outputs from a subset of models within an ensemble. Additionally, positional encodings and input embeddings are crucial. The absolute positional encodings PE for position pos in <cit.> are calculated using sinusoidal functions with varying frequencies:



   PE(pos, 2i) = sin(pos/10000^2i/d) 

   PE(pos, 2i+1) =  cos(pos/10000^(2i)/d) 

where i represents the encoding dimension and d the total number of dimensions. This method was chosen because it allows relative positions, potentially as significant as absolute positions, to be a linear function of the absolute position encodings.



Bidirectional Encoder Representations from Transformers(BERT) <cit.>:
The encoder of the transformer architecture generates contextual word embeddings through a masked language model pre-training objective and self-supervised learning. In this approach, the model is tasked with predicting randomly selected, masked input tokens based on their surrounding context, thus providing it with bidirectional information—that is, access to tokens both before and after the masked words. Unlike traditional next-word prediction methods, where no tokens following the predicted word are presented. Additionally, the model is required to determine whether a pair of sentences (A,B) are consecutive sentences from the same document or two unrelated sentences. Once pre-trained via self-supervised methods, the model can be fine-tuned for specific tasks using labeled data.

The original BERT model has undergone various improvements, for instance, <cit.> minimized BERT's computational demands, and <cit.> extended training durations, utilized longer sequences, larger batches, and more data. These enhancements have led to more robust and generalizable representations.







Text-to-Text Transfer Transformer (T5)<cit.> 
considers all text-based language models as systems that produce output text from an input text. This differs from BERT<cit.> through the use of causal masking in training, which blocks the network from seeing "future" tokens of the target. Additionally, T5 diverges in its pre-training tasks.



BART<cit.> is a denoising autoencoder used for pretraining sequence-to-sequence models with a conventional transformer-based machine translation architecture. It has proven effective in tasks such as language generation, translation, and comprehension. The training involves corrupting text using a variety of noising functions, including token deletion, masking, sentence permutation, and document rotation. The learning process involves reconstructing the original text from its corrupted form. The variety of noising techniques utilized can be attributed to BART's adaptation and generalization of concepts from previous models like BERT and GPT<cit.>, combining a bi-directional encoder (similar to BERT) with an autoregressive decoder (similar to GPT).



XLNet 
<cit.> 
This approach merges the benefits of autoregressive modeling, as seen in GPT, where it predicts the next token, with the denoising auto-encoding capabilities of BERT<cit.>, which involves reconstructing x from a noisy input x̂ created by masking words in x. It achieves this by using a permutation language model that samples a permutation of Z=z_0,z_1,...,z_T-1 of the sequence (0,1,2,...,T-1) aiming for the objective:
maxp(u_z_T|u_z_0,...,u_z_T-1)
Importantly, the inputs themselves are not actually permuted, which would be impractical and inconsistent with later fine-tuning tasks. Instead, the permutation influences the attention mask to ensure that the factorization order determined by Z is preserved.

The Vision Transformer <cit.> 
splits an images into small patches, each of which is flattened and embedded linearly along with position embeddings. These embedded vectors for each patch are then processed by a standard transformer encoder.

The Swin Transformer <cit.> 
constructs hierarchical feature maps in computer vision applications instead of just a single resolution feature map. Additionally, it limits the computation of self-attention to within a local window, which helps reduce the overall computation time. 
  


PaLM 2: The original PaLM model<cit.>, containing 540 billion parameters, parallels other prominent models like GPT-3 in scale. Its technical advancements primarily focus on the scalability of model training, enabling a single model to be trained efficiently across tens of thousands of accelerator chips. Slight modifications were made to the original transformer architecture<cit.>, such as incorporating SwiGLU activations, represented by Swish(xW)·xV , where Swish is defined in Eq. <ref>. Additional changes include alternative positional embeddings optimized for longer sequences, multi-query attention for faster computation, the elimination of biases for improved training stability, and shared input-output embeddings.

PaLM 2<cit.>, the more advanced successor of PaLM, distinguishes itself by utilizing a different mixture of datasets that incorporate a broader range of languages and domains, such as programming languages and mathematics. Although it uses the traditional transformer architecture, it operates with a smaller model size yet employs more computational resources for training. Moreover, it broadens its pre-training objectives beyond simple next-word or masked-word prediction, incorporating a variety of tasks to enhance its performance.



OpenAI's GPT to GPT-3 on to ChatGPT 3.5 and GPT-4o: 
The first version of Generative Pre-trained Transformer (GPT) is based only on the transformer's decoder to predict tokens one after the other. GPT<cit.> initially undergoes unsupervised pre-training, typically followed by supervised fine-tuning. The pre-training process involves a large corpus of tokens U=(u_0,u_1,...,u_n-1) and focuses on maximizing the probability of predicting subsequent tokens based on prior ones:
L(U)=∑_i logp(u_i|u_i-k,...,u_i-1) where k represents the context window size, and the conditional probability is computed using a neural network, specifically a multi-layer transformer decoder<cit.> that omits the encoder as per <cit.>. Additionally, this method reduces the memory usage of the attention mechanism.

Building on the foundational GPT model, GPT-2 <cit.> introduces a few key enhancements, such as relocating layer normalization to the beginning of each sub-block and adding an additional normalization after the last self-attention block. Moreover, the initialization of residual weights has been adjusted, and there has been an increase in the vocabulary, context window, and batch sizes.

GPT-3<cit.> maintains a structure nearly identical to GPT-2, but it expands dramatically in scale, with more than 100 times the number of parameters and variations in the volume of training data.

(Chat)GPT-3.5 <cit.> is closely related to InstructGPT<cit.>, designed with a focus on adhering to user directives. InstructGPT refines GPT-3's capabilities through a dual-phase fine-tuning approach: (i) guided by demonstrations from labelers using supervised learning and (ii) leveraging human evaluations of model responses in a reinforcement learning framework. ChatGPT adheres to this methodology, where (i) supervised learning involves human AI trainers engaging in scripted dialogues, where humans act as both the user and the AI agent. These dialogues are subsequently merged with the InstructGPT conversations, which have been reformatted into dialogues. In Phase (ii), AI trainers assess the quality of responses generated by ChatGPT, choosing from multiple possibilities for a randomly chosen model-generated text. This ranking informs the reinforcement learning phase to refine the model’s performance. 

The technical specifications of GPT-4, the follow-up to ChatGPT-3.5, have not been officially released as per the technical report<cit.>, although some unofficial sources <cit.> claim to have insights. The report suggests that GPT-4 retains many characteristics of ChatGPT 3.5 but extends its capabilities to multi-modal functions, allowing it to process images as well. It highlights significant enhancements in training efficiency and a notable achievement in predicting the performance of large-scale models based on the outcomes of smaller models, which may have been trained on less data. This aspect is particularly crucial given the significant impact of computational costs and time on the development of extensive deep learning models.

Later models by OpenAI such as GPT-4 Turbo and GPT-4o <cit.> focused on improving efficiency and response times and multi-modality, including audio, with comparable performance or modest gains on benchmarks. While there are no technical details available, models were likely down-sized and the amount of training was increased, possibly with new or improved data.



Meta's Open source LLama series: LLama is Meta's open-source model family of LLMs, ranging from LLama<cit.>, LLama-2<cit.> on to LLama-3.1<cit.>. The LLama-3 paper also discussed multi-modal experiments. The largest model LLama 3.1 405B achieves similar performance as commercial models such as GPT4o and Claude Sonnet 3.5<cit.>. All models are dense transformers rather than MoE to improve training stability. Improvements, in particular for LLama-3 originated from scaling data, models and training and changes in post-processing, e.g, using direct preference optimization (DPO)<cit.> rather than reinforcement learning. That is, LLama-3 included no major architectural changes compared to prior LLama models and, in turn, to the original transformer architecture or self-supervised pre-training proceducre. Minor updates were the usage of grouped query attention <cit.> and preventing self-attention between different documents (using an attention mask). Architectural details including hyperparameters are provided in <cit.>.





Gemini, Claude and other models:
The Gemini family has been briefly described in a technical report <cit.> and the version 1.5 family in <cit.>. However, technical details are mostly missing. It is also based on the transformer architecture by Vaswani et al. with improvements in architecture such as multi-query attention<cit.> and optimization to help training at scale and improve efficiency. It is multi-modal including video. The version 1.5 improved long-context understanding of inputs and it introduced a sparse mixture of expert (MoE) transformer-based model. The company `Mistral.AI' also launched Mixtral, an opensource MoE model<cit.>. GPT-4 is also alleged to be based on MoE with eight experts <cit.>. Though Anthropic's Claude models are claimed to perform well, there was no official information on its architecture or training but only a model card <cit.>.













 §.§ Trends and Alternatives to Transformers
 
The original transformer architecture is still the key model for LLMs and a key driver for generative AI – as elaborated in the prior section <ref>. The prior section also highlighted a few trends that emerged in the last few years to improve transformers. Many focus on computational aspects by altering existing transformers, e.g., using a MoE to reduce computation or altering attention mechanisms, while others are not driven by computation in particular and suggest more radical ideas, e.g, by combining older models like RNNs with new elements of transformers. 



Combining Sequence Models and Transformers:
That is, there is a stream of research that aims to take elements from sequence models such as RNNs and LSTMs to transformers. However, one alleged success factor for transformers in the first place is their parallelizability in contrast to classical sequence models.

<cit.> proposed xLSTM, which improves on the classical LSTM cell by using exponential gating with memory mixing and a new memory structure. It allows for improved parallel training. While the results are promising, they stem mostly from smaller models (1.3B) and, thus, need to be validated using larger models.

Receptance weighted key value (RWKV) <cit.> combines the well parallelizable training of transformers with the fast inference of RNNs. For inference they achieve linear time complexity in sequence length S and memory needs linear in the (embedding) dimension. In contrast, classical transformer require a factor S more for both time and space(memory). The architecture bares similarity with an ordinary transformer but they use a variant of linear attention to focus on channels rather than the classical dot-product token attention.

Retentive networks<cit.> replace multi-head attention with multi-scale retention. They derive it from a recurrent model by adding context awareness. In a recurrent setting, a state s is updated using a matrix Q. Context awareness means that the matrix Q is also mulitplied by the context X derived from the current input. They also perform matrix diagonalization to obtain an easily parallizable formulation.

State-space models -  Replacing attention with a merger of CNNs and RNNs:

State-space models (SSM) are well known in natural science dating back more 60 years<cit.>. They can be written as x'(t)=Ax(t)+Bu(t), y(t)=Cx(t)+Du(t) with matrixes A, B, C, and D. They have been adopted for sequence modeling<cit.>, where they can be interpreted as “a combination of recurrent neural networks (RNNs) and convolutional neural networks (CNNs), with inspiration from classical state space models”<cit.>.  In particular, <cit.> focuses on long range dependencies leading to the so-called structured state space sequence model (S4) <cit.>. Technically, they apply a low-rank correction to matrix A allowing the matrix to be diagonalized stably. Special SSM layers have also been proposed for LLMs achieving comparable performance on small models <cit.>. A major evolution is MAMBA<cit.>, which neither requires attention nor MLPs. It introduces a selection mechanism that allows to select data in an input-dependent manner. They also create a hardware aware algorithm and simplify prior SSM architectures by merging with the MLP layer into a single block.


Forgoing attention based on MLPs only and gated convolutions:
One of the simplest architectures relies on multi-layer perceptrons (MLP)s only. The MLP-mixer<cit.> has been shown to achieve state-of-the-art results in computer vision without convolutions and self-attention. It relies on MLPs that are applied on spatial locations or feature channels instead. It also maintaining other common architectural elements such as skip-connections and layer normalization. Recent work has also improved on MLPs for mixing, e.g., by using Monarch matrices <cit.>.


A drop-in replacement for attention is Hyena <cit.>. It is based on long convolutions and data-controlled gating. The paper also discusses three factors that must be fulfilled by attention alternatives but are often missed, i.e., (i) data control: Attention is a linear data controlled operator, (ii) sublinear parameter scaling: parameters (of attention layers) does not depend on sequence length, and (iii) unrestricted context: interaction between any tokens is possible, e.g., there is no locality.

Models to systems - Integrating planning and external tools:
While a single LLM is often seen or believed to be capable of doing any task, there are a number of approaches that aim at building systems rather than improving only the deep learning model in isolation. That is, LLMs are enhanced with other components that interact to improve its performance. For example, <cit.> claim that large language models (LLMs) cannot plan but can generate ideas that are critiqued by separate models, which may be domain-specific and not LLMs. Ideas like Retrieval Augmented Generation (RAG)<cit.> have altered the way how LLMs process prompts by first enhancing them with external knowledge, which helped to reduce hallucinations. The usage of external tools by the LLM such as calculators has also improved the performance of LLM-based systems for many tasks<cit.>.



 §.§ Far-out models and ideas
 
The prior section contained some interesting ideas that have arleady shown promising outcomes compared to transformers in empirical evaluations. However, one might also seek to leverage more revolutionary ideas that have been proposed in the context of deep learning but have not yet yielded the desired success.

Capsule networks - Returning vectors instead of scalars:
Capsule networks have been proposed in 2011<cit.>. The essential idea is that a neuron outputs a vector rather than a scalar. Hinton et al. motivate the usage of a vector output in the context of computer vision. They argue that a (visual) feature is often characterized by multiple instantation parameters such as position, scale, lighting, etc. that resemble a vector rather than just a scalar. A capsule learns an implicitly defined visual entity across some limited transformations such as lighting or scaling. It returns the probability that the entity is present and a set of instantation parameters (e.g., lighting or scaling). They argue that a key advantage of this approach is the recognition of whole-part relationships, e.g., to detect a face one might identify a mouth and nose and both must have appropriate scaling and orientation. The training of the network requires pairs of transformed images to learn capsules that can extract pose parameters from inputs. Capsules themselves might consist of classical elements such as dense layers and gates. Capsule networks have been further improved since their introcduction in 2011<cit.>. They managed to achieve good performance on simple daasets such as MNIST but failed to deliver close to state-of-the-art performance on challenging datasets such as ImageNet.

Spiking neural networks - Adding temporality through non-synchronous activating neurons:
Spiking neural networks <cit.> exist for more than 25 years, achieving remarkable, but no state-of-the-art results. Spiking neural networks add a temporal dimension to neural networks. A neuron activates once its “potential” exceeds a threshold through transmission of spiking signals to its adjacent neurons. The spike increases their potential. Potentials decrease upon sending spikes but also spontaneously over time (without transmission). Thus, in contrast to non-spiking networks, transmission of signals in the network can occur in non-periodic, non-synchronized cycles among neurons. As such the time, when a signal occurs also resembles information in addition to the actual information in the spike. This effectively allows to perform time-coding as common distributed systems <cit.>. 

Neurosymbolic AI - Leveraging domain specific (symbolic) languages:
There is also an (ongoing) debate on the value of symbolic approaches in conjunction with neural networks relying on continuous representations. For example, in 2017 a neuro symbolic approach for program synthesis was proposed<cit.> that can generate program code in a domain specific language from input-output examples. While the synthesis appears trivial compared to current program skills of LLMs, the paper still contains a number of interesting ideas, e.g., related to how to search the space of possible programs and how to incrementally refine programs. Other neuro-symbolic approaches have also performed concept extraction based on neural networks and used domain specific languages to perform quasi-symbolic program execution <cit.>. A comprehensive overview with a focus on NLP is given in <cit.>. 


Embrassing ideas from the mathematics community:
Classical ideas from mathematics have also entered the domain of deep learning. For example, residual neural networks (ResNets) <cit.> essentially solve ordinary differential equations(ODE) <cit.>. A simple ODE is dy/dt=f(y,t) and the inital condition y(t_0)=y_0. Its solution y_n at discretized time steps can be computed (recursively) using the Euler method: 
    y_n+1=y_n+f(y_n,t_n)· (t_n+1-t_n)

A layer n in a residual network essentially performs the same computation:
y_n+1=y_n+f(y_n,t_n)
Thus, a sequence of residual blocks can be interpreted as a solution of the ODE with the Euler method with the initial condition y(0)=X, where X is the input to the network. Viewing ResNets as ODEs allows to utilize mathematical tools such as ODE solvers and, in turn, might also have benefits with respect to memory and parameter efficiency. The idea of reformulating learning systems has also led to performance improvements <cit.>. That is, e.g., <cit.> introduce time-continuous RNNs described as a linear first order system.

Embrassing ideas from neuroscience and psychology:
While AI does not seek to rebuild the brain, (articial) neural networks structurally still share similarity with the real neural networks in our heads. Thus, almost any finding in the neuroscience community might be used as an inspiration for improving artificial neural networks. The main hinderness to adopting neuroscience in deep learning research is “that we simply do not have enough information about the brain to use it as a guide”<cit.>. For instance, the bain has (neurological) feedback loops: A neuron receives and processes a signal, and can then send a response back to the original source. While numerous attempts have been made to leverage this idea, overall success has been limited, i.e., it is essentially a call for more research. For example, <cit.> argued that feedback (or recurrent circuits) are critical for fast object identification. In another idea from psychology, a reflective architecture was introduced that leverages explanations computed from gradients (GRADCAM) during learning but also during inference<cit.> drawing inspirations from Kahneman's fast thinking (System 1) and slow (reflective) thinking (System 2).  The idea of ordering samples by difficulty to improve learning of deep learning models, i.e., curriculuum learning <cit.>, can be traced back to the idea of shaping in animal training (e.g., <cit.>).











§ LOSS FUNCTIONS AND OPTIMIZATION

We elaborate on fundamental loss functions and optimizers.





 §.§ Loss Functions

Loss functions, as reviewed in <cit.>, typically include several terms including a regularization term. While these functions are generally tailored to specific tasks, certain universal principles can be applied across various tasks. It is common to combine multiple loss terms using a weighted approach. Frequently, enhancements to previous studies are achieved merely by modifying the loss function.
 

The Triplet Loss 
<cit.> was introduced for Siamese networkss, though its roots trace back to earlier work<cit.>. The overarching principle involves comparing a given input with both a positive and a negative input, aiming to maximize the association with positively related inputs and minimize it with negative ones. It operates on input pairs (x,y), each processed by a distinct but architecturally identical network.  The goal is to maximize the joint probability p(x,y) for all pairs (x,y):
L(𝒱_p,𝒱_n)    =-1/|𝒱_p|·|𝒱_n| ∑_x ∈𝒱_p∑_y ∈𝒱_n logp(x,y)  

   =-1/|𝒱_p|·|𝒱_n| ∑_x ∈𝒱_p∑_y ∈𝒱_n log(1+e^x-y)  

Here, 𝒱_p and 𝒱_n are the positive and negative score set respectively.














The Supervised Contrastive Loss<cit.>  
groups clusters of points from the same class together in the embedding space while pushing apart samples from different classes. It is designed to utilize label information more effectively than cross-entropy loss, optimizing the separation and aggregation based on class identity.

ℒ_i^sup   =-1/2N_ỹ_i-1·

   ∑_j=1^2N1_i≠j·1_ỹ_i=ỹ_j·logexp(z_i·z_j/τ)/∑_k=1^2N1_i≠k·exp(z_i·z_k/τ)
 
where N_ỹ_i denotes the total number of images in the minibatch with the same label ỹ_i as the anchor i. The overall loss is calculated as the sum of the losses for all anchor points  i, i.e., ℒ=∑_iℒ_i^sup. This loss formulation is particularly advantageous for supervised learning due to its:

    
  * Ability to generalize across an arbitrary number of positive examples.
    
  * Increased contrastive effectiveness as the number of negative examples grows.


The Cycle Consistency Loss<cit.> 
is specifically designed for unpaired image-to-image translation using generative adversarial networks. It facilitates the learning of mappings between two distinct image domains X and Y. Optimizing the loss supports the learning of mappings G: X → Y and F: Y → X so that one reverses the other, i.e., F(G(x))≈ x and G(F(y)) ≈ y.
L(G,F)   = 𝔼_x ∼p_data(x)[||F(G(x)) - x||_1]

   + 𝔼_y ∼p_data(y)[||G(F(y)) - y||_1]


Focal Loss<cit.> 
modifies the standard cross-entropy loss to concentrate learning efforts on hard-to-classify samples. It incorporates a factor (1-p)^γ, where p is the probability of a sample from the cross entropy loss and γ is a freely adjustable parameter.
L(p)=(1-p)^γlog(p)



 §.§ Regularization


Regularization techniques, as reviewed in <cit.>, are critically beneficial for deep learning applications. Explicit regularization incorporates an additional loss term R(f) for the network f into the loss function L(x) for data (x_i,y_i) with a balancing parameter λ.
min_f ∑_i L(x_i,y_i)+λR(f)
Implicit regularization encompasses all other forms of regularization, such as early stopping or employing a robust loss function. Common methods like L2-regularization and dropout<cit.>, which involves setting the activations of a random subset of neurons to zero, are among the most extensively used techniques.



Entropy Regularization 
<cit.> is designed to promote diversity, particularly through asynchronous methods in deep reinforcement learning <cit.>. This approach enhances the diversity of actions in reinforcement learning by avoiding excessive optimization towards only a small portion of the environment. Entropy is calculated based on the probability distribution of actions as determined by the policy as:
 H(x)=∑_x π(x)·log(π(x))






Path Length Regularization <cit.> 
<cit.> focuses on enhancing the image quality for generative adversarial networks by ensuring consistent step lengths in the latent space and changes in the generated images. That is, changes in the latent space representation lead to (proportional) changes in the generated image (and vice versa). The principle behind this is to facilitate a fixed-size step in the latent space 𝒲  to yield a consistent, non-zero change in image magnitude. This approach aims to improve the conditioning of GANs, which can simplify the process of architecture search and generator inversion. Gradients, with respect to 𝐰∈𝒲, originating from random directions in the image space should be almost equal in length independent of 𝐰 or the image space direction. The local metric scaling characteristics of the generator g: 𝒲→𝒴 are stated by the Jacobian matrix 𝐉_𝐰 = δg(𝐰)/δ𝐰. The formulation for the regularizer is:

 𝔼_𝐰,𝐲 ∼𝒩(0, 𝐈) (||𝐉^𝐓_𝐰𝐲||_2 - a)^2

where y corresponds to random images with pixel values that follow a normal distribution, and w ∼ f(z), where z is also normally distributed. The constant a is defined as the exponential moving average of ||𝐉^𝐓_𝐰𝐲||_2. The paper also avoids the computationally expensive, explicit computation of the Jacobian.

DropBlock<cit.> 
 selectively drops correlated regions within feature maps instead of dropping features independently. This method is particularly effective for convolutional neural networks, where feature maps tend to show spatial correlations and a real-world feature typically corresponds to a contiguous spatial area within these maps.

R_1 Regularization 
<cit.> penalizes the  discriminator in generative adversarial networks based on the gradient to stabilize training:

R_1(ψ) = γ/2E_p_D(x)[||∇D_ψ(x)||^2]
Technically, the regularization term imposes penalties on gradients that are orthogonal to the data manifold, aiming to refine the learning process.

 




 §.§ Optimization
 
Optimization, as explored in <cit.>, involves determining the best set of network parameters to minimize the loss function. The two most widely recognized techniques are stochastic gradient descent (SGD) and Adam. Neither technique consistently surpasses the other across all scenarios in terms of generalization performance. SGD has historical roots extending back to the 1950s <cit.>, whereas Adam was developed more recently, in 2014 <cit.>.


Adafactor 
 <cit.> enhances the efficiency of the Adam optimization algorithm by reducing its memory requirements. This is achieved by keeping only row- and column-wise statistics of parameter matrices, instead of storing detailed per-element information.
 
Layerwise adaptive large batch optimization (LAMB)<cit.> 
builds on Adam and accelerates training by utilizing large mini-batches. It applies per-dimension and layerwise normalization to further enhance the optimization process.









Decoupled Weight Decay Regularization for ADAM: 


AdamW<cit.> leverage a seemingly trivial observation: The original Adam optimizer updates weights with (L2-)regularization after calculating the gradients for Adam. However, it intuitively makes more sense that moving averages of gradients should exclude regularization.







RAdam and AMSGrad: 
Both techniques address the convergence issues associated with Adam. Rectified Adam (RAdam)<cit.> adjusts the variance of the adaptive learning rate, which is initially large. This approach is akin to the warm-up heuristic, where small initial learning rates are beneficial. AMSGrad<cit.> uses the maximum of past squared gradients instead of the exponential average to enhance stability and convergence.


Stochastic Weight Averaging: 
Naive averaging of weights from different epochs during stochastic gradient descent, using either a constant or cycling learning rate, has been shown to improve performance<cit.>.

Two Time-scale Update Rule(TTUR): 
TTUR<cit.> enhances the training of generative adversarial networks by employing separate learning rates for the discriminator and the generator. When the generator is fixed, the discriminator can reach a local minimum. This approach remains effective even if the generator converges slowly, for instance, by using a smaller learning rate. By allowing the generator to more deeply assimilate feedback from the discriminator before exploring new regions, this method aids in the convergence of GANs and can lead to improved performance.
Besides ensuring convergence, the performance may also improve since the discriminator must first learn new patterns before they are transferred to the generator. In contrast, a generator which is overly fast, drives the discriminator steadily into new regions without capturing its gathered information.













Sharpness-Aware Minimization<cit.> 
minimizes both the loss value and sharpness, which enhances generalization. It seeks parameters within neighborhoods of low loss values, rather than focusing solely on parameters that individually exhibit low loss. The loss function is defined as:

min_w max_||ϵ||_p≤ρ L(w+ϵ)






§ SELF, SEMI-SUPERVISED AND CONTRASTIVE LEARNING
 
Semi-supervised learning utilizes a large volume of unlabeled data alongside a small amount of labeled data, as reviewed in <cit.>. Self-supervised learning, on the other hand, generates (pseudo) labels from artificial tasks, reducing the need for manually labeled data. Both approaches alleviate the burden of collecting human-annotated data. Combining self-supervised (pre-)training with fine-tuning on a small human-labeled dataset can achieve state-of-the-art results. This paradigm has significantly expanded in recent years (surveyed in <cit.>). It is often integrated with contrastive learning, which aims to learn the distinction between similar and dissimilar data. By automatically distorting data to varying degrees, creating "pseudo-labeled" data for self-supervised learning becomes straightforward.
















The simple framework for contrastive learning (SimCLR)<cit.> 
maximizes the agreement between two inputs derived from different augmentations of the same data sample. Augmentations can include random cropping, color distortions, and Gaussian blur. A standard ResNet<cit.> is used to obtain representation vectors, which are then further processed through a simple MLP before applying the contrastive loss.

Bootstrap Your Own Latent (BYOL) 
<cit.> employs both an online network and a target network, each with identical architectures comprising an encoder, a projector, and a predictor, but with separate weights. The parameters of the target network are updated as an exponential moving average of the online network's parameters. The task of the online network is to predict the representation of the target network given an augmentation of the same input.




Barlow Twins<cit.> 
rely on an objective function that aims to reduce cross-correlation C between outputs for a set of image Y^A and their distorted versions Y^B as close to the identity as possible, i.e., the loss (including  λ as a tuning parameter) is:

use an objective function aiming to minimize the cross-correlation 
C between outputs for a set of images Y^A and their distorted versions Y^B, bringing it as close to the identity matrix as possible. The loss function, incorporating λ as a tuning parameter, is given by:
L=∑_i (1-C_i,i)^2+λ·∑_i∑_j≠i C_i,j^2



Momentum Contrast (MoCo) 
<cit.> constructs a dynamic dictionary via an encoder using unsupervised contrastive learning. During training, it performs look-ups, ensuring that an encoded query closely matches its corresponding encoded key while being dissimilar to other keys. The dictionary functions as a queue of data samples: for each mini-batch, new encoded samples are added, and the oldest mini-batch is dequeued. The key encoder's parameters are updated using a momentum-based moving average of the query encoder, promoting consistency over time.




Noisy Student: 
<cit.> outlines a method where an CNN-based EfficientNet model is initially trained on labeled data. This trained model then acts as a teacher, generating pseudo labels for unlabeled images. Subsequently, a larger model is trained on the combined labeled and pseudo-labeled data. This process is iteratively repeated, with each student model becoming the teacher for the next iteration. During the training of the student model, techniques like dropout and data augmentation are employed to introduce noise, making the learning process more challenging and allowing the student to surpass the performance of the teacher.

FixMatch <cit.> 
predicts the label of a weakly-augmented image. If the confidence in this prediction exceeds a certain threshold, the model is then trained to produce the same label for a strongly-augmented version of the image.







§ ARCHITECTURES AND LAYERS

We elaborate on four key types of layers: activation layers, skip connections, normalization layers, and attention layers. This is followed by a discussion of various contemporary architectures based on transformers and graph neural networks.





 §.§ Activation

Activation functions are typically non-linear and significantly influence gradient flow and learning. Early activation functions, such as sigmoid and tanh, were commonly used from the 1960s through the early 2000s. However, these functions can cause training difficulties in deep networks due to the vanishing gradient problem when they saturate. The introduction of the rectified linear unit (ReLU) in 2010<cit.> was a breakthrough. Although the original ReLU remains widely used, transformer architectures have introduced other activation functions and ReLU variants. Most of these alternatives share the qualitative behavior of ReLU, where outputs for negative inputs are of small magnitude and outputs for positive inputs are unbounded (see <cit.> for a survey).

Gaussian Error Linear Units (GELU)<cit.>  
These functions weigh inputs by their percentile (ReLUs only use the sign). The activation function is the product of the input and the standard Gaussian cumulative distribution function Φ(x), i.e.,
GELU(x)= x·Φ(x)






The Mish activation<cit.> 
originates from systematic search inspired by Swish and ReLU: 
   f(x)=x·tanh(soft^+(x))

    with  soft^+(x):=ln(1+e^x)

In contrast, the Swish activation<cit.>  is:
   f(x)=x·sigmoid(βx)  Here β is a learnable parameter.









 §.§ Skip connections

Skip connections were introduced for residual networks<cit.>. In their simplest form, the output y for an input x of a block L of layers with a skip connection is y(x)=L(x)+x.
The term "residual" was used in the original paper because the layer 
L is tasked with learning a residua L(x)=H(x)-x rather than the desired mapping H itself. Since then, skip connections have been utilizied in a number of variations.



ResNeXt Block<cit.>: 
This split-transform-merge approach for residual blocks involves evaluating a set of residual blocks concurrently and then aggregating their outputs back into one output.
A Dense Block<cit.> 
receives inputs from all preceding layers with matching feature map sizes and connects to all such subsequent layers.

Inverted Residual Block<cit.>: 
By reversing the channel width sequence to a narrow-wide-narrow order from the original wide-narrow-wide configuration<cit.>, and using depthwise convolutions for the wide layer, parameters are reduced, and residual blocks execute more quickly. Furthermore, the activation function of the last layer within a block is skipped.









 §.§ Normalization

With the advent of batch normalization<cit.>, the concept of normalization has significantly enhanced training speed, stability, and generalization in neural networks including almost all architectures. However, its necessity is debated<cit.>; for certain applications, careful initialization and learning rate adjustments may render normalization partially redundant.

The principle behind normalization is to transform a value x to a normalized value x̃, by subtracting the mean μ and scaling by the standard deviation σ, i.e.,
x̃ = x-μ/σ.
Normalization approaches differ in the computation of μ and σ, e.g., μ and σ can be computed across different channels. 

Layer Normalization: 
Normalization statistics are calculated using summed inputs<cit.> for a layer L with |L| neurons as:
   μ= 1/|L| ∑_i=0^|L|-1 a_i abcd σ=  √(1/|L| ∑_i=0^|L|-1 (a_i-μ)^2) 
Unlike batch normalization, this method does not limit the batch size and eliminates inter-batch dependencies, making it suitable even for batch sizes as small as one.

LayerScale<cit.> 
has been established for transformers as a per-channel multiplication of outputs of a residual block with a diagonal matrix:

   x_l'= x_l+diag(λ_1,...,λ_d)·SA(η(x))

   x_l+1= x_l'+diag(λ_1,...,λ_d)·FFN(η(x))

SA is the self-attention layer, FFN is the feed forward network, and η the layer-normalisation (see Figure <ref>).

Instance Normalization<cit.> 
calculates for a four-dimensional input, such as an image of height H, width W, channels C, and batch size T:
   μ_t,c = 1/HWT ∑_t<T,w<W,h<H x_t,c,w,h

   σ_t,c =  √(1/HWT ∑_t<T,w<W,h<H (x_t,c,w,h-μ_t,c)^2)

This method can be applied, for example, to normalize image contrast. Various adaptations exist, including a version that scales according to weight norms<cit.>.







 §.§ Attention

Attention mechanisms, explored in <cit.>, enable the learning of relevance scores for inputs, mimicking cognitive attention processes. This allows certain parts of the inputs to be highlighted as highly important, while others may be ignored as irrelevant. Often, the importance of a specific input is influenced by its context; for example, the significance of a word in a text document usually relies on the words surrounding it. 

Scaled Dot-Product Multi-Head Attention <cit.>: 

The use of dot products, coupled with down-scaling, has been highly effective in calculating attention scores. 
Attention takes a query Q, a key K and a value V as inputs and outputs an attention score:
Att(Q, K, V) = softmax(QK^T/√(d_k))·V




Employing multiple, independent attention mechanisms in parallel enables the system to focus on different aspects of the input simultaneously. In the case of multi-head attention, matrices W are learned to facilitate this process. Formally:

   MultiHead(Q, K, V) = [h_0,…,h_n-1]W_0

   where  head h_i = Att (QW_i^Q, KW_i^K, VW_i^V ) 

Factorized (Self-)Attention <cit.> 
decreases both the computational and memory demands compared to the original “full” self-attention<cit.>, which allows every element to attend to every other prior input. In factorized self-attention, attention is limited to a subset of input elements. Formally, an output matrix is generated using a matrix of input embeddings  X and the connectivity pattern S={S_1,...,S_n}, where S_i is the set of indices of input vectors attended to by the ith output vector.

   FacAtt(X, S) = (A(𝐱_i,S_i))_i∈[1,n]

   a(𝐱_i, S_i) = softmax((W_q𝐱_i)K^T_S_i/√(d))·V_S_i 

   K_Si = (W_k𝐱_j)_j∈S_i abc V_S_i = (W_v𝐱_j)_j∈S_i 

For full self-attention S^F_i:={j|j≠ i}, i.e., all indexes to inputs prior to the ith input. In contrast, factorized self-attention has p separate attention heads, where the mth head is given by a subset A_i^(m)⊂ S^F_i  with S_i=A_i^(m).
For strided self-attention:
    A_i^(1)={t,t+1,...i}  for  t=max(0,i-l)

   A_i^(2)={j:(i-j)l=0}
This pattern is adequate, when structure is similar to the stride-like images. For data without a periodic structure like text, fixed attention can be preferable:

   A_i^(1)={j: ⌊j/l⌋= ⌊i/l ⌋}

    A_i^(2)={j: j l ∈{t,t+1,...l}}
 where t=l-c with hyperparameter c. For instance, with a stride of 128 and c=8, all future positions greater than 128 can attend to positions 120-128, all greater 256 to 248-256, and so on.

Multi- and Grouped Query Attention<cit.> 
Multi-query attention(MQA)<cit.> uses multiple query but only one key head and one value head, which differs from standard muli-head attention. This helps in lowering memory bandwidth for loading keys and values but it reduce quality and training stability. MQA can be added to trained models without MQA with (relatively) little compute <cit.>. Grouped query attention aims at striking a balance between a single and many heads. Per head they use multiple queries rather than just one. This allows to lower quality gaps.

A Residual Attention Network (RAN)<cit.> 
capitalizes on the concept of skip connections and is composed of two branches: a mask branch and a trunk branch. The trunk branch is responsible for processing features and can be any type of network, while the mask branch determines the weights for these features. The output of the attention module is
H_i,c(x)=(1+M_i,c(x))·F_i,c(X) 
where i is a spatial position and c is a channel. M(x) should be approximatedly 0, H(x) approximates original features F(x).

Large Kernel Attention<cit.> 
break down a (large-scale) convolution into three more manageable components: a depth-wise dilated convolution, a non-dilated depth-wise convolution, and a 1x1 convolution across channels. Subsequently, an attention map is derived from the outputs of these convolutions.


Sliding Window Attention<cit.> 
This approach focuses on enhancing the efficiency of attention mechanisms in terms of both time and memory by reducing the number of input pairs considered. Specifically, for a designated window size w, each token attends to w/2  tokens on either side.





 §.§ Graph Neural Networks 

Graph neural networks, reviwed in <cit.>, can be viewed as an extension of CNNs and transformers to accommodate graph-structured data. These networks process information represented as nodes interconnected by edges. We delve into various graph models, focusing on how to derive node embeddings that are applicable to downstream tasks.


Graph Convolutional Networks (GCN)
<cit.> use CNNs for semi-supervised learning. They approximate spectral graph convolutions using polynomials of order k, which a CNN can compute with k linear layers. This also implies that dependencies are localized, i.e., only up to nodes of distance k from a node.

Scalable Feature Learning for Networks
(Node2Vec)<cit.> is designed to learn feature vectors that effectively preserve the neighborhood characteristics of nodes within a graph. This method employs random walks to generate samples of neighborhoods, allowing it to capture and represent nodes based on their roles or the communities to which they belong.

Graph Attention Networks
<cit.> utilizes masked self-attention layers that enable nodes to adaptively focus on the features of their neighboring nodes. Specifically, node j assigns importance scores to the features of node i based on their connectivity. The use of masking ensures that only connected node pairs are considered. Unlike GCNs, this method allows for different levels of importance to be assigned to nodes within the same neighborhood. Additionally, it avoids the expensive matrix operations associated with eigendecompositions.




TuckER
<cit.> focuses on using factorization for link prediction within a knowledge graph, where knowledge is structured as (subject, relation, object) triplets. The objective is to predict the existence of a relationship between two entities. This approach models the graph as a binary tensor, with the dimensions representing subjects, relations, and objects. Tucker decompositions are then employed to break down this binary tensor into a core tensor and separate embedding matrices for subjects, relations, and objects.

Embedding by Relational Rotation 
(RotatE)<cit.> is used for predicting missing links in knowledge graphs, similar to the previously described TuckER<cit.>, but with a focus on modeling additional relational properties like composition and inversion. In this method, entities are embedded in a complex space, and relations are treated as element-wise rotations. These rotations are optimized to effectively transition from one entity to another within the complex embedding space.


Graph Transformer
<cit.>   extends the original transformer architecture to handle graph structures by implementing attention mechanisms that consider the connectivity of each node's neighborhood. This adaptation generalizes the concept of position encoding to suit graph data. The model also replaces layer normalization with batch normalization and introduces the capability to learn representations for edges, in addition to node representations.




§ DISCUSSION

Our findings suggest that despite many small and creative innovations since the original transformer architecture, there have not been any significant "breakthrough" discoveries that have led to much better leaderboard results. The improvements on models such as large language models within the last few years have been characterized by the enlargement of existing networks such as GPT, the increase of data volume (and quality), focus on computational efficiency, and a shift towards self-supervised learning. This could indicate a need for more daring approaches to research rather than incremental improvements of existing works and it raises the question: 
Are transformers all we can do?

Combining different elements or pursuing more radical ideas as outlined in this work could be one way to go beyond transformers. But advancing novel ideas such as state-space models and ideas that have existed for a while such as capsule networks, but have not lived up to their hopes, is not an easy task. We have identified a few general patterns that have been proven effective in many areas and might help in accomplishing this task: 

    
  * Multi-X refers to the strategy of deploying the same component repeatedly in parallel configurations, such as employing several residual blocks (ResNeXt) or utilizing multi-head attention mechanisms or multiple models as for mixture of experts. This concept is closely aligned with the principles of "ensemble learning."
    
  * Higher order layers involve more complex operations than the linear layers and simple ReLU functions typically used in classical CNNs and MLPs. Examples of these more advanced layers include Mish or attention layers, which facilitate deeper and more nuanced processing. 

    
  * Data controlled gating is an instance of a higher order layer. The idea of data controlled gating, e.g., depending on some (possibly transformed) input X, another input X' is deemed relevant or ignored, seems powerful. This idea can be found in attention but also in LSTM layers and GELU, where we have “self-gating”. On a larger scale also mixture of experts perform data controlled gating and, more generally, in transistors. Data controlled gating can also often be interpreted as Weighing functions, which involves the use of parameterized functions to weigh inputs. Instead of simply aggregating inputs, these functions assign weights to them, often derived from learned parameters. These functions act as "gates," allowing the flow of information only within a specific range of input parameters.
    
    
  * Moving average refers to the technique of averaging weights, as seen in methods like SGD and BYOL.
    
  * Decompose  refers to breaking down matrices into simpler components, as exemplified by methods like TuckER and large kernel attention.
    


Furthermore, the transformer seems to violate the “no free lunch theorem” by Wolpert saying that no model can be best for all (generative) tasks. While fine-tuning models on specific data can yield better (transformer) models for specific tasks, it is not clear, whether specialized architectures might substantially outperform the general transformer in certain sets of tasks, where it seems to outperform. This is analogous to activation functions, where no activation outperforms on all datasets and within all models <cit.>. 


Our survey focused on key design elements in building deep learning models. Taking a practical approach, we chose to ignore theoretical works, which should be further explored in future studies. For example, <cit.> derived a theoretical framework for dynamical systems allowing to better compare model classes such as linear attention and state space models. 


Our survey intentionally focused on more recent, yet well-established works, which could be seen as either a strength or a limitation. The selection of papers for the deep learning part was guided by a prominent platform that offers leaderboards. The rise of such platforms, which allow the upload of papers and models and provide information on citations and rankings, offers a fresh perspective compared to traditional survey methods that often choose papers more arbitrarily. Although this approach benefits readers looking for "what works well and what is very promising," it may overlook innovative ideas that need more research to fully demonstrate their potential. This could contribute to the "winner-takes-all" dynamic, reinforcing already successful ideas. Furthermore, such platforms might undergo a hype-cycle, e.g., become very popular for a while before vanishing. Nevertheless, given the vast number of papers, some form of selection is essential for conducting a comprehensive survey of deep learning.

We recognize that online platforms offering leaderboards and related features are highly valuable to the research community and should continue to be developed. However, we found that manual verification was necessary, such as double-checking relevance with Google Scholar citations and reviewing surveys and papers, to identify works and methods that were not accurately listed on the platform.





§ CONCLUSIONS

We have provided a concise yet thorough overview of the deep learning design landscape, with a focus on transformers and their potential successors. Key works from various significant areas that have emerged in recent years have been summarized. We believe that our holistic overview in a single paper can establish connections that may inspire novel ideas. Additionally, we have identified four patterns that characterize many improvements in this field. To further advance the development of artificial intelligence, it is crucial to generate fundamentally new and successful approaches, as recent improvements, though numerous and often very creative, have primarily been incremental.

















 








splncs04





















 Paper Title: Machine learning surrogates for efficient hydrologic modeling: Insights from stochastic simulations of managed aquifer recharge.


A first systematic characterization of cataclysmic variables in surveysTable <ref> is only available in electronic form at the CDS 
    A.D. Schwope1
K. Knauff1
J. Kurpas1,2
M. Salvato3
B. Stelzer4
L. Stütz1
D. Tubín-Arenas1,2

    August 5, 2024
===================================================================================================================================



§ ABSTRACT 
	Process-based hydrologic models are invaluable tools for understanding the terrestrial water cycle and addressing modern water resources problems. However, many hydrologic models are computationally expensive and, depending on the resolution and scale, simulations can take on the order of hours to days to complete. While techniques such as uncertainty quantification and optimization have become valuable tools for supporting management decisions, these analyses typically require hundreds of model simulations, which are too computationally expensive to perform with a process-based hydrologic model. To address this gap, we propose a hybrid modeling workflow in which a process-based model is used to generate an initial set of simulations and a machine learning (ML) surrogate model is then trained to perform the remaining simulations required for downstream analysis. As a case study, we apply this workflow to simulations of variably saturated groundwater flow at a prospective managed aquifer recharge (MAR) site. We compare the accuracy and computational efficiency of several ML architectures, including deep convolutional networks, recurrent neural networks, vision transformers, and networks with Fourier transforms. Our results demonstrate that ML surrogate models can achieve under 10% mean absolute percentage error and yield order-of-magnitude runtime savings over processed-based models. We also offer practical recommendations for training hydrologic surrogate models, including implementing data normalization to improve accuracy, using a normalized loss function to improve training stability and downsampling input features to decrease memory requirements.









§ INTRODUCTION

Future water security hinges on effective water resources management. Process-based hydrologic and reactive transport models, which simulate water flow and solute transport <cit.>, are frequently used to forecast future changes in the hydrologic cycle and to optimize water management strategies. When paired with uncertainty quantification and parameter estimation methods, these models can support decision-making in diverse scenarios such as flood forecasting, contaminant transport, nutrient availability and managed aquifer recharge (MAR). However, these techniques often require hundreds of model simulations in which uncertain input parameters are varied randomly from one simulation to the next. While each individual simulation is deterministic – producing one fixed output for a given set of input parameters – the ensemble of simulations is stochastic because of the random variations in model input. When performed at the scale and resolution required for locally relevant policy decisions <cit.>, these stochastic approaches result in large computational loads, even on state-of-the-art supercomputers.

To bridge this gap, recent studies have proposed a hybrid modeling workflow that capitalizes on both the high accuracy of process-based hydrologic models and the computational efficiency of modern machine learning (ML) models <cit.>. In this workflow, an initial batch of simulations is generated using a process-based hydrologic model. An ML surrogate is then trained on this initial set of simulations and used to perform the remaining model runs. For example, <cit.> evaluated the accuracy of two- and three-dimensional convolutional surrogate models for simulating overland flow in a simple tilted V catchment. This benchmark test problem <cit.> simulates 2D surface water flow through a simplified domain that consists of two inclined planes that converge in a central channel. Across simulations, the authors varied six uncertain input parameters related to rainfall and surface runoff, all of which were assumed to be spatially homogeneous across the domain. Of the six architectures tested, those that were provided with explicit temporal information during training exhibited the best performance, though accuracy decreased when the simulations in the test set contained different parameter distributions than those in the training set <cit.>. Building off of this work, <cit.> trained recurrent and highway networks <cit.> to emulate surface-subsurface flows within two watersheds spanning 6001236km^2. As compared to a process-based hydrologic model, the emulator exhibits low (<0.1) relative bias for streamflow and total groundwater storage and can produce simulations 42 times faster.

While these early studies highlight the potential for ML surrogate models to accelerate hydrologic simulations, many questions remain for practitioners interested in applying ML emulators to water resources problems. For example, how many process-based simulations are required to train a high-fidelity surrogate model? Given the high dimensionality associated with spatial uncertainty, does the minimum training set size change when varying spatially heterogeneous input parameters across stochastic simulations? How does the choice of loss function impact performance? How important is data normalization? And, given the computational expense associated with training many ML architectures, when is it more computationally efficient to perform simulations exclusively with a process-based hydrologic code instead of an ML surrogate?

In this paper, we investigate these questions and present practical considerations for using ML surrogates to perform stochastic hydrologic simulations. As a case study, we focus on 3D simulations of variably saturated groundwater flow during managed aquifer recharge (MAR), the intentional replenishment of depleted aquifers through recharge basins and injection wells. We investigate seven different surrogate model architectures and evaluate the accuracy with which each one can reproduce output from a parallel, integrated hydrologic code (ParFlow-CLM). Importantly, we explore the impacts of spatial uncertainty on surrogate performance by varying parameters both spatially across the domain and between simulations. We then evaluate the memory and runtime required for each surrogate architecture as compared to the process-based hydrologic code. We also identify a range of implementation techniques that improve model performance, including data normalization, loss normalization and downsampling of high-dimensional inputs using an autoencoder.



§ MATERIAL AND METHODS


In this section, we formalize the hybrid modeling workflow (<ref>) and introduce the case study (<ref>). Next, we describe the process-based hydrologic code (<ref>) and the surrogate model architectures (<ref>). We then describe data preprocessing (<ref>) and detail the experimental setup for training and evaluating each ML surrogate (<ref>). We conclude with a summary of experiments, enumerating the ML models trained to assess our approach (<ref>).



 §.§ Proposed hybrid workflow


The goal of this hybrid workflow is to harness the fast output rate of ML models to efficiently generate a large volume of simulations for downstream hydrologic analysis. Our task is to execute N simulations, generating N simulation outputs from N distinct sets of simulation inputs. As illustrated in Figure <ref>, we employ a traditional processed-based hydrologic model (ParFlow-CLM) to generate a training set 𝒟_train comprising simulation input–output pairs. Subsequently, an ML surrogate model is trained on these examples and used to generate simulation outputs for the remaining N - |𝒟_train| simulations, where |𝒟_train| is the number of examples in 𝒟_train. If the ML surrogate provides comparable accuracy to the process-based model, the combined group of N simulations can then be used for further analyses, such as uncertainty quantification, parameter estimation and resource optimization.





 §.§ The case study: Managed aquifer recharge site


We apply this hybrid workflow to a single case study in which hydrologic simulations are used to evaluate a recharge efficiency at a prospective managed aquifer recharge (MAR) site. MAR is a water management strategy whereby land managers deliberately flood parcels of land or inject water into wells to replenish underlying aquifers. Recent work has shown that MAR is a valuable tool for mitigating groundwater depletion in many regions of the globe <cit.>, but evaluating prospective recharge sites remains challenging as subsurface heterogeneity can significantly impact recharge rates <cit.>. Hydrologic and reactive transport simulations are effective tools for quantifying outcomes of interest at prospective MAR sites, such as changes in groundwater storage <cit.> or the fate of subsurface contaminants <cit.>. However, generating these simulations comes with a notable computational expense, as the time required for a single simulation can range from hours to days depending on model scale and resolution. Thus, MAR site selection is a logical case study that could benefit from a computationally efficient hybrid modeling workflow.

In this case study, we focus on a prospective MAR site in Tulare County, California, that measures 800m×400m. The site is underlain by a 45m thick vadose zone that consists of interbedded layers of sand, silt and clay. The unconfined aquifer extends to 150m depth, below which the Corcoran Clay, a regional aquitard, limits vertical flow <cit.>. The goal of the simulations at this site is to quantify the increase in saturated zone storage that will occur within two years of inundating the site with a 0.8m recharge event.

To evaluate recharge efficiency at this site, <cit.> performed several hundred stochastic simulations of MAR using ParFlow-CLM (<ref>). The model domain is discretized into a 120 × 80 × 25 rectilinear grid (1200m×800m×150m in the x, y and z directions, respectively), utilizing a uniform horizontal cell size of 10m and variable cell thicknesses in the z direction (1.02m at the surface and 9m at 90m depth). Specified head boundaries corresponding to the depth to the water table (45m) are applied on the sides of the domain. A no-flow boundary is implemented on the bottom of the domain and variable forcing is applied to the top of the domain based on the modeling stage, as explained at the end of this section.

The domain is parameterized through a stochastic geophysical-geostatistical workflow <cit.>. Because subsurface heterogeneity can exert a dominant control on recharge efficiency, we parameterize the domain using a subsurface electrical resistivity model derived from a towed transient electromagnetic (tTEM) survey of the site. The tTEM system is a hydrogeophysical tool that can map meter-scale variations in electrical resistivity, a property directly related to sediment lithology. We first generate 600 gridded realizations of the tTEM-derived resistivity model using sequential Gaussian simulation, a stochastic technique for populating a rectilinear grid with a random variable. We then compare vertical profiles of electrical resistivity to collocated profiles of sediment type collected through cone penetration tests. Using the bootstrapping procedure of <cit.>, we transform each resistivity realization to a realization of coarse fraction, a metric that describes the fraction of coarse-grained (sand and gravel) sediment within each grid cell. The remainder of sediment within the cell is assumed to be fine-grained (silt and clay). For example, a cell with a coarse fraction of 0.5 is 50% coarse-grained material (either 100% sand, 100% gravel, or some combination of the two) and 50% fine-grained material (some combination of silt and clay). To assign hydrogeologic parameters (hydraulic conductivity, porosity, etc.) to each coarse fraction realization, we assume that only two types of sediment exist within a single realization — coarse-grained and fine-grained — and that each cell is a mix of the two types. We then sample from broad distributions to assign parameter values to each end member and use a weighted averaging procedure to calculate values for cells that are a mix of the two end members. The input parameter distributions are derived from a mix of literature and site-specific data. For more information about these parameter values, we refer the reader to the more complete description provided by <cit.>.

Each MAR simulation consists of three consecutive stages. In Stage 1, a constant recharge forcing equal to the long-term natural recharge rate is applied to the top of the domain for 160,000 days (∼440 years), which is enough time for each simulation to reach steady state. In Stage 2, we apply one year of meteorological and irrigation forcing using hourly measurements from the southern Central Valley. These two spin-up stages initialize water content in the vadose zone and establish realistic soil moisture profiles prior to implementing recharge. Finally, Stage 3 simulates recharge by inundating the orchard with 0.8m of water in a single winter season (applied in either 1, 4, 8 or 16 individual recharge events), alternating between periods of flooding and no flooding. In between inundation events, we apply meteorologic and irrigation forcing to the top of the domain.

Once each simulation has completed all three stages, we quantify the increase in saturated zone storage induced by MAR. Each simulation produces spatiotemporally variable output (transient pressure fields, represented as a time series of 120 × 80 × 25 rectilinear grids) from a combination of temporally static, spatially variable input (e.g., 3D fields of porosity or saturated hydraulic conductivity, represented as 120× 80 × 25 rectilinear grids) and temporally variable, spatially homogeneous input (e.g., ambient air temperature). The number of output time steps varies depending on the modeling stage, with a finer temporal resolution in later modeling stages. We output pressure fields every 1000 days in Stage 1 (160 output time steps), every 2 days in Stage 2 (183 output time steps) and every 2 days in Stage 3 (366 output time steps). Note that the number of time step outputs here simply reflects the frequency at which the model outputs 3D pressure fields and not the temporal resolution of the underlying solver. From these 3D transient pressure fields, we calculate other system states (e.g., water content in each grid cell) and quantify the change in saturated zone storage.




 §.§ The high-fidelity process-based model: ParFlow-CLM


In the first phase of our proposed framework (<ref>), a process-based hydrologic model is used to generate an initial set of simulations, which are then used to train an ML surrogate model. In this case study, we perform these process-based simulations using ParFlow-CLM. ParFlow-CLM couples the integrated hydrologic code ParFlow with the Common Land Model (CLM) to solve partial differential equations that describe water and energy fluxes over and beneath the land surface. ParFlow solves the Richards' equation for variably saturated subsurface flow using a Newton-Krylov approach and the shallow-wave equations for overland flow using a kinematic wave approximation. CLM, meanwhile, uses a mass transfer approach to compute water and energy fluxes at the land surface <cit.>, including irrigation, evapotranspiration and precipitation. To accurately simulate the impacts of diurnal temperature and radiation fluctuations on water fluxes across the land surface, we run ParFlow-CLM using an hourly time step in Stages 2 and 3. For a detailed description of the governing equations, we refer readers to previous work describing ParFlow-CLM <cit.>.





 §.§ The machine learning models


In the second phase of this hybrid framework (<ref>), we train an ML surrogate on the initial set of simulations generated by the process-based model. In total, we evaluate the performance of seven different machine learning surrogate model architectures. The architectures include recurrent and highway networks developed for spatiotemporal forecasting <cit.>, vision transformers <cit.>, convolutional nerual networks and U-shaped networks paired with Fourier transforms <cit.>. We selected these architectures because some of them have been successfully applied in previous hydrologic surrogate modeling studies <cit.>, while others are some of the most common architectures used for spatiotemporal forecasting in related fields like computer vision.

The ML surrogate architectures can be categorized into three types based on their method of prediction (Table <ref>, Figure <ref>). First, the Non-sequential models emulate the entire simulation in a single forward pass (i.e., the emulator produces 3D pressure fields for all output time steps in one run of the model). By contrast, the One-step models emulate one output time step (producing a single 3D pressure field) per forward pass. These models then use the output from the previous time step to perform another forward pass, repeating until all 3D pressure fields have been generated. Finally, the Recurrent models are similar to the one-step models in that they emulates one output time step per forward pass. However, they use memory cells to retain hidden states between forward passes, so that information can be stored between output time steps. As discussed above, the output frequency varies between model stages (see <ref>), which results in a different shape input tensor for each model stage. Because most ML architectures require a consistent shape input tensor across all training examples, we train a separate surrogate for each model stage. The ML models are summarized below, though complete details are provided in the Supplementary material (Section <ref>).






  
Non-sequential models
We experiment with three non-sequential models. Since non-sequential models produce 3D pressure fields for all output time steps in a single forward past, they are four-dimensional; i.e., they receive four-dimensional inputs — x, y, z and time — and generate four-dimensional outputs (3D transient pressure fields). The non-sequential models are:

    
  * CNN4d. The CNN4d is a three-layer convolutional neural network that consists of four-dimensional convolutional layers, each followed by a four-dimensional batch normalization layer <cit.> to allow for fast and stable training, a leaky ReLU activation <cit.> to introduce nonlinearities, and a dropout layer <cit.> to prevent overfitting.
    

    
  * ViT4d. The ViT4d is a vision transformer <cit.> that uses a four-dimensional convolutional layer to transforms the input tensor into a sequence of four-dimensional patches. This sequence of patches is then fed into a self-attention module <cit.> that calculates the importance of each patch relative to every other patch, allowing the model to learn both spatial and temporal dependencies regardless of their distance from each other.
    
    
  * U-FNO4d. The U-FNO4d is based on the three-dimensional U-FNO implementation originally proposed by <cit.>, which consists of a U-shaped network paired with Fourier neural operators <cit.>. While a full description of the U-FNO architecture is beyond the scope of this paper, we provide a brief description below. The U-FNO architecture (Figure <ref>) consists of three Fourier layers followed by three U-Fourier layers, bookended by linear projection layers. Each Fourier layer contains two information pathways: one through a Fourier and inverse Fourier transform separated by a linear transformation and another through a simple linear layer. These two information pathways are merged with a residual connection. The U-Fourier layer is essentially identical to the Fourier layer except there is a third information pathway: a U-Net consisting of a sequence of five convolutional layers followed by three transposed convolutional layers. Again, the information pathways are merged with a residual connection. We expand the original, three-dimensional U-FNO architecture to four dimensions by replacing three-dimensional convolutional layers in the U-Fourier layers with four-dimensional convolutional layers and by adding a fourth dimension with an identical maximum number of nodes to the Fourier and inverse Fourier transforms.




  
One-step models
One-step models, unlike non-sequential models, receive three-dimensional inputs (x, y and z) and predict one time step at a time. The output of the previous time step is reused as input for the next time step and this rolling forward process continues until the final time step. Note that these models receive no explicit temporal information during training, such as day of year or time step size. These one-step models include:


    
  4. CNN3d. The CNN3d model, like the CNN4d model, acts as our baseline one-step model.
    Constructed almost identically to the CNN4d model, the CNN3d model mainly differs in that it employs three-dimensional convolutions instead of four-dimensional ones.
    
    
  5. ViT3d. The ViT3d model mirrors the ViT4d model but employs a three-dimensional convolutional layer to generate three-dimensional patches.
    
  6. U-FNO3d. Our U-FNO3d model replicates the U-FNO architecture presented by <cit.> and is the same as the U-FNO4d described above, except that it uses three-dimensional convolutional layers instead of four-dimensional layers.




  
Recurrent models
Like one-step models, one forward pass of a recurrent model produces three-dimensional output for a single time step. The key distinction is that the recurrent model retains hidden states between forward passes. We experiment with one such model:

    
  7. PredRNN++. This recurrent model is an improved predictive recurrent neural network (PredRNN++), which was originally developed by <cit.>. The PredRNN++ architecture (Figure <ref>) pairs causal long short-term memory networks (causal LSTMs) with a gradient highway units (GHUs). Causal LSTMs contain a spatiotemporal memory mechanism designed to improve short-term spatial correlations, while GHUs connect future outputs with distant inputs for improved long-term modeling capabilities. Each causal LSTM contains distinct cells designed to store spatial information (ℳ) and temporal information (𝒞) from one time step to the next, as well as hidden states (ℋ), which are used to make predictions (Figure <ref>). Within the PredRNN++, which contains multiple causal LSTMs layers, each layer has its own unique temporal memory (𝒞) and hidden states (ℋ). The spatial memory (ℳ) is shared across all layers passed sequentially between them within a single time step. To allow the model to learn long-term dependencies, the GHU layer has its own memory cell (𝒵), which bypasses the deeper causal LSTM layers and provides an alternate route for information to flow through the network. Given that the original PredRNN++ architecture was designed to handle two-dimensional inputs, we modify the implementation to handle three-dimensional inputs by replacing all two-dimensional convolutional layers with three-dimensional convolutional layers. Detailed descriptions of causal LSTMs and gradient highway unit  can be found elsewhere <cit.>.




 §.§ Data preprocessing


To train each ML surrogate model, we designate the inputs of the ParFlow-CLM simulations (e.g., 3D porosity fields, transient meteorological forcing, etc.) as our training features and the outputs of the ParFlow-CLM simulations (transient, 3D pressure fields) as our training targets. However, prior to training, we first perform several steps of preprocessing. In particular, we perform feature selection to curate a list of salient input features, we broadcast input features across the x, y, z and time dimensions to standardize input and output formats, and we downsample certain inputs to reduce memory requirements for training.

margin=2cm

1.1

    Summary of input and target fields. 


    
                        Varies between     Spatially     Temporally     3cAppears in Stage

        9-11
           Field name    Units     μ     σ     simulations     variable     variable     1     2     3 

        lightgray
           10lInputs

        1.     Horizontal Hydraulic Conductivity     m/hr     0.0936     0.270
                                     

        2.     Vertical Hydraulic Conductivity     m/hr     0.0151    0.0465
                                     

        3.     Porosity     1     0.338     0.0585
                                     

        4.     Residual Saturation     1     3.417     1.15
                                     

        5.     van Genuchten α     1/m     3.41     1.15
                                     

        6.     van Genuchten n     1     0.365     0.201
                                     

        7.     Incoming Short-Wave Radiation     W/m^2     202.     86.5
                                       

        8.     Incoming Long-Wave Radiation     W/m^2     323.     22.3
                                       

        9.     Precipitation Rate     mm/s     2.85e-05     1.74e-05
                                       

        10.     Air Temperature     K     290.     6.14
                                       

        11.     West-to-East Wind Speed     m/s     0.525     0.844
                                       

        12.     South-to-North Wind Speed     m/s     -0.371     0.902
                                       

        13.     Barometric Pressure     pa     100,000     246.
                                       

        14.     Specific Humidity     kg/kg     6.40e-03     5.00e-04
                                       

        15.     Cell Volume     m^3     600.     1130
                                      

        16.     Initial Pressure Head Field1    m     6.40     18.4
                /                     

        17.     Inundation Forcing     m/s     0.232     0.294
                                      

        lightgray
           10lTargets

        1.     Pressure Head     m     6.20     21.2
                                    

        

    
   
   
   
  1 At the start of model Stage 1 (model spin up to steady state), pressure across the domain is in hydrostatic equilibrium with the water table at 45 m depth. For Stages 2 and 3, the initial pressure head field is initialized as the final pressure field output from the previous stage.
   
   








  §.§.§ Feature selection


We select 17 input fields as the input features and one output field as our supervised learning target (Table <ref>). These input fields include hydrogeologic parameters (fields 1-6 in Table <ref>), meteorologic forcing (fields 7-14), information about the rectilinear model grid (field 15), initial conditions (field 16) and the recharge forcing applied at the surface (field 17).

The process-based hydrologic model receives additional input information not included in Table <ref>, such as cell dimensions and time step sizes. However, we omit many of these fields from our input features because they are temporally invariant and constant across all training examples. Thus, each surrogate architecture likely learns this information implicitly during training. Some fields, such as cell volume and the initial pressure head for Stage 1, are constant across all simulations, but are included nonetheless to support future studies seeking to modify these parameters.



  §.§.§ Broadcasting


Broadcasting streamlines manipulation of inputs and targets through the use of large, combined tensors. The input features (Table <ref>) include fields that remain constant in time, fields that constant in space and fields that are constant in both space and time. Broadcasting simply repeats the values of each field across dimensions where that field remains constant, which ensures that the tensor for each input feature is four-dimensional (x, y, z and time). For example, we broadcast air temperature at each time step — which does not vary spatially across the domain — along the x, y and z dimensions. Similarly, given that saturated hydraulic conductivity does not change from one time step to the next, we repeat the 3D hydraulic conductivity field from the first time step across all time steps. This approach standardizes the dimensions of all input fields and allows them to be combined into one large tensor for input into each architecture.



  §.§.§ Downsampling


While broadcasting standardizes input dimensions, it increases the total size of the input features and, by extension, the memory allocated to each training example. The memory required to train an ML model is an important component of computational efficiency; if the input and output for a single training example exceeds the memory capacity of a single graphics processing unit (GPU), then the model will have to be trained across multiple GPUs, which will increase the total number of processor-hours used during training. Modern hydrologic model domains can include anywhere from 10^5 to >10^8 total grid cells <cit.>. At this scale, after broadcasting each input feature across the x, y, z and time dimensions and adding in the size of the training targets (i.e., the transient 3D pressure fields from ParFlow-CLM), the memory allocated to a single training example can quickly exceed 100 GB, which is larger than the memory capacity of most modern GPUs. To address this challenge, we limit the size of a single training example through temporal and spatial downsampling.

For model stages 1 and 2, we apply temporal downsampling. These model stages are only used to initialize water content throughout the vadose zone. Because intermediate pressure fields are not used to calculate recharge efficiency, decreasing the temporal resolution does not limit downstream analyses. For each stage, we sample 16 output time steps from the spin up period using a geometric sampling scheme:


    {⌊∑_k=1^t r^k-1⌋ : ∑_k=1^16 r^k-1 = 160 }_t=1^16.


The length of 16 is chosen so that a single training example fits on our training hardware, while the geometric sampling scheme prioritizes time step outputs toward the beginning of the model stage, when changes in pressure head are most pronounced. Though the final steady-state pressure field at the end of each stage is the only output used in subsequent modeling stages, we calculate loss across multiple time steps from this stage to ensure that the model simulates physically meaningful processes (i.e., gradual changes in pressure over time). This also forces the model to output intermediate pressure fields; visually examining these intermediate outputs also allows the user to more easily diagnose factors contributing to poor model performance. While this geometric sampling scheme results in a variable time step size within an individual simulation, the output frequency is consistent across all training examples. Thus, the data-driven ML models should implicitly learn these temporal patterns during training.


To accurately quantify recharge efficiency, the output from the Stage 3 surrogate models must be temporally high-resolution. Thus, instead of applying temporal downsampling, we apply spatial downsampling to the Stage 3 input features and training targets. For the input features, we reduce dimensionality using an average pooling layer with a 20× 20× 1 window and a stride of 4× 4× 1. This layer simply calculates the mean of each input feature over the 20× 20× 1 window, reducing the size of the input tensor in the x and y directions. The z dimension remains unchanged, however, in order to preserve vertical resolution given that vertical flow is important for this case study.

To downsample the Stage 3 training targets, we use an autoencoder. The autoencoder is a separate neural network that is trained to reproduce the spatially variable pressure field from a single time step, but it is forced to store this information in an intermediate, lower-dimensional space (Figure <ref>). Each surrogate model is then trained on the lower-dimensional representation of the 3D pressure fields, known as the encoded targets (Figure <ref>). We train a single autoencoder that is used for all surrogate models. To calculate surrogate performance on the test set, we then decode the surrogate model's predictions using the autoencoder and calculate error metrics relative to the high-resolution pressure fields output by ParFlow-CLM. This workflow allows us to train our surrogate model in the reduced dimensional space, minimizing memory requirements, while retaining the high spatial resolution of our predictions for use in downstream tasks, such as calculating recharge efficiency. For a detailed overview of the autoencoder architecture, please refer to Section <ref> in the Supplementary material.





 §.§ Model training and evaluation


In this section, we discuss the training setup used for all surrogate models, the hardware used to calculate computational efficiency and the evaluation criteria used to quantify model performance.



  §.§.§ Training configuration
  Each model undergoes training for 150 epochs with a learning rate of 0.0001, using Adam optimization <cit.>. Due to GPU memory constraints, we restrict the batch size to 1. We employ a normalized L2 loss function:


    L(y_i, h(x_i))= ||w · (y_i - h(x_i))||_2/||y_i||_2


where y_i is the ground-truth output for the ith training example, x_i is the input for the ith training example, h(x_i) is the predicted output, w defines a depth-wise weighting scheme and  ||·||_2 is the L2 norm. We choose this normalized loss function because it is particularly effective when there is large variance in the data and because previous hydrologic surrogate modeling work suggests that it provides faster convergence than the L1 loss <cit.>. Given that managed aquifer recharge efficiency is largely governed by unsaturated flow processes <cit.>, accurately simulating pressure changes within the vadose zone is more important than simulating pressure changes deep within the saturated zone. Thus, we define the weights w such that they more severely penalize pressure errors closer to the surface than at depth (Figure <ref>).

To investigate the impact of our choice of loss function on model performance, we train additional CNN3d surrogates on Stage 1 simulations using the mean square error loss (i.e., the square of the numerator of Equation <ref>). Due to computational constraints, we could not train duplicates of each surrogate architecture using the mean square error loss, so we only perform this comparison with the CNN3d because it has the fastest training time.

We perform a random 80%-10%-10% (217-28-28) training–validation–test split and all performance metrics are reported based on the test set unless otherwise noted. Features and labels are normalized using their respective minimum and maximum values from the training set. We normalize all splits by the training set's minimums and maximums to prevent bias in validation and test results introduced by distribution information from one set to another. To assess the effects of data normalization on ML model accuracy, we train two additional Stage 1 CNN3d models: one using Z-score normalization and one without any data normalization. As discussed above, we only perform these tests using the CNN3d because of its short training time.



  §.§.§ Hardware environment
  We train all ML models on a single NVIDIA A100 Tensor Core graphics processing unit (GPU) with 80 GB memory. All ParFlow-CLM simulations were performed on AMD Epyc 7543 central processing units (CPUs). Depending on the model stage, ParFlow-CLM simulations were run either on 4 MPI ranks with 1 CPU core per rank (Stage 1 and Stage 3) or 6 MPI ranks with 1 CPU core per rank (Stage 2).



  §.§.§ Model accuracy evaluation
 
We evaluate each surrogate model using the mean absolute percentage error (MAPE) between the predicted pressure head time-series and the ground-truth (ParFlow-CLM) pressure head time-series in a cell-by-cell manner. Because we train a different surrogate for each modeling stage, we report different MAPE values for each stage. As with the loss function, we weight MAPE by depth to ensure that each model's performance accurately reflects its ability to simulate vadose zone flow (Figure <ref>). We also evaluate model performance when each surrogate is run sequentially across all three stages, such that the final pressure head output by the surrogate from Stage 1 is the initial pressure head (field 16 in Table <ref>) used for the Stage 2 surrogate model and the final pressure head output of Stage 2 is the initial pressure head for Stage 3. We call this technique end-to-end (E2E) evaluation. E2E provides a more realistic evaluation of surrogate performance in a real-world application.

Training an accurate ML surrogate requires a sufficiently large training set, but at a certain point, the cost of increasing the training set size may outweigh any benefit in improved prediction performance. To evaluate this relationship and estimate the minimum number of process-based simulations required to train an accurate surrogate, we train multiple versions of each Stage 1 surrogate model while using smaller and smaller training sets. In addition to the baseline training set size of 245 examples used for training (with a 217-28 training-validation split), we use training set sizes of 196 (174-22 split), 147 (130-17), 98 (87-11), 49 (43-6), 25 (22-3), 13 (11-2), 8 (7-1), 5 (4-1), and 3 (2-1) training examples while maintaining the test set constant at 28 examples. Because ML models typically exhibit a power-law relationship between model error and training set size <cit.>, we then fit a power function to these data (y=am^b, where m is the number of training examples, y is the test set MAPE and a and b are regression coefficients). Lastly, we differentiate am^b with respect to m, which yields abm^b-1. This expression represents the instantaneous change in MAPE given an increase in training set size, which we use to compare each model's sensitivity to training set size.



  §.§.§ Model runtime evaluation

In addition to evaluating ML model accuracy, we quantify the total compute time needed to generate simulations with this hybrid modeling workflow. Since model runtime is largely dependent on the hardware on which it is executed, we strictly adhere to the hardware environment described in <ref> when evaluating model runtime. To simplify analysis and minimize additional training, we only consider models trained on Stage 1 simulations in this section.

First, we benchmark runtime by defining the time needed to generate N Stage 1 simulations using the processed-based hydrologic model, ParFlow-CLM, as

    T_PF = N t_PF,

where t_PF denotes the CPU-hours taken by ParFlow-CLM to generate a single Stage 1 simulation (i.e., the wall clock time multiplied by the number of CPU cores used for the parallel simulation). Though runtime varies across stochastic ParFlow-CLM simulations (1.07–19.37 CPU-hours for Stage 1 simulations), we calculate t_PF here as the mean runtime across all Stage 1 simulations (4.17 CPU-hours).

We also establish the number of GPU-hours required to output N Stage 1 simulations using an ML model as

    T_ML = T_train + (N - 𝒟_train) t_ML,

where T_train denotes the training time needed to reach the minimum MAPE on the validation set, t_ML denotes the time taken by the trained ML model to generate one Stage 1 simulation, and 𝒟_train represents the minimum training set size necessary for the ML model to achieve an MAPE of less than 10% — the threshold for “highly accurate forecasting” <cit.>. Due to differences in each surrogate model architecture, the minimum training set size (𝒟_train) differs for each model. To find this value, we train multiple versions of each surrogate model while varying training set sizes using a binary search approach. The test set (28 simulations) is held constant during this procedure and is used to evaluate models, ensuring they fall below the 10% MAPE threshold. We then calculate 𝒟_train as the smallest training set that still allows for a test set MAPE below 10%. Note that 𝒟_train includes both training and validation examples with an 80%-10% train–validation split, so each model trains on 8/9·𝒟_train examples and validates using 1/9·𝒟_train examples.

Given that the training set is generated by a process-based model, the total compute time used to generate N simulations in this hybrid workflow includes both Equation <ref> and the time to generate the training set:

    T_training set = 𝒟_train· t_PF.


Equations <ref>, <ref>, and <ref> have different units (CPU-, GPU-, and CPU-hours, respectively) because the ML model and the process-based hydrologic model run on different types of processors. Depending on the computing resources available to an individual modeler, these units might not be equivalent. Nonetheless, it is  valuable to compare these compute times in order to evaluate the efficiency of a given surrogate model against a traditional process-based model. By assuming that 1 GPU-hour = 1 CPU-hour, we quantify the compute time saved using this hybrid modeling workflow as

    T_saved   = N t_PF - (T_training set + T_train + (N - |𝒟_train|) t_ML)
       = (t_PF - t_ML)_slope N - (|𝒟_train| (t_PF - t_ML) + T_train)_intercept,


where T_saved is in generic processor-hours. The slope term in Equation <ref> indicates that the time saved by employing an ML surrogate model becomes more pronounced when t_ML is significantly lower than t_PF. Furthermore, the intercept term indicates that time savings begin sooner, i.e., at smaller values of N, if the required training set is small and the training time is low, as expected.



 §.§ Summary of experiments

In summary, we trained six different groups of surrogate models, each for a different numerical experiment:

    
  * We train all seven surrogate architectures — the CNN4d, ViT4d, U-FNO4d, CNN3d, ViT3d, U-FNO3d, and PredRNN++ (<ref>) — on Stage 1, 2, and 3 ParFlow-CLM simulations, amounting to 21 trained models. We also train an autoencoder for all Stage 3 models. This totals 22 trained models.
    
  * We train two additional Stage 1 CNN3d models using alternative data normalization techniques (<ref>). One of the additional models does not use any data normalization, and the other uses Z-score normalization. The Stage 1 CNN3d model is chosen over other stages and models for its efficiency in training and evaluation.
    
  * We train six additional Stage 1 CNN3d models using different combinations of loss function and data normalization (<ref>). Three CNN3d models use the normalized L2 loss with min-max, Z-score, and no data normalization, and three CNN3d models use the mean squared error loss with min-max, Z-score and no data normalization.
    
  * To evaluate the effect of training set size on model accuracy, we train all seven ML models with nine Stage 1 training sets of decreasing sizes, as described in <ref>. This experiment includes 63 trained models in total.
    
  * To evaluate the ML models' runtime efficiency, we re-train all seven ML models with variable Stage 1 training set sizes, using a binary search technique to find the minimum training set size that achieves <10% test MAPE. The number of models trained at this stage varies according to the efficiency of the search algorithm and the test MAPE results from previous experiments but ranged from 2 to 7 across the seven architectures. We then compare the runtimes of the seven models trained on their respective minimum Stage 1 training sets with the procedure described in <ref>.





§ RESULTS AND DISCUSSION


Balancing speed and accuracy, the hybrid modeling framework achieves runtime benefits with minimal accuracy trade-offs. In <ref>, we first present the predictive accuracy of each ML model when trained on the full training set before examining the impacts of training set size and normalization on model accuracy and training stability. In <ref>, we compare the computational efficiency of each ML surrogate with a baseline approach that relies solely on the process-based hydrologic model. Lastly, we discuss the importance and challenges of model interpretability (<ref>) and potential limitations of this framework (<ref>).



 §.§ Model accuracy




Each ML surrogate model produces pressure fields that are visually similar to those produced by the process-based hydrologic model. For example, a comparison of ParFlow and the PredRNN++ shows that both models produce similarly-shaped heterogeneous wetting fronts during Stage 1 (Figure <ref>), with monotonically increasing pressure values below the front. Plotting the cell-by-cell absolute error between the two models reveals that differences are greatest near the wetting front (Figure <ref>c). However, absolute pressure differences are generally less than a few meters, which is of a reasonable range given the sharp hydraulic gradients that arise near infiltration wetting fronts.

All seven ML models achieve <10% MAPE on the test set (Figure <ref> and Table <ref>), which is the threshold for “highly accurate forecasting” <cit.>. The PredRNN++ displays the lowest error for Stage 1 and end-to-end (E2E) predictions, while the CNN3d and ViT3d perform the best on Stage 2 and Stage 3, respectively. Even though the non-sequential models have more parameters than their one-step counterparts (Table <ref>), they do not exhibit an appreciable increase in predictive performance. The lone recurrent model (the PredRNN++), however, outperforms the others and is better able to learn the temporal dependencies that develop as each hydrologic simulation progresses through time. The large memory cells retained between time steps likely provide ample degrees of freedom for the model to store spatially heterogeneous information over time.



Overall, prediction error is the highest for Stage 1, with a mean MAPE of 5.6% across all ML models, as opposed to 0.13% and 0.42% for Stages 2 and 3, respectively. Previous modeling work at this site has shown that managed aquifer recharge efficiency is most strongly controlled by vadose zone water content prior to inundation <cit.>. The low model error for Stages 2 and 3 makes sense, then, given that Stage 2 and 3 are provided the ground-truth pressure field output by ParFlow-CLM at the end of Stage 1 and 2, respectively, as initial pressure head fields during training.

Alternatively, the higher Stage 1 MAPE could result from its large time scale (<ref> and <ref>). While Stage 2 and 3 span 1 and 2 years, respectively, Stage 1 spans over 400 years, with more dramatic changes in pressure between outputs. Pressure head changes across large time scales may be more difficult to learn than smaller, more incremental changes, potentially contributing to the poorer performance of Stage 1 models.

End-to-end (E2E) predictions, on the other hand, do not receive ground-truth pressure fields between stages; the predicted pressure at the end of one stage is used as input for the ML surrogate in the next stage. Overall, the MAPE for E2E predictions is typically lower than that of Stage 1 and similar in magnitude to that of Stage 3 (Figure <ref>). The similarity between Stage 3 and E2E error likely arises because E2E error is averaged equally over the entire time series and Stage 3 has the most time steps. However, not all models follow this trend; the CNN3d and CNN4d both exhibit E2E MAPE higher than 1%, with a 3.1× and 6.6 × respective increases compared to their Stage 3 MAPEs. Plotting the E2E MAPE per time step in Figure <ref> reveals that the CNN3d and CNN4d exhibit the highest MAPE per time step among other models in Stage 3. This heightened difference between Stage 3 MAPE and Stage 3 MAPE in E2E evaluation could indicate that the CNN3d and CNN4d both exhibit a strong dependence on the initial pressure head at the start of a model stage. The curves for all other models follow the same general shape, suggesting that these models may have learned similar patterns in the Stage 3 data that have resulted in similar accuracy.

Collectively, the results in this section suggest that ML surrogate models can simulate variably saturated groundwater flow with a similar level of accuracy as process-based hydrologic models, supporting the feasibility of applying a hybrid modeling framework to this case study.





  §.§.§ Impact of training set size on accuracy


While the PredRNN++ exhibits the lowest E2E error, it also demonstrates the second lowest sensitivity to training set size. Figure <ref> reveals that the ML models, in general, exhibit a power-law relationship between training set size and MAPE, which is typical for most deep learning architectures <cit.>. This power-law relationship demonstrates diminishing returns when increasing training set size beyond a certain point. To explore these diminishing returns more closely, we plot the derivative of the best-fit curve for this power-law relationship (dMAPE/dm(m)) in Figure <ref>. The dMAPE/dm(m) values for the PredRNN++ remain consistently close to 0 and the model is able to achieve ∼5% MAPE with as few as 25 training examples (Figure <ref>). In fact, almost all models exhibit nearly consistent MAPE values after 50–100 training examples. While the models presented in Figure <ref> were trained on 245 examples (with a 217-28 training-validation split), these results suggests that future practitioners could achieve similar performance by training on fewer examples. However, the relationship between training set size and MAPE is highly dependant on the specific problem of interest, so future practitioners should use caution when extending these results to other sites or other scenarios.





  §.§.§ Impact of normalization on accuracy


Both the min-max data normalization and the normalized loss function (<ref>) improve model performance, resulting in lower MAPE and a more stable training process. When trained on a Stage 1 dataset normalized with a min-max procedure, the CNN3d achieves 4.8% MAPE, which is an order of magnitude lower than the CNN3d trained without any normalization (48.9% MAPE) and three times lower than the model trained with a Z-score normalized dataset (15.3% MAPE; Figure <ref>).  We attribute this improved performance to the variability in the input dataset (Table <ref>), which contains several features that span multiple orders of magnitude. In such cases, normalization improves accuracy by constraining features to a consistent range. In addition, some of the input features are not necessarily normally distributed. Because min-max normalization preserves distribution information — as opposed to Z-score normalization, which standardizes each input feature to have a mean of 0 and a standard deviation of 1 — it provides this distribution information to each ML model, which may improve model accuracy.

While normalizing input data improves model accuracy, loss normalization increases training stability. Comparing MAPE on the validation set across all training epochs, we see that using a normalized loss generally reduces the number of sharp spikes in MAPE, indicating that a normalized loss allows for more stable descent and reliable training convergence. One reason loss normalization may have such a large impact in this case is our use of small batch sizes. To limit memory requirements, we train each ML model using a batch size of 1. However, at small batch sizes, the gradient is calculated over fewer examples (or in this case, a single example), which leads to noisy loss curves and increases the likelihood that gradient descent can get stuck in a local minimum. Loss normalization helps offset these drawbacks by ensuring that the gradients are of consistent magnitude across the training set. This effect is most noticeable in models that use either no data normalization or Z-score data normalization, suggesting that loss normalization most benefits models with inadequate data normalization.

Due to computational constraints, we only perform these analyses with the CNN3d. However, we expect that results would be similar or even more pronounced for other architectures; the CNN3d has built-in batch normalization layers that increase training stability, while most other architectures do not. Overall, these results underscore the importance of both data normalization and loss normalization in enhancing model performance and training stability for hydrologic surrogate models.






 §.§ Computational efficiency


While each ML surrogate model achieves <10% MAPE, the models vary widely in the amount of time and memory they require to produce simulations. One attribute they share, however, is that all ML surrogate models offer significant runtime advantages over the process-based hydrologic model (Figure <ref>). Figure <ref> reveals positive slopes across all lines. Recall that the slope for these plots are calculated as the difference in time taken by ParFlow-CLM and the ML model in generating a single Stage 1 simulation (Equation <ref>). All lines appear to exhibit identical slopes because each ML surrogate outputs simulations orders of magnitude faster than the process-based hydrologic model (t_PF = 15,017 CPU-seconds/simulation), so the difference in t_PF - t_ML across the ML models is miniscule. Nonetheless, positive slopes for all lines signify that all seven ML models produce simulations at a faster rate than ParFlow-CLM. The ViT4d technically exhibits the steepest slope, producing simulations the fastest (t_ML =0.115 GPU-seconds/simulation), but, due to the large training set size required for the ViT4d, time savings only appear for the ViT4d after 246 simulations. Closer examination of the x-intercepts reveals that 6 out of 7 of the ML models achieve a runtime advantage over ParFlow-CLM when the desired simulation output count is as low as 17. We provide a detailed overview of all time measurements in Table <ref>, including distinct values of 𝒟_train (the minimum size training set required to achieve <10% MAPE, which strongly impacts the computational efficiency of each model.



When accounting for both training time and forward simulation time, the U-FNO4d is the most efficient ML model. To perform 500 simulations, the U-FNO4d requires a total of 13.1 processor-hours (the sum of Equations <ref> and <ref>), compared to the second-most efficient ML model, the PredRNN++, which requires 13.3 processor-hours. These computational times are orders of magnitude smaller than the time it would take to generate 500 simulations exclusively with ParFlow-CLM (2090 processor-hours). These findings, expressed in Figure <ref>, affirm the tangible benefits of this hybrid framework, which outpaces a naive approach of solely relying on ParFlow-CLM. We believe this computational speedup arises from two factors. First, ParFlow-CLM uses an iterative Newton-Krylov approach to solve the coupled system of nonlinear partial differential equations at each time step. This iterative scheme requires many Jacobian evaluations and floating point operations per time step, while the ML surrogates only require forward passes, consisting of fixed calculations determined by the model architecture and model weights. Second, the ML surrogate models can take arbitrarily large time steps without compromising convergence due to their data-driven nature, while ParFlow-CLM must use an hourly time step to accurately represent meteorology-driven water fluxes across the land surface. Other process-based models are subject to similar constraints — such as the Courant–Friedrichs–Lewy condition — which limit their computational efficiency relative to ML surrogates. Since ML surrogates can learn temporal patterns in training data, they are unbound by time step size and can thereby complete simulations at faster rates.



In addition to runtime, the memory required to train an ML model is an important component of computational efficiency. While the memory cells in the PredRNN++ allow the model to retain information from one time step to the next, these large tensors require significant memory allocation (Table <ref>). Using a batch size of 1, the PredRNN++ requires 15.1× more GPU memory to train (31.7 GB) than the ViT3d (2.1 GB), the least memory-intensive architecture. At this allocation size, the PredRNN++ may be too memory-intensive to train on some modern GPUs. Similarly, all three non-sequential models (CNN4d, ViT4d and U-FNO4d) require more memory than their one-step counterparts (CNN3d, ViT3d and U-FNO3d), though this increase in memory allocation does not necessarily translate to an increase in accuracy (Figure <ref>). Ultimately, even though some architectures, such as the PredRNN++, can achieve low MAPE with only a few training examples, they may not be flexible enough to adapt to all compute environments due to their high memory requirements.



 §.§ Model interpretability


Another crucial factor to consider when selecting an ML architecture within a hybrid modeling framework is model interpretability. Error recognition is more straightforward with an interpretable surrogate model and can enhance trust among practitioners. The PredRNN++ architecture, for example, contains 𝒞 and ℳ memory cells (Figure <ref>) that are intended to serve distinct roles, with the former facilitating the transfer of temporal information and the latter facilitating the transfer of spatial information <cit.>. However, in practice, deciphering the values stored in these cells and validating their role in each model has proven challenging (Figure <ref>). Each memory cell is the same size as the model domain (120 × 80 × 25), with two memory cells for each of the four causal LSTM layers in the PredRNN++. Given that these memory cells change between each time step and between each evaluation example, manually inspecting and interpreting 𝒞 and ℳ quickly becomes cumbersome. In other words, the hidden states of the PredRNN++ model and, consequently, the internal mechanisms of the model, prove challenging to interpret.



In addition to the PredRNN++, other ML architectures have been shown to be interpretable when applied in different domains and may prove useful in hydrologic surrogate modeling. For example, previous studies have indicated that hierarchical convolutional layers, as employed in CNN3d, can learn hierarchical image information <cit.>. Furthermore, research on transformers like those employed by the ViT3d has demonstrated that attention mechanisms within these models may also learn hierarchical information <cit.>. When combined with conventional ML interpretability methods such as saliency maps, deconvolutional networks, guided backpropagation, and feature occlusion, each of these mechanisms could serve as a check to ensure that a surrogate model is behaving as expected. Future work is needed to explore the interpretablility of each of these architectures when applied to hydrologic simulations.



 §.§ Limitations


While each ML model can reliably reproduce transient pressure fields generated by a process-based model, future researchers should carefully tune models to limit spatial and temporal variability in model performance. For example, when loss is calculated evenly across the entirety of the domain, ML models may not accurately simulate changes in pressure head in the most important portions of the domain. In Figure <ref>, which visualizes one time step from a PredRNN++ emulation of Stage 1, model errors are predominantly concentrated at and above the wetting front. Even though the loss function is weighted to more severely penalize pressure changes within the vadose zone (Figure <ref>), the high MAPE in this area suggests that the ML model still cannot fully resolve sharp hydraulic gradients near the wetting front. Future applications could employ more complex weighted loss functions to ensure that each surrogate most accurately simulates the processes of interest.

The full results that display ML model performance on all stages in Figure <ref> reveal a similar challenge in capturing temporal changes in pressure head. The performance in Stage 1 for all models consistently lags behind that of Stage 2 and Stage 3. Because these later stages simulate shorter time periods (one year in Stage 2 and two years in Stage 3, as opposed to 440 years in Stage 1), pressure head changes over the model stage are less pronounced and their temporal relationships are thereby easier to learn. Thus, while this case study achieves low MAPE, future applications should carefully adapt the implementation strategies in this study (for example, the choice of loss function, data normalization technique and input features) to their problem of interest, especially when simulating model domains characterized by abrupt variations in pressure head across either time or space.



§ CONCLUSION


Machine learning surrogate models offer an efficient approach for generating large sets of hydrologic simulations. All seven of the ML surrogate models produce transient pressure fields that are comparable (below 10% mean absolute percentage error, MAPE) to those produced by the process-based hydrologic model. These transient pressure fields can then be used in downstream tasks, such as quantifying changes in groundwater storage. The improved predictive recurrent neural network (PredRNN++) exhibits the highest accuracy when simulating all three model stages (0.36% MAPE), though other architectures also perform well (0.52–2.4% MAPE). Interestingly, he non-sequential models, which are deeper, have more parameters and compute pressure at all time steps simultaneously, do not exhibit improved accuracy over the one-step models, which compute pressure for each time step sequentially and with no memory between steps.

This hybrid modeling framework also yields substantial runtime benefits (0.115–2.38 processor-seconds/simulation, as opposed to 3,860–69,700 processor-seconds/simulation with a process-based model for Stage 1 simulations) with errors consistently below 10%. Even when accounting for the time it takes to train each model, 6 out of 7 ML surrogate models exhibit runtime advantages when generating as few as 17 hydrologic simulations. Computational savings increase when performing larger numbers of simulations, with 159× speed-up when generating 500 simulations with the U-FNO4d, for example.

Additionally, we identify key practices for efficiently training accurate hydrologic surrogate models, showing that min-max data normalization and loss normalization increase surrogate model accuracy and training stability. Temporal and spatial downsampling, meanwhile, improve computational efficiency by decreasing the memory required to train each model.

Overall, this study offers encouraging results for applying machine learning surrogate models to real-world hydrologic problems, supporting future advancements in hydrologic modeling and simulation techniques.



  §.§.§ Acknowledgements

We thank the Stanford Research Computing Center for providing computational resources and support that contributed to these research results. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1656518. Additional research funding was provided by the Realizing Environmental Innovation Program (REIP) at the Stanford Woods Institute for the Environment.












§ OVERVIEW

This supplementary file contains additional details on the methods used to develop each ML surrogate model, including layer-by-layer information for each surrogate model (<ref>), diagrams of recent model architecture innovations (<ref>), and details on the autoencoder used to compress 3D pressure fields (<ref>). We also present additional results referenced in the main text, including complete MAPE results across all models and stages (<ref>) and a discussion of any error introduced by the autoencoder (<ref>).



§ MATERIALS AND METHODS





 §.§ Model layer tables


1.1
    r l l l 
             Layer    Output shape     Parameter count

        1.     Linear     (-1,120,80,25,64)     576

        2.     Conv3d/BatchNorm3d/ReLU/Dropout     (-1,64,120,80,25)     512,192

        3.     Conv3d/BatchNorm3d/ReLU/Dropout     (-1,128,120,80,25)     1,024,384

        4.     Conv3d/BatchNorm3d/ReLU/Dropout     (-1,64,120,80,25)     1,024,384

        5.     Linear     (-1,120,80,25,1)     65

    CNN3d layer details given (-1, 120, 80, 25, 8) input.
    

    r l l l 
             Layer    Output shape     Parameter count

        1.     Linear     (-1,1,120,80,25,64)     576

        2.     Conv4d/BatchNorm4d/ReLU/Dropout     (-1,64,1,120,80,25)     2,560,128

        3.     Conv4d/BatchNorm4d/ReLU/Dropout     (-1,128,16,120,80,25)     5,120,256

        4.     Conv4d/BatchNorm4d/ReLU/Dropout     (-1,64,16,120,80,25)     5,120,256

        5.     Linear     (-1,16,120,80,25,1)     65

    CNN4d layer details given (-1, 16, 120, 80, 25, 8) input.
    

    r l l l
            Layer    Output shape     Parameter count

        1.     Padding     (-1,120,80,32,8)     0

        2.     Linear     (-1,120,80,32,32)     256

        3.     Linear     (-1,120,80,32,32)     1056

        4.     Fourier3d/Conv1d/Add/ReLU     (-1,32,120,80,32)     4,097,056

        5.     Fourier3d/Conv1d/Add/ReLU     (-1,32,120,80,32)     4,097,056

        6.     Fourier3d/Conv1d/Add/ReLU     (-1,32,120,80,32)     4,097,056

        7.     Fourier3d/Conv1d/U-Net3d/Add/ReLU     (-1,32,120,80,32)     4,618,720

        8.     Fourier3d/Conv1d/U-Net3d/Add/ReLU     (-1,32,120,80,32)     4,618,720

        9.     Fourier3d/Conv1d/U-Net3d/Add/ReLU     (-1,32,120,80,32)     4,618,720

        10.   Linear   (-1,120,80,32,128)     4224

        11.   Linear   (-1,120,80,32,1)     129

        12.   Depadding   (-1,120,80,25,1)     0

        UFNO3d layer details given (-1, 120, 80, 25, 8) inputs.
    

    r l l l
            Layer    Output shape     Parameter count

        1.     Padding     (-1,16,120,80,32,8)     0

        2.     Linear     (-1,16,120,80,32,32)     256

        3.     Fourier4d/Conv1d/Add/ReLU     (-1,32,16,120,80,32)     40,961,056

        4.     Fourier4d/Conv1d/Add/ReLU     (-1,32,16,120,80,32)     40,961,056

        5.     Fourier4d/Conv1d/Add/ReLU     (-1,32,16,120,80,32)     40,961,056

        6.     Fourier4d/Conv1d/U-Net4d/Add/ReLU     (-1,32,16,120,80,32)     42,852,832

        7.     Fourier4d/Conv1d/U-Net4d/Add/ReLU     (-1,32,16,120,80,32)     42,852,832

        8.     Fourier4d/Conv1d/U-Net4d/Add/ReLU     (-1,32,16,120,80,32)     42,852,832

        9.   Linear   (-1,16,120,80,32,128)     4224

        10.   Linear   (-1,16,120,80,32,1)     129

        11.   Depadding   (-1,16,120,80,25,1)     0

        UFNO4d layer details given (-1, 16, 120, 80, 25, 8) inputs.
    

    r l l l
            Layer    Output shape     Parameter count

        1.     Linear     (-1,120,80,25,64)     512

        2.     Conv3d     (-1,64, 5, 5, 5)     7,864,320

        3.     PositionalEncoding     (125, -1, 64)     0

        4.     TransformerEncoder     (125, -1, 64)     2,249,216

        5.     Linear     (-1, 125, 1920)     124,800

        6.     Reshape     (-1, 120, 80, 25, 1)     0

        ViT3d layer details given (-1, 120, 80, 25, 8) inputs.
    

    r l l l
            Layer    Output shape     Parameter count

        1.     Linear     (-1,16, 120,80,25,64)     512

        2.     Conv4d     (-1,64, 8, 5, 5, 5)     15,728,704

        3.     PositionalEncoding     (1000, -1, 64)     0

        4.     TransformerEncoder     (1000, -1, 64)     2,249,216

        5.     Linear     (-1, 1000, 3840)     249,600

        6.     Reshape     (-1, 16, 120, 80, 25, 1)     0

        ViT4d layer details given (-1, 16, 120, 80, 25, 8) inputs.
    

    r l l l
            Layer    Output shape     Parameter count

        1.     Linear     (-1, 120,80,25,32)     256

        2.     CausalLSTM3d     (-1,32, 120, 80, 25)     2,690,752

        3.     GHU     (-1,32, 120, 80, 25)     512,128

        4.     CausalLSTM3d     (-1,32, 120, 80, 25)     2,690,752

        5.     CausalLSTM3d     (-1,32, 120, 80, 25)     2,690,752

        6.     CausalLSTM3d     (-1,32, 120, 80, 25)     2,690,752

        7.     Linear     (-1,120, 80, 25, 1)     33

        PredRNN++ layer details given (-1, 120, 80, 25, 8) inputs.
    




 §.§ Model architecture diagrams




  §.§.§ U-FNO






  §.§.§ PredRNN++






 §.§ Autoencoder details
 
The autoencoder is a convolutional neural network containing four layers and around 676,000 parameters, which makes the autoencoder much smaller than each surrogate model. Two of the four layers are encoding convolutional layers, which squeezes the input into compact spatial dimensions, and the remaining two are decoding transposed convolutional layers, which recovers the original spatial resolution. All convolutional layers are 2D, using the z dimension as the channel dimension. They employ a (7, 7)-sized convolutional kernel, a stride of (2, 2), and a hidden size of 128.  The convolutional layers are separated by ReLU activation functions and batch normalization layers.




 §.§ Implicit depth-wise loss and MAPE weighting






§ RESULTS




 §.§ Mean absolute percentage error across all model stages







 §.§ Error introduced by the autoencoder


The autoencoder, though a vital component in training Stage 3 models, may introduce inaccuracies. We attempt to disentangle the errors from the surrogate model and the errors from the autoencoder in Figure <ref>, focusing on the CNN3d for its efficiency in training. Computing MAPE between the predictions ŷ taken directly from the autoencoder, i.e., ŷ← h_dec(h_enc(y)), and the simulation targets reveals that the autoencoder can reproduce its own input with 0.28% error on the held-out test set (Figure <ref>, Config A). We expect that this autoencoder error contributes to Stage 3 error, since we use the autoencoder in training Stage 3 models; indeed, when computing the MAPE between decoded Stage 3 model predictions and simulation targets, this error, at 0.29%, is greater than the autoencoder error alone. It is surprising, then, when we use the autoencoder to encode and decode the targets and use these encoded and decoded targets as our new evaluation references, our percent error decreases to 0.11%. This shift indicates that the autoencoder introduces error by compressing the space of the target encodings. In other words, our autoencoder encodes targets in a way that makes it more difficult to distinguish between unique targets once they have been encoded. Consequently, during the decoding step, essential information distinguishing unique targets is not fully recovered. Exploring improved autoencoder architectures may help to mitigate these challenges and enhance the overall performance our hybrid framework in future applications.




unsrtnat
  





























 Paper Title: Lyapunov weights to convey the meaning of time in physics-informed neural networks.









Université Paris Dauphine - PSL

CEREMADE, 
 Place du Marechal de Lattre de Tassigny, Paris 75016, FRANCE



[url]https://turinici.com

Gabriel.Turinici@dauphine.fr



§ ABSTRACT 
Time is not a dimension as the others. In Physics-Informed Neural Networks (PINN) several proposals attempted to adapt the time sampling or time weighting to take into account the specifics of this special dimension. But these proposals are not principled and need guidance to be used. We explain here theoretically why the Lyapunov exponents give actionable insights and propose a weighting scheme to automatically adapt to chaotic, periodic or stable dynamics. We characterize theoretically the best weighting scheme under computational constraints as a cumulative exponential integral of the local Lyapunov exponent estimators and show that it performs well in practice under the regimes mentioned above.




 physics-informed neural networks; causal weighting in PINNs; Lyapunox exponent








§ TIME IN PINNS : INTRODUCTION AND PREVIOUS WORKS


Physics-informed Neural Networks introduced in <cit.> proved to be a very successful paradigm invoked to solve complex mathematical equations such as time evolutions 
<cit.> <cit.> possibly in high dimensions <cit.>, 
 partial differential equations <cit.>
 or control systems <cit.>; this framework exploits the power of neural networks (including automatic differentiation) to represent the main unknown function which is the solution of some equation involving its derivatives. The PINNs can also incorporate different other elements such as (possibly noisy) measurements on the system or further data such as controls.

To improve PINN performance several leads have been followed; 
Sharma and Shankar
 <cit.> studied meshless discretization, Cho et al. <cit.> consider separable network architecture while Wang et al. <cit.> analyze 
best norm to express the loss. We borrow here from all these perspectives but we focus on time-dependent problems and more specifically on how the sampling of time points (or their weighting in the loss) impacts the overall performance. The time dimension has been long recognized to possess particular characteristics that deserve special attention. In a highly influential work on 
Allen-Cahn and Cahn-Hilliard  
phase field equations <cit.> Wight and Zhao introduce various sampling strategies in both space and time while McClenny and Braga-Neto
<cit.> make weights trainable which requires additional computational time; in the latter work the influence of the weights on the training loss is mediated by a function that becomes a hyper-parameter to be specified by the user.
In a related approach, 
<cit.> Wang et al. remarked that, especially for chaotic or near chaotic systems, the causality of the time dimension has to be respected and enforced to obtain good results. In particular they proposed that earlier times be given more weight in the loss functional, proportional to the exponential of the cumulative sum of the accumulated errors (up to that point in time, see formula (<ref>) below); a hyper-parameter denoted ϵ remains to be chosen and in practice it is iterated in a prescribed list. The procedure also requires some special termination criterion.  
On the other hand 
<cit.> Penwarden et al. also studied the time dimension and propose a scalable framework for causal sweeping strategies in PINNs. This and many other contribution recognized the necessity to combine PINNs sequentially in order to solve a problem set on a large time interval. While we agree with this perspective, we search here only for the best time weighting scheme on each of these individual intervals ; as such, our procedure can be combined with any other time sweeping protocol. Moreover, as the previous protocols have been rather prescriptive and proposed ad-hoc choices of time weighting, we focus here on the design of principled approaches to orient the choice of  weights.  Our procedure is on one hand flexible enough to recognize the main regime the evolution takes place into (stable, chaotic, periodic) and on the other hand it does not require additional hyper-parameters to adjust, which was a more laborious part of the previous approaches.

The balance of the paper is the following: in section <ref> we introduce the general framework and in section <ref> we give the main theoretical insights that motivate our procedure. The numerical experiments are the object of section <ref>, followed in sections <ref> and <ref> by concluding remarks. 




§ PRESENTATION OF THE FRAMEWORK
 
To set notations let us recall very briefly how PINNs operate. Suppose that the evolution equation is to be solved is written in the form :

    ∂_t u    = _t u
    
    
    u(0,x)     =u_0(x),  ∀ x ∈Ω
    
    u(t,x)    = u_bc(t,x),  ∀ t∈[0,T], x ∈∂Ω.

where u(·,·) is the main unknown, t is the time variable and x the spatial variable, 
_t is some operator possibly involving the 
spatial derivatives of u (e.g., for the heat equation _t = ∂_xx), u_0(·) and u_bc(·,·) are the initial and boundary conditions respectively. When the equation is set in finite dimension such as the Lorenz system in section <ref> then Ω will be discrete and ∂Ω = ∅. The solution u will be searched in the form of a neural network (NN) taking two inputs t and x and outputting an approximation _θ(t,x) of u(t,x) ; θ are the
NN parameters, see figure <ref>.
In order to simplify the presentation we will moreover assume as in <cit.> and related works, that we can construct the network _θ such that _θ satisfies exactly the initial and boundary conditions. For instance for the initial conditions one can simply shift the output of the NN :
_θ(t,x) ↦_θ(t,x)-_θ(0,x)+u_0(x)· t
as in <cit.> or  

    _θ(t,x) ↦_θ(t,x)-_θ(0,x)+u_0(x).

 as in <cit.>.
In order to train the NN a loss is formulated that includes as main ingredient the equation error w(t,x) defined as w(t,x) = ∂_t _θ(t,x) - _t _θ(t,x) or equivalently :

    ∂_t _θ(t,x) = _t _θ(t,x) + w(t,x),  _θ(0,x) = u_0(x).

Our use of this last equation is slightly different from the literature, see also figure <ref> : since the derivatives are here exactly computed using the automatic differentiation capabilities of the NN, the equation 
(<ref>) is also satisfied exactly. So we can consider the error w as being the main output and _θ as the solution of 
(<ref>) that is available at no additional cost.


The NN is trained to minimize a loss functional; the original proposal in <cit.> is 
∫_0^T ∫_Ω | w(t,x)|^2 dx dt but, later on, particular weighting schemes appeared that proposed a loss of the form :

    (θ) = ∫_0^T ∫_Ωρ(t) | w(t,x)|^2 dx dt,

with the weight ρ(t) ≥ 0 to be chosen in order to speed up the convergence and improve the quality of the output. For instance <cit.> use 
a discrete form of 

    ρ(t) = e^-ϵ∫_0^t w(t,x)^2_L^2_x

for some ϵ >0 ; such a choice will give more weight to low values of t. Another proposal is 
<cit.> who sets  
 ρ(t) = e^-ϵ t for some ϵ∈; the intuitive reason is that errors at earlier times will accumulate into larger errors in the final output so  the solution at initial times should be computed more precisely. Although these proposals are inducing better numerical properties, they remain somehow ad-hoc. We investigate below in a principled way 
 how the weights ρ contribute to the quality of the result and set it accordingly.





§ THEORETICAL RESULTS
 

To design time weighting schemes we need to know how the equation error w(t,·) at time t will impact the accuracy of the final result _θ(T,·). There is no general answer to this question but we will use a close concept, namely the Lyapunov exponent which describes  quantitatively 
the rate of separation of infinitesimally close trajectories. The Lyapunov exponent originates from the dynamic system analysis and, for some 1D ordinary differential equation z'(t) = f(t,z(t)) describes how two solutions, starting from z^1(0) and z^2(0) diverge in the limit of large times t; more precisely the Lyapunov exponent is the coefficient λ∈ such that | z^1(t)-z^2(t)| ∼ e^λ t | z^1(0)-z^2(0)| (for large t). The determination of the exact value of the Lyapunov exponent is a difficult task in dynamical systems and ever more so in evolution PDE, but we will retain this idea. This allows to state :

Let u, _θ as in  (<ref>) and (<ref>) respectively. Let λ(t) be such that 

    ⟨_t(u(t,·))- _t(_θ(t,·)), u(t,·)- _θ(t,·) ⟩/u- _θ^2≤λ(t).

Then

    u(T,·)- _θ(T,·)≤∫_0^T e^∫_t^Tλ(s)dsw(t,·) dt.

Here the norm is euclidean when Ω is finite and L^2 norm otherwise.

 Denote e(t) = u(t,·)- _θ(t,·). Using (<ref>) and (<ref>)  we obtain:

    1/2d/dte(t)^2 = ⟨d/dt e(t), e(t)⟩
    =  ⟨_t(u)- _t(_θ)- w(t,·), e(t)⟩
       ≤λ (t) ⟨  e(t), e(t)⟩ -
    ⟨ w(t,·), e(t)⟩≤λ (t)  e(t)^2 + w(t,·)· e(t)

This means that d/dte(t)≤λ (t)e(t) + w(t,·), and, using the notation Λ(t) = e^-∫_0^t λ(s) ds
we obtain d/dt(Λ(t)  e(t) ) ≤Λ(t) w(t,·) ; hence, since e(0) = 0, by integration from 0 to T we get 
Λ(T)  e(T) ≤∫_0^T Λ(t)  w(t,·) dt which gives the conclusion.


The result already gives qualitative insight into how the error at the final time T depends on errors at intermediary times t≤ T, which is the main idea of Lyapunov exponents. Let us take the simple case _t(u) = λ u with λ∈ constant; in this case 
(<ref>) is in fact an equality.
As in <cit.> we note that for a chaotic or divergent system, which is characterized by a strictly positive Lyapunov exponent i.e., Re(λ)>0 the trajectories diverge exponentially with respect to error in the equation (<ref>) or in the data. In this case the precision has to be large for t near zero because the factor e^λ (T-t) will multiply it resulting in large error in the final state. On the contrary, for values of t close to T the factor e^λ (T-t) is relatively small and a larger error in the equation can be tolerated because it will be multiplied by a smaller factor. What is interesting to note is that this also works the other way around for Re(λ)<0 which characterize  systems converging to stable equilibria. In this case initial errors in the equation will be "washed away" by the stable dynamics and are less important and estimation (<ref>) says that we can manage errors increasing exponentially with T-t as long as the exponent is smaller than |Re(λ)| !
Finally, for the (quasi-) periodic dynamics |λ|=1, all times t≤ T contribute equally to the final error and no special weighting should be enforced. 

To add more actionable quantitative details to the discussion above we 
will see now how this translates into weighting schemes for ρ(t); changing the weight ρ(t) indicates to the training algorithm that some values of t should be given priority when minimizing the error loss w(t,·). Of course, more computational resources are available better the algorithm will converge, so we will make our reasoning assuming a given computational level of error i.e., will assume that we can employ numerical algorithms that  make ∫_0^T ρ(t)  w(t,·)^2 dt small enough (but not zero), say equal to C_p and inquire which is the best weighting scheme for this class of errors. The idea is to see how well the choice of ρ guarantees a good behavior of the final error at time T. Note that the training will only minimize the loss without regards as to which of the possible functions w are selected, as long as 
 ∫_0^T ρ(t)  w(t,·)^2 dt reaches the desired level C_p. So here we will assume worse case scenario and see what is the ρ
 whose worse case scenario is the best. This will be formalized as looking for the ρ such that 
 {max_∫_0^T ρ(t)  w(t,·)^2 dt ≤ C_p∫_0^T e^∫_t^T λ(s)ds  w(t,·) dt }
is smaller, see proposition below.
To gain in generality we will accept weighting schemes with irregular densities and write for a law η on [0,T] with density ρ (maybe not a proper function) _τ∼η w(τ,·)^2 dt instead of 
 ∫_0^T ρ(t)  w(t,·)^2 dt.
 For technical reasons we will need the law η to have finite second order moment which we write η∈^2([0,T]) and we assume w is bounded.
The following result gives precise information on the best weighting to be used :

Let λ(s):[0,T] → with 
∫_0^T e^∫_0^t λ(s)ds  dt < ∞ and w such that 
t↦ w(t,·) is bounded. 


Then the  problem

    min_η∈^2([0,T]){max_w; _τ∼η[ w(τ,·)^2 ] ≤ C_p∫_0^T e^∫_t^T λ(s)ds  w(t,·) dt }

admits an optimum which is attained in some η^opt. Moreover the minimum η^opt has a density ρ^opt(t) which is proportional to 
e^- ∫_0^t λ(s) ds that is :

    ρ^opt(t) = e^- ∫_0^t λ(s) ds/∫_0^T e^- ∫_0^t λ(s) ds dt.



We first suppose that ρ is regular enough and look for the solution.
Denote Λ(t) = e^- ∫_0^t λ(s) ds and write 
∫_0^T e^∫_t^T λ(s)ds  w(t,·) dt  = 
1/Λ(T)∫_0^T Λ(t)  w(t,·) dt. Dismissing the constant 
1/Λ(T) we are thus maximizing ∫_0^T Λ(t)  w(t,·) dt under the constraint _τ∼η[ w(τ,·)^2 ] ≤ C_p.
The Cauchy's inequality informs that 

    ( ∫_0^T Λ(t)  w(t,·) dt )^2 ≤_τ∼η[ w(τ,·)^2 ] 
     _τ∼η[(Λ^2(τ)/ ρ^2(τ) ) ]

with equality only when Λ(τ)/ √(ρ(τ)) and √(ρ(t)) w(τ,·) are proportional i.e. 
w(τ,·) is proportional to  Λ(t)/ρ(t). In this case the maximal value is ∫_0^T Λ^2(t) / ρ(t) dt. 
So for a given ρ we cannot guarantee a final error better than
∫_0^T Λ^2(t) / ρ(t) dt (up to a multiplicative constant depending on C_p and Λ(T) but not on ρ).
Now the choice of ρ should set this integral to the smallest value possible under the constraint ∫ρ(t) dt = 1  and ρ≥ 0. Use again the Cauchy inequality to see that 

    ∫_0^T Λ^2(t) / ρ(t) dt  = 
    ∫_0^T ρ(t) dt ∫_0^T Λ^2(t) / ρ(t) dt ≥(∫_0^T Λ(t) dt)^2,

which shows that we cannot do better than (∫_0^T Λ(t) dt)^2, 
with equality when Λ^2(t) / ρ(t) and ρ(t) are proportional i.e.
ρ(t) is proportional to Λ(t), which is the conclusion. 

For any other, possibly non smooth, choice of ρ one can follow the same reasoning by regularization : take smooth approximations of ρ which, by the arguments above turn out to not be better than ρ^opt so passing to the limit one obtains that no non-smooth ρ can do better. 

So the best weights ρ(t) are proportional to 
e^- ∫_0^t λ(s) ds; this quantity is calculated from the values of λ(t). Ideally, one would like to have :

    λ(t) = ⟨_t(_θ(t,·)) - _t(u(t,·)), _θ(t,·)- 
        u(t,·)⟩/_θ(t,·)- u(t,·) ^2,

i.e., the λ that is realizing equality in (<ref>). But the right hand side depends on the exact solution which is the unknown and has to be approximated.  
We will estimate the λ(s) by taking simply :







    λ_n(t) ∼⟨_t(_θ_n(t,·))-_t(0), _θ_n(t,·)
        ⟩/_θ_n(t,·)  ^2

where θ_n is the value of the NN parameters after the n-th training step (_θ_n is the solution candidate at this iteration). 
This amounts to approximating the exact solution (which is unknown) to zero. Although this approximation seems rough, it the most universal to be made and in practice it gives good results because we do not want exact values for λ_n(t) but some general information e.g., on its sign and monotonicity. Of course, if additional information on the exact solution u(t,x) is available this could be incorporated into  (<ref>). 
Furthermore, note  that for the simple situation when 

_t(u) = _t(u) + ξ_t + α_t u,

with _t a anti-symmetric operator, ξ_t independent of u and α_t ∈ the choice (<ref>) results in λ_n(t) = α_t (for all n) which is exact and optimal.



§ NUMERICAL EXPERIMENTS
 

The code for the numerical experiments will be provided on Github https://github.com/gabriel-turinici. 



 §.§ Lorenz system
 

The Lorenz system is a set of ordinary differential equations that describes a simplified model of atmospheric convection. It exhibits chaotic behavior and has been widely studied in the field of dynamical systems; it is described by the following set of ordinary differential equations:

    x'(t) = σ (y - x), 	 
    y'(t) = x (ρ - z) - y, 	 
    z'(t) = x y - β z,







where x, y, and z are the state variables, and σ, ρ, and β are parameters.
Coherent with the literature <cit.> we will take final time T=0.5[In the literature time goes up to T=20 but in practice it is implemented as sequential solves on 40 time intervals of size 0.5.] and use the classical parameter set (initially studied by Lorenz) 
(σ, ρ, β) = (10, 28, 8/3) and the initial state 
(x, y, z)(0)=(1,1,1). This test case is notoriously difficult to solve with PINN  (the default scheme does not even converge, see below) because of its very high sensitivity with respect to errors in the data and equation.










The reference solution, considered exact, is the one obtained with  routine with default settings. 
For the initial condition we used the shift  in (<ref>). 
The NN has 5 hidden fully connected layers of 20 neurons each ('tanh' activation) and a final FC layer with output dimension 3 (to match the unknown dimension) with no activation. All layer initializations are Glorot Uniform. The NN has 1'783 trainable parameters (which compares very favorably with 1'054'723 training parameters used by <cit.> who employ 5 hidden FC layers with 512 neurons). We used a uniform time grid of 256 points (as in <cit.>) and 2000 iterations of the Adam optimizer; initially we tested with the default learning rate (0.001) but to accelerate convergence we then chose everywhere a learning rate of 0.01 (only exception being  figure <ref>  where we compare these two learning rates). All other optimizer parameters are the TensorFlow version 2.15 defaults. A run of the procedure takes 80 seconds on a T4 GPU.

We first show convergence results in figure <ref>. It is seen that the loss reaches low values (and will improve if more iterations are allowed); at the same time we plot (right figure) also the error at final time T between the reference solution and the PINN solution, that also behave well and shows that convergence occurred. This is to be compared with figure <ref> where the numerical solution shows high quality match with the baseline solution. We also checked (not shown here) that given more iteration the quality improves further (but then we cannot distinguish graphically the two).

We now pass to the inner workings of our procedure. First in figure <ref> (right) we note that the equation error is indeed of intuitive form: since the system is very sensitive to errors, it is best to set error to be lower for small t and larger at final times. 
This is coherent with the weights selected (automatically) by our algorithm, depicted in figure <ref> left, that are indeed decreasing. The weight assignment is the result of the estimators (<ref>) plotted in figure <ref> right. We plot both our estimator and the ground truth (<ref>) (that requires knowledge of the exact solution). The agreement is remarkable and it is seen that our estimator (<ref>) captures the good order of magnitude and sign while retaining a more smooth behavior.

As discussed above, we also investigated the influence of the learning rates; we plot in figure <ref> left 
a comparison of the losses during the optimization with two different learning rates, 0.001 (TensorFlow Adam optimized default) and 0.01. As expected from the general theory of stochastic optimization it is seen that the larger learning rate gives faster convergence but also larger final variance (oscillations) at the solution. 

Finally, in figure <ref> right we plot a confidence interval for the loss (learning rate set to 0.01 for all) after 1000 independent runs of the procedure. The variability is due to the initialization of the NN layers and is normal to have runs taking longer time. For all quantiles the decrease in loss is systematic. 








We now turn to comparing these results with equivalent results for two other time weighting algorithms, <cit.> (cumulative exponential error loss (<ref>)) and <cit.> (exponential weighting ρ(t) ∼ e^-λ t). 
We do not even compare with uniform time-weighting (which is the default PINN protocol) because this is known to not work well on Lorenz (and most chaotic) systems. The interested reader can nevertheless find in the supplementary material section <ref> this result.

For both we used exactly the same NN and optimizer settings.
We seeded the random number generation of numpy and TensorFlow with the same seed as in the previous case so all procedures start from the same initial network weights.
The 'causal' procedure seem to not have converged yet cf. figure <ref> even if we gave it 5 times more iterations. We did so because this procedure needs to change the value of ϵ in a schedule involving 5 preset levels in the list 0.01,0.1, 1, 10, 100. To each level we gave 2000 iterations (which is the total number of iterations used by our procedure). We checked that if given even more time the protocol does indeed converges. Obviously it attempts to exploit the same idea of using larger weights for t small resulting in lower equations errors at t small and larger errors for t large (cf. figure <ref> rightmost plot).
In figure <ref> left we plot the convergence of the procedure (the 5 ϵ levels are visible on left plot). Note that, since the Lorenz system is close to chaotic, significant equation errors of the early stages of the evolution equation destroy completely any confidence in the solution for final time T as is shown in figure <ref> right, where the final error between the exact and numerical solution does not appear to improve much. The loss is indeed decreasing but it does not convey enough relevant information to orient the convergence. See supplementary material for further explanation of the weights behavior in this procedure, which appear sensitive to the precise number of iterations run for each  value of ϵ.




We compare now with the exponential time weighting scheme in  <cit.>. The procedure has a hyper-parameter λ  that give the exponential decay of the weights. We tested several choices of parameters 
but will only show results for two of them,
λ=10 (which has the good order of magnitude) and 
λ=13.8; this latter value was chosen to have a posteriori the same average decay as our procedure (and thus should give comparable results). The results in figure  <ref> for λ=10 show that the procedure is sensitive to the choice of the λ parameter; when this parameter is given by some 'oracle' then the solution is much improved. When λ is farther away from the optimal value the results degrade.




 §.§ The Burgers' equation
 
The Burgers' equation models the one-dimensional flow of viscous  fluid and reads :

    ∂ u/∂ t + u ∂ u/∂ x = ν∂^2 u/∂ x^2,  t∈ ]0,T],  x∈]-1,1[ 
        u(0,x) = -sin(π x) 
        u(t,± 1)=0,  ∀ t ∈ [0,T].

where u(x, t) is the fluid velocity, ν=0.01/π is a real positive constant called the viscosity coefficient, x and t are spatial and temporal variables respectively. Total time is T=1.0.
These are supplemented by the initial 
(<ref>) and boundary (<ref>) conditions. For the initial condition we used the shift described in  (<ref>); on the other hand we did nothing special for the boundary conditions which were implemented by adding a corresponding term in the loss as it is classic in these cases <cit.>.

For the 'exact' baseline, as this problem is known to be difficult to solve, a first proposal was obtained with a  
finite difference scheme combining two half-step  Crank-Nicholson propagators for the viscosity term  with a Lax-Friedrichs  propagator for the transport part. A different computation can be proposed through the exact formula (2.5) used by <cit.> combined with Hermite polynomial quadrature ; see the supplementary material  <ref> for all scheme details and analytic formula. We checked that both agree up to 1.0e-4 and used quadrature analytic formula if precision beyond this was required.

As in <cit.> and <cit.> 
the NN architecture uses 8 fully connected hidden layers of 20 neurons each (3021 parameters to train) and learning rate of 0.01; we define a uniform grid of 50 points in time and 25 in space (which gives a total number of interior collocation points of 1250, almost an order of magnitude less than in the references cited). Procedure is run for 3000 iterations in 387 seconds on a T4 GPU.

We plot in figure <ref> the results for our procedure, the causal procedure <cit.> and the standard PINN <cit.> approach. Compared with the Lorenz system this is a non-chaotic case and all algorithms will eventually converge and give good results. The plot confirms that our procedure has some merits in using insights from the system during the initial iterations and improve efficacy. Data in supplementary material shows that this is realized by some counter-intuitive behavior of over-weighting final instants when t is close to T because at this point that the discontinuity is created (and detected by our procedure).







§ LIMITATIONS
 
Although the theoretical and empirical results appear encouraging, further work could improve some aspects of the procedure. The most apparent is the estimation of the Lyapunov exponents, which is now done in a very crude way; this could be refined or, if not possible, at least construct some trust regions to inform when these estimations are reliable or not.

Another aspect is  the interaction between the weights dynamic and the solution. This is common to all adaptive weights procedures but it may concern some cases where the weights given by (<ref>) induce instabilities in the solution preventing convergence (in the same vein as the well-known GAN "mode collapse" problem). An analysis of whether this can happen or how to prevent it could be required.

Finally, for the problems where the canonical, simple implementation of the PINN procedure already works well this algorithm may cause slight numerical computational overhead.



§ CONCLUSION
 

In this work, we presented a new approach to treat the time dimension within the framework of Physics-Informed Neural Network. We started from the observation that time instants are  not permutable and depending on the regime (chaotic, periodic, or stably convergent) errors in earlier instants affect differently the quality of the outcome. We proposed a theoretical explanation based on Lyapunov exponents and then a new algorithm built from this insight.

The procedure was then tested on two different cases, one chaotic (Lorenz system) and one stable but that induces singularities (Burgers' equation), both known to require careful numerical treatment. The proposed algorithm not only showcases robustness and practical efficiency but also addresses some limitations of earlier works that lacked a principled and theoretically grounded foundation. Of course, while our procedure is robust across various benchmarks, it is designed as a complementary addition to the existing methods rather than a one-size-fits-all solution. We believe this work contributes meaningfully to the field, offering a solid foundation for future research and application development.












§ SUPPLEMENTARY MATERIAL




 §.§ Standard PINN (no time weighting) results for the Lorenz system
 

As announced in section <ref> we plot in figure <ref> the solution of the procedure for the situation when the weights are not optimized at all but left all uniform. This case was allowed 30'000 iterations (10 times more than our standard setting) but did not even start to converge.





 §.§ Exact solution for Burgers' equation

To compare the PINN solution with a reference ground truth we solved Burger's equation with a finite difference scheme and compared with an analytic formula with Hermite-Gauss integration.



  §.§.§ Finite differences scheme for Burgers' equation

The finite difference scheme works on a spatial domain
discretized with nx = 400+1 spatial grid points (counting both segment extremities ± 1), 
i.e. Δ x =2/400 
and nt = 700 time steps (Δ t = 1/700);  each time step is implementing the following split-operator technique :

- first a Crank-Nicholson (CN) scheme over a Δ t/2 time step (taking into account the boundary conditions) for the diffusion part i.e. the solution by CN of the equation ∂_t u = ν∂_xxu 

- then a Δ t step of a Lax-Friedrichs scheme for the viscosity term, i.e. for the equation
∂_t u + u ∂_x u = 0.

- then a final Crank-Nicholson (CN) scheme over a Δ t/2 time step (taking into account the boundary conditions) for the diffusion part

The splitting technique is known to be of high order in time <cit.> 
which leaves the Crank-Nicholson part (second order in time) and Lax-Friedrichs scheme (first order in time and  space) as limiting factors. As for stability it is known that CN is unconditionally stable for the heat diffusion and since the matrix for the Lax-Friedrichs scheme is tridiagonal by Gersgorin theorem it is stable when max(|u|) ·Δ t/ Δ x<1.

In practice, if V∈^nx+1  is the current solution with V_j the component  representing the solution at point x=-1 + j·Δ x), the CN propagation for a Δ t/2 time step means solving the linear system in V^next : 

    V^next_j- V^next_j/Δ t/2 = ν/2(
         V_j+1+ V_j-1-2 V_j/Δ x^2  +
        V^next_j+1+ V^next_j-1-2 V^next_j/Δ x^2), 
     V^next_0=0=V^next_nx.


The  Lax-Friedrichs scheme means replacing V with :

    V^next_j- V_j+1+ V_j-1/2/Δ t +  (
         V_j+1^2/2- V_j-1^2/2/2 Δ x)=0, 
     V^next_0=0=V^next_nx.

Once this is solved the resulting 2D solution matrix is used as input to a 2D interpolation routine to obtain a linear interpolation function that is able to output values for any other point not necessarily on the space-time grid used by the finite difference resolution.



  §.§.§ Analytic formula for the solution of the Burgers' equation


Another way to find a solution is to used the analytic formula (2.5) from <cit.>. It reads

    u(t,x) = ∫_ u_0(x-η) ζ(x-η) e^-η^2/(4ν t) d η/∫_ζ(x-η) e^-η^2/(4ν t) d η,  where ζ(y) = e^-cos(π y) / (2 πν).

The integrals can be computed using Hermite-Gauss quadrature and in practice we used 50-th order formulas after a change of variables η = η' √(4 ν t ). For t < 10^-10 we used a first order Taylor development with respect to t using the exact formula for the time derivative from the Burgers' equation.




 §.§ Behavior of weights of the causal procedure (Lorenz system)

 
To explore the convergence properties of the causal protocol <cit.> we looked at the history of the weights used by the procedure during the resolution of the Lorenz system, see comments in section <ref>; these are plotted in figure <ref>.
The algorithm's protocol iterates the ϵ parameter ; at each change the weights associated with latter times are smaller which causes deterioration of the quality there. So the algorithm can only advance to the next value of ϵ only after the equation is satisfactory solved at initial times.





 §.§ Weights for Burgers' equation

We compare in figure <ref> the final weights of the three procedures considered for the Burgers' equation, we note that our proposal over-weights final times which is when the discontinuity appears. This is not the case of the other procedures ; this behavior explains the results in last columns of figure <ref> (rows 2 and 3) that clearly lack precision at that point in space.





 Paper Title: Coarse Correspondence Elicit 3D Spacetime Understanding in Multimodal Language Model.



Quantized electrical, thermal, and spin transports of non-Hermitian clean and dirty two-dimensional topological insulators and superconductors
    Bitan Roy
    August 5, 2024
==============================================================================================================================================

§ ABSTRACT 
Multimodal language models (MLLMs) are increasingly being implemented in real-world environments, necessitating their ability to interpret 3D spaces and comprehend temporal dynamics. Despite their potential, current top models within our community still fall short in adequately understanding spatial and temporal dimensions.
We introduce , a simple, training-free, effective, and general-purpose visual prompting method to elicit 3D and temporal understanding in multimodal LLMs.
Our method uses a lightweight tracking model to find object correspondences between frames in a video or between sets of image viewpoints. It selects the most frequent object instances and visualizes them with markers with unique IDs in the image.
With this simple approach, we achieve state-of-the-art results on 3D understanding benchmarks including ScanQA (+20.5%) and a subset of OpenEQA (+9.7%), and on long-form video benchmarks such as EgoSchema (+6.0%). We also curate a small diagnostic dataset to evaluate whether MLLMs can reason about space from a described viewpoint other than the camera viewpoint. Again, improves spatial perspective-taking abilities but we highlight that MLLMs struggle with this task.
Together, we demonstrate that our simple prompting method can significantly aid downstream tasks that require 3D or temporal reasoning.





§ INTRODUCTION



Multimodal LLMs (MLLMs) today are part of larger applications in the physical world through devices like smartphones, smart glasses, and robots <cit.>. Augmented with vision encoders <cit.>, proprietary models like GPT-4o <cit.> and Gemini-Pro <cit.> have recently been demoed to understand users' physical environment as well as reason about their actions over time.
Despite their promise, research benchmarks have continued to highlight that our community's best models still lack sufficient 3D understanding of space and time <cit.>.
Popular 3D (e.g. ScanQA <cit.> and OpenEQA <cit.>) and video (e.g. EgoSchema <cit.>) benchmarks report that MLLMs struggle when asked to reason about spatial relationships between objects (“Is the couch to the right of the door”?) or remember temporal events in long videos (“How does [the person] ensure the completion of their primary goal in the video?”)

Many recent efforts have begun incorporating 3D or temporal information to improve reasoning.
For instance, new methods finetune MLLMs with specialized structures or representations <cit.>. 
Unfortunately, these methods require an open-sourced model with weights that can be modified.
In parallel, visual prompting mechanisms (e.g. set-of-marks <cit.> and 3DAxiesPrompts <cit.>) augment image inputs with visualizations to guide the MLLM's attention to specific 2D regions in an image. Visual prompting methods so far show improvements for 2D understanding but not yet for 3D and temporal reasoning.
Worse, these efforts have focused either on 3D understanding or temporal understanding—not both. 
Even if visual prompting methods could add useful information, encoding that information  meaningfully in a video for an MLLM is difficult:
Proprietary MLLMs are developed primarily using image inputs and not videos; when presented with a video, they sparsify and preprocess videos before sending it to the MLLM.



To address the above challenges, We introduce , a propose a simple, effective, and general-purpose visual prompting method to evoke 3D and temporal understanding in multimodal LLMs. Given a video or a set of images of a space, transforms the images/frames to help MLLMs reason.
First, we use a lightweight tracking model <cit.> to extract object correspondences for the same instance across multiple frames at the video's original high frame rate.
Next, we sparsify the videos to reduce the number of frames we pass to the MLLM. 
Finally, we visualize the top-K objects that appear in the most number of frames, sorted by the number of pixels they occupy. We use marks and segmentation outlines with unique IDs to highlight these objects (Figure <ref>).
Notice that these images are modified independent of any underlying questions or tasks.
Finally, we pass these modified images, along with the question to an MLLM.
In our experiments, we apply our visual prompting method to GPT-4V, GPT-4O, and other MLLMs.

Our framework consistently achieves significant improvements over state-of-the-art results on ScanQA <cit.> and OpenEQA <cit.>, two 3D understanding benchmarks, and EgoSchema <cit.>, a long video understanding benchmark. On ScanQA, our framework allows GPT-4V <cit.> to surpass specially designed and fine-tuned methods <cit.> in a zero-shot manner with fewer views. 
Additionally, the rapid advancement of MLLMs provides the potential for our method to elicit other emergent understanding abilities. For instance, with GPT-4O, our method enables 3D spatial understanding from extremely sparse views, reducing the computational cost and expense associated with MLLMs.

To further test for 3D spatial reasoning, we create a small diagnostic benchmark called . 
Current benchmarks assess a model's 3D spatial awareness from the viewpoint of the camera-holding observer. In cognitive science, the spatial orientation test (SOT) is commonly used to evaluate spatial intelligence in children <cit.>. It measures spatial perspective-taking, which is the capacity to envision how an object or scene looks from a different viewpoint <cit.>.
contains 10 scenes with videos from different perspectives. Using these scenes, we test whether MLLMs can reason about spatial relationships from different perspectives.
We show that improves the GPT-4 models' abilities. But overall, we find that their performance is at random chance.


In summary, we propose , a general visual prompting method that combines lightweight tracking models to induce 3D and temporal reasoning in any MLLMs. Our results across 4 3D, temporal, and our new spatial cognition benchmarks verify that our method works out-of-the-box. 
We also demonstrated that even the most advanced MLLMs have significant shortcomings in spatial understanding that are not easily overcome. We hope this work can help MLLMs better understand the physical world we live in.


§ RELATED WORK



Multimodal language models
Multimodal LLMs<cit.> integrate vision encoders <cit.> into large LLMs <cit.>, allowing them to directly reason over visual input. Many proprietary models, such as GPT-4 <cit.>, Gemini <cit.>, and Claude <cit.>, as well as open-source models like the LLaVA series <cit.> and BLIP series <cit.>, have made significant progress in 2D vision-language tasks like image captioning <cit.> and visual question answering (VQA) <cit.>. Beyond these language-related tasks, many newer attempts applying MLLMs to applications such as autonomous driving <cit.> and robotics <cit.>. Many of these tasks require understanding the 3D space in which they are deployed and reason about how things are changing temporally. We improve the 3D space-time capabilities of such models.

Visual prompting.
Effective prompting has been widely proven to improve LLMs across multiple domains. Methods, such as chain-of-thought prompting <cit.>, force the model to reason before answering a question.
For multimodal LLMs, methods such as Red-circle prompting <cit.> and Set-of-marks <cit.> can enhance the grounding abilities of CLIP <cit.> and GPT-4V.
PIVOT <cit.> employs visual prompting combined with iterative VQA to induce GPT-4V to generate outputs for robotics control. 3DAxies <cit.> enhances GPT-4V's ability to use numerical expressions to describe 3D relationships of objects in a single image by annotating a scaled 3D coordinate system on the image.
Unlike these works, prompts MLLMs to understand the spatial relationships within a complete 3D scene from an image sequence. 

Video understanding.
Videos carry rich information about both the 3D structure as well as temporal changes in the physical world. To perform better long-horizon reasoning, work has begun incorporating video inputs into MLLMs. 

By replacing the image encoder in MLLMs with a video encoder <cit.>, recent work has improved performance on video dense captioning <cit.> and videoQA <cit.>.
To further advance the understanding of temporal relationships in videos, EgoSchema <cit.> introduced a benchmark for long video understanding, which is more challenging than previous video-language benchmarks.
Meanwhile, understanding 3D spatial relationships in videos received relatively less attention. 3D-LLM <cit.> converts multiview images into 3D point clouds and then feeds them into LLMs, demonstrating better results on the ScanQA <cit.> benchmark for 3D understanding. OpenEQA <cit.> is also a benchmark dedicated to evaluating MLLM's understanding of 3D physical space, with outputs that are more open-vocabulary compared to ScanQA.
In this paper, we propose a framework that does not require any training in modifying MLLMs; it extracts meaningful information from videos using off-the-shelf tracking models and achieves state-of-the-art performance on the benchmarks mentioned. 

Visual correspondences.
Visual correspondences have been a vital area of research in computer vision for a few decades. Applications such as Structure-from-Motion<cit.> utilize correspondences to better reconstruct 3D scenes. In the past, we relied on handcrafted features like SIFT <cit.> or SURF <cit.> to obtain good correspondence. Today, features extracted from deep models <cit.> can also provide increasingly accurate correspondences. Generally, people aim to achieve precise geometric and semantic correspondences at the pixel level. However, in this paper, we use coarse visual correspondence to prompt MLLMs, which can be easily obtained from off-the-shelf video tracking models <cit.>.





§ METHOD

We introduce , a visual prompting method that allows MLLMs to reason about 3D space and time. 

Problem formulation. Given a question 𝒬 and a sequence or set of observations in an environment [I_1, …, I_n], 
our aim is to design a visual prompt 𝒫(…) that modifies the input image set. These image inputs don't have to be a video. They can also represent a set of images of a scene from multiple viewpoints.
We evaluate the prompt by measuring its utility in prompting an MLLM ℳ:

    [I_1^', …, I_n^'] = P([I_1, …, I_n])
       𝒜̂ = ℳ(([I_1^', …, I_n^']), 𝒬)

We compare the generated answer 𝒜̂ with the ground truth 𝒜.


In our framework, the MLLMs can be any general-purpose model without requiring any special architecture or training for 3D or temporal understanding. Our aim is to develop a prompting strategy that allows models to improve such capabilities without any training (Figure <ref>).




 §.§ Coarse Correspondence

Our prompting method, , contains four steps: (1) tracking correspondences, (2) sparsify frames, (3) selecting, and (4) visualizing coarse correspondences. 

(1) Tracking correspondences. Given n input images, [I_1, …, I_n], we first use an off-the-shelf video object tracking model, such as Tracking Anything <cit.>. This model extracts class-agnostic instance segmentation masks (M_1, …, M_n ) for each image. Each M_i is a H × W dimensional matrix where H and W are the height and width of the input image I_i. Each pixel location in M_i contains an instance ID, indicating which instance the pixel at that position belongs to within the image sequence. 


(2) Sparsify frames.
Since most MLLMs contain a large number of parameters, directly using them to process long image sequences is very computationally intensive. Additionally, proprietary MLLMs like GPT-4O can also incur significant costs if the number of image tokens that need to be processed increases. Reducing the number of input images might lose vital information necessary for MLLMs. 

strikes a balance in this tradeoff by extracting meaningful video object tracks (a relatively cheaper operation) from high-frame-rate image sequences, and then samples a few image inputs along with the tracks, to retain—and even improve—performance while reducing the MLLM's computation cost.
From this extracted video object tracks, we perform temporal downsampling, retaining only m << n uniformly sampled images and their corresponding masks, denoted as [I_s_1, …, I_s_m] and [M_s_1, …, M_s_m], where s_i ∈{1, …, n}.
This downsampling reduces the number of images we feed into ℳ. 


(3) Selecting coarse correspondences.
Prompting an MLLM with all the detected correspondences results in information overload. In fact, our ablations (discussed in Sec <ref>) find that adding all the correspondences reduces the MLLM's performance.
Therefore, we select a subset of prominent instances to retain. We select the prominent instances of the top-K objects that co-occur in the most number of frames.
We first calculate the occurrence frequency and area sum of each unique instance ID in the retained m masks using the following equation:

    ℱreq(ID)    = ∑_i=s_1^s_m1_{ID∈ M_i}, 
    𝒜rea(ID)    = ∑_i=s_1^s_m∑_p ∈ M_i1_{ID = p}.

Then, we first sort all instance IDs in descending order based on ℱreq(ID). If there are ties, we further sort based on 𝒜rea(ID). Finally, we retain the top k instance IDs as tracklets, denoted as [T_1, …, T_k], to visualize for MLLMs.

(4) Visualizing coarse correspondences.
For each set of obtained correspondence relationships, we visualize the correspondences directly in the image as a marker. Specifically, for each identified primary instance ID T_i, if it exists in the mask M_s_j of a retained image I_s_j, we overlay a mark with a fixed size and shape labeled with T_i at the position (x̅_ij, y̅_ij) on I_s_j to produce I_s_j^'. The specific placement position can be easily obtained by:

    (x̅_ij, y̅_ij) = ∑_(x,y) (x, y) ·1_{M_s_j(x,y) = T_i}/∑_(x,y)1_{M_s_j(x,y) = T_i}


Naturally, we can overlay not just the markers but also the segmentation outlines or even the segmentation masks associated with each retained prominent instance. We explore these ablations later. In the end, we obtain the prompted image sequence [I_1^', …, I_m^'], which is then used as the input to MLLMs. 

We refer to our method as Coarse because of the following: first, we only visually prompt for instance-level correspondences and not point-level correspondences. 
Second, the instance-level correspondences are extracted using off-the-shelf tracking models. Despite not being perfectly precise, they still help MLLMs build a better 3D model of the environment. Third, we only visualize a handful of prominent corresponding instances.











§ EXPERIMENTS

We evaluate the utility of across a number of tasks that require understanding 3D space (ScanQA <cit.> in <ref> and OpenEQA <cit.>) in <ref>) as well as temporal events (EgoSchema <cit.> in <ref>). 
Across all three benchmarks, we augment GPT-4V and GPT-4O with and evaluate its zero-shot performance.
We show that not only significantly improves the base GPT models but that the improvements establish new state-of-the-art results across all our evaluations. All experiments are conducted with A100 80G GPUs.


Existing benchmarks evaluate the 3D spatial awareness of a model from the perspective of the camera-wielding observer. To further demonstrate the utility of , we introduce a new benchmark: . Our benchmark tests whether models are able to reason about 3D space from the perspective of an imaginary observer at another location in their field of view.
Again, we show that significantly improves GPT-4{V,O}'s abilities.



 §.§ Spatial understanding with ScanQA







The ScanQA dataset <cit.>.
The ScanQA dataset consists of 41363 questions about 800 scenes, including 32337 unique questions. Its validation set contains 4675 questions about 71 scenes. Because the dataset is derived from the original ScanNet dataset <cit.>, it also contains 3D localization of all objects in the scene. 
Questions in ScanQA require basic recognition, 3D localization, and 3D embodied capabilities <cit.>. 
The validation set contains two ground-truth answers per question for evaluation with models that produce free-form answers.

Baselines. We evaluate by augmenting both GPT-4{V,O}, Gemini and Claude models. We also compare against other general-purpose multimodal LLMs such as LLaVA <cit.>, flamingo <cit.>, and BLIP2 <cit.>. We also consider 3D-LLM which was trained with 3D awareness <cit.>. 

Metrics. Following prior works, we adopt  BLEU <cit.> scores, METEOR <cit.>, ROUHE-L <cit.> and CIDEr<cit.> as our evaluation metrics

Results.
As shown in <ref>, compared with raw input to GPT-4V and GPT-4O, constantly improves the recently released GPT-4o's overall performance by a significant 5.7 BLEU-2, 3.2 METEOR, 6.5 ROUHE-L, and 15 CIDEr points. 

Compared with other vision-language models, it is clear that helps improve 3D understanding for different GPTs constantly; GPT-4V and GPT-4O both fail to achieve competitive performance without but surpass the state-of-the-art models with it. 






 §.§ Spatial understanding with OpenEQA



The OpenEQA dataset <cit.>.
OpenEQA is an open-vocabulary dataset benchmarking spatial environment understanding and embodied reasoning. We evaluate on OpenEQA's EM-EQA data split, which contains over 1600 high-quality human-generated questions collected from various real-world environments.
The subset tests the episodic memory of an agent moving through a 3D environment over time.
This task uses video inputs, from which we sample only K frames at equal intervals as input to the GPT models. We use K=8 frames for GPT-4V and only K=4 for GPT-4O, which we find to be the best performance empirically for the GPT models without . We adopt accuracy as our metric.

Baselines. We compare against language-only models to account for language bias (LLaMA2 <cit.>), commonly used general-purpose multimodal LLMs (GPT-4 <cit.>, Claude3 <cit.>, Gemini-Pro <cit.>, GPT-4V with 15 and 50 frames.

Results.
again achieves state-of-the-art performance while utilizing significantly fewer frames than all previous works (Table <ref>). This substantial reduction in the number of views highlights the efficiency of our approach during inference time. 
The improved performance, coupled with reduced computational overhead, charts a potential future for using for embodied AI tasks.
Despite our marks being generated unconditioned on the questions, they still improve performance.


  
  [t]0.48
    
      Models     Frame     Accuracy 

      LLaMA2 <cit.>     0     28.3 

      GPT-4 <cit.>     0     33.5 

      Claude3 <cit.>     20     36.3 

      Gemini-Pro <cit.>     15     44.9 

      GPT-4V <cit.>     15     54.6 

      GPT-4V <cit.>     50     55.3 

      Human     Full     86.8 

      GPT-4V     8     44.8 

      gray!20 GPT-4V+CC     8     58.5 

      GPT-4O     4     49.4 

      gray!20 GPT-4O+CC     4     59.1 

      
    tableComparisons on EM-EQA setting of OpenEQA . Our method further enhances the embodied ability of MLLMs and exceeds previous methods by a large margin.
    
  
  [t]0.48
    
      Models     Frame     Subset 

      LongViviT <cit.>     256     56.8 

      MC-ViT-L <cit.>     128+     62.6 

      LLoVi <cit.>     180     58.3 

      VideoAgent <cit.>     8.4     60.2 

      MVU <cit.>     16     60.3 

      VideoAgent <cit.>     -     62.8 

      LangRepo <cit.>     -     66.2 

      GPT-4V     8     64.2 

      gray!20 GPT-4V+CC     8     67.4 

      GPT-4O     8     67.2 

      gray!20 GPT-4O+CC     8     73.2 

      
    tableComparisons on EgoSchema validation set. improves existing MLLMs and surpasses previous finetuned models in a zero-shot manner.
    
  

  


 §.§ Temporal understanding



The EgoSchema dataset <cit.>. EgoSchema aims at benchmarking a model's long video understanding ability. Due to budget constraints, we limit this evaluation to 500 questions from the validation set. For the GPT models' input, we sample K=8 image frames uniformly from the video input.

Results. 
demonstrates state-of-the-art performance, significantly outperforms existing approaches in a zero-shot manner (Table <ref>) with fewer frames. The reduction in frame usage underscores 's ability to extract relevant temporal information more effectively, suggesting the possibility of using it for other video tasks in the future.




 §.§ The benchmark for Spatial Orientation Test





In cognitive science, the spatial orientation test (SOT) is a widely adopted examination for spatial intelligence in children <cit.>.
The SOT assesses spatial perspective-taking—the ability to imagine how an object or scene would appear from a perspective different from the current camera viewpoint. Numerous human studies <cit.> have shown that this ability is closely related to the development of spatial intelligence in children. In this section,
We adopt this test to evaluate multimodal LLMs.


Data curation. 
We manually curated ten real-world scenes, both indoor and outdoor, using different mobile devices at various viewpoints. We instructed 10 human participants to take two videos in their environment from two viewpoints. When in each viewpoint, they were asked to remain in place as they laterally pan their mobile devices to scan their 3D environment. 
From 20 collected scenes, we filtered to and retained 10 scenes that satisfied the following four criteria:
First, we could uniquely describe one viewpoint from the perspective of the other and vice-versa. For example, in Figure <ref>, we define the other viewpoints as 'a person stepping out of an elevator.'
Second, we ensured that no single frame captured the entire 3D space, ensuring that models can not short-cut answers using any single view.
Third, all scans move the camera from left to right.
Fourth, to avoid privacy concerns, we ensured that no people appeared in the videos.
Each scene scan lasts between 3 to 5 seconds. 

For each scene, we designed five carefully crafted questions, each asking the model to determine if one object is to the left or to the right of another from a specific viewpoint. The first three questions are from the observer's (camera's) perspective, while the final two describe the perspective in language, thereby, testing for a model's spatial perspective-taking ability. 

Human performance on these questions is 100%. 
We design questions to have a bias towards asking about relationships between objects that appear in the first last frame of the scan, ensuring that the has to use multiple frames to answer.
In total, across the 10 scenes, has a modest 50 questions.



Results. 

significantly improves GPT-4O in perspective-taking ability by a significant 21.2% (Table <ref>). Notably, when using only the first and last frames, as illustrated in Figure <ref>, our method enables GPT-4O to understand the 3D spatial structure represented by the images using minimal overlap, whereas GPT-4O alone performs only slightly better than random guessing.

We further investigate two striking findings. 
First, the GPT models exhibit a bias where they perform better when the camera moves from left to right. But if the order flipped, they perform worse. We report both the L->R and R->L camera pans and also report their harmonic mean. 
Second, when we isolate the performance on the 2 perspective-taking questions per scene in Figure <ref>. While improves GPT-4O's perspective-taking capability, these results are bittersweet since they still perform worse than random guessing.





§ ANALYSIS




Here, we explore the various design decisions implicit in our method.


How does differ from other visual prompting methods? Our proposed method calculates and highlights correspondences between images, aiming to elicit 3D and temporal understanding.
Other visual prompting methods (namely Set-of-Mark, 3DAxiesPrompts, and Chain-of-thought <cit.>) can also be viewed as alternative prompting methods. We compare against them on ScanQA qualitatively in <Ref>.

The orange part of <Ref> shows our Coarse Correspondence labels are recognized by GPT-4V. 
The output answer provides evidence that our coarse correspondence helps GPT-4V develop a mental 3D model of the scene.
Set-of-Marks <cit.> provides no spatial corresponding information and therefore is unhelpful.
The Axis labels in 3DAxies<cit.> can be easily misrecognized by GPT-4V, leading to misleading spatial information.
Though Chain-of-Thought <cit.> helps
identify objects, it fails to resolve the “spatial perspective-taking” issue. 




Why use coarse instead of dense correspondences?

Instead of filtering and retaining only a handful of coarse correspondences, one ablation we considered is the possibility of using all dense correspondence. Unfortunately, we find that excessively overlaying too many instance marks can degrade performance (Table <ref>) as they occlude the visual content in the images.

How large should the marks be?
We inject the correspondences into MLLMs by overlaying the marks into images. 
We empirically find an optimal mark size (where `px' represents the mark's diameter in pixels) in Table <ref>. Marks that are too small tend to be ignored while those that are too large occlude visual content.

What shape should the marks be?
We further studied the appearance of the marks. In addition to red circles with white text, we experimented with adding segmentation outlines and segmentation masks. As shown in Table <ref>, using segmentation outlines enhances object grounding. However, using segmentation marks occludes visual content and reduces performance.







§ DISCUSSION


Limitations.
We would like to point out two limitations of our method. First, our method relies on off-the-shelf video tracking models to obtain instance-level correspondences. 
Although the performance of tracking models has significantly improved with the advent of tools like SAM <cit.>, achieving good results on long-form in-the-wild videos remains challenging. 
This is particularly evident on the 180-second EgoSchema benchmark, where Track-Anything often loses track of objects after 100 seconds, leading to inconsistent instance segmentation masks between the beginning and end of the video clip. 
Despite observing consistent and significant improvements on EgoSchema, we believe that accurate correspondence would further enhance the benefits of our approach.
Second, we find that our method does not yet improve the 3D understanding of open-source models like LLaVA <cit.>. We speculate that there are two reasons for this: either LLaVA may not recognize the marks we overlay on the images or LLaVA may not effectively handle multiple image sequences appropriately. We find that LLaVA often focuses on the content in the last input image. There is a need to develop open-source models with visual promptable capabilities and with multi-image inputs.


Conclusion.
We propose a framework called prompting. By using off-the-shelf video tracking models at a high frame rate to obtain class-agnostic instance-level correspondences, we enable better recognition and reasoning with very few frames. 
Our method is easy to use, doesn't require any training, and performs zero-shot with strong models like GPT-4.
enables GPT to reason about the 3D structure from very sparse views.
We demonstrated consistent and significant improvements across multiple spatial and temporal understanding benchmarks.
We also identified that even GPT struggles with perspective-taking capability, a fundamental component of human visual intelligence.
Regardless, 's results highlight its potential for robotics and embodied AI, which require 3D or temporal understanding.











§ ACKNOWLEDGEMENT

We thank Xiaojuan Wang, Luming Tang, Ruitao Zhang, Yuntian Deng for helpful discussions, feedback and collecting data. This project is partially funded by Amazon Science. 

unsrt








§ APPENDIX



§ BROADER IMPACT

Our method aims at improving the trustworthiness and reliability of deployment of MLLMs in real world application, including but not limited to Vision Pro, autonomous driving, and also humanoid robots. To have a virtual assistant like JARVIS in Marvel films, it's necassry to align the understanding of vision-language model with human's understanding, so that we can ensure safe application of these applications. 
Further, we are committed to reducing the carbon emissions produced by these models. By employing our coarse correspondence prompting method, we use a much smaller tracking module to reduce the number of input used as input to large GPT model. Besides, we also improve the speed and lower the cost of calling OpenAI API to understand a 3d scene. This enables democratize MLLMs so that more people and small companies can create their own real-world applications based on GPT-4V. We hope our work can make large AI models more effectively used for social good.

Still, we would like to point out that with the development of MLLMs, increased reliance on advanced MLLMs could also lead to a reduction in human skills, especially in interpreting and interacting with visual content. Over-dependence on these models might erode critical thinking and analytical abilities in the long term.





§ MORE DISCUSSIONS

Relation to SlowFast
SlowFast <cit.> is a framework for video recognition that includes two parallel pathways: a Slow pathway that captures motion information at a high frame rate and a Fast pathway that captures semantic information at a low frame rate. The information from both pathways is fused through lateral connections for downstream video recognition tasks.
In a way, our coarse correspondence prompting can be seen as another form of SlowFast. However, unlike SlowFast, where the Slow and Fast pathways operate in parallel, our framework operates sequentially. First, it captures low-level, class-agnostic motion information at a high frame rate using a lightweight tracking model. Then, at a low frame rate, it performs recognition and reasoning requiring semantic understanding using larger MLLMs. The two stages are bridged through visual prompting.
Moreover, while SlowFast learns a representation of the input video for pure vision recognition tasks such as action classification and detection, our coarse correspondence framework aims to better understand the 3D spatial structure and temporal information contained in the input video to achieve spatiotemporal perception and reasoning simultaneously.

Eulerian vs Lagrangian If deep learning-based methods represent camera or object motion in videos from an Eulerian viewpoint—i.e., expressing how features at fixed locations evolve over time through a multi-dimensional tensor—then our framework adds a Lagrangian viewpoint to this representation. The Lagrangian viewpoint describes the trajectories of entities moving through space and time in the video. Previously, the Lagrangian viewpoint in video descriptions has been shown to better aid human action recognition <cit.>. Here, we demonstrate that it can more generally help 
MLLMs understand the 4D spatiotemporal context represented in videos.




§ QUALITATIVE CASE STUDY





To further demonstrate the effectiveness of our proposed Coarse Correspondence under sparse image input,
we defined two challenging tasks and one qualitative case study for each task.

The results of these case studies are shown in Fig. <ref>.
Detailed illustration of the results are provided in the figure captions.
The first case study is about the task of Duplicate Objects Counting,
where the model needs to count the number of objects in a 3D scene.
Only equipped with coarse correspondence can GPT-4V get a comprehensive understanding of the 3D scene, excludes the duplicate objects, and give the right answer.The second case study is about the task of Relative Location Modeling,
where the model needs to understand the relative location of objects in a 3D scene.
It is obvious that without the correspondence markers, GPT-4V fails to response from 3D perspective with only raw 2D images.These case studies demonstrate that our proposed Coarse Correspondence can elicit
MLLMs in understanding 3D scenes from sparse image inputs.







§ USER-FRIENDLY INTERACTIONS

We also prove that our Coarse Correspondence method works well with 
hand-crafted correspondence marks as shown in  <Ref>. 
This demonstrates that our method is highly user-friendly for utilizing proprietary multimodal language models, such as GPT-4O, in web interfaces. Users can easily complete prompts by marking correspondence relationships on images. Moreover, the marks can be diverse and flexible. This also proves the robustness of our method, as the marks are style-agnostic, as long as they convey the visual correspondence knowledge.












 Paper Title: Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversion.





0000-0002-4238-7560

 ETH Zürich & DisneyResearch|Studios
 Zürich
 Switzerland
manuel.kansy@inf.ethz.ch

0000-0002-2315-5166

 DisneyResearch|Studios
 Zürich
 Switzerland

jacek.naruniec@disneyresearch.com

0000-0003-1473-1878

 DisneyResearch|Studios
 Zürich
 Switzerland

christopher.schroers@disneyresearch.com

0000-0002-6838-9775

 ETH Zürich & DisneyResearch|Studios
 Zürich
 Switzerland

christopher.schroers@disneyresearch.com

0000-0003-1196-5425

 DisneyResearch|Studios
 Zürich
 Switzerland

romann.weber@disneyresearch.com

§ ABSTRACT 

Recent years have seen a tremendous improvement in the quality of video generation and editing approaches. While several techniques focus on editing appearance, few address motion. Current approaches using text, trajectories, or bounding boxes are limited to simple motions, so we specify motions with a single motion reference video instead. We further propose to use a pre-trained image-to-video model rather than a text-to-video model. This approach allows us to preserve the exact appearance and position of a target object or scene and helps disentangle appearance from motion.

Our method, called motion-textual inversion, leverages our observation that image-to-video models extract appearance mainly from the (latent) image input, while the text/image embedding injected via cross-attention predominantly controls motion. We thus represent motion using text/image embedding tokens. By operating on an inflated motion-text embedding containing multiple text/image embedding tokens per frame, we achieve a high temporal motion granularity. Once optimized on the motion reference video, this embedding can be applied to various target images to generate videos with semantically similar motions.

Our approach does not require spatial alignment between the motion reference video and target image, generalizes across various domains, and can be applied to various tasks such as full-body and face reenactment, as well as controlling the motion of inanimate objects and the camera. We empirically demonstrate the effectiveness of our method in the semantic video motion transfer task, significantly outperforming existing methods in this context.









    
    < g r a p h i c s >

    1
    73.65mm
      [width=0.0875]7figures/teaser/body/frames_input/001014 
    
    73.65mm
      [width=0.0875, poster=3]7figures/teaser/body/frames/001014 
    
    62.55mm
      [width=0.0875, poster=3]7figures/teaser/body/frames_domain/001014 
    
    51.45mm
      [width=0.0875, poster=3]7figures/teaser/body/frames_multiple/001014 
    
    51.45mm
      [width=0.0875]7figures/teaser/body/frames_visualization/001014 
    
    35.6mm
      [width=0.0875]7figures/teaser/body_second/frames_input/001014 
    
    35.6mm
      [width=0.0875, poster=3]7figures/teaser/body_second/frames/001014 
    
    24.6mm
      [width=0.0875]7figures/teaser/face/frames_input/001014 
    
    24.6mm
      [width=0.0875, poster=6]7figures/teaser/face/frames/001014 
    
    13.7mm
      [width=0.0875]7figures/teaser/camera/frames_input/001014 
    
    13.7mm
      [width=0.0875, poster=6]7figures/teaser/camera/frames/001014 
    
    2.8mm
      [width=0.0875]7figures/teaser/handcrafted/frames_input/001014 
    
    2.8mm
      [width=0.0875, poster=8]7figures/teaser/handcrafted/frames/001014 
    
    We encode the motion of a reference video into a novel motion-text embedding using a frozen, pre-trained image-to-video diffusion model. This optimized motion-text embedding can then be applied to different starting images to generate videos with semantically similar motions. The general nature of our motion representation allows for successful motion transfer even when objects are not spatially aligned, across various domains, and for multiple objects. Additionally, our method supports multiple types of motions, including full-body, face, camera, and even hand-crafted motions. 

    1 Please refer to the supplementary material for corresponding videos for all figures of this paper. 
    A figure made up of two blocks. In the top block, the rough model architecture is shown. A motion-text embedding is extracted from a motion reference of a person doing jumping jacks, and this motion is then applied to three different input images and to an unconditional appearance (motion visualization). The generated videos all show a jumping jack motion even if the objects are not spatially aligned, from different domains and for multiple objects. In the bottom block, a number of motion input and generated output videos are shown for full-body reenactment, face reenactment, hand-crafted motions, and camera motions.
    



Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversion
    Romann M. Weber
    August 5, 2024
===============================================================================





§ INTRODUCTION
 

Since the advent of diffusion models, video generation and editing methods have significantly improved. However, controlling the motions generated by these models remains challenging. For instance, Stable Video Diffusion <cit.>, a state-of-the-art image-to-video diffusion model, offers little practical control over motion. The primary methods to modify motion involve altering the diffusion process's random seed or adjusting micro-conditioning inputs like frame rate, but these options are not easily interpretable[The authors demonstrate camera motion control by fine-tuning motion-specific LoRA <cit.> modules; however, we consider this an extension rather than a feature of the base model.].

Existing methods for controlling motion with sparse control signals like text <cit.>, boxes <cit.>, or trajectories <cit.> are limited to simple motions in most practical scenarios. On the other hand, dense motion trajectories <cit.> may leak the motion reference video's spatial structure, thus often failing in unaligned scenarios.

Many reference-based motion transfer methods fine-tune model components (e.g., using LoRA <cit.>) on one or several motion reference videos <cit.>. This often results in appearance leakage, where the model overfits to the reference video's appearance and fails to generalize to new target object appearances. Motion transfer techniques that use the inversion-then-generation framework <cit.> attempt to replicate the reference video's structure in the generated video. However, this approach can be problematic when there are significant differences between the locations and geometries of the reference and target objects, leading to misaligned semantic features.

Most of the one-shot reference-based methods produce videos with motions that are mostly spatially aligned with the motion reference video, i.e., they follow the layout as well as the subject scale and position of the reference video. We thus argue that many of these works <cit.> can be considered as an advanced form of appearance transfer rather than motion transfer. 
Lastly, most existing methods are based on text-to-video models and thus cannot preserve the exact appearance and background of a given target image.

We propose to transfer the semantic motion of a motion reference video to a given target image. Specifically, given motion reference video V_R and target image I_T, we want to generate a video V̂ such that M_sem(V̂) = M_sem(V_R), where M_sem is the semantic motion, and A(V̂) = A(I_T), where A is the appearance and spatial layout. Importantly, the generated motion should match the semantics of the motion reference video rather than the exact same spatial motions. For example, we want to be able to generate a video of a given subject doing jumping jacks on the left side of the video even if the subject in the motion reference video is in the center.  Fig. <ref> shows exemplary results of our method, including motion transfers to multiple (misaligned) objects.

We identify two key challenges: appearance leakage from the motion reference video and object misalignment. To tackle appearance leakage, we employ an image-to-video rather than a text-to-video (or an inflated text-to-image) model and do not fine-tune the model. To the best of our knowledge, we are the first to use an image-to-video model for general motion transfer. To address object misalignments between the motion reference video and the target image, we introduce a novel motion representation that eliminates the need for spatial alignment by not having a spatial dimension in the first place.

Our motion representation is based on our observation that image-to-video models extract the appearance predominantly from the (latent) image input, whereas the text/image embedding injected via cross-attention mostly controls the motion. We therefore propose to represent motion with several text/image embedding tokens, together referred to as motion-text embedding, that we optimize on a given motion reference video. Thereby, our inflated motion-text embedding enables us to preserve the timing of the motion video very precisely, which is crucial for applications such as visual dubbing. Our approach, named motion-textual inversion, is general in nature and works for various types of motions and objects. Perhaps surprising at first, it turns out that while words are not ideal for describing motions, their embeddings can describe highly complex motions exceptionally well, as shown in our experiments.

To summarize, our contributions are:

    
  * We introduce the semantic video motion transfer task in an image-to-video setting.
    
  * We observe that text/image embeddings of image-to-video diffusion models store and affect motion and leverage them as a general and compact motion representation.
    
  * We propose motion-textual inversion, a novel method that optimizes multiple text/image embedding tokens on a motion reference video and transfers the learned motion to target images.
    
  * We demonstrate superior performance over existing motion transfer approaches.




§ RELATED WORK
 



 §.§ Domain-Specific Reenactment
 

Reenactment has been a significant research area, but much of the focus has been on domain-specific approaches like face reenactment <cit.> and human full-body motion transfer <cit.>. While these methods perform well, their architectures and training data are tailored to specific domains, making it challenging to adapt them for use across multiple domains. Current general image animation methods <cit.> also still require domain-specific training, so they cannot directly solve our task.

Since our aim was to introduce a general method, our design choices differ from domain-specific models. For instance, many reenactment techniques rely on keypoints, which are derived either from pre-trained, domain-specific landmark detectors <cit.> or learned in an unsupervised manner <cit.>. Although the latter approach is more flexible, it still requires a separate training for each domain, which is cumbersome. Additionally, this approach presents challenges in determining keypoint placement for unseen domains during inference (e.g., inanimate objects), cross-domain transfers (e.g., human-to-animal), and misaligned objects. We therefore choose an implicit motion representation rather than an explicit one. 

Given the impressive cross-domain translation capabilities of diffusion models, as demonstrated in works like <cit.>, we utilize a diffusion-based video foundation model for our general task to take advantage of its extensive and general priors.



 §.§ Video Generation


Following the rise of text-to-image diffusion models <cit.>, video generation models have also greatly improved in quality in recent years. Many text-to-video methods start with a pre-trained text-to-image model and inflate it by adding and training temporal convolution and attention blocks after each corresponding spatial block <cit.>. Similarly, many image-to-video diffusion models use a pre-trained text-to-image <cit.> or text-to-video <cit.> model as a starting point. They then adapt the model to the image-to-video task by conditioning the model on the image, e.g., by adding <cit.> or concatenating <cit.> it to the noisy input. The text embedding input from the pre-trained model is either kept <cit.> or replaced with an image embedding input <cit.>. While training a custom video generation model provides the most freedom in terms of design choices, it is very expensive in terms of computation and data. Even fine-tuning video models requires substantial resources, so we decided to use a pre-trained diffusion model, Stable Video Diffusion <cit.>, and keep it frozen. Additionally, we aim for our method to be applicable to a wide range of motions and subjects. In contrast, approaches that involve training the model often focus on a single type of motion, such as human full-body motion <cit.>.



 §.§ Video Motion Editing with Explicit Motions




  §.§.§ Based on Sparse Control Signals


In theory, the motion of all video generation models that have a text input can simply be controlled by text <cit.>, but this approach struggles with complex motions in practice. For more precise spatial control, recent methods use bounding boxes, either with training <cit.> or without <cit.>, and trajectories <cit.>, but they rely on consistent spatial alignment for effective motion transfer. Similarly, keypoints are another option for describing motions <cit.>, but they suffer from the challenges outlined in Section <ref>. Additionally, some methods focus specifically on camera motions <cit.> or combine camera and bounding box motions <cit.>. However, all these approaches are either limited to simple motions or require significant effort to specify complex ones. For instance, a bounding box can specify an object's location (e.g., a person) but not the detailed motion within it (e.g., doing jumping jacks), and modeling complex motion with part-based boxes quickly becomes impractical.



  §.§.§ Based on Dense Control Signals


Dense control signals, such as motion vectors <cit.> and depth maps <cit.>, allow for a more precise motion specification. However, using them for general motion transfer is challenging because they also encode information about image and object structure. This can result in unnatural motions when there is a mismatch between the structures of the target image and the reference video as shown in <cit.>.



 §.§ Video Motion Editing with Implicit Motions


This subsection covers methods for implicitly representing and transferring motion from a reference video. Fine-tuning approaches store motion in model weights, whereas inversion-then-generation methods store motion in model features and attention maps. Some techniques combine both paradigms. When the layout of the subjects in the reference and generated videos match, a given transfer can be seen as either changing the appearance to match the target image or altering the motion to match the reference video. Our focus is on motion transfer where the layouts do not align, a less explored area in the literature, as discussed in Section <ref>.



  §.§.§ Fine-Tuning
 

Many fine-tuning methods are inspired by image customization techniques like DreamBooth <cit.> and LoRA <cit.>. Loosely speaking, the idea is to fine-tune the parts of the model responsible for motion but avoid training the parts responsible for appearance. Tune-A-Video <cit.> inflates a text-to-image model by adding spatio-temporal attention and only trains some parts of the attention layers. Similarly, <cit.> only fine-tunes parts of the model and further focuses the training more on earlier denoising steps to emphasize learning the general motion rather than fine appearance details. MotionDirector <cit.> proposes a dual-path LoRA architecture and an appearance-debiased temporal loss to disentangle appearance from motion. Similarly, DreamVideo <cit.>, MotionCrafter <cit.>, and Customize-A-Video <cit.> have separate branches for appearance and motion. VMC <cit.> adapts temporal attention layers using a motion distillation strategy with residual vectors between consecutive noisy latent frames as the motion reference. 

Fine-tuning a model carries the risk of appearance leakage, which is why many of the aforementioned methods focus on preventing it. We find that using an image-to-video model instead of a text-to-video model largely avoids these problems. LAMP <cit.> is the most similar method to ours in that sense, but they adapt a pre-trained text-to-image model to the image-to-video task and fine-tune it only briefly. In contrast, we employ a pre-trained, large-scale image-to-video model to leverage stronger priors for better generalization.




  §.§.§ Inversion-then-Generation
 

The inversion-then-generation framework, initially developed for image editing <cit.>, involves first inverting a reference video into “noise” using methods like DDIM <cit.> to enable reconstruction through backward diffusion. Thereby, features such as self-attention maps are extracted from the reference video and then injected into the diffusion process of the video being generated. These features either directly replace existing features <cit.> or are incorporated into a loss function <cit.>, ensuring the generated video has a similar structure. Numerous methods have been proposed within this framework for video appearance editing <cit.> and video motion editing <cit.>, mainly differing in their inversion techniques and feature choices.

The methods mentioned above face several inherent issues in motion transfer tasks. Most notably, they often assume or enforce that the features of the reference and target videos are identical, which leads to problems when generating videos with different geometries or spatial layouts. Some methods attempt to address this by collapsing the spatial dimension of features before using them in a loss <cit.>, but they still typically produce motions with similar directions in pixel space. This limits control and diversity and can produce less natural results. Furthermore, these approaches require tuning numerous hyperparameters (choice of feature, layers, time steps) and necessitate inverting the video, which is challenging for high guidance scales <cit.> and when using few time steps <cit.>.




  §.§.§ With Different Spatial Layout
 

To avoid being restricted to the layout of a single motion reference video, some methods use multiple motion videos <cit.>. However, our goal is to transfer motion with precise temporal alignment to the reference video. This would require multiple temporally-aligned videos, which are often impractical to obtain. Additionally, many motion editing methods with spatial variations <cit.> use text to define the subject's appearance instead of an image, resulting in videos that only roughly match the input image. The approach in <cit.> is most similar to ours as it keeps the model frozen and learns a motion embedding like we do, but it also suffers from the above limitation.



§ METHOD



[width=0.09, poster=3]7figures/i2v_ignores_text/white_horse_frames/001014

[width=0.09, poster=3]7figures/i2v_ignores_text/white_horse_but_pink_text_frames/001014




[width=0.09, poster=3]7figures/horse_clip_swap_1024_576/white_horse_frames/0114

[width=0.09, poster=3]7figures/horse_clip_swap_1024_576/white_horse_with_pink_horse_emb_frames/0114

[width=0.09, poster=3]7figures/horse_clip_swap_1024_576/pink_horse_with_white_horse_emb_frames/0114

[width=0.09, poster=3]7figures/horse_clip_swap_1024_576/pink_horse_frames/0114



We propose to transfer the semantic motion of a motion reference video to a given target image by motion-textual inversion. We thereby optimize a set of text/image embedding tokens, which we refer to as motion-text embedding, for the motion reference video using a pre-trained image-to-video diffusion model. 



 §.§ Preliminaries
 

Diffusion models <cit.> consist of two processes. In the forward process, Gaussian noise is iteratively added to a clean data sample x_0 until it is approximately pure noise. In the reverse process, starting with pure noise x_T, a learnable denoiser D_θ iteratively removes noise to obtain a sample that matches the original data distribution p_data. We follow the continuous-time framework <cit.>, where the denoiser is trained via denoising score matching: 

    𝔼_(x_0, c) ∼ p_data(x_0, c),(σ, n) ∼ p(σ, n)[λ_σ ||D_θ(x_0 + n; σ, c) - x_0||^2_2],

where x_0 is a clean data sample and c an arbitrary conditioning signal from the original data distribution p_data; p(σ, n) = p(σ)𝒩(n; 0, σ^2), where p(σ) is a probability distribution over noise levels σ, and n is noise; and λ_σ: ℝ_+ →ℝ_+ is a weighting function. The denoiser D_θ is parameterized as

    D_θ(x; σ) = c_skip(σ)x + c_out(σ)F_θ(c_in(σ)x; c_noise(σ)),

where F_θ is the neural network to be trained; c_skip(σ) modulates the skip connection; c_out and c_in scale the output and input magnitudes respectively; and c_noise maps noise level σ into a conditioning input for F_θ. For more details, please refer to <cit.>.

Latent diffusion models <cit.> operate in the latent space rather than in pixel space to reduce computation and thus enable higher resolutions. First, an encoder ℰ produces a compressed latent z = ℰ(x). Then, we perform the diffusion process over z. Lastly, a decoder 𝒟 reconstructs the latent features back into pixel space.[To maintain consistency in notation, we use x for the diagrams and method description, even though the diffusion process actually occurs in latent space.]

Stable Video Diffusion (SVD) <cit.> is a video latent diffusion model trained in three stages: 1. A text-to-image model <cit.> is trained or fine-tuned on (image, text) pairs. 2. The diffusion model is inflated by inserting temporal convolution and attention layers following <cit.> and then trained on (video, text) pairs. 3. The diffusion model is refined on a smaller subset of high-quality videos with exact model adaptations and inputs depending on the task (text-to-video, image-to-video, frame interpolation, multi-view generation). For image-to-video generation, the task is to produce a video given its starting frame. The starting frame is supplied to the model in two places: as a CLIP <cit.> image embedding via cross-attention (replacing the CLIP text embedding from the text-to-video pre-training) and as a latent repeated across frames and concatenated channel-wise to the video input. Additionally, the model is micro-conditioned on the frame rate, motion amount, and strength of the noise augmentation (applied to first frame latent).



 §.§ Motivation
 

Transferring the motion of a reference video to a given target poses two key challenges, which our design solves quite naturally.



  §.§.§ Challenge 1: Appearance Leakage
 

When fine-tuning a text-to-video model to learn the motion from a single reference video, there is a risk of overfitting to the reference video's appearance, which can prevent the model from producing the correct target appearance during inference. We demonstrate that using a frozen image-to-video model can preserve the target appearance without any of the special mechanisms from literature described in Section <ref>. 

By design, image-to-video models generate videos given a starting frame, so generated frames tend to preserve the appearance of the input image. We observe that image-to-video models primarily derive the appearance from the image (latent) input, even with an additional text input, as shown in Fig. <ref>. This is likely because the model can directly copy (latent) pixels from the first frame instead of hallucinating them from the sparse text input. This strong reliance on the image input reduces the chance of the reference video's appearance leaking through. To further minimize the risk of appearance leakage, we keep the model's weights frozen, so they cannot possibly store the reference video appearance. This also helps retain the rich video understanding and generalization capabilities of the pre-trained model.





  §.§.§ Challenge 2: Handling Object Misalignment.
 

Our goal is to generate videos where subjects perform the same semantic actions, even if they are in different spatial locations or orientations. Handling misaligned objects is especially important when using image-to-video models because the subject's position is determined by the input image, which typically does not match the position in the motion reference video. 

As discussed in Section <ref>, existing methods using the inversion-then-generation framework inject features from the motion reference video into the generated video, making it closely follow the reference structure. Arguably, these methods do not copy the motion at its origin but rather the per-frame structure that results from a motion (e.g., rough object positions). For the general, unaligned case, these features would first need to be aligned spatially to avoid injecting the structure in the wrong place. This alignment is challenging since the final positions in the generated video are unknown during the diffusion process as they depend on the motion.

We forgo the alignment problem by representing motions with text or image embedding tokens that do not have a spatial dimension in the first place. Our novel motion representation was motivated by the observation shown in Fig. <ref>. While SVD generated walking motions for an image of a real horse, it generated no object but mostly camera motion for an image of a pink toy horse[Image was generated using <cit.>.], perhaps because the model learned that toys do not move. Recall that SVD has the first frame as input in two places: as image latent and as CLIP <cit.> image embedding. When using the image latent of the real horse but the CLIP embedding of the toy horse, the horse in the generated video does not move. Inversely, the toy horse starts walking when using the CLIP embedding of the real horse, implying that the CLIP embedding affects the motion. We believe that these embeddings are not just affecting the motion but are actually the main origin of the motion. 

Our intuition for why the text/image embeddings determine the motion (which may be surprising at first) is as follows: Videos can be divided into appearance and motion. Appearance is tied to the spatial arrangement of pixels, making it easier to extract it from spatial inputs like image latents. Motion depends on how pixels change over time, requiring a more global, semantic understanding. Thus, it is more effective to modify motion using image embeddings, which contain more semantic information, have no spatial dimension, and are injected in multiple places of the model. Furthermore, SVD was initially trained as a text-to-video model, with CLIP text embeddings describing motions like “standing”, “walking”, or “running,” incentivizing the model to control motion through cross-attention inputs to effectively denoise training videos.



 §.§ Motion-Textual Inversion
 

While using embeddings from different images can alter the generated motion, it does not transfer the motion robustly. Moreover, selecting a specific frame to define a desired motion is difficult since motion is rarely captured by a single frame. To address this, we propose optimizing the embedding based on a given motion reference video, which bears some resemblance to textual inversion <cit.>. In analogy to textual inversion, we name our method motion-textual inversion[In our implementation, it is actually an image embedding, but we refer to it as “motion-textual inversion” since SVD's image and text embeddings share the same CLIP space, and other I2V methods use text embeddings instead. Also, it feels more intuitive to represent motions as text rather than an image.]. Note, however, that our method has a completely different goal: using embeddings to encode video motion rather than image appearance.

Fig. <ref> shows a high-level overview of our method. Given a single motion reference video x_0 containing F frames, we optimize the motion-text embedding m directly by minimizing the diffusion model loss from Equation <ref>, keeping the diffusion model frozen:

    m^* = marg min    𝔼_(x_0, c) ∼ p_data(x_0, c),(σ, n) ∼ p(σ, n)
        [λ_σ ||D_θ(x_0 + n; σ, m, c) - x_0||^2_2],

where c encompasses all remaining conditionings of SVD (e.g., first frame latent, time/noise step, and micro-conditionings). All other symbols are defined in Equations <ref> and <ref>.



 §.§ Motion-Text Embedding and Cross-Attention Inflation
 



Cross-attention allows the model to dynamically attend to different tokens (∼ words in text-to-image and text-to-video) depending on the current features or context. It is computed as follows:

    Attention(Q, K, V) = M V = softmax(QK^T/√(d_a)) V, 
     
        Q = φ_i(z_t) W_Q,i,   K = m W_K,i,   V = m W_V,i,

where Q, K, V are the queries, keys, and values respectively; M is the attention map; d_a is the dimension used in the attention operation; φ_i(z_t) is an intermediate representation of the level i features with C_i channels; m is the motion-text embedding (or text/image embedding e in case of baseline SVD) with embedding dimension d; and W_Q,i∈ℝ^C_i × d_a, W_K,i∈ℝ^d × d_a, and W_V,i∈ℝ^d × d_a are learned weight matrices for queries, keys, and values respectively.

The image embedding of SVD only has one token. This results in a degenerate case for the cross-attention operation where all entries of the attention map M are 1, as shown in Fig. <ref>. The model thus attends 100% to that single token and applies its value to all spatial and temporal locations. 



  §.§.§ Multiple Tokens


Using the same value V for all locations limits the extent to which the embedding can change the motion. We therefore propose to have N tokens (instead of 1) to attend to, recovering the scenario from the text-to-image or text-to-video pre-training. This allows the model to dynamically attend to different tokens depending on the features, e.g., using different values for the background and foreground as seen in the spatial cross-attention maps in Fig. <ref>.


[width=0.08, poster=2]7figures/motion_vis/cond_000000_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_000000_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_000000_frames_2/001014

[width=0.08, poster=2]7figures/motion_vis/cond_000200_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_000200_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_000200_frames_2/001014

[width=0.08, poster=2]7figures/motion_vis/cond_000500_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_000500_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_000500_frames_2/001014

[width=0.08, poster=2]7figures/motion_vis/cond_001000_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_001000_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_001000_frames_2/001014

[width=0.08, poster=2]7figures/motion_vis/cond_002000_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_002000_frames/001014

[width=0.08, poster=2]7figures/motion_vis/uncond_002000_frames_2/001014





  §.§.§ Different Tokens per Frame


For the spatial cross-attention, the default SVD broadcasts the image embedding across all frames. We argue that we can obtain a higher temporal granularity of the motion by using a different set of tokens for each frame, i.e., F × N tokens[Note that we always use the same F frames of the motion reference video when optimizing the motion-text embedding.]. Having different sets of tokens leads to different sets of keys and values for each frame. Different keys allow the model to attend to different spatial locations for different frames, e.g., attending to the arm of a person in one frame but to the leg in another frame. Different values allow the model to apply different changes to the features for different frames, e.g., applying a vector to inform the model to shift pixels in one direction in one frame but another direction in another frame. This is visualized in Fig. <ref>, where the spatial cross-attention maps differ greatly between frames because they use different tokens. 

For the temporal cross-attention, the default SVD broadcasts the image embedding across all spatial locations. Inflating this analogously to the spatial case would mean inflating the embedding across the spatial dimensions. Learning different tokens for different spatial locations is nontrivial (spatial dimensions change depending on the input resolution and the level i) and would likely cause issues with the alignment (see Section <ref>). Furthermore, in our empirical experiments, the temporal cross-attention seemed to have a smaller effect on the generated motion than the spatial cross-attention. Therefore, we decided to keep the number of tokens of our temporal motion-text embedding at N but learn them separately from the F × N tokens of the spatial motion-text embedding. Our combined motion-text embedding thus has (F+1) × N tokens per motion reference video. Section <ref> describes the tensor operations and dimensions of our motion-text embedding and cross-attention inflation in detail.

To give an intuitive analogy for our motion-text embedding inflation, think of building a house. Instead of using a single tool for every part of the house, it is more efficient to have N different tools depending on the spatial location on a given floor – like a hammer for the floor and a drill for the wall. Moreover, each of the F floors of the house might need a different set of tools. For example, the roof requires different tools compared to the walls. Similarly, in our approach, we use multiple tokens to handle different aspects of the motion. 





§ EXPERIMENTS




 §.§ Implementation Details


We base our implementation on the 14-frame version of Stable Video Diffusion (SVD) <cit.>, but our method could in theory be applied to other image-to-video methods that have a text/image embedding input. Per default, we use N=5 different tokens for each of the F=14 frames, so a total of (14+1) × 5 = 75 tokens for the motion-text embedding. Please refer to Section <ref> for the remaining hyperparameters and implementation details.

Our data consists of motion reference videos and target images from internal data sets as well as target images generated with Stable Diffusion XL <cit.>. We only use one reference video for the optimization per motion and can apply this motion to various target images.



 §.§ Motion-Text Embedding Analysis


SVD was pre-trained as a text-to-video model and dropped the image (latent) input for some percentage of training iterations for classifier-free guidance <cit.>. We find that SVD can produce somewhat reasonable videos with the image (latent) input zeroed out and only the CLIP <cit.> image embedding as input, especially if we increase the classifier-free guidance scale. We can use this to visualize our learned motion-text embedding with an unconditional appearance[Note that the visualization is not always easily interpretable, depending on the motion, the optimization iteration, and the seed.]. Fig. <ref> shows motion visualizations of our motion-text embedding for a “jumping jacks” motion after different numbers of optimization iterations and the generated videos for a given target image side-by-side. Starting around iteration 500, a person doing a “jumping jacks” motion can be seen in the visualizations. Beyond 1000 iterations, the motion visualizations become more abstract, but the generated motions in the conditional case remain of high quality. Notably, the appearance and position of the people do not match those of the motion reference video (from Fig. <ref>). Furthermore, the position of the people is different in the conditional and unconditional videos, but all videos have a similar semantic motion. This demonstrates that our motion-text embedding neither encodes the appearance nor the exact spatial positioning of the objects extensively, likely for reasons described in Section <ref>.



 §.§ Qualitative Evaluation



[width=0.08, poster=0]7figures/qual_eval/fullbody_jumping_jacks/frames_input/001014

[width=0.08, poster=0]7figures/qual_eval/face_nodding/frames_input/001014

[width=0.08, poster=0]7figures/qual_eval/camera_medium/frames_input/001014

[width=0.08, poster=4]7figures/qual_eval/fullbody_jumping_jacks/frames_svd/001014

[width=0.08, poster=3]7figures/qual_eval/face_nodding/frames_svd/001014

[width=0.08, poster=6]7figures/qual_eval/camera_medium/frames_svd/001014

[width=0.045, poster=4]7figures/qual_eval/fullbody_jumping_jacks/frames_vc/001014

[width=0.045, poster=3]7figures/qual_eval/face_nodding/frames_vc/001014

[width=0.045, poster=6]7figures/qual_eval/camera_medium/frames_vc/001014

[width=0.045, poster=4]7figures/qual_eval/fullbody_jumping_jacks/frames_md/001014

[width=0.045, poster=3]7figures/qual_eval/face_nodding/frames_md/001014

[width=0.045, poster=6]7figures/qual_eval/camera_medium/frames_md/001014

[width=0.08, poster=4]7figures/qual_eval/fullbody_jumping_jacks/frames_ours/001014

[width=0.08, poster=3]7figures/qual_eval/face_nodding/frames_ours/001014

[width=0.08, poster=6]7figures/qual_eval/camera_medium/frames_ours/001014



As our baseline, we use SVD <cit.> without any adaptations. Since it does not have a motion control input, we do not expect it to follow the correct motion given only the starting image in most cases. However, we show it anyway to showcase the quality and motions of typical SVD outputs. To our knowledge, our method is the first method to address the general motion transfer task in the image-to-video setting. Therefore, there are few methods to choose from that permit a direct and fair comparison. We considered the two most similar classes of methodology and compared our method with a representative of each class. Methods within these classes tend to have certain drawbacks in common:

    
  * Image-to-video methods with explicit motion representation: Without proper alignment, these methods transfer spatial but not semantic motion and may leak the reference video's structure. Since it is unclear how to automatically extract sparse motion inputs, we compare to VideoComposer <cit.>, which uses dense motion vectors.
    
  * Text-to-video methods with implicit motion representation: These methods do not directly take a target image input, compromising the preservation of the target's appearance and layout.  We compare to MotionDirector <cit.> since it learns the target image's appearance, which we expect to perform better than specifying the appearance with text alone.


We use the official implementations. For the models requiring a text input, we use BLIP <cit.> to generate image captions and SpaceTimeGPT <cit.> to generate video captions.

Fig. <ref> shows motion transfer results for three videos. The SVD baseline produces high-quality videos for the robot and waterfall, but their motions do not match the reference videos. For the face video, SVD introduces artifacts and changes the person's identity with head movement. In our experience, SVD often generates static objects with moving cameras, which users might find unexpected or frustrating. We suggest that motion transfer methods, like ours, can help generate more natural and diverse motions. VideoComposer fails to transfer motions successfully, alters appearances, and sometimes introduces strong artifacts. This issue is inherent to its class and arises from the dense motion inputs, which implicitly encode the reference video's structure that may not align with the input image. Since proper alignment is difficult to achieve in our general (and potentially cross-domain) setting, we find it easier to use an implicit motion representation without a spatial dimension, like ours, for general semantic video motion transfer. MotionDirector correctly transfers motion in the head-nodding example but fails to produce the correct motion for the other two. Furthermore, the appearance does not match the target image exactly. For example, the robot is rotated, the person's identity changes, and the waterfall also looks differently. This limitation stems from text-to-video methods not having a direct image input and relying on learning the appearance (or predicting it from text). Thus, we believe that image-to-video models have inherent advantages over text-to-video models when animating a given image. Our method uniquely preserves the input image's appearance and layout while successfully transferring the semantic motion of the video.



 §.§ Results



[width=0.09, poster=0]7figures/results/fullbody_drunk/frames_input/001014

[width=0.09, poster=0]7figures/results/fullbody_animal/frames_input/001014

[width=0.09, poster=3]7figures/results/fullbody_drunk/frames/001014

[width=0.09, poster=3]7figures/results/fullbody_animal/frames/001014

[width=0.09, poster=0]7figures/results/face_opening_mouth/frames_input/001014

[width=0.09, poster=0]7figures/results/face_tilting_up/frames_input/001014

[width=0.09, poster=3]7figures/results/face_opening_mouth/frames/001014

[width=0.09, poster=2]7figures/results/face_tilting_up/frames/001014

[width=0.09, poster=0]7figures/results/camera_birdseye_rotate/frames_input/001014

[width=0.09, poster=0]7figures/results/camera_object_following/frames_input/001014

[width=0.09, poster=3]7figures/results/camera_birdseye_rotate/frames/001014

[width=0.09, poster=3]7figures/results/camera_object_following/frames/001014

[width=0.09, poster=0]7figures/results/handcrafted_robot_jumping/frames_input/001014

[width=0.09, poster=0]7figures/results/handcrafted_passing_front/frames_input/001014

[width=0.09, poster=4]7figures/results/handcrafted_robot_jumping/frames/001014

[width=0.09, poster=5]7figures/results/handcrafted_passing_front/frames/001014



Our motion representation is highly versatile, allowing our method to work with nearly any type of object and motion, as demonstrated in Fig. <ref> and Fig. <ref>. Notably, we do not require a spatial alignment, so our motion-text embedding can be applied to images where objects have different positions and orientations from those in the reference video. For instance, in the sixth row on the right side of Fig. <ref>, the camera tracks the moving camper van in a similar way to how it tracks the car in the fifth row, despite their misalignment. Furthermore, our method applies the motion to all semantically reasonable objects simultaneously “for free”. It can even handle simple hand-crafted motions, enabling artists to sketch a motion (e.g., drawing stick figures) and apply it to complex scenes. For more results, including joint subject and camera motion as well as extreme cross-domain transfers and applying the same motion to multiple target images, please refer to Section <ref>. 



 §.§ Ablation Study



[width=0.07, poster=0]7figures/ablation_study/frames_input/001014

[width=0.07, poster=4]7figures/ablation_study/frames_same_clip_1/001014

[width=0.07, poster=4]7figures/ablation_study/frames_same_clip_15/001014

[width=0.07, poster=4]7figures/ablation_study/frames_clip_1/001014

[width=0.07, poster=4]7figures/ablation_study/frames_default/001014

[width=0.07, poster=4]7figures/ablation_study/frames_clip_15/001014




[width=0.07, poster=0]7figures/failure_cases/quality_svd/frames_input/001014

[width=0.07, poster=3]7figures/failure_cases/quality_svd/frames/001014

[width=0.07, poster=0]7figures/failure_cases/structure_leak/frames_input/001014

[width=0.07, poster=4]7figures/failure_cases/structure_leak/frames/001014

[width=0.07, poster=0]7figures/failure_cases/partial_motion_transfer/frames_input/001014

[width=0.07, poster=4]7figures/failure_cases/partial_motion_transfer/frames/001014

Our proposed motion-text embedding inflation is crucial for high-quality motion transfers. Fig. <ref> illustrates different settings for the motion-text embedding size. With only one token, only a small portion of the motion is transferred. Increasing the number of tokens but sharing them across all frames offers some improvement. However, the key factor is having different tokens per frame. Both rows 2 and 3 use 15 tokens, but the version with different tokens per frame performs significantly better. This is logical since the motion-text embedding can adapt to each frame, which is particularly beneficial for complex motions. Increasing the number of tokens per frame further improves performance slightly, but it eventually saturates, so we default to using N=5.



 §.§ Limitations


Fig. <ref> shows typical failure cases of our method. Since we do not fine-tune the model, our method inherits the priors and quality of our pre-trained image-to-video model. We observed that the SVD baseline often struggles with object motions, as can be seen in the head example in Fig. <ref>, where the appearance changes throughout the video. Our method's results have similar issues: in the first example of Fig. <ref>, the identity of the target person changes when he moves his head to the side. We believe our motion-text embedding does not exacerbate these issues or temporal inconsistencies, as it primarily instructs the model on the desired motion without altering the rest of the model. Often, it seems that the model attempts to produce the desired motion, but its priors are insufficient to generate a satisfactory result. SVD also does not seem to be able to handle some combinations of motions and given input images, likely because they fall outside of the range of the training data set. When the domain gap between motion reference video and target image is too large, our method may leak the structure of the motion reference video into the generated video. In the second example of Fig. <ref>, when applying a laid-back walking style to a kangaroo, the kangaroo starts walking, but its feet and overall structure become more human-like. Lastly, we found that some motions are not transferred or to a smaller extent. This is especially visible if a video has multiple motions, where the more fine-grained motion is sometimes not transferred. In the third example of Fig. <ref>, the person pretends to squat down and type on a keyboard. The dinosaurs in the generated video do squat down, but their hands do not move. We hypothesize that fine-grained motions are also a general limitation of SVD. Overall, we expect better results of our method as image-to-video models improve.

We believe our approach is more accessible than approaches that require substantial training or fine-tuning. Nevertheless, our approach does require an optimization procedure that takes around one hour on an NVIDIA Tesla A100 (80 GB) GPU per motion. We encourage future work in reducing this per-motion optimization time as well as trying to learn a model to directly predict motion-text embeddings from motion reference videos. 






§ CONCLUSION


We introduce the general task of transferring the semantic motion of a reference video to any target image. We observe and exploit inherent advantages of image-to-video over text-to-video models for this task and find that text/image embedding tokens are well-suited as a motion representation. Specifically, our method, motion-textual inversion, optimizes an inflated version of the text/image embedding for a given motion reference video. Due to its general nature, this motion can then be applied to a wide number of objects and domains. Our method thus enables completely novel applications and takes a significant step towards being able to reenact anything.



§ ACKNOWLEDGEMENTS


We would like to thank Michael Bernasconi, Dominik Borer, Jakob Buhmann, and Daniela Kansy for providing motion videos as well as all participants featured in our internal datasets. We also want to give special thanks to Michael Bernasconi, Vukasin Bozic, Karlis Briedis, Pascal Chang, Guilherme Haetinger, Christopher Otto, Lucas Relic, Seyedmorteza Sadat, and Agon Serifi for their valuable and insightful discussions throughout the project.



ACM-Reference-Format





§ BROADER IMPACT AND ETHICS


To the best of our knowledge, our method is the first that can reenact a wide array of objects and motions given a target image and motion reference video without training domain-specific models. We believe this represents a significant advancement in controllable video generation, as our approach can address multiple existing domain-specific scenarios within a single framework and even facilitate entirely new applications. That said, we acknowledge the potential for misuse of reenactment methods like ours, such as creating realistic deepfakes or videos depicting individuals or objects performing specified, potentially inappropriate actions. We strongly condemn such misuse and advocate for implementing safety mechanisms and procedures in real-world applications. Additionally, we support ongoing research into detecting fake videos to mitigate these risks.




§ IMPLEMENTATION DETAILS
 



 §.§ High-Level Overview of the Implementation


To aid in reproducibility, we list the main steps of our method's implementation below:


    
  * [Only once] Take pre-trained Stable Video Diffusion (SVD) <cit.> and adapt code to inflate motion-text embedding and cross-attention. See high-level description in Section <ref> and details in Section <ref>.
    
  * Initialize motion-text embedding of shape (F + 1) x N x d. See Section <ref>.
    
  * Repeat until convergence:
    
        
  * Load same F frames of reference video in data loader for each iteration.
        
  * Augment data. See Section <ref>.
        
  * Input noisy version of frames, motion-text embedding, and other inputs into SVD.
        
  * Apply loss from Equation <ref> to update motion-text embedding.
    
    
  * Save motion-text embedding.
    
  * For all target images:
    
        
  * Input learned motion-text embedding along with new target image to inflated SVD during inference to generate video with motion from reference video.
    




 §.§ Hyperparameters
 

Our implementation builds up on the diffusers implementation <cit.> of Stable Video Diffusion (SVD) <cit.>. We use the default parameters of the 14-frame version of SVD (e.g., micro-conditionings) unless specified otherwise. Like SVD, we generally employ a classifier-free guidance <cit.> scale that increases linearly from 1 to 3 across the frame axis. For the motion visualization (unconditional image input), however, we use a higher scale, i.e., increasing linearly from 1 to 10, to improve the visibility of the objects. We initialize the F=14 sets of N=5 tokens for the spatial cross-attention with the CLIP image embedding token of each corresponding frame and the N=5 tokens for the temporal cross-attention with the mean of the CLIP image embedding tokens across all frames. We additionally add Gaussian noise 𝒩(0, 0.1) to the combined motion-text embedding during initialization. In our experience, the initialization does not affect the results significantly, so other initializations are equally reasonable. During optimization, we always pick the same F frames of a given video and apply the same spatial and color augmentations to all frames[For horizontal camera motions, we turn of horizontal flipping]. Since most of the video motion is determined in noisy diffusion steps, we shift the noise schedule towards higher noise values (from P_mean=1.0, P_std=1.6 to P_mean=2.8, P_std=1.6 where log σ∼𝒩(P_mean, P_std^2) to speed up the optimization. We use Adam <cit.> with a learning rate of 1e-2 for 1000 iterations with a batch size of 1. 





 §.§ Hardware Requirements and Runtime


The optimization for a motion reference video with a resolution of 1024 × 576 takes around 45 GB of GPU memory and around one hour on an NVIDIA Tesla A100 (80 GB) GPU. The inference takes less than one minute per video. Note that our implementation is not optimized extensively for memory or runtime.



 §.§ Motion-Text Embedding and Cross-Attention Inflation
 



This section provides more implementation details for the motion-text embedding and cross-attention inflation described in Section <ref>. Fig. <ref> shows the spatial and temporal cross-attention layers of the default Stable Video Diffusion (SVD) <cit.> and our inflated version along with their tensor dimensions.

The image embedding of the default SVD consists of a single token and has dimensions B × 1 × d, where B is the batch size (in our implementation typically 1 when optimizing the motion-text embedding and 2 during inference because of classifier-free guidance) and d is the CLIP <cit.> embedding dimension. For spatial cross-attention, the image embedding is broadcast to dimensions (B*F) × 1 × d, i.e., the same token is used for all F frames. This results in an attention map M of dimensions (B*F) × (H_i*W_i) × 1 where H_i and W_i are the spatial heights and widths respectively, and C_i is the number of channels of level i of the diffusion model. Notably, due to the softmax operation and the last dimension being 1, every value of the attention map is 1. This means that each spatial location attends 100% to the single token. Similarly, for temporal cross-attention, the image embedding is broadcast from dimensions of B × 1 × d to dimensions (B*H_i*W_i) × 1 × d, eventually leading to an attention map M of dimensions (B*H_i*W_i) × F × 1  where every value is 1. Having only one token thus leads to a degenerate case of the cross-attention where Attention(Q, K, V) = V (broadcasted) and many of the components (e.g., queries and keys) have no effect on the result. 



  §.§.§ Multiple Tokens


To avoid the above degenerate case and instead be able to dynamically attend to different tokens, we extend the token dimension from 1 to N where N is a hyperparameter. For spatial cross-attention, this results in an attention map M of dimensions (B*F) × (H_i*W_i) × N where, in general, each spatial location has different values ≠ 1 for the N different tokens. Similarly, the temporal cross-attention map M has dimensions (B*H_i*W_i) × F × N with values ≠ 1. Since SVD was pre-trained using multiple text embedding tokens as input, the code can already handle multiple tokens, so mainly the initialization of the motion-text embedding as well as some input dimensions have to be adapted slightly. 



  §.§.§ Different Tokens per Frame


As explained in Section <ref>, we propose to learn different sets of tokens per frame for the spatial cross-attention to obtain a higher temporal granularity of the motion. The default SVD implementation broadcasts the embedding from dimensions B × N × d across all frames to (B*F) × N × d (where N=1 originally). We instead learn a larger spatial motion-text embedding of dimensions B × F × N × d and reshape it to (B*F) × N × d. We keep the dimensions of the temporal motion-text embedding at B × N × d and learn it separately. Therefore, the dimensions of the combined spatial and temporal motion-text embedding is B × (F+1) × N × d. 



§ ADDITIONAL RESULTS
 

Fig. <ref> shows that our method does not only apply the rough motion category but also its style, even in cases where the domains differ vastly, e.g., transferring the motion of a horse to a cereal box. Furthermore, these examples demonstrate that our method can transfer joint subject and camera motion. Fig. <ref> shows that our method generates the same semantic rather than spatial motion by applying the same learned motion to a flipped target image. Fig. <ref> shows additional results of our method, where we apply the same motion to different target images.  


[width=0.09, poster=0]7figures/style/trot/frames_input/001014

[width=0.09, poster=0]7figures/style/canter/frames_input/001014

[width=0.09, poster=3]7figures/style/trot/frames/001014

[width=0.09, poster=3]7figures/style/canter/frames/001014

[width=0.09, poster=3]7figures/style/trot/frames_2/001014

[width=0.09, poster=3]7figures/style/canter/frames_2/001014

[width=0.09, poster=3]7figures/style/trot/frames_3/001014

[width=0.09, poster=3]7figures/style/canter/frames_3/001014

[width=0.09, poster=3]7figures/style/trot/frames_4/001014

[width=0.09, poster=3]7figures/style/canter/frames_4/001014




[width=0.09, poster=0]7figures/alignment/frames_input/001014

[width=0.09, poster=4]7figures/alignment/frames/001014

[width=0.09, poster=4]7figures/alignment/frames_2/001014




[width=0.09, poster=0]7figures/results_extra/fullbody_jumping_jacks/frames_input/001014

[width=0.09, poster=0]7figures/results_extra/fullbody_animal_chewing/frames_input/001014

[width=0.09, poster=4]7figures/results_extra/fullbody_jumping_jacks/frames/001014

[width=0.09, poster=4]7figures/results_extra/fullbody_animal_chewing/frames/001014

[width=0.09, poster=4]7figures/results_extra/fullbody_jumping_jacks/frames_2/001014

[width=0.09, poster=4]7figures/results_extra/fullbody_animal_chewing/frames_2/001014

[width=0.09, poster=0]7figures/results_extra/face_nodding/frames_input/001014

[width=0.09, poster=0]7figures/results_extra/face_tilting_down/frames_input/001014

[width=0.09, poster=3]7figures/results_extra/face_nodding/frames/001014

[width=0.09, poster=4]7figures/results_extra/face_tilting_down/frames/001014

[width=0.09, poster=3]7figures/results_extra/face_nodding/frames_2/001014

[width=0.09, poster=4]7figures/results_extra/face_tilting_down/frames_2/001014

[width=0.09, poster=0]7figures/results_extra/camera_medium/frames_input/001014

[width=0.09, poster=0]7figures/results_extra/camera_left/frames_input/001014

[width=0.09, poster=4]7figures/results_extra/camera_medium/frames/001014

[width=0.09, poster=4]7figures/results_extra/camera_left/frames/001014

[width=0.09, poster=4]7figures/results_extra/camera_medium/frames_2/001014

[width=0.09, poster=4]7figures/results_extra/camera_left/frames_2/001014

[width=0.09, poster=0]7figures/results_extra/handcrafted_robot_jumping/frames_input/001014

[width=0.09, poster=0]7figures/results_extra/handcrafted_zooming_rect/frames_input/001014

[width=0.09, poster=4]7figures/results_extra/handcrafted_robot_jumping/frames/001014

[width=0.09, poster=4]7figures/results_extra/handcrafted_zooming_rect/frames/001014

[width=0.09, poster=4]7figures/results_extra/handcrafted_robot_jumping/frames_2/001014

[width=0.09, poster=4]7figures/results_extra/handcrafted_zooming_rect/frames_2/001014








 Paper Title: Unlocking Fair Use in the Generative AI Supply Chain: A Systematized Literature Review.



[
Unlocking Fair Use in the Generative AI Supply Chain: A Systematized Literature Review]

















Amruta MahuliMPI-SP
Asia BiegaMPI-SP












MPI-SP Max Planck Institute for Security and Privacy, Bochum, Germany



Amruta Mahuliamruta.mahuli@mpi-sp.org 
Asia Biegaasia.biega@mpi-sp.org

















  

§ ABSTRACT 
Through a systematization of generative AI (GenAI) stakeholder goals and expectations, this work seeks to uncover what value different stakeholders see in their contributions to the GenAI supply line. This valuation enables us to understand whether fair use advocated by GenAI companies to train model progresses the copyright law objective of promoting science and arts. While assessing the validity and efficacy of the fair use argument, we uncover research gaps and potential avenues for future works for researchers and policymakers to address. 




§ INTRODUCTION

One of the demands made and accepted on account of the Writers’ Guild of America (WGA) protests[Art. 72 of Memorandum of Agreement for the 2023 WGA Theatrical and Television Basic Agreement: <https://www.wga.org/uploadedfiles/contracts/2023_mba_moa.pdf>] included disregarding generative artificial intelligence (GenAI) outputs as source material while determining writing credits and compensation. This demand highlighting the importance of credit and compensation for creative content creators was also voiced by Mr. Ashley Irwin during his congressional testimony[House holds hearing to examine the intersection of generative AI and copyright law — 05/17/23. Retrieved June 3, 2024 from <https://www.youtube.com/watch?v=isTgXmzoaXc>]. He reasoned that Art 1 section 8(8) of the US Constitution[U.S. Constitution. Art. I, § 8(8): <https://constitution.congress.gov/browse/essay/artI-S8-C8-1/ALDE_00013060/>] granted rights to authors and inventors to promote Science and arts, thus calling for policies that would regulate AI companies to generate their model using the ‘3Cs: consent, credit and compensation’ for creative workers (creatives). 

On the other hand, GenAI companies like OpenAI[OpenAI. 2019. Comment by OpenAI Regarding Intellectual Property Protection for Artificial Intelligence Innovation. Department of Commerce, United States Patent and Trademark Office. Retrieved December 22, 2023 from <https://www.uspto.gov/sites/default/files/documents/OpenAI_RFC-84-FR-58141.pdf>] are advocating for training models with copyrighted material as fair use based on how ‘transformative’ GenAI outputs will be, relying on Campbell v. Acuff-Rose Music[Campbell v. Acuff-Rose Music, 510 U.S. 569 (1994)] where the Supreme Court averred that “...the goal of copyright, to promote science and the arts, is generally furthered by the creation of transformative works.” 

Both arguments stem from the goal and justification for intellectual property law: to promote science and arts <cit.>. Ideally, these justifications form two sides of the same coin. However, in this case, and many lawsuits filed by creatives, they stand on opposing ends[Andersen et al v. Stability AI Ltd. et al, Docket No. 3:23-cv-00201 (N.D. Cal. Jan 13, 2023):  <https://docs.justia.com/cases/federal/district-courts/california/candce/3:2023cv00201/407208/67>][Main Sequence, Ltd. et al v. Dudesy, LLC et al, Docket No. 2:2024-cv-00711(Cal. Jan 25, 2024):  <https://s3.documentcloud.org/documents/24377081/carlin-lawsuit.pdf>][Tatjana Paterno and Layna Deneen. 2024. AI Threats Emerge In Music Publishers’ Battle With Big Tech. Hollywood Reporter. Retrieved June 3, 2024 from <https://www.hollywoodreporter.com/business/business-news/ai-threats-music-publishers-big-tech-1235767692/>][Blake Brittain and Blake Brittain. 2024. Microsoft, OpenAI hit with new lawsuit by authors over AI training. Reuters. Retrieved June 3, 2024, from <https://www.reuters.com/legal/microsoft-openai-hit-with-new-lawsuit-by-authors-over-ai-training-2024-01-05/>][Winston Cho. 2024. Scarlett Johansson, OpenAI Voice: Lawsuit for Actors Threatened. Hollywood Reporter. Retrieved June 3, 2024 from <https://www.hollywoodreporter.com/business/business-news/scarlett-johansson-ai-legal-threat-1235905899/>]. On the one hand, the creatives’ argument grounds itself in Lockean rhetoric of one's moral right to reap the benefits of one's labor. Conversely, OpenAI employs utilitarian rhetoric that views copyright as an incentivization system to enhance public welfare and advancement<cit.>. 

The challenge for the judiciary and the policymakers lies in how they ideologically interpret these copyright law arguments, given that they both hold ground. In addition to this challenge of interpreting cogent arguments, economically, in 2021, the core copyright industries in the US added a value of $1.810 trillion, which was 7.76% of the U.S. GDP <cit.>. Additionally, employment in the total copyright industries was more than 9.6 million creative workers, representing almost 5% of the total US employment <cit.>, making balancing this thin line of authorship rights and advancing cultural and scientific progress in society even more difficult. 

Therefore, this systematization of existing user studies draws from the conceptual framework of value similarity to corroborate the objectives of copyright law and the fair use doctrine for training GenAI models with the attributes valued by the stakeholders in their contributions to the GenAI supply chain. This paper begins by providing an overview of US copyright law and the philosophical groundings of copyright law. This section also maps the GenAI supply chain with the relevant stakeholders across every stage of the GenAI supply chain. Section 3 describes the method used in this paper to enable the analysis in Section 4. Section 5 discusses the results of the study. With the hope of uncovering research gaps and new future research directions, Section 6 concludes by reviewing the findings and proposing immediate computational challenges for better protection and implementation of the existing intellectual property frameworks, particularly in the context of GenAI.






§ METHODS



 §.§ Data Collection

This systematic literature review follows the PRISM statement while synthesizing existing Human-Computer Interaction (HCI) literature in English within the last 10 years since 2014 from 01/01/2014 to 04/30/2024 <cit.>.

We began by mapping stakeholders to different stages of the ML supply chain by relying on the supply chain conceptualized by Lee et al. <cit.> as a structuring framework. Given the complexity and possibilities of involved actors, Lee et al. <cit.>, for the sake of structure and clarity, our mapping of the stakeholders is approximate and has been summarized in Table 1.
 

We experimented with different keyword combinations based on our review goals and research questions to identify search query combinations to retrieve relevant existing literature. Given that our research questions seek to gauge the elements of value in the stakeholders’ contributions and ownership perceptions, we generalize by using different synonyms of the identified search strings where the star symbol (*) represents a wild card character, for example, recruit* could mean recruited or recruitment or recruit. We yielded 5895 papers using the following 4 sets of search strings for our search:

    
  * [Search across the paper for: "writer*" OR “knowledge workers” OR author* OR storyteller OR novelist* OR poet* OR co-author* OR “coauthor*” OR “cocreator” OR “co-creator” OR “creator” OR “artist” OR “dataset curat*” OR “database curat*” OR “dataset creat*” OR “database creat*” OR “ML practitioner*” OR “developer” OR “software developer” OR “ML developer” OR “computer scientist*” OR “data scientist*” OR “UX practitioner*” OR “UX designer*” OR “user*”] AND [Search within the abstract for: "user studies" OR "survey" OR "interview*" OR "recruit*"] AND [Search across the paper for: "goals" OR "expectation*" OR "own*" OR "incentiv*" OR "motiv*"]
    
  * [[Search across the paper for: "designer"] AND [Search with the abstract for: "user studies" OR "survey" OR "interview*"] OR "recruit*"] AND [Search across the paper for: "goals" OR "expectation*" OR "own*" OR "incentiv*" OR "motiv*"] AND [Search across the paper for: "creativ*" OR "ux"]]
    
  * [[Search across the paper for: “GenAI" OR "generative artificial intelligence" OR "generative AI" OR "generative output"] AND [Search with the abstract for: "user studies" OR "survey" OR  "interview*"] OR "recruit*"] AND [Search across the paper for: "intellectual property" OR "copyright" OR  "patent"] OR "law"]]
    
  * [[Search across the paper for: "stakeholder*"] AND [Search within abstract for: "user studies" OR "survey" OR "interview*" OR "recruit*"] AND [Search across the paper for: "goals" OR "expectation*" OR "own*" OR "incentiv*" OR "motiv*"] AND [Search across the paper for: "machine learning" OR "ML" OR "AI"] OR "artificial intelligence"]]


To be eligible for our study, the publications met the following criteria:

    
  * Articles must be peer-reviewed full papers that were published over the last 10 years, limiting the results to the first 100 searches. Given the dynamic and fast-paced innovation in the machine learning and artificial intelligence domain, we believe that reviewing peer-reviewed full papers from the last 10 years will allow us to synthesize an understanding of the stakeholder perspectives.
    
  * The selected articles must be full research papers that used qualitative user interviews as a research method for their research. To gauge stakeholder perspective on subjective issues such as valuation and expectations, we believe that user interviews and survey-based studies and papers have the potential to act as surrogates of authority to gain information from stakeholders. Therefore, we did not include full papers based on quantitative methods, non-user studies papers, posters, doctoral theses, books, book chapters, panel discussion reports, abstracts, or work-in-progress papers.
    
  * The papers must relate to user perspectives on the thought process or workflow processes for generating works of authorship or generative artificial intelligence. Alternatively, we include papers relating to stakeholders’ perceptions of ownership regarding their contributions.





Based on these criteria, out of the 5895 yielded papers, papers that did not meet the criteria, duplicates, and full papers that could not be found were removed to result in 855 papers. Given that this paper is a scoping study for the systematic literature review of these 855 papers, it restricts its scope. It systematically reviews papers from the past 3 years, from 2022 to April 2024, resulting in 338 papers.  A more thorough manual elimination process using title and abstract analysis was conducted to ensure that only the most relevant papers were reviewed. This round of elimination resulted in 56 papers selected papers. 31 papers were randomly selected for this systematic scoping literature review. After reviewing the selected papers, 10 were eliminated as they did not meet the inclusion/exclusion criteria mentioned in this section, resulting in 20 full papers forming the final dataset.



 §.§ Data Analysis

Our units of analysis were the first 100 peer-reviewed, published full papers that came through the search as mentioned earlier queries, inclusion, and exclusion criteria from the ACM Guide to Computing literature library. We then analyzed the final filtered papers using MaxQDA 24, a qualitative and quantitative data analysis software. We used descriptive and in-vivo coding techniques to code the filtered papers for the following research questions: What do stakeholders in the GenAI supply think they value in their contributions to the GenAI supply chain?
We used affective methods such as in-vivo coding and value coding<cit.> by assigning codes to attributes that interviewed stakeholders, in their interview responses, mention or specify as incentivized or are considered ‘good’. The assigned codes were then analyzed using versus coding as an inspiration with labor theory and utilitarian theory as dichotomous groups.
Although we paid particular attention to the findings/results sections of the filtered papers to capture the interviewed stakeholders’ first-hand perceptions, we also carefully read through all the sections where stakeholders or the authors of the filtered papers mentioned any valuation or ownership perceptions.



§ RESULTS

Using value coding and in-vivo coding, we identified over 600 codes, structured into 8 groups. The grounding for each theme with each stakeholder group is illustrated in Fig. 2:

It must be noted that some of the identified themes do not correspond directly to our research questions. However, it is indicative of a successful thematic analysis that all the identified themes do not simply correspond to the research questions <cit.>.  We have not collected and analyzed any descriptive statistics or metadata of the selected reviewed papers other than those required in the search terms, and inclusion and exclusion criteria.



What do stakeholders in the GenAI supply think they value in their contributions to the GenAI supply chain?


 §.§ Value added by creatives while creating expressive work

Interpersonal communication and sense of community: 
Some creatives believe they add value when their expressive work enables interpersonal communication and a sense of community. The text-to-image artist in one study <cit.> claimed that discovery (of new concepts) and a sense of community is rewarding for them as an artist:  “Everything is more fun when you can share it.” This emphasis on communicating ideas is also reflected in other studies<cit.> emphasizing the human need to communicate and share ideas:  “We are humans. We want to communicate others how we feel, what ideas we have and that’s what it is. It’s an accelerator for human ideation and being able to reach.” In addition, participants in one study <cit.> highlight that their expressive work is a medium to “maintain closeness with their audiences across platforms”. 

Impact and activism: 
Some creatives believe that the value of expressive work lies in the impact and activism it inspires in the artists and the public. One of the participants in the study<cit.>  noted that creating expressive work is  “not just having fun ... but [actually] making something that has some power.” A participant from another study<cit.> interviewing creatives on TikTok stated, “And then January 6, happened. I was like, I understand this intricately. As someone who has studied white nationalism–as someone who has studied insurrections and fascism. Part of what I want to do is educate people to keep themselves safe. And every now and then I post book updates, every now and then I post [anonymized] content. But now it’s mostly news. It’s mostly talking about data transparency; it’s mostly talking about keeping people safe from threats they didn’t even know existed.” 

Vulnerabilities and lived experience: 
Interviews in studies show that creatives add value by tapping into their vulnerabilities and lived experiences to create expressive works. Simpson et al<cit.>, in their paper, noted that  'scripts, films, and produces videos on TikTok about surviving suicide, the intention behind creating these videos is not to receive attention and acclaim, but to help cope with their grief and help others cope with theirs.' A creative in the same study<cit.> noted that “just being yourself” garners more success. A writer interviewed for a paper<cit.> to understand feedback networks emphasized the importance of the lived experience of their peers as an important factor in dictating whom the writer would ask for feedback: 
 "Since my brother is also Indian if I want to know how something reads to another Indian person, I will show him. But then if I’m writing a story about girlhood, I’ll send it to my friend Jen, who also writes about girlhood."
 Another text-to-image artist in the same study<cit.> noted that “Art doesn’t live in a vacuum, nobody starts from scratch, everything is based on something else.” 

Expertise, ideation, and implementation of creative ideas: 
Expertise, ideation, and implementation of creative ideas bring value to creatives' contributions while creating expressive content. Participants in a study<cit.> consider the artist’s background and education relevant when using text-to-image generation in the creative process. They state that:  “artistic intention is required to guide the interaction, prompt engineering benefits from knowledge of art history styles and artistic techniques, and assessing the system’s output requires an intuitive understanding of aesthetics and rules to achieve e.g. color harmony.”  A virtual influencer interviewed in a study<cit.> noted that being a virtual influencer is more accessible because  “you don’t have to be pretty to be a Virtual Influencer, just smart and be a good storyteller.”  Another artist in a study<cit.> shared:  "As a professional artist, shading, lighting, and the process after flatting is an eminent way to express my uniqueness. If I see the new features [that automate shadowing and lighting], I may feel an occupational crisis"  thus, emphasizing the importance of skill and creative imagination. A creative interviewed in a study focussed on multi-platform content creation<cit.> stated that he  “understood that his audiences across platforms have different interests, so he curated different granularity of gaming topics of video content for each of his channels” showcasing his niche by curating and implementing ideas into different kind of content for his many channels.



 §.§ Value added by ML practitioners while developing ML models

Managing end-users needs: 
ML practitioners in a study<cit.> shared that  "user needs to be the basis of the entire design.”  In another study<cit.>, participants corroborated this requirement when they stated that  “the success of the deployment does not include satisfying the functional or performance requirements but also how it is satisfying the target users.”  However, in the former study<cit.>, some participants expressed their trust in their abilities to map user needs, and some participants noted that information about the user and their needs is often collected by conversing with the client and not the users. This paper<cit.> also stated that in 9 out of the 12 companies, the technical AI development team maps users' needs. In the remaining 3 companies, UX designers were responsible for mapping user needs.

Documentation and explainability: 
Only one ML practitioner in the reviewed literature<cit.> shared that they also need to “version each model and be able to connect them to datasets they were trained with”.

Expertise and ability to meet clients’ needs: 
ML practitioners add value to the ML supply chain through expert domain knowledge and their ability to meet clients’ needs. A participant in a study<cit.> noted that  “In addition to the AI development, AI teams are responsible for the need assessment, early-phase interaction design, and communications with the client.”
 Although companies often either built their models in-house or used third-party models<cit.>, model development, itself, was considered a highly iterative process where 'a lot of time is spent on tweaking the model parameters to achieve best possible performance.'<cit.> 8 out of 12 participants in a study<cit.> noted that they do not have established processes when designing a model. Due to AI's uncertainty and dynamic nature, the ways of working strongly depend on the data, product, used AI technique, and the client<cit.>.  A participant in another study<cit.> also corroborates this uncertainty when they expressed: "When [the ML model is] multiplying enormous matrices together and then applying some non-linear functions to them a hundred times in a row, how can anyone ultimately know what is happening there?" 



 §.§ Value added by UX practitioner while developing ML models’ interface

Catering to end-users needs:
Interviews with some ML practitioners in the reviewed study<cit.> indicate that UX practitioners are responsible for managing users’ needs: an ML practitioner noted, “If we talk about the end-user, UI is the way to answer their needs. It’s hard for me to come up with any user-related practices during the early phase of development.”  
 These needs include ensuring simplicity, ease of use, and easy-to-test interface for AI models<cit.>. A UX practitioner in a study<cit.> highlighted the importance of simplicity and ease of testing while designing the proof of concept for an interface, he noted:  “We probably want to start with something minimal that we can actually test with users.” 

Accuracy, flexibility, and explainability of ML models:
The weight of user needs to ensure the accuracy, flexibility, and explainability of ML models falls upon the UX practitioners. A survey<cit.> studying the collaborative practices and tools of UX practitioners noted that accuracy and flexibility were two common goals for UX practitioners. When comparing the workflow for designing AI and non-AI users, the participants in the same study<cit.> noted that although the workflow for both user interfaces was not that different, they had to design to ensure user consent and awareness:  “When pure UX is concerned, I don’t see it as significantly different from any other user experience. I think that the user consent and awareness [of AI being used] needs to be a little bit more, but apart from that, in terms of experience, it needs to be like anything else I guess.” 

Designing, creating, and editing content:
Similar to creatives, UX practitioners add value to the ML supply chain by designing, creating, and editing content by applying their domain knowledge and expertise. UX practitioner participants in a study on text-to-image AI use<cit.> noted that GenAI allows them to side-step their artists when they are too busy, unavailable, or when their engagement is too time-consuming:  "We definitely can use just straight-out stuff from Midjourney and give them to an artist instead and they’ll be super happy now. This is a good idea and they can, like copy, paste and basically paint over stuff then maturing, and then we can use that already.".  In addition to creating and editing content, interviewees in a study<cit.> described several other considerations while designing an interface that highlights and applies their expertise which includes  “how the language of the VUI is ‘built’ i.e., its ‘components’, how content is delivered, and how temporality (e.g., length of speech) shapes design (P8). Interviewees argued that language components such as syntax, grammatical rules, semantics, and lexical morphology dramatically influence the user experience of voice interfaces (P6) and therefore provided a go-to set of reflective considerations during VUI design.”  



 §.§ Value derived by users from GenAI outputs:

Content that yields joy and entertainment:
Users interviewed in the reviewed studies noted that they value expressive content that yields enjoyment. A participant in a study<cit.> on blind visual art patrons noted,  “Exploring art is simple to me, I feel relaxed and connected to the environment while I am in art galleries. It is very similar to watching a movie or playing a game...”   
 For some patrons in the same study<cit.>,  “The goal for me to enjoy art is to understand the story and cultural background of different art pieces from different periods. Paintings and pictures are very language independent that you do not need to understand the language that the artists speak to appreciate their paintings..”   

Activism and messaging of expressive content: 
In addition to enjoyment, participants in another study<cit.> enjoyed expressive content for the activism and message of the expressive content. It must be noted that this study sought to understand the relationship between end-users and virtual influencers where some end-users stated that they followed the virtual influencer for  “the message they give, the art style, and mostly the stories they’re communicating."  Another participant in the same study<cit.> liked the positive message on environmental sustainability that the virtual influencers created. Some participants in this study benefitted from the endorsement that the virtual influencers advertised noting  "I’ll also pay more attention to the endorsement and likely form an opinion about the endorsing company that this must be a ’cool’ and ’in touch’ company for having an awareness of this.”     

Content that is novel, useful, and non-obvious: 
Interviewees in a study noted that the participants characterized good ideas as having at least two of the properties of novelty, usefulness, or nonobviousness (but not always all three)<cit.>.  Participants in another study<cit.> surveying virtual influencer followers validated the importance of novelty and nonobviousness:  “The limitations that traditional influencers face is simply the fact that they are human. As an actual living being, there are certain boundaries to the things they can do. On the other hand, a virtual influencer is just the creation of a human. It can and will do anything one can dream of. This makes their content unique, be it Reels, posts, or stories.". Another participant in the same study<cit.> noted,  “Maybe because I know they are fake, and someone is working really hard to make a fake person??? It’s like the "artist" or "creator" or whatever of the virtual influencer is showcasing their abilities/progression. Like an artist would, posting their art.” 
; A participant in the same study<cit.> stated  “Virtual influencers are liked for their aesthetic. Being virtual is sort of part of their thing.”

Connecting with the persona and interpersonal relationship with the creator: 
While being virtual adds value to the creativity of virtual influencers, it also causes some of the participants to be emotionally detached from them, thus enabling them to engage more confidently with the influencers<cit.>. One participant in a study<cit.> noted that  “With a VI account, I comment more easily because it’s lighter, I never comment on other accounts because I figure that stars or other celebrities can read and I don’t necessarily seek more interaction with them. With a VI account, I don’t really think about the person behind the account because I figure the person is more detached, for bee influencing for example it’s just a small part of their job I guess.” This detachment and sense of anonymity, however, renders a greater responsibility on the creatives as users believe<cit.> that such creatives can voice their opinions more freely and engage on more sensitive issues because they are not human and therefore do not have to worry about repercussions such as public hate:  “I think that, if she would live in our world and she is up to date about things happening in our society, she would probably give an unfiltered and pure opinion because she is not a real person. She will be able to tell things a normal person wouldn’t want to say. . .if you can use an AI to say things humans are going to get canceled for then why not. . . Social media is very toxic and people can be hated based on their opinion. Wouldn’t it be great to use AI for sensitive topics?”



§ ANALYSIS AND DISCUSSION



 §.§ Does copyright law protect the attributes valued by stakeholders in GenAI supply chain?



For our analysis, we rely on the concept of value similarity introduced by <cit.>. In <cit.> paper, a simulation experiment imitating the value similarity model proved that similarity in values or attributes can be a driving force for cooperation and trust in the regulation and design of public policy. Given this context, we attempt to understand the value similarity between the values behind exclusive rights enshrined by copyright law and the attributes valued by stakeholders in the GenAI system, a summary of which can be found in Table. 

Creatives’ value addition in terms of expertise, novelty in ideation, and implementation of those ideas are protected under copyright law and are appreciated by end-users consuming the implementation of these ideas. Both creatives and end-users appreciate enjoyable, impactful content that promotes advocacy. Given the expressive nature of this type of content, it is protected under copyright law. 

Creatives’ value additions by doing work that fosters interpersonal communications with their fans/followers and a sense of community are also appreciated by end-users. However, these attributes are not protected by copyright law. Creatives also add value when they share their life experiences and vulnerabilities with the end-users. However, this kind of input is not protected by copyright law.

ML practitioners make a distinction between end-users' needs and clients’ needs. They add value to the ML supply chain when communicating and meeting clients’ needs. Their work also adds value to the supply chain through their expertise and ensuring adequate documentation and explainability of the model. Although ML practitioners’ expertise is protected through the models, they develop along with their work process of documenting and ensuring the explainability of the model through the document itself, their ability to communicate with the clients and meet the clients’ needs remains unprotected under copyright law.
The UX practitioners’ value addition in terms of meeting end-users' needs through the UX design and interface itself can be protected under copyright law, along with the contributions they make when they design, create, and edit content. However, their contribution when they ensure flexibility in the model accuracy and agility remains unprotected.  

Recommendations: Given this gap between the values protected under copyright laws and the attributes valued by stakeholders, this paper, therefore, calls for the need to reinvent copyright law with computational solutions that go beyond the letter of the law but encapsulate the spirit of the law. In particular, given the analysis of these studied papers, as future work, researchers may consider developing tools that offer a communal environment for fostering interpersonal communication and a sense of community with other stakeholders in the ecosystem. Researchers may also consider creating tools that enhance the GenAI infrastructure by supporting creative autonomy and providing nuanced technical means of self-expression while taking into account stakeholder needs. 



 §.§ Does the advocacy of fair use doctrine enable the progress of copyright law’s objectives?

The analyzed papers also show that ML practitioners believe that data processes such as training and fine-tuning models are the most challenging aspects of their workflow. With GenAI companies like OpenAI advocating for training models as fair use and reports suggesting that researchers may run out of data to train, it is pertinent to ask whether using the fair use doctrine to train GenAI models is, in fact, producing transformative content that is enhancing or even promoting the enhancement of public knowledge of science and arts.

OpenAI, in its submission to the Copyright Office [OpenAI. 2019. Comment by OpenAI Regarding Intellectual Property Protection for Artificial Intelligence Innovation. Department of Commerce, United States Patent and Trademark Office. Retrieved December 22, 2023 from <https://www.uspto.gov/sites/default/files/documents/OpenAI_RFC-84-FR-58141.pdf>], clarified that AI systems learn patterns from the training corpus and use those patterns to generate ‘novel’ expressive works which ‘share some commonality with works in the corpus subject to the model learning from ‘enormous number of works’ . Furthermore, OpenAI argues that such infringement “is an unlikely accidental outcome.” However, a report by the US Congressional Research Service[Congressional Research Service. 2023. Generative Artificial Intelligence and Copyright Law. US Congress. <https://sgp.fas.org/crs/misc/LSB10922.pdf>]  brings forth a contradicting narrative by the Getty Images lawsuit [Getty Images (US) Inc & Ors v Stability AI Ltd [2023] EWHC 3090 (Ch)] that alleges “Stable Diffusion at times produces images that are highly similar to and derivative of the Getty Images.” This report also brings attention to a paper<cit.> that found “a significant amount of copying” in less than 2% of the images created by Stable Diffusion where the authors claimed that their methodology “likely underestimates the true rate” of copying . While 2% may seem like a significantly small number, and it can be agreed that generative AI does have the potential to create transformative work, extant literature<cit.> indicates that ML practitioners who develop and train these models face difficulties predicting whether the outcomes of such AI models will be original and transformative for society at all:  “When [the ML model is] multiplying enormous matrices together and then applying some non-linear functions to them a hundred times in a row, how can anyone ultimately know what is happening there?” 

These characteristics of ML models being black boxes, their unpredictability, AI mimicry, and overfitting are also impacting the drive of creatives to create expressive works. In a survey of more than 1000 artists<cit.> where more than 95% of the artists post their art online, 53% of them anticipate reducing or removing their online artwork if they have not done so already. Out of these artists, 55% of them believe reducing their online presence will significantly impact their careers. One participant stated,  “AI art has unmotivated myself from uploading more art and made me think about all the years I spent learning art.” 78% of artists anticipate AI mimicry would impact their job security, and this percentage increases to 94% for the job security of newer artists. Further, 24% of artists believe AI art has already impacted their job security, and an additional 53% expect to be affected within the next 3 years.  This impact on creatives' jobs is also seen in the accepted demands of the WGA which highlights the precarious artistic and financial security in the creative industry due to the advent of generative AI models. 

The argument of fair use falls flat if we apply Leaffer's considerations for fair use doctrine<cit.>  which include a) that the use must not impose economic harm to the original copyright owners and b) that the use is one that provides something new and potentially valuable to society. Firstly, GenAI, in its present version, has raised significant apprehensions amongst the original creatives content creators regarding the economic harms they face due to GenAI outputs.  Secondly, it is evident from the extant literature that although GenAI has the potential to create transformative work, as of the submission of this paper, GenAI is still plagued with issues of AI mimicry, overfitting, and excess memorization. 

Recommendations: Therefore, unless these problems of AI mimicry, overfitting and excess memorization are resolved computationally, GenAI will continue to create pattern-based content that is neither original nor novel, failing to validate its claim for eligibility for the fair use exception.



 §.§ Conclusion

This systematic literature review reveals attributes that stakeholders in the ML supply chain believe are incentivized in their contributions. Although GenAI models and their outputs find their origins for intellectual property ownership in the contributions of the data curators, the lack of studies that focus on data curators’ perspectives in this regard highlights a gap in the existing literature. This paper also uncovers computational gaps that expose cracks in the GenAI companies' call for training data for GenAI models to be considered fair use. It also recognizes the need to reinvent and complement copyright law with computational solutions for better implementation of the letter and spirit of the law. It, therefore, spells out computational problems in GenAI models and outputs for researchers and practitioners that demand immediate resolution for better implementation and enforcement of intellectual property laws and governing ownership and stakeholder value beyond the protections afforded by copyright.

icml2024



















 Paper Title: Diagnostic Runtime Monitoring with Martingales.





  When Audits and Recounts Distract from Election Integrity:
  The 2020 U.S. Presidential Election in Georgia
    
 Philip B. Stark10000-0002-3771-9604 

    Received ..... .., ....; accepted ......, ....
==============================================================================================================

empty
empty



§ ABSTRACT 
Machine learning systems deployed in safety-critical robotics settings must be robust to distribution shifts. However, system designers must understand the cause of a distribution shift in order to implement the appropriate intervention or mitigation strategy and prevent system failure. In this paper, we present a novel framework for diagnosing distribution shifts in a streaming fashion by deploying multiple stochastic martingales simultaneously. We show that knowledge of the underlying cause of a distribution shift can lead to proper interventions over the lifecycle of a deployed system. Our experimental framework can easily be adapted to different types of distribution shifts, models, and datasets. We find that our method outperforms existing work on diagnosing distribution shifts in terms of speed, accuracy, and flexibility, and validate the efficiency of our model in both simulated and live hardware settings. 



 




§ INTRODUCTION



Modern learning-enabled systems deployed in the real world are susceptible to failure if they encounter test-time inputs that do not follow the same distribution as the training-time inputs. However, the complexity of these learned systems leads to significant challenges in diagnosing problems that arise during operation <cit.>. Runtime monitors can alert users when issues arise, but pinpointing the underlying cause of the issue is difficult. For instance, a problem could stem from a suboptimal training process, during which the model was not exposed to a sufficiently diverse or representative dataset, leading to poor generalization capabilities. Alternatively, the problem might originate from the operational environment, where real-time conditions differ from those encountered during training (e.g. due to sensor degradation or environmental changes).




As an example, consider a camera-driven robot such as an autonomous aircraft using a PID controller to taxi along the centerline of a runway, where learned models estimate the cross-track and heading errors relative to that centerline. In this setting, problems that could cause the robot to fail include new situations that the robot has not previously encountered (e.g. the autonomous aircraft taking off from a runway at a new airport), or sensor degradation. In the case of new situations that the robot has not previously encountered, there is an offline problem (i.e. the unrepresentative training data), and we expect the distribution of both the system inputs and system outputs to change. For example, a new airport will look different than the original airport, so the input images will change; the width of the new runway will be different, so the distribution of deviations from the centerline will also change. In the case of sensor degradation, there is an online operational problem (the system that's running has changed), and we expect the distribution of system inputs to change, but not the distribution of system outputs. The input images may look grainier, but the distribution of the cross-track and heading errors will remain the same.  

In the example described above, using two runtime monitors that issue alerts in case of distribution shift — one over the system inputs and one over the system outputs — could differentiate between the two types of problems. More generally, if system designers identify that certain features correspond to specific issues, they can strategically place targeted runtime monitors over those features. This approach facilitates a more efficient troubleshooting process, since the activated monitors provide useful insights into which aspects of the system are contributing to the issue. 
By quickly detecting the cause of an issue, an appropriate intervention can then be applied. For example, in the case of a new runway at a new airport, the appropriate intervention is to collect additional data and perform weighted retraining. In the case of sensor degradation, the appropriate intervention is to replace the sensor. 

In this work, we present a framework for determining the cause of a distribution shift, using multiple martingale-based runtime monitors based on the method described in <cit.>. As in <cit.>, we focus on episodic situations (e.g. for a plane repeatedly taxiing down a runway during a continuous deployment, each taxiing sequence can be considered an episode drawn from a task distribution). Our method can be applied to any online learning setting, such as deploying autonomous vehicles, recognizing spoken language, or evaluating the performance of large language models. An overview of our system is shown in Figure <ref>.  

The contributions of our work are as follows: 1) We introduce a method that quickly alerts users when a distribution shift has occurred and enables rapid diagnosis of the underlying problem. 2) Our approach detects the cause of a distribution shift faster than prior work, and leads to better outcomes over the lifecycle of a robot. 3) We empirically evaluate our approach on photorealistic simulations of an autonomous aircraft taxiing down a runway with a camera perception module and in hardware on a free-flyer space robotics testbed for vision-based navigation. In these experiments, our method detects the cause of a distribution shift up to five times faster than prior work, and our method is effective at eliminating system failures due to causal misidentification experienced by a baseline system.




§ BACKGROUND AND RELATED WORK




 §.§ Distribution Shift Detection




  
Traditional Approaches. 
The challenge of detecting distribution shift has been explored by both the machine learning and statistics communities. Traditional approaches typically rely on statistical hypothesis testing to assess whether the test-time distribution differs from the training distribution. Most of these methods focus on covariate shift, where the distribution of input data changes while the conditional distribution of labels given inputs remains constant  <cit.>. Some methods have also been proposed for label shift, where the distribution of labels changes but the conditional distribution of inputs given labels remains the same. For example,  <cit.> propose a maximum likelihood algorithm for detecting and correcting label shift; <cit.> use an efficient weight estimator to provide a generalization bound for the label shift problem; <cit.> introduce black box predictors and a score-based test statistic for detecting label shift. However, these methods are generally designed for offline (batch) scenarios, and applying them online in a robotics setting may either compromise their statistical guarantee or lead to statistical inefficiency. Furthermore, these methods focus on detecting only a single type of distribution shift. 




  
Martingale-Based Approaches. 
A martingale (Definition <ref>) is a stochastic process (a sequence of random variables) where the conditional expectation of the next value, given all previous values, is the same as the most recent value <cit.>. Doob's Inequality (Proposition <ref>), states that the probability that a martingale grows very large is very small <cit.>.



 
A martingale is a sequence of random variables M_1, M_2, …, such that E[|M_n|] < ∞ and
E[M_n+1 | M_1, …, M_n] = M_n for all n.




For a martingale M_n indexed by an interval [0, N], and for any positive real number C, it holds that

    Pr[sup_0 ≤ n ≤ N M_n ≥ C] ≤E[max(M_N,0)]/C.



An approach for detecting distribution shift in online settings, introduced by <cit.>, uses conformal martingales to test for exchangeability. 
Other works following this approach include <cit.>. These methods apply conformal prediction to obtain p-values for each test sample, which are then used to construct a martingale. When the martingale exhibits significant growth, it indicates a probable distribution shift. <cit.> introduce different p-value calculations for different types of distribution shifts. While these works demonstrate good efficiency on lower dimensional data (e.g. on the USPS dataset <cit.>, which contains 11-dimensional feature vectors), they often struggle in more complex or higher-dimensional robotics settings, such as those involving image data. 



More recently, <cit.> introduce a learned, martingale-based runtime monitor that detects distribution shifts quickly for high-dimensional data, with guarantees limiting the number of false alarms. This method trains a neural network model to distinguish between older and more recent samples, and issues a warning when the model is consistently able to predict recency. A key component of this method is a martingale-based test statistic designed to grow if the distribution shifts. 
However, this work by default applies a martingale monitor only over the system inputs, making it impossible to distinguish between different underlying problems. In our work, we introduce a framework using multiple learned, martingale-based runtime monitors, and apply them over different feature spaces to improve problem diagnosis.



 §.§ System Fault Diagnosis

Many works leverage deep learning for system fault detection, using large models to detect one type of system fault <cit.>. However, these papers deploy large models with long training times that are impractical for live robotics systems, and they focus on diagnosing faults in specific mechanical systems in offline settings that are not easily generalizable. Our method, meanwhile, works with lightweight models suitable for online robotics settings. 

Other learned methods using simpler models for system fault detection include <cit.>.
However, these methods generally deal with low-dimensional data and require access to the true labels. Our method, meanwhile, works with high-dimensional data without access to true labels.



Some methods for fault diagnosis in online settings include <cit.>; however, these works focus on anomaly detection (i.e. experiencing a single rare event), while our work focuses on gradual distributional shifts.




§ IDENTIFYING TYPES OF DISTRIBUTION SHIFT
 



We present a framework for quickly identifying distribution shifts in episodic robotics settings that provides actionable information for targeted interventions.  Our method builds on the work of <cit.> and also leverages stochastic martingale-based runtime monitors.  Our critical contribution is to use multiple martingale-based monitors, deployed simultaneously with each monitoring a different type of distribution shift.  This enables our system to detect distribution shifts more quickly, and moreover the monitors each probe a distinct intermediate result of the robotics system.  This enables system designers to identify system components that are behaving abnormally before failure of the entire robotics system, develop appropriate interventions, and differentiate between types of distribution shift. 





 §.§ Problem Setup 

Consider an autonomous system that interacts with the environment.  The system's behavior is defined by function f: 𝒳→𝒴 which maps sensor inputs x^t ∈𝒳 at time t to action y^t ∈𝒴.  This function can be expressed as the composition of K functions, f = f_K ∘…∘ f_2 ∘ f_1, with K-1 intermediate results I_1, I_2, ... I_K-1, where I_1 = f_1(x), 
I_k = f_k(I_k-1)  for  1 < k < K, 

y = f_K(I_K-1).




In our problem setup we are given a dataset of N historical inputs sampled from a single distribution, D_orig = (X_1, X_2, ⋯, X_N).  Each X_i represents a sequence of sensor inputs recorded during an episode of the robot interacting with the environment, e.g. X_i = (x_i^1, x_i^2, …, x_i^l_i).  During the robot's deployment, in a potentially novel environment, we observe new sequences of sensor inputs D_new = (X_N+1, X_N+2, …).  Again each X_i in D_new represents a sequence of inputs recorded during an episode of the robot interacting with the environment.  

As shown in <cit.> it is possible to design a series of test functions over system inputs


    ψ^1_j: D_orig, X_N+1, ⋯, X_j↦{ T, F }
       ∀ j = N + 1, N + 2, …,

where the output T(rue) indicates that we have identified a distribution shift over the system inputs, while F(alse) indicates that we have failed to identify a distribution shift over the system inputs.

In this work we additionally aim to design test functions over the K-1 intermediate results and the final action.  These test functions will issue an output alert when a distribution shift is identified over the intermediate results or the final action.  The test functions will take the form

    ψ^k_j: D_orig, f^1 → k_j (X_N+1), ⋯, f^1 → k_j (X_j) ↦{ T, F }
       ∀ j = N + 1, N + 2, …,

where 1 ≤ k < K for test functions over the intermediate results and k=K for the test function over the output actions.  We define f^1 → k_j : 𝒳^l_j→𝒴^l_j as [That is, the domain of f^1 → k_j is the l_j-fold Cartesian product of 𝒳 and the codomain is the l_j-fold Cartesian product of 𝒴, where l_j is the length of episode j.]

    f^1 → k_j(X_j) := ( 
       f_k ∘…∘ f_2 ∘ f_1 (x^1_j), 
       f_k ∘…∘ f_2 ∘ f_1 (x^2_j), 
       ⋮
       f_k ∘…∘ f_2 ∘ f_1 (x^l_j_j)
    ).



Additionally, each test function is designed to be ϵ-sound (<cit.>), meaning that if the historical data D_orig and the new data D_new are drawn from the same distribution, then with high probability (1 - ϵ) the test will not issue a warning.  Formally, when D_new and D_orig are exchangeable, then 

    [∃ j, ψ^k_j (D_orig, f^1 → k_j (X_N+1), ⋯, f^1 → k_j (X_j) ) = T ] ≤ϵ.






 §.§ Proposed Framework 


The key intuition behind martingale-based monitors is that it is impossible for any predictor to reliably distinguish between two samples drawn from the same distribution.  Consider drawing two samples, one from D_new and the other from from D_orig.  If there has been no distribution shift (so these distributions are in fact the same), then no predictor can do better than random chance (e.g. a Bernoulli variable with parameter p = 0.5).

We adapt the martingale-based monitor defined in <cit.> to design the test functions ψ^1_j over system inputs.  This monitor relies on a neural network f^1_NN : 𝒳×𝒳→{0, 1} that is trained to predict whether two inputs come from the same distribution.  Additionally we design test functions ψ^k_j for k = 1, …, K over all K-1 intermediate results I_1, I_2, ... I_K-1 and over the final output action.  For the test functions over intermediate results, the neural network is modified to take as input two intermediate results, f^k_NN : I_k × I_k →{0, 1}.  
For the monitor over final output actions, the neural network is modified to take as input two actions, f^K_NN : 𝒴×𝒴→{0, 1}.  Note that our method is agnostic to the specific choice of model architecture (but for details about the model architectures that we used in our experiments, refer to the Appendix.

To design the test functions, let us first define the indicator variables

    Z^k(j) = 
        
            1if  f^k_NN predicts correctly for test example  j 
    
            0otherwise.

Then let us define the martingales

    M^k_n  = (e^t · S^k_n)/((q + p e^t)^n),

where S^k_n = ∑_j=1^n Z^k(j), p = q = 0.5, and we use t = 1 (Equation 4 in <cit.>).  Using these martingales we define the test functions as

    ψ^k_j( D_orig, X_N+1, ⋯, X_j) = 
        
            Tif  M^k_n  ≥ C 
    
            Fotherwise.

Using a threshold of C = 100 we are guaranteed a false positive rate of ≤ 0.01 for each test function because (Lemma 2 in <cit.>)

    Pr[sup_0 ≤ n ≤ N M^k_n  ≥ C] ≤1/C.


This allows our framework to propose a mitigation strategy tailored to the specific failure mode identified by the martingale monitor that signaled the distribution shift. 

In order to identify a particular type of distribution shift, we need to construct different martingales, each with representative inputs for our test functions. As an illustrative example, we consider distribution shifts that may occur for images taken by an external camera, as shown in Figure <ref>. During deployment, we may encounter sensor degradation, which is a shift that results in a change to the input images, so we feed in representative input features to the classifier. If a distribution shift is detected using these input features, we know that the distribution of our input images has changed, and a good mitigation strategy would be to replace the sensors. Similarly, if we encounter a new environment, such as a new airport, our output labels will change, as the plane taxis with a new trajectory. Therefore, we feed in output features to the classifier, and if we detect a distribution shift using the output features, we know that the distribution of our output labels has changed. Here, a good mitigation strategy would be to gather additional data and perform a weighted retraining. The key insight is to use knowledge of a likely type of distribution to inform the relevant chosen features.



§ EXPERIMENTS 


  
We empirically evaluate our method on photorealistic simulations of an autonomous aircraft taxiing down a runway using camera-based perception and in hardware on a free-flyer space robotics testbed for vision-based navigation. We show that our method detects distribution shifts and identifies the type of distribution shift more quickly than existing methods. We also show that the knowledge of distribution type provided by our method allows us to use targeted interventions, which leads to better results than generic interventions, and better outcomes over the robot lifecycle compared to a scheduled maintenance approach. 




 §.§ Experiment 1: Fast categorization of distribution shifts

We first demonstrate that our method is able to detect different types of distribution shifts faster than prior methods. To evaluate our method, we use data from the photorealistic X-Plane 11 flight simulator where an autonomous aircraft uses a PID controller to taxi along the centerline of the runway. 


Next, we introduce two types of distribution shifts to the data: first, we introduce Gaussian noise to the images to simulate sensor degradation, and second, we add 200 new sequences of images of the plane taxiing at a new airport, KJFK, that was not in the training dataset (see example images in Figure <ref>). During deployment, images are drawn from either the original sequences, the sequences with added noise, or the sequences from a new runway. To detect whether a shift has occurred and categorize the type of shift, we deploy two stochastic martingales simultaneously to act as runtime monitors. An alert is raised when the martingales reach a threshold of 100. If no alert is raised after 200 samples, the experiment is terminated. The experiment is repeated 100 times.



We compare our method against the method described by Vovk in <cit.>, which we will refer to as the conformal martingale (CM) method. For the CM method, we use the nearest distance nonconformity score and the ratio nonconformity score respectively for the two martingales. For each type of distribution shift, we record the number of iterations before an alert is issued by our method and the CM method. The results are summarized in Table <ref>. A representative plot showing martingale growth for our method and the CM method is shown in Figure <ref>.


In every scenario, both methods are able to detect the correct distribution shift when one exists (sensor degradation, new environment) and do not issue false alerts when there is no distribution shift, i.e., we empirically observe no false negatives or false positives. Additionally, in every scenario, our method issues an alert corresponding to the correct shift in a fewer number of iterations compared to the CM method, showing that our method is able to detect and categorize distribution shifts more quickly than existing methods. 




 §.§ Experiment 2: Impact of targeted interventions

Next, we look at how the knowledge of the type of distribution shift enables us to choose the right intervention, which leads to better performance compared to a generic intervention. To evaluate the effect of interventions, we use the same X-Plane dataset as the previous experiment. Images are sampled from the original sequences, the sequences with sensor degradation, or the sequences from a new runway (refer to example images in Figure <ref>). 


If an alert is raised, i.e., a distribution shift is detected by one of our martingales, an intervention is applied. If the alert indicates that sensor degradation has occurred, the correct targeted intervention is to replace the sensor. On the other hand, if the alert indicates that the environment has shifted, the correct targeted intervention is to perform weighted retraining to prioritize learning the most recent samples. We also evaluate a generic intervention, which is retraining the model with equal weight for all samples. After the intervention, the experiment is continued for 100 more samples. Mean squared error (MSE) of the predicted cross-track and heading distance is recorded before the distribution shift and after the intervention. We also record the MSE if no intervention is applied. The comparison of MSE for different types of interventions (over 100 trials) is shown in Table <ref>.

Unsurprisingly, the model predictions degrade after a distribution shift occurs, so if no intervention is applied, the MSE increases significantly (from 0.17 to 0.885 or 0.771, depending on the shift). A generic retraining of the model reduces the MSE somewhat. However, the greatest reduction in MSE is achieved if the type of shift is identified and the correct corresponding intervention is applied. 



 §.§ Experiment 3: System lifecycle performance compared to a scheduled maintenance approach

Next, we show that applying the correct targeted intervention for each type of distribution shift not only improves model performance in the short-term, but leads to overall better performance and greater safety over the system lifecycle. We posit that a standard approach to maintain lifelong performance, in the absence of runtime monitors for different types of distribution shifts, might be to adopt a standard maintenance schedule, where all interventions are scheduled at a fixed cadence, e.g., every γ iterations. We compare such a maintenance schedule against our method, where only targeted interventions are applied, and only when the corresponding shift has been detected.




We see that our method of only applying targeted interventions when a distribution shift is detected often achieves a lower MSE over the lifecycle as compared to the maintenance schedule, despite the maintenance schedule running every intervention at a fixed cadence. When the maintenance frequency (γ) is equal to the rate at which shifts occur (λ), our method achieves a lower MSE. Only when maintenance is carried out at double the frequency, i.e., γ is half of λ, does the maintenance schedule outperform our method in terms of lower MSE. 



More importantly, our method reduces the frequency of crashes (in fact, completely prevents crashes in this experiment) in all cases while the maintenance schedule occasionally leads to crashes, i.e., on average 3-5 crashes over 1000 iterations. This is because our method uses runtime monitors to quickly detect that a distribution shift has occurred and applies the correct intervention to prevent excessive degradation of the predictions. On the other hand, maintenance is carried out at a fixed cadence, so in some instances where a shift occurs soon after a scheduled maintenance has been completed, that shift can quickly cause errors to snowball and lead to a crash within 45 iterations, before the next scheduled maintenance can take place. Thus, we see that timely and targeted runtime monitoring leads to better performance and fewer failures over the system lifecycle. 




 §.§ Experiment 4: Tailoring custom martingales to specific types of shifts


Custom martingales can be designed to detect other specific types of shifts. As an example, we define a new martingale to detect a shift in brightness of the images, since a brightness shift is a common trigger for existing runtime monitors. This martingale uses the same CNN classifier as before, but it is trained by using the feature vector from the layer of a ResNet-50 architecture that most closely corresponds to image brightness. Specifically, we evaluate all 50 feature vectors from the intermediate layers of the ResNet-50 model and choose the feature vector that changes the most with changes in image brightness. This experiment reinforces one of the advantages of our method: system designers with insights into a likely type of distribution shift can construct a monitor to alert an issue if such a shift occurs.  




On average, all three martingales usually detect their corresponding distribution shift faster than the other two martingales. It is worth noting that a change in brightness also induces a shift in image distribution. Therefore, the brightness martingale detects the shift first 89% of the time, taking, on average, 17.92 iterations to issue an alert. The other 11% of time, however, the sensor degradation martingale detects a shift first, because the input distribution is changing as well. Similarly, a shift such as sensor degradation not only changes the distribution of input images but also that of labels, so eventually an alert is raised by the environment shift martingale as well.  Most notably, the brightness martingale only issues an alert for a shift in brightness and not for the other two shifts. This demonstrates that it is possible to design custom martingales that act as runtime monitors for specific predefined distribution shifts, allowing system designers to use their domain expertise to design the most informative runtime monitors. 





 §.§ Experiment 5: Categorizing distribution shifts on a hardware free-flyer space robotics testbed


We also demonstrate the effectiveness of our method on a hardware free-flyer space robotics testbed with input from a forward-facing Intel Realsense D455 camera mounted on the side, as pictured in Figure <ref>. The free-flyer is a cold gas thruster-actuated 2D mobile robot that floats almost frictionlessly on a smooth granite table, developed to simulate zero-g or zero-friction conditions in aerospace robotics applications <cit.>.







We introduce distributions shifts similar to the previous experiments. We simulate sensor degradation by adding Gaussian noise to the images, an example of which is shown in Figure <ref>. We simulate environment shift by changing the starting position (and therefore, the resulting trajectory) of the free-flyer, an example of which is shown in Figure <ref>. We measure how many time-steps it takes to detect a distribution shift, comparing both our method and the CM method using nearest neightbor nonconformity scores <cit.>. The results are summarized in Table <ref>.

For both the sensor degradation and environment shift, our method detects a shift faster than the CM method. Additionally, neither method issues a false positive, i.e., neither raises an alarm when a shift has not occurred. These results show that our method more rapidly warns us of different types of distribution shifts compared to existing methods and can be used for runtime monitoring in safety-critical contexts.






§ CONCLUSION



In this paper, we introduce a novel framework for diagnosing the cause of a distribution shift, and experimentally validate the speed and adaptability of our method to different types of distribution shifts, models, and datasets. We demonstrate that knowledge of the underlying cause of a distribution shift allows the system designer to induce an appropriate mitigation strategy, and applying our method outperforms a traditional maintenance schedule while reducing cost.  
 

In future work, we would like to explore combining our method with other strategies to prevent performance degradation in real-time systems <cit.>, and understanding how to combine possibly correlated monitors to achieve stronger system performance guarantees.




isrr2024_format/styles/bibtex/spbasic




§ APPENDIX





§ OVERVIEW: TYPES OF DISTRIBUTION SHIFT

    In this section, we clarify the context of our distribution shifts with respect to the well-defined mathematical notions of covariate, concept, and label shift. In a supervised machine learning setting, if X is the input and Y is the output, the training data for a model is a set of samples from the distribution P(X,Y), and the model is learning to approximate the distribution P(Y|X) <cit.>. By the definition of conditional probability,
    
    P(X,Y)  = P(Y|X)P(X) 
     
                    = P(X|Y)P(Y)
 
    In a covariate shift, P(X) changes, but P(Y|X) remains the same, In a label shift, P(Y) changes, but P(X|Y) remains the same. In a concept shift, P(Y|X), but P(X) remains the same <cit.>. However, in practice, multiple shifts may occur jointly, as a change in the P(X) distribution can affect the P(Y) distribution, and vice versa. In this paper, the distribution shifts that were induced do not necessarily fall cleanly into one of these mathematical categories. For example, the sensor degradation shift in Section <ref> has a change in the P(X) distribution (i.e. a change in the distribution of input images) which our method detects, but no additional information about the P(Y) distribution or P(Y|X) distribution. The new environment shift has a change in P(Y) (i.e. a change in the distribution of trajectory labels) which our method detects, but no additional information about the P(X) distribution or P(X|Y) distribution.

    Note that the nature of the distribution shift detection problem differs from that of the anomaly detection problem, as distribution shift detection focuses on identifying a change in the distribution of samples (and it is arguably impossible to make a distributional claim without multiple samples of evidence), whereas anomaly detection focuses on identifying a single unusual sample or rare event. Moreover, the intervention method for these two tasks differs significantly. A safety intervention or fallback should be triggered immediately (i.e. within an episode) in response to a sudden change for an unsafe anomalous event. However, interventions for distribution shifts can occur on a somewhat longer timescale (e.g. model retraining), and an alert about distributional shift should inform decision-making about successive deployments.

    The main advantage of our method is as follows: Based on which monitor signaled the distribution shift first, system designers can focus their efforts on improving specific components of the system or gathering additional data. For instance, if an issue is caused by sensor degradation, specific monitors will issue an alert and a good mitigation strategy would be to replace the sensors; if the issue stems from encountering a new environment, other monitors will issue an alert and a good mitigation strategy would be to gather additional data and perform a weighted retraining. In theory, a system designer could apply every possible mitigation strategy each time any distribution shift is encountered, but in practice doing so is costly and inefficient. Ideally, we want to find the best intervention strategy as quickly as possible, which requires that we are able to identify each particular type of distribution shift.
    


§ X-PLANE EXPERIMENT DETAILS 



In this section, we provide further details on the X-Plane experimental setup, including the model architecture that was used and the methodology with which the distribution shifts were induced. 



 §.§ Model Architecture

For all X-Plane experiments, we use the model architecture outlined in Fig <ref> with the Adam Optimizer, a batch size of 32, a fixed learning rate of 1e-4, and a binary cross-entropy loss function. As mentioned in the paper, our method is agnostic to the model used for inference, and we used a simple model to demonstrate the efficacy of our method on small models. For each type of distribution shift, we trained a model with essentially this architecture, except that the input to the model is the cross-track and heading error in the environment shift case and a feature vector of size 100,352 (2048*7*7) in the brightness shift case (this feature vector is the flattened output of one of the intermediate convolutional layers of a ResNet-50 model). The code for running these simulations will be released upon acceptance of this paper.   




 §.§ Experiment Details: Sensor Degradation

Using NASA's XPlaneConnect Python API, 1000 simulated video sequences are generated, each corresponding to a taxiing episode of the aircraft on the runway at KSVC airport <cit.>. Each taxiing sequence consists of approximately 30 images of size 200x360x3. These sequences occur at randomly initialized times between 8:00am and 10:00am. We train a classifier to distinguish the “more recent” image on these image sequences, by randomly sampling one image from each sequence. The distance to the centerline is estimated by a pre-trained DNN, using images from an outboard camera mounted on the plane. 

In the sensor degradation experiments, we gradually add Gaussian noise to a set of images in the unseen test set. In the initial experiments, we induce Gaussian noise with a kernel size of (15,15) and a σ value that gradually increases. For Experiments 1, 2, 4 and 5, we increase σ by 1 at each time step until it reaches the value of 50. With σ=50, the Gaussian noise becomes so extreme that our predictor causes the plane to go off the runway after approximately 105 timesteps. For Experiment 3, we increase σ by 2.5 until it reaches a value of 100. With σ=100, we observe a crash after approximately 45 timesteps. We chose a more significant σ value in Experiment 3 to simulate a crash if either method does not fix a shift fast enough. When the σ value reaches 50 and 100 respectively, we continuously sample images blurred with these values. See Fig. <ref> for example images.



 Note that it takes around 30 minutes to train the initial model on a Macbook Pro M1 chip on the X-Plane dataset of approximately 30,000 images for the sensor degradation shift, and around 2 minutes to retrain after each update step. Each sequence consists of approximately 30 images of size 200x360x3, with 1000 samples observed. 800 of these were used in the training set, while the other 200 were modified according to the Gaussian blur rule defined above. 
 




 §.§ Experiment Details: Environment Shift

In the environment shift experiments, we sampled 200 sequences of images from a new runway, KFJK. We induce the same sinusoidal trajectories as before, except with a larger amplitude in order to account for the larger width of the runway. 
In this case, the two linear layers of the neural network had dimension (4, 512) and (512,1), which took approximately 10 minutes to train initially on all 30,000 images, and 45 seconds for each model update. If a warning signal is not issued, the plane will crash after approximately 85 time steps as the predictor will cause the plane to taxi off the runway. 



 §.§ Experiment Details: Brightness Shift

In the brightness shift experiments, we sampled 200 sequences with the brightness of the image reduced by 50%. If a warning signal is not issued, the plane will crash after approximately 120 time steps. The model we used for this method is the same architecture as in Fig. <ref>, except that the input to the model is a feature vector from an intermediate layer of a pre-trained ResNet-50 model, rather than the image itself.
 
In order to determine which intermediate layer to use, we constructed a martingale from each intermediate layer of ResNet-50 and ran them simultaneously in order to determine which layer corresponded most closely to image brightness. We experimentally determined that Layer 49 was the fastest in detecting the brightness shift, so we used that martingale when running all of the rest of our experiments. 



 §.§ Experiment Details: No Shift

We again take 1000 video sequences of a plane taxiing down a runway using the X-Plane 11 simulator. All sequences occur without a distribution shift, with approximately 30 images of size 200x360x3 in each sequence. We run all of our martingales on these sequences to experimentally show that we issue no false positive warnings when there is no shift.



 §.§ Experiment Details: System Lifecycle

We simulate the system lifecycle by constructing sequences of 1000 iterations from the X-Plane dataset and inducing either sensor degradation or environment shift to the data according to a Poisson process with parameter λ corresponding to the expected rate of occurrence of the shift. As before, sensor degradation is simulated by adding noise to the images and environment shift is simulated by introducing images from the new runway. The shifts are large enough that if an intervention is not applied within 45 iterations after the shift occurs, the degraded model predictions cause the aircraft to deviate off the runway and crash. 









§ FREE-FLYER HARDWARE EXPERIMENTS




We collect data from multiple episodes of the free-flyer moving from a starting position to the docking target. This data consists of 9000 camera images from 30 episodes (each image is of size 360x640x3) as well as the associated ground truth relative pose of the free-flyer with respect to the docking target at each iterations, which is recorded using a motion capture system above the free-flyer testbed. These images are used to train a classifier to distinguish the “more recent" image. An example image is shown in Figure <ref>.

In these experiments, the free-flyer performs a learning-based visual servoing task that emulates autonomous spacecraft docking. Images from the onboard camera are used to guide the free-flyer from a starting point towards a static visual target (as shown in Figure <ref>), which is a pattern defined by the International Docking System Standard for spacecraft docking adapters. 

R0.5
    
    < g r a p h i c s >

    Martingale values growing to indicate a distribution shift, specifically sensor degradation, in a real-time robotics setting recorded on the free-flyer hardware. The martingale associated with our method grows much faster than the martingale associated with the CM method, demonstrating that our method is able to issue an alert faster than existing methods.
    


To collect the initial image data, we positioned the robot at or around a fixed starting point on the granite table while ensuring that the visual target stayed within the field of view of the camera. For the new environment distribution shift, we collected data by positioning the robot at a different starting point about one meter away from the original starting point. We captured 30 sequences from each starting point with around 300 images per sequence for a total of 18,000 images, as well as the corresponding x,y position of the freeflyer using the Optitrack motion capture system. For the sensor degradation experiment, we add Gaussian noise to the data from the original starting point using the same process as described above for the X-Plane experiments with σ=50. For both sets of experiments, we use the model architecture shown in Fig. <ref>, except that instead of predicting the cross-track and heading error in the environment shift case, we predict the x,y position of the free-flyer on the granite table relative to the visual servoing target. A representative plot of the martingale growth for our method and the CM method is shown in Figure <ref>.









 Paper Title: Taming the Frequency Factory of Sinusoidal Networks.




Enhancing Deep Hedging of Options with Implied Volatility Surface Feedback InformationWe thank John Hull and Zissis Poulos for valuable feedback about this work. Gauthier is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC, RGPIN-2019-04029), a professorship funded by HEC Montréal, and the HEC Montréal Foundation. Godin is funded by NSERC (RGPIN-2024-04593).
 
    [
     
August 5, 2024
=============================================================================================================================================================================================================================================================================================================================================================================================================




§ ABSTRACT 
This work investigates the structure and representation capacity of sinusoidal MLPs, which have recently shown promising results in encoding low-dimensional signals.
This success can be attributed to its smoothness and high representation capacity.
The first allows the use of the network's derivatives during training, enabling regularization. 
However, defining the architecture and initializing its parameters to achieve a desired capacity remains an empirical task. This work provides theoretical and experimental results justifying the capacity property of sinusoidal MLPs and offers control mechanisms for their initialization and training. We approach this from a Fourier series perspective and link the training with the model's spectrum.
Our analysis is based on a harmonic expansion of the sinusoidal MLP, which says that the composition of sinusoidal layers produces a large number of new frequencies expressed as integer linear combinations of the input frequencies (weights of the input layer). We use this novel identity to initialize the input neurons which work as a sampling in the signal spectrum. We also note that each hidden neuron produces the same frequencies with amplitudes completely determined by the hidden weights. Finally, we give an upper bound for these amplitudes, which results in a bounding scheme for the network's spectrum during training.







§ INTRODUCTION




Implicit neural representations (INRs) have emerged as a powerful method to represent media objects, such as images and surfaces.
They contrast with the classical techniques as they encode the data continuously on the parameters by mapping the coordinates to the signal values.
For example, in the case of an image, an INR f receives a point (x, y) and returns the  value f(x, y). In this case, the network's domain is low-dimensional, which differs from classical machine learning representations, where the input is a discrete image. Another difference is that we would like the INR to fit the input data as close as possible, as in the problem of approximating signals using radial basis functions.

As noted by <cit.>, since multilayer perceptrons (MLPs) activated by  cannot introduce high frequencies, they are not capable of representing detailed signals. 
To avoid getting stuck in a local minimum of the loss function where the INR does not contain high frequencies, they project the input coordinates onto a list of sines and cosines (Fourier feature mapping). 
This projection allows the INR to bypass the spectral bias <cit.> and represent detailed signals.

In parallel, <cit.> presented an initialization scheme for sinusoidal MLPs, which were considered difficult to train due to the initialization leading to undesirable local minima <cit.>. Their scheme ensures stability and convergence during training. 
An observation that partially justifies the representation power of a sinusoidal INR is that its first layer acts as a Fourier feature mapping <cit.>.

Several works used sinusoidal INRs to represent images <cit.>, surfaces <cit.>, animation <cit.>, and radiance fields <cit.>. They have empirically demonstrated that such INRs have a high capacity to represent fine details. However, they also show well-known limitations <cit.>, such as controlling the signal bandlimit and determining the appropriate parameters to efficiently represent a given signal.

This work considers an alternative approach for such problems through a novel trigonometric identity (Theorem <ref>) which resembles a Fourier series of a sinusoidal neuron.
We use this formula to understand the network's capacity by investigating how the layer's composition compacts information. 
In addition, we explore practical issues of initialization in sinusoidal INRs providing a reinterpretation of SIREN's scheme and ways of improving it.
For this, we note that the expansion of each sinusoidal neuron produces a large number of new frequencies in terms of the first layer weights (input frequencies). 
We also prove that the corresponding amplitudes are bounded by a term depending only on the INR's hidden weights (Theorem <ref>).
Finally, we apply this result to control the bandlimit of the network spectrum during training.
In summary, our contributions are:

    
  * We present a trigonometric identity that says that each sinusoidal neuron expands as a wide sum of sines with its new frequencies being an integer linear combination of the input frequencies. 
    The corresponding amplitudes are given by the hidden weights. 
    
  * We provide an upper bound for the amplitudes in the expansion of a sinusoidal neuron, which offers tools to control the resulting signal using the hidden weights.
    
  * We use the trigonometric identity to initialize sinusoidal INRs. First, we initialize the input neurons using the fact that they determine the possible INR spectrum.
    Then, we initialize the hidden weights, taking into account that they determine the corresponding amplitudes.
    
    
  * We use the amplitude's upper bound to control the bandlimit of a sinusoidal INR. For this, we design schemes to bound the hidden weights during training.
    
    These boundings, along with the initialization generate frequencies progressively around the input frequencies.








§ RELATED WORKS




Implicit neural representation (INRs). INR <cit.> is a trending topic in the machine learning community that is used to learn highly detailed signals in low-dimension domains.
Current INR architectures use Fourier feature mappings <cit.> or sinusoidal activation functions <cit.> to bypass the spectral bias <cit.> common in  multilayer perceptrons.
The high representation capacity of sinusoidal INRs has motivated their use to represent a wide range of media objects. Examples include audio <cit.>, images <cit.>, face morphing <cit.>, signed distance functions <cit.>, displacement fields <cit.>, surface animation <cit.>, multi-resolution signals <cit.>, among others.
Most of these works exploit the derivatives of sinusoidal INRs in the loss functions.


Initialization.
Considering sinusoidal activation functions in neural networks is a classical problem <cit.>; however, these INRs have been regarded as difficult to train <cit.>. <cit.> overcome this by presenting a specific initialization scheme that allows training sinusoidal INRs, avoiding instability and ensuring convergence.

Despite these advances, in practice, the initialization of such INRs remains an empirical task. In this work, inspired by Fourier series theory, we present a novel initialization method that allows us to train INRs with bandlimit. 
Recently, several works have addressed the representation problem of sinusoidal INRs from different perspectives. <cit.> approached this problem by observing that the first layer of a sinusoidal INR is similar to a Fourier feature mapping. 
Here, in addition to improving the initialization of sinusoidal INRs, we present a training scheme for bounding the spectra of these networks.




Control of spectrum.
One of the main drawbacks of sinusoidal INRs is the lack of frequency control. <cit.> addressed this issue by multiplying the input frequencies by a factor ω_0 to scale them to an appropriate range. However, even an appropriate choice of ω_0 could result in a noisy reconstruction since we have no control guarantees over the network spectrum.
We avoid the ω_0 problem by providing a novel initialization for the first layer. This is inspired by the fact that the frequencies generated by the network are determined by the input layer. Additionally, we provide a bounding scheme during training considering that the hidden layers define the frequency amplitudes.
Other works <cit.> employed multiplicative filter networks (MFNs) <cit.> to control the bandlimit by applying a filter over the spectrum. However, this strategy is prone to introducing reconstruction artifacts. It is important to note that MFNs are fundamentally different from neural networks as they are not modeled by the composition of sinusoidal layers, which is the focus of this work.




§ A FOURIER SERIES INTERPRETATION OF SINUSOIDAL INRS





This work addresses the problem of obtaining an efficient initialization and a training scheme with controlled bandlimit for the frequencies of sinusoidal INRs. This section presents the mathematical definitions and some novel formulas for approaching such tasks.








Sinusoidal INRs demonstrated a high representational capacity with only a few hidden layers <cit.>. 
Thus, to understand how the composition of its layers are compacting the information, we propose to investigate a sinusoidal INR f:^d→ consisting of a single hidden layer S:^m→^n, with m,n∈.
Specifically, we consider f to be expressed as f(x)= L∘S∘D(x),
with S(x) = sin(Wx + b), where W∈^n× m is the hidden matrix, and b∈^n the bias.
The input layer D(x)=sin(ωx+φ) projects the input x into a list of harmonics (input neurons) D_i(x)=sin(ω_i x+φ_i) with frequencies ω=(ω_1,…, ω_m)∈^m× d and shifts φ∈^m.
Finally, L(x)=Cx+d is a linear layer.
We show empirically that such architecture has enough capacity for our purposes.

We now present some properties of the sinusoidal layers, which play key roles in the representation, and give a reinterpretation of them in terms of the network parameters.

First, note that the hidden neurons h(x) := S∘D(x) are defined by composing the dictionary D with the hidden sinusoidal layer which results in a list with elements 
h_i(x)=sin(∑_j=1^mW_ijsin( ω_jx+φ_j)+b_i).



We now present a key result of this work, which states that we can linearize a hidden neuron by a sum of sines with frequencies and amplitudes completely determined by ω and .  

    Each neuron h_i of a sinusoidal INR has an amplitude-phase expansion of the form
    
    h_i(x) = 
                ∑_k∈^mα_k sin(β_k x + λ_k),    where   β_k = ∑_j k_j ω_j,   λ_k=∑_j k_jφ_j + b_i,

    and α_k = ∏_j=1^mJ_k_j(W_ij) is the product of the Bessel functions of the first kind.


The Bessel functions J_k appear in the Fourier series of sin(asin(x)) <cit.> and Theorem <ref> is a generalization of this result.
The complete proof adn details can be found in Appex <ref>, and Appedx  <ref> gives a generalization for more hidden layers.


[13]r0.42
     

    
    < g r a p h i c s >

    

    Comparison between INRs with zero and one hidden layers. Training with one hidden layer is significantly faster (50s vs 13m), uses half the parameters (33842 vs 67803), and produces higher quality results (34.4dB vs 21.1dB).
    

We list some consequences of (<ref>).
First, the layer composition introduces a wide number of frequencies β_k depending only on ω, and shifts given by the input shift φ and the hidden bias b.
Precisely, truncating the expansion by summing over k_∞≤ B∈, [k_∞ denotes max{k_1, …, k_m}.] implies that each neuron h_i could learn (2B+1)^m -1/2 non-null frequencies.
This frequency factory
 explains why composing sinusoidal layers may compact data information. Indeed, Fig <ref> shows that training with fewer parameters using a narrow sinusoidal INR with one hidden layer performs better than a wide INR with no hidden layer.
Additionally, we explore this factory to present an initialization of the INR's input neurons in Sec <ref>.

Secondly, note that the weights W fully determine the amplitude α_k of each harmonic sin( β_k x+λ_k) in the expansion. 
Additionally, we can derive the corresponding sine-cosine form of (<ref>),


    h_i(x) 
       =∑_k∈^mA_ksin(
            β_kx
        ) +B_kcos(
            β_kx
        ),    with   A_k=α_kcos(λ_k),    B_k=α_ksin(λ_k).

Note that the generated frequencies are independent of the index i, thus the hidden neurons share the same harmonics: sin(β_kx), cos(β_kx).
Since different linear combinations of the input frequencies may correspond to a single generated frequency, (<ref>) isn't (yet) the Fourier transform of the network's output. In other words, we could have two integer vectors k, l∈^m such that β_k=β_l. In Appex <ref>, we show how to aggregate those frequencies to obtain the final Fourier transform.

A natural issue regarding the frequency factory is the existence of B>0 such that α_k is small for k_∞≥ B.
We can use the formula of α_k to determine a bounding, showing that if W_∞<2 then the coefficients A_k and B_k decay exponentially as k_∞ grows.

  
    The magnitude of the amplitude α_k is bounded by
    ∏_j=1^m( |W_ij|2)^|k_j|1/|k_j|!.
    


Sec <ref> presents a bandlimit control mechanism during training based on Theorem <ref>.



§ INITIALIZATION AND FREQUENCY BOUNDING


This section presents an initialization and frequency control schemes for sinusoidal INRs.
Our initialization considers integer input frequencies (based on a Fourier series interpretation) and uses Theorem <ref> to study the spectral content of layer composition.
Then, we use Theorem <ref> to control the bandlimit of sinusoidal INRs during training.


 §.§ Initializing the input layer is a sampling in the spectral domain




[11]r0.17
     

     [b]0.17
         
    < g r a p h i c s >

         
     
        
        

First, we initialize ω with integer frequencies by setting ω_j ∈2π/p^d
and freezing them during training, where p∈^+ will be the INR's period.
This guarantees that the input neurons D are p-periodic.
Then, Theorem <ref> ensures that such periodicity is preserved by the layer activation. On the right, we show a comparison between SIREN and our initialization with p=2, both supervised only inside the red square (image from <cit.>).
This gives a proof of the empirical results about periodicity in <cit.>.
Additionally, (<ref>) implies that we can rewrite the INR as:




    f(x) =∑_k∈^mC,A_ksin(β_kx) + C, B_kcos(β_kx) + d





Since the generated frequencies β_k only depend on the frozen parameters ω, the training is responsible for learning the amplitudes of the sine-cosine series in (<ref>).








Assume that the ground truth has a bandlimit B.
First, note that we can sample ω directly on the cube [-B, B]^d, since (<ref>) indicates that the chosen frequencies appear in the spectrum of f. 

However, it also shows that the composition of layers produces an exponential number of new frequencies. These may extrapolate the cube
since they are integer combinations of the input frequencies.
Therefore, we propose to sample ω_j in 2π/p[-b, b]^d, with 0<b<B a bandlimit for the input frequencies. 
Indeed, as the hidden matrix  is initialized with small values (|W_ij|<2, see Sec <ref>), the magnitudes of A_k and B_k decrease as k_∞ increases.
This is a consequence of Theorem <ref>:



    |α_k| ≤∏_j=1^m( |W_ij|/2)/|k_j|!^|k_j|≤1/|k_1|!…|k_m|!  for   i=1,…,n.

Thus, the generated frequencies β_k with small k have more influence over the expansion (<ref>), mimicking  Fourier series theory.
This motivates us to initialize the input frequencies close to the origin; we found b=B/3 to work well with the experiments.
Fig <ref> shows that it is necessary, as higher b introduces noise.
However, in next section we avoid this problem by bounding the INR's spectrum.



Note that if ω_j is an input frequency, -ω_j is also present in the initialization. 

This is a consequence of sin(ω_j+φ_j)=cos(φ_j)sin(ω_j)-sin(φ_j)sin(-ω_j+π/2) being a linear combination of harmonics with frequencies ω_j, -ω_j.





Then, we only need to sample ω_j in half of the cube [-b, b]^d.
Also, from (<ref>) we have that the final bias d represents the amplitude of the frequency (0,0), hence we do not initialize it in ω.
Finally, since D()=sin(ω+φ), we initialize φ uniformly in [-π/2, π/2].












[11]r0.75
     
     
     

         
    < g r a p h i c s >

         

        INR spectrum during training. The input frequencies ω in green. During training new frequencies appear in a neighborhood of ω. The red dot indicates a high input frequency ω_j and the red arrows show the generated frequencies k_jω_j, k_j=1, 2, 3.
    

We now discuss an initialization of the frequencies ω∈^m× d.
First, note that (<ref>) indicates that only frequencies β_k with k small can have high amplitudes. Then we initialize ω such that it determines a sampling on the spectral domain.




Specifically, we split [-b, b]^d in regions L=[-l, l]^d of low and H=[-b, b]^d∖L of high frequencies, l∈(0,b).


When ω_j∈L, small variations in k_j results in a frequency near β_k. Indeed, Fig <ref>b shows how low frequencies (green square) introduce frequencies around the input frequencies (Fig <ref>a).
On the other hand, if ω_j∈H then β_k+e_j is far away from β_k; we observe this in Fig <ref>b-e, where ω_j is the red point and the arrows show the generated frequencies k_jω_j.














[8]r0.45


     [b]0.22
         
    < g r a p h i c s >

         
     
     [b]0.22
         
    < g r a p h i c s >

         
     
     

    Choosing ω as the cartesian product of the odd frequencies w/wo (1,0), (0,1).
    

When sampling frequencies in L, we include (1,0) and (0,1). This guarantees that any frequency could be generated and prevents the appearance of sub-periods (see Appex <ref>).
Fig <ref> shows an example in which this condition is violated, leading to an unappropriated reconstruction. 
For the higher frequencies in H, we sample in a grid-like fashion, uniformly covering the half square K. 








 §.§ Frequency bounding and hidden layer initialization






Sinusoidal INRs can represent detailed signals due to the exponential number of frequencies generated by the sine activation.
However, they could manifest as noise in the reconstruction.
Here, we use Theorem <ref> to introduce a bounding 
scheme to control the INR bandlimit during training.
[13]r0.55

     [b]0.55
         
    < g r a p h i c s >

     
     

    Training for 6000 epochs with b=30 and bounds c=0.1, 0.2, 0.5. The bound over the spectrum restricts the appearance of high frequencies.
    

Then, we use the relationship between the hidden matrix , generated frequencies β_k, and ω to define the bounding by blocks.
Next, we initialize  in a range defined by the bounds, improving performance.
Finally, we define a new architecture that allows us to learn the bounds.
Precisely,
using Theorem <ref>, we have that the amplitudes of the generated frequencies are limited by:

    |α_k|≤(_i_∞2)^k_11/∏_j=1^m|k_j|!.



This implies that if _∞≤2 then the amplitude of a generated frequency β_k rapidly decreases when the coordinates of k grow. 
Thus, forcing _∞≤2 during training reduces the appearance of  β_k with high |k|_∞ in the reconstruction.
Therefore, we use (<ref>) to define a bounding scheme that controls the INR bandlimit so that the generated frequencies saturate around small multiples of the input frequencies. 
It simply clamps the coefficients W_ij by a bound parameter c∈(0,2]. 

This procedure is applied at each epoch, enforcing _∞≤c.

Fig <ref> presents the results of bounding  with different bounds c.


We initialize  with values in [-c, c] using a normal distribution with mean 0 and standard deviation c/3.
Observe that 
SIREN <cit.> also controls the bandlimit but only at initialization, since the weights of  are sampled in (-√(6/m), √(6/m)). 
In other words, since _∞<2 (for m>1) at initialization, they do not introduce high frequencies at the beginning of the training. Additionally, we note a faster convergence when training with bounding using our initialization (see Appex <ref>).




In Sec <ref> we split the input frequencies ω in two sets: the low (ω_j_∞≤l) and high (ω_j_∞>l) frequencies.
Now, observe that each input frequency ω_j is associated with the j-column 
 of .
Then, by bounding the weights of  related to high frequencies, we restrict the amplitudes of β_k that could exceed the Nyquist limit B. Conversely, since low frequencies are used to span around ω, we consider a higher bounding for their columns.
Precisely, we define two parameters c_1 and c_2 to bound the weights W_ij corresponding to ω_j in L and H, respectively,





    ω_j_∞≤l →  W_ij=(W_ij, [-c_1,c_1]);     ω_j_∞>l →  W_ij=(W_ij, [-c_2,c_2]).





Analogously, we initialize  such that each column is normal distributed with mean 0 and standard deviation c_l3, with c_l the bound corresponding to that column.


Finally, we present a scheme to learn the bounding parameters c_j during training. For this, we bound each j-column W_*j of the hidden matrix W using a tanh activation followed by multiplication with a trainable bounding parameter c_j, that is, tanh(W_*j)c_j. Clearly, the resulting column is bounded by c_j. In other words, we are replacing the hidden layer 
sin(Wx + b) by





    sin(tanh(W)Cx + b)  whereC is a diagonal matrix with each jj-entry being c_j.





To prevent the bounds from getting too big, we use Theorem <ref> to define a regularization ℒ_reg=∑c_j.
Otherwise, the hidden weights virtually grow unbounded and the INR overfits as in the usual training.







§ ADDITIONAL EXPERIMENTS AND COMPARISONS






This section presents experiments regarding the initialization and bounding of frequencies of an INR f.
Our code is based on the MRNet <cit.> repository, using the images of the Kodak dataset resized to [-1,1]^2. 
Since our architecture is determined by the hidden matrix ∈^n× m, we use n and m to define f. 
We initialize the network using the scheme given in Sec <ref>.

We use Adam's optimizer <cit.> with a fixed learning rate 1e-4.
For the quantitative evaluations, we use the mean and standard deviation of the PSNR evaluated on the test set (10% random pixels).
Besides comparing the reconstruction quality of our INR f, we also evaluate the resulting analytical gradient ∇ f to show that our method adds high order regularity in the signal reconstruction. All experiments were performed in a GPU NVIDIA TITAN X (Pascal) with 12 GB.





 §.§ Initialization






[10]r0.5



    
    2-7
    1l|     2c|30%     2c|50%     2c70%                            
 1-7 l
                              Signal            Grad            Signal            Grad            1cSignal     1cGrad 
 21     35.2     26.9     35.2     26.6     35.0     26.0 
 42     35.1     26.3     34.8     24.6     33.8     22.2 
 64     34.2     25.2     33.3     21.6     30.8     15.0 
 
    

    Training an INR with 176K params. for 400 epochs. 
    Results indicate that it is better to sample more ω_j∈L and choose a smaller l.
    
    

First, we compare our ω initialization against the uniform distribution. We obtained a 4dB improvement of PNSR on average (with a std of 1.6dB). 
We also present an ablation over the parameter l which splits the square [-b, b]^ 2 in regions of low (L) and high (H) frequencies (see Sec <ref>).
Here we consider b=85 and l=21, 42, 64, 

and sample ω with 30%, 50%, and 70% of coordinates in H.
Table <ref> shows the performance of the INR, with the signal (grad.) mean std of 1.9dB (3.1dB).

[7]r0.5


     [b]0.5
         
    < g r a p h i c s >

     
     

    Reconstructions with large l present artifacts (ovefitting) on the 10% unsupervised points. 
    
    
    

This experiment considered all the images in the dataset.
Observe that all values of the row l=64 have less PSNR, showing that an INR with very high frequencies performs worse even for small percentages (30%). 
Similarly, the performance worsens when we increase the number of high frequencies (last column).



We now test the influence of high frequencies by training an INR of size m=360, n=512 with b=256 over 90% of the ground truth pixels. We vary the parameter l=60, 120, 220 that defines low frequencies.


Increasing the number of higher frequencies, the INR suffers from noise and overfitting. This can be noted in the reconstruction on the unsupervised pixels and gradients in Fig <ref>.



 §.§ Bounding




In Sec <ref> we presented a scheme to bound the INR spectrum.
Here, we show additional experiments to demonstrate that this improves the signal and has a great benefit over the unsupervised gradient reconstruction.
Then, we give a comparison between our fixed and trained bounding schemes.


[12]r0.5


         
    < g r a p h i c s >

    
    Comparison of training wo/w bounds. Observe that training using bounds preserves high order information of the signal.
    
    
    

Fig <ref> shows a comparison between wo/w our bounding scheme to reconstruct an image of size 512^2. To demonstrate the impact of our scheme, we sample the input frequencies using b=256 and l=100. 
Without bounding (left), we effectively reconstruct the signal at the supervised pixels (30.8 dB), but it results in a noisy gradient (15.8 dB). That is, we are overfitting the signal. On the other hand, when bounding the hidden weights, we observe significant improvements in both the signal (35.1 dB) and its gradient (27.8 dB).
Our bounding scheme acts like a filter in both signal and gradient reconstruction.

[12]r0.5


    
    4-7
                                                                2lsignal     2|lgrad 
 1lb     1lc_1     c_2     1|lNB       WB       1|lNB        WB   
 1l256      1l1.5        0.05      1|l32.6           34.8        1|l14.5          25.8    
 1l190      1l1.5       0.05       1|l34.0           35.9       1|l18.5           26.1  
 1l190      1l1.5       0.2       1|l34.0            34.4      1|l18.5           21.0    
 
    
    

        Training using our bounding scheme (WB) improves the signal and grad. reconstruction. The choice of c_2 greatly impacts the grad. fidelity. The std of the difference between each image signal (grad) quality wo/w bounding (NB) is 1.4dB (1.6dB) on average. Evaluations performed on the test set (10% of pixels).
    

We perform a quantitative comparison, varying the parameters b and c_2, to numerically test our scheme on the dataset. Table <ref> shows an improvement in the signal for all cases. Furthermore, the unsupervised gradient of the image is better learned when using bounds, showing an average increase of 9.6dB against the standard training.
Also, the last two lines of the table show the impact of the parameters c_1, c_2 over the signal and gradient reconstructions. 
We consider training with c_1>c_2 since we are prioritizing low input frequencies in the spectrum generation.


Otherwise, this may imply a generation of high frequencies, possibly bypassing the Nyquist limit and resulting in a worse gradient reconstruction.







[13]r0.69


         
    < g r a p h i c s >

    
    

    Comparison of training with fixed/learned bounds. Learning the bound of each frequency allows an adjustment of the signal spectrum resulting in better signal/gradient reconstructions. Each blue point in the diagram represents a pair (ω_j, c_j).
    
    
    

Next, we show how we can learn each bound c_j. 
As described in Sec <ref>, we can incorporate a learnable c_j for each j-column of the hidden matrix through a modified layer (<ref>). 
Fig <ref> shows a comparison between our fixed bound scheme using initial un-tuned c_1=1.0 and c_2=0.6 (left) and the learned bound scheme (middle). 


As expected, training lowers the bounds for high frequencies (H) while maintaining or increasing the bounds of low frequencies (L). Thus, we achieve better reconstruction in both the signal and gradient.
In Fig <ref> (right), we note that low frequencies (left of the green line) have significant bounds, showing their importance during training.
Conversely, the bounds of frequencies in H rapidly decrease as they grow higher.
These schemes allow us to initialize with higher frequencies than B/3, accelerating convergence.







 §.§ Comparisons


We first compare our method with SIREN, which requires choosing the frequency parameter b,[Denoted as ω_0 in <cit.>.] a critical and often empirical task. Our approach uses b to control the generation of new frequencies. To demonstrate this, we perform a comparison by varying b as well as the number of input (m) and hidden (n) neurons.

First, consider the case where we have no information about the bandlimit of the underlying signal and can only rely on the Nyquist limit B. Fig <ref> presents a comparison between SIREN (1^∘ row) and our method (2^∘ row) with b=B. Note that SIREN produces very noisy reconstructions during training, while ours learns faster and does not suffer from noise. This is due to our bounding scheme, which limits the generation of high frequencies and provides more control during training (Sec <ref>). In contrast, SIREN initializes its parameters such that the spectrum is initially bounded but does not maintain this guarantee during training.










Next, we consider a tuned b for SIREN to show that even under this condition, our method provides a better training scheme for sinusoidal INRs. Fig <ref> presents the comparison varying the number of input (m) and hidden (n) neurons. We consider the cases m<n (1^∘ row) and m>n (2^∘ row). Note that in both cases, our method performs better in the signal/gradient reconstructions. Also, we observe more robust training using our scheme, as seen in the loss functions in the 3^∘ column.













The experiments above showed that initializing SIREN with a high bandlimit (b= B) results in overfitting (Fig <ref>). Conversely, our method effectively learned the lower frequencies first and performed better than SIREN with the same configuration and a tuned b (Fig <ref>). Next, we numerically evaluate these experiments, providing a quantitative comparison with SIREN and BACON on the Kodak dataset.

We compare different architectures (m, n) and spectrum bandlimits b, evaluating the results in terms of signal and gradient PSNR over the supervised (Train) and unsupervised (Test) pixels. Table <ref> presents these evaluations. As expected, SIREN performs well with b=B/3. In this case, our method is perceptually similar to SIREN but improves gradient reconstruction and consistently outperforms BACON.

When b=171, our method demonstrates the best quality in both signal and gradient, showing greater robustness to spectral noise and superior performance even with b≥B/3. Finally, with b=256, our method surpasses SIREN in all cases. For BACON, our method achieves better signal reconstruction, with comparable gradients. However, the following experiment shows that BACON's reconstruction has visible defects in this scenario.

Note that we do not supervise the training with gradient information.


















Lastly, Fig <ref> presents a qualitative comparison between our bounding scheme and BACON.
We consider the three initial bandlimits b=85, 171, 256 as in Table <ref>.
We note that BACON truncates the spectrum in a similar way as the 0 hidden layer INR (see Fig <ref>), 
resulting in ringing artifacts.
Moreover, observe that it presents a loss of contrast as the bandlimit increases.
Conversely, ours operates as a soft filter providing better reconstructions in all cases.









§ CONCLUSIONS AND LIMITATIONS







We presented a study based on Fourier theory for sinusoidal INRs and used it to propose new schemes for both initialization and spectral control. 
Positively, our theoretical contributions generalize to different domains and aim to provide a better understanding of sinusoidal INRs for all applications. 
While we limited our experiments to images to demonstrate our theoretical claims; future works include using our schemes for other signals.
We also have focused our studies on a single hidden layer architecture leaving the investigation of deep architectures for a follow-up work.
In that sense, we are certainly considering future research aiming to extrapolate our results to general architectures and domains. Thus, we consider this work as a first step to unravel the powerful expressiveness of sinusoidal INRs.











plainnat











§ PROOFS OF THEOREMS



In this section, we prove Theorems <ref> and <ref>.
We consider the INR input to be 1D since the general case is analogous. 
Let f(x)= L∘S∘D(x) be a sinusoidal INR 
with S(x) = sin(Wx + b), where W∈^n× m is the hidden matrix, and b∈^n the bias.
The input layer D(x)=sin(ωx+φ) projects the input x into a list of harmonics with frequencies ω=(ω_1,…, ω_m)∈^m and shifts φ∈^m.
Finally, L is a linear layer.

The ith hidden neuron can be written as follows: 

    h_i(x)
    =sin(
        ∑_j=1^m W_ijsin(
            ω_jx+φ_j
        )+ b_i
    )


Note that, h_i receives a list of m input neurons D_j(x)=sin(ω_jx+φ_j) that are combined with the weights W_ij and activated by sin. 

Before presenting the expansion of h_i (Theorem <ref>), let us recall the (Fourier) expansion of a neuron with width 1 and no bias <cit.>:

    sin(asin(x))=∑_k∈ oddJ_k(a) sin(kx),  with   J_k(a)=∫_0^πcos(kt-asin(t)) dt.

The functions J_k are the Bessel functions of the first kind.
The expansion of h_i generalizes (<ref>). 







Its proof consists of verifying (<ref>) as well as a similar formula using the cosine as the activation function:

    cos(
        ∑_j=1^m W_ijsin(
            ω_jx+φ_j
        )+ b_i
    )
    = ∑_k∈^mα_kcos(
        k̇ωx +φ+ b_i
    ).

Now, we present the proof of Theorem <ref>, that gives the harmonic expansion (<ref>) of the neuron h_i,

    h_i(x) = 
                ∑_k∈^mα_k sin(β_k x + λ_k),   with  β_k = ∑_j k_j ω_j,  λ_k=∑_j k_jφ_j + b_i,  α_k=∏_j=1^mJ_k_j(W_ij).


For simplicity, we denote W_ij by a_j.

The proof is by induction in m. For the base case m=1, we prove sin(a sin(y)+b)=∑_k∈ J_k(a) sin(ky+b) with y=ω x+ φ.
For this, we use the expansion in (<ref>) and its cosine analogous expansion cos(asin(y))=∑ J_l(a) sin(ly), here the sum is over the even numbers. Thus, applying the trigonometric identity sin(a+b)=sin(a)cos(b)+cos(a)sin(b) we obtain:

    sin(a sin(y)+b)   =sin(a sin(y))cos(b)+cos(a sin(y))sin(b)
       =∑_k∈ oddJ_k(a) sin(ky)cos(b) + ∑_l∈ evenJ_l(a) cos(ly)sin(b)
       =∑_k∈ oddJ_k(a) sin(ky+b) + ∑_l∈ evenJ_l(a) sin(ly+b)
       =∑_k∈ J_k(a) sin(ky+b).

In the third equality we combined the formula sin(a)cos(b)=sin(a+b)+sin(a-b)/2 and the fact that J_-k(a)=(-1)^kJ_k(a) to rewrite the summations.
The proof of the formula using the cosine as an activation function is similar.

Assume that the formulas hold for m-1, with m>1, we prove (<ref>) holds for m (the induction step). Again, we denote y_j=ω_j x+ φ_j and b_i as b for simplicity.


    sin(∑_j=1^m a_jsin(y_j)+ b)   =sin(∑_j=1^m-1 a_jsin(y_j)+ b)cos(a_msin(y_m))
       +cos(∑_j=1^m-1 a_jsin(y_j)+ b)sin(a_msin(y_m))
        
       =∑_l∈^m-1,  k∈ evenα_lJ_k(a_m)sin(l̇y+ b)cos(ky_m)
       +∑_l∈^m-1,  k∈ oddα_lJ_k(a_m)cos(l̇y+ b)sin(ky_m)
       =∑_k∈^mα_ksin(k̇y+ b)

We use the induction hypothesis in the second equality and an argument similar to the one used in the base case to rewrite the harmonic sum. Again, the cosine activation function case is analogous.


<cit.> presented a similar formula for MLPs activated by polynomial functions.
While it is evident that the sine function can be approximated by a polynomial using Taylor series, our formula requires no approximations.
In addition to providing a simple proof, we also derive the analytical expressions for the amplitudes. These expressions enable us to compute upper bounds (Theorem <ref>) for the new frequencies α_k in terms of k and .
Next, we prove such inequality,

    α_k=∏_j=1^m( |W_ij|2)^|k_j|1/|k_j|!.



Again, we denote W_ij by a_j to simplify notation. Theorem <ref> says that
α_k=∏_j=1^mJ_k_j(a_j).
To estimate an upper bound for this number, we use the following inequality <cit.>, which gives an upper bound for the Bessel functions J_i(a).

    J_i(a)<(a/2)^i/i! ,    i>0,   a>0.

Observe that this inequality also holds for a<0 since J_i(-a)=J_i(a).
Therefore, replacing (<ref>) in ∏_j=1^mJ_k_j(a_j) and using J_-i(a)=J_i(a) results in the desired inequality.




§ TOWARDS COMPUTING THE FOURIER SERIES OF A PERIODIC INR



Here, we consider the INR f to have ^2 as domain, however, it could be easily extended to other domains.
Observe that a given frequency 𝔣 may appear several times as generated frequencies β_k=k^⊤ω with different coefficients k in the expansion (<ref>).
Here, we provide a characterization of all k∈^m such that k^⊤ω=2π/p𝔣, where p is the period 
of the sinusoidal network. This leads to an algorithm to compute the Fourier series of the periodic INR f.

To guarantee that a sinusoidal INR f initialized with ω= 2π/p[f_1, f_2]∈2π/p^m×2 can represent an arbitrary image, we need to determine when it can reconstruct a given frequency 𝔣∈^2.
This is equivalent to find a solution k∈^m to the system of Diophantine equations

    [f_1, f_2]^⊤k = 𝔣  for  𝔣∈^2.



Let B = U[f_1, f_2]^⊤V be the Smith normal form of [f_1, f_2]^⊤. Assuming e=U𝔣
such that e_i/B_ii∈ for i=1,2, then all integer solutions of (<ref>) have the form
V[[ e_1/B_11 e_2/B_22      l_1      ...    l_m-2 ]]^T
with l_1,..., l_m-2 arbitrary integers <cit.>.

Now, recall that we are considering ω_1=2π/p(1,0) and ω_2=2π/p(0,1), thus the Smith normal form and unimodular matrices U, V are given by,  

    B = [𝕀2 | 0_2× m-2],  U=I_2,   and  V = 
    [                                             𝕀2                                                -f_13    ...     -f_1m
    
    -f_23    ...     -f_2m;                                      0_m-2 × 2                                                                                          𝕀m-2 ].

Then e=𝔣 and the divisibility condition is satisfied since B_ii=1. Therefore, any frequency 𝔣∈^2 can be represented by f, and the integer solutions of (<ref>) are given by


    K(𝔣) = 
    [ 𝔣_1 - ∑_k=1^m-2 l_k f_1,k+2; 𝔣_2 - ∑_k=1^m-2 l_k f_2,k+2;                         l_1;                           ⋮;                       l_m-2 ]: l_1, ..., l_m-2∈.


Using K(𝔣) and (<ref>), we obtain the Fourier series of f by aggregating all coefficients associated with each frequency 𝔣∈^2. That is, the Fourier coefficients are given by 
A_𝔣=∑_k∈K(𝔣)C,A_k and B_𝔣=∑_k∈K(𝔣)C,B_k. 

To find a similar formula for the general case of ω∈^m× d, we must consider ω_i=e_i for i=1,...,d. Then, the aggregation of coefficients is analogous.



§ ON THE SUB-PERIODICITY OF SINUSOIDAL INRS


Observe that initializing a sinusoidal INR f using input neurons with period p implies that f is also periodic with period p (Theorem <ref>).
However, the initialization could force the INR to have sub-periods. In that case, it would never be able to fit a signal with fundamental period p.
We derive a condition to avoid such a problem.
First, recall that Theorem <ref> says that a hidden neuron 
h(x) =
sin(
    _i sin(
        ω+φ
    ) +b_i
), with ω = 2π/p[f_1, f_2] for some f_1, f_2∈^m, can be expressed as:

    h(x) = ∑_k∈^mα_ksin(β_kx + λ_k)

with frequencies and phase-shifts given by β_k=k^⊤ω and λ_k= k^⊤φ+b_i. Therefore,
β_kx = 2π/p(k, f_1x + k, f_2y).
Now, suppose that h has sub-period p/q in x-axis and p/s in y-axis (q,s∈^+), that is, h(x,y)=h(x+p/q, y+p/s).
However, (<ref>) implies

    h(x+p/q, y+p/s) = ∑_k∈^mα_ksin(β_kx + λ_k + 2πk, f_1/q+f_2/s).



Thus, since sin(x+2π k )=sin(x) for k∈, we have that Expansions <ref> and <ref> are equal only if k, f_1/q+f_2/s∈ for all k∈^m. Therefore, h (and consequently f) has sub-periods only if there exists q>1 or s>1 such that f_1/q+f_2/s∈^m.

An important consequence of our initialization is that the addition of the input frequencies (1,0), (0,1) to [f_1, f_2] prevents the appearance of sub-periods.
Indeed, we define the first elements of [f_1, f_2] as _11=1, _21=0, _12=0, and _22=1.
Hence, this implies that
_11/q+_21/s=1/q∈ if and only if q=1, and _12/q+_22/s=1/s∈ if and only if s=1.

Fig <ref> illustrates three cases where the choice of the input frequencies ω results in a bad reconstruction: 
(a) [f_1, f_2] with odd frequencies and period p=2; (b) [f_1, f_2] defined by the cartesian product of {1, 10, 100} and p=2, and (c)  [f_1, f_2] with odd frequencies and period p=3.
Fig <ref>c shows that in the case p=3 there is an overlap of copies generated by the periodicity.
This suggests an inconsistency between the selected period p and the resulting period of f.
Adding the frequencies (0,1) and (1,0) solves this problem, see Fig <ref> in the main text.





§ EXTENSION OF THEOREM 1 TO DEEPER NETWORKS


This section extends Theorem <ref> to multiple hidden layers. By recursively applying the argument from the proof of the theorem, we can establish the generalization using induction. We demonstrate the case of an INR with two hidden layers for simplicity, however, the general case is analogous.

Let f(x) = L∘S_1∘S_0∘D(x) with S_i(x) = sin(W^(i)x + b^(i)). Then, a neuron of the last hidden layer can be written as 
h(x) = sin(
        asin(
            W^(0)sin(
                ωx + φ
            ) + b^(0))+ b^(1)), where a is a row of the hidden matrix ^(1).
Observe that by denoting y^(0)=W^(0)sin(           ωx + φ) + b^(0) we can apply (<ref>) as in the case d=1 as follows:

    h(x) 
         = sin(
            ∑_i=1^n a_j sin(
                y_i^(0)) + b^(1))
         = ∑_l∈^nα_l^(1)sin(
                l̇y^(0) + b^(1))


where α^(1)_l=∏_i=1^nJ_l_i(a_i). When replacing the term y^(0) in the sine of the last expression, it can be written as a composition of sines:

    sin(
            l, y^(0) + b^(1))
            = sin( 
            ∑_j=1^ml, _*j^(0)sin(
                ω_jx+φ_j
                ) + l, b^(0) + b^(1)) 
        = ∑_k∈^mα_k^(0)sin(
                k, ωx + k, φ + l, b^(0) + b^(1))

Here, α^(0)_k=∏_j=1^mJ_k_j(l, _*j^(0)).
The first equality is obtained by substituting y^(0) and aggregating the terms appropriately, the second one is given, again, by (<ref>) used in the proof of Theorem <ref>. Then, denoting
α_k, l = α_l^(1)α_k^(0) and λ_k, l = k, φ + l, b^(0) + b^(1)
we obtain the desired equality:

    h(x)
         = ∑_(l,k)∈^n× mα_k, lsin(
             k, ωx + λ_k, l)

Observe that several properties drawn from the case of an INR with a single hidden layer hold for the general case. First, the amplitudes α_k, l depend only on the hidden matrices, and still product of Bessel functions. On the other hand, only the biases of the model influence the shifts λ_k, l. This may imply that not all biases are necessary during training.


We emphasize that adding more hidden layers doesn't introduce new frequencies, as they are, once again, defined by k, ω. Then, deep sinusoidal networks just fine tune the amplitudes of the generated frequencies, which are completely determined by the choice of ω.



§ ADDITIONAL EXPERIMENTS




 §.§ Initialization in the half cube

[9]r0.13
    

    
    < g r a p h i c s >

    
    

We showed that initializing a frequency ω_i implies that its negative -ω_i also appears in the spectrum.
Here, we use this fact to avoid sampling duplicated frequencies in [-b, b]^2 to obtain a better reconstruction.  
We test the capacity of two INRs with the same architecture m=102, n=512, but with different initializations for their frequencies ω (blue and white dots in Fig <ref>a) and ω' (white/green dots in Fig <ref>b):

    ω =[(k,l)|   k,l∈{1, 3, 4, 7, 10, 20}]  and  ω' =[(k,l)|   (k,l)∈ω with  k>0] ⊔η.


Where η are additional frequencies (in green) sampled in the Cartesian product of the list [0, 1, 2, 3, 4, 7, 10, 20]. 
Fig <ref>c shows the PSNR of the resulting networks with respect to the iterations.
The results show that adding the new frequencies η results in a slight but consistent improvement in the PSNR during training.




 §.§ Initialization of hidden layer


In Section <ref> we proposed an initialization for the hidden matrix . The following experiment shows that an initialization that considers the bounding limit for  may grant a faster convergence.
In Fig <ref>, we initialize the INRs of size m=416, n=1024 with bandlimit b=82 and bounds c_1=1.0, c_2=0.2. The INR trains during only 10 epochs to observe the speed of convergence and fits an image with resolution 1024^2.
Observe that the INRs that considered the bounds to initialize their hidden layer  (2^∘ row) present a better reconstruction. This can also be observed in the zoom-ins of images, where our method presents more details.






 §.§ Learned bounds

In Section <ref> we presented a novel scheme to bound the sinusoidal INR during training. Furthermore, we also proposed an architecture that allows us to learn those bounds during training. In this section, we analyze the coherence of the resulting bounds as we vary the signal bandlimit.

We define several INRs with size m=n=416, initial bounds with value 0.5 and bandlimits b = 34, 85, 171, 256. Then, we fit them to images with resolution 512^2 using our scheme of learnable bounds. Fig <ref> shows the bounds trained after 400 epochs. Note that all plots have a green, dashed line that divides frequencies into low (L) and high (H).





Observe that in Fig <ref>a the low frequencies are saturated around a height of 0.37, showing that their use is reduced from the initial 0.5. Conversely, the amplitude of frequencies in H increases as the frequency gets higher. This shows that the INR finds it necessary to increase the amplitudes of high frequencies for a good reconstruction. 
For the case when b=85 (Fig <ref>b), training started discriminating which of the low frequencies contributes more to the reconstruction; this is observed by the spreading of bounds. As for the high frequencies, we observe a more uniform behavior, which may indicate that all frequencies may be contributing the same amount of information during training. This is also an indicator that the parameter b=85 (B/3) is a good bandlimit to train the INR.

Now, in Fig <ref>c-d, we can observe even more extreme behaviors. Low frequencies exhibit multiple distinct bounds, while high frequencies are suppressed to prevent noise generation.
The patterns depicted in the figures align with our theory, confirming that we are effectively learning a bound that serves as a mechanism for spectral control.



