TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model
Defu Cao Wen Ye Yizhou Zhang Yan Liu



Review and Novel Formulae for Transmittance and Reflectance of Wedged Thin Films on absorbing Substrates
    [
    
========================================================================================================




§ ABSTRACT 


With recent advances in building foundation models for texts and video data, there is a surge of interest in foundation models for time series.  A family of models have been developed,  utilizing  a temporal auto-regressive generative Transformer architecture, whose effectiveness has been proven in Large Language Models.  While the empirical results are  promising,  almost all existing time series foundation models have only been tested on well-curated “benchmark” datasets very similar to texts. However, real-world time series exhibit unique challenges, such as variable channel sizes across domains, missing values, and varying signal sampling intervals due to the multi-resolution nature of real-world data. Additionally, the uni-directional nature of temporally auto-regressive decoding limits the incorporation of domain  knowledge, such as physical laws expressed as partial differential equations (PDEs). 
To address these challenges, we introduce the Time Diffusion Transformer (), a general foundation model for time series that employs a denoising diffusion paradigm instead of temporal auto-regressive generation. leverages the Transformer architecture to capture temporal dependencies and employs diffusion processes to generate high-quality candidate samples without imposing stringent assumptions on the target distribution via novel masking schemes and a channel alignment strategy.
Furthermore, we propose a finetuning-free model editing strategy that allows the seamless integration of external knowledge during the sampling process without updating any model parameters.
Extensive experiments conducted on a varity of tasks such as forecasting, imputation, and anomaly detection, demonstrate the  effectiveness of TimeDiT. 






§ INTRODUCTION




Time series analysis is pivotal in a diverse set of AI applications, such as natural science <cit.>, social science <cit.>,sustainability <cit.>,health <cit.>, etc. These applications root in diverse domains  <cit.>, leading to time series with various distributions <cit.> and a divers set of analysis tasks, such as forecasting <cit.>, imputation <cit.>, anomaly detection <cit.>, etc. Even though considerable progress has been made in developing specialized models optimized for specific scenarios and individual tasks, an open question remains: Can a single time series foundation model excel across domains?  Recent initiatives have explored the possibility of universal time series models on zero-shot setting <cit.>, drawing inspiration from large pre-trained language models in natural language processing(NLP) and computer vision(CV), such as GPT<cit.>, CLIP <cit.>, which are known for their robust transfer learning capabilities. However, due to the fundamentally different semantics between text/images and time series data, the unique challenges of achieving a truly flexible and general-purpose time series model remain an open problem. 

Recently, the emergence of LLMs like GPT-4 <cit.> and LLaMA <cit.> suggests the potential for building time series foundation models enabling a general solution to handle multiple time series distributions. 
Previous attempts typically build upon the transformer backbone, which has achieved state-of-the-art performance on various time series tasks, particularly in modeling long-term dependencies.
However, the tokenization of time series data for transformers is especially sensitive to variations in data sources and sampling rates. Previous tokenization approaches with different schemes including token patching <cit.>; discretization tokens <cit.> and tokens based on time series features <cit.> have either fragmented the global information or have been constructed in a manner that inherently loses important information. Most, if not all, of them employ a channel independence strategy <cit.> or focus solely on univariate time series.  Channel independence strategy, though beneficial in certain contexts, often overlooks the complex inter-temporal and cross-feature dependencies in practical applications and thus presents an opportunity for optimization <cit.>. 

Moreover, compared with texts and images, time series exhibit unique characteristics such as missing values <cit.>, irregular sampling <cit.>, multi-resolution <cit.>, etc. 
To address these challenges, a foundation model for time series must be capable of demonstrating flexibility across different scales to handle diverse inputs with varying distributions. However, these unique natures and challenges are not covered by the popular well-curated benchmark datasets<cit.>. As a result, most existing works, which are developed and evaluated primarily on these datasets, may not fully address the complexities encountered in real-world time series applications. In addition, time series processes are often governed by underlying physical principles <cit.> and can be guided by domain-specific textual information <cit.>. However, integrating these diverse sources of information into a unified model poses further challenges, as the model must effectively leverage the relevant physics context while adapting to the unique characteristics and distributions of each domain. Addressing each of these issues requires innovative approaches in data preprocessing, model architecture, and training strategies to create models that can seamlessly handle the diverse and complex nature of time series data.

Current approaches to time series modeling lack a unified framework for handling the aforementioned diverse and imperfect data inputs, frequently prioritizing performance on well-curated benchmarks over addressing real-world challenges. Diffusion models, such as DDPM <cit.>, offer a promising solution by framing data generation as a series of conditional transformations, effectively recasting density estimation as sequential reconstruction. Unlike autoregressive methods that generate future tokens sequentially, diffusion models can directly produce high-quality samples through a reverse denoising process. This process can be analogized to solving partial differential equations (PDEs), allowing for the natural incorporation of physics-based knowledge. 
This capability, combined with transformers' strength in capturing temporal dependencies, presents an opportunity to develop a more versatile and robust time series foundation model. Such a hybrid approach could effectively address the complexities of real-world data while maintaining the flexibility to adapt to various forecasting tasks and data conditions.


In this work, we introduce —a diffusion transformer-based foundation model equipped with a standardized training pipeline for different shapes of input time series and tailored for diverse distributions and downstream tasks. TimeDiT leverages the Transformer architecture's inherent ability to capture temporal dependencies through its attention mechanisms, while also benefiting from the scalability that allows for increased model capacity crucial for complex time series tasks. By adopting a diffusion model approach, TimeDiT treats time series holistically, avoiding the error accumulation issues common in autoregressive solutions. The model incorporates a novel comprehensive mask mechanism that enables a single, unified foundation model to handle multiple tasks without additional modules or parameters. This design naturally addresses real-world challenges such as multi-resolution data and missing values. During the sampling stage, TimeDiT introduces an innovative strategy to incorporate physics knowledge as an energy-based prior, supported by theoretical guarantees. This approach guides the reverse diffusion process using physics-based constraints, including partial differential equations, resulting in generated samples that adhere to known physical laws and domain-specific requirements, thereby enhancing sample quality and model applicability across various scientific and engineering contexts.


TimeDiT's performance is rigorously evaluated through an extensive experimental setup encompassing over 20 diverse datasets from domains including traffic, weather, finance, etc. The model is benchmarked against more than 25 open-source baselines, ranging from linear-based models to diffusion-based models, transformer-based models, and other forecasting foundation models. These comprehensive experiments cover multiple challenging time series tasks, including in-domain and zero-shot probabilistic forecasting, imputation, anomaly detection, and synthetic data generation. TimeDiT demonstrated state-of-the-art or highly competitive results across these tasks, showcasing its effectiveness and efficiency as a foundation model for various time series applications. Notably, TimeDiT achieved new state-of-the-art CRPS_sum scores on the Electricity and Traffic datasets for probabilistic forecasting. In addition, the results on zero-shot experiments show that our model can be used as a foundation model even without fine-tuning, although fine-tuning may be necessary in some cases. Furthermore, TimeDiT's scalability and adaptability are evident in its ability to incorporate external knowledge, such as physical constraints, during the sampling stage. This feature allows for the generation of samples that better conform to known physical laws and domain-specific requirements.  This combination of state-of-the-art performance, adaptability across diverse tasks, scalability, and the ability to incorporate domain-specific knowledge positions TimeDiT as a powerful and versatile foundation model, capable of addressing a wide spectrum of time series challenges and opening new avenues for advanced time series analysis across various fields.


In summary, our contributions can be summarized as three unfolds:


    
  * We introduce TimeDiT, a novel diffusion transformer-based foundation model for time series analysis. By combining the strengths of diffusion models and transformers, our approach offers a flexible architecture adaptable to various downstream tasks. The model incorporates a comprehensive mask mechanism for reconstruction pretraining and task-specific fine-tuning, ensuring a standardized training pipeline capable of handling diverse input shapes and distributions.
   
    
  * Unlike autoregressive approaches, TimeDiT addresses real-world challenges in time series data by directly processing multivariate inputs and employing a denoising process to generate cohesive target time series. This method effectively handles issues such as missing values and multi-resolution data. In addition, TimeDiT can generate time series that adhere to known physical laws and domain-specific requirements, enhancing its applicability in scientific and engineering contexts.
   
    
  * Evaluated on over multiple datasets across different domains and tasks, TimeDiT achieves state-of-the-art or competitive results. It excels in probabilistic forecasting, imputation, anomaly detection, and data generation, showcasing its versatility as a foundation model in both in-domain and zero-shot settings.
    





§ RELATED WORK



  
General Purpose Time Series Model
In the past decades, researchers have excelled in designing sophisticated models for specific time series analysis tasks <cit.>. However, in recent years, the emergence of large language models has inspired the development of general-purpose time series models <cit.> and the field of time series has seen tremendous exploration efforts towards foundation models.  <cit.> simply encoded time series as strings while <cit.> converted time series into language representations by alignment. <cit.> and <cit.> further incorporated decomposition technique and prompt design and generalizes to unseen data and multimodal scenarios. <cit.> worked towards foundation model from a probabilistic perspective but only considered univariate time series only which rarely appears in real-life. Additionally, many studies started to follow a two-stage training paradigm of pretraining and finetuning <cit.>. However, these works mainly focused on the forecasting task only <cit.>. <cit.> first adapted GPT2 as a general-purpose time series analysis model and extended it to various time series tasks. <cit.> leveraged VQVAE as a tokenizer for transformer to handle time series tasks and <cit.> employed a scaling and quantization technique to embed time series. For more detailed literatures of the general-purpose time series model, please refer to recent surveys and position paper <cit.>





  
Diffusion models for Time Series
Despite the growing interest of diffusion models in various scenarios <cit.>, the use of diffusions in time series analysis is less explored compared to pre-trained language models and transformers.  Most existing studies also focused solely on forecasting and the choice of backbone model also varies among VAE<cit.>, RNN<cit.>, and transformer. CSDI <cit.> utilized diffusion model for time series imputation. <cit.> incorporated decomposition into diffusion model to improve interoperability. 
Although  <cit.> build a diffusion pipeline for multiple tasks with refinement, they still train different models for each task. To the best of our knowledge, there has been no exploration of leveraging unified diffusion models for a comprehensive set of time series tasks yet. Please refer to  <cit.> for a comprehensive literature review on diffusion models for time series analysis. 



§ PRELIMINARIES



 §.§ Diffusion Model

In recent years, diffusion models have emerged as a promising approach in generative modeling. A diffusion process is a Markov chain that incrementally adds Gaussian noise to data over a sequence of steps, effectively destroying  the data structure in forward process and destroying the data structure in backward structure.


The forward process adds noise to the data 𝐱_0 over a series of timesteps t according to a variance schedule β_t, resulting in a set of noisy intermediate variables 𝐱_1, 𝐱_2, …, 𝐱_T. Each subsequent 𝐱_t is derived from the previous step by applying Gaussian noise:

    q(𝐱_t |𝐱_t-1) = 𝒩(𝐱_t; √(1 - β_t)𝐱_t-1, β_t 𝐈)




The reverse process aims to denoise the noisy variables step by step, sampling each 𝐱_t-1 from the learned distribution p_θ(𝐱_t-1|𝐱_t). This distribution, modeled by a neural network parameterized by θ, approximates the Gaussian distribution:

    p_θ(𝐱_t-1|𝐱_t) = 𝒩(𝐱_t-1; μ_θ(𝐱_t, t), Σ_θ(𝐱_t, t))

By iterating this reverse process from t=T down to t=0, the model gradually reconstructs the original data from noise. The reverse process learns to predict the mean and covariance of each intermediate distribution, effectively approximating the original data distribution.






§ METHODOLOGY


In this section, we present our main contributions: the proposed foundation model, , a diffusion model with transformer backbone designed for multiple time series tasks, along with uniform masking strategies and incorporation of physics knowledge and textual information as an extension. We first outline the uniformed problem setting for multiple down-stream tasks and offer an in-depth examination of the model architecture. Subsequently, we delve into the training pipeline with mask strategies, which help to build the training scheme in self-supervised learning for time series. Next, we present how to incorporate external information to improve the model's performance during both the training and inference stages. By doing so, can generate samples that better conform to real-world requirements and enhance its performance on various downstream tasks. These extensions showcase the flexibility and adaptability of our proposed model, making it a powerful tool for a wide range of time series applications.



 §.§ Problem Definition


We denote a multivariate time series as 𝐗 = {x_i,j}∈ℝ^K × L, where K is the number of features and L is the length of the time series. Each individual entry x_i,j represents the j-th feature at time step l, for i ∈{1, …, K} and j ∈{1, …, L}. 
We define an observation mask 𝐌_𝐨𝐛𝐬 = {m_i,j}∈{0, 1}^K × L, where m_i,j = 0 if x_i,j is missing, and m_i,j = 1 if x_i,j is observed.
Let  𝐱_0^obs∈ X^obs denote the observed subsequence; 𝐱_0^tar denote the target subsequence of 𝐱_0^obs which could be forecast target or imputation target or the whole sequence depending on the task. Let  𝐱_0^con denote the unmasked partial observations in 𝐱_0^obs which acts like conditions for the masked area  𝐱_0^tar.  Let us use all subscripts of x to denote diffusion timestamp, and a subscript of 0 means no noise has been applied to the original data. Formally, the goal of our task is to approximate the true conditional data distribution given the conditional information
q_𝐗(𝐱_0^ta|𝐱_0^con)
with a model distribution p_θ(𝐱^tar_0 |𝐱^con_0), which can be calculated by a diffusion model with conditional information:

    p_θ(𝐱_0: T^tar|𝐱_0^con) := p(𝐱_T^tar) ∏_t=1^T p_θ(𝐱_t-1^tar|𝐱_t^tar, 𝐱_0^con), 𝐱_T^tar∼𝒩(0, 𝐈), where
    
    p_θ(𝐱_t-1^tar|𝐱_t^tar, 𝐱_0^con) := 𝒩(𝐱_t-1^tar ; μ_θ(𝐱_t^tar, t |𝐱_0^con), σ_θ(𝐱_t^tar, t |𝐱_0^con) 𝐈).

The mask mechanism 𝐌 plays a critical role in identifying the positions of 𝐱_0^con and 𝐱_0^tar. By leveraging these positional differences, our model adeptly adapts to various downstream tasks, including forecasting, imputation, anomaly detection, etc, within a unified framework. 



 §.§ Time Series Diffusion Transformer

Figure <ref> shows the overall framework of . We first establish the 𝐌_𝐨𝐛𝐬 and 𝐱_0^obs based on the given input from different distributions with multivariate sequences, missing value and multi-resolution by injecting placeholders to standardize the input shape across different time series, facilitating more efficient and consistent processing.
Then, the unified time series mask unit adapts to diverse time series scenarios and builds the 𝐱_0^con, 𝐌 and 𝐱_0^tar, with shape  ℝ^B × L × K,
 to help learn robust representations in a self-supervised manner by reconstructing the original sequence through denoising 𝐱_T^tar. After that, the embedding layer directly treats 𝐱_0^con and 𝐱_0^tar as  without any patching, as the diffusion process is designed to handle multivariate input and operate in a continuous token space.
 By preserving the integrity of the input time series, ensures that the model can effectively capture and utilize the rich information contained within the data. The block's attention mechanism is designed to autonomously learn cross-channel and temporal correlations through end-to-end training. 


  
Standardize Pipeline 
We introduce placeholders within the input sequences to standardize the input shape across different time series, accounting for varying channel numbers K and sequence length L. Specifically, we define the maximum channel number K_max such that any input with channel k<K_max is padded to have K^max channels while any input with more than K_max channels are segmented into ⌈k/K_max⌉ blocks of inputs where each block has K_max channels and undergoes independent processing. This segmentation allows our model to manage high-dimensional data efficiently, reducing computational overhead and maintaining relative positional integrity of the data and consistency across inputs. Additionally, for any input with sequence length less than the designated maximum length L_max, we pad the sequence in the front to achieve the desired length. This standardization is essential for establishing a uniform input structure that enhances processing efficiency and consistency.



  
Time Series Mask Unit 
We propose a unified time series mask mechanism that includes a variety of masks that seamlessly integrates with the model during self-supervised  task agnostic pre-training and task specific fine-tuning to cater to diverse time series scenarios. The time series mask unit generate four types of masks: reconstruction mask, stride mask, block mask, and random mask. Firstly, the task-agnostic pre-training aims to improve the overall time series representation by encouraging the model to learn robust and generalizable features from the input data. Secondly, the task-specific training is designed specifically for the most common downstream tasks, including forecasting and imputation, enabling the model to adapt to the unique requirements of each task. 


As shown in Figure <ref> right top, given 𝐱∈ℝ^K× L, the random mask M^R can be generated by:

    𝐌^R(x,r) =  1     z_i,j>r, z∈ℝ^K× L, z∼ Uniform(0,1) 
     0,     otherwise,

where r is the mask ratio. In addition, for task-specific training and inference, we allow the user to supply customized imputation masks that could simulate the naturally missing data and multi-resolution cases. 

Block mask 𝐌^B can be generated via:

    𝐌^B(x,l) =  1     j<L-l, 
     0,     otherwise,

where l is the predicted length. We can randomly select l during pretraining and use the designated prediction length during the finetuning and inference stage for specific experiment settings.

Stride mask 𝐌^S, a variant of 𝐌^B,  is placed intermittently within the series and is defined as follows:


    𝐌^S(x, n_blocks) = 
     
    1    ⌊j/b⌋ 2 = 0 
     
    0    otherwise


where n_block is the number of blocks into which the sequence is divided; b = ⌈L/n_blocks⌉ is the length of each block; j is the index of the sequence. 𝐌^Sis designed for task-agnostic pretraining further to enhance the model's representation ability by integrating information across non-contiguous parts of the series. 


In addition, reconstruction mask 𝐌^Rec = 0 is used for tasks including synthetic data generation and anomaly detection, where we can directly generate synthetic data or obtain an anomaly score for each temporal position based on the difference between original and reconstructed series. 


For the pretraining stage, we random select one conditional mask type from 𝐌={𝐌^R, 𝐌^S, 𝐌^B, 𝐌^Rec} for each instance. 's goal is to reconstruct the 𝐱^tar, defined as 𝐱_0 × (J - 𝐌) where J is unit matrix. The target sequence is the masked portion of the original sequence. In the finetuning and inference stage, the choice of mask is tailored to align with the specific requirements of the user. This flexibility allows to apply the most appropriate masking strategy based on the context of the task and application.




  
Condition Injection
Instead of following  <cit.> to integrate diffusion timestep and label information (texts in out case) through the layer normalization preceeding the attention block, we add the diffusion timestep and texts information to the target noise as these are universal information to the series. 

Given that utilizes a transformer-based architecture, a straightforward and intuitive approach is to include conditional information directly as part of the input sequence by concatenation as done in latent diffusion  <cit.>. However, we empirically found out that controlling mean and variance through layer normalization is a stronger form of conditional information injection. We incorporate the partial observations x_0^con through adaptive layer normalization to control the scale and shift of the x_0^tar. This design choice is motivated by the face that the scale and shift of the partial observations are more relevant to the target observations. This integration can be expressed as 

    AdaLN(h,c) = c_scaleLayerNorm(h) +c_shift

where h is the hidden state and c_scale and c_shift are the scale and shift parameters derived from the partial observations. We perform temporal attention in the self-attention block to capture the temporal dependency within the input. 



  
Inference   We perform pre-training and finetuning for forecasting and imputation. We exclude the pre-training process from anomaly detection and synthetic generation because the two tasks are very dataset specific and do not necessarily benefit from learning distributions beyond the target dataset. Let n represent the number of samples generated for each prediction, which we set to n=10 (n=30 for forecasting tasks) in our experimental setup at inference time. We use the median of these n predictions as the final prediction, providing the added benefit of obtaining a confidence interval for 's predictions. To prevent channel padding from affecting the generated samples, we mask out the invalid channels during sampling at each diffusion timestep so that does not falsely treat information in the non-valid channels as meaningful information. Padding is applied at the beginning of the temporal dimension to ensure that the most relevant information remains at the end, thereby mitigating the effect of padding.



 §.§ Physics-Informed  TimeDiT




Physics principles are fundamental in shaping the evolution of temporal signals observed in real-world phenomena, such as climate patterns and oceanographic data. Therefore, it is essential to integrate physical knowledge into foundational time series models. In this section, we propose a strategy to incorporate physics knowledge as an energy-based prior for during inference, which iteratively refines the reverse diffusion process. By guiding the denoising process during inference with gradients derived from physical laws represented by partial differential equations (PDEs), the integration of this knowledge can significantly enhance the quality of the generated samples.

A generic form of a physical law represented as a PDE that describes the evolution of a continuous temporal signal 𝐱(𝐮,t) over a spatial coordinate 𝐮 is given by:

    ∂𝐱/∂ t = F(t,𝐱,𝐮,∂𝐱/∂𝐮_i, ∂^2𝐱/∂𝐮_i ∂𝐮_j, …)


Based on this PDE representation of physical knowledge, the consistency between the predicted time series 𝐱^tar and the physics knowledge can be quantified using the following squared residual function:

    K(𝐱^tar;F) = -||∂𝐱^tar/∂ t-F(t,𝐱^tar,𝐮,∂𝐱^tar/∂𝐮_i, ∂^2𝐱^tar/∂𝐮_i ∂𝐮_j, …)||^2_2


This function reaches its maximum when the predicted time series is perfectly consistent with the physical model, resulting in a residual of 0. Using this metric K, physics knowledge can be integrated into a probabilistic time series foundation model p(𝐱^tar|𝐱^con) by solving the following optimization problem to obtain a refined model q(𝐱^tar|𝐱^con): 
    
    q(𝐱^tar|𝐱^con) = max_q𝔼_𝐱^tar∼ qK(𝐱^tar;F) - α D_KL(q(𝐱^tar|𝐱^con)||p(𝐱^tar|𝐱^con))

where the first term represents the aforementioned physics knowledge metric, and the second term controls the divergence between q(𝐱^tar|𝐱^con) and p(𝐱^tar|𝐱^con).
The following theorem provides a closed-form solution to the above optimization problem:


    The optimal q(𝐱^tar|𝐱^con) in Eq.<ref> is the Boltzmann distribution defined on the following energy function:
    
    E(𝐱^tar;𝐱^con) = K(𝐱^tar;F)+αlog p(𝐱^tar|𝐱^con)

    in other words, the optimal q(𝐱^tar|𝐱^con) is:
    
    q(𝐱^tar|𝐱^con) = 1/Zexp(K(𝐱^tar;F)+αlog p(𝐱^tar|𝐱^con)),

where Z = ∫exp(K(𝐱^tar;F)+αlog p(𝐱^tar|𝐱^con))d𝐱^tar is the partition function.


The theorem illustrates that sampling from the Boltzmann distribution defined in Eq.  <ref>, is analogous to incorporating physics knowledge into model edition. In the context of diffusion models, this distribution can be effectively sampled using Langevin dynamics <cit.>:
    
    𝐱^tar_j+1   = 𝐱^tar_j + ϵ∇log q(𝐱^tar|𝐱^con) + √(2ϵ)σ, σ∼𝒩(0,1)
       =𝐱^tar_j+ϵ∇ K(𝐱^tar_j;𝐱^con)+αϵ∇log p(𝐱^tar_j|𝐱^con) + √(2ϵ)σ, σ∼𝒩(0,1)


In diffusion model, precisely calculate the likelihood log p(𝐱^tar|𝐱^con) is intractable. To tackle this issue, following previous works <cit.>, we approximate likelihood with the objective to edit the pre-trained diffusion model:

    log p(𝐱^tar|𝐱^con) = -𝔼_ϵ, t [||ϵ_θ(𝐱^tar,t;𝐱^con)-ϵ||^2]


The approximation presented above constitutes the optimizable component of the evidence lower bound(ELBO). Algorithm <ref> summarizes the comprehensive model editing process. 



§ EXPERIMENTS


To comprehensively assess our time series foundation model, we evaluate on a diverse set of tasks that reflect real-world challenges and applications. We begin by testing the model's performance in practical scenarios and its ability to integrate domain knowledge. This includes handling missing data and multi-resolution forecasting on customized datasets, which allows us to evaluate the model's robustness in situations that frequently occur in real-world applications, as well as physics-informed modeling crucial for scientific and engineering domains<cit.>, which uses 6 practical partial differential equations (PDEs). We then assess the model's capabilities in well-established benchmarking tasks across various fields such as finance, healthcare, and industrial monitoring. These tasks include forecasting on Solar, Electricity, Traffic, Taxi, and Exchange datasets<cit.> to evaluate temporal dependency modeling, imputation on ETTh, ETTm, Weather and Electricity datasets<cit.> to assess the handling of missing data, 
anomaly detection on MSL, SMAP, SWaT, SMD, and PSM datasets<cit.> to gauge sensitivity to unusual patterns, 
and synthetic data generation on Stock, Air Quality, and Energy datasets<cit.> to test understanding of underlying distributions. By evaluating these diverse tasks, we can demonstrate that our model truly serves as a foundation for various time series applications, potentially reducing the need for task-specific models.









 §.§ Practical Scenarios: Missing Data and Multi-Resolution Forecasting



To simulate more realistic scenarios in time series tasks, we introduced two additional challenges: missing values and multi-resolution data. These conditions are common in real-world applications and test a model's robustness and adaptability.
For the missing value scenario, we created datasets with various missing ratios, simulating incomplete data often encountered in practice. In the multi-resolution setting, we sampled each individual time series within the multivariate dataset at different resolutions, reflecting the diverse sampling frequencies often present in real-world data collection.

Figure  <ref> illustrates TimeDiT's performance in realistic scenarios, showcasing its effectiveness across different sampling frequencies on the Exchange dataset. In Figure  <ref> (a), we observe TimeDiT's superior performance in handling missing data. As the missing ratio increases from 5% to 50%, TimeDiT maintains the lowest CRPS_sum across all scenarios, indicating its robustness to data gaps. The performance gap between TimeDiT and other models widens as the missing ratio increases, highlighting its effectiveness in more challenging conditions.
Figure  <ref> (b) demonstrates TimeDiT's ability to manage multi-resolution data, where it maintains a clear performance advantage as the number of different sampling resolutions increases from 2 to 6. This demonstrates its ability to effectively integrate and forecast time series data sampled at varying frequencies.
These findings underscore TimeDiT's potential as a practical and versatile tool for time series forecasting in diverse, challenging scenarios that more closely resemble real-world applications.





 §.§ Domain Knowledge Integration: Physics-Informed TimeDiT
 

A key advantage of our approach is the ability to directly incorporate physics knowledge into the pretrained foundation model without additional fine-tuning. This is possible because the diffusion model reconstructs the entire process, allowing for seamless integration of PDE-based constraints during inference. By encoding the known physical laws governing the system into the sampling process, we can guide the model towards more physically consistent and accurate predictions.
In this section, we evaluate how effectively our pre-trained foundation model can integrate physics-informed knowledge into time series forecasting without the need for fine-tuning. We study four 1D partial differential equations (PDEs) from  <cit.>: general Navier-Stokes Equations, Kolmogorov Flow (a specific case of Navier-Stokes Equations), Advection Equations, Burgers Equations, Diffusion Soeption and Computational Fluid Dynamics (CFD). These equations are used to generate synthetic data with random initial conditions, and we apply diffusion models to forecast time series based on data from a historical window.

Table <ref> presents the results, including both mean error and error bars. The table clearly demonstrates that our proposed model editing solution, which incorporates physics knowledge, significantly outperforms previous sampling strategies such as DDPM <cit.>, DDIM <cit.>, and TS Diffusion's Self-Guidance <cit.>. By leveraging domain-specific physical information, our approach achieves substantial performance improvements over these baselines, highlighting the effectiveness of integrating physics-informed priors into the diffusion model sampling process.
This performance gain underscores the potential of combining pretrained foundation models with domain-specific knowledge. Our method offers a flexible framework for enhancing time series forecasting in scientific and engineering applications where the underlying physical laws are partially known. It demonstrates that by bridging the gap between data-driven approaches and physics-based modeling, we can achieve more accurate and physically consistent predictions without the computational overhead of retraining or fine-tuning the entire model. The ability to easily incorporate physics knowledge into a pretrained foundation model represents a significant advance in the field of scientific machine learning. It opens up new possibilities for rapid adaptation to specific physical systems and phenomena, potentially accelerating research and discovery in various scientific domains.






 §.§ Forecasting on Full-shot Setting and Zero-shot Setting


In the forecasting task, we conduct two types of experiments. First, we compare our proposed TimeDiT with baselines in a full-shot setting, where models are trained and tested on separate datasets. This approach evaluates their performance on conventional time series forecasting tasks, ensuring that models can effectively learn and generalize from complete data. Second, we assess TimeDiT as a foundation model in a zero-shot setting, comparing it to previous transformer-based time series models. This setting is crucial as it tests the model's ability to generalize and adapt to entirely new datasets without prior exposure, highlighting its robustness and versatility. 
Together, these experiments provide a holistic view of TimeDiT's capabilities, addressing both specialized performance and broad applicability in time series forecasting.













Table  <ref> presents the full-shot forecasting results, comparing TimeDiT with state-of-the-art models in two categories: deterministic forecasting models, which are trained with the Student's t-distribution head to support probabilistic results, and inherently probabilistic time series forecasting models, including diffusion-based models, such as CSDI and non-diffusion-based, such as GP-copula. Our model achieves the lowest CRPS_sum on four datasets and the second-best performance on the Taxi dataset.  In the zero-shot setting, TimeDiT is compared with the open-sourced foundation models including TEMPO <cit.>, which is pre-trained with Student's t-distribution head to support probabilistic results, Moirai <cit.> and LagLLama <cit.> in Table <ref>. TimeDiT's ability to outperform other open-source foundation models in most cases is particularly noteworthy, as it suggests that TimeDiT can be effectively applied to a wide range of time series forecasting tasks across different domains with minimal adaptation.  








 §.§ Imputation Task

We conduct experiments on six benchmark time-series datasets: ETTh1, ETTh2, ETTm1, ETTm2, Electtricity, and Weather. We use random mask ratios {12.5%, 25%, 37.5%, 50%} following previous studies' settings with sequence length set to 96. We finetune on model checkpoints pretrained on solar, traffic, exchange, taxi, Huawei cloud, air quality, and weather (different from the evaluation weather data). Table <ref> shows the imputation result averaged over the four mask ratios. achieves the best performance on most dataset. obtaining 10 first places out of the 12 evaluations while the remaining baselines obtained 2 first place count in total. In particular, achived a 39% reduction in MSE and 22% reduction in MAE compared to the strongest baseline on ETTh1 dataset. For full result on each mask ratio, please refer to section <ref>.








 §.§ Anomaly Detection Task


We conduct experiments on five real-world datasets from industrial applications: MSL, SMAP, SWaT, SMD, and PSM.  The diffusion model, renowned for its proficiency in distribution learning, may inadvertently overfit by reconstructing anomalies alongside normal data points. To counteract this, we opted to bypass pretraining and introduced spectral residue (SR) transformation at the preprocessing stage of . This transformation helps to conceal points most likely to be anomalies and their immediate neighbors. The number of neighbors affected is controlled by the hyperparameter n_neighbor. The SR method utilizes Fourier Transformation to convert the original time series into a saliency map, thereby amplifying abnormal points, as detailed in <cit.>. For additional information about this transformation, please see section <ref>. 
Consistent with prior methodologies, we set the sequence length to be 100 identify anomalies using the 99th percentile of reconstruction errors. During evaluations, we apply standard anomaly adjustments as suggested by <cit.>. As demonstrated in table <ref>, outperforms baseline models on four of the five datasets. In particular, 23.03 points of improvement in terms of F1 score on the SMAP dataset compared to the strongest baseline.










 §.§ Synthetic Generation Task






We conduct experiments to synthesize multivariate time series and evaluate performance using the discriminative score and predictive score metrics under a "train on synthetic test on real" experimental setup with sequence length set to 24  <cit.>. We finetune on model checkpoints pretrained on etth1, etth2, ettm1, ettm2, electricity, and weather. Table <ref> shows the result on synthetic generation where TimeDiT, in general, consistently generates more realistic synthetic samples compared to baselines, even on high-dimensional energy datasets. This demonstrates TimeDiT's strength in complex time series synthesis. We visualize synthesis performance using PCA and t-SNE in Appendix <ref>. As shown in Figure <ref>, TimeDiT's samples markedly overlap the original data distribution better than other methods. Qualitative and quantitative results confirm TimeDiT's superior ability to model intricate characteristics for realistic time series synthesis, even on multidimensional, complex datasets.  







§ CONCLUSION


In this paper, we introduced , a pioneering approach to creating a versatile and robust foundation model for various time series tasks under practical scenarios. By integrating transformer architecture with diffusion model, effectively captures temporal dependencies and addresses real world challenges unique to time series regarding multi-resolution and missing values as well as incorporating external knowledge. Our innovative masking strategies allow for a consistent training framework adaptable to diverse tasks such as forecasting, imputation, and anomaly detection and synthetic data generation. Extensive experiments demonstrated the strong performance of on both practical scenarios and standard benchmarks. However, we recognize some limitations. We primarily explored common sequence lengths and did not assess 's performance on very long sequences. While we have introduced randomness in prediction length and feature numbers up to a maximum, we aim to develop more scalable solutions for highly variable multivariate time series. Additionally, our understanding of how different types of external information contribute to performance is still developing. For future work, we envision several key directions: enhancing scalability to improve 's ability to handle practical time series with varying multivariate numbers; developing techniques for seamless multi-modal integration, allowing to leverage diverse data sources for improved performance across different tasks; and extending 's capabilities to effectively process and analyze very long time series sequences, addressing a critical need in many real-world applications.









§ DATASETS


    
  * The ETT datasets <cit.>[ETT: <https://github.com/zhouhaoyi/ETDataset>] include electricity load data at various resolutions (ETTh & ETTm) from two different electricity stations.
    
  * The Weather dataset comprises 21 meteorological indicators collected in Germany over the span of one year.
    
  * The Electricity dataset provides information on electricity consumption.
    
  * The SMD dataset <cit.> includes multivariate time-series data collected from server machines in a data center. It typically contains metrics such as CPU usage, memory usage, and disk activity.
    
  * The PSM dataset <cit.>is used for predictive maintenance and includes sensor data from industrial machines. It often contains readings such as temperature, pressure, and vibration over time.
    
  * The MSL dataset <cit.> comes from the Mars Science Laboratory mission, specifically the Curiosity rover. It includes telemetry data from the rover's sensors and systems.
    
  * The SWaT dataset <cit.> originates from a scaled-down water treatment testbed designed to reflect a real-world water treatment process. It includes sensor and actuator data collected over time.
    
  * The SMAP dataset <cit.>comes from NASA's Soil Moisture Active Passive (SMAP) mission, which measures soil moisture and freeze/thaw state. It includes time-series data from multiple sensors aboard the SMAP satellite.
    
  * The Sine dataset <cit.> is synthetically generated by sinusoidal waves.
    
  * The Air Quality dataset [Air Quality: <https://archive.ics.uci.edu/dataset/360/air+quality>]contains hourly averaged readings from five metal oxide chemical sensors integrated into an Air Quality Chemical Multisensor Device. This device was positioned at road level in a highly polluted area of an Italian city. Data were collected from March 2004 to February 2005, making it the longest freely available record of on-field air quality chemical sensor responses.
    
  * The Stock dataset [Stock: <https://finance.yahoo.com/quote/GOOG>] contains daily historical Google stocks data from 2004 to 2019. 
    
  * The UCI Appliances Energy prediction dataset [Energy: <https://archive.ics.uci.edu/ml/datasets>]consists of multivariate, continuous-valued measurements including numerous temporal features measured at close intervals.
    
  * The Cloud dataset: The Huawei cloud datasets contain serverless traces <cit.>. Following <cit.>, we selected 8 time series containing metrics based on the minute-frequency occurrences of the top 10 functions over a period of 141 days. The metrics included in these series are: Function delay; Platform delay; CPU usage; Memory usage; CPU limit; Memory limit; Instances; Requests. The functions were chosen based on their median occurrences throughout the dataset.
    
  * The Weather_2 dataset: The Weather_2 dataset comprises hourly climate time series data collected near Monash University, Clayton, Victoria, Australia, from January 2010 to May 2021. It includes series for temperature, dewpoint temperature, wind speed, mean sea level pressure, relative humidity, surface solar radiation, surface thermal radiation, and total cloud cover <cit.>.






§ FORECASTING EXPERIMENT SETTING


For the forecasting task, we utilized five widely-used open datasets to evaluate probabilistic time series forecasting performance. These datasets were collected in GluonTS  <cit.> and have been previously employed in  <cit.>:


    
  * Solar[Solar: <https://www.nrel.gov/grid/solar-power-data.html>]: Hourly solar power production records from 137 stations in Alabama State, as used in  <cit.>.
   
  * Electricity[Electricity: <https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014>]: Hourly time series of electricity consumption for 370 customers, as used in  <cit.>.
   
  * Traffic[Traffic_nips: <https://archive.ics.uci.edu/dataset/204/pems_sf>]: Hourly occupancy rates of 963 San Francisco freeway car lanes, with values between 0 and 1  <cit.>.
   
  * Taxi[Taxi: <https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data>]: Half-hourly spatio-temporal time series of New York taxi rides taken at 1,214 locations, using data from January 2015 for training and January 2016 for testing, as proposed in  <cit.>.
   
  * Exchange rate[Exchange: <https://github.com/laiguokun/multivariate-time-series-data>]: Daily exchange rates between 8 currencies, namely Australia,
the United Kingdom, Canada, Switzerland, China, Japan, New Zealand, and Singapore, as used in   <cit.>.



Table  <ref> summarizes the characteristics of each dataset. The task for these datasets is to predict the future L_2 steps given the observed L_1 steps. We set L_1 and L_2 values based on previous studies  <cit.>. For training, we randomly selected L_1 + L_2 consecutive time steps as a single time series and designated the last L_2 steps as forecasting targets. We adhered to the train/test splits used in previous studies and utilized the last five samples of the training data as validation data.

For the full-shot setting, we trained separate models on different datasets. Due to the large number of features in multivariate time series, we adopted subset sampling of features for training. For each input, we split them into subsets based on their order. If the last subset was smaller than the fixed shape, we applied padding to ensure equal input sizes across all subsets. In the multi-resolution setting, we used different resolutions identified by the resolution number, which corresponded to different sampling rates for the exchange features. It is worth to note that the aforementioned strategy was also employed for zero-shot training, as the input feature length varied across datasets.




§ TRAINING DETAILS

The codebase for TimeDiT is modified from <https://github.com/facebookresearch/DiT>, where they provide different model sizes, including Small (S), Big (B), Large (L), Extra Large (XL) <cit.>.
In our training, we used Adam optimizer with a training rate of 0.0001 without weight decay. Batch size is set to 512. The maximum channel number K_max is set to 40. All experiments are run on NVIDIA A100 GPUs. The zero-shot foundation model was trained on the ETT, weather, illness, air quality, and cloud datasets and used for different downstream tasks. We will include more available time series datasets to develop a more robust time series foundation model as the future work.




§ METRICS




  
MAE describes the mean absolute error that measures the absolute difference between ground truth and prediction. 

    MAE = 1/n∑_i=1^n |y_i - ŷ_i|



  
MSE describes the mean squared difference between ground truth and prediction. 
    MSE = 1/n∑_i=1^n (y_i - ŷ_i)^2



  
RMSE is the sqaure root of MSE. 

    RMSE = √(1/n∑_i=1^n (y_i - ŷ_i)^2)



  
Discriminative score Following TimeGAN, we train a post-hoc time-series classification model (by optimizing a 2-layer LSTM) to distinguish between sequences from the original and generated datasets. First, each original sequence is labeled real, and each generated sequence is labeled not real. Then, an off-the-shelf (RNN) classifier is trained to distinguish between the two classes as a standard supervised task. We then report the classification error on the held-out test set.


  
Predictive Score Following TimeGAN, we train a post-hoc sequence-prediction model (by optimizing a 2-layer LSTM) to predict next-step temporal vectors over each input sequence. Then, we evaluate the trained model on the original dataset. Performance is measured in terms of the mean absolute error (MAE); for event-based data, the MAE is computed as the absolute value of 1 - estimated probability that the event occured. 


  
Computations of CRPS


We explain the definition and calculation of the CRPS metric. The continuous ranked probability score (CRPS)  assesses how well an estimated probability distribution F aligns with an observation x. It is defined as the integral of the quantile loss Λ_α(q, z) := (α - 1_z<q)(z - q) over all quantile levels α∈ [0, 1]:


    CRPS(F^-1, x) = ∫_0^1 2Λ_α(F^-1(α), x)   dα


where 1 represents the indicator function. We then calculated quantile losses for quantile levels discretized in 0.05 increments. Thus, we approximated CRPS as follows:


    CRPS(F^-1, x) ≈1/19∑_i=1^19 2Λ_i · 0.05(F^-1(i · 0.05), x).


Next, we computed the normalized average CRPS for all features and time steps:


    ∑_k,lCRPS(F^-1_k,l, x_k,l)/∑_k,l |x_k,l|


where k and l denote the features and time steps of the imputation targets, respectively.

CRPS_sum measures CRPS for the distribution F of the sum of all K features, calculated by:


    ∑_lCRPS(F^-1, ∑_k x_k,l)/∑_k,l |x_k,l|


where ∑_k x_k,l is the total of the forecasting targets for all features at time point l.





§ SYNTHETIC GENERATION
 
We use 80% of all data for training and evaluate on the same data. For the air quality dataset, previous methods did not carefully the -200 values as placeholder for missing values. In our experiment, we masked all the -200 values for and baselines that support masks. For baselines that do not support mask, we replace -200 with the mean value. Minmax scaler is used for all models. Figure <ref>, <ref>,<ref>,<ref> shows the PCA plots for all datasets and baselines. The visual comparison also validates the superiority of . 






§ IMPUTATION
 




§ ANOMALY DETECTION
 




 §.§ SR processing for Anomaly Detection



  
Spectral Residue The SR Transformation involves the following equations. Table <ref> shows the full anomaly detection results. 

    A(f) = Amplitude(F(x))


    P(f) = Phase(F(x))


    L(f) = log(A(f))


    AL(f) = h_q(f) · L(f)


    R(f) = L(f) - AL(f)


    S(x) = F^-1(exp(R(f) + iP(f)))



§ PHYSICS EQUATIONS

The Burgers Equation is:

    ∂ u/∂ t + u∂ u/∂ x - v∂^2 u/∂ x^2 = 0

where v is the diffusion term. We set the v (diffusion term) as 0.1 and randomly sample a combination of sine waves as initial status

The Advection Equation is:

    ∂ u/∂ t + c∂ u/∂ x = 0

where c is the advection speed. We set the c as 1.0 and randomly placed Gaussian peaks as initial status


The diffusion-reaction Equation is:

    ∂ u/∂ t - D∂^2 u/∂ x^2 -R(u)= 0

where D is the diffusion coefficient and R(u) is the reaction term. Here, we apply a linear reaction term R(u) = -k· u, where k is the reaction speed. We set the D as 1.0, k as 0.1, and a Gaussian distribution with random parameters as initial status.

The Kolmogrov Flow is a specific case of NS equation. More specifically, it is described by:


    𝐮(x, y, z, t) = ( -∂ψ/∂ y, ∂ψ/∂ x, 0 )


where the psi is the flow function. It is usually set as:


    ψ(x, y, z, t) = A sin(kx) cos(zy + ω t)


where A,k,w are hyperparameters.




§ PROOF OF THE THEOREM


Let us consider the objective function:
    
    O(q(y|x))    = 𝔼_y∼ q(y|x)K(y) - α D_KL(q(y|x)||p(y|x))
       =𝔼_y∼ q(y|x)K(y) - α∫_yq(y|x)log(q(y|x)/p(y|x))dy
       =∫_yq(y|x)[K(y) + αlog p(y|x) -αlog q(y|x)]dy

    We try to find the optimal q(y|x) through Lagrange multipliers. The constraint of the above objective function is that q(y|x) is a valid ∫_y q(y|x)dy=1. Thus, the Lagrangian is:
    
    L(q(y|x),λ)    = ∫_yq(y|x)[K(y) + αlog p(y|x) -αlog q(y|x)]dy - λ (∫_y q(y|x)dy-1)
       =∫_y q(y|x)[K(y) + αlog p(y|x) -αlog q(y|x)-λ q(y|x)]dy + λ

    We define f(q(y|x),y,λ)=q(y|x)[K(y) + αlog p(y|x) -αlog q(y|x) - λ] + λ h(y)], where h(y) can be the density function of any fixed distribution defined on the support set of y. Therefore, L(q(y|x),λ) = ∫_y f(q(y|x),y,λ)dy.
    According to Euler-Lagrange equation, when the above Lagrangian achieve extreme point, we have:
    
    ∂ f/∂ q = K(y) + αlog p(y|x) -αlog q(y|x) - λ - α = 0

    Thus, we have:
    
    αlog q(y|x)    = K(y) + αlog p(y|x) - log q(y|x) - λ - α
    
            q(y|x)    = exp(1/αK(y) + log p(y|x) - λ/α - 1)
       =1/exp(λ/α +1)exp(1/αK(y) + log p(y|x))

    Meanwhile, since ∫_y q(y|x)dy=1, we have:
    
    ∫_y exp(1/αK(y) + log p(y|x) - λ/α - 1)dy   =1
    1/exp(λ/α +1)∫_y exp(1/αK(y) + log p(y|x))dy   =1

    Thus, we have exp(λ/α +1) = ∫_y exp(1/αK(y) + log p(y|x))dy=Z, leading to:
    
    q(y|x) = 1/Zexp(K(y)+αlog p(y|x)), Z = ∫exp(K(y)+αlog p(y|x))dy




