Solving Video Inverse Problems Using Image Diffusion Models
Taesung Kwon Jong Chul Ye


BMI Prediction from  Handwritten English Characters Using a Convolutional Neural Network





    N. T. Diba1, N. Akter2, S. A. H. Chowdhury3
Dept. of Electronics & Telecommunication Engineering 

Rajshahi University of Engineering & Technology

Rajshahi, Bangladesh

nishattasnimediba09@gmail.com1,
 nasrinmanha123@gmail.com2, arif.1968.ruet@gmail.com3
J. E. Giti4
Dept. of Electrical & Electronic Engineering 

Rajshahi University of Engineering & Technology

Rajshahi, Bangladesh

jishan.e.giti@gmail.com4

    September 9, 2024
===============================================================================================================================================================================================================================================================================================================================================================================================================================





§ ABSTRACT 
Recently, diffusion model-based inverse problem solvers (DIS) have emerged as state-of-the-art approaches for addressing inverse problems, including image super-resolution, deblurring, inpainting, etc. 
However, their application to video inverse problems arising from spatio-temporal degradation remains largely unexplored due to the challenges in training video diffusion models.
To address this issue, here we introduce an innovative video inverse solver that leverages only image diffusion models.
Specifically, by
drawing inspiration from the success of the recent decomposed diffusion sampler (DDS), 
our method treats the time dimension of a video as the batch dimension of image diffusion models and solves spatio-temporal optimization problems within denoised spatio-temporal batches derived from each image diffusion model.
Moreover, we introduce a batch-consistent diffusion sampling strategy that encourages consistency across batches by synchronizing the stochastic noise components in image diffusion models. 
Our approach synergistically combines batch-consistent sampling with simultaneous optimization of denoised spatio-temporal batches at each reverse diffusion step, resulting in a novel and efficient diffusion sampling strategy for video inverse problems.
Experimental results demonstrate that our method effectively addresses various spatio-temporal degradations in video inverse problems, achieving state-of-the-art reconstructions.
Project page: 




§ INTRODUCTION




Diffusion models <cit.> represent the state-of-the-art generative modeling by learning the underlying data distribution p() to produce realistic and coherent data samples from the learned distribution p_θ().
In the context of Bayesian inference, the parameterized prior distribution p_θ() can be disentangled from the likelihood p(|), which denotes the probability of observing  given . This seperation facilitates the derivation of the posterior distribution p_θ(|) ∝ p_θ()p(|).

Diffusion model-based inverse problem solvers (DIS) <cit.> leverage this property, enabling the unconditional diffusion models to solve a wide range of inverse problems. 
They achieve this by conditional sampling from the posterior distribution p_θ(|), effectively integrating information from both the forward physics model and the measurement . 
This approach allows for sophisticated and precise solutions to complex inverse problems, introducing the power and flexibility of diffusion models in practical applications.


Despite extensive DIS research on a wide range of image inverse problems such as super-resolution, colorization, inpainting, compressed sensing,  deblurring, and so on <cit.>, the application of these approaches to video inverse problems, particularly those involving spatio-temporal degradation, has received relatively less attention.
Specifically, 
in time-varying data acquisition systems, various forms of motion blur often arise due to the camera or object motions <cit.>, which can be modeled as a temporal PSF convolution of motion dynamics. These are often associated with spatial degradation caused by noise, camera defocus, and other factors.
Specifically, the spatio-temporal degradation process can be  formulated as:
 
    = ()+

with

    = [ [1]   ⋯ [N] ],   = [ [1]   ⋯ [N] ],   = [ [1]   ⋯ [N] ],

where  [n],[n] and [n] denote the n-th frame ground-truth image,
measurement, and additive noise, respectively; N is the number of temporal frames, and 
 refers to the operator that describes the spatio-temporal degradation process. 
The spatio-temporal degradation introduces complexities that image diffusion priors cannot fully capture, as image diffusion priors are primarily designed to handle spatial features rather than temporal dynamics. 
Employing video diffusion models <cit.> could address these issues, but 
poses significant implementation challenges for video inverse problems,  due to the  difficulty
of training video diffusion models for various applications.

Contrary to the common belief that a pre-trained video diffusion model is necessary for solving video inverse problems, here we propose a radically different method that addresses video inverse problems using only image diffusion models.
Inspired by the success of the decomposed diffusion sampler (DDS) <cit.>, which simplifies DIS by formulating it as a Krylov subspace-based optimization problem for denoised images via Tweedie's formula at each reverse sampling step, 
we treat the time dimension of a video as the batch dimension of image diffusion models
and solve spatio-temporal optimization problems using the batch of denoised temporal frames from image diffusion models.
However, treating each frame of the video as a separate sample in the batch dimension can lead to inconsistencies between temporal frames. To mitigate this, we introduce  the batch-consistent sampling strategy
that controls the stochastic directional component (e.g., initial noise or additive noise) of each image diffusion model during the reverse sampling process, encouraging the temporal consistency along the batch dimension. 
By synergistically combining batch-consistent sampling with the simultaneous optimization of the spatio-temporal denoised batch, our approach effectively addresses a range of spatio-temporal inverse problems, including spatial deblurring, super-resolution, and inpainting.
Our contribution can be summarized as follows.


  * We introduce an innovative video inverse problem solver using pre-trained image diffusion models by solving spatio-temporal optimization problems within the batch of denoised frames.

  * We develop a batch-consistent sampling strategy to ensure temporal consistency by synchronizing stochastic noise components in image diffusion models.

  * Extensive experiments confirm that our method generates state-of-the-art results for various video inverse problems.





§ BACKGROUND



Diffusion models.
Diffusion models <cit.> attempt to model the data distribution p_data()  based on  a latent variable model

    p_θ(_0) = ∫ p_θ(_0:T)  d_1:T,   where    p_θ(_0:T) := p_θ(_T) ∏_t=1^T p_θ^(t)(_t-1|_t)

where the _1:T are noisy latent variables defined by the Markov chain with Gaussian transitions

    q(_t|_t-1) = 𝒩(_t|√(β_t)_t-1, (1 - β_t) I),   
        q(_t|_0) = 𝒩(_t|√(α̅_t)_0, (1 - α̅_t) I).

Here, the noise schedule β_t is an increasing sequence of t, with α̅_t := ∏_i=1^t α_i, α_i := 1 - β_i. 
Training of diffusion models amounts to training a multi-noise level residual denoiser:

    min_θ__t ∼ q(_t|_0), _0 ∼ p_data(_0), ϵ∼𝒩( 0, I)[ϵ_θ^(t)(_t) - ϵ_2^2] .


Then, sampling from (<ref>) can be implemented by ancestral sampling, which iteratively performs

    _t-1 =   1/√(α_t)(_t - 1 - α_t/√(1 - α̅_t)ϵ_θ^*^(t)(_t) ) + β̃_tϵ

where β̃_t := 1 - α̅_t-1/1 - α̅_tβ_t
and 
θ^* refers to the optimized parameter from  (<ref>).
On the other hand, DDIM <cit.> accelerates the sampling based on
non-Markovian assumption. Specifically, the sampling iteratively performs

    _t-1 =   √(α̅_t-1)_t +  √(1-α̅_t-1)

where 

    _t :=
        1/√(α̅_t)(_t - √(1 - α̅_t)ϵ_θ^*^(t)(_t)  ),  := √(1-α̅_t-1-η^2β̃^2_t)ϵ_θ^*^(t)(_t)  + ηβ̃_tϵ/√(1-α̅_t-1)

Here, _t is the denoised estimate of _t that is derived from Tweedie’s formula <cit.>.
Accordingly, DDIM sampling can be expressed as a two-step manifold transition: (i) the noisy sample _t∈ℳ_t transits to clean manifold ℳ by deterministic estimation using Tweedie's formula, (ii) a subsequent transition from clean manifold to next noisy manifold ℳ_t-1 occurs by adding noise ,
which is composed of the deterministic noise ϵ_θ^*^(t)(_t) and the stochastic noise ϵ.


Diffusion model-based inverse problem solvers.

For a given loss function ℓ() which often stems from  the likelihood
for
measurement consistency, the goal of DIS is to address the following optimization problem

    min_∈ℓ()

where 
 represents the clean data manifold sampled from unconditional distribution p_0().
Consequently, it is essential to find a way that minimizes cost while also identifying the correct manifold.


Recently, <cit.> proposed a general technique called diffusion posterior sampling (DPS), where the updated estimate from the noisy sample _t ∈_t is constrained to stay on the same noisy manifold _t. 
This is achieved by computing the manifold constrained gradient (MCG) <cit.> on a noisy sample _t∈_t.
The resulting algorithm can be stated as follows:

    _t-1   = √(α̅_t-1)( -  γ_t ∇__tℓ ())+ √(1-α̅_t-1),

where γ_t>0 denotes the step size.
Under the linear manifold assumption <cit.>, this allows precise transition to _t-1.
Unfortunately, the computation of MCG  requires computationally expensive backpropagation and is often unstable.  

In a subsequent work, <cit.> shows that   
under the same linear manifold assumption in DPS,
the one step update by -  γ_t ∇_ℓ () 
are guaranteed to remain within a linear subspace, thus obviating the need for explicit computation of the MCG and
leading to a simpler approximation:
 
    _t-1   ≃√(α̅_t-1)( -  γ_t ∇_ℓ ())+ √(1-α̅_t-1) .


Furthermore, instead of using a one-step gradient update, <cit.> demonstrated that multi-step update using Krylov subspace methods, such as the conjugate gradient (CG) method, guarantees that the intermediate steps lie in the linear subspace. This approach improves the convergence of the optimization problem without incurring additional neural function evaluations (NFE). This method, often referred to as decomposed diffusion sampling (DDS), bypasses the computation of the MCG and improves the convergence speed, making it stable and suitable for large-scale medical imaging inverse problems <cit.>.




§ VIDEO INVERSE SOLVER USING IMAGE DIFFUSION MODELS




 §.§ Problem formulation


Using the forward model (<ref>) and the optimization framework in (<ref>), the video inverse problem can be formulated as

    min_∈ℓ():= - ()^2

where   denotes the spatio-temporal volume of the clean image composed of N temporal frames as defined in (<ref>), and 
 represents the clean video manifold  sampled from unconditional distribution p_0().
Then, a naive application of the one-step gradient within the DDS framework can be formulated by
 
    _t-1   = √(α̅_t-1)(_t -  γ_t ∇__tℓ (_t))+ √(1-α̅_t-1) .

where  and  refer to Tweedie's formula and noise in the spatio-temporal volume, respectively, which are defined by  

    :=
        1/√(α̅_t)(_t - √(1 - α̅_t)ℰ_θ^*^(t)(_t)  ),  := √(1-α̅_t-1-η^2β̃^2_t)ℰ_θ^*^(t)(_t)  + ηβ̃_tℰ/√(1-α̅_t-1)

Here, _t refers to the spatio-temporal volume at the t-th reverse diffusion step  and
ℰ∼∏_i=1^N𝒩(0,).
Although the formula (<ref>) is a direct extension of the image-domain counterpart (<ref>),
the main technical challenge lies in training the video diffusion model ℰ_θ^(t), which is required for the formula (<ref>). Specifically, the video diffusion model is trained by

    min_θ__t ∼ q(_t|_0), _0 ∼ p_data(_0), ℰ∼∏_i=1^N𝒩(0,)[ℰ_θ^(t)(_t) - ℰ_2^2] ,

which requires large-scale video training data and computational resources beyond the scale of training image diffusion models. 
Therefore, the main research motivation is to propose an innovative method that can bypass the need for computationally extensive video diffusion models.




 §.§ Batch-consistent reconstruction with DDS


Consider 
a batch of 2D diffusion models along the temporal direction:

    E_θ^(t)(_t):= [ ϵ_θ^*^(t)(_t[1])                ⋯ ϵ_θ^*^(t)(_t[N]) ]

where ϵ_θ^*^(t) represents an image diffusion model.
Suppose that  E_θ^(t)(_t) is used for (<ref>).
Since unconditional reverse diffusion is entirely determined by (<ref>), the
generated video 
 is then
fully controlled by the behavior of the image diffusion models.
Thus, we investigate the limitations of using a batch of image diffusion models compared to using a video diffusion model
and explore ways to mitigate these limitations.

Recall that for the reverse sampling of each image diffusion model, the stochastic transitions occur from two sources: (i) the initialization and (ii) re-noising.
Accordingly,   in batch-independent sampling, where each image diffusion model is initialized with independent random noise and re-noised with independent additive noise, it is difficult to impose any temporal consistency in video generation so that
each generated temporal frame may represent different content from each other (see Fig. <ref>(a)).
Conversely, in batch-consistent sampling, where each image diffusion model is initialized with the same noise and re-noised with the same additive noise, the generated frames from the unconditional diffusion model should be trivially reduced to identical images (see Fig. <ref>(b)). This dilemma is why separate video diffusion model training using (<ref>) was considered necessary for effective video generation.



One of the most important contributions of this paper is demonstrating that the aforementioned dilemma can be readily mitigated in conditional diffusion sampling originated from inverse problems. Specifically, inspired by the DDS formulation in (<ref>),  we propose a method that employs a batch-consistent sampling scheme to ensure temporal consistency and introduces temporal diversity from the conditioning steps.
More specifically,  the denoised image for each frame is computed individually using Tweedie's formula via  image diffusion models:

    ^b :=
        1/√(α̅_t)(_t - √(1 - α̅_t)E_θ^*^(t)(_t)  )

where we use the superscript ^b to represent the batch-consistency and
 E_θ^*^(t) is a batch of image diffusion models defined by (<ref>).
Here, the image diffusion models are initialized with the same random noises to ensure temporal consistency.
Subsequently, the denoised spatio-temporal batch is perturbed as a whole by applying the l-step
conjugate gradient (CG)  to optimize
the data consistency term from the spatio-temporal degradation. 
This can be formally represented by

    _t :=  _∈^b + 𝒦_l - ()^2

where 𝒦_l denotes the l-dimensional Kyrlov subspace 
associated with the given inverse problem <cit.>.
The multistep CG can 
diversify each temporal frame according to the condition and achieve faster convergence than a single gradient step.
The resulting solution ensures that the loss function from the spatio-temporal degradation process can be minimized with coherent but frame-by-frame distinct reconstructions. 
Finally,  the reconstructed spatio-temporal volume from the CG is renoised with additive noise as:

    _t-1   = √(α̅_t-1)_t + √(1-α̅_t-1)^b .

where

    ^b:= √(1-α̅_t-1-η^2β̃^2_t)E_θ^*^(t)(_t)  + ηβ̃_tℰ^b/√(1-α̅_t-1)

Here, ℰ^b denotes the additive random noise from 𝒩(0,).
 In contrast
to  ℰ in (<ref>), which is composed of frame-independent random noises,
we impose batch consistency by adding the same random noises to each temporal frame to ensure temporal consistency.
In summary, the proposed batch-consistent sampling and frame-dependent perturbation through multistep CG ensure that the sampling trajectory of each frame, starting from the same noise initialization, gradually diverges from each other  during reverse sampling to meet the spatio-temporal
data consistency. The
geometric illustration of the sampling path evolution is shown in Fig. <ref>(c). The detailed illustration of the intermediate sampling process of our method is shown in Fig. <ref>.
Additionally, the pseudocode implementation is given in Algorithm <ref>.












§ EXPERIMENTS





In this section, we conduct  thorough comparison studies to demonstrate the efficacy of the proposed method
in addressing spatio-temporal degradations.
Specifically, we consider two types of loss functions for video inverse problems:

    ℓ():= - ()^2,  ℓ_TV():= - ()^2+λ TV()
 
where the first loss is from (<ref>)
and  TV() denotes the total variation loss along the temporal direction.

Then, classical optimization methods are used as the baselines for comparison to minimize each loss function. Specifically, the stand-alone Conjugate Gradient (CG) method is employed to minimize ℓ(), while the Alternating Direction Method of Multipliers (ADMM) is used to minimize ℓ_TV(). Additionally, diffusion-based methods are utilized as baselines to minimize the loss functions in
(<ref>).
Specifically, DPS <cit.> is used to minimize ℓ(). However, instead of relying on 3D diffusion models, we use 2D image diffusion models, similar to our proposed methods, to ensure that backpropagation for MCG computation can be performed through 2D diffusion models.
Second, we employ DiffusionMBIR <cit.> to minimize ℓ_TV(), also using 2D image diffusion models. Unlike the original DiffusionMBIR, which applies TV along the z-direction, we apply TV along the temporal direction.


To test various spatio-temporal degradations, we select the temporal degradation in time-varying data acquisition systems, which is represented as PSF convolution along temporal dimension <cit.>. 
We select three types of PSFs: (i) uniform PSF with widths of 7, (ii) uniform PSF with widths of 13, and (iii) Gaussian PSF with a standard deviation of 1.0. Each kernel is convolved along the temporal dimension with the ground truth video to produce the measurements. Note that convolving uniform PSF with widths of 7 and 13 correspond to averaging 7 and 13 frames, respectively.
Furthermore, we combined temporal degradation and various spatial degradations to demonstrate various combinations of spatio-temporal degradations.
For spatio-temporal degradations, we fix a temporal degradation as a convolving uniform PSF with a width of 7 and add various spatial degradations to the video. 
These spatial degradations include (i) deblurring using a Gaussian blur kernel with a standard deviation σ of 2.0, (ii) super-resolution through a 4× average pooling, and (iii) inpainting with random masking at a ratio r of 0.5 (For specific implementation details of degradations, see Appendix <ref>).







We conduct our experiments on the DAVIS dataset <cit.>, which includes a wide variety of videos covering multiple scenarios. The pre-trained unconditional 256×256 image diffusion model from ADM <cit.> is used directly without fine-tuning and additional networks. All videos were normalized to the range [0, 1] and split into 16-frame samples of size 256×256. A total of 338 video samples were used for evaluation. More preprocessing details are described in the Appendix <ref>.

For quantitative comparison, we focus on the following two widely used standard metrics: peak signal-to-noise-ratio (PSNR) and structural similarity index (SSIM) <cit.> with further evaluations with two perceptual metrics - Learned Perceptual Image Patch Similarity (LPIPS) <cit.> and Fréchet Video Distance (FVD) <cit.>. FVD results are displayed scaled by 10^-3 for easy comparison.
For all proposed methods, we employ l = 5, η = 0.15 for 20 NFE in temporal degradation tasks, and l = 5, η = 0.8 for 100 NFE in spatio-temporal degradation tasks unless specified otherwise.




 §.§ Results

We present the quantitative results of the temporal degradation tasks in Table. <ref>. The table shows that the proposed method outperforms the baseline methods by large margins in all metrics. 
The large margin improvements in FVD indicate that the proposed method successfully solves inverse problems with temporally consistent reconstruction.
Fig. <ref> shows the qualitative reconstruction results for temporal degradations . The proposed method restores much finer details compared to the baselines and demonstrates robustness across various temporal PSFs.
In contrast, as shown in Fig. <ref>, while DPS performs well in reconstructing uniform PSFs with a kernel width of 7, it fails to accurately reconstruct frame intensities as the kernel becomes wider or more complex as shown in the bottom figures, leading to significant drops in Table <ref>.
DiffusionMBIR ensures temporal consistency and performs well for static scenes, but it struggles with dynamic scenes in the video.
In the same context, ADMM-TV produces unsatisfactory results for dynamic scenes.





The results of the spatio-temporal degradations are presented in Table <ref> and Fig. <ref>. 
Even with additional spatial degradations, the proposed method consistently outperforms baseline methods. 
On the other hand, DPS often produces undesired details, as shown in Fig. <ref>. 
DiffusionMBIR fails to restore fine details in dynamic scenes. 
Specifically, in the 3^rd row of Fig. <ref>, DiffusionMBIR restores the static mural painting but fails to capture the motion of the person. 
This is because
TV regularizer often disrupts the restoration of dynamic scenes. 
In this context, our method ensures temporal consistency without the need for a TV regularizer.
Furthermore, thanks to the consistent performance even at low NFE, the proposed method achieves a dramatic 10× to 50× acceleration in reconstruction time. 
For handling temporal degradation with 20 NFE, the proposed diffusion model-based inverse problem solver can now achieve speeds exceeding 1 FPS.




 §.§ Ablation study



Effect of CG updates. Experimental results demonstrate the tangential CG updates in video space on the denoised manifold are key elements in solving spatio-temporal degradations. 
Here, we compare the proposed method with a stand-alone CG method to demonstrate its impact within the solver.
We applied the same CG iterations as in the proposed method but excluded the diffusion updates. As shown in Fig. <ref>, while the stand-alone CG method nearly solves the video inverse problem, it leaves residual artifacts, as seen in the first row, or fails to fully resolve spatial degradation, as shown in the second row. 
In contrast, the proposed method generates natural and fully resolved frames. 
This indicates that the diffusion update in the proposed method refines the unnatural aspects of the CG updates. 

Effect of batch-consistent sampling.
Fig. <ref> illustrates the inter-batch difference within the denoised manifold ℳ during the reverse diffusion process. The blue plot shows results from our full method, while the green and orange plots represent results without stochasticity control and with gradient descent (GD) updates instead of conjugate gradient (CG) updates, respectively. Notably, GD converges more slowly than CG.
Our method consistently achieves low inter-batch difference (i.e., high inter-batch similarity), ensuring batch-consistent reconstruction and precise reconstructions. In contrast, the absence of stochasticity control or the use of GD updates results in higher difference (i.e., lower similarity), leading to less consistent sampling. The intermediate samples  in Fig. <ref> and reconstruction results in Table <ref> further confirm that our method outperforms the others in producing batch-consistent results.
Further experimental results and ablation studies are illustrated in Appendix <ref>. 






§ CONCLUSION


In this work, we introduce an innovative video inverse problem solver that utilizes only image diffusion models. Our method leverages the time dimension of video as the batch dimension in image diffusion models, integrating video inverse optimization within the Tweedie denoised manifold. We combine batch-consistent sampling with video inverse optimization at each reverse diffusion step, resulting in a novel and efficient solution for video inverse problems. Extensive experiments on temporal and spatio-temporal degradations demonstrate that the proposed method achieves superior quality while being faster than previous DIS methods, even reaching speeds exceeding 1 FPS.




iclr2024_conference





§ EXPERIMENTAL DETAILS




 §.§ Implementation of Degradations


For spatio-temporal degradations, we applied temporal degradation followed by spatial degradation sequentially. 
We utilize spatial degradation operations for super-resolution, inpainting, and deblurring as specified in the official implementation from <cit.> and <cit.>. For super-resolution, we employ 4× average pooling as the forward operator . For inpainting, we use a random mask to eliminate half of the pixels for both the forward operator . In deblurring, we apply a Gaussian blur with a standard deviation (σ) of 2.0 and a kernel width of 13 as the forward operator .



 §.§ Data preprocessing details

We conducted every experiment using train/val sets of DAVIS 2017 dataset <cit.>. 480p resolution dataset has a spatial resolution of 480×640. Therefore, to avoid spatial distortion, the frames were first center cropped to 480×480, then resized to a resolution of 256×256. The resizing was performed using the `resize' function from the `cv2' library. After that, all videos were normalized to the range [0, 1].
In the temporal dimension, the video was segmented into chunks of 16 frames starting from the first frame. Any remaining frames that did not form a complete set of 16 were dropped. Through this process, a total of 338 video samples were obtained. The detailed data preprocessing code and the preprocessed Numpy files have all been open-sourced.



 §.§ Comparative methods

DiffusionMBIR <cit.>. 
For DiffusionMBIR,
we use the same pre-trained image diffusion model <cit.> with 1000 NFE sampling.
The optimal ρ and λ values are obtained
through grid search within the ranges [0.001, 10] and [0.0001, 1], respectively. The values are set to (ρ, λ) = (0.1, 0.001) for temporal degradation, and (ρ, λ) = (0.01, 0.01) for spatio-temporal degradation.

DPS <cit.>.
For DPS, 
we use the same pre-trained image diffusion model <cit.> with 1000 NFE sampling. 
The optimal step size ζ is obtained through grid search within the range [0.01, 100].
The value is set to ζ = 30 for both temporal degradation and spatio-temporal degradation.
Memory issues exist when performing DPS sampling more than 5 batch sizes in NVIDIA GeForce RTX 4090 GPU with VRAM 24GB. Therefore, we divide 16-frame videos into 4-frame videos and use them for all DPS experiments.

ADMM-TV. Following the protocol of <cit.>, we optimize the following objective

    ^∗ = argmin1/2 - ^2_2 + λ_1

where = [_t, _h, _w], which corresponds to the classical TV. t, h, and w represent temporal, height, and width directions, respectively. The outer iterations of ADMM are solved with 30 iterations and the inner iterations of CG are solved with 20 iterations, which are identical settings to <cit.>. We perform a grid search to find the optical parameter values that produce the most visually pleasing solution. The parameter is set to (ρ, λ) = (1, 0.001). We set initial  as zeros.




§ FURTHER EXPERIMENTAL RESULTS





 §.§ VRAM-efficient Sampling


The proposed method is VRAM-efficient, treating video frames as batches in the image diffusion model for sampling. As shown in Table <ref>, the method can reconstruct an 8-frame video at 256x256 resolution using less than 11GB of VRAM, which is feasible on GPUs like the GTX 1080Ti or RTX 2080Ti (11GB VRAM). With a single RTX 4090 GPU (24GB VRAM), it can reconstruct a 32-frame video at the same resolution.





 §.§ Ablation study of stochasticity

Experimental results show that synchronizing stochastic noise along batch direction enables batch-consistent reconstruction, offering an effective solution for video inverse problems. 
While it is theoretically possible to achieve batch-consistent sampling with η set to 0 (by eliminating stochastic noise), our empirical findings, as shown in Table. <ref>, indicate that incorporating stochastic noise is beneficial for video reconstruction, particularly in cases involving spatio-temporal degradations. 
Consequently, in our experiments, the optimal η value was determined through a grid search.










 §.§ Detailed visualizations of experimental results




















