DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos
Wenbo Hu Xiangjun Gao Xiaoyu Li Sijie Zhao Xiaodong Cun Yong Zhang Long Quan Ying Shan


[
Target Detection in Sea Clutter with Application to Spaceborne SAR Imaging 



    Shahrokh Hamidi

Department of Electrical and Computer Engineering

University of Waterloo

Waterloo, Ontario, Canada 

shahrokh.hamidi@uwaterloo.ca

    September 9, 2024
=========================================================================================================================================================



    type=figure
    

    
    < g r a p h i c s >

    

    figure
    We innovate DepthCrafter, a novel video depth estimation approach, by leveraging video diffusion models. 
    It can generate temporally consistent long depth sequences with fine-grained details for open-world videos, without the need for additional information such as camera poses or optical flow.
    
    
    


]


[1] Joint first authors.
[2] Corresponding authors.

§ ABSTRACT 
    Despite significant advancements in monocular depth estimation for static images, estimating video depth in the open world remains challenging, since open-world videos are extremely diverse in content, motion, camera movement, and length.
    
    We present DepthCrafter, an innovative method for generating temporally consistent long depth sequences with intricate details for open-world videos, without requiring any supplementary information such as camera poses or optical flow. 
    
    DepthCrafter achieves generalization ability to open-world videos by training a video-to-depth model from a pre-trained image-to-video diffusion model, through our meticulously designed three-stage training strategy with the compiled paired video-depth datasets.
    
    Our training approach enables the model to generate depth sequences with variable lengths at one time, up to 110 frames, and harvest both precise depth details and rich content diversity from realistic and synthetic datasets.
    
    We also propose an inference strategy that processes extremely long videos through segment-wise estimation and seamless stitching.
    
    Comprehensive evaluations on multiple datasets reveal that DepthCrafter achieves state-of-the-art performance in open-world video depth estimation under zero-shot settings.
    
    Furthermore, DepthCrafter facilitates various downstream applications, including depth-based visual effects and conditional video generation.
    
    
    


§ INTRODUCTION


Depth estimation from monocular images or videos, serving as the bridge linking 2D observations and the 3D world, has been a long-standing fundamental problem in computer vision.

It plays a crucial role in a wide range of downstream applications, such as mixed reality, AI-generated content, autonomous driving, and robotics <cit.>.

The inherent ambiguity makes it extremely challenging, as the observed information from a single view is insufficient to determine the depth of a scene uniquely.


With recent advances in foundation models, we have witnessed significant progress in depth estimation from monocular images <cit.>.


However, all these methods are tailored for static images, without considering the temporal information in videos.

Temporal inconsistency, or flickering, would be observed when directly applying them to videos, as shown in Fig. <ref>.

Native video depth estimation methods <cit.> typically optimize a temporally consistent depth sequence in 3D space from a pre-trained image depth model, with a given or learnable calibrated camera poses.

Their performance is sensitive to both the proportion of dynamic content and the quality of the camera poses. 

Yet, videos in the open world are diverse in content, motion, camera movement, and length, making these methods hard to perform well in practice.

Moreover, the required camera poses are often non-trivial to obtain in open-world videos, particularly for long videos and videos with abundant dynamic content.







In this paper, we aim to generate temporally consistent long depth sequences with high-fidelity details for open-world videos, without requiring any additional information, , camera poses, optical flow, .

Observing the strong capability of diffusion models in generating various types of videos <cit.>, we propose a novel approach, named DepthCrafter, to leverage the video diffusion model for video depth estimation, while maintaining the generalization ability to open-world videos.

We train our DepthCrafter, a video-to-depth model, from a pre-trained image-to-video diffusion model, using our compiled paired video-depth datasets, which are in two styles, realistic and synthetic, where the realistic dataset provides rich content diversity and the synthetic dataset offers precise depth details.

On the aspect of temporal context, existing video diffusion models can only produce a fixed and small number of frames at a time, , 25 frames in SVD <cit.>.

However, this is often too short for open-world video depth estimation to accurately arrange depth distributions throughout the video.

Considering both the respective advantages of the two-styled datasets and the requirement of variable long temporal context, we present a three-stage training strategy to progressively train certain layers of the diffusion model on different datasets with variable lengths.

By doing so, we can adapt the diffusion model to generate depth sequences with variable lengths at one time, up to 110 frames, and harvest both the precise depth details and rich content diversity.

To further enable estimating depth sequences for extremely long videos in the open world, we design an inference strategy to process the video in overlapped segments and seamlessly stitch them together.






























We extensively evaluate our DepthCrafter on multiple datasets under zero-shot settings.

Both qualitative and quantitative results demonstrate that our DepthCrafter achieves state-of-the-art performance in open-world video depth estimation, outperforming existing methods by a large margin.

Besides, we demonstrate that our DepthCrafter facilitates various downstream applications, including depth-based visual effects and conditional video generation.

Our contributions can be summarized below:

    
  * We innovate DepthCrafter, a novel method to generate temporally consistent long depth sequences with fine-grained details for open-world videos, outperforming existing approaches by a large margin.
    
    
  * We present a three-stage training strategy to enable generating depth sequences with a long and variable temporal context, up to 110 frames. It also allows us to harvest both the precise depth details and rich content diversity from synthetic and realistic datasets. 
    
    
  * We design an inference strategy to segment-wisely process videos beyond 110 frames and seamlessly stitch them together, enabling depth estimation for extremely long videos.
    






§ RELATED WORK



Monocular depth estimation.
Deep neural networks have dominated monocular depth estimation <cit.> for their superior performance.

Nevertheless, the generalization ability to diverse open-world scenes is challenging due to the limited training data.

To this end, MiDaS <cit.> presents an affine-invariant loss for training on mixed datasets.

Depth-Anything (V2) <cit.> followed this idea and proposed to train the model on both labeled and large-scale unlabeled images, achieving good generalization ability.

Marigold <cit.> and GeoWizard <cit.> leverage the diffusion priors to realize zero-shot transfer to unseen datasets.

Besides, a stream of methods focus on estimating metric depth, such as ZoeDepth <cit.>, UniDepth <cit.>, and Metric3D <cit.>.

However, all these methods are tailored for static images, while our work aims to estimate temporally consistent depth sequences from open-world videos.







Video depth estimation.
Compared to single-image depth estimation, video depth additionally requires temporal consistency.

Existing methods could be categorized into two classes: test-time optimization and feed-forward prediction.

Test-time optimization methods <cit.> involve an optimization procedure for each video during inference, typically requiring camera poses or optical flow.

This type of method usually can produce consistent video depth, but the required camera poses may limit their applicability to open-world videos.

Feed-forward prediction methods directly predict depth sequences from videos <cit.>, , DeepV2D <cit.> combines camera motion estimation with depth estimation, MAMO <cit.> leverages memory attention, and NVDS <cit.> introduces a plug-and-play stabilization network.

However, due to the limited training data and model capacity, these methods often fail to address the in-the-wild videos with diverse content. 

By leveraging video diffusion priors and our designed three-stage training strategy, our method demonstrates the ability to perform open-world video depth estimation.




Video diffusion models.
Diffusion models <cit.> have achieved high-fidelity image generation results from text descriptions benefiting from web-scale aligned image-text datasets. 

Consequently, these models have been extended to generating videos of various types from text or images <cit.>.

Among these methods, VDM <cit.> presents the first results on video generation using diffusion models, while Sora <cit.> has shown remarkable results in this area. 

SVD <cit.> provides the popular open-source pre-trained models for image-to-video. 

Trained on a well-curated video dataset, SVD can generate high-quality videos and is used as the model prior for various video-related tasks. 

In this paper, we leverage the video diffusion model for high-fidelity consistent video depth estimation by taking the input video as the condition.

Concurrent to our work, ChronoDepth <cit.> also explores video depth estimation with video diffusion priors.

However, ChronoDepth only supports a short temporal context, 10 frames, which is insufficient to accurately arrange depth distributions throughout the video.

In contrast, our method not only supports variable-length temporal context, up to 110 frames, but also can estimate depth sequences for extremely long videos.








§ METHOD



Given an open-world video, 𝐯∈ℝ^T × H × W × 3, our goal is to estimate temporally consistent depth sequences, 𝐝∈ℝ^T × H × W, with fine-grained details.





Considering the diversity of open-world videos in content, motion, camera movement, and length, the challenges to achieving our goal are threefold:
1.) a comprehensive understanding of video content for generalization ability;
2.) a long and variable temporal context to arrange the entire depth distributions accurately and keep temporal consistency;
and 3.) the ability to process extremely long videos.

As shown in Fig. <ref>, we tackle these challenges by formulating the video depth estimation as a conditional diffusion generation problem to model the conditional distribution p(𝐝  |  𝐯), training a video-to-depth model from a pre-trained image-to-video diffusion model through a meticulously designed three-stage training strategy with compiled paired video-depth datasets, and crafting an inference strategy to process extremely long videos through segment-wise estimation and seamless stitching.







 §.§ Preliminaries of Video Diffusion Models


Diffusion models <cit.> learn the data distribution p(𝐱) by a forward diffusion process to gradually noise the data to a target distribution, the Gaussian distribution, and a reverse denoising process to iteratively recover the data from the noise by a learned denoiser.

In this paper, our study is conducted based on the stale video diffusion (SVD) <cit.>, which is a famous open-source video diffusion model.

SVD adopts the EDM-framework <cit.> for the noise schedule and denoising process.

The diffusion process is achieved by adding σ_t^2-variance Gaussian noise to the data 𝐱_0 ∼ p(𝐱):

    𝐱_t = 𝐱_0 + σ_t^2 ϵ,   ϵ∼𝒩(0, 𝐈),

where 𝐱_t ∼ p(𝐱;σ_t) is the data with noise level σ_t.

When σ_t is large enough (σ_max),
the distribution would be indistinguishable from the Gaussian distribution.


Based on this fact, the diffusion model starts from a high-variance Gaussian noise ϵ∼𝒩(0, σ_max^2𝐈) and gradually denoises it towards σ_0 = 0 to generate the data.

The denoiser D_θ is a learnable function that tries to predict the clean data, 𝐱̃_0 = D_θ(𝐱_t; σ_t ).

Its training objective is the denoising score matching:

    𝔼_𝐱_t ∼ p(𝐱;σ_t), σ_t ∼ p(σ)
            [λ_σ_t‖ D_θ(𝐱_t; σ_t; 𝐜) - 𝐱_0 ‖_2^2 ],

where p(σ) is the noise level distribution during training, 𝐜 denotes the conditioning information, and λ_σ_t is the weight for the denoising loss at time t.

To promote the learning, EDM adopts the preconditioning strategy <cit.>, to parameterize the denoiser D_θ as:

    D_θ   (𝐱_t; σ_t; 𝐜) = 
       c_skip(σ_t)  𝐱_t + 
            c_out(σ_t)  
            F_θ( 
                c_in𝐱_t;  c_noise(σ_t); 𝐜),

where F_θ is implemented as a learnable U-Net <cit.>, and c_in, c_out, c_skip, and c_noise are preconditioning functions.





















 §.§ Formulation with Diffusion Models





Latent space transformation.
To generate high-resolution depth sequences without sacrificing computational efficiency, we adopt the framework of Latent Diffusion Models (LDMs) <cit.> that perform in a low-dimensional latent space, rather than the original data space.

The transformation between the latent and data spaces is achieved by a Variational Autoencoder (VAE) <cit.>, which was originally designed for encoding and decoding video frames in SVD <cit.>.

Fortunately, we found it can be directly used for depth sequences with only a negligible reconstruction error, which is similar to the observation in Marigold <cit.> for image depth estimation.

As shown in Fig. <ref>, the latent space transformation is formulated as:

    𝐳^(𝐱) = ℰ(𝐱),   𝐱̂ = 𝒟(𝐳^(𝐱)),

where 𝐱 is either the video 𝐯 or the depth sequence 𝐝, 𝐳^(𝐱) is the latent representation of the data, 𝐱̂ is the reconstructed data, ℰ and 𝒟 are encoder and decoder of the VAE, respectively.

For the depth sequence, we replicate it three times to meet the 3-channel input format of the encoder in VAE and average the three channels of the decoder output to obtain the final latent of the depth sequence.

Following the practice in image depth estimation <cit.>, we adopt the relative depth, the affine-invariant depth, which is normalized to [0, 1].

But differently, our predicted depth sequence shares the same scale and shift across frames, rather than a per-frame normalization, which is crucial for maintaining temporal consistency.





Conditioning on the video.
SVD is an image-to-video diffusion model that generates videos conditioned on a single image.

The conditional image is fed into the U-Net in two ways, , concatenating its latent to the input latent, and injecting its CLIP <cit.> embedding to the intermediate features via cross-attention.

Yet, our DepthCrafter involves the generation of depth sequences conditioned on video frames in a frame-to-frame fashion.

Therefore, we adapt the conditioning mechanism in SVD to meet our video-to-depth generation task.

As shown in Fig. <ref>, given the encoded latent of depth sequence 𝐳^(𝐝) and video frames 𝐳^(𝐯) from Eq. (<ref>), we concatenate the video latent to the input noisy depth latent frame-wisely, rather than only the first frame, to condition the denoiser for generating the depth sequence.

For high-level semantic information, we embed the video frames using CLIP and then inject the embeddings in a frame-to-frame manner to the denoiser via cross-attention.

Compared to the original conditioning mechanism, our adapted conditioning provides more comprehensive information from the video frames to the denoiser, which significantly improves the alignment between the generated depth sequences and the video content, as well as the temporal consistency.










 §.§ Training


To train our DepthCrafter, we need a large amount of high-quality paired video-depth sequences.

Although there are several video depth datasets available, , KITTI <cit.>, Scannet <cit.>, VDW <cit.>, DynamicReplica <cit.>, and MatrixCity <cit.>, they are either lacking high-quality depth annotations or restricted to a specific domain, , driving scenes, indoor scenes, or synthetic scenes.


Dataset construction.
To this end, we compiled paired datasets of two styles, realistic and synthetic, where the realistic dataset is large-scale and diverse, while the synthetic dataset is miniature but fine-grained and accurate.

The realistic dataset is constructed from a large number of binocular videos with a wide range of scene and motion diversity.

We cut the videos according to scene changes, and apply the state-of-the-art video stereo matching method, , BiDAStereo <cit.>, to generate temporally consistent depth sequences.

Finally, we obtained ∼200K paired video-depth sequences with the length of 50-200 frames.

The synthetic dataset is a combination of the DynamicReplica <cit.> and MatrixCity <cit.> datasets, which contains ∼3K fine-grained depth annotations with a length of 150 frames.



Challenges of variable long temporal context.
Different from image depth estimation which can determine the distribution of relative depth from a single frame, the video depth estimation requires a long temporal context to arrange the depth distributions accurately for the entire video and keep the temporal consistency.

Besides, the model should support variable-length estimation as the length of open-world videos may vary significantly.

However, existing open-source video diffusion models can only generate a fixed small number of frames at a time, , 25 frames in SVD <cit.>.

It is non-trivial to adapt the pre-trained model to meet this requirement, as directly fine-tuning it with long sequences is memory-consuming, for example, a modern GPU with 40GB memory can only support the training of a 25-frame sequence in SVD.






Three-stage training.
Considering both the two-style paired datasets and the long temporal context requirement, we design a three-stage training strategy to harvest the variety of video content, the precise depth details, as well as the support for long and variable sequences.

As shown in Fig. <ref>, we train our DepthCrafter from the pre-trained SVD in three stages.

We first train it on our large realistic dataset to adapt the model to the video-to-depth generation task.

The sequence length in this stage is randomly sampled from [1, 25] frames, such that the model can learn to generate depth sequences with variable lengths.

In the second stage, we only fine-tune the temporal layers of the model still on our large realistic dataset, with the sequence length randomly sampled from [1, 110] frames.

The reason why we only fine-tune the temporal layers is that the temporal layers are more sensitive to the sequence length while the spatial layers are already adapted to the video-to-depth generation task in the first stage, and doing so significantly reduces memory consumption compared to fine-tuning the full model.

The long temporal context in this stage enables the model to precisely arrange the entire depth distributions for long and variable sequences.

In the third stage, we fine-tune the spatial layers of the model on our small synthetic dataset, with a fixed sequence length of 45 frames since the model has already learned to generate depth sequences with variable lengths in the first two stages and tuning the spatial layers would not affect the temporal context.


As the depth annotations in the synthetic dataset are more accurate and fine-grained, the model can learn more precise depth details in this stage.

The three-stage training strategy makes our DepthCrafter capable of generating high-quality depth sequences for open-world videos with variable lengths.











 §.§ Inference for Extremely Long Videos


Although the model can estimate depth sequences up to the length of 110 frames after training, it is still far from long enough for open-world videos, which can even contain hundreds or thousands of frames.

To this end, we design an inference strategy to infer extremely long depth sequences in a segment-wise manner and seamlessly stitch them together to form the entire depth sequence.

As shown in Fig. <ref>, we first divide the video into overlapped segments, whose lengths are up to 110 frames.

Then we estimate the depth sequences for each segment.

Rather than purely initializing the input latent with Gaussian noise ϵ∼𝒩(0, σ_max^2𝐈), we initialize the latent of the overlapped frames by adding noise to the denoised latent from the previous segment, to anchor the scale and shift of the depth distributions.

Finally, to further ensure the temporal smoothness across segments, we craft a mortise-and-tenon style latent interpolation strategy to stitch consecutive segments together, inspired by <cit.>.

Specifically, we interpolate the latent of the overlapped frames o_i from the two segments with the interpolation weights w_i and 1-w_i, respectively, where w_i is linearly decreased from 1 to 0.

The final estimated depth sequence is obtained by decoding the stitched latent segments with the decoder 𝒟 in the VAE.

With the training and inference strategies, our DepthCrafter can generate temporally consistent long depth sequences for open-world videos.









§ EXPERIMENTS





 §.§ Implementation

We implemented our DepthCrafter based on SVD <cit.>, using the diffusers <cit.> library. 

We train our model at the resolution of 320 × 640 for efficiency, but we can estimate depth sequences at any resolution, , 576 × 1024, during inference.

We use the Adam optimizer <cit.> with a learning rate of 1×10^-5 and a batch size of 8.

The number of iterations in the three stages of training is 80K, 40K, and 10K, respectively.

We employed eight NVIDIA A100 GPUs for training, with a total training time of about five days.

We also adopt the classifier-free guidance <cit.> to improve the details of the generated depth sequences.

The number of denoising steps is set to 25 for all experiments.

















 §.§ Evaluation


Evaluation datasets. 
We evaluate our model on four video datasets, a single-image dataset, as well as the DAVIS dataset <cit.> and in-the-wild videos for qualitative results.

All the evaluation videos were not included in our training process.

Sintel <cit.> is a synthetic dataset with precise depth labels, featuring dynamic scenes with diverse content and camera motion.

It contains 23 sequences with the length of around 50 frames each in the training set.

ScanNet v2 <cit.> is an indoor dataset with depth maps obtained from a Kinect sensor.

For evaluation purposes, we employed the test set, which includes 100 RGB-D video sequences of various scenes.

We extracted 90 frames from each sequence at a rate of 15 frames per second.

Since ScanNet v2 contains only static indoor scenes, we further introduced 5 dynamic indoor RGB-D videos with a length of 110 frames each from the Bonn <cit.> dataset to better evaluate the performance of our model on dynamic scenes.

KITTI <cit.> is a street-scene outdoor dataset for autonomous driving, with sparse metric depths captured by a LiDAR sensor.

We adopted the validation set, which includes 13 scenes, and extracted 13 videos from it with a length of 110 frames each.

Besides, we also evaluated our model for single-image depth estimation on the NYU-v2 <cit.> dataset, which contains 654 images in the test split.

These datasets cover a wide range of scenes, including synthetic and realistic scenes, indoor and outdoor scenes, and static and dynamic scenes, to evaluate the generalization ability of our model across various open-world scenarios.



    




Evaluation metrics.
Following conventional practice in relative depth estimation <cit.>, we align the estimated depth maps with the ground truth using a scale and shift before calculating the metrics.

Different from previous methods that optimize the scale and shift individually for each frame, we optimize a shared scale and shift across the entire video, which is more challenging but necessary for video depth estimation to ensure temporal consistency.

We calculate two metrics: AbsRel ↓ (absolute relative error: |𝐝̂-𝐝| / 𝐝) and δ_1↑ (percentage of max(𝐝/𝐝̂, 𝐝̂/𝐝) < 1.25), which are widely used in the literature <cit.>.


Quantitative results.
We compare our DepthCrafter with the representative methods for both single-image and video depth estimation, Marigold <cit.>, Depth-Anything <cit.>, Depth-Anything-V2 <cit.>, NVDS <cit.>, and ChronoDepth <cit.>.

As shown in Tab. <ref>, our DepthCrafter achieves state-of-the-art performance in all four video datasets, thanks to the powerful open-world video undederstanding capability of the video diffusion models and the three-stage training strategy that leverages both realistic and synthetic datasets.

For Sintel and KITTI, characterized by significant camera motion and fast-moving objects, our DepthCrafter outperforms the previous strongest Depth-Anything (V2) model tremendously in terms of both the AbsRel and δ_1 metrics, (0.697-0.564)/0.564=23.6% improvement in δ_1 on Sintel.

For indoor datasets like Scannet and Bonn, featuring minimal camera motion and roughly the same room scales, Depth-Anything has exhibited strong performance. 

Nevertheless, we still have some performance enhancements over Depth-Anything, (0.130-0.125)/0.130=3.8% improvement in AbsRel on Scannet.

Note that the sequence length of these datasets varies from 50 to 110 frames, and our model can generalize well across different video lengths.




Qualitative results.
To further demonstrate the effectiveness of our model, we present the qualitative results on video depth estimation from the DAVIS dataset <cit.>, Sora generated videos <cit.>, and open-world videos, including human actions, animals, architectures, cartoons, and games, where the sequence length varies from 90 to 195 frames.

As shown in Fig. <ref>, we show the temporal profiles of the estimated depth sequences in the red line position by slicing the depth values along the time axis, to better visualize the temporal consistency of the estimated depth sequences, following the practice in <cit.>.

We can observe that our DepthCrafter can produce temporally consistent depth sequences with fine-grained details across various open-world videos, while both NVDS and Depth-Anything exhibit zigzag artifacts in the temporal profiles, indicating the flickering artifacts in the estimated depth sequences.

These results demonstrate the effectiveness of our DepthCrafter in generating temporally consistent long depth sequences with high-fidelity details for open-world videos.
 



Single-image depth estimation.
Although our model is designed for video depth estimation, it can also perform single-image depth estimation, as our DepthCrafter can estimate video depth of any length.

As shown in Tab. <ref>, our DepthCrafter achieves competitive performance in single-image depth estimation on the NYU-v2 dataset.

Since the depth labels in the NYU-v2 dataset are sparse and noisy, we also provide the qualitative results in Fig. <ref> to demonstrate the effectiveness of our model in estimating depth maps from static images.

We can observe that our DepthCrafter can even produce more detailed depth maps than Depth-Anything-V2, which is the existing state-of-the-art single-image depth estimation model.

These results demonstrate the ability of our DepthCrafter for processing both video and single-image depth estimation tasks.







 §.§ Ablation Studies








Effectiveness of the three-stage training strategy.
We first ablate the effectiveness of the three-stage training strategy by evaluating the performance of our model at the end of each stage on the Sintel dataset <cit.>, since it contains precise depth annotations on dynamic scenes.

From Tab. <ref>, we can observe that the performance of our model almost improves as the training progresses, indicating the effectiveness of the three-stage training strategy.

Although the AbsRel metric slightly increases in stage 2, the δ_1 metric consistently improves, and stage 2 is essential for supporting the long temporal context up to 110 frames.

Effectiveness of the inference strategy.
To ablate the effectiveness of our inference strategy components, we consider these variants:
baseline, which independently infers each segment and directly averages the overlapped frames; 
+ initialization, which contains the same initialization of overlapped latents as our method, but without the stitching process;
+ initialization & stitching, which is our full method.

We visually compare the temporal profiles of the estimated depth sequences of these variants in Fig. <ref>.

We can observe the overlapped jaggies in both the static regions (pointed by the yellow arrow) and the dynamic regions (pointed by the green arrow) in temporal profiles of the “baseline” method, which indicates the flickering artifacts.

The “+ initialization” method can alleviate the flickering artifacts in the static regions, but still has jaggies in the dynamic regions, while our full method can produce smooth depth sequences in both static and dynamic regions.





 §.§ Applications





Our DepthCrafter can facilitate various downstream applications, , foreground matting, depth slicing, fog effects, and depth-conditioned video generation, by providing temporally consistent depth sequences with fine-grained details for open-world videos. 

We show example results of fog effects and depth-conditioned video generation in Fig. <ref>, while more visual effects results are available in our website.

For the fog effect, we blend the fog map with the input video frames based on the depth values to simulate varying transparency levels in fog.

And many recent conditioned video generation models <cit.> employ depth maps as the structure conditions for video generation or editing.

We adopt Control-A-Video <cit.> and video depth of our method as conditions to generate a video with prompts “a rider walking through stars, artstation”.

The visual effects of these applications rely heavily on the accuracy and consistency of the video depth, which demonstrates the wide applicability of our DepthCrafter in various downstream tasks.










§ CONCLUSION
 

We present DepthCrafter, a novel method for open-world video depth estimation by leveraging video diffusion models.

It can generate temporally consistent depth sequences with fine-grained details for video width diverse content, motion, and camera movement, without requiring any additional information.

It also supports videos of variable lengths, ranging from one frame (static image) to extremely long videos.

This is achieved through our meticulously designed three-stage training strategy, compiled paired video-depth datasets, and an inference strategy.

Extensive evaluations have demonstrated that DepthCrafter achieves state-of-the-art performance in open-world video depth estimation under zero-shot settings.

It also facilitates various downstream applications, including depth-based visual effects and conditional video generation.

There are still some limitations to be addressed in the future, such as the expensive computation and memory cost, which is due to the large model size and the iterative denoising process in the diffusion model.






    ieeenat_fullname
    





