---
license: apache-2.0
language:
- en
tags:
- magicdrive
- image-generation
- controllable-generation
- autonomous-driving
---

# MagicDrive-t

[ArXiv](https://arxiv.org/abs/2310.02601) | [Web](https://gaoruiyuan.com/magicdrive/) | [License](https://github.com/cure-lab/MagicDrive/blob/video/LICENSE) | [GitHub](https://github.com/cure-lab/MagicDrive/tree/video) (`video`-branch)

This repository contains 60-frame driving-view video generation model checkpoint.

- We train this model by loading from the 16-frame 224x400 checkpoint (we interpolate the temporal positional embeddings from 16 to 60), and train for 80k steps.
- Please check `hydra/overrides.yaml` for command overrides in the training config.

> MagicDrive: Street View Generation with Diverse 3D Geometry Control <br>
> [Ruiyuan Gao](https://gaoruiyuan.com/)<sup>1\*</sup>, [Kai Chen](https://kaichen1998.github.io/)<sup>2\*</sup>, [Enze Xie](https://xieenze.github.io/)<sup>3^</sup>, [Lanqing Hong](https://scholar.google.com.sg/citations?user=2p7x6OUAAAAJ&hl=en)<sup>3</sup>, [Zhenguo Li](https://scholar.google.com/citations?user=XboZC1AAAAAJ&hl=en)<sup>3</sup>, [Dit-Yan Yeung](https://sites.google.com/view/dyyeung)<sup>2</sup>, [Qiang Xu](https://cure-lab.github.io/)<sup>1^</sup><br>
> <sup>1</sup>CUHK <sup>2</sup>HKUST <sup>3</sup>Huawei Noah's Ark Lab <br>
> <sup>\*</sup>Equal Contribution <sup>^</sup>Corresponding Authors

<div>
  <p class="top_p">Generated video A.</p>
  <video id="4470_rm2" muted preload controls playsinline autoplay loop height="100%">
    <source src="https://gaoruiyuan.com/magicdrive/static/60f/4470_rm2.mp4" type="video/mp4" />
  </video>
  <p class="top_p">More cars in A on the right and front.</p>
  <video id="4470_rm15" muted preload controls playsinline autoplay loop height="100%">
    <source src="https://gaoruiyuan.com/magicdrive/static/60f/4470_rm15.mp4" type="video/mp4" />
  </video>
  <p class="top_p"><br></p>

  <p class="top_p">Generated video B</p>
  <video id="5664_sunny" muted preload controls playsinline autoplay loop height="100%">
    <source src="https://gaoruiyuan.com/magicdrive/static/60f/5664_sunny.mp4" type="video/mp4" />
  </video>
  <p class="top_p">Change B to night.</p>
  <video id="5664_night" muted preload controls playsinline autoplay loop height="100%">
    <source src="https://gaoruiyuan.com/magicdrive/static/60f/5664_night.mp4" type="video/mp4" />
  </video>
  <p class="top_p">Remove the car on the left in B.</p>
  <video id="5664_sunny_n1" muted preload controls playsinline autoplay loop height="100%">
    <source src="https://gaoruiyuan.com/magicdrive/static/60f/5664_sunny_n1.mp4" type="video/mp4" />
  </video>
  <p class="top_p"><br></p>

  <p class="top_p">Generated video C.</p>
  <video id="9357_sunny_n1" muted preload controls playsinline autoplay loop height="100%">
    <source src="https://gaoruiyuan.com/magicdrive/static/60f/9357_sunny_n1.mp4" type="video/mp4" />
  </video>
  <p class="top_p">Remove the car on the right in C.</p>
  <video id="9357_sunny_n2" muted preload controls playsinline autoplay loop height="100%">
    <source src="https://gaoruiyuan.com/magicdrive/static/60f/9357_sunny_n2.mp4" type="video/mp4" />
  </video>
  <p class="top_p">Change C to a rainy day.</p>
  <video id="9357_rain_n1" muted preload controls playsinline autoplay loop height="100%">
    <source src="https://gaoruiyuan.com/magicdrive/static/60f/9357_rain_n1.mp4" type="video/mp4" />
  </video>
  <p class="top_p">Change C to a rainy day and remove the car on the right.</p>
  <video id="9357_rain_n2" muted preload controls playsinline autoplay loop height="100%">
    <source src="https://gaoruiyuan.com/magicdrive/static/60f/9357_rain_n2.mp4" type="video/mp4" />
  </video>
</div>

For more information, please refer to our GitHub: https://github.com/cure-lab/MagicDrive/tree/video (i.e., the video branch of MagicDrive repo)