Update README.md

README.md

@@ -43,7 +43,7 @@ torchrun --nproc_per_node $parallel run_parallel.py \
     --infer_steps 50 \
     --save_path ./results \
     --cfg_scale 9.0 \
-    --motion_score 5 \
+    --motion_score 5.0 \
     --time_shift 12.573
 ```
 ## Motion Control
@@ -115,135 +115,167 @@ The default motion_score = 5 is suitable for general use. If you need more stabi
   </tr>
 </table>
 
+## 5. Benchmark
 
-## Table of Contents
-
-1. [Introduction](#1-introduction)
-2. [Model Summary](#2-model-summary)
-3. [Model Download](#3-model-download)
-4. [Model Usage](#4-model-usage)
-5. [Benchmark](#5-benchmark)
-6. [Online Engine](#6-online-engine)
-7. [Citation](#7-citation)
-8. [Acknowledgement](#8-ackownledgement)
-
-## 1. Introduction
-We present **Step-Video-T2V**, a state-of-the-art (SoTA) text-to-video pre-trained model with 30 billion parameters and the capability to generate videos up to 204 frames. To enhance both training and inference efficiency, we propose a deep compression VAE for videos, achieving 16x16 spatial and 8x temporal compression ratios. Direct Preference Optimization (DPO) is applied in the final stage to further enhance the visual quality of the generated videos. Step-Video-T2V's performance is evaluated on a novel video generation benchmark, **Step-Video-T2V-Eval**, demonstrating its SoTA text-to-video quality compared to both open-source and commercial engines.
-
-## 2. Model Summary
-In Step-Video-T2V, videos are represented by a high-compression Video-VAE, achieving 16x16 spatial and 8x temporal compression ratios. User prompts are encoded using two bilingual pre-trained text encoders to handle both English and Chinese. A DiT with 3D full attention is trained using Flow Matching and is employed to denoise input noise into latent frames, with text embeddings and timesteps serving as conditioning factors. To further enhance the visual quality of the generated videos, a video-based DPO approach is applied, which effectively reduces artifacts and ensures smoother, more realistic video outputs.
-
-<p align="center">
-  <img width="80%" src="assets/model_architecture.png">
-</p>
-
-### 2.1. Video-VAE
-A deep compression Variational Autoencoder (VideoVAE) is designed for video generation tasks, achieving 16x16 spatial and 8x temporal compression ratios while maintaining exceptional video reconstruction quality. This compression not only accelerates training and inference but also aligns with the diffusion process's preference for condensed representations.
-
-<p align="center">
-  <img width="70%" src="assets/dcvae.png">
-</p>
-
-### 2.2. DiT w/ 3D Full Attention
-Step-Video-T2V is built on the DiT architecture, which has 48 layers, each containing 48 attention heads, with each head’s dimension set to 128. AdaLN-Single is leveraged to incorporate the timestep condition, while QK-Norm in the self-attention mechanism is introduced to ensure training stability. Additionally, 3D RoPE is employed, playing a critical role in handling sequences of varying video lengths and resolutions.
-
-<p align="center">
-  <img width="80%" src="assets/dit.png">
-</p>
-
-### 2.3. Video-DPO
-In Step-Video-T2V, we incorporate human feedback through Direct Preference Optimization (DPO) to further enhance the visual quality of the generated videos. DPO leverages human preference data to fine-tune the model, ensuring that the generated content aligns more closely with human expectations. The overall DPO pipeline is shown below, highlighting its critical role in improving both the consistency and quality of the video generation process.
-
-<p align="center">
-  <img width="100%" src="assets/dpo_pipeline.png">
-</p>
-
-
-
-## 3. Model Download
-| Models   | 🤗Huggingface    |  🤖Modelscope |
-|:-------:|:-------:|:-------:|
-| Step-Video-T2V | [download](https://huggingface.co/stepfun-ai/stepvideo-t2v) | [download](https://www.modelscope.cn/models/stepfun-ai/stepvideo-t2v)
-| Step-Video-T2V-Turbo (Inference Step Distillation) | [download](https://huggingface.co/stepfun-ai/stepvideo-t2v-turbo) | [download](https://www.modelscope.cn/models/stepfun-ai/stepvideo-t2v-turbo)
+We build [Step-Video-TI2V-Eval](https://github.com/stepfun-ai/Step-Video-T2V/blob/main/benchmark/Step-Video-T2V-Eval), a new benchmark designed for the text-driven image-to-video generation task. The dataset comprises 178 real-world and 120 anime-style prompt-image pairs, ensuring broad coverage of diverse user scenarios. To achieve comprehensive representation, we developed a fine-grained schema for data collection in both categories.
 
 
-## 4. Model Usage
-### 📜 4.1  Requirements
+<table border="0" style="width: 100%; text-align: center; margin-top: 1px;">
+  <tr>
+    <th style="width: 20%;">vs. OSTopA</th>
+    <th style="width: 20%;">vs. OSTopB</th>
+    <th style="width: 20%;">vs. CSTopC</th>
+    <th style="width: 20%;">vs. CSTopD</th>
+  </tr>
+  <tr>
+    <td>37-63-79</td>
+    <td>101-48-29</td>
+    <td>41-46-73</td>
+    <td>92-51-18</td>
+  </tr>
+  <tr>
+    <td>40-35-44</td>
+    <td>94-16-10</td>
+    <td>52-35-47</td>
+    <td>87-18-17</td>
+  </tr>
+  <tr>
+    <td>46-92-39</td>
+    <td>43-71-64</td>
+    <td>45-65-50</td>
+    <td>36-77-47</td>
+  </tr>
+  <tr>
+    <td>42-61-18</td>
+    <td>50-35-35</td>
+    <td>29-62-43</td>
+    <td>37-63-23</td>
+  </tr>
+  <tr>
+    <td>52-57-49</td>
+    <td>71-40-66</td>
+    <td>58-33-69</td>
+    <td>67-33-60</td>
+  </tr>
+  <tr>
+    <td>75-17-28</td>
+    <td>67-30-24</td>
+    <td>78-17-39</td>
+    <td>68-41-14</td>
+  </tr>
+  <tr>
+    <th colspan="4">Total Score</th>
+  </tr>
+  <tr>
+    <td>292-325-277</td>
+    <td>426-240-228</td>
+    <td>303-258-321</td>
+    <td>387-283-179</td>
+  </tr>
+</table>
+<p style="text-align: center;"><strong>Table 1: Comparison with baseline TI2V models using Step-Video-TI2V-Eval.</strong></p>
 
-The following table shows the requirements for running Step-Video-T2V model (batch size = 1, w/o cfg distillation) to generate videos:
 
-|     Model    |  height/width/frame |  Peak GPU Memory | 50 steps w flash-attn | 50 steps w/o flash-attn |
-|:------------:|:------------:|:------------:|:------------:|:------------:|
-| Step-Video-T2V   |        544px992px204f      |  77.64 GB | 743 s | 1232 s |
-| Step-Video-T2V   |        544px992px136f      |  72.48 GB | 408 s | 605 s |
 
-* An NVIDIA GPU with CUDA support is required. 
-  * The model is tested on four GPUs.
-  * **Recommended**: We recommend to use GPUs with 80GB of memory for better generation quality.
-* Tested operating system: Linux
-* The self-attention in text-encoder (step_llm) only supports CUDA capabilities sm_80 sm_86 and sm_90
+[VBench](https://arxiv.org/html/2411.13503v1) is a comprehensive benchmark suite that deconstructs “video generation quality” into specific, hierarchical, and disentangled dimensions, each with tailored prompts and evaluation methods. We utilize the VBench-I2V benchmark to assess the performance of Step-Video-TI2V alongside other TI2V models. 
 
-### 🔧 4.2 Dependencies and Installation
-- Python >= 3.10.0 (Recommend to use [Anaconda](https://www.anaconda.com/download/#linux) or [Miniconda](https://docs.conda.io/en/latest/miniconda.html))
-- [PyTorch >= 2.3-cu121](https://pytorch.org/)
-- [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads)
-- [FFmpeg](https://www.ffmpeg.org/)
-- 
-```bash
-git clone https://github.com/stepfun-ai/Step-Video-TI2V.git
-conda create -n stepvideo python=3.10
-conda activate stepvideo
 
-cd StepFun-StepVideo
-pip install -e .
 
-```
 
-###  🚀 4.3 Inference Scripts
-- We employed a decoupling strategy for the text encoder, VAE decoding, and DiT to optimize GPU resource utilization by DiT. As a result, a dedicated GPU is needed to handle the API services for the text encoder's embeddings and VAE decoding.
-```bash
-python api/call_remote_server.py --model_dir where_you_download_dir &  ## We assume you have more than 4 GPUs available. This command will return the URL for both the caption API and the VAE API. Please use the returned URL in the following command.
 
-parallel=4  # or parallel=8
-url='127.0.0.1'
-model_dir=where_you_download_dir
-
-torchrun --nproc_per_node $parallel run_parallel.py \
-    --model_dir $model_dir \
-    --vae_url $url \
-    --caption_url $url  \
-    --ulysses_degree  $parallel \
-    --prompt "男孩笑起来" \
-    --first_image_path ./assets/demo.png \
-    --infer_steps 50 \
-    --save_path ./results \
-    --cfg_scale 9.0 \
-    --motion_score 5 \
-    --time_shift 12.573
-```
-
-###  🚀 4.4 Best-of-Practice Inference settings
-Step-Video-T2V exhibits robust performance in inference settings, consistently generating high-fidelity and dynamic videos. However, our experiments reveal that variations in inference hyperparameters can have a substantial effect on the trade-off between video fidelity and dynamics. To achieve optimal results, we recommend the following best practices for tuning inference parameters:
-
-| Models   | infer_steps   | cfg_scale  | time_shift | num_frames |
-|:-------:|:-------:|:-------:|:-------:|:-------:|
-| Step-Video-T2V | 30-50 | 9.0 |  13.0 | 204
-| Step-Video-T2V-Turbo (Inference Step Distillation) | 10-15 | 5.0 | 17.0 | 204 |
-
-
-## 5. Benchmark
-We are releasing [Step-Video-T2V Eval](https://github.com/stepfun-ai/Step-Video-T2V/blob/main/benchmark/Step-Video-T2V-Eval) as a new benchmark, featuring 128 Chinese prompts sourced from real users. This benchmark is designed to evaluate the quality of generated videos across 11 distinct categories: Sports, Food, Scenery, Animals, Festivals, Combination Concepts, Surreal, People, 3D Animation, Cinematography, and Style.
-
-## 6. Online Engine
-The online version of Step-Video-T2V is available on [跃问视频](https://yuewen.cn/videos), where you can also explore some impressive examples.
+<table border="0" style="width: 100%; text-align: center; margin-top: 1px;">
+  <tr>
+    <th style="width: 20%;">Scores</th>
+    <th style="width: 20%;">Step-Video-TI2V (motion=10)</th>
+    <th style="width: 20%;">Step-Video-TI2V (motion=5)</th>
+    <th style="width: 20%;">OSTopA</th>
+    <th style="width: 20%;">OSTopB</th>
+  </tr>
+  <tr>
+    <td><strong>Total Score</strong></td>
+    <td><strong>87.98</strong></td>
+    <td>87.80</td>
+    <td>87.49</td>
+    <td>86.77</td>
+  </tr>
+  <tr>
+    <td><strong>I2V Score</strong></td>
+    <td>95.11</td>
+    <td><strong>95.50</strong></td>
+    <td>94.63</td>
+    <td>93.25</td>
+  </tr>
+  <tr>
+    <td>Video-Text Camera Motion</td>
+    <td>48.15</td>
+    <td><strong>49.22</strong></td>
+    <td>29.58</td>
+    <td>46.45</td>
+  </tr>
+  <tr>
+    <td>Video-Image Subject Consistency</td>
+    <td>97.44</td>
+    <td><strong>97.85</strong></td>
+    <td>97.73</td>
+    <td>95.88</td>
+  </tr>
+  <tr>
+    <td>Video-Image Background Consistency</td>
+    <td>98.45</td>
+    <td>98.63</td>
+    <td><strong>98.83</strong></td>
+    <td>96.47</td>
+  </tr>
+  <tr>
+    <td><strong>Quality Score</strong></td>
+    <td><strong>80.86</strong></td>
+    <td>80.11</td>
+    <td>80.36</td>
+    <td>80.28</td>
+  </tr>
+  <tr>
+    <td>Subject Consistency</td>
+    <td>95.62</td>
+    <td><strong>96.02</strong></td>
+    <td>94.52</td>
+    <td><strong>96.28</strong></td>
+  </tr>
+  <tr>
+    <td>Background Consistency</td>
+    <td>96.92</td>
+    <td>97.06</td>
+    <td>96.47</td>
+    <td><strong>97.38</strong></td>
+  </tr>
+  <tr>
+    <td>Motion Smoothness</td>
+    <td>99.08</td>
+    <td><strong>99.24</strong></td>
+    <td>98.09</td>
+    <td>99.10</td>
+  </tr>
+  <tr>
+    <td>Dynamic Degree</td>
+    <td>48.78</td>
+    <td>36.58</td>
+    <td><strong>53.41</strong></td>
+    <td>38.13</td>
+  </tr>
+  <tr>
+    <td>Aesthetic Quality</td>
+    <td>61.74</td>
+    <td><strong>62.29</strong></td>
+    <td>61.04</td>
+    <td>61.82</td>
+  </tr>
+  <tr>
+    <td>Imaging Quality</td>
+    <td>70.17</td>
+    <td>70.43</td>
+    <td><strong>71.12</strong></td>
+    <td>70.82</td>
+  </tr>
+</table>
 
-## 7. Citation
-```
-@misc{
-}
-```
 
-## 8. Acknowledgement
-- We would like to express our sincere thanks to the [xDiT](https://github.com/xdit-project/xDiT) team for their invaluable support and parallelization strategy. 
-- Our code will be integrated into the official repository of [Huggingface/Diffusers](https://github.com/huggingface/diffusers).
-- We thank the [FastVideo](https://github.com/hao-ai-lab/FastVideo) team for their continued collaboration and look forward to launching inference acceleration solutions together in the near future.
+<p style="text-align: center;"><strong>Table 3: Comparison with two open-source TI2V models using VBench-I2V.</strong></p>