Init

README.md

@@ -8,12 +8,16 @@ tags:
 - video generation
 library_name: diffusers
 ---
+
 # MoviiGen 1.1
------
 
-[**MoviiGen 1.1: Towards Cinematic-Quality Video Generative Models**]("") <be>
+[![HuggingFace](https://img.shields.io/badge/🤗%20Hugging%20Face-Model-blue)](https://huggingface.co/ZuluVision/MoviiGen1.1)
+[![GitHub stars](https://img.shields.io/github/stars/ZulutionAI/MoviiGen1.1?style=social)](https://github.com/ZulutionAI/MoviiGen1.1/stargazers)
+
 
-In this repository, we present **MoviiGen 1.1**, a cutting-edge video generation model that excels in cinematic aesthetics and visual quality. This model is a fine-tuning model based on the **Wan2.1**. Based on comprehensive evaluations by 11 professional filmmakers and AIGC creators, including industry experts, across 60 aesthetic dimensions, **MoviiGen 1.1** demonstrates superior performance in key cinematic aspects:
+[**MoviiGen 1.1: Towards Cinematic-Quality Video Generative Models**]("https://huggingface.co/ZuluVision/MoviiGen1.1") <be>
+
+In this repository, we present **MoviiGen 1.1**, a cutting-edge video generation model that excels in cinematic aesthetics and visual quality. This model is a fine-tuning model based on the Wan2.1. Based on comprehensive evaluations by 11 professional filmmakers and AIGC creators, including industry experts, across 60 aesthetic dimensions, **MoviiGen 1.1** demonstrates superior performance in key cinematic aspects:
 
 - 👍 **Superior Cinematic Aesthetics**: **MoviiGen 1.1** outperforms competitors in three critical dimensions: atmosphere creation, camera movement, and object detail preservation, making it the preferred choice for professional cinematic applications. 
 - 👍 **Visual Coherence & Quality**: MoviiGen 1.1 excels in clarity (+14.6%) and realism (+4.3%), making it ideal for high-fidelity scenarios such as real-scene conversion and portrait detail. Wan2.1 stands out in smoothness and overall visual harmony, better suited for tasks emphasizing composition, coherence, and artistic style. Both models have close overall scores, so users can select MoviiGen 1.1 for clarity and realism, or Wan2.1 for style and structural consistency.
@@ -23,30 +27,26 @@ In this repository, we present **MoviiGen 1.1**, a cutting-edge video generation
 
 This repository features our latest model, which establishes new benchmarks in cinematic video generation. Through extensive evaluation by industry professionals, it has demonstrated exceptional capabilities in creating high-quality visuals with natural motion dynamics and consistent aesthetic quality, making it an ideal choice for professional video production and creative applications.
 
-
 ## Video Demos
 
-| <video width="320" controls><source src="assets/79_1920*1056_seed3732225395.mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="assets/150_1920*1056_seed1674457713.mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="assets/143_1920*1056_seed3114534932.mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="assets/94_1920*1056_seed3693446494.mp4" type="video/mp4">Your browser does not support the video tag.</video> |
+| <video width="320" controls><source src="https://huggingface.co/ZuluVision/MoviiGen1.1/resolve/main/assets/79_1920*1056_seed3732225395.mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="https://huggingface.co/ZuluVision/MoviiGen1.1/resolve/main/assets/150_1920*1056_seed1674457713.mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="https://huggingface.co/ZuluVision/MoviiGen1.1/resolve/main/assets/143_1920*1056_seed3114534932.mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="https://huggingface.co/ZuluVision/MoviiGen1.1/resolve/main/assets/94_1920*1056_seed3693446494.mp4" type="video/mp4">Your browser does not support the video tag.</video> |
 |--------|--------|--------|--------|
+| <video width="320" controls><source src="https://huggingface.co/ZuluVision/MoviiGen1.1/resolve/main/assets/23_1920*1056_seed3934691816.mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="https://huggingface.co/ZuluVision/MoviiGen1.1/resolve/main/assets/13_1920*1056..mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="https://huggingface.co/ZuluVision/MoviiGen1.1/resolve/main/assets/26_1920*1056..mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="https://huggingface.co/ZuluVision/MoviiGen1.1/resolve/main/assets/39_1920*1056..mp4" type="video/mp4">Your browser does not support the video tag.</video> |
+||
+| <video width="320" controls><source src="https://huggingface.co/ZuluVision/MoviiGen1.1/resolve/main/assets/100_1920*1056_seed2949593166.mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="https://huggingface.co/ZuluVision/MoviiGen1.1/resolve/main/assets/54_1920*1056..mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="https://huggingface.co/ZuluVision/MoviiGen1.1/resolve/main/assets/107_1920*1056_seed525896597.mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="https://huggingface.co/ZuluVision/MoviiGen1.1/resolve/main/assets/94_1920*1056_seed3693446494.mp4" type="video/mp4">Your browser does not support the video tag.</video> |
 
-| <video width="320" controls><source src="assets/23_1920*1056_seed3934691816.mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="assets/13_1920*1056..mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="assets/26_1920*1056..mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="assets/39_1920*1056..mp4" type="video/mp4">Your browser does not support the video tag.</video> |
-|--------|--------|--------|--------|
-
-| <video width="320" controls><source src="assets/100_1920*1056_seed2949593166.mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="assets/54_1920*1056..mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="assets/107_1920*1056_seed525896597.mp4" type="video/mp4">Your browser does not support the video tag.</video> | <video width="320" controls><source src="assets/94_1920*1056_seed3693446494.mp4" type="video/mp4">Your browser does not support the video tag.</video> |
-|--------|--------|--------|--------|
 
 ## 🔥 Latest News!!
 
-* Feb 22, 2025: 👋 We've released the inference code and weights of Wan2.1.
+* May 12, 2025: 👋 We've released the inference code and **training code** and weights of MoviiGen1.1.
 
-
-## Quickstart
+## 💡 Quickstart
 
 #### Installation
 Clone the repo:
 ```
-git clone https://github.com/Wan-Video/Wan2.1.git
-cd Wan2.1
+git clone https://github.com/ZulutionAI/MoviiGen1.1.git
+cd MoviiGen1.1
 ```
 
 Install dependencies:
@@ -55,46 +55,91 @@ Install dependencies:
 pip install -r requirements.txt
 ```
 
-
 #### Model Download
 
-| T2V-14B       |      🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B)          | Supports both 720P and 1080P
-
-> 💡Note: The 14B model is capable of generating videos at 720P resolution. However, due to limited training at this resolution, the results are generally less stable compared to 480P. For optimal performance, we recommend using 480P resolution.
-
+T2V-14B  Model: 🤗 [Huggingface](https://huggingface.co/ZuluVision/MoviiGen1.1) 
+MoviiGen1.1 model supports both 720P and 1080P.
 
 Download models using huggingface-cli:
 ```
 pip install "huggingface_hub[cli]"
-huggingface-cli download Wan-AI/Wan2.1-T2V-14B --local-dir ./Wan2.1-T2V-14B
+huggingface-cli download ZuluVision/MoviiGen1.1 --local-dir ./MoviiGen1.1
 ```
 
-#### Run Text-to-Video Generation
+## 🛠️ Training
+
+### Training Framework
+
+Our training framework is built on [FastVideo](https://github.com/hao-ai-lab/FastVideo), with custom implementation of sequence parallel to optimize memory usage and training efficiency. The sequence parallel approach allows us to distribute the computational load across multiple GPUs, enabling efficient training of large-scale video generation models.
+
+#### Key Features:
 
-This repository supports two Text-to-Video models (14B) and two resolutions (720P and 1080P). The parameters and configurations for these models are as follows:
+- **Sequence Parallel & Ring Attention**: Our custom implementation divides the temporal dimension across multiple GPUs, reducing per-device memory requirements while maintaining model quality.
+- **Efficient Data Loading**: Optimized data pipeline for handling high-resolution video frames (Latent Cache and Text Embedding Cache). 
+- **Multi Resolution Training Bucket**: Support for training at multiple resolutions.
+- **Mixed Precision Training**: Support for BF16/FP16 training to accelerate computation.
+- **Distributed Training**: Seamless multi-node, multi-GPU training support.
 
+### Data Preprocessing
+
+We cache the videos and corresponding text prompts as latents and text embeddings to optimize the training process. This preprocessing step significantly improves training efficiency by reducing computational overhead during the training phase.
+
+```bash
+cd scripts/data_preprocess
+bash scripts/data_preprocess/preprocess.sh
 ```
-python generate.py  --task t2v-14B --size 1920*1080 --ckpt_dir ./MoviiGen1.1 --prompt "Indoors, in a hospital hallway during the daytime, a middle-aged Caucasian female doctor with her brown hair in a bun, wearing a white doctor's coat over a grey shirt, a gold necklace, and stud earrings, with a blue lanyard and glasses around her neck, is conversing with a man whose back is to the camera (only his back and part of his profile are visible; he wears a dark jacket over a light-colored hooded sweatshirt). She alternates between looking down and looking up at him, her expression serious and concerned. The hallway background features beige walls and dark door frames, looking relatively empty, with natural light entering from a window at the end. The camera employs a static medium close-up shot, filmed from a slightly low angle, with the focus consistently on the doctor. The overall color palette is cool-toned, with a slightly greyish tint. Soft, primarily natural side lighting from the background window creates a calm and serious atmosphere. Visual characteristics include soft lighting, digital noise, high definition, a slightly blue tint, indoor lighting, modern aesthetic, and low saturation."
+Example Data Format:
+
+training_data.json
+```json
+[
+    {
+        "cap": "your prompt",
+        "path": "path/to/your/video.mp4",
+        "resolution": {
+            "width": 3840,
+            "height": 2160
+        },
+        "fps": 23.976023976023978,
+        "duration": 1.4180833333333331
+    },
+    ...
+]
+```
+merge.txt:
+```txt
+relative_path_to_json_dir, training_data.json
 ```
 
-##### Runing local gradio
+Output Json:
+
+video_caption.json
+```json
+[
+    {
+        "latent_path": "path/to/your/latent.pt",
+        "prompt_embed_path": "path/to/your/prompt_embed.pt",
+        "length": 12
+    },
+    ...
+]
+```
 
+### Train
+```bash
+bash scripts/train/finetune.sh
 ```
-cd gradio
-python t2v_14B_singleGPU.py --ckpt_dir ./MoviiGen1.1
+
+**When multi-node training, you need to set the number of nodes and the number of processes per node manually.** We provide a sample script for multi-node training.
+
+```bash
+bash scripts/bash/finetune_multi_node.sh
 ```
 
+
 ## Manual Evaluation
 
 <div style="display: flex; justify-content: space-between;">
     <div style="flex: 1; margin-right: 10px;"><img src="assets/movie_asethetic.png" alt="Movie Aesthetic Evaluation" style="width: 100%;" /></div>
     <div style="flex: 1;"><img src="assets/visual_quality.png" alt="Movie Aesthetic Evaluation" style="width: 100%;" /></div>
-</div>
-
-
-
-## Community Contributions
-- [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) provides more support for Wan, including video-to-video, FP8 quantization, VRAM optimization, LoRA training, and more. Please refer to [their examples](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/wanvideo).
-
-
-
+</div>