Start fresh

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,5 @@
+app1.py
+app2.py
+demo_utils1.py
+tmp
+models

README.md

@@ -0,0 +1,141 @@
+---
+title: "RelightVid"
+emoji: "💡"
+colorFrom: "blue"
+colorTo: "green"
+sdk: "gradio"  # 你的项目使用的 SDK (gradio / streamlit / docker)
+app_file: "app.py"  # 你的主程序文件
+---
+
+
+<!-- # <img src="assets/icon.png" style="vertical-align: -14px;" :height="50px" width="50px"> RelightVid -->
+# RelightVid
+
+**[RelightVid: Temporal-Consistent Diffusion Model for Video Relighting](https://arxiv.org/abs/2501.16330)**
+</br>
+[Ye Fang](https://github.com/Aleafy)\*,
+[Zeyi Sun](https://github.com/SunzeY)\*,
+[Shangzhan Zhang](https://zhanghe3z.github.io/),
+[Tong Wu](https://wutong16.github.io/),
+[Yinghao Xu](https://justimyhxu.github.io/),
+[Pan Zhang](https://panzhang0212.github.io/),
+[Jiaqi Wang](https://myownskyw7.github.io/),
+[Gordon Wetzstein](https://web.stanford.edu/~gordonwz/),
+[Dahua Lin](http://dahua.site/)
+
+<p style="font-size: 0.6em; margin-top: -1em">*Equal Contribution</p>
+<p align="center">
+<a href="https://arxiv.org/abs/2501.16330"><img src="https://img.shields.io/badge/arXiv-Paper-<color>"></a>
+<a href="https://sunzey.github.io/Make-it-Real"><img src="https://img.shields.io/badge/Project-Website-red"></a>
+<a href="https://www.youtube.com/watch?v=_j-t8592GCM"><img src="https://img.shields.io/static/v1?label=Demo&message=Video&color=orange"></a>
+<a href="" target='_blank'>
+<img src="https://visitor-badge.laobi.icu/badge?page_id=Aleafy.RelightVid&left_color=gray&right_color=blue">
+</a>
+</p>
+
+
+![Demo](./assets/demo.gif)
+
+
+## 📜 News
+🚀 [2024/6/8] We release our [inference pipeline of Make-it-Real](#⚡-quick-start), including material matching and generation of albedo-only 3D objects.
+
+🚀 [2024/6/8] [Material library annotations](#📦-data-preparation) generated by GPT-4V and [data engine](#⚡-quick-start) are released!
+
+🚀 [2024/4/26] The [paper](https://arxiv.org/abs/2404.16829) and [project page](https://sunzey.github.io/Make-it-Real) are released!
+
+## 💡 Highlights
+- 🔥 We first demonstrate that **GPT-4V** can effectively **recognize and describe materials**, allowing our model to precisely identifies and aligns materials with the corresponding components of 3D objects.
+- 🔥 We construct a **Material Library** containing thousands of materials with highly
+detailed descriptions readily for MLLMs to look up and assign.
+- 🔥 **An effective pipeline** for texture segmentation, material identification and matching, enabling the high-quality application of materials to
+3D assets.
+
+## 👨‍💻 Todo
+- [ ] Evaluation for Existed and Model-Generated Assets (both code & test assets)
+- [ ] More Interactive Demos (huggingface, jupyter) 
+- [x] Make-it-Real Pipeline Inference Code
+- [x] Highly detailed Material Library annotations (generated by GPT-4V) 
+- [x] Paper and Web Demos
+
+## 💾 Installation
+
+   ```bash
+   git clone https://github.com/Aleafy/RelightVid.git
+   cd RelightVid
+
+   conda create -n relitv python=3.10 
+   conda activate relitv
+   
+   pip install torch==2.1.2 torchvision==0.16.2 --index-url https://download.pytorch.org/whl/cu118
+   pip install -r requirements.txt
+   ```
+
+
+
+
+
+## 📦 Data Preparation
+ 1. **Annotations**: in `data/material_lib/annotations` [folder](data/material_lib/annotations), include:
+    - Highly-detailed descriptions by GPT-4V: offering thorough descriptions of the material’s visual characteristics and rich semantic information.
+    - Category-tree: Divided into a hierarchical structure with coarse and fine granularity, it includes over 80 subcategories.
+ 2. **PBR Maps**: You can download the complete PBR data collection at [Huggingface](https://huggingface.co/datasets/gvecchio/MatSynth/tree/main), or download the data used in our project at [OpenXLab](https://openxlab.org.cn/datasets/YeFang/MatSynth/tree/main) (Recommended). (If you have any questions, please refer to [issue#5](https://github.com/Aleafy/Make_it_Real/issues/5))
+ 3. **Material Images(optinal)**: You can download the material images file [here](https://drive.google.com/file/d/1ob7CV6JiaqFyjuCzlmSnBuNRkzt2qMSG/view?usp=sharing), to check and visualize the material appearance.
+
+<pre>
+Make_it_Real
+└── data
+    └── material_lib
+        ├── annotations
+        ├── mat_images
+        └── pbr_maps
+            └── train
+                ├── Ceremic
+                ├── Concrete
+                ├── ...
+                └── Wood
+</pre>
+
+
+
+## ⚡ Quick Start
+#### Inference
+```bash
+python main.py --obj_dir <object_dir> --exp_name <unique_exp_name> --api_key <your_own_gpt4_api_key>
+```
+- To ensure proper network connectivity for GPT-4V, add proxy environment settings in [main.py](https://github.com/Aleafy/Make_it_Real/blob/feb3563d57fbe18abbff8d4abfb48f71cc8f967b/main.py#L18) (optional). Also, please verify the reachability of your [API host](https://github.com/Aleafy/Make_it_Real/blob/feb3563d57fbe18abbff8d4abfb48f71cc8f967b/utils/gpt4_query.py#L68).
+- Result visualization (blender engine) is located in the `output/refine_output` dir. You can compare the result with that in `output/ori_output`. 
+
+#### Annotation Engine
+
+```bash
+cd scripts/gpt_anno
+python gpt4_query_mat.py
+```
+`Note`: Besides functinoning as annotation engine, you can also use this code ([gpt4_query_mat.py](https://github.com/Aleafy/Make_it_Real/blob/main/scripts/gpt_anno/gpt4_query_mat.py)) to test the GPT-4V connection simply.
+
+<!-- [annotation code](scripts/gpt_anno) -->
+<!-- #### Evalutation -->
+
+
+
+## ❤️ Acknowledgments
+- [MatSynth](https://huggingface.co/datasets/gvecchio/MatSynth/tree/main): a Physically Based Rendering (PBR) materials dataset, which offers extensive high-resolusion tilable pbr maps to look up.
+- [TEXTure](https://github.com/TEXTurePaper/TEXTurePaper): Wonderful text-guided texture generation model, and the codebase we built upon.
+- [SoM](https://som-gpt4v.github.io/): Draw visual cues on images to facilate GPT-4V query better.
+- [Material Palette](https://github.com/astra-vision/MaterialPalette): Excellent exploration of material extraction and generation, offers good insights and comparable setting.
+
+## ✒️ Citation
+If you find our work helpful for your research, please consider giving a star ⭐ and citation 📝
+```bibtex
+@misc{fang2024makeitreal,
+      title={Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials}, 
+      author={Ye Fang and Zeyi Sun and Tong Wu and Jiaqi Wang and Ziwei Liu and Gordon Wetzstein and Dahua Lin},
+      year={2024},
+      eprint={2404.16829},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+

app.py

@@ -0,0 +1,365 @@
+import os
+import gradio as gr
+import numpy as np
+from enum import Enum
+import db_examples
+import cv2
+
+# from demo_utils1 import *
+
+from misc_utils.train_utils import unit_test_create_model
+from misc_utils.image_utils import save_tensor_to_gif, save_tensor_to_images
+import os
+from PIL import Image
+import torch
+import torchvision
+from torchvision import transforms
+from einops import rearrange
+import imageio
+import time
+
+from torchvision.transforms import functional as F
+
+import os
+
+# 推理设置
+from pl_trainer.inference.inference import InferenceIP2PVideo
+from tqdm import tqdm
+
+# 下载文件
+os.makedirs('models', exist_ok=True)
+filename = "models/iclight_sd15_fbc.safetensors"
+
+# if not os.path.exists(filename):
+#     original_path = os.getcwd()
+#     base_path = './models'
+#     os.makedirs(base_path, exist_ok=True)
+
+#     # 直接在代码中写入 Token（注意安全风险）
+#     GIT_TOKEN = "955b8ea91095840b76fe38b90a088c200d4c813c"
+#     repo_url = f"https://YeFang:{GIT_TOKEN}@code.openxlab.org.cn/YeFang/RIV_models.git"
+
+#     try:
+#         if os.system(f'git clone {repo_url} {base_path}') != 0:
+#             raise RuntimeError("Git 克隆失败")
+#         os.chdir(base_path)
+#         if os.system('git lfs pull') != 0:
+#             raise RuntimeError("Git LFS 拉取失败")
+#     finally:
+#         os.chdir(original_path)
+
+def tensor_to_pil_image(x):
+    """
+    将 4D PyTorch 张量转换为 PIL 图像。
+    """
+    x = x.float()  # 确保张量类型为 float
+    grid_img = torchvision.utils.make_grid(x, nrow=4).permute(1, 2, 0).detach().cpu().numpy()
+    grid_img = (grid_img * 255).clip(0, 255).astype("uint8")  # 将 [0, 1] 范围转换为 [0, 255]
+    return Image.fromarray(grid_img)
+
+def frame_to_batch(x):
+    """
+    将帧维度转换为批次维度。
+    """
+    return rearrange(x, 'b f c h w -> (b f) c h w')
+
+def clip_image(x, min=0., max=1.):
+    """
+    将图像张量裁剪到指定的最小和最大值。
+    """
+    return torch.clamp(x, min=min, max=max)
+
+def unnormalize(x):
+    """
+    将张量范围从 [-1, 1] 转换到 [0, 1]。
+    """
+    return (x + 1) / 2
+
+
+# 读取图像文件
+def read_images_from_directory(directory, num_frames=16):
+    images = []
+    for i in range(num_frames):
+        img_path = os.path.join(directory, f'{i:04d}.png')
+        img = imageio.imread(img_path)
+        images.append(torch.tensor(img).permute(2, 0, 1))  # Convert to Tensor (C, H, W)
+    return images
+
+def load_and_process_images(folder_path):
+    """
+    读取文件夹中的所有图片，将它们转换为 [-1, 1] 范围的张量并返回一个 4D 张量。
+    """
+    processed_images = []
+    transform = transforms.Compose([
+        transforms.ToTensor(),
+        transforms.Lambda(lambda x: x * 2 - 1)  # 将 [0, 1] 转换为 [-1, 1]
+    ])
+    for filename in sorted(os.listdir(folder_path)):
+        if filename.endswith(".png"):
+            img_path = os.path.join(folder_path, filename)
+            image = Image.open(img_path).convert("RGB")
+            processed_image = transform(image)
+            processed_images.append(processed_image)
+    return torch.stack(processed_images)  # 返回 4D 张量
+
+def load_and_process_video(video_path, num_frames=16, crop_size=512):
+    """
+    读取视频文件中的前 num_frames 帧，将每一帧转换为 [-1, 1] 范围的张量，
+    并进行中心裁剪至 crop_size x crop_size，返回一个 4D 张量。
+    """
+    processed_frames = []
+    transform = transforms.Compose([
+        transforms.CenterCrop(crop_size),       # 中心裁剪
+        transforms.ToTensor(),
+        transforms.Lambda(lambda x: x * 2 - 1)  # 将 [0, 1] 转换为 [-1, 1]
+    ])
+    
+    # 使用 OpenCV 读取视频
+    cap = cv2.VideoCapture(video_path)
+    
+    if not cap.isOpened():
+        raise ValueError(f"无法打开视频文件: {video_path}")
+
+    frame_count = 0
+
+    while frame_count < num_frames:
+        ret, frame = cap.read()
+        if not ret:
+            break  # 视频帧读取完毕或视频帧不足
+        
+        # 转换为 RGB 格式
+        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
+        image = Image.fromarray(frame)
+        
+        # 应用转换
+        processed_frame = transform(image)
+        processed_frames.append(processed_frame)
+        
+        frame_count += 1
+
+    cap.release()  # 释放视频资源
+
+    if len(processed_frames) < num_frames:
+        raise ValueError(f"视频帧不足 {num_frames} 帧，仅找到 {len(processed_frames)} 帧。")
+
+    return torch.stack(processed_frames)  # 返回 4D 张量 (帧数, 通道数, 高度, 宽度)
+
+
+def clear_cache(output_path):
+    if os.path.exists(output_path):
+        os.remove(output_path)
+    return None
+
+
+#! 加载模型
+# 配置路径和加载模型
+config_path = 'configs/instruct_v2v_ic_gradio.yaml'
+diffusion_model = unit_test_create_model(config_path).cuda()
+
+# 加载模型检查点
+ckpt_path = 'models/pytorch_model.bin' #! change
+ckpt = torch.load(ckpt_path, map_location='cpu')
+diffusion_model.load_state_dict(ckpt, strict=False)
+
+# import pdb; pdb.set_trace()
+
+# # 更改全局临时目录
+# new_tmp_dir = "./demo/gradio_bg"
+# os.makedirs(new_tmp_dir, exist_ok=True)
+
+# import pdb; pdb.set_trace()
+
+def save_video_from_frames(image_pred, save_pth, fps=8):
+    """
+    将 image_pred 中的帧保存为视频文件。
+
+    参数:
+    - image_pred: Tensor，形状为 (1, 16, 3, 512, 512)
+    - save_pth: 保存视频的路径，例如 "output_video.mp4"
+    - fps: 视频的帧率
+    """
+    # 视频参数
+    num_frames = image_pred.shape[1]
+    frame_height, frame_width = 512, 512  # 目标尺寸
+    fourcc = cv2.VideoWriter_fourcc(*'mp4v')  # 使用 mp4 编码格式
+
+    # 创建 VideoWriter 对象
+    out = cv2.VideoWriter(save_pth, fourcc, fps, (frame_width, frame_height))
+
+    for i in range(num_frames):
+        # 反归一化 + 转换为 0-255 范围
+        pred_frame = clip_image(unnormalize(image_pred[0][i].unsqueeze(0))) * 255
+        pred_frame_resized = pred_frame.squeeze(0).detach().cpu()  # (3, 512, 512)
+        pred_frame_resized = pred_frame_resized.permute(1, 2, 0).numpy().astype("uint8")  # (512, 512, 3)
+
+        # Resize 到 256x256
+        pred_frame_resized = cv2.resize(pred_frame_resized, (frame_width, frame_height))
+
+        # 将 RGB 转为 BGR（因为 OpenCV 使用 BGR 格式）
+        pred_frame_bgr = cv2.cvtColor(pred_frame_resized, cv2.COLOR_RGB2BGR)
+
+        # 写入帧到视频
+        out.write(pred_frame_bgr)
+
+    # 释放 VideoWriter 资源
+    out.release()
+    print(f"视频已保存至 {save_pth}")
+
+
+# 伪函数占位（生成空白视频）
+def dummy_process(input_fg, input_bg):
+    # import pdb; pdb.set_trace()
+    fg_tensor = load_and_process_video(input_fg).cuda().unsqueeze(0)
+    bg_tensor = load_and_process_video(input_bg).cuda().unsqueeze(0) # (1, 16, 4, 64, 64)
+
+    cond_fg_tensor = diffusion_model.encode_image_to_latent(fg_tensor)  # (1, 16, 4, 64, 64)
+    cond_bg_tensor = diffusion_model.encode_image_to_latent(bg_tensor)
+    cond_tensor = torch.cat((cond_fg_tensor, cond_bg_tensor), dim=2)
+
+    # 初始化潜变量
+    init_latent = torch.randn_like(cond_fg_tensor)
+
+    inf_pipe = InferenceIP2PVideo(
+        diffusion_model.unet, 
+        scheduler='ddpm',
+        num_ddim_steps=20
+    )
+
+    EDIT_PROMPT = 'change the background'
+    VIDEO_CFG = 1.2
+    TEXT_CFG = 7.5
+    text_cond = diffusion_model.encode_text([EDIT_PROMPT])  # (1, 77, 768)
+    text_uncond = diffusion_model.encode_text([''])
+    latent_pred = inf_pipe(
+        latent=init_latent,
+        text_cond=text_cond,
+        text_uncond=text_uncond,
+        img_cond=cond_tensor,
+        text_cfg=TEXT_CFG,
+        img_cfg=VIDEO_CFG,
+    )['latent']
+
+    image_pred = diffusion_model.decode_latent_to_image(latent_pred) # (1,16,3,512,512)
+    output_path = os.path.join(new_tmp_dir, f"output_{int(time.time())}.mp4")
+    # clear_cache(output_path)
+    
+    save_video_from_frames(image_pred, output_path)
+    # import pdb; pdb.set_trace()
+    # fps = 8
+    # frames = []
+    # for i in range(16):
+    #     pred_frame = clip_image(unnormalize(image_pred[0][i].unsqueeze(0))) * 255
+    #     pred_frame_resized = pred_frame.squeeze(0).detach().cpu() #(3,512,512)
+    #     pred_frame_resized = pred_frame_resized.permute(1, 2, 0).detach().cpu().numpy().astype("uint8") #(512,512,3) np
+    #     Image.fromarray(pred_frame_resized).save(save_pth)
+
+    # # 生成一个简单的黑色视频作为示例
+    # output_path = os.path.join(new_tmp_dir, "output.mp4")
+    # fourcc = cv2.VideoWriter_fourcc(*'mp4v')
+    # out = cv2.VideoWriter(output_path, fourcc, 20.0, (512, 512))
+
+    # for _ in range(60):  # 生成 3 秒的视频（20fps）
+    #     frame = np.zeros((512, 512, 3), dtype=np.uint8)
+    #     out.write(frame)
+    # out.release()
+
+    return output_path
+
+# 枚举类用于背景选择
+class BGSource(Enum):
+    UPLOAD = "Use Background Video"
+    UPLOAD_FLIP = "Use Flipped Background Video"
+    LEFT = "Left Light"
+    RIGHT = "Right Light"
+    TOP = "Top Light"
+    BOTTOM = "Bottom Light"
+    GREY = "Ambient"
+
+# Quick prompts 示例
+quick_prompts = [
+    'beautiful woman',
+    'handsome man',
+    'beautiful woman, cinematic lighting',
+    'handsome man, cinematic lighting',
+    'beautiful woman, natural lighting',
+    'handsome man, natural lighting',
+    'beautiful woman, neo punk lighting, cyberpunk',
+    'handsome man, neo punk lighting, cyberpunk',
+]
+quick_prompts = [[x] for x in quick_prompts]
+
+# Gradio UI 结构
+block = gr.Blocks().queue()
+with block:
+    with gr.Row():
+        gr.Markdown("## IC-Light (Relighting with Foreground and Background Video Condition)")
+
+    with gr.Row():
+        with gr.Column():
+            with gr.Row():
+                input_fg = gr.Video(label="Foreground Video", height=370, width=370, visible=True)
+                input_bg = gr.Video(label="Background Video", height=370, width=370, visible=True)
+
+            prompt = gr.Textbox(label="Prompt")
+            bg_source = gr.Radio(choices=[e.value for e in BGSource],
+                                 value=BGSource.UPLOAD.value,
+                                 label="Background Source", type='value')
+
+            example_prompts = gr.Dataset(samples=quick_prompts, label='Prompt Quick List', components=[prompt])
+            bg_gallery = gr.Gallery(height=450, object_fit='contain', label='Background Quick List', value=db_examples.bg_samples, columns=5, allow_preview=False)
+            relight_button = gr.Button(value="Relight")
+
+            with gr.Group():
+                with gr.Row():
+                    num_samples = gr.Slider(label="Videos", minimum=1, maximum=12, value=1, step=1)
+                    seed = gr.Number(label="Seed", value=12345, precision=0)
+                with gr.Row():
+                    video_width = gr.Slider(label="Video Width", minimum=256, maximum=1024, value=512, step=64)
+                    video_height = gr.Slider(label="Video Height", minimum=256, maximum=1024, value=640, step=64)
+
+            with gr.Accordion("Advanced options", open=False):
+                steps = gr.Slider(label="Steps", minimum=1, maximum=100, value=20, step=1)
+                cfg = gr.Slider(label="CFG Scale", minimum=1.0, maximum=32.0, value=7.0, step=0.01)
+                highres_scale = gr.Slider(label="Highres Scale", minimum=1.0, maximum=3.0, value=1.5, step=0.01)
+                highres_denoise = gr.Slider(label="Highres Denoise", minimum=0.1, maximum=0.9, value=0.5, step=0.01)
+                a_prompt = gr.Textbox(label="Added Prompt", value='best quality')
+                n_prompt = gr.Textbox(label="Negative Prompt", value='lowres, bad anatomy, bad hands, cropped, worst quality')
+                normal_button = gr.Button(value="Compute Normal (4x Slower)")
+
+        with gr.Column():
+            result_video = gr.Video(label='Output Video', height=600, width=600, visible=True)
+
+    # 输入列表
+    # ips = [input_fg, input_bg, prompt, video_width, video_height, num_samples, seed, steps, a_prompt, n_prompt, cfg, highres_scale, highres_denoise, bg_source]
+    ips = [input_fg, input_bg]
+
+    # 按钮绑定处理函数
+    # relight_button.click(fn=lambda: None, inputs=[], outputs=[result_video])
+
+    relight_button.click(fn=dummy_process, inputs=ips, outputs=[result_video])
+    
+    normal_button.click(fn=dummy_process, inputs=ips, outputs=[result_video])
+
+    # 背景库选择
+    def bg_gallery_selected(gal, evt: gr.SelectData):
+        # import pdb; pdb.set_trace()
+        # img_path = gal[evt.index][0]
+        img_path = db_examples.bg_samples[evt.index]
+        video_path = img_path.replace('frames/0000.png', 'cropped_video.mp4')
+        return video_path
+
+    bg_gallery.select(bg_gallery_selected, inputs=bg_gallery, outputs=input_bg)
+
+    # 示例
+    # dummy_video_for_outputs = gr.Video(visible=False, label='Result')
+    gr.Examples(
+        fn=lambda *args: args[-1],
+        examples=db_examples.background_conditioned_examples,
+        inputs=[input_fg, input_bg, prompt, bg_source, video_width, video_height, seed, result_video],
+        outputs=[result_video],
+        run_on_click=True, examples_per_page=1024
+    )
+
+# 启动 Gradio 应用
+# block.launch(server_name='0.0.0.0', server_port=10002, share=True)
+block.launch(share=True)

configs/instruct_v2v.yaml

@@ -0,0 +1,149 @@
+expt_dir: experiments
+expt_name: instruct_v2v_ic
+trainer_args:
+  max_epochs: 10
+  accelerator: "gpu"
+  devices: [0]
+  limit_train_batches: 2048
+  limit_val_batches: 5 #! 这边限制了每个epoch只跑多少个batch的validation
+  # strategy: "ddp"
+  strategy: "deepspeed_stage_2"
+  accumulate_grad_batches: 128 #! 注意一下这个值
+  check_val_every_n_epoch: 1 #! check一下这个值是不是和记录有关。。。
+diffusion:
+  target: pl_trainer.instruct_p2p_video.InstructP2PVideoTrainerTemporal
+  params:
+    beta_schedule_args:
+      beta_schedule: scaled_linear
+      num_train_timesteps: 1000
+      beta_start: 0.00085
+      beta_end: 0.012
+      clip_sample: false
+      thresholding: false
+    prediction_type: epsilon
+    loss_fn: l2
+    optim_args:
+      lr: 1e-5
+    unet_init_weights: #! 注意一下, 完全可以从iv2v的ckpt开始train
+      - pretrained_models/instruct_pix2pix/diffusion_pytorch_model.bin # 这边sd加载的是ip2p的
+      - pretrained_models/Motion_Module/mm_sd_v15.ckpt # motion module, 推测加载的是animatediff的
+    vae_init_weights: pretrained_models/instruct_pix2pix/vqvae.ckpt
+    text_model_init_weights: pretrained_models/instruct_pix2pix/text.ckpt
+    scale_factor: 0.18215
+    guidance_scale: 5 # not used
+    ddim_sampling_steps: 20
+    text_cfg: 7.5
+    img_cfg: 1.2
+    cond_image_dropout: 0.1
+    prompt_type: edit_prompt
+unet:
+  target: modules.video_unet_temporal.unet.UNet3DConditionModel
+  params:
+    in_channels: 4 #! 8-> 4
+    out_channels: 4
+    act_fn: silu
+    attention_head_dim: 8
+    block_out_channels: 
+      - 320
+      - 640
+      - 1280
+      - 1280
+    cross_attention_dim: 768
+    down_block_types: 
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - DownBlock3D
+    up_block_types: 
+      - UpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+    downsample_padding: 1
+    layers_per_block: 2
+    mid_block_scale_factor: 1
+    norm_eps: 1e-05
+    norm_num_groups: 32
+    sample_size: 64
+    use_motion_module: true
+    motion_module_resolutions:
+    - 1
+    - 2
+    - 4
+    - 8
+    motion_module_mid_block: false
+    motion_module_decoder_only: false
+    motion_module_type: Vanilla
+    motion_module_kwargs:
+      num_attention_heads: 8
+      num_transformer_block: 1
+      attention_block_types:
+      - Temporal_Self
+      - Temporal_Self
+      temporal_position_encoding: true
+      temporal_position_encoding_max_len: 32
+      temporal_attention_dim_div: 1
+vae:
+  target: modules.kl_autoencoder.autoencoder.AutoencoderKL
+  params:
+    embed_dim: 4
+    ddconfig:
+      double_z: true
+      z_channels: 4
+      resolution: 256
+      in_channels: 3
+      out_ch: 3
+      ch: 128
+      ch_mult:
+        - 1
+        - 2
+        - 4
+        - 4
+      num_res_blocks: 2
+      attn_resolutions: []
+      dropout: 0.0
+    lossconfig:
+      target: torch.nn.Identity
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true
+data:
+  batch_size: 1
+  val_batch_size: 1
+  train:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAug
+    params: #注意修改一下training的路径，和相关加载的代码, 比如说没有meta.yaml这些参数怎么搞
+      root_dirs: 
+        - /home/fy/Code/instruct-video-to-video/data_train/Girl
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+      is_train: True
+  val:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAug
+    params:
+      root_dirs: 
+        - /home/fy/Code/instruct-video-to-video/data_train/Girl
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+callbacks:
+  - target: pytorch_lightning.callbacks.ModelCheckpoint
+    params:
+      dirpath: "${expt_dir}/${expt_name}"
+      filename: "{epoch:04d}"
+      monitor: epoch
+      mode: max
+      save_top_k: 5
+      save_last: true
+  - target: callbacks.instruct_p2p_video.InstructP2PLogger
+    params:
+      max_num_images: 1
+      # accumulate_grad_batches: 128
+    require_wandb: true
+  - target: pytorch_lightning.callbacks.DeviceStatsMonitor

configs/instruct_v2v_ic.yaml

@@ -0,0 +1,130 @@
+expt_dir: experiments
+expt_name: instruct_v2v_ic
+trainer_args:
+  max_epochs: 10
+  accelerator: "gpu"
+  devices: [0]
+  limit_train_batches: 2048
+  limit_val_batches: 5 #! 这边限制了每个epoch只跑多少个batch的validation
+  # strategy: "ddp"
+  strategy: "deepspeed_stage_2"
+  accumulate_grad_batches: 128 #! 注意一下这个值
+  check_val_every_n_epoch: 1 #! check一下这个值是不是和记录有关。。。
+diffusion:
+  target: pl_trainer.instruct_p2p_video.InstructP2PVideoTrainerTemporal
+  params:
+    beta_schedule_args:
+      beta_schedule: scaled_linear
+      num_train_timesteps: 1000
+      beta_start: 0.00085
+      beta_end: 0.012
+      clip_sample: false
+      thresholding: false
+    prediction_type: epsilon
+    loss_fn: l2
+    optim_args:
+      lr: 1e-5
+    unet_init_weights: #! 注意一下, 完全可以从iv2v的ckpt开始train
+      - unet/diffusion_pytorch_model.safetensors # iclight, unet, sf tensor
+      - pretrained_models/Motion_Module/mm_sd_v15.ckpt # motion module, 推测加载的是animatediff的
+      - pretrained_models/iclight/iclight_sd15_fbc.safetensors # iclight lora weights
+    base_path: /home/fy/.cache/huggingface/hub/models--stablediffusionapi--realistic-vision-v51/snapshots/19e3643d7d963c156d01537188ec08f0b79a514a
+    # vae_init_weights: pretrained_models/instruct_pix2pix/vqvae.ckpt
+    # text_model_init_weights: pretrained_models/instruct_pix2pix/text.ckpt #! 这两个可以直接设为None, 从from_pretrained中加载
+    scale_factor: 0.18215
+    guidance_scale: 5 # not used
+    ddim_sampling_steps: 20
+    text_cfg: 7.5
+    img_cfg: 1.2
+    cond_image_dropout: 0.1
+    prompt_type: edit_prompt
+unet:
+  target: modules.video_unet_temporal.unet.UNet3DConditionModel
+  params:
+    in_channels: 4 #! change:8->12 iclight 改为12 注意一下...
+    out_channels: 4
+    act_fn: silu
+    attention_head_dim: 8
+    block_out_channels: 
+      - 320
+      - 640
+      - 1280
+      - 1280
+    cross_attention_dim: 768
+    down_block_types: 
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - DownBlock3D
+    up_block_types: 
+      - UpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+    downsample_padding: 1
+    layers_per_block: 2
+    mid_block_scale_factor: 1
+    norm_eps: 1e-05
+    norm_num_groups: 32
+    sample_size: 64
+    use_motion_module: false #!!! 这边test iclight的时候可以不用motion module 即False
+    motion_module_resolutions:
+    - 1
+    - 2
+    - 4
+    - 8
+    motion_module_mid_block: false
+    motion_module_decoder_only: false
+    motion_module_type: Vanilla
+    motion_module_kwargs:
+      num_attention_heads: 8
+      num_transformer_block: 1
+      attention_block_types:
+      - Temporal_Self
+      - Temporal_Self
+      temporal_position_encoding: true
+      temporal_position_encoding_max_len: 32
+      temporal_attention_dim_div: 1
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true
+data:
+  batch_size: 1
+  val_batch_size: 1
+  train:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAug
+    params: #注意修改一下training的路径，和相关加载的代码, 比如说没有meta.yaml这些参数怎么搞
+      root_dirs: 
+        - /home/fy/Code/instruct-video-to-video/data_train/Girl
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+      is_train: True
+  val:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAug
+    params:
+      root_dirs: 
+        - /home/fy/Code/instruct-video-to-video/data_train/Girl
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+callbacks:
+  - target: pytorch_lightning.callbacks.ModelCheckpoint
+    params:
+      dirpath: "${expt_dir}/${expt_name}"
+      filename: "{epoch:04d}"
+      monitor: epoch
+      mode: max
+      save_top_k: 5
+      save_last: true
+  - target: callbacks.instruct_p2p_video.InstructP2PLogger
+    params:
+      max_num_images: 1
+      # accumulate_grad_batches: 128
+    require_wandb: true
+  - target: pytorch_lightning.callbacks.DeviceStatsMonitor

configs/instruct_v2v_ic_gradio.yaml

@@ -0,0 +1,81 @@
+diffusion:
+  target: pl_trainer.instruct_p2p_video.InstructP2PVideoTrainerTemporal
+  params:
+    beta_schedule_args:
+      beta_schedule: scaled_linear
+      num_train_timesteps: 1000
+      beta_start: 0.00085
+      beta_end: 0.012
+      clip_sample: false
+      thresholding: false
+    prediction_type: epsilon
+    loss_fn: l2
+    optim_args:
+      lr: 1e-5
+    # base_path: models/realistic_v51
+    base_path: stablediffusionapi/realistic-vision-v51
+    # unet_init_weights: #! 注意一下, 完全可以从iv2v的ckpt开始train
+    #   - diffusion_pytorch_model.safetensors # iclight, unet, sf tensor
+    #   - relvid_mm_sd15_fbc.pth
+    #   - iclight_sd15_fbc.safetensors # iclight lora weights
+    # base_path: stablediffusionapi/realistic-vision-v51
+    # vae_init_weights: pretrained_models/instruct_pix2pix/vqvae.ckpt
+    # text_model_init_weights: pretrained_models/instruct_pix2pix/text.ckpt #! 这两个可以直接设为None, 从from_pretrained中加载
+    scale_factor: 0.18215
+    guidance_scale: 5 # not used
+    ddim_sampling_steps: 20
+    text_cfg: 7.5
+    img_cfg: 1.2
+    cond_image_dropout: 0.1
+    prompt_type: edit_prompt
+unet:
+  target: modules.video_unet_temporal.unet.UNet3DConditionModel
+  params:
+    in_channels: 4 #! change:8->12 iclight 改为12 注意一下...
+    out_channels: 4
+    act_fn: silu
+    attention_head_dim: 8
+    block_out_channels: 
+      - 320
+      - 640
+      - 1280
+      - 1280
+    cross_attention_dim: 768
+    down_block_types: 
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - DownBlock3D
+    up_block_types: 
+      - UpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+    downsample_padding: 1
+    layers_per_block: 2
+    mid_block_scale_factor: 1
+    norm_eps: 1e-05
+    norm_num_groups: 32
+    sample_size: 64
+    use_motion_module: true #!!! 这边test iclight的时候可以不用motion module 即False
+    motion_module_resolutions:
+    - 1
+    - 2
+    - 4
+    - 8
+    motion_module_mid_block: false
+    motion_module_decoder_only: false
+    motion_module_type: Vanilla
+    motion_module_kwargs:
+      num_attention_heads: 8
+      num_transformer_block: 1
+      attention_block_types:
+      - Temporal_Self
+      - Temporal_Self
+      temporal_position_encoding: true
+      temporal_position_encoding_max_len: 32
+      temporal_attention_dim_div: 1
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true

configs/instruct_v2v_ic_inference.yaml

@@ -0,0 +1,79 @@
+diffusion:
+  target: pl_trainer.instruct_p2p_video.InstructP2PVideoTrainerTemporal
+  params:
+    beta_schedule_args:
+      beta_schedule: scaled_linear
+      num_train_timesteps: 1000
+      beta_start: 0.00085
+      beta_end: 0.012
+      clip_sample: false
+      thresholding: false
+    prediction_type: epsilon
+    loss_fn: l2
+    optim_args:
+      lr: 1e-5
+    unet_init_weights: #! 注意一下, 完全可以从iv2v的ckpt开始train
+      - unet/diffusion_pytorch_model.safetensors # iclight, unet, sf tensor
+      - pretrained_models/Motion_Module/mm_sd_v15.ckpt # motion module, 推测加载的是animatediff的
+      - pretrained_models/iclight/iclight_sd15_fbc.safetensors # iclight lora weights
+    base_path: /mnt/petrelfs/fangye/.cache/huggingface/hub/models--stablediffusionapi--realistic-vision-v51/snapshots/19e3643d7d963c156d01537188ec08f0b79a514a
+    # vae_init_weights: pretrained_models/instruct_pix2pix/vqvae.ckpt
+    # text_model_init_weights: pretrained_models/instruct_pix2pix/text.ckpt #! 这两个可以直接设为None, 从from_pretrained中加载
+    scale_factor: 0.18215
+    guidance_scale: 5 # not used
+    ddim_sampling_steps: 20
+    text_cfg: 7.5
+    img_cfg: 1.2
+    cond_image_dropout: 0.1
+    prompt_type: edit_prompt
+unet:
+  target: modules.video_unet_temporal.unet.UNet3DConditionModel
+  params:
+    in_channels: 4 #! change:8->12 iclight 改为12 注意一下...
+    out_channels: 4
+    act_fn: silu
+    attention_head_dim: 8
+    block_out_channels: 
+      - 320
+      - 640
+      - 1280
+      - 1280
+    cross_attention_dim: 768
+    down_block_types: 
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - DownBlock3D
+    up_block_types: 
+      - UpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+    downsample_padding: 1
+    layers_per_block: 2
+    mid_block_scale_factor: 1
+    norm_eps: 1e-05
+    norm_num_groups: 32
+    sample_size: 64
+    use_motion_module: true #!!! 这边test iclight的时候可以不用motion module 即False
+    motion_module_resolutions:
+    - 1
+    - 2
+    - 4
+    - 8
+    motion_module_mid_block: false
+    motion_module_decoder_only: false
+    motion_module_type: Vanilla
+    motion_module_kwargs:
+      num_attention_heads: 8
+      num_transformer_block: 1
+      attention_block_types:
+      - Temporal_Self
+      - Temporal_Self
+      temporal_position_encoding: true
+      temporal_position_encoding_max_len: 32
+      temporal_attention_dim_div: 1
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true

configs/instruct_v2v_ic_inference_hdr.yaml

@@ -0,0 +1,80 @@
+diffusion:
+  target: pl_trainer.instruct_p2p_video.InstructP2PVideoTrainerTemporalText
+  params:
+    beta_schedule_args:
+      beta_schedule: scaled_linear
+      num_train_timesteps: 1000
+      beta_start: 0.00085
+      beta_end: 0.012
+      clip_sample: false
+      thresholding: false
+    prediction_type: epsilon
+    loss_fn: l2
+    optim_args:
+      lr: 1e-5
+    unet_init_weights: #! 注意一下, 完全可以从iv2v的ckpt开始train
+      - unet/diffusion_pytorch_model.safetensors # iclight, unet, sf tensor
+      - pretrained_models/Motion_Module/mm_sd_v15.ckpt # motion module, 推测加载的是animatediff的
+      - pretrained_models/iclight/iclight_sd15_fc.safetensors # iclight lora weights
+    base_path: /mnt/petrelfs/fangye/.cache/huggingface/hub/models--stablediffusionapi--realistic-vision-v51/snapshots/19e3643d7d963c156d01537188ec08f0b79a514a
+    # vae_init_weights: pretrained_models/instruct_pix2pix/vqvae.ckpt
+    # text_model_init_weights: pretrained_models/instruct_pix2pix/text.ckpt #! 这两个可以直接设为None, 从from_pretrained中加载
+    scale_factor: 0.18215
+    guidance_scale: 5 # not used
+    ddim_sampling_steps: 20
+    text_cfg: 7.5
+    img_cfg: 1.2
+    cond_image_dropout: 0.1
+    prompt_type: edit_prompt
+    hdr_train: True
+unet:
+  target: modules.video_unet_temporal.unet.UNet3DConditionModel
+  params:
+    in_channels: 4 #! change:8->12 iclight 改为12 注意一下...
+    out_channels: 4
+    act_fn: silu
+    attention_head_dim: 8
+    block_out_channels: 
+      - 320
+      - 640
+      - 1280
+      - 1280
+    cross_attention_dim: 768
+    down_block_types: 
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - DownBlock3D
+    up_block_types: 
+      - UpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+    downsample_padding: 1
+    layers_per_block: 2
+    mid_block_scale_factor: 1
+    norm_eps: 1e-05
+    norm_num_groups: 32
+    sample_size: 64
+    use_motion_module: true #!!! 这边test iclight的时候可以不用motion module 即False
+    motion_module_resolutions:
+    - 1
+    - 2
+    - 4
+    - 8
+    motion_module_mid_block: false
+    motion_module_decoder_only: false
+    motion_module_type: Vanilla
+    motion_module_kwargs:
+      num_attention_heads: 8
+      num_transformer_block: 1
+      attention_block_types:
+      - Temporal_Self
+      - Temporal_Self
+      temporal_position_encoding: true
+      temporal_position_encoding_max_len: 32
+      temporal_attention_dim_div: 1
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true

configs/instruct_v2v_ic_inference_text.yaml

@@ -0,0 +1,79 @@
+diffusion:
+  target: pl_trainer.instruct_p2p_video.InstructP2PVideoTrainerTemporalText
+  params:
+    beta_schedule_args:
+      beta_schedule: scaled_linear
+      num_train_timesteps: 1000
+      beta_start: 0.00085
+      beta_end: 0.012
+      clip_sample: false
+      thresholding: false
+    prediction_type: epsilon
+    loss_fn: l2
+    optim_args:
+      lr: 1e-5
+    unet_init_weights: #! 注意一下, 完全可以从iv2v的ckpt开始train
+      - unet/diffusion_pytorch_model.safetensors # iclight, unet, sf tensor
+      - pretrained_models/Motion_Module/mm_sd_v15.ckpt # motion module, 推测加载的是animatediff的
+      - pretrained_models/iclight/iclight_sd15_fc.safetensors # iclight lora weights
+    base_path: /mnt/petrelfs/fangye/.cache/huggingface/hub/models--stablediffusionapi--realistic-vision-v51/snapshots/19e3643d7d963c156d01537188ec08f0b79a514a
+    # vae_init_weights: pretrained_models/instruct_pix2pix/vqvae.ckpt
+    # text_model_init_weights: pretrained_models/instruct_pix2pix/text.ckpt #! 这两个可以直接设为None, 从from_pretrained中加载
+    scale_factor: 0.18215
+    guidance_scale: 5 # not used
+    ddim_sampling_steps: 20
+    text_cfg: 7.5
+    img_cfg: 1.2
+    cond_image_dropout: 0.1
+    prompt_type: edit_prompt
+unet:
+  target: modules.video_unet_temporal.unet.UNet3DConditionModel
+  params:
+    in_channels: 4 #! change:8->12 iclight 改为12 注意一下...
+    out_channels: 4
+    act_fn: silu
+    attention_head_dim: 8
+    block_out_channels: 
+      - 320
+      - 640
+      - 1280
+      - 1280
+    cross_attention_dim: 768
+    down_block_types: 
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - DownBlock3D
+    up_block_types: 
+      - UpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+    downsample_padding: 1
+    layers_per_block: 2
+    mid_block_scale_factor: 1
+    norm_eps: 1e-05
+    norm_num_groups: 32
+    sample_size: 64
+    use_motion_module: true #!!! 这边test iclight的时候可以不用motion module 即False
+    motion_module_resolutions:
+    - 1
+    - 2
+    - 4
+    - 8
+    motion_module_mid_block: false
+    motion_module_decoder_only: false
+    motion_module_type: Vanilla
+    motion_module_kwargs:
+      num_attention_heads: 8
+      num_transformer_block: 1
+      attention_block_types:
+      - Temporal_Self
+      - Temporal_Self
+      temporal_position_encoding: true
+      temporal_position_encoding_max_len: 32
+      temporal_attention_dim_div: 1
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true

configs/instruct_v2v_ic_pexels.yaml

@@ -0,0 +1,133 @@
+expt_dir: experiments
+expt_name: instruct_v2v_ic_pexels_text_bgdrop_0.3_trystepckpt
+trainer_args:
+  max_epochs: 1
+  accelerator: "gpu"
+  devices: [0] #! change to get more cards
+  limit_train_batches: 2048
+  limit_val_batches: 1 #! 这边限制了每个epoch只跑多少个batch的validation
+  # strategy: "ddp"
+  strategy: "deepspeed_stage_2"
+  # autotune_only_on_rank_zero: true # 确保只有一个进程执行调优表操作 
+  accumulate_grad_batches: 1 #! 注意一下这个值
+  check_val_every_n_epoch: 1 #! check一下这个值是不是和记录有关。。。
+  # precision: 16 # 启用半精度 (FP16)
+diffusion:
+  target: pl_trainer.instruct_p2p_video.InstructP2PVideoTrainerTemporal
+  params:
+    beta_schedule_args:
+      beta_schedule: scaled_linear
+      num_train_timesteps: 1000
+      beta_start: 0.00085
+      beta_end: 0.012
+      clip_sample: false
+      thresholding: false
+    prediction_type: epsilon
+    loss_fn: l2
+    optim_args:
+      lr: 1e-5
+    unet_init_weights: #! 注意一下, 完全可以从iv2v的ckpt开始train
+      - unet/diffusion_pytorch_model.safetensors # iclight, unet, sf tensor
+      - pretrained_models/Motion_Module/mm_sd_v15.ckpt # motion module, 推测加载的是animatediff的
+      - pretrained_models/iclight/iclight_sd15_fbc.safetensors # iclight lora weights
+    base_path: /mnt/petrelfs/fangye/.cache/huggingface/hub/models--stablediffusionapi--realistic-vision-v51/snapshots/19e3643d7d963c156d01537188ec08f0b79a514a
+    # vae_init_weights: pretrained_models/instruct_pix2pix/vqvae.ckpt
+    # text_model_init_weights: pretrained_models/instruct_pix2pix/text.ckpt #! 这两个可以直接设为None, 从from_pretrained中加载
+    scale_factor: 0.18215
+    guidance_scale: 5 # not used
+    ddim_sampling_steps: 20
+    text_cfg: 7.5
+    img_cfg: 1.2
+    cond_image_dropout: 0.1
+    cond_text_dropout: 0.1
+    prompt_type: edit_prompt
+unet:
+  target: modules.video_unet_temporal.unet.UNet3DConditionModel
+  params:
+    in_channels: 4 #! change:8->12 iclight 改为12 注意一下...
+    out_channels: 4
+    act_fn: silu
+    attention_head_dim: 8
+    block_out_channels: 
+      - 320
+      - 640
+      - 1280
+      - 1280
+    cross_attention_dim: 768
+    down_block_types: 
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - DownBlock3D
+    up_block_types: 
+      - UpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+    downsample_padding: 1
+    layers_per_block: 2
+    mid_block_scale_factor: 1
+    norm_eps: 1e-05
+    norm_num_groups: 32
+    sample_size: 64
+    use_motion_module: true #!!! 这边test iclight的时候可以不用motion module 即False
+    motion_module_resolutions:
+    - 1
+    - 2
+    - 4
+    - 8
+    motion_module_mid_block: false
+    motion_module_decoder_only: false
+    motion_module_type: Vanilla
+    motion_module_kwargs:
+      num_attention_heads: 8
+      num_transformer_block: 1
+      attention_block_types:
+      - Temporal_Self
+      - Temporal_Self
+      temporal_position_encoding: true
+      temporal_position_encoding_max_len: 32
+      temporal_attention_dim_div: 1
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true
+data:
+  batch_size: 1
+  val_batch_size: 1
+  train:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAugPexels
+    params: # 注意修改一下training的路径，和相关加载的代码, 比如说没有meta.yaml这些参数怎么搞
+      root_dirs: 
+        - /mnt/petrelfs/fangye/test/instruct-video-to-video_1019/data_train_pexels/rmbg_data
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+      is_train: True
+  val:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAugPexels
+    params:
+      root_dirs: 
+        - /mnt/petrelfs/fangye/test/instruct-video-to-video_1019/data_train_pexels/rmbg_data
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+callbacks:
+  - target: pytorch_lightning.callbacks.ModelCheckpoint
+    params:
+      dirpath: "${expt_dir}/${expt_name}"
+      filename: "{step:06d}"
+      every_n_train_steps: 10
+      save_last: True
+      # every_n_train_steps: 10
+  - target: callbacks.instruct_p2p_video.InstructP2PLogger
+    params:
+      max_num_images: 1
+      expt_name: instruct_v2v_ic_pexels_text_bgdrop_0.3_trystepckpt
+      # accumulate_grad_batches: 128
+    require_wandb: true
+  - target: pytorch_lightning.callbacks.DeviceStatsMonitor

configs/instruct_v2v_ic_pexels_hdr.yaml

@@ -0,0 +1,147 @@
+expt_dir: experiments
+expt_name: instruct_v2v_ic_pexels_text_hdr_test_lr0.5_aug_lossc_fix_bs1 #! 注意传入log里面, 不要每次修改
+trainer_args:
+  max_epochs: 10
+  accelerator: "gpu"
+  devices: [0,1,2,3,4,5,6,7] #! change to get more cards
+  limit_train_batches: 2048
+  limit_val_batches: 3 #! 这边限制了每个epoch只跑多少个batch的validation
+  # strategy: "ddp"
+  strategy: "deepspeed_stage_2"
+  # autotune_only_on_rank_zero: true # 确保只有一个进程执行调优表操作 
+  accumulate_grad_batches: 128 #! 注意一下这个值  256->128
+  check_val_every_n_epoch: 1 #! check一下这个值是不是和记录有关。。。
+  # precision: 16 # 启用半精度 (FP16)
+diffusion:
+  target: pl_trainer.instruct_p2p_video.InstructP2PVideoTrainerTemporalText
+  params:
+    beta_schedule_args:
+      beta_schedule: scaled_linear
+      num_train_timesteps: 1000
+      beta_start: 0.00085
+      beta_end: 0.012
+      clip_sample: false
+      thresholding: false
+    prediction_type: epsilon
+    loss_fn: l2
+    optim_args:
+      lr: 1e-5 #! 原来是1e-5
+    unet_init_weights: #! 注意一下, 完全可以从iv2v的ckpt开始train
+      - unet/diffusion_pytorch_model.safetensors # iclight, unet, sf tensor
+      - pretrained_models/Motion_Module/mm_sd_v15.ckpt # motion module, 推测加载的是animatediff的
+      - pretrained_models/iclight/iclight_sd15_fc.safetensors # iclight lora weights
+    base_path: /mnt/petrelfs/fangye/.cache/huggingface/hub/models--stablediffusionapi--realistic-vision-v51/snapshots/19e3643d7d963c156d01537188ec08f0b79a514a
+    # vae_init_weights: pretrained_models/instruct_pix2pix/vqvae.ckpt
+    # text_model_init_weights: pretrained_models/instruct_pix2pix/text.ckpt #! 这两个可以直接设为None, 从from_pretrained中加载
+    scale_factor: 0.18215
+    guidance_scale: 5 # not used
+    ddim_sampling_steps: 20
+    text_cfg: 7.5
+    img_cfg: 1.2
+    hdr_cfg: 7.5
+    cond_image_dropout: 0.1
+    cond_text_dropout: 0.1
+    cond_hdr_dropout: 0.1
+    ic_condition: fg
+    hdr_train: True
+    prompt_type: edit_prompt
+unet:
+  target: modules.video_unet_temporal.unet.UNet3DConditionModel
+  params:
+    in_channels: 4 #! change:8->12 iclight 改为12 注意一下...
+    out_channels: 4
+    act_fn: silu
+    attention_head_dim: 8
+    block_out_channels: 
+      - 320
+      - 640
+      - 1280
+      - 1280
+    cross_attention_dim: 768
+    down_block_types: 
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - DownBlock3D
+    up_block_types: 
+      - UpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+    downsample_padding: 1
+    layers_per_block: 2
+    mid_block_scale_factor: 1
+    norm_eps: 1e-05
+    norm_num_groups: 32
+    sample_size: 64
+    use_motion_module: true #!!! 这边test iclight的时候可以不用motion module 即False
+    motion_module_resolutions:
+    - 1
+    - 2
+    - 4
+    - 8
+    motion_module_mid_block: false
+    motion_module_decoder_only: false
+    motion_module_type: Vanilla
+    motion_module_kwargs:
+      num_attention_heads: 8
+      num_transformer_block: 1
+      attention_block_types:
+      - Temporal_Self
+      - Temporal_Self
+      temporal_position_encoding: true
+      temporal_position_encoding_max_len: 32
+      temporal_attention_dim_div: 1
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true
+data:
+  batch_size: 1
+  val_batch_size: 1
+  train:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAugPexelsHDR
+    params: # 注意修改一下training的路径，和相关加载的代码, 比如说没有meta.yaml这些参数怎么搞
+      root_dirs: #! 注意root_dirs已经更改
+        # - /mnt/petrelfs/fangye/test/instruct-video-to-video_1019/data_train_pexels/rmbg_data
+        - /mnt/hwfile/mllm/sunzeyi/iclight_video/rendered_data_rgb_fixlast
+      hdr_dir: /mnt/hwfile/mllm/sunzeyi/iclight_video/haven_hdr_rgb
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+      is_train: True
+      ic_condition: fg
+  val:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAugPexelsHDR
+    params:
+      root_dirs: 
+        # - /mnt/petrelfs/fangye/test/instruct-video-to-video_1019/data_train_pexels/rmbg_data
+        - /mnt/hwfile/mllm/sunzeyi/iclight_video/rendered_data_rgb_fixlast
+      hdr_dir: /mnt/hwfile/mllm/sunzeyi/iclight_video/haven_hdr_rgb
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+      ic_condition: fg
+callbacks:
+  - target: pytorch_lightning.callbacks.ModelCheckpoint
+    params:
+      dirpath: "${expt_dir}/${expt_name}"
+      # filename: "{epoch:04d}"
+      filename: "{step:06d}"
+      every_n_train_steps: 1
+      save_last: false
+      # monitor: epoch
+      # mode: max
+      # save_top_k: 3
+      # save_last: false
+  - target: callbacks.instruct_p2p_video.InstructP2PLogger
+    params:
+      max_num_images: 1
+      expt_name: instruct_v2v_ic_pexels_text_hdr_test_lr0.5_aug_lossc_fix_bs1
+      # accumulate_grad_batches: 128
+    require_wandb: true
+  - target: pytorch_lightning.callbacks.DeviceStatsMonitor

configs/instruct_v2v_ic_pexels_text.yaml

@@ -0,0 +1,137 @@
+expt_dir: experiments
+expt_name: instruct_v2v_ic_pexels_text_fg #! 注意传入log里面, 不要每次修改
+trainer_args:
+  max_epochs: 5
+  accelerator: "gpu"
+  devices: [0] #! change to get more cards
+  limit_train_batches: 2048
+  limit_val_batches: 1 #! 这边限制了每个epoch只跑多少个batch的validation
+  # strategy: "ddp"
+  strategy: "deepspeed_stage_2"
+  # autotune_only_on_rank_zero: true # 确保只有一个进程执行调优表操作 
+  accumulate_grad_batches: 256 #! 注意一下这个值
+  check_val_every_n_epoch: 1 #! check一下这个值是不是和记录有关。。。
+  # precision: 16 # 启用半精度 (FP16)
+diffusion:
+  target: pl_trainer.instruct_p2p_video.InstructP2PVideoTrainerTemporalText
+  params:
+    beta_schedule_args:
+      beta_schedule: scaled_linear
+      num_train_timesteps: 1000
+      beta_start: 0.00085
+      beta_end: 0.012
+      clip_sample: false
+      thresholding: false
+    prediction_type: epsilon
+    loss_fn: l2
+    optim_args:
+      lr: 1e-5
+    unet_init_weights: #! 注意一下, 完全可以从iv2v的ckpt开始train
+      - unet/diffusion_pytorch_model.safetensors # iclight, unet, sf tensor
+      - pretrained_models/Motion_Module/mm_sd_v15.ckpt # motion module, 推测加载的是animatediff的
+      - pretrained_models/iclight/iclight_sd15_fc.safetensors # iclight lora weights
+    base_path: /mnt/petrelfs/fangye/.cache/huggingface/hub/models--stablediffusionapi--realistic-vision-v51/snapshots/19e3643d7d963c156d01537188ec08f0b79a514a
+    # vae_init_weights: pretrained_models/instruct_pix2pix/vqvae.ckpt
+    # text_model_init_weights: pretrained_models/instruct_pix2pix/text.ckpt #! 这两个可以直接设为None, 从from_pretrained中加载
+    scale_factor: 0.18215
+    guidance_scale: 5 # not used
+    ddim_sampling_steps: 20
+    text_cfg: 7.5
+    img_cfg: 1.2
+    cond_image_dropout: 0.1
+    cond_text_dropout: 0.075
+    ic_condition: fg
+    prompt_type: edit_prompt
+unet:
+  target: modules.video_unet_temporal.unet.UNet3DConditionModel
+  params:
+    in_channels: 4 #! change:8->12 iclight 改为12 注意一下...
+    out_channels: 4
+    act_fn: silu
+    attention_head_dim: 8
+    block_out_channels: 
+      - 320
+      - 640
+      - 1280
+      - 1280
+    cross_attention_dim: 768
+    down_block_types: 
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - DownBlock3D
+    up_block_types: 
+      - UpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+    downsample_padding: 1
+    layers_per_block: 2
+    mid_block_scale_factor: 1
+    norm_eps: 1e-05
+    norm_num_groups: 32
+    sample_size: 64
+    use_motion_module: true #!!! 这边test iclight的时候可以不用motion module 即False
+    motion_module_resolutions:
+    - 1
+    - 2
+    - 4
+    - 8
+    motion_module_mid_block: false
+    motion_module_decoder_only: false
+    motion_module_type: Vanilla
+    motion_module_kwargs:
+      num_attention_heads: 8
+      num_transformer_block: 1
+      attention_block_types:
+      - Temporal_Self
+      - Temporal_Self
+      temporal_position_encoding: true
+      temporal_position_encoding_max_len: 32
+      temporal_attention_dim_div: 1
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true
+data:
+  batch_size: 1
+  val_batch_size: 1
+  train:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAugPexels
+    params: # 注意修改一下training的路径，和相关加载的代码, 比如说没有meta.yaml这些参数怎么搞
+      root_dirs: 
+        - /mnt/petrelfs/fangye/test/instruct-video-to-video_1019/data_train_pexels/rmbg_data
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+      is_train: True
+      ic_condition: fg
+  val:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAugPexels
+    params:
+      root_dirs: 
+        - /mnt/petrelfs/fangye/test/instruct-video-to-video_1019/data_train_pexels/rmbg_data
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+      ic_condition: fg
+callbacks:
+  - target: pytorch_lightning.callbacks.ModelCheckpoint
+    params:
+      dirpath: "${expt_dir}/${expt_name}"
+      filename: "{epoch:04d}"
+      monitor: epoch
+      mode: max
+      save_top_k: 5
+      save_last: false
+  - target: callbacks.instruct_p2p_video.InstructP2PLogger
+    params:
+      max_num_images: 1
+      expt_name: instruct_v2v_ic_pexels_text_fg
+      # accumulate_grad_batches: 128
+    require_wandb: true
+  - target: pytorch_lightning.callbacks.DeviceStatsMonitor

configs/instruct_v2v_ic_pexels_text_hdr.yaml

@@ -0,0 +1,137 @@
+expt_dir: experiments
+expt_name: instruct_v2v_ic_pexels_text_fg #! 注意传入log里面, 不要每次修改
+trainer_args:
+  max_epochs: 5
+  accelerator: "gpu"
+  devices: [0] #! change to get more cards
+  limit_train_batches: 2048
+  limit_val_batches: 1 #! 这边限制了每个epoch只跑多少个batch的validation
+  # strategy: "ddp"
+  strategy: "deepspeed_stage_2"
+  # autotune_only_on_rank_zero: true # 确保只有一个进程执行调优表操作 
+  accumulate_grad_batches: 256 #! 注意一下这个值
+  check_val_every_n_epoch: 1 #! check一下这个值是不是和记录有关。。。
+  # precision: 16 # 启用半精度 (FP16)
+diffusion:
+  target: pl_trainer.instruct_p2p_video.InstructP2PVideoTrainerTemporalText
+  params:
+    beta_schedule_args:
+      beta_schedule: scaled_linear
+      num_train_timesteps: 1000
+      beta_start: 0.00085
+      beta_end: 0.012
+      clip_sample: false
+      thresholding: false
+    prediction_type: epsilon
+    loss_fn: l2
+    optim_args:
+      lr: 1e-5
+    unet_init_weights: #! 注意一下, 完全可以从iv2v的ckpt开始train
+      - unet/diffusion_pytorch_model.safetensors # iclight, unet, sf tensor
+      - pretrained_models/Motion_Module/mm_sd_v15.ckpt # motion module, 推测加载的是animatediff的
+      - pretrained_models/iclight/iclight_sd15_fc.safetensors # iclight lora weights
+    base_path: /mnt/petrelfs/fangye/.cache/huggingface/hub/models--stablediffusionapi--realistic-vision-v51/snapshots/19e3643d7d963c156d01537188ec08f0b79a514a
+    # vae_init_weights: pretrained_models/instruct_pix2pix/vqvae.ckpt
+    # text_model_init_weights: pretrained_models/instruct_pix2pix/text.ckpt #! 这两个可以直接设为None, 从from_pretrained中加载
+    scale_factor: 0.18215
+    guidance_scale: 5 # not used
+    ddim_sampling_steps: 20
+    text_cfg: 7.5
+    img_cfg: 1.2
+    cond_image_dropout: 0.1
+    cond_text_dropout: 0.075
+    ic_condition: fg
+    prompt_type: edit_prompt
+unet:
+  target: modules.video_unet_temporal.unet.UNet3DConditionModel
+  params:
+    in_channels: 4 #! change:8->12 iclight 改为12 注意一下...
+    out_channels: 4
+    act_fn: silu
+    attention_head_dim: 8
+    block_out_channels: 
+      - 320
+      - 640
+      - 1280
+      - 1280
+    cross_attention_dim: 768
+    down_block_types: 
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - DownBlock3D
+    up_block_types: 
+      - UpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+    downsample_padding: 1
+    layers_per_block: 2
+    mid_block_scale_factor: 1
+    norm_eps: 1e-05
+    norm_num_groups: 32
+    sample_size: 64
+    use_motion_module: true #!!! 这边test iclight的时候可以不用motion module 即False
+    motion_module_resolutions:
+    - 1
+    - 2
+    - 4
+    - 8
+    motion_module_mid_block: false
+    motion_module_decoder_only: false
+    motion_module_type: Vanilla
+    motion_module_kwargs:
+      num_attention_heads: 8
+      num_transformer_block: 1
+      attention_block_types:
+      - Temporal_Self
+      - Temporal_Self
+      temporal_position_encoding: true
+      temporal_position_encoding_max_len: 32
+      temporal_attention_dim_div: 1
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true
+data:
+  batch_size: 1
+  val_batch_size: 1
+  train:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAugPexels
+    params: # 注意修改一下training的路径，和相关加载的代码, 比如说没有meta.yaml这些参数怎么搞
+      root_dirs: 
+        - /mnt/petrelfs/fangye/test/instruct-video-to-video_1019/data_train_pexels/rmbg_data
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+      is_train: True
+      ic_condition: fg
+  val:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAugPexels
+    params:
+      root_dirs: 
+        - /mnt/petrelfs/fangye/test/instruct-video-to-video_1019/data_train_pexels/rmbg_data
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+      ic_condition: fg
+callbacks:
+  - target: pytorch_lightning.callbacks.ModelCheckpoint
+    params:
+      dirpath: "${expt_dir}/${expt_name}"
+      filename: "{epoch:04d}"
+      monitor: epoch
+      mode: max
+      save_top_k: 5
+      save_last: false
+  - target: callbacks.instruct_p2p_video.InstructP2PLogger
+    params:
+      max_num_images: 1
+      expt_name: instruct_v2v_ic_pexels_text_fg
+      # accumulate_grad_batches: 128
+    require_wandb: true
+  - target: pytorch_lightning.callbacks.DeviceStatsMonitor

configs/instruct_v2v_ic_test.yaml

@@ -0,0 +1,132 @@
+expt_dir: experiments
+expt_name: instruct_v2v_ic
+trainer_args:
+  max_epochs: 5
+  accelerator: "gpu"
+  devices: [0,1,2,3]
+  limit_train_batches: 2048
+  limit_val_batches: 1 #! 这边限制了每个epoch只跑多少个batch的validation
+  # strategy: "ddp"
+  strategy: "deepspeed_stage_2"
+  # autotune_only_on_rank_zero: true # 确保只有一个进程执行调优表操作 
+  accumulate_grad_batches: 32 #! 注意一下这个值
+  check_val_every_n_epoch: 1 #! check一下这个值是不是和记录有关。。。
+  # precision: 16 # 启用半精度 (FP16)
+diffusion:
+  target: pl_trainer.instruct_p2p_video.InstructP2PVideoTrainerTemporal
+  params:
+    beta_schedule_args:
+      beta_schedule: scaled_linear
+      num_train_timesteps: 1000
+      beta_start: 0.00085
+      beta_end: 0.012
+      clip_sample: false
+      thresholding: false
+    prediction_type: epsilon
+    loss_fn: l2
+    optim_args:
+      lr: 1e-5
+    unet_init_weights: #! 注意一下, 完全可以从iv2v的ckpt开始train
+      - unet/diffusion_pytorch_model.safetensors # iclight, unet, sf tensor
+      - pretrained_models/Motion_Module/mm_sd_v15.ckpt # motion module, 推测加载的是animatediff的
+      - pretrained_models/iclight/iclight_sd15_fbc.safetensors # iclight lora weights
+    base_path: /mnt/petrelfs/fangye/.cache/huggingface/hub/models--stablediffusionapi--realistic-vision-v51/snapshots/19e3643d7d963c156d01537188ec08f0b79a514a
+    # vae_init_weights: pretrained_models/instruct_pix2pix/vqvae.ckpt
+    # text_model_init_weights: pretrained_models/instruct_pix2pix/text.ckpt #! 这两个可以直接设为None, 从from_pretrained中加载
+    scale_factor: 0.18215
+    guidance_scale: 5 # not used
+    ddim_sampling_steps: 20
+    text_cfg: 7.5
+    img_cfg: 1.2
+    cond_image_dropout: 0.1
+    prompt_type: edit_prompt
+unet:
+  target: modules.video_unet_temporal.unet.UNet3DConditionModel
+  params:
+    in_channels: 4 #! change:8->12 iclight 改为12 注意一下...
+    out_channels: 4
+    act_fn: silu
+    attention_head_dim: 8
+    block_out_channels: 
+      - 320
+      - 640
+      - 1280
+      - 1280
+    cross_attention_dim: 768
+    down_block_types: 
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - DownBlock3D
+    up_block_types: 
+      - UpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+    downsample_padding: 1
+    layers_per_block: 2
+    mid_block_scale_factor: 1
+    norm_eps: 1e-05
+    norm_num_groups: 32
+    sample_size: 64
+    use_motion_module: true #!!! 这边test iclight的时候可以不用motion module 即False
+    motion_module_resolutions:
+    - 1
+    - 2
+    - 4
+    - 8
+    motion_module_mid_block: false
+    motion_module_decoder_only: false
+    motion_module_type: Vanilla
+    motion_module_kwargs:
+      num_attention_heads: 8
+      num_transformer_block: 1
+      attention_block_types:
+      - Temporal_Self
+      - Temporal_Self
+      temporal_position_encoding: true
+      temporal_position_encoding_max_len: 32
+      temporal_attention_dim_div: 1
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true
+data:
+  batch_size: 1
+  val_batch_size: 1
+  train:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAug
+    params: # 注意修改一下training的路径，和相关加载的代码, 比如说没有meta.yaml这些参数怎么搞
+      root_dirs: 
+        - /mnt/petrelfs/fangye/test/instruct-video-to-video_1019/data_train_v2
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+      is_train: True
+  val:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAug
+    params:
+      root_dirs: 
+        - data_train
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+callbacks:
+  - target: pytorch_lightning.callbacks.ModelCheckpoint
+    params:
+      dirpath: "${expt_dir}/${expt_name}"
+      filename: "{epoch:04d}"
+      monitor: epoch
+      mode: max
+      save_top_k: 5
+      save_last: true
+  - target: callbacks.instruct_p2p_video.InstructP2PLogger
+    params:
+      max_num_images: 1
+      # accumulate_grad_batches: 128
+    require_wandb: true
+  - target: pytorch_lightning.callbacks.DeviceStatsMonitor

configs/instruct_v2v_inference.yaml

@@ -0,0 +1,98 @@
+diffusion:
+  target: pl_trainer.instruct_p2p_video.InstructP2PVideoTrainerTemporal
+  params:
+    beta_schedule_args:
+      beta_schedule: scaled_linear
+      num_train_timesteps: 1000
+      beta_start: 0.00085
+      beta_end: 0.012
+      clip_sample: false
+      thresholding: false
+    prediction_type: epsilon
+    loss_fn: l2
+    unet_init_weights: 
+      - pretrained_models/instruct_pix2pix/diffusion_pytorch_model.bin
+      - pretrained_models/Motion_Module/mm_sd_v15.ckpt
+    vae_init_weights: pretrained_models/instruct_pix2pix/vqvae.ckpt
+    text_model_init_weights: pretrained_models/instruct_pix2pix/text.ckpt
+    optim_args:
+      lr: 1e-5
+    scale_factor: 0.18215
+    guidance_scale: 5 # not used
+    ddim_sampling_steps: 20
+    text_cfg: 7.5
+    img_cfg: 1.2
+    cond_image_dropout: 0.1
+    prompt_type: edit_prompt
+unet:
+  target: modules.video_unet_temporal.unet.UNet3DConditionModel
+  params:
+    in_channels: 8
+    out_channels: 4
+    act_fn: silu
+    attention_head_dim: 8
+    block_out_channels: 
+      - 320
+      - 640
+      - 1280
+      - 1280
+    cross_attention_dim: 768
+    down_block_types: 
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - DownBlock3D
+    up_block_types: 
+      - UpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+    downsample_padding: 1
+    layers_per_block: 2
+    mid_block_scale_factor: 1
+    norm_eps: 1e-05
+    norm_num_groups: 32
+    sample_size: 64
+    use_motion_module: true
+    motion_module_resolutions:
+    - 1
+    - 2
+    - 4
+    - 8
+    motion_module_mid_block: false
+    motion_module_decoder_only: false
+    motion_module_type: Vanilla
+    motion_module_kwargs:
+      num_attention_heads: 8
+      num_transformer_block: 1
+      attention_block_types:
+      - Temporal_Self
+      - Temporal_Self
+      temporal_position_encoding: true
+      temporal_position_encoding_max_len: 32
+      temporal_attention_dim_div: 1
+vae:
+  target: modules.kl_autoencoder.autoencoder.AutoencoderKL
+  params:
+    embed_dim: 4
+    ddconfig:
+      double_z: true
+      z_channels: 4
+      resolution: 256
+      in_channels: 3
+      out_ch: 3
+      ch: 128
+      ch_mult:
+        - 1
+        - 2
+        - 4
+        - 4
+      num_res_blocks: 2
+      attn_resolutions: []
+      dropout: 0.0
+    lossconfig:
+      target: torch.nn.Identity
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true

configs/instruct_v2v_ori.yaml

@@ -0,0 +1,147 @@
+expt_dir: experiments
+expt_name: instruct_v2v
+trainer_args:
+  max_epochs: 10
+  accelerator: "gpu"
+  devices: [0]
+  limit_train_batches: 2048
+  limit_val_batches: 1
+  # strategy: "ddp"
+  strategy: "deepspeed_stage_2"
+  accumulate_grad_batches: 256
+  check_val_every_n_epoch: 5
+diffusion:
+  target: pl_trainer.instruct_p2p_video.InstructP2PVideoTrainerTemporal
+  params:
+    beta_schedule_args:
+      beta_schedule: scaled_linear
+      num_train_timesteps: 1000
+      beta_start: 0.00085
+      beta_end: 0.012
+      clip_sample: false
+      thresholding: false
+    prediction_type: epsilon
+    loss_fn: l2
+    optim_args:
+      lr: 1e-5
+    unet_init_weights: 
+      - pretrained_models/instruct_pix2pix/diffusion_pytorch_model.bin
+      - pretrained_models/Motion_Module/mm_sd_v15.ckpt
+    vae_init_weights: pretrained_models/instruct_pix2pix/vqvae.ckpt
+    text_model_init_weights: pretrained_models/instruct_pix2pix/text.ckpt
+    scale_factor: 0.18215
+    guidance_scale: 5 # not used
+    ddim_sampling_steps: 20
+    text_cfg: 7.5
+    img_cfg: 1.2
+    cond_image_dropout: 0.1
+    prompt_type: edit_prompt
+unet:
+  target: modules.video_unet_temporal.unet.UNet3DConditionModel
+  params:
+    in_channels: 8
+    out_channels: 4
+    act_fn: silu
+    attention_head_dim: 8
+    block_out_channels: 
+      - 320
+      - 640
+      - 1280
+      - 1280
+    cross_attention_dim: 768
+    down_block_types: 
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - DownBlock3D
+    up_block_types: 
+      - UpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+    downsample_padding: 1
+    layers_per_block: 2
+    mid_block_scale_factor: 1
+    norm_eps: 1e-05
+    norm_num_groups: 32
+    sample_size: 64
+    use_motion_module: true
+    motion_module_resolutions:
+    - 1
+    - 2
+    - 4
+    - 8
+    motion_module_mid_block: false
+    motion_module_decoder_only: false
+    motion_module_type: Vanilla
+    motion_module_kwargs:
+      num_attention_heads: 8
+      num_transformer_block: 1
+      attention_block_types:
+      - Temporal_Self
+      - Temporal_Self
+      temporal_position_encoding: true
+      temporal_position_encoding_max_len: 32
+      temporal_attention_dim_div: 1
+vae:
+  target: modules.kl_autoencoder.autoencoder.AutoencoderKL
+  params:
+    embed_dim: 4
+    ddconfig:
+      double_z: true
+      z_channels: 4
+      resolution: 256
+      in_channels: 3
+      out_ch: 3
+      ch: 128
+      ch_mult:
+        - 1
+        - 2
+        - 4
+        - 4
+      num_res_blocks: 2
+      attn_resolutions: []
+      dropout: 0.0
+    lossconfig:
+      target: torch.nn.Identity
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true
+data:
+  batch_size: 1
+  val_batch_size: 1
+  train:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAug
+    params:
+      root_dirs: 
+        - video_ptp/raw_generated
+        - video_ptp/raw_generated_webvid
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+  val:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAug
+    params:
+      root_dirs: 
+        - video_ptp/raw_generated
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+callbacks:
+  - target: pytorch_lightning.callbacks.ModelCheckpoint
+    params:
+      dirpath: "${expt_dir}/${expt_name}"
+      filename: "{epoch:04d}"
+      monitor: epoch
+      mode: max
+      save_top_k: 5
+      save_last: true
+  - target: callbacks.instruct_p2p_video.InstructP2PLogger
+    params:
+      max_num_images: 1
+    require_wandb: true

configs/test_textmodel.yaml

@@ -0,0 +1,7 @@
+diffusion:
+  params:
+    base_path: /home/fy/.cache/huggingface/hub/models--stablediffusionapi--realistic-vision-v51/snapshots/19e3643d7d963c156d01537188ec08f0b79a514a
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true

configs/test_vae.yaml

@@ -0,0 +1,21 @@
+vae:
+  target: modules.kl_autoencoder.autoencoder.AutoencoderKL
+  params:
+    embed_dim: 4
+    ddconfig:
+      double_z: true
+      z_channels: 4
+      resolution: 256 #先暂时256
+      in_channels: 3
+      out_ch: 3
+      ch: 128
+      ch_mult:
+        - 1
+        - 2
+        - 4
+        - 4
+      num_res_blocks: 2
+      attn_resolutions: []
+      dropout: 0.0
+    lossconfig:
+      target: torch.nn.Identity

configs/test_vae_ori.yaml

@@ -0,0 +1,28 @@
+diffusion:
+  params:
+    base_path: /home/fy/.cache/huggingface/hub/models--stablediffusionapi--realistic-vision-v51/snapshots/19e3643d7d963c156d01537188ec08f0b79a514a
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true
+vae:
+  target: modules.kl_autoencoder.autoencoder.AutoencoderKL
+  params:
+    embed_dim: 4
+    ddconfig:
+      double_z: true
+      z_channels: 4
+      resolution: 256 #先暂时256
+      in_channels: 3
+      out_ch: 3
+      ch: 128
+      ch_mult:
+        - 1
+        - 2
+        - 4
+        - 4
+      num_res_blocks: 2
+      attn_resolutions: []
+      dropout: 0.0
+    lossconfig:
+      target: torch.nn.Identity

configs/tmp_ic.yaml

@@ -0,0 +1,130 @@
+expt_dir: experiments
+expt_name: instruct_v2v_ic
+trainer_args:
+  max_epochs: 10
+  accelerator: "gpu"
+  devices: [0]
+  limit_train_batches: 2048
+  limit_val_batches: 5 #! 这边限制了每个epoch只跑多少个batch的validation
+  # strategy: "ddp"
+  strategy: "deepspeed_stage_2"
+  accumulate_grad_batches: 128 #! 注意一下这个值
+  check_val_every_n_epoch: 1 #! check一下这个值是不是和记录有关。。。
+diffusion:
+  target: pl_trainer.instruct_p2p_video.InstructP2PVideoTrainerTemporal
+  params:
+    beta_schedule_args:
+      beta_schedule: scaled_linear
+      num_train_timesteps: 1000
+      beta_start: 0.00085
+      beta_end: 0.012
+      clip_sample: false
+      thresholding: false
+    prediction_type: epsilon
+    loss_fn: l2
+    optim_args:
+      lr: 1e-5
+    unet_init_weights: #! 注意一下, 完全可以从iv2v的ckpt开始train
+      - unet/diffusion_pytorch_model.safetensors # iclight, unet, sf tensor
+      - pretrained_models/Motion_Module/mm_sd_v15.ckpt # motion module, 推测加载的是animatediff的
+      - /mnt/petrelfs/fangye/IC-Light/models/iclight_sd15_fbc.safetensors # iclight lora weights
+    base_path: /mnt/petrelfs/fangye/.cache/huggingface/hub/models--stablediffusionapi--realistic-vision-v51/snapshots/19e3643d7d963c156d01537188ec08f0b79a514a
+    # vae_init_weights: pretrained_models/instruct_pix2pix/vqvae.ckpt
+    # text_model_init_weights: pretrained_models/instruct_pix2pix/text.ckpt #! 这两个可以直接设为None, 从from_pretrained中加载
+    scale_factor: 0.18215
+    guidance_scale: 5 # not used
+    ddim_sampling_steps: 20
+    text_cfg: 7.5
+    img_cfg: 1.2
+    cond_image_dropout: 0.1
+    prompt_type: edit_prompt
+unet:
+  target: modules.video_unet_temporal.unet.UNet3DConditionModel
+  params:
+    in_channels: 4 #! change:8->12 iclight 改为12 注意一下...
+    out_channels: 4
+    act_fn: silu
+    attention_head_dim: 8
+    block_out_channels: 
+      - 320
+      - 640
+      - 1280
+      - 1280
+    cross_attention_dim: 768
+    down_block_types: 
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - CrossAttnDownBlock3D
+      - DownBlock3D
+    up_block_types: 
+      - UpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+      - CrossAttnUpBlock3D
+    downsample_padding: 1
+    layers_per_block: 2
+    mid_block_scale_factor: 1
+    norm_eps: 1e-05
+    norm_num_groups: 32
+    sample_size: 64
+    use_motion_module: true #! 这边test iclight的时候可以不用motion module 即False
+    motion_module_resolutions:
+    - 1
+    - 2
+    - 4
+    - 8
+    motion_module_mid_block: false
+    motion_module_decoder_only: false
+    motion_module_type: Vanilla
+    motion_module_kwargs:
+      num_attention_heads: 8
+      num_transformer_block: 1
+      attention_block_types:
+      - Temporal_Self
+      - Temporal_Self
+      temporal_position_encoding: true
+      temporal_position_encoding_max_len: 32
+      temporal_attention_dim_div: 1
+text_model:
+  target: modules.openclip.modules.FrozenCLIPEmbedder
+  params:
+    freeze: true
+data:
+  batch_size: 1
+  val_batch_size: 1
+  train:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAug
+    params: #注意修改一下training的路径，和相关加载的代码, 比如说没有meta.yaml这些参数怎么搞
+      root_dirs: 
+        - /home/fy/Code/instruct-video-to-video/data_train/Girl
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+      is_train: True
+  val:
+    target: dataset.videoP2P.VideoPromptToPromptMotionAug
+    params:
+      root_dirs: 
+        - /home/fy/Code/instruct-video-to-video/data_train/Girl
+      num_frames: 16
+      zoom_ratio: 0.2
+      max_zoom: 1.25
+      translation_ratio: 0.7
+      translation_range: [0, 0.2]
+callbacks:
+  - target: pytorch_lightning.callbacks.ModelCheckpoint
+    params:
+      dirpath: "${expt_dir}/${expt_name}"
+      filename: "{epoch:04d}"
+      monitor: epoch
+      mode: max
+      save_top_k: 5
+      save_last: true
+  - target: callbacks.instruct_p2p_video.InstructP2PLogger
+    params:
+      max_num_images: 1
+      # accumulate_grad_batches: 128
+    require_wandb: true
+  - target: pytorch_lightning.callbacks.DeviceStatsMonitor

db_examples.py

@@ -0,0 +1,133 @@
+
+bg_samples = [
+    'demo/clean_bg_extracted/22/frames/0000.png',
+    'demo/clean_bg_extracted/23/frames/0000.png',
+    'demo/clean_bg_extracted/27/frames/0000.png',
+    'demo/clean_bg_extracted/33/frames/0000.png',
+    'demo/clean_bg_extracted/47/frames/0000.png',
+    'demo/clean_bg_extracted/39/frames/0000.png',
+    'demo/clean_bg_extracted/59/frames/0000.png',
+    'demo/clean_bg_extracted/55/frames/0000.png',
+    'demo/clean_bg_extracted/58/frames/0000.png',
+    'demo/clean_bg_extracted/57/frames/0000.png', #42
+    'demo/clean_bg_extracted/8/frames/0000.png',
+    'demo/clean_bg_extracted/9/frames/0000.png',
+    'demo/clean_bg_extracted/10/frames/0000.png',
+    'demo/clean_bg_extracted/14/frames/0000.png',
+    'demo/clean_bg_extracted/62/frames/0000.png'
+] # 准备大概 15 个 background视频
+
+
+background_conditioned_examples = [
+    [
+        "demo/clean_fg_extracted/14/cropped_video.mp4",
+        "demo/clean_bg_extracted/22/cropped_video.mp4",
+        "beautiful woman, cinematic lighting",
+        "Use Background Video",
+        512,
+        768,
+        12345,
+        "static_fg_sync_bg_visualization_fy/14_22_100fps.mp4",
+    ],
+    [
+        "demo/clean_fg_extracted/14/cropped_video.mp4",
+        "demo/clean_bg_extracted/55/cropped_video.mp4",
+        "beautiful woman, cinematic lighting",
+        "Use Background Video",
+        512,
+        768,
+        12345,
+        "static_fg_sync_bg_visualization_fy/14_55_100fps.mp4",
+    ],
+    [
+        "demo/clean_fg_extracted/15/cropped_video.mp4",
+        "demo/clean_bg_extracted/27/cropped_video.mp4",
+        "beautiful woman, cinematic lighting",
+        "Use Background Video",
+        512,
+        768,
+        12345,
+        "static_fg_sync_bg_visualization_fy/15_27_100fps.mp4",
+    ],
+    [
+        "demo/clean_fg_extracted/18/cropped_video.mp4",
+        "demo/clean_bg_extracted/23/cropped_video.mp4",
+        "beautiful woman, cinematic lighting",
+        "Use Background Video",
+        512,
+        768,
+        12345,
+        "static_fg_sync_bg_visualization_fy/18_23_100fps.mp4",
+    ],
+    # [
+    #     "demo/clean_fg_extracted/18/cropped_video.mp4",
+    #     "demo/clean_bg_extracted/33/cropped_video.mp4",
+    #     "beautiful woman, cinematic lighting",
+    #     "Use Background Video",
+    #     512,
+    #     768,
+    #     12345,
+    #     "static_fg_sync_bg_visualization_fy/18_33_100fps.mp4",
+    # ],
+    [
+        "demo/clean_fg_extracted/22/cropped_video.mp4",
+        "demo/clean_bg_extracted/39/cropped_video.mp4",
+        "beautiful woman, cinematic lighting",
+        "Use Background Video",
+        512,
+        768,
+        12345,
+        "static_fg_sync_bg_visualization_fy/22_39_100fps.mp4",
+    ],
+    # [
+    #     "demo/clean_fg_extracted/22/cropped_video.mp4",
+    #     "demo/clean_bg_extracted/59/cropped_video.mp4",
+    #     "beautiful woman, cinematic lighting",
+    #     "Use Background Video",
+    #     512,
+    #     768,
+    #     12345,
+    #     "static_fg_sync_bg_visualization_fy/22_59_100fps.mp4",
+    # ],
+    [
+        "demo/clean_fg_extracted/9/cropped_video.mp4",
+        "demo/clean_bg_extracted/8/cropped_video.mp4",
+        "beautiful woman, cinematic lighting",
+        "Use Background Video",
+        512,
+        768,
+        12345,
+        "static_fg_sync_bg_visualization_fy/9_8_100fps.mp4",
+    ],
+    [
+        "demo/clean_fg_extracted/9/cropped_video.mp4",
+        "demo/clean_bg_extracted/9/cropped_video.mp4",
+        "beautiful woman, cinematic lighting",
+        "Use Background Video",
+        512,
+        768,
+        12345,
+        "static_fg_sync_bg_visualization_fy/9_9_100fps.mp4",
+    ],
+    [
+        "demo/clean_fg_extracted/9/cropped_video.mp4",
+        "demo/clean_bg_extracted/10/cropped_video.mp4",
+        "beautiful woman, cinematic lighting",
+        "Use Background Video",
+        512,
+        768,
+        12345,
+        "static_fg_sync_bg_visualization_fy/9_10_100fps.mp4",
+    ],
+    # [
+    #     "demo/clean_fg_extracted/9/cropped_video.mp4",
+    #     "demo/clean_bg_extracted/14/cropped_video.mp4",
+    #     "beautiful woman, cinematic lighting",
+    #     "Use Background Video",
+    #     512,
+    #     768,
+    #     12345,
+    #     "static_fg_sync_bg_visualization_fy/9_14_100fps.mp4",
+    # ],
+
+]