Duplicate from MeiGen-AI/MeiGen-MultiTalk

Co-authored-by: F <[email protected]>

.gitattributes

@@ -0,0 +1,38 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/logo.png filter=lfs diff=lfs merge=lfs -text
+assets/logo2.jpeg filter=lfs diff=lfs merge=lfs -text
+assets/pipe.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,88 @@
+---
+license: apache-2.0
+language:
+- en
+- zh
+tags:
+- video generation
+- conversational video generation
+- talking human video generation
+pipeline_tag: image-to-video
+---
+
+<p align="center">
+  <img src="assets/logo2.jpeg" alt="MultiTalk" width="300"/>
+</p>
+
+# MeiGen-MultiTalk • Audio-Driven Multi-Person Conversational Video Generation
+
+
+
+<p align="left">
+  <a href="https://meigen-ai.github.io/multi-talk/">
+    <img
+      src="https://img.shields.io/badge/MultiTalk-Website-0A66C2?logo=safari&logoColor=white" style="display: inline-block; vertical-align: middle;"
+      alt="MultiTalk Website"
+    />
+  </a>
+  <a href="https://arxiv.org/abs/2505.22647">
+    <img
+      src="https://img.shields.io/badge/MultiTalk-Paper-red?logo=arxiv&logoColor=red" style="display: inline-block; vertical-align: middle;"
+      alt="MultiTalk Paper on arXiv"
+    />
+  </a>
+  <a href="https://github.com/MeiGen-AI/MultiTalk" target="_blank" style="margin: 2px;">
+      <img 
+        alt="Github" src="https://img.shields.io/badge/MultiTalk-Codebase-536af5?color=536af5&logo=github" style="display: inline-block; vertical-align: middle;"
+        alt="MultiTalk Codebase"
+      />
+  </a>
+  
+</p>
+
+> We present **MultiTalk**, an open-source audio-driven multi-person conversational video generation model with the state-of-the-art lip synchronization accuracy. 
+> ​​Key features:​​
+> - 💬 ​​Realistic Conversations​​ - Supports single & multi-person generation
+> - 👥 ​​Interactive Character Control​​ - Direct virtual humans via prompts
+> - 🎤 ​​Generalization Performances​​ - Supports the generation of cartoon character and singing 
+> - 📺 ​​Resolution Flexibility​​: 480p & 720p output at arbitrary aspect ratios
+> - ⏱️ **Long Video Generation**: Support video generation up to 15 seconds
+
+
+This repository hosts the model weights for **MultiTalk**. For installation, usage instructions, and further documentation, please visit our [GitHub repository](https://github.com/MeiGen-AI/MultiTalk).
+
+
+
+
+
+
+## Method
+We propose a novel framework, MultiTalk, for audio-driven multi-person conversational video generation. We investigate several schemes for audio injection and introduce
+the Label Rotary Position Embedding (L-RoPE) method. By assigning identical labels to audio embeddings and video latents, it effectively activates specific regions within the audio cross-attention
+map, thereby resolving incorrect binding issues. To localize the region of the specified person, we introduce the adaptive person localization by computing the similarity 
+between the features of the given region of a person in the reference image and all the features of the whole video. 
+
+<p align="left"><img src="assets/pipe.png" width="80%"></p>
+
+
+
+
+
+## Citation
+If you find our work helpful, please cite us.
+
+```
+@article{kong2025let,
+  title={Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation},
+  author={Kong, Zhe and Gao, Feng and Zhang, Yong and Kang, Zhuoliang and Wei, Xiaoming and Cai, Xunliang and Chen, Guanying and Luo, Wenhan},
+  journal={arXiv preprint arXiv:2505.22647},
+  year={2025}
+}
+```
+
+
+
+## License Agreement
+The models in this repository are licensed under the Apache 2.0 License. We claim no rights over the your generated contents, granting you the freedom to use them while ensuring that your usage complies with the provisions of this license. You are fully accountable for your use of the models, which must not involve sharing any content that violates applicable laws, causes harm to individuals or groups, disseminates personal information intended for harm, spreads misinformation, or targets vulnerable populations. 
+
+

multitalk.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4b48e2eb148e2407711dfc29ef411820094e5684435d5791a6d34b53fe9e1db
+size 9947889040