Delete EchoMimic

EchoMimic/.gitignore

@@ -1,4 +0,0 @@
-ffmpeg-4.4-amd64-static
-pretrained_weights
-output
-__pycache__

EchoMimic/.ipynb_checkpoints/Untitled-checkpoint.ipynb

@@ -1,6 +0,0 @@
-{
- "cells": [],
- "metadata": {},
- "nbformat": 4,
- "nbformat_minor": 5
-}

EchoMimic/LICENSE

@@ -1,201 +0,0 @@
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright [yyyy] [name of copyright owner]
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.

EchoMimic/README.md

@@ -1,285 +0,0 @@
-<h1 align='center'>EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditioning</h1>
-
-<div align='center'>
-    <a href='https://github.com/yuange250' target='_blank'>Zhiyuan Chen</a><sup>*</sup>&emsp;
-    <a href='https://github.com/JoeFannie' target='_blank'>Jiajiong Cao</a><sup>*</sup>&emsp;
-    <a href='https://github.com/octavianChen' target='_blank'>Zhiquan Chen</a><sup></sup>&emsp;
-    <a href='https://github.com/lymhust' target='_blank'>Yuming Li</a><sup></sup>&emsp;
-    <a href='https://github.com/' target='_blank'>Chenguang Ma</a><sup></sup>
-</div>
-<div align='center'>
-    *Equal Contribution.
-</div>
-
-<div align='center'>
-Terminal Technology Department, Alipay, Ant Group.
-</div>
-<br>
-<div align='center'>
-    <a href='https://badtobest.github.io/echomimic.html'><img src='https://img.shields.io/badge/Project-Page-blue'></a>
-    <a href='https://huggingface.co/BadToBest/EchoMimic'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Model-yellow'></a>
-    <a href='https://huggingface.co/spaces/BadToBest/EchoMimic'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Demo-yellow'></a>
-    <a href='https://www.modelscope.cn/models/BadToBest/EchoMimic'><img src='https://img.shields.io/badge/ModelScope-Model-purple'></a>
-    <a href='https://www.modelscope.cn/studios/BadToBest/BadToBest'><img src='https://img.shields.io/badge/ModelScope-Demo-purple'></a>
-    <a href='https://arxiv.org/abs/2407.08136'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a>
-    <a href='assets/echomimic.png'><img src='https://badges.aleen42.com/src/wechat.svg'></a>
-</div>
-
-## &#x1F4E3; &#x1F4E3; Updates
-* [2024.08.02] 🔥 EchoMimic is now available on [huggingface](https://huggingface.co/spaces/BadToBest/EchoMimic) with A100 GPU. Thanks Wenmeng Zhou@ModelScope.
-* [2024.07.25] 🔥🔥🔥 Accelerated models and pipe on **Audio Driven** are released. The inference speed can be improved by **10x** (from ~7mins/240frames to ~50s/240frames on V100 GPU)
-* [2024.07.23] 🔥 EchoMimic gradio demo on [modelscope](https://www.modelscope.cn/studios/BadToBest/BadToBest) is ready.
-* [2024.07.23] 🔥 EchoMimic gradio demo on [huggingface](https://huggingface.co/spaces/fffiloni/EchoMimic) is ready. Thanks Sylvain Filoni@fffiloni.
-* [2024.07.17] 🔥🔥🔥 Accelerated models and pipe on **Audio + Selected Landmarks** are released. The inference speed can be improved by **10x** (from ~7mins/240frames to ~50s/240frames on V100 GPU)
-* [2024.07.14] 🔥 [ComfyUI](https://github.com/smthemex/ComfyUI_EchoMimic) is now available. Thanks @smthemex for the contribution. 
-* [2024.07.13] 🔥 Thanks [NewGenAI](https://www.youtube.com/@StableAIHub) for the [video installation tutorial](https://www.youtube.com/watch?v=8R0lTIY7tfI).
-* [2024.07.13] 🔥 We release our pose&audio driven codes and models.
-* [2024.07.12] 🔥 WebUI and GradioUI versions are released. We thank @greengerong @Robin021 and @O-O1024 for their contributions.
-* [2024.07.12] 🔥 Our [paper](https://arxiv.org/abs/2407.08136) is in public on arxiv.
-* [2024.07.09] 🔥 We release our audio driven codes and models.
-
-## Gallery
-### Audio Driven (Sing)
-
-<table class="center">
-    
-<tr>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/d014d921-9f94-4640-97ad-035b00effbfe" muted="false"></video>
-    </td>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/877603a5-a4f9-4486-a19f-8888422daf78" muted="false"></video>
-    </td>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/e0cb5afb-40a6-4365-84f8-cb2834c4cfe7" muted="false"></video>
-    </td>
-</tr>
-
-</table>
-
-### Audio Driven (English)
-
-<table class="center">
-    
-<tr>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/386982cd-3ff8-470d-a6d9-b621e112f8a5" muted="false"></video>
-    </td>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/5c60bb91-1776-434e-a720-8857a00b1501" muted="false"></video>
-    </td>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/1f15adc5-0f33-4afa-b96a-2011886a4a06" muted="false"></video>
-    </td>
-</tr>
-
-</table>
-
-### Audio Driven (Chinese)
-
-<table class="center">
-    
-<tr>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/a8092f9a-a5dc-4cd6-95be-1831afaccf00" muted="false"></video>
-    </td>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/c8b5c59f-0483-42ef-b3ee-4cffae6c7a52" muted="false"></video>
-    </td>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/532a3e60-2bac-4039-a06c-ff6bf06cb4a4" muted="false"></video>
-    </td>
-</tr>
-
-</table>
-
-### Landmark Driven
-
-<table class="center">
-    
-<tr>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/1da6c46f-4532-4375-a0dc-0a4d6fd30a39" muted="false"></video>
-    </td>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/d4f4d5c1-e228-463a-b383-27fb90ed6172" muted="false"></video>
-    </td>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/18bd2c93-319e-4d1c-8255-3f02ba717475" muted="false"></video>
-    </td>
-</tr>
-
-</table>
-
-### Audio + Selected Landmark Driven
-
-<table class="center">
-    
-<tr>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/4a29d735-ec1b-474d-b843-3ff0bdf85f55" muted="false"></video>
-    </td>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/b994c8f5-8dae-4dd8-870f-962b50dc091f" muted="false"></video>
-    </td>
-    <td width=30% style="border: none">
-        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/955c1d51-07b2-494d-ab93-895b9c43b896" muted="false"></video>
-    </td>
-</tr>
-
-</table>
-
-**（Some demo images above are sourced from image websites. If there is any infringement, we will immediately remove them and apologize.）**
-
-## Installation
-
-### Download the Codes
-
-```bash
-  git clone https://github.com/BadToBest/EchoMimic
-  cd EchoMimic
-```
-
-### Python Environment Setup
-
-- Tested System Environment: Centos 7.2/Ubuntu 22.04, Cuda >= 11.7
-- Tested GPUs: A100(80G) / RTX4090D (24G) / V100(16G)
-- Tested Python Version: 3.8 / 3.10 / 3.11
-
-Create conda environment (Recommended):
-
-```bash
-  conda create -n echomimic python=3.8
-  conda activate echomimic
-```
-
-Install packages with `pip`
-```bash
-  pip install -r requirements.txt
-```
-
-### Download ffmpeg-static
-Download and decompress [ffmpeg-static](https://www.johnvansickle.com/ffmpeg/old-releases/ffmpeg-4.4-amd64-static.tar.xz), then
-```
-export FFMPEG_PATH=/path/to/ffmpeg-4.4-amd64-static
-```
-
-### Download pretrained weights
-
-```shell
-git lfs install
-git clone https://huggingface.co/BadToBest/EchoMimic pretrained_weights
-```
-
-The **pretrained_weights** is organized as follows.
-
-```
-./pretrained_weights/
-├── denoising_unet.pth
-├── reference_unet.pth
-├── motion_module.pth
-├── face_locator.pth
-├── sd-vae-ft-mse
-│   └── ...
-├── sd-image-variations-diffusers
-│   └── ...
-└── audio_processor
-    └── whisper_tiny.pt
-```
-
-In which **denoising_unet.pth** / **reference_unet.pth** / **motion_module.pth** / **face_locator.pth** are the main checkpoints of **EchoMimic**. Other models in this hub can be also downloaded from it's original hub, thanks to their brilliant works:
-- [sd-vae-ft-mse](https://huggingface.co/stabilityai/sd-vae-ft-mse)
-- [sd-image-variations-diffusers](https://huggingface.co/lambdalabs/sd-image-variations-diffusers)
-- [audio_processor(whisper)](https://openaipublic.azureedge.net/main/whisper/models/65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9/tiny.pt)
-
-### Audio-Drived Algo Inference 
-Run the python inference script:
-
-```bash
-  python -u infer_audio2vid.py
-  python -u infer_audio2vid_pose.py
-```
-
-### Audio-Drived Algo Inference On Your Own Cases 
-
-Edit the inference config file **./configs/prompts/animation.yaml**, and add your own case:
-
-```bash
-test_cases:
-  "path/to/your/image":
-    - "path/to/your/audio"
-```
-
-The run the python inference script:
-```bash
-  python -u infer_audio2vid.py
-```
-
-### Motion Alignment between Ref. Img. and Driven Vid.
-
-(Firstly download the checkpoints with '_pose.pth' postfix from huggingface)
-
-Edit driver_video and ref_image to your path in demo_motion_sync.py, then run
-```bash
-  python -u demo_motion_sync.py
-```
-
-### Audio&Pose-Drived Algo Inference
-Edit ./configs/prompts/animation_pose.yaml, then run
-```bash
-  python -u infer_audio2vid_pose.py
-```
-
-### Pose-Drived Algo Inference
-Set draw_mouse=True in line 135 of infer_audio2vid_pose.py. Edit ./configs/prompts/animation_pose.yaml, then run
-```bash
-  python -u infer_audio2vid_pose.py
-```
-
-### Run the Gradio UI
-
-Thanks to the contribution from @Robin021:
-
-```bash
-
-python -u webgui.py --server_port=3000
-
-```
-
-## Release Plans
-
-|  Status  | Milestone                                                                | ETA |
-|:--------:|:-------------------------------------------------------------------------|:--:|
-|    ✅    | The inference source code of the Audio-Driven algo meet everyone on GitHub   | 9th July, 2024 |
-|    ✅    | Pretrained models trained on English and Mandarin Chinese to be released | 9th July, 2024 |
-|    ✅    | The inference source code of the Pose-Driven algo meet everyone on GitHub   | 13th July, 2024 |
-|    ✅    | Pretrained models with better pose control to be released                | 13th July, 2024 |
-|    ✅    | Accelerated models to be released                                        | 17th July, 2024 |
-|    🚀    | Pretrained models with better sing performance to be released            | TBD |
-|    🚀    | Large-Scale and High-resolution Chinese-Based Talking Head Dataset       | TBD |
-
-## Acknowledgements
-
-We would like to thank the contributors to the [AnimateDiff](https://github.com/guoyww/AnimateDiff), [Moore-AnimateAnyone](https://github.com/MooreThreads/Moore-AnimateAnyone) and [MuseTalk](https://github.com/TMElyralab/MuseTalk) repositories, for their open research and exploration. 
-
-We are also grateful to [V-Express](https://github.com/tencent-ailab/V-Express) and [hallo](https://github.com/fudan-generative-vision/hallo) for their outstanding work in the area of diffusion-based talking heads.
-
-If we missed any open-source projects or related articles, we would like to complement the acknowledgement of this specific work immediately.
-
-## Citation
-
-If you find our work useful for your research, please consider citing the paper :
-
-```
-@misc{chen2024echomimic,
-  title={EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditioning},
-  author={Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, Chenguang Ma},
-  year={2024},
-  archivePrefix={arXiv},
-  primaryClass={cs.CV}
-}
-```
-
-## Star History
-
-[![Star History Chart](https://api.star-history.com/svg?repos=BadToBest/EchoMimic&type=Date)](https://star-history.com/?spm=5176.28103460.0.0.342a3da23STWrU#BadToBest/EchoMimic&Date)

EchoMimic/Untitled.ipynb

@@ -1,315 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "4a8767df-feea-4bda-8bfb-ab07f667cd11",
-   "metadata": {},
-   "outputs": [
-    {
-     "ename": "ModuleNotFoundError",
-     "evalue": "No module named 'cv2'",
-     "output_type": "error",
-     "traceback": [
-      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
-      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
-      "Cell \u001b[0;32mIn[1], line 12\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mdatetime\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m datetime\n\u001b[1;32m     10\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpathlib\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Path\n\u001b[0;32m---> 12\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mcv2\u001b[39;00m\n\u001b[1;32m     13\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mnumpy\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mnp\u001b[39;00m\n\u001b[1;32m     14\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mtorch\u001b[39;00m\n",
-      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'cv2'"
-     ]
-    }
-   ],
-   "source": [
-    "#!/usr/bin/env python\n",
-    "# -*- coding: UTF-8 -*-\n",
-    "'''\n",
-    "webui\n",
-    "'''\n",
-    "\n",
-    "import os\n",
-    "import random\n",
-    "from datetime import datetime\n",
-    "from pathlib import Path\n",
-    "\n",
-    "import cv2\n",
-    "import numpy as np\n",
-    "import torch\n",
-    "from diffusers import AutoencoderKL, DDIMScheduler\n",
-    "from omegaconf import OmegaConf\n",
-    "from PIL import Image\n",
-    "from src.models.unet_2d_condition import UNet2DConditionModel\n",
-    "from src.models.unet_3d_echo import EchoUNet3DConditionModel\n",
-    "from src.models.whisper.audio2feature import load_audio_model\n",
-    "from src.pipelines.pipeline_echo_mimic import Audio2VideoPipeline\n",
-    "from src.utils.util import save_videos_grid, crop_and_pad\n",
-    "from src.models.face_locator import FaceLocator\n",
-    "from moviepy.editor import VideoFileClip, AudioFileClip\n",
-    "from facenet_pytorch import MTCNN\n",
-    "import argparse\n",
-    "\n",
-    "import gradio as gr\n",
-    "\n",
-    "default_values = {\n",
-    "    \"width\": 512,\n",
-    "    \"height\": 512,\n",
-    "    \"length\": 1200,\n",
-    "    \"seed\": 420,\n",
-    "    \"facemask_dilation_ratio\": 0.1,\n",
-    "    \"facecrop_dilation_ratio\": 0.5,\n",
-    "    \"context_frames\": 12,\n",
-    "    \"context_overlap\": 3,\n",
-    "    \"cfg\": 2.5,\n",
-    "    \"steps\": 30,\n",
-    "    \"sample_rate\": 16000,\n",
-    "    \"fps\": 24,\n",
-    "    \"device\": \"cuda\"\n",
-    "}\n",
-    "\n",
-    "ffmpeg_path = os.getenv('FFMPEG_PATH')\n",
-    "if ffmpeg_path is None:\n",
-    "    print(\"please download ffmpeg-static and export to FFMPEG_PATH. \\nFor example: export FFMPEG_PATH=/musetalk/ffmpeg-4.4-amd64-static\")\n",
-    "elif ffmpeg_path not in os.getenv('PATH'):\n",
-    "    print(\"add ffmpeg to path\")\n",
-    "    os.environ[\"PATH\"] = f\"{ffmpeg_path}:{os.environ['PATH']}\"\n",
-    "\n",
-    "\n",
-    "config_path = \"./configs/prompts/animation.yaml\"\n",
-    "config = OmegaConf.load(config_path)\n",
-    "if config.weight_dtype == \"fp16\":\n",
-    "    weight_dtype = torch.float16\n",
-    "else:\n",
-    "    weight_dtype = torch.float32\n",
-    "\n",
-    "device = \"cuda\"\n",
-    "if not torch.cuda.is_available():\n",
-    "    device = \"cpu\"\n",
-    "\n",
-    "inference_config_path = config.inference_config\n",
-    "infer_config = OmegaConf.load(inference_config_path)\n",
-    "\n",
-    "############# model_init started #############\n",
-    "## vae init\n",
-    "vae = AutoencoderKL.from_pretrained(config.pretrained_vae_path).to(\"cuda\", dtype=weight_dtype)\n",
-    "\n",
-    "## reference net init\n",
-    "reference_unet = UNet2DConditionModel.from_pretrained(\n",
-    "    config.pretrained_base_model_path,\n",
-    "    subfolder=\"unet\",\n",
-    ").to(dtype=weight_dtype, device=device)\n",
-    "reference_unet.load_state_dict(torch.load(config.reference_unet_path, map_location=\"cpu\"))\n",
-    "\n",
-    "## denoising net init\n",
-    "if os.path.exists(config.motion_module_path):\n",
-    "    ### stage1 + stage2\n",
-    "    denoising_unet = EchoUNet3DConditionModel.from_pretrained_2d(\n",
-    "        config.pretrained_base_model_path,\n",
-    "        config.motion_module_path,\n",
-    "        subfolder=\"unet\",\n",
-    "        unet_additional_kwargs=infer_config.unet_additional_kwargs,\n",
-    "    ).to(dtype=weight_dtype, device=device)\n",
-    "else:\n",
-    "    ### only stage1\n",
-    "    denoising_unet = EchoUNet3DConditionModel.from_pretrained_2d(\n",
-    "        config.pretrained_base_model_path,\n",
-    "        \"\",\n",
-    "        subfolder=\"unet\",\n",
-    "        unet_additional_kwargs={\n",
-    "            \"use_motion_module\": False,\n",
-    "            \"unet_use_temporal_attention\": False,\n",
-    "            \"cross_attention_dim\": infer_config.unet_additional_kwargs.cross_attention_dim\n",
-    "        }\n",
-    "    ).to(dtype=weight_dtype, device=device)\n",
-    "\n",
-    "denoising_unet.load_state_dict(torch.load(config.denoising_unet_path, map_location=\"cpu\"), strict=False)\n",
-    "\n",
-    "## face locator init\n",
-    "face_locator = FaceLocator(320, conditioning_channels=1, block_out_channels=(16, 32, 96, 256)).to(dtype=weight_dtype, device=\"cuda\")\n",
-    "face_locator.load_state_dict(torch.load(config.face_locator_path))\n",
-    "\n",
-    "## load audio processor params\n",
-    "audio_processor = load_audio_model(model_path=config.audio_model_path, device=device)\n",
-    "\n",
-    "## load face detector params\n",
-    "face_detector = MTCNN(image_size=320, margin=0, min_face_size=20, thresholds=[0.6, 0.7, 0.7], factor=0.709, post_process=True, device=device)\n",
-    "\n",
-    "############# model_init finished #############\n",
-    "\n",
-    "sched_kwargs = OmegaConf.to_container(infer_config.noise_scheduler_kwargs)\n",
-    "scheduler = DDIMScheduler(**sched_kwargs)\n",
-    "\n",
-    "pipe = Audio2VideoPipeline(\n",
-    "    vae=vae,\n",
-    "    reference_unet=reference_unet,\n",
-    "    denoising_unet=denoising_unet,\n",
-    "    audio_guider=audio_processor,\n",
-    "    face_locator=face_locator,\n",
-    "    scheduler=scheduler,\n",
-    ").to(\"cuda\", dtype=weight_dtype)\n",
-    "\n",
-    "def select_face(det_bboxes, probs):\n",
-    "    ## max face from faces that the prob is above 0.8\n",
-    "    ## box: xyxy\n",
-    "    if det_bboxes is None or probs is None:\n",
-    "        return None\n",
-    "    filtered_bboxes = []\n",
-    "    for bbox_i in range(len(det_bboxes)):\n",
-    "        if probs[bbox_i] > 0.8:\n",
-    "            filtered_bboxes.append(det_bboxes[bbox_i])\n",
-    "    if len(filtered_bboxes) == 0:\n",
-    "        return None\n",
-    "    sorted_bboxes = sorted(filtered_bboxes, key=lambda x:(x[3]-x[1]) * (x[2] - x[0]), reverse=True)\n",
-    "    return sorted_bboxes[0]\n",
-    "\n",
-    "def process_video(uploaded_img, uploaded_audio, width, height, length, seed, facemask_dilation_ratio, facecrop_dilation_ratio, context_frames, context_overlap, cfg, steps, sample_rate, fps, device):\n",
-    "\n",
-    "    if seed is not None and seed > -1:\n",
-    "        generator = torch.manual_seed(seed)\n",
-    "    else:\n",
-    "        generator = torch.manual_seed(random.randint(100, 1000000))\n",
-    "\n",
-    "    #### face musk prepare\n",
-    "    face_img = cv2.imread(uploaded_img)\n",
-    "    face_mask = np.zeros((face_img.shape[0], face_img.shape[1])).astype('uint8')\n",
-    "    det_bboxes, probs = face_detector.detect(face_img)\n",
-    "    select_bbox = select_face(det_bboxes, probs)\n",
-    "    if select_bbox is None:\n",
-    "        face_mask[:, :] = 255\n",
-    "    else:\n",
-    "        xyxy = select_bbox[:4]\n",
-    "        xyxy = np.round(xyxy).astype('int')\n",
-    "        rb, re, cb, ce = xyxy[1], xyxy[3], xyxy[0], xyxy[2]\n",
-    "        r_pad = int((re - rb) * facemask_dilation_ratio)\n",
-    "        c_pad = int((ce - cb) * facemask_dilation_ratio)\n",
-    "        face_mask[rb - r_pad : re + r_pad, cb - c_pad : ce + c_pad] = 255\n",
-    "        \n",
-    "        #### face crop\n",
-    "        r_pad_crop = int((re - rb) * facecrop_dilation_ratio)\n",
-    "        c_pad_crop = int((ce - cb) * facecrop_dilation_ratio)\n",
-    "        crop_rect = [max(0, cb - c_pad_crop), max(0, rb - r_pad_crop), min(ce + c_pad_crop, face_img.shape[1]), min(re + r_pad_crop, face_img.shape[0])]\n",
-    "        face_img = crop_and_pad(face_img, crop_rect)\n",
-    "        face_mask = crop_and_pad(face_mask, crop_rect)\n",
-    "        face_img = cv2.resize(face_img, (width, height))\n",
-    "        face_mask = cv2.resize(face_mask, (width, height))\n",
-    "\n",
-    "    ref_image_pil = Image.fromarray(face_img[:, :, [2, 1, 0]])\n",
-    "    face_mask_tensor = torch.Tensor(face_mask).to(dtype=weight_dtype, device=\"cuda\").unsqueeze(0).unsqueeze(0).unsqueeze(0) / 255.0\n",
-    "    \n",
-    "    video = pipe(\n",
-    "        ref_image_pil,\n",
-    "        uploaded_audio,\n",
-    "        face_mask_tensor,\n",
-    "        width,\n",
-    "        height,\n",
-    "        length,\n",
-    "        steps,\n",
-    "        cfg,\n",
-    "        generator=generator,\n",
-    "        audio_sample_rate=sample_rate,\n",
-    "        context_frames=context_frames,\n",
-    "        fps=fps,\n",
-    "        context_overlap=context_overlap\n",
-    "    ).videos\n",
-    "\n",
-    "    save_dir = Path(\"output/tmp\")\n",
-    "    save_dir.mkdir(exist_ok=True, parents=True)\n",
-    "    output_video_path = save_dir / \"output_video.mp4\"\n",
-    "    save_videos_grid(video, str(output_video_path), n_rows=1, fps=fps)\n",
-    "\n",
-    "    video_clip = VideoFileClip(str(output_video_path))\n",
-    "    audio_clip = AudioFileClip(uploaded_audio)\n",
-    "    final_output_path = save_dir / \"output_video_with_audio.mp4\"\n",
-    "    video_clip = video_clip.set_audio(audio_clip)\n",
-    "    video_clip.write_videofile(str(final_output_path), codec=\"libx264\", audio_codec=\"aac\")\n",
-    "\n",
-    "    return final_output_path\n",
-    "  \n",
-    "with gr.Blocks() as demo:\n",
-    "    gr.Markdown('# EchoMimic')\n",
-    "    gr.Markdown('![]()')\n",
-    "    with gr.Row():\n",
-    "        with gr.Column():\n",
-    "            uploaded_img = gr.Image(type=\"filepath\", label=\"Reference Image\")\n",
-    "            uploaded_audio = gr.Audio(type=\"filepath\", label=\"Input Audio\")\n",
-    "        with gr.Column():\n",
-    "            output_video = gr.Video()\n",
-    "\n",
-    "    with gr.Accordion(\"Configuration\", open=False):\n",
-    "        width = gr.Slider(label=\"Width\", minimum=128, maximum=1024, value=default_values[\"width\"])\n",
-    "        height = gr.Slider(label=\"Height\", minimum=128, maximum=1024, value=default_values[\"height\"])\n",
-    "        length = gr.Slider(label=\"Length\", minimum=100, maximum=5000, value=default_values[\"length\"])\n",
-    "        seed = gr.Slider(label=\"Seed\", minimum=0, maximum=10000, value=default_values[\"seed\"])\n",
-    "        facemask_dilation_ratio = gr.Slider(label=\"Facemask Dilation Ratio\", minimum=0.0, maximum=1.0, step=0.01, value=default_values[\"facemask_dilation_ratio\"])\n",
-    "        facecrop_dilation_ratio = gr.Slider(label=\"Facecrop Dilation Ratio\", minimum=0.0, maximum=1.0, step=0.01, value=default_values[\"facecrop_dilation_ratio\"])\n",
-    "        context_frames = gr.Slider(label=\"Context Frames\", minimum=0, maximum=50, step=1, value=default_values[\"context_frames\"])\n",
-    "        context_overlap = gr.Slider(label=\"Context Overlap\", minimum=0, maximum=10, step=1, value=default_values[\"context_overlap\"])\n",
-    "        cfg = gr.Slider(label=\"CFG\", minimum=0.0, maximum=10.0, step=0.1, value=default_values[\"cfg\"])\n",
-    "        steps = gr.Slider(label=\"Steps\", minimum=1, maximum=100, step=1, value=default_values[\"steps\"])\n",
-    "        sample_rate = gr.Slider(label=\"Sample Rate\", minimum=8000, maximum=48000, step=1000, value=default_values[\"sample_rate\"])\n",
-    "        fps = gr.Slider(label=\"FPS\", minimum=1, maximum=60, step=1, value=default_values[\"fps\"])\n",
-    "        device = gr.Radio(label=\"Device\", choices=[\"cuda\", \"cpu\"], value=default_values[\"device\"])\n",
-    "\n",
-    "    generate_button = gr.Button(\"Generate Video\")\n",
-    "\n",
-    "    def generate_video(uploaded_img, uploaded_audio, width, height, length, seed, facemask_dilation_ratio, facecrop_dilation_ratio, context_frames, context_overlap, cfg, steps, sample_rate, fps, device):\n",
-    "\n",
-    "        final_output_path = process_video(\n",
-    "            uploaded_img, uploaded_audio, width, height, length, seed, facemask_dilation_ratio, facecrop_dilation_ratio, context_frames, context_overlap, cfg, steps, sample_rate, fps, device\n",
-    "        )        \n",
-    "        output_video= final_output_path\n",
-    "        return final_output_path\n",
-    "\n",
-    "    generate_button.click(\n",
-    "        generate_video,\n",
-    "        inputs=[\n",
-    "            uploaded_img,\n",
-    "            uploaded_audio,\n",
-    "            width,\n",
-    "            height,\n",
-    "            length,\n",
-    "            seed,\n",
-    "            facemask_dilation_ratio,\n",
-    "            facecrop_dilation_ratio,\n",
-    "            context_frames,\n",
-    "            context_overlap,\n",
-    "            cfg,\n",
-    "            steps,\n",
-    "            sample_rate,\n",
-    "            fps,\n",
-    "            device\n",
-    "        ],\n",
-    "        outputs=output_video\n",
-    "    )\n",
-    "parser = argparse.ArgumentParser(description='EchoMimic')\n",
-    "parser.add_argument('--server_name', type=str, default='0.0.0.0', help='Server name')\n",
-    "parser.add_argument('--server_port', type=int, default=7680, help='Server port')\n",
-    "args = parser.parse_args()\n",
-    "\n",
-    "# demo.launch(server_name=args.server_name, server_port=args.server_port, inbrowser=True)\n",
-    "\n",
-    "if __name__ == '__main__':\n",
-    "    #demo.launch(server_name='0.0.0.0')\n",
-    "    demo.launch(server_name=args.server_name, server_port=args.server_port, inbrowser=True, share=True)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.13"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

EchoMimic/assets/driven_videos/a.mp4

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:f4889c1ecc27aea2be958f67ad417bd0b11b6626d5ca1edd692c52a3fe1e73ee
-size 1798555

EchoMimic/assets/driven_videos/b.mp4

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:97c5cc480eecf21130b5d17807f23242b67861b58a36f308df7159aa5d0b8dc0
-size 2072709

EchoMimic/assets/driven_videos/c.mp4

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:228736987a683026e3220ed1bca8775ec44692e75cbddbb4c132903d90b022a5
-size 2340154

EchoMimic/assets/test_pose_demo_audios/movie_0_clip_0.wav

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:eb6994f9ce1788e7672d2489fb05e21459e0869e44202634a503e2a556ac939c
-size 1771130

EchoMimic/assets/test_pose_demo_pose/0.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:79f459214a9284b9a80b38c3fb64c9d24890fe5472a0220f6c0327742c0cc856
-size 11625

EchoMimic/assets/test_pose_demo_pose/1.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:acb82a39521d5d774f67546e94aaf2cf743e2f16e4894d7819241dd290c004ee
-size 11625

EchoMimic/assets/test_pose_demo_pose/10.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:a30a8350432635fc65861c30c4433b03f0ddff615628356cd8e6eeb36bd0c985
-size 11625

EchoMimic/assets/test_pose_demo_pose/100.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:425a4c9428f09e1b73da0b31d4aa5df50421085b18dbc3eccfdf85ab40e97770
-size 11625

EchoMimic/assets/test_pose_demo_pose/101.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:d06bc52421328959f16fc95e7495767c982870a88e1ec78d681cc06dc7bd4f09
-size 11625

EchoMimic/assets/test_pose_demo_pose/102.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:135ee941d0c7bf27c1808d19a2e56a89e4f96a5b89c3f9c8793682cf2a707bf0
-size 11625

EchoMimic/assets/test_pose_demo_pose/103.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:8f0a3b12714d02f933cca212689d5727fd5a87e3fdd29c7ed6d403a0367a2312
-size 11625

EchoMimic/assets/test_pose_demo_pose/104.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:43bec0c1442aaba6a20ec49aa472fbd35ce3b42f24567f698a92ade73de6fc50
-size 11625

EchoMimic/assets/test_pose_demo_pose/105.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:4143b52e0430a7ae75d2a9785bdea5505ae2936e47531273086961a2f63c2cac
-size 11625

EchoMimic/assets/test_pose_demo_pose/106.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:10396bf15c0fa9bd7e3d925aced0009ae74c3353daa5076dd74bacad37572603
-size 11625

EchoMimic/assets/test_pose_demo_pose/107.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:7ebcf965e910b130f48f1c1dd35b68176f13e1d7f546681eaf37a959b0325ebb
-size 11625

EchoMimic/assets/test_pose_demo_pose/108.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:25546b95ca775dc94bd9439ab748cacab1d13b8ddae2a35c931a92fc8669c22f
-size 11625

EchoMimic/assets/test_pose_demo_pose/109.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:16068bbde36d595f268b2ec37baa6b7dcea935b50345d989066f4d8d7d2aac1f
-size 11625

EchoMimic/assets/test_pose_demo_pose/11.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:e534fe28d74fb88578fee00b40aa3b368065bb67e6e240664024f7bd5fc37b64
-size 11625

EchoMimic/assets/test_pose_demo_pose/110.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:49bd7320a4d2eb64bf04eff284eacc124a3bdfd3313f825d092788d42c3f6ad6
-size 11625

EchoMimic/assets/test_pose_demo_pose/111.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:d6eea05553a2f4ef6ec9c51c5973ab576d86cd58c26a3777084b1baccfe61b44
-size 11625

EchoMimic/assets/test_pose_demo_pose/112.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:333655dcb1cabb37f217569690e09fa884e8c6634f62610b1234476a2cddd177
-size 11625

EchoMimic/assets/test_pose_demo_pose/113.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:338363e0f2b69e23d29be2aa9952e8de4a8d79ae27f0b015caded1a1231b74a4
-size 11625

EchoMimic/assets/test_pose_demo_pose/114.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:c4099b77a8fb45d1e8f09d43af6e46d7bb59a60734f7f37f12e1cff31c07b91a
-size 11625

EchoMimic/assets/test_pose_demo_pose/115.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:e5611bce0729c35305f3b67b0b7492786a109f071c06e26600009b6464238d81
-size 11625

EchoMimic/assets/test_pose_demo_pose/116.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:a79cac90622ee10de3a4d1b2fe1373155ead1fe325bba1c5fa12763fa3800718
-size 11625

EchoMimic/assets/test_pose_demo_pose/117.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:b3ada492fe7572daff2a8ac567c86f771bf90b5bd3147e8ad799c73ea83dbfb4
-size 11625