| # WarpTuber | |
| A real-time face animation tool that brings static images and videos to life using your webcam. | |
| Tutorial: https://www.youtube.com/watch?v=w1_KY5E7Aig | |
| ## Overview | |
| WarpTuber allows you to animate portraits or character images by mapping your facial expressions and movements to them in real-time. It supports both static images and video animations, making it perfect for virtual avatars, content creation, and streaming. | |
| ## Quick Start | |
| 1. Connect your webcam | |
| 2. Choose one of the following launch options: | |
| - `camera_image.bat` - For animating a static image | |
| - `camera_video.bat` - For using a video animation with facial expressions | |
| ## Setup Requirements | |
| - Windows 10/11 | |
| - Webcam | |
| - Python environment (included in the `venv` folder) | |
| - NVIDIA GPU with CUDA support (recommended) | |
| - Python local install | |
| ## Usage Instructions | |
| ### Using Static Images (`camera_image.bat`) | |
| Use this mode when you want to animate a single static portrait or character image with your facial movements. | |
| 1. Place your image in `assets/examples/source/` and name it `main.jpg` | |
| 2. Run `camera_image.bat` | |
| 3. When prompted, select your webcam index (usually 0 for the default webcam) | |
| 4. The application will open with your animated image | |
| ### Using Video Animations (`camera_video.bat`) | |
| Use this mode when you want to combine a pre-recorded animation/idle loop with your facial expressions. | |
| 1. Place your video in `assets/examples/source/` and name it `main.mp4` | |
| 2. Run `camera_video.bat` | |
| 3. When prompted, select your webcam index (usually 0 for the default webcam) | |
| 4. The application will open with your video animation that responds to your facial movements | |
| ### File Requirements | |
| - **Static Image**: Must be named `main.jpg` and placed in `assets/examples/source/` | |
| - **Animation Video**: Must be named `main.mp4` and placed in `assets/examples/source/` | |
| For best results: | |
| - Images should be clear portraits with visible facial features | |
| - Videos should be looping animations with consistent lighting | |
| - Both should have dimensions of at least 512x512 pixels | |
| ### Image Compatibility and Facial Landmark Detection | |
| WarpTuber relies on facial landmark detection to animate images. Not all images will work properly, especially: | |
| - Highly stylized anime or cartoon characters | |
| - Images with unusual facial proportions | |
| - Artwork with abstract facial features | |
| - Images with poor lighting or low contrast | |
| - Side-profile portraits (faces should be mostly front-facing) | |
| If you encounter an error message like `"no face in [image path]! exit!"`, this means the system failed to detect facial landmarks in your image. | |
| **Solutions for facial landmark detection issues:** | |
| 1. **Try a different image**: Use photographs or realistic illustrations with clear facial features | |
| 2. **Enable animal mode**: For cartoon characters, try running with the `--animal` parameter: | |
| ``` | |
| camera_image.bat --animal | |
| ``` | |
| 3. **Adjust the image**: Edit your image to make facial features more prominent: | |
| - Increase contrast around eyes, nose, and mouth | |
| - Ensure the face is well-lit and centered | |
| - Crop the image to focus more on the face | |
| - Convert stylized characters to a more realistic style using AI tools | |
| 4. **Pre-process with face enhancement**: Use photo editing software to enhance facial features before using with WarpTuber | |
| Note that even with these adjustments, some highly stylized images may not work with the current facial landmark detection system. | |
| ## Advanced Configuration | |
| ### Command Line Parameters | |
| Both batch files support additional parameters: | |
| - `--src_image [path]` - Specify a custom source image/video path | |
| - `--animal` - Enable animal face mode | |
| - `--paste_back` - Enable background preservation (enabled by default) | |
| - `--interactive` - Enable interactive controls (enabled by default) | |
| - `--advanced_ui` - Enable advanced UI controls | |
| Example: `camera_image.bat --src_image assets/my_custom_folder/portrait.jpg --animal` | |
| ### TensorRT Optimization | |
| This repository includes pre-compiled TensorRT models for optimal performance. If you encounter issues with the included models, you may need to compile your own: | |
| 1. Navigate to the `checkpoints` directory and locate any existing `.trt` files: | |
| - Human model files: `checkpoints/liveportrait_onnx/*.trt` | |
| - Animal model files: `checkpoints/liveportrait_animal_onnx/*.trt` | |
| - Example path: `C:\FLivePort\WarpTuber\checkpoints\liveportrait_onnx\stitching_lip.trt` | |
| 2. Delete these `.trt` files if you're experiencing compatibility issues | |
| 3. Run `scripts/all_onnx2trt.bat` to recompile all models | |
| - This will convert all ONNX models to TensorRT format optimized for your specific GPU | |
| - The conversion process may take several minutes to complete | |
| Note: Compiling your own TensorRT models is only necessary if the pre-compiled models don't work on your system. The compilation process creates optimized models specifically for your GPU hardware. | |
| ## Troubleshooting | |
| - If no webcams are detected, ensure your camera is properly connected and not in use by another application | |
| - If the animation appears laggy, try closing other GPU-intensive applications | |
| - If you encounter model loading errors, try running the `scripts/all_onnx2trt.bat` script to compile models for your specific GPU | |
| - If you see the error `"no face in driving frame"`, ensure your webcam can clearly see your face with good lighting | |
| ## License | |
| MIT License | |
| Copyright (c) 2024 Kuaishou Visual Generation and Interaction Center | |
| Permission is hereby granted, free of charge, to any person obtaining a copy | |
| of this software and associated documentation files (the "Software"), to deal | |
| in the Software without restriction, including without limitation the rights | |
| to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | |
| copies of the Software, and to permit persons to whom the Software is | |
| furnished to do so, subject to the following conditions: | |
| The above copyright notice and this permission notice shall be included in all | |
| copies or substantial portions of the Software. | |
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | |
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | |
| FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | |
| AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | |
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | |
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | |
| SOFTWARE. | |
| --- | |
| **Important Note**: | |
| - The code of InsightFace is released under the MIT License. | |
| - The models of InsightFace are for non-commercial research purposes only. | |
| - If you want to use the WarpTuber project for commercial purposes, you should remove and replace InsightFace's detection models to fully comply with the MIT license. | |
| ## Acknowledgements | |
| This project is based on [FasterLivePortrait](https://github.com/warmshao/FasterLivePortrait) by [warmshao](https://github.com/warmshao), which is an optimized implementation of the original [LivePortrait](https://github.com/KwaiVGI/LivePortrait) technology developed by the Kuaishou Technology team. | |
| The original LivePortrait technology was created by Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang from Kuaishou Technology and academic institutions. | |
| WarpTuber is a customized implementation that simplifies the usage of this technology with a focus on ease of use for content creators and streamers. | |
| Key technologies used: | |
| - Face landmark detection and tracking | |
| - Neural rendering for portrait animation | |
| - TensorRT optimization for real-time performance | |
| Special thanks to the original developers for making this technology accessible and open source. | |
| For more information about the original research, please visit [liveportrait.github.io](https://liveportrait.github.io/). |