MotionCrafter: One-Shot Motion Customization of Diffusion Models

The essence of a video lies in its dynamic motions, including character actions, object movements, and camera movements. While text-to-video generative diffusion models have recently advanced in creating diverse contents, controlling specific motions through text prompts remains a significant challenge. A primary issue is the coupling of appearance and motion, often leading to overfitting on appearance. To tackle this challenge, we introduce <PRE_TAG>MotionCrafter</POST_TAG>, a novel one-shot instance-guided motion customization method. <PRE_TAG>MotionCrafter</POST_TAG> employs a <PRE_TAG>parallel spatial-temporal architecture</POST_TAG> that injects the <PRE_TAG>reference motion</POST_TAG> into the <PRE_TAG>temporal component</POST_TAG> of the base model, while the <PRE_TAG>spatial module</POST_TAG> is independently adjusted for character or style control. To enhance the <PRE_TAG>disentanglement of motion and appearance</POST_TAG>, we propose an innovative <PRE_TAG>dual-branch motion disentanglement</POST_TAG> approach, comprising a motion disentanglement loss and an <PRE_TAG>appearance prior enhancement strategy</POST_TAG>. During training, a frozen base model provides <PRE_TAG>appearance normalization</POST_TAG>, effectively separating appearance from motion and thereby preserving diversity. Comprehensive quantitative and qualitative experiments, along with user preference tests, demonstrate that <PRE_TAG>MotionCrafter</POST_TAG> can successfully integrate dynamic motions while preserving the <PRE_TAG>coherence</POST_TAG> and <PRE_TAG>quality</POST_TAG> of the base model with a wide range of <PRE_TAG>appearance generation capabilities</POST_TAG>. Project page: https://zyxelsa.github.io/homepage-motioncrafter. Codes are available at https://github.com/zyxElsa/<PRE_TAG>MotionCrafter</POST_TAG>.