Human-VDM: Learning Single-Image 3D Human Gaussian Splatting from Video Diffusion Models
Zhibin Liu Haoye Dong Aviral Chharia Hefeng Wu


[

[
    [
    
=====





    type=figure
    
    < g r a p h i c s >

    

    figureHuman-VDM for generating 3D humans from a single image. Given a single RGB human image, Human-VDM aims to generate high-fidelity 3D human. Human-VDM preserves face identity, delivers realistic texture, ensures accurate geometry, and maintains a valid pose of the generated 3D human, surpassing the current state-of-the-art models.
    

    


]

§ ABSTRACT 
Generating lifelike 3D humans from a single RGB image remains a challenging task in computer vision, as it requires accurate modeling of geometry, high-quality texture, and plausible unseen parts. Existing methods typically use multi-view diffusion models for 3D generation, but they often face inconsistent view issues, which hinder high-quality 3D human generation. To address this, we propose Human-VDM, a novel method for generating 3D human from a single RGB image using Video Diffusion Models. Human-VDM provides temporally consistent views for 3D human generation using Gaussian Splatting. It consists of three modules: a view-consistent human video diffusion module, a video augmentation module, and a Gaussian Splatting module. First, a single image is fed into a human video diffusion module to generate a coherent human video. Next, the video augmentation module applies super-resolution and video interpolation to enhance the textures and geometric smoothness of the generated video. Finally, the 3D Human Gaussian Splatting module learns lifelike humans under the guidance of these high-resolution and view-consistent images. Experiments demonstrate that Human-VDM achieves high-quality 3D human from a single image, outperforming state-of-the-art methods in both generation quality and quantity.





§ INTRODUCTION

Generating 3D humans from a single RGB image has gained significant attention in recent years due to its versatile applications in filmmaking, video games, human-robotic interaction, etc. However, existing approaches for 3D human generation largely rely on multi-view diffusion models, which often suffer from inconsistent views and lead to artifacts. To address this problem, we propose a 3D Human Gaussian Splatting framework that allows users to generate 3D humans from a single 2D image input while ensuring accurate geometry and realistic appearance. However, generating 3D humans using only a single RGB image presents a significant challenge due to its inherent ambiguity, which necessitates inferring unseen geometry and appearance that are not directly captured in a 2D image.

Current approaches address this challenge by incorporating parametric human shape models, such as SCAPE <cit.> and SMPL <cit.>. However, these methods exclusively focus on reconstructing the human shape, neglecting the appearance details crucial for a fully realistic 3D representation. Earlier works, like PIFu <cit.>, attempted to address this gap with a data-driven approach. They used CycleGAN <cit.> and residual blocks <cit.> trained on image-3D pairs. However, such methods often struggle with novel appearances or poses mainly due to the lack of sufficient 3D training information. Subsequent methods, such as ECON <cit.> and 2K2K <cit.>, enhanced performance by incorporating depth or normal estimation into the generation process. SIFU <cit.> proposed a 3D human generation method using a side-view based Transformer with 3D aware Refinement. Despite the improvements, these methods often lack detail or result in inaccurate geometry, particularly with high-resolution input images.

Recently, SiTH <cit.> integrated a generative diffusion model into the 3D human generation pipeline to produce realistic textures and geometries, especially in unobserved regions. Ultraman <cit.> introduced a multi-view image generation model that helped in providing essential appearance priors aiding the generation process. Although diffusion models <cit.>, trained on extensive image datasets, have demonstrated potential for creating 3D humans, multi-view diffusion often struggles with generating view-consistent images and tends to introduce artifacts in the generated 3D humans.

This paper proposes Human-VDM, a novel Gaussian Splatting framework for generating 3D humans from a single image using video diffusion models. Human-VDM is comprised of three distinct modules: a view-consistent human video diffusion module, a video augmentation module, and a 3D human Gaussian Splatting module. Human-VDM first generates a `view-consistent' human video, then enhances the quality of the frames through super-resolution and video frame interpolation, and finally employs 3D Gaussian Splatting (3DGS) <cit.> to effectively generate the 3D human model. 

Initially, we fine-tune SV3D <cit.>, a latent video diffusion model specifically designed for generating object videos, to enable it to generate view-consistent human videos. However, a direct application of video diffusion models <cit.> to the 3D human generation can result in geometric artifacts and blurry textures. Additionally, the generated video consists of only 21 frames at a low resolution of 576 × 576, which is insufficient for high-quality 3D human generation. To provide more view-consistent frames and realistic texture for 3D human generation, we carefully designed a video augmentation module that includes super-resolution and frame interpolation components. The generated human video is enhanced through this module by undergoing super-resolution and frame interpolation, which results in smooth, high-quality frames at a resolution of 1080 × 1080. Lastly, we introduce a 3D human Gaussian splatting module to generate realistic 3D human models. For this, we utilize SMPL <cit.> along with an optimizable feature tensor training strategy to optimize the parameters of the 3D Gaussians, thereby generating a high-quality 3D human from a single image. Figure <ref> and <ref> demonstrate that Human-VDM achieves state-of-the-art (SOTA) performance and generates realistic 3D humans from a single-view RGB image input. Our contributions can be summarized as follows:


  * We propose a novel single-view 3D human generation framework that leverages the human video diffusion model to produce view-consistent human frames.

  * We carefully designed a video augmentation model that consists of super-resolution and video frame interpolation to enhance the quality of the generated video.

  * We introduce an effective Gaussian Splatting framework for 3D human reconstruction with offset prediction.

  * Extensive experiments demonstrate that the proposed Human-VDM can generate realistic 3D humans from single-view images, outperforming state-of-the-art methods in both quality and effectiveness.






§ RELATED WORKS
 

3D Human Generation. PIFu <cit.> was among the first methods to introduce pixel-aligned features and neural fields <cit.> for reconstructing human figures from images by fitting parametric human shape models such as SMPL <cit.> and SCAPE <cit.>. PIFuHD <cit.> further enhanced this framework with high-resolution normal guidance. Subsequent methods improved upon this initial approach by integrating additional human body priors. For instance, PaMIR <cit.> and ICON <cit.> utilized skinned body models to guide the reconstruction process, while ARCH <cit.>, ARCH++ <cit.>, and CAR <cit.> extended this approach by mapping global coordinates into canonical coordinates, enabling reposing. PHOHRUM <cit.> and S3F <cit.> introduced techniques to disentangle shading and albedo, facilitating relighting. Concurrently, another set of methods replaced neural representations with traditional Poisson surface reconstruction <cit.>. Despite these advancements, such approaches have been primarily tailored to human bodies and often struggle with the complex topologies of loose clothing. To address this limitation, ECON <cit.> and 2K2K <cit.> integrated depth or normal estimation to enhance the reconstruction process. More recently, Ultraman <cit.> introduced a model to map texture  thereby optimizing the texture details thus helping to maintain the color consistency during the final reconstruction. SIFU <cit.> also proposed a novel approach that combined the 3D Consistent Texture Refinement pipeline with a side-view Decoupling Transformer.



3D Human Generation with Diffusion models. Diffusion models <cit.> trained on large image datasets have exhibited remarkable capabilities in generating 3D objects from text prompts. Earlier works, such as Fantasia3d <cit.> and Magic3d <cit.>, predominantly followed an optimization-based workflow where 3D representations, such as NeRF <cit.>, were updated through neural rendering <cit.>. Although a few studies, such as TeCH <cit.>, adapted this workflow for 3D human reconstruction, they struggled to achieve accurate appearance and geometric representations of the human body due to the inherent ambiguities in text prompt condition. Recently, SiTH <cit.> integrated a generative diffusion model to produce full-body texture and geometry, including unobserved regions, within the reconstruction workflow. However, these methods still face challenges in capturing detailed clothing. In this paper, we leverage a video diffusion model (VDM) to generate an orbital video for 3D human reconstruction.



§ HUMAN-VDM


Given a single RGB image I of a person, Human-VDM aims to generate its 3D human model (see Figure <ref>). Human-VDM comprises several key modules: (i) the Human Video Diffusion module, (ii) the Video augmentation module, which includes the super-resolution and frame interpolation sub-modules, and (iii) the Human Gaussian Splatting module. First, the Human Video Diffusion module generates view-consistent videos of the input image. This video is then processed by the Video Augmentation module, where super-resolution enhances the resolution to 1080×1080, while video frame interpolation (VFI) smoothens the video frames. Finally, the augmented video is fed into the Human Gaussian Splatting module to generate a high-fidelity 3D human model.



 §.§ Human Video Diffusion Module


To generate the video V̂, we input the front image of a human, denoted as I, into a latent video diffusion model which we fine-tuned for high-quality human video generation. We specifically use SV3D <cit.>, a latent video diffusion model designed for generating videos from a single image, capable of producing consistent multi-view images. However, since SV3D was originally designed for reconstructing general objects, its generated video quality for human body images is not satisfactory. Therefore, to enhance its capability for human video generation, we fine-tuned SV3D on Thuman 2.0 <cit.> dataset which includes a variety of high-quality human body scans. SV3D produces a raw orbital video, V̂ = [f̂_1, f̂_2, f̂_3, …, f̂_21], with a resolution of 576 × 576, illustrating the human from different viewpoints. The videos generated by the fine-tuned SV3D exhibit superior shape, appearance, and detailed rendering of areas not directly captured in a 2D image. We represent this generation process as follows:

    V̂ = SV3D(I),

where `SV3D' denotes the generative process of the fine-tuned SV3D model.






 §.§ Video Augmentation Module


The 21-frame human video V̂, with a resolution of 576 × 576, has limited expressive capacity for detailed 3D human reconstruction. To address this, we introduce the Video Augmentation Module, which includes super-resolution and frame interpolation. Super-resolution helps in improving the quality of textures while video frame interpolation improves the geometric smoothness of the 3D human and the quality of the previously invisible areas.



Video Super-resolution sub-module. For image super-resolution on each frame of V̂, we employ CodeFormer <cit.>, a transformer-based model designed primarily for enhancing facial image resolution. CodeFormer performs Low Quality (LQ) to High Quality (HQ) mapping by first learning a discrete codebook and an HQ decoder D_H through self-reconstruction learning. During Codebook Lookup, a transformer and an LQ encoder E_L are additionally introduced to accurately model the cookbook code combination. For facial images, increasing the resolution of each frame of V̂ by 4× and then resizing it to 1080×1080 yields clear and realistic images that significantly benefit 3D reconstruction. Similarly, we increase the resolution of each frame in the raw orbital video V̂ by 4× and resize it to 1080×1080, resulting in a high-resolution video V^'=[f^'_1,f^'_2,...,f^'_21] with improved texture quality. This process is formulated as follows:

    f^'_i = Resize(SuperResolution (f̂_i)),   1≤ i≤ 21,

where `SuperResolution' denotes the operation of CodeFormer, while `Resize' denotes the operation of resizing the image to 1080×1080.



Video Frame Interpolation (VFI) sub-module. To enhance video consistency and interpolate frames, we employ PerVFI <cit.>. VFI provides additional visual information from diverse angles, improving the geometric smoothness of the 3D human and the quality of the invisible areas. PerVFI performs perception-oriented VFI and inputs two reference frame images I_0 and I_1 to reconstruct intermediate frames. First, bidirectional optical flows, i.e., F_0→1 and F_1→0 are estimated using a motion estimator. Additionally, two encoders capture multi-scale features. These features are then blended using asymmetric synergistic blending to obtain intermediate features f_t. These features are finally decoded to obtain the intermediate frame using a conditional flow generator, which samples from a normal distribution. We input the 21-frame high-resolution video frames V^' into PerVFI, resulting in an 81-frame high-resolution augmented video V=[f_1,f_2, ..., f_81]. This is formulated as follows:

    f = VFI(f^'_j),   1≤ j≤ 81,

where `VFI' denotes the frame interpolation operation.





 §.§ 3D Human Gaussian Splatting Module

We leverage 3D Gaussian Splatting <cit.> to model the 3D human from the augmented human video V. 3D Gaussian Splatting employs point-based representation, which facilitates high-quality real-time rendering by modeling the 3D object as a collection of parameterized static 3D Gaussians. Each Gaussian is characterized by a color c ∈ℝ^3, a 3D center position x ∈ℝ^3, opacity α∈ℝ, a 3D scaling factor s ∈ℝ^3, and a 3D rotation q ∈ℝ^4.

In this module, we incorporate an appearance network in conjunction with an optimizable feature tensor to enhance the representation of 3D Gaussian models refined from video data <cit.>. For each i^th frame f_i in the augmented video V, we first extract the SMPL model of the human body. We then sample points on the surface of this model and map their positions onto a UV position map, denoted by m. We introduce an optimizable feature tensor to capture the appearance of the reconstructed human. The parameters for each Gaussian are predicted by a Gaussian parameter decoder using the optimizable feature concatenated with m as input. These predictions form the 3D Gaussians in the canonical space. Using Linear Blend Skinning (LBS), these canonical 3D Gaussians can be reposed into motion space for rendering. This is formulated as follows:

    m    = M(θ̃,β) 
    
        P    = Decode(cat(t,m)),
    
        f_i^r    = Splatting(LBS(D,J(β),θ̂_i),P),

where θ̃ is the pose parameters of the SMPL model in canonical space and β is the average shape parameters calculated from V, respectively. M is the operation of mapping the positions of the sampled points on the surface of the SMPL model onto a UV map; t denotes the optimizable feature tensor, Decode means the process of decoding the aligned feature tensors to predict the parameters of Gaussians P. D = T(β) + dT denotes the locations of 3D Gaussians in canonical space, formed by adding corrective point displacements dT on the template mesh surface T(β), J(β) produces 3D joint locations, θ̂_i represents the refined pose parameter optimized from θ_i, which denotes the pose parameters obtained from f_i, `LBS' is the operation of Linear Blend Skinning; `Splatting' denotes the render process, resulting in a rendered image f_i^r. 








Training Objectives. For formulating the loss function, we take the current frame image f_i as the ground truth and calculate the loss with the rendered image f_i^r for optimization. This is formulated as follows:

    ℒ   = λ_RGBℒ_RGB + 
            λ_SSIMℒ_SSIM + λ_LPIPSℒ_LPIPS
        +λ_Offsetℒ_Offset
            + λ_Scaleℒ_Scale
            + λ_fℒ_f,
 
where ℒ_RGB is the L1-loss between the ground truth and the rendered frame. ℒ_SSIM and ℒ_LPIPS denotes the SSIM and LPIPS losses, respectively. ℒ_Offset, ℒ_Scale and ℒ_f calculate the L2-norm of predicted offsets and scales, and the feature map, respectively. The weight coefficients λ_RGB, λ_SSIM, λ_LPIPS, λ_Offset, λ_Scale and λ_f, are set to 0.8, 0.2, 0.2, 10, 1.0 and 1.0 respectively.




§ EXPERIMENTS AND RESULTS



Dataset. Most works use the popular Thuman 2.0 dataset <cit.>, which comprises 2,500 high-quality human body scans, each accompanied by a detailed 3D model and texture mapping. The dataset includes a wide range of action poses and provides the SMPL-X <cit.> parameters along with corresponding grids.

Evaluation Metrics. Following previous works on 3D human generation, we use the four major metrics to evaluate the performance of Human-VDM. These include CLIP-Similarity <cit.>, LPIPS (Learned Perceptual Image Patch Similarity) <cit.>, SSIM <cit.> and PSNR. CLIP <cit.> measures the similarity between two images, providing a more representative evaluation of image feature similarity. LPIPS <cit.>, measures differences based on learned perceptual image patch similarity, aligning more closely with human perception. Likewise, SSIM (Structural Similarity Index) <cit.> is used to compare the luminance, contrast, and structure between two images. Lastly, PSNR (Peak Signal-to-Noise Ratio) assesses image quality based on pixel-level error, making it an error-sensitive evaluation metric.

Training details. To produce high-quality human videos, we fine-tuned SV3D using the Thuman 2.0 dataset <cit.> to enhance its 3D human video generation capabilities. We selected 475 samples from Thuman 2.0, excluding those used in subsequent quantitative comparisons. For each sample, 21 images were rendered from various angles following <cit.>. All images corresponding to a sample are rendered at the same horizontal position with a constant angular interval of 360/21 degree to ensure the consistency of rendered multi-view images. The first rendered image of each body was employed as the input, while the remaining images served as ground truth for fine-tuning SV3D. We freeze the image encoder and decoder of the original SV3D <cit.> model and optimize the U-Net weights <cit.>. The learning rate was set to  and fine-tuned on one NVIDIA A800 GPU with a batch size of 13.



 §.§ Qualitative Comparison
 
Figure <ref> presents the qualitative 3D human generation results from Human-VDM on a variety of input images that differ in gender, body posture, lighting, color, and clothing styles. The results demonstrate Human-VDM's significant performance with high appearance consistency, texture, and geometry qualities. Next, we compare Human-VDM with recent SOTA works on single-image based 3D human generation (see Figure <ref>), including PIFu <cit.>, PaMIR <cit.>, TeCH <cit.>, Ultraman <cit.>, SiTH <cit.> and SIFU <cit.>. Compared to Human-VDM, PaMIR <cit.> exhibits significant shortcomings in the geometry of the generated 3D human, e.g., the body of the generated human is incomplete for the first image. On the other hand, TeCH <cit.>, PIFu <cit.>, and SiTH <cit.> reconstruct remarkable geometries but contain apparent artifacts. Likewise, SIFU <cit.> displays misalignment in character motion and suboptimal texture quality on the back of the generated human. While Ultraman <cit.> obtains good geometry but fails to predict the realistic appearance of unseen view. Therefore, the proposed Human-VDM outperforms SOTA models in terms of texture quality and appearance consistency.



 §.§ Quantitative Comparison
 

Following previous methods <cit.>, we randomly selected 50 samples from Thuman 2.0 <cit.>. Four views of the ground truth (GT), i.e., front, back, left, and right, were used to compute scores between the reconstructed results and the GT across these views. As reported in Table <ref>, Human-VDM achieves the lowest LPIPS and the highest CLIP score, indicating that the rendered images produced by our method are highly consistent with the input images. Additionally, Human-VDM achieves the highest SSIM and PSNR scores, further demonstrating that the rendered images of the generated 3D human are most closely aligned with the ground truth. All reported scores demonstrate the superiority of the proposed Human-VDM over existing SOTA methods.



 §.§ User Study
 

The discussed metrics may not always fully capture the quality of generated 3D humans in terms of realism and other details. Thus following previous works, a user preference study was conducted to evaluate the performance of Human-VDM against existing SOTA methods. We compare Human-VDM with six recent SOTA models using 10 different samples, each with four views of generated 3D humans in different samples. For each sample, 30 volunteers were asked to vote on their impressions regarding four key aspects: geometry quality, texture quality, face quality, and overall quality. For a fair comparison, the results for the other six SOTA models were generated using their official code, with all settings left at their default values. As shown in Table <ref>, the proposed Human-VDM surpasses SOTA models in the aforementioned aspects. 

Most volunteers considered Human-VDM to generate the best results, especially in terms of geometry and texture. Though Human-VDM does not particularly dominate in face quality relatively, it performs the best face consistency with the input image as shown in Figure <ref>. More than 53% of the volunteers confirm that Human-VDM outperforms other SOTA models, which confirms Human-VDM's superiority.



 §.§ Ablation Study
 
We performed ablation studies by systematically excluding various components to assess the effectiveness of the proposed modules through both quantitative and qualitative comparisons. For this analysis, we randomly selected 30 samples from the Thuman 2.0 dataset <cit.>. We compared the full model with the variants excluding the proposed modules using the CLIP Similarity <cit.>, SSIM <cit.>, LPIPS <cit.>, and PSNR metrics. The evaluation covered rendered results from four viewpoints: front, back, left, and right. We additionally report results solely for the front view as well. Table <ref> presents the quantitative comparisons, while the qualitative visual comparisons are illustrated in Figure <ref>.

Quantitative results demonstrate that the proposed full model achieves superior CLIP Similarity and SSIM across both the single view and four views. The visual ablation results further establish that the 3D human generated by the full model exhibits more photorealistic textures and precise geometry. Results produced without finetuned SV3D are less lifelike and realistic since the videos generated by the original SV3D are not satisfactory. Without Super-Resolution, the video frames are not distinct enough for the Human Gaussian Splatting module, which results in blurs and artifacts of the reconstructed humans. Due to the lack of features presented by only 21 frames, results generated without frame interpolation are not good enough yet, which has apparent artifacts in novel views. This confirms the significance and contribution of the video augmentation module. In general, the finetuned SV3D provides high-quality human orbital video for realistic reconstruction; the super-resolution module enhances the quality of video frames to generate more distinct results, and the VFI module enables the model to generate remarkable results in novel views. Although the full model shows a slight decrease in LPIPS and PSNR, the visual results indicate that the 3D human reconstructed by the complete model is of higher quality. Overall, the full model achieves better performance i.e., when including the proposed components. This confirms the effectiveness of the proposed modules.





§ CONCLUSION AND FUTURE WORK


We propose a novel 3DGS-based framework for generating 3D humans from a single RGB image leveraging human video diffusion models. We first generate a view-consistent orbital video around the human and then augment the video through super-resolution and video frame interpolation. Finally, we reconstruct a remarkable 3D human using 3D Gaussian with the enhanced video. Both quantitative and qualitative experiments demonstrate that Human-VDM excels in generating 3D humans from a single image, outperforming state-of-the-art methods.



Limitations and Future works. Human-VDM has two limitations. First, it is challenging to accurately generate precise finger geometry due to the intricate and small size of finger poses. Second, applying large video diffusion models limits the model's overall ability to achieve a real-time 3D human generation. Future works can focus on addressing these limitations by enhancing geometry generation for complex and small finger poses, as well as developing more efficient models that can achieve real-time 3D human generation.





[
Supplementary Material


]

In the supplementary material, we provide a more detailed explanation of the model architecture, as well as training specifics, such as loss function weights, dataset descriptions, and definitions of the evaluation metrics. Additionally, we include further visual results and an analysis of failure cases.



§ MODEL ARCHITECTURE DETAILS




 §.§ Human Video Diffusion Module


Module Architecture. The Video Diffusion Module of Human-VDM is based on  SV3D <cit.>. SV3D's architecture builds upon SVD <cit.> and consists of a UNet <cit.> model with multiple layers. Each layer comprises a sequence of 1 residual block with Conv3D layers, followed by spatial and temporal transformer blocks integrated with attention layers. After being embedded into the latent space via the visual autoencoder (VAE) of SVD, the conditioning image is concatenated with the noisy latent state input z_t at noise timestep t before being fed into the UNet. The CLIP-embedding <cit.> matrix of the input image is provided to the cross-attention layers of each transformer block <cit.>, serving as the key and value, with the layer's feature acting as the query. Along with the diffusion noise timestep, the camera trajectory is also incorporated into the residual blocks. The camera pose angles e_i and a_i are first embedded into the position embeddings. These camera pose embeddings are then concatenated, linearly transformed, and combined with the noise timestep embedding. The composite embedding is fed into every residual block, where it is added to the block’s output after another linear transformation to match the feature size.



Static Orbits. The original SV3D model <cit.> consists of two main orbits: (1) the static orbit and (2) the dynamic orbit. Our study utilizes the static orbit, where the camera moves around the object at evenly spaced azimuth angles while maintaining the same elevation angle as in the conditioning image.



Fine-tuning SV3D for Human Video Diffusion. The original SV3D is fine-tuned upon SVD-xt <cit.> on the Objaverse dataset <cit.>, which contains synthetic 3D objects covering a wide diversity. For each object,  <cit.> renders 21 frames around it on a random color background at 576×576 resolution, field-of-view of 33.8 degrees. We adopt the same rendering strategy for the Thuman 2.0 dataset <cit.> to fine-tune SV3D for high-quality human video generation.



 §.§ Video Augmentation Module


Video Super-Resolution sub-module. 
CodeFormer <cit.> is a transformer-based model <cit.> to enhance the resolution of human images. Upon learning a discrete codebook, an encoder E_H embed the high-quality human image I_h ∈ℝ^H× W× 3 as a compressed feature Z_h ∈ℝ^m× n× d by an encoder E_H. Each “pixel” in Z_h is then replaced by the nearest entry in the learnable codebook 𝒞 = c_k ∈ℝ^d^N_k=0. Afterward, the quantized feature Z_c ∈ℝ^m× n× d along with the code token sequence s ∈0, ⋯, N-1^m· n are produced as the following:

    Z_c^(i,j)   =min_c_k∈𝒞Z_h^(i,j)-c_k_2,
       s^(i,j)   =min_kZ_h^(i,j)-c_k_2.

Given Z_c, the high-quality human image I_rec is reconstructed by the decoder D_H. The m × n code token sequence, denoted as s, constitutes a novel latent discrete representation, which encodes the specific indices corresponding to entries in the learned codebook, i.e., Z^(i,j)_c = c_k when s^(i,j) = k.

Subsequently, with the codebook ℛ and decoder D_H held constant, a Transformer module <cit.> is introduced for predicting the code sequence, capturing the global human composition from low-quality inputs. To extract the low-quality features Z_l ∈ℝ^m × n × d using E_L, the features are first unfolded to m · n vectors Z_l^v ∈ℝ^(m · n) × d, which are subsequently fed into the Transformer. In the transformer, the s^th self-attention block performs the below operation:

    X_s+1=Softmax(Q_sK_s)V_s+X_s,

where X_0 = Z^v_l. X_s is used to get the queries Q, key K, and value V through linear layers.



Video Frame Interpolation (VFI) sub-module. PerVFI is a novel model of frame interpolation. Given two reference frame images, I_0 and I_1 ∈ℝ^H× W× 3, with height H and width W, PerVFI is designed for reconstructing the intermediate frame I_t within the target time t ∈ (0, 1). It incorporates an asymmetric synergistic blending (ASB) module and a conditional normalizing flow-based generator.

After estimating bidirectional optical flows, PerVFI presents a pyramidal architecture, which can better capture multiscale information to extract features at different scales.
Specifically, a feature encoder E_θ is used to encode the two images into pyramid features with L levels, which can be denoted as f_i=E_θ(I_i), i=0,1. Subsequently, a feature blending module, denoted as B_θ, blends the pyramidal features to produce intermediate pyramid features. Afterward, a conditional normalizing flow-based generator G_ϕ, which is invertible, decodes f_t into the output frame I_t. The output is formulated as I_t=G^-1_ϕ(r;f_t), where r ∼𝒩(0,τ) ∈ℝ^H × W × 3 represents a variable drawn from a normal distribution with a temperature parameter τ; f_t is the feature pyramid with L levels.









 §.§ 3D Human Gaussian Splatting Module


In 3D Gaussian, human appearances are determined by point displacements dT and properties P. Modeling dynamic human appearances involves estimating these evolving properties. We propose a dynamic appearance network coupled with an optimizable feature tensor to effectively capture dynamic human appearances across various poses. The dynamic appearance network is designed to learn a mapping from a 2D manifold representing the underlying human shape to the dynamic properties of 3D Gaussians as follows:

    f_ϕ:𝒮^2∈ℝ^3→ℝ^7,

the 2D human manifold 𝒮^2 is depicted by a UV positional map I ∈ℝ^H× W ×3, where each valid pixel stores the position (x, y, z) of one point on the posed body surface. The final predictions consist of per point offset Δ𝐱̂∈ℝ^3, color 𝐜̂∈ℝ^3, and scale ŝ∈ℝ.

Human poses θ and translations t estimated from monocular videos are usually inaccurate. Hence, the 3D Gaussians reposed in motion space may be inaccurately represented, potentially resulting in unsatisfactory rendering outcomes. To address this issue, we jointly optimize human motions and appearances. We update the estimated body poses and translations by calculating (Δθ,Δ𝐭) to refine human motions, which can be formulated as follows:

    Θ̂=(θ+Δθ,𝐭+Δ𝐭).

We modify θ in the equation of animatable Gaussians in the main article using Θ̂ to render the proposed animatable 3D Gaussians differentiable with respect to the motion conditions. Finally, the current frame image is taken as the ground truth to calculate the loss with the rendered image. 



 §.§ Training Objectives

We use the current frame image, i.e., f_i, and the rendered image, i.e., f_i^r, for supervising the Human-VDM model. The total loss consists of six different loss functions which include ℒ_RGB, ℒ_SSIM, ℒ_LPIPS, ℒ_Offset, ℒ_Scale and ℒ_f. In this section, we describe the loss functions in greater detail.


ℒ_RGB is the L1-loss between the ground truth and the rendered frame and is formulated as:

    ℒ_RGB(x,y)=1/HW∑_h,w^HW|y_hw-x_hw|,


ℒ_SSIM <cit.>, or the Structural Similarity Index Metric Loss is a perceptual metric to measure the similarity between two images, taking luminance, contrast, and structure into account. We define the SSIM loss as follows:


    ℒ_SSIM(x,y)    = 1 - SSIM(x,y) 
       =1 - (2μ_xμ_y+c_1)(2σ_xy+c_2)/(μ_x^2+μ_y^2+c_1)(σ_x^2+σ_y^2+c_2),

where μ_x and μ_y stands for the mean of x and y; σ_x and σ_y represent the variance of x and y, while σ_xy denote the covariance of x and y.

ℒ_LPIPS <cit.> measures image similarity, which evaluates the perceptual difference between two images through deep learning models. In this paper, we utilize AlexNet <cit.> for extracting features of images. We calculate ℒ_LPIPS as:

    ℒ_LPIPS(x,y)=∑_l1/H_lW_l∑_h,w||w_l⊙(f̂_xhw^l-f̂_yhw^l)||_2^2,

where f̂_xhw^l represents the feature output of image x in layer l at the pixel hw, and f̂_yhw^l means the same of image y. w_l is a trainable parameter in layer l.

ℒ_Offset, ℒ_Scale and ℒ_f  calculate the L2-norm of the feature map, predicted offsets and scales on the canonical surface, respectively. We formulate them as follows:

    ℒ_Offset=1/N∑_i=1^N(Δx̂_̂î)^2,

where Δ x_i denote the predicted offset of i^th gaussian.

    ℒ_Scale=1/N∑_i=1^N(ŝ_̂î)^2,

where s_i denotes the predicted scale of i^th gaussian.

    ℒ_f=1/F∑_i=1^F(t_i)^2,

where t_i denotes the optimized feature.



§ IMPLEMENTATION DETAILS


In this section, we present additional details on the model implementation. The Gaussian decoder is implemented as an MLP. A total of 202,738 Gaussians were initially sampled on the surface of the canonical SMPL model. The adjustable coefficient w, which presents the reliance on input low-quality image, is set to 0.7 in the Super-Resolution module. For each sample, we train the dynamic appearance network on a single NVIDIA RTX 3090 GPU for 1000 epochs with a batch size of 2. The learning rate of the network is set to .



§ ADDITIONAL RESULTS


In this section, we present additional results, including in-the-wild testing and failure cases.



 §.§ In-the-wild visual results


To demonstrate the superiority of Human-VDM, we provide more visual comparison results. This includes additional results as shown in Figure <ref>, including results on challenging in-the-wild cases illustrated in Figure <ref>.



 §.§ Failure Cases

In this subsection, we present several cases of failure in Human-VDM.
Although Human-VDM performs exceptionally well in generating 3D humans from a single RGB image, it still has a few limitations and failure cases, as discussed in the main text. Figure <ref> shows the failure cases of Human-VDM. For example, when the human in the input image interacts with their hands against their body, some artifacts may appear at the contact region.

