Industrial manipulators do not collapse under their own weight when powered off due to the friction in their joints. Although these mechanism are effective for stiff position control of pick-and-place, they are inappropriate for legged robots that must rapidly regulate compliant interactions with the environment. However, no metric exists to quantify the robot's performance degradation due to mechanical losses in the actuators and transmissions.
This paper provides a fundamental formulation that uses the mechanical efficiency of transmissions to quantify the effect of power losses in the mechanical transmissions on the dynamics of a whole robotic system.

We quantitatively demonstrate the intuitive fact that the apparent inertia of the robots increase in the presence of joint friction. We also show that robots that employ high gear ratio and low efficiency transmissions can statically sustain more substantial external loads. We expect that the framework presented here will provide the fundamental tools for designing the next generation of legged robots that can effectively interact with the world.

§ INTRODUCTION

The mechanical losses in the transmissions of individual joints govern the system-level dynamics of robots. For instance, conventional industrial manipulators behave like statues when they are powered off; in other words, they are non-backdrivable. The characteristics that determine this behavior are the low mechanical efficiency and high friction in the gearboxes. Although these mechanical transmissions have been successfully utilized in industrial manipulators for stiff position control, they are not appropriate for compliant force control due to their large mechanical impedance <cit.>. Therefore, to enable legged robots to control their contact forces with the environment, one must analyze how the power efficiency of the mechanical transmissions utilized in robots governs the overall dynamics. Towards this goal, this paper introduces a framework for studying how the energetic losses at joint-level propagate to the dynamic behavior of the robot at system-level.

No existing design metric can quantitatively describe the performance degradation of a robotic system due to the mechanical losses in actuators or transmissions. The absence of such metric hinders the selection of an optimal mechanical transmission for legged robots. For instance, no existing design guideline provides a clear choice between a compact and low efficiency strain wave gearbox and a bulkier, but higher efficiency, planetary gearbox used in many legged robots <cit.>.

This unanswered question arose in the design of  the humanoid robot TELLO, shown in Fig. <ref> and in the video <cit.>, and motivated this study.

Related work in the literature analysed the fundamental behavior of mechanical transmission and their impact on actuator backdrivability.

Giberti provides the optimal choice of an actuator-reducer pair of one-degree-of-freedom (DoF) system, considering joint level efficiency <cit.>.
Similarly, Wang demonstrates that gearboxes present directional efficiency, which means that the mechanical losses are different if actuators operates within positive or negative work regimes <cit.>. Wensing investigates how joint-level apparent inertia decreases backdrivability, and thus, degrades the impact mitigation capability of the whole system <cit.>. Singh studies how the mass distribution of a robot's leg influences the propagation of impact from the ground to the torso <cit.>. Kim demonstrated a light-weighted robot arm of high backdrivability with a novel tension-amplification transmission <cit.>. However, the influence of the mechanical efficiency of transmissions on the whole system still remains unstudied.

Energy flow diagram of a robotic system showing the dissipation of energy in  actuators and transmissions. The energy conversions are always accompanied by energy losses such as Joule heating or friction.

(a) A forward-driving scenario of a wedge-block model analogous to a geared transmission. (b) A backward-driving scenario (c)  The forward- and backward-driving scenarios of a rotor-manipulator model that represents a typical geared transmission (d) A dissipative force $f_d$ associated with the slip $\Delta v$ at the contact point $C$ of a typical geared transmission <cit.>

The main contribution of this paper is to provide a fundamental formulation to quantify the effect of power losses in the mechanical transmissions on the dynamics of the whole robotic system. As a product of this formulation, we propose an augmented equation of motion that embeds the mechanical efficiency in the system's inertia matrix and the vectors of bias force, the sum of Coriolis and gravitational force, and generalized forces. The system-level impact of the individual efficiencies is demonstrated by computing the (task-space) generalized inertia ellipsoid <cit.> and force capability of the robot's end-effector. Two core results are obtained from this study. First, the perceived inertia of the robot at the end-effector increases as the efficiency decreases. Second, the capability of the robot to generate or resist force depends on the direction of the energy flow in the system.

This paper is organized as follows. First, we study a simple representative example to understand how transmission efficiency affects the dynamics of a one-DoF system in forward-driving (FWD) and backward-driving (BWD) scenarios. Next, the kinematic structure and constraints of rigid body system are highlighted to trace the energy loss inherent to the mechanical transmission. In Section IV, the generalized dynamics of a rigid-body system with mechanical power losses in kinematic structure  is obtained. The proposed equation of motion enables the re-derivation of a conventional design criteria, such as the generalized inertia tensor and the force capability. Finally, a case study of a 2-DoF leg qualitatively validates the proposed formulation.

§ A SIMPLE MODEL OF DISSIPATIVE DYNAMICS

This section investigates how the power efficiency of a mechanical transmission is embedded into the equation of motion using a simple wedge-block model shown in Fig. <ref>(a) and (b). This model extracts the essential behavior of the complex sliding and pushing dynamics of the gear teeth meshing in the rotor-manipulator model in Fig. <ref>(c) and (d). The wedge-block model is designed to inherit mechanical characteristics of rotor-manipulator model; (i) frictional loss is dictated by the geometry of meshing between two bodies, and (ii) the dynamics is different depending on the direction of energy flow.
This flow falls into two categories:

* FWD (Fig. <ref>(a)) occurs when pushing the wedge $m$ with a force $f_u$ and, consequently, moving the block $M$. This is equivalent to commanding a motor torque $\tau_\theta$ to drive the link of a manipulator.

* BWD (Fig. <ref>(b)) occurs when energy flows in the opposite direction by pushing the block $M$ with a force $f_x$ to drive the wedge $m$. This is equivalent to an external force applied to the manipulator's end-effector to backdrive the rotor through the mechanical transmission.

The analogy between the wedge-block and rotor-manipulator models is described here. First, the displacements of the block $x$ and the wedge $u$ are coupled by the slope with incline $\alpha$. In this case, $\tfrac{1}{\cos\alpha}$ is the mechanical advantage that increases with the slope angle. This is expressed by the constraint

\begin{align*}
g(x, u) & = -x +({\cos\alpha}){u}  \quad (\alpha \in [0, \tfrac{\pi}{2}))
\end{align*}

Similarly, the angular displacements of the rotor $\phi$ and the manipulator joint $\psi$ are kinematically coupled by gear ratio ${N}$ which is the ratio of input velocity to output velocity.

\begin{align*}
g(\psi, \phi) &= -\psi + \tfrac{1}{N}\phi  \quad (N \ge 1 ).
\end{align*}

Second, we study the dynamics of different energy flow scenarios, where the normal force $\lambda$ and sliding friction $f_d$ maintain their orientation and magnitude. However, the sign of the friction flips according to the movement of wedge, which contributes to the asymmetric dynamic behavior in FWD and BWD cases. As a consequence, the example renders different mechanical efficiencies in the FWD and BWD scenarios, similarly to geared transmissions <cit.>. The power loss inherent to the gear meshing mechanics largely contributes to the asymmetry of the dynamics. This means that the power loss, mechanical efficiency, apparent inertia, and input power distribution are different if the motors are driving the manipulator or if an external force is back-driving the actuators. The rest of this section derives and discuss the dynamics of forward and back-driving cases of the wedge-block model in Fig. <ref>.

§.§ Meshing forces

Let the redundant coordinates be $\bm s = [x \quad u]^\top$. The dynamic equations of motion of the wedge-block model in Fig. <ref>(a) and (b) follows

\begin{gather*}
\bM H_s \ddot{ \bm s} - \bM A^\top \lambda = \bm f + \bm f_d,
\end{gather*}

from standard Lagrangian formulation, with

\begin{gather*}
\bM H_s \!=\! \bmat{M \!&\! 0 \\ 0 \!&\! m}\!\!,\,  \bM A^\top \!= \!\bmat{-1 \\ \cos\alpha}\!\!,\,  \bm f \!=\! \bmat{f_x \\ f_u}\!\!,\, \bm f_d \!=\! \bmat{0 \\ \pm \mu \sin\alpha \lambda}\!\!,
\end{gather*}

where $\bm f_d$ is the dissipative force and $\bM A =\pdif{g}{\bm s}$ is the constraint Jacobian that represents the mechanical advantage that distributes the constraint force. We introduce the concept of meshing forces $\bm r$, which are the sum of dissipative forces $\bm f_d$ and the constraint forces $\bM A^\top \lambda$.

\begin{gather*}
\bm r \coloneqq \left(\bM A^\top \lambda + \bm{f}_d \right) = \bM H_s \ddot{\bm s} - \bm{f }.
\end{gather*}

This rearrangement groups the contact forces that transmits power between bodies. Writing the $x$ and $u$ components of the meshing force $\bm r$, it becomes clear that the dry friction $\pm \mu\sin\alpha\lambda$ contributes to the asymmetricity of the dynamics:

\begin{align}
\bmat{r_x \\ r_u} = \bmat{-1 \\ \cos \alpha (1\pm \mu \sin \alpha )}\lambda = \bmat{M\ddot x - f_x \\ m \ddot u -f_u}.
\label{eqn:exDyn}
\end{align}

With (<ref>), the mechanical efficiency $\eta$, the ratio of output power $(\hat{x}) $ to input power $(\hat{u})$, is described,

\begin{gather}
\eta = - \frac{ r_x dx}{ r_u du} = \begin{cases} \tfrac{1}{\eta_b} = 1+\mu \tan \alpha & (\textrm{BWD}),\\ \eta_f = {1-\mu \tan \alpha }  & (\textrm{FWD}).\end{cases} \label{effdef}
\end{gather}

This result aligns with the standard description of the bidirectional efficiency of geared transmissions that is a function of the friction coefficient, contact geometry, and the direction of energy flow <cit.>.

§.§ Model Reduction and Efficiency-Null

First, since coordinates $x$ and $u$ in (<ref>) are related by the constraint $g$, the redundant coordinates $\bm s$ are projected onto a minimal coordinate $x$ using the constraint nullspace matrix $\bM K=[1\quad \sec\alpha]^\top$ such that $\bM{AK}=0$,

\begin{gather*}
d\bm s = \bM K dx.
\end{gather*}

Next, the Lagrangian multiplier $\lambda$ is cancelled out from (<ref>).
Conservative formulations assume that constraints do not dissipate power because the constraint force and its tangent motion are orthogonal  <cit.>, <cit.>. For dissipative systems, we find an alternative nullity by rearranging (<ref>) and using an efficiency matrix $\bM E_r \!\coloneqq\! \diag{(1, \eta)}$,

\begin{gather}
r_x dx + \eta r_u du = \bmat{dx\\ du}^{\!\top\!\!} \!\underbrace{\bmat{1 \!&\! 0 \\0 \!&\! \eta}}_{= \bM E_r} \!\bmat{r_x \\ r_u} = dx \bM K^\top \bM E_r \bm r = 0.  \label{dZ}
\end{gather}

We formalize this nullity as efficiency-null ($\delta Z \!=\! 0$) which defines the orthogonality between the meshing force and its tangent motion,

\begin{gather}
\delta Z \coloneqq  \bM K^\top \bM E_r \bm r = 0. \label{orth}
\end{gather}

§.§ Asymmetric Dynamics and Mechanical Impedance

Finally, by multiplying $\bM K^\top \bM E_r $ on both sides of (<ref>) and applying $\ddot{\bm s}=\bM K \ddot x $, the meshing force or the Lagrangian multiplier is cancelled out. As a result, we obtain the dynamics of an 1-DoF model:

\begin{gather*}
\bM K^\top \bM E_r \bm r = \bM K^\top \bM E_r ( \bM H_s \bM K \ddot x - \bm f ) = 0,
\end{gather*}

\begin{align}
\textrm{FWD:}& \:\: \left(M + \eta_f \frac{m}{\cos^2 \alpha}\vphantom{\frac{1}{\eta_b}}\right)\ddot{x} = f_x-\eta_f\hat{f}_u \label{eqn:exDynFwdFinal},
\\
\textrm{BWD:}& \:\:\left( M + \frac{1}{\eta_b} \frac{m}{\cos^2 \alpha}\right)\ddot{x} = f_x - \frac{1}{\eta_b}\hat{f}_u, \label{eqn:exDynBwdFinal}
\end{align}

where $\hat{f}_u$ is the force on $u-$direction, $f_u$, projected onto the $\hat{x}$ coordinate frame, $\hat{f}_u = \tfrac{f_u}{\cos\alpha}$. We utilize the Laplace transform to obtain the mechanical impedance $\bM X(s)$ in the frequency domain $s$. We assume that $\hat{f}_u$ is the only force applied in the FWD case and $f_x$ is the only force exerted in the BWD case.

FWD:     X_f (s)=f̂_u (s)/ẋ (s)    =    ( 1/η_fM + m/cos^2 α)s,

BWD:     X_b (s)=f_x (s)/ẋ (s)     =    ( M + 1/η_b m/cos^2 α)s.

§.§ Discussion

The results from the wedge-block model highlights the fundamental asymmetry of dissipative dynamics; FWD and BWD dynamics are differently affected by friction $\mu$ and gear ratio $\tfrac{1}{\cos\alpha}$. The unique properties of the dissipative dynamics of the wedge-block model are summarized as follows:

* Efficiency: The backward efficiency $\eta_b$ is always smaller than forward efficiency $\eta_f$. And both are negatively affected by larger gear ratio and friction.

* Non-backdrivability: There is a limiting case where the system becomes non-backdrivable, $\mu \tan\alpha\!=\!1$ ($\eta_b\!=\!0$).

* Apparent inertia: Both the large gear ratio and the low backward efficiency increases apparent inertia in the BWD case.

* Impedance: For both the FWD and BWD cases, the impedance of the system increases with the degradation of mechanical efficiency.

The results demonstrate that the overall BWD dynamics are more negatively affected by low efficiency. This occurs because, first, the apparent inertia of the wedge is proportional to the inverse of the backward efficiency. Second, the right hand side of (<ref>) illustrates that any unmodeled friction in the transmission, such as stiction, is amplified by the gear ratio and the inverse of the backward efficiency to resist external forces, which degrades the backdrivability of the mechanism. We note that these properties are also observed in the geared transmissions as reported in <cit.>. Extending the discussion to the rotor-manipulator model, the negative impact of low transmission efficiency is more likely to affect robots with light-weight links and high gear ratios.

§ GENERALIZATION OF THE DISSIPATIVE DYNAMICS

This section introduces mathematical terms and a graphical tool that are useful to generalize a multi-DoF robotic system with transmissions. First, the angles of rotor, motor, and joint are defined in addition to the mappings between them. Next, we conceptualize transmissions by formulating kinematic constraints that determine the mechanical connectivity between rigid bodies. Finally, we trace the frictional losses in the transmissions and discuss the topological propagation of power.

§.§ Kinematic Topology and Constraints

This section provides the mathematical definitions for two core concepts: the speed reduction and the actuation topology. Speed reduction is often realized by employing pairs of meshed gears or belt-driven pulleys with different diameters. The actuation topology describes the topology of the mechanism that distributes power from the motors to the joints. The actuation topology, or the kinematic structure <cit.>, is used to classify robots as, for example, serial or parallel mechanisms. We define these two concepts as transformations between coordinate angles. For that, three sequential states (angular positions) are introduced:

* Rotor angle $\bm \phi$ is the angular position of the motor rotor before a gearbox or a reduction mechanism;

* Motor angle $ \bm \psi$ is the angular position of the output of the actuator after a reduction mechanism; and

* Joint angle $ \bm q $ is the angle between the structural links of the robotic system.

The mappings between the states are defined as:

* Reduction $\bM R$ is the generalization of the speed reduction, which maps the displacement of the rotor angle $\bm \phi$ to the motor angle $ \bm \psi$; and

* Actuation topology $\bM D$ represents the kinematic topology of a transmission, which maps the motor angle $ \bm \psi$ to the joint angle $ \bm q $.

Fig. <ref>(a) and (b) illustrates an actuation topology of a typical 2-DoF parallelogram mechanism in which the speed reduction is represented by a single-stage gearbox. Similar mechanisms have been used in the legged robots ATRIAS <cit.> and Minitaur <cit.>. The two motors are fixed to the base and drive the links $\textrm{S1}$ and $\textrm{P1}$ of the parallelogram mechanism. As a consequence, the input power propagates to the end-effector via links $\textrm{P2}$ and $\textrm{S2}$.

Each motor has a gearbox that reduces the rotation of the rotor to motor angle $\psi_i\!=\!\tfrac{\phi_i}{N_i}$ for $i \!\in\! \{1,2\}$, where ${N_i}$ are gear ratios. The first joint is driven by the first actuator ($q_1\!=\!\psi_1$), while the second joint angle is controlled by both motors ($q_2\!=\!\psi_2\!-\!\psi_1$). Following our previous definitions, the reduction and actuation topology of this manipulator are given by:
R_par =
1N_1     0

0     1N_2
],    D_par =
1     0

-1     1

If the actuation topology represented a serial mechanism in which the motors are directly mounted on the joints, the matrix $\bM D_\textrm{ser}$ would be $2\times2$ identity matrix, while the reduction matrix would remain unchanged, $\bM R_\textrm{ser}\!=\!\bM R_\textrm{par}$.
Next, the kinematic constraints in actuators and transmissions, are represented with the reduction and actuation topology matrices:

g(q,  ϕ) = q - D R ϕ.

Finally, we include the task-space coordinate of the end-effector $\bm x$ and its Jacobian $\bM J$ to obtain the sequential transformations of generalized coordinates,
d ϕ       d ψ       d q        d x.

To simplify the discussion, it is assumed that the matrices $\bM R$ and $\bM D$ are dimensionless and invertible. Readers can refer to <cit.> for the analysis of nonlinear constraints.

The coordinate angles in a 2-DoF robotic arm are visualized.

(a) A single stage gearbox in an actuator (b) A 2-DoF robotic arm using parallelogram (c) The kinematic tree outlines the topology of a transmission and the kinematic constraints associated with power losses.

§.§ Rigid Body Systems and Kinematic Tree

We employ the kinematic tree of a rigid body system [1] to trace the energy losses associated with the kinematic constraints in a transmission mechanism. Mechanical connections between rigid bodies are categorized into two types: (i) free joints, which define the parent-child relations and (ii) dissipative couplings, which represent a kinematic constraint that bears power loss. For example, Fig. <ref>(c) is the kinematic tree of a parallelogram mechanism, showing energy transfer from rotors $\textrm{(R1, R2)}$ to structural bodies $\textrm{(S1, S2)}$. The dissipative couplings are represented by dashed arrows to clearly illustrate that the power losses depend on the direction of power delivery.
In order to avoid kinematic loops that would complicate the analysis of the dynamics, we assume that some components, such as $\textrm{P1}$ and $\textrm{P2}$, in a transmission are massless.

§ THE DYNAMICS OF DISSIPATIVE RIGID-BODY SYSTEMS

The dissipative dynamics of a general robotic system is obtained similarly to the solution for the simple example using the definitions from section <ref>. The goals are to (i) create a kinematic tree of rigid-bodies with redundant states, (ii) define kinematic couplings between rigid-bodies written as constraints, (iii) collect meshing forces given by dissipative forces and constraint forces, and (iv) reduce the model by projecting its dynamics onto the tangent space to eliminate the meshing forces.

The general model of a legged robot is comprised of a $b$-DoF floating-base (the torso) and a serial chain of $m$-links (the limbs) <cit.>. The limbs are driven by $m$-rotors through mechanical transmissions as defined by a reduction matrix $\bM R_m$ and an actuator topology matrix $\bM D_m$.
We define $\bm q_b\in \R^{b}$ as the $b$-DoF coordinates for the free floating-base, and $\bm \phi_m, \bm q_m\in \R^{m}$ as the rotor angles and joint angles. Next, the constraint in the transmission is denoted as

\begin{equation}
\bm g(\bm q_m, \bm \phi_m ) = \bm q_m -\bM D_m \bM R_m \bm \phi_m.    \label{eqn:genConstr}
\end{equation}

The generalized coordinates of the redundant model and the reduced-order model are $\bm s = [\bm q_b ^\top, \bm q_m ^\top, \bm \phi_m^\top ]^\top$ and $\bm q = [\bm q_b ^\top, \bm q_m ^\top ]^\top$. The constraint Jacobian $\bM A$ and its nullspace matrix $\bM K$ such that $d\bm s \!=\! \bM K d\bm q$ and that $\bM A\bM K \!=\! \bm 0 $ are written

\begin{align*}
\bM A = \mpdif{\bm{g}}{\bm s} = \bmat{\; \bm 0_{m\times b} & \bm1_{m}& -\bM{D}_m \bM{R}_m \;\;} \in \R^{m\times(b+2m)},
\\
% [4pt]
\bM K = \bmat{\;\;\bm 1_b & \bm 0 \\ \;\;\bm 0 & \bm 1_m \\ \;\; \bM 0 & (\bM{D}_m \bM{R}_m)^{-1}} \in \R^{(b+2m)\times (b+m)},
\end{align*}

where $\bm1_{n}$ is the $n\times n$ identity for any $n\in \mathbb{N}$. The equation of motion of the redundant system is given as
H_s s̈ +c_s -A^⊤λ_m = τ_s + τ_d,

where $\bM H_s$ and $\bm c_s$  are inertia matrix and bias force, $\bm \lambda_m \!\in\! \R^{m}$ are the Lagrange multipliers, $\bm \tau_s$ are the generalized forces, and $\bm \tau_d \!= \![0, \! \cdots \!, 0, \tau_{d,1}, \!\cdots \!, \tau_{d,m}]^\top \! \in \! \R^{b+2m}$ are generalized dissipative forces in the transmission.
The forces due to the constraints are combined as meshing forces

\begin{gather}
\bm r = \bM A^\top \bm \lambda_m + \bm \tau_d = \bM H_s \ddot{\bm s} + \bm c_s -\bm \tau_s.
\end{gather}

We assume that for all $m$-kinematic couplings of power efficiencies $\eta_1, \cdots, \eta_m$, their associated efficiency-nulls $\delta Z_1, \cdots, \delta Z_m$ are all equal to zero. By stacking efficiency-nulls, we obtain an orthogonality between the meshing force and its tangent motion at system-level,

\begin{gather}
\bM K^\top \bM E_s \bm r =\bM 0, \label{sysdZ}
\end{gather}

where $\bM E_s \!\coloneqq \!\diag{(1, \!\cdots\!, 1, \eta_1, \!\cdots\!, \eta_m)}\!\in\! \R^{(b+2m)\times(b+2m)}$.

The equation of the motion of the redundant system (<ref>) is reduced to $(b+m)$-dimensional system by left multiplying $\bM{K}^\top \bM{E}_s$ to the meshing force $\bm r$, and projecting $\ddot {\bm s}$ onto the reduced coordinates $\ddot {\bm q}$ with $\ddot{\bm s} = \bM K \ddot {\bm q}$. This procedure yields a dissipative equation of motion of a robotic system,

\begin{gather}
\bM K^\top \bM E_s \bM H_s \bM K\ddot{\bm q}  + \bM{K}^\top \bM E_s \bm c_s = \bM J^\top \bm f_{\textrm{ext}} + \bmat{\bm 0_{b\times m} \\ \bM B_m \bM E_m}\bm\tau_\phi,  \label{eqn:dissipativeDyn}
\end{gather}

where $\bM J \!\in\! \R^{n \times(b+m) }$ is the contact Jacobian of an external force at end-effector $\bm f_{\textrm{ext}}\!\!\in\! \R^n$ , $\bM B_m \!\coloneqq\! \pmat{\bM D_m \bM R_m}^{\!-\!\top}$ is a distribution matrix, $\bM E_m \!\coloneqq\! \diag{(\eta_1, \!\cdots\!, \eta_m)}$ is the efficiency matrix, and $\bm \tau_{\phi}\!\in \! \R^m$ are torques applied to the rotors. We assume that there are no forces applied to the floating-base and that each end-effector contacts the environment at a single point. The result takes the form of the conventional manipulator equations of motion <cit.>, with the transmission efficiencies embedded into the inertia and Coriolis matrices, and generalized forces term.

§.§ Application of the formulation

In contrast with the conservative formulation, the dissipative equation of motion makes evident the concept of energy flow in the system. There are different expressions if the actuators are driving the robot with torques $\bm \tau_\phi$ (FWD) or if the limbs are being accelerated by external forces $\bm f_{\textrm{ext}}$ (BWD). This concept of asymmetric transmission dynamics leads to separate design criteria if the robot actuators are expected to perform positive or negative work, and hence, conventional metrics can be re-derived <cit.>. We illustrate this characteristic by analyzing the design of a 2-DoF leg and compute its generalized inertia ellipsoid  <cit.>, its force capability or tip force bounds <cit.>, and its ability to mitigate shock loads due to impacts between the foot and the ground. The latter is defined by the Impact Mitigation Factor (IMF) <cit.>, which ranges from zero to one and quantifies the inertial backdrivability of the mechanism.

§.§.§ Generalized Inertia Tensor

The generalized inertia tensor (GIT) or the generalized inertia ellipsoid describes the inertia felt at the end-effector frame of a robot. We propose that the generalized inertia tensor of a dissipative system depends on the efficiency of transmissions and the direction of energy flow. Due to the dependency of the transmission dynamics on the direction of energy flow, the end-effector inertia perceived by the motors will be different than the inertia perceived by an external force back-driving the robot. The calculation of Backward-GIT is identical to the conventional GIT, since the external force is not distorted by transmission efficiency. However, the propagation of the actuation torque is different in the case of Forward-GIT. To accelerate the end-effector, the actuators exert a virtual task-space force $\hat{\bm f}_{\textrm{task}}$ given by

\begin{gather}
\bm{\tau_\phi} = \bM B_m^{-1} \bM J ^\top {\bm {\hat{f}_{\textrm{task}}}}. \label{VirtualForce}
\end{gather}

For the above equation to hold in the forward-driving cases, it is assumed that the system is fully-actuated or fixed-based ($b=0$).
Equation (<ref>) provides and (<ref>) the Forward-GIT.
Backward-GIT         (  J H^-1   J^⊤  )^-1,

Forward-GIT        (  J H^-1  B_m E_m B_m ^-1 J^⊤ )^-1,

where $\bM H \!=\! \bM K ^\top \bM E_s \bM H_s \bM K$.
Analogous to the forward and backward impedance described in equations (<ref>) and (<ref>), the Forward- and Backward-GIT are larger in respect to the GIT due to the lower efficiency of transmission.

§.§.§ Task-Space Force Capability

The task-space force capability (FC) estimates the maximum contact force that the robot can produce at the end-effector. Conventionally, this concept shows that this force is limited by the maximum torque that the actuators can generate. However, we propose that FC also depends on the mechanical efficiency of the robot's transmission. To better understand this concept, assume that a legged robot semi-statically interacts with the ground to support it's own body weight. In the forward-driving case, the leg motors must overcome the forces due to the robot mass plus the dissipative forces in the transmission. Hence, the lower the efficiency of the transmission, the higher the motor input effort must be to lift the robot. However, in the BWD case, the gravitational torques due to the robot weight must drive the motors through the transmission. Interestingly, in this scenario, the lower efficiency (due to high friction) helps the robot to passively support its own body. Thus, legged robots or industrial manipulators which employ highly geared and low efficiency actuators do not collapse under their own body weight when powered off. This phenomena is captured by Asymmetric-FC that combines Forward-FC and Backward-FC . The Asymmetric-FC estimates the robot's ability to quasi-statically support or resist external forces,

\begin{gather}
\textrm{Asymmetric-FC} =      {\bM J_m^{-\top}} \bM B_m \bM E_m \convhull({\bm \tau}_{{\phi}}),
\end{gather}

where we assume that the torque on rotors $\bm \tau_\phi$ is bounded and its feasible region is represented by a convex-hull. The Asymmetric-FC only employs the limb's contact Jacobian $\bM J_m \in \R ^{3\times m}$, and not that of the whole system. If a  robot has redundant actuators or is in a singular pose, the Asymmetric-FC can be obtained by linear programming <cit.>.

§.§ Effect of dissipative dynamics

We present two analysis of the design of a 2-DoF  planar leg composed of a serial actuation mechanism. First, we investigate the effect of transmission efficiencies on the task-space (foot) inertia matrix using the Forward- and Backward-GIT, and the force capability using the Asymmetric-FC. Next, the Asymmetric-FC and the IMF in the vertical $\hat z$ direction is computed to analyze the dynamic response of the leg to external contact forces. The Asymmetric-FC is normalized by the conventional FC to measure the relative difference due to the internal friction in the transmissions.

The leg, shown in Fig. <ref>(a), is equipped with two identical motors that can exert up to $20 \textrm{ N}\!\cdot \!\textrm{m}$ after identical speed reduction of $20:1$. Motors are attached to the hip ($ q_1$) and the knee ($ q_2$) joint. The thigh and the shin have the identical mass $m\!=\!2\textrm{ kg}$ and length $L\!=\!0.4\textrm{ m}$. Their center of mass are located at the links' midpoint. The base is modeled as a uniform planar square of dimensions $L_b\!=\! 0.5\textrm{ m}$ and mass $m_b\!=\!15\textrm{ kg}$. The base and the motors are initially at rest, with joint angles, $q_1\!=\!q_2\!=\frac{\pi}{3}$. The reflected inertia of a rotor is set identical to the inertia of a connected link at its center of mass. We assume a typical forward efficiency of single-stage gearboxes of $0.8$ and $0.7$ for the hip and knee transmission. The backward efficiency is a function of forward efficiency $\eta_f$ as displayed in Fig. <ref>(a) and calculated by the equation (38) of <cit.>.

We note that the backward efficiency is always smaller than the forward efficiency, and converges to zero when $\eta_f \xrightarrow{} 0.499$.

We observe the behavior of the asymmetric transmission dynamics in Fig. <ref>. First, both ellipsoids of Forward- and Backward-GIT in Fig. <ref>(b) are always larger than that of GIT.
Intuitively, driving forces need to inject more energy into the system to compensate for the frictional losses in the transmissions, which renders larger inertia.

(a) The 2-DoF floating base robot for the design study (b) The forward and backward generalized inertia ellipsoids of the 2-DoF Leg (c) The asymmetric force capabilities of the 2-DoF Leg

(a) The relation between forward and backward efficiencies. The backward efficiency converges to zero at $\eta_f=0.499$. (b) The $\hat{z}$-directional forward and backward force capabilities normalized by force capability. (c) The $\hat{z}$-directional IMF of 2-DoF Leg

In contrast, the Forward- and the Backward-FC in Fig. <ref>(c) show opposite tendencies. As the transmission efficiency decreases, the Forward-FC linearly decreases while Backward-FC diverges to infinity when $\eta_f\xrightarrow{}0.499$. Consequently, the leg can quasi-statically withstand more substantial load.

Finally, Fig. <ref> shows the FC, Forward- and Backward-FC, and IMF of the foot in the vertical $\hat z$ direction. The result conveys the trade-off between the Backward-FC and the IMF. As the actuators' efficiency decrease, the $\hat z-$Backward-FC in Fig. <ref>(b) increases, while $\hat z-$IMF in Fig. <ref>(c) decreases. In other words, large mechanical losses in the transmissions allow the robot to sustain more substantial static forces, but also degrade the machine's ability to mitigate shock loads from impacts with the ground.

§ CONCLUSION

This paper investigates how the dissipative forces in actuators and transmissions propagate to the dynamics of a whole robot. We present a framework that uses the mechanical efficiency to augment the inertia, Coriolis, and generalized force terms in the equation of motion.
We show how the individual efficiency of transmission influences not only the inertia felt at the end-effector, but also its capability of applying forces to and resisting disturbances from the environment. We expect that roboticists will use this formulation to add a tunable variable, the mechanical efficiency, to optimize the design of multi-body systems.
For instance, designers may exploit the beneficial effects of low efficiency to gain more static load-bearing capability, or to minimize the negative effects of friction for dynamic tasks with impacts.


# An Implementation of Multimodal Fusion System for Intelligent Digital Human
Generation
††thanks:  This work was supported in part by Undergraduate Training Program
for Innovation and Entrepreneurship, China University of Mining and
Techonology (No.202310290164Y).

Yingjie Zhou, Yaodong Chen, Kaiyue Bi, Lian Xiong, Hui Liu  School of
Information and Control Engineering, China University of Mining and
Technology, Xuzhou, China  {zyj2000, chenyaodong, bikaiyue, xionglian,
<EMAIL_ADDRESS>
###### Abstract

With the rapid development of artificial intelligence (AI), digital humans
have attracted more and more attention and are expected to achieve a wide
range of applications in several industries. Then, most of the existing
digital humans still rely on manual modeling by designers, which is a
cumbersome process and has a long development cycle. Therefore, facing the
rise of digital humans, there is an urgent need for a digital human generation
system combined with AI to improve development efficiency. In this paper, an
implementation scheme of an intelligent digital human generation system with
multimodal fusion is proposed. Specifically, text, speech and image are taken
as inputs, and interactive speech is synthesized using large language model
(LLM), voiceprint extraction, and text-to-speech conversion techniques. Then
the input image is age-transformed and a suitable image is selected as the
driving image. Then, the modification and generation of digital human video
content is realized by digital human driving, novel view synthesis, and
intelligent dressing techniques. Finally, we enhance the user experience
through style transfer, super-resolution, and quality evaluation. Experimental
results show that the system can effectively realize digital human generation.
The related code is released at
https://github.com/zyj-2000/CUMT_2D_PhotoSpeaker.

###### Index Terms:

digital human, AI, deep learning, multimodality, multimedia information
processing

##  I Introduction

Digital humans are computer-simulated virtual images with human appearance
characteristics, movement and expression, and have a wide range of application
prospects in many fields such as medicine, film and virtual reality. They are
considered as the entrance to the metaverse [1]. Especially in recent years,
with the rapid development of computer graphics, hardware equipment and
display technology, digital humans have not only become more vivid images but
also more intelligent, able to help people or independently complete specific
tasks. However, traditional digital human generation often includes processes
such as modeling, driving, and rendering. The whole process is cumbersome and
highly dependent on the designer’s professional skills, aesthetic perception,
and design experience. It consumes a lot of human resources and time costs.
With the growing demand for digital humans, the shortcomings of traditional
digital human generation technology are gradually exposed. On the other hand,
thanks to the rapid development of deep learning and artificial intelligence
(AI), artificial intelligence generated content (AIGC) has been able to
generate text, images, and even multimedia content that is very similar to
human-created content, improving the efficiency of practitioners in various
industries. For this reason, it can be believed that the introduction of AI
will effectively improve the efficiency of digital human generation. It is
necessary and highly significant to design an intelligent digital human
generation system.

Figure 1: Classification of digital human. The selected digital humans are
from the DDH-QA [2] and SJTU-H3D [3] databases. Figure 2: The framework of the
proposed intelligent digital human generation system.

As shown in the Fig. 1, from the data dimension of the digital human itself,
the common digital human can be divided into two-dimensional (2D) digital
human and three-dimensional (3D) digital human. Between them, the 2D digital
human itself is only a flat image, which usually can only be presented to the
audience in the form of 2D media, such as images and videos. In addition, the
research in related fields is more mature. In the last five years, many
databases of 2D digital faces [4, 5, 6, 7, 8, 9, 10, 11] have been proposed to
further promote the development of the field. Currently, there are two types
of mainstream driving methods for 2D faces. Person dependent face-driving
methods [12, 13, 14, 15] are designed for specific faces, thus enabling fine-
grained control with higher naturalness. Correspondingly, these advantages
also lead to the limited application of person dependent methods, which are
not well compatible with other faces. The person independent methods [16, 17,
18, 19, 20, 21, 22], on the other hand, although not as effective as the
person dependent methods in driving specific characters, are able to capture a
wider range of patterns with more data, effectively addressing the issue of
generalizability. Another type of digital human often explored in academia is
the 3D digital human. A 3D digital human can be represented by point clouds,
meshes or voxels, allowing the audience to view the digital human from
multiple viewpoints. Thanks to the structural characteristics of 3D digital
humans, they can either be rendered to generate 2D media in a specific
viewpoint or placed in virtual reality (VR) for immersive experience. At
present, some databases have been established in the academic field to address
this issue, such as NoW [23], FaceScape [24, 25], Human3.6M [26, 27], ZJU-
Mocap [28], and BEAT [29] in the area of 3D digital humans, VOCA [30] and
MultiFace [31] in the area of speech-driven, and DHHQA [32], DDH-QA [2], and
SJTU-H3D [3] in the area of quality assessment of 3D digital humans. On this
basis, the research of 3D digital humans is gradually deepened.

In this paper, we mainly focus on the 2D digital humans and use AI to provide
a multimodal, interactive, 3D, stylized intelligent digital human generation
technology. Specifically, the whole system contains three modules: the
preprocessing module, generation module and post-processing module. The
preprocessing module supports the input of rich media data consisting of text,
speech and character images. After the input text is answered by language
model, the speech can be obtained either through text-to-speech conversion or
by cloning from the timbre of the input speech. The difference between the two
methods is that the latter can preserve the phonological characteristics of
the input speech. Besides, this system adopts an age transformer to customize
the age for the person image, which enriches the user experience. The
generation module supports person independent and person dependent digital
human driving methods to modify the appearance of the digital human and
generate the animation. Particularly, this system realizes the modification of
digital humans’ clothes. The post-processing module is mainly responsible for
the style transfer and quality improvement of the generated animation. The
final digital human animation can also be assessed for quality in advance
through the quality evaluation model before flowing to the market. By
observing the generated digital human animations, the system integrates a
variety of intelligent models, which can quickly and effectively realize the
generation of digital humans and meet the needs of different users. Therefore,
the main contributions of this paper are as follows:

  * •

An effective and feasible design method for an intelligent digital human
generation system with multimodal fusion is proposed.

  * •

An open-source system for intelligent digital human generation with multimodal
fusion realized by combining existing technologies.

  * •

The current status of the development of digital human-related technologies is
reviewed, and application scenarios are given for this system.

##  II Proposed Method

In this section, we specifically discuss the proposed method. The framework of
the proposed method is schematically shown in Fig. 2. It includes
preprocessing module, generation module and post-processing module.

###  II-A Preprocessing Module

Considering that the digital human generation system is a multimodal fusion
information processing system, the preprocessing module processes the input
rich media data separately for the input. Given the input media data type $M$:

$M\in\\{W,A,P\\},$ (1)

where $W$ denotes the input text, which is mainly used for the user to
interact with the digital image, and $A$ denotes the input target audio, which
is mainly used for voiceprint extraction and speech cloning. If the user does
not provide the audio $A$, the system will output the digital human media with
fixed voice through the text-to-speech conversion. $P$ denotes the input RGB
character image, which is the main driving object. In this paper, the user
input text is interactively responded to through the language model and the
whole process can be described as follows:

$\begin{gathered}\mathcal{W}{\text{ =
}}Q([{W_{n}},W{R_{n-1}},\cdots,W{R_{1}}],U),\\\ {R_{n}}{\text{ =
}}\operatorname{Re}(\mathcal{W}),\\\ \end{gathered}$ (2)

where $\mathcal{W}$ denotes a POST access request, ${W_{n}}$ denotes the input
text of the user during the $n$th round of interaction, ${R_{n}}$ denotes the
response text of language model during the $n$th round of interaction,
${WR_{n}}$ denotes the $n$th round of interaction, including a pair of input
and response text, $U$ denotes the IP of the deployed language model,
$Q(\cdot)$ denotes the process of a local host sending a POST access request
to deployed language model, $[{W_{n}},W{R_{n-1}},\cdots,W{R_{1}}]$ denotes the
request body of the POST, and $\operatorname{Re}(\cdot)$ denotes the process
where a response is fed back from the language model during the $n$th round of
interaction. The response generated by the language model further selects
whether or not to perform extraction of voiceprint and speech cloning based on
whether or not the target audio data $A$ is input. Thus, the whole process of
speech generation can be described as:

$S=G_{s}(R_{n})\parallel G_{c}(R_{n},\mathbb{F}),$ (3)

where $G_{s}(\cdot)$ and $G_{c}(\cdot)$ denote the process of text-to-speech
conversion and speech cloning, respectively, $\parallel$ indicates the
selection of one specific content from multiple options, $\mathbb{F}$ denotes
the acoustic features extracted from the target audio $A$, and $S$ is the
digital human speech generated with reference to the text $R_{n}$. For the
input digital human image $P$, the appearance of the image is selected and
modified by age transformation. The whole process can be described as:

$\begin{gathered}{I_{a}}=Age(P),\\\
{I_{a}}=[{i_{1}},{i_{2}},{i_{3}},\cdots,{i_{k}}],\\\ \end{gathered}$ (4)

where $Age(\cdot)$ denotes the process of age transformation, $I_{a}$ denotes
the set of digital human images corresponding to each age, and $i_{k}$ denotes
the digital human appearance image corresponding to the $k$th age.

###  II-B Generation Module

The generation module performs clothing modification and face driving for the
synthesized speech and selected digital human images, and also can widen the
observation field of the digital human through the novel view synthesis of
monocular RGB images to make the digital human 3D. First of all, the free
dressing of the digital human can be realized through human posture detection
and clothing matching. Specifically, the whole dressing process can be
described as:

$\begin{gathered}{I_{c}}={C_{T}}(i),\\\ i\in{I_{a}},\\\ \end{gathered}$ (5)

where ${C_{T}}(\cdot)$ denotes the process of digital human dressing, $i$
denotes the image selected from the digital human age image set $I_{a}$, and
$I_{c}$ denotes the digital human clothing image set. After that, two common
driving methods are designed for the face driving of digital human. Between
them, person independent driving method is simpler and can directly generate
the digital human animation using the input speech and image, while person
dependent driving method can only be driven for a specific person image, so it
is necessary to use the action retarget after driving a specific person image
in order to obtain the desired digital human animation. Overall, the whole
driving process can be described as:

$\begin{gathered}\hat{i}\in{I_{a}}\cup{I_{c}},\\\
{V_{m}}={\mathfrak{M}_{i}}(\hat{i},S)||({\mathfrak{M}_{d}}(S)\oplus{\mathfrak{M}_{t}}(\hat{i})),\\\
\end{gathered}$ (6)

where, ${\mathfrak{M}_{i}}(\cdot)$, ${\mathfrak{M}_{d}}(\cdot)$ and
${\mathfrak{M}_{t}}(\cdot)$ denote the adoption of person independent, person
dependent driving method and action retarget method, respectively, $\oplus$
denotes the successive realization of two processes, $\hat{i}$ denotes the
selected digital human image, $S$ is the digital human driving speech, and
$V_{m}$ is the generated digital human animation. In particular, this system
realizes the expansion of the viewpoint of the digital human through the novel
view synthesis to form a 3D visual effect. This function is optional for users
and the whole process can be described as:

${V_{3{\text{d}}}}=3D({V_{m}}),$ (7)

where $3D(\cdot)$ denotes the 3D effect realized by the novel view synthesis
and ${V_{3{\text{d}}}}$ denotes the synthesized 3D digital human video.

###  II-C Post-processing Module

In order to further enhance the quality of the user experience with the
digital human product, this system performs post-processing on the generated
digital human animation. It is worth stating that the post-processing module
is not mandatory to be used but merely provides more selectivity for users.
Specifically, the module designs three parts, namely, style transfer, super-
resolution, and quality assessment, respectively. Among them, the style
transfer model is able to transfer the features of a specific style obtained
through learning to the input media. The process can be chosen according to
the actual needs and described as:

${V_{s}}=\Psi({V_{m}}||{V_{3d}}),$ (8)

where $\Psi(\cdot)$ denotes the process of style transfer, and ${V_{s}}$
denotes the stylized video. In addition, the super-resolution technique can
effectively improve the quality of digital human animation, which can be
described as:

${V_{SR}}=\varphi({V_{m}}||{V_{3d}}||{V_{s}}),$ (9)

where $\varphi(\cdot)$ denotes the process of super-resolving the animation,
and ${V_{SR}}$ denotes the processed video. Finally, before the digital human
video is presented to the viewers, the quality assessment model may perform an
objective evaluation of the generated video based on the a priori knowledge of
the digital human video, which may be described as:

$Score=QA({V_{m}}||{V_{3d}}||{V_{s}}||{V_{SR}}),$ (10)

where $QA(\cdot)$ denotes the process of evaluating a digital human video
using an objective quality assessment algorithm, and $Score$ denotes the
result of the evaluation. The result can provide guidance for the improvement
and optimization of the digital human generation system.

##  III Experiments

(a) Multimodal digital human face animation

(b) Age-transformed digital human face animation

(c) 3D digital human face animation

(d) Stylized digital human face animation

Figure 3: Visual effects of generating digital human face animations.

(a) Multimodal digital human animation

(b) Dress-up digital human animation

(c) Stylized digital human animation

Figure 4: Visual effects of generating digital human animations.

###  III-A Experimental Setup

In order to further validate the feasibility of the proposed method, we have
implemented the method with the prior art, including the large language model
ChatGLM [33], the text-to-speech conversion package
eSpeaker111https://espeak.sourceforge.net/, the speech cloning model
MockingBird222https://github.com/babysor/MockingBird, the age transformer
model SAM [34], the person dependent driving method LiveSpeechPortraits [35],
the action retarget Thin-Plate-Spline-Motion-Model [36], the person
independent driving method SadTalker [37], cloth modification model VITON-HD
[38], style transfer model DCT-Net [39], super resolution model BasicVSR++
[40, 41], quality assessment model VSFA [42]. In our experiments, we trained
and tested our proposed method on a server using Intel i7-7700K CPU @ 4.20GHz,
32GB RAM and NVIDIA 2080TI GPU.

###  III-B Generating Effects

This section tests the visual effects of generating digital human animations,
including digital face animations and digital human animations. In digital
face generation, four different digital human images were selected from
StyleGAN database333https://github.com/a312863063/generators-with-stylegan2.
Specifically, the generation of multimodal digital faces, age-transformed
digital faces, 3D digital faces, and stylized digital faces by different audio
was tested and the results are shown in Fig. 3. As for digital human, we chose
three digital human images from HumanAlloy444https://humanalloy.com/, VITON-HD
[38] and Cgtrader555https://www.cgtrader.com/. Then, the generation of
multimodal digital human, dress-up digital human, and stylized digital human
by different audio was tested. The results are shown in Fig. 4. Combining Fig.
3 and Fig. 4, it can be concluded that the proposed method can effectively
realize the generation of digital human face and full-body digital human. In
addition, the method realizes the effects of age conversion, 3D, style
transfer, and dress-up by prior art, which enriches the user experience.

###  III-C Performance indicators

To further validate the generative effectiveness of the proposed method, we
also tested seven generated digital human videos mentioned above using image
and video quality assessment methods. In terms of image quality evaluation
metrics, CPBD [43] for evaluating blur and CGIQA [44] for evaluating CG
animation were selected. Besides, two advanced video methods were chosen,
including VSFA [42] and FastVQA [45]. In this case, the raw output of CPBD was
recorded, while the outputs of the CGIQA, VSFA and FAST-VQA, were all
normalized. The higher the four metrics, the higher the quality of the
generated digital human video. It is worth noting that the image quality
assessment method evaluates each frame of the video and finally the average
result is recorded. All methods are performed using source code provided by
the authors. The test results are shown in Table I. As can be seen in Table I,
the generated digital human videos are clear enough and have some aesthetic
value. In addition, most of the generated videos achieved satisfactory
performance on the two advanced video quality evaluation metrics. This further
proves the effectiveness of the proposed method.

TABLE I: PERFORMANCE RESULTS FOR GENERATED DIGITAL HUMAN ANIMATIONS.  Index | Name | Image Quality | Video Quality
---|---|---|---
CPBD$\uparrow$ | CGIQA$\uparrow$ | VSFA$\uparrow$ | FAST-VQA$\uparrow$
Fig. LABEL:sub@fig2:a | Short hair man face | 0.2800 | 0.6015 | 0.8510 | 0.8589
Fig. LABEL:sub@fig2:b | Short hair woman face | 0.3391 | 0.6164 | 0.8880 | 0.8802
Fig. LABEL:sub@fig2:c | Long hair man face | 0.4736 | 0.5622 | 0.8445 | 0.8694
Fig. LABEL:sub@fig2:d | Long hair woman face | 0.4100 | 0.5688 | 0.9893 | 0.6294
Fig. LABEL:sub@fig3:a | Nash | 0.4134 | 0.4692 | 0.8724 | 0.7659
Fig. LABEL:sub@fig3:b | Fashion model | 0.4628 | 0.6215 | 0.8647 | 0.7579
Fig. LABEL:sub@fig3:c | Woman | 0.4471 | 0.5763 | 0.8878 | 0.6094

##  IV Conclusion

In this paper, we propose a method for implementing an intelligent digital
human generation system with multimodal fusion. The proposed method performs
separate preprocessing for text, speech, and image. Then, the preprocessed
media-rich data is used to generate vivid digital human videos through digital
human driving methods, intelligent dressing technology, and novel view
synthesis. Finally, to enhance the user experience, the post-processing module
offers three optional functions: style transfer, super-resolution, and quality
assessment. Furthermore, we have developed an effective and feasible
multimodal fusion system for intelligent digital human generation using
existing technologies based on the proposed method. The system has been
validated with different images and audio, demonstrating its ability to
achieve the expected results. Given the features of this system, we anticipate
revolutionary applications in various fields such as entertainment, film,
games, and clothing sales.

## References

  * [1] Y. Zhou, Z. Zhang, W. Sun, X. Min, X. Ma, and G. Zhai, “A no-reference quality assessment method for digital human head,” in _2023 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2023, pp. 36–40.
  * [2] Z. Zhang, Y. Zhou, W. Sun, W. Lu, X. Min, Y. Wang, and G. Zhai, “Ddh-qa: A dynamic digital humans quality assessment database,” in _2023 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2023, pp. 2519–2524.
  * [3] Z. Zhang, W. Sun, Y. Zhou, H. Wu, C. Li, X. Min, X. Liu, G. Zhai, and W. Lin, “Advancing zero-shot digital human quality assessment through text-prompted evaluation,” _arXiv preprint arXiv:2307.02808_ , 2023.
  * [4] K. Wang, Q. Wu, L. Song, Z. Yang, W. Wu, C. Qian, R. He, Y. Qiao, and C. C. Loy, “Mead: A large-scale audio-visual dataset for emotional talking-face generation,” in _European Conference on Computer Vision_. Springer, 2020, pp. 700–717.
  * [5] A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, “Voxceleb: Large-scale speaker verification in the wild,” _Computer Speech & Language_, vol. 60, p. 101027, 2020.
  * [6] A. Nagrani, J. S. Chung, and A. Zisserman, “Voxceleb: a large-scale speaker identification dataset,” _arXiv preprint arXiv:1706.08612_ , 2017.
  * [7] J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker recognition,” _arXiv preprint arXiv:1806.05622_ , 2018.
  * [8] J. S. Chung and A. Zisserman, “Lip reading in the wild,” in _Computer Vision–ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13_. Springer, 2017, pp. 87–103.
  * [9] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Deep audio-visual speech recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 12, pp. 8717–8727, 2018.
  * [10] J. Son Chung, A. Senior, O. Vinyals, and A. Zisserman, “Lip reading sentences in the wild,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6447–6456.
  * [11] J. Chung and A. Zisserman, “Lip reading in profile,” in _ritish Machine Vision Conference, 2017_. British Machine Vision Association and Society for Pattern Recognition, 2017.
  * [12] S. Suwajanakorn, S. M. Seitz, and I. Kemelmacher-Shlizerman, “Synthesizing obama: learning lip sync from audio,” _ACM Transactions on Graphics (ToG)_ , vol. 36, no. 4, pp. 1–13, 2017.
  * [13] K. Sato, T. Nose, A. Ito _et al._ , “Hmm-based photo-realistic talking face synthesis using facial expression parameter mapping with deep neural networks,” _Journal of Computer and Communications_ , vol. 5, no. 10, p. 50, 2017.
  * [14] P. P. Filntisis, A. Katsamanis, and P. Maragos, “Photorealistic adaptation and interpolation of facial expressions using hmms and aams for audio-visual speech synthesis,” in _2017 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2017, pp. 2941–2945.
  * [15] R. Kumar, J. Sotelo, K. Kumar, A. De Brebisson, and Y. Bengio, “Obamanet: Photo-realistic lip-sync from text,” _arXiv preprint arXiv:1801.01442_ , 2017\.
  * [16] R. Huang, P. Lai, Y. Qin, and G. Li, “Parametric implicit face representation for audio-driven facial reenactment,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 12 759–12 768.
  * [17] W. Zhong, C. Fang, Y. Cai, P. Wei, G. Zhao, L. Lin, and G. Li, “Identity-preserving talking face generation with landmark and appearance priors,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 9729–9738.
  * [18] J. Guan, Z. Zhang, H. Zhou, T. Hu, K. Wang, D. He, H. Feng, J. Liu, E. Ding, Z. Liu _et al._ , “Stylesync: High-fidelity generalized and personalized lip sync in style-based generator,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 1505–1515.
  * [19] Y. Gao, Y. Zhou, J. Wang, X. Li, X. Ming, and Y. Lu, “High-fidelity and freely controllable talking head video generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 5609–5619.
  * [20] W. Li, L. Zhang, D. Wang, B. Zhao, Z. Wang, M. Chen, B. Zhang, Z. Wang, L. Bo, and X. Li, “One-shot high-fidelity talking-head synthesis with deformable neural radiance field,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 17 969–17 978.
  * [21] J. Wang, X. Qian, M. Zhang, R. T. Tan, and H. Li, “Seeing what you said: Talking face generation guided by a lip reading expert,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 14 653–14 662.
  * [22] G. Hwang, S. Hong, S. Lee, S. Park, and G. Chae, “Discohead: Audio-and-video-driven talking head generation by disentangled control of head pose and facial expressions,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [23] S. Sanyal, T. Bolkart, H. Feng, and M. J. Black, “Learning to regress 3d face shape and expression from an image without 3d supervision,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7763–7772.
  * [24] H. Yang, H. Zhu, Y. Wang, M. Huang, Q. Shen, R. Yang, and X. Cao, “Facescape: a large-scale high quality 3d face dataset and detailed riggable 3d face prediction,” in _Proceedings of the ieee/cvf conference on computer vision and pattern recognition_ , 2020, pp. 601–610.
  * [25] H. Zhu, H. Yang, L. Guo, Y. Zhang, Y. Wang, M. Huang, Q. Shen, R. Yang, and X. Cao, “Facescape: 3d facial dataset and benchmark for single-view 3d face reconstruction,” _arXiv preprint arXiv:2111.01082_ , 2021.
  * [26] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, “Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 36, no. 7, pp. 1325–1339, 2013.
  * [27] C. Ionescu, F. Li, and C. Sminchisescu, “Latent structured models for human pose estimation,” in _2011 International Conference on Computer Vision_. IEEE, 2011, pp. 2220–2227.
  * [28] S. Peng, Y. Zhang, Y. Xu, Q. Wang, Q. Shuai, H. Bao, and X. Zhou, “Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 9054–9063.
  * [29] H. Liu, Z. Zhu, N. Iwamoto, Y. Peng, Z. Li, Y. Zhou, E. Bozkurt, and B. Zheng, “Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis,” in _European Conference on Computer Vision_. Springer, 2022, pp. 612–630.
  * [30] D. Cudeiro, T. Bolkart, C. Laidlaw, A. Ranjan, and M. J. Black, “Capture, learning, and synthesis of 3d speaking styles,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 101–10 111.
  * [31] C.-h. Wuu, N. Zheng, S. Ardisson, R. Bali, D. Belko, E. Brockmeyer, L. Evans, T. Godisart, H. Ha, A. Hypes, T. Koska, S. Krenn, S. Lombardi, X. Luo, K. McPhail, L. Millerschoen, M. Perdoch, M. Pitts, A. Richard, J. Saragih, J. Saragih, T. Shiratori, T. Simon, M. Stewart, A. Trimble, X. Weng, D. Whitewolf, C. Wu, S.-I. Yu, and Y. Sheikh, “Multiface: A dataset for neural face rendering,” in _arXiv_ , 2022. [Online]. Available: https://arxiv.org/abs/2207.11243
  * [32] Z. Zhang, Y. Zhou, W. Sun, X. Min, Y. Wu, and G. Zhai, “Perceptual quality assessment for digital human heads,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [33] A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia _et al._ , “Glm-130b: An open bilingual pre-trained model,” _arXiv preprint arXiv:2210.02414_ , 2022.
  * [34] Y. Alaluf, O. Patashnik, and D. Cohen-Or, “Only a matter of style: Age transformation using a style-based regression model,” _ACM Trans. Graph._ , vol. 40, no. 4, 2021. [Online]. Available: https://doi.org/10.1145/3450626.3459805
  * [35] Y. Lu, J. Chai, and X. Cao, “Live speech portraits: real-time photorealistic talking-head animation,” _ACM Transactions on Graphics (TOG)_ , vol. 40, no. 6, pp. 1–17, 2021.
  * [36] J. Zhao and H. Zhang, “Thin-plate spline motion model for image animation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 3657–3666.
  * [37] W. Zhang, X. Cun, X. Wang, Y. Zhang, X. Shen, Y. Guo, Y. Shan, and F. Wang, “Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 8652–8661.
  * [38] S. Choi, S. Park, M. Lee, and J. Choo, “Viton-hd: High-resolution virtual try-on via misalignment-aware normalization,” in _Proc. of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2021.
  * [39] Y. Men, Y. Yao, M. Cui, Z. Lian, and X. Xie, “Dct-net: Domain-calibrated translation for portrait stylization,” vol. 41, no. 4. ACM New York, NY, USA, 2022, pp. 1–9.
  * [40] K. C. Chan, S. Zhou, X. Xu, and C. C. Loy, “BasicVSR++: Improving video super-resolution with enhanced propagation and alignment,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2022.
  * [41] ——, “On the generalization of BasicVSR++ to video deblurring and denoising,” _arXiv preprint arXiv:2204.05308_ , 2022.
  * [42] D. Li, T. Jiang, and M. Jiang, “Quality assessment of in-the-wild videos,” in _Proceedings of the 27th ACM International Conference on Multimedia_ , 2019, pp. 2351–2359.
  * [43] N. D. Narvekar and L. J. Karam, “A no-reference image blur metric based on the cumulative probability of blur detection (cpbd),” _IEEE Transactions on Image Processing_ , vol. 20, no. 9, pp. 2678–2683, 2011.
  * [44] Z. Zhang, W. Sun, Y. Zhou, J. Jia, Z. Zhang, J. Liu, X. Min, and G. Zhai, “Subjective and objective quality assessment for in-the-wild computer graphics images,” _arXiv preprint arXiv:2303.08050_ , 2023.
  * [45] H. Wu, C. Chen, J. Hou, L. Liao, A. Wang, W. Sun, Q. Yan, and W. Lin, “Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling,” in _European Conference on Computer Vision_. Springer, 2022, pp. 538–554.