# LASR: Learning Articulated Shape Reconstruction from a Monocular Video

Gengshan Yang1 Work done in an internship at Google.  Deqing Sun2 Varun
Jampani2 Daniel Vlasic2 Forrester Cole2 Huiwen Chang2
Deva Ramanan1 William T. Freeman2 and Ce Liu2
1Carnegie Mellon University, 2Google Research

###### Abstract

Remarkable progress has been made in 3D reconstruction of rigid structures
from a video or a collection of images. However, it is still challenging to
reconstruct nonrigid structures from RGB inputs, due to its under-constrained
nature. While template-based approaches, such as parametric shape models, have
achieved great success in modeling the “closed world” of known object
categories, they cannot well handle the “open-world” of novel object
categories or outlier shapes. In this work, we introduce a template-free
approach to learn 3D shapes from a single video. It adopts an analysis-by-
synthesis strategy that forward-renders object silhouette, optical flow, and
pixel values to compare with video observations, which generates gradients to
adjust the camera, shape and motion parameters. Without using a category-
specific shape template, our method faithfully reconstructs nonrigid 3D
structures from videos of human, animals, and objects of unknown classes. Our
code is available at lasr-google.github.io.

Figure 1: Top: Sample input video frames and articulated shapes recovered by
our method (LASR). Bottom: Comparison with existing methods, where the input
to each method (either video or image) is denoted at the top left, and the
shape template being used is denoted at the bottom right of each result. Many
existing approaches on nonrigid shape reconstruction heavily rely on category-
specific 3D shape templates, such as SMPL for human [34, 36] and SMAL for
quadrupeds [6, 59]. In contrast, LASR jointly recovers the object shape,
articulation, and camera parameters from a monocular video without using
category-specific shape templates. By relying on generic shape and motion
priors, LASR applies to a wider range of nonrigid shapes and yields high-
fidelity 3D reconstructions: It recovers both humps of the camel, which are
missing from other methods. It also recovers the silk ribbon of the dancer (as
denoted by the blue box), which confuses SMPLify-X and VIBE as the right arm.
Please refer to video results on our project page.

##  1 Introduction

Perceiving and modeling the geometry and dynamics of 3D entities is an open
research problem in computer vision and has numerous applications. One
fundamental challenge is the under-constrained nature of the problem: from
limited 2D image measurements, there exist multiple interpretations of the
geometry and motion of the 3D world.

A recent and promising trend for addressing this challenge is exploiting data
priors, which have proven quite successful for high-level vision tasks, such
as image classification and object detection [13, 32]. However, in contrast to
high-level vision tasks, it is often costly to obtain 3D annotations for real-
world entities. For example, SMPL [34] is learned from thousands of registered
3D scans of human. SMAL [59] is learned from scans of animal toys and a
manually rigged mesh model. It involves nontrivial efforts to collect such
data for an arbitrary object category. Therefore, existing methods often fail
to capture objects of novel or unknown classes, and hallucinate an average 3D
structure based on the category shape prior, as shown in Fig. LASR: Learning
Articulated Shape Reconstruction from a Monocular Video.

Interestingly, remarkable progress has been made in the field of SLAM and
structure-from-motion without relying on strong shape priors by taking
advantage of multiview data recordings. However, such results are limited to
static scenes. We explore an intermediate regime between these two extremes:
Can one reconstruct an articulated shape from video data without relying on
template priors?

Why videos? To reconstruct 3D object shape from images, prior work learns
category-specific shape models either from 3D data [17, 42] or from 2D
supervision, such as object silhouette and keypoints in a large image
collection [10, 18, 23, 31]. However, 3D data are generally difficult to
acquire at a large scale due to sensor design. Although it is easier to
collect images of the same category, enforcing multiview constraints is often
challenging, due to ambiguities of associating 2D observations across
instances and under different viewpoints [11, 40]. Video serves as an
alternative to depth scans and image collections – videos are easier to
acquire, and provide well-defined multiview constraints on the 3D shape of the
same instance.

Why optical flow? To solve the inverse problem, prior work discussed various
forms of 2D constraints or supervision, such as object silhouette, texture, 2D
keypoints, and semantic parts [4, 18, 23, 31]. Why should motion be treated as
a first-class citizen? Besides that optical flow naturally encodes
correspondences, it provides more fine-grained information than keypoints as
well as semantic parts. Different from long-range point tracks, which is the
classic input for NRSfM [43], optical flow can be obtained more reliably [48,
53] over two consecutive frames.

Why not nonrigid SfM? NRSfM deals with a problem similar to ours: given a set
of 2D point trajectories depicting a deformable object in a collection of
images, the goal is to recover the 3D object shape and pose (i.e., relative
camera position) in each view. Usually, trajectories of 2D points are
factorized into low-rank shape and motion matrices [8, 19, 27] without using
3D shape templates. Although NRSfM is able to deal with generic shapes, it
requires reliable long-term point tracks or keypoint annotations, which are
challenging to acquire densely in practice [43, 45, 47].

Proposed approach: Instead of inferring 3D shapes from category-specific image
collections or point trajectories, we build an articulated shape model from a
monocular video of an object. Recent progress in differentiable rendering
allows one to recast the problem as an analysis-by-synthesis task: we solve
the inverse graphics problem of recovering the 3D object shape (including
spacetime deformations) and camera trajectories (including intrinsics) to fit
video observations, such as object silhouette, raw pixels, and optical flow.
An overview of the pipeline is shown in Fig. 2.

Contributions: We propose a method for articulated shape reconstruction from a
monocular video that does not require a prior template or category
information. It takes advantage of dense two-frame optical flow to overcome
the inherent ambiguity in the nonrigid structure and motion estimation
problem. It automatically recovers a nonrigid shape under the constraints of
rigid bones under linear-blend skinning. It combines coarse-to-fine re-meshing
with soft-symmetric constraints to recover high-quality meshes. Our method
demonstrates state-of-the-art reconstruction performance in the BADJA animal
video dataset [6], strong performance against model-based methods on humans,
and higher accuracy on two animated animals than A-CSM [28] and SMALify [6]
that use shape templates.

Table 1: Related work in nonrigid shape reconstruction. (1)Model-based optimization and regression methods. (2)Category-specific mesh reconstruction. (3)Template-free approaches. S: single view. V: video or multi-view data. I: images. J2: 2D joints. J3: 3D joints. M: 2D masks. V3: 3D scans. C: camera matrices. F: optical flow. MF: multi-frame optical flow. quad: quadruped animals. †:Only representative categories are listed. | Method | category | template | test-input | train
---|---|---|---|---|---
(1) |  SMPLify [7] | human | SMPL | S:J2,M | None
VIBE [26] | human | SMPL | V:I | J2,J3
SMALify [6] | quadx5 | SMAL | V:J2,M | None
SMALR [58] | quadx12 | SMAL | S:J2,M | None
SMALST [57] | zebra | SMAL | S:I | J2,V3
WLDO [5] | dog | SMAL | S:I | J2,M
(2) |  CMR [23] |  bird† | SfM-hull | S:I | J2,M,C
UCMR [18] |  bird† | cate-mesh | S:I | M
UMR [31] |  bird† | None | S:I | M
IMR [49] | animals | cate-mesh | S:I | M
A-CSM [28] | animals | cate-mesh | S:I | M
VMR [30] | animals | cate-mesh | V:M | None
(3) |  PIFuHD [42] | human | None | S:I | V3
NRSfM [2, 12] | any | None | V:J2 | None
N-NRSfM [45] | any | None | V:MF,M | None
WSD [10] |  dolphin† | cate-mesh | V:J2,M | None
A3DC [39] | any | None | V:stroke | None
LASR (Ours) | any | None | V:F,M | None
Figure 2: Method overview. Given a monocular video of an object, we jointly
recover the object’s rest shape ${\bf S}$, skinning weights ${\bf W}$,
articulation ${\bf D_{t}}$, and camera parameters ${\bf K_{t}}$ by solving an
inverse graphics problem through gradient-based optimization. The object rest
shape is represented by a mesh (initialized from a sphere) and articulated at
each frame under linear blend skinning (Sec. 3.2). The time-varying
parameters, including focal length, object root transformation and
articulation, are parameterized by a neural basis, i.e., convolutional network
given image inputs (Sec. 3.4). At each iteration, we randomly sample pairs of
consecutive frames and forward-render object silhouette, texture, and two-
frame optical flow using a differentiable renderer (Sec. 3.1). The renderings
are compared against raw pixels, segmentation and optical flow estimated by
off-the-shelf networks to generate gradients signals and update the model
(Sec. 3.3).

##  2 Related Work

Below and in Tab. LABEL:tab:taxonomy, we discuss related work of nonrigid
shape recovery according to priors being used (shape template, category prior,
or generic shape and motion priors).

Model-based reconstruction: Model-based reconstruction leverages a parametric
shape model to solve the under-constrained 3D shape and pose estimation
problem. A large body of work in 3D human and animal reconstruction uses such
parametric shape models [34, 36, 52, 58, 59], which are learned from 3D scans
of human or animal toys [34, 59], and allow one to recover the 3D shape given
very few annotations at test time (2D keypoints and silhouettes). Recently,
model-based regression methods are developed to predict model-specific shape
and pose parameters from a single image or video [3, 5, 26, 57], usually
trained with ground-truth 3D data generated by such parametric shape models.
Although model-based methods achieve great success in reconstructing “closed-
world” objects of known category and rich 3D data, it is challenging to apply
to unknown object categories, or categories with limited 3D data.

Category mesh reconstruction: A recent trend is to reduce supervision from 3D
or multi-view capturing to 2D annotations, such as keypoints and object
silhouettes [18, 23, 31]. Such methods often take advantage of category
priors, including a collection of images from the same category, and category-
specific shape templates [28, 49]. Recent progress makes single-view
reconstruction of birds and other common categories possible without 3D
annotation. However, the single view reconstruction is usually coarse and
lacks instance-specific details. Recent work adapts category-specific models
to a test-time video [30], but still does not handle objects of unknown
classes.

Template-free reconstruction: Among the template-free methods, PIFu [41, 42]
learns to predict an implicit shape representation for clothed human
reconstruction, but requires ground-truth 3D shapes to train. A3DC [39]
reconstructs articulated animals from videos, but requires involved user
stroke interactions. Without requiring 3D data or user interactions, NRSfM
factorizes a set of 2D keypoints or point trajectories into the 3D object
shape and pose in each view assuming “low-rank” shape or deformation [2, 12,
19]. Recently, deep networks have been applied to learn such factorization of
specific categories from 2D annotations [27, 35, 51]. Close to our approach,
Neural Dense NRSfM (N-NRSfM) [45] learns a video-specific shape and
deformation model from dense 2D point tracks. However, such methods are
limited by the accuracy of 2D trajectory inputs, which is challenging to
estimate in real-world sequences when large motion occurs [43, 45, 47].

##  3 Approach

Problem: Given a monocular video $\\{I_{t}\\}$ with an object of interest
(indicated by a segmentation mask $\\{S_{t}\\}$), we tackle the nonrigid 3D
shape and motion estimation problem, which includes estimating (1) ${\bf S}$:
the rest shape of the object, (2) ${\bf D_{t}}$: the time-varying
articulations as well as the object root body transformations, and (3) ${\bf
K_{t}}$: the camera intrinsics.

Overview: Figure 2 illustrates the overview of our method. Motivated by recent
progress in differentiable rendering and self-supervised shape learning [23,
33], we cast the nonrigid 3D shape and motion estimation problem as an
analysis-by-synthesis task. Despite the under-constrained nature of this
problem, we hypothesize that, by giving appropriate video measurements, a
“low-rank” shape and motion can be solved up to an unknown scale. Model
parameters ${\bf X=\\{S,D_{t},K_{t}\\}}$ are updated (via gradient descent) to
minimize the difference between the rendered output ${\bf Y}\\!=\\!{\bf
f}({\bf X})$ and ground-truth video measurements ${\bf Y^{*}}$ at _test_ time
(Sec. 3.1). To deal with the fundamental ambiguities in object shape,
deformation and camera motion, we seek (1) a ”low-rank” but expressive
parameterization of deformation (Sec. 3.2), (2) rich constraints provided by
optical flow and raw pixels, and (3) appropriate regularization of object
shape deformation and camera motion (Sec. 3.3).

###  3.1 Forward-synthesis model

We first introduce the forward synthesis model. Given a frame index $t$ and
model parameters ${\bf X}$, we synthesize the measurements of the
corresponding frame pair $\\{t,t+1\\}$, including color images renderings
$\\{\hat{I}_{t},\hat{I}_{t+1}\\}$, object silhouettes renderings
$\\{\hat{S}_{t},\hat{S}_{t+1}\\}$ and forward-backward optical flow renderings
$\\{{\bf\hat{u}}^{+}_{t},{\bf\hat{u}}^{-}_{t+1}\\}$.

Rendering pipeline: We represent object shape as a triangular mesh ${\bf
S}=\\{{\bf\bar{V}},{\bf\bar{C}},{\bf F}\\}$ with vertices
${\bf\bar{V}}\in\mathbb{R}^{N\times 3}$, vertex colors
${\bf\bar{C}}\in\mathbb{R}^{N\times 3}$ and a fixed topology ${\bf
F}\in\mathbb{R}^{M\times 3}$. To model time-varying articulations ${\bf
D_{t}}$, we have

$\footnotesize{\bf V}_{t}={\bf G_{0,t}}({\bar{\bf V}}+\Delta{\bf V_{t}})\\\ $
(1)

where $\Delta{\bf V_{t}}$ is a per-vertex motion field applied to the rest
vertices ${\bf\bar{V}}$ , and ${\bf G_{0,t}}=\begin{pmatrix}[c|c]{\bf
R_{0}}&{\bf T_{0}}\end{pmatrix}{\bf{}_{t}}$ is an object root body
transformation matrix (index 0 is used to differentiate from bone
transformations indexed from 1 in Sec. 3.2). Finally, we apply a perspective
projection ${\bf K_{t}}$ before rasterization, where principal point
$(p_{x},p_{y})$ is assumed to be constant and focal length $f_{t}$ varies over
time to deal with zoom-in/out.

Shaders: We render object silhouette and color images with a differentiable
renderer [33]. Color images are rendered given per-vertex appearance $\bar{\bf
C}$ and constant ambient light. To synthesize the forward flow ${\bf
u}^{+}_{t}$, we take surface positions ${\bf V}_{t}$ corresponding to each
pixel in frame $t$, compute their locations ${\bf V}_{t+1}$ in the next frame,
then take the difference of their projections:

$\footnotesize\begin{pmatrix}u_{x,t}^{+}\\\
u_{y,t}^{+}\end{pmatrix}=\begin{pmatrix}{\bf P}^{(1)}_{t}{\bf V}_{t}/{\bf
P}^{(3)}_{t}{\bf V}_{t}\\\ {\bf P}^{(2)}_{t}{\bf V}_{t}/{\bf P}^{(3)}_{t}{\bf
V}_{t}\end{pmatrix}-\begin{pmatrix}{\bf P}^{(1)}_{t+1}{\bf V}_{t+1}/{\bf
P}^{(3)}_{t+1}{\bf V}_{t+1}\\\ {\bf P}^{(2)}_{t+1}{\bf V}_{t+1}/{\bf
P}^{(3)}_{t+1}{\bf V}_{t+1}\end{pmatrix},$ (2)

where ${\bf P}^{(i)}$ denotes the $i$th row of the projection matrix ${\bf
P}$.

###  3.2 Articulation Modeling

Unknowns vs constraints: Similar to NRSfM, we analyse the number of unknowns
and constraints to solve the inverse problem. Given $T$ frames of a video, we
have

$\displaystyle\text{\\# Unknowns}=$ $\displaystyle 3N\quad+\quad$
$\displaystyle 3NT\quad+\quad$ $\displaystyle 6T\quad+\quad\quad$
$\displaystyle(T+2),$ (3) $\displaystyle({\bf\bar{V}})$
$\displaystyle(\Delta{\bf V})$ $\displaystyle({\bf R_{0}},{\bf T_{0}})$
$\displaystyle({\bf K})$

which grows linearly with the number of vertices. Motivated by NRSfM [12] that
uses low-rank shape and motion basis to deal with the exploding solution
space, we seek an expressive but low-rank representation of shape and motion.

Linear-blend skinning: Instead of modeling deformation as per-vertex motion
$\Delta{\bf V_{t}}$ [18, 23, 31], we adopt a linear-blend skinning model (LBS)
[28, 29] to constrain vertex motion by blending $B$ rigid “bone”
transformations $\\{{\bf G_{1}},\cdots,{\bf G_{B}}\\}$, which reduces the
number of parameters and makes optimization easier. Besides bone
transformations, the LBS model defines a skinning weight matrix ${\bf
W}\in\mathbb{R}^{B\times N}$ that attaches the vertices of rest shape vertices
${\bf\bar{V}}$ to the set of bones. Each vertex is transformed by linearly
combining the weighted bone transformations in the object coordinate frame and
then transformed to the camera coordinate frame,

$\footnotesize{\bf V}_{i,t}={\bf G_{0,t}}(\sum\nolimits_{b}{\bf W}_{b,i}{\bf
G}_{b,t}){\bar{\bf V}}_{i}$ (4)

where $i$ is the vertex index, $b$ is the bone index. Unlike A-CSM [28] that
only learns articulation, we learn skinning weights and time-varying bone
transformations jointly.

Parametric skinning model: Inspired by the work on surface editing and local
3D shape learning [15, 46], we model the skinning weights as a mixture of
Gaussians with $B$ components. The probability of assigning vertex $i$ to
component $b$ is given as

$\footnotesize W_{b,i}=Ce^{-\frac{1}{2}({\bf v}_{i}-{\bf J}_{b})^{T}{\bf
Q}_{b}({\bf v}_{i}-{\bf J}_{b})},$ (5)

where ${\bf J}_{b}\in\mathbb{R}^{3}$ is the center of b-th Gaussian, ${\bf
Q}_{b}$ is the corresponding precision matrix that determines the orientation
and radius of a Gaussian, and $C$ is a normalization factor that ensures the
probabilities of assigning a vertex to different Gaussians sum up to one.
${\bf W\rightarrow\\{Q,J\\}}$ is optimized. Note that the mixture of Gaussian
models not only reduces the number of parameters for skinning weights from
$NB$ to $9B$, but also guarantees smoothness, the benefits of which are
empirically validated in our experiments (Tab. LABEL:tab:aba). The number of
shape and motion parameters now becomes

$\displaystyle\text{\\# Unknowns}=$ $\displaystyle 3N\,+\,$ $\displaystyle
6BT\qquad+\quad$ $\displaystyle 9B\quad+\,$ $\displaystyle\quad 6T\quad+\quad$
$\displaystyle(T+2),$ (6) $\displaystyle({\bf\bar{V}})$ $\displaystyle({\bf
G_{1\dots B}})$ $\displaystyle({\bf J},{\bf Q})$ $\displaystyle({\bf
R_{0}},{\bf T_{0}})$ $\displaystyle({\bf K})$

which grows linearly the number of frames and bones.

Figure 3: Coarse-to-fine reconstruction from S0 to S3. The learned centers of
3D Gaussians are shown as colored dots.

###  3.3 Self-supervised Learning from a Video

We exploit rich supervision signals from dense optical flow and raw pixels, as
well as shape and motion regularizers to further constrain the problem.

Reconstruction Losses: The supervision for our analysis-by-synthesis pipeline
includes silhouette loss, optical flow loss, texture loss, and perceptual
loss. Given a pair of rendered outputs $(\hat{S}_{t},\hat{{I}}_{t},\hat{{\bf
u}}_{t})$ and measurements $(S_{t},{I}_{t},{\bf u}_{t})$, the inverse graphics
loss is computed as,

$\displaystyle L_{\textrm{IG}}=$
$\displaystyle\beta_{1}||\hat{S}_{t}^{i}-S_{t}||^{2}_{2}+\beta_{2}\sigma_{t}||\hat{{\bf
u}}_{t}^{i}-{\bf u}_{t}||_{2}$
$\displaystyle+\beta_{3}||\hat{{I}}_{t}^{i}-{I}_{t}||_{1}$ (7)
$\displaystyle+\beta_{4}\text{\emph{pdist}}(\hat{{I}}_{t},{I}_{t})$

where $\\{\beta_{1},\cdots,\beta_{4}\\}$ are weights empirically chosen,
$\sigma_{t}$ is the normalized confidence map for flow measurement, and
_pdist_ $(\cdot,\cdot)$ is the perceptual distance [55] measured by an AlexNet
pretrained on ImageNet. Applying L2 norm loss to optical flow is empirically
better than squared L2 loss, and we hypothesize the reason being that the
former is more tolerant to outliers in the observed flow fields.

Shape and motion regularization: We exploit generic shape and temporal
regularizers to constrain the problem. A Laplacian operator is applied to the
rest mesh to enforce smooth surfaces,

$\displaystyle L_{\textrm{shape}}=||\bar{\bf
V}_{i}-\frac{1}{|N_{i}|}\sum_{j\in N_{i}}\bar{\bf V}_{j}||^{2}.$ (8)

Motion regularization includes an ARAP (as-rigid-as-possible) deformation term
and a least deformation term. The ARAP term encourages natural deformation
[46, 49],

$\displaystyle L_{\textrm{ARAP}}=\sum_{i=1}^{V}\sum_{j\in N_{i}}|\ ||{\bf
V}_{i,t}-{{\bf V}}_{j,t}||_{2}-||{\bf V}_{i,t+1}-{\bf V}_{j,t+1}||_{2}\ |.$
(9)

The least deformation term encourages the deformation from the rest shape to
be small [23],

$\displaystyle L_{\textrm{least-motion}}=\sum_{i=1}^{V}||{\bf
V}_{i,t}-\bar{\bf V}_{i}||_{2},$ (10)

which discourages arbitrarily large deformations and reduces ambiguities in
joint object root body pose and articulation recovery.

Soft-symmetry constraints: To exploit the reflectional symmetry structure
exhibited in common objects, we pose a soft-symmetry constraint along the
symmetry plane $({\bf n}^{*},0)$ at an arbitrary frame $t^{*}$. The symmetry
plane is initialized from visual inspection and jointly optimized. We
encourage the rest shape to be similar to its reflection,

$\displaystyle L_{\textrm{symm-shape}}=L_{\text{cham}}(\\{\bar{\bf V},{\bf
F}\\},\\{{\bf H}\bar{\bf V},{\bf F}\\})$ (11)

where ${\bf H}={\bf I}-2{\bf n}_{*}{\bf n}_{*}^{T}$ is the Householder
reflection matrix, and the Chamfer distance ($L_{\text{cham}}$) is computed as
bidirectional pixel-to-face distances. For the centers of Gaussian control
points ${\bf J}$, we also have

$\displaystyle L_{\textrm{symm-bone}}=L_{\text{cham}}(\bar{\bf J},{\bf
H}\bar{\bf J}).$ (12)

The total loss is a weighted sum of all losses with the weights empirically
chosen and fixed for all experiments.

Figure 4: Visual comparison on near-rigid DAVIS sequences: scooter, soapbox,
and car. COLMAP reconstructs only the visible rigid part, while our method
faithfully reconstruct both the rigid object and near-rigid person.

###  3.4 Implementation Details

Neural basis for time-varying parameters: Instead of optimizing explicit time-
varying parameters $\\{{\bf D_{t}},{\bf K_{t}}\\}$, we parameterize those as
predictions from a convolutional network (ResNet-18 [21]) given an input image
${\bf I}_{t}$,

$\displaystyle\psi_{w}({\bf I}_{t})$ $\displaystyle=({\bf K},{\bf G}_{0},{\bf
G}_{1},{\bf G}_{2},\cdots,{\bf G}_{B})_{t},$ (13)

where one parameter is predicted for focal length, four parameters are
predicted for each bone rotation parameterized by quaternion, and three
numbers are predicted for each translation, adding to $1+7(B+1)$ numbers in
total at each frame. The weights are initialized with ImageNet [13] pre-
training and then optimized by LASR for each test video. Intuitively, the
network learns a joint basis for cameras and poses that is empirically much
easier to optimize than the raw parameters (Tab. LABEL:tab:aba).

Silhouette and flow measurements Our approach assumes that a reliable
segmentation of the foreground object is given, which can be manually
annotated [37], or estimated using instance segmentation and tracking methods
[25, 56]. Our method requires reasonable optical flow estimation, which can be
provided by state-of-the-art flow estimators [48, 53] trained on a mixture of
datasets [1]. Notably, LASR recovers from some bad flow initialization and
obtains better long-term correspondences (Tab. LABEL:tab:PCK).

Coarse-to-fine reconstruction We adopt a coarse-to-fine strategy to
reconstruct high-quality meshes inspired by Point2Mesh [20]. S0: We first
assume a rigid object and optimize the rest shape and cameras $\\{{\bf S},{\bf
G_{0,t}},{\bf K_{t}}\\}$ for 20 epochs. The rest shape is initialized from a
subdivided icosahedron projected onto a sphere. S1-S3: We perform iso-surface
extraction and re-meshing [22] to fix mesh self-intersections and long edges.
After remeshing, the number of vertices and the number of bones increase, as
shown in Fig. 3. The centers of Gaussian control points are initialized by
running K-means on the vertices coordinates. We then jointly optimize all
parameters $\\{{\bf S},{\bf D_{t}},{\bf K_{t}}\\}$ for 10 epochs. The above
procedure is repeated three times (S1-S3).

##  4 Experiments

Setup: Due to the difficulty of obtaining 3D ground truth for nonrigid objects
in the real world, we evaluate 2D keypoint transfer accuracy as a proxy of 3D
reconstruction quality on real videos. We additionally evaluate 3D
reconstruction accuracy on objects with ground-truth meshes.

Figure 5: Keypoint transfer between frame 2 and frame 70 of the camel video.
The distance between tranferred keypoint and target annotation is represented
by the radius of circles. A correct transfer is marked with green and a wrong
transfer is marked with blue. Our method transfers keypoint more accurately
than baselines.

###  4.1 2D Keypoint Transfer on Animal Videos

Dataset: We test our method on an animal video dataset, BADJA [6], which
provides nine real animal videos with 2D keypoint and mask annotations,
derived from the DAVIS video segmentation dataset [37] and online stock
footage. It includes three videos of dogs, two videos of horsejump, and one
video of camel, cow, bear as well as impala. We report quantitative results on
one video per-category and show the reconstruction of the rest in the sup.
mat.

Metric: To approximate the accuracy of 3D shape and articulation recovery, we
adopt percentage of correct keypoint transfer (PCK-T) [23, 28, 54] metric.
Given a reference and target image pair with 2D keypoint annotations, the
reference keypoint is transferred to the target image, and labeled as
“correct” if the transferred keypoint is within some threshold distance
$d_{th}=0.2\sqrt{|S|}$ from the target keypoint, where $|S|$ is the area of
the ground-truth silhouette [6]. In practice, we transfer points by re-
projection from the reference frame to the target frame given the articulated
shape and camera pose estimations. If the back-projected keypoint lies outside
the reconstructed mesh, we re-project its nearest neighbor that intersects the
mesh. The accuracy is averaged over all T(T-1) pairs of frames.

Baselines: We compare with state-of-the-art methods for animal reconstruction
and refer to Tab. LABEL:tab:taxonomy for a taxonomy. SMALST [57] is a model-
based regressor trained for zebras. It takes an image as input and predicts
shape, pose and texture for the SMAL [59] model. UMR [31] is a category-
specific shape estimator trained for several categories, including birds,
horses and other categories that have a large collection of annotated images.
We report the performance of the horse model since the models of other animal
categories are not available. A-CSM [28] learns a category-specific canonical
surface mapping and articulations from an image collection. At test time, it
takes an image as input and predicts the articulation parameters of a rigged
template mesh. It provides 3D templates for 27 animal categories and an
articulation model for horses, which is used throughout the experiments.
SMALify [6] is a model-based optimization approach that fits one of five
categories (including cat, dog, horse, cow and hippo) of SMAL models to a
video or a single image. We provide all the video frames with ground-truth
keypoint and mask annotations. Close to our setup, N-NRSfM [45] trains a
video-specific model for object shape, deformation and camera parameters from
multi-frame optical flow estimations [14]. Finally, we include a detection-
based method, OJA [6], which trains an hourglass network to detect animal
keypoints (indicated by Detector), and post-process the joint cost maps with a
proposed optimal assignment algorithm. The results of PCK are taken from the
paper [6] without recomputing PCK-T.

Table 2: 2D Keypoint transfer accuracy on BADJA. ${}^{\text{(1)}}$Model-based regression. ${}^{\text{(2)}}$Category-specific reconstruction. ${}^{\text{(3)}}$Free-form reconstruction. Methods with† do not reconstruct 3D shape. Results with∗ indicates the method is not designed for such category. Best results are underlined, and bolded if reconstruct a 3D shape.  Method | camel | dog | cows | horse | bear
---|---|---|---|---|---
${}^{\text{(1)}}$SMALST [57] |  49.7∗ |  12.8∗ |  59.7∗ |  10.4∗ |  67.2∗
${}^{\text{(2)}}$A-CSM [28] |  60.2∗ |  24.5∗ |  65.7∗ | 21.5 |  39.7∗
${}^{\text{(2)}}$UMR [31] |  35.1∗ |  38.5∗ |  68.1∗ | 32.4 |  56.9∗
${}^{\text{(3)}}$N-NRSfM [45] | 67.8 | 17.9 | 70.0 | 8.7 | 60.2
${}^{\text{(3)}}$LASR (Ours) | 81.9 | 65.8 | 83.7 | 49.3 | 85.1
${}^{\text{(3)}}$ +Auto-mask | 78.9 | 59.5 | 82.7 | 42.2 | 82.6
†Static | 51.9 | 13.0 | 55.5 | 8.8 | 58.6
†Detector [6] | 73.3 | 66.9 | 89.2 | 26.5 | 83.1
†OJA [6] | 87.1 | 66.9 | 94.7 | 24.4 | 88.9
†Flow-VCN [53] | 47.9 | 25.7 | 60.7 | 14.4 | 63.8

Results: Qualitative results of 3D shape reconstruction are shown in Fig.
LASR: Learning Articulated Shape Reconstruction from a Monocular Video and
Fig. 6, where we compare with UMR, A-CSM and SMALify on the camel, bear and
dog video. Quantitative results of keypoint transfer are shown in Tab.
LABEL:tab:PCK. Given that all 3D reconstruction baselines are category-
specific and might not provide the exact model for some categories (such as
camel), we pick up the best model or template for each animal video. Compared
with 3D reconstruction baselines, LASR is better for all categories, even on
the categories the baselines are trained for (e.g., LASR: 49.3 vs UMR: 32.4 on
horsejump-high). Replacing the GT masks with an object segmentor, PointRend
[25], the performance of LASR (‘+Auto-mask’ in Tab. LABEL:tab:PCK) drops, but
is still better than all the reconstruction baselines. Compared to detection-
based methods, our accuracy is higher on the horsejump video, and close to the
baseline on other videos. LASR also shows a large improvement compared to the
initial optical flow (81.9% vs 47.9% for camel), especially between long-range
frames as shown in Fig. 5.

Figure 6: Comparison on BADJA bear and dog videos. The reconstruction of the first frame of the video is shown from two viewpoints. Compared to UMR that also does not use a shape template, LASR reconstructs more fine-grained geometry. Compared to A-CSM and SMALify that use a shape template, LASR recovers instance-specific details, such as the fluffy tail of the dog, and a more natural pose. Table 3: Mesh reconstruction error in terms of Chamfer distance on our animated object dataset. To ensure comparable results over objects, ground-truth shapes are rescaled such that the maximum distance between vertices is 10. Best results are bolded. “-” means a method does not apply to a particular sequence.  Method |  dancer $\downarrow$ |  dog $\downarrow$ |  horse $\downarrow$ |  golem $\downarrow$
---|---|---|---|---
SMPLify-X [36] | 0.26 | - | - | -
VIBE [26] | 0.22 | - | - | -
A-CSM [28] | - | 0.38 | 0.26 | -
SMALify [6] | - | 0.51 | 0.41 | -
PIFuHD [42] | 0.28 | - | - | -
UMR [31] | - | 0.44 | 0.42 | -
LASR (Ours) | 0.35 | 0.28 | 0.23 | 0.16

###  4.2 Mesh Reconstruction on Articulated Objects

Dataset: To evaluate mesh reconstruction accuracy, we collect a video dataset
of five articulated objects with ground-truth mesh and articulation, including
one dancer video from AMA (Articulated Mesh Animation dataset) [50], one
German shepherd video, one horse video, one eagle video and one stone golem
video from TurboSquid. We also include a rigid object, Keenan’s spot to
evaluate performance on rigid object reconstruction and ablation for the S0
stage.

Metric: Most prior work on mesh reconstruction assumes given camera
parameters. However, both the camera and the geometry are unknown in our case,
which leads to ambiguities in evaluation, including scale ambiguity (exists
for all monocular reconstruction) as well as the depth ambiguity (exists for
weak perspective cameras as used in UMR, A-CSM, VIBE, etc.). To factorize out
the unknown camera matrices, we align two meshes with a 3D similarity
transformation solved by ICP. Then, the bidirectional Chamfer distance is
adopted as the evaluation metric. We follow prior work [16, 38] to randomly
sample 10k points uniformly from the surface of predicted and ground-truth
meshes, and compute the average distance between the nearest neighbor for each
point in the corresponding point cloud.

Baselines: Besides A-CSM, SMALify, and UMR for animal reconstruction, we
compare with SMPLify-X, VIBE, and PiFUHD for human reconstruction. SMPLify-X
[36] is a model-based optimization method for expressive human body capture.
We use the female SMPL model for the dancer sequence, and provide the keypoint
inputs estimated from OpenPose [9]. VIBE [26] is a state-of-the-art model-
based video regressor for human pose and shape inference. PIFuHD is a state-
of-the-art free-form 3D shape estimator for clothed humans. It takes a single
image as input and predicts an implicit shape representation, which is
converted to a mesh by the marching cube algorithm. To compare with SMALify on
dog and horse, we manually annotate 18 keypoints per-frame, and initialize
with the corresponding shape template.

Table 4:  Ablation study with mesh reconstruction error. S0 | ref. |  ${}^{\text{(1)}}$w/o flow |  ${}^{\text{(2)}}$w/o Lcan |  ${}^{\text{(3)}}$w/o CNN
---|---|---|---|---
spot | 0.05 | 0.55 | 0.61 | 0.63
S0-S3 | ref. |  ${}^{\text{(4)}}$w/o LBS |  ${}^{\text{(5)}}$w/o C2F |  ${}^{\text{(6)}}$w/o GMM
dog | 0.28 | 0.68 | 0.59 | 0.34

Results The visual comparison on human and animals are shown in Fig. LASR:
Learning Articulated Shape Reconstruction from a Monocular Video and Fig. 7
respectively. We report the quantitative results in Tab. LABEL:tab:mesh. On
the dog video, our method is better than all the baselines (0.28 vs A-CSM:
0.38), possibly because A-CSM and UMR are not trained specifically for dogs
(although A-CSM uses a wolf template), and SMALify cannot reconstruct a
natural 3D shape from limited keypoint and silhouette annotations. For the
horse video, our method is slightly better than A-CSM, which uses a horse
shape template, and outperforms other baselines. For the dancer sequence, our
method is not as accurate as baseline methods (0.35 vs VIBE: 0.22), which is
expected given that all baselines either use a well-designed human model, or
have been trained with 3D human mesh data, while LASR does not have access to
3D human data. For the stone golem video, our method is the only one that
reconstructs a meaningful shape. Although the stone golem has a similar shape
to a human’s, OpenPose does not detect joints correctly, leading to the
failure of SMALify-X, VIBE and PiFUHD.

Figure 7: Shape and articulation reconstruction results on synthetic dog and
horse sequences. We also visualize Chamfer distances measured from the ground
truth to the reconstructed mesh on top of each result, and yellow indicates
high error. Compared to UMR, LASR successfully reconstructs the four legs of
the horse. Compared to template-based methods (A-CSM and SMALify), LASR
successfully reconstructs the instance-specific details (ears and tails of the
dog) and recovers a more natural articulation. The reference is shown at the
left corner and the template mesh used is shown in the bottom right boxes.
Figure 8: Left: Ablation study on camera and rigid shape optimization (S0).
Removing the optical flow loss introduces large errors in camera pose
estimation and therefore the overall geometry is not recovered. Removing the
canonicalization loss leads to worse camera pose estimation, and therefore the
symmetric shape constraint is not correctly enforced. Finally, if we directly
optimize the camera poses without using a convolutional network, it converges
much slower and does not yield an ideal shape within the same iterations.
Right: Ablation study on articulated shape optimization (S1-S3). We show the
reconstructed articulated shape at the middle frame (t=8) from two viewpoints.
Without LBS model, although the reconstruction looks plausible from the
visible view, it does not recover the full geometry due to the redundant
deformation parameters and lack of constraints. Without coarse-to-fine re-
meshing, fine-grained details are not recovered. Replacing GMM skinning
weights (9xB parameters) with an NxB matrix leads to extra limbs and tails on
the reconstruction.

Qualitative results on DAVIS videos: To examine the performance on arbitrary
real-world objects, we use five DAVIS videos, including dance-twirl, scooter-
board, soapbox, car-turn, mallard-fly, and a cat video captured by us and
segmented by PointRend. The comparison with COLMAP [44], a template-free SfM-
MVS pipeline, is shown in Fig. 4. More results are available in the sup. mat.

Ablation study: We investigate the effect of different design choices on the
rigid “spot” and animated dog sequences. The videos are rendered into T=15
frames given ambient light and a camera rotating around the object by 90
degrees at zero elevation. Besides color images, we render silhouette and
optical flow as the supervision. Results are shown in Fig. 8 and quantitative
results are reported in Tab. LABEL:tab:aba. In terms of camera parameter
optimization and rigid shape reconstruction (S0), we find it beneficial to use
(1)optical flow as supervision signals, (2)canonicalization of symmetry plane,
and (3)CNN as an implicit representation for camera parameters. For
articulated shape reconstruction (S1-S3), it is critical to use (4)linear
blend skinning, (5)coarse-to-fine re-meshing, and (6)parametric skinning
model.

Limitations: Empirically, LASR struggles to estimate surfaces that are not
visible in any input view and fails at heavy occlusions that are missed by
mask annotations. Its efficiency also needs improvement, as it takes less than
one hour for rigid objects and a few hours for nonrigid shapes on a single
GPU.

##  5 Conclusion

We present LASR, a template-free approach for articulated shape reconstruction
from a monocular video. LASR faithfully reconstructs individual objects from
diverse categories (such as human, camel, dog, bear, etc.) without relying on
category-specific shape templates, making it applicable to a wide range of
scenarios. We hope that LASR will enable more progress in articulated shape
reconstruction.

## References

  * [1] Robust vision challenge.  www.robustvision.net, 2020.
  * [2] Antonio Agudo, Melcior Pijoan, and Francesc Moreno-Noguer.  Image collection pop-up: 3d reconstruction and clustering of rigid and non-rigid categories.  In CVPR, pages 2607–2615, 2018.
  * [3] Marc Badger, Yufu Wang, Adarsh Modh, Ammon Perkes, Nikos Kolotouros, Bernd Pfrommer, Marc Schmidt, and Kostas Daniilidis.  3D bird reconstruction: a dataset, model, and shape recovery from a single view.  In ECCV, 2020.
  * [4] Simon Baker, Takeo Kanade, et al.  Shape-from-silhouette across time part i: Theory and algorithms.  IJCV, 62(3):221–247, 2005.
  * [5] Benjamin Biggs, Ollie Boyne, James Charles, Andrew Fitzgibbon, and Roberto Cipolla.  Who left the dogs out: 3D animal reconstruction with expectation maximization in the loop.  In ECCV, 2020.
  * [6] Benjamin Biggs, Thomas Roddick, Andrew Fitzgibbon, and Roberto Cipolla.  Creatures great and smal: Recovering the shape and motion of animals from video.  In ACCV, pages 3–19. Springer, 2018.
  * [7] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J. Black.  Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image.  In ECCV, 2016.
  * [8] Christoph Bregler, Aaron Hertzmann, and Henning Biermann.  Recovering non-rigid 3d shape from image streams.  In CVPR, volume 2, pages 690–696. IEEE, 2000.
  * [9] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields.  TPAMI, 2019.
  * [10] Thomas J Cashman and Andrew W Fitzgibbon.  What shape are dolphins? building 3d morphable models from 2d images.  PAMI, 35(1):232–244, 2012.
  * [11] Christopher B Choy, JunYoung Gwak, Silvio Savarese, and Manmohan Chandraker.  Universal correspondence network.  In NeurIPS. 2016.
  * [12] Yuchao Dai, Hongdong Li, and Mingyi He.  A simple prior-free method for non-rigid structure-from-motion factorization.  IJCV, 107(2):101–122, 2014.
  * [13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, pages 248–255. Ieee, 2009.
  * [14] Ravi Garg, Anastasios Roussos, and Lourdes Agapito.  A variational approach to video registration with subspace constraints.  IJCV, 2013.
  * [15] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser.  Local deep implicit functions for 3d shape.  In CVPR, pages 4857–4866, 2020.
  * [16] Justin Johnson Georgia Gkioxari, Jitendra Malik.  Mesh r-cnn.  ICCV, 2019.
  * [17] Georgia Gkioxari, Jitendra Malik, and Justin Johnson.  Mesh r-cnn.  In ICCV, pages 9785–9795, 2019.
  * [18] Shubham Goel, Angjoo Kanazawa, and Jitendra Malik.  Shape and viewpoints without keypoints.  In ECCV, 2020.
  * [19] Paulo FU Gotardo and Aleix M Martinez.  Non-rigid structure from motion with complementary rank-3 spaces.  In CVPR, pages 3065–3072. IEEE, 2011.
  * [20] Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or.  Point2mesh: A self-prior for deformable meshes.  arXiv preprint arXiv:2005.11084, 2020.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [22] Jingwei Huang, Hao Su, and Leonidas Guibas.  Robust watertight manifold surface generation method for shapenet models.  arXiv preprint arXiv:1802.01698, 2018.
  * [23] Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik.  Learning category-specific mesh reconstruction from image collections.  In ECCV, 2018.
  * [24] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [25] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick.  Pointrend: Image segmentation as rendering.  In CVPR, pages 9799–9808, 2020.
  * [26] Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black.  Vibe: Video inference for human body pose and shape estimation.  In CVPR, June 2020.
  * [27] Chen Kong and Simon Lucey.  Deep non-rigid structure from motion.  In ICCV, pages 1558–1567, 2019.
  * [28] Nilesh Kulkarni, Abhinav Gupta, David F Fouhey, and Shubham Tulsiani.  Articulation-aware canonical surface mapping.  In CVPR, pages 452–461, 2020.
  * [29] John P Lewis, Matt Cordner, and Nickson Fong.  Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation.  In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 165–172, 2000.
  * [30] Xueting Li, Sifei Liu, Shalini De Mello, Kihwan Kim, Xiaolong Wang, Ming-Hsuan Yang, and Jan Kautz.  Online adaptation for consistent mesh reconstruction in the wild.  In NeurIPS, 2020.
  * [31] Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang, and Jan Kautz.  Self-supervised single-view 3d reconstruction via semantic consistency.  ECCV, 2020.
  * [32] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, pages 740–755. Springer, 2014.
  * [33] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li.  Soft rasterizer: A differentiable renderer for image-based 3d reasoning.  ICCV, Oct 2019.
  * [34] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: A skinned multi-person linear model.  ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1–248:16, Oct. 2015.
  * [35] David Novotny, Nikhila Ravi, Benjamin Graham, Natalia Neverova, and Andrea Vedaldi.  C3dpo: Canonical 3d pose networks for non-rigid structure from motion.  In ICCV, pages 7688–7697, 2019.
  * [36] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  In CVPR, 2019.
  * [37] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  In CVPR, pages 724–732, 2016.
  * [38] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  arXiv:2007.08501, 2020.
  * [39] Bernhard Reinert, Tobias Ritschel, and Hans-Peter Seidel.  Animated 3d creatures from single-view video by skeletal sketching.  In Proceedings of Graphics Interface 2016, GI 2016, pages 133–141. Canadian Human-Computer Communications Society / Société canadienne du dialogue humain-machine, 2016.
  * [40] Ignacio Rocco, Mircea Cimpoi, Relja Arandjelović, Akihiko Torii, Tomas Pajdla, and Josef Sivic.  Neighbourhood consensus networks.  In NeurIPS, 2018.
  * [41] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In ICCV, pages 2304–2314, 2019.
  * [42] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In CVPR, 2020.
  * [43] Peter Sand and Seth Teller.  Particle video: Long-range motion estimation using point trajectories.  IJCV, 80(1):72, 2008.
  * [44] Johannes Lutz Schönberger, True Price, Torsten Sattler, Jan-Michael Frahm, and Marc Pollefeys.  A vote-and-verify strategy for fast spatial verification in image retrieval.  In ACCV, 2016.
  * [45] Vikramjit Sidhu, Edgar Tretschk, Vladislav Golyanik, Antonio Agudo, and Christian Theobalt.  Neural dense non-rigid structure from motion with latent space constraints.  In ECCV, pages 204–222. Springer, 2020.
  * [46] Robert W Sumner, Johannes Schmid, and Mark Pauly.  Embedded deformation for shape manipulation.  In ACM SIGGRAPH 2007 papers, pages 80–es. 2007.
  * [47] Narayanan Sundaram, Thomas Brox, and Kurt Keutzer.  Dense point trajectories by gpu-accelerated large displacement optical flow.  In ECCV, pages 438–451. Springer, 2010.
  * [48] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  2020\.
  * [49] Shubham Tulsiani, Nilesh Kulkarni, and Abhinav Gupta.  Implicit mesh reconstruction from unannotated image collections, 2020\.
  * [50] Daniel Vlasic, Ilya Baran, Wojciech Matusik, and Jovan Popović.  Articulated mesh animation from multi-view silhouettes.  In SIGGRAPH 2008, pages 1–9. 2008.
  * [51] Chaoyang Wang, Chen-Hsuan Lin, and Simon Lucey.  Deep nrsfm++: Towards 3d reconstruction in the wild.  arXiv preprint arXiv:2001.10090, 2020.
  * [52] Donglai Xiang, Hanbyul Joo, and Yaser Sheikh.  Monocular total capture: Posing face, body, and hands in the wild.  In CVPR, pages 10965–10974, 2019.
  * [53] Gengshan Yang and Deva Ramanan.  Volumetric correspondence networks for optical flow.  In NeurIPS, pages 794–805, 2019.
  * [54] Yi Yang and Deva Ramanan.  Articulated pose estimation with flexible mixtures-of-parts.  In CVPR, pages 1385–1392. IEEE, 2011.
  * [55] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, pages 586–595, 2018.
  * [56] Tianfei Zhou, Shunzhou Wang, Yi Zhou, Yazhou Yao, Jianwu Li, and Ling Shao.  Motion-attentive transition for zero-shot video object segmentation.  In AAAI, pages 13066–13073, 2020.
  * [57] Silvia Zuffi, Angjoo Kanazawa, Tanya Berger-Wolf, and Michael Black.  Three-d safari: Learning to estimate zebra pose, shape, and texture from images “in the wild”.  In ICCV, pages 5358–5367. IEEE.
  * [58] Silvia Zuffi, Angjoo Kanazawa, and Michael J. Black.  Lions and tigers and bears: Capturing non-rigid, 3D, articulated shape from images.  In CVPR, 2018.
  * [59] Silvia Zuffi, Angjoo Kanazawa, David Jacobs, and Michael J. Black.  3D menagerie: Modeling the 3D shape and pose of animals.  In CVPR, July 2017.

##  6 Appendix

###  6.1 Implementation details

Training details: We include details of the hyper-parameters used for training
in Tab. LABEL:tab:hp.

Video pre-processing: We provide details for video pre-processing. To ensure
enough object motion between adjacent frames, we use a heuristic rule that
skips the next frame when the average magnitude of measured flow within the
object silhouette is lower than 0.05 in the clip space.

###  6.2 Notations

A summary of the notations is listed in Tab. LABEL:tab:notations.

Table 5: Choices of hyper-parameters for training. Name | Value
---|---
Optimization parameters
Network architecture of $\phi_{w}$ |  ResNet-18 (ImageNet-pretrained) [21]
Optimizer |  Adam [24]
Learning rate for $\phi_{w}$ | $1\times 10^{-4}$
Learning rate for other params. | $5\times 10^{-3}$
Batch size | 8 image pairs
Loss weight $\\{\beta_{1},\dots,\beta_{4}\\}$ | $\\{0.5,0.5,2,5\times 10^{-3}\\}$
Measurement pre-processing
Crop center | Center of object bounding box
Crop size |  1.2 $\times$ longest edge
Resized to | $256\times 256$
Table 6: Table of notations used in this work. Symbol | Description
---|---
Numbers
$T$ | Number of frames in the input video
$M$ | Number of faces in the mesh
$N$ | Number of vertices in the mesh
$B$ | Number of bones for LBS
Measurements
$I_{t}$ | Input RGB image at time $t$
$S_{t}$ | Input or measured object silhouette image at time $t$
${\bf u}^{+}_{t}$ | Input or measured forward optical flow map from time $t$ to $t+1$
${\bf u}^{-}_{t}$ | Input or measured backward optical flow map from time $t$ to $t-1$
${\bf Y^{*}}$ | Union of all measurements {$I_{t}$, $S_{t}$, ${\bf u}^{+}_{t}$, ${\bf u}^{-}_{t}$}
Renderings
$\hat{I}_{t}$ | Rendered color image of the object at time $t$
$\hat{S}_{t}$ | Rendered object silhouette image at time $t$
${\bf\hat{u}}^{+}_{t}$ | Rendered forward optical flow map of the object from time $t$ to $t+1$
${\bf\hat{u}}^{-}_{t}$ | Rendered backward optical flow map of the object from time $t$ to $t-1$
${\bf Y}$ | Union of all renderings {$\hat{I}_{t}$, $\hat{S}_{t}$, ${\bf\hat{u}}^{+}_{t}$, ${\bf\hat{u}}^{-}_{t}$}
Variables
$f_{t}$ | Focal length of the camera at time $t$
${\bf K_{t}}$ | Intrinsic matrix of a simple pinhole camera (with zero skew and square pixel) at time $t$
${\bf R_{0,t}}$ | Object root body rotation matrix $\in SO(3)$ at time $t$
${\bf T_{0,t}}$ | Object root body translation vector at time $t$
${\bf G_{0,t}}$ | Object root body transformation at time $t$, ${\bf G_{0,t}}=\begin{pmatrix}[c|c]{\bf R_{0}}&{\bf T_{0}}\end{pmatrix}{\bf{}_{t}}$
${\bf R_{1\dots B,t}}$ | Bone rotations from the rest pose to time $t$
${\bf T_{1\dots B,t}}$ | Bone rotations from the rest pose to time $t$
${\bf G_{1\dots B,t}}$ | Bone transformations from the rest pose to time $t$, ${\bf G_{i,t}}=\begin{pmatrix}[c|c]{\bf R_{i}}&{\bf T_{i}}\end{pmatrix}{\bf{}_{t}},i\in\\{1\dots,B\\}$
${\bf D_{t}}$ | Union of camera and bone transformations $\\{\bf G_{0,t},\dots,G_{B,t}\\}$
${\bf P_{t}}$ | Projection matrix of the camera at time $t$, ${\bf P_{t}}={\bf K_{t}}{\bf G_{0,t}}$
$\Delta{\bf V_{t}}$ | Vertex motion from the rest shape to time $t$
Parameters
$(p_{x},p_{y})$ | Principal point of the camera
${\bf\bar{V}}_{i}$ | Position of the i-th vertex of the mesh in the rest pose (or mean shape)
${\bf\bar{C}}_{i}$ | Color of the i-th vertex of the mesh
${\bf S}$ | Union of all mesh parameters, ${\bf S}=\\{{\bf\bar{V}},{\bf\bar{C}},{\bf F}\\}$
${\bf J}_{b}$ | Position of the center of the b-th bone (or Gaussian component)
${\bf Q}_{b}$ | Precision matrix of b-th bone (or Gaussian component)
${\bf W}$ | Skinning weights matrix, W=$\\{{\bf J},{\bf Q}\\}$
$\phi_{w}$ | Weights of the convolutional camera and pose network
${\bf n^{*}}$ | Normal vector of the symmetry plane in the canonical frame
${\bf X}$ | Union of all Parameters
Constants
${\bf F}$ | Faces of the mesh
${\boldsymbol{\beta}}$ | Weights between the losses
${\bf H}$ | Householder transformation matrix describing reflection about the y-z plane
${\bf n}_{0}$ | Unit vector towards to the x axis
Others
S0 | Training stage 0: optimize for $\\{\phi_{w}({f_{t},\bf G_{0,t}}),p_{x},p_{y},{\bf n}^{*},{\bf\bar{V}},{\bf\bar{C}}\\}$
S1-3 | Training stage 1 to 3: optimize for $\\{\phi_{w}(f_{t},{\bf G_{{\bf 0\dots B},t}}),p_{x},p_{y},{\bf n}^{*},{\bf\bar{V}},{\bf\bar{C}},{\bf J},{\bf Q}\\}$