\begin{table*}[!t]
\centering
\scalebox{0.9}{
\centering
\begin{tabular}{c|c|c}
\hline
& layer description & output dimensions  \\ \hline \hline
layer index & input volume & D $\times$ H $\times$ W $\times$ (64+1) \\ \hline
1-2 & (kernel size 3, out channel 32, stride 1, dilation 1) $\times$ 2 & D $\times$ H $\times$ W $\times$ 32 \\
3 & (kernel size 3, out channel 32, stride 2, dilation 1) & D/2 $\times$ H/2 $\times$ W/2 $\times$ 32 \\
4-5 & (kernel size 3, out channel 32, stride 1, dilation 1) $\times$ 2 & D/2 $\times$ H/2 $\times$ W/2 $\times$ 32 \\
6 & (kernel size 3, out channel 32, stride 2, dilation 1) & D/4 $\times$ H/4 $\times$ W/4 $\times$ 32 \\
7-9 & (kernel size 3, out channel 32, stride 1, dilation 2) $\times$ 3 & D/4 $\times$ H/4 $\times$ W/4 $\times$ 32 \\
10 & (kernel size 3, out channel 64, stride 2, dilation 1) & D/8 $\times$ H/8 $\times$ W/8 $\times$ 64 \\
11-13 & (kernel size 3, out channel 64, stride 1, dilation 2) $\times$ 3 & D/8 $\times$ H/8 $\times$ W/8 $\times$ 64 \\
14 & (kernel size 3, out channel 64, stride 2, dilation 1) & D/16 $\times$ H/16 $\times$ W/16 $\times$ 64 \\
15-17 & (kernel size 3, out channel 64, stride 1, dilation 2) $\times$ 3 & D/16 $\times$ H/16 $\times$ W/16 $\times$ 64 \\
\hline
\end{tabular}
} 
\vspace{3mm}
\caption{Architecture of the geometry branch's sparse 3D CNN. Each convolution layer is followed by 1D batch normalization and ReLU.}
\label{tab:sparse_3d_cnn}
\end{table*}
\author{
Hongsuk Choi$^{*1,3}$
\and
Gyeongsik Moon$^{*2}$
\and
Matthieu Armando$^3$
\and
Vincent Leroy$^3$
\and
Kyoung Mu Lee$^1$
\and
Gr\'egory Rogez$^3$
\and
\\
$^1$ Dept. of ECE \& ASRI, Seoul National University, Korea
\\
$^2$ Meta Reality Labs Research
\hspace{1.1cm}
$^3$ NAVER LABS Europe
\\
\small \texttt {redstonepo@gmail.com, mks0601@fb.com, kyoungmu@snu.ac.kr}\\ \small \texttt {\{matthieu.armando,vincent.leroy,gregory.rogez\}@naverlabs.com}
}

\subsection{Backbone}
First, a backbone extracts image features that will be used to condition the neural radiance field. Similarly to~\cite{yu2021pixelnerf}, this allows MonoNHR to learn priors relevant to human rendering, which ultimately enables our method to work on subjects that were not seen during training.
We use a ResNet18~\cite{he2016deep} as backbone.
It takes a masked human image $\mathbf{I} \in \mathbb{R}^{3 \times 256 \times 256}$ as input and produces a feature map $\mathbf{F} \in \mathbb{R}^{c_p \times 128 \times 128}$, with
$c_p=64$ the feature's channel dimension. An off-the-shelf semantic part segmentation method, PGN~\cite{gong2018instance}, is used to segment the subjects in the images.
The feature map $\mathbf{F}$ is used in two ways:

\noindent\textbf{Mesh feature.} Following NHP~\cite{kwon2021neural}, we sample per-vertex image features from $\mathbf{F}$, given SMPL~\cite{loper2015smpl} mesh vertices.
We project these 3D vertices onto the image plane and use bilinear interpolation to sample the corresponding image features. 
Then, we concatenate the per-vertex image features and the vertices' root joint (~\textit{i.e.}, pelvis)-relative depths, to distinguish the features of both occluded and occluding parts.
The concatenated feature is called the \textcolor[RGB]{98,163,59}{mesh feature} and denoted as \textcolor[RGB]{98,163,59}{$\mathbf{M} \in \mathbb{R}^{M \times (c_p+1)}$}, where $M$ denotes the number of SMPL mesh vertices. We feed these to the geometry branch.

\subsection{Mesh Inpainter}~\label{sec:mesh_inpainter}
This module reconstructs the colors of mesh vertices from a monocular image. 
It is a simple MLP network that consists of two layers. 
It takes the mesh feature $\mathbf{M}$ as input and regresses the vertices' colors $\hat{\mathbf{K}} \in \mathbb{R}^{M \times 3}$.
$\hat{\mathbf{K}}$ is only used during training.
The vertices' GT colors $\mathbf{K}$ are obtained by 1.
projecting the vertices to multi-view images considering visibility, 2. sampling the colors from each image with bilinear interpolation and 3. averaging the visible vertices' sampled colors.
The visibility of vertices are obtained by rasterizing mesh vertices.

\noindent\textbf{Sparse 3D convolution.} 
We work with a sparse 3D volume, in which lie the sparse features of $\mathbf{M}$.
This sparse 3D volume is fed to a sparse 3D convolutional neural network (CNN)~\cite{graham2018spconv} that 
extracts multi-scale feature volumes, denoted mesh volumes.
From each mesh volume, we sample the feature corresponding to $\mathbf{x}$ based on its 3D camera coordinates using trilinear interpolation to get the \textcolor[RGB]{252,22,39}{voxel feature $\mathbf{v} \in \mathbb{R}^{c_v}$}, where
$c_v=192$ is the feature's channel dimension.
The voxel feature $\mathbf{v}$ is used only for the geometry estimation; therefore, it is a \emph{geometry-dedicated feature}.

\section{Reproducibility}
\label{supp:rep}
\subsection{MonoNHR}
\noindent\textbf{Backbone.}
The backbone takes the input image $\mathbf{I} \in \mathbb{R}^{3 \times 256 \times 256}$ and produces the feature map $\mathbf{F} \in \mathbb{R}^{64 \times 128 \times 128}$.
We use ResNet18~\cite{he2016deep} from which we removed the classifier head, as MonoNHR's backbone network architecture.
Following pixelNeRF~\cite{yu2021pixelnerf}, the backbone extracts multiple feature maps at different scales, right after the first pooling operation of ResNet18 and after ResNet 3 layers.
The multi-scale feature maps have the following shapes (channel dimension $\times$ height $\times$ width) :
\\
1. $64 \times 128 \times 128$\\
2. $64 \times 64 \times 64$\\
3. $128 \times 32 \times 32$\\
4. $256 \times 16 \times 16$\\

\noindent\textbf{Mesh inpainter.}
The mesh inpainter takes the mesh feature $\mathbf{M} \in \mathbb{R}^{M \times (64+1)}$ as input and estimates the vertices' colors $\hat{\mathbf{K}} \in \mathbb{R}^{M \times 3}$, where $M$ is the number of SMPL~\cite{loper2015smpl} mesh vertices, which is 6890.
The mesh inpainter consists of two MLP layers with a ReLU~\cite{nair2010rectified} activation between them.
The first layer reduces the channel dimension from (64+1) to 32, and the second layer reduces it to 3, which corresponds to RGB channels.

\noindent\textbf{Geometry branch - Sparse 3D convolution.} 
The geometry branch first builds a sparse 3D volume with the mesh feature $\mathbf{M}$, given a SMPL mesh similarly to NB~\cite{peng2021neural} and NHP~\cite{kwon2021neural}. 
We obtain the 3D bounding box of the SMPL mesh in the input camera's coordinate system with $5cm$ padding along the depth, height, and width dimensions.
The 3D bounding box is transformed into a discretized volume with a voxel size of $5mm \times 5mm \times 5mm$, where the depth, height, and width ($D \times H \times W$) depend on the SMPL mesh.
We calculate the discretized coordinates of mesh vertices, which correspond to the discretized volume's voxels, and fill the voxels with $\mathbf{M}$ to get the sparse 3D volume.
The sparse 3D volume is fed to a sparse 3D CNN~\cite{graham2018spconv}. 
We describe the architecture of this sparse 3D CNN in Table~\ref{tab:sparse_3d_cnn}.
The output features of the 5th, 9th, 13th, and 17th layers are the mesh volumes defined in Section 3.3 of the main manuscript.
We sample the features of 3D query points from each mesh volume and concatenate the sampled features to get the voxel feature $\mathbf{v} \in \mathbb{R}^{192}$, which is the geometry-dedicated feature.
{"value": "3", "type": "Other"} {"value": "6", "type": "Other"} {"value": "10", "type": "Other"} {"value": "14", "type": "Other"}