# S2F2: Self-Supervised High Fidelity Face Reconstruction from Monocular Image

Abdallah Dib Junghyun Ahn∗ Cédric Thébault Philippe-Henri Gosselin Louis
Chevallier

InterDigital R&I Equal contribution

###### Abstract

We present a novel face reconstruction method capable of reconstructing
detailed face geometry, spatially varying face reflectance from a single
monocular image. We build our work upon the recent advances of DNN-based auto-
encoders with differentiable ray tracing image formation, trained in self-
supervised manner. While providing the advantage of learning-based approaches
and real-time reconstruction, the latter methods lacked fidelity. In this
work, we achieve, for the first time, high fidelity face reconstruction using
self-supervised learning only. Our novel coarse-to-fine deep architecture
allows us to solve the challenging problem of decoupling face reflectance from
geometry using a single image, at high computational speed. Compared to state-
of-the-art methods, our method achieves more visually appealing
reconstruction.

Figure 1:  Given a single image, our method achieves appealing 3D face
reconstruction and estimates a dense detailed face geometry, spatially varying
face reflectance (diffuse and specular albedos) and high frequency scene
illumination.

##  1 Introduction

Fast, robust and high fidelity 3D face reconstruction has a wide range of
applications in many domains such as interactive face editing, video-
conferencing, XR, Metaverse applications, and visual effects for movies post-
production. Several approaches such as [1, 2, 3, 4, 5, 6] achieve high
fidelity reconstruction, but require complex hardware setups (multi-view,
lightstage). They are therefore not easily usable in most of the
aforementioned applications. Significant progress was made to achieve high
quality reconstruction from monocular image/video using optimization-based
frameworks. Such methods [7, 8, 9, 10] are generally slow, of limited
robustness and not suitable for interactive scenarios. Also, their
performances in challenging conditions (non-uniform lighting, extreme poses)
are limited.

Deep-based analysis-by-synthesis approaches have been investigated to leverage
the generalization capabilities of machine learning. However, these methods
[11, 12, 13, 14] generally sacrifice reconstruction quality. Methods combining
CNNs with differentiable rendering trained in a self-supervised manner have
been introduced by Tewari et al. [11, 12, 15]. These methods directly regress
the parameters of a statistical morphable model and achieve real-time
performance but fall short of the quality and fidelity because their estimated
geometry and reflectance are bound by the statistical prior space which limits
its generalization regarding the real diversity of face geometry and
reflectance.

More recently, many works aim to improve the realism and fidelity of deep-
based methods by capturing either detailed geometry or reflectance but not
both, which we discuss next. First, to capture detailed geometry, and because
of the complexity of the problem, several methods rely on ground truth dataset
obtained either from multi-view reconstruction setup (and/or lightstage) [16,
17, 18, 19], from synthetic data [20, 21] or from a mixture of both [22, 23].
Feng et al. [24] is the only self-supervised method that captures detailed
geometry. However, this method only captures medium-scale geometry details and
misses high-frequency geometry variations. Additionally, its estimated
reflectance is restricted to the statistical prior space which limits its
generalization regarding the real diversity in face geometry and reflectance.
Second, and to improve the reflectance, Dib et al. [25] combined ray tracing
and self-supervised learning to capture medium-scale reflectance details.
However, this method restricts the estimated geometry to a parametric face
model preventing high-frequency facial details (such as wrinkles, folds…) to
be captured. To our knowledge, there is no existing self-supervised methods
that can jointly estimate detailed geometry and reflectance.

The first contribution of this work, is the introduction of the first self-
supervised method that jointly estimate detailed geometry and reflectance.
This is accomplished via our novel coarse-to-fine architecture, with an
adapted training strategy which allows our method to efficiently solve the
ambiguous and complex problem of separating detailed geometry from reflectance
from a single image taken under uncontrolled lighting conditions.

The second contribution, is the combination, for the first time, of
differentiable ray tracing with vertex-based renderer at training time to
overcome the problem of edge discontinuities of the ray tracing. This allow
our method to benefit from both renderers. On one hand, ray tracing accurately
models self-shadows and on the second hand, the vertex-based renderer
evaluates correctly the whole geometry including boundaries. This leads to a
significant improvement in the estimated geometry compared to Dib et al. [25]
that uses only ray tracing.

Finally, the aforementioned contributions enable to take a big leap forward in
reconstruction quality for self-supervised methods and lead to superior face
reconstruction when compared to recent state-of-the art methods. To our
knowledge, this is the first time a self-supervised method reaches this level
of fidelity and realism. Our robust face attributes estimation (diffuse,
specular and normal) leads to practical applications such as face attribute
editing and relighting.

##  2 Related works

Methods such as [1, 3, 4, 5] deliver high-fidelity face reconstruction from
multi-view or light-stage setup, but they are generally expensive and not
applicable for in-the-wild conditions (many cameras, specific lighting). In
this work, we are interested in face reconstruction/tracking methods that only
use image or video as input and do not require any external hardware setup
beyond the camera. These methods can be split into two categories:
optimization-based and learning-based approaches.
Geometry and reflectance modeling Statistical 3D Morphable Models - 3DMM - is
the main building block for a wide range of optimization-based and learning-
based methods [26, 27, 28, 29]. This statistical model adds a lot of structure
and priors to face reconstruction problem from monocular image or video and
makes it tractable. However, due to the low-dimensional space of 3DMM, subject
specific medium and high frequency geometry and albedo details cannot be
modeled. Additionally, the skin reflectance model of 3DMM can only model the
diffuse albedo and may bake shadows/specularity in the albedo. [30] proposes a
drop-in replacement for the basic lambertian reflectance model of 3DMM
incorporating a diffuse and specular priors. In this work, we base our
reconstruction on the 3DMM geometry with the statistical diffuse and specular
prior of [30] and we train a novel multi-stage deep network to capture fine
diffuse and geometry details.

Most of optimization-based methods like [9, 7, 8, 31, 32, 10, 33, 34] rely on
the same 3DMM parametrization, they provide generally precise reconstruction
at the expense of a high computation cost and are sensitive to difficult
lighting conditions.

Among the learning-based methods, deep convolution neural networks (CNN) are
effective at direct face reconstruction [13, 14, 11, 12, 15, 35, 21, 36, 37,
38, 39, 40, 41]. Tewari et al. [11] proposed the first self-supervised
autoencoder-like method to estimate face attributes based on 3DMM. The
advantage of these self-supervised methods is that they can be trained on
large corpus of unlabeled images. However they generally fall short of
reconstruction precision because of their simplified underlying scene
parameterization (pure-Lambertian BRDF to model skin reflectance and low-order
spherical harmonics to model light). Their inability to model self-shadows is
also a possible reason for their instability under challenging lighting
conditions. Dib et al. [25] proposes a self-supervised method that
significantly improves over these methods and solves many of these
limitations. For instance, it uses a cook-torrance BRDF to model skin
reflectance, and captures personalized albedos outside the statistical prior
space. It also uses a differentiable ray tracing to model self-shadows.
However the reconstructed geometry of their method is still limited by 3DMM
space.
Detailed geometry reconstruction Capturing fine detailed geometry on top of
global face structure is a pre-condition to achieve high fidelity face
reconstruction. However, because of the complexity of the problem, methods
such as [6] uses specific hardware setup which is not applicable in-the-wild.
Others, such Cao et al. [42] relies on ground truth dense data [1], or data
captured using a lightstage (or multi-view) such as [19, 43, 44, 16, 17, 18].
However, acquiring such ground truth data is not always possible.

Optimization based methods like [45, 7, 9] use shape-from-shading [46] to
capture fine geometry details. However these methods may not generalize well
and are computationally expensive.

Some deep-based methods rely partially on synthetic data ([20, 47, 22]) or a
mix of labeled and unlabeled data [23] to capture fine detailed geometry. The
bias introduced by these methods may impede the resulting precision due to the
mismatch with real data distribution. They also do not estimate face
reflectance. Sengupta et al. [21] uses synthetic data to train their network
which estimates a normal map and skin reflectance (limited to pure-lambertian
BRDF) but does not capture high frequency geometry details. More recently,
Feng et al. [24] learns an ’expression-dependent’ displacement model in-the-
wild and is the only method that relies only on unlabeled images for end-to-
end training. However this method only captures medium-frequency displacement
map and their estimated reflectance is restricted to the statistical albedo
prior space which limits their reconstruction quality. To our knowledge, our
method is the first self-supervised method that jointly estimates: geometry at
high frequency, spatially varying personalized skin reflectance with diffuse
and specular albedos and high frequency illumination from a single monocular
image.
Differentiable rendering Differentiable rendering is a key block in the
context of analysis-by-rendering and several implementations exist. Tewari et
al. [11] proposed an efficient vertex-based differentiable rendering that can
only handle pure Lambertian BRDF and cannot capture self-shadows. Works such
as [48] propose a differentiable shadow computation method for this type of
renderer.

Dib et al. [49, 10] introduced a method which uses differentiable ray tracing
for face reconstruction within a classic optimization framework. The key
advantage of ray tracing over vertex-based renderer is the capacity of ray
tracing to handle self-shadows where a visibility mask is calculated for each
surface point with respect to each light during direct illumination pass.
However, differentiable ray tracing is computationally expensive and memory
consuming. Recently, Dib et al. [25] uses differentiable ray tracing in
conjunction with a deep neural architecture for direct face reconstruction. In
this scheme, inference does not incur the speed penalty of ray tracing and
delivers near real-time performance with robust reconstruction in challenging
lighting conditions. Regarding the optimization process, a limitation of ray
tracing is the noise on gradients originating at the objects boundaries as
they are sampled by very few points. Solutions exist but remain expensive
([50, 51]). In this work, we combine a vertex-based renderer with a ray
tracing renderer together with a deep neural architecture that takes advantage
of both. On one hand, ray tracing accurately models self-shadows, and on the
second hand, the vertex-based renderer evaluates the whole geometry including
the edges.

##  3 Method

Figure 2: Our network architecture, trained end-to-end in a self-supervised
manner, estimates face attributes (reflectance and detailed geometry) in a
coarse-to-fine fashion (refer to section 3).

Our goal is to obtain a high fidelity face reconstruction with faithful
separation between reflectance and geometry attributes. To solve this ill-
posed problem, we propose a novel multi-stage deep architecture, wherein
different stages progressively refine the reconstruction.

Our network architecture, depicted in Figure 2, is composed of 3 stages
denoted: ’Coarse’, ’Medium’ and ’Fine’. The ’Coarse’ reconstruction relies on
the statistical geometry and albedo priors space. This base reconstruction
lacks important geometry and albedo (diffuse and specular) details because it
is restricted by the low dimensional space of the underlying model. The
’Medium’ stage improves the previously estimated albedos without the
limitations of the statistical prior. Finally, the ’Fine’ stage adds diffuse
albedo and fine geometry details.

In the next, we introduce the scene attributes used by our formulation and we
describe the network architecture.

###  3.1 Scene attributes

Face geometry Shape identity is modeled using [26, 34]’s statistical face
model, and is given by
$\mathsf{e}=\mathsf{a}_{s}+\mathsf{\Sigma}_{s}\mathsf{\alpha}$. $\mathsf{e}$
is a vector of face geometry vertices with $N$ vertices. The identity-shape
space is spanned by $\mathsf{\Sigma}_{s}\in\mathbb{R}^{3N\times K_{s}}$
composed of $K_{s}=80$ principal components of this space.
$\mathsf{\alpha}\in\mathbb{R}^{K_{s}}$ weights each coefficient of the 3DMM
and $\mathsf{a}_{s}\in\mathbb{R}^{3N}$ is the mean face mesh vertices. We use
linear blendshapes to represent face expressions over the neutral identity
$\mathsf{e}$: $\mathsf{v}=\mathsf{e}+\mathsf{\Sigma}_{e}\mathsf{\delta}$,
where $\mathsf{v}$ is the final vertex position displaced from $\mathsf{e}$ by
blendshape weights vector $\mathsf{\delta}\in\mathbb{R}^{K_{e}}$ and
$\mathsf{\Sigma}_{e}\in\mathbb{R}^{3N\times K_{e}}$ composed of $K_{e}=75$
components of the expression space.
Face reflectance Similar to [25], we use a simplified Cook-Torrance BRDF [52,
53] with a constant roughness term. This BRDF model has the advantage of
modelling specular reflections compared to the pure Lambertian BRDF. For each
vertex, we define a diffuse $\mathsf{c}_{i}\in\mathbb{R}^{3}$ and a specular
$\mathsf{s}_{i}\in\mathbb{R}^{3}$ albedos. The statistical diffuse
$\mathsf{c}\in\mathbb{R}^{3N}$ and specular $\mathsf{s}\in\mathbb{R}^{3N}$
albedos are obtained from statistical prior of [30], where
$\mathsf{c}=\mathsf{a}_{r}+\mathsf{\Sigma}_{r}\mathsf{\beta}$ and
$\mathsf{s}=\mathsf{a}_{b}+\mathsf{\Sigma}_{b}\mathsf{\beta}$ with
$\mathsf{\Sigma}_{r},\mathsf{\Sigma}_{b}\in\mathbb{R}^{3N\times K_{r}}$ are
the PCA for diffuse and specular reflectance with $K_{r}=80$. $\mathsf{a}_{r}$
and $\mathsf{a}_{b}$ are the average skin diffuse and specular reflectance. We
use the same coefficient $\beta$ to sample the diffuse and specular albedos as
in [30].
Illumination Similar to [25], we use nine spherical harmonics (SH) bands to
model light. Dib et al. [10] showed that this high order SH parameterization
provides a better light and shadows estimation when used with ray tracing
compared to the widely used low-order 3 SH bands. An environment map of
$64\times 64$ is derived from SH to use with ray tracing. We define
$\gamma\in\mathbb{R}^{9\times 9\times 3}$ the light coefficients.
Camera We use the pinhole camera model and define
$\phi=\\{\mathsf{T},\mathsf{R}\\}$ as the camera parameters with rotation
$\mathsf{R}\in\mathsf{SO}(3)$ and translation $\mathsf{T}\in\mathbb{R}^{3}$.

###  3.2 Coarse stage

On Figure 2, the network E projects the input image $\mathcal{I}$ into the
latent scene representation followed by a fully connected layer that predicts
the semantic attributes vector
$\chi=\\{\mathsf{\alpha},\mathsf{\delta},\mathsf{\phi},\mathsf{\gamma},\mathsf{\beta}\\}$.
Diffuse $\mathcal{D}$ and specular $\mathcal{S}$ texture representations are
derived from $\beta$. These parameters are fed to a differentiable ray tracer
(DRT) and to a vertex-wise differentiable renderer (DR) to generate two images
$\mathcal{I}_{S}^{1}$ and $\mathcal{I}_{R}^{1}$ respectively. The following
loss function is minimized during training:

$\mathsf{E}_{d}(\chi)+\mathsf{E}_{p}(\mathsf{\alpha},\mathsf{\beta})+\mathsf{E}_{b}(\mathsf{\delta}),$
(1)

where $\mathsf{E}_{d}$ is the data term equal to:

$\mathsf{E}_{d}(\chi)=\mathsf{E}_{ph}^{S}(\chi)+w_{dr}\mathsf{E}_{ph}^{R}(\chi)+w_{lm}\mathsf{E}_{land}(\chi),$
(2)

with $\mathsf{E}_{ph}^{S}$ is the pixel-wise photo-consistency loss between
input and ray traced pixels
$\mathsf{p}_{i},\mathsf{p}_{i}^{S}\in\mathbb{R}^{3}$:

$\mathsf{E}_{ph}^{S}(\chi)=\sum_{i}|\mathsf{p}_{i}^{S}(\chi)-\mathsf{p}_{i}|,$
(3)

where $\mathsf{p}_{i}^{S}=\mathcal{F}(\chi)$, with $\mathcal{F}$, the Monte
Carlo estimator of the rendering equation [54]. $\mathsf{E}_{ph}^{R}$ is the
vertex-based photo-consistency loss between the projected mesh and the
corresponding pixels in the input image, defined as follows:

$\mathsf{E}_{ph}^{R}(\chi)=\sum_{i=1}^{N}|\mathcal{B}(\mathsf{n_{i}},\mathsf{c_{i}},\mathsf{R_{i}})-\mathcal{I}(\Pi\circ\mathsf{C(\mathsf{v}_{i}}))|,$
(4)

where $N$ is the number of vertices, $\mathsf{C(\mathsf{v_{i}}})$ is the
projection of vertex $\mathsf{v_{i}}$ in the real image, equal to:
$\mathsf{R}^{-1}(\mathsf{v_{i}}-\mathsf{T})$. $\Pi$ is the perspective camera
matrix that projects a 3D vertex to a 2D pixel. $\mathcal{B}$ is the final
irradiance equal to the sum of the diffuse and specular terms weighted by the
specular intensity $s_{i}$ (details on $\mathcal{B}$ in supplementary material
section A). $\mathsf{E}_{land}$ is the landmark loss, which measures the
distance between the $L=68$ predicted facial landmarks and the projection of
their corresponding vertex on the input image. These landmarks are obtained
using the landmarks detector of [55]. $\mathsf{E}_{p}$ is the statistical
prior that regularizes against implausible face geometry and reflectance [10].
$\mathsf{E}_{b}(\mathsf{\delta})$ is a soft-box constraint that maintains
$\mathsf{\delta}$ in the range $[0,1]$.
Edge discontinuities Ray tracing can naturally models self-shadows by building
a visibility mask for each surface point with respect to each light. However,
the major drawback of differentiable ray tracing is the discontinuities along
geometry edges ([50, 51]). In fact, when solving for the rendering equation
via Monte Carlo ray tracing [54], very few points are sampled on these areas.
As a result, back-propagation fails to handle geometry edges accurately during
the optimization. Several solutions have been proposed to overcome this
limitation but they are generally very computationally expensive ([48, 51]).
For instance, [51] explicitly samples the geometry edges, which extremely
penalizes the training time as it needs to calculate the silhouette edges of
the geometry. While landmarks are mainly used to guide the training, in the
particular case of ray tracing they can help mitigating the aforementioned
limitation. However the geometry is not as precise as it could be. As an
efficient solution, we introduce in this work a new loss term
$\mathsf{E}_{ph}^{R}$ (eq. 4) which relies on a vertex-based differentiable
renderer. This leads to a more accurate reconstruction, by taking advantage of
ray tracing (which can model self-shadows) and vertex-based rendering (for
better gradients on geometry edges) without significant cost. For instance, it
only takes $370$ ms to process (forward-backward) an image using our method
compared to $42$ seconds for the method of [51].

###  3.3 Medium stage

The albedos (diffuse and specular) and geometry estimated by the previous
stage are bound by the statistical prior space and can only capture low
frequency components of the skin reflectance and geometry. Our goal is to
obtain personalized albedos (outside this space) with detailed geometry.
Estimating these parameters jointly in a self-supervised manner is
challenging. For this, we proceed with a coarse-to-fine strategy and we start
by capturing personalized medium diffuse and specular albedos outside the
statistical prior space. The challenge here is to avoid mixing diffuse and
specular albedos and also to avoid baking unexplained shadows in these
albedos. For this, we use the same technique of Dib et al. [25] which
estimates personalized shadow-free albedos. For this, we train two additional
decoders, $\mathbf{D_{d}}$ and $\mathbf{D_{s}}$, in a self-supervised way,
that estimate diffuse $\Delta_{d}$ and specular $\Delta_{s}$ increments to be
added on top of the previously estimated textures, $\mathcal{D}$ and
$\mathcal{S}$, respectively. The resultant textures, $\mathcal{\hat{D}}$ and
$\mathcal{\hat{S}}$, are used to generate a new image $\mathcal{I}_{S}^{2}$.
We note that the second stage has only access to latent space of E allows this
stage to focus on separating medium diffuse from specular albedo without
worrying about high frequency geometry details that are discarded naturally by
design. We define
$\hat{\chi}=\\{\mathsf{\alpha},\mathsf{\delta},\mathsf{\phi},\mathsf{\gamma},\mathcal{\hat{D}},\mathcal{\hat{S}}\\}$
and we minimize the following loss function:

$\displaystyle\mathsf{E}_{d}(\hat{\chi})+\mathsf{E}_{sc}(\mathcal{\hat{A}},\mathcal{A})+w_{m}\mathsf{E}_{m}(\mathcal{\hat{A}})+w_{b}\mathsf{E}_{b}(\mathcal{\hat{A}}),$
(5)

where
$\mathsf{E}_{sc}(\mathcal{\hat{A}},\mathcal{A})=w_{s}\mathsf{E}_{s}(\mathcal{\hat{A}})+w_{c}\mathsf{E}_{c}(\mathcal{\hat{A}},\mathcal{A})$,
and $\mathcal{A}$ is either $\mathcal{D}$ or $\mathcal{S}$. $\mathsf{E}_{s}$
and $\mathsf{E}_{c}$ are the symmetry and consistency regularizers (similar to
[25]) used to avoid baking residual shadows in the personalized albedos.
$\mathsf{E}_{m}$ is a constraint term that ensures local smoothness at each
vertex, with respect to its first ring neighbors.

###  3.4 Fine stage

While the previous stage allows to obtain more personalized diffuse and
specular albedos, these albedos remain generally blurry and still miss
details. Also the geometry is restricted to the low-dimensional space of 3DMM.
For this, we leverage the U-net based architecture (with skip connections)
which are very efficient at capturing these fine details. First, using the
pose and the geometry produced by the first stage, we project the input image
$\mathcal{I}$ in the uv-space. This projection, denoted as $\mathcal{I}_{p}$,
is passed to two U-net networks, $\mathbf{U_{G}}$ and $\mathbf{U_{D}}$.
$\mathbf{U_{G}}$ predicts a normal map $\hat{\Delta}_{n}$ used to displace the
original normal vectors of the coarse mesh. We denote $\mathcal{\bar{M}}$ the
final normal map used for shading, where each vector $\mathsf{\bar{m}}_{i}$ in
$\mathcal{\bar{M}}$ equal to:
$\mathsf{\bar{m}}_{i}=\mathsf{T}_{i}\bigotimes\mathsf{\bar{n}}_{i}$, with
$\mathsf{\bar{n}}_{i}$ sampled from $\hat{\Delta}_{n}$ and $\mathsf{T}_{i}$ is
a column-wise matrix that stack the original normal $\mathsf{n}_{i}$, tangent
$\mathsf{t}_{i}$ and bi-tangent $\mathsf{b}_{i}$ vectors in the camera
coordinate system (more on normal mapping with ray tracing in [56]).
$\mathbf{U_{D}}$ predicts an increment $\hat{\Delta}_{d}$ that is added to the
estimated diffuse albedo $\mathcal{\hat{D}}$ of the previous stage. We denote
$\mathcal{\bar{D}}$ as the resulting texture.

We define
$\bar{\chi}=\\{\mathsf{\alpha},\mathsf{\delta},\mathsf{\phi},\mathsf{\gamma},\mathcal{\bar{M}},\mathcal{\bar{D}}\\}$
and we train $\mathbf{U_{G}}$ and $\mathbf{U_{D}}$ in a self-supervised manner
by minimizing the following loss function:

$\displaystyle\mathsf{E}_{d}(\bar{\chi})+\mathsf{E}_{sc}(\mathcal{\bar{D}},\mathcal{\hat{D}})+w_{m}^{f}\mathsf{E}_{m}(\mathcal{\bar{A}})+w_{b}^{f}\mathsf{E}_{b}(\mathcal{\bar{A}}),$
(6)

where
$\mathsf{E}_{sc}(\mathcal{\bar{D}},\mathcal{\hat{D}})=w_{s}^{f}\mathsf{E}_{s}(\mathcal{\bar{D}})+w_{c}^{f}\mathsf{E}_{c}(\mathcal{\bar{D}},\mathcal{\hat{D}})$,
and $\mathcal{\bar{A}}$ is either $\mathcal{\bar{D}}$ or $\mathcal{\bar{M}}$.
The regularization terms $\mathsf{E}_{s}$ and $\mathsf{E}_{c}$ play an
important role in avoiding baking unexplained shadows in diffuse texture
$\mathcal{\bar{D}}$ in case our lighting model did not recover the light
correctly. Also these regularizers contribute in producing a good separation
between diffuse and geometry details. However, they sacrifice some albedo
details (please refer to the ablation section 5). Finally, we note that we
experimented using an additional U-net to capture a specular increment
$\hat{\Delta}_{s}$ (similar to the diffuse) but we did not obtain substantial
improvements in the reconstruction quality.

###  3.5 Training strategy

We proceed with the following training strategy. We first train $\mathbf{E}$
(with the fully-connected layer) for 30 epochs, in a non-supervised manner, to
directly regress $\chi$ by minimizing eq. 1. Next, we train $\mathbf{D_{d}}$,
$\mathbf{D_{s}}$, and $\mathbf{E}$ for 10 epochs while minimizing the loss in
eq. 5. We follow the same training strategy proposed by [25] to separate
diffuse and specular albedos, which consists in starting with a high
regularization value of $w_{c}$ for diffuse texture (in eq. 5), and then in
progressively relaxing its value during training to allow for the diffuse
albedo to capture more details. Next, we fix $\mathbf{D_{d}}$ and
$\mathbf{D_{s}}$, then we train $\mathbf{U_{G}},\mathbf{U_{D}}$ and
$\mathbf{E}$ for 5 epochs, to estimate a normal map and an enhanced diffuse
map respectively, by minimizing the photo-consistency loss in eq. 6. To avoid
over-fitting one component, and to obtain a plausible separation between these
attributes, we start with a high weight for $w_{c}^{f}$ and we progressively
relax this constraint to allow $\mathcal{\bar{D}}$ to capture more albedo
details.

Finally, we note that the vertex-based loss in eq. 4 is used in the whole
training process with the goal to assist and guide the pixel-wise photo-
consistency loss of ray tracing (eq. 3) at different stages, so to alleviate
the problem of noisy edge gradients that ray tracing exhibits.

Input Overlay Final Shading Normal Diffuse Specular

Figure 3: Results with challenging face details. From left to right: Input
image, overlay of our final reconstruction on the input image, final
reconstruction, shading, normal, diffuse and specular.

##  4 Results

For training, we use CelebA dataset [57] where images were cropped to
$256\times 256$. Output textures of different networks have the same
resolution as the input image. We implemented our network using PyTorch [58].
Ray tracing is based on the method of [51]. During training, we use 8 samples
per pixels for ray tracing. More implementation details are in supp. material,
section B.

Figure S2F2: Self-Supervised High Fidelity Face Reconstruction from Monocular
Image, Figure 3 and supp. material (section E) show successful face
reconstruction of more than 100 subjects with challenging face details,
extreme lighting conditions, challenging head pose/expression and different
skin type (makeup, beards)… For all these subjects, our method successfully
estimates personalized albedos and captures fine detailed geometry, which
leads to appealing reconstruction at high fidelity. Even in challenging
lighting conditions, our method successfully estimates shadow-free maps
(diffuse, specular and normal). All this, at very high computational speed,
where at inference, our method takes $131$ ms to process an image on a Nvidia
RTX 2080 Ti. We note that while ray tracing penalizes our training time, it is
not needed at inference time and our estimated attributes are compatible with
existing rendering engines. Finally, our robust estimation of scene light,
face reflectance and geometry provides explicit controls over the face
attributes which leads to practical applications such as face attribute
editing (aging) and relighting as shown in supp. material (section E).
Finally, in supp. video, we also show reconstruction on video sequence.

Figure 4: ablation studies (section 5)

##  5 Ablation

Importance of vertex-based renderer In this experiment, we study the
importance of the vertex-based renderer to overcome the problem of noisy edge
gradients of the ray tracer. For this, we trained E by dropping the vertex-
based renderer based loss term (eq 4) from equation 2. We compare the
estimated mesh to the one that use the full energy term (ray tracing + vertex-
wise). The results in Figure 4 a) show that the estimated meshes using both
the vertex-based renderer and ray tracer are more accurate than the ones
obtained using ray tracing only (especially around the mouth edges).
Quantitatively, we evaluate both methods on 100 subjects from Facescape
dataset [19] with various type of facial expressions. We compute the vertex
position error with respect to ground truth mesh, and we obtain 2.288/1.671 mm
(mean error/std deviation) for the ’vertex-based + ray tracing’ compared to
2.831/1.782 mm for ’ray tracing only’ which show that combining ray tracing
with the vertex-based renderer improves the reconstructed geometry. We note
that reason why the improvement may look small is that ’vertex-based + ray
tracing’ aims to improve the geometry on very small area around the edges.
Regularization In this experiment, we study the importance of the symmetry and
consistency regularizers ($\mathsf{E}_{sc}$) used in equation 6 to separate
the diffuse and geometric details faithfully. For this, we trained
$\mathbf{U_{G}}$ and $\mathbf{U_{D}}$ by dropping these two regularizers. For
both subjects in Figure 4 b), without these regularizers, some geometric
details get baked in the albedo and leads to sub-optimal separation. Adding
these regularizers produce more convincing separation. While these
regularizers play an important role in obtaining a correct separation between
diffuse and geometry details, they sacrifice some albedo details.
Multi-Stage reconstruction In this experiment we show the importance of the
’Medium’ and ’Fine’ stages to improve the realism of the ’Coarse’
reconstruction. Figure 4 c) shows the reconstruction obtained from the
’Coarse’ stage with the estimated statistical albedo priors. For the ’Medium’
stage, we show the corresponding reconstruction and the enhanced diffuse and
specular albedos. For the ’Fine’ stage, we show the final reconstruction with
the final diffuse and normal maps. We note that the diffuse albedo in ’Medium’
stage is generally blurry and lacks some details and is significantly enhanced
in the ’Fine’ stage. Also, the detailed geometry captured in the ’Fine’ stage
significantly improves the realism of the final reconstruction.
Quantitatively, we calculate the SSIM between the reconstruction of each stage
and the original input image. On 1000 images, we obtain an average of
$0.89/0.91/\textbf{0.97}$ for the coarse/medium/fine stages respectively
(higher is better). This shows that the ’Fine’ stage significantly improves
the fidelity of the reconstruction.

Figure 5: Comparison against Chen et al. [17], Yang et al. [19] and Feng et
al. [24]

##  6 Comparison

In this section, we compare, qualitatively and quantitatively, our method to
the state-of-the-art methods.

###  6.1 Qualitative comparison

Figure 5 shows comparison against Chen et al. [17], Yang et al. [19] and Feng
et al. [24]. For [17] and [24], results are from authors’ open implementation.
For [19] results are generated by the authors. The methods of [17] and [19]
use ground truth (GT) data to train their generative network while our method
and [24] are self-supervised methods and do not require any GT data. For all
subjects, our method successfully capture most of geometry details especially
for the top subjects that present challenging details. These details are
barely captured or missed by other methods. Also our method shows better
results on the wrinkles formed by the zygomaticus muscles (smile wrinkles
around nose and mouth). Our method has a significantly better shape and
expression recovery than all other methods, especially around the mouth (as
highlighted in red rectangles), where the expression is incorrectly captured
by other methods.

Figure 6: Comparisons against Dib et al. [25] (top subject from [25])

Compared to Dib et al. [25] (Figure 6), our method achieves robustness against
challenging lighting conditions similar to [25] and produces shadow free
albedos (first two subjects). In addition, our method estimates better diffuse
map and captures detailed geometry, while [25] restricts the geometry
reconstruction to the low-dimensional space of 3DMM. This leads to a superior
and high fidelity reconstruction of our method compared to [25] (highlighted
in red rectangle). Finally, for third subject, our method that combine ray
tracing and vertex-based rendering has a better expression recovery around the
mouth than [25] that uses only ray tracing, which also confirms our earlier
ablation study (highlighted in green rectangle).

Figure 7 show comparaison against the method of Abrevaya et al. [23] on
subjects with challenging facial details. The method of Abrevaya et al. [23]
uses a combination of labeled and unlabeled data to train the network that
predicts normal map of the face. It also produces a complete normal map for
the entire head (including eyes and moth interior) while our method restricts
the reconstruction to the frontal region of the face (without eyes and mouth
interior). However, our method captures more facial details (especially
arround the eyes) than [23]. Finally we note that [23] only predicts a normal
map (in camera space and not in uv space), and other face attributes are not
estimated. Our method, estimates rich face attributes maps in uv space
(normal, diffuse and specular), face geometry and scene light.

Figure 7: Comparison against Abrevaya et al. [23]

More comparisons against other recent methods can be found in supp. material,
section C.

###  6.2 Quantitative comparison

Geometric evaluation We first compare our estimated geometry to the state-of-
the-art methods of Chen et al. [17], Feng et al. [24], and Dib et al. [25] on
the Superfaces dataset [59] composed of 20 high resolution 3D ground truth
(GT) face meshes (Table 1 and Figure 8). Table 1 reports, for each method, on
all subjects, the average error $\mu$ and standard deviation $\sigma$ for
vertex position error. As shown in Figure 8, the same mask is used for all
methods. Feng et al.[24] achieves slightly better results than our method on
average error while ours has a smaller standard deviation. Our method achieves
better score than Chen et al. [17] and Dib et al. [25]. As depicted in Figure
8, our approach measures lowest error around the mouth. Finally, we note that
our method, which combines ray tracing with vertex-based renderer, has lower
error than Dib et al.[25] that only uses ray tracing, which again confirms our
earlier ablation study (Section 5).

Table 1: Position error (mean/stdev) on Superfaces dataset [59]

Chen et al. [17] Feng et al. [24] Dib et al. [25] Ours Mean err $\mu$ (mm)
1.847 1.234 1.379 1.287 Stdev $\sigma$ (mm) 1.512 1.206 1.159 1.025

We also evaluate our method on the NoW dataset [60], which is composed of 80
subjects with a total of 1702 images. Results are reported in Table 2. We note
that this dataset only evaluates mesh in neutral pose, so expression accuracy
is not evaluated. Nevertheless, our method achieves competitive results and a
better score than Dib et al. [25]. Feng et al. [24] achieves the top score
(more scores against other methods are on NoW website).

Figure 8: Vertex position error compared to GT mesh of Superfaces dataset
[59].

We note that for both datasets, for our method, the mesh used for comparison
is the base 3DMM geometry estimated by the ”coarse” stage as our method only
estimates a normal map to model the finer details.

Table 2: Position error (median/mean/stdev) on Now dataset [60]

Feng et al. [24] Dib et al. [25] Ours Median err (mm) 1.09 1.26 1.24 Mean err
(mm) 1.38 1.57 1.54 Stdev (mm) 1.18 1.31 1.29

Normal evaluation We also compare our estimated normal map to Chen et al. [17]
and Feng et al. [24] on Emily [61] and on realistic 3D head model [62] which
we call ’Male014’ in the next. Results are reported in Figure 9 and Table 3.
Since each method uses a different UV map parametrization, the comparison was
done on the rendered image and not on the unwrapped texture using the same
mask (as shown in Figure 9). For ’Male014’, our method has the lowest mean
error. For Emily, [24] has a better average error but our method has a lower
standard deviation. Our method scores the best in error percentage under
$20^{\circ},25^{\circ}$ and $30^{\circ}$ for both subjects except for Emily
$20^{\circ}$. We note that the method of Chen et al. [17] does not correctly
recover the mouth expression of both subjects (Figure 9). Finally, the last
column in Figure 9 shows the final reconstruction of our method overlaid on
the input image.

Table 3: Mean angular error (degrees) and percentage of errors below
$20^{\circ},25^{\circ}$ and $30^{\circ}$. (Top-row: Male014; Bottom-row:
Emily)

Name Mean$\pm$Std $<$ 20∘ $<$ 25∘ $<$ 30∘ Ours 22.9$\pm$15.3 51.6$\%$ 67.0$\%$
77.1$\%$ Feng et al. [24] 24.2$\pm$14.5 43.8$\%$ 59.7$\%$ 72.4$\%$ Chen et al.
[17] 26.2$\pm$15.6 39.4$\%$ 55.8$\%$ 68.5$\%$ Ours 16.8$\pm$ 9.9 71.2$\%$
84.7$\%$ 92.8$\%$ Feng et al. [24] 15.2$\pm$12.2 77.2$\%$ 84.5$\%$ 89.4$\%$
Chen et al. [17] 17.0$\pm$13.9 72.4$\%$ 82.7$\%$ 88.2$\%$

##  7 Limitations, Future works and Conclusion

Limitations and Future works First, separating light color from skin color
using a single image is not solved in this work and remains an open challenge,
therefore, our method may sometimes bake some albedo color in the estimated
light, that is why our shading may look reddish sometimes. Second, since we do
not use symmetry and consistency regularizers for the normal map, some shadows
may get baked in the normal map when the method fails to recover to correct
light of the scene. Our experiments show that adding these regularizers can
mitigate this but sacrifices important geometry details. Third, regularizers
used in eq. 6 allow our method to obtain a correct separation between diffuse
albedo and geometry details but this is at the expense of some albedo details
(cf. section 5). Finally, our method cannot handle occlusions, so face props
(such as glasses) get baked in the estimated maps. Work such [63] is important
to tackle this. As future work, our method does not connect expression to the
geometry details as in [24] which is important to obtain realistic animated
rigs. Such a function could leverage the rich representation produced by our
pipeline. Finally, our method estimates view and light dependent face
attributes and can be extended to video/multi-view based reconstruction which
can significantly improve the estimated facial attributes and alleviate a lot
of ambiguity introduced by using a single image only. Some limitations are
shown in supp. material, section D.

Figure 9: Estimated normal map compared to GT normal.

Conclusion In this work, we push to a new level the principle of analysis-by-
rendering within self-supervised learning framework by refining the modelling,
the training as well as the rendering stages. First, by combining, for the
first time, ray tracing with the vertex-based renderer at training time, to
solve the problem of edge-discontinuities of ray tracing which significantly
improves the overall geometry and suggests an improvement on the original ray
tracing algorithm. Second, by introducing a coarse-to-fine deep architecture,
with adapted training strategy, that solve, for the first time, the highly
challenging problem of separating detailed face attributes (reflectance and
geometry) from a single image, within a self-supervised setup, and under
uncontrolled lighting conditions. Compared to recent state-of-the art, our
method achieves superior reconstruction quality and produces more visually
appealing results and define a new baseline for self-supervised monocular face
reconstruction methods ’in-the-wild’.

##  8 Acknowledgements

We would like to thank authors in [24], [17] and [23] for sharing their code
source publicly and authors of [19] for processing our images with their
method.

## References

  * [1] Thabo Beeler, Fabian Hahn, Derek Bradley, Bernd Bickel, Paul Beardsley, Craig Gotsman, Robert W Sumner, and Markus Gross.  High-quality passive facial performance capture using anchor frames.  In ACM Transactions on Graphics (TOG), volume 30, page 75. ACM, 2011\.
  * [2] Chenglei Wu, Kiran Varanasi, Yebin Liu, Hans-Peter Seidel, and Christian Theobalt.  Shading-based dynamic shape refinement from multi-view video under general illumination.  In 2011 International Conference on Computer Vision, pages 1108–1115. IEEE, 2011.
  * [3] Levi Valgaerts, Chenglei Wu, Andrés Bruhn, Hans-Peter Seidel, and Christian Theobalt.  Lightweight binocular facial performance capture under uncontrolled lighting.  ACM Trans. Graph., 31(6):187–1, 2012.
  * [4] Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec.  Multiview face capture using polarized spherical gradient illumination.  In ACM Transactions on Graphics (TOG), volume 30, page 129. ACM, 2011.
  * [5] P Gotardo, J Riviere, D Bradley, et al.  Practical Dynamic Facial Appearance Modeling and Acquisition.  ACM SIGGRAPH Asia, 2018.
  * [6] Jérémy Riviere, Paulo Gotardo, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler.  Single-shot high-quality facial geometry and skin appearance capture.  2020\.
  * [7] Pablo Garrido, Levi Valgaert, Chenglei Wu, and Christian Theobalt.  Reconstructing detailed dynamic face geometry from monocular video.  ACM Transactions on Graphics, 2013.
  * [8] Supasorn Suwajanakorn, Ira Kemelmacher-Shlizerman, and Steven M Seitz.  Total moving face reconstruction.  In European Conference on Computer Vision, pages 796–812. Springer, 2014.
  * [9] Pablo Garrido, Michael Zollhöfer, Dan Casas, Levi Valgaerts, Kiran Varanasi, Patrick Perez, and Christian Theobalt.  Reconstruction of Personalized 3D Face Rigs from Monocular Video.  {ACM} Trans. Graph. (Presented at SIGGRAPH 2016), 35(3):28:1–28:15, 2016.
  * [10] Abdallah Dib, Gaurav Bharaj, Junghyun Ahn, Cédric Thébault, Philippe-Henri Gosselin, Marco Romeo, and Louis Chevallier.  Practical face reconstruction via differentiable ray tracing.  Computer Graphics Forum, 2021.
  * [11] Ayush Tewari, Michael Zollöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Theobalt Christian.  MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction.  In The IEEE International Conference on Computer Vision (ICCV), 2017\.
  * [12] Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Pérez, and Christian Theobalt.  Self-supervised multi-level face model learning for monocular reconstruction at over 250 hz.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2549–2559. IEEE Computer Society, 2018.
  * [13] Luan Tran and Xiaoming Liu.  Nonlinear 3d face morphable model.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7346–7355, 2018.
  * [14] Luan Tran, Feng Liu, and Xiaoming Liu.  Towards high-fidelity nonlinear 3d face morphable model.  In In Proceeding of IEEE Computer Vision and Pattern Recognition, Long Beach, CA, June 2019.
  * [15] A Tewari, F Bernard, P Garrido, G Bharaj, et al.  FML: Face Model Learning from Videos.  CVPR, 2019.
  * [16] S Yamaguchi, S Saito, et al.  High-fidelity Facial Reflectance and Geometry Inference from an Unconstrained Image.  ACM TOG, 2018.
  * [17] Anpei Chen, Zhang Chen, Guli Zhang, Kenny Mitchell, and Jingyi Yu.  Photo-realistic facial details synthesis from single image.  In Proceedings of the IEEE International Conference on Computer Vision, pages 9429–9439, 2019.
  * [18] Alexandros Lattas, Stylianos Moschoglou, Baris Gecer, Stylianos Ploumpis, Vasileios Triantafyllou, Abhijeet Ghosh, and Stefanos Zafeiriou.  Avatarme: Realistically renderable 3d facial reconstruction” in-the-wild”.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 760–769, 2020.
  * [19] Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao.  Facescape: A large-scale high quality 3d face dataset and detailed riggable 3d face prediction.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [20] Matan Sela, Elad Richardson, and Ron Kimmel.  Unrestricted facial geometry reconstruction using image-to-image translation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1576–1585, 2017.
  * [21] Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo, and David W Jacobs.  Sfsnet: Learning shape, reflectance and illuminance of facesin the wild’.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6296–6305, 2018.
  * [22] Xiaoxing Zeng, Xiaojiang Peng, and Yu Qiao.  Df2net: A dense-fine-finer network for detailed 3d face reconstruction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2315–2324, 2019.
  * [23] Victoria Fernández Abrevaya, Adnane Boukhayma, Philip HS Torr, and Edmond Boyer.  Cross-modal deep face normals with deactivable skip connections.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4979–4989, 2020.
  * [24] Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart.  Learning an animatable detailed 3d face model from in-the-wild images.  ACM Transactions on Graphics (TOG), 40(4):1–13, 2021.
  * [25] Abdallah Dib, Cedric Thebault, Junghyun Ahn, Philippe-Henri Gosselin, Christian Theobalt, and Louis Chevallier.  Towards high fidelity monocular face reconstruction with rich reflectance using self-supervised learning and ray tracing.  In The IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [26] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3D faces.  In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 187–194. ACM Press/Addison-Wesley Publishing Co., 1999.
  * [27] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM Transactions on Graphics (ToG), 36(6):194, 2017.
  * [28] Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al.  3d morphable face models-past, present, and future.  ACM Transactions on Graphics (TOG), 39(5):1–38, 2020.
  * [29] Michael Zollhöfer, Justus Thies, Darek Bradley, Pablo Garrido, Thabo Beeler, Patrick Pérez, Marc Stamminger, Matthias Nießner, and Christian Theobalt.  State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications.  2018\.
  * [30] William AP Smith, Alassane Seck, Hannah Dee, Bernard Tiddeman, Joshua B Tenenbaum, and Bernhard Egger.  A morphable face albedo model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5011–5020, 2020.
  * [31] Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler.  An anatomically-constrained local deformation model for monocular face capture.  ACM transactions on graphics (TOG), 35(4):1–12, 2016.
  * [32] Curtis Andrus, Junghyun Ahn, Michele Alessi, Abdallah Dib, Philippe Gosselin, Cédric Thébault, Louis Chevallier, and Marco Romeo.  Facelab: Scalable facial performance capture for visual effects.  In The Digital Production Symposium, DigiPro ’20, New York, NY, USA, 2020. Association for Computing Machinery.
  * [33] Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner.  Face2face: Real-time face capture and reenactment of rgb videos.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2387–2395, 2016.
  * [34] Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Luthi, Sandro Schönborn, and Thomas Vetter.  Morphable face models-an open framework.  In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 75–82. IEEE, 2018.
  * [35] Tatsuro Koizumi and William A. P. Smith.  Look ma, no landmarks! unsupervised, model-based dense face alignment.  In IEEE European Conference on Computer Vision (ECCV), 2020.
  * [36] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [37] Stylianos Ploumpis, Evangelos Ververas, Eimear O’Sullivan, Stylianos Moschoglou, Haoyang Wang, Nick Pears, William Smith, Baris Gecer, and Stefanos P Zafeiriou.  Towards a complete 3d morphable model of the human head.  IEEE transactions on pattern analysis and machine intelligence, 2020\.
  * [38] Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael J Black.  Learning to regress 3d face shape and expression from an image without 3d supervision.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7763–7772, 2019.
  * [39] Xiaoguang Tu, Jian Zhao, Zihang Jiang, Yao Luo, Mei Xie, Yang Zhao, Linxiao He, Zheng Ma, and Jiashi Feng.  Joint 3d face reconstruction and dense face alignment from a single image with 2d-assisted self-supervised learning.  In The IEEE International Conference on Computer Vision (ICCV), 2019\.
  * [40] Jianzhu Guo, Xiangyu Zhu, and Zhen Lei.  3ddfa.  https://github.com/cleardusk/3DDFA, 2018.
  * [41] Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Z Li.  Towards fast, accurate and stable 3d dense face alignment.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [42] Chen Cao, Derek Bradley, Kun Zhou, and Thabo Beeler.  Real-time high-fidelity facial performance capture.  ACM Transactions on Graphics (ToG), 34(4):1–9, 2015.
  * [43] Timur Bagautdinov, Chenglei Wu, Jason Saragih, Pascal Fua, and Yaser Sheikh.  Modeling facial geometry using compositional vaes.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3877–3886, 2018.
  * [44] Loc Huynh, Weikai Chen, Shunsuke Saito, Jun Xing, Koki Nagano, Andrew Jones, Paul Debevec, and Hao Li.  Mesoscopic Facial Geometry Inference Using Deep Neural Networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8407–8416, 2018.
  * [45] Luo Jiang, Juyong Zhang, Bailin Deng, Hao Li, and Ligang Liu.  3d face reconstruction with geometry details from a single image.  IEEE Transactions on Image Processing, 27(10):4756–4770, 2018.
  * [46] Ruo Zhang, Ping-Sing Tsai, James Edwin Cryer, and Mubarak Shah.  Shape-from-shading: a survey.  IEEE transactions on pattern analysis and machine intelligence, 21(8):690–706, 1999.
  * [47] Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel.  Learning detailed face reconstruction from a single image.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1259–1268, 2017.
  * [48] Linjie Lyu, Marc Habermann, Lingjie Liu, Ayush Tewari, Christian Theobalt, et al.  Efficient and differentiable shadow computation for inverse problems.  arXiv preprint arXiv:2104.00359, 2021.
  * [49] Abdallah Dib, Gaurav Bharaj, Junghyun Ahn, Cedric Thebault, Philippe-Henri Gosselin, and Louis Chevallier.  Face reflectance and geometry modeling via differentiable ray tracing.  ACM SIGGRAPH European Conference on Visual Media Production (CVMP), 2019.
  * [50] Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob.  Reparameterizing discontinuous integrands for differentiable rendering.  Transactions on Graphics (Proceedings of SIGGRAPH Asia), 38(6), Dec. 2019.
  * [51] Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen.  Differentiable monte carlo ray tracing through edge sampling.  ACM Trans. Graph., 37(6):222:1–222:11, Dec. 2018.
  * [52] Robert L Cook and Kenneth E. Torrance.  A reflectance model for computer graphics.  ACM Transactions on Graphics (TOG), 1(1):7–24, 1982.
  * [53] Bruce Walter, Stephen Marschner, Hongsong Li, and Kenneth Torrance.  Microfacet models for refraction through rough surfaces.  pages 195–206, 01 2007.
  * [54] James T. Kajiya.  The rendering equation.  In Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’86, pages 143–150, New York, NY, USA, 1986. ACM.
  * [55] Adrian Bulat and Georgios Tzimiropoulos.  How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks).  In International Conference on Computer Vision, 2017.
  * [56] Eric Veach.  Robust Monte Carlo methods for light transport simulation.  Stanford University, 1998.
  * [57] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  * [58] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in PyTorch.  2017\.
  * [59] Stefano Berretti, Alberto Del Bimbo, and Pietro Pala.  Superfaces: A super-resolution model for 3d faces.  In Andrea Fusiello, Vittorio Murino, and Rita Cucchiara, editors, Computer Vision – ECCV 2012. Workshops and Demonstrations, pages 73–82, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg.
  * [60] Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael Black.  Learning to regress 3D face shape and expression from an image without 3D supervision.  In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 7763–7772, June 2019.
  * [61] Emily.  The wikihuman project.  https://vgl.ict.usc.edu/Data/DigitalEmily2/, 2017.  Accessed: 2020-05-21.
  * [62] 3D Scan Store.  Captured assets for digital artists.  www.3dscanstore.com, 2021.  [Online; accessed 14-November-2021].
  * [63] Anh Tuan Tran, Tal Hassner, Iacopo Masi, Eran Paz, Yuval Nirkin, and Gérard Medioni.  Extreme 3d face reconstruction: Seeing through occlusions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3935–3944, 2018.
  * [64] Dhruv Mahajan, Ravi Ramamoorthi, and Brian Curless.  A theory of frequency domain invariants: Spherical harmonic identities for brdf/lighting transfer and image consistency.  IEEE transactions on pattern analysis and machine intelligence, 30(2):197–213, 2007.
  * [65] Xuaner Zhang, Jonathan T. Barron, Yun-Ta Tsai, Xiuming Zhang, Ren Ng, and David E. Jacobs.  Portrait shadow manipulation.  volume 39, 2020.

##  Appendix A Vertex-based implementation

Here we provide the implementation details for the final irradiance
$\mathcal{B}$ used by the vertex-based renderer (please refer to eq. 4 in main
document).

The final irradiance $\mathcal{B}$ is equal to the sum of the diffuse and
specular terms weighted by the specular intensity $s_{i}\in\mathbb{R}$:

$\displaystyle\mathcal{B}(\mathsf{n_{i}},\mathsf{c_{i}},\mathsf{R_{i}})=(1-\mathsf{s_{i}})\cdot\mathcal{B}_{d}(\mathsf{n_{i}},\mathsf{c_{i}})+\mathsf{s_{i}}\cdot\mathcal{B}_{s}(\mathsf{R_{i}})$
(7)

The diffuse irradiance $\mathcal{B}_{d}$, is obtained by multiplying the SH
coefficients, $B_{lm}$ of the light with SH coefficients ($A_{l}$) of the
half-cosine function ([64]):

$\displaystyle\mathcal{B}_{d}(\mathsf{n_{i}},\mathsf{c_{i}})=\mathsf{c_{i}}\cdot\sum_{l=0}^{8}\sum_{m=-l}^{l}A_{l}\cdot
B_{lm}\cdot Y_{lm}(\mathsf{n_{i}})$ (8)

with $\mathsf{c}_{i}\in\mathbb{R}^{3}$ and $\mathsf{n}_{i}\in\mathbb{R}^{3}$
are the diffuse albedo and normal vector for each vertex respectively.

Similarly, The specular irradiance is obtained using a spatial convolution of
the SH light coefficients with the BRDF kernel of the roughness, which is
constant in our simplified Cook-Torrance BRDF model:

$\displaystyle\mathcal{B}_{s}(\mathsf{R_{i}})=\sum_{l=0}^{8}\sum_{m=-l}^{l}S_{l}\cdot
B_{lm}\cdot Y_{lm}(\mathsf{R_{i}}),$ (9)

with $\mathsf{R_{i}}$ is the reflection direction of the viewing vector
$\mathsf{W_{i}}$ with respect to the surface normal, and $S_{l}$ are the SH
coefficients of the BRDF function [64].

##  Appendix B Implementation details

We use PyTorch [58] with Cuda-enabled backend gpu. Two GPUs were used for
training with a total of 24GB of memory. We used Adam optimizer with default
parameters. Celeba dataset [57] is used for training with 3K images kept for
validation. Images are aligned and cropped to a 256x256 resolution. Because
ray tracing is generally slow, it takes 10 hours to do a single epoch.
However, our method does not need ray tracing at test time and achieves fast
inference ($131$ ms to process an image). We use 8 samples per pixel for ray
tracing and a batch size of 8. For $\mathbf{E}$, we use a pre-trained
ResNet-152. The architecture for $\mathbf{D_{d}}$ and $\mathbf{D_{s}}$ is
given in Table 4. For $\mathbf{U_{G}}$ and $\mathbf{U_{D}}$, we use a U-net
with skip connections with a pre-trained vgg16 backbone from
here111https://github.com/mkisantal/backboned-unet. The last layer of
$\mathbf{U_{D}}$, $\mathbf{U_{G}}$, $\mathbf{D_{d}}$ and $\mathbf{D_{s}}$ is
initialized to output zero increment at the beginning of the training.
Landmarks weight $w_{lm}=0.1$, vertex-based renderer weight $w_{dr}=0.5$. For
the ’Medium’ stage, smoothness regularizer $w_{m}=0.0001$ for diffuse and
specular albedos. The symmetry regularizer $w_{s}=20$ for medium diffuse and
specular albedos. Consistency regularizer for specular albedo $w_{c}=0.01$.
For diffuse albedo, we start with a value of $0.2$ and we relax it by a factor
of $2$ at each epoch. For the fine layer, the smoothness regularizer
$w_{m}^{f}=0.0001$ for diffuse and normal maps. The symmetry regularizer
$w{s}^{f}=10$ for the final diffuse. The consistency regularizer, we start
with $w_{c}^{f}=1.0$ and we relax it by a factor of two at each epoch.

Layer | Architecture
---|---
1 | ct(256, 160, 3),bn=ELU,c2d(160, 256, 1),bn,ELU
2 | ct(256, 256, 3),bn,ELU=c2d(256, 128, 3),bn,ELU,c2d(128, 192, 3),bn,ELU
3 | ct(192, 192, 3),bn,ELU,c2d(192, 96, 3),bn,ELU,c2d(96, 128, 3),bn,ELU
4 | ct(128, 128, 3),bn,ELU,c2d(128, 64, 3),bn,ELU,c2d(64, 64, 3),bn,ELU
5 | ct(64, 64, 3),bn,ELU,c2d(64, 32, 3),bn,ELU,c2d(32, 42, 3),bn,ELU
6 | ct(42, 42, 3),bn,ELU,c2d(42, 21, 3),bn,ELU,c2d(21, 3, 3),Tanh

Table 4: Architecture for $\mathbf{D_{d}}$ and $\mathbf{D_{s}}$ (ct:
ConvTranspose2d, c2d: Conv2D, bn:BatchNorm2d (pytorch).

##  Appendix C More qualitative comparison

In this section, we show more qualitative comparison against state-of-the art
methods.

Figure 10 and Figure 11 show comparison results against the method of Sela et
al. [20] and Richardson et al. [47]. Subjects are taken from authors’
manuscript. We note that these two methods only estimate detailed geometry
while our method estimates geometry, reflectance and scene light.

Figure 10: Comparison against Sela et al. [20]. Subjects from authors’
manuscript. Figure 11: Comparison against Richardson et al. [47]. Subjects
from authors’ manuscript.

##  Appendix D Limitations example

Figure 12 shows some limitations of our method. The description of the
examples are as follows:

  * •

Subject a): Under external (foreign) shadows, our method fails to recover the
light and leads to shadows baking in the estimated normal map. Tackling
foreign shadows is a challenging problem, and the method such as Zhang et al.
[65] described about the difficulties of handling this issue. We also note
that our spherical harmonics based lighting model has its own limitations
because it can only model infinite lights.

  * •

Subject b): Separating light color from skin color using a single image
remains challenging and our method does not solve for this. As a result, some
skin color can get baked in the estimated light (represented here as an
environment map).

  * •

Subject c) and d): Occlusions and face props can get baked in the estimated
maps. In case of severe occlusions (subject d), our method may fail to
correctly estimate the geometry (highlighted in red box).

Figure 12: Limitations of our method.

##  Appendix E Results and relighting/aging applications

  * •

Figure 3 and Figure 13 show reconstruction for subjects with challenging
details on the face.

  * •

Figure 14 show reconstruction for subjects under challenging lighting
conditions.

  * •

Figure 15 and Figure 16 show reconstruction for subjects with make-up, beards
and face props.

  * •

Figures 17, 18, 19 and 20 show reconstruction for various subjects with
different ethnicity, skin color, difficult expression, challenging head pose.

For all these subjects, our method generalizes very well and achieves
appealing 3D reconstruction with high fidelity compared to the input image
(first column vs second column). It also successfully estimates meaningful
face attributes with faithful separation between reflectance and detailed
geometry. Even under challenging lighting conditions, our method estimates
plausible face reconstruction and produces shadow-free maps. In the supp.
video, we show animated reconstruction with relighting and also reconstruction
from video sequence.
Figure 21 shows relighting of different subjects under novel lighting
conditions (second and third column). Because our method can successfully
estimates shadow-free face attributes, and faithfully separates reflectance
from detailed geometry, all this allow our method to perform relighting even
for subjects under challenging light (last two subjects in Figure 21). The
last three columns in Figure 21 show the estimated detailed geometry by our
method rendered with OpenGL under different viewing angles.

Another practical applications for our method are face attribute(s) editing,
attribute(s) transfer, aging and de-aging… We show in Figure 22 an ’Aging’
application that consists simply on transferring the estimated normal map from
source (A) to target (B).

Input Overlay Final Shading Normal Diffuse Specular

Figure 13: Results with challenging face details. From left to right: Input
image, overlay of our final reconstruction on the input image, final
reconstruction, shading, normal, diffuse and specular.

Input Overlay Final Shading Normal Diffuse Specular

Figure 14: Results with challenging lighting conditions. From left to right:
Input image, overlay of our final reconstruction on the input image, final
reconstruction, shading, normal, diffuse and specular.

Input Overlay Final Shading Normal Diffuse Specular

Figure 15: Results for subjects with make-up, beards and face props. From left
to right: Input image, overlay of our final reconstruction on the input image,
final reconstruction, shading, normal, diffuse and specular.

Input Overlay Final Shading Normal Diffuse Specular

Figure 16: Results for subjects with make-up, beards and face props. From left
to right: Input image, overlay of our final reconstruction on the input image,
final reconstruction, shading, normal, diffuse and specular.

Input Overlay Final Shading Normal Diffuse Specular

Figure 17: Results for subjects with different ethnicity, skin color,
difficult expression and challenging head pose. From left to right: Input
image, overlay of our final reconstruction on the input image, final
reconstruction, shading, normal, diffuse and specular.

Input Overlay Final Shading Normal Diffuse Specular

Figure 18: Results for subjects with different ethnicity, skin color,
difficult expression and challenging head pose. From left to right: Input
image, overlay of our final reconstruction on the input image, final
reconstruction, shading, normal, diffuse and specular.

Input Overlay Final Shading Normal Diffuse Specular

Figure 19: Results for subjects with different ethnicity, skin color,
difficult expression and challenging head pose. From left to right: Input
image, overlay of our final reconstruction on the input image, final
reconstruction, shading, normal, diffuse and specular.

Input Overlay Final Shading Normal Diffuse Specular

Figure 20: Results for subjects with different ethnicity, skin color,
difficult expression and challenging head pose. From left to right: Input
image, overlay of our final reconstruction on the input image, final
reconstruction, shading, normal, diffuse and specular. Figure 21: Our robust
face attributes estimation gives explicit control over these attributes and
allow for relighting even for subjects under challenging lighting conditions
(last two subjects). The last three columns show the estimated geometry by our
method for each subject rendered with OpenGL under different viewing angles.
Figure 22: Face attributes edition: Our robust face attributes estimation
allow for practical applications such as normal transfer from source (A) to
target (B) which leads to aging/de-aging (here we show aging). Please note the
wrinkles/folds that appears on the aged face.


# Non-Integer-Oversampling Digital Signal Processing for Coherent Passive
Optical Networks

Haide Wang, Ji Zhou, Jinyang Yang, Jianrui Zeng, Weiping Liu, Changyuan Yu,
Fan Li, and Zhaohui Li This work was supported in part by the National Natural
Science Foundation of China (62005102); Hong Kong Scholars Program
(XJ2021018). (Corresponding authors: Ji Zhou, e-mail: zhouji@jnu.edu.cn), (H.
Wang and J. Yang contributed equally to this work).H. Wang, J. Zhou, J. Yang,
J. Zeng, and W. Liu are with the Department of Electronic Engineering, College
of Information Science and Technology, Jinan University, Guangzhou 510632,
China. J. Zhou and C. Yu are with the Department of Electronic and Information
Engineering, The Hong Kong Polytechnic University, Hong Kong.F. Li and Z. Li
are with the Guangdong Provincial Key Laboratory of Optoelectronic Information
Processing Chips and Systems, Sun Yat-sen University, Guangzhou 510275, China
and Southern Marine Science and Engineering Guangdong Laboratory (Zhuhai),
Zhuhai, China.

###### Abstract

Beyond 100G passive optical networks (PONs) will be required to meet the ever-
increasing traffic demand in the future. Coherent optical technologies are the
competitive solutions for the future beyond 100G PON but also face challenges
such as the high computational complexity of digital signal processing (DSP).
A high oversampling rate in coherent optical technologies results in the high
computational complexity of DSP. Therefore, DSP running in a non-integer-
oversampling below 2 samples-per-symbol (sps) is preferred, which can not only
reduce computational complexity but also obviously lower the requirement for
the analog-to-digital converter. In this paper, we propose a non-integer-
oversampling DSP for meeting the requirements of coherent PON. The proposed
DSP working at 9/8-sps and 5/4-sps oversampling rates can be reduced by 44.04%
and 40.78% computational complexity compared to that working at the 2-sps
oversampling rate, respectively. Moreover, a 400-Gb/s-net-rate coherent PON
based on digital subcarrier multiplexing was demonstrated to verify the
feasibility of the non-integer-oversampling DSP. There is almost no penalty on
the receiver sensitivity when the non-integer-oversampling DSP is adopted. In
conclusion, the non-integer-oversampling DSP shows great potential in the
future coherent PON.

##  I Introduction

It is foreseeable that the beyond 100G passive optical networks (PONs) will be
required to meet the ever-increasing traffic demand in the future [1, 2].
Coherent optical technologies are currently moving towards the access network
from core and metro networks, which are the competitive solutions for the
future beyond 100G PON [3, 4]. However, coherent optical technologies have a
high cost for the future 100G PON [5]. Simplified coherent optical
technologies are widely studied, such as a single-polarization heterodyne
receiver and the single Mach-Zehnder modulator at the optical network unit
(ONU) rather than the full coherent transceivers [6]. Furthermore, digital
subcarrier multiplexing (DSCM) is another appealing solution to implement
coherent PON in an efficient and economical manner [7, 8]. Coherent DSCM
technology allows the low-bandwidth transceivers at ONU, which is preferred
for the cost-sensitive PON scenario.

Coherent PON using DSCM can support massive and low-latency connectivity by
providing time-and-frequency division multiple access (TFDMA). A diagram of
the future TFDMA-coherent PON is shown in Fig. 1, where the allocated
subcarriers of the DSCM signal from the optical line terminal (OLT) can be
selected by ONUs. Therefore, the burden of high cost for TFDMA-coherent PON
would be alleviated by using the low-bandwidth transceivers. However, the high
computational complexity of digital signal processing (DSP) is still a great
challenge for TFDMA-coherent PON [9]. To reduce the high computational
complexity, simplified DSP is an important solution [10, 11]. The conventional
adaptive equalizer can be split into a 1-tap butterfly finite impulse response
(FIR) filter and two non-butterfly FIR filters to halve the tap number of
equalizers [12]. In a simplified carrier phase recovery (CPR), the phase
estimation of the X polarization can be shared with the Y polarization, which
removes the phase estimation of the Y polarization [13].

A high oversampling rate in coherent optical technologies increases the number
of samples to be processed, which is another factor that results in the high
computational complexity of DSP [14, 15, 16]. Therefore, the non-integer-
oversampling DSP is another effective solution for coherent PON to reduce
computational complexity. The non-integer-oversampling DSP requires an
oversampling rate below two samples-per-symbol (sps) [17, 18, 19], which
directly lowers the computational complexity by reducing the total number of
processing samples. Moreover, the non-integer-oversampling DSP can lower the
high sampling rate requirement of the analog-to-digital converter (ADC), which
would also reduce the cost and is highly preferred for the ONUs in coherent
PON. However, the non-integer-oversampling DSP for TFDMA-coherent PON using
DSCM still lacks adequate research.

Figure 1: Diagram of the future TFDMA-coherent PON using DSCM, where the
coherent optical technologies are moving from the core and metro networks
towards the access network. The allocated subcarriers of the DSCM signal can
be selected by ONUs using the low-bandwidth transceivers.

In this paper, we propose a non-integer-oversampling DSP to reduce the
computational complexity for TFDMA-coherent PON using DSCM. The non-integer-
oversampling DSP works at an oversampling rate close to 1 sps, which reduces
the computational complexity of DSP as well as the high sampling rate
requirement of the ADC. The main contributions of the work are as follows:

  * •

The non-integer-oversampling DSP is proposed for coherent PON, which mainly
includes the non-integer-oversampling frequency offset estimation (FOE),
timing recovery (TR), frame synchronization, and polyphase fractionally-spaced
multiple-input-multiple-output (MIMO) equalizer.

  * •

A coherent PON using DSCM is demonstrated to verify the feasibility of the
non-integer-oversampling DSP. Compared to the DSP at a 2-sps oversampling
rate, the computational complexity of the proposed DSP at 9/8-sps and 5/4-sps
oversampling rates can be reduced by 44.04% and 40.78% almost without penalty
on receiver sensitivity, respectively.

The remainder of this paper is organized as follows. In Section II, the
principle of non-integer-oversampling DSP is given, which includes the non-
integer-oversampling coarse FOE, TR, frame synchronization and fine FOE,
polyphase fractionally-spaced MIMO equalizer. In Section III, the experimental
setups of an $8$-Gbaud/subcarrier (SC)$\times$8-SCs DSCM-based coherent PON
are introduced, which adopts the proposed non-integer-oversampling DSP. The
experimental results and discussions are given in Section IV. Finally, the
paper is concluded in Section V.

##  II Principle of non-integer-oversampling DSP

In this section, the principle of the proposed non-integer-oversampling DSP
will be introduced, including the coarse FOE, TR, frame synchronization, fine
FOE, MIMO equalizer, and CPR. The non-integer-oversampling coarse FOE and TR
use the frequency-domain (FD) characteristics of the training sequence to
solve the frequency offset and timing error. Since frame synchronization and
fine FOE rely on the time-domain (TD) characteristics of the training
sequence, the subsequent DSP is executed in the TD. Therefore, the FD and TD
algorithms are combined in non-integer-oversampling DSP.

###  II-A Non-integer-oversampling coarse FOE

The coarse FOE is implemented using a training sequence with the two symmetric
frequency tones at $\pm 1/2T$, where $T$ is the symbol period. If there is the
frequency offset $\Delta f$, the two frequency tones will locate at
$1/2T+\Delta f$ and $-1/2T+\Delta f$. Therefore, the coarse FOE is the average
of the two tones. The coarse frequency offset is recovered by a cyclic shift
of the frequency components, which is transformed by discrete Fourier
transform (DFT). The FD algorithms are implemented using the overlap-save
method with an overlap rate $\eta$ $(\eta>1)$.

When the signal is oversampled at a $K/M$-sps oversampling rate, the accuracy
of the coarse FOE can be calculated as

$accuracy=\frac{K}{MNT}$ (1)

where $N$ is the DFT size and $M<K$. Although a lower sampling rate enables
more accurate FOE, the high-frequency components of one sideband would be
filtered due to the frequency offset. For the Nyquist-shaped signal with a
roll-off factor of $\beta$, the minimum oversampling rate without performance
penalty can be calculated as

$K/M_{min}=1+\beta.$ (2)

Therefore, the higher the sampling rate, the greater the frequency offset the
signal can tolerate. Although increasing the size of DFT makes FOE more
accurate, it also makes computational complexity higher. Thus, a small-size
DFT-based FOE requires a fine estimation of the residual frequency offset.

###  II-B Non-integer-oversampling timing recovery

The frequency-domain Godard TR can work at a non-integer oversampling rate
close to 1 sps. The magnitude spectrum $X$ of the signal after match filtering
with a roll-off factor $\beta$ is shown in Fig. 2(a). Fig. 2(b) shows the
spectrum shifted by $(1-M/K)\times N$. Fig. 2(c) shows the spectral parts
$|X_{k}|$, $|X_{k+(1-M/K)N}|$, and their overlap. The multiplication between
the spectral parts is zero except $k\in[d,a]$, which can be calculated as

$d=\frac{(1-\beta)M}{2K}N,$ (3) $a=\frac{(1+\beta)M}{2K}N-1.$ (4)

Therefore, for the non-integer-oversampling Godard timing error detector, the
timing error is estimated as [20, 21]

$\hat{\tau}_{\varepsilon}=\sum_{k=\frac{(1-\beta)M}{2K}N}^{\frac{(1+\beta)M}{2K}N-1}\operatorname{Im}\left[X_{k}\cdot
X_{k+(1-M/K)N}^{*}\right]$ (5)

where $\operatorname{Im}(\cdot)$ denotes the imaginary part of a complex. The
DFT size $N$ should choose a value that allows the upper and lower bounds to
be integers. Then the signal is transferred to the TD by inverse DFT (IDFT)
and the subsequent algorithms are performed in the TD.

Figure 2: (a) Magnitude spectrum of the received signal with a non-integer
oversampling rate. (b) Magnitude spectrum of the received signal shifted by
$M/K\times N$. (c) The overlap of two spectral parts of the spectrum.

###  II-C Frame synchronization and fine FOE

The frame synchronization can be implemented based on the stacked timing
metric after resampling the non-integer-oversampled signal to 1 sps [22]. Then
the synchronization position $p$ of the oversampled signal is obtained by

$p=\lfloor p_{1}\times K/M\rfloor$ (6)

where $\lfloor\cdot\rfloor$ denotes the round-down operation and $p_{1}$ is
the synchronization position of the signal at 1 sps. The fractionally-spaced
delay $D$ caused by the round-down operation can be calculated by

$D=\left[\operatorname{mod}\left(p,K\right)-1\right]/M$ (7)

where $\operatorname{mod}\left(p,K\right)$ denotes the remainder of $p$
divided by $K$.

After the frame synchronization, a fine FOE is implemented to eliminate the
residual frequency offset using a periodic training sequence, which can be
represented as $\left[\bm{S},\bm{S},\bm{S}\right]$. The length of the sequence
$\bm{S}$ is $L_{s}$. The fine FOE can be calculated as [23]

$\displaystyle\Delta f$ $\displaystyle=\frac{R_{s}}{4\pi
L_{s}^{2}}\sum_{k=1}^{L_{s}}\arg\left[R_{2}(k)\times
R_{1}^{*}(k)+R_{3}(k)\times R_{2}^{*}(k)\right]$ (8)
$\displaystyle=\frac{R_{s}}{4\pi
L_{s}^{2}}\sum_{k=1}^{L_{s}}\arg\left[2\left|S(t)|_{t=k+\tau}\right|^{2}\cdot\exp\left(j\frac{2\pi
L_{s}}{R_{s}}\Delta f(k)\right)\right]$

where $R_{s}$ is the baud rate and $\tau$ is a residual sampling phase. The
$\arg(\cdot)$ represents the operation of taking the angle of a complex value
and $(\cdot)^{\mathbf{*}}$ denotes the conjugate operation.
$\left[\bm{R}_{1},\bm{R}_{2},\bm{R}_{3}\right]$ denotes the received training
sequence $\left[\bm{S},\bm{S},\bm{S}\right]$ after TR and the resampling to 1
sps. It will have almost no influence on the fine FOE if there is a residual
sampling phase $\tau$.

Figure 3: Polyphase fractionally-spaced MIMO equalizer. Figure 4: Experimental
setups of (a) OLT and (b) ONU at the $8$-Gbaud/SC$\times 8$-SCs DSCM-based
coherent PON. Inset (i) is the electrical spectrum of the DSCM signal at the
transmitter. Insets (ii) and (iii) are the received spectrum of the allocated
subcarrier 5 and subcarrier 8, respectively. DSCM signal is broadcast to ONU,
which selects the allocated subcarrier by tuning the wavelength of LO.

###  II-D Polyphase fractionally-spaced MIMO equalizer

The fractionally-spaced equalizer banks suffer from poor convergence because
of $M$ independent fractionally-spaced equalizers [24]. Fig. 3 shows the
structure of the polyphase fractionally-spaced MIMO equalizer. The proposed
fractionally-spaced MIMO equalizer consists of two polyphase filters and a
$(M/K\times T)$-spaced MIMO equalizer. The polyphase filters consists of a
bank of $M$ fractional delay filters $C_{k}(z)$ and $k=0,1,\cdots,M-1$.
Polyphase filters compensate for the fractionally delay $D$ caused by frame
synchronization and the fixed delay between the non-integer oversampled
samples and the transmitted symbols [25].

The received signal $R_{\mathrm{X,Y}}\left(n\right)$ is downsampled to 1 sps,
which can be expressed as

$In_{\mathrm{X,Y}}\left(n\right)=R_{\mathrm{X,Y}}\left(\lfloor n\times
K/M\rfloor\right).$ (9)

The output of the polyphase filters using Lagrange’s interpolation on the X or
Y polarization can be expressed as

$p_{\mathrm{X,Y}}(n)=\sum_{i=-L}^{L}C_{k}(i)\cdot
In_{\mathrm{X,Y}}\left(n-i\right)$ (10)

where $L$ is the half-tap length of the polyphase filters. $C_{k}(i)$ is the
tap coefficients of the $i$-th tap of the $k$-th fractional delay filter.
$C_{k}(i)$ can be calculated as

$C_{k}(i)=\prod_{j=-L/2,j\neq i}^{L/2}\frac{[\mu(k)]-j}{i-j}.$ (11)

$k$ can be calculated as

$k=\operatorname{mod}(\lfloor n\times K/M\rfloor,K-1).$ (12)

The fractionally-spaced MIMO equalizer uses $\bm{p_{X,Y}}$ as the input to
compensate for the channel impairments. The output $\bm{Out_{X,Y}}$ of the
proposed fractionally-spaced MIMO equalizer is at the rate of $1/T$, which can
be expressed as

$Out_{X}(n)=\sum_{i=0}^{L_{1}-1}H_{XX}(i)\cdot
p_{\mathrm{X}}(n-i)+H_{XY}(i)\cdot p_{\mathrm{Y}}(n-i),$ (13)
$Out_{Y}(n)=\sum_{i=0}^{L_{1}-1}H_{YY}(i)\cdot
p_{\mathrm{Y}}(n-i)+H_{YX}(i)\cdot p_{\mathrm{X}}(n-i),$ (14)

where $L_{1}$ is the tap length of the MIMO equalizer. The output of the
equalizer is at 1 sps. Finally, CPR is implemented for the equalized signal to
recover the signal.

##  III Experimental setups

The experimental setups of an $8$-Gbaud/SC$\times 8$-SCs DSCM-based coherent
PON are shown in Fig. 4. At the transmitter side, the signal is generated by
using the transmitter (Tx) DSP. Firstly, the bit sequence is mapped to the 16
quadrature amplitude modulation (16QAM) symbols. There are 9137 symbols in one
frame at each subcarrier, including a 416-symbol training sequence (TS) and
273 pilot symbols. The pilot symbols are periodically inserted into the
payload for a low-complexity CPR. The power of each subcarrier is optimized to
balance the signal-to-noise ratio (SNR). Then the signal is pulse shaped by a
root-raised cosine (RRC) filter with a 0.1 roll-off factor and shifted to the
central frequency of each subcarrier. Finally, the DSCM signal is generated by
subcarrier multiplexing and uploaded to a 96 GSa/s-arbitrary waveform
generator (AWG). Inset (i) is the electrical spectrum of the DSCM signal.
After being amplified by the electrical amplifiers (EAs), the analog signal is
modulated by a dual-polarized in-phase/quadrature modulator (DP I/Q-Mod.) on
an optical carrier @ $\sim$1545.123 nm. The output power of the modulator is
about $-14$ dBm. An erbium-doped optical fiber amplifier (EDFA) is used as a
booster amplifier to increase the launch optical power to 10dBm but causes
about $-1.5$ dB penalty on receiver sensitivity. The optical signal is
launched into a 20km standard single-mode fiber (SSMF).

A variable optical attenuator is used at the receiver side to adjust the
received optical power (ROP). A tunable ECL is used as the local oscillator
(LO), which has a linewidth less than 100 kHz and $\sim$12.31 dBm output
power. Insets (ii) and (iii) of Fig. 4 show the received spectrum of the
allocated subcarrier 5 and subcarrier 8, respectively. Each of the eight
subcarriers can be selected by tuning the wavelength of LO. An integrated
coherent receiver (ICR) is used to mix the optical signal with LO and convert
it into an electrical signal and a real-time oscilloscope (RTO, Keysight
UXR0594AP) digitizes the signal. Then the digital signal is resampled to
9/8-sps, 5/4-sps, and 2 sps, respectively. Finally, the received signal is
recovered by the receiver-end DSP, including frame detection, coarse FOE,
match filtering, and TR performed in the FD, while the frame synchronization,
fine FOE, equalization, and CPR in the TD. The line rate of the DSCM-based
coherent PON is 512 Gb/s (i.e. $8$-Gbaud/SC$\times 8$-SCs $\times 4$
bits/symbol/polarization $\times$ 2 polarization $=512$ Gb/s). When the 20%
soft-decision forward error correction (SD-FEC) is considered, the net rate is
approximately 400 Gb/s.

After the frame detection, coarse FOE is implemented. Then the signal is
filtered by an RRC match filter. TR is implemented based on the non-integer-
oversampling Godard timing error detector. After the frame synchronization and
fine FOE, a fractionally-spaced MIMO equalizer is used to compensate for the
channel impairments. The output signal of the equalizer is at 1 sps. The tap
coefficients of the fractionally-spaced MIMO equalizer are converged using the
least mean square (LMS) algorithm. The phase noise $\psi_{\text{Pilot}}$ is
estimated using the periodically inserted pilot symbols and the residual phase
noise $\psi_{\text{Res}}$ is estimated using a $(2Q+1)$-tap average filter at
1 sps. Then the total phase noise $(\psi_{\text{Pilot}}+\psi_{\text{Res}})$ is
compensated for the baud-rate equalized signal and fed to the decision-
directed LMS algorithm to track the tap coefficients, which requires CPR
before the calculation of the error function. Finally, the recovered 16QAM
symbols are de-mapped to the bits.

Figure 5: The estimated timing error without and with the sampling phase
initialization of the TR at (a) 9/8-sps, (b) 5/4-sps, and (c) 2-sps
oversampling rate, respectively.

##  IV Experimental results and discussions

To ensure the upper and lower bounds of Eq. (5) to be integers, the DFT size
$N$ is 252, 250, and 256 for the 9/8-sps, 5/4-sps, and 2-sps oversampling TR,
respectively. The overlap size is set to 54, 50, and 64, which results in the
overlap rate $\eta$ of 1.27, 1.25, and 1.33, respectively. After the detection
of the training sequence with the two symmetric frequency tones at $\pm
1/2T+\Delta f$, the coarse FOE is implemented to compensate for the offset
$\Delta f$ by a cyclic shift of the frequency components. After match
filtering, TR is executed. Fig. 5 shows the estimated timing error without and
with the sampling phase initialization of the TR at (a) 9/8-sps, (b) 5/4-sps,
and (c) 2-sps oversampling rates, respectively. Godard TR algorithm can
estimate the sampling phase $\Delta T$ accurately. An initial sampling phase
is estimated using the same training sequence for coarse FOE, which can
accelerate the convergence of TR [22]. The sampling phase initialization
ensures the validity of frame synchronization, which is performed
subsequently. Under the same symbol length, the lower the oversampling rate,
the fewer samples need to be processed. Therefore, the non-integer-
oversampling TR can reduce the high computational complexity for coherent PON.

The training sequence for frame synchronization follows the training sequence
for frame detection and coarse FOE. As a result, after frame detection and
coarse FOE, the approximate position of the training sequence can be obtained
for frame synchronization. Therefore, only the subsequent $L_{f}$
($L_{f}=100$) symbols are required by the sliding window for frame
synchronization. The training sequence is a periodic sequence with a length of
64 symbols. Frame synchronization is realized by calculating the stacked
timing metric after resampling all the oversampled signals to 1 sps. There are
nine equally spaced peaks in the stacked timing metric as shown in Fig. 6. The
fifth peak is the highest peak without a plateau, which indicates the accurate
synchronization position $p_{1}$. After resampling the 9/8-sps, 5/4-sps, and
2-sps oversampled signals to 1 sps, the stacked timing metrics are almost
identical. Since the sampling phase error can affect the frame
synchronization, the identical timing metrics also mean that the non-integer-
oversampling TR is effective. Then the synchronization position $p$ of the
oversampled signal is calculated as the Eq. (6).

Figure 6: The stacked timing metric calculated at 1 sps for frame
synchronization after the TR at 9/8-sps, 5/4-sps, and 2-sps oversampling rate,
respectively. Figure 7: MSE and BER using fractionally-spaced MIMO equalizers
with 9/8-sps, 5/4-sps, and 2-sps oversampling rates, respectively. Figure 8:
BER performance versus ROP of the coherent PON using 9/8-sps, 5/4-sps, and
2-sps oversampling DSP from subcarrier 1 to subcarrier 8, respectively. The
dashed line represents the 20% SD-FEC limit.

[b]  DFT/IFFT Match Filter Timing Recovery Frame Synchronization Fine FOE MIMO
Equalization Pilot-based CPR Mul. $\eta NK/M$ $\eta K/M$ $\eta(\beta+K/M)$
$(4L_{s}+4)L_{f}/F$ $(4L_{s}+1)/F+K/M$ $(2L+1)$ \+ $2L_{1}$ $P/F+2Q+1$ Add.
$\eta(N-1)K/M$ 0 $\eta(\beta-K/M/N)$ $(3L_{s}+2)L_{f}/F$ $(4L_{s}-1)/F$
$2L+2(L_{1}-1)$ $2Q$ $\eta$: overlap rate, $N$: DFT/IDFT size, $K/M$:
oversampling rate, $\beta$: roll-off factor, $L_{s}$: the size of sliding
window, $L_{f}$: length of symbols for frame synchronization, $F$: symbol
number at one data frame, $L$: the half-tap length of the polyphase filter,
$L_{1}$: tap length of the MIMO equalizer, $P$: number of pilot symbols, $Q$:
half-length of the average filter.

TABLE I: Computational complexity per symbol in terms of complex-valued
multiplication (Mul.) and addition (Add.).

After frame synchronization and fine FOE, the fractionally-spaced MIMO
equalizer is trained using the LMS algorithm. The half-tap length $L$ of the
polyphase filter is 14 and 7 for DSP with 9/8-sps and 5/4-sps oversampling
rates, respectively. For DSP with an integer-oversampling rate, the polyphase
filters are not required, which means that the half-tap length $L$ of the
polyphase filter is zero. The tap length $L_{1}$ of MIMO equalizer for the
9/8-sps, 5/4-sps, and 2-sps oversampling rates are optimized to 9, 9, and 11,
respectively. Fig. 7 shows the mean square error (MSE) of fractionally-spaced
MIMO equalizers and the bit-error rate (BER) using DSP with 9/8-sps, 5/4-sps,
and 2-sps oversampling rates, respectively. Three MSEs can achieve convergence
within approximately 256 training symbols. After equalization, the signal is
downsampled to the baud rate. CPR is implemented at 1 sps using pilot symbols
and an average filter with the half-length $Q$ of $7$. BER can be reflected by
MSE curves. When the training length is approximately 256, the BER after the
non-integer-oversampling DSP does not decrease significantly, which is close
to the BER of the 2-sps oversampling DSP. Therefore, there is almost no
penalty on the BER performance.

Figure 8 shows the BER performance of all the subcarriers versus ROP of the
400-Gb/s-net-rate coherent PON using 9/8-sps, 5/4-sps, and 2-sps oversampling
DSP from subcarrier 1 to subcarrier 8, respectively. The dashed line
represents the 20% SD-FEC limit. Due to the low baud rate of each subcarrier
in TFDMA-coherent PON using DSCM, the relatively low dispersion caused by the
20-km SSMF for each 8-Gbaud subcarrier can be compensated by only the MIMO
equalizer. Therefore, no additional chromatic dispersion compensation module
is required, which can greatly alleviate the DSP burdens for ONUs. BERs of all
the eight subcarriers of the DSCM signal using DSP working at 9/8-sps,
5/4-sps, and 2-sps oversampling rates can achieve below the 20% SD-FEC limit
when the ROP is no less than $-27$ dBm. Using the proposed non-integer-
oversampling DSP, BER can be close to that of the 2-sps oversampling DSP with
almost no penalty on the receiver sensitivity.

The computational complexity per symbol in terms of complex-valued
multiplication (Mul.) and addition (Add.) of the non-integer-oversampling DSP
is listed in Table I. The oversampling sampling rate $K/M$ leads to a
different total number of samples to be processed. Therefore, the main
difference in computational complexity of the DSP with different over-sampling
rates lies in the algorithms related to the oversampling sampling rate $K/M$,
including all FD algorithms and the TD equalization. However, frame
synchronization, fine FOE, and CPR are all implemented in the TD at 1 sps,
which leads to the same computational complexity for those parts of DSP with
different sampling rates. The coarse FOE only requires a cyclic shift of the
frequency components but no multiplication and addition. DFT and IDFT would
account for the major part of the computational complexity of the non-integer-
oversampling DSP. Therefore, reducing the oversampling rate $K/M$ is an
important way to lower the computational complexity of DSP if the DFT size $N$
and the overlap rate $\eta$ are almost the same for all the oversampling
rates.

Figure 9: Computational complexity per symbol of the DSP with 9/8-sps,
5/4-sps, and 2-sps oversampling rates, respectively.

The exact computational complexity per symbol (i.e., the total number of
complex-valued multiplication and addition) of the DSP with 9/8-sps, 5/4-sps,
and 2-sps oversampling rates is shown in Fig. 9. Due to the reduction of the
total number of processing samples, the overall computational complexity per
symbol of the non-integer-oversampling DSP can be greatly reduced. Compared to
the DSP working at a 2-sps oversampling rate, the complex-valued
multiplication of the DSP working at the 9/8-sps and 5/4-sps oversampling rate
is reduced by 44.04% and 40.78%, respectively. Moreover, the complex-valued
addition is reduced by 44.26% and 41.01%, respectively. Therefore, the non-
integer-oversampling DSP can reduce the computational complexity for coherent
PON. The high sampling rate requirement is also lowered, which will reduce the
cost and is highly preferred for ONUs. It is worth mentioning that although
the 9/8-sps oversampling DSP has a lower computational complexity compared to
the 5/4-sps oversampling DSP, it needs more fractional delay filters in the
polyphase filters, which leads to a more complex structure.

##  V Conclusions

In this paper, we propose a non-integer-oversampling DSP to reduce the
computational complexity for TFDMA-coherent PON. The proposed DSP mainly
includes the non-integer-oversampling FOE, TR, frame synchronization, and
polyphase fractionally-spaced MIMO equalizer. Compared to the 2-sps
oversampling rate, the computational complexity of the proposed algorithm
working at 9/8-sps and 5/4-sps oversampling rates can be reduced by 44.04% and
40.78%, respectively. Moreover, the experimental results of a 400-Gb/s-net-
rate coherent PON using DSCM show that there is almost no penalty on the
receiver sensitivity. The proposed non-integer-oversampling DSP can reduce the
computational complexity and lower the high sampling rate requirement of ADC,
which is preferred in coherent PON. To further reduce the computational
complexity of DSP for coherent PON, the simplified non-integer-oversampling
DSP is worth studying. Therefore, the proposed non-integer-oversampling DSP
shows great potential in the future coherent PON.

## References

  * [1] M. S. Faruk, X. Li, D. Nesset, I. N. Cano, A. Rafel, and S. J. Savory, “Coherent passive optical networks: why, when, and how,” _IEEE Communications Magazine_ , vol. 59, no. 12, pp. 112–117, 2021.
  * [2] J. Zhou, J. He, X. Lu, G. Wang, Y. Bo, G. Liu, Y. Huang, L. Li, C. Yang, H. Wang, _et al._ , “100G fine-granularity flexible-rate passive optical networks based on discrete multi-tone with PAPR optimization,” _Journal of Optical Communications and Networking_ , vol. 14, no. 11, pp. 944–950, 2022.
  * [3] J. Zhang and Z. Jia, “Coherent Passive Optical Networks for 100G/$\lambda$-and-Beyond Fiber Access: Recent Progress and Outlook,” _IEEE Network_ , vol. 36, no. 2, pp. 116–123, 2022.
  * [4] L. A. Campos, Z. Jia, H. Zhang, and M. Xu, “Coherent optics for access from P2P to P2MP,” _Journal of Optical Communications and Networking_ , vol. 15, no. 3, pp. A114–A123, 2023.
  * [5] J. Zhou, Q. Sui, and Z. Li, “Non-orthogonal discrete multi-tone: toward higher spectral efficiency for optical networks,” _IEEE Communications Magazine_ , vol. 59, no. 10, pp. 70–75, 2021.
  * [6] Z. Xing, K. Zhang, X. Chen, F. Qiguang, K. Zheng, Y. Zhao, Z. Dong, J. Zhou, T. Gui, Z. Ye, and L. Li, “First Real-time Demonstration of 200G TFDMA Coherent PON using Ultra-simple ONUs,” in _Optical Fiber Communication Conference_. Optica Publishing Group, 2023, pp. Th4C–4.
  * [7] J. Zhang, Z. Jia, H. Zhang, M. Xu, J. Zhu, and L. A. Campos, “Rate-flexible single-wavelength TFDM 100G coherent PON based on digital subcarrier multiplexing technology,” in _2020 Optical Fiber Communications Conference and Exhibition (OFC)_. IEEE, 2020, p. W1E.5.
  * [8] M. Xu, Z. Jia, H. Zhang, L. A. Campos, and C. Knittle, “Intelligent Burst Receiving Control in 100G Coherent PON with 4$\times$ 25G TFDM Upstream Transmission,” in _Optical Fiber Communication Conference_. Optica Publishing Group, 2022, pp. Th3E–2.
  * [9] R. Nagarajan, I. Lyubomirsky, and O. Agazzi, “Low power DSP-based transceivers for data center optical fiber communications (invited tutorial),” _Journal of Lightwave Technology_ , vol. 39, no. 16, pp. 5221–5231, 2021.
  * [10] R. Zhang, Y.-W. Chen, K. Kuzmin, and W. I. Way, “Intra-data center 120Gbaud/DP-16QAM self-homodyne coherent links with simplified coherent DSP,” in _Optical Fiber Communication Conference_. Optica Publishing Group, 2022, pp. W1G–1.
  * [11] J. Song, Y. Li, J. Qiu, X. Hong, H. Guo, Z. Yang, and J. Wu, “Low-complexity FPGA implementation of 106.24 Gbps DP-QPSK coherent optical receiver with fractional oversampling rate based on one FIR filter for resampling, retiming and equalizing,” _Journal of Lightwave Technology_ , 2023.
  * [12] K. Matsuda, R. Matsumoto, and N. Suzuki, “Hardware-efficient adaptive equalization and carrier phase recovery for 100-Gb/s/$\lambda$-based coherent WDM-PON systems,” _Journal of Lightwave Technology_ , vol. 36, no. 8, pp. 1492–1497, 2018.
  * [13] N. Suzuki, H. Miura, K. Mochizuki, and K. Matsuda, “Simplified digital coherent-based beyond-100G optical access systems for B5G/6G,” _Journal of Optical Communications and Networking_ , vol. 14, no. 1, pp. A1–A10, 2022.
  * [14] N. Kaneda and A. Leven, “Coherent polarization-division-multiplexed QPSK receiver with fractionally spaced CMA for PMD compensation,” _IEEE Photonics technology letters_ , vol. 21, no. 4, pp. 203–205, 2008.
  * [15] T. Duthel, C. Fludger, D. van den Borne, C. Schulien, E.-D. Schmidt, T. Wuth, E. De Man, G. Khoe, and H. de Waardt, “Impairment tolerance of 111Gbit/s POLMUX-RZ-DQPSK using a reduced complexity coherent receiver with a T-spaced equaliser,” in _33rd European Conference and Exhibition of Optical Communication_. VDE, 2007, pp. 1–2.
  * [16] S. J. Savory, “Digital coherent optical receivers: Algorithms and subsystems,” _IEEE Journal of selected topics in quantum electronics_ , vol. 16, no. 5, pp. 1164–1179, 2010.
  * [17] C. Malouin, B. Zhang, A. Wagner, S. Khatana, E. Ibragimov, H. Jiang, and T. Schmidt, “Sub-rate sampling in 100 Gb/s coherent optical receivers,” in _2010 Conference on Optical Fiber Communication (OFC/NFOEC), collocated National Fiber Optic Engineers Conference_. IEEE, 2010, p. OThT3.
  * [18] B. Baeuerle, A. Josten, M. Eppenberger, D. Hillerkuss, and J. Leuthold, “Low-complexity real-time receiver for coherent Nyquist-FDM signals,” _Journal of Lightwave Technology_ , vol. 36, no. 24, pp. 5728–5737, 2018\.
  * [19] M. Arikawa and K. Hayashi, “Frequency-domain adaptive MIMO filter with fractional oversampling using stochastic gradient descent for long-haul transmission over coupled 4-core fibers,” _Optics Express_ , vol. 31, no. 8, pp. 13 104–13 124, 2023.
  * [20] A. Josten, B. Baeuerle, E. Dornbierer, J. Boesser, D. Hillerkuss, and J. Leuthold, “Modified Godard timing recovery for non-integer oversampling receivers,” _Applied Sciences_ , vol. 7, no. 7, p. 655, 2017.
  * [21] A. Josten, B. Bäuerle, M. Eppenberger, E. Dornbierer, D. Hillerkuss, and J. Leuthold, “168 Gb/s line rate real-time PAM receiver enabled by timing recovery with 8/7 oversampling in a single FPGA,” in _Optical Fiber Communication Conference_. Optical Society of America, 2017, pp. W2A–24.
  * [22] H. Wang, J. Zhou, Z. Xing, Q. Feng, K. Zhang, K. Zheng, X. Chen, T. Gui, L. Li, J. Zeng, _et al._ , “Fast-Convergence Digital Signal Processing for Coherent PON using Digital SCM,” _Journal of Lightwave Technology_ , 2023\.
  * [23] M. Morelli and U. Mengali, “An improved frequency offset estimator for OFDM applications,” in _1999 IEEE Communications Theory Mini-Conference (Cat. No. 99EX352)_. IEEE, 1999, pp. 106–109.
  * [24] S. F. A. Shah, L. Wang, C. Li, and Z. Zhang, “Low-complexity design of noninteger fractionally spaced adaptive equalizers for coherent optical receivers,” _IEEE Signal Processing Letters_ , vol. 23, no. 9, pp. 1289–1293, 2016.
  * [25] S. F. A. Shah, C. Li, and Z. Zhang, “Low-complexity fractionally spaced equalizer with non-integer sub-symbol sampling for coherent optical receivers,” in _Signal Processing in Photonic Communications_. Optical Society of America, 2015, pp. SpT3D–5.