Kögel, Ibrahim, Kallies, Findeisen

Rolf Findeisen, ,
TU Darmstadt, Germany

# Safe Hierarchical Model Predictive Control and Planning for Autonomous
Systems

Markus Kögel  Mohamed Ibrahim  Christian Kallies  Rolf Findeisen  Laboratory
for Systems Theory and Automatic Control, Otto von Guericke University,
Magdeburg, Germany  Institute of Flight Guidance,
German Aerospace Center (DLR), Braunschweig, Germany  Control and Cyber-
Physical Systems Laboratory, Technical University of Darmstadt, Germany
<EMAIL_ADDRESS>
(26 January 2022; 2022; 2022)

###### Abstract

[Abstract]Planning and control for autonomous vehicles usually are
hierarchical separated. However, increasing performance demands and operating
in highly dynamic environments requires an frequent re-evaluation of the
planning and tight integration of control and planning to guarantee safety. We
propose an integrated hierarchical predictive control and planning approach to
tackle this challenge. Planner and controller are based on the repeated
solution of moving horizon optimal control problems. The planner can choose
different low-layer controller modes for increased flexibility and performance
instead of using a single controller with a large safety margin for collision
avoidance under uncertainty. Planning is based on simplified system dynamics
and safety, yet flexible operation is ensured by constraint tightening based
on a mixed-integer linear programming formulation. A cyclic horizon tube-based
model predictive controller guarantees constraint satisfaction for different
control modes and disturbances. Examples of such modes are a slow-speed
movement with high precision and fast-speed movements with large uncertainty
bounds. Allowing for different control modes reduces the conservatism, while
the hierarchical decomposition of the problem reduces the computational cost
and enables real-time implementation. We derive conditions for recursive
feasibility to ensure constraint satisfaction and obstacle avoidance to
guarantee safety and ensure compatibility between the layers and modes.
Simulation results illustrate the efficiency and applicability of the proposed
hierarchical strategy.

###### keywords:

Model Predictive Control, Planning, Hierarchies, Safety, Autonomous Systems,
Obstacle avoidance

††articletype: Research Article

##  1 Introduction

Autonomous vehicles, such as drones, mobile robots, or autonomous
transportation systems, are by now used in a wide range of applications,
spanning from geo-surveillance, agricultural application, logistics, or search
and rescue operations 1, 2. Often, the autonomous vehicle has to achieve a
task, like to move from a start to a goal position, while avoiding obstacles
in a dynamically changing environment, compare Figure 1.

Figure 1: Planning and control for autonomous vehicles in uncertain,
dynamically changing environments, e.g., moving obstacles or wind, with
collision avoidance.

Safe operation, i.e., collision avoidance, under all circumstances is
challenging 3, 4, 5. Typically, the task is hierarchically separated into a
planning task performed once or repeatedly on a slow time scale. It provides a
reference or path to a lower-layer control layer that operates on a fast time
scale to counteract disturbances, model uncertainty, or react to (fast)
environmental changes, c.f. Figure 2 (a), see also. 6, 7, 8, 9, 10, 11, 12, 13

Many planning approaches for autonomous vehicles exist, see, e.g., 1, 14, 15,
16 and references their-in. They, for example, exploit deterministic or
heuristic search strategies, or reformulate the problem as a mathematical
optimization problem. However, most of them do not allow considering detailed
vehicle dynamics, environmental conditions, or disturbances explicitly.

((a)) Typical decomposition of planning and control.

((b)) Safe planning by constraint tightening/constraint back off considering a
single, fixed tracking control mode.

Figure 2: (a) Hierarchical separation of the planning and control layers; (b)
Safe planning with a single tracking control mode.

Counteracting significant uncertainties, e.g., due to changing environmental
conditions like wind, or operating in a dynamically evolving environment,
requires frequent replanning and tightly integrating control and planning.
This has attracted significant research over the past years: A multi-rate
hierarchical approach consisting of three control layers operating at
different sampling times is presented in 6. A Markov decision problem is
exploited on the highest planning layer, which feeds into the
planning/reference generation layer, exploiting a model predictive control
formulation. At the lowest layer, a tracking controller is considered that
uses a barrier function approach. In 7 an approach to co-design the planner
and control is proposed, which allows considering vehicle dynamics and
kinematic constraints. The tracking error of the controller is directly
considered in the planner. A multi-layer control framework was presented in
10, where the optimization-based reference planner used a low fidelity model.
At the same time, a feedback controller tracks the planned trajectory with a
specific error bound. The state and input constraints are taken into account
in the planning layer. The tracking control law and its error bounds are
parameterized and computed offline using sum-of-squares programming. The
approach allows maintaining safety by taking the tracking error bound into
account to achieve maximum permissiveness of the planning layer. A two-layer
model predictive control framework is presented in 12, where the upper-layer
controller determines the state trajectories over the entire mission while
guaranteeing the satisfaction of the system constraint during operation. The
lower-layer controller modifies the determined trajectories to improve
reference tracking. In 13, a stable hierarchical model predictive control
(MPC) is introduced using an inner loop reference model and contracting
constraint sets for guaranteeing the overall stability. The inner-loop
controller tracks the output of a prescribed reference model.

((a)) Proposed moving horizon planning and control exploiting multiple
controller operation regions/modes.

((b)) Reduced conservatism due to different tracking controller operation
regions/modes.

Figure 3: Proposed planning and control scheme with multiple tracking
controller operation regions/modes.

We propose a tightly integrated hierarchical planning and control approach to
reduce conservatism while being computational feasible. Both planner and
controller are based on the repeated solution of moving horizon optimal
control problems. In particular, both layers are based on robust MPC 17, 18,
19, 20, 21 formulations. The planning- and the lower-layer controller agree on
“contracts” (safety corridors), guaranteeing consistency and compatibility
between the layers. Consequently, this ensures robust satisfaction of the
constraints such as collision avoidance even in the presence of disturbances.
The planner can choose different operating modes, corresponding to different
accuracy’s that the lower-layer controller can achieve in the closed-loop. In
comparison, existing hierarchical formulations often rely on a single yet
conservative tracking controller mode, c.f., Fig 2.

A specific example of such modes is a slow-speed movement with high precision
and a fast-speed movement with large uncertainty bounds, c.f., 3(b). As
illustrated in Fig. 3(a) (b), the two control modes enable the planner to
provide a collision-free path by switching online between different velocity
ranges and corresponding controller parametrization.

We propose a moving horizon formulation for the planning layer, resulting in
an efficiently solvable mixed-integer linear programming (MILP). The tracking
control for the vehicle is achieved by a cyclic horizon MPC 20, 22, which
operates with a high sampling frequency. The tracking controller provides
safety bounds for each operation mode, despite disturbances. We derive
conditions for recursive feasibility to ensure constraint satisfaction and
obstacle avoidance. Simulation results illustrate the efficiency and
applicability of the proposed hierarchical strategy. Especially, it is shown
how different modes reduce the conservatism, while the decomposition of the
control problem limits the computational cost and enables real-time
implementation while providing guarantees.

The presented results are based on the results presented in 23. They are
expanded with respect to less conservative conditions and formulations and the
overall approach is evaluated considering a quadcopter operating in a 3D
environment.

The main contributions of this work are threefold:

  1. 1.

First, we outline a planning approach operating on a moving horizon, which
allows us to consider different tracking controller modes. We show how this
approach can be reformulated as a MILP, which can be efficiently solved in
real-time.

  2. 2.

Second, we design a low-layer cyclic horizon tube MPC controller, which
provides safety bounds, i.e., tubes, for the different modes, e.g.,
velocities. This controller guarantees in combination with the planning layer
constraint satisfaction and collision avoidance.

  3. 3.

Finally, we demonstrate our approach, considering a quadcopter operating in a
3D environment.

The remainder of the paper is organized as follows. Section 2 presents the
problem setup. Section 3 outlines the hierarchy planning and control scheme,
where Section 3.1 presents the MILP moving horizon planning problem. Section
3.2 describes the low-layer cyclic horizon tube MPC and Section 3.3 outlines
the switching between the operating modes. Section 4 presents the simulation
example and results for an quadcopter, while Section 5 recapitulates our
findings, conclusions, and outlooks.

_Notation_ For two given sets $\mathbb{X},\mathbb{Y}\subset\mathbb{R}^{n}$,
the Minkowski set sum $\oplus$ and the Minkowski set difference $\ominus$ are
defined by:
$\mathbb{X}\oplus\mathbb{Y}\triangleq\\{x+y|x\in\mathbb{X},y\in\mathbb{Y}\\}$,
$\mathbb{X}\ominus\mathbb{Y}\triangleq\\{x|x\oplus\mathbb{Y}\subseteq\mathbb{X}$.
We use $\operatorname{rem}$ to denote the remainder function of Euclidean
division, $\mathbf{1}$ denotes a column vector with $1$ in each entry, and
$\|x\|_{Q}^{2}=x^{T}Qx$.

##  2 Problem Formulation

We consider the control of an autonomous vehicle, which should move (drive,
fly, etc.) from a starting point, $x(0)$, to a goal, $x_{\text{goal}}$, while
avoiding obstacles and satisfying constraints despite uncertainties and
disturbances, see Fig. 1. We assume that the vehicle dynamics are subject to
unknown but bounded additive disturbances and that they are governed by

$\displaystyle x(k+1)=\,$ $\displaystyle Ax(k)+Bu(k)+w(k),$ (1a)
$\displaystyle y(k)=\,$ $\displaystyle Cx(k),$ (1b) $\displaystyle x(k)\in\,$
$\displaystyle\mathbb{X},\,\,u(k)\in\mathbb{U}.$ (1c)

Here $x(k)\in\mathbb{R}^{n}$ is the vehicle’s state, $u(k)\in\mathbb{R}^{m}$
is the applied control input, and the output $y(k)\in\mathbb{R}^{p}$, while
$w(k)$ is an unknown, but bounded disturbance. The state $x(k)$ and the input
$u(k)$ need to satisfy constraints: they are restricted to the sets
$\mathbb{X}$ and $\mathbb{U}$, which are both closed and convex.

_Obstacle avoidance:_ Beside the constraints (1c) on the state $x(k)$ and
input $u(k)$ of the autonomous vehicle, we want to achieve obstacle avoidance,
which we formulate in terms of the output $y(k)$ (e.g. the position). We
assume that there are $H$ obstacles and that each obstacle is modeled as a
bounded set of the form $\mathbb{O}_{\ell}=\\{y|E_{\ell}y<f_{\ell}\\}$, where
$E_{\ell}\in\mathbb{R}^{q_{\ell}\times p}$. So, $\mathbb{O}_{\ell}$ is the
interior of a convex, compact polytope. In the most simple case (box
obstacles) $E_{\ell}=[I\,\,-I]$. Consequently, to avoid that the vehicle
collides or touches the obstacles we require that:

$\displaystyle
y(k)\notin\mathbb{O},\,\,\mathbb{O}=\\{E_{\ell}y(k)<f_{\ell},\,\,l=1,\ldots,H\\},$
(2)

where $\mathbb{O}$ is the collection of all $H$ obstacles. Clearly, the set of
admissible output/position $y(k)\notin\mathbb{O}$, defined in (2), is non-
convex, but contains its boundaries. One can also formulate it as

$\displaystyle\forall\ell\in\\{1,\ldots,H\\},\,\,\exists
a\in\\{1,...,q_{\ell}\\}:\ E_{\ell,a}y(k)\geq f_{\ell,a}.$ (3)

This formulation allows to use a MILP framework with the so-called big-M
approach, which enables handling the obstacles systematically, see Section
3.3.

_Disturbance bounds and operating regions/modes:_ The disturbance $w(k)$, and
its bounds, may depend in parts also on the vehicle’s state $x(k)$ and/or the
applied control input $u(k)$. This might, for example be due to model
uncertainties, which are captured as disturbances. For example, if the vehicle
is operated at a lower speed, then the (worst case) disturbance might be
smaller then at high speed.

Specifically, we consider discrete sets of disturbance bounds for different
low-layer/tracking controllers operating modes, i.e., we assume that there are
$N_{w}$ low-layer controller operating modes: (Operating region/mode dependent
disturbance bounds) There are $N_{w}$ low-layer controller operating
regions/modes defined by the state and input sets $U_{i}$ and $X_{i}$ such
that if $x(k)\in\mathbb{X}_{i}\subseteq\mathbb{X}$ and
$u(k)\in\mathbb{U}_{i}\subseteq\mathbb{U}$, the uncertainty can be bounded by
$w(k)\in\mathbb{W}_{i}$, where the sets $\mathbb{X}_{i}$, $\mathbb{U}_{i}$ are
convex and closed polytopes and $\mathbb{W}_{i}$ is a convex and compact
polytope. Note that the sets $\mathbb{X}_{i}$ (and $\mathbb{U}_{i}$) may
overlap, i.e. one can have that
$\mathbb{X}_{i}\cap\mathbb{X}_{j}\neq\emptyset$ for $i\neq j$.

##  3 Robust Hierarchical Moving Horizon Planning and Control

We focus on a hierarchical two-layer planning and control decomposition as
depicted in Fig. 3(a), to achieve safe and collision-free motion of the
autonomous vehicle to the goal. Both layers utilize (robust) MPC formulations
for constraint satisfaction under uncertainties.

To provide guarantees despite the hierarchical decomposition of the problem in
a planning and control layer, we utilize the concept of “contracts”, inspired
by 24, 25, 26. Loosely speaking, a contract specifies the achievable precision
in terms of a bound on the disturbance the lower-layer tracking controller can
achieve for a specific operating region/mode $i$ \- a particular set of states
and inputs. In the proposed approach, the moving horizon planner provides a
reference path and selects the operating region the controller should operate
in. To calculate a safe passage - satisfy the constraints (1c) and to avoid
all obstacles (3) - the planner takes the uncertainty bounds corresponding to
the different operating regions directly into account.

In detail the upper-layer planner calculates the reference based on a model
with simplified dynamics of the form

$\displaystyle x_{p}(k_{p}+1)=A_{p}x_{p}(k_{p})+B_{p}u_{p}(k_{p}),$ (4)

where $k_{p}$ is the planning time index. The generated reference path and
measurements of the vehicle’s state are used by the lower-layer controller to
calculate the control input $u(k)$, see Fig. 2. The lower-layer control loop
aims to efficiently counteract the disturbances, to ensure that the vehicle
follows the planned reference with a specified accuracy, to satisfy the
contract, and to guarantee constraint satisfaction. As the obstacles are
handled by the planner, the tracking controller does not need to consider
them, which enables a fast and efficient implementation as non-convex
constraints are avoided.

We consider that the planning is done slowly, all $M\ (>1)$ time steps. So we
have

$\displaystyle k=k_{p}\cdot M,\quad A_{p}=A^{M},\quad
B_{p}=\sum_{i=0}^{M-1}A^{i}B.$ (5)

We assume that the the real dynamics (1a) and the "planning dynamics" (4)
satisfy: (Controllability of the planning and control dynamics) The pairs
$(A,B)$ and $(A_{p},B_{p})$ are controllable. Note that if $(A_{p},B_{p})$ is
controllable, then also $(A,B)$. In principle, $(A_{p},B_{p})$ could be
uncontrollable even if $(A,B)$ is controllable, see 27.

The contracts ensure the planner that the lower-layer controller can bound the
tracking error by

$\displaystyle x\big{(}(k_{p}+1)\cdot
M\big{)}-x_{p}(k_{p}+1)\in\mathbb{Z}_{i},$ (6)

for $i=1,\ldots,N_{w}$, where the sets $\mathbb{Z}_{i}$ are convex, compact
polytopes and $N_{w}$ is the number of the operating mode.

The lower-layer controller can guarantee the constraints, if
$x\in\mathbb{X}_{i}$ and $u\in\mathbb{U}_{i}$, i.e., that the state and the
input are inside the operation region $i$, see Assumption 2. Additionally,
constraints due to the obstacle avoidance requirements and the different
sampling times need to be satisfied, which are introduced in s second step. We
assume that the contracts - the operating regions/modes - are designed offline
and known by both control layers. They depend on the design of of the low-
layer controller and the operation mode $i$, i.e., the (partly) selectable
uncertainty bound $\mathbb{W}_{i}$ on the disturbance $w$. Summarizing:
Utilizing the idea of contracts - different operating modes - enables the
planner to utilize and take the capability of the low-layer control loop into
account for the computation of the reference. Consequently, the planner
calculates and sends to the low-layer controller a reference and selects via
the choice of $\mathbb{Z}_{i}$ the maximum allowed tracking discrepancy. In
other words, the reference planner can switch between different operation
modes, in order to improve the performance, as illustrated in Fig. 3(b).

###  3.1 Upper-layer: Moving Horizon Reference Planning

The reference planer should guarantee constraint satisfaction, including
obstacle avoidance, despite the presence of disturbances and uncertainties in
combination with a suitably designed lower-layer control loop, compare Fig.
3(a).

The key idea is to incorporate the concept of contracts - different operating
modes - into mathematical programming based moving horizon planning schemes28,
9, 29, 30, 31. To do so, the reference planning problem
$\mathcal{P}\big{(}x(k_{p}M)\big{)}$ is formulated on a moving horizon as a
optimization problem:

$\displaystyle\min_{\\{x_{p}\\},\\{u_{p}\\},i}J_{p}\big{(}\\{x_{p}\\},\\{u_{p}\\}\big{)}$
, (7a) s.t. $\displaystyle i$ $\displaystyle\in\\{1,\ldots,N_{w}\\}$ (7b)
$\displaystyle x(k_{p}M)-x_{p}(k_{p}|k_{p})$ $\displaystyle\in\mathbb{Z}_{i},$
(7c) $\displaystyle\forall j$ $\displaystyle\in\\{0,\ldots,N-1\\}:$
$\displaystyle x_{p}(k_{p}+j+1|k_{p})$
$\displaystyle=A_{p}x_{p}(k_{p}+j|k_{p})+B_{p}u_{p}(k_{p}+j|k_{p}),$ (7d)
$\displaystyle\forall j$ $\displaystyle\in\\{0,\ldots,N-1\\}:$ $\displaystyle
x_{p}(k_{p}+j|k_{p})$ $\displaystyle\in\mathbb{X}_{i}\ominus\mathbb{Z}_{i},$
(7e) $\displaystyle\forall j$ $\displaystyle\in\\{0,\ldots,N-1\\}:$
$\displaystyle u_{p}(k_{p}+j|k_{p})$
$\displaystyle\in\mathbb{U}_{i}\ominus{K}\mathbb{Z}_{i},$ (7f)
$\displaystyle\forall j$ $\displaystyle\in\\{0,\ldots,N-1\\}:$ $\displaystyle
Cx_{p}(k_{p}+j|k_{p})$
$\displaystyle\notin\mathbb{O}\oplus(-C)\mathbb{Z}_{i},$ (7g)
$\displaystyle\forall j$ $\displaystyle\in\\{0,\ldots,N-1\\}:$
$\displaystyle\big{(}x_{p}(k_{p}+j|k_{p}),u_{p}(k_{p}+j|k_{p})\big{)}$
$\displaystyle\in\mathbb{I}_{i},$ (7h) $\displaystyle x_{p}(k_{p}+N|k_{p})$
$\displaystyle\in\mathbb{X}_{i}^{f}.$ (7i)

Here $N$ denotes the planning horizon, $i$ the operation mode, and
$(k_{p}+j|k_{p})$ corresponds to the prediction of a value at time $k_{p}+j$
made at time $k_{p}$. The constraint (7c) represent an initial constraint at
the begin of the planning time $k_{p}$. Eq. (7d) represent the vehicle
dynamics used by the planning layer. While the constraints (7e,7f) are the
tightened state and input constraints, where $K$ is a control gain, which is
discussed later in detail. For the output constraints (7g) the constraint
tightening corresponds to an enlargement of the obstacles, see Fig. 3(b).

To allow for an efficient reformulation of (7) as an MILP, c.f. Section 3.3,
we consider the following cost function

$\displaystyle
J_{\text{p}}\big{(}\\{x_{p}\\},\\{u_{p}\\}\big{)}=\|x_{\text{\tiny
goal}}-x_{p}(k_{p}+N)\|_{\infty}+\sum_{j=k_{p}}^{k_{p}+N-1}\alpha_{x}\|{x}_{p}(j)\|_{\infty}+\alpha_{u}\|{u}_{p}(j)\|_{\infty}.$

The stage cost penalizing the state $x_{p}$ and control input $u_{p}$ with
different weights $(\alpha_{x}\geq 0,\alpha_{u}\geq 0)$, respectively. The
terminal cost penalizes the vehicle’s distance at the end of the planning
horizon to the goal $x_{\text{goal}}$.

The inter-sample constraints (7h) and the terminal constraint (7i) depend on
the operation mode $i$ and are non-convex. We make the following assumption
with respect to the inter-sample constraints (7h) (Inter-sample constraints)
The inter-sample constraints (7h) determined by the sets $\mathbb{I}_{i}$ are
such that, if $(x_{p},u_{p})\in\mathbb{I}_{i}$, then for $\ell=1,\ldots,M-1$
it holds

$\displaystyle A^{\ell}x_{p}+\sum\limits_{m=0}^{\ell-1}A^{m}Bu_{p}$
$\displaystyle\in\mathbb{X}_{i}\ominus\mathbb{Z}_{i},$ (8a) $\displaystyle
C(A^{\ell}x_{p}+\sum\limits_{m=0}^{\ell-1}A^{m}Bu_{p})$
$\displaystyle\notin\mathbb{O}\oplus(-C)\mathbb{Z}_{i}.$ (8b)

This assumption guarantees that the lower-layer/tracking control loop,
operating at the faster time scale, can always satisfy the constraints. A
straightforward choice is to choose $\mathbb{I}_{i}$ directly as (8), which
can lead to a large number of constraints and thus might increase the
computational effort. Note that depending on the actual dynamics (4) certain
constraints in (7h) might be redundant and thus also the overall optimization
(7) and can be removed without changing the solution of the optimization
problem, e.g. using physical insight into the system dynamics or with the
procedure presented in 32.

For the terminal sets $\mathbb{X}^{f}_{i}$ we assume that they are positive
invariant sets for (4) satisfying all constraints: (Terminal sets and terminal
control laws) The terminal control laws $\kappa^{f}_{i}(x_{p})$ and the
terminal sets $\mathbb{X}^{f}_{i}$ are such that $x_{p}\in\mathbb{X}^{f}_{i}$
implies:

$\displaystyle A_{p}x_{p}+B_{p}\kappa^{f}_{i}(x_{p})$
$\displaystyle\in\mathbb{X}_{i}^{f},$ (9a) $\displaystyle x_{p}$
$\displaystyle\in\mathbb{X}_{i}\ominus\mathbb{Z}_{i},$ (9b)
$\displaystyle\kappa^{f}_{i}(x_{p})$ $\displaystyle\in\mathbb{U}_{i}\ominus
K\mathbb{Z}_{i},$ (9c)
$\displaystyle\big{(}x_{p},\kappa^{f}_{i}(x_{p})\big{)}$
$\displaystyle\in\mathbb{I}_{i},$ (9d) $\displaystyle Cx_{p}$
$\displaystyle\notin\mathbb{O}\oplus(-C)\mathbb{Z}_{i}.$ (9e)

Clearly, the terminal sets $\mathbb{X}^{f}_{i}$ are non-convex due to the
presence of the inter-sample constraints (9d) and the obstacle avoidance
condition (9e). A possible choice are admissible, nominal steady states
$x_{p}=A_{p}x_{p}+B_{p}\kappa^{f}_{i}(x_{p})$ for the terminal control sets
and the corresponding inputs as terminal control laws. For autonomous
vehicles, these are basically all points where the vehicle can stop its motion
safely. These points can also be determined for systems with complex dynamics
such as unmanned aerial vehicles. Note that the terminal control laws
$\kappa^{f}_{i}(x_{p})$ are fictitious and never implemented.

The upper-layer planning algorithm solves the optimization problem
$\mathcal{P}\big{(}x(k_{p}M)\big{)}$ (7). Based on the optimal solution it
sends to the lower-layer controller the chosen operation mode $i^{\star}$ and
an inter-sampled reference

$\displaystyle
x_{\text{ref}}(k_{p}M+j)=A^{j}x_{p}^{\star}(k_{p}|k_{p})+\sum\limits_{m=0}^{j}A^{m-1}Bu_{p}^{\star}(k_{p}|k_{p}),\,\,j=0,\ldots,M.$
(10)

Clearly,
$\big{(}x_{p}^{\star}(k_{p}|k_{p}),u_{p}^{\star}(k_{p}|k_{p})\big{)}\in\mathbb{I}_{i}$
implies that the reference satisfies

$\displaystyle
Cx_{\text{ref}}(k_{p}M+j)\notin\mathbb{O}\oplus(-C)\mathbb{Z}_{i},\,\,j=0,\ldots,M,$
(11)

which means that the planned reference $x_{\text{ref}}$ robustly avoids
obstacles, it satisfies the condition (2). Using the idea of contracts between
both layers, we can guarantee the following.

######  Proposition 3.1.

(Recursive feasibility of the upper layer planning) Let Assumptions 2-3.1 hold
and assume that the lower-layer controller guarantees the contract (6) for the
reference (10). If the planning optimization problem
$\mathcal{P}\big{(}x(k_{p}M)\big{)}$ (7) is feasible, then also
$\mathcal{P}\big{(}x((k_{p}+1)M)\big{)}$ is feasible.

######  Proof 3.2.

The key idea for the proof is that the upper-layer planning MPC (in
combination with the contracts) corresponds to a tube-based MPC using robust
invariant sets, compare 17, 19, 21.

We denote the optimal solution of $\mathcal{P}\big{(}x(k_{p}M)\big{)}$ by
$x_{p}^{\star}(k_{p}|k_{p})$, …, $u_{p}^{\star}(k_{p}|k_{p})$, …, $i^{\star}$.
Let us consider the following guess as solution for the optimization problem
$\mathcal{P}\big{(}x((k_{p}+1)M)\big{)}$

$\displaystyle i$ $\displaystyle=i^{\star},$ $\displaystyle\forall
j\in\\{1,\ldots,N\\}:x_{p}(k_{p}+j|k_{p}+1)$
$\displaystyle=x^{\star}_{p}(k_{p}+j|k_{p}),$ $\displaystyle
x_{p}(k_{p}+N+1|k_{p}+1)$
$\displaystyle=A_{p}x^{\star}_{p}(k_{p}+N|k_{p})+B_{p}\kappa^{f}_{i}\big{(}x^{\star}_{p}(k_{p}+N|k_{p})\big{)},$
$\displaystyle\forall m\in\\{1,\ldots,N-1\\}:u_{p}(k_{p}+m|k_{p}+1)$
$\displaystyle=u^{\star}_{p}(k_{p}+m|k_{p}),$ $\displaystyle
u_{p}(k_{p}+N|k_{p}+1)$
$\displaystyle=\kappa^{f}_{i}\big{(}x^{\star}_{p}(k_{p}+N|k_{p})\big{)}.$

Note that this guess is based on the previous solution, the selected operation
mode $i^{\star}$ and the terminal control law $\kappa_{i}^{f}$. We need to
verify that this guess is feasible (but it might be possibly suboptimal) for
the optimization problem $\mathcal{P}\big{(}x((k_{p}+1)M)\big{)}$. Using the
contract (6), i.e. the guarantee on the lower-layer control loop, we have that
$x\big{(}(k_{p}+1)M\big{)}-x_{p}(k_{p}+1|k_{p}+1)\in\mathbb{Z}_{i}$, i.e. (7c)
holds for $\mathcal{P}\big{(}x((k_{p}+1)M)\big{)}$. The above initial guess
satisfies the constraints (7d), (7e)-(7h) for $1\leq j<N-1$ for the
optimization problem $\mathcal{P}\big{(}x(k_{p}M)\big{)}$, thus also the
similar constraints for $\mathcal{P}\big{(}x((k_{p}+1)M)\big{)}$. Finally,
using the conditions on the terminal sets and terminal control laws in
Assumption 3.1 imply that also the remaining constraints of
$\mathcal{P}\big{(}x((k_{p}+1)M)\big{)}$, i.e. (7d)-(7h) with $j=N-1$ and (7i)
are satisfied.

######  Remark 3.3.

(Planning without feedback) _In ( 7) the initial constraint (7c) provides a
feedback between the planning state $x_{p}$ and the real state $x$ for all
$k_{p}$. One can only enforce the constraint (7c) at the initial time
($k_{p}=0$) and replace it by the simpler equality constraint
$x^{\star}_{p}(k_{p}+1|k_{p})=x_{p}(k_{p}+1|k_{p}+1)$ for $k_{p}>0$. This
would remove the feedback from the plant to the upper-layer planner. This has
the advantage that it avoids the need to wait for plant feedback for the
planning and thus could enable computationally less restrictive planning, but
it would also decrease the control performance._

The optimization problem $\mathcal{P}$ (7) is non-convex, but it can be
reformulated as an MILP, which can be efficiently solved, see Section 3.3.

###  3.2 Lower-layer: Cyclic Horizon Robust Model Predictive Tracking Control

The lower-layer controller tracks the generated reference based on the
(faster) dynamics of the real system and needs to guarantee the contracts (6).
Also at the lower-layer we use the concept of robust, tube-based MPC 17, 19,
20, 21, but we rely on growing tubes 20 instead of tubes based on robust
positive invariance 21 as in the upper-layer. The proposed tube-based MPC of
the lower-layer is based on a nominal prediction dynamics (nominal state $z$,
nominal input $v$), which starts from the real state $x(k)$ at the current
time $k$:

$\displaystyle z(k+j+1|k)$ $\displaystyle=Az(k+j|k)+Bv(k+j|k),$ (12a)
$\displaystyle z(k|k)$ $\displaystyle=x(k).$ (12b)

The effect of disturbances $w(k+j)$ onto the closed loop is taken into account
using a fictitious, auxiliary control law of the form

$\displaystyle u(k+j|k)=v(k+j|k)+K\big{(}x(k+j)-z(k+j|k)\big{)}.$ (13)

The control gain $K$ in this affine feedback is chosen such that $A+BK$ is
Schur stable. The auxiliary control law is utilized to determine sets to bound
the difference $e(k+j|k)=x(k+j)-z(k+j|k)$ between the real system state $x$
and the predictions $z$ made using (12). In detail, for the $i$-th operation
mode the error bounds satisfy $e(k+j|k)\in\mathbb{E}_{i}(j)$ where

$\displaystyle\mathbb{E}_{i}(j+1)=(A+BK)\mathbb{E}_{i}(j)\oplus\mathbb{W}_{i},$
$\displaystyle\mathbb{E}_{i}(0)=\\{0\\}.$ (14)

Note that the size of the sets $\mathbb{E}_{i}$ monotonically increases with
$j$, i.e. $\mathbb{E}_{i}(j)\subseteq\mathbb{E}_{i}(j+1)$. However, for any
$j\geq 0$ we have that $\mathbb{E}_{i}(j)\subseteq\mathbb{Z}_{i}$, where
$\mathbb{Z}_{i}$ is the (minimum) robust positive invariant set, compare 19,
satisfying:

$\displaystyle\mathbb{Z}_{i}\supseteq(A+BK)\mathbb{Z}_{i}\oplus\mathbb{W}_{i}.$
(15)

In the lower-layer MPC we predict until the next planning instant utilizing a
cyclic horizon $L_{k}=M-\operatorname{rem}(k,M)$, see 22. For the case that
$k$ is a multiple of $M$, we have $L_{k}=M$. Otherwise, $L_{k}$ is smaller
than $M$, but $k+L_{k}$ is a multiple of $M$. Consequently, the horizon
shrinks between two planning instants and is increased at the next planning
instant again to length $M$.

The lower-layer MPC predicts and optimizes nominal state and input sequences

$\displaystyle\mathbf{z}(k)=\\{z(k|k),\ldots,z(k+L_{k}|k)\\},\quad\mathbf{v}(k)=\\{v(k|k),\ldots,v(k+L_{k}-1|k)\\},$
(16)

based on the nominal dynamics (12) and subject to satisfaction of the
constraints

$\displaystyle z(k+j|k)$
$\displaystyle\in\mathbb{X}_{i}\ominus\mathbb{E}_{i}(j),$ (17a) $\displaystyle
v(k+j|k)$ $\displaystyle\in\mathbb{U}_{i}\ominus K\mathbb{E}_{i}(j),$ (17b)
$\displaystyle C\left(z(k+j|k)-x_{\text{ref}}(k+j)\right)$ $\displaystyle\in
C\left(\mathbb{Z}_{i}\ominus\mathbb{E}_{i}(j)\right),$ (17c) $\displaystyle
z(k+L_{k}|k)-x_{\text{ref}}(k+L_{k})$
$\displaystyle\in\mathbb{Z}_{i}\ominus\mathbb{E}_{i}(L_{k}).$ (17d)

Note that these constraints include the convex state and input constraints
(1c). In contrast, the non-convex obstacle avoidance constraints are taken
into account using the concept of contracts. Basically, the lower-layer
controller needs to enforce the guaranteed accuracy with respect to the output
(condition (17c)) or even the full state at the end of the prediction
(condition (17d)). Note that, the constraints (12), (17) are convex.

The lower-layer MPC penalizes the deviation error from the reference
$x_{\text{ref}}$ and utilizes the convex cost function

$\displaystyle J_{\text{t}}\big{(}\mathbf{z}(k),\mathbf{v}(k)\big{)}=$
$\displaystyle\sum_{j=k}^{k+L_{k}-1}\|x_{\text{ref}}(j)-z(j|k)\|_{Q}^{2}+\|v(j|k)\|_{R}^{2}+\|x_{\text{ref}}(k+L_{k})-z(k+L_{k}|k)\|_{P}^{2},$
(18)

where the matrices $Q\in\mathbb{R}^{n\times n}$, $P\in\mathbb{R}^{n\times n}$
and $R\in\mathbb{R}^{m\times m}$ are positive definite and represent the
weighting for the inter-sample states, the final state and the inputs,
respectively.

The applied control input $u(k)=v^{\star}(k|k)$ is given by solving the
optimization problem $\mathcal{L}\big{(}x(k),\\{x_{\text{ref}}\\},i,k\big{)}$

$\displaystyle\min\limits_{\mathbf{z}(k),\mathbf{v}(k)}J_{\text{t}}\big{(}\mathbf{z}(k),\mathbf{v}(k)\big{)}\mbox{
s.t.~{}\eqref{LL_c_1},~{}\eqref{LL_c_2}.}$ (19)

This optimization problem depends on the current state available at the lower-
layer as well as the reference and the operation mode determined by the upper-
layer planner. The resulting optimization problem is a convex quadratic
program (QP) and has, in addition, a special structure, which allows its
efficient solution, even on computationally limited hardware, see e.g. 33, 34.

For the lower-layer, we can derive the following properties assuming that the
upper-layer reference is chosen suitably.

######  Proposition 3.4.

(Constraint satisfaction and obstacle avoidance) Let Assumptions 2 and 3.1
hold and assume that the reference $x_{\text{ref}}$ is given by (10) and
satisfies (11). If the lower-layer MPC problem
$\mathcal{L}\big{(}x(k),\\{x_{\text{ref}}\\},i,k\big{)}$ (19) is feasible,
then the constraints (1c) are satisfied:
$x(k)\in\mathbb{X}_{i}\subseteq\mathbb{X}$ and
$u(k)\in\mathbb{U}_{i}\subseteq\mathbb{U}$ and the obstacles are avoided, i.e.
(3) holds.

######  Proof 3.5.

Combining (12), (13) and (17) we have that $z^{\star}(k|k)=x(k)$,
$v^{\star}(k|k)=u(k)$ and $Cz^{\star}(k|k)=y(k)$. Together with
$\mathbb{E}_{i}(0)=\\{0\\}$ and Assumption 2 this implies that the state and
input constraints (1c) are satisfied. (17c) together with (14) yield that
$y(k)-Cx_{\text{ref}}(k)\in C\mathbb{Z}_{i}$. Combined with (7g) (11) this
implies that the avoidance constraint $y_{k}\notin\mathbb{O}$ (3) holds.

Note that if the upper-layer planner provides the reference $x_{\text{ref}}$
(10), then also (11) holds.

######  Proposition 3.6.

(Recursive feasibility of the overall control scheme) Let Assumptions 2-3.1
hold. If the planning problem (7) is feasible for $x(0)$, then the
optimization problems (7), (19) remain feasible for the closed loop system
consisting of the upper-layer moving horizon planner (7), (10), the lower-
layer controller (19) and the uncertain plant dynamics (1).

######  Proof 3.7.

The proof consists of three parts: first it is shown that feasibility of the
planning problem $\mathcal{P}\big{(}x(k_{p}M)\big{)}$ (7) implies feasibility
of the low-layer optimization problem (19)
$\mathcal{L}\big{(}x(k_{p}M),\\{x_{\text{ref}}\\},i^{\star},k_{p}M\big{)}$.
Afterwards we verify that feasibility of
$\mathcal{L}\big{(}x(k),\\{x_{\text{ref}}\\},i,k\big{)}$ (19) implies: if
$k+1$ is not a multiple of $M$, feasibility of
$\mathcal{L}\big{(}x(k+1),\\{x_{\text{ref}}\\},i,k+1\big{)}$ (19) (part 2) and
otherwise feasibility of the planning problem
$\mathcal{P}\big{(}x(k+1)\big{)}$ (7) (part 3).

1) If $\mathcal{P}\big{(}x(k_{p}M)$ (7) is feasible, then the following
(suboptimal) input trajectory and state sequence for the lower-layer problem
$\mathcal{L}\big{(}x(k_{p}M),\\{x_{\text{ref}}\\},i^{\star},k_{p}M\big{)}$

$\displaystyle z(k|k)=$ $\displaystyle x(k)$ $\displaystyle v(k+j|k)=$
$\displaystyle
u_{p}^{\star}(k_{p}|k_{p})+K(z(k+j|k)-x_{\text{ref}}(k+j|k)),\,\,j=0,\ldots,M-1,$
$\displaystyle z(k+j+1|k)=$ $\displaystyle
Az(k+j|k)+Bv(k+j|k),\,\,j=0,\ldots,M-1,$

satisfies all constraints of (19) due to the inter-sample constraints
$\mathbb{I}_{i}$ and the consistent constraint tightening utilized at both
layers, i.e., the definition of $\mathbb{E}_{i}$ and $\mathbb{Z}_{i}$, see
(14) and (15) and that $\mathbb{E}_{i}\subseteq\mathbb{Z}_{i}$.

2) In the case that $k+1$ is not a multiple of $M$, i.e. no planning takes
place, the horizon $L_{k}$ shrinks. Due to the design of the set
$\mathbb{E}_{i}(j)$ (14) a feasible nominal state trajectory $\mathbf{z}(k+1)$
and a nominal input trajectory $\mathbf{v}(k+1)$ for
$\mathcal{L}\big{(}x(k+1),\\{x_{\text{ref}}\\},i,k+1\big{)}$, satisfying (12)
and (17), can be obtained from the solution of
$\mathcal{L}\big{(}x(k),\\{x_{\text{ref}}\\},i,k\big{)}$:

$\displaystyle z(k+j|k+1)=$ $\displaystyle z^{\star}(k+j|k)+(A+BK)^{j-1}w(k),$
$\displaystyle v(k+j|k+1)=$ $\displaystyle
v^{\star}(k+j|k)+K(A+BK)^{j-1}w(k).$

3) Finally, if $k+1$ is a multiple of $M$, i.e. the planning problem is solved
at $k+1$, then feasibility of
$\mathcal{L}\big{(}x(k),\\{x_{\text{ref}}\\},i,k\big{)}$, in particular (17d)
implies that $x(k+1)-x_{\text{ref}}(k+1)\in\mathbb{Z}_{i^{\star}}$. This
together with Proposition 3.1 yields that the planning problem (7) is feasible
at $k+1$.

######  Remark 3.8.

(Adaption of sets) _We assume that state and input constraints
$\\{\mathbb{X}_{i},\mathbb{U}_{i}\\}$ and the tubes/contracts
$\\{\mathbb{E}_{i},\mathbb{Z}_{i}\\}$ are determined offline. The proposed
approach can in principle be extended to allow an adaption of these sets. This
could be useful for example to consider the influence of varying weather
conditions. We do not consider such an extension in this work._

######  Remark 3.9.

(Relaxing condition (17c)) _In the optimization problem ( 19) the difference
between the real/predicted output $y_{k}$/$z_{k}$ and its reference
$Cx_{\text{ref}}(k)$ is restricted to the set $C\mathbb{Z}_{i}\ominus
C\mathbb{E}_{i}$, compare (17c). This restriction is used to enable guarantees
on the obstacle avoidance constraints (2). However, if the vehicle
position/output (or /certain directions of it) is at time instant $k$ far way
from an obstacle, then this restriction can be conservative. In principle, it
is possible to relax these constraints by generating online based on the
solution of the upper-layer sets $\mathbb{Y}_{k+j}$ resulting in less
conservative output constraints: $Cz(k+j|k)\in\mathbb{Y}_{k+j}\ominus
C\mathbb{E}_{i}$._

######  Remark 3.10.

(More general tube scheme) _We use a basic tube scheme with a single gain $K$
and focused on linear dynamics. One could use a more general scheme with
multiple gains, see 35, or a more complex tube control law, see e.g. 36, 37.
Also, an extension to nonlinear lower-layer dynamics is in principle possible
using for example 38, 39._

###  3.3 MILP solution of the planning problem

In the following, we discuss how the non-convex optimization problem
$\mathcal{P}$ (7) can be reformulated using the big-M method 9, 29 into an
MILP. Note that $\mathcal{P}$ (7) is non-convex due to two reasons: firstly,
due to the operation mode $i$, and secondly the non-convex obstacles avoidance
constraints (3) result in the non-convex constraints (7f) - (7i).

_Scheduling of operation modes_ : In the proposed hierarchical scheme the
operating modes of the vehicle given in the form of different constraint sets
$\mathbb{X}_{i}$ and $\mathbb{U}_{i}$ directly influences the uncertainty
bounds $w\in\mathbb{W}_{i}$, see Assumption 2. The lower-layer controller
guarantees constraint satisfaction and guarantees bounds on the tracking error
in form of a set $\mathbb{Z}_{i}$, which depends on the operation mode. So,
the sets appearing in the initial constraint (7c) and the tightened
state/input constraints (7e), (7f) are of the form

$\displaystyle\mathbb{Z}_{i}$ $\displaystyle\equiv\\{x|F^{z}_{i}x\leq
G^{z}_{i}\\},\quad\forall i\in\\{1,\ldots,N_{w}\\},$
$\displaystyle\mathbb{X}_{i}\ominus\mathbb{Z}_{i}$
$\displaystyle\equiv\\{x|F^{x}_{i}x\leq G^{x}_{i}\\},\quad\forall
i\in\\{1,\ldots,N_{w}\\},$ $\displaystyle\mathbb{U}_{i}\ominus
K\mathbb{Z}_{i}$ $\displaystyle\equiv\\{u|F^{u}_{i}u\leq
G^{u}_{i}\\},\quad\forall i\in\\{1,\ldots,N_{w}\\}.$

As result, this contract provides the planner an extra degree of freedom to
reduce the planning conservatism by switching between different operating
modes of the lower-layer controller.

We use the so called big-M method to formulate the mode scheduling as:

$\displaystyle F^{z}_{i}x-x_{p}\leq$ $\displaystyle
G^{z}_{i}+M_{\text{big}}\big{(}1-d_{i}\big{)}\mathbf{1},\quad\forall
i\in\\{1,\ldots,N_{w}\\},$ $\displaystyle F^{x}_{i}x\leq$ $\displaystyle
G^{x}_{i}+M_{\text{big}}\big{(}1-d_{i}\big{)}\mathbf{1},\quad\forall
i\in\\{1,\ldots,N_{w}\\},$ $\displaystyle F^{u}_{i}u\leq$ $\displaystyle
G^{u}_{i}+M_{\text{big}}\big{(}1-d_{i}\big{)}\mathbf{1},\quad\forall
i\in\\{1,\ldots,N_{w}\\},$ $\displaystyle\sum_{i=1}^{N_{w}}d_{i}=$
$\displaystyle 1.$

Here we use a large positive number $M_{\text{big}}$ to deactivate the
constraints of the $i$-th mode by relaxing its constraints using the binary
decision variable $d_{i}$. The last constraint guarantees that exactly one
mode is active in the planning.

_Obstacle avoidance constraints:_ For each mode, the tracking error set
$\mathbb{Z}_{i}$ is used to enlarge boundaries of the obstacles $\mathbb{O}$,
compare (7). The enlarged, non-convex avoidance constraints (3) can be
rewritten/over-approximated by

$\displaystyle\forall j=0,\ldots,N-1,\,\forall,\ell=1\ldots,H:\exists a\in
1,\ldots,q_{j}\mbox{ s.t. }E_{\ell,a}Cx_{p}(k_{p}+j|k_{p})\geq
f_{\ell,a}^{i}.$

In this case one can enforce this constraint for the active mode $i$ by using
additional binary variables $b^{i}_{\ell,a}(j)$ by requiring for
$j=0,\ldots,N-1$ and $\ell=1\ldots,H$ that

$\displaystyle E_{\ell,a}Cx_{p}(k_{p}+j|k_{p})$ $\displaystyle\geq
f_{\ell,a}^{i}-M_{\text{big}}b^{i}_{\ell,a}(j),$
$\displaystyle\sum\limits_{a=1}^{q_{j}}b^{i}_{\ell,a}(j)$ $\displaystyle\leq
q^{\ell}-d_{i}.$

Here we impose an extra constraint to ensure that at least one active
constraint for mode $i$, where $q^{\ell}$ is number of faces of each obstacle.
Loosely speaking the planner can choose different low-layer controller modes
$i$ to adjust the obstacle boundary $f_{\ell,a}^{i}$ by changing the operation
modes. This is illustrated in Fig. 6(b) considering that the operation modes
corresponds to different velocities.

##  4 Unmanned Aerial Vehicle Example

We consider a quadcopter that should fly from a starting to a goal point
without hitting obstacles, c.f. Figure 4.

Figure 4: Illustration of the quadcopter state, forces, and moments described
in the earth ($p_{x}$, $p_{y}$, $p_{z}$, $\theta$, $\varphi$, $\phi$) and body
frame. The considered inputs are the lateral and longitudinal moments are
provided by the four rotors resulting in the forces $F_{i}$.

A linearized model, as presented in 40 based on a nonlinear dynamic model is
used, c.f. Appendix A. The states of the quadcopter are roll and pitch angles
$(\phi,\theta)$, roll and pitch rates $(w_{x},w_{y})$, 3D position
$(p_{x},p_{y},p_{z})$, and 3D velocities $(v_{x},v_{y},v_{z})$. The linearized
model has three inputs: $u_{\text{lon}}=\theta_{c}$, $u_{\text{lat}}=\phi_{c}$
and $u_{\text{alt}}=T_{c}$. The input and states are constrained to:

$\displaystyle\left[\begin{smallmatrix}-1.5\,\text{m/s}\\\
-\pi/4\,\text{rad}\\\
-\pi/4\,\text{rad}\end{smallmatrix}\right]\leq\left[\begin{smallmatrix}v_{x},\,v_{y},\,v_{z}\\\
\phi,\,\theta,\\\
\phi_{c},\,\theta_{c}\end{smallmatrix}\right]\leq\left[\begin{smallmatrix}1.5\,\text{m/s}\\\
\pi/4\,\text{rad}\\\ \pi/4\,\text{rad}\end{smallmatrix}\right].$ (20)

The model is discretized with a sampling time of $T_{p}=0.5s$ for the planning
problem and $T_{t}=0.05s$, i.e. $M=10$, for the cyclic horizon MPC controller
(5). YALMIP 41 is used to implement and formulate the planning and control
problems, exploiting Gurobi 42 for the solution of the optimization problems.
The required sets are calculated via the MPT toolbox 43.

Figure 5 shows simulations for the case that only a single low layer control
mode is used.

Figure 5: Disturbance free closed-loop simulation results for a single MPC
controller mode.

As can be seen, for a small planning horizon of $N_{p}=15$ no path around the
obstacle can be found. Only an increase of the planning horizon to $N_{p}=30$,
or the removal of the vertical velocity constraint on $v_{z}$ allows the
controlled vehicle to reach the goal. Note that in all cases no collisions
occur. The (maximum) computation times to solve the planning problem are
$\text{T}_{\text{com}}=$ 0.4 s for $N=15$, and $\text{T}_{\text{com}}=$ 60 s
for $N=30$, which is well above the desired re-planning time of 0.5 s.111The
computation times are carried out on an Intel${}^{\circledR}$ CoreTM i7-8550U
CPU which operates on 1.99GHz..

Figure 6 shows simulation results using two operation modes for the low-layer
MPC controller. They correspond to a fast operation mode, given by
$-1.5\,\text{m/s}\,\leq(v_{x},\,v_{y})\leq 1.5\,\text{m/s}$ and a slow
operation mode given by $-1.0\,\text{m/s}\,\leq(v_{x},\,v_{y})\leq
1.0\,\text{m/s}$. The fast operation mode corresponds to a large uncertainty
set, while slow operation mode leads to a smaller uncertainty bound
$\mathbb{W}_{2}$ and thus smaller sets $\mathbb{E}_{2}$ and $\mathbb{Z}_{2}$.

((a)) 3D and top down view.

((b)) Selected operation mode, input, and speed profile

Figure 6: (a): Using two operating modes allows the quadcopter to reach the
goal on a direct way. (b): The reference planner switches between the two
operation modes, resulting in different velocity constraints and uncertainty
bounds.

Figure 7 shows simulation results for large wind disturbances, which are not
considered in the controller explicitly. As can be seen, the hierarchical
control strategy is able to achieve the goal, while avoiding the obstacles and
satisfying the input constraints.

((a)) 3D and top down view.

((b)) Selected operation mode, input, and speed profile

Figure 7: (a): Strong wind disturbance case: the reference planner allows to
reach the goal and avoid obstacles despite strong wind disturbance. (b): The
reference planner switches between the two operation modes, resulting in
different velocity constraints and uncertainty bounds.

Summarizing, introducing additional control modes allows avoiding conservative
behavior while satisfying constraints and being computationally feasible.

##  5 Conclusions and outlook

We propose a tightly integrated hierarchical predictive control and planning
approach. Both planner and controller repeatedly solve moving horizon optimal
control problems. The upper- and lower-layers exploit different “contracts”
(guaranteed uncertainty bounds), i.e., the planner can choose other low-layer
controller regions/modes instead of using a fixed safety corridor to capture
the controller capabilities. For example, the planner can choose slow-speed
movement with high precision or a fast speed with large uncertainty bounds.

The planning reference is determined by solving a moving horizon optimization
problem considering a simplified model. It exploits constraints tightening,
which represents the lower-layer tracking capabilities in the form of the
precision contracts. The resulting planning algorithm can be reformulated as a
MILP, allowing for efficient and reliable solutions. The low-layer tube-based
MPC controller utilizes a cyclic horizon and results in a convex optimization
problem. It guarantees constraint satisfaction and the desired tracking
accuracy for the different modes. Moreover, it operates at a faster time
scale. We derived conditions that ensure compatibility between the planning
and control layers to guarantee recursive feasibility and ensure the
satisfaction of constraints and obstacle avoidance.

Simulation results demonstrated the efficiency and applicability of the
proposed hierarchical strategy. First, the contract option provides
significant advantages, e.g., it leads to a less conservative solution.
Moreover, the hierarchical decomposition of the challenging vehicle
control/planning problem leads to a decrease in the computational cost. It
allows the implementation of robust control on-board while providing
guarantees.

Possible extension are the consideration of ellipsoidal tube MPC methods 39,
44. In this case, the lower controller online sends the tube parameterization
to the upper layer. Therefore, the planner can predict a possible uncertainty
evaluation over the planning horizon.

We also aim to experimentally evaluate the approach, implementing the upper-
layer planner and the lower-layer MPC controller on computationally limited
systems.

## References

  * 1 Paden B, Čáp M, Yong SZ, Yershov D, Frazzoli E. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on intelligent vehicles 2016; 1(1): 33–55.
  * 2 Schwarting W, Alonso-Mora J, Rus D. Planning and decision-making for autonomous vehicles. Annual Review of Control, Robotics, and Autonomous Systems 2018; 1: 187–210.
  * 3 Dadkhah N, Mettler B. Survey of motion planning literature in the presence of uncertainty: Considerations for UAV guidance. Journal of Intelligent & Robotic Systems 2012; 65(1-4): 233–246.
  * 4 Kingston Z, Moll M, Kavraki LE. Sampling-based methods for motion planning with constraints. Annual review of control, robotics, and autonomous systems 2018; 1: 159–185.
  * 5 Ibrahim M. Real-Time Moving Horizon Planning and Control of Aerial Systems under Uncertainties. Phd thesis. Otto von Guericke University, Institute for Automation Engineering; 2020.
  * 6 Rosolia U, Singletary A, Ames AD. Unified multi-rate control: from low level actuation to high level planning. arXiv preprint arXiv:2012.06558 2020\.
  * 7 Pant YV, Yin H, Arcak M, Seshia SA. Co-design of control and planning for multi-rotor UAVs with signal temporal logic specifications. In: Proc. American Control Conference. IEEE. ; 2021: 4209–4216.
  * 8 Broderick JA, Tilbury DM, Atkins EM. Optimal coverage trajectories for a UGV with tradeoffs for energy and time. Autonomous Robots 2014: 257–271.
  * 9 Ibrahim M, Matschek J, Morabito B, Findeisen R. Hierarchical model predictive control for autonomous vehicle area coverage. IFAC-PapersOnLine 2019; 52(12): 79–84.
  * 10 Yin H, Bujarbaruah M, Arcak M, Packard A. Optimization based planner–tracker design for safety guarantees. In: Proc. American Control Conference. IEEE. ; 2020: 5194–5200.
  * 11 Ibrahim M, Matschek J, Morabito B, Findeisen R. Improved area covering in dynamic environments by nonlinear model predictive path following control. IFAC-PapersOnLine 2019; 52(15): 418–423.
  * 12 Koeln J, Alleyne A. Two-Level Hierarchical Mission-Based Model Predictive Control. In: American Control Conference. IEEE. ; 2018: 2332-2337.
  * 13 Vermillion C, Menezes A, Kolmanovsky I. Stable hierarchical model predictive control using an inner loop reference model and $\lambda$-contractive terminal constraint sets. Automatica 2014; 50(1): 92–99.
  * 14 Goerzen C, Kong Z, Mettler B. A survey of motion planning algorithms from the perspective of autonomous UAV guidance. Journal of Intelligent and Robotic Systems 2010; 57(1-4): 65.
  * 15 Quan L, Han L, Zhou B, Shen S, Gao F. Survey of UAV motion planning. IET Cyber-systems and Robotics 2020; 2(1): 14–21.
  * 16 Mac TT, Copot C, Tran DT, De Keyser R. Heuristic approaches in robot path planning: A survey. Robotics and Autonomous Systems 2016; 86: 13–28.
  * 17 Grüne L, Pannek J. Nonlinear model predictive control.  Springer .  2017\.
  * 18 Findeisen R, Allgöwer F. An introduction to nonlinear model predictive control. 21st Benelux Meeting on Systems and Control 2002; 11: 119–141.
  * 19 Rawlings JB, Mayne DQ, Diehl M. Model predictive control: theory, computation, and design. 2.  Nob Hill Publishing .  2017\.
  * 20 Chisci L, Rossiter JA, Zappa G. Systems with persistent disturbances: predictive control with restricted constraints. Automatica 2001; 37(7): 1019–1028.
  * 21 Mayne DQ, Seron MM, Raković S. Robust model predictive control of constrained linear systems with bounded disturbances. Automatica 2005; 41(2): 219–224.
  * 22 Kögel M, Findeisen R. Stability of NMPC with cyclic horizons. IFAC Proceedings Volumes 2013; 46(23): 809–814.
  * 23 Ibrahim M, Kögel M, Kallies C, Findeisen R. Contract-based hierarchical model predictive control and planning for autonomous vehicle. IFAC-PapersOnLine 2020; 53(2): 15758–15764.
  * 24 Bäthge T, Kögel M, Di Cairano S, Findeisen R. Contract-based Predictive Control for Modularity in Hierarchical Systems. IFAC-PapersOnLine 2018; 51(20): 499–504.
  * 25 Lucia S, Kögel M, Findeisen R. Contract-based predictive control of distributed systems with plug and play capabilities. IFAC-PapersOnLine 2015; 48(23): 205–211.
  * 26 Blasi S, Kögel M, Findeisen R. Distributed Model Predictive Control Using Cooperative Contract Options. IFAC-PapersOnLine 2018; 51(20): 448–454.
  * 27 Ogata K. Discrete-time control systems.  Prentice-Hall .  1995\.
  * 28 Trodden P, Richards A. Multi-vehicle cooperative search using distributed model predictive control. In: AIAA Guidance, Navigation and Control Conference and Exhibit. AIAA. ; 2008: 7138
  * 29 Ibrahim M, Kallies C, Findeisen R. Learning-Supported Approximated Optimal Control for Autonomous Vehicles in the Presence of State Dependent Uncertainties. In: Proc. European Control Conf. IEEE. ; 2020: 338-343.
  * 30 Pinto SC, Afonso RJ. Risk Constrained Navigation Using MILP-MPC Formulation. IFAC-PapersOnLine 2017; 50(1): 3586–3591.
  * 31 Richards A, How JP. Robust variable horizon model predictive control for vehicle maneuvering. International Journal of Robust and Nonlinear Control: IFAC-Affiliated Journal 2006; 16(7): 333–351.
  * 32 Kerrigan EC. Robust constraint satisfaction: Invariant sets and predictive control. Phd thesis. University of Cambridge, Cambridge, UK; 2001\.
  * 33 Wang Y, Boyd S. Fast model predictive control using online optimization. IEEE Transactions on Control Systems Technology 2009; 18(2): 267–278.
  * 34 Zometa P, Kögel M, Findeisen R. $\mu$AO-MPC: A free code generation tool for embedded real-time linear model predictive control. In: IEEE. ; 2013: 5320–5325.
  * 35 Kögel M, Findeisen R. Robust MPC with reduced conservatism by blending multiples tubes. In: Proc. American Control Conference. IEEE. ; 2020.
  * 36 Raković SV, Kouvaritakis B, Cannon M, Panos C, Findeisen R. Fully parameterized tube MPC. IFAC Proceedings Volumes 2011; 44(1): 197–202.
  * 37 Raković SV. Invention of prediction structures and categorization of robust MPC syntheses. In: IFAC Conf. Nonlinear Model Predictive Control. IFAC. ; 2012: 245–273.
  * 38 Kögel M, Findeisen R. Discrete-time robust model predictive control for continuous-time nonlinear systems. In: Proc. American Control Conf. IEEE. ; 2015: 924–930.
  * 39 Villanueva ME, Li JC, Feng X, Chachuat B, Houska B. Computing ellipsoidal robust forward invariant tubes for nonlinear MPC. IFAC-PapersOnLine 2017; 50(1): 7175–7180.
  * 40 Alexis K, Papachristos C, Siegwart R, Tzes A. Robust model predictive flight control of unmanned rotorcrafts. J. of Intelligent & Robotic Systems 2016; 81(3-4): 443–469.
  * 41 Löfberg J. YALMIP: A toolbox for modeling and optimization in MATLAB. In: Proc. IEEE Int. Symp. on Comp. Aided Cont. Syst. Design. IEEE. ; 2004: 284–289.
  * 42 Gurobi Optimization, LLC . Gurobi Optimizer Reference Manual. https://www.gurobi.com; 2022.
  * 43 Kvasnica M, Grieder P, Baotić M. Multi-Parametric Toolbox (MPT).; 2004.
  * 44 Hu H, Feng X, Quirynen R, Villanueva ME, Houska B. Real-time tube MPC applied to a 10-state quadrotor model. In: Proc. American Control Conf. IEEE. ; 2018: 3135–3140.

##  Appendix A Quadcopter model

The quadcopter states and inputs are represented in two different coordinate
systems, e.g., earth and body fixed frame, see Fig. 4. The resulting nonlinear
dynamics are given by 40:

$\displaystyle\begin{bmatrix}m_{t}\mathbf{I}_{3\times 3}&{0}\\\
{0}&\mathbf{I}\end{bmatrix}\begin{bmatrix}\dot{{V}}\\\
{W}\end{bmatrix}+\begin{bmatrix}{W}\times m_{t}{V}\\\
{W}\times\mathbf{I}{W}\end{bmatrix}=\begin{bmatrix}{F}\\\ {T}\end{bmatrix},$
(21)

where $m_{t}$ and ${I}$ are the mass and inertia matrix, ${V}$ and ${W}$ are
the linear and angular velocities expressed in the body-fixed frame. ${F}$ and
${T}$ are the applied forces and moments. The model is linearized assuming
decoupling of the translational and the attitude dynamics 40, leading to

$\displaystyle\dot{x}_{\text{3D}}$
$\displaystyle=A_{\text{3D}}x_{\text{3D}}+B_{\text{3D}}u_{\text{3D}},$ (22a)
$\displaystyle y_{\text{3D}}$
$\displaystyle=C_{\text{3D}}x_{\text{3D}}+D_{\text{3D}}u_{\text{3D}},$ (22b)

with:
$x_{\text{3D}}=[x^{\top}_{\text{lon}}\,x^{\top}_{\text{lat}}\,x^{\top}_{\text{alt}}]^{\top},\quad
u_{\text{3D}}=[u^{\top}_{\text{lon}}\,u^{\top}_{\text{lat}}\,u^{\top}_{\text{alt}}]^{\top},$
$A_{\text{3D}}=\left[\begin{smallmatrix}A_{\text{lon}}&0&0\\\
0&A_{\text{lat}}&0\\\ 0&0&A_{\text{alt}}\end{smallmatrix}\right],$
$B_{\text{3D}}=\left[\begin{smallmatrix}B_{\text{lon}}&0&0\\\
0&B_{\text{lat}}&0\\\ 0&0&B_{\text{alt}}\end{smallmatrix}\right].$ The
corresponding longitudinal, lateral and vertical sub-dynamics are given by:

$\displaystyle\dot{x}_{\text{lon}}$
$\displaystyle=A_{\text{lon}}x_{\text{lon}}+B_{\text{lon}}u_{\text{lon}},$
(23a) $\displaystyle\dot{x}_{\text{lat}}$
$\displaystyle=A_{\text{lat}}x_{\text{lat}}+B_{\text{lat}}u_{\text{lat}},$
(23b) $\displaystyle\dot{x}_{\text{alt}}$
$\displaystyle=A_{\text{alt}}x_{\text{alt}}+B_{\text{alt}}u_{\text{alt}},$
(23c)

with the matrices $A_{\text{lon}}=\left[\begin{smallmatrix}0&1&0&0\\\
0&-\lambda_{x}&-g&0\\\ 0&0&0&1\\\
0&0&-a_{w_{x},\theta}&-a_{w_{x},w_{x}}\end{smallmatrix}\right],$
$B_{\text{lon}}=\left[\begin{smallmatrix}0\\\ 0\\\ 0\\\
b_{x}\end{smallmatrix}\right],$ where longitudinal state is
$x_{\text{lon}}=[p_{x}\,v_{x}\,\theta\,w_{x}]^{\top}$, with the input
$u_{\text{lon}}=\theta_{c}$. The lateral state is
$x_{\text{lat}}=[p_{y}\,v_{y}\,\phi\,w_{y}]^{\top}$, with the input
$u_{\text{lat}}=\phi_{c}$ and the matrices
$A_{\text{lat}}=\left[\begin{smallmatrix}0&1&0&0\\\ 0&-\lambda_{y}&g&0\\\
0&0&0&1\\\ 0&0&-a_{w_{y},\phi}&-a_{w_{y},w_{y}}\end{smallmatrix}\right],$
$B_{\text{lat}}=\left[\begin{smallmatrix}0\\\ 0\\\ 0\\\
b_{y}\end{smallmatrix}\right].$ Finally, the matrices for vertical (altitude)
dynamics are given by $A_{\text{alt}}=\left[\begin{smallmatrix}0&1\\\
0&-\lambda_{z}\end{smallmatrix}\right],$
$B_{\text{alt}}=\left[\begin{smallmatrix}0\\\
b_{z}\end{smallmatrix}\right],$with the vertical state
$x_{\text{alt}}=[p_{z}\,v_{z}]^{\top}$, and the input is
$u_{\text{alt}}=T_{c}$. The overall UAV states are the roll and pitch angles
$(\phi,\theta)$, roll and pitch rates $(w_{x},w_{y})$, the positions
$(p_{x},p_{y},p_{z})$, and the velocities $(v_{x},v_{y},v_{z})$. The
parameters
$\lambda_{x},\lambda_{y},\lambda_{z},a_{w_{x},\theta},a_{w_{x},w_{x}},a_{w_{y},\phi},a_{w_{y},w_{y}},b_{x},b_{y},b_{z}$
can be found in 40.


smequation

$\BODY$ (1)

smequation*

$\BODY$

# SIDER : Single-Image Neural Optimization for Facial Geometric Detail
Recovery

Aggelina Chatziagapi1 Equal Contribution  ShahRukh Athar1††footnotemark:
Francesc Moreno-Noguer2 Dimitris Samaras1

1Stony Brook University
2Institut de Robòtica i Informàtica Industrial, CSIC-UPC

###### Abstract

We present SIDER (Single-Image neural optimization for facial geometric DEtail
Recovery), a novel photometric optimization method that recovers detailed
facial geometry from a single image in an unsupervised manner. Inspired by
classical techniques of coarse-to-fine optimization and recent advances in
implicit neural representations of 3D shape, SIDER combines a geometry prior
based on statistical models and Signed Distance Functions (SDFs) to recover
facial details from single images. First, it estimates a coarse geometry using
a morphable model represented as an SDF. Next, it reconstructs facial geometry
details by optimizing a photometric loss with respect to the ground truth
image. In contrast to prior work, SIDER does not rely on any dataset priors
and does not require additional supervision from multiple views, lighting
changes or ground truth 3D shape. Extensive qualitative and quantitative
evaluation demonstrates that our method achieves state-of-the-art on facial
geometric detail recovery, using only a single in-the-wild image.

##  1 Introduction

The study of the 3D geometry of the human face is a problem of great interest
in computer graphics and vision communities. Early approaches to recover the
3D structure of the human face were based on morphable models [1], where the
face shape, expression and texture are optimized with respect to a given
image. However, due to the low dimensionality of the identity and expression
subspaces, morphable models are unable to capture facial geometric details,
such as wrinkles and skin-folds. With the advent of deep learning, it has now
become possible to train networks on large datasets [5, 24, 21], in order to
regress the 3D shape of the face along with its geometric details. Still, the
generalization ability of these methods is limited, mainly due to the lack of
diversity of the data they are trained on [7].

Recent advances in implicit neural representations of 3D shape [29, 22, 8]
have made it possible to learn rich details of the 3D geometry of an object.
The geometry can be represented by Signed Distance Functions (SDFs) [29, 22,
8], Unsigned Distance Functions (USDFs) [3] or occupancies [15] that are
parameterized using Multilayer Perceptrons (MLPs). The high representational
power of MLPs, along with the use of positional encoding [16], facilitates the
reconstruction of rich geometric details. Additionally, if the geometry is
represented as an SDF, recent works [17, 29] have proposed ways that allow the
cheap calculation of derivatives of the geometry through implicit
differentiation, making gradient learning significantly more tractable. These
implicit representations are learnt using either multi-view supervision [29,
17] or partial 3D data [22, 8]. However, in the absence of 3D data or multi-
view supervision, like when one has access only to a single face image, it
might not be possible to train such models as they may collapse into trivial
solutions.

In this work, we propose SIDER, a novel photometric optimization method that
recovers facial geometric details from a single face image. SIDER uses an
unsupervised coarse-to-fine optimization scheme that does not require any
ground truth 3D, multi-view or varying light-source supervision. Since
optimization of SDFs using a single image is prone to trivial solutions, SIDER
first learns a coarse approximation of the 3D face geometry using a morphable
model fit to the input image by a standard landmark fitting pipeline. Next,
this SDF is optimized by minimizing the photometric loss with respect to the
given image via implicit differentiation [29, 17]. After converging, SIDER
outputs an SDF that represents the 3D face shape along with its geometric
details (see Fig 1). We show, both quantitatively and qualitatively, that
SIDER significantly outperforms the current state-of-the-art in detailed face
reconstruction by recovering facial geometric details that are realistic and
have high fidelity to the input image.

To summarize, our contributions are as follows:

  * •

We propose SIDER, a method that recovers facial geometric details from a
single face image in an unsupervised manner.

  * •

We propose a novel coarse-to-fine optimization scheme that leverages a
classical morphable model representation as a prior to prevent degenerate
solutions of the SDF and is optimized using an unsupervised photometric loss.

  * •

We achieve state-of-the-art performance in facial geometry reconstruction from
single in-the-wild images.

##  2 Related Work

In this section, we describe recent related works in facial geometry
estimation, facial geometric details recovery and implicit representations of
3D shapes.

Figure 1: Overview of SIDER: Left: First SIDER learns a coarse geometry,
represented as an SDF, by using FLAME mesh [13] fit to the input image as
supervision Right: Next, it recovers facial geometric details by optimizing
the SDF w.r.t. the photometric loss on the input image.

Facial Geometry Estimation. One of the first widely used methods for 3D shape
reconstruction of the human face was based on statistical 3D face models that
can fit a given image, going back to the original 3D Face Morphable Model [1]
(3DMMs). However, the underlying PCA-based representation for shape and
expression of 3DMMs is not flexible enough to represent fine facial geometric
details such as wrinkles, skin folds and skin bumps. Recent methods [24, 25,
27, 26, 31, 2, 4, 10, 12, 6] leverage the power of deep-learning and large-
scale image and video datasets to regress parameters that generate realistic
3D reconstructions or learn complex representations for face shape, expression
and texture. These methods produce more realistic results than traditional
3DMM fitting, but they still cannot capture fine facial details and need large
datasets to train on. In contrast, we propose a neural coarse-to-fine
optimization scheme to recover facial geometric details from single images and
no-supervision.

Geometric Facial Details Estimation. The past few years have witnessed a
significant improvement in the realism of reconstructed 3D face geometry and
the quality of facial details. In [19], a CNN (CoarseNet) first regresses a
rough geometry of the face, and then another CNN (FineNet) estimates facial
details using a coarse depth map and input images. In [21], the regressed
correspondence and depth maps are registered onto a template mesh, which is
further refined to generate the detailed facial geometry. In [28] facial
details are modelled with bump maps on top of a 3DMM base. DF2Net [30] uses
multiple refinement steps to reconstruct detailed geometric structure. First,
a coarse depth is predicted, which is then refined by an F-Net. The refined
depth along with the input image is then given as input to a specially
designed Finer-Net that outputs the final recovered facial details. DECA [5]
uses a differentiable renderer to perform 3D face reconstruction and recover
the detailed face geometry. The facial geometric details are represented as a
UV-map of vertex displacements of a FLAME mesh[13]. DECA is trained on a large
dataset of close to 2 million images with a subset of them being paired. In
contrast to the aforementioned methods, SIDER does not require an exorbitantly
large dataset for training and can be used on single in-the-wild images.

Implicit Neural Representations. Representing 3D shapes implicitly with neural
networks, more specifically using Multi-Layer Perceptrons (MLPs), has led to
the development of methods that are able to reconstruct a large variety of 3D
shape with rich details [29, 22, 8, 15, 17, 14, 23, 18, 3]. In [18], authors
train an SDF using instance specific meshes. Once trained, a latent space for
the instances is learnt, making it possible to sample a large variety of
shapes represented as SDFs. In [8], the authors propose a regularizer, the
eikonal constraint, that allows learning SDFs from sparse 3D samples in the
form of a point cloud. The authors of [29] use the eikonal constraint proposed
by [8] to learn 3D shapes of objects using multi-view supervision along with
object masks. They also propose an expression for the derivative of the
intersection point of sphere tracing with respect to the MLP parameters that
matches the real derivative up to the first order. SIDER uses the expression
for the derivative of the intersection point proposed by [29], in order to
back-propagate gradients to its geometry network and recover the facial
geometric details. However, unlike the aforementioned methods, SIDER leverages
on a prior estimated from statistical models, which constrains the
optimization and allows recovery of facial geometric details from single
images without any multi-view supervision or ground truth 3D data.

##  3 SIDER

Figure 2: SIDER in action: Here we show the intermediate output of SIDER as it
is trained. The first column contains an overlay of the initial FLAME [13]
geometry on top of the image (top row) and the normals of the initial
geometry. The subsequent columns show the rendered RGB (top row) and the
normals of the learnt geometry (bottom row) as SIDER is trained. Once
converged, we get a facial geometry with high quality details (last column,
bottom row) and its photrealistic rendering (last column, top row).

Given a single image, SIDER aims to extract from it facial geometric details,
such as wrinkles and skin folds. SIDER uses a two-stage neural optimization
approach to extract these details. In the first stage, we leverage the FLAME
morphable model [13] prior, in order to learn the coarse geometry of the face,
which is represented as an SDF. Next, we optimize this SDF w.r.t the
photometric loss of the provided image, $\mathbf{I}$, in order to learn the
facial geometric details. In the following, we elaborate the workings and the
training process of SIDER.

###  3.1 Architecture

As shown in the overview Fig. 1, SIDER consists of two MLPs: a geometry
network $f_{\theta}$ and a rendering network $g_{\omega}$. The geometry MLP,
$f_{\theta}(\cdot)$ represents the face shape (along with the facial geometric
details) as an SDF. More specifically, for any point $\mathbf{x}$:

$f_{\theta}(\mathbf{x})=\\{f_{\theta}^{\text{SDF}}(\phi(\mathbf{x})),\boldsymbol{\gamma}_{f}^{\mathbf{x}}\\}$
(2)

where $f_{\theta}^{\text{SDF}}(\mathbf{x})$ is the SDF predicted at
$\mathbf{x}$, $\boldsymbol{\gamma}_{f}^{\mathbf{x}}$ is a feature vector
predicted at $\mathbf{x}$ that is used as input to the rendering network
$g_{\omega}$, and $\phi$ is the positional encoding.

The rendering network, $g_{\omega}(\cdot)$, predicts the RGB value of a point
$\mathbf{x}$ as follows:

$g_{\omega}(\phi(\mathbf{x}),\mathbf{n}_{\mathbf{x}},\phi(\mathbf{v}),\boldsymbol{\gamma}_{f}^{\mathbf{x}})=\\{R,G,B\\}$
(3)

where $\mathbf{x}$ are the point’s coordinates, $\mathbf{n}_{\mathbf{x}}$ is
the normal at point $\mathbf{x}$, $\mathbf{v}$ is the viewing direction and
$\boldsymbol{\gamma}_{f}^{\mathbf{x}}=f_{\theta}(\mathbf{x})$ is a feature
vector predicted by the geometry network $f_{\theta}$ at $\mathbf{x}$.

###  3.2 Learning the coarse geometry

Given a face image $\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$, SIDER first
learns a coarse geometry of the face using a morphable model prior. FLAME is
fit to the face in $\mathbf{I}$ using standard landmark fitting [5, 9]

$\underset{\boldsymbol{\alpha}_{\text{shape}},\boldsymbol{\alpha}_{\text{exp}},\boldsymbol{\alpha}_{\text{pose}},\textbf{cam}}{\text{min}}||L_{FLAME}^{i}-L_{gt}^{i}||;\quad\forall
i$ (4)

where $L_{FLAME}^{i}$ is the position of the $i$’th landmark of the FLAME
model [13], $L_{gt}^{i}$ is the position of the $i$’th landmark predicted by
3DDFA [9] (which we treat as ground truth),
$\boldsymbol{\alpha}_{\text{shape}},\boldsymbol{\alpha}_{\text{exp}},\boldsymbol{\alpha}_{\text{pose}}$
are the shape, expression and pose parameters of the FLAME model [13] and cam
are the camera parameters. Once FLAME [13] is fit to the image, we train an
MLP, $f_{\theta}$, to represent this coarse mesh as an SDF by minimizing the
following:

$\text{Loss}_{geo}(\mathbf{x})=||f_{\theta}^{\text{SDF}}(\phi(\mathbf{x}))-\text{SDF}_{GT}(\mathbf{x})||;\quad\forall\mathbf{x}\in\mathcal{P}$
(5)

where $\mathcal{P}$ is a set of randomly chosen points in space in the
neighborhood of the FLAME mesh, $\mathbf{x}=\\{x,y,z\\}$ is a point in space,
$\phi$ is the positional encoding and $\text{SDF}_{GT}(\cdot)$ is the ground
truth SDF to the coarse mesh. Since the FLAME face mesh is an open and single
surface layer, an SDF cannot be defined on it directly (there is no region
where the distance is negative, since there is no ‘inside’). Therefore, in
order to define the SDF, we consider the face mesh to be volume of ‘thickness’
$\boldsymbol{\epsilon}$. This allows us to define the ground-truth SDF as
follows

$\text{SDF}_{GT}(\mathbf{x})=\text{Point2Mesh}(\mathbf{x})-\frac{\boldsymbol{\epsilon}}{2}$
(6)

where Point2Mesh is the point-to-mesh distance function and
$\boldsymbol{\epsilon}$ is a small number denoting the thickness of the mesh.
Additionally, the geometry network is regularized using the eikonal loss:

$\text{Loss}_{eik}(\mathbf{x})=\mathbb{E}_{\mathbf{x}}(||\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})||-1)^{2}$
(7)

The full loss on the geometry network is

$\text{Loss}^{f_{\theta}}_{coarse}(\mathbf{x})=\text{Loss}_{geo}(\mathbf{x})+\lambda\text{Loss}_{eik}(\mathbf{x})$
(8)

where $\lambda$ is a regularization coefficient.

Figure 3: Ablation Study: Here we ablate the contributions of the normals and
the feature vector as input to the gradient network (see Eq. (10)). The first
column is the image on which SIDER is optimized. (a) Geometry learnt without
using the feature vector as input. (b) Geometry learnt without using the
normals as input, (c) Geometry learnt without using both feature vector and
normals as input. The last column shows the result of SIDER that uses both the
normals and the feature vector as input to the rendering network.

###  3.3 Recovering the Facial Geometric Details

Once $f_{\theta}$ has been trained to approximate the coarse geometry, we use
the provided image, $\mathbf{I}$, to fine-tune $f_{\theta}$ and recover the
facial geometric details. In order to render the SDF $f_{\theta}$, we use
sphere-tracing along with a rendering network $g_{\omega}$. Rays are shot into
the scene from the camera center $\mathbf{o}$, and the intersection of these
rays with the face mesh is estimated using sphere-tracing and implicit
differentiation [29]. The colors of the intersecting points are predicted
using $g_{\omega}$. More specifically, consider a ray,
$\mathbf{r}=\mathbf{o}+\mathbf{v}t$, with viewing direction $\mathbf{v}$ and
surface intersection point $\hat{\mathbf{x}}$. The RGB color at
$\hat{\mathbf{x}}$ is calculated as follows:

$\mathbf{c_{\hat{\mathbf{x}}}}=g_{\omega}(\phi(\hat{\mathbf{x}}),\mathbf{n}_{\hat{\mathbf{x}}},\phi(\mathbf{v}),\boldsymbol{\gamma}_{f}^{\hat{\mathbf{x}}})$
(9)

where $\mathbf{c_{\hat{\mathbf{x}}}}$ is the predicted RGB color at point
$\hat{\mathbf{x}}$, $\mathbf{n}_{\hat{\mathbf{x}}}$ is the normal at point
$\hat{\mathbf{x}}$ and $\boldsymbol{\gamma}_{f}^{\hat{\mathbf{x}}}$ is a
feature vector predicted by the geometry network, $f_{\theta}$ at
$\hat{\mathbf{x}}$. The facial geometric details are recovered by jointly
optimizing the geometry network $f_{\theta}$ and the rendering network
$g_{\omega}$ with respect to the photometric loss as follows:

$\underset{\theta,\omega}{\text{min
}}\text{Loss}_{color}(\mathbf{c_{\hat{\mathbf{x}}}},\mathbf{c}_{\mathbf{p}}^{gt})=||\mathbf{c_{\hat{\mathbf{x}}}}-\mathbf{c}_{\mathbf{p}}^{gt}||$
(10)

where $\mathbf{c}_{\mathbf{p}}^{gt}$ is the ground truth pixel color. The
gradients to the geometry network $f_{\theta}$, are calculated using implicit
differentiation [29]. Additionally, in order to ensure that $f_{\theta}$ does
not drift too far away from the face shape, it is regularized using the coarse
SDF from Sect 3.2. The complete loss of the geometry network is:

$\begin{split}\text{Loss}^{f_{\theta}}_{detail}(\mathbf{c_{\hat{\mathbf{x}}}},\mathbf{c}_{\mathbf{p}}^{gt})=&||\mathbf{c_{\hat{\mathbf{x}}}}-\mathbf{c}_{\mathbf{p}}^{gt}||\\\
&+\lambda_{1}||f_{\theta}(\phi(\mathbf{x}))-\text{SDF}_{GT}(\mathbf{x})||\\\
&+\lambda_{2}\text{Loss}_{eik}\end{split}$ (11)

where $\lambda_{1},\lambda_{2}$ are the regularization coefficients.

##  4 Experiments

In this section, we evaluate SIDER’s ability to recover facial geometric
details from single in-the-wild images. We present both qualitative and
quantitative results, and compare our proposed approach to the current state
of the art.

Figure 4: Reconstructions by SIDER: Above we show the results of SIDER on
images from the FFHQ [11] dataset. The first column is the input image with
respect to which SIDER is optimized. The second column is the overlay of
coarse FLAME [13] mesh on the input image. The third column is the render of
the detailed geoemtry learnt by SIDER. The fourth column is the depth. The
fifth column are the normals of the deetailed face geometry learnt by SIDER
and the last column is the overlay of the learnt detailed geometry on top of
the input image. Figure 5: Cumulative error plot. Reconstruction error on the
NoW validation set for the different methods. Figure 6: Comparison to state-
of-the-art methods: Here we compare again prior art: Pix2vertex [21], DF2Net
[30], FLAME fitting [13] and DECA (w/ details) [5]. Input images taken from
FFHQ. Figure 7: Detailed comparison to DECA (w/ details) [5]: Here we compare
the facial geometric details generated by DECA (red insets) and SIDER (green
insets). For the image in the top row, we see that SIDER faithfully recovers
the details on the forehead, around the left eye and around the left corner of
the lip. In contrast, DECA [5] smoothens the details on the forehead and fails
to recover the details around the left eye and the left corner of the lip.
Similarly, for the image in row 2, SIDER is able to accurately recover the
details on the forehead and around the left eye, while DECA [5] fails to do
so.

###  4.1 Implementation Details

SIDER learns detailed facial geometry given only a single image. We perform
extensive experiments on images from three publicly available datasets, namely
FFHQ [11], ALFW2000 [32], and the NoW challenge dataset [20]. We choose
challenging images that contain both subtle and more complicated facial
geometric details, including wrinkles and skin folds. The images are resized
to 256x256 resolution before fitting the FLAME model.

The geometry network, $f_{\theta}(\cdot)$, consists of 8 linear layers and a
skip connection from the input to the fourth layer, similarly to [29]. The
rendering network, $g_{\omega}(\cdot)$, consists of 4 linear layers. Its input
is non-linearly mapped to learn high frequencies [16]. Each layer of both MLPs
includes 512 hidden units. We first train the geometry network for 1000 epochs
to learn the coarse geometry. Then, we fine-tune it via photometric
optimization and jointly train the rendering network for around 200-300
epochs, until the loss is not decreasing further. We use Adam optimizer with a
learning rate of $10^{-4}$.

###  4.2 Ablation Study

Gradients from the photometric loss of Eq. (10) are back-propagated into the
geometry network $f_{\theta}$ via the feature vector, the normals and the
intersection point $\hat{x}$ (see [29]). In this section, we ablate the
contribution of the feature vector and the normals by alternatively removing
one or the other, and finally both, from the input to the rendering network
$g_{\omega}$. Fig. 3 shows the results of the ablation study, the first image
from the left is the input image on which SIDER was trained, (a) shows the
results when the feature vector is removed from the input to $g_{\omega}$, (b)
shows the results when the normals are removed from the input, (c) shows the
results when both are removed, and the last image shows the results of the
full model. Without the feature vector as input (see Fig. 3 (a)) the geometry
network does not recover any facial geometric details and the geometry is very
smooth. In contrast, without the normals as input (see Fig. 3 (b)), the
geometry network does seem to recover some details but the overall
reconstruction suffers significantly. Without either the normals or the
feature vector as input (see Fig. 3 (c)) learning completely fails. Using both
the normals and the feature vector (see the last column of Fig. 3) as input to
the rendering network, $g_{\omega}$, allows SIDER to recover high quality
facial geometric details without compromising overall reconstruction quality.

###  4.3 Quantitative Evaluation

We evaluate the accuracy of the detailed reconstruction of SIDER on the NoW
challenge dataset [20]. We compare it with the performance of recent state-of-
the-art methods, namely Pix2vertex [21], DF2Net [30], and DECA [5]. We choose
NoW for our quantitative evaluation, since it includes 3D ground truth scans
and provides a standard evaluation protocol. It measures the distance from all
reference scan vertices to the closest point in the reconstructed mesh
surface, after rigidly aligning scans and reconstructions. Because Pix2vertex
[21] and DF2Net [30] reconstruct a smaller part of the face than the other
methods, we ensure that the ground truth face scan is cropped to a circular
area (same for all the methods) that would not exceed the smallest
reconstructed mesh.

As shown on Table 1, SIDER outperforms the current state-of-the-art, i.e. DECA
[5], and other methods by a healthy margin. The mean and median error of SIDER
is 0.89 mm and 0.66 mm respectively; both of which are less than that of DECA
[5], which has a median error 1.19 mm and mean error of 1.47 mm. Additionally,
the standard deviation of SIDER is lower than that of DECA [5], 0.88 mm as
compared to 1.25 mm of DECA [5]. Similarly, the cumulative error plots in Fig.
5 demonstrate the significant advantage of SIDER compared to the other
methods.

Method | Median (mm) | Mean (mm) | Std (mm)
---|---|---|---
Pix2vertex [21] | 1.93 | 2.74 | 2.85
DF2Net [30] | 1.84 | 2.51 | 3.40
DECA [5] | 1.19 | 1.47 | 1.25
SIDER (Ours) | 0.66 | 0.89 | 0.88
Table 1: Quantitative results. Reconstruction error on the NoW validation set.

###  4.4 Qualitative Evaluation

We next provide an extensive qualitative evaluation of SIDER on in-the-wild
images from datasets for which no ground truth is available.

In Fig. 4, we demonstrate qualitative results of SIDER on images from the FFHQ
dataset [11]. The columns of the figure show correspondingly: the input image,
the FLAME fitting, the learnt render (RGB output), the depth, the normals, and
finally the overlay of the reconstructed face, with geometric details, on top
of the original image. As can be seen, SIDER recovers accurately the geometric
details for each input image. For example, in the first row of Fig. 4, the
geometric details arising on the left side of the forehead due to the raised
eyebrow are captured faithfully. Similarly, in the second row we see the
wrinkling around the mouth is realistically recovered.

In Fig. 2, we show the RGB output and normals during the joint training of the
geometry and rendering networks. We can see how the networks gradually learn a
detailed facial geometry represented as an SDF.

In Fig. 6, we qualitatively compare the results of SIDER to state-of-the-art
detailed facial reconstruction methods, namely Pix2vertex [21], DF2Net [30],
and DECA [5]. The first column corresponds to the input image, the second
column contains the results of Pix2vertex [21], the third contains the results
of DF2Net [30], the fourth column contains the results of simple FLAME fitting
[13] (used as ground truth of the coarse geometry during the first stage), the
fifth column contains the results of DECA with details [5], the current state-
of-the-art, and the last column contains the results of our method. Note that
DECA results are masked with FLAME, in order to illustrate only the face
region for fair comparison with the other methods. As can be seen, SIDER is
able to recover significantly more detail than the other methods. Pix2vertex
[21] generates detailed reconstructions that are quite smooth and misses large
visible geometric details such at the skin fold on the right side of the lip
of the image in row 1 or the wrinkles on the forehead of the image in row 4.
The results of DF2Net [30], while recovering greater geometric details than
Pix2vertex [21], are still prone to errors. For example, it misses the
wrinkles on the forehead of the image in row 2 and recovers incorrect details
on the forehead from the image in row 4. The produced meshes also contain
artifacts, e.g. on the right part of the head in rows 1 and 2, or include
extreme curvature, e.g. in the eyes region. The face shape is also affected by
lighting to a great extent (e.g. shaded part in row 2).

DECA [5], the current state-of-the-art in facial geometric details recovery,
generates better reconstructions than both Pix2vertex [21] and DF2Net [30],
however it is unable to recover fine geometric details. For example, the skin
fold on the left side of the mouth of the image in row 1 is captured by both
our method, SIDER, and DF2Net [30]. In contrast, DECA [5] is unable to recover
it. Similarly, the geometric details under the left eye of the image in row 2
are not recovered by DECA [5]. SIDER, however, is able to accurately recover
them. The skin folds around the eyes and around the lips of the image in row 3
are faithfully recovered by SIDER, but DECA [5] is unable to reconstruct them.
In Fig. 7, we show a direct comparison of the facial geometric details
generated by SIDER and DECA [5]. Insets in green show details generated by
SIDER and insets in red show details generated by DECA [5]. For the image in
row 1, we see that SIDER is faithfully able to recover the details on the
forehead, around the eyes, and around the mouth. In contrast, the details
recovered by DECA [5] are over-smoothed and inaccurate. Similarly, for the
image in row 2, we see that the details recovered by SIDER have greater
fidelity to the input than the details recovered by DECA [5].

In summary, as can be seen from the qualitative results in both Fig. 4, Fig. 7
and Fig. 6, SIDER is able to reconstruct high-quality facial geometric details
from single images, more accurately and with greater fidelity to the input
image than competing methods.

##  5 Conclusion

In this work we present SIDER, a method for high-fidelity detailed 3D face
reconstruction from a single image that can be trained in an unsupervised
manner. Our approach combines the best from classical statistical models and
recent implicit neural representations. The former is used to obtain a coarse
shape prior, and the latter provides high frequency detail of the geometry, by
only optimizing over a photometric loss computed w.r.t. the input image. A
thorough quantitative and qualitative evaluation, shows that SIDER outperforms
current state-of-the-art by a significant margin. A limitation of our current
approach is that it still cannot handle details like hair or beards. This is
because the photometric loss for these regions would require sub-pixel
accuracy. In the future, we will explore alternatives for addressing this type
of high-frequency details.

## References

  * [1] Volker Blanz, Thomas Vetter, et al.  A morphable model for the synthesis of 3d faces.  1999\.
  * [2] James Booth, Epameinondas Antonakos, Stylianos Ploumpis, George Trigeorgis, Yannis Panagakis, and Stefanos Zafeiriou.  3d face morphable models” in-the-wild”.  In CVPR. IEEE, 2017.
  * [3] Julian Chibane, A. Mir, and Gerard Pons-Moll.  Neural unsigned distance fields for implicit function learning.  In NeurIPS, 2020.
  * [4] Pengfei Dou, Shishir K Shah, and Ioannis A Kakadiaris.  End-to-end 3d face reconstruction with deep neural networks.  In CVPR, 2017.
  * [5] Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart.  Learning an animatable detailed 3D face model from in-the-wild images.  volume 40, 2021.
  * [6] Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, and William T. Freeman.  Unsupervised training for 3d morphable model regression.  In CVPR, 2018.
  * [7] Markos Georgopoulos, Yannis Panagakis, and Maja Pantic.  Investigating bias in deep face analysis: The kanface dataset and empirical study, 2020.
  * [8] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman.  Implicit geometric regularization for learning shapes.  In ICML, pages 3569–3579. 2020.
  * [9] Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Z Li.  Towards fast, accurate and stable 3d dense face alignment.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [10] Aaron S Jackson, Adrian Bulat, Vasileios Argyriou, and Georgios Tzimiropoulos.  Large pose 3d face reconstruction from a single image via direct volumetric cnn regression.  In ICCV, 2017.
  * [11] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, 2019.
  * [12] Hyeongwoo Kim, Michael Zollhöfer, Ayush Tewari, Justus Thies, Christian Richardt, and Christian Theobalt.  Inversefacenet: Deep monocular inverse face rendering.  In CVPR, 2018.
  * [13] Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4D scans.  ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6), 2017\.
  * [14] Shichen Liu, S. Saito, Weikai Chen, and H. Li.  Learning to infer implicit surfaces without 3d supervision.  In NeurIPS, 2019.
  * [15] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In CVPR, 2019.
  * [16] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  2020\.
  * [17] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In CVPR, 2020.
  * [18] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In CVPR, June 2019.
  * [19] Elad Richardson, Matan Sela, Roy Or-El, and R. Kimmel.  Face reconstruction from a single image.  In CVPR, 2017.
  * [20] Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael Black.  Learning to regress 3d face shape and expression from an image without 3d supervision.  In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [21] Matan Sela, Elad Richardson, and Ron Kimmel.  Unrestricted facial geometry reconstruction using image-to-image translation.  In ICCV, 2017.
  * [22] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  In NeurIPS, 2020.
  * [23] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler.  Neural geometric level of detail: Real-time rendering with implicit 3D shapes.  2021\.
  * [24] Ayush Tewari, Florian Bernard, Pablo Garrido, Gaurav Bharaj, Mohamed Elgharib, Hans-Peter Seidel, Patrick Pérez, Michael Zöllhofer, and Christian Theobalt.  Fml: Face model learning from videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10812–10822, 2019.
  * [25] Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Pérez, and Christian Theobalt.  Self-supervised multi-level face model learning for monocular reconstruction at over 250 hz.  In CVPR, 2018.
  * [26] Luan Tran, Feng Liu, and Xiaoming Liu.  Towards high-fidelity nonlinear 3d face morphable model.  In In Proceeding of IEEE Computer Vision and Pattern Recognition, Long Beach, CA, June 2019.
  * [27] Luan Tran and Xiaoming Liu.  Nonlinear 3d face morphable model.  In IEEE Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, June 2018.
  * [28] Anh Tu$\hat{a}^{\prime}$n Tr$\hat{a}$n, Tal Hassner, Iacopo Masi, Eran Paz, Yuval Nirkin, and Gérard Medioni.  Extreme 3d face reconstruction: Seeing through occlusions.  In CVPR, 2018.
  * [29] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman.  Multiview neural surface reconstruction by disentangling geometry and appearance.  Advances in Neural Information Processing Systems, 33, 2020.
  * [30] Xiaoxing Zeng, Xiaojiang Peng, and Yu Qiao.  Df2net: A dense-fine-finer network for detailed 3d face reconstruction.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2315–2324, 2019.
  * [31] Wenbin Zhu, HsiangTao Wu, Zeyu Chen, Noranart Vesdapunt, and Baoyuan Wang.  Reda:reinforced differentiable attribute for 3d face reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [32] Xiangyu Zhu, Zhen Lei, Junjie Yan, Dong Yi, and Stan Z. Li.  High-fidelity pose and expression normalization for face recognition in the wild.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 787–796, 2015.

##  SIDER : Single-Image Neural Optimization for Facial Geometric Detail
Recovery
\- Supplementary -

Figure 8: Reconstructions by SIDER: SIDER is a novel photometric optimization
method that recovers, from a single image, detailed facial geometry without
any 3D, multi-view or multiple image supervision. As shown above, the details
recovered by SIDER such as wrinkles, skin folds and skin bumps are realistic
and have high fidelity to the input image. Please note the enlarged insets of
the geometric details on the forehead and the eyes (both on the center and on
the right). The image is taken from NoW dataset [20]

##  6 More Reconstruction Results

In this section, we provide more reconstruction results by SIDER. In Fig. 8
and Fig. 9, we show sample reconstructions of SIDER from the NoW dataset [20]
and the FFHQ dataset [11] respectively. In Fig. 8, we see that the geometric
details on the forehead and on around the eyes (both on the center and on the
right) are accurately recovered. In Fig. 9, we see that the geometric details
on the forehead and on around the mouth are accurately recovered.

In Fig. 10 and Fig. 11, we compare the results of SIDER with prior art. In the
second, third and seventh row of Fig. 10, we see that SIDER is accurately able
to recover the geometric details around the mouth. In the fourth and and
eighth row of Fig. 10, we see that SIDER is accurately able to recover the
geometric details on the forehead. Similarly, in seventh and eighth row of
Fig. 11 we see accurate recovery of facial geometric details by SIDER around
both the forehead and the mouth despite the extreme head-pose. In summary,
SIDER is able to recover realistic facial geometric details with high fidelity
to the input.

Figure 9: Reconstructions by SIDER: As shown above, the details recovered by
SIDER such as wrinkles, skin folds and skin bumps are realistic and have high
fidelity to the input image. Please note the enlarged insets of the geometric
details on the forehead and mouth of all three images. The images are taken
from the FFHQ dataset [11] Figure 10: Comparison to state-of-the-art methods:
Here we compare SIDER against prior art. From left to right we see results of
Pix2vertex [21], DF2Net [30], FLAME fitting [13], DECA (with details) [5] and
SIDER. Blank entries indicate that the particular method did not return any
reconstruction. As can be seen, across all images, SIDER recovers high quality
facial geometric details that have a high fidelity to the input image. Note
the accurate recovery of facial geometric details around the mouth of the
images in the second, third and seventh row. Figure 11: Comparison to state-
of-the-art methods: Here we compare SIDER against prior art. From left to
right we see results of Pix2vertex [21], DF2Net [30], FLAME fitting [13], DECA
(with details) [5] and SIDER. Blank entries indicate that the particular
method did not return any reconstruction. As can be seen, across all images,
SIDER recovers high quality facial geometric details that have a high fidelity
to the input image. Note the accurate recovery of facial geometric details on
the forehead of the images in the seventh and the eighth row despite the
extreme head-pose.