# Two-step dynamic obstacle avoidance

Fabian Hart  Martin Waltz<EMAIL_ADDRESS>Ostap Okhrin

###### Abstract

Dynamic obstacle avoidance (DOA) is a fundamental challenge for any autonomous
vehicle, independent of whether it operates in sea, air, or land. This paper
proposes a two-step architecture for handling DOA tasks by combining
supervised and reinforcement learning (RL). In the first step, we introduce a
data-driven approach to estimate the collision risk of an obstacle using a
recurrent neural network, which is trained in a supervised fashion and offers
robustness to non-linear obstacle movements. In the second step, we include
these collision risk estimates into the observation space of an RL agent to
increase its situational awareness. We illustrate the power of our two-step
approach by training different RL agents in a challenging environment that
requires to navigate amid multiple obstacles. The non-linear movements of
obstacles are exemplarily modeled based on stochastic processes and periodic
patterns, although our architecture is suitable for any obstacle dynamics. The
experiments reveal that integrating our collision risk metrics into the
observation space doubles the performance in terms of reward, which is
equivalent to halving the number of collisions in the considered environment.
Furthermore, we show that the architecture’s performance improvement is
independent of the applied RL algorithm.

###### keywords:

deep reinforcement learning , supervised learning , dynamic obstacle avoidance
, local path planning

[1]organization=Technische Universität Dresden, Chair of Econometrics and
Statistics, esp. in the Transport Sector,city=Dresden, postcode=01062,
country=Germany [2]organization=Center for Scalable Data Analytics and
Artificial Intelligence (ScaDS.AI), city=Dresden/Leipzig, country=Germany

##  1 Introduction

Autonomous vehicles represent a promising avenue for enhancing safety and
optimizing energy and fuel efficiency within traffic systems, making them a
vital element of future sustainable transportation networks (Feng et al.,
2023). While the general public often fixates on self-driving cars (Badue et
al., 2021), the potential benefits of autonomous vehicles extend to other
modes of transportation, such as autonomous vessels (Negenborn et al., 2023)
and aircraft systems (Ribeiro et al., 2020). Regardless of whether a vehicle
is under human or autonomous control, the paramount concern in all traffic
operations remains ensuring safety by preventing collisions with other
vehicles and stationary obstacles (Kuchar and Yang, 2000; Huang et al., 2020).

We focus on entities operating in non-lane-based traffic environments,
irrespective of whether they steer a vessel, fly an aircraft, command a drone,
or control any other vehicle. These entities are referred to as _agents_.
Furthermore, we will call traffic participants close to the agent _obstacles_.
Generally, an agent’s objective is to reach a specific destination safely. A
real-world system typically relies on a perception-planning-control (PPC)
modularization (Sauer et al., 2018; Fossen, 2021). The perception module
includes sensor systems such as LiDAR, radar, or camera to localize the agent
and relevant obstacles in the environment accurately (Liu et al., 2016).
Moreover, considering the noise and uncertainties associated with real-world
sensors, advanced state estimation techniques are often applied (Kandepu et
al., 2008). The planning module processes the information from the perception
module and generates a safe and feasible local path in the form of a set of
waypoints (Marin-Plaza et al., 2018). Crucially, the path planning should
consider the maneuverability of both the agent and the obstacles. Lastly, the
control module translates high-level commands derived from the local path into
low-level actuator control commands (Vagale et al., 2021).

In this paper, we consider the task of dynamic obstacle avoidance (DOA,
Falanga et al. 2020), which we define as the challenge of executing real-time
collision avoidance maneuvers while facing obstacles with non-zero velocity
vectors. Following the formalism of Ribeiro et al. (2020), we consider a
distributed setting where no communication between traffic participants takes
place, which is a practically occurring situation across traffic domains. DOA
can be attributed to local path planning in the PPC modularization, and the
methodological repertoire for DOA is diverse, as shown by the extensive
reviews of Kuchar and Yang (2000), Ribeiro et al. (2020), Pandey et al.
(2017), and Polvara et al. (2018). In the following, we focus on the emerging
method of deep reinforcement learning (DRL, Sutton and Barto 2018) due to its
remarkable successes in recent years in various application domains (Vinyals
et al., 2019; Bellemare et al., 2020; Fawzi et al., 2022). The DRL agent is
trained through trial-and-error interaction with its traffic environment in
our DOA task. In particular, it receives an observation of the traffic
situation, takes a steering or acceleration action, and obtains a numerical
reward signal that should be maximized. We emphasize that several researchers
from different traffic domains have successfully applied DRL approaches to DOA
tasks; see Zhao and Liu (2021), Brittain and Wei (2022), Wang et al. (2019),
Isufaj et al. (2022), Waltz and Okhrin (2023), Waltz et al. (2023), Xu et al.
(2022), and Everett et al. (2021).

Several contributions emphasize the strong dependence of the RL algorithms’
performances on the precise selection of the features of the observation
vector (Mousavi et al., 2016; Suárez-Varela et al., 2019; Li, 2019). These
features build the basis for the agent’s decision-making and should contain
relevant information to handle the task at hand successfully. In DOA tasks,
the observation features frequently consist of the position and velocity of
the agent and nearby obstacles (Everett et al., 2018; Brittain and Wei, 2022;
Everett et al., 2021). In real-world situations, we acknowledge that certain
rule sets, like the Convention on the International Regulations for Preventing
Collisions at Sea, explicitly require estimating collision risk with other
traffic participants (Rule 7, International Maritime Organization 1972).
Leveraging these considerations, we aim to additionally include collision risk
(CR) metrics into the observation of the DRL agent.

We emphasize that a similar approach has been pursued by some prior works
(Chun et al., 2021; Xu et al., 2022; Waltz et al., 2023). However, these
studies share the limitation of assuming a _linear motion_ of nearby obstacles
during the CR computation. For instance, popular CR measures based on the
closest point of approach (CPA, Lenart 1983) like the distance between the
vehicles at the CPA, denoted $d^{\rm CPA}$, and the time until reaching the
CPA, denoted $t^{\rm CPA}$, are calculated assuming constant course and speed
of the obstacle. While these widely used metrics offer great interpretability
and constitute an essential tool for real-world captains and pilots, future
turns or accelerations of the obstacles are not considered, leaving a
significant risk of misinterpreting a given scenario.

Besides, a different strand of literature has focused on supervised learning
approaches to CR estimation (Rawson and Brito, 2023). For example, Strickland
et al. (2018) train a deep predictive model from images for CR assessment in
autonomous driving. Zheng et al. (2020) built a support vector machine
classifier to model the probability of a collision. Further, Zhang et al.
(2020) process a traffic image via a convolutional neural network based on
labels from an analytical collision risk model. The growing capabilities of
supervised learning techniques thus hold considerable promise for CR
estimation.

In this paper, we propose to merge the worlds of traditional highly
interpretable CPA-based measures and supervised CR estimation to construct a
comprehensive DRL agent for DOA tasks. In particular, our contributions are as
follows:

  * •

We propose a _data-driven_ approach to generate an estimate of the $d^{\rm
CPA}$ and $t^{\rm CPA}$ with obstacles. More precisely, we train an LSTM
network (Hochreiter and Schmidhuber, 1997) to estimate the future trajectories
of obstacles in a supervised fashion (LeCun et al., 2015), and compute the CR
metrics subsequently. This procedure can generalize across situations and is
robust to non-linear obstacle behavior.

  * •

We augment the _observation space_ of an RL algorithm to include the newly
estimated CR metrics, enabling the agent to assess and solve various encounter
situations robustly. Moreover, this proposal can be seen in context to recent
advances from the robotics domain (Lee et al., 2019) since our CR estimator
acts as an additional observation processing module.

  * •

We showcase the potential of our _two-step architecture_ (Step 1: CR
estimation, Step 2: DRL) for handling DOA tasks, thereby underscoring its
suitability for implementation inside a local path planning unit of a PPC
modularization. More precisely, we define a traffic-type independent
environment where an agent needs to avoid dynamic obstacles and quantify the
advantage in terms of the reward difference of the proposal in comparison to a
’conventional’ DRL agent for obstacle avoidance. Figure 1 visualizes the
architecture.

The paper is structured as follows: Section 2 provides a concise and formal
problem description of the considered DOA task. Section 3 displays the
estimation of the CR metrics, while the reinforcement learning (RL)
environment and algorithms are presented in Section 4. Section 5 shows the
results, and Section 6 concludes this article. The source code for this paper
is publicly available at GitHub (Waltz and Paulig, 2022).

Figure 1: The proposed two-level architecture for DOA tasks. The supervised
learning module predicts the future trajectories of relevant obstacles while
accounting for possible non-linear motions. These predictions are used to
generate estimates of the $d^{\rm CPA}$ and $t^{\rm CPA}$, which are
concatenated with the remaining features from the environment to build the
observation vector for the DRL agent.

##  2 Problem description

We consider a two-dimensional DOA task detached from a specific transportation
context. We define a point-mass agent with a constant longitudinal speed for
simplification while the lateral acceleration can be controlled. Since this
study aims at a realistic setting with non-linearly moving obstacles, we
generically define an obstacle trajectory as:

$\displaystyle\vec{p}_{t}$ $\displaystyle=\vec{p}_{t,\rm lin}+\vec{p}_{t,\rm
non},$ (1)

where $\vec{p}_{t}=(x_{t},y_{t})^{\top}\in\mathbb{R}^{2}$ is the position of
the obstacle at time step $t$. The terms $\vec{p}_{t,\rm lin}$ and
$\vec{p}_{t,\rm non}$ denote the linear and non-linear components of the
obstacle trajectory, respectively. More precisely, the linear part is defined
as:

$\displaystyle\vec{p}_{t,\rm lin}$
$\displaystyle=\vec{p}_{0}+\dot{\vec{p}}\,t,$ (2)

with the initial obstacle position
$\vec{p}_{0}=(x_{0},y_{0})^{\top}\in\mathbb{R}^{2}$ and the constant,
obstacle-dependent velocity
$\dot{\vec{p}}=(\dot{x},\dot{y})^{\top}\in\mathbb{R}^{2}$. Equation (2)
ensures that the obstacle moves in a specific direction. At the same time, the
non-linear part $\vec{p}_{t,\rm non}$ can capture course deviations due to
environmental effects or the obstacle’s own avoidance maneuvers. Importantly,
different practical behaviors of obstacles can be simulated via different
specifications of $\vec{p}_{t,\rm non}$. Throughout this study, we exemplarily
consider two different kinds of non-linear obstacle movements: _stochastic_
and _periodic_ trajectories.

Stochastic trajectories. The first non-linear specification is based on a
stochastic two-dimensional auto-regressive process (Tsay, 2010):

$\displaystyle A_{t+1}$ $\displaystyle=\phi_{\rm stoch}A_{t}+u_{t,\rm
stoch},\quad\text{where}\quad u_{t,\rm
stoch}\sim\mathcal{N}_{2}\left[\begin{pmatrix}0\\\
0\end{pmatrix},\begin{pmatrix}\sigma_{\rm stoch}^{2}&0\\\ 0&\sigma_{\rm
stoch}^{2}\end{pmatrix}\right],$ (3)

with initial value $A_{0}=(0,0)^{\top}$, auto-regressive parameter $\phi_{\rm
stoch}$, and variance $\sigma_{\rm stoch}^{2}$ of the centered noise stemming
from the multivariate normal distribution. Addressing the noisy nature of the
process, we perform an exponential smoothing of (3):

$\displaystyle A_{t,\rm smooth}$ $\displaystyle=\beta_{\rm
stoch}A_{t}+(1-\beta_{\rm stoch})A_{t-1,\rm smooth},\quad\text{where}\quad
A_{0,\rm smooth}=0,$ (4)

with the smoothing factor $\beta_{\rm stoch}=0.2$. Moreover, to account for
different obstacle velocities, we scale the stochastic trajectory:

$\displaystyle\vec{p}_{t,\rm non}^{\,(1)}=\frac{\|\dot{\vec{p}}\|}{v_{\rm
max}}A_{t,\rm smooth},$ (5)

with maximum obstacle speed $v_{\rm max}$.

Periodic trajectories. The second obstacle behavior considered in this study
incorporates a sinus pattern with additional noise:

$\displaystyle\vec{p}_{t,\rm non}^{\,(2)}=\begin{pmatrix}\dot{y}\\\
-\dot{x}\end{pmatrix}A_{\rm sin}\sin\left({\frac{2\pi t}{T_{\rm
sin}}}\right)+u_{t,\rm sin},\quad\text{where}\quad u_{t,\rm
sin}\sim\mathcal{N}_{2}\left[\begin{pmatrix}0\\\
0\end{pmatrix},\begin{pmatrix}\sigma_{\rm sin}^{2}&0\\\ 0&\sigma_{\rm
sin}^{2}\end{pmatrix}\right],$ (6)

with amplitude $A_{\rm sin}$, period $T_{\rm sin}$, and variance $\sigma_{\rm
sin}^{2}$.

Figures 3 and 4 visualize examples of the stochastic and periodic
trajectories, respectively, in blue color. Throughout this study, we use a
simulation step size of $\Delta t=5\,\mathrm{s}$ for simulating obstacle and
agent movement. Crucially, to train a _collision risk estimator_ that
estimates CR metrics for each obstacle and for each time step, we impose the
following assumption:

######  Assumption 1: Behavior stability.

The behavior of the obstacles used to train the collision risk estimator is
similar to the obstacle behavior in the simulation environment.

##  3 Step 1: Supervised learning

This section describes the supervised learning step of our approach as
depicted in Figure 1. For each obstacle, we predict its future trajectory
based on previous observations and compute the associated collision risk
subsequently.

###  3.1 Basics

Supervised learning is the most common form of machine learning and describes
the task of learning a mapping from potentially high-dimensional input samples
to corresponding target vectors based on labeled data (LeCun et al., 2015;
Zhou et al., 2021). Paired with (deep) neural networks as function
approximators, supervised learning approaches have shown remarkable successes
in various real-world problems such as speech recognition (Nassif et al.,
2019) or object detection (Dhillon and Verma, 2020). In this study, we
consider the regression task of having data pairs
$D=\left\\{(\mathbf{X}_{i},\mathbf{Y}_{i})\right\\}_{i=1}^{N}$, where
$\mathbf{X}_{i}\in\mathbb{R}^{n}$, $\mathbf{Y}_{i}\in\mathbb{R}^{m}$, and
learning the parameter vector $\theta$ of a neural network
$f_{\theta}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ by minimizing the mean
squared error
$MSE(D;\theta)=N^{-1}\sum_{i=1}^{N}\left[f_{\theta}(\mathbf{X}_{i})-\mathbf{Y}_{i}\right]^{\top}\left[f_{\theta}(\mathbf{X}_{i})-\mathbf{Y}_{i}\right]$
with the Adam optimizer (Kingma and Ba, 2014).

###  3.2 Trajectory prediction

The input features $\mathbf{X}$ contain information about $h=10$ previous
positions of the obstacle $(\vec{p}_{t-9},\dots,\vec{p}_{t})^{\top}$, from
which the neural network should approximate the next position $\vec{p}_{t+1}$.
We construct the neural network $f_{\theta}$ with an LSTM-layer (Hochreiter
and Schmidhuber, 1997) of 64 hidden units. The output is forwarded through
another hidden layer with 64 neurons and ReLU activation to yield output in
$\mathbb{R}^{2}$. The training progress of the network is depicted in Figure 2
for both defined non-linear obstacle movements.

Figure 2: Training of the trajectory prediction module for periodic and
stochastic obstacle trajectories.

After the training is completed, we perform subsequent one-step-ahead
forecasts to predict an obstacle’s _entire trajectory_ for a time horizon
$T=100$. More precisely, we always take the last ten positions, possibly
including predicted ones, to predict the following position iteratively. To
illustrate the training results, we provide a representative example for a
predicted trajectory in Figure 3 for the stochastic obstacle movement and
Figure 4 for the periodic movement. The blue dots represent the ground truth
trajectory, from which only the first ten dots, shown in green, are observed.
These observations are used to iteratively compute the predicted trajectory,
which is shown in orange. Since we assume that not all previous positions of
an obstacle are observable, we predict past positions as well. These past
obstacle positions are later used to estimate if the agent already passed and
obstacle; see Section 3.3. Since the obstacle trajectories defined in (5) and
(6) are symmetric, we can use the same model to predict past positions.

To quantitatively evaluate the prediction accuracy, we compute the root mean
squared error (RSME) between predicted and real trajectories for both obstacle
movements, as shown in Figure 5, with different quantiles. We can see that the
prediction accuracy is relatively high for short-term predictions, but
expectedly increases with increased forecasting time.

Figure 3: Trajectory prediction for stochastic obstacle movement with last
observations in green and ground truth data in blue. Figure 4: Trajectory
prediction for periodic obstacle movement with last observations in green and
ground truth data in blue. Figure 5: RMSE between predicted trajectory and
real trajectory with 25%-, 50%-, 75%-, and 90%-quantile.

###  3.3 Collision risk estimation

After predicting the trajectory of an obstacle for a time horizon $T=100$,
which equals a forecast time of $500\,\mathrm{s}$ based on a simulation step
size of $\Delta t=5\,\mathrm{s}$, we can estimate the CPA by numerically
finding the minimum between the predicted obstacle and the predicted agent
trajectory. For the agent, we thereby assume a linear movement based on its
current velocity vector. An example is illustrated in Figure 6 for a periodic
movement of the obstacle, where the blue curve represents the estimated
distance between the agent and obstacle based on these predictions. Naturally,
these distance estimates exhibit the periodic oscillations of the obstacle
trajectory. To reduce the variance in the estimation of $d^{\rm CPA}$ and
$t^{\rm CPA}$ caused by this pattern, we apply a symmetric moving average
filter on the estimated distance:

$\hat{d}_{j,\rm smooth}=\frac{1}{2n+1}\sum_{j=-n}^{n}\hat{d}_{j},$

where $n=10$. The variables $\hat{d}_{j}$ and $\hat{d}_{j,\rm smooth}$ denote
the distance estimates between agent and obstacle at step $j$ before and after
smoothing, respectively. The resulting estimates for $d^{\rm CPA}$ and $t^{\rm
CPA}$ using the smoothed distance are depicted in orange color in Figure 6.
Note that negative values for the $t^{\rm CPA}$ are possible if the agent
already passed the obstacle.

Figure 6: Real and smoothed distance between a predicted obstacle and
predicted agent trajectory for a periodic obstacle movement.

##  4 Step 2: Reinforcement Learning

###  4.1 Basics

RL aims at solving sequential decision tasks in which an agent interacts with
an environment under the objective to maximize a numerical reward signal
(Sutton and Barto, 2018). Formally, we consider Markov Decision Processes
(MDP, Puterman 1994) consisting of a state space $\mathcal{S}$, an action
space $\mathcal{A}$, an initial state distribution
$T_{0}:\mathcal{S}\rightarrow[0,1]$, a state transition probability
distribution
$\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$, a
reward function
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and a
discount factor $\gamma\in[0,1)$. At each time step $t$, the agent receives a
state information $S_{t}\in\mathcal{S}$, selects an action
$A_{t}\in\mathcal{A}$, gets an instantaneous reward $R_{t+1}$, and transitions
based on the environmental dynamics $\mathcal{P}$ to the next state
$S_{t+1}\in\mathcal{S}$. In practical applications, the full-state information
is rarely available due to sensor limitations, delays, noise, or other
disturbances. Addressing this issue, a generalization called Partially
Observable Markov Decision Processes (POMDP, Kaelbling et al. 1998) introduces
two additional components: the observation space $\mathcal{O}$ and the
observation function
$\mathcal{Z}:\mathcal{S}\times\mathcal{A}\times\mathcal{O}\rightarrow[0,1]$.
In a POMDP, the agent does not receive the new state $S_{t+1}$ directly, but
instead an observation $O_{t+1}\in\mathcal{O}$, which is generated with
probability $P(O_{t+1}|S_{t+1},A_{t})$ by the observation function
$\mathcal{Z}$. Consequently, a POMDP is a Hidden Markov Model with actions and
the observations are used for learning. In the following, we use capital
notation, e.g., $S_{t}$, to indicate random variables and small notation,
e.g., $s_{t}$ or $s$, to describe their realizations.

Objective of the agent in the MDP scenario is to optimize for a policy
$\pi:\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$, a mapping from states to
probability distributions over actions, that maximizes the expected return,
which is the expected discounted cumulative reward, from the start state:
$E_{\pi}\left[\sum_{k=0}^{\infty}\gamma^{k}R_{k+1}\right|S_{0}]$. Common
practice is the definition of action value functions $Q^{\pi}(s,a)$, which are
the expected return when starting in state $s$, taking action $a$, and
following policy $\pi$ afterwards:
$Q^{\pi}(s,a)=E_{\pi}\left[\sum_{k=0}^{\infty}\gamma^{k}R_{t+k+1}|S_{t}=s,A_{t}=a\right]$.
Crucially, in an MDP there is a deterministic optimal policy
$\pi^{*}(s)=\operatorname*{arg\,max}_{a\in\mathcal{A}}Q^{*}(s,a)$ if the state
space is finite or countably infinite (Puterman, 1994). The policy $\pi^{*}$
is connected with optimal action-values $Q^{*}(s,a)=\max_{\pi}Q^{\pi}(s,a)$.
The general control objective of RL is to find or approximate $\pi^{*}$.

###  4.2 Recurrent reinforcement learning

As described in Section 4.1, only observations $o_{t}$ rather than full states
$s_{t}$ are available in the POMDP case. One popular approach to handle this
scenario is the construction of belief states, which are distributions over
the real states the agent might be in, given the observation so far. This
approach requires a model of the environment and is computationally demanding
(Heess et al., 2015). An alternative approach might be to stack past
observations together (Mnih et al., 2015), but this quickly increases the
input dimension and limits the agent to a fixed number of frames. Finally, a
further approach is to incorporate recurrency into the function approximators
of model-free algorithms, which was shown to yield strong performances (Ni et
al., 2021). The recurrency enriches the agent’s decision making by extracting
information from past observations, potentially yielding an improved ability
to solve problems without access to the complete state vector.

In this study, we follow the latter approach and use the LSTM-TD3 algorithm of
Meng et al. (2021), which adds LSTM layers to the off-policy, actor-critic TD3
algorithm of Fujimoto et al. (2018). Crucially, the LSTM-TD3 uses a
deterministic policy $\mu$ and defines the past history $h_{t}^{l}$ of length
$l$ at time step $t$ defined as:

$h_{t}^{l}=\begin{cases}o_{t-l},\ldots,o_{t-1}&\text{if \hskip
3.55658pt}l,t\geq 1.\\\ o_{0}&\text{else}.\end{cases}$

The zero-valued dummy observation $o_{0}$ is of the same dimension as a
regular observation. Note that the definition of $h_{t}^{l}$ slightly differs
from Meng et al. (2021) since we do not include past actions in the history.
Furthermore, we set $l=10$ throughout the paper, analogous to the history
length of the trajectory prediction from Section 3. The algorithm disassembles
both actor and critic into different sub-components. Precisely, there is a
memory extraction (MEM) part in the function approximators, $Q^{\rm MEM}$ and
$\mu^{\rm MEM}$, respectively, that processes the history. In parallel, the
current feature extraction (CFE) components $Q^{\rm CFE}$ and $\mu^{\rm CFE}$
process the observation of the current step $o_{t}$. Finally, the output of
both MEM and CFE are concatenated and fed into the perception integration (PI)
components $Q^{\rm PI}$ and $\mu^{\rm PI}$. These aggregate the extracted
pieces of information and yield the final result. The network design of our
LSTM-TD3 implementation is included in Figure 1 and formalized as follows:

$\displaystyle Q(o_{t},a_{t},h_{t}^{l})$ $\displaystyle=Q^{\rm
PI}\left\\{Q^{\rm MEM}(h_{t}^{l})\bowtie Q^{\rm CFE}(o_{t},a_{t})\right\\},$
$\displaystyle\mu(o_{t},h_{t}^{l})$ $\displaystyle=\mu^{\rm
PI}\left\\{\mu^{\rm MEM}(h_{t}^{l})\bowtie\mu^{\rm CFE}(o_{t})\right\\},$

where $\bowtie$ is the concatenation operator. The optimization of the
algorithm is identical to the TD3. Moreover, to ensure the robustness of our
architecture with respect to the algorithm, we apply the SAC algorithm of
Haarnoja et al. (2018a, b), which uses a stochastic actor and automated tuning
of the algorithm’s temperature parameter. We construct the network design
analogous to the LSTM-TD3 networks and refer to the approach in the following
as LSTM-SAC.

###  4.3 Training environment

We aim to develop a challenging training environment for DOA where the agent
has to anticipate several obstacle trajectories to avoid collisions. To solely
focus on the collision avoidance task and prevent the agent from making
strategic decisions, we define an environment where obstacles should be passed
in a predefined fashion, so-called _passing rules_. Moreover, research on RL-
based DOA indicates that the final performance increases when the collision
risk in training is high (Hart and Okhrin, 2022), which we consider
appropriately.

We consider a set of obstacles $\mathcal{M}=\\{1,\ldots,N_{\rm obstacle}\\}$,
where $N_{\rm obstacle}=10$ is the total number of obstacles in the
environment and the obstacle dynamics follow (1). We further define an agent
with position $\vec{p}_{t,\rm agent}=(x_{t,\rm agent},y_{t,\rm agent})^{\top}$
and velocity $\dot{\vec{p}}_{t,\rm agent}=(\dot{x}_{\rm agent},\dot{y}_{t,\rm
agent})^{\top}$ at time step $t$, where $\dot{x}_{\rm agent}$ denotes a
constant longitudinal velocity component. The RL agent controls the lateral
velocity component via the lateral acceleration $\ddot{y}_{t,\rm agent}$.
Formally, the agent computes an action $a_{t}\in[-1,1]$ each time step $t$
that is mapped to the lateral acceleration:

$\ddot{y}_{t,\rm agent}=a_{y,\rm max}a_{t},$

where $a_{y,\rm max}$ defines the maximal lateral acceleration for the agent.
The agent receives an observation vector each time step $t$, defined as:

$o_{t}=\begin{pmatrix}\frac{\ddot{y}_{t,\rm agent}}{a_{y,\rm
max}},\frac{\dot{\vec{p}}_{t,\rm agent}}{v_{\rm max}},\frac{\vec{p}_{t,\rm
agent}-\vec{p}_{t,i}}{p_{\rm scale}},\frac{\hat{d}^{\rm CPA}_{t,i}}{d^{\rm
CPA}_{\rm scale}},\frac{\hat{t}^{\rm CPA}_{t,i}}{t^{\rm CPA}_{\rm
scale}}\end{pmatrix}^{\top},$ (7)

where $v_{\rm max}$ is the maximum speed and $\vec{p}_{t,i}$, $\hat{d}^{\rm
CPA}_{t,i}$, and $\hat{t}^{\rm CPA}_{t,i}$ denote the position, estimated
$d^{\rm CPA}$, and estimated $t^{\rm CPA}$ of the $i$-th obstacle,
respectively. Crucially, the estimates $\hat{d}^{\rm CPA}_{t,i}$ and
$\hat{t}^{\rm CPA}_{t,i}$ are thereby computed using the supervised learning
module introduced in Section 3. The constants $p_{\rm scale}$, $d^{\rm
CPA}_{\rm scale}$, and $t^{\rm CPA}_{\rm scale}$ are used for scaling. Since
the computation of $\hat{d}^{\rm CPA}_{t,i}$ and $\hat{t}^{\rm CPA}_{t,i}$ is
demanding, we update those values only every 10th simulation step. We
emphasize that (7) slightly abuses notation and contains information about all
obstacles $i\in\mathcal{M}$. Consequently, $o_{t}$ is of dimension
$3+4N_{\text{obstacle}}$.

The passing rule for each obstacle is indicated by the index within the
observation vector. The first three observations are reserved for agent-
related observations, the following $4N_{\rm obstacle}/2$ observations are
reserved for obstacles with passing rule ’right’, and the last $4N_{\rm
obstacle}/2$ for obstacles with passing rule ’left’. Within these blocks of
the same passing rule, obstacles are sorted according to ascending estimated
$\hat{t}^{\rm CPA}$. To validate the proposed approach, we additionally
configure a baseline agent with a reduced observation space, neglecting
$\hat{d}^{\rm CPA}_{t,i}$ and $\hat{t}^{\rm CPA}_{t,i}$, and with sorting the
obstacles according to ascending euclidean distance to the agent.

Following recent RL literature (Silver et al., 2018), we aim to keep the
reward function as simple as possible. Therefore, we penalize the agent only
when passing an obstacle on the wrong side:

$r_{t+1}=\begin{cases}-1,&\text{passing an obstacle contrary to the respective
passing rule,}\\\ 0&\text{else.}\end{cases}$ (8)

B provides details about the initialization of the agent and the obstacles and
other environment configurations. Figure 7 exemplary depicts a time interval
of a training episode, where blue and green colors refer to obstacles with
different passing rules.

Figure 7: Training environment with several obstacle trajectories (green and
blue) and a possible collision-free agent trajectory (red).

##  5 Results

We compare our proposed two-step architecture with a training procedure
without incorporating the estimates of $d^{\rm CPA}$ and $t^{\rm CPA}$ in the
observation vector. Crucially, to demonstrate the proposal’s invariance to the
selected DRL algorithm, we run the same experiments on two different actor-
critic algorithms: the LSTM-TD3 of Meng et al. (2021), which uses a
determinstic actor, and the LSTM-SAC as described in Section 4, which uses a
stochastic actor. All experiments are conducted in the same training
environment, which is described in Section 4.3. Hyperparameters shared by both
algorithms are set equal; see Table 1. Figure 8 displays the training results,
where we mark agents trained with CR estimates from the supervised learning
module with the prefix ’SL’. The results are averaged over ten independent
runs of each experiment, and we include point-wise confidence intervals for
robustness.

Following (8), the absolute value of the test return of an episode can be
immediately interpreted as the number of collisions. Thus, including the
collision risk estimates based on supervised learning roughly halves the
number of collisions. This performance boost is independent of the used
algorithm. Notably, the high variance observed in the SL-LSTM-TD3 algorithm
can be attributed to two out of the ten training runs, during which the
training process remains stagnant for an extended period around an initial
return value of -10.

Figure 8: Performance comparison of agents that use the estimates for $d^{\rm
CPA}$ and $t^{\rm CPA}$ (prefix ’SL’) with baseline agents without the SL
subroutine for stochastic (left) and periodic (right) obstacle movement. The
y-axis displays the test return during an episode. The results are averaged
over ten independent runs and the shaded areas are 95% confidence intervals.

We visualize an exemplary episode for stochastic and periodic obstacle
movements in Figures 9 and 10, respectively, to give further intuition about
how the trained agents perform in the obstacle avoidance environment. Since
both RL algorithms perform return-wise equally well, we show only the
trajectory of the two configurations of the LSTM-TD3. The agent trajectories
are depicted in red and yellow, and obstacle trajectories in blue and green,
respectively, according to the imposed passing rules. The violations of the
passing rules are marked with a black square. We can observe that the agent
with the additional collision risk estimates (prefix ’SL’) chooses the better
trajectory, resulting in no violation of passing rules. On the other hand, the
baseline agent seems unable to accurately predict the trajectory of some
obstacles, resulting in clear violations of passing rules.

Figure 9: Agent trajectories in the obstacle avoidance environment with
stochastic obstacle movement, black squares indicating a violation of passing
rules. Figure 10: Agent trajectories in the obstacle avoidance environment
with periodic obstacle movement, black squares indicating a violation of
passing rules.

##  6 Conclusion

Future sustainable transportation systems can benefit from autonomous vehicles
in terms of safety and energy efficiency, particularly in non-lane-based
domains such as air or maritime traffic. Such real-world autonomous systems
necessitate robust local path planning capabilities, including the critical
task of avoiding collisions with dynamic obstacles. This work proposes a two-
step architecture that leverages recent advances from supervised learning and
reinforcement learning. The first step captures non-linear movements of
obstacles and estimates collision risk metrics in a supervised fashion, while
the second step uses these estimates to enrich the observation space of an RL
agent.

Our approach has demonstrated a significant enhancement in the collision
avoidance capabilities of the respective autonomous agent, thereby
constituting a valuable contribution to the realm of autonomous vehicles.
Importantly, our learning architecture exhibits versatility by not being
confined to a specific set of obstacle dynamics. In our upcoming research, we
aim to validate this architecture using a diverse range of real-world obstacle
behavior data and implementing it across different traffic domains. Moreover,
the current approach focuses on providing point estimates of the collision
risk with obstacles. In future work, we will consider the uncertainties
connected with the collision risk estimates since these uncertainties are
likely to be decision-relevant for real-life operations.

## Acknowledgment

This work was partially funded by BAW - Bundesanstalt für Wasserbau
(Mikrosimulation des Schiffsverkehrs auf dem Niederrhein). The authors are
grateful to the Center for Information Services and High Performance Computing
at TU Dresden for providing its facilities for high throughput calculations.

## References

  * Badue et al. (2021) Badue, C., Guidolini, R., Carneiro, R.V., Azevedo, P., Cardoso, V.B., Forechi, A., Jesus, L., Berriel, R., Paixao, T.M., Mutz, F., et al., 2021.  Self-driving cars: A survey.  Expert Systems with Applications 165, 113816.
  * Bellemare et al. (2020) Bellemare, M.G., Candido, S., Castro, P.S., Gong, J., Machado, M.C., Moitra, S., Ponda, S.S., Wang, Z., 2020.  Autonomous navigation of stratospheric balloons using reinforcement learning.  Nature 588, 77–82.
  * Brittain and Wei (2022) Brittain, M., Wei, P., 2022.  Scalable autonomous separation assurance with heterogeneous multi-agent reinforcement learning.  IEEE Transactions on Automation Science and Engineering 19, 2837–2848.
  * Chun et al. (2021) Chun, D.H., Roh, M.I., Lee, H.W., Ha, J., Yu, D., 2021.  Deep reinforcement learning-based collision avoidance for an autonomous ship.  Ocean Engineering 234, 109216.
  * Dhillon and Verma (2020) Dhillon, A., Verma, G.K., 2020.  Convolutional neural network: a review of models, methodologies and applications to object detection.  Progress in Artificial Intelligence 9, 85–112.
  * Everett et al. (2018) Everett, M., Chen, Y.F., How, J.P., 2018.  Motion planning among dynamic, decision-making agents with deep reinforcement learning, in: International Conference on Intelligent Robots and Systems, IEEE. pp. 3052–3059.
  * Everett et al. (2021) Everett, M., Chen, Y.F., How, J.P., 2021.  Collision avoidance in pedestrian-rich environments with deep reinforcement learning.  IEEE Access 9, 10357–10377.
  * Falanga et al. (2020) Falanga, D., Kleber, K., Scaramuzza, D., 2020.  Dynamic obstacle avoidance for quadrotors with event cameras.  Science Robotics 5, eaaz9712.
  * Fawzi et al. (2022) Fawzi, A., Balog, M., Huang, A., Hubert, T., Romera-Paredes, B., Barekatain, M., Novikov, A., R Ruiz, F.J., Schrittwieser, J., Swirszcz, G., et al., 2022.  Discovering faster matrix multiplication algorithms with reinforcement learning.  Nature 610, 47–53.
  * Feng et al. (2023) Feng, S., Sun, H., Yan, X., Zhu, H., Zou, Z., Shen, S., Liu, H.X., 2023.  Dense reinforcement learning for safety validation of autonomous vehicles.  Nature 615, 620–627.
  * Fossen (2021) Fossen, T.I., 2021.  Handbook of Marine Craft Hydrodynamics and Motion Control, 2nd Edition.  John Wiley & Sons.
  * Fujimoto et al. (2018) Fujimoto, S., Hoof, H., Meger, D., 2018.  Addressing function approximation error in actor-critic methods, in: International Conference on Machine Learning, PMLR. pp. 1587–1596.
  * Haarnoja et al. (2018a) Haarnoja, T., Zhou, A., Abbeel, P., Levine, S., 2018a.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, in: International Conference on Machine Learning, PMLR. pp. 1861–1870.
  * Haarnoja et al. (2018b) Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al., 2018b.  Soft actor-critic algorithms and applications.  arXiv preprint arXiv:1812.05905 .
  * Hart and Okhrin (2022) Hart, F., Okhrin, O., 2022.  Enhanced method for reinforcement learning based dynamic obstacle avoidance by assessment of collision risk.  URL: https://arxiv.org/abs/2212.04123, doi:10.48550/ARXIV.2212.04123.
  * Heess et al. (2015) Heess, N., Hunt, J.J., Lillicrap, T.P., Silver, D., 2015.  Memory-based control with recurrent neural networks.  arXiv preprint arXiv:1512.04455 .
  * Hochreiter and Schmidhuber (1997) Hochreiter, S., Schmidhuber, J., 1997.  Long short-term memory.  Neural computation 9, 1735–1780.
  * Huang et al. (2020) Huang, Y., Chen, L., Chen, P., Negenborn, R.R., Van Gelder, P., 2020.  Ship collision avoidance methods: State-of-the-art.  Safety science 121, 451–473.
  * International Maritime Organization (1972) International Maritime Organization, 1972.  COLREG: Convention on the International Regulations for Preventing Collisions at Sea.
  * Isufaj et al. (2022) Isufaj, R., Omeri, M., Piera, M.A., 2022.  Multi-UAV conflict resolution with graph convolutional reinforcement learning.  Applied Sciences 12, 610.
  * Kaelbling et al. (1998) Kaelbling, L.P., Littman, M.L., Cassandra, A.R., 1998.  Planning and acting in partially observable stochastic domains.  Artificial Intelligence 101, 99–134.
  * Kandepu et al. (2008) Kandepu, R., Foss, B., Imsland, L., 2008.  Applying the unscented kalman filter for nonlinear state estimation.  Journal of Process Control 18, 753–768.
  * Kingma and Ba (2014) Kingma, D.P., Ba, J., 2014.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980 .
  * Kuchar and Yang (2000) Kuchar, J.K., Yang, L.C., 2000.  A review of conflict detection and resolution modeling methods.  IEEE Transactions On Intelligent Transportation Systems 1, 179–189.
  * LeCun et al. (2015) LeCun, Y., Bengio, Y., Hinton, G., 2015.  Deep learning.  Nature 521, 436–444.
  * Lee et al. (2019) Lee, M.A., Zhu, Y., Srinivasan, K., Shah, P., Savarese, S., Fei-Fei, L., Garg, A., Bohg, J., 2019.  Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks, in: International Conference on Robotics and Automation, IEEE. pp. 8943–8950.
  * Lenart (1983) Lenart, A.S., 1983.  Collision threat parameters for a new radar display and plot technique.  The Journal of Navigation 36, 404–410.
  * Li (2019) Li, Y., 2019.  Reinforcement learning applications.  arXiv preprint arXiv:1908.06973 .
  * Liu et al. (2016) Liu, Z., Zhang, Y., Yu, X., Yuan, C., 2016.  Unmanned surface vehicles: An overview of developments and challenges.  Annual Reviews in Control 41, 71–93.
  * Marin-Plaza et al. (2018) Marin-Plaza, P., Hussein, A., Martin, D., Escalera, A.d.l., 2018.  Global and local path planning study in a ros-based research platform for autonomous vehicles.  Journal of Advanced Transportation 2018.
  * Meng et al. (2021) Meng, L., Gorbet, R., Kulić, D., 2021.  Memory-based deep reinforcement learning for pomdps, in: International Conference on Intelligent Robots and Systems, IEEE. pp. 5619–5626.
  * Meyer et al. (2020) Meyer, E., Robinson, H., Rasheed, A., San, O., 2020.  Taming an autonomous surface vehicle for path following and collision avoidance using deep reinforcement learning.  IEEE Access 8, 41466–41481.
  * Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al., 2015.  Human-level control through deep reinforcement learning.  Nature 518, 529–533.
  * Mousavi et al. (2016) Mousavi, S.S., Schukat, M., Howley, E., 2016.  Deep reinforcement learning: an overview, in: Proceedings of SAI Intelligent Systems Conference, Springer. pp. 426–440.
  * Nassif et al. (2019) Nassif, A.B., Shahin, I., Attili, I., Azzeh, M., Shaalan, K., 2019.  Speech recognition using deep neural networks: A systematic review.  IEEE access 7, 19143–19165.
  * Negenborn et al. (2023) Negenborn, R.R., Goerlandt, F., Johansen, T.A., Slaets, P., Valdez Banda, O.A., Vanelslander, T., Ventikos, N.P., 2023.  Autonomous ships are on the horizon: here’s what we need to know.  Nature 615, 30–33.
  * Ni et al. (2021) Ni, T., Eysenbach, B., Salakhutdinov, R., 2021.  Recurrent model-free RL is a strong baseline for many POMDPs.  arXiv preprint arXiv:2110.05038 .
  * Pandey et al. (2017) Pandey, A., Pandey, S., Parhi, D., 2017.  Mobile robot navigation and obstacle avoidance techniques: A review.  International Robotics & Automation Journal 2, 96–105.
  * Polvara et al. (2018) Polvara, R., Sharma, S., Wan, J., Manning, A., Sutton, R., 2018.  Obstacle avoidance approaches for autonomous navigation of unmanned surface vehicles.  The Journal of Navigation 71, 241–256.
  * Puterman (1994) Puterman, M.L., 1994.  Markov Decision Processes: Discrete Stochastic Dynamic Programming.  John Wiley & Sons.
  * Rawson and Brito (2023) Rawson, A., Brito, M., 2023.  A survey of the opportunities and challenges of supervised machine learning in maritime risk analysis.  Transport Reviews 43, 108–130.
  * Ribeiro et al. (2020) Ribeiro, M., Ellerbroek, J., Hoekstra, J., 2020.  Review of conflict resolution methods for manned and unmanned aviation.  Aerospace 7, 79.
  * Sauer et al. (2018) Sauer, A., Savinov, N., Geiger, A., 2018.  Conditional affordance learning for driving in urban environments, in: Conference on Robot Learning, PMLR. pp. 237–252.
  * Silver et al. (2018) Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al., 2018.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  Science 362, 1140–1144.
  * Strickland et al. (2018) Strickland, M., Fainekos, G., Amor, H.B., 2018.  Deep predictive models for collision risk assessment in autonomous driving, in: International Conference on Robotics and Automation, IEEE. pp. 4685–4692.
  * Suárez-Varela et al. (2019) Suárez-Varela, J., Mestres, A., Yu, J., Kuang, L., Feng, H., Barlet-Ros, P., Cabellos-Aparicio, A., 2019.  Feature engineering for deep reinforcement learning based routing, in: International Conference on Communications, IEEE. pp. 1–6.
  * Sutton and Barto (2018) Sutton, R.S., Barto, A.G., 2018.  Reinforcement Learning: An Introduction.  Cambridge: The MIT Press.
  * Treiber and Kesting (2013) Treiber, M., Kesting, A., 2013.  Traffic Flow Dynamics: Data, Models and Simulation.  Springer-Verlag Berlin Heidelberg.
  * Tsay (2010) Tsay, R.S., 2010.  Analysis of Financial Time Series.  New Jersey: John Wiley & Sons.
  * Vagale et al. (2021) Vagale, A., Oucheikh, R., Bye, R.T., Osen, O.L., Fossen, T.I., 2021.  Path planning and collision avoidance for autonomous surface vehicles I: a review.  Journal of Marine Science and Technology 26, 1292–1306.
  * Vinyals et al. (2019) Vinyals, O., Babuschkin, I., Czarnecki, W.M., Mathieu, M., Dudzik, A., Chung, J., Choi, D.H., Powell, R., Ewalds, T., Georgiev, P., et al., 2019.  Grandmaster level in starcraft II using multi-agent reinforcement learning.  Nature 575, 350–354.
  * Waltz and Okhrin (2023) Waltz, M., Okhrin, O., 2023.  Spatial–temporal recurrent reinforcement learning for autonomous ships.  Neural Networks 165, 634–653.
  * Waltz and Paulig (2022) Waltz, M., Paulig, N., 2022.  RL Dresden Algorithm Suite.  https://github.com/MarWaltz/TUD_RL.
  * Waltz et al. (2023) Waltz, M., Paulig, N., Okhrin, O., 2023.  2-level reinforcement learning for ships on inland waterways.  arXiv preprint arXiv:2307.16769 .
  * Wang et al. (2019) Wang, C., Wang, J., Shen, Y., Zhang, X., 2019.  Autonomous navigation of uavs in large-scale complex environments: A deep reinforcement learning approach.  IEEE Transactions on Vehicular Technology 68, 2124–2136.
  * Xu et al. (2022) Xu, X., Lu, Y., Liu, G., Cai, P., Zhang, W., 2022.  COLREGs-abiding hybrid collision avoidance algorithm based on deep reinforcement learning for usvs.  Ocean Engineering 247, 110749.
  * Zhang et al. (2020) Zhang, W., Feng, X., Goerlandt, F., Liu, Q., 2020.  Towards a convolutional neural network model for classifying regional ship collision risk levels for waterway risk analysis.  Reliability Engineering & System Safety 204, 107127.
  * Zhao and Liu (2021) Zhao, P., Liu, Y., 2021.  Physics informed deep reinforcement learning for aircraft conflict resolution.  IEEE Transactions on Intelligent Transportation Systems 23, 8288–8301.
  * Zheng et al. (2020) Zheng, K., Chen, Y., Jiang, Y., Qiao, S., 2020.  A SVM based ship collision risk assessment algorithm.  Ocean Engineering 202, 107062.
  * Zhou et al. (2021) Zhou, X., Liu, H., Pourpanah, F., Zeng, T., Wang, X., 2021.  A survey on epistemic (model) uncertainty in supervised learning: Recent advances and applications.  Neurocomputing 489, 449–465.

##  Appendix A RL algorithm hyperparameters

Table 1: List of hyperparameters of the RL algorithms. Parameters with a † are solely used by the LSTM-TD3, whereas * refers to LSTM-SAC. Hyperparameter | Value
---|---
Discount factor | 0.99
Batch size | 32
Loss function | Mean squared error
Replay buffer size | $10^{6}$
Learning rate actor | $0.0001$
Learning rate critic | $0.0001$
Optimizer | Adam
History length $h$ | 10
Number of units per hidden layer | 128
Target noise† | 0.2
Target noise clip† | 0.5
Policy update delay† | 2
Initial temperature* | 0.2
Learning rate temperature* | 0.0001

##  Appendix B Environment details

At the beginning of a training episode, we initialize the agent’s dynamics to
zero, except the longitudinal speed $\dot{x}_{\rm agent}$, that is sampled
uniformly at random from the interval $[1\,\mathrm{m/s},v_{\rm max}]$. The
Euler and ballistic methods are used to update the agent’s lateral speed and
the positions for the agent and obstacles at time step $t+1$ (Treiber and
Kesting, 2013). Exemplary for the agent, we have:

$\displaystyle\dot{y}_{t+1,\rm agent}$ $\displaystyle=\dot{y}_{t,\rm
agent}+\ddot{y}_{t+1,\rm agent}\Delta t,$ (9) $\displaystyle\vec{p}_{t+1,\rm
agent}$ $\displaystyle=\vec{p}_{t,\rm agent}+\frac{\dot{\vec{p}}_{t,\rm
agent}+\dot{\vec{p}}_{t+1,\rm agent}}{2}\Delta t,$ (10)

with $\Delta t=5\,\mathrm{s}$ corresponding to the simulation step size. One
episode is defined as $N_{\rm steps}=500$ steps of updating.

For each obstacle, we define a passing rule. The set of obstacles that should
only be passed, from the perspective of the agent, on the right side in the
lateral direction is denoted as $\mathcal{M}_{\rm
right}=\\{1,\ldots,N_{\text{obstacle}}/2\\}$. Consequently, the remaining
obstacles should be passed left and are denoted $\mathcal{M}_{\rm
left}=\\{N_{\text{obstacle}}/2+1,\ldots,N_{\text{obstacle}}\\}$.

Next, a procedure to update the trajectories of obstacles needs to be defined
so that the agent has to face threatening obstacles constantly. Therefore, we
replace obstacles that have already been passed. To specify this condition, we
introduced the helper variable $TTC_{t,i}$ as the agent’s time-to-collision
with an obstacle $i\in\mathcal{M}$ in the longitudinal direction at time step
$t$, where the time-to-collision is computed based on the obstacle’s linear
trajectory part (2). Negative values for $TTC_{t,i}$ relate to obstacles that
have already passed the agent in the longitudinal direction. If for two
obstacles $k,l\in\mathcal{M_{\rm right}}$ holds: $TTC_{t,k}<0$, $TTC_{t,l}<0$,
and $TTC_{t,k}<TTC_{t,l}$, we replace obstacle $k$ as shown in Figure 11. Its
new time-to-collision is randomly sampled from:

$TTC_{t,k}\sim\mathcal{U}\left(\max_{j\in\mathcal{M_{\rm
right}}}(TTC_{t,j}),\max_{j\in\mathcal{M_{\rm right}}}(TTC_{t,j})+\Delta
TTC_{\rm max}\right),$ (11)

where $\Delta TTC_{\rm max}$ is the maximal temporal distance for the new
placement of an obstacle. This parameter affects the number of obstacles being
passed in a certain time interval and is, therefore, a crucial design element
of this environment. The same replacement procedure is applied for obstacles
with passing rule ’left.’

Figure 11: Replacement of an obstacle (obstacle $1$) since two obstacles with
the same passing rule already passed the agent (negative time-to-collision).
The obstacle’s new $TTC_{t,1}$ is set uniformly at random in the time interval
colored turquoise with the length of $\Delta TTC_{\rm max}$.

Having computed the new value for $TTC_{t,i}$ for a replaced obstacle
$i\in\mathcal{M}$ at time step $t$, the new position and velocity need to be
determined. First, we draw values for the constant velocity
$\dot{\vec{p}}_{i}=(\dot{x}_{i},\dot{y}_{i})^{\top}$ of the linear trajectory
part from uniform distributions:

$\displaystyle\dot{x}_{i}$ $\displaystyle\sim\mathcal{U}(-v_{x,\rm
max},v_{x,\rm max}),$ (12) $\displaystyle\dot{y}_{i}$
$\displaystyle\sim\mathcal{U}(-v_{y,\rm max},v_{y,\rm max}).$ (13)

Second, the new longitudinal position can be set as follows:

$x_{t,i}=(\dot{x}_{t,\rm agent}-\dot{x}_{i})TTC_{t,i}+x_{t,\rm agent}.$ (14)

Third, having the lateral speed of the replaced obstacle set, we generate the
new lateral position with the help of a predefined, stochastic trajectory
$y_{t,\rm traj}$, inspired by Meyer et al. (2020). This lateral trajectory is
computed at the beginning of an episode and is based on a smoothed AR(1)
process, whose parameters reflect the kinematics of the agent. Figure 12 shows
a replacement situation identical to Figure 11 and illustrates how this
trajectory is used to define the new lateral position for a replaced obstacle.
One can think of this process as an exemplary trajectory the agent has to
follow to avoid collisions with obstacles. In the following, we define the
smoothed AR(1) process and give a detailed explanation of the replacement of
an obstacle based on that process.

Figure 12: Replacement of an obstacle identical to the situation in Figure 11
but with additional information about lateral positions of obstacles.

The AR(1) process is defined as:

$B_{t+1}=\phi_{\rm traj}B_{t}+u_{t},\quad\text{where}\quad
u_{t}\sim\mathcal{N}(0,\sigma_{\rm traj}^{2}),$ (15)

with auto-regressive parameter $\phi_{\rm traj}$ and variance $\sigma_{\rm
traj}^{2}$. The parameters have been designed to model a lateral trajectory
the agent can approximately follow under acceleration and velocity constraints
represented by $a_{y,\rm max}$ and $v_{\rm max}$. To reduce the noise, we
exponentially smooth the AR(1) process:

$\displaystyle y_{t,\rm traj}=\begin{cases}B_{0},&\text{for }t=0,\\\
\beta_{\rm traj}B_{t}+(1-\beta_{\rm traj})y_{t-1,\rm traj},&\text{for
}t>0,\end{cases}$ (16)

where $\beta_{\rm traj}$ defines the smoothing factor. Based on this
trajectory and having already computed $TTC_{t,i}$, $x_{t,i}$, $\dot{x}_{i}$,
and $\dot{y}_{i}$ via (11), (12), (13), and (14), one more step is needed to
set the new lateral position $y_{t,i}$ for a replaced obstacle
$i\in\mathcal{M}$ at time step $t$.

We define $\Delta y_{i}$ as the absolute difference between an obstacle’s
lateral position $y_{t,i}$ and the defined trajectory $y_{t,\rm traj}$ when
agent and obstacle are at the same longitudinal position ($TTC_{t,i}=0$):

$\Delta y_{i}=|y_{t,i}-y_{t,\rm traj}|\quad\text{for}\quad TTC_{t,i}=0,$ (17)

shown yellow in Figure 12. To force our agent to move approximately along the
trajectory $y_{t,\rm traj}$, the positional difference $\Delta y_{i}$ should
be small, thus being another crucial design parameter to adjust the complexity
of the environment. Every time an obstacle $i$ is replaced, the variable
$\Delta y_{i}$ is sampled from a normal distribution:

$\Delta y_{i}\sim\mathcal{N}(\mu_{\Delta y},\sigma_{\Delta y}^{2}),$ (18)

and lower-bounded to $\Delta y_{\rm min}$:

$\Delta y_{i}=\max(\Delta y_{\rm min},\Delta y_{i}).$ (19)

By changing the parameters $\mu_{\Delta y}$, $\sigma_{\Delta y}^{2}$, and
$\Delta y_{\rm min}$, one can adjust how close the obstacles are coming to the
trajectory $y_{t,\rm traj}$ when obstacle and agent are at the same
longitudinal position. Table 2 contains the chosen values for those
parameters. Finally, the lateral position for obstacles $i_{\rm
R}\in\mathcal{M_{\rm right}}$ is computed via:

$y_{t,i_{\rm R}}=y_{t,\rm traj}+\Delta y_{i_{\rm R}}-\dot{y}_{i_{\rm
R}}TTC_{t,i_{\rm R}},$ (20)

and for obstacles $i_{\rm L}\in\mathcal{M_{\rm left}}$ via:

$y_{t,i_{\rm L}}=y_{t,\rm traj}-\Delta y_{i_{\rm L}}-\dot{y}_{i_{\rm
L}}TTC_{t,i_{\rm L}}.$ (21)

Figure 12 shows the final lateral position and time-to-collision for a
replaced obstacle as a filled circle. The linear trajectory part is
represented as a blue or green arrow. Having defined the linear trajectory
part of an obstacle, we can now add the non-linear trajectory part based on
(1).

Table 2: Environment parameters Parameter |  Description |  Value
---|---|---
$N_{\text{obstacle}}$ |  number of obstacles |  10
$a_{y,\rm max}$ |  agent’s maximum lateral acceleration |  $0.01\,\mathrm{m/s^{2}}$
$v_{\rm max}$ |  agent’s maximum speed |  $5\,\mathrm{m/s}$
$\Delta t$ |  simulation step size |  $5\,\mathrm{s}$
$N_{\rm steps}$ |  number of episode steps |  500
$p_{\rm scale}$ |  scaling factor for observation |  $3000\,\mathrm{m}$
$d^{\rm CPA}_{\rm scale}$ |  scaling factor for observation |  $400\,\mathrm{m}$
$t^{\rm CPA}_{\rm scale}$ |  scaling factor for observation |  $300\,\mathrm{s}$
$\Delta TTC_{\rm max}$ |  maximal temporal distance for replacing an obstacle |  $300\,\mathrm{s}$
$\phi_{\rm stoch}$ |  AR(1) process parameter |  $0.9$
$\phi_{\rm traj}$ |  AR(1) process parameter |  $0.99$
$\sigma^{2}_{\rm stoch}$ |  normal distribution variance |  $225\,\mathrm{m^{2}}$
$\sigma^{2}_{\rm traj}$ |  normal distribution variance |  $800\,\mathrm{m^{2}}$
$\sigma^{2}_{\rm sin}$ |  normal distribution variance |  $16\,\mathrm{m^{2}}$
$\beta_{\rm stoch}$ |  smoothing factor |  $0.2$
$\beta_{\rm traj}$ |  smoothing factor |  $0.03$
$\mu_{\Delta y}$ |  normal distribution mean |  $100\,\mathrm{m}$
$\sigma_{\Delta y}^{2}$ |  normal distribution variance |  $2500\,\mathrm{m^{2}}$
$\Delta y_{\rm min}$ |  minimum bound for $\Delta y$ |  $40\,\mathrm{m}$
$A_{\rm sin}$ |  periodic trajectory amplitude |  $15\,\mathrm{s}$
$T_{\rm sin}$ |  periodic trajectory period |  $20\,\mathrm{s}$


using deep learning.

#### Entropy across layers.

We present our first experiments on the study of the information propagation
within the network, where some intriguing results appear. This phenomenon is
illustrated in Figure 11, where we examine the entropy across the layers after
the network has been trained. We introduce two different measures of entropy,
depicted in the two graphs of the figure. In first place, we consider the
well-known Shannon entropy, denoted as $H(E)$, which quantifies the
information content of a discrete random variable $E$, distributed according
to a discrete probability measure $p:\Omega\to[0,1]$ such that $p(e)=p(E=e)$.
The Shannon entropy is computed as follows:

$H(E)=\mathbb{E}\big{[}-\log(p(E)\big{]}=\sum_{e\in E}-p(e)\log(p(e))$

In our context, the random variable of interest is
$E=\sum_{j=1}^{N}1\\!\\!1_{|X_{0}^{i}-X_{0}^{j}|\leq\epsilon}$, where
$X_{0}^{i}$ represents a generic image from the MNIST dataset. Additionally,
we introduce another measure of entropy, denoted as $\mathcal{E}$, which
quantifies the probability that the dataset can be partitioned into ten
clusters corresponding to the ten different digits. This quantity has been
introduced in [21] and it is defined as

$\mathcal{E}=\mathbb{P}\left(X\in\bigcup_{i=1}^{k}B_{\varepsilon}(X_{0}^{i})\right),$

where $\varepsilon>0$ is a small radius, and $X_{0}^{1},\ldots,X_{0}^{k}$ are
samplings from the dataset. Figure 11 suggests the existence of a distinct
pattern in the variation of information entropy across the layers, which
offers a hint for further investigations.

Figure 11: Left: Shannon entropy across layers. Right: Our measure of entropy
across layers.

Let us first focus on the Shannon entropy: as the layers’ dimensionality
decreases in the encoding phase, there is an expected decrease of entropy,
reflecting the compression and reduction of information in the lower-
dimensional representation. The bottleneck layer, where the dimension is kept
constant, represents a critical point where the entropy reaches a minimum.
This indicates that the information content is highly concentrated and
compressed in this latent space. Then, during the decoding phase, the Shannon
entropy do not revert to its initial value but instead exhibit a slower
increase. This behavior suggests that the network retains some of the learned
structure and information from the bottleneck layer. Something similar happens
for the second measure of entropy: at the beginning, the data is unlikely to
be highly clustered, since two distinct images of the same digit may be quite
distant one from the other. In the inner layers, this probability increases
until it reaches its maximum (rather close to $1$) in the bottleneck, where
the data can then be fully partitioned into clusters of radius $\epsilon$. As
for the Shannon entropy, the information from the bottleneck layer is retained
during the decoding phase, which is why the entropy remains constant for a
while and then decreases back in a slower manner.
It is worth noticing that in both cases the entropy does not fully return to
its initial level. This might be attributed to the phenomenon of mode
collapse, where the network fails to capture the full variability in the input
data and instead produces similar outputs for different inputs, hence inducing
some sort of implicit bias. Mode collapse is often considered undesirable in
generative models, as it hinders the ability to generate diverse and realistic
samples. However, in the context of understanding data structure and
performing clustering, the network’s capability to capture the main modes or
clusters of the data can be seen as a positive aspect. The network learns to
extract salient features and represent the data in a compact and informative
manner, enabling tasks such as clustering and classification. Further
investigation is needed to explore the relationship between the observed
entropy patterns, mode collapse, and the overall performance of the network on
different tasks.

### Acknowledgments

The authors would like to thank Prof. Giuseppe Savaré for the fruitful
discussions during his permanence in Munich. Moreover, the authors are
grateful to Dr. Oleh Melnyk for the suggestion on the extension of the
dynamical model to the U-net architecture.

This work has been funded by the German Federal Ministry of Education and
Research and the Bavarian State Ministry for Science and the Arts. C.C. and
M.F. acknowledge also the partial support of the project “Online Firestorms
And Resentment Propagation On Social Media: Dynamics, Predictability and
Mitigation” of the TUM Institute for Ethics in Artificial Intelligence and of
the DFG Project “Implicit Bias and Low Complexity Networks” within the DFG SPP
2298 “Theoretical Foundations of Deep Learning”. A.S. acknowledges the partial
support from INdAM-GNAMPA.

##  Appendix A Appendix

######  Lemma A.1 (Boundedness of trajectories).

Let us consider the controlled system

$\dot{x}={\mathcal{F}}(t,x,\theta),\quad x(0)=x_{0},$

where
${\mathcal{F}}:[0,T]\times\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}^{n}$
satisfies Assumptions 1, and $\theta\in L^{2}([0,T],\mathbb{R}^{m})$. Then,
for every $R>0$ and any $x_{0}\in B_{R}(0)$, we have that $x(t)\in
B_{\overline{R}}(0)$ for every $t\in[0,T]$, where
$\overline{R}=(R+L_{R}(1+\|\theta\|_{L^{1}}))e^{L_{R}(1+\|\theta\|_{L^{1}})}$.

###### Proof.

According to Assumption 1$-(ii)$ on ${\mathcal{F}}$, the trajectories can be
bounded as follows:

$\begin{split}|x(t)|&\leq|x_{0}|+\int_{0}^{t}|{\mathcal{F}}(s,x(s),\theta(s))|\,ds\leq|x_{0}|+L_{R}\int_{0}^{t}(1+|x(s)|)(1+|\theta(s)|)\,ds\end{split}$

for every $t\in[0,T]$. Using Gronwall’s lemma, it follows that

$|x(t)|\leq\Big{(}|x_{0}|+L_{R}(1+\left\lVert\theta\,\right\rVert_{L^{1}})\Big{)}e^{L_{R}(1+\left\lVert\theta\,\right\rVert_{L^{1}})}.$

∎

######  Lemma A.2 (Flow’s dependency on initial datum).

For every $t\in[0,T]$, let us consider the flow mapping
$\Phi^{\theta}_{(0,t)}:\mathbb{R}^{d}\to\mathbb{R}^{d}$ defined in (2.2) and
driven by the control $\theta\in L^{2}([0,T],\mathbb{R}^{m})$. Let us assume
that the controlled dynamics
${\mathcal{F}}:[0,T]\times\mathbb{R}^{d}\times\mathbb{R}^{m}\to\mathbb{R}^{d}$
satisfies Assumption 1. Then, for every $R>0$, and every $x_{1},x_{2}\in
B_{R}(0)$, it follows that

$|\Phi^{\theta}_{(0,t)}(x_{1})-\Phi^{\theta}_{(0,t)}(x_{2})|\leq
e^{L_{\overline{R}}(1+\left\lVert\theta\,\right\rVert_{L1})}|x_{1}-x_{2}|,$

where $\overline{R}$ is defined as in Lemma A.1, and $L_{\overline{R}}$ is
prescribed by Assumption 1-($ii$).

###### Proof.

Let us denote with $t\mapsto x_{1}(t),t\mapsto x_{2}(t)$ the solutions of
(2.1) driven by $\theta$ and starting, respectively, from
$x_{1}(0)=x_{1},x_{2}(0)=x_{2}$. Then, for every $t\in[0,T]$, we have

$\displaystyle|x_{1}(t)-x_{2}(t)|$
$\displaystyle\leq|x_{1}-x_{2}|+\int_{0}^{t}|{\mathcal{F}}(s,x_{1}(s),\theta(s))-{\mathcal{F}}(s,x_{2}(s),\theta(s))|\,ds$
$\displaystyle\leq|x_{1}-x_{2}|+L_{\overline{R}}\int_{0}^{t}(1+|\theta(s)|)|x_{1}(s)-x_{2}(s))|\,ds,$

by using Assumption 1$-(ii)$. As before, the statement follows from Gronwall’s
Lemma. ∎

######  Lemma A.3 (Flow’s dependency on time).

Under the same assumptions and notations as in Lemma A.2, for every $R>0$, for
every $x\in B_{R}(0)$ and for every $\theta\in L^{2}([0,T],\mathbb{R}^{m})$,
we have that

$|\Phi^{\theta}_{(0,t_{2})}(x)-\Phi^{\theta}_{(0,t_{1})}(x)|\leq
L_{\overline{R}}(1+\overline{R})(1+\left\lVert\theta\,\right\rVert_{L^{2}})|t_{2}-t_{1}|^{\frac{1}{2}}$

for every $0\leq t_{1}<t_{2}\leq T$, where $\overline{R}$ is defined as in
Lemma A.1, and $L_{\overline{R}}$ is prescribed by Assumption 1-($ii$).
Moreover, if $\theta\in L^{2}([0,T],\mathbb{R}^{m})\cap
L^{\infty}([0,T],\mathbb{R}^{m})$, then, for every $0\leq t_{1}<t_{2}\leq T$,
it holds:

$|\boldsymbol{\Phi}^{\theta}_{(0,t_{2})}(x)-\boldsymbol{\Phi}^{\theta}_{(0,t_{1})}(x)|\leq
L_{\overline{R}}(1+\overline{R})(1+\left\lVert\theta\,\right\rVert_{L^{2}})|t_{2}-t_{1}|.$

###### Proof.

If we denote by $t\mapsto x(t)$ the solution of (2.1) driven by the control
$\theta$, then

$|x(t_{2})-x(t_{1})|\leq\int_{t_{1}}^{t_{2}}|{\mathcal{F}}(s,x(s),\theta(s)|\,ds\leq\int_{t_{1}}^{t_{2}}L_{\overline{R}}(1+\overline{R})(1+|\theta(s)|)\,ds.$

The thesis follows by using Cauchy-Schwarz for $\theta\in L^{2}$, or from
basic estimates if $\theta\in L^{\infty}$. ∎

######  Lemma A.4 (Flow’s dependency on controls).

For every $t\in[0,T]$, let
$\Phi^{\theta_{1}}_{(0,t)},\Phi^{\theta_{2}}_{(0,t)}:\mathbb{R}^{d}\to\mathbb{R}^{d}$
be the flows defined in (2.2) and driven, respectively, by
$\theta_{1},\theta_{2}\in L^{2}([0,T],\mathbb{R}^{m})$. Let us assume that the
controlled dynamics
${\mathcal{F}}:[0,T]\times\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}^{n}$
satisfies Assumption 1. Then, for every $R>0$ and for every $x\in B_{R}(0)$,
it holds that

$|\Phi^{\theta_{1}}_{(0,t)}(x)-\Phi^{\theta_{2}}_{(0,t)}(x)|\leq
L_{\overline{R}}(1+\left\lVert\theta_{1}\,\right\rVert_{L^{2}}+\left\lVert\theta_{2}\,\right\rVert_{L^{2}})e^{L_{\overline{R}}(1+\left\lVert\theta_{1}\,\right\rVert_{L^{1}})}\left\lVert\theta_{1}-\theta_{2}\,\right\rVert_{L^{2}},$

where $\overline{R}$ is defined as in Lemma A.1, and $L_{\overline{R}}$ is
prescribed by Assumption 1-($ii$).

###### Proof.

By using Assumption 1$-(ii),(iii)$ and the triangle inequality, we obtain that

$\displaystyle|\Phi^{\theta_{1}}_{(0,t)}(x)-\Phi^{\theta_{2}}_{(0,t)}(x)|$
$\displaystyle\leq\int_{0}^{t}|{\mathcal{F}}(s,x_{1}(s),\theta_{1}(s))-{\mathcal{F}}(s,x_{2}(s),\theta_{2}(s))|\,ds$
$\displaystyle\leq\int_{0}^{t}|{\mathcal{F}}(s,x_{1}(s),\theta_{1}(s))-{\mathcal{F}}(s,x_{2}(s),\theta_{1}(s))|\,ds$
$\displaystyle\quad+\int_{0}^{t}|{\mathcal{F}}(s,x_{2}(s),\theta_{1}(s))-{\mathcal{F}}(s,x_{2}(s),\theta_{2}(s))|\,ds$
$\displaystyle\leq
L_{\overline{R}}\int_{0}^{t}(1+\theta_{1}(s))|x_{1}(s)-x_{2}(s)|\,ds+L_{\overline{R}}(1+\left\lVert\theta_{1}\,\right\rVert_{L^{2}}+\left\lVert\theta_{2}\,\right\rVert_{L^{2}})\left\lVert\theta_{1}-\theta_{2}\,\right\rVert_{L^{2}}.$

The statement follows again by applying Gronwall’s Lemma. ∎

######  Proposition A.5 (Differentiability with respect to trajectories
perturbations).

Let us assume that the controlled dynamics ${\mathcal{F}}$ satisfies
Assumptions 1-2. Given an admissible control $\theta\in
L^{2}([0,T],\mathbb{R}^{m})$ and a trajectory $t\mapsto
x(t)=\Phi_{(0,t)}^{\theta}(x_{0})$ with $x_{0}\in B_{R}(0)$, let
$\xi:[0,T]\to\mathbb{R}^{d}$ be the solution of the linearized problem

$\begin{cases}\dot{\xi}(t)=\nabla_{x}{\mathcal{F}}(t,x(t),\theta(t))\xi(t),\\\
\xi(\overline{t})=v,\end{cases}$ (A.1)

where $\overline{t}\in[0,T]$ is the instant of perturbation and $v$ is the
direction of perturbation of the trajectory. Then, for every
$t\in(\overline{t},T)$, it holds

$|\Phi_{(\overline{t},t)}^{\theta}(x(\overline{t})+\epsilon
v)-\Phi_{(\overline{t},t)}^{\theta}(x(\overline{t}))-\epsilon\xi(t)|\leq
C|v|^{2}\epsilon^{2}$

where $C$ is a constant depending on
$T,R,\left\lVert\theta\,\right\rVert_{L^{2}}$.

###### Proof.

For $t\geq\overline{t}$, let us denote with $t\mapsto
y(t):=\Phi_{(\overline{t},t)}^{\theta}(x\left(\overline{t})+\epsilon v\right)$
the solution of the modified problem, obtained by perturbing the original
trajectory with $\epsilon v$ at instant $\overline{t}$. Then, since $\xi$
solves (A.1), we can write

$\displaystyle|y(t)-x(t)-\epsilon\xi(t)|$
$\displaystyle=|\Phi_{(\overline{t},t)}^{\theta}(x(\overline{t})+\epsilon
v)-\Phi_{(\overline{t},t)}^{\theta}(x(\overline{t}))-\epsilon\xi(t)|$
$\displaystyle\leq\int_{\overline{t}}^{t}|{\mathcal{F}}(s,y(s),\theta(s))-{\mathcal{F}}(s,x(s),\theta(s))-\epsilon\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s))\xi(s)|ds$
$\displaystyle\leq\int_{\overline{t}}^{t}|{\mathcal{F}}(s,y(s),\theta(s))-{\mathcal{F}}(s,x(s),\theta(s))-\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s))(y(s)-x(s))|ds$
$\displaystyle\quad+\int_{\overline{t}}^{t}|\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s))||y(s)-x(s)-\epsilon\xi(s)|ds$
$\displaystyle\leq\int_{\overline{t}}^{t}\left[\int_{0}^{1}|\nabla_{x}{\mathcal{F}}(s,x(s)+\tau(y(s)-x(s)),\theta(s)-\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s)||y(s)-x(s)|d\tau\right]ds$
$\displaystyle\quad+\int_{\overline{t}}^{t}|\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s))||y(s)-x(s)-\epsilon\xi(s)|ds$

for every $t\geq\overline{t}$. We now address the two integrals separately.
Using Assumption 2$-(iv)$ and the result of Lemma A.4, we obtain the following
bound

$\displaystyle\int_{\overline{t}}^{t}\left[\int_{0}^{1}|\nabla_{x}{\mathcal{F}}(s,x(s)+\tau(y(s)-x(s))),\theta(s)-\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s))||y(s)-x(s)|d\tau\right]ds$
$\displaystyle\leq\int_{\overline{t}}^{t}L_{\overline{R}}(1+|\theta(s)|^{2})\frac{1}{2}|y(s)-x(s)|^{2}ds$
$\displaystyle\leq\frac{1}{2}L_{\overline{R}}\left(1+\|\theta\|_{L^{2}}^{2}\right)e^{2L_{\overline{R}}(1+\left\lVert\theta\,\right\rVert_{L^{1}})}|\epsilon
v|^{2}$

Similarly, for the second integral, owing to Assumption 2$-(iv)$, we can
compute:

$\displaystyle\int_{\overline{t}}^{t}|\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s))||y(s)-x(s)-\epsilon\xi(s)|ds\leq\int_{\overline{t}}^{t}L_{\overline{R}}(1+|\theta(s)|^{2})(1+\overline{R})|y(s)-x(s)-\epsilon\xi(s)|ds$

Finally, by combining the two results together and using Gronwall’s Lemma, we
prove the statement. ∎

######  Proposition A.6 (Differentiability with respect to control
perturbations).

Consider the solution $\xi$ of the linearized problem

$\begin{cases}\dot{\xi}(t)=\nabla_{x}{\mathcal{F}}(t,x^{\theta}(t),\theta(t))\xi(t)+\nabla_{\theta}{\mathcal{F}}(t,x^{\theta}(t),\theta(t))\nu(t)\\\
\xi(0)=0\end{cases}$ (A.2)

where the control $\theta$ is perturbed at the initial time with
$\theta+\epsilon\nu$, when starting with an initial datum $x_{0}\in B_{R}(0)$.
Then,

$|\Phi_{(0,t)}^{\theta+\epsilon\nu}(x_{0})-\Phi_{(0,t)}^{\theta}(x_{0})-\epsilon\xi(t)|\leq
C||\nu||_{L^{2}}^{2}\epsilon^{2}$ (A.3)

where $C$ is a constant depending on
$T,\overline{R},L_{\overline{R}},\left\lVert\theta\,\right\rVert_{L^{1}}$.
Moreover, we have that for every $t\in[0,T]$

$\xi(t)=\int_{0}^{t}\mathcal{R}^{\theta}_{(s,t)}(x_{0})\cdot\nabla_{\theta}{\mathcal{F}}(s,x^{\theta}(s),\theta(s))\nu(s)\,ds,$
(A.4)

where $\mathcal{R}^{\theta}_{(s,t)}(x_{0})$ has been defined in (3.10).

###### Proof.

We first observe that the dynamics in (A.2) is affine in the $\xi$ variable.
Moreover, Assumptions 1-2 guarantee that the coefficients are $L^{1}$-regular
in time. Hence, from the classical Caratheodory Theorem we deduce the
existence and the uniqueness of the solution of (A.2). Finally, the identity
(A.4) follows as a classical application of the resolvent map
$(\mathcal{R}^{\theta}_{(s,t)}(x_{0})_{s,t\in[0,T]}$ (see ,e.g., in [10,
Theorem 2.2.3]).

Let us denote with $t\mapsto x(t)$ and $t\mapsto y(t)$ the solutions of Cauchy
problem (2.1) corresponding, respectively, to the admissible controls $\theta$
and $\theta+\epsilon\nu$. We observe that, in virtue of Lemma A.1, we have
that there exists $\overline{R}>0$ such that $x(t),y(t)\in
B_{\overline{R}}(0)$ for every $t\in[0,T]$. Then, recalling the definition of
the flow map provided in (2.2), we compute

$\displaystyle|y(t)-x(t)-\epsilon\xi(t)|$
$\displaystyle=|\Phi_{(0,t)}^{\theta+\epsilon\nu}(x_{0})-\Phi_{(0,t)}^{\theta}(x_{0})-\epsilon\xi(t)|$
$\displaystyle\leq\int_{0}^{t}|{\mathcal{F}}(s,y(s),\theta(s)+\epsilon\nu(s))-{\mathcal{F}}(s,x(s),\theta(s))-\epsilon\dot{\xi}(s)|\,ds$
$\displaystyle\leq\int_{0}^{t}|{\mathcal{F}}(s,y(s),\theta(s)+\epsilon\nu(s))-{\mathcal{F}}(s,x(s),\theta(s)+\epsilon\nu(s))$
$\displaystyle\qquad\qquad-\epsilon\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s)+\epsilon\nu(s))\cdot(y(s)-x(s))|\,ds$
$\displaystyle\quad+\int_{0}^{t}|{\mathcal{F}}(s,x(s),\theta(s)+\epsilon\nu(s))-{\mathcal{F}}(s,x(s),\theta(s))-\epsilon\nabla_{\theta}{\mathcal{F}}(s,x(s),\theta(s))\cdot\nu(s)|\,ds$
$\displaystyle\quad+\int_{0}^{t}|\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s)+\epsilon\nu(s))-\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s))||y(s)-x(s)|ds$
$\displaystyle\quad+\int_{0}^{t}|\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s))||y(s)-x(s)-\epsilon\xi(s)|\,ds.$

We now handle each term separately:

$\begin{split}&\int_{0}^{t}|{\mathcal{F}}(s,y(s),\theta(s)+\epsilon\nu(s))-{\mathcal{F}}(s,x(s),\theta(s)+\epsilon\nu(s))-\epsilon\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s)+\epsilon\nu(s))(y(s)-x(s))|\,ds\\\
&\leq\int_{0}^{t}\left[\int_{0}^{1}L_{\overline{R}}(1+|\theta(s)+\epsilon\nu(s)|^{2})\tau|y(s)-x(s)|^{2}d\tau\right]ds\\\
&\leq
L_{\overline{R}}^{3}(1+\left\lVert\theta\,\right\rVert_{L^{2}}+\epsilon\left\lVert\nu\,\right\rVert_{L^{2}})^{4}e^{2L_{\overline{R}}(1+\left\lVert\theta\,\right\rVert_{L^{1}})}\left\lVert\nu\,\right\rVert^{2}_{L^{2}}\epsilon^{2}\end{split}$
(A.5)

where we used Assumption 2$-(iv)$ and Lemma A.4. By using Assumption 2$-(v)$,
we obtain the following bounds for the second integral:

$\begin{split}&\int_{0}^{t}|{\mathcal{F}}(s,x(s),\theta(s)+\epsilon\nu(s))-{\mathcal{F}}(s,x(s),\theta(s))-\nabla_{\theta}{\mathcal{F}}(s,x(s),\theta(s))\cdot\epsilon\nu(s)|\,ds\\\
&\leq\int_{0}^{t}\left[\int_{0}^{1}L_{\overline{R}}|\nu(s)|^{2}\epsilon^{2}\tau
d\tau\right]ds=\frac{1}{2}L_{\overline{R}}\left\lVert\nu\,\right\rVert_{L^{2}}^{2}\epsilon^{2}.\end{split}$
(A.6)

Similarly, the third integral can be bounded by using Assumption 2$-(vi)$ and
Lemma A.4, and it yields

$\begin{split}&\int_{0}^{t}|\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s)+\epsilon\nu(s))-\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s))||y(s)-x(s)|\,ds\\\
&\leq\int_{0}^{t}L_{\overline{R}}(1+|\theta(s)|+\epsilon|\nu(s)|)\epsilon|y(s)-x(s)||\nu(s)|\,ds\\\
&\leq
L_{\overline{R}}^{2}(1+\left\lVert\theta\,\right\rVert_{L^{2}}+\epsilon\left\lVert\nu\,\right\rVert_{L^{2}})^{2}e^{L_{\overline{R}}(1+\left\lVert\theta\,\right\rVert_{L^{1}})}\left\lVert\nu\,\right\rVert_{L^{2}}^{2}\epsilon^{2}.\end{split}$
(A.7)

Finally, the fourth integral can be bounded using Assumption 2$-(iv)$ as
follows:

$\begin{split}\int_{0}^{t}|\nabla_{x}{\mathcal{F}}(s,x(s),\theta(s))||y(s)-x(s)-\epsilon\xi(s)|ds&\leq\int_{0}^{t}L_{\overline{R}}(1+\overline{R})(1+|\theta(s)|^{2})|y(s)-x(s)-\epsilon\xi(s)|\,ds.\end{split}$
(A.8)

Hence, by combining (A.5), (A.6), (A.7) and (A.8), the thesis follows from
Gronwall Lemma. ∎

######  Proposition A.7 (Properties of the resolvent map).

Let us assume that the controlled dynamics ${\mathcal{F}}$ satisfies
Assumptions 1-2. Given an admissible control $\theta\in
L^{2}([0,T],\mathbb{R}^{m})$ and a trajectory $t\mapsto
x(t)=\Phi_{(0,t)}^{\theta}(x)$ with $x\in B_{R}(0)$, for every $\tau\in[0,T]$
the resolvent map
$\mathcal{R}^{\theta}_{(\tau,\cdot)}(x):[0,T]\to\mathbb{R}^{d\times d}$ is the
curve $s\mapsto\mathcal{R}^{\theta}_{(\tau,s)}(x_{0})$ that solves

$\begin{cases}\frac{d}{ds}\mathcal{R}^{\theta}_{(\tau,s)}(x)=\nabla_{x}{\mathcal{F}}(s,\Phi_{(0,s)}^{\theta}(x),\theta(s))\cdot\mathcal{R}_{(\tau,s)}^{\theta}(x)&\mbox{for
a.e. }s\in[0,T],\\\
\mathcal{R}_{(\tau,\tau)}^{\theta}(x)=\mathrm{Id}.\end{cases}$ (A.9)

Then for every $\tau,s\in[0,T]$, there exists a constant $C_{1}$ depending on
$T,R,\left\lVert\theta\,\right\rVert_{L^{2}}$ such that

$|\mathcal{R}^{\theta}_{(\tau,s)}(x)|:=\sup_{v\neq
0}\frac{|\mathcal{R}^{\theta}_{(\tau,s)}(x)\cdot v|}{|v|}\leq C_{1}.$ (A.10)

Moreover, for every $x,y\in B_{R}(0)$, there exists a constant $C_{2}$
depending on $T,R,\left\lVert\theta\,\right\rVert_{L^{2}}$ such that

$|\mathcal{R}^{\theta}_{(\tau,s)}(x)-\mathcal{R}^{\theta}_{(\tau,s)}(y)|:=\sup_{v\neq
0}\frac{|\mathcal{R}^{\theta}_{(\tau,s)}(x)\cdot
v-\mathcal{R}^{\theta}_{(\tau,s)}(y)\cdot v|}{|v|}\leq C_{2}|x-y|.$ (A.11)

Finally, if $\theta_{1},\theta_{2}$ satisfy
$\|\theta_{1}\|,\|\theta_{2}\|\leq\rho$, then there exists a constant $C_{3}$
depending on $T,R,\rho$ such that

$|\mathcal{R}^{\theta_{1}}_{(\tau,s)}(x)-\mathcal{R}^{\theta_{2}}_{(\tau,s)}(x)|:=\sup_{v\neq
0}\frac{|\mathcal{R}^{\theta_{1}}_{(\tau,s)}(x)\cdot
v-\mathcal{R}^{\theta_{2}}_{(\tau,s)}(x)\cdot v|}{|v|}\leq
C_{3}\|\theta_{1}-\theta_{2}\|_{L^{2}}.$ (A.12)

###### Proof.

We first prove the boundedness of the resolvent map. Let us fix
$v\in\mathbb{R}^{d}$ with $v\neq 0$, and let us define
$\xi(s):=\mathcal{R}^{\theta}_{(\tau,s)}(x)\cdot v$ for every $s\in[0,T]$.
Then, in virtue of Assumption 2$-(vi)$, we have:

$|\xi(s)|\leq|\xi(\tau)|+\int_{\tau}^{s}\left|\nabla_{x}{\mathcal{F}}(\sigma,\Phi_{(0,\sigma)}^{\theta}(x),\theta(\sigma))\right||\xi(\sigma)|\,d\sigma\leq|v|+L_{\overline{R}}\int_{0}^{t}(1+\theta(\sigma)^{2})|\xi(\sigma)|\,d\sigma,$

and, by Gronwall’s Lemma, we deduce (A.10). Similarly as before, given $x,y\in
B_{R}(0)$ and $v\neq 0$, let us define
$\xi^{x}(s):=\mathcal{R}^{\theta}_{(\tau,s)}(x)\cdot v$ and
$\xi^{y}(s):=\mathcal{R}^{\theta}_{(\tau,s)}(y)\cdot v$ for every $s\in[0,T]$.
Then, we have that

$\begin{split}|\xi^{x}(s)-\xi^{y}(s)|&\leq\int_{\tau}^{s}\left|\nabla_{x}{\mathcal{F}}(\sigma,\Phi_{(0,\sigma)}^{\theta}(x),\theta(\sigma))\xi^{x}(\sigma)-\nabla_{x}{\mathcal{F}}(\sigma,\Phi_{(0,\sigma)}^{\theta}(y),\theta(\sigma))\xi^{y}(\sigma)\right|\,d\sigma\\\
&\leq\int_{\tau}^{s}\left|\nabla_{x}{\mathcal{F}}(\sigma,\Phi_{(0,\sigma)}^{\theta}(x),\theta(\sigma))-\nabla_{x}{\mathcal{F}}(\sigma,\Phi_{(0,\sigma)}^{\theta}(y),\theta(\sigma))\right||\xi^{y}(\sigma)|\,d\sigma\\\
&\quad+\int_{\tau}^{s}\left|\nabla_{x}{\mathcal{F}}(\sigma,\Phi_{(0,\sigma)}^{\theta}(x),\theta(\sigma))\right||\xi^{x}(\sigma)-\xi^{y}(\sigma)|\,d\sigma\\\
&\leq
C_{1}|v|\int_{\tau}^{s}L_{\overline{R}}(1+\theta(\sigma)^{2})\left|\Phi_{(0,\sigma)}^{\theta}(x)-\Phi_{(0,\sigma)}^{\theta}(y)\right|\,d\sigma\\\
&\quad+\int_{\tau}^{s}L_{\overline{R}}(1+\theta(\sigma)^{2})|\xi^{x}(\sigma)-\xi^{y}(\sigma)|\,d\sigma,\end{split}$

where we used (A.10) and Assumption 2-($iv$). Hence, combining Lemma A.2 with
Gronwall’s Lemma, we deduce (A.11). Finally, we prove the dependence of the
resolvent map on different controls $\theta_{1},\theta_{2}\in
L^{2}([0,T];\mathbb{R}^{m})$. Given $x\in B_{R}(0)$ and $v\neq 0$, let us
define $\xi^{\theta_{1}}(s):=\mathcal{R}^{\theta_{1}}_{(\tau,s)}(x)\cdot v$
and $\xi^{\theta_{2}}(s):=\mathcal{R}^{\theta_{2}}_{(\tau,s)}(x)\cdot v$ for
every $s\in[0,T]$. Then, we compute

$\displaystyle|\xi^{\theta_{1}}(s)-\xi^{\theta_{2}}(s)|$
$\displaystyle\leq\int_{\tau}^{s}\left|\nabla_{x}{\mathcal{F}}(\sigma,\Phi_{(0,\sigma)}^{\theta_{1}}(x),\theta_{1}(\sigma))\xi^{\theta_{1}}(\sigma)-\nabla_{x}{\mathcal{F}}(\sigma,\Phi_{(0,\sigma)}^{\theta_{2}}(x),\theta_{2}(\sigma))\xi^{\theta_{2}}(\sigma)\right|\,d\sigma$
$\displaystyle\leq\int_{\tau}^{s}\left|\nabla_{x}{\mathcal{F}}(\sigma,\Phi_{(0,\sigma)}^{\theta_{1}}(x),\theta_{1}(\sigma))-\nabla_{x}{\mathcal{F}}(\sigma,\Phi_{(0,\sigma)}^{\theta_{2}}(x),\theta_{2}(\sigma))\right||\xi^{\theta_{1}}(\sigma)|\,d\sigma$
$\displaystyle\quad+\int_{\tau}^{s}\left|\nabla_{x}{\mathcal{F}}(\sigma,\Phi_{(0,\sigma)}^{\theta_{2}}(y),\theta(\sigma))\right||\xi^{\theta_{1}}(\sigma)-\xi^{\theta_{2}}(\sigma)|\,d\sigma$
$\displaystyle\leq
C_{1}|v|\int_{\tau}^{s}L_{\overline{R}}(1+\theta_{1}(\sigma)^{2})\left|\Phi_{(0,\sigma)}^{\theta_{1}}(x)-\Phi_{(0,\sigma)}^{\theta_{2}}(x)\right|\,d\sigma$
$\displaystyle\quad+C_{1}|v|\int_{\tau}^{s}L_{\overline{R}}(1+|\theta_{1}(\sigma)|+|\theta_{2}(\sigma)|)|\theta_{1}(\sigma)-\theta_{2}(\sigma)|\,d\sigma$
$\displaystyle\quad+\int_{\tau}^{s}L_{\overline{R}}(1+\theta(\sigma)^{2})|\xi^{\theta_{1}}(\sigma)-\xi^{\theta_{2}}(\sigma)|\,d\sigma,$

where we used Assumption 2$-(iv)$-$(vi)$. ∎

## References

  * [1] Luigi Ambrosio, Luis Caffarelli, Michael G Crandall, Lawrence C Evans, Nicola Fusco and Luigi Ambrosio “Transport equation and Cauchy problem for non-smooth vector fields” In _Calculus of Variations and Nonlinear Partial Differential Equations: With a historical overview by Elvira Mascolo_ Springer, 2008, pp. 1–41
  * [2] Luigi Ambrosio, Nicola Gigli and Giuseppe Savaré “Gradient flows: in metric spaces and in the space of probability measures” Springer Science & Business Media, 2005
  * [3] Martin Arjovsky, Soumith Chintala and Léon Bottou “Wasserstein generative adversarial networks” In _International conference on machine learning_ , 2017, pp. 214–223 PMLR
  * [4] J.-P. Aubin “Viability Theory” Birkhäuser Boston, MA, 2010 URL: doi.org/10.1007/978-0-8176-4910-4
  * [5] Yoshua Bengio, Patrice Simard and Paolo Frasconi “Learning long-term dependencies with gradient descent is difficult” In _IEEE transactions on neural networks_ 5.2 IEEE, 1994, pp. 157–166
  * [6] Martin Benning, Elena Celledoni, Matthias J Ehrhardt, Brynjulf Owren and Carola-Bibiane Schönlieb “Deep learning as optimal control problems: Models and numerical methods” In _arXiv preprint arXiv:1904.05657_ , 2019
  * [7] Benoît Bonnet, Cristina Cipriani, Massimo Fornasier and Hui Huang “A measure theoretical approach to the Mean-field Maximum Principle for training NeurODEs” In _Nonlinear Analysis_ 227 Elsevier, 2023, pp. 113161
  * [8] Benoît Bonnet and Hélène Frankowska “Viability and Exponentially Stable Trajectories for Differential Inclusions in Wasserstein Spaces” In _2022 IEEE 61st Conference on Decision and Control (CDC)_ , 2022, pp. 5086–5091 IEEE
  * [9] Benoît Bonnet-Weill and Hélène Frankowska “On the Viability and Invariance of Proper Sets under Continuity Inclusions in Wasserstein Spaces”, 2023
  * [10] Alberto Bressan and Benedetto Piccoli “Introduction to the mathematical theory of control” American institute of mathematical sciences Springfield, 2007
  * [11] Leon Bungert, Tim Roith, Daniel Tenbrinck and Martin Burger “Neural architecture search via Bregman iterations” In _arXiv preprint arXiv:2106.02479_ , 2021
  * [12] Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert and Elliot Holtham “Reversible architectures for arbitrarily deep residual neural networks” In _Proceedings of the AAAI conference on artificial intelligence_ 32.1, 2018
  * [13] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt and David K Duvenaud “Neural ordinary differential equations” In _Advances in neural information processing systems_ 31, 2018
  * [14] F.L. Chernousko and A.A. Lyubushin “Method of successive approximations for solution of optimal control problems” In _Optimal Control Applications and Methods_ 3.2 Wiley Online Library, 1982, pp. 101–114
  * [15] Hung-Hsu Chou, Johannes Maly and Holger Rauhut “More is less: Inducing sparsity via overparameterization” In _Information and Inference: A Journal of the IMA_ 12.3 Oxford University Press, 2023, pp. iaad012
  * [16] Jose A. Costa and Alfred O. Hero “Learning intrinsic dimension and intrinsic entropy of high-dimensional datasets” In _2004 12th European Signal Processing Conference_ , 2004, pp. 369–372
  * [17] Francesco Denti, Diego Doimo, Alessandro Laio and Antonietta Mira “The generalized ratios intrinsic dimension estimator” In _Scientific Reports_ 12.1 Nature Publishing Group UK London, 2022, pp. 20005
  * [18] Weinan E “A proposal on machine learning via dynamical systems” In _Communications in Mathematics and Statistics_ 1.5, 2017, pp. 1–11
  * [19] Weinan E, Jiequn Han and Qianxiao Li “A mean-field optimal control formulation of deep learning” In _arXiv preprint arXiv:1807.01083_ , 2018
  * [20] Carlos Esteve, Borjan Geshkovski, Dario Pighin and Enrique Zuazua “Large-time asymptotics in deep learning” In _arXiv preprint arXiv:2008.02491_ , 2020
  * [21] Massimo Fornasier, Pascal Heid and Giacomo Sodini “In preparation”, 2023
  * [22] Borjan Geshkovski and Enrique Zuazua “Turnpike in optimal control of PDEs, ResNets, and beyond” In _Acta Numerica_ 31 Cambridge University Press, 2022, pp. 135–263
  * [23] Sebastian Goldt, Marc Mézard, Florent Krzakala and Lenka Zdeborová “Modeling the influence of data structure on learning in neural networks: The hidden manifold model” In _Physical Review X_ 10.4 APS, 2020, pp. 041044
  * [24] Ian Goodfellow, Yoshua Bengio and Aaron Courville “Deep Learning” http://www.deeplearningbook.org MIT Press, 2016
  * [25] Ian J Goodfellow, Jonathon Shlens and Christian Szegedy “Explaining and harnessing adversarial examples” In _arXiv preprint arXiv:1412.6572_ , 2014
  * [26] Eldad Haber and Lars Ruthotto “Stable architectures for deep neural networks” In _Inverse problems_ 34.1 IOP Publishing, 2017, pp. 014004
  * [27] Jack K Hale “Ordinary differential equations” In _Wiley-Interscience_ , 1969
  * [28] Kaiming He and Jian Sun “Convolutional neural networks at constrained time cost” In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 5353–5360
  * [29] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun “Deep residual learning for image recognition” In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [30] Geoffrey E Hinton, Simon Osindero and Yee-Whye Teh “A fast learning algorithm for deep belief nets” In _Neural computation_ 18.7 MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info …, 2006, pp. 1527–1554
  * [31] Gerasimos E Ladas and Vangipuram Lakshmikantham “Differential equations in abstract spaces” Academic press, 1972
  * [32] Elizaveta Levina and Peter Bickel “Maximum likelihood estimation of intrinsic dimension” In _Advances in neural information processing systems_ 17, 2004
  * [33] Qianxiao Li, Long Chen and Cheng Tai “Maximum principle based algorithms for deep learning” In _arXiv preprint arXiv:1710.09513_ , 2017
  * [34] Hailiang Liu and Peter Markowich “Selection dynamics for deep neural networks” In _Journal of Differential Equations_ 269.12 Elsevier, 2020, pp. 11540–11574
  * [35] Iuri Macocco, Aldo Glielmo, Jacopo Grilli and Alessandro Laio “Intrinsic dimension estimation for discrete metrics” In _Physical Review Letters_ 130.6 APS, 2023, pp. 067401
  * [36] Song Mei, Theodor Misiakiewicz and Andrea Montanari “Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit” In _Conference on Learning Theory_ , 2019, pp. 2388–2464 PMLR
  * [37] Lev Semenovich Pontryagin “Mathematical theory of optimal processes” CRC press, 1987
  * [38] Olaf Ronneberger, Philipp Fischer and Thomas Brox “U-net: Convolutional networks for biomedical image segmentation” In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , 2015, pp. 234–241 Springer
  * [39] David E Rumelhart, Geoffrey E Hinton and Ronald J Williams “Learning internal representations by error propagation”, 1985
  * [40] Yoshiyuki Sakawa and Yuji Shindo “On global convergence of an algorithm for optimal control” In _IEEE Transactions on Automatic Control_ 25.6 IEEE, 1980, pp. 1149–1153
  * [41] Filippo Santambrogio “{Euclidean, metric, and Wasserstein} gradient flows: an overview” In _Bulletin of Mathematical Sciences_ 7 Springer, 2017, pp. 87–154
  * [42] Alessandro Scagliotti “A gradient flow equation for optimal control problems with end-point cost” In _Journal of Dynamical and Control Systems_ Springer, 2022, pp. 1–48
  * [43] Alessandro Scagliotti “Deep Learning approximation of diffeomorphisms via linear-control systems” In _Mathematical Control and Related Fields_ 13.3 Mathematical ControlRelated Fields, 2023, pp. 1226–1257
  * [44] Alessandro Scagliotti “Optimal control of ensembles of dynamical systems” In _ESAIM: Control, Optimisation and Calculus of Variations_ 29 EDP Sciences, 2023, pp. 22
  * [45] Ferdia Sherry, Elena Celledoni, Matthias J Ehrhardt, Davide Murari, Brynjulf Owren and Carola-Bibiane Schönlieb “Designing Stable Neural Networks using Convex Analysis and ODEs” In _arXiv preprint arXiv:2306.17332_ , 2023
  * [46] Matthew Thorpe and Yves Gennip “Deep limits of residual neural networks” In _Research in the Mathematical Sciences_ 10.1 Springer, 2023, pp. 6
  * [47] Yijia Zheng, Tong He, Yixuan Qiu and David P Wipf “Learning Manifold Dimensions with Conditional Variational Autoencoders” In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 34709–34721