# Collective motion driven by nutrient consumption

Pierre-Emmanuel Jabin P.–E. Jabin. Department of Mathematics and Huck
Institutes, Pennsylvania State University, State College, PA 16801, USA.
Email<EMAIL_ADDRESS>Benoît Perthame Sorbonne Université, CNRS, Université
de Paris, Inria, Laboratoire Jacques-Louis Lions UMR7598, F-75005 Paris. Email
<EMAIL_ADDRESS>has received funding from the
European Research Council (ERC) under the European Union’s Horizon 2020
research and innovation programme (grant agreement No 740623). P.E.J. is
partially supported by NSF DMS Grants DMS-2049020, DMS-2205694, and
DMS-2219397.

###### Abstract

A classical problem describing the collective motion of cells, is the movement
driven by consumption/depletion of a nutrient. Here we analyze one of the
simplest such model written as a coupled Partial Differential
Equation/Ordinary Differential Equation system which we scale so as to get a
limit describing the usually observed pattern. In this limit the cell density
is concentrated as a moving Dirac mass and the nutrient undergoes a
discontinuity.

We first carry out the analysis without diffusion, getting a complete
description of the unique limit. When diffusion is included, we prove several
specific a priori estimates and interpret the system as a heterogeneous
monostable equation. This allow us to obtain a limiting problem which shows
the concentration effect of the limiting dynamics.

2010 Mathematics Subject Classification. Primary : 35B25. Secondary: 35B36,
35D40, 35K57, 35B25, 35Q92, 92C17.
Keywords and phrases. Asymptotic analysis; Pattern formation; Reaction-
diffusion equations.

##  1 Introduction

A classical problem describing the collective motion of cells, is the movement
driven by consumption/depletion of a nutrient [11, 15, 16]. The simplest
description uses a number density of cells $u_{\varepsilon}$ and a nutrient
concentration $v_{\varepsilon}$. It is written

$\begin{cases}\partial_{t}u_{\varepsilon}-\varepsilon\partial^{2}_{xx}u_{\varepsilon}=\frac{1}{\varepsilon}u_{\varepsilon}(v_{\varepsilon}-\mu),\qquad
t\geq 0,\;x\in\mathbb{R},\\\
\partial_{t}v_{\varepsilon}=-u_{\varepsilon}v_{\varepsilon},\end{cases}$ (1)

completed with initial data $u_{\varepsilon}^{0}$, $v_{\varepsilon}^{0}$, such
that

$\varepsilon u_{\varepsilon}^{0}\in L^{1}_{+}(\mathbb{R}),\qquad 0<v_{m}\leq
v_{\varepsilon}^{0}\leq v_{M}<\infty,\qquad v_{m}<\mu<v_{M}.$

We have introduced a parameter $\varepsilon$ which measures the time scale of
the cell random motion compared to nutrient consumption. Our interest here is
when this parameter is small because it is a case when a pattern is produced
under the form of a high concentration of cells despite the parabolic
character of Equation (1). In fact this phenomena is closely related to
concentration effects in non-local semi-linear parabolic equations as studied
intensively recently, see [9, 17, 6, 14] and the references therein. This
analogy leads us to postulate that $u_{\varepsilon}$ concentrate as Dirac
masses at points where $v_{\varepsilon}$ undergoes a discontinuity.
The scale proposed here, which is chosen to produce a distinguished limit, is
usual for semi-linear diffusion equations and has been studied for local
problems in classical works, [10, 1]. The most efficient method is to use the
Hopf-Cole transform and viscosity solutions of Hamilton-Jacobi equations [8,
7]. We restrict our analysis to one dimension to explain the solution
structure as depicted in Figure 1 but significant parts of our analysis can be
extended to several dimensions.

Several related studies can be mentioned. Coupling Ordinary and Partial
Differential Equation is rather classical in different areas: for pattern
formation, see [12] (study of existence and stability of stationary
solutions), modeling of two species dynamics with an unmotile specie [5, 18]
for instance. Traveling waves with a non-motile phase have also been studied,
see [19] and the references therein. However we are not aware of any
analytical study related to the scaling proposed here.

Figure 1:  Traveling wave solution of Equation (1). In blue/solid line the
component $u_{\varepsilon}$. In red/dashed line, the nutrient
$v_{\varepsilon}$.

Our approach combines a reformulation of the problem which leads to a
heterogeneous monostable equation for $v_{\varepsilon}$ and the standard Hopf-
Cole transform as mentioned above. This a convenient tool to represent the
Dirac concentration of $u_{\varepsilon}$ under the form
$\exp(-\varphi(t,x)/\varepsilon)$ where $\varphi$ behaves like a quadratic
function.

In Section 2, we begin with a simpler case where we omit the diffusion on
$u_{\varepsilon}$, arriving to a system of ordinary differential equations
which can be solved nearly explicitly. This allows us to introduce the tools
which are used in Section 3 where we state and prove the concentration effect.
Several related questions are detailed in appendices: the particular case of
traveling waves, and some Sobolev regularity results.

##  2 The problem without diffusion

We begin with the simpler case where diffusion is ignored and where we can
give a complete description while introducing the main tools for the general
problem. We are reduced to a system of two differential equations with a
parameter $x$, which solutions however behave as a front propagation in space,
namely

$\begin{cases}\partial_{t}u_{\varepsilon}(t,x)=\frac{1}{\varepsilon}u_{\varepsilon}(v_{\varepsilon}-\mu),\qquad
t\geq 0,\;x\in\mathbb{R},\\\
\partial_{t}v_{\varepsilon}=-u_{\varepsilon}v_{\varepsilon}.\end{cases}$ (2)

We define

$Q(v)=v-\mu\ln v.$

We assume there are constants $Q_{m}$ and $Q_{M}$ such that

$\begin{cases}\text{For }x<0,\quad v_{\varepsilon}^{0}(x)<\mu,\qquad\text{and
for }x>0\quad v_{\varepsilon}^{0}(x)>\mu,\\\
Q(v^{0}_{\varepsilon}(x))+\varepsilon u^{0}_{\varepsilon}\leq Q_{M},\qquad
Q(v^{0}_{\varepsilon}(x))>Q_{m}>Q(\mu),\\\ v_{\varepsilon}^{0}\to
v^{0}\quad\text{pointwise for }x\neq 0.\end{cases}$ (3)

In other words $v^{0}_{\varepsilon}$ has an initial increasing discontinuity
at $x=0$. For the cell density, we assume that in the weak sense of measures,
as $\varepsilon\to 0$,

$\begin{cases}u_{\varepsilon}^{0}=\exp(\frac{\varphi_{\varepsilon}^{0}}{\varepsilon})\rightharpoonup\rho^{0}\delta(x)\quad\text{in
the weak sense of measures},\\\
\varphi_{\varepsilon}^{0}\to\varphi^{0}\quad\text{in
}C(\mathbb{R}),\qquad\varphi^{0}<0\text{ for }x\neq 0.\end{cases}$ (4)

In this framework, we can describe the behavior of solutions as follows

###### Theorem 1

Assume (3)– (4). The solution of (2) has limits
$u_{\varepsilon}\rightharpoonup u$ (weak measures) and $v_{\varepsilon}\to v$
(strongly in $L^{p}_{loc}$) and, for $x>0$ there is a time $\tau(x)$ and
$v^{0}_{-}(x)<\mu<v^{0}(x)$ such that
(i) $\;v(t,x)=v^{0}_{-}(x)$ for $t<\tau(x)$, $v(t,x)=v^{0}(x)$ for
$t>\tau(x)$,
(ii) $u(t,x)=[\ln v^{0}(x)-\ln v^{0}_{-}(x)]\delta(t-\tau(x))$,
(iii) $\tau(x)=-\frac{\varphi^{0}(x)}{v^{0}(x)-\mu}$ with $\varphi$ defined by
(8).

For $x<0$, we have $v(t,x)=v^{0}(x)$.

Proof. A remarkable property of the system (2) is the identity

$\partial_{t}[\varepsilon u_{\varepsilon}+v_{\varepsilon}-\mu\ln
v_{\varepsilon}]=0,$

which implies

$\varepsilon
u_{\varepsilon}+Q(v_{\varepsilon})=Q(v_{\varepsilon}^{0}(x))+\varepsilon
u^{0}_{\varepsilon}\leq Q_{M},\quad\forall x\in\mathbb{R}.$ (5)

Using this inequality and the fact that $v_{\varepsilon}$ decreases, we first
conclude for $x<0$, $v_{\varepsilon}-\mu<0$ and thus $u_{\varepsilon}(t,x)\to
0$ as $\varepsilon\to 0$, therefore $v_{\varepsilon}(t,x)\to v^{0}(x)$

For $x>0$, we conclude from (5) and assumption (3), that there are constants
such that

$v_{m}\leq v_{\varepsilon}(t,x)\leq v_{M}.$ (6)

Also, integrating the second equation of (2), we have for all $T>0$,

$\int_{0}^{T}u_{\varepsilon}(t,x)dt=\ln v_{\varepsilon}^{0}(x)-\ln
v_{\varepsilon}(T,x)\leq C(T),$

and since we expect that $u_{\varepsilon}$ is a concentrated measure, we
define $\varphi_{\varepsilon}=\varepsilon\ln u_{\varepsilon}$. It satisfies

$\partial_{t}\varphi_{\varepsilon}=v_{\varepsilon}-\mu\quad\text{ is bounded
in $t$ and $x$}.$ (7)

We can argue $x$ by $x$ and define as $\varepsilon\to 0$ (after extraction of
subsequences, but the uniqueness of the limit shows that it is the full
family) the strong limits

$v_{\varepsilon}(t,x)\to v(t,x),\qquad\varphi_{\varepsilon}\to\varphi(t,x).$

We can also define the weak limit

$\qquad u_{\varepsilon}(t,x)\rightharpoonup m(t,x)\geq
0\in{\mathcal{M}}^{1}(0,T).$

These limits satisfy the equations obtained passing to the limit in (2) and
(7)

$\begin{cases}\partial_{t}\varphi(t,x)=v-\mu,\qquad\varphi(t,x)\leq 0\\\
\partial_{t}\ln(v)=-m(t,x),\qquad\text{supp}(m)\subset\\{\varphi(t)=0\\}.\\\
Q(v(t,x))=Q(v^{0}(x)).\end{cases}$ (8)

Because of this last equality, $v(t,x)$ belongs to one of the two branches
$v^{0}_{-}(x)<v^{0}_{+}(x):=v^{0}(x)$ of roots of $Q(v(t,x))=Q(v^{0}(x))$.
Therefore $v-\mu$ is away from $0$ and there is a first time $\tau(x)$ such
that $\varphi\big{(}\tau(x),x\big{)}=0$ and $\varphi(t,x)<0$ for
$t\neq\tau(x)$. This time $\tau(x)$ is also the jump time from one branch to
the other for $v$ as stated in (i).

We can also compute, from the above equation on $\ln v$, and this gives (ii),
namely

$u_{\varepsilon}=-\partial_{t}\ln v_{\varepsilon}\rightharpoonup u=[\ln
v_{+}(x)-\ln v_{-}(x)]\delta(t-\tau(x)).$

The time $\tau(x)$ is fully identified from the limiting system: By
integrating the equation on $\varphi(t,x)$, we find that

$\varphi(t,x)=\left\\{\begin{aligned} &\varphi(0,x)+t\,(v^{0}(x)-\mu),\quad
t<\tau(x),\\\
&\varphi(0,x)+\tau(x)(v^{0}(x)-\mu)+(t-\tau(x))\,(v_{-}^{0}(x)-\mu),\quad
t>\tau(x).\\\ \end{aligned}\right.$

For $x>0$, as $v^{0}_{-}(x)<\mu<v^{0}(x)$, $\varphi(t,x)$ is strictly
increasing in time for $t<\tau(x)$, and strictly decreasing for $t>\tau(x)$.

Furthermore by the second equation, we have that $\varphi(\tau(x),x)=0$. Hence
this characterizes $\tau(x)$ as

$\tau(x)=-\frac{\varphi^{0}(x)}{v^{0}(x)-\mu}.$

This gives (iii) and identifies completely the limiting solution $x$ by $x$.

##  3 The full problem

When diffusion is included, the previous analysis, which uses fundamentally
$x$ by $x$ convergence, does not apply and the assumptions have to take into
account the diffusion term. We also make more general assumptions. For the
initial data, we assume that

$u^{0}_{\varepsilon}>0,\qquad\varepsilon u^{0}_{\varepsilon}\leq
C,\qquad\int_{\mathbb{R}}u^{0}_{\varepsilon}dx\leq C,$ (9)

and, $\varphi^{0}_{\varepsilon}=\varepsilon\ln u^{0}_{\varepsilon}$ satisfies

$\varepsilon\partial^{2}_{xx}\varphi^{0}_{\varepsilon}\leq
C,\qquad|\partial_{x}\varphi^{0}_{\varepsilon}|\leq
C,\qquad\varphi^{0}_{\varepsilon}(x)\geq-C(1+|x|),$ (10)
$\varepsilon\,|\partial^{2}_{xx}\ln v^{0}_{\varepsilon}|\leq
C,\qquad|\partial_{x}\ln v^{0}_{\varepsilon}|+|\ln v^{0}_{\varepsilon}|\leq
C.$ (11)

Then, recalling the definition $Q(v)=v-\mu\ln v$, we also use

$w=\ln v,\qquad{\widetilde{Q}}(w)=Q(v)=e^{w}-\mu w,$ (12)

and we can define uniquely two smooth branches of initial data
$w^{0}_{\pm,\varepsilon}$ by

${\widetilde{Q}}(w^{0}_{\varepsilon})={\widetilde{Q}}(w^{0}_{-,\varepsilon})={\widetilde{Q}}(w^{0}_{+,\varepsilon}),\qquad
w^{0}_{-,\varepsilon}\leq\ln\mu\leq w^{0}_{+,\varepsilon}.$ (13)

Note that assumption (11) provides corresponding bounds on
$w^{0}_{\varepsilon}$ and $w^{0}_{\pm,\varepsilon}$, which are

$|w^{0}_{\varepsilon}|+|w^{0}_{\pm,\varepsilon}|\leq
C,\quad|\partial_{x}w^{0}_{\varepsilon}|+|\partial_{x}w^{0}_{\pm,\varepsilon}|\leq
C,\quad\varepsilon\,|\partial_{xx}^{2}w^{0}_{\varepsilon}|+\varepsilon|\partial_{xx}^{2}w^{0}_{\pm,\varepsilon}|\leq
C.$ (14)

And we also use, for an unessential result, the stronger condition

$\varepsilon\partial^{2}_{xx}(w^{0}_{\varepsilon}-w^{0}_{-,\varepsilon})+u^{0}_{\varepsilon}\leq
C.$ (15)

Finally, we use the notation $|W|_{-}=\max(0,-W)$.

Our main theorem now reads

###### Theorem 2

Assume that $u_{\varepsilon}^{0}\in C^{2}\cap L^{1}(\mathbb{R})$ and that
assumptions (9)–(11) hold. The solution of (1) has limits
$u_{\varepsilon}\rightharpoonup u$ (weak measures) and $v_{\varepsilon}\to v$
(strongly in $L^{p}_{loc}$) and, there is a time $\tau(x)$ and
$v^{0}_{-}(x)<\mu<v^{0}(x)$ such that
(i) $\;v(t,x)=v^{0}_{-}(x)$ for $t<\tau(x)$, $v(t,x)=v^{0}(x)$ for
$t>\tau(x)$,
(ii) $u(t,x)=[\ln v^{0}(x)-\ln v^{0}_{-}(x)]\delta(t-\tau(x))$.

Compared to the case without diffusion in Theorem 1, there is no simple
explicit formula for the jump time $\tau(x)$. However, here it is also
characterized by $\varphi(\tau(x),x)=0$ for the solution of an Eikonal
equation.

The end of this section is devoted to the proof of Theorem 1.

Our analysis is based on the observation that $\partial_{t}\ln
v_{\varepsilon}=-u_{\varepsilon}$, thus, from (1), we get the identity

$\partial_{t}[\varepsilon u_{\varepsilon}+v_{\varepsilon}-\mu\ln
v_{\varepsilon}+\varepsilon^{2}\partial^{2}_{xx}\ln v_{\varepsilon}]=0,$

and consequently

$\varepsilon\partial_{t}\ln
v_{\varepsilon}-\varepsilon^{2}\partial^{2}_{xx}\ln
v_{\varepsilon}-Q(v_{\varepsilon})=-Q(v^{0}_{\varepsilon})-\varepsilon
u^{0}_{\varepsilon}-\varepsilon^{2}\partial^{2}_{xx}\ln v^{0}_{\varepsilon},$

which we can write in terms of $w_{\varepsilon}$ as

$\varepsilon\partial_{t}w_{\varepsilon}-\varepsilon^{2}\partial^{2}_{xx}w_{\varepsilon}-{\widetilde{Q}}(w_{\varepsilon})=-{\widetilde{Q}}(w^{0}_{\varepsilon})-\varepsilon
u^{0}_{\varepsilon}-\varepsilon^{2}\partial^{2}_{xx}w^{0}_{\varepsilon}.$ (16)

Notice that this is a monostable equation of Fisher/KPP type, where the steady
states depend on $x$.

### A priori estimates on $v_{\varepsilon}$.

###### Lemma 3

The inequalities hold

$0<C\leq w_{\varepsilon}(t,x)\leq w^{0}_{+,\varepsilon},$ (17)

and for any $R$, there is a constant $C_{R}$ such that,

$\int_{|x|\leq
R}|{\widetilde{Q}}(w_{\varepsilon})-{\widetilde{Q}}(w^{0}_{\varepsilon})|{1{\rm
I}}_{\\{w_{\varepsilon}-w^{0}_{-,\varepsilon}\leq 0\\}}\,dx\leq
C_{R}\,\varepsilon,\qquad\forall t\geq 0.$ (18)

Finally, with the additional assumption (15), we have
$-C\sqrt{\varepsilon}+w^{0}_{-,\varepsilon}\leq w_{\varepsilon}(t,x)$.

Proof. For the first statement, on the one hand, we note that we necessarily
have that for any $x$, $w_{\varepsilon}^{0}(x)=w^{0}_{-,\varepsilon}(x)$ or
$w_{\varepsilon}^{0}(x)=w^{0}_{+,\varepsilon}(x)$. In both cases, that implies
that $w_{\varepsilon}^{0}(x)\leq w^{0}_{+,\varepsilon}(x)$. Since
$v_{\varepsilon}$ is non-increasing in time, so is $w_{\varepsilon}$ and

$w_{\varepsilon}(t,x)\leq w^{0}_{\varepsilon}(x)\leq
w^{0}_{+,\varepsilon}(x).$

On the other hand, as $\partial_{t}w_{\varepsilon}\leq 0$, we can deduce from
(16), that

$-\varepsilon^{2}\partial^{2}_{xx}(w_{\varepsilon}-w^{0}_{-,\varepsilon})+[{\widetilde{Q}}(w^{0}_{-,\varepsilon})-{\widetilde{Q}}(w_{\varepsilon})]\geq-\varepsilon^{2}\partial^{2}_{xx}(w^{0}_{\varepsilon}-w^{0}_{-,\varepsilon})-\varepsilon
u^{0}_{\varepsilon}.$ (19)

Using the maximum principle and assumptions (9), (11), we conclude that at a
minimum value of $w_{\varepsilon}-w^{0}_{-,\varepsilon}$, the quantity
${\widetilde{Q}}(w^{0}_{-,\varepsilon})-{\widetilde{Q}}(w_{\varepsilon})$ is
controlled from below and the lower bound on $w_{\varepsilon}$ follows.

Furthermore, from the usual convex inequalities, we also observe that

$-\varepsilon^{2}\partial^{2}_{xx}(w_{\varepsilon}-w^{0}_{-,\varepsilon})_{-}+[{\widetilde{Q}}(w_{\varepsilon})-{\widetilde{Q}}(w^{0}_{-,\varepsilon})]{1{\rm
I}}_{\\{w_{\varepsilon}-w^{0}_{-,\varepsilon}<0\\}}\leq\varepsilon\,u^{0}_{\varepsilon}+\varepsilon^{2}|\partial^{2}_{xx}(w^{0}_{\varepsilon}-w^{0}_{-,\varepsilon})|.$
(20)

Because the set $\\{W\,s.t.\,W-w^{0}_{-,\varepsilon}<0\\}$ lies in the
decreasing branch of ${\widetilde{Q}}$, the quantity
$[{\widetilde{Q}}(w_{\varepsilon})-{\widetilde{Q}}(w^{0}_{-,\varepsilon})]{1{\rm
I}}_{\\{w_{\varepsilon}-w^{0}_{-,\varepsilon}<0\\}}$ is positive. We integrate
against some smooth non-negative $\psi$ with $\psi=1$ on $B(0,R)$ and $\psi$
compactly supported in $B(0,2R)$. Since we have already proved that
$w_{\varepsilon}-w^{0}_{-,\varepsilon}$ is bounded, we obtain the estimate

$\int_{|x|\leq
R}|{\widetilde{Q}}(w_{\varepsilon})-{\widetilde{Q}}(w^{0}_{\varepsilon})|{1{\rm
I}}_{\\{w_{\varepsilon}-w^{0}_{-,\varepsilon}\leq 0\\}}\,dx\leq
C_{R}\,\varepsilon^{2}+\varepsilon\,\int_{|x|\leq
R}u_{\varepsilon}^{0}\,dx+\varepsilon^{2}\,\int_{|x|\leq
R}|\partial^{2}_{xx}(w^{0}_{\varepsilon}-w^{0}_{-,\varepsilon})|\,dx.$

Using assumption (9) and assumption (11) concludes the second point of the
lemma.

We may also use the specific assumption (15) in (20), we obtain that

$-\varepsilon^{2}\partial^{2}_{xx}(w_{\varepsilon}-w^{0}_{-,\varepsilon})_{-}+[{\widetilde{Q}}(w_{\varepsilon})-{\widetilde{Q}}(w^{0}_{-,\varepsilon})]{1{\rm
I}}_{\\{w_{\varepsilon}-w^{0}_{-,\varepsilon}<0\\}}\leq C\varepsilon.$

Recalling that
$[{\widetilde{Q}}(w_{\varepsilon})-{\widetilde{Q}}(w^{0}_{-,\varepsilon})]{1{\rm
I}}_{\\{w_{\varepsilon}-w^{0}_{-,\varepsilon}<0\\}}$ is positive, we conclude
that

$Q(w_{\varepsilon})-{\widetilde{Q}}(w^{0}_{-,\varepsilon}){1{\rm
I}}_{\\{w_{\varepsilon}-w^{0}_{-,\varepsilon}<0\\}}\leq C\varepsilon,$

and thus the third statement of Lemma 3 holds, namely
$w_{\varepsilon}-w_{-,\varepsilon}\geq-C\,\sqrt{\varepsilon}$.

### Concentration dynamics of $u_{\varepsilon}$.

We turn to the study of $u_{\varepsilon}$ and begin with some simple
estimates.
$\bullet$ Since $u_{\varepsilon}=-\partial_{t}w_{\varepsilon}$, and using the
bound (17), we find

$\sup_{x}\int_{0}^{T}u_{\varepsilon}(t,x)dt=\sup_{x}[w_{\varepsilon}^{0}(x)-w_{\varepsilon}(x,T)]\leq
C(T),$ (21)

and thus, integrating the equation on $u_{\varepsilon}$, we also get

$\varepsilon\,\int_{\mathbb{R}}u_{\varepsilon}(t,x)\,dx=\varepsilon\,\int_{\mathbb{R}}u^{0}_{\varepsilon}(x)\,dx+\int_{\mathbb{R}}\int_{0}^{t}u_{\varepsilon}(s,x)(v_{\varepsilon}(s,x)-\mu)ds\,dx\leq
C(t).$ (22)

$\bullet$ Next, we use the Hopf-Cole transform

$\varphi_{\varepsilon}(t,x)=\varepsilon\ln u_{\varepsilon}(t,x).$

As usual, we compute that $\varphi_{\varepsilon}$ satisfies the Eikonal
equation

$\partial_{t}\varphi_{\varepsilon}=\varepsilon\partial^{2}_{xx}\varphi_{\varepsilon}+|\partial_{x}\varphi_{\varepsilon}|^{2}+v_{\varepsilon}-\mu.$
(23)

We are going to show some uniform bounds on $\varphi_{\varepsilon}$.

###### Lemma 4

We have

$\partial_{t}\varphi_{\varepsilon}(t,x)\leq C,\qquad\forall
x\in\mathbb{R},\;t\geq 0$ (24)
$\|\partial_{x}\varphi_{\varepsilon}(t)\|_{L^{\infty}(\mathbb{R})}\leq
C,\qquad\|\partial_{t}\varphi_{\varepsilon}\|_{L^{p}_{t,x}}\leq
C(p),\quad\forall p\in[1,\infty),$ (25)
$-C(t)(1+|x|)\leq\varphi_{\varepsilon}(t,x)\leq
2\,\varepsilon\,\ln\frac{1}{\varepsilon}+C(t)\,\varepsilon,\qquad\forall
x\in\mathbb{R},\;t\geq 0.$ (26)

Proof. For the time derivative, differentiating (23) and using the equation on
$v_{\varepsilon}$, we find

$\partial_{t}(\partial_{t}\varphi_{\varepsilon})=\varepsilon\partial^{2}_{xx}(\partial_{t}\varphi_{\varepsilon})+2\partial_{x}\varphi_{\varepsilon}\partial_{x}(\partial_{t}\varphi_{\varepsilon})-u_{\varepsilon}v_{\varepsilon}\leq\varepsilon\partial^{2}_{xx}(\partial_{t}\varphi_{\varepsilon})+2\partial_{x}\varphi_{\varepsilon}\partial_{x}(\partial_{t}\varphi_{\varepsilon}),$

so that the maximum principle gives
$\partial_{t}\varphi_{\varepsilon}(t,x)\leq\max_{x}\partial_{t}\varphi_{\varepsilon}^{0}(x)$
which gives (24) thanks to the assumption (10).

Next, we prove the Lipschitz bound. Consider any point $x_{\varepsilon}$ that
is a maximum in $x$ of $\partial_{x}\varphi_{\varepsilon}$ at any time $t$
(standard arguments apply if the maximum is not achieved, see [7, 2]). Then
$\partial_{xx}^{2}\varphi_{\varepsilon}(x_{\varepsilon})=0$ and we conclude,
still using (24), that

$|\partial_{x}\varphi_{\varepsilon}(t,x_{\varepsilon})|^{2}=\partial_{t}\varphi_{\varepsilon}(t,x_{\varepsilon})+v_{\varepsilon}-\mu\leq
C.$ (27)

Once $\partial_{x}\varphi_{\varepsilon}\in L^{\infty}_{t,x}$ uniformly,
standard parabolic estimates provide a uniform bound on
$\partial_{t}\varphi_{\varepsilon}$ in $L^{p}_{t,x}$ for any $1\leq p<\infty$.

Finally, since $\varphi_{\varepsilon}$ is uniformly Lipschitz in $x$, let
$x_{\varepsilon}$ be a maximum of $\varphi_{\varepsilon}$, then

$\varphi_{\varepsilon}(t,x)\geq\max\varphi_{\varepsilon}(t,.)-C\,|x-x_{\varepsilon}|,$

so that, using (22),

$\frac{C}{\varepsilon}\geq\int_{\mathbb{R}}u_{\varepsilon}(t,x)\,dx\geq\int_{\mathbb{R}}e^{\max\varphi_{\varepsilon}(t,.)/\varepsilon}\,e^{-C\,|x-x_{\varepsilon}|/\varepsilon}\,dx\geq\frac{\varepsilon}{C}\,e^{\max\varphi_{\varepsilon}(t,.)/\varepsilon},$

which proves the upper bound in (26). The lower bound relies, as it is
standard [2, 7, 3], on the construction of a sub-solution. Here one can
immediately check that $-C(t+1)-\frac{Cx^{2}}{\sqrt{1+|x|^{2}}}$ will work.

### Compactness of $v_{\varepsilon}$.

We introduce the quantity $\Phi(x,w)$, defined up to a constant, by

$\Phi_{w}(x,w)=|{\widetilde{Q}}(w^{0}_{\varepsilon}(x))-{\widetilde{Q}}(w)|\geq
0.$

###### Lemma 5

With assumptions (11)–(9), we have

$\sup_{x\in\mathbb{R},\,0\leq t\leq
T}|\partial_{x}w_{\varepsilon}|\leq\frac{C_{T}}{\varepsilon},\qquad\int_{0}^{T}\\!\\!\int_{|x|\leq
R}\big{|}\partial_{x}\Phi(x,w_{\varepsilon}(t,x))\big{|}\leq C_{T,R}.$

Consequently, by monotonicity of $\Phi$ in $w$, $v_{\varepsilon}$ is locally
compact in $L^{p}((0,\infty)\times\mathbb{R})$ for any $1\leq p<\infty$.

###### Remark 6

It is also possible to conclude from this lemma that $w_{\varepsilon}$ is
uniformly bounded in $L^{1}([0,\ T],\;W^{\theta,1}(\mathbb{R}))$ for some
$\theta>0$; see Appendix B.

Proof. First of all, calculate

$\partial_{t}\partial_{x}w_{\varepsilon}=-\partial_{x}u_{\varepsilon}=-\frac{\partial_{x}\varphi_{\varepsilon}}{\varepsilon}\,u_{\varepsilon},$

which yields, from the Lipschitz bound on $\varphi_{\varepsilon}$ in (27) and
the estimate (21),

$|\partial_{x}w_{\varepsilon}(t,x)|\leq|\partial_{x}w_{\varepsilon}^{0}(x)|+\frac{C}{\varepsilon}\,\int_{0}^{t}u_{\varepsilon}(s,x)\,ds\leq\frac{C(t)}{\varepsilon}.$

Next, we write
$\partial_{x}[\Phi(x,w_{\varepsilon}(t,x))]=\partial_{x}\Phi(x,w_{\varepsilon})+[{\widetilde{Q}}(w_{\varepsilon}^{0})-{\widetilde{Q}}(w_{\varepsilon})]\partial_{x}w_{\varepsilon}$.
Since $\partial_{x}\Phi(x,w_{\varepsilon})$ is bounded in $L^{\infty}$, and
thanks to the second bound in Lemma 3, we conclude that

$\int_{0}^{T}\\!\\!\int_{|x|\leq R}|\partial_{x}\Phi(x,w_{\varepsilon})|\leq
C_{T,R}+\varepsilon\sup_{|x|\leq R,\,0\leq t\leq
T}|\partial_{x}w_{\varepsilon}|\int_{0}^{T}\\!\\!\int_{|x|\leq
R}|\frac{Q(v^{0}_{\varepsilon})-Q(v_{\varepsilon})}{\varepsilon}|\leq
C_{T,R}.$ (28)

Since $\partial_{t}w_{\varepsilon}\leq 0$ and $w_{\varepsilon}$ bounded
provide us with compactness in time, we conclude that
$\Phi(x,w_{\varepsilon})$ is compact and thus converges a.e. By monotonicity
of $\Phi$ in $w_{\varepsilon}$, we also conclude that $w_{\varepsilon}$
converges a.e.

### Convergence as $\varepsilon\to 0$.

We are now ready to study the limit as $\varepsilon$ vanishes.
$\bullet$ The bounds in Lemma 4 show that $\varphi_{\varepsilon}$ is locally
compact in $C(\mathbb{R}_{+}\times\mathbb{R})$ and hence, after extraction of
a subsequence that we still denote by $\varepsilon$, there exists $\varphi$
which is Lipschitz in space and with time derivatives in $L^{p}$ such that

$\|\varphi_{\varepsilon}-\varphi\|_{L^{\infty}((0,T)\times(-R,R))}\to
0,\quad\mbox{as}\ \varepsilon\to 0,\qquad\forall T>0,\;R>0.$

We note that

$-C(1+t)-C|x|\leq\varphi(t,x)\leq 0.$ (29)

$\bullet$ From Lemma 5, we also conclude that $v_{\varepsilon}$ converges
locally; for any $1\leq p<\infty$,

$v_{\varepsilon}\to v,\quad w_{\varepsilon}\to w=\ln v\quad\text{in}\quad
L^{p}((0,T)\times(-R,R)),\qquad\forall T>0,\;R>0.$

$\bullet$ From the bound (21), we can also extract a subsequence such that, in
the weak sense of measures,

$u_{\varepsilon}\rightharpoonup
u,\quad\text{in}\quad\mathcal{M}((0,T)\times(-R,R)),\qquad\forall T>0,\;R>0.$

We may pass to the limit, in distributional sense, in Equations (1) and get,
as $\varepsilon\to 0$,

$u_{\varepsilon}v_{\varepsilon}\rightharpoonup
u\mu,\qquad\partial_{t}w=-u\qquad\partial_{t}v=\text{w-lim
}(-u_{\varepsilon}v_{\varepsilon})=-u\mu,$

which expresses that the concentration of the measure $u_{\varepsilon}$ is
exactly at the point where $v_{\varepsilon}=\mu$. And from Lemma 3, we know
that

${\widetilde{Q}}(w)={\widetilde{Q}}(w^{0}),\quad i.e.,\quad
w(t,x)=w^{0}_{-}(x)\quad or\quad w^{0}_{+}(x).$

Since $w$ is non-decreasing in time, we conclude that, for all
$x\in\mathcal{S}$, a subset of $\mathbb{R}$ where $w^{0}=w^{0}_{-}$, there is
a unique time $\tau(x)$ such that $w$ jumps from $w^{0}_{-}(x)$ to
$w^{0}_{+}(x)$ (with $\tau(x)=\infty$ for $x<0$), and

$u(t,x)=[w^{0}_{+}(x)-w^{0}_{-}(x)]\delta(t-\tau(x)){1{\rm
I}}_{\\{x\in\mathcal{S}\\}},\qquad\varphi(\tau(x),x)=0.$

### Open questions.

Uniqueness for the limit problem, which we proved when diffusion is ignored
(Section 2, is an open question in full generality. In particular it seems
hard to determine more properties about the set $\mathcal{S}$, which depends
on the initial data. In the monotone case when $w^{0}(x)=w^{0}_{-}(x)$ for
$x<0$ and $w^{0}(x)=w^{0}_{+}(x)$ for $x>0$ with $u^{0}=[[w^{0}]]\delta(x)$,
one can expect $\tau(x)$ be invertible and to obtain a front located at some
value $x=X(t)$.

Acknowledgments. The authors would like to thank the Institut Henri Poincaré
for its support and hospitality during the program “Mathematical modeling of
organization in living matter”. The authors also thank similarly the Isaac
Newton Institute for Mathematical Sciences, Cambridge, for support and
hospitality during the program ”Frontiers in kinetic theory: connecting
microscopic to macroscopic scales - KineCon 2022” where work on this paper was
undertaken. This work was supported by EPSRC grant no EP/K032208/1. B.P. has
received funding from the European Research Council (ERC) under the European
Union’s Horizon 2020 research and innovation programme (grant agreement No
740623). P. E. Jabin is partially supported by NSF DMS Grants DMS-2049020,
DMS-2205694, and DMS-2219397.

##  Appendix A Traveling wave

Traveling waves are an intuitive way to understand, in a very particular case,
the general behavior of system (1). considering solutions of the form
$u_{\varepsilon}(x-\sigma t)$, $v_{\varepsilon}(x-\sigma t)$,
$w_{\varepsilon}=\ln v_{\varepsilon}$. Recalling the notation
${\widetilde{Q}}(w)=e^{w}-\mu w$, we arrive at an equation on the single
quantity $w_{\varepsilon}(y)$,

$-\sigma\varepsilon
w_{\varepsilon}^{\prime}-\varepsilon^{2}w_{\varepsilon}^{\prime\prime}={\widetilde{Q}}(w_{\varepsilon})-A,$

with the conditions at infinity

$w_{\varepsilon}(-\infty)=w_{-}<\ln\mu,\qquad
w_{\varepsilon}(+\infty)=w_{+}>\ln\mu,\qquad
A={\widetilde{Q}}(w_{-})={\widetilde{Q}}(w_{+}).$

This is just a Fisher/KPP monostable equation with $w_{+}$ the unstable state
and we know from the general theory [13] that there is a traveling wave with
minimal speed $\sigma_{*}$ which is characterized by the property of a double
root for the polynomial

$-\sigma\varepsilon\lambda-\varepsilon^{2}\lambda^{2}={\widetilde{Q}}^{\prime}(w_{+}),$

that is $\sigma_{*}=2\sqrt{{\widetilde{Q}}^{\prime}(w_{+})}$. In our analysis,
this value $\sigma_{*}$ also appears in the limit of Equation (23), that is
the Eikonal equation

$\partial_{t}\varphi=|\partial_{x}\varphi|^{2}+v-\mu,$

which for the traveling wave problem generates a solution $\varphi(x-\sigma
t)$ with

$\sigma\varphi^{\prime}(y)=|\varphi^{\prime}(y)|^{2}+v_{\pm}(y)-\mu,$

where $v_{\pm}(y)=v_{-}$ for $y<0$ and $v_{\pm}(y)=v_{+}$ for $y>0$. The
limiting minimal speed traveling wave solution is

$\varphi(y)=\begin{cases}p_{-}y<0\qquad&\text{for}\quad y<0,\\\
p_{+}y<0\qquad&\text{for}\quad y>0,\end{cases}$

and $p_{+}$ is the double root of the polynomial
$-\sigma_{*}\lambda-2\lambda^{2}={\widetilde{Q}}^{\prime}(w_{+})=v_{+}-\mu$.
This approach based on the concentration as a Dirac measure of
$u_{\varepsilon}$ differs (but is restricted to one dimension) from the
general front propagation theory in [1, 4] based on the quantity
$v_{\varepsilon}$.

##  Appendix B A Sobolev estimate

We may use the bound (28) to obtain Sobolev regularity on $v_{\varepsilon}$ by
controlling

$\sup_{|h|\leq 1}\int_{0}^{T}\\!\\!\int_{|x|\leq
R}\frac{|v_{\varepsilon}(x+h)-v_{\varepsilon}(x)|}{|h|^{\theta}}\,dx\,dt,$
(30)

for some appropriate value of $\theta$.

This requires to be a bit more precise on the set where the initial data
$v_{\varepsilon}$ crosses $\mu$. Specifically, we assume that there exists
some constants $C>0$ and $\kappa>0$ such that for any $\delta>0$

$|\\{x,\;|v_{\varepsilon}^{0}(x)-\mu|\leq\delta\\}|\leq C\,\delta^{\kappa}.$
(31)

Observe that when $|h|\leq\varepsilon^{1/(1-\theta)}$ then by the Lipschitz
bound on $v_{\varepsilon}$ (which follows immediately from that on
$w_{\varepsilon}$), then

$\int_{0}^{T}\\!\\!\int_{|x|\leq
R}\frac{|v_{\varepsilon}(x+h)-v_{\varepsilon}(x)|}{|h|^{\theta}}\,dx\,dt\leq
C_{T,R}\,\|\partial_{x}v_{\varepsilon}\|_{L^{\infty}}\,|h|^{1-\theta}\ \leq
C_{R},$

so that we can limit ourselves to $h\geq\varepsilon^{1/(1-\theta)}$.

For some $\alpha>0$ which we later relate to $\theta$ and some constant $C$,
denote

$\begin{split}&\Omega_{-}=\\{(t,x)\in[0,\ T]\times
B(0,R),\;w^{0}_{-,\varepsilon}-|h|^{\alpha}\leq
w_{\varepsilon}(t,x)\leq\ln\mu-|h|^{\alpha/2}\\},\\\
&\Omega_{+}=\\{(t,x)\in[0,\ T]\times
B(0,R),\;w_{\varepsilon}(t,x)\geq\ln\mu+|h|^{\alpha/2}\\},\\\
&\Omega_{0}=\\{(t,x)\in[0,\ T]\times B(0,R),\;\ln\mu-|h|^{\alpha/2}\leq
w_{\varepsilon}(t,x)\leq\ln\mu+|h|^{\alpha/2}\ \text{or}\
w_{\varepsilon}(t,x)\leq w^{0}_{-,\varepsilon}-|h|^{\alpha}\\}.\end{split}$

Observe that when $(t,x)\in\Omega_{0}$ then

$\begin{split}&\sup_{\varepsilon^{1/(1-\theta)}\leq|h|\leq
1}\int_{(t,x)\in\Omega_{0}\ \text{or}\
(t,x+h)\in\Omega_{0}}\frac{|v_{\varepsilon}(t,x+h)-v_{\varepsilon}(t,x)|}{|h|^{\theta}}\,dx\,dt\\\
&\qquad\leq 2\,\sup_{\varepsilon^{1/(1-\theta)}\leq|h|\leq
1}\frac{\|v_{\varepsilon}\|}{|h|^{\theta}}\,\int_{(t,x)\in\Omega_{0}\
\text{or}\ (t,x+h)\in\Omega_{0}}dxdt\\\ \end{split}$

We note that if $w_{\varepsilon}(t,x)\leq
w^{0}_{-,\varepsilon}(x)-|h|^{\alpha}$ then

$|{\widetilde{Q}}(w_{\varepsilon})-{\widetilde{Q}}(w_{\varepsilon}^{0})|\geq|h|^{\alpha}/C.$

Similarly if $\ln\mu-|h|^{\alpha/2}\leq
w_{\varepsilon}(t,x)\leq\ln\mu+|h|^{\alpha/2}$ but
$w_{\varepsilon}^{0}(x)<\ln\mu-2\,|h|^{\alpha/2}$ or
$w_{\varepsilon}^{0}(x)>\ln\mu+2|h|^{\alpha/2}$, then we have again

$|{\widetilde{Q}}(w_{\varepsilon})-{\widetilde{Q}}(w_{\varepsilon}^{0})|\geq|h|^{\alpha}/C,$

as ${\widetilde{Q}}(w)$ has a minimum at $w=\ln\mu$ but is strictly convex.

Hence by (31)

$\begin{split}|\Omega_{0}|&\leq\frac{C}{|h|^{\alpha}}\,\int_{0}^{T}\\!\\!\int_{B(0,R)}|{\widetilde{Q}}(w_{\varepsilon})-{\widetilde{Q}}(w_{\varepsilon}^{0})|\,dx\,dt+\\{x,\;|w_{\varepsilon}^{0}(x)-\ln\mu|\leq
2\,|h|^{\alpha/2}\\}\\\
&\leq\frac{C}{|h|^{\alpha}}\,\int_{0}^{T}\\!\\!\int_{B(0,R)}|{\widetilde{Q}}(w_{\varepsilon})-{\widetilde{Q}}(w_{\varepsilon}^{0})|\,dx\,dt+C\,|h|^{\kappa\,\alpha/2}.\end{split}$

We therefore obtain that

$\begin{split}&\sup_{\varepsilon^{1/(1-\theta)}\leq|h|\leq
1}\int_{(t,x)\in\Omega_{0}\ \text{or}\
(t,x+h)\in\Omega_{0}}\frac{|v_{\varepsilon}(t,x+h)-v_{\varepsilon}(t,x)|}{|h|^{\theta}}\,dx\,dt\\\
&\qquad\leq\sup_{\varepsilon^{1/(1-\theta)}\leq|h|\leq
1}\frac{C}{|h|^{\theta}}\,\frac{1}{|h|^{\alpha}}\,\int_{0}^{T}\\!\\!\int_{B(0,R)}|{\widetilde{Q}}(w_{\varepsilon})-{\widetilde{Q}}(w_{\varepsilon}^{0})|\,dx\,dt+C\,\frac{|h|^{\kappa\,\alpha/2}}{|h|^{\theta}}\leq
C_{T,R},\end{split}$

by Lemma 3 and provided that $\kappa\,\alpha/2\geq\theta$ leading us to take
$\alpha=2\,\theta/\kappa$ and
$(\alpha+\theta)/(1-\theta)=(1+2/\kappa)\,\theta/(1-\theta)\leq 1$. It is
always possible to satisfy these inequalities provided that
$\theta/(1-\theta)\leq\frac{1}{1+2/\kappa}$.

We can consequently exclude the case where $(t,x)\in\Omega_{0}\ \text{or}\
(t,x+h)\in\Omega_{0}$ when bounding (30).

The $BV$ bound (28) also directly controls the case where $(t,x)\in\Omega_{-}\
\text{and}\ (t,x+h)\in\Omega_{+}$ (or vice-versa). Indeed in that case, we
necessarily have that $w_{+,\varepsilon}^{0}\geq
w_{-,\varepsilon}^{0}+2\,|h|^{\alpha/2}$ and
$\partial_{w}\Phi(x,w_{\varepsilon})={\widetilde{Q}}(w_{\varepsilon}^{0})-{\widetilde{Q}}(w_{\varepsilon})$
has a sign between $w_{-,\varepsilon}$ and $w_{+,\varepsilon}$ so that there
exists a constant $C$ s.t. (again at least locally)

$|\Phi(x,w^{0}_{-,\varepsilon})-\Phi(x,w^{0}_{+,\varepsilon})|\geq\frac{|h|^{3\,\alpha/2}}{C}.$

By our definitions of $\Omega_{-}$ and $\Omega_{+}$ and taking $C$ large
enough, this implies that

$|\Phi(x,w_{\varepsilon}(t,x+h))-\Phi(x,w_{\varepsilon}(t,x))|\geq\frac{|h|^{3\,\alpha/2}}{C}\quad\text{if}\
(t,x)\in\Omega_{-}\ \text{and}\ (t,x+h)\in\Omega_{+}.$

Therefore

$\begin{split}&\int_{(t,x)\in\Omega_{-}\ \text{and}\
(t,x+h)\in\Omega_{+}}\frac{|v_{\varepsilon}(t,x+h)-v_{\varepsilon}(t,x)|}{|h|^{\theta}}\,dx\,dt\leq
2\,\frac{\|v_{\varepsilon}\|}{|h|^{\theta}}\,\int_{(t,x)\in\Omega_{-}\
\text{and}\ (t,x+h)\in\Omega_{+}}dx\\\
&\qquad\leq\frac{C}{|h|^{\theta+3\alpha/2}}\,\int_{0}^{T}\\!\\!\int_{B(0,R)}|\Phi(x,w_{\varepsilon}(t,x+h))-\Phi(x,w_{\varepsilon}(t,x))|\,dx\,dt\leq
C_{T,R}\,|h|^{1-\theta-3\alpha/2},\end{split}$

by (28). This is bounded as long as $\theta+3\alpha/2=\theta\,(1+3/\kappa)\leq
1$.

We are finally able to focus on the case where for example both
$(t,x)\in\Omega_{-}$ and $(t,x+h)\in\Omega_{-}$. Note again that
$\partial_{w}\Phi(x,w_{\varepsilon})={\widetilde{Q}}(w_{\varepsilon}^{0})-{\widetilde{Q}}(w_{\varepsilon})$
vanishes once with a change of sign at $w_{\varepsilon}=w_{-,\varepsilon}$ for
$(t,x)\in\Omega_{-}$. Therefore $w\to\Phi(x,w)$ is injective on
$w\in[w^{0}_{-,\varepsilon},\;\ln\mu-|h|^{\alpha/2}]$ for some constant $C$
and

$|w_{1}-w_{2}|^{3}\leq C\,|\Phi(x,w_{1})-\Phi(x,w_{2})|,\quad
w_{1},\,w_{2}\in[w^{0}_{-,\varepsilon},\;\ln\mu-|h|^{\alpha/2}].$

Since $w_{\varepsilon}\geq w^{0}_{-,\varepsilon}-|h|^{\alpha}$ on
$\Omega_{-}$, this implies that for both $(t,x)\in\Omega_{-}$ and
$(t,x+h)\in\Omega_{-}$, we have

$|w_{\varepsilon}(t,x)-w_{\varepsilon}(t,x+h)|^{3}\leq
C\,|\Phi(x,w_{\varepsilon}(t,x))-\Phi(x,w_{\varepsilon}(t,x+h))|+C\,\varepsilon^{\alpha}.$

By a straightforward Hölder inequality, we get

$\begin{split}&\int_{(t,x)\in\Omega_{-}\ \text{and}\
(t,x+h)\in\Omega_{+}}\frac{|v_{\varepsilon}(t,x+h)-v_{\varepsilon}(t,x)|}{|h|^{\theta}}\,dx\,dt\\\
&\quad\leq C_{T,R}\,\left(\int_{(t,x)\in\Omega_{-}\ \text{and}\
(t,x+h)\in\Omega_{-}}\frac{|w_{\varepsilon}(t,x+h)-w_{\varepsilon}(t,x)|^{3}}{|h|^{3\theta}}\,dx\,dt\right)^{1/2}\\\
&\quad\leq C_{T,R}\,\left(\int_{(t,x)\in\Omega_{-}\ \text{and}\
(t,x+h)\in\Omega_{-}}\frac{|\Phi(x,w_{\varepsilon}(t,x))-\Phi(x,w_{\varepsilon}(t,x+h))|+|h|^{\alpha}}{|h|^{3\theta}}\,dx\,dt\right)^{1/2}\leq
C_{T,R},\end{split}$

by (28) again, provided that $3\,\theta\leq 1$ and $\alpha\geq 3\,\theta$.
Since we chose $\alpha=2\,\theta/\kappa$, this last inequality holds provided
that $\kappa\leq 2/3$, which we can always impose.

To summarize, we have obtained the desired bound (30) provided that
$\theta\leq 1/3$ and $\theta\leq 1/(1+3/\kappa)$ together with
$\theta/(1-\theta)\leq 1/(1+2/\kappa)$.

## References

  * [1] G. Barles, L. C. Evans, and P. E. Souganidis.  Wavefront propagation for reaction diffusion systems of PDE.  Duke Math. J., 61(3):835–858, 1990.
  * [2] Guy Barles.  Solutions de viscosité des équations de Hamilton-Jacobi, volume 17 of Mathématiques & Applications (Berlin).  Springer-Verlag, Paris, 1994.
  * [3] Guy Barles, Sepideh Mirrahimi, and Benoît Perthame.  Concentration in Lotka-Volterra parabolic or integral equations: a general convergence result.  Methods Appl. Anal., 16(3):321–340, 2009.
  * [4] Guy Barles and Panagiotis E. Souganidis.  A remark on the asymptotic behavior of the solution of the KPP equation.  C. R. Acad. Sci. Paris Sér. I Math., 319(7):679–684, 1994.
  * [5] K. Böttger, H. Hatzikirou, A. Chauviere, and A. Deutsch.  Investigation of the migration/proliferation dichotomy and its impact on avascular glioma invasion.  Math. Model. Nat. Phenom., 7(1):105–135, 2012.
  * [6] Nicolas Champagnat and Pierre-Emmanuel Jabin.  The evolutionary limit for models of populations interacting competitively via several resources.  J. Differential Equations, 251(1):176–195, 2011.
  * [7] M. G. Crandall, L. C. Evans, and P.-L. Lions.  Some properties of viscosity solutions of Hamilton-Jacobi equations.  Trans. Amer. Math. Soc., 282(2):487–502, 1984.
  * [8] Michael G. Crandall and Pierre-Louis Lions.  Viscosity solutions of Hamilton-Jacobi equations.  Trans. Amer. Math. Soc., 277(1):1–42, 1983.
  * [9] O. Diekmann, P.-E. Jabin, S. Mischler, and B. Perthame.  The dynamics of adaptation: an illuminating example and a Hamilton-Jacobi approach.  Theor. Popul. Biol., 67(4):257–271, 2005.
  * [10] W. H. Fleming and P. E. Souganidis.  PDE-viscosity solution approach to some problems of large deviations.  Annali della Scuola Normale Superiore di Pisa-Classe di Scienze, 13(2):171–192, 1986.
  * [11] I. Golding, Y. Kozlovsky, I. Cohen, and E. Ben-Jacob.  Studies of bacterial branching growth using reaction-diffusion models for colonial development.  Physica A, 260:510–554, 1998.
  * [12] Alexandra Köthe, Anna Marciniak-Czochra, and Izumi Takagi.  Hysteresis-driven pattern formation in reaction-diffusion-ODE systems.  Discrete Contin. Dyn. Syst., 40(6):3595–3627, 2020.
  * [13] King-Yeung Lam and Yuan Lou.  Reaction-Diffusion Equations: Theory and Applications.  Lecture Notes on Mathematical Modelling in the Life Sciences. Springer International Publishing, To appear.
  * [14] Alexander Lorz, Sepideh Mirrahimi, and Benoît Perthame.  Dirac mass dynamics in multidimensional nonlocal parabolic equations.  Comm. Partial Differential Equations, 36(6):1071–1098, 2011.
  * [15] M. Mimura, H. Sakagushi, and M. Matsushita.  Reaction-diffusion modelling of bacterial colony patterns.  Physica A, 282:283–303, 2000.
  * [16] J. D. Murray.  Mathematical biology. II, volume 18 of Interdisciplinary Applied Mathematics.  Springer-Verlag, New York, third edition, 2003.  Spatial models and biomedical applications.
  * [17] Benoît Perthame and Guy Barles.  Dirac concentrations in Lotka-Volterra parabolic PDEs.  Indiana Univ. Math. J., 57(7):3275–3301, 2008.
  * [18] Christina Surulescu and Michael Winkler.  Does indirectness of signal production reduce the explosion-supporting potential in chemotaxis-haptotaxis systems? Global classical solvability in a class of models for cancer invasion (and more).  European J. Appl. Math., 32(4):618–651, 2021.
  * [19] Kate Fang Zhang and Xiao-Qiang Zhao.  Asymptotic behaviour of a reaction-diffusion model with a quiescent stage.  Proc. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci., 463(2080):1029–1043, 2007.


# Self-Supervised Adversarial Imitation Learning

1st Juarez Monteiro1 0000-0002-8831-5343 Pontifícia Universidade Católica do
RS
Porto Alegre, Brazil
<EMAIL_ADDRESS>2nd Nathan Gavenski1 3rd Felipe Meneguzzi
0000-0003-0578-3086 King’s College London
London, England
<EMAIL_ADDRESS>0000-0003-3549-6168 University of
Aberdeen
Pontifícia Universidade Católica do RS
Aberdeen, Scotland
<EMAIL_ADDRESS>4th Rodrigo C. Barros  0000-0002-0782-9482
Pontifícia Universidade Católica do RS
Porto Alegre, Brazil
<EMAIL_ADDRESS>
###### Abstract

Behavioural cloning is an imitation learning technique that teaches an agent
how to behave via expert demonstrations. Recent approaches use self-
supervision of fully-observable unlabelled snapshots of the states to decode
state pairs into actions. However, the iterative learning scheme employed by
these techniques is prone to get trapped into bad local minima. Previous work
uses goal-aware strategies to solve this issue. However, this requires manual
intervention to verify whether an agent has reached its goal. We address this
limitation by incorporating a discriminator into the original framework,
offering two key advantages and directly solving a learning problem previous
work had. First, it disposes of the manual intervention requirement. Second,
it helps in learning by guiding function approximation based on the state
transition of the expert’s trajectories. Third, the discriminator solves a
learning issue commonly present in the policy model, which is to sometimes
perform a ‘no action’ within the environment until the agent finally halts.

###### Index Terms:

Imitation Learning, Adversarial Learning, Learning from Observation, Self-
Supervised Learning

11footnotetext: These authors contributed equally to the work.

##  I Introduction

Learning by observing is an intrinsic human ability that we have been able to
rely on since childhood. We can learn tasks by watching a video teaching us
how to cook or how to play a specific video game. Learning from demonstrations
allows humans to learn tasks from proficient sources and apply the newly-
acquired knowledge in different domains, similar tasks, or after adapting it
to their own reality, i.e., different body sizes and proportions.
Occasionally, learning a task by observing a specialist can be difficult. We
can watch tennis players performing their best moves, but it is not a simple
task for us to break down their actions into straightforward instructions to
learn them properly.

In Machine Learning (ML), we refer to the technique of learning from a teacher
as Imitation Learning (IL). It consists of an agent learning from the actions
of a known teacher in order to solve a given task [1]. The learning agent must
be able to achieve the goal or conclude the task which it was trained for.
Recent approaches try to approximate the human learning experience by
exploring a strategy where the agent needs no explicit label of the actions
performed by the teacher to imitate them. This strategy of learning without
explicitly receiving the teacher’s actions (labels) or learning by observing
is called in the literature Learning from Observation (LfO) [2, 3, 4, 5].

LfO emerges with improvements in efficiency [6] and generalisation [4],
overcoming the need for fine-grained information found in annotated
trajectories or in complex reward functions. Since they are more effective,
LfO methods require fewer teacher snapshots, which helps mitigate the lack-of-
labels problem in the available training data. The fact that we can use LfO
with smaller amounts of (unsupervised) data gives us the opportunity to
explore problems where data is scarce or costly to collect, e.g., autonomous
vehicles [7]. The existing methods for LfO often rely on learning how to map
the state-action transition in a self-supervised manner. This, in turn,
requires access to the test environment and multiple deliberations from the
agent, so we properly map the state-action transitions.

LfO strategies are frequently benchmarked by measuring performance and
efficiency following a specific formalisation [3, 4, 5]. One can also evaluate
imitation by comparing the trajectories that both the agent and the teacher
have taken in order to solve a given task. Both perspectives have their
issues. With the traditional measures, an agent can diverge from the teacher
and still achieve the same reward, even though it is performing in a
completely different way from what was expected. Indeed, sparse rewards in an
environment make identifying proficient behaviour non-trivial. By only
comparing rewards and not trajectories or intent, an agent might reach the
same reward as its proficient counterpart, though perhaps missing part of the
desired behaviour that it should account for. We can derive a similar example
for the second perspective. Suppose the agent is acting within a maze
environment, where the agent and the teacher follow practically the same
trajectory, but in the end, the agent does not achieve a final state,
resulting in a totally different reward.

##  II Related Work

Behavioural Cloning (BC) [8] is one of the most straightforward techniques for
Imitation Learning [9]. BC uses teacher trajectories containing the state
$s_{t}$ and the action $a_{t}$ of a given task at time $t$ to create a policy
$\pi=P(a\mid s_{t})$. BC presents consistent results in the aspect of episodic
rewards by training a policy in a supervised manner. However, it comes with a
high cost for wide state-space scenarios, requiring sufficient trajectories to
make the policy generalise for unknown states.

To solve this issue, recent approaches in IL [3, 10, 4, 5] employ strategies
that do not require labelled data provided by a teacher. Torabi et al. [3]
designed a LfO model-based strategy called Behavioural Cloning from
Observation (BCO), which learns to imitate using a self-supervised strategy
that does not require a teacher to provide annotated data. BCO starts by
learning the state-action transition to build a predictor capable of guessing
what action occurred in a given pair of states $S_{t}$ and $S_{t+1}$. It then
uses such a model to label the teacher trajectories. Next, all automatically-
labelled data is used to train a policy in a supervised fashion. Even though
the authors present better results than supervised methods that use the
teacher’s original labels, the approach lacks a proper exploration technique,
leading to a situation in which it repeatedly finds itself stuck in endless
states.

Imitating Unknown Policies via Exploration (IUPE) [5] is an approach that uses
sampling and exploration mechanisms to solve the efficiency issue. After each
interaction, the inverse dynamics model $\mathcal{M}$ learns the most likely
action given a state-transition in a supervised fashion, and then it trains a
policy with the pseudo-labelled ($\mathcal{M}$’s outputs) as teacher’s
actions. IUPE weighs the random and policy samples to create a dataset similar
to the teacher’s transitions. Finally, its exploration mechanism uses the
softmax distribution over its outputs to perform weighed sampling over all
actions from both models. By avoiding the usage of its maximum a posteriori
(MAP) estimation, it creates a stochastic policy that dynamically changes its
exploration ratio according to each model’s output. The downside of using IUPE
is its need for hand-crafted goal-aware functions, which require prior domain
knowledge, to retrieve intermediate samples capable of approximating the
initial random state-action pairs to the proficient ones.

Ho and Ermon [11] propose Generative Adversarial Imitation Learning (GAIL), an
approach that uses adversarial training to solve that same problem. GAIL
requires a smaller teacher dataset compared to other approaches. Nevertheless,
the method needs extensive interactions with the environment. Torabi et al.
[10] find inspiration on GAIL to design a Generative Adversarial Network (GAN)
[12] named Generative Adversarial Imitation from Observation (GAIfO). GAIfO
tries to learn a policy by creating a mechanism to distinguish if the source
of the data is from a teacher or provided by the model. By training the model
to understand what is the next state $S_{t+1}$ from a given state-action pair
$S_{t}$ and $a$, GAIfO produces a policy that has similar behaviour to a
teacher. Although GAIfO has yielded significantly better results than GAIL, by
reducing the number of samples necessary to train a policy, it falls in the
same issue as GAIL, where the number of interactions with the environment is a
bottleneck.

##  III Problem Formulation

We formalise Imitation Learning assuming an environment defined by a Markov
Decision Process (MDP), which is represented by a five-tuple
$M=\\{S,A,T,r,\gamma\\}$ [13, Ch 3.], where $S$ is the state-space, $A$ is the
action space, $T$ is the transition model, $r$ is the immediate reward
function, and $\gamma$ is the discount factor. Solving an MDP yields a
stochastic policy $\pi(a|s)$ with a probability distribution over actions for
an agent in state $s$ that needs to take a given action $a$. _Imitation from
observation_ (IfO) [3] aims to learn the inverse dynamics of the agent,
$\mathcal{M}_{a}^{s_{t},s_{t+1}}=P(a|s_{t},s_{t+1})$, i.e., the probability
distribution of each action $a$ when the agent transitions from state $s_{t}$
to $s_{t+1}$. While we assume the environment is an MDP, in imitation learning
the agent has no access to an explicit reward signal. The actions performed by
the teacher are unknown, so we want to find an imitation policy from a set of
state-only demonstrations of the teacher
$\mathcal{T}=\\{\zeta_{1},\zeta_{2},\ldots,\zeta_{N}\\}$, where $\zeta$ is a
state-only trajectory $\\{s_{0},s_{1},\ldots,s_{N}\\}$.

A classic self-supervised IL approach in the LfO area focuses on using two
models to learn to imitate. The models are the Inverse Dynamic Model
($\mathcal{M}$) and the policy model $\pi_{\theta}$. The Inverse Dynamic Model
is responsible for learning to predict $P(a\mid s_{t},s_{t+1})$, which action
caused the transition between a given pair of states ($s_{t},s_{t+1}$). After
training the ($\mathcal{M}$) model, it is now possible to predict the most
possible action taken by the teachers in their collected trajectories that
will be used in the policy $\pi_{\theta}$ training [3, 4]. Using the pseudo-
labels $\hat{a}$ to replace the potential actions that the teachers might have
taken builds a natural exploration for LfO methods which helps the agent
acquire the capability of generalisation by accessing different states from
the original trajectory and acquiring knowledge about state transitions [14].
A learner can iterate over this self-supervised pipeline to reduce the error
coming from both $\mathcal{M}$ and $\pi_{\theta}$ models.

LfO strategies offer a more data-efficient approach to imitation learning,
since they can learn similar policies to other methods that require labelled
data. Specifically, we consider policies to be similar not only in terms of
similar returns, but also policies that generate similar trajectories in the
state space. This is in contrast to the literature in imitation learning,
which often measures similarities by looking at the returns alone for any
given task [3, 4, 5]. However, just because an agent achieves similar returns
in a given task, this does not mean the agent is actually imitating observed
behaviour if the policies generate radically different trajectories.

##  IV Self-supervised Adversarial Imitation Learning

Figure 1: Self-supervised Adversarial Imitation Learning (SAIL) training
pipeline proposed in this paper. All models are initialised with random
weights, and the agent interacts with the environments to collect data, so
$\mathcal{M}$ learns state transitions. Afterwards, given state-only
demonstrations, the $\mathcal{M}$ creates self-supervised labels to use with
its policy in a supervised manner. The updated policy then interacts with the
environment and collects new samples to be used in a new iteration by the
state transition model. Unlike previous work, SAIL uses a discriminator to
classify whether samples should be appended to the initial samples and a
generative model to update its policy to act more like its observation
samples. SAIL repeats steps (ii) from (viii) for a specific amount of epochs
or until both models stop improving.

In this paper, we create the Self-supervised Adversarial Imitation Learning
(SAIL), an IL method that interleaves self-supervised and adversarial learning
to create a policy based on observation of a teacher without any use of
labelled snapshots. SAIL uses an exploration mechanism based on previous work
[5, 15] to explore when it is unsure of the teacher’s actions, and a
discriminator model to classify whether the policy trajectory is similar to a
teachers’ one. SAIL comprises four different models:  (i) $\mathcal{M}$, which
predicts an action given a state transition $P(\hat{a}\mid s_{t},s_{t+1})$;
(ii) a policy model $\pi_{\theta}$ that uses the self-supervised labels
$\hat{a}$ to mimic a teacher, given a state $P(\hat{a}\mid s_{t})$;  (iii) a
generative model $\mathcal{G}$ conditioned by prediction from $\pi_{\theta}$
and the current state $P(s_{t+1}\mid s_{t},\tilde{a})$; and  (iv) a
discriminator model $\mathcal{D}$ to discriminate $\pi_{e}$ and
$\pi_{\theta}$, creating better samples for $\mathcal{M}$ and updating weights
$\theta$ when the policy is not similar to its proficient counterpart.

Algorithm 1 SAIL

1:Initialise model $\mathcal{M}_{\theta}$ as a random approximator

2:Initialise policy $\pi_{\theta}$ with random weights

3:Initialise generative model $\mathcal{G}$ with random weights

4:Initialise discriminator $\mathcal{D}$ with random weights

5:Generate $I^{s}$ using $\pi_{\theta}$ $i\leftarrow 1$ to epochs

6:Improve $\mathcal{M}_{\theta}$ by $\Call{supervised}{I^{s}}$

7:Use $\mathcal{M}_{\theta}$ with $\mathcal{T}^{e}$ to predict $\hat{A}$

8:Improve $\pi_{\theta}$ and $\mathcal{G}$ by
$\Call{behaviouralCloning}{\mathcal{T}^{e},\hat{A}}$

9:Use $\pi_{\theta}$ to solve environments $E$

10:Append samples $I^{pos}\leftarrow(s_{t},\tilde{a}_{t},s_{t+1})$

11:Append $I^{s}\leftarrow\forall i\in
I^{pos}\;|\;\mathcal{D}(\mathcal{G}(I^{pos}_{i}))=1$

Algorithm 1 provides an overview of the learning procedure of SAIL. We first
initialise all four models with random weights (Line 1-4). With $\pi_{\theta}$
randomly initialised, we can use this model as a random agent and collect all
samples required (Line 5). Considering that IL methods have no knowledge about
optimal or the teacher’s action distributions, these random samples help
$\mathcal{M}$ to classify each state transition without any biases since the
randomly generated ones are equally distributed. Once all samples consisting
of $(s_{t},a,s_{t+1})$ tuples are appended to $I^{s}$, SAIL trains its inverse
dynamic model in a supervised manner (Function $\Call{supervised}{}$ in Line
6). With its updated weights $\theta$, $\mathcal{M}$ predicts all pseudo-
labels $\hat{A}$ to all teacher’s transitions in $\mathcal{T}^{e}$ (Line 7).
As the model’s weights might not be optimal in each iteration, SAIL implements
an exploration mechanism that allows for $\mathcal{M}_{\theta}$ to deviate
from its MAP by sampling from its predictions using softmax distributions of
the same output as weights. This mechanism allows SAIL to dynamically explore
other labels when unsure (with more uniform distributed MAP values) and
exploit once its MAPS values are farther apart. After creating all self-
supervised labels, we train $\pi_{\theta}$ using a behavioural cloning
approach (Function $\Call{behaviouralCloning}{}$ in Line 8). Unlike other
behavioural cloning approaches, SAIL also uses a generative model
$\mathcal{G}$ to predict its next state, conditioned by the action predicted
by $\pi_{\theta}$. Hence, it also updates $\mathcal{G}$ during
$\pi_{\theta}$’s training (we further explain the learning process of the
generative model in Section IV-B). Afterwards, SAIL uses $\pi_{\theta}$ to
create new samples that might help $\mathcal{M}$ better approximate the
unknown ground-truth actions from $\mathcal{T}^{e}$ (Line 9). Finally, SAIL
appends to $I^{s}$ all samples that $\mathcal{D}$ could not differentiate
between $\mathcal{T}^{e}$ and $\mathcal{G}(I^{pos})$ (Line 11). We aim to
discard trajectories that could result in $\mathcal{M}$ getting stuck in bad
local minima and update $\pi_{\theta}$ to correct some behaviours that
$\mathcal{D}$ uses to differentiate between teacher and student. We better
explain how $\mathcal{D}$ benefits SAIL in Section IV-A.

###  IV-A Goal-aware function

Developing a goal-aware function can be a complex task. Environments in the
agent literature have different meanings for what a goal is. Environments
typically have one of two different types of tasks:  1. maintenance; or  2.
achievement [16, Chapter 2].  Environments with an achievement task define a
clear end goal, such as MountainCar [17] – where an agent has to reach a flag
located on top of a mountain with an accumulated reward $\geqslant-110$. Since
agents in an IL context have no access to the reward signal, we must consider
the number of steps an agent performs before reaching its objective. As
environments grow in complexity, such a function will become harder to encode.
By contrast, maintenance task environments usually define a set of states that
an agent should not reach. For example, Ant [18], where an agent walks as far
as possible without reaching angles that it classifies as ‘falling’. While
others, such as CartPole [19], define a stopping criterion, e.g., when its
pole reaches a certain angle, and an optimal threshold, e.g., maintain its
task for 195 consecutive steps. Thus, encoding a goal-aware function creates a
degree of unwanted complexity in a learning algorithm.

By only using samples that reach a goal, SAIL acquires examples that have
‘some’ degree of optimality, and approximates $I$’s samples from
$\mathcal{T}^{e}$ [5]. Nevertheless, classifying whether samples are close to
$\mathcal{T}^{e}$ might be difficult. First, defining what indicates a sample
being close to a proficient teacher is hard. If we consider a stationary agent
(which SAIL is), we might discard samples that allow $\mathcal{M}$ to
accurately predict transitions due to their distance to $\mathcal{T}^{e}$
states alone. Therefore, to achieve better policies, SAIL needs a goal-aware
function that allows for $\mathcal{M}$ to deal with sub-optimal samples.

To remove the usage of hand-crafted goal-aware functions, SAIL uses a
discriminator $\mathcal{D}$ to discriminate between $\pi_{\theta}$ and
$\pi_{e}$. By doing so, SAIL eliminates all human intervention and gains a
non-greedy sampling mechanism by using a model to classify which agent created
a trajectory. Moreover, since $\mathcal{D}$ starts with random weights (Line
4), it allows samples that did not reach a ‘goal’, i.e., sub-optimal samples,
to be appended to $I^{s}$. However, considering that SAIL works under LfO
constraints (not having access to $\pi_{e}$ actions), there is a need to
create a mechanism that can discriminate between teachers’ and students’
state-only trajectories. Using state-only trajectories from $\pi_{\theta}$ and
$\pi_{e}$, and an adversarial learning approach (Equation 1), SAIL allows
$\pi_{\theta}$ via $\mathcal{G}$ to be updated from its gradient flow from
$\mathcal{D}$.

$\begin{split}\min_{\mathcal{M}\cup\pi}\max_{\mathcal{D}}&SAIL(\mathcal{M},\pi,\mathcal{G},\mathcal{D})=\\\
&\mathbb{E}_{r\sim R(\pi_{\epsilon},Env)}[log(\mathcal{D}(r))]+\\\
&\mathbb{E}_{r^{\prime}\sim
R(\pi_{\theta},Env)}[log(1-\mathcal{D}(\mathcal{G}(r^{\prime},\pi_{\theta}(r^{\prime}))))]\end{split}$
(1)

Considering that SAIL uses a few samples (in the form of
$\mathcal{T}^{e}\cup\mathcal{T}^{\pi}$), it is important to avoid
$\mathcal{D}$ overfitting. For that, we record all $\pi_{\theta}$ trajectories
in a replay buffer $RB$ and only sample a few trajectories from both sets at
each iteration,
$RB_{(s_{t},\cdots,s_{t+n})}\sim\mathcal{T}^{e}\cup\mathcal{T}^{\pi}$. By only
using a sub-set from each sample pool of trajectories, SAIL avoids overfitting
$\mathcal{D}$ during first iterations (where $\pi_{\theta}$’s trajectories are
considerably different).

###  IV-B Generative model

SAIL uses a generative model in two different steps. Firstly during Function
$\Call{behavioralCloning}{}$; and afterwards, when selecting which samples
should be appended to $I^{s}$ Although SAIL adds a new model to its pipeline,
in these situations, it benefits in two folds:  (i) intrinsically encodes
environments physics in $\pi_{\theta}$; and  (ii) it updates $\pi_{\theta}$
when using $\mathcal{D}$ via its gradient flow.  Which we believe far
surpasses the overhead cost created by using a generative model.

The first benefit results from SAIL updating $\mathcal{G}$ weights using
Equation 2. Thus, when using the generative model in Line 8, SAIL allows
$\mathcal{L}_{\mathcal{G}}$ to update $\pi_{\theta}$ to create actions that
would correctly condition $\mathcal{G}$ to generate correct state transitions
and equal to those observed. Moreover, by learning how to properly decode from
$(s_{t},\tilde{a_{t}})\mapsto\hat{s_{t+1}}$, the generative model becomes a
forward dynamics model, which helps $\pi_{\theta}$ encode some of the
environments’ dynamics, e.g., physics. We hypothesise that it is also possible
only to update $\mathcal{G}$ weights when $\pi_{\theta}$ correctly predicts a
self-supervised label. However, this creates two different problems. The first
problem is that not always $\mathcal{M}$ will be correct. Thus, $\pi_{\theta}$
being correct about $\hat{a}$ might not be indicative of how accurate it is in
conditioning $\mathcal{G}$. The second problem originates from the fact that
$\mathcal{M}$ might stop predicting some actions when being stuck in local
minima [5]. In this scenario, $\mathcal{G}$ will not update for state
transitions for the action not being predicted. Therefore, $\mathcal{G}$ will
update $\pi_{\theta}$ fewer times, resulting in less exploration, since
updating $\pi_{\theta}$ weights with two different objectives can also help it
to no be stuck at local minima.

$\mathcal{L}_{\mathcal{G}}=-\frac{1}{N}\left[\sum_{i=1}^{N}s_{i+1}\cdot
log(\mathcal{G}(s_{i},\pi_{\theta}(s_{i}))\right]$ (2)

The second benefit originates from the fact that SAIL has an adversarial
training mechanism in its pipeline. Hence, $\mathcal{D}$ directly updates
$\pi_{\theta}$ weights via gradient flow when it correctly discriminates
between teacher and student (Equation 1). This behaviour is beneficial because
updating $\pi_{\theta}$ through $\mathcal{D}$ allows for the agent to have a
direct temporal signal, i.e., where it deviates from its teachers’
observations. Consequently, since SAIL maintains all original behavioural
cloning techniques, it creates agents that mimic teachers’ trajectories more
accurately while not losing performance.

Both generative update moments allow SAIL to have more precise trajectories
(further explored in Section V), as well as more trajectories that are similar
to their source (discussed in Section VI-B). We believe having trajectories
closer to the teachers is beneficial since it avoids unwanted biases due to
previous methods only using the performance metrics, which does not carry
behaviour meaning [20].

##  V Experimental Results

We tested SAIL and all baselines described in Section II with four different
environments:  (i) CartPole;  (ii) MountainCar;  (iii) Acrobot; and  (iv)
LunarLander.  We use OpenAI Gym [21] versions for all environments. Figure
LABEL:fig:envs illustrates a single frame for each of these environments,
while Section V-A gives a brief description of all environments and SAIL
neural network topology.

###  V-A Implementation and Metrics

We follow Gavenski et al. [5] implementation for our agents. Therefore,
$\pi_{\theta}$ is a Multi-Layer Perceptron (MLP) model with $2$ hidden layers
with $32$ neurons and $2$ self-attention modules after each layer.
$\mathcal{M}$ is an MLP with $2$ hidden layers with $32$ neurons, $2$ self-
attention modules and $2$ Layer Normalisation layers. $\mathcal{G}$ is an MLP
with $2$ hidden layers with $2\times(\mid s\mid+1)$ neurons, where $\mid
s\mid$ is the size of the environment state vector, and no self-attention or
normalisation layers. $\mathcal{D}$ is a Long Short Term Memory [22] with $2$
layers, $32$ neurons each, and dropout of $50\%$. The official SAIL
implementation can be found at: https://github.com/NathanGavenski/SAIL.

To measure our experiments, we will use two main metrics: Average Episodic
Reward (AER), and Performance ($\mathcal{P}$) [11] metrics. AER is the average
of all accumulated rewards for a consecutive amount of tries in each
environment. Performance is returned by calculating the average reward for
each run scaled to be within $[0,1]$, where zero is a behaviour compatible
with a random policy ($\pi_{\xi}$) reward, and one a behaviour compatible with
the teacher ($\pi_{\varepsilon}$).

$\mathbb{P}=\frac{\sum_{i=1}^{E}\frac{\pi_{\phi}(e_{i})-\pi_{\xi}(e_{i})}{\pi_{\varepsilon}(e_{i})-\pi_{\xi}(e_{i})}}{E}$
(3)

It is possible for a model to achieve scores $<0$ if it has the worst
performance than a random policy and $>1$ if the model can perform better than
its teacher. We do not use accuracy for evaluation since achieving high
accuracy in Imitation Learning tasks does not guarantee good results in
solving a task.

We now briefly describe all environments used in this work.

  * $\bullet$

CartPole-v1 is an environment where an agent moves a car sideways, applying
force to a single pole. The goal is to prevent the pole from falling over. The
space state has four dimensions:car position, car velocity, pole angle, and
pole velocity at tips. The agent receives a single reward point every time the
pole remains upright. Barto [19] describes solving CartPole as getting an
average reward of $195$ over $100$ consecutive trials.

  * $\bullet$

MountainCar-v0 environment consists of a car situated in a valley. The agent
needs to learn to leverage potential energy by driving up the opposite hill
until completing the goal. The state-space has two continuous attributes:
velocity and position and three discrete action spaces: left, neutral, and
right. A reward of $-1$ is provided for every time step until the goal
position of $0.5$ is reached. The first state starts in a random position with
no velocity. Moore [17] defines solving MountainCar as getting an average
reward of $-110$ over $100$ consecutive trials.

  * $\bullet$

Acrobot-v1, based on Sutton’s work [23], is an environment where an agent has
two joints and two links. The joint between the two links is actuated. The
state space consists of: $\\{\cos\theta_{1},\sin\theta_{1},$
$\cos\theta_{2},\sin\theta_{2},\theta_{1},\theta_{2}\\}$, and the action space
consists of the $3$ possible forces. The goal is to move the end of the lower
link up to a given height. Although Acrobot is an achievement task
environment, it does not have a specified reward threshold.

  * $\bullet$

LunarLander-v2, created by Klimov [21], is an environment where an agent needs
to land on the moon under low gravity conditions. The state space is
continuous, and the action space is discrete. There are four actions: do
nothing, move left, right, and reduce the falling velocity. All actions have a
reward of $-1$, except for do nothing state, which is $-0.3$. A positive value
is returned when the agent moves in the right direction (always at $0,0$
coordinates). LunarLander-v2 is solved when the agent receives a reward of
$200$ over $100$ constitutive trials.

TABLE I: SAIL and baselines results for all environments. Algorithm | Metric | CartPole | MountainCar | Acrobot | LunarLander
---|---|---|---|---|---
Random | AER | $21,92\pm$ | $-200\pm 0$ | $-499.36\pm$ | $-170.47\pm$
$\mathcal{P}$ | $0$ | $0$ | $0$ | $0$
Expert | AER | $500\pm 0$ | $-98.03\pm 8.17$ | $-74.85\pm 8.61$ | $256.79\pm 21.38$
$\mathcal{P}$ | $1$ | $1$ | $1$ | $1$
BC | AER | $218.53\pm 160.71$ | $-102.06\pm 4.23$ | $-80.21\pm 3.61$ | $63.05\pm 79.50$
$\mathcal{P}$ | $0.37$ | $0.97$ | $0.99$ | $0.63$
GAIL | AER | $302.03\pm 158.96$ | $-200\pm 0$ | $-274.27\pm 116.85$ | $120.21\pm 28.03$
$\mathcal{P}$ | $0.41$ | $0$ | $0.54$ | $0.66$
GAIfO | AER | $\mathbf{500\pm 0}$ | $-200\pm 0$ | $-128.20\pm 15.88$ | $\mathbf{200\pm 29.95}$
$\mathcal{P}$ | $\mathbf{1}$ | $0$ | $0.85$ | $\mathbf{0.86}$
IUPE | AER | $\mathbf{500\pm 0}$ | $-166.97\pm 18.34$ | $\mathbf{-75.65\pm 12.85}$ | $-81.34\pm 74.5$
$\mathcal{P}$ | $\mathbf{1}$ | $0.32$ | $\mathbf{1}$ | $0.21$
SAIL | AER | $\mathbf{500\pm 0}$ | $\mathbf{-99.35\pm 1.84}$ | $-78.84\pm 0.41$ | $183.62\pm 5.63$
$\mathcal{P}$ | $\mathbf{1}$ | $\mathbf{0.99}$ | $0.99$ | $0.83$

###  V-B Results

Table I shows results for all baselines and SAIL in four different
environments. SAIL’s performance is closer to the teacher’s reward in almost
all environments (CartPole, MountainCar and Acrobot), and it performs worst in
the LunarLander environment, resulting in the best algorithm throughout all
environments. When comparing SAIL to other methods, we observe it yields a
lower standard deviation ($\leqslant 1$) in the first three environments, with
a higher deviation for the LunarLander environment ($5.63$). We believe these
lower deviations are due to the gradient flow from $\mathcal{D}$ into
$\pi_{\theta}$ discussed in Section IV-A, and $\mathcal{G}$ intrinsically
encoding each environment physics into $\pi_{\theta}$ without a direct signal
(Section IV-B). By properly encoding physics in its policy, SAIL achieves a
behaviour that helps its agent yield similar results to the teacher since it
has knowledge on how $s_{t+1}$ should be given $a$ and $s_{t}$. Moreover,
$\mathcal{D}$ allows $\pi_{\theta}$ to have a temporal signal in its
trajectory, helping to correct any unwanted/divergent behaviour that helps
$\mathcal{D}$ discriminate against teacher and student.

Conversely, SAIL does not achieve the best results for Acrobot and
LunarLander. For the Acrobot environment, SAIL achieves an accumulated reward
lower than IUPE ($3.19$ lower). However, its standard deviation is even lower
than behavioural cloning, which had labelled snapshots during its training. We
believe this is the case for this environment because Acrobot rewards hectic
behaviour from the agent, which IUPE deeply beneficiates given its exploration
mechanism, while $\mathcal{D}$ incentives SAIL to have more consistent
trajectories. We hypothesise that for cases where SAIL has a higher
exploration ratio, reducing the gradient from its adversarial phase would be
beneficial, avoiding it getting into an exploitative phase too soon. As for
the LunarLander environment, SAIL achieves a similar result to GAIfO, which
had $16.38$ more accumulated reward, but with $24.32$ more deviation points.
We believe this is the case for SAIL due to it learning the proficient
behaviour much more closely than the other IL counterparts. If we compare BC’s
performance, we observe that by having the finer-grained information of all
actions, the results dramatically changed – which we draw the comparison to
SAIL’s results. For the LunarLander environment, IUPE has the worst and only
negative result. We believe that such a result originates from the fact that
LunarLander optimal behaviour highly correlates to the agent and goal
initialisation, which IUPE lacks mechanisms to understand from its proficient
source.

Finally, we observe that for environments with more relation between states
(i.e., carrying momentum), such as MountainCar, SAIL performs best. While most
baselines yielded policies closer or equal to a random one, SAIL achieved a
performance $\approx 1$. During experimentation, we observe that most other
methods require the agent to be in a specific state, i.e., stopped or with
almost no movement force. However, we did not notice a similar behaviour since
SAIL receives information from its discriminator model.

##  VI Discussion

In this section, we consider the following:  (i) how SAIL learns with
different amounts of samples; and  (ii) how $\pi_{\theta}$ behaves compared to
its performance and discriminator accuracy.  The first case allows us to
understand the trade-off between having more or fewer trajectories than those
presented in Section V. Understanding this balance is essential, so SAIL
learns with as fewer samples as possible and, therefore, faster. We
investigate the second case to understand how much $\pi_{\theta}$ approximates
from $\pi_{e}$ trajectories. IL works use performance and AER as metrics but
do not consider how similar the policy is to its teacher.

###  VI-A Sample Efficiency

TABLE II: Results for SAIL with different sample sizes. Environment | Trajectories | $\mathcal{P}$ | $AER~{}(avg)$ | $AER~{}(\min)$ | $AER~{}(\max)$ | $SD$
---|---|---|---|---|---|---
CartPole | $1$ | $0.55$ | $1.55$ | $86.23$ | $457.30$ | $\pm 133.98$
$25$ | $1$ | $500$ | $500$ | $500$ | $\pm 0$
$50$ | $1$ | $500$ | $500$ | $500$ | $\pm 0$
$75$ | $1$ | $500$ | $500$ | $500$ | $\pm 0$
$100$ | $1$ | $500$ | $500$ | $500$ | $\pm 0$
MountainCar | $1$ | $0$ | $-200$ | $-200$ | $-200$ | $\pm 0$
$25$ | $0.87$ | $-109.96$ | $-114.40$ | $-103.60$ | $\pm 4.21$
$50$ | $0.96$ | $-101.78$ | $-103.70$ | $-99.11$ | $\pm 2.02$
$75$ | $0.99$ | $-99.35$ | $-102.10$ | $-97.38$ | $\pm 1.84$
$100$ | $0.98$ | $-101.31$ | $-109.90$ | $-97.87$ | $\pm 4.93$
Acrobot | $1$ | $0.98$ | $-87.47$ | $-112.20$ | $-77.75$ | $\pm 14.03$
$25$ | $0.99$ | $-77.95$ | $-79.75$ | $-76.78$ | $\pm 1.16$
$50$ | $0.99$ | $-79.33$ | $-80.33$ | $-78.23$ | $\pm 0.96$
$75$ | $0.99$ | $-78.84$ | $-79.46$ | $-78.46$ | $\pm 0.41$
$100$ | $0.99$ | $-79.36$ | $-80.72$ | $-76.43$ | $\pm 1.73$
LunarLander | $1$ | $0.31$ | $-30.13$ | $-76.95$ | $55.28$ | $\pm 52.28$
$25$ | $0.86$ | $196.50$ | $148$ | $242.20$ | $\pm 36.86$
$50$ | $0.83$ | $133.19$ | $-24.84$ | $207.30$ | $\pm 98.04$
$75$ | $0.83$ | $183.62$ | $175.20$ | $188.90$ | $\pm 5.63$
$100$ | $0.81$ | $151.26$ | $10.72$ | $204.50$ | $\pm 79.45$

We observe that SAIL inherits two different behaviours from behavioural
cloning methods. Firstly, it requires a sample size bigger than a single
trajectory to learn how to achieve a performance similar to its teacher. This
is no surprise since $1$ trajectory barely provides any information for SAIL
to learn how to encode different states and generalise significantly in each
environment [24]. The second behaviour SAIL inherits is it fails to scale
according to the number of samples due to compounding error [25]. As the
samples grow, the policy diverges less from its observed trajectories,
decreasing the agent’s performance. Therefore, behavioural cloning methods
need to find the correct number of trajectories they should use. Conversely,
as Table II shows, SAIL achieves a performance close to $1$ with $25$
episodes, only decreasing its standard deviation for each row consecutively
and increasing again with $100$ trajectories. We believe that SAIL achieves
this result due to the updates from $\mathcal{G}$ and, therefore,
$\mathcal{D}$. Using a different objective, e.g., recreating trajectories
closer to the teachers’, SAIL results in fewer episodes needed than previous
methods would because $\pi_{\theta}$ is less dependent on its LfO objective.

Nevertheless, SAIL achieves these results also due to using a smaller
discriminator model, which comes with the price of being sequential. We
hypothesise that to keep the sample size small, SAIL cannot use a larger
sequence model, such as Transformers [26], since the high number of parameters
means that the discriminator model would easily overfit on its small dataset.
We believe that increasing SAIL efficiency would require further
experimentation regarding the augmentation of observations since performing
modification of teacher samples requires prior domain knowledge so as not to
augment tuples into impossible or undesired transitions.

TABLE III: Discriminator’s accuracy ($\mathcal{D}$), Generator’s loss ($\mathcal{G}$) and Policy’s performance ($\pi_{\theta}$) for each environment. Environment |  $\mathcal{D}$’s Accuracy (%) |  $\mathcal{G}$’s Loss |  $\pi_{\theta}$’s Performance
---|---|---|---
CartPole | $49.44\pm 1.12$ | 0.0015 | 1
MountainCar | $49.76\pm 0.83$ | 0.0029 | 0.98
Acrobot | $50.13\pm 3.21$ | 0.8685 | 0.99
LunarLander | $49.08\pm 1.28$ | 0.0308 | 0.81

###  VI-B Imitation Behaviour

IL metrics usually rely only on the accumulated reward of the agent to judge
how well a policy learned to mimic its teacher’s behaviour. Most works, such
as this, use $\mathcal{P}$ as a metric, which, we believe, fails to show IL
agents’ intricacies (such as trajectory). By only using the reward signal to
evaluate these agents, we can not conclude that the agent behaves like its
teacher. There might be a proficient behaviour desired in an environment that
is not encoded in the reward function or a stochastic behaviour not apparent
in its observation. Therefore, $\mathcal{P}$ may fail to measure a divergence
in trajectory since the accumulated reward might be equal. Let us use a maze
environment as an example, where there are always two trajectories with equal
lengths, but in one path, there is a slight chance of the floor breaking. In
this case, a conservative agent might never take the trajectory with the
faulty floor. On the other hand, a more aggressive agent will not consider
this when making its way through the maze. An imitation learning agent that
learns from the conservative policy might inherit the bias of never stepping
into the floor that might break. However, the performance will not measure
this behaviour, even if the student becomes closer to the aggressive policy.

To avoid this issue, we analyse not only $\pi_{\theta}$’s performance but also
$\mathcal{D}$ accuracy and $\mathcal{G}$ error. If $\pi_{\theta}$ achieves
higher performance, while $\mathcal{D}$ has high accuracy and $\mathcal{G}$
has a lower error, it means that although $\pi_{\theta}$ correctly encoded
proficient behaviour, it has a trajectory that is not close from its teacher.
Moreover, if $\pi_{\theta}$ achieves a performance close to $1$, and
$\mathcal{D}$ has lower accuracy and $\mathcal{G}$ has a higher error, it
means that $\pi_{\theta}$ yields the same reward as its teacher, but its
generator learned how to encode next states to ‘fool’ $\mathcal{D}$, but does
not follow the observations correctly. Therefore, we are looking for a
scenario where $\pi_{\theta}$ has $\mathcal{P}\approx 1$, $\mathcal{D}$ an
accuracy $\approx 50\%$ and a lower error for its $\mathcal{G}$ during
$\Call{behaviouralClonning}{}$ (Line 8 in Algorithm 1. Table III shows all
three metrics results for all four environments used in this work.

Figure 2: Error rate for $5$ different runs in Acrobot environment.

We observe that for all environments, when $\pi_{\theta}$ yields its best
result, $\mathcal{D}$ has results equal to a random model, e.g., it is not
able to discriminate whether a trajectory comes from teacher or student.
Additionally, when we compare $\mathcal{G}$ error during its first learning
phase, where it tries to recreate the next state from a teacher’s trajectory,
it has a lower error, meaning it correctly learned how to encode state
transitions. Therefore, SAIL yields a policy consistent with proficient
rewards while keeping a consistent trajectory with its learned source. We note
that error is highly contextual to each environment state encodings. For
example, Acrobot’s states is a vector with $6$ values, which consists of $4$
values varying from $[-1,1]$ and two values from $[-12.57,12.57]$ and
$[28.27,28.27]$. Figure 2 displays different error margins for $5$ executions
of SAIL. It is possible to note that, although $\mathcal{G}$ has a higher
error rate for the Acrobot environment, its initial value ($\approx 2.5$) is
higher due to the environment encoding characteristics. Thus, its higher error
rate (when compared with all other environments), contextually, could be
considered a lower margin rate.

##  VII Conclusion

In this paper, we developed a novel LfO approach that uses an adversarial
module to learn how to imitate the behaviour of teachers without accessing its
actions and is capable of achieving state-of-the-art results in performance
and efficiency. We evaluate our model under different amounts of sampled
behaviour and compare it with various baselines from the literature. SAIL
(Self-supervised Adversarial Imitation Learning) achieves significant results
in both Performance and AER with fewer samples for two main reasons. First, we
use an adversarial mechanism to better approximate our model’s behaviour with
the teacher’s. Such a mechanism is connected to our end-to-end model and
iteratively learns through a loss error, which makes the model achieve higher
returns. Second, SAIL uses an exploration technique that helps the model
collect the best data for each interaction, resulting in a model’s convergence
in fewer steps. Finally, we evaluate and discuss the metrics by which we can
measure how much a policy actually imitates sampled behaviour. To that end, we
carry out an ablation study in Section VI where we show that SAIL is capable
of achieving proficient rewards while still ‘fooling’ its discriminator,
including analysis through our obtained results.

In future work, we aim to test our method in a variety of different
environments to understand better SAIL’s ability to imitate other agent
behaviour vis-à-vis other baselines from literature. We believe that by
changing our model’s topology, we can test with environments that represent
their states with images, which could result in learning from teachers by
collecting available videos from the internet (e.g., YouTube) since we do not
need previous knowledge of actions used by the teachers. And further
investigate how we can measure proficient behaviour in different environments
for a better understanding of possible emergent behaviour by the agent, and
its impact on real-world applications.

##  VIII Acknowledgements

This work was supported by UK Research and Innovation [grant number
EP/S023356/1], in the UKRI Centre for Doctoral Training in Safe and Trusted
Artificial Intelligence (www.safeandtrustedai.org) and made possible via
King’s Computational Research, Engineering and Technology Environment (CREATE)
[27].

## References

  * [1] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne, “Imitation learning: A survey of learning methods,” _ACM Computing Surveys_ , vol. 50, no. 2, pp. 21:1–21:35, 2017.
  * [2] Y. Liu, A. Gupta, P. Abbeel, and S. Levine, “Imitation from observation: Learning to imitate behaviors from raw video via context translation,” in _Proceedings of ICRA 2018_ , 2018, pp. 1118–1125.
  * [3] F. Torabi, G. Warnell, and P. Stone, “Behavioral cloning from observation,” in _Proceedings of IJCAI’18_ , 2018, pp. 4950–4957.
  * [4] J. Monteiro, N. Gavenski, R. Granada, F. Meneguzzi, and R. C. Barros, “Augmented behavioral cloning from observation,” in _Proceedings of the 2020 International Conference on Neural Networks_ , ser. IJCNN 2020, Proceedings of the 2020 International Conference on Neural Networks. IEEE, Jul 2020, pp. 1–8. [Online]. Available: https://arxiv.org/abs/2004.13529
  * [5] N. Gavenski, J. Monteiro, R. Granada, F. Meneguzzi, and R. C. Barros, “Imitating unknown policies via exploration,” _arXiv preprint arXiv:2008.05660_ , 2020.
  * [6] Z. Zhu, K. Lin, B. Dai, and J. Zhou, “Off-policy imitation learning from observations,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 402–12 413, 2020.
  * [7] L. Le Mero, D. Yi, M. Dianati, and A. Mouzakitis, “A survey on imitation learning techniques for end-to-end autonomous vehicles,” _IEEE Transactions on Intelligent Transportation Systems_ , 2022.
  * [8] D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” in _Proceedings of the 1st Conference on Neural Information Processing Systems_ , ser. NIPS 1988. Proceedings of the 1st Conference on Neural Information Processing Systems, 1988, pp. 305–313.
  * [9] B. Zheng, S. Verma, J. Zhou, I. W. Tsang, and F. Chen, “Imitation learning: Progress, taxonomies and challenges,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–16, 2022.
  * [10] F. Torabi, G. Warnell, and P. Stone, “Generative adversarial imitation from observation,” in _I3 Workshop at ICML 2019_ , 2019.
  * [11] J. Ho and S. Ermon, “Generative adversarial imitation learning,” in _Proceedings of NIPS 2016_ , 2016, pp. 4565–4573.
  * [12] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Proceedings of NIPS’14_ , 2014, pp. 2672–2680.
  * [13] R. S. Sutton and A. G. Barto, _Reinforcement Learning_ , 2nd ed. The MIT Press, 2018.
  * [14] N. S. Gavenski, “Self-supervised imitation learning from observation,” Master’s thesis, Pontifícia Universidade Católica do Rio Grande do Sul, 2021.
  * [15] R. Kidambi, J. Chang, and W. Sun, “Mobile: Model-based imitation learning from observation alone,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 28 598–28 611.
  * [16] M. Wooldridge, _An introduction to multiagent systems_. John wiley & sons, 2009.
  * [17] A. W. Moore, “Efficient memory-based learning for robot control,” University of Cambridge, Tech. Rep., 1990.
  * [18] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” _arXiv preprint arXiv:1506.02438_ , 2015.
  * [19] A. G. Barto, R. S. Sutton, and C. W. Anderson, “Neuronlike adaptive elements that can solve difficult learning control problems,” _IEEE transactions on systems, man, and cybernetics_ , vol. 1, no. 5, pp. 834–846, Sep 1983.
  * [20] N. Gavenski, J. Monteiro, A. Medronha, and R. C. Barros, “How resilient are imitation learning methods to sub-optimal experts?” in _Intelligent Systems_ , J. C. Xavier-Junior and R. A. Rios, Eds. Cham: Springer International Publishing, 2022, pp. 449–463.
  * [21] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” _arXiv preprint arXiv:1606.01540_ , 2016.
  * [22] A. Graves and A. Graves, “Long short-term memory,” _Supervised sequence labelling with recurrent neural networks_ , pp. 37–45, 2012.
  * [23] R. S. Sutton, “Generalization in reinforcement learning: Successful examples using sparse coarse coding,” in _Advances in Neural Information Processing Systems_ , D. Touretzky, M. Mozer, and M. Hasselmo, Eds., vol. 8. MIT Press, 1995.
  * [24] H. M. Le and Y. Yue, “Imitation learning tutorial,” 2018, iCML Presentation. [Online]. Available: https://sites.google.com/view/icml2018-imitation-learning
  * [25] G. Swamy, S. Choudhury, J. A. Bagnell, and S. Wu, “Of moments and matching: A game-theoretic framework for closing the imitation gap,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 10 022–10 032.
  * [26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [27] K. C. L. e Research team, “King’s computational research, engineering and technology environment (create),” 2023. [Online]. Available: https://doi.org/10.18742/rnvf-m076