$\displaystyle=\frac{|\mathcal{C}\cap\mathcal{H}|}{|\mathcal{C}|}\delta\leq\delta,$

which leads to the desired result. ∎

The next lemma provides the main intermediate result we need to prove Theorem
2.

######  Lemma 7.

Suppose Robust Fed-SEL is run by every honest client. Moreover, suppose
$|\mathcal{C}_{j}\cap\mathcal{J}|\leq f$ and $|\mathcal{C}_{j}|\geq
3f+1,\forall j\in\mathcal{[}N]$. On event $\mathcal{E_{H}}$, the following
then hold:

  * (i)

$\bar{a}_{j}=j_{1},\forall j\in[N]$.

  * (ii)

In very round $k$,

$\bar{r}^{(U)}_{j}(k)\in\texttt{Conv}\left(\\{\tilde{r}^{(U)}_{i}(k)\\}_{i\in\bar{\mathcal{C}}_{j}\cap\mathcal{H}}\right);\hskip
5.69054pt\bar{r}^{(L)}_{j}(k)\in\texttt{Conv}\left(\\{\tilde{r}^{(L)}_{i}(k)\\}_{i\in\bar{\mathcal{C}}_{j}\cap\mathcal{H}}\right),\forall
j\in[N],$ (45)

where we use $\texttt{Conv}(\Omega)$ to represent the convex hull of a set of
points $\Omega$.

###### Proof.

Fix a client group $j\in[N].$ Based on Lemma 2, note that when each client
$i\in\mathcal{C}_{j}\cap\mathcal{H}$ runs Algo. 1 on $\mathcal{A}_{j}$, the
local subroutine will eventually terminate on event $\mathcal{E_{H}}$, and
client $i$ will output $a_{i}=j_{1}$, i.e., each honest client $i$ in
$\mathcal{C}_{j}$ will correctly identify the locally best arm $j_{1}$ in
$\mathcal{A}_{j}$ at the end of Phase I. Since
$|\mathcal{C}_{j}\cap\mathcal{J}|\leq f$ and $|\mathcal{C}_{j}|\geq 3f+1$, at
least $2f+1$ clients from $\mathcal{C}_{j}$ are guaranteed to report to the
server at the end of Phase I. Among them, since at most $f$ can be
adversarial, observe that the majority voting operation in line 2 of Robust
Fed-SEL yields $\bar{a}_{j}=j_{1}$. This establishes the first claim of the
lemma.

For the second claim, let us start by noting that on $\mathcal{E_{H}}$, the
number of adversaries in the set $\bar{\mathcal{C}}_{j}$ is at most
$f_{j}=f-|\mathcal{C}^{\prime}_{j}|+|\bar{\mathcal{C}}_{j}|$. This follows
from the fact that on $\mathcal{E_{H}}$, any client
$i\in\mathcal{C}^{\prime}_{j}$ who reports an arm other than $j_{1}$ must be
adversarial, i.e., on $\mathcal{E_{H}}$,
$\mathcal{C}^{\prime}_{j}\setminus\bar{\mathcal{C}}_{j}\subseteq\mathcal{J}$.
Moreover, observe that

$\displaystyle 2f_{j}+1$
$\displaystyle=2\left(f-|\mathcal{C}^{\prime}_{j}|+|\bar{\mathcal{C}}_{j}|\right)+1$
(46) $\displaystyle=2\left(f-(2f+1)+|\bar{\mathcal{C}}_{j}|\right)+1$
$\displaystyle=\left(|\bar{\mathcal{C}}_{j}|-(2f+1)\right)+|\bar{\mathcal{C}}_{j}|$
$\displaystyle\leq\left(|\mathcal{C}^{\prime}_{j}|-(2f+1)\right)+|\bar{\mathcal{C}}_{j}|$
$\displaystyle=|\bar{\mathcal{C}}_{j}|.$

With these basic observations in place, we are now ready to analyze the Trim
operation outlined in line 6 of Robust Fed-SEL. To establish the claim that

$\bar{r}^{(U)}_{j}(k)\in\texttt{Conv}\left(\\{\tilde{r}^{(U)}_{i}(k)\\}_{i\in\bar{\mathcal{C}}_{j}\cap\mathcal{H}}\right),$

suppose
$\texttt{Trim}\left(\\{\tilde{r}^{(U)}_{i}(k)\\}_{i\in\bar{\mathcal{C}}_{j}},f_{j}\right)=\tilde{r}^{(U)}_{w}(k)$,
where $w\in\bar{\mathcal{C}}_{j}$. If $w\in\mathcal{H}$, then the claim holds
trivially. Now suppose $w$ is adversarial, i.e., $w\in\mathcal{J}$. Since
$|\bar{\mathcal{C}}_{j}\cap\mathcal{J}|\leq f_{j}$, and
$|\bar{\mathcal{C}}_{j}|\geq 2f_{j}+1$, there must exist honest clients
$u,v\in\bar{\mathcal{C}}_{j}\cap\mathcal{H}$, such that
$\tilde{r}^{(U)}_{v}(k)\leq\tilde{r}^{(U)}_{w}(k)\leq\tilde{r}^{(U)}_{u}(k)$.
This establishes the claim about $\bar{r}^{(U)}_{j}(k)$ in Eq. (45); the claim
regarding $\bar{r}^{(L)}_{j}(k)$ follows the exact same argument. ∎

The above lemma tells us that with high probability, (i) the server will
correctly identify the locally best arm from each arm-set, and (ii) the
thresholds used for eliminating client-groups are “not too corrupted” by the
adversaries. We now show how these two properties feature in the proof of
Theorem 2.

###### Proof.

(Theorem 2) Throughout the proof, we will condition on the event
$\mathcal{E_{H}}$ defined in Eq. (43). Based on Lemma 6, recall that
$\mathcal{E_{H}}$ occurs with probability at least $1-\delta$. As in the proof
of Theorem 1, we start by arguing that client group $1$ always remains active
(recall that $a^{*}=1\in\mathcal{A}_{1}$). To see why this is true, fix any
$j\in[N]$, and observe that for each
$i\in\bar{\mathcal{C}}_{j}\cap\mathcal{H}$ and every round $k$,

$\displaystyle\tilde{r}^{(L)}_{i}(k)$
$\displaystyle=\tilde{r}_{i}(k)-2\alpha_{j}(\bar{t}_{i}+kH)$ (47)
$\displaystyle\overset{(a)}{\leq}\hat{r}_{i,\bar{a}_{j}}(\bar{t}_{i}+kH)+\frac{\alpha_{j}(\bar{t}_{i}+kH)}{2}-2\alpha_{j}(\bar{t}_{i}+kH)$
$\displaystyle\overset{(b)}{=}\hat{r}_{i,j_{1}}(\bar{t}_{i}+kH)-\frac{3}{2}\alpha_{j}(\bar{t}_{i}+kH)$
$\displaystyle\overset{\mathcal{E}_{\mathcal{H}}}{\leq}{r_{j_{1}}}+\alpha_{j}(\bar{t}_{i}+kH)-\frac{3}{2}\alpha_{j}(\bar{t}_{i}+kH)$
$\displaystyle<r_{j_{1}}.$

In the above steps, (a) follows from (i) the fact that by definition of the
set $\bar{\mathcal{C}}_{j}$, every client $i$ in $\bar{\mathcal{C}}_{j}$
reports $a_{i}=\bar{a}_{j}$, and (ii) by using the bound on the quantization
error in Eq. (26). Based on Lemma 7, the majority voted arm $\bar{a}_{j}$ is
$j_{1}$ on event $\mathcal{E_{H}}$; this leads to $(b)$. Using the same
reasoning, we can show that for each
$i\in\bar{\mathcal{C}}_{j}\cap\mathcal{H}$ and every round $k$,

$\tilde{r}^{(U)}_{i}(k)>r_{j_{1}}$ (48)

on event $\mathcal{E_{H}}$. Appealing to (45) from Lemma 7, and using (47) and
(48), we conclude that on $\mathcal{E_{H}}$,

$\bar{r}^{(U)}_{1}(k)>r_{1};\hskip
11.38109pt\bar{r}^{(L)}_{j}(k)<r_{j_{1}},\forall j\in[N]\setminus\\{1\\}.$

From lines 7 and 8 of Robust Fed-SEL, we immediately note that client-group
$1$ never gets eliminated. To show that Robust Fed-SEL is consistent, it
remains to argue that on $\mathcal{E_{H}}$, every client-group
$j\in[N]\setminus\\{1\\}$ eventually gets eliminated. We claim that for group
$j\in[N]\setminus\\{1\\}$ to get eliminated, the following is a set of
sufficient conditions:

$\displaystyle\alpha_{j}(\bar{t}_{i}+kH)$
$\displaystyle\leq\frac{\Delta_{1j_{1}}}{8},\forall
i\in\bar{\mathcal{C}}_{j}\cap\mathcal{H},\textrm{and}$ (49)
$\displaystyle\alpha_{1}(\bar{t}_{i}+kH)$
$\displaystyle\leq\frac{\Delta_{1j_{1}}}{8},\forall
i\in\bar{\mathcal{C}}_{1}\cap\mathcal{H}.$

Suppose the above conditions hold. Then, based on the same reasoning we used
to arrive at Eq. (30), we can show that for each
$i\in\bar{\mathcal{C}}_{j}\cap\mathcal{H}$,

$\displaystyle\tilde{r}^{(U)}_{i}(k)\leq
r_{j_{1}}+\frac{7}{16}\Delta_{1j_{1}},$ (50)

and for each $i\in\bar{\mathcal{C}}_{1}\cap\mathcal{H}$,

$\tilde{r}^{(L)}_{i}(k)\geq r_{1}-\frac{7}{16}\Delta_{1j_{1}}.$ (51)

Appealing once again to the second claim of Lemma 7, and using (50) and (51),
we conclude that if the conditions in (49) hold, then

$\bar{r}^{(L)}_{1}(k)\geq r_{1}-\frac{7}{16}{\Delta_{1j_{1}}}\hskip
2.84526pt\textrm{and}\hskip 2.84526pt\bar{r}^{(U)}_{j}(k)\leq
r_{j_{1}}+\frac{7}{16}{\Delta_{1j_{1}}},$ (52)

implying $\bar{r}^{(L)}_{1}(k)>\bar{r}^{(U)}_{j}(k)$. From line 8 of Robust
Fed-SEL, this is sufficient for client-group $j$ to get eliminated. Thus,
Robust Fed-SEL will eventually terminate on $\mathcal{E_{H}}$ with
$\bar{\Psi}(k+1)=\\{1\\}$, for some round $k$. Consistency then follows from
Lemma 7 by noting that on $\mathcal{E_{H}}$, $\bar{a}_{1}=a^{*}=1$.

The sample- and communication-complexity bounds of Robust Fed-SEL are of the
same order as that of Fed-SEL, and can be derived in exactly the same way as
in Theorem 1. We now briefly explain how this can be done. Suppose we want to
compute an upper bound on the number of rounds client group
$j\in[N]\setminus\\{1\\}$ remains active. Start by noting that at the end of
Phase I, the following bound applies uniformly to every
$i\in\bar{\mathcal{C}}_{j}\cap\mathcal{H}$:

$\alpha_{j}(\bar{t}_{i})<\frac{2}{9}\Delta^{*}_{j}<\frac{\sigma_{j1}\Delta_{1j_{1}}}{8}.$

This follows from Lemma 3. For each
$i\in\bar{\mathcal{C}}_{j}\cap\mathcal{H}$, let $R^{(j)}_{i1}$ be the first
round such that

$\alpha_{j}(\bar{t}_{i}+kH)\leq\frac{\Delta_{1j_{1}}}{8}.$

We can obtain an upper bound on $R^{(j)}_{i1}$ for each
$i\in\bar{\mathcal{C}}_{j}\cap\mathcal{H}$ exactly as in Theorem 1, and show
that

$\max_{i\in\bar{\mathcal{C}}_{j}\cap\mathcal{H}}R^{(j)}_{i1}\leq\frac{1}{H}\left(\frac{128}{{(\Delta^{*}_{j})}^{2}}(\sigma^{2}_{j1}-1)\log{\frac{128\bar{c}_{j}}{\Delta^{2}_{1j_{1}}}}+\frac{256}{{(\Delta^{*}_{j})}^{2}}\log{\sigma_{j1}}+O\left(\frac{1}{\Delta^{2}_{1j_{1}}}\log\log{\frac{\bar{c}_{j}}{\Delta^{2}_{1j_{1}}}}\right)\right)+1,\hskip
2.84526pt\textrm{if}\hskip 2.84526pt\sigma_{j1}>1,$

and $\max_{i\in\bar{\mathcal{C}}_{j}\cap\mathcal{H}}R^{(j)}_{i1}=1$, if
$\sigma_{j1}\leq 1.$ With
$R_{j1}\triangleq\max_{i\in\bar{\mathcal{C}}_{j}\cap\mathcal{H}}R^{(j)}_{i1}$,
observe that the expression for $R_{j1}$ exactly resembles that for $R_{i1}$
in Eq. (5) of Theorem 1.131313Recall that in the context of Theorem 1, each
client group contained only one agent, and hence the index $i$ was used for
both client groups and agents. Proceeding as we did in Theorem 1, we can now
derive an upper bound on the number of rounds group $j$ remains active. The
rest of the proof mirrors that of Theorem 1, and hence we omit details. ∎

##  Appendix C Proof of Theorem 3

In this section, we will analyze the peer-to-peer setting described in Section
6. As before, we need a “good event” on which we will base our reasoning. For
the setting we consider here, the event $\mathcal{E_{H}}$ described in Eq.
(43) will serve such a purpose. Recall that $\mathbb{P}(\mathcal{E_{H}})\geq
1-\delta.$

Consider any honest client $i\in\mathcal{C}_{j}.$ The argument we present next
applies identically to honest clients in other client groups as well. Since
$i\in\mathcal{C}_{j}$, running Algo. 1 on $\mathcal{A}_{j}$ with
$\beta_{j}(t)=6\alpha_{j}(t)$ yields

$a_{i}=j_{1};\hskip
2.84526pt|\hat{r}_{i,a_{i}}(\bar{t}_{i})-r_{j_{1}}|=|\hat{r}_{i,j_{1}}(\bar{t}_{i})-r_{j_{1}}|\leq\alpha_{j}(\bar{t}_{i})<\frac{c}{(c-2)^{2}}\Delta^{*}_{j}=\frac{3}{8}\Delta^{*}_{j}$
(53)

on event $\mathcal{E_{H}}$. The above claims follow directly from Lemma’s 2
and 3. Thus, based on Line 1 of Algo. 4, we have $\mathbf{I}_{i}(j)=j_{1}$,
and $\mathbf{R}_{i}(j)=\hat{r}_{i,j_{1}}(\bar{t}_{i})$, where
$\hat{r}_{i,j_{1}}(\bar{t}_{i})$ satisfies the bound in Eq. (53).

Now for client $i$ to identify the globally best arm, it must eventually
receive accurate information about every other arm-set
$\ell\in[N]\setminus\\{j\\}$ from its neighbors, despite the presence of
adversaries in the network. The next lemma shows how the strong-robustness
property in Definition 2 contributes to this cause.

######  Lemma 8.

Suppose Algo. 4 is run by every $i\in\mathcal{H}$. Moreover, suppose the
adversary model is $f$-local, and $\mathcal{G}$ is strongly-robust w.r.t.
$\mathcal{C}_{j},\forall j\in[N]$. Consider any $j\in[N]$. Then, on the event
$\mathcal{E_{H}}$, every client $i\in\mathcal{V}\setminus\mathcal{C}_{j}$
receives information about arm-set $j$ from at least $2f+1$ neighbors in
$\mathcal{N}_{i}$.

###### Proof.

We prove this lemma by contradiction. Fix any $j\in[N]$. Suppose that on the
event $\mathcal{E_{H}}$, there exists a non-empty set
$\mathcal{B}\subseteq\mathcal{V}\setminus\mathcal{C}_{j}$ of clients who never
receive information about arm-set $j$ from $2f+1$ neighbors. Based on the
strong-robustness property in Def. 2, there exists a client $i\in\mathcal{B}$
with at least $3f+1$ neighbors in $\mathcal{V}\setminus\mathcal{B}$. Consider
the set
$\mathcal{Q}_{i,j}=\mathcal{N}_{i}\cap\\{\mathcal{V}\setminus\mathcal{B}\\}$.
Based on the strong-robustness property and the $f$-local assumption,
$|\mathcal{Q}_{i,j}\cap\mathcal{H}|\geq 2f+1$. Now a client $k$ in
$\mathcal{Q}_{i,j}\cap\mathcal{H}$ can either belong to $\mathcal{C}_{j}$ or
not. If $k\in\mathcal{Q}_{i,j}\cap\mathcal{H}$ belongs to $\mathcal{C}_{j}$,
then on the event $\mathcal{E_{H}}$, Algo. 1 will terminate at client $k$.
Thus, client $k$ will eventually transmit
$\\{\mathbf{I}_{k}(j),\mathbf{R}_{k}(j)\\}$ to every out-neighbor in
$\mathcal{N}^{+}_{k}$, including client $i$. Even if
$k\in\mathcal{Q}_{i,j}\cap\mathcal{H}$ does not belong to $\mathcal{C}_{j}$,
since $k\notin\mathcal{B}$, client $k$ must have received information about
arm-set $j$ from $2f+1$ in-neighbors. Thus, based on lines 2-4 of Algo. 4, it
must transmit $\\{\mathbf{I}_{k}(j),\mathbf{R}_{k}(j)\\}$ to client $i$ at
some point of time. We conclude that on the event $\mathcal{E_{H}}$, client
$i$ will eventually receive information about arm-set $j$ from every neighbor
in $\mathcal{Q}_{i,j}\cap\mathcal{H}$. This leads to the desired contradiction
and concludes the proof. ∎

Our next goal is to prove that on $\mathcal{E_{H}}$, for each $j\in[N]$, every
honest client $i\in\mathcal{V}\setminus\mathcal{C}_{j}$ is able to correctly
identify the best arm $j_{1}$ in $\mathcal{A}_{j}$, and obtain a reasonable
estimate of its mean $r_{j_{1}}$. We establish these properties in the next
lemma.

######  Lemma 9.

Suppose Algo. 4 is run by every $i\in\mathcal{H}$. Moreover, suppose the
adversary model is $f$-local, and $\mathcal{G}$ is strongly-robust w.r.t.
$\mathcal{C}_{j},\forall j\in[N]$. On event $\mathcal{E_{H}}$, the following
then hold $\forall j\in[N]$:

  * (i)

$\mathbf{I}_{i}(j)=j_{1},\forall i\in\mathcal{H}$.

  * (ii)

$|\mathbf{R}_{i}(j)-r_{j_{1}}|\leq(3/8)\Delta^{*}_{j},\forall
i\in\mathcal{H}.$

###### Proof.

Fix any $j\in[N]$. If $i\in\mathcal{C}_{j}\cap\mathcal{H}$, then each of the
above claims hold based on the arguments we presented at the beginning of this
section; see Eq. (53). Thus, we only need to prove these claims for clients in
$\\{\mathcal{V}\setminus\mathcal{C}_{j}\\}\cap\mathcal{H}$.

To do so, let us first associate a notion of “activation” with Algo. 4. For
each client $i\in\mathcal{C}_{j}\cap\mathcal{H}$, we say that it gets
activated when the local subroutine (Algo. 1) terminates at client $i$. For
each $i\in\\{\mathcal{V}\setminus\mathcal{C}_{j}\\}\cap\mathcal{H}$, we say it
gets activated upon receiving information about arm-set $j$ from $2f+1$
neighbors in $\mathcal{N}_{i}$. From Lemma’s 2 and 8, observe that all honest
clients eventually get activated on the event $\mathcal{E_{H}}$. Accordingly,
on the event $\mathcal{E_{H}}$, let $\tau_{1}<\tau_{2}<\cdots<\tau_{K}$ denote
the activation times for clients in
$\\{\mathcal{V}\setminus\mathcal{C}_{j}\\}\cap\mathcal{H}$.141414Note that
multiple clients may get activated at the same time-step. Moreover, let
$\mathcal{H}^{(j)}_{\tau_{s}}$ represent those clients in
$\\{\mathcal{V}\setminus\mathcal{C}_{j}\\}\cap\mathcal{H}$ that get activated
at time-step $\tau_{s},s\in[K]$.

We prove the result by inducting on the activation times. For the base case,
consider any client $i$ in the set $\mathcal{H}^{(j)}_{\tau_{1}}$. Recall that
$\mathcal{N}^{\prime}_{i,j}\subseteq\mathcal{N}_{i}$ are the first $2f+1$
neighbors who report about arm-set $j$ to client $i$. From the rules of Algo.
4, notice that an honest client $k$ transmits
$\\{\mathbf{I}_{k}(j),\mathbf{R}_{k}(j)\\}$ only when it gets activated. Thus,
it must be that
$\mathcal{N}^{\prime}_{i,j}\cap\mathcal{H}\subseteq\mathcal{C}_{j}\cap\mathcal{H}$.
Moreover, since the adversary model is $f$-local,
$|\mathcal{N}^{\prime}_{i,j}\cap\mathcal{H}|\geq f+1$. We conclude that at
least $f+1$ honest clients in $\mathcal{C}_{j}$ report arm $j_{1}$ to client
$i$. Based on the majority voting operation in line 3 of Algo. 4, it follows
that $\mathbf{I}_{i}(j)=j_{1}$.

Observe also that on the event $\mathcal{E_{H}}$, any client
$k\in\mathcal{N}^{\prime}_{i,j}$ that reports an arm other than arm $j_{1}$ to
client $i$ must be adversarial. Following the same reasoning as in Lemma 7, we
can then establish that (i) $|\bar{\mathcal{N}}_{i,j}|\geq 2f_{i,j}+1$, where
$f_{i,j}=f-|\mathcal{N}^{\prime}_{i,j}|+|\bar{\mathcal{N}}_{i,j}|$, and
$\bar{\mathcal{N}}_{i,j}$ represents those clients in
$\mathcal{N}^{\prime}_{i,j}$ that report arm $j_{1}$; and (ii)
$|\bar{\mathcal{N}}_{i,j}\cap\mathcal{J}|\leq f_{i,j}$. Using the same
arguments used to arrive at Eq. (45), we can then show that

$\mathbf{R}_{i}(j)\in\texttt{Conv}\left(\\{\mathbf{R}_{k}(j)\\}_{k\in\bar{\mathcal{N}}_{i,j}\cap\mathcal{H}}\right)\implies\mathbf{R}_{i}(j)\in\texttt{Conv}\left(\\{\mathbf{R}_{k}(j)\\}_{k\in{\mathcal{C}}_{j}\cap\mathcal{H}}\right),$
(54)

since
$\bar{\mathcal{N}}_{i,j}\cap\mathcal{H}\subseteq{\mathcal{C}}_{j}\cap\mathcal{H}$.
For client $i$, claim (ii) of the lemma then follows immediately from the fact
that on event $\mathcal{E_{H}}$,
$|\mathbf{R}_{k}(j)-r_{j_{1}}|\leq(3/8)\Delta^{*}_{j},\forall
k\in{\mathcal{C}}_{j}\cap\mathcal{H}$. This completes the base case.

For the induction hypothesis, suppose for all
$i\in\bigcup_{s\in[m]}\mathcal{H}^{(j)}_{\tau_{s}}$, $m\in[K-1]$, it holds
that $\mathbf{I}_{i}(j)=j_{1}$, and

$\mathbf{R}_{i}(j)\in\texttt{Conv}\left(\\{\mathbf{R}_{k}(j)\\}_{k\in{\mathcal{C}}_{j}\cap\mathcal{H}}\right).$
(55)

Now consider a client $i\in\mathcal{H}^{(j)}_{\tau_{m+1}}$. Based on the rules
of Algo. 4, and the definition of activation times, it must be that

$\mathcal{N}^{\prime}_{i,j}\cap\mathcal{H}\subseteq\\{\mathcal{C}_{j}\cap\mathcal{H}\\}\cup\\{\bigcup_{s\in[m]}\mathcal{H}^{(j)}_{\tau_{s}}\\}.$

Using the above fact, $|\mathcal{N}^{\prime}_{i,j}\cap\mathcal{H}|\geq f+1$,
the induction hypothesis, and the same reasoning as the base case, it is easy
to see that $\mathbf{I}_{i}(j)=j_{1}$, and that $\mathbf{R}_{i}(j)$ satisfies
the inclusion in Eq. (55). This completes the induction hypothesis and the
proof. ∎

We are now ready to complete the proof of Theorem 3.

###### Proof.

(Theorem 3) To complete the proof of Theorem 3, it remains to argue that for
each honest client $i$, $\mathbf{R}_{i}(1)>\mathbf{R}_{i}(j),\forall
j\in[N]\setminus\\{1\\}$. By way of contradiction, suppose there exists some
client $i\in\mathcal{H}$ and some $j\in[N]\setminus\\{1\\}$, such that
$\mathbf{R}_{i}(j)\geq\mathbf{R}_{i}(1)$. We then have

$\displaystyle\mathbf{R}_{i}(j)\geq\mathbf{R}_{i}(1)$ (56)
$\displaystyle\overset{(a)}{\implies}$ $\displaystyle
r_{j_{1}}+\frac{3}{8}\Delta^{*}_{j}\geq r_{1}-\frac{3}{8}\Delta^{*}_{1}$
$\displaystyle\implies$
$\displaystyle\Delta_{1j_{1}}\leq\frac{3}{8}(\Delta^{*}_{1}+\Delta^{*}_{j})$
$\displaystyle\overset{(b)}{\implies}$
$\displaystyle\Delta_{1j_{1}}\leq\frac{3}{8}\left(\sigma_{1j}+\sigma_{j1}\right)\Delta_{1j_{1}}$
$\displaystyle\overset{(c)}{\implies}$
$\displaystyle\Delta_{1j_{1}}\leq\frac{3}{4}\Delta_{1j_{1}},$

which leads to the desired contradiction. Here, (a) follows from the second
claim of Lemma 9, (b) follows from the definition of arm-heterogeneity indices
in Eq. (4), and (c) follows from the assumption on arm-heterogeneity in Eq.
(8). This completes the proof. ∎


# Universal aspects of droplet spreading dynamics in Newtonian and non-
Newtonian Fluids

B. Gorin  [<EMAIL_ADDRESS>G. Di Mauro  [  gabrielle.di-
<EMAIL_ADDRESS>D. Bonn  [<EMAIL_ADDRESS>H. Kellay  [
<EMAIL_ADDRESS>
###### Abstract

Droplet impacts are common in many applications such as coating, spraying, or
printing; understanding how droplets spread after impact is thus of utmost
importance. Such impacts may occur with different velocities on a variety of
substrates. The fluids may also be non-Newtonian and thus possess different
rheological properties. How the different properties such as surface roughness
and wettability, droplet viscosity and rheology as well as interfacial
properties affect the spreading dynamics of the droplets and the eventual drop
size after impact are unresolved questions.

Most recent work focuses on the maximum spreading diameter after impact and
uses scaling laws to predict this. In this paper we show that a proper
rescaling of the spreading dynamics with the maximum radius attained by the
drop, and the impact velocity leads to a unique single and thus universal
curve for the variation of diameter versus time. The validity of this
universal functional shape is validated for different liquids with different
rheological properties as well as substrates with different wettabilities.
This universal function agrees with a recent model that proposes a closed set
of differential equations for the spreading dynamics of droplets.

University of Amsterdam] Van der Waals Zeeman Institute, University of
Amsterdam,1018 XE Amsterdam, The Netherlands [University of Bordeaux]
Laboratoire Ondes et Matière d’Aquitaine, Université de Bordeaux, 33400
Talence, France University of Amsterdam] Van der Waals Zeeman Institute,
University of Amsterdam,1018 XE Amsterdam, The Netherlands University of
Amsterdam] Van der Waals Zeeman Institute, University of Amsterdam,1018 XE
Amsterdam, The Netherlands University of Bordeaux] Laboratoire Ondes et
Matière d’Aquitaine, Université de Bordeaux, 33400 Talence, France [University
of Bordeaux] Institut Universitaire de France, 75005 Paris, France

##  1 Introduction

Droplet impacts are routinely studied josserand_drop_2016 to understand a wide
range of phenomena ranging from spraying, coating, printing
hoath2016fundamentals, glasser_tuning_2019, xie_delamination_2021,
lohse_fundamental_2022 to self cleaning and anti-icing
thievenaz_solidification_2019. Typical examples abound from every day life or
in industrial applications: a droplet of rain impacting a glass pane
cebeci_aircraft_2003, the deposition of a molten metal droplet for electronic
printing fukanuma_porosity_1994, tavakoli_spreading_2014,
gielen_solidification_2020, or droplet impacts of polymers or surfactants
solutions usually used for a more controlled deposition
bergeron_controlling_2000, bartolo_retraction_2005, bartolo_dynamics_2007,
jalaal_gel-controlled_2018, hoffman_controlling_2021. It is thus central to
understand droplet spreading and give it a proper description. The difficulty
of understanding droplet impacts stems from the fact that this process
involves many different contributions such as viscous dissipation, surface
tension effects, surface roughness, wetting and spreading as well as the
dynamics of the contact line. It is therefore arduous to propose a simple
description for the spreading dynamics upon impact and describe how the
diameter of the drop $D(t)$ evolves with time $t$ biance_first_2004,
eggers_drop_2010, lagubeau_spreading_2012, eddi_short_2013,
wildeman_spreading_2016 . Many studies focus on the maximum spreading diameter
$D_{max}$ and propose different scaling laws using arguments of energy
conservation madejski_droplets_1983, clanet_maximal_2004, ukiwe_maximum_2005,
laan_maximum_2014, lee_modeling_2016, wei_maximum_2021 . Recently, for the
purpose of describing the dynamics of spreading of Newtonian fluids, a theory
proposed by Gordillo et.al riboux_maximum_2016, gordillo_theory_2019, garcia-
geijo_spreading_2021 gives a solution for $D(t)$ by solving a system of
equations describing the spreading of an impacting droplet. This model is
based on modelling the spreading droplet as a thick rim followed by a thin
liquid film connected to the bulk of the drop. Using mass and momentum
conservation between the rim, the liquid film and the droplet, a set of
coupled differential equations is proposed. Solving these equations with the
proper boundary conditions leads to a complete description of $D(t)$ for
Newtonian drops impacting a smooth surface. In this work, we carry out
experiments of droplet spreading dynamics on a variety of fluids, with
different rheological properties, impacting at different velocities substrates
with different wettabilities. Our results show that a simple rescaling of
$D(t)$ with the maximum diameter $D_{max}$ and a rescaling of the time scale
by a characteristic time $D_{max}/V_{0}$ where $V_{0}$ is the impact velocity,
leads to a single master curve describing the spreading dynamics of droplet
impacts independently of the substrate, interfacial or rheological properties.
Our scaling function agrees with the recent theory of Gordillo et.al
gordillo_theory_2019 . Our work therefore shows that the diameter $D(t)$ of
spreading droplets depends only on the maximum spreading diameter $D_{max}$
and the impact velocity $V_{0}$.

##  2 Methods and materials

We perform a series of droplet impact experiments with different liquids
(water, hexadecane, aqueous polymer solutions and Carbopol solutions)
impacting different surfaces. The experimental setup is as follows: a syringe
with a needle is fixed at a certain height which sets the impact velocity
$V_{0}$ (varied between $1m.s^{-1}$ and $4m.s^{-1}$). The syringe is connected
to a syringe pump to release droplets using very low injection fluxes. Droplet
sizes $D_{0}$ can be varied between $1.80$ and $4.5mm$ (all droplet diameters
are given in the figure descriptions). The Reynolds and Weber numbers are
defined as $Re=\frac{\rho V_{0}D_{0}}{\eta}$ and $We=\frac{\rho
V_{0}^{2}D_{0}}{\sigma}$ where $\rho$, $\eta$ and $\sigma$ are respectively
the density, the viscosity and the surface tension of the liquid. Different
surfaces are used: smooth hydrophilic and hydrophobic microscope glass slides
(hydrophobic coating with Octadecyltrichlorosilane) as well as parafilm. All
experiments are done at room temperature $T=23\degree C$ and $RH=40\%$.
Droplet impacts are recorded at high framerates (up to 19000 fps) using a fast
camera (Phantom V640) aligned with the substrate ( see figure 1a,b and c).

ht!

Figure 1: (a) experimental setup for measuring droplet impacts on a flat
surface. (b) water droplet ($D_{0}=3.57\;mm$, $V_{0,a}=2.29\;m.s^{-1}$,
$Re=8100$, $We=260$) impacting a hydrophilic glass surface. (c) water droplet
($D_{0}=3.64\;mm$, $V_{0}=2.17\;m.s^{-1}$, $Re=7900$, $We=240$) impacting a
hydrophobic glass surface. Snapshots are taken at 0-1-2-3-4-5-8 ms after
impact. Scale bar : 2 mm

The liquids used for the droplets are pure water, hexadecane and two types of
non-Newtonian fluids. We use aqueous solutions of polymer (polyethylene oxide
also called PEO) of molecular weight of $4.10^{6}g.mol^{-1}$ at concentrations
varying between $10\;ppm$ and $5000\;ppm$. A small amount of isopropanol
(about 1 wt %) is added to avoid polymer aggregation. Experiments are done
using a yield stress fluid consisting of aqueous solutions of Carbopol at
different concentrations. In all cases, we focus our study on drop impacts
with high Reynolds numbers meaning that the spreading is mainly driven by
droplet inertia. Further, drop impact velocities are chosen in such a way that
splashing is avoided. The two Non-Newtonian fluids are used in order to test
the role of the liquid rheology. The presence of polymer renders the viscosity
of the liquid shear rate dependent and elastic polymer flow interactions can
take place ingremeau_stretching_2013, bonn_viscoelastic_1997. Further
experiments using Carbopol solutions are intended to examine how yield stress
fluids affect the spreading louvet_nonuniversality_2014,
martouzet_dynamic_2021 .

The rheology of the polymer solutions (PEO) was measured using an Anton Paar
rheometer with a cone plate geometry (two cones were used, angle $0.5\degree$
and diameter 60 $mm$ and angle $1\degree$ and diameter 50$mm$). For the
Carbopol solutions the flow curves were measured using a rough cone-plate
geometry (diameter = 40 $mm$ , angle = $1\degree 58^{\prime}$ , truncation
gap= 59 $\mu m$) on a controlled stress rheometer ARG2 (TA - Instruments). The
relaxation time for the 5000 ppm PEO aqueous solutions using the Carreau Model
is $\tau_{r}\sim 0.040\;s$. For Carbopol solutions, we use the same definition
as proposed by Luu.et.al luu_drop_2009 to estimate a characteristic time as :
$\tau_{r}=(\frac{k}{G})^{1/n}$ where $k$ is the fluid consistency (in
$Pa.s^{n}$), G the elastic shear modulus (in $Pa.$) and $n$ the flow index.

Figure 2: Rheology of (a) PEO at different concentrations and (b) Carbopol
solutions at two different concentrations. The flow curves are fit to the
Hershel Bulkley model $\sigma=\sigma_{ys}+kx^{n}$. Here $x$ is the shear rate,
$n$ the flow index, $k$ the consistency parameter, and $\sigma_{ys}$ is the
yield stress of the solution. The values of the parameters are given in the
figure.

The Deborah number is defined as the ratio of the relaxation time $\tau_{r}$
deduced from the rheology of the system and the observation time
$\tau=\frac{R_{0}}{V_{0}}$: $De=\frac{\tau_{r}}{\tau}$. For all impacts $De$
was much greater than 1 (10 for the Carbopol solutions and 40 for the 5000ppm
PEO solution at a velocity of 1m/s) so our drops behave like a fluid during
the impact.

In general, when a droplet impacts a flat smooth surface, it follows three
different regimes : An initial stage where the droplet spreads very quickly
until it reaches a maximum spreading diameter $D_{max}$. In principle, during
this stage, surface wettability is negligible compared to droplet inertia.
When the droplet reaches its maximum spreading diameter, it enters a stage
where wetting plays a role and the droplet can either retract or continue
spreading tanner_spreading_1979 depending on the surface wettability.


# Time fractional exact controllability

Paola Loreti Dipartimento di Scienze di Base e Applicate per l’Ingegneria,
Sapienza Università di Roma, Via Antonio Scarpa 16, 00161 Roma (Italy);
e-mail<EMAIL_ADDRESS>Daniela Sforza Dipartimento di Scienze
di Base e Applicate per l’Ingegneria, Sapienza Università di Roma, Via Antonio
Scarpa 16, 00161 Roma (Italy); e-mail<EMAIL_ADDRESS>
###### Abstract

Our purpose is to adapt the Hilbert Uniqueness Method by J.-L. Lions in the
case of fractional diffusion-wave equations. The main difficulty is to
determine the right shape for the adjoint system, suitable for the procedure
of HUM.

Keywords: Caputo and Riemann–Liouville fractional derivatives; fractional
diffusion-wave equations; Hilbert Uniqueness Method.

MSC 2010: Primary 35R11; Secondary 47G20.

##  1 Introduction

In this paper we investigate the fractional differential equation

$\sideset{{}^{C}}{{}_{0+}^{\alpha}}{\mathop{D}}u=\Delta u$ (1)

where the symbol $\sideset{{}^{C}}{{}_{0+}^{\alpha}}{\mathop{D}}u$ denotes the
Caputo derivative

$\sideset{{}^{C}}{{}_{0+}^{\alpha}}{\mathop{D}}u(t)=\frac{1}{\Gamma(2-\alpha)}\int_{0}^{t}(t-s)^{1-\alpha}u{{}^{\prime\prime}}(s)\
ds$

defined in [2]. We assume that the order $\alpha$ belongs to $(1,2)$, so (1)
is commonly called fractional diffusion-wave equation. For a comparison
between Caputo derivatives and Riemann-Liouville derivatives see the survey
[12].

In [14] the well-posedness has been established by means of a Fourier series
approach that involves Mittag-Leffler functions. Also in [11] the proof of
trace regularity properties for the weak solutions of (1) is based on the
representation like Fourier expansions. The paper [13] drew attention to the
fact that fractional wave equations rule the propagation of mechanical
diffusive waves in viscoelastic media.

The aim of this paper is to illustrate the Hilbert Uniqueness Method for (1),
introduced by J.-L. Lions in [9]. Several scientists show interest for HUM
since it gives a general procedure to get exact controllability for
distributed systems when the control acts on the boundary of the domain. This
method is based on the idea that the observability of the adjoint problem is
sufficient for the exact controllability of the assigned problem. The
observability is also a necessary condition, for detailed argumentations see
[7].

Our task is to consider the following problem: for a domain $\Omega$ and
$T>0$, we have to control the system

$\left\\{\begin{array}[]{l}\displaystyle\sideset{{}^{C}}{{}_{0+}^{\alpha}}{\mathop{D}}u=\Delta
u\qquad\text{in }\ (0,T)\times\Omega,\\\ \\\ u(0)=u_{t}(0)=0\qquad\text{in }\
\Omega\,,\end{array}\right.$ (2)

by means of a function that acts on a boundary’s subset.

We point out that, due to the nonlocal nature of the problem, HUM allows one
to achieve exact reachability rather than exact controllability.

The main difficulty in carrying out HUM for (2) is to find the right form for
the adjoint system. Precisely, we have to change the Caputo derivative
$\sideset{{}^{C}}{{}_{0+}^{\alpha}}{\mathop{D}}$ into the Riemann-Liouville
derivative $\sideset{}{{}_{T-}^{\alpha}}{\mathop{D}}$ and consider as adjoint
system associated with (2) the following problem

$\begin{cases}\displaystyle\sideset{}{{}_{T-}^{\alpha}}{\mathop{D}}w=\Delta
w\qquad\text{in }\ (0,T)\times\Omega,\\\ \\\ w=0,\qquad\text{on }\
(0,T)\times\partial\Omega,\end{cases}$

with final data

$\sideset{}{{}_{T-}^{\alpha-2}}{\mathop{D}}w(T)=w_{0}\,,\qquad\sideset{}{{}_{T-}^{\alpha-1}}{\mathop{D}}w(T)=w_{1}\qquad\text{in
}\ \Omega\,.$

We remark that the question of the exact reachability is still unanswered,
since inverse and direct inequalities have to be proved.

Another adaptation of HUM to nonlocal problems with regular integral kernels
can be found in [10].

In Section 2 we set up notations and terminology. Section 3 provides a
detailed exposition of the Hilbert Uniqueness Method for system (2).

##  2 Preliminaries

Let $f\in L^{1}(0,T)$ $(T>0)$ be. The following definitions for fractional
integrals and derivatives will be used throughout the paper.

  * –

Riemann–Liouville fractional integrals $I_{0+}^{\alpha}f$ and
$I_{T-}^{\alpha}f$ of order $\alpha>0$:

$I_{0+}^{\alpha}f(t)=\frac{1}{\Gamma(\alpha)}\int_{0}^{t}(t-s)^{\alpha-1}f(s)\
ds\qquad\mbox{a.e.}\ t\in(0,T),$ (3)
$I_{T-}^{\alpha}f(t)=\frac{1}{\Gamma(\alpha)}\int_{t}^{T}(s-t)^{\alpha-1}f(s)\
ds,\qquad\mbox{a.e.}\ t\in(0,T),$ (4)

where $\Gamma(\alpha)=\int_{0}^{\infty}t^{\alpha-1}e^{-t}\ dt$ is the Euler
gamma function.

  * –

Caputo fractional derivative $\sideset{{}^{C}}{{}_{0+}^{\alpha}}{\mathop{D}}f$
of order $\alpha\in(1,2)$ when $f^{\prime}$ is absolutely continuous:

$\sideset{{}^{C}}{{}_{0+}^{\alpha}}{\mathop{D}}f(t)=I_{0+}^{2-\alpha}f{{}^{\prime\prime}}(t)=\frac{1}{\Gamma(2-\alpha)}\int_{0}^{t}(t-s)^{1-\alpha}f{{}^{\prime\prime}}(s)\
ds\,.$ (5)

  * –

Riemann–Liouville fractional derivative
$\sideset{}{{}_{T-}^{\alpha}}{\mathop{D}}f$ of order $\alpha\in(n-1,n)$,
$n=0,1,2$:

$\sideset{}{{}_{T-}^{\alpha}}{\mathop{D}}f(t)=\Big{(}\frac{d}{dt}\Big{)}^{n}I_{T-}^{n-\alpha}f(t)=\frac{1}{\Gamma(n-\alpha)}\Big{(}\frac{d}{dt}\Big{)}^{n}\int_{t}^{T}(s-t)^{n-\alpha-1}f(s)\
ds.$ (6)

In particular, we note that for $-1<\alpha<0$, that is $n=0$, we have

$\sideset{}{{}_{T-}^{\alpha}}{\mathop{D}}f=I_{T-}^{-\alpha}f.$ (7)

We will use later a well-known property (see e.g. [6]):

$\int_{0}^{T}I_{0+}^{\alpha}f(t)g(t)\ dt=\int_{0}^{T}f(t)I_{T-}^{\alpha}g(t)\
dt\,.$ (8)

##  3 Time fractional exact controllability

Our aim is to adapt the Hilbert Uniqueness Method by J.-L. Lions [9] for
fractional differential equations.

Let $T>0$ and $\Omega$ be a bounded domain of class $C^{2}$ in
${\mathbb{R}}^{N}$, $N\geq 1$, with boundary $\partial\Omega$ given by
$\partial\Omega=\partial\Omega_{1}\cup\partial\Omega_{2}$ and
$\partial\Omega_{1}\cap\partial\Omega_{2}=\emptyset$. We consider the
fractional diffusion-wave equation with $\alpha\in(1,2)$:

$\sideset{{}^{C}}{{}_{0+}^{\alpha}}{\mathop{D}}u(t,x)=\Delta
u(t,x)\qquad(t,x)\in(0,T)\times\Omega,$ (9)

with null initial conditions

$u(0,x)=u_{t}(0,x)=0\qquad x\in\Omega\,,$ (10)

and boundary conditions,

$u(t,x)=g(t,x)\quad(t,x)\in(0,T)\times\partial\Omega_{1},\qquad
u(t,x)=0\quad(t,x)\in(0,T)\times\partial\Omega_{2}\,.$ (11)

We give the definition for a reachability problem: fixed $u_{0}\in
L^{2}(\Omega)$ and $u_{1}\in H^{-1}(\Omega)$, we have to determine $g\in
L^{2}(0,T;L^{2}(\partial\Omega_{1}))$ such that the weak solution $u$ of
problem (9)-(11) verifies the final conditions

$u(T,x)=u_{0}(x)\,,\quad u_{t}(T,x)=u_{1}(x)\,,\qquad x\in\Omega\,.$ (12)

One can solve such type of problems by the Hilbert Uniqueness Method. To this
end, we proceed as follows. Given
$(w_{0},w_{1})\in(C^{\infty}_{c}(\Omega))^{2}$, we introduce the adjoint
system of (9), that is

$\begin{cases}\displaystyle\sideset{}{{}_{T-}^{\alpha}}{\mathop{D}}w(t,x)=\Delta
w(t,x)\qquad(t,x)\in(0,T)\times\Omega,\\\ \\\
w(t,x)=0\qquad(t,x)\in(0,T)\times\partial\Omega\,,\end{cases}$ (13)

with final data

$\sideset{}{{}_{T-}^{\alpha-2}}{\mathop{D}}w(T,\cdot)=w_{0}\,,\qquad\sideset{}{{}_{T-}^{\alpha-1}}{\mathop{D}}w(T,\cdot)=w_{1}\,.$
(14)

The above problem is well-posed by means of a spectral representation and the
Mittag-Leffler functions, see e.g. [6, 3, 5, 8, 4].

Thanks to the regularity of the final data, the solution $w$ of (13)–(14) is
regular enough to take in (11) $g=\partial_{\nu}w$. So the nonhomogeneous
problem (9)-(11) can be written in the form

$\left\\{\begin{array}[]{l}\displaystyle\sideset{{}^{C}}{{}_{0+}^{\alpha}}{\mathop{D}}u(t,x)=\Delta
u(t,x)\qquad(t,x)\in(0,T)\times\Omega,\\\ \\\ u(0,x)=u_{t}(0,x)=0\qquad
x\in\Omega\,,\\\ \displaystyle u(t,x)=\partial_{\nu}w(t,x),\ \
(t,x)\in(0,T)\times\partial\Omega_{1},\quad u(t,x)=0\ \
(t,x)\in(0,T)\times\partial\Omega_{2}\,.\end{array}\right.$ (15)

As in the non integral case, it can be proved that problem (15) admits a
unique solution $u$. We can introduce a linear operator defined as

$\Psi(w_{0},w_{1})=\big{(}-u_{t}(T,\cdot),u(T,\cdot)\big{)},\qquad(w_{0},w_{1})\in(C^{\infty}_{c}(\Omega))^{2}\,.$
(16)

We will prove that

$\langle\Psi(w_{0},w_{1}),(w_{0},w_{1})\rangle_{L^{2}(\Omega)}=\int_{0}^{T}\int_{\partial\Omega_{1}}\big{|}\partial_{\nu}w(t,x)\big{|}^{2}\
dxdt\,.$ (17)

To this end, we multiply the equation in (15) by the solution $w(t,x)$ of the
adjoint system (13)-(14) and integrate on $(0,T)\times\Omega$, that is

$\int_{0}^{T}\int_{\Omega}\sideset{{}^{C}}{{}_{0+}^{\alpha}}{\mathop{D}}u(t,x)w(t,x)\
dxdt=\int_{0}^{T}\int_{\Omega}\Delta u(t,x)w(t,x)\ dxdt\,.$ (18)

If we apply (8), then we obtain for any $x\in\Omega$

$\int_{0}^{T}\sideset{{}^{C}}{{}_{0+}^{\alpha}}{\mathop{D}}u(t,x)w(t,x)\
dt=\int_{0}^{T}I_{0+}^{2-\alpha}u_{tt}(t,x)w(t,x)\
dt=\int_{0}^{T}u_{tt}(t,x)I_{T-}^{2-\alpha}w(t,x)\ dt\,.$ (19)

Integrating twice by parts and taking into account that
$u(0,\cdot)=u_{t}(0,\cdot)=0$, we have

$\int_{0}^{T}u_{tt}(t,x)I_{T-}^{2-\alpha}w(t,x)dt=u_{t}(T,x)I_{T-}^{2-\alpha}w(T,x)-\int_{0}^{T}u_{t}(t,x)\frac{\partial}{\partial
t}I_{T-}^{2-\alpha}w(t,x)dt\\\
=u_{t}(T,x)I_{T-}^{2-\alpha}w(T,x)-u(T,x)\frac{\partial}{\partial
t}I_{T-}^{2-\alpha}w(T,x)+\int_{0}^{T}u(t,x)\sideset{}{{}_{T-}^{\alpha}}{\mathop{D}}w(t,x)\
dt\,.$ (20)

Since $-1<\alpha-2<0$, we can use (7) to get

$I_{T-}^{2-\alpha}w(T,x)=\sideset{}{{}_{T-}^{\alpha-2}}{\mathop{D}}w(T,x).$

Moreover, $0<\alpha-1<1$, so by (6) for $n=1$ we have

$\frac{\partial}{\partial
t}I_{T-}^{2-\alpha}w(T,x)=\sideset{}{{}_{T-}^{\alpha-1}}{\mathop{D}}w(T,x)\,.$

Combining (19) with (20) yields

$\int_{0}^{T}\sideset{{}^{C}}{{}_{0+}^{\alpha}}{\mathop{D}}u(t,x)w(t,x)\ dt\\\
=u_{t}(T,x)\sideset{}{{}_{T-}^{\alpha-2}}{\mathop{D}}w(T,x)-u(T,x)\sideset{}{{}_{T-}^{\alpha-1}}{\mathop{D}}w(T,x)+\int_{0}^{T}u(t,x)\sideset{}{{}_{T-}^{\alpha}}{\mathop{D}}w(t,x)\
dt\,.$ (21)

On the other hand, integrating twice by parts with respect to the variable $x$
we have

$\int_{\Omega}\Delta u(t,x)w(t,x)dx=\int_{\Omega}u(t,x)\triangle w(t,x)\
dx-\int_{\partial\Omega}u(t,x)\partial_{\nu}w(t,x)\ dx\,.$ (22)

Putting (21) and (22) into (18) we get

$\int_{0}^{T}\int_{\Omega}u(t,x)\sideset{}{{}_{T-}^{\alpha}}{\mathop{D}}w(t,x)\
dxdt+\int_{\Omega}u_{t}(T,x)\sideset{}{{}_{T-}^{\alpha-2}}{\mathop{D}}w(T,x)dx-\int_{\Omega}u(T,x)\sideset{}{{}_{T-}^{\alpha-1}}{\mathop{D}}w(T,x)dx\\\
=\int_{0}^{T}\int_{\Omega}u(t,x)\triangle w(t,x)\
dxdt-\int_{0}^{T}\int_{\partial\Omega}u(t,x)\partial_{\nu}w(t,x)\ dxdt\,.$

We recall that $w$ is the solution of the adjont problem (13): keeping
$\sideset{}{{}_{T-}^{\alpha}}{\mathop{D}}w=\Delta w$ in mind, from the above
equation we have

$\int_{\Omega}u_{t}(T,x)\sideset{}{{}_{T-}^{\alpha-2}}{\mathop{D}}w(T,x)dx-\int_{\Omega}u(T,x)\sideset{}{{}_{T-}^{\alpha-1}}{\mathop{D}}w(T,x)dx=-\int_{0}^{T}\int_{\partial\Omega}u(t,x)\partial_{\nu}w(t,x)\
dxdt\,.$

By the final data (14) of $w$ and the boundary conditions satisfied by $u$,
see (15), we deduce that

$\langle\Psi(w_{0},w_{1}),(w_{0},w_{1})\rangle_{L^{2}(\Omega)}=\int_{\Omega}u(T,x)w_{1}dx-\int_{\Omega}u_{t}(T,x)w_{0}dx=\int_{0}^{T}\int_{\partial\Omega_{1}}\big{|}\partial_{\nu}w(t,x)\big{|}^{2}\
dxdt\,,$

hence (17).

Define a semi-norm on the space $\big{(}C^{\infty}_{c}(\Omega)\big{)}^{2}$:
for $(w_{0},w_{1})\in\big{(}C^{\infty}_{c}(\Omega)\big{)}^{2}$ we set

$\|(w_{0},w_{1})\|^{2}_{F}:=\int_{0}^{T}\int_{\partial\Omega_{1}}\big{|}\partial_{\nu}w(t,x)\big{|}^{2}\
dxdt\,.$ (23)

We observe that $\|\cdot\|_{F}$ is a norm if and only if the following
uniqueness theorem holds.

###### Theorem 3.1

If $w$ is the solution of problem (13)–(14) such that

$\partial_{\nu}w(t,x)=0\,,\qquad\forall(t,x)\in[0,T]\times\partial\Omega_{1}\,,$

then

$w(t,x)=0\qquad\forall(t,x)\in[0,T]\times\Omega\,.$

If theorem 3.1 holds true, then we can define the Hilbert space $F$ as the
completion of $\big{(}C^{\infty}_{c}(\Omega)\big{)}^{2}$ for the norm (23).
Moreover, the operator $\Psi$ extends uniquely to a continuous operator,
denoted again by $\Psi$, from $F$ to the dual space $F^{\prime}$ in such a way
that $\Psi:F\to F^{\prime}$ is an isomorphism.

In conclusion, if we prove the uniqueness result given by theorem 3.1 and

$F=H^{1}_{0}(\Omega)\times L^{2}(\Omega)$

with the equivalence of the respective norms, then we can solve the
reachability problem (9)–(12) taking $(u_{0},u_{1})\in L^{2}(\Omega)\times
H^{-1}(\Omega)$.

## References

  * [1]
  * [2] M. Caputo, Linear model of dissipation whose Q is almost frequency independent-II, in Geophysical Journal International, 13 (1967) 529–539.
  * [3] N. Heymans, I. Podlubny, Physical interpretation of initial conditions for fractional differential equations with Riemann-Liouville fractional derivatives. Rheologica Acta, 45 (2006), 765–771.
  * [4] S. Hristova, R. Agarwal, D. O’Regan, Explicit solutions of initial value problems for systems of linear Riemann-Liouville fractional differential equations with constant delay. Adv. Difference Equ. (2020), Paper No. 180, 18 pp.
  * [5] D. Idczak, R. Kamocki, On the existence and uniqueness and formula for the solution of R-L fractional Cauchy problem in ${\mathbb{R}}^{n}$. Fract. Calc. Appl. Anal. 14 (2011), 538–553.
  * [6] A.A. Kilbas, H.M. Srivastava, J.J. Trujillo, _Theory and Applications of Fractional Differential Equations_. Elsevier, Amsterdam (2006).
  * [7] V. Komornik, Exact controllability and stabilization. The multiplier method, RAM: Research in Applied Mathematics, Masson, Paris; John Wiley and Sons, Ltd., Chichester, 1994.
  * [8] K. Li, J. Peng, J. Jia, Cauchy problems for fractional differential equations with Riemann-Liouville fractional derivatives. J. Funct. Anal. 263 (2012), 476–510.
  * [9] J.-L. Lions, Contrôlabilité exacte, perturbations et stabilisation de systèmes distribués I-II, Recherches en Mathématiques Appliquées, 8-9 (1988), Masson, Paris.
  * [10] P. Loreti, D. Sforza, Reachability problems for a class of integro-differential equations, J. Differential Equations 248 (2010), 1711–1755.
  * [11] P. Loreti, D. Sforza, Fractional diffusion-wave equations: hidden regularity for weak solutions. Fract. Calc. Appl. Anal. 24 (2021), 1015–1034.
  * [12] F. Mainardi, R. Gorenflo, Time-fractional derivatives in relaxation processes: a tutorial survey. Fract. Calc. Appl. Anal. 10 (2007), 269–308.
  * [13] F. Mainardi, Fractional diffusive waves in viscoelastic solids, in: J.L. Wegner, F.R. Norwood (Eds.), Nonlinear Waves in Solids, ASME/AMR, Fairfield (1995) 93–97.
  * [14] K. Sakamoto, M. Yamamoto, Initial value/boundary value problems for fractional diffusion-wave equations and applications to some inverse problems. _J. Math. Anal. Appl._ 382, No 1 (2011), 426–447.


Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
10.1109/ACCESS.2017.DOI

Corresponding author: Yongseob Lim (e-mail<EMAIL_ADDRESS>and Gyeungho
Choi (e-mail: ghchoi@dgist.ac.kr).
*JINU PAHK and JUNGSEOK SHIM are co-first authors.

# Effects of Sim2Real Image Translation on Lane Keeping Assist System in CARLA
Simulator

JINU PAHK*1  JUNGSEOK SHIM*1, MINHYEOK BAEK1, YOUNGSEOB LIM2, AND GYEUNGHO
CHOI3  Daegu Gyeongbuk Institute of Science and Technology, Daegu 42988, South
Korea (e-mail: kyg132, shim2751<EMAIL_ADDRESS> Department of
Robotics Engineering, Daegu Gyeongbuk Institute of Science and Technology,
Daegu 42988, South Korea  Department of Interdisciplinary Engineering, Daegu
Gyeognbuk Institute of Science and Technology, Daegu 42988, South Korea

###### Abstract

Autonomous vehicle simulation has the advantage of testing algorithms in
various environment variables and scenarios without wasting time and
resources, however, there is a visual gap with the real-world. In this paper,
we trained DCLGAN to realistically convert the image of the CARLA simulator
and evaluated the effect of the Sim2Real conversion focusing on the LKAS (Lane
Keeping Assist System) algorithm. In order to avoid the case where the lane is
translated distortedly by DCLGAN, we found the optimal training hyperparameter
using FSIM (feature-similarity). After training, we built a system that
connected the DCLGAN model with CARLA and AV in real-time. Then, we collected
data (e.g. images, GPS) and analyzed them using the following four methods.
First, image reality was measured with FID, which we verified quantitatively
reflects the lane characteristics. CARLA images that passed through DCLGAN had
smaller FID values than the original images. Second, lane segmentation
accuracy through ENet-SAD was improved by DCLGAN. Third, in the curved route,
the case of using DCLGAN drove closer to the center of the lane and had a high
success rate. Lastly, in the straight route, DCLGAN improved lane restoring
ability after deviating from the center of the lane as much as in reality.

###### Index Terms:

Intelligent vehicles, Vehicle driving, Autonomous vehicles, Lane Keeping
Assist System, Lane detection, GAN, DCLGAN, FID, Autonomous vehicle
simulation, CARLA, Software-in-the-loop

=-15pt

##  I Introduction

Currently, the autonomous driving industry is expanding at a rapid pace.
Autonomous driving has been proposed as a solution to traffic accidents that
cause 1.25 million deaths worldwide every year [1]. However, in the United
States alone, a total of 392 traffic accidents occurred from July 2021 to May
2022 due to Level 2 autonomous driving, and these reports raised the need for
securing the safety of autonomous driving technology [2].
Autonomous driving simulation is drawing attention as a new testbed. Unlike
reality, the simulation can implement complex situations, such as traffic
accidents and specific weather, that are hard to reproduce in the real-world
[3, 4, 5]. This technology gives train datasets about abnormal situations and
will prevent traffic accidents more perfectly. Therefore, many studies have
been conducted for modeling vehicles in simulation [6, 7, 8].
[!t](topskip=0pt, botskip=0pt, midskip=0pt)[height=1.75in]fig1.png Full
framework for identifying Sim2Real image gaps and driving gaps from the CARLA-
AV-GAN loop.
In order to adapt proven algorithms from virtual-world to real-world, reducing
the gap between simulation and reality is an inevitable task. This gap is
caused by several factors such as the unreality of sensor models (e.g.,
camera, LiDAR, GPS, and IMU), mismatch of dynamic models, and noise in the
real environment. Among the factors, the camera sensor’s input images are the
main cause of the gap because this sensor accounts for the largest portion of
driving. The image’s gap which is a difference in visual realism between the
real domain and the simulation domain would make driving results unreliable.
Moreover, correcting this gap only by adjusting the simulation parameters is
impossible. Therefore, simulation developers and users often have used a high-
level graphic engine or imported a deep learning model to make the visual
contents of the simulation look more realistic.
To make images from the two domains similar, Generative Adversarial Networks
(GAN) [9] has been adopted. With the development of the GAN structure, the
model has begun to be applied to various fields such as colorizing gray image
[10] and restoring image quality [11]. Moreover, many studies have attempted
to make virtual world images similar to reality [12, 13]. Although the study
on improving the graphic quality of the virtual world is active, autonomous
driving tests with these photorealistic techniques have not been dealt with
yet.
The purpose of this paper is to show the possibility of applying GAN to the
autonomous vehicle (AV) simulation not just translating images, but also
driving tests. We used CARLA [14] for AV simulation. Dual Contrastive Learning
Generative Adversarial Networks (DCLGAN) [15] was applied because DCLGAN
showed the better performance to translate lane geography characteristics than
CycleGAN [16] without mode collapse. Moreover, we explored a way to train
DCLGAN using FSIM to prevent lane distortion.
For this purpose, the data were collected from the three domains: real-world,
CARLA with DCLGAN (hereafter referred to as CwD), and CARLA without DCLGAN
(hereafter referred to as Cw/oD). The real-world data were obtained from the
Korea Intelligent Automotive Parts Promotion Institute (KIAPI) and the
virtual-world data were obtained from a map the same as KIAPI. The types of
data were camera images and GPS data. To show the DCLGAN contributed to making
more realistic driving results, a method to quantify the Sim2Real gap is
needed. We propose the following four methods: First, FID(Fréchet Inception
distance) [17] measures the reality of the virtual-world’s images by
calculating differences in density of two distributions of imageset. Second,
lane segmentation performance was measured and compared. Third, we calculated
the driving orbital difference between the real-world and the simulation
with/without DCLGAN by using GPS data. Finally, we measured how well the
vehicle returns to its original lane when it is out of the center.
Experimental results show that the performance of our proposed CwD method is
more effective in practical use than without DCLGAN.

##  II Related Works

###  II-A CARLA

CARLA is an autonomous driving research simulator. We used the CARLA simulator
for the implementation of a vehicle and conducting environments by using the
following functions. First, ROS-bridge was provided. ROS-bridge enables the
connection between real-vehicle and virtual-vehicle. Second, CARLA allowed us
to spawn various sensors and configure specific parameters such as the
camera’s FoV, resolution, and gamma. Lastly, custom vehicles can be
implemented through Unreal Engine 4 (UE4), the game engine of CARLA. We
imported a 3D model of the same size as the Hyundai IONIQ Electric for spawn
sensors in the same position as reality.

###  II-B Generative Adversarial Network

Generative Adversarial Network consists of a generator that creates fake
images following real images’ distribution, and a discriminator that
distinguishes between the fake images and the real images. As the two modules
compete repeatedly, GAN can generate more realistic images. However, the GAN
model has a limitation that it necessarily requires paired datasets for
learning. CycleGAN solves the problem by adding the reverse mapping that
returns the fake image to the original image. Whereas cycle consistency loss
has a drawback that the fake images have low diversity. Contrastive Learning
for Unpaired Image-to-Image Translation (CUT) and Dual Contrastive Learning
GAN tried to deal with this issue through continuous learning. Especially,
Similarity Dual Contrastive Learning GAN (SIMDCL) with similarity loss added
to DCLGAN suffers less from the mode collapse by allocating embeddings as much
as the number of data domains. We mainly used SIMDCL, considering the
characteristics of our dataset, which has low diversity and a high risk of
mode collapse.

###  II-C Fréchet Inception Distance

FID is an evaluation metric for a generative model such as Generative
Adversarial Networks (GAN), text-to-image, etc. The distance between the fake
images and the real images is measured by comparing the extracted features
from the pre-trained Inception v3 model with the Imagenet dataset. In this
paper, the measured distance presented how realistic the generative model
creates an image. FID is calculated as (1) using the mean of the variance of
features ($\mu$) and covariance($\sum$).

$FID=\left|\mu_{T}-\mu_{G}\right|^{2}+Tr(\sum_{T}+\sum_{F}-2(\sum_{T}\sum_{G})^{\frac{1}{2}}).$
(1)

###  II-D ENet-SAD

ENet-SAD [18] has the characteristic of self-learning, which delivers and
enhances the representation top-down by distilling the attention map of the
lower layer. Thanks to this distillation technique, Enet-SAD can be computed
faster without additional annotation or inference even with much fewer
parameters compared to other CNN-based lane detection models such as SCNN.
Since the IoU of the ground truth lane and lane segmentation prediction
results were better than the previous model, our LKAS system uses ENet-SAD.

###  II-E TuSimple Evaluation

Tusimple [19] introduces the lane segmentation accuracy formula as (2). Cclip
is the number of true points in each clip and Sclip is the number of the
ground truth points in the clip. We used this evaluation method to compare
each domain’s accuracy.

$Accuracy=\frac{\sum_{clip}C_{clip}}{\sum_{clip}S_{clip}}.$ (2)

##  III Environment Setting

###  III-A Real-world Setting

####  III-A1 Specification

TABLE I describes the sensors and hardware’s specifications. Fig. III-A1 shows
the location of the sensor. Although the camera’s maximum resolution is
1632x1248 pixels, 808x620 was used by binning. The vehicle was the Hyundai
IONIQ Electric modified for applying the drive-by-wire system.[h](topskip=0pt,
botskip=0pt, midskip=0pt)[height=1in]fig2.pngVehicle top view and sensor
location (left). Sensor installation on the test vehicle (right).

Hardware Component | Purpose | Model | Specifications
---|---|---|---
Main PC |  Vehicle Control |  Nuvo -8108GC -NX |  CPU: Intel Xeon Processor E-2176G Memory: 32GB DDR4 GPU: GeForce RTX-2080Ti 11GB
Deep Learning PC |  Running deep learning algorithm |  Custom PC |  CPU: Intel I7 9700F Memory: 32GB DDR4 GPU: NVIDIA Titan X 12GB
Simulation PC |  Playing CARLA Calculating GAN model |  Custom PC |  CPU: AMD Ryzen 9 3950x Memory: 64GB DDR4 GPU: GeForce RTX-3090 24GB
FLIR Camera |  Lane recognition |  BFS-PGE -19S4C-C |  FoV: 76∘ FPS: 50 Gamma: 0.8 Resolution: 1632x1248 (808x620 after binning)
GPS (RTK GNSS) |  Checking the vehicle trajectory | MRP-2000 |  RTK Accuracy: 0.010m + 1ppm CEP Rate: up to 10Hz
TABLE I: Sensors and Hardware’s specification.

####  III-A2 LKAS algorithm

We have Autonomous driving algorithms and architecture that were licensed by
MOLIT (Ministry of Land, Infrastructure, and Transport) of the Republic of
Korea. This license required Autonomous Emergency Braking, Smart Cruise
Control, and Lane Keeping Assist (LKA). Among the technologies we have, only
LKA was used in the experiment.
Previous paper[21, 22] proposed an lane keeping algorithm. This algorithm
worked in two stages. The first stage was lane segmentation. In Deep Learning
PC, Enet-SAD was used to segment the lane. This Enet-SAD was trained by
datasets from the CULANE Dataset[23] and our laboratory. The second stage was
lane detection. The lane detection transformed the segmented image to a bird’s
eye view, then calculated lane center points. After these stages, Main PC
adjusts steer and velocity by using pure pursuit through the result of lane
detection. In this paper, lane keeping was performed in KIAPI and compared
with the data from the actual vehicle and the virtual vehicle.

####  III-A3 Proving Ground

The real-world test data was collected at the Korea Intelligent Automotive
parts Promotion Institute (KIAPI), located in Daegu, Korea. Among the various
test circuits, the high-speed circuit was used. The high-speed circuit is a
telephone receiver-shaped and consists of a three-lane straight line and bank.
Fig. III-A3 shows the actual appearance of the facility. [h](topskip=0pt,
botskip=0pt, midskip=0pt)[width=3.3in]fig3.pngKIAPI satellite picture.

###  III-B CARLA Setting

####  III-B1 Camera and Vehicle

CARLA provides various sensors and attributes. Attributes make it possible to
set the sensor’s specifications. The performance of the camera mentioned in
TABLE I was imported such as FoV or Gamma. Then, the camera was spawned
according to Fig. III-A1 position. The CARLA image received from the imported
camera and the real-world image can be seen in Fig. III-B1. [h](topskip=0pt,
botskip=0pt, midskip=0pt)[width=3.3in]fig4.pngCARLA image (left) and Real-
world image (right).

####  III-B2 Map

To make a custom map, CARLA needed map geometry information in .fbx format and
OpenDRIVE information in .xodr format. For these, We used RoadRunner
recommended by CARLA as a map editor and OpenStreetMap (OSM) as map data. OSM
has latitude and longitude as vector data and RoadRunner made OSM possible to
be map geometry information and OpenDRIVE information.

Parameters |  Specification
---|---
Road Width (m) |  3.5
Line Length (m) |  4.5
Line Spacing (m) |  4
Line Width (m) |  0.125
Slope (∘) |  30 (first lane) 0 (second, third lane)
TABLE II: Road parameters’ specification.

The process of making the KIAPI map was as follows. First, as can be seen in
Fig. II(a) we dragged the desired area from the satellite map in OSM and
export the file in .osm format. Second, as shown in Fig. II(b), the latitude
and longitude were corrected by comparing the osm file exported from JOSM with
the GPS file of actual measurement data. After that, the calibrated osm file
was imported into RoadRunner, and the RoadRunner file set the road width,
lanes, and slope of the road according to the information announced in KIAPI
[24]. Fig. II(c) describes this process and TABLE II shows the specification
of each parameter. Lastly, this file was exported in Carla format such as fbx
and xodr format files.

[h](topskip=0pt, botskip=0pt, midskip=0pt)[width=2.0in]fig5.pngThe process of
creating a virtual map from the OSM data of KIAPI. (a) Exporting map data from
OSM. (b) PG osm file edited on JOSM. (c) KIAPI map produced in RoadRunner.

####  III-B3 Time

The experiment with the same time period as the real-world is impossible
because of the calculation time of GAN. As a solution, we used synchronous
time. According to CARLA documentation, “On synchronous mode, the server waits
for a client tick, a ”ready to go” message, before updating to the following
simulation step” [25, Sec.2]. It means that We can set the attribute to wait
for a step until the vehicle control command arrived. Therefore, the Client
server would wait for a cycle described in 3.B.3. section.

####  III-B4 Network Connection between CARLA and AV

Three computers were used for the experiments, each called the Main PC, Deep
Learning PC (DL PC), and Simulation PC. TABLE I shows specification. Main PC
decided on vehicle control such as steering value and accelerator value. DL PC
computed deep learning algorithms such as the LKA algorithm. Simulation PC
played CARLA simulation. 10 Gbps cables were used to connect three PCs. We
selected ROS as a middleware. TABLE III describes the ROS version for each
computer. For connecting different versions of ROS 2 between DL PC and
Simulation PC, Cyclone DDS is applied to Simulation PC and Fast RTPS is used
to DL PC.

PC Name |  Ubuntu Ver. |  ROS Ver.
---|---|---
Main PC |  18.04 LTS |  Melodic
Deep Learning PC |  18.04 LTS |  Dashing
Simulation PC |  20.04 LTS |  Foxy
TABLE III: Each PC’s Ubuntu version and ROS version.

Fig. III shows flow of the three computers interactions. The title of the box
means the topic name. First, in the Simulation PC, the camera sensor from
CARLA gave an image to GAN. The output from the GAN model was sent to the DL
PC. Second, in the DL PC, lane_segmentation function was performed and gave
segmentation data to lane_detect function. The lane_detect function found
center points of the lane and sent the output points to the Main PC. Third, in
the Main PC, lane_driving function got vehicle status from the Simulation PC
and center points from DL PC. By utilizing these values, the lane_driving
function determined desired steer value. move_car function calculates proper
steer, acceleration, and brake values. Then the function sent them to the
Simulation PC for control of the vehicle. [h!](topskip=0pt, botskip=0pt,
midskip=0pt)[width=3.3in]fig6.png CARLA-AV-GAN Connection overview.

##  IV Approach Method

###  IV-A Lane Characteristics Difference Measurement using FID

It is necessary to guarantee whether the difference in characteristics of
lanes can be quantitatively measured. As shown in Fig. III-B1, the lanes in
the real-world are clearer and thicker. To check whether the FID used to
quantify the Sim2Real difference in this paper can identify the difference
between these lanes, we made several maps. These maps were based on KIAPI map
and varied the thickness, texture, length, and distance of lanes using
RoadRunner. Fig. IV-A shows these characteristic differences. For the texture,
LaneMarking1, 2, and 3, which are lane materials provided by RoadRunner, were
used. The material used in LaneMarking1 is the sharpest and 3 is the faintest.
All other variables in RoadRunner were fixed except for lanes.
After that, the vehicle was summoned using the RoadRunner Scenario function
and then driven on the KIAPI map. While the car was driving in the center of
the road, 25-second videos were recorded on each map with specific
characteristics of the lane. By extracting each frame of the recorded video,
about 700 imageset was created for each map, and FID operation was performed
between the maps using this. At this time, the FID code was written based on
PyTorch and the dimensionality of features was set to 2048 (default) [26].
To compare only the thickness of lanes, we set the lane material to
LaneMarking1, set the length and spacing of lanes to (10m, 10m), and increased
the lane thickness from 0.125m to 0.15m, 0.175m, 0.2m. While going, the FID
between each other was measured. Also, to compare the texture, the thickness
was fixed to 0.125m and the length and spacing of the lanes were fixed to
(10m, 10m). To compare the length and spacing of the lanes, the thickness was
fixed to 0.125m and the material to LaneMarking1.

[h](topskip=0pt, botskip=0pt, midskip=0pt)[width=3.3in]fig7.png Front camera
image in RoadRunner Simulation Play according to (a) the thickness of lanes,
(b) the material of lane texture, (c) the length of the lane and the distance
between lanes.

TABLE IV shows the FID measurements between each map. The fact that the value
increases toward the upper right of the table mean that the FID actually
appears larger as the characteristics of the lanes are quantitatively farther
from each other. For example, the thickness change result (TABLE 3 (a)) shows
that the FID is smaller when the thickness of the lane is quantitatively
close, and vice versa. When compared with the image of the lane with a
thickness of 0.125 m, it can be seen that the FID increases from 3.4 (0.15 m)
to 7.6 (0.175m) to 11.2 (0.2 m). For the texture of the lane (TABLE 3(b)), the
imageset with the clearest lane (LaneMarking1) was farther from the imageset
with the most blurry lane (LaneMarking3) than the distance from the imageset
with the second clearest lane (LaneMarking2). (85.1 vs 31.6) The same results
(TABLE 3(c)) were obtained for the length and spacing of the lanes. This is
thought to be because the Inception v3 model abstracts the characteristics
such as thickness, texture, length, and spacing of lanes well and extracts
them as features. This tendency confirms that FID is suitable as a numerical
value to quantitatively measure the difference in characteristics of the
lanes.

Thickness | 0.125m | 0.15m | 0.175m | 0.2m
---|---|---|---|---
0.125m | 0 | 3.36 | 7.57 | 11.17
0.15m |  | 0 | 4.18 | 6.71
0.175m |  |  | 0 | 3.12
0.2m |  |  |  | 0

(a)

Texture |  Lane Marking1 |  Lane Marking2 |  Lane Marking3
---|---|---|---
LaneMarking1 | 0 | 31.60 | 85.14
LaneMarking2 |  | 0 | 44.05
LaneMarking3 |  |  | 0

(b)

Length, Spacing | (10m, 10m) | (5m, 5m) | (3m, 3m)
---|---|---|---
(10m, 10m) | 0 | 15.54 | 31.91
(5m, 5m) |  | 0 | 11.09
(3m, 3m) |  |  | 0

(c)

TABLE IV: FID measurement value between imagesets with (a) different
thicknesses of lanes, (b) the material of texture of lanes (c) the lane length
and space between lanes.

###  IV-B Training DCLGAN

####  IV-B1 Training Method

We tried to reduce this Sim2Real gap measured by FID with DCLGAN. The data of
the real-world domain was collected from the high-speed circuit of KIAPI and
the data of the CARLA domain was collected from the OSM map. Both were
collected while driving at a speed of 30 km/h in the second lane.
The CARLA domain contains 5498 images as trainsets and 611 images as testsets,
and the real-world domain contains 6929 images as train set and 770 images as
the test set. In order to match the channels of the FLIR camera images used in
the AV, the order of the CARLA image channels was changed from RGB to BGR.
The images used for training were loaded with a size of 600x600 and cropped to
a size of 300x300. The model was selected as SIMDCL and the rest of the
options kept the DCLGAN default settings. Additionally, $\lambda_{NCE,X}$, a
hyperparameter of the loss function, was determined through the process of 2).

####  IV-B2 Determination of Hyperparameter using FSIM

As seen in Fig. IV-B2, if the image passes through the DCLGAN model, the lane
in CARLA may not maintain its original shape and may be distorted enough to
affect lane-keeping driving. This lane distortion occurs when the generator of
the GAN does not fully learn the data distribution and is fixed only in a
specific biased direction to generate an image[15]. Also, the imbalance
between the straight line data and curve data of the lane can be the cause.

[h](topskip=0pt, botskip=0pt, midskip=0pt)[width=1.5in]fig8.png Mode collapse
occurred when training with the option of DCLGAN model as default. Lane is
distorted and all images are similar.

To solve the problem of distortion of the suboptimal shape, we adjusted the
coefficients of the loss function of the generative model. (3) shows the loss
function of DCLGAN, and (4) shows the loss function of SimDCL [11]. As in (5),
the X part of the PatchNCEloss term is associated with a generator that
converts X(CARLA)→Y(Real-world). PatchNCELoss calculates Noise Contrastive
Estimation(NCE) Loss by receiving the positive pair patch corresponding to
query, negative patches, and $H^{l}_{Y}(F^{l}_{enc}(G(x)))$, the $G(x)$ re-
embedded with $F:Y{\rightarrow}X$. We checked whether better results can be
obtained by changing the coefficient such as $\lambda_{NCE,X}$ when the
direction in the direction or shape of a lane can be observed.

$\displaystyle L_{DCLGAN}(G,F,D_{X},D_{Y},H_{X},H_{Y})$ $\displaystyle=$
$\displaystyle\lambda_{GAN}(L_{GAN}(G,D_{Y},X,Y)$ $\displaystyle+$
$\displaystyle L_{GAN}(F,D_{X},X,Y)$ $\displaystyle+$
$\displaystyle\lambda_{NCE}L_{PatchNCE_{X}}(G,H_{X},H_{Y},X)$ $\displaystyle+$
$\displaystyle\lambda_{NCE}L_{PatchNCE_{Y}}(F,H_{X},H_{Y},Y)$ $\displaystyle+$
$\displaystyle\lambda_{idt}L_{identity}(G,F).$ (3) $\displaystyle
L_{SimDCL}(G,F,D_{X},D_{Y},H_{X},H_{Y})$ $\displaystyle=$
$\displaystyle\lambda_{GAN}(L_{GAN}(G,D_{Y},X,Y)$ $\displaystyle+$
$\displaystyle L_{GAN}(F,D_{X},X,Y)$ $\displaystyle+$
$\displaystyle\lambda_{NCE}L_{PatchNCE_{X}}(G,H_{X},H_{Y},X)$ $\displaystyle+$
$\displaystyle\lambda_{NCE}L_{PatchNCE_{Y}}(F,H_{X},H_{Y},Y)$ $\displaystyle+$
$\displaystyle\lambda_{sim}L_{sim}(G,F,H_{X},H_{Y},H_{1},H_{2},H_{3},H_{4})$
$\displaystyle+$ $\displaystyle\lambda_{idt}L_{identity}(G,F).$ (4)
$\displaystyle L_{PatchNCE_{X}}(G,H_{X},H_{Y},X)$ $\displaystyle=$
$\displaystyle E_{x\sim
X}\sum^{L}_{l=1}\sum^{S_{l}}_{s=1}l(\widehat{z^{s}_{l}},z^{s}_{l},z^{S\setminus
s}_{l})$ (5)

[b](topskip=0pt, botskip=0pt, midskip=0pt)[width=2.5in]fig9.png A Cw/oD image
cropped to 808x245 size and a CwD image generated therefrom for FSIM
measurement.

In this case, FSIM [18] was used to evaluate the degree of lane distortion of
the generated image in the case of each adjusted hyperparameter. The edge map
similarity between the original image and the fake image is measured from the
phase congruency map extracted based on frequency information and the gradient
magnitude map extracted based on the gradient of pixel values. To compare the
edge information of only lanes, as shown in Fig. IV-B2, 808x245 sized images
excluding the background and car bonnet were cropped from each 808x620 image
of the CARLA test imageset, and the FSIM of each pair of the Cw/oD image and
the CwD image was measured. A model with proper $\lambda_{NCE,X}$ that
maintains the original lane shape or direction well will have a high FSIM
value. The weight determined in this way is then used for FID measurement. The
relationship and differences between FSIM and FID are shown in Fig. V. FSIM
can directly measure the similarity between two paired images, but since FID
compares the distribution of images, it is necessary to compare imagesets when
measuring FID.

TABLE V shows the average FSIM value in each $\lambda_{NCE,X}$. In all cases,
the weight that seems to have the least lane distortion among the weights
trained more than 15 times was selected, and 611 images of the test set of the
CARLA domain were used. For FSIM measurements, quality_metrics. FSIM of the
image_similarity_measures Python package was used [18]. $\lambda_{NCE,X}$
value 3, which has the highest FSIM value 0.451 and is judged to have the
least distortion, was determined as the optimal condition.

$\lambda_{NCE,X}$ | 1 | 2 | 3 | 4
---|---|---|---|---
FSIM | 0.439 | 0.423 | 0.451 | 0.403
TABLE V: Mean FSIM values (611 pairs of paired Cw/oD images & CwD images) for
each $\lambda_{NCE,X}$.

[h](topskip=0pt, botskip=0pt, midskip=0pt)[width=3.3in]fig10.png A diagram of
the relationship between Cw/oD images, CwD images, and real-world images with
measuring FSIM and FID from them. When a CwD image is generated from each
CARLA image, the FSIM is measured from each pair. FID is measured between
unpaired CwD imageset and real-world imageset to measure how realistic the
generated images are.

###  IV-C Extracting Ground Truth from Lane Segmentation Images

To obtain CARLA’s ground truth data automatically, we performed several works.
Fig. IV-C shows this process. First, we created the same map except for the
shape of the line. This map’s line was fully connected shown as Fig. IV-C(a).
Then, the vehicle ran in the line-connected map while recording its movement
and collecting segmentation images. Second, the recording was turned on in the
original map, and saved the test images. Then, We found the specific pixel
color belonging to the line and counted the number of points for each h_sample
which is the height value that lanes can exist. Lastly, line order was
designated.
This algorithm was implemented as described in Algorithm 1 only for the second
road. When searching based on height, the number of possible points in one
height is 2, 3, and 4 [t!](topskip=0pt, botskip=0pt,
midskip=0pt)[width=3.3in]fig11.pngThe segmentation image from line connected
map (a), extracting the specific pixel color (b), and ground truth points (b).
Red lines in (b) show the example of each case for Algorithm 1.

Algorithm 1 Automatically Extracting Ground Truth

Input Segmentation Images
Output Ground Truth Data

1:$h\\_sample\leftarrow$ the list of height values that lines can exist

2:$line\\_rgb\leftarrow$ the RGB value of segmented line

3:$w\leftarrow$ the width of images

4:$GT\leftarrow$ the 2D array of ground truth data. Row corresponds to
h_sample and column corresponds to line number

5:for $i=h\\_sample$ do

6: for $j=1,2,...,w$ do

7: Put $w$ in the $GT[i]$ that

8: satisfies Input${[i][j]=line\\_rgb}$

9: end for

10: if $len(GT[i])=2$ then

11: Set line 2, 3

12: else

13: if $len(GT[i])=3$ then

14: if $GT[i][1]<w/2$ then

15: Set line 1, 2, 3

16: else

17: line 2, 3, 4

18: end if

19: else

20: Set line 1, 2, 3, 4

21: end if

22: end if

23:end for

[hbt!](topskip=0pt, botskip=0pt, midskip=0pt)[width=3.3in]fig12.pngResults of
extracting ground truth (green points) on the original map images. (top red
line of Fig. IV-C(b)). Two points mean line 2 and line 3 in the left order
(bottom red line of Fig. IV-C(b)). The three points mean line 2, line 3, and
line 4 if a second point is on the left based on the middle of the image
(middle red line of Fig. IV-C(b)). Conversely, if the second point is on the
right, it means line 1, line 2, and line 3. Lastly, The four points mean line
1, line 2, line 3, and line 4.

##  V Experiments and Results

To show the usefulness of GAN in simulation driving tests, three methods were
used to analyze data. First, FID measured the reality of virtual-world images
by calculating differences in the density of two distributions. FID showed
DCLGAN made CARLA images to realistic. Second, lane segmentation accuracy
proved ENet-SAD understood realistic images easier. Third, GPS data showed CwD
images made driving results more realistic. We selected two road types; curved
roads, and straight roads. Then, the curved road was analyzed by using Root
Mean Square Error (RMSE) and the straight road was considered by the ability
of lane restoring.

###  V-A FID score

Cw/oD2Real image gap was obtained by measuring the FID between 5498 CARLA
images without DCLGAN and 6929 real-world images, and CwD2Real image gap was
obtained by measuring the FID between 5298 images passed through DCLGAN and
6929 real-world images. This relationship is depicted in Fig. V-A.
[hbt!](topskip=0pt, botskip=0pt, midskip=0pt)[width=2.4in]fig14.png Cw/oD2Real
gap (Cw/oD vs real-world imageset) and CwD2Real gap (CwD vs real-world
imageset) measured by FID
As a result of training by setting $\lambda_{NCE,X}$ to 3, the weight(epoch
26) obtained with FSIM value of 0.451 was used. An example of an image
translated using this weight is shown in Fig. V-A. The lanes of CARLA are
inherently blurry and indistinguishable from the surrounding roads, whereas
the lanes of the real-world domain are clear and distinctly visible in the
distance. This DCLGAN weight translates the image by well reflecting the
difference in lane characteristics between these domains without distorting
the shape of the lane.
TABLE VI is the FID measurement of the CARLA images before and after
conversion using the corresponding weights. Considering the lack of diversity
in our dataset, it was also measured when the dimensionality of the feature
was 64, 192, 768, not only 2048. In the case of CARLA domain, 5498 images that
were used as trainsets were used, and in the case of real-world domain, 6929
images were used in the same way. Additionally, 4585 separate images were used
for Real vs Real comparison. In all cases, the FID value of CwD was lower than
that of Cw/oD. (7.34 vs 1.20, 24.10 vs 4.67, 2.05 vs 1.17, 321.97 vs 240.80)
That is, as we saw earlier, DCLGAN changed the CARLA image characteristics
such as road color and lane clarity to fit the real-world domain, and it was
possible to quantitatively confirm the reduction of the gap with FID.

[h](topskip=0pt, botskip=0pt, midskip=0pt)[width=2.9in]fig15.png From left,
Cw/oD, paired CwD, and real-world lane image for comparison. The lane on the
real-world domain side is clearer, it can be easily distinguished from the
surroundings, and the far side can be seen well.

Feature Dimensionality | Real vs Cw/oD | Real vs CwD | Real vs Real
---|---|---|---
64 | 7.34 | 1.20 | 0.01
192 | 24.10 | 4.67 | 0.05
768 | 2.05 | 1.17 | 0.15
2048 | 321.97 | 240.80 | 86.50
TABLE VI: Real vs Cw/oD, Real vs CwD, Real vs Real FID values for each
dimensionality of the feature. The FID values of the CwD images subjected to
DCLGAN were always smaller.

###  V-B Lane Segmentation Accuracy

|  Cw/oD |  CwD |  Real
---|---|---|---
Accuracy(%) |  82.85 |  84.94 |  89.73
TABLE VII: Cw/oD, CwD, Real-world Lane segmentation accuracy.

Lane segmentation accuracy in each domain was evaluated followed by the
TuSimple evaluation method. Prediction results were obtained using ENet-SAD
used for LKAS mentioned 3.A.2. TABLE VII. shows the specific results and Fig.
VII describes specific flow.
This result means that CwD has become similar to the image of reality. This is
because DCLGAN corrected the color of CARLA’s roads and incomplete cracked
lanes. Moreover, DCLGAN modified CARLA images looks like train image dataset
from real-world. This made Enet-SAD easier to understand.

[h](topskip=0pt, botskip=0pt, midskip=0pt)[width=3.3in]fig16.png Prediction
results were got from Enet-SAD. Then accuracy was calculated followed by
TuSimple evaluation method.

###  V-C Difference of GPS Orbit

To compare the driving test results, we collected GPS data and analyzed them
by calculating RMSE. For data collection, The LKAS algorithm was tested in
CARLA. The vehicle was spawned and turned on LKAS. The starting position was
(East, North) = (44549054.51, 394557162.27) as UTM. After finishing a lap, we
repeated this 20 times. Fig. V-C shows the overall setting for GPS and Fig.
V-C shows the overall driving orbital in each domain. After data collection,
to utilize RMSE, we selected two road types (slightly curved road, and steep
curved road) and two sections for each road type. In Fig. V-C, (a), (b), (c),
and (d) described each section. Then, The RMSE between the ideal path
connecting the central points of the road and each driving path was
calculated. More specifically, there are two ways to calculate RMSE: distance
east (x) and distance north (y). Table N shows the mean RMSE values of 10
experiments of CwD and Cw/oD for each section. [h](topskip=0pt, botskip=0pt,
midskip=0pt)[width=3.3in]fig13.pngGPS setting and method flow for each domain.

As shown in TABLE VIII, CwD ran closer to the center route than Cw/oD.
According to section 1 and section 4, the steep curved roads, CwD was more
stable. However, in section 2 and section 3 which are slightly curved roads,
CwD and Cw/oD were almost the same. Moreover, CwD finished whole laps but
Cw/oD failed two times. These results mean that Cw/oD can cause bad driving
results because of lane segmentation accuracy, especially on the steep curved
road which is the most important part of the map. [hbt!](topskip=0pt,
botskip=0pt, midskip=0pt)[width=6.6in]fig18.png GPS orbit of CwD, Cw/oD, and
center path. (a) Position of each section for RMSE calculation. (b), (c), (d),
(e) Magnified view of each section.

| Section1 | Section 2 | Section 3 | Section 4 |
---|---|---|---|---|---
| x (m) | y (m) | x (m) | y (m) | x (m) | y (m) | x (m) | y (m) | Success Rate ($\%$)
CwD | 0.753 | 1.941 | 0.232 | 1.579 | 0.182 | 0.111 | 0.550 | 0.998 | 100
Cw/oD | 1.115 | 2.099 | 0.252 | 1.574 | 0.192 | 0.118 | 0.719 | 1.449 | 71.4
Cw/oD - CwD | 0.337 | 0.158 | 0.021 | -0.055 | 0.003 | 0.001 | 0.139 | 0.481 |
TABLE VIII: The degree to which CwD and Cw/oD deviate from the center of the
road in each session. CwD drove a complete lap in all cases, showing a 100%
success rate. Cw/oD was 80%.

###  V-D LANE RESTORING

In this section, we tried to evaluate how well a vehicle can return when it
veers out of its lane. If CwD recognizes the lane better than Cw/oD, the
ability to return to the existing lane within reasonable time when it deviates
from the lane should be better. To confirm this, a lane-keeping driving test
was conducted after locating the car at +-90cm, 120cm, 130cm, 140cm, and 150cm
in the direction perpendicular to the lane from the common driving start GNSS
location (35.6524837116667, 128.397828661667). (+ is right, - is left) In
common, if the vehicle starts from its own location and returns to the
original lane within 2 minutes and continues to keep the lane stably, it was
judged as a success. The results are shown in Fig. V-D and TABLE IX. As shown
in TABLE IX, Cw/oD failed when starting by moving 1.5 m, 1.4 m to the right,
and 1.2 m, 1.3 m, 1.4 m, and 1.5 m to the left. However, CwD succeeded in all
but the right 1.5m, left 1.3m, 1.4m, and 1.5m, and as shown in Fig. V-D, the
path returning to the original lane was confirmed even under relatively
extreme conditions such as +1.4m and -1.3m. Compared to Cw/oD, it can be seen
that CwD has better restoring force to return to the center of the lane by
better recognizing the lane even when it is above the lane away from the
center of the lane.

[!t](topskip=0pt, botskip=0pt, midskip=0pt)[width=3.3in]fig17.png Lane
restoring GPS path; Red (X), Blue (O).

Distance (m) | 1.5 | 1.4 | 1.3 | 1.2 | 0.9 | -0.9 | -1.2 | -1.3 | -1.4 | -1.5
---|---|---|---|---|---|---|---|---|---|---
Cw/oD | X | X | O | O | O | O | X | X | X | X
CwD | X | O | O | O | O | O | O | O | X | X
TABLE IX: Lane return success or not according to the distance from the center
of the lane.

##  VI Conclusion

This paper proposed the issues about unrealistic images when the algorithm was
tested in simulation and the solution of this problem by utilizing DCLGAN
model. The images extracted from CARLA’s camera were converted to the real
domain. Then, FID quantitatively showed that it became more similar to
reality, especially in our KIAPI high-speed circulation environment. through
the Lane segmentation accuracy, we showed that the image of DCLGAN is similar
to the reality, of our train dataset, so that Enet-SAD had a better
understanding of the lane. Finally, we evaluated the difference in the overall
trajectory with the difference in GPS orbitals as area and compared the
driving trajectory of critical parts through RMSE (x) and RMSE (y). These
indicated that CwD is more similar to the driving result in reality. Future
work will concentrate on applying AEB and SCC and implementing dynamic models
more precisely through CARSIM.

## References

  * [1] World Bank, “The High Toll of Traffic Injuries: Unacceptable and Preventable,” Washington, DC, USA, 2018 [Online]. Available: http://hdl.handle.net/10986/29129
  * [2] National Highway Traffic Safety Administration, ”Standing General Order on Crash Reporting for Level 2 Advanced Driver Assistance Systems,” Washington, DC, USA, DOT-HS-813-325, 2022 [Online]. Available: https://www.nhtsa.gov/sites/nhtsa.gov/files/2022-06/ADAS-L2-SGO-Report-June-2022.pdf
  * [3] R. Gutierrez, J. F. Arango, C. Gomez-Huelamo, L. M. Bergasa, R. Barea, and J. Araluce, “Validation method of a self-driving architecture for unexpected pedestrian scenario in Carla Simulator,” 2021 IEEE Intelligent Vehicles Symposium (IV), 2021\.
  * [4] D. J. Fremont, E. Kim, Y. V. Pant, S. A. Seshia, A. Acharya, X. Bruso, P. Wells, S. Lemke, Q. Lu, and S. Mehta, “Formal scenario-based testing of autonomous vehicles: From simulation to the Real World,” 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), 2020\.
  * [5] C. Gómez-Huélamo, J. Del Egido, L. M. Bergasa, R. Barea, E. López-Guillén, F. Arango, J. Araluce, and J. López, “Train here, drive there: Simulating real-world use cases with fully-autonomous driving architecture in Carla Simulator,” Advances in Intelligent Systems and Computing, pp. 44–59, 2020.
  * [6] Y. Chen, S. Chen, T. Xiao, S. Zhang, Q. Hou, and N. Zheng, “Mixed test environment-based vehicle-in-the-loop validation - A new testing approach for Autonomous Vehicles,” 2020 IEEE Intelligent Vehicles Symposium (IV), 2020\.
  * [7] C. Brogle, C. Zhang, K. L. Lim, and T. Braunl, “Hardware-in-the-loop autonomous driving simulation without real-time constraints,” IEEE Transactions on Intelligent Vehicles, vol. 4, no. 3, pp. 375–384, 2019.
  * [8] S. Stević, M. Krunić, M. Dragojević, and N. Kaprocki, “Development of adas perception applications in Ros and ‘software-in-the-loop’ validation with Carla Simulator,” Telfor Journal, vol. 12, no. 1, pp. 40–45, 2020.
  * [9] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Nets,” in Advances in neural information processing systems (NIPS), 2014, pp. 2672–2680.
  * [10] Y. Cao, Z. Zhou, W. Zhang, and Y. Yu, “Unsupervised diverse colorization via generative adversarial networks,” arXiv.org, 01-Jul-2017. [Online]. Available: https://arxiv.org/abs/1702.06674. [Accessed: 22-Nov-2022].
  * [11] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-realistic single image super-resolution using a generative adversarial network,” arXiv.org, 25-May-2017. [Online]. Available: https://arxiv.org/abs/1609.04802. [Accessed: 22-Nov-2022].
  * [12] E. Yurtsever, D. Yang, I. M. Koc, and K. A. Redmill, “Photorealism in driving simulations: Blending generative adversarial image synthesis with rendering,” IEEE Transactions on Intelligent Transportation Systems, pp. 1–10, 2022.
  * [13] S. R. Richter, H. A. AlHaija, and V. Koltun, “Enhancing photorealism enhancement,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–1, 2022.
  * [14] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An Open Urban Driving Simulator,” the 1st Annual Conference on Robot Learning (CoRL), 2017\.
  * [15] J. Han, M. Shoeiby, L. Petersson, and M. A. Armin, “Dual contrastive learning for unsupervised image-to-image translation,” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021\.
  * [16] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” 2017 IEEE International Conference on Computer Vision (ICCV), 2017\.
  * [17] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium”, the 31st International Conference on Neural Information Processing Systems(NIPS), 2017\.
  * [18] Y. Hou, Z. Ma, C. Liu, and C. C. Loy, “Learning lightweight lane detection CNNS by self-attention distillation,” 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019\.
  * [19] TuSimple, ”Tusimple benchmark”, GitHub repository, https://github.com/TuSimple/tusimple-benchmark, 2017.
  * [20] L. Zhang, L. Zhang, X. Mou and D. Zhang, ”FSIM: A Feature Similarity Index for Image Quality Assessment,” in IEEE Transactions on Image Processing, vol. 20, no. 8, pp. 2378-2386, Aug. 2011, doi: 10.1109/TIP.2011.2109730.
  * [21] H. Bae, G. Lee, J. Yang, G. Shin, G. Choi, and Y. Lim, “Estimation of the closest in-path vehicle by low-channel LIDAR and camera sensor fusion for Autonomous Vehicles,” Sensors, vol. 21, no. 9, p. 3124, 2021.
  * [22] E. Seo, S. Lee, G. Shin, H. Yeo, Y. Lim, and G. Choi, “Hybrid tracker based optimal path tracking system of autonomous driving for Complex Road Environments,” IEEE Access, vol. 9, pp. 71763–71777, 2021.
  * [23] X. Pan, J. Shi, P. Luo, X. Wang, and X. Tang, “Spatial as deep: Spatial CNN for traffic scene understanding,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018.
  * [24] “Proving ground,” KIAPI. http://www.kiapi.or.kr/sub02/sub01_02.php?sid
=2&page_num=&skey=&sval=&mode= (Accessed Sep. 20, 2022).

  * [25] “Synchrony and time-step,” CARLA Simulator. https://carla.readthedocs.io/en/latest/adv_synchrony_timestep/ (Accessed: 05-Nov-2022).
  * [26] Maximilian Seitzer, “pytorch-fid”, GitHub repository, https://github.com/mseitzer/pytorch-fid, 2020.