# On Tsirelson pairs of $\mathrm{C}^{*}$-algebras

Isaac Goldbring and Bradd Hart  Department of Mathematics
University of California, Irvine, 340 Rowland Hall (Bldg.# 400), Irvine, CA
92697-3875<EMAIL_ADDRESS>http://www.math.uci.edu/ isaac Department of
Mathematics and Statistics, McMaster University, 1280 Main St., Hamilton ON,
Canada L8S 4K1<EMAIL_ADDRESS>http://ms.mcmaster.ca/ bradd/

###### Abstract.

We introduce the notion of a Tsirelson pair of $\mathrm{C}^{*}$-algebras,
which is a pair of $\mathrm{C}^{*}$-algebras for which the space of quantum
strategies obtained by using states on the minimal tensor product of the pair
is dense in the space of quantum strategies obtained by using states on the
maximal tensor product. We exhibit a number of examples of such pairs that are
“nontrivial” in the sense that the minimal tensor product and the maximal
tensor product of the pair are not isomorphic. For example, we prove that any
pair containing a $\mathrm{C}^{*}$-algebra with Kirchberg’s QWEP property is a
Tsirelson pair. We then introduce the notion of a $\mathrm{C}^{*}$-algebra
with the Tsirelson property (TP) and establish a number of closure properties
for this class. We also show that the class of $\mathrm{C}^{*}$-algebras with
the TP forms an elementary class (in the sense of model theory), but that this
class does not admit an “effective” axiomatization.

I. Goldbring was partially supported by NSF grant DMS-2504477. B. Hart was
partially supported by the NSERC

##  1\. Introduction

Tsirelson’s problem in quantum information theory (see [16, 17]) asks whether
or not, for any given pair of natural numbers $k,n\in\mathbb{N}$ with $k,n\geq
2$, the set $C_{qc}(k,n)$ of quantum commuting correlations coincides with the
set $C_{qa}(k,n)$ of quantum asymptotic correlations. The former set consists
of correlation matrices $p(a,b|x,y)\in[0,1]^{k^{2}n^{2}}$ (suggestively
written as conditional probabilities for their connections with nonlocal
games) for which there exists a Hilbert space $\mathcal{H}$, POVMS
$A^{x}=(A^{x}_{1},\ldots,A^{x}_{n})$ and $B^{y}=(B^{y}_{1},\ldots,B^{y}_{n})$
on $\mathcal{H}$ for all $x,y\in[k]:=\\{1,\ldots,k\\}$ satisfying
$[A^{x}_{a},B^{y}_{b}]:=A^{x}_{a}B^{y}_{b}-B^{y}_{b}A^{x}_{a}=0$ for all
$x,y\in[k]$ and $a,b\in[n]$, and a unit vector $\xi\in\mathcal{H}$ for which
$p(a,b|x,y)=\langle A^{x}_{a}B^{y}_{b}\xi,\xi\rangle$. (Precise definitions of
POVMs and any other undefined notions will be given later in the paper.) The
latter set consists of the subset of those quantum commuting strategies that
can be approximated by quantum commuting strategies for which the Hilbert
space $\mathcal{H}$ can be decomposed as
$\mathcal{H}_{A}\otimes\mathcal{H}_{B}$ and for which the POVMS $A^{x}$
(respectively $B^{y}$) act only on $\mathcal{H}_{A}$ (respectively only on
$\mathcal{H}_{B}$). It is a consequence of the landmark result
$\operatorname{MIP}^{*}=\operatorname{RE}$ [9] that there are pairs $(k,n)$
for which $C_{qa}(k,n)$ is a proper subset of $C_{qc}(k,n)$, thus providing a
negative answer to Tsirelson’s problem in general.

Work of Fritz [4] and independently Junge et. al. [10] allows one to recast
Tsirelson’s problem in terms of states on tensor products of certain
$\mathrm{C}^{*}$-algebras. (It is this connection that allows one to conclude
a negative solution to Kirchberg’s QWEP problem-and thus, ultimately, a
negative solution to the Connes Embedding Problem-from a negative solution to
Tsirelson’s problem.) Indeed, set $\mathbb{F}(k,n)$ to be the group freely
generated by $k$ elements of order $n$ and let $C^{*}(\mathbb{F}(k,n))$ denote
its universal group $\mathrm{C}^{*}$-algebra. It was established that, for any
$p(a,b|x,y)\in[0,1]^{k^{2}n^{2}}$, one has:

  * •

$p\in C_{qa}(k,n)$ if and only if there are POVMs $A^{x}$ and $B^{y}$ in
$C^{*}(\mathbb{F}(k,n))$ and a state $\phi$ on $C^{*}(\mathbb{F}(k,n))\otimes
C^{*}(\mathbb{F}(k,n))$ such that

$p(a,b|x,y)=\phi(A^{x}_{a}\otimes B^{y}_{b});$

  * •

$p\in C_{qc}(k,n)$ if and only if there are POVMs $A^{x}$ and $B^{y}$ in
$C^{*}(\mathbb{F}(k,n))$ and a state $\phi$ on
$C^{*}(\mathbb{F}(k,n))\otimes_{\max}C^{*}(\mathbb{F}(k,n))$ such that

$p(a,b|x,y)=\phi(A^{x}_{a}\otimes B^{y}_{b}).$

Throughout this paper, $\otimes$ always denotes the minimal tensor product of
$\mathrm{C}^{*}$-algebras.

The preceding fact suggests looking at correlations of the form
$\phi(A^{x}_{a}\otimes B^{y}_{b})$, where $A^{x}$ and $B^{y}$ are POVMs from
some fixed pair $(C,D)$ of $\mathrm{C}^{*}$-algebras and $\phi$ is a state
either on $C\otimes D$ or $C\otimes_{\max}D$; in fact, we will consider the
closures of these two sets of correlations, which we will denote by
$C_{\min}(C,D,k,n)$ and $C_{\max}(C,D,k,n)$ respectively. (In general, we see
no reason why these sets of correlations should be closed, although they do
happen to be so for the case that $C=D=C^{*}(\mathbb{F}(k,n))$.) It is
standard fare that $C_{\min}(C,D,k,n)\subseteq C_{\max}(C,D,k,n)$,
$C_{\min}(C,D,k,n)\subseteq C_{qa}(k,n)$, and $C_{\max}(C,D,k,n)\subseteq
C_{qc}(k,n)$; for the convenience of the reader, proofs of these facts will be
given in Section 3.

Motivated by the above discussion, we will call a pair of
$\mathrm{C}^{*}$-algebras $(C,D)$ a Tsirelson pair if
$C_{\min}(C,D,k,n)=C_{\max}(C,D,k,n)$ for all $(k,n)$. Denoting by
$\mathbb{F}_{\infty}$ the free group on a countably infinite set of generators
and using the fact that the canonical projection
$C^{*}(\mathbb{F}_{\infty})\to C^{*}(\mathbb{F}(k,n))$ admits a ucp lift for
each $(k,n)$, the fact that Tsierelson’s problem has a negative answer implies
that $(C^{*}(\mathbb{F}_{\infty}),C^{*}(\mathbb{F}_{\infty}))$ is not a
Tsirelson pair.

Following Pisier, a pair of $\mathrm{C}^{*}$-algebras $(C,D)$ is called a
nuclear pair if $C\otimes D=C\otimes_{\max}D$. Clearly, any nuclear pair is a
Tsirelson pair. We show, however, that the class of Tsirelson pairs is much
larger than the class of nuclear pairs. For example, we show that if one
element of the pair has Kirchberg’s QWEP property, then the pair is a
Tsirelson pair. This follows from a more general fact, namely that the class
of Tsirelson pairs is closed under “quotients.”

We then move on to studying $\mathrm{C}^{*}$-algebras with the Tsirelson
property (TP), which we define to be those $\mathrm{C}^{*}$-algebras which
form a Tsirelson pair with any other $\mathrm{C}^{*}$-algebra. Continuing with
our analogy with nuclear pairs, $\mathrm{C}^{*}$-algebras with the TP are the
analog of nuclear $\mathrm{C}^{*}$-algebras in this context. By the previous
paragraph, the class of $\mathrm{C}^{*}$-algebras with the TP includes the
class of $\mathrm{C}^{*}$-algebras with the QWEP; we leave open the question
of whether or not these classes coincide. We establish a variety of closure
properties of the class of $\mathrm{C}^{*}$-algebras with the TP and show that
they form an elementary class (in the sense of model theory) but that this
class does not admit any “effective” set of axioms. As a consequence, we show
that the class of $\mathrm{C}^{*}$-algebras without the TP is not closed under
ultraproducts.

We are grateful to Michał Banacki, Narutaka Ozawa, Vern Paulsen, Christopher
Schafhauser, David Sherman, and Aaron Tikuisis for useful discussions and
comments about this work.

##  2\. Preliminaries

###  2.1. WEP, QWEP, and LP

Recall that if $C\subseteq D$ are $\mathrm{C}^{*}$-algebras, a weak
conditional expectation from $D$ onto $C$ is a ucp map $D\to C^{**}$
restricting to the canonical inclusion $C\hookrightarrow C^{**}$. If there is
a weak conditional expectation from $D$ onto $C$, then we say that $C$ is
relatively weakly injective (r.w.i.) in $D$. It is known that $C$ is r.w.i. in
$D$ if and only if: for any $\mathrm{C}^{*}$-algebra $E$, we have that
$C\otimes_{\max}E\subseteq D\otimes_{\max}E$, that is, the canonical map
$C\otimes_{\max}E\to D\otimes_{\max}E$ is isometric. $C$ has the weak
expectation property (WEP) if it is r.w.i. in any $\mathrm{C}^{*}$-algebra
containing it.

Recalling the notion of a nuclear pair from the introduction, Kirchberg’s
fundamental result from [11] on $\mathrm{C}^{*}$-algebras with WEP reads as
follows:

######  Fact 2.1.

$C$ has the WEP if and only if $(C,C^{*}(\mathbb{F}_{\infty}))$ is a nuclear
pair.

Recall also that a $\mathrm{C}^{*}$-algebra $C$ has the QWEP property if it is
a quotient of a $\mathrm{C}^{*}$-algebra with the WEP.

The $\mathrm{C}^{*}$-algebra $C$ has the lifting property (LP) if: given any
ucp map $\Phi:C\to D/J$, where $J$ is an ideal of some
$\mathrm{C}^{*}$-algebra $D$, there is a ucp map $\Psi:C\to D$ such that
$\Phi=\pi\circ\Psi$, where $\pi:D\to D/J$ is the canonical quotient map.

We will need the following recent characterization of the LP due to Pisier
[14]:

######  Fact 2.2.

$D$ has the LP if and only if: for any family $(C_{i})_{i\in I}$ of
$\mathrm{C}^{*}$-algebras and any ultrafilter $\mathcal{U}$ on $I$, the
canonical map
$(\prod_{\mathcal{U}}C_{i})\otimes_{\max}D\to\prod_{\mathcal{U}}(C_{i}\otimes_{\max}D)$
is isometric.

###  2.2. Ultraproduct states

Throughout this paper, $\mathcal{U}$ denotes a nonprincipal ultrafilter on
some index set $I$. Given a family of $\mathrm{C}^{*}$-algebras $(C_{i})_{i\in
I}$, we define their ultraproduct with respect to $\mathcal{U}$ to be the
quotient of the $\mathrm{C}^{*}$-algebra $\prod_{i\in I}C_{i}$ of all
uniformly bounded sequences by the ideal of elements $(A_{i})_{i\in I}$ for
which $\lim_{\mathcal{U}}\|A_{i}\|=0$. It is well-known that
$\prod_{\mathcal{U}}C_{i}$ is a $\mathrm{C}^{*}$-algebra once again. Given
$(A_{i})_{i\in I}\in\prod_{i\in I}C_{i}$, we denote its coset in
$\prod_{\mathcal{U}}C_{i}$ by $(A_{i})_{\mathcal{U}}$. When $C_{i}=C$ for all
$i\in I$, we refer to the ultraproduct as the ultrapower of $C$ with respect
to $\mathcal{U}$ and denote it by $C^{\mathcal{U}}$. There is a canonical
embedding of $C$ into $C^{\mathcal{U}}$, called the diagonal embedding, given
by mapping $c\in C$ to the coset of the sequence indexed by $I$ with constant
value $c$.

If $\phi_{i}\in S(C_{i})$ is a state on $C_{i}$ for each $i\in I$, we let the
ultraproduct state (also known in the literature as the limit state)
$\phi:=(\phi_{i})_{\mathcal{U}}$ on $\prod_{\mathcal{U}}C_{i}$ be defined by
$\phi((A_{i})_{\mathcal{U}}):=\lim_{\mathcal{U}}\phi_{i}(A_{i})$.

We will need the following lemma later in this paper:

######  Lemma 2.3.

For any ultraproduct $\prod_{\mathcal{U}}C_{i}$ of $\mathrm{C}^{*}$-algebras,
the set of ultraproduct states is weak*-dense in the set of all states on
$\prod_{\mathcal{U}}C_{i}$.

###### Proof.

Suppose, towards a contradiction, that this is not the case. Let $X$ denote
the weak*-closure of the set of ultraproduct states in
$S(\prod_{\mathcal{U}}C_{i})$. If $X\neq S(\prod_{\mathcal{U}}C_{i})$, then by
the Hahn-Banach Separation Theorem, there is a self-adjoint element
$A\in\prod_{\mathcal{U}}C_{i}$ with $\|A\|=1$ such that $\psi(A)=0$ for all
$\psi\in X$. Write $A=(A_{i})_{\mathcal{U}}$ with each $A_{i}$ a self-adjoint
element of $C_{i}$. For each $i\in I$, take $\psi_{i}\in S(C_{i})$ such that
$\psi_{i}(A_{i})=\|A_{i}\|$. Setting $\psi:=(\psi_{i})_{\mathcal{U}}\in X$, we
have that $1=\|A\|=\lim_{\mathcal{U}}\|A_{i}\|=\psi(A)$, a contradiction. ∎

###  2.3. POVMs in $\mathrm{C}^{*}$-algebras

Given a $\mathrm{C}^{*}$-algebra $C$, a positive operator-valued measure or
POVM of length $n$ in $C$ is a finite collection $A_{1},\ldots,A_{n}$ of
positive elements in $C$ such that $A_{1}+\cdots+A_{n}=I$.

It is well-known that there is a 1-1 correspondence between POVMs in $A$ of
length $n$ and ucp maps $\Phi:\mathbb{C}^{n}\to A$, where $A_{1},\ldots,A_{n}$
as above corresponds to the ucp map $\Phi$ given by $\Phi(e_{i})=A_{i}$ for
$i=1,\ldots,n$, where $e_{1},\ldots,e_{n}$ is the standard basis for
$\mathbb{C}^{n}$.

######  Lemma 2.4.

Suppose that $A_{1},\ldots,A_{n}$ is a POVM in the $\mathrm{C}^{*}$-algebra
$C$ and $\pi:D\to C$ is a surjective *-homomorphism. Then there is a POVM
$B_{1},\ldots,B_{n}$ in $D$ such that $\pi(B_{i})=A_{i}$ for $i=1,\ldots,n$.

###### Proof.

Let $\Phi:\mathbb{C}^{n}\to C$ be the ucp map given by $\Phi(e_{i})=A_{i}$ for
$i=1,\ldots,n$. Since $\mathbb{C}^{n}$ has the LP, there is a ucp map
$\Psi:\mathbb{C}^{n}\to D$ such that $\Phi=\pi\circ\Psi$. Setting
$B_{i}:=\Psi(e_{i})$, we have that $B_{1},\ldots,B_{n}$ is the desired POVM in
$D$. ∎

A special case of the previous lemma is the following:

######  Proposition 2.5.

Suppose that $(C_{i})_{i\in I}$ is a family of $\mathrm{C}^{*}$-algebras and
$\mathcal{U}$ is an ultrafilter on $I$. Let $C:=\prod_{\mathcal{U}}C_{i}$.
Then for any POVM $A_{1},\ldots,A_{n}$ of length $n$ from $C$, there are POVMs
$A_{1,i},\ldots,A_{n,i}$ of length $n$ in $C_{i}$ for each $i\in I$ such that
$A_{a}=(A_{a,i})_{\mathcal{U}}$ for all $a\in[n]$.

In model theoretic terms, the previous proposition says that the set of POVMs
of length $n$ is definable in the theory of $\mathrm{C}^{*}$-algebras. The
previous proposition together with a standard “compactness and contradiction”
argument yield the following corollary. However, for our purposes in the last
section, it behooves us to give a more direct and explicit proof.

######  Corollary 2.6.

For any $n\geq 1$, $\epsilon>0$, and sequence $A_{1},\ldots,A_{n}$ of positive
elements from a $\mathrm{C}^{*}$-algebra $C$ for which
$\|\sum_{i=1}^{n}A_{i}-I\|<\frac{\epsilon}{2}$, there is a POVM
$B_{1},\ldots,B_{n}$ in $C$ with $\|A_{i}-B_{i}\|<\epsilon$ for all
$i=1,\ldots,n$.

###### Proof.

First suppose that $\sum^{n}_{i=1}A_{i}\leq I$. In this case, the desired POVM
can be obtained by letting $B_{i}=A_{i}$ for $i<n$ and
$B_{n}=A_{n}+(I-\sum^{n}_{i=1}A_{i})$.

Suppose, on the other hand, that $\sum^{n}_{i=1}A_{i}\geq I$ and set
$C_{i}=\frac{1}{1+\frac{\epsilon}{2}}A_{i}$. We then have that
$\|A_{i}-C_{i}\|\leq\frac{\epsilon}{2}$ and, by functional calculus, that
$\sum^{n}_{i=1}C_{i}\leq I$. By the first case, there is a POVM
$B_{1},\ldots,B_{n}$ such that $\|B_{i}-C_{i}\|<\frac{\epsilon}{2}$ for each
$i=1,\ldots,n$, whence $\|A_{i}-B_{i}\|<\epsilon$, as desired. ∎

##  3\. Tsirelson pairs of $\mathrm{C}^{*}$-algebras

We now precisely define the correlation sets mentioned in the introduction.

######  Definition 3.1.

Given $\mathrm{C}^{*}$-algebras $C$ and $D$, we let $C_{\min}(C,D,k,n)$
(respectively $C_{\max}(C,D,k,n))$ denote the _closure_ of the set of
correlations of the form
$\phi(A^{x}_{a}\otimes B^{y}_{b})$, where $A^{1},\ldots,A^{k}$ are POVMs of
length $n$ from $C$, $B^{1},\ldots,B^{k}$ are POVMs of length $n$ from $D$,
and $\phi$ is a state on $C\otimes D$ (respectively a state on
$C\otimes_{\max}D$).

######  Remark 3.2.

In the preceding definition, we took closures as in many arguments, having
closed sets of correlations is preferable. It is not clear to us if taking the
closure is necessary (although we suspect that it is in general).

The following is well-known, but we include a proof for the sake of the
reader. For the proof, we recall that the sets $C_{qa}(k,n)$ and $C_{qc}(k,n)$
are closed, convex subsets of $[0,1]^{k^{2}n^{2}}$ for all $(k,n)$.

######  Lemma 3.3.

For any pair of $\mathrm{C}^{*}$-algebras $C$ and $D$ and any $(k,n)$, we
have:

  1. (1)

$C_{\min}(C,D,k,n)\subseteq C_{\max}(C,D,k,n)$.

  2. (2)

$C_{\min}(C,D,k,n)\subseteq C_{qa}(k,n)$.

  3. (3)

$C_{\max}(k,n)\subseteq C_{qc}(k,n)$.

  4. (4)

If $C^{\prime}$ and $D^{\prime}$ are $\mathrm{C}^{*}$-algebras for which
$C\subseteq C^{\prime}$ and $D\subseteq D^{\prime}$, then

$C_{\min}(C,D,k,n)\subseteq C_{\min}(C^{\prime},D^{\prime},k,n).$

###### Proof.

(1) follows from the fact that any state on $C\otimes D$ induces a state on
$C\otimes_{\max}D$ via the canonical surjection $C\otimes_{\max}D\to C\otimes
D$. To see (2), fix faithful representations $C\subseteq B(\mathcal{H})$ and
$D\subseteq B(\mathcal{K})$, whence $C\otimes D\subseteq
B(\mathcal{H}\otimes\mathcal{K})$ is a faithful representation. Now given any
state $\phi$ on $C\otimes D$, POVMs $A^{x}$ and $B^{y}$ from $C$ and $D$
respectively, and $\epsilon>0$, there are positive real numbers
$\lambda_{1},\ldots,\lambda_{m}$ with $\sum_{j=1}^{m}\lambda_{j}=1$ and unit
vectors $\psi_{1},\ldots,\psi_{m}\in\mathcal{H}\otimes\mathcal{K}$ for which

$|\phi(A^{x}_{a}\otimes
B^{y}_{b})-\sum_{j=1}^{m}\lambda_{j}\langle(A^{x}_{a}\otimes
B^{y}_{b})\psi_{j},\psi_{j}\rangle|<\epsilon$

for all $x,y\in[k]$ and $a,b\in[n]$ (see, for example, [4, Proposition B.5]).
It remains to use the fact that $C_{qa}$ is a closed, convex set.

(3) is an immediate consequence of the GNS construction while (4) follows from
the fact that $C\otimes D\subseteq C^{\prime}\otimes D^{\prime}$ and any state
on $C\otimes D$ extends to a state on $C^{\prime}\otimes D^{\prime}$. ∎

######  Definition 3.4.

We say that a pair $(C,D)$ of $\mathrm{C}^{*}$-algebras is a Tsirelson pair if
${C_{\min}(C,D,k,n)}={C_{\max}(C,D,k,n)}$ for all $(k,n)$.

Note that every nuclear pair is a Tsirelson pair. Our next goal is to give a
useful reformulation of the notion of Tsirelson pair. We first need a
preliminary lemma. Recall that, for $n\in\mathbb{N}$, a
$\mathrm{C}^{*}$-algebra $C$ is called $n$-subhomogeneous if all irreducible
representations of $C$ have dimension at most $n$. $C$ is called
subhomogeneous if it is $n$-subhomogeneous for some $n\in\mathbb{N}$. Recall
also that subhomogeneous $\mathrm{C}^{*}$-algebras are nuclear.

######  Lemma 3.5.

If $C$ is not subhomogeneous, then for every $m\in\mathbb{N}$, there is a ucp
embedding of $M_{m}(\mathbb{C})$ in $C$.

###### Proof.

Fix $m\in\mathbb{N}$. By assumption, there is an irreducible representation of
$C$ on some Hilbert space $\mathcal{H}$ with $\dim(\mathcal{H})>m$. Consider
an orthogonal projection $p\in B(\mathcal{H})$ of rank $m$ and consider the
ucp map $T:C\to M_{m}(\mathbb{C})$ given by $T(c)=pcp$. Then $T$ restricts to
a surjective *-homomorphism on the multiplicative domain for $T$ by the
Kadison Transitivity theorem. Since $M_{m}(\mathbb{C})$ has the lifting
property, we can find a ucp left inverse of $T$, establishing the lemma. ∎

######  Proposition 3.6.

The pair $(C,D)$ is a Tsirelson pair if and only if
$C_{\max}(C,D,k,n)\subseteq C_{qa}(k,n)$ for all $(k,n)$.

###### Proof.

The forward direction is immediate from Lemma 3.3(2). For the converse
direction, fix $(k,n)$ and suppose that $C_{\max}(C,D,k,n)\subseteq
C_{qa}(k,n)$. If either $C$ or $D$ are subhomogeneous, then $(C,D)$ is a
nuclear pair and hence a Tsirelson pair. Thus we may assume that neither $C$
nor $D$ are subhomogeneous. In this case, by Lemma 3.5, for each
$m\in\mathbb{N}$, there is a u.c.p. embedding $\Phi:M_{m}(\mathbb{C})\otimes
M_{m}(\mathbb{C})\to C\otimes D$. Consequently,
$C_{qa}(k,n)\subseteq{C_{\min}(C,D,k,n)}$ for all $(k,n)$. By Lemma 3.3(1) and
the assumption, we see that
${C_{\min}(C,D,k,n)}={C_{\max}(C,D,k,n)}=C_{qa}(k,n)$, and thus $(C,D)$ is a
Tsirelson pair. ∎

The proof of the previous proposition motivates the following definition:

######  Definition 3.7.

The pair $(C,D)$ is said to be a strong Tsirelson pair if

${C_{\min}(C,D,k,n)}={C_{\max}(C,D,k,n)}=C_{qa}(k,n)$

for all $(k,n)$.

######  Corollary 3.8.

Given a pair $(C,D)$ of $\mathrm{C}^{*}$-algebras, exactly one of the
following occurs:

  * •

$(C,D)$ is not a Tsirelson pair.

  * •

One of $C$ or $D$ is subhomogeneous, in which case $(C,D)$ is a nuclear pair
(and thus a Tsirelson pair), but not a strong Tsirelson pair.

  * •

$(C,D)$ is a strong Tsirelson pair.

###### Proof.

Given the proof of Proposition 3.6, the only assertion that needs to be
established is that if one of $C$ or $D$ is subhomogeneous, then $(C,D)$ is
not a strong Tsirelson pair. While there is perhaps a more elementary proof of
this statement, we use Theorem 5.2 below and thus defer the proof until after
that theorem. ∎

######  Remark 3.9.

One might wonder about the condition that ${C_{\max}(C,D,k,n)}=C_{qc}(k,n)$
for all $(k,n)$. This happens, for example, when
$C=D=C^{*}(\mathbb{F}_{\infty})$. (This follows from the fact that the
canonical map $C^{*}(\mathbb{F}_{\infty})\to C^{*}(\mathbb{F}(k,n))$ has a ucp
lift together with the characterizations of $C_{qa}(k,n)$ and $C_{qc}(k,n)$ in
terms of states on the minimal and maximal tensors products of
$C^{*}(\mathbb{F}(k,n))$ with itself.) However, by the negative solution to
Tsirelson’s problem, we have that no such pair can be a Tsirelson pair.

We end this section with a useful closure property of the class of Tsirelson
pairs. To state this, we call a pair $(C^{\prime},D^{\prime})$ a quotient of
the pair $(C,D)$ if $C^{\prime}$ is a quotient of $C$ and $D^{\prime}$ is a
quotient of $D$.

######  Proposition 3.10.

The set of Tsirelson pairs is closed under taking quotients.

###### Proof.

This follows immediately from Lemma 2.4. ∎

##  4\. $\mathrm{C}^{*}$-algebras with the Tsirelson property

######  Definition 4.1.

We say that a $\mathrm{C}^{*}$-algebra $C$ has the Tsirelson property (TP) if
$(C,D)$ is a Tsirelson pair for any $\mathrm{C}^{*}$-algebra $D$.

Recall that a $\mathrm{C}^{*}$-algebra $C$ has WEP if and only if
$(C,C^{*}(\mathbb{F}_{\infty}))$ is a nuclear pair. The following lemma is the
analog for the TP and Tsirelson pairs.

######  Lemma 4.2.

For a $\mathrm{C}^{*}$-algebra $C$, the following are equivalent:

  1. (1)

$C$ has the TP.

  2. (2)

$(C,D)$ is a Tsirelson pair for every separable $\mathrm{C}^{*}$-algebra $D$.

  3. (3)

$(C,C^{*}(\mathbb{F}_{\infty}))$ is a Tsirelson pair.

###### Proof.

It is clear that (1) implies (2). To prove that (2) implies (1), fix $(k,n)$,
an arbitrary $\mathrm{C}^{*}$-algebra $D$, POVMs $A^{x}$ and $B^{y}$ in $C$
and $D$ respectively of length $n$, and a state $\phi$ on $C\otimes_{\max}D$.
Let $D^{\prime}$ be the subalgebra of $D$ generated by the $B^{y}_{b}$’s. We
have a canonical map $C\otimes_{\max}D^{\prime}\to C\otimes_{\max}D$, which
induces a state $\phi^{\prime}$ on $C\otimes_{\max}D^{\prime}$. Thus
$\phi(A^{x}_{a}\otimes B^{y}_{b})=\phi^{\prime}(A^{x}_{a}\otimes B^{y}_{b})$
and the latter correlation belongs to $C_{qa}(k,n)$ by assumption.

(2) clearly implies (3). The implication (3) implies (2) follows from
Proposition 3.10. ∎

We note the following closure properties of the class of
$\mathrm{C}^{*}$-algebras with the TP:

######  Proposition 4.3.

The class of $\mathrm{C}^{*}$-algebras with the TP is closed under quotients,
r.w.i. subalgebras, and ultraproducts.

###### Proof.

Closure under quotients follows from Proposition 3.10. Closure under r.w.i.
subalgebras follows from the definitions. We now prove closure under
ultraproducts. Suppose that $(C_{i})_{i\in I}$ is a family of
$\mathrm{C}^{*}$-algebras with the TP and $\mathcal{U}$ is an ultrafilter on
$I$. Set $C:=\prod_{\mathcal{U}}C_{i}$. By Lemma 4.2, it suffices to show that
$(C,C^{*}(\mathbb{F}_{\infty}))$ is a Tsirelson pair. Fix $k$ and $n$ and for
each $x,y\in[k]$, consider a POVM $A^{x}$ in $C$ of length $n$. By Proposition
2.5, for each $x\in[k]$, there is a POVM $A^{x}_{1,i},\ldots,A^{x}_{n,i}$ in
$C_{i}$ such that $A^{x}_{a}=(A^{x}_{a,i})_{\mathcal{U}}$ for all $a\in[n]$.
Fix also POVMs $B^{y}$ in $C^{*}(\mathbb{F}_{\infty})$ and a state $\phi\in
S(C\otimes_{\max}C^{*}(\mathbb{F}_{\infty}))$. Set
$p(a,b|x,y):=\phi(A^{x}_{a}\otimes B^{y}_{b})$. By Fact 2.2, the canonical map
$C\otimes_{\max}C^{*}(\mathbb{F}_{\infty})\to\prod_{\mathcal{U}}(C_{i}\otimes_{\max}C^{*}(\mathbb{F}_{\infty}))$
is isometric, whence we can consider $\phi$ as a state on the latter algebra
by extension. By Lemma 2.3, given $\epsilon>0$, there are states $\phi_{i}\in
S(C_{i}\otimes_{\max}C^{*}(\mathbb{F}_{\infty}))$ such that
$|\lim_{\mathcal{U}}\phi_{i}(A^{x}_{a,i}\otimes
B^{y}_{b}))-p(a,b|x,y)|<\epsilon$. By assumption, the correlations
$p_{i}(a,b|x,y):=\phi_{i}(A^{x}_{a,i}\otimes B^{y}_{b})$ belong to
$C_{qa}(k,n)$, whence so does $p$ by the previous sentence, letting $\epsilon$
tend to $0$. ∎

The following corollary yields a large class of $\mathrm{C}^{*}$-algebras with
the TP:

######  Corollary 4.4.

If $C$ has the QWEP property, then $C$ has the TP.

###### Proof.

This follows from the closure of the class of $\mathrm{C}^{*}$-algebras with
the TP under quotients together with the fact that every
$\mathrm{C}^{*}$-algebra with the WEP has the TP. To see this latter
statement, suppose that $C$ has the WEP; it suffices to show that
$(C,C^{*}(\mathbb{F}_{\infty}))$ has the TP, which is indeed the case since
this pair is a nuclear pair by Fact 2.1(1). ∎

######  Question 4.5.

Is there an example of a $\mathrm{C}^{*}$-algebra with the TP but not the QWEP
property?

Given that the class of $\mathrm{C}^{*}$-algebras with the QWEP is closed
under direct products, one potential approach to Question 4.5 would be to
establish a negative answer to the next question:

######  Question 4.6.

Is the class of $\mathrm{C}^{*}$-algebras with the TP closed under direct
products?

Since the class of $\mathrm{C}^{*}$-algebras with the TP is closed under
quotients, if it were closed under direct products, then it would in fact be
closed under arbitrary reduced products, that is, quotients of direct products
by arbitrary filters.

Just like the QWEP property, the TP is closed under inductive limits:

######  Proposition 4.7.

If $(C_{i})_{i\in I}$ is a directed family of $\mathrm{C}^{*}$-algebras with
the TP, then so is the direct limit $\varinjlim C_{i}$.

###### Proof.

Set $C:=\varinjlim C_{i}$. Fix $k$ and $n$ and a $\mathrm{C}^{*}$-algebra $D$.
For each $x,y\in[k]$, consider POVMs $A^{x}$ and $B^{y}$ of length $n$ in $C$
and $D$ respectively and a state $\phi\in S(C\otimes_{\max}D)$. By Corollary
2.6, given $\epsilon>0$, there is $i\in I$ and POVMs $\bar{A}^{x}$ in $C_{i}$
of length $n$ such that the image of $\bar{A}^{x}_{a}$ in $C$ under the
canonical map is within $\epsilon$ of $A^{x}_{a}$ for each $x\in[k]$ and
$a\in[n]$. Moreover, the canonical map of $C_{i}$ into $C$ induces a map
$C_{i}\otimes_{\max}D\to C\otimes_{\max}D$, inducing a state $\phi^{\prime}$
on $C_{i}\otimes_{\max}D$. It follows that the correlation
$p^{\prime}(a,b|x,y):=\phi^{\prime}(\bar{A}^{x}_{a}\otimes B^{y}_{b})$ is
within $\epsilon$ of the correlation $p(a,b|x,y):=\phi(\bar{A}^{x}_{a}\otimes
B^{y}_{b})$. Since $C_{i}$ has the TP, $p^{\prime}\in C_{qa}(k,n)$. Since
$\epsilon$ is arbitrary, it follows that $p\in C_{qa}(k,n)$, whence $(C,D)$ is
a Tsirelson pair. ∎

We list one further closure property of the class of $\mathrm{C}^{*}$-algebras
with the TP (see [2, Exercise 2.3.11] for the analogous property of the class
of nuclear $\mathrm{C}^{*}$-algebras):

######  Proposition 4.8.

Suppose that $C$ is a $\mathrm{C}^{*}$-algebra with the property that, for any
finite subset $F$ of $C$ and any $\epsilon>0$, there is a
$\mathrm{C}^{*}$-algebra $C^{\prime}$ with the TP and u.c.p. maps
$\psi_{1}:C\to C^{\prime}$ and $\psi_{2}:C^{\prime}\to C$ such that
$\|(\psi_{2}\circ\psi_{1})(c)-c\|<\epsilon$ for all $c\in F$. Then $C$ also
has the TP.

###### Proof.

Fix a $\mathrm{C}^{*}$-algebra $D$. Fix also $k$ and $n$ and, for each
$x,y\in[k]$, POVMs $A^{x}$ and $B^{y}$ in $C$ and $D$ respectively of length
$n$. Fix also a state $\phi\in S(C\otimes_{\max}D)$; we wish to show that
$p(a,b|x,y):=\phi(A^{x}_{a}\otimes B^{y}_{b})$ belongs to $C_{qa}(k,n)$. Fix
$\epsilon>0$ and take a $\mathrm{C}^{*}$-algebra $C^{\prime}$ with the TP and
u.c.p. maps $\psi_{1}:C\to C^{\prime}$ and $\psi_{2}:C^{\prime}\to C$ with
respect to the finite set $F:=\\{A^{x}_{a}\ :\ x\in[k],a\in[n]\\}$ and the
given $\epsilon$. Let $\bar{A}^{x}_{a}:=\psi_{1}(A^{x}_{a})$ and note that
each $\bar{A}^{x}$ is a POVM in $C^{\prime}$. Let
$\phi^{\prime}:=\phi\circ(\psi_{2}\otimes\operatorname{id}_{D})$, a state on
$C^{\prime}\otimes_{\max}D$. Let
$p^{\prime}(a,b|x,y):=\phi^{\prime}(\bar{A}^{x}_{a}\otimes B^{y}_{b})$. By
assumption, $p^{\prime}\in C_{qa}(k,n)$. Since $\|p-p^{\prime}\|<\epsilon$ and
$\epsilon$ was arbitrary, it follows that $p\in C_{qa}(k,n)$, as desired. ∎

######  Remark 4.9.

It appears to be an open question as to whether the conclusion of Proposition
4.8 holds if one replaces “TP” by “QWEP.” This might perhaps lead to a way to
separate these two classes as asked in Question 4.5.

######  Definition 4.10.

A $\mathrm{C}^{*}$-algebra $C$ has the strong Tsirelson property (STP) if it
has the TP and is not subhomogeneous.

######  Lemma 4.11.

A $\mathrm{C}^{*}$-algebra $C$ has the STP if and only if $(C,D)$ is a strong
Tsirelson pair for every non-subhomogeneous $\mathrm{C}^{*}$-algebra $D$.

######  Lemma 4.12.

The $\mathrm{C}^{*}$-algebras with STP are closed under r.w.i. subalgebras and
ultraproducts.

###### Proof.

This follows from Proposition 4.2 and the fact that the class of
$\mathrm{C}^{*}$-algebras that are not subhomogeneous is closed under r.w.i.
subalgebras and ultraproducts. ∎

##  5\. Model-theoretic connections

######  Theorem 5.1.

Both the class of $\mathrm{C}^{*}$-algebras with the TP and the class of
$\mathrm{C}^{*}$-algebras with the STP are elementary.

###### Proof.

We first deal with the case of $\mathrm{C}^{*}$-algebras with the TP. We must
show that this class is closed under ultraproducts and ultraroots. The first
item was established in Proposition 4.3. Since any $\mathrm{C}^{*}$-algebra is
r.w.i. in its ultrapower (see [5, Proposition 2(2)]), closure under ultraroots
also follows from Proposition 4.3.

Since the class of $\mathrm{C}^{*}$-algebras that are not subhomogeneous is
axiomatizable (see [3, Subsection 2.5.5]), the second statement of the theorem
follows from the first. ∎

By Proposition 4.7, both of the aforementioned classes are inductive.

While one might wish for an “explicit” axiomatization of the class of
$\mathrm{C}^{*}$-algebras with the TP or STP, the next result will show that
this cannot be the case.

######  Theorem 5.2.

There can be no effectively axiomatizable theory $T$ in the language of pairs
of $\mathrm{C}^{*}$-algebras such that all models of $T$ are Tsirelson pairs
and at least one model of $T$ is a strong Tsirelson pair.

###### Proof.

Suppose, towards a contradiction, that such a theory $T$ exists. Work now in
the language of pairs of $\mathrm{C}^{*}$-algebras expanded by a new unary
predicate symbol. One can then consider the effectively axiomatizable theory
$T^{\prime}$ whose models are of the form $(C,D,P)$, where $(C,D)$ is a model
of $T$ and $P(c,d)=\phi(c\otimes d)$ for some state $\phi$ on
$C\otimes_{\max}D$. That this class of structures is axiomatizable (and
effectively so) is due to the fact that states on the maximal tensor product
are just extensions of unital linear functionals on the algebraic tensor
product that are positive on the algebraic tensor product.

One can now reach a contradiction to the main result of [9] as follows. We
assume familiarity with nonlocal games and their quantum and commuting values
as defined in [9]. Given a nonlocal game $\mathfrak{G}$ with $k$ questions,
$n$ answers, probability distribution $\pi$, and decision predicate $D$, one
can consider the first-order sentence $\sigma_{\mathfrak{G}}$ in the language
of $T^{\prime}$ given by

$\sup_{A}\sup_{B}\sum_{(x,y)\in[k]}\pi(x,y)\sum_{(a,b)\in[n]}D(x,y,a,b)P(A^{x}_{a},B^{y}_{b}).$

Here, it is understood that $A$ ranges over $k$-tuples of POVMs of length $n$
from the first $\mathrm{C}^{*}$-algebra in the pair while $B$ ranges over
$k$-tuples of POVMs of length $n$ from the second $\mathrm{C}^{*}$-algebra in
the pair. That these quantifications are allowed, that is, that the sets in
question are definable, follows from Proposition 2.5. Since the modulus of
definability of these sets is explicit (in fact, by Corollary 2.6, it is
$\epsilon\mapsto\frac{\epsilon}{2}$), $\sigma_{\mathfrak{G}}$ is effectively
describable from the description of $\mathfrak{G}$. (See [6, Section 2] for
more on these matters.)

The assumptions on the theory $T$ imply that

$\sup\\{\sigma_{\mathfrak{G}}^{(C,D,P)}\ :\ (C,D,P)\models
T^{\prime}\\}=\operatorname{val}^{*}(\mathfrak{G}),$

where $\operatorname{val}^{*}(\mathfrak{G})$ is the quantum entangled value of
$\mathfrak{G}$. Since $T^{\prime}$ is effectively axiomatizable, the
Completeness Theorem for continuous logic implies that one can effectively
enumerate upper bounds for the left-hand side of the above display, whence
also for $\operatorname{val}^{*}(\mathfrak{G})$. Since effective lower bounds
for $\operatorname{val}^{*}(\mathfrak{G})$ always exist, this allows one to
effectively approximate $\operatorname{val}^{*}(\mathfrak{G})$, which, by the
main result of [9], would allow one to decide the halting problem, yielding a
contradiction. ∎

######  Remark 5.3.

The assumption of at least one model that is a strong Tsirelson pair is
necessary as the class of pairs of abelian $\mathrm{C}^{*}$-algebras is
clearly effectively axiomatizable and consists entirely of nuclear (and thus
Tsirelson) pairs.

######  Remark 5.4.

We can now finish the proof of Corollary 3.8. Recall that we claimed that if
one $C$ or $D$ is subhomogeneous, then $(C,D)$ is not a strong Tsirelson pair.
Suppose, towards, a contradiction that $C$ is $n$-subhomogeneous and $(C,D)$
is a strong Tsirelson pair. Then letting $T$ denote the theory of pairs of
$\mathrm{C}^{*}$-algebras for which the first element of the pair is
$n$-subhomogeneous, we see that $T$ is effectively axiomatizable (this follows
from the effective axiomatization of $n$-subhomogeneous
$\mathrm{C}^{*}$-algebras given in [3, Subsection 2.5.4]), all models of $T$
are nuclear pairs (and thus Tsirelson pairs), and has at least one model that
is a strong Tsirelson pair (by our contradiction assumption). This contradicts
the statement of Theorem 5.2.

######  Corollary 5.5.

There can be no effectively axiomatizable theory $T$ of
$\mathrm{C}^{*}$-algebras, all of whose models have the TP, and which has at
least one model that has the STP. In particular, neither the elementary class
of $\mathrm{C}^{*}$-algebras with the TP nor the elementary class of
$\mathrm{C}^{*}$-algebras with the STP are effectively axiomatizable.

The following consequence of the previous corollary improves upon [1, Theorem
2.2].

######  Corollary 5.6.

There can be no effectively axiomatizable theory $T$ of
$\mathrm{C}^{*}$-algebras, all of whose models have the QWEP property, and
which has at least one model that is not subhomogeneous.

###### Proof.

This follows immediately from Corollaries 4.4 and 5.5. ∎

Using Corollary 5.5, an argument identical to that establishing [1, Corollary
2.4] establishes the following:

######  Corollary 5.7.

The class of $\mathrm{C}^{*}$-algebras without the (S)TP is not closed under
ultraproducts.

##  6\. Conflicts of interests statements

The authors have no competing interests to declare that are relevant to the
content of this article.

## References

  * [1] J. Arulseelan, I. Goldbring, and B. Hart, _The undecidability of having QWEP_ , preprint.
  * [2] N. Brown and N. Ozawa, _C*-algebras and finite-dimensional appoximations_ , AMS Graduate Studies in Mathematics 88.
  * [3] I. Farah, B. Hart, M. Lupini, L. Robert, A. Tikuisis, A. Vignati and W. Winter, _Model Theory of $\mathrm{C}^{*}$-algebras_, Memoirs of the AMS 271(2021).
  * [4] T. Fritz. _Tsirelson’s problem and Kirchberg’s conjecture_ , Reviews in Mathematical Physics, 24(05):1250012 (2012).
  * [5] I. Goldbring, _Model theory and the QWEP problem_ , Illinois Journal of Mathematics 59 (2015), 795-799.
  * [6] I. Goldbring and B. Hart, _The universal theory of the hyperfinite II 1 factor is not computable_, preprint. arXiv 2006.05629.
  * [7] I. Goldbring and T. Sinclair, _Omitting types in operator systems_ , Indiana University Mathematics Journal 66 (2017), 821-844.
  * [8] U. Haagerup and S. Thorbjornsen, _A new application of random matrices: $\operatorname{Ext}(C^{*}_{\operatorname{red}}(\mathbb{F}_{2}))$ is not a group_, Annals of Mathematics 162 (2005), 711-775.
  * [9] Z. Ji, A. Natarajan, T. Vidick, J. Wright and H. Yuen, MIP* = RE, preprint, arxiv 2001.04383
  * [10] M. Junge, M. Navascues, C. Palazuelos, D. Perez-Garcia, V. Scholz, and R. Werner. Connes’ embedding problem and Tsirelson’s problem. Journal of Mathematical Physics, 52(1):012102, 2011.
  * [11] E. Kirchberg, _On nonsemisplit extensions, tensor products and exactness of group $\mathrm{C}^{*}$-algebras_, Invent. Math. 112 (1993), no. 3, 449–489.
  * [12] N. Ozawa, _Dixmier approximation and symmetric amenability for $\mathrm{C}^{*}$-algebras_, J. Math. Sci. Univ. Tokyo 20 (2013), 349–374. 66 (2017), 821-844.
  * [13] by same author, _About the Connes embedding conjecture_ , Japanese Journal of Mathematics, 8(1):147–183, 2013.
  * [14] G. Pisier, _On the lifting property for $\mathrm{C}^{*}$-algebras_, to appear in the Journal of Noncommutative Geometry.
  * [15] by same author, _Tensor products of $\mathrm{C}^{*}$-algebras and operator spaces: The Connes-Kirchberg problem_, London Mathematical Society Student Texts Book 96 (2020).
  * [16] B. Tsirelson. _Some results and problems on quantum Bell-type inequalities_ , Hadronic J. Suppl. 8 (1993), 329- 345\.
  * [17] B. Tsirelson. _Bell inequalities and operator algebras_ , available at https://www.tau .ac.il/ tsirel/download/bellopalg.pdf


# Scenario-Based Curriculum Generation for Multi-Agent Autonomous Driving

Axel Brunnbauer1, Luigi Berducci1, Peter Priller2, Dejan Nickovic3, Radu
Grosu1 1 CPS, Technische Universität Wien (TU Wien), Austria2 AVL List GmbH3
Austrian Institute of Technology, AITa Correspondence:
<EMAIL_ADDRESS>
###### Abstract

The automated generation of diverse and complex training scenarios has been an
important ingredient in many complex learning tasks. Especially in real-world
application domains, such as autonomous driving, auto-curriculum generation is
considered vital for obtaining robust and general policies. However, crafting
traffic scenarios with multiple, heterogeneous agents is typically considered
as a tedious and time-consuming task, especially in more complex simulation
environments. In our work, we introduce MATS-Gym, a Multi-Agent Traffic
Scenario framework to train agents in CARLA, a high-fidelity driving
simulator. MATS-Gym is a multi-agent training framework for autonomous driving
that uses partial scenario specifications to generate traffic scenarios with
variable numbers of agents. This paper unifies various existing approaches to
traffic scenario description into a single training framework and demonstrates
how it can be integrated with techniques from unsupervised environment design
to automate the generation of adaptive auto-curricula. The code is available
at https://github.com/AutonomousDrivingExaminer/mats-gym.

##  I Introduction

Figure 1:  This multi-agent scenario illustrates an intersection where five
vehicles navigate according to assigned routes, with three pedestrians
observed on the sidewalk adjacent to the ego vehicle. A visual representation
of the simulation is depicted in the center, while the scenario description
from which the simulation parameters are sampled is described on the right. On
the left, we depict other scenarios sampled from the same Scenic description.

Autonomous driving (AD) stands as a groundbreaking frontier in transportation
technology, offering the promise of safer and more efficient roadways.
However, achieving fully autonomous vehicles has been a long-standing goal for
researchers across generations. As recent advancements enable the deployment
of more vehicles in cities worldwide, extensive testing in a diverse range of
scenarios becomes essential. For every kilometer driven by AD systems on real-
world roads, they traverse several orders of magnitude more in simulation.
Thus, access to high-fidelity simulation environments plays a crucial role in
advancing the field of AD. Open-source simulation software like CARLA [1] has
empowered researchers and practitioners to evaluate and train AD algorithms in
realistic environments. Despite this, creating diverse and lifelike traffic
scenarios, incorporating varying numbers of agents, remains a laborious and
time-consuming task. To tackle this challenge, scenario specification
languages like Scenic [2] and
OpenSCENARIO111https://www.asam.net/standards/detail/openscenario/ have been
introduced. These scenario specification approaches streamline the process of
defining logical traffic scenarios and are well-integrated into simulation
environments such as CARLA. However, we observe that (1) infrastructure aiding
the training and evaluation of AD stacks is scattered across numerous ad hoc
implementations, (2) integration of scenario specifications and simulation
environments lacks support for training in multi-agent systems and (3) limited
support for approaches targeting open-ended curriculum generation. To address
these shortcomings, we propose MATS-Gym, an open-source multi-agent training
framework that leverages existing traffic scenario-specification approaches
for CARLA. Additionally, we demonstrate how MATS-Gym can be used to implement
novel auto-curriculum approaches [3] for generating open-ended traffic
scenarios with adaptive difficulty. Hence, the key contributions of MATS-Gym
are:

  1. 1.

A multi-agent, scenario-based AD training and evaluation framework for the
CARLA simulator.

  2. 2.

Experimental evaluation to demonstrate MATS-Gym’s practicality for multi-agent
learning and ability to generate auto-curricula from scenario descriptions.

  3. 3.

Extensive comparison and classification in the context of AD simulation
frameworks.

###  I-A Illustrative Scenario

In this scenario, as depicted in Figure 1, five vehicles converge on a four-
way intersection, each following a predefined route toward individual goal
locations. Additionally, three pedestrians traverse the sidewalk near the
intersection. All vehicles are equipped with sensors, including cameras,
odometry, and velocity sensors. Despite the simplicity of this setting, the
potential variations are vast, depending on initial conditions such as the
number and types of actors and their behaviors. Effective training needs
exposure to diverse scenario variations for robust generalization. Moreover,
meticulous selection of progressively complex challenges is crucial but
requires expert knowledge. In the following, we introduce MATS-gym and
showcase its effectiveness in generating scenarios of varying difficulty.

##  II Related Work

TABLE I: Comparison of various traffic scenario frameworks Framework | Scenario Specification | Realism | Traffic Types | Training
---|---|---|---|---
Sampling | Scriptable | Sensors | Visual | Physics | Highway | Urban | RL | MARL
HighwayEnv [4] | ✗ | $\sim$ | ✗ | ✗ | ✗ | ✓ | $\sim$ | ✓ | ✓
BARK [5] | ✗ | ✓ | $\sim$ | ✗ | ✗ | ✓ | $\sim$ | ✓ | ✗
CommonRoads [6, 7] | ✗ | ✓ | ✗ | ✗ | ✓ | ✓ | $\sim$ | ✓ | ✗
SMARTS [8] | ✓ | ✓ | $\sim$ | ✗ | ✓ | ✓ | $\sim$ | ✓ | ✓
MetaDrive [9] | ✓ | $\sim$ | ✓ | ✗ | $\sim$ | ✓ | ✓ | ✓ | ✓
DI Drive [10] | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗
MACAD-Gym [11] | ✗ | $\sim$ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
MATS-Gym (ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

###  II-A Training Frameworks for AD

We compare our approach to other frameworks commonly utilized for training and
evaluating AD agents. We identified several popular frameworks and assess them
based on several key aspects crucial for our use-case. These aspects include
ease and expressivity of scenario specifications, simulation realism,
supported traffic types, and integrated training interfaces. Table I
summarizes the comparison.

HighwayEnv [4], BARK [5], SMARTS [8], and the CommonRoads Suite [6, 7] offer
alternatives to full-fledged game-engine simulations like CARLA. They focus on
modeling vehicle dynamics and motion planning, with CommonRoads providing more
sophisticated multi-body dynamics compared to HighwayEnv and BARK. While
simpler environments are computationally efficient, they lack complex sensor
models and realistic observations. These frameworks mainly involve vehicles
and lack pedestrian or object modeling found in urban traffic. BARK and
CommonRoads allow scriptable scenarios, while HighwayEnv supports custom
scenarios through code re-implementation. SMARTS allows to model urban traffic
scenarios with traffic lights and traffic signs. However, since the focus is
not on graphical realism, those scenarios can not reach the visual diversity
of traffic scenes in CARLA, which is necessary to train and evaluate vision
based AD stacks.

MetaDrive [9] is a training framework that builds on the Panda3D engine. It
offers a wide range of features such as procedural road generation and
incorporating scenarios from traffic datasets. However, scenarios can not be
specified by behavioral building blocks or predefined sub-scenarios. Moreover,
its engine limits the structural variety of environments that can be achieved
in CARLA.

DI Drive Gym [10] and MACAD Gym [11] are both training frameworks designed
around CARLA. DI Drive Gym focuses on single-agent autonomous driving systems
and interfaces with ScenarioRunner scenarios, lacking support for training
multi-agent systems or scenario generation via sampling. MACAD Gym, supports
multi-agent training but offers limited scenario generation capabilities,
allowing only manual specification of initial conditions.

In this comparison, we focus on relevant aspects for learning-based approaches
in realistic urban traffic scenarios. We acknowledge our criteria may not be
complete and there might be aspects that could be relevant in other use cases.
However, we tried to cover a broad spectrum of frameworks that could be used
for training and evaluation.

###  II-B Curriculum Learning

Generating safety-critical scenarios is an active area of research and has
been addressed in numerous works in recent years [12]. However, training
agents in complex environments requires presenting levels at the _right
difficulty_ , matching their capabilities to generate meaningful learning
signals. Curriculum Learning (CL) and Auto CL (ACL) methods have emerged to
progressively expose agents to more complex environments. While traditional CL
involves manually designing curricula, ACL automates this process by
dynamically presenting suitable scenarios based on agent performance. For an
overview, we refer to [13].

Previous works also applied ACL to AD scenarios [14, 15]. A novel research
direction in ACL is Unsupervised Environment Design (UED), where task creation
itself is treated as an optimization problem. This is formalized through
Underspecified Partially Observable Markov Decision Processes (UPOMDP),
allowing for flexible reparameterization of environments during runtime. A
UPOMDP is a tuple

$\mathcal{M}=\langle
A,O,\Theta,{S}^{\mathcal{M}},{\mathcal{T}}^{\mathcal{M}},{\mathcal{I}}^{\mathcal{M}},{\mathcal{R}}^{\mathcal{M}},\gamma\rangle,$

which differs from a standard POMDP only in having a set of free parameters
$\Theta$, which can be chosen at any time during runtime. Recent works have
explored different algorithms for UED. Prioritized Level Replay (PLR) utilizes
domain randomization to generate parameters and a level replay buffer to
resample scenarios proportional to their regret, a measure of suboptimality
[16]. PAIRED [17] optimizes the level generator via reinforcement learning
(RL) to maximize regret, while REPAIRED [3] combines PLR with the level
generation approach from PAIRED in a Dual Curriculum Design (DCD) framework.
Other approaches use evolutionary algorithms to optimize the generator and
include a level editor for environment modifications [18, 19].

##  III MATS-Gym: Architecture and Features

MATS-Gym is a multi-agent training and evaluation framework that allows to
generate diverse traffic scenarios in CARLA, a high-fidelity traffic
simulator. The framework is designed to reconcile scenario execution engines,
such as Scenic [2] and ScenarioRunner [20], and training frameworks in the AD
research community. We model the interaction between the environment and the
agents as a Partially Observable Stochastic Game (POSG) where all agents take
actions simultaneously at each timestep. Thus, a natural choice for the
environment interface is the PettingZoo API [21], the multi-agent version of
OpenAI Gym [22].

###  III-A Architecture

Figure 2: MATS-Gym Architecture: We build on ScenarioRunner the scenario
execution infrastructure. Blue: our contributions, Green: existing
infrastructure from ScenarioRunner. Figure 3:  Action abstractions include
low-level control, waypoints for local planning, and macro-actions for
behavior planning.

We build MATS-Gym around the core of ScenarioRunner, which allows us to
leverage their existing infrastructure to specify scenarios, define event
observers and execute scenarios, as depicted in Figure 2. Our framework serves
as an execution handler and training interface that extends the capabilities
of those scenarios to multi-agent environments. Furthermore, we provide a set
of wrappers to enhance observation and action spaces in order to facilitate
versatile training environments. Additionally, we also provide a task
infrastructure that allows to easily define reward functions and termination
conditions. One of the design goals of MATS-Gym is interoperability with
current scenario engines for CARLA. Compatibility with ScenarioRunner and thus
OpenSCENARIO is inherently built in. As a consequence, our framework can be
also used as a training and evaluation framework for the CARLA AD
Challenge222https://leaderboard.carla.org/challenge/. Furthermore, we connect
another complementary scenario specification engine for CARLA by implementing
an adapter for Scenic, a probabilistic programming language for building
complex scenarios in various domains. Scenic comes with a rich set of building
blocks for specifying distributions over traffic scenarios that adhere to
predefined constraints.

###  III-B Observations, Actions and Tasks

MATS-Gym offers comprehensive infrastructure for agent state retrieval,
providing ground truth information such as position, velocity vectors, and key
traffic events like violations and collisions. Its modular design allows for
flexible specification of additional observation types. It offers a versatile
infrastructure for configuring sensor suites for agents, akin to the CARLA AD
Challenge, supporting various sensors such as cameras, LiDAR, and Radar. It
extends the CARLA BirdEyeView [23] framework for bird’s-eye view observations,
constructing occupancy gridmaps around agents encoding lane markings, traffic
lights, signs, and other road users. Agents can also access road network
information and vector-based representations of the road layout. We account
for different type of tasks by providing action spaces at various levels of
abstraction. On the lowest level, agents can issue throttle, breaking and
steering commands at high frequencies. On a higher level, agents can instead
provide a target waypoint which is tracked by a PID controller for smooth
driving trajectories. We also provide a discrete action space for high-level
driving commands, suitable for route planning, behavior prediction, or multi-
agent interaction. Fig. 3 visualizes the available action spaces. Furthermore,
MATS-Gym allows for defining custom tasks and offers pre-defined tasks related
to autonomous driving, including route following, infraction avoidance, and
driving comfort, crucial for designing reward functions in reinforcement
learning approaches.

##  IV Scenario Based Curriculum Generation

In the following, we will outline our approach to use Scenic’s parametrizable
scenario specifications as a way to build an adaptive scenario generation
procedure. We adopt a dual-curriculum design algorithm based on REPAIRED [3],
a recent algorithm combining PLR with an adaptive level generator. At each
timestep, we either generate a new level by sampling environment parameters
from $\tilde{\pi}$ or we sample a replay level from our PLR buffer. The buffer
samples levels with a probability proportional to their regret and a staleness
metric, as described in the original approach. Then, we sample a trajectory
from the environment to update the student policy $\pi$ and to compute the
estimated regret for the current level, which is then used to update the
buffer and the environment generator. Algorithm 1 outlines this approach.

Differently than the original, we do not use an antagonist agent to estimate
the regret, due to computational constraints. Instead, we use the _Maximum
Monte Carlo_ [3] regret estimator, formally described as:

$\frac{1}{T}\sum_{t=0}^{T}R_{\text{max}}-V(s_{t}).$ (1)

This regret formulation compares the maximum achieved return with the average
state value for each episode. We choose this alternative to avoid training
another policy just for regret estimation, which would entail twice as much
simulation steps. Furthermore, for the level generator optimization, we use
the tree-structured Parzen estimator algorithm, implemented in the
hyperparameter optimization framework Optuna [24] instead of RL.

Algorithm 1 MATS-REPAIRED

1: Input: policy $\pi$, environment generator $\tilde{\pi}$, buffer $\Lambda$

2: while not converged do

3: Sample replay decision $d\sim P_{D}(d)$

4: if d=0 then

5: Generate environment parameters $\theta\sim\tilde{\pi}$.

6: Insert $\theta$ into buffer $\Lambda$.

7: else

8: Sample replay level, $\theta\sim\Lambda$.

9: end if

10: Collect trajectory $\tau$ using policy $\pi$.

11: Compute regret $R$ using Eq. 1.

12: Update policy $\pi$ with $\tau$.

13: Update level buffer: $\Lambda\leftarrow(\theta,R)$.

14: if level was generated then

15: Update environment generator $\tilde{\pi}$ with $R$.

16: end if

17: end while

##  V Experiments

We conduct two experiments in which we show that MATS-Gym can be used to (1)
train multiple agents in an implicit coordination task and (2) for auto-
curriculum generation. Both experiments share the base scenario in which
agents are tasked with successfully navigating along a route through an
intersection in urban environments, similar to the one introduced in the
illustrative example. The reward function for this task is a combination of a
progress-based and a cruise speed reward:

$r_{t}=r_{\text{progress}}+r_{\text{cruise}},$ (2)

where $r_{\text{progress}}=p_{t}-p_{t-1}$ is the progress made along the route
since the last time-step and
$r_{\text{cruise}}=min(\frac{v_{t}}{v_{\text{target}}},1)$ rewards the agent
to drive at a cruise velocity $v_{target}$.

For the optimization of the agent policy we use Proximal Policy Optimization
(PPO) [25]. The agent receives birdview observations as input and can
therefore observe the drivable road area, lane markings, route and other
traffic participants.

###  V-A Learning in Different Action Spaces

Figure 4:  Learning curves for I-PPO under different action definitions and
the impact on episodic return, collisions and route completion. Performance
reports mean and standard deviation over 5 consecutive policy updates of the
same run.

In this experiment, we demonstrate our framework’s usability for multi-agent
training and discuss how the choice of actions affects the training of
cooperative agents.

Scenario Description. Based on the initial scenario, we let four vehicles
navigate through a four-way intersection, each with a designated route and a
time limit to reach their destination. The starting conditions and goals vary
per episode as outlined in Scenic. To succeed, the agent must manage its
vehicle, collaborate with other traffic participants, and follow the
prescribed route. Termination occurs when all agents are stopped.

Training Setup. Each vehicle is governed by an independent policy, which
perceives a birdview observation encoding the road layout, vehicle positions
relative to the agent, and route information. To study the impact of actions
spaces, we consider the same task definition with continuous, waypoint and
macro actions. Each of the proposed actions have different control
frequencies: continuous actions are repeated twice, waypoint actions are
executed for 5 steps, and macro actions persist for 10 steps with 0.05s per
step. We train the policies with Independent PPO [26] and account for the
different action frequencies by fixing the training budget to $175$ policy
updates, each over a batch of $2048$ transitions.

Results. Figure 4 shows the learning curves with respect to average episodic
reward, collisions and route completions, evaluated over the batch of training
data for each policy update. We aggregate the metrics over $k$-size bins of
policy updates ($k=5$), reporting the mean and standard deviation.

We observe different characteristics of the learned policies. Low-level and
waypoint target actions incur a high number of collisions, especially in early
training stages. On the other hand, macro actions make use of a basic lane
keeping and collision avoidance controller, which prevents agents from leaving
drivable areas and avoids most of the accidents, leading to a low collision
rate throughout the training. This restriction comes at the cost of frequent
deadlocks, which result in a low route completion rate. Less restrictive
action spaces allow agents to leave the predefined lane to conduct evasive
maneuvers, leading to higher average route completion in later training
stages.

This experiment emphasizes how the action space design profoundly shapes
emergent behavior and affects the learning task’s difficulty. Careful
modelling of the problem, encompassing observations and actions, is crucial
for multi-agent learning and serves as a key feature in training frameworks.
Importantly, it demonstrates that there isn’t a universally optimal level of
abstraction for the action space; instead, the choice depends on the problem
at hand.

###  V-B Scenario Based Environment Design

Figure 5:  Learning curves of average episodic return, route completion during
training and evaluation over 3 seeds with one standard deviation. We also
report the average regret of the level buffers of PLR and our DCD approach
over timesteps.

In this experiment, we assess our DCD approach’s ability to align the training
scenario distribution with the agent’s capabilities. We examine its impact on
agent performance and the distribution of generated levels. Additionally, we
compare its effectiveness with PLR and basic domain randomization. We analyze
the evolution of sampled scenario parameters for each approach and evaluate
their adaptation to the agent’s capabilities, particularly in the early stages
of training when the agent struggles with complex scenarios.

Scenario Description. We parameterize the base scenario introduced before with
a variety of discrete and continuous variables, which characterize the
difficulty of the generated scenes. Intuitively, learning to follow a straight
path through an empty intersection is easier than performing an unprotected
left turn in a busy intersection where other traffic participants are driving
recklessly. In our experiment, we define the following parameters that can be
sampled:

  * •

Assigned route.

  * •

Number of other vehicles in the intersection.

  * •

Target speed for the other vehicles’ controllers.

  * •

Whether the other vehicles ignores others recklessly.

  * •

Whether the other vehicles respect traffic lights.

Results.

Figure 6:  Evolution of the parameters that are sampled during the training
with DCD (top) and domain randomization (bottom). For each parameter, we
report mean and standard deviation over the batch of training data, indicate
the direction of major difficulty ($\uparrow$), and the linear interpolation
to highlight the general trend (red). We observe that DCD steers the parameter
distribution towards configurations that are supposedly easier to solve:
straight crossings, fewer and safer NPCs, etc. This leads us to the conclusion
that DCD adapts the training distribution faster to the performance level of
the agent.

In Figure 5, we compare learning curves over 150K environment steps. Both PLR
and DR show similar performance in terms of episodic returns and route
completion during training, indicating comparable scenario difficulty. DCD
demonstrates a notable increase in training signal, suggesting adaptation by
generating progressively easier scenarios in the initial stages of training.
We evaluate all approaches on the same 12 hold-out levels, representing
different maneuvers with varying numbers of NPCs. Although the performance on
the hold-out set is comparable in early stages of the training, DCD and PLR do
not suffer as much from performance drops in later stages.

Figure 6 provides insight into the progression of scenario parameters
throughout training. DCD shifts the scenario distribution towards less
challenging levels, characterized by straight maneuvers for the agent and a
low number of vehicles in intersections. For comparison, we also show how DR
naturally maintains a uniform distribution over environment parameters.

We also investigate the distribution of level parameters in the replay buffers
of PLR and REPAIRED. Figure 7 depicts the average regret associated with
various parameter combinations at four distinct checkpoints during the
training process. For visualization purposes, we focus on combinations
involving the maneuver type and the number of other vehicles (NPCs) present in
the intersection. Observing the data, it becomes evident that the average
regrets of Dual-Curriculum Design (DCD) levels tend to be higher and exhibit a
narrower distribution across fewer parameter combinations compared to those of
PLR alone. This trend can be attributed to the presence of the adaptive level
sampler, which facilitates expedited convergence towards parameter
combinations that are more pertinent to the task at hand.

Our experiments showcase the effectiveness of automatic curriculum design in
aligning scenario generation with the agent’s capabilities. Additionally,
optimizing the scenario generation process, rather than just the resampling
procedure, accelerates the adaptation of the level distribution. Through this
experiment, we aim to demonstrate that integrating UED approaches into MATS-
Gym facilitates future research in curriculum generation for AD applications.

Figure 7: For each pair of maneuver type and number of NPCs, we average the
regrets of all levels with the same parameters in the buffer at four
checkpoints throughout the training. We observe that the environment
generation policy of DCD leads to more narrowly concentrated regret on fewer
level configurations, suggesting faster adaption of the level sampling.

##  VI Conclusion

In this work, we present MATS-Gym, a multi-agent training framework capable of
generating scenario-based auto-curricula for AD tasks in CARLA. MATS-Gym is
open to multiple, commonly used frameworks for traffic scenario specification
and sampling, including ScenarioRunner, OpenSCENARIO and Scenic. Leveraging
Scenic enables sampling from scenario distributions and the integration with
UED approaches. This compatibility resembles a promising way to generate more
relevant and realistic training scenarios for AD. We demonstrate the usability
of our framework in two experiments of multi-agent training and automatic
curriculum generation, respectively. By introducing MATS-Gym and demonstrating
its application in various experiments, we contribute to ongoing efforts in
advancing multi-agent training for autonomous driving.

For future work, we intend to explore the potential of MATS-Gym in adversarial
training within multi-agent systems and the integration of generative models
for traffic scenario generation.

## References

  * [1] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An open urban driving simulator,” in _Proceedings of the 1st Annual Conference on Robot Learning_ , ser. Proceedings of Machine Learning Research, S. Levine, V. Vanhoucke, and K. Goldberg, Eds., vol. 78. PMLR, 2017, pp. 1–16. [Online]. Available: https://proceedings.mlr.press/v78/dosovitskiy17a.html
  * [2] D. J. Fremont, T. Dreossi, S. Ghosh, X. Yue, A. L. Sangiovanni-Vincentelli, and S. A. Seshia, “Scenic: A language for scenario specification and scene generation,” in _Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation_ , 2019, pp. 63–78. [Online]. Available: http://arxiv.org/abs/1809.09310
  * [3] M. Jiang, M. Dennis, J. Parker-Holder, J. Foerster, E. Grefenstette, and T. Rocktäschel, “Replay-guided adversarial environment design.” [Online]. Available: http://arxiv.org/abs/2110.02439
  * [4] E. Leurent, “An environment for autonomous driving decision-making,” publication Title: GitHub repository. [Online]. Available: https://github.com/eleurent/highway-env
  * [5] J. Bernhard, K. Esterle, P. Hart, and T. Kessler, “BARK: Open behavior benchmarking in multi-agent environments,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 6201–6208. [Online]. Available: https://ieeexplore.ieee.org/document/9341222/
  * [6] M. Althoff, M. Koschi, and S. Manzinger, “CommonRoad: Composable benchmarks for motion planning on roads,” in _2017 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2017, pp. 719–726. [Online]. Available: http://ieeexplore.ieee.org/document/7995802/
  * [7] X. Wang, H. Krasowski, and M. Althoff, “CommonRoad-RL: A configurable reinforcement learning environment for motion planning of autonomous vehicles,” in _IEEE International Conference on Intelligent Transportation Systems (ITSC)_ , 2021.
  * [8] M. Zhou, J. Luo, J. Villella, Y. Yang, D. Rusu, J. Miao _et al._ , “SMARTS: Scalable multi-agent reinforcement learning training school for autonomous driving,” publication Title: Proceedings of the 4th Conference on Robot Learning (CoRL). [Online]. Available: https://arxiv.org/abs/2010.09776
  * [9] Q. Li, Z. Peng, Z. Xue, Q. Zhang, and B. Zhou, “Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning,” _arXiv preprint arXiv:2109.12674_ , 2021.
  * [10] D.-d. Contributors, “DI-drive: OpenDILab decision intelligence platform for autonomous driving simulation.” [Online]. Available: https://github.com/opendilab/DI-drive
  * [11] P. Palanisamy, “Multi-agent connected autonomous driving using deep reinforcement learning,” _2020 International Joint Conference on Neural Networks (IJCNN)_ , pp. 1–7, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:207852404
  * [12] W. Ding, C. Xu, M. Arief, H. Lin, B. Li, and D. Zhao, “A survey on safety-critical driving scenario generation—a methodological perspective,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 24, no. 7, pp. 6971–6988, 2023.
  * [13] R. Portelas, C. Colas, L. Weng, K. Hofmann, and P.-Y. Oudeyer, “Automatic curriculum learning for deep rl: a short survey,” in _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence_ , ser. IJCAI’20, 2021.
  * [14] Z. Qiao, K. Muelling, J. M. Dolan, P. Palanisamy, and P. Mudalige, “Automatically generated curriculum based reinforcement learning for autonomous vehicles in urban environment,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_ , 2018, pp. 1233–1238.
  * [15] L. Anzalone, P. Barra, S. Barra, A. Castiglione, and M. Nappi, “An end-to-end curriculum learning approach for autonomous driving scenarios,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 10, pp. 19 817–19 826, 2022.
  * [16] M. Jiang, E. Grefenstette, and T. Rocktäschel, “Prioritized level replay.” [Online]. Available: http://arxiv.org/abs/2010.03934
  * [17] M. Dennis, N. Jaques, E. Vinitsky, A. Bayen, S. Russell, A. Critch, and S. Levine, “Emergent complexity and zero-shot transfer via unsupervised environment design.” [Online]. Available: http://arxiv.org/abs/2012.02096
  * [18] J. Parker-Holder, M. Jiang, M. Dennis, M. Samvelyan, J. Foerster, E. Grefenstette, and T. Rocktäschel, “Evolving curricula with regret-based environment design.” [Online]. Available: http://arxiv.org/abs/2203.01302
  * [19] I. Mediratta, M. Jiang, J. Parker-Holder, M. Dennis, E. Vinitsky, and T. Rocktäschel, “Stabilizing unsupervised environment design with a learned adversary.” [Online]. Available: http://arxiv.org/abs/2308.10797
  * [20] “ScenarioRunner — Traffic scenario definition and execution engine,” https://github.com/carla-simulator/scenario_runner, [Accessed 12-03-2024].
  * [21] J. Terry, B. Black, N. Grammel, M. Jayakumar, A. Hari, R. Sullivan _et al._ , “PettingZoo: Gym for multi-agent reinforcement learning,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 15 032–15 043. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2021/file/7ed2d3454c5eea71148b11d0c25104ff-Paper.pdf
  * [22] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” _arXiv preprint arXiv:1606.01540_ , 2016.
  * [23] M. Martyniak, “carla-birdeye-view,” [Accessed 12-03-2024]. [Online]. Available: https://github.com/deepsense-ai/carla-birdeye-view
  * [24] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next-generation hyperparameter optimization framework,” 2019.
  * [25] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017.
  * [26] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of ppo in cooperative multi-agent games,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 24 611–24 624, 2022.