# Supermultiplet of $\beta-$ deformations from twistors

Segundo P. Milián  Instituto de Física Teórica, Universidade Estadual Paulista
R. Dr. Bento Teobaldo Ferraz 271, Bloco II- Barra Funda
CEP:01140-070-São Paulo, Brasil

###### Abstract

We consider the supermultiplet of linearized beta-deformation of
$\mathcal{N}=4$ Super Yang-Mills(SYM). It was previously studied on the
gravitational side. We study the supermultiplet of beta-deformations on the
field theory side and we compare two finite-dimensional representations of
$psl(4|4,\bf{R})$ algebra. We show that they are related by an intertwining
operator. We develop a twistor-based approach which could be useful for
studying other finite-dimensional and nonunitary representations in AdS/CFT
correspondence.

###### Contents

  1. 1 Introduction
  2. 2 Marginal Deformations in $\mathcal{N}=4$ SYM theory and $psl(4|4,{\bf{R}})$
    1. 2.1 The superalgebra $psl(4|4,\bf{R})$
      1. 2.1.1 Oscillator representation of $psl(4|4,\bf{R})$ superalgebra
    2. 2.2 $\mathcal{N}=4$ SYM fields in terms of oscillators and supertwistors
      1. 2.2.1 From oscillators representation to supertwistors
      2. 2.2.2 Single trace operators
  3. 3 The structure of the supermultiplet
    1. 3.1 Descendants multiplets from $\mathcal{O}[B]$
      1. 3.1.1 Action of $q$ and $\bar{q}$ on $\mathcal{O}[B]$
      2. 3.1.2 Action of $s$ and $\bar{s}$ onto $\mathcal{O}[B]$
  4. A $\mathcal{O}[B]$ is conformal invariant

##  1 Introduction

Much progress in order to understand the AdS/CFT correspondence has been
achieved since it was formulated [1, 2]. An important tool is the study of
small perurbations. In the CFT side they correspond to deformations of the SYM
action.

Small perturbations in the supergravity(SUGRA) side correspond to small
fluctuations of the classical SUGRA fields around their "vacuum" values in the
maximally symmetric background solution, $AdS_{5}\times S^{5}$. There was
extensive study of small fluctuations [3]. However, even at the linearized
level, the research was mainly focused on unitary representations [4]. We feel
that non-unitary representations are also important and have not been
sufficiently studied in this context. In particular, there are finite-
dimensional representations [5]. To the best of our knowledge, the
classification of finite-dimensional representations is missing.

The beta-deformation is probably the simplest example of a finite-dimensional
representation, it is very well studied on the SUGRA side [6], [8], [9], [7].
On the field theory side, it is a particular case of the deformation studied
by Leigh and Strassler [10]. It preserves $\mathcal{N}=1$ SUSY and depends on
two complex parameters.

However, the full supermultiplet has never been studied on the field theory
side. And we fill this gap with the present paper. Moreover, we develop a new
method, which is hopefully useful also for other non-unitary representations.
Namely, we consider the deformation of the SYM action, the one considered in
[10]. It is evaluated on the tensor product of singleton representations. We
will use the oscillator representation of the singleton representation of
$psl(4|4,\bf{R})$ [11]; for a review of the superconformal algebra see [12].

The $\mathcal{N}=4$ SYM admits a formulation in twistor space [13], see also
[14]. A vector in the singleton representation can be understood as a wave
function of the free field on ${\bf{R}}^{2,2}$ supported on the
$\alpha$-plane. This method allows us to reduce the problem to finding the
fininite-dimensional invariant subspace in the tensor product of three
singleton representations. Basically, in order to classify deformations of the
SYM action, it is enough to evaluate them on tensor products of the spaces of
solutions of free equations, which are dual to singletons.

Plan of the paper We make a review of $\mathcal{N}=4$ SYM theory and
$psl(4|4,\bf{R})$ superalgebra. Namely, we focus on the oscillator
representation of $psl(4|4,\bf{R})$, which we call oscillator picture. We also
review the relation between the oscillator picture and supertwistors. We
discuss how to write single trace operators on both pictures in Section 2.
Then in Section 3 we present the structure of the supermultiplet of
$\beta-$deformation on the field theory side. Using the oscillator picture we
show the conformal invariance of $\mathcal{O}[B]$ in Appendix A.

##  2 Marginal Deformations in $\mathcal{N}=4$ SYM theory and
$psl(4|4,{\bf{R}})$

${\mathcal{N}=4}$ SYM is a field theory with the maximal amount of
supersymmetry in four dimensions and is unique [15], for an extended review
see Chapter 3 of [16]. The action of this field theory is

$\displaystyle S_{4}$ $\displaystyle=$ $\displaystyle-\int
d^{4}x\big{(}\frac{1}{4}F^{(a)}_{\mu\nu}F^{\mu\nu(a)}+\frac{1}{4}D_{\mu}\phi^{AB(a)}D^{\mu}\phi_{AB}^{(a)}+i\bar{\psi}^{(a)}_{A}\bar{\sigma}^{\mu}D_{\mu}\psi^{(a)A}+\frac{\sqrt{2}}{2}g_{YM}f_{abc}\psi^{\alpha
A(a)}\phi_{AB}^{(b)}\psi_{\alpha}^{B(c)}$ (1) $\displaystyle+$
$\displaystyle\frac{g^{2}_{YM}}{16}f_{abc}f_{ade}\phi^{AB(b)}\phi^{CD(c)}\phi_{AB}^{(d)}\phi_{CD}^{(e)}-\frac{\sqrt{2}}{2}g_{YM}f_{abc}\bar{\psi}^{(a)}_{\dot{\beta}A}\phi^{AB(b)}\bar{\psi}^{\dot{\beta}(c)}_{B}\big{)},$

where capital letters run from $1$ to $4$.

$\mathcal{N}=4$ SYM admits a formulation in terms of $\mathcal{N}=1$ language,
one of the four fermions is combined with the gauge field to form the vector
superfield. The remaining fermions are combined with three complex scalar, to
form three chiral superfield. The superpotential $\mathcal{W}$ is given by

$\mathcal{W}=\frac{g_{YM}\sqrt{2}}{3!}\epsilon_{ijk}f_{abc}\Phi^{i}_{a}\Phi^{j}_{b}\Phi^{k}_{c},$
(2)

where $i,j,k$ indices running from $1$ to $3$, $\Phi^{i}$ denote the three
chiral superfields and $a,b,c$ denote gauge group SU(N) indices.

In this work we are interested in study deformations of this field theory. To
begin with, let us start with the deformation studied by Leigh and Strassler
[10]. The superpotential (2) is deformed to

$\displaystyle\mathcal{W}\mapsto\mathcal{W}+\frac{1}{6}h_{ijk}d_{abc}\Phi^{i}_{a}\Phi^{j}_{b}\Phi^{k}_{c},$
(3)

where $d_{abc}=\operatorname{Tr}(T_{a}\\{T_{b},T_{c}\\})$. The action at
linear order in $h$ is

$\displaystyle S=S^{\text{free}}_{4}-\frac{1}{4}\int
d^{4}xB^{[MN]}_{(IJ)}d_{abc}\psi^{(a)I}\psi^{(b)J}\phi_{MN}^{(c)}+\text{h.c.}\phantom{-},$
(4)

the deformation is conformal invariant at classical level for all values of
$h_{ijk}$. $S^{\text{free}}_{4}$ denotes the action of $\mathcal{N}=4$ SYM in
free field theory. Here the coupling $B^{[MN]}_{(IJ)}$ obeys the tracelessness
condition [7]

$\displaystyle B^{[MN]}_{(IN)}$ $\displaystyle=$ $\displaystyle 0,$ (5)

where $B^{[mn]}_{(ij)}=h_{ijl}\epsilon^{lmn}$, with latin indices running from
$1$ to $3$. We will see that this condition is crucial for matching with the
field theory side.

###  2.1 The superalgebra $psl(4|4,\bf{R})$

The symmetry group of the action (1) is $PSU(2,2|4)$. Local operators of
$\mathcal{N}=4$ SYM are arranged in multiplets of the algebra
$psl(4|4,\bf{R})$ [17]. The generators of the algebra $psl(4|4,\bf{R})$ are
the $sl(2)\times sl(2)$ rotations
$l_{\beta}^{\;\alpha},\;\dot{l}_{\dot{\beta}}^{\;\dot{\alpha}}$, the $sl(4)$
rotations, the translation $p_{\alpha\dot{\alpha}}$, the conformal generator
$k^{\dot{\alpha}\alpha}$, the supersymmetry generators $q_{A\alpha}$,
$\bar{q}^{A}_{\dot{\alpha}}$, the superconformal generators $s^{A\alpha}$,
$\bar{s}^{\dot{\alpha}}_{A}$ as well as dilatation generator $D$.

Using $\mathcal{N}=1$ language to write down the action $S_{4}$, it breaks the
original ${\mathcal{R}}$ symmetry into $SU(3)\times U(1)$ .

####  2.1.1 Oscillator representation of $psl(4|4,\bf{R})$ superalgebra

The oscillator method was developed in order to construct unitary irreducible
representations of non-compact groups [11] in terms of its maximal compact
subgroup. Namely, the generators of the $gl(4|4,\bf{R})$ superalgebra can be
represented in terms of two sets of bosonic oscillators
$(a^{\alpha},\;a^{\dagger}_{\alpha})$,
$(b^{\dot{\alpha}},b^{\dagger}_{\dot{\alpha}})$ with
$\alpha,\;\dot{\alpha}=1,2$ and one set of fermionic oscillator
$(c_{A},\;c^{\dagger A})$ with $A=1,2,3,4$. The non-vanishing relations of
commutation are

$\displaystyle[a^{\alpha},a^{\dagger}_{\beta}]=\delta^{\alpha}_{\phantom{-}\beta},\phantom{--}[b^{\dot{\alpha}},b^{\dagger}_{\dot{\beta}}]$
$\displaystyle=$
$\displaystyle\delta^{\dot{\alpha}}_{\phantom{-}\dot{\beta}}\phantom{--}\\{c_{A},c^{\dagger
B}\\}=\delta_{A}^{\phantom{-}B}.$ (6)

The $gl(4|4,\bf{R})$ generators can be written as products of two oscillators,
the supercharges, translations and conformal transformations are

$\displaystyle q_{A\alpha}$ $\displaystyle\;=\;$ $\displaystyle
c_{A}a^{\dagger}_{\alpha},\;\;\phantom{--}s^{A\alpha}\;=\;c^{A\dagger}a^{\alpha},$
$\displaystyle\bar{q}^{A}_{\dot{\alpha}}$ $\displaystyle=$ $\displaystyle
c^{A\dagger}b^{\dagger}_{\dot{\alpha}},\;\;\phantom{--}\bar{s}^{\dot{\alpha}}_{A}\;=\;b^{\dot{\alpha}}c_{A},$
$\displaystyle p_{\alpha\dot{\alpha}}$ $\displaystyle=$ $\displaystyle
a^{\dagger}_{\alpha}b^{\dagger}_{\dot{\alpha}},\;\;\phantom{--}k^{\dot{\alpha}\alpha}\;=\;b^{\dot{\alpha}}a^{\alpha}.$
(7)

The dilatation generator reads as follows

$\displaystyle
D\;=\;1+\frac{1}{2}a^{\dagger}_{\gamma}a^{\gamma}+\frac{1}{2}b^{\dagger}_{\dot{\gamma}}b^{\dot{\gamma}},$
(8)

and the $sl(2)\times sl(2)$ and $sl(4)$ rotation generators are

$\displaystyle l_{\beta}^{\;\;\alpha}$ $\displaystyle=$ $\displaystyle
a^{\dagger}_{\beta}a^{\alpha}-\frac{1}{2}\delta_{\beta}^{\;\alpha}a^{\dagger}_{\gamma}a^{\gamma},$
$\displaystyle\bar{l}_{\dot{\beta}}^{\;\;\dot{\alpha}}$ $\displaystyle=$
$\displaystyle
b^{\dagger}_{\dot{\beta}}b^{\dot{\alpha}}-\frac{1}{2}\delta^{\;\dot{\alpha}}_{\dot{\beta}}b^{\dagger}_{\dot{\gamma}}b^{\dot{\gamma}},$
$\displaystyle r^{A}_{\;\;B}$ $\displaystyle=$ $\displaystyle c^{\dagger
A}c_{B}-\frac{1}{4}\delta^{A}_{\;B}c^{\dagger D}c_{D}.$ (9)

Also there are two $gl(1)$ generators

$\displaystyle C$ $\displaystyle=$
$\displaystyle\;1-\frac{1}{2}a^{\dagger}_{\gamma}a^{\gamma}+\frac{1}{2}b^{\dagger}_{\dot{\gamma}}b^{\dot{\gamma}}-\frac{1}{2}c^{\dagger
D}c_{D}$ $\displaystyle B$ $\displaystyle=$
$\displaystyle\;-1+\frac{1}{2}a^{\dagger}_{\gamma}a^{\gamma}-\frac{1}{2}b^{\dagger}_{\dot{\gamma}}b^{\dot{\gamma}},$
(10)

which are the central charge and outer derivation, respectively.

All fields in $\mathcal{N}=4$ SYM are uncharged with respect to the central
charge $C$, therefore can be dropped. This procedure leads to
$sl(4|4,\bf{R})$. The generator $B$ does not appear in commutators in
$sl(4|4,\bf{R})$ and can be projected out, giving the algebra
$psl(4|4,\bf{R})$.

###  2.2 $\mathcal{N}=4$ SYM fields in terms of oscillators and supertwistors

Here we give a brief description of how we can write down solutions of free
SYM theory in both oscillator representation and supertwistors. Also we
present an equivalence between oscillator and supertwistor pictures. We finish
this section writing composite operators in both pictures.

To write down the free field components of $\mathcal{N}=4$ SYM in the
oscillator picture. Let be $|0\rangle$ an invariant non-physical vacuum under
$psl(4|4,\bf{R})$, it is annhilated by $c_{A}$, $b^{\dot{\alpha}}$ and
$a^{\alpha}$ [4].

Let us define scalar fields as

$\displaystyle\phi^{AB}\;=\;c^{\dagger A}c^{\dagger B}|0\rangle,$ (11)

the another fields without derivatives can be obtained by applying the SUSY’s
generators $q_{A\alpha}$ and $\bar{q}^{A\dot{\alpha}}$ on (11). They read as
follows

$\displaystyle\psi^{D}_{\alpha}\sim a^{\dagger}_{\alpha}c^{\dagger
D}|0\rangle\phantom{--},\phantom{--}\bar{\psi}_{M\dot{\alpha}}\sim\epsilon_{MNBD}b^{\dagger}_{\dot{\alpha}}c^{\dagger
N}c^{\dagger B}c^{\dagger D}|0\rangle,$ (12)

and the self-dual and antiself-dual of the field strength are given by

$\displaystyle f_{\alpha\beta}$ $\displaystyle\sim$ $\displaystyle
a^{\dagger}_{\alpha}a^{\dagger}_{\beta}|0\rangle\phantom{--}\text{and}\phantom{--}\bar{f}_{\dot{\alpha}\dot{\beta}}\sim\epsilon_{ABDE}b^{\dagger}_{\dot{\alpha}}b^{\dagger}_{\dot{\beta}}c^{A\dagger}c^{B\dagger}c^{D\dagger}c^{E\dagger}|0\rangle,$
(13)

where the tildes means up to a proportionality constant.

####  2.2.1 From oscillators representation to supertwistors

The set of oscillators introduced above can be written in terms of
supertwistors variables. Supertwistors $\mathcal{Z}$ parametrize the space
$\bf{RP}^{3|4}$ [13]

$\displaystyle\mathcal{Z}\;=\;\left(\begin{array}[]{c}\lambda_{\alpha}\\\
\mu^{\dot{\alpha}}\\\ \psi^{A}\end{array}\right),$ (17)

where $\lambda_{\alpha}$ and $\mu^{\dot{\alpha}}$ are two-components bosons
and $\Psi^{A}$ is a four-components fermions. A twistor
$(\lambda_{\alpha},\mu^{\dot{\alpha}})$ defines a two-dimensional isotropic
subspaces of ${\bf{R}}^{2,2}$

$\displaystyle\mu^{\dot{\alpha}}+\lambda_{\alpha}x^{\dot{\alpha}\alpha}\;=\;0,$
(18)

which are called $\alpha-$planes. In the supersymmetric case, $\psi^{A}$
define also a plane in $\theta$ space

$\displaystyle\psi^{A}+\theta^{A\alpha}\lambda_{\alpha}\;=\;0,$ (19)

where $x^{\dot{\alpha}\alpha}$ and $\theta^{A\alpha}$ are coordinates in
superMinkowski spacetime $\mathbb{\bf M}^{4|8}$.

The oscillators are related to the the variables $\lambda,\,\mu$ and $\psi$,
say the twistor picture, by

$\displaystyle
a^{\dagger}_{\rho}\;\rightarrow\;\lambda_{\rho},\;\;b^{\dot{\alpha}}\rightarrow\mu^{\dot{\alpha}},\;\;\text{and}\;\;c^{\dagger
A}\;\rightarrow\;\psi^{A},$ (20)

and so on. The generators of $psl(4|4,\bf{R})$ can be written in terms of
supertwistor variables leading to first-order differential operators [13].

On-shell $\mathcal{N}=4$ SYM fields can be described by a scalar superfield
$\Phi(\lambda,\mu,\eta)$ [18]:

$\displaystyle\Phi(\mathcal{Z})\;=\;\tilde{f}+\eta_{A}\tilde{\psi}^{A}+\frac{1}{2!}\eta_{A}\eta_{B}\tilde{\phi}^{AB}+\frac{1}{3!}\eta_{A}\eta_{B}\eta_{C}\epsilon^{ABCD}\tilde{\bar{\psi}}_{D}+\eta_{1}\eta_{2}\eta_{3}\eta_{4}\tilde{\bar{f}}.$
(21)

Notice that $\tilde{f}$ and $\tilde{\bar{f}}$ are independent of $\eta$ and
that all the fields are in the twistor picture. The scalar $\Phi(\mathcal{Z})$
in twistor languague [19, 20] is written as

$\displaystyle\Phi(\mathcal{Z})\;=\;2\pi
i\int_{\mathbb{C}}\frac{ds}{s}e^{s(\mu^{\dot{\alpha}}\bar{p}_{\dot{\alpha}}+\eta_{A}\xi^{A})}\bar{\delta}^{2}(s\lambda_{\alpha}-p_{\alpha}),$
(22)

see Appendix C of [19] for a derivation of (22) In order to recover the
spacetime dependence, we need to Penrose transform
$\tilde{f},\;\tilde{\psi}^{A},\;\tilde{\phi}^{AB},\;\tilde{\bar{\psi}}$ and
$\tilde{\bar{f}}$. For a review see appendix A of [13].

The scalar field (11) is given by

$\displaystyle\phi^{AB}(x)\;=\;\int\frac{\lambda^{\alpha}d\lambda_{\alpha}}{2\pi
i}\left.\frac{\partial}{\partial\eta_{A}}\frac{\partial}{\partial\eta_{B}}\Phi(\mathcal{Z})\right|_{\eta=0},$
(23)

roughly speaking, we can state that the non-physical vacuum $|0\rangle$ can be
related to (22) up to Penrose transform. The another free fields (12) and (13)
are given as follows

Fields |  Oscillators |  Stwistors
---|---|---
$\Psi^{D}_{\alpha}(x)$ |  $\mathcal{R}a^{\dagger}_{\alpha}c^{\dagger D}|0\rangle$ |  $\int\frac{\lambda^{\alpha^{\prime}}d\lambda_{\alpha^{\prime}}}{2\pi i}\lambda_{\alpha}\left.\frac{\partial}{\partial\eta_{D}}\Phi(\mathcal{Z})\right|_{\eta=0}$
$\bar{\psi}_{M\dot{\alpha}}(x)$ |  $\epsilon_{MNBD}\mathcal{R}b^{\dagger}_{\dot{\alpha}}c^{\dagger N}c^{\dagger B}c^{\dagger D}|0\rangle$ |  $\epsilon_{MNBD}\int\frac{\lambda^{\alpha^{\prime}}d\lambda_{\alpha^{\prime}}}{2\pi i}\frac{\partial}{\partial\mu^{\dot{\alpha}}}\left.\frac{\partial}{\partial\eta_{N}}\frac{\partial}{\partial\eta_{B}}\frac{\partial}{\partial\eta_{D}}\Phi(\mathcal{Z})\right|_{\eta=0}$
$f_{\alpha\beta}(x)$ |  $\mathcal{R}a^{\dagger}_{\alpha}a^{\dagger}_{\beta}|0\rangle$ |  $\int\frac{\lambda^{\alpha^{\prime}}d\lambda_{\alpha^{\prime}}}{2\pi i}\lambda_{\alpha}\lambda_{\beta}\left.\Phi({\mathcal{Z}})\right|_{\eta=0}$
$\bar{f}_{\dot{\alpha}\dot{\beta}}(x)$ |  $\epsilon_{ABDE}\mathcal{R}b^{\dagger}_{\dot{\alpha}}b^{\dagger}_{\dot{\beta}}c^{A\dagger}c^{B\dagger}c^{D\dagger}c^{E\dagger}|0\rangle$ |  $\epsilon_{ABDE}\int\frac{\lambda^{\alpha^{\prime}}d\lambda_{\alpha^{\prime}}}{2\pi i}\frac{\partial}{\partial\mu^{\dot{\alpha}}}\frac{\partial}{\partial\mu^{\dot{\beta}}}\left.\frac{\partial}{\partial\eta_{A}}\frac{\partial}{\partial\eta_{B}}\frac{\partial}{\partial\eta_{D}}\frac{\partial}{\partial\eta_{E}}\Phi(\mathcal{Z})\right|_{\eta=0}$

where $\mathcal{R}=e^{ip\cdot x}$ is the translation operator.

####  2.2.2 Single trace operators

However in gauge theories with gauge group $SU(N)$, e.g. $\mathcal{N}=4$ SYM.
We need the product of these fields leading to composite operators. In this
work we are interested in single trace composite operators.

In this section, we explain how to write down single trace operators in terms
of oscillators. We set up the following notation.

  1. 1.

If $\psi_{1}$ and $\psi_{2}$ are fermions then
$\psi_{1}\bullet\psi_{2}={1\over
2}(\psi_{1}\otimes\psi_{2}-\psi_{2}\otimes\psi_{1})$, 111$\bullet$ means the
super-symmetrized tensor product.

  2. 2.

if $\psi_{1}$ and $\phi_{2}$ are bosons then $\phi_{1}\bullet\phi_{2}={1\over
2}(\phi_{1}\otimes\phi_{2}+\phi_{2}\otimes\phi_{2})$, and

  3. 3.

if $\phi$ is boson and $\psi$ is fermion then $\phi\bullet\psi={1\over
2}(\phi\otimes\psi+\psi\otimes\phi)$.

At linear order in $h$, say $\mathcal{O}[B]$. The multiplet in the oscillator
picture is

$\displaystyle\mathcal{O}[B]$ $\displaystyle=$ $\displaystyle\int
d^{4}x\prod_{j=1}^{3}\mathcal{R}^{(j)}B^{[MN]}_{(IJ)}\Psi^{(IJ)}_{[MN]},$ (24)

where $\Psi^{(IJ)}_{[MN]}$ is given by

$\Psi^{(IJ)}_{[MN]}=\;a^{\dagger}_{[1}c^{\dagger(I}|0\rangle\bullet
a^{\dagger}_{2]}c^{\dagger J)}|0\rangle\bullet\epsilon_{MNKL}c^{\dagger
K}c^{\dagger L}|0\rangle.$ (25)

The index $j$ denotes in which singleton representation acts the
$\mathcal{R}^{(j)}$ operator. (24) is invariant under conformal
transformation. In Appendix A we show this fact using the oscillator
representation.

In the twistor picture (24) is given by

$\displaystyle\mathcal{O}[B]\;=\;\int
d^{4}x\;\prod^{3}_{i=1}\int\frac{\lambda^{\alpha^{\prime}}_{(i)}d\lambda_{\alpha^{\prime}(i)}}{2\pi
i}B^{[MN]}_{(IJ)}\Psi^{(IJ)}_{[MN]},$ (26)

where $\Psi^{(IJ)}_{[MN]}$ reads as

$\displaystyle\Psi^{(IJ)}_{[MN]}\;=\;\epsilon_{MNAB}\epsilon^{\alpha\beta}\lambda_{\beta(1)}\left.\frac{\partial}{\partial\eta_{I(1)}}\Phi^{(1)}(\mathcal{Z})\right|_{\eta=0}\;\lambda_{\alpha(2)}\left.\frac{\partial}{\partial\eta_{J(2)}}\Phi^{(2)}(\mathcal{Z})\right|_{\eta=0}\;\left.\frac{\partial}{\partial\eta_{A(3)}}\frac{\partial}{\partial\eta_{B(3)}}\Phi^{(3)}(\mathcal{Z})\right|_{\eta=0}.$
(27)

We see that (26) is finite on-shell. We can evaluate it on the product of
three off-shell fields and the integral is convergent 222We thank Prof. Andrei
Mikhailov for making clear this point.. This means that (26) defines an
element of the dual space to the tensor product of three singleton
representations.

##  3 The structure of the supermultiplet

In this section we describe the full supermultiplet of $\beta-$deformation in
the field theory side. Instead of working in both oscillator picture and/or
twistor picture, we just work out in the oscillator picture.

###  3.1 Descendants multiplets from $\mathcal{O}[B]$

It is known that $(c^{\dagger I}c_{M}\wedge c^{\dagger J}c_{N})_{0}$, which is
$(g\wedge g)_{0}$ corresponds to $\Psi_{[MN]}^{(IJ)}$ [7].

Below we list in a table the set of deformations arising from the multiplet
$\mathcal{O}[B]$.

Table 1: Descendants from $\mathcal{O}[B]$ $(g\wedge g)_{0}$ |  Field theory side
---|---
$c^{\dagger I}c_{M}\wedge c^{\dagger J}c_{N}$ |  $\Psi_{[MN]}^{(IJ)}$
$a^{\dagger}_{\rho}c_{M}\wedge c^{\dagger J}c_{N}$ |  $(q_{A\rho})\Psi_{[MN]}^{(IJ)}$
$a^{\dagger}_{\rho}c_{M}\wedge a^{\dagger}_{\varrho}c_{N}$ |  $(q_{B\varrho})(q_{A\rho})\Psi_{[MN]}^{(IJ)}$
$0$ |  $(q_{[D|\varpi|})(q_{B|\varrho|})(q_{A]\rho})\Psi_{[MN]}^{(IJ)}$
$b^{\dagger}_{\dot{\alpha}}c^{\dagger I}\wedge\;c^{\dagger J}c_{N}$ |  $(\bar{q}^{A}_{\dot{\alpha}})\Psi_{[MN]}^{(IJ)}$
$b^{\dagger}_{\dot{\alpha}}c^{\dagger I}\wedge\;b^{\dagger}_{\dot{\beta}}c^{\dagger J}$ |  $(\bar{q}^{B}_{\dot{\beta}})(\bar{q}^{A}_{\dot{\alpha}})\Psi_{[MN]}^{(IJ)}$
$0$ |  $(\bar{q}^{[D}_{\dot{\rho}})(\bar{q}^{B}_{\dot{\beta}})(\bar{q}^{A]}_{\dot{\alpha}})\Psi_{[MN]}^{(IJ)}$
$b^{\dot{\alpha}}c_{M}\wedge c^{\dagger J}c_{N}$ |  $(\bar{s}^{\dot{\alpha}}_{A})\Psi_{[MN]}^{(IJ)}$
$b^{\dot{\alpha}}c_{M}\wedge b^{\dot{\beta}}c_{N}$ |  $(\bar{s}^{\dot{\beta}}_{B})(\bar{s}^{\dot{\alpha}}_{A})\Psi_{[MN]}^{(IJ)}$
$0$ |  $(\bar{s}^{\dot{\rho}}_{D})(\bar{s}^{\dot{\beta}}_{B})(s^{\dot{\alpha}}_{A})\Psi_{[MN]}^{(IJ)}$
$a^{\alpha}c^{\dagger I}\wedge c^{\dagger J}c_{N}$ |  $(s^{\alpha A})\Psi_{[MN]}^{(IJ)}$
$a^{\alpha}c^{\dagger I}\wedge a^{\beta}c^{\dagger J}$ |  $(s^{\beta B})(s^{\alpha A})\Psi_{[MN]}^{(IJ)}$
$0$ |  $(s^{\rho D})(s^{\beta B})(s^{\alpha A})\Psi_{[MN]}^{(IJ)}$

On the left hand side we listed the representations of $psl(4,4|\bf{R})$
namely $(g\wedge g)_{0}$. The AdS/CFT correspondence implies that those
representations are related to the field theory by a certain intertwining
operator. In the field theory side representations of the algebra comes from
the subspace in the tensor product of three singletons.

####  3.1.1 Action of $q$ and $\bar{q}$ on $\mathcal{O}[B]$

Here and the following subsection we describe the supermultiplet of
$\beta-$deformation in the field theory side.

From the supersymmetry algebra, we know that $q_{A\rho}$ and
$\bar{q}^{A}_{\dot{\rho}}$ commute with $P_{\mu}$. They act just on
$\Psi^{(IJ)}_{[MN]}$.

To begin with, $q_{A\rho}\Psi^{(IJ)}_{[MN]}$ is

$\displaystyle
q_{A\rho}\Psi^{(IJ)}_{[MN]}\;=\;(c_{A}a^{\dagger}_{\rho})\Psi^{(IJ)}_{[MN]}\;\equiv\;$
$\displaystyle
2\delta_{A}^{(I}\;a^{\dagger}_{\rho}a^{\dagger}_{[1}|0\rangle\bullet
a^{\dagger}_{2]}c^{\dagger J)}|0\rangle\bullet\epsilon_{MNKL}c^{\dagger
K}c^{\dagger L}|0\rangle$
$\displaystyle+2\;a^{\dagger}_{[1}c^{\dagger(I}|0\rangle\bullet
a^{\dagger}_{2]}c^{\dagger
J)}|0\rangle\bullet\epsilon_{AMNL}a^{\dagger}_{\rho}c^{\dagger L}|0\rangle,$
(28)

where the right hand side (rhs) is

$\displaystyle
2\delta_{A}^{(I}\;f_{\rho[1}\;\psi_{2]}^{J)}\phi_{MN}+2\epsilon_{AMNL}\psi^{(I}_{[1}\psi^{J)}_{2]}\psi^{L}_{\rho},$
(29)

in terms of oscillators. To be more clear, what we get is

$\displaystyle\mathcal{O}_{A\rho}$ $\displaystyle\equiv$ $\displaystyle\;\int
d^{4}xB^{[MN]}_{(IJ)}\prod_{j=1}^{3}\mathcal{R}^{(j)}q_{A\rho}\Psi^{(IJ)}_{[MN]}$
(30) $\displaystyle=$ $\displaystyle\int
d^{4}xB^{[MN]}_{(IJ)}(\delta_{A}^{I}\;\epsilon^{\alpha\beta}f_{\rho\beta}\;\psi_{\alpha}^{J}\phi_{MN}+\epsilon^{\alpha\beta}\epsilon_{AMNL}\psi^{I}_{\beta}\psi^{J}_{\alpha}\psi^{L}_{\rho}).$

The result of acting with a second SUSY generator $q_{B\varrho}$ on (3.1.1) is

$\displaystyle(c_{B}a^{\dagger}_{\varrho})(c_{A}a^{\dagger}_{\rho})\Psi^{(IJ)}_{[MN]}\;=\;$
$\displaystyle
2\delta_{A}^{(I}\;a^{\dagger}_{\rho}a^{\dagger}_{[1}|0\rangle\bullet
a^{\dagger}_{2]}a^{\dagger}_{\varrho}\;\delta^{J)}_{B}|0\rangle\bullet\epsilon_{MNKL}c^{\dagger
K}c^{\dagger L}|0\rangle$
$\displaystyle-4\delta_{A}^{(I}\;a^{\dagger}_{\rho}a^{\dagger}_{[1}|0\rangle\bullet
a^{\dagger}_{2]}c^{\dagger
J)}|0\rangle\bullet\epsilon_{BMNL}a^{\dagger}_{\varrho}c^{\dagger L}|0\rangle$
$\displaystyle+4\delta^{(I}_{B}\;a^{\dagger}_{[1}a^{\dagger}_{\varrho}|0\rangle\bullet
a^{\dagger}_{2]}c^{\dagger
J)}|0\rangle\bullet\epsilon_{AMNL}a^{\dagger}_{\rho}c^{\dagger L}|0\rangle$
$\displaystyle+2\;a^{\dagger}_{[1}c^{\dagger(I}|0\rangle\bullet
a^{\dagger}_{2]}c^{\dagger
J)}|0\rangle\bullet\epsilon_{ABMN}a^{\dagger}_{\rho}a^{\dagger}_{\varrho}|0\rangle,$
(31)

it leads us to the following

$\displaystyle\mathcal{O}_{(\varrho B),(\rho A)}$ $\displaystyle\equiv$
$\displaystyle\int
d^{4}xB^{[MN]}_{(IJ)}\prod_{j=1}^{3}\mathcal{R}^{(j)}(c_{B}a^{\dagger}_{\varrho})(c_{A}a^{\dagger}_{\rho})\Psi^{(IJ)}_{[MN]}$
(32) $\displaystyle=$ $\displaystyle\int
d^{4}xB^{[MN]}_{(IJ)}\epsilon^{\alpha\beta}\big{[}\delta^{(I}_{A}\delta^{J)}_{B}f_{\rho\beta}f_{\alpha\varrho}\phi_{MN}-2\epsilon_{BMNL}\delta^{(I}_{A}f_{\rho\beta}\psi_{\alpha}^{J)}\psi_{\varrho}^{L}$
$\displaystyle+$ $\displaystyle
2\epsilon_{AMNL}\delta^{(I}_{B}f_{\varrho\beta}\psi^{J)}_{\alpha}\psi^{L}_{\rho}+\epsilon_{ABMN}\psi_{\beta}^{(I}\psi_{\alpha}^{J)}f_{\rho\varrho}\big{]},$

where the terms in brackets come from (3.1.1).

Symmetry of $qq\Psi$

Antisymmetrization in $A$ and $B$ on (3.1.1) gives us

$\displaystyle\epsilon^{PQAB}(c_{B}a^{\dagger}_{\varrho})(c_{A}a^{\dagger}_{\rho})\Psi^{(IJ)}_{[MN]}\;=\;0+\text{mod}(\Psi^{(IJ)}_{[MJ]}),$

this means that

$\displaystyle
q_{[A|(\rho|}q_{|B]|\varrho)}\Psi^{(IJ)}_{[MN]}\;=\;0+\text{traces}.$ (33)

The above result implies that

$\displaystyle q_{[D|\varpi|}q_{B|\varrho|}q_{A]\rho}\Psi^{(IJ)}_{[MN]}$
$\displaystyle=$ $\displaystyle 0.$ (34)

Therefore we conclude that there are just two descendants (30) and (32)
obtained from (24) by applying once and twice $q$, respectively.

Acting with $\bar{q}$ on $\Psi^{(IJ)}_{[MN]}$

The action of $\bar{q}^{A}_{\dot{\alpha}}$ and
$\bar{q}^{B}_{\dot{\beta}}\bar{q}^{A}_{\dot{\alpha}}$ on $\Psi^{(IJ)}_{[MN]}$
reads as follow

$\displaystyle(c^{\dagger
A}b^{\dagger}_{\dot{\alpha}})\Psi^{(IJ)}_{[MN]}\;=\;$ $\displaystyle
2\;b^{\dagger}_{\dot{\alpha}}a^{\dagger}_{[1}c^{\dagger A}c^{\dagger
I}|0\rangle\bullet a^{\dagger}_{2]}c^{\dagger
J)}|0\rangle\bullet\epsilon_{MNKL}c^{\dagger K}c^{\dagger L}|0\rangle$
$\displaystyle+\;a^{\dagger}_{[1}c^{\dagger(I}|0\rangle\bullet
a^{\dagger}_{2]}c^{\dagger
J)}|0\rangle\bullet\epsilon_{MNKL}b^{\dagger}_{\dot{\alpha}}c^{\dagger
A}c^{\dagger K}c^{\dagger L}|0\rangle,$ (35)

and

$\displaystyle(c^{\dagger B}b^{\dagger}_{\dot{\beta}})(c^{\dagger
A}b^{\dagger}_{\dot{\alpha}})\Psi^{(IJ)}_{[MN]}\;=\;$ $\displaystyle
2\;b^{\dagger}_{\dot{\beta}}b^{\dagger}_{\dot{\alpha}}a^{\dagger}_{[1}c^{\dagger
B}c^{\dagger A}c^{\dagger(I}|0\rangle\bullet a^{\dagger}_{2]}c^{\dagger
J)}|0\rangle\bullet\epsilon_{MNKL}c^{\dagger K}c^{\dagger L}|0\rangle$
$\displaystyle-2\;b^{\dagger}_{\dot{\alpha}}a^{\dagger}_{[1}c^{\dagger
A}c^{\dagger(I}|0\rangle\bullet a^{\dagger}_{2]}c^{\dagger
J)}|0\rangle\bullet\epsilon_{MNKL}b^{\dagger}_{\dot{\beta}}c^{\dagger
B}c^{\dagger K}c^{\dagger L}|0\rangle$
$\displaystyle+\;2a^{\dagger}_{[1}b^{\dagger}_{\dot{\beta}}c^{\dagger
B}c^{\dagger(I}|0\rangle\bullet a^{\dagger}_{2]}c^{\dagger
J)}|0\rangle\bullet\epsilon_{MNKL}b^{\dagger}_{\dot{\alpha}}c^{\dagger
A}c^{\dagger K}c^{\dagger L}|0\rangle$
$\displaystyle+a^{\dagger}_{[1}c^{\dagger(I}|0\rangle\bullet
a^{\dagger}_{2]}c^{\dagger
J)}|0\rangle\bullet\epsilon_{MNKL}b^{\dagger}_{\dot{\beta}}b^{\dagger}_{\dot{\alpha}}c^{\dagger
B}c^{\dagger A}c^{\dagger K}c^{\dagger L}|0\rangle,$ (36)

where we have used the momentum conservation and on-shell condition to get
(3.1.1). Those results lead us to

$\displaystyle\mathcal{O}^{A}_{\dot{\alpha}}$ $\displaystyle\equiv$
$\displaystyle\int
d^{4}x\prod^{3}_{i=1}\mathcal{R}^{(i)}B^{[MN]}_{(IJ)}(c^{\dagger
A}b^{\dagger}_{\dot{\alpha}})\Psi^{(IJ)}_{[MN]}$ (37) $\displaystyle=$
$\displaystyle\int
d^{4}xB^{[MN]}_{(IJ)}\epsilon^{\alpha\beta}[\partial_{\beta\dot{\alpha}}\phi^{AI}\psi_{\alpha}^{J}\phi_{MN}+\frac{1}{2}\psi^{I}_{\beta}\psi^{J}_{\alpha}\delta^{[PA]}_{MN}\bar{\psi}_{P\dot{\alpha}}],$

and

$\displaystyle\mathcal{O}^{BA}_{\dot{\beta}\dot{\alpha}}$
$\displaystyle\equiv$ $\displaystyle\int
d^{4}x\prod^{3}_{i=1}\mathcal{R}^{(i)}B^{[MN]}_{(IJ)}(c^{\dagger
B}b^{\dagger}_{\dot{\beta}})(c^{\dagger
A}b^{\dagger}_{\dot{\alpha}})\Psi^{(IJ)}_{[MN]}$ (38) $\displaystyle=$
$\displaystyle\int
d^{4}xB^{[MN]}_{(IJ)}\epsilon^{\alpha\beta}\big{[}\epsilon^{ABPI}\partial_{\beta(\dot{\alpha}}\bar{\psi}_{\dot{\beta})P}\psi^{J}_{\alpha}\phi_{MN}-\partial_{\beta\dot{\alpha}|}\phi^{AI}\psi^{J}_{\alpha}\delta^{PB}_{MN}\bar{\psi}_{\dot{\beta}P}$
$\displaystyle+$
$\displaystyle\partial_{\beta\dot{\beta}|}\phi^{BI}\psi^{J}_{\alpha}\delta^{PA}_{MN}\bar{\psi}_{\dot{\alpha}P}+\frac{1}{2}\psi^{I}_{\beta}\psi^{J}_{\alpha}\delta^{AB}_{MN}\bar{f}_{(\dot{\beta}\dot{\alpha})}\big{]},$

where the terms in brackets of (37) and (38) come from (3.1.1) and (3.1.1),
respectively.

Acting with three times $\bar{q}$ on $\Psi$, i.e. $(c^{\dagger
D}b^{\dagger}_{\dot{\rho}})(c^{\dagger B}b^{\dagger}_{\dot{\beta}})(c^{\dagger
A}b^{\dagger}_{\dot{\alpha}})\Psi^{(IJ)}_{[MN]}$, following an
antisymmetrization in $A$, $B$ and $D$, we got

$\displaystyle(c^{\dagger[D}b^{\dagger}_{(\dot{\rho}})(c^{\dagger
B}b^{\dagger}_{\dot{\beta}})(c^{\dagger
A]}b^{\dagger}_{\dot{\alpha})})\Psi^{(IJ)}_{[MN]}\;=\;0,$ (39)

this implies that the only descendants obtained from $\mathcal{O}[B]$ after
applying once and twice $\bar{q}$ are (37) and(38).

These computations were almost straightforward since $q$ and $\bar{q}$ commute
with $\mathcal{R}^{(j)}$. This will not happen with $s^{A\alpha}$ and
$\bar{s}^{\dot{\alpha}}_{A}$, since the commutator of them with
$\mathcal{R}^{(j)}$ is non-zero.

####  3.1.2 Action of $s$ and $\bar{s}$ onto $\mathcal{O}[B]$

The commutator of $\bar{s}$ with $p_{\mu}$ is proportional to $q$, it implies
the following result

$\displaystyle[\bar{s}^{\dot{\alpha}}_{A},\mathcal{R}^{(j)}]=-\frac{i}{2}\mathcal{R}^{(j)}x^{\dot{\alpha}\omega}a^{\dagger}_{\omega}c_{A},$
(40)

where $x^{\dot{\alpha}\omega}$ is the contraction
$\bar{\sigma}_{\mu}^{\dot{\alpha}\omega}x^{\mu}$. Therefore the action of
$\bar{s}^{\dot{\alpha}}_{A}$ on $\mathcal{O}[B]$ is

$\displaystyle\tilde{\mathcal{O}}^{\dot{\alpha}}_{A}\;\equiv\;(b^{\dot{\alpha}}c_{A})\mathcal{O}[B]\;=\;$
$\displaystyle-\frac{i}{2}\int
d^{4}x\prod_{j=1}^{3}\mathcal{R}^{(j)}B^{[MN]}_{(IJ)}x^{\dot{\alpha}\omega}(a^{\dagger}_{\omega}c_{A})\Psi^{(IJ)}_{[MN]},$
(41)

where the rhs is

$\displaystyle-i\int
d^{4}xB^{[MN]}_{(IJ)}x^{\dot{\alpha}\omega}\epsilon^{\alpha\beta}\\{\delta_{A}^{I}\;f_{\omega\beta}\;\psi_{\alpha}^{J}\phi_{MN}+\epsilon_{AMNL}\psi^{I}_{\beta}\psi^{J}_{\alpha}\psi^{L}_{\omega}\\},$
(42)

in terms of oscillators.

The result of acting $\bar{s}$ twice on $\mathcal{O}[B]$ is

$\displaystyle\tilde{\mathcal{O}}^{\dot{\beta}\dot{\alpha}}_{BA}\;\equiv\;(b^{\dot{\beta}}c_{B})(b^{\dot{\alpha}}c_{A})\mathcal{O}[B]\;=\;$
$\displaystyle-\frac{1}{4}\int
d^{4}x\prod_{j=1}^{3}\mathcal{R}^{(j)}B^{[MN]}_{(IJ)}x^{\dot{\alpha}\omega}x^{\dot{\beta}\rho}(a^{\dagger}_{\rho}c_{B})(a^{\dagger}_{\omega}c_{A})\Psi^{(IJ)}_{[MN]},$
(43)

where the rhs is

$\displaystyle-\int
d^{4}xB^{[MN]}_{(IJ)}x^{\dot{\alpha}\omega}x^{\dot{\beta}\rho}\epsilon^{\alpha\beta}\\{\delta^{I}_{A}\delta^{J}_{B}f_{\omega\beta}f_{\alpha\rho}\phi_{MN}-2\epsilon_{BMNL}\delta^{I}_{A}f_{\omega\beta}\psi_{\alpha}^{J}\psi_{\rho}^{L}$
$\displaystyle\;+\;2\epsilon_{AMNL}\delta^{I}_{B}f_{\rho\beta}\psi^{J}_{\alpha}\psi^{L}_{\omega}+\epsilon_{ABMN}\psi_{\beta}^{I}\psi_{\alpha}^{J}f_{\omega\rho}\\}.$
(44)

Symmetry of $\bar{s}\bar{s}\mathcal{O}[B]$

Antisymmetrization in $A$ and $B$ on (43) lead us to

$\displaystyle\bar{s}^{(\dot{\beta}}_{[B}\bar{s}^{\dot{\alpha})}_{A]}\mathcal{O}[B]=0,$
(45)

it implies that

$\displaystyle\bar{s}^{(\dot{\varrho}}_{[D}\bar{s}^{\dot{\beta}}_{B}\bar{s}^{\dot{\alpha})}_{A]}\mathcal{O}[B]=0.$
(46)

Therefore, we conclude that the only descendants obtained from (24) after
applying once a twice the generator $\bar{s}$ are (41) and (43), respectively.

Acting $s$ on $\mathcal{O}[B]$

The action of the superconformal generator $s^{A\alpha}$ on $\mathcal{O}[B]$
reads as follows

$\displaystyle\tilde{\mathcal{O}}^{\alpha A}$ $\displaystyle\equiv$
$\displaystyle(a^{\alpha}c^{\dagger A})\mathcal{O}[B]$ (47) $\displaystyle=$
$\displaystyle\int
d^{4}x\prod^{3}_{j=1}\mathcal{R}^{(j)}B^{[MN]}_{(IJ)}\Big{(}-ix^{\dot{\omega}\alpha}a^{\dagger}_{[1}b^{\dagger}_{\dot{\omega}}c^{\dagger
A}c^{\dagger(I}|0\rangle\bullet a^{\dagger}_{2]}c^{\dagger
J)}|0\rangle\bullet\epsilon_{MNKL}c^{\dagger K}c^{\dagger L}|0\rangle$
$\displaystyle-$
$\displaystyle\frac{i}{2}a^{\dagger}_{[1}c^{\dagger(I}|0\rangle\bullet
a^{\dagger}_{2]}c^{\dagger
J)}|0\rangle\bullet\epsilon_{MNKL}x^{\dot{\omega}\alpha}b^{\dagger}_{\dot{\omega}}c^{\dagger
A}c^{\dagger K}c^{\dagger L}|0\rangle$ $\displaystyle+$ $\displaystyle
2\delta^{\alpha}_{\;[1}\;c^{\dagger A}c^{\dagger(I}|0\rangle\bullet
a^{\dagger}_{2]}c^{\dagger J)}|0\rangle\bullet\epsilon_{MNKL}c^{\dagger
K}c^{\dagger L}|0\rangle\Big{)},$

where $x^{\dot{\omega}\alpha}$ comes from the commutator of $s^{A\alpha}$ with
$\mathcal{R}^{(j)}$. The rhs of Eqn. (47) is just

$\displaystyle\int
d^{4}xB^{[MN]}_{(IJ)}\epsilon^{\beta\rho}\\{\delta^{\alpha}_{\;\rho}\phi^{AI}\psi^{J}_{\beta}\phi_{MN}-\frac{i}{2}x^{\dot{\omega}\alpha}(\partial_{\rho\dot{\omega}}\phi^{AI}\psi_{\beta}^{J}\phi_{MN}+\frac{1}{2}\psi^{I}_{\beta}\psi^{J}_{\rho}\delta_{MN}^{PA}\bar{\psi}_{P\dot{\omega}})\\},$
(48)

in the oscillator picture.

$\tilde{\mathcal{O}}^{\alpha A}$ (47) is a descendant obtained from (24) and
also $s^{B\varrho}s^{A\alpha}\mathcal{O}[B]$, say
$\tilde{\mathcal{O}}^{(B\varrho),(A\alpha)}$, is. However, the structure of
$\tilde{\mathcal{O}}^{(B\varrho),(A\alpha)}$ is not showed explicitly
here.They are just the two descendants from (24), since it can be proved that

$\displaystyle s^{[D|\varpi|}s^{B|\varrho|}s^{A]\alpha}\mathcal{O}[B]\;=\;0.$
(49)

Those results described the structure of the supermultiplet of deformation on
the field theory side and they were listed in the Table 1.

However we state that still there are another descendant multiplets which can
be obtained from (24). Bellow, we list more results .

$(g\wedge g)_{0}$ |  Field theory side
---|---
$a^{\dagger}_{\varrho}b^{\dagger}_{\dot{\alpha}}\wedge c^{\dagger J}c_{N}-b^{\dagger}_{\dot{\alpha}}c^{\dagger I}\wedge a^{\dagger}_{\varrho}c_{N}$ |  $(q_{B\varrho})(\bar{q}^{A}_{\dot{\alpha}})\Psi_{[MN]}^{(IJ)}$
$\;\;\;\;\;\;b^{\dagger}_{\dot{\alpha}}a^{\dagger}_{[\varrho}\wedge a^{\dagger}_{\varpi]}c_{N}$ |  $(q_{C\varpi})(q_{B\varrho})(\bar{q}^{A}_{\dot{\alpha}})\Psi_{[MN]}^{(IJ)}$
$\;\;\;\;\;\;\;0$ |  $(q_{D\vartheta})(q_{C\varpi})(q_{B\varrho})(\bar{q}^{A}_{\dot{\alpha}})\Psi_{[MN]}^{(IJ)}$
$a^{\dagger}_{\varrho}b^{\dagger}_{\dot{\alpha}}\wedge b^{\dagger}_{\dot{\beta}}c^{\dagger J}-b^{\dagger}_{\dot{\alpha}}c^{\dagger I}\wedge a^{\dagger}_{\varrho}b^{\dagger}_{\dot{\beta}}$ |  $(q_{D\varrho})(\bar{q}^{B}_{\dot{\beta}})(\bar{q}^{A}_{\dot{\alpha}})\Psi_{[MN]}^{(IJ)}$
$\;\;\;\;\;\;b^{\dagger}_{\dot{\alpha}}a^{\dagger}_{[\varrho}\wedge a^{\dagger}_{\varpi]}b^{\dagger}_{\dot{\beta}}$ |  $(q_{E\varpi})(q_{D\varrho})(\bar{q}^{B}_{\dot{\beta}})(\bar{q}^{A}_{\dot{\alpha}})\Psi_{[MN]}^{(IJ)}$
$\;\;\;\;\;\;0$ |  $q\;q\;q(\bar{q}^{B}_{\dot{\beta}})(\bar{q}^{A}_{\dot{\alpha}})\Psi_{[MN]}^{(IJ)}$
$a^{\alpha}b^{\dot{\alpha}}\wedge c_{N}c^{\dagger J}-b^{\dot{\alpha}}c_{M}\wedge a^{\alpha}c^{\dagger J}$ |  $(s^{\alpha B})(\bar{s}^{\dot{\alpha}}_{A})\Psi^{(IJ)}_{[MN]}$
$\;\;\;\;\;\;b^{\dot{\alpha}}a^{[\alpha}\wedge a^{\beta]}c^{\dagger J}$ |  $(s^{\beta D})(s^{\alpha B})(\bar{s}^{\dot{\alpha}}_{A})\Psi^{(IJ)}_{[MN]}$
$\;\;\;\;\;\;\;0$ |  $(s^{\rho E})(s^{\beta D})(s^{\alpha B})(\bar{s}^{\dot{\alpha}}_{A})\Psi^{(IJ)}_{[MN]}$
$a^{\alpha}b^{\dot{\alpha}}\wedge b^{\dot{\beta}}c_{N}-b^{\dot{\alpha}}c_{M}\wedge a^{\alpha}b^{\dot{\beta}}$ |  $(s^{\alpha D})(\bar{s}^{\dot{\beta}}_{B})(\bar{s}^{\dot{\alpha}}_{A})\Psi^{(IJ)}_{[MN]}$
$\;\;\;\;\;\;b^{\dot{\alpha}}a^{[\alpha}\wedge a^{\beta]}b^{\dot{\beta}}$ |  $(s^{\beta E})(s^{\alpha D})(\bar{s}^{\dot{\beta}}_{B})(\bar{s}^{\dot{\alpha}}_{A})\Psi^{(IJ)}_{[MN]}$
$\;\;\;\;\;\;0$ |  $sss(\bar{s}^{\dot{\beta}}_{B})(\bar{s}^{\dot{\alpha}}_{A})\Psi^{(IJ)}_{[MN]}$
$b_{\dot{\beta}}^{\dagger}b^{\dot{\alpha}}\wedge c^{\dagger J}c_{N}-b^{\dot{\alpha}}c_{M}\wedge b^{\dagger}_{\dot{\beta}}c^{\dagger J}$ |  $(\bar{q}^{B}_{\dot{\beta}})(\bar{s}^{\dot{\alpha}}_{A})\Psi^{(IJ)}_{[MN]}$
$\;\;\;\;\;\;b_{[\dot{\beta}}^{\dagger}b^{\dot{\alpha}}\wedge b^{\dagger}_{\dot{\rho}]}c^{\dagger J}$ |  $(\bar{q}^{D}_{\dot{\rho}})(\bar{q}^{B}_{\dot{\beta}})(\bar{s}^{\dot{\alpha}}_{A})\Psi^{(IJ)}_{[MN]}$
$\;\;\;\;\;\;0$ |  $(\bar{q}^{E}_{\dot{\varrho}})(\bar{q}^{D}_{\dot{\rho}})(\bar{q}^{B}_{\dot{\beta}})(\bar{s}^{\dot{\alpha}}_{A})\Psi^{(IJ)}_{[MN]}$
$b^{\dagger}_{\dot{\rho}}b^{\dot{\alpha}}\wedge b^{\dot{\beta}}c_{N}-b^{\dot{\alpha}}c_{M}\wedge b^{\dagger}_{\dot{\rho}}b^{\dot{\beta}}$ |  $(\bar{q}^{D}_{\dot{\rho}})(\bar{s}^{\dot{\beta}}_{B})(\bar{s}^{\dot{\alpha}}_{A})\Psi_{[MN]}^{(IJ)}$
$\;\;\;\;\;\;b^{\dagger}_{[\dot{\rho}}b^{\dot{\alpha}}\wedge b^{\dagger}_{\dot{\varrho}]}b^{\dot{\beta}}$ |  $(\bar{q}^{E}_{\dot{\varrho}})(\bar{q}^{D}_{\dot{\rho}})(\bar{s}^{\dot{\beta}}_{B})(\bar{s}^{\dot{\alpha}}_{A})\Psi_{[MN]}^{(IJ)}$
$\;\;\;\;\;\;0$ |  $\bar{q}\bar{q}\bar{q}(\bar{s}^{\dot{\beta}}_{B})(\bar{s}^{\dot{\alpha}}_{A})\Psi_{[MN]}^{(IJ)}$
$a^{\dagger}_{\beta}a^{\alpha}\wedge c^{\dagger J}c_{N}-a^{\alpha}c^{\dagger I}\wedge a^{\dagger}_{\beta}c_{N}$ |  $(q_{B\beta})(s^{\alpha A})\Psi^{(IJ)}_{[MN]}$
$\;\;\;\;\;\;a^{\dagger}_{[\beta}a^{\alpha}\wedge a^{\dagger}_{\rho]}c_{N}$ |  $(q_{D\rho})(q_{B\beta})(s^{\alpha A})\Psi^{(IJ)}_{[MN]}$
$\;\;\;\;\;\;0$ |  $(q_{E\varrho})(q_{D\rho})(q_{B\beta})(s^{\alpha A})\Psi^{(IJ)}_{[MN]}$
$a^{\dagger}_{\rho}a^{\alpha}\wedge a^{\beta}c^{\dagger J}-a^{\alpha}c^{\dagger I}\wedge a^{\dagger}_{\rho}a^{\beta}$ |  $(q_{D\rho})(s^{\beta B})(s^{\alpha A})\Psi^{(IJ)}_{[MN]}$
$\;\;\;\;\;\;a^{\dagger}_{[\rho}a^{\alpha}\wedge a^{\dagger}_{\varrho]}a^{\beta}$ |  $(q_{E\varrho})(q_{D\rho})(s^{\beta B})(s^{\alpha A})\Psi^{(IJ)}_{[MN]}$
$\;\;\;\;\;\;0$ |  $qqq(s^{\beta B})(s^{\alpha A})\Psi^{(IJ)}_{[MN]}$

The results listed in this table are possible, too. For instance, let us
choose

$\displaystyle
sss(\bar{s}^{\dot{\beta}}_{B})(\bar{s}^{\dot{\alpha}}_{A})\Psi^{(IJ)}_{[MN]},$
(50)

using relation of commutations. It can be written as

$\displaystyle
ss(\bar{s}^{\dot{\beta}}_{B})(\bar{s}^{\dot{\alpha}}_{A})s^{\star\circ}\Psi^{(IJ)}_{[MN]}-\delta^{\star}_{A}ss(\bar{s}^{\dot{\beta}}_{B})k^{\dot{\alpha}\circ}\Psi^{(IJ)}_{[MN]}+\delta^{\star}_{B}ssk^{\dot{\beta}\circ}(\bar{s}^{\dot{\alpha}}_{A})\Psi^{(IJ)}_{[MN]},$
(51)

let us recall that the complete expresion in the field theory side is (24),
for a brievity let us say $\Psi^{(IJ)}_{[MN]}$. In Appendix A we check that
(24) is invariant under the generator of conformal transformations
$k^{\dot{\alpha}\alpha}$. Moreover the generators $k$ and $\bar{s}$ commutes.
Therefore the last two terms in (51) vanish. With this procedure we have the
following equality

$\displaystyle
sss(\bar{s}^{\dot{\beta}}_{B})(\bar{s}^{\dot{\alpha}}_{A})\Psi^{(IJ)}_{[MN]}\;=\;ss(\bar{s}^{\dot{\beta}}_{B})(\bar{s}^{\dot{\alpha}}_{A})s^{\star\circ}\Psi^{(IJ)}_{[MN]}\;=\;(\bar{s}^{\dot{\beta}}_{B})(\bar{s}^{\dot{\alpha}}_{A})[sss\Psi^{(IJ)}_{[MN]}]\;=\;0,$
(52)

where we have used the result listed in table 1.

With the same reasoning we get additional representations of the
$psl(4|4,\bf{R})$ algebra, in subspaces of the tensor product of three
singleton representations.

Acknowledgments

I am gratefull to Prof. Andrei Mikhailov for suggesting the problem and for
his advice. The current work was supported by the Brazilian CNPq scholarship
and in part by FAPESP grant 2014/18634-9.

##  Appendix A $\mathcal{O}[B]$ is conformal invariant

Here we check that $\mathcal{O}[B]$ is invariant under conformal
transformations. To do that we act by the generator of conformal
transformation $k^{\dot{\varrho}\varrho}$ on it, first we need the following
result

$\displaystyle[a^{\varrho}b^{\dot{\varrho}},\mathcal{R}^{(j)}]\;=\;-\frac{i}{2}\mathcal{R}^{(j)}\big{(}x^{\varrho\alpha^{\prime}}a^{\dagger}_{\alpha^{\prime}}a^{\varrho}+x^{\dot{\alpha}\varrho}b^{\dagger}_{\dot{\alpha}}b^{\dot{\varrho}}+x^{\dot{\varrho}\varrho}-\frac{i}{2}x^{\dot{\varrho}\alpha^{\prime}}x^{\dot{\rho}\varrho}a^{\dagger}_{\alpha^{\prime}}b^{\dagger}_{\dot{\rho}}\big{)}.$
(53)

Therefore $(a^{\varrho}b^{\dot{\varrho}})\mathcal{O}[B]$ is given by

$\displaystyle\int
d^{4}x\Bigg{(}\Big{(}\big{(}\frac{3}{2}\frac{\partial}{\partial
p^{(1)}_{\varrho\dot{\varrho}}}\prod_{i=1}^{3}\mathcal{R}^{(i)}+\\{\frac{\partial}{\partial
p^{(1)}_{\alpha^{\prime}\dot{\varrho}}}\frac{\partial}{\partial
p^{(1)}_{\varrho\dot{\rho}}}\prod_{i=1}^{3}\mathcal{R}^{(i)}\\}p^{(1)}_{\alpha^{\prime}\dot{\rho}}\big{)}c^{A\dagger}a_{[1}^{\dagger}|0\rangle\Big{)}\bullet
c^{B\dagger}a_{2]}^{\dagger}|0\rangle\bullet\epsilon_{ABMN}c^{M\dagger}c^{N\dagger}|0\rangle$
$\displaystyle\;+\;c^{A\dagger}a_{[1}^{\dagger}|0\rangle\bullet\Big{(}\big{(}\frac{3}{2}\frac{\partial}{\partial
p^{(2)}_{\varrho\dot{\varrho}}}\prod_{i=1}^{3}\mathcal{R}^{(i)}+\\{\frac{\partial}{\partial
p^{(2)}_{\alpha^{\prime}\dot{\varrho}}}\frac{\partial}{\partial
p^{(2)}_{\varrho\dot{\rho}}}\prod_{i=1}^{3}\mathcal{R}^{(i)}\\}p^{(2)}_{\alpha^{\prime}\dot{\rho}}\big{)}c^{B\dagger}a_{2]}^{\dagger}|0\rangle\Big{)}\bullet\epsilon_{ABMN}c^{M\dagger}c^{N\dagger}|0\rangle$
$\displaystyle\;+\;c^{A\dagger}a_{[1}^{\dagger}|0\rangle\bullet
c^{B\dagger}a_{2]}^{\dagger}|0\rangle\bullet\big{(}\frac{\partial}{\partial
p^{(3)}_{\varrho\dot{\varrho}}}\prod_{i=1}^{3}\mathcal{R}^{(i)}+\\{\frac{\partial}{\partial
p^{(3)}_{\alpha^{\prime}\dot{\varrho}}}\frac{\partial}{\partial
p^{(3)}_{\varrho\dot{\rho}}}\prod_{i=1}^{3}\mathcal{R}^{(i)}\\}p^{(3)}_{\alpha^{\prime}\dot{\rho}}\big{)}\epsilon_{ABMN}c^{M\dagger}c^{N\dagger}|0\rangle\Bigg{)},$

where we have used (53) and antisymmetrization properties. Taking the integral
we have

$\displaystyle\Big{(}\big{(}\frac{3}{2}\frac{\partial}{\partial
p^{(1)}_{\varrho\dot{\varrho}}}\delta(\sum^{3}_{i=1}p^{(i)})+\\{\frac{\partial}{\partial
p^{(1)}_{\alpha^{\prime}\dot{\varrho}}}\frac{\partial}{\partial
p^{(1)}_{\varrho\dot{\rho}}}\delta(\sum_{i=1}^{3}p^{(i)})\\}p^{(1)}_{\alpha^{\prime}\dot{\rho}}\big{)}c^{A\dagger}a_{[1}^{\dagger}|0\rangle\Big{)}\bullet
c^{B\dagger}a_{2]}^{\dagger}|0\rangle\bullet\epsilon_{ABMN}c^{M\dagger}c^{N\dagger}|0\rangle$
$\displaystyle\;+\;c^{A\dagger}a_{[1}^{\dagger}|0\rangle\bullet\Big{(}\big{(}\frac{3}{2}\frac{\partial}{\partial
p^{(2)}_{\varrho\dot{\varrho}}}\delta(\sum_{i=1}^{3}p^{(i)})+\\{\frac{\partial}{\partial
p^{(2)}_{\alpha^{\prime}\dot{\varrho}}}\frac{\partial}{\partial
p^{(2)}_{\varrho\dot{\rho}}}\delta(\sum_{i=1}^{3}p^{(i)})\\}p^{(2)}_{\alpha^{\prime}\dot{\rho}}\big{)}c^{B\dagger}a_{2]}^{\dagger}|0\rangle\Big{)}\bullet\epsilon_{ABMN}c^{M\dagger}c^{N\dagger}|0\rangle$
$\displaystyle\;+\;c^{A\dagger}a_{[1}^{\dagger}|0\rangle\bullet
c^{B\dagger}a_{2]}^{\dagger}|0\rangle\bullet\big{(}\frac{\partial}{\partial
p^{(3)}_{\varrho\dot{\varrho}}}\delta(\sum_{i=1}^{3}p^{(i)})+\\{\frac{\partial}{\partial
p^{(3)}_{\alpha^{\prime}\dot{\varrho}}}\frac{\partial}{\partial
p^{(3)}_{\varrho\dot{\rho}}}\delta(\sum_{i=1}^{3}p^{(i)})\\}p^{(3)}_{\alpha^{\prime}\dot{\rho}}\big{)}\epsilon_{ABMN}c^{M\dagger}c^{N\dagger}|0\rangle.$
(54)

From
$\delta(\sum_{i=1}^{3}p^{(i)})\\}\sum_{j=1}^{3}p^{(j)}_{\alpha^{\prime}\dot{\rho}}=0$,
we get the following identity

$\displaystyle\frac{1}{4}\sum_{l=1}^{3}\sum_{j=1}^{3}g_{lj}\\{\frac{\partial}{\partial
p^{(l)}_{\alpha^{\prime}\dot{\varrho}}}\frac{\partial}{\partial
p^{(j)}_{\varrho\dot{\rho}}}\delta(\sum_{i=1}^{3}p^{(i)})\\}\sum_{k=1}^{3}p^{(k)}_{\alpha^{\prime}\dot{\rho}}+\sum_{k=1}^{3}\sum_{l=1}^{3}g_{kl}\\{\frac{\partial}{\partial
p^{(l)}_{\varrho\dot{\varrho}}}\delta(\sum_{i=1}^{3}p^{(i)})\\}\;=\;0,$

let $g_{lm}$ be $a_{l}\delta^{l}_{m}$, so the last identity reduces to

$\displaystyle\frac{1}{4}\big{(}\sum_{l=1}^{3}a_{l}\big{)}\\{\frac{\partial}{\partial
p^{(m)}_{\alpha^{\prime}\dot{\varrho}}}\frac{\partial}{\partial
p^{(m)}_{\varrho\dot{\rho}}}\delta(\sum_{i=1}^{3}p^{(i)})\\}\sum_{k=1}^{3}p^{(k)}_{\alpha^{\prime}\dot{\rho}}+\sum_{k=1}^{3}a_{k}\\{\frac{\partial}{\partial
p^{(k)}_{\varrho\dot{\varrho}}}\delta(\sum_{i=1}^{3}p^{(i)})\\}\;=\;0,$ (55)

there is no sumation on $m$ index, Eqn. (A) can be written as

$\displaystyle\mathcal{O}(c^{A\dagger}a_{[1}^{\dagger}|0\rangle\bullet
c^{B\dagger}a_{2]}^{\dagger}|0\rangle\bullet\epsilon_{ABMN}c^{M\dagger}c^{N\dagger}|0\rangle)$
(56)

where $\mathcal{O}$ is the identity in (55), with $a_{1}=a_{2}=\frac{3}{2}$
and $a_{3}=1$. Therefore we have checked that $\mathcal{O}[B]$ is invariant
under conformal transformations.

## References

  * [1] J. M. Maldacena, Int. J. Theor. Phys. 38, 1113 (1999) [Adv. Theor. Math. Phys. 2, 231 (1998)] doi:10.1023/A:1026654312961 [hep-th/9711200].
  * [2] O. Aharony, S. S. Gubser, J. M. Maldacena, H. Ooguri and Y. Oz, Phys. Rept. 323, 183 (2000) doi:10.1016/S0370-1573(99)00083-6 [hep-th/9905111].
  * [3] H. J. Kim, L. J. Romans and P. van Nieuwenhuizen, Phys. Rev. D 32, 389 (1985). doi:10.1103/PhysRevD.32.389
  * [4] M. Gunaydin and N. Marcus, Class. Quant. Grav. 2, L11 (1985). doi:10.1088/0264-9381/2/2/001
  * [5] A. Mikhailov, JHEP 1112, 005 (2011) doi:10.1007/JHEP12(2011)005 [arXiv:1105.2231 [hep-th]].
  * [6] O. Lunin and J. M. Maldacena, JHEP 0505, 033 (2005) doi:10.1088/1126-6708/2005/05/033 [hep-th/0502086].
  * [7] O. A. Bedoya, L. I. Bevilaqua, A. Mikhailov and V. O. Rivelles, Nucl. Phys. B 848, 155 (2011) doi:10.1016/j.nuclphysb.2011.02.012 [arXiv:1005.0049 [hep-th]].
  * [8] S. Frolov, JHEP 0505, 069 (2005) doi:10.1088/1126-6708/2005/05/069 [hep-th/0503201].
  * [9] P. A. Grassi and J. Kluson, JHEP 0703, 033 (2007) doi:10.1088/1126-6708/2007/03/033 [hep-th/0611151].
  * [10] R. G. Leigh and M. J. Strassler, Nucl. Phys. B 447, 95 (1995) doi:10.1016/0550-3213(95)00261-P [hep-th/9503121].
  * [11] I. Bars and M. Gunaydin, Commun. Math. Phys. 91, 31 (1983). doi:10.1007/BF01206048
  * [12] N. Beisert, Lett. Math. Phys. 99, 529 (2012) doi:10.1007/s11005-011-0479-8 [arXiv:1012.4004 [hep-th]].
  * [13] E. Witten, Commun. Math. Phys. 252, 189 (2004) doi:10.1007/s00220-004-1187-3 [hep-th/0312171].
  * [14] R. Boels, L. J. Mason and D. Skinner, JHEP 0702, 014 (2007) doi:10.1088/1126-6708/2007/02/014 [hep-th/0604040].
  * [15] R. Grimm, M. Sohnius and J. Wess, Nucl. Phys. B 133, 275 (1978). doi:10.1016/0550-3213(78)90303-6
  * [16] E. D’Hoker and D. Z. Freedman, hep-th/0201253.
  * [17] V. K. Dobrev and V. B. Petkova, Phys. Lett. B 162, 127 (1985). doi:10.1016/0370-2693(85)91073-1
  * [18] N. Berkovits, JHEP 1004, 067 (2010) doi:10.1007/JHEP04(2010)067 [arXiv:0910.1684 [hep-th]].
  * [19] T. Adamo and L. Mason, Phys. Rev. D 86, 065019 (2012) doi:10.1103/PhysRevD.86.065019 [arXiv:1103.1352 [hep-th]].
  * [20] F. Cachazo, P. Svrcek and E. Witten, JHEP 0409, 006 (2004) doi:10.1088/1126-6708/2004/09/006 [hep-th/0403047].


# Solving The Lunar Lander Problem under Uncertainty using Reinforcement
Learning
††thanks: 978-1-7281-6861-6/20/$31.00 ©2020 IEEE

Soham Gadgil  Computer Science Department
Stanford University
<EMAIL_ADDRESS>Yunfeng Xin  Electrical Engineering Department
Stanford University
<EMAIL_ADDRESS>Electrical Engineering Department
Stanford University
<EMAIL_ADDRESS>and Chengzhe Xu

###### Abstract

Reinforcement Learning (RL) is an area of machine learning concerned with
enabling an agent to navigate an environment with uncertainty in order to
maximize some notion of cumulative long-term reward. In this paper, we
implement and analyze two different RL techniques, Sarsa and Deep Q-Learning,
on OpenAI Gym’s LunarLander-v2 environment. We then introduce additional
uncertainty to the original problem to test the robustness of the mentioned
techniques. With our best models, we are able to achieve average rewards of
170+ with the Sarsa agent and 200+ with the Deep Q-Learning agent on the
original problem. We also show that these techniques are able to overcome the
additional uncertainties and achieve positive average rewards of 100+ with
both agents. We then perform a comparative analysis of the two techniques to
conclude which agent performs better.

###### Index Terms:

Deep Reinforcement Learning, Neural Networks, Q-Learning, Robotics, Control

##  I Introduction

Over the past several years, reinforcement learning [1] has been proven to
have a wide variety of successful applications including robotic control [2,
3]. Different approaches have been proposed and implemented to solve such
problems [4, 5]. In this paper, we solve a well-known robotic control problem
– the lunar lander problem – using different reinforcement learning
algorithms, and then test the agents’ performances under environment
uncertainties and agent uncertainties. The problem is interesting since it
presents a simplified version of the task of landing optimally on lunar
surface, which in itself has been a topic of extensive research [6, 7, 8]. The
added uncertainties are meant to model the situations that a real spacecraft
might face while landing on the lunar surface.

##  II Problem Definition

We aim to solve the lunar lander environment in the OpenAI gym kit using
reinforcement learning methods.111Our code is available at
https://github.com/rogerxcn/lunar_lander_project The environment simulates the
situation where a lander needs to land at a specific location under low-
gravity conditions, and has a well-defined physics engine implemented.

The main goal of the game is to direct the agent to the landing pad as softly
and fuel-efficiently as possible. The state space is continuous as in real
physics, but the action space is discrete.

##  III Related Work

There has been previous work done in solving the lunar lander environment
using different techniques. [9] makes use of modified policy gradient
techniques for evolving neural network topologies. [10] uses a control-model-
based approach that learns the optimal control parameters instead of the
dynamics of the system. [11] explores spiking neural networks as a solution to
OpenAI virtual environments.

These approaches show the effectiveness of a particular algorithm for solving
the problem. However, they do not consider additional uncertainty. Thus, we
aim to first solve the lunar lander problem using traditional Q-learning
techniques, and then analyze different techniques for solving the problem and
also verify the robustness of these techniques as additional uncertainty is
added.

Figure 1: Visualization of the lunar lander problem.

##  IV Model

###  IV-A Framework

The framework used for the lunar lander problem is gym, a toolkit made by
OpenAI [12] for developing and comparing reinforcement learning algorithms. It
supports problems for various learning environments, ranging from Atari games
to robotics. The simulator we use is called Box2D and the environment is
called LunarLander-v2.

###  IV-B Observations and State Space

The observation space determines various attributes about the lander.
Specifically, there are 8 state variables associated with the state space, as
shown below:

$\displaystyle state\rightarrow\left\\{\begin{array}[]{ll}\text{$x$ coordinate
of the lander}\\\ \text{$y$ coordinate of the lander}\\\ \text{$v_{x}$, the
horizontal velocity}\\\ \text{$v_{y}$, the vertical velocity}\\\
\text{$\theta$, the orientation in space}\\\ \text{$v_{\theta}$, the angular
velocity}\\\ \text{Left leg touching the ground (Boolean)}\\\ \text{Right leg
touching the ground (Boolean)}\end{array}\right.$

All the coordinate values are given relative to the landing pad instead of the
lower left corner of the window. The $x$ coordinate of the lander is $0$ when
the lander is on the line connecting the center of the landing pad to the top
of the screen. Therefore, it is positive on the right side of the screen and
negative on the left. The $y$ coordinate is positive at the level above the
landing pad and is negative at the level below.

###  IV-C Action Space

There are four discrete actions available: do nothing, fire left orientation
engine, fire right orientation engine, and fire main engine. Firing the left
and right engines introduces a torque on the lander, which causes it to
rotate, and makes stabilizing difficult.

###  IV-D Reward

Defining a proper reward directly affects the performance of the agent. The
agent needs to maintain both a good posture mid-air and reach the landing pad
as quickly as possible. Specifically, in our model, the reward is defined to
be:

$\begin{split}\text{Reward}(s_{t})=-100*(d_{t}-d_{t-1})-100*(v_{t}-v_{t-1})\\\
-100*(\omega_{t}-\omega_{t-1})+\text{ hasLanded}(s_{t})\end{split}$ (1)

where $d_{t}$ is the distance to the landing pad, $v_{t}$ is the velocity of
the agent, and $\omega_{t}$ is the angular velocity of the agent at time $t$.
$hasLanded()$ is the reward function of landing, which is a linear combination
of the boolean state values representing whether the agent has landed softly
on the landing pad and whether the lander loses contact with the pad on
landing.

With this reward function, we encourage the agent to lower its distance to the
landing pad, decrease the speed to land smoothly, keep the angular speed at
minimum to prevent rolling, and not to take off again after landing.

Figure 2: Performance comparison between Sarsa agent under naive state
discretization and random agent.

##  V Approaches

###  V-A Sarsa

Since we do not encode any prior knowledge about the outside world into the
agent and the state transition function is hard to model, Sarsa [13] seems to
be a reasonable approach to train the agent using an exploration policy. The
update rule for Sarsa is:

$Q(s_{t},a_{t})=Q(s_{t},a_{t})+\\\
\alpha[r_{t}+Q(s_{t+1},a_{t+1})-Q(s_{t},a_{t})]$ (2)

At any given state, the agent chooses the action with the highest Q value
corresponding to:

$\displaystyle a=argmax_{a\in actions}Q(s,a)$ (3)

From the equation we can see that we need to discretize the states and assign
Q values for each state-action pair, and that we also need to assign a policy
to balance exploitation and exploration since Sarsa is an on-policy algorithm.

Intuitively, a simple exploration policy can be the $\epsilon$-greedy policy
[14], where the agent randomly chooses an action with probability $\epsilon$
and chooses the best actions with probability $1-\epsilon$. A simple way of
discretizing the state is to divide each continuous state variable into
several discrete values. However, as shown in Fig. 2, the result shows that
the agent can only reach marginally better performance than a random agent,
and cannot manage to get positive rewards even after 10,000 episodes of
training. It then becomes obvious that we cannot simply adopt these algorithms
out-of-the-box, and we need to tailor them for our lunar lander problem.

Figure 3: Diagram showing the state discretization and generalization scheme
of the x coordinate state variable.

####  V-A1 State Discretization

There are six continuous state variables and two boolean state variables, so
the complexity of the state space is on the order of $O(n^{6}\times 2\times
2)$, where $n$ is the number of discretized values for each state variable.
Thus, even discretizing each state variable into 10 values can produce 400,000
states, which is far too large to explore within a reasonable number of
training episodes. It explains the poor performance we were seeing previously.
Therefore, we need to devise a new discretization scheme for these states.

Specifically, we examine the initial Q values learned by the agent and observe
that the agent wants to move to the right when it is in the left half of the
space, and move to the left when it is in the right half of the space. As a
result, all the x coordinates far from the center can be generalized into one
single state because the agent will always tend to move in one direction (as
shown in Fig. 3), which helps reduce the state space.

Therefore, we define the discretization of the x coordinate at any state with
the discretization step size set at 0.05 as:

$\displaystyle
d(x)=\min(\lfloor\frac{n}{2}\rfloor,\max(-\lfloor\frac{n}{2}\rfloor,\frac{x}{\text{0.05}}))$
(4)

The same intuition of generalization is applied to other state variables as
well. In total, we experiment with 4 different ways of discretizing and
generalizing the states with different number of discretization levels.

####  V-A2 Exploration Policy

As the size of the state space exceeds 10,000 even after the process of
discretization and generalization, the probability $\epsilon$ in the
$\epsilon$-greedy policy needs to be set to a fairly large number ($\sim$20%)
for the agent to explore most of the states within a reasonable number of
episodes. However, this means that the agent will pick a sub-optimal move once
every five steps.

In order to minimize the performance impact of the $\epsilon$-greedy policy
while still getting reasonable exploration, we decay $\epsilon$ in different
stages of training. The intuition is that the agent in the initial episodes
knows little about the environment, and thus more exploration is needed. After
an extensive period of exploration, the agent has learned enough information
about the outside world, and needs to switch to exploitation so that the Q
value for each state-action pair can eventually converge.

Specifically, the epsilon is set based on the following rules:

$\displaystyle\epsilon=\left\\{\begin{array}[]{ll}0.5&\text{\\#Iteration}\in[0,100)\\\
0.2&\text{\\#Iteration}\in[100,500)\\\ 0.1&\text{\\#Iteration}\in[500,2500)\\\
0.01&\text{\\#Iteration}\in[2500,7500)\\\
0&\text{\\#Iteration}\in[7500,10000)\end{array}\right.$

###  V-B Deep Q-Learning

Since there is model uncertainty in the problem, Q-learning is another
approach which can be used to solve the environment. For this problem, we use
a modified version of Q-learning, called deep Q-learning (DQN) [15, 16] to
account for the continuous state space. The DQN method makes use of a multi-
layer perceptron, called a Deep Q-Network (DQN), to estimate the Q values. The
input to the network is the current state (8-dimensional in our case) and the
outputs are the Q values for all state-action pairs for that state. The
Q-Learning update rule is as follows:

$Q(s,a)=Q(s,a)+\alpha(r+\gamma\max\limits_{a^{\prime}}Q(s^{\prime},a^{\prime})-Q(s,a))$
(5)

The optimal $Q$-value $Q^{*}(s,a)$ is estimated using the neural network with
parameters $\theta$. This becomes a regression task, so the loss function at
iteration $i$ is obtained by the temporal difference error:

$\displaystyle\mathcal{L}_{i}(\theta_{i})$
$\displaystyle=\mathbbm{E}_{(s,a)\sim\rho(.)}[(y_{i}-Q(s,a;\theta_{i}))^{2}]$
(6)

where

$\displaystyle
y_{i}=\mathbbm{E}_{s^{\prime}\sim\mathcal{E}}[r+\gamma\max\limits_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta_{i-1})]$
(7)

Here, $\theta_{i-1}$ are the network parameters from the previous iteration,
$\rho(s,a)$ is a probability distribution over states $s$ and actions $a$, and
$\mathcal{E}$ is the environment the agent interacts with. Gradient descent is
then used to optimise the loss function and update the neural network weights.
The neural network parameters from the previous iteration, $\theta_{i-1}$, are
kept fixed while optimizing $\mathcal{L}_{i}(\theta_{i})$. Since the next
action is selected based on the greedy policy, Q-learning is an off policy
algorithm [17].

One of the challenges here is that the successive samples are highly
correlated since the next state depends on the current state and action. This
is not the case in traditional supervised learning problems where the
successive samples are i.i.d. To tackle this problem, the transitions
encountered by the agent are stored in a replay memory $\mathcal{D}$. Random
minibatches of transitions $\\{s,a,r,s^{\prime}\\}$ are sampled from
$\mathcal{D}$ during training to train the network. This technique is called
experience replay [18].

We use a 3 layer neural network, as shown in Fig. 4, with 128 neurons in the
hidden layers. We use $ReLU$ activation for the hidden layers and $LINEAR$
activation for the output layer. The number of hidden neurons are chosen based
on analyzing different values, as shown in section $6.2.1$. The learning rate
used is $0.001$ and the minibatch size is $64$.

$\vdots$$\vdots$$\vdots$$\vdots$$s_{1}$$s_{2}$$s_{3}$$s_{8}$$h_{1}$$h_{128}$$h_{1}$$h_{128}$$a_{1}$$a_{4}$InputlayerHiddenlayerHiddenlayerOuputlayer
Figure 4: Neural Network used for Deep Q-Learning

####  V-B1 Exploration Policy

Similar to Sarsa, an improved $\epsilon$-greedy policy is used to select the
action, with $\epsilon$ starting at 1 to favor exploration and decaying by
$0.996$ for every iteration, until it reaches 0.01, after which it stays the
same.

##  VI Experiments

###  VI-A The Original Problem

The goal of the problem is to direct the lander to the landing pad between two
flag poles as softly and fuel efficiently as possible. Both legs of the lander
should be in contact with the pad while landing. The lander should reach the
landing pad as quickly as possible, while maintaining a stable posture and
minimum angular speed. Also, once landed, the lander should not take off
again. In the original problem, uncertainty is added by applying a randomized
force to the center of the lander at the beginning of each iteration. This
causes the lander to be pushed in a random direction. The lander must recover
from this force and head to the landing pad.

We experiment with tackling the original problem using Sarsa and deep
Q-learning as described in our approach section, and our observations are
demonstrated in section $7$.

###  VI-B Introducing Additional Uncertainty

After solving the original lunar lander problem, we analyze how introducing
additional uncertainty can affect the performance of the agents and evaluate
their robustness to different uncertainties.

####  VI-B1 Noisy Observations

Retrieving the exact state of an object in a physical world can be hard, and
we need to rely on noisy observations such as a radar to infer the real state
of the object. Thus, instead of directly using the exact observations provided
by the environment, we add a zero-mean Gaussian noise of scale 0.05 into our
observation of the location of the lander. The standard deviation is
deliberately set to 0.05, which corresponds to our discretization step size.
Specifically, for each observation of $x$, we sample a random zero-mean
Gaussian noise

$s\sim\mathcal{N}(\mu=0,\sigma=0.05)$ (8)

and add the noise to the observation, so that the resulting random variable
becomes

$Observation(x)\sim\mathcal{N}(x,0.05)$ (9)

We then evaluate the resulting performance of two agents: one using the
original Q values from the original problem, and the other using the Q values
trained under such noisy observations.

We notice that we can frame this problem as a POMDP (Partially Observable
Markov Decision Process), and compare its performance with the two Sarsa
agents mentioned above. We calculate the alpha vector of each action using
one-step lookahead policy using the Q values from the original problem, and
calculate the belief vector using the Gaussian probability density function

$\text{PDF}(x)=\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(o(x)-x)^{2}}{2\sigma^{2}}}$
(10)

This way, we can get the expected utility of each action under uncertainty by
taking the inner product of the corresponding alpha vector and the belief
vector. The resulting policy simply picks the action with the highest expected
utility.

Notice that we could have used transition probabilities of locations to assist
in determining the exact location of the agent. However, after experimenting
with different transition probability functions, we concluded that the
transition probability in a continuous space is very hard to model, and naive
transition probability functions will cause the agent to perform even worse
than the random agent.

Figure 5: Performance comparison of our state discretization and policy scheme
(green), naive state discretization and policy scheme (blue), and random agent
(grey).

####  VI-B2 Random Engine Failure

Another source of uncertainty in the physical world can be random engine
failures due to the various unpredictable conditions in the agent’s
environment. The model needs to be robust enough to overcome such failures
without impacting performance too much. To simulate this, we introduce action
failure in the lunar lander. The agent takes the action provided $80\%$ of the
time, but $20\%$ of the time the engines fail and it takes no action even
though the provided action is firing an engine.

####  VI-B3 Random Force

Uncertainty can also come from unstable particle winds in the space such as
solar winds, which result in random forces being applied to the center of the
lander while landing. The model is expected to manage the random force and
have stable Q maps with enough fault tolerance.

We apply a random force each time the agent interacts with the environment and
modify the state accordingly. The force is sampled from a Gaussian
distribution for better simulation of real-world random forces. The mean and
variance of the Gaussian distribution are set in proportion to the engine
power to avoid making the random force either too small to have any effect on
the agent or too large to maintain control of the agent.

Figure 6: Performance comparison of different state discretization schemes in
Sarsa.

##  VII Results and Analysis

###  VII-A Sarsa

####  VII-A1 The Original Problem

Fig. 5 shows the average reward acquired (over the previous 10 episodes) by
the the random agent and the Sarsa agent with naive state discretization and
our customized discretization scheme. With a naive discretization which
quantizes each state variable into 10 equal steps, the agent cannot learn
about the outside world very effectively even after 10,000 episodes of
training. Due to the huge state space, the acquired reward is only marginally
better than the random agent. In contrast, with our customized discretization
scheme which combines small-step discretization and large-step generalization,
the agent is able to learn the Q values rapidly and gather positive rewards
after 500 episodes of training. The results show that for continuous state
spaces, proper discretization and generalization of the states helps the agent
learn the values faster and better.

Fig. 6 also shows how different discretization schemes affect the learning
speed and the final performance the agent is able to achieve. The notation
$aXbY$ is used to denote that the the x coordinate is discretized into $a$
levels while the y coordinate is discretized into $b$ levels. The results
indicate that as the number of discretization levels increases, the agent in
general learns the Q values more slowly, but is able to achieve higher
performance after convergence. The discretization scheme 5X4Y does slightly
better than the scheme 7X5Y, indicating that further discretization will not
help.

####  VII-A2 Handling Noisy Observations

Fig. 7 shows the results after feeding the noisy observations under three
agents. The first agent uses the Q values learned from the original problem
under discretization scheme 5X4Y and takes the noisy observations as if they
were exact. The second agent is re-trained under the noisy environment using
the noisy observations. The third agent uses the Q values learned from the
original problem, but uses one-step lookahead alpha vectors to calculate the
expected reward for each action.

Each data point in Fig. 7 represents the average acquired reward in 10
episodes and can help eliminate outliers. The results show that the POMDP
agent (POMDP in Fig. 7) receives the highest average rewards and outperforms
the other agents. Of the other two agents, the agent trained under noisy
observations (Trained Q in Fig. 7) fails to generalize information from these
episodes.

In general, there is a significant performance impact in terms of average
acquired rewards with the added uncertainty of noisy observation, and such a
result is expected: when the agent is close to the center of the space, a
noisy x observation can significantly change the action which the agent picks.
For example, when $x=0.05$, a noisy observation has a 15.87% probability of
flipping the sign so that $x<0$ according the Gaussian cumulative distribution
function

$\text{CDF}(x)=\frac{1}{\sqrt{2\pi}\times
0.05}\int_{-\infty}^{-0.05}e^{-\frac{(x-0.05)^{2}}{2\times 0.05^{2}}}dx$ (11)

This means that the noise has a decent chance of tricking the agent into
believing that it is in the left half of the screen while it is in fact in the
right half of the screen. Therefore, the agent will pick an action that helps
the agent move right, instead of original optimal action of moving left.

The POMDP agent has the correct learned Q value and takes the aforementioned
sign-flipping observation scenario into account using the belief vector, which
explains why it is performing the best by getting the highest average rewards.
The agent trained under noisy observation, however, is learning the wrong Q
value in the wrong state due to the noisy observation and does not take the
noisy observation into account. Thus, it is performing the worst of all three
agents.

Figure 7: Comparison of different agent performance under noisy observations.
Figure 8: Comparison of Sarsa agent performance under different random forces.

####  VII-A3 Handling Random Force

Fig. 8 shows the result after applying different random forces to the Sarsa
agent under state discretization of 5X4Y.

In the experiments, we introduce three kinds of random forces: regular, medium
and large. In the regular cases, we ensure that random forces do not go too
much beyond the agent’s engine power. In the medium case, we relax that
constraint, and in the large case, we ensure that the agent cannot control
itself because the random force becomes much larger than the engine power. The
details are described as follows:

1) regular$\\_$00: mean equals 0 and variance equals $engine\\_power/3$

2) regular$\\_$01: mean equals 0 on the x-axis and $engine\\_power/6$ on the
y-axis, variance equals $engine\\_power/3$

3) regular$\\_$10: mean equals $engine\\_power/6$ on the x-axis and 0 on the
y-axis, variance equals $engine\\_power/3$

4) regular$\\_$11: mean equals $engine\\_power/6$ on both x-axis and y-axis,
variance equals $engine\\_power/3$

5) medium: mean equals $engine\\_power$ on both x-axis and y-axis, variance
equals $engine\\_power*3$

6) large: mean equals $engine\\_power*2$ on both x-axis and y-axis, variance
equals $engine\\_power*5$

The result suggests that agents would perform well and offset the effect of
the random force in regular cases, while in the medium and large cases where
random forces are more likely to exceed the maximum range engines could
compensate, there would be an obvious reduction in reward indicating that the
random forces make landing harder. The results reveal the fact that Sarsa
agents have learned a robust and smooth Q map where similar state-action pairs
would have similar Q value distributions. Slight state variations caused by
random forces would have small influences on Q value and action selection,
which increases the fault tolerance of the agent.

When state variations become too large, Q maps would be noisy and total
rewards would decrease, in which case agents would tend to lose control
because of the random forces.

Figure 9: Comparison of Sarsa agent performance under engine failure

####  VII-A4 Handling Random Engine Failure

Fig. 9 shows the results after adding engine failure to the original
environment. The first agent uses Q values learned from the original problem
under discretization scheme 5X4Y and the second agent is retrained in the
environment with engine failure. The agent reusing Q values (Vanilla Q in Fig.
9) does well and is able to achieve positive average rewards of 100+. This
shows that the Sarsa agent using the discretization scheme is robust enough to
recover from sudden failures in the engines. The retrained Sarsa agent
(Trained Q in Fig. 9) is not able to achieve rewards as high as the vanilla Q
agent, indicating that the agent is not able to generalize information from
these episodes.

Again, there is a performance drop with the added uncertainty of engine
failure. This is understandable, since an engine failure causes the lander to
behave unexpectedly and the effect gets compounded as more engine failures
take place.

###  VII-B Deep Q-Learning

####  VII-B1 The Original Problem

Fig. 10 shows the average reward obtained (over the previous 10 episodes) by
the DQN agent for different number of neurons in the hidden layers of the
neural network used for predicting the Q-values. At the start, the average
reward is negative since the agent is essentially exploring all the time and
collecting more transitions for storing in memory $\mathcal{D}$. As the agent
learns, the Q-values start to converge and the agent is able to get positive
average reward. For both the plots, the DQN agent is able to converge pretty
quickly, doing so in $\sim$ 320 iterations. The DQN implementation performs
well and results in good scores. However, the training time for the DQN agent
is quite long ($\sim 8$ hours). It is observed that the neural network with
$128$ neurons in the hidden layers reaches the highest converged average
rewards, and is chosen as the hyperparameter value.

Figure 10: Average reward obtained by DQN agent for different number of hidden
neurons Figure 11: Comparison of DQN agent performance under different random
forces

####  VII-B2 Handling Random Force

Fig. 11 shows the result after applying different random forces described in
section VII-A3 to the DQN agent. The result looks similar to the one obtained
with the Sarsa agent. The agent performs well in the the regular cases and the
obtained reward curves are very similar. In the the medium and the large
cases, the average reward drops and the agent performs the worst for the large
case, and is barely able to get rewards above 100. This suggests that the
agent is able to adapt well to slight deviations in the state of the lander
due to the random forces and the rewards obatined are almost equal to the ones
obtained in the original environment. However, when the random forces become
too large, the agent is not able to overcome the effect of the forces to land
optimally, and there is a reduction in the rewards obtained.

Figure 12: Comparison of DQN agent performance under noisy observations

####  VII-B3 Handling Noisy Observations

Fig. 12 shows a comparison between the agent using Q-values learned from the
original problem (Vanilla DQN) and the retrained DQN agent with noisy
observations (Trained DQN). The DQN agent trained under noisy observations is
able to obtain positive rewards but fails to perform as well as the vanilla
DQN agent. This can also be because a noisy x observation close to the center
of the environment can significantly affect the action chosen by the agent.
However, unlike the Sarsa agent which fails to generalize any information, the
retrained DQN agent is able to capture some information about the environment
even with noisy observations, which can be seen by the upward trend of the
reward curve.

####  VII-B4 Handling Random Engine Failure

Fig. 13 shows a comparison between the agent using Q-values learned from the
original problem (Vanilla DQN), the re-trained DQN agent when there are engine
failures, and the original DQN agent when there are no engine failures. For
all the plots, the number of neurons in the hidden layers of the neural
network is 128. The vanilla DQN agent performs well on the new environment
with random engine failures and is able to obtain positive average rewards of
$100+$. This shows that even without retraining, the original agent is able to
adjust to the uncertainty.

For the re-trained agents, at the start, the curves are similar since this is
the exploration phase. Both agents are taking random actions and engine
failure does not affect the reward much. However, in the later iterations, as
the agents learn the environment, the lander without engine failure achieves
higher average reward and is less erratic as compared to the lander with
engine failure. This is expected since the random engine failures require the
agent to adjust to the unexpected behavior while trying to maximise reward.
However, even with engine failure, the agent shows the same increasing trend
in average reward as the original problem and is able to achieve positive
rewards around $100$. Also, both the retrained agents achieve higher average
rewards than the vanilla DQN. This shows that the DQN agent is able to
estimate the optimal Q-values well even with the added uncertainty.

Figure 13: Comparison of DQN agents under engine failure

###  VII-C Comparative Analysis

Based on the results of testing the agents in different environments, we can
observe that both the Sarsa and DQN agents are able to achieve rewards of 100+
even in cases of added uncertainties, except for the retrained Sarsa agent
under noisy observations, in which case the POMDP agent performs well. This
shows that both the methods work well in practice and are robust enough to
handle environments with added uncertainty.

Comparing the results of the two methods, it can be observed that the DQN
agent with a 3-layer neural network of 128 hidden neurons consistently gets
higher average rewards, both for the original problem and for the problems
with added uncertainty, than the Sarsa agent under the 5X4Y discretization
scheme. This can be because with the Sarsa agent, we lose information about
the state on discretization, which can affect how well the agent learns the
environment. The DQN agent doesn’t discretize the state space and uses all the
information that the environment provides. Also, Q-learning is an off policy
algorithm in which it learns the optimal policy using the absolute greedy
policy by selecting the next action which maximises the Q-value. Since this is
a simulation, the agent’s performance during the training process doesn’t
matter. In such situations, the DQN agent would perform better since it learns
an optimal greedy policy which we switch to eventually.

However, the DQN agent seems to be more erratic than the Sarsa agent,
especially in the environment with noisy observations. There are drops in the
acquired average reward for both the agents, which can be because of the
randomness associated with the original environment and the added
uncertainties. These drops are more frequent in the DQN agent than the Sarsa
agent, which shows that even though the DQN agent is able to achieve higher
average rewards, it is not as stable as the Sarsa agent. Also, the DQN agent
takes twice as long to train as the Sarsa agent.

## Conclusion

In conclusion, we observe that both the Sarsa and the DQN agents perform well
on the orignal lunar lander problem. When we introduce additional uncertainty,
both agents are able to adapt to the new environments and achieve positive
rewards. However, the re-trained Sarsa agent fails to handle noisy
observations. This is understandable since the noisy observations affect the
underlying state space and the agent isn’t able to generalize information from
its environment during training. The POMDP agent performs well with noisy
observations and is able to get positive average rewards since it makes use of
belief vectors to model a distribution over the possible next states. Overall,
the DQN agent performs better than the Sarsa agent.

For future work, we would like to combine the different uncertainties together
and analyze how the different agents perform. This will provide a more
holistic overview of the robustness of different agents.

## References

  * [1] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [2] J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,” _The International Journal of Robotics Research_ , vol. 32, no. 11, pp. 1238–1274, 2013.
  * [3] M. Riedmiller, T. Gabel, R. Hafner, and S. Lange, “Reinforcement learning for robot soccer,” _Autonomous Robots_ , vol. 27, no. 1, pp. 55–73, 2009.
  * [4] L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: A survey,” 1996.
  * [5] C. Szepesvári, “Algorithms for reinforcement learning,” _Synthesis lectures on artificial intelligence and machine learning_ , vol. 4, no. 1, pp. 1–103, 2010.
  * [6] T. Brady and S. Paschall, “The challenge of safe lunar landing,” in _2010 IEEE Aerospace Conference_. IEEE, 2010, pp. 1–14.
  * [7] D.-H. Cho, B.-Y. Jeong, D.-H. Lee, and H.-C. Bang, “Optimal perilune altitude of lunar landing trajectory,” _International Journal of Aeronautical and Space Sciences_ , vol. 10, no. 1, pp. 67–74, 2009.
  * [8] X.-L. Liu, G.-R. Duan, and K.-L. Teo, “Optimal soft landing control for moon lander,” _Automatica_ , vol. 44, no. 4, pp. 1097–1103, 2008.
  * [9] A. Banerjee, D. Ghosh, and S. Das, “Evolving network topology in policy gradient reinforcement learning algorithms,” in _2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP)_ , Feb 2019, pp. 1–5.
  * [10] Y. Lu, M. S. Squillante, and C. W. Wu, “A control-model-based approach for reinforcement learning,” 2019.
  * [11] C. Peters, T. Stewart, R. West, and B. Esfandiari, “Dynamic action selection in openai using spiking neural networks,” 2019. [Online]. Available: https://www.aaai.org/ocs/index.php/FLAIRS/FLAIRS19/paper\\\/view/18312/17429
  * [12] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016.
  * [13] M. J. Kochenderfer, _Decision making under uncertainty: theory and application_. MIT press, 2015.
  * [14] E. Rodrigues Gomes and R. Kowalczyk, “Dynamic analysis of multiagent q-learning with $\epsilon$-greedy exploration,” in _Proceedings of the 26th Annual International Conference on Machine Learning_ , ser. ICML ’09. New York, NY, USA: ACM, 2009, pp. 369–376. [Online]. Available: http://doi.acm.org/10.1145/1553374.1553422
  * [15] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” 2013.
  * [16] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, p. 529, 2015.
  * [17] C. J. Watkins and P. Dayan, “Q-learning,” _Machine learning_ , vol. 8, no. 3-4, pp. 279–292, 1992.
  * [18] L.-J. Lin, “Reinforcement learning for robots using neural networks,” Carnegie-Mellon Univ Pittsburgh PA School of Computer Science, Tech. Rep., 1993\.