#  Renormalization groupoids in algebraic topology

Jack Morava  Department of Mathematics, The Johns Hopkins University,
Baltimore, Maryland 21218<EMAIL_ADDRESS>
###### Abstract.

Continuing work begin in [19, 27], we interpret the Hurewicz homomorphism for
Baker and Richter’s noncommutative complex cobordism spectrum $M\xi$ in terms
of characteristic numbers (indexed by quasi-symmetric functions) for complex-
oriented quasitoric manifolds, and show that automorphisms or cohomology
operations on this representation are defined by the ‘renormalization’ Hopf
algebra ${\bf N}_{*}$ of formal diffeomorphisms at the origin of the
noncommutative line, previously considered (over ${\mathbb{Q}}$) in quantum
electrodynamics [3].

The resulting structure can be presented in purely algebraic terms, as a
groupoid scheme over ${\mathbb{Z}}$ defined by a coaction of ${\bf N}_{*}$ on
the ring ${\sf N}_{*}$ of noncommutative symmetric functions. We sketch some
applications to symplectic toric manifolds, combinatorics of simplicial
spheres, and statistical mechanics.

Introduction

In [1, 2] Andrew Baker and Birgit Richter defined a remarkable noncommutative
analog $M\xi$ of the complex cobordism spectrum $M{\rm U}$. The homology
$H_{*}(M{\rm U},{\mathbb{Z}})$ is a free polynomial algebra with one generator
in each even degree, while $H_{*}(M\xi,{\mathbb{Z}})$ is a free associative
algebra with one generator in each even degree. The homology of $M{\rm U}$ can
be identified with the algebra of ‘characteristic numbers’ of complex-oriented
manifolds (defined for example by the Chern-Weil theory of integrals of
curvature forms), while $H_{*}(M\xi,{\mathbb{Z}})$ can be identified with
invariants defined by quasi-symmetric functions of the line bundles associated
to complex-oriented quasi-toric manifolds.

In the late 1960s, work of Landweber and Novikov led to an analysis of the
cohomology operations in complex cobordism in terms of what has come to be
understood as a ‘Hopf algebroid’ structure

$M{\rm U}_{*}M{\rm U}\cong M{\rm U}_{*}\otimes{\bf S}_{*}\;,$

where ${\bf S}_{*}$ is the commutative but noncocommutative Hopf algebra
representing the group of formal diffeomorphisms of the line at the origin,
known previously (over ${\mathbb{Q}}$) to combinatorialists as the Faa di
Bruno algebra. An easy corollary identifies

$H_{*}(M{\rm U}\wedge M{\rm U},{\mathbb{Z}})\cong H_{*}(M{\rm
U},{\mathbb{Z}})\otimes{\bf S}_{*}$

in terms of that algebra, and the ring of characteristic numbers.

It is fair to say that the noncommutative ring $M\xi_{*}M\xi$ defining the
cohomology operations in $M\xi_{*}$ is not yet well-understood, but Baker and
Richter show that it injects into $H_{*}(M\xi\wedge M\xi,{\mathbb{Z}})$, and
that this injection becomes an isomorphism over ${\mathbb{Q}}$. The purpose of
this note is to identify

$H_{*}(M\xi\wedge M\xi.{\mathbb{Z}})\cong H_{*}(M\xi,{\mathbb{Z}})\otimes{\bf
N}_{*}$

in terms of a remarkable generalization ${\bf N}_{*}$ of ${\bf S}_{*}$,
neither commutative nor cocommutative, already known (over ${\mathbb{Q}}$) to
physicists [3, 4] as the ‘renormalization’ Hopf algebra of formal
diffeomorphisms at the origin of the noncommutative line.

Since the work of Quillen it has become clear that it is useful to think of
complex cobordism in terms of the moduli object for one-dimensional formal
group laws. As a first step toward a deeper understanding of $M\xi$, we
suggest an interpretation of the algebra $H_{*}({\mathbb{C}}P^{\infty}\wedge
M\xi,{\mathbb{Z}})$ as functions on a noncommutative ‘grand canonical
ensemble’ of characteristic numbers of symplectic quasitoric manifolds, with
morphisms defined by noncommutative renormalization.

§I Some algebra

[This section reviews the basic properties of the algebra of symmetric
functions (used in the theory of characteristic classes) following [21], and
of the Hopf algebra representing the group of formal diffeomorphisms of the
line at the origin used in cobordism theory [31]. It then summarizes some work
of Brouder, Frabetti, Krattenthaler, and Schmitt [3, 4] on a (neither
commutative nor cocommutative) Hopf algebra representing formal
diffeomorphisms of the noncommutative line, used in renormalization theory.]

§1 The unique self-dual irreducible positive graded Hopf algebra

1.1 Let ${\mathbb{Z}}[x_{*}]={\mathbb{Z}}_{k\geq 1}[x_{k}]$ be the graded
polynomial algebra on generators $x_{k}$ of degree $|x_{k}|=2$; then the
subalgebra

${\mathbb{Z}}[e_{*}]={\mathbb{Z}}_{k\geq 1}[e_{k}]:={\sf
S}_{*}\subset{\mathbb{Z}}[x_{*}]$

of elementary symmetric ‘functions’ [21]( I §2.7 p 22) is generated by
elements $e_{k}$ of degree $2k$, defined by the formal series

$e(T)=\prod_{k\geq 1}(1+x_{k}T)=\sum_{j\geq 0}e_{j}T^{j}\in{\sf
S}_{*}[[T]]\;.$

Similarly,

$h(T)=e(-T)^{-1}=\prod_{k\geq 1}(1-x_{k}T)^{-1}=\sum_{j\geq
0}h_{j}T^{j}\in{\sf S}_{*}[[T]]\;$

defines the ‘complete’ symmetric functions of degree $|h_{k}|=2k$; thus
$e_{0}=h_{0}=1,\;e_{1}=h_{1}\dots$. If $I=1^{i_{1}}2^{i_{2}}\cdots r^{i_{r}}$
is an (unordered) partition of $|I|=\sum_{1\leq k\leq r}ki_{k}$ into $r(I)=r$
parts, let

$e_{I}:=\prod_{1\leq k\leq r}e_{k}^{i_{k}}\in{\sf S}_{2|I|}\;,$

and similarly for $h_{I}$, etc. Elements of the form $e_{I}$ (or $h_{I}$ or
$p_{I}$, see [21](I §6) and further below) provide bases for ${\sf S}_{*}$.

The coproduct $\Delta_{\sf S}:{\sf S}_{*}\to{\sf
S}_{*}\otimes_{\mathbb{Z}}{\sf S}_{*}$ defined by

$\Delta e(T)=e(T)\otimes e(T)$

(undecorated $\otimes$ signifies $\otimes_{\mathbb{Z}}$) and the antipode

$\chi_{\sf S}(e_{n})=\sum_{|I|=n}(-1)^{r(I)}e_{I}$

make ${\sf S}_{*}$ into a binomial Hopf algebra. The group-valued Witt functor

$A\mapsto({\rm{sp}\;}{\sf S})(A)={\rm Hom}_{\rm alg}({\sf S},A)$

on commutative rings (ignoring the grading)111sp is the canonical
contravariant functor from a category to its opposite sends a ring
homomorphism $\alpha:{\sf S}\to A$ to the (multiplicatively invertible) power
series

$\alpha(e)(T)=\sum_{k\geq 0}\alpha(e_{k})T^{k}\in(1+TA[[T]])^{\times}\;;$

the lost grading can be recovered from an action of the multiplicative group,
as suggested below.

1.2 P Hall’s positive definite inner product [21](§1.4 p 63, 1.5 ex 25 p 91)
on ${\sf S}_{*}$ defines a canonical isomorphism with its dual Hopf algebra,
rendering some applications to topology confusing. The (primitive) power sums

$p(-T):=\sum_{k\geq 1}p_{k}T^{k-1}=e(T)^{-1}\cdot e^{\prime}(T)\in{\sf
S}_{*}[[T]]$

define an orthogonal basis $p_{I}$ for ${\sf S}_{*}\otimes{\mathbb{Q}}$, while
the monomial symmetric functions

$m_{I}:=\sum_{\sigma\in\Sigma_{r}}x^{i_{1}}_{\sigma(1)}\cdots
x^{i_{r}}_{\sigma(r)}\in{\sf S}_{2|I|}$

are dual to the $h_{I}$. The cohomology of the classifying space for stable
complex vector bundles (see II 1.2) can be identified with ${\sf S}_{*}$, with
the Chern class $c_{k}\in H^{2k}(B{\rm U},{\mathbb{Z}})$ corresponding to
$e_{k}$.

The operation which sends a stable vector bundle to its dual sends $e_{k}$ to
$(-1)^{k}e_{k}$, while the Whitney sum inverse $V\mapsto-V$ interchanges
$e_{k}$ and $(-)^{k}h_{k}$, defining a kind of adjoint involution
$\omega:V\mapsto-V^{*}$. Hall’s duality $h_{I}\leftrightarrow m_{I}$ is a Hopf
algebra involution on ${\sf S}_{*}$.

§2. Formal diffeomorphisms of the line at 0

2.1 When $A$ is a commutative algebra, the set

$({\rm{sp}\;}{\bf S})(A)=\\{t(T)=\sum_{k\geq 0}t_{k}T^{k+1}\in
A[[T]]\>|\>t_{k}\in A\\}$

is a noncommutative monoid under composition
$t^{\prime},t\mapsto(t^{\prime}\circ t)(T)=t^{\prime}(t(T))$ of formal power
series. If $t_{0}$ is invertible, an elementary induction shows that $t$ has a
unique two-sided compositional inverse $t^{-1}$ with $(t^{-1}\circ
t)(T)=(t\circ t^{-1})(T)=T$. It follows that the localized polynomial algebra

${\bf S}_{*}:=t_{0}^{-1}{\mathbb{Z}}_{k\geq 0}[t_{k}]:={\mathbb{Z}}[t_{0}^{\pm
1}]\otimes\tilde{{\bf S}}_{*}$

has a noncocommutative coproduct which can be expressed as

$(\Delta_{\bf S}t)(T)=(t\otimes 1)((1\otimes t)(T))\in({\bf S}_{*}\otimes{\bf
S}_{*})[[T]]\;;$

the composition inverse defines an antipode $\chi_{\bf S}$ on ${\bf S}_{*}$
given explicitly by Lagrange’s reversion formula, defining a commutative but
noncocommutative Hopf algebra. Enlarging $\tilde{{\bf S}}_{*}$ to ${\bf
S}_{*}$ encodes the grading as a coaction of the Hopf algebra of the
multiplicative groupscheme (with coproduct $\Delta t_{0}=t_{0}\otimes_{0}$).
This can be suppressed by taking $t_{0}=1$.

The noncommutative dual algebra ${\bf S}^{*}$ appears in algebraic topology as
the analog of the Steenrod algebra for complex cobordism, where it is known as
the Landweber-Novikov algebra. Quillen’s theorem interprets ${\bf S}_{*}$ as
the Hopf algebra representing the group of coordinate transformations of one-
dimensional formal group laws.

The noncommutative groupscheme ${\rm{sp}\;}{\bf
S}=\mathbb{G}_{m}\times{\rm{sp}\;}\tilde{{\bf S}}$ (i.e. of formal
diffeomorphisms at the origin of the line over ${\rm{sp}\;}{\mathbb{Z}}$) is
closely related to the combinatorialists’ Faa di Bruno algebra: or, more
precisely, to the analogous group of formal diffeomorphisms of the line over
${\mathbb{Q}}$, but now parametrized by Taylor-MacLaurin coordinates
$t^{(n)}(0)=n!t_{n-1}$.

2.2 There is a right action

${\rm{sp}\;}{\sf S}\times_{{\rm{sp}\;}{\mathbb{Z}}}{\rm{sp}\;}{\bf
S}\to{\rm{sp}\;}{\sf S}$

of the noncommutative groupscheme ${\rm{sp}\;}{\bf S}$ on the commutative
groupscheme ${\rm{sp}\;}{\sf S}$, defined by composition

$e(T),t(T)\mapsto(e\circ t)(T)$

of power series, i.e. by $\psi_{\sf S}(e(T))=(e\otimes 1)(t(1\otimes T))$,
representing a coaction

$\psi_{\sf S}:{\sf S}_{*}\to{\sf S}_{*}\otimes{\bf S}_{*}$

making ${\sf S}_{*}$ into a Hopf ${\bf S}_{*}$ \- algebra comodule. This
structure can be interpreted as defining a ‘right unit’ $\eta_{R}=\psi_{\sf
S}$ and a ‘left unit’ $\eta_{L}={\bf 1}_{\sf S}\otimes 1$ for a (split) Hopf
algebroid [31]

${\sf S}_{\bullet}{\bf S}:=({\sf S}_{*},{\sf S}_{*}\otimes{\bf S}_{*}):{\sf
S}_{*}\Rightarrow{\sf S}_{*}\otimes{\bf S}_{*}$

with coproduct

$({\sf S}_{*}\otimes{\bf S}_{*})\to{\sf S}_{*}\otimes({\bf S}_{*}\otimes{\bf
S}_{*})\cong({\sf S}_{*}\otimes{\bf S}_{*})\otimes_{{\sf S}_{*}}({\sf
S}_{*}\otimes{\bf S}_{*})\;.$

These morphisms extend to define a cosimplicial (Amitsur) algebra

${\mathcal{A}}^{\bullet}_{{\sf S}_{*}}({\sf S}_{*}\otimes{\bf S}_{*}):0\to{\sf
S}_{*}\Rightarrow{\sf S}_{*}\otimes{\bf S}_{*}\Rrightarrow{\sf
S}_{*}\otimes({\bf S}_{*}\otimes{\bf S}_{*})\dots\;,$

a cobar construction or codescent complex [16](§5)

${\mathcal{A}}^{\bullet}_{C}(B):0\to C\Rightarrow B\Rrightarrow
B\otimes_{C}B\dots$

such that
$D\otimes_{C}{\mathcal{A}}^{\bullet}_{C}(B)\cong{\mathcal{A}}^{\bullet}_{D}(D\otimes_{C}B)$,
e.g.

${\mathcal{A}}^{\bullet}_{{\sf S}_{*}}({\sf S}_{*}\otimes{\bf S}_{*})\cong{\sf
S}_{*}\otimes{\mathcal{A}}^{\bullet}_{\mathbb{Z}}({\bf S}_{*})\;.$

Such constructions underly (for example) the Adams-Novikov spectral sequences
of homotopy theory. In the dual language of schemes

${\rm{sp}\;}{\mathcal{A}}^{\bullet}_{{\sf S}_{*}}({\sf S}_{*}\otimes{\bf
S}_{*})\simeq[{\rm{sp}\;}{\sf S}{/\\!\\!/}{\rm{sp}\;}{\bf S}]$

represents a kind of pre-stack: the (untopologized) groupoid-valued functor on
commutative algebras defined by the action of coordinate transformations on
the symmetric functions.

§3. Noncommutative and quasisymmetric functions

3.1 The free associative graded cocommutative (but noncommutative) binomial
Hopf algebra

${\sf N}_{*}={\mathbb{Z}}_{k\geq 1}\langle Z_{k}\rangle={\mathbb{Z}}\langle
Z_{*}\rangle$

of noncommutative symmetric functions222As in 2.1, the grading on ${\sf
N}_{*}$ can be encoded by adjoining a central unit $Z_{0}$ of degree zero
(i.e. with coproduct $\Delta Z(T)=Z(T)\otimes Z(T),\;Z(T)=\sum_{k\geq
0}Z_{k}T^{k+1}$) is dual to a commutative but not cocommutative algebra

${\sf Q}_{*}\subset{\mathbb{Z}}[x_{*}]$

of quasisymmetric functions: an ordered partition

${\bf I}=i_{1}+\cdots+i_{r}$

of the integer $|I|$ into $r$ nonempty parts (there are $2^{|I|-1}$ such
things) defines a basis element [1],[15](§4)

$m_{\bf I}=\sum_{0<n_{1}<\cdots<n_{r}}\prod_{1\leq j\leq
r}x_{n_{j}}^{i_{j}}\in{\sf Q}_{2|I|}\subset{\mathbb{Z}}[x_{*}]$

dual to the basis element $Z_{\bf I}=Z_{i_{1}}\cdots Z_{i_{r}}\in{\sf
N}_{2|I|}$. We have

$\chi_{\sf N}(Z_{n})=\sum_{|I|=n}(-1)^{r(I)}Z_{\bf I}$

for the antipode of ${\sf N}$. According to Ditters’ conjecture, ${\sf Q}_{*}$
is a free commutative algebra; over ${\mathbb{Q}}$ it is polynomial, generated
by elements of degree $2n$ indexed by Lyndon words of degree $n$, with
multiplicative structure defined by a certain shuffle product on the basis
$\\{m_{\bf I}\\}$. The quasisymmetric functions have interesting relations
with Koszul duality [24](§3) and number theory [6](§4.4), but these will not
be pursued here. The surjection

${\sf N}_{*}\ni Z_{k}\mapsto e_{k}\in{\sf S}_{*}$

of Hopf algebras defined by abelianization dualizes to a monomorphism ${\sf
S}_{*}\to{\sf Q}_{*}$.

3.2 Although composition of power series over noncommutative rings is not
associative in general [2], there is a remarkable generalization of the Hopf
algebra of formal diffeomorphisms of the line in the noncommutative context.
Brouder, Frabetti, and Krattenthaler [3](Thm 2.4) define a Hopf algebra ${\bf
N}_{*}$ with coproduct

$(\Delta_{\bf N}Z)(T)={\rm
res}_{U=0}\;Z(U)\otimes_{\mathbb{Z}}(U-Z(T))^{-1}\in({\bf
N}\otimes_{\mathbb{Z}}{\bf N})_{*}[[T]]$

on generators of the underlying algebra ${\sf N}_{*}$ via a formal version

${\rm res}_{U=0}\;(\prod_{i\in{\mathbb{Z}}}a_{i}U^{i}):=a_{-1}$

of Cauchy’s residue333Here and throughout, variables such as $T,U,\dots$ will
always be central, of cohomological degree two. which elegantly simplifies
proof of coassociativity: expanding the right-hand side, we have

${\rm
res}_{U=0}\;U^{-1}Z(U)\otimes_{\mathbb{Z}}(1-U^{-1}Z(T))^{-1}=\sum_{k\geq
0}{\rm res}_{U=0}\;U^{-k-1}Z(U)\otimes Z(T)^{k}$ $=\sum_{k\geq
1}Z_{k-1}\otimes_{\mathbb{Z}}Z(T)^{k}=(Z\otimes 1)((1\otimes Z)(T))\;,$

i.e.

$\Delta_{\bf N}Z_{k}=\sum_{|I|+j=k}Z_{j}\otimes Z_{\bf I}\;.$

Their Theorem 2.14 provides an explicit formula for the antipode $\chi_{\bf
N}$ of the resulting Hopf algebra. Similarly, their §3 shows that the
coproduct, regarded as an algebra homomorphism

$\Delta_{\bf N}:{\sf N}_{*}\to{\sf N}_{*}\otimes{\bf N}_{*}\;,$

is compatible with the binomial coalgebra structure on ${\sf N}_{*}$, making
it a Hopf algebra comodule over ${\bf N}_{*}$ [18](III §7). As in §2.2, this
can be reformulated as the assertion that

${\sf N}_{\bullet}{\bf N}:=({\sf N}_{*},{\sf N}_{*}\otimes{\bf N}_{*}):{\sf
N}_{*}\Rightarrow{\sf N}_{*}\otimes{\bf N}_{*}$

is a (noncommutative, split) Hopf algebroid over ${\mathbb{Z}}$, with ${\sf
S}_{\bullet}{\bf S}$ as its abelianization.

§4 A digression, on renormalization Hopf algebras

4.0 I owe Michiel Hazewinkel thanks for drawing attention to the Hopf algebra
${\bf N}_{*}$ and its applications in the formalization of quite classical
quantum electrodynamics. It is a part of a wider literature (largely over
${\mathbb{Q}}$) expressed in terms of combinatorial Hopf algebras of trees,
graphs, posets, and related structures, and to place it appropriately in that
enormous field is not practical here; but its many possible generalizations
deserve at least some mention. This section collects a few key ideas from work
of Brouder and Schmitt on a general class of ‘renormalization bialgebras’
$T(T({\bf B})^{+})$ constructed functorially as cotensor algebras on a graded
bialgebra ${\bf B}$. When ${\bf B}={\mathbb{Z}}$ is the trivial bialgebra,

$T({\bf B})^{+}=\\{\otimes^{n}e\>|\>n\geq 1\\}$

is the augmentation ideal of the (co)tensor algebra $T({\bf
B})={\mathbb{Z}}[e]$, identifying $Z_{n}$ with $(\otimes^{n}e)\in
T^{2n}(T({\bf B}))$.

4.1 A certain partially ordered monoid $C$ of ‘compositions’ – ordered
partitions as in §3.1, but with $\emptyset$ allowed as identity element –
plays a basic role in [4]: concatenation, e.g.

$(1+1)*(2+3)=(1+1+2+3)$

is its multiplication operation. The iterated cotensor algebra $T(T({\bf
B})^{+})$ is shown to be naturally $C$-graded, and Brouder and Schmitt
associate to $\rho,\sigma\in C$, certain restriction and contraction coalgebra
morphisms

$T^{\sigma}(T({\bf B})^{+})\to T^{\sigma|\rho}(T({\bf B})^{+})\;,$
$T^{\sigma}(T({\bf B})^{+})\to T^{\sigma/\rho}(T({\bf B})^{+})$

of a sort familiar (for example) in the study of Hopf algebras of trees and
graphs. On generators $u\in T^{\tau}({\bf B}))$ with

$\Delta u=\sum u^{\prime}_{i}\otimes u^{\prime\prime}_{i}\;,$

they define a new, ‘renormalization’ coproduct

${\boldsymbol{\Delta}}u=\sum_{\sigma\leq\tau}u^{\prime}_{i}|\sigma\otimes
u^{\prime\prime}_{i}/\sigma\;,$

which then extends to define a new bialgebra structure [3](§2.3 Th 1) on
$T(T({\bf B})^{+})$.

4.2 Constructions of this sort are ubiquitous in work on combinatorial Hopf
algebras, but vary in details, which are omitted here. If $\varepsilon$ is the
counit of ${\bf B}$, then elements of the form $(x)-\varepsilon(x)$ generate a
two-sided ideal $(J)\subset T(T({\bf B})^{+})$, such that the quotient
bialgebra $T(T({\bf B})^{+})/(J)$ is a Hopf algebra [3](§5.1). This ideal is
trivial for ${\bf B}={\mathbb{Z}}$, and so recovers (in different notation)
the construction of ${\bf N}_{*}$.

§II Some topology

§1 The Hurewicz homomorphism

[This section reviews the theory of characteristic classes and numbers for
complex cobordism in the language of §I. Its point is that the noncommutative
cobordism spectrum $M\xi$ is not yet well-understood, but the associated
theory of characteristic numbers is surprisingly accessible.]

1.1 A topological group $G$, regarded as a topological category and hence as a
simplicial space, has a canonical classifying space $BG$ for $G$-bundles as
its topological realization. For example, the circle group ${\mathbb{T}}={\rm
U}(1)$ has B${\mathbb{T}}\simeq{\mathbb{C}}P^{\infty}$, with polynomial
cohomology

$H^{*}({\mathbb{C}}P^{\infty},{\mathbb{Z}})\cong{\mathbb{Z}}[c],\;|c|=2$

generated by the Chern class of a complex line bundle. ${\mathbb{T}}$ being
commutative, the product map ${\mathbb{T}}\times{\mathbb{T}}\to{\mathbb{T}}$
is a group homomorphism, making $H^{*}(B{\mathbb{T}},{\mathbb{Z}})$ a
primitively generated Hopf algebra. Its dual

$H_{*}(B{\mathbb{T}},{\mathbb{Z}})={\mathbb{Z}}_{k\geq 1}[b_{(k)}]$

is a divided power algebra, with generators satisfying

$b_{(n)}\cdot b_{(m)}=(n,m)\>b_{(n+m)}\;.$

A theorem of IM James identifies the stable homotopy type of the loopspace
$\Omega\Sigma X$ (of the reduced suspension of a simply-connected space) with
that of the free topological monoid

$\Omega\Sigma X\sim\bigvee_{k\geq 1}X^{\wedge k}$

generated by the pointed space $X$. [The free commutative topological monoid
on $X$ is the infinite symmetric product ${\rm SP}^{\infty}X$ of Thom and
Dold.] For example, a level one projective representation of the loop group
$L{\rm SU}(2))$ defines a morphism

$\Omega{\rm SU}(2)\cong\Omega\Sigma S^{2}\to{\mathbb{C}}P^{\infty}$

of loop spaces, with an induced morphism

$H_{*}(\Omega
S^{3},{\mathbb{Z}})={\mathbb{Z}}[\beta]\to{\mathbb{Z}}[b_{(*)}]=H_{*}({\mathbb{C}}P^{\infty},{\mathbb{Z}})$

of Hopf algebras sending $\beta\in\pi_{2}(\Omega S^{3})$ to $b_{(1)}$. [This
notation is meant to suggest Boltzmann’s thermodynamic beta; see 1.3.2 below.]

Similarly, the homology of ${\mathbb{C}}P^{\infty}$ is torsion-free, so the
Künneth theorem implies that

$H_{*}(\Omega\Sigma{\mathbb{C}}P^{\infty},{\mathbb{Z}})\cong{\mathbb{Z}}\langle
Z_{*}\rangle\cong{\sf N}_{*}$

is isomorphic to the cofree cotensor coalgebra on
$\tilde{H}_{*}({\mathbb{C}}P^{\infty},{\mathbb{Z}})$, with generator $b_{(k)}$
corresponding to $Z_{k}$, defining a basis $Z_{\bf I}$ as in I 3.1 and [1].

1.2 Block diagonal or Whitney sum composition

${\rm U}(n)\times{\rm U}(m)\to{\rm U}(n+m)$

defines a homotopy commutative- and -associative law on the monoid

$\coprod_{n\geq 0}B{\rm U}(n)\;,$

whose group completion

$\Omega B(\coprod_{n\geq 0}B{\rm U}(n))\simeq{\mathbb{Z}}\times B{\rm
U}\supset 0\times B{\rm U}$

has a classifying space for stable complex vector bundles as identity
component. The maximal toruses ${\mathbb{T}}^{n}\subset{\rm U}(n)$ then define
Borel’s Hopf algebra isomorphiam

$H^{*}(B{\rm U},{\mathbb{Z}})\cong{\sf S}_{*}\;.$

[5], while the map $B{\rm U}(1)\to B{\rm U}$ defines a polynomial basis

$H_{2k}(B{\rm U}(1),{\mathbb{Z}})\ni b_{(k)}\mapsto b_{k}\in H_{2k}(B{\rm
U},{\mathbb{Z}})$

with $b(T)=\sum_{i\geq 0}b_{k}T^{k+1}$ satisfying $\Delta b(T)=b(T)\otimes
b(T)$. The associated primitives $\check{p}_{k}$ (as in I 1.2) then satisfy
$c_{i}\cdot\check{p}_{k}=\delta_{i,k}$.

The universal complex vector bundle $\xi_{n}\to B{\rm U}(n)$ defines the Thom
spectrum

$M{\rm U}:=\\{S^{n}\wedge\xi^{+}_{m}=S^{n}\wedge M{\rm U}(n)\to M{\rm
U}(n+m)=\xi^{+}_{n+m}\\}$

representing the complex cobordism ring

$M{\rm U}_{*}=\pi_{*}M{\rm U}=\lim_{k\to\infty}\pi_{*+k}M{\rm U}(k)$

of compact complex-oriented manifolds. Hurewicz’s ring homomorphism

$\pi_{*}M{\rm U}\to H_{*}(M{\rm U},{\mathbb{Z}})\cong H_{*}(B{\rm
U},{\mathbb{Z}})\cong{\sf S}_{*}$

takes the homotopy groups of the composition

$M{\rm U}\simeq M{\rm U}\wedge S^{0}\to M{\rm U}\wedge H{\mathbb{Z}}$

of morphisms of ring spectra defined by the identity map

$[S^{0}\to H{\mathbb{Z}}]=1\in\tilde{H}^{0}(S^{0},{\mathbb{Z}})\;.$

(where $S^{0}$ is the sphere spectrum and $H{\mathbb{Z}}$ is the integral
Eilenberg - Mac Lane spectrum). This extends to a natural characteristic
number homomorphism

$h_{M{\rm U}}:M{\rm U}^{*}(X)\to H^{*}(X,{\sf S}_{*})$

of multiplicative cohomology functors, familiar (over ${\mathbb{R}}$) from
Chern-Weil theory.

1.3.1 A cobordism class $\\{X\\}\in M{\rm U}_{2k}$ can be interpreted as the
equivalence class of a compact smooth submanifold
$X\subset{\mathbb{R}}^{2k+N},\;N\gg 0$, with complex normal bundle classified
by

$\nu_{X}:X\to B{\rm U}(N)\to B{\rm U}\;.$

The Thom - Pontryagin collapse construction

$S^{*}\to{\mathbb{R}}^{*}/({\mathbb{R}}^{*}-\nu(X))=\nu(X)^{+}\to\xi^{+}_{*}$

defines a map

$\textstyle{H_{*}(S^{*})\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\nu_{X*}}$$\textstyle{H_{*}(\xi^{+}_{*})\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{H_{*}(M{\rm
U})\cong H_{*}(B{\rm U})\cong{\sf S}_{*}}$

sending the generator of $H_{*}(S^{*})$ to the Hurewicz image of $\\{X\\}$. A
dual construction sends $\\{X\\}$ to the system of integrals of the tangential
Chern classes $c_{I}(\tau_{X})$ with dim $X=2|I|$, regarded as a linear
functional on ${\sf S}_{*}$. Hall duality equates this with the linear
functional

$h_{M{\rm U}}\\{X\\}:m_{I}\mapsto(-1)^{|I|}m_{I}(\nu_{X})\cap[X]\in{\sf
S}_{*}$

defined by the monomial characteristic classes of the normal bundle.

1.3.2 Because $M{\rm U}_{*}\to H_{*}(M{\rm U},{\mathbb{Z}})$ is a homomorphism
of torsion-free rings, it is convenient to work with their rationalizations.
Quillen’s theorem identifies the completed Hopf algebra $M{\rm
U}^{*}{\mathbb{C}}P^{\infty}$ with Lazard’s universal one-dimensional formal
group law, and over ${\mathbb{Q}}$ such group laws are classified by their
logarithms. Miščenko’s theorem

$\log_{M{\rm U}}(T)=\sum_{k\geq 1}\\{{\mathbb{C}}P_{k-1}\\}\frac{T^{k}}{k}\in
M{\rm U}^{*}_{\mathbb{Q}}[[T]]$

identifies this logarithm; on the other hand

${h_{M{\rm U}}}:M{\rm U}^{*}({\mathbb{C}}P^{\infty})\to
H^{*}({\mathbb{C}}P^{\infty},H_{*}(B{\rm U},{\mathbb{Z}}))\cong
H^{*}({\mathbb{C}}P^{\infty},{\sf S}_{*})$

expresses the formal group law in terms of the exponential

${h_{M{\rm U}}}(X+_{M{\rm U}}Y)=b(b^{-1}(X)+b^{-1}(Y))\in{\sf S}_{*}[[X,Y]]$

defined $b(T)$ as above. Note that

$h_{M{\rm U}}\\{{\mathbb{C}}P_{n}\\}=(n+1)\>\chi_{\sf S}(b_{n})\;,$

see [18](28.5.012).

In [32] Ravenel and Wilson show that the canonical inclusions

$\beta_{k}=\\{{\mathbb{C}}P_{k}\subset{\mathbb{C}}P^{\infty}\\}\in M{\rm
U}_{2k}({\mathbb{C}}P^{\infty})$

generate the complex bordism of ${\mathbb{C}}P^{\infty}$, modulo the relation

$\beta(X+_{M{\rm U}}Y)=\beta(X)\cdot\beta(Y)$

(with $\beta(T)=\sum_{k\geq 0}\beta_{k}T^{k+1}$). Working rationally, it
follows that

$\log(\beta\circ b)(T)=b_{(1)}T$

and hence (recalling that $b_{(1)}$ is the image of $\beta$ as in 1.4)

$\beta(T)=\exp(\beta\log_{M{\rm U}}(T))\in({\sf
S}^{\mathbb{Q}}_{*}[\beta])[[T]]\;.$

As suggested by Friedrich and McKay [11](Prop 4.2) this resembles formally the
canonical partition function in statistical mechanics; see further in §III.

1.4 As in I 2.1, the moduli object $[{\rm{sp}\;}M{\rm
U}_{*}{/\\!\\!/}{\rm{sp}\;}S_{*}]$ for one-dimensional group laws can thus be
identified with the stack defined by the now-classical Hopf algebroid

$M{\rm U}_{\bullet}M{\rm U}:M{\rm U}_{*}\Rightarrow M{\rm U}_{*}M{\rm U}\;;$

the Hurewicz or characteristic number map

$h_{M{\rm U}}:[{\rm{sp}\;}H_{*}M{\rm U}{/\\!\\!/}{\rm{sp}\;}{\bf
S}_{*}]\subset[{\rm{sp}\;}M{\rm U}_{*}{/\\!\\!/}{\rm{sp}\;}{\bf S}_{*}]$

then becomes the inclusion of the stratum of formal group laws of additive
type. It pulls back to an isomorphism over
${\rm{sp}\;}{\mathbb{Q}}\subset{\rm{sp}\;}{\mathbb{Z}}$.

§2 The Baker-Richter spectrum $M\xi$

2.1 Following 1.2, group completion of the monoidal map

$\coprod_{n\geq 0}B{\rm U}(1)^{\times n}\to\coprod_{n\geq 0}B{\rm U}(n)$

defines a morphism ${\rm BR}:\Omega\Sigma B{\rm U}(1)\to{\mathbb{Z}}\times
B{\rm U}$ of $A_{\infty}$ spaces. Pulling back the stable universal bundle
$\xi$ over $0\times B{\rm U}$ defines Baker and Richter’s $A_{\infty}$
spectrum

$M\xi:=\\{S^{n}\wedge{\rm BR}^{*}\xi^{+}_{m}\to{\rm BR}^{*}\xi^{+}_{n+m}\\}$

together with an abelianization morphism $M\xi\to M{\rm U}$. A remarkable
theorem [1](§7) shows that localization at a prime splits $M\xi$ as a wedge of
suspensions of the Brown-Peterson spectrum, even though $M\xi$ is not an
$M{\rm U}$-module spectrum; but it is complex-orientable, and possesses
natural Thom isomorphisms.

We will however make no use of this local structure here; our focus is the
ring-spectrum $M\xi\wedge H{\mathbb{Z}}$, i.e. on the integral homology and
cohomology

$H_{*}(M\xi,{\mathbb{Z}})\cong{\sf N}_{*},\;H^{*}(M\xi,{\mathbb{Z}})\cong{\sf
Q}_{*}$

and their relation to $M\xi$ through its Hurewicz homomorphism. Baker and
Richter show that

$M\xi_{*}=\pi_{*}M\xi\to H_{*}(M\xi,{\mathbb{Z}})$

is injective, and an isomorphism after rationalization. Following 1.3 and
[19], we can interpret the Hurewicz map as a noncommutative characteristic
number homomorphism

${h_{M\xi}}:M\xi^{*}(Y)\to H^{*}(Y,H_{*}(M\xi,{\mathbb{Z}}))\cong H^{*}(Y,{\sf
N}_{*})\cong{\rm Hom}({\sf Q}_{*},H^{*}(Y,{\mathbb{Z}}))\;.$

Under Quillen’s conventions [30] we can assume that $Y$ is a smooth manifold,
and interpret an element $\\{X\\}\in M\xi^{2i}(Y)$ as the cobordism class of a
map $[\nu:X\to Y]$ between manifolds, with stable normal bundle

$v_{X}\cong\oplus L_{i}:X\to\bigvee_{n\geq 0}B{\rm U}(1)_{+}^{\times}$

presented as a direct sum of stable complex line bundles. Then
${h_{M\xi}}\\{X\\}$ is represented by the linear functional

$m_{\bf I}\mapsto(-1)^{i}\nu_{!}(m_{\bf I}(c(L_{*})))\in
H^{2i}(Y,{\mathbb{Z}})$

on ${\sf Q}_{2i}$ (where $\nu_{!}$ is the covariant pushforward cohomology
homomorphism induced by the complex-oriented map $\nu$).

Remark The right unit

$\eta_{R}:{\bf N}_{*}\to{\sf N}_{*}\otimes{\sf N}_{*}\cong{\rm Hom}({\sf
Q}_{*},{\sf N}_{*})$

of I §2.3 can be defined on generators by

$\eta_{R}(Z_{k})(m_{\bf I})=\sum_{i+j=k}m_{\bf I}(Z_{i})\cdot Z_{j}\;,$

agreeing with the coproduct formula in I 3.2.

2.2 The Euler class $x_{\xi}\in M\xi^{2}{\mathbb{C}}P^{\infty}$ provides a
(noncentral [2]) coordinate for kind of noncommutative one-dimensional formal
group structure on $M\xi^{*}{\mathbb{C}}P^{\infty}$. Its Hurewicz image

$h_{M\xi}(x_{\xi})=\sum_{k\geq 0}Z_{k}T^{k+1}=Z(T)\in
H_{*}(M\xi,{\mathbb{Z}})[[T]]\subset{\sf N}_{*}[[T]]$

defines a (completed) Hopf coproduct

${\boldsymbol{\Delta}}_{{\mathbb{C}}P^{\infty}}(T)=\sum_{k\geq
0}Z_{k}\cdot(\chi_{\bf N}Z)(T\otimes 1)+(\chi_{\bf N}Z)(1\otimes
T))^{k+1}\in{\sf N}_{*}[[T\otimes 1,1\otimes T]]\;,$

cf. Zassenhaus’s noncommutative binomial theorem.

The Thom isomorphism $M\xi_{*}M\xi\cong
M\xi_{*}\Omega\Sigma{\mathbb{C}}P^{\infty}$ (together with a Künneth argument
for torsion-free spaces) then defines a cosimplicial noncommutative algebra

${\mathcal{A}}^{\bullet}_{M\xi_{*}}(M\xi_{*}M\xi):0\to M\xi_{*}\Rightarrow
M\xi_{*}M\xi\Rrightarrow M\xi_{*}M\xi\otimes_{M\xi_{*}}M\xi_{*}M\xi\dots$

as in I 1.2. In view of the remark above, the Hurewicz morphism

$M\xi\wedge M\xi\to(M\xi\wedge H{\mathbb{Z}})\wedge_{H{\mathbb{Z}}}(M\xi\wedge
H{\mathbb{Z}})\simeq M\xi\wedge M\xi\wedge H{\mathbb{Z}}$

induces an injective homomorphism

${\mathcal{A}}^{\bullet}_{M\xi_{*}}(M\xi_{*}M\xi)\to{\mathcal{A}}^{\bullet}_{H_{*}M\xi}(H_{*}M\xi\otimes
H_{*}M\xi)\cong{\mathcal{A}}^{\bullet}_{{\sf N}_{*}}({\sf N}_{*}\otimes{\bf
N}_{*})\;,$

of cosimplicial algebras defined by homotopy groups: dually, a morphism

$[{\rm{sp}\;}{\sf N}{/\\!\\!/}{\rm{sp}\;}{\bf
N}]\to{\rm{sp}\;}{\mathcal{A}}^{\bullet}_{M\xi_{*}}(M\xi_{*}M\xi)$

of pre-stacks, which becomes an isomorphism over ${\mathbb{Q}}$. Thinking of
commutative objects as a subclass of noncommutative ones defines an
abelianization morphism

$[{\rm{sp}\;}{\sf S}{/\\!\\!/}{\rm{sp}\;}{\bf S}]\to[{\rm{sp}\;}{\sf
N}{/\\!\\!/}{\rm{sp}\;}{\bf N}]\;.$

It is tempting to imagine the descent object
${\mathcal{A}}^{\bullet}_{M\xi}(M\xi\wedge M\xi)$ as an approximation to a
noncommutative analog of the sphere spectrum.

§III Examples and remarks

1 A $2n$-dimensional prequantized toric manifold [9, 12, 14, 27] is an
omnioriented complex-oriented manifold (i.e. with a preferred decomposition of
its stable tangent bundle as a sum of complex lines) defined by an effective
action of a torus ${\mathbb{T}}^{n}$, together with an equivariant complex
line bundle $L$ with connection, such that the curvature form
$\omega(\nabla_{L})$ is symplectic; it is roughly analogous to a compactified
system of harmonic oscillators.

Such a manifold defines an element of $M\xi_{2n}{\mathbb{C}}P^{\infty}$ and
thus has characteristic numbers in $H_{*}({\mathbb{C}}P^{\infty},{\sf
N}_{*})\cong{\sf N}_{*}[b_{(*)}]$. Similarly toric varieties, considered as
orbifolds [19, 20], have characteristic numbers in ${\sf
N}_{*}\otimes{\mathbb{Q}}$, while Hamiltonian toric varieties have similar
invariants in $M\xi_{*}{\mathbb{C}}P^{\infty}\otimes{\mathbb{R}}\cong
M\xi_{*}\otimes{\mathbb{R}}[\beta]$. Projective toric varieties [13] provide
examples; in particular, chemical reaction networks [8] with $k$ nodes and
deficiency zero [27](§1.3.2) have characteristic invariant $\sum_{{\bf
I}=k}Z_{\bf I}$.

2 The equivariant cohomology of the moment-angle complex associated to a
simplicial (e.g. Gorenstein [27](§1.2.2)) complex can be identified with its
Stanley-Reisner face ring, which thus has (quasi-symmetric) characteristic
classes and numbers. This defines a homomorphism [19] from the
(noncommutative) ring of simplicial spheres, with join as composition, to
${\bf N}_{*}$; the boundary of the $k$-simplex yields the example above.

3 Work in the theory of free, i.e. noncommutative, probability [28], [17](Th
3.1),[29] suggests that

$(\chi_{\bf N}Z)(-T)\in{\bf N}_{*}[[T]]$

generalizes the cumulant generating function [10, 22] of classical statistical
mechanics, along the lines suggested in II 1.3.2 above.

## References

  * [1] A Baker, B Richter, Quasisymmetric functions from a topological point of view, Math. Scand. 103 (2008) 208 -– 242, https://arxiv.org/abs/math/0605743
  * [2] —–, —–, Some properties of the Thom spectrum over loop suspension of complex projective space, in An alpine expedition through algebraic topology 1 -– 12, Contemp. Math. 617, AMS 2014, https://arxiv.org/abs/1207.4947
  * [3] C Brouder, A Frabetti, C Krattenthaler, Non-commutative Hopf algebra of formal diffeomorphisms, Adv. Math. 200 (2006) 479 –- 524, https://arxiv.org/abs/math/0406117
  * [4] —–, W Schmitt, Renormalization as a functor on bialgebras, J Pure Appl. Algebra 209 (2007) 477 -– 495, https://arxiv.org/abs/hep-th/0210097
  * [5] H Cartan, Démonstration homologique des théorémès de périodicité de Bott, II, Séminaire Henri Cartan, tome 12 (1959 – 1960) exp.17, http://www.numdam.org/volume/SHC_1959-1960__12/
  * [6] P Cartier, A primer of Hopf algebras, in Frontiers in number theory, physics, and geometry II 537 -– 615, Springer (2007), http://preprints.ihes.fr/2006/M/M-06-40.pdf
  * [7] P Conner, E Floyd, The relation of cobordism to $K$-theories, Springer LNM 28 (1966)
  * [8] G Craciun, A Dickenstein, A Shiu, B Sturmfels, Toric dynamical systems, J. Symbolic Comput. 44 (2009) 1551 –- 1565, https://arxiv.org/abs/0708.3431
  * [9] M Davis, T Januszkiewicz, Convex polytopes, Coxeter orbifolds and torus actions, Duke Math. J. 62 (1991) 417 –- 451
  * [10] R Friedrich, J McKay, Free probability and complex cobordism, C R Math Acad Sci So R Can 33 (2011) 116 –- 122
  * [11] ——, —–, Formal groups, Witt vectors, and free probability, https://arxiv.org/abs/1204.6522
  * [12] VL Ginzburg, Calculation of contact and symplectic cobordism groups, Topology 31 (1992) 767 -– 773
  * [13] V Guillemin, Kaehler structure on toric varieties, J Diff Geo 40 (1994) 285 - 309
  * [14] V Guillemin, VL Ginzburg, Y Karshon, Moment maps, cobordisms, and Hamiltonian group actions, AMS Mathematical Surveys and Monographs 98 (2002)
  * [15] M Hazewinkel, Symmetric functions, noncommutative symmetric functions, and quasisymmetric functions, Acta Appl. Math. 75 (2003) 55 – 83, https://arxiv.org/abs/0410468
  * [16] K Hess, A general framework for homotopic descent and codescent, https://arxiv.org/abs/1001.1556
  * [17] M Josuat-Vergés, F Menous, JC Novelli, JY Thibon, Free cumulants, Schroder trees, and operads, Adv in App Math 88 (2017) 92 – 119, https://arxiv.org/abs/1604.04759
  * [18] C Kassel, Quantum groups, Springer Graduate Texts 155 (1995)1995.
  * [19] N Kitchloo, J Morava, The Baker-Richter spectrum as cobordism of quasitoric manifolds, https://arxiv.org/abs/1201.3127
  * [20] E Lerman, S Tolman, Hamiltonian torus actions on symplectic orbifolds and toric varieties, Trans. AMS 349 (1997) 4201 -– 4230, https://arxiv.org/abs/dg-ga/9511008
  * [21] I Macdonald, Symmetric functions and Hall polynomials, 2nd ed, Oxford Classic Texts in the Physical Sciences (2008)
  * [22] M Marcolli, Motivic information, https://arxiv.org/abs/1712.08703
  * [23] A Mathew, Climbing Mount Bourbaki, https://amathew.wordpress.com/2012/05/28/
  * [24] J Morava, Homotopy-theoretically enriched categories of noncommutative motives, Res. Math. Sci. 2 (2015), Art. 8, 16 pp., https://arxiv.org/abs/1402.3693
  * [25] —–, On formal groups and geometric quantization, https://arxiv.org/abs/1905.06181
  * [27] —–, Topological invariants of some chemical reaction networks, https://arxiv.org/abs/1910.12609
  * [28] A Nica, R Speicher, Lectures on the combinatorics of free probability, LMS Lecture Note Series 335, Cambridge (2006)
  * [29] JC Novelli, JY Thibon, Hopf algebras and dendriform structures arising from parking functions, Fund. Math. 193 (2007) 189 – 241, https://arxiv.org/abs/math/0511200
  * [30] D Quillen, Elementary proofs of some results of cobordism theory using Steenrod operations, Adv. in Math. 7 (1971) 29 -– 56
  * [31] D Ravenel, Complex cobordism and stable homotopy groups of spheres, Pure and Applied Mathematics 121, Academic Press (1986)
  * [32] —–, W S Wilson, The Hopf ring for complex cobordism, J Pure Appl Algebra 9 (1976/77) 241 -– 280


# Count-Based Temperature Scheduling for Maximum Entropy Reinforcement
Learning

Dailin Hu  Pieter Abbeel  Department of Electrical Engineering and Computer
Science, University of California, Berkeley  Roy Fox  Department of Computer
Science, University of California, Irvine

###### Abstract

Maximum Entropy Reinforcement Learning (MaxEnt RL) algorithms such as Soft
Q-Learning (SQL) and Soft Actor–Critic trade off reward and policy entropy,
which has the potential to improve training stability and robustness. Most
MaxEnt RL methods, however, use a constant tradeoff coefficient (temperature),
contrary to the intuition that the temperature should be high early in
training to avoid overfitting to noisy value estimates and decrease later in
training as we increasingly trust high value estimates to truly lead to good
rewards. Moreover, our confidence in value estimates is state-dependent,
increasing every time we use more evidence to update an estimate. In this
paper, we present a simple state-based temperature scheduling approach, and
instantiate it for SQL as Count-Based Soft Q-Learning (CBSQL). We evaluate our
approach on a toy domain as well as in several Atari 2600 domains and show
promising results.

##  1 Introduction

Deep Reinforcement Learning (RL) methods that use neural-network function
approximators to learn control policies have shown great performance in
domains including games (Mnih et al., 2015; Silver et al., 2017), robotic
control (Haarnoja et al., 2017), and autonomous driving (Sallab et al., 2017).
The training of such a function approximator, however, is often very sensitive
to hyperparameter tuning in different environments (Henderson et al., 2018).
Recent work in Maximum Entropy Reinforcement Learning (MaxEnt RL) (Fox et al.,
2016; Haarnoja et al., 2018) trades off maximizing the policy values with its
entropy, producing policies that are more robust to disturbances in the
environment dynamics and reward function (Haarnoja et al., 2017; Eysenbach &
Levine, 2021).

MaxEnt RL follows from the maximum entropy principle (Jaynes, 2003), which
states that one should prefer maximally uninformed solutions, subject to the
available evidence. MaxEnt RL therefore maximizes average policy entropy,
subject to an attainable level of policy value. The Lagrangian of this
constrained optimization problem trades off the traditional RL objective of
maximum policy value with the expected policy entropy. The Lagrange multiplier
is an _inverse-temperature_ parameter $\beta$ that determines the relative
importance of the value and entropy terms, and corresponds to a level of value
that we believe we can attain. Intuitively, in early stages of training,
$\beta$ should be lower to introduce higher stochasticity as we have low
confidence that the current value estimates are attainable. During training,
we become increasingly more confident in our value estimates, and we increase
$\beta$ such that the policy stochasticity decreases and eventually approaches
zero. As $\beta\to\infty$, the conventional RL learning objective is
recovered.

Most MaxEnt RL algorithms, such as Soft Q-Learning (SQL) (Haarnoja et al.,
2017), use a constant temperature throughout training. The log-partition
function in SQL’s soft Bellman backup operator allows it to put higher weight
on the policy entropy when the temperature is high, or approximate reward
maximization arbitrarily well as the temperature decreases to 0. Selecting a
constant inverse-temperature $\beta$ may not reflect the changing confidence
of the value estimates throughout learning. Even when there exists a constant
$\beta$ that works well, it is domain-dependent, which makes it hard to tune
as a hyperparameter. Soft Actor–Critic (SAC) (Haarnoja et al., 2018) adjusts
the temperature $\beta^{-1}$ automatically with stochastic gradient descent,
but often has poor performance in discrete action spaces (Christodoulou,
2019). Fox et al. (2016) proposes a linear inverse-temperature schedule, in
which $\beta_{i}=\kappa i$ in training step $i$. This corresponds to the
intuition that $\beta$ should increase throughout learning, but the
coefficient $\kappa$ remains a domain-dependent hyperparameter to be tuned.

Importantly, our confidence in SQL’s value estimates is very much state-
dependent. Value estimates that have been updated more times, from more data,
are more reliable. States that are more often experienced by early exploration
will have their value estimates improve in accuracy, and their $\beta$ should
increase. Later in learning, some of these states may not be encountered as
often, becoming less important to train, and their $\beta$ should not increase
as fast. When exploration reaches novel states, whose value estimates are less
reliable, their $\beta$ should start very low again, even late in training.

In this paper, we describe a simple state-dependent temperature scheduling
method for MaxEnt RL based on a pseudo-count of state value updates derived
from a CTS density model (Ostrovski et al., 2017). As a concrete instance of
this method, adapted to the SQL algorithm, we present the Count-Based Soft
Q-Learning (CBSQL) algorithm. Evaluating CBSQL on 6 popular games on the Atari
2600 platform suggests that our scheduling method improves over DQN and SQL
with a fixed temperature, and display the potential to reach state-of-the-art
performance with Rainbow integration (Hessel et al., 2018).

##  2 Preliminaries

We address MaxEnt RL in discrete action spaces. In this section, we will
introduce the notation and framework of MaxEnt RL.

###  2.1 Reinforcement Learning

We consider environments modeled as a Markov Decision Process (MDP). We
describe this process by a tuple $\langle\mathcal{S},\mathcal{A},p,r\rangle$,
where $\mathcal{S}$ represents a state space and $\mathcal{A}$ represents a
discrete action space.
$p(s_{t+1}|s_{t},a_{t}):\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$
represents the probability distribution of the next state $s_{t+1}$ given the
current state $s_{t}$ and action $a_{t}$.
$r_{t}=r(s_{t},a_{t}):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$
describes the reward for each transition $t$. The discounted return $R$, for a
discount factor $0\leq\gamma<1$, is defined as $R=\sum_{t}\gamma^{t}r_{t}$.

A reinforcement learning agent learns a policy $\pi(a_{t}|s_{t})$ for
interacting with its environment. The agent and the environment jointly induce
a distribution $p_{\pi}(\xi)$ over trajectories
$\xi=s_{0},a_{0},s_{1},a_{1},\ldots$. It is also convenient to define a state
distribution
$p^{\gamma}_{\pi}(s)=(1-\gamma)\sum_{t}\gamma^{t}p_{\pi}(s_{t}=s)$, describing
the distribution of $s_{t}$ at a time $t$ distributed geometrically with
parameter $1-\gamma$. This satisfies $\mathbb{E}_{\xi\sim
p_{\pi}}[R(\xi)]=\tfrac{1}{1-\gamma}\mathbb{E}_{s\sim
p^{\gamma}_{\pi}}[\mathbb{E}_{(a|s)\sim\pi}[r(s,a)]]$.

###  2.2 Maximum Entropy Reinforcement Learning

The MaxEnt RL objective augments the standard reinforcement learning objective
of maximizing expected discounted rewards by adding an entropy term

$\pi^{*}=\arg\max_{\pi}\mathbb{E}_{s\sim
p^{\gamma}_{\pi}}\left[\mathbb{E}_{(a|s)\sim\pi}[r(s,a)]+\tfrac{1}{\beta}\mathbb{H}[\pi(\cdot|s)]\right],$
(1)

where $\beta$ is an _inverse-temperature_ parameter that controls the
stochasticity of the optimal policy by determining the relative importance
between the reward and policy entropy $\mathbb{H}[\pi]$. In early stages of
training, $\beta$ should intuitively be assigned a small value to induce
stochasticity, and in later stages of training $\beta\rightarrow\infty$ to
approach the deterministic behavior that optimizes the standard reinforcement
learning objective.

###  2.3 Soft Q-Learning (SQL) and the Mellowmax Operator

RL methods often involve the state–action value function
$Q(s,a)=\mathbb{E}[R|s_{0}=s,a_{0}=a]$ and the Bellman operator
$\mathcal{B}[Q](s,a)=r(s,a)+\gamma\mathbb{E}_{(s^{\prime}|s,a)\sim
p}[\max_{a^{\prime}}Q(s^{\prime},a^{\prime})]$. The MaxEnt RL objective
suggests a _soft_ Bellman operator (Rubin et al., 2012)

$\displaystyle\mathcal{B}[Q](s,a)$
$\displaystyle=r(s,a)+\gamma\mathbb{E}_{(s^{\prime}|s,a)\sim
p}\left[\max_{\pi}\left(\mathbb{E}_{(a^{\prime}|s^{\prime})\sim\pi}[Q(s^{\prime},a^{\prime})]+\tfrac{1}{\beta}\mathbb{H}[\pi(\cdot|s^{\prime})]\right)\right]$
(2) $\displaystyle=r(s,a)+\gamma\mathbb{E}_{(s^{\prime}|s,a)\sim
p}\left[\tfrac{1}{\beta}\log\sum_{a^{\prime}\in\mathcal{A}}\exp(\beta
Q(s^{\prime},a^{\prime}))\right]\qquad\forall
s\in\mathcal{S},a\in\mathcal{A},$ (3)

where the optimizer in (2) is the softmax policy $\pi(a|s)\propto\exp(\beta
Q(s,a))$, and the log-sum-exp expression in (3) is the log-partition function,
also called the mellowmax operator (Asadi & Littman, 2017). Mellowmax is non-
decreasing in $\beta$ (Kim et al., 2019) and a non-expansion for a fixed
$\beta$ under the supremum norm (Fox et al., 2016), and as $\beta\to\infty$ it
converges to pure maximization.

Soft Q-Learning (Fox et al., 2016; Haarnoja et al., 2017) uses a model-free
empirical estimate of the soft Bellman operator as the target for learning a Q
function. When the Q function has a tabular representation, the contraction
property of the soft Bellman operator guarantees convergence to the operator’s
fixed point, the softmax of which is the optimal policy in (1) (Fox et al.,
2016).

##  3 Count-Based Temperature Scheduling

###  3.1 Motivation

Empirical evidence from SQL suggests that a state-independent linear
temperature scheduling, i.e. using $\beta_{i}=\kappa i$ in iteration $i$, can
achieve good performance (Fox et al., 2016; Grau-Moya et al., 2018). Fox
(2019) proposes a closed-form state-dependent expression for the temperature
that completely eliminates bias in entropy-regularized value updates in two-
action environments. If the gap between the action values $Q(s,a=0)-Q(s,a=1)$
has Gaussian uncertainty with mean $\mu_{A}$ and variance $\sigma^{2}_{A}$,
the prescribed inverse-temperature is
$\beta(s)=\frac{2\mu_{A}}{\sigma^{2}_{A}}$. The Gaussian assumption is
motivated by the central limit theorem, which also suggests that the
asymptotic behavior of $\beta$ is to grow linearly in the number of data
points used to estimate the value gap at each state.

We propose Count-Based Soft Q-Learning (CBSQL), an algorithm based on SQL that
uses a state-dependent temperature schedule in which $\beta(s)$ grows linearly
with the number of times that the algorithm updates $Q(s,a)$, for any action
$a$. Formally, let $n(s,a)$ be the count of sampled data points of the form
$(s,a,r,s^{\prime})$ used thus far in value updates. Then the inverse-
temperature in CBSQL is $\beta(s)=\kappa\sum_{a}n(s,a)$, with $\kappa>0$ a
constant hyperparameter.

In addition to the aforementioned empirical and theoretical evidence
supporting linear count-based scheduling, more insight can be gained by
comparing two families of successful RL algorithms. The first family,
consisting of such algorithms as G-Learning (Fox et al., 2016), SQL (Haarnoja
et al., 2017), Path Consistency Learning (PCL) (Nachum et al., 2017a), and
Soft Actor–Critic (SAC) (Haarnoja et al., 2018), aims to learn a policy that
is the softmax of a value function $Q(s,a)$ with a low temperature. In
iteration $i$, the policy target is

$\pi_{i}(a|s)\propto\pi_{0}(a|s)\exp\beta_{i}(s)Q_{i}(s,a)\qquad\forall
s\in\mathcal{S},a\in\mathcal{A},$ (4)

with $\pi_{0}$ the uniform policy, and the policy is either updated toward the
target (4), or set equal to it. Note that in their original formulations, most
of these algorithms use a constant state-independent $\beta$. The second
family of algorithms, consisting of Relative Entropy Policy Search (REPS)
(Peters et al., 2010), $\Psi$-learning (Rawlik et al., 2010), Trust Region
Policy Optimization (TRPO) (Schulman et al., 2015), and Trust-PCL (Nachum et
al., 2017b), aims to use value estimates to gradually update the softmax
policy. Instead of the policy entropy, which is the Kullback–Leibler (KL)
divergence from a uniform prior policy, these algorithms consider a KL term
with the current policy as the prior for the update. Moreover, instead of a
low temperature $\beta^{-1}$, these algorithms place a large coefficient
$\kappa^{-1}$ on the KL term, to induce small updates in a “trust region”. In
iteration $i$, the policy update is therefore

$\pi_{i}(a|s)\propto\pi_{i-1}(a|s)\exp\kappa_{i}(s)Q_{i}(s,a)\propto\pi_{0}(a|s)\exp\left(\sum_{j\leq
i}\kappa_{j}(s)Q_{j}(s,a)\right).$ (5)

The two types of learning processes (4) and (5) can have different properties,
because $Q_{i}$ can change significantly between iterations. For the sake of
our intuitive argument here, imagine that $Q$ could be approximated by a
constant, and that the inverse-temperatures $\kappa_{i}(s)$ used in trust-
region algorithms were also a constant $\kappa$. Then combining the above two
equations, we have

$\beta_{i}(s)\approx\sum_{j\leq i}\kappa_{j}(s)\approx\kappa i,$ (6)

where $\kappa$ is a small constant, and $i$ is the number of times that the
update equation (5) is applied in state $s$. In practice, updates are not
applied to all actions in state $s$ at the same time, leading to the heuristic
definition $i=\sum_{a}n(s,a)$.

###  3.2 Pseudo-counts

Directly recording the number of times that the value update has been applied
to state $s$ is not useful in practical settings, where the state space is
large, and most states will rarely be visited more than once. Moreover, a
tabular mapping from $s$ to $i$ would fail to capture the similarity between
different states that a Q function approximator does leverage, and would
vastly underestimate the effective number of times that $Q(s,a)$ has been
updated in states similar to $s$.

Instead, we use a pseudo-count method derived by Bellemare et al. (2016) from
a simplified pixel-level CTS density model (Bellemare et al., 2014). Define
$\rho$ to be the CTS density model on the state space $\mathcal{S}$. Let
$\rho_{k}(s)$ be the probability assigned by the model to $s\in\mathcal{S}$
after $k$ updates of the model. Let $\rho^{\prime}_{k}(s)$ be the probability
that the model would assign to $s$ if it were updated on $s$ one more time.
The pseudo-count can then be defined as

$n_{k}(s)=\frac{\rho_{k}(s)(1-\rho^{\prime}_{k}(s))}{\rho^{\prime}_{k}(s)-\rho_{k}(s)}.$
(7)

Note that $k$ is different from $i$ in the last section and represents the
total number of updates in all states. The effective number of times that
state $s$ has been updated is its pseudo-count $n_{k}(s)$, suggesting the
state-dependent inverse-temperature

$\beta(s)=\kappa\cdot n_{k}(s),$ (8)

where $\kappa$ is a small constant. Throughout our experiments, we set
$\kappa=0.01$.

In summary, we propose a model-free reinforcement learning algorithm that, on
experience $(s,a,r,s^{\prime})$, uses the SQL update rule

$Q(s,a)\leftarrow
r+\gamma\operatorname*{mellowmax}_{a^{\prime};\beta(s^{\prime})}Q(s^{\prime},a^{\prime})=r+\tfrac{\gamma}{\beta(s^{\prime})}\log\tfrac{1}{|\mathcal{A}|}\sum_{a^{\prime}\in\mathcal{A}}\exp(\beta(s^{\prime})Q(s^{\prime},a^{\prime})).$
(9)

In tabular experiments, we also update

$n(s)\leftarrow n(s)+1,$ (10)

whereas in deep-learning experiments, we update the density model with state
$s$, and consider the pseudo-count (7). The inverse-temperature is then
scheduled as $\beta(s)=\kappa\cdot n(s)$. We present the pseudocode in Alg 1.

Algorithm 1 Count-Based Soft Q Learning (SQL)

Initialize Q network parameters $\theta$

Initialize target Q network $\overline{\theta}\leftarrow\theta$

Initialize an empty replay buffer $\mathcal{D}\leftarrow\emptyset$

Initialize a density model $\rho$

for each iteration do

for each step $t$ in the rollout do

In state $s_{t}$, sample action $a_{t}$ from the $\epsilon$-greedy policy for
$Q_{\theta}(s_{t},\cdot)$

Execute action $a_{t}$ and observe reward $r_{t}$ and new state $s_{t+1}$

Store the transition $(s_{t},a_{t},r_{t},s_{t+1})$ into the replay buffer
$\mathcal{D}$

end for

for each gradient step do

Sample random batch $(s,a,r,s^{\prime})$ from $\mathcal{D}$

$\beta\leftarrow\kappa\cdot\frac{\rho(s^{\prime})(1-\rho^{\prime}(s^{\prime}))}{\rho^{\prime}(s^{\prime})-\rho(s^{\prime})}$

$y=r+\gamma\operatorname*{mellowmax}_{a^{\prime};\beta}Q_{\bar{\theta}}(s^{\prime},a^{\prime})$

Perform gradient descent on $(y-Q_{\theta}(s,a))^{2}$

Update the density model with state $s$

Every target_freq steps, update $\bar{\theta}\leftarrow\theta$

end for

end for

##  4 Experiments

In this section, we evaluate our approach in tabular representation on a toy
domain, and in deep learning on six popular Atari 2600 environments.

###  4.1 Tabular Representation

We compare our approach to DQN and SQL on a toy domain of noisy transitions in
a chain-linked graph. This environment is small enough to directly find the
optimal policy.

We define a noisy chain-walk problem with five states, each state with two
possible actions (Figure 1(a)). The agent always starts at state 0 and each
episode ends after 5 steps. The agent receives a reward of $+1$ when it takes
action 1 at state 4, and a reward of $-0.1$ when it takes any action at any
other states. The reward that agent receives is always corrupted with an
additive Gaussian noise of $\mathcal{N}(0,1)$.

The optimal expected reward is $0.6$. Figure 1(b) shows the rewards averaged
over 1000 runs comparing Q-learning, SQL with different constant inverse-
temperature parameters $\beta\in\\{10,100,1000\\}$, and CBSQL with a tabular
representation. These results suggest that CBSQL can converge significantly
faster and obtain higher rewards than both SQL and Q learning within 300
episodes of training in this simple domain.

(a) Domain Demonstration

(b) Rewards averaged over 10000 runs

Figure 1: Noisy Chain-Walk Problem

###  4.2 Neural Network Representation

We evaluate our method on six Atari 2600 game from the arcade learning
environment (Bellemare et al., 2013) and compare it with DQN and SQL under the
same hyperparameter setting from Mnih et al. (2015). We train the agents on 3M
frames and record the rewards. CBSQL appears to outperform DQN and fixed-
temperature SQL in 4 of the 6 games (Table 1, Figure 2).

Game | DQN | SQL($\beta=100$) | SQL($\beta=1000$) | CBSQL
---|---|---|---|---
Breakout | 5.9 ($\pm$5.9) | 5.9 ($\pm$4.5) | 5.1 ($\pm$4.7) |  8.2 ($\pm$6.1)
Freeway | 21.0 ($\pm$1.5) | 14.6 ($\pm$8.5) | 22.56 ($\pm$4.7) |  25.82 ($\pm$4.9)
Pong | 1.93 ($\pm$2.6) |  17.83 ($\pm$2.2) | 16.31 ($\pm$2.7) | 17.56 ($\pm$2.0)
Q*bert | 568.4 ($\pm$1101.9) | 828 ($\pm$1411.7) | 564.5 ($\pm$1097.5) |  875.3 ($\pm$1254.6)
Seaquest | 13.5 ($\pm$24.1) | 4 ($\pm$60.2) | 17.2 ($\pm$24.0) |  84.6 ($\pm$60.2)
Space Invaders | 132.7 ($\pm$113.2) |  158.9 ($\pm$128.5) | 132.25 ($\pm$118.4) | 138.9 ($\pm$112.8)
Table 1: DQN, fixed-temperature SQL, and CBSQL average rewards ($\pm$ standard
deviation). Raw scores are averaged over the last 100 testing episodes across
3 runs. Figure 2: CBSQL results compared with DQN and fixed-temperature SQL,
without Rainbow. Rewards are averaged over 3 runs.

DQN by itself is a powerful algorithm, but in recent years more extensions to
it have been proposed that greatly improve its performance. Interestingly, SQL
and CBSQL can be combined with many of these extensions, allowing us to
compare these methods after integrating popular DQN extensions. We integrate
CBSQL with Rainbow DQN (Hessel et al., 2018), a state-of-the-art reinforcement
learning algorithm for memoryless agents, which includes multi-step targets
(Sutton & Barto, 2018), double Q-learning (Hasselt, 2010), prioritized
experience replay (Schaul et al., 2015), dueling networks (Wang et al., 2016),
distributional RL (Bellemare et al., 2017), and noisy networks (Fortunato et
al., 2017). All of these methods can be straightforwardly applied to soft
Q-learning, with the exception of multi-step targets and distributional RL,
which we discuss next.

Multi-step learning. Multi-step targets with a well-tuned number of steps $n$
can lead to faster learning in on-policy RL algorithms (Sutton & Barto, 2018)
by trading off the bias and variance of the return estimates (Kearns & Singh,
2000). The n-step truncated return at time $t$ is
$r^{(n)}_{t}=\sum_{k=0}^{n-1}\gamma^{k}r(s_{t+k},a_{t+k})$. Hessel et al.
(2018) defined a multi-step variant of DQN by minimizing the alternative loss
$(r_{t}^{(n)}+\gamma^{n}\max_{a\in\mathcal{A}}Q_{\bar{\theta}}(s_{t+n},a)-Q_{\theta}(s_{t},a_{t}))^{2}$
and demonstrated through empirical experiments that n-step targets can
outperform single-step targets in DQN, despite the off-policy experience
providing biased estimates of the n-step return. In SQL with inverse-
temperature $\beta$, the n-step truncated return is

$\tilde{r}^{(n)}_{t}=r^{(n)}_{t}+\tfrac{1}{\beta}\sum_{k=1}^{n-1}\gamma^{k}\mathbb{H}[\pi(\cdot|s_{t+k})].$
(11)

Unfortunately, empirical policy entropy estimates are often very noisy. It is
also unclear which $\beta$ should be used in off-policy estimates of (11).
These considerations call for further study, and in this work we simply use
1-step returns for SQL and CBSQL.

Distributional RL. Unlike conventional RL which estimates the expected return,
distributional RL estimates the distribution of the return at time $t$ over
the stochasticity of the environment and the policy. The estimator uses a
fixed $N$-dimensional vector $\bm{z}$ of values spaced evenly along the range
$[v_{\min},v_{\max}]$ of possible returns. The distribution of
$Q(s_{t},a_{t})$ is then represented by a categorical distribution
$\bm{p}_{\theta}(s_{t},a_{t})$ over the values of $\bm{z}$. Distributional DQN
updates $\bm{p}_{\theta}(s,a)$ by minimizing its KL-divergence from a
projected target distribution induced by the categorical distribution
$\bm{p}_{\bar{\theta}}(s^{\prime},a^{*})$ over the values $r+\gamma\bm{z}$.
Here the action $a^{*}$ is chosen greedily with
$a^{*}=\arg\max_{a^{\prime}}\bm{z}^{\intercal}\bm{p}_{\bar{\theta}}(s^{\prime},a^{\prime})$.
We adapt distributional RL to soft Q-learning and CBSQL by defining a policy
distribution of

$\pi(a^{\prime}|s^{\prime})=\frac{\exp\beta(s^{\prime})\bm{z}^{\intercal}\bm{p}_{\bar{\theta}}(s^{\prime},a^{\prime})}{\sum_{\bar{a}^{\prime}}\exp\beta(s^{\prime})\bm{z}^{\intercal}\bm{p}_{\bar{\theta}}(s^{\prime},\bar{a}^{\prime})}$

over the values
$r+\gamma\left(\bm{z}-\tfrac{1}{\beta}\mathbb{D}[\pi(\cdot|s^{\prime})\|\pi_{0}]\right)$.

We train the Rainbow-variations of SQL and CBSQL as described above over 500K
frames each for the atari game Breakout and Q*bert against Rainbow-DQN(See Fig
3). Rainbow-CBSQL out-performs Rainbow DQN in both these environments.

Figure 3: CBSQL results compared with DQN and fixed-temperature SQL, with Rainbow. Rewards are averaged over 5 runs. Game | DQN | SQL($\beta=100$) | SQL($\beta=1000$) | CBSQL
---|---|---|---|---
Breakout | 10.2 ($\pm$ 4.9) | 5.5 ($\pm$ 3.6) | 3.6 ($\pm$ 2.1) |  39.9 ($\pm$ 15.4)
Q*bert | 356.0 ($\pm$ 202.8 ) | 30.0 ($\pm$ 60.0) | 1.0 ($\pm$ 4.9) |  370.0 ($\pm$ 43.0)
Table 2: CBSQL results compared with DQN and SQL with fixed temperatures,
combined with Rainbow. Raw scores averaged over last 50 testing episodes
across 5 runs.

##  5 Conclusion

In this paper, we presented a simple method for temperature scheduling in soft
Q learning which could be potentially applied to other maximum entropy
reinforcement learning algorithms. We showed through empirical experiments
that our method can outperform DQN and SQL with or without the rainbow
framework.

## References

  * Asadi & Littman (2017) Kavosh Asadi and Michael L Littman.  An alternative softmax operator for reinforcement learning.  In _International Conference on Machine Learning_ , pp. 243–252. PMLR, 2017.
  * Bellemare et al. (2014) Marc Bellemare, Joel Veness, and Erik Talvitie.  Skip context tree switching.  In _International conference on machine learning_ , pp. 1458–1466. PMLR, 2014.
  * Bellemare et al. (2013) Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ , 47:253–279, 2013.
  * Bellemare et al. (2016) Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos.  Unifying count-based exploration and intrinsic motivation.  _CoRR_ , abs/1606.01868, 2016.  URL http://arxiv.org/abs/1606.01868.
  * Bellemare et al. (2017) Marc G. Bellemare, Will Dabney, and Rémi Munos.  A distributional perspective on reinforcement learning.  _CoRR_ , abs/1707.06887, 2017.  URL http://arxiv.org/abs/1707.06887.
  * Christodoulou (2019) Petros Christodoulou.  Soft actor-critic for discrete action settings.  _arXiv preprint arXiv:1910.07207_ , 2019.
  * Eysenbach & Levine (2021) Benjamin Eysenbach and Sergey Levine.  Maximum entropy rl (provably) solves some robust rl problems.  _arXiv preprint arXiv:2103.06257_ , 2021.
  * Fortunato et al. (2017) Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al.  Noisy networks for exploration.  _arXiv preprint arXiv:1706.10295_ , 2017.
  * Fox (2019) Roy Fox.  Toward provably unbiased temporal-difference value estimation.  In _Optimization Foundations for Reinforcement Learning Workshop at NeurIPS_ , 2019.
  * Fox et al. (2016) Roy Fox, Ari Pakman, and Naftali Tishby.  Taming the noise in reinforcement learning via soft updates.  _arXiv preprint arXiv:1512.08562_ , 2016.
  * Grau-Moya et al. (2018) Jordi Grau-Moya, Felix Leibfried, and Peter Vrancx.  Soft q-learning with mutual-information regularization.  In _International conference on learning representations_ , 2018.
  * Haarnoja et al. (2017) Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine.  Reinforcement learning with deep energy-based policies.  In _International Conference on Machine Learning_ , pp. 1352–1361. PMLR, 2017.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al.  Soft actor-critic algorithms and applications.  _arXiv preprint arXiv:1812.05905_ , 2018.
  * Hasselt (2010) Hado Hasselt.  Double q-learning.  _Advances in neural information processing systems_ , 23:2613–2621, 2010.
  * Henderson et al. (2018) Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger.  Deep reinforcement learning that matters.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32, 2018.
  * Hessel et al. (2018) Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.  Rainbow: Combining improvements in deep reinforcement learning.  In _Thirty-second AAAI conference on artificial intelligence_ , 2018\.
  * Jaynes (2003) Edwin T Jaynes.  _Probability theory: The logic of science_.  Cambridge university press, 2003.
  * Kearns & Singh (2000) Michael J Kearns and Satinder P Singh.  Bias-variance error bounds for temporal difference updates.  In _COLT_ , pp. 142–147. Citeseer, 2000.
  * Kim et al. (2019) Seungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris.  Deepmellow: removing the need for a target network in deep q-learning.  In _Proceedings of the Twenty Eighth International Joint Conference on Artificial Intelligence_ , 2019.
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  _nature_ , 518(7540):529–533, 2015.
  * Nachum et al. (2017a) Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans.  Bridging the gap between value and policy based reinforcement learning.  _arXiv preprint arXiv:1702.08892_ , 2017a.
  * Nachum et al. (2017b) Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans.  Trust-pcl: An off-policy trust region method for continuous control.  _arXiv preprint arXiv:1707.01891_ , 2017b.
  * Ostrovski et al. (2017) Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos.  Count-based exploration with neural density models.  In _International conference on machine learning_ , pp. 2721–2730. PMLR, 2017.
  * Peters et al. (2010) Jan Peters, Katharina Mulling, and Yasemin Altun.  Relative entropy policy search.  In _Twenty-Fourth AAAI Conference on Artificial Intelligence_ , 2010\.
  * Rawlik et al. (2010) Konrad Rawlik, Marc Toussaint, and Sethu Vijayakumar.  Approximate inference and stochastic optimal control.  _arXiv preprint arXiv:1009.3958_ , 2010.
  * Rubin et al. (2012) Jonathan Rubin, Ohad Shamir, and Naftali Tishby.  Trading value and information in mdps.  In _Decision Making with Imperfect Decision Makers_ , pp. 57–74. Springer, 2012.
  * Sallab et al. (2017) Ahmad EL Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani.  Deep reinforcement learning framework for autonomous driving.  _Electronic Imaging_ , 2017(19):70–76, 2017.
  * Schaul et al. (2015) Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized experience replay.  _arXiv preprint arXiv:1511.05952_ , 2015.
  * Schulman et al. (2015) John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz.  Trust region policy optimization.  In _International conference on machine learning_ , pp. 1889–1897. PMLR, 2015.
  * Silver et al. (2017) David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.  Mastering the game of go without human knowledge.  _nature_ , 550(7676):354–359, 2017.
  * Sutton & Barto (2018) Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Wang et al. (2016) Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas.  Dueling network architectures for deep reinforcement learning.  In _International conference on machine learning_ , pp. 1995–2003. PMLR, 2016.

##  Appendix A Hyperparameters

Hyper-parameter | value
---|---
Discount factor $\gamma$ | 0.99
Exploration $\epsilon$ | 0.01
Learning rate | 1
Table 3: Hyper-parameters for tabular experiments. Hyper-parameter | value
---|---
Grey-scaling | True
Observation down-sampling | (84, 84)
Frames stacked | 4
Reward clipping | [-1, 1]
Discount factor $\gamma$ | 0.99
Initial exploration | 1
Final exploration | 0.1
Final exploration frame | 1000000
Learning rate | 0.00025
Replay buffer size | 1000000
Minibatch size | 32
Q network channels | [32, 64, 64]
Q network filter size | $8\times 8,4\times 4,3\times 3$
Q network stride | 4, 2, 1
Q network hidden units | 512
Table 4: Hyper-parameters for DQN, SQL and CBSQL on Atari 2600. The values of all the hyper-parameters are based on the work from Mnih et al. (2015). Hyper-parameter | value
---|---
Grey-scaling | True
Observation down-sampling | (84, 84)
Frames stacked | 4
Reward clipping | [-1, 1]
Discount factor $\gamma$ | 0.99
Learning rate | 0.0000625
Replay buffer size | 1000000
Minibatch size | 32
Q network channels | [32, 32, 64]
Q network filter size | $8\times 8,4\times 4,3\times 3$
Q network stride | 4, 2, 1
Q network hidden units | 512
Noisy net $\sigma_{0}$ | 0.5
Multi-step returns $n$ | 4 for DQN, 1 for SQL and CBSQL
Distributional atoms | 51
Distributional min/max values | [-10, 10]
Table 5: Hyper-parameters for DQN, SQL and CBSQL with Rainbow Integration on
Atari 2600. The values of all the hyper-parameters are based on the work from
Hessel et al. (2018).