# Accumulating Evidence for the Associate Production of
a Neutral Scalar with Mass around 151 GeV

Andreas Crivellin<EMAIL_ADDRESS>Physik-Institut, Universität
Zürich, Winterthurerstrasse 190, CH–8057 Zürich, Switzerland  Paul Scherrer
Institut, CH–5232 Villigen PSI, Switzerland  CERN Theory Division, CH–1211
Geneva 23, Switzerland  Yaquan Fang<EMAIL_ADDRESS>Institute of High
Energy Physics, 19B, Yuquan Road, Shijing District, Beijing, China, 100049
University of Chinese Academy of Sciences (CAS), 19A Yuquan Road, Shijing
District, Beijing, China, 100049  Oliver Fischer
<EMAIL_ADDRESS>Department of Mathematical Sciences, University
of Liverpool, Liverpool, L69 7ZL, UK  Abhaya Kumar<EMAIL_ADDRESS>School of Physics and Institute for Collider Particle Physics, University of
the Witwatersrand, Johannesburg, Wits 2050, South Africa.  Mukesh Kumar
<EMAIL_ADDRESS>School of Physics and Institute for Collider Particle
Physics, University of the Witwatersrand, Johannesburg, Wits 2050, South
Africa.  Elias Malwa<EMAIL_ADDRESS>School of Physics and Institute for
Collider Particle Physics, University of the Witwatersrand, Johannesburg, Wits
2050, South Africa.  Bruce Mellado<EMAIL_ADDRESS>School of Physics
and Institute for Collider Particle Physics, University of the Witwatersrand,
Johannesburg, Wits 2050, South Africa.  iThemba LABS, National Research
Foundation, PO Box 722, Somerset West 7129, South Africa.  Ntsoko Rapheeha
<EMAIL_ADDRESS>School of Physics and Institute for Collider
Particle Physics, University of the Witwatersrand, Johannesburg, Wits 2050,
South Africa.  Xifeng Ruan<EMAIL_ADDRESS>School of Physics and
Institute for Collider Particle Physics, University of the Witwatersrand,
Johannesburg, Wits 2050, South Africa.  Qiyu Sha<EMAIL_ADDRESS>Institute
of High Energy Physics, 19B, Yuquan Road, Shijing District, Beijing, China,
100049  University of Chinese Academy of Sciences (CAS), 19A Yuquan Road,
Shijing District, Beijing, China, 100049

###### Abstract

In recent years, hints for “multi-lepton anomalies” have been accumulated by
the analysis of Large Hadron Collider (LHC) data, pointing towards the
existence of beyond the Standard Model (SM) Higgs bosons: a new scalar
particle $S$ with a mass $m_{S}$ in the range between $130\,$GeV and
$160\,$GeV, produced from the decay of a heavier new scalar particle, $H$.
Motivated by this observation, we perform a search for the signatures of $S$
within this mass region, which has been studied by CMS and ATLAS as a by-
product of the SM Higgs searches in the side-bands of the kinematic regions.
Combining the $\gamma\gamma$ and $Z\gamma$ channels, with associated leptons,
di-jets, bottom quarks and missing energy, we obtain a local (global)
significance of 5.1$\sigma$ (4.8$\sigma$) for a mass of $m_{S}=151.5$ GeV and
provide the preferred ranges for the corresponding (fiducial) cross sections.
This is a strong indication for a scalar resonance $S$ decaying into photons,
and, to a lesser extent to $Z\gamma$, in association with missing energy, jets
or leptons. Hints for the decays into, or production in association with,
bottom quarks are statistically less significant. In order to test this
hypothesis, we propose a search for $H\rightarrow\gamma\gamma
b\overline{b},\tau^{+}\tau^{-}b\overline{b}$ in asymmetric configurations that
has not yet been performed by ATLAS and CMS.

††preprint: CERN-TH-2021-129, ICPP-057, PSI-PR-21-21, ZU-TH 38/21, LTH 1267

##  I Introduction

The discovery of the Brout-Englert-Higgs boson ($h$) Higgs (1964a); Englert
and Brout (1964); Higgs (1964b); Guralnik et al. (1964) at the LHC by ATLAS
Aad et al. (2012) and CMS Chatrchyan et al. (2012) has opened a new chapter in
particle physics. Measurements so far indicate that this $125\,$GeV boson has
properties compatible with those predicted by the SM Chatrchyan et al. (2013);
Aad et al. (2013). However, this does not exclude the existence of additional
scalar bosons as long as their possible mixing with the SM Higgs is
sufficiently small, such that the properties of the latter remain to a good
approximation unchanged.

In fact, in recent years the so-called “multi-lepton anomalies” emerged as
deviations from the SM predictions in several analyses of multi-lepton final
states from ATLAS and CMS, as observed in Refs. von Buddenbrock et al. (2018);
Buddenbrock et al. (2019); von Buddenbrock et al. (2020); Hernandez et al.
(2021). These hints for New Physics (NP) are statistically most compelling in
non-resonant di-leptons (both opposite and same sign) final states as well as
in the three-lepton channel (with and without the presence of $b$-tagged
jets). These signatures can be explained by the decay of a neutral scalar $H$
into a lighter one $S$ and the SM Higgs von Buddenbrock et al. (2016), i.e.
$H\rightarrow Sh,SS$, as realized e.g. within the 2HDM+$S$ model (also called
N2HDM) He et al. (2009); Grzadkowski and Osland (2010); Chen et al. (2014);
Muhlleitner et al. (2017); Krause et al. (2017).111Interestingly, the model
can explain anomalies in astro-physics (the positron excess of AMS-02 Aguilar
et al. (2019) and the excess in gamma-ray fluxes from the galactic center
measured by Fermi-LAT Ackermann et al. (2017)) if it is supplemented by a Dark
Matter candidate Beck et al. (2021). Furthermore, it can be easily extended
Sabatta et al. (2020) to account for the $4.2\sigma$ anomaly $g-2$ of the muon
Abi et al. (2021); Aoyama et al. (2020).

The explanation of the multi-lepton anomalies in the 2HDM+$S$ framework
requires the mass of $S$ to be between $130\,$GeV and $160\,$GeV von
Buddenbrock et al. (2018). Fortunately, this mass range is covered in the side
bands of CMS and ATLAS searches for the SM Higgs. Therefore, the published
analyses of di-photon, $Z\gamma$ and $b\overline{b}$ resonances Sirunyan et
al. (2021); ATL (2020); Aad et al. (2020); Sirunyan et al. (2020); Aad et al.
(2021a); Sirunyan et al. (2018a); Sirunyan et al. (2018b); ATLAS Collaboration
(2020) can be used to search for a signal of $S$. For the search of di-photon
resonances both inclusive distributions, as well as analyses including the
associate production of leptons, ($b$-) jets and missing transverse energy
($E^{T}_{miss}$) are available. Importantly, it is possible to combine the
information from these channels without specifying an explicit model, such
that the only assumption is a scalar particle $S$ with an associate production
mechanism.

In this article we present a combined fit of the different channels, briefly
discuss the implications of the different signals for the properties of $S$,
and suggest a new search for $S$ via the search for asymmetric $\gamma\gamma
b\overline{b},\tau^{+}\tau^{-}b\overline{b}$ signals.

##  II Methodology

As outlined in the introduction, we use CMS and ATLAS analyses of SM Higgs
decays which cover (implicitly) also the search for other Higgs-like
resonances from a scalar $S$ in the side bands. In fact, depending on the
specific channel, searches can cover scalar masses ($m_{S}$) up to 180 GeV.
However, since some searches only reach $160\,$GeV, we will use the region
between 140 GeV (in order to avoid overlap with SM Higgs signals) and 155 GeV
as the use of a side-band fit further reduces the range within one can search
for a resonance. Importantly, this happens to be the mass range motivated by
the multi-lepton anomalies.

Let us summarize the CMS and ATLAS searches used in our analysis:

$S(Z(\ell^{+}\ell^{-})\gamma)+\ell$: $m_{S}$ is reconstructed from the
invariant mass of the $Z\gamma$ pair and $S$ is assumed to be produced in
association with one additional lepton (other than the leptons originating
from the decays of the $Z$). This data is extracted from Fig. 5 in Ref.
Sirunyan et al. (2018b).

$S$($\gamma\gamma$)+$E^{T}_{miss}$: In these channels $m_{S}$ is reconstructed
from the invariant mass of $\gamma\gamma$ and $S$ is produced in association
with $E^{T}_{miss}$. The data is taken from Fig. 6 in Ref. Aad et al. (2021a)
and Fig. 3 in Ref. Sirunyan et al. (2018a).

$S$($b\overline{b}$)+$E^{T}_{miss}$: $m_{S}$ is reconstructed from the
invariant mass of $b\overline{b}$ and $S$ is produced in association with
$E^{T}_{miss}$ originating from the decay of $S$ to invisible final
states.222It should be mentioned that there is a small over-lap between this
category and the SM channel $t\overline{t}h$. However, the pollution of the
former by the latter is very small (about 1%). The data is taken from Fig. 14
(a) in Ref. ATLAS Collaboration (2020).

$S$($\gamma\gamma$)+$b$ jets: $m_{S}$ decays to two photons and is produced in
association with $b$ quarks (which, in the 2HDM+$S$ model, could originate
from $S$ but also from $h$ if $H\to Sh$ is non-negligible). The data is
obtained from Fig. 2 (top-right) in Ref. Aad et al. (2020) and Fig. 2 in Ref.
Sirunyan et al. (2020).

$S$($\gamma\gamma$)+ $V(W,Z)$: $S$ decays to two photons and is produced in
association with a $W$ or a $Z$ boson. The corresponding data is given in Fig.
15 bottom-left in Ref. Sirunyan et al. (2021) and Fig. 9 (c) and (d) in Ref.
ATL (2020).

$S(\gamma\gamma)$ inclusive: Here $m_{S}$ is reconstructed from the invariant
mass of the photon pair while the search is quasi-inclusive, however, vector
boson fusion, $W$ and $Z$ as well as top quark associated production are
excluded. Note that there is no veto on missing energy, but that this channel
covers only a very tiny phase space of the quasi-inclusive final search. (see
Fig. 15 top-left in Ref. Sirunyan et al. (2021) and Fig. 9 (a) in Ref. ATL
(2020)).

For each category discussed above, we model the background via a function:

$f(m;b,\\{a\\})=(1-m)^{b}(m)^{a_{0}+a_{1}\log(m)}\,,$ (1)

where $a_{0,1}$ and $b$ are free parameters (different for each category) and
$m$ is the invariant mass of the distribution, e.g. the di-photon mass. This
corresponds to the background-only hypothesis and the goodness of the
corresponding fit results in the p-value of the SM.

In order to search for a signal in each category, we add to the background the
function in Eq. (1) a double-sided-crystal-ball function:

$N\cdot\begin{cases}e^{-t^{2}/2}&\mbox{if $-\alpha_{\rm Low}\leq
t\leq\alpha_{\rm High}$}\\\ \frac{e^{-0.5\alpha_{\rm
Low}^{2}}}{\left[\frac{\alpha_{\rm Low}}{n_{\rm Low}}\left(\frac{n_{\rm
Low}}{\alpha_{\rm Low}}-\alpha_{\rm Low}-t\right)\right]^{n_{\rm
Low}}}&\mbox{if $t<-\alpha_{\rm Low}$}\\\ \frac{e^{-0.5\alpha_{\rm
High}^{2}}}{\left[\frac{\alpha_{High}}{n_{\rm High}}\left(\frac{n_{\rm
High}}{\alpha_{\rm High}}-\alpha_{\rm High}+t\right)\right]^{n_{\rm
High}}}&\mbox{if $t>\alpha_{\rm High}$}.\end{cases}$ (2)

Here $N$ is a normalization parameter, $t=(m-m_{S})/\sigma_{CB}$ with
$\sigma_{CB}$ the width of the Gaussian part of the function, $m$ is the
invariant mass of the distribution and $m_{S}$ the mass of the new resonance
we are interested in (which is the same for all channels). $\alpha_{\rm
Low}$($\alpha_{\rm High}$) is the point where the Gaussian becomes a power law
on the low (high) mass side and we set $\alpha_{\rm Low}=\alpha_{\rm
High}=1.5$. $n_{\rm Low}$ ($n_{\rm High}$) is the exponent of this power law
and $n_{\rm Low}$ ($n_{\rm High}$) is set to 5 (9). The fit results are quite
insensitive to the specific choice of $\alpha_{\rm Low}$($\alpha_{\rm High}$)
or $n_{\rm Low}$ ($n_{\rm High}$). As we assume that the physical width of $S$
is much smaller than the detector resolution of the respective channels,
$\sigma_{CB}$ is thus determined by the detector resolution:
$\sigma_{CB}=1.5\,$GeV for the di-photon and $Z\gamma$ channel, and
$\sigma_{CB}=14\,$GeV for the $b\overline{b}$ channel.

For any given final state (category), the relative contribution of sub-
categories to the total signal strength is taken to be proportional to the
observed excess in each spectrum. For some of the spectra, this is obtained by
a $N_{S}/(N_{S}+N_{B})$ or ${\rm ln}(1+N_{S}/N_{B})$ re-weighting, where
$N_{S}$ is the number of SM Higgs events in the sub-category in the original
analysis and $N_{B}$ is the number of background events under the Higgs boson
peak. The spectra are then unweighted using a constant factor $N^{h}_{\rm
unweighted}/N^{h}_{\rm weighted}$, where $N^{h}_{\rm(un)weighted}$ is the
number of SM Higgs (125 )GeV signal evens obtained in the CMS or ATLAS
analyses. The simultaneous fit of $m_{S}$ using all channels therefore takes
into account the different integrated luminosity in each analysis. This is
then implemented into the likelihood ratio formalism, relaying on the software
used as well for the discovery of the SM Higgs boson Chatrchyan et al. (2013);
Aad et al. (2013).

Figure 1: Fits to the spectra of the different categories extracted from Refs.
Sirunyan et al. (2021); ATL (2020); Aad et al. (2020); Sirunyan et al. (2020);
Aad et al. (2021a); Sirunyan et al. (2018a); Sirunyan et al. (2018b); ATLAS
Collaboration (2020) (see text for details). The shown number of events are
re-weighted by the ratio of signal over background at $151$ GeV for each
channel.

##  III Analysis and Results

Table 1 shows the extracted total cross section for the quasi-inclusive di-
photon analysis, the limit on the total $4\ell$ cross section (not included in
the signal analysis) and the fiducial cross sections for all other associate
production channels.

We define the signal yield as $Y=\epsilon\cdot\sigma_{S}\cdot L$, where
$\sigma_{S}$ is the total or fiducial cross section, depending on the
analysis, the (channel dependent) luminosity $L$ and the efficiency
$\epsilon$, which parametrizes analysis-specific selection criteria and the
geometric acceptance of the detector. Based on the efficiencies quoted in the
different ATLAS and CMS analyses, we assume that the value of $\epsilon$ is on
average 59% for the di-photon channel and 24% for the $Z\gamma$ channel.
Therefore, $Y$ corresponds to the number of observed signal events in a given
channel.

The spectra of the channels in which the excesses are most significant are
shown in Fig. 1 and the combination of all channels in Fig. 2, excluding the
$b\overline{b}$ channel due to the large resolution. Note that the background
curves here do not exactly correspond to the SM hypothesis but rather to Eq.
(1) in the combined fit to Eq. (1) and Eq. (2). For improved visualization,
each plot combines the spectra from ATLAS and CMS (if available) for the same
final states by using a signal over background ($N_{S}/N_{B}$) re-weighting,
such that $N_{S}$ is the number of signal at the peak ($\pm 3\,$GeV) of the
crystal-ball function and $N_{B}$ are the corresponding background events
within this range.

Combining all spectra (being a function of a common mass $m_{S}$) the local
$p$-value that a background-only hypothesis is true is shown in Fig. 3 as a
function of $m_{S}$. The maximal local significance for a resonance is
5.1$\sigma$ at $m_{S}=151.5\,$GeV, where it varies by 1$\sigma$ when shifting
the mass by $1$ GeV. Taking into account the look-else-where effect due to the
mass scan between 140 GeV to 155 GeV, the significance is reduced to
4.8$\sigma$.333There is also a 2.3$\sigma$ excess (locally) reported by ATLAS
in the search for fully hadronic final states originating from $S\rightarrow
VV$ Aad et al. (2021b). This excess was not included in our fit, since the
spectrum was not given in a form usable for our analysis. We observe that the
di-photon final states, in particular in channels with associated
$E^{T}_{miss}$, contribute most to the excess. Furthermore, while there is a
preference for a decay of $S$ to $Z\gamma$ and $b\overline{b}$ the
significance for the latter is, with $1.2\sigma$, quite small. We remark that
the limits on the total cross section in the four-lepton final states
(extracted from Refs. ATLAS Collaboration (2018); CMS (2019)) show no hint for
$S\to ZZ$.

Figure 2:  The combination of fits to all the categories from Sirunyan et al.
(2021); ATL (2020); Aad et al. (2020); Sirunyan et al. (2020); Aad et al.
(2021a); Sirunyan et al. (2018a); Sirunyan et al. (2018b); ATLAS Collaboration
(2020) (see text). The data points are $N_{S}/N_{B}$ weighted.  Figure 3:  The
combined local $p$-value as a function of $m_{S}$ using inputs from Sirunyan
et al. (2021); ATL (2020); Aad et al. (2020); Sirunyan et al. (2020); Aad et
al. (2021a); Sirunyan et al. (2018a); Sirunyan et al. (2018b); ATLAS
Collaboration (2020) (see text).

The significance reported here is obtained with a signal strength that varies
among different categories and therefore quite independent of the production
mechanism of the diphoton resonance. It is important to realise that an
extraction of the total cross sections from the fiducial ones would be model
dependent, especially in the case of $E^{T}_{miss}$ searches. Nonetheless,
assuming that the signal is generated together with $E^{T}_{miss}$, and that
the latter is within the cuts that were applied in the analysis, the total
cross sections should not differ from the fiducial ones by more than a factor
2. We observe that the signal yields obtained in the different categories are
consistent with di-boson signatures, such as $H\rightarrow SS^{*}$. In this
case, in order to allow for a $E^{T}_{miss}$ of more than 90 GeV, $m_{H}>240$
GeV is required and if $m_{H}$ were above the TeV scale, the efficiency of the
search would be significantly affected (due to overproduced $E^{T}_{miss}$).

Let us therefore interpret the results in Table 1 assuming that $S$ is pair
produced by the decays of $H$ with a mass around 270 GeV (as suggested by the
multi lepton anomalies) via $pp\to H\rightarrow SS^{*}$. We remark that also
the process including $H\rightarrow Sh^{(*)}$ is possible in principle, but we
will ignore it for now. The properties of $S$ can be determined approximately
from the relative signal strengths of the considered channels: It should
clearly decay to photons, and to a lesser extent to $Z\gamma$, and it should
decay into invisible final states in order to produce missing energy. Further
hints for $S$ decaying into, or it being produced in association with, bottom
quarks are less significant statistically, and therefore only optional.

The individual signal strengths of the here considered channels could imply
that the couplings of $S$ to SM articles are induced via the mixing with $h$.
In this case $S$ would ‘inherit’ the branching ratios of a would-be Higgs
boson with a mass of 151 GeV, and one would expect ${\rm Br}[S\to bb]\approx
14\%$, ${\rm Br}[S\to\tau\tau]\approx 0.16\%$, ${\rm
Br}[S\to\gamma\gamma]\approx 1.3\times 10^{-3}$, ${\rm Br}[S\to
Z\gamma]\approx 2.2\times 10^{-3}$, ${\rm Br}[S\to WW]\approx 72\%$ and ${\rm
Br}[S\to ZZ^{*}]\approx 8.1\%$ Andersen et al. (2013), with a possible re-
scaling due to the additional decay of $S$ into some invisible final state.
Taking into account that ${\rm Br}[Z\to\mu^{+}\mu^{-}]={\rm Br}[Z\to
e^{+}e^{-}]\approx 3.36\%$, one can estimate that in this case ${\rm Br}[S\to
4\ell]\approx 0.3{\rm Br}[S\to\gamma\gamma]$ which is in conflict with the
limit on $S\to 4\ell$ and thus disfavours the mixing hypothesis. This suggests
the possibility that the couplings of $S$ to photons and the $Z$ might be
loop-induced, for example by vector-like fermions (see e.g. Franceschini et
al. (2016); Falkowski et al. (2016); Benbrik et al. (2016); Chao et al.
(2017)), one of which could be neutral and give rise to the $E^{T}_{miss}$
signatures. In this case, one would, however, not naturally expect a large
${\rm Br}[S\to b\overline{b}]$.

Channel | Cross sec. [fb] | Obs. lim. | Exp. lim.
---|---|---|---
$S(\gamma\gamma)$ |  4.1$\pm$2.6 | 9.0 | 5.4
$S(\gamma\gamma)$ \+ $E^{T}_{miss}>$90 GeV |  0.42$\pm$0.13 | 0.65 | 0.22
$S(\gamma\gamma)$+$V\rightarrow jj$ |  0.28$\pm$0.28 | 0.80 | 0.57
$S(\gamma\gamma)$ \+ $b$-jets |  0.08$\pm$0.08 | 0.22 | 0.16
$S(\ell\ell\gamma)$ \+ $V\rightarrow\ell\nu$ or $\ell\ell$ |  1.3$\pm$0.7 | 2.8 | 1.7
$S(b\overline{b})$ \+ 150$<E^{T}_{miss}<$250 GeV |  0.90$\pm$0.79 | 2.2 | 1.6
$S(4\ell)$ |  0$\pm$0.15 | 0.28 | 0.33
Table 1:  Extracted cross sections in fb for each final state considered here
(see main text for details). The observed and expected limits on fiducial
cross-section at 95% confidence level are also provided. For the second and
the last category total cross sections are quoted while for the associate
production channels fiducial ones are given. The $S\to 4\ell$ is not included
in the fit but rather given as a constraint.

##  IV Asymmetric $\gamma\gamma b\overline{b}$ searches

In order to verify or falsify the hypothesis of sizable $S\to b\overline{b}$
or $H\to Sh$ rates, one could search for $H\rightarrow
SS\rightarrow\gamma\gamma b\overline{b}$ and $H\rightarrow
Sh\rightarrow\gamma\gamma b\overline{b}$ final states. These are very
promising signatures as they have the highest sensitivity for di-Higgs
searches due to a good balance between the di-photon triggering efficiency,
the triggering of the invariant mass spectra Sirunyan et al. (2019a); Aaboud
et al. (2018).

Let us illustrate this for $H\rightarrow SS\rightarrow\gamma\gamma
b\overline{b}$. Assuming $m_{H}=270\,$GeV, the dominant branching ratio being
$H\to SS^{*}$ forces one of the singlet scalars to be off-shell.444We notice
that in other possible decay channels like $H\to Sh$ and $H\to hh$ the
daughter particles are on-shell, which offers different search strategies in
the final states. This type of resonant $\gamma\gamma b\overline{b}$ searches
have not been performed by the LHC experiments. Here, two corners of the
phase-space are devised to study asymmetric configurations:
$m_{\gamma\gamma}\in(145,155)$ GeV and $m_{b\overline{b}}\in(70,120)$ GeV to
isolate $H\rightarrow S(\rightarrow\gamma\gamma)S^{*}(\rightarrow
b\overline{b})$; $m_{\gamma\gamma}\in(90,120)$ GeV and
$m_{b\overline{b}}\in(120,160)$ GeV to isolate $H\rightarrow S(\rightarrow
b\overline{b})S^{*}(\rightarrow\gamma\gamma)$. In order to predict the
resulting signatures, we perform a Monte-Carlo simulation of $pp$ collisions
at the LHC. The events corresponding to the signal and SM backgrounds are
generated using Madgraph5 Alwall et al. (2014) with the NNPDF3.0 parton
distribution functions Ball et al. (2015). The UFO model files required for
the Madgraph analysis have been obtained from FeynRules Alloul et al. (2014)
after a proper implementation of the model. Following this partonlevel
analysis, the parton showering and hadronization are performed using Pythia
Sjostrand et al. (2006). We use Delphes(v3) de Favereau et al. (2014) for the
corresponding detector level simulation after the showering/hadronization. The
jet construction at this level has been performed using Fastjet Cacciari et
al. (2012) which involves the anti-$K_{T}$ jet algorithm with radius $R=0.5$,
transverse momentum $p_{T}>20$ GeV and pseudorapidity $\left|\eta\right|<2.5$.
Photons are required to have $p_{T}>20$ GeV and $\left|\eta\right|<2.5$

Figure 4: Expected yield of signal and background $\gamma\gamma b\overline{b}$
events for 500 fb-1 of integrated luminosity with one experiment at the LHC.
The upper and and lower graphs correspond to the $H\rightarrow
S(\rightarrow\gamma\gamma)S^{*}(\rightarrow b\overline{b})$ and $H\rightarrow
S(\rightarrow b\overline{b})S^{*}(\rightarrow\gamma\gamma)$ searches,
respectively.

Figure 4 shows the expected signal and background yields in $\gamma\gamma
b\overline{b}$ final state for one LHC experiment and 500 fb-1 of integrated
luminosity. The first configuration displays significantly better signal-to-
background rates due to the excellent di-photon invariant mass resolution.
Here the benchmark signal cross-section $\sigma(pp\rightarrow H\rightarrow
SS^{*}\rightarrow\gamma\gamma b\overline{b})=2$ fb is assumed, which includes
the two configurations. The background includes the contribution from
$\gamma\gamma$ in association with $c\overline{c}$ and light quarks. The
$b$-jet tagging efficiency for $p_{T}=30$ GeV is assumed to be 68%. In this
setup a combined significance of over 7$\sigma$ could be achieved per LHC
experiment for 500 fb-1 of integrated luminosity, assuming the current best
fit to data is confirmed.

Interestingly, as the LEP experiments reported a mild excess (2.3$\sigma$) in
the search for a scalar boson ($S^{\prime}$) Barate et al. (2003) using the
process $e^{+}e^{-}\rightarrow Zh(\rightarrow b\overline{b})$ at 98 GeV for
the invariant $b\overline{b}$ mass, asymmetric $\gamma\gamma b\overline{b}$
final states could also originate from the decay $H\to SS^{\prime}$. This is
further supported by the CMS result reporting similar excesses with Run 1 data
and 35.9 fb-1 of Run 2 data Sirunyan et al. (2019b), with a local significance
of 2.8$\sigma$ at 95.3 GeV. In this context, t should be noted that searches
with asymmetric configurations of $H\rightarrow\tau^{+}\tau^{-}b\overline{b}$
are also substantiated.

##  V Conclusions and Outlook

In this article we accumulated evidence for the associate production of new
scalar particle $S$ (assumably via the decay of a heavier boson $H$) with a
mass $m_{S}=151.5$ GeV by combining the side-band data from CMS and ATLAS
searches for the SM Higgs. Including all channels involving photons, $Z$
bosons, $b$-quarks and/or missing energy, we find a global significance of
$4.8\sigma$ (locally $5.1\sigma$) for a mass range between $130\,$GeV and
$160\,$GeV.

While the hints that this new scalar is decaying into photons (and to a lesser
extent into the $Z\gamma$) are strong, in particular in association with
$E^{T}_{miss}$, decays of $S$ into, or its associate production with, bottom
quarks are preferred but still optional. In order to test the decay channel
$S\to b\overline{b}$ we suggest to perform asymmetric searches for the final
state $\gamma\gamma b\overline{b},\tau^{+}\tau^{-}b\overline{b}$, which also
allows an assessment of the branching ratio ${\rm Br}[H\to Sh]$. Furthermore,
such signals could be related to the LEP/CMS excess in associate
$b\overline{b}$ production and point towards a boson $S^{\prime}$ at $\approx
96\,$GeV that could lead to $H\to S^{(\prime)}S^{\prime}$.

Our analysis opens up new directions in particle physics. First of all, $S$
could be related to the multi-lepton anomalies. However, due to the absence of
a signal in $S\to ZZ^{*}$, an alternative mechanism for the lepton production,
such as $S\to NN$, where $N$ has the quantum numbers of a right-handed
neutrino, is possible. In such a setup, i.e. the 2HDM+$S$ extended with right-
handed neutrinos, the anomalous muon $g-2$ could be explained via a chiral
enhancement Crivellin et al. (2018); Chun and Mondal (2020); Ferreira et al.
(2021) and in case of a non-minimal flavour structure, also the intriguing
hints for NP in $b\to s\ell^{+}\ell^{-}$ (above the $7\,\sigma$ level Algueró
et al. (2021); Altmannshofer and Stangl (2021); Alok et al. (2020); Hurth et
al. (2021); Ciuchini et al. (2021)) could be explained Li et al. (2018); Marzo
et al. (2019); Iguro and Omura (2018); Crivellin et al. (2019). Furthermore,
in the context of the 96 GeV $S^{\prime}$ excess Barate et al. (2003);
Sirunyan et al. (2019b) searches for $H\to
S^{(\prime)}(\to\gamma\gamma,b\bar{b})S(\to{\rm invisible})$ are interesting.

###### Acknowledgements.

The authors are grateful for support from the South African Department of
Science and Innovation through the SA-CERN program and the National Research
Foundation for various forms of support. The work of A.C. is supported by a
professorship grant of the Swiss National Science Foundation (No.
PP00P21_76884).

## References

  * Higgs (1964a) P. W. Higgs, Phys. Lett. 12, 132 (1964a).
  * Englert and Brout (1964) F. Englert and R. Brout, Phys. Rev. Lett. 13, 321 (1964).
  * Higgs (1964b) P. W. Higgs, Phys. Rev. Lett. 13, 508 (1964b).
  * Guralnik et al. (1964) G. S. Guralnik, C. R. Hagen, and T. W. B. Kibble, Phys. Rev. Lett. 13, 585 (1964).
  * Aad et al. (2012) G. Aad et al. (ATLAS), Phys. Lett. B 716, 1 (2012), eprint 1207.7214.
  * Chatrchyan et al. (2012) S. Chatrchyan et al. (CMS), Phys. Lett. B 716, 30 (2012), eprint 1207.7235.
  * Chatrchyan et al. (2013) S. Chatrchyan et al. (CMS), Phys. Rev. Lett. 110, 081803 (2013), eprint 1212.6639.
  * Aad et al. (2013) G. Aad et al. (ATLAS), Phys. Lett. B 726, 120 (2013), eprint 1307.1432.
  * von Buddenbrock et al. (2018) S. von Buddenbrock, A. S. Cornell, A. Fadol, M. Kumar, B. Mellado, and X. Ruan, J. Phys. G 45, 115003 (2018), eprint 1711.07874.
  * Buddenbrock et al. (2019) S. Buddenbrock, A. S. Cornell, Y. Fang, A. Fadol Mohammed, M. Kumar, B. Mellado, and K. G. Tomiwa, JHEP 10, 157 (2019), eprint 1901.05300.
  * von Buddenbrock et al. (2020) S. von Buddenbrock, R. Ruiz, and B. Mellado, Phys. Lett. B 811, 135964 (2020), eprint 2009.00032.
  * Hernandez et al. (2021) Y. Hernandez, M. Kumar, A. S. Cornell, S.-E. Dahbi, Y. Fang, B. Lieberman, B. Mellado, K. Monnakgotla, X. Ruan, and S. Xin, Eur. Phys. J. C 81, 365 (2021), eprint 1912.00699.
  * von Buddenbrock et al. (2016) S. von Buddenbrock, N. Chakrabarty, A. S. Cornell, D. Kar, M. Kumar, T. Mandal, B. Mellado, B. Mukhopadhyaya, R. G. Reed, and X. Ruan, Eur. Phys. J. C 76, 580 (2016), eprint 1606.01674.
  * He et al. (2009) X.-G. He, T. Li, X.-Q. Li, J. Tandean, and H.-C. Tsai, Phys. Rev. D 79, 023521 (2009), eprint 0811.0658.
  * Grzadkowski and Osland (2010) B. Grzadkowski and P. Osland, Phys. Rev. D 82, 125026 (2010), eprint 0910.4068.
  * Chen et al. (2014) C.-Y. Chen, M. Freid, and M. Sher, Phys. Rev. D 89, 075009 (2014), eprint 1312.3949.
  * Muhlleitner et al. (2017) M. Muhlleitner, M. O. P. Sampaio, R. Santos, and J. Wittbrodt, JHEP 03, 094 (2017), eprint 1612.01309.
  * Krause et al. (2017) M. Krause, D. Lopez-Val, M. Muhlleitner, and R. Santos, JHEP 12, 077 (2017), eprint 1708.01578.
  * Aguilar et al. (2019) M. Aguilar et al. (AMS Collaboration), Phys. Rev. Lett. 122, 041102 (2019), URL https://link.aps.org/doi/10.1103/PhysRevLett.122.041102.
  * Ackermann et al. (2017) M. Ackermann, M. Ajello, A. Albert, W. B. Atwood, L. Baldini, J. Ballet, G. Barbiellini, D. Bastieri, R. Bellazzini, E. Bissaldi, et al., The Astrophysical Journal 840, 43 (2017), ISSN 1538-4357, URL http://dx.doi.org/10.3847/1538-4357/aa6cab.
  * Beck et al. (2021) G. Beck, M. Kumar, E. Malwa, B. Mellado, and R. Temo (2021), eprint 2102.10596.
  * Sabatta et al. (2020) D. Sabatta, A. S. Cornell, A. Goyal, M. Kumar, B. Mellado, and X. Ruan, Chin. Phys. C 44, 063103 (2020), eprint 1909.03969.
  * Abi et al. (2021) B. Abi et al. (Muon g-2), Phys. Rev. Lett. 126, 141801 (2021), eprint 2104.03281.
  * Aoyama et al. (2020) T. Aoyama et al., Phys. Rept. 887, 1 (2020), eprint 2006.04822.
  * Sirunyan et al. (2021) A. M. Sirunyan et al. (CMS), JHEP 07, 027 (2021), eprint 2103.06956.
  * ATL (2020) (2020).
  * Aad et al. (2020) G. Aad et al. (ATLAS), Phys. Rev. Lett. 125, 061802 (2020), eprint 2004.04545.
  * Sirunyan et al. (2020) A. M. Sirunyan et al. (CMS), Phys. Rev. Lett. 125, 061801 (2020), eprint 2003.10866.
  * Aad et al. (2021a) G. Aad et al. (ATLAS) (2021a), eprint 2104.13240.
  * Sirunyan et al. (2018a) A. M. Sirunyan et al. (CMS), JHEP 09, 046 (2018a), eprint 1806.04771.
  * Sirunyan et al. (2018b) A. M. Sirunyan et al. (CMS), JHEP 11, 152 (2018b), eprint 1806.05996.
  * ATLAS Collaboration (2020) ATLAS Collaboration (2020), URL https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/PAPERS/HIGG-2018-51/.
  * Aad et al. (2021b) G. Aad et al. (ATLAS), Phys. Rev. Lett. 126, 121802 (2021b), eprint 2010.06548.
  * ATLAS Collaboration (2018) ATLAS Collaboration, Eur. Phys. J. C 78, 293 (2018), eprint 1712.06386.
  * CMS (2019) (2019).
  * Andersen et al. (2013) J. R. Andersen et al. (LHC Higgs Cross Section Working Group) (2013), eprint 1307.1347.
  * Franceschini et al. (2016) R. Franceschini, G. F. Giudice, J. F. Kamenik, M. McCullough, A. Pomarol, R. Rattazzi, M. Redi, F. Riva, A. Strumia, and R. Torre, JHEP 03, 144 (2016), eprint 1512.04933.
  * Falkowski et al. (2016) A. Falkowski, O. Slone, and T. Volansky, JHEP 02, 152 (2016), eprint 1512.05777.
  * Benbrik et al. (2016) R. Benbrik, C.-H. Chen, and T. Nomura, Phys. Rev. D 93, 055034 (2016), eprint 1512.06028.
  * Chao et al. (2017) W. Chao, R. Huo, and J.-H. Yu, Eur. Phys. J. Plus 132, 27 (2017), eprint 1512.05738.
  * Sirunyan et al. (2019a) A. M. Sirunyan et al. (CMS), Phys. Lett. B 788, 7 (2019a), eprint 1806.00408.
  * Aaboud et al. (2018) M. Aaboud et al. (ATLAS), JHEP 11, 040 (2018), eprint 1807.04873.
  * Alwall et al. (2014) J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H. S. Shao, T. Stelzer, P. Torrielli, and M. Zaro, JHEP 07, 079 (2014), eprint 1405.0301.
  * Ball et al. (2015) R. D. Ball et al. (NNPDF), JHEP 04, 040 (2015), eprint 1410.8849.
  * Alloul et al. (2014) A. Alloul, N. D. Christensen, C. Degrande, C. Duhr, and B. Fuks, Comput. Phys. Commun. 185, 2250 (2014), eprint 1310.1921.
  * Sjostrand et al. (2006) T. Sjostrand, S. Mrenna, and P. Z. Skands, JHEP 05, 026 (2006), eprint hep-ph/0603175.
  * de Favereau et al. (2014) J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaître, A. Mertens, and M. Selvaggi (DELPHES 3), JHEP 02, 057 (2014), eprint 1307.6346.
  * Cacciari et al. (2012) M. Cacciari, G. P. Salam, and G. Soyez, Eur. Phys. J. C 72, 1896 (2012), eprint 1111.6097.
  * Barate et al. (2003) R. Barate et al. (LEP Working Group for Higgs boson searches, ALEPH, DELPHI, L3, OPAL), Phys. Lett. B 565, 61 (2003), eprint hep-ex/0306033.
  * Sirunyan et al. (2019b) A. M. Sirunyan et al. (CMS), Phys. Lett. B 793, 320 (2019b), eprint 1811.08459.
  * Crivellin et al. (2018) A. Crivellin, M. Hoferichter, and P. Schmidt-Wellenburg, Phys. Rev. D 98, 113002 (2018), eprint 1807.11484.
  * Chun and Mondal (2020) E. J. Chun and T. Mondal, JHEP 11, 077 (2020), eprint 2009.08314.
  * Ferreira et al. (2021) P. M. Ferreira, B. L. Gonçalves, F. R. Joaquim, and M. Sher (2021), eprint 2104.03367.
  * Algueró et al. (2021) M. Algueró, B. Capdevila, S. Descotes-Genon, J. Matias, and M. Novoa-Brunet, in _55th Rencontres de Moriond on QCD and High Energy Interactions_ (2021), eprint 2104.08921.
  * Altmannshofer and Stangl (2021) W. Altmannshofer and P. Stangl (2021), eprint 2103.13370.
  * Alok et al. (2020) A. K. Alok, S. Kumbhakar, and S. Uma Sankar (2020), eprint 2001.04395.
  * Hurth et al. (2021) T. Hurth, F. Mahmoudi, and S. Neshatpour, Phys. Rev. D 103, 095020 (2021), eprint 2012.12207.
  * Ciuchini et al. (2021) M. Ciuchini, M. Fedele, E. Franco, A. Paul, L. Silvestrini, and M. Valli, Phys. Rev. D 103, 015030 (2021), eprint 2011.01212.
  * Li et al. (2018) S.-P. Li, X.-Q. Li, Y.-D. Yang, and X. Zhang, JHEP 09, 149 (2018), eprint 1807.08530.
  * Marzo et al. (2019) C. Marzo, L. Marzola, and M. Raidal, Phys. Rev. D 100, 055031 (2019), eprint 1901.08290.
  * Iguro and Omura (2018) S. Iguro and Y. Omura, JHEP 05, 173 (2018), eprint 1802.01732.
  * Crivellin et al. (2019) A. Crivellin, D. Müller, and C. Wiegand, JHEP 06, 119 (2019), eprint 1903.10440.


# Implicit Offline Reinforcement Learning via Supervised Learning

Alexandre Piché
ServiceNow Research
Mila, Université de Montréal
&Rafael Pardiñas, David Vázquez
ServiceNow Research
&Igor Mordatch
Google Brain
Chris J. Pal
ServiceNow Research
Canada CIFAR AI Chair
Mila, Polytechnique Montréal
Correspondence to<EMAIL_ADDRESS>
###### Abstract

Offline _Reinforcement Learning (RL) via Supervised Learning_ is a simple and
effective way to learn robotic skills from a dataset collected by policies of
different expertise levels. It is as simple as supervised learning and
Behavior Cloning (BC), but takes advantage of return information. On datasets
collected by policies of similar expertise, implicit BC has been shown to
match or outperform explicit BC. Despite the benefits of using implicit models
to learn robotic skills via BC, offline _RL via Supervised Learning_
algorithms have been limited to explicit models. We show how implicit models
can leverage return information and match or outperform explicit algorithms to
acquire robotic skills from fixed datasets. Furthermore, we show the close
relationship between our implicit methods and other popular _RL via Supervised
Learning_ algorithms to provide a unified framework. Finally, we demonstrate
the effectiveness of our method on high-dimension manipulation and locomotion
tasks.

##  1 Introduction

Large datasets of varied interactions combined with Offline RL algorithms are
a promising direction to learning robotic skills safely (Levine et al., 2020).
A practical and straightforward approach to leveraging large and varied
robotics datasets is to convert the RL problem into a supervised learning
problem (Emmons et al., 2021; Chen et al., 2021). _RL via Supervised Learning_
(RvS) algorithms can be seen as return-conditional, -filtered, or -weighted
BC. Here, we use RvS to refer to policies where the action distribution is
conditioned on a return defined by the user or generated by the agent.

_Reinforcement Learning via Supervised Learning_ algorithms are as simple as
BC ones, but since they take advantage of the return information, they can
leverage sub-optimal interactions. In order to be effective for datasets
collected by policies of different expertise levels, these algorithms must
model multi-modal joint distributions, e.g., over the actions and the return.
Previous RvS methods either discretized the variables and could use
multinomial distributions (Chen et al., 2021) or ignored the multi-modality of
the variables and used simple distributions such as a Gaussian distribution
(Kumar et al., 2019). This paper shows that implicit models can be excellent
for modeling multi-modal distributions without the need to discretize the
return and action variables. In addition, implicit models have been shown to
better extrapolate and model discontinuities than explicit models allowing
them to capture complex robotic behaviors (Florence et al., 2021). Despite the
advantages of using implicit models, RvS algorithms have been limited to
explicit models.

In this work, we bridge the gap between implicit models and RvS, and propose
the first implicit RvS (IRvS) algorithm. We show that our IRvS algorithm is
closely related to other RvS algorithms but has the advantages of 1) not
requiring the user or a second neural network to specify a target return, and
2) not requiring continuous outputs to be discretized to capture multi-modal
distributions. Furthermore, we demonstrate the superiority of IRvS over RvS to
interpolate, model discontinuity, and handle higher-dimension action spaces in
an environment with linear dynamics. Furthermore, using the same objective as
IRvS, we derive a novel RvS algorithm formulation where the target return is
obtained by maximizing the exponential tilt of the return. Finally, we show
that our method achieves state-of-the-art on the difficult ADROIT and
FrankaKitchen manipulation tasks and is competitive with Offline RL and RvS
algorithms on a suite of robotic locomotion environments.

##  2 Preliminaries

(a) Explicit RvS.

(b) Our Implicit RvS (IRvS) approach.

Figure 1: Comparison of implicit and explicit _Reinforcement Learning via
Supervised Learning_ (RvS). Implicit models can be more straightforward and
use a single neural network. In contrast, explicit models require either a
second neural networks (not depicted here) or the user to specify a task-
specific target return $\hat{G}$.

Offline Reinforcement Learning. We consider a Markov Decision Process (MDP)
$({\mathcal{S}},{\mathcal{A}},{\mathcal{P}},{\mathcal{R}})$ with states
${\mathbf{s}}\in{\mathcal{S}}$, actions ${\mathbf{a}}\in{\mathcal{A}}$,
transition function
${\mathcal{P}}({\mathbf{s}}^{\prime}|{\mathbf{s}},{\mathbf{a}})$, and reward
function $r={\mathcal{R}}({\mathbf{s}},{\mathbf{a}})$. We aim to learn a
policy $\pi:{\mathcal{S}}\rightarrow{\mathcal{A}}$ maximizing the expected
return $\mathbb{E}_{\tau\sim\pi}\big{[}G(\tau)\big{]}$, where
$G(\tau)=\sum_{t=0}^{H}r_{t}$. Contrary to “online” RL agents that can
interact with the environment, we focus on agents that must learn a policy
exclusively from a dataset ${\mathcal{D}}$ of trajectories
$\tau=({\mathbf{s}}_{0},{\mathbf{a}}_{0},r_{0},\ldots,{\mathbf{s}}_{H},{\mathbf{a}}_{H},r_{H})$
collected under a behavior policy $\mu$. For the rest of the paper, we work
with the tuple $({\mathbf{s}}_{i},{\mathbf{a}}_{i},G_{i})$, where
$G_{i}=\sum_{t=i}^{H}r_{i}$ is the cumulative return starting from state-
action $({\mathbf{s}}_{i},{\mathbf{a}}_{i})$.

Implicit Models and Energy-Based Models. We define an _implicit model_ as a
general function approximator $E:\mathbb{R}^{D}\rightarrow\mathbb{R}$ where
inference can be performed via optimization of
$\hat{y}=\arg\min_{\mathbf{y}}E_{\theta}(x,y)$. In this work, we focus on
_Energy Based Models_ (LeCun et al., 2006) (EBMs). EBMs associate scalar
energy to a configuration of variables $({\mathbf{x}},{\mathbf{y}})$. The
model is trained to predict lower energy on the observed data than on the
unobserved one. We use an EBMs to model the conditional density

$p_{\bm{\theta}}({\mathbf{y}}|{\mathbf{x}})=\frac{\exp(-E_{\bm{\theta}}({\mathbf{x}},{\mathbf{y}}))}{Z_{\bm{\theta}}}$
(1)

where $E_{\bm{\theta}}$ is the energy and the normalizing constant
$Z_{\bm{\theta}}=\int_{\mathbf{y}}\exp(-E_{\bm{\theta}}({\mathbf{x}},{\mathbf{y}}))d{\mathbf{y}}$.
Training the density model can be performed using the InfoNCE loss function
(Oord et al., 2018) defined as

$\displaystyle{\mathcal{L}}({\bm{\theta}})$
$\displaystyle=\sum_{i=1}^{N}-\log(\tilde{p}_{\bm{\theta}}({\mathbf{y}}_{i}\mid{\mathbf{x}},\\{\tilde{{\mathbf{y}}}_{i}^{j}\\}_{j=1}^{N_{\text{neg.}}}))$
(2)
$\displaystyle\tilde{p}_{\bm{\theta}}({\mathbf{y}}_{i}\mid{\mathbf{x}},\\{\tilde{{\mathbf{y}}}_{i}^{j}\\}_{j=1}^{N_{\text{neg.}}})$
$\displaystyle=\frac{\exp({-E_{\bm{\theta}}({\mathbf{x}}_{i},{\mathbf{y}}_{i})})}{\exp({-E_{\bm{\theta}}({\mathbf{x}}_{i},{\mathbf{y}}_{i})})+\sum_{j=1}^{N_{\text{neg.}}}\exp({-E_{\bm{\theta}}({\mathbf{x}}_{i},\tilde{{\mathbf{y}}}^{j}_{i})})}.$

where $\\{\tilde{{\mathbf{y}}}_{i}^{j}\\}_{j=1}^{N_{\text{neg.}}}$ is a set of
negative counter-examples. Negative sampling can be performed by minimizing
$\tilde{{\mathbf{y}}}_{i}^{j}=\arg\min_{{\mathbf{y}}}E_{\bm{\theta}}({\mathbf{x}}_{i},{\mathbf{y}})$
using stochastic gradient Langevin dynamics (SGLD) (Welling & Teh, 2011), see
Algorithm 1 for details. Finally, prediction can be done by minimizing the
energy

$\hat{{\mathbf{y}}}=\arg\min_{{\mathbf{y}}\in{\mathcal{Y}}}E_{\bm{\theta}}({\mathbf{x}},{\mathbf{y}})$
(3)

which can also be done via SGLD.

Implicit Behavior Cloning. Behavior Cloning (Pomerleau, 1988) is a simple and
popular way to acquire robotic skills from a dataset of expert demonstrations.
The policy is usually modeled via an explicit model and learned by minimizing
the mean squared error loss
$\mathbb{E}_{{\mathbf{s}},{\mathbf{a}}\sim{\mathcal{D}}}\big{[}({\mathbf{a}}-f_{\bm{\theta}}({\mathbf{s}}))^{2}\big{]}$
to clone the behavior policy $\mu$. It is particularly effective when the
dataset consists of expert trajectories. Florence et al. (2021) introduced
_Implicit Behavior Cloning_ where they leverage an EBM to learn a policy from
offline data. They showed that EBMs could capture multi-modal distribution,
interpolate, and model discontinuities making them particularly effective on
continuous control tasks. The EBM is trained to learn the conditional density

$p_{\bm{\theta}}({\mathbf{a}}|{\mathbf{s}})=\frac{\exp(-E_{\bm{\theta}}({\mathbf{s}},{\mathbf{a}}))}{Z_{\bm{\theta}}}.$
(4)

The decision-making is performed by minimizing the energy

$\hat{{\mathbf{a}}}=\arg\min_{{\mathbf{a}}\in{\mathcal{A}}}E_{\bm{\theta}}({\mathbf{s}},{\mathbf{a}}).$
(5)

which can be minimized using SGLD. The advantages of EBMs allowed IBC to
outperform explicit methods by 100% in some environments. While IBC performs
particularly well on expert data, it does not leverage return information,
enabling it to leverage more diverse non-expert data. This is an important
limitation, as sub-optimal data might be easier to obtain, have better state
coverage and result in better representation than expert data.

##  3 Implicit Offline RL via Supervised Learning

###  3.1 Method

Inspired by the successes of EBMs on robotic tasks, we introduce _Implicit
Reinforcement Learning via Supervised Learning_ (IRvS) to improve on the IBC
algorithm by modeling the dependencies between the state, action, and return
variables. Modeling these dependencies would allow the robotics community to
use implicit methods on varied datasets instead of being limited to datasets
of expert demonstrations. Specifically, we model the following state
conditional distribution

$\displaystyle
p_{\bm{\theta}}({\mathbf{a}},G|{\mathbf{s}})=\frac{\exp(-E_{\bm{\theta}}({\mathbf{s}},{\mathbf{a}},G))}{Z_{\bm{\theta}}}.$
(6)

We now have access to a differentiable joint density model
$p_{\bm{\theta}}({\mathbf{a}},G\mid{\mathbf{s}})$ from which we can sample
action and return pairs using SGLD. Doing so results in actions likely under
the training dataset irrespective of their associated returns, which is
equivalent to the IBC algorithm. Instead, we want to bias sampling towards
actions with high returns.

To sample action with high return, we instead sample from the _exponential
tilt_ (Asmussen & Glynn, 2007) in $G$ of the state conditional distribution.
Specifically, we use the exponential tilt in $G$ defined as

$\displaystyle p_{\bm{\theta}}({\mathbf{a}},G\mid{\mathbf{s}}_{t};\eta)$
$\displaystyle=p_{\bm{\theta}}({\mathbf{a}},G\mid{\mathbf{s}}_{t})\exp(\eta^{-1}G-\kappa(\eta)),$
(7)

where $\kappa(\eta)$ is the _cumulant generating function_
$\log\mathbb{E}[e^{\eta^{-1}G}]$ and $\eta$ controls the trade-off between the
likeliness of an action-return pair under the model and the magnitude of the
return. We note that the exponential tilt density is still differentiable and
can be efficiently sampled via SGLD. Specifically, the policy (and its
associated return) can be obtained by maximizing the log density of the
exponential tilt

$\displaystyle\hat{{\mathbf{a}}},\hat{G}=\operatorname*{arg\,min}_{{\mathbf{a}}\in{\mathcal{A}},G\in{\mathcal{G}}}E_{\bm{\theta}}({\mathbf{s}},{\mathbf{a}},G)-\eta^{-1}G,$
(8)

where we can safely ignore the normalizing constant $\kappa(\eta)$.
Informally, the above scheme generates high, but plausible returns under the
model. For the rest of the paper, we omit $\kappa(\eta)$ to allege the
notation.

###  3.2 Didactic Example

(a) IRvS $\eta^{-1}=0$. In Figure i), we observe that the policy is the same
as the goal agnostic policy that collected the data. In Figure ii), we observe
that IRvS can successfully model the 4 possible non-discounted returns. In
Figure iii), we observe that IRvS can successfully capture the joint
distribution over actions and returns and their dependencies, where
$a=\arctan\frac{\Delta y}{\Delta x}$.

(b) IRvS $\eta^{-1}=5.0$. In Figure i), we observe that IRvS policy is now
leading towards the largest goal for most of the state space. In Figure ii),
we observe that the exponential tilt of the return distribution is biased
towards the highest reward. In Figure iii), we observe that the mass is
concentrated over the return of 1 and actions near $\pi/4$ which are the
optimal actions and returns.

Figure 2: Effect of $\eta^{-1}$. This experiment demonstrates how IRvS can
leverage sub-optimal demonstrations to achieve near-optimal behaviors by
increasing the parameter $\eta^{-1}$. The change of behavior caused by
increasing $\eta^{-1}$ from 0 to 3 can be observed in the policy, the return
prediction, and their joint distribution.

We build a simple didactic example to better understand how the hyper-
parameter $\eta^{-1}$ changes the behavior of the IRvS algorithm. The
environment has linear dynamics and goals located at each corner of the room.
Each goal yields a different reward (-1, -1/4, 1/2, 1), and the dataset is
collected under a behavior policy that visits each goal with equal
probability. The dataset thus consists of 25% of optimal demonstrations and
75% sub-optimal ones. The 2d actions $(\Delta_{x},\Delta_{y})$ are represented
as $a=\arctan\frac{\Delta y}{\Delta x}$.

In Figure 2(a), we study the behavior of IRvS when $\eta^{-1}=0$ and is
equivalent to IBC. Since the resulting policy is reward-agnostic, we expect
the policy learned by IRvS to visit each goal with equal probability which can
indeed be observed in the first sub-figure. The IRvS algorithm also
successfully models the distribution of non-discounted returns which are
possible to reach from the starting state. We note that IRvS can model the
continuous return distribution, while RvS would need the returns to be
discretized to capture the multi-modality of the distribution, see Figure 3.
Finally, we observe that the joint distribution over returns and actions is
successfully captured in the third figure.

In Figure 2(b), we study the impact of increasing $\eta^{-1}$ to 3. In the
first sub-figure, we observe that the policy is now heading towards the
largest reward for most of the state space. In the second sub-figure, the
distribution mass over return has now mostly shifted towards larger returns
which are likely under the dataset, i.e., +1. In the third sub-figure, we
observe that the most likely action is $\pi/4$ which is pointing towards the
best goal.

##  4 Related Work

###  4.1 Implicit Models and Reinforcement Learning

We note that other works have used EBMs in RL. Interestingly, most previous
works treat the $Q$-value as the energy and do not model how likely the
$Q$-value is (or the return in our case). Sallans & Hinton (2004) parametrize
the Restricted Boltzmann Machine (Freund & Haussler, 1991; Welling et al.,
2004) negative energy function as the Q value. And more recently, Haarnoja et
al. (2017) learn a $Q$ function via the (soft-) Bellman equation, then treat
the $Q$ function as the negative energy function, which is sampled via a
stochastic neural network. Recently, EBM techniques have been used to learn
conservative $Q$-values (Kostrikov et al., 2021).

###  4.2 Connection to Return Conditional Policies

Figure 3: For RvS, we can use the exponential tilt of a learned distribution
over the return to obtain a distribution biased towards higher returns. As we
increase $\eta^{-1}$, the mass shifts towards higher returns. A limitation of
these RvS methods is that the continuous distribution of return has to be
discretized as shown here. While our proposed implicit method does not require
the return to be discretized.

Offline _RL via supervised learning_ is often done by learning a return
conditional policy (Srivastava et al., 2019; Kumar et al., 2019; Emmons et
al., 2021; Chen et al., 2021). Specifically, the policy
$p({\mathbf{a}}|{\mathbf{s}},G)$ is conditioned on the return $G$, which can
easily be learned via supervised learning and does not require _Temporal-
Difference learning_ (Sutton, 1988). At test time, the policy requires a
return-to-go to be conditioned on. It is often treated as a task-dependent
hyper-parameter (Chen et al., 2021; Emmons et al., 2021) or as a biased
estimate of the return (a scalar instead of a distribution) estimated by a
second neural network (Kumar et al., 2019).

Inspired by the exponential tilt formulation used for IRvS, we propose instead
to model the state conditional distribution over returns
$p(G\mid{\mathbf{s}})$ using supervised learning by discretizing the returns,
see Figure 3. Similarly to IRvS, the return to go can be obtained by
maximizing the exponential tilt

$\hat{G}_{t}=\arg\max_{G}p(G\mid{\mathbf{s}}_{t})\exp(\eta^{-1}G-\kappa(\eta))$
(9)

resulting in a high return which is also likely under the learned model. The
three distributions are depicted in Figure 3. Thus, we argue that if the
action is obtained by maximizing the learned likelihood of the model, e.g.,
outputting the mean of a Gaussian distribution,

$\hat{{\mathbf{a}}}=\arg\max_{\mathbf{a}}p_{\bm{\theta}}({\mathbf{a}}\mid{\mathbf{s}}_{t},\hat{G}_{t}).$
(10)

Thus, RvS and IRvS obtain the actions by maximizing the same objective, and
their main differences are in the modeling choices. First, using an implicit
model is simpler than an explicit one since it requires only a single neural
network, $p_{\bm{\theta}}({\mathbf{a}},G|{\mathbf{s}})$, compared to two
networks for an explicit model, i.e.,
$p_{\bm{\theta}}({\mathbf{a}}\mid{\mathbf{s}}_{t},\hat{G}_{t})$ and
$p_{\bm{\theta}}(\hat{G}_{t}\mid{\mathbf{s}}_{t})$, see Figure 1. Second,
implicit models can model rich continuous distributions making them a natural
tool for control tasks, while explicit models must discretize continuous
variables or assume a parametric distribution such as the Gaussian to model
the multi-modal action and return distributions. In summary, by modeling the
dependency between the state, action, and return with an implicit model, we
bridge an important gap between the popular IBC and RvS methods.

Concurrently, Lee et al. (2022) introduced a binary “expert” variable, which
leads to a similar way to select the return-to-go on, which the policy is
conditioned on. They then demonstrated impressive performance using a
Transformer on the Atari suite. In order to model the distribution over the
return-to-go, they discretize the variable and model it with a categorical
distribution. We note that introducing a binary “expert” variable is closely
related to the concept of “optimality” in control-as-inference (Levine, 2018).

##  5 Experimental Results

###  5.1 Advantages of Implicit RvS Over Explicit RvS

(a) The agent must reach the left-most goal.

(b) Near discontinuities, IRvS reaches both goals, while RvS fails to reach
either goal and converges to the purple point.

(c) Difference in performance between implicit RvS and (explicit) RvS.

Figure 4: Comparison between Implicit and Explicit RvS method. Figure a) shows
how the agent changes focus from one goal to the other depending on which one
is placed more to the left. Figure b) depicts how implicit vs explicit RvS
behave when the goal to reach is unclear. Figure c) measure the performance of
implicit and explicit RvS method along extrapolation capabilities, environment
dimensionality, and policy discontinuities.

In order to understand the advantages of implicit RvS over explicit RvS, we
proposed a simple navigation task with linear dynamics, where the agent has to
reach the left-most goal, i.e. the goal with the smallest first dimension. The
optimal policy while being relatively simple is _discontinuous_ as the left-
most goal changes. For example, in Figure 4(a), slightly moving the bottom
goal left results in a completely different optimal action.

Experimental settings. For all experiments, we keep the number of
demonstrations constant at 100,000. The agent’s initial position
${\mathbf{s}}$ is sampled uniformly from $[-1,1]^{D}$. The 2 goals during
training are independently sampled uniformly from $[-0.1,0.1]^{D}$. The state
is composed of the agent’s position and two goals, resulting in a state of
dimensions $3\times D$, The optimal action is
${\mathbf{a}}^{*}=g^{*}-{\mathbf{s}}$ which is also of dimension $D$, where
$g^{*}$ is the left-most goal. To better understand the advantages of implicit
models over explicit ones, we measure i) the performance as we increase the
dimensionality of the environment, ii) the ability to model discontinuous
policies, and iii) the generalization capabilities of the model as we change
the goal sampling distribution.

Extrapolation. Implicit models have been shown to extrapolate more effectively
than explicit ones. This environment demonstrates this is also the case for
implicit versus explicit RvS. To study extrapolation, the dataset is composed
of demonstrations where the goals are placed near the origin, i.e., $g$ is
sampled on $[-0.1,0.1]^{D}$. At the same time, we can increase the goal
sampling radius ($\epsilon$) at test time to measure the extrapolation
capabilities of the model. In Figure 4(c) a) we observe that implicit RvS
extrapolates more effectively than explicit RvS to changes in goal radius much
larger than observed during training.

High-dimensions. As we increase the environment’s dimensionality but keep the
dataset’s size constant, the training demonstrations are, on average, further
away from what is observed at test time. Thus increasing the dimensions of the
environment increases the difficulty of the task. In Figure 4(c) b), we can
observe that implicit RvS performance degrades more gracefully than explicit
RvS. Similar behavior has also been observed in Florence et al. (2021).

Discontinuities. Looking at Figure 4(a), we can see that as the bottom goal
slide to the right, the optimal action switches from going to the bottom goal
to the top one: making the optimal policy highly discontinuous. This
represents a challenge for explicit models as they can hardly fit
discontinuous functions and instead fit intermediate values. We can measure
the models ability to handle discontinuous functions by controlling the
relative position of the second goal. Specifically, instead of sampling two
goals independently (as done above), we can set the second goal’s first
dimension equal to the first goal’s first dimension with a small amount of
uniform noise sampled from $[-\delta,\delta]$. In Figure 4(b), we observe
qualitatively the behavior of the two models for $\delta$=1e-4. We observe
that the (explicit) RvS policy converges to the purple point which is between
the two goals and rarely reaches one of them, while IRvS policy reaches the
correct goal half the time. Finally, in Figure 4(c), we quantify the
differences between the policies by measuring their success rate for different
$\delta$. We see that IRvS outperforms RvS for all $\delta$.

###  5.2 Continuous Control Dataset

We compare both IRvS and our version of RvS (as defined by Equation 9 and
Equation 10) to implicit and explicit state-of-the-art algorithms on a suite
of offline RL tasks provided by d4rl (Fu et al., 2020). Specifically, we
compare our methods to conservative Q-learning (Kumar et al., 2020), to the
implicit algorithms IBC and IBC w/ RWR proposed by Florence et al. (2021), and
to the explicit algorithms BC from Fu et al. (2020), and _RvS hand-tuned_ (the
target returns were selected by hand on a task-level) from Emmons et al.
(2021).

####  5.2.1 Experiment settings.

We fine-tuned the hyper-parameter $\eta^{-1}$ for our algorithm on 3 seeds,
see Section A.3. Then we run 3 seeds on the whole suite where we evaluate our
algorithms for 20 episodes, for task-level results see Table 1. As in Emmons
et al. (2021) we define $G$ as the average future reward. Additionally, we
normalize each state coordinate to have mean 0 and standard deviation 1, and
the actions and the return $G$ to lie in $[-1,1]$.

ADROIT. The ADROIT benchmark (Kumar et al., 2013; Rajeswaran et al., 2017)
consists of 4 environments (relocate, pen, door, hammer) and 3 datasets
(human, cloned, expert) for a total of 12 tasks. The agent controls a 24-DoF
anthropomorphic manipulator and must solve challenging and dynamic dexterous
manipulation. The first environment relocation requires the agent to move a
ball to a randomized target. Similarly, the pen environment requires the agent
to reposition a pen to match the orientation of a randomized target. The door
environment requires the agent to undo the latch and swing the door open.
Finally, the hammer environment requires the agent to pick up a hammer and
drive a nail into a board. The human dataset is comprised of 25 human
demonstrations. The expert dataset is comprised of 5000 trajectories sampled
from an expert policy. The cloned dataset is comprised of 2500 trajectories
from the expert policy and 2500 trajectories from a behavior cloning policy
trained on the human demonstrations.

Mujoco. The offline mujoco benchmark (Todorov et al., 2012) consists of 3
locomotion environments (halfcheetah, walker2d, and hopper) and 3 datasets
(medium, medium-replay, medium-expert) for a total of 9 tasks. The hopper
environment consists of a one-legged robot system with 11-dimensional state
space and 3 degrees of freedom (DoF), halfcheetah is a cheetah robot with
11-dimensional state pace and 6-DoF, finally, walker2d has a 17-dimensional
state space and 6-DoF. The medium datasets consist of $10^{6}$ samples, the
medium-expert of $2\times 10^{6}$ samples, and the medium replay of
$~{}10^{5}$ samples.

FrankaKitchen. The FrankaKitchen (Gupta et al., 2019) (kitchen) environment
places a 9-DoF robot in a realistic kitchen environment where it can interact
freely with different objects, e.g. opening the microwave and various
cabinets, moving the kettle, and turning on the lights and burners. The task
consists of reaching the desired configuration of the objects in the scene.
The kitchen benchmark consists of 3 different tasks (complete, partial, and
mixed). The complete dataset includes demonstrations of all 4 target sub-tasks
being completed in order. The partial dataset includes consists of multiple
other tasks being performed including the 4 target sub-tasks being completed
in sequence. Finally, the mixed dataset consists of various sub-tasks, but the
4 target sub-tasks are never completed in sequence together.

|  |  | IRvS (Ours) | IRvS (Ours) | BC | CQL | RvS | IBC w/ RWR | RvS (Ours) | RvS (Ours) | IBC
---|---|---|---|---|---|---|---|---|---|---|---
|  |  | $\eta^{-1}=1$ | Best $\eta$ |  |  | Hand-tuned |  | $\eta^{-1}=10$ | Best $\eta$ |
adroit | cloned | door | 6.3 | 13.0 | -0.1 | 0.4 | - | 0.1 | 0.1 | 0.1 | -0.1
|  | hammer | 3.9 | 3.9 | 0.8 | 2.1 | - | 0.3 | 2.8 | 3.1 | 0.3
|  | pen | 76.7 | 90.2 | 56.9 | 39.2 | - | 81.1 | 80.3 | 81.4 | 64.5
|  | relocate | 0.1 | 0.7 | -0.1 | -0.1 | - | 0.1 | 0.0 | 0.1 | 0.0
| expert | door | 103.7 | 105.8 | 34.9 | 101.5 | - | 104.4 | 105.4 | 105.5 | 101.7
|  | hammer | 127.2 | 127.2 | 125.6 | 86.7 | - | 118.0 | 127.0 | 127.0 | 123.4
|  | pen | 128.1 | 138.1 | 85.1 | 107.0 | - | 123.0 | 127.5 | 127.5 | 116.2
|  | relocate | 102.5 | 107.5 | 101.3 | 95.0 | - | 106.3 | 106.5 | 106.7 | 102.3
| human | door | 15.6 | 16.5 | 0.5 | 9.9 | - | 13.2 | 2.9 | 4.4 | 11.0
|  | hammer | 1.5 | 1.7 | 1.5 | 4.4 | - | 1.2 | 1.9 | 2.3 | 1.3
|  | pen | 84.3 | 84.3 | 34.4 | 37.5 | - | 56.7 | 62.9 | 76.6 | 71.4
|  | relocate | 0.3 | 0.3 | 0.0 | 0.2 | - | 0.1 | 0.0 | 0.1 | 0.1
kitchen | complete | kitchen | 65.4 | 78.8 | 33.8 | 43.8 | 1.5 | 69.1 | 45.8 | 46.9 | 76.2
| mixed | kitchen | 67.9 | 67.9 | 47.5 | 51.0 | 1.1 | 52.1 | 46.5 | 47.9 | 39.6
| partial | kitchen | 55.4 | 67.1 | 33.8 | 49.0 | 0.5 | 52.9 | 51.5 | 51.5 | 42.5
mujoco | medium | halfcheetah | 41.7 | 42.1 | 36.1 | 49.1 | 41.6 | 41.6 | 42.8 | 42.8 | 41.5
|  | hopper | 61.7 | 61.7 | 29.0 | 64.6 | 60.2 | 54.1 | 59.4 | 59.4 | 53.3
|  | walker2d | 64.9 | 69.5 | 6.6 | 82.9 | 71.7 | 64.3 | 70.4 | 70.4 | 39.7
| medium-expert | halfcheetah | 93.5 | 93.5 | 35.8 | 85.8 | 92.2 | 92.5 | 92.2 | 92.2 | 51.4
|  | hopper | 101.6 | 101.6 | 111.9 | 102.0 | 101.7 | 102.1 | 91.1 | 94.9 | 91.4
|  | walker2d | 107.2 | 107.2 | 6.4 | 109.5 | 106.0 | 105.8 | 107.8 | 108.3 | 97.8
| medium-replay | halfcheetah | 34.4 | 34.4 | 38.4 | 47.3 | 38.0 | 37.4 | 40.0 | 40.0 | 26.6
|  | hopper | 25.0 | 27.2 | 11.8 | 97.8 | 73.5 | 17.2 | 61.7 | 63.0 | 1.8
|  | walker2d | 52.9 | 52.9 | 11.3 | 86.1 | 60.6 | 26.6 | 51.8 | 51.8 | 16.0

Table 1: Normalized score per task.. In general, IRvS performs well on high-
dimensional manipulation tasks where the dataset has been collected by
policies on varying expertise levels. While CQL and explicit methods perform
well on the locomotion tasks. All algorithms perform well on the adroit expert
dataset.

####  5.2.2 Results

ADROIT. On 11 of the 12 datasets, IRvS obtains or matches the best
performance. We also note that, excluding IRvS, IRvS $\eta^{-1}=1$ obtains the
best performance on 6 of the 12 datasets. With the exception of Conservative
Q-learning (CQL) (Kumar et al., 2020) obtaining the best score on hammer-
human, implicit methods outperform explicit ones on most tasks. The gap
between IRvS and RvS is particularly noticeable for the human and cloned
datasets hinting that implicit methods perform better for high-dimensional
control tasks combined with datasets collected by policies of different
proficiency.

Mujoco. CQL obtains the best score on 8 of the 9 datasets while IRvS
$\eta^{-1}=1$ obtains the best score on halfcheetah medium-expert. These
results suggest that TD learning and _trajectory stitching_ are crucial on the
mujoco benchmark to obtain the best performance. Interestingly, IRvS and _RvS
hand-tuned_ (Emmons et al., 2021) obtain almost identical performance on the
medium and medium-expert datasets, while RvS obtains much better scores on the
medium-replay datasets. It is worth noting that our implementation of RvS
$\eta^{-1}=10$ achieves roughly the same score as _RvS hand-tuned_ on most of
the locomotion tasks, where _RvS hand-tuned_ condition on the following hand-
selected return: 110 for the medium-expert dataset, 90 for hopper,
walker2d-medium-replay, and walker2d-medium, 60 for hopper-medium, and 40 for
halfcheetah-medium, medium-replay.

FrankaKitchen. IRvS reaches the best score on all 3 datasets. Additionally, if
excluding IRvS, IRvS $\eta^{-1}=1$ reaches the best score on 2 of the 3 tasks,
while IBC (Florence et al., 2021) obtains the best score on kitchen complete -
which is well suited for behavior cloning. Surprisingly, RvS $\eta^{-1}=10$
outperforms by a wide margin _RvS hand-tuned_. Overall, implicit methods
outperform explicit methods for all kitchen tasks.

Figure 5: Effectiveness of $\eta^{-1}$ to incorporate return information. We
report the performance of IRvS as a function of $\eta^{-1}$ on the medium-
expert datasets. For walker2d, we observe that increasing $\eta^{-1}$ slightly
improves the performance, but does not impact performance too much past a
certain point. For halfcheetah, we see that increasing $\eta^{-1}$
dramatically improves the performance. For hopper, we observe that increasing
$\eta^{-1}$ improves the performance up to a point, then the return falls
below the behavior cloning performance ($\eta^{-1}=0$). These observations
confirm that $\eta^{-1}$ allows us to trade-off between the likelihood of a
behavior and its potential return. We denote $\eta^{-1}=1$ (the best overall
hyper-parameter across all tasks) by red vertical lines.

Sensitivity Analysis. In Figure 5, we observe that for every task the best
performance is obtained with $\eta^{-1}>0$. Thus confirming the effectiveness
of $\eta^{-1}$ to incorporate return information by outperforming IBC
($\eta^{-1}=0$). Interestingly, increasing $\eta^{-1}$ beyond a point does not
impact the performance for walker2d. However, increasing $\eta^{-1}$ beyond a
point decreases the performance on hopper, thus implying that some tasks
cannot be solved by naively conditioning on the maximum observed return.
Finally, while the best performance is reached using different $\eta^{-1}$, we
found that using $\eta^{-1}=1$ (red vertical line) for all tasks results in
good overall performance. Additional figures on the sensitivity of $\eta^{-1}$
can be observed in Section A.3.

##  6 Conclusion

This work bridges an essential gap between implicit models and RvS methods. By
bridging this gap, practitioners will be able to enjoy the advantages of
implicit models to learn robotic skills from an offline dataset of varied
expertise. Overall, IRvS is conceptually simpler than previous explicit RvS
methods as it requires a single network and does not require quantizing the
return. In addition to being simpler, we have shown that IRvS can better
capture discontinuities, extrapolate, and handle higher dimensional problems
than explicit RvS. Additionally, we use insight from IRvS to derive a simpler
RvS algorithm where the target return is controlled via a temperature $\eta$
instead of being specified by hand or treated as a scalar. Furthermore, we
have demonstrated performance matching or exceeding state-of-the-results on
challenging offline RL benchmarks, particularly on the high-dimensional
manipulation tasks where IRvS greatly outperforms the baselines. Finally, we
have performed a sensitivity analysis where we have shown that our newly
introduced hyper-parameter $\eta^{-1}$ can be set to 1 on all tasks without
substantially degrading performance.

## References

  * Asmussen & Glynn (2007) Søren Asmussen and Peter W Glynn.  _Stochastic simulation: algorithms and analysis_ , volume 57.  Springer, 2007.
  * Chen et al. (2021) Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  _arXiv preprint arXiv:2106.01345_ , 2021.
  * Emmons et al. (2021) Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine.  Rvs: What is essential for offline rl via supervised learning?  _arXiv preprint arXiv:2112.10751_ , 2021.
  * Florence et al. (2021) Pete Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson.  Implicit behavioral cloning.  _arXiv preprint arXiv:2109.00137_ , 2021.
  * Freund & Haussler (1991) Yoav Freund and David Haussler.  Unsupervised learning of distributions on binary vectors using two layer networks.  _Advances in neural information processing systems_ , 4, 1991.
  * Fu et al. (2020) Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  _arXiv preprint arXiv:2004.07219_ , 2020.
  * Gupta et al. (2019) Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman.  Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning.  _arXiv preprint arXiv:1910.11956_ , 2019.
  * Haarnoja et al. (2017) Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine.  Reinforcement learning with deep energy-based policies.  In _International Conference on Machine Learning_ , pp. 1352–1361. PMLR, 2017.
  * Kostrikov et al. (2021) Ilya Kostrikov, Ashvin Nair, and Sergey Levine.  Offline reinforcement learning with implicit q-learning.  _arXiv preprint arXiv:2110.06169_ , 2021.
  * Kumar et al. (2019) Aviral Kumar, Xue Bin Peng, and Sergey Levine.  Reward-conditioned policies.  _arXiv preprint arXiv:1912.13465_ , 2019.
  * Kumar et al. (2020) Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.  Conservative q-learning for offline reinforcement learning.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Kumar et al. (2013) Vikash Kumar, Zhe Xu, and Emanuel Todorov.  Fast, strong and compliant pneumatic actuation for dexterous tendon-driven hands.  In _2013 IEEE international conference on robotics and automation_ , pp. 1512–1519. IEEE, 2013.
  * LeCun et al. (2006) Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang.  A tutorial on energy-based learning.  _Predicting structured data_ , 1(0), 2006.
  * Lee et al. (2022) Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, et al.  Multi-game decision transformers.  _arXiv preprint arXiv:2205.15241_ , 2022.
  * Levine (2018) Sergey Levine.  Reinforcement learning and control as probabilistic inference: Tutorial and review.  _arXiv preprint arXiv:1805.00909_ , 2018.
  * Levine et al. (2020) Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Pomerleau (1988) Dean A Pomerleau.  Alvinn: An autonomous land vehicle in a neural network.  _Advances in neural information processing systems_ , 1, 1988.
  * Rajeswaran et al. (2017) Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine.  Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.  _arXiv preprint arXiv:1709.10087_ , 2017.
  * Sallans & Hinton (2004) Brian Sallans and Geoffrey E Hinton.  Reinforcement learning with factored states and actions.  _The Journal of Machine Learning Research_ , 5:1063–1088, 2004.
  * Srivastava et al. (2019) Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Jaśkowski, and Jürgen Schmidhuber.  Training agents using upside-down reinforcement learning.  _arXiv preprint arXiv:1912.02877_ , 2019.
  * Sutton (1988) Richard S Sutton.  Learning to predict by the methods of temporal differences.  _Machine learning_ , 3(1):9–44, 1988.
  * Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ international conference on intelligent robots and systems_ , pp. 5026–5033. IEEE, 2012.
  * Welling & Teh (2011) Max Welling and Yee W Teh.  Bayesian learning via stochastic gradient langevin dynamics.  In _International Conference on Machine Learning_ , 2011.
  * Welling et al. (2004) Max Welling, Michal Rosen-Zvi, and Geoffrey E Hinton.  Exponential family harmoniums with an application to information retrieval.  _Advances in neural information processing systems_ , 17, 2004.

##  Appendix A Appendix

###  A.1 Stochastic Gradient Langevin Dynamics

Algorithm 1 Stochastic Gradient Langevin Dynamics

1:Initial state ${\mathbf{s}}$

2:Initial action
${\mathbf{a}}\sim{\mathcal{U}}({\mathbf{a}}_{\text{min}},{\mathbf{a}}_{\text{max}})$

3:Initial return $G\sim{\mathcal{U}}(-1,1)$

4:Learning rate $\alpha$

5:_# Density Estimation_

6:for $i\in\\{1,\ldots,100\\}$ do

7:
$g\leftarrow\nabla_{({\mathbf{a}},G)}E_{\theta}({\mathbf{s}},{\mathbf{a}},G)$

8: $u\leftarrow\text{clip}(\frac{1}{2}g+\epsilon,-0.5,0.5)$,
$\epsilon\sim{\mathcal{N}}(0,1)$

9: $({\mathbf{a}},G)\leftarrow({\mathbf{a}},G)-\alpha u$

10:end for

11:Return: $({\mathbf{s}},{\mathbf{a}},G)$

###  A.2 Hyper-parameters

For IRvS, all the hyper-parameters with the exception of the learning rate are
the same as the ones reported by Florence et al. (2021). For our
implementation of RvS, we closely followed the hyper-parameters reported by
Emmons et al. (2021).

Hyper-parameter | Chosen Value | Swept Value
---|---|---
EBM variant | Langevin |
batch size | 512 | 512
learning rate | 1e-3 | 1e-4, 5e-4, 1e-3
learning rate decay | 0.99 |
learning rate decay steps | 100 |
network size (width x depth) | 512x8 |
activation | ReLU | swish, leaky ReLU, ReLU
regularization | none | none, layer norm
layer type | spectral norm | regular, spectral norm
train counter-examples | 8 |
action boundary buffer | 0.05 |
gradient penalty | final step only |
gradient margin | 1.0 |
Langevin iterations | 100 |
Langevin learning rate init. | 0.5 |
Langevin learning rate final | 1e-5 |
Langevin polynomial decay power | 2 |
Langevin delta action clip | 0.5 |
Langevin noise scale | 0.5 |
Langevin 2nd iteration learning rate | 1e-5 |
Didactic example gradient steps | 2000 |
Navigation task gradient steps | 10000 |
Continuous control gradient steps | 100000 |
Table 2: Hyper parameter for IRvS Hyper-parameter | Chosen Value | Swept Value
---|---|---
batch size | 16384 | 16384
learning rate | 1e-3 | 1e-4, 5e-4, 1e-3
network size (width x depth) | 2048x2 |
activation | ReLU | swish, leaky ReLU, ReLU
Navigation task gradient steps | 10000 |
Continuous control gradient steps | 10000 |
Scales | 1e-5 | 1e-4, 1e-5, 1e-6
Number of atoms | 101 | 21, 51, 101
Table 3: Hyper parameter for RvS

CQL Results. All the CQL results are from Fu et al. (2020), except for the
mujoco results which are from Emmons et al. (2021), since the results were
from the previous simulator version.

IBC and filter-IBC. We have reproduced the results from Florence et al. (2021)
such that IBC, IRvS, and IBC w/ RWR share the same code-base.

RvS. We report the results RvS results from Emmons et al. (2021) as RvS, we
note that they condition on return of 110 for the medium-expert dataset, 90
for hopper, walker2d-medium-replay, and walker2d-medium, 60 for hopper-medium,
and 40 for halfcheetah-medium, medium-replay. Additionally, we reported the
RvS $\eta^{-1}=10$, where the action selection and return prediction are
performed as described in Equation 10 and Equation 9 respectively.

###  A.3 Sensitivy Analysis

Figure 6: Mujoco locomotion tasks. Hyper-parameter selection was performed on
these 3 seeds, and final results in Table 1 are reported on 3 new seeds.
Figure 7: Adroit manipulation tasks. Hyper-parameter selection was performed
on these 3 seeds, and final results in Table 1 are reported on 3 new seeds.
Figure 8: Kitchen manipulation tasks. Hyper-parameter selection was performed
on these 3 seeds, and final results in Table 1 are reported on 3 new seeds.


* [6] Docherty2016 S.SimonDocherty and D.DavidPym 20160 Simon Docherty David Pym Intuitionistic Layered Graph Logic, International Joint Conference on Automated Reasoning — IJCAR 16, Springer,2016, pp.469–486. docherty simon pym david 2016 intuitionistic layered graph logic
  * [7] DP2018 S.SimonDocherty and D.DavidPym 20180 Modular Tableaux Calculi for Separation Theories, Foundations of Software Science and Computation Structures — FOSSACS 21 (Christel Baier Ugo Dal Lago, editors), Springer International Publishing,2018, pp.441–458. docherty simon pym david 2018 modular tableaux calculi for separation theories
  * [8] Docherty2018 S.SimonDocherty and D. J.David J.Pym 2018a0 Simon Docherty David J. Pym A Stone-type Duality Theorem for Separation Logic via its Underlying Bunched Logics, Electronic Notes in Theoretical Computer Science, vol.336(2018), pp.101–118. docherty simon pym david j 2018 a stone type duality theorem for separation logic via its underlying bunched logics
  * [9] Docherty2018a S.SimonDocherty and D. J.David J.Pym 2018b0 Intuitionistic Layered Graph Logic: Semantics and Proof Theory, Logical Methods in Computer Science, vol.14(2018), pp.1–36. docherty simon pym david j 2018 intuitionistic layered graph logic semantics and proof theory
  * [10] Docherty2019a S.SimonDocherty and D. J.David J.Pym 20190 Stone-Type Dualities for Separation Logics, Logical Methods in Computer Science, vol.15(2019), pp.1–40. docherty simon pym david j 2019 stone type dualities for separation logics
  * [11] Dummett2000 M. A. E.Michael A. E.Dummett 20000 Michael A. E. Dummett Elements of Intuitionism, Oxford Logic Guides, vol. 39, Clarendon Press,2000. dummett michael a e 2000 elements of intuitionism
  * [12] Gabbay1998 D. M.Dov M.Gabbay 19980 Dov M. Gabbay Fibring Logics, Oxford Logic Guides, vol. 38, Clarendon Press,1998. gabbay dov m 1998 fibring logics
  * [13] Galmiche2019 D.DidierGalmiche, M.MichelMarti, and D.DanielMéry 20190 Didier Galmiche, Michel Marti, Daniel Méry Relating Labelled and Label-Free Bunched Calculi in BI Logic, Automated Reasoning with Analytic Tableaux and Related Methods — Tableaux 28 (Serenella Cerrito Andrei Popescu, editors), Springer International Publishing,2019, pp.130–146. galmiche didier marti michel mery daniel 2019 relating labelled and label free bunched calculi in bi logic
  * [14] Galmiche2005 D.DidierGalmiche, D.DanielMéry, and D.DavidPym 20050 Didier Galmiche, Daniel Méry, David Pym The Semantics of BI and Resource Tableaux, Mathematical Structures in Computer Science, vol.15(2005), no.6, p.1033–1088. galmiche didier mery daniel pym david 2005 the semantics of bi and resource tableaux
  * [15] Samsonschrift A. V.Alexander V.Gheorghiu, S.SimonDocherty, and D. J.David J.Pym 20210 Alexander V. Gheorghiu, Simon Docherty, David J. Pym Reductive Logic, Coalgebra, and Proof-search: A Perspective from Resource Semantics, Samson Abramsky on Logic and Structure in Computer Science and Beyond (A. Palmigiano M. Sadrzadeh, editors), Springer Outstanding Contributions to Logic Series, Springer,2021, to appear. gheorghiu alexander v docherty simon pym david j 2021 reductive logic coalgebra and proof search a perspective from resource semantics
  * [16] Gheorghiu2021 A. V.Alexander V.Gheorghiu and S.SoniaMarin 20210 Alexander V. Gheorghiu Sonia Marin Focused Proof-search in the Logic of Bunched Implications, Foundations of Software Science and Computation Structures - FOSSACS 24 (Stefan Kiefer Christine Tasson, editors), Lecture Notes in Computer Science, vol. 12650, Springer,2021, pp.247–267. gheorghiu alexander v marin sonia 2021 focused proof search in the logic of bunched implications
  * [17] Komendantskaya2010 E.EkaterinaKomendantskaya, G.GuyMcCusker, and J.JohnPower 20110 Ekaterina Komendantskaya, Guy McCusker, John Power Coalgebraic Semantics for Parallel Derivation Strategies in Logic Programming, International Conference on Algebraic Methodology and Software Technology — AMAST 13, vol. 13,2011, pp.111–127. komendantskaya ekaterina mccusker guy power john 2011 coalgebraic semantics for parallel derivation strategies in logic programming
  * [18] kowalski1979book R.RobertKowalski 19790 Robert Kowalski Logic for Problem Solving, Artificial Intelligence Series, vol. 7, Elsevier North Holland, Inc.,1979. kowalski robert 1979 logic for problem solving
  * [19] Kowalski1971 R.RobertKowalski and D.DonaldKuehner 19710 Robert Kowalski Donald Kuehner Linear Resolution with Selection Function, Artificial Intelligence, vol.2(1971), no.3, pp.227–260. kowalski robert kuehner donald 1971 linear resolution with selection function
  * [20] Kreisel G.GeorgKreisel 19580 Georg Kreisel Elementary Completeness Properties of Intuitionistic Logic with a Note on Negations of Prenex Formulae, , vol.23(1958), no.3, pp.317–330. kreisel georg 1958 elementary completeness properties of intuitionistic logic with a note on negations of prenex formulae
  * [21] Kripke1965 S. A.Saul A.Kripke 19650 Saul A. Kripke Semantical Analysis of Intuitionistic Logic I, Formal Systems and Recursive Functions (John N. Crossley Michael A. E. Dummett, editors), Studies in Logic and the Foundations of Mathematics, vol. 40, Elsevier,1965, pp.92–130. kripke saul a 1965 semantical analysis of intuitionistic logic i
  * [22] Milner1984 R.RobinMilner 19840 Robin Milner The Use of Machines to Assist in Rigorous Proof, Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences, vol.312(1984), no.1522, pp.411–422. milner robin 1984 the use of machines to assist in rigorous proof
  * [23] Negri2005 S.SaraNegri 20050 Sara Negri Proof Analysis in Modal Logic, Journal of Philosophical Logic, vol.34(2005), no.5, pp.507–544. negri sara 2005 proof analysis in modal logic
  * [24] OHearn1999 P. W.Peter W.O’Hearn and D. J.David J.Pym 19990 Peter W. O’Hearn David J. Pym The Logic of Bunched Implications, ,(1999), pp.215–244. ohearn peter w pym david j 1999 the logic of bunched implications
  * [25] Pym2002 D. J.David J.Pym 20020 David J. Pym The Semantics and Proof Theory of the Logic of Bunched Implications, Applied Logic Series, vol. 26, Springer, 012002. pym david j 2002 the semantics and proof theory of the logic of bunched implications
  * [26] Pym2004bi D. J.David J.Pym, P. W.Peter W.O’Hearn, and H.HongseokYang 20040 David J. Pym, Peter W. O’Hearn, Hongseok Yang Possible Worlds and Resources: The Semantics of BI, Theoretical Computer Science, vol.315(2004), no.1, p.257–305. pym david j ohearn peter w yang hongseok 2004 possible worlds and resources the semantics of bi
  * [27] Pym2004 D. J.David J.Pym and E.EikeRitter 20040 David J. Pym Eike Ritter Reductive Logic and Proof-search: Proof theory, Semantics, and Control, Oxford Logic Guides, vol. 45, Clarendon Press,2004. pym david j ritter eike 2004 reductive logic and proof search proof theory semantics and control
  * [28] Read1988 S.StephenRead 19880 Stephen Read Relevant Logic, Basil Blackwell,1988. read stephen 1988 relevant logic
  * [29] routley1973semantics R.RichardRoutley and R.RobertMeyer 19730 Richard Routley Robert Meyer The Semantics of Entailment, Studies in Logic and the Foundations of Mathematics, vol. 68, Elsevier,1973, pp.199–243. routley richard meyer robert 1973 the semantics of entailment
  * [30] schroeder P.PeterSchroeder-Heister 20180 Peter Schroeder-Heister Proof-Theoretic Semantics, The Stanford Encyclopedia of Philosophy (Edward N. Zalta, editor), Metaphysics Research Lab, Stanford University, Spring 2018 ed.,2018. schroeder heister peter 2018 proof theoretic semantics
  * [31] Gentzen M. E.M. E.Szabo 19691 M. E. Szabo (editor) The Collected Papers of Gerhard Gentzen, North-Holland Publishing Company,1969. szabo m e 1969 the collected papers of gerhard gentzen
  * [32] Troelstra A. S.A. S.Troelstra and H.H.Schwichtenberg 20000 A. S. Troelstra H. Schwichtenberg Basic Proof Theory, Cambridge Tracts in Theoretical Computer Science, Cambridge University Press,2000. troelstra a s schwichtenberg h 2000 basic proof theory
  * [33] urquhart1972semantics A.AlasdairUrquhart 19720 Alasdair Urquhart Semantics for Relevant Logics, , vol.37(1972), no.1, pp.159–169. urquhart alasdair 1972 semantics for relevant logics
  * [34]