[1,2]Duncan R. Lorimer

These authors contributed equally to this work.

These authors contributed equally to this work.

[1]Department of Physics & Astronomy, West Virginia University, White Hall,
Morgantown, 26501-6315, WV, USA

2]Center for Gravitational Waves and Cosmology, West Virginia University,
Chestnut Ridge Research Building, Morgantown, 26506, WV, USA

3]Centre for Astrophysics and Supercomputing, Swinburne University of
Technology, Hawthorn, 3122, Victoria, Australia

# The discovery and significance of fast radio bursts

<EMAIL_ADDRESS>Maura A. McLaughlin<EMAIL_ADDRESS>Matthew Bailes<EMAIL_ADDRESS>*  [  [

###### Abstract

In 2007 we were part of a team that discovered the so-called “Lorimer Burst”,
the first example of a new class of objects now known as fast radio bursts
(FRBs). These enigmatic events are only a few ms in duration and occur at
random locations on the sky at a rate of a few thousand per day. Several
thousand FRBs are currently known. While it is now well established that they
have a cosmological origin, and about 10% of all currently known sources have
been seen to exhibit multiple bursts, the origins of these enigmatic sources
are currently poorly understood. In this article, we review the discovery of
FRBs and present some of the highlights from the vast body of work by an
international community. Following a brief overview of the scale of the
visible Universe in §1, we describe the key moments in radio astronomy (§2)
that led up to the discovery of the Lorimer burst (§3). Early efforts to find
more FRBs are described in §4 which led to the discovery of the first
repeating source (§5). In §6, as we close out on the second decade of FRBs, we
outline some of the many open questions in the field and look ahead to the
coming years where many surprises are surely in store.

###### keywords:

fast radio bursts, compact objects, radio astronomy, transient astrophysics

##  1 Introduction to the Universe

For millennia humans have gazed upon the night sky and pondered what our place
is in the Universe and what laws govern it? The unaided human eye can only see
about 6000 stars but for many years their distances, lives and death were a
mystery. It wasn’t until astronomers applied mathematics to observational data
that our understanding of the Universe and the night sky started to progress
rapidly. Following the Copernican Revolution which inspired Tycho Brahe’s
precision measurements of the positions of the planets, Johannes Kepler used
Brahe’s work to understand their trajectories, and developed his famous three
laws of planetary motion. A further revolution in our understanding of the
Universe occurred when Galileo made the first recorded observations of the
night sky with a telescope in the early 1600s. His observations revealed that
Jupiter was orbited by four major moons, and this displaced the prevailing
view that Earth was the centre of the Universe. Four centuries of observations
have followed, and the astronomical and theoretical breakthroughs are
astonishing. The heavens have changed from a static background with pinpricks
of light and wandering planets to a dynamic and violent Universe where
phenomenal instruments are revealing the innermost workings of complex systems
involving stars, planets, black holes, white dwarfs and neutron stars.

Our Earth is only 40,000 km in circumference, but a million of them would
comfortably fit inside our Sun. Fortunately the Sun has almost constant
warmth, with an estimated remaining lifetime of about 5 billion years, and
this has enabled the evolution of life on Earth. But the Sun is a fairly
average star in the Universe, and will eventually puff off its outer layers
when it becomes a red giant leaving behind a white dwarf, the ashes of its
core. Some stars are much more massive than the Sun and have more violent
deaths. Rigel, for instance, is about 20 times the mass of the Sun and will
die in a violent supernova explosion, leaving behind a neutron star. Neutron
stars are only about 10 km in radius but weigh over half a million Earth
masses. A teaspoon of a neutron star material weighs more than the whole of
humanity. Even larger stars collapse to form black holes, and these can be in
excess of 50 solar masses. Black holes are so dense that not even light can
escape their extreme gravity.

In 1054 AD Chinese astronomers witnessed the supernova explosion that created
the Crab Nebula when a massive star like Rigel died. For a few weeks the
remnant was so bright it could be seen during the day. At the heart of the
nebula was a neutron star, the famous Crab pulsar. It rotates every 33 ms and
emits radiation all across the electromagnetic spectrum, from radio waves to
gamma-rays.

The Milky Way is full of the remnants of exploded stars, and comprises $\sim
400$ billion suns, as well as gas and dust. But the Milky Way is only one
galaxy in the Universe, which is thought to contain many 10s–100s of billions
of galaxies of different sizes. Our Milky Way has a pancake-like structure
with a central bulge and bar, and spiral arms. It is very flat in one
dimension, and extends for something like 100,000 light years in diameter. Its
halo is sparsely populated by stars that have been ejected from the disk and
is mainly composed of older stars. It is referred to as a spiral galaxy.

One of our closest neighbours is the Andromeda galaxy, similar in shape and
size to the Milky Way. Other galaxies lacked the spin to collapse into spiral
galaxies, and have a more ellipsoidal shape. Their stars tend to be “red and
dead”, as their star formation era has long-since passed. Astronomers call
these galaxies “ellipticals”. Although less luminous, smaller fragments of
galaxy formation abound, and our two nearest neighbours are known as the
Magellanic Clouds. These small, irregular galaxies often exhibit a haphazard
shape, are still producing new stars and orbit the Milky Way every billion
years or so. They are archetypal “irregular” galaxies.

Astronomers have mapped the distribution of galaxies in the wider universe and
find that they are consistent with numerical simulations of a once very smooth
universe that has both expanded and had gas clouds collapse to form galaxies.
The galaxies cluster along filaments and form groups. Sometimes they collide,
triggering new waves of star formation. At the cores of galaxies, supermassive
black holes accrete matter and spew forth relativistic particles in powerful
jets. These are known as quasars [61].

Between the galaxies the density of atoms drops to only about one per cubic
metre, over 10,000 times less dense than that of galaxies. Light and radio
waves traverse the enormous distances across the Universe with a small
probability of interaction. This property has enabled astronomers to see the
Universe age by looking back in time. On its long journey, visible light
largely ignores the ionised gas between the galaxies, making its composition
difficult to study. Modern astrophysics is driven by our desire to learn about
what classes of objects exist, such as planets, white dwarfs, stars, quasars,
neutron stars and black holes, but also how they live and die, and how they
can be detected. Astronomers also attempt to measure the mass and composition
of the Universe, its age and dimension, as well as what physical laws are at
play. It turns out that fast radio bursts have an important role to play in
developing our broader understanding of the cosmos.

##  2 The transient radio sky

Although optical astronomy has existed since humans first gazed at the
heavens, radio astronomy is less than 100 years old. The first radio
telescopes were not sensitive to short timescale phenomena; they focused on
making maps of the radio sky that were unchanging and static. With the
discovery of quasars in 1964, however, that changed. Because the angular sized
of quasars are so small, quasars are observed to “scintillate”, or twinkle as
their radio waves travel through interstellar gas. This effect, due to
diffraction of the waves through the interstellar medium, causes their radio
brightness to fluctuate with time.

Figure 1: Pen chart recording showing the detection of pulses from the
original pulsar B1919+21 first seen during the Cambridge survey [33].

In 1967, Jocelyn Bell Burnell was a graduate student working with Prof.
Anthony Hewish at Cambridge University in the UK. Her thesis project was to
study this phenomenon of quasar scintillation [6]. To do this, she used a
chart recorder that recorded the intensity of radio emission over the sky to
search for the types of variations expected from scintillating quasars. One
day, Bell Burnell saw a signal that occurred on a much shorter timescale than
expected for quasar scintillation. It also did not look like human-made radio
frequency interference, and it appeared at the same local sidereal time each
day, meaning that it was coming from space. Soon, Bell Burnell and Hewish
found that the signal consisted of regularly spaced pulses, with a separation
of exactly 1.337 seconds, as shown in Figure 1. This short timescale meant
that the signal must be coming from a very small object. Bell Burnell and
Hewish jokingly nicknamed it “LGM 1”, or “Little Green Man 1”, as this period
was too short to come from anything like a normal star and could have been
from aliens [33].

Soon Bell Burnell and Hewish found more of these objects, in different parts
of the sky, confirming that they were a new class of astronomical sources.
Following the suggestion of a journalist, these “pulsating radio sources” were
henceforth referred to as pulsars. Not long after, Thomas Gold and Franco
Pacini independently suggested that these pulses were neutron stars born in
the supernova explosions of massive stars. Soon thereafter, when the 33-ms
Crab pulsar was discovered at the centre of the Crab supernova remnant in 1968
[65] and the 89-ms pulsar was found at the centre of the Vela supernova
remnant [44], this theory was confirmed.

Pulsars are fascinating objects with extreme properties. Because of the
conservation of angular momentum, they rotate very rapidly, with spin periods
ranging from 1.4 milliseconds to tens of seconds. They also have extremely
high magnetic field strengths of $10^{8}$ to $10^{14}$ Gauss, trillions of
times higher than the magnetic field of the Earth. They are extremely dense,
with masses up to twice as high as the Sun but with radii of only 10 km. This
is an equivalent density to taking all the people on Earth and squeezing them
into a thimble! Pulsars’ rapid spin periods and high magnetic fields lead to
acceleration of particles above their magnetic poles, and this produces beamed
radio emission. Pulsars therefore act like interstellar lighthouses; every
time the pulsar beam swings past our line of sight, we detect a pulse.

One very important property of pulsar emission is that it is dispersed. This
means that the lower frequencies of the radio pulse arrive later than the
higher frequencies. The time delay, $\Delta t$, between a pulse at a high
frequency, $\nu_{\rm hi}$, compared to a lower frequency, $\nu_{\rm lo}$, is
given approximately by

$\Delta t\simeq 4150~{}{\rm s}\times\left[\left(\frac{\nu_{\rm lo}}{\rm
MHz}\right)^{-2}-\left(\frac{\nu_{\rm hi}}{\rm
MHz}\right)^{-2}\right]\times\left(\frac{{\rm DM}}{{\rm cm}^{-3}~{}{\rm
pc}}\right),$

where the “dispersion measure” DM is the integrated column density of free
electrons along the line of sight to the source. Dispersion occurs due to
interactions of the radio photons with electrons and other charged particles
in the interstellar medium, the gas in between the pulsars and Earth. On one
hand, dispersion is a nuisance, as it means we must correct for this sweep
before searching for pulsars in radio data. On the other hand, it is a
valuable tool, as the amount of this dispersive sweep, coupled with a model
for Galactic electron density, gives us a handle on how far away a pulsar is.
We have estimated distances to many pulsars through this method.

After the discovery of the first pulsars, astronomers soon realised that
searches that were sensitive to a pulsar’s time-averaged emission were much
more efficient and sensitive that those that searched for individual, single
pulses. In the 1970s, searches that used Fourier transforms to measure excess
power at particular rotation frequencies became commonplace [15], and
astronomers generally stopped searching data for the single, dispersed pulses
through which the first few pulsars were discovered. Despite this, one special
class of pulsars could be detected with higher sensitivity through their
individual single pulses - giant pulsing pulsars. These pulsars, of which the
Crab pulsar is the first-discovered and most well-known example, emit single
pulses that are occasionally 100s or even 1000s of times brighter than the
average pulse. These pulses are very narrow and occur randomly in time, as
shown in Figure 2.

Figure 2: This observation, carried out in the 5–6 GHz radio frequency band,
is among the highest time resolution views of emission seen in the radio sky
so far [32]. These nanosecond scale pulses are being emitted by the Crab
pulsar. Credit: Tim Hankins.

The discovery of the Crab pulsar using the 300 ft Green Bank telescope through
its giant pulses [65] and subsequent identification of the 33 ms period using
the 1000 ft Arecibo telescope [26] generated significant interest in the
prospect of finding extragalactic sources similar to the Crab. Following
prophetic remarks by Bondi [13], Colgate and Noerdlinger [25] and Cavallo and
Ventura [17] discussed some ideas about emission mechanisms for such sources.
Also on the theoretical side, Ginzburg [29] highlighted the prospects for
using extragalactic sources to map out the electron content of intergalactic
space in a similar way to what was then being done with the growing sample of
pulsars to map out the interstellar medium of the Milky Way [49].

Following a number of attempts to find extragalactic radio pulses at radio
wavelengths [24, 37, 34, 16, 57, 27], Linscott and Erkes [46] announced the
discovery of radio pulses detected from the direction of the giant elliptical
galaxy M87. However, in spite of a number of follow-up searches [67, 31, 52,
66], no further radio pulses were ever seen from M87. While it was never
conclusively determined whether the events seen by Linscott and Erkes were
from highly variable sources in M87 or some form of interference, we note that
the galaxy is strongly suspected to contain detectable sources [43]. Large-
area searches for extragalactic radio pulses in the intervening years
continued to yield null results [1, 9, 38]. Meanwhile, Palmer [56] had
emphasized the importance of dispersion in the detection of cosmological radio
transients and searches for ‘prompt’ radio emission from gamma-ray bursts were
being carried out [see, for example, 30, 4]

As a graduate student at Cornell, one of the authors (Maura McLaughlin) and
her advisor Jim Cordes had a goal to detect pulsars in other galaxies. At that
time, nearly all of the pulsars known were in the Milky Way, with just a
handful in the Small and Large Magellanic Clouds, nearby satellite galaxies to
the Milky Way. McLaughlin and Cordes [53] realised that probably the only way
to detect pulsars in galaxies further than these was to search for extremely
bright giant pulses, as opposed to using the Fourier transform searches that
were almost exclusively used for searches at the time. They developed
methodology and code that they applied to a search to the local group galaxy
M33 (the Triangulum galaxy), which is roughly 1 Mpc away. While they found no
bursts, they had developed the tools that could be used to carry out other
searches for single pulses.

A breakthrough in the field came shortly after the Arecibo searches when,
applying this single-pulse code to data from the Parkes Multibeam Pulsar
Survey, McLaughlin et al [54] found roughly a dozen pulsars that were not
detectable in previous Fourier transform searches. These objects, which the
authors nicknamed “rotating radio transients” (RRATs), emitted a very small
number of detectable pulses over the course of the 35-min search observations,
meaning they could not be detected through a Fourier transform search but
could be through a search for single pulses. After the discovery of the RRATs,
pulsar astronomers world-wide began to use the single-pulse search technique
routinely as part of pulsar searches. Despite their sporadicity, the
properties of the RRAT single pulses were not significantly different from the
properties of normal pulsar single pulses. However, looking at the properties
of these objects in this way raised a tantalising question: what other types
of objects could exist in this newly revealed single-pulse phase space?

##  3 Discovery of the Lorimer Burst

While all the work that led up to this point prepared us to find the first
fast radio burst, it took the act of applying these tools analysing archival
datasets to make the discovery. Shortly after two of the authors (Maura
McLaughlin and Duncan Lorimer) arrived at West Virginia University (WVU), as
assistant professors of physics in 2006, they began to work with students on a
variety of projects. One of these was to look through recently published data
from Parkes looking at the Magellanic Clouds [50]. The Parkes data had already
been extensively searched for periodic pulses from pulsars, but represented a
self-contained experiment as a place to look for bright individual pulses. At
that time, computational resources were sufficiently developed for it to be
feasible to process the data on a desktop computer over a period of a few
months. This was adequate to allow then undergraduate student Ash Narkevic to
manually inspect the results of the data analysis at a pace that worked well
with their studies within an academic year. The diagnostic plots were
essentially representations of the data as a function of dispersion measure,
DM, versus time. It was Ash’s task to look for signals that were statistically
significant and occurred at non-zero DM. Fortunately the Parkes data set
contained a number of known pulsars in the Magellanic Clouds that were bright
enough such that they could be seen as individual pulses as shown in Figure 3.

Figure 3: Single-pulse search output showing individual pulses from PSR
B0529–66 in the Large Magellanic cloud observed in the 1.2–1.5 GHz frequency
band. In this visualisation, the pulses are seen as the discrete vertical
stripes at a constant DM of 100 cm-3 pc.

Ash would meet with Duncan for an hour once a week to review findings from the
previous week’s processing. Most of the signals were relatively
straightforward to categorise: a form of interference, an already known pulsar
in the data, or something that perhaps was right at the edge of the
sensitivity threshold, and too faint to confidently assert the presence of a
signal. One day, Ash found something that didn’t fit these categories. The
discovery plot is shown in Figure 4.

Figure 4: Single-pulse search output showing the Lorimer burst which appears
as the bright localised signal at a DM of 375 cm-3 pc. These data were
collected in the 1.2–1.5 GHz frequency band.

As can be seen, the presence of a bright individual pulse is seen at a DM of
375 cm-3 pc at around 1750 s after the start of the observation. The only
other signals present in this graph are the vertical stripes seen at 7400 and
8000 s; the vertical stripes seen for these are the hallmarks of strong
locally generated interference. The new signal was very well defined in DM
space, as shown by the increasing and decreasing widths of the symbols in
Figure 4. Further inspection of this signal showed that the sky position was
significantly offset from the Small Magellanic Cloud. This observation had
been collected as part of an effort to cover both the clouds using the Parkes
multibeam receiver. This 13-pixel radio camera allows users to observe larger
fields than possible using a single beam that was available to most telescopes
back at that time. Because of the way the field of the Small Magellanic Cloud
was covered, this particular pointing was looking mostly away from the cloud
itself and into the distant Universe. The DM that we observed for this pulse
about 10 times the expected value from the Milky Way in that direction. The
pulse appeared to be firmly extragalactic in origin.

These initial checks also showed the presence of the signal in one other pixel
in the multibeam system. During a visit to Parkes in Spring 2007, Duncan
shared this discovery with the third author of this paper (Matthew Bailes).
Upon even further scrutiny while they were together, it was realised that the
signal was strongly present in one further beam of the system. It was
originally not registered by the code as the pulse was so bright that it
saturated the digitizer levels in the system and the data preparation software
had erroneously flagged that particular signal as interference! All in all,
the pulse’s signal can be detected in 4 out of 13 of the multibeam pixels (one
quite weakly) and it had an anomalously high DM. Visualisations of the pulse
in terms of frequency versus time, along with an example pulse from a known
pulsar with a similar DM are shown in Figure 5.

Figure 5: Waterfall plots showing radio frequency versus time for the bright
pulsar PSR J1707–4053 (left) and the Lorimer burst seen in two beams of the
multibeam receiver (centre and right).

The dispersion of the pulse provides a strong constraint on its distance.
After subtracting off the expected DM from the Milky Way, there is about 330
cm-3 pc of “extragalactic dispersion” produced mostly by the free electrons in
the intergalactic medium. Based on results from other astronomical surveys, it
is known that the density of the IGM is roughly one electron per cubic metre.
A back-of-the-envelope calculation (DM divided by mean density) then gives an
implied distance of 330 Mpc. Using slightly more sophisticated calculations
[47], we suggested that the distance could be as high as 1 Gpc. It appeared
that the pulse originated from a source at cosmological distances. In addition
to the observed dispersion, we also were able to see an evolution of the pulse
width with frequency, with the lower-frequency components of the pulse being
more scattered than their higher-frequency counterparts. The morphology of the
pulse was consistent with its propagation through a turbulent ionised medium.

Figure 6: Individual pulses from a variety of different coherent radio sources
displayed as luminosity versus pulse width times observing frequency. The
highly luminous nature of the Lorimer burst made it clearly anomalous at the
time of its discovery. Figure credit: Chris Flynn and Manisha Caleb.

Given such a high implied distance, the inferred luminosity of the new source,
colloquially known as the “Lorimer burst”, was phenomenally bright. As shown
in Figure 6, it was something like 10 to 12 orders of magnitude brighter than
anything known at the time. The Lorimer burst was apparently the prototype of
a new class of distant radio transient source. Very much like the discovery of
pulsars some 40 years earlier, this serendipitous discovery had a wealth of
implications some of which were outlined in the discovery paper [47].

Beyond its anomalous luminosity, we realised that (like pulsars) the short
pulse duration implied a coherent emission mechanism with a very high
brightness temperature. While considerably uncertain due to the lack of a
direct measurement of distance to this source, we inferred source energetics
of the order of $10^{33}$ J. Light travel time arguments limited the size of
the emitting region to be less than 1500 km. Not only was the source very
small, bright and very far away, our observations to date implied an all-sky
rate of hundreds of similar sources per day over the sky. Finally, in spite of
over 90 hr of follow-up, no further bursts were seen from this location.

Taken together, these lines of evidence suggested a source mechanism that was
catastrophic in nature — perhaps a neutron star inspiral, supernova, or gamma-
ray burst. With statistics of one object, we realised that these ideas were
speculative at best. Nevertheless, in our paper and accompanying cover letter
to the editor of Science, we boldly posited this source as a prototype and
eagerly encouraged further searches of other archival datasets by ourselves
and other groups.

The months following the discovery publication in Science were incredibly
exciting. There was a lot of interest in both the popular press and the
astrophysical community. Naturally, many questions were asked. Why was the
source so bright? Where was the more numerous population of fainter sources?
In spite of our best arguments in favour of an astrophysical origin, could the
source be due to something else that we had not contemplated? A nearby
anomalously dispersed object, or some local interference? Only further follow-
up observations could really answer these and other questions.

##  4 Early follow-ups

Radio astronomers soon pored through old radio surveys of the sky to look for
further examples of the Lorimer Burst, and also commenced new surveys looking
further into the universe for them. We had observed the location of the
Lorimer Burst for 90 hours, but not seen any repeat bursts. New surveys failed
to find other examples, and there was some alarm when one of Matthew’s PhD
students (Sarah Burke-Spolaor, now a faculty member at WVU) found some sources
of interference [14] at the Parkes site with dispersion characteristics
bearing some similarities to the Lorimer Burst. These were dubbed “Perytons”
named after the mythical Greek elk that casts a human shadow. These mysterious
sources cast doubts on the celestial nature of the Lorimer Burst. The Perytons
were imperfectly dispersed signals, clearly close to the telescope site, and
worryingly close to the Lorimer Burst’s DM of 375 cm-3 pc.

For over five years no further extragalactic radio bursts were identified, and
the community started to express scepticism as to their reality. Hopes
resurfaced when a team led by Evan Keane [39] found a putative burst near the
plane of our Galaxy, but it was unclear whether it was part of our own
galaxy’s stellar population or not due to our imperfect knowledge of the
density of the interstellar medium along the Galactic plane.

Just prior to the discovery of the Lorimer Burst, the High Time Resolution
Universe (HTRU) collaboration had formed with the aim of using new digital
instrumentation to more effectively search for millisecond pulsars in the
Milky Way with the Parkes radio telescope [40]. The new digital
instrumentation had about 8 times the frequency resolution of the old Parkes
filterbanks, higher time resolution and up to 8-bit resolution.

Dan Thornton was a PhD student under the supervision of Ben Stappers at the
University of Manchester [69] and was assigned the search for Lorimer Bursts
by the collaboration. His initial efforts were proving frustrating, as radio
interference was masking any putative bursts. His advisor suggested he just
set a high detection threshold, and, remarkably, a 50$\sigma$ burst was
detected near a DM of 1000 cm-3 pc well away from the Galactic plane. The new
digital instrumentation allowed very precise measurement of the burst’s
dispersion and scattering time scale power law indices, which were very near
the expected values of –2 (dispersion) and –4 (scattering) expected for
celestial sources. Spurred on by his discovery, Thornton soon found another
three examples, all with different dispersion measures, all far from the
Galactic plane, and all with vertical components of dispersion measures well
in excess of any known radio pulsar. The cosmological population of radio
bursts had been discovered [70]. The HTRU collaboration suggested the name
“fast radio bursts” or FRBs, a name that has stuck to this day. These
discoveries spurred many different teams into action. Laura Spitler and
collaborators soon discovered [63] the first FRB with the Arecibo telescope in
archival data. It had occurred back in 2012 and was dubbed FRB 121102 (on UTC
date 2 Nov 2012). Matthew was greatly relieved to see a burst at another
facility other than Parkes, and when she previewed its waterfall (dispersion)
plot at Manchester he felt like leaping on stage and hugging her for joy!

Parkes remained a fruitful source of FRBs, the HTRU collaboration published
another 5, one with a DM beyond 1600 cm-3 pc that appeared to consist of two
components, and other groups started to find them both at Parkes and other
facilities like the Green Bank Telescope [51]. The GBT burst possessed a high
rotation measure, consistent with it emerging from a magnetised environment.

At the Molonglo Observatory, Bailes et al [3] developed machine learning
algorithms to identify bursts in real time and enable post facto coherent
dedispersion that revealed that FRBs could consist of multiple narrow
components, with timescales less than a few 10s of microseconds [28]. These
small timescales meant that the source size could only extend tens of
kilometres and pointed to an origin in the magnetospheres of neutron stars.
One Parkes FRB had a dispersion measure of over 2000 cm-3 pc and was
potentially from a source so distant that the Universe was much less than half
of its current age when it was emitted. The field was however transformed when
routine follow-up observations of the only known Arecibo burst were conducted
in 2015.

##  5 The repeating FRB — a “minor point of interest”

All of the FRBs discovered up until the time of the Arecibo FRB had been
subject to many hours of follow-up observations with Parkes and other
telescopes, with no repeated bursts found. It was becoming almost generally
accepted that these were “one-off” phenomena when additional bursts were
discovered from FRB 121102. A Canadian graduate student working under the
supervision of Vicky Kaspi at McGill University on the Arecibo survey named
Paul Scholz jokingly titled the email announcing this result in November 2015
as “A minor point of interest regarding the Spitler burst”. These bursts
occurred at the same DM and sky position as the original FRB, and so there was
no doubt they were associated. As more and more bursts were detected from this
FRB, it was clear that they showed an astonishing diversity of properties.
Some were narrow, some broader. Some peaked in brightness at high frequencies,
and some at low frequencies. Some had a single pulse component, while some
appeared to have multiple components [64]. Further multi-wavelength
observations of this source soon followed [62] and it was shown that the
arrival time distribution of the radio bursts was highly non-Poissonian.

The repetition of this FRB made it possible to observe its position with an
interferometer to localise it much more precisely than possible with a single-
dish telescope like Arecibo. Observations with the VLA by Chatterjee et al
[18] revealed that the burst was associated with a galaxy. This was very
important for several reasons. First, these bursts were definitely
astrophysical and coming from stellar populations. Second, as shown in Fig. 7,
the galaxy was detected optically by Tendulkar et al [68], providing a
measurement of the redshift, $z=0.19$. Hence, for the first time the DM of a
burst could be used to constrain the electron density of the intergalactic
medium. Third, the kind of galaxy it came from provided important clues to its
origin. In this case, FRB 121102 originated in a dwarf galaxy with a high star
formation rate, indicating that the burst came from stellar populations such
as neutron stars. It was also found that the bursts from FRB 121102 had one of
the highest rotation measures ever observed [55] which means it came from a
very magnetic, dense environment.

Figure 7: Follow-up of FRB 121102 showing the identification of a host galaxy
at $z=0.19$ (inset).

After the discovery of this FRB, each year brought more and more FRB
discoveries, with this number exploding once the CHIME telescope in Canada
came online in 2018 [19]. This telescope began discovering FRBs at a very high
rate, and there are currently 536 CHIME-discovered FRBs in the public domain
[22] with several thousand more to be released soon. As more FRBs began to be
detected, more puzzles emerged. Although around 10% of all currently known
FRBs exhibit repeat bursts, an analysis of the CHIME sample shows that the
true repeater fraction is probably closer to aroud 3% [23]. One of the
repeating FRBs detected with CHIME appeared to show a 16-day periodicity in
its active cycle [20]. More recently, there are hints that the original
repeater FRB 121102 demonstrates a roughly 180 day periodicity [59]. Periods
of this timescale could be indicative of binary motion, and several models
have been proposed. One attractive model involves modulation as a neutron star
travels through the wind of its stellar companion [2].

##  6 Recent breakthroughs and the future

The current state of the field of FRBs is very active. Barely a day goes by
without a new paper appearing discussing either a new result, or trying to
make sense of some of the many observational breakthroughs that are being
presented. Among the recent breakthroughs is the discovery of an FRB-like
pulse from a known magnetar in our Galaxy, SGR 1935+2154. This incredibly
bright pulse was spotted by CHIME [21] and subsequently also by the STARE2
experiment [12] and quickly localised to a magnetar which was known to be
undergoing an enhanced period of activity at the time (April 28, 2020). With a
peak flux density of over 1 MJy, the equivalent luminosity of this pulse was
several thousand times brighter than anything observed from the Crab pulsar,
and only a factor of 30 fainter than a typical cosmological FRB pulse. It
therefore appears to make a strong case for magnetars as plausible sources of
FRBs. Magnetars have now long been known as sources of energetic emission and
the 2021 Shaw Astronomy
Prize111https://www.shawprize.org/laureates/2021-astronomy Laureates, Prof.
Vicky Kaspi and Prof. Chryssa Kouveliotou, were recognized for their
contributions to our understanding of the magnetar population in the Galaxy.

A simple picture in which all FRBs originate from young magnetars, however,
does not seem to be sufficient to explain the diversity of environments in
which FRBs are being found. The sheer number of FRB discoveries being made by
CHIME (around 3 per day on average) is leading to a variety of FRB locations.
While we see one source located in a spiral arm of a nearby galaxy, a very
intriguing discovery is FRB 20200120E in the globular cluster associated with
the nearby galaxy M81 [11, 41]. Globular clusters are among the oldest stellar
populations, and regions where young magnetars are not expected due to the
lack of any recent star formation activity. As recently pointed out by Kremer
et al [42], magnetars and other young neutron stars could form in these
environments through the mergers of white dwarf binaries. As discussed by
Kremer et al [42], this could also explain why our own Galaxy’s globular
cluster system harbours a few young neutron stars which are otherwise very
difficult to reconcile with the older population of millisecond pulsars
currently seen. Further observational and theoretical studies in this area are
eagerly anticipated in the future.

At the time of writing, March 2024, almost 50 FRBs have been conclusively
associated with host galaxies. To date, there appears to be no statistically
significant difference between the hosts of repeating versus non-repeating
FRBs. In general, FRBs are found in galaxies where there are at least moderate
amounts of star formation. They tend not to be found in the so-called “red and
dead” galaxies where star formation has ceased [10]. When tracked as a
function of redshift, FRBs generally seem to trace the evolution of star
formation in galaxies, at least out to redshifts around 0.5. A statistically
significant sample at higher redshifts is currently lacking, with the highest
redshift FRB to date being found at $z=1$ [60].

One of the amazing aspects of FRB studies is that they act as probes of the
intervening matter, even if one does not fully understand their source
populations, which we currently do not! All that is needed is a sample of FRBs
with well-determined redshifts. From this, it is possible to effectively count
the number of electrons along different sight lines in the Universe and
measure the electron density directly. As one journalist, Joe Palca, points
out: very much in the same way as we use dipsticks to keep track of the oil
content of our cars, astronomers can in effect use FRBs as “intergalactic
dipsticks” to measure the electrons in the universe. Although conceptually
simple, this turns out to be a very challenging exercise. The first team to do
it was led by J-P Macquart using a sample of FRBs that was very carefully
obtained using the ASKAP telescope [48]. This required the coordination of a
large team of astronomers to get both the radio measurements and optical
redshifts necessary. The result, shown in Figure 8, is now known as the
Macquart relation, in honour of J-P’s memory. J-P passed away far too soon,
but left the astronomical community with an amazing result first anticipated
by Ginzburg [29].

Figure 8: The Macquart relation, as established for the first FRBs with robust
host galaxy identifications and redshifts. The correlation between DM and
redshift provides a compelling detection of baryons in the intergalactic
medium. Credit: J-P Macquart.

The scatter around the Macquart relation contains information about the
turbulence of the interstellar gas around galaxy clusters and is now being
studied in great detail [5]. We anticipate a significant increase in the
sample of FRBs with measured redshifts in the coming years.

Among the many exciting results from individual sources that are now coming to
light are the detections of thousands of bursts from selected FRB repeaters,
in particular using FAST (Five Hundred Metre Aperture Spherical Telescope) in
China [45]. One very recent result by the FAST group shows that the bright
FRB-like pulses from the Galactic magnetar appear at random rotational phases
of the neutron star [71]. The fainter radio pulsar-like pulses are seen only
in a well defined rotational phase window. This potentially explains why, to
date, no repeating FRB has shown a periodicity that is conclusively connected
to a pulsar-like rotation.

One of the great debates that is currently ongoing in the FRB community is
whether all FRBs repeat or not? As shown in the first CHIME catalogue, the
morphological appearance of repeating versus one-off FRBs does appear to be
very different: repeaters occupy only a fraction of the observable band, and
produce broader pulses on average compared to the one of sources which tend to
be narrower pulses that are broad band in nature [58]. Efforts to unify this
dichotomy as some sort of geometrical effect (as has been proposed, for
example, with active galactic nuclei), are challenging. All three of the
authors of this paper have different opinions about the nature of repeating
versus non-repeating FRBs!

Regardless of all the mysteries that are still in play, one thing is very
clear: as we look at the current state of the transient sky in terms of the
“luminosity — pulse duration” diagram shown in Fig. 9, we see clear groupings
of sources, but also lots of gaps in the parameter space.

Figure 9: Individual pulses from a variety of different sources displayed as
luminosity versus pulse width times observing frequency. The Lorimer burst is
no longer an anomalously bright object when compared to the other FRBs known.
As we note in the text, there is still much parameter space to be explored.
Further exciting discoveries in the transient radio sky are anticipated.
Credit: Chris Flynn and Manisha Caleb.

These gaps indicate opportunities and challenges to the FRB community. We are
not, for example, currently able to resolve pulses well enough to rule out a
class of extremely short duration FRBs (dubbed ultrafast FRBs by some).
Following predictions about a substantial population of ultra-long period
Galactic magnetars [7], and the fact that most searches to date are less
sensitive to long-duration pulses, examples of long period (minutes or more)
transients in the Galaxy are now being found [35, 36]. It is important to
recognize these current shortcomings and opportunities as we plan future
experiments. Further discussion of this emerging population can be found in
Beniamini et al [8].

In closing, the future of FRBs is very exciting thanks to all of the
developments that have taken place since our discovery in 2007. As mentioned
above, many of the most recent results are challenging our understanding of
the population which leads to a lot of open questions. We feel very fortunate
to have played a role in sparking this whole community and continue to enjoy
participating in it.

Acknowledgements

We are grateful to a number of individuals for their roles in this work and
organisations for supporting it over the last 17 years. Special thanks to
Froney Crawford and Ash Narkevic for their contributions to the discovery. To
the Shaw Foundation, and to those that nominate awardees and serve on our
committees, thank you for highlighting the significance of the discovery of
FRBs. Matthew has received significant support from Swinburne University of
Technology and the Australian Research Council. Maura and Duncan are grateful
for the sustained support from WVU, the National Science Foundation, the
Research Corporation for Science Advancement and the West Virginia NASA Space
Grant Consortium as well as the West Virginia Higher Education Policy
Commission. We are grateful to our colleagues at the Australia Telescope
National Facility, and the National Radio Astronomy Observatory for their
support. Duncan would like to thank Enrico Massaro and Frank Drake for
alerting him to some of the key references and developments concerning
searches for extragalactic radio pulses prior to the discovery of FRBs and Paz
Beniamini for highlighting the importance of long-period pulsars in the
Galaxy. We are very grateful to Fabio Santos and Elias Brinks at Springer for
their patience as we put this manuscript together, and to the reviewer, Vicky
Kaspi, for very useful feedback to an earlier draft of the manuscript.
Finally, huge thanks to all of our students and colleagues who have been a
pleasure to work with and a constant source of inspiration.

## References

  *   * Amy et al [1989] Amy SW, Large MI, Vaughan AE (1989) A search for transient events at 843 MHz. PASA 8(2):172–175. 10.1017/S1323358000023262
  * Bailes [2022] Bailes M (2022) The discovery and scientific potential of fast radio bursts. Science 378(6620):abj3043. 10.1126/science.abj3043, arXiv:2211.06048 [astro-ph.HE]
  * Bailes et al [2017] Bailes M, Jameson A, Flynn C, et al (2017) The UTMOST: A Hybrid Digital Signal Processor Transforms the Molonglo Observatory Synthesis Telescope. PASA 34:e045. 10.1017/pasa.2017.39, arXiv:1708.09619 [astro-ph.IM]
  * Balsano [1999] Balsano RJ (1999) A search for radio emission coincident with gamma-ray bursts. PhD thesis, Princeton University, New Jersey
  * Baptista et al [2023] Baptista J, Prochaska JX, Mannings AG, et al (2023) Measuring the Variance of the Macquart Relation in z-DM Modeling. arXiv e-prints arXiv:2305.07022. 10.48550/arXiv.2305.07022, arXiv:2305.07022 [astro-ph.CO]
  * Bell Burnell [1969] Bell Burnell SJ (1969) The measurement of radio source diameters using a diffraction method. PhD thesis, University of Cambridge, UK
  * Beniamini et al [2020] Beniamini P, Wadiasingh Z, Metzger BD (2020) Periodicity in recurrent fast radio bursts and the origin of ultralong period magnetars. MNRAS 496(3):3390–3401. 10.1093/mnras/staa1783, arXiv:2003.12509 [astro-ph.HE]
  * Beniamini et al [2023] Beniamini P, Wadiasingh Z, Hare J, et al (2023) Evidence for an abundant old population of Galactic ultra-long period magnetars and implications for fast radio bursts. MNRAS 520(2):1872–1894. 10.1093/mnras/stad208, arXiv:2210.09323 [astro-ph.HE]
  * Benz and Paesold [1998] Benz AO, Paesold G (1998) A search for prompt radio emission of gamma-ray bursts. AAP 329:61–67
  * Bhandari et al [2022] Bhandari S, Heintz KE, Aggarwal K, et al (2022) Characterizing the Fast Radio Burst Host Galaxy Population and its Connection to Transients in the Local and Extragalactic Universe. Astrophys J 163(2):69. 10.3847/1538-3881/ac3aec, arXiv:2108.01282 [astro-ph.HE]
  * Bhardwaj et al [2021] Bhardwaj M, Gaensler BM, Kaspi VM, et al (2021) A Nearby Repeating Fast Radio Burst in the Direction of M81. ApJL 910(2):L18. 10.3847/2041-8213/abeaa6, arXiv:2103.01295 [astro-ph.HE]
  * Bochenek et al [2020] Bochenek CD, Ravi V, Belov KV, et al (2020) A fast radio burst associated with a Galactic magnetar. Nature 587(7832):59–62. 10.1038/s41586-020-2872-x, arXiv:2005.10828 [astro-ph.HE]
  * Bondi [1970] Bondi H (1970) Astronomy of the Future. QJRAS 11:443
  * Burke-Spolaor et al [2011] Burke-Spolaor S, Bailes M, Ekers R, et al (2011) Radio Bursts with Extragalactic Spectral Characteristics Show Terrestrial Origins. ApJ 727(1):18. 10.1088/0004-637X/727/1/18, arXiv:1009.5392 [astro-ph.CO]
  * Burns and Clark [1969] Burns WR, Clark BG (1969) Pulsar Search Techniques. AAP 2:280–287
  * Cavallo and Jelley [1975] Cavallo G, Jelley JV (1975) Why are no radio pulses associated with the bursts of celestial gamma-rays? ApJL 201:L113–L115. 10.1086/181954
  * Cavallo and Ventura [1972] Cavallo G, Ventura A (1972) Restrictions on the Short Time Constant Astrophysics. Astrophys Sp Sci 19(2):431–439. 10.1007/BF00645726
  * Chatterjee et al [2017] Chatterjee S, Law CJ, Wharton RS, et al (2017) A direct localization of a fast radio burst and its host. Nature 541(7635):58–61. 10.1038/nature20797, arXiv:1701.01098 [astro-ph.HE]
  * CHIME/FRB Collaboration et al [2018] CHIME/FRB Collaboration, Amiri M, Bandura K, et al (2018) The CHIME Fast Radio Burst Project: System Overview. ApJ 863(1):48. 10.3847/1538-4357/aad188, arXiv:1803.11235 [astro-ph.IM]
  * CHIME/FRB Collaboration et al [2020a] CHIME/FRB Collaboration, Amiri M, Andersen BC, et al (2020a) Periodic activity from a fast radio burst source. Nature 582(7812):351–355. 10.1038/s41586-020-2398-2, arXiv:2001.10275 [astro-ph.HE]
  * CHIME/FRB Collaboration et al [2020b] CHIME/FRB Collaboration, Andersen BC, Bandura KM, et al (2020b) A bright millisecond-duration radio burst from a Galactic magnetar. Nature 587(7832):54–58. 10.1038/s41586-020-2863-y, arXiv:2005.10324 [astro-ph.HE]
  * CHIME/FRB Collaboration et al [2021] CHIME/FRB Collaboration, Amiri M, Andersen BC, et al (2021) The First CHIME/FRB Fast Radio Burst Catalog. ApJS 257(2):59. 10.3847/1538-4365/ac33ab, arXiv:2106.04352 [astro-ph.HE]
  * Chime/FRB Collaboration et al [2023] Chime/FRB Collaboration, Andersen BC, Bandura K, et al (2023) CHIME/FRB Discovery of 25 Repeating Fast Radio Burst Sources. ApJ 947(2):83. 10.3847/1538-4357/acc6c1, arXiv:2301.08762 [astro-ph.HE]
  * Colgate and Blevins [1973] Colgate SA, Blevins BA (1973) Supernova observational radio telescope. In: Bulletin of the American Astronomical Society, p 284
  * Colgate and Noerdlinger [1971] Colgate SA, Noerdlinger PD (1971) Coherent Emission from Expanding Supernova Shells. ApJ 165:509. 10.1086/150918
  * Comella et al [1969] Comella JM, Craft HD, Lovelace RVE, et al (1969) Crab Nebula Pulsar NP 0532\. Nature 221(5179):453–454. 10.1038/221453a0
  * Cortiglioni et al [1981] Cortiglioni S, Mandolesi N, Morigi G, et al (1981) A Systematic Search for Radio Pulses Associated with Gamma-Ray Bursts. Astrophys Sp Sci 75(1):153–161. 10.1007/BF00651392
  * Farah et al [2018] Farah W, Flynn C, Bailes M, et al (2018) FRB microstructure revealed by the real-time detection of FRB170827. MNRAS 478(1):1209–1217. 10.1093/mnras/sty1122, arXiv:1803.05697 [astro-ph.HE]
  * Ginzburg [1973] Ginzburg VL (1973) Possibility of Determining Intergalactic Gas Density by Radio Observations of Flares of Remote Sources. Nature 246(5433):415. 10.1038/246415a0
  * Green et al [1995] Green DA, Dessenne CAC, Warner PJ, et al (1995) A Search for Prompt Radio Emission from GRBs. ApSS 231(1-2):281–284. 10.1007/BF00658632
  * Hankins et al [1981] Hankins TH, Campbell DB, Davis MM, et al (1981) Searches for the radio millipusles from M 87 Virgo A. ApJL 244:L61–L64. 10.1086/183480
  * Hankins et al [2003] Hankins TH, Kern JS, Weatherall JC, et al (2003) Nanosecond radio bursts from strong plasma turbulence in the Crab pulsar. Nature 422(6928):141–143. 10.1038/nature01477
  * Hewish et al [1968] Hewish A, Bell SJ, Pilkington JDH, et al (1968) Observation of a Rapidly Pulsating Radio Source. Nature 217(5130):709–713. 10.1038/217709a0
  * Huguenin and Moore [1974] Huguenin GR, Moore EL (1974) A Search for Isolated Radiofrequency Pulses. ApJL 187:L57. 10.1086/181395
  * Hurley-Walker et al [2022] Hurley-Walker N, Zhang X, Bahramian A, et al (2022) A radio transient with unusually slow periodic emission. Nature 601(7894):526–530. 10.1038/s41586-021-04272-x
  * Hurley-Walker et al [2023] Hurley-Walker N, Rea N, McSweeney SJ, et al (2023) A long-period radio transient active for three decades. Nature 619(7970):487–490. 10.1038/s41586-023-06202-5
  * Jelley et al [1974] Jelley JV, Meikle WPS, Drever RWP, et al (1974) An Experiment to Search for Prompt Emissions from Supernovae at Microwave Frequencies. In: Cosmovici CB (ed) Supernovae and Supernova Remnants, p 61, 10.1007/978-94-010-2166-1_7
  * Katz et al [2003] Katz CA, Hewitt JN, Corey BE, et al (2003) A Survey for Transient Astronomical Radio Emission at 611 MHz. PASP 115(808):675–687. 10.1086/375568, arXiv:astro-ph/0304260 [astro-ph]
  * Keane et al [2012] Keane EF, Stappers BW, Kramer M, et al (2012) On the origin of a highly dispersed coherent radio burst. MNRAS 425(1):L71–L75. 10.1111/j.1745-3933.2012.01306.x, arXiv:1206.4135 [astro-ph.SR]
  * Keith et al [2010] Keith MJ, Jameson A, van Straten W, et al (2010) The High Time Resolution Universe Pulsar Survey - I. System configuration and initial discoveries. MNRAS 409(2):619–627. 10.1111/j.1365-2966.2010.17325.x, arXiv:1006.5744 [astro-ph.HE]
  * Kirsten et al [2022] Kirsten F, Marcote B, Nimmo K, et al (2022) A repeating fast radio burst source in a globular cluster. Nature 602(7898):585–589. 10.1038/s41586-021-04354-w, arXiv:2105.11445 [astro-ph.HE]
  * Kremer et al [2023a] Kremer K, Fuller J, Piro AL, et al (2023a) Connecting the young pulsars in Milky Way globular clusters with white dwarf mergers and the M81 fast radio burst. MNRAS 525(1):L22–L27. 10.1093/mnrasl/slad088, arXiv:2305.11933 [astro-ph.HE]
  * Kremer et al [2023b] Kremer K, Li D, Lu W, et al (2023b) Prospects for Detecting Fast Radio Bursts in the Globular Clusters of Nearby Galaxies. ApJ 944(1):6. 10.3847/1538-4357/acabbf, arXiv:2210.04907 [astro-ph.HE]
  * Large et al [1968] Large MI, Vaughan AE, Mills BY (1968) A Pulsar Supernova Association? Nature 220(5165):340–341. 10.1038/220340a0
  * Li et al [2021] Li D, Wang P, Zhu WW, et al (2021) A bimodal burst energy distribution of a repeating fast radio burst source. Nature 598(7880):267–271. 10.1038/s41586-021-03878-5, arXiv:2107.08205 [astro-ph.HE]
  * Linscott and Erkes [1980] Linscott IR, Erkes JW (1980) Discovery of millisecond radio bursts from M 87\. ApJL 236:L109–L113. 10.1086/183209
  * Lorimer et al [2007] Lorimer DR, Bailes M, McLaughlin MA, et al (2007) A Bright Millisecond Radio Burst of Extragalactic Origin. Science 318(5851):777. 10.1126/science.1147532, arXiv:0709.4301 [astro-ph]
  * Macquart et al [2020] Macquart JP, Prochaska JX, McQuinn M, et al (2020) A census of baryons in the Universe from localized fast radio bursts. Nature 581(7809):391–395. 10.1038/s41586-020-2300-2, arXiv:2005.13161 [astro-ph.CO]
  * Manchester et al [1969] Manchester RN, Murray JD, Radhakrishnan V (1969) Pulsar H-Line Absorption and Dispersion in the Interstellar Medium. Astrophys Lett 4:229
  * Manchester et al [2006] Manchester RN, Fan G, Lyne AG, et al (2006) Discovery of 14 Radio Pulsars in a Survey of the Magellanic Clouds. ApJ 649(1):235–242. 10.1086/505461, arXiv:astro-ph/0604421 [astro-ph]
  * Masui et al [2015] Masui K, Lin HH, Sievers J, et al (2015) Dense magnetized plasma associated with a fast radio burst. Nature 528(7583):523–525. 10.1038/nature15769, arXiv:1512.00529 [astro-ph.HE]
  * McCulloch et al [1981] McCulloch PM, Ellis GRA, Gowland GA, et al (1981) Failure to detect millisecond radio pulses from M 87. ApJL 245:L99–L101. 10.1086/183533
  * McLaughlin and Cordes [2003] McLaughlin MA, Cordes JM (2003) Searches for Giant Pulses from Extragalactic Pulsars. ApJ 596(2):982–996. 10.1086/378232, arXiv:astro-ph/0304365 [astro-ph]
  * McLaughlin et al [2006] McLaughlin MA, Lyne AG, Lorimer DR, et al (2006) Transient radio bursts from rotating neutron stars. Nature 439(7078):817–820. 10.1038/nature04440, arXiv:astro-ph/0511587 [astro-ph]
  * Michilli et al [2018] Michilli D, Seymour A, Hessels JWT, et al (2018) An extreme magneto-ionic environment associated with the fast radio burst source FRB 121102\. Nature 553(7687):182–185. 10.1038/nature25149, arXiv:1801.03965 [astro-ph.HE]
  * Palmer [1993] Palmer DM (1993) Radio Dispersion as a Diagnostic of Gamma-Ray Burst Distances. ApJL 417:L25. 10.1086/187085
  * Phinney and Taylor [1979] Phinney S, Taylor JH (1979) A sensitive search for radio pulses from primordial black holes and distant Supernovae. Nature 277(5692):117–118. 10.1038/277117a0
  * Pleunis et al [2021] Pleunis Z, Good DC, Kaspi VM, et al (2021) Fast Radio Burst Morphology in the First CHIME/FRB Catalog. ApJ 923(1):1. 10.3847/1538-4357/ac33ac, arXiv:2106.04356 [astro-ph.HE]
  * Rajwade et al [2020] Rajwade KM, Mickaliger MB, Stappers BW, et al (2020) Possible periodic activity in the repeating FRB 121102. MNRAS 495(4):3551–3558. 10.1093/mnras/staa1237, arXiv:2003.03596 [astro-ph.HE]
  * Ryder et al [2023] Ryder SD, Bannister KW, Bhandari S, et al (2023) A luminous fast radio burst that probes the Universe at redshift 1. Science 382(6668):294–299. 10.1126/science.adf2678, arXiv:2210.04680 [astro-ph.HE]
  * Schmidt [1963] Schmidt M (1963) 3C 273 : A Star-Like Object with Large Red-Shift. Nature 197(4872):1040. 10.1038/1971040a0
  * Scholz et al [2016] Scholz P, Spitler LG, Hessels JWT, et al (2016) The Repeating Fast Radio Burst FRB 121102: Multi-wavelength Observations and Additional Bursts. ApJ 833(2):177. 10.3847/1538-4357/833/2/177, arXiv:1603.08880 [astro-ph.HE]
  * Spitler et al [2014] Spitler LG, Cordes JM, Hessels JWT, et al (2014) Fast Radio Burst Discovered in the Arecibo Pulsar ALFA Survey. ApJ 790(2):101. 10.1088/0004-637X/790/2/101, arXiv:1404.2934 [astro-ph.HE]
  * Spitler et al [2016] Spitler LG, Scholz P, Hessels JWT, et al (2016) A repeating fast radio burst. Nature 531(7593):202–205. 10.1038/nature17168, arXiv:1603.00581 [astro-ph.HE]
  * Staelin and Reifenstein [1968] Staelin DH, Reifenstein IEdward C. (1968) Pulsating Radio Sources near the Crab Nebula. Science 162(3861):1481–1483. 10.1126/science.162.3861.1481
  * Suresh et al [2021] Suresh A, Chatterjee S, Cordes JM, et al (2021) An Arecibo Search for Fast Radio Transients from M87. ApJ 920(1):16. 10.3847/1538-4357/ac1672, arXiv:2107.09687 [astro-ph.HE]
  * Taylor et al [1981] Taylor JH, Backus PR, Damashek M (1981) No radio pulses from M 87. ApJL 244:L65–L68. 10.1086/183481
  * Tendulkar et al [2017] Tendulkar SP, Bassa CG, Cordes JM, et al (2017) The Host Galaxy and Redshift of the Repeating Fast Radio Burst FRB 121102. ApJL 834(2):L7. 10.3847/2041-8213/834/2/L7, arXiv:1701.01100 [astro-ph.HE]
  * Thornton [2013] Thornton D (2013) The High Time Resolution Radio Sky. PhD thesis, University of Manchester, UK
  * Thornton et al [2013] Thornton D, Stappers B, Bailes M, et al (2013) A Population of Fast Radio Bursts at Cosmological Distances. Science 341(6141):53–56. 10.1126/science.1236789, arXiv:1307.1628 [astro-ph.HE]
  * Zhu et al [2023] Zhu W, Xu H, Zhou D, et al (2023) A radio pulsar phase from SGR J1935+2154 provides clues to the magnetar FRB mechanism. Science Advances 9(30):eadf6198. 10.1126/sciadv.adf6198, arXiv:2307.16124 [astro-ph.HE]


# Where2Start: Leveraging initial States for Robust and Sample-Efficient
Reinforcement Learning

Pouya Parsa 111These authors contributed equally to this work
<EMAIL_ADDRESS>Raoof Zare Moayedi 111These authors contributed equally
to this work
<EMAIL_ADDRESS>Mohammad Bornosi 111These authors contributed equally
to this work
<EMAIL_ADDRESS>Mohammad Mahdi Bejani
<EMAIL_ADDRESS>
###### Abstract

The reinforcement learning algorithms that focus on how to compute the
gradient and choose next actions, are effectively improved the performance of
the agents. However, these algorithms are environment-agnostic. This means
that the algorithms did not use the knowledge that has been captured by
trajectory. This poses that the algorithms should sample many trajectories to
train the model. By considering the essence of environment and how much the
agent learn from each scenario in that environment, the strategy of the
learning procedure can be changed. The strategy retrieves more informative
trajectories, so the agent can learn with fewer trajectory sample. We propose
Where2Start algorithm that selects the initial state so that the agent has
more instability in vicinity of that state. We show that this kind of
selection decreases number of trajectories that should be sampled that the
agent reach to acceptable reward. Our experiments shows that Where2Start can
improve sample efficiency up to 8 times. Also Where2Start can combined with
most of state-of-the-art algorithms and improve that robustness and sample
efficiency significantly.

##  1 Introduction

Despite the remarkable success achieved by deep reinforcement learning (RL),
its practical application often encounters significant hurdles, primarily
stemming from its vulnerability in real-world settings. The performance of
many RL methods tends to falter when confronted with disparities between
training and testing scenarios, giving rise to critical safety and security
concerns. At the heart of the challenge lies the imperative to develop
policies robust enough to withstand disturbances and environmental changes,
particularly in addressing the pervasive ’reality gap’ problem. This issue
encapsulates the formidable task of designing systems capable of effectively
navigating the transition from controlled simulated training environments to
the intricate and unpredictable conditions of the real world.

One notable limitation in current RL systems is the random selection of
initial states for each training episode. This method overlooks the essential
characteristics of the environment in the short term, resulting in policies
that adeptly handle common scenarios but neglect fundamental yet rare
situations and catastrophic phenomena [1]. Consider a self-driving car
navigating through a city devoid of cars and pedestrians. Random placement in
the city often allows the agent, maintaining a fixed speed and untouched
steering, to achieve a high reward. However, crucial actions such as turning
and changing lanes, while less frequent, remain vital. If the agent encounters
difficulties in changing lanes but excels at turning, it should be exposed to
more lane-changing scenarios than turning scenarios during training. This
targeted exposure guides the agent toward mastering more sophisticated
maneuvers, minimizing the risk of catastrophic events and enhancing overall
learning.

This paper introduces a novel approach, named Where2Start, aimed at training
an agent with fewer episodes by strategically selecting initial states where
the agent is prone to suboptimal actions. By doing so, the agent is compelled
to learn from more challenging scenarios. Additionally, by excluding less
informative states, the number of sampled trajectories significantly
decreases. Our approach is versatile and can be seamlessly integrated with a
wide array of state-of-the-art approaches to achieve superior results in fewer
steps. The foundational concept of Where2Start is based on stability.
Therefore, we propose a criteria that represents the stability of policy
entire states.

##  2 Related Work

###  2.1 Robust RL

The task of formulating robust strategies that can secure high rewards amidst
adversarial environmental interferences, a notion commonly known as robust RL,
has been examined in scholarly research. This involves creating policies that
not only perform well under normal conditions but also maintain their
performance when faced with unexpected or adverse changes in the environment
[2][3][4].

Several studies have attempted to leverage adversarial attacks on neural
networks to create disturbances in environmental dynamics. For instance, these
works utilize the gradient of the critic network to generate disturbances that
have meaningful long-term effects. This approach allows for the exploration of
how these disturbances impact the performance and robustness of reinforcement
learning algorithms in various environments [5][6][7]. However, this line of
research does not encompass scenarios or environments where the features of
the environment are not directly correlated with observations. This leaves a
gap in understanding how reinforcement learning algorithms perform when
environmental features and observations are decoupled.

Several studies are exploring ways to improve the robustness of systems by
introducing an adversary into the environment, turning the problem into a
zero-sum game. This setup is seen as a duel between two players: the
protagonist, who is learning to perform a task, and the adversary, who is
learning to disrupt the environment to make the protagonist fail. The
adversary can cause various disruptions, such as creating disturbances that
alter the environment’s natural progression and behavior, or manipulating
physical obstacles to change the protagonist’s navigation routes or
strategies. The ultimate goal is to train the protagonist to become more
robust against the adversary’s disruptions. [8][9][10][11][12].This
methodology does present certain challenges. For instance, there isn’t a clear
correlation between the states of the adversary and the protagonist in these
environments. This is akin to the feature engineering of the adversary’s
states, which can introduce complexity and potential inaccuracies into the
system. It’s crucial to address these issues to ensure the effectiveness and
reliability of the approach.

In our research, we explore a unique scenario where the agent is confronted
with complexities in its observations, making it difficult to manage the
random noise inherent in these observations. This environment presents unique
challenges as the agent is expected to perform efficiently despite the
unpredictability and variations introduced by the noise. This situation is
particularly relevant as it closely mirrors real-world conditions. It’s not
uncommon for an agent’s sensors to experience issues or malfunctions, which
can further complicate the agent’s tasks. These complications can arise from
various factors such as environmental conditions, hardware limitations, or
even software glitches. The agent, therefore, needs to be robust and
adaptable, capable of making accurate decisions even when the input data is
noisy or incomplete.

###  2.2 Sample Efficieny

The upcoming topic will focus on the issue of how efficiently samples are used
in reinforcement learning. The aim in reinforcement learning that is efficient
in terms of data usage, especially when dealing with a Markov decision process
without any prior knowledge, is to find a policy that achieves a specific goal
with the fewest interactions. For a agent, it’s a significant challenge to
learn an effective policy with the least number of interactions, especially
when there’s no prior knowledge. Data-efficient methods have been proposed to
focus on policy evaluation and enhancement stages. These methods usually use
random techniques or fixed strategies to sample initial states. It’s important
to note that having a more informative initial states can help in learning a
more accurate controller, which can lead to fewer interactions.

In many real world scenarios, each interaction with the environment comes at a
cost, and it is desirable for deep reinforcement learning (RL) algorithms to
learn with a minimal amount of samples[13]. Operating real-world physical
systems, such as robots, can be expensive, which makes it crucial to learn
with the fewest possible number of real-world experiments. This is where the
concept of data-efficient reinforcement learning comes into play. It aims to
develop a policy that can achieve the desired outcome with minimal
interactions, thereby reducing the cost and time associated with numerous
trials. The challenge lies in the fact that these systems often lack prior
knowledge, making it difficult to establish an effective policy from the
outset. However, by utilizing data-efficient methods and having a more
informative initial states, we can expedite the learning of a precise
controller, ultimately leading to a decrease in the number of required real-
world interactions.

The issue of the impact of initial states dates back to the early years of
reinforcement learning. In [14] a reset distribution has been suggested, which
determines the next state based on a specific distribution. Extensive research
has been carried out to identify the most critical states for effective
training.

Some approaches limit the initial state to a specific set of states. This can
be seen, for example, in [15]. They utilized a collection of demonstration
states, initiating each episode by resetting to a state from a demonstration.
Interestingly, their agent does not exactly replicate the demonstrated
behavior. Instead, it is capable of discovering new and innovative solutions
that the human demonstrator might not have considered. This leads to achieving
a higher score on Montezuma’s Revenge than what was obtained using previously
published methods.

Other research has adopted an approach where a memory of the most significant
previous states is retained, such as [16][17]. As an illustration, [17]
presents a technique where the agent’s past trajectories are recorded. They
then revisit those intersections that seem to hold the potential for revealing
new insights, and restart their investigation from those locations.

Moreover, there exists a body of research focused on goal-oriented problems,
which employs a "reverse" training approach. This method progressively learns
to achieve the goal from a variety of starting points that are incrementally
distanced from the goal [18].

Our research stands out from others, especially regarding sample efficiency,
due to our unique methodology. We assign a score to each state before
initiating each episode, reflecting its level of uncertainty or sensitivity.
This strategy is somewhat akin to randomly selecting the initial state, but it
yields significantly better convergence than random sampling. We commence
exploration from the state with the highest uncertainty or sensitivity,
enabling us to develop a policy that concentrates on the most informative or
sensitive regions. In addition, we offer a comprehensive framework that
accommodates both off-policy and on-policy algorithms. This is a contrast to
some previous works that only cater to specific scenarios such as tabular MDPs
[19][20][21].

##  3 Background

###  3.1 Relative Conditional Number

The relative condition number is a metric that quantifies the potential
variation in the output of a function due to a small perturbation in the
input. This measure is particularly useful in environments with noise, as it
allows for consistent output generation for an input and its slightly altered,
or noisy, version. Essentially, it provides a mechanism to assess the
stability of a function’s output in the presence of minor fluctuations in the
input, making it a valuable tool in maintaining output integrity amidst input
uncertainties. For a function $f$ with several variables, we could define the
relative condition number as:

$\frac{\lVert J(X)\rVert.\lVert X\rVert}{\lVert f(X)\rVert}$ (1)

where $J(X)$ is the jacobian matrix of partial derivaties of $f$ at $x$.It is
important to note that in this study, we have defined the function
$f:R^{n}\rightarrow R$. Consequently, the Jacobian matrix is equivalent to its
gradient. Therefore, the relative condition number that we utilize is as
follows:

$\frac{\lVert\nabla f(X)\rVert.\lVert X\rVert}{\lVert f(X)\rVert}$ (2)

In the context of our research, we apply the relative condition number in a
unique manner. We designate the function ’f’ to correspond to the value
function, and then calculate the gradient in relation to the parameters of the
policy network. This methodology allows us to effectively leverage the
relative condition number within our research framework. The expression
related to this methodology for each state $s_{t}$ is provided below.

$\frac{\lVert\nabla_{\theta}Value_{\pi}(s_{t})\rVert.\lVert
s_{t}\rVert}{\lVert Value_{\pi}(s_{t})\rVert}$ (3)

Furthermore, In our analysis, we disregard the term $\lVert s_{t}\rVert$, as
it does not hold as much significance as the other quantities. The final
metric is presented as follows:

$\frac{\lVert\nabla_{\theta}Value_{\pi}(s_{t})\rVert}{\lVert
Value_{\pi}(s_{t})\rVert}$ (4)

###  3.2 GP

Gaussian processes offer a method for directly contemplating the overarching
characteristics of functions that could align with our data. They enable us to
integrate features such as rapid variation, periodicity, conditional
independencies, or translation invariance. The procedure commences with the
definition of a prior distribution over potential reasonable functions. This
prior is not in search of functions that match the dataset, but rather it aims
to specify plausible overarching characteristics of the solutions, like their
rate of variation with inputs. Upon data observation, this prior aids in
inferring a posterior distribution over functions that could align with the
data. Posterior samples can be utilized for making predictions by averaging
the values of every potential sample function from the posterior. The
posterior mean can be employed for point predictions, and a representation of
uncertainty can also be provided to indicate confidence in the predictions. In
this study, we employ two metrics to determine the uncertainty or sensitivity
of various random states. Utilizing these metrics, we then apply a Gaussian
Process (GP) model to fit our data. Essentially, we calculate a score based on
these metrics for the entire state space. The state with the highest score is
then selected. This approach allows us to systematically evaluate and select
the most significant states based on our established criteria.

###  3.3 Soft Actor Critic

SAC incentivizes stochastic exploration through entropy regularization on its
policy, this random action noise is insufficient for thoroughly exploring
complex environments. Simply increasing entropy leads to undirected state
space coverage, wasting samples in already-visited regions. Intelligently
searching the state space requires more than injecting random noise into the
policy. SAC lacks directed goals for seeking out novel and uncertain states.
To enable efficient exploration, the agent needs logic for identifying regions
of unpredictability and information gain, setting intrinsic goals for visiting
these areas. This targeted search across the state space could uncover new
states faster than entropy-driven noise. Overall, SAC’s core algorithm does
not include mechanisms for focused, information-maximizing exploration.

##  4 Method

In this section, we detail our methodologies for bolstering the robustness of
our model against environmental noise. Despite comprehensive experiments,
conventional off-policy reinforcement learning algorithms such as Soft Actor-
Critic (SAC) often struggle to achieve robustness against random environmental
disturbances. To tackle this, we put forth two distinct strategies. These
strategies are specifically designed to enhance the robustness of SAC, thereby
making it more resilient to environmental noise. The specifics of these
strategies will be elaborated in the following sections.

###  4.1 The issue of environmental noise

To begin with, we try to understand how the model could handle environmental
noise. When some perturbation is applied to the agent’s observation, the agent
should perform as before. Additionally, the agent cannot handle some states in
the environment or behave differently in them because it does not learn how to
act in them. Seeing a variety of situations is therefore crucial to training
an agent effectively and robustly. As a result, the agent is aware of how to
behave in different situations, and when there is noise in the observation,
which causes a different state for the agent, the agent should know how to
react appropriately.

###  4.2 Random selection of initial states

We propose an initial strategy that involves a more extensive random selection
of starting states, marking a departure from traditional methods. While many
existing reinforcement learning algorithms also employ random selection, they
typically limit their initial states to a confined region within the
environment. This restricted scope of random noise may not fully explore the
state space, potentially limiting the model’s resilience against random
disturbances. Our approach, on the other hand, champions a wider selection of
initial states with the goal of bolstering the model’s robustness. Owing to
this comprehensive exploration, our model encounters a variety of situations
and is better equipped to handle noisier settings. It’s important to note that
this broader random selection is not available in existing baseline
algorithms.

###  4.3 A new metric for measuring the importance of states

In this section, we focus on a key measure of significance: the Relative
condition number. This measure is used to assess the stability of the system.
A lower condition number indicates a more stable system, while a higher number
suggests potential instability. This metric provides a comprehensive
understanding of the system’s behavior and performance. To identify the state
with the highest score using this method, we initially select various states
at random and compute the specified scores for these states. Following this,
we implement a Gaussian process on the states to estimate scores across the
entire state space. The state yielding the highest score, as determined by the
output of the Gaussian process, is then selected. This approach allows us to
efficiently explore the state space and identify states of interest based on
their scores.1 2

Algorithm 1 Calculate Condition Number of Value Function

Input: Observations state $S_{0}$, policy parameters $\theta$, Q-function
parameters $\phi$, number of sampled actions $n$

$\mu,\sigma\leftarrow\pi_{\theta}(S_{0})$

$\text{sampled actions}\leftarrow\mathcal{N}(\mu,\sigma,\text{n})$

$\text{sampled actions probability}\leftarrow\text{calculate
probability}(\mu,\sigma,\text{sampled actions})$

$\text{value function}(S_{0})\leftarrow\sum_{\text{action in sampled
actions}}\text{Q}_{\phi}(S_{0},\text{action})\cdot\text{sampled actions
probability}[\text{action}]$

$\text{gradients}\leftarrow\nabla_{\theta}\text{value function}(S_{0})$

output: $\text{condition
number}\leftarrow\frac{||\text{gradients}||}{||\text{value function}||}$

Algorithm 2 Training on-stability policy gradient

Input: state discretization $S_{0}$, variance threshold $V$, number of epochs
$E$

$X_{\text{train}}\leftarrow S_{0}$

$Z_{\text{train}}\leftarrow\text{normalize}\hskip 2.84526ptX_{\text{train}}$

for epoch$=E,E-1,\cdots,1$ do

$Z_{\text{test}}\leftarrow\text{shift }Z_{\text{train}}$

$X_{\text{train}}\leftarrow\text{unnormalize }Z_{\text{train}}$

$Y_{\text{train}}\leftarrow\text{calculate metric of }\pi(X_{\text{train}})$

$X_{\text{test}}\leftarrow\text{unnormalize }Z_{\text{test}}$

$Y_{\text{test}}\leftarrow\text{calculate metric }\pi(X_{\text{test}})$

$\text{model }m\leftarrow\text{fit gaussian process on }Z_{\text{train}}\text{
and }Y_{\text{train}}$

$\text{variance, mean}\leftarrow\text{inference of }m\text{ on
}Z_{\text{test}}$

if $\text{max(variance) }>V$ then

$i\leftarrow\text{argmax(variance)}$

initial_state $\leftarrow X_{\text{test}_{i}}$

else

$i\leftarrow\text{argmax(mean)}$

initial_state $\leftarrow X_{\text{test}_{i}}$

end if

obs $\leftarrow$ reset env with the initial_state

while True do

$\text{action}\leftarrow\pi(\text{obs})$

$\text{obs, reward, done, }_{\text{ }}\leftarrow\text{env.step(action)}$

if done then

break

end if

end while

$\pi\leftarrow\text{update policy's parameters}$

$Z_{\text{train}}\leftarrow Z_{\text{test}}$

end for

##  5 Experiments

We designed some experiments to evaluate the performance and measure the
impacts of Our proposed method compared to state-of-the-art reinforcement
learning algorithms. To demonstrate this enhancement we conducted extensive
benchmarking experiments analyzing the performance of SAC [22] versus On-
StabilitySAC using the condition number metric across three distinct
environments. The environments examined were the OpenAI benchmarks
Pendulum-v1, MountainCarContinuous-v0, and Deepmind’s Swimmer-v3. Pendulum-v1
represents a typical environment that is readily solved by common algorithms.
MountainCarContinuous-v0 provides a sparse reward landscape, often trapping
algorithms in local minima as exemplified by SAC (1). Swimmer-v3 offers a high
dimensional observation space inducing substantial computational complexity.
Note : In comparison to the baseline SAC, our method, SAC + RandomSelection,
expands the exploration range from the initial state to cover the entire
observation space while opting for random selection

###  5.1 Robustness to different noise regimes

Performance of SAC versus On-StabilitySAC using the condition number metric
across the mentioned environments under four different noise regimes. The
noise types tested were $L_{0}$, $L_{2}$, $L_{\text{infinity}}$, and Gaussian
noise, with the first three representing Lp noise models. These experiments
aimed to demonstrate our model’s robustness across diverse noise types.
Overall, our contributions illustrate enhanced stability and noise resistance
compared to established RL techniques on representative benchmark environments
and noise models.1

(a)  (b)  (c)  (d)

Figure 1: Our experiments underscored the challenge of noise robustness in
MountainCarContinuous, a sparse reward environment. Despite the low
dimensional observation space, the SAC algorithm failed to identify the global
minimum in MountainCarContinuous. This inability to converge highlights the
difficulty reinforcement learning algorithms face in overcoming local minima
and sparse rewards. Our model demonstrates superior noise resilience in this
setting, reaching the global minimum under L0, L2, Linfinity, and Gaussian
noise regimes.

###  5.2 Effect of exploration metric

Our results indicate that the choice of exploration metric significantly
impacts measured performance. We compared our model using two approaches for
selecting initial states: random selection versus selection based on the
condition number metric. This benchmarking demonstrated substantially
different outcomes depending on the exploration metric used. When initial
states were chosen randomly, our model showed modest improvements compared to
baseline algorithms. However, conditioning initial states on the condition
number revealed dramatic enhancements in stability and noise resilience.2

(a) Pendulum

(b) MountainCar

(c) Swimmer

Figure 2: We compared the convergence rates across metrics in Pendulum-v1,
MountainCarContinuous-v0, and Swimmer-v3 benchmark environments. The x-axis
indicates training time in thousands of timesteps, while the y-axis shows the
mean cumulative reward over 100 evaluation episodes. The condition number
metric achieves superior cumulative rewards given equal interactions with the
environment. Dashed lines demonstrate the substantial additional training time
required for alternative metrics to reach parity with the condition number’s
performance.

###  5.3 Scalability and Sample efficiency

Duo to evaluate our model’s data efficiency we take advantage of the training
curves of our proposed method, on-stability SAC utilizing the condition number
metric compared to the baseline SAC conditioning initial state on Random
selection on both the Swimmer and MountainCarContinuous environments.3

(a) Swimmer (b) MountainCarContinuous (c) Pendulum

Figure 3:  On-Stability SAC achieves substantially higher cumulative rewards
over the course of training compared to SAC. Additionally, On-Stability SAC
reaches its maximum performance in approximately 150,000 timesteps, while SAC
requires over 300,000 interactions to converge. These results validate our
approach’s ability to attain superior asymptotic performance with fewer
environmental interactions on complex tasks. The condition number metric
enables more efficient learning, allowing our algorithm to surpass baseline
SAC’s performance with nearly 50% less experience.

Also by having the advantage of sample efficiency search, we have to compare
the order of computation time per one episode of models, to ensure how
scalable methods are versus complex environments.1

| Computation Time | Sample Efficiency
---|---|---
Environments | Random Selection | Condition Number | Random Selection | Condition Number
Pendulum-v1 | X | 15X | A | 2A
MountainCarContinuous-v0 | Y | 4.85Y | - | -
Swimmer-v3 | Z | 95Z | B | 8B
Table 1: On-Stability SAC + ConditionNumber, while exhibiting sample
efficiency in search environments, is associated with a high computation time
due to the complexity of environments. This complexity could potentially
overshadow data efficiency in search, making it important to consider in
particularly complex environments.

A comprehensive examination of environmental complexity, spanning from
Pendulum-v1 to Swimmer-v3, reveals a heightened density in the observation
space due to an increased number of dimensions. Consequently, the exploration
of numerous states becomes essential to identify an optimal initial state. (It
is noteworthy that the quantity of states considered for metric calculation
exhibits an exponential relationship with the observation space’s dimension.)
As a result, the computational cost associated with our metric calculation is
conditioned upon the extensive application of the backward() function on the
value function of each state. This, in turn, leads to significant time-
consuming training. Nonetheless, there is a conceptual approach wherein
leveraging the backward function throughout the training process by baseline
algorithms enables the calculation of gradients for the value function,
allowing for their efficient reuse.

Now, by incorporating sample efficiency and scalability through the reuse
concept, we can assert that our method is poised to be applied effectively in
complex environments.

##  6 Conclusion

In this study, we introduced the innovative approach Where2Start to tackle
pivotal challenges in deep reinforcement learning (RL), with a particular
focus on mitigating vulnerabilities during the transition from simulated
training environments to real-world applications. The persistent ’reality gap’
dilemma highlights the imperative for RL systems to dynamically adapt to the
intricacies and unpredictabilities inherent in diverse scenarios.

Where2Start takes a strategic approach to selecting initial states for
training episodes based on the agent’s suboptimal actions, providing a
promising avenue for bolstering RL robustness. The method systematically
exposes the agent to challenging scenarios influenced by conditional numbers,
fostering a more comprehensive understanding of its environment.

Moreover, Where2Start contributes to efficiency gains by significantly
reducing the number of sampled trajectories. This efficiency, coupled with the
adaptability of our approach to seamlessly integrate with various state-of-
the-art methods, positions Where2Start as a valuable tool for achieving
superior RL results in a condensed number of training steps. Notably, our
experiments reveal that Where2Start can converge to a better sub-optimal agent
up to 8 times faster than common approaches. However, it is essential to
acknowledge that the computation cost of the conditional number at the start
of each episode is up to 95 times higher than common approaches, a critical
consideration for future research endeavors.

The emphasis on stability within our approach establishes a solid foundation
for ongoing advancements in RL research and applications. As we navigate the
evolving landscape of artificial intelligence, Where2Start signifies a
significant stride toward more reliable, safer, and efficient RL systems,
setting the stage for broader and more impactful real-world implementations.

## References

  * [1] Robert M French.  Catastrophic forgetting in connectionist networks.  Trends in cognitive sciences, 3(4):128–135, 1999.
  * [2] Marek Petrik and Reazul Hasan Russel.  Beyond confidence regions: Tight bayesian ambiguity sets for robust mdps.  Advances in neural information processing systems, 32, 2019.
  * [3] Reazul Hasan Russel, Bahram Behzadian, and Marek Petrik.  Entropic risk constrained soft-robust policy optimization.  arXiv preprint arXiv:2006.11679, 2020.
  * [4] Esther Derman, Matthieu Geist, and Shie Mannor.  Twice regularized mdps and the equivalence between robustness and regularization.  Advances in Neural Information Processing Systems, 34:22274–22287, 2021.
  * [5] Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary.  Robust deep reinforcement learning with adversarial attacks.  arXiv preprint arXiv:1712.03632, 2017.
  * [6] Lucas Schott, Hatem Hajri, and Sylvain Lamprier.  Improving robustness of deep reinforcement learning agents: Environment attack based on the critic network.  In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2022.
  * [7] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel.  Adversarial attacks on neural network policies.  arXiv preprint arXiv:1702.02284, 2017.
  * [8] Xiaobai Ma, Katherine Driggs-Campbell, and Mykel J Kochenderfer.  Improved robustness and safety for autonomous vehicle control with adversarial reinforcement learning.  In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1665–1671. IEEE, 2018.
  * [9] Johannes Heinrich, Marc Lanctot, and David Silver.  Fictitious self-play in extensive-form games.  In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 805–813, Lille, France, 07–09 Jul 2015. PMLR.
  * [10] Johannes Heinrich and David Silver.  Deep reinforcement learning from self-play in imperfect-information games.  arXiv preprint arXiv:1603.01121, 2016.
  * [11] Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, and Volkan Cevher.  Robust reinforcement learning via adversarial training with langevin dynamics.  Advances in Neural Information Processing Systems, 33:8127–8138, 2020.
  * [12] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta.  Robust adversarial reinforcement learning.  In International Conference on Machine Learning, pages 2817–2826. PMLR, 2017.
  * [13] Vincent François-Lavet, Peter Henderson, Riashat Islam, Marc G Bellemare, Joelle Pineau, et al.  An introduction to deep reinforcement learning.  Foundations and Trends® in Machine Learning, 11(3-4):219–354, 2018.
  * [14] Sham Kakade and John Langford.  Approximately optimal approximate reinforcement learning.  2002\.
  * [15] Tim Salimans and Richard Chen.  Learning montezuma’s revenge from a single demonstration.  arXiv preprint arXiv:1812.03381, 2018.
  * [16] Arash Tavakoli, Vitaly Levdik, Riashat Islam, Christopher M Smith, and Petar Kormushev.  Exploring restart distributions.  arXiv preprint arXiv:1811.11298, 2018.
  * [17] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune.  Go-explore: a new approach for hard-exploration problems.  arXiv preprint arXiv:1901.10995, 2019.
  * [18] Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel.  Reverse curriculum generation for reinforcement learning.  In Conference on robot learning, pages 482–495. PMLR, 2017.
  * [19] Trey Smith and Reid Simmons.  Focused real-time dynamic programming for mdps: Squeezing more out of a heuristic.  01 2006.
  * [20] Andrew G. Barto, Steven J. Bradtke, and Satinder P. Singh.  Learning to act using real-time dynamic programming.  Artificial Intelligence, 72(1):81–138, 1995.
  * [21] H. Brendan McMahan, Maxim Likhachev, and Geoffrey J. Gordon.  Bounded real-time dynamic programming: Rtdp with monotone upper bounds and performance guarantees.  In Proceedings of the 22nd International Conference on Machine Learning, ICML ’05, page 569–576, New York, NY, USA, 2005. Association for Computing Machinery.
  * [22] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In International conference on machine learning, pages 1861–1870. PMLR, 2018.