# STATUS OF THE HIE-ISOLDE PROJECT AT CERN

M.A. Fraser  Y. Kadi  A.P. Bernardes  Y. Blumenfeld  E. Bravin  S. Calatroni
R. Catherall
B. Goddard  D. Parchet  E. Siesling  W. Venturini Delsolaro  G. Vandoni  D.
Voulot  L.R. Williams

CERN  Geneva  Switzerland

###### Abstract

The HIE-ISOLDE project represents a major upgrade of the ISOLDE nuclear
facility with a mandate to significantly improve the quality and increase the
intensity and energy of radioactive nuclear beams produced at CERN. The
project will expand the experimental nuclear physics programme at ISOLDE by
focusing on an upgrade of the existing Radioactive ion beam EXperiment (REX)
linac with a 40 MV superconducting linac comprising thirty-two niobium-on-
copper sputter-coated quarter-wave resonators housed in six cryomodules. The
new linac will raise the energy of post-accelerated beams from 3 MeV/$u$ to
over 10 MeV/$u$. The upgrade will be staged to first deliver beam energies of
5.5 MeV/$u$ using two high-$\beta$ cryomodules placed downstream of REX,
before the energy variable section of the existing linac is replaced with two
low-$\beta$ cryomodules and two additional high-$\beta$ cryomodules are
installed to attain over 10 MeV/$u$ with full energy variability above 0.45
MeV/$u$. An overview of the project including a status summary of the
different R&D activities and the schedule will outlined.

##  1 INTRODUCTION

The High Intensity and Energy (HIE) ISOLDE project [1] aims at several
important upgrades of the present ISOLDE radioactive beam facility at CERN.
The main focus lies in the energy upgrade of the post-accelerated radionuclide
beams from 3 MeV/$u$ up to over 10 MeV/$u$ through the addition of
superconducting (SC) quarter-wave resonators (QWRs) operating at 101.28 MHz.
This will open the possibility of many new types of experiments including
transfer reactions throughout the nuclear chart.

The project also includes a design study that aims at improving the target and
front-end part of ISOLDE to fully benefit from upgrades of the existing CERN
proton injectors, e.g. LINAC4 and upgrade in energy of the PS Booster. This
improvement combined with upgrades to the RILIS laser ion source and the
radiofrequency quadrupole (RFQ) cooler and buncher (ISCOOL) will lead to an
increase of radioactive beam intensities of up to an order of magnitude. The
beam emittance will be improved with the implementation of ISCOOL placed after
a pre-separator but before a new High-Resolution Separator (HRS). The new HRS,
based on the latest magnet technology, will have sufficient mass resolution to
permit isobaric separation. ISCOOL will also permit a tailoring of the time
structure of the beam, removing the dependence on the proton beam time
structure and diffusion-effusion properties of the target and ion source
units. Highly charged ions will be provided for REX and other users through an
improved low energy stage of REX-ISOLDE and a possible installation of an
upgraded Electron Beam Ion Source (EBIS) charge breeder.

The linac upgrade will be staged in order to deliver higher beam energies to
the experiments as soon as possible, with future upgrade stages ensuring a
wide range of energy variability and providing an optional $\sim$100 ns bunch
spacing. The first stage of the upgrade involves the design, construction,
installation and commissioning of two cryomodules downstream of REX, the
existing post-accelerator. These cryomodules will each house five high-$\beta$
($\beta_{g}=10.3\%$) SC cavities and one SC solenoid. Extra cryomodules will
be added to the beam line in a modular fashion until all six cryomodules,
including two cryomodules housing six low-$\beta$ ($\beta_{g}=6.3\%$) SC
cavities and two SC solenoids, are online. The upgrade will be completed with
a final stage that will see the linac extended in order to make room to pre-
bunch the beam into the existing RFQ accelerator at a sub-harmonic frequency
below 101.28 MHz, allowing the bunch spacing to be increased without
significant loss in transmission; time-of-flight particle detection will then
be viable at the experiments. Also foreseen is a beam chopper to reject the
background of populated satellite bunches either side of the main sub-harmonic
beam pulses. The staged installation of the linac is shown schematically in
Figure 1.

Figure 1: A schematic of the staged installation of the HIE-ISOLDE linac
(Existing REX structures: RFQ, IHS: 20-gap IH-structure, 7GX: 7-gap split-ring
cavities, 9GP: 9-gap IH-structure).

The pre-bunching/chopping scheme is under study and still to be finalised,
e.g. chopping could occur before RFQ instead of after.

The beam dynamics studies [2, 3] arrived at a compact lattice, based on
cavities operating at a gradient of 6 MV/m,111Cavity active length
definitions: $\textrm{low-}\beta=195~{}\textrm{mm}$ and
$\textrm{high-}\beta=300~{}\textrm{mm}$. designed both due to the limited
space in the experimental hall and to maximise the dynamical acceptance and
optical performance of the machine. These considerations influenced the
specification of the SC solenoids, alignment system, beam diagnostics system,
steering magnets and cryomodules.

The superconducting linac is designed to provide a total accelerating voltage
of 39.6 MV with an average synchronous phase $\phi_{s}$ of $-20$ deg, i.e. the
minimum voltage required in order to achieve a final energy of at least 10
MeV/$u$ with the heaviest beams that have a mass-to-charge state $A/q=4.5$.
The normal conducting 20-gap IH-structure (IHS) limits the $A/q$ acceptance of
the machine to $2.5\leq A/q\leq 4.5$. The energy range accessible with the new
SC linac is shown for the each stage in Figure 2, including the decelerated
beam that opens a range of energy not previously accessible with REX because
of the fixed velocity profile of the IHS.

(a) Stages 1 and 2a

(b) Stage 2b

Figure 2: Energy reach of post-accelerated beams for each stage of the upgrade
as a function of the $A/q$ acceptance.

A research and development programme looking at all the different aspects of
the SC linac started in 2008 and continued throughout 2011. In particular, the
R&D effort has focused on the development of the high-$\beta$ cavity, for
which it has been decided to adopt technology based on copper cavities
sputter-coated with niobium. The required infrastructure has been determined
and the integration of the SC linac and High Energy Beam Transfer (HEBT) lines
inside the existing experimental hall has been finalised.

The project has been approved by CERN and its implementation started in
January 2010. An overview of the project and its timeline will be given below.

##  2 PHYSICS MOTIVATION

The current REX maximum energy of 3 MeV/$u$ largely restricts the physics
program to measurements of collective degrees of freedom through single-step
Coulomb excitation and probing single-particle degrees of freedom of light
nuclei through direct nuclear reactions. The provision of exotic beams with
energies up to 10 MeV/$u$ will greatly expand the application of these methods
to all ISOLDE beams [4]. Complete measurements of low-lying collective
structure will be possible and, for particle states, unambiguous physical
quantities can be extracted. The versatility of the accelerator, with variable
energy down to 0.45 MeV/$u$, will facilitate measurements of rates of
reactions that are the building blocks of nucleosynthesis chains, while the
higher energy beams from the new linac will also allow prototype studies of
new collective modes using electromagnetic probes and studies of exotic
proton-rich and neutron-rich nuclei using fusion and highly-damped binary
collisions.

The installation of the TSR storage ring [5] at HIE-ISOLDE has been approved
and provides unique opportunities by being located at an ISOL facility.
Reaction experiments could benefit from an increased luminosity from being in-
ring, through multiple beam passes ($\sim$1 MHz), or from extracting electron-
cooled beams for ultra-high resolution studies of heavy nuclei and Coulex
studies. The TSR could also be used as a beam stretcher to accumulate the
pulsed beam delivered by the EBIS and extract it as a d.c. beam over a few
seconds, alleviating problems from the high instantaneous rates encountered
with pulsed beams.

The 34 letters of intent submitted in 2010 for experiments at HIE-ISOLDE
include 284 participants. The demand for the linac upgrade is reflected in 88%
of the letters requesting higher beam energies that will be provided by the
HIE upgrade.

##  3 INTEGRATION AT ISOLDE

The new HIE-ISOLDE SC linac will require a major increase of equipment to the
existing facility’s infrastructure. Two new surface buildings will be
constructed in order to house the helium compressor station and the helium
refrigerator cold box, see Figure 3.

Figure 3: New HIE-ISOLDE buildings.

Ground breaking started at the end of last summer (2011) with the preparation
of the site and the construction of a new secure access point to the
experimental hall for personnel and material while HIE-ISOLDE construction
work is on-going. Presently, civil engineering works are in progress for the
construction of the new buildings adjacent to the existing ISOLDE experimental
hall. The construction works should be completed by the third quarter of 2012
after which the installation of the electrical systems and main services will
take over. The cryogenic station installation, shown in Figure 4, will start
in the second quarter of 2013. The He liquefier will be installed in a
separate light construction building as close as possible to the linac in
order to minimise the length of the LHe distribution line. This will enable an
easier and more stable operation of the cryogenic system. The cryogenic system
includes a cryogenic transfer line that will link the cold box to the
different interconnecting “jumper” boxes, feeding from the top, the six
cryomodules of the new SC linac.

Figure 4: Layout of the HIE-ISOLDE facility.

The shielding tunnel will be installed in its entirety for the full linac
along with the cryogenic transfer line, see Figure 5, whereas the linac and
high energy beam transfer lines will be installed in stages. A new HEBT,
discussed below, will bring the beam into the existing extension of the ISOLDE
experimental hall [6].

Figure 5: Linac shielding tunnel and cryogenic transfer line.

##  4 CRYOMODULE DESIGN

The cavities will be grouped in common cryomodules with six cavities and two
focusing solenoids for the low-$\beta$ cryomodules, and five cavities and one
focusing solenoid for the high-$\beta$ cryomodules. Figure 6 shows a 3D model
of the high-$\beta$ cryomodule. In order to simplify the mechanical design and
assembly, and to minimise the drift length between cavities and the overall
length of the machine, a common vacuum was chosen for the beam and cryogenic
insulation. The solenoids need to be aligned with a stringent precision of
$\pm 0.15$ mm (i.e. $\pm 1\sigma$, where the simulated Gaussian error
distribution was truncated at $\pm 3\sigma$) and a system of independent
adjustment, useable under vacuum and at operational temperatures, of the
solenoid with respect to the cavity string is foreseen. An active position
monitoring system based on BCAM CCD cameras [7] is under development. The
active components will be cooled to 4.5 K in two stages using gaseous and
liquid He. Insulation will be guaranteed by a heat screen at 75 K. A vacuum of
$10^{-8}$ mbar after cryopumping is necessary for optimal operation. To ensure
the cleanest vacuum conditions, the vacuum system will be entirely dry, with
turbo-molecular pumps backed by dry scrolls. In addition, automatic procedures
with staged limitation of conductance, both for slow pump-down in the viscous
regime and for venting with dry nitrogen, will be applied. A detailed
description of the high-$\beta$ cryomodule design can be found in [8].

Figure 6: 3D model of the high-$\beta$ cryomodule (beam direction is from
right to left).

##  5 CAVITY DEVELOPMENT

The development of the SC cavities started in 2008 and focused on the
high-$\beta$ cavity design, which will be installed first. The cavities are
based on the niobium sputtered copper technology pioneered at INFN-LNL. They
are specified to reach a nominal accelerating field of 6 MV/m on axis over an
active length of 300 mm. A $Q$-value of $5\times 10^{8}$ is necessary to reach
this field with a power dissipation of 10 W. In recent years much effort has
been put in the design of a coating facility and the optimisation of the
coating process. Since 2011 a single-cavity test-stand has been operational at
CERN [9] working in tandem with the sputtering development to improve the
quality factor of the niobium sputtered cavities. The latest tests have shown
$Q$-values of $1.5\times 10^{8}$ at 6 MV/m; close to the design goal. Further
details can be found in [10].

##  6 BEAM DIAGNOSTICS

A beam instrumentation R&D programme is ongoing to provide a solution for the
HIE-ISOLDE beam diagnostic system [11]. The space available in the regions
between the cryomodules, in which a dipole steering magnet and vacuum valves
are also situated, is very constrained, see Figure 7. Dimensioned at 58 mm in
the longitudinal direction, the box is extremely compact and is designed to
operate in the stray-field of the adjacent steering magnet. More details of
the air-cooled steering magnet design specified at 6 T mm can be found in
[12].

Figure 7: 3D model of the region between cryomodules.

An octagonal solution for the main body of the diagnostic box was chosen to
facilitate multiple functionalities in the compact space available for the
box, see Figure 8. The same design will be used in the HEBT lines, numbering
fifteen units in the first stage of the upgrade. Up to five different devices
can be actuated in-and-out of the beam in the transverse plane, with a beam
profiling functionality included in two of these five devices as standard: a
scanning slit that profiles the beam in front of a Faraday cup. The Faraday
cup has been designed with a length of just 20 mm and a sensitivity from
$1-500$ pA. The prototype system will be delivered to CERN by Added Value
Solutions (AVS) this summer for testing of the Faraday cup with beam before
the Long Shutdown of the CERN accelerator complex commences (December 2012
until April 2014).

Figure 8: 3D model of the short diagnostic box, courtesy of AVS technology.

A longitudinal beam diagnostics system based on a silicon detector has been
developed for the fast, and eventually automated, phasing of the large number
of cavities that will accompany the upgrade. The system is capable of
providing both relative phase and energy measurements. The proof-of-principle
has been tested at REX by phasing the third 7-gap resonator with the system.
For more details the reader is referred to [13]. A time-of-flight system for
absolute energy measurements is also being considered.

##  7 COMMISSIONING AND OPERATION

In preparation for the linac upgrade various recommissioning steps have been
undertaken to better understand the optimum working points of the existing
accelerating structures, including bead-pull measurements, emittance
measurements and end-to-end beam dynamics simulations benchmarked to
measurements, described in [3]. The emittance delivered by REX was shown to be
compatible with the acceptance of the superconducting linac in both the
transverse and longitudinal phase-space planes, see [14] and [15] for the
details of the respective measurements. The longitudinal acceptance of the SC
linac is compared to the phase-space of the beam delivered by REX in Stages 1
and 2b of the upgrade in Figure 9. The normalised 90% transverse emittance was
measured as less than 0.3 $\pi\textrm{ mm mrad}$ and the 86% longitudinal
emittance was measured as 1.5 $\pi\textrm{ ns keV/}u$.

(a) Stage 1 ($W_{0}=2.85~{}\textrm{MeV}/u$).

(b) Stage 2b ($W_{0}=1.2~{}\textrm{MeV}/u$).

Figure 9: Simulated beam distribution (consistent with measurements) in
longitudinal phase-space at entry to the SC linac, compared to the acceptance
(shown in white).

To date, transverse emittance measurements using the NTG emittance rig [16]
have been limited to those made on high intensity ($\sim\textrm{nA}$) pilot
beams, which are generated by leaking gas into the EBIS. It is likely that the
beam emittance is degraded with respect to the radioactive beam emittance due
to the disruptive space-charge effects produced by the higher beam intensity
and resulting compensation of the radial focusing force provided to the
trapped ions in the electron beam of the source. For this reason a high
sensitivity emittance measurement system using single-particle detectors is
being developed to measure the radioactive beam emittance using slits and
detectors housed in adjacent diagnostic boxes that were shown previously. An
emittance measurement will also be attempted this summer at lower beam
intensity ($\sim$10 pA) by avoiding the use of slits to sample the phase-space
in the emittance rig and instead using the three-gradient method to
reconstruct the phase-space distribution by tomography; the beam size at the
wire-grid of the emittance rig will be varied as a function of the gradient of
an upstream quadrupole.

Due to the short nature of REX experimental runs (typically three to ten days)
and the need for frequent energy and beam changes dictated by the physics
programme, REX operation requires fast and reliable set-ups and the
possibility to switch between beams or change the energy within a few hours.
The size and complexity of the machine will increase with the energy upgrade,
e.g. the number of cavities will increase from seven in the present linac to
thirty-five in the final version of the HIE linac. The cavity phases will also
become dependent on $A/q$ as the full voltage provided by the QWRs can be
exploited and the velocity profile changes as a function of $A/q$ in the SC
part of the linac. For this reason an automatic phasing procedure is foreseen
using the above mentioned silicon detector either by tracking the relative
energy change or, when the 100 ns bunch spacing becomes available, using a
time-of-flight method.

Software applications for the control system are being developed in
preparation for the commissioning phase, including the possibility of setting
machine parameter values directly from beam simulation codes and vice versa.
Automated optimisation software is also being investigated to phase the linac
quickly and some initial tests have already been done on the existing linac.

##  8 HIGH ENERGY BEAM TRANSFER LINES AND EXPERIMENTAL AREAS

The Miniball segmented germanium array and T-REX experimental setups already
operational at REX will be used intensively when the first beams are delivered
in 2015.

Figure 10: Stage 1 of the HIE upgrade with two high-$\beta$ cryomodules and
HEBT lines added to REX.

Only the first two high-$\beta$ cryomodules will be installed in the tunnel,
downstream of the existing REX linac and an open beam line will also be
available for traveling experiments, see Figure 10.

The following year it is foreseen to install a helical-orbit spectrometer that
will focus on transfer reactions across the entire nuclear chart, made
possible by the upgraded beam energy. The helical-orbit spectrometer will
demand a bunch spacing of the order of 100 ns and a scheme to pre-bunch before
the RFQ at a sub-harmonic of its natural frequency is foreseen to satisfy this
requirement. An active-target experiment for resonant scattering and transfer
reactions is also under consideration.

The HEBT design [6, 11] is highly standardised with only a single type of
quadrupole and dipole magnet needing procurement; the exceptions are the 22.5
deg dipoles on the transfer line to the second experimental station that are
already procured and being refurbished. Each 90 deg bend is a standardised
double-bend achromat consisting of two 45 deg dipoles and a single dispersion
suppressing quadrupole. The HEBT is based on a periodic system of doublet
cells with a unit length equal to the length of the high-$\beta$ cryomodule.
As a result, the installation of the staged upgrade can occur in a modular
fashion by the replacement of doublet cells with cryomodules. The HEBT will be
extended in a second stage of the installation foreseen during 2017/18 to
accommodate a third experimental station. Some space has been reserved behind
the third experimental station for the installation of a magnetic spectrometer
in the future. The layout of the HEBT is shown in Figure 11.

Figure 11: Layout of the HEBT lines in the ISOLDE experimental hall.

Vacuum in the low $10^{-7}~{}\textrm{mbar}$ range is obtained by
turbomolecular pumps, backed by dry, hermetical multiroots, connected to the
common effluent collector and the air ventilation evacuation. The turbopumps
are installed at dedicated ports on the diagnostic boxes, which constitute the
major source of length-specific outgassing. Vacuum sectorisation is kept
minimal, with the goal of separating in distinct vacuum sectors the different
beam lines, as well as to cut the experimental areas from the beam transfer
sections.

The extension of the HEBT in the final stage permits a connection to the TSR
storage ring that is expected to become operational at CERN during this
period, see Figure 12.

Figure 12: First-floor extension for the TSR behind the experimental hall
(shown in orange) connecting to the HEBT via the inclined beam line (shown in
red).

##  9 SAFETY ASPECTS

The main hazards that will be introduced by the linac upgrade come from
X-rays, neutrons and the cryogenic system. A radiation measurement station has
been integrated into the single-cavity test-stand to collect dose rate data
specific to the HIE-ISOLDE cavities. These data will be used to confirm the
sizing of the tunnel shielding. An access system with key is already used to
prevent access during linac operation. When one key is removed from a key box
the RF is stopped and the amplifiers cannot be restarted until all keys are
back in a key box [17]. A similar system is foreseen for the HIE-ISOLDE tunnel
access.

An incident beam of heavy ions at an energy of 10 MeV/$u$ is higher than the
Coulomb barrier allowing neutron production. A dose rate of 0.1 $\textrm{mSv
h}^{-1}~{}\textrm{pnA}^{-1}$ is expected at 1 m, at an angle of 90 deg from
vicinity of the beam loss [18]. During operation and setting-up, a maximum
beam intensity of $\sim$ppA is expected in the experimental hall and free
access will be possible.

Redundant safety valves will be installed on the He tanks of the cryomodules
to mitigate for the cryogenic hazard associated with the 900 litres of LHe
inside the six cryomodules. These safety valves will be routed outside the
tunnel in order to reduce exposure to an oxygen deficiency hazard during
technical access to the tunnel while the cryomodules are cold.

Finally, pumping on radioactive beams implies a risk of radioactive
contamination of the vacuum system, demanding an entirely hermetical system,
with collection and evacuation of the effluent gases before controlled release
to the atmosphere.

##  10 PLANNING AND SCHEDULING

If the carefully planned schedule is respected, the various experiments
currently receiving beams from ISOLDE will not suffer from the upgrade works.
Civil engineering work and installation of the main services such as power,
ventilation and cooling will take place while the ISOLDE facility is running
for the experiments. To guarantee a minimum of perturbation to the operation
of the facility, the main services will be connected and the existing services
modified only during the CERN Long Shutdown. Civil engineering work inside the
experimental hall, such as the construction of the new tunnel, as well as the
move of the existing Miniball experiment to its new position will also be
carried out during this period of shutdown.

Start-up of the low energy (60 keV) part of the ISOLDE facility, excluding the
REX post-accelerator, is foreseen for April 2014 as normal. However, at that
moment the HIE-ISOLDE linac and HEBT line will still be under construction
with the installation of the first two high-$\beta$ cryomodules and the
transfer line elements running into the summer of 2014. Beam commissioning at
5.5 MeV/$u$ is planned for the end of the 2014. The remaining two high-$\beta$
cryomodules will be installed in a second stage in 2016, increasing the beam
energy to 10 MeV/$u$, together with an additional bend in the HEBT providing
the users with a third experimental station (and making possible the
connection of the TSR storage ring). A pre-bunching scheme before the RFQ with
chopper and the two low-$\beta$ cryomodules completing the HIE-ISOLDE linac
are foreseen for a later stage after 2017.

##  11 OUTLOOK

The main technological options of the linac are now fixed and most components
are in their final design or prototyping phase. The HEBT design has been
finalised and the components are now in the specification or prototyping
phase. The infrastructure installation is under way and should be completed by
mid-2013. The cryogenic lines and first phase of the linac and transfer lines
installation is planned for 2014. The linac commissioning at 5.5 MeV/$u$ is
foreseen for late 2014 and the first physics should take place in early 2015.

##  12 ACKNOWLEDGMENTS

This paper summarises the work of several teams: The ISOLDE Collaboration, the
HIE-ISOLDE project team, and numerous groups at CERN within the accelerator
and technology sector.

We acknowledge funding from the Swedish Knut and Alice Wallenberg Foundation
(KAW 2005-0121) and from the Belgian Big Science program of the FWO (Research
Foundation Flanders) and the Research Council K.U. Leuven.

We would also like to acknowledge the receipt of fellowships from the CATHI
Marie Curie Initial Training Network (EU-FP7-PEOPLE-2010-ITN Project number
264330) and the COFUND-CERN Marie Curie Fellowship programme.

Support from the Spanish Programme Industry for Science from CDTI is also
acknowledged.

## References

  * [1] M. Lindroos, P.A. Butler, M. Huyse and K. Riisager, “HIE-ISOLDE,” _Nucl. Instrum. Meth. B_ 266, pp. 4687-4691 (2008).
  * [2] M.A. Fraser, R.M. Jones, M. Pasini,“Beam Dynamics Design Studies of a Superconducting Radioactive Ion Beam Postaccelerator,” _Phys. Rev. ST Accel. Beams_ 14, p. 020102 (2011).
  * [3] M.A. Fraser, “Beam Dynamics Studies of the ISOLDE Post-Accelerator for the High Intensity and Energy Upgrade,” PhD Thesis (University of Manchester), Vol. 13 in EuCARD Series (2012).
  * [4] K. Riisager, P. Butler, M. Huyse, and R. Krücken. “HIE-ISOLDE: The scientific opportunities,” CERN Report, CERN-2007-008 (2007).
  * [5] M. Grieser, Yu. Litvinov, R. Raabe, K. Blaum, Y. Blumenfeld, P. Butler, F. Wenander, _et al._ , “Storage Ring at HIE-ISOLDE,” _Eur. Phys. J. ST_ 207, pp. 1-117 (2012).
  * [6] D. Voulot, E. Bravin, M.A. Fraser, B. Goddard, Y. Kadi, D. Lanaia, A. Parfenova, A. Sosa, F. Zocca, “HIE-ISOLDE SC linac: operational aspects and commissioning preparation,” _Proc. of IPAC’12_ , to be published.
  * [7] K. Hashemi, BCAM Home Page, Brandeis University, http://alignment.hep.brandeis.edu/Devices/BCAM/.
  * [8] L. Williams, A. Bouzoud, N. Delruelle, J.-C. Gayde, Y. Leclercq, M. Pasini, J.-P. Tock, G. Vandoni, “Design of the High Beta Cryomodule for the HIE-ISOLDE Upgrade at CERN,” _Proc. of IPAC’11_ , (MOPC105) (2011).
  * [9] O. Capatina, J.-P. Brachet, G. Cuccuru, M. Pasini, T. Renaglia, M. Therasse, B. Vullierme, “Cryostat for Testing HIE-ISOLDE Superconducting RF Cavities,” _Proc. of IPAC’11_ (MOPC103) (2011).
  * [10] M. Therasse, O. Brunner, S. Calatroni, J.K. Chambrillon, B. Delaup, M. Pasini,“HIE-ISOLDE SRF Development Activities at CERN,” _Proc. of IPAC’11_ (MOPC104) (2011).
  * [11] A. Sosa, E. Bravin, M.A. Fraser, J. Galipienzo, Y. Kadi, P. Noguera, D. Voulot, C. Welsch, F. Wenander, F. Zocca, “Beam Instrumentation for the HIE-ISOLDE Linac at CERN,” _Proc. of IPAC’12_ , to be published.
  * [12] J. Bauche, “Compact Inter-cryomodule Combined Corrector Magnets for the HIE-ISOLDE Project at CERN,” _Proc. of MT-22_ (2011).
  * [13] F. Zocca, M.A. Fraser, E. Bravin, M. Pasini, D. Voulot, F. Wenander, _Nucl. Instrum. Meth. A_ 672, pp. 21 28 (2012).
  * [14] D. Voulot, “REX-linac Emittance Measurements,” CERN-BE-Note-2009-012 OP (2009).
  * [15] M.A. Fraser, F. Zocca, R.M. Jones, M. Pasini, P.A. Posocco, D. Voulot, and F. Wenander. “Longitudinal Emittance Measurements at REX-ISOLDE,” _Nucl. Instrum. and Meth. A_ 663, pp. 1-9 (2012).
  * [16] J. Dietrich, I. Mohos, J. Häuser, G. Riehl, F. Wenander, “High Sensitive Beam Emittance Analyzer,” _Proc. of EPAC’02_ , pp. 1864-1866 (2002).
  * [17] D. Voulot, “Safety file - REX-ISOLDE Linac shielding tunnel,” CERN-EDMS 1016852, to be published.
  * [18] W.F. Ohnesorge, “Heavy ion target area fast neutron dose equivalent rates,” _Health Phys._ 39, pp. 633-636 (1980).


tcb@breakable

# Automatic Curriculum Learning with Gradient Reward Signals

Ryan Campbell111Contributions from Ryan Campbell: wrote all of the code; ran
all of the experiments; generated all of the visuals and figures; formalized
the teacher-student ACL algorithm; thought of idea of using gradient reward
signals; writeup
University of California, Berkeley
<EMAIL_ADDRESS>Junsang Yoon222Contributions from Junsang Yoon:
Project direction and framework, idea generation. Analysis for experiments,
writeup.
University of California, Berkeley
<EMAIL_ADDRESS>
This study333https://github.com/riensou/automatic_curriculum_learning explores
an approach to Automatic Curriculum Learning (ACL) in reinforcement learning
(RL), focusing on the integration of gradient norm reward signals. Traditional
ACL methods primarily rely on predefined metrics that may not adequately
capture the intricacies of learning dynamics. Our work proposes a method where
the ‘teacher’ algorithm adapts the curriculum based on the gradient norm
information from the ‘student’ model, hypothesizing that these signals offer a
more refined insight into the student’s learning progression.

We developed a framework (Teacher-Student ACL) where the teacher dynamically
adjusts the learning curriculum, guided by the gradient norms of the student’s
model. This approach allows for a more responsive and tailored learning
experience. The experimental setup utilized the PointMaze, AntMaze, and
AdroitHandRelocate environments. These diverse settings provided a robust
platform to assess the versatility and effectiveness of our proposed method.

Our experiments showed that incorporating gradient norm-based rewards
significantly impacts the learning curve of the student model. In PointMaze,
we observed a marked acceleration in learning, with the student models
achieving proficiency more rapidly with the use of a teacher trained on
gradient reward signals compared to without any teacher. In the
AdroitHandRelocate environment, the student models demonstrated not only
faster learning but also improved evaluation return scores. These results
suggest that gradient norm signals offer a more nuanced understanding of
learning progression, enabling the teacher to devise more effective and
challenging curriculums.

The implications of our findings are substantial for the future of curriculum
learning in RL. The ability of the gradient norm signals to provide detailed
insights into the learning process opens new possibilities for more efficient
training of RL models. This could lead to advancements in various
applications, from autonomous systems to complex decision-making tasks.
Furthermore, our approach underscores the importance of adaptive and dynamic
curriculum strategies in enhancing the learning capabilities of AI systems.

Our study establishes the potential of gradient norm reward signals. Future
research could explore the integration of this approach in other learning
paradigms, further expanding its applicability and impact. Additionally,
comparisons to other ACL methods that use other reward signals could lead to
further understanding of the efficacy of gradient reward signals.

##  1 Introduction

Curriculum learning has emerged as a significant paradigm in artificial
intelligence, offering a structured and efficient approach to training machine
learning models. This concept, drawing inspiration from the way humans learn,
involves progressively training models on tasks of increasing complexity.
Curriculum learning stands on the premise that learning is more effective when
it follows a certain order of increasing difficulty. Just as a human student
learns better when concepts are introduced from simple to complex, machine
learning models to benefit from a similar approach. This method contrasts with
the traditional random presentation of data, offering a more guided and
focused learning trajectory. [7] [8]

In domains such as natural language processing and computer vision, curriculum
learning has shown to improve model performance, especially in dealing with
complex tasks. By starting with simpler, easier-to-learn examples and
gradually introducing more challenging data, models can develop a robust
understanding of the underlying patterns, leading to better generalization and
performance.

Recent developments in RL focus on dynamic curriculum strategies that adjust
based on the agent’s performance, optimizing the training process and
potentially leading to breakthroughs in more sophisticated and adaptive RL
systems. Our work attempts to explore the intricacies and benefits of applying
curriculum learning principles to reinforcement learning environments. Our
desired benefit is that by starting with less complex tasks, agents can
quickly grasp basic skills, which can be incrementally built upon for
increasingly complex tasks. In particular, we are most interested in being
able to automatically generate “good curriculums.” While curriculum learning
is met with widespread success, there is no clear evaluation method on the
effectiveness of a given curriculum. [4] [6]

##  2 Related Works

###  2.1 Curriculum Learning for Language Modeling

In the domain of language modeling, curriculum learning has been effectively
employed to enhance model training and performance. The application of
curriculum learning applied to Neural machine translation achieved notable
improvements in translation quality. More recently, this approach has been
adapted to the pre-training phases of transformer models, enhancing their
efficiency in handling complex language tasks. [7]

###  2.2 Curriculum Learning for Vision

In the field of computer vision, curriculum learning has also shown promising
results in fields such as object recognition, where models were gradually
exposed to increasingly complex objects and scenes. Incremental learning of
features and its integration with convolutional neural networks, led to more
effective training and better generalization in tasks like image
classification and object detection. [7]

###  2.3 Curriculum Learning for Reinforcement Learning

In reinforcement learning, curriculum learning has been employed to structure
the learning process of agents, beginning with simpler and progressively
moving to more complex environments. This step-by-step approach has not only
improved learning efficiency but also enhanced policy generalization in
agents, as evidenced in various studies. Its applications range from
sequencing tasks for robotic learning to adjusting difficulty levels in game
environments, contributing to more robust and adaptable learning processes.
Recent developments in RL focus on dynamic curriculum strategies that adjust
based on the agent’s performance, optimizing the training process and
potentially leading to breakthroughs in more sophisticated and adaptive RL
systems. [4]

The work of curriculum generation is an evolving work in RL as well. One way
of curriculum generation is a regret-based curriculum, adapting the training
distribution over the parameters of an environment, constantly producing
levels at the frontier of an agent’s capabilities, resulting in curricula that
start simple but become increasingly complex.

##  3 Formal Background

###  3.1 Automatic Curriculum Learning

In ACL, there is a student and a teacher. The student can be thought of as a
regular DRL agent, while the teacher can be thought of as a meta-learner
attempting to maximize the sample efficiency of the student.

Consider the Markov Decision Process (MDP) of the student, represented as
$(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\rho_{0})$. Here
$\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$\mathcal{P}$ is the transition function, $\mathcal{R}$ is the reward
function, and $\rho_{0}$ is the distribution of initial states. In standard
DRL algorithms, the goal is to learn a policy $\pi_{\theta}$.

Consider the following ACL problem: learn a task-selection function
$\pi_{\phi}:\mathcal{H}\to\mathcal{T}$, where $\mathcal{H}$ contains
information about the student’s progress and $\mathcal{T}$ is space of tasks
assignable by the teacher [5]. In this paper, we use the student’s rewards
from previous assignments as $\mathcal{H}$ and initial states, $\rho_{0}$, as
tasks assignable by the teacher. Thus, this problem can be understood as
another MDP for the teacher. States in this MDP are represented by initial
state and reward pairs, $(\rho_{0},r)$. Actions in this MDP are given by
initial state assignments to the student, $\rho_{0}$. Finally, the rewards are
given by some measure of learning progress by the student, denoted by
$\nabla\theta$ for now.

###  3.2 Teacher-Student ACL Algorithm

Now that the student and teacher MDPs have been formalized, consider how one
might go about training the teacher agent. Let $A_{S}$ and $A_{T}$ be DRL
algorithms, where $A_{T}$ is an on-policy DRL algorithm. Use the teacher to
generate the student dataset, $\mathcal{D}_{S}$, and facilitate student
learning by using $A_{S}$. The outcome of learning from $A_{S}$ can then be
used to construct the teacher dataset, $\mathcal{D}_{T}$. Next, use $A_{T}$ to
facilitate teacher learning. Repeat this process of student learning followed
by teacher learning until the student has achieved some threshold set
beforehand. See Algorithm 1 for a more concrete explanation of this process,
or Figure 1 for a visualization.

11:Input: DRL Algorithms $A_{S}$, $A_{T}$

22:Output: $\pi_{\theta}$, $\pi_{\phi}$

3:Repeat:

4: Repeat $k$ times:

5: $\mathcal{D}_{S}=\pi_{\phi}(\mathcal{H})$

6: $\nabla\theta=A_{S}(\pi_{\theta},\mathcal{D}_{S})$

7: $\theta\leftarrow\theta-\alpha_{S}\nabla\theta$

8:
$\mathcal{D}_{T}=\left\\{(\rho_{0},r)_{i},(\rho_{0})_{i},(\nabla\theta)_{i}\right\\}_{i=1}^{k}$

9: $\nabla\phi=A_{T}(\pi_{\phi},\mathcal{D}_{T})$

10: $\phi\leftarrow\phi-\alpha_{T}\nabla\phi$

11:Return $\pi_{\theta}$, $\pi_{\phi}$

Algorithm 1 Teacher-Student ACL Figure 1: The Teacher-Student ACL process

The hope is that the teacher will learn which environments increase the rate
of student learning, thus reducing the number of samples needed otherwise.
This is distinct from learning an efficient exploration scheme however, as the
teacher controls starting states rather than actions.

##  4 Methodology

###  4.1 Implementing Teacher-Student ACL

Many design choices are left to made from Algorithm 1 before actually
implementing anything. The two DRL Algorithms $A_{S}$ and $A_{T}$, as well as
the metric of learning progress, $\nabla\theta$ are all important decisions
that need to be made. For $A_{S}$ we used soft actor-critic and for $A_{T}$ we
used policy gradient. For learning progress, many metrics have been used in
the past such as student loss, model complexity increases, and others [3]. We
focused on two novel measures of learning progress, both based on the student
gradients encountered during training on the teacher’s assignments. These
metrics of learning progress are as follows:

  1. 1.

Average gradient size during episode:

$\frac{1}{T}\sum_{t=1}^{T}\|\left(\nabla\theta\right)_{t}\|_{2}$

  2. 2.

Size of overall episode gradient:

$\frac{1}{T}\left\|\sum_{t=1}^{T}\left(\nabla\theta\right)_{t}\right\|_{2}$

The first metric, the average gradient size during episode, accounts for how
much the student was able to learn at each step starting from the initial
state given by the teacher. This may be better than the alternative as it
still values steps that slightly counteract each other. The second metric, the
size of the overall episode gradient, accounts for the total change that comes
from the episode starting from the initial state given by the teacher.

###  4.2 Training Data

Our research utilizes OpenAI gym environments [1], particularly the PointMaze,
AntMaze, and AdroitHandRelocate environments [2], as seen in Figure 2. The
PointMaze and AntMaze environments are designed for testing reinforcement
learning algorithms in navigation and pathfinding tasks. The
AdroitHandRelocate environment is a more complicated environment that is
designed to test reinforcement learning algorithms in spatial awareness and
decision-making. Additionally, the potential starting states of the
AdroitHandRelocate environment are much higher-dimensional than the mazes,
giving the teacher more control over the student’s curriculum.

(a) PointMaze_OpenDense-v3

(b) AntMaze_OpenDense-v4

(c) AdroitHandRelocate-v1

Figure 2: The three different gym environments used in this paper

###  4.3 Model Architecture / Configurations

All our experiments were performed with the following model configurations.
These can be found in more depth in the corresponding yaml files.

  * •

Total Steps: 1,000,000

  * •

Batch Size: 256

  * •

Hidden Layer Size: 256

  * •

Number of Layers: 3

  * •

Discount Factor: 0.99

  * •

Temperature: 0.05

###  4.4 Smoothing

Each experiment ran for at least $750,000$ steps, and many values were logged
frequently. Thus, many graphs look very chaotic and are hard to interpret on
their own. Thus, the following smoothing technique was used:
$\tilde{y}_{i}=\lambda\tilde{y}_{i-1}+(1-\lambda)y_{i}$. Larger $\lambda$
values (closer to $1$) were used whenever the data was logged more frequently.
The raw, unsmoothed graphs can all be found in the appendix.

##  5 Findings

To evaluate the effectiveness of teaching, we employed several experiments
comparing the students with (1) a teacher with average gradient per time-step
as a reward, (2) a teacher with overall episode gradient as a reward, and (3)
no teacher. Our findings suggest that curriculum learning with a teacher using
gradients as a reward signal is effective. This effect is pronounced after
label smoothing, as described in the above section. For most experiments,
either one or both teaching methods were shown to outperform the no-teacher
baseline, either converging to the peak faster, or having a bigger peak
overall. Subsections below discuss these results in more depth. Dissecting the
curriculum learned by the teacher showed a clear improvement in the teacher in
generating harder tasks over time, which we approximate by the average goal
distance that the teacher’s generated curriculum provides.

###  5.1 PointMaze Performance

As shown in Figure 3, the train return and eval return converge faster with a
teacher than compared to without a teacher. The different teachers tested are
the teachers trained with different rewards.

While each agent is able to converge, it does seem that the agent with the
teacher using average gradient per time-step is slightly faster than the
teacher using overall episode gradient. This is more clear from the eval
return, while the train return they seem to be indistinguishable.

(a) Eval Return on PointMaze environment

(b) Train Return on PointMaze environment

Figure 3: PointMaze

###  5.2 AntMaze Performance

Figure 4: Teacher vs No Teacher on AntMaze environment

We were unable to get a significant outcome employing teaching on the AntMaze
environment. As exhibited by Figure 4, the agents with a teacher and no
teacher all performed similarly after a million environment steps. Note that
slight variations of $A_{S}$ were also tested (reparametrize, reinforce), but
no significant learning was able to happen regardless. This suggests that the
AntMaze environment is either too challenging for the capacity of the models
used or the learning algorithms employed.

###  5.3 AdroitHandRelocate Performance

In the AdroitHandRelocate environment, the teacher using average gradient per
time-step outperformed both the agent with the teacher using overall episode
gradient and no teacher, as shown in Figure 5. The overall episode gradient
over-performed no teacher throughout the environment steps, until converging
towards a similar return near the end.

(a) Eval Return on AdroitHandRelocate environment

(b) Train Return on AdroitHandRelocate environment

Figure 5: AdroitHandRelocate

###  5.4 Teacher Effect on Gradients

As shown in Figure 6, the student’s gradient norms varied over time in ways
that depended on the type of teacher present. In both the PointMaze and
AdroitHandRelocate environments, the agent with the teacher with average
gradient per time-step rewards started with the lowest gradients, but
eventually caught up to the agent with the teacher with overall episode
gradient rewards.

This is surprising, as the agent with the teacher with average gradient per
time-step as rewards performed the best in terms of eval return. Thus, this
seems to indicate that while the gradient norms are smaller, they are more
precise and in the correct direction.

(a) Gradient norms on PointMaze environment

(b) Gradient norms on AdroitHandRelocate environment

Figure 6: Gradient norms over time for different teachers

###  5.5 Visualizing the Learned Curriculum

In order to visualize the learned curriculum of the teacher, the distance of
the ball to the target was tracked in the AdroitHandRelocate environment.
Intuitively, one would expect an easier curriculum to put the ball further and
further away as the agent gets better at moving it, and this is what can be
seen in Figure 7.

Figure 7: Curriculum learned by teacher with respect to goal distance

This aligns with the intuitive reasoning that harder tasks are associated with
farther goals, meaning that the teacher correctly learns to generate more
difficult curriculum over time, to improve the performance of the agent.

Additionally, it should be noted that the teacher had control over other
things such as initial hand position and velocity, so the teacher did more
than just move the ball away.

##  6 Conclusion

Through the implementation of our Teacher-Student ACL algorithm, we have
created a novel instance of using student model gradients to effectively train
a teacher model. The teacher model, guided by the feedback from the student’s
gradients, succeeded in significantly enhancing the student’s learning
process. This is evident in the marked improvements in sample efficiency and
peak return, surpassing the outcomes when the student model operates without
the influence of the teacher.

A deeper examination of the curriculum devised by the teacher model reveals a
sophisticated understanding of the learning trajectory. The progression in
goal distances set by the teacher aligns remarkably with the improving
performance of the agent, underscoring the teacher model’s capacity to
intuitively and dynamically adjust the learning path to optimize outcomes.
This insight into curriculum development is a testament to the effectiveness
of our gradient-norm-based approach in shaping a more adept and responsive
teaching strategy.

Looking ahead, this study opens several exciting avenues for future
exploration. One particularly promising direction is the investigation of
different reward signals based on gradient norms. Focusing on the gradients of
specific layers within the agent’s network, such as the final layer, might
yield intriguing insights and further enhance the teacher’s capability to
fine-tune the learning process. Moreover, the current restriction to initial
state control for the teacher can be seen as a starting point rather than a
limitation. Expanding the teacher’s control to encompass more aspects of the
learning environment could provide a richer dataset for the teacher to learn
from. Such an expansion of control parameters could lead to a more
comprehensive and effective training regime, potentially unlocking new levels
of performance and efficiency in student models.

In conclusion, our research presents a meaningful advancement in the use of
gradient norms for ACL. This approach not only improves the performance of
student models in reinforcement learning but also opens up new possibilities
for refining training processes. Future investigations could explore different
gradient-based reward signals and expand the teacher’s control within the
learning environment, potentially leading to more nuanced and effective
training methods. This study contributes to the ongoing evolution of AI
training strategies, suggesting new paths for exploration and development.

## References

  * [1] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym, 2016.
  * [2] Rodrigo de Lazcano, Kallinteris Andreas, Jun Jet Tai, Seungjae Ryan Lee, and Jordan Terry.  Gymnasium robotics, 2023.
  * [3] Alex Graves, Marc G. Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu.  Automated curriculum learning for neural networks, 2017.
  * [4] Fengchun Liu, Tong Zhang, Chunying Zhang, Lu Liu, Liya Wang, and Bin Liu.  A review of the evaluation system for curriculum learning.  Electronics, 12(7), 2023.
  * [5] Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer.  Automatic curriculum learning for deep rl: A short survey, 2020.
  * [6] Clément Romac, Rémy Portelas, Katja Hofmann, and Pierre-Yves Oudeyer.  Teachmyagent: a benchmark for automatic curriculum learning in deep rl.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9052–9063. PMLR, 18–24 Jul 2021.
  * [7] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe.  Curriculum learning: A survey, 2022.
  * [8] Xin Wang, Yudong Chen, and Wenwu Zhu.  A survey on curriculum learning.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555–4576, 2022.

## Appendix

### A: Teacher Updates and Buffer Clearing

The following are two hyperparameter settings that were studied that seemed to
not make too much of an impact, but are nonetheless interesting:

  1. 1.

number of teacher updates: determines the number of times to perform an update
to the teacher network each time it is updated

  2. 2.

clear buffer: toggles whether or not to clear the student’s replay buffer
after each time the teacher gives a new initial state

(a) Eval return on AdroitHandRelocate

(b) Train return on AdroitHandRelocate

Figure 8: Different number of teacher updates with clear buffer Figure 9:
Gradient norm for different number of teacher updates with clear buffer

(a) Eval return on AdroitHandRelocate

(b) Train return on AdroitHandRelocate

Figure 10: clear buffer vs no clear buffer on AdroitHandRelocate

### B: Unsmoothed Graphs

(a) Unsmoothed graph of Figure 4

(b) Unsmoothed graph of Figure 5(a)

(c) Unsmoothed graph of Figure 5(b)

Figure 11: Unsmoothed graphs of figures

(a) Unsmoothed graph of Figure 6(b)

(b) Unsmoothed graph of Figure 3(a)

(c) Unsmoothed graph of Figure 3(b)

Figure 12: Unsmoothed graphs of figures

(a) Unsmoothed graph of Figure 6(a)

(b) Unsmoothed graph of Figure 8(a)

(c) Unsmoothed graph of Figure 8(b)

Figure 13: Unsmoothed graphs of figures

(a) Unsmoothed graph of Figure 9

(b) Unsmoothed graph of Figure 10(a)

Figure 14: Unsmoothed graphs of figures

(a) Unsmoothed graph of Figure 10(b)

(b) Unsmoothed graph of Figure 7

Figure 15: Unsmoothed graphs of figures