# ENN’s Roadmap for Proton-Boron Fusion Based on Spherical Torus

Min-sheng Liu*, Hua-sheng Xie**, Yu-min Wang, Jia-qi Dong, Kai-ming Feng,
Xiang Gu, Xian-li Huang, Xin-chen Jiang, Ying-ying Li, Zhi Li, Bing Liu, Wen-
jun Liu, Di Luo, Yueng-Kay Martin Peng, Yue-jiang Shi, Shao-dong Song, Xian-
ming Song, Tian-tian Sun, Mu-zhi Tan, Xue-yun Wang, Yuan-ming Yang, Gang Yin,
Han-yue Zhao and ENN fusion team  Email<EMAIL_ADDRESS><EMAIL_ADDRESS>Hebei Key Laboratory of Compact Fusion, Langfang 065001,
China  ENN Science and Technology Development Co., Ltd., Langfang 065001,
China

###### Abstract

Energy iNNovation (ENN) Science and Technology Development Co. is committed to
generating fusion energy in an environmentally friendly and cost-effective
manner, which requires abundant aneutronic fuel. Proton-boron ( p-11B or p-B)
fusion is considered an ideal choice for this purpose. Recent studies have
suggested that p-B fusion, although challenging, is feasible based on new
cross-section data, provided that a hot ion mode and high wall reflection can
be achieved to reduce electron radiation loss. The high beta and good
confinement of the spherical torus (ST) make it an ideal candidate for p-B
fusion. By utilizing the new spherical torus energy confinement scaling law, a
reactor with a major radius $R_{0}=4$ m, central magnetic field $B_{0}=6$ T,
central temperature $T_{i0}=150$ keV, plasma current $I_{p}=30$ MA, and hot
ion mode $T_{i}/T_{e}=4$ can yield p-B fusion with $Q>10$. A roadmap for p-B
fusion has been developed, with the next-generation device named EHL-2. EHL
stands for ENN He-Long, which literally means “peaceful Chinese dragon”. The
main target parameters include $R_{0}\simeq 1.05$ m, $A\simeq 1.85$,
$B_{0}\simeq 3$ T, $T_{i0}\simeq 30$ keV, $I_{p}\simeq 3$ MA, and
$T_{i}/T_{e}\geq 2$. The existing ST device EXL-50 was simultaneously upgraded
to provide experimental support for the new roadmap, involving the
installation and upgrading of the central solenoid, vacuum chamber, and
magnetic systems. The construction of the upgraded ST fusion device, EXL-50U,
was completed at the end of 2023, and it achieved its first plasma in January
2024. The construction of EHL-2 is estimated to be completed by 2026.

##  I Introduction

The quest for controlled fusion energy has been a longstanding dream for
humanity since the 1950s, primarily due to its carbon-free nature, high unit
energy, and abundance of resources. However, the challenges associated with
achieving this goal are both scientific and engineering-related. Several books
(cf. Chen2011 ) and papers (cf. Wurzel2022 ) review the progress in this
field.

The development of a fusion energy reactor involves three essential steps: (1)
selection of fusion fuels; (2) selection of a confinement approach; and (3)
selection of a method to utilize or convert the energy produced. Over the past
70 years, much attention has been focused on the second step, and the
feasibility or progress of a confinement approach can be described by the
product of three physical parameters: ion density ($n_{i}$), ion temperature
($T_{i}$), and energy confinement time ($\tau_{E}$), commonly referred to as
the Lawson criteria Lawson1957 ; Wurzel2022 . To confine plasmas efficiently,
various approaches have been considered. Among these, magnetic confinement,
which uses a magnetic field to confine the plasma, and inertial confinement,
which initiates nuclear fusion reactions by compressing and heating fuel
targets, are considered the two most promising techniques. Significant
progress has been made in both magnetic confinement fusion and inertial
fusion. For example, in JET’s 1997 Deuterium-Tritium (DT) experiments, a Q
value, defined as the ratio between the output fusion power and the input
power, of 0.67 was achieved Gibson1998 ; Reinders2021 . In the latest DT
experiments in 2021, a value of 0.33 was achieved for 5 seconds, setting a new
record for magnetic confined fusion energy output with 59 MJ Gibney2022 . In
recent experiments at NIF, a Q value of approximately 1.5 was achieved with
fusion energy output around 3.15 MJ Tollefson2022 , indicating that more
fusion power was produced than input power, or ignition was achieved in
inertial confinement fusion.

Considerable effort has been dedicated to magnetic confinement fusion. Various
configurations have been proposed to confine plasmas, including tokamaks
Wesson2004 , stellarators Nespoli2022 , field reversed configuration (FRC)
Steinhauer2011 , and magnetic mirrors Post1987 . Tokamaks and stellarators are
toroidal devices that utilize twisted magnetic fields to confine high-
temperature plasma. Both configurations have demonstrated much higher plasma
parameters than other magnetic confinement configurations to date. In
stellarators, twisted magnetic field lines are created using external coils,
while in tokamaks, field lines are twisted by a combination of the toroidal
magnetic field generated by toroidal field coils (TF) and the poloidal
magnetic field generated by poloidal field coils (PF) and the plasma current.
Tokamaks are generally easier to construct than stellarators, and over one
hundred tokamaks have been built and extensively studied both experimentally
and theoretically. This has provided a solid foundation for the design of
next-generation tokamaks such as ITER Ikeda2010  and CFETR Chan2015 .

In addition to conventional tokamaks with a large aspect ratio, defined as
$A=R_{0}/a$ where $R_{0}$ and $a$ denote the major and minor radius,
respectively, the spherical tokamak/torus (ST) with a small aspect ratio has
been proposed Peng1986 . The small aspect ratio significantly influences
magnetohydrodynamic (MHD) stabilities, allowing ST to operate with high
$\beta_{T}$, the ratio of plasma pressure to toroidal magnetic pressure. This
enables a more efficient use of the magnetic field, benefiting compact
devices. Experiments have been conducted on START Gryaznevich1998 , MAST
Sykes2001 , NSTX Menard2003 , and Globus-M Gusev2009 . Recent experimental
results from the Globus-M2 device have demonstrated that energy confinement in
ST can be significantly increased by doubling the toroidal magnetic field
Kurskiev2022 . Furthermore, a new scaling law for ST has been developed,
showing promising energy confinement time results at high toroidal magnetic
fields, i.e., $\tau_{E}\propto I_{p}^{0.53}B_{T}^{1.05}$ Kurskiev2022 . One of
the main drawbacks of ST is the limited space within the central solenoid (CS)
coil, making it challenging to increase the toroidal field. However, recent
advances in high-temperature superconductor (HTS) technology have made it
possible to achieve high magnetic fields in compact devices Creely2023 . ST40,
designed with a magnetic field of 3T, has operated at a toroidal field (TF) up
to 2 T at $R_{0}$ = 40 cm, although the TF magnet is made from Cu and is
liquid nitrogen (LN2) cooled Gryaznevich2022 . Recently, ion temperature
plasmas of around 9.6 keV ($>100$ million degrees Kelvin) were achieved in
ST40 McNamara2023 , representing a significant milestone for compact fusion
devices. The UK has also initiated the ambitious Spherical Tokamak for Energy
Production (STEP) program (https://step.ukaea.uk/), which aims to develop a
compact prototype reactor by 2040 capable of delivering a net electric power
of $P_{el}>$100 MW to the gridMeyer2022 .

The p-11B fusion has been studied both theoretically Nevins2000 ; Geser2020 ;
Putvinskia2019  and experimentally. On the experimental side, research has
been conducted using laser-produced plasmas Margarone2022  and particle
accelerators employing beam-target fusion Stave2011 . Recently, clear
experimental measurements of $\alpha$ particles produced by p-11B fusion in
the Large Helical Device (LHD) have been achieved. This involved using high-
energy neutral beams and boron powder injection in high-temperature fusion
plasma Magee2023 . These findings represent the first experimental
demonstration of p-11B fusion in a magnetically confined plasma.

The significant accomplishments mentioned above have significantly enhanced
confidence in the potential of commercial fusion energy. However, for fusion
energy to become more competitive with other sources, such as fission and
solar energy, considerations beyond the physics aspects are necessary.
Achieving commercial fusion energy in the near future requires careful
consideration of various options. In this work, we describe ENN’s chosen
roadmap for fusion energy, encompassing the selection of fuels, confinement
approaches, and a research plan for the next 10-15 years.

This paper is organized as follows: In Sec. II, we discuss the choice of
fuels. In Sec. III, we describe a newly built system code that demonstrates
the feasibility of a proton-Boron fusion energy reactor based on a spherical
torus (ST). In Sec. IV, we summarize the key technologies that need
resolution. In Sec. V, we present the ENN fusion roadmap. Finally, in Sec. VI,
we provide a summary and discuss the findings.

##  II Choice of the Fuels

In order to achieve commercial fusion energy, it is essential to utilize eco-
friendly fusion sources. This implies a need for low neutron yield or
aneutronic fusion. Additionally, the cost per unit of fusion power must be
sufficiently low to compete with other energy sources such as wind,
photovoltaic (PV), and fission. To fulfill these requirements, the fuel must
be abundant and accessible, energy conversion must be high, and the technology
must be suitable for the distributed energy market.

A comparison of the advantages and disadvantages of different fusion fuels is
presented in Table 1. D-T fusion stands out as the easiest reaction to achieve
fusion energy gain due to its high reaction cross section at relatively low
temperatures, as well as its high energy yield for each reaction. Currently,
the mainstream roadmap for commercial fusion is based on D-T fusion.

However, D-T fusion poses significant challenges, particularly concerning
energetic neutron shielding and tritium breeding. Currently, there are no
materials available for effective neutron shielding, and achieving a tritium
breeding rate (TBR) larger than 1.2, which is necessary, may be difficult in
the short term. The 6Li resource for tritium breeding is also limited.
Concerns about D-T fusion for energy have been raised from an engineering
perspective Lidsky1983 ; Reinders2021 . These concerns, combined with the
challenges mentioned above, make D-T fusion challenging to compete with
fission Lidsky1983 ; Lerner2023 .

D-D fusion, on the other hand, is relatively easy to achieve due to its large
reaction cross-section and abundance of fuel. However, it also produces high-
energy neutrons. D-3He fusion produces fewer neutrons and has a relatively
large reaction cross-section, making it an attractive choice for fusion.
Nevertheless, the disadvantages of D-3He fusion include the side effects of
D-D fusion and the scarcity of 3He storage on Earth. If 3He can be mined from
the moon, it may make D-3He fusion more feasible, although the reserves and
cost of 3He remain open questions in the short term.

Figure 1: The triple products (a) and the thermal fusion reactivity (b) for
D-T, D-D, D-3He, and p-11B fusion, and the strategy to make p-11B fusion
possible (c).

The p-11B fusion is an attractive fusion choice for several reasons. From the
fuel perspective, abundant boron is readily available on Earth. Regarding
fusion products, p-11B fusion exclusively generates charged $\alpha$
particles, which have the potential for active control and direct electricity
generation through charge separation. It’s important to note that side
reactions, such as p+11B$\rightarrow$ n+11C, producing low-energy neutrons
with an energy of 2.765 MeV, exist. The fraction of this fusion reaction is
approximately $10^{-5}$, and the low-energy neutrons are more manageable.
These properties align with the requirements for commercial fusion. However,
it should be acknowledged that the reaction threshold for p-11B fusion is
exceptionally high due to its relatively low fusion cross-section. The
criteria for ignition and fusion reactivity for various fusion reactions are
illustrated in Figure 1. Notably, the required criteria for p-11B fusion are
three orders of magnitude higher than those for D-T fusion. The fusion
reactivity, $\langle\sigma v\rangle$, significantly improves when the
effective temperature, $T_{eff}=(m_{1}T_{2}+m_{2}T_{1})/(m_{1}+m_{2})$,
exceeds 200 keV. Two sets of different cross-section data are utilized in
p-11B fusion reactivity calculations Nevins1998 ; Sikora2016 . Despite these
challenges, it is believed that high ion temperatures can be achieved by
utilizing effective ion cyclotron range of frequency (ICRF) heating and high-
energy negative ion-based neutral beam injection (NNBI) Ikeda2006 . Notably,
both TFTR Hawryluk1998  and JT-60 Ishida1996  tokamaks have achieved ion
temperatures $T_{i}\simeq 45$ keV.

After this analysis, choosing p-11B as the fuel for commercial fusion appears
reasonable. The primary reason is not that it is easy, but rather that we have
limited choices. Due to the scarcity of fuels, options like D-T and D-3He are
not viable. Additionally, to avoid dealing with neutrons, D-D fusion is not a
preferred choice.

Table 1: The comparison between different fusion fuels. Nuclear reaction | Advantages | Disadvatages |  Rough fuel price (RMB/g) |  Consumption in 2050 | Rough reserves
---|---|---|---|---|---
D-T | Easiest way to fusion |  High energy neutron wall material protection, Tritium breeding | T: 10 million | 3000 tons | 10 kg
D-D | Relatively easy |  High energy neutrons, peak reaction cross section low | D: 30 | 10000 tons | 45 trillion tons
D-3He |  few neutrons, low neutron energy |  3He expensive, D-D side effects |  3He: 1 million | 3000 tons |  produced by nuclear reactions, mined on the moon
p-11B |  With few neutrons and cheap raw fuels, $\alpha$ particles direct energy coversion |  Extreamely high reaction threshold |  11B: 5 | 10000 tons |  170 million tons

##  III System code calculations

System codes utilizing scaling laws from experiments have been developed and
employed Shimada2007 ; Costley2015 . However, to design a spherical tokamak
using p-11B as fuel, a new system code is required. This section will
introduce our new system code based on the updated spherical tokamak scaling.

###  III.1 Flow chart of the code

The new code is developed and streamlined from our original system code
Cai2022 . The flow chart of the code is illustrated in Figure 2.

The input parameters encompass the size of the vacuum vessel or the shape of
the plasma (explained in detail in subsection III.2), density and temperature
profiles, toroidal magnetic field ($B_{0}$), plasma current ($I_{p}$), and
energy confinement time ($\tau_{E}$). The profiles are expressed as

$\displaystyle n(x)=n_{0}(1-x^{2})^{S_{n}},$ (1) $\displaystyle
T(x)=T_{0}(1-x^{2})^{S_{T}},$ (2)

where $n_{0}$ and $T_{0}$ are the density and temperature on the magnetic
axis, and $S_{n}$ and $S_{T}$ are the shape factors of density and
temperature. Here, $x=r/a$ represents the normalized radial displacement,
where $r$ is the displacement from the magnetic axis along the radial
direction, and $a$ is the minor radius. The profiles can exhibit more H-mode-
like behavior when decreasing the shape factors, i.e., having a steep gradient
in the edge region. By using these expressions, the volume-averaged and line-
averaged plasma density and temperature can be calculated as follows:

$\displaystyle\langle
n\rangle=\int_{0}^{1}n_{0}\left(1-x^{2}\right)^{S_{n}}2xdx=\frac{n_{0}}{1+S_{n}},$
(3) $\displaystyle\langle
T\rangle=\int_{0}^{1}T_{0}\left(1-x^{2}\right)^{T_{n}}2xdx=\frac{T_{0}}{1+S_{T}},$
(4) $\displaystyle\langle
n\rangle_{l}=\int_{0}^{1}n_{0}\left(1-x^{2}\right)^{S_{n}}dx=\frac{\sqrt{\pi}}{2}\frac{\Gamma(S_{n}+1)}{\Gamma(S_{n}+1.5)}n_{0},$
(5) $\displaystyle\langle
T\rangle_{l}=\int_{0}^{1}T_{0}\left(1-x^{2}\right)^{S_{T}}dx=\frac{\sqrt{\pi}}{2}\frac{\Gamma(S_{T}+1)}{\Gamma(S_{T}+1.5)}T_{0}.$
(6)

The energy confinement time is also an input parameter in the new code,
enabling the calculation of heating power and fusion power.

The output parameters of the new code include plasma size, averaged density
and temperature ($n_{avg}$ and $T_{avg}$), stored energy ($W_{th}$), fusion,
$\alpha$ particle power, and radiation power ($P_{fus}$, $P_{\alpha}$, and
$P_{rad}$). Combining the fusion power and energy confinement time allows the
determination of the required heating power ($P_{heat}$) from the power
balance equation, and the fusion energy gain factor ($Q_{fus}$) can be
calculated. The plasma $\beta_{T}$ can be determined using profiles and
toroidal magnetic field. Finally, scaling factors such as Greenwald density
fraction ($n_{G}$), H factors ($H_{98}$ and $H_{ST}$), L-H mode transition
power ($P_{LH}$), safety factor ($q$), and normalized beta ($\beta_{N}$) can
be computed.

Figure 2: The flow chart of the system code. A major difference from other
system codes is that certain scaling law-relevant parameters, such as
$\tau_{E}$, are set as input parameters to avoid sensitivity.

###  III.2 Plasma shape

The geometry of the plasmas is illustrated in Figure 3. The shape of the
plasma can be fixed when parameters such as the major radius ($R_{0}$), aspect
ratio ($A$), triangularity ($\delta$), elongation ($\kappa$), and gap between
the plasma and the vacuum wall ($g$) are specified. The initial design of the
vacuum vessel can be determined based on these parameters. The plasma volume
can be calculated using

$\displaystyle V_{p}=2\pi^{2}\kappa\left(R_{0}-\delta
a\right)a^{2}+\frac{16}{3}\delta\pi\kappa a^{3}.$ (7)

The volume of the device is calculated by using

$\displaystyle
V_{d}=2\pi^{2}\kappa\left(R_{0}-\delta\left(a+g\right)\right)\left(a+g\right)^{2}+\frac{16}{3}\delta\pi\kappa(a+g)^{3}.$
(8)

The surface area of the plasma, $S_{p}$, and the surface area of the device,
$S_{w}$, are

$\displaystyle
S_{p}=\left(4\pi^{2}\frac{R_{0}}{a}\kappa^{0.65}-4\kappa\delta\right)a^{2},$
(9) $\displaystyle
S_{w}=\left(4\pi^{2}\frac{R_{0}}{a+g}\kappa^{0.65}-4\kappa\delta\right)(a+g)^{2}.$
(10)

Figure 3: The plasma shape.

It should be noted that we have set the major radius and the magnetic axis at
the center of the plasma, i.e., $R_{0}=(R_{max}+R_{min})/2$. In realistic
plasma equilibrium, the magnetic axis would experience a shift outside due to
the Shafranov shift. We omit this difference for simplification.

###  III.3 Species

There are multiple species inside the plasma, including two ions for fusion
reactions, helium particles, and other impurities, with their densities
represented by $n_{1}$, $n_{2}$, $n_{He}$, and $n_{imp}$, respectively, and
the charge numbers represented by $Z_{1}$, $Z_{2}$, $Z_{He}$, and $Z_{imp}$.
The ion and electron density can be determined according to the quasi-neutral
condition, i.e.,

$\displaystyle n_{i}=\frac{n_{1}+n_{2}}{1+\delta_{12}}+n_{He}+n_{imp},$ (11)
$\displaystyle
n_{e}=\frac{n_{1}Z_{1}+n_{2}Z_{2}}{1+\delta_{12}}+n_{He}Z_{He}+n_{imp}Z_{imp},$
(12)

where $\delta_{12}=1$ when fusion particles 1 and 2 are identical, while
$\delta_{12}=0$ when they are different. The average and effective charge
numbers can be calculated by

$\displaystyle Z_{i}=\frac{n_{e}}{n_{i}},$ (13) $\displaystyle
Z_{eff}=\frac{\frac{n_{1}Z_{1}^{2}+n_{2}Z_{2}^{2}}{1+\delta_{12}}+n_{He}Z_{He}^{2}+n_{imp}Z_{imp}^{2}}{n_{e}}.$
(14)

The effective density of the two fusion ions is presented by

$n_{12}=\frac{n_{1}+n_{2}}{1+\delta_{12}},$ (15)

and the ratio of the fusion ions is $f_{12}=n_{12}/n_{i}$. The ratios of the
helium ions and impurities are represented by $f_{He}=n_{He}/n_{i}$ and
$f_{imp}=n_{imp}/n_{i}$, respectively. Furthermore, the ratios of the two
separate fusion ions can be calculated by $x_{1}=n_{1}/n_{12}$ and
$x_{2}=n_{2}/n_{12}$. Under this definition, and together with plasma quasi-
neutrality, the following relations need to be satisfied,

$\displaystyle f_{12}+f_{He}+f_{imp}=1,$ (16)
$\displaystyle\frac{f_{12}(x_{1}Z_{1}+x_{2}Z_{2})}{1+\delta_{12}}+f_{He}Z_{He}+f_{imp}Z_{imp}=1.$
(17)

Assuming that the temperature of the ions is the same, while the temperature
ratio of the electrons and ions is fixed at $f_{T}$, i.e., $T_{e}=f_{T}T_{i}$,
the effect of the ion and electron temperature ratio can be evaluated.

###  III.4 Fusion power

The fusion power of the two fusion ions can be calculated by

$P_{fus}=\frac{1}{1+\delta_{12}}n_{1}n_{2}\langle\sigma v\rangle
Y=\frac{Y}{1+\delta_{12}}n_{10}n_{20}\Phi V_{p},$ (18)

where $Y$ is the total energy released from the fusion reaction, and $\sigma$
represents the cross-section of the two fusion ions. Here $n_{10}$ and
$n_{20}$ are the densities at the magnetic axis for the two fusion ions.
Considering the radial distribution of the ion temperature, the fusion
reactivity can be calculated as follows,

$\Phi=2\int_{0}^{1}(1-x^{2})^{2S_{n}}\langle\sigma v\rangle xdx.$ (19)

where $\langle\sigma v\rangle$ is related to the effective ion temperature
$T_{i}(x)$. Assuming there is a potential enhancement factor $f_{\sigma}$
Xie2023 , for non-Maxwellian distributed ions or from other effects, the
fusion rate can be presented by

$\langle\sigma v\rangle=f_{\sigma}\langle\sigma v\rangle_{M},$ (20)

where $\langle\sigma v\rangle_{M}$ can be used to calculate the fusion rate
for a Maxwellian ion temperature distribution. In this way, the effect of the
fusion rate enhancement can be evaluated.

The fusion equation for D-T fusion is

${\rm D+T}\rightarrow^{4}_{2}{\rm He(3.5MeV)}+^{1}_{0}n{\rm(14.1MeV)}.$ (21)

Here $Y=17.6$ MeV, and the formulas for D-T fusion cross-section are used
Bosch1992 .

The fusion equation for p-${}^{11}B$ fusion is

${\rm p+}^{11}{\rm B}\rightarrow 3^{4}_{2}{\rm He(8.7MeV)}.$ (22)

Here $Y=8.7$ MeV, and the fusion cross-section is obtained from the fitting
data by Nevins and Swain Nevins2000 .

After calculating the fusion power, the fusion gain factor can be obtained by

$Q=\frac{P_{fus}}{P_{aux}},$ (23)

where $P_{aux}$ is the auxiliary power used to sustain the plasma.

###  III.5 Power balance

The power balance equation is given as

$\frac{dW_{th}}{dt}=P_{\alpha}+P_{aux}-P_{brem}-P_{cycl}-\frac{W_{th}}{\tau_{E}}=0,$
(24)

where $W_{th}$ is the plasma thermal energy, and $P_{\alpha}$ is the heating
power by $\alpha$ particles. $P_{brem}$ and $P_{cycl}$ are the bremsstrahlung
and cyclotron radiation loss power, respectively.

In the non-relativistic case, the bremsstrahlung power can be evaluated using
the following expression Nevins1998

$\displaystyle P_{brem}$ (25)
$\displaystyle=C_{B}n_{e0}^{2}\sqrt{k_{B}T_{e0}}V_{p}\left\\{Z_{eff}\left[\frac{1}{1+2S_{n}+0.5S_{T}}+\right.\right.$
$\displaystyle\frac{0.7936}{1+2S_{n}+1.5S_{T}}\frac{k_{B}T_{e0}}{m_{e}c^{2}}+\left.\frac{1.874}{1+2S_{n}+2.5S_{T}}\left(\frac{k_{B}T_{e0}}{m_{e}c^{2}}\right)^{2}\right]$
$\displaystyle+\left.\frac{3}{\sqrt{2}(1+2S_{n}+1.5S_{T})}\frac{k_{B}T_{e0}}{m_{e}c^{2}}\right\\}(MW),$
(26)

where $C_{B}=5.34\times 10^{-43}$, $k_{B}T_{e0}$ and $m_{e}c^{2}$ are in keV,
and the electron density in $m^{-3}$. Here $k_{B}$ and $m_{e}$ are the
Boltzmann constant and electron rest mass, respectively.

The cyclotron power can be obtained from Kukushikin2009

$\displaystyle P_{cycl}$ $\displaystyle=$ $\displaystyle 4.14\times
10^{-7}B_{T0}^{2.5}(1-R_{w})^{0.5}V_{p}a_{eff}^{-0.5}\times$ (27)
$\displaystyle
n_{eff}^{0.5}T_{eff}^{2.5}\left(1+\frac{2.5T_{eff}}{511}\right)(MW).$

Here, $R_{w}$ denotes the reflection rate of the wall, and it is important to
reuse the cyclotron power by reflecting it back to the plasma. The effective
density, $n_{eff}$, the effective minor radius, $a_{eff}$, and the effective
temperature, $T_{eff}$, are defined as

$\displaystyle n_{eff}=\langle n_{e}\rangle=\frac{n_{e0}}{1+S_{n}},$ (28)
$\displaystyle
a_{eff}=a\kappa^{0.5},T_{eff}=T_{e0}\int_{0}^{1}(1-x^{2})^{S_{T}}dx.$ (29)

Note that the definition of $T_{eff}$ is different from $\langle
T_{e}\rangle$.

The energy confinement time, $\tau_{E}$, is a critical parameter in magnetic
confinement fusion and serves as an input parameter in the new system code.
The scaling law for the energy confinement time in conventional tokamaks is
described by the IPB98(y,2) model Shimada2007 , expressed as

$\tau_{E}^{CT}=0.0562I_{p}^{0.93}B_{T}^{0.15}P_{loss}^{-0.69}n_{e}^{0.41}M^{0.19}R_{0}^{1.97}\epsilon^{0.58}\kappa^{0.78}$
(30)

where $M$ is the average mass number.

For spherical tokamaks, the energy confinement time can be described by the
following scaling law Kurskiev2022

$\tau_{E}^{ST}=0.066I_{p}^{0.53}B_{T}^{1.05}P_{loss}^{-0.58}n_{e}^{0.65}R_{0}^{2.66}\kappa^{0.78}.$
(31)

Both of these scaling laws are used in the new system code to compare the
different confinement factors between conventional tokamaks and spherical
tokamaks. The H-factor, defined as the ratio of the required energy
confinement time to the scaling, is utilized to evaluate the confinement of
the tokamak. It can be calculated as follows,

$\displaystyle H_{98}=\frac{\tau_{E}}{\tau_{E}^{CT}},$ (32) $\displaystyle
H_{ST}=\frac{\tau_{E}}{\tau_{E}^{ST}}.$ (33)

###  III.6 Output parameters

The toroidal beta can be calculated using

$\beta_{T}=\frac{2\mu_{0}k_{B}\int(n_{i}T_{i}+n_{e}T_{e})dV}{B_{T}^{2}}=\frac{2\mu_{0}k_{B}(n_{i}T_{i}+n_{e}T_{e})}{(1+S_{n}+S_{T})B_{T}^{2}},$
(34)

where $\mu_{0}$ is the permeability of free space.

The safety factor is calculated by default using Costley2015

$q=\frac{2\pi\kappa
B_{T}a^{2}\kappa}{\mu_{0}R_{0}I_{p}}=\frac{5B_{T}a^{2}\kappa}{R_{0}I_{p}},$
(35)

and also with ST correction

$q^{*}=\frac{\pi(1+\kappa^{2})B_{T}a^{2}\kappa}{\mu_{0}R_{0}I_{p}}.$ (36)

Generally, the safety factor should be larger than 2 to avoid disruption. The
normalized beta of the plasma can be calculated using

$\beta_{N}=100\beta_{T}\frac{aB_{T}}{I_{p}},$ (37)

and it is required that $\beta_{N}<12/A$ to stabilize instabilities.

###  III.7 Code validation

The ITER design parameters are employed to benchmark the new system code, and
the comparison results are presented in Table 2. The outcomes obtained from
the new simplified system code closely align with the results reported by
Costley Costley2015  for the same input parameters.

Table 2: Comparison of the new system code with Costley’s results for ITER parameters. ITER parameters | Costley15 | ENN model
---|---|---
Central ion temperature, $T_{i0}$ (keV) | 25 | 25
Central density, $n_{e0}$ (m-3) | $0.77\times 10^{20}$ | $0.77\times 10^{20}$
Confinement time, $\tau_{E}$ (s) | 2.0 | 2.0
Beta, $\beta$ | 0.02 | 0.02
Magnetic field, $B_{0}$ (T) | 5.18 | 5.18
Major radius, $R_{0}$ (m) | 6.35 | 6.35
Aspect ratio, $A$ | 3.43 | 3.43
Plasma currrent, $I_{p}$ (MA) | 9.2 | 9.2
Heating power, $P_{heat}$ (MW) | 70 | 70.4
Fusion power, $P_{fus}$ (MW) | 350 | 356
Fusion gain, $Q$ | 5 | 5.1

##  IV Technologies to be resolved

The general strategy to make p-11B fusion possible is shown in Fig. 1c, i.e.,
reduce the requirement by increasing the reactivity and reducing the
radiation, and increasing the triple products. In pursuit of p-11B fusion, it
is imperative to enhance the triple product, particularly by elevating the ion
temperature and lowering the reaction threshold conditions. While high $T_{i}$
is essential for increased fusion reactivity, it is crucial to manage electron
bremsstrahlung loss power, which can surpass fusion power if $T_{e}$ matches
$T_{i}$ . Hence, hot ion modes with $T_{i}/T_{e}=4$ are deemed necessary
Cai2022 .

Augmenting the fusion reactivity becomes feasible with a larger fusion
reaction cross-section. Additional physical mechanisms, such as nonthermalized
distribution Xie2023 ; Kong2024  and potential avalanche processes Hora2017
in p-11B fusion reactions, hold promise for further boosting fusion
reactivity. The primary scientific requirements for realizing p-11B fusion are
succinctly outlined in Figure 4.

Using the system code described in Section III, the optimal parameters for
achieving p-11B fusion gain ($Q>1$) are as follows: magnetic field strength
($B_{0}$) in the range of $4-7$ T, ion temperature to electron temperature
ratio ($T_{i}/T_{e}$) of $\geq 3$, ion temperature ($T_{i0}$) between
$100-300$ keV, electron density ($n_{e0}$) of $1-3\times 10^{20}$ m-3, and
energy confinement time ($\tau_{E}$) ranging from $20-50$ s. Additionally, if
the potential fusion enhancement factor $f_{\sigma}$ is high (e.g.,
$f_{\sigma}>2$), the other requirements can be significantly reduced. By
utilizing the new ST energy confinement scaling law, a reactor with major
radius ($R_{0}$) of $4$ m, central magnetic field ($B_{0}$) of $6$ T, central
temperature ($T_{i0}$) of $150$ keV, plasma current ($I_{p}$) of $30$ MA, and
hot ion mode ($T_{i}/T_{e}$) of $4$ can yield p-11B fusion with $Q>10$.

The attainment of an extremely high p-11B fusion gain is challenging. To
illustrate the difficulty, we compare the cases of Conventional Tokamaks (CT)
and Spherical Tokamaks (ST) in Table 3 with a set of highly optimized
conditions for p-11B fusion, specifically $f_{\sigma}=5$ and $T_{i}/T_{e}=4$.
The primary distinctions between CT and ST lie in the confinement scaling law
and plasma beta. As a result, we observe that CT can only achieve $Q\simeq 3$
with $B_{0}=12$ T, necessitating an even higher H factor of $H_{98}$=3.41. In
contrast, ST can achieve $Q=30$ with $H_{ST}\simeq 1$.

Figure 4: Key issues for spherical torus p-B fusion. Primary factors can
interact with each other. If one technological breakthrough is achieved, other
technical requirements may be lowered accordingly.

Benefiting from the high $\beta_{T}$ and favorable energy confinement time of
ST, the required magnetic field and heating power are significantly lower
compared to conventional tokamaks. Consequently, spherical tokamaks emerge as
more suitable candidates for p-11B fusion. However, it’s crucial to note that
currently, the plasma current, temperature, and energy confinement time of
spherical tokamaks are considerably lower than those of conventional tokamaks
and need validation at high parameters. Verifying the confinement properties
at elevated plasma parameters constitutes a crucial task in ENN’s fusion
roadmap. Additionally, it is emphasized that other magnetic confinement
approaches typically exhibit even lower confinement time scaling than tokamaks
and, therefore, may not be a prioritized option in ENN’s strategy.

Table 3: The comparison of parameters of a CT and ST from the system code. Parameters | CT | ST
---|---|---
Average ion temperature, $T_{i,avg}$(keV) | 33 | 33
Average electron density, $n_{e,avg}(10^{20}m^{-3})$ | 1.65 | 0.66
Confinement time, $\tau_{E}(s)$ | 5 | 19.3
Beta, $\beta_{T}$ | 0.038 | 0.32
Central Magnetic field, $B_{0}(T)$ | 12 | 2.6
Major radius, $R_{0}(m)$ | 4 | 3.2
Fusion Gain, $Q$ | 3.2 | 30
Aspect ratio, $A$ | 3.5 | 1.7
$n_{0}/n_{G}$, | 1 | 0.76
Safty factor, q | 3.27 | 2.16
Elongation, $\kappa$, | 2.5 | 3.3
Fusion power, $P_{fus}(MW)$, | 241 | 107
Heating power, $P_{fus}(MW)$, | 74.5 | 3.56
Plasma currrent, $I_{p}(MA)$ | 15 | 22
H factor | ($H_{98}$) 3.41 | ($H_{ST}$)0.98
$T_{i}/T_{e}$ | 4 | 4
p-11B Fusion enhancement factor $f_{\sigma}$ | 5 | 5

After summarizing the key scientific issues, the key technological challenges
for ST p-11B fusion can also be outlined accordingly:

  * •

High heat load materials: The compact size of ST is advantageous for the
commercialization of fusion energy but results in a higher particle and energy
flux on the plasma facing components, especially the divertor. Effective
management of these challenging conditions requires the development of new
materials or the implementation of innovative design strategies.

  * •

High power supply: In addition to the power supply for the toroidal and
poloidal magnetic coils, increased power demands for heating and current drive
actuators are anticipated.

  * •

High efficient ion heating: The ion temperature required for p-11B fusion is
substantially higher than that for D-T fusion, necessitating advanced ion
heating methods. Promising approaches include Neutral Beam Injection (NNBI) at
high energies with long-duration operation or Ion Cyclotron Resonance Heating
(ICRF). However, mature technologies for these methods are yet to be fully
developed.

  * •

High-temperature superconductivity: Although the inherent properties of ST
reduce the demand for a strong toroidal magnetic field, advancements in high-
temperature superconducting materials hold the potential to optimize the
engineering design of ST, making them more efficient for fusion applications.

  * •

Low aspect ratio engineering: The use of a low aspect ratio in fusion reactors
offers benefits such as improved plasma confinement and reduced tokamak size.
However, it imposes limitations on central solenoid space, affecting available
volt-seconds for ohmic heating and plasma feedback control. Addressing this
challenge requires the development of new technologies for effective heating
and control in low aspect ratio fusion systems.

  * •

Advanced divertor: Controlled production of $\alpha$ particles in p-11B fusion
enables direct electric power generation. Realizing this potential requires
new conceptual designs and experimental validations to ensure the feasibility
and practicality of this approach.

##  V ENN fusion roadmap

In this section, we will outline a new commercial fusion roadmap for p-11B
utilizing spherical torus.

The roadmap for ENN fusion research is depicted in Figure 5. A noteworthy
milestone in this progression is the establishment of the central solenoid-
free spherical torus, EXL-50, which commenced operations in 2018 (detailed in
Shi2022 ). EXL-50’s primary focus is the exploration of current drive
techniques through Electron Cyclotron Resonance Heating (ECRH). A major
upgrade is on the horizon for the EXL-50 device, involving the incorporation
of a central solenoid and a doubling of the toroidal magnetic field strength.
This enhancement plan also encompasses improvements to NBI, ICRF, low hybrid
wave heating, and ECRH. These enhancements are anticipated to facilitate the
achievement of hot ion mode discharges with ion temperatures surpassing 1.0
keV. Furthermore, the subsequent project, EXL-50U (Fig.6), is designed to
develop crucial technologies for ST p-11B fusion. It also aims to conduct
feasibility studies covering both the physics and engineering aspects for
EHL-2 (Fig.7), which is slated to be the next-generation machine in the
ongoing fusion research initiative.

In the initial phase of the commercial fusion roadmap for p-11B utilizing a
spherical torus, a dedicated research and development platform, EHL-2, will be
constructed. The primary objectives of Phase I include achieving the p-11B
thermal fusion reaction and validating the existence of hot ion modes
characterized by a high ion-to-electron temperature ratio ($T_{i}/T_{e}\geq
2$) at elevated ion temperatures. Additionally, the platform will focus on
experimental validation, specifically measuring and confirming the presence of
$\alpha$ particles resulting from the fusion reaction. The research effort
will explore novel physics phenomena influenced by a high magnetic field
($B_{T}\simeq$ 3T), conduct in-depth studies of energetic particle physics,
and develop control methods for managing these energetic particles.
Diagnostics related to energetic particles will be established and refined.
The platform will also serve as a testing ground for a new divertor design
that facilitates direct energy conversion, aiming for improved efficiency in
harnessing fusion energy. Furthermore, Phase I will involve fine-tuning and
optimizing strategies for achieving ion heating and current drive with maximum
efficiency. Through these targeted objectives, Phase I aims to lay the
groundwork for subsequent phases in the pursuit of realizing commercial p-11B
fusion using a spherical torus.

Phase II of the commercial fusion roadmap for p-11B using a spherical torus
will concentrate on elevating plasma parameters to closely emulate the
conditions required for practical fusion. The construction of a new spherical
torus, EXL-3, will be a key aspect of this phase, aiming to achieve a
significant level of p-11B fusion power generation. The primary objective
during Phase II is to increase the ion temperature, with specific targets set
at 70 keV for EHL-3A and 140 keV for EHL-3B. These targets bring the ion
temperature closer to the required levels ($T_{i}$) for p-11B fusion.
Additionally, Phase II will involve a detailed analysis of engineering
constraints that arise under fusion-related conditions, ensuring the safe and
efficient operation of the fusion reactor. A critical aspect of this analysis
will be the testing and selection of first-wall materials capable of
withstanding the harsh nuclear environment associated with fusion reactions.

Phase III represents the apex of efforts towards achieving commercial p-11B
fusion. The primary emphasis during this phase will be on optimizing the
fusion process for practical and economically viable energy generation. As
part of Phase III, a comprehensive strategy for cost reduction will be
explored and implemented to enhance the commercial viability of p-11B fusion
technology.

Figure 5: The ENN’s roadmap for ST p-11B fusion. Figure 6: The upgrade of EXL-50 to EXL-50U was completed in 2023. EXL-50U achieved its first plasma in January 2024. Figure 7: Preliminary design of EHL-2.  Table 4: The preliminary target parameters for ENN ST devices. Parameters | EXL-50 | EXL-50U | EHL-2 | EHL-3A | EXL-3B
---|---|---|---|---|---
Average / Central ion temperature, $T_{i0}$(keV) | -/1.0 | -/6.0 | -/30 | -/70 | 46.2/140
Average / Central electron density, $n_{e}(10^{20}m^{-3})$ | -/0.2 | -/1.0 | -/1.3 | -/1.5 | 0.83/2.5
Confinement time, $\tau_{E}(s)$ | - | 0.07 | 0.5 | 5 | 25
Beta, $\beta$ | - | 0.1 | 0.11 | 0.18 | 0.24
Central Magnetic field, $B_{0}(T)$ | 0.46 |<EMAIL_ADDRESS>| 3 | 4 | 4
Major radius, $R_{0}(m)$ | 0.58 | 0.6-0.8 | 1.05 | 2 | 3.2
Aspect ratio, $A$ | 1.4 | 1.5-1.85 | 1.85 | 1.8 | 1.7
Heating power, $P_{fus}(MW)$ | 3 |  $>$3 | 17 | 60 | 6.83+133
Plasma currrent, $I_{p}(MA)$ | 0.5 | 0.9 | 3.0 | 10 | 25
$T_{i}/T_{e}$ | - | 1.5 | 3 | 3 | 4

The preliminary design parameters for each device during each phase can be
found in Table 4. The realization of commercial p-11B fusion is anticipated to
begin around 2035 after the successful construction and operation of these
devices. Details of the EHL-2 physics and engineering designs will be
published in the following 1-2 years. The construction cost of EHL-2 is
estimated to be around 5 billion RMB, with the aim of obtaining the first
plasma around 2027.

##  VI Summary and Discussion

In this paper, we present ENN’s roadmap for achieving p-11B fusion utilizing
spherical torus (ST) technology. We conduct a comprehensive comparison of the
advantages and disadvantages of different fusion reactions, ultimately
selecting p-11B fusion due to its exceptional potential for commercialization.
Our analysis highlights several key factors that make p-11B fusion a promising
candidate. First, the abundance of boron fuel on Earth is expected to
significantly lower the cost of p-11B fusion compared to other fusion
reactions. Additionally, the absence of a breeding blanket requirement, which
is necessary for D-T fusion, simplifies the design and reduces the overall
size of the fusion device. Moreover, the aneutronic nature of the p-11B fusion
reaction alleviates the need for a dense screen blanket, further contributing
to size reduction. This opens up the possibility of direct energy conversion
or direct electric power generation, contingent upon the controlled production
of only charged $\alpha$ particles.

A simplified system code has been successfully developed, enabling the
evaluation of fusion reactor performance. This code takes various input
parameters into account, such as plasma shape, density and temperature
profiles, magnetic field strength, plasma current, and energy confinement
time. In return, it provides crucial output parameters, including plasma size,
average density and temperature, fusion power, radiation loss power, heating
power, fusion gain, $\beta_{T}$ (the ratio of plasma pressure to magnetic
pressure), and scaling factors. To assess plasma confinement and performance,
scaling laws for energy confinement time have been applied to both
conventional tokamak (CT) and spherical torus (ST). A specific comparison
between CT and ST has been conducted. The results of this comparison
demonstrate that ST offer certain advantages over CT. Specifically, ST
requires lower magnetic field strengths and reduced heating power, making them
more feasible and potentially easier to construct for achieving the desired
fusion conditions.

Fusion reactivity indeed increases with higher ion temperatures, which is
advantageous for achieving successful fusion reactions. However, it’s crucial
to note that as temperatures rise, so does the electron bremsstrahlung power
loss. When the electron bremsstrahlung power loss surpasses the fusion power
at the same ion temperature, it becomes essential to operate in hot ion mode.
Both CT and ST face the requirement of maintaining ion temperatures higher
than electron temperatures ($T_{i}/T_{e}\geq 4$) to realize p-11B fusion. This
condition is essential to achieve the necessary conditions for successful
fusion energy gain. To make p-11B fusion more attainable, methods that enhance
fusion reactivity, such as maintaining a non-thermalized ion distribution, can
prove highly beneficial.

The technologies needed for p-11B fusion can be summarized after the physical
requirements have been reviewed. These include high heat load materials, high-
power supply, highly efficient ion heating, high-temperature superconductivity
technology, low aspect ratio engineering, and advanced divertor design.

The roadmap for ENN’s p-11B fusion using ST is outlined. Two STs, namely
EXL-50 (a central solenoid-free ST) and EXL-50U (an upgraded version of
EXL-50), are crucial in establishing the foundation for ST p-11B fusion. Phase
I of the roadmap has been initiated, focusing on evaluating key scientific and
technological aspects, including hot ion mode experiments and validating ST
confinement scaling laws at high plasma parameters. A new spherical torus,
EHL-2, is being constructed during this phase, with the goal of realizing the
thermal p-11B fusion reaction and establishing the physical basis for p-11B
fusion. Following the completion of Phase I, Phase II will concentrate on
improving parameters and overcoming engineering constraints to scale up the
fusion reactor. EHL-3, a new ST with plasma parameters closer to those
required for p-11B fusion, will be built. In Phase III, preparations for
commercial fusion will include exploring methods to reduce costs.

The roadmap for ENN’s pursuit of p-11B fusion using ST is detailed and
consists of several key phases. Here’s a summary of the roadmap:

Phase I - Establishing the Scientific and Technological Basis:

  * •

Operation of two STs, EXL-50 (a central solenoid-free ST) and EXL-50U (an
upgrade of EXL-50).

  * •

Evaluation of the key focuses on scientific and technological aspects.

  * •

Conducting hot ion mode experiments to validate their feasibility.

  * •

Verification of ST confinement scaling laws at high plasma parameters.

  * •

Construction of a spherical torus, EHL-2, dedicated to realizing the p-11B
fusion reaction.

  * •

Establishing the physical foundation for p-11B fusion.

Phase II - Parameter Improvement and Engineering Overcoming:

  * •

Concentration on enhancing plasma parameters and addressing engineering
challenges.

  * •

Building EHL-3 (in two steps with EHL-3A and EHL-3B), another ST with plasma
parameters approaching those required for p-11B fusion.

  * •

Preparing the groundwork for scaling up the fusion reactor.

Phase III - Preparing for Commercial Fusion:

  * •

Focusing on cost reduction strategies and preparations for commercial fusion.

  * •

Exploring methods to make p-11B fusion economically viable.

This comprehensive roadmap outlines the stepwise progression from scientific
exploration to technological development and, ultimately, the pursuit of
commercial p-11B fusion, showcasing ENN’s commitment to advancing fusion
energy research.

In the past five years, ENN has explored various fusion approaches,
considering almost all types of fusion technologies. The findings and key
challenges of commercializing fusion energy have been summarized in a
reportXie2023a . The major conclusion drawn is that achieving commercial
fusion energy is highly challenging, and the success of fusion energy depends
on various factors, ranging from the choice of fuel to the confinement
approach. As a private company, ENN’s choices are further constrained. For
instance, ENN prefers not to deal with neutron-related challenges. If, in the
next stages of ENN’s roadmap, p-B fusion proves to be impractical, the
alternative choice would be D-D-He3 fusion. This fusion process involves the
catalyzed fusion of deuterium-deuterium (D-D) with a small amount of 3He to
sustain the reactor, thereby reducing the Lawson criterion requirements.
Additionally, handling neutrons in this fusion process is considered to be
less challenging. Hence, the research and development efforts on spherical
torus technology at ENN remain relevant, as D-D-He3 fusion is being explored
as a potential alternative to p-B fusion. The focus is on achieving radiation
control, high beta, and good confinement in pursuit of viable and sustainable
fusion energy solutions.

## References

  * (1) F. F. Chen, An Indispensable Truth: How Fusion Power Can Save the Planet, Springer, 2011.
  * (2) S. E. Wurzel and S. C. Hsu, Progress toward fusion energy breakeven and gain as measured against the Lawson criterion, Phys. Plasmas 29, 062103 (2022).
  * (3) J. D. Lawson, “Some Criteria for a Useful Thermonuclear Reactor,” Tech. Rep. GP/R 1807 (Atomic Energy Research Establishment, 1955) https: //www.euro-fusion.org/fileadmin/user_upload/Archive/wp-content/uploads/2012/10/dec05-aere-gpr1807.pdf. Lawson, J. D., Some Criteria for a Power Producing Thermonuclear Reactor, Proceedings of the Physical Society. Section B, 1957, 70, 6.
  * (4) A. Gibson and JET Team, Deuterium–tritium plasmas in the Joint European Torus (JET): Behavior and implications, Physics of plasmas 5, 1839 (1998).
  * (5) L. J. Reinders, The Fairy Tale Of Nuclear Fusion, 2021, Springer.
  * (6) E. Gibney, Nuclear-fusion reactor smashes energy record, Nature, 602, 371 (2022).
  * (7) J. Tollefson and E. Gibney, Nuclear-fusion lab achieves ’ingnition’: what dose it mean?, Nature 612, 597 (2022).
  * (8) J. Wesson, Tokamak Third edition, Oxford Press, ISBN: 9780198509226 (2004).
  * (9) F. Nespoli, S. Masuzaki, K. Tanaka, N. Ashikawa, M. Shoji, E. P. Gilson, R. Lunsford, T. Oishi, K. Ida, M. Yoshinuma, Y. Takemura, T. Kinoshita, G. Motojima, N. Kenmochi, G. Kawamura, C. Suzuki, A. Nagy, A. Bortolon, N. A. Pablant, A. Mollen, N. Tamura, D. A. Gates and T. Morisaki, Observation of a reduced-turbulence regime with boron powder injection in a stellarator, Nature Physics, 18, 350 (2022)
  * (10) Loren. C. Stainhauer, Review of field-reversed configurations, Physics of Plasmas 18, 070501 (2011).
  * (11) R. F. Post, The magnetic mirror approach to fusion, Nuclear Fusion, 27, 1579 (1987).
  * (12) K. Ikeda, ITER on the road to fusion energy, Nuclear Fusion, 50, 014002 (2010).
  * (13) V. S. Chan A.E. Costley, B.N. Wan, A.M. Garofalo and J.A. Leuer, Evaluation of CFETR as a Fusion Nuclear Science Facility using multiple system codes, Nuclear Fusion, 55, 023017 (2015)
  * (14) Y-K. M. Peng and D. J. Strickler, Nucler Fusion, 26, 769 (1986).
  * (15) M. Gryaznevich, R. Akers, P. G. Carolan, N. J. Conway, D. Gates, A. R. Field, T. C. Hender, I. Jenkins, R. Martin, M. P. S. Nightingale, C. Ribeiro, D. C. Robinson, A. Sykes, M. Tournianski, M. Valovic, and M. J. Walsh, Achievement of record $beta$ in the START spherical tokamak, Physical Review Letters, 80, 3972 (1998).
  * (16) A. Sykes, J.-W. Ahn, R. Akers, E. Arends, P. G. Carolan, G. F. Counsell, S. J. Fielding, M. Gryaznevich, R. Martin, M. Price, C. Roach, V. Shevchenko, M. Tournianski, M. Valovic, M. J. Walsh, H. R. Wilson, and MAST Team, First physics results from the MAST Mega-Amp Spherical Tokamak, Physics of plasmas 8, 2101 (2001).
  * (17) J. E. Menard, M.G. Bell, R.E. Bell, E.D. Fredrickson, D.A. Gates, S.M. Kaye, B.P. LeBlanc, R. Maingi, D. Mueller, S.A. Sabbagh, D. Stutman, C.E. Bush, D.W. Johnson, R. Kaita, H.W. Kugel, R.J. Maqueda, F. Paoletti, S.F Paul, M. Ono, Y.-K.M. Peng, C.H. Skinner, E.J. Synakowski and the NSTX Research Team, $beta$-limitting MHD instabilities in improved-performance NSTX spherical torus plasma, Nuclear Fusion 43, 330 (2003).
  * (18) V. K. Gusev, S.E. Aleksandrov, V. Kh Alimov, I.I. Arkhipov, B.B. Ayushin, A.G. Barsukov, B. Ya Ber, F.V. Chernyshev, I.N. Chugunov, A.V. Dech, et al, Overview of results obtained at the Globus-M spherical tokamak, Nuclear Fusion 49, 104021 (2009).
  * (19) G. S. Kurskiev, V.K.Gusev, N.V. Sakharov, Yu.V. Petrov, N.N. Bakharev, I.M. Balachenkov, A.N.Bazhenov, F.V. Chernyshev, N.A. Khromov, E.O. Kiselev, S.V. Krikunov, V.B. Minaev, et al, Energy confinement in the sphrical tokamak Globus-M2 with toroidal magnetic field reaching 0.8T, Nuclear Fusion 62, 016011 (2022).
  * (20) A. J. Creely, D. Brunner, R. T. Mumgaard, M. L. Reinke, M. Segal, B. N. Sorbom and M. J. Greenwald, SPARC as a platform to advance tokamak science, Phys. Plasmas 30, 090601 (2023).
  * (21) M. Gryaznevich, TE. Ltd Physics Team and TE. Ltd HTS Team for Tokamak Energy Ltd, Experiments on ST40 at high magnetic field, Nuclear Fusion 62, 042008 (2022)
  * (22) S. A. M. McNamara, O. Asunta, J. Bland, P.F. Buxton, C. Colgan, A. Dnestrovskii, M. Gemmell, M. Gryaznevich, D. Hoffman, F. Janky, et al, Achievement of ion temperatures in excess of 100 million degrees Kelvin in the compact high-field spherical tokamak ST40, Nucl. Fusion 63 (2023) 054002.
  * (23) (Meyer H for the STEP Plasma Control, Heating, Current Drive Team and Contributors) 2022 The physics of the preferred plasma scenario for STEP, Proc. 48th EPS conf. on Plasma Physics (Online) (available at: https://indico.fusenet.eu/event/28/contributions/302/ attachments/316/991/EPS2022_paper_HMeyer_v3.pdf)
  * (24) F. A. Geser and Mauro Valente, A theoretical model for the cross section of the proton-boron fusion nuclear reaction, Radiat. Phys. Chem., 167, 108224 (2020).
  * (25) W. M. Nevins and R. Swain, The thermonuclear fusion rate coefficient for p- 11 B reactions, Nuclear Fusion, 40, 4, 865 (2000).
  * (26) S. V. Putvinskia, D. D. Ryutov and P. N. Yushmanov, Fusion reactivity of the pB11 plasma revisited, Nucl. Fusion 59, 076018 (2019).
  * (27) D. Margarone, J. Bonvalet, L. Giuffrida, A. Morace, V. Kantarelou, M. Tosca, D. Raffestin, P. Nicolai, A. Picciotto, Y. Abe, et al, In-target proton-boron nuclear fusion using a PW-class leaser, Appl. Scie. 12, 1444 (2022).
  * (28) S. Starve, M.W. Ahmed, R.H. France III, S.S. Henshaw, B. Müller, B.A. Perdue, R.M. Prior, M.C. Spraker, H.R. Weller, Understanding the 11B(p,$\alpha$)$\alpha\alpha$ raction at the 0.675 mev resonance, Phys. Lett. B, 686,26 (2011).
  * (29) R. M. Magee, K. Ogawa, T. Tajima, I. Allfrey, H. Gota, P. McCarroll, S. Ohdachi, M. Isobe, S. Kamio, V. Klumper, H. Nuga, M. Shoji, S. Ziaei, M. W. Binderbauer and M. Osakabe, First measurements of p11B fusion in a magnetically confined plasms, Nature Communications, 14, 955 (2023).
  * (30) L. M. Lidsky, The trouble with fusion, MIT Technol Rev, 1–12 (1983).
  * (31) E. J. Lerner, S. M. Hassan, I. Karamitsos-Zivkovic and R. Fritsch, What are the fastest routes to fusion energy?, Phys. Plasmas 30, 120602 (2023).
  * (32) W. M. Nevins, A Review of Confinement Requirements for Advanced Fuels, Journal of Fusion Energy, 17, 1, 25 (1998).
  * (33) M. H. Sikora and H. R. Weller, A New Evaluation of the ${}^{11}B(p,\alpha)\alpha\alpha$ Reaction Rates, Journal of Fusion Energy, 35, 3, 538 (2016).
  * (34) Y. Ikeda, N. Umeda, N. Akino, N. Ebisawa, L. Grisham, M. Hanada, A. Honda, T. Inoue, M. Kawai, M. Kazawa, et al, Present status of the negative ion based NBI system for long pulse operation on JT-60U, Nuclear Fusion, 46, S211 (2006).
  * (35) R. J. Hawryluk, S. Batha, W. Blanchard, M. Beer, M. G. Bell, R. E. Bell, H. Berk, S. Bernabei, M. Bitter, B. Breizman, N. L. Bretz, et al, Fusion plasma experiments on TFTR: A 20 year retrospective, Phys. Plasmas, 5, 1577 (1998).
  * (36) S. Ishida, Y. Neyatani, Y. Kamada, and JT-60 Team, High performance experiments in JT-60U high current divertor discharges, Proc. 16th Inter. Conf. on Fusion Energy, Montreal, 1, 315 (1996).
  * (37) M. Shimada, D.J. Campbell, V. Mukhovatov, M. Fujiwara, N. Kirneva, K. Lackner, M. Nagami, V.D. Pustovitov, N. Uckan, J. Wesley, et al, Progress in the ITER Physics Basis- Chapter 1: Overview and summary, Nuclear Fusion, 47, S1 (2007).
  * (38) A. E. Costley, J. Hugill1 and P.F. Buxton, On the power and size of tokamak fusion pilot plants and reactors, Nuclear Fusion, 55, 033001 (2015).
  * (39) J. Q. Cai, H. S. Xie, Y. Li, M. Tuszewski, H. B. Zhou and P. P. Chen, A Study of the Requirements of p-B11 Fusion Reactor by Tokamak System, Code, Fusion Science and Technology, 78:2, 149-163 (2022).
  * (40) H. S. Xie, M. Z. Tan, D. Luo, Z. Li and B. Liu, Fusion reactivities with drift bi-Maxwellian ion velocity distributions, Plasma Phys. Control. Fusion 65 (2023) 055019.
  * (41) H. S. Bosch and G. M. Hale, Improved formulas for fusion cross-sections and thermal reactivities, Nuclear Fusion, 32, 4, 611 (1992).
  * (42) A. B. Kukushikin and P. V. Minashin, Generalization of Trubnikov formula for electron cyclotron total power loss in Tokamak-Reactors XXXVI International Coference on Plasma Physics and CF, Feb. 9-13, 2009, Zvenigorod.
  * (43) H. Z. Kong, H. S. Xie, B. Liu, M. Z. Tan, D. Luo, Z. Li and J. Z. Sun, Enhancement of fusion reactivity under non-Maxwellian distributions: effects of drift-ring-beam, slowing-down, and kappa super-thermal distributions,Plasma Phys. Control. Fusion 66 (2024) 015009.
  * (44) H. Hora, S. Eliezer, G.J. Kirchhoff, N. Nissim, J.X. Wang, P. Lalousis, Y.X. Xu, G.H. Miley, J.M. Martinez-Val, W. McKenzie and J. Kirchhoff, Road map to clean energy using laser beam ignition of boron-hydrogen fusion, Laser and Particle Beams, 35, 4, 730 (2017).
  * (45) Y. J. Shi, B Liu, S.D. Song, Y.Y Song, X.M. Song, B.W. Tong, S.K. Cheng, W.J. Liu, M.Y. Wang, T.T. Sun, D. Guo, S.J. Li, Y.Y. Li, B. Chen, X. Gu, J.Q. Cai, D. Luo, D. Banerjee, X. Zhao, Y.M. Yang, W.W. Luo, P.H. Zhou, Y. Wang, A. Ishida, T. Maekawa, M.S. Liu, B.S. Yuan, Y.-K. M. Peng and the EXL-50 Team, Solenoid-free current drive via ECRH in EXL-50 spherical torus plasma, Nuclear Fusion , 62, 086047 (2022).
  * (46) H. S. Xie, Introduction to Fusion Ignition Principles: Zeroth Order Factors of Fusion Energy Research (in Chinese), USTC Press, Hefei, 2023. (English version in prepare, http://hsxie.me/fusionbook/)


XX XX 8 May/June

# Why Accuracy Is Not Enough: The Need for Consistency in Object Detection

Caleb Tung  Abhinav Goel  Fischer Bordwell  Nick Eliopoulos  Xiao Hu  Yung-
Hsiang Lu  {tung3, goel39, fbordwel, neliopou, hu440<EMAIL_ADDRESS>-
Purdue University  George K. Thiruvathukal<EMAIL_ADDRESS>- Loyola University
Chicago

(2021)

###### Abstract

Object detectors are vital to many modern computer vision applications.
However, even state-of-the-art object detectors are not perfect. On two images
that look similar to human eyes, the same detector can make different
predictions because of small image distortions like camera sensor noise and
lighting changes. This problem is called inconsistency. Existing accuracy
metrics do not properly account for inconsistency, and similar work in this
area only targets improvements on artificial image distortions. Therefore, we
propose a method to use non-artificial video frames to measure object
detection consistency over time, across frames. Using this method, we show
that the consistency of modern object detectors ranges from 83.2% to 97.1% on
different video datasets from the Multiple Object Tracking Challenge. We
conclude by showing that applying image distortion corrections like .WEBP
Image Compression and Unsharp Masking can improve consistency by as much as
5.1%, with no loss in accuracy.

††journal: Multimedia

Much of modern computer vision relies on object detectors. An object detector
is a Deep Neural Network (DNN) that takes an image as the input and then
identifies the locations and types of objects found in that image. Across
scientific disciplines, object detectors are increasingly ubiquitous. From
electronic package sorting in e-commerce to collision detection in traffic
monitoring, from remote sensing in low-orbit satellites to automated MRI
screening in the fight against cancer: object detectors are driving an entire
frontier of technology. With so many critical applications, object detectors
need to be consistently accurate. Modern object detectors use different
architectures (e.g., single-shot, R-CNN, etc.) and training methods (e.g.,
multitask loss, neural architecture search, etc.) to achieve state-of-the-art
accuracy on popular image datasets like Microsoft COCO (Common Objects in
Context) [1].

## The Consistency Problem

Even though object detectors are carefully tested for accuracy, this article
observes that consistency is also a valuable metric that receives less
attention in literature. As we discuss later, common image datasets make it
challenging to test for consistency.

(a)

(b)

Figure 1: State-of-the-art object detector Mask-RCNN is inconsistent on two
images taken 0.03s apart, even though both images look alike. In (a), the
woman is missed (red dashed-line box) while the man is detected (green solid
box). In (b), the reverse is true. (All other people are detected correctly in
both images, giving an average 3/4 accuracy in both images.)

Accuracy typically measures how often an object detector is correct on
average. This information is partially deficient because it does not capture
the variation in an object detector’s performance when input images are
similar. Since accuracy is reported as an average, there could be multiple
ways an object detector achieves a given accuracy, some of which may be less
desirable. For example, in Fig. 1, a state-of-the-art object detector (Mask-
RCNN [2]) detects three out of four people per image. The accuracy is the same
on average (3/4), yet the detector behaves inconsistently: it misses a
different person in each image.

Inconsistent behavior becomes cause for concern in vision applications that
demand strict performance guarantees. For example, a collision prevention
algorithm with 95% accuracy that misses the same 5% of objects allows one to
investigate the cause of the 5% error more easily, because the defective
behavior is consistent. However, an algorithm that behaves inconsistently,
missing different objects across each image in the test, is much harder to
troubleshoot.

This article investigates object detector consistency as a method to augment
existing accuracy metrics. Consistency measures the difference in predictions
from an object detector across similar images. We explore different methods to
quantify consistency, ultimately choosing a metric that tracks a detector’s
behavior on time-series images from the MOT Challenge [3]—a dataset originally
intended to benchmark object tracking. As shown in Fig. 2, we find that state-
of-the-art detectors (Mask-RCNN, Faster-RCNN [4], RetinaNet [5], SSD [6])
exhibit inconsistent behavior. In our experiments, we observe up to an average
of 17% inconsistent detections. We evaluate methods to improve consistency and
present a selection of methods (lossy image compression, gamma boosting, etc.)
that successfully raise consistency by up to 5%.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2: Different object detectors behave inconsistently on images that look
visually similar. Left images are taken 0.03s before the right. Red, dashed-
line boxes are missed detections, green solid boxes are correct detections.
(a)-(b) Faster-RCNN. (c)-(d) RetinaNet. (e)-(f) SSD.

## What accuracy measures

Accuracy is determined by comparing the predictions of an object detector
against the ground truth in a dataset of images. Standard accuracy metrics
like mean Average Precision (mAP) usually consider two factors simultaneously:
(1) how much of the ground truth the detector successfully predicts, and (2)
how many of the detector’s predictions were incorrect. To get perfect
accuracy, each of the predictions an object detector makes must be correct: it
cannot blanket the image with guesses.

The mAP accuracy metric is defined in terms of precision and recall using
true/false positives and negatives (see Tab. 1). Precision and recall are
defined in Eqn. 1 and 2, respectively. mAP is calculated for a given image
dataset by integrating the area under the precision-versus-recall curve with
respect to recall (Eqn. 3).

Table 1: Terminology used for calculating object detection accuracy. | Object detected | Object not detected
---|---|---
Object exists |  | True Positive
---
(TP)
| False Negative
---
(FN)
| Object does
---
not exist
| False Positive
---
(FP)
| True Negative
---
(TN)

$\texttt{precision }p=\frac{TP}{TP+FP}$ (1)

$\texttt{recall }r=\frac{TP}{TP+FN}$ (2)

$\texttt{accuracy }=\int_{0}^{1}p(r)dr$ (3)

## Accuracy does not measure consistency

When accuracy is not 100%, there may be more to the story (for context, the
state-of-the-art Mask-RCNN reaches 63% mAP on the COCO dataset). Accuracy does
not describe whether the detector is consistent. Consider an object detector
that achieves 50% accuracy on a set of images, where each image is slightly
different but still contains the same objects. One might hope that the
detector would predict consistently as shown in Fig. 3a-b. However, it is
still possible for other detector predictions (Fig. 3c-d) to achieve the same
50% accuracy, albeit inconsistently.

(a)

(b)

(c)

(d)

Figure 3: Accuracy does not capture consistency. For object detections shown
in similar-looking images, (a)-(b) has the same average accuracy (i.e. 1/2
correct) as (c)-(d). (a)-(b) has consistent detections (same object detected
in both images), but (c)-(d) is inconsistent (different objects detected in
both images).

Additionally, simply reporting fine-grained accuracy statistics does not
capture consistency. It is true that we can infer more information about the
neural network’s consistency by reporting additional statistics like the
standard deviation or variance of per-image accuracy across a dataset of
similar images. However, reporting the standard deviation still does not
reveal the Fig. 3c-d case discussed above, where the detector inconsistently
detects and misses different objects in each frame even though the number of
objects remains the same.

As we showed earlier in Fig. 2, inconsistent behavior exists even in popular
object detectors. Thus, accuracy does not always communicate the full picture
of a detector’s performance because it does not explicitly describe
consistency. Inconsistency may have severe consequences in applications that
involve safety.

## Related work

There is a growing number of studies to improve computer vision, but they do
not focus on consistency. These efforts can largely be grouped into two
categories: (1) adversarial attacks and (2) synthetic image distortions.

### Adversarial attacks

Adversarial attacks present a significant challenge to neural networks.
Goodfellow, et al. [7] demonstrate that a well-trained image classifier can be
tricked into misclassifying an image by slightly perturbing the values of the
pixels. In a typical adversarial attack, an algorithm makes minute, calculated
adjustments to the pixels of a correctly predicted image until the neural
network makes an incorrect prediction. The final image, known as the
adversarial sample, appears to human eyes as very similar to the original
image. Several different adversarial samples are shown in Fig. 4.

(a)

(b)

(c)

(d)

Figure 4: Similar-looking adversarial samples generated from an image in the
MNIST handwriting dataset. (a) Original image. Correctly classified as “8”
with 0.9 confidence. (b) Fast-Gradient Sign Method adversarial sample. Wrongly
classified as “4” with 0.9 confidence [7]. (c) Jacobian Saliency Map Attack
adversarial sample. Wrongly classified as “9” with 0.6 confidence [8]. (d)
DeepFool adversarial sample. Wrongly classified as “4” with 0.83 confidence
[9].

Existing methods to defend against adversarial attacks often involve some
combination of (1) including adversarial samples during network training [7],
(2) transforming input images into a lower-dimensional space before feeding
the neural networks [10], and (3) filtering adversarial samples through a
custom neural network before they reach the main network [11].

### Synthetic image distortions and data augmentation

Computer vision models can make incorrect predictions on images from natural
sources (i.e., not manipulated for adversarial attacks) [12]. Two images of
the same scene can be captured by the same camera less than a second apart,
yet the predictions of a neural network on those two visually similar images
can differ dramatically. The small differences between the two images are
called image distortions, and they can be caused by a range of factors,
including ambient light level and camera sensor noise [13].

To make neural networks more robust against image distortions, data
augmentation is widely used. The networks are trained on datasets that are
modified, or synthetically distorted, to emulate the natural distortions.
Common synthetic image distortions (shown in Fig. 5) include perturbing the
pixels with Gaussian noise, adding artificial motion blur, adjusting color
saturation and brightness, and even adding computer-generated fog, snow, and
rain. [12]

(a)

(b)

(c)

(d)

Figure 5: Examples of synthetic image distortions. (a) Original image. (b)
Increased brightness. (c) Motion blur. (d) Artificial fog.

The discussed previous work uses only accuracy as the metrics, without
explicitly measuring consistency. Although some publications [14, 15] explore
ways to measure the robustness of a neural network beyond accuracy, they
neither concretely define consistency nor propose solutions. This article aims
to do both.

## Considerations when measuring consistency

In this article, we define consistency as a metric of how differently an
object detector behaves on similar images:

If images appear similar to the human eye, an object detector should
consistently detect the same objects.

Since accuracy does not always quantify consistency, we now discuss the
considerations needed to properly capture consistency.

### Use Video/Time-Series Data: consistency measurements require similar test
images

A meaningful metric for consistency should use input images that are already
consistent. Consider Fig. 6a, b. Those pictures appear similar to the human
eye, so we would expect consistent performance from an object detector.
However, if the images are significantly different (Fig. 6a, c), it is
meaningless to use them to make claims about consistency.

(a)

(b)

(c)

Figure 6: Consistency is only meaningful when measured across images that look
similar (a), (b) (where (b) is taken a second later and camera autofocus is
slightly blurred). Consistency is meaningless on images that look different
(a), (c). Popular datasets use images that look different, making them
appropriate for measuring accuracy, but less suited for measuring consistency.

As Tung, et al. [13] observe, popular image datasets (e.g., ImageNet,
Microsoft COCO) are filled with visually dissimilar images. Thus, those
datasets are largely unsuitable for consistency testing. Instead, those
authors recommend consecutive frames from videos as a better source of
visually similar images. Although adversarial attacks and artificial image
transformations can also be used to generate consistency test data from such
popular datasets, Gu, et al. [15] report that such techniques do not well
represent image distortions that would occur naturally. Instead, this paper
uses consecutive frames from video to test consistency - this way, any
inconsistencies would be caused by natural image distortions.

## Quantifying image similarity

This article emphasizes the importance of using similar-looking images to test
for consistency. We use consecutive frames from videos in the MOT Challenge
because such frames are taken up to 30 times per second, ensuring that
adjacent frames look similar. Beyond the scope of this article, other popular
image similarity measurement methods attempt to relate two images beyond raw
pixel value differences. The Structural Similarity Index (SSIM) [16]
identifies structural details about the images and then compares the details.
This means that SSIM identifies a noisy version of an image as similar to the
original, even though raw pixel values are very different. Beyond SSIM,
techniques such as the Feature Similarity Index (FSIM) [17] and deep-learning
driven comparison extract low-level features to better approximate the way
humans compare images.

### Use MOT Ground Truth: Consistency measurements need additional labels

When benchmarking an object detector, bounding box and class ground truth
labels are sufficient to report accuracy, but they cannot reveal all
inconsistencies. In particular, that ground truth cannot show whether the same
objects were detected between two similar images (the above Fig. 3c-d
problem): the ground truth is identifier-agnostic. Thus, we need a way to
check whether objects from two images are the same, so that consistency can be
measured.

We choose to use per-image object identifier (Object ID) ground truth labels
to keep track of objects during measurement. Each unique object is assigned
the same Object ID across the dataset.

## Proposed method of measuring consistency

Based on our prior discussion of accuracy vs. consistency, we present a method
that specifically tracks whether an object detector detects the same objects,
given visually similar, time-series images. Our source of visually similar
images is the Multi-Object Tracking (MOT) Challenge [3], consisting of high-
quality videos from various datasets.

The pairwise consistency $C_{i,j}$ refers to the object detector’s consistency
on a pair of images $I_{i}$ and $I_{j}$. It is calculated as shown in Eqn. 4.
If the detector is perfectly consistent, $C_{i,j}$ is 1. If it is entirely
inconsistent, then $C_{i,j}$ is 0. As shown in Fig. 7, consistency looks to
capture whether objects were detected in one image and missed in another.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 7: Consistency tracks whether an object is detected in one image and
missed in another similar-looking image. This complements accuracy
measurements. (a)-(b) 100% accurate (all objects correctly detected in both
images), 100% consistent (nothing was missed in one image and detected in
another). (c)-(d) 50% accurate (only one of two objects detected in each
image), 100% consistent (nothing was missed in one image and detected in
another). (e)-(f) 50% accurate (only one of two objects detected in each
image), 0% consistent (any objects detected in one image was missed in the
other). (g)-(h) 0% accurate (nothing detected), 100% consistent (nothing was
missed in one image and detected in another). Improving accuracy does not
necessarily imply improving consistency and vice versa.

$C_{i,j}=\frac{|G_{i}\cap G_{j}|-|M_{i,j}|-|M_{j,i}|}{|G_{i}\cap G_{j}|}$ (4)

(a)

(b)

Figure 8: Visualization of Eq. 4, where (a) is $I_{i}$, (b) is $I_{j}$ and the
green boxes indicate object detections. $G_{i}\cap G_{j}$ contains objects A
and B since they appear in both $I_{i},I_{j}$. Objects C and D are not
included in calculations because they do not appear in both images. Because
object B is detected in $I_{i}$ but missed in $I_{j}$, so $M_{i,j}$ contains
object B. No shared boxes are detected in $I_{j}$ and missed in $I_{i}$, so
$M_{j,i}=\emptyset$. Thus, consistency $C_{i,j}=(2-1-0)/2=0.5$.

Eqn. 4 is explained using the example in Fig. 8. Fig. 8a is image $I_{i}$, and
Fig. 8b is image $I_{j}$. $G_{i}$ is the set of $I_{i}$’s ground truth Object
IDs {A, B, C}, and $G_{j}$ is the set of $I_{j}$’s ground truth Object IDs {D,
A, B}. $M_{i,j},M_{j,i}$ captures the objects that were inconsistently
detected as follows: $M_{i,j}$ is the set of ground truth Object IDs that
satisfy the following conditions: (1) the ground truth box is present in both
images $I_{i},I_{j}$ (i.e. in $G_{i}\cap G_{j}$), (2) the object detector
detected the object in frame $I_{i}$, and (3) the object detector missed the
object in frame $I_{j}$. So $M_{i,j}$ is object B, while $M_{j,i}$ is empty.

If an object is present in both images, yet is detected in only one of the
images, than consistency should decrease. Taken together, $M_{i,j},M_{j,i}$
captures consistency decreases in $I_{i},I_{j}$. This also implies that if
both $|M_{i,j}|,|M_{j,i}|$=0, the detector can be said to be consistent.

Bounding boxes predicted by the object detector are eligible for consideration
in $M_{i,j},M_{j,i}$ calculations only after being filtered through non-
maximum suppression (we use the common IoU threshold = 0.5) and a confidence
threshold of 0.7 (see sidebar: “IoU and non-maximum suppression”).

## IoU and non-maximum suppression

IoU (Intersection-over-Union) is a common measurement in object detection,
used to determine if two bounding boxes overlap sufficiently to be counted as
the same object. It is calculated by dividing the area of two bounding boxes’
overlap by the area of the union of the two boxes. If IoU = 1, then the boxes
perfectly overlap. If IoU = 0, the boxes have no overlap. In literature, an
IoU threshold of 0.5 is typically used [13] to decide if two bounding boxes
sufficiently overlap. Non-maximum suppression is a common application of IoU
to filter an object detector’s predictions so that only the “best” ones
remain. It finds all overlapping predicted bounding boxes (determined by the
IoU threshold) and then filters out the ones that have the same object class
and lower confidence scores.

We measure consistency across a given video $V$ with $N$ frames by measuring
pairwise consistency between each pair of adjacent frames in the video, and
then averaging the results across all $N-1$ pairs. This is expressed in Eqn.
5.

$C_{V}=\frac{1}{N-1}\sum_{i=1}^{N-1}C_{i,i+1}$ (5)

As shown earlier in Fig. 7, consistency and accuracy can work together to
supply more information about object detection performance than either could
on its own. Improving accuracy does not necessarily imply improved consistency
and vice versa. Colloquially, one might say that consistency indicates how
similarly an object detector behaves on two similar images, while accuracy
indicates whether that behavior is desirable (detects everything consistently)
or undesirable (misses everything consistently).

Finally, note that we choose to compare bounding box object IDs instead of
output feature maps to calculate consistency. This is because comparing
feature maps requires arbitrary similarity metrics - selecting an appropriate
metric is itself an open problem. Additionally, bounding boxes are already
used for accuracy measurements; re-using boxes for consistency will make it
more convenient for researchers to take consistency measurements.

## Consistency of modern object detectors

We present the consistency of several state-of-the-art object detectors,
showing that they exhibit inconsistent behavior. We demonstrate using highly
accurate two-stage object detectors (Faster-RCNN and Mask-RCNN) as well as the
faster, but less accurate, single-shot detectors (RetinaNet and SSD). We use
Facebook’s official, pretrained models from their torchvision Python package.
Measurements are taken on the videos found in the MOT Challenge.

(a)

(b)

Figure 9:  Object detector consistency (our method, (a)) and accuracy (mAP,
(b)) measured on the different videos in the MOT Challenge. Videos are: AR-6:
ADL-Rundle-6, AR-8: ADL-Rundle-8, E-B: ETH-Bahnhof, E-P: ETH-Pedcross2, E-S:
ETH-Sunnyday, K-13: KITTI-13, K-17: KITTI-17, P-S: PETS09-S2L1, T-C: TUD-
Campus, T-S: TUD-Stadtmitte, V-2: Venice-2. State-of-the-art object detectors
exhibit inconsistent behavior, ranging from 83.2% to 97.1% consistency. The
two-stage models are both more consistent (a) and more accurate (b) than their
single-shot counterparts.

As shown in Fig. 9, all object detectors exhibit some inconsistent behavior,
ranging from 83.2% to 97.1% consistency ($C_{V}$ as calculated in Eqn. 5). We
also see that the two-stage models are more accurate and more consistent.

## Toward consistent object detection

Object detector inconsistency is caused by the detector missing an object. As
described in Related Work, missed detections can be caused by adversarial
attacks and image distortions. Because the MOT Challenge is not an adversarial
dataset, we expect the inconsistencies to be caused by image distortions
naturally present in the dataset. Therefore, we apply different image
distortion corrections from literature to compare their efficacies at
improving consistency.

## Improving consistency via training

Common training techniques such as data augmentation and dropout are known to
improve a model’s robustness to image transformations and distortions. Despite
these techniques being employed to produce our pretrained models, we find that
inconsistencies still persist. To improve consistency, we use post-training
image distortion corrections because of their accessibility. However, emerging
training methods appear promising for improving consistency. Zhang et al.
propose weakly-supervised, context-based techniques [18, 19] that gather
context for the scene (e.g., optical flow and prior knowledge) to provide
additional information to an object detector - this information could
stabilize the detector and improve consistency over time-series data from
videos. Other techniques project neural network features into low-dimensional
representations during training [10] \- this could improve consistency on
images of similar objects taken from different angles, lighting, etc.

1) Gaussian Denoise (GD). Random sensor noise and shot noise can decrease
detection performance. As demonstrated by Kang et al. [20], we apply a
normalized Gaussian filter to all images in the dataset in an attempt to
reduce the noise.

2) Horizontal Flip (HF). Zhang, et al. [14] note that the slight translation
of an object in an image could cause a previously misdetected object to become
correctly detected. Further, Yin, et al. [21] observe that horizontally
flipping an image can mitigate the detrimental effects of noise on a neural
network. Thus, we horizontally flip all images.

3) WEBP Compression (WC). Compressing an image using the lossy JPEG format is
already known to defeat adversarial attacks [12]. Yin, et al. [21] find that
because WEBP compression introduced loop filtering, it is even better suited
to breaking down the structures in an image that result from synthetic image
distortions. Thus, we apply WEBP compression to the dataset images using a
compression quality factor of 30 (Yin, et al. find that lower quality factors
allow neural networks to perform better).

4) Unsharp Mask (UM). Tung, et al. [13] explain that if an object is moving,
the motion blur can make it harder for a network to extract features along the
object’s edges. The Unsharp Mask is a linear image processing technique
commonly used to remove blur. The technique first identifies a set of blurry
details by subtracting a further-blurred image from the original. The details
are then emphasized in the original image. We apply the Unsharp Mask to the
dataset to reduce the blur on object edges.

5) Gamma Correction (GC). Yeu, et al. [22] show that artificially increasing
an image’s contrast and perceived brightness can help object detectors like
Faster-RCNN perform better, particularly when the detectors are trained on
daytime images. Gamma Correction is a common brighten/contrast technique that
is driven by the Power Law expression; we use it on the dataset as well.

Table 2: Avg. Consistency Improvements |  | Faster-
---
RCNN
| Mask-
---
RCNN
RetinaNet | SSD
GD | 0% | -0.3% | 0% | -0.6%
HF | -5.3% | -5.4% | -7.3% | -10.1%
WC | 0.6% | 0.5% | 0.7% | 0.4%
UM | 3.6% | 2.6% | 3.0% | 1.1%
WC+UM | 5.1% | 3.0% | 3.2% | 1.3%
GC | 0.1% | 0.1% | 0.4% | 0.1%
Table 3: Avg. Accuracy Improvements |  | Faster-
---
RCNN
| Mask-
---
RCNN
RetinaNet | SSD
GD | 2.1% | 2.4% | -0.6% | -1.1%
HF | -19.3% | -19.4% | -25.5% | -28.4%
WC | 1.5% | 1.8% | 0.5% | 0.5%
UM | 2.0% | 3.2% | 8.3% | 3.6%
WC+UM | 3.2% | 4.1% | 8.6% | 3.9%
GC | 0.1% | -0.5% | -0.7% | -0.1%

Tab. 2 shows the average improvement across the MOT Challenge in terms of
consistency percentage points (i.e. a table entry of Y% means that it raises
consistency from X% to X+Y%), when the different distortion corrections are
applied. Similarly, Tab. 3 shows the accuracy improvement.

We see that both WEBP Compression (WC) and Unsharp Mask (UM) improve both
consistency and accuracy for all object detectors. Applying both effects at
the same time (WC+UM) gives a further overall improvement. In fact, the
example inconsistencies in Fig. 1 and 2 are resolved using WC+UM. Gaussian
Denoise (GD) and Horizontal Flip (HF) both degrade consistency and accuracy
(likely because applying these effects on relatively un-noisy images degrades
the feature structure of the images [17]).

Finally, we note that improvements in consistency do not always equate to
improvements in accuracy. Gamma Correction (GC) improves consistency, but
degrades accuracy: in other words, the detector is consistently worse on the
GC data, as described earlier in Figure 7.

## Conclusion and Future Work

Object detectors are vital to many modern computer vision applications.
However, even state-of-the-art object detectors exhibit inconsistent behavior
when the input undergoes small changes. This inconsistent behavior is not
fully captured by existing measurement tools; accuracy metrics and popular
image datasets cannot measure whether the same objects are detected
consistently. We devise a consistency measurement method that uses images from
videos and object ID labels. Our method compliments accuracy measurement.
Using this method, we show that object detectors have consistency ranging from
83.2% up to 97.1%, depending on the input data. Additionally, applying image
distortion corrections like WEBP Compression and Unsharp Masking can improve
consistency by as much as 5.1%. There is still room for improvement by the
community. We only explore post-training methods to raise consistency. Future
exploration should explore training-aware consistency improvements and further
investigate the relationship between accuracy and consistency.

## Acknowledgments

This project is supported in part by the National Science Foundation
OAC-2104709. Any opinions, findings, and conclusions or recommendations
expressed in this material are those of the author(s) and do not necessarily
reflect the views of the National Science Foundation.

## Authors

Caleb Tung is a doctoral student in the Elmore Family School of Electrical and
Computer Engineering at Purdue University. Caleb’s research is on designing
energy-efficient neural network operations for computer vision on embedded
devices. Outside the lab, Caleb can be found at the piano bench or somewhere
in a good book.

Abhinav Goel is a doctoral candidate in the Elmore Family School of Electrical
and Computer Engineering at Purdue University. Abhinav’s research interests
are in the area of energy-efficient artificial intelligence systems.

Fischer Bordwell is a Master’s student in the Elmore Family School of
Electrical and Computer Engineering at Purdue University. Fischer’s research
is on utilizing computer vision to reduce misinformation in the news.

Nick Eliopoulos is currently is a doctoral student the Elmore Family School of
Electrical and Computer Engineering at Purdue University. Outside of
academics, Nick enjoys drawing and cooking.

Xiao Hu is an M.S. student in the Elmore Family School of Electrical and
Computer Engineering at Purdue University. His research broadly connects to
the fields of human-computer interaction, low-power computer vision, fairness
in artificial intelligence, machine learning, and full-stack web development.

George K. Thiruvathukal is a professor and chairperson in the Department of
Computer Science of Loyola University Chicago and visiting computer scientist
at Argonne National Laboratory in the Leadership Computing Facility. His
research interests include parallel computing, software engineering, and
computer vision.

Yung-Hsiang Lu is a professor in the Elmore Family School of Electrical and
Computer Engineering at Purdue University. He is also the director of Purdue’s
John Martinson Engineering Entrepreneurial Center. His research topics include
computer systems, computer vision, and embedded systems. He is a Fellow of the
IEEE and Distinguished Scientist of the ACM.

## References

  * [1] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common Objects in Context,” in Computer Vision – ECCV 2014 (D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, eds.), Lecture Notes in Computer Science, (Cham), pp. 740–755, Springer International Publishing, 2014.
  * [2] K. He, G. Gkioxari, P. Dollar, and R. Girshick, “Mask R-CNN,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–1, 2018.
  * [3] L. Leal-Taixé, A. Milan, I. Reid, S. Roth, and K. Schindler, “MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking,” arXiv:1504.01942 [cs], Apr. 2015.  arXiv: 1504.01942.
  * [4] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” in Advances in Neural Information Processing Systems 28 (C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, eds.), pp. 91–99, Curran Associates, Inc., 2015.
  * [5] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, “Focal Loss for Dense Object Detection,” pp. 2980–2988, 2017.
  * [6] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “SSD: Single Shot MultiBox Detector,” in Computer Vision – ECCV 2016 (B. Leibe, J. Matas, N. Sebe, and M. Welling, eds.), Lecture Notes in Computer Science, (Cham), pp. 21–37, Springer International Publishing, 2016.
  * [7] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and Harnessing Adversarial Examples,” in 2015 International Conference on Learning Representations, Mar. 2015.
  * [8] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, “Universal Adversarial Perturbations,” pp. 1765–1773, 2017.
  * [9] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks,” pp. 2574–2582, 2016.
  * [10] T. d’Orsi, P. K. Kothari, G. Novikov, and D. Steurer, “Sparse PCA: Algorithms, Adversarial Perturbations and Certificates,” in 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS), pp. 553–564, Nov. 2020.  ISSN: 2575-8454.
  * [11] S. Gu and L. Rigazio, “Towards Deep Neural Network Architectures Robust to Adversarial Examples,” arXiv:1412.5068 [cs], Apr. 2015\.  arXiv: 1412.5068.
  * [12] S. Dodge and L. Karam, “Understanding how image quality affects deep neural networks,” in 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–6, June 2016.
  * [13] C. Tung, M. R. Kelleher, R. J. Schlueter, B. Xu, Y. Lu, G. K. Thiruvathukal, Y. Chen, and Y. Lu, “Large-Scale Object Detection of Images from Network Cameras in Variable Ambient Lighting Conditions,” in 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pp. 393–398, Mar. 2019.
  * [14] R. Zhang, “Making Convolutional Networks Shift-Invariant Again,” in International Conference on Machine Learning, pp. 7324–7334, May 2019\.
  * [15] K. Gu, B. Yang, J. Ngiam, Q. Le, and J. Shlens, “Using Videos to Evaluate Image Model Robustness,” arXiv:1904.10076 [cs], Aug. 2019.  arXiv: 1904.10076.
  * [16] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, pp. 600–612, Apr. 2004.  Conference Name: IEEE Transactions on Image Processing.
  * [17] L. Zhang, L. Zhang, X. Mou, and D. Zhang, “FSIM: A Feature Similarity Index for Image Quality Assessment,” IEEE Transactions on Image Processing, vol. 20, pp. 2378–2386, Aug. 2011.  Conference Name: IEEE Transactions on Image Processing.
  * [18] D. Zhang, J. Han, L. Yang, and D. Xu, “SPFTN: A Joint Learning Framework for Localizing and Segmenting Objects in Weakly Labeled Videos,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, pp. 475–489, Feb. 2020.  Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
  * [19] D. Zhang, J. Han, L. Zhao, and D. Meng, “Leveraging Prior-Knowledge for Weakly Supervised Object Detection Under a Collaborative Self-Paced Curriculum Learning Framework,” International Journal of Computer Vision, vol. 127, pp. 363–380, Apr. 2019.
  * [20] D. Kang, Y. Sun, D. Hendrycks, T. Brown, and J. Steinhardt, “Testing Robustness Against Unforeseen Adversaries,” arXiv:1908.08016 [cs, stat], June 2020.  arXiv: 1908.08016.
  * [21] Z. Yin, H. Wang, J. Wang, J. Tang, and W. Wang, “Defense against adversarial attacks by low-level image transformations,” International Journal of Intelligent Systems, vol. 35, no. 10, pp. 1453–1466, 2020.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/int.22258.
  * [22] Y. H. Yeu, M. I. Shapiai, Z. H. Ismail, and H. Fauzi, “Investigation on Different Color Spaces on Faster RCNN for Night-Time Human Occupancy Modelling,” in 2019 IEEE 7th Conference on Systems, Process and Control (ICSPC), pp. 118–121, Dec. 2019.