# Richness out of smallness: a Possible Staged Blueprint on Future Colliders

Meng Lu$\ast$1<EMAIL_ADDRESS>Qiang Li$\ast$2<EMAIL_ADDRESS>Zhengyun
You1<EMAIL_ADDRESS>Ce Zhang2<EMAIL_ADDRESS>1\. School of
Physics, Sun Yat-Sen University, Guangzhou 510275, China  2\. State Key
Laboratory of Nuclear Physics and Technology, School of Physics, Peking
University, Beijing, 100871, China

###### Abstract

Novel collision methods and rich phenomena are crucial to keeping high-energy
collision physics more robust and attractive. In this document, we present a
staged blueprint for future high-energy colliders: from neutrino-neutrino
collision, neutrino-lepton collision to electron-muon and muon-muon
collisions. Neutrino beam from TeV scale muons is a good candidate to enrich
high-energy collision programs and can serve as a practical step toward a
high-energy muon collider, which still requires tens of years of R$\&$D.
Neutrinos-neutrinos collision provides a promising way to probe heavy Majorana
neutrinos and effective neutrino mass; neutrino and antineutrino annihilation
into Z boson has a huge cross-section at 10K pb level; leptons-neutrinos
collision benefits W boson mass precision measurements. With only a minimal
amount of integrated luminosity, one can envision the “Richness out of
smallness”. This document summarizes the current status and the roadmap
towards the muon-muon collider with less challenging techniques required
through intermediate facilities, where a wide variety of physics goals could
be achieved. A (preparatory) laboratory on novel colliders could attract vast
international interests and collaborations.

##  I Introduction

The discovery of the Higgs boson at the LHC in 2012 [1] symbolizes that
particle physics is entering a key period. On one hand, direct searches for
new physics beyond the Standard Model (BSM, such as supersymmetry and extra
dimensions) through the Higgs portal receive intense attention. On the other
hand, rich progress has been made on heavy flavor and electroweak measurements
from the LHC and other experiments, which deepens the scope of precision tests
on the SM, and stimulates indirect searches for BSM with the effective field
method in a bottom-up approach [2].

Recent years have witnessed several significant anomalies or hints of possible
new physics BSM. First, the LHCb Collaboration, in a test of lepton flavor
universality using B$\rightarrow$K$ll$, reports a measurement that deviates by
3.1 standard deviations from the SM prediction [14]. Second, the latest result
from the Muon $g-2$ Experiment at Fermilab has pushed the world average of the
muon anomalous magnetic moment measurements to 4.2 standard deviations away
from the SM prediction [15]. Most recently, the CDF II collaboration has
reported a measurement of the W gauge boson mass [18], ${\rm M}_{\rm W}^{\rm
CDF}=80.433\pm 0.009\,\,{\rm GeV}$, which is $7.2$ standard deviations away
from the SM prediction of ${\rm M}_{\rm W}^{\rm SM}=80.357\pm 0.006\,\,{\rm
GeV}$ [19]. Numerous theoretical studies attempt to accommodate these
anomalies, which may or may not require a modification of the SM.

In the next stage, the LHC will enter the HL-LHC phase after 2025-2027 [3, 4,
5], and will collect in total around 3000 fb-1 of data in a period of 10
years, which can help deepen our understanding of fundamental physics. In
addition, HEP communities have had intense discussions on the target and
strategy for future colliders (see e.g. [6, 7]). Various options include,
electron-positron collider at the collision energy from 250 GeV to 3 TeV [8,
9, 10, 11, 12], hadron collider at 100 TeV scale, and TeV scale muon colliders
[13], etc. These future colliders are aiming at precision measurement of Higgs
properties and searching for new physics at higher energy scales. The
International Linear Collider (ILC) costs around 10B dollars; the 100 km
double-ring circular electron-positron collider (CEPC) and the Future Circular
Collider (FCC) cost less but are space-consuming due to energy loss from
synchrotron radiation.

Muons suffer less synchrotron energy loss by 8 orders of magnitude than
electrons and positrons, which leads to the fact that a TeV scale muon
collider can be kept as small as O(km) in circumference. muon collider has a
much cleaner environment and larger effective center of mass energy with
respect to the collision energy than the hadron collider. It is also sensitive
directly to muon-related new physics. Recently, due to the LHCb lepton flavor
universality and Fermilab muon $g-2$ anomalies, interest in muon colliders has
revived [16].

It is generally believed that a muon collider still needs decades of research
and challenging development, especially, on how to achieve high quality
(intensity and emittance) beam and mitigate the beam-induced background (BIB)
effects from muon decays. Muon beams are usually achieved in proton or
positron interactions on target, which have both pros and cons on beam cooling
or intensity. BIB is crucial in the physics program at a muon collider [17],
for which usually a nozzle or a timing detector is introduced to mitigate such
effect.

TeV scale muon beams emit bunches of collimated decay products and thus can
provide neutrino beams, which have a great potential for application in high-
energy collision programs. With only a small amount of integrated luminosity,
interesting phenomena from neutrino collisions can be observed towards the
“Richness out of smallness”. For example, neutrino and antineutrino annihilate
into Z boson with a huge cross-section at 10K pb level. It also opens doors to
many new topics, such as searching for resonances that decay into neutrinos.
One can also collide neutrinos with neutrinos to probe heavy Majorana
neutrinos (HMN) and effective neutrino mass. Furthermore, leptons and
neutrinos can collide into W bosons, which can be also observed with a very
small amount of integrated luminosity and benefit W boson mass precision
measurements.

Through all these stages, a rich variety of physics goals could be achieved,
and may also be useful intermediate steps as the study of the challenging
technologies towards the muon-muon collider by the energy frontier community.

Section II elaborates the physics potentials of these collision schemes.
Technical considerations are presented in Section III. Status and prospects
are presented in Section IV. The work is summarized in Section V.

##  II Physics Potentials

Neutrinos are among the most abundant and least understood of all particles in
the SM that make up our universe. Observation of neutrino oscillations
confirms that at least two types of SM neutrinos have a tiny, but strictly
nonzero, mass. The upper limits on each neutrino mass come from many
experiments including cosmology and direct measurements [19]. For example, the
Karlsruhe Tritium Neutrino (KATRIN) experiment [20] finds an upper limit
$m_{\tilde{\nu}_{\mathrm{e}}}<0.8\,\,\text{eV}$ at the 90% C.L. for the
electron anti-neutrino $\tilde{\nu}_{\mathrm{e}}$. On the other hand, the
direct mass limit on electron neutrinos and muon neutrinos are relatively much
looser [19]. The simplest formalism in which neutrino masses can arise is
through a dimension-5 operator as shown by Weinberg [21], which extends the SM
Lagrangian with

$\displaystyle{\cal L}_{5}=\left(\ {\rm
C}_{5}^{\ell\ell^{\prime}}/\Lambda\right)\big{[}\Phi\\!\cdot\\!\overline{L}^{c}_{\ell}\big{]}\big{[}L_{\ell^{\prime}}\\!\\!\cdot\\!\Phi\big{]},$
(1)

where $\ell,~{}\ell^{\prime}$ are the flavors of the leptons, which can be
electrons, muons or taus; $\Lambda$ is the relevant new physics scale; ${\rm
C}_{5}^{\ell\ell^{\prime}}$ is a flavor-dependent Wilson coefficient;
$L_{\ell}^{T}=(\nu_{\ell},\ell)$ is the left-handed lepton doublet; and $\Phi$
is the SM Higgs doublet with a vacuum expectation value
$v=\sqrt{2}\langle\Phi\rangle\approx 246\,\text{GeV}$. The Weinberg operator
generates the Majorana neutrino masses as
${\color[rgb]{0,0,0}m_{\ell\ell}=C_{5}^{\ell\ell}v^{2}/\Lambda}=\left|\sum_{i}U_{\mathrm{\ell}i}^{2}m_{i}\right|$,
and introduces lepton number violation (LNV).

The ultraviolet (UV) completion of the Weinberg operator can be realised in
the context of “see-saw” models [22], assuming the existence of hypothetical
heavy states, for example the HMN in the type-I seesaw model. Searches for
neutrinoless double beta decay in the decays of heavy nuclei have placed
strong limit, i.e., $m_{\ell\ell}<0.08-0.18\,\text{eV}$ at the 90% C.L. [23].

Figure 1: Energy fraction distributions of products emitted from 1000 GeV and
200 GeV muon beams.

The physics potential of novel collider schemes is elaborated in a step-by-
step manner. We start with using a TeV scale
$\mu^{+}\rightarrow\mathrm{e}^{+}\nu_{\mathrm{e}}\tilde{\nu}_{\mathrm{\mu}}$
beam. Fig. 1 shows the energy distributions of muon decay products from a muon
beam with energy at $1\,\text{TeV}$ and 200 GeV. As the decay angle $\theta$
goes like $\theta\sim 10^{-4}/{\rm E(TeV)}$, the muon decay products will be
more collimated with increasing beam energy [24, 25]. With TeV scale
$\mu^{+}\rightarrow\mathrm{e}^{+}\nu_{\mathrm{e}}\tilde{\nu}_{\mathrm{\mu}}$
and $\mu^{-}\rightarrow\mathrm{e}^{-}\tilde{\nu}_{\mathrm{e}}\nu_{\mu}$ beams
from two sides, there appears the collisions

$\displaystyle\nu_{\mathrm{e}}\tilde{\nu}_{\mathrm{e}}\rightarrow\mathrm{Z}\rightarrow\mu^{+}\mu^{-}.$
(2)

Figure 2: Example diagrams for neutrino antineutrino annihilation into Z boson
and SM-like Z’ boson.

Relevant Feynman diagrams for neutrino antineutrino annihilation into Z and
SM-like Z’ boson are shown as in Fig. 2.

To simulate these processes, we implemented the neutrino energy fraction
function [24] from $200\,\text{GeV}$ muon decay in MadGraph5_aMC@NLO [26]. The
cross section reads 320 pb after requiring the final state muon to satisfy
$p_{\mathrm{T}}>20$ GeV and $|\eta|<3.0$. As for the same process but with
hadronic Z decay, the cross section will be around 5200 pb. Such large cross
section can compensate for the luminosity limitation. For example, with a tiny
integrated luminosity of about $10^{-5}$$~{}\mathrm{fb^{-1}}$, one can already
expect to observe direct neutrino collisions through process. 2 and further
can probe the $\mathrm{Z}\nu\bar{\nu}$ couplings [27], or search for possible
$\nu\bar{\nu}$ resonance. Figure. 3 shows the outgoing muon energy
distributions for neutrino antineutrino annihilation into Z and SM-like Z’
bosons, with Z’ mass set as $150\,\text{GeV}$ and narrow width.

Figure 3: Outgoing muon energy distributions for neutrino antineutrino
annihilation into Z and SM-like Z’ bosons, with Z’ mass set as
$150\,\text{GeV}$ and narrow width.

On the other hand, we can also consider neutrino neutrino collision, which is
specifically sensitive to the Weinberg operator and Majorana neutrino mass.
With $1\,\text{TeV}$
$\mu^{+}\rightarrow\mathrm{e}^{+}\nu_{\mathrm{e}}\tilde{\nu}_{\mathrm{\mu}}$
beams from two sides, some of the main physics processes can be shown as below
(where we only consider $\nu_{\mathrm{e}}$ for simplicity):

$\displaystyle\nu_{\mathrm{e}}\nu_{\mathrm{e}}\rightarrow\mathrm{H}\mathrm{H}$
(3)
$\displaystyle\nu_{\mathrm{e}}\nu_{\mathrm{e}}\rightarrow\mathrm{Z}\mathrm{Z}\,,\mathrm{Z}\mathrm{H}$
(4)
$\displaystyle\nu_{\mathrm{e}}\nu_{\mathrm{e}}\rightarrow\nu_{\mathrm{e}}\nu_{\mathrm{e}}\mathrm{H},$
(5)
$\displaystyle\nu_{\mathrm{e}}\nu_{\mathrm{e}}\rightarrow\nu_{\mathrm{e}}\nu_{\mathrm{e}}\mathrm{Z}\mathrm{Z}\,,\nu_{\mathrm{e}}\nu_{\mathrm{e}}\mathrm{W}\mathrm{W},$
(6)
$\displaystyle\nu_{\mathrm{e}}\nu_{\mathrm{e}}\rightarrow\nu_{\mathrm{e}}\nu_{\mathrm{e}}\mathrm{Z}\mathrm{H}\,,\nu_{\mathrm{e}}\nu_{\mathrm{e}}\mathrm{H}\mathrm{H},$
(7) $\displaystyle\nu_{\mathrm{e}}\nu_{\mathrm{e}}\rightarrow
e^{-}e^{-}\mathrm{W}^{+}\mathrm{W}^{+},$ (8)

The first two are generated using the so called Phenomenological Type I Seesaw
model [28], with HMN mass scanned from several to hundred TeV, and the HMN
mixing element set as $V_{eN}=0.01$ (where N stands for neutrino) by default.
The latter three arise from so called vector boson fusion processes in the SM.
The cross sections read: 11 fb for process. 3 and negligibly small for
process. 4, with HMN mass ${\rm M}_{\rm{N}}=20$ TeV; 133 fb for process. 5; 14
fb for process. 6; 0.17fb for process. 7; 27 fb for process. 8 after requiring
the final state electron to satisfy $p_{\mathrm{T}}>10$ GeV and $|\eta|<5.0$.

Based on a simulation study [24], we expect for 20 TeV HMN, the 95% C.L. limit
on $V_{eN}$ can be set around 0.01, which surpasses current best results from
the CMS experiments by two orders of magnitude [29].

With a TeV scale
$\mu^{+}\rightarrow\mathrm{e}^{+}\nu_{\mathrm{e}}\tilde{\nu}_{\mathrm{\mu}}$
beam, while the collision beams from the other side are $\mathrm{e}^{-}$,
$\mathrm{e}^{+}$ and $\mu^{-}$, respectively, some of the main physics
processes can be shown as below:

$\displaystyle\mathrm{e}^{+}\mathrm{e}^{-}\rightarrow\mathrm{Z}^{0(*)},\,\,\,\nu_{\mathrm{e}}\mathrm{e}^{-}\rightarrow\nu_{\mathrm{e}}\mathrm{e}^{-},\,\,\,\tilde{\nu}_{\mathrm{\mu}}\mathrm{e}^{-}\rightarrow\tilde{\nu}_{\mathrm{\mu}}\mathrm{e}^{-},$
(9)
$\displaystyle\nu_{\mathrm{e}}\mathrm{e}^{+}\rightarrow\mathrm{W}^{+(*)},\,\,\,\tilde{\nu}_{\mathrm{\mu}}\mathrm{e}^{+}\rightarrow\tilde{\nu}_{\mathrm{\mu}}\mathrm{e}^{+},\,\,\,\tilde{\nu}_{\mathrm{\mu}}\mathrm{e}^{+}\rightarrow\tilde{\nu}_{\mathrm{e}}\mu^{+},$
(10)
$\displaystyle\tilde{\nu}_{\mathrm{\mu}}\mu^{-}\rightarrow\mathrm{W}^{-(*)},\,\,\,\nu_{\mathrm{e}}\mu^{-}\rightarrow\nu_{\mathrm{e}}\mu^{-},\,\,\,\nu_{\mathrm{e}}\mu^{-}\rightarrow\mathrm{e}^{-}\nu_{\mu}.$
(11)

We are especially interested in
$\nu_{\mathrm{e}}\mathrm{e}^{+}\rightarrow\mathrm{W}^{+(*)}$, which has a
cross section that depends on $\mathrm{M_{W}}$. Based on a simulation study
done in Ref. [24] and as shown in Fig. 4, we expect a 10 MeV accuracy on
$\mathrm{M_{W}}$ can be achieved with an integrated luminosity of only
0.1$~{}\mathrm{fb^{-1}}$, which is already comparable with the CDF measurement
result [18]. Other potential physics results from such a neutrino lepton
collider include a search for leptophilic gauge bosons, and studies of
neutrino scattering processes to probe the $\mathrm{Z}\nu\nu$ couplings.

Figure 4: Distributions of the outgoing muon energy from
$\nu_{\mathrm{e}}\mathrm{e}^{+}\rightarrow\mathrm{W}^{+(*)}\rightarrow\nu_{\mu}\mu^{+}$
at two collision scenarios: a neutrino beam arising from a 1000 (500) GeV muon
beam, and a 3 (5) GeV positron beam. Clearly visible differences are seen
between the $\mathrm{M_{W}}=80.2$, $80.4$ and $80.6$ GeV cases. The figure
above also shows the energy comparison of output muon when the income muon
energy of 1000 GeV varies by 0.5 GeV at $\mathrm{M_{W}}=80.4\mathrm{GeV}$.
Ratios are defined as distributions in other colors divided by the
distribution in black (${\rm E}_{\mu}^{\mathrm{income}}=$1000 (500) GeV and
$\mathrm{M_{W}}=80.4\mathrm{GeV}$). Error bars include only the statistical
errors.

In addition to neutrino-neutrino and neutrino-lepton collisions, one can
extend to electron muon and muon muon collisions [24]. The collision of an
electron and muon beam leads to less physics background compared with either
an electron-electron or a muon-muon collider, since electron-muon interactions
proceed mostly through higher order vector boson fusion and vector boson
scattering processes, as shown in Fig. 5. The asymmetric collision profile
results in collision products that are boosted towards the electron beam side,
which can be exploited to reduce BIB from the muon beam to a large extent.

With these discussions, one can imagine a lepton collider complex, starting
from colliding order 10 GeV electron and muon beams for the first time in
history and to probe charged lepton flavor violation, then to be upgraded to a
collider with 50-100 GeV electron and 1-3 TeV muon beams to measure Higgs
properties and search for new physics, and finally to be transformed to a TeV
scale muon muon collider. The cost should vary from order 100 millions to a
few billion dollars corresponding to different stages, which make the funding
situation more practical (see also report on electron-muon collider from
[30]).

Figure 5: Cross section dependence on center-of-mass energy of six dominant
physics processes at an electron-muon collider.

All these novel types of collisions will give us rich phenomena unexplored
before and a rich variety of physics goals could be achieved. For neutrino-
neutrino or neutrino-lepton collisions, only a tiny or small integrated
luminosity is needed to yield wonderful physics results. We envision them as
“Richness out of smallness”, in partial answer to great P. W. Anderson’s “More
is different”.

##  III Technical considerations

A neutrino collision complex from high-energy muon beams was discussed in
Refs. [24, 25]. The illustration of the proposed neutrino beam and collider is
shown in Fig. 6. The muon beam is accelerated in the circular section in the
upper left and then extracted into the rectangular section in the lower right.
During each cycle, the muon beam will be squeezed due to Lorentz contraction
and then pass through an arc ($L_{c}$) and linear sections ($L_{l}$), and emit
bunches of collimated neutrinos finally. The electrons from the muon decays
can either be shielded or used for energy calibration through collision with
positrons from the other side. Using the two rings instead of one ring here
allows for more flexibility to accommodate crowded bunches with different time
or space gaps.

Figure 6: An illustration of the proposed neutrino beam and collider. Muons
are accelerated in the circular section in the upper left, and then extracted
into the rectangular section in the lower right. In one of the long edges of
the rectangle, the neutrinos emitted from the muon decays are formed into a
collimated beam. A small modulation of the muon decay angle through vertical
bending, symbolized by the squiggly line, is used to focus the neutrino beam.

The instantaneous luminosity of a neutrino collider would be limited by two
main factors: 1) the intensity of the neutrino beam compared with the incoming
muon beam is suppressed by roughly $L_{l}/L_{c}\sim 0.1$, i.e., the fraction
of the collider ring circumference occupied by the production straight section
[25], 2) the neutrino beam spread, which may still be kept at 10 to 100
microns at the interaction point, by applying a small modulation on muon decay
angle through vertical bending to achieve more focused neutrino beam [35].

We provide more details using the formula for the instantaneous luminosity,

$\displaystyle{\cal L}={N_{\rm beam1}N_{\rm beam2}\over
4\pi\sigma_{x}\sigma_{y}}f_{\rm rep},$ (12)

where $f_{\rm rep}$ is the rate of collisions and is typically 100 kHz (40
MHz) for lepton colliders (hadron colliders), and $N_{\rm beam1,2}$ are the
number of particles in each bunch which can be taken as $\sim
10^{11}\text{--}10^{12}$ [36, 37, 38], $\sigma_{x}$ and $\sigma_{y}$ are the
beam sizes. Take the LHC as an example [39], with $f_{\rm rep}=40$ MHz,
$\sigma_{x,y}=16$ microns, and $N_{\rm beam1,2}=10^{11}$, one can get ${\cal
L}=10^{34}$ cm-2s-1. As for TeV muon colliders [40, 41], with $f_{\rm
rep}=100$ KHz, $\sigma_{x,y}\lesssim 10$ microns, and $N_{\rm
beam1,2}=10^{12}$, then ${\cal L}=10^{33}\text{--}10^{34}$ cm-2s-1. As for the
neutrino neutrino collisions discussed above, there are further suppression
factors from linear over arc ratio ($L^{2}_{l}/L^{2}_{c}\sim 1/100$) with the
exact value depending on the realistic design as shown in Fig. 6, and the
neutrino beam spread which can be around $r_{s}\sim 1000$ microns for
$L_{l}\sim$ 10 to 100 meters. Taking all these into account, a realistic
instantaneous luminosity for neutrino neutrino collisions can reach around
${\cal L}=10^{28}$ cm-2s-1 level. Although it is a small number, however, to
reach the discovery threshold of neutrino antineutrino annihilation process
$\nu_{\mathrm{e}}\tilde{\nu}_{\mathrm{e}}\rightarrow\mathrm{Z}$, a tiny
integrated luminosity of about $10^{-5}$$~{}\mathrm{fb^{-1}}$ is needed, i.e.,
several days of data taking.

To be more practical, we consider the neutrino neutrino collision profile as
shown in Fig. 7, where muon beams flying the linear path (symbolized by
$L_{l}$ to $L_{0}$, where $L_{0}$ is a cut-off parameter defined by the muon
beam size ) radiate neutrinos approximately along a cone with polar angle as
$\theta$. We then estimate the instantaneous luminosity for neutrino neutrino
collisions as below:

$\displaystyle{\cal L}$
$\displaystyle=\frac{L^{2}_{l}}{L^{2}_{c}}\int^{L_{l}}_{L_{0}}\frac{N_{\rm
beam1}N_{\rm beam2}f_{\rm rep}}{L^{2}_{l}\times(4\times 2\pi
x\tan^{2}\theta)}\times dx$
$\displaystyle=\frac{L^{2}_{l}}{L^{2}_{c}}\frac{N_{\rm beam1}N_{\rm
beam2}f_{\rm rep}}{8\pi L^{2}_{l}\tan^{2}\theta}\times\ln(L_{l}/L_{0}),$ (13)

with $L_{l}\tan\theta\sim r_{s}$, and there appears as an enhanced factor of
$\ln(L_{l}/L_{0})/2\sim 2-5$, and thus can further increase the instantaneous
luminosity for neutrino neutrino collisions. Note $L_{0}$ is a cut-off
parameter in above integration formula and defined by the muon beam size,
which can be at the order of 1-10 cm and thus may relax the stringent
requirement on beam cooling of nominal muon collider being pursued.

Figure 7: The neutrino neutrino collision profile, where muon beams flying the
linear path radiate neutrinos approximately along a cone with polar angle as
$\theta$.

The lepton beam from the other collision side is of lower energy with a few
GeV, and the quality can be improved by many high-current high-frequency
techniques. Assuming the neutrinos emitted by TeV scale muon beams and
electron energies around 5 GeV, the instantaneous luminosity can reach
$10^{32}$ to $10^{33}\ {\rm{cm}}^{-2}{\rm{s}}^{-1}$ level. In the following
study, we assume the integrated luminosity to be around 10 to 100
$~{}\mathrm{fb^{-1}}$ in 10 years. The interesting physics process from
neutrino antineutrino annihilation has a huge production cross-section and is
expected to be observed with 200 GeV muon beam and a very tiny integrated
luminosity of data. Therefore the requirement of the muon collider on beam
quality can be relaxed to a large extent.

As for the electron muon and muon muon colliders [24], the physical
infrastructure needed to generate electron and muon beams with energies below
100 GeV would be either a small linear or circular accelerator. A muon beam at
TeV scale can be achieved in a kilometer-sized storage ring. Specifically to
our interest, 200 GeV and 1 TeV muon beams are crucial for two important
physics processes, i.e., neutrino-antineutrino annihilation into Z boson, and
neutrino-neutrino scattering through lepton flavor violation (LFV) process
into two Higgs, respectively.

A set-up that can generate both electron and positron beams, or both muon and
anti-muon beams, or polarized beams, should be considered to further enrich
the physics outcome. Such a lepton collider complex can start from colliding
order 10 GeV electron and muon beams for the first time in history and probe
charged LFV (CLFV), then be upgraded to a collider with 50–100 GeV electron
and 1-3 TeV muon beams to measure Higgs properties and probe BSM physics
including CLFV, and finally to be transformed to a TeV scale muon-muon
collider. Note such a lepton collider complex can also serve as a muon source
at various energy scales for general scientific research.

##  IV Status and Prospects

Muon beams are usually achieved in proton or positron interactions on target,
although there have been progress made recently [32, 33, 34], which are still
facing challenges to overcome on beam cooling or intensity. As a
countermeasure, we can also consider those important physics goals with low
luminosity requirements.

Several related neutrino scattering experiments have been proposed in the last
few decades, including NuTeV [43], NuMAX [44], NuSOnG [45], and nuSTORM [46].
Their motivations include, e.g., making precision neutrino interaction cross
section measurements, or searching for neutrino related non-SM physics.
However, a head-on neutrino lepton collider at the 100 GeV scale is proposed
here for the first time, with rich physics potential.

We are planning to work closely with the international Muon Collider
Collaboration [31] and closely follow domestic and international development
on muon physics.

The underlying Deliberation Document [42] of the 2020 Update of the European
Strategy for Particle Physics stipulates that, in addition to high field
magnets, the accelerator R$\&$D roadmap should contain: “…an international
design study for a muon collider, as it represents a unique opportunity to
achieve a multi-TeV energy domain beyond the reach of $e^{+}e^{-}$ colliders,
and potentially within a more compact circular tunnel than for a hadron
collider. The biggest challenge remains to produce an intense beam of cooled
muons, but novel ideas are being explored”. The international Muon Collider
Collaboration [31] is now leading the R$\&$D study, to establish whether the
investment into a full Conceptual Design Report (“CDR”) and demonstrator for a
muon collider is scientifically justified. The collaboration consists of tens
of renowned institutes and universities over the world, and is aiming to
finalize this study in 5-10 years. In particular, the study is focused on the
high-energy frontier and considers options with a centre-of-mass energy of 3
TeV and of 10 TeV or more. Potential synergies with other projects are also
being pursued.

Our program currently and for the next 2-3 years will be focusing on R$\&$D,
especially physics feasibility study and simulation software. On beam,
hardware and luminosity targets, we are following closely related colliders’
designs (including CEPC and muon colliders). One of the key points of this
program is on simulation and validation of muon decay secondary beams,
especially on luminosity optimization over neutrino beams. Once fixing the
muon source and acceleration options, we will try first to realise 100-200 GeV
muon beam and secondary neutrino beams’ collision. We anticipate to achieve
bunches of important physics results with only a tiny data of
$10^{-5}$$~{}\mathrm{fb^{-1}}$. Afterwards, we will upgrade to 1 TeV muon beam
and neutrino collisions.

We have already achieved a series of physics results with attention, on
neutrino neutrino collisions, neutrino lepton collisions, electron muon and
muon muon collisions [24]. As for the physics analysis parts, one can use the
Monte-Carlo generators such as MadGraph and Pythia to simulate the events at
parton and hadron level, and cross-checked with the Whizard generator widely
used by the CEPC experiment. Afterwards, fast simulations with Delphes or full
simulations with GEANT4 and FLUKA will be studied in detail, which will also
help detector designs for the novel type of colliders.

##  V Summary

In this document, we propose a staged blueprint on future High Energy
Colliders: from neutrino neutrino collision, neutrino lepton collision, to
electron muon and muon muon collisions.

Our physics target and development strategy are of novelty and may enjoy the
first mover advantage. In particular, we propose for the first time the
concept of neutrino neutrino collisions, which can serve as a pre-step of a
high energy muon collider. To be more practical, one may start from neutrino
collisions: notice a tiny or small integrated luminosity is needed for
neutrino neutrino or neutrino lepton collisions to achieve wonderful physics
results as described above. Accordingly, the requirement on muon beams may not
be so stringent.

We will work closely with international and domestic programs on muon source
and acceleration, including the CSNS/EMus, HIAF and CiADs, etc [42]. On the
other hand, as the requirement on muon beam quality is relatively loose for
neutrino collisions , we can also exploit the position driven muon source
method, LEMMA [33]. Besides, Chinese scientists have made many original and
important contributions on several key technologies, such as proton target,
slow muon and muonium generation, etc [42].

We anticipate our program can sustain for over 50 years. Due to the program’s
large scale, the most important tasks in the next few years will be finalizing
a detailed conceptual white book. We believe such a (preparatory) Laboratory
on novel colliders could attract vast international interests and
collaborations.

###### Acknowledgements.

This work is supported in part by the National Natural Science Foundation of
China under Grants No. 12150005, No. 12075004 and No. 12061141002, by MOST
under grant No. 2018YFA0403900.

## References

  * [1] F. Gianotti, CERN Seminar, Update on the Standard Model Higgs searches in ATLAS, July, 4 2012. ATLAS-CONF-2012-093; J. Incandela, CERN Seminar, Update on the Standard Model Higgs searches in CMS, July, 4 2012; The ATLAS Collaboration, Phys. Lett. B 716, 1 (2012); The CMS Collaboration, Phys. Lett. B 716, 30 (2012).
  * [2] https://twiki.cern.ch/twiki/bin/view/CMSPublic/PhysicsResultsCombined
  * [3] High-Luminosity Large Hadron Collider (HL-LHC) Technical Design Report CERN-2017-007-M.
  * [4] CMS Technical Design Report for Phase-2 upgrade, CERN-LHCC-2015-10.
  * [5] ATLAS and CMS Collaborations, arXiv:1902.10229.
  * [6] Physics Briefing Book for the European Strategy for Particle Physics, arXiv:1910.11775.
  * [7] https://home.cern/news/news/physics/particle-physicists-update-strategy-future-field-europe
  * [8] https://linearcollider.org/
  * [9] https://clic.cern/
  * [10] http://cepc.ihep.ac.cn/
  * [11] https://fcc.web.cern.ch/Pages/default.aspx
  * [12] https://indico.fnal.gov/event/43963/contributions/190544
  * [13] https://cerncourier.com/a/sketching-out-a-muon-collider/
  * [14] LHCb Collaboration, Nature Phys. 18 (2022) 3, 277-282.
  * [15] Muon g-2 Collaboration, Phys. Rev. Lett. 126, 141801 (2021).
  * [16] Chiara Aimè et. al., arXiv:2203.07256.
  * [17] Nazar Bartosik et.al., JINST 15 (2020) 05, P05001.
  * [18] T. Aaltonen et al. [CDF], Science 376, no.6589, 170-176 (2022).
  * [19] Particle Data Group, Progress of Theoretical and Experimental Physics, Volume 2020, Issue 8, August 2020, 083C01.
  * [20] M. Aker et al. [KATRIN], Nature Phys. 18, no.2, 160-166 (2022).
  * [21] S. Weinberg, Phys. Rev. Lett. 43, 1566-1570 (1979).
  * [22] P. Minkowski, Phys. Lett. 67B (1977) 421–428; T. Yanagida, Conf. Proc. C7902131 (1979) 95–99; T. Yanagida, Prog. Theor. Phys. 64 (1980) 1103; M. Gell-Mann, P. Ramond, and R. Slansky, Conf. Proc. C790927 (1979) 315–321; R. N. Mohapatra and G. Senjanović, Phys. Rev. Lett. 44 (1980) 912–915.
  * [23] M. Agostini et al. [GERDA], Phys. Rev. Lett. 125, no.25, 252502 (2020); A. Gando et al. [KamLAND-Zen], Phys. Rev. Lett. 117, no.8, 082503 (2016).
  * [24] S. Qian, T. Yang, S. Deng, J. Xiao, L. Gao, A. M. Levin, Q. Li, M. Lu and Z. You, [arXiv:2205.15350 [hep-ph]]; T. Yang, S. Qian, S. Deng, J. Xiao, L. Gao, A. M. Levin, Q. Li, M. Lu and Z. You, [arXiv:2204.11871 [hep-ph]]; S. Qian, C. Li, Q. Li, F. Meng, J. Xiao, T. Yang, M. Lu and Z. You, JHEP 12, 047 (2021) doi:10.1007/JHEP12(2021)047 [arXiv:2109.01265 [hep-ph]]; T. Yang, S. Qian, Z. Guan, C. Li, F. Meng, J. Xiao, M. Lu and Q. Li, Phys. Rev. D 104, no.9, 093003 (2021) doi:10.1103/PhysRevD.104.093003 [arXiv:2107.13581 [hep-ph]]; M. Lu, A. M. Levin, C. Li, A. Agapitos, Q. Li, F. Meng, S. Qian, J. Xiao and T. Yang, Adv. High Energy Phys. 2021, 6693618 (2021) doi:10.1155/2021/6693618 [arXiv:2010.15144 [hep-ph]].
  * [25] B. J. King, AIP Conf. Proc. 530, no.1, 142-164 (2000).
  * [26] J. Alwall et.al., JHEP 07, 079 (2014).
  * [27] S. Davidson, S. Forte, P. Gambino, N. Rius and A. Strumia, JHEP 02, 037 (2002).
  * [28] B. Fuks et.al., Phys. Rev. D 103, no.5, 055005 (2021).
  * [29] CMS Collaboration, CMS-PAS-EXO-21-003.
  * [30] CERN Courier, 2021 March-April.
  * [31] https://muoncollider.web.cern.ch
  * [32] MICE Collaboration, Nature volume 578, pages 53–59 (2020).
  * [33] D. Alesini et.al., arXiv:1905.05747.
  * [34] D. Schulte et.al., CERN Yellow Rep.Monogr. 1 (2022) 145-183.
  * [35] K. Long et.al., Nature Phys. 17, no.3, 289-292 (2021).
  * [36] C. Adolphsen et al. (ILC Collaboration) [arXiv:1306.6353[physics.acc-ph]]
  * [37] A. Abada et al. (FCC Collaboration), Eur.Phys. J.ST 228 (2019) no.2, 261-623
  * [38] CEPC Study Group, IHEP-CEPC-DR-2018-01, IHEP-AC-2018-01 [arXiv:1809.00285[physics.acc-ph]]
  * [39] https://www.lhc-closer.es/taking_a_closer_look_at_lhc/0.luminosity
  * [40] J. P. Delahaye, M. Diemoz, K. Long, B. Mansoulié, N. Pastrone, L. Rivkin, D. Schulte, A. Skrinsky and A. Wulzer, [arXiv:1901.06150 [physics.acc-ph]].
  * [41] F. Bossi and P. Ciafaloni, JHEP 10, 033 (2020) doi:10.1007/JHEP10(2020)033 [arXiv:2003.03997 [hep-ph]].
  * [42] Jian Tang, Liang Li, Ye Yuan, Physics 50, (2021) 4 (in Chinese).
  * [43] https://cds.cern.ch/record/2721371/files/CERN-ESU-016-2020
  * [44] G. P. Zeller et al. [NuTeV], Phys. Rev. Lett. 88, 091802 (2002) [erratum: Phys. Rev. Lett. 90, 239902 (2003)].
  * [45] J. P. Delahaye et al., JINST 13, T06003 (2018).
  * [46] T. Adams et al. [NuSOnG], Int. J. Mod. Phys. A 24, 671-717 (2009).
  * [47] P. Kyberd et al. [nuSTORM], [arXiv:1206.0294 [hep-ex]]; C. C. Ahdida et al. doi:10.17181/CERN.FQTB.O8QN


ThinResNet: A New Baseline for Structured Convolutional Networks Pruning

Hugo Tessier

IMT Atlantique

Lab-STICC, UMR CNRS 6285
29238 Brest, France

Ghouti Boukli Hacene

Sony Europe

R&D Center, Stuttgart Laboratory 1

70327 Stuttgart, Germany

Vincent Gripon

IMT Atlantique

Lab-STICC, UMR CNRS 6285
29238 Brest, France

Pruning is a compression method which aims to improve the efficiency of neural networks by reducing their number of parameters while maintaining a good performance, thus enhancing the performance-to-cost ratio in nontrivial ways. Of particular interest are structured pruning techniques, in which whole portions of parameters are removed altogether, resulting in easier to leverage shrunk architectures.
Since its growth in popularity in the recent years, pruning gave birth to countless papers and contributions, resulting first in critical inconsistencies in the way results are compared, and then to a collective effort to establish standardized benchmarks.
However, said benchmarks are based on training practices that date from several years ago and do not align with current practices.
In this work, we verify how results in the recent literature of pruning hold up against networks that underwent both state-of-the-art training methods and trivial model scaling.
We find that the latter clearly and utterly outperform all the literature we compared to, proving that updating standard pruning benchmarks and re-evaluating classical methods in their light is an absolute necessity.
We thus introduce a new challenging baseline to compare structured pruning to: ThinResNet.

Results for ResNet-50 on the ImageNet ILSVRC2012 dataset.

Results on CIFAR-10. All reference methods use ResNet-56. We provide results for both ThinResNet-56 and ThinResNet-20 to compare the influence of width and depth. The ratio of remaining operations and parameters of ThinResNet-20 is shown while keeping ResNet-56 as the reference, so that a regular ResNet-20 of width 16 is shown as having virtually undergone pruning.

§ INTRODUCTION

Being the state of the art in countless domains, such as computer vision [6], language processing [4] or image generation [36], deep neural networks remain computationally expensive for both learning and inference, which tends to make them unsuitable for many applications, as well as environmentally unfriendly.
This is the reason why a large literature emerged to compress such networks; their goal is usually to maximize a performance-to-cost ratio, where the cost can account for computations, latency, bandwidth, memory, energy usage...
Among them, the field of neural networks pruning [16] is particularly active, and counts numerous publications each year.
Structured pruning [25], where whole portions of the networks are removed altogether, is especially promising, as it leads to more reliable cost reduction compared to non-structured methods.

However, some papers have raised concerns about how pruning methods compare to each others: lack of consistency between benchmarks[3], techniques that may not work well on actual hardware [33], or even unexpected side-effects that make it counter-productive [43], pruning has had many problems in the past, that many papers in the literature have endeavored to solve.

Globally, the field of pruning suffers from not being built on proper theoretical foundations, as well as from missing reliable benchmarks. More precisely, theoretical foundations are based on wild assumptions, such as using tailor expansions, which only hold for small perturbations whereas pruning methods typically nullify a lot of parameters in considered architectures, and only hold for making sure the performance right after pruning remains high, while most if not all of pruning methods include training steps after removing parameters.
Consequently, it is not absurd to question the effectiveness of proposed methods, which is often taken for granted in the literature.
This is why we decided, as a first step, to propose a new, fresh baseline which to compare structured pruning to, based on popular architecture/dataset pairs in the literature (i.e. ResNet-50 [17]/ImageNet ILSVRC2012 [37] and ResNet-56/CIFAR-10 [22]).
This new baseline is both simple and challenging, in order for future pruning methods to showcase their true efficiency.

We therefore propose ThinResNet, that are simply ResNets with a uniformly reduced proportion of channels in each layer (a.k.a. “width”), but trained using state-of-the-art methods.
Such reduced architectures are not only trivial to generate, but also contain no hidden cost [44].
We also release trained models on CIFAR10, CIFAR100 and ImageNet ILSVRC2012, for various target flops and number of parameters, that can be directly used for fine tuning, feature extraction or direct classification on considered classes.

We compare our baseline to a wide array of methods from the literature, that are usually tested on outdated training procedures, in order to showcase the order of magnitude of the gap between these benchmarks and what is actually possible with modern training methods.
We not only state that such a comparison is not unfair, the ultimate goal of pruning staying to maximize the performance-to-cost ratio, but also necessary, because what is measured for a certain baseline performance may not translate well once using more modern training procedures.

We therefore encourage future works to use our ThinResNet as their main point of comparison, as long as no better performing methods are available.
We made both our code and pretrained models available to download on our Github page: <https://github.com/brain-bzh/baseline-pruning>.

To summarize, our contributions are the following:

* We introduce a simple baseline for structured pruning methods, for image classification with ResNets,

* We show that this baseline clearly outperform results from existing literature, and we describe how benchmarks in the field should be updated, for both practical and theoretical reasons,

* We provide the code and the trained models on our Github, including a gradient of ThinResNet ranging from 4Gflops (25M parameters) down to 48Mflops (570k parameters) pretrained on ImageNet ILSVRC2012 and ready to be fine-tuned or deployed for edge applications.

§ BACKGROUND

Compression of neural networks generated a lot of interest during the last decade, and many types of compression methods have been introduced, such as quantization [7], distillation [19], clustering [39] or the one we focus on here: pruning, or the removal of parts of neural networks.

Initially born in the late 80's [23], pruning grew in popularity with the work of Han et al. [16] in 2015.
Since then, many different methods have been published, which can mainly be divided into two different categories: non-structured and structured pruning.

Non-structured pruning simply involves pruning isolated parameters without consideration for the particular layout of the pruning masks and how easy it will be to leverage them on hardware.
This type of pruning has two main advantages: 1) its simplicity makes room for more experimental and theoretical experimentation [16, 11, 34] and 2) its fine-grained nature allows more easily to reach extremely high pruning rates [45], which is especially interesting since some libraries are able to leverage non-structured sparsity only at the condition of exceeding, for example, 95% of zeros in the case of cuSPARSE [47].
Another, more marginal advantage, is that the introduced redundancy in values among parameters allows for a better compression through, for example, Huffman coding [15], which can be interesting for some low-power devices [49].

Structured pruning, instead, prunes larger type of structures, such as whole channels, which has numerous advantages: 1) since it results in a reduction in the architecture itself, any framework and hardware can leverage it, 2) it does not only reduce the number of parameters but also the operations required, as well as the size of intermediate representations, thus reducing the memory utilization during inference, 3) there are enough channels in a network to allow for a reasonably fine-grained and efficient pruning, compared to layers or blocks.
These significant advantages made this type of pruning (“channel” or “filter” pruning, loosely referred as “structured” pruning as it became the standard by default) especially popular very quickly in the literature [25, 31], and the topic of interest for our paper.

Whatever the type of pruning, another aspect has to be taken account of: its distribution across layers.
Indeed, while pruning parameters or filters, it is possible either to set a same criterion on all layers at once, which is called “global” pruning [16, 31], or to set manually a given “local” pruning rate on each layer independently, whether it is the same rate (“uniform”) or not (“non-uniform”) [25].
Theoretically, global pruning should bring better results, as it may not restrict itself to the predefined architectures fixed by local, manual pruning, which could be suboptimal.

However, some contributions have raised doubts about the efficiency of pruning against more trivial baselines trained from scratch [32, 12], or about its ability to produce efficient architectures [44, 43].
Finally, in 2020, Blalock et al. [3] raised the alarm about the consistency of benchmarks in the literature, warning about how few papers actually bothered to compare on the same pairs of architectures and datasets and under the same training conditions: everything remained to be standardized.
They therefore proposed ShrinkBench[https://github.com/JJGO/shrinkbench], to help such an effort, which revealed fruitful as many papers use it nowadays.
Since then, two pairs became very widespread for comparison between methods: ResNet-56/CIFAR-10, performing between 93.5% and 94.5% Top-1 accuracy and ResNet-50/ImageNet ILSVRC2012, performing around 76.15% Top-1 accuracy (cf. Table <ref> and <ref>).
It is debatable whether these choices are the best to test the effectiveness of pruning methods, which is beyond the scope of this paper.

§ METHODOLOGY

Our goal is to confront the literature of structured pruning with what modern training methods are able to provide as the simplest baseline possible, i.e. networks reduced from the start and trained from scratch.
Not only is such a method reminiscent of the concerns of Liu et al. [32] and Gale et al. [12], but it also has the merit of being unequivocal, as we propose the simplest way possible to emulate the behavior of local and uniform channel pruning.

Training conditions

We used the same hyperparameters as those shown on the Pytorch blog[<https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/>].
Even though they were designed for training on ImageNet ILSVRC2012, we found that they delivered comparable improvements when training on CIFAR-10.
Here is a brief summary of the training conditions we used:

* Batch Size: 1024

* Steps: 750,000, adding 5 epochs of warmup

* Optimizer: SGD, with a weight decay of $2\times10^{-5}$ (except for batch-normalization layers that are spared) and a momentum of $0.9$

* Scheduler: linear during the warmup, then cosine annealing

* Learning rate: $5\times10^{-3}$ at the beginning, then $0.5$ at the end of the warmup, then decreases until it reaches 0

* Criterion: cross-entropy loss, with a label smoothing [40] factor of $0.1$

* Data Augmentation:

* TrivialAugment [35]

* Random Erasing [56, 8], with a probability of 0.1

* Mixup [53] with a Beta(0.2, 0.2) distribution

* Cutmix [51] with a Beta(1.0, 1.0) distribution

* FixRes [46] mitigation, reducing the resolution during training from $224\times224$ to $176\times176$

* Exponential Moving Average (EMA), every 32 steps, with a decay of $0.99998$

* Inference Resize: during validation, input images are resized from $224\times224$ to $232\times232$

* Interpolation: bilinear

However, we do not apply Repeated Augmentation [20, 2], as we found that it brings too little improvement for a prohibitive cost in term of training time.

We share our code and provide the trained models whose results are reported in Table <ref>

. We ran each experiment under deterministic conditions, using the same seed 0. We did not report the best accuracy across epochs, but only that of the EMA model at the very last epoch, even when it is not the best result available.

While counting the number of remaining parameters can seem trivial, the case of batch-normalization layers can be ambiguous, as they are eventually fused with convolution layers after training. However, since counting their weights or not has a negligible impact on the ratio of remaining parameters, we kept them.
Concerning the number of operations, we used the torchinfo[<https://github.com/TylerYep/torchinfo>] utility, that gave us counts of “mult-adds” operations (a.k.a. MACs), even though most papers report counts of FLOPs. Since FLOPs are usually almost exactly twice the number of MACs, and since we report pruning ratios instead of raw numbers, we consider the difference between the two to be negligible.

Pseudo-Pruning Strategies

We simply varied the width (i.e. the number of channels) of each convolutional layer in the architecture proportionally, which emulates the resulting architecture of a strictly uniform local structured pruning strategy.
Besides its simplicity, this method has a great advantage: it brings no discrepancies at all between layers, as can be the case when performing pruning [44].

Choice of the ResNet Variant

Although the architecture of ResNet-50 is absolutely standardized, there are actually two competing variants for ResNets dedicated to CIFAR-10, such as ResNet-56 and ResNet-20: their shortcut modules can contain either a $1\times1$ convolution layer or a padding operation.
The padding variant is the one used in ShrinkBench (sourced from another popular repository[<https://github.com/akamaster/pytorch_resnet_cifar10>]).
However, these padding operations are much more constraining when doing any kind of structured pruning that is not uniform, because of the channel dependencies between layers [44].
Even though this did not pose problem with our own pseudo-pruning strategy, it may actually be a significant burden to the rest of the literature.
After having verified that replacing the paddings by the other variant did not impact the performance at all, while increasing the number of parameters by less than 1%, we decided to use the variant using convolutions in order to advocate for its use over that of the padding one.

§ RESULTS

Choice of reference methods

The methods we chose to compare to were selected among the publications of various international conferences.
Were kept those which matched the following criteria: 1) the papers provide results for at least ResNet-56/CIFAR-10 or ResNet-50/ImageNet, 2) they provide their results in the form of a table, since it is almost impossible to obtain exact values otherwise (e.g. from a graph or a figure), 3) the absolute accuracy of each result is provided (or, at least, possible to infer from a baseline), and 4) the pruning rate, in term of operations or parameters, is indicated or possible to infer.
Each of these criteria, that are essential to extract exploitable results from the papers without having to reproduce the experiments, ended up excluding a large portion of papers, so that our final selection only represent a minority of the papers we reviewed.

Structured Pruning on ImageNet ILSVRC2012

Figure <ref> compare results from the literature of structured pruning to our own, for ResNet-50 trained on ImageNet ILSVRC2012.
Results from the reference methods are directly extracted from the original papers (as for all figures in this article).
Since multiple papers report results in term of operations but not parameters, some references are missing for the curve showing the ratio of remaining parameters.
Detailed results of reference methods are reported in Table <ref>, and our own results are shown in Table <ref>.

Structured Pruning on CIFAR-10

Figure <ref> compare results from the literature of structured pruning to our own, for both ResNet-56 and ResNet-20, trained on CIFAR-10.
The reason why we tested two different architectures was to distinguish the impact of width and depth on the efficiency of the architecture and whether the two reduction strategies would lead to different results.
Some references are also missing when showing remaining parameters.
Detailed results of reference methods are reported in Table <ref>, and our own results are shown in Table <ref>.

§ DISCUSSIONS

§.§ A New Baseline

As can be seen in Figure <ref> and <ref>, our results for ResNet-50, ResNet-56 and ResNet-20 beat their counterparts from the literature by a wide margin.
Since these points were obtained using the simplest method possible, while providing networks that are actually reduced without any ambiguity about the efficiency of their implementation on hardware, they can serve as a relevant baseline to beat for future pruning methods.
This is why we provide both the code, to obtain them, and the networks themselves to download on the dedicated Github page.

§.§ The Importance of Proper Training

The implications of our work are not only practical, but also theoretical.
Indeed, the question is not just “what are the best available compressed networks ?” but also “are the current benchmarks reliable, to tell apart methods that work or not ?”.
As mentioned by Tessier et al. [42], an insufficient post-pruning retraining can lead to erroneous conclusions.
Therefore, even though our results are obviously insufficient to tell that pruning does not work, it evidences that, in the absence of a proper, up-to-date benchmark, it is not possible to tell that the previous results would scale the same way once properly retrained.

§.§ New Benchmark Paradigms

All of this means that sticking to out-of-date training conditions, for the sake of making comparison easier, may not turn out to be such a good idea.
Indeed, not only is the accuracy-to-operations ratio (or, more generally, performance-to-cost ratio) not the only thing that matters in the end, from an applicative point of view, but, as we mentioned, old benchmarks can create erroneous conclusions.
Therefore, even though some criteria need to be fixed, such as the performance and cost metrics, the way to present results and make code available, and the datasets on which to compare methods, everything else should likely not be fixed, and instead be kept the most up-to-date possible.

This also means that, ultimately, we should likely not stick to simple ResNets, as we did ourself (even though we considered testing other architectures to be out of the scope of this work).
Indeed, why should we prune them, if other architectures are available while being more efficient than even compressed ResNets, especially if we consider pruning as a subcategory of Neural Architecture Search [42]?
However, keeping at least one reference network is still useful, not only because it still helps making methods easier to compare, but also because it can serve as a sanity check: if a method looks to work well for a given custom architecture but not at all on ResNet-50/ImageNet, then there may be some overlooked variables explaining the good results.

§ CONCLUSION

In this contribution, we propose to compare results from the literature of structured pruning to a very simple baseline method that uses modern training hyperparameters.
It turns out that our results largely outmatch all the references we compared to, and shows that the literature needs to update its benchmarks.
We therefore made our compressed networks available to download, as well as our code, to serve as a new baseline to beat for future pruning methods.
We encourage the literature not only to compare to our results for future methods, but also to renew previous experiments to verify if conclusions that were to be drawn from old benchmarks do scale well to new ones, or if all previous discussions in the literature were unfortunately just a byproduct of insufficient retraining methods.

[1]
Manoj Alwani, Yang Wang, and Vashisht Madhavan.
Decore: Deep compression with reinforcement learning.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 12349–12359, 2022.

[2]
Maxim Berman, Hervé Jégou, Andrea Vedaldi, Iasonas Kokkinos, and
Matthijs Douze.
Multigrain: a unified image embedding for classes and instances.
arXiv preprint arXiv:1902.05509, 2019.

[3]
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag.
What is the state of neural network pruning?
Proceedings of machine learning and systems, 2:129–146, 2020.

[4]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al.
Language models are few-shot learners.
Advances in neural information processing systems,
33:1877–1901, 2020.

[5]
Linhang Cai, Zhulin An, Chuanguang Yang, Yangchun Yan, and Yongjun Xu.
Prior gradient mask guided pruning-aware fine-tuning.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 36, pages 140–148, 2022.

[6]
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu
Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al.
Symbolic discovery of optimization algorithms.
arXiv preprint arXiv:2302.06675, 2023.

[7]
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David.
Binaryconnect: Training deep neural networks with binary weights
during propagations.
Advances in neural information processing systems, 28, 2015.

[8]
Terrance DeVries and Graham W Taylor.
Improved regularization of convolutional neural networks with cutout.
arXiv preprint arXiv:1708.04552, 2017.

[9]
Yuan Cao Di Jiang and Qiang Yang.
On the channel pruning using graph convolution network for
convolutional neural network acceleration.
In Proc. Int. Joint Conf. Artif. Intell, volume 7, pages
3107–3113, 2022.

[10]
Sara Elkerdawy, Mostafa Elhoushi, Hong Zhang, and Nilanjan Ray.
Fire together wire together: A dynamic pruning approach with
self-supervised mask prediction.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 12454–12463, 2022.

[11]
Jonathan Frankle and Michael Carbin.
The lottery ticket hypothesis: Finding sparse, trainable neural
arXiv preprint arXiv:1803.03635, 2018.

[12]
Trevor Gale, Erich Elsen, and Sara Hooker.
The state of sparsity in deep neural networks.
arXiv preprint arXiv:1902.09574, 2019.

[13]
Alireza Ganjdanesh, Shangqian Gao, and Heng Huang.
Interpretations steered network pruning via amortized inferred
saliency maps.
In European Conference on Computer Vision, pages 278–296.
Springer, 2022.

[14]
Shangqian Gao, Feihu Huang, Yanfu Zhang, and Heng Huang.
Disentangled differentiable network pruning.
In European Conference on Computer Vision, pages 328–345.
Springer, 2022.

[15]
Song Han, Huizi Mao, and William J Dally.
Deep compression: Compressing deep neural networks with pruning,
trained quantization and huffman coding.
arXiv preprint arXiv:1510.00149, 2015.

[16]
Song Han, Jeff Pool, John Tran, and William Dally.
Learning both weights and connections for efficient neural network.
Advances in neural information processing systems, 28, 2015.

[17]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 770–778, 2016.

[18]
Zhiqiang He, Yaguan Qian, Yuqi Wang, Bin Wang, Xiaohui Guan, Zhaoquan Gu, Xiang
Ling, Shaoning Zeng, Haijiang Wang, and Wujie Zhou.
Filter pruning via feature discrimination in deep neural networks.
In European Conference on Computer Vision, pages 245–261.
Springer, 2022.

[19]
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.
Distilling the knowledge in a neural network.
arXiv preprint arXiv:1503.02531, 2015.

[20]
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel
Augment your batch: better training with larger batches.
arXiv preprint arXiv:1901.09335, 2019.

[21]
Minsoo Kang and Bohyung Han.
Operation-aware soft channel pruning using differentiable masks.
In International Conference on Machine Learning, pages
5122–5131. PMLR, 2020.

[22]
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.

[23]
Yann LeCun, John Denker, and Sara Solla.
Optimal brain damage.
Advances in neural information processing systems, 2, 1989.

[24]
Seunghyun Lee and Byung Cheol Song.
Ensemble knowledge guided sub-network search and fine-tuning for
filter pruning.
In European Conference on Computer Vision, pages 569–585.
Springer, 2022.

[25]
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf.
Pruning filters for efficient convnets.
arXiv preprint arXiv:1608.08710, 2016.

[26]
Yawei Li, Kamil Adamczewski, Wen Li, Shuhang Gu, Radu Timofte, and Luc
Van Gool.
Revisiting random channel pruning for neural network compression.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 191–201, 2022.

[27]
Yawei Li, Shuhang Gu, Christoph Mayer, Luc Van Gool, and Radu Timofte.
Group sparsity: The hinge between filter pruning and decomposition
for network compression.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 8018–8027, 2020.

[28]
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong
Tian, and Ling Shao.
Hrank: Filter pruning using high-rank feature map.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 1529–1538, 2020.

[29]
Mingbao Lin, Rongrong Ji, Yuxin Zhang, Baochang Zhang, Yongjian Wu, and
Yonghong Tian.
Channel pruning via automatic structure search.
arXiv preprint arXiv:2001.08565, 2020.

[30]
Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang
Ye, Feiyue Huang, and David Doermann.
Towards optimal structured cnn pruning via generative adversarial
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 2790–2799, 2019.

[31]
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui
Learning efficient convolutional networks through network slimming.
In Proceedings of the IEEE international conference on computer
vision, pages 2736–2744, 2017.

[32]
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell.
Rethinking the value of network pruning.
arXiv preprint arXiv:1810.05270, 2018.

[33]
Xiaolong Ma, Sheng Lin, Shaokai Ye, Zhezhi He, Linfeng Zhang, Geng Yuan,
Sia Huat Tan, Zhengang Li, Deliang Fan, Xuehai Qian, et al.
Non-structured dnn weight pruning—is it beneficial in any platform?
IEEE transactions on neural networks and learning systems,
33(9):4930–4944, 2021.

[34]
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov.
Variational dropout sparsifies deep neural networks.
In International Conference on Machine Learning, pages
2498–2507. PMLR, 2017.

[35]
Samuel G Müller and Frank Hutter.
Trivialaugment: Tuning-free yet state-of-the-art data augmentation.
In Proceedings of the IEEE/CVF international conference on
computer vision, pages 774–782, 2021.

[36]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn
High-resolution image synthesis with latent diffusion models.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 10684–10695, 2022.

[37]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma,
Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.
Imagenet large scale visual recognition challenge.
International journal of computer vision, 115:211–252, 2015.

[38]
Haopu Shang, Jia-Liang Wu, Wenjing Hong, and Chao Qian.
Neural network pruning by cooperative coevolution.
arXiv preprint arXiv:2204.05639, 2022.

[39]
Guillaume Soulié, Vincent Gripon, and Maëlys Robert.
Compression of deep neural networks on the fly.
In Artificial Neural Networks and Machine Learning–ICANN 2016:
25th International Conference on Artificial Neural Networks, Barcelona,
Spain, September 6-9, 2016, Proceedings, Part II 25, pages 153–160.
Springer, 2016.

[40]
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew
Rethinking the inception architecture for computer vision.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 2818–2826, 2016.

[41]
Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang
Scop: Scientific control for reliable neural network pruning.
Advances in Neural Information Processing Systems,
33:10936–10947, 2020.

[42]
Hugo Tessier.
Convolutional neural networks pruning and its application to
embedded vision systems.
PhD thesis, Ecole nationale supérieure Mines-Télécom
Atlantique Bretagne Pays de la Loire, 2023.

[43]
Hugo Tessier, Vincent Gripon, Mathieu Léonardon, Matthieu Arzel, David
Bertrand, and Thomas Hannagan.
Energy consumption analysis of pruned semantic segmentation networks
on an embedded gpu.
In International Conference on System-Integrated Intelligence,
pages 553–563. Springer, 2022.

[44]
Hugo Tessier, Vincent Gripon, Mathieu Léonardon, Matthieu Arzel, David
Bertrand, and Thomas Hannagan.
Leveraging structured pruning of convolutional neural networks.
In 2022 IEEE Workshop on Signal Processing Systems (SiPS),
pages 1–6. IEEE, 2022.

[45]
Hugo Tessier, Vincent Gripon, Mathieu Léonardon, Matthieu Arzel, Thomas
Hannagan, and David Bertrand.
Rethinking weight decay for efficient neural network pruning.
Journal of Imaging, 8(3):64, 2022.

[46]
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou.
Fixing the train-test resolution discrepancy.
Advances in neural information processing systems, 32, 2019.

[47]
Zhuliang Yao, Shijie Cao, Wencong Xiao, Chen Zhang, and Lanshun Nie.
Balanced sparsity for efficient dnn inference on gpu.
In Proceedings of the AAAI conference on artificial
intelligence, volume 33, pages 5676–5683, 2019.

[48]
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang.
Gate decorator: Global filter pruning method for accelerating deep
convolutional neural networks.
Advances in neural information processing systems, 32, 2019.

[49]
Hamoud Younes, Hugo Le Blevec, Mathieu Léonardon, and Vincent Gripon.
Inter-operability of compression techniques for efficient deployment
of cnns on microcontrollers.
In International Conference on System-Integrated Intelligence,
pages 543–552. Springer, 2022.

[50]
Sixing Yu, Arya Mazaheri, and Ali Jannesari.
Topology-aware network pruning using multi-stage graph embedding and
reinforcement learning.
In International conference on machine learning, pages
25656–25667. PMLR, 2022.

[51]
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and
Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with
localizable features.
In Proceedings of the IEEE/CVF international conference on
computer vision, pages 6023–6032, 2019.

[52]
Edouard Yvinec, Arnaud Dapogny, Matthieu Cord, and Kevin Bailly.
Singe: Sparsity via integrated gradients estimation of neuron
Advances in Neural Information Processing Systems,
35:35392–35403, 2022.

[53]
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.
mixup: Beyond empirical risk minimization.
arXiv preprint arXiv:1710.09412, 2017.

[54]
Miao Zhang, Li Wang, David Campos, Wei Huang, Chenjuan Guo, and Bin Yang.
Weighted mutual learning with diversity-driven model compression.
Advances in Neural Information Processing Systems,
35:11520–11533, 2022.

[55]
Shaochen Zhong, Guanqun Zhang, Ningjia Huang, and Shuai Xu.
Revisit kernel pruning with lottery regulated grouped convolutions.
In International Conference on Learning Representations, 2021.

[56]
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.
Random erasing data augmentation.
In Proceedings of the AAAI conference on artificial
intelligence, volume 34, pages 13001–13008, 2020.

Our results for the pairs ThinResNet-50/ImageNet ILSVRC2012, ThinResNet-56/CIFAR-10 and ThinResNet-20/CIFAR-10. We report the width (the number of channels in the initial embedding), the Top-1 accuracy (Acc.) in percents, the percentage of remaining operations (R.O.) as well as that of remaining parameters (R.P.).
We also report the latency (Lat.) in milliseconds.
These values were measured using a script that we provide in our Github repository.
The measurements were performed on Intel Xeon Silver 4208 CPU, as doing so gave more steady results compared to GPU.
Each experiment involved running inferences during 1000 seconds, and we report the average of all the  points generated during this duration.

Results from the literature of structured pruning, for ResNet-56 on CIFAR-10. Values are directly extracted from the original papers. We report the width (the number of channels in the initial embedding), the Top-1 accuracy (Acc.) in percents, the percentage of remaining operations (R.O.) as well as that of remaining parameters (R.P.).

Results from the literature of structured pruning, for ResNet-50 on ImageNet ILSVRC2012. Values are directly extracted from the original papers. We report the width (the number of channels in the initial embedding), the Top-1 accuracy (Acc.) in percents, the percentage of remaining operations (R.O.) as well as that of remaining parameters (R.P.).