# Charged spinning fermionic configurations and a mass gap

Vladimir Dzhunushaliev<EMAIL_ADDRESS>Department of Theoretical
and Nuclear Physics, Al-Farabi Kazakh National University, Almaty 050040,
Kazakhstan  Institute of Nuclear Physics, Almaty 050032, Kazakhstan
Academician J. Jeenbaev Institute of Physics of the NAS of the Kyrgyz
Republic, 265 a, Chui Street, Bishkek 720071, Kyrgyzstan  Laboratory for
Theoretical Cosmology, International Centre of Gravity and Cosmos, Tomsk State
University of Control Systems and Radioelectronics (TUSUR), Tomsk 634050,
Russia  Vladimir Folomeev<EMAIL_ADDRESS>Institute of Nuclear Physics,
Almaty 050032, Kazakhstan  Academician J. Jeenbaev Institute of Physics of the
NAS of the Kyrgyz Republic, 265 a, Chui Street, Bishkek 720071, Kyrgyzstan
Laboratory for Theoretical Cosmology, International Centre of Gravity and
Cosmos, Tomsk State University of Control Systems and Radioelectronics
(TUSUR), Tomsk 634050, Russia

###### Abstract

We consider a self-consistent axially symmetric system supported by a
classical nonlinear spinor field minimally coupled to electric and magnetic
Maxwell fields. The presence of the nonlinearity of the spinor field ensures
the existence of a minimum positive energy of the system (a mass gap), of a
minimum charge (a charge gap), and of a minimum magnetic moment. In turn, the
presence of the electric charge results in qualitative changes in the behavior
of physical characteristics of the systems under consideration as compared
with the case of an electrically neutral spinor field. It is shown that, with
a suitable choice of free system parameters, there exists a regular finite-
energy particlelike solution describing a localized spinning object whose
physical parameters correspond to the main characteristics of an
electron/positron (including the spin equal to $1/2$), but with the
characteristic size comparable to the corresponding Compton wavelength. Also,
we show that four local Dirac equations are equivalent to two nonlocal
equations.

nonlinear spinor field; spinning particlelike solutions; mass, charge, and
magnetic moment gaps; total angular momentum

###### pacs:

11.90.+t,11.15.-q

##  I Introduction

Nonlinear equations describing various physical systems have been the object
of numerous investigations in different aspects. The bulk of such studies have
been mainly focused on a consideration of the nonlinear Shrödinger and Klein-
Gordon equations involving different potentials. The first of these equations
permits one to describe various phenomena and processes within condensed
matter physics, nonlinear optics, atomic and mathematical physics. In turn,
the Klein-Gordon equation is widely used in modeling various particlelike
objects in condensed matter and mathematical physics, including strongly
gravitating systems.

Much less attention was paid to investigations of the nonlinear Dirac
equation. Such an equation was initially introduced by D. Ivanenko
Ivanenko:1938 . Subsequently, it was analyzed in the works Finkelstein:1951zz
; Finkelstein:1956 , where possible forms of nonlinear terms were suggested.
Following these ideas, W. Heisenberg tried to employ this equation as a
fundamental equation suitable for describing the properties of an electron
Heisenberg . Later, one of forms of the nonlinear Dirac equation was employed
for an approximate description of the properties of hadrons (this approach is
called the Nambu-Jona-Lasinio model Nambu:1961 ; for a review, see Ref.
Volkov:2006 ). In turn, bearing in mind that the systems with a nonlinear
spinor field contain a mass gap Finkelstein:1951zz ; Finkelstein:1956 ;
Soler:1970xp ; Ranada:1973hna , in Ref. Ranada:1974hx , the authors tried to
describe extended particles (hadrons) possessing the smallest possible energy.
On the other hand, in the case of fermions with zero bare mass, Ref.
Gross:1974jv  suggests a toy model of quark confinement in quantum
chromodynamics. In addition, the nonlinear Dirac equations may be used as
effective theories in various fields of atomic, nuclear, particle, and
gravitational physics Alvarez:1981yh ; Ionescu:1987jr ; Esteban:1995 ;
Esteban:2002 ; Zecca:2002dq ; Bronnikov:2004uu ; Bronnikov:2009na ;
Adanhounme:2012cm ; Saha:2016cbu ; Dzhunushaliev:2018jhj ;
Dzhunushaliev:2019kiy ; Dzhunushaliev:2019uft ; Bronnikov:2019nqa .

In quantum chromodynamics, there is a well-known problem to prove the
existence of a minimum value of the mass, i.e., of a mass gap, in non-Abelian
quantum Yang-Mills theory. This problem is very nontrivial and has not been
solved yet. One of possible approaches towards solving this problem might be a
consideration of simpler problems when quantum systems are replaced by some
approximate classical systems. In this case, if one could show that for such
classical configurations a mass gap might occur, this could be treated as a
possible indication for the existence of the mass gap in quantum systems. As
shown in our previous investigations Dzhunushaliev:2019ham ;
Dzhunushaliev:2020qwf ; Dzhunushaliev:2021apa ; Dzhunushaliev:2022pkz , in the
systems supported by classical non-Abelian fields coupled to nonlinear spinor
fields, there is the possibility of obtaining a mass gap. From this point of
view, such classical systems may be thought of as approximately describing
realistic quantum systems.

The systems considered by us earlier in Refs. Dzhunushaliev:2019ham ;
Dzhunushaliev:2020qwf ; Dzhunushaliev:2021apa ; Dzhunushaliev:2022pkz  are
spherically symmetric. Obviously, in the more general case the spherical
symmetry can already be violated, for example, because of the presence of
magnetic Maxwell and/or color fields. Correspondingly, this requires a
generalization of the above models. As a first step in this direction, one can
consider a simplified situation where the system with a nonlinear spinor field
contains only Abelian fields. Consistent with this, the present paper studies
a system supported by a classical nonlinear spinor field minimally coupled to
Maxwell electric and magnetic (dipole) fields. Due to the presence of the
dipole magnetic field, the system is inevitably axisymmetric, and therefore a
consideration will be carried out in a general form without any simplifying
assumptions as to the smallness of the electromagnetic fields, as was done
earlier, for example, in Refs. Ranada:1973hna ; Ranada:1974hx . This will
enable us to study the cases where the contribution to the energy-momentum
tensor coming from the electromagnetic fields can be comparable to that of the
spinor field. It will be shown that such a contribution results in qualitative
changes in the physical characteristics of the systems under consideration.

Notice here that in the present paper we consider a system supported by a
classical spinor field. Following Ref. ArmendarizPicon:2003qk , such a field
is meant to be a set of four complex-valued spacetime functions transforming
according to the spinor representation of the Lorentz group. In turn,
realistic spin-1/2 particles must evidently be described by quantum spinor
fields, and it is furthermore believed that there is no classical limit for
quantum spinor fields. However, classical spinors can be thought of as arising
from some effective description of more complex quantum systems (for arguments
in favor of the possibility of the existence of classical spinors, see Ref.
ArmendarizPicon:2003qk ).

##  II The model

We consider localized configurations consisting of a spinor field $\psi$
minimally coupled to Maxwell fields. The corresponding total Lagrangian for
such a system can be represented in the form (we use natural units with
$c=\hbar=1$ throughout)

$L_{\text{tot}}=\frac{\imath}{2}\left(\bar{\psi}\gamma^{\mu}\psi_{;\mu}-\bar{\psi}_{;\mu}\gamma^{\mu}\psi\right)-m\bar{\psi}\psi-F(S)-\frac{1}{4}F_{\mu\nu}F^{\mu\nu},$
(1)

where $m$ is a bare mass of the fermion, $F(S)$ is in general an arbitrary
nonlinear term with the invariant $S$ (see below), and the electromagnetic
field tensor $F_{\mu\nu}=\partial_{\mu}A_{\nu}-\partial_{\nu}A_{\mu}$. The
semicolon denotes the covariant derivative defined as
$\psi_{;\mu}=[\partial_{\mu}+1/8\,\omega_{ab\mu}\left(\gamma^{a}\gamma^{b}-\gamma^{b}\gamma^{a}\right)+\imath\,QA_{\mu}]\psi$
with $\gamma^{a}$ being the Dirac matrices in flat space; the term
$\imath\,QA_{\mu}\psi$ describes the interaction between the spinor and
Maxwell fields with the coupling constant $Q$. In turn, the Dirac matrices in
curvilinear coordinates, $\gamma^{\mu}=e_{a}^{\phantom{a}\mu}\gamma^{a}$, are
obtained using the tetrad $e_{a}^{\phantom{a}\mu}$, and $\omega_{ab\mu}$ is
the spin connection [for its definition, see Ref. Lawrie2002 , Eq. (7.135)].
In the above expressions, $\mu,\nu=0,1,2,3$ are spacetime indices and
$a,b=0,1,2,3$ are tetrad indices. In what follows, we use the Weyl
representation of the Dirac matrices,

$\gamma^{0}=\begin{pmatrix}0&1\\\
1&0\end{pmatrix},\quad\gamma^{k}=\begin{pmatrix}0&\sigma^{k}\\\
-\sigma^{k}&0\end{pmatrix},$

where $k=1,2,3$ and $\sigma^{k}$ are the Pauli matrices.

Varying the action with the Lagrangian (1) with respect to the spinor field
and to the vector potential $A_{\mu}$, we derive the corresponding Dirac and
Maxwell field equations

$\displaystyle\imath\gamma^{\mu}\psi_{;\mu}-m\psi-\frac{\partial
F}{\partial\bar{\psi}}$ $\displaystyle=$ $\displaystyle 0,$ (2)
$\displaystyle\frac{1}{\sqrt{-g}}\frac{\partial}{\partial
x^{\nu}}\left(\sqrt{-g}F^{\mu\nu}\right)$ $\displaystyle=$
$\displaystyle-Q\bar{\psi}\gamma^{\mu}\psi.$ (3)

In the present paper, we take the following simplest self-interaction term

$F(S)=-\frac{\lambda}{2}\left(\bar{\psi}\psi\right)^{2},$

where $\lambda$ is some free nonlinearity parameter. Classical spinor fields
with such a nonlinearity have been considered, for instance, in Refs.
Finkelstein:1951zz ; Finkelstein:1956 ; Soler:1970xp ; Ranada:1973hna ;
Ranada:1974hx ; Dzhunushaliev:2018jhj ; Dzhunushaliev:2019kiy ;
Dzhunushaliev:2019uft .

From the Lagrangian (1), one can also obtain the corresponding energy-momentum
tensor of the system under consideration (already in a symmetric form)

$T_{\mu}^{\nu}=\frac{\imath}{4}g^{\nu\rho}\left[\bar{\psi}\gamma_{\mu}\psi_{;\rho}+\bar{\psi}\gamma_{\rho}\psi_{;\mu}-\bar{\psi}_{;\mu}\gamma_{\rho}\psi-\bar{\psi}_{;\rho}\gamma_{\mu}\psi\right]-\delta_{\mu}^{\nu}L_{\text{sp}}-F^{\nu\rho}F_{\mu\rho}+\frac{1}{4}\delta_{\mu}^{\nu}F_{\alpha\beta}F^{\alpha\beta}.$
(4)

Taking into account the Dirac equation (2) and the corresponding adjoint
equation for $\bar{\psi}$, the Lagrangian for the spinor field appearing in
Eq. (4) becomes

$L_{\text{sp}}=-F(S)+\frac{1}{2}\left(\bar{\psi}\frac{\partial
F}{\partial\bar{\psi}}+\frac{\partial F}{\partial\psi}\psi\right).$

In the present paper, we take the stationary Ansatz for the spinor field in
the form similar to that of Ref. Herdeiro:2019mbz ,

$\psi=e^{\imath\left(M\varphi-\Omega t\right)}\begin{pmatrix}\psi_{1}\\\
\psi_{2}\\\ \psi_{2}^{*}\\\ \psi_{1}^{*}\end{pmatrix},$ (5)

where $\Omega$ is the spinor frequency, $M$ is a half-integer parameter (the
azimuthal number). For our purposes, it is convenient to represent the
components of the spinor appearing in (5) in the following form:

$\psi_{1}=\frac{1}{2}\left[X+Y+\imath\left(V+W\right)\right],\quad\psi_{2}=\frac{1}{2}\left[X-Y+\imath\left(V-W\right)\right],$

where the functions $X,Y,V$, and $W$ depend only on the spherical coordinates
$r$ and $\theta$, and the line element is

$ds^{2}=dt^{2}-dr^{2}-r^{2}\left(d\theta^{2}+\sin^{2}\theta
d\varphi^{2}\right).$

The Ansatz for the Maxwell field is taken to be

$A_{\mu}=\\{\phi(r,\theta),0,0,\sigma(r,\theta)\\},$ (6)

i.e., it contains an electric and a magnetic potentials. This Ansatz implies
the presence of the following nonzero components of the electric and magnetic
fields:

$E_{r}=-\frac{\partial\phi}{\partial r},\quad
E_{\theta}=-\frac{\partial\phi}{\partial\theta},\quad
H_{r}=-\frac{\csc\theta}{r^{2}}\frac{\partial\sigma}{\partial\theta},\quad
H_{\theta}=\csc\theta\frac{\partial\sigma}{\partial r}.$ (7)

##  III Equations and solutions

Substituting the Ansätze (5) and (6) in the field equations (2) and (3), one
can obtain the following set of six partial differential equations:

$\displaystyle\tilde{X}_{,x}+\frac{\tilde{X}}{x}-\frac{\tilde{W}_{,\theta}}{x}-\frac{\cot\frac{\theta}{2}}{2x}\tilde{W}+\tilde{Q}\left(-\frac{\csc\theta}{x}\tilde{W}\tilde{\sigma}+\tilde{V}\tilde{\phi}\right)-\left(1+\tilde{\Omega}\right)\tilde{V}+U_{2}\tilde{V}$
$\displaystyle=0,$ (8)
$\displaystyle\tilde{Y}_{,x}+\frac{\tilde{Y}}{x}-\frac{\tilde{V}_{,\theta}}{x}+\frac{\tan\frac{\theta}{2}}{2x}\tilde{V}+\tilde{Q}\left(\frac{\csc\theta}{x}\tilde{V}\tilde{\sigma}-\tilde{W}\tilde{\phi}\right)-\left(1-\tilde{\Omega}\right)\tilde{W}+U_{2}\tilde{W}$
$\displaystyle=0,$ (9)
$\displaystyle\tilde{V}_{,x}+\frac{\tilde{V}}{x}+\frac{\tilde{Y}_{,\theta}}{x}+\frac{\cot\frac{\theta}{2}}{2x}\tilde{Y}+\tilde{Q}\left(\frac{\csc\theta}{x}\tilde{Y}\tilde{\sigma}-\tilde{X}\tilde{\phi}\right)-\left(1-\tilde{\Omega}\right)\tilde{X}+U_{2}\tilde{X}$
$\displaystyle=0,$ (10)
$\displaystyle\tilde{W}_{,x}+\frac{\tilde{W}}{x}+\frac{\tilde{X}_{,\theta}}{x}-\frac{\tan\frac{\theta}{2}}{2x}\tilde{X}+\tilde{Q}\left(-\frac{\csc\theta}{x}\tilde{X}\tilde{\sigma}+\tilde{Y}\tilde{\phi}\right)-\left(1+\tilde{\Omega}\right)\tilde{Y}+U_{2}\tilde{Y}$
$\displaystyle=0,$ (11)
$\displaystyle\tilde{\phi}_{,xx}+\frac{2}{x}\tilde{\phi}_{,x}+\frac{1}{x^{2}}\tilde{\phi}_{,\theta\theta}+\frac{\cot\theta}{x^{2}}\tilde{\phi}_{,\theta}+\tilde{Q}\,U_{1}$
$\displaystyle=0,$ (12)
$\displaystyle\tilde{\sigma}_{,xx}+\frac{1}{x^{2}}\tilde{\sigma}_{,\theta\theta}-\frac{\cot\theta}{x^{2}}\tilde{\sigma}_{,\theta}+2\,\tilde{Q}\,x\sin\theta\,U_{3}$
$\displaystyle=0,$ (13)

where

$U_{1}=\tilde{X}^{2}+\tilde{Y}^{2}+\tilde{V}^{2}+\tilde{W}^{2},\quad
U_{2}=\tilde{X}^{2}-\tilde{Y}^{2}-\tilde{V}^{2}+\tilde{W}^{2},\quad
U_{3}=\tilde{X}\tilde{Y}+\tilde{V}\tilde{W}$

and the azimuthal number is taken to be $M=1/2$ throughout the paper. These
equations are written in terms of the following dimensionless variables:
$x=mr$, $\tilde{\Omega}=\Omega/m$, $\tilde{Q}=Q/\left(m\sqrt{\lambda}\right)$,
$\tilde{X},\tilde{Y},\tilde{V},\tilde{W}=\sqrt{\lambda/m}\,X,Y,V,W$,
$\tilde{\phi}=\sqrt{\lambda}\,\phi$, $\tilde{\sigma}=m\sqrt{\lambda}\,\sigma$.
The lower indices denote differentiation with respect to the corresponding
coordinate. Notice that these equations do not explicitly contain the
nonlinearity parameter $\lambda$ and are invariant with respect to multiplying
the spinor functions by $-1$. That is, the system contains only two free
parameters, $\tilde{\Omega}$ and $\tilde{Q}$, whose values will be varied to
obtain solutions describing configurations with different physical
characteristics.

###  III.1 Physical quantities

Let us now write down expressions for some physically interesting parameters
of the systems under consideration. The total dimensionless mass of the system
can be found in the form

$\tilde{M}_{\text{tot}}\equiv m\lambda
M_{\text{tot}}=2\pi\int_{0}^{\infty}dx\int_{0}^{\pi}d\theta\,\tilde{T}_{t}^{t}x^{2}\sin\theta,$
(14)

where the dimensionless $(^{t}_{t})$-component of the energy-momentum tensor
(4) is

$\tilde{T}_{t}^{t}=\frac{1}{2}\left[\tilde{\phi}_{,x}^{2}+\frac{\csc^{2}\theta}{x^{2}}\tilde{\sigma}_{,x}^{2}+\frac{1}{x^{2}}\tilde{\phi}_{,\theta}^{2}+\frac{\csc^{2}\theta}{x^{4}}\tilde{\sigma}_{,\theta}^{2}+2\left(\tilde{\Omega}-\tilde{Q}\tilde{\phi}\right)U_{1}+U_{2}^{2}\right].$
(15)

The total dimensionless angular momentum

$\tilde{J}_{\text{tot}}\equiv m^{2}\lambda
J_{\text{tot}}=-2\pi\int_{0}^{\infty}dx\int_{0}^{\pi}d\theta\,\tilde{T}_{\varphi}^{t}x^{2}\sin\theta,$
(16)

where the dimensionless $(^{t}_{\varphi})$-component of the energy-momentum
tensor (4) is

$\begin{split}\tilde{T}_{\varphi}^{t}=&\tilde{\phi}_{,x}\tilde{\sigma}_{,x}+\frac{1}{x^{2}}\tilde{\phi}_{,\theta}\tilde{\sigma}_{,\theta}-\frac{1}{4}\left(1+2\,\tilde{Q}\,\tilde{\sigma}\right)U_{1}+x\sin\theta\left(\tilde{\Omega}-\tilde{Q}\tilde{\phi}\right)U_{3}\\\
&+\frac{1}{2}\sin\theta\left(\tilde{W}\tilde{X}-\tilde{V}\tilde{Y}\right)+\frac{1}{4}\cos\theta\left(\tilde{X}^{2}-\tilde{Y}^{2}+\tilde{V}^{2}-\tilde{W}^{2}\right).\end{split}$
(17)

The occurrence of a nonzero angular momentum is due to the presence in the
system of (i) a single fermion possessing an intrinsic angular momentum; and
(ii) the crossed electric and magnetic fields. For this reason, in analogy to
quantum particles possessing the quantum-mechanical spin, such configurations
can be treated as spinning ones.

The total dimensionless Noether charge

$\tilde{Q}_{\text{tot}}\equiv
m^{2}\lambda\,Q_{\text{tot}}=2\pi\int_{0}^{\infty}dx\int_{0}^{\pi}d\theta\,\tilde{j}^{t}x^{2}\sin\theta,$
(18)

where the dimensionless temporal component of the current density
$\tilde{j}^{t}=U_{1}$. Note here that the normalization condition
$Q_{\text{tot}}=1$ corresponds to one-particle solutions; in this case the
coupling constant $Q$ will correspond to an electric charge of the system, and
below we will be interested mostly in such configurations.

Magnetic moment of the system under investigation can be calculated in a
standard way by considering the electric current flowing perpendicular to a
meridional plane (see, e.g., the textbook Blokh ). As a result, one can obtain
the following expression for the dimensionless magnetic dipole moment:

$\tilde{\mu}_{m}\equiv
m^{2}\sqrt{\lambda}\,\mu_{m}=-2\pi\tilde{Q}\int_{0}^{\infty}dx\int_{0}^{\pi}d\theta\,U_{3}x^{3}\sin^{2}\theta.$
(19)

Finally, the dimensionless gyromagnetic ratio $g$, expressed in units of
$q_{e}/\left(2M_{\text{tot}}\right)$ [where $q_{e}$ is the electric charge,
see Eq. (22) below], is defined from the relation

$\mu_{m}=g\frac{q_{e}}{2}\frac{J_{\text{tot}}}{M_{\text{tot}}}\quad\Rightarrow\quad
g=2\frac{\tilde{\mu}_{m}\tilde{M}_{\text{tot}}}{\tilde{q}_{e}\tilde{J}_{\text{tot}}},$
(20)

where $\tilde{q}_{e}\equiv m\sqrt{\lambda}\,q_{e}$ is the dimensionless
electric charge.

###  III.2 Boundary conditions and a numerical approach

We will seek globally regular finite-energy nodeless solutions of the set of
six partial differential equations (8)-(13). To do this, it is necessary to
impose appropriate boundary conditions for the spinor and Maxwell fields. The
behavior of solutions of Eqs. (8)-(13) in the vicinity of the boundaries of
the domain of integration implies the following boundary conditions:

$\begin{split}&\left.\frac{\partial\tilde{X}}{\partial
x}\right|_{x=0}=\left.\frac{\partial\tilde{W}}{\partial
x}\right|_{x=0}=\left.\frac{\partial\tilde{\phi}}{\partial
x}\right|_{x=0}=0,\left.\tilde{Y}\right|_{x=0}=\left.\tilde{V}\right|_{x=0}=\left.\tilde{\sigma}\right|_{x=0}=0;\\\
&\left.\tilde{X}\right|_{x=\infty}=\left.\tilde{Y}\right|_{x=\infty}=\left.\tilde{V}\right|_{x=\infty}=\left.\tilde{W}\right|_{x=\infty}=\left.\tilde{\phi}\right|_{x=\infty}=\left.\tilde{\sigma}\right|_{x=\infty}=0;\\\
&\left.\frac{\partial\tilde{X}}{\partial\theta}\right|_{\theta=0}=\left.\frac{\partial\tilde{V}}{\partial\theta}\right|_{\theta=0}=\left.\frac{\partial\tilde{\phi}}{\partial\theta}\right|_{\theta=0}=0,\left.\tilde{Y}\right|_{\theta=0}=\left.\tilde{W}\right|_{\theta=0}=\left.\tilde{\sigma}\right|_{\theta=0}=0;\\\
&\left.\frac{\partial\tilde{Y}}{\partial\theta}\right|_{\theta=\pi}=\left.\frac{\partial\tilde{W}}{\partial\theta}\right|_{\theta=\pi}=\left.\frac{\partial\tilde{\phi}}{\partial\theta}\right|_{\theta=\pi}=0,\left.\tilde{X}\right|_{\theta=\pi}=\left.\tilde{V}\right|_{\theta=\pi}=\left.\tilde{\sigma}\right|_{\theta=\pi}=0.\end{split}$

For numerical computations, it is convenient to introduce the compactified
radial coordinate

$\bar{x}=\frac{x}{1+x}$ (21)

in order to map the infinite interval $[0,\infty)$ to the finite region
$[0,1]$. The results of numerical computations for axisymmetric systems
presented below have been obtained using the Intel MKL PARDISO sparse direct
solver and the CESDSOL library, and also verified for some particular cases
using the package FIDISOL fidisol . These packages provide an iterative
procedure for obtaining an exact solution starting from some approximate
solution (an initial guess). As the initial guess, it is possible to use the
solutions describing configurations in the absence of electric and magnetic
fields Soler:1970xp . The equations (8)-(13) have been solved on a grid of
$200\times 100$ points which covers the integration region $0\leq\bar{x}\leq
1$ [given by the compactified radial coordinate (21)] and
$0\leq\theta\leq\pi$.

###  III.3 Numerical solutions

Figure 1:  The total mass of the system $\tilde{M}_{\text{tot}}$ as a function
of the spinor frequency $\tilde{\Omega}$ for different values of the coupling
constant $\tilde{Q}$. The inset shows the dependence of the total mass on
$\tilde{Q}$ for different values of $\tilde{\Omega}$ (shown by the numbers
near the points) corresponding to the location of the mass gap.

In contrast to the case of a linear spinor field, in the nonlinear case, there
is a family of solutions, depending continuously on two parameters – the
frequency $\tilde{\Omega}$ and the coupling constant $\tilde{Q}$, whose values
completely determine all physical characteristics of the configurations under
consideration. To illustrate this, Fig. 1 shows the spectrum of the total mass
(14) of the systems under investigation as a function of $\tilde{\Omega}$ for
some fixed values of $\tilde{Q}$. It is seen from these graphs that the
behavior of the dependence $\tilde{M}_{\text{tot}}(\tilde{\Omega})$ is largely
determined by the value of the coupling constant $\tilde{Q}$. Namely, the
numerical calculations indicate that:

  * (i)

When $\tilde{Q}=0$, the total mass diverges as $\tilde{\Omega}\to 1$. In turn,
for small $\tilde{\Omega}$, a rapid increase of $\tilde{M}_{\text{tot}}$ also
occurs, and one may expect that in the limit $\tilde{\Omega}\to 0$ the total
mass will tend to infinity as well. But in this limit a calculation of the
mass using the integral (14) is a difficult technical problem, and we cannot
verify this assumption by direct calculation.

  * (ii)

When $\tilde{Q}\neq 0$, the total mass in the limit $\tilde{\Omega}\to 1$ is
already finite. In turn, there is some nonzero value $\tilde{\Omega}<1$ for
which one can still perform numerical calculations. In doing so, one can
observe that the mass demonstrates a rapid increase
($|\partial\tilde{M}_{\text{tot}}/\partial\tilde{\Omega}|\gg 1$); this can be
regarded as an indication that there is some critical value
$\tilde{\Omega}_{\text{crit}}$ for which the mass will eventually tend to
infinity.

  * (iii)

A distinctive feature of the configurations from (i) and (ii) is the presence
of a minimum of the mass for all values of $\tilde{Q}$ lying in the interval
$-0.27\lesssim\tilde{Q}\leq 0$. This minimum corresponds to the presence in
the system of a mass gap where
$\partial\tilde{M}_{\text{tot}}/\partial\tilde{\Omega}=0$. In turn, for
$\tilde{Q}\lesssim-0.27$, such a mass gap is already absent: the total mass
demonstrates a gradual decrease as $\tilde{\Omega}$ increases, and eventually
$\tilde{M}_{\text{tot}}$ reaches some minimum value as $\tilde{\Omega}\to 1$.

  * (iv)

The dependence of the total mass on the coupling constant $\tilde{Q}$ for
different values of $\tilde{\Omega}$ corresponding to the location of the mass
gap is shown in the inset of Fig. 1. It is seen from this inset that for the
system with $\tilde{Q}=0$ (and correspondingly without the magnetic field),
$\tilde{\Omega}\approx 0.936$ (cf. Ref. Ranada:1973hna ). On the other hand,
there exists a maximum possible value of the coupling constant
$|\tilde{Q}|\approx 0.27$ for which the frequency $\tilde{\Omega}\to 1$. For
larger (modulus) values of $\tilde{Q}$ the curve
$\tilde{M}_{\text{tot}}(\tilde{\Omega})$ has no minimum already, that is, the
derivative $\partial\tilde{M}_{\text{tot}}/\partial\tilde{\Omega}$ is nowhere
equal to zero, and correspondingly the mass gap is absent.

  * (v)

There is some critical value of the coupling constant
$\tilde{Q}_{\text{crit}}$ approaching that the interval
$\tilde{\Omega}_{\text{crit}}\leq\tilde{\Omega}\leq 1$ (where the solutions do
exist) becomes narrower, and eventually $\tilde{\Omega}_{\text{crit}}\to 1$
and the complete set of solutions with different $\tilde{\Omega}$ degenerates
to the only solution with $\tilde{\Omega}=1$. Numerical calculations show that
$\tilde{Q}_{\text{crit}}\approx-0.3119$, and in this limit the total mass
$\tilde{M}_{\text{tot}}\approx 97.13$.

  * (vi)

For all these spinning systems, a straightforward computation shows that the
total angular momentum $J_{\text{tot}}$ from Eq. (16) and the total Noether
charge $Q_{\text{tot}}$ from Eq. (18) are related by
$J_{\text{tot}}=\frac{1}{2}Q_{\text{tot}}$, although the angular momentum
density and the Noether charge density are not proportional.

  * (vii)

From the form of Eqs. (8)-(13), it is evident that the solutions under
consideration are invariant with respect to a change in the sign
$\tilde{Q}\to-\tilde{Q},\tilde{\phi}\to-\tilde{\phi},\tilde{\sigma}\to-\tilde{\sigma}$,
that is, they may describe systems with the coupling constant opposite in sign
and the same physical characteristics.

Note here that, in order to apply the results obtained above for a description
of one particle, it is necessary to normalize the solutions so that the
Noether charge $Q_{\text{tot}}$ from Eq. (18) would be equal to 1. In this
case the coupling constant $Q$ will correspond to an electric charge, i.e.,
$Q=q_{e}$. This one-particle condition can be fulfilled by a suitable choice
of the free system parameters $\lambda$ and $m$. However, in doing so, one
should bear in mind that in this case there will be its own particular set of
the parameters $\lambda$ and $m$ for every point in the
$\left[\tilde{M}_{\text{tot}}-\tilde{\Omega}\right]$-plane, i.e., different
points of the plane will correspond to different models.

We conclude this subsection with the expression for the effective radial
pressure $p_{r}\equiv-T_{r}^{r}$. Using the energy-momentum tensor (4) and the
Dirac equations (8)-(11), it can be shown that the radial pressure contains
the terms

$p_{r}\sim\frac{\lambda}{2}\left(X^{2}-Y^{2}-V^{2}+W^{2}\right)^{2}-Q\phi\left(X^{2}+Y^{2}+V^{2}+W^{2}\right)\ldots$

This implies the following physical meaning of the nonlinearity parameter: the
case of $\lambda>0$ corresponds to the attraction and the case of $\lambda<0$
to the repulsion. Correspondingly, for the configurations considered above,
the attraction of the spinor field related to the choice of positive values of
the nonlinearity parameter provides a counter-balance to the effective
repulsion due to the presence of the electric charge.

###  III.4 Asymptotic behavior

For completeness of analysis of the numerical solutions obtained above, let us
write down analytical expressions for asymptotic solutions. The Maxwell
equations (12) and (13) have the following asymptotic ($x\to\infty$) behavior
of the electric and magnetic fields:

$\tilde{\phi}\approx\frac{1}{4\pi}\frac{\tilde{q}_{e}}{x},\quad\tilde{\sigma}\approx-\frac{1}{4\pi}\frac{\tilde{\mu}_{m}}{x}\sin^{2}\theta.$
(22)

Using these expressions, the numerical values of the electric charge
$\tilde{q}_{e}$ and of the magnetic moment $\tilde{\mu}_{m}$ can be found in
the form

$\tilde{q}_{e}=-4\pi\lim_{x\to\infty}x^{2}\frac{\partial\tilde{\phi}}{\partial
x}=-4\pi\lim_{\bar{x}\to
1}\bar{x}^{2}\frac{\partial\tilde{\phi}}{\partial\bar{x}},\quad\tilde{\mu}_{m}=4\pi\lim_{x\to\infty}\frac{x^{2}}{\sin^{2}\theta}\frac{\partial\tilde{\sigma}}{\partial
x}=4\pi\lim_{\bar{x}\to
1}\frac{\bar{x}^{2}}{\sin^{2}\theta}\frac{\partial\tilde{\sigma}}{\partial\bar{x}}.$
(23)

Note that the value of $\tilde{\mu}_{m}$ calculated using the above formula
coincides with that of obtained using Eq. (19).

In turn, the asymptotic behavior of the spinor fields follows from the Dirac
equations (8)-(11),

$\tilde{X}\approx-2\cos\frac{\theta}{2}\,g(x),\quad\tilde{Y}\approx
2\sin\frac{\theta}{2}\,f(x),\quad\tilde{V}\approx
2\cos\frac{\theta}{2}\,f(x),\quad\tilde{W}\approx-2\sin\frac{\theta}{2}\,g(x).$

The form of the functions $f(x)$ and $g(x)$ appearing here depends on the
value of $\tilde{\Omega}$. Namely, for $0<\tilde{\Omega}<1$, we have

$f(x)\approx
f_{\infty}\frac{e^{-\sqrt{1-\tilde{\Omega}^{2}}\,x}}{x}x^{-\frac{1}{\sqrt{1-\tilde{\Omega}^{2}}}\frac{\tilde{Q}\tilde{q}_{e}}{4\pi}},\quad
g(x)\approx
f_{\infty}\sqrt{\frac{1+\tilde{\Omega}}{1-\tilde{\Omega}}}\,\frac{e^{-\sqrt{1-\tilde{\Omega}^{2}}\,x}}{x}x^{-\frac{1}{\sqrt{1-\tilde{\Omega}^{2}}}\frac{\tilde{Q}\tilde{q}_{e}}{4\pi}},$
(24)

where $f_{\infty}$ is an integration constant. In the case of
$\tilde{\Omega}=1$, we have

$f(x)\approx
f_{\infty}\frac{e^{-\sqrt{\frac{2}{\pi}\tilde{Q}\tilde{q}_{e}\,x}}}{x^{5/4}},\quad
g(x)\approx
f_{\infty}\sqrt{\frac{8\pi}{\tilde{Q}\tilde{q}_{e}}}\,\frac{e^{-\sqrt{\frac{2}{\pi}\tilde{Q}\tilde{q}_{e}\,x}}}{x^{3/4}}.$

Notice here that regular solutions with $\tilde{\Omega}=1$ are only possible
in the presence of the charge.

Figure 2:  The dependence of the physical quantities on the spinor frequency
$\Omega$ for a fixed $\tilde{Q}=-0.04452$. Left panel: the graphs for the
total mass $\tilde{M}_{\text{tot}}$ from Eq. (14), for the angular momentum
$\tilde{J}_{\text{tot}}$ from Eq. (16), and for the charge $\tilde{q}_{e}$
from Eq. (23). Middle panel: the same quantities, but in the dimensional form
and with $Q_{\text{tot}}=1$ (normalized values). Right panel: the gyromagnetic
ratio (20). The vertical dashed lines correspond to the minimum of the mass
(the mass gap) located at the point $\tilde{\Omega}\approx 0.937$.

###  III.5 Particular example: an “electron”

Figure 3:  The dimensionless spinor functions $P_{+}$ and $P_{-}$ from (33)
and (34), energy density (15), angular momentum density (17), charge density
$\tilde{j}^{t}=U_{1}$, physical component of the current density
$\tilde{j}^{\varphi}=-2\,\tilde{Q}\,U_{3}$, electric,
$\tilde{\vec{E}}\equiv\sqrt{\lambda}/m\vec{E}$, and magnetic,
$\tilde{\vec{H}}\equiv\sqrt{\lambda}/m\vec{H}$, field strengths for the system
with $\tilde{Q}=-0.04452$ located at the mass gap (cf. Fig. 2). The plots are
made in a meridional plane $\varphi=\text{const.}$ spanned by the coordinates
$\rho=x\sin\theta$ and $z=x\cos\theta$. Since the system is
$\mathbb{Z}_{2}$-symmetric with respect to the equatorial plane $z=0$, the
electromagnetic field strength distributions are shown only for $z>0$.

The freedom in the choice of values of the parameters $\lambda$ and $m$
enables us to model various objects. In doing so, a choice of a specific value
of $\tilde{\Omega}$ can be made on the basis of a physically reasonable
assumption that an energetically stable system must possess a minimum energy
(or, equivalently, a minimum mass $\tilde{M}_{\text{tot}}$). Consistent with
this, consider, for example, the case where the coupling constant $\tilde{Q}$
is understood to be so chosen that at a minimum of the curve
$\tilde{M}_{\text{tot}}(\tilde{\Omega})$ the electric charge of the system
$q_{e}$ would be equal to the charge of an electron. The corresponding
dependencies $\tilde{M}_{\text{tot}}(\tilde{\Omega})$ and
$\tilde{q}_{e}(\tilde{\Omega})$ are shown in the left panel of Fig. 2. It is
seen that the minimum of the mass curve (the maximum of the charge curve) is
located at $\tilde{\Omega}\approx 0.937$, and the total mass and Noether
charge are

$M_{\text{tot}}=\frac{47.512}{m\lambda},\quad
Q_{\text{tot}}=\frac{46.255}{m^{2}\lambda}.$

This yields the dimensional mass $M_{\text{tot}}=1.027\,m\,Q_{\text{tot}}$
(see the middle panel of Fig. 2). For a normalized solution,
$Q_{\text{tot}}=1$; correspondingly, there is a mass renormalization of
$2.7\%$. Then, in order to make the total mass of the system $M_{\text{tot}}$
equal to the electron mass $m_{e}$, it is necessary to take $m=m_{e}/1.027$.
This in turn leads to the corresponding renormalization of the magnetic moment
and change in the value of the gyromagnetic ratio $g$, whose graph is shown in
the right panel of Fig. 2. For the case under consideration, $g\approx 2.083$.
Thus we have a configuration with the mass $M_{\text{tot}}=0.511\,\text{MeV}$
and charge $q_{e}=-0.3028$ equal to the mass and charge of an electron, but
with $g>2$.

Also, for a normalized solution, i.e., when $Q_{\text{tot}}=1$, the quantum-
mechanical angular momentum, which is determined using the operator of the
total angular momentum,

$\hat{M}_{z}=\hat{L}_{z}+\hat{S}_{z},$ (25)

(here $\hat{L}_{z}=-\imath\partial_{\varphi}$ is the operator which projects
the orbital angular momentum on the $z$-axis and $\hat{S}_{z}$ is the operator
which projects the spin on the $z$-axis), defines the value of the total
angular momentum as an eigenvalue $M_{z}$,

$\hat{M}_{z}\psi=M_{z}\psi.$

For the spinor (5), this eigenvalue is $M_{z}=1/2$, and it coincides with that
calculated using the integral formula (16) (cf. the value of $J_{\text{tot}}$
from the middle panel of Fig. 2). It is worth noting that for the solutions
that are not normalized to unity this coincidence no longer occurs.

The characteristic size of such a charged configuration supported by the
spinor field can be estimated from the asymptotic behavior of the field (24)
as

$r_{\text{ch}}\sim\frac{1}{\sqrt{1-\tilde{\Omega}^{2}}\,m_{e}}.$

For $\tilde{\Omega}\approx 0.937$, this yields $r_{\text{ch}}\sim
10^{-10}\,\text{cm}$, a value that is comparable in order of magnitude to the
electron Compton wavelength.

Note that the spinor functions $\tilde{X},\tilde{Y},\tilde{V},$ and
$\tilde{W}$ appearing in the Dirac equations (8)-(11) are neither even nor odd
functions with respect to the equatorial plane $\theta=\pi/2$. Nevertheless,
the system possesses a $\mathbb{Z}_{2}$ symmetry with respect to this plane;
this can be shown by considering the corresponding combinations of the spinor
functions (see Appendix A). To demonstrate this fact in a pictorial way, the
upper row of Fig. 3 shows the graphs of the functions $P_{+}$ and $P_{-}$ from
Eqs. (33) and (34), respectively. Also, this figure shows the corresponding
$\mathbb{Z}_{2}$-symmetric distributions of the components of the energy-
momentum tensor (4) and current density, as well as the electric and magnetic
field strengths defined by the expressions (7). The structure of the magnetic
field strength corresponds to an axially symmetric dipole field sourced by the
current associated with the spinor field given on the right-hand side of Eq.
(3). The radial distribution of the current and the magnitude of the magnetic
field are determined by the value of the coupling constant $Q$. In turn, the
structure of the electric field strength corresponds to a negative charge with
force lines directed toward the center of the configuration.

In conclusion, note that by choosing $m=m_{\mu}/1.027$, where $m_{\mu}$ is the
muon mass, we get characteristics typical for a muon/antimuon.

##  IV Conclusions

The main purpose of the present paper is to study self-consistently the
influence that an electromagnetic field has on a system supported by a
nonlinear spinor field. To this end, we generalized the configurations
considered in Refs. Soler:1970xp ; Ranada:1973hna  by including
nonperturbatively electric and magnetic (dipole) Maxwell fields to take
account of their backreaction on the physical characteristics of the system.
In such a generalized case, the presence in the system of the dipole magnetic
field requires a consideration of an axisymmetric problem.

In the absence of electromagnetic fields, an important distinctive feature of
the systems supported by nonlinear spinor fields is the presence of a mass
gap. For such systems, all solutions are parameterized by one parameter – the
spinor frequency $\tilde{\Omega}$, and regular stationary solutions describing
configurations with finite values of various physical parameters (for
instance, of a total mass) do exist only for the values of $\tilde{\Omega}$
lying in the range $0<\tilde{\Omega}<1$, whereas for $\tilde{\Omega}\to 0$ and
$\tilde{\Omega}\to 1$ the total mass diverges. The inclusion of the Maxwell
fields results in the appearance of one more free parameter – the coupling
constant $\tilde{Q}$. For such a two-parametric system, we have considered all
permissible values of the parameters $\tilde{\Omega}$ and $\tilde{Q}$ for
which regular spinning solutions do exist. The studies of the present work
indicate that there are the following qualitative changes in the
characteristics of the configurations as compared with the electrically
neutral ($\tilde{Q}=0$) case:

  * •

Apart from the mass and angular momentum gaps, the system also contains the
charge and magnetic moment gaps located at the same values of $\tilde{\Omega}$
as the mass gap (see Fig. 2 and cf. Ref. Dzhunushaliev:2022pkz ).

  * •

For a nonzero coupling constant $\tilde{Q}$, there is some critical value of
the spinor frequency $\tilde{\Omega}_{\text{crit}}>0$ that restricts the range
of permissible values of $\tilde{\Omega}$ from the left. As in the case
without Maxwell fields, at this boundary, the total masses of the system
diverge for all permissible values of $\tilde{Q}$. As $\tilde{Q}$ increases
(modulus), the value of $\tilde{\Omega}_{\text{crit}}$ increases as well, and
there is a finite critical value $|\tilde{Q}_{\text{crit}}|\approx 0.3119$ for
which $\tilde{\Omega}_{\text{crit}}\to 1$, i.e., the set of solutions with
different $\tilde{\Omega}$ degenerates to the only solution with
$\tilde{\Omega}=1$.

  * •

For $0<|\tilde{Q}|<|\tilde{Q}_{\text{crit}}|$ and as $\tilde{\Omega}\to 1$,
the total mass of the system, in contrast to the case without Maxwell fields,
remains always finite. That is, regular solutions exist in the frequency range
of $\tilde{\Omega}_{\text{crit}}<\tilde{\Omega}\leq 1$, and the magnitude of
$\tilde{\Omega}_{\text{crit}}$ is completely determined only by the value of
the coupling constant $\tilde{Q}$.

  * •

There is a maximum possible value of $|\tilde{Q}|\approx 0.27$ above which the
aforementioned gaps are already absent in the system.

As a possible application of the above results, we have considered the case
where the coupling constant $\tilde{Q}$ is to be so chosen that the electric
charge of the system located at the mass/charge gap would be equal to the
charge of an electron (or of a positron when $-\tilde{Q}$ is changed into
$\tilde{Q}$) (see Sec. III.5). Then, by choosing an appropriate value of the
bare mass $m$ in the Dirac equation (2), one can also get the total mass of
the system, equal to the mass of an electron. In turn, the total angular
momentum $J_{\text{tot}}$ and the total Noether charge $Q_{\text{tot}}$ are
related by $J_{\text{tot}}=\frac{1}{2}Q_{\text{tot}}$ (as it also takes place
for all other spinning systems with permissible values of $\tilde{Q}$ and
$\tilde{\Omega}$ considered in the present paper), and when $Q_{\text{tot}}=1$
(i.e., for a normalized solution), $J_{\text{tot}}$ coincides with the
eigenvalue $M_{z}=1/2$ of the operator of the total angular momentum (25).
Also, the gyromagnetic ratio for such configuration is $g\approx 2.083$ (cf.
the electron for which $g\approx 2$) and the characteristic size is
$r_{\text{ch}}\sim 10^{-10}\,\text{cm}$.

## Acknowledgments

This research was funded by the Committee of Science of the Ministry of
Science and Higher Education of the Republic of Kazakhstan (Grant No.
BR21881941).

##  Appendix A Symmetry of the field equations

When $\theta$ is replaced by $\pi-\theta$, the equations (8)-(13) take the
form

$\displaystyle\tilde{X}_{,x}+\frac{\tilde{X}}{x}+\frac{\tilde{W}_{,\theta}}{x}-\frac{\tan\frac{\theta}{2}}{2x}\tilde{W}+\tilde{Q}\left(-\frac{\csc\theta}{x}\tilde{W}\tilde{\sigma}+\tilde{V}\tilde{\phi}\right)-\left(1+\tilde{\Omega}\right)\tilde{V}+U_{2}\tilde{V}=$
$\displaystyle 0,$ (26)
$\displaystyle\tilde{Y}_{,x}+\frac{\tilde{Y}}{x}+\frac{\tilde{V}_{,\theta}}{x}+\frac{\cot\frac{\theta}{2}}{2x}\tilde{V}+\tilde{Q}\left(\frac{\csc\theta}{x}\tilde{V}\tilde{\sigma}-\tilde{W}\tilde{\phi}\right)-\left(1-\tilde{\Omega}\right)\tilde{W}+U_{2}\tilde{W}=$
$\displaystyle 0,$ (27)
$\displaystyle\tilde{V}_{,x}+\frac{\tilde{V}}{x}-\frac{\tilde{Y}_{,\theta}}{x}+\frac{\tan\frac{\theta}{2}}{2x}\tilde{Y}+\tilde{Q}\left(\frac{\csc\theta}{x}\tilde{Y}\tilde{\sigma}-\tilde{X}\tilde{\phi}\right)-\left(1-\tilde{\Omega}\right)\tilde{X}+U_{2}\tilde{X}=$
$\displaystyle 0,$ (28)
$\displaystyle\tilde{W}_{,x}+\frac{\tilde{W}}{x}-\frac{\tilde{X}_{,\theta}}{x}-\frac{\cot\frac{\theta}{2}}{2x}\tilde{X}+\tilde{Q}\left(-\frac{\csc\theta}{x}\tilde{X}\tilde{\sigma}+\tilde{Y}\tilde{\phi}\right)-\left(1+\tilde{\Omega}\right)\tilde{Y}+U_{2}\tilde{Y}=$
$\displaystyle 0,$ (29)
$\displaystyle\tilde{\phi}_{,xx}+\frac{2}{x}\tilde{\phi}_{,x}+\frac{1}{x^{2}}\tilde{\phi}_{,\theta\theta}+\frac{\cot\theta}{x^{2}}\tilde{\phi}_{,\theta}+\tilde{Q}\,U_{1}=$
$\displaystyle 0,$ (30)
$\displaystyle\tilde{\sigma}_{,xx}+\frac{1}{x^{2}}\tilde{\sigma}_{,\theta\theta}-\frac{\cot\theta}{x^{2}}\tilde{\sigma}_{,\theta}+2\,\tilde{Q}\,x\sin\theta\,U_{3}=$
$\displaystyle 0.$ (31)

All the functions appearing here depend on $\pi-\theta$:
$\tilde{X}(x,\pi-\theta),\tilde{Y}(x,\pi-\theta),\tilde{V}(x,\pi-\theta),\tilde{W}(x,\pi-\theta),\phi(x,\pi-\theta),\sigma(x,\pi-\theta)$,
while the combinations $U_{1},U_{2},$ and $U_{3}$ remain unchanged. A
comparison of the equations (26) and (11) enables one to conclude that
$\tilde{W}(x,\theta)=\tilde{X}(x,\pi-\theta)$. Similarly, one can show that
such a symmetry is valid for all functions:

$\tilde{X}(x,\theta)=\tilde{W}(x,\pi-\theta),\quad\tilde{Y}(x,\theta)=\tilde{V}(x,\pi-\theta),\quad\tilde{V}(x,\theta)=\tilde{Y}(x,\pi-\theta),\quad\tilde{W}(x,\theta)=\tilde{X}(x,\pi-\theta).$
(32)

This enables us to rewrite four local Dirac equations (8)-(11) in the form of
two nonlocal equations

$\begin{split}\tilde{X}(x,\theta)_{,x}+\frac{\tilde{X}(x,\theta)}{x}-\frac{\tilde{X}(x,\pi-\theta)_{,\theta}}{x}-\frac{\cot\frac{\theta}{2}}{2x}\tilde{X}(x,\pi-\theta)+\tilde{Q}\left[-\frac{\csc\theta}{x}\tilde{X}(x,\pi-\theta)\tilde{\sigma}+\tilde{Y}(x,\pi-\theta)\tilde{\phi}\right]&\\\
-\left(1+\tilde{\Omega}\right)\tilde{Y}(x,\pi-\theta)+U_{2}\tilde{Y}(x,\pi-\theta)=&0,\\\
\tilde{Y}(x,\theta)_{,x}+\frac{\tilde{Y}(x,\theta)}{x}-\frac{\tilde{Y}(x,\pi-\theta)_{,\theta}}{x}+\frac{\tan\frac{\theta}{2}}{2x}\tilde{Y}(x,\pi-\theta)+\tilde{Q}\left[\frac{\csc\theta}{x}\tilde{Y}(x,\pi-\theta)\tilde{\sigma}-\tilde{X}(x,\pi-\theta)\tilde{\phi}\right]&\\\
-\left(1-\tilde{\Omega}\right)\tilde{X}(x,\pi-\theta)+U_{2}\tilde{X}(x,\pi-\theta)=&0.\end{split}$

In turn, taking into account the properties of the functions
$\tilde{X},\tilde{Y},\tilde{V}$, and $\tilde{W}$ given in Eq. (32), one can
see that there are the following symmetric and antisymmetric functions:

$\displaystyle P_{+}(x,\theta)=$
$\displaystyle\tilde{X}(x,\theta)+\tilde{W}(x,\theta)=\tilde{X}(x,\theta)+\tilde{X}(x,\pi-\theta)=\tilde{W}(x,\theta)+\tilde{W}(x,\pi-\theta)=P_{+}(x,\pi-\theta),$
(33) $\displaystyle P_{-}(x,\theta)=$
$\displaystyle\tilde{X}(x,\theta)-\tilde{W}(x,\theta)=\tilde{X}(x,\theta)-\tilde{X}(x,\pi-\theta)=\tilde{W}(x,\pi-\theta)-\tilde{W}(x,\theta)=-P_{-}(x,\pi-\theta),$
(34) $\displaystyle Q_{+}(x,\theta)=$
$\displaystyle\tilde{Y}(x,\theta)+\tilde{V}(x,\theta)=\tilde{Y}(x,\theta)+\tilde{Y}(x,\pi-\theta)=\tilde{V}(x,\theta)+\tilde{V}(x,\pi-\theta)=Q_{+}(x,\pi-\theta),$
(35) $\displaystyle Q_{-}(x,\theta)=$
$\displaystyle\tilde{Y}(x,\theta)-\tilde{V}(x,\theta)=\tilde{Y}(x,\theta)-\tilde{Y}(x,\pi-\theta)=\tilde{V}(x,\pi-\theta)-\tilde{V}(x,\theta)=-Q_{-}(x,\pi-\theta).$
(36)

## References

  * (1) D. D. Ivanenko, Notes to the theory of interaction via particles, Sov. Phys. JETP 13, 141(1938).
  * (2) R. Finkelstein, R. LeLevier, and M. Ruderman, Nonlinear Spinor Fields, Phys. Rev. 83, 326 (1951).
  * (3) R. Finkelstein, C. Fronsdal, and P. Kaus, Nonlinear Spinor Field, Phys. Rev. 103, 1571 (1956).
  * (4) W. Heisenberg, Introduction to the Unified Field Theory of Elementary Particles (Max-Planck-Institut für Physik und Astrophysik, Interscience Publisher, London, 1966).
  * (5) Y. Nambu and G. Jona-Lasinio, Dynamical model of elementary particles based on an analogy with superconductivity. I, Phys. Rev. 122, 345 (1961).
  * (6) M.K. Volkov and A.E. Radzhabov, The Nambu-Jona-Lasinio model and its development. Phys. Usp. 49, 551 (2006).
  * (7) M. Soler, Classical, stable, nonlinear spinor field with positive rest energy, Phys. Rev. D 1, 2766 (1970).
  * (8) A. F. Ranada and M. Soler, Perturbation theory for an exactly soluble spinor model in interaction with its electromagnetic field, Phys. Rev. D 8, 3430-3433 (1973).
  * (9) A. F. Ranada, M. F. Ranada, M. Soler, and L. Vazquez, Classical Electrodynamics of a Nonlinear Dirac Field with Anomalous Magnetic Moment, Phys. Rev. D 10, 517 (1974).
  * (10) D. J. Gross and A. Neveu, Dynamical Symmetry Breaking in Asymptotically Free Field Theories, Phys. Rev. D 10, 3235 (1974).
  * (11) A. Alvarez and B. Carreras, Interaction Dynamics for the Solitary Waves of a Nonlinear Dirac Model, Phys. Lett. A 86, 327-332 (1981).
  * (12) D. C. Ionescu, J. Reinhardt, B. Muller, W. Greiner, and G. Soff, Nonlinear extensions of the Dirac equation and their implications in QED, Phys. Rev. A 38, 616 (1988).
  * (13) M. J. Esteban and E. Sere, Stationary states of the nonlinear Dirac equation: a variational approach, Comm. Math. Phys. 171, 323 (1995).
  * (14) M. J. Esteban and E. Sere, An overview on linear and nonlinear Dirac equations, Discrete Contin. Dyn. Syst. 8, 381 (2002).
  * (15) A. Zecca, Dirac equation in space-time with torsion, Int. J. Theor. Phys. 41, 421-428 (2002).
  * (16) K. A. Bronnikov, E. N. Chudaeva, and G. N. Shikin, Self-gravitating string-like configurations of nonlinear spinor fields, Gen. Relativ. Gravit. 36, 1537 (2004).
  * (17) K. A. Bronnikov and J. P. S. Lemos, Cylindrical wormholes, Phys. Rev. D 79, 104019 (2009).
  * (18) V. Adanhounme, A. Adomou, F. P. Codo, and M. N. Hounkonnou, Nonlinear spinor field equations in gravitational theory: Spherical symmetric soliton-like solutions, J. Mod. Phys. 3, 935 (2012).
  * (19) B. Saha, Nonlinear spinor field in isotropic space-time and dark energy models, Eur. Phys. J. Plus 131, 242 (2016).
  * (20) V. Dzhunushaliev and V. Folomeev, Dirac stars supported by nonlinear spinor fields, Phys. Rev. D 99, 084030 (2019).
  * (21) V. Dzhunushaliev and V. Folomeev, Dirac star in the presence of Maxwell and Proca fields, Phys. Rev. D 99, 104066 (2019).
  * (22) V. Dzhunushaliev and V. Folomeev, Dirac star with SU(2) Yang-Mills and Proca fields, Phys. Rev. D 101, 024023 (2020).
  * (23) K. A. Bronnikov, Y. P. Rybakov, and B. Saha, Spinor fields in spherical symmetry: Einstein-Dirac and other space-times, Eur. Phys. J. Plus 135, 124 (2020).
  * (24) V. Dzhunushaliev, V. Folomeev, and A. Makhmudov, Non-Abelian Proca-Dirac-Higgs theory: Particlelike solutions and their energy spectrum, Phys. Rev. D 99, 076009 (2019).
  * (25) V. Dzhunushaliev, V. Folomeev, and A. Serikbolova, Monopole solutions in SU(2) Yang-Mills-plus-massive-nonlinear-spinor-field theory, Phys. Lett. B 806, 135480 (2020).
  * (26) V. Dzhunushaliev, N. Burtebayev, V. N. Folomeev, J. Kunz, A. Serikbolova, and A. Tlemisov, Mass gap for a monopole interacting with a nonlinear spinor field, Phys. Rev. D 104, 056010 (2021).
  * (27) V. Dzhunushaliev, V. Folomeev, and D. Berkimbayev, Charge gap in SU(3) Yang–Mills-plus-nonlinear-spinor-field theory, Eur. Phys. J. C 83, 546 (2023).
  * (28) C. Armendariz-Picon and P. B. Greene, Spinors, inflation, and nonsingular cyclic cosmologies, Gen. Relativ. Gravit. 35, 1637 (2003).
  * (29) I. Lawrie, A Unified Grand Tour of Theoretical Physics (Institute of Physics Publishing, Bristol, 2002).
  * (30) C. Herdeiro, I. Perapechka, E. Radu, and Y. Shnir, Asymptotically flat spinning scalar, Dirac and Proca stars, Phys. Lett. B 797, 134845 (2019).
  * (31) D. I. Blokhintsev, Quantum mechanics (D. Reidel Publishing Company, Dordrecht, Holland, 1964).
  * (32) W. Schönauer and R. Weiß, Efficient vectorizable PDE solvers, J. Comput. Appl. Math. 27, 279 (1989).


# Learning Metrics from Teachers: Compact Networks for Image Embedding

Lu Yu1,2, Vacit Oguz Yazici2,3, Xialei Liu2, Joost van de Weijer2, Yongmei
Cheng1, Arnau Ramisa3
1 School of Automation, Northwestern Polytechnical University, Xi’an, China
2 Computer Vision Center, Universitat Autonoma de Barcelona, Barcelona, Spain
3 Wide-Eyes Technologies, Barcelona, Spain
<EMAIL_ADDRESS><EMAIL_ADDRESS>aramisa@wide-
eyes.it

###### Abstract

Metric learning networks are used to compute image embeddings, which are
widely used in many applications such as image retrieval and face recognition.
In this paper, we propose to use network distillation to efficiently compute
image embeddings with small networks. Network distillation has been
successfully applied to improve image classification, but has hardly been
explored for metric learning. To do so, we propose two new loss functions that
model the communication of a deep teacher network to a small student network.
We evaluate our system in several datasets, including CUB-200-2011, Cars-196,
Stanford Online Products and show that embeddings computed using small student
networks perform significantly better than those computed using standard
networks of similar size. Results on a very compact network (MobileNet-0.25),
which can be used on mobile devices, show that the proposed method can greatly
improve Recall@1 results from 27.5% to 44.6%. Furthermore, we investigate
various aspects of distillation for embeddings, including hint and attention
layers, semi-supervised learning and cross quality distillation. 111Code is
available at https://github.com/yulu0724/EmbeddingDistillation.

##  1 Introduction

Deep neural networks obtain impressive performance for many computer vision
applications, some of which have subsequently been turned into products for
the general population. However, the applicability of these techniques is
often limited by their high computational cost. To reduce network traffic and
server costs, as well as for scalability, it is desirable to place as much of
the computation as possible on the end-user side of the application. However,
this is often a mobile device with limited computing power and battery life,
and thus cannot compute large networks in real-time. This creates a strong
demand for methods that transfer the knowledge from large networks to smaller
ones, but without a significant drop in performance.

One important class of deep networks learns feature embeddings. To be
successful, feature embeddings must preserve semantic similarity, i.e. items
deemed similar by users must be close in the embedding space, despite
significant visual differences such as point of view, illumination, or image
quality. To bridge this gap between the semantic and visual domains, pairs or
triplets of related and unrelated items are used to teach the network how to
organize the output embedding space [5, 35, 12]. Embeddings were found
efficient on the tasks of out-of-distribution detection [19] and transfer
learning [26]. Furthermore, embedding networks are essential for computer
vision, as evidenced by the large variety of tasks in which they are used,
including feature-based object retrieval [9], face recognition [25], feature
matching [6], domain adaptation [27], weakly supervised learning [36], ranking
[35], or zero-shot learning [32].

Large networks are known to provide excellent feature embeddings [23, 25], but
are often impractical for real-life applications, as mentioned before. To
obtain efficient neural networks, research has focused on two main research
directions: network compression and network distillation. Network
compremossion reduces the number of parameters in the network [17, 10], while
network distillation uses a teacher-student setup in which a, typically large,
teacher network is used to guide a small student network [2, 11]. This is done
by using a loss function that minimizes cross-entropy between the outputs of
the student and teacher network for classification. The main idea underlying
network distillation is that uncertainty in the estimates of the teacher
network, e.g. about whether an image contains a cat and dog, provides relevant
information for the student. There are several differences between network
compression and knowledge distillation. First of all, the underlying
assumption of network compression is that the knowledge of the network is in
the weights, whereas knowledge distillation assumes that the knowledge of the
network is in the activations which arise from particular data. A second
important difference is that compression algorithms typically end up with a
similar network architecture than the initial large network but with less
parameters (i.e. same number of layers, and layer types). In contrast, network
distillation puts no restrictions on the student network design. Therefore, we
focus on network distillation techniques for the efficient computation of
feature embeddings with small networks.

In this paper, we use network distillation to obtain efficient networks to
learn feature embeddings. We propose two different ways of teaching metrics to
students: one based on an _absolute teacher_ , where the student aims to
produce the same embeddings as the teacher, and one based on a _relative
teacher_ , where the teacher communicates only the distances between pairs of
data points to the student. Using the CUB-200-2011 (birds), Cars-196 and
Stanford Online Products datasets, we show that network distillation can
significantly improve retrieval performance compared to directly training the
student network on the data. We also found that the relative teacher
consistently outperforms the absolute teacher. We evaluate various aspects of
knowledge distillation, namely the usage of hint and attention layers, and the
possibility to train from unlabelled data. We also show that a teacher with
access to high-quality images can be used to improve embeddings learned with a
student network with access to low-quality images.

##  2 Related work

There is a large number of works on metric learning, see for example the
survey [16]. Here we focus on metric learning using deep networks.

Metric Learning Initially metric learning with deep networks was based on
Siamese architecture with contrastive loss [5]. Later Triplet networks were
proposed which allow more local modifications of the embedding space, and do
not require that all the observations of the same class collapse to the same
point [12, 35]. The progress of Siamese and Triplet networks has been hampered
because of the pair (or triplet) sampling problem which arises from the huge
potential space from which pairs can be sampled. For instance, in a dataset
with $N$ samples, $N^{2}$ pairs could be possibly sampled, and it is therefore
unfeasible to consider all of them. Therefore, hard negative mining was
proposed to focus only on the pairs which induced the highest loss [29], with
the expectation that the network would learn the most from them.
Unfortunately, this led to severe overfitting in many cases, and semi-hard
negative mining was introduced as a solution [25]. However, both hard and
semi-hard negative mining have a high computational cost, which led several
authors to limit the hard negative mining process to the current mini-batch
[18, 31, 32, 36].

Network Distillation Bucila et al. [2] compress a large network into a small
one. Their method aims to approximate a large teacher network with a single
fast and compact student network. This was further improved by Hinton et al.
[11] by moving the teacher signal from the logits (just before the softmax) to
the probabilities (after the softmax), and introducing temperature scaling to
increase the influence of small probabilities. With these improvements, they
achieved some surprising results on MNIST, and also showed that the acoustic
model of a heavily used commercial system could be significantly improved by
distilling the joint knowledge of an ensemble of models into a single one.
FitNet [24] introduced hint layers with additional losses on intermediate
layers of the network to communicate knowledge of the teacher to the student.
They show that this helps to train deep and thin networks, which cannot be
trained from scratch without teacher supervision. In addition, these students
can outperform the teacher network while using less memory. Zhang et al. [38]
show that a group of students, without a teacher, which are jointly trained
with similar losses as between teacher and student, can outperform standard
ensemble learning. Network distillation can also be used to compress multiple
teachers into a single student network [8]. Most literature on network
distillation focuses on image classification, but recently several works have
investigated applying the theory to object detection [3] and pedestrian
detection [28].

Only two works have previously addressed knowledge distillation for
embeddings. Chen et al. [4] bring the ’learning to rank’ technique into deep
metric learning for knowledge transfer. It is formalized as a rank matching
problem between teacher and student networks. Their list-wise loss can easily
overflow due to the product operation when computing the probability of the
permutation, which severely limits the batch size that can be used. PKT [21]
applies a different approach where they model the interactions between the
data samples in the feature space as a probability distribution. In our
experiments we show that our proposed relative teacher outperforms the
DarkRank and PKT significantly.

##  3 Preliminaries

In this paper, we will apply network distillation to metric learning networks.
This section will briefly introduce both.

Figure 1: Graphical illustration of the two knowledge distillation losses we
propose for metric learning. $L_{KD}^{abs}$ aims to minimize the distance
between the student and teacher embedding of the same image. $L_{KD}^{rel}$
compares the distance in the embedding of the teacher between two images, with
the distance of the same two images in the student embedding. It aims to make
the two distances as similar as possible.

###  3.1 Metric Learning

A fundamental step in most computer vision applications is transforming the
initial representation of the images (i.e. pixels) into another one with more
desirable properties. This process is often denoted as feature extraction, and
projects the images to a high-level representation that captures the semantic
characteristics relevant to the task. How images are organized in this high-
level representation is crucial to the success of many applications. For
example, image retrieval, k-NN or Nearest Class Mean classifiers are directly
based on the distances between these high-level image representations. Metric
learning addresses this problem and intends to map the input feature
representations to an embedding space where the L2 distance correlates with
the desired notion of similarity. In this work we will focus on deep, or end-
to-end, metric learning, where the whole feature extraction network is trained
jointly to generate the best possible representation.

Siamese networks map data to an output space where distance represents the
semantic dissimilarity between the images [1, 5]. Triplet networks were
proposed by Hoffer et al. [12] based on the work of Wang et al. [35]. In
contrast to Siamese networks, they use triplets formed by an anchor ($x_{a}$),
a positive instance ($x_{p}$) and a negative instance ($x_{n}$), as input. The
anchor and the positive instances correspond to the same category, while the
negative instance is from a different one. The objective is to guarantee that
the negative instance is further away from the anchor than the positive (plus
a margin $m$). The Triplet loss is given by:

$L_{T}=\max(0,d_{+}-d_{-}+m),$ (1)

where $d_{+}$ and $d_{-}$ are the distances between the anchor and the
positive and negative instances respectively. The Triplet network imposes only
_local_ constraints on the output embedding, which can simplify convergence
when compared to the Siamese network, reportedly harder to train. In the
experiments we will show results of network distillation for embeddings
learned with triplet losses.

###  3.2 Network Distillation

Figure 2: Illustration of difference between absolute and relative teacher.
(left) Example of four data points in the embedding space of teacher. We
consider two samples from two classes (indicated by square and star). (middle
and right) show the absolute and relative loss for two student embeddings $S1$
and $S2$ (the teacher location of the points is given in dashed lines). The
(right) embedding is preferable since it is exactly equal to the teacher
(except for a translation). This is only appreciated by the relative teacher,
whereas the absolute teacher assigns equal loss to both.

Network distillation [11, 24] aims to transfer the knowledge of a large
teacher network $T$ to a small student network $S$. The objective for network
distillation for classification networks is defined as:

$L_{KD}=H(y_{true},P_{\tau=1}^{S})+\lambda H(P_{\tau}^{T},P_{\tau}^{S}),$ (2)

where $\lambda$ is used to balance the importance of two cross-entropy losses
$H$: the first one corresponds to the traditional loss between the predictions
of the student network and the ground-truth labels $y_{true}$, and the second
one between the _annealed_ probability outputs of the student and teacher
networks. This loss encourages the student to make similar predictions as the
teacher network. The information of the teacher $P_{\tau}^{T}$ could be more
valuable than the ground truth $y_{true}$ for the student network, because it
also contains information of which classes could possibly be confused with the
true label for a particular image. More precisely, $P_{\tau}^{T}$ and
$P_{\tau}^{S}$ are:

$P_{\tau}^{T}={\rm{softmax}}(\frac{a_{T}}{\tau}),\;P_{\tau}^{S}={\rm{softmax}}(\frac{a_{S}}{\tau}),$
(3)

where $a_{S}$ and $a_{T}$ are the (pre-softmax) activations of the student and
teacher networks respectively, and temperature $\tau$ is a relaxation which is
introduced to soften the signal arising from the output of the networks. It
was found that for complex classification tasks $\tau=1$ obtained good results
[3]. $P_{\tau=1}^{S}$ is equal to the output of the standard student network
without any temperature scaling.

##  4 Distillation for Metric Learning

Wide and deep networks with large amounts of parameters are known to obtain
excellent results [30], however they are very time consuming and memory
demanding. Network distillation is proven to be one of the solutions to handle
this problem in the classification field [11]. In this section we extend the
theory of knowledge distillation to networks that aims to project images into
an embedding space. In addition we will discuss the incorporation of hint and
attention transfer between student and teacher.

###  4.1 Knowledge distillation for embedding networks

Traditional network distillation has focused on networks which perform
classification, and are trained with a cross-entropy loss [11, 24]. During
training the output class distribution produced by the student is enforced to
be close to that of the teacher. This is shown to obtain much better results
than directly training the student on the available data; the main reason for
this performance difference is that confusions between classes of the teacher
reveal relevant information to the student, thereby providing a richer
training signal than ground truth labels would provide [11].

Here we extend the technique of knowledge distillation to networks that are
used to project input data into an embedding (from now on called _embedding
networks_). These embeddings are then typically used to perform distance
computation. For example, to provide a ranked list of similar data (ordered
according to the distance). For knowledge distillation it is important to
consider what is the knowledge that is contained in the embedding network. One
could consider the actual embedding (meaning the coordinates of the embedding)
to be the knowledge of the network. Another point of view would be to consider
the distances which are computed based on the embedding network to be the
actual knowledge, since this is actually the main purpose of the embedding
network. We will consider both these points of view and design two different
teachers: one teacher, called _absolute teacher_ which teaches the exact
coordinates to the student and one teacher, called _relative teacher_ which
only teaches the distance between data pairs to the student.

In the first approach, the absolute teacher, we directly minimize the distance
between the student ($F^{S}$) and teacher ($F^{T}$) embeddings. This is done
by minimizing:

$L_{KD}^{abs}=\left\|{F^{S}\left({x_{i}}\right)-F^{T}\left({x_{i}}\right)}\right\|,$
(4)

where $\left\|.\right\|$ refers to the Frobenius norm.

As a second approach, we consider the relative teacher, which enforces the
student network to learn any embedding as long as it results in similar
distances between the data points. This is done by minimizing the following
loss:

$L_{KD}^{rel}=\left|d^{S}-d^{T}\right|,$ (5)

where $d^{S}$ and $d^{T}$ are, respectively, the distances between the student
and teacher embeddings of images $x_{i}$ and $x_{j}$:

$\begin{array}[]{l}d^{S}=\left\|{F^{S}\left({x_{i}}\right)-F^{S}\left({x_{j}}\right)}\right\|,\\\
d^{T}=\left\|{F^{T}\left({x_{i}}\right)-F^{T}\left({x_{j}}\right)}\right\|,\end{array}$
(6)

The minimization loss in Eq. 5 is equal to the loss used in the classical
problem of multidimensional scaling (MDS) [7]. There the dissimilarities
between points is known and the goal is to find coordinates for the points in
some (low-dimensional) space where the dissimilarities between the points is
equal to their dissimilarity.

Figure 3: Schematics of teacher-student hint/attention transfer.

A graphical illustration which shows the relevant distances that are used by
the absolute and relative teacher is provided in Fig. 1. The teacher networks
are frozen during training of the student network. The absolute teacher
minimizes the distance between the student and teacher embedding for each
training sample. In case of the relative teacher, one should consider pairs of
data points, since during training the student network is optimized to obtain
similar distances between instances of data points.

As reported by several other authors [11, 38], we train the student network by
simultaneously considering the standard metric learning loss $L_{ML}$ (see Eq.
1) and the loss $L_{KD}^{T}$ imposed by the teacher, according to:

$L=L_{ML}+\lambda L_{KD}^{T},$ (7)

where $T\in\left\\{{abs,rel}\right\\}$ and $\lambda$ is a trade-off parameter
between the different losses, which is learned by cross validation.

In Fig. 2 an illustration of the two distillation losses is given for two
different student embeddings, indicated by $S1$ and $S2$. The embedding $S2$
is preferable because it is equal to the teacher embedding except for a
translation which does not influence the ranking of data points. The S1
embedding actually changes the relation between samples, and would not obtain
similar results as the teacher network. However, if we consider the absolute
loss for these two scenarios we see it assigns equal loss to both embeddings.
The relative loss does correctly assign a lower (zero) loss to the S2
embedding. By focusing on the relevant parameter (namely the distance), we
expected that relative teachers are able to better guide the student to a
similar embedding than the student network.

###  4.2 Learning from hints and attention

In this section we consider two techniques that have shown to improve results
for distillation of classification networks. The techniques we consider are:
the introduction of hint layers [24] and the usage of attention [37]. Both
were proposed to improve the learning of student networks. We are interested
to know if these techniques also generalize to knowledge distillation for
embedding networks.

Romero et al. [24] propose to improve knowledge distillation by introducing an
additional loss on intermediate representations learned by the teacher (called
hints). The loss which incorporates the hint layers is given by:

$L_{hint}=\left\|{F^{S}_{hint}\left({x_{i}}\right)-F^{T}_{hint}\left({x_{i}}\right)}\right\|,$
(8)

where $F^{T}_{hint}\in{R}^{w\times h\times d}$ where $w$, $h$ and $d$ are
dimensions of the activation map of the hint layer.

In this work [24], they first train the network until the hint loss, and then
train the whole network only based on the distillation loss. In contrast, we
propose to learn with both losses simultaneously, as was also done in [3, 28].
Combining the knowledge distillation loss of either the absolute or relative
teacher we would obtain as a final objective function:

$L=L_{ML}+\lambda L_{KD}^{T}+\mu L_{hint},$ (9)

where $T\in\left\\{{abs,rel}\right\\}$ and $\mu$ is used to balance the
relative weight of the hint loss.

Zagoruyko and Komodakis [37] improve the performance of student networks by
forcing them to mimic intermediate attention maps of a powerful teacher
network. Attention maps convey what spatial locations in the image are
considered relevant to the teacher network for its interpretation.
Communicating this information can therefore guide the student network in
learning the task at hand. They propose to compute activation-based spatial
attention according to:

$A_{sum}^{T}(x_{i})=\sum_{l=1}^{C^{k}}\left|F^{T}_{kl}(x_{i})\right|^{2},$
(10)

where $F^{T}_{kl}(x_{i})\in{R}^{w\times h}$ refers to the $l$-th map of the
activation of the $k$-th layer for image $i$. Here $C^{k}$ denotes the number
of feature maps in the $k$-th layer of the teacher net. We use
$\left|.\right|$ to refer to the pixel-wise absolute value, as a results
$A_{sum}^{T}(x_{i})\in{R}^{w\times h}$. A similar equation is used to compute
$A_{sum}^{S}(x_{i})$ from the student activation maps $F^{S}$. The attention
loss is then defined as:

$L_{AT}=\left\|\frac{A_{sum}^{T}(x_{i})}{\left\|A_{sum}^{T}(x_{i})\right\|_{2}}-\frac{A_{sum}^{S}(x_{i})}{\left\|A_{sum}^{S}(x_{i})\right\|_{2}}\right\|.$
(11)

This enforces the student to assign its attention to the same locations which
were deemed important by the teacher.

The full objective function for the attention based metric learning network
becomes:

$L=L_{ML}+\lambda L_{KD}^{T}+\kappa L_{AT},$ (12)

where $\kappa$ defines the relative weight of the attention loss.

In Fig. 3 we show how the hint and attention layers are incorporated between a
ResNet-101 teacher and a ResNet-18 student network. Both hint and attention
losses are applied on multiple layers222For the student we take the output of
each block, and compare it to the last but one layer for each block of the
teacher. The dimensionality of these layers is the same.. Results for this
scheme will be presented in the experimental section.

Table 1: Retrieval Performance on the CUB-200-2011 and Cars-196 dataset.
’ML’:metric learning loss, ’hint’:hint loss; ’AT’: attention loss; KD (abs):
absolute teacher loss; KD (rel):relative teacher loss

| CUB-200-2011 | Cars-196
---|---|---
Recall@K | 1 | 2 | 4 | 8 | 16 | 1 | 2 | 4 | 8 | 16
Student (ResNet-18) | 51.7 | 63.7 | 74.2 | 83.7 | 90.9 | 46.7 | 59.5 | 71.6 | 82.3 | 90.6
PKT [21] | 53.1 | 64.2 | 75.4 | 84.6 | 91.6 | 46.9 | 59.9 | 72.1 | 82.8 | 90.8
DarkRank [4] | 56.2 | 67.8 | 77.2 | 85.0 | 91.5 | 74.3 | 83.6 | 90.0 | 94.2 | 96.9
ML+KD (abs) | 54.9 | 66.5 | 76.5 | 85.0 | 91.3 | 70.6 | 80.7 | 88.0 | 93.2 | 96.0
ML+KD (rel) | 58.0 | 69.0 | 79.4 | 87.8 | 93.6 | 76.6 | 85.4 | 91.2 | 95.0 | 97.3
ML+KD (abs)+hint | 55.0 | 66.5 | 76.6 | 84.9 | 91.1 | 71.3 | 81.2 | 88.1 | 92.7 | 95.9
ML+KD (rel)+hint | 57.4 | 68.8 | 79.1 | 87.4 | 93.1 | 76.4 | 85.5 | 91.3 | 95.1 | 97.2
ML+KD (abs)+AT | 55.0 | 66.3 | 76.9 | 85.3 | 91.8 | 71.1 | 81.3 | 88.3 | 93.1 | 96.0
ML+KD (rel)+AT | 58.1 | 69.2 | 79.6 | 85.3 | 91.3 | 76.4 | 85.7 | 91.7 | 95.0 | 97.2
Teacher (ResNet-101) | 58.9 | 70.4 | 80.7 | 88.2 | 93.5 | 74.8 | 83.6 | 89.9 | 93.8 | 96.5

##  5 Experimental Results

We show results on several benchmark datasets. Our method is implemented with
the PyTorch framework [22]. We will release a GitHub page with code upon
acceptation.

###  5.1 Retrieval on Fine-grained Datasets

Table 2: Comparison on Stanford Online Products dataset.  | Stanford Online Products
---|---
Recall@K | 1 | 10 | 100 | 1000
Student (ResNet-18) | 61.7 | 78.6 | 90.2 | 96.8
ML+KD (abs) | 68.0 | 82.7 | 92.1 | 97.4
ML+KD (rel) | 67.7 | 83.0 | 92.0 | 97.2
Teacher (ResNet-101) | 69.5 | 84.4 | 93.1 | 97.9

Datasets: We evaluate our framework for the task of image retrieval on three
fine-grained datasets:

  * •

CUB-200-2011: this dataset was introduced in [34]. It has 200 classes with
$11,788$ images in total.

  * •

Cars-196: this dataset contains $16,185$ images of $196$ cars classes and was
introduced in [15].

  * •

Stanford Online Products: this dataset introduced in [32] contains $120,053$
images of $22,634$ products collected from eBay.com.

Example images of CUB-200-2011 and Cars-196 are shown in Fig. 5. We follow the
evaluation protocol proposed in [32]. By excluding some classes from the
training of the embedding, we can evaluate at testing time how good the
embedding generalizes to unseen classes. Therefore, the first half of classes
are used for training and the remaining half for testing. For instance, on
CUB-200-2011 dataset, 100 classes (5,864 images) are for training and the
remaining 100 classes (5,924 images) are for testing. We divide the training
set into 80% as training and 20% as validation.

Experimental Details: For these experiments, we use a ResNet-101 as the
teacher network and a ResNet-18 as the student network (see also Fig. 3). The
comparison of the number of parameters of these two networks is shown in Table
4. After the average pooling layer, a linear 512-dimensional embedding layer
is added and the triplet loss is used for training both teacher and student
networks. The Adam [14] optimizer is used with a learning rate of $1e^{-5}$,
and a mini-batch of 32 images. We apply hard negative mining [29] on triplet
loss. For preprocessing, we follow the previous work [20], we resize all
images to 256$\times$256, and crop 224$\times$224 patches randomly. Horizontal
flip is used for data augmentation. We fine-tune both student and teacher
networks from pre-trained ImageNet models with the same preprocessing. During
test time, we only use the 224$\times$224 pixel center crop to predict the
final feature representation used for retrieval. The optimal parameters are
selected according to the performance on the validation set for all of the
experiments. We use the whole training set to retrain with optimal parameters
for a fixed number of epochs.

For evaluation, we use the Recall@$K$ metric [32]: each image in the test set
is projected using the trained network, and if one of the $K$ closest images
in the embedding space has the same label, it is considered as a positive
result. The final score is the fraction of positive results obtained on all
the test images. Furthermore, the reported results in all tables are the
average over three repeated experiments.

Baselines: We start by considering the results of the student and teacher
network in Table 1 and Table 2. Not surprisingly, the teacher network is able
to leverage the additional capacity to learn better embeddings. On the
CUB-200-2011 dataset, we obtain a $R@1$ accuracy of 58.9% for the teacher and
51.7% for the student. This is consistent for the other evaluated recall
levels, although the gap narrows with higher $K$. This shrinking performance
gap is mirrored in the Car-196 dataset, with the teacher net attaining a 28.1%
better $R@1$, and a 5.9% better $R@16$. On the Stanford Online Products
dataset, the gap between the teacher and student network is 7.8%.

We also compare our method with the DarkRank method [4] and PKT [21]333For
these results we used the code made available by the authors.. The experiments
show that the relative teacher network significantly outperforms DarkRank and
PKT , obtaining 1.8% and 4.9% more on CUB-200-2011 and 2.3% and 29.7% on
Cars-196.

Absolute and Relative Loss: Next we incorporate the additional knowledge
distillation losses to the student metric learning objective (indicated by
_ML+KD_). Table 1 shows that results improve for every dataset and recall
level, regardless of the loss used. The performance improvement at Recall@1 is
3.2% and 6.3% respectively for the absolute and relative teacher on
CUB-200-2011. On Cars-196 we see a similar behavior, again the relative
teacher is outperforming the absolute teacher. The student trained with the
relative teacher has an stunning performance gain of almost 30.0%. It is
interesting to note that the relative teacher even outperforms the teacher by
1.8% while having less parameters. On Stanford Online Product dataset, both
absolute and relative teacher obtain similar results, and outperform the
direct training of the student network with 6.0%. In conclusion, the proposed
distillation methods consistently manage to improve the performance of student
network, especially those trained with the relative teacher.

Figure 4: R@1 as a function of $\lambda$ on CUB-200-2011 dataset.

Hint and attention losses: Here we investigate if hint [24] and attention [37]
layers are beneficial for knowledge distillation of embedding networks (see
also Section 4.2). We combine them with our proposed absolute and relative
losses according to Eq. 9 and Eq. 12. The results are summarized in Table 1.
We found that adding a hint layer was not stable. This is probably because the
hint layer is similar as the absolute teacher forcing the network to learn the
exact same embedding as the teacher, and therefore only helps when combined
with the absolute teacher. Adding attention layers in general provided a small
gain but the gain was not as large as reported for classification networks
[37].

Sensitivity to $\lambda$ As shown in Figure 4, we compare R@1 performance as a
function of different $\lambda$ values on the validation set of CUB-200-2011
for both absolute and relative teachers. It is noteworthy that the relative
teacher has a stable performance on a large range of the trade-off parameter
$\lambda$, while the absolute teacher only works in a very narrow range. It
suggests that in practice the selection of the $\lambda$ parameter is not that
essential for the relative teacher.

Table 3: Semi-supervised results on CUB-200-2011.

| Recall@$K$ | 1 | 2 | 4
---|---|---|---|---
50% labeled | Student (ResNet-18) | 51.0 | 63.0 | 74.0
ML+KD (abs) | 51.7 | 63.2 | 73.9
ML+KD (rel) | 56.0 | 66.7 | 77.6
| Teacher (ResNet-101) | 58.1 | 70.0 | 80.1
50% labeled | (ML+KD (abs))/KD (abs) | 53.9 | 65.2 | 75.8
+50% unlabeled | (ML+KD (rel))/KD (rel) | 57.2 | 68.0 | 78.2
50% unlabeled | only KD (abs) | 49.8 | 60.8 | 71.0
only KD (rel) | 55.5 | 67.0 | 77.4

###  5.2 Semi-Supervised Learning

One of the interesting properties of network distillation is that it allows
for the usage of unlabeled data. This was observed by [38] and we apply this
idea here to distillation for embedding networks. The knowledge distillation
losses of Eqs. 4 and 5 do not require any labels. Knowing the estimation of
the teacher network for unlabeled data can help the student network to better
approximate the teacher network. In addition, the existing problems of pair
sampling can be avoided in semi-supervised learning for the student network
because for the unlabeled images we do not apply the triplet loss as it
requires labels. In the experiments we evaluate the benefit of adding
unlabeled data to the student network for embedding learning on the
CUB-200-2011 dataset.

We randomly select half of the training images per class as labeled data, and
consider the rest as unlabeled data. Thus, here we have two teacher-student
learning mechanisms, one is used on the labeled training set with both the
ground truth annotations and information transferred from the teacher, and the
other one is applied to the unlabeled training set with only information from
the teacher by means of a distillation loss. The results of this experiment
can be seen in Table 3. The first row (50% labeled) shows our approach using
only the remaining labeled data, with similar performance as in the previous
experiments: the performance obtained by the relative teacher is closer to
teacher network and better than the absolute teacher. In the second row we add
the remaining 50% of unlabeled data. This leads to improved Recall@K with both
losses, but especially for the relative teacher.

Figure 5:  Example images from two fine-grained datasets CUB-200-2011 and
Cars-196 used in our experiments. The top row shows examples of high-quality
images and the bottom row shows examples of the corresponding low-quality
images.

Finally, in the third row, we consider the case where we have access to a
trained teacher network, but no labelled data at all, to train the student
network. Here we would like to highlight the results of the relative teacher,
since it manages to increase performance by 4.5% compared to the student
network trained with 50% of labeled training data.

###  5.3 Very Small Student Networks

Table 4: Parameter Comparison of Different Networks .

Network | ResNet-101 | ResNet-18 | MobileNet-0.25
---|---|---|---
Parameters |  $\sim$48.1 M |  $\sim$11.3 M |  $\sim$0.3 M

MobileNets [13] are efficient but light-weight networks that can be easily
matched to the design requirements for mobile and embedded vision
applications. To show the potential of our method on very small networks, we
propose to use the MobileNet-0.25 (0.25 is the width multiplier) network as
our student network and use ResNet-101 as the teacher network. The number of
parameters per network is given in Table 4. We can see that the number of
parameters of MobileNet-0.25 is almost 40 times smaller than that of the
previous student network (ResNet-18) and 160 times smaller than that of the
teacher network (ResNet-101).

Table 5 shows retrieval performance results on CUB-200-2011 with
MobileNet-0.25 as the student network. We can observe that the Recall@K for
$K=1,2$ of the teacher network is almost 2 times higher than the student
network. After our relative teacher is applied, the performance gain is 17.1%
at Recall@1 and 15.5% at Recall@16 higher compared to the original student
model.

Table 5: Performance on CUB-200-2011 with MobileNet-0.25.

Recall@K 1 2 4 8 16 Student (MobileNet-0.25 ) 27.5 35.8 46.0 58.5 70.6 ML+KD
(rel) 44.6 56.0 66.4 77.3 86.1 Teacher (ResNet-101) 58.9 70.4 80.7 88.2 93.5

Table 6: Cross quality results on the CUB-200-2011 and Cars-196 datasets with
low resolution and unlocalized object degradations.

|  | Low Resolution | Unlocalized
---|---|---|---
| Recall@$K$ | 1 | 2 | 4 | 1 | 2 | 4
CUB-200-2011 | Student (ResNet-18) | 44.4 | 54.7 | 65.3 | 43.6 | 54.5 | 66.9
ML+KD (abs) | 45.7 | 56.8 | 68.3 | 43.0 | 54.5 | 66.1
ML+KD (rel) | 46.2 | 57.4 | 68.6 | 45.9 | 57.9 | 69.3
Teacher (ResNet-18) | 53.7 | 65.2 | 74.7 | 54.8 | 67.2 | 78.7
Cars-196 | Student (ResNet-18) | 37.5 | 50.0 | 62.6 | 54.0 | 67.3 | 78.2
ML+KD (abs) | 58.6 | 70.7 | 80.7 | 57.7 | 70.4 | 80.6
ML+KD (rel) | 58.9 | 71.0 | 81.1 | 61.9 | 74.4 | 84.2
Teacher (ResNet-18) | 71.0 | 81.2 | 88.7 | 67.8 | 79.1 | 87.9

###  5.4 Cross Quality Distillation

As an additional experiment we do the distillation of embeddings to transfer
knowledge between different domains. This was originally proposed in a
classification setting by Su et al. [33] who, in order to improve the
recognition on low-quality data, use distillation with a teacher trained with
high-quality data. The student then is trained with the low-quality data and
the guidance from the teacher which has access to the high-quality data. Here
we will apply cross quality distillation with the proposed losses for metric
learning. Since in this experiment the objective is not to reduce the number
of parameters but to bridge a domain gap, we use the same architecture
(ResNet-18) for the teacher and the student. To train the embeddings we use
triplet loss and, as in the previous experiments, we train the students with
two teachers: relative and absolute.

We consider two cross quality distillation experiments on CUB-200-2011 and
Cars-196. The first experiment considers low and high-resolution images. To
get the low-resolution images, we downsample them to 50 x 50 and then upsample
them again to 224 x 224 (see examples in Fig. 5). The second experiment
considers unlocalized signal degradation, where the input images are cropped
according to the given bounding boxes for the teachers, but not cropped for
the students. The results can be seen in Table 6.

It can be seen that incorporating the additional knowledge distillation losses
improves the results for most settings, with the relative teachers
consistently surpassing the absolute ones, as in the previous experiments. The
improvement by the distillation is more noticeable on the Cars-196 dataset
which is also observed in [33]. Since it is a more challenging dataset which
has cars with different colors belonging to the same category, the information
provided by the teacher becomes more crucial.

##  6 Conclusions

We have investigated network distillation with the aim of computing efficient
image embedding networks. We have proposed two losses with the aim to
communicate the teacher network knowledge to the student network. We evaluate
our approach on several datasets, and report significant improvements: we
obtain a 6.3% gain on CUB-200-2011, a 29.9% gain on Cars-196 and a 6.3% gain
on Stanford Online Products for Recall@$1$ when compared to a student network
of the exact same capacity which was trained without a teacher network.
Furthermore, we apply our distillation loss to MobileNet-0.25. It greatly
improves the Recall@1 by 17.1%. We also verify the benefit of adding unlabeled
data for embedding learning. In addition, we demonstrate that an embedding
learned on high-quality images can be used to improve the student network
which has only access to low quality images.

Acknowledgement This work was supported by TIN2016-79717-R of the Spanish
Ministry, the CERCA Program and the Industrial Doctorate Grant 2016 DI 039 of
the Ministry of Economy and Knowledge of the Generalitat de Catalunya. Xialei
Liu acknowledges the Chinese Scholarship Council (CSC) grant No.201506290018.
We also acknowledge the generous GPU support from NVIDIA.

## References

  * [1] J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah.  Signature verification using a ”siamese” time delay neural network.  In Advances in Neural Information Processing Systems, pages 737–744, 1994.
  * [2] C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil.  Model compression.  In Proceedings of the 12th international conference on Knowledge discovery and data mining, pages 535–541. ACM, 2006.
  * [3] G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker.  Learning efficient object detection models with knowledge distillation.  In Advances in Neural Information Processing Systems, pages 742–751, 2017.
  * [4] Y. Chen, N. Wang, and Z. Zhang.  Darkrank: Accelerating deep metric learning via cross sample similarities transfer.  In Proceedings of the Conference on Artificial Intelligence, 2018\.
  * [5] S. Chopra, R. Hadsell, and Y. LeCun.  Learning a similarity metric discriminatively, with application to face verification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 1, pages 539–546. IEEE, 2005.
  * [6] C. B. Choy, J. Gwak, S. Savarese, and M. Chandraker.  Universal correspondence network.  In Advances in Neural Information Processing Systems, pages 2414–2422, 2016.
  * [7] T. F. Cox and M. A. Cox.  Multidimensional scaling.  Chapman and hall/CRC, 2000.
  * [8] J. Gao, Z. Li, R. Nevatia, et al.  Knowledge concentration: Learning 100k object classifiers in a single cnn.  In arXiv preprint arXiv:1711.07607, 2017.
  * [9] A. Gordo, J. Almazán, J. Revaud, and D. Larlus.  Deep image retrieval: Learning global representations for image search.  In Proceedings of the European Conference on Computer Vision, pages 241–257. Springer, 2016.
  * [10] S. Han, H. Mao, and W. J. Dally.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  In International Conference on Learning Representations, 2016.
  * [11] G. Hinton, O. Vinyals, and J. Dean.  Distilling the knowledge in a neural network.  In Advances in Neural Information Processing Systems, 2014.
  * [12] E. Hoffer and N. Ailon.  Deep metric learning using triplet network.  In International Workshop on Similarity-Based Pattern Recognition, pages 84–92. Springer, 2015.
  * [13] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [14] D. P. Kingma and L. Ba.  J. adam: a method for stochastic optimization.  In International Conference on Learning Representations, 2015.
  * [15] J. Krause, M. Stark, J. Deng, and L. Fei-Fei.  3d object representations for fine-grained categorization.  In IEEE International Conference on Computer Vision Workshops, pages 554–561, 2013.
  * [16] B. Kulis et al.  Metric learning: A survey.  Foundations and Trends® in Machine Learning, 5(4):287–364, 2013.
  * [17] Y. LeCun, J. S. Denker, and S. A. Solla.  Optimal brain damage.  In Advances in Neural Information Processing Systems, pages 598–605, 1990.
  * [18] X. Liu, J. van de Weijer, and A. D. Bagdanov.  Rankiqa: Learning from rankings for no-reference image quality assessment.  In Proceedings of the International Conference on Computer Vision, 2017.
  * [19] M. Masana, I. Ruiz, J. Serrat, J. van de Weijer, and A. M. Lopez.  Metric learning for novelty and anomaly detection.  In Proceedings of the British Machine Vision Conference, 2018.
  * [20] M. Opitz, G. Waltner, H. Possegger, and H. Bischof.  Bier-boosting independent embeddings robustly.  In International Conference on Computer Vision (ICCV), 2017.
  * [21] N. Passalis and A. Tefas.  Learning deep representations with probabilistic knowledge transfer.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 268–284, 2018.
  * [22] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.  Automatic differentiation in pytorch.  2017\.
  * [23] A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson.  Cnn features off-the-shelf: an astounding baseline for recognition.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 512–519, 2014.
  * [24] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio.  Fitnets: Hints for thin deep nets.  In International Conference on Learning Representations, 2015.
  * [25] F. Schroff, D. Kalenichenko, and J. Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 815–823, 2015.
  * [26] T. Scott, K. Ridgeway, and M. C. Mozer.  Adapted deep embeddings: A synthesis of methods for k-shot inductive transfer learning.  In Advances in Neural Information Processing Systems, pages 76–85, 2018.
  * [27] O. Sener, H. O. Song, A. Saxena, and S. Savarese.  Learning transferrable representations for unsupervised domain adaptation.  In Advances in Neural Information Processing Systems, pages 2110–2118, 2016.
  * [28] J. Shen, N. Vesdapunt, V. N. Boddeti, and K. M. Kitani.  In teacher we trust: Learning compressed models for pedestrian detection.  In arXiv preprint arXiv:1612.00478, 2016.
  * [29] E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and F. Moreno-Noguer.  Discriminative learning of deep convolutional feature point descriptors.  In Proceedings of the International Conference on Computer Vision, pages 118–126, 2015.
  * [30] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  In International Conference on Learning Representations, 2015.
  * [31] K. Sohn.  Improved deep metric learning with multi-class n-pair loss objective.  In Advances in Neural Information Processing Systems, pages 1857–1865, 2016.
  * [32] O. H. Song, Y. Xiang, S. Jegelka, and S. Savarese.  Deep metric learning via lifted structured feature embedding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4004–4012, 2016.
  * [33] J.-C. Su and S. Maji.  Adapting models to signal degradation using distillation.  In Proceedings of the British Machine Vision Conference, 2017.
  * [34] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The caltech-ucsd birds-200-2011 dataset.  Computation & Neural Systems Technical Report, CNS-TR-2011-001, 2011.
  * [35] J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and Y. Wu.  Learning fine-grained image similarity with deep ranking.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1386–1393, 2014.
  * [36] X. Wang and A. Gupta.  Unsupervised learning of visual representations using videos.  In Proceedings of the International Conference on Computer Vision, pages 2794–2802, 2015.
  * [37] S. Zagoruyko and N. Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  In International Conference on Learning Representations, 2016.
  * [38] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu.  Deep mutual learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.