# Optical manipulation of Rashba-split 2-Dimensional Electron Gas

M. Michiardi<EMAIL_ADDRESS>Quantum Matter Institute, University of
British Columbia, Vancouver, BC V6T 1Z4, Canada  Department of Physics &
Astronomy, University of British Columbia, Vancouver, BC V6T 1Z1, Canada  Max
Planck Institute for Chemical Physics of Solids, Dresden, Germany  F. Boschini
Quantum Matter Institute, University of British Columbia, Vancouver, BC V6T
1Z4, Canada  Department of Physics & Astronomy, University of British
Columbia, Vancouver, BC V6T 1Z1, Canada  Centre Énergie Matériaux
Télécommunications, Institut National de la Recherche Scientifique, Varennes,
Québec J3X 1S2, Canada  H.-H. Kung  Quantum Matter Institute, University of
British Columbia, Vancouver, BC V6T 1Z4, Canada  Department of Physics &
Astronomy, University of British Columbia, Vancouver, BC V6T 1Z1, Canada  M.
X. Na  Quantum Matter Institute, University of British Columbia, Vancouver, BC
V6T 1Z4, Canada  Department of Physics & Astronomy, University of British
Columbia, Vancouver, BC V6T 1Z1, Canada  S. K. Y. Dufresne  Quantum Matter
Institute, University of British Columbia, Vancouver, BC V6T 1Z4, Canada
Department of Physics & Astronomy, University of British Columbia, Vancouver,
BC V6T 1Z1, Canada  A. Currie  Quantum Matter Institute, University of British
Columbia, Vancouver, BC V6T 1Z4, Canada  Department of Physics & Astronomy,
University of British Columbia, Vancouver, BC V6T 1Z1, Canada  G. Levy
Quantum Matter Institute, University of British Columbia, Vancouver, BC V6T
1Z4, Canada  Department of Physics & Astronomy, University of British
Columbia, Vancouver, BC V6T 1Z1, Canada  S. Zhdanovich  Quantum Matter
Institute, University of British Columbia, Vancouver, BC V6T 1Z4, Canada
Department of Physics & Astronomy, University of British Columbia, Vancouver,
BC V6T 1Z1, Canada  A. K. Mills  Quantum Matter Institute, University of
British Columbia, Vancouver, BC V6T 1Z4, Canada  Department of Physics &
Astronomy, University of British Columbia, Vancouver, BC V6T 1Z1, Canada  D.
J. Jones  Quantum Matter Institute, University of British Columbia, Vancouver,
BC V6T 1Z4, Canada  Department of Physics & Astronomy, University of British
Columbia, Vancouver, BC V6T 1Z1, Canada  J. L. Mi  Department of Chemistry,
Aarhus University, 8000 Aarhus C, Denmark  B. B. Iversen  Department of
Chemistry, Aarhus University, 8000 Aarhus C, Denmark  Ph. Hofmann  Department
of Physics and Astronomy, Interdisciplinary Nanoscience Center, Aarhus
University, 8000 Aarhus C, Denmark  A. Damascelli<EMAIL_ADDRESS>Quantum Matter Institute, University of British Columbia, Vancouver, BC V6T
1Z4, Canada  Department of Physics & Astronomy, University of British
Columbia, Vancouver, BC V6T 1Z1, Canada

††preprint: APS/123-QED

In spintronic devices, the two main approaches to actively control the
electrons’ spin degree of freedom involve either static magnetic or electric
fields [1, 2, 3]. An alternative avenue relies on the application of optical
fields to generate spin currents, which promises to bolster spin-device
performance allowing for significantly faster and more efficient spin logic.
To date, research has mainly focused on the optical injection of spin currents
through the photogalvanic effect [4, 5], and little is known about the direct
optical control of the intrinsic spin splitting [6, 7]. Here, to explore the
all-optical manipulation of a material’s spin properties, we consider the
Rashba effect at a semiconductor interface [8, 9]. The Rashba effect has long
been a staple in the field of spintronics owing to its superior tunability,
which allows the observation of fully spin-dependent phenomena, such as the
spin-Hall effect, spin-charge conversion, and spin-torque in semiconductor
devices [10, 11]. In this work, by means of time and angle-resolved
photoemission spectroscopy (TR-ARPES), we demonstrate that an ultrafast
optical excitation can be used to manipulate the Rashba-induced spin splitting
of a two-dimensional electron gas (2DEG) engineered at the surface of the
topological insulator Bi2Se3. We establish that light-induced photovoltage and
charge carrier redistribution – which in concert modulate the spin-orbit
coupling strength on a sub-picosecond timescale – can offer an unprecedented
platform for achieving all optically-driven THz spin logic devices.

Spintronics has the potential to deliver computational devices that are less
volatile, faster, and more energy efficient with respect to their electronic
counterparts. However, the need to control the spin degree of freedom in a
fast and efficient manner is challenging, as the field required to flip the
electron’s spin in magnetic materials is often prohibitively high. Spin-orbit
coupling (SOC) effects, such as the Rashba effect, allow the formation of
spin-polarized electron states without a magnetic moment, thereby
circumventing this limitation. In particular, the Rashba effect breaks the
spin degeneracy at semiconductor interfaces, resulting in quasi-particle bands
of opposite spin texture that are offset in momentum. An example of a Rashba-
split quasi-free electron state with effective mass $m^{*}$ is shown in Fig.
1a. To the first order, its dispersion relation is given by:

$E=\frac{\hbar^{2}k^{2}}{2m^{*}}\pm\alpha_{R}k.$ (1)

Here, the parameter $\alpha_{R}$ is the strength of the Rashba SOC (RSOC) in
the system, and it depends on the atomic SOC as well as the electric field
perpendicular to the surface (E⟂). Experimentally, $\alpha_{R}$ can be
extracted from the detailed dispersion of the spin-split subbands: the energy
splitting of the subbands is given by $\Delta E_{R}=2\alpha_{R}k$, and can be
seen as a momentum-dependent Zeeman splitting caused by the pseudo-magnetic
field – or Rashba field –
$\textbf{B}_{R}\propto\textbf{k}\times\textbf{E}_{\perp}$; correspondingly,
the momentum splitting is given by $\Delta k_{R}=2\alpha_{R}m^{*}/\hbar^{2}$.

We illustrate the inner workings of these parameters with the paradigmatic
example of the spin field-effect transistor (spinFET), depicted in Fig. 1b. In
the pioneering concept of Datta and Das [12], a Rashba-split two-dimensional
electron gas (2DEG) in a channel of length $L$ is sandwiched between two spin-
polarized leads. As electrons transit the Rashba 2DEG in the direction
perpendicular to the Rashba field $\textbf{B}_{R}$, their spin precesses,
acquiring a phase $\Delta\Theta=\Delta k_{R}L$. Switching between the 0/1
logic operation – corresponding to the low/high resistance state in the device
– is achieved by tuning $\Delta k_{R}$ such that the electron spin at $L$
aligns to that of the drain lead. As $\Delta k_{R}$ is proportional to
$\alpha_{R}$, the operation of such a device relies primarily on the
possibility to tune the RSOC, typically realized by gating the 2DEG.

In spintronic devices such as the spinFET, the prospect to replace the gate
with an optical field prompts the development of even faster and more
efficient hybrid opto-spintronics. To this end, previous works have
demonstrated the generation of spin-polarized currents in Rashba and
topological states through the photogalvanic effect, as well as the ultrafast
switching of spin orientation in antiferromagnets [13, 14, 15, 16, 4, 17, 18].
Here, we show that light can change the RSOC strength, effectively
manipulating the Rashba spin-transport properties on an engineered 2DEG. The
proposed mechanism is as follows: in the presence of a band-bending surface
potential, an above-gap optical excitation drives a charge redistribution
along the axis perpendicular to the surface. This charge redistribution
creates an ultrafast photovoltage, which then reliably alters the RSOC
strength ($\alpha_{R}$) of the 2DEG system on a sub-picosecond timescale. We
employ time- and angle-resolved photoemission spectroscopy (TR-ARPES) to track
the evolution of the RSOC strength through the dispersion of the Rashba 2DEGs.
By directly measuring $\Delta k_{R}$ and $\Delta E_{R}$ as a function of pump-
probe delay, we unambiguously extract the evolution of $\alpha_{R}$.

Among the materials that can host Rashba-split 2DEGs, bismuth-based
topological insulators (TI) are an ideal platform: 2DEGs can be induced on the
surface of TIs by applying a positive surface bias or chemical gating [19, 20,
21, 22]. The combination of the strong atomic SOC in TIs with surface gating
generates a substantial Rashba effect in the 2DEGs, allowing one to finely
resolve the spin splitting. In an ideal TI, only the topological surface state
(TSS) – recognizable by its linear dispersion across the bandgap – crosses the
Fermi level ($E_{F}$), and all charge carriers belong to the TSS [23]. As
represented in Fig. 2a, the application of a sufficient positive bias at the
surface induces a strong band bending, leading to the creation of 2DEGs in the
form of surface confined quantum well states (QWSs). While the TSS
wavefunction extends only within a few layers from the surface and does not
depend on the shape of the surface potential, the wavefunction of the QWS
does, and extends comparatively deeper into the bulk [20]. The difference in
spatial extent between the TSS and QWS wavefunctions allows us to extract the
behavior specific to QWSs, as opposed to the behavior of surface states in
general.

Our experimental approach is depicted in Fig. 2b. We choose p-doped Bi2Se3 to
host QWSs, as the hole doping provides a lower bulk conductivity in this
material. The 2DEGs are prepared by depositing a controlled amount of alkali
atoms on the surface, leading to a population of conduction band-derived
states that are spin-split by the Rashba effect. Increasing the concentration
of deposited atoms is analogous to raising the surface bias, which introduces
a higher surface charge density and stronger band bending. The system is then
optically excited with a near-infrared (1.55 eV) ”pump” pulse and its response
is probed by photoemission using a UV (6.2 eV) pulse at variable time delay
$\Delta$t [24, 25]. The result of such a TR-ARPES experiment is summarized in
Fig. 2c over a long range of delays. The left and right panels show the ARPES
spectra at negative delay (-100 ps), and 500 ps after the pump arrival,
respectively. The central panel presents the evolution of the states at the
Brillouin zone center (black dashed lines in the left panel). Before the pump
arrival (-100 ps), the system is in equilibrium; the Fermi level is crossed by
the linear topological surface state (TSS) and a single parabolic band,
nominally the first quantum well state (QWS1). Here, the Rashba-splitting is
just barely discernible, owing to the moderate chemical gating. At zero-delay,
electrons are optically excited into unoccupied states and subsequently decay
into a quasi-thermalized state [26, 27]. Remarkably, we see that QWS1 is
pushed to lower energies after the excitation, and a second band becomes
populated. This second band (shown also in the spectrum at 500 ps) is in fact
the second quantum well state (QWS2), which emerges following an increase in
surface charge density. It is worth noting that the aforementioned
photovoltage induced by the pump pulse also affect the kinetic energy of
photoemitted electrons [28, 29, 30]. This manifests as a rigid shift of the
ARPES spectra that can be accounted for by a simple subtraction; henceforth,
we refer all energy scales to the electron quasi-Fermi level EFn, extracted by
fitting a Fermi-Dirac distribution to the photoemission intensity around the
TSS Fermi vector(details can be found in the supplementary information).

In pursuance of determining the impact of the optical excitation on the Rashba
effect, we perform TR-ARPES on a sample with a higher concentration of
deposited alkali atoms, so that energy and momentum splittings are better
distinguished. The results of this experiment are shown in Fig. 3. In panel a,
the dispersion is shown for three pump-probe delays ($\Delta$t=-0.5, 0, and 8
ps). We observe that both QWSs (parabolic bands) are populated before pump
arrival, with energy minima at -114 and -13 meV, and QWS1 exhibits a visible
and strong Rashba splitting. Differential ARPES maps
[$I(\mathbf{k},E,t)-I(\mathbf{k},E,-0.5$ ps)] of the 0 and 8 ps delays are
also shown, highlighting the pump-induced modification of the QWSs. At time
zero, the optical excitation creates an electron population (depletion) above
(below) EFn, but shows no appreciable change in dispersion. However, at 8 ps,
while the TSS shows no significant change, both QWSs shift downwards in
response to an increase in surface charge, similarly to what was reported in
Fig. 2(c). This is further emphasized in Fig 3b where the time dependent
energy shifts of the QWSs at Brillouin zone center are displayed. For both
QWSs, the energy minimum shows a fluctuation at short timescales before
eventually settling to lower energy. We fit the curves in Fig. 3b with a
phenomenological model that includes two exponentially decaying processes,
shown in purple and cyan respectively (note that the latter curve appears flat
because of a long decay time). The first process acts to increase the QWSs’
energy, peaking at approximately 1.5 ps after the pump excitation, and
decaying within 3 ps. This characteristic timescale follows that of the
electron population above the Fermi level in TIs [26]; therefore, we attribute
this process to a non-thermal (NT) effect given by the presence of hot
carriers close to EFn. The second and more interesting process is a long-
lasting shift of the QWSs to lower energy, that emerges as a consequence of an
increase in the surface electron population and variation of the electrostatic
environment. This process – as we will show in detail – is given by a
photovoltage (PV) effect, and it is the central mechanism of this work. The
effect of the PV arises within a few hundred femtoseconds and alters the
energy and density of the QWSs over hundreds of picoseconds.

As the PV alters the electrostatic environment at the surface, we expect it
will have an impact on the Rashba effect as well. For a quantitative look at
the momentum splitting, we plot in Fig. 3c the momentum distribution curves
(MDCs) of QWS1 at EFn, in equilibrium (before the pump arrival) and 8 ps after
the excitation. The MDCs span the two spin-polarized bands on the right-hand
side of QWS1 (see red dashed line in Fig. 3a) and are referenced to the Fermi
momentum of the inner branch ($k_{F1}$). The MDC peak locations are indicated
by dashed lines; in comparing the equilibrium (purple) and post-excitation
(blue) MDCs, we observe that the momentum splitting of the carriers is reduced
from (26.0 $\pm$ 0.5)$\times 10^{-3}$ to (22.5 $\pm$ 0.5)$\times 10^{-3}$ Å-1.
A similar result is observed for the energy splitting: in Fig. 3d, we plot the
energy distribution curves (EDCs), for the same two delays, along the cut
shown in Fig. 3a; we find that, whereas the outer branch of QWS1 maintains its
position, the inner branch moves to lower energy, leading to a reduction of
the energy splitting $\Delta E_{R}$ by approximately 13 meV. The additional
shoulder observed in the EDC at 8 ps arises from QWS2, which also moves to
significantly lower energy. The simultaneous reduction of both $\Delta k_{R}$
and $\Delta E_{R}$ is a clear indication of an optically driven change of the
Rashba spin-orbit coupling strength $\alpha_{R}$, and excludes modifications
of the electron dispersion and effective mass as relevant contributions.

The full temporal dynamics of the Rashba effect in QWS1 under optical
excitation is given in Fig. 3e, where the splitting in momentum $\Delta k_{R}$
(in orange), as well as the RSOC strength obtained following Eq. 1, are shown
on the left and right y-axis, respectively. We observe that $\Delta k_{R}$
decreases immediately after the excitation, and – after a fluctuation lasting
less than 3 ps – is effectively reduced to a seemingly constant value. The
change in the RSOC strength is about 15%, decreasing from a value of 0.76
$\pm$ 0.02 to 0.66 $\pm$ 0.02 eVÅ. Similar values of $\alpha_{R}$ are obtained
by performing an analogous analysis on $\Delta E_{R}(k)$, plotted in green;
here, the EDC fit between 0 and 2 ps could not be performed due to the
presence of a highly non-thermal electronic distribution. Our data outline a
scenario where an opportune optical pulse changes the RSOC strength, in a
manner similar to a static electric field, on a picosecond time-scale.

The observation of an increased surface electron density in concert with a
decrease in RSOC is, however, nontrivial, as the two quantities typically
increase/decrease correspondingly. Thus, a satisfactory explanation for the
observed effect of the pump excitation, requires one to consider the detailed
variation of the surface electric field in relation to the spatial electron
distribution. To this end, we build a model to capture the salient aspects of
the experimental results. We begin by calculating the band bending of the
system at equilibrium, which can be described by a one-dimensional model in
the out of plane direction $x$. The potential profile $V(x)$ is calculated by
solving the Poisson equation within a modified Thomas-Fermi approximation [31,
32]. The binding energy and wavefunction of the QWSs is computed a posteriori
by solving the Schrödinger equation within the calculated $V(x)$. For our
simulation, all material-specific parameters for the calculation are taken
from Refs. 33, 34, 35, 36, and the surface potential $V_{0}$ is determined
empirically by the shift of the TSS Dirac point induced by chemical gating
(see Supplementary Information).

We present the calculated equilibrium potential and QWSs’ energy minima in
Fig. 4a. The space charge region (SCR) spans more than 30 nm, and QWS1 and
QWS2 are partially populated, replicating the experimental observations of
Fig. 3. Following an optical excitation across the band gap, the generated
electron-hole pairs within the SCR are swept apart by the electric field ESCR,
which pushes the negative charges towards the surface and the positive charges
towards the bulk [37]. The electrons and holes become spatially separated over
tens of nanometers, giving rise to a long-lasting photovoltage field EPV of
the opposite sign to ESCR. This effectively softens the band bending, pushing
the surface potential V0 to less negative values, as shown in Fig. 4b. The new
shallower surface potential drives both the QWSs and the $E_{Fn}$ to higher
energy. To accommodate the surplus of surface electric charge, the $E_{Fn}$
shifts further upwards, resulting in the QWSs moving to more negative energies
when plotted with respect to $E_{Fn}$, as seen in the TR-ARPES data (details
in Supplementary Information). It must be noted that, as small surface state-
induced band bending is a common feature in semiconductors, surface PV has
previously been observed in pristine TIs [38, 39, 40]. However, while this is
technologically relevant for TIs – because it leads to spin-polarized
diffusion currents – the TSS only undergoes a rigid shift in energy under the
surface PV. The 2DEGs, on the other hand, are much more sensitive to the shape
and magnitude of the confining potential $V(x)$, and by extension, the PV.

The full temporal dynamics of the QWSs has been simulated by introducing a
pair of effective photo-charges in the system, which approximates the
collective charge motion via a center of mass approach [41]. At each time
step, the charge distribution, electric field and EFn are reevaluated. The
time evolution of the QWSs’ energy with respect to the EFn – shown in Fig. 4c
– is in good qualitative agreement with experimental data. Both QWS1 and QWS2
fall to more negative values almost instantaneously at positive delays.
Consistent with experimental observations, the energy shift of QWS2 is larger
than that of QWS1. This is a manifestation of the shallower confining
potential, which allows for a smaller energy difference between consecutive
QWSs. Lastly, the RSOC strength and, by extension, Rashba-splitting for QWS1
are calculated at each step from Eq. 1. The evolution of $\Delta k_{R}$ is
also consistent with experimental findings (Fig. 4d), confirming that the
reduction in the Rashba strength is due to the PV-induced softening of the
surface potential.

With the exception of the fluctuation at early delays given by the presence of
hot carriers, our simple model succeeds in reproducing all salient features of
the experimental data, proving that a photovoltage is responsible for the
observed behavior. The small quantitative deviations can be attributed to the
model simplicity and approximations, such as the omission of time-dependent
changes in the screening and dielectric properties of the material.
Nevertheless, the model captures the fundamental observations of the
experiment and provides a clear explanation of the underlying mechanisms. The
simulated spectral function in the inset of Fig. 4d showcases the calculated
dispersion of QWS1 at the two representative time delays, highlighting the
faithful reproduction of the TR-ARPES data. Finally, the PV model can also
account for the long timescale (950 ps) needed to recover equilibrium
conditions (Fig. 2c), as the spatial separation of electrons and holes in the
SCR drastically reduces the recombination rate. Since the return to
equilibrium is ultimately determined by the charge carriers’ diffusion from
the illuminated area, the lifetime of the PV effect could in principle be
tuned by varying the size of the pump beam.

In conclusion, we demonstrated that light can be used to control the Rashba
spin splitting and, by extension, the spin transport properties in
semiconductor devices. Specifically, an optically driven photovoltage can be
used to manipulate the surface band dispersion and electron distribution at
ultrafast (picosecond) timescales. The specific application of this technique
on 2DEGs to tune the Rashba-splitting on a picosecond timescale is an
important benchmark for the development of optically controlled spin devices.
We review our findings in the context of the spinFET discussed in Fig. 1b: the
observed variation of $\Delta k_{R}$ in QWS1, about 3.5$\times 10^{-3}$ Å-1,
translates into a difference in spin precession angle of $\pi$ after $<$100 nm
travel distance, making this effect already appreciable in devices of such
length, where ballistic transport can be achieved. It is important to
emphasize that, while this study is performed on a TI platform, the underlying
physics does not require topological non-triviality and is universal to
semiconductors. The effect of the PV on the Rashba strength can be enhanced
producing 2DEGs with higher effective masses, while surface gating and pump
fluence can be utilized as tuning parameters.

Methods : Samples of Ca-doped Bi2Se3 are synthesized as described in Ref. 42.
Here, Ca acts as acceptor atom, positively doping Bi2Se3 which is normally
found to be n-doped due to Se vacancies. The samples are cleaved in vacuum at
pressures lower than $7\cdot 10^{-11}$ mbar, and kept at a temperature of 15 K
during evaporation and measurements. 2DEGs are induced by evaporating K (Fig.
2) or Li (Fig. 3) in situ on the cleaved sample surface. The TR-ARPES
experiments are performed at QMI’s UBC-Moore Center for Ultrafast Quantum
Matter [43, 44], with 1.55 and 6.2 eV photons for pump and probe,
respectively. Both pump and probe have linear horizontal polarization
(parallel to the analyzer slit direction). The pump (probe) beam radius is 150
$\mu m$ (100 $\mu m$), and the pump fluence is 40 and 80 $\mu J/cm^{2}$ for
experiments represented in Fig. 2 and 3, respectively. Pump and probe were
collinear with an incidence angle of 45 degrees with respect to the sample
normal. Energy and temporal resolution are 17 meV and 250 fs, respectively, as
determined by the width of the gold Fermi edge and of the combined pump-probe
dynamics of the pump induced direct population peak in Bi2Se3 [26]. For the
band bending model, the Poisson equation was solved numerically employing a
modified Thomas-Fermi approximation, which intrinsically accounts for
modulation of the charge density due to confinement-induced quantization,
without the need for numerically heavy self-consistent calculations [45, 32].
The Schrödinger equation was solved numerically with the Numerov algorithm
[46]. The code is available at Ref. 47.

Acknowledgements : We would like to thank M. Bianchi and P. D. C. King for the
fruitful discussions. This research was undertaken thanks in part to funding
from the Max Planck-UBC-UTokyo Centre for Quantum Materials and the Canada
First Research Excellence Fund, Quantum Materials and Future Technologies
Program. This project is also funded by the Gordon and Betty Moore
Foundation’s EPiQS Initiative, Grant GMBF4779 to A.D. and D.J.J.; the Killam,
Alfred P. Sloan, and Natural Sciences and Engineering Research Council of
Canada’s (NSERC’s) Steacie Memorial Fellowships (A.D.); the Alexander von
Humboldt Foundation (A.D.); the Canada Research Chairs Program (A.D.); NSERC,
Canada Foundation for Innovation (CFI); British Columbia Knowledge Development
Fund (BCKDF); the VILLUM FONDEN via the Centre of Excellence for Dirac
Materials (Grant No. 11744); and the CIFAR Quantum Materials Program.

## References

  * [1] Oestreich, M. _et al._ Spintronics: Spin electronics and optoelectronics in semiconductors.  _Advances in Solid State Physics_ 41, 173–186 (2001).
  * [2] Gomonay, O., Jungwirth, T. & Sinova, J.  Concepts of antiferromagnetic spintronics.  _Physica Status Solidi (RRL) - Rapid Research Letters_ 11, 1700022 (2017).
  * [3] Chuang, P. _et al._ All-electric all-semiconductor spin field-effect transistors.  _Nature Nanotechnology_ 10, 35–39 (2015).
  * [4] McIver, J. W., Hsieh, D., Steinberg, H., Jarillo-Herrero, P. & Gedik, N.  Control over topological insulator photocurrents with light polarization.  _Nature Nanotechnology_ 7, 96–100 (2012).
  * [5] Gmitra, M. & Fabian, J.  Graphene on transition-metal dichalcogenides: A platform for proximity spin-orbit physics and optospintronics.  _Physical Review B_ 92, 155403 (2015).
  * [6] Sheremet, A. S., Kibis, O. V., Kavokin, A. V. & Shelykh, I. A.  Datta-and-Das spin transistor controlled by a high-frequency electromagnetic field.  _Physical Review B_ 93, 165307 (2016).
  * [7] Cheng, L. _et al._ Optical manipulation of Rashba spin–orbit coupling at SrTiO3-based oxide interfaces.  _Nano Letters_ 17, 6534–6539 (2017).
  * [8] Bychkov, Y. A. & Rashba, E. I.  Properties of a 2D electron gas with a lifted spectrum degeneracy.  _Journal of Experimental and Theoretical Physics Letters_ 78–81 (1984).
  * [9] Rashba, E.  Spin–orbit coupling and spin transport.  _Physica E: Low-dimensional Systems and Nanostructures_ 34, 31–35 (2006).
  * [10] Bercioux, D. & Lucignano, P.  Quantum transport in Rashba spin-orbit materials: a review.  _Reports on Progress in Physics_ 78, 106001 (2015).
  * [11] Soumyanarayanan, A., Reyren, N., Fert, A. & Panagopoulos, C.  Emergent phenomena induced by spin–orbit coupling at surfaces and interfaces.  _Nature_ 539, 509–517 (2016).
  * [12] Datta, S. & Das, B.  Electronic analog of the electro‐optic modulator.  _Applied Physics Letters_ 56, 665–667 (1990).
  * [13] Yuan, H. _et al._ Generation and electric control of spin–valley-coupled circular photogalvanic current in WSe2.  _Nature Nanotechnology_ 9, 851–857 (2014).
  * [14] Liu, X. _et al._ Circular photogalvanic spectroscopy of Rashba splitting in 2D hybrid organic–inorganic perovskite multiple quantum wells.  _Nature Communications_ 11, 323 (2020).
  * [15] Kimel, A. V., Kirilyuk, A., Tsvetkov, A., Pisarev, R. V. & Rasing, T.  Laser-induced ultrafast spin reorientation in the antiferromagnet TmFeO3.  _Nature_ 429, 850–853 (2004).
  * [16] Nova, T. F. _et al._ An effective magnetic field from optically driven phonons.  _Nature Physics_ 13, 132–136 (2017).
  * [17] Wang, J., Zhu, B.-F. & Liu, R.-B.  Proposal for direct measurement of a pure spin current by a polarized light beam.  _Physical Review Letters_ 100, 086603 (2007).
  * [18] Zhou, B. & Shen, S.-Q.  Deduction of pure spin current from the linear and circular spin photogalvanic effect in semiconductor quantum wells.  _Physical Review B_ 75, 045339 (2007).
  * [19] Bianchi, M. _et al._ Coexistence of the topological state and a two-dimensional electron gas on the surface of Bi2Se3.  _Nature Communications_ 1, 128 (2010).
  * [20] Bahramy, M. _et al._ Emergent quantum confinement at topological insulator surfaces.  _Nature Communications_ 3, 1159 (2012).
  * [21] Zhu, Z.-H. _et al._ Rashba spin-splitting control at the surface of the topological insulator Bi2Se3.  _Physical Review Letters_ 107, 186405 (2011).
  * [22] Michiardi, M. _et al._ Strongly anisotropic spin-orbit splitting in a two-dimensional electron gas.  _Physical Review B_ 91, 035445 (2015).
  * [23] Barreto, L. _et al._ Surface-dominated transport on a bulk topological insulator.  _Nano Letters_ 14, 3755–3760 (2014).
  * [24] Perfetti, L. _et al._ Time evolution of the electronic structure of 1T-TaS2 through the insulator-metal transition.  _Physical Review Letters_ 97, 067402 (2006).
  * [25] Freericks, J. K., Krishnamurthy, H. R. & Pruschke, T.  Theoretical description of time-resolved photoemission spectroscopy: Application to pump-probe experiments.  _Physical Review Letters_ 102, 136401 (2009).
  * [26] Sobota, J. _et al._ Ultrafast electron dynamics in the topological insulator Bi2Se3 studied by time-resolved photoemission spectroscopy.  _Journal of Electron Spectroscopy and Related Phenomena_ 195, 249–257 (2014).
  * [27] Hajlaoui, M. _et al._ Ultrafast surface carrier dynamics in the topological insulator Bi2Te3.  _Nano Letters_ 12, 3532–3536 (2012).
  * [28] Ciocys, S., Morimoto, T., Moore, J. E. & Lanzara, A.  Tracking surface photovoltage dipole geometry in Bi2Se3 with time-resolved photoemission.  _Journal of Statistical Mechanics: Theory and Experiment_ 2019, 104008 (2019).
  * [29] Yang, S.-L., Sobota, J. A., Kirchmann, P. S. & Shen, Z.-X.  Electron propagation from a photo-excited surface: implications for time-resolved photoemission.  _Applied Physics A_ 116, 85–90 (2014).
  * [30] Tanaka, S.-i.  Utility and constraint on the use of pump-probe photoelectron spectroscopy for detecting time-resolved surface photovoltage.  _Journal of Electron Spectroscopy and Related Phenomena_ 185, 152–158 (2012).
  * [31] Paasch, G. & Übensee, H.  A Modified Local Density Approximation. Electron Density in Inversion Layers.  _Physica Status Solidi (b)_ 113, 165–178 (1982).
  * [32] King, P. D. C., Veal, T. D. & McConville, C. F.  Nonparabolic coupled Poisson-Schrödinger solutions for quantized electron accumulation layers: Band bending, charge profile, and subbands at InN surfaces.  _Physical Review B_ 77, 125305 (2008).
  * [33] Analytis, J. G. _et al._ Bulk Fermi surface coexistence with Dirac surface state in Bi2Se3: A comparison of photoemission and Shubnikov–de Haas measurements.  _Physical Review B_ 81, 205407 (2010).
  * [34] Martinez, G. _et al._ Determination of the energy band gap of Bi2Se3.  _Scientific Reports_ 7, 6891 (2017).
  * [35] Nechaev, I. A. _et al._ Evidence for a direct band gap in the topological insulator Bi2Se3 from theory and experiment.  _Physical Review B_ 87, 121111 (2013).
  * [36] Gao, Y.-B., He, B., Parker, D., Androulakis, I. & Heremans, J. P.  Experimental study of the valence band of Bi2Se3.  _Physical Review B_ 90, 125204 (2014).
  * [37] Kronik, L. & Shapira, Y.  Surface photovoltage phenomena: theory, experiment, and applications.  _Surface Science Reports_ 37, 1–206 (1999).
  * [38] Yoshikawa, T. _et al._ Bidirectional surface photovoltage on a topological insulator.  _Physical Review B_ 100, 165311 (2019).
  * [39] Sánchez-Barriga, J. _et al._ Laser-induced persistent photovoltage on the surface of a ternary topological insulator at room temperature.  _Applied Physics Letters_ 110, 141605 (2017).
  * [40] Ciocys, S. _et al._ Manipulating long-lived topological surface photovoltage in bulk-insulating topological insulators Bi2Se3 and Bi2Te3.  _npj Quantum Materials_ 5, 16 (2020).
  * [41] Mora-Seró, I., Dittrich, T., Garcia-Belmonte, G. & Bisquert, J.  Determination of spatial charge separation of diffusing electrons by transient photovoltage measurements.  _Journal of Applied Physics_ 100, 103705 (2006).
  * [42] Bianchi, M. _et al._ Robust surface doping of Bi2Se3 by rubidium intercalation.  _ACS Nano_ 6, 7009–7015 (2012).
  * [43] Mills, A. K. _et al._ Cavity-enhanced high harmonic generation for extreme ultraviolet time- and angle-resolved photoemission spectroscopy.  _Review of Scientific Instruments_ 90, 083001 (2019).
  * [44] Boschini, F. _et al._ Collapse of superconductivity in cuprates via ultrafast quenching of phase coherence.  _Nature Materials_ 17, 416–420 (2018).
  * [45] Trott, S., Trott, M. & Nakov, V.  Modified Thomas‐Fermi Approximation. A Surprisingly Good Tool for the Treatment of Semiconductor Layer Structures Including Various Two‐Dimensional Systems.  _Phys Status Solidi B_ 177, 389–395 (1993).
  * [46] Vigo-Aguiar, J. & Ramos, H.  A variable-step Numerov method for the numerical solution of the Schrödinger equation.  _Journal of Mathematical Chemistry_ 37, 255–262 (2005).
  * [47] Michiardi, M.  P-MTFA-S band bending simulation (2021).  URL https://github.com/okio-mm/P-MTFA-S_band_bending_simulation.
  * [48] Yang, H.-B. _et al._ Emergence of preformed Cooper pairs from the doped Mott insulating state in Bi2Sr2CaCu2O8+δ.  _Nature_ 456, 77–80 (2008).

Figure 1: Rashba spin-orbit coupling in two-dimensional electron gas. a:
Rashba spin-orbit-coupling (RSOC) splits a free electron state into two
subbands carrying opposite spin texture (red and blue). The splitting of the
free electron state in both energy ($\Delta E_{R}$) and momentum ($\Delta
k_{R}$) is proportional to the RSOC strength $\alpha_{R}$, which is tunable
with an electric field $E_{\perp}$. This Rashba splitting locks the electron’s
spin to its momentum. b: Fundamental design of a spin field-effect transistor
(spinFET) in which spin-polarized electrons are injected from a source into a
Rashba 2DEG and collected with a ferromagnetic drain. Due to the momentum-
dependent splitting of Rashba 2DEGs, charges traversing from source to drain
feel an effective magnetic field, $B_{R}$, proportional to $\alpha_{R}$,
perpendicular to their direction of motion, causing their spin to precess. The
spin polarization of carriers changes by the angle $\Delta\Theta=\Delta
k_{R}\cdot L$, where L is the length of the 2DEG. Modulating $\Delta k_{R}$ –
conventionally via an electric field – switches the spinFET between a state of
high {0} and low {1} resistance. Figure 2:  TR-ARPES of surface-gated
topological insulators. a: Representation of the surface and bulk electronic
structure in a surface-gated topological insulator as a function of momentum,
energy, and distance from the surface (the side view displays the momentum
integrated projection of the band structure, where CB and VB are the
conduction and valence bands). Two-dimensional electron gases (2DEGs) taking
the form of spatially confined quantum well states (QWSs) are created by a
sufficiently large positive bias applied to the surface. The dispersion,
Rashba-splitting, and spatial extent of the 2DEGs depend on the detailed shape
of the band bending. Here, the band bending pushes the two lowest QWSs (blue
and green) below the Fermi energy. b: TR-ARPES experiment on p-type Bi2Se3;
the cleaved sample is gated in situ by alkali atom deposition. A near-infrared
(1.55 eV) ”pump” pulse perturbs the system, and a UV (6.2 eV) pulse is used to
probe the electronic structure by ARPES. The time delay ($\Delta t$) between
pump and probe pulses is varied to resolve the electron dynamics. c: Temporal
evolution of the QWSs in p-doped Bi2Se3 plotted relative to the electron
quasi-Fermi level EFn. The left panel shows the ARPES spectra of all surface
states before pump arrival (-100 ps); in the center panel photoemission
intensity integrated around the Brillouin zone center (black dashed lines) is
shown as a function of time (pump and probe are overlapped at time zero, red
dashed line). We observe that a second QWS emerges after the pump excitation;
the right panel shows the dispersion at 500 ps, characterized by two partially
populated QWSs. Figure 3: Ultrafast response of Rashba QWSs to optical
perturbation. a: Snapshots of the ARPES dispersion at negative time delay
(before pump arrival), at time zero (pump and probe fully overlapped), and at
8 ps. The latter two are accompanied by differential spectra obtained by
subtracting the spectrum acquired at around -0.5 ps; the blue (red) color is
indicative of a pump-induced decrease (increase) of photoemission intensity.
The strong red signal below the quasi-Fermi level at 8 ps reveals a change in
dispersion for the QWSs that does not appear in the TSS. b: Temporal evolution
of the energy minimum for QWS1 and QWS2 extracted from fitting the ARPES data
at $k_{\parallel}=0$. The fit to the experimental data is shown as a solid
black line and stems from the sum of two contributions (cyan and purple shaded
areas), each consisting of a finite rise-time step function and an exponential
decay. The positive contribution, in purple, is defined as a non-thermal (NT)
process, which is short-lived and acts to increase the QWSs’ energy; the
negative contribution, in cyan, is the result of a photovoltage (PV) effect,
which is an almost instantaneous effect that acts to decrease the QWSs energy
over long timescales c: Momentum distribution curves (MDC) profiles across the
right branch of QWS1 (horizontal dashed line in a) at EFn at equilibrium
(purple) and after 8 ps (blue) relative to the Fermi wave-vector of the inner
branch $k_{F1}$; the solid lines are Voigt fits to the data. The momentum
splitting $\Delta k_{R}$ is the distance between two peaks of the same MDC,
and it is dynamically reduced with the pump excitation from (26 $\pm$
0.5)$\times 10^{-3}$ (at $\Delta t<$ 0) to (22.5 $\pm$ 0.5)$\times 10^{-3}$
Å-1 (at $\Delta t$ = 8 ps). d: Energy distribution curves (EDC) profiles
across the inner and outer branch of QWS1 at $k_{\parallel}$ = 0.05 Å-1
(vertical dashed line in a) before pump arrival (purple) and after 8 ps
(blue); the solid lines are fits to the experimental curves using Voigt
profiles. The clear reduction of the energy splitting $\Delta E_{R}$ (similar
to the momentum splitting in b) is induced by the optical excitation. EDC and
MDC profiles in b and c have been deconvolved by the energy resolution via
Lucy-Richardson algorithm (Ref. 48) for better clarity. e: Temporal evolution
of the RSOC strength $\alpha_{R}$ in QWS1 calculated from Eq. 1 and extracted
by fitting the momentum (orange) and energy (green) splitting at several time
delay;. $\alpha_{R}$ is reduced by about 0.1 eVÅ at 8 ps with respect to
equilibrium. The values of $\Delta k_{R}$ at EFn are explicitly plotted
against the left axes, which shows a reduction of about 15% within the first 3
ps due to the optical excitation; the splitting does not change relevantly
thereafter in the investigated time interval. Figure 4: Simulations of the
band bending and quantum-well state dynamics. a: Band-bending profile for a
surface biased p-doped Bi2Se3; the surface boundary condition $V_{0}$ is given
by the shift of the Dirac point after potassium evaporation, and the bending
is calculated solving the Poisson equation within a modified Thomas-Fermi
approximation. The energy minima of QWS1 and QWS2 are solutions of the
Schrödinger equation within the confining potential, and well reproduce
experimental observations. At zero-delay, an optical excitation creates free
electron-hole pairs that are swept apart by the built-in electric field of the
space-charge region, ESCR. b: After 8 ps the charge separation between
electrons and holes generates a photovoltage whose electric field (EPV)
opposes ESCR and softens the band bending, causing the QWSs to shift upwards.
Concomitantly, the increasing surface electron density shifts the EFn upwards.
c: The calculated energy minima of QWS1 and QWS2 with respect to EFn are
obtained from a time-dependent calculation of the band bending and QWSs’
energy levels; the observed decrease at time zero is given by the change of
the electrostatic environment and carrier redistribution across SCR. d: The
simulated Rashba momentum splitting ($\Delta k_{R}$) at the Fermi level for
QWS1; the Rashba strength $\alpha_{R}$ is given on the right axes. The
photovoltage-induced reduction of the surface electric field in the system is
responsible for the decrease of $\alpha_{R}$ and $\Delta k_{R}$. Inset:
Simulated spectral functions of QWS1 before and 8 ps after the pump
excitation, constructed from the results of the dynamical simulation. The
simulated data confirms the two main observation from the TR-ARPES experiment,
an increase in electron population accompanied by a decrease in the spin-
splitting.


#  Transfer Learning Based Efficient Traffic Prediction with Limited Training
Data

Sajal Saha, Anwar Haque, and *Greg Sidebottom  Department of Computer Science
University of Western Ontario, London, ON, Canada
*Juniper Networks, Kanata, ON, Canada
Email:{ssaha59<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Efficient prediction of internet traffic is an essential part of Self
Organizing Network (SON) for ensuring proactive management. There are many
existing solutions for internet traffic prediction with higher accuracy using
deep learning. But designing individual predictive models for each service
provider in the network is challenging due to data heterogeneity, scarcity,
and abnormality. Moreover, the performance of the deep sequence model in
network traffic prediction with limited training data has not been studied
extensively in the current works. In this paper, we investigated and evaluated
the performance of the deep transfer learning technique in traffic prediction
with inadequate historical data leveraging the knowledge of our pre-trained
model. First, we used a comparatively larger real-world traffic dataset for
source domain prediction based on five different deep sequence models:
Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), LSTM Encoder-
Decoder (LSTM_En_De), LSTM_En_De with Attention layer (LSTM_En_De_Atn), and
Gated Recurrent Unit (GRU). Then, two best-performing models, LSTM_En_De and
LSTM_En_De_Atn, from the source domain with an accuracy of 96.06% and 96.05%
are considered for the target domain prediction. Finally, four smaller traffic
datasets collected for four particular sources and destination pairs are used
in the target domain to compare the performance of the standard learning and
transfer learning in terms of accuracy and execution time. According to our
experimental result, transfer learning helps to reduce the execution time for
most cases, while the model’s accuracy is improved in transfer learning with a
larger training session.

###### Index Terms:

deep sequence model, internet traffic, IP traffic prediction, ISP, transfer
learning

##  I Introduction

Over the past decade, internet traffic growth has been accelerated by the
rapid and widespread development of network technologies such as the Internet
of Things (IoT), Industrial IoT (IIoT), 5G, and cloud computing. Also,
according to the Cisco Annual Internet Report [1], there is an indication of a
massive number of internet users by 2023 (approximately 5.3 billion). With the
advancement of network and communication technologies and the increment of
potential users, it is inevitable to design a more intelligent and self-
organizing network (SON) capable of adapting the dynamic behavior and taking
preemptive actions. The ISP (Internet Service Provider) network traffic is a
critical metric for assessing network load and performance, and achieving a
precise traffic prediction can be a suitable technique for proactive network
management [2]. Most network management operations, such as resource
allocation, short-term traffic scheduling or re-routing, long-term capacity
planning, network architecture, and network anomaly detection, need an
accurate traffic prediction tool. However, real-world internet traffic
prediction is a challenging area. The main reasons are as follows [3]:

####  I-1 Data heterogeneity

Generally, internet traffic is heterogeneous in the temporal dimension, and a
different graphical region can present different traffic patterns at different
time-period. Therefore, it is challenging to propose a generic prediction
model that can predict other traffic datasets with equal accuracy.

####  I-2 Anomalous data

Real-world internet traffic is sensitive to various external and internal
factors that produce non-stationary complex traffic patterns. Also, those
events can exhibit data points that are not in the range of the data
distribution, hindering the model generalization capability during the
deployment. So it is crucial to handle the outliers or anomalous data points
before using them to train the prediction model.

####  I-3 Data scarcity

For better prediction on the testing set, it is necessary to train the
prediction model with a massive dataset so that model can generalize the
pattern well. But in reality, it is challenging to manage a large historical
dataset to train the prediction model, which leads to poor performance.

Predicting internet traffic is commonly thought of as a time series
forecasting problem that can be tackled by using traditional statistical
techniques such as ARIMA [4], SARIMA, Holt-Winter [5], etc. But these models
cannot predict the non-linear component of the actual internet traffic. With
the advancement of artificial intelligence, machine learning[6] and deep
learning[7] based traffic prediction models achieve superior performance
improvement over the classical approaches. However, modern machine learning
and deep learning algorithms require a massive amount of historical data to
learn the general pattern in the traffic. Unfortunately, it is quite difficult
to ensure enough data for building a new model separately for different
geographical or network sectors. Also, it is unrealistic to design and deploy
an individual model for each network segment for a larger ISP as the process
is expensive and time-consuming.

Transfer learning [8] comes into play to solve the problem stated earlier by
using the knowledge from the existing prediction model to devise a new model
for another dataset. This is the key technology for the industrialization of
the deep learning solution. The transfer learning approach consists of three
methods: parameter transfer, domain adaption, and multi-task learning. The
parameter transfer technique uses the learned parameters from the source data
and uses this knowledge to build a model for the target data. This technique
can help in two ways: re-use knowledge for quick learning and model smaller
datasets. The main contributions of this work are:

  * •

Analyzing different deep sequence models in the source domain to identify the
best performing model for the target domain.

  * •

Investigating transfer learning technique on the target domain consisting of
multiple datasets based on the best two models from the source domain.

  * •

Comparing the performance of standard learning and transfer learning in terms
of average accuracy and execution time in the target domain for comparatively
smaller datasets.

This paper is organized as follows. Section II describes the literature review
of current traffic prediction using machine learning models. Section III
presents the methodology, including dataset description, deep learning models
explanation, anomaly identification process, and experiment details. Section
IV summarizes the different deep learning methods’ performance and draws a
comparative picture among prediction models with and without outliers in the
dataset. Finally, section V concludes our paper and sheds light on future
research directions.

##  II Literature Review

Wu, Qiong, et al. [9] proposed a novel mobile traffic prediction framework
that combines the parameter-transfer [10] and domain adaption [11] approaches
from deep transfer learning to enhance the model performance with a smaller
dataset. The framework functionality is divided into two main parts: build the
target prediction model with a massive dataset and then use the pre-trained
model knowledge from the source domain, which faces the data-scarcity problem.
Furthermore, they applied a GAN-based approach to solving the domain shift
problem due to different data distribution between source and target domain.
According to their experiment, the GAN-based domain adaption helps their model
leverage the knowledge from the source domain to the target domain, giving a
better prediction for a smaller dataset. However, the quality of the generated
data using GAN is not presented. Moreover, since GAN is suffering to reach a
stable training point [12], it is unclear how their GAN performs in generating
samples.

Li, Ning, et al. [13] proposed a satellite traffic prediction model based on
Gated Recurrent Unit (GRU) architecture that uses the transfer learning and
particle filter algorithm for better prediction with a smaller dataset and
lower training time. According to their experimental environment, they used
similar distribution for the source and target domain in transfer learning. It
is unclear how their prediction model will perform if the source and target
distribution are asymmetric. The distribution is unlikely to be similar for
the source and target domain in the real world, and that’s why it is crucial
to validate the model performance with data coming from a different
distribution than that source domain. A wireless cellular traffic prediction
model has been proposed by Zeng, Qingtian, et al. [14], which is trained based
on a cross-domain dataset. They also used the already trained model’s
parameters for the target domain by adjusting the parameter’s values or
transferring the learned features to improve the model accuracy. The
experimental results showed the outperformance of the model with the transfer
learning capability than the model having no transfer learning. In [15],
Dridi, Aicha, et al. proposed a transfer-learning based deep learning model
for time series classification and prediction. The transfer-learning technique
is adapted in their model mainly for two reasons: better prediction with a
smaller dataset and re-adaption of the already trained model for another
domain. Their experimental results showed an outperformance of transfer
learning in time-series prediction. However, their source and target domain
data are drawn from the same distribution, which is unlikely to happen in the
real world.

Huang, Yunjie, et al. [16] proposed a spatial-temporal traffic prediction
model has been proposed with transfer learning capability to predict urban
traffic from small to medium-sized cities. They applied graph neural networks
to model the traffic network’s spatial information, while the temporal
information was modeled using a deep sequence model called GRU. Two different
datasets have been used as a source and a target domain in the experiment. The
experimental results showed the advantage of transfer learning in predicting a
smaller dataset. In [3], the authors proposed an intelligent cellular traffic
prediction model based on a graph convolutional network with an attention
mechanism. The temporal component of the traffic data has been captured using
a convolutional neural network (CNN). They also used the power of transfer
learning for better prediction on a large scale. Their experimental results
showed the effectiveness of transfer learning in reducing the training cost
and re-using the knowledge from the already trained model.

The current works show the usefulness of the transfer learning method in
traffic prediction. But we found a lack of investigation in the performance
comparison of the deep sequence model such as RNN and its varieties in real-
world traffic prediction based on deep transfer learning. In this work, a
comprehensive analysis of different deep sequence models has been performed
for networks with limited training data based on transfer learning. In
addition, we evaluate the performance using five different real-world internet
traffic datasets.

Figure 1: High level presentation of methodology

##  III Methodology

In this section, we first introduce the real IP traffic dataset and the
proprocessing steps used in our experiment in subsection III-A. Then, we
describe deep transfer learning technique in subsection III-B. After that, we
explain deep learning model background in subsection III-C. The model
performance evaluation metrics are described in subsection III-D. Finally, we
summarize the configuration of our experimental environment in subsection
III-E.

###  III-A Dataset and Preprocessing Steps

Real internet traffic telemetry on several high-speed interfaces has been used
for this experiment. The data are collected every five minutes for a recent
thirty days time period. A total of five different datasets are used in our
experiment. The source domain dataset, Dataset A, consists of 8563 data
samples and it is used to build the predictive model for the source task. The
other four datasets, Dataset B, C, D, and E are comparatively smaller in size,
having 363, 369, 358, and 365 data instances, respectively. We used these four
datasets for the predictive task in target domain.

####  III-A1 Data Windowing

Time series data need to be expressed in the proper format for supervised
learning. Generally, the time-series data consists of several tuples (time,
value), which is inappropriate for feeding them into the machine learning
model. So, we restructured our original time series data using the sliding
window technique. The sliding window technique is illustrated in Fig. 2. For
example, three historical points ($X$) called features are used to predict the
next data point ($y$) known as a target in the given sliding window example.

Figure 2: Data windowing

###  III-B Deep Transfer Learning

Transfer learning is an deep learning or machine learning optimization
approach in which knowledge is transferred from one domain to another similar
domain. We can formally define transfer learning in terms in terms of domain
and task [17]. There are two domains such as source and target domain involved
in transfer learning while the domain consists of a feature space $X$ and
probability distribution $P(X)$ where $X=\\{x_{1},x_{2},...,x_{n}\\}\in X$.
The task $T$ in transfer learning is consists of label space $Y$ and an
objective function $f:X\longrightarrow Y$ while $X$ is a feature space for
particular domain, $\\{X,P(X)\\}$. In transfer learning, the source domain
$D_{S}$ and target domain $D_{T}$ consists of two different task $T_{S}$ and
$T_{T}$ and the purpose of the transfer learning is to assist the task in
target domain to perform better using the knowledge in $D_{S}$ and $T_{S}$.

Figure 3: Actual traffic vs predicted traffic in source domain using
LSTM_En_De model

###  III-C Predictive Models

####  III-C1 Recurrent Neural Network (RNN)

At each step, the RNN produces the current output using the current sequence
information and the prior sequence information. In the final stage, the model
learns knowledge about all previous data points in the series. Text mining,
audio classification, language modelling, time series analysis, and other
sequential data applications benefit from the RNN paradigm.

####  III-C2 Gated Recurrent Unit (GRU)

The GRU has been proposed to tackle the RNN model’s short-time memory problem.
In this architecture, the gate idea is employed to govern the flow of
information between two adjacent cells. The update gate in the GRU model
determines whether or not the previous cell output should be transferred to
the next cell. A gate is a mathematical unit that can decide the relevance of
data and whether it should be saved or not. The GRU model has two gates that
function on updating the cell state: update gate and reset gate.

####  III-C3 Long Short-Term Memory (LSTM)

The LSTM serves a similar purpose to the GRU model. Along with the update and
reset gates, there are two more gates named forget and output gate. The LSTM
has more control over information flow between network cells. The LSTM network
is widely used for classifying and predicting time-series data. It solves the
classic RNN model’s vanishing gradient problem and outperforms it in terms of
performance.

####  III-C4 LSTM Encoder-Decoder (LSTM_En_De)

The sequence-to-sequence model is a model that can predict an output sequence
from an input sequence. It is made up of two recurrent neural networks, one of
which is an encoder and the other a decoder. The encoder turns the input
sequence into a fixed-length context vector, which the decoder receives. The
decoder takes the context vector and the encoder’s end state as input and
produces a series of outputs.

####  III-C5 LSTM Encoder-Decoder with Attention Layer (LSTM_EN_DE_Atn)

This model, also known as the sequence-to-sequence model, may predict an
output sequence given an input sequence. However, the encoder-decoder model
cannot identify significant contextual relationships from extended sequence
data, lowering model performance and accuracy. The encoder-decoder model’s
extra attention layer[18], on the other hand, can detect relevance in the long
sequence data.

###  III-D Evaluation Metrics

We used Mean Absolute Percentage Error (MAPE) to estimate the performance of
our traffic forecasting models. The performance metric identifies the
deviation of the predicted result from the original data. For example, MAPE
error represents the average percentage of fluctuation between the actual
value and predicted value. Therefore, we can define our performance metric
mathematically as follow:

$MAPE=\dfrac{1}{n}\sum_{i=1}^{n}\bigg{|}\dfrac{p_{i}-o_{i}}{o_{i}}\bigg{|}\times
100\%$ (1) $Accuracy=(100-MAPE)\%$ (2)

Here, $p_{i}$ and $o_{i}$ are predicted and original value respectively and
$n$ is the total number of test instance.

###  III-E Software and Hardware Preliminaries

We used Python and deep learning library TensorFlow-Keras[19] to conduct the
experiments. Our computer has the configuration of Intel (R) i5-9500T
<EMAIL_ADDRESS>8GB memory, and a 64-bit Windows operating system.

##  IV Results and Discussion

###  IV-A Experimental setup

As we mentioned earlier, our main research objectives are to find out the
effectiveness of deep transfer learning in real-world traffic prediction with
limited data and high volatility. To conduct this experiment, we used five
different internet traffic datasets extracted from real-world telemetry. But
all the datasets are not equivalent according to the data volume; four of them
have a smaller number of data instances compared to the other. We used dataset
$A$ to build our source traffic prediction model to transfer knowledge to the
target domain. The other datasets from $B$ to $E$ are considered for the
target domain as they have a smaller number of historical data for learning.
So, our experiment has two main parts: i.) design source prediction model on
dataset $A$ ii) transfer the source domain knowledge to design predictive
models for target domains based on dataset $A$, dataset $B$, dataset $C$, and
dataset $D$ and compare the performance with the standard learning to
determine the effectiveness of deep transfer learning in real-world traffic
prediction.

TABLE I: Performance summary for all model in source domain Model | Accuracy (%)
---|---
RNN | 92.49
LSTM | 94.97
GRU | 93.51
LSTM_En_De | 96.06
LSTM_En_De_Atn | 96.05
TABLE II: Results summary in target domain using LSTM Encoder Decoder Model | Dataset B | Dataset C | Dataset D | Dataset E
---|---|---|---|---
|  | Standard
---
Learning
| Transfer
---
Learning
| Standard
---
Learning
| Transfer
---
Learning
| Standard
---
Learning
| Transfer
---
Learning
| Standard
---
Learning
| Transfer
---
Learning
Epoch |  | Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
| Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
| Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
| Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
| Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
| Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
| Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
| Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
250 | 83.50 | 33.15 | 84.08 | 27.03 | 70.94 | 33.44 | 76.77 | 27.44 | 73.54 | 37.99 | 76.24 | 26.94 | 82.68 | 42.81 | 84.43 | 27.75
200 | 82.72 | 26.48 | 84.26 | 23.11 | 72.88 | 27.38 | 76.97 | 23.39 | 74.32 | 31.40 | 78.43 | 23.41 | 82.95 | 35.76 | 84.13 | 23.65
150 | 84.19 | 21.28 | 84.36 | 19.27 | 77.24 | 22.10 | 77.00 | 19.48 | 74.59 | 24.59 | 80.25 | 19.13 | 84.17 | 27.53 | 84.39 | 19.46
100 | 83.37 | 15.24 | 82.41 | 14.97 | 77.25 | 15.60 | 75.60 | 15.09 | 68.89 | 17.74 | 82.91 | 14.84 | 84.38 | 18.94 | 82.13 | 14.97
50 | 86.42 | 9.52 | 81.06 | 10.68 | 77.47 | 10.01 | 73.17 | 10.87 | 79.42 | 10.90 | 79.39 | 10.71 | 84.58 | 11.60 | 80.91 | 10.72
TABLE III: Results summary in target domain using LSTM Encoder Decoder with Attention layer Model | Dataset B | Dataset C | Dataset D | Dataset E
---|---|---|---|---
|  | Standard
---
Learning
| Transfer
---
Learning
| Standard
---
Learning
| Transfer
---
Learning
| Standard
---
Learning
| Transfer
---
Learning
| Standard
---
Learning
| Transfer
---
Learning
Epoch |  | Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
| Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
| Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
| Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
| Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
| Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
| Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
| Avg.
---
Acc.
(%)
| Avg.
---
Time
(S)
250 | 83.55 | 22.51 | 83.53 | 23.82 | 74.10 | 25.14 | 75.19 | 25.22 | 71.91 | 28.23 | 77.71 | 24.45 | 82.42 | 28.37 | 83.76 | 24.17
200 | 83.44 | 18.07 | 83.68 | 20.11 | 72.07 | 20.62 | 74.78 | 20.55 | 75.01 | 23.16 | 81.70 | 19.94 | 84.77 | 22.96 | 83.62 | 20.07
150 | 84.98 | 15.06 | 83.77 | 15.70 | 75.42 | 16.16 | 75.79 | 16.54 | 72.61 | 18.43 | 83.57 | 15.95 | 84.36 | 19.12 | 83.49 | 15.70
100 | 83.72 | 10.83 | 80.85 | 11.53 | 76.46 | 11.73 | 74.96 | 12.14 | 70.32 | 13.35 | 84.29 | 11.70 | 85.02 | 13.45 | 80.46 | 11.61
50 | 86.80 | 7.24 | 78.80 | 7.23 | 77.43 | 7.73 | 72.93 | 7.78 | 79.07 | 8.54 | 81.43 | 7.06 | 84.91 | 8.64 | 78.54 | 7.19

###  IV-B Result analysis

####  IV-B1 Source domain prediction model

We applied several deep sequence models such as RNN and their variants in this
phase, including LSTM, LSTM En_De, LSTM En_De_Atn, and GRU, to identify the
best performing models for our source domain. All our model training continued
for 100 epochs with a batch size of 16. The prediction accuracy for all the
prediction models on dataset $A$ is summarized in Table I. According to our
experimental results, RNN is the worst performing model with an accuracy of
92.49%, which is improved by more than 2% and 1% after applying LSTM and GRU,
respectively. Since LSTM and GRU can retain the information from the longer
sequence, it is expected to have a better performance than RNN for final
prediction. Finally, we achieved our best prediction result using LSTM_En_De
with an accuracy of 96.06%, which is around a 4% improvement compared to the
RNN model. The LSTM_En_De_Atn also gave us around 4% higher accuracy (96.0%)
when compared with the RNN model, which is very close to the LSTM En_De model
performance. Fig. 3 depicts the actual traffic vs. predicted traffic by the
best-performing model in the source domain. According to our experiment, the
encoder-decoder architecture-based models performed better, and we considered
our two best-performing models in the second step of our investigation. We
used two source domain model-based LSTM En_De and LSTM En_De_Atn to compare
and validate the performance of deep transfer learning.

####  IV-B2 Target domain prediction model

In this phase, we design a predictive model for datasets $B$, $C$, $D$, and
$E$ using both standard learning and transfer learning approaches. For both
strategies, 70% of the data has been used for the training, while the
remaining 30% has been applied to test the model. Two best-performing models,
such as LSTM En_De and LSTM En_De_Atn from the source domain, are used to
design the target domain’s predictive models. Also, we trained the target
domain models for different epoch lengths to identify the best training
settings and make a comparative analysis between standard learning and
transfer learning in terms of total training time and accuracy. In the case of
transfer learning, we freeze the reused layers during the first ten epochs
with a more significant learning rate of 0.001, allowing the new layers to
learn reasonable weights. Then we unfreeze the reused layers and decrease the
learning rate to 0.0001 to avoid damaging the reused weights. Finally, to
calculate the average training time and average accuracy of the prediction in
the target domain, we executed the experiment five times and took the
corresponding metric’s average.

Figure 4: LSTM_En_De model accuracy comparison Figure 5: LSTM_En_De model
execution time comparison

The details results for four different data sets are presented in Table II and
Table III , respectively, for the LSTM En_De and LSTM En_De_Atn model. The
average accuracy (Avg. Acc.) and average total training time in second (Time
(S)) indicates the mean accuracy and mean training time for five different
runs. Fig. 4 and Fig. 5 depicted the comparative analysis of the average
accuracy and average training time between standard learning and transfer
learning for LSTM_En_De model. We presented the comparative analysis of the
average accuracy and average training time between standard learning and
transfer learning for LSTM_En_De_Atn model in Fig. 6 and Fig. 7. For the LSTM
En_De model, the model training time is much lesser in transfer learning than
standard learning for all target domain’s datasets. The training time gap
between the two learning strategies is increased with the epoch size, which
indicates that transfer learning has the chance of quick learning in case of
longer training. Also, the graphs indicate better accuracy with transfer
learning for the first three epochs settings, such as 250, 200, and 150, for
all four datasets. Our results indicate a similar pattern in the LSTM
En_De_Atn model, although the gap between average time is smaller than in the
LSTM En_De model. Transfer learning gave us better training time or better
prediction accuracy for all settings and performed better for more significant
epochs in both metrics.

Figure 6: LSTM_En_De_Atn model accuracy comparison Figure 7: LSTM_En_De_Atn
model execution time comparison

##  V Conclusion

In this work, we evaluate the effectiveness of the deep transfer learning
technique in real-world internet traffic prediction for the network with
smaller training data. It is practically challenging to arrange a large
dataset for efficient model training. Furthermore, designing an individual
model for each particular host in the large network is infeasible due to time
and resource complexity. Therefore, a deep transfer learning based traffic
prediction methodology is proposed that is expected to provide better
performance with a comparatively smaller dataset. Our experiment used one
large dataset for the source domain and four smaller datasets for the target
domain. A total of five different deep learning models such as RNN, LSTM,
LSTM_En_De, LSTM_En_De_Atn, and GRU were implemented to analyze our source
domain traffic. Then we transferred knowledge from the best two prediction
models (LSTM_En_De and LSTM_En_De_Atn) in the source domain to enhance the
prediction task’s performance in the target domain. As a result, the target
domain learning became significantly faster with the transfer learning
approach than the standard learning for most of the target domain’s datasets.
Also, transfer learning improves the model accuracy when it is trained for the
larger epoch. According to our experiment, deep transfer learning is an
efficient approach to predict the non-stationary and non-linear real-world
internet traffic even with a smaller size by re-using the knowledge. In the
future, we would like to extend this work by adding the multi-step prediction
to evaluate transfer learning performance for both single-step and multi-step
forecasting.

## References

  * [1] “Cisco annual internet report a white paper 2020,” Mar 2020. [Online]. Available: https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.html
  * [2] Y. Han, Y. Jing, and K. Li, “Multi-step prediction for the network traffic based on echo state network optimized by quantum-behaved fruit fly optimization algorithm,” in _2017 29th Chinese Control And Decision Conference (CCDC)_. IEEE, 2017, pp. 2270–2274.
  * [3] X. Zhou, Y. Zhang, Z. Li, X. Wang, J. Zhao, and Z. Zhang, “Large-scale cellular traffic prediction based on graph convolutional networks with transfer learning,” _Neural Computing and Applications_ , pp. 1–11, 2022\.
  * [4] S. Wheelwright, S. Makridakis, and R. J. Hyndman, _Forecasting: methods and applications_. John Wiley & Sons, 1998\.
  * [5] G. E. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung, _Time series analysis: forecasting and control_. John Wiley & Sons, 2015.
  * [6] S. Saha, A. Haque, and G. Sidebottom, “Towards an ensemble regressor model for anomalous isp traffic prediction,” 2022, (in press).
  * [7] ——, “Deep sequence modeling for anomalous isp traffic prediction,” 2022, (in press).
  * [8] K. Weiss, T. M. Khoshgoftaar, and D. Wang, “A survey of transfer learning,” _Journal of Big data_ , vol. 3, no. 1, pp. 1–40, 2016.
  * [9] Q. Wu, K. He, X. Chen, S. Yu, and J. Zhang, “Deep transfer learning across cities for mobile traffic prediction,” _IEEE/ACM Transactions on Networking_ , 2021.
  * [10] S. J. Pan and Q. Yang, “A survey on transfer learning,” _IEEE Transactions on knowledge and data engineering_ , vol. 22, no. 10, pp. 1345–1359, 2009.
  * [11] M. Wang and W. Deng, “Deep visual domain adaptation: A survey,” _Neurocomputing_ , vol. 312, pp. 135–153, 2018.
  * [12] L. Weng, “From gan to wgan,” _arXiv preprint arXiv:1904.08994_ , 2019.
  * [13] N. Li, L. Hu, Z.-L. Deng, T. Su, and J.-W. Liu, “Research on gru neural network satellite traffic prediction based on transfer learning,” _Wireless Personal Communications_ , vol. 118, no. 1, pp. 815–827, 2021.
  * [14] Q. Zeng, Q. Sun, G. Chen, H. Duan, C. Li, and G. Song, “Traffic prediction of wireless cellular networks based on deep transfer learning and cross-domain data,” _IEEE access_ , vol. 8, pp. 172 387–172 397, 2020.
  * [15] A. Dridi, H. Afifi, H. Moungla, and C. Boucetta, “Transfer learning for classification and prediction of time series for next generation networks,” in _ICC 2021-IEEE International Conference on Communications_. IEEE, 2021, pp. 1–6.
  * [16] Y. Huang, X. Song, S. Zhang, and J. James, “Transfer learning in traffic prediction with graph neural networks,” in _2021 IEEE International Intelligent Transportation Systems Conference (ITSC)_. IEEE, 2021, pp. 3732–3737.
  * [17] Y.-P. Lin and T.-P. Jung, “Improving eeg-based emotion classification using conditional transfer learning,” _Frontiers in human neuroscience_ , vol. 11, p. 334, 2017.
  * [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [19] F. Chollet _et al._ , “Keras,” https://keras.io, 2015.