#  RuO2: a puzzle to be solved

Andriy Smolyanyuk<EMAIL_ADDRESS>Institute of Solid State
Physics, TU Wien, 1040 Vienna, Austria  Igor I. Mazin<EMAIL_ADDRESS>George
Mason University, Department of Physics & Astronomy and Center for Quantum
Science and Engineering, Fairfax, USA  Laura Garcia-Gassull  Institut für
Theoretische Physik, Goethe-Universität Frankfurt, 60438 Frankfurt am Main,
Germany  Roser Valentí  Institut für Theoretische Physik, Goethe-Universität
Frankfurt, 60438 Frankfurt am Main, Germany

###### Abstract

Altermagnetism is a topic that has lately been gaining attention and the RuO2
compound is among one of the most studied altermagnetic candidates. However,
the survey of available literature on RuO2 properties suggests that there is
no consensus about the magnetism of this material. By performing density
functional theory calculations, we show that the electronic properties of
stoichiometric RuO2 are described in terms of a smaller Hubbard $U$ within
DFT+$U$ than the value required to have magnetism. We further argue that Ru
vacancies can actually aid the formation of a magnetic state in RuO2. This in
turn suggests that a characterization of the amount of Ru vacancies in
experimental samples might help the resolution of the controversy between the
different experimental results.

##  I Introduction

In recent years the topic of altermagnetism has been gaining attention, with
significant efforts directed towards finding new altermagnetic materials [1,
2]. Altermagnetism is defined as a magnetic phase with symmetry-driven
compensated net magnetization, where the symmetry operation responsible for
this magnetic phase is neither inversion nor translation. A material
exhibiting these properties combines characteristics of both ferromagnetism
and antiferromagnetism. Furthermore, in regards to the electronic band
structure, the bands in this phase are non-spin-degenerate, leading to
intriguing applications.

Among the various proposed materials as altermagnetic candidates, RuO2 is
attracting much attention. However, the magnetism in this system is still in
itself a controversial topic. On the one hand, the absence of a discernible
phase transition in the heat capacity [3, 4] and the resistivity data suggests
that RuO2 is a Pauli paramagnet [5, 6, 7, 8, 9]. On the other hand, the
existence of an antiferromagnetic configuration has been reported by resonant
X-ray scattering [10] and neutron diffraction [11]. However, the latter
measurements reported a rather small local magnetization value (0.05
$\mu_{B}$). Additionally, there have been observations of a sizeable anomalous
Hall effect, consistent with a considerably larger magnetization [12, 13].

Unfortunately, the available neutron diffraction data on RuO2 [11] are not
sufficient to confidently resolve the controversy on the magnetization, for
the reasons described below. The main issue is that the quality of the
magnetic component of the fit in these experiments depends on the quality of
the structural refinement. In reference [11], the authors mention the
possibility of a structural distortion in the rutile phase accompanied by
antiferromagnetic order. However, Ref. [11] was unable to find a distorted
structure that would fit both unpolarized and polarized neutron diffraction
data, while the powder X-ray diffraction patterns are consistent with the
undistorted rutile structure (see the crystal structure depicted in Fig. 1).
To address this problem, Ref. [11] employed density functional theory (DFT)
calculations in an attempt to find such a structure. A distorted $2\times
2\times 2$ rutile supercell was optimized in both the non-magnetic and
antiferromagnetic states, and the rutile structure was obtained as the ground
state. The available computational data on the lattice dynamics in RuO2 [14,
15] confirm that the rutile structure is dynamically stable.

The absence of a structural phase transition is indirectly confirmed by other
measurements. Two independent electron transport measurements, one conducted
up to 300 K [8] and one up to 1000 K [9], show no changes in resistivity that
could be caused by a structural phase transition. Both data sets are well
described by a model that has three contributions to the resistivity: the
electron-phonon interaction with acoustic (Bloch-Grünesien) and optical modes,
along with a term arising from electron-electron scattering. Moreover, there
is no indication of a structural phase transition in the heat capacity
measurements, which was measured up to $\sim$340 K [3] and $\sim$1050 K [4].
This same conclusion is supported by the available measurements of thermal
expansion [16].

Based on the refinement using the rutile structure, the extracted magnetic
moment per Ru atom is 0.23 $\mu_{B}$ for unpolarized neutron diffraction and
0.05 $\mu_{B}$ for polarized neutron diffraction measurements [11].
Furthermore, there is no evidence of a phase transition to an
antiferromagnetic phase, neither in the susceptibility data of Ref. 11 nor in
earlier measurements [5, 6, 7]. Additionally, nuclear magnetic resonance
measurements strongly suggest the absence of long-range magnetic order. This
conclusion is supported by the absence of any contribution from Ru d electrons
in both the Knight shift and relaxation rate, as well as the absence of any
hyperfine splitting [17]. The authors of this paper point out that, overall,
the resonant magnetic properties closely resemble those of nonmagnetic Ru
metal.

Figure 1: Crystal structure of RuO2: Ru atoms are shown in red and blue
(different colors denote different spin orientations), O atoms are shown in
teal.

The controversy among the different experiments suggests that the existence of
antiferromagnetic (and hence altermagnetic) order in RuO2 is rather fragile,
likely sample-dependent, and possibly present in only a fraction of the sample
volume. In order to gain a better microscopic understanding of the magnetism
(or lack thereof) in this material, we have systematically investigated the
magnetic states of RuO2 employing density functional theory (DFT), both with
and without a Hubbard $U$ correction applied to the Ru $d$-orbitals. Our
tentative conclusion is that the perfectly ordered, stoichiometric RuO2 is
likely nonmagnetic, consistent with numerous experiments above. On top of
that, a modest hole doping, for instance, by creating Ru vacancies (a common
defect in this class of materials, cf. Ref. [18] that found 5% vacancies in
their RuO2 samples) promotes the RuO2 to a magnetic state of exactly the same
symmetry as suggested in Ref. 11 and utilized in Refs. [19, 20].

The amount of Ru vacancies is liable to vary from sample to sample, and even
from one batch to another, depending on the growth procedure, and may even be
nonuniform over a sample. This could explain the discrepancy between different
experiments and leads us to conclude that a characterization of the Ru
vacancies in the samples may be key to know about the magnetic character of
RuO2.

##  II Discussion

Figure 2: Total energy (left) and local magnetization at the Ru site (right)
as a function of $U_{eff}=U{-}J$ are explored in two sets of calculations. In
one set (“increasing $U_{eff}$”), denoted with $\times$ (energy results) and
$\Box$ symbols (magnetization results) the calculations were done starting
from $U=0$ eV and progressively increasing $U$ in each subsequent calculation.
In the other set (“decreasing $U_{eff}$”), denoted with $+$ and $\ast$
symbols, the direction of the calculation was reversed. The calculations are
without SOC contributions.

The first concern to cover is whether stoichiometric RuO2 is magnetic or not.
To account for the possible effects of electronic correlations in this system,
we perform DFT+$U$ computations. In Fig. 2, we plot the dependency of the
local magnetic moment at the Ru site for an anti-parallel spin orientation (a
parallel orientation, as well as various magnetic arrangements with $q\neq 0$,
are invariably higher in energy) over a range of $U$ values. As seen from the
overlap of the magnetic moment for small values of $U$ in the plot, RuO2 is
non-magnetic up to a critical value, $U_{eff}=U-J$ ${\sim}1.06$ eV. After
this, there is a discontinuous jump (see below the explanation) of the value
of the magnetic moment to ${\sim}0.5$ $\mu_{B}$. This jump is an order of
magnitude larger than the 0.05 $\mu_{B}$ obtained from polarized neutron
scattering measurements [11], and more than twice larger than the 0.23
$\mu_{B}$ value fitted to unpolarized data (claimed to be less reliable and
contaminated by unknown structural factors).

Moreover, $U_{eff}>1$ eV is rather large for this good-metallic, strongly-
hybridized, 4$d$ system. For comparison, first principles calculations of
$U_{eff}$ for the ruthenium-based spin-orbit Mott insulators $\alpha$-RuCl3,
RuBr3, and RuI3 gave estimates of 2 to 1 eV [21]. Considering the metallic
screening occurring in RuO2, it is expected that its $U_{eff}$ will be
noticeably smaller than the values given above. This leads us to conclude that
for stoichiometric RuO2 a smaller $U_{eff}$ is likely more realistic to
describe its properties than the required one to have magnetism, and therefore
stoichiometric RuO2 is most probably non-magnetic.

Figure 3: Projected non-magnetic density of states onto Ru $d$-orbitals: blue,
orange, cyan, red, and teal are used to depict $z^{2}$, $x^{2}{-}y^{2}$, $xy$,
$xz$ and $yz$ orbitals respectively. The coordinate system is aligned with
Ru-O bonds.

Moving on to analyzing the projected density of states (DOS) (see Fig. 3), we
observe that the main contribution to the DOS around the Fermi level comes
from the $xz$/$yz$ Ru $d$-orbitals. The value of the DOS at the Fermi level is
relatively low and flat in its vicinity. This causes the Stoner criterion for
ferromagnetism to be very hard to fulfill.

Zone-center antiferromagnetism, as in RuO2, obeys a modified Stoner criterion,
where instead of the uniform susceptibility, $\chi(\mathbf{q}=0)=N(0)$ at some
finite reciprocal lattice vectors appears, $\chi(\mathbf{G}\neq 0)$, but, it
is quite obvious that highly dispersive bands at the Fermi level and low DOS
are rather unfavorable in this case as well.

Another interesting aspect in the DOS is the narrow peak below the Fermi level
coming from the $xy$ Ru orbitals. If the Fermi level were shifted closer to
this peak, it could potentially trigger a magnetic transition from the current
non-magnetic state to a state with a significant magnetic moment.

Figure 4: Dependence of the local atomic magnetization $m$ on hole doping (an
undoped case corresponds to 56 electrons per cell, i.e., per two formula
units) and the effective Hubbard parameter $U_{eff}=U{-}J$. Isolines for
$m=0.05$, $0.10$, and $0.15$ $\mu_{B}$ are depicted by white lines (with no
attempts to smooth the plotted lines).

The shift of the Fermi level can be accomplished through hole doping, changing
the electron occupation. The only other alternative for generating a magnetic
order without doping is to increase $U$, and thus the effective Stoner
parameter $I_{eff}=I+(U-J)/5$ (see Ref. [22]) until the Stoner criterion
($I_{eff}>J$) is satisfied. To check this hypothesis we did a series of
DFT+$U$ calculations varying both the value of $U_{eff}$ and the number of
electrons. Figure 4 shows the value of the local magnetic moment at the Ru
site as a function of $U_{eff}$ and the number of electrons per unit cell (two
formula units). The isoline with $m=0.05$ $\mu_{B}$, corresponding to the
measured value from Ref. 11, is highlighted. One can see that there is a
rather stable ground state for $\sim 0.1$ hole/Ru doping, within a reasonable
range of $U_{eff}\lesssim 1$ eV. For the same $U_{eff}$, a larger doping of
0.4 hole/Ru, corresponding to 10% of Ru vacancies, generated a local magnetic
moment of $m=0.2$ $\mu_{B}$. It is worth pointing out that the discontinuous
jump in the calculated magnetic moment for the undoped compound as a function
of $U_{eff}$ is immediately understood from the DOS on Fig. 3. That is
because, in order to get a stable magnetic solution, the exchange splitting
(proportional to $U_{eff}$) must reach the threshold (around 1 eV, from Fig.
3) corresponding to the separation between the $xy$ band and the Fermi level.

Note that the data in Fig. 4 merely show a trend of the system to attain a
magnetic moment with hole doping, but the preferred magnetic orientation may
depend on the doping as well. In particular, when spin-orbit coupling (SOC) is
accounted for, our DFT+$U$ calculations with $U_{eff}$=1.4 eV show that the
magnetic anisotropy energy $E_{100}-E_{001}$ changes linearly with doping and
there is a transition from the easy axis along the $c$ direction towards an
easy plane at around 0.2 hole/Ru. However, the full sampling of magnetic
ground states as a function of $U_{eff}$ and hole doping is out of the scope
of this work.

Figure 5: Magnetic anisotropy as function of the angle in degrees, when
rotating the Neel vector in the indicated planes.

The next issue is to focus on the magnetic ground state when the system is in
the regime where magnetization is permitted. To address this question, we
initially calculated, using VASP [23, 24, 25, 26], the energy difference
between the ferromagnetic (FM) and altermagnetic (AM) stoichiometric RuO2
configurations. The calculations were done without considering SOC effects
while setting the value of $U_{eff}$ to 1.3 and 1.4 eV and we analyzed the
magnetization of both configurations. Interestingly, the AM configuration
converged to $M_{Ru}=0.66$ and 0.78 $\mu_{B}$ for each $Ru$ atom in the cell,
respectively. In contrast, the FM ones essentially collapsed, yielding a total
magnetization of $M_{tot}=0.015$ (0.038) $\mu_{B}$ per Ru atom.
Correspondingly, the AM energy was lower than the FM one by 3.3 (9.3) meV/Ru.
Altogether, these results indicate that the altermagnetic configuration has
the lowest energy. Besides that, the spin-spirals with $\vec{q}=(0,0,q)$ and
$\vec{q}=(q,0,0)$ were checked leading to $q=0$ as the lowest energy state in
both cases.

We also checked the calculated magnetic anisotropy and compared it with the
experiment. Including spin-orbit coupling, we found that the $c$ axis is the
easy axis along the direction, in agreement with experiment[11], as seen in
Fig. 5.

Thus, the magnetic ground state is characterized by an antiparallel alignment
along the $c$ axis of the magnetic moments of two Ru atoms. This description
can be described by the magnetic space group P4′2/mnm′ (BNS 136.499).

##  III Conclusions

Our DFT calculations show that, for a realistic value of $U_{eff}$ (using Ru
based insulators as reference), the stoichiometric RuO2 compound is non-
magnetic. However, hole doping due to Ru vacancies can induce a phase
transition to the antiferromagnetic phase even for small values of $U_{eff}$.
This observation may be a key to reconcile different, strongly mutually
contradicting experiments. If our conjecture is correct, every experimental
work on RuO2 must begin with a careful characterization of the O and Ru
content. Moreover, a systematic experimental investigation of magnetic
properties as a function of the O and Ru content is absolutely necessary. One
verifiable corollary is that with controllable Ru vacancies one should be able
to observe the antiferromagnetic transition in thermodynamics, transport and
magnetometry.

##  IV Computational Details

Computations were done using density functional theory (DFT) in the
generalized gradient approximation (GGA) with the Perdew-Burke-Ernzerhof [27,
28] functional as implemented in the VASP [23, 24, 25, 26] package employing
the projector augmented wave method (PAW) [29, 30], Ru_sv and O (or O_h, for
structural optimization) pseudopotentials were used. The energy cutoff was set
to 400 eV (for the test purpose, selected calculations were performed with a
900 eV cutoff) and the 8x8x12 (12x12x18 for testing) k-points Monkhorst-Pack
grid [31, 32] was used. For selected calculations, a cross-check using Wien2k
was used.

##  V Acknowledgments

We thank Libor Šmejkal and Huibo Cao for the discussions. A.S. was supported
by the Austrian Science Fund (FWF) through the project P33571 “BandITT”.
L.G-G. and R.V. were supported by the Deutsche Forschungsgemeinschaft (DFG,
German Research Foundation) for funding through TRR 288 – 422213477 (project
B05). I.M. was supported by the Army Research Office under Cooperative
Agreement Number W911NF- 22-2-0173. He also acknowledges Heraeus Foundation
for supporting his visits to University of Frankfurt. Some of the images in
the paper were created using VESTA software [33].

## References

  * Šmejkal  _et al._ [2022] L. Šmejkal, J. Sinova, and T. Jungwirth, Emerging research landscape of altermagnetism, Phys. Rev. X 12, 040501 (2022).
  * Mazin [2022] I. Mazin, Editorial: Altermagnetism—a new punch line of fundamental magnetism, Phys. Rev. X 12, 040002 (2022).
  * Cordfunke  _et al._ [1989] E. H. P. Cordfunke, R. J. M. Konings, E. F. Westrum, and R. Shaviv, The thermophysical and thermochemical properties of RuO${}_{\textrm{2}}$ from 0 to 1000 K, Journal of Physics and Chemistry of Solids 50, 429 (1989).
  * O’Neill and Nell [1997] H. S. C. O’Neill and J. Nell, Gibbs free energies of formation of RuO${}_{\textrm{2}}$, IrO${}_{\textrm{2}}$, and OsO${}_{\textrm{2}}$: A high-temperature electrochemical and calorimetric study, Geochimica et Cosmochimica Acta 61, 5279 (1997).
  * Guthrie and Bourland [1931] A. N. Guthrie and L. T. Bourland, Magnetic Susceptibilities and Ionic Moments in the Palladium and Platinum Groups, Physical Review 37, 303 (1931), publisher: American Physical Society.
  * Ryden and Lawson [1970] W. D. Ryden and A. W. Lawson, Magnetic Susceptibility of IrO${}_{\textrm{2}}$ and RuO${}_{\textrm{2}}$, The Journal of Chemical Physics 52, 6058 (1970).
  * Fletcher  _et al._ [1968] J. M. Fletcher, W. E. Gardner, B. F. Greenfield, M. J. Holdoway, and M. H. Rand, Magnetic and other studies of ruthenium dioxide and its hydrate, Journal of the Chemical Society A: Inorganic, Physical, Theoretical , 653 (1968), publisher: The Royal Society of Chemistry.
  * Lin  _et al._ [2004] J. J. Lin, S. M. Huang, Y. H. Lin, T. C. Lee, H. Liu, X. X. Zhang, R. S. Chen, and Y. S. Huang, Low temperature electrical transport properties of RuO${}_{\textrm{2}}$ and IrO${}_{\textrm{2}}$ single crystals, Journal of Physics: Condensed Matter 16, 8035 (2004).
  * Glassford and Chelikowsky [1994] K. M. Glassford and J. R. Chelikowsky, Electron transport properties in RuO${}_{\textrm{2}}$ rutile, Physical Review B 49, 7107 (1994), publisher: American Physical Society.
  * Zhu  _et al._ [2019] Z. H. Zhu, J. Strempfer, R. R. Rao, C. A. Occhialini, J. Pelliciari, Y. Choi, T. Kawaguchi, H. You, J. F. Mitchell, Y. Shao-Horn, and R. Comin, Anomalous Antiferromagnetism in Metallic RuO${}_{\textrm{2}}$ Determined by Resonant X-ray Scattering, Physical Review Letters 122, 017202 (2019), publisher: American Physical Society.
  * Berlijn  _et al._ [2017] T. Berlijn, P. C. Snijders, O. Delaire, H.-D. Zhou, T. A. Maier, H.-B. Cao, S.-X. Chi, M. Matsuda, Y. Wang, M. R. Koehler, P. R. C. Kent, and H. H. Weitering, Itinerant Antiferromagnetism in RuO${}_{\textrm{2}}$, Physical Review Letters 118, 077201 (2017), publisher: American Physical Society.
  * Feng  _et al._ [2022] Z. Feng, X. Zhou, L. Šmejkal, L. Wu, Z. Zhu, H. Guo, R. González-Hernández, X. Wang, H. Yan, P. Qin, X. Zhang, H. Wu, H. Chen, Z. Meng, L. Liu, Z. Xia, J. Sinova, T. Jungwirth, and Z. Liu, An anomalous Hall effect in altermagnetic ruthenium dioxide, Nature Electronics 5, 735 (2022), number: 11 Publisher: Nature Publishing Group.
  * Tschirner  _et al._ [2023] T. Tschirner, P. Keßler, R. D. G. Betancourt, T. Kotte, D. Kriegner, B. Buechner, J. Dufouleur, M. Kamp, V. Jovic, L. Smejkal, J. Sinova, R. Claessen, T. Jungwirth, S. Moser, H. Reichlova, and L. Veyrat, Saturation of the anomalous Hall effect at high magnetic fields in altermagnetic RuO2 (2023), arXiv:2309.00568 [cond-mat].
  * Bohnen  _et al._ [2007] K.-P. Bohnen, R. Heid, O. d. l. P. n. Seaman, B. Renker, P. Adelmann, and H. Schober, Lattice dynamics of RuO${}_{\textrm{2}}$: Theory and experiment, Physical Review B 75, 092301 (2007).
  * Uchida  _et al._ [2020] M. Uchida, T. Nomoto, M. Musashi, R. Arita, and M. Kawasaki, Superconductivity in Uniquely Strained RuO${}_{\textrm{2}}$ Films, Physical Review Letters 125, 147001 (2020), publisher: American Physical Society.
  * Touloukian  _et al._ [1977] Y. Touloukian, R. Kirby, E. Taylor, T. Lee, THERMOPHYSICAL, and E. P. I. A. C. L. IN,  _Thermophysical Properties of Matter - the TPRC Data Series. Volume 13. Thermal Expansion - Nonmetallic Solids_ (Defense Technical Information Center, 1977).
  * Mukuda  _et al._ [1999] H. Mukuda, K. Ishida, Y. Kitaoka, K. Asayama, R. Kanno, and M. Takano, Spin fluctuations in the ruthenium oxides RuO${}_{\textrm{2}}$, SrRuO${}_{\textrm{3}}$, CaRuO${}_{\textrm{3}}$ and Sr${}_{\textrm{2}}$RuO${}_{\textrm{4}}$ probed by Ru NMR, Physical Review B 60, 12279 (1999), publisher: American Physical Society.
  * Diulus  _et al._ [2021] J. T. Diulus, B. Tobler, J. Osterwalder, and Z. Novotny, Thermal oxidation of ru (0001) to RuO${}_{\textrm{2}}$ (110) studied with ambient pressure x-ray photoelectron spectroscopy, Journal of Physics D: Applied Physics 54, 244001 (2021).
  * Zhou  _et al._ [2023] X. Zhou, W. Feng, R. W. Zhang, L. Smejkal, J. Sinova, Y. Mokrousov, and Y. Yao, Crystal thermal transport in altermagnetic RuO${}_{\textrm{2}}$, (2023), arXiv preprint arXiv:2305.01410.
  * Šmejkal  _et al._ [2022] L. Šmejkal, A. Marmodoro, K. H. Ahn, R. Gonzalez-Hernandez, I. Turek, S. Mankovsky, and T. Jungwirth, Chiral magnons in altermagnetic RuO${}_{\textrm{2}}$, (2022), arXiv preprint arXiv:2211.13806.
  * Kaib  _et al._ [2022] D. A. S. Kaib, K. Riedl, A. Razpopov, Y. Li, S. Backes, I. I. Mazin, and R. Valentí, Electronic and magnetic properties of the RuX${}_{\textrm{3}}$ (X=Cl, Br, I) family: two siblings—and a cousin?, npj Quantum Materials 7, 75 (2022).
  * Petukhov  _et al._ [2003] A. G. Petukhov, I. I. Mazin, L. Chioncel, and A. I. Lichtenstein, Correlated metals and the $\mathrm{LDA+U}$ method, Phys. Rev. B 67, 153106 (2003).
  * Kresse and Hafner [1993] G. Kresse and J. Hafner, Ab initio molecular dynamics for liquid metals, Phys. Rev. B 47, 558 (1993), publisher: American Physical Society.
  * Kresse and Hafner [1994] G. Kresse and J. Hafner, Ab initio molecular-dynamics simulation of the liquid-metal–amorphous-semiconductor transition in germanium, Phys. Rev. B 49, 14251 (1994), publisher: American Physical Society.
  * Kresse and Furthmüller [1996a] G. Kresse and J. Furthmüller, Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set, Computational Materials Science 6, 15 (1996a).
  * Kresse and Furthmüller [1996b] G. Kresse and J. Furthmüller, Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set, Phys. Rev. B 54, 11169 (1996b), publisher: American Physical Society.
  * Perdew  _et al._ [1996] J. P. Perdew, K. Burke, and M. Ernzerhof, Generalized Gradient Approximation Made Simple, Physical Review Letters 77, 3865 (1996), publisher: American Physical Society.
  * Perdew  _et al._ [1997] J. P. Perdew, K. Burke, and M. Ernzerhof, Generalized Gradient Approximation Made Simple [Phys. Rev. Lett. 77, 3865 (1996)], Physical Review Letters 78, 1396 (1997), publisher: American Physical Society.
  * Blöchl [1994] P. E. Blöchl, Projector augmented-wave method, Physical Review B 50, 17953 (1994), publisher: American Physical Society.
  * Kresse and Joubert [1999] G. Kresse and D. Joubert, From ultrasoft pseudopotentials to the projector augmented-wave method, Physical Review B 59, 1758 (1999), publisher: American Physical Society.
  * Monkhorst and Pack [1976] H. J. Monkhorst and J. D. Pack, Special points for Brillouin-zone integrations, Physical Review B 13, 5188 (1976), publisher: American Physical Society.
  * Pack and Monkhorst [1977] J. D. Pack and H. J. Monkhorst, ”Special points for Brillouin-zone integrations”—a reply, Physical Review B 16, 1748 (1977), publisher: American Physical Society.
  * Momma and Izumi [2011] K. Momma and F. Izumi, VESTA3 for three-dimensional visualization of crystal, volumetric and morphology data, Journal of Applied Crystallography 44, 1272 (2011).


11institutetext: School of Computing, Newcastle University, Newcastle upon
Tyne, UK
<EMAIL_ADDRESS>
# Generating Synthetic Data for Real World Detection of DoS Attacks in the IoT

Luca Arnaboldi  Charles Morisset

###### Abstract

Denial of service attacks are especially pertinent to the internet of things
as devices have less computing power, memory and security mechanisms to defend
against them. The task of mitigating these attacks must therefore be
redirected from the device onto a network monitor. Network intrusion detection
systems can be used as an effective and efficient technique in internet of
things systems to offload computation from the devices and detect denial of
service attacks before they can cause harm. However the solution of
implementing a network intrusion detection system for internet of things
networks is not without challenges due to the variability of these systems and
specifically the difficulty in collecting data. We propose a model-hybrid
approach to model the scale of the internet of things system and effectively
train network intrusion detection systems. Through bespoke datasets generated
by the model, the IDS is able to predict a wide spectrum of real-world
attacks, and as demonstrated by an experiment construct more predictive
datasets at a fraction of the time of other more standard techniques.

##  1 Introduction

A Denial of Service (DoS) attack targets the availability of a device or
network [16], with the intent of disrupting system usability. The most common
method is referred to as Flooding DoS [16], and may be used as an attempt to
deplete the devices’ resources including memory, bandwidth and/or battery. A
DoS attack against an Internet of Things (IoT) network has the potential to be
significantly more detrimental than one against a standard network, this
increased vulnerability is due in part to the low computational power and
battery power characteristic of IoT devices [22].

The extant literature has delineated several potential approaches that may be
effective in the mitigation of a DoS attack [23]. They widely speaking fall
into two categories, host based (e.g. Client Puzzles) which puts the
computational effort on the device and network based (e.g. firewall) which
offloads the computational effort to a remote server or more powerful device
within the system. However many of these approaches may not scale well in the
IoT as computational power, heterogeneity and the large scale of these systems
are all limiting factors that deplete the available choices. One approach that
sidesteps many of these standard detriments is a Intrusion Detection System
(IDS) bespoke to the IoT system to protect. An IDS is a monitor placed on the
network that analyses incoming messages to detect attacks and/or unwanted
traffic. They are trained using system behaviour data and use these patterns
to make the detection.

Organizations and researchers alike have widely recognised the advantages of
adapting IDSs as the norm to monitor against DoS attacks on their systems
[17]. Standard approaches used to train IDSs include using a database of known
attacks (misuse detection) and testing systems to create a “benchmark”
behaviour and flag any anomaly as a potential attack (anomaly detection) [15].
Implementing an IDS within an IoT network however faces multiple challenges:
Firstly, it is usually challenging to establish a benchmark behaviour in
dynamic IoT systems as devices may constantly shift, new devices might join
and behaviours might change [10], which might prevent using anomaly detection;
Secondly, protocols can vary from one network to another, which necessitates
data collection to be bespoke to an individual system [11]; And thirdly, a
misuse detection can be time consuming to enforce, since collecting data
unique to a system and for each attack is time consuming [15] and some system
changes can require data (or part of the data) to be collected from scratch
(e.g. interactive smart homes where devices can change frequently).

To address the second and third challenges, we present a novel modelling
approach. In brief, our model is a Markov Decision Process (MDP), representing
the IoT network, the attackers, and some processes monitoring the security
metrics under consideration. A trace of the model (corresponding to a sequence
of actions of the MDP) should match a trace of the actual system, and vice
versa, such that it becomes possible to train a IDS for the actual system on
the traces of the model. The main strengths of our approach is the ability to
easily represent various configurations for the IoT network as well as
multiple types of attackers. MDPs have some key advantages: they have
substantial tool support such as PRISM Model Checker [13], they rely on
probabilities and non-determinism to recreate systems and they provide the
ability to find the optimum paths through the system using the reward
function. Through the reward function we create traces of behaviour that mimic
attacks on systems by assigning rewards to successful (damaging) behaviour.
Our results highlighted that through this methodology we were able to
consistently produce datasets that resulted in accurate IDSs (detecting
attacks on real world systems) and that could be trained in a fraction of the
time. The core contributions of this paper are 1) A model of an IoT system
that enables the generation of synthetic data sets of network behaviour 2)
Modelling of attack behaviour against a system to train a real world IDS 3) A
quantitative analysis and validation of this model against a real world
implementation of the same system to validate our methodology.

The paper is split into the following sections; In section 2 we discuss the
related work; In section 3 the problem overview is discussed; In section 4 we
introduce our IoT system model and attacks model that generates the network
behaviour; In section 5 we highlight our assessment methodology; In section 6
we discuss the setup for the experiment; Section 7 provides an analysis of our
results and section 8 concludes and discusses future work.

##  2 Related Work

###  2.1 DoS Attacks on IoT Systems

DoS attacks have long been one of the most common and dangerous threats in any
internet system. These attacks become even more dangerous as the IoT spreads
across a vast amount of spectra and parts of life including safety critical
and potentially life endangering ones such as IoT Healthcare and Intelligent
Transportation Systems.

The extant literature highlights several new DoS attacks against IoT system
taking advantage of unique qualities and IoT infrastructures [14, 19, 8]. One
such attack, battery drain attack focuses on exhausting the devices battery
power as replacing it might be costly, difficult and lead to extensive periods
of downtime. These kinds of attack are very subtle as the behaviour of the
attacker might not necessarily mimic more common attacks such as pure
flooding, they attempt to find battery intensive operations (not necessarily
malicious) and repeat them until the device is out of power. This is only one
specific example of the literature cited above, however, what all of the above
have in common is that they are specialised in their intent of disrupting IoT
devices and many of the current detection systems do not account for them
[19]. The literature highlights that there is a constant evolution of attacks,
as can be seen using resources such as ExploitDB [21]. When filtering for IoT
attacks we can observe that there is a huge increase in the spectrum of
attacks targeting these systems.

These upwards trends in combination with the expansion of the IoT across
various field makes a good argument for a simple way to observe the impact of
these attacks. A formalised model would allow for intuitive means to observe
and quantify these attacks as well as better defend these systems by
generating network behaviour bespoke to them.

###  2.2 Intrusion Detection Systems

The growing use of internet services in the past few years have facilitated an
increase in DoS attacks. Despite the best preventative measures, DoS attacks
have been successfully carried out against various companies and organizations
enforcing the need for better prevention/detection mechanisms. This is
partially due to the vast new avenues of attack (often unique to IoT) that
signature based schemes such as SNORT [18] struggle to detect. Further work
attempts a more scalable approach that models behaviour of a network
(stationary or non-stationary) and labels abnormal packets as a potential
anomaly [6]. Limitations of this approach are a large number of false
positives as well as lack of information regarding the attack (e.g. the
specific vulnerability the attack relies on) as opposed to a signature based
IDS that is able to tell you exactly what rule is broken.

The approach suggested in this paper allows for a mixture of these approaches
tackling the limitations of both works. By modelling behaviour of a system,
one can detect any anomaly similar to the second approach and by modelling
various attacks it can also provide accurate data of the system behaviour
whilst being targeted, allowing for less false positives. To predict “unknown”
attacks, the modelling approach uses a stochastic attacker that attempts
different behaviours allowed by the system policy. Using this data it can
create a wide range of attack signatures and simulate an attacker probing the
system.

###  2.3 Modelling IoT Systems

Several papers address modelling IoT, adopting various different approaches.
Fruth [9] examines various properties of a wireless network protocol namely
connectivity and energy power through PRISM, including quantifying the battery
drainage of certain randomized protocols. In previous work [4] we model basic
flooding DoS attacks through PRISM and look at the effectiveness of different
attack strains and mitigation techniques in defending systems of
interconnected IoT devices.

Our proposed method combines these approaches to recreate an accurate
representation of system behaviour and represent a wide range of DoS attacks.
PRISM has been widely used as a excellent method to evaluate and verify models
of IoT systems and protocols, combining these two models by adapting both the
system models and the attack models we successfully model the behaviour and
general properties of a bespoke IoT system. We then use the inbuilt
verification capabilities to ensure correctness relative to mimicking system
behaviour by establishing benchmarks and tests. PRISM and its inbuilt
simulation capabilities allows to simulate attacks against the verified model.

##  3 Formulation of Problem

An IDS is in essence an evaluator that can establish whether a set of network
packets entering the system is malicious or not, by using what it has
“learned” from previous data. In order to successfully train an IDS for a
bespoke system, a security professional needs to therefore collect large
quantities of data. The problem with this is that to gather this data there
are several options each with several drawbacks [15]: 1) make use of known
attack datasets to train the IDS 2) make use of existing IDS (e.g. Snort -
Lightweight Intrusion Detection for Networks [18]) or 3) make use of an
exploit database and simulate attacks on your own system as a pen testing
approach. This latter approach is by far the most precise [15, 7] as it allows
to search for bespoke attacks to the IoT network and construct a dataset which
is unique and effective for the specific system.

Whilst this approach produces the best suiting dataset it has some major
drawbacks. Firstly, one must find and implement the attacks, which is a
difficult process that might take a very long time[15]. Secondly, one would
need to cause major disruptions to one’s own network by running the attacks
which might obstruct work and productivity. One of the many difficulties in
detecting attacks on systems through the use of IDS is that one cannot
(easily) predict potential attacker behaviour, or rather it is very difficult
to classify an attack if its behavior differs from known attacks.

We formulate the problem as the following: Firstly, is it possible to overcome
some of these difficulties and train an IDS for a specific IoT system making
use of a model? The model would need to be able to produce similar results of
the third approach, but would have the advantage that it could run parallel to
the real system without causing downtime (Fig.1) . Secondly, by making use of
non-determinism and probabilistic behavior could the modelling approach
recreate behavior that mimics that of an attacker probing and finding
weaknesses in the system?

Figure 1: Running model along-side real system to generate further datasets

##  4 IoT System Model

The intent of the modelled system is in essence to produce traces of behavior
that correspond to the behaviour of real devices. A trace of a model under
attack should be a subset of the full (finite) model trace. The traces are
however limited by the drain of battery either by standard behavior or by
attacker behavior, as devices out of battery stop performing actions. This
means that going from a set of traces one can reconstruct a data file of what
has taken place in the system. The traces can be used by the IDS to observe
patterns of behavior and set out rules to use against real-world attackers.

We model the system as a synchronization of three core components: a set of
devices, a set of monitors (each assigned to a device) and an attacker. The
other aspect was measuring the impact of the actions on the system,
specifically their effect on the devices battery and ability to operate
successfully. Whilst several process calculi are available to represent traces
of processes we customize the trace semantics of standard process calculus, as
there are several further features we need to capture in order to be able to
produce descriptive datasets. Several process calculi achieve the notion of
communication however this achieves the effect of two processes performing an
action simultaneously, we wish to capture the effect of a device _sending_ an
action and the other device _receiving_ it. Looking at common process algebras
such as CSP by Hoare [12], the semantics of the traces are a set of action of
the processes. These kinds of traces allow for a human reader to understand
the way the system operates. However an IDS can draw very little information
from these traces and they fail to capture the concept of messages through the
network. Specifically we need to be able to capture the intercommunication
between devices at each transition within the trace (as per a log in a real
system). This meant that output needed to hold further information rather than
just the action taking place, the rules and the way the system was constructed
was built around making a descriptive dataset.

The system as a whole is a tuple $\Phi=(D,M,T)$ where $D=\\{D_{1},..,D_{n}\\}$
is a set of devices, $M=\\{M_{1},...,M_{n}\\}$ is a set of monitors
calculating properties of their corresponding device and
$T=\\{t_{i},...,t_{n}\\}$ is a set of times to calculate the changes over time
of the system as a consequence of actions being triggered. We also introduce
means to model an attacker as a malicious device.

###  4.1 Device Model

Given a global set of actions $\gamma$, a device $D$ is a pair $(A,P)$, where
$A\subseteq\gamma\times[0,1]$ is the set of active actions, where $(a,p)\in A$
means the process chooses action $a$ with probability $p$, and such that
$\sum\\{p\mid(a,p)\in A\\}=1$; and $P\subseteq\gamma$ is the set of passive
actions.

In order to recreate the full spectrum of potential system behaviours we model
the set of actions of the device as the full capabilities of the real world
device. This allows to capture the full set of abilities of its behavior and
increases the accuracy of the benchmarking. This also eases the addition of
further devices as they are simply modelled with the full send and receive
action spectrum without the need to alter the rest of the system. The behavior
of a device is in the form of a guarded communication, which in our model
means that the communication is reliant on a set of conditions being true in
order to be triggered. An action $a$ in the device can only be triggered to
begin a communication if it doesn’t violate the capabilities of the system,
such as remaining battery and time per message. This allows for realistic
device behaviour, mimicking the patterns and constrictions of a real system.

###  4.2 Monitor Model

A monitor is the part of the system that enables its correct functioning as
well as monitoring dangerous behavior. It calculates the shifts in battery of
the various actions and synchronises with the devices to ensure correctness. A
monitor $M$ controls value $\lambda$, where $\lambda$ is the remaining battery
of the device. Given a global set of battery drains $\Omega$ the $\lambda$ is
measured as a quantity that is linearly drained by a $\omega_{a}$ where
$\omega_{a}\in\Omega$ is a constant battery drain of an action, the monitor
will update its $\lambda$ value to $\lambda^{\prime}$ after each corresponding
device action. The drain of each action is a fixed value calculated from the
real world device, as such each action is associated to a single device only.

######  Rule 1.

Given two devices $D_{1}=(A_{1},P_{1})$ and $D_{2}=(A_{2},P_{2})$, a
communication initiated by device $D_{1}$ on an active action $a$, triggering
corresponding receive action $\bar{a}$ in $D_{2}$, with an associated
probability $p$ takes the form:

$\frac{(a,p)\in A_{1}\ \ \bar{a}\in P_{2}\ \
p>0}{(A_{1},P_{1})||(A_{2},P_{2})\xrightarrow{(a,p)}(A_{1},P_{1})||(A_{2},P_{2})}$

######  Rule 2.

Given monitors $M_{1}$ and $M_{2}$ holding battery values $\lambda_{1}$ and
$\lambda_{2}$, devices $D_{1}$ and $D_{2}$ are controlled by their respective
monitors. The monitors calculate the drain in battery caused by action $a$ and
$\bar{a}$ from constant drain values $\omega_{a}$ and $\omega_{\bar{a}}$ in
the form:

$\frac{D_{1}||D_{2}\xrightarrow{(a,p)}D_{1}||D_{2}\quad\lambda_{1}>\omega_{a}\quad\lambda_{2}>\omega_{\bar{a}}}{\lambda_{1}\vartriangleright
D_{1}||\lambda_{2}\vartriangleright
D_{2}\xrightarrow{(a,p)}{(\lambda_{1}-\omega_{a})\vartriangleright
D_{1}||(\lambda_{2}-\omega_{\bar{a}})\vartriangleright D_{2}}}$

These measurements can further aid the IDS in making informed decisions
regarding the impact of the various actions in the system and were used to
quantify the effectiveness of the attacker. Through this synthetic data the
IDS will get a wide range of attacker behaviour that will lead to system
failure, including potentially unknown attacker behaviour. Doing a similar
approach without the model would require attacking one’s own system and
implementing an attack to collect data as per a penetration test (these
approaches were compared in the experiments in section 5).

###  4.3 Traces of the system

We differentiate each transition as a network packet running through the
system, checked by the monitor of the device. Therefore they must be unique
and fit all the possible behaviours of the device. As each action belongs to a
single device it enables the corresponding devices to be uniquely identified.

######  Rule 3.

Given two devices controlled by their monitors in the form:
$M_{1}\vartriangleright D_{1}$ as $CD_{1}$ and $M_{2}\vartriangleright D_{2}$
as $CD_{2}$, and taking the total set of devices $X$, then the transition
between CD1, CD2, taking system time t and being performed with probability p
takes the form:

$\frac{CD_{1}||CD_{2}\xrightarrow{(a,p)}CD_{1}^{\prime}||CD_{2}^{\prime}\ \
}{(t,CD_{1}||CD_{2}||X)\xrightarrow{(a,p,t)}(t+(t_{a}+t_{\bar{a}}),CD_{1}^{\prime}||CD_{2}^{\prime}||X)}$

In the computational view we compose a trace of the system inductively as a
set of transitions in between states, where prefix is the prior transitions
and the diagram describes a single transition in the form:

$\overbrace{\\{\\}}^{\text{prefix}}$$\overbrace{\bullet}^{\text{state}}$
$\overbrace{[a,p,t^{\prime}]}^{\text{transition}}$
$\overbrace{\bullet}^{\text{state'}}$$\downarrow$$\Updownarrow$$\downarrow$$\begin{smallmatrix}M_{1}&D_{1}&T_{1}\\\
\vdots&\vdots&\vdots\\\
M_{n}&D_{n}&T_{n}\end{smallmatrix}$$\begin{smallmatrix}\exists\ D_{i}\ \ni\
b_{i}\ >\ \omega_{a}\\\ \ \wedge\ \ \exists\ D_{j}\ \ni\ b_{j}\ >\
\omega_{\bar{a}}\\\ \wedge\ (a,p)\ \in\ D_{i}\ \wedge\ \bar{a}\ \in\
D_{j}\end{smallmatrix}$$\begin{smallmatrix}M_{1}^{\prime}&T_{1}^{\prime}\\\
\vdots&\vdots\\\ M_{n}^{\prime}&T_{n}^{\prime}\end{smallmatrix}$
$\begin{smallmatrix}\forall\ k\ if\ k\in\\{D_{i},D_{j}\\}\\\
\lambda_{k}^{\prime}\mathrel{-}=\;(\omega_{a}+\omega_{\bar{a}})\ \\\
\wedge\;t^{\prime}\ \mathrel{+}=\ (t_{a}\;+\;t_{\bar{a}})\\\ if\
k\not\in\\{D_{i},D_{j}\\}\ M_{k}^{\prime}=M_{k}\end{smallmatrix}$

The output of the system is a set of transitions following the semantics
described. By generating the outputs of the system as the full behavior
spectrum, the model can describe everything that can take place in the system.
By updating the probabilities we can cater to the specifics of the underlying
system behavior and make use of this to find unusual or potentially malicious
behavior. The rules can expand to include a wide array of behaviours and
specifics to regulate devices actions and when they can be activated. These
can include complex policies on whether actions can be activated at a specific
time or whether some actions have higher priority allowing for very specific
behaviour to be modelled.

Running Example: We show an example composed of: devices $D_{x}$,$D_{y}$ and
$D_{z}$, corresponding monitors $M_{x}$,$M_{y}$ and $M_{z}$, and global time
$t$. Each device has different actions that are synchronized with some other
devices. The monitors have battery values for the devices and each device has
a set $\Omega_{i}\in\Omega$ of action drains. Transitions follow the described
rules to construct the traces. Note that they do not represent the full
possible set of traces but rather two simulations of the system until devices
are drained.

Table 1: Example system model and its outputs Devices : |  $\boldsymbol{D_{x}}=(A_{x},P_{x})$ where
---|---
|  $A_{x}=\\{(read_{xy},0.3),(write_{xy},0.5),(read_{xz},0.2)\\}$ and
$P_{x}=\\{\overline{read_{zx}}\\}$
|  $\boldsymbol{D_{y}}=(A_{y},P_{y})$ where
|  $A_{y}=\\{(write_{yz},0.8),(read_{yz},0.2)\\}$ and
$P_{y}=\\{\overline{read_{xy}},\overline{write_{xy}},\overline{read_{zy}}\\}$
|  $\boldsymbol{D_{z}}=(A_{z},P_{z})$ where
|  $A_{z}=\\{(read_{zx},0.1),(read_{zy},0.9)\\}$ and
$P_{z}=\\{\overline{read_{xz}},\overline{write_{yz}},\overline{read_{yz}}\\}$
Monitors |  $\boldsymbol{M_{x}}\ni\lambda_{x}=5$ and $\boldsymbol{Drains_{x}}$ $\ni$ $\Omega_{A_{x}}=(1,3,1)$ $\wedge$ $\Omega_{P_{x}}=(1)$
& Drains: |  $\boldsymbol{M_{y}}\ni\lambda_{y}=8$ and $\boldsymbol{Drains_{y}}$ $\ni$ $\Omega_{A_{y}}=(2,4)$ $\wedge$ $\Omega_{P_{y}}=(1,2,1)$
|  $\boldsymbol{M_{z}}\ni\lambda_{z}=2$ and $\boldsymbol{Drains_{z}}$ $\ni$
$\Omega_{A_{z}}=(1,1)$ $\wedge$ $\Omega_{P_{z}}=(1,2,1)$
Trace 1: |  $[write_{yz},.8,30]$ $[write_{xy},.5,50]$ $[read_{xy},.3,65]$
Trace 2: |  $[read_{xz},.2,8]$ $[read_{xz},.2,16]$ $[read_{xy},.3,31]$ $[read_{xy},.3,46]$ $[read_{xy},.3,61]$

###  4.4 Attacker Model

An attacker synchronizes with a subset of actions of the device. When an
attacker synchronizes on the device the monitor will synchronize on that
action and calculate the respective drainage. The monitor keeps track of all
these measurements for its respective device. Implementing the model in a tool
like PRISM allows us to make use of Probabilistic Computation Tree Logic
(PCTL) [5] to calculate various conditions of pertinence to the system, to
compute the optimal attack path, and to simulate traces of the model.

An attacker’s intent is to behave in a manner that shortens the traces of the
system by draining the value of battery in the monitor in the most efficient
way possible. To model the attacker we made use of non-deterministic behavior
in order to allow for anything to take place at any point. The advantage of
non-determinism is that it allows for a system to arrive to an outcome using
various routes. This can be manipulated to find optimal routes through the
system and simulate varied behaviour. Unlike devices that are restricted by
time and batteries of the devices they model, we allow for the attacker to
have different levels of power to simulate various attacker strengths. An
attacker, like the devices, has a set of unique actions $A_{A}$ a, however
unlike other devices does not have a set of passive actions as it sits outside
the connectivity of devices and cannot receive messages. An Attacker may
synchronise with any device in the system, and the set of actions
$a_{A_{i}}\in A_{A}$ each correspond to different types of attacks in the real
system. To expand further on the actions of the attacker, these should be very
flexible and we make allowance for any action that can take place in the
system (only restricted by the setup and protocols).

Each action label will correspond to an attack message from the real attacker
and can be converted for the log file. For our specific example, each action
in the attacker corresponds to the attacker in our experiment sending
different packets/targeting different parts of the system as per the attacker
experiment in section 5. Beyond actions it is important for us to be able to
monitor the behaviour of attackers looking at how many actions an attacking
device can perform at a time T (whether by assuming a real attacker device or
by simulating different powers of attack). This is highlighted by measurements
of the system we implemented that were then modelled in the monitor of each
device. The other information to keep track of is: the choices the attacker
makes to take down the devices, as these are important behavioural patterns
for the IDS to use and can give us insight on potential vulnerabilities as
well as unknown attacker behaviours.

Unlike with the devices (whose intention is to cover the full spectrum of
possible behaviours with the attacker), we are particularly interested in
targeted behaviour. The attack actions therefore encompasses behaviours which
are particularly damaging to the system (e.g. causes large drain of battery to
the devices). As opposed to probabilistic behaviour we use non-determinism to
find paths of behaviour that are particularly rewarding in terms of time taken
to take down the system and in terms on lowering system usability (e.g.
message throughput). To model non-determinism we remove the probabilities from
the attacker action. This differs from probabilistic behavior because the non-
deterministic choice between process $A$ and $D_{x}$ is resolved at the moment
the first action takes place. Conversely in the case of a probabilistic choice
is done before the actions takes place [3], so if there is a conflict in the
system where both probabilistic actions and non-deterministic actions exist
the probabilistic action is resolved first. By not associating a probability
to an action we allow for the strategy of the attacker device to vary
depending on what we are looking for in the system. Given a policy regulating
the behaviour (corresponding to the available attack types) we allow for any
action to take place at any point. This can be combined with a set of rules to
find the trace of behaviour that allows to follow all the rules and yet still
drain the battery as quickly as possible within these restrictions. Instead of
a probability each action has an associated reward, and one can use this to
find the path of most reward (or the best strategy to take down the system).

The non-determinism in combination with the reward structure $time$ is used to
find the optimum attacker strategy, or the most rewarding trace through the
system. In PCTL it is written as $R\\{``time"\\}min=?[\ \textrm{F}\;power=0\
]$ or the minimum time for the variable power (referring to battery levels) to
reach 0. The value “time” is a variable calculated by the time for a single
message to be sent by the attacker and cumulated for each message sent before
the power reaches zero calculated in microseconds and the power drain is
calculated by the formulas in section 5. These reward structures allow for
simulated attack strategies that an hypothetical attacker might make to take
down the modelled IoT system. Not all attacks rely on speed and intensity to
take down the system, as highlighted by the running example in 1 where the
longer trace (Trace 2) is faster, so we model different rewards and observe
different attacker behaviours. We can find generate traces of less detectable
attack by associating an predictability score to an action and therefore
keeping the behavior varied and realistic whilst still optimizing time. This
can scale to several scenarios. We use these “optimized” traces to create a
large dataset that mimics different kinds of attackers.

##  5 Experiment Methodology

To evaluate the effectiveness of the models we tested and compared the
modelled system in 4.1 with the more standard approach described previously.
Both the approaches output was used to train an IDS. The IDSs were then used
to predict attack behavior. The verification was on the following basis: 1)
Accuracy on unknown attack detection; 2) Ability to mimic devices behavior and
smart attackers. The setup of the experiment was the following:

Experiment - Device Setup :  We set up a small IoT network in the lab and then
modelled it to compare the results and to test out the effectiveness of our
model in creating synthetic dataset. For the sake of testing we kept the setup
simple to display the tool as the thing that needs to scale and not the
system. Once the simple model is created it is trivial to add more (similar)
devices, whilst implementing a new system in the real world can be very time
consuming. We implemented a sensor network consisting of two devices. Each
device had the following actions; they took sensor readings and then could
_send_ it to the other device at any time; they could also _request_ the
sensor data from the other device at any point. The devices used simple HTTP
protocol for communication, and the behaviour was stored in Apache log format.
To accurately represent the devices and to create smart attackers, several
measures needed to be obtained. Both devices were equipped with a Mh3500
battery. We made a basic assumption that the devices are on constantly. We
argue this is a correct assumption as due to our attack the device is
constantly in log mode and therefore never in sleep mode. Beyond this
assumption we calculated time to send a message/log a message, baseline
battery usage, percentage increase in battery usage under different DoS
strains (taken this value and dividing it by messages processed for second)
and battery drain per message.

Experiment - Attacker: To validate the model we implemented a common DoS
attack both in the real world and in the model. Our attack of choice was HULK,
a DoS attacking tool which relies on several obfuscation techniques. In order
to not be spotted whilst still outputting intense strain enough to take down
systems very quickly [2]. The attack specifies it has the following
properties: 1) obfuscation of source client - this is achieved by using a list
of known user agents, and for every request that is constructed, the user
agent is a random value out of the known list, 2) reference forgery - the
referrer that points at the request is obfuscated and points into either the
host itself or some major pre-listed websites, 3) stickiness - using some
standard Http command to try and ask the server to maintain open connections
by using Keep-Alive with variable time window and 4) unique transformation of
URL - to eliminate caching and other optimization tools, they crafted custom
parameter names and values and they are randomized and attached to each
request. The tool was able to take down a web server within minutes from just
a single host. Seeing as IoT devices will have less capabilities than any web
server we hypothesized that this would be a good attack to use as its
properties make for a good dataset that is not straightforward to detect.
These properties and obfuscations led to different combinations of message
structure that we used in the non-deterministic attacker.

To measure the time it takes per message we measure how many messages can be
sent within a time period. This helps evaluate the accuracy in respect to the
real world of our test attacker. In order to measure voltage usage across the
different IoT devices, we attached an extra component in between the battery
supply and the device to take the readings required. To measure battery
drainage we utilized IoT battery lifespan estimator tool by Farnell [1]. This
was used in combination with a variance we introduced on top of the
calculator, to represent attack intensity and change to current. Through this
we were able to estimate the different drains of the devices as an outcome of
the actions they performed. We created datasets utilising three approaches and
compared each dataset in two different experiments.

The first dataset (RWD) was constructed from data from the real system. We
implemented the system of devices and the real-world attack and monitored the
behavior of the system. The data was logged across a period of twelve hours
and used to train the first IDS. The second approach was a naive approach, we
constructed a synthetic dataset (ND) without attacking the system but rather
attempting random behavior. This gave a comparison of the model with a
different synthetic dataset this will help evaluate the effectiveness of the
IDS predictions as they effectively should be random guesses. And finally, we
followed our proposed approach (MD) following section 4.

###  5.1 Experiment 1

As our dataset relies on stochastic events and actions, we created three
datasets from the approach and evaluated each one to benchmark its
effectiveness, a mean score was taken. Whilst our model is able to recreate
very large datasets quickly we choose to keep the dataset size uniform across
the initial experiment to get a fair comparison against the other two
datasets. The comparison was based on accuracy of prediction against unknown
attacks given IDSs trained with each of the datasets. The unknown dataset
consisted of real world data of the systems behavior whilst being targeted by
attacks that we had not modelled nor contained in the RWD. To measure accuracy
we made use of the F score. The F score is a measure of a predictors accuracy,
it is a measure of its precision over recall (a measure which takes in
consideration both false positives and false negatives).

###  5.2 Experiment 2

The second experiment we ran was to test the effectiveness of the model in
creating large quantities of behavior and the ability to readjust in case of
network reconfiguration. We used deep learning classifiers catered to large
datasets and created a much more efficient IDS purely through synthetic data.
One of the core strengths of our approach is that once the model is setup the
datasets are very easy to generate and we wanted to test whether this, in
combination with our smart attackers, will lead to the ability to train better
performing IDS.

##  6 Experiment Setup

To perform experiments described in section 5 we implement a Python framework
that runs through the various steps required to test the IDSs: data
generation, data processing, standardization and setting up of the IDS’s
classifiers. This automatic framework prepares the datasets and trains the
IDSs so that we may perform Experiment 1 and 2. It is implemented using the
scikit-learn machine learning libraries.

###  6.1 Data Generation

Achieving a rich descriptive dataset was paramount in training an effective
IDS. Through the outputted model traces we were able to generate a dataset of
different transitions through the modelled system. These traces were
descriptive enough for a machine learning algorithms to construct rules about
negative behavior through supervised learning. The traces of the model
correspond to the real system behavior and each transition was labelled as
either normal or abnormal behavior, therefore they can be used to make
informed decisions about the system. For instance, if the model traces of the
attacker continuously target a device, the IDS can interpret this as a weak
point and set a rule to limit this behavior, as this could correspond to the
behavior of a real world attacker.

###  6.2 Data Processing & Standardization

To allow for data to be interpreted by machine learning algorithms it needs to
go through a process of standardization. This is often due to categorical non-
numeric features or continuous features. The data provided by most if not all
internet protocols is categorical (e.g. agent names and method calls). As
such, in order to evaluate it we first needed to go through an initial phase
of pre-processing. The intent of pre-processing is to render the data machine
readable whilst preserving patterns. The process we adapted was the process of
binarisation. Binarisation allocates a numeric value to each unique feature
for example if dealing with HTTP codes GET would become 0001, POST 0010,
DELETE 0100 and PUT 1000. This allows for the features to maintain their
patterns and their predictive power and be used normally. This initial step
was applied to both the real world dataset and the naive synthetic dataset.
This step was however not required for the model dataset as it already
produced numeric features rather than categorical ones for efficiency.

###  6.3 Classifiers

The classifiers we implemented represented the IDSs. We choose to use two
separate classifiers to get a better evaluation of the results. Each dataset
was used to train two IDSs and then all the IDSs were tested against a new
dataset of attack to establish their predictive power and the strength of the
datasets.

The first classifier we implemented was Multi Layer Perceptron (MLP) Neural
Network. An MLP consists of at least three layers of nodes. Except for the
input nodes, each node is a neuron that uses a non-linear activation function
[24]. MLP utilizes a supervised learning technique called back propagation for
training. Its multiple layers and non-linear activation distinguish MLP from a
linear perceptron. A linear perceptron is a function that can decide whether
an input, represented by a vector of numbers, belongs to some specific class
or not. Combining several together in an MLP and adjusting the functions and
weights you build a statistically accurate classifier. The result is a non-
linear perceptron that is able to classify non-linear classes.

The second classifier used was a Decision Tree Classifier. A decision tree is
a decision support tool that uses a tree-like graph or model of decisions and
their possible consequences, including chance event outcomes, resource costs,
and utility. It is one way to display an algorithm that only contains
conditional control statements [20]. Decision tree learning uses a decision
tree (as a predictive model) to go from observations about an item
(represented in the branches) to conclusions about the item’s target value
(represented in the leaves). The rules in the branches are automatically
constructed from the training data which is labelled. Using these rules it
will be able to take in the test data and run it until it reaches an end node
corresponding to a class (either DoS attack or normal behaviour).

##  7 Results

Following the evaluation criteria in section 5 and recreating the model
described in section 4, we generated and tested three model datasets against
our benchmarks of the naive dataset and the real world dataset. Beyond the
accuracy of the results, we make an argument for feasibility and re usability
of our approach. The results were acquired by initially training two
classifiers for each dataset, these were trained with 20,000 samples of which
10% were attacks. The classifiers were then evaluated on an unknown and
unlabelled real world dataset of 100,000 samples of which 20% were attacks (of
two different unknown types). The classifiers then attempted to label the new
dataset to predict which ones were attacks.

###  7.1 Experiment 1 - Results

The neural network trained on the real world dataset proved to be very
accurate with a 85.5% prediction accuracy. On the other hand the model dataset
trained predictor whilst still high, suffered from some degree of variance
($79.7\pm 6.3\%$). What was of most interest however was the predictions
outputted by the naive dataset of 0.9%. This combined with the relatively
inconsistent results of the synthetic dataset ($\pm 6.3\%$) make a case for
over fitting. Over fitting is the scenario in which a model is trained so
specifically to the training data that it is no longer classifying DoS attacks
and normal behaviour of the system but rather focusing solely on the training
data and learning on patters unique to the dataset not the system. This is
quite common in Neural Networks as they perform best with very large
quantities of data [24], which for this part of the experiment we did not
have.

The results of the decision tree, contrasting to neural networks do not suffer
from the same inadequacy of over fitting and do not necessarily need large
amounts of data. This was mirrored by the results, as the model datasets all
performed to very similar standards and the added randomness traces which
might have disrupted the neural network made for a more ample rule set
resulting in near perfect predicting power in the model dataset ($98.8\pm
0.6$). The real world data which did not look at the possibility of random
behaviour only achieved 77% accuracy and the random dataset had a predictive
power of near 50% as expected.

###  7.2 Experiment 2 - Results

We observed that our approach of using non-determinism to recreate attack
traces was particularly effective for the rule based classifier however led to
disruption during the back-propagation process of the neural network, as non-
standardized data can create uneven results. This time using the much larger
dataset of 100,000 transitions,the results were a lot more accurate (97.1%)
than previously, confirming our hypothesis.

As highlighted by this example our model has one key advantage over the
traditional approach. Data generation is fast and efficient. If we wanted to
improve the training of the IDS used on the real world dataset to a similar
level of accuracy, it would take several days of data collection and
consumption of resources (electricity, system downtime etc). We argue that
whilst the initial effort of creating a model might be time consuming and
perhaps not as intuitive for a potential system administrator, the phase of
dataset generation makes up for this effort both for speed and predicting
power of the IDS.

##  8 Conclusion & Future work

Our case study and proposed methodology has shown very promising results. We
have shown that generating synthetic datasets of DoS attacks in IoT networks
through this tool is both effective and efficient. We believe that the ability
for this approach to scale easily to multiple devices and protocols in
combination with its strong predictive power makes a very good argument for
its usage across various IoT networks. Our argument for scalability of this
approach is two fold, firstly it scales well in terms of costs as you can make
assessment prior to implementing the system and secondly, we can bypass
several of the downsides of verification (in terms of state space) as we focus
on simulation. Perhaps the most useful feature of our proposed approach is
that it allows for the construction of datasets to be very efficient even if a
device is added or the system is reconfigured. As this is a prominent concern
in dynamic IoT systems this advantage is quite significant.

In this paper we included a case study of a single attack which worked very
well. Our future work envisions the ability to model further attacks from a
database to create an extensive set of attacks to create a much more
predictive dataset. We envision that the ability to relatively easily plug and
play any IoT system in combination with implemented corpus of attacks, could
turn into a tool that generates synthetic datasets of attacks to train bespoke
IDSs for any IoT system.

## References

  * [1] Farnell element14, calculating battery life in iot applications (2017), http://uk.farnell.com/calculating-battery-life-in-iot-applications
  * [2] Hulk, web server dos tool - confessions of a dangerous mind (Feb 2013), http://www.sectorix.com/2012/05/17/hulk-web-server-dos-tool/
  * [3] Andova, S.: Probabilistic process algebra. Technische Universiteit Eindhoven (2002)
  * [4] Arnaboldi, L., Morisset, C.: Quantitative analysis of dos attacks and client puzzles in iot systems. In: Security and Trust Management STM 2017
  * [5] Baier, C., Katoen, J.P., Larsen, K.G.: Principles of model checking. MIT press (2008)
  * [6] Bhuyan, M.H., Bhattacharyya, D.K., Kalita, J.K.: Network anomaly detection: methods, systems and tools. IEEE communications surveys & tutorials (2014)
  * [7] Böhme, R., Félegyházi, M.: Optimal information security investment with penetration testing. In: International Conference on Decision and Game Theory for Security. pp. 21–37. Springer (2010)
  * [8] Buennemeyer, T.K., Gora, M., Marchany, R.C., Tront, J.G.: Battery exhaustion attack detection with small handheld mobile computers. In: Portable Information Devices, 2007.
  * [9] Fruth, M.: Formal methods for the analysis of wireless network protocols. Oxford University (2011)
  * [10] Gubbi, J., Buyya, R., Marusic, S., Palaniswami, M.: Internet of things (iot): A vision, architectural elements, and future directions. Future generation computer systems (2013)
  * [11] Guillen, E., Sánchez, J., Paez, R.: Inefficiency of ids static anomaly detectors in real-world networks. Future Internet 7(2), 94–109 (2015)
  * [12] Hoare, C.A.R.: Communicating sequential processes. Communications of the ACM 21(8), 666–677 (1978)
  * [13] Kwiatkowska, M., Norman, G., Parker, D.: Prism: Probabilistic symbolic model checker. In: International Conference on Modelling Techniques and Tools for Computer Performance Evaluation (2002)
  * [14] Liang, L., Zheng, K., Sheng, Q., Huang, X.: A denial of service attack method for an iot system. In: Information Technology in Medicine and Education (ITME), 2016 8th International Conference on. pp. 360–364. IEEE (2016)
  * [15] Mell, P., Hu, V., Lippmann, R., Haines, J., Zissman, M.: An overview of issues in testing intrusion detection systems (2003)
  * [16] Mirkovic, J., Dietrich, S., Dittrich, D., Reiher, P.: Internet Denial of Service: Attack and Defense Mechanisms (Radia Perlman Computer Networking and Security). Prentice Hall PTR, Upper Saddle River, NJ, USA (2004)
  * [17] Mukkamala, S., Janoski, G., Sung, A.: Intrusion detection using neural networks and support vector machines. In: Neural Networks, 2002. IJCNN’02. Proceedings of the 2002 International Joint Conference on. vol. 2, pp. 1702–1707. IEEE (2002)
  * [18] Roesch, M., et al.: Snort: Lightweight intrusion detection for networks. In: Lisa. vol. 99, pp. 229–238 (1999)
  * [19] Roman, R., Zhou, J., Lopez, J.: On the features and challenges of security and privacy in distributed internet of things. Computer Networks (2013)
  * [20] Safavian, S.R., Landgrebe, D.: A survey of decision tree classifier methodology. IEEE transactions on systems, man, and cybernetics 21(3), 660–674 (1991)
  * [21] Security, O.: Exploitdb, offensive security’s exploit database archive (2009), https://www.exploit-db.com/
  * [22] Suo, H., Wan, J., Zou, C., Liu, J.: Security in the internet of things: a review. In: Computer Science and Electronics Engineering (ICCSEE), 2012 international conference on. vol. 3, pp. 648–651. IEEE (2012)
  * [23] Talpade, R., Madhani, S., Mouchtaris, P., Wong, L.: Mitigating denial of service attacks (Jan 29 2003), uS Patent App. 10/353,527
  * [24] Zhang, G.P.: Neural networks for classification: a survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 30(4), 451–462 (2000)