# Global and local bifurcations, three-dimensional Henon maps and discrete
Lorenz attractors

I. Ovsyannikov<EMAIL_ADDRESS><EMAIL_ADDRESS>University
of Bremen, MARUM, Department of Mathematics, Germany.  Lobachevsky State
University of Nizhny Novgorod, ITMM, Russia

###### Abstract

Lorenz attractors play an important role in the modern theory of dynamical
systems. The reason is that they are robust, i.e. preserve their chaotic
properties under various kinds of perturbations. This means that such
attractors can exist in applied models and be observed in experiments. It is
known that discrete Lorenz attractors can appear in local and global
bifurcations of multidimensional diffeomorphisms. However, to date only
partial cases were investigated. In this paper bifurcations of homoclinic and
heteroclinic cycles with quadratic tangencies of invariant manifolds are
studied. A full list of such bifurcations, leading to the appearance of
discrete Lorenz attractors is provided. In addition, with help of numerical
techniques, it was proved that if one reverses time in the diffeomorphisms
described above, the resulting systems also have such attractors. This result
is an important step in the systematic studies of chaos and hyperchaos.

> Dynamical chaos is a topic that attracts a high interest of researchers as
> dynamical models with complex behaviour can be widely met in applications.
> This relates, for example, to climate models Lorenz (1986); Palmer (1998),
> fluid mechanics Pedlosky and Frenzen (1980); Pedlosky (2011); Eilertsen and
> Magnan (2018), nonholonomic rigid body mechanics Gonchenko, Gonchenko, and
> Kazakov (2013), laser dynamics Vladimirov and Volkov (1993) and many others.
> Usually, the presence of chaos in a dynamical system is connected to the
> existence of strange attractors. According to Afraimovich and Shilnikov
> Aframovich and Shilnikov (1983), they can be divided into two main types –
> genuine strange attractors and quasiattractors. Many of well-known types of
> chaotic attractors, such as Hénon-like attractors, spiral attractors,
> Rössler attractors, attractors in the Chua circuits etc., are
> quasiattractors in the sense that arbitrary small perturbations of them lead
> to appearance of stable periodic orbits. This is not the case for genuine
> strange attractors, they exist in open domains in the space of dynamical
> systems and even in the case when they are structurally unstable (e.g. the
> Lorenz attractor), stable periodic orbits are not born in bifurcations.

##  I Introduction

Among the known genuine strange attractors there are hyperbolic attractors,
Lorenz attractors and wild hyperbolic attractors. The latter are remarkable by
the fact that they, unlike the previous ones, allow homoclinic tangencies and
therefore they belong to Newhouse domains Newhouse, Palis, and Takens (1983);
Gonchenko, Turaev, and Shilnikov (1993). In addition, such attractors are
stable, closed and chain-transitive invariant sets. Chain-transitivity means
that any point of attractor $\Lambda$ is admissible by $\varepsilon$-orbits
from any other point of $\Lambda$; stability means the existence of an open
absorbing domain containing the attractor such that any orbit entering the
domain tends to $\Lambda$ exponentially fast. All above constitutes the
definition of an attractor by Ruelle and Conley Ruelle (1981); Conley (1978).

An example of a wild spiral (hyperbolic) attractor was presented first by
Turaev and Shilnikov Turaev and Shilnikov (1998). Another important example of
a wild hyperbolic attractor is the discrete Lorenz attractor which appears, in
particular, in the Poincare maps for periodically perturbed flows with Lorenz
attractors Turaev and Shilnikov (2008). They were also observed in
applications, such as the nonholonomic rattleback model Gonchenko, Gonchenko,
and Kazakov (2013) and two-component convection Eilertsen and Magnan (2018).
It is well-known that the classical Lorenz attractor does not allow homoclinic
tangencies Afraimovich, Bykov, and Shilnikov (1977, 1982), however the latter
can appear under small non-autonomous periodic perturbations, this is a method
to make a wild hyperbolic attractor out of the Lorenz attractor. The reason
why stable periodic orbits do not arise from bifurcations of these tangencies,
is that Lorenz-like attractors possess a pseudo-hyperbolic structure, and this
property is preserved under small perturbations. Recall that the pseudo-
hyperbolic structure exists in map $f$ if its differential $Df$ in the
restriction onto the absorbing domain ${\cal D}$ of attractor $\Lambda$ admits
for any point $x\in{\cal D}$ an invariant splitting of form $E^{ss}_{x}\oplus
E^{uc}_{x}$, such that $Df$ is strongly contracting along directions $E^{ss}$
and expands volumes in transversal to $E^{ss}$ sections $E^{uc}$ (see Refs.
Turaev and Shilnikov, 1998, 2008 for details), and this splitting continuously
depend on the point $x$.

One of the peculiarities of discrete Lorenz attractors is that such attractors
can be born at local bifurcations of periodic orbits having three or more
multipliers lying on the unit circle. Thus the corresponding attractors can be
found in particular models which have a sufficient number of parameters to
provide the mentioned degeneracy. The following 3D Hénon map

$\bar{x}=y,\;\;\bar{y}=z,\;\;\bar{z}=M_{1}+Bx+M_{2}y-z^{2}$ (1)

which depends on three parameters $M_{1}$, $M_{2}$ and $B$ and has constant
Jacobian $B$, is an example of such a model. In Refs. Gonchenko  _et al._ ,
2005; Gonchenko, Meiss, and Ovsyannikov, 2006; Gonchenko  _et al._ , 2013 it
was shown that map (1) possesses a discrete Lorenz attractor in some open
parameter domain near point $(M_{1},B,M_{2})=(1/4,1,1)$, where the map has a
fixed point with the triplet $(-1,-1,+1)$ of multipliers. Recently in Ref.
Gonchenko  _et al._ , 2021a, it was proved that map (1) also has discrete
Lorenz attractors in the orientation reversing case $B<0$, near the
codimension-three bifurcation $(M_{1},M_{2},B)=(7/4,-1,-1)$, when the map has
a fixed point with eigenvalues $(i,-i,-1)$. The properties of these attractors
were studied in Ref. Gonchenko  _et al._ , 2021b. Note that the second iterate
of the map in this case has a fixed point with multipliers $(-1,-1,+1)$, but,
unlike the orientable case, they form three Jordan blocks instead of two, so
the bifurcations that happen for $B>0$ and $B<0$ are principally different.

These results immediately imply the birth of discrete Lorenz attractors in
systems where map (1) appears, e.g. as a Poincare map. This, in particular,
happens in global (homoclinic and heteroclinic) bifurcations. The first such
example was considered in Ref. Gonchenko, Meiss, and Ovsyannikov, 2006 for a
homoclinic tangency to a saddle-focus. Later analogous results were obtained
for heteroclinic cycles containing saddle-foci Gonchenko, Shilnikov, and
Turaev (2009); Gonchenko and Ovsyannikov (2010, 2013) and homoclinic and
heteroclinic cycles consisting of saddles and having additional degeneracies,
such that non-simple homoclinic (heteroclinic) orbits Gonchenko, Ovsyannikov,
and Tatjer (2014); Ovsyannikov (2017), or a resonance condition on the
eigenvalues at the fixed point Gonchenko and Ovsyannikov (2017). Note that the
presence of saddle-foci or degeneracies of certain kinds in these cases is a
very important condition for the birth of Lorenz-like attractors as it
prevents from the existence of lower-dimensional center manifolds and makes
the dynamics to be effectively three-dimensional (see Refs. Turaev, 1996;
Gonchenko, Shilnikov, and Turaev, 1993; Tatjer, 2001). Another important
condition was imposed on the values of the Jacobian of the map evaluated in
the fixed points. It is based on the fact that the orbits under consideration
may spend unboundedly large number of iterations in the neighbourhoods of
saddle fixed points. In the homoclinic case this means that if the Jacobian is
separated from unity, the phase volumes near such orbits will be either
unboundedly expanded or unboundedly contracted, and the dynamics will have
effective dimension less than three. In the same way, for the heteroclinic
cases it is necessary to demand that all the Jacobians are not simultaneously
contracting ($<1$) or simultaneously expanding ($>1$). Thus, in order to get
Lorenz attractors in bifurcations of heteroclinic cycles, one needs to
consider “contracting-expanding” or “mixed” cases.

The results mentioned above concern homoclinic and heteroclinic
$(2,1)$–cycles, i.e. those containing fixed points with a two-dimensional
stable manifold and one-dimensional unstable manifold. In a three-dimensional
space two other kinds of cycles are possible: $(1,2)$–cycles, consisting of
fixed points of type $(1,2)$, and heterodimensional cycles, having saddles of
both stability types. Heterodimensional cycles are out of scope of this paper.
Every system with a $(1,2)$–cycle can be regarded as an inverse to a system
with a $(2,1)$–cycle, in which the first return map is the 3D Henon map (1).
This immediately means that the first return map in $(1,2)$–cycles will be
close up to small terms to the inverse map of (1), this is the following
three-dimensional Henon map:

$\bar{x}=y,\;\;\bar{y}=z,\;\;\bar{z}=\hat{M}_{1}+\hat{B}x+\hat{M}_{2}z-y^{2}.$
(2)

This map was obtained first in Ref. Gonchenko, Shilnikov, and Turaev, 1993 as
a first return map along a homoclinic orbit to a four-dimensional saddle-focus
of $(2,2)$ type. As map (2) is an inverse to (1), there exist domains in the
space of parameters $(\hat{M}_{1},\hat{M}_{2},\hat{B})$, in which it possesses
a discrete Lorenz repeller Gonchenko  _et al._ (2005, 2021a, 2021b). However,
Lorenz attractors have not been found in this map before.

In the present paper, bifurcations, leading to the birth of discrete Lorenz
attractors in homoclinic and heteroclinic cycles of type $(2,1)$ and $(1,2)$
with quadratic tangencies of invariant manifolds, are studied. The full list
of such bifurcations is presented, namely homoclinic (or heteroclinic,
consisting of two fixed points) cycles, having

  * •

saddle-focus fixed points;

  * •

non-simple homoclinic (heteroclinic) orbits;

  * •

resonant fixed points.

This list includes known results for orientable maps Gonchenko, Meiss, and
Ovsyannikov (2006); Gonchenko, Shilnikov, and Turaev (2009); Gonchenko and
Ovsyannikov (2010, 2013); Gonchenko, Ovsyannikov, and Tatjer (2014);
Ovsyannikov (2017); Gonchenko and Ovsyannikov (2017), which were extended here
to non-orientable maps. The latter is possible due to the recent result
Gonchenko  _et al._ (2021a) that map (1) possesses discrete Lorenz attractors
also in the non-orientable case $B<0$.

In addition, new cases are considered, when the cycle contains resonant fixed
points that undergo the Belyakov transition from saddle to saddle-focus. This
happens when at the bifurcation moment the stable multipliers have
multiplicity two, and under small perturbations such a pair splits either in
two different real eigenvalues or in a complex-conjugate pair.

The results of the paper are the following. First, for $(2,1)$–cycles it is
shown that in the space of dynamical systems the original system is a limit of
a sequence of open subsets containing systems with discrete Lorenz attractors,
Theorem 1. The proof is based on the fact (Lemma 3) that the first return map
near a homoclinic or heteroclinic cycle can be represented in the form of
three-dimensional Henon map (1), which has the discrete Lorenz attractor, see
Refs. Gonchenko  _et al._ , 2005; Gonchenko, Meiss, and Ovsyannikov, 2006;
Gonchenko  _et al._ , 2013, 2021a. Then, it is proved numerically (Lemma 2),
that the inverse to (1) map (2) possesses the discrete Lorenz attractor near
certain period-$6$ points. This result implies that systems with
$(1,2)$–cycles are also limits of sequences of open subsets in which systems
have discrete Lorenz attractors

The paper is organised as follows. Section II contains the statement of the
problem, main definitions, and also the main results, Theorems 1 and 2 are
formulated there. In Section III the birth of discrete Lorenz attractors in
the inverse 3D Henon map (2) is studied. In Section IV the first return map is
constructed. For all cases under consideration, local and global maps are
written. At the end of the Section, there is the rescaling Lemma 3, stating
that the first return map for all $(2,1)$–cycles coinsides with the 3D Henon
map (1).

Figure 1:  A heteroclinic cycle consisting of two saddles, with a quadratic
tangency of manifolds.

##  II Statement of the problem and main results

In this section two classes of problems are set up: the study of bifurcations
in homoclinic and heteroclinic cycles. These two classes are defined by
corresponding conditions A–C, and fall into subcases, defined by an additional
condition D.

Let a $C^{r}$-smooth three-dimensional diffeomorphism $f_{0}$, $r\geq 4$,
satisfy the following conditions:

I. The homoclinic case.

  * A.

$f_{0}$ has a fixed point $O$ with multipliers
$(\lambda_{1},\lambda_{2},\gamma)$, where $|\lambda_{1,2}|<1<|\gamma|$;

  * B.

$|J(O)|\equiv|\lambda_{1}\lambda_{2}\gamma|=1$;

  * C.

Invariant manifolds $W^{u}(O)$ and $W^{s}(O)$ have a quadratic tangency at the
points of a homoclinic orbit $\Gamma_{0}$.

II. The heteroclinic case.

  * A.

$f_{0}$ has two fixed points $O_{1}$ and $O_{2}$ of type $(2,1)$, i.e. each
$O_{j}$ has multipliers $(\lambda_{(j)1},\lambda_{(j)2},\gamma_{(j)})$ with
$|\lambda_{(j)1,(j)2}|<1<|\gamma_{(j)}|$ for $j=1,2$.

  * B.

The absolute value of the Jacobian of $f_{0}$ is less than one in one fixed
point and greater than one in another one. Without loss of generality,
$|J(O_{1})|\equiv|\lambda_{(1)1}\lambda_{(1)2}\gamma_{(1)}|<1$ and
$|J(O_{2})|\equiv|\lambda_{(2)1}\lambda_{(2)2}\gamma_{(2)}|>1$.

  * C.

There exists a heteroclinic cycle such that one-dimensional stable manifolds
$W^{u}(O_{1})$ intersect transversely two-dimensional stable manifolds
$W^{s}(O_{2})$ in the points of hereroclinic orbits $\Gamma_{12}$ and unstable
manifold $W^{u}(O_{2})$ has a quadratic tangency with stable manifold
$W^{s}(O_{1})$ at the points of a non-transversal heteroclinic orbit
$\Gamma_{21}$.

The goal of this paper is the study of bifurcations of single-round periodic
orbits, lying in a some small neighborhood of the homoclinic or the
heteroclinic cycle defined by conditions A–C. The main attention is paid to
codimension-three bifurcations leading to the appearance of discrete Lorenz
attractors. For this purpose, the first return map $T$ along the cycle is
constructed, such that single-round periodic orbits become fixed points of map
$T$. This map is three-dimensional, however, in some circumstances it can have
lower-dimensional invariant submanifolds, that prevent from the existence of
Lorenz-like attractors. To avoid this, the effective dimension of the problem
Turaev (1996) should be kept equal to three. This is achieved by imposing an
additional condition D, which gives the following subcases.

D.I. The homoclinic case

  * 1.

Point $O$ is a saddle-focus, i.e. $\lambda_{1,2}=\lambda e^{\pm i\varphi}$,
$0<\varphi<\pi$, see Ref. Gonchenko, Meiss, and Ovsyannikov, 2006;

  * 2.

The quadratic tangency at $\Gamma_{0}$ is non-simple (the definitions are
given below), see Ref. Gonchenko, Ovsyannikov, and Tatjer, 2014:

    * 2.a.

the homoclinic orbit undergoes an inclination flip;

    * 2.b.

the homoclinic orbit undergoes an orbit flip;

  * 3.

The multipliers at $O$ satisfy the following conditions:

    * 3.a.

an alternating resonance $\lambda_{1}=-\lambda_{2}=\lambda$, see Ref.
Gonchenko and Ovsyannikov, 2017;

    * 3.b.

a Belyakov transition from saddle to saddle-focus
$\lambda_{1}=\lambda_{2}=\lambda$.

D.II. The heteroclinic case

  * 1.

One of the points $O_{1}$ and $O_{2}$, or they both, are saddle-foci, see
Refs. Gonchenko, Shilnikov, and Turaev, 2009; Gonchenko and Ovsyannikov, 2010,
2013;

  * 2.

One of the heteroclinic orbits is non-simple, see Ref. Ovsyannikov, 2017:

    * 2.a.

the non-transversal orbit $\Gamma_{21}$ undergoes an inclination flip;

    * 2.b.

the non-transversal orbit $\Gamma_{21}$ undergoes an orbit flip;

    * 2.c.

the transversal orbit $\Gamma_{12}$ undergoes an orbit flip;

  * 3.

The stable multipliers at one fixed point $O_{j}$, $j=1$ or $2$, satisfy the
following condition

    * 3.a.

an alternating resonance $\lambda_{(j)1}=-\lambda_{(j)2}=\lambda$;

    * 3.b.

a Belyakov transition from saddle to saddle-focus
$\lambda_{(j)1}=\lambda_{(j)2}=\lambda$.

The notation of the cases is the following: the Roman numbers I and II denote
the homoclinic and heteroclinic cases respectively, and arabic numbers with
letters denote the subcases given by condition D. For example, case I.2.a is a
homoclinic tangency with an inclination flip to a saddle fixed point, and case
II.1 is a heteroclinic cycle consisting of a saddle and a saddle-focus or two
saddle-foci. All the required definitions are given later in this section.

The following lemma states that fulfilment of at least one subcase of D is a
necessary condition for discrete Lorenz attractors to appear in bifurcations.

###### Lemma 1

Assume that $f_{0}$ satisfies conditions A–C, but condition D is not
fulfilled. Then in small generic unfoldings $f_{\mu}$ of $f_{0}$, the first
return map along the homoclinic or heteroclinic cycle possesses a global
invariant manifold of dimension lower than three.

Diffeomorphisms close to $f_{0}$ and satisfying conditions A–D compose locally
connected bifurcation surfaces in the space of $C^{r}$-diffeomorphisms. They
have codimension one in case II.1 codimension two in cases I.1, II.2–3 and
codimension three in cases I.2–3.

Generic bifurcations of systems close to $f_{0}$ are studied in three-
parametric unfoldings, that are, the families of diffeomorphisms $f_{\mu}$,
$\mu=(\mu_{1},\mu_{2},\mu_{3})$ such that
$\left.f_{\mu}\right|_{\mu=0}=f_{0}$. The first parameter $\mu_{1}$ is
selected as the splitting distance of the quadratic tangency (homoclinic or
heteroclinic), defined by condition C. The second parameter $\mu_{2}$ controls
condition D such that in the cases I–II.1, when one or more saddle-foci are
present, $\mu_{2}$ is monotonically related to the complex argument
$\varphi_{j}$ of stable multipliers of one saddle-focus $O_{j}$:
$\mu_{2}=\varphi_{j}-\varphi_{0}$, and in all other cases $\mu_{2}$ unfolds
the corresponding degeneracy given by condition D.

The third parameter $\mu_{3}$ controls the Jacobians at the saddle points,
such that in the homoclinic case I it is responsible to a deviation of the
Jacobian from unity:

$\mu_{3}=|J(O)|-1,$ (3)

and in the heteroclinic case II it is a deviation of a functional $S$:

$\mu_{3}=S(f_{\mu})-S(f_{0}),$ (4)

where

$S=-\frac{\ln|J(O_{1})|}{\ln|J(O_{2})|}.$

Note that when the system has a homoclinic or heteroclinic cycle containing
saddle-foci (cases I–II.1), there exist continuous invariants of topological
conjugacy on the set of non-wandering orbits ($\Omega$-moduli), and the most
important moduli are the angular arguments Gonchenko (2000) of complex
eigenvalues. This is the reason why at least one of them should be chosen as a
control parameter $\mu_{2}$. The condition B on Jacobians are essential for
all cases, as the orbits lying in a small neighbourhood of the homoclinic or
heteroclinic cycle, may spend arbitrary large number of iterations near saddle
points, and if the absolute value of the Jacobian in the homoclinic case I
will be separate from $1$, then the phase volumes will be unboundedly
contracted or expanded, thus no three-dimensional effective dynamics and
Lorenz-like attractors will be possible. The same applies to heteroclinic
cycles, just here it is reuquired that there exists a point $O_{1}$, where the
phase volumes are contracted, $|J(O_{1})|<1$ and a point $O_{2}$, near which
they are expanded, $|J(O_{2})|>1$ (the so-called, contracting-expanding, or
mixed case). Controlling the number of iterations the orbit spends near these
two points, one can avoid unbounded contractions or expansions, i.e. existence
of lower-dimensional invariant manifolds.

When both fixed points are saddles (the eigenvalues are real, see Fig. 1), one
needs to impose some additional bifurcation conditions, in order to prevent
from existence of lower-dimensional center manifolds. They can be separated
into global and local ones.

###  II.1 Global degeneracies (non-simple homoclinic and heteroclinic orbits).

Before giving definitions, recall some facts from the normal hyperbolicity
theory. Let $O$ be a saddle fixed point of type $(2,1)$ with eigenvalues
$|\lambda_{2}|<|\lambda_{1}|<1<|\gamma|$ and $U_{0}$ be some small
neighbourhood of it. It is known Gonchenko, Shilnikov, and Turaev (2008a);
Gonchenko and Shilnikov (1990, 1993); Shilnikov  _et al._ (1998) that
diffeomorphism $\left.f_{\mu}\right|_{U_{0}}$ for each small $\mu$ can be
represented in some $C^{r}$-smooth local coordinates $(x_{1},x_{2},y)$ as
follows (the so-called main normal form):

$\begin{array}[]{l}\bar{x}_{1}\;=\;\lambda_{1}(\mu)x_{1}+\tilde{H}_{2}(y,\mu)x_{2}+O(\|x\|^{2}|y|)\\\
\bar{x}_{2}\;=\;\lambda_{2}(\mu)x_{2}+\tilde{R}_{2}(x,\mu)+\tilde{H}_{4}(y,\mu)x_{2}+O(\|x\|^{2}|y|)\\\
\bar{y}\;=\;\gamma(\mu)y+O(\|x\||y|^{2}),\\\ \end{array}$ (5)

where $\tilde{H}_{2,4}(0,\mu)=0$, $\tilde{R}_{2}(x,\mu)=O(\|x\|^{2})$. In
coordinates (5) the invariant manifolds of saddle fixed point $O$ are locally
straightened: stable $W^{s}_{loc}(O):\\{y=0\\}$, unstable
$W^{u}_{loc}(O):\\{x_{1}=0,\;x_{2}=0\\}$ and strong stable
$W^{ss}_{loc}(O):\\{x_{1}=0,\;y=0\\}$.

According to Refs. Shilnikov  _et al._ , 1998; Hirsch, Pugh, and Shub, 1977,
an important role in dynamics is played by an extended unstable manifold
$W^{ue}(O)$, see Fig. 2. By definition, it is a two-dimensional invariant
manifold, that is tangent to the leading stable direction (corresponding to
$\lambda_{1}$) at the saddle point and contains unstable manifold $W^{u}(O)$.
Unlike the previous ones, the extended unstable manifold is not uniquely
defined and its smoothness is, generally speaking, only $C^{1+\varepsilon}$.
Locally, $W^{ue}_{loc}(O)=W^{ue}(O)\cap U_{0}$, and the equation of
$W^{ue}_{loc}(O)$ has the form $x_{2}=\varphi(x_{1},y)$, where
$\varphi(0,y)\equiv 0$ and $\varphi^{\prime}_{x_{1}}(0,0)=0$. Note that
despite the fact that $W^{ue}(O)$ is non-unique, all such manifolds have the
same tangent plane at each point of $W^{u}(O)$.

Another essential fact is the existence of the strong stable invariant
foliation, see Figure 2. In $W^{s}(O)$ there exists a one-dimensional strong
stable invariant submanifold $W^{ss}(O)$, which is $C^{r}$–smooth and touches
at $O$ the eigenvector corresponding to the strong stable (nonleading)
multiplier $\lambda_{2}$. Stable manifold $W^{s}(O)$ is foliated near $O$ by
the leaves of invariant foliation $F^{ss}$ which is $C^{r}$-smooth, unique and
contains $W^{ss}(O)$ as a leaf.

Figure 2:  Invariant structures near a saddle fixed point $O$. A part of the
strong stable foliation $F^{ss}$ containing the strong stable manifold
$W^{ss}$ and a piece of one of the extended unstable manifolds $W^{ue}$
containing $W^{u}$ and being transversal to $W^{ss}$ at $O$.

Now consider a pair of saddle fixed points $O_{1}$ and $O_{2}$, and orbit
$\Gamma_{21}$ in the points of which manifolds $W^{u}(O_{2})$ and
$W^{s}(O_{1})$ have a quadratic tangency (heteroclinic in case II, when
$O_{1}\neq O_{2}$, and homoclinic if points $O_{1}$ and $O_{2}$ coincide, case
I). Each of the points possesses invariant manifolds and foliations described
above. Let $U_{01}\ni O_{1}$ and $U_{02}\ni O_{2}$ be some small
neighbourhoods of the fixed points, $M_{1}^{+}\in W_{loc}^{s}(O_{1})\subset
U_{01}$ and $M_{2}^{-}\in W_{loc}^{u}(O_{2})\subset U_{02}$ be two points of
$\Gamma_{21}$ and $\Pi_{1}^{+}\subset U_{01}$ and $\Pi_{2}^{-}\subset U_{02}$
their respective neighborhoods. Note that there exists some integer $q_{N}$
such that $M_{1}^{+}=f_{0}^{q_{1}}(M_{2}^{-})$. The global map along
$\Gamma_{21}$ is defined as
$T_{21}:\Pi_{2}^{-}\to\Pi_{1}^{+}=\left.f^{q_{1}}\right|_{\Pi_{2}^{-}}$.

###### Definition 1

The homoclinic or heteroclinic tangency of $W^{u}(O_{2})$ and $W^{s}(O_{1})$
is called simple if image $T_{21}(P^{ue}(M_{2}^{-}))$ of tangent plane
$P^{ue}(M_{2}^{-})$ to $W^{ue}(O_{2})$ intersects transversely the leaf
$F_{1}^{ss}(M_{1}^{+})$ of invariant foliation $F_{1}^{ss}$, containing point
$M_{1}^{+}$. Otherwise, such a quadratic tangency is called non-simple.
Following Ref. Tatjer, 2001, there may be two generic cases of non-simple
homoclinic (heteroclinic) tangencies:

Orbit flip. Surface $T_{21}(P^{ue}(M_{2}^{-}))$ is transversal to plane
$W^{s}_{loc}(O_{1})$ but is tangent to line $F_{1}^{ss}(M_{1}^{+})$ at point
$M_{1}^{+}$, fig.3 $($a$)$.

Inclination flip. Surfaces $T_{21}(P^{ue}(M_{2}^{-}))$ and
$W^{s}_{loc}(O_{1})$ have a quadratic tangency at $M_{1}^{+}$ and curves
$T_{21}(W^{u}_{loc}(O_{2})\cap\Pi_{2}^{-})$ and $F_{1}^{ss}(M_{1}^{+})$ have a
general intersection, fig.3 $($b$)$.

The existence of local extended unstable manifold $W^{ue}_{loc}(O_{2})$
implies that near fixed point $O_{2}$ the dynamics is effectively two-
dimensional (the restriction onto $W^{ue}_{loc}(O_{2})$) plus the strong
contraction in the transverse direction. If the homoclinic or heteroclinic
tangency is simple, then under forward-time further iterations the image of
$T_{21}(P^{ue}(M_{2}^{-}))$ will tend to become tangent to
$W^{ue}_{loc}(O_{1})$ at point $O_{1}$. This implies the existence of a global
two-dimensional center manifold along $\Gamma_{21}$. When the tangency is non-
simple, this global center manifold does not exist.

###### Remark 1

The names for these two degeneracies are taken analogous to the continuous-
time case, when the corresponding degeneracies also prevent from existence of
global two-dimensional center manifolds. The inclination flip corresponds to
the case when the extended unstable manifold of $O_{2}$ has a quadratic
tangency with the stable manifold of $O_{1}$, similarly to the discrete-time
case. Orbit flip in flows occurs when the unstable separatrix of $O_{2}$ comes
to $O_{1}$ along its strong stable direction, that is $W^{ue}(O_{2})\supset
W^{ss}(O_{1})$, and thus the tangent plane to $W^{ue}(O_{2})$ in any point of
the homoclinic $($heteroclinic$)$ orbit is also tangent to $W^{ss}(O_{1})$. In
discrete-time systems we call it an orbit flip when $W^{ue}(O_{2})$ is tangent
to any leaf of foliation $F^{ss}(O_{1})$ (in particular, but not necessary, it
can be $W^{ss}(O_{1})$ itself). Then this tangency will be preserved under the
forward iterations of the map, therefore images of $T_{21}(P^{ue}(M_{2}^{-}))$
will be always transverse to $W^{ue}_{loc}(O_{1})$ and no global center
manifold will exist.

Figure 3:  Two types of the non-simple quadratic (homoclinic or heteroclinic)
tangency: (a) Inclination flip: $W^{ue}(O_{2})$ is tangent to
$W^{s}_{loc}(O_{1})$ and curves $W^{u}(O_{2})$ and $F^{ss}(M_{1}^{+})$ have a
general intersection at $M_{1}^{+}$; (b) Orbit flip: $W^{ue}(O_{2})$ is
transversal to $W^{s}_{loc}(O_{1})$ and touches leaf $F^{ss}(M_{1}^{+})$

In the similar way, one defines orbit flip for a transversal heteroclinic
orbit $\Gamma_{12}$ (the non-simple heteroclinic intersection). Consider two
points $M_{1}^{-}\in U_{01}$ and $M_{2}^{+}\in U_{02}$, of $\Gamma_{12}$ and
their small respective neighbourhoods $\Pi_{1}^{-}\subset U_{01}$ and
$\Pi_{2}^{+}\subset U_{02}$. Again, there exists some integer $q_{2}$ such
that $M_{2}^{+}=f_{0}^{q_{2}}(M_{1}^{-})$ so that the global map from $U_{01}$
to $U_{2}$ is defined as
$T_{12}:\Pi_{1}^{-}\to\Pi_{2}^{+}=\left.f_{\mu}^{q_{2}}\right|_{\Pi_{1}^{-}}$.
Let $P^{ue}(M_{1}^{-})$ be the tangent plane to $W^{ue}(O_{1})$ at $M_{1}^{-}$
and $F_{2}^{ss}(M_{2}^{+})$ be the leaf of invariant foliation $F_{2}^{ss}$ on
$W^{s}(O_{2})$ passing through $M_{2}^{+}$.

###### Definition 2

The heteroclinic intersection of $W^{u}(O_{1})$ and $W^{s}(O_{2})$ is called
simple if image $T_{12}(P^{ue}(M_{1}^{-}))$ and leaf $F_{2}^{ss}(M_{2}^{+})$
intersect transversely. If this condition is not fulfilled the heteroclinic
intersection is non-simple, and undergoes an orbit flip, see fig. 4.

Figure 4:  A non-simple heteroclinic intersection (orbit flip) of
$W^{u}(O_{1})$ and $W^{s}(O_{2})$.

###  II.2  Local degeneracies.

Now consider the case when all fixed points in the heteroclinic cycle are
saddles and all connections are simple. The existence of extended unstable
manifold $W^{ue}$ and strong stable manifold $W^{ss}$ is a robust property –
they persists under small parametric perturbation, and with an absence of non-
simple global orbits this immediately implies the existence of a global lower-
dimensional center manifold along the homoclinic (heteroclinic) cycle. To
prevent from this, local bifurcations can occur at one of the saddle points,
such that $W^{ue}$ and $W^{ss}$ near this point do not exist at all, cases
I–II.3. The first bifurcation is the resonance condition when the stable
eigenvalues have the same absolute value, but different signs,
$\lambda_{1}=-\lambda_{2}=\lambda$, cases I–II.3.a. Under small perturbations
here the strong stable manifold appears, but in alternating directions: when
$|\lambda_{1}|<|\lambda_{2}|$, $W^{ss}$ is tangent to the eigendirection
corresponding to $\lambda_{1}$, and to the eigendirection corresponding to
$\lambda_{2}$ otherwise.

Figure 5:  A Belyakov-like transition. When $\mu_{2}>0$ (left), the stable
eigenvalues are real, and when $\mu_{2}<0$ (right) they form a complex-
conjugate pair.

The second bifurcation is analogous to the Belyakov resonance for the
continuous-time case Belyakov (1980), that is the boundary between saddle and
saddle-focus, cases I-II.3.b. At the bifurcation moment the stable multiplier
has multiplicity two: $\lambda_{1}=\lambda_{2}=\lambda$, and under small
perturbations such a degenerate saddle becomes a saddle with real eigenvalues
or a saddle-focus, see fig. 5.

###### Remark 2

Another type of saddle to saddle-focus transition is possible in dimensions
higher than three, when a complex-conjugate pair of stable eigenvalues
$\lambda_{2}e^{\pm i\varphi}$ coincides in absolute value with real stable
eigenvalue $\lambda_{1}$, i.e. $|\lambda_{1}|=\lambda_{2}=\lambda$. In
continuous time such a homoclinic bifurcation was studied recently in Ref.
Kalia, Kuznetsov, and Meier, 2019, where it was called the 3DL-bifurcation.
Under small perturbations, manifold $W^{ss}$ appears with alternating
direction and dimension, namely in the case when $|\lambda_{1}|<\lambda_{2}$
it is one-dimensional and tangent to the $\lambda_{1}$ eigendirection, and
when $|\lambda_{1}|>\lambda_{2}$, strong stable manifold $W^{ss}$ is two-
dimensional and tangent to the eigendirections corresponding to complex
eigenvalues $\lambda_{2}e^{\pm i\varphi}$. This case is out of scope of the
current paper and will be studied separately.

###  II.3 Main theorems

The main result of the paper is given by the following

###### Theorem 1

Let $f_{0}$ satisfy conditions A–D and $f_{\mu}$ be the three-parametric
unfolding of $f_{0}$ defined above. Then, in any neighbourhood of the origin
in the parameter space there exist infinitely many accumulating to $\mu=0$
domains $\delta_{k},\;k=\\{k_{1},k_{2}\\}$, such that diffeomorphism $f_{\mu}$
has a discrete Lorenz attractor for $\mu\in\delta_{k}$.

The proof of Theorem 1 is based on the fact that for sufficiently large $k$
the first return map can be transformed to the form of three-dimensional Henon
map (1) plus asymptotically small terms, see Lemma 3 for more details.
According to Ref. Gonchenko  _et al._ , 2005, map (1) possesses the discrete
Lorenz attractor is some open domain $V$ in the parameter space. Varying
indices $k_{1},k_{2}$ unboundedly, one gets that in the space of original
parameters this domains correspond to a sequence of domains $V_{k}$ that
accumulate to point $\mu=0$.

###### Theorem 2

Let $f_{0}$ satisfy conditions A–D and $f^{-1}_{\mu}$ be the three-parametric
unfolding of its inverse map $f^{-1}_{0}$. Then, in any neighbourhood of the
origin in the parameter space there exist infinitely many accumulating to
$\mu=0$ domains $\delta_{k},\;k=\\{k_{1},k_{2}\\}$, such that diffeomorphism
$f^{-1}_{\mu}$ has a discrete Lorenz attractor for $\mu\in\delta_{k}$.

##  III Three-dimensional Henon maps and discrete Lorenz attractors

In this section the main attention is paid to the extension of results
obtained in Refs. Gonchenko  _et al._ , 2005 and Gonchenko  _et al._ , 2013 on
the birth of discrete Lorenz attractors in local bifurcations of three-
dimensional maps. In these papers an analysis of a certain local codimension-
three bifurcation was performed, namely when a fixed point possesses
multipliers $(-1,-1,+1)$. The reason why this bifurcation is relevant to
Lorenz attractors is that the pair of $-1$ multipliers in a single Jordan
block provides the same symmetry $(x,y)\to(-x,-y)$ as in the Lorenz system.
Also, bifurcations of triply degenerate equilibria with the same symmetry lead
to birth of Lorenz attractors in continuous-time systems Shilnikov, Shilnikov,
and Turaev (1993).

In 3D Henon map (1), a fixed point with eigenvalues $(-1,-1,+1)$ exists when
$(M_{1},M_{2},B)=(-1/4,1,1)$. In paper Gonchenko  _et al._ (2005) the normal
form of this bifurcation was approximated by an ODE system, such that for some
$T$ a time-$T$ shift of the flow of that system coincided with map (1) up to
arbitrary small terms. It was also shown that this system coincides with the
Shimizu-Morioka system after rescaling of coordinates, parameters and time:

$\dot{X}=Y,\;\;\dot{Y}=X(1-Z)-\lambda Y,\;\;\dot{Z}=-\alpha Z+X^{2}.$ (6)

The latter has the Lorenz attractor in some open region of parameters, as
proved with assistance of computer, using interval arythmetics in Ref.
Capiński, Turaev, and Zgliczyński, 2018. Thus, the original Henon map can be
regarded as the time-$T$ shift of a periodically perturbed ODE system with a
Lorenz attractor. From Refs. Gonchenko  _et al._ , 2005, 2013 it follows that
map (1) possesses a discrete-time analogue of the classical Lorenz attractor.

The results above are applicable to orientation preserving maps, as in the
proof the Jacobian $B$ varies near $+1$. When the original map $f_{0}$ is non-
orientable, it is possible that the first return map will also reverse
orientation, and parameter $B$ will take only negative values. Then the
results above will not apply to such cases, and in order to find Lorenz-like
attractors, one needs to look for another parameter domain. Recently, this
case was studied in Ref. Gonchenko  _et al._ , 2021a. The authors considered a
fixed point with eigenvalues $(i,-i,-1)$, which exists in map (1) for
$(M_{1},M_{2},B)=(7/4,-1,-1)$, and showed that the flow normal form near this
bifurcation point possesses a new, 4-winged strange attractor of Lorenz type,
which they call the “Simo’s angel”. Numerically such an attractor was observed
in Ref. Gonchenko  _et al._ , 2005 for $(M_{1},M_{2},B)=(1.77,-0.925,-0.95)$.
Note that the fixed point with eigenvalues $(i,-i,-1)$ has eigenvalues
$(-1,-1,+1)$ for the second iterate of the map; however, there is no Jordan
block, so the normal form differs significantly from the Shimizu-Morioka
system (6).

In Refs. Gonchenko  _et al._ , 2013; Ovsyannikov and Turaev, 2017 the results
of Ref. Gonchenko  _et al._ , 2005 were extended to a wider class of maps and
a more generic criterion of existence. It was shown that near a fixed point
with multipliers $(-1,-1,+1)$ the map can be represented as the following
normal form:

$\begin{array}[]{rcl}\bar{u}_{1}&=&-u_{1}-u_{2}\\\
\bar{u}_{2}&=&-u_{2}+au_{1}u_{3}+a_{1}u_{2}u_{3}+O(\|u\|^{3})\\\
\bar{u}_{3}&=&u_{3}+bu_{1}^{2}+b_{1}u_{2}^{2}+b_{2}u_{1}u_{2}+b_{3}u_{3}^{2}+O(\|u\|^{3}).\\\
\end{array}$ (7)

By Lemma 3.1 from Ref. Gonchenko  _et al._ , 2013, if $ab>0$, the discrete
Lorenz attractors is born in generic perturbations of system (7), as in this
case the flow approximation of it is the Shimizu-Morioka system. From the
proof of the Lemma it can be easily seen that when $ab<0$, the flow normal
form can be also transformed to the Shimizu-Morioka system, but with negative
scaling of time. This means that in the normal form a discrete Lorenz repeller
is born in generic perturbations. Also it is wotrh to mention, that Lemma 3.1
provides a simple criterion of existence of Lorenz attractors or repellers –
this fact follows immediately from the signs of two coefficients $a$ and $b$
of normal form (7). This will be used below for the study towards the proof of
Conjecture 2.

Now consider the inverse map $f_{0}^{-1}$, i.e. the diffeomorphism having a
homoclinic or heteroclinic cycle composed of $(1,2)$ saddle or saddle-focus
fixed points and one quadratic tangency of invariant manifolds. The first
return map $X\to F(X)$ along the cycle after an appropriate rescaling of
coordinates and parameters can be brought to the form of an inverse to (1) map
(2). The correspondence between the parameters is
$\displaystyle\hat{B}=\frac{1}{B},\;\;\hat{M}_{1}=\frac{M_{1}}{B^{2}},\;\;\hat{M}_{2}=-\frac{M_{2}}{B}$.
This map is also well-known in homoclinic dynamics Gonchenko, Shilnikov, and
Turaev (1993); Tatjer (2001); Gonchenko, Shilnikov, and Turaev (1996, 2008b).
The conservative dynamics of both Henon maps (the case when $B=1$) was studied
in Ref. Lomelli and Meiss, 1999.

As map (2) is the inverse to (1), it automatically follows that it has a
discrete Lorenz repeller, which appears under perturbations of a fixed point
with multipliers $(-1,-1,+1)$. The “Simo’s angel” near a fixed point with
multipliers $(i,-i,-1)$ is also a repeller here for $\hat{B}<0$. It means,
that in order to find Lorenz-like attractors in map (2), one should look at
not only fixed points, but also periodic orbits. That is, to consider
$n$-periodic points such that in the $n$-th iterate of map (2) it is a fixed
point with multipliers $(-1,-1,+1)$ and a Jordan block.

###### Lemma 2

There exist parameter values, for which map $($2$)$ has periodic points of
period $6$ such that its $6$-th iterate $F^{6}$ has a fixed point with
multipliers $(-1,-1,+1)$. The normal form of this bifurcation is $($7$)$ with
$ab>0$.

Proof
Theoretical computations show that such periodic points do not exist for
periods $2$ and $3$. For periods $4$ and $5$, numerical computations show that
there exist parameter values such that $4$th and $5$th iterates of map (2)
have fixed points with eigenvalues $(-1,-1,+1)$, but in all of them, the
coefficients of normal form (7) give $ab<0$, this means that discrete Lorenz
repellers appear near these points. Now consider period-$6$ orbits. They it
will be orbits consisting of $6$ points $Z_{1}$–$Z_{6}$ with coordinates

$(z_{1},z_{2},z_{3})\to(z_{2},z_{3},z_{4})\to(z_{3},z_{4},z_{5})\to(z_{4},z_{5},z_{6})\to(z_{5},z_{6},z_{1})\to(z_{6},z_{1},z_{2})\to(z_{1},z_{2},z_{3}),$

which satisfy the following system of equations:

$\begin{array}[]{l}z_{1}=\hat{M}_{1}+\hat{M}_{2}z_{6}+\hat{B}z_{4}-z_{5}^{2},\qquad\qquad
z_{2}=\hat{M}_{1}+\hat{M}_{2}z_{1}+\hat{B}z_{5}-z_{6}^{2}\\\
z_{3}=\hat{M}_{1}+\hat{M}_{2}z_{2}+\hat{B}z_{6}-z_{1}^{2},\qquad\qquad
z_{4}=\hat{M}_{1}+\hat{M}_{2}z_{3}+\hat{B}z_{1}-z_{2}^{2}\\\
z_{5}=\hat{M}_{1}+\hat{M}_{2}z_{4}+\hat{B}z_{2}-z_{3}^{2},\qquad\qquad
z_{6}=\hat{M}_{1}+\hat{M}_{2}z_{5}+\hat{B}z_{3}-z_{4}^{2}.\\\ \end{array}$ (8)

Each of $6$-periodic points $Z_{1}$–$Z_{6}$ are fixed points of the sixth
iteration of map $F$, i.e. $x\to F^{6}(x)$. Next we will determine the
equations that guarantee that matrix $D(F^{6})$ possesses eigenvalues
$(-1,-1,+1)$ at these points. Namely, condition

$\det D(F^{6})(Z_{1})\equiv\hat{B}^{6}=1$ (9)

ensures that the product of the eigenvalues is equal to $1$, condition

$trD(F^{6})(Z_{1})\equiv tr\left(DF(Z_{6})\circ DF(Z_{5})\circ DF(Z_{4})\circ
DF(Z_{3})\circ DF(Z_{2})\circ DF(Z_{1})\right)=-1$ (10)

makes the sum of the eigenvalues to be equal to $-1$, and the third one

$\det(D(F^{6})(Z_{1})-id)=0$ (11)

means that $D(F^{6})(Z_{1})$ has an eigenvalue $+1$. Then equations (9) and
(10) imply that the product of the rest two multiplies is $1$, and their sum
is $-2$, so they both are equal to $-1$.

Formulas (8)–(11) define $9$ equations for $9$ unknowns
$z_{1},\ldots,z_{6},\hat{M}_{1},\hat{M}_{2},\hat{B}$, so the system of
equations is well-posed. One of numerical solutions of this system is

$\begin{array}[]{cc}z_{1}=1.1109087187819051,&z_{2}=0.5430803496704105,\\\
z_{3}=-0.018564282101437988,&z_{4}=-1.0126053862814206,\\\
z_{5}=-0.3759675295870319,&z_{6}=-0.6947447970072144,\\\
\hat{M}_{1}=0.3974562084897318,&\hat{M}_{2}=0.2271356235631268,\\\
\hat{B}=-1.\end{array}$ (12)

For the parameter values given by (12), the $6$-th iterate of the map near
point $(z_{1},z_{2},z_{3})$ can be written as normal form (7) with
$a=-0.0555732$ and $b=-1.6955$. According to Ref. Gonchenko  _et al._ , 2013,
Lemma 3.1, a discrete Lorenz attractor (of period $6$) is born in system (2)
near this bifurcation point for the orientation reversing case, as
$\hat{B}=-1$.

For the orientation preserving map (2), i.e. when $\hat{B}>0$, another
numerical solution of equations (8)–(11) was found:

$\begin{array}[]{cc}z_{1}=0.913442745966901,&z_{2}=1.220643948207064,\\\
z_{3}=1.3256709760748737,&z_{4}=1.1287783775951246,\\\
z_{5}=0.7765991221464961,&z_{6}=0.6638157026635255,\\\
\hat{M}_{1}=-0.9336687216264129,&\hat{M}_{2}=1.99067193080051,\\\
\hat{B}=1.\end{array}$ (13)

Normal form (7) has in this case coefficients $a=-0.107789$ and $b=-0.769823$,
thus $ab>0$, and near this bifurcation point also a period-$6$ discrete Lorenz
attractor is born.

##  IV The first return map and the rescaling lemma

Consider $U$ — a sufficiently small fixed neighborhood of the homoclinic or
heteroclinic cycle under consideration. It is a union of small neighbourhoods
${\textbf{U}_{0}}=U_{01}\cup U_{02}$ of the fixed points and small
neighbourhoods $U_{m}$ of all points of homoclinic or heteroclinic orbits
$\Gamma_{12}$ and $\Gamma_{21}$, that do not belong to ${\textbf{U}_{0}}$.
Note that there exists only a finite number of such points and neighbourhoods
$U_{m}$. Each single-round periodic orbit that lies entirely in $U$, has
exactly one intersection point with each of $U_{m}$ and all remaining its
points lie in ${\textbf{U}_{0}}$.

For each saddle $O_{j}$, $j=1,2$, select two points: $M_{j}^{+}\in
W^{s}_{loc}(O_{i})$ and $M_{j}^{-}\in W^{u}_{loc}(O_{i})$ and their respective
neighbourhoods $\Pi_{j}^{+},\Pi_{j}^{-}\subset U_{0j}$. The restriction of
diffeomorphism $f_{\mu}$ onto neighbourhoods $U_{0j}$ are called local maps
$T_{0j}$. Begin iterating $\Pi_{j}^{+}$ under the action of $T_{0j}$. Starting
from some number $\bar{k}_{j}$ images $T_{0j}^{k}\Pi_{j}^{+}$,
$k>\bar{k}_{j}$, will have nonempty intersections with $\Pi_{j}^{-}$. As
discussed in section II, there exist numbers $q_{1,2}$ such that
$M_{2}^{+}=f_{0}^{q_{2}}(M_{1}^{-})$, $M_{1}^{+}=f_{0}^{q_{1}}(M_{2}^{-})$.
For all small $\mu$ global maps are defined as
$T_{12}:\Pi_{1}^{-}\to\Pi_{2}^{+}=\left.f_{\mu}^{q_{2}}\right|_{\Pi_{2}^{-}}$,
$T_{21}:\Pi_{2}^{-}\to\Pi_{1}^{+}=\left.f_{\mu}^{q_{1}}\right|_{\Pi_{1}^{-}}$.
Now for every $k=(k_{1},k_{2})$, where $k_{j}>\bar{k}_{j}$, $j=1,2$, the first
return maps $T_{k}:V_{k}\to\Pi_{1}^{+}$ are defined as $T_{k}=T_{21}\circ
T_{02}^{k_{2}}\circ T_{12}\circ T_{01}^{k_{1}}$, where
$V_{k}\subset\Pi_{1}^{+}$ is a subdomain such that
$T_{01}^{k_{1}}(V_{k})\subset\Pi_{1}^{-}$, $T_{12}\circ
T_{01}^{k_{1}}(V_{k})\subset\Pi_{2}^{+}$, $T_{02}^{k_{2}}\circ T_{12}\circ
T_{01}^{k_{1}}(V_{k})\subset\Pi_{2}^{-}$, and
$T_{k}(V_{k})\subset\Pi_{1}^{+}$.

In order to write the first return map in coordinates, the local and global
maps should be represented in the most suitable form.

###  IV.1 Local maps

Table 1:  Local maps near fixed points of different types NN | Fixed point | Case # | The local map
---|---|---|---
1. | Saddle | I.2, II | (14)
2. | Saddle-focus | I–II.1 | (15)
3. | Resonant alternating saddle | I–II.3.a | (16)
4. | Resonant Belyakov saddle | I–II.3.b | (22)

In this subsection formulas for multiple iterations of local maps,
$T_{0j}^{k_{j}}$ are derived for different types of fixed points: saddle,
saddle-focus, saddle with the alternating resonance, saddle with the Belyakov
resonance. The summary with references to formulas is given in Table 1.

For a saddle fixed point $O_{j}$ with eigenvalues $\lambda_{1j}$,
$\lambda_{2j}$, $\gamma_{j}$, $|\lambda_{2j}|<|\lambda_{1j}|$, the local map
can be brought to the main normal form (5). This gives us the following
formula for its $k$-th iteration (see Refs. Shilnikov  _et al._ , 1998;
Gonchenko and Shilnikov, 1993 for details):

$\begin{array}[]{l}x_{1k}=\lambda_{(j)1}^{k}x_{10}+\hat{\lambda}_{j}^{k}\xi^{j}_{1k}(x_{0},y_{k},\mu),\\\
x_{2k}=\hat{\lambda}_{j}^{k}\xi^{j}_{2k}(x_{0},y_{k},\mu),\\\
y_{0}=\gamma_{(j)}^{-k}y_{k}+\hat{\gamma}_{j}^{-k}\xi^{j}_{3k}(x_{0},y_{k},\mu).\end{array}$
(14)

Here $0<|\lambda_{(j)2}|\leq\hat{\lambda}_{j}<|\lambda_{(j)1}|$,
$\hat{\gamma}_{j}>|\gamma_{(j)}|$, functions $\xi^{j}_{mk}$ and their
derivatives up to the order $(r-2)$ are uniformly bounded, and their higher
order derivatives tend to zero.

Case I–II.1. When $O_{j}$ is a saddle-focus with eigenvalues
$\lambda_{(j)}e^{\pm i\varphi_{j}}$, $\gamma_{(j)}$, where $i^{2}=-1$, the
$k$-th iteration of the local map has the form

$\begin{array}[]{l}(x_{1k},x_{2k})^{\top}=\lambda_{(j)}^{k}R_{k\varphi_{j}}(x_{10},x_{20})^{\top}+\hat{\lambda}_{j}^{k}\xi^{j}_{1k}(x_{0},y_{k},\mu),\\\
y_{0}=\gamma_{(j)}^{-k}y_{k}+\hat{\gamma}_{j}^{-k}\xi^{j}_{2k}(x_{0},y_{k},\mu),\end{array}$
(15)

where $R_{\psi}$ is the rotation matrix of angle $\psi$.

I–II.3.a. For the case of a resonant saddle with eigenvalues
$\lambda_{(j)1}(0)=-\lambda_{(j)2}(0)=\lambda_{(j)}$ and $\gamma_{(j)}$, the
$k$-th iteration can be written as

$\begin{array}[]{l}x_{1k}=\lambda_{(j)1}^{k}x_{10}+\hat{\lambda}_{j}^{k}\xi^{j}_{1k}(x_{0},y_{k},\mu)\\\
x_{2k}=\lambda_{(j)2}^{k}x_{20}+\hat{\lambda}_{j}^{k}\xi^{j}_{2k}(x_{0},y_{k},\mu)\\\
y_{0}=\gamma_{(j)}^{-k}y_{k}+\hat{\gamma}_{j}^{-k}\xi^{j}_{3k}(x_{0},y_{k},\mu),\end{array}$
(16)

where $0<\hat{\lambda}_{j}<|\lambda_{(j)}|$. Parameter $\mu_{2}$ unfolds the
resonance condition:

$\frac{\lambda_{(j)1}}{\lambda_{(j)2}}=-1+\mu_{2}.$ (17)

I–II.3.b. In the case of the Belyakov-type bifurcation
$\lambda_{(j)1}(0)=\lambda_{(j)2}(0)=\lambda_{(j)}$, in order to construct
smooth parmetric families, it is not possible to use canonical Jordan forms
for saddle and saddle-focus Arnold (1971), as these two normal forms can not
be smoothly conjugated at the bifurcation moment. One of the possible smooth
conjugating parametric families is given by the following formula:

$Df_{\mu}(O_{j})=\left(\begin{array}[]{cc}A_{s}&0\\\
0&\gamma_{(j)}(\mu)\end{array}\right),\;\;A_{s}=\left(\begin{array}[]{cc}\lambda_{(j)}(\mu)&1\\\
\mu_{2}&\lambda_{(j)}(\mu)\end{array}\right).$ (18)

When $\mu_{2}>0$, the linearization matrix has real stable eigenvalues
$\lambda_{(j)}(\mu)\pm\sqrt{\mu_{2}}$, and when $\mu_{2}<0$, they form a
complex-conjugate pair $\lambda_{(j)}(\mu)\pm i\sqrt{-\mu_{2}}$. The $k$-th
power of matrix $A_{s}$ has the following form:

$A_{s}^{k}=\lambda_{(j)}(\mu)^{k}\left(1-\frac{\mu_{2}}{\lambda_{(j)}^{2}(\mu)}\right)^{k/2}\left(\begin{array}[]{cc}C_{k}(\mu)&S_{k}(\mu)\\\
\mu_{2}S_{k}(\mu)&C_{k}(\mu)\end{array}\right),$ (19)

where

$C_{k}=\left\\{\begin{array}[]{l}\cosh{k\varphi_{j}},\;\mu_{2}\geq 0\\\
\cos{k\varphi_{j}},\;\mu_{2}<0\end{array}\right.,\;\;S_{k}=\left\\{\begin{array}[]{l}\frac{\sinh{k\varphi_{j}}}{\sqrt{\mu_{2}}},\;\mu_{2}>0\\\
0,\;\mu_{2}=0\\\
-\frac{\sin{k\varphi_{j}}}{\sqrt{-\mu_{2}}},\;\mu_{2}<0,\end{array}\right.$
(20)

with

$\varphi_{j}=\left\\{\begin{array}[]{l}$arctanh${\frac{\sqrt{\mu_{2}}}{\lambda_{j}}},\;\mu_{2}\geq
0\\\
-\arctan{\frac{\sqrt{-\mu_{2}}}{\lambda_{j}}},\;\mu_{2}<0\end{array}\right.,$
(21)

and the $k$-th iteration of the local map is written as:

$\begin{array}[]{l}(x_{1k},x_{2k})^{\top}=A_{s}^{k}(x_{10},x_{20})^{\top}+\hat{\lambda}_{j}^{k}\xi^{j}_{1k}(x_{0},y_{k},\mu),\\\
y_{0}=\gamma_{(j)}^{-k}y_{k}+\hat{\gamma}_{j}^{-k}\xi^{j}_{2k}(x_{0},y_{k},\mu),\end{array}$
(22)

where $\hat{\lambda}_{j}<\lambda_{(j)}$.

All possible types of fixed points and the references to the corresponding
formulas for the local maps are given in Table 1.

###  IV.2 Global maps

Recall that the global map in the homoclinic cases maps neighbourhood
$\Pi^{-}$ to $\Pi^{+}$, and in the heteroclinic cases they map $\Pi_{1}^{-}$
to $\Pi_{2}^{+}$ and $\Pi_{2}^{-}$ to $\Pi_{1}^{+}$. Assume that the
homoclinic or heteroclinic points at $\mu=0$ have the following coordinates:
$M_{j}^{-}(0,0,y_{(j)}^{-})$ and $M_{j}^{+}(x_{(j)1}^{+},x_{(j)2}^{+},0)$,
where $x_{(j)1}^{+}$, $x_{(j)2}^{+}$ and $y_{(j)}^{-}$ depend on parameters,
and $(x_{(j)1}^{+})^{2}+(x_{(j)2}^{+})^{2}\neq 0$, $y_{(j)}^{-}\neq 0$. The
global maps are written as Taylor expansions near points $M_{j}^{+}$.

####  IV.2.1 Transversal heteroclinic intersections

Table 2:  Global maps for transversal intersections NN |  Connection |  Case # | Genericity conditions
---|---|---|---
1. |  Saddle $\to$ saddle, no orbit flip |  II | (28)
2. |  Saddle $\to$ saddle, orbit flip |  II.2.c | (27)
3. |  Saddle $\to$ saddle-focus
Saddle-focus $\to$ saddle
Saddle-focus $\to$ saddle-focus |  II.1 | None
4. |  Resonant alternating saddle $\to$ saddle |  II.3.a | (29)
5. |  Saddle $\to$ resonant alternating saddle |  II.3.a | (30)
6. |  Saddle $\to$ resonant Belyakov saddle |  II.3.b | (31)
7. |  Resonant Belyakov saddle $\to$ saddle |  II.3.b | (28)

Consider first transversal heterolinic intersections along orbit $\Gamma_{12}$
that appear in case II. Unstable manifold $W^{u}(O_{1})$ in $\Pi_{1}^{-}$ has
equation $x_{(1)1}=x_{(1)2}=0$, and under the action of global map $T_{12}$ it
is transformed to a curve that intersect transversely stable manifold
$W^{s}(O_{2})$, which locally in $\Pi_{2}^{+}$ has equation $y_{(2)}=0$. Thus
one can write $T_{12}$ as follows:

$\begin{array}[]{rcl}x_{(2)1}-x_{(2)1}^{+}&=&a_{11}^{(1)}x_{(1)1}+a_{12}^{(1)}x_{(1)2}+b_{1}^{(1)}(y_{(1)}-y_{(1)}^{-})+O(\|x_{(1)}\|^{2}+|y_{(1)}-y_{(1)}^{-}|^{2})\\\
x_{(2)2}-x_{(2)2}^{+}&=&a_{21}^{(1)}x_{(1)1}+a_{22}^{(1)}x_{(1)2}+b_{2}^{(1)}(y_{(1)}-y_{(1)}^{-})+O(\|x_{(1)}\|^{2}+|y_{(1)}-y_{(1)}^{-}|^{2})\\\
y_{(2)}&=&y^{+}_{(1)}+c_{1}^{(1)}x_{(1)1}+c_{2}^{(1)}x_{(1)2}+d^{(1)}(y_{(1)}-y_{(1)}^{-})+O(\|x_{(1)}\|^{2}+|y_{(1)}-y_{(1)}^{-}|^{2}).\\\
\end{array}$ (23)

Here all coefficients depend smoothly on parameters, and $y^{+}_{(1)}(0)=0$,
$d^{(1)}(0)\neq 0$, as the intersection is transversal. Map $T_{12}$ is a
diffeomorphism, therefore its Jacobian $DT_{12}$ at $M^{-}_{1}$ is non-
degenerate, i.e.

$\det
DT_{12}=\det\left(\begin{array}[]{ccc}a_{11}^{(1)}&a_{12}^{(1)}&b_{1}^{(1)}\\\
a_{21}^{(1)}&a_{22}^{(1)}&b_{2}^{(1)}\\\
c_{1}^{(1)}&c_{2}^{(1)}&d^{(1)}\end{array}\right)\neq 0.$ (24)

In case II.2.c the transversal intersection has an additional degeneracy at
the bifurcation moment — an orbit flip, in coordinates the condition of a
simple and non-simple heteroclinic orbit $\Gamma_{12}$ is obtained as follows.
The equation of extended unstable manifold $W^{ue}_{loc}(O_{1})$ is
$x_{(1)2}=0$, and the leaf $F^{ss}(M^{+}_{2})$ passing through point
$M^{+}_{2}$ is locally a straight line $\\{x_{(2)1}=x^{+}_{(2)1}$,
$y_{(2)}=0\\}$ with direction vector $l^{ss}=(0,1,0)^{\top}$. Tangent plane
$P^{ue}(M^{-}_{1})$ has equation $x_{(1)2}=0$, and its image under the action
of global map $T_{12}$ has at $\mu=0$ the following parametric equation:

$\begin{array}[]{rcl}x_{(2)1}-x_{(2)1}^{+}&=&a_{11}^{(1)}x_{(1)1}+b_{1}^{(1)}(y_{(1)}-y_{(1)}^{-})+O(\|x_{(1)}\|^{2}+|y_{(1)}-y_{(1)}^{-}|^{2})\\\
x_{(2)2}-x_{(2)2}^{+}&=&a_{21}^{(1)}x_{(1)1}+b_{2}^{(1)}(y_{(1)}-y_{(1)}^{-})+O(\|x_{(1)}\|^{2}+|y_{(1)}-y_{(1)}^{-}|^{2})\\\
y_{(2)}&=&c_{1}^{(1)}x_{(1)1}+d^{(1)}(y_{(1)}-y_{(1)}^{-})+O(\|x_{(1)}\|^{2}+|y_{(1)}-y_{(1)}^{-}|^{2}).\\\
\end{array}$ (25)

At point $M^{+}_{2}$ it has two linearly independent tangent vectors
$l_{1}=(a_{11}^{(1)},a_{21}^{(1)},c_{1}^{(1)})^{\top}$ and
$l_{2}=(b_{1}^{(1)},b_{2}^{(1)},d^{(1)})^{\top}$. Curve $F^{ss}(M^{+}_{2})$
and surface $T_{12}(P^{ue}(M^{-}_{1}))$ will be tangent at point $M^{+}_{2}$
if vectors $l_{1}$, $l_{2}$ and $l^{ss}$ are linearly dependent, this happens
when

$\left.A_{11}^{(1)}(\mu)\right|_{\mu=0}=\left.\left(a_{11}^{(1)}(\mu)-\frac{b_{1}^{(1)}(\mu)c_{1}^{(1)}(\mu)}{d^{(1)}(\mu)}\right)\right|_{\mu=0}=0.$
(26)

So in case II.2.c, when the heteroclinic orbit connecting saddles $O_{1}$ and
$O_{2}$ is non-simple, parameter $\mu_{2}$ is introduced to unfold the orbit
flip degeneracy as

$\mu_{2}\equiv A_{11}^{(1)}(\mu).$ (27)

When transversal heteroclinic orbit $\Gamma_{12}$ is simple, it should satisfy
the non-degeneracy condition

$A_{11}^{(1)}(0)\neq 0.$ (28)

If $O_{1}$ is a saddle with an alternating resonance (case II.3.a), then due
to switching of leading and non-leading directions for small $\mu$,
$\Gamma_{12}$ will be simple if

$A_{11}^{(1)}(0)\neq
0,\;\;\left.A_{12}^{(1)}(\mu)\right|_{\mu=0}=\left.\left(a_{12}^{(1)}(\mu)-\frac{b_{1}^{(1)}(\mu)c_{2}^{(1)}(\mu)}{d^{(1)}(\mu)}\right)\right|_{\mu=0}\neq
0.$ (29)

If $O_{2}$ is a saddle with an alternating resonance, the genericity
conditions are

$A_{11}^{(1)}(0)\neq
0,\;\;\left.A_{21}^{(1)}(\mu)\right|_{\mu=0}=\left.\left(a_{21}^{(1)}(\mu)-\frac{b_{2}^{(1)}(\mu)c_{1}^{(1)}(\mu)}{d^{(1)}(\mu)}\right)\right|_{\mu=0}\neq
0.$ (30)

If $O_{1}$ is a saddle undergoing the Belyakov transition (case II.3.b), then
the leading stable direction at $O_{1}$ tends to the $x_{(1)1}$ axis as
$\mu_{2}\to+0$, so that condition (28) guarantees the absence of orbit flips
in small perturbations. If $O_{2}$ undergoes the Belyakov transition, then its
non-leading direction tends to the $x_{(2)1}$ axis in the limit
$\mu_{2}\to+0$. In this case the heteroclinic orbit will be simple if

$A_{21}^{(1)}(0)\neq 0.$ (31)

All possible cases of transverse intersections together with the references to
the corresponding non-degeneracy conditions are summarized in Table 2.

####  IV.2.2 Quadratic homoclinic and heteroclinic tangencies

Table 3:  Global maps for quadratic tangencies NN |  Connection |  Case # | Genericity conditions
---|---|---|---
1. |  Saddle $\to$ saddle, simple tangency |  II | (40)
2. |  Saddle $\to$ saddle, inclination flip |  I–II.2.a | (37), (39)
3. |  Saddle $\to$ saddle, orbit flip |  I–II.2.b | (38), (39)
4. |  Saddle $\to$ saddle-focus |  II.1 | (41)
5. |  Saddle-focus $\to$ saddle |  II.1 | None
6. |  Saddle-focus $\to$ saddle-focus |  I–II.1 | None
7. |  Saddle $\to$ resonant alternating saddle
Resonant alternating saddle $\to$ saddle
Resonant alternating saddle $\to$ itself |  I–II.3.a | (42)
8. |  Saddle $\to$ resonant Belyakov saddle
Resonant Belyakov saddle $\to$ itself |  I–II.3.b | (43)
9. |  Resonant Belyakov saddle $\to$ saddle |  II.3.b | (40)

The nontransversal heteroclinic (homoclinic) orbit connects fixed points
$O_{2}$ and $O_{1}$. When $\mu=0$, global map $T_{21}$ transforms a piece of
unstable manifold $W^{u}(O_{2})\cap\Pi^{-}_{2}$ with equation $x=0$ into a
curve tangent at point $M^{+}_{1}$ to surface $W^{s}(O_{1})\cap\Pi^{+}_{1}$
with equation $y=0$. Then for all small $\mu$ global map $T_{21}$ is written
as

$\begin{array}[]{rcl}\bar{x}_{(1)1}-x_{(1)1}^{+}&=&a_{11}^{2}x_{(2)1}+a_{12}^{2}x_{(2)2}+b_{1}^{2}(y_{(2)}-y_{(2)}^{-})+O(\|x_{(2)}\|^{2}+|y_{(2)}-y_{(2)}^{-}|^{2})\\\
\bar{x}_{(1)2}-x_{(1)2}^{+}&=&a_{21}^{2}x_{(2)1}+a_{22}^{2}x_{(2)2}+b_{2}^{2}(y_{(2)}-y_{(2)}^{-})+O(\|x_{(2)}\|^{2}+|y_{(2)}-y_{(2)}^{-}|^{2})\\\
\bar{y}_{(1)1}&=&y^{+}_{(2)}+c_{1}^{2}x_{(2)1}+c_{2}^{2}x_{(2)2}+d^{2}(y_{(2)}-y_{(2)}^{-})^{2}+O(\|x_{(2)}\|^{2}+|y_{(2)}-y_{(2)}^{-}|^{3})\\\
\end{array}$ (32)

The left hand side variables are denoted here as
$(\bar{x}_{(1)},\bar{y}_{(1)})$ to indicate that the image of $T_{21}$ lies in
$\Pi_{1}^{+}$ and these coordinates also represent the iteration of the first
return map $T_{k}$ from $\Pi_{1}^{+}$ to itself. All coefficients here depend
smoothly on parameters, and when $\mu=0$ we have $y^{+}_{(2)}(0)=0$ and
$d^{2}(0)\neq 0$, as the tangency is quadratic at the bifurcation moment. The
Jacobian of the global map $DT_{21}$ at $M^{-}_{2}$ is non-degenerate, that is

$\det DT_{21}=\det\left(\begin{array}[]{ccc}a_{11}^{2}&a_{12}^{2}&b_{1}^{2}\\\
a_{21}^{2}&a_{22}^{2}&b_{2}^{2}\\\ c_{1}^{2}&c_{2}^{2}&0\end{array}\right)\neq
0$ (33)

When $\mu\neq 0$, value $y^{+}_{(2)}(\mu)$ is the splitting distance of the
quadratic tangency up to $o\|\mu\|$ terms, so it is taken as the splitting
parameter:

$\mu_{1}\equiv y^{+}_{(2)}(\mu).$ (34)

Now write in coordinates the conditions of simple and non-simple quadratic
tangencies. Consider saddle fixed points $O_{2}$ and $O_{1}$ such that all
their eigenvalues are real and do not satisfy resonance conditions from cases
I–II.3. The equation of extended unstable manifold $W^{ue}_{loc}(O_{2})$ is
$x_{(2)2}=0$, and the leaf $F^{ss}(M^{+}_{1})$ passing through point
$M^{+}_{1}$ is locally a straight line $\bar{x}_{(1)1}=x^{+}_{(1)1}$,
$\bar{y}_{(1)}=0$ with direction vector $l^{ss}=(0,1,0)^{\top}$. The image of
tangent plane $P^{ue}(M^{-}_{2})$ under the action of global map $T_{21}$ has
the following parametric equation:

$\begin{array}[]{rcl}\bar{x}_{(1)1}-x_{(1)1}^{+}&=&a_{11}^{2}x_{(2)1}+b_{1}^{2}(y_{(2)}-y_{(2)}^{-})+O(\|x_{(2)}\|^{2}+|y_{(2)}-y_{(2)}^{-}|^{2})\\\
\bar{x}_{(1)2}-x_{(1)2}^{+}&=&a_{21}^{2}x_{(2)1}+b_{2}^{2}(y_{(2)}-y_{(2)}^{-})+O(\|x_{(2)}\|^{2}+|y_{(2)}-y_{(2)}^{-}|^{2})\\\
\bar{y}_{(1)}&=&c_{1}^{2}x_{(2)1}+d^{2}(y_{(2)}-y_{(2)}^{-})^{2}+O(\|x_{(2)}\|^{2}+|y_{(2)}-y_{(2)}^{-}|^{2}).\\\
\end{array}$ (35)

At point $M^{+}_{1}$ it has two linearly independent tangent vectors
$l_{1}=(a_{11}^{2},a_{21}^{2},c_{1}^{2})^{\top}$ and
$l_{2}=(b_{1}^{2},b_{2}^{2},0)^{\top}$. Curve $F^{ss}(M^{+}_{1})$ and surface
$T_{21}(P^{ue}(M^{-}_{2}))$ will be tangent at point $M^{+}_{1}$ if vectors
$l_{1}$, $l_{2}$ and $l^{ss}$ are linearly dependent, this happens when

$\left.b_{1}^{2}(\mu)c_{1}^{2}(\mu)\right|_{\mu=0}=0.$ (36)

So here naturally two possibilities appear for the quadratic tangency to be
non-simple. In the inclination flip cases I–II.2.a, surfaces
$T_{21}(P^{ue}(M^{-}_{2}))$ and $W^{s}(O_{1})$ are tangent to each other (fig.
3 (a)), therefore vectors $l_{1}$ and $l_{2}$ both lie in $W^{s}(O_{1})$, thus

$c_{1}^{2}(0)=0,\;b_{1}^{2}(0)\neq 0,$ (37)

and in the orbit flip cases I–II.2.b, when surface $T_{21}(P^{ue}(M^{-}_{2}))$
is transverse to $W^{s}(O_{1})$, (fig. 3 (b)), it follows that

$b_{1}^{2}(0)=0,\;c_{1}^{2}(0)\neq 0,$ (38)

which implies that vectors $l_{ss}$ and $l_{2}$ are parallel.

For these types of degeneracies the second unfolding parameter $\mu_{2}$ is
introduced as

$\mu_{2}=\left\\{\begin{array}[]{rl}c_{1}^{2}(\mu)&{\rm in\;cases\;\textbf{I--
II.2.a}}\\\ b_{1}^{2}(\mu)&{\rm in\;cases\;\textbf{I--
II.2.b}}\end{array}\right.$ (39)

When the quadratic tangency is simple the condition of absence of non-simple
tangencies at the bifurcation moment and in small perturbations should be
written. If points $O_{1}$ and $O_{2}$ are saddles, and they do not satisfy
resonance conditions from cases cases I–II.3, then

$b_{1}^{2}(0)\neq 0,\;c_{1}^{2}(0)\neq 0.$ (40)

If point $O_{2}$ is a saddle and $O_{1}$ is a saddle-focus, only the
inclination flip degeneracy is possible, when manifold $W^{ue}(O_{2})$ is
tangent to stable manifold $W^{s}(O_{1})$. To avoid this, one needs:

$c_{1}^{2}(0)\neq 0.$ (41)

If one of the points $O_{1}$ and $O_{2}$ at the bifurcation moment is a saddle
with the alternating resonance, cases I–II.3.a, then either the direction of
$W^{ue}(O_{2})$, or the direction of $W^{ss}(O_{1})$ may alternate when $\mu$
varies, thus the quadratic tangency is simple if

$b_{1}^{2}(0)\neq 0,\;b_{2}^{2}(0)\neq 0,\;c_{1}^{2}(0)\neq
0,\;c_{2}^{2}(0)\neq 0.$ (42)

If point $O_{2}$ satisfies the Belyakov condition, case II.3.b, then
inequalities (40) should be fulfilled to avoid non-simple tangencies, and if
$O_{1}$ satisfies the Belyakov condition (this also inlcludes the homoclinic
case I.3.b), the quadratic tangency will be simple if

$b_{2}^{2}(0)\neq 0,\;c_{1}^{2}(0)\neq 0.$ (43)

All possible cases of quadratic tangencies together with the references to the
corresponding non-degeneracy conditions are summarized in Table 3.

###### Lemma 3

(The rescaling lemma) Let $f_{\mu_{1},\mu_{2},\mu_{3}}$ be the family under
consideration. Then, in the space $(\mu_{1},\mu_{2},\mu_{3})$ there exist
infinitely many regions $\Delta_{i}$ in the homoclinic case I and
$\Delta_{ij}$ in the heteroclinic case II accumulating to the origin as
$i,j\to\infty$, such that the first return map in appropriate rescaled
coordinates and parameters is asymptotically $C^{r-1}$-close to one of the
following limit maps.

1) In the orbit flip cases I–II.2.b:

$\begin{array}[]{l}\bar{X}_{1}\;=\;-BX_{2}+M_{2}Y,\;\;\bar{X}_{2}\;=\;Y,\;\;\bar{Y}=M_{1}-X_{1}-Y^{2},\end{array}$
(44)

2) In all other cases:

$\begin{array}[]{l}\bar{X}_{1}\;=\;Y,\;\;\bar{X}_{2}\;=\;X_{1},\;\;\bar{Y}=M_{1}+M_{2}X_{1}+BX_{2}-Y^{2},\end{array}$
(45)

Thus, the rescaled first return map in almost all cases is exactly the 3D
Henon map (1). In cases I–II.2.b in system (44) we make an additional change
of coordinates $X_{1new}=X_{1}-M_{2}X_{2}$ and scale $X_{1}$ by $(-B)$,
bringing it again to the form (45).

The relations between old and new parameters are the following.

$M_{1}\sim\left\\{\begin{array}[]{rl}\mu_{1}\gamma^{2i}&{\rm
in\;case\;\textbf{I}}\\\ \mu_{1}\gamma_{(1)}^{2i}\gamma_{(2)}^{2j}&{\rm
in\;case\;\textbf{II}}.\end{array}\right.$ (46)

When $i,j\to\infty$, with sufficiently small variations of parameter $\mu_{1}$
one gets arbitrary finite values of parameter $M_{1}$.

$B\sim\left\\{\begin{array}[]{rl}J^{i}(O)\det DT_{1}&{\rm
in\;case\;\textbf{I}}\\\ J^{i}(O_{1})J^{j}(O_{2})\det DT_{12}\det DT_{21}&{\rm
in\;case\;\textbf{II}}.\end{array}\right.$ (47)

Based on formulas (3) and (4), by small variations of parameter $\mu_{3}$
parameter $B$ takes arbitrary finite values. If the original diffeomorphism
$f_{0}$ is orientation preserving, $B$ takes only positive values, if $f_{0}$
is orientation reversing, then $B$ takes either only positive or only negative
values, depending on the orientability of the first return map.

$M_{2}\sim\left\\{\begin{array}[]{rl}\lambda_{1}^{i}\gamma^{i}\cos(i\varphi+\theta)&{\rm
in\;case\;\textbf{I.1}}\\\
\lambda_{(1)1}^{i}\gamma_{(1)}^{i}\lambda_{(2)1}^{j}\gamma_{(2)}^{j}\cos(i\varphi_{1}+\theta_{1})\cos(j\varphi_{2}+\theta_{2})&{\rm
in\;case\;\textbf{II.1}},\end{array}\right.$ (48)

where $\theta$, $\theta_{1}$ and $\theta_{2}$ smoothly depend on parameters
and $\mu_{2}$ is varied in the way that the trigonometric function stays close
to zero. At the same time, according to formulas (3) and (4), the coefficients

$\lambda_{1}^{i}\gamma^{i}\sim\lambda_{2}^{-i}\;\;{\rm
and}\;\;\lambda_{(1)1}^{i}\gamma_{(1)}^{i}\lambda_{(2)1}^{j}\gamma_{(2)}^{j}\sim\lambda_{(1)2}^{-i}\lambda_{(2)2}^{-j}$

are asymptotically large when $i,j\to\infty$. Thus parameter $M_{2}$ takes
arbitrary finite values.

$M_{2}\sim\left\\{\begin{array}[]{rl}\mu_{2}\lambda_{1}^{i}\gamma^{i}&{\rm
in\;case\;\textbf{I.2}}\\\
\mu_{2}\lambda_{(1)1}^{i}\gamma_{(1)}^{i}\lambda_{(2)1}^{j}\gamma_{(2)}^{j}&{\rm
in\;case\;\textbf{II.2}}.\end{array}\right.$ (49)

Again, for $i,j\to\infty$ and sufficiently small $\mu_{2}$ parameter $M_{2}$
takes arbitrary finite values.

$M_{2}\sim\left\\{\begin{array}[]{rl}\displaystyle\lambda_{1}^{i}\gamma^{i}\left((-1+\mu_{2})^{i}+A\right)&{\rm
in\;case\;\textbf{I.3.a}}\\\
\displaystyle\lambda_{(1)1}^{i}\gamma_{(1)}^{i}\lambda_{(2)1}^{j}\gamma_{(2)}^{j}\left((-1+\mu_{2})^{k}+A\right)&{\rm
in\;case\;\textbf{II.3.a}}.\\\ \end{array}\right.$ (50)

Here value $A$ smoothly depends on the parameters, and $A\neq 0$ when $\mu=0$.
The power $k$ denotes $i$ or $j$ depending on which saddle point, $O_{1}$ or
$O_{2}$, satisfies the resonance condition. Here, to make $M_{2}$ finite,
parameter $\mu_{2}$ is varied near such values, where
$\left((-1+\mu_{2})^{k}+A\right)$ becomes zero. To achieve this, the parity of
$k$ is taken appropriately, depending on the sign of $A$.

$M_{2}\sim\left\\{\begin{array}[]{rl}\displaystyle\lambda_{1}^{i}\gamma^{i}\left(\frac{A}{\sqrt{-\mu_{2}}}\cos(i\varphi+\theta)\right)&{\rm
in\;case\;\textbf{I.3.b}}\\\
\displaystyle\lambda_{(1)1}^{i}\gamma_{(1)}^{i}\lambda_{(2)1}^{j}\gamma_{(2)}^{j}\left(\frac{A}{\sqrt{-\mu_{2}}}\cos(k\varphi+\theta)\right)&{\rm
in\;case\;\textbf{II.3.b}}.\end{array}\right.$ (51)

This formula is valid only when $\mu_{2}<0$, which means that the saddle point
having a stable eigenvalue with multiplicity two (the Belyakov resonance),
becomes a saddle-focus. Here $A$ and $\theta$ smoothly depend on the
parameters, moreover $A\neq 0$ when $\mu=0$. Exponent $k$ is $i$ or $j$
depending on which saddle point, $O_{1}$ or $O_{2}$, satisfies the resonance
condition. The angle variable $\varphi$ is given by formula (21). Varying a
small $\mu_{2}$ near one of the zeros of the trigonometric function, and at
the same time, keeping it away from zero, one get parameter $M_{2}$ taking
arbitrary finite values.

## Acknowledgements

This paper is a contribution to the project M7 (Dynamics of Geophysical
Problems in Turbulent Regimes) of the Collaborative Research Centre TRR 181
“Energy Transfer in Atmosphere and Ocean” funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) – Projektnummer
274762653. The paper is also supported by the grant of the Russian Science
Foundation 19-11-00280. I especially thank S. Gonchenko and D. Turaev for the
idea of considering bifurcations of periodic points in the inverse 3D Henon
map (2). In addition, I am grateful to Jean-Luc Doumont for the seminars on
scientific writing, that helped me to significantly improve the manuscript.

## Data Availability Statement

The data that support the findings of this study (proofs, pictures) are placed
in the body of the text. If some extra requirements appear, they should be
addressed to the corresponding author.

## References

  * Lorenz (1986) E. N. Lorenz, “The index cycle is alive and well,” in  _Namias Symposium_ , SIO Reference 86-17, edited by J. O. Roads, Scripps Institution of Oceanography (California. Univ., La Jolla, 1986) pp. 188–196.
  * Palmer (1998) T. Palmer, “Nonlinear dynamics and climate change: Rossby’s legacy,” Bulletin of the American Meteorological Society 79, 1411–1423 (1998).
  * Pedlosky and Frenzen (1980) J. Pedlosky and C. Frenzen, “Chaotic and periodic behavior of finite-amplitude baroclinic waves,” Journal of the Atmospheric Sciences 37, 1177–1196 (1980).
  * Pedlosky (2011) J. Pedlosky, “The nonlinear downstream development of baroclinic instability,” Journal of Marine Research 69, 705–722 (2011).
  * Eilertsen and Magnan (2018) J. S. Eilertsen and J. F. Magnan, “Asymptotically exact codimension-four dynamics and bifurcations in two-dimensional thermosolutal convection at high thermal Rayleigh number: Chaos from a quasi-periodic homoclinic explosion and quasi-periodic intermittency,” Physica D: Nonlinear Phenomena 382–383, 1–21 (2018).
  * Gonchenko, Gonchenko, and Kazakov (2013) A. S. Gonchenko, S. V. Gonchenko, and A. O. Kazakov, “Richness of chaotic dynamics in nonholonomic models of a Celtic stone,” Regul. Chaotic Dyn. 18, 521–538 (2013).
  * Vladimirov and Volkov (1993) A. G. Vladimirov and D. Y. Volkov, “Low-intensity chaotic operations of a laser with a saturable absorber,” Optics Communications 100, 351–360 (1993).
  * Aframovich and Shilnikov (1983) V. S. Aframovich and L. P. Shilnikov, “Strange attractors and quasiattractors,” in  _Nonlinear Dynamics and Turbulence_ , edited by G. I. Barenblatt, G. Iooss, and D. D. Joseph (Pitman, Boston, 1983) p. 1–34.
  * Newhouse, Palis, and Takens (1983) S. E. Newhouse, J. Palis, and F. Takens, “Bifurcations and stability of families of diffeomorphisms,” Publ. Math. Inst. Haute Etudes Scientifiques 57, 5–72 (1983).
  * Gonchenko, Turaev, and Shilnikov (1993) S. V. Gonchenko, D. V. Turaev, and L. P. Shilnikov, “On the existence of newhouse regions near systems with non-rough Poincare homoclinic curve (multidimensional case),” Russian Acad. Sci. Dokl. Math. 329, 404–407 (1993).
  * Ruelle (1981) D. Ruelle, “Small random perturbations of dynamical systems and the definition of attractors,” Comm. Math. Phys. 82, 137–151 (1981).
  * Conley (1978) C. Conley, “Isolated invariant sets and the morse index,” in  _CBMS Regional Conference Series Math._ , Vol. 38 (Am. Math. Soc., Providence, 1978).
  * Turaev and Shilnikov (1998) D. V. Turaev and L. P. Shilnikov, “An example of a wild strange attractor,” Sb. Math. 189, 137–160 (1998).
  * Turaev and Shilnikov (2008) D. V. Turaev and L. P. Shilnikov, “Pseudo-hyperbolisity and the problem on periodic perturbations of lorenz-like attractors,” Russian Dokl. Math. 467, 23–27 (2008), [Doklady Mathematics 77, 1 (2008), 17–22].
  * Afraimovich, Bykov, and Shilnikov (1977) V. S. Afraimovich, V. V. Bykov, and L. P. Shilnikov, “The origin and structure of the Lorenz attractor,” Sov. Phys. Dokl. 22, 253–255 (1977).
  * Afraimovich, Bykov, and Shilnikov (1982) V. S. Afraimovich, V. V. Bykov, and L. P. Shilnikov, “On attracting structurally unstable limit sets of Lorenz attractor type,” Trans. Mosc. Math. Soc. 44, 153–216 (1982).
  * Gonchenko  _et al._ (2005) S. V. Gonchenko, I. I. Ovsyannikov, C. Simó, and D. Turaev, “Three-dimensional Hénon-like maps and wild Lorenz-like attractors,” International Journal of Bifurcation and Chaos 15, 3493–3508 (2005).
  * Gonchenko, Meiss, and Ovsyannikov (2006) S. V. Gonchenko, J. D. Meiss, and I. I. Ovsyannikov, “Chaotic dynamics of three-dimensional Hénon maps that originate from a homoclinic bifurcation,” Regul. Chaotic Dyn. 11, 191–212 (2006).
  * Gonchenko  _et al._ (2013) S. V. Gonchenko, A. S. Gonchenko, I. I. Ovsyannikov, and D. V. Turaev, “Examples of Lorenz-like attractors in Hénon-like maps,” Mat. Model. of Nat. Phenom 8, 48–70 (2013).
  * Gonchenko  _et al._ (2021a) S. Gonchenko, E. Karatetskaya, A. Kazakov, K. Safonov, and D. Turaev, (2021a), (in preparation).
  * Gonchenko  _et al._ (2021b) S. Gonchenko, A. Gonchenko, A. Kazakov, and E. Samylina, “On discrete Lorenz-like attractors,” Chaos 31, 023117 (2021b).
  * Gonchenko, Shilnikov, and Turaev (2009) S. V. Gonchenko, L. Shilnikov, and D. Turaev, “On global bifurcations in three-dimensional diffeomorphisms leading to wild Lorenz-like attractors,” Regul. Chaotic Dyn. 14, 137–147 (2009).
  * Gonchenko and Ovsyannikov (2010) S. V. Gonchenko and I. I. Ovsyannikov, “On bifurcations of three-dimensional diffeomorphisms with a non-transversal heteroclinic cycle containing saddle-foci,” Rus. J. Nonlinear Dynamics 6, 61–77 (2010).
  * Gonchenko and Ovsyannikov (2013) S. V. Gonchenko and I. I. Ovsyannikov, “Global bifurcations of three-dimensional diffeomorphisms leading to Lorenz-like attractors,” Mat. Model. of Nat. Phenom 8, 71–83 (2013).
  * Gonchenko, Ovsyannikov, and Tatjer (2014) S. V. Gonchenko, I. I. Ovsyannikov, and J. C. Tatjer, “Birth of discrete Lorenz attractors at the bifurcations of 3D maps with homoclinic tangencies to saddle points,” Regular and Chaotic Dynamics 19, 495–505 (2014).
  * Ovsyannikov (2017) I. Ovsyannikov, “On birth of discrete Lorenz attractors under bifurcations of 3D maps with nontransversal heteroclinic cycles,” https://arxiv.org/abs/1705.04621 (2017), submitted to Regular and Chaotic Dynamics.
  * Gonchenko and Ovsyannikov (2017) S. V. Gonchenko and I. I. Ovsyannikov, “Homoclinic tangencies to resonant saddles and discrete Lorenz attractors,” Discrete and continuous dynamical systems, series S 10, 273–288 (2017).
  * Turaev (1996) D. V. Turaev, “On dimension of nonlocal bifurcational problems,” Int. J. of Bifurcation and Chaos 6, 919–948 (1996).
  * Gonchenko, Shilnikov, and Turaev (1993) S. V. Gonchenko, L. P. Shilnikov, and D. V. Turaev, “Dynamical phenomena in multidimensional systems with a structurally unstable homoclinic poincare curve,” Russian Acad. Sci. Dokl. Math. 47, 410–415 (1993).
  * Tatjer (2001) J. C. Tatjer, “Three-dimensional dissipative diffeomorphisms with homoclinic tangencies,” Ergod. Th. & Dynam. Sys. 21, 249–302 (2001).
  * Gonchenko (2000) S. Gonchenko, “Dynamics and moduli of $\Omega$-conjugacy of 4D-diffeomorphisms with a structurally unstable homoclinic orbit to a saddle-focus fixed point,” AMS Transl. Math. 200, 107–134 (2000).
  * Gonchenko, Shilnikov, and Turaev (2008a) S. V. Gonchenko, L. P. Shilnikov, and D. V. Turaev, “On dynamical properties of multidimensional diffeomorphisms from Newhouse regions: I,” Nonlinearity 21, 923–972 (2008a).
  * Gonchenko and Shilnikov (1990) S. V. Gonchenko and L. P. Shilnikov, “Invariants of $\Omega$-conjugacy of diffeomorphisms with a nontransversal homoclinic orbit,” Ukr. Math. J. 42, 134–140 (1990).
  * Gonchenko and Shilnikov (1993) S. V. Gonchenko and L. P. Shilnikov, “On moduli of systems with a nontransversal Poincare homoclinic orbit,” Russian Acad. Sci. Izv. Math. 41, 417–445 (1993).
  * Shilnikov  _et al._ (1998) L. P. Shilnikov, A. L. Shilnikov, D. V. Turaev, and L. O. Chua,  _Methods of Qualitative Theory in Nonlinear Dynamics, Part I_ , World Scientific Series on Nonlinear Science A, Vol. 4 (World Scientific, 1998).
  * Hirsch, Pugh, and Shub (1977) M. W. Hirsch, C. C. Pugh, and M. Shub,  _Invariant manifolds_ , Lecture Notes in Math, Vol. 583 (Springer-Verlag, Berlin Heidelberg, 1977).
  * Belyakov (1980) L. A. Belyakov, “On a bifurcation set in systems with a homoclinic curve of a saddle,” Math. Notes 28, 910–916 (1980).
  * Kalia, Kuznetsov, and Meier (2019) M. Kalia, Y. Kuznetsov, and H. Meier, “Homoclinic saddle to saddle-focus transitions in 4D systems,” Nonlinearity 32, 2024 (2019).
  * Shilnikov, Shilnikov, and Turaev (1993) A. L. Shilnikov, L. P. Shilnikov, and D. V. Turaev, “Normal forms and Lorenz attractors,” Int. J. of Bifurcation and Chaos 3, 1123–1139 (1993).
  * Capiński, Turaev, and Zgliczyński (2018) M. Capiński, D. Turaev, and P. Zgliczyński, “Computer assisted proof of the existence of the Lorenz attractor in the Shimizu-Morioka system,” Nonlinearity 31, 5410–5440 (2018).
  * Ovsyannikov and Turaev (2017) I. Ovsyannikov and D. Turaev, “Analytic proof of the existence of the Lorenz attractor in the extended lorenz model,” Nonlinearity 30, 115–137 (2017).
  * Gonchenko, Shilnikov, and Turaev (1996) S. V. Gonchenko, L. P. Shilnikov, and D. V. Turaev, “Dynamical phenomena in systems with structurally unstable Poincare homoclinic orbits,” Interdisc. J. Chaos 6, 15–31 (1996).
  * Gonchenko, Shilnikov, and Turaev (2008b) S. V. Gonchenko, L. P. Shilnikov, and D. V. Turaev, “On dynamical properties of multidimensional diffeomorphisms from newhouse regions. I,” Nonlinearity 21, 923–972 (2008b).
  * Lomelli and Meiss (1999) H. E. Lomelli and J. D. Meiss, “Heteroclinic orbits and transport in a perturbed, integrable standard map,” Phys. Lett. A 269, 309–318 (1999).
  * Arnold (1971) V. I. Arnold, “On matrices depending on parameters,” Russian Math. Surveys 26, 29–43 (1971).


# Transformer Networks for Data Augmentation of Human Physical Activity
Recognition

Sandeep Ramachandra  Ubiquitous Computing, University of SiegenHölderlinstr.
357076<EMAIL_ADDRESS>,  Alexander Hoelzemann
Ubiquitous Computing, University of SiegenHölderlinstr. 357076
<EMAIL_ADDRESS>and  Kristof Van Laerhoven
0000-0001-5296-5347 Ubiquitous Computing, University of SiegenHölderlinstr.
357076<EMAIL_ADDRESS>
###### Abstract.

Data augmentation is a widely used technique in classification to increase
data used in training. It improves generalization and reduces amount of
annotated human activity data needed for training which reduces labour and
time needed with the dataset. Sensor time-series data, unlike images, cannot
be augmented by computationally simple transformation algorithms. State of the
art models like Recurrent Generative Adversarial Networks (RGAN) are used to
generate realistic synthetic data. In this paper, transformer based generative
adversarial networks which have global attention on data, are compared on
PAMAP2 and Real World Human Activity Recognition data sets with RGAN. The
newer approach provides improvements in time and savings in computational
resources needed for data augmentation than previous approach.

Data Augmentation, Human Activity Recognition, Self Attention

††copyright: none††ccs: Computing methodologies Neural networks††ccs:
Computing methodologies Machine learning algorithms††ccs: Human-centered
computing Ubiquitous computing

##  1\. Introduction

Deep learning networks have progressed substantially, both due to improvements
in algorithms as well as availability of powerful computing hardware. These
networks, however, require large amounts of data to train well, and such large
benchmark datasets are not available for some domain, and human activity
recognition from wearable sensors is such a field.

Data augmentation in the context of deep learning, is a technique used to
increase the number of input data by adding modified data or by creating
artificial data which are similar to real data (Steven Eyobu and Han, 2018).
Data augmentation acts as a regularizer, leading to generalized neural
networks which give better results on unseen data. Image data can for instance
be augmented by techniques such as geometric transformations or color space
transformations which does not take much computational resources. Timeseries
data like the raw output of inertial sensors is harder to augment, since any
changes made to timeseries data due to augmentation cannot be visually
identified. This is more complicated in human activity datasets since many
activities like standing and sitting are very similar and any change may alter
what activity the classifier identifies the data with. Annotation of collected
data is a labour-intensive task and takes a long time for a sizable dataset
(Abedin et al., 2020). The development of Generative Adversarial Networks
(GAN) lead to neural networks outputting real-like data from random inputs.
This is used to augment limited data. However, GANs do not always converge due
to mode collapse arising from alternating gradient descent.

This work introduces a GAN architecture that is based on Transformer Networks.
We evaluate our architecture with a recently published one (Hoelzemann et al.,
2021) that is based on Long Short-Term Memory (LSTM). LSTMs has the
disadvantage that they process incoming data sequentially. However,
Transformers process data in a parallel stream which helps to speed up
training drastically. These Transformer’s parameters grow quadratically with
input size and can outstrip available GPU memory. However, for human activity
that have small input sizes, this is not a big concern.

We validated our network with leave-one-subject-out (LOSO) cross-Validation
and on two publicly available datasets, PAMAP2 (Reiss and Stricker, 2012) and
Real World Human Activity Recognition (RWHAR) (Sztyler and Stuckenschmidt,
2016). For the PAMAP2 dataset, we used 7 common activities with 27 sensor
channels (3 dimensions each for accelerometer, gyroscope and magnetometer in 3
Inertial Measurement Units). For the RWHAR dataset, the dataset consists of 8
activities with 6 sensor channels (3 dimensions for accelerometer and
gyroscope of the smartphone attached to the chest).

##  2\. Related Work

In sequence to sequence transformations like translating between two
languages, transformers outperforms LSTM networks in both resources and in
performance metrics (Gehring et al., 2017), (Vaswani et al., 2017). In Human
activity recognition, Mahmud et al. (Mahmud et al., 2020) and Murahari et al.
(Murahari and Plötz, 2018) uses transformers to capture the spatial temporal
context from the feature space of sensor reading sequence and classify the
sensor data.The transformer networks performs very well in this field as well.
This suggests that the use of transformers in data augmentation can be
beneficial.

Data augmentation is a topic under research in human activity recognition.
Alawneh et al. (Alawneh et al., 2021) makes a case that using data
augmentation in human activity recognition improves accuracy of trained
classifiers. Li et al. (Li et al., 2020) uses a convolutional GAN since
Recurrent GANs do not converge consistently, to generate synthetic human
activity physical data which were distinguishable by visualization techniques.
Hoelzemann et al. (Hoelzemann et al., 2021) used a state of the art Recurrent
GAN (RGAN), a GAN with an LSTM layer, for data augmentation with the resulting
artificial data verified to be similar to original data.

Transformers in data augmentation has been explored in Zhang et al. (Zhang et
al., 2019) which utilises self attention from transformers to explore long
range dependencies in internal representation of images. Another network
focusing on synthesizing images is the GANBERT or GAN with a bidirectional
encoder representation from transformer (Shin et al., 2020) which uses self
attention to generate difficult medical images like that from MRI and PET
scans.

##  3\. Methodology

Figure 1. Flowchart of the data augmentation process for human activity
recognition.

  1. (1)

Ready the dataset for loading : This means that the dataset has been cleaned
and loaded up in a format that is ready for the training and validation steps.

  2. (2)

Train the validation model : The main objective is to train a model of very
high quality to classify signals to classes. The validation model chosen here
is one DeepConvLSTM model (Ordóñez and Roggen, 2016) and one transformer model
(Wu et al., 2020) for each of the datasets.

  3. (3)

Train the GAN models : In this work, an LSTM based Recurrent GAN and a
transformer based GAN is compared. New models are trained for each class, so
the dataset needs to be separated by the class.

  4. (4)

Validate artificial signal using validation model : The synthesized data from
the generator needs to be validated using F1 score from the trained
classifier.

###  3.1. GAN Models

(a) RGAN Generator

(b) RGAN Discriminator

Figure 2. Architecture of state of the art Recurrent Generative Adversarial
Networks(RGAN). The trapezoid of convolution depict the increase or decrease
in channel size based on the divergent or convergent shape of trapezoid
respectively.

The LSTM based Recurrent GAN (RGAN) (see figure 2) in both generators and in
discriminators, has convolutions going into and coming out of the stacked LSTM
layer (Hoelzemann et al., 2021). The input to the generator is a 1D vector of
random numbers sampled from a standard normal distribution. The input to the
discriminator is a vector of shape cxL with c channels and L length (the
PAMAP2 dataset has a shape of 27x100 and RWHAR has 6x50). The channels of the
input are is progressively increased using 1D convolutions (with padding) to
increase the number of trainable parameters in the LSTM as well as increase
the receptive field on original data. The stacked LSTM observes the trends in
the data sequentially and its hidden state is used to classify the input data
to N classes via a fully connected layer. There is a dropout before the fully
connected layer providing regularisation during training. The fully connected
layer is realised by a 1D convolutional layer. The hyperparameters in this
model are the noise length, generator LSTM layers, and discriminator’s LSTM’s
hidden size and layers.

(a) TGAN Generator

(b) TGAN Discriminator

Figure 3. Architecture of proposed Transformer Generative Adversarial
Networks(TGAN). The attention is realised by a stacked scaled dot product
attention (Vaswani et al., 2017). The feedforward network takes the form of 2
linear layers going from input size to a fixed size and then returning to
original input size.

The Transformer GAN (TGAN) (see figure 3) follows a similar approach to the
encoder of the traditional transformer encoder layer (Vaswani et al., 2017).
The inputs to both discriminator and generator are same as the ones for
Recurrent GAN. The input is not processed sequentially as in Recurrent layers
but in parallel. To give the network positional awareness, a regular signal,
typically sine/cosine signal, is added to the input. This is then passed to
the multi head self attention blocks which is realised by a scaled dot product
attention.

(1) $Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V$

The inputs Query(Q), Key(K), and Value(V) are obtained by a 1 to 1 1D
convolution of input for each of the three. $d_{k}$ is a scaling factor for
the attention. The multi head attention block are followed by skip connections
and then a layer normalization. Then a feedforward network in the shape of two
fully connected layers going from input shape to a fixed dimension and then
back to input shape. This is followed by another skip connection and layer
normalization. This forms the encoder part of the network which is repeated
sequentially so the network can learn the features of the input better.
Finally, a fully connected layer realised by 1D convolutions with dropout is
added so that the intermediary encoding done by the encoder is converted to
the synthetic data in case of generator and to real/synthetic classification
in case of discriminator. The hyperparameters available are the number of
heads of the self attention layer of generator and discriminator, the number
of encoder layers stacked in the generator and discriminator, the dimension of
feedforward network, and the length of the noise vector.

##  4\. Results and Evaluation

(a) PAMAP2 DeepConvLSTM

(b) RWHAR DeepConvLSTM

(c) PAMAP2 Transformer

(d) RWHAR Transformer

Figure 4. The confusion matrices in percentages for all trained models and
both datasets. The overall validation F1 score of the models are (a) 96.21%
(b) 91.8% (c) 97% (d) 92.6%. Table 1 shows the sizes of the trained validation
models. The transformer classifiers are smaller, faster and have better
validation F1 scores than DeepConvLSTM models.

The models are trained with the given hyperparameters chosen by manual search
through the configuration space (Ramachandra, 2021). Figure 4 shows the
confusion matrices for the trained models. The Transformer outperforms the
DeepConvLSTM in classification problems, both in time and in validation F1
scores (see figure 4. In each dataset, the transformer, despite being a
smaller model consistently does better than the DeepConvLSTM model. In the
classes where it is worse, it still performs very good. The margin of false
classifications in those cases is only marginally lower.

The Transformer seems like a clear choice from figure 4 for use in the GAN as
a validation model. However in actual use, this is not the case. RGAN appears
to prefer DeepConvLSTM models and the Transformer GAN prefers a Transformer
classifier models. They give higher validation F1 scores when paired with its
preferred model and may not even train to completion with the other model.
This may be due to the GAN learning the distribution which can pass the
validation. The GANs seem to be able to learn the distribution needed when
paired with a similar recurrent based layers i.e the LSTMs of the GAN can
learn the distribution needed for DeepConvLSTM but not of the self attention
layer and likewise for Transformer GAN. This is a curious observation of the
behaviour of GAN, although it is not conclusively proven in this work. This is
why there are 4 models trained for 4 data augmentation experiments.

Dataset | Model |  | Parameter
---
Size
| Augmented
---
Data
| Speed up
---
(Time per
epoch)
PAMAP2 | RGAN | G : 17.6M | Yes |  1x (13 secs)
D : 697K
V : 3.00M
TGAN | G : 765K | Yes |  2.6x (5 secs)
D : 7.80M
V : 984K
RWHAR | RGAN | G : 2.10M | No* |  1x (24 secs)
D : 1.00M
V : 11.5M
TGAN | G : 4.40M | Yes |  3x (8 secs)
D : 2.20M
V : 10.3M
Table 1. Table of the performance of trained GAN models.RGAN stands for
Recurrent GAN and TGAN stands for Transformer GAN. G is Generator, D is
Discriminator, and V is Validation model. For speedup, the RGAN times are used
as baseline for calculations for each dataset. The times and speeds shown are
the average for the first activity only. The parameter sizes are truncated to
first decimal place and are in thousands(K) and millions(M). * - The generated
data for all activities in dataset did not satisfy the ¿95% validation model
F1 score. The validation model training speed for transformer model was 4 to
4.5 times the speed of DeepConvLSTM model.

(a) PAMAP2 Recurrent GAN

(b) RWHAR Recurrent GAN

(c) PAMAP2 Transformer GAN

(d) RWHAR Transformer GAN

Figure 5. Relation between real data and each model’s synthetic data is shown
using Pearson correlation coefficient. Mean of 10 samples for each are taken.
Each sensor channel of the mean real data is compared to the corresponding
sensor channel of the generated data. PAMAP2 has 27 channels - 3 IMUs with 3D
Accelerometer, 3D gyroscope and 3D magnetometer. RWHAR has 6 channels - 1 IMU
with 3D accelrometer and 3D gyroscope.

Table 1 shows the performances of the 4 GANs which have been trained. The
table lists the parameter size for the first try of the GAN training for
generator and discriminator. The GAN models are so large as they were not
optimized with a hyperparameter search as they have very large training time.
The given hyperparameters are the first ones which made the GAN network to
converge. The times per epoch is the average time taken for the model to
complete the training step and validation step. To calculate the speed up on
using the model, the time of the baseline model is divided by time taken by
the model. The advantage of using the transformer is immediately apparent. The
time taken for each epoch for the Transformer GAN in spite of being bigger
than Recurrent GAN, is much smaller than Recurrent GAN. All but one of the
models trained the augmented data for all activities successfully. For the
Recurrent GAN trained on RWHAR dataset, the GAN produced augmented data which
produced 91% F1 score on the validation model which is still a respectable
output.

The synthetic data generated from each GAN looks different from the real data
but since the shown samples are just one of many real samples, it is difficult
to make any conclusions regarding the data. So the mean of ’X’ samples of real
data and the mean of ’X’ samples of syntheticdata are used in the calculation
of Pearson correlation coefficient to make a simple comparison. The
coefficient is a measure of the relationship between the two variables. The
formula is

(2) $r=\frac{N\sum xy-(\sum x)(\sum y)}{\sqrt{[N\sum x^{2}-(\sum x)^{2}][N\sum
y^{2}-(\sum y)^{2}]}}$

where r is the Pearson correlation coefficient of the two variables x and y,
and N is the number of pairs of scores in x and y. The coefficient is scaled
from -1 to 1 where 1 denotes that the two variables are proportional to each
other and -1 denotes inverse proportionality. 0 means the two variables are
not related to one another. Ideally, very closely related data like a real
sample and perfect syntheticdata should have a coefficient of 1.

Each axes of the data is compared to one another using the formula and the
resulting coefficients are plotted for each activity. The figure 5 shows the
graphs for the 4 dataset-model combinations. Any correlation of 0.8 and above
correlates very well for use in data augmentation. This means that for PAMAP2,
the Transformer GAN works splendidly with all activities of good quality, far
better than Recurrent GAN in all activities. However there appears to be some
issues with the RWHAR dataset for both models. Recurrent GAN fails in 3
activities and Transformer GAN in 4 activities. Activity 2(climbing down) is
appears to be problematic since both have below 0.2 and transformer appears to
be negatively correlated. This seems to be linked to one of the subjects of
the dataset who wore their sensor upside down which has had a detrimental
effect on both models but the transformer GAN is more severely afflicted. This
suggests that the Transformer GAN is more sensitive to outlying data.

##  5\. Conclusions and outlook

This paper proposed a Transformer-based GAN for data augmentation of human
activity data, in order to provide a speedup benefit over the existing
Recurrent GAN and a performance benefit over a Convolutional GAN. This GAN is
used along with a validation model which is used to verify that the output of
the GAN is identifiable. The Transformer GAN is compared with Recurrent GAN
using two public datasets, PAMAP2 and Real World Human Activity Recognition.
In validation models, Transformer model outperformed DeepConvLSTM model in
both F1 scores and in model parameter sizes. The Transformer GAN provides a 2
- 3 times speedup over Recurrent GAN in both datasets. The performance of the
Transformer GAN is as good as if not better than the Recurrent GAN though it
appears to be more sensitive to errant input data than Recurrent GAN. For that
reason we are able to state that we (1) implemented the model and it generates
data as needed for training (2) beat current benchmarks for data augmentation
of human activity data with respect to the GAN training time (3) pointed out
the importance to ensure that the produced data is variable but still belong
to the original distribution of the input data and therefore recognizable by
the classification network.

###### Acknowledgements.

We thank our anonymous reviewers for their many insightful suggestions and
feedback.

## References

  * (1)
  * Abedin et al. (2020) Alireza Abedin, Farbod Motlagh, Qinfeng Shi, Hamid Rezatofighi, and Damith Ranasinghe. 2020.  Towards Deep Clustering of Human Activities from Wearables. In _Proceedings of the 2020 International Symposium on Wearable Computers_ (Virtual Event, Mexico) _(ISWC ’20)_. Association for Computing Machinery, New York, NY, USA, 1–6.  https://doi.org/10.1145/3410531.3414312
  * Alawneh et al. (2021) Luay Alawneh, Tamam Alsarhan, Mohammad Al-Zinati, Mahmoud Al-Ayyoub, Yaser Jararweh, and Hongtao Lu. 2021.  Enhancing human activity recognition using deep learning and time series augmented data.  , 16 pages.
  * Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017.  Convolutional Sequence to Sequence Learning.  http://arxiv.org/abs/1705.03122 arXiv: 1705.03122.
  * Hoelzemann et al. (2021) Alexander Hoelzemann, Nimish Sorathiya, and Kristof Van Laerhoven. 2021.  Data Augmentation Strategies for Human Activity Data Using Generative Adversarial Neural Networks.  , 8-13 pages.  https://doi.org/10.1109/PerComWorkshops51409.2021.9431046
  * Li et al. (2020) Xi’ang Li, Jinqi Luo, and Rabih Younes. 2020.  ActivityGAN: Generative Adversarial Networks for Data Augmentation in Sensor-Based Human Activity Recognition. In _Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers_ (Virtual Event, Mexico) _(UbiComp-ISWC ’20)_. Association for Computing Machinery, New York, NY, USA, 249–254.  https://doi.org/10.1145/3410530.3414367
  * Mahmud et al. (2020) Saif Mahmud, M Tanjid Hasan Tonmoy, and Kishor Kumar Bhaumik. 2020.  Human Activity Recognition from Wearable Sensor Data Using Self-Attention.  , 8 pages.
  * Murahari and Plötz (2018) Vishvak S. Murahari and Thomas Plötz. 2018.  On Attention Models for Human Activity Recognition. In _Proceedings of the 2018 ACM International Symposium on Wearable Computers_ (Singapore, Singapore) _(ISWC ’18)_. Association for Computing Machinery, New York, NY, USA, 100–103.  https://doi.org/10.1145/3267242.3267287
  * Ordóñez and Roggen (2016) Francisco Javier Ordóñez and Daniel Roggen. 2016.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition.  https://doi.org/10.3390/s16010115
  * Ramachandra (2021) Sandeep Ramachandra. 2021\.  Implementation and Evaluation of Transsformer Networks to augment sensor based Human Activity Data.  https://github.com/sandeep-189/Data-Augmentation.git.
  * Reiss and Stricker (2012) Attila Reiss and Didier Stricker. 2012.  Introducing a new benchmarked dataset for activity monitoring.  , 108–109 pages.
  * Shin et al. (2020) Hoo-Chang Shin, Alvin Ihsani, Swetha Mandava, Sharath Turuvekere Sreenivas, Christopher Forster, Jiook Cha, and Alzheimer’s Disease Neuroimaging Initiative. 2020.  GANBERT: Generative Adversarial Networks with Bidirectional Encoder Representations from Transformers for MRI to PET synthesis.  http://arxiv.org/abs/2008.04393 arXiv: 2008.04393.
  * Steven Eyobu and Han (2018) Odongo Steven Eyobu and Dong Seog Han. 2018.  Feature Representation and Data Augmentation for Human Activity Classification Based on Wearable IMU Sensor Data Using a Deep LSTM Neural Network.  https://doi.org/10.3390/s18092892
  * Sztyler and Stuckenschmidt (2016) Timo Sztyler and Heiner Stuckenschmidt. 2016.  On-body localization of wearable devices: An investigation of position-aware activity recognition.  , 9 pages.  https://doi.org/10.1109/PERCOM.2016.7456521
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention Is All You Need.  arXiv:1706.03762 [cs.CL]
  * Wu et al. (2020) Neo Wu, Bradley Green, Xue Ben, and Shawn O’Banion. 2020\.  Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case.  http://arxiv.org/abs/2001.08317 arXiv: 2001.08317.
  * Zhang et al. (2019) Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2019.  Self-Attention Generative Adversarial Networks.  http://arxiv.org/abs/1805.08318 arXiv: 1805.08318.