# Irregular Fixation: I. Fixed points and librating orbits of the Brown
Hamiltonian

Evgeni Grishin1,2
1School of Physics and Astronomy, Monash University, Clayton, VIC 3800,
Australia
2OzGrav: Australian Research Council Centre of Excellence for Gravitational
Wave Discovery, Clayton, VIC 3800, Australia Contact e-mail:
<EMAIL_ADDRESS>
###### Abstract

In hierarchical triple systems, the inner binary is slowly perturbed by a
distant companion, giving rise to large-scale oscillations in eccentricity and
inclination, known as von-Zeipel-Lidov-Kozai (ZLK) oscillations. Stable
systems with a mild hierarchy, where the period ratio is not too small,
require an additional corrective term, known as the Brown Hamiltonian, to
adequately account for their long-term evolution. Although the Brown
Hamiltonian has been used to accurately describe the highly eccentric systems
on circulating orbits where the periapse completes a complete revolution, the
analysis near its elliptical fixed points had been overlooked. We derive
analytically the modified fixed points including the Brown Hamiltonian and
analyse its librating orbits (where the periapse motion is limited in range).
We compare our result to the direct three-body integrations of millions of
orbits and discuss the regimes of validity. We numerically discover the
regions of orbital instability, allowed and forbidden librating zones with a
complex, fractal, structure. The retrograde orbits, where the mutual
inclination is $\iota>90\ \rm deg$, are more stable and allowed to librate for
larger areas of the parameter space. We find numerical fits for the librating-
circulating boundary. Finally, we discuss the astrophysical implications for
systems of satellites, stars and compact objects. In a companion paper
(Grishin, 2024), we apply our formalism to the orbits of irregular satellites
around giant planets.

###### keywords:

planets and satellites: dynamical evolution and stability – celestial
mechanics – chaos – stars: kinematics and dynamics

††pubyear: 2024††pagerange: Irregular Fixation: I. Fixed points and librating
orbits of the Brown Hamiltonian–References

##  1 Introduction

Triple and multiple systems are ubiquitous and appear in many astrophysical
settings, from Solar-System asteroids (Marchis et al., 2005), to multiple
planetary and stellar systems, (Martin & Triaud, 2014; Wagner et al., 2016),
to multiple stellar and compact objects (where most massive stars are in
triples, Offner et al., 2023), and supermassive black holes on galactic scales
(Deane et al., 2014). The dynamical evolution of three bodies under the
gravitational force – the three body problem – is a notoriously chaotic
problem with no analytical solution (Poincaré, 1892). For strongly chaotic
systems a statistical approach is generally preferred to yield a probabilistic
outcome of the scattering in terms of the final states (Monaghan, 1976a, b).
Only recently, explicit dependence on the orbital elements had been found with
various methods (Stone & Leigh, 2019; Ginat & Perets, 2021b, a; Kol, 2021b, a;
Manwadkar et al., 2021).

For hierarchical systems, the inner binary of semi-major axis $a_{1}$ is
perturbed by a distant tertiary at semi-major axis $a_{2}\gg a_{1}$, which is
typically on a much longer orbit. This separation of timescales allows
hierarchical systems to be stable. In addition, it is natural to expand the
interaction Hamiltonian in power series in multipoles, where the semi-major
axis ratio $a_{1}/a_{2}\ll 1$ is the small parameter. Truncating at the
leading quadrupole ($\propto(a_{1}/a_{2})^{2}$) order and orbit-averaging over
both the inner and outer orbits (e.g. Harrington, 1968) leads to a simplified
problem where the typical timescale is a much longer, secular timescale. An
integrable, analytical, approximate solution in the test-particle limit (where
one of the inner masses is small compared to the outer mass, Kinoshita &
Nakai, 2007; Lubow, 2021)111Formally, the condition is that the angular
momentum of the inner binary is much smaller than the angular momentum of the
outer binary, since the feedback on the outer binary is proportional to the
angular momentum ratio. Soderhjelm (1975) calls this approximation ’immovable
wide orbit limit’. can be obtained. A key feature is that for systems where
the mutual inclination $\iota$ between the inner and outer orbital planes is
large enough, the inner orbital eccentricity grows significantly, which is
known as the von-Zeipel-Lidov-Kozai effect (ZLK) (von Zeipel, 1910; Lidov,
1962; Kozai, 1962). Adding the higher order octupole
($\propto(a_{1}/a_{2})^{3}$) terms render the system to chaotic again, and it
is usually studied numerically by solving the secular equations of motion
(e.g. Naoz et al., 2013).

The latter double-averaging approximation was first tested when the apsidal
and nodal precession of the Moon was inconsistent with the secular theory,
which was off by a factor of $\sim 2$. Taking into account the relative motion
of the Earth after one Lunar period and expanding for almost coplanar and
circular orbits, the corrected longitude of pericentre advance
$\dot{\varpi}_{\leftmoon}$ is (Brouwer & Clemence, 1961; Tremaine, 2023)

$\frac{d\varpi_{\leftmoon}}{d\tau}=\frac{3}{4}\epsilon+\frac{225}{32}\epsilon^{2}+\frac{4071}{128}\epsilon^{3}+\frac{265493}{2048}\epsilon^{4}+\mathcal{O}\left(\epsilon^{5}\right)$
(1)

where $\tau=t/2\pi P_{2}$ is a normalised timescale, $P_{1}$ and $P_{2}$ are
the inner and outer orbital periods, and $\epsilon=P_{1}/P_{2}\approx 1/12$ is
the period ratio. Thus the quadratic $\epsilon^{2}$ term is nearly equal to
the linear $\epsilon$ term.

While the Lunar theory had been developed for low inclination and
eccentricity, discovering highly inclined and eccentric exoplanets and
hundreds of irregular satellites required a more general development.
Retrograde orbits have been observed to be more stable (Innanen, 1979, 1980;
Hamilton & Burns, 1991), while the exact dependence on the mutual inclination
was studied relatively recently (Grishin et al., 2017; Tory et al., 2022). The
disturbing function for irregular satellites had been expanded in Ćuk & Burns
(2004), and generalised by Luo et al. (2016) for any mass ratio where they
find an ’effective potential’. Tremaine (2023) had recently shown that the
aforementioned (and other related) derivations are equivalent to the
pioneering work of Brown (Brown, 1936a, b, c), and coined the effective
potential as ’Brown Hamiltonian’. The equivalence relies on an intricate gauge
freedom in the averaging procedure. We note that it is also possible to evolve
the ’single averaged’ equations, where the outer period is the typical
timescale (e.g. Liu & Lai, 2018). Generalised analogues of Brown’s Hamiltonian
for higher-order perturbations theory (but less practical) are given by Beaugé
et al. (2006); Lei et al. (2018); Lei (2019).

The importance of the Brown Hamiltonian was highlighted in the context of
highly eccentric orbits. Grishin et al. (2018) had developed analytical
expressions for the modified maximal eccentricity and critical inclination,
which were later extended beyond the test particle limit (Mangipudi et al.,
2022). Close approaches and disruptions in low-hierarchy triples were studied
in Haim & Katz (2018) and Mushkin & Katz (2020), respectively, while extreme
eccentricities have been invoked in white dwarf collision rates (Kushnir et
al., 2013). The modified formulae were widely used in many applications, from
Kuiber-Belt collisions (Grishin et al., 2020; Rozner et al., 2020), to compact
object mergers and gravitational waves (Fragione et al., 2019). Other external
driving forces such as the cluster and the galactic tidal potential for wide
binaries and triples had been also recently studied (Hamilton & Rafikov, 2019;
Bub & Petrovich, 2020; Grishin & Perets, 2022).

For the double-averaging approximation ($\epsilon=0$), for initial
eccentricity $e_{0}$, fixed points in the eccentricity $e_{1}$ – argument of
pericentre $\omega_{1}$ phase space appear (for allowed mutual inclination
values, $\cos^{2}\iota\leq 3/5$) at $\omega_{1}=\pi/2$ and
$e_{1}^{2}=1-\sqrt{(5/3)(1-e_{0}^{2})}\cos\iota$ (Carruba et al., 2002;
Antognini, 2015; Hamers, 2021)222Both Carruba et al. (2002) and Antognini
(2015) use the notation $\Theta=(1-e^{2})\cos^{2}I$, while we’ll write it it
terms of $j_{z}=\sqrt{\Theta}$.. Fixed points in the planetary limit were
studied by Hansen & Naoz (2020). Klein & Katz (2023) studied the dynamics of a
ZLK cycle near a fixed point for a slowly precessing outer orbit and later
generalised to libration far from the fixed point in Klein & Katz (2024).

Besides the highly eccentric cases, the analytical properties of the Brown
Hamiltonian were overlooked. In this paper, we explore the fixed point and the
phase space of the Brown Hamiltonian. We explore the regions where libration
is allowed and uncover its complex, fractal structure and find numerical fit
for the boundary far enough from instability. In a companion paper (paper II)
we will focus on the motion of irregular satellites in the Solar system.

The paper is organised as follows: In sec. 2 we derive the equations of motion
of the Brown Hamiltonian in Delaunay elements. We then compute the fixed
points in $e_{1}$ \- $\omega_{1}$ space. In sec. 3 we compare our analytic
results to direct three body integrations. We uncover the parameter space for
librating, circulating and unstable orbit and its complex, fractal boundary.
We also provide a numerical fit for the boundary between stable circulating
and librating orbits. Finally in sec. 4 we conclude our results and discuss
the limitations and future work.

##  2 Modified fixed points

###  2.1 Hierarchical systems

Consider a hierarchical system with masses $m_{0},m_{1}\ll m_{2}$, orbital
separations $a_{1}\ll a_{2}$ and other orbital elements denoted by subscript
$1$ for the inner orbit and $2$ for the outer orbit. The double-averaged
quadrupole Hamiltonian is (Naoz, 2016)

$\displaystyle{\mathcal{H}}_{{\rm sec}}$
$\displaystyle\equiv\mathcal{\langle\langle
H}\rangle\rangle=C\left[1-6e_{1}^{2}-3(1-e_{1}^{2})\cos^{2}\iota+15e_{1}^{2}\sin^{2}\iota\sin^{2}\omega_{1}\right]$
$\displaystyle C$ $\displaystyle\equiv\frac{G\mu_{\rm
in}m_{2}}{8a_{2}\left(1-e_{2}^{2}\right)^{3/2}}\left(\frac{a_{1}}{a_{2}}\right)^{2},$
(2)

where $\mu_{\rm in}=m_{0}m_{1}/(m_{0}+m_{1})$ is the reduced mass of the inner
binary, $\omega_{1}$ is the argument of pericentre and $\iota$ is the mutual
inclination between the orbits.

We can change the variables into the Delauanay elements

$\displaystyle L_{1}$ $\displaystyle=\mu_{{\rm
in}}\sqrt{G\left(m_{0}+m_{1}\right)a_{1}}\qquad L_{2}=\mu_{{\rm
out}}\sqrt{G\left(m_{0}+m_{1}+m_{2}\right)a_{2}}$ $\displaystyle G_{1}$
$\displaystyle=L_{1}\sqrt{1-e_{1}^{2}}\qquad G_{2}=L_{2}\sqrt{1-e_{2}^{2}}$
$\displaystyle H_{1}$ $\displaystyle=G_{1}\cos i_{1}\qquad H_{2}=G_{2}\cos
i_{2}$ (3)

where $\mu_{{\rm out}}=\left(m_{0}+m_{1}\right)m_{2}/(m_{0}+m_{1}+m_{2})$ and
$i_{i}$ are the inclinations with respect to the invariable plane.

Generally, the mutual inclination is given by $\cos\iota=\cos i_{1}\cos
i_{2}+\sin i_{i}\sin i_{2}\cos(\Omega_{1}-\Omega_{2})$ where $\Omega_{i}$ are
the arguments of the ascending nodes. The Delaunay elements are strictly
defined only on the invariable plane (where the total angular momentum is
parallel to the $\hat{\bm{z}}$ direction Soderhjelm, 1975; Naoz et al., 2013),
where the nodes can be eliminated, $\Omega_{2}-\Omega_{1}=\pi$, and the mutual
inclination reduces to $\cos\iota=\cos(i_{1}+i_{2})$.

####  2.1.1 Mild hierarchy

The double averaging approach involves averaging over the inner, and then over
the outer orbit. This approach ignores the relative movement of the outer body
after one orbital period, and induces errors of the order of $P_{1}/P_{2}$.
Ćuk & Burns (2004) studied the corrections expanding the disturbing function
in terms of $P_{1}/P_{2}$. (Luo et al., 2016) generalised the analysis to
general three body systems. The additional term in Luo et al. (2016) (their
Eq. 39) is

$\displaystyle{\mathcal{H}}_{\rm B}$ $\displaystyle=-\epsilon_{{\rm
SA}}C\frac{27}{8}j_{z}\left(\frac{(1-j_{z}^{2})}{3}+8e_{1}^{2}-5e_{1}^{2}\sin^{2}\omega_{1}\sin^{2}\iota\right)+\mathcal{O}(e_{2}^{2}).$
(4)

Here,

$\epsilon_{\rm
SA}=\left(\frac{a_{1}}{\ell_{2}}\right)^{3/2}\left(\frac{m_{2}^{2}}{(m_{0}+m_{1})(m_{0}+m_{1}+m_{2})}\right)^{1/2}=\frac{2P_{2}}{3\pi\tau_{\rm
sec}}$ (5)

is the parameter that measures the relative accuracy of the double-averaging
approximation, and $\ell_{2}=a_{2}(1-e_{2}^{2})$ is the semi-latus rectum. The
deviation between single-averaged and double-averaged approximations is
proportional to $\epsilon_{\rm SA}$. In the limit of $e_{2}\ll 1$ and
$m_{2}\gg m_{1},m_{0}$, we have $\epsilon_{\rm
SA}=(m_{2}a_{1}^{3}/m_{2}a_{2}^{3})^{1/2}=P_{1}/P_{2}$, which is just
$\epsilon$ from Eq. 1 for the Lunar case.

In the right-hand side of Eq. 5, the secular timescale is given by

$\tau_{\rm
sec}=\frac{L_{1}}{6C}=\frac{4(m_{0}+m_{1})^{1/2}a_{1}^{3/2}}{G^{1/2}m_{2}\ell_{2}^{3}}$
(6)

Note that it differs by $4/3$ from the definition of Luo et al. (2016). This
is an order-of-magnitude estimate. Formally, one needs to integrate over the
closed loop

$\tau_{\rm sec}=\oint{\rm d}t=\oint\left(\frac{{\rm d}X}{{\rm
d}t}\right)^{-1}{\rm d}X$ (7)

Where $X\propto G_{1}^{\alpha}\propto(1-e_{1}^{2})^{\alpha/2}$ is related to
the action $G_{1}$. Antognini (2015) used $\alpha=1$ in the test particle
limit, while Hamers (2021) and Hamilton & Rafikov (2019) used $\alpha=2$ for
comparable masses for for a more general tidal galactic field, respectively.

The Hamiltonian in Eq. 4 has appeared in several different forms and is known
as the Brown Hamiltonian (Brown, 1936a, b, c). Recently, (Tremaine, 2023)
showed that the various forms in the literature are equivalent due to a gauge-
freedom. Moreover, choosing the right gauge leads to a much simpler expression
for the eccentric term. We keep the outer orbit circular to simplify the
calculation.

Figure 1: Values of the total Hamiltonian $\mathcal{H}_{\rm
sec}+\mathcal{H}_{B}$ at the fixed point normalised at the separatrix. For
each $\epsilon_{\rm SA},j_{z}$ we plot $\mathcal{H}_{\rm
fixed}=\mathcal{H}(e_{\rm fix},\omega=\pi/2|\epsilon_{\rm SA},j_{z})$ minus
the value at the separatrix (Eq. 19). Left: Prograde orbits, $j_{z}>0$. Right:
Retrograde orbits. The grey triangles are the initial conditions in Fig. 2,
and 3 where the red points are the initial conditions for Fig. 5, and 6.

The total Hamiltonian and the equations of motion can be written in terms of
the Delaunay elements:

$\displaystyle\mathcal{H}$ $\displaystyle=\mathcal{H}_{\rm
sec}(G_{1},\omega_{1},H_{1},\Omega_{1})+\mathcal{H}_{\rm
B}(G_{1},\omega_{1},H_{1},\Omega_{1})$
$\displaystyle=C\left[6\frac{G_{1}^{2}}{L_{1}^{2}}-3\frac{H_{1}^{2}}{L_{1}^{2}}+15\left(1-\frac{{G_{1}^{2}}}{L_{1}^{2}}-\frac{H_{1}^{2}}{G_{1}^{2}}+\frac{{H_{1}^{2}}}{L_{1}^{2}}\right)\sin^{2}\omega_{1}\right]$
$\displaystyle-\epsilon_{{\rm
SA}}C\frac{27}{8}j_{z}\left[\frac{(1-j_{z}^{2})}{3}+8+(-8+5\sin^{2}\omega_{1})\frac{G_{1}^{2}}{L_{1}^{2}}\right.$
$\displaystyle+\left.5\sin^{2}\omega_{1}\left(\frac{H_{1}^{2}}{G_{1}^{2}}-1-j_{z}^{2}\right)\right]+\mathcal{O}(e_{2}^{2}),$
(8)

The first term gives as the standard ZLK equations of motion (e.g. Liu et al.,
2015). The second term gives us the additional contribution from Brown’s
Hamiltonian. For the argument of pericentre, we have

$\displaystyle\left.\frac{d\omega_{1}}{dt}\right|_{\rm B}$
$\displaystyle=\frac{d\mathcal{H}_{\rm B}}{dG_{1}}=-\epsilon_{{\rm
SA}}C\frac{27}{8}j_{z}\left(-16\frac{G_{1}}{L_{1}^{2}}+10\sin^{2}\omega_{1}\left(\frac{G_{1}}{L_{1}^{2}}-\frac{H_{1}^{2}}{G_{1}^{3}}\right)\right)$
$\displaystyle=-\epsilon_{{\rm
SA}}\frac{6C}{L_{1}\sqrt{1-e_{1}^{2}}}\frac{9}{8}j_{z}\left(-8(1-e_{1}^{2})+5\sin^{2}\omega_{1}(1-e_{1}^{2})\right.$
$\displaystyle-\left.5\cos^{2}\iota\sin^{2}\omega_{1}\right).$ (9)

For a sanity check we will take $\omega_{1}=\pi/2$ and $e_{1}=0$ to get

$\tau_{\rm sec}\frac{d\omega_{1}}{dt}=9\epsilon_{{\rm
SA}}\cos\iota\left(1-\frac{5}{8}\sin^{2}\iota\right).$ (10)

When the total (quadrupole and Brown) Hamiltonians are added, we get back Eq.
38 of Ćuk & Burns (2004) (a prefactor of $3/4$ is missing in the first two
terms, which appears in their Eq. 4). For almost coplanar orbit,
$\sin\iota\approx 0$ and the typical apsidal advance rate is
$\dot{\omega}_{1}=2+9\epsilon_{\rm SA}=2(1+9\epsilon_{\rm SA}/2)$. Similarly,
the nodal precession rate can also be derived from the Hamiltonian and is
$\dot{\Omega}_{1}=-9\epsilon_{\rm SA}/32$, so the precession of the longitude
of pericentre is $\dot{\varpi}_{1}=2+(9+9/24)\epsilon_{\rm
SA}=2+225\epsilon_{\rm SA}/24$, which is compatible with earlier works on the
Lunar theory (the secular timescale is defined up to a multiplicative factor
of $3/4$).

For completeness, the equation of motion for the eccentricity is

$\displaystyle\left.\tau_{\rm sec}\frac{de_{1}}{dt}\right|_{\rm B}$
$\displaystyle=\tau_{\rm
sec}\frac{\sqrt{1-e_{1}^{2}}}{L_{1}e_{1}}\frac{dG_{1}}{dt}=-\tau_{\rm
sec}\frac{\sqrt{1-e_{1}^{2}}}{\tau_{\rm
sec}L_{1}e_{1}}\frac{\partial\mathcal{H}_{{\rm B}}}{\partial\omega_{1}}$
$\displaystyle=\frac{45}{64}\epsilon_{{\rm
SA}}(1-e_{1}^{2})e_{1}\cos\iota\sin^{2}\iota\cos 2\omega_{1}.$ (11)

Usually, the equations of motion are presented in terms of the vector elements
(e.g. Luo et al., 2016). Will (2021) derived the equations of motion in terms
of the orbital elements directly from the Laplace-Largange secular equations
(their Eq. 2.33). However, the derivative of $\omega_{1}$ is not clearly
stated, but rather given by
$\dot{\omega}_{1}=\dot{\varpi}_{1}-\dot{\Omega}_{1}\cos i_{1}$, where
$\dot{\varpi}_{1}$ and $\dot{\Omega}_{1}$ are provided333Note that this is
different from the usual definition of the longitude of pericentre
$\varpi=\omega+\Omega$. Also eliminating $i_{1}$ is not trivial.. Soderhjelm
(1975) also derived Brown’s Hamiltonian in Delaunay variables, but presented
the equations of motion only for small $e_{1}\ll 1$ (which is applicable for
eclipsing binaries).

Our equations reduce to the ones of Soderhjelm (1975) for low $e_{1}$ and to
our knowledge, this is the first time Eq. 9 was derived explicitly for any
$e_{1}$.

###  2.2 Fixed points

Restoring back to orbital elements in Eq. 9 and adding the ZLK secular
contribution leads to

$\displaystyle\dot{\omega}_{1}$
$\displaystyle=2(1-e_{1}^{2})+5\sin^{2}\omega_{1}(e_{1}^{2}-\sin^{2}\iota)$
(12) $\displaystyle-9\epsilon_{\rm
SA}j_{z}\left(-(1-e_{1}^{2})+\frac{5}{8}\sin^{2}\omega_{1}(1-e_{1}^{2})-\frac{5}{8}\cos^{2}\iota\sin^{2}\omega_{1}\right).$

Setting $\omega_{1}=\pi/2$, and defining $x^{2}=1-e_{1}^{2}$,
$\cos\iota=j_{z}/x^{1/2}$ we get

$\displaystyle 0$
$\displaystyle=2x^{2}-5x^{2}+5\frac{j_{z}^{2}}{x^{2}}-9\epsilon_{\rm
SA}j_{z}\left(-x^{2}+\frac{5}{8}x^{2}-\frac{5}{8}\frac{j_{z}^{2}}{x^{2}}\right)$
$\displaystyle=3\left(-1+\frac{9}{8}\epsilon_{\rm
SA}j_{z}\right)x^{2}+5\left(1+\frac{9}{8}\epsilon_{\rm
SA}j_{z}\right)\frac{j_{z}^{2}}{x^{2}},$ (13)

and the solution is

$x^{2}=\sqrt{\frac{5(1+\frac{9}{8}\epsilon_{\rm
SA}j_{z})}{3(1-\frac{9}{8}\epsilon_{\rm SA}j_{z})}}|j_{z}|,$ (14)

or in terms of the eccentricity

$\displaystyle e_{\rm fix}=\sqrt{1-\sqrt{\frac{5(1+\frac{9}{8}\epsilon_{\rm
SA}j_{z})}{3(1-\frac{9}{8}\epsilon_{\rm SA}j_{z})}}|j_{z}|}.$ (15)

The associated inclination at the fixed point is

$\cos\iota_{\rm fix}=\frac{j_{z}}{x^{1/2}}={\rm
sign}(j_{z})\left(\frac{3}{5}\right)^{1/4}\sqrt{\frac{1-\frac{9}{8}\epsilon_{\rm
SA}j_{z}}{{1+\frac{9}{8}\epsilon_{\rm SA}j_{z}}}}|j_{z}|^{1/2}.$ (16)

Note that for the hierarchical limit ($\epsilon_{\rm SA}=0$), we get

$e_{\rm fix}=\sqrt{1-\left(\frac{5}{3}\right)^{1/2}|j_{z}|},$ (17)

and the associated inclination at the fixed point is

$\cos\iota_{\rm fix}=\frac{j_{z}}{x^{1/2}}={\rm
sign}(j_{z})\left(\frac{3}{5}\right)^{1/4}|j_{z}|^{1/2},$ (18)

which is the result of Antognini (2015). We note that analytic solutions have
been obtained in Kinoshita & Nakai (2007) and the results for the general
quadrupole problem (beyond the test particle limit) have been recently
extended in Hamers (2021).

###  2.3 Phase space and individual orbits

The dynamics are revealed in more detail upon inspecting the total Hamiltonian
$\mathcal{H}_{\rm sec}+\mathcal{H}_{B}$ from Eq. 2.1 and 4. The maximal value
of the Hamiltonian is attained at the fixed point (if exists) for each value
of $\epsilon_{\rm SA}$, $j_{z}$. The value itself is unimportant (Hamiltonians
are always defined up to a constant), but an important quantity is the
distance to the separatrix444Strictly speaking, a separatrix curve passes
through an unstable (hyperbolic) fixed point, similar to the pendulum. The
unstable fixed point is at the origin in rectangular coordinates
$(e_{1}\cos\omega_{1},e_{1}\sin\omega_{1})$. The timescale also diverges
logarithmically (e.g. Eq. 48 of Antognini, 2015). – a critical curve that
separates librating and circulating orbits.

Figure 2: Phase space of the total Hamiltonian $\mathcal{H}_{\rm
sec}+\mathcal{H}_{B}$ for the prograde cases. Top panels show only
$\mathcal{H}_{\rm sec}$, while the lower panel have both terms with
$\epsilon_{\rm SA}=0.2$. The colour and lines show constant energy curves. The
thick red line is the separatrix which separates circulating and librating
orbits. The Hamiltonian is normalised by $C=1$. Figure 3: Same as Fig. 2, but
for retrograde orbits.

Fig. 1 shows the distance of the maximal value $\mathcal{H}_{\rm
fixed}=\mathcal{H}(e_{\rm fix},\omega=\pi/2|\epsilon_{\rm SA},j_{z})$ from the
separatrix

$\mathcal{H}_{\rm sep}=1-3j_{z}^{2}-\frac{9\epsilon_{\rm
SA}}{8}j_{z}(1-j_{z}^{2}),$ (19)

as function of $\epsilon_{\rm SA}$, and $j_{z}$. We see that the trend in the
prograde orbits (left) is different from the trend in the retrograde orbits
(right), where the distance is decreasing (increasing) with larger
$\epsilon_{\rm SA}$ for prograde (retrograde) orbits. This is another
manifestation of the symmetry breaking that occurs when introducing the Brown
Hamiltonian (e.g. Innanen, 1979; Grishin et al., 2017, 2018). For decreasing
$|j_{z}|$, further away from the bifurcation point, the distance is
increasing, as expected. Very low $|j_{z}|$ are expected to have the largest
distance

Although it is tempting to try and determine whether an orbit will librate or
circulate, the situation is more complex, especially for larger $\epsilon_{\rm
SA}$. The modified averaging procedure that is used to obtain
$\mathcal{H}_{B}$ leaves out higher order terms and osculating elements. In
other words, the quantities $j_{z}$, $\epsilon_{\rm SA}$ are conserved only on
secular timescales, but fluctuate on shorter tiemscales. One can decompose,
e.g. $j_{z}$ to mean and fluctuating contributions, $\bar{j}_{z}+\delta
j_{z}$. The mean $\bar{j}_{z}$ is conserved because $\mathcal{H}_{\rm
sec}+\mathcal{H}_{B}$ is independent of the nodal angle $\Omega_{1}$. The
fluctuating elements vary with period $P_{2}$. Although an analytic envelope
for the strength of the amplitude of $\delta j_{z}$ can be estimated (e.g. Luo
et al., 2016), it is challenging to draw definite conclusions on the nature of
the orbits without exact N-body integrations. Nevertheless, we suggest a
procedure that finds an empirical criterion for librating orbits that relies
on the difference $\mathcal{H}_{\rm fixed}-\mathcal{H}_{\rm sep}$ and the
analytic envelope of $\delta j_{z}$ and apply it to irregular satellites in
paper II (Grishin, 2024).

Fig. 2 shows the constant energy curves in the phase space for prograde orbits
($j_{z}>0$) with and without the Brown Hamiltonian. The Hamiltonian is
normalised by $C=1$. The numerically obtained fixed point is consistent with
our Eq. 15 (blue dots). The thick red line is the separatrix.

Fig. 3 shows the constant energy curves for retrograde orbits. We see that
increasing $\epsilon_{\rm SA}$ could lead to ’birth’ of fixed points, contrary
to the prograde case where an existing fixed point may be eliminated.

In choosing the initial conditions in Fig 2 and 3 we show how the morphology
of the phase space changes once the correction from the Brown Hamiltonian are
introduced. We see, for example, that the general volume of the librating zone
is decreasing for prograde orbit (or the fixed point completely vanishes) for
increasing $\epsilon_{\rm SA}$. On the contrary, retrograde orbits ar
increasing their librating volume phase space with increasing $\epsilon_{\rm
SA}$ (or a fixed point and libration zone can be born). The minimal value of
the Hamiltonian can be obtained analytically but is of little importance.
Large negative value will result in almost co-planar motion on almost constant
eccentricity. For a given $j_{z}$ the maximum eccentricity allowed is
$\sqrt{1-j_{z}^{2}}$ on a circular orbit.

Figure 4: Left: Eccentricity of the fixed point as a function of
$\epsilon_{\rm SA}$ for various $j_{z}$ values. Right: The associated
inclination at the fixed point. The dashed lines correspond to retrograde
orbits where $j_{z}\to-j_{z}$. The retrograde inclination is understood as
$\iota_{\rm fix}^{\rm retro}=180^{\circ}-\iota_{\rm fix}$.

Fig. 4 shows the eccentricity (left) and associated inclination (right) of the
fixed point as a function of $\epsilon_{\rm SA}$, for various values of
$j_{z}$. The retrograde orbits are dashed where $j_{z}\to-j_{z}$ and
$\iota_{\rm fix}-180^{\circ}-\iota_{\rm fix}$. We see that the eccentricity
decreases for prograde orbits and increases for retrograde ones. This
asymmetry stems from the Coriolis asymmetry (Innanen, 1980) and comes from odd
powers of $j_{z}\propto\cos\iota$, which also affects the maximal eccentricity
and critical inclination for ZLK oscillations (Grishin et al., 2018). For low
$j_{z}$ (both positive and negative) the changes are mild, which is due to the
scaling of $1-e_{\rm fix}$ with $j_{z}$, so it doesn’t affect too much the
highly eccentric orbits (but even the correction from Brown Hamiltonian could
be broken down if $\sqrt{1-e_{\rm max}^{2}}\lesssim\epsilon_{\rm SA}^{2}$, see
Grishin et al., 2018 for details).

To summarise, the addition of the Brown Hamiltonian induce bifurcations which
change the morphology of the phase space for fixed $j_{z}$, and varying
$\epsilon_{\rm SA}$ leading to birth and death of ZLK resonance: For prograde
orbits, with increasing $\epsilon_{\rm SA}$, the fixed point could vanish
(e.g. for $j_{z}=0.7$, maroon line), while for retrograde orbits the fixed
point could be in areas which are inaccessible with quadrupole approximation
only (e.g. dashed black line).

##  3 Comparison to N-body integrations

We’ve seen that the deviation from the double-averaging approximation linearly
increases with $\epsilon_{\rm SA}$. Moreover, systems with larger
$\epsilon_{\rm SA}$ also tend to be more unstable: If an inner orbit is
parametrised in terms of the Hill radius, $a_{1}=kr_{\rm
H}=ka_{2}((m_{0}+m_{1})/3m_{2})^{1/3}$, then $\epsilon_{\rm
SA}=3^{-1/2}k^{3/2}$. The critical Hill stability limit is around $k\approx
0.5$ for prograde orbits (Grishin et al., 2017; Tory et al., 2022), so the
maximal $\epsilon_{\rm SA}\approx 0.2$.

Even if the system is dynamically stable, mildly hierarchical systems also
exhibit additional fluctuations, both in osculating elements on shorter
timescales and higher-order corrections to the Brown Hamiltonian. Therefore,
the averaged, constant energy $\mathcal{H}_{\rm sec}+\mathcal{H}_{B}$ is
subjected to short-term fluctuations due to the osculating elements (see e.g.
sec. 3.1 of Luo et al., 2016). The amplitude of the osculating elements is
also proportional to $\epsilon_{\rm SA}$. At some point, $\epsilon_{\rm SA}$
is large enough such that an orbit that starts near the fixed point could
cross the separatrix and become circulating.

Figure 5: Comparison to N-body simulations for the prograde case. The grey
points are snapshots of one orbit at different times, which starts at the
fixed point. Black points are results of secular evolution with
$\mathcal{H}_{\rm sec}$, the green points is the secular evolution with
$\mathcal{H}_{\rm B}$, which remains fixed. The top panels show librating
orbits, while the bottom panels show orbits that escape and circulating. The
colours and lines are the same as in Fig. 2 and 3.

Figure 6: Similar to Fig. 5 but with retrograde orbits. Figure 7: Phase map of
N-body integration. The plot include $360$ initial values of
$j_{z}\in[-0.9,0.9]$ and $291$ values of $\epsilon_{\rm SA}\in[0.01,0.3]$, a
total of $104,760$ integrations. The end time is $t_{\rm end}=n_{\rm c}P_{\rm
out}\epsilon_{\rm SA}^{-1}$, where $n_{\rm c}=10$ for $\epsilon_{\rm SA}<0.15$
and $n_{\rm c}=100$ if $\epsilon_{\rm SA}\geq 0.15$. This ensures that dozens
of secular timescales are integrated, but also saves integration time for the
more stable systems. Green dots are librating orbits, navy are circulating
orbits, and pink are unstable orbits. The dashed white lines are the numerical
fits of Eq. 20. Red points are individual integration shown in Fig. 5 and 6.
Red triangle is the orbit in Fig. 10. Boxed are areas of high-resolution
further insets: see Fig. 8 for the red insets and Fig. 9 for the black inset
with longer integration time of $n_{\rm c}=1000$.

Figure 8:  High resolution insets on the boundaries. Top row: firsts insets,
with $201\times 201=40,401$ ($201\times 351=70,551$) integrations for the
retrograde (prograde) inset in the top (bottom) left. Bottom row: additional
inset of the left panels with $331\times 331=109,591$ ($291\times
441=128,331$) initial conditions. Figure 9: Inset of Fig. 7 at higher
resolution of $101\times 101=10201$ orbits integrated for longer times of
$1000n_{\rm c}$.

###  3.1 Individual orbits

In order to explore the critical transition to circulating orbits, we compare
the results of sec. 2 with direct N-body integrations. All the integrations
were computed using the REBOUND N-body code, with the IAS15 adaptive timestep
integrator, accurate to machine precision for billions of orbits (Rein &
Spiegel, 2014).

Unless stated otherwise, for all orbits we vary $\epsilon_{\rm SA}$ and
$j_{z}$ to get the fixed points’ solution for $e_{\rm fix},\iota_{\rm
fix},\omega_{1}=\pi/2$ and $a_{1}$ which corresponds to $\epsilon_{\rm SA}$.
The other orbital elements are for the outer binary are $a_{2}=5.203\ \rm au$,
$m_{2}=m_{J}$ (all other angles are zero). We keep the outer orbit circular
for simplicity. For the inner orbit, the mean anomaly is $\pi$ to start the
orbit at apocentre, and $\Omega_{1}=\pi/4$, to ensure that the osculating
$j_{z}$ is equal to its averaged value (Grishin et al., 2018). The interaction
time is $t_{\rm end}=10P_{\rm out}\epsilon_{\rm SA}^{-1}$, which ensures that
the simulations are integrated to about $\mathcal{O}(10)$ secular timescales.

####  3.1.1 Prograde orbits

In Fig. 5 we show similar energy contours as in Fig. 2, but with the grey dots
as recorded from direct three body integrations for various initial
conditions. To clarify, each panel has only one orbit. The osculating orbital
elements tend to fill up a fraction of the phase space, but the averaged value
of the energy is retained. The recorded orbital elements are osculating
elements, which fluctuate about the size of $\epsilon_{\rm SA}$ around the
mean orbital elements (Luo et al., 2016).

We’ve also ran the same initial conditions with the secular code SecuLab
(Grishin & Perets, 2022), which includes a variant of the Brown Hamiltonian
and solves the equations of motion from Luo et al. (2016) but for arbitrary
reference frame. The black points are where $\mathcal{H}_{\rm B}$ is turned
off ($\epsilon_{\rm SA}=0$), which does not fit within the separatrix in the
left panels. The green points are the results of the secular evolution, which
remains fixed.

We see that for $j_{z}=0.71$ the critical $\epsilon_{\rm SA}$ for which the
orbit escapes is $\epsilon_{\rm SA}\approx 0.07$, while for $j_{z}=0.5$ it is
around $\epsilon_{\rm SA}\approx 0.16$. The associated inclinations for the
top panel from Eq. 16 are $45.02\ \rm deg$ and $53.97\ \rm deg$ for
$j_{z}=0.71$ and $j_{z}=0.5$, respectively, and for the bottom panel $45.47\
\rm deg$ and $55.345\ \rm deg$ for $j_{z}=0.71$ and $j_{z}=0.5$, respectively.

####  3.1.2 Retrograde orbits

Fig. 6 shows the numerical integrations for retrograde orbits, where
$j_{z}=-0.71,-0.77,-0.798$. We see that the retrograde orbits are more stable,
with larger values of $\epsilon_{\rm SA}$ allowed. This is consistent with the
stability analysis (Grishin et al., 2017; Tory et al., 2022), however, the
latter studies have generally assumed an initially circulating orbit with low
eccentricity. The associated inclinations for the top panel from Eq. 16 are
$147.49\ \rm deg,152.61\ \rm deg$ and $147.79\ \rm deg$ for
$j_{z}=-0.71,-0.77$ and $j_{z}=-0.8$, respectively, and for the bottom panel
$153.26\ \rm deg,153.4\ \rm deg$ and $148.62\ \rm deg$ for $j_{z}=-0.71,-0.77$
and $j_{z}=-0.8$, respectively. We also see that the secular approximation
completely missed out the libration in all the cases.

In Fig. 5 and 6 we show how a small increase in $\epsilon_{\rm SA}$ leads to a
qualitative change in the N-body integration. Our analytic theory is
insufficient to describe the validity of the existence of fixed point and it
requires detailed numerical investigation, which is done in the next section.
We note that individual time evolution of various quantities for several
irregular satellites, and the boundaries for the osculating energy, are shown
in paper II of this series (Grishin, 2024).

###  3.2 Parameter space exploration

In order to qualitatively explore the possibility of librating orbits at large
$\epsilon_{\rm SA}$ values, we integrate a large sample of orbits. We
uniformly initialise $361$ values of $j_{z}\in[-0.9,0.9]$ and $291$ values of
$\epsilon_{\rm SA}\in[0.01,3]$ and set an orbit at the fixed point in $e_{\rm
fix},\iota_{\rm fix}$ given by Eq. 15 and 16 and $\omega_{1}=\pi/2$. If the
fixed point doesn’t exist, the initial eccentricity will be zero (and the
orbit can only be circulating or unstable). The timescale for each orbit is
$t_{\rm end}=n_{\rm c}P_{\rm out}\epsilon_{\rm SA}^{-1}$, where $n_{\rm c}=10$
for $\epsilon_{\rm SA}<0.15$, and $n_{\rm c}=100$, for $\epsilon_{\rm SA}\geq
0.15.$. This ensures that at least several secular timescales are integrated
and longer timescales are tested. For lower $\epsilon_{\rm SA}$, values,
$n_{\rm c}$ is lower to ensure reasonable integration times. The entire grid
takes $\sim 2$ days to run on a 8-core machine. The planet is initialised in a
circular orbit at a separation of $5.203\ \rm au$ from the Sun to mimic
Jupiter. We deem the orbit unstable if $d>3a_{1}$ where $d$ is the
instantaneous separation. We classify the orbits as circulating if at any
point $\sin\omega_{1}<-0.2$ (since librating orbits will be constrained to
$\omega_{1}\in(0,\pi)$, or $\sin\omega_{1}>0$). Choosing $-0.2$ is a
compromise between making sure that librating orbits are not misclassified and
reducing the run time of a system that completes a full circulation.

Fig. 7 shows the result of the integration. We see that most of the orbits are
able to remain in the librating zone. The transition in the retrograde case is
between $-0.8\leq j_{z}\leq-0.75$, while in the prograde case the transition
is more dependent on $\epsilon_{\rm SA}$, varying from $j_{z}\sim 0.5$ for
large $\epsilon_{\rm SA}$ to $j_{z}\sim 0.75$ for small $\epsilon_{\rm SA}$.
For stable regions, we can find numerically the transition between the
librating and circulating orbits. We find the following numerical fit

$\epsilon_{{\rm SA}}(j_{z})=\begin{cases}0.4(\sqrt{3/5}-j_{z})^{0.7}&j_{z}>0;\
\epsilon_{{\rm SA}}\in[0.01,0.16]\\\ 0.08+0.5(0.8+j_{z})^{0.52}&j_{z}<0;\
\epsilon_{{\rm SA}}\in[0.08,0.23]\\\ 0.08-0.54(0.8+j_{z})^{0.52}&j_{z}<0;\
\epsilon_{{\rm SA}}\in[0.01,0.08]\end{cases},$ (20)

which is plotted in dashed white lines in Fig. 7. Note that the retrograde fit
could be multi-valued in $\epsilon_{\rm SA}$ for fixed $j_{z}$, and the
minimal $j_{z}$ s attained at $\epsilon_{\rm SA}\approx 0.08$.

We also plot additional insets of Fig. 7 with higher resolution. Fig. 8 shows
higher resolution grids of the red rectangles. An additional second zoom-in is
presented in the right panels. We see that higher resolution revealed
additional structure, which indicates a fractal boundary.

Fig. 9 shows the insets of the black rectangle, run for longer times of
$1000n_{\rm c}$. We see that the longer integration times make more orbits
unstable, but most of the librating orbits (green) are retained. A Large
number of the circulating orbit is becoming unstable. We also checked the
stability of a few initially circulating orbits (e.g. $\epsilon_{\rm
SA}=0.22,j_{z}=-0.69$), which become unstable after $n_{c}\approx 20,000$
cycles. We suspect that the transition to circulation could ba a pathway to
long-term instability for orbits near the stability limit. Further simulations
with longer integration times are required to explore this transition and will
be studied elsewhere.

Figure 10: Highly inclined orbital integration. The period ratio, or
$\epsilon_{\rm SA}=0.3$. The different panels show the evolution of the
argument of pericentre $\omega_{1}$, eccentricity $e_{1}$, instantaneous
separation $r$ and mutual inclination $\iota$.

Another interesting feature is the stable, highly inclined orbits around
$\epsilon_{\rm SA}\approx 0.3$. We Show in Fig. 10 the orbital evolution of a
particular orbit with $\epsilon_{\rm SA}=0.3,j_{z}=-0.02$. we integrated the
orbit for $n_{c}=100$ secular cycles. We also ran a longer integration for
$n_{c}=10^{5}$ secular cycles and verified that the orbit remained stable.

Although the orbit is chaotic, the argument of pericentre is librating and
stable. The reason may lie in the fact that $\mathcal{H}_{\rm
fixed}-\mathcal{H}_{\rm sep}$ is larger for lower vales of $j_{z}$ (see the
trend in Fig. 2). Thus, even highly perturbed orbits could retain a libration
motion of $\omega_{1}$. Additional work is required to tackle these orbits. In
practice, the large radial extent (left bottom panel) will cause a satellite
to either collide with the planet or to be scattered with other satellites, so
it is likely to be lost either via collision or ejection.

##  4 Summary and conclusions

We studied analytically and numerically the long-term evolution of triple
systems of mild hierarchy, where the period ratio between the inner and outer
orbits is not too small. Contrary to other studies, we focused on the fixed
points of the Brown Hamiltonian in eccentricity $e_{1}$ and argument of
pericentre $\omega_{1}$ phase space. Our conclusions are summarised below:

  * •

We have calculated for the first time the fixed points of the Brown
Hamiltonian (Eq. 15-16 and Fig. 4). The fixed point are expressed in terms of
dimensionless quantities $j_{z}=\sqrt{1-e_{1}^{2}}\cos\iota$ and
$\epsilon_{\rm SA}$ (Eq. 5), and could be applied to wide range of
astrophysical applications.

  * •

We’ve integrated numerically several orbits to test the limitation of our
analytic approach and visualise the morphology of the orbits (Fig. 5-6. We
found that the retrograde librating orbits extend to large values of $|j_{z}|$
and $\epsilon_{\rm SA}$, and for marginal orbits the escape usually occurs in
the low eccentricity stage.

  * •

We have numerically explored the boundary between circulating, librating and
unstable orbits (Fig. 7-8). We found a complex, fractal boundary. Retrograde
orbits tend to be more stable (a known result for initially circular orbits,
Grishin et al., 2017; Tory et al., 2022) and librating retrograde orbits tend
to occupy a larger region of parameter space. We provide a numerical fit for
the librating-circulating boundary $\epsilon_{\rm SA}(j_{z})$ in Eq. 20, and
find that the critical $j_{z}$ has a larger range for prograde orbits, while
retrograde librate can be recorded both for larger $|j_{z}|$ and
$\epsilon_{\rm SA}$.

Future work will expand the Brown Hamiltonian formalism beyond the test
particle limit where the three masses are comparable, and/or where the outer
orbit is eccentric. Also, the exact transition to circulating orbit and the
timescale for transition could also behave chaotically and are interesting
future research directions.

Extra apsidal advance from other, non-Keplerian forces may quench the ZLK
cycles. Considering additional non-Keplerian forces (either conservative or
dissipative) will be relevant for other applications in stellar and compact
object dynamics. For example, general-relativity and tidal evolution could
affect the long-term evolution and the final fate of triple systems.

In paper II we focus on the long-term evolution of irregular satellites in the
Solar system. We will construct a condition on the energy of the satellite
from Brown’s Hamiltonian and the analytic fluctuating terms. We’ll confirm
that the known librating satellites satisfy our criterion, and find several
new librating satellites.

## Acknowledgements

EG thanks Alessandro Trani, Isobel Romero-Shaw, Mor Rozner, Rosemary Mardling
and Scott Tremaine for stimulating discussions and comments on the manuscript.
EG acknowledges support from ARC grant FT190100574 (CI: Mandel).

## Data Availability

No data used in this work. The methods are publicly available in the GitHub
repository.

## Appendix A: Libration-circulations boundary

Figure 11:  Additional semi-analytical to fit the librating-circulating
binary, zoomed in on the prograde boundary (left) and the retrograde boundary
(right). In addition to the dashed black line from Eq. 20 (originally dashed
white in Fig. 7), we plot also a fit based on setting $e_{\rm fix}=0$ (dashed
blue), and on the energy fluctuations (dashed red). See text fro details. The
transparent green and blue grid have the same meaning as in Fig. 7 and 8.

In Fig. 7 we used an empirical fit based on numerical integration (Eq. 20).
Here we attempt two additional semi-analytic fits:

i) Bifurcation point: We take $e_{\rm fix}=0$ from Eq. 15. This yields the
relation

$\epsilon_{\rm
SA}=\frac{8}{9j_{z}}\frac{1-\frac{5}{3}j_{z}^{2}}{1+\frac{5}{3}j_{z}^{2}}$
(21)

ii) Energy fluctuation: We vary the total Hamiltonian at the fixed point
$\mathcal{H}_{\rm fixed}$ to $d\mathcal{H}_{\rm fixed}=\mathcal{H}(e_{\rm
fix},\omega=\pi/2|\epsilon_{\rm SA},j_{z}\pm\Delta j_{z})$. Here, $\Delta
j_{z}$ is the typical $j_{z}$ energy fluctuation which is proportional to
$\epsilon_{\rm SA}$ (see details and exact expressions Luo et al., 2016 and
paper II). We pick the sign in $j_{z}\pm\Delta j_{z}$ which gives the smaller
$d\mathcal{H}_{\rm fixed}$. We then plot the the contour where
$d\mathcal{H}_{\rm fixed}$ equals the energy at separatrix.

Fig. 11 shows the various fits with the numerical data. We see that all the
fits converge for $\epsilon_{\rm SA}=0$ to $j_{z}=\sqrt{3/5}\approx 0.775$ and
also have the same first order expansion of $\epsilon_{\rm SA}(j_{z})$.
However, the curved are markedly diverge for $\epsilon_{\rm SA}\gtrsim 0.5$.
The energy condition (ii) performs slightly better, but quickly diverges too
(dashed red lines), while the bifurcation point (i) is ever further (dashed
blue lines) from the empirical fit. Although the general trend is correct, the
analytical fits tend to diverge for larger $\epsilon_{\rm SA}$, which requires
further investigation.

A condition that uses a variant of the energy condition is developed in paper
II and applied to the available data on irregular satellites.

## References

  * Antognini (2015) Antognini J. M. O., 2015, MNRAS, 452, 3610
  * Beaugé et al. (2006) Beaugé C., Nesvorný D., Dones L., 2006, AJ, 131, 2299
  * Brouwer & Clemence (1961) Brouwer D., Clemence G. M., 1961, Methods of celestial mechanics
  * Brown (1936a) Brown E. W., 1936a, MNRAS, 97, 56
  * Brown (1936b) Brown E. W., 1936b, MNRAS, 97, 62
  * Brown (1936c) Brown E. W., 1936c, MNRAS, 97, 116
  * Bub & Petrovich (2020) Bub M. W., Petrovich C., 2020, ApJ, 894, 15
  * Carruba et al. (2002) Carruba V., Burns J. A., Nicholson P. D., Gladman B. J., 2002, Icarus, 158, 434
  * Ćuk & Burns (2004) Ćuk M., Burns J. A., 2004, AJ, 128, 2518
  * Deane et al. (2014) Deane R. P., et al., 2014, Nature, 511, 57
  * Fragione et al. (2019) Fragione G., Grishin E., Leigh N. W. C., Perets H. B., Perna R., 2019, MNRAS, 488, 47
  * Ginat & Perets (2021a) Ginat Y. B., Perets H. B., 2021a, arXiv e-prints, p. arXiv:2108.01085
  * Ginat & Perets (2021b) Ginat Y. B., Perets H. B., 2021b, Physical Review X, 11, 031020
  * Grishin (2024) Grishin E., 2024, MNRAS, submitted
  * Grishin & Perets (2022) Grishin E., Perets H. B., 2022, MNRAS, 512, 4993
  * Grishin et al. (2017) Grishin E., Perets H. B., Zenati Y., Michaely E., 2017, MNRAS, 466, 276
  * Grishin et al. (2018) Grishin E., Perets H. B., Fragione G., 2018, MNRAS, 481, 4907
  * Grishin et al. (2020) Grishin E., Malamud U., Perets H. B., Wandel O., Schäfer C. M., 2020, Nature, 580, 463
  * Haim & Katz (2018) Haim N., Katz B., 2018, MNRAS, 479, 3155
  * Hamers (2021) Hamers A. S., 2021, MNRAS, 500, 3481
  * Hamilton & Burns (1991) Hamilton D. P., Burns J. A., 1991, Icarus, 92, 118
  * Hamilton & Rafikov (2019) Hamilton C., Rafikov R. R., 2019, MNRAS, 488, 5512
  * Hansen & Naoz (2020) Hansen B. M. S., Naoz S., 2020, MNRAS, 499, 1682
  * Harrington (1968) Harrington R. S., 1968, AJ, 73, 190
  * Innanen (1979) Innanen K. A., 1979, AJ, 84, 960
  * Innanen (1980) Innanen K. A., 1980, AJ, 85, 81
  * Kinoshita & Nakai (2007) Kinoshita H., Nakai H., 2007, Celestial Mechanics and Dynamical Astronomy, 98, 67
  * Klein & Katz (2023) Klein Y. Y., Katz B., 2023, ApJ, 953, L10
  * Klein & Katz (2024) Klein Y. Y., Katz B., 2024, AJ, 167, 80
  * Kol (2021a) Kol B., 2021a, arXiv e-prints, p. arXiv:2107.12372
  * Kol (2021b) Kol B., 2021b, Celestial Mechanics and Dynamical Astronomy, 133, 17
  * Kozai (1962) Kozai Y., 1962, AJ, 67, 591
  * Kushnir et al. (2013) Kushnir D., Katz B., Dong S., Livne E., Fernández R., 2013, ApJ, 778, L37
  * Lei (2019) Lei H., 2019, MNRAS, 490, 4756
  * Lei et al. (2018) Lei H., Circi C., Ortore E., 2018, MNRAS, 481, 4602
  * Lidov (1962) Lidov M. L., 1962, Planet. Space Sci., 9, 719
  * Liu & Lai (2018) Liu B., Lai D., 2018, ApJ, 863, 68
  * Liu et al. (2015) Liu B., Muñoz D. J., Lai D., 2015, MNRAS, 447, 747
  * Lubow (2021) Lubow S. H., 2021, MNRAS, 507, 367
  * Luo et al. (2016) Luo L., Katz B., Dong S., 2016, MNRAS, 458, 3060
  * Mangipudi et al. (2022) Mangipudi A., Grishin E., Trani A. A., Mandel I., 2022, ApJ, 934, 44
  * Manwadkar et al. (2021) Manwadkar V., Kol B., Trani A. A., Leigh N. W. C., 2021, MNRAS, 506, 692
  * Marchis et al. (2005) Marchis F., Descamps P., Hestroffer D., Berthier J., 2005, Nature, 436, 822
  * Martin & Triaud (2014) Martin D. V., Triaud A. H. M. J., 2014, A&A, 570, A91
  * Monaghan (1976a) Monaghan J. J., 1976a, MNRAS, 176, 63
  * Monaghan (1976b) Monaghan J. J., 1976b, MNRAS, 177, 583
  * Mushkin & Katz (2020) Mushkin J., Katz B., 2020, MNRAS, 498, 665
  * Naoz (2016) Naoz S., 2016, ARA&A, 54, 441
  * Naoz et al. (2013) Naoz S., Farr W. M., Lithwick Y., Rasio F. A., Teyssandier J., 2013, MNRAS, 431, 2155
  * Offner et al. (2023) Offner S. S. R., Moe M., Kratter K. M., Sadavoy S. I., Jensen E. L. N., Tobin J. J., 2023, in Inutsuka S., Aikawa Y., Muto T., Tomida K., Tamura M., eds, Astronomical Society of the Pacific Conference Series Vol. 534, Protostars and Planets VII. p. 275 (arXiv:2203.10066), doi:10.48550/arXiv.2203.10066
  * Poincaré (1892) Poincaré H., 1892, Les méthodes nouvelles de la mécanique céleste, doi:10.3931/e-rara-421.
  * Rein & Spiegel (2014) Rein H., Spiegel D. S., 2014, MNRAS, 446, 1424
  * Rozner et al. (2020) Rozner M., Grishin E., Perets H. B., 2020, MNRAS, 497, 5264
  * Soderhjelm (1975) Soderhjelm S., 1975, A&A, 42, 229
  * Stone & Leigh (2019) Stone N. C., Leigh N. W. C., 2019, Nature, 576, 406
  * Tory et al. (2022) Tory M., Grishin E., Mandel I., 2022, Publ. Astron. Soc. Australia, 39, e062
  * Tremaine (2023) Tremaine S., 2023, MNRAS, 522, 937
  * Wagner et al. (2016) Wagner K., Apai D., Kasper M., Kratter K., McClure M., Robberto M., Beuzit J.-L., 2016, Science, 353, 673
  * Will (2021) Will C. M., 2021, Phys. Rev. D, 103, 063003
  * von Zeipel (1910) von Zeipel H., 1910, Astronomische Nachrichten, 183, 345


# AraLegal-BERT: A pretrained language model for Arabic Legal text

Muhammad AL-Qurishi, Sarah AlQaseemi  Riad Soussi
Elm Company,
Research Department,
Riyadh 12382, Saudia Arabia

###### Abstract

The effectiveness of the BERT model on multiple linguistic tasks has been well
documented. On the other hand, its potentials for narrow and specific domains
such as Legal, have not been fully explored. In this paper, we examine how
BERT can be used in the Arabic legal domain and try customizing this language
model for several downstream tasks using several different domain-relevant
training and testing datasets to train BERT from scratch. We introduce the
AraLegal-BERT, a bidirectional encoder Transformer-based model that have been
thoroughly tested and carefully optimized with the goal to amplify the impact
of NLP-driven solution concerning jurisprudence, legal documents, and legal
practice. We fine-tuned AraLegal-BERT and evaluated it against three BERT
variations for Arabic language in three natural languages understanding (NLU)
tasks. The results show that the base version of AraLegal-BERT achieve better
accuracy than the general and original BERT over the Legal text.

AraLegal-BERT: A pretrained language model for Arabic Legal text

Muhammad AL-Qurishi, Sarah AlQaseemi and Riad Soussi Elm Company, Research
Department, Riyadh 12382, Saudia Arabia

##  1 Introduction

The impressive performance of Bidirectional Encoder Representations from
Transformers (BERT) Devlin et al. (2018) inspired numerous authors to try and
improve on the original BERT. This type of follow-up research goes in several
different directions, including the development of specific solutions for
various thematic domains. This is necessary because the vocabulary used in
some fields significantly deviates from the language used for everyday
purposes and may include specific meanings of certain phrases or atypical
relationships between contextual elements. This problem can be partially
resolved with domain-specific adjustments of the training process – a good
example of this approach is demonstrated in the work by Chalkidis et al.
(2020), who created Legal-Bert specifically for mining text related to English
legal text, improving the output of the standard Transformer algorithm within
this domain. Another successful adaptation of the BERT concept was performed
by Beltagy et al. (2019); Lee et al. (2020), who created models that were pre-
trained on a compilation of scientific and Biomedical data from various
fields, achieving significantly better performance on scientifically-related
NLP tasks. Those examples show that BERT is far from a finished model at the
moment, and that its current effectiveness could be enhanced even further, at
least when it comes to relatively narrowly defined tasks.

For Arabic language, several BERT-based models have consistently demonstrated
superior performance on numerous linguistic tasks that require semantic
understanding, outperforming all benchmarks on public datasets such as Arabic
NER corpus (ANERcorp) or Arabic Reading Comprehension Dataset (ARCD), such as
the works presented by Antoun et al. (2020); Abdul-Mageed et al. (2020) and
mBERT by Google Devlin et al. (2018). This is largely a consequent of
efficient transfer learning inherent in this model, which comes at a high
computational price as this approach requires huge collections of training
examples, followed by fine-tuning for specific downstream tasks. One great
advantage of BERT is that the training phase can be skipped since it’s
possible to take already pre-trained version of the model and train it
further. However, it has been shown by Chalkidis et al. (2020); Beltagy et al.
(2019); Lee et al. (2020) that generalist approach to pre-training doesn’t
work well when BERT needs to be used within a domain with highly specific
terminology, such as legal, science or medicine. There are two possible
responses to this issue, namely, to continue specializing an already pre-
trained model or to start training it from the beginning with relevant
material from the domain.

In this paper, we build a language model from scratch based on the original
Bert Devlin et al. (2018), this model is specific to Arabic legal texts with
the aim of improving the performance of state-of-the-art in most of the tasks
in language understanding and processing, especially Arabic legal texts. We
believe that it is important to take into account the specific nature of legal
documents and terminology, which affects the way sentences and paragraphs are
constructed in this field. In fact, the extent of formal and semantic
differences is such that some authors describe the linguistic content used for
legal matters as almost a language of its own Chalkidis et al. (2020).

By focusing on one Arabic legal text area, this paper attempts to reveal how
NLP model can be adjusted to fit any thematic domain. Based on our
experiments, we can confirm that pre-training BERT with examples from the
Arabic legal domain from scratch provides a better foundation for working with
documents containing Arabic legal vocabulary than using vanilla version of the
algorithm.

We introduce the AraLegal-BERT, a Transformer-based model that have been
thoroughly tested and carefully optimized with the goal to amplify the impact
of NLP-driven solution concerning jurisprudence, legal documents, and legal
practice. We fine-tuned AraLegal-BERT and evaluated it against three BERT
variations for Arabic language in three NLU tasks. The results shows that the
base version of AraLegal-BERT achieves better accuracy than the general and
original BERT over the Legal text. AraLegal-BERT is a particularly efficient
model that can match the output of computationally more demanding models while
reaching its findings faster and expending far less resources. Consequently,
it was observed that the base version of the model can reach comparable
accuracy levels to larger versions of the large general and orginal BERT when
they are trained with domain-relevant examples similar to ones use for testing
our model.

##  2 Related work

In a very short time, Transformers Vaswani et al. (2017) architecture has
become the golden standard of machine learning methodologies in the field of
linguistics Wolf et al. (2020); Su et al. (2022). Unparalleled success of BERT
in combination with its flexibility lead to proliferation of tools based on
it, created with a more narrowly defined vocabulary Young et al. (2018); Brown
et al. (2020). AraBERT Antoun et al. (2020); Abdul-Mageed et al. (2020), are
examples of such specialization, and given the number of Arabic speakers
worldwide it could have considerable practical value. Since the model is
trained for some of the most commonly used NLP tasks and proven effective
across regional variations in morphology and syntax, those language models
have a chance to become the default tool for analyzing Arabic text. Pre-
training and fine tuning procedures that were described in this work may not
be the most optimal ones, but the output of the localized model clearly
indicated that the initial approach was correct. With further refinement, the
model could become reliable enough for a wide range of real world
applications. On the other hand, these models are based on data most of which
were collected from the Modern Standard Arabic, and these language models may
fail when the language changes to colloquial dialects Abdelali et al. (2021).
In addition, the performance of these language models may be affected when
dealing with a language for a specific domain with special terms such as
scientific, medical and legal terms Yeung (2019).

The majority of BERT domain-specific forms are related to scientific or
medical texts, as well as legal but all are in English language Beltagy et al.
(2019); Lee et al. (2020); Chalkidis et al. (2020). In a study by Alsentzer et
al. (2019) the main area of interest was clinical practice, so the authors
developed two different variations by pre-training basic BERT and BIOBERT with
examples from this domain, with positive results in both cases. Another
interesting project of this kind was conducted by Beltagy et al. (2019),
resulting in the creation of SCIBERT, a whole branch of variations optimized
for use with scientific documents. In this case, two different optimization
strategies were tested, including additional training and training from
scratch using documents that include scientific terminology, with both
approaches creating measurable improvements. A study by Chalkidis et al.
(2020) included pre-training of Transformer models for English legal text with
comparing three possible approaches for adaptation of BERT to thematic content
niches – 1) using the vanilla model without any modifications; 2) introducing
pre-training with datasets that contain examples from the target domain on top
of standard training; and 3) Using only domain-relevant samples to train BERT
from scratch.

Practically all of the BERT adaptations that include fine-tuning use the same
approach for the selection of hyper-parameters as outlined in the original
BERT formulation without even questioning it. Another research gap is observed
when it comes to the possibility of using shallower models for executing
domain-specific tasks. It could be argued that the impressive generalization
capacity characteristic for deep models with lots of layers is wasted when the
model works within a narrowly defined field, where linguistic rules are more
streamlined and the vocabulary volume is more limited. Despite the fact that
BERT has been the most successful deep learning model for many tasks related
to the legal sphere, so far there have been no publicized attempts to develop
a variation unique for this type of content especially in Arabic, which was
largely a motivation for this study. Therefore, to the best of our knowledge,
this is the first work to build a BERT-based language model for legal texts in
Arabic.

##  3 AraLegal-BERT: Transformer Model Pre-trained with Arabic Legal Text

In order to be able to optimize BERT for working with Arabic legal documents,
we followed the same procedures in the original BERT model Devlin et al.
(2018); however, for Arabic language we followed the same procedure in AraBERT
Antoun et al. (2020).

###  3.1 Dataset

Due to relative scarcity of publically available large-scale resources in
Arabic lagal text, the training dataset had to be manually collected from
several sources and included many regional variations. All the collected
documents are in Arabic language related to several different sub-fields of
legal practice, such as legislative documents, judicial documents, contracts
and legal agreements, Islamic rules and Fiqh. All data was collected from
public sources, and the final size of the dataset was 4.5GB. The final size of
the training set after the removal of duplicates was approximately 13.7
million sentences. Table 1 describes the used dataset.

Table 1: The dataset used to train AraLegal-BERT

Type | Sample Size | Desc
---|---|---
Books | 6K | Master and PhD theses, research papers, magazines, dictionaries and Fiqh books
Cases | 336K | Legal Cases in KSA and Gulf countries which consists of copy rights, design rights, facts and appealing
Terms and laws | 3K | Laws and regulations in KSA and Gulf countries
others | 5K | Reports and studies, academic courses, forms, reports, contracts

###  3.2 AraLegal-BERT

This version of the model was created by following up on the original pre-
training with additional steps involving textual material from the Arabic
legal domain. The authors of the original BERT indicated that 100,000 steps
would be sufficient, but in our implementation the model was trained with up
to half million steps to determine how extended pre-training with narrowly
focused data samples affects performance on various linguistic tasks. The pre-
training of BERT base Devlin et al. (2018) with general content involves a lot
more steps, which is why the model tends to be most proficient with a
vocabulary containing around 30,000 words commonly found in everyday language.
It was presumed that with extended training using domain-focused examples this
tendency can be partially reversed, with a positive impact on model accuracy.

Before we start the training, data pre-processing was performed and in this
part we followed the same procedure in Antoun et al. (2020). To account for
the unique nature of Arabic prefixes, sub-word segmentation was performed to
separate all tokens into prefixes, stems, and suffixes as explained in
Abdelali et al. (2016). This resulted in a vocabulary of around 64K words,
which was used to pre-train the model and create AraLegal-BERT. Then we traind
our model on masked language modeling (MLM) task where 15% of the entire input
sequence were used as tokens as 80% of them were masked, 10 percent were
replaced with a random token, and only 10% were left in their natural state.
This procedure allows the algorithms to derive conclusions based on entire
words rather than just linguistic elements, which is better suited for the
Arabic Language.

##  4 Experimental procedure

###  4.1 Pre-training stage

AraLegal-BERT was trained for approximately 50 epochs, involving a total of
half million steps, which is similar to the original BERT pre-training
procedure. We have trained our model in Elm research center using NVIDIA DGX1
with 8 GPUs. Batch size was set at 8 per gpu, so the total train batch size
(w. parallel, distributed & accumulation) is 512. The maximum sequence length
is 512 tokens and a learning rate ranging from $1e-5$ to $5e-5$.

###  4.2 Fine-tuning

The authors of BERT Devlin et al. (2018) proposed an approach for determining
the most optimal parameters for fine-tuning that is based on a search within a
limited range. In this concept, the learning rate, length of training, size of
training batch and dropout rate are all either fixed or can take one of a few
possible discreet values. While no particular reason has been given for this
approach, it has been widely replicated in studies dealing with BERT
derivatives Wehnert et al. (2022); Rogers et al. (2020). Since those
parameters don’t always produce the best results and their use may still
result in a model remaining undertrained, an alternative strategy was adopted
for choosing the upper limit of training epochs that tracks validation loss
and terminates training only when the conditions are met.

####  4.2.1 Legal text classification

The samples within the experimental dataset were collected from two main
portals. The first data set was collected from the Scientific Judicial
Portal(SJP)111https://sjp.moj.gov.sa/, which is run by the of Ministry of
Justice in Saudi Arabia. SJP is the largest specialized information database
in the field of the judiciary in the Kingdom of Saudi Arabia. It is the ideal
solution for specialists in the judicial and legal field, including judges,
lawyers, trained lawyers, academics, prosecutors, graduate students, and
others. The second data we collected was from the Board of Grievances(BoG)
portal222https://www.bog.gov.sa/en/ScientificContent/Pages/default.aspx, which
is as they stated in their website: "The Board tested a judiciary academic
series in the name of (judge library) and its distribution among the Board
judges (hard copy and soft copy) to increase cognitive formation with them, a
state which its effect shall be reflected on the judiciary verdicts they
issue, including academic references in administrative, commercial and penal
judiciary formed of 32 volumes in addition to judiciary verdicts".

Since the existing documents in both datasets can belong to several categories
depending on the submission details, they are suitable for the task of long
length legal document classification. Three different classes of documents
were selected from the SJP dataset and ten classes from the BoG dataset. Since
practically all documents from certain classes are headlined in the same way,
for the classification task it was necessary to truncate the parts of the
document containing easily identifiable indicators of the class. Due to this
omission, the model needs to analyze the entire content instead of simply
deriving the conclusion based on the first few lines. This modification was
implemented with all classes.

####  4.2.2 Keyword Extraction

Unfortunately, compared to English and some other Latin languages, there is no
ready-made and well-prepared data for research purposes, especially in the
field of natural languages processing of legal texts in Arabic. Therefore, we
built our data in this task with the help of professionals in the Arab legal
field. This dataset consists of approximately 8K legal documents containing
main keywords, extracted manually by these professionals. We did a
preprocessing and cleaning of the data and extracted about 37640 sentences
containing both keywords and others. The average sentence length was no more
than 65 words, as we did a sentence segmentation process so that the sentence
does not lose its meaning or break. We tagged the keywords in the sentence
with the number 1 and the others with the number 0.

####  4.2.3 Named Entity Recognition

This dataset also was created inside the research department at Elm company.
It contains more than 311K sentences including thousands of distinct entities
of seventeen different sequence tags that labeled manually by multiple human
annotators as of part of our CourtNLP project in Elm
research333https://www.elm.sa/en/research-and-innovation/Pages/Research.aspx.
All the used classes are shown in Figure 1. The main objective of NER
procedure is to associate a label belonging to a particular class to each
included word. It’s important to note that some complex named entities could
stretch across multiple words, but are always contained in a single sentence.
The predominant sentence representation form used in this field is IOB, with
words that start a name of an entity marked with B, internally located words
marked as I, and other tokens marked with O.

Figure 1: The main NER tags Table 2: The overall results of all fine-tuned models in the legal text classification task on BoG Dataset Model / Macro-Average | Precision | Recall | F1-score
---|---|---|---
Arabertv2-Large Antoun et al. (2020) | 0.850387 | 0.810795 | 0.827078
ARBERT Abdul-Mageed et al. (2020) | 0.802514 | 0.821973 | 0.812820
mBERT Devlin et al. (2018) | 0.702017 | 0.635928 | 0.598267
AraLegal-BERT (base) | 0.89276 | 0.89173 | 0.89098
Table 3: The overall results of all fine-tuned models in the legal text classification task on SJP Dataset Model / Macro-Average | Precision | Recall | F1-score
---|---|---|---
Arabertv2-Large Antoun et al. (2020) | 0.885678 | 0.886816 | 0.884516
ARBERT Abdul-Mageed et al. (2020) | 0.837714 | 0.834804 | 0.843827
mBERT Devlin et al. (2018) | 0.814763 | 0.780226 | 0.782501
AraLegal-BERT (base) | 0.92395 | 0.92133 | 0.92210

##  5 Results

###  5.1 Impact of pre-training

We trained two models from scratch, the first one is a base model that
contains 12 layers and the second one is a large model consists of 24 layers,
as in the original BERT. As could be anticipated, full-sized with 24-layer
model that was trained from scratch has a much better ability to fulfill the
pre-training objectives than the base model with 12-layer. However, after the
pre-training stage is completed the base model displays similar level of loss
to original BERT that is trained on general datasets. In particular, it is
faster to adjust to narrowly defined niches, while can be a great advantage
for domain-focused applications such legal. For this reason, the content of
the training set plays into the selection of the suitable training method. We
did not experiment on the large model yet, and all the fine-tuning results
were based on the base model since we find that it gives much better accuracy
than the general Arabic BERT models in the three defined NLU tasks.

###  5.2 Results discussion

We have divided the datasets into training, validation and testing datasets
with respect to all of the three tasks. Here we will discuss the testing
results. The evaluation was conducted using standardized hyperparameters such
as batch size and sequence length, with several different datasets suitable
for legal text classification, keyword extraction, and named entity
recognition.

In the first task of legal text classification, the best option in this task
is determined based on experimental results, for example, in this way we found
that multiple strategies can be used to get around BERT’s limit to sequence
length of 512 tokens, but the ‘head & tails’ strategy where only the first 128
and the last 382 tokens are kept performs the best such as the work in Sun et
al. (2019). Tables 2 and 3 summarize the over all results of our model
compared to three BERT variations for Arabic language on the classification
task with two datasets namely, SJP and BoG. On the dataset of BoG, AraLegal-
BERT, outperformed the all models by 0.7% in F1-Macro average higher than the
highest model of the three, which is ARABERT-v2large. Similarly, our model
also outperformed the rest of the models in the SJP dataset, with a good
difference between it and ArabBERTv2-large, which is approximately 0.4% in
F1-Macro average.

On the other hand, as for the tasks related to the named enity recognition as
well as the keyword extraction, we followed the same procedure that was done
in our previous work Al-Qurishi and Souissi (2021), taking into account that
no new layers were added to the model, a linear layer was used to make the
word and sequence-level tagging. In these two tasks, the results are more
different, and there is a significant difference between the performance of
AraLegal-BERT and the rest of the models, where AraLegal-BERT outperformed
ARABERT-v2large Antoun et al. (2020) with a difference of almost 21% in the
F1-Macro average in the task of extracting named entities as depects in Table
5, while the difference is much greater in the keyword extraction task, where
AraLegal-BERT outperformed the highest model which is ARBERT Abdul-Mageed et
al. (2020) with a significant difference equal to almost 26% in the F1-Macro
average as shown in Table 4.

We denote that the general BERT models were not only less in the F1-macro
score, but also in the recall-macro score, where they are not able to retrieve
most of the required words compared to AraLegal-BERT. Finally, it is worth
noting that the used AraLegal-BERT model is the base version; nevertheless, it
has outperformed the rest of the models in all the three defined tasks with
less memory size and faster performance as well as good accuracy score.

Table 4: The overall results of all fine-tuned models in the Keywords Extraction Task Model / Macro-Average | Precision | Recall | F1-score
---|---|---|---
Arabertv2-Large Antoun et al. (2020) | 0.765979 | 0.470098 | 0.582625
ARBERT Abdul-Mageed et al. (2020) | 0.789460 | 0.524715 | 0.630420
mBERT Devlin et al. (2018) | 0.721356 | 0.375075 | 0.493533
AraLegal-BERT (base) | 0.93481 | 0.84449 | 0.88736
Table 5: The overall results of all fine-tuned models in the Named Entity Recognition Task Model / Macro-Average | Precision | Recall | F1-score
---|---|---|---
Arabertv2-Large Antoun et al. (2020) | 0.891934 | 0.450073 | 0.598261
ARBERT Abdul-Mageed et al. (2020) | 0.889916 | 0.413266 | 0.564423
mBERT Devlin et al. (2018) | 0.886825 | 0.326475 | 0.477254
AraLegal-BERT (base) | 0.89848 | 0.73644 | 0.80943

##  6 Conclusions and future work

Based on the findings of our experiments, the pre-training of BERT for a
specific domain is better than the general language models in certain NLU
task. For this reason, we present AraLegal-BERT model that were trained
exclusively for Arabic legal texts and are able to make very accurate
predictions on three main NLU tasks, namely Legal text classification, named
entity recognition and keywords extraction. Crucially, the level of difficulty
of the task is correlated with the gains from choosing the right training
strategy, as the importance of domain-specific vocabulary and semantics
becomes more pronounced. The tested version of AraLegal-BERT model is the
base, cost-efficient version suitable for a broad range of Arabic legal
textual applications. In the near future, future work will focus on additional
possibilities for pre-training other models such as Electra and Roberta models
for several NLU tasks in the Arabic legal domain with small, base and large
versions.

## Acknowledgements

This work was supported and funded by Elm Company Research department.

## References

  * Abdelali et al. (2016) Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016.  Farasa: A fast and furious segmenter for arabic.  In _Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations_ , pages 11–16.
  * Abdelali et al. (2021) Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021\.  Pre-training bert on arabic tweets: Practical considerations.  _arXiv preprint arXiv:2102.10684_.
  * Abdul-Mageed et al. (2020) Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020.  Arbert & marbert: deep bidirectional transformers for arabic.  _arXiv preprint arXiv:2101.01785_.
  * Al-Qurishi and Souissi (2021) Muhammad Saleh Al-Qurishi and Riad Souissi. 2021.  Arabic named entity recognition using transformer-based-crf model.  In _Proceedings of The Fourth International Conference on Natural Language and Speech Processing (ICNLSP 2021)_ , pages 262–271.
  * Alsentzer et al. (2019) Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019.  Publicly available clinical bert embeddings.  _arXiv preprint arXiv:1904.03323_.
  * Antoun et al. (2020) Wissam Antoun, Fady Baly, and Hazem Hajj. 2020.  Arabert: Transformer-based model for arabic language understanding.  _arXiv preprint arXiv:2003.00104_.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  Scibert: A pretrained language model for scientific text.  _arXiv preprint arXiv:1903.10676_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chalkidis et al. (2020) Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020.  Legal-bert: The muppets straight out of law school.  _arXiv preprint arXiv:2010.02559_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020.  A primer in bertology: What we know about how bert works.  _Transactions of the Association for Computational Linguistics_ , 8:842–866.
  * Su et al. (2022) Xing Su, Shan Xue, Fanzhen Liu, Jia Wu, Jian Yang, Chuan Zhou, Wenbin Hu, Cecile Paris, Surya Nepal, Di Jin, et al. 2022.  A comprehensive survey on community detection with deep learning.  _IEEE Transactions on Neural Networks and Learning Systems_.
  * Sun et al. (2019) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019.  How to fine-tune bert for text classification?  In _China national conference on Chinese computational linguistics_ , pages 194–206. Springer.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wehnert et al. (2022) Sabine Wehnert, Shipra Dureja, Libin Kutty, Viju Sudhi, and Ernesto William De Luca. 2022.  Applying bert embeddings to predict legal textual entailment.  _The Review of Socionetwork Strategies_ , 16(1):197–219.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_ , pages 38–45.
  * Yeung (2019) Chin Man Yeung. 2019.  Effects of inserting domain vocabulary and fine-tuning bert for german legal language.  Master’s thesis, University of Twente.
  * Young et al. (2018) Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2018.  Recent trends in deep learning based natural language processing.  _ieee Computational intelligenCe magazine_ , 13(3):55–75.