Discovery potentials of double-charm tetraquarks

Qin<EMAIL_ADDRESS>corresponding author, Yin-Fa Shen School of
physics, Huazhong University of Science and Technology, Wuhan 430074, China

Fu-Sheng<EMAIL_ADDRESS>corresponding author

School of Nuclear Science and Technology, and Frontiers Science Center for
Rare Isotopes, Lanzhou University, Lanzhou 730000, China
Lanzhou Center for Theoretical Physics, and Key Laboratory of Theoretical
Physics of Gansu Province, Lanzhou University, Lanzhou 730000, China
Center for High Energy Physics, Peking University, Beijing 100871, China

###### Abstract

In this study, we investigate the discovery potential of double-charm
tetraquarks $T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$. We find that their
production cross sections at the LHCb with $\sqrt{s}=13$ TeV reach
$\mathcal{O}(10^{4})$ pb, which indicates that the LHCb has collected
$\mathcal{O}(10^{8})$ such particles. Through the decay channels of
$T^{\\{cc\\}}_{[\bar{u}\bar{d}]}\to D^{+}K^{-}\pi^{+}$ or $D^{0}D^{+}\gamma$
(if stable) or $T^{\\{cc\\}}_{[\bar{u}\bar{d}]}\to D^{0}D^{*+}\to
D^{0}D^{0}\pi^{+}$ (if unstable), it is highly expected that they get
discovered at the LHCb in the near future. We also discuss the productions and
decays of the double-charm tetraquarks at future Tera-$Z$ factories.

##  1 Introduction

Doubly-heavy tetraquarks including $T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$,
$T^{\\{bc\\}}_{[\bar{q}\bar{q}^{\prime}]}$, and
$T^{\\{bb\\}}_{[\bar{q}\bar{q}^{\prime}]}$ have drawn considerable attention
from theorists after the discovery of the first double-charm hadron
$\Xi_{cc}^{++}$ [1]. The doubly-heavy tetraquarks can shed light on strong
dynamics and greatly help us identify the nature of the exotic $XYZ$ states or
structures, e.g., cusps or true resonances. In addition, several studies of
both heavy quark symmetry and lattice [2, 3, 4, 5, 6, 7, 8] suggest that some
of them have masses below the threshold and thus can only decay weakly. If
such weakly-decaying tetraquarks are found, they will provide evidence of
compact diquarks as building blocks of hadronic matter. Among these doubly-
heavy tetraquarks, $T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$ is the most
promising one to be observed in experiments in the near future, because both
its production rate and detection efficiency are the highest at the LHCb. In
this work, we analyze the potential of finding
$T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$ at the LHCb and also future
$Z$-factories [9, 10], evaluating their production cross sections at the two
kinds of facilities and proposing possible golden channels according to their
decay properties.

Decay properties of the doubly-heavy tetraquarks
$T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$ rely on whether their masses are
above or below the thresholds $D^{0}D^{*+}$ and $D^{*}D_{s}^{+}$. If they are
stable particles below the threshold, they will mainly decay radiatively (or
even weakly according to [11]) and thus have a relatively long lifetime, which
will considerably suppress the background and make the experimental search
easier. In contrast, if they are unstable particles above the thresholds333If
the particles are slightly below the thresholds, they can also decay strongly
and thus unstable. For convenience, we also classify this special case as
”above threshold”., they will decay strongly and still have a good chance to
be found at the LHCb using a method similar to that in [12]. Their masses have
been calculated in many different procedures, such as QCD sum rule [13, 14,
15, 16, 17], lattice [18], Bethe-Salpeter equation approach [11], heavy quark
symmetry [3, 19] and several QCD inspired models [20, 21, 22, 23, 24, 25, 26,
27, 28, 29, 2, 30, 31, 32]. The results are listed in Table 1 in form of the
mass differences of $T^{\\{cc\\}}_{\bar{n}\bar{n}^{\prime}}$ and
$T^{\\{cc\\}}_{\bar{n}\bar{s}}$ ($n^{(\prime)}=u$ or $d$) from their
thresholds $D^{0}D^{*+}$ and $D^{*}D_{s}^{+}$, respectively. These results are
far from conclusive because some of the calculations suggest they lie above
the thresholds while the others are against it. As a result, the
$T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$ decay channels are analyzed case by
case.

Reference | [21] | [22] | [23] | [24] | [25] | [26] | [13] | [18] | [3] | [19] | [15]
---|---|---|---|---|---|---|---|---|---|---|---
$T^{\\{cc\\}}_{\bar{n}\bar{n}^{\prime}}$ | -79 | -96 | +53 | -150 | +166 | +60 | - | AT | +102 | +88 | +25
$T^{\\{cc\\}}_{\bar{n}\bar{s}}$ | -9 | -56 | +128 | +94 | +255 | +166 | +143 | AT | +179 | +181 | -15
Reference | [11] | [27] | [28] | [29] | [2] | [30] | [14] | [31] | [20] | [32] |
$T^{\\{cc\\}}_{\bar{n}\bar{n}^{\prime}}$ | -215 | BT | -149 | -182 | +7 | +98 | +91 | +125 | AT | AT |
Table 1: Theoretical predictions on the mass differences of
$T^{\\{cc\\}}_{\bar{n}\bar{n}^{\prime}}$ and $T^{\\{cc\\}}_{\bar{n}\bar{s}}$
from their thresholds $D^{0}D^{*+}$ and $D^{*}D_{s}^{+}$, respectively, in
units of MeV. The quarks $n,n^{\prime}$ = $u$ or $d$. For some works without
explicit numerical results, we use ”AT” and ”BT” denoting ”above threshold”
and ”below threshold”, respectively.

The rest of the paper is organized as follows. In section 2, we study the
production of $T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$ at both the LHCb and
future $Z$-factories. According to our estimation of the production cross
sections, the LHCb has produced $\mathcal{O}(10^{8})$ double-charm tetraquarks
to date, and this number will increase by almost one order of magnitude by the
end of Run 4 [33, 34]. It is also shown that a Tera-$Z$ factory will produce
$\mathcal{O}(10^{6})$ double-charm tetraquarks in a very clean environment. In
section 3, possible decay channels of
$T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$ are discussed, in three cases with
the double-charm tetraquarks either above or below the $DD^{*}$ and $DD\gamma$
thresholds. For the LHCb, we propose some decay channels to search for the
double-charm tetraquarks, because they have relatively large branching ratios,
and more importantly, all of their final-state particles have high detection
efficiencies, e.g., $T^{\\{cc\\}}_{[\bar{u}\bar{d}]}\to D^{0}D^{*+}\to
D^{0}D^{0}\pi^{+}$ and $T^{\\{cc\\}}_{[\bar{u}\bar{d}]}\to D^{+}K^{-}\pi^{+}$.
In contrast, future $Z$-factories have an advantage over the LHCb in searching
for other decay channels containing a photon, e.g.,
$T^{\\{cc\\}}_{[\bar{u}\bar{d}]}\to D^{0}D^{+}\gamma$. We conclude our study
in section 4.

##  2 Production rate

Following the approach proposed in [35, 36, 37], we estimate the production
rate of double-charm hadrons $H_{cc}$ at the LHCb and future $Z$-factories.
The basic idea is as follows. First, the underlying processes for their
production at the quark level are $pp\to cc\bar{c}\bar{c}+X$ and
$e^{+}e^{-}\to Z\to cc\bar{c}\bar{c}$ at these two kinds of facilities,
respectively. The two charm quarks must stay close enough to form a $cc$
diquark jet, which further fragments into different kinds of double-charm
hadrons. The invariant mass of the two heavy quarks $m_{QQ^{\prime}}$ is used
to parameterize their collinear level, and if $m_{QQ^{\prime}}$ is smaller
than some cut-off value $M_{QQ^{\prime}}(\Delta M)\equiv
m_{Q}+m_{Q^{\prime}}+\Delta M$, we regard the two heavy quarks as a
$QQ^{\prime}$ diquark that will eventually produce a doubly-heavy hadron. In
contrast, if $m_{QQ^{\prime}}$ exceeds $M_{QQ^{\prime}}(\Delta M)$, the two
heavy quarks fragment separately. The parameter $\Delta M$ can be determined
by matching the partonic $b\bar{c}$ production simulation to the experimental
measurements of the $B_{c}$ meson production cross sections [38], and it has
been found by [35, 37] that

$\displaystyle\Delta
M=\left\\{\begin{array}[]{l}\left(2.0^{+0.5}_{-0.4}\right)\text{GeV, for
LHCb,}\\\ \left(2.7^{+1.3}_{-0.5}\right)\text{GeV, for Z factories.}\\\
\end{array}\right.$ (3)

Next, we will discuss the production of the doubly-heavy hadrons at the LHCb
and future $Z$-factories separately.

LHCb: For the analysis of $H_{cc}$ production at the LHC with $\sqrt{s}=13$
TeV, we generate $10^{5}$ $pp\to cc\bar{c}\bar{c}+X$ events via the Monte
Carlo generator MadGraph5$\\_$aMC@NLO [39] at the next-to-leading-order level
showered by Pythia8 [40, 41]. It is found that the total cross section
$\sigma(pp\to cc\bar{c}\bar{c}+X)\approx 4.9\times 10^{7}$ pb, and after
applying the invariant mass cut in (3), we obtain

$\displaystyle\sigma(p+p\to H_{cc}+X)=(3.1^{+1.7}_{-0.7})\times 10^{5}\
\text{pb}\;,$ (4)

in the transverse momentum range $4<p_{\text{T}}<15$ GeV and the
pseudorapidity range $2<\eta<4.5$.

The double-charm hadrons $H_{cc}$ include both the double-charm baryons
$\Xi_{cc}^{++}(ccu)$, $\Xi_{cc}^{+}(ccd)$, and $\Omega_{cc}^{-}(ccs)$ and the
double-charm tetraquarks $T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$ and their
excited states. Therefore, we still need to know the relative fractions of
$H_{cc}$ to these hadrons. It is difficult to calculate these non-perturbative
fragmentation ratios, while we can approximately borrow the corresponding
ratios of a $b$-quark jet under the heavy quark - heavy diquark symmetry. Such
ratios have been measured by the LHCb [42, 43] as

$\begin{split}\left[\frac{f_{\Lambda_{b}^{0}}}{f_{d}}\right](p_{\text{T}})=&\
(0.151\pm 0.030)+\exp\left[(-0.57\pm 0.11)-(0.095\pm
0.016)p_{\text{T}}\text{(GeV)}\right]\;,\\\
\left[\frac{f_{s}}{f_{d}}\right](p_{\text{T}})=&\ (0.263\pm 0.008)+((-17.6\pm
2.1)\times 10^{-4})\cdot p_{\text{T}}\text{(GeV)}\;,\end{split}$ (5)

which are $p_{\text{T}}$ dependent. In addition, we assume that for states
containing the same valence quarks, the ground state fraction is approximately
$r_{g}=0.48\pm 0.08$ [44]. Then, convoluting the $p_{\text{T}}$-distribution
of the $cc$-diquark jets fetched from the simulated events with the
$p_{\text{T}}$-dependent fragmentation functions (5), we estimate the direct
production cross sections of the double-charm tetraquarks to be

$\displaystyle\sigma(pp\to T^{\\{cc\\}}_{[\bar{u}\bar{d}]}+X)=$
$\displaystyle\ (24^{+14}_{-7})\ \text{nb}\;,$ $\displaystyle\sigma(pp\to
T^{\\{cc\\}}_{[\bar{u}\bar{s}]}+X)=\sigma(pp\to
T^{\\{cc\\}}_{[\bar{d}\bar{s}]}+X)=$ $\displaystyle\ (6.0^{+3.5}_{-1.7})\
\text{nb}\;.$ (6)

The $p_{\text{T}}$ distribution of the $pp\to
T^{\\{cc\\}}_{[\bar{u}\bar{d}]}+X$ events from the simulation is displayed in
Figure 1. In contrast, because almost all the excited states of $\Xi_{cc}$ and
$\Omega_{cc}$ decay into their ground states, we do not need to multiply the
$r_{g}$ ratio for their production cross sections, which read

$\displaystyle\sigma(pp\to\Xi_{cc}^{++}+X)=\sigma(pp\to\Xi_{cc}^{+}+X)=$
$\displaystyle\ (103^{+56}_{-22})\ \text{nb}\;,$
$\displaystyle\sigma(pp\to\Omega_{cc}^{+}+X)=$ $\displaystyle\
(26^{+14}_{-6})\ \text{nb}\;.$ (7)

Figure 1: Projected $p_{\text{T}}$-dependence of the tetraquark production
cross section in $pp\to T^{\\{cc\\}}_{[\bar{u}\bar{d}]}+X$ at the LHC for
$\sqrt{s}=13$ TeV with $2<\eta<4.5$.

The $\Xi_{cc}$ production cross section has been calculated in the framework
of non-relativistic QCD as $\sigma(\Xi_{cc}^{++})=\sigma(\Xi_{cc}^{+})\approx
62$ nb [45], with the experimental cuts $p_{\text{T}}>$ 4 GeV and $|\eta|\leq
1.5$. To compare with this result, we reset the same cuts and find
$\sigma(\Xi_{cc}^{++})=\sigma(\Xi_{cc}^{+})\approx 100$ nb. Taking into
account the large uncertainties, they agree with each other. In addition, the
$\Xi_{cc}^{++}$ production has been studied by the LHCb, with a relative
production rate given as [46]

$\displaystyle{\sigma(\Xi_{cc}^{++})\times\mathcal{B}(\Xi_{cc}^{++}\to\Lambda_{c}^{+}K^{-}\pi^{+}\pi^{+})\over\sigma(\Lambda_{c}^{+})}=(2.22\pm
0.27\pm 0.29)\times 10^{-4}\;,$ (8)

choosing the cuts $4<p_{\text{T}}<15$ GeV and $2<\eta<4.5$. From an LHCb
measurement with $\sqrt{s}=7$ TeV [47], it can be extracted that
$\sigma(\Lambda_{c}^{+}+\Lambda_{c}^{-})\approx 38\mu b$ with
$4<p_{\text{T}}<8\ \text{GeV},2.0<\eta<4.5$ (the $p_{\text{T}}>8$ GeV
contributions are negligible). By comparing the $D$ meson production at 13 TeV
[48] and 7 TeV [47], one expects that the $\Lambda_{c}$ production rate with
$\sqrt{s}=13$ TeV should exhibit an enhancement of approximately 50%, i.e.,
$\sigma(\Lambda_{c}^{+}+\Lambda_{c}^{-})\approx 57\mu b$. Assuming
$\mathcal{B}(\Xi_{cc}^{++}\to\Lambda_{c}^{+}K^{-}\pi^{+}\pi^{+})\in[5,20]\%$
as suggested in [49], one can solve from (8) that
$\sigma(\Xi_{cc}^{++})\in[30,130]$ nb, which is also consistent with our
result given by (2).

The LHCb collaboration has collected approximately 9 fb-1 of data up to now,
which indicates ${\cal O}(10^{8})$ $T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$
particles. If reconstructed by proper decay channels, as we analyze in the
next section, they have a good chance to be discovered in the near future.

$Z$-factories: For future $Z$-factories which are designed to produce
$10^{12}$ or more $Z$ bosons, we also generate $10^{5}$ $e^{+}e^{-}\to Z\to
cc\bar{c}\bar{c}$ events via MadGraph5 [39] and Pythia8 [40, 41]. We find that
the total cross section $\sigma(e^{+}e^{-}\to cc\bar{c}\bar{c})\approx 120$
pb, and after the diquark jet condition $m_{cc}<M_{cc}(\Delta M)$ is applied,
the decay branching ratio of the $Z$ boson into double-charm hadrons is found
to be

$\displaystyle\mathcal{B}(Z\to H_{cc}+X)=(10.1^{+7.9}_{-2.7})\times
10^{-5}\;.$ (9)

Analogous to the analysis of the LHCb, we assume that the fragmentation
fraction $f((cc)_{\text{jet}}\to
T^{\\{cc\\}}_{[\bar{u}\bar{d}]}+X)/f((cc)_{\text{jet}}\to\Xi_{cc}+X)$ is close
to two times $f_{\Lambda_{b}}/(f_{B_{u}}+f_{B_{d}})=0.11\pm 0.02$ [50],
$f_{s}/f_{d}\approx 0.25$, and the ground-state fraction is still
$r_{g}=0.48\pm 0.08$ [44]. With these approximations, the branching ratios of
the $Z$ boson decays into double-charm tetraquarks are estimated to be

$\displaystyle\mathcal{B}(Z\to T^{\\{cc\\}}_{[\bar{u}\bar{d}]}+X)=$
$\displaystyle\ (4.1^{+3.4}_{-1.5})\times 10^{-6}\;,$
$\displaystyle\mathcal{B}(Z\to
T^{\\{cc\\}}_{[\bar{u}\bar{s}]}+X)=\mathcal{B}(Z\to
T^{\\{cc\\}}_{[\bar{d}\bar{s}]}+X)=$ $\displaystyle\ (1.0^{+0.8}_{-0.4})\times
10^{-6}\;.$ (10)

We also give the $Z$ decay branching ratios to the ground-state double-charm
baryons, including those secondarily decaying from their excited states,

$\displaystyle\mathcal{B}(Z\to\Xi_{cc}^{++}+X)=\mathcal{B}(Z\to\Xi_{cc}^{+}+X)=$
$\displaystyle\ (3.9^{+3.1}_{-1.1})\times 10^{-5}\;,$
$\displaystyle\mathcal{B}(Z\to\Omega_{cc}^{+}+X)=$ $\displaystyle\
(1.0^{+0.8}_{-0.3})\times 10^{-5}\;.$ (11)

It can be seen that our estimation for the $\Xi_{cc}$ production rate at
$Z$-factories is consistent with the NRQCD calculation
$\mathcal{B}(Z\to\Xi_{cc}+X)\approx(1.7^{+1.3}_{-0.6})\times 10^{-5}$ [51].
Therefore, it is expected that future Tera-$Z$ factories will produce
$\mathcal{O}(10^{6})$ $T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$ particles.
Owing to the clean background of electron-positron collisions, future
$Z$-factories have the advantage of measuring some of the decay channels of
$T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$ containing neutral particles such
as photons in their final states, while it is difficult for hadron colliders.
More details are given in the next section.

##  3 Decay channels

In addition to the production rates, the decaying processes are another
important issue in the experimental searches for the double-charm tetraquarks.
An example of the importance of decaying processes is the discovery of the
first doubly charmed baryon. With the prediction of the most favorable decay
channels of $\Xi_{cc}^{++}\to\Lambda_{c}^{+}K^{-}\pi^{+}\pi^{+}$ and
$\Xi_{c}^{+}\pi^{+}$ [49], the LHCb collaboration observed $\Xi_{cc}^{++}$ via
the above two modes [1, 52].

The decaying properties depend on the masses of
$T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$. Above the $DD^{*}$ thresholds,
they will decay strongly. In contrast, between the $DD^{*}$ and $DD\gamma$
thresholds, they will decay radiatively; below the $DD\gamma$ thresholds, they
will decay weakly. Since the theoretical predictions on the masses are very
different from each other, as shown in Table 1, we will discuss the decay
processes separately according to the above cases.

In the case that the masses of double-charm tetraquarks are above the
thresholds of $DD^{*}$, they are unstable and will decay strongly. The
decaying processes include $T^{\\{cc\\}}_{[\bar{u}\bar{d}]}\to
D^{0}D^{*+}~{}\text{or}~{}D^{+}D^{*0}$, $T^{\\{cc\\}}_{[\bar{u}\bar{s}]}\to
D^{0}D_{s}^{*+}~{}\text{or}~{}D_{s}^{+}D^{*0}$, and
$T^{\\{cc\\}}_{[\bar{d}\bar{s}]}\to
D^{+}D_{s}^{*+}~{}\text{or}~{}D_{s}^{+}D^{*+}$. The reconstructions of
$D^{*0}$ and $D_{s}^{*+}$ require the reconstructions of neutral particles
$\pi^{0}$ or $\gamma$, which will reduce the detection efficiencies
significantly. Then, the favorable decay processes of the LHCb are

$\displaystyle T^{\\{cc\\}}_{[\bar{u}\bar{d}]}\to D^{0}D^{*+}\to
D^{0}D^{0}\pi^{+},~{}~{}~{}\text{and}~{}~{}~{}T^{\\{cc\\}}_{[\bar{d}\bar{s}]}\to
D_{s}^{+}D^{*+}\to D_{s}^{+}D^{0}\pi^{+}.$ (12)

If their masses are above the $DD\gamma$ thresholds but below the $DD^{*}$
thresholds, they will decay radiatively, such as
$T^{\\{cc\\}}_{[\bar{u}\bar{d}]}\to D^{0}D^{+}\gamma$,
$T^{\\{cc\\}}_{[\bar{u}\bar{s}]}\to D^{0}D_{s}^{+}\gamma$, and
$T^{\\{cc\\}}_{[\bar{d}\bar{s}]}\to D^{+}D_{s}^{+}\gamma$. These channels are
expected to be better measured at future $Z$-factories.

In the case of $T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$ below the $DD\gamma$
thresholds, they can only decay weakly and are thus stable particles. This is
different from the exotic states discovered in experiments so far, which decay
strongly. The scales of heavy quark weak decays, at $\mathcal{O}$(GeV), are
different from the ones of strong decays at the scale of hundreds of MeV.
Thus, it will provide another way to distinguish the nature of hadrons to
study the weak decays of such particles. Since they can only weakly decay,
they must be long-lived particles and cannot be kinematical effects or
coupled-channel effects. Due to large non-perturbative contributions, it is
difficult to precisely predict the branching fractions of weak decays of
charmed hadrons. Considering the Cabibbo-favored processes with charged
particles in the final states, the most favorable processes of weak decays at
the LHCb are

$\displaystyle T^{\\{cc\\}}_{[\bar{u}\bar{d}]}$ $\displaystyle\to
D^{+}K^{-}\pi^{+},$ $\displaystyle T^{\\{cc\\}}_{[\bar{u}\bar{s}]}$
$\displaystyle\to D_{s}^{+}K^{-}\pi^{+},\quad\text{or}\quad
D^{0}\pi^{+},\quad\text{or}\quad D^{+}K^{+}K^{-},$ (13) $\displaystyle
T^{\\{cc\\}}_{[\bar{d}\bar{s}]}$ $\displaystyle\to
D^{+}K^{+}K^{-}\pi^{+},\quad\text{or}\quad D^{+}\pi^{+}.$

Considering a more abundant production of $T^{\\{cc\\}}_{[\bar{u}\bar{d}]}$
compared to the ones containg a strange quark, we will discuss the possible
signal events of $T^{\\{cc\\}}_{[\bar{u}\bar{d}]}$ in experiments. If it
decays strongly, it decays into $D^{0}D^{*+}$ with $D^{*+}\to D^{0}\pi^{+}$,
$D^{0}\to K^{-}\pi^{+}$, and $K^{-}\pi^{+}\pi^{+}\pi^{-}$. The total branching
fraction would be $\mathcal{O}(10^{-2})$. In case it is a stable particle, the
branching fraction of $T^{\\{cc\\}}_{[\bar{u}\bar{d}]}\to D^{+}K^{-}\pi^{+}$
would be of the order of $10\%$. Considering the further decay $D^{+}\to
K^{-}\pi^{+}\pi^{+}$, the total branching fraction of the weak decay of
$T^{\\{cc\\}}_{[\bar{u}\bar{d}]}$ would be $\mathcal{O}(10^{-2})$ as well. The
order of the branching fractions of $T^{\\{cc\\}}_{[\bar{u}\bar{d}]}$ decays
is thus the same as the observed $\Xi_{cc}^{++}$. Comparing with the
production rates between double-charm tetraquarks and baryons and considering
approximately $2\times 10^{3}$ events of $\Xi_{cc}^{++}$ with the current LHCb
data, the signal yields of $T^{\\{cc\\}}_{[\bar{u}\bar{d}]}$ would be
$\mathcal{O}(10^{2})$ at LHCb, either for a strongly or weakly decaying
$T^{\\{cc\\}}_{[\bar{u}\bar{d}]}$. This number will reach
$\mathcal{O}(10^{3})$ at LHCb Run III. Therefore, it is expected that the
double-charm tetraquark will be observed at the LHCb in the near future.
Although the production rates are smaller at the future Z factories, it is
also expected to be observed at the Tera-Z factories due to smaller
backgrounds.

##  4 Conclusion

To search for double-charm tetraquarks
$T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$, we study their productions and
decay channels at the LHCb and also future $Z$-factories. It is found that the
LHCb has already collected plenty of events for the discovery of
$T^{\\{cc\\}}_{[\bar{u}\bar{d}]}$. In case
$T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$ are above the $DD^{*}$ thresholds,
we propose that the decay channel $T^{\\{cc\\}}_{[\bar{u}\bar{d}]}\to
D^{0}D^{*+}\to D^{0}D^{0}\pi^{+}$ should be analyzed at the LHCb. In contrast,
in case that $T^{\\{cc\\}}_{[\bar{q}\bar{q}^{\prime}]}$ are below the
$DD\gamma$ thresholds, the decay channel $T^{\\{cc\\}}_{[\bar{u}\bar{d}]}\to
D^{+}K^{-}\pi^{+}$ should be analyzed at the LHCb. Although these double-charm
tetraquarks will probably be discovered at the LHCb, there are still some
decay channels containing neutral particles in their final states, e.g.,
$D^{0}D^{+}\gamma$, which are better to be studied at future $Z$-factories.

## Acknowledgement

The authors are grateful to Ji-Bo He and Xiao-Rui Lyu for useful discussions.
This work is supported by Natural Science Foundation of China under grant No.
12005068 and 11975112.

## Additional note

After the first version of this paper appeared on arxiv.org, with e-Print No.
2008.08026 [hep-ph], and during the preparation of its submission to Chinese
Physics C, the observation of the first double-charm tetraquark, a
$T^{\\{cc\\}}_{[\bar{u}\bar{d}]}$-like particle, was reported in the talks
”Highlights from the LHCb Experiment” given by Franz Muheim and ”Recent LHCb
results on exotic meson candidates” given by Ivan Polyakov at the European
Physical Society Conference on high energy physics 2021. For details, see
https://indico.desy.de/event/28202/contributions/102717/ and
https://indico.desy.de/event/28202/contributions/105627/. Subsequently, the
corresponding papers were released on arxiv [53, 54]. It turns out that the
observed double-charm tetraquark is an unstable particle dominated by strong
decays, and indeed, it was discovered through the channel
$T^{\\{cc\\}}_{[\bar{u}\bar{d}]}\to D^{0}D^{*+}\to D^{0}D^{0}\pi^{+}$, as we
suggested in this paper. Moreover, the signal yield at the LHCb was reported
to be $N=117\pm 16$, which is very consistent with our prediction,
$\mathcal{O}(10^{2})$, using the current LHCb data.

## References

  * [1] R. Aaij et al. [LHCb], Phys. Rev. Lett. 119, no.11, 112001 (2017) [arXiv:1707.01621 [hep-ex]].
  * [2] M. Karliner and J. L. Rosner, Phys. Rev. Lett. 119, no.20, 202001 (2017) [arXiv:1707.07666 [hep-ph]].
  * [3] E. J. Eichten and C. Quigg, Phys. Rev. Lett. 119, no.20, 202002 (2017) [arXiv:1707.09575 [hep-ph]].
  * [4] A. Francis, R. J. Hudspith, R. Lewis and K. Maltman, Phys. Rev. Lett. 118, no.14, 142001 (2017) [arXiv:1607.05214 [hep-lat]].
  * [5] P. Bicudo, M. Cardoso, A. Peters, M. Pflaumer and M. Wagner, Phys. Rev. D 96, no.5, 054510 (2017) [arXiv:1704.02383 [hep-lat]].
  * [6] P. Junnarkar, M. Padmanath and N. Mathur, EPJ Web Conf. 175, 05014 (2018) [arXiv:1712.08400 [hep-lat]].
  * [7] T. Mehen, Phys. Rev. D 96, no.9, 094028 (2017) [arXiv:1708.05020 [hep-ph]].
  * [8] A. Czarnecki, B. Leng and M. B. Voloshin, Phys. Lett. B 778, 233-238 (2018) [arXiv:1708.04594 [hep-ph]].
  * [9] J. B. Guimarães da Costa et al. [CEPC Study Group], [arXiv:1811.10545 [hep-ex]].
  * [10] M. Bicer et al. [TLEP Design Study Working Group], JHEP 01, 164 (2014) [arXiv:1308.6176 [hep-ex]].
  * [11] G. Q. Feng, X. H. Guo and B. S. Zou, [arXiv:1309.7813 [hep-ph]].
  * [12] R. Aaij et al. [LHCb], JHEP 07, 035 (2019) [arXiv:1903.12240 [hep-ex]].
  * [13] M. L. Du, W. Chen, X. L. Chen and S. L. Zhu, Phys. Rev. D 87, no.1, 014003 (2013) [arXiv:1209.5134 [hep-ph]].
  * [14] F. S. Navarra, M. Nielsen and S. H. Lee, Phys. Lett. B 649, 166-172 (2007) [arXiv:hep-ph/0703071 [hep-ph]].
  * [15] Z. G. Wang and Z. H. Yan, Eur. Phys. J. C 78, no.1, 19 (2018) [arXiv:1710.02810 [hep-ph]].
  * [16] S. S. Agaev, K. Azizi, B. Barsbay and H. Sundu, Nucl. Phys. B 939, 130-144 (2019) [arXiv:1806.04447 [hep-ph]].
  * [17] S. S. Agaev, K. Azizi and H. Sundu, Phys. Rev. D 99, no.11, 114016 (2019) [arXiv:1903.11975 [hep-ph]].
  * [18] A. Francis, R. J. Hudspith, R. Lewis and K. Maltman, Phys. Rev. D 99, no.5, 054505 (2019) [arXiv:1810.10550 [hep-lat]].
  * [19] E. Braaten, L. P. He and A. Mohapatra, Phys. Rev. D 103, no.1, 016001 (2021) [arXiv:2006.08650 [hep-ph]].
  * [20] J. Carlson, L. Heller and J. A. Tjon, Phys. Rev. D 37, 744 (1988).
  * [21] S. H. Lee and S. Yasui, Eur. Phys. J. C 64, 283-295 (2009) [arXiv:0901.2977 [hep-ph]].
  * [22] S. Q. Luo, K. Chen, X. Liu, Y. R. Liu and S. L. Zhu, Eur. Phys. J. C 77, no.10, 709 (2017) [arXiv:1707.01180 [hep-ph]].
  * [23] J. B. Cheng, S. Y. Li, Y. R. Liu, Z. G. Si and T. Yao, Chin. Phys. C 45, no.4, 043102 (2021) [arXiv:2008.00737 [hep-ph]].
  * [24] C. Deng, H. Chen and J. Ping, Eur. Phys. J. A 56, no.1, 9 (2020) [arXiv:1811.06462 [hep-ph]].
  * [25] Q. F. Lü, D. Y. Chen and Y. B. Dong, Phys. Rev. D 102, no.3, 034012 (2020) [arXiv:2006.08087 [hep-ph]].
  * [26] D. Ebert, R. N. Faustov, V. O. Galkin and W. Lucha, Phys. Rev. D 76, 114015 (2007) [arXiv:0706.3853 [hep-ph]].
  * [27] S. Pepin, F. Stancu, M. Genovese and J. M. Richard, Phys. Lett. B 393, 119-123 (1997) [arXiv:hep-ph/9609348 [hep-ph]].
  * [28] G. Yang, J. Ping and J. Segovia, Phys. Rev. D 101, no.1, 014001 (2020) [arXiv:1911.00215 [hep-ph]].
  * [29] Y. Tan, W. Lu and J. Ping, Eur. Phys. J. Plus 135, no.9, 716 (2020) [arXiv:2004.02106 [hep-ph]].
  * [30] W. Park, S. Noh and S. H. Lee, Acta Phys. Polon. B 50, 1151-1157 (2019) [arXiv:1809.05257 [nucl-th]].
  * [31] M. Karliner and J. L. Rosner, Phys. Rev. D 102, no.9, 094016 (2020) [arXiv:2008.05993 [hep-ph]].
  * [32] R. Zhu, X. Liu, H. Huang and C. F. Qiao, Phys. Lett. B 797, 134869 (2019) [arXiv:1904.10285 [hep-ph]].
  * [33] R. Aaij et al. [LHCb], Eur. Phys. J. C 73, no.4, 2373 (2013) [arXiv:1208.3355 [hep-ex]].
  * [34] I. Bediaga et al. [LHCb], CERN-LHCC-2012-007.
  * [35] A. Ali, A. Y. Parkhomenko, Q. Qin and W. Wang, Phys. Lett. B 782, 412-420 (2018) [arXiv:1805.02535 [hep-ph]].
  * [36] A. Ali, A. Y. Parkhomenko, Q. Qin and W. Wang, Phys. Part. Nucl. Lett. 16, no.5, 481-485 (2019).
  * [37] A. Ali, Q. Qin and W. Wang, Phys. Lett. B 785, 605-609 (2018) [arXiv:1806.09288 [hep-ph]].
  * [38] R. Aaij et al. [LHCb], Phys. Rev. Lett. 114, 132001 (2015) [arXiv:1411.2943 [hep-ex]].
  * [39] J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H. S. Shao, T. Stelzer, P. Torrielli and M. Zaro, JHEP 07, 079 (2014) [arXiv:1405.0301 [hep-ph]].
  * [40] T. Sjostrand, S. Mrenna and P. Z. Skands, JHEP 05, 026 (2006) [arXiv:hep-ph/0603175 [hep-ph]].
  * [41] T. Sjöstrand, S. Ask, J. R. Christiansen, R. Corke, N. Desai, P. Ilten, S. Mrenna, S. Prestel, C. O. Rasmussen and P. Z. Skands, Comput. Phys. Commun. 191, 159-177 (2015) [arXiv:1410.3012 [hep-ph]].
  * [42] R. Aaij et al. [LHCb], JHEP 08, 143 (2014) [arXiv:1405.6842 [hep-ex]].
  * [43] R. Aaij et al. [LHCb], Phys. Rev. D 104, no.3, 032005 (2021) [arXiv:2103.06810 [hep-ex]].
  * [44] M. Niiyama et al. [Belle], Phys. Rev. D 97, no.7, 072005 (2018) [arXiv:1706.06791 [hep-ex]].
  * [45] C. H. Chang, C. F. Qiao, J. X. Wang and X. G. Wu, Phys. Rev. D 73, 094022 (2006) [arXiv:hep-ph/0601032 [hep-ph]].
  * [46] R. Aaij et al. [LHCb], Chin. Phys. C 44, no.2, 022001 (2020) [arXiv:1910.11316 [hep-ex]].
  * [47] R. Aaij et al. [LHCb], Nucl. Phys. B 871, 1-20 (2013) [arXiv:1302.2864 [hep-ex]].
  * [48] R. Aaij et al. [LHCb], JHEP 03, 159 (2016) [arXiv:1510.01707 [hep-ex]].
  * [49] F. S. Yu, H. Y. Jiang, R. H. Li, C. D. Lü, W. Wang and Z. X. Zhao, Chin. Phys. C 42, no.5, 051001 (2018) [arXiv:1703.09086 [hep-ph]].
  * [50] M. Tanabashi et al. [Particle Data Group], Phys. Rev. D 98, no.3, 030001 (2018).
  * [51] J. Jiang, X. G. Wu, Q. L. Liao, X. C. Zheng and Z. Y. Fang, Phys. Rev. D 86, 054021 (2012) [arXiv:1208.3051 [hep-ph]].
  * [52] R. Aaij et al. [LHCb], Phys. Rev. Lett. 121, no.16, 162002 (2018) [arXiv:1807.01919 [hep-ex]].
  * [53] R. Aaij et al. [LHCb], [arXiv:2109.01038 [hep-ex]].
  * [54] R. Aaij et al. [LHCb], [arXiv:2109.01056 [hep-ex]].


# Neural-based classification rule learning for sequential data

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Marine Collery1,2 , Philippe
Bonnard1, François Fages2 & Remy Kusters1,3
1IBM France Lab, 2Inria Saclay, 3IBM Research
Corresponding author<EMAIL_ADDRESS>
###### Abstract

Discovering interpretable patterns for classification of sequential data is of
key importance for a variety of fields, ranging from genomics to fraud
detection or more generally interpretable decision-making. In this paper, we
propose a novel differentiable fully interpretable method to discover both
local and global patterns (i.e. catching a relative or absolute temporal
dependency) for rule-based binary classification. It consists of a
convolutional binary neural network with an interpretable neural filter and a
training strategy based on dynamically-enforced sparsity. We demonstrate the
validity and usefulness of the approach on synthetic datasets and on an open-
source peptides dataset. Key to this end-to-end differentiable method is that
the expressive patterns used in the rules are learned alongside the rules
themselves.

##  1 Introduction

During the last decades, machine learning and in particular neural networks
have made tremendous progress on classification tasks for a variety of fields
such as healthcare, fraud detection or entertainment. They are able to learn
from various data types ranging from images to timeseries and achieve
impressive classification accuracy. However, they are difficult or impossible
to understand by a human. Recently, explaining those black-box models has
attracted considerable research interest under the field of Explainable AI
(XAI). However, as stated by Rudin (2019), those aposteriori approaches are
not the solution for high stakes decision-making and more interest should be
placed on learning models that are interpretable in the first place.

Rule-based methods are interpretable, human-readable and have been widely
adopted in different industrial fields with Business Rule Management Systems
(BRMS). In practice however, those rules are manually written by experts. One
of the reasons manually-written rule models cannot easily be replaced with
learned rule models is that rule-base learning models are not able to learn as
expressive rules with higher-level concepts and complex grammar (Kramer,
2020). Moreover, due to the lack of latent representations, rule-based
learning methods underperform w.r.t. state-of-the-art neural networks (Beck &
Fürnkranz, 2021).

Classical classification rule learning algorithms (Cohen, 1995; Breiman et
al., 1984; Dash et al., 2018; Lakkaraju et al., 2016; Su et al., 2016) as well
as neural-based approaches to learn rules (Qiao et al., 2021; Kusters et al.,
2022) (or logical expressions with Riegel et al. (2020)) do not provide the
grammar required to learn classification rules on sequential data. Numerous
approaches for learning classification rules on sequential data in the field
of sequential pattern mining have been studied in the past such as Egho et al.
(2015); Zhou et al. (2013); Holat et al. (2014) but with a different goal in
mind : improve the performance of extracted patterns for a fixed rule grammar
as opposed to extending the rule grammar. Another domain of research focuses
on training binary neural networks to obtain more computational efficient
model storing, computation and evaluation efficiency (Geiger & Team, 2020;
Helwegen et al., 2019). It comes with fundamental optimization challenges
around weights updates and gradient computation.

In this paper, we bridge three domains and introduce a binary neural network
to learn classification rules on sequential data. We propose a differentiable
rule-based classification model for sequential data where the conditions are
composed of sequence-dependent patterns that are discovered _alongside_ the
classification task itself. More precisely, we aim at learning a rule of the
following structure: $\textit{if}\textrm{ pattern }\textit{then class
=}\textrm{ }1\textrm{ {else class} = }0$. In particular we consider two types
of patterns: local and global patterns as introduced in Aggarwal (2002) that
are in practice studied independently with a local and a global model. A local
pattern describes a subsequence at a specific position in the sequence while a
global pattern is invariant to the location in the sequence (Fig 2). The
network, that we refer to as Convolutional Rule Neural Network (CR2N), builds
on top of a base rule model that is comparable to rule models for tabular data
presented in Qiao et al. (2021); Kusters et al. (2022).

The contributions of this paper are the following: i) We propose a
convolutional binary neural network that learns classification rules together
with the sequence-dependent patterns in use. ii) We present a training
strategy to train a binarized neural network while dynamically enforcing
sparsity. iii) We show on synthetic and real world datasets the usefulness of
our architecture with the importance of the rule grammar and the validity of
our training process with the importance of sparsity.

##  2 Base Rule Model

The base rule model we invoke is composed of three consecutive layers (Fig 1).
The two last layers respectively mimic logical AND and OR operators (Qiao et
al., 2021; Kusters et al., 2022). On top of these layers, we add an additional
layer that is specific for categorical input data and corresponds to an OR
operator for each categorical variable over every possible value it can take.

$x_{1}$$x_{2}$$x_{3}$$x_{1}$$x_{2}$$h_{1}$$h_{2}$$h_{3}$$h_{4}$$h_{5}$$h_{2}$$y$StackedOR
layerAND layerOR
layer$\left(\sum_{k}n_{k},K\right)$$(K,H)$$(H,1)$$x_{c_{1}}=A_{1}$$\dots$$B_{1}$$C_{1}$$D_{1}$$x_{c_{2}}=A_{2}$$\dots$$B_{2}$$C_{2}$$D_{2}$$x_{c_{3}}=A_{3}$$\dots$$B_{3}$$C_{3}$$D_{3}$$B_{1}$$D_{2}$$C_{3}$$B_{1}$
and $D_{2}$$D_{2}$ and $C_{3}$ if ($B_{1}$ and $D_{2}$) or ($D_{2}$ and
$C_{3}$) then 1 else 0 Figure 1: Example of a trained base rule model
architecture for the rule if ($B_{1}$ and $D_{2}$) or ($D_{2}$ and $C_{3}$)
then 1 else 0 on 3 categorical variables $x_{c_{1}}$, $x_{c_{2}}$ and
$x_{c_{3}}$ ($x_{c_{k}}\in\\{A_{k},B_{k},C_{k},D_{k}\\}$). For simplicity, the
truth value of $x_{c_{1}}=B_{1}$ is replaced by $B_{1}$ for example. Plain
(dotted) lines represent activated (masked) weights. An example evaluation of
the model is represented with the filled neurons (neuron=1) for the binary
input $x_{c_{1}}=B_{1}$, $x_{c_{2}}=D_{2}$ and $x_{c_{3}}=A_{3}$.

The AND layer takes binary features (which are atomic boolean formulae) as
input and outputs to the OR layer. The output of the OR layer is mapped to the
classification label $y$. These layers have binary weights specifying the
nodes that are included in the respective boolean expression (conjunction or
disjunction). In other words, this network implements the evaluation of a DNF
and has a direct equivalence with a binary classification rule like
$\textit{if }(A\land B)\lor C\textit{ then class =}\textrm{ }1\textrm{ {else
class} = }0$, where $A,B$ and $C$ are binary input features (atoms in logical
terms). In this paper, we focus on supervised binary classification where we
predict the label $y\in\\{0,1\\}$ given input data ${\bm{x}}$.

The base rule model is illustrated in Fig 1 and is composed of three binary
neural layers.

  * •

Input neurons ${\bm{x}}$, are binarized input features of size $K$
(${\bm{x}}_{c}$ are one-hot encoded categorical input features of size
${\bm{n}}$).

  * •

Hidden neurons ${\bm{h}}$, are conjunctions of the input features of size $H$.

  * •

Output neuron ${\bm{y}}$, is a disjunction of the (hidden) conjunctions.

We assign to each of boolean operations, i.e. AND and OR operations, a binary
weight (${\bm{W}}_{and}$ and ${\bm{W}}_{or}$ respectively) that plays the role
of a mask to filter nodes with regards to their respective logical operation.
For the sake of simplicity, we did not extend the model with a logical NOT
operation.

The disjunction operation is implemented as,

$y=\min({\bm{W}}_{or}{\bm{h}},1).$ (1)

If none of the neurons $h$ are activated then $y=0$, and $y=1$ if at least one
is.

For the conjunction operation, we use the De Morgan’s law that express the
conjunction with the OR operator $A\land B=\neg(\neg A\lor\neg B)$. Combined
with Eq 1, we obtain:

${\bm{h}}=\neg(\min({\bm{W}}_{and}(\neg{\bm{x}}),\mathbf{1}))=\mathbf{1}-\min({\bm{W}}_{and}(\mathbf{1}-{\bm{x}}),\mathbf{1}).$
(2)

#### StackedOR Input Layer

As defined previously, the AND layer takes binary input features as input. In
this paper, we propose to add an additional layer for categorical data. A
categorical variable ${\bm{x}}_{c}$ can take one value $\alpha_{i}^{c}$ out of
a fixed number of possible values $n$ e.g.
$\\{\alpha_{0},\dots,\alpha_{3}\\}=\\{A,B,C,D\\}$. Without any additional
layer, it requires a one-hot encoding to be provided as input to the AND
layer. Binary inputs $x_{c}=A$ and $x_{c}=B$ are then given as input to the
AND layer that can in theory represent the impossible expression $x_{c}=A\land
x_{c}=B$ i.e the model has to learn the hidden categorical relationship
between the one-hot encoded variables. To prevent learning a distribution we
already know, we deepen the model with the addition of a stacked architecture
of OR layers as input of the AND layer as shown in Fig 1. This structure is
defined by $K$ weights, ${\bm{W}}_{stack}^{k}$, for each input categorical
variables and will be referred to as the StackedOR layer with
${\bm{W}}_{stack}$ weights.

To conclude, the base rule model is composed of a StackedOR layer for
categorical variables, a logical AND layer and an OR layer. The formal grammar
that this architecture can express is specified with the following production
rules (see Appendix A for the full grammar):

$\begin{array}[]{rll}\text{rule}&\rightarrow&\texttt{\text@underline{if}}\text{
base expression}\texttt{\text@underline{then class =}}\text{ }1\text{
}\texttt{\text@underline{else class =}}\text{ }0\\\ \text{base
expression}&\rightarrow&\text{conjunction $|$ conjunction $\lor$ base
expression }\\\ \text{conjunction}&\rightarrow&\text{predicate $|$ predicate
$\land$ conjunction}\\\ \textrm{predicate}&\rightarrow&\text{categorical
expression $|$ literal}\\\ \textrm{categorical
expression}&\rightarrow&\text{categorical literal $|$ categorical literal
$\lor$ categorical expression }\\\ \textrm{categorical
literal}&\rightarrow&\text{\text{\text@underline{$x_{c}=\alpha^{c}_{0}$}} $|$
\ldots$|$ \text{\text@underline{$x_{c}=\alpha^{c}_{n_{c}}$}} (or simply
\text{\text@underline{$\alpha^{c}_{0}$}} $|$ \ldots$|$
\text{\text@underline{$\alpha^{c}_{n_{c}}$}})}\\\
\textrm{literal}&\rightarrow&\text{\text{\text@underline{$x_{1}$}} $|$
\text{\text@underline{$x_{2}$}} $|$ \ldots}\end{array}$ (3)

This grammar is also limited by the model architecture: conjunction contains
at most one occurrence of each predicate and the total number of
conjunction(s) is bounded by the number of hidden nodes.

##  3 Convolutional Rule Neural Network

Our main contribution is to extend the base rule model for sequential data. We
apply the base rule model as a 1D-convolutional window of fixed length
$l\in\mathbb{N}$ over a sequence and retrieve all outputs as input for an
additional disjunctive layer which we refer to as the ConvOR layer as shown in
Fig 2 111A natural extension for sequential data of the base rule architecture
would be to extend it with an explicit recursion of the base rule model,
similar to a RNN. This approach was tested but faced the same limitations as
any classical RNNs, i.e., vanishing gradients and only captures short-term
dependencies.. The base rule model learns a DNF over the window size length
and the ConvOR layer indicates where along the sequence that logical
expression is true. If the evaluation of the logical expression is true all
along the sequence then it can be described as a _global pattern_ , otherwise
the learned pattern represents a _local pattern_.

The model input is now of size $\sum_{k}l\times n_{k}$ and output of StackedOR
layer (or input of the AND layer) is $l\times K$. Other dimensions are not
impacted. For simplicity in the following, $K$ is fixed to $1$ i.e. input data
is composed of one categorical variable evolving sequentially. The method is
still valid for $K>1$. Fig 1 is also still valid with a change of index, $k$
is now referring to the position in the window of size $l$.

$t$ABDACBD y$local$$global$ Evaluation of base rule model on every window
ConvOR layer Filter: Base Rule Model ($B$ at t-2 and $D$ at t-1) or ($D$ at
t-1 and $C$ at t-0) ($B$ at t-5 and $D$ at t-4) or ($D$ at t-4 and $C$ at t-3)
$B$-$D$ or $D$-$C$ in sequence Figure 2: Example of a trained CR2N
architecture. The base rule model is applied as 1D-convolutional window over
the sequence (i.e. sliding window). The resulting boolean values are given as
input of the ConvOR layer which indicates through its activated weights where
along the sequence the expression learned by the base model is true. The
output of the ConvOR layer is mapped to the label of the sequence $y$. For
local patterns, the base model expression needs to be shifted accordingly to
the ConvOR layer weights. For a real-domain application like fraud detection,
by providing meaning to B, C and D, we could have for example if “receiving a
transaction of amount X”($B$) is followed by “emitting a transaction of amount
X” ($D$) or “emitting a transaction of amount X”($D$) is followed by “closing
the bank account”($C$) then class=fraud.

With this approach, different sequence-dependent expressions can be extracted
and their nature depends on the weights of the ConvOR layer (Fig 2). If all
the weights, ${\bm{W}}_{conv}$, of the ConvOR layer are activated (i.e. equal
to 1), the logical expression learned by the base model is true in all the
sequence: a _global_ pattern is learned. If only some of the weight of the
ConvOR layer are activated, the logical expression learned by the base model
is valid only in the window associated to that weight: a _local_ pattern is
learned. The base model logical expression is modified accordingly to match
that shift (see example in Fig 2 with a shift of 3 sequential steps).

The obtained weights thus translate to a rule grammar with the following
production rules:

$\begin{array}[]{lll}\textrm{rule}&\rightarrow&\textit{if}\textrm{ expression
}\textit{then class =}\textrm{ }1\textrm{ {else class} = }0\\\
\textrm{expression}&\rightarrow&\textrm{local pattern $|$ global pattern}\\\
\end{array}$ (4)

We introduce $t$, the position when the last observation in a sequence was
made. With $t$ being our reference, in a sequence of size $N\in\mathbb{N}$,
$t-i$ refers to the moment of the $i^{\textrm{th}}$ observation before $t$
($0\leq i\leq N-1$). $A,B,C$ and $D$ are toy binary input possible values for
our categorical variable $x_{c}$ (they cannot be activated simultaneously at
the same position $t$ in the sequence). With those definitions, we list below
examples of different sequence-dependent expressions that can be expressed
with the proposed architecture (see Fig 2):

A local pattern is an expression composed of predicates that are true at a
specific position $i$, for example $A$ at t-15. Based on Eq 3 we have:

$\begin{array}[]{lll}\textrm{local pattern}&\rightarrow&\textrm{base
expression}\\\ \textrm{predicate}&\rightarrow&\textrm{categorical expression
}\textit{at t-}i\textrm{ }|\textrm{ }\textrm{literal }\textit{at t-}i.\\\
\end{array}$ (5)

A global pattern is an expression describing the presence of a pattern
anywhere in the sequence, for example $B$-$D$ in sequence is a global pattern
where “$-$” sign refers to “followed by” and “$*$” correspond to any unique
literal (equivalent to $\forall i\in[0;N-1$], $B$ at t-i-1 and $D$ at t-i) .
If inputs are sequences of characters, global patterns can be compared to
simple regular expressions supporting the logical OR (metacharacter‘[ ]’).
Based on Eq 3 we have:

$\begin{array}[]{lll}\textrm{global pattern}&\rightarrow&\textrm{base
expression}\\\ \text{conjunction}&\rightarrow&\text{predicate $|$ * $|$
predicate $-$ conjunction}\\\ \end{array}$ (6)

Additional special cases can be pointed out such as the learning of a global
pattern over an interval (e.g. $B$-*-$D$ in window [t-6; t-3]) or the learning
of sequence characteristics dependant expression such as 4 $\leq$
len(sequence) $\leq$ 6 based on the sequence length (not shown on Fig 2 but it
corresponds to a specific case where the base model has learned an always true
rule). Also, it is important to note that base expression and conjunction in
both grammars are bounded by the fixed window size $l$.

To ensure full equivalence between the model and rule, sequences boundaries
need to be considered, especially for global patterns. All sequences are
padded on both ends with a sequence of 0 of size $l-1$ (not shown for
simplicity on Fig 2). Also sequences of different lengths are supported by
creating a model based on the maximal available sequence length $M$ in the
data and padding shorter sequences with a sequence of 0 of appropriate length.
ConvOR layer input size is then $M+l-1$.

With this one architecture we can model both local and global patterns.
However for optimization reasons detailed below, we choose to differentiate
the two into two distinct models: a local and a global model. The ConvOR layer
weights for the global model are set and fixed to 1 during training.

##  4 Training strategy

To overcome training challenges attributed to binarized neural networks
(Geiger & Team, 2020), we use latent weights and enforce sparsity dynamically.
We define a loss function that penalizes complex rules and the model is
trained via automatic differentiation (Pytorch) with Adam optimizer.

#### Latent weights

The binary model parameters introduced above (${\bm{W}}_{and}$,
${\bm{W}}_{or}$, ${\bm{W}}_{stack}$, ${\bm{W}}_{conv}$) are trained indirectly
via the training of a continuous parameter $loc$ which is activated
(binarized) by a sigmoid function (Kusters et al., 2022). With such binary
weights and continuous relaxation Eq 1 and 2 are differentiable with nonzero
derivatives (Kusters et al., 2022). As opposed to when using a straight
through estimator (Qiao et al., 2021), non-zero gradients are ensured during
the backward pass. To overcome training limitations, we use a hard concrete
distribution (Qiao et al., 2021; Louizos et al., 2018). It rescales the
weights and the random variable introduced during training prevents from
obtaining local minima (Appendix B). Weight values are in $[0,1]$ during
training, while for testing and rule extraction, a Heaviside is applied to
them ($\geq 0.5$) to ensure strict binarization.

#### Loss function

We define the loss function $\mathcal{L}$ composed of a mean-squared error
component along with a regularization term that penalize the complexity of the
rule,

$\displaystyle\mathcal{L}$ $\displaystyle=\mathcal{L}_{mse}+\lambda\Pi$ (7)

That regularization term $\Pi$, or penalty, evaluates the number of terminal
conditions in the rule. In practice we use $\lambda=10^{-5}$. For a
$\rm{layer}_{n}$ of input size $I$ and output size $O$, the number of terminal
conditions per output corresponds to the weighted sum of the number of
terminal conditions of each output of the previous layer i.e.
$\mathbf{\Pi}_{\rm{layer}_{n}}=\sum_{I}\mathbf{W}_{\rm{layer}_{n}}\mathbf{\Pi}_{\rm{layer}_{n-1}}$,
a vector of size $O$. For the first layer, the StackedOR layer,
$\mathbf{\Pi}_{stack}$ is defined as the sum over the input dimension of the
weights and we can then express the number of terminal conditions of the base
rule model $\Pi_{base}$.

$\displaystyle\mathbf{\Pi}_{\rm{layer}_{0}}=\mathbf{\Pi}_{stack}$
$\displaystyle=\begin{bmatrix}\sum\mathbf{W}_{stack}^{1}\\\ \vdots\\\
\sum\mathbf{W}_{stack}^{K}\end{bmatrix},$ $\displaystyle\qquad\Pi_{base}$
$\displaystyle=\sum\mathbf{W}_{or}\sum\mathbf{W}_{and}\mathbf{\Pi}_{stack}$
(8)

For optimizing $\Pi$ for local patterns, we have to minimize the activated
ConvOR layer weights. For global patterns, we want them to all be activated. A
condition could be set on the sum of ConvOR layer weights (Eq 9) to shift from
one optimization problem to the other but with loss of continuity and thus
differentiability (interesting values of $\tau$ being $M+l-1$, the ConvOR
layer input size, that would correspond to all ConvOR layer weights being
equal to 1, or $M-l+1$ that would allow for $2(l-1)$ weights to be 0, and
corresponds to the padding required for properly accounting for sequence
boundaries (Section 3)).

$\displaystyle\Pi_{local}$ $\displaystyle=\Pi_{base}\sum\mathbf{W}_{conv},$
$\displaystyle\qquad\Pi_{global}$ $\displaystyle=\Pi_{base},$
$\displaystyle\qquad\Pi^{*}=\begin{cases}\Pi_{global}&\mathrm{if}\sum\mathbf{W}_{conv}\geq\tau\\\
\Pi_{local}&\text{otherwise}\end{cases}$ (9)

Due to non continuity of $\Pi^{*}$ in Eq 9, we choose to have two models with
the same architecture for the two cases: the local and the global model
respectively more relevant for their associated pattern. For the local model,
all weights are trainable and $\Pi=\Pi_{local}$. For the global model, weights
in the ConvOR layer are fixed and set to 1, and $\Pi=\Pi_{global}$.

#### Enforced sparsity

Sparsity of the model is crucial to learn concise expressions, the model needs
to generalize without observing _all_ possible instances at training time. The
first requirement for that matter is sparsity in the base rule model. In
addition to the regularization term in the loss function, we propose to use a
sparsify-during-training method (Hoefler et al., 2021) and dynamically enforce
sparsity in weights from 0% to an end rate $r_{f}$ set to 99% in our case (Lin
et al., 2020). Sparsify-during-training method can also benefit the quality of
the training in terms of convergence by correcting for approximation errors
due to premature pruning in early iterations but is highly dependant on the
sparsification schedule (Hoefler et al., 2021).

Every 16 iterations $s$ and for a total of $s_{f}$ training iterations, every
trainable weight is pruned with a binary mask, $\mathbf{m}$ (of size of its
associated weight and applied with Hadamard product ($\odot$)) (Lin et al.,
2020; Zhu & Gupta, 2017). We propose a mask based on the maximum of weight
magnitude $loc$ and pruning rate $r$ (Zhu & Gupta, 2017) making the assumption
that it contributes to generalization (Eq 10). This strategy can be more
aggressive than state-of-art contributions (Lin et al., 2020) due to its
dependency to the $loc$ maximum value. During training, the model with the
highest prediction accuracy on validation dataset and the highest sparsity
(evaluated at each epoch) is kept.

$\displaystyle r$
$\displaystyle=r_{f}-r_{f}\left(1-\frac{s}{s_{f}}\right)^{3},$
$\displaystyle\qquad m_{i,j}$ $\displaystyle=\lvert loc_{i,j}\rvert\geq
r\times\mathrm{max_{i,j}}(loc),$
$\displaystyle\qquad\hat{\mathbf{W}}=\mathbf{W}\odot\mathbf{m}$ (10)

Additional training optimizations have been tested out such as for example
using a binarized optimizer (Helwegen et al., 2019; Geiger & Team, 2020),
adding a scheduled cooling on the sigmoid of the binarized weights,
alternating the training of each layer every few epochs (Qiao et al., 2021) or
using a learning rate scheduler. Those techniques are not presented here but
would be of interest for improving results on specific datasets.

##  5 Experiments

In order to evaluate the validity and usefulness of this method, we apply it
to both synthetic datasets and UCI membranolytic anticancer peptides dataset
(Grisoni et al., 2019; Dua & Graff, 2017).

#### Synthetic Datasets

Table 1: Ground truth applied on sequences of letters (A to F) to generate synthetic unbalanced datasets 1, 2, 3 and 4 along with the distributions of the positive class. In the patterns, $t$ refers to the position when the last observation in a sequence was made. Balanced datasets with same ground truth are generated and are referred to as the dataset number followed by the letter b (Appendix D).  Ground truth | # | Distribution (%)
---|---|---
C at t-4 | 1 | 14.2
A at t-6 and C at t-4 | 2 | 1.5
(A at t-6 and C at t-4) or (B at t-5 and C at t-3) | 3 | 3.6
B-D in sequence | 4 | 20.4

We propose 8 synthetic datasets based on 4 ground truth expressions in both
balanced and unbalanced distributions for discovering simple binary
classification rules with local or global patterns as shown in Table 1. There
are 1000 sequences of letters (A to F) of different length from 4 to 14
letters in each of them (Mean around 9$\pm$3). Generation is detailed in
Appendix D.

#### Peptides Dataset

Besides the synthetic datasets, real-world UCI anticancer peptides dataset
composed of labeled one-letter amino acid sequences, is used (Grisoni et al.,
2019; Dua & Graff, 2017). The multi-classification problem is transformed into
a binary classification problem is the same manner as Nwegbu et al. (2022)
(see Appendix C). Sequence length are from 5 to 38 letters (Mean: 17 $\pm$
5.5) and positive class distribution is 79%.

#### Experimental Setting

All datasets are partitioned in a stratified fashion with 60% for training,
20% for validation and 20% for testing datasets and we use a batch size of 100
sequences. The hidden size in the base rule model is set to the double of the
input size of the AND layer (which is the window size of the convolution).
More details on experimental setting can be found in Appendix E. At each epoch
(200 in total), we evaluate the model against the validation dataset and keep
the model with the highest accuracy and in case of equality the model with
lowest penalty. For each experiment, we run the algorithm 10 times with
different weights initializations. Resulting metrics are averaged over these
runs.

We run the experiments with two different window sizes (3 and 6) for the CR2N
convolution filter size. We compare the two versions of the architecture: the
local and global models described in Section 4 and study three different
dynamic pruning strategies: none, dynamic enforced sparsity from epoch=0 and
from epoch=30 (arbitrary).

##  6 Results

Figure 3: Representations of key results obtained on the synthethic datasets. Error bars represent the standard deviation over the 10 executions with different weights initializations. Full results are available in Appendix F. Table 2: Performance metrics obtained for the different models, window size and pruning strategy on the peptides dataset, along with the standard deviations over the 10 executions with different weights initializations. (Bal. Acc.: balanced accuracy, Epoch: best epoch). Model | Window | Pruning | Accuracy | Bal. Acc. | Penalty ($\Pi$) | Epoch
---|---|---|---|---|---|---
global | 3 | No | 88.9 $\pm$ 5.0 | 75.3 $\pm$ 12.7 | 58.7 $\pm$ 35.9 | 105 $\pm$ 60
Yes | 85.3 $\pm$ 5.3 | 67.3 $\pm$ 14.3 | 19.4 $\pm$ 15.9 | 23 $\pm$ 20
30 | 88.7 $\pm$ 5.0 | 75.4 $\pm$ 12.9 | 46.9 $\pm$ 28.0 | 48 $\pm$ 22
6 | No |  91.2 $\pm$ 1.0 |  81.8 $\pm$ 2.0 | 220.0 $\pm$ 56.5 | 92 $\pm$ 82
Yes | 89.5 $\pm$ 3.6 | 77.6 $\pm$ 9.3 | 132.7 $\pm$ 93.9 | 33 $\pm$ 15
30 |  91.0 $\pm$ 1.3 |  82.0 $\pm$ 2.8 |  97.3 $\pm$ 60.6 | 71 $\pm$ 19
local | 3 | No | 90.0 $\pm$ 3.7 | 78.7 $\pm$ 9.7 | 694.3 $\pm$ 269.3 | 77 $\pm$ 48
Yes |  90.3 $\pm$ 1.5 |  81.6 $\pm$ 1.6 |  885.2 $\pm$ 328.7 | 25 $\pm$ 8
30 | 89.2 $\pm$ 5.2 | 76.3 $\pm$ 13.2 | 674.7 $\pm$ 483.9 | 49 $\pm$ 13
6 | No |  92.1 $\pm$ 1.0 |  83.5 $\pm$ 2.5 |  3.9k $\pm$ 1.4k | 91 $\pm$ 58
Yes | 88.0 $\pm$ 4.7 | 74.6 $\pm$ 12.4 | 1.0k $\pm$ 1.0k | 34 $\pm$ 16
30 |  91.7 $\pm$ 1.5 |  83.5 $\pm$ 2.6 | 1.8k $\pm$ 0.6k | 49 $\pm$ 10

#### Rule grammar and expressivity

The importance of the rule model expressivity can be seen concretely by
comparing the different patterns the local and global models have learned for
dataset 3b for example: 1\. (A or B)-*-(C)-*-*-(A or B or C or D or E or F) in
sequence (global, no pruning, window size=6), and 2. (B at t-5 and C at t-3)
or (A at t-6 and C at t-4) (local, pruning, window size=6). In the first case,
the grammar is not appropriate to model the data (as a reminder, the global
model is a constrained version of the local model) as opposed to the local
model that learned the perfect rule. In practice, on real data, obtained
patterns, such as “if (D or E or G or H or I or N or Q or T or Y)-*-*-*-(D or
E or G or I or N or Q or S or T or V or Y) in sequence” obtained for labelling
‘invalid-virtual’ peptides, can be explored further by a domain expert. Black
box approaches do not provide such insights.

It is also highlighted by comparing experiments with local or global model and
experiments with different window sizes. First of all, the accuracy of the
local model is higher compared to the global model on balanced synthetic
datasets 1, 2 and 3 (Fig 3(a)). For balanced and unbalanced dataset 4, both
models achieve very high accuracies ($>95\%$). However, as shown on Fig 3(b),
it is at the cost of rule complexity for the local approach with averaged
penalty values higher than 60 (and standard deviation higher than 50) compared
to lower than 10 for the global model (and standard deviation lower than 5).
It points out that the local model in that case requires on average at least
more than 6 times more terminal conditions in the learned rule than the global
model for comparable accuracies, but also that the weights initial states have
a huge impact on the rule complexity when the rule grammar is not expressive
enough (with no pruning). Those results are confirmed on real-world dataset
with the peptides dataset, accuracies between the local and global models
especially for a window size of 6 are comparable. However there is an order of
magnitude difference for the penalty, global approach being more concise. It
is important to note that by architecture the global approach has less weights
to train and thus a much lower maximum penalty.

Datasets 2b and 3b benefit from a bigger window size (highly expected for
dataset 3/3b due to ground truth pattern size) as shown in Fig 3(c).
Accuracies are also higher with window size 6 than 3 for the peptides dataset
at the cost of also higher penalties (Table 2).

The more expressive the model is i.e. the more patterns it can model, training
limitations aside, the better for the performance. Of course any black box
neural network with no such 1-1 rule mapping constrained architecture would
reach 100% accuracy, but it is that mapping in particular that makes the model
relevant, expressive and fully interpretable. Also, the best performances in
accuracy for the peptides dataset ($\sim 91\%$) are comparable to the best
results ($\sim 92\%$) obtained from classification with single kernels when
applied to that same dataset in Nwegbu et al. (2022), our model providing an
additional fully-interpretable property. The presented model is also flexible
due to its logical equivalence and can be inputted into other logical layers
for deeper architectures to extend the rule grammar (Beck & Fürnkranz, 2021).
It can also be extended for timeseries, temporal aggregates or multi-
classification problems. Other rule grammar extensions can be inspired by
Linear Temporal Logic domain and regular expression pattern mining (De Giacomo
et al., 2022). However the more expressive the model is the more attention is
required for training and rule complexity.

#### Sparsity and training strategy

The importance of the model sparsity is pointed out by the experiments with
different pruning strategies. First, looking at training scenarios, both on
synthethic and peptide datasets, experiments with sparsity-during-training
approaches reach the best model faster on average than without (lower best
epoch Fig 3(d)). Then, regarding the performance in terms of accuracy, we can
differentiate two cases: balanced and unbalanced datasets. Training of
unbalanced datasets is more affected by the aggressive dynamic pruning
strategy than balanced datasets with a drop in average of around 0.2 in
accuracy for dataset 1 compared to an equivalent accuracy for balanced dataset
1 for example (Fig 3(e)). The pruning strategy starting after 30 epochs is
preferred in both cases. Average accuracies with a pruning strategy not
starting immediately (30 epochs) are comparable to the ones obtain without
pruning for balanced datasets. In terms of rule complexity, penalty values are
lower with pruning and even lower when starting after 30 epochs in most cases
(Fig 3(f)).

With our pruning strategy (Eq 10), we make the assumption that lower positive
$loc$ values are associated to overfitting or redundancy by taking into
account that values closer to 0, i.e. on the sigmoid slope, are more likely to
shift thus less ‘certain’. As pointed out in early work by Prechelt (1997),
the dynamic pruning strategy helps to overcome possible lower generalization
ability compared to a fixed pruning which could explain cases of better
performance (peptide dataset local model window size of 3 for example).
Prechelt proposed a different pruning strategy based on a generalization loss
to characterize the amount of overfitting. When this strategy is relevant in
more general cases and can be applied to many different networks, our strategy
is tailored for minimizing positive trainable parameter values.

Sparsity of the model is also induced via the regularization term $\Pi$ in the
loss function $\mathcal{L}$ (Eq 7). While this method is parameterized with a
relative importance of sparsity for training optimization and provides an
uncontrolled target sparsity, a dynamic pruning strategy is easier to control
for both target sparsity and accuracy but is highly dependent on the pruning
schedule (Hoefler et al., 2021).

An interesting point is made by Hoefler et al. (2021) about the convolutional
operator that ‘can be seen as a sparse version of fully-connected layers’.
That level of forced sparsity in our model is therefore defined by the fixed
window size model parameter with respect to the maximum sequence length. The
ideal sparser window size would be the size of the maximum temporal hidden
pattern in distribution that can only be approximated with external or expert
knowledge and/or tuned with trial and error.

With or without a dynamic pruning strategy, for highly unbalanced datasets (2
and 3), experiments have shown that the training strategy of the model is not
suitable. Indeed most of them, label everything with the majority class (50%
balanced accuracy). It corresponds to the specific case of learning an empty
rule (penalty=0) (Fig 3(a,c,e)). For unbalanced datasets 1 and 4, their best
models do not reach on average the same accuracies as in their balanced
versions.

Overall this training strategy is both the key and the main limitation of our
approach: it can provide a sharp concise rule with minimal redundancy and
simplified logical expression but it is highly dependent on numerous model,
training and pruning parameters and is not suited as is for highly unbalanced
datasets.

##  7 Conclusion

To conclude, we presented a 1D-convolutional neural architecture to discover
local and global patterns in sequential data while learning binary
classification rules. This architecture is fully differentiable, interpretable
and requires sparsity that is enforced dynamically. One main limitation is its
dependence to the window size and sparsity scheduler parameters. Further work
will consist in integrating this block into more complex architectures to
augment the expressivity of the learned rules as well as extending it for
multi-classification.

### Author Contributions

M.C. and R.K. designed the model. R.K. encouraged M.C. to investigate the use
of convolutions and supervised the findings of this work. P.B. and F.F. helped
supervise the project. M.C. validated the training strategy and carried out
the implementation and the experiments. M.C. wrote the manuscript. All authors
provided critical feedback and helped shape the manuscript.

### Acknowledgments

We would like to thank Shubham Gupta for helpful discussions and constructive
feedback as well as Yusik Kim for reviewing the manuscript. This work has been
partially funded by the French government as part of project PSPC AIDA
2019-PSPC-09, in the framework of the “Programme d’Investissement d’Avenir”.

## References

  * Aggarwal (2002) Charu C. Aggarwal.  On effective classification of strings with wavelets.  In _Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’02, pp. 163–172, New York, NY, USA, July 2002. Association for Computing Machinery.  ISBN 978-1-58113-567-1.  doi: 10.1145/775047.775071.
  * Beck & Fürnkranz (2021) Florian Beck and Johannes Fürnkranz.  Beyond DNF: First Steps towards Deep Rule Learning.  In Broňa Brejová, Lucie Ciencialová, Martin Holeňa, František Mráz, Dana Pardubská, Martin Plátek, and Tomáš Vinař (eds.), _Proceedings of the 21st Conference Information Technologies – Applications and Theory (ITAT 2021)_ , volume 2962 of _CEUR Workshop Proceedings_ , pp. 61–68, Hotel Heľpa, Nízke Tatry and Muránska planina, September 2021\. CEUR.
  * Breiman et al. (1984) Leo Breiman, Jerome Friedman, Charles J. Stone, and R. A. Olshen.  _Classification and Regression Trees_.  Taylor & Francis, January 1984.  ISBN 978-0-412-04841-8.
  * Chomsky (1956) Noam Chomsky.  Three models for the description of language.  _IRE Transactions on information theory_ , 2(3):113–124, 1956.  Publisher: IEEE.
  * Cohen (1995) William W. Cohen.  Fast Effective Rule Induction.  In _In Proceedings of the Twelfth International Conference on Machine Learning_ , pp. 115–123. Morgan Kaufmann, 1995\.
  * Dash et al. (2018) Sanjeeb Dash, Oktay Gunluk, and Dennis Wei.  Boolean Decision Rules via Column Generation.  In _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.
  * De Giacomo et al. (2022) Giuseppe De Giacomo, Marco Favorito, Jianwen Li, Moshe Y. Vardi, Shengping Xiao, and Shufang Zhu.  LTLf Synthesis as AND-OR Graph Search: Knowledge Compilation at Work.  In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence_ , pp. 2591–2598, Vienna, Austria, July 2022. International Joint Conferences on Artificial Intelligence Organization.  ISBN 978-1-956792-00-3.  doi: 10.24963/ijcai.2022/359.
  * Dua & Graff (2017) Dheeru Dua and Casey Graff.  _UCI Machine Learning Repository_.  University of California, Irvine, School of Information and Computer Sciences, 2017.
  * Egho et al. (2015) Elias Egho, Dominique Gay, Marc Boulle, Nicolas Voisine, and Fabrice Clerot.  A Parameter-Free Approach for Mining Robust Sequential Classification Rules.  In _2015 IEEE International Conference on Data Mining_ , pp. 745–750, Atlantic City, NJ, November 2015. IEEE.  ISBN 978-1-4673-9504-5.  doi: 10.1109/ICDM.2015.87.
  * Geiger & Team (2020) Lukas Geiger and Plumerai Team.  Larq: An open-source library for training binarized neural networks.  _Journal of Open Source Software_ , 5(45):1746, January 2020.  doi: 10.21105/joss.01746.
  * Glorot & Bengio (2010) Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , pp. 249–256. JMLR Workshop and Conference Proceedings, March 2010.
  * Grisoni et al. (2019) Francesca Grisoni, Claudia S. Neuhaus, Miyabi Hishinuma, Gisela Gabernet, Jan A. Hiss, Masaaki Kotera, and Gisbert Schneider.  De novo design of anticancer peptides by ensemble artificial neural networks.  _Journal of Molecular Modeling_ , 25(5):112, April 2019.  ISSN 0948-5023.  doi: 10.1007/s00894-019-4007-6.
  * Helwegen et al. (2019) Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang-Ting Cheng, and Roeland Nusselder.  Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Hoefler et al. (2021) Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste.  Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks.  _Journal of Machine Learning Research_ , 22(241):1–124, 2021.
  * Holat et al. (2014) Pierre Holat, Marc Plantevit, Chedy Raissi, Nadi Tomeh, Thierry Charnois, and Bruno Cremilleux.  Sequence Classification Based on Delta-Free Sequential Patterns.  In _2014 IEEE International Conference on Data Mining_ , pp. 170–179, Shenzhen, China, December 2014. IEEE.  ISBN 978-1-4799-4302-9 978-1-4799-4303-6.  doi: 10.1109/ICDM.2014.154.
  * Kramer (2020) Stefan Kramer.  A Brief History of Learning Symbolic Higher-Level Representations from Data (And a Curious Look Forward).  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence_ , pp. 4868–4876, Yokohama, Japan, July 2020. International Joint Conferences on Artificial Intelligence Organization.  ISBN 978-0-9992411-6-5.  doi: 10.24963/ijcai.2020/678.
  * Kusters et al. (2022) Remy Kusters, Yusik Kim, Marine Collery, Christian de Sainte Marie, and Shubham Gupta.  Differentiable Rule Induction with Learned Relational Features.  _arXiv:2201.06515 [cs, stat]_ , January 2022.
  * Lakkaraju et al. (2016) Himabindu Lakkaraju, Stephen H. Bach, and Jure Leskovec.  Interpretable Decision Sets: A Joint Framework for Description and Prediction.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pp. 1675–1684, San Francisco California USA, August 2016. ACM.  ISBN 978-1-4503-4232-2.  doi: 10.1145/2939672.2939874.
  * Lin et al. (2020) Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi.  Dynamic model pruning with feedback.  In _International Conference on Learning Representations_ , 2020.
  * Louizos et al. (2018) Christos Louizos, Max Welling, and Diederik P. Kingma.  Learning sparse neural networks through L0 regularization.  In _International Conference on Learning Representations_ , 2018.
  * Nwegbu et al. (2022) Nnanyelugo Nwegbu, Santosh Tirunagari, and David Windridge.  A novel kernel based approach to arbitrary length symbolic data with application to type 2 diabetes risk.  _Scientific Reports_ , 12:4985, March 2022.  ISSN 2045-2322.  doi: 10.1038/s41598-022-08757-1.
  * Prechelt (1997) Lutz Prechelt.  Connection pruning with static and adaptive pruning schedules.  _Neurocomputing_ , 16(1):49–61, July 1997.  ISSN 0925-2312.  doi: 10.1016/S0925-2312(96)00054-9.
  * Qiao et al. (2021) Litao Qiao, Weijia Wang, and Bill Lin.  Learning accurate and interpretable decision rule sets from neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pp. 4303–4311, 2021.
  * Riegel et al. (2020) Ryan Riegel, Alexander Gray, Francois Luus, Naweed Khan, Ndivhuwo Makondo, Ismail Yunus Akhalwaya, Haifeng Qian, Ronald Fagin, Francisco Barahona, Udit Sharma, Shajith Ikbal, Hima Karanam, Sumit Neelam, Ankita Likhyani, and Santosh Srivastava.  Logical Neural Networks, June 2020.
  * Rudin (2019) Cynthia Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1(5):206–215, May 2019.  ISSN 2522-5839.  doi: 10.1038/s42256-019-0048-x.
  * Su et al. (2016) Guolong Su, Dennis Wei, Kush R. Varshney, and Dmitry M. Malioutov.  Interpretable Two-level Boolean Rule Learning for Classification, June 2016.
  * Zhou et al. (2013) Cheng Zhou, Boris Cule, and Bart Goethals.  Itemset Based Sequence Classification.  In David Hutchison, Takeo Kanade, Josef Kittler, Jon M. Kleinberg, Friedemann Mattern, John C. Mitchell, Moni Naor, Oscar Nierstrasz, C. Pandu Rangan, Bernhard Steffen, Madhu Sudan, Demetri Terzopoulos, Doug Tygar, Moshe Y. Vardi, Gerhard Weikum, Camille Salinesi, Moira C. Norrie, and Óscar Pastor (eds.), _Advanced Information Systems Engineering_ , volume 7908, pp. 353–368. Springer Berlin Heidelberg, Berlin, Heidelberg, 2013.  ISBN 978-3-642-38708-1 978-3-642-38709-8.  doi: 10.1007/978-3-642-40988-2˙23.
  * Zhu & Gupta (2017) Michael Zhu and Suyog Gupta.  To prune, or not to prune: exploring the efficacy of pruning for model compression, November 2017.  URL http://arxiv.org/abs/1710.01878.  arXiv:1710.01878 [cs, stat].

##  Appendix A Context-Free Grammar

#### Context-free grammar (Chomsky, 1956)

A context free-grammar is a 4-tuple $G=(V_{T},V_{N},S,R)$ where

  * $-$

$V_{T}$, is finite set of terminals or terminal elements in the language that
form the alphabet of the language.

  * $-$

$V_{N}$, disjoint from $V_{T}$, is a finite set of non terminal elements
(variables) that define a sub-language of $L$. We note $V=V_{N}\cup V_{T}$,
the vocabulary of the grammar.

  * $-$

$S\in V_{N}$, is the start symbol or variable that defines the whole sentence.

  * $-$

$R$ is a finite set of rules or production rules of the form $A\rightarrow w$
with $A\in V_{N}$ and $w\in V^{*}$

In the following, we have 3 different types of terminal elements on the syntax
level:

  * •

reserved words, distinguished with the following style : reserved

  * •

signs, such as for example $-,*,...$

  * •

other terminal elements that are defined prior to the grammar, distinguished
with the following style: terminal

Production rules presented in the paper (Eq 3, Eq 4, Eq 5 and Eq 6) define
grammars when associated to values for $V_{T},V_{N}$ and $S$.

Here is an example for the base rule model grammar with production rules in Eq
3.

$\begin{array}[]{lll}V_{T}&=&\left\\{\texttt{\text@underline{if}},\texttt{\text@underline{then
class = 1 else class =
0}},\land,\lor,=\right\\}\cup\left\\{\text{\text@underline{$x_{1}$}},\text{\text@underline{$x_{2}$}},\dots\right\\}\cup\\\
&&\left\\{\text{\text@underline{$x_{c_{1}}=\alpha^{1}_{0}$}},\dots,\text{\text@underline{$x_{c_{1}}=\alpha^{1}_{n_{1}}$}},\dots,\text{\text@underline{$x_{c_{k}}=\alpha^{k}_{n_{k}}$}}\right\\}\\\
V_{N}&=&\\{\text{rule},\text{base
expression},\text{conjunction},\text{predicate},\text{categorical
expression},\\\ &&\text{categorical literal},\text{literal}\\}\\\
S&=&\left\\{\text{rule}\right\\}\\\ \end{array}$ (11)

##  Appendix B Hard concrete distribution (Louizos et al., 2018)

Parameters are set as follows: $\beta=2/3$, $\zeta=1.1$ and $\gamma=-0.1$.

$\begin{array}[]{lll}u\sim
U(0,1),&s=\sigma((log(u)-log(1-u)+loc)/\beta),&\hat{s}=s*(\zeta-\gamma)+\gamma\end{array}$
(12) $W=\min(\max(\hat{s},0),1)$ (13)

##  Appendix C Peptides Dataset

UCI anticancer peptides dataset (Grisoni et al., 2019) (Available on Dua &
Graff (2017)) is composed of one-letter amino acid sequences (of variable
length) and each sequence is labeled with its anticancer activity on breast
cancer cell lines. The dataset provides 4 classes with the following
distribution: 83 inactive-exp, 750 inactive-virtual, 98 moderately active and
18 very active. Sequences lengths range from 5 to 38 letters (Mean: 17 $\pm$
5.5). We transform this multi-classification problem into a binary
classification problem (as done in Nwegbu et al. (2022)). Class ‘inactive-
virtual’ is the positive class (750) and all the other are combined as the
negative class (199). No other processing of the data is necessary and we
leave it as is.

##  Appendix D Synthetic Datasets Generation

Balanced datasets are generated randomly with the same ground truth as
unbalanced datasets. Then, they are upsampled until the minority class
represents half of the goal dataset size and appropriate number of majority
class are randomly removed.

##  Appendix E Experimental Setting

The $loc$-parameters for weights computation are initialized with xavier
uniform initialization method (Glorot & Bengio, 2010). The loss function is
described in Eq 7 and depends on the MSE loss and regularization coefficient
$\lambda=10^{-5}$. The adam optimizer is used with a fixed learning rate set
to $0.1$ and a run consists of 200 epochs.

Experiments were run on CPU on a MacBookPro18,2 (2021) with Apple M1 Max chip,
10 Cores, 32 GB of RAM and running macOS Monterey Version 12.4.

##  Appendix F Full experiment results

Table 3: Performance metrics obtained for the different models, window size and pruning strategy on the synthetic datasets. Values are followed by the standard deviation over the 10 executions with different weights initializations. (Bal Acc: balanced accuracy, Epoch: best epoch). Dataset | Model | Window | Pruning | Bal. Acc. | Penalty | Epoch
---|---|---|---|---|---|---
1 | global | 3 | No | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 3 $\pm$ 1
Yes | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 3 $\pm$ 2
30 | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 2 $\pm$ 1
6 | No | 54.9 $\pm$ 3.9 | 9.5 $\pm$ 8.1 | 109 $\pm$ 64
Yes | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 1
30 | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 1
local | 3 | No | 100.0 $\pm$ 0.0 | 1.8 $\pm$ 1.5 | 63 $\pm$ 24
Yes | 90.0 $\pm$ 20.0 | 0.8 $\pm$ 0.4 | 56 $\pm$ 47
30 | 85.0 $\pm$ 22.9 | 0.7 $\pm$ 0.5 | 41 $\pm$ 27
6 | No | 100.0 $\pm$ 0.0 | 5.8 $\pm$ 5.6 | 96 $\pm$ 50
Yes | 80.9 $\pm$ 23.6 | 1.1 $\pm$ 1.4 | 26 $\pm$ 18
30 | 95.0 $\pm$ 15.0 | 0.9 $\pm$ 0.3 | 79 $\pm$ 43
1b | global | 3 | No | 70.7 $\pm$ 4.0 | 5.1 $\pm$ 1.6 | 81 $\pm$ 27
Yes | 68.5 $\pm$ 6.6 | 5.8 $\pm$ 2.8 | 47 $\pm$ 31
30 | 71.6 $\pm$ 3.5 | 5.0 $\pm$ 1.3 | 60 $\pm$ 16
6 | No | 76.5 $\pm$ 3.4 | 37.4 $\pm$ 20.3 | 108 $\pm$ 49
Yes | 70.2 $\pm$ 4.3 | 12.5 $\pm$ 5.5 | 29 $\pm$ 15
30 | 72.7 $\pm$ 5.1 | 15.4 $\pm$ 6.9 | 53 $\pm$ 13
local | 3 | No | 100.0 $\pm$ 0.0 | 3.0 $\pm$ 3.1 | 70 $\pm$ 47
Yes | 100.0 $\pm$ 0.0 | 1.0 $\pm$ 0.0 | 49 $\pm$ 39
30 | 100.0 $\pm$ 0.0 | 1.0 $\pm$ 0.0 | 54 $\pm$ 30
6 | No | 100.0 $\pm$ 0.0 | 12.6 $\pm$ 10.1 | 116 $\pm$ 52
Yes | 100.0 $\pm$ 0.0 | 1.0 $\pm$ 0.0 | 77 $\pm$ 24
30 | 100.0 $\pm$ 0.0 | 1.0 $\pm$ 0.0 | 87 $\pm$ 27
2 | global | 3 | No | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 3 $\pm$ 1
Yes | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 3 $\pm$ 1
30 | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 2 $\pm$ 1
6 | No | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 1
Yes | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 0
30 | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 1
local | 3 | No | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 1
Yes | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 1
30 | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 1
6 | No | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 1
Yes | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 0
30 | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 0
2b | global | 3 | No | 90.5 $\pm$ 0.0 | 3.7 $\pm$ 2.1 | 44 $\pm$ 37
Yes | 85.7 $\pm$ 9.8 | 2.3 $\pm$ 0.6 | 39 $\pm$ 21
30 | 90.5 $\pm$ 0.0 | 2.0 $\pm$ 0.0 | 51 $\pm$ 24
6 | No | 94.2 $\pm$ 0.6 | 19.8 $\pm$ 7.7 | 89 $\pm$ 36
Yes | 90.5 $\pm$ 0.0 | 2.0 $\pm$ 0.0 | 71 $\pm$ 16
30 | 91.0 $\pm$ 1.5 | 12.4 $\pm$ 13.0 | 61 $\pm$ 26
local | 3 | No | 97.9 $\pm$ 2.6 | 2.8 $\pm$ 1.2 | 58 $\pm$ 42
Yes | 97.4 $\pm$ 2.6 | 1.5 $\pm$ 0.5 | 43 $\pm$ 28
30 | 98.0 $\pm$ 2.4 | 1.6 $\pm$ 0.5 | 54 $\pm$ 29
6 | No | 100.0 $\pm$ 0.0 | 12.5 $\pm$ 7.4 | 105 $\pm$ 56
Yes | 100.0 $\pm$ 0.0 | 2.0 $\pm$ 0.0 | 51 $\pm$ 21
30 | 100.0 $\pm$ 0.0 | 2.4 $\pm$ 1.2 | 68 $\pm$ 22
3 | global | 3 | No | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 2 $\pm$ 1
Yes | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 2 $\pm$ 1
30 | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 2 $\pm$ 1
6 | No | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 1
Yes | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 0
30 | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 0
local | 3 | No | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 1
Yes | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 2 $\pm$ 1
30 | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 1
6 | No | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 0
Yes | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 1 $\pm$ 1
30 | 50.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0 | 2 $\pm$ 1
3b | global | 3 | No | 79.8 $\pm$ 8.1 | 4.6 $\pm$ 2.2 | 98 $\pm$ 51
Yes | 70.5 $\pm$ 8.1 | 3.1 $\pm$ 1.4 | 48 $\pm$ 31
30 | 80.4 $\pm$ 9.2 | 2.8 $\pm$ 0.6 | 57 $\pm$ 36
6 | No | 90.2 $\pm$ 1.4 | 26.2 $\pm$ 15.3 | 120 $\pm$ 51
Yes | 77.4 $\pm$ 7.4 | 14.1 $\pm$ 8.0 | 35 $\pm$ 13
30 | 87.7 $\pm$ 3.1 | 11.5 $\pm$ 6.6 | 61 $\pm$ 10
local | 3 | No | 89.8 $\pm$ 6.3 | 8.5 $\pm$ 3.9 | 95 $\pm$ 64
Yes | 86.2 $\pm$ 8.0 | 6.1 $\pm$ 8.2 | 50 $\pm$ 34
30 | 92.5 $\pm$ 4.6 | 4.2 $\pm$ 1.8 | 64 $\pm$ 22
6 | No | 97.1 $\pm$ 1.2 | 15.1 $\pm$ 7.9 | 123 $\pm$ 49
Yes | 95.7 $\pm$ 2.9 | 10.1 $\pm$ 12.9 | 43 $\pm$ 8
30 | 96.8 $\pm$ 0.9 | 7.6 $\pm$ 2.9 | 69 $\pm$ 21
4 | global | 3 | No | 100.0 $\pm$ 0.0 | 2.2 $\pm$ 0.6 | 42 $\pm$ 28
Yes | 90.0 $\pm$ 20.0 | 1.6 $\pm$ 0.8 | 28 $\pm$ 14
30 | 95.0 $\pm$ 15.0 | 1.8 $\pm$ 0.6 | 42 $\pm$ 25
6 | No | 100.0 $\pm$ 0.0 | 4.3 $\pm$ 3.4 | 72 $\pm$ 35
Yes | 86.2 $\pm$ 21.3 | 1.7 $\pm$ 0.9 | 33 $\pm$ 20
30 | 100.0 $\pm$ 0.0 | 2.0 $\pm$ 0.0 | 43 $\pm$ 15
local | 3 | No | 98.7 $\pm$ 0.4 | 28.6 $\pm$ 14.5 | 108 $\pm$ 40
Yes | 59.1 $\pm$ 12.9 | 3.4 $\pm$ 5.1 | 30 $\pm$ 37
30 | 77.1 $\pm$ 20.7 | 12.5 $\pm$ 8.7 | 58 $\pm$ 39
6 | No | 99.1 $\pm$ 0.6 | 68.8 $\pm$ 51.0 | 109 $\pm$ 31
Yes | 52.1 $\pm$ 5.2 | 3.3 $\pm$ 5.2 | 11 $\pm$ 13
30 | 79.3 $\pm$ 19.2 | 21.9 $\pm$ 17.5 | 57 $\pm$ 24
4b | global | 3 | No | 100.0 $\pm$ 0.0 | 2.7 $\pm$ 1.1 | 16 $\pm$ 8
Yes | 99.2 $\pm$ 2.4 | 2.1 $\pm$ 0.3 | 29 $\pm$ 18
30 | 100.0 $\pm$ 0.0 | 2.0 $\pm$ 0.0 | 42 $\pm$ 32
6 | No | 100.0 $\pm$ 0.0 | 6.7 $\pm$ 3.8 | 90 $\pm$ 52
Yes | 99.2 $\pm$ 2.4 | 4.7 $\pm$ 8.1 | 50 $\pm$ 22
30 | 100.0 $\pm$ 0.0 | 2.2 $\pm$ 0.6 | 69 $\pm$ 25
local | 3 | No | 100.0 $\pm$ 0.0 | 39.9 $\pm$ 17.5 | 76 $\pm$ 38
Yes | 99.0 $\pm$ 2.0 | 29.2 $\pm$ 9.4 | 60 $\pm$ 14
30 | 98.7 $\pm$ 4.0 | 35.9 $\pm$ 17.3 | 53 $\pm$ 14
6 | No | 100.0 $\pm$ 0.0 | 122.6 $\pm$ 100.2 | 107 $\pm$ 42
Yes | 88.2 $\pm$ 7.8 | 103.9 $\pm$ 77.3 | 44 $\pm$ 16
30 | 98.0 $\pm$ 3.9 | 79.0 $\pm$ 55.9 | 68 $\pm$ 12