# Continual Learning for Unsupervised Anomaly Detection
in Continuous Auditing of Financial Accounting Data

Hamed Hemati1, Marco Schreyer1, Damian Borth 1

###### Abstract

International audit standards require the direct assessment of a financial
statement’s underlying accounting journal entries. Driven by advances in
artificial intelligence, deep-learning inspired audit techniques emerged to
examine vast quantities of journal entry data. However, in regular audits,
most of the proposed methods are applied to learn from a comparably stationary
journal entry population, e.g., of a financial quarter or year. Ignoring
situations where audit relevant distribution changes are not evident in the
training data or become incrementally available over time. In contrast, in
continuous auditing, deep-learning models are continually trained on a stream
of recorded journal entries, e.g., of the last hour. Resulting in situations
where previous knowledge interferes with new information and will be entirely
overwritten. This work proposes a continual anomaly detection framework to
overcome both challenges and designed to learn from a stream of journal entry
data experiences. The framework is evaluated based on deliberately designed
audit scenarios and two real-world datasets. Our experimental results provide
initial evidence that such a learning scheme offers the ability to reduce
false-positive alerts and false-negative decisions.

## Introduction

According to the International Standards in Auditing (ISA), auditors are
obliged to collect reasonable assurance that an issued financial statement is
free from fraud that causes a material misstatement (AICPA 2002; IFAC 2009).
At the same time, the Association of Certified Fraud Examiners (ACFE)
estimates that organizations lose 5% of their annual revenues due to fraud
(ACFE 2020). The term fraud refers to ‘the abuse of one’s occupation for
personal enrichment through the deliberate misuse of an organization’s
resources or assets’ (Wells 2017). A similar study revealed that approx. 30%
of the respondents experienced losses between $100,000 and $5 million due to
fraud (PwC 2020). The study also showed that financial statement fraud caused
the highest median loss of the surveyed fraud schemes.111The ACFE study
encompasses an analysis of 2,504 cases of occupational fraud surveyed between
July 2019 and September 2019 in 125 countries. The PwC study encompasses over
5,000+ respondents that experienced economic crime in the last 24 months. To
detect such fraud, the ISA demands auditors to examine a statement’s
underlying accounting transactions, usually referred to as journal entries
(CAQ 2008; IFAC 2009). Nowadays, organizations record vast quantities of
journal entries in Accounting Information Systems (AIS) or more general
Enterprise Resource Planning (ERP) systems (Grabski, Leech, and Schmidt 2011).
Figure 1 depicts an exemplary hierarchical view of the journal entry recording
process in designated database tables of an AIS system. In general, such
recorded entries originate from non-stationary activities due to the dynamics
of organizational processes, e.g., the (i) introduction, (ii) redesign, or
(iii) discontinuation of business processes. Therefore, the detection of fraud
in non-stationary journal entry distributions remains a challenging task
requiring significant efforts and resources.

Figure 1: Exemplary view of an Accounting Information System (AIS) that
records distinct layer of abstractions, namely (1) the business process, (2)
the accounting and (3) technical journal entry information in designated
tables.

In order to conduct fraud, perpetrators need to deviate from regular journal
entry posting patterns. Such a divergence is then recorded in a small fraction
of journal entries that exhibit ‘anomalous’ attribute values. Nowadays,
auditors apply a wide range of Computer Assisted Audit Techniques (CAATs) when
examining journal entries. These techniques often encompass rule-based
analyses, such as the detection of unusual posting times (Fay and Negangard
2017) or multiple vendor bank account changes (Coderre 2009). Furthermore,
statistical analyses are applied, such as the Benford’s Law (Benford 1938) or
extreme value analysis (Bolton and Hand 2002). Driven by the rapid
technological advances of artificial intelligence, deep-learning (LeCun,
Bengio, and Hinton 2015) enabled audit techniques have recently emerged in
financial audits (Sun 2019; Nonnenmacher and Gómez 2021).

At the same time, the application of deep-learning in modern audit engagements
results, among others, in the challenge of (i) stationary training setups and
(ii) catastrophic forgetting. The first challenge originates from the
situation that in regular auditing deep-learning models are learned from a
limited and therefore stationary population of journal entries. Typically the
training only encompasses entries posted throughout the in-scope audit period,
e.g., the current financial quarter or year. Such a learning setup exhibits
the drawback that relevant distribution changes might not be evident in the
training data. The second challenge originates from the fact that deep-
learning based audit models are prone to Catastrophic Forgetting (CF)
(McCloskey and Cohen 1989), i.e., training a model with new information
interferes with previously learned knowledge. Typically, in continuous
auditing (Vasarhelyi and Halper 1991; Vasarhelyi, Alles, and Kogan 2004) the
model training encompasses a constant stream of entries recorded within
incremental time intervals, e.g., the recent hour or day. This phenomenon
leads, in the worst-case, to a situation in which previous knowledge will be
entirely overwritten by new one.

In order to mitigate false-positive alerts or false-negative decisions in
real-world audits, it is paramount to learn important audit relevant
information that becomes incrementally available over time. For example, an
organizational change in previous financial years might be relevant when
concluding on a recently detected audit finding. This holds in particular for
applying unsupervised anomaly detection techniques to audit large-scale
accounting data. Recently, the idea of Continual Learning (CL), also referred
to as lifelong learning, regained attention (Chen and Liu 2018). CL denotes a
learning paradigm in which the assumption is made that data will become
available over time. We think accommodating new knowledge over time while
retaining previously learned information is essential for future deep-learning
enabled audit techniques. In this work, inspired by the successes of CL (Sun
2019; Zhai, Liu, and Sun 2019; Lee and Lee 2020), we investigate whether such
a CL setup can be utilized in financial audits to detect accounting anomalies?
In summary, we present the following contributions:

  * •

Learnability \- We propose and evaluate a novel learning framework to
continuously learn from real-world journal entry for unsupervised anomaly
detection.

  * •

Transferability \- We demonstrate that the continual learning techniques
outperform the re-training on an exhaustive dataset and the sequential fine-
tuning.

The remainder of this work is structured as follows: First, we provide an
overview of related work. Following, we present the proposed continual
learning framework for unsupervised anomaly detection in financial audits.
Afterwards, the experimental setup is described and results are outlined. The
work concludes with a discussion and summary.

Figure 2: The proposed continual anomaly detection framework that continuously
learns from a stream of accounting data experiences $\\{E_{i}\\}_{i=1}^{N}$
(e.g., financial quarters or years) by accommodating new information while
preserving previous knowledge.

## Related Work

Lately, techniques that build on (deep) machine learning have been gradually
applied to different audit tasks (Sun 2019; Cho et al. 2020). In parallel, the
idea of continual learning triggered a sizable body of machine learning
research (Parisi et al. 2019; Delange et al. 2021). In the following, we focus
our literature review on (i) CAATs applied in financial audits and (ii)
continual deep-learning techniques.

Computer Assisted Audit Techniques: With the evolution of AIS systems, the
development of CAATs has been of central relevance in auditing. Various
techniques have been proposed to audit structured accounting data, such as
(un-) supervised machine-learning approaches (Bay et al. 2006; McGlohon et al.
2009; Jans, Lybaert, and Vanhoof 2010; Thiprungsri and Vasarhelyi 2011;
Argyrou 2012, 2013; Hu et al. 2020), statistical analyzes (Debreceny and Gray
2010; SEOW, Pan, and Suwardy 2016), rule-based procedures (Khan et al. 2009,
2010, 2014; Islam et al. 2010), business process mining (Jans et al. 2011;
Jans, Alles, and Vasarhelyi 2014; Werner and Gehrke 2015; Werner 2016), and
the combination thereof (Baader and Krcmar 2018; Stephan, Lahann, and Fettke
2021). Recently, deep-learning techniques have emerged in auditing. Such
techniques encompass, autoencoder neural networks (Schreyer et al. 2017;
Schultz and Tropmann-Frick 2020; Nonnenmacher et al. 2021; Zupan et al. 2020;
Schreyer et al. 2020) to detect accounting anomalies and learn representative
audit samples. Lately, self-supervised learning has been proposed to learn
multi-task audit models (Schreyer, Sattarov, and Borth 2021).

Continual Deep-Learning Techniques: The challenge of CF in training neural
networks, has been studied extensively (French 1999; Díaz-Rodríguez et al.
2018; Toneva et al. 2018; Lesort, Caccia, and Rish 2021). To mitigate CF,
several approaches have been proposed, broadly classified as (i) rehearsal,
(ii) regularization, and (iii) dynamic architecture techniques. Rehearsal
based techniques (Rolnick et al. 2018; Isele and Cosgun 2018; Chaudhry et al.
2019) replay samples from past experiences. Thereby, samples are saved in a
buffer and replayed to conduct knowledge distillation (Rebuffi et al. 2017).
Other methods use gradients of previous tasks to mitigate gradient
interference (Lopez-Paz and Ranzato 2017; Chaudhry et al. 2018). Furthermore,
generative models are used to generate synthesized samples for rehearsal (Shin
et al. 2017). Regularization based techniques consolidate previously acquired
knowledge when optimizing the model parameters. Learning without Forgetting
(LwF) (Li and Hoiem 2017), use knowledge distillation to regularize parameter
updates. Elastic Weight Consolidation (EWC) (Kirkpatrick et al. 2017) directly
regularizes the parameters based on their importance concerning previous data.
Similarly to EWC, in Synaptic Intelligence (Zenke, Poole, and Ganguli 2017)
the model parameter importance is regularized but in a separate parameter
optimization step. Dynamic architecture methods prevent forgetting by the
deliberate reuse of (Mendez and Eaton 2021) or increase in (Rusu et al. 2016)
model parameters.

Concluding from the reviewed literature, this work presents, to the best of
our knowledge, the first analysis on the application of continual deep-
learning methods for the purpose of real-world financial statement audits.

## Methodology

Let  $X=\\{x_{1},x_{2},x_{3},...,x_{N}\\}$ formally define a population of
$i=1,2,3,...,N$ journal entries. Each individual entry, denoted by
$x_{i}=\\{x_{i}^{1},x_{i}^{2},...,x_{i}^{M};x_{i}^{1},x_{i}^{2},...,x_{i}^{K}\\}$
, consists of  $j=1,2,3,...,M$ categorical accounting attributes and
$l=1,2,3,...,K$ numerical accounting attributes. The individual attributes
describe the journal entries details, e.g., the entries’ posting type, posting
date, amount, general-ledger. When examining journal entries, recorded in
large-scaled ERP systems, two characteristics can be observed: First, the
journal entry attributes exhibit a high variety of distinct attribute values,
e.g., due to the high number of vendors or distinct posting amounts. Second,
journal entries exhibit a plethora of attribute value correlations, e.g., a
document type usually posted in combination with a certain general ledger
account. Derived from this observation we distinguish two classes of anomalous
entries (Breunig et al. 2000), namely global and local accounting anomalies:

  * •

Global Anomalies correspond to journal entries that exhibit unusual or rare
individual attribute values, e.g., rarely used ledgers, or unusual posting
times. Such anomalies are often simple to detect since they are created
unintentionally in the context of regular posting activities. Hence, global
anomalies are usually associated with a high error risk.

  * •

Local Anomalies correspond to journal entries that exhibit unusual or rare
attribute value correlations, e.g., unusual co-occurrences of general ledgers,
posting types and user accounts. Such anomalies are often difficult to detect
since perpetrators aim to imitate regular posting activities. Hence, local
anomalies are usually associated with a high fraud risk.

Ultimately, auditors aim to learn a model that detects both classes of
anomalies using a continual anomaly detection framework. In the following, we
describe the main components of our proposed framework. A schematic view of
the framework is illustrated in Fig. 2, encompassing an anomaly detection and
a continual learning setup.

Anomaly Detection Setup: Hawkins et al. (2002) proposed the application of
Autoencoder Networks (AEN’s), a special type of feed-forward multi-layer
neural network for unsupervised anomaly detection. Formally, AEN’s are
comprised of two nonlinear functions referred to as encoder and decoder
network (Hinton and Salakhutdinov 2006) that are jointly trained to
reconstruct a given input. The encoder network $q_{\psi}(\cdot)$ maps an input
$x_{i}\in\mathcal{R}^{k}$ to a code vector $z_{i}\in\mathcal{R}^{m}$ referred
to as latent representation, where usually $k>m$. Subsequently, this
representation is mapped back by the decoder network $p_{\phi}(\cdot)$ to a
reconstruction $\tilde{x}_{i}\in\mathcal{R}^{k}$ of the original input space.
In an attempt to achieve $x\approx\tilde{x}$ the AEN is trained to minimize
the dissimilarity of a given journal entry $x_{i}$ and its reconstruction
$\tilde{x}_{i}=p_{\phi}(q_{\psi}(x_{i}))$ as faithfully as possible. Thereby,
the training objective is to learn a set of optimal model parameters
$\theta^{*}=\\{\psi^{*}\cup\,\phi^{*}\\}$ by minimizing the AEN’s
reconstruction loss ($\mathcal{L}_{Rec}$), formally denoted as:

$\arg\min_{\phi,\psi}\|x_{i}-p_{\phi}(q_{\psi}(x_{i}))\|,$ (1)

where $i=1,2,3,...,N$. Upon successful AEN training, the model is used to
obtain the individual reconstruction error of each journal entry. In a real-
world audit scenario, entries that correspond to a high reconstruction error
will then be selected for subsequent substantial examination.

Continual Learning Setup: In Continual Learning (CL) a model $f_{\theta}$,
e.g., an autoencoder $f_{\theta}\\!:q_{\psi}\oplus p_{\phi}$, is trained in a
setup where data  $D=\bigcup_{i=1}^{M}D_{i}$ is observed as a stream of $M$
disjoint experiences  $\\{E_{i}\\}_{i=1}^{M}$ . In the context of auditing,
the journal entry data $D_{i}$ of a particular experience $E_{i}$ originates
from $N$ organizational posting activities $A_{j}$ as defined by
$D_{i}=\bigcup_{j=1}^{N}A_{j}$ . A model trained on the data of the i-th
experience is denoted as  $f^{(i)}_{\theta}$ , where $\theta$ defines the
model parameters. The models reconstruction loss can then be derived by:

$\mathcal{L}_{Rec}^{(i)}=\sum_{j=1}^{N}{\mathcal{L}_{Rec}(f_{\theta}^{(i)},A_{j})},$
(2)

where $A_{j}$ denotes the organizational posting activities of experience
$E_{i}$.

In this work three CL setups are utilized to audit financial accounting data,
namely (i) Sequential Fine-tuning, (ii) EWC, and (iii) Experience Replay. In
each setup an anomaly detection model is incrementally trained on a stream of
$M-1$ past experiences $\\{E_{i}\\}_{i=1}^{M-1}$. For each experience $E_{i}$,
the model parameters $\theta$ are initialized with the optimal model parameter
learned at the previous experience $E_{i-1}$. In the Sequential Fine-Tuning
(SFT) setup, the parameters $\theta$ are optimized, as defined by:

$\theta^{*(i)}=\arg\min_{\theta}{\mathcal{L}(f(\theta),D_{i}|\theta_{init}=\theta^{*(i-1)})},$
(3)

where $\theta^{*(i)}$ denotes the optimal parameters obtained for the previous
experience $E_{i}$. In the regularization based Elastic Weight Consolidation
(EWC) setup, the parameters $\theta$ are optimized, as defined by:

$\theta^{*(i)}=\arg\min_{\theta}{\widehat{\mathcal{L}}(f(\theta),D_{i}|\theta_{init}=\theta^{*(i-1)})},$
(4)

where $\theta^{*(i)}$ denotes the optimal parameters obtained for the previous
experience $E_{i}$. To update the model’s parameters in a given experience,
EWC (Kirkpatrick et al. 2017) is applied by determining the Fisher information
matrix $F$ according to
$\hat{\mathcal{L}}(\theta)=\mathcal{L}_{Rec}(\theta)+\sum_{k}{\frac{\lambda}{2}}F_{k}(\theta_{i}-\tilde{\theta}_{k})^{2}$
, where $k$ denotes the model parameters of the current model and
$\tilde{\theta}$ the optimal parameters of the previous experience. In the
rehearsal based Experience Replay (ER) setup, the parameters $\theta$ are
optimized, as defined by:

$\theta^{*(i)}=\arg\min_{\theta}{\mathcal{L}(f(\theta),\widehat{D}_{i}|\theta_{init}=\theta^{*(i-1)})},$
(5)

where $\theta^{*(i)}$ denotes the optimal parameters obtained for the previous
experience $E_{i}$. To update the model’s parameters in a given experience,
samples of past experiences are replayed. Therefore, a sample buffer
$\mathcal{B}$ of capacity $N_{B}$ is employed. The buffer comprises an equal
number of  $\lfloor\frac{N_{B}}{M}\rfloor$ samples for each observed
experience.

We evaluate the continual learning setups against two baselines, namely (i)
single experience and (ii) joint experience learning. In Single Experience
Learning (SEL) only the journal entries of the current in-scope audit
experience $E_{M}$ are used to train $f_{\theta}$. Such a learning setup is
commonly applied in contemporary audits. The model parameters $\theta$ are
randomly initialized and optimized, as defined by:

$\theta^{*(i)}=\arg\min_{\theta}{\mathcal{L}(f(\theta),D_{i}|\theta_{init}={Rand})},$
(6)

where $\theta^{*(i)}$ denotes the optimal parameters obtained after training
on $E_{i}$. We view this setup as performance upper-bound in our experiments.
In Joint Experience Learning (JEL) the journal entries of all previous
experiences  $\\{E_{i}\\}_{i=1}^{M-1}$ and the current in-scope experience
$E_{M}$ are used to train $f_{\theta}$ from scratch. However, such a learning
setup is not desirable in contemporary audit engagements due to time-budget
constraints. The model parameters $\theta$ are randomly initialized and
optimized, as defined by:

$\theta^{*(i)}=\arg\min_{\theta}{\mathcal{L}(f(\theta),\\{D_{1},...,D_{i}\\}|\theta_{init}={Rand})},$
(7)

where $\theta^{*(i)}$ denotes the optimal parameters obtained after training
on $E_{i}$. Nowadays, this setup is commonly rejected due to time-budget
constraints of audit engagements. We view this setup as performance lower-
bound in our experiments.

## Experimental Setup

In this section, we describe the dataset, data pre-processing, audit scenarios
and experimental details to continually learn from real-world accounting data.

(a)

Single Experience Learning (SEL)

(b)

Joint Experience Learning (JEL)

(c)

Sequential Fine-Tuning (SFT)

(d)

Experience Replay (ER)

Figure 3: Average reconstruction losses $\mathcal{L}_{Rec}$ in the in-scope
audit period $E_{M}$ of dataset $D^{A}$ for the decaying target department
(purple bar) and the four departments exhibiting the highest loss (non-purple
bars). It can be observed that in CL the linearly decaying target department

$A^{\prime}$

results in a significantly reduced number of loss magnitude based false-
positive alerts.

Datasets and Data Preparation:  To evaluate the learning capabilities of the
proposed continual learning framework we use two publicly available datasets
of real-world financial payment data.222Due to the general confidentiality of
journal entry data, we evaluate the proposed methodology based on two public
available real-world datasets to allow for the reproducibility of our results.
The datasets exhibit high similarity to real-world ERP accounting data, e.g.,
typical manual payments or payment run records.

  * •

The City of Philadelphia
payments333https://www.phila.gov/2019-03-29-philadelphias-initial-release-of-
city-payments-data/ denoted as  $D^{A}$ encompass a total of 238,894 payments
generated by 58 distinct city departments and comprises $10$ categorical and
$1$ numerical attribute(s).

  * •

The City of Chicago payments444https://data.cityofchicago.org/Administration-
Finance/Payments/s4vu-giwb/ denoted as  $D^{B}$ encompass a total of 399,158
payments generated by 54 distinct city departments and comprises of $6$
categorical and $1$ numerical attribute(s).

For both datasets, let  $\\{A_{j}\\}_{j=1}^{N}$ denote the set of payments
generating city departments. For each dataset we conduct the following pre-
processing steps: First, the $\tau$ departments exhibiting the highest number
of payments  $\\{A^{*}_{k}\\}_{k=1}^{\tau}$ are determined. Second, for each
city department  $A^{*}_{k}$ a set of  $\eta$ payments is randomly sampled
$x_{i}\sim\\{A^{*}\\}_{k=1}^{\tau}$ , where  $i=1,..,\eta$ . Third, the
sampled payment’s attribute values are transformed according to:

  * •

The categorical attribute values of  $x_{i}^{j}$ are converted into one-hot
numerical tuples of bits  $\hat{x}_{i}^{j}\in\\{0,1\\}^{\upsilon}$ , where
$\upsilon$ denotes the number of unique attribute values in  $x^{j}$ .

  * •

The numerical attribute values of  $x_{i}^{l}$ are scaled, according to
$\hat{x}_{i}^{l}=(x_{i}^{l}-\text{min}(x^{l})/(\text{max}(x^{l})-\text{min}(x^{l}))$
, where the min and max are obtained over all attribute values in  $x^{l}$ .

We set the number of departments  $\tau=10$ and number of samples
$\eta=10,000$ in all our experiments to derive the pre-processed journal
entries  $\hat{x}\in\hat{X}$ , where
$\hat{x}_{i}=\\{\hat{x}_{i}^{1},\hat{x}_{i}^{2},...,\hat{x}_{i}^{M};\hat{x}_{i}^{1},\hat{x}_{i}^{2},...,\hat{x}_{i}^{K}\\}$
. Ultimately, each payment  $\hat{x}_{i}\in D^{A}$ resulted in  5,588 encoded
dimensions, while each payment  $\hat{x}_{i}\in D^{B}$ in  7,486 encoded
dimensions.

Figure 4: Journal entry distribution for distinct activities

$A_{1},A_{2},...,A_{\tau}$

over progressing experiences (top) and target department decaying scenarios

$A_{S1}^{\prime},A_{S2}^{\prime},A_{S3}^{\prime}$

(bottom).

Continual Audit Scenarios:  We create a stream of $N$ experiences denoted by
$\\{E\\}_{i=1}^{M}$ where each experience exhibits its own dataset  $D_{i}$ .
For each $D_{i}$ a number of  $\rho\cdot\frac{\eta}{N}$ pre-processed payments
is randomly sampled from each $\hat{x}_{i}\sim A^{*}_{k}$, where
$\rho\in[0.9;1.0]$ and $k=1,..,\tau$. In a real-world audit setup the
experiences may correspond to distinct time periods, e.g., a financial quarter
or year. Furthermore, we assume that the most recent experience $E_{M}$
defines the in-scope period of the audit with journal entry data $D_{N}$. In
this initial work, we evaluate two scenarios observable when examining journal
entry data in financial audits, namely (i) discontinued operations and (ii)
anomalous postings. Figure 4 illustrates the distribution of sampled journal
entries for distinct activities over progressing experiences.

(a)

Sequential Fine-Tuning (SFT)

(b)

Elastic Weight Consolidation (EWC)

(c)

Experience Replay (ER)

Figure 5: Reconstruction losses $\mathcal{L}_{Rec}$ of the $\tau=10$
departments in dataset $D_{A}$ over a stream of

$\\{E\\}_{i=1}^{M}$

experiences for distinct CL setups. The dashed blue line corresponds to
reconstruction loss of the linearly decaying target department

$A^{\prime}$

. It can be observed that ER results in a significantly reduced number of
false-positives over the progressing experiences.

The first discontinued operations scenario, refers to a situation in which an
organizational activity or department is discontinued, e.g., due to a shut
down a divestiture. Thereby, the number of corresponding journal entries
vanishes over time. Using stationary deep-learning enabled audit techniques
such a scenario regularly results in a high number of false-positive alerts
within the in-scope audit period. To establish such a scenario a to be
discontinued department  $A^{\prime}\in\\{A^{*}_{k}\\}_{k=1}^{\tau}$ is
selected. Subsequently, a decaying number of journal entries is randomly
sampled from  $A^{\prime}$ with progressing experiences. We investigated three
discontinuation scenarios, resulting in a (i) linear (  $A_{S1}^{\prime}$ ),
(ii) exponential (  $A_{S2}^{\prime}$ ), and (iii) instant (
$A_{S3}^{\prime}$ ) reduction of journal entries.

The second anomalous postings scenario, refers to a situation in which
‘anomalous’ journal entries are posted within the in-scope period of an audit,
e.g., due to error or fraudulent activities. Using stationary deep-learning
enabled audit techniques such a scenario regularly results in an increased
number of undetected false-negative observations within the in-scope audit
period. To establish such a scenario, a small number of $\alpha_{1}$ local (
$A_{P1}$ ) and $\alpha_{2}$ global (  $A_{P2}$ ) accounting anomalies is
injected into the dataset $D_{N}$ of the last experience.555To create both
classes of anomalies, we built upon the Faker project, which is publicly
available via the following URL: https://github.com/joke2k/faker. For both
classes of of injected anomalies we set $\alpha_{1}=10$ and $\alpha_{2}=10$ in
all our experiments.

Autoencoder Learning Setup: The encoder $q_{\psi}$ and a decoder $p_{\theta}$
network use Leaky-ReLU activation functions (Xu et al. 2015) with scaling
factor $\alpha=0.4$ except in the bottleneck (last) layer where tanh
linearities (non-linearities) are applied. Table 1 depicts the architectural
details of the models used in our experiments. Both AEN parameter sets $\psi$
and $\phi$ are initialized as described by Glorot and Bengio (2010). The
models are trained for 500 epochs, using a batch size of $b=128$ journal
entries, and early stopping once the loss converges. We use Adam optimization
(Kingma and Ba 2014) with $\beta_{1}=0.9$, $\beta_{2}=0.999$. Furthermore, we
compute the binary cross-entropy error of a given encoded journal entry
$\hat{x}_{i}$, as defined by
$\mathcal{L}_{Rec}(\tilde{x}_{i},\hat{x}_{i})=\frac{1}{N}\sum_{i=1}^{N}\tilde{x}_{i}\cdot\log(\hat{x}_{i})+(1-\tilde{x}_{i})\cdot\log(1-\hat{x}_{i})$
, where  $\tilde{x}_{i}=p_{\phi}(q_{\psi}(\hat{x}_{i}))$ denotes the i-th
reconstructed entry.

Table 1: Number of neurons per layer $\eta$ of the encoder $q_{\psi}$ and decoder $p_{\phi}$ network that constitute the distinct AEN architectures used in our experiments. Net |  $\eta$ = 1 | 2 | 3 | 4 | 5 | 6 | 7
---|---|---|---|---|---|---|---
$q_{\psi}(z|\hat{x})$ | 128 | 64 | 32 | 16 | 8 | 4 | 2
$p_{\phi}(\hat{x}^{\prime}|z)$ | 2 | 4 | 8 | 16 | 32 | 64 | 128

Continual Learning Setup: We set the number of experiences  $N=10$ in all our
experiments. To find optimal parameters in EWC and ER we conducted a grid
search of the hyper-parameter space. For EWC we found that $\lambda=50$
preserves an optimal degree of plasticity. In ER learning we set $N_{B}=500$
which seemed sufficient for both datasets. We reset the optimizer parameters
upon each experience to avoid past experience information transfer through the
optimizer state. We used Avalanche (Lomonaco et al. 2021) to implement our
experiments and published the code on GitHub. 666
https://github.com/GitiHubi/deepContinualAuditing

## Experimental Results and Discussion

In this section, we present the experimental continual learning results for
both continual audit scenarios, namely (i) discontinued operations and (ii)
anomalous postings.

The experimental results of the discontinued operations scenario are presented
in Tab. 2. For both datasets $D^{A}$ and $D^{B}$ it can be observed that the
average reconstruction loss difference of the target department $A^{\prime}$
and the department exhibiting the highest loss $\tilde{A}$, as defined by
$\Delta_{FP}:=\mathcal{L}_{Rec}(A_{S}^{\prime})-\mathcal{L}_{Rec}(\tilde{A})$
. A negative (positive) $\Delta_{FP}$ indicates a reduction (an increase) of
false-positive alerts originating from the decaying target department within
the in-scope audit period $E_{M}$. Thereby, JEL and SEL provide a lower- and
upper-bound on the false-positive alerts. The obtained results demonstrate
that CL mitigates the risk of false-positive alerts while saving data and
computational resources. When comparing SFT, EWC, and ER for the decaying
setups, as also illustrated in Fig. 5, the best performance is achieved by ER.
Thereby, the exponential $A_{S2}^{\prime}$ and instant $A_{S3}^{\prime}$ decay
results in a more challenging CL setup.

Table 2: Average reconstruction loss $\mathcal{L}_{Rec}$ difference
$\Delta_{FP}$ at $E_{M}$ between the decaying target department and the
department with the highest loss in both datasets $D^{A}$ and $D^{B}$.
Discontinued Operations - Philadelphia Payments $D^{A}$
---
Method | $\Delta_{FP}(A_{S1}^{\prime})$ | $\Delta_{FP}(A_{S2}^{\prime})$ | $\Delta_{FP}(A_{S3}^{\prime})$
SEL | 1.45 $\pm$ 0.115 | 1.40 $\pm$ 0.064 | 1.45 $\pm$ 0.062
JEL | -1.36 $\pm$ 0.047 | -1.36 $\pm$ 0.047 | -1.36 $\pm$ 0.047
SFT | 0.56 $\pm$ 0.112 | 1.39 $\pm$ 0.401 | 1.82 $\pm$ 0.284
EWC | 0.91 $\pm$ 0.400 | 1.31 $\pm$ 0.313 | 2.03 $\pm$ 0.218
ER |  -1.08 $\pm$ 0.371 |  0.02 $\pm$ 0.539 |  0.02 $\pm$ 0.089
Stdv. $\pm$ originates from parameter initialization using five random seeds.
Discontinued Operations - Chicago Payments $D^{B}$
---
Method | $\Delta_{FP}(A_{S1}^{\prime})$ | $\Delta_{FP}(A_{S2}^{\prime})$ | $\Delta_{FP}(A_{S3}^{\prime})$
SEL | 0.66 $\pm$ 0.101 | 0.66 $\pm$ 0.053 | 0.61 $\pm$ 0.099
JEL | -0.96 $\pm$ 0.153 | -0.96 $\pm$ 0.153 | -0.96 $\pm$ 0.153
SFT | 0.78 $\pm$ 0.268 | 1.71 $\pm$ 1.207 | 1.35 $\pm$ 0.213
EWC | 1.18 $\pm$ 0.168 | 1.12 $\pm$ 0.184 | 1.10 $\pm$ 0.230
ER |  -0.05 $\pm$ 1.514 |  -0.50 $\pm$ 0.192 |  -0.19 $\pm$ 0.243
Stdv. $\pm$ originates from parameter initialization using five random seeds.

The experimental results of the anomalous postings scenario and a linear
decaying target department are presented in Tab. 3. For both datasets $D^{A}$
and $D^{B}$ it can be observed that SEL, the average reconstruction error
$\mathcal{L}_{Rec}$ of the decaying target department $A^{\prime}$ and the
local anomalies $A_{P1}$ exhibit a similar error magnitude within the in-scope
audit period $E_{M}$. Such a scenario increases the risk of not detecting
local anomalies and might result in false-negative decisions. The obtained
results demonstrate that CL mitigates this risk by forward transfer of
knowledge from previous experiences. This observation is also reflected by the
loss difference
$\Delta_{FN}:=\mathcal{L}_{Rec}(A^{\prime})-\mathcal{L}_{Rec}(A_{P1})$ of the
target department $A^{\prime}$ and the local anomalies $A_{P1}$. When
comparing SFT, EWC, and ER, ER achieves the best performance.

Table 3: Average reconstruction loss $\mathcal{L}_{Rec}$ at $E_{M}$ of the
decaying target department $A^{\prime}$ and the injected local $A_{P1}$ and
global $A_{P2}$ anomalies in both datasets $D^{A}$ and $D^{B}$. Anomalous
Postings - Philadelphia Payments $D^{A}$
---
Method | $\mathcal{L}_{Rec}(A^{\prime})\downarrow$ | $\mathcal{L}_{Rec}(A_{P1})\uparrow$ | $\mathcal{L}_{Rec}(A_{P2})\uparrow$
SEL | 6.59 $\pm$ 0.125 | 7.03 $\pm$ 0.130 | 12.21 $\pm$ 0.208
JEL | 2.04 $\pm$ 0.042 | 36.71 $\pm$ 5.948 | 18.12 $\pm$ 7.399
SFT | 3.87 $\pm$ 0.141 | 34.10 $\pm$ 6.025 | 12.99 $\pm$ 2.286
EWC | 4.25 $\pm$ 0.339 | 29.97 $\pm$ 4.761 | 13.22 $\pm$ 1.145
ER |  2.24 $\pm$ 0.292 |  47.69 $\pm$ 6.482 |  16.64 $\pm$ 5.176
Stdv. $\pm$ originates from parameter initialization using five random seeds.
Anomalous Postings - Chicago Payments $D^{B}$
---
Method | $\mathcal{L}_{Rec}(A^{\prime})\downarrow$ | $\mathcal{L}_{Rec}(A_{P1})\uparrow$ | $\mathcal{L}_{Rec}(A_{P2})\uparrow$
SEL | 4.27 $\pm$ 0.097 | 3.55 $\pm$ 0.171 | 4.53 $\pm$ 0.053
JEL | 1.30 $\pm$ 0.145 | 16.75 $\pm$ 4.036 | 4.71 $\pm$ 0.454
SFT | 3.01 $\pm$ 0.288 | 13.03 $\pm$ 3.845 | 3.93 $\pm$ 0.169
EWC | 3.42 $\pm$ 0.185 | 12.20 $\pm$ 1.958 | 5.18 $\pm$ 1.709
ER |  2.26 $\pm$ 1.154 |  22.23 $\pm$ 5.439 |  6.25 $\pm$ 1.857
Stdv. $\pm$ originates from parameter initialization using five random seeds.

## Conclusion

In this work, we proposed the first CL inspired framework to detect accounting
anomalies in the context of financial statement audits. Based on two
deliberately designed audit scenarios, we demonstrated the ability of our
method to continuously learn data distribution shifts observable in datasets
of real-world city payments. Our experimental results provide initial evidence
that CL enabled anomaly detection improve stationary baseline detection
techniques in terms of false-positive alerts and false-negative decisions.

## References

  * ACFE (2020) ACFE. 2020.  _Report to the Nations - Global Study on Occupational Fraud and Abuse_.  Association of Certified Fraud Examiners (ACFE).
  * AICPA (2002) AICPA. 2002.  _Statement on Auditing Standards No. 99.: Consideration of Fraud in a Financial Statement Audit_.
  * Argyrou (2012) Argyrou, A. 2012.  Auditing Journal Entries Using Self-Organizing Map.  In _Proceedings of the Eighteenth Americas Conference on Information Systems_.
  * Argyrou (2013) Argyrou, A. 2013.  Auditing Journal Entries Using Extreme Vale Theory.  In _Proceedings of the 21st European Conference on Information Systems_.
  * Baader and Krcmar (2018) Baader, G.; and Krcmar, H. 2018.  Reducing False Positives in Fraud Detection: Combining the Red Flag Approach with Process Mining.  _International Journal of Accounting Information Systems_ , 31: 1–16.
  * Bay et al. (2006) Bay, S.; Kumaraswamy, K.; Anderle, M. G.; Kumar, R.; Steier, D. M.; Blvd, A.; and Jose, S. 2006.  Large Scale Detection of Irregularities in Accounting Data.  In _Sixth International Conference on Data Mining_ , 75–86. IEEE.
  * Benford (1938) Benford, F. 1938.  The Law of Anomalous Numbers.  _Proceedings of the American Philosophical Society_ , 78(4): 551–572.
  * Bolton and Hand (2002) Bolton, R. J.; and Hand, D. J. 2002.  Statistical fraud detection: A review.  _Statistical science_ , 235–249.
  * Breunig et al. (2000) Breunig, M. M.; Kriegel, H.-P.; Ng, R. T.; and Sander, J. 2000.  LOF: Identifying Density-based Local Outliers.  In _Proceedings of the 2000 ACM SIGMOD international conference on Management of data_ , 93–104.
  * CAQ (2008) CAQ. 2008.  _Practice Aid for Testing Journal Entries and Other Adjustments Pursuant to AU Section 316_.  Center for Audit Quality (CAQ).
  * Chaudhry et al. (2018) Chaudhry, A.; Ranzato, M.; Rohrbach, M.; and Elhoseiny, M. 2018.  Efficient Lifelong Learning with A-GEM.  _arXiv preprint arXiv:1812.00420_.
  * Chaudhry et al. (2019) Chaudhry, A.; Rohrbach, M.; Elhoseiny, M.; Ajanthan, T.; Dokania, P. K.; Torr, P. H.; and Ranzato, M. 2019.  Continual learning with tiny episodic memories.
  * Chen and Liu (2018) Chen, Z.; and Liu, B. 2018.  Lifelong machine learning.  _Synthesis Lectures on Artificial Intelligence and Machine Learning_ , 12(3): 1–207.
  * Cho et al. (2020) Cho, S.; Vasarhelyi, M. A.; Sun, T.; and Zhang, C. 2020.  Learning from Machine Learning in Accounting and Assurance.  _Journal of Emerging Technologies in Accounting_ , 17(1): 1–10.
  * Coderre (2009) Coderre, D. 2009.  _Computer-aided Fraud Prevention and Detection: A Step-by-step Guide_.  John Wiley & Sons.
  * Debreceny and Gray (2010) Debreceny, R. S.; and Gray, G. L. 2010.  Data Mining Journal Entries for Fraud Detection: An Exploratory Study.  _International Journal of Accounting Information Systems_ , 11(3): 157–181.
  * Delange et al. (2021) Delange, M.; Aljundi, R.; Masana, M.; Parisot, S.; Jia, X.; Leonardis, A.; Slabaugh, G.; and Tuytelaars, T. 2021.  A Continual Learning Survey: Defying Forgetting in Classification Tasks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Díaz-Rodríguez et al. (2018) Díaz-Rodríguez, N.; Lomonaco, V.; Filliat, D.; and Maltoni, D. 2018.  Don’t Forget, There is More than Forgetting: New Metrics for Continual Learning.  _arXiv preprint arXiv:1810.13166_.
  * Fay and Negangard (2017) Fay, R.; and Negangard, E. M. 2017.  Manual Journal Entry Testing: Data Analytics and the Risk of Fraud.  _Journal of Accounting Education_ , 38: 37–49.
  * French (1999) French, R. M. 1999.  Catastrophic Forgetting in Connectionist Networks.  _Trends in Cognitive Sciences_ , 3(4): 128–135.
  * Glorot and Bengio (2010) Glorot, X.; and Bengio, Y. 2010.  Understanding the Difficulty of Training Deep Feedforward Neural Networks.  In _Proceedings of the thirteenth International Conference on Artificial Intelligence and Statistics_ , 249–256.
  * Grabski, Leech, and Schmidt (2011) Grabski, S. V.; Leech, S. A.; and Schmidt, P. J. 2011.  A Review of ERP Research: A Future Agenda for Accounting Information Systems.  _Journal of Information Systems_ , 25(1): 37–78.
  * Hawkins et al. (2002) Hawkins, S.; He, H.; Williams, G.; and Baxter, R. 2002.  Outlier Detection using Replicator Neural Networks.  In _International Conference on Data Warehousing and Knowledge Discovery_ , 170–180. Springer.
  * Hinton and Salakhutdinov (2006) Hinton, G. E.; and Salakhutdinov, R. R. 2006.  Reducing the Dimensionality of Data with Neural Networks.  _science_ , 313(5786): 504–507.
  * Hu et al. (2020) Hu, H.; Sun, T.; Vasarhelyi, M. A.; and Zhang, M. 2020.  A Machine Learning Approach of Measuring Audit Quality: Evidence From China.  _Available at SSRN 3732563_.
  * IFAC (2009) IFAC. 2009.  _International Standards on Auditing 240, The Auditor’s Responsibilities Relating to Fraud in an Audit of Financial Statements_.
  * Isele and Cosgun (2018) Isele, D.; and Cosgun, A. 2018.  Selective Experience Replay for Lifelong Learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Islam et al. (2010) Islam, A. K.; Corney, M.; Mohay, G.; Clark, A.; Bracher, S.; Raub, T.; and Flegel, U. 2010.  Fraud Detection in ERP Systems using Scenario Matching.  _IFIP Advances in Information and Communication Technology_ , 330: 112–123.
  * Jans, Alles, and Vasarhelyi (2014) Jans, M.; Alles, M. G.; and Vasarhelyi, M. A. 2014.  A Field Study on the Use of Process Mining of Event Logs as an Analytical Procedure in Auditing.  _The Accounting Review_ , 89(5): 1751–1773.
  * Jans, Lybaert, and Vanhoof (2010) Jans, M.; Lybaert, N.; and Vanhoof, K. 2010.  Internal Fraud Risk Reduction: Results of a Data Mining Case Study.  _International Journal of Accounting Information Systems_ , 11(1): 17–41.
  * Jans et al. (2011) Jans, M.; Van Der Werf, J. M.; Lybaert, N.; and Vanhoof, K. 2011.  A Business Process Mining Application for Internal Transaction Fraud Mitigation.  _Expert Systems with Applications_ , 38(10): 13351–13359.
  * Khan et al. (2014) Khan, R.; Clark, A.; Mohay, G.; and Suriadi, S. 2014.  Detecting Fraud Using Transaction Frequency Data.  _Information Technology in Industry_ , 2(3): 79–88.
  * Khan et al. (2009) Khan, R.; Corney, M.; Clark, A.; and Mohay, G. 2009.  A Role Mining inspired Approach to Representing user Behaviour in ERP Systems.  In _Proceedings of Asia Pacific Industrial Engineering & Management Systems Conference 2009_, 2541–2552. APIEMS Society.
  * Khan et al. (2010) Khan, R.; Corney, M.; Clark, A.; and Mohay, G. 2010.  Transaction Mining for Fraud Detection in ERP Systems.  _Industrial Engineering and Management Systems_ , 9(2): 141–156.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  ADAM: A Method for Stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kirkpatrick et al. (2017) Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; et al. 2017.  Overcoming Catastrophic Forgetting in Neural Networks.  _Proceedings of the National Academy of Sciences_ , 114(13): 3521–3526.
  * LeCun, Bengio, and Hinton (2015) LeCun, Y.; Bengio, Y.; and Hinton, G. 2015.  Deep learning.  _nature_ , 521(7553): 436.
  * Lee and Lee (2020) Lee, C. S.; and Lee, A. Y. 2020.  Clinical Applications of Continual Learning Machine Learning.  _The Lancet Digital Health_ , 2(6): e279–e281.
  * Lesort, Caccia, and Rish (2021) Lesort, T.; Caccia, M.; and Rish, I. 2021.  Understanding Continual Learning Settings with Data Distribution Drift Analysis.  _arXiv preprint arXiv:2104.01678_.
  * Li and Hoiem (2017) Li, Z.; and Hoiem, D. 2017.  Learning Without Forgetting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 40(12): 2935–2947.
  * Lomonaco et al. (2021) Lomonaco, V.; Pellegrini, L.; Cossu, A.; Carta, A.; Graffieti, G.; Hayes, T. L.; Lange, M. D.; Masana, M.; Pomponi, J.; van de Ven, G.; Mundt, M.; She, Q.; Cooper, K.; Forest, J.; Belouadah, E.; Calderara, S.; Parisi, G. I.; Cuzzolin, F.; Tolias, A.; Scardapane, S.; Antiga, L.; Amhad, S.; Popescu, A.; Kanan, C.; van de Weijer, J.; Tuytelaars, T.; Bacciu, D.; and Maltoni, D. 2021\.  Avalanche: an End-to-End Library for Continual Learning.  In _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition_ , 2nd Continual Learning in Computer Vision Workshop.
  * Lopez-Paz and Ranzato (2017) Lopez-Paz, D.; and Ranzato, M. 2017.  Gradient Episodic Memory for Continual Learning.  _Advances in Neural Information Processing Systems_ , 30: 6467–6476.
  * McCloskey and Cohen (1989) McCloskey, M.; and Cohen, N. J. 1989.  Catastrophic Interference in connectionist Networks: The Sequential Learning Problem.  In _Psychology of Learning and Motivation_ , volume 24, 109–165. Elsevier.
  * McGlohon et al. (2009) McGlohon, M.; Bay, S.; Anderle, M. G. M.; Steier, D. M.; and Faloutsos, C. 2009\.  SNARE: A Link Analytic System for Graph Labeling and Risk Detection.  In _KDD’09: 15th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 1265–1273.
  * Mendez and Eaton (2021) Mendez, J. A.; and Eaton, E. 2021.  Lifelong Learning of Compositional Structures.  In _International Conference on Learning Representations_.
  * Nonnenmacher and Gómez (2021) Nonnenmacher, J.; and Gómez, J. M. 2021.  Unsupervised Anomaly Detection for Internal Auditing: Literature Review and Research Agenda.  _The International Journal of Digital Accounting Research_ , 21(27): 1–22.
  * Nonnenmacher et al. (2021) Nonnenmacher, J.; Kruse, F.; Schumann, G.; and Marx Gómez, J. 2021.  Using Autoencoders for Data-Driven Analysis in Internal Auditing.  In _Proceedings of the 54th Hawaii International Conference on System Sciences_ , 5748.
  * Parisi et al. (2019) Parisi, G. I.; Kemker, R.; Part, J. L.; Kanan, C.; and Wermter, S. 2019.  Continual lifelong learning with neural networks: A review.  _Neural Networks_ , 113: 54–71.
  * PwC (2020) PwC. 2020.  _Fighting Fraud: A never-ending Battle, The Global Economic Crime Survey 2020_.  PricewaterhouseCoopers LLP.
  * Rebuffi et al. (2017) Rebuffi, S.-A.; Kolesnikov, A.; Sperl, G.; and Lampert, C. H. 2017.  ICARL: Incremental Classifier and Representation Learning.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2001–2010.
  * Rolnick et al. (2018) Rolnick, D.; Ahuja, A.; Schwarz, J.; Lillicrap, T. P.; and Wayne, G. 2018.  Experience Replay for Continual Learning.  _arXiv preprint arXiv:1811.11682_.
  * Rusu et al. (2016) Rusu, A. A.; Rabinowitz, N. C.; Desjardins, G.; Soyer, H.; Kirkpatrick, J.; Kavukcuoglu, K.; Pascanu, R.; and Hadsell, R. 2016.  Progressive Neural Networks.  _arXiv preprint arXiv:1606.04671_.
  * Schreyer, Sattarov, and Borth (2021) Schreyer, M.; Sattarov, T.; and Borth, D. 2021.  Multi-view Contrastive Self-Supervised Learning of Accounting Data Representations for Downstream Audit Tasks.  In _Proceedings of the International Conference on Artificial Intelligence (ICAIF)’21_. Association of Computing Machinery (ACM).
  * Schreyer et al. (2017) Schreyer, M.; Sattarov, T.; Borth, D.; Dengel, A.; and Reimer, B. 2017.  Detection of Anomalies in Large Scale Accounting Data using Deep Autoencoder Networks.  _arXiv preprint arXiv:1709.05254_.
  * Schreyer et al. (2020) Schreyer, M.; Sattarov, T.; Gierbl, A. S.; Reimer, B.; and Borth, D. 2020.  Learning Sampling in Financial Statement Audits using Vector Quantised Variational Autoencoder Neural Networks.  In _Proceedings of the International Conference on Artificial Intelligence (ICAIF)’20_. Association of Computing Machinery (ACM).
  * Schultz and Tropmann-Frick (2020) Schultz, M.; and Tropmann-Frick, M. 2020.  Autoencoder Neural Networks versus External Auditors: Detecting Unusual Journal Entries in Financial Statement Audits.  In _Proceedings of the 53rd Hawaii International Conference on System Sciences_.
  * SEOW, Pan, and Suwardy (2016) SEOW, P. S.; Pan, G.; and Suwardy, T. 2016.  Data Mining Journal Entries for Fraud Detection: A Replication of Debreceny and Gray’s (2010) Techniques.  _Journal of Forensic and Investigative Accounting_ , 8(3): 501.
  * Shin et al. (2017) Shin, H.; Lee, J. K.; Kim, J.; and Kim, J. 2017.  Continual Learning with Deep Generative Replay.  _arXiv preprint arXiv:1705.08690_.
  * Stephan, Lahann, and Fettke (2021) Stephan, S.; Lahann, J.; and Fettke, P. 2021.  A Case Study on the Application of Process Mining in Combination with Journal Entry Tests for Financial Auditing.  In _Proceedings of the 54th Hawaii International Conference on System Sciences_ , 5718.
  * Sun (2019) Sun, T. 2019.  Applying Deep Learning to Audit Procedures: An Illustrative Framework.  _Accounting Horizons_ , 33(3): 89–109.
  * Thiprungsri and Vasarhelyi (2011) Thiprungsri, S.; and Vasarhelyi, M. A. 2011.  Cluster Analysis for Anomaly Detection in Accounting Data: An Audit Approach.  _International Journal of Digital Accounting Research_ , 11.
  * Toneva et al. (2018) Toneva, M.; Sordoni, A.; Combes, R. T. d.; Trischler, A.; Bengio, Y.; and Gordon, G. J. 2018.  An Empirical Study of Example Forgetting During Deep Neural Network Learning.  _arXiv preprint arXiv:1812.05159_.
  * Vasarhelyi, Alles, and Kogan (2004) Vasarhelyi, M. A.; Alles, M. G.; and Kogan, A. 2004.  Principles of Analytic Monitoring for Continuous Assurance.  _Journal of Emerging Technologies in Accounting_ , 1(1): 1–21.
  * Vasarhelyi and Halper (1991) Vasarhelyi, M. A.; and Halper, F. B. 1991.  The Continuous Audit of Online Systems.  In _Auditing: A Journal of Practice and Theory_. Citeseer.
  * Wells (2017) Wells, J. T. 2017.  _Corporate Fraud Handbook: Prevention and Detection_.  John Wiley & Sons.
  * Werner (2016) Werner, M. 2016.  Process Model Representation Layers for Financial Audits.  In _2016 49th Hawaii International Conference on System Sciences (HICSS)_ , 5338–5347. IEEE.
  * Werner and Gehrke (2015) Werner, M.; and Gehrke, N. 2015.  Multilevel Process Mining for Financial Audits.  _IEEE Transactions on Services Computing_ , 8(6): 820–832.
  * Xu et al. (2015) Xu, B.; Wang, N.; Chen, T.; and Li, M. 2015.  Empirical Evaluation of Rectified Activations in Convolutional Network.  _arXiv preprint arXiv:1505.00853_.
  * Zenke, Poole, and Ganguli (2017) Zenke, F.; Poole, B.; and Ganguli, S. 2017.  Continual Learning through Synaptic Intelligence.  In _International Conference on Machine Learning_ , 3987–3995. PMLR.
  * Zhai, Liu, and Sun (2019) Zhai, M.; Liu, H.; and Sun, F. 2019.  Lifelong Learning for Scene Recognition in Remote Sensing Images.  _IEEE Geoscience and Remote Sensing Letters_ , 16(9): 1472–1476.
  * Zupan et al. (2020) Zupan, M.; Budimir, V.; Letinic, S.; et al. 2020.  Journal Entry Anomaly Detection Model.  _Intelligent Systems in Accounting, Finance and Management_ , 27(4): 197–209.


marginparsep has been altered.
topmargin has been altered.
marginparpush has been altered.
The page layout violates the ICML style. Please do not change the page layout,
or include packages like geometry, savetrees, or fullpage, which change it for
you. We’re not able to reliably undo arbitrary changes to the style. Please
remove the offending package(s), or layout-changing commands and try again.

NuCLR: Nuclear Co-Learned Representations

Ouail Kitouni* 1  Niklas Nolte* 1  Sokratis Trifinopoulos 1  Subhash
Kantamneni 1  Mike Williams 1

††footnotetext: *Equal contribution 1Institute for Artificial Intelligence and
Fundamental Interactions (IAIFI), MIT, USA. Correspondence to: Mike Williams
<EMAIL_ADDRESS>
Accepted after peer-review at the 1st workshop on Synergy of Scientific and
Machine Learning Modeling, SynS & ML ICML, Honolulu, Hawaii, USA. July, 2023.
Copyright 2023 by the author(s).

###### Abstract

We introduce Nuclear Co-Learned Representations (NuCLR), a deep learning model
that predicts various nuclear observables, including binding and decay
energies, and nuclear charge radii. The model is trained using a multi-task
approach with shared representations and obtains state-of-the-art performance,
achieving levels of precision that are crucial for understanding fundamental
phenomena in nuclear (astro)physics. We also report an intriguing finding that
the learned representations of NuCLR exhibit the prominent emergence of
crucial aspects of the nuclear shell model, namely the shell structure,
including the well-known magic numbers, and the Pauli Exclusion Principle.
This suggests that the model is capable of capturing the underlying physical
principles, and that our approach has the potential to offer valuable insights
into nuclear theory.

##  1 Introduction

The nucleus is the incredibly small and dense region at the center of an atom,
consisting of protons and neutrons bound together by the strong nuclear force.
Despite having been discovered over a century ago—and the fundamental theory
of the strong nuclear force (quantum chromodynamics, QCD) having been
discovered 50 years ago—we still lack a precise quantitative understanding of
nuclear physics. In principle all nuclear properties are calculable from the
theory of QCD, though in practice such calculations are intractable for all
but the smallest few nuclei.

Qualitatively, a major breakthrough in understanding nuclei came just after
World War II with the development of the nuclear shell model by Goeppert-Mayer
and Jensen (Nobel Prize in Physics, 1963), which analogous to the atomic shell
model describes the structure of the nucleus in terms of quantum energy
states. A key component of this model is the Pauli Exclusion Principle, that
no two identical fermions, e.g. nucleons (protons or neutrons), can occupy the
same quantum state. Since nucleons have spin 1/2, two such particles can
occupy the same energy state, one with spin up and the other with spin down,
leading to even numbers of protons and neutrons being preferred (more strongly
bound). As in the atomic shell model, the most stable states are those with
filled shells. Therefore, when adding nucleons to a nucleus, there are magic
numbers where the binding energy of the next nucleon is substantially less
than the previous one, i.e. there are configurations that are much more
tightly bound due to having a filled shell. The shells for protons and
neutrons are filled independently since they are distinguishable particles;
however, the number of protons does affect the neutron quantum states and vice
versa. There are well known magic numbers such as 2, 8, 20, 28, 50, and 82 for
both proton and neutron number, $Z$ and $N$, respectively, and in addition
$N=126$.

Figure 1: Most important principal components of the neutron embedding
representations from (left) early and (right) late in the training. The left
panel shows that the most crucial aspects of the nuclear shell model, namely
the shell structure, including the magic numbers, and the Pauli Exclusion
Principle, arise already early in the training. (The Pauli principle is
evident with the even (odd) numbers represented as negative (positve) values
of PC2. The ends of the even chains of numbers are the magic numbers where
each nuclear shell become full.) In the right panel, the even-odd split now
occurs in PC dimension 4, hence is not shown. The shell structure has grown
into 3-dimensional spirals, with the largest 4 magic numbers all occurring at
local maxima in PC2 and each shell represented as one revolution around an
approximately conic surface. Interpreting this spiral structure is ongoing
work.

A major goal of the modern nuclear physics community is to accurately predict
nuclear properties, among which binding energies and charge radii are some of
the most important. Traditionally, analytical models have been used, motivated
by the known physics—but simplified to make the calculations tractable—and
augmented with empirical input to improve their agreement with data Goriely et
al. (2001); Bender et al. (2003); Geng et al. (2005); Möller et al. (2012);
Wang et al. (2014).111With an increase in computational power nuclear ab
initio methods have recently gained momentum Navratil et al. (2009); Hagen et
al. (2014); Ekström et al. (2015); Novario et al. (2023). The most accurate of
these models, referred to as WS4 Wang et al. (2014), is able to predict
nuclear binding energies and charge radii with a precision of about 300 keV
and 0.02 fm, respectively. More recently, machine learning (ML) models have
received some attention due to their superior performance potential. Thus far,
most ML models were designed for a _single-task learning_ (STL) purpose,
namely predicting binding energies Niu & Liang (2018); Wu et al. (2021); Niu &
Liang (2022); Wu et al. (2022) or charge radii Utama et al. (2016); Wu et al.
(2020); Dong et al. (2022); Ma & Zhang (2022). An exception is the kernel
ridge regression model of Wu et al. (2022) which utilizes _multi-task
learning_ (MTL) to simultaneously predict both binding and separation
energies, achieving a world-leading precision of about 140 keV; n.b. this
model uses the WS4 predictions as important inputs. ML models have yet to
substantially improve upon analytic-model predictions for charge radii. We
note that the precision needed for understanding fundamental phenomena in
nuclear (astro)physics, are below 100 keV for binding energies and 0.01 fm for
charge radii;e.g. for $r$-process nucleosynthesis see Martin et al. (2016);
Mumpower et al. (2016).

A major drawback of the previous ML-based approaches is that they lack
interpretability. The nuclear shell model is intuitive and surprisingly
accurate given its simplicity. We propose an ML-based approach designed to
learn a task-independent representation of the nucleus, analogous to the
human-learned nuclear shell model picture. To decouple the nuclear
representation from task-specific model features, we train to predict a
variety of nuclear properties in an MTL setting, all based on the same nuclear
representation model, referred to as Nuclear Co-Learned Representations
(NuCLR). By exploiting all information available about nuclei, we show that we
are able to build a more meaningful model and improve prediction quality that
achieves state-of-the-art performance. More importantly, as shown in Fig. 1 we
can clearly identify the most important aspects of the nuclear shell model in
the latent representations of the model that are shared between all tasks,
such as the shell structure, including the well-known magic numbers, and the
Pauli Exclusion Principle.

##  2 More Tasks, More Information

Improving generalization of a prediction task can be achieved in many ways,
including by obtaining more or more precise data. However, collecting such
data is often expensive, and in the case of nuclear data technologically
challenging or even infeasible. Another avenue is exploiting the joint
information between different known and measured properties of the nucleus to
increase the effective amount of data available for any specific prediction
task. This can be achieved by training jointly on all tasks. To exploit data
correlations over multiple tasks, a prediction model needs to process data
through dependent channels. For neural networks, that most often means sharing
weights and layers between tasks. To what degree a model should share
processing channels between tasks depends on how similar those tasks are. In
the limit of tasks with no joint information, there is no evident benefit to
joint training.

In the MTL literature there exist many ways to either manually or
automatically control information sharing Crawshaw (2020). In order to
introduce and motivate choices made for this MTL task, we first show proof of
concept with a toy model. The task is to predict the result of the following
arithmetic binary functions of two input real numbers: $a+b$, $|a-b|$,
$(a+b)^{3/2}$, $\log(a+b+1)$, and exp$(-\sqrt{a+b}/5)$. The input numbers are
treated like symbols or tokens in a language task by embedding them randomly
into a high-dimensional space. Once embedded, they lose all their relational
properties. The representation vectors are trainable parameters during the
learning process that allows the architecture to arrange them in a beneficial
way. Schematically, the architecture for this toy-model regression processes
inputs is as follows: embed both numbers, concatenate these embeddings, then
send the resulting vector through a stack of residual blocks.

In the regime of limited data, we find that using multiple tasks improves
generalization on all tasks simultaneously. Figure 2 shows the performance
obtained when training on individual targets (single-task, ST) versus joint
training on all tasks at once (multi-task, MT), where the latter is superior
for all tasks. Here, the MT model has the same layout as the ST models except
that the last layer of the model has an output for each task. Note how this
architecture hard-codes a high degree of shared computation between tasks.
Regardless of the regression target, input numbers undergo the same
transformations up to the last layer.

Figure 2: For the toy-model arithmetic problem, the multi-task with embeddings
(MTE) model generalizes better than the multi task (MT) and single task (ST)
models on all tasks in the regime of limited data. Here, only 400 examples for
each task are used, 10% for training and the rest for testing.

Next, we show that far superior performance is achieved by using a different
approach to multi-tasking. Traditionally in MTL, architectures define
hyperparameters to control the degree of weight sharing for different tasks.
Taking inspiration from language models—which do implicit multi-tasking by
conditioning on the task as an additional input—we instead treat the binary
operations as trainable embeddings in the same way as the input numbers, and
concatenate all three embedding vectors for processing through the model. As
in the MT model, the last layer has one output per task. Our approach lets the
model decide to which degree tasks are similar and how computations should be
shared by moving the task embeddings closer together or farther apart in
latent space. Figure 2 shows that this model—labelled MTE, shown schematically
in Fig. 3—yields superior performance over both single-target and naive multi-
target training.

Figure 3: MTE architecture: input-data and task embeddings are concatenated,
projected, and passed through a sequence of residual blocks. Different readout
heads output predictions for each task. The fact that the same input-data
embeddings are used for all tasks encourages structure formation in the
embedding space which encodes task-independent information.

Figure 4: Residuals for NuCLR predictions compared to experimental
measurements for (left) binding energy (keV) and (right) charge radius (fm).
The vast majority of our predictions are highly precise, especially from the
$5^{\rm th}$ shell up ($N,Z>28$) where the RMS of our binding energy
predictions is only 79 keV cf. the binding energies themselves which are
$\mathcal{O}(7000)\cdot(Z+N)$ keV.

##  3 Predicting Nuclear Properties

We now use the MTE approach presented in the previous section to predict
nuclear properties. We base our training on the AME2020 dataset Wang et al.
(2021) for the energy-based quantities and on the compilation of Angeli &
Marinova (2013) for the charge radii, in both cases using all well-measured
nuclei with $Z,N>8$.222More details on the dataset, model, and validation
procedure are provided in an appendix. The only inputs are three embeddings:
proton and neutron number, $Z$ and $N$, and the task. We choose the following
nuclear properties to predict: binding energy per nucleon; charge radius;
neutron and proton separation energies, defined as the energy required to
remove one neutron or proton from the nucleus; and the energy available for
$\alpha$, $\beta$, electron-capture (EC), and $\beta$-$n$ decays. All targets
are scaled to be contained in the interval [0,1]. In addition, for binding
energies we predict the residuals relative to the semi-empirical Bethe-
Weizsäcker (BW) mass formula, a direct consequence of the famous liquid-drop
model of the nucleus Weizsacker (1935); Bethe & Bacher (1936). The BW formula
itself has a prediction precision of about $3~{}\rm MeV$, which is 30 times
worse than our target performance. Removing the BW predictions does not
greatly affect the prediction performance; however, it does affect the
interpretability (discussed later).

When evaluating our models, we must avoid prediction biases such as
correlations between the separation energies of a nucleus and the binding
energies of its neighboring nuclei, which could be used to artificially obtain
excellent performance Wu et al. (2022); e.g., it is possible to directly
obtain (and memorize) the binding energy, $B(Z,N)$, from
$B(Z,N)=B(Z,N-1)+S_{n}(Z,N)$ where $S_{n}$ denotes the neutron separation
energy. Therefore, we train many independent models (100-fold cross-
validation), where for each, we withhold 1% of the nuclei from training to be
used for validation. In addition, we also remove the separation energies of
the neighboring nuclei for each nucleus in a validation sample.

All inputs are embedded into a 1024-dimensional space. The model consists of
one linear layer that takes the concatenated embeddings in $3\times 1024$
dimensions and maps them back to 1024 dimensional space, 4 residual blocks
with 2 layers of width 1024 each, and one final output layer with one output
for each task. We use SiLU activation after each linear layer. This results in
a total of $11\,869\,194$ parameters. We train each model for 50 000 epochs
with learning rate of 0.01, weight decay of 0.01, and a batch size of 4096.
The learning rate is cosine-annealed to $10^{-5}$ during the training.

Figure 4 shows the residuals for our predictions for binding energy and charge
radius. The vast majority of our predictions are highly precise, especially
for larger nuclei. The root-mean-square (RMS) of the prediction performance of
our model on the various tasks is as follows: 130 keV for binding energy;
0.011 fm for charge radius; 130 keV and 140 keV for neutron and proton
separation energies; and 139 keV, 180 keV, 179 keV, 196 keV for the energy
available for $\alpha$, $\beta$, EC, and $\beta$-$n$ decays. We believe that
these results establish a new state of the art. In addition, we note that to
get competitive performance, previous works considered more human-designed
inputs, for instance binding energy predictions of high-precision analytical
models Niu & Liang (2022); Wu et al. (2022). We could likely improve our
results by also including such inputs, especially for smaller nuclei; however,
our focus here was to demonstrate the ability to self learn the nuclear shell
structure in an interpretable way.

##  4 Gaining Understanding via Embeddings

The deep learning literature suggests that the crucial phenomenon of
_capability emergence_ arises as one scales up the number of parameters,
dataset size, and compute across modalities, and in particular, this
phenomenon is extremely important in language models Brown et al. (2020); Wei
et al. (2022). A plausible explanation for the performance gains attributed to
this phenomenon is the emergence of specialized neural circuits and higher-
quality representations. Similarly, since we use the same nuclear embeddings
to tackle a range of tasks, we expect our training procedure to lead to the
emergence of physically-meaningful representations. Furthermore, if most of
the information stored in the embeddings lies on a low-dimensional manifold,
then the learned solution could be interpretable, allowing us to gain both
confidence in and understanding from the model.

To study the embeddings, we project them down into a low-dimensional space via
a principal component analysis (PCA). Figure 1 (left) shows the first two
principle components of the neutron embeddings at an early stage in the
training. We clearly see that the model has learned to separate the even and
odd values of $N$ (vertically on the plot). Recall that $N$ is not given to
the machine as an integer, that fact is obscured by providing $N$ as a
randomly initialized vector in a 1024-dimensional space. This clear separation
of even and odd is a striking statement that it has learned the consequences
of the Pauli Exclusion Principle. Furthermore, already at this early stage of
training we see chains of connected numbers arise with clear breaks between
the chains. The break points in the even chains are the well-known magic
numbers for $N$, i.e. these break points occur where the nuclear shells become
full. Therefore, we conclude that the earliest stages of learning correspond
to building embeddings that represent the famous nuclear shell model!

Figure 1 (right) shows the $N$ embeddings at the end of training, where the
clear even-odd split due to the Pauli principle remains. Interestingly, we see
that the chains of numbers attributed to the shell structure have evolved into
3-dimensional spirals, where the numbers from each shell are roughly
equidistant around a circle in the 2-dimensional subspace with the different
shells separated along the third PCA dimension. There are many potential
explanations for this sturcture, though more detailed study is required to
understand this structure. Nevertheless, this spiral structure appears crucial
to NuCLR being able to provide state-of-the-art precision for nuclear
predictions. Finally, we note that the final embeddings and the observable
predictions obtained both with and without the use of the imprecise BW
binding-energy predictions as inputs are similar. The primary benefit of using
the BW predictions is the clear emergence of the nuclear shell model early in
the training.

##  5 Discussion and Future Directions

NuCLR achieves state-of-the-art performance predicting nuclear observables
using a multi-task approach with shared representations. Amazingly, these
learned representations exhibit the prominent emergence of the most crucial
aspects of the nuclear shell model, suggesting the model is capable of
capturing the underlying physical principles. Future directions include more
detailed study of the embedding structures, and more generally how to
encourage embedding as much information as possible into an interpretable low-
dimensional manifold. On the nuclear-physics side, NuCLR could potentially be
used to make accurate predictions about many exciting topics in nuclear
(astro)physics, including $r$-process nucleosynthesis Burbidge et al. (1957),
the nuclear neutron skin and its consequences for the structure of neutron
stars Brown (2000); Horowitz & Piekarewicz (2001); Gandolfi et al. (2012), the
exploration of the boundaries of the nuclear landscape Erler et al. (2012),
and of exotic phenomena such as halo nucleii Nortershauser et al. (2009) and
shape coexistence Heyde et al. (1983); Wood et al. (1992), and the CKM
unitarity puzzle Seng et al. (2018); Belfatto & Trifinopoulos (2023); Seng
(2023).

## Broader Impact

A generalized view of the method proposed here is that the scientific endeavor
itself is a representation-learning problem. Nature presents us with high-
dimensional seemingly unorganized data, from which scientists attempt to find
low-dimensional representations of the relevant information. Once a low-
dimensional representation is found, a model can be built that is not only
precise but also trustworthy because we can understand how its predictions
arise. In many scientific applications, ML models can easily achieve superior
performance but they lack interpretability. Our method re-formulates the ML
approach to science in a way that is more aligned with how traditional human-
led science works. We showed that if most of the information stored in the
embeddings lies on a low-dimensional manifold, then the ML solution could be
interpretable, allowing us to gain both confidence in and understanding from
the ML model. In this way, we not only obtain more precise models—but we can
potentially learn from the machine. We believe that our approach could be
applied to any scientific problem where understanding what the ML model has
learned is desired. Finally, we do not foresee any ethical concerns associated
with our work.

## Acknowledgement

We are grateful to Yotam Soreq and Ben Ohayon for the very useful comments.
This work was supported by NSF grant PHY-2019786 (The NSF AI Institute for
Artificial Intelligence and Fundamental Interactions, http://iaifi.org/). ST
is also supported by the Swiss National Science Foundation - project n.
P500PT_203156.

## Appendix: Additional NuCLR Details

Dataset: The dataset is built from the nuclei in the AME2020 collection, which
contains both experimentally measured nuclear data and model predictions for
as-yet-unobserved nuclei. Only experimentally measured nuclei with $Z,N>8$ are
considered when evaluating model performance; however, predictions for
unobserved nuclei are used in the training, which helps in regions where there
is little experimental information. In total, about 3k nuclei are used in the
training and testing. Note that most nuclei do not have measurements for all
of the properties that we predict, e.g., only around one third of the AME2020
nuclei posses a charge-radius measurement. For more details, the AME2020
sample can be inspected interactively at https://www-
nds.iaea.org/relnsd/vcharthtml/VChartHTML.html.

Model: Figure 3 shows a schematic diagram of the model that takes as input the
embedding vectors for proton and neutron number, along with the specific
prediction task. During a forward pass, all 3 inputs (proton number, neutron
number, task ID) are embedded in a $1024$-dimensional space using a learned
embedding function. These are then concatenated to form a $3\times
1024$-dimensional vector. This vector is passed through a set of residual
blocks of size $1024$, ending in one readout linear layer that transforms the
$1024$-dimensional penultimate activation into a vector with entries for each
task. Thus, branching for multi-tasking happens on two levels: (1) the task
embedding and (2) the last-layer routing. One data point is defined as ($Z$,
$N$, Task ID), and the loss for that point considers only the task-relevant
output of the network.

Cross-Validation: We use 100-fold cross validation because neighboring nuclei
must be excluded in the training (see Wu et al. (2022), and discussion in the
main text above). For each nucleus in the validation set, all neighbor
energies are removed from the training sample, leading to $4\mbox{--}5\times$
validation set cardinality removed from the training data in each fold. This
requires that each validation set be kept small to allow for a large enough
training sample. Thus, a large number of folds are needed so that each nucleus
appears in at least one validation sample. We note that the previous state-of-
the-art model for predicting nuclear properties Wu et al. (2022) does a much
higher number of folds; they train one model per nucleus.

## References

  * Angeli & Marinova (2013) Angeli, I. and Marinova, K. P.  Table of experimental nuclear ground state charge radii: An update.  _Atom. Data Nucl. Data Tabl._ , 99(1):69–95, 2013\.  doi: 10.1016/j.adt.2011.12.006.
  * Belfatto & Trifinopoulos (2023) Belfatto, B. and Trifinopoulos, S.  The remarkable role of the vector-like quark doublet in the Cabibbo angle and $W$-mass anomalies.  2 2023.
  * Bender et al. (2003) Bender, M., Heenen, P.-H., and Reinhard, P.-G.  Self-consistent mean-field models for nuclear structure.  _Rev. Mod. Phys._ , 75:121–180, 2003.  doi: 10.1103/RevModPhys.75.121.
  * Bethe & Bacher (1936) Bethe, H. A. and Bacher, R. F.  Nuclear Physics A. Stationary States of Nuclei.  _Rev. Mod. Phys._ , 8:82–229, 1936.  doi: 10.1103/RevModPhys.8.82.
  * Brown (2000) Brown, B. A.  Neutron radii in nuclei and the neutron equation of state.  _Phys. Rev. Lett._ , 85:5296–5299, 2000.  doi: 10.1103/PhysRevLett.85.5296.
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Burbidge et al. (1957) Burbidge, M. E., Burbidge, G. R., Fowler, W. A., and Hoyle, F.  Synthesis of the elements in stars.  _Rev. Mod. Phys._ , 29:547–650, 1957.  doi: 10.1103/RevModPhys.29.547.
  * Crawshaw (2020) Crawshaw, M.  Multi-task learning with deep neural networks: A survey.  _arXiv preprint arXiv:2009.09796_ , 2020.
  * Dong et al. (2022) Dong, X.-X., An, R., Lu, J.-X., and Geng, L.-S.  Novel Bayesian neural network based approach for nuclear charge radii.  _Phys. Rev. C_ , 105(1):014308, 2022.  doi: 10.1103/PhysRevC.105.014308.
  * Ekström et al. (2015) Ekström, A., Jansen, G. R., Wendt, K. A., Hagen, G., Papenbrock, T., Carlsson, B. D., Forssén, C., Hjorth-Jensen, M., Navrátil, P., and Nazarewicz, W.  Accurate nuclear radii and binding energies from a chiral interaction.  _Phys. Rev. C_ , 91(5):051301, 2015.  doi: 10.1103/PhysRevC.91.051301.
  * Erler et al. (2012) Erler, J., Birge, N., Kortelainen, M., Nazarewicz, W., Olsen, E., Perhac, A., and Stoitsov, M. V.  The limits of the nuclear landscape.  _Nature_ , 486:509–512, 2012.
  * Gandolfi et al. (2012) Gandolfi, S., Carlson, J., and Reddy, S.  The maximum mass and radius of neutron stars and the nuclear symmetry energy.  _Phys. Rev. C_ , 85:032801, 2012.  doi: 10.1103/PhysRevC.85.032801.
  * Geng et al. (2005) Geng, L.-S., Toki, H., and Meng, J.  Masses, deformations and charge radii: Nuclear ground-state properties in the relativistic mean field model.  _Prog. Theor. Phys._ , 113:785–800, 2005.  doi: 10.1143/PTP.113.785.
  * Goriely et al. (2001) Goriely, S., Tondeur, F., and Pearson, J. M.  A Hartree-Fock Nuclear Mass Table.  _Atom. Data Nucl. Data Tabl._ , 77:311–381, 2001.  doi: 10.1006/adnd.2000.0857.
  * Hagen et al. (2014) Hagen, G., Papenbrock, T., Hjorth-Jensen, M., and Dean, D. J.  Coupled-cluster computations of atomic nuclei.  _Rept. Prog. Phys._ , 77(9):096302, 2014.  doi: 10.1088/0034-4885/77/9/096302.
  * Heyde et al. (1983) Heyde, K., van Isacker, P., Waroquier, M., Wood, J. L., and Meyer, R. A.  Coexistence in odd-mass nuclei.  _Phys. Rept._ , 102:291–393, 1983.  doi: 10.1016/0370-1573(83)90085-6.
  * Horowitz & Piekarewicz (2001) Horowitz, C. J. and Piekarewicz, J.  Neutron star structure and the neutron radius of Pb-208.  _Phys. Rev. Lett._ , 86:5647, 2001.  doi: 10.1103/PhysRevLett.86.5647.
  * Ma & Zhang (2022) Ma, J.-Q. and Zhang, Z.-H.  Improved phenomenological nuclear charge radius formulae with kernel ridge regression *.  _Chin. Phys. C_ , 46(7):074105, 2022.  doi: 10.1088/1674-1137/ac6154.
  * Martin et al. (2016) Martin, D., Arcones, A., Nazarewicz, W., and Olsen, E.  Impact of nuclear mass uncertainties on the $r$-process.  _Phys. Rev. Lett._ , 116(12):121101, 2016.  doi: 10.1103/PhysRevLett.116.121101.
  * Möller et al. (2012) Möller, P., Myers, W. D., Sagawa, H., and Yoshida, S.  New Finite-Range Droplet Mass Model and Equation-of-State Parameters.  _Phys. Rev. Lett._ , 108(5):052501, 2012.  doi: 10.1103/PhysRevLett.108.052501.
  * Mumpower et al. (2016) Mumpower, M. R., Surman, R., McLaughlin, G. C., and Aprahamian, A.  The impact of individual nuclear properties on $r$-process nucleosynthesis.  _Prog. Part. Nucl. Phys._ , 86:86–126, 2016.  doi: 10.1016/j.ppnp.2015.09.001.  [Erratum: Prog.Part.Nucl.Phys. 87, 116–116 (2016)].
  * Navratil et al. (2009) Navratil, P., Quaglioni, S., Stetcu, I., and Barrett, B. R.  Recent developments in no-core shell-model calculations.  _J. Phys. G_ , 36:083101, 2009.  doi: 10.1088/0954-3899/36/8/083101.
  * Niu & Liang (2018) Niu, Z. M. and Liang, H. Z.  Nuclear mass predictions based on Bayesian neural network approach with pairing and shell effects.  _Phys. Lett. B_ , 778:48–53, 2018.  doi: 10.1016/j.physletb.2018.01.002.
  * Niu & Liang (2022) Niu, Z. M. and Liang, H. Z.  Nuclear mass predictions with machine learning reaching the accuracy required by r-process studies.  _Phys. Rev. C_ , 106(2):L021303, 2022.  doi: 10.1103/PhysRevC.106.L021303.
  * Nortershauser et al. (2009) Nortershauser, W. et al.  Nuclear Charge Radii of Be-7, Be-9, Be-10 and the one-neutron halo nucleus Be-11.  _Phys. Rev. Lett._ , 102:062503, 2009.  doi: 10.1103/PhysRevLett.102.062503.
  * Novario et al. (2023) Novario, S. J., Lonardoni, D., Gandolfi, S., and Hagen, G.  Trends of Neutron Skins and Radii of Mirror Nuclei from First Principles.  _Phys. Rev. Lett._ , 130(3):032501, 2023.  doi: 10.1103/PhysRevLett.130.032501.
  * Seng (2023) Seng, C.-Y.  Model-Independent Determination of Nuclear Weak Form Factors and Implications for Standard Model Precision Tests.  _Phys. Rev. Lett._ , 130(15):152501, 2023.  doi: 10.1103/PhysRevLett.130.152501.
  * Seng et al. (2018) Seng, C.-Y., Gorchtein, M., Patel, H. H., and Ramsey-Musolf, M. J.  Reduced Hadronic Uncertainty in the Determination of $V_{ud}$.  _Phys. Rev. Lett._ , 121(24):241804, 2018.  doi: 10.1103/PhysRevLett.121.241804.
  * Utama et al. (2016) Utama, R., Chen, W.-C., and Piekarewicz, J.  Nuclear charge radii: Density functional theory meets Bayesian neural networks.  _J. Phys. G_ , 43(11):114002, 2016.  doi: 10.1088/0954-3899/43/11/114002.
  * Wang et al. (2021) Wang, M., Huang, W. J., Kondev, F. G., Audi, G., and Naimi, S.  The AME 2020 atomic mass evaluation (II). Tables, graphs and references.  _Chin. Phys. C_ , 45(3):030003, 2021.  doi: 10.1088/1674-1137/abddaf.
  * Wang et al. (2014) Wang, N., Liu, M., Wu, X., and Meng, J.  Surface diffuseness correction in global mass formula.  _Phys. Lett. B_ , 734:215–219, 2014.  doi: 10.1016/j.physletb.2014.05.049.
  * Wei et al. (2022) Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ , 2022.
  * Weizsacker (1935) Weizsacker, C. F. V.  Zur Theorie der Kernmassen.  _Z. Phys._ , 96:431–458, 1935.  doi: 10.1007/BF01337700.
  * Wood et al. (1992) Wood, J. L., Heyde, K., Nazarewicz, W., Huyse, M., and van Duppen, P.  Coexistence in even-mass nuclei.  _Phys. Rept._ , 215:101–201, 1992.  doi: 10.1016/0370-1573(92)90095-H.
  * Wu et al. (2020) Wu, D., Bai, C. L., Sagawa, H., and Zhang, H. Q.  Calculation of nuclear charge radii with a trained feed-forward neural network.  _Phys. Rev. C_ , 102(5):054323, 2020.  doi: 10.1103/PhysRevC.102.054323.
  * Wu et al. (2021) Wu, X. H., Guo, L. H., and Zhao, P. W.  Nuclear masses in extended kernel ridge regression with odd-even effects.  _Phys. Lett. B_ , 819:136387, 2021.  doi: 10.1016/j.physletb.2021.136387.
  * Wu et al. (2022) Wu, X. H., Lu, Y. Y., and Zhao, P. W.  Multi-task learning on nuclear masses and separation energies with the kernel ridge regression.  _Phys. Lett. B_ , 834:137394, 2022.  doi: 10.1016/j.physletb.2022.137394.