# Deep learning methods for Hamiltonian parameter estimation and magnetic
domain image generation in twisted van der Waals magnets

Woo Seok Lee  1ST Biotherapeutics, Seongnam 13493, Republic of Korea  Taegeun
Song  Department of Data Information and Physics, Kongju National University,
Kongju 32588, Republic of Korea  Kyoung-Min Kim  Center for Theoretical
Physics of Complex Systems, Institute for Basic Science, Daejeon 34126,
Republic of Korea<EMAIL_ADDRESS>
###### Abstract

The application of twist engineering in van der Waals magnets has opened new
frontiers in the field of two-dimensional magnetism, yielding distinctive
magnetic domain structures. Despite the introduction of numerous theoretical
methods, limitations persist in terms of accuracy or efficiency due to the
complex nature of the magnetic Hamiltonians pertinent to these systems. In
this study, we introduce a deep-learning approach to tackle these challenges.
Utilizing customized, fully connected networks, we develop two deep-neural-
network kernels that facilitate efficient and reliable analysis of twisted van
der Waals magnets. Our regression model is adept at estimating the magnetic
Hamiltonian parameters of twisted bilayer CrI3 from its magnetic domain images
generated through atomistic spin simulations. The generative model excels in
producing precise magnetic domain images from the provided magnetic
parameters. The trained networks for these models undergo thorough validation,
including statistical error analysis and assessment of robustness against
noisy injections. These advancements not only extend the applicability of
deep-learning methods to twisted van der Waals magnets but also streamline
future investigations into these captivating yet poorly understood systems.

twisted van der Waals magnets, CrI3, magnetic domains, deep neural network,
regression/generative model

###### Contents

  1. I Introduction
  2. II Methods
    1. II.1 Dataset generation
    2. II.2 Deep learning algorithms
      1. II.2.1 Regression model
      2. II.2.2 Generative model
    3. II.3 Statistical error analysis
    4. II.4 Generation of noisy test sets
  3. III Results
    1. III.1 Regression model: parameter estimation from domain images
      1. III.1.1 Validation of trained networks
      2. III.1.2 Statistical error analysis: error correlation between $J$ and $A$
      3. III.1.3 Robustness against noise injections
    2. III.2 Generative model: domain image generation from parameters
      1. III.2.1 Validation of trained networks
      2. III.2.2 Statistical error analysis: error magnification near NCD–MD transitions
      3. III.2.3 Prediction performance in averaged magnetization
  4. IV Discussion
  5. A Dataset preparation
  6. B Neural network architectures

##  I Introduction

Recent discoveries in the field of twist engineering of two-dimensional (2D)
magnetism have unveiled unique magnetic domain structures, including nanoscale
magnetic domain arrays [1, 2, 3, 4, 5]. The qualitative features of these
structures find their explanation in the context of exotic interlayer magnetic
coupling induced by moiré patterns [6, 7, 8, 9, 10]. However, due to the
complex nature of the magnetic coupling in the systems [9], the quantitative
description and analysis of these structures pose formidable challenges within
conventional theoretical frameworks. Specifically, effective model
approximations, such as the continuum model [6, 11, 12] and the free-layer-
substrate model [13, 14, 15, 16, 7, 17, 18], exhibit substantial limitations
in accuracy. Alternative atomistic spin simulations may offer improved
accuracy, but they are hindered by their resource-intensive nature [8, 9, 10,
19, 20, 21]. Consequently, it is crucial to develop theoretical frameworks
that are both efficient and reliable to overcome these constraints and propel
advancements in this field.

In this study, we introduce a deep-learning approach that facilitates
efficient and reliable analysis of twisted van der Waals magnets. Deep
learning has showcased remarkable success in tackling scientific challenges
across various physical systems [22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
33, 34, 35, 36]. Specifically, the application of deep neural network (DNN)
techniques to 2D magnetic systems has proven highly effective in extracting
magnetic Hamiltonian parameters from magnetic domain images [22]. Furthermore,
these techniques have demonstrated applicability to experimentally observed
domain images within the same research. These results validate DNN techniques
as valuable tools for comprehending 2D magnetic domain structures.

Figure 1: Schematic diagrams illustrating two deep neural network models
developed in this study. (a) The regression model for estimating magnetic
Hamiltonian parameters from input magnetic domain images. (b) The generative
model for producing predicted magnetic domain images based on input
parameters.

We develop two specialized DNN kernels designed to analyze 2D magnetic domain
structures within twisted van der Waals magnets, such as nanoscale magnetic
domain arrays found in chromium triiodide (CrI3) [1, 3, 2, 4, 5, 6, 16, 7, 17,
8, 18, 9, 10, 12, 19, 20]. Our regression model, illustrated in Fig. 1(a),
accurately estimates the parameters of the magnetic Hamiltonian from magnetic
domain images generated through atomistic spin simulations. This model
resolves an inverse problem associated with inferring the parameters from the
observed data, a feat beyond the reach of established theoretical methods [6,
11, 12, 13, 10, 14, 16, 7, 15, 17, 18, 8, 9, 19, 20]. Our generative model,
depicted in Fig. 1(b), excels at producing magnetic domain images based on
predefined magnetic parameters. This model efficiently handles the complexity
of the magnetic Hamiltonian without compromising accuracy, thereby mitigating
the need for resource-intensive atomistic spin simulations [8, 9, 10, 19, 20,
21]. These advancements not only extend the applicability of DNN techniques to
twisted van der Waals magnets but also streamline future investigations into
these captivating systems.

##  II Methods

###  II.1 Dataset generation

Figure 2: Twisted bilayer CrI3 and its magnetic domain images employed in this
study. (a) Moiré superlattice for a twist angle $\theta=5.08$°. The colored
circles denote local stacking patterns, including AA (green), AB (blue), BA
(cyan), and monoclinic (red) [9]. (b) Modulation of the local interlayer
exchange energy ($J_{i}^{\perp}=\sum_{j}J_{ij}^{\perp}$) calculated for
$\theta=1.02$°. Red and blue colors indicate antiferromagnetic and
ferromagnetic (FM) interlayer exchange couplings, respectively. (c) Zero
temperature magnetic phase diagram for $A=0.2$ meV as a function of $\theta$
and intralayer exchange ($J$), showing the FM phase (white), noncollinear
domain (NCD) phase (yellow), and magnetic domain (MD) phase (green). (d–e)
Magnetic configurations illustrating nanoscale magnetic domain arrays in the
NCD (d) and MD phases (e), respectively. Left and right panels in each plot
correspond to the top and bottom layers, respectively. The color wheel depicts
the in-plane magnetization directions, and the grayscale indicates the out-of-
plane magnetization directions. In (d), $J=2$ meV, $A=0.2$ meV and
$\theta=1.89$° are utilized. In (e), $J=2$ meV, $A=0.2$ meV and $\theta=1.02$°
are utilized.

To generate magnetic domain images, we employ twisted bilayer CrI3 [6, 16, 7,
17, 8, 18, 9, 10, 12] in commensurate moiré superlattice structures (Fig.
2(a)). The magnetic behavior of this system is effectively described using a
Heisenberg spin model [16, 8, 18, 9]:

$\displaystyle H=$ $\displaystyle-J\sum_{l=t,b}\sum_{\langle
i,j\rangle}\bm{S}_{i}^{l}\cdot\bm{S}_{j}^{l}-A\sum_{l=t,b}\sum_{i}\big{(}\bm{S}_{i}^{l}\cdot\hat{z}\big{)}^{2}+\sum_{i,j}J_{ij}^{\perp}\bm{S}_{i}^{t}\cdot\bm{S}_{j}^{b}.$
(1)

Here, $\bm{S}_{i}^{l}$ represents the spin at site $i$ on the top layer
($l=t$) and the bottom layer ($l=b$). $J>0$ represents intralayer
ferromagnetic (FM) exchange interactions. $A>0$ represents single-ion
anisotropy favoring out-of-plane magnetization. $J_{ij}^{\perp}$ represents
interlayer exchange interactions derived from ab-initio calculations on
bilayer CrI3 [9]. The coupling type of $J_{ij}^{\perp}$ varies between
antiferromagnetic and FM depending on the local stacking pattern in the moiré
superlattice (Fig. 2(b)).

Figure 2(c) illustrates three magnetic phases inherent in this spin model [8,
9]: (i) the FM phase, (ii) the noncollinear domain (NCD) phase, and (iii) the
magnetic domain (MD) phase. The FM phase exhibits a perpendicular, uniform
spin configuration. In the NCD phase, magnetic domains form in both layers,
with spins tilted horizontally in opposite directions (Fig. 2(d)). On the
other hand, in the MD phase, domains form in one layer with opposite out-of-
plane polarization compared to the background FM region (Fig. 2(e)). In the
other layer, the magnetization largely maintains an FM order, but slight spin
tilting occurs around localized domain wall regions. Both NCD and MD phases
exhibit moiré periodicity, leading to the periodic domain patterns (Fig.
2(d–e)), which we refer to as “nanoscale magnetic domain arrays” [20].

We generate magnetic domain images through atomistic spin simulations [9]
using Eq. (1). These simulations incorporate randomly generated parameter sets
($\theta$, $J$, $A$) spanning from 1.01° to 3.89° for $\theta$, 1 to 10 meV
for $J$, and 0.01 to 0.3 meV for $A$, encompassing
$(J,A)\approx(2.0~{}\textrm{meV},0.2~{}\textrm{meV})$ pertinent to CrI3 [37].
The values of $J_{ij}^{\perp}$ are fixed, determining the overall energy scale
of the magnetic parameters. For each parameter set, we obtain a magnetic
ground state and generate two images for each state, one for the top and the
other for the bottom layer. These images depict out-of-plane magnetization
within a fixed length dimension of 100 nm. Additionally, we consistently omit
the FM phase from the dataset to prevent potential overfitting issues [38],
albeit resulting in biased parameter distributions. Consequently, our dataset
comprises 17,292 paired images, capturing morphological modifications of
nanoscale magnetic domain arrays within the NCD and MD phases corresponding to
the parameter variations. Further details about the dataset generation process
can be found in Appendix A.

###  II.2 Deep learning algorithms

####  II.2.1 Regression model

We implement our regression model using a customized, fully connected network.
Three networks are employed, each predicting one assigned parameter,
collectively yielding the entire parameter set ($\theta$, $J$, $A$). Each
network consists of four hidden layers, employing the Gaussian error linear
unit (GELU) activation function [39]. The input layer of the network receives
two images, each sized at $(200,200)$, illustrating magnetic domains at the
top and bottom layers. These inputs separately undergo dimensional reduction
to $512$ features within the first two hidden layers, which are then
concatenated within a single list with a length of $1024$. This list is
subsequently processed through the two remaining hidden layers. The final
output layer yields a predicted parameter using a linear activation function.
For a visual presentation of the network architecture, refer to Appendix B.

We train the networks to minimize the discrepancy between predicted and actual
Hamiltonian parameters, employing mean squared error (MSE) as a loss function.
The Adam optimizer is employed with a dynamic learning rate ranging from
$10^{-4}$ to $10^{-3}$, and a batch size of $128$ was chosen to facilitate
efficient gradient updates. Additionally, we incorporate early stopping as a
regularization technique to prevent overfitting and enhance generalization
performance during training [38]. These carefully considered choices in
training parameters are made to ensure optimal performance. Two randomly
partitioned sets (13,836 samples and 3,459 samples) from the full dataset are
utilized as training and test sets.

####  II.2.2 Generative model

Our generative model is implemented using a customized, fully connected
network. The network consists of two hidden layers, employing the GELU
activation function. In the input layer, the network receives three parameters
($\theta$, $J$, $A$), which are processed through two hidden layers, and
culminated in a list of 77,618 features in the output layer. This list is
subsequently reshaped into two pixel arrays with dimensions (197, 197),
representing magnetic domain images that delineate the out-of-plane components
of spins in both the top and bottom layers. For a detailed visual
representation of the network architecture, please refer to Appendix B.

We train the network to minimize the discrepancy in pixel values between true
and predicted magnetic domain images, utilizing the mean absolute error (MAE)
as our chosen loss function. The detailed training process aligns with the
regression model, involving the Adam optimizer with a dynamic learning rate
ranging from $10^{-4}$ to $10^{-3}$, a batch size of $128$, and the
implementation of early stopping as a regularization technique. The generative
model is trained and tested using the same training and test datasets that are
used in the regression model.

###  II.3 Statistical error analysis

Our statistical error analysis in both regression and generative models
involves normalizing $\theta$, $J$, and $A$ as follows:

$\displaystyle\theta_{N}=$
$\displaystyle\;\frac{\theta-\textrm{min}(\theta)}{\textrm{max}(\theta)-\textrm{min}(\theta)},$
$\displaystyle J_{N}=$
$\displaystyle\;\frac{J-\textrm{min}(J)}{\textrm{max}(J)-\textrm{min}(J)},$
$\displaystyle A_{N}=$
$\displaystyle\;\frac{A-\textrm{min}(A)}{\textrm{max}(A)-\textrm{min}(A)}.$
(2)

This normalization ensures that all parameters fall within the $[0,1]$ range,
facilitating consistent treatment across different parameters [22].
Furthermore, the Euclidean distance from each data point to the phase boundary
is calculated using the following expression:

$\displaystyle
R=\sqrt{(\theta_{N}-\theta_{N}^{*})^{2}+(J_{N}-J_{N}^{*})^{2}+(A_{N}-A_{N}^{*})^{2}}.$
(3)

Here, $(\theta_{N},J_{N},A_{N})$ and $(\theta_{N}^{*},J_{N}^{*},A_{N}^{*})$
represent a data point in the test set and its corresponding nearest point on
the FM–NCD or NCD–MD phase boundary.

In the regression model, the MAE values for each normalized parameter are
calculated as:

$\displaystyle\Delta\theta_{N}=$
$\displaystyle\;\left|\theta_{N}^{\textrm{Pred.}}-\theta_{N}^{\textrm{True}}\right|,$
$\displaystyle\Delta J_{N}=$
$\displaystyle\;\left|J_{N}^{\textrm{Pred.}}-J_{N}^{\textrm{True}}\right|,$
$\displaystyle\Delta A_{N}=$
$\displaystyle\;\left|A_{N}^{\textrm{Pred.}}-A_{N}^{\textrm{True}}\right|,$
(4)

where ‘True’ and ‘Pred.’ indicate the true and predicted values in the test
set, respectively. Furthermore, our principal component analysis [40, 22]
incorporates error vectors, defined as:

$\displaystyle X=(\Delta\theta_{N},\Delta J_{N},\Delta A_{N}).$ (5)

The covariance matrix of the error vectors is computed using the following
expression:

$\displaystyle K_{ij}=E[(X_{i}-E[X_{i}])(X_{j}-E[X_{j}])].$ (6)

Here, $X_{i,j}$ for $i,j=1,2,3$ represents each component in the error vector
$X$. $E[X_{i}]$ indicates the average value of $X_{i}$ among different samples
in the test set. The eigenvectors and eigenvalues of $K_{ij}$ are computed,
and the principal components are selected based on the magnitude of their
eigenvalues. Variance ratios are computed for each component using
$\frac{\lambda_{i}}{\sum_{i=1}^{3}\lambda_{i}}\times 100$, where $\lambda_{i}$
denotes the eigenvalue for each component.

###  II.4 Generation of noisy test sets

We generate noisy test sets from the original test set by introducing
Gaussian-white noises according to the equation:

$\displaystyle z_{i}=z_{i}^{0}+\delta z_{i}.$ (7)

Here, $z_{i}^{0}$ and $z_{i}$ represent the pixel values at the $i$-th pixel
in the original and noise-injected images, respectively. The injected noise
$\delta z_{i}$ follows a Gaussian distribution:

$\displaystyle P(\delta
z_{i})=\frac{1}{\sigma\sqrt{2\pi}}\exp{\bigg{(}-\frac{\delta
z_{i}^{2}}{2\sigma^{2}}\bigg{)}}.$ (8)

The variance $\sigma^{2}$ of the distribution is effectively controlled by
adjusting a signal-to-noise ratio (SNR), defined as
$R_{\textrm{SNR}}=P_{\textrm{signal}}/P_{\textrm{noise}}$. Here,
$P_{\textrm{signal}}=\frac{1}{N_{\textrm{pixel}}}\sum_{i=1}^{N_{\textrm{pixel}}}\big{|}z_{i}^{0}\big{|}^{2}$
and
$P_{\textrm{noise}}=\frac{1}{N_{\textrm{pixel}}}\sum_{i=1}^{N_{\textrm{pixel}}}\big{|}\delta
z_{i}\big{|}^{2}$ represent signal and noise power, respectively. The
adjustment using $R_{\textrm{SNR}}$ ensures consistency across different
images with various $P_{\textrm{signal}}$ levels [22]. Given
$P_{\textrm{signal}}$ obtained from a noiseless image and a specified value of
$R_{\textrm{SNR}}$, we determine $\sigma^{2}$ through the relationship:

$\displaystyle\sigma^{2}=P_{\textrm{signal}}/R_{\textrm{SNR}},$ (9)

which is derived from $\sigma^{2}\approx P_{\textrm{noise}}$, valid for a
sufficiently large $N_{\textrm{pixel}}$, the number of pixels in the image. In
the generation process of noisy datasets, the SNR is systematically varied
over a wide range of $R_{\textrm{SNR}}$ using a decibel scale:

$\displaystyle R_{\textrm{SNR}}^{\textrm{dB}}=10\log_{10}(R_{\textrm{SNR}}),$
(10)

where $R_{\textrm{SNR}}^{\textrm{dB}}$ is tuned from $-5$ dB to $60$ dB.

##  III Results

###  III.1 Regression model: parameter estimation from domain images

####  III.1.1 Validation of trained networks

Figure 3: Performance of the trained networks in the regression model. (a–c)
Parameter estimation results using the test set. The $x$-axis denotes the true
values used in the simulation, while the $y$-axis represents the model’s
estimated values. The yellow lines ($y=x$) represent the ideal predictions.
Evaluation metrics include Pearson correlation coefficients (labeled as
Pearson), mean squared error (MSE), and mean absolute percentage error (MAPE).
(d–f) Profiles of the parameter estimation errors derived from (a–c). Green
bars depict the count of samples ($N_{\textrm{training}}$) within each
parameter range in the training set. Blue bars represent the MAPE values for
the estimated parameters. Error bars signify the standard errors of the MAPE
values. The units of degree, meV, and meV are utilized for $\theta$, $J$, and
$A$, respectively.

Figure 3 presents the parameter estimation results from our trained networks
on the test set. The $\theta$ estimation demonstrates precise agreement
between predicted and true values, evidenced by high-level scores in
evaluation metrics (Fig. 3(a)). Further analysis reveals consistently low mean
absolute percentage errors (MAPEs) across the entire $\theta$ range (Fig.
3(d)). Particularly noteworthy is the observation that MAPE values remain
below 0.1 in the small twist angle regime ($\theta\lesssim 1.5$°). This
capability indicates the network’s proficiency in estimating $\theta$,
underscoring its promising applications to small twist angle systems, which
are frequently involved in experimental investigations [1, 3, 2, 4, 5].

Estimations for $J$ and $A$ also exhibit consistent linear relationships
between predicted and true values, showing small MAPE values of 1.22 and 2.23
for $J$ and $A$, respectively (Fig. 3(b–c)). Further examination reveals that
the MAPE value for $J$ and $A$ tend to increase as their true values decrease,
while high accuracy is maintained for sufficiently large $J$ and $A$ values
(Fig. 3(e–f)). Specifically, in the vicinity of
$(J,A)\approx(2.0~{}\textrm{meV},0.2~{}\textrm{meV})$ pertinent to CrI3 [37],
the MAPE values stand at small numbers, 1.9 and 0.7 for $J$ and $A$,
respectively. These results showcase the networks’ reliable predictive
capabilities, specifically in magnetic systems with moderate values of $J$ and
$A$, including CrI3.

####  III.1.2 Statistical error analysis: error correlation between $J$ and
$A$

Figure 4: Profiles of parameter-estimation errors concerning parameter
variations. The $x$-axis denotes the normalized parameters ($\theta_{N}$,
$J_{N}$, and $A_{N}$). Blue bars represent the sample count
($N_{\textrm{training}}$) in the training set. Red bars depict the mean
absolute error (MAE) values ($\Delta\theta_{N}$, $\Delta J_{N}$, and $\Delta
A_{N}$). Error bars indicate the standard errors of the MAE values.

To delve deeper into the errors in parameter estimation, we conduct a
statistical error analysis on these errors. Figure 4 illustrates the profiles
of the MAE values ($\Delta\theta_{N}$, $\Delta J_{N}$, $\Delta A_{N}$) and the
sample count ($N_{\textrm{training}}$) in the training set, concerning the
variation of the normalized parameters ($\theta_{N}$, $J_{N}$, $A_{N}$). Our
finding reveals that the MAE values tend to increase as their corresponding
values of $N_{\textrm{training}}$ decrease (Fig. 4(a,e,i)). This tendency
demonstrates that sample scarcity is one of the primary factors contributing
to these errors, aligning with common expectations in machine learning
modeling. In addition to this, we notice an unusual rise in $\Delta A_{N}$
within a range characterized by a small $A_{N}$ value, despite an increase in
$N_{\textrm{training}}$ (Fig. 4(i)). One of the possible explanations for this
is that the rapid alterations in magnetic domain structures within this range
[9] make it challenging for the networks to learn these patterns from the
limited dataset.

We observe no discernible correlations between $\Delta\theta_{N}$–$J_{N}$,
$\Delta\theta_{N}$–$A_{N}$, $\Delta J_{N}$–$\theta_{N}$, and $\Delta
A_{N}$–$\theta_{N}$ (Fig. 4(b,c,d,g)), suggesting that variations in $J_{N}$
and $A_{N}$ do not significantly affect the estimation of $\theta_{N}$, and
vice versa. In contrast, noticeable correlations exist between $\Delta
J_{N}$–$A_{N}$ and $\Delta A_{N}$–$J_{N}$ (Fig. 4(f,h)), indicating mutual
impacts on their parameter estimation. These contrasting tendencies may stem
from the distinct roles among different parameters in determining nanoscale
magnetic domain arrays; while $\theta_{N}$ solely dictates the arrays’
periodicity, $J_{N}$ and $A_{N}$ collectively influence the detailed
morphology of individual domains [9]. Consequently, these results suggest that
the networks recognize these two factors—the periodicity and domain shape of
the domain arrays—and process them independently, despite not being explicitly
incorporated into the algorithm. In other words, our model effectively
exploits the networks’ capability to learn from data without explicit manual
programming, which is one of the primary benefits of deep learning techniques.

Figure 5: Visualization of error vectors and their principal components. Blue
dots denote the error vectors ($\Delta\theta_{N}$, $\Delta J_{N}$, $\Delta
A_{N}$), while red dots indicate their projection onto the respective planes.
Cyan, magenta, and yellow arrows denote the first, second, and third principal
components, given by $(-0.017,-0.227,0.974)$, $(0.019,-0.974,-0.226)$, and
$(0.974,-0.226,0.021)$, respectively. The numbers next to each arrow specify
the variance ratio of the corresponding component.

To gain deeper insights into the correlation among errors, we conduct a
principal component analysis on the error vectors ($\Delta\theta_{N},\Delta
J_{N},\Delta A_{N}$), as depicted in Fig. 5. Notably, the error distribution
appears highly concentrated within a two-dimensional subspace delineated by
$\Delta J_{N}$ and $\Delta A_{N}$, reflecting the relatively large values of
$\Delta J_{N}$ and $\Delta A_{N}$ compared to $\Delta\theta_{N}$ (Fig.
3(d–f)). Moreover, within this subspace, the distribution of $\Delta J_{N}$
and $\Delta A_{N}$ manifests notable asymmetry, as demonstrated by the
principal component $(-0.017,-0.227,0.974)$, which accounts for 77.86% of the
observed variance. This predominant axis, intertwined with $\Delta J_{N}$ and
$\Delta A_{N}$, signals a significant relationship between the two errors
[22], succinctly expressed as:

$\displaystyle\Delta J_{N}\approx-0.233\Delta A_{N}.$ (11)

This relationship suggests a tendency for underestimation in $\Delta J_{N}$ to
coincide with overestimation in $\Delta A_{N}$, and vice versa. Moreover, this
correlation may elucidate the observed error magnification in $\Delta J_{N}$
and $\Delta A_{N}$ (Fig. 3(e–f)), where errors in one variable propagate and
exacerbate those in another, resulting in more pronounced deviations [41].

The observed correlation between $\Delta J_{N}$ and $\Delta A_{N}$ may offer
valuable physical insights into the system. Specifically, it suggests an
intricate interplay between $J$ and $A$ in shaping the detailed morphology of
nanoscale magnetic domain arrays. One conceivable scenario is that $J$ and $A$
operate interchangeably, i.e., an increase in one parameter compensates for a
decrease in the other, thereby preserving the overall structure. This
possibility aligns with previous findings in the interaction between magnetic
dipolar interactions and single-ion anisotropy [22]. While elucidating the
precise mechanism lies beyond the scope of this study, the specific
relationship uncovered here may provide valuable insights into this pursuit.

Figure 6: Profiles of parameter-estimation errors near phase boundaries. The
$x$-axis denotes the Euclidean distance from each data point to the FM–NCD or
NCD–MD phase boundary ($R_{\textrm{FM–NCD}}$ or $R_{\textrm{NCD–MD}}$). Blue
bars depict the sample count ($N_{\textrm{training}}$) in the training set.
Red bars depict the MAE values ($\Delta\theta_{N}$, $\Delta J_{N}$, and
$\Delta A_{N}$). Error bars signify the standard errors of the MAE values.

For a thorough understanding of the errors, we evaluate the network’s
performance near phase boundaries, where sharp variations in magnetic domain
structures can significantly affect parameter estimation. Figure 6 illustrates
the profiles of MAE values concerning the Euclidean distance from each data
point to the FM–NCD or NCD–MD phase boundaries that quantify the degree of
separation from these phase boundaries. We observe a notable increase in MAE
values as the Euclidean distance to the FM–NCD phase boundary decreases (Fig.
6(a–c)), indicating amplified errors near this boundary. This trend contrasts
with the growing number of samples within this region, suggesting that the
errors are more influenced by the physical aspects of this transition rather
than sample scarcity. However, no discernible correlation is found between the
MAE values and the Euclidean distance to the NCD–MD phase boundary (Fig.
6(d–f)). Consequently, we infer that the FM–NCD transition plays a pivotal
role in magnifying errors in parameter estimation.

From these results, we deduce that errors in parameter estimation within the
regression model may stem from three potential sources of magnification: (i)
limited sample availability in the training set, (ii) error correlation, and
(iii) the FM–NCD transition. Although the hierarchy among these factors and
their potential interplay lie beyond the scope of this study, these
identifications offer valuable insights for implementing meticulous treatments
aimed at enhancing predictive accuracy.

####  III.1.3 Robustness against noise injections

Figure 7: Performance of the trained networks in the regression model in the
presence of noises. (a–c) Parameter estimation results derived from noisy test
sets in comparison to the noiseless original test set (denoted by Clean).
Here, $R_{\textrm{SNR}}^{\textrm{dB}}$ denotes the signal-to-noise ratio,
effectively reflecting the noise level on a decibel scale. The $x$-axis
denotes the true values used in the simulation, while the $y$-axis represents
the model’s estimated values. The pink lines ($y=x$, denoted by True)
represent the ideal predictions. (d–f) MAPE values for the estimated
parameters in different $R_{\textrm{SNR}}^{\textrm{dB}}$ levels. The markers
denote MAPE values for each parameter. The red dashed line depicts the MAPE
values for the Clean test set.

For real-world applications, including noisy experimental data, we evaluate
the regression model’s performance under noisy conditions. Figure 7 presents
the parameter estimation results on noisy test sets obtained from the networks
trained on the noiseless training set. The $\theta$ estimation sustains a
consistent linear relationship between predicted and true values across the
entire $\theta$ range (Fig. 7(a)). Moreover, the MAPE value remains
steadfastly low at 5% to the highest noise level of
$R_{\textrm{SNR}}^{\textrm{dB}}=-5$ dB (Fig. 7(d)). Consequently, this result
supports that the network retains its proficiency in accurately extracting
$\theta$ from magnetic domain images containing substantial noises.

Estimation for $J$ and $A$ also display consistent linear relationships
between predicted and true values (Fig. 7(b–c)), although their precision
considerably degrades as the noise level increases. Specifically, within a
weak noise regime ($R_{\textrm{SNR}}^{\textrm{dB}}\geq 20$ dB), the MAPE
values are sustained at relatively small values less than $4\%$ and $6\%$ for
$J$ and $A$, respectively (Fig. 7(e–f)). However, upon entering a strong noise
regime ($R_{\textrm{SNR}}^{\textrm{dB}}<20$ dB), the MAPE values steeply
increase, reaching considerable levels exceeding $20\%$ for both $J$ and $A$
at $R_{\textrm{SNR}}^{\textrm{dB}}=10$ dB. Consequently, the networks’
capability to make meaningful predictions in $J$ and $A$ is practically
restricted to $R_{\textrm{SNR}}^{\textrm{dB}}=20$ dB.

###  III.2 Generative model: domain image generation from parameters

####  III.2.1 Validation of trained networks

Figure 8: Performance of the trained network in the generative model. (a–b)
Examples of predicted magnetic domain images corresponding to (a) the NCD and
(b) MD phases. The color scale indicates the magnitude and direction of out-
of-plane magnetization. The first and second columns display the true and
predicted images, and the third column shows the MAE values of the out-of-
plane magnetization between the two images. The first and second rows
represent the top and bottom layers, respectively. The parameter sets
$(\theta,J,A)=$ (3.15, 5.36, 0.14) and $(\theta,J,A)=$ (1.16, 6.00, 0.04) are
utilized for (a) and (b), respectively. (c–e) Profiles of the average MAE
values obtained from the predicted images in the test set. Green bars depict
the count of samples ($N_{\textrm{training}}$) within each parameter range in
the training set. Blue bars represent the average MAE values of the predicted
images. Error bars signify the standard errors of the average MAE values. The
degree, meV, and meV units are utilized for $\theta$, $J$, and $A$,
respectively.

Figure 8 illustrates the magnetic domain image prediction results from our
trained network on the test set. The predicted images precisely replicate the
intricate pattern of the nanoscale magnetic domain arrays in the true images,
capturing details such as the domains’ location, shape, and size (the first
and second columns in Fig. 8(a–b)). Additionally, the magnitude of local
magnetization exhibits precise agreement between the two images over the
majority of the region (the third column in Fig. 8(a–b)), as evidenced by
small average MAE values of approximately $\sim 0.002$ and $\sim 0.007$ for
the NCD and MD phases, respectively. This precise agreement persists
consistently across the entire parameter ranges, with average MAE values
remaining below $0.03$ (Fig. 8(c–e)). These observations affirm the proficient
performance of the trained network in accurately predicting magnetic domain
images from given parameters.

Despite its overall decent performance, the network shows a notable
discrepancy in predicting magnetic domain images between the NCD and MD
phases. In the NCD phase, the MAE value between true and predicted images
remains consistently low, less than $0.02$ (the third column in Fig. 8(a)).
Conversely, in the MD phase, relatively large MAE values exceeding $0.3$
appear in a layer where magnetic domains form (the third column in Fig. 8(b)).
A closer examination reveals a concentration of these high MAE values within
the local domain-wall regions, where sharp alterations in magnetization occur.
This is in contrast to the NCD phase, which lacks domain walls and exhibits
relatively smooth variations across the entire region. This observation leads
us to posit that the pronounced discrepancy in the MD phase arises from the
network’s challenge in learning sharp variations within domain wall regions
from the limited dataset.

####  III.2.2 Statistical error analysis: error magnification near NCD–MD
transitions

Figure 9: Profiles of domain-image-prediction errors near phase boundaries.
Blue bars depict the sample count in the training set. The $x$-axis indicates
the Euclidean distance from each data point to the phase boundary. Red bars
represent the average MAE values for out-of-plane magnetization between true
and predicted images. Error bars signify the standard errors of the average
MAE values.

To delve deeper into the errors in domain image generation, we evaluate the
network’s performance near phase boundaries. Figure 9 illustrates the profiles
of average MAE values between true and predicted images concerning the
Euclidean distance from each data point to the phase boundaries. We observe no
discernible correlation between the average MAE value and the distance to the
FM–NCD phase boundary (Fig. 9(a)). However, the average MAE value demonstrates
a striking correlation with the distance to the NCD–MD phase boundary, showing
a steep increase near the phase boundary (Fig. 9(b)). This trend is
particularly compelling, given the rising number of samples within this
region, and indicates that the nature of errors is deeply rooted in the
physical aspects of the transition rather than a mere statistical origin, such
as sample scarcity. These observations strongly suggest that the NCD–MD
transition serves as a primary factor magnifying the errors in domain image
generation, warranting further investigation.

One conceivable scenario is that the discontinuous nature of the NCD–MD
transition exacerbates errors near this boundary. During this transition, a
magnetic domain array undergoes a discontinuous transformation [6, 8, 9]. The
degree of change, quantified by its net magnetization, tends to increase with
a higher value of $A$ [9]. In a large $A$ regime, this change may surpass the
network’s capacity to effectively capture, leading to the magnification of
errors. This scenario gains support when observing the magnification of errors
with increasing $A$ (Fig. 8(e)). Consequently, it becomes evident that
meticulous attention is required to mitigate the impacts of the abrupt change
across the NCD–MD transition and uphold the accuracy of the generative model
in this scenario.

####  III.2.3 Prediction performance in averaged magnetization

Figure 10: Performance of the trained network in the generative model in
predicting averaged magnetization. (a–c) Profiles of MAPE values of averaged
magnetization ($M$). The $x$-axis denotes the true values used in the
simulation. Red bars represent the MAPE values for averaged magnetization.
Error bars signify the standard errors of the MAPE values. (d–e) Profiles of
MAE values of averaged magnetization near phase boundaries. The $x$-axis
indicates the Euclidean distance from each data point to the phase boundary.
Red bars represent the MAE values for averaged magnetization ($\Delta M$).
Error bars signify the standard errors of the MAE values.

To assess the versatility of the generative model, we evaluate its performance
in predicting averaged magnetization, a practical indicator for discerning
nanoscale magnetic domain arrays in experimental investigations [1, 3, 2, 4,
5]. Figure 10(a–c) presents the profiles of MAPE values between true and
predicted averaged magnetization ($M$), concerning the parameter variations.
Our findings reveal consistently low MAPE values across all parameter ranges.
Particularly noteworthy is the observation that MAPE values remain below $0.3$
in the small twist angle regime ($\theta\lesssim 1.5$°) (Fig. 10(a)) and below
$0.5$ near $(J,A)\approx(2.0\textrm{meV},0.2~{}\textrm{meV})$ pertinent to
CrI3 (Fig. 10(b–c)). This capability indicates the network’s proficiency in
estimating averaged magnetization, underscoring its promising applications to
small twist angle systems and the pivotal material CrI3, which are frequently
investigated experimentally [1, 3, 2, 4, 5].

To delve deeper into the errors, we evaluate the network’s performance near
the phase boundaries. Figure 10(d–e) illustrates the profiles of MAE values
between true and predicted averaged magnetization ($\Delta M$), concerning the
Euclidean distance to the phase boundaries. We observe no discernible
correlation between $\Delta M$ and the distance to the FM–NCD phase boundary
(Fig. 10(d)). On the other hand, a striking increasing tendency of $\Delta M$
emerges near the NCD–MD phase boundary (Fig. 10(e)). These observations align
completely with the previous findings in domain image prediction of the
generative model. Consequently, the considerations regarding error magnifying
factors in that context apply similarly to the case of averaged magnetization,
suggesting that the discontinuous nature of the NCD–MD transition serves as a
primary factor in magnifying errors in predicting averaged magnetization.

##  IV Discussion

We have demonstrated the efficacy of our trained networks in accurately
predicting magnetic Hamiltonian parameters based on magnetic domain images
within twisted bilayer magnets and vice versa. The reliable performance of our
DNN methods positions them as efficient tools for analyzing the intricate
magnetic behaviors inherent in these systems. Specifically, these methods have
direct applications in the study of twisted bilayer CrI3 [1, 2, 6, 16, 7, 17,
8, 18, 9, 10, 15] and other related systems incorporating different van der
Waals magnetic materials [6, 7, 18, 9, 21]. Furthermore, our methods possess
proficiency in the efficient analysis of twisted multilayer superlattice
structures, which typically require increased computing resources for handling
such large-sized superlattices [3, 4, 5, 19]. Additionally, the networks’
robust performance against noise injections suggests promising applications to
experimental data [1, 2, 3, 4, 5].

Future research prospects encompass expanding our methods to different moiré-
induced magnetic orders in various twisted magnetic systems, including the
exploration of the coexisting multiple zigzag antiferromagnetic orders in
twisted bilayer -RuCl3 [21]. Additionally, incorporation of topological
defects such as skyrmions [13, 14, 16, 15, 7, 8, 17, 12, 9, 21] and merons
[20] into the magnetic domain images holds promise for gaining valuable
insights into the generation of topological defects within twisted van der
Waals magnets, a captivating aspect that remains unexplored in this study.

We highlight that the pioneering deep-learning approach established in this
study opens avenues for the application of deep-learning techniques in twisted
van der Waals magnets. This significant breakthrough not only deepens our
understanding of the intricate magnetic behaviors within these captivating
systems but also holds great promise for accelerating advancements in this
swiftly evolving field.

## Acknowledgements

We would like to thank Hyun Kim and Se Kwon Kim for sharing their insights.
K.-M.K. was supported by the Institute for Basic Science in the Republic of
Korea through the project IBS-R024-D1.

## References

  * Song  _et al._ [2021] T. Song, Q.-C. Sun, E. Anderson, C. Wang, J. Qian, T. Taniguchi, K. Watanabe, M. A. McGuire, R. Stöhr, D. Xiao, T. Cao, J. Wrachtrup, and X. Xu, Direct visualization of magnetic domains and moiré magnetism in twisted 2D magnets, Science 374, 1140 (2021).
  * Xu  _et al._ [2022] Y. Xu, A. Ray, Y.-T. Shao, S. Jiang, K. Lee, D. Weber, J. E. Goldberger, K. Watanabe, T. Taniguchi, D. A. Muller, K. F. Mak, and J. Shan, Coexisting ferromagnetic–antiferromagnetic state in twisted bilayer CrI3, Nat. Nanotechnol. 17, 143 (2022).
  * Xie  _et al._ [2022] H. Xie, X. Luo, G. Ye, Z. Ye, H. Ge, S. H. Sung, E. Rennich, S. Yan, Y. Fu, S. Tian, H. Lei, R. Hovden, K. Sun, R. He, and L. Zhao, Twist engineering of the two-dimensional magnetism in double bilayer chromium triiodide homostructures, Nat. Phys. 18, 30 (2022).
  * Xie  _et al._ [2023] H. Xie, X. Luo, Z. Ye, Z. Sun, G. Ye, S. H. Sung, H. Ge, S. Yan, Y. Fu, S. Tian, H. Lei, K. Sun, R. Hovden, R. He, and L. Zhao, Evidence of non-collinear spin texture in magnetic moiré superlattices, Nat. Phys. 19, 1150 (2023).
  * Cheng  _et al._ [2023] G. Cheng, M. M. Rahman, A. L. Allcca, A. Rustagi, X. Liu, L. Liu, L. Fu, Y. Zhu, Z. Mao, K. Watanabe, T. Taniguchi, P. Upadhyaya, and Y. P. Chen, Electrically tunable moiré magnetism in twisted double bilayers of chromium triiodide, Nat. Electron. 6, 434 (2023).
  * Hejazi  _et al._ [2020] K. Hejazi, Z.-X. Luo, and L. Balents, Noncollinear phases in moiré magnets, Proc. Natl. Acad. Sci. U.S.A. 117, 10721 (2020).
  * Akram  _et al._ [2021] M. Akram, H. LaBollita, D. Dey, J. Kapeghian, O. Erten, and A. S. Botana, Moiré skyrmions and chiral magnetic phases in twisted CrX3 (X = I, Br, and Cl) bilayers, Nano Lett. 21, 6633 (2021).
  * Zheng [2023] F. Zheng, Magnetic skyrmion lattices in a novel 2D-twisted bilayer magnet, Adv. Func. Mater. 33, 2206923 (2023).
  * Kim  _et al._ [2023] K.-M. Kim, D. H. Kiem, G. Bednik, M. J. Han, and M. J. Park, Ab initio spin hamiltonian and topological noncentrosymmetric magnetism in twisted bilayer cri3, Nano Lett. 23, 6088 (2023).
  * Yang  _et al._ [2023] B. Yang, Y. Li, H. Xiang, H. Lin, and B. Huang, Moiré magnetic exchange interactions in twisted magnets, Nat. Comput. Sci. 3, 314 (2023).
  * Hejazi  _et al._ [2021] K. Hejazi, Z.-X. Luo, and L. Balents, Heterobilayer moiré magnets: Moiré skyrmions and commensurate-incommensurate transitions, Phys. Rev. B 104, L100406 (2021).
  * Shaban  _et al._ [2023] P. S. Shaban, I. S. Lobanov, V. M. Uzdin, and I. V. Iorsh, Skyrmion dynamics in moiré magnets, Phys. Rev. B 108, 174440 (2023).
  * Tong  _et al._ [2018] Q. Tong, F. Liu, J. Xiao, and W. Yao, Skyrmions in the moiré of van der Waals 2D magnets, Nano Lett. 18, 7194 (2018).
  * Akram and Erten [2021] M. Akram and O. Erten, Skyrmions in twisted van der waals magnets, Phys. Rev. B 103, L140406 (2021).
  * Ray and Das [2021] S. Ray and T. Das, Hierarchy of multi-order skyrmion phases in twisted magnetic bilayers, Phys. Rev. B 104, 014410 (2021).
  * Xiao  _et al._ [2021] F. Xiao, K. Chen, and Q. Tong, Magnetization textures in twisted bilayer CrX3 (X=Br, I), Phys. Rev. Res. 3, 013027 (2021).
  * Ghader  _et al._ [2022] D. Ghader, B. Jabakhanji, and A. Stroppa, Whirling interlayer fields as a source of stable topological order in moiré CrI3, Commun. Phys. 5, 192 (2022).
  * Fumega and Lado [2023] A. O. Fumega and J. L. Lado, Moiré-driven multiferroic order in twisted CrCl3, CrBr3 and CrI3 bilayers, 2D Materials 10, 025026 (2023).
  * Kim and Park [2023] K.-M. Kim and M. J. Park, Controllable magnetic domains in twisted trilayer magnets, Phys. Rev. B 108, L100401 (2023).
  * Kim  _et al._ [2024] K.-M. Kim, G. Go, M. J. Park, and S. K. Kim, Emergence of stable meron quartets in twisted magnets, Nano Lett. 24, 74 (2024).
  * Akram  _et al._ [2024] M. Akram, J. Kapeghian, J. Das, R. Valentí, A. S. Botana, and O. Erten, Theory of moiré magnetism in twisted bilayer $\alpha$-rucl3, Nano Lett. 24, 890 (2024).
  * Kwon  _et al._ [2020] H. Y. Kwon, H. G. Yoon, C. Lee, G. Chen, K. Liu, A. K. Schmid, Y. Z. Wu, J. W. Choi, and C. Won, Magnetic hamiltonian parameter estimation using deep learning techniques, Sci. Adv. 6, eabb0872 (2020).
  * Wright  _et al._ [2022] L. G. Wright, T. Onodera, M. M. Stein, T. Wang, D. T. Schachter, Z. Hu, and P. L. McMahon, Deep physical neural networks trained with backpropagation, Nature 601, 549 (2022).
  * Schmidhuber [2015] J. Schmidhuber, Deep learning in neural networks: An overview, Neural Networks 61, 85 (2015).
  * Pfau  _et al._ [2020] D. Pfau, J. S. Spencer, A. G. D. G. Matthews, and W. M. C. Foulkes, Ab initio solution of the many-electron schrödinger equation with deep neural networks, Phys. Rev. Res. 2, 033429 (2020).
  * Senior  _et al._ [2019] A. W. Senior, R. Evans, J. Jumper, J. Kirkpatrick, L. Sifre, T. Green, C. Qin, A. Žídek, A. W. R. Nelson, A. Bridgland, H. Penedones, S. Petersen, K. Simonyan, S. Crossan, P. Kohli, D. T. Jones, D. Silver, K. Kavukcuoglu, and D. Hassabis, Protein structure prediction using multiple deep neural networks in the 13th critical assessment of protein structure prediction (casp13), Proteins: Structure, Function, and Bioinformatics 87, 1141 (2019).
  * Carleo and Troyer [2017] G. Carleo and M. Troyer, Solving the quantum many-body problem with artificial neural networks, Science 355, 602 (2017).
  * Cai and Liu [2018] Z. Cai and J. Liu, Approximating quantum many-body wave functions using artificial neural networks, Phys. Rev. B 97, 035116 (2018).
  * Singh and Han [2019] V. K. Singh and J. H. Han, Application of machine learning to two-dimensional dzyaloshinskii-moriya ferromagnets, Phys. Rev. B 99, 174426 (2019).
  * Kwon  _et al._ [2019] H. Y. Kwon, N. J. Kim, C. K. Lee, and C. Won, Searching magnetic states using an unsupervised machine learning algorithm with the heisenberg model, Phys. Rev. B 99, 024423 (2019).
  * Lee and Ahn [2020] W. S. Lee and K.-H. Ahn, Heterogeneous trp channel model of a chordotonal neuron might explain drosophila hearing, J. Korean Phys. Soc. 76, 118 (2020).
  * Lee  _et al._ [2021a] W. S. Lee, J. Jo, and T. Song, Machine learning for the diagnosis of early-stage diabetes using temporal glucose profiles, J. Korean Phys. Soc. 78, 373 (2021a).
  * Lee  _et al._ [2021b] W. S. Lee, H. Kim, A. N. Cleland, and K.-H. Ahn, Fast frequency discrimination and phoneme recognition using a biomimetic membrane coupled to a neural network, Bioinspir. Biomim. 16, 026012 (2021b).
  * Lee and Flach [2020] W. S. Lee and S. Flach, Deep learning of chaos classification, Mach. Learn.: Sci. Technol. 1, 045019 (2020).
  * Easaw  _et al._ [2023] N. Easaw, W. S. Lee, P. S. Lohiya, S. Jalan, and P. Pradhan, Estimation of correlation matrices from limited time series data using machine learning, J. Comput. Sci 71, 102053 (2023).
  * Miyazaki [2023] Y. Miyazaki, Equivariant neural networks for spin dynamics simulations of itinerant magnets, Machine Learning: Science and Technology 4, 045006 (2023).
  * Chen  _et al._ [2018] L. Chen, J.-H. Chung, B. Gao, T. Chen, M. B. Stone, A. I. Kolesnikov, Q. Huang, and P. Dai, Topological spin excitations in honeycomb ferromagnet ${\mathrm{cri}}_{3}$, Phys. Rev. X 8, 041028 (2018).
  * Hawkins [2004] D. M. Hawkins, The problem of overfitting, J. Chem. Inf. Comput. 44, 1 (2004).
  * Hendrycks and Gimpel [2023] D. Hendrycks and K. Gimpel, Gaussian error linear units (gelus) (2023), arXiv:1606.08415 [cs.LG] .
  * Jolliffe [1986] I. Jolliffe,  _Principal Component Analysis_ (Springer Verlag, 1986).
  * Shalev-Shwartz and Ben-David [2014] S. Shalev-Shwartz and S. Ben-David,  _Understanding Machine Learning: From Theory to Algorithms_ (Cambridge University Press, USA, 2014).

##  Appendix A Dataset preparation

Figure 11: Magnetic phase identifications for full datasets obtained in the
atomistic spin simulation. (a) Out-of-plane magnetization ($M$) across both
top and bottom layers as a function of intralayer exchange coupling ($J$),
twist angle ($\theta$), and single-ion anisotropy ($A$). (b) Difference
between top and bottom layer magnetizations ($\Delta M$) as a function of the
same three parameters. (c) Visual representation for identified magnetic
phases as a function of the three parameters. The identification is based on
the results on (a–b) and the classification scheme presented in Eq. (A).
Black, green, and red markers indicate the FM, NCD, and MD phases,
respectively.

Our atomistic spin simulations involved 23,000 randomly generated parameter
sets ($\theta$, $J$, $A$), as illustrated in Fig. 11. We computed the magnetic
ground state for each parameter set by employing 50 different random initial
spin configurations and relaxing each configuration using an iterative
optimization method [20]. During the relaxation, periodic boundary conditions
were imposed on the boundary of the moiré supercell. The magnetic ground state
is determined by selecting the lowest energy configuration among the resulting
energy-minimized spin configurations.

To classify the magnetic ground states obtained in the simulation, we
calculated the following order parameters:

$\displaystyle M$
$\displaystyle=\frac{1}{2}(M_{z}^{\textrm{top}}+M_{z}^{\textrm{bot}})$ (12a)
$\displaystyle\Delta M$
$\displaystyle=|M_{z}^{\textrm{top}}-M_{z}^{\textrm{bot}}|.$ (12b)

Here, $M_{z}^{\textrm{top}}$ and $M_{z}^{\textrm{bot}}$ represent the averaged
out-of-plane magnetization in the top and bottom layers, respectively. They
are defined as:

$\displaystyle M_{z}^{\textrm{top}}$
$\displaystyle=\frac{1}{N_{\textrm{top}}}\sum_{i\in\textrm{top}}n_{i}^{z},$
$\displaystyle M_{z}^{\textrm{bot}}$
$\displaystyle=\frac{1}{N_{\textrm{bottom}}}\sum_{i\in\textrm{bottom}}n_{i}^{z},$
(13)

Here, $n_{i}^{z}$ denotes out-of-plane components of a normalized vector
$\bm{n}_{i}=(n_{i}^{x},n_{i}^{y},n_{i}^{z})$ for spin at $i$-site,
$N_{\textrm{top}}$ and $N_{\textrm{bottom}}$ denote the numbers of spins on
the top and bottom layers, respectively. The calculation results for $M$ and
$\Delta M$ are presented in Fig. 11(a–b). The magnetic ground states are
categorized based on the following criteria:

$\displaystyle\textrm{FM Phase}:~{}~{}M=1,~{}~{}\Delta M=0,$
$\displaystyle\textrm{NCD Phase}:M<1,~{}~{}\Delta M=0,$
$\displaystyle\textrm{MD Phase}:~{}~{}M<1,~{}~{}\Delta M>0.$ (14)

The classification of the magnetic ground states using Eq. (A) is presented in
Fig. 11(c)

Utilizing this result, we consistently omitted FM states from our dataset.
Consequently, we prepared 17,292 magnetic ground states within the NCD and MD
phases, which were utilized to generate 17,292 paired images in our dataset.
In the process of generating images, the supercell of moiré superlattices is
first repeated and then tailored to a fixed length dimension of 100 nm. This
regularization ensures uniformity across moiré superlattices of different
sizes. The generated images have dimensions of (200, 200).

##  Appendix B Neural network architectures

Figure 12: Neural network structure of the regression model. The model takes
two pixel arrays (‘Top’ and ‘Bottom’) as input and produces three parameters
$(\theta,J,A)$ as output. The model comprises three independent networks
(denoted as ‘network_1’, ‘network_2’, ‘network_3’) for regressing the three
parameters. Each network has two input layers (‘input_1’ and ‘input_2’), two
flattening layers (‘flatten’ and ‘flatten_1’), and four hidden layers (‘h1_1’,
‘h2_1’ and ‘h1_2’, ‘h2_2’), with two dedicated to each input. Additionally, it
has a concatenating layer (‘concatenate’), two more hidden layers (‘h3’ and
‘h4’) for the concatenated data, and an output layer (‘out’).

Figure 12 provides a detailed illustration of the neural network structure
employed in the regression model. The model incorporates three neural networks
(labeled as ‘network_1’, ‘network_2’, ‘network_3’) for regressing three
parameters $(\theta,J,A)$ from given two pixel arrays (‘Top True’ and ‘Bottom
True’). These arrays are derived from the first quadrant of full images sized
at (200, 200), obtained in the simulations. This preprocessing step ensures
efficiency in processing while maintaining consistency due to the four-fold
rotational symmetry inherent in each image. Each network accepts the two
arrays through dedicated input layers (‘input_1’ and ‘input_2’), flattening
them into two lists (‘concatenate’). Subsequently, the two lists undergo
independent processing through two consecutive hidden layers (‘h1_1’, ‘h2_1’
and ‘h1_2’, ‘h2_2’ for each list), ultimately reducing to 512 features each,
combined into a single list sized at 1024 (‘concatenate’). This list then
undergoes processing through two consecutive hidden layers (‘h3’, ‘h4’),
resulting in a further reduction to 512 features. The output layer (‘out’) of
the neural network regresses these features to predict a single parameter from
the set $(\theta,J,A)$.

Figure 13: Neural network structure of the generative model. The generative
model takes three parameters $(\theta,J,A)$ as input and produces two pixel
arrays (labeled as ‘Top’ and ‘Bottom’) as output. The network consists of an
input layer (denoted as ‘input_1’), two hidden layers (‘dense’ and ‘dense_1’),
and an output layer (‘dense_2’).

Figure 13 provides a detailed illustration of the neural network structure
employed in the generative model. The input layer (labeled as ‘input_1’)
processes a list of three parameters $(\theta,J,A)$. This input undergoes
further processing through two hidden layers (‘dense’ and ‘dense_1’) and the
output layer (‘dense_2’). The result is a list of 77,618 floating-point
numbers representing as many features. Subsequently, this output list is
reshaped into two pixel arrays with dimensions (197, 197), serving as the
predicted domain array images (‘Top Pred.’ and ‘Bottom Pred.’).


Abstraction via exemplars? A representational case study on lexical category
inference in BERT
Kanishka Misraτ,π and Najoung Kimβ
τThe University of Texas at Austin, πPurdue University, βBoston University

Linguistic abstractions are often posited as the mechanism by which language
learners generalize to novel expressions. However, Ambridge [1] argues that
such abilities arise not through linguistic abstraction but instead via stored
exemplars encountered during learning. One counterargument [2], among many
[3], takes inspiration from the unprecedented success of neural network
language models (LMs) such as BERT [4]. Specifically, LMs deviate from both
the radical exemplar account and the pure abstraction-based account—their
parameters store a summarized form of their training exemplars as guided by
their training objective (word-prediction in context), rather than storing
individual exemplars or encoding abstractions explicitly. Empirically, this
has shown to give rise to generalization behaviors aligning with abstract
linguistic structures [5]. Abstraction could therefore be viewed as arising
via the (compressed) encoding of exemplars.

In this work, we provide further empirical evidence for the abstraction-via-
exemplars account, by presenting a case study on lexical category inference in
BERT, an LM that is trained to predict missing words in fill-in-the-blank
(fitb) inputs (e.g., I  you. $\rightarrow$ “like”). We adapt the experiments
of Kim and Smolensky [6, hereafter K&S], who tested BERT’s generalization of
the usage of novel tokens belonging to either noun/verb/adj/adverb, from
exposure to a single observation Their tests are inspired by developmental
linguistic studies that probe for lexical category abstraction in children
[7]. Briefly, K&S used pairs (see Figure 1) of fitb stimuli that disambiguate
the category of the novel token and trained the model’s embeddings of the
novel tokens only (initialized with random-values), keeping all other
parameters constant. They then tested BERT’s accuracy for predicting each of
the novel tokens on a set of fitb inputs (N=100 per category) that are
lexically disjoint from the training examples, and found overall above-chance
performance indicating desirable generalization patterns. Using
representations learned by this method, we perform two analyses that relate
the representational change in the novel tokens’ embeddings with BERT’s
generalization patterns on the K&S experiment. First, we track the movement of
the embeddings in two-dimensional space (obtained using Principal Component
Analysis) as they are updated during training. Figure 2 suggests that the
final states of the embeddings of the novel tokens move closer in two-
dimensional space to that of known, unambiguous category exemplars (N=500 per
category). That is, regardless of initialization, wug (Figure 1) becomes more
noun-like, while dax becomes more adjective-like. To further analyze this
behavior, we take multiple novel tokens (N=20 per category) and assign them
values randomly sampled from regions in two-dimensional space populated by
category exemplars. We then project them into BERT’s embedding space, yielding
novel token representations with values informed by two-dimensional encoding
of category regions. Evaluating BERT to predict these novel tokens on the K&S
testing set—without any additional training—yields substantially above-chance
performance across multiple category pairs (Table 1).

Together, these analyses suggest that representational movement towards
regions of known category exemplars can result in behavior that is consistent
with successful generalization to novel expressions. Overall, this lends
empirical support to the claim that abstraction and exemplar based accounts
need not be at odds, and that abstraction-like behavior can emerge in learners
that encode training exemplars.

(498 words)

Figure 1:  Experimental setting proposed by Kim and Smolensky [6], illustrated
with noun vs. adj.  Figure 2:  Left: Relative movement of the novel token
representations with respect to known category exemplars for each category
after training on the K&S experiments. 0.0 indicates no movement. Right:
Movement (indicated by arrow) of a novel token’s two-dimensional
representation from its initial state in the adj–noun experiment. Points
indicate known, unambiguous adjectives and nouns.

Category Pair | Accuracy
---|---
adj–adverb | $0.93_{\pm 0.03}$
adj–verb | $0.70_{\pm 0.06}$
adverb–verb | $0.87_{\pm 0.05}$
noun–adj | $0.80_{\pm 0.08}$
noun–adverb | $0.89_{\pm 0.04}$
noun–verb | $0.81_{\pm 0.08}$

Table 1:  Accuracies (with 95% CI) on the test set of K&S [6] obtained by
randomly sampling values from two-dimensional regions of category-exemplars
(e.g., see Fig 1-Right) which are projected to serve as BERT embeddings for
novel, unseen tokens (N=20 each). Chance performance is 0.50.

## References

  * Ambridge [2020a] Ben Ambridge.  Against stored abstractions: A radical exemplar model of language acquisition.  _First Language_ , 40(5-6):509–559, 2020a.
  * Mahowald et al. [2020] Kyle Mahowald, George Kachergis, and Michael C Frank.  What counts as an exemplar model, anyway? A commentary on Ambridge (2020).  _First Language_ , 40(5-6):608–611, 2020.
  * Ambridge [2020b] Ben Ambridge.  Abstractions made of exemplars or ‘You’re all right, and I’ve changed my mind’: Response to commentators.  _First Language_ , 40(5-6):640–659, 2020b.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional Transformers for language understanding.  In _NAACL 2019_ , pages 4171–4186, 2019.
  * Goldberg [2019] Yoav Goldberg.  Assessing BERT’s syntactic abilities.  _arXiv preprint arXiv:1901.05287_ , 2019.
  * Kim and Smolensky [2021] Najoung Kim and Paul Smolensky.  Testing for grammatical category abstraction in neural language models.  _Proceedings of the Society for Computation in Linguistics_ , 4(1):467–470, 2021.
  * Höhle et al. [2004] Barbara Höhle, Jürgen Weissenborn, Dorothea Kiefer, Antje Schulz, and Michaela Schmitz.  Functional elements in infants’ speech processing: The role of determiners in the syntactic categorization of lexical elements.  _Infancy_ , 5(3):341–353, 2004.