[1,2]Samuel P. Gleason

[1]Jim Ciston

1]National Center for Electron Microscopy Facility, Molecular Foundry,
Lawrence Berkeley National Laboratory, Berkeley, CA, USA

2]Department of Chemistry, University of California, Berkeley, 94720, CA, USA

3]Center for Functional Nanomaterials, Brookhaven National Laboratory, Street,
Upton, 100190, NY, USA

# Prediction of the Cu Oxidation State from EELS and XAS Spectra Using
Supervised Machine Learning

<EMAIL_ADDRESS>Deyu Lu<EMAIL_ADDRESS>[  [  [

###### Abstract

Electron energy loss spectroscopy (EELS) and X-ray absorption spectroscopy
(XAS) provide detailed information about bonding, distributions and locations
of atoms, and their coordination numbers and oxidation states. However,
analysis of XAS/EELS data often relies on matching an unknown experimental
sample to a series of simulated or experimental standard samples. This limits
analysis throughput and the ability to extract quantitative information from a
sample. In this work, we have trained a random forest model capable of
predicting the oxidation state of copper based on its L-edge spectrum. Our
model attains an $R^{2}$ score of 0.89 and a root mean square valence error of
0.21 on simulated data. It has also successfully predicted experimental L-edge
EELS spectra taken in this work and XAS spectra extracted from the literature.
We further demonstrate the utility of this model by predicting simulated and
experimental spectra of mixed valence samples generated by this work. This
model can be integrated into a real time EELS/XAS analysis pipeline on
mixtures of copper containing materials of unknown composition and oxidation
state. By expanding the training data, this methodology can be extended to
data-driven spectral analysis of a broad range of materials.

###### keywords:

Machine Learning, EELS, XAS, Cu, Spectral Analysis

## Introduction

Due to their wide range of accessible oxidation states and materials
applications, the ability to determine the oxidation state of third row
transition metals is essential to a wide variety of applications. These
include the development of catalysts [1], photovoltaic devices [2], and
biotechnology [3]. Core level spectroscopy is often used to probe transition
metal oxidation states, and two main types are electron energy loss
spectroscopy (EELS) and X-ray absorption spectroscopy (XAS). EELS provides
detailed atomic scale information, such as oxidation state, coordination
number and local symmetry of a nanomaterial [4, 5]. When probing
nanomaterials, EELS is often combined with scanning transmission electron
microscopy (STEM). In STEM-EELS, an electron beam is scanned over an area of a
sample and a full spectrum is acquired and stored at each probe position. This
technique is particularly valuable in the study of nanomaterials due to its
combination of high spacial and high energy resolution. [6, 7, 8]. Like EELS,
XAS has also attained wide usage in determining oxidation state and local
environment in nanomaterials [9, 10, 11]. XAS, however, is typically limited
to a spacial resolution of a few nanometers [12], rather than the sub angstrom
spacial resolution possible with STEM-EELS [13]. The main advantages of XAS
compared to EELS for core-loss spectroscopy are the ability to attain higher
signal to noise ratios (SNR) and higher energy resolution, particularly at
higher excitation energies [14], and functionality on thicker samples for hard
x-ray excitation [15]. Due to the myriad use cases for both techniques, they
are commonly applied to the nanoscale study of materials containing third row
transition metals. However, since EELS and XAS spectra encode the electronic
properties of the sample in an abstract way, extracting physical descriptors
is a non-trivial task in spectral analysis.

Therefore, quantitative spectral analysis is often the rate limiting step in
materials characterization, and can typically only be conducted by trained
experts. This is especially true of L-edge spectra of transition metals, where
variations in oxidation state can manifest in small shifts in edge location,
L2/L3 ratio and peak width that are not immediately obvious to a non expert,
particularly for samples containing multiple oxidation states [16]. Oxidation
state assignment is typically done by mapping the unknown spectrum to known
experimental or simulated standards, a process which can be time intensive and
requires significant domain knowledge. Particularly challenging to analyze are
mixed valence materials, which are often interpreted as combinations of
spectra of integer valence structures [17]. The prevailing solution to this
problem is to fit integer valence spectra to the unknown spectrum using least
squares. This allows a user to input known standards and determine the
coefficients of a linear combination of the standard spectra that reproduce
the experimental spectrum [18, 19, 17, 20].

Least squares fitting has allowed quantitative oxidation state analysis of
mixed valence samples, and is widely implemented as the state-of-the-art
procedure for quantitative analysis of unlabeled XAS/EELS L-edge data.
However, in the case of experimental standards, it has a few serious
limitations. First, this procedure requires fresh standards to be taken for
each instrument, and often each day, as changes in detector setup and
alignment can lead to non trivial changes in the spectra. Second, this
procedure is highly sensitive to experimental variation in the acquisition of
the standard samples. Contamination with materials of other oxidation state,
surface oxidation and beam damage can have a significant impact on the shape
of the standard spectrum, and therefore interfere with the fitting of the
unknown sample. Additionally, inconsistencies in standard spectrum processing,
such as baseline subtraction or the incomplete deconvolution of multiple
scattering from the standard sample, can have a similar impact. Third, the
presence of any oxidation state or coordination environment unaccounted for by
the standards will not only be missed by the prediction of the makeup of the
material, potentially missing an important fundamental discovery, but will
also lead to an inaccurate representation of the oxidation state as the
standard components are forced to represent a signal not originating from any
of them. In a similar vein, experimental standards must be taken for every
material expected to be present in order to perform the oxidation state
analysis. For example, a standard for CuO may not be suitable for an
experiment involving CuS due to non trivial differences between the spectra,
although they are both a Cu(II) oxidation state [21, 22]. Simulated standards
suffer from fewer experimental limitations, but instead are limited by the
level of approximations used in the theory and often can not perfectly
reproduce experimental spectra. This can cause systematic errors leading to
significant misidentifications, particularly when applied to noisy
experimental spectra or experimental spectra more challenging to simulate. It
is rare for simulated standards for L-edge transition metal spectra to be
quantitatively accurate enough to fit an unknown experimental spectrum using
least squares fitting [23]. Instead, these are used to qualitatively match
components of an unknown spectrum. Therefore, there is a need for a procedure
that can determine oxidation state from XAS/EELS L-edge data that is more
robust than the least squares fitting of a handful of standard spectra.

An avenue for a more broadly applicable automated analysis procedure is
machine learning (ML). Despite some recent advancements in automated L-edge
XAS/EELS analysis of transition metals using ML approaches [24], overall, the
transition metal K-edge has received more focus from the ML community [25, 26,
27]. Numerical analysis of L-edge transition metal XAS/EELS data has mainly
been performed using principle component analysis (PCA) to reduce the
dimensionality of the spectrum. This field has been well developed, comprising
numerous applications of PCA on L-edge XAS/EELS data [28, 29, 30, 31].
Additionally, PCA dimenionality reduction procedures have been used to
successfully de-noise low SNR core loss EELS data [32, 33, 34]. PCA has also
been extended into analysis of oxidation states. Applying component analysis
to a mixed valence XAS/EELS spectrum can result in components that mimic the
unique oxidation states present. This can be used as a qualitative estimation
of the different oxidation states present in a sample, however, it is
difficult to ensure each of the resulting components match the pure form of an
oxidation state. Therefore, the lack of rigorous physical interpretation of
the components makes any quantitative analysis challenging [35].

Supervised machine learning approaches have found success predicting oxidation
states in manganese and iron samples, using neural networks and support vector
machines [36, 37, 38]. However, these models were trained on a small subset of
materials and, with the exception of [37] on Mn spectra, only focused on
integer valence states. Therefore, the more complicated question of L-edge
spectra oxidation state regression of an arbitrary Cu material containing a
wide range of oxidation states has not been thoroughly explored. The lack of
focus on mixed valence structures generally is especially notable, as such a
model is necessary to analyze an in-situ experiment where 1000s of spectra are
generated quickly with minor variations in oxidation state.

Figure 1: A flow chart containing the four components of constructing the
training data and random forest model. First, data is extracted from the
Materials Project and scaled, aligned and processed to ensure internal
consistency and accuracy to experiments. The colored boxes in I show how the
materials project classifies the materials extracted and simulated by this
work. Second, the spectra are labeled by their oxidation state using the
Materials Project oxidation state function ”get valence”. Third, the dataset
is augmented by creating mixture spectra made up of linear combinations of
integer valence spectra. Fourth, the random forest model is trained and
validated using test simulated data and experimental reference samples [39].

This work has developed a supervised ML model capable of conducting a
regression task on an unlabeled Cu L2,3-edge XAS/EELS spectrum and predicting
the average oxidation state. The L2,3-edge was selected as the focus due to
the prohibitively high energy of the transition metal K-edge for electron
detectors. We utilized the simulated L-edge XAS spectra of transition metals
stored in the Materials Project [40, 39] as a seed to construct our training
set. Despite the differing physical origins of XAS vs EELS, with XAS caused by
excitation from a photon and EELS by an electron, under the long wave-length
limit and dipole approximation, both spectroscopic methods involve evaluating
the same transition matrix element. Therefore, a model trained on XAS data is
able to effectively predict EELS data [41, 42] for features where the
quadrupole contribution is not significant.

Cu was selected as the focus of this work due to the myriad applications of Cu
nanomaterials. Specifically, Cu nanoparticles (CuNPs) are used in
antimicrobial agents [43], catalysts [44] and renewable energy devices,
particularly the electrochemical reduction of CO2 [45]. Examining the
oxidation state of Cu nanomaterials is critical to their function, as CuNP
preparation procedures can lead to unintended surface oxidation that disrupts
many of their applications [43]. Additionally, the major trends in Cu L-edge
spectra can be captured accurately in Cu metal, Cu2O and CuO using the
multiple scattering 111In this case multiple scattering refers to the
interference of multiple scattering paths, not to be confused with sequential
inelastic events originating from the same excitation source. method
implemented in the FEFF9 code [46, 40]. Figure S1a-c shows good agreement in
the L2-L3 spacing and well preserved intensity ratios between the L2 and L3
peaks. Fine detail such as the splitting of the L3 peak in Figure S1a is
demonstrated as well. The limitations of this method include the treatment of
the partially filled 3$d$ bands in Cu(II), where the many-body effects, such
as multiplet effects, require higher levels of theory beyond the mean-field
level [46, 23]. This can produce some spurious artifacts in the simulations,
such as the L3 shoulder in the CuO simulation (Figure S1c) which is not
present in the experimental sample. Although the quadrupole contribution can
play an important role in pre-edge features, distinct spectral features in the
main edge regions are found to be sensitive to the oxidation state from
feature importance analysis. Therefore, neglecting the quadrupole contribution
will not have a significant impact in this analysis. The overall success of
FEFF9 in producing Cu L-edge spectra allows Cu materials to serve as a model
system for this type of automated analysis procedure. In this work we present
a framework for predicting the Cu oxidation state that can be readily extended
to other transition metals by acquiring a volume of corresponding simulated
XAS data.

Figure 2: The performance of the random forest model on the test set of
simulated data. (a) $R^{2}$ plot, where each spot’s size is proportional to
the number of spectra at that point and its color corresponds to the
prediction’s standard deviation. (b) histogram of the absolute errors, with
the vertical green line showing the location of the root mean square error
(RMSE) and the vertical red line showing the location of the mean absolute
error (MAE).(c) feature importance of the random forest model plotted on the
same energy axis as the spectra.

## Results and Discussion

### Performance on Simulated Spectra

Our RF model shows a high level of accuracy on a test set of simulated data.
Figure 2a shows the $R^{2}$ plot of the predictions of this test set, which
contains roughly 2400 spectra. The $R^{2}$ for this model is 0.89, and shows a
visible high degree of correlation across all the well represented oxidation
states. The largest errors come from integer valence misprediction, most
commonly when a Cu(0) or Cu(I) spectrum is predicted as mixed valence.
However, as shown in Figure 2a, these mispredictions can often be
differentiated from the accurate predictions by using the prediction standard
deviation (described in the methods section). The feature importance plot from
Figure 2c offers insights into the origin of these errors. The model takes a
small amount of information from the pre-edge and then bases its prediction
mostly on the location and shape of the L3 peak. As Cu(0) and Cu(I) have L3
peaks at almost exactly the same energy, these are harder to differentiate
than Cu(II), which is red shifted by roughly 3 eV. Despite this difficulty,
Cu(0) and Cu(I) are accurately identified far more often than they are
mispredicted, as shown in Figure 2a. As can be seen from Figure 2b, a full
integer miss, i.e. a Cu(0) spectrum incorrectly called a Cu(I) spectrum,
essentially never occurs. What is even more encouraging in Figure 2a and 2b is
the simulated mixture samples are frequently predicted with a high degree of
accuracy, showing this model has significant potential in predicting mixed
valence samples. Figure S5 shows the model’s prediction on the individual
spectra used to build the mixed valence dataset, and illustrates that they are
a representative sample of our integer spectra, comprising many spectra
predicted accurately with a small number of larger misses between Cu(0) and
Cu(I). It should be noted that the construction of the mixture dataset
occurred before model training. Therefore, the random split of the entire
dataset, including mixtures, between training and testing data placed some of
the individual integer oxidation state spectra visualized in Figure S5 in the
augmented mixed-valent training set. In no cases were the same mixed, or
integer, spectra present in both the training and test sets.

### Model Uncertainty Metric

In this work we have developed a method for quantifying the uncertainty in our
RF model’s prediction. This is done by examining the predictions of each of
the 500 decision trees which comprise the random forest as well as the
averaged value used as the final prediction. This uncertainty analysis is
visualized by generating a prediction histogram, as shown in Figure 1 (IV) and
Figure 3d-3f. Beyond the qualitative spread of predictions shown in the
prediction histograms, the uncertainty can be understood quantitatively by
calculating the standard deviation of these predictions. This is indicated by
the horizontal green line in the prediction histogram plots shown in Figures 1
and 3, and is used here as the RF model’s internal uncertainty measurement. To
leverage this quantitative uncertainty, the standard deviation can be used to
filter out predictions that are highly uncertain, and therefore presumably
less accurate. Figure S6 illustrates this concept, where a standard deviation
threshold was imposed, and all predictions with a standard deviation higher
than this value were discarded due to their high uncertainty. The standard
deviation can be used as a powerful tool in determining significantly
inaccurate predictions on simulated data, as can be seen when the threshold is
set at 0.35 (red rectangle in Figure S6a and S6b). When this threshold is
used, 9% of the predictions of our test set are higher than the threshold and
discarded (Figure S6a). However, imposing this threshold causes the RMSE of
the remaining 91% of our test set to decrease 20% from the full test set value
of 0.21 to 0.17 (Figure S6b). Therefore, the 9% of the test set discarded by
this method is comprised of predictions less accurate than average by a
significant margin, showcasing the utility of this uncertainty metric in
informing the accuracy of the model’s predictions for unknown samples.

Figure 3: The performance of the random forest model on experimental Cu oxide
EELS standards collected in this work. The top row (a, b, c) shows the raw
spectrum with the cumulative spectrum as an insert. The bottom row (d, e, f)
shows the prediction histograms for each spectrum, where the grey bars
correspond to the number of decision trees predicting values over that range,
the red dashed line shows the prediction, the blue solid line shows the
labeled value, and the green horizontal line shows the standard deviation of
the individual decision trees predictions.

### Validation Using Experimental Spectra

To test the RF model’s validity when applied to experiments, we used the model
to predict a set of metallic Cu and Cu oxide standards. The simulated spectra
corresponding to these standards were left out of both the training and test
sets previously discussed. These standards were smoothed using a Savitzy-Golay
filter with a window size of 1.5 eV and a polynomial order of 3. From Figure
S10 it can be seen that the level of smoothing makes virtually no difference
in prediction accuracy, with the only small difference coming from Cu(0),
which is likely due to the high level of noise in the raw spectrum of that
sample (Figure S7). The smoothing window of 1.5 eV was selected as the default
method due to observations that it removed the vast majority of the noise but
also preserved the overall shape of the spectrum. From Figure 3e and 3f it can
be seen that the model has a high degree of accuracy when predicting Cu(I) and
Cu(II), rendering essentially perfect predictions for each of these standards.
However, Figure 3d shows the Cu(0) standard appears to be slightly over
estimated, as it is predicted at roughly 0.3. There are likely two factors
responsible for this observation. First, as has been discussed above, random
forest models average predictions across individual decision trees, in this
case 500. Therefore, it will always be more challenging for this model to
predict Cu(0) as exactly zero, as all materials have non-negative valence.
Consequently, any spread in the predictions will result in an overestimate. It
is worth noting that in Figure 2a this issue rarely manifests in our
predictions on test simulated data. However, due to increased uncertainty
around experimental spectra due to minor differences between the experimental
and simulated spectra, it is not unreasonable to suspect this feature could
prove to be a greater factor when applied to experiments. It is also worth
nothing that Figure 3d shows that the mode of our prediction histogram is
Cu(0) by a factor of two over the next highest bar. Additionally, a second
factor may also partially explain this overestimate, which is that our Cu(0)
likely experienced some surface oxidation. Therefore, it may be assumed that
this material no longer had a true oxidation state of zero at the time of
measurement. This is reflected in the spectrum, which can be seen to have
visibly taken on some additional Cu(I) character relative to simulated Cu(0)
and Cu(0) observed in XAS studies taken from the literature (Figure S8, [21])
Therefore, we believe that this prediction of a mixed valence material closer
to Cu(0) than Cu(I) matches our experimental realities and a detailed
examination of the experimental spectrum.

Figure 4: The impacts of energy misalignment on the prediction for EELS
spectra taken in this work (a, b, and c) and XAS spectra extracted from the
literature (d, e, f) [21]. The spectra are shifted horizontally on the energy
axis by the amount indicated in the x axis but are not changed in any other
way. The scatter plot color corresponds to the prediction’s standard
deviation.

Given that the edge position of our training data has been manually aligned to
our experimental spectra, it is reasonable to inquire how significant an
impact an energy misalignment will have. To explore this, we created a set of
experimental spectra where the onset energy was shifted by controlled amounts
and tracked how this shift impacted the oxidation state prediction (Figure 4).
From Figure 4a we see that the energy misalignment has the greatest impact on
the Cu(0) sample, and an offset of greater than -0.2 eV causes an inflection
point where the prediction jumps from 0.3 to nearly 0.5. Interestingly,
misalignment in the positive direction has a far less dramatic impact, and an
energy shift of +0.5 eV produces essentially no change in the prediction. The
opposite trend is observed in the Cu(I) sample from Figure 4b, where a
negative shift produces little change in the prediction, while an inflection
point occurs with a positive shift of greater than +0.2 eV. In Figure 4c,
however, we see that Cu(II)’s prediction is virtually independent of shift
plus/minus 1 eV, which is likely explained by the greater than 2 eV gap
between the onset energy of Cu(II) vs Cu(I) and Cu(0).

To further examine the utility of our model when applied to experimental
spectra, and to further study the impact of absolute energy shift on a model
that was aligned to our experimental samples, an additional experimental
validation was done using an extracted set of XAS spectra of Cu oxides [21].
This set of spectra has been measured to be -0.9 eV for Cu metal and -1.2 eV
for Cu2O and CuO (Figure S1) shifted from the experimental spectra used to
validate this model, and provides a test case for how the model will respond
to spectra with their energy axes significantly misaligned. From Figure 4d-f,
we can see that our ML model produces excellent results for the XAS spectra
when they are correctly aligned to our training data (red line in Figure 4d-f)
and the results are robust even when the raw spectra are predicted, which are
severely misaligned (black line in Figure 4d-f). When such a misalignment has
occurred, the Cu(I) and Cu(II) spectra are predicted with near perfect
accuracy, while the Cu(0) spectrum appears to be slightly over estimated,
returning a prediction of 0.4 when the correct alignment prediction is 0.25.
It is worth reflecting this prediction is still a slight overestimate,
although closer to zero than our experimental EELS spectrum shown in Figure
3d, reflecting this model’s slight propensity to overestimate Cu(0). With
these observations, it is clear that the ML model trained on properly aligned
spectra can achieve highly accurate results on spectra with significant energy
misalignment. Additionally, a potential avenue to determine the true alignment
location is to vary the energy axis and seek out regions of consistent
stability and low prediction standard deviation, as these regions are clearly
associated with more accurate predictions for all our experimental data.

Figure 5: Performance of the random forest model on experimental mixed valence
spectra. (a) mixtures of Cu(0) and Cu(I), while (b) shows mixtures of Cu(I)
and Cu(II). The scatter plot color corresponds to the prediction’s standard
deviation. The dashed line indicates the location of a perfect prediction.

### Prediction of Experimental Mixed Valence Samples

Post successful proof of concept for our model on standard experimental
samples, we turn our attention to a more valuable, but also more challenging,
experimental case, samples of mixtures of different oxidation states. As shown
in Figure 2a, our model has already demonstrated a high degree of accuracy on
simulated mixed valence samples. Additionally, we show how smooth variance in
simulated mixed valence materials excluded from the training data is captured
by our model by showing simulated mixtures of Cu(0), Cu(I) and Cu(II) in
Figure S7. The important test for the utility of this model in experimental
spectra is how well this process works on experimental mixtures of oxidation
states. Due to the difficulty in engineering a system with smoothly varying
mixed valence states, and inherent uncertainties in quantifying such a system,
we have generated mixed valence experimental spectra through linear
combinations of our standard samples. The labeled value for these experimental
mixtures is determined by multiplying their oxidation state by their
contribution to the final mixture spectrum, as was done with the labeling for
the simulated mixtures. However, in this case, the labeled oxidation state for
the standard samples was taken to be the prediction shown in Figure 3, rather
than the nominal value. For example, a mixture of 40% Cu(0) standard and 60%
Cu2O standard would be calculated as follows:

$0.31*0.4+1.08*0.6=0.772$ (1)

This is because our model predicts the Cu(0) standard to be 0.31 and the Cu2O
standard to be 1.08 and, as has already been mentioned above, we consider this
mixed valence value for our Cu(0) to be more representative than a label of
zero valence. The results are shown in Figure 5. From Figure 5a-b, we see both
plots contain regions of high accuracy, particularly for mixtures of Cu(I) and
Cu(II) (Figure 5b). These mixtures are accurately predicted to within less
than 0.1 in around half of the mixture samples. However, we can see that the
absolute accuracy has sections of low accuracy, particularly at inflection
points where the prediction is changing quickly. This is particularly true for
mixtures of Cu(0) and Cu(I) (Figure 5a), where the inflection region drives
the prediction into a region of overestimation which is not recovered until
the mixture becomes entirely Cu(I). However, the overall trend of the
prediction is correct, as in both Figure 5a and b the higher valence sample is
identified as such until a pure sample is predicted, regardless of any
absolute inaccuracies in the prediction. Both mixed valence cases tend
overestimate the oxidation state when the higher oxidation state sample
comprises greater than 50% of the mixture. Likely due to the similar locations
of the L3 peaks for Cu(0) and Cu(I), it has particular trouble differentiating
nearly even mixtures of these materials. For future work, it is possible to
introduce an additional empirical correction to the RF model’s prediction
based on the trends in Figure 5.

Figure 6: Random forest model performance on simulated data augmented by
Poisson noise. The standard deviation of the Poisson distribution used to
generate the noise is shown in the x axis of each plot. The error bars denote
the standard deviation of the RMSE/R2 across 100 random states for that noise
standard deviation value.

### Impact of Noise on Simulated Data

To test the impact of noise on the simulated data, random Poisson noise was
added to each simulated spectrum in the test set to produce a test set
augmented by noise. To ensure that this process echoed our approach on
experimental spectra as much as possible, the simulated spectra, which are on
a 0.1 eV resolution, were re-sampled using scipy’s 1d interpolation function
with a higher resolution of 0.03 eV, matching that of our experimental
samples. Noise was then added to the interpolated spectra, and these spectra
were then smoothed in the same method as the experimental spectra and
integrated to produce a cumulative spectrum (Figure S7). These spectra were
then predicted by the model to test its accuracy on noisy data.

As shown in Figure 6a, the simulated data are relatively sensitive to noise
augmentation, and the addition of a small amount of Poisson noise resulted in
an increase in RMSE from 0.21 to 0.3 as compared to results from the noiseless
spectra. Further increase in noise led to an even larger RMSE, however the
decline in accuracy becomes less sharp than the initial slope. A similar trend
is seen in Figure 6b for $R^{2}$, where a drop in $R^{2}$ is observed after
adding a small amount of noise, however this decline is less sharp than the
increase in RMSE, and adding additional noise has a more pronounced decline on
$R^{2}$ than subsequent noise does on RMSE. Despite this observation, the
noise level of our experimental spectra, which are noticeably larger than the
simulated low noise case, do not appear to suffer as much as these simulated
noisy spectra (Figure S7). Additionally, the selection of the random seed for
the addition of noise appears to have a significant impact on the overall
accuracy of the noisy test set. This is shown with the error bars in Figure 6a
and 6b, which represent the standard deviation across 100 different random
noise seed states. The presented RMSEs and $R^{2}$s are the average values
across these 100 random states. A detailed examination of the noise profiles
for these higher error random states shows that in these spectra the region
around the baseline experiences noise spikes that mimic features around the
baseline region, similar to how an inaccurate power law subtraction of an EELS
spectrum baseline appears. This observation further enforces that the accuracy
of this model relies heavily on the accurate identification and subtraction of
the baseline.

## Conclusion

In this work, we have built a random forest model trained on simulated L-edge
XAS spectra which is capable of predicting the oxidation state of copper based
on its L-edge XAS/EELS spectrum. We have also developed a database of Cu XAS
spectra containing 3500 unique materials that have been accurately aligned to
experimental spectra, and augmented this database with 6000 simulated mixture
spectra. Our random forest model attains an $R^{2}$ of 0.89 on simulated data
with an RMSE of 0.21 and has been shown to accurately predict experimental
spectra taken from our home institution and from the literature. Additionally,
this model has proven successful predicting mixed valence samples, showing its
applicability to track Cu oxidation state in in-situ experiments where
oxidation state is changing fluidly as a reaction occurs. Beyond this model’s
utility to Cu materials, we have also developed a broader methodology which
can be extended to the analysis of other materials by acquiring a spectral
database of accurate simulated L-edge spectra for the corresponding material.

## Methods

### Training Set Generation

In this work, simulated FEFF9 XAS spectra of Cu materials were extracted from
the Materials Project. This initial extraction produced a dataset of site
averaged spectra for 1533 materials, which contains the 59 materials shown in
Figure 1I labeled as neither predicted stable nor synthesized [40]. To
increase the volume of our training data, an additional 2000 structures were
selected by searching the Materials Project for all Cu containing materials
that had either been previously synthesized or were predicted to be stable by
theory [39]. This choice screens a broad material space that is likely
accessible to experiments. We computed 2000 site averaged spectra using the
_Lightshow_ workflow [47] and FEFF9 [46]. The combination of this augmentation
step and the initial extraction of L-edge spectra already generated by the
Materials Project provided 1199 materials that both have been experimentally
synthesized and are predicted to be stable (Figure 1I). For each structure,
unique Cu sites are determined by the space group symmetry. Then site specific
spectra were calculated using FEFF9. The L2 and L3 spectra for each site were
combined into the L2,3 spectrum by summing the L2 and L3 spectra, after first
interpolating onto the same energy grid (Figure S2). The site averaged
spectrum is calculated from the weighted sum of site-specific spectra based on
the multiplicity of the unique sites in the unit cell. The oxidation state of
the site specific spectra were determined using the Materials Project’s “get
valences” function [39]. Despite this averaging procedure, greater than 93% of
the site averaged spectra retained integer valence. When FEFF9 failed to
converge for some, but not all, of the sites in a material, converged site
spectra were averaged leaving out the failed spectra.

To prepare our training set of 3500 site averaged spectra, several additional
steps were performed. This workflow is summarized in Figure 1. First, spectra
were interpolated to ensure they were all on a 0.1 eV energy resolution.
Second, the non uniformity in the energy range of the L3 edge, specifically at
the starting point, was addressed by fitting a 6th order polynomial to connect
the lowest energy point to [925, 0] (i.e., vanishing intensity at 925 eV) for
every spectrum (see Figure S3). The spectra were then aligned to ensure their
onset edges matched those seen by experimental EELS Cu materials, as it was
observed that there was a systematic misalignment occurring in the absolute
energy of the L2,3 edge.

To accomplish this alignment, two systematic errors were corrected. First, a
high degree of onset energy variability was observed within an oxidation
state, especially for zero valence materials. Second, the absolute energy of
the simulated spectrum was several eV off from experimental standards. Both of
these issues were fixed simultaneously by our automated alignment procedure.
By subtracting the predicted Fermi energy from the simulated spectra and
adding an experimental reference energy taken from our home instrument,
correct energy alignments across the three main oxidation states for Cu
materials, Cu(0), Cu(I) and Cu(II) were established. The experimental
reference energies were obtained using three reference standards (Cu metal,
Cu2O, and CuO) taken in this work, and the same shift values were used for all
materials with the same oxidation state in our spectral database. For the
small subset of materials that were classified as mixed valence, they were
aligned based on whichever integer oxidation state they were closest to.

Our spectral dataset was then augmented by generating simulated mixed valence
samples (see step III in Figure 1, Figure S4). To accomplish this, 300 random
sets of spectra were drawn from the integer dataset, each draw taking a random
Cu(0), Cu(I) and Cu(II) site averaged spectrum. Each of these 300 sets of 3
integer spectra were then linearly combined to mimic mixed valence structures.
For each set of three spectra, 20 random fractions of each material were
combined to produce a simulated mixed valence spectrum. To ensure an even
spread of mixed valences, 100 sets were combinations of Cu(0) and Cu(I), 100
were combinations of Cu(I) and Cu(II), and 100 were combinations of Cu(0),
Cu(I) and Cu(II). This mixture produced a final dataset of roughly 9500
spectra with data well distributed from Cu(0) to Cu(II) (step III in Figure 1,
Figure S4).

To achieve the best ML model performance, we have tested different spectral
representations, including the spectrum itself, its first and second
derivative, and the cumulative integral of the spectrum. We found that the
best model performance was achieved with the cumulative integral with
intensity normalized to 1. In addition, using the cumulative integral,
referred to as a cumulative spectrum from this point on, as input feature can
ensure consistency in the absolute scale of the EELS spectrum. This
representation can simplify intensity scaling, as experimental post processing
decisions and noise can create a high degree of variability in spectral
intensity. The cumulative spectrum approach is insensitive to the absolute
scale of the spectrum, although it does require an accurate identification and
subtraction of the baseline for experimental spectra.

### Random Forest Modeling

Random forest (RF) models for this work were trained using Scikit-learn’s
RandomForestRegressor model [48]. The number of trees was fixed at 500, with
all features available and max depth unfixed. The dataset was split into train
and test components using a 75/25 random train test split function from
Scikit-learn. The structure of this model allows for the input of a raw
spectrum of arbitrary min and max energy and energy scale. The model then
takes the input spectrum and interpolates it to a 0.1 eV resolution from 925
to 970 eV to ensure the consistency of the energy grid used in the training
data. Spectral smoothing is then applied using a Savitzy-Golay filter from
scipy [49]. The smoothing step is done before the interpolation provided that
the inputted spectrum is on an evenly spaced energy scale. The cumulative
operation on the spectrum is then performed and this spectrum is the input of
the model. The trained RF model is an ensemble of 500 individually trained
decision trees, and returns the predictions of each decision tree. A simple
average of inferred valence values from each tree is taken as the final
prediction. The standard deviation of these 500 predictions can approximate
the model’s internal confidence in its prediction, and is visualized in the
prediction histogram in Figures 1, 3 and S7, the last of which illustrates the
entirety of the processing steps performed on an input spectrum.

### Experimental EELS

To validate the utility of this model on experimental data, experimental EELS
spectra of standard reference samples were measured, including Cu metal, Cu2O
and CuO. Cu metal was purchased from Sigma-Aldrich with 99.999% purity. Cu2O
and CuO were purchased from Sigma-Aldrich with 99.99% purity. The Cu2O sample
was measured using a vacuum holder to prevent oxidation. However, the Cu metal
sample was not delivered in a vacuum sealed container, and under the
assumption that surface oxidation had already occurred, a vacuum holder was
not used for this sample. Using the TEAM I microscope, a double-corrected
Thermo Fisher Titan microscope, we acquired monochromated reference data for
these samples at roughly 0.2 meV resolution. Data were collected at 300kV with
a semi-convergence angle of 17 mrad and a collection angle of 82 mrad. All
data was collected using a Gatan Continuum spectrometer equipped with a K3-IS
detector operated in electron counting mode. Spectra were baseline subtracted
using the GMS Digital Micrograph software package, and spectra were taken
using dual EELS to dynamically remove shifts in the reference elastic energy
and deconvolved with the simultaneously measured zero-loss region to mitigate
artifacts from electrons experiencing multiple scattering events. The
deconvolution of multiple scattering is essential to ensure the experimental
EELS spectra are comparable to XAS.

## Data and Code Availability

The spectral dataset and the code to generate and analyze the random forest
model presented in this study are available upon request.

## Author Contributions

SPG took the experimental data, generated the simulated XAS dataset, conducted
the machine learning training and analysis and wrote the manuscript. DL
provided training and expertise necessary to generate the simulated dataset.
JC provided experimental EELS knowledge, microscope training, led the
collaboration and designed the scope of this work. All authors read, edited
and approved the final manuscript.

## Acknowledgment

This work was primarily funded by the US Department of Energy in the program
“4D Camera Distillery: From Massive Electron Microscopy Scattering Data to
Useful Information with AI/ML.” Work at the Molecular Foundry was supported by
the Office of Science, Office of Basic Energy Sciences, of the U.S. Department
of Energy under Contract No. DE-AC02-05CH11231. This research used Theory and
Computation resources of the Center for Functional Nanomaterials (CFN), which
is a U.S. Department of Energy Office of Science User Facility, at Brookhaven
National Laboratory under Contract No. DE-SC0012704. The training and
instrumentation support necessary to operate the TEAM I microscope and acquire
the experimental EELS data presented in this work was provided by Dr Chengyu
Song.

## References

  *   * Dalle et al. [2019] Dalle, K.E., Warnan, J., Leung, J.J., Reuillard, B., Karmel, I.S., Reisner, E.: Electro- and Solar-Driven Fuel Synthesis with First Row Transition Metal Complexes. Chemical Reviews 119(4), 2752–2875 (2019) https://doi.org/10.1021/acs.chemrev.8b00392
  * Mccusker [2019] Mccusker, J.K.: Electronic structure in the transition metal block and its implications for light harvesting. Science 363, 484–488 (2019)
  * Johnstone et al. [2015] Johnstone, T.C., Suntharalingam, K., Lippard, S.J.: Third row transition metals for the treatment of cancer. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 373(2037) (2015) https://doi.org/10.1098/rsta.2014.0185
  * Batson [1993] Batson, P.E.: Simultaneous STEM imaging and electron energy-loss spectroscopy with atomic-column sensitivity. Nature 366, 727–728 (1993)
  * Browning et al. [1993] Browning, N.D., Chisholm, M.F., Pennycook, S.J.: Atomic-resolution chemical analysis using a scanning transmission electron microscope. Nature 366, 143–146 (1993)
  * Yang et al. [2014] Yang, G., Cheng, S., Li, C., Zhong, J., Ma, C., Wang, Z., Xiang, W.: Investigation of the oxidation states of Cu additive in colored borosilicate glasses by electron energy loss spectroscopy. Journal of Applied Physics 116(22) (2014) https://doi.org/10.1063/1.4903955
  * Gazquez et al. [2017] Gazquez, J., Sanchez-Santolino, G., Biskup, N., Roldan, M.A., Cabero, M., Pennycook, S.J., Varela, M.: Applications of STEM-EELS to complex oxides. Elsevier Ltd (2017). https://doi.org/10.1016/j.mssp.2016.06.005
  * Kociak and Stephan [2014] Kociak, M., Stephan, O.: Mapping plasmons at the nanometer scale in an electron microscope. Royal Society of Chemistry (2014). https://doi.org/10.1039/c3cs60478k
  * Bai et al. [2022] Bai, Y., Wu, Y., Zhou, X., Ye, Y., Nie, K., Wang, J., Xie, M., Zhang, Z., Liu, Z., Cheng, T., Gao, C.: Promoting nickel oxidation state transitions in single-layer NiFeB hydroxide nanosheets for efficient oxygen evolution. Nature Communications 13(1) (2022) https://doi.org/10.1038/s41467-022-33846-0
  * Kubin et al. [2018] Kubin, M., Guo, M., Kroll, T., Lochel, H., Kallman, E., Baker, M.L., Mitzner, R., Gul, S., Kern, J., Fohlisch, A., Erko, A., Bergmann, U., Yachandra, V., Yano, J., Lundberg, M., Wernet, P.: Probing the oxidation state of transition metal complexes: a case study on how charge and spin densities determine Mn L-edge X-ray absorption energies. Chemical Science 9(33), 6813–6829 (2018) https://doi.org/10.1039/c8sc00550h
  * Henderson et al. [2014] Henderson, G.S., Groot, F.M.F., Moulton, B.J.A.: X-ray absorption near-edge structure (XANES) spectroscopy. In: Spectroscopic Methods in Mineralogy and Materials Sciences, pp. 75–138. De Gruyter, ??? (2014). https://doi.org/10.2138/rmg.2014.78.3
  * Yu et al. [2021] Yu, L., Li, M., Wen, J., Amine, K., Lu, J.: (S)TEM-EELS as an advanced characterization technique for lithium-ion batteries. Royal Society of Chemistry (2021). https://doi.org/10.1039/d1qm00275a
  * Batson et al. [2002] Batson, P.E., Dellby, N., Krivanek, O.L.: Sub-aångstrom resolution using aberration corrected electron optics. Nature 418(6898), 617–620 (2002) https://doi.org/10.1038/nature00972
  * Hart et al. [2023] Hart, J.L., Lang, A.C., Li, Y., Shahrezaei, S., Alix-Williams, D.D., Falk, M.L., Mathaudhu, S.N., Frenkel, A.I., Taheri, M.L.: Revealing local order via high energy EELS. Materials Today Nano 21 (2023) https://doi.org/10.1016/j.mtnano.2022.100298
  * Akgul et al. [2014] Akgul, G., Akgul, F.A., Ufuktepe, Y.: Investigation of thickness dependence on electronic structures of iron and nickel thin films by L-edge X-ray absorption spectroscopy. Vacuum 99, 211–215 (2014) https://doi.org/10.1016/j.vacuum.2013.06.003
  * Keast et al. [2001] Keast, V.J., Scott, A.J., Brydson, R., Williams, D.B., Bruley, J.: Electron energy-loss near-edge structure a tool for the investigation of electronic structure on the nanometre scale. Journal of Microscopy 203, 135–175 (2001)
  * Cressey et al. [1993] Cressey, G., Henderson, C.M.B., Van Der Laan, G.: Use of L-edge X-ray Absorption Spectroscopy to Characterize Multiple Valence States of 3d Transition Metals; a New Probe for Mineralogical and Geochemical Research. Phys Chem Minerals 20, 111–119 (1993)
  * Zhang et al. [2010] Zhang, S., Livi, K.J.T., Gaillot, A.C., Stone, A.T., Veblen, D.R.: Determination of manganese valence states in (Mn3+, Mn 4+) minerals by electron energy-loss spectroscopy. American Mineralogist 95(11-12), 1741–1746 (2010) https://doi.org/10.2138/am.2010.3468
  * Van Aken and Liebscher [2002] Van Aken, P.A., Liebscher, B.: Quantification of ferrous/ferric ratios in minerals: New evaluation schemes of Fe L23 electron energy-loss near-edge spectra. Physics and Chemistry of Minerals 29(3), 188–200 (2002) https://doi.org/10.1007/s00269-001-0222-6
  * Laurence A. J. Garvie and Peter R. Buseck [1998] Laurence A. J. Garvie, Peter R. Buseck: Ratios of ferrous to ferric iron from nanometre-sized areas in minerals. letters to nature 396, 667–670 (1998)
  * Grioni et al. [1992] Grioni, M., Van Acker, J.F., Czyzyk, M.T., Fuggle, J.C.: Unoccupied electronic structure and core-hole effects in the x-ray-absorption spectra of Cu20. PHYSICAL REVIEW B 45, 3309–3318 (1992)
  * Shang et al. [2020] Shang, H., Zhou, X., Dong, J., Li, A., Zhao, X., Liu, Q., Lin, Y., Pei, J., Li, Z., Jiang, Z., Zhou, D., Zheng, L., Wang, Y., Zhou, J., Yang, Z., Cao, R., Sarangi, R., Sun, T., Yang, X., Zheng, X., Yan, W., Zhuang, Z., Li, J., Chen, W., Wang, D., Zhang, J., Li, Y.: Engineering unsymmetrically coordinated Cu-S1N3 single atom sites with enhanced oxygen reduction activity. Nature Communications 11(1) (2020) https://doi.org/10.1038/s41467-020-16848-8
  * Groot [2005] Groot, F.D.: Multiplet effects in X-ray spectroscopy (2005). https://doi.org/10.1016/j.ccr.2004.03.018
  * Timoshenko et al. [2017] Timoshenko, J., Lu, D., Lin, Y., Frenkel, A.I.: Supervised Machine-Learning-Based Determination of Three-Dimensional Structure of Metallic Nanoparticles. Journal of Physical Chemistry Letters 8(20), 5091–5098 (2017) https://doi.org/10.1021/acs.jpclett.7b02364
  * Carbone et al. [2019] Carbone, M.R., Yoo, S., Topsakal, M., Lu, D.: Classification of local chemical environments from x-ray absorption spectra using supervised machine learning. Physical Review Materials 3(3) (2019) https://doi.org/10.1103/PhysRevMaterials.3.033604
  * Torrisi et al. [2020] Torrisi, S.B., Carbone, M.R., Rohr, B.A., Montoya, J.H., Ha, Y., Yano, J., Suram, S.K., Hung, L.: Random forest machine learning models for interpretable X-ray absorption near-edge structure spectrum-property relationships. npj Computational Materials 6(1) (2020) https://doi.org/%****␣main.bbl␣Line␣500␣****10.1038/s41524-020-00376-6
  * Zheng et al. [2018] Zheng, C., Mathew, K., Chen, C., Chen, Y., Tang, H., Dozier, A., Kas, J.J., Vila, F.D., Rehr, J.J., Piper, L.F.J., Persson, K.A., Ong, S.P.: Automated generation and ensemble-learned matching of X-ray absorption spectra. npj Computational Materials 4(1) (2018) https://doi.org/10.1038/s41524-018-0067-x
  * Bonnet [1998] Bonnet, N.: Multivariate statistical methods for the analysis of microscope image series: applications in materials science. Journal of Microscopy 190, 2–18 (1998)
  * Noel Bonnet [2000] Noel Bonnet: Artificial Intelligence and Pattern Recognition Techniques in Microscope Image Processing and Analysis. Advances in imaging and electron physics 114, 1–77 (2000)
  * Bosman et al. [2006] Bosman, M., Watanabe, M., Alexander, D.T.L., Keast, V.J.: Mapping chemical and bonding information using multivariate analysis of electron energy-loss spectrum images. Ultramicroscopy 106(11-12 SPEC. ISS.), 1024–1032 (2006) https://doi.org/10.1016/j.ultramic.2006.04.016
  * Kotula and Keenan [2006] Kotula, P.G., Keenan, M.R.: Application of multivariate statistical analysis to STEM X-ray spectral images: Interfacial analysis in microelectronics. In: Microscopy and Microanalysis, vol. 12, pp. 538–544 (2006). https://doi.org/10.1017/S1431927606060636
  * Cueva et al. [2012] Cueva, P., Hovden, R., Mundy, J.A., Xin, H.L., Muller, D.A.: Data processing for atomic resolution electron energy loss spectroscopy. In: Microscopy and Microanalysis, vol. 18, pp. 667–675 (2012). https://doi.org/%****␣main.bbl␣Line␣600␣****10.1017/S1431927612000244
  * Bonnet and Nuzillard [2005] Bonnet, N., Nuzillard, D.: Independent component analysis: A new possibility for analysing series of electron energy loss spectra. Ultramicroscopy 102(4), 327–337 (2005) https://doi.org/10.1016/j.ultramic.2004.11.003
  * Bosman et al. [2007] Bosman, M., Keast, V.J., García-Muñoz, J.L., D’Alfonso, A.J., Findlay, S.D., Allen, L.J.: Two-dimensional mapping of chemical information at atomic resolution. Physical Review Letters 99(8) (2007) https://doi.org/10.1103/PhysRevLett.99.086102
  * Li et al. [2023] Li, R., Jiang, X., Zhou, C., Topsakal, M., Nykypanchuk, D., Attenkofer, K., Stacchiola, D.J., Hybertsen, M.S., Stavitski, E., Qu, X., Lu, D., Liu, M.: Deciphering phase evolution in complex metal oxide thin films via high-throughput materials synthesis and characterization. Nanotechnology 34(12) (2023) https://doi.org/10.1088/1361-6528/acad09
  * Chatzidakis and Botton [2019] Chatzidakis, M., Botton, G.A.: Towards calibration-invariant spectroscopy using deep learning. Scientific Reports 9(1) (2019) https://doi.org/10.1038/s41598-019-38482-1
  * Ji et al. [2023] Ji, Z., Hu, M., Xin, H.L.: MnEdgeNet for accurate decomposition of mixed oxidation states for Mn XAS and EELS L2,3 edges without reference and calibration. Scientific Reports 13(1), 14132 (2023) https://doi.org/10.1038/s41598-023-40616-5
  * del Pozo-Bueno et al. [2023] del-Pozo-Bueno, D., Kepaptsoglou, D., Peiró, F., Estradé, S.: Comparative of machine learning classification strategies for electron energy loss spectroscopy: Support vector machines and artificial neural networks. Ultramicroscopy 253, 113828 (2023) https://doi.org/10.1016/j.ultramic.2023.113828
  * Jain et al. [2013] Jain, A., Ong, S.P., Hautier, G., Chen, W., Richards, W.D., Dacek, S., Cholia, S., Gunter, D., Skinner, D., Ceder, G., Persson, K.A.: Commentary: The materials project: A materials genome approach to accelerating materials innovation. American Institute of Physics Inc. (2013). https://doi.org/10.1063/1.4812323
  * Chen et al. [2021] Chen, Y., Chen, C., Zheng, C., Dwaraknath, S., Horton, M.K., Cabana, J., Rehr, J., Vinson, J., Dozier, A., Kas, J.J., Persson, K.A., Ong, S.P.: Database of ab initio L-edge X-ray absorption near edge structure. Scientific Data 8(1) (2021) https://doi.org/%****␣main.bbl␣Line␣750␣****10.1038/s41597-021-00936-5
  * R.F. Egerton [2011] R.F. Egerton: Electron Energy-Loss Spectroscopy in the Electron Microscope, 3rd edn. Springer, ??? (2011)
  * Moreno et al. [2007] Moreno, M.S., Jorissen, K., Rehr, J.J.: Practical aspects of electron energy-loss spectroscopy (EELS) calculations using FEFF8. Micron 38(1), 1–11 (2007) https://doi.org/10.1016/j.micron.2006.03.011
  * Bhagat et al. [2021] Bhagat, M., Anand, R., Sharma, P., Rajput, P., Sharma, N., Singh, K.: Review—Multifunctional Copper Nanoparticles: Synthesis and Applications. ECS Journal of Solid State Science and Technology 10(6), 063011 (2021) https://doi.org/10.1149/2162-8777/ac07f8
  * Gawande et al. [2016] Gawande, M.B., Goswami, A., Felpin, F.X., Asefa, T., Huang, X., Silva, R., Zou, X., Zboril, R., Varma, R.S.: Cu and Cu-Based Nanoparticles: Synthesis and Applications in Catalysis. American Chemical Society (2016). https://doi.org/10.1021/acs.chemrev.5b00482
  * Laffont et al. [2006] Laffont, L., Wu, M.Y., Chevallier, F., Poizot, P., Morcrette, M., Tarascon, J.M.: High resolution EELS of Cu-V oxides: Application to batteries materials. Micron 37(5), 459–464 (2006) https://doi.org/10.1016/j.micron.2005.11.007
  * John J. Rehr et al. [2010] John J. Rehr, Joshua J. Kas, Fernando D. Vila, Micah P. Prange, Kevin Jorissen: Parameter-free calculations of X-ray spectra with FEFF9. Physical Chemistry Chemical Physics 12, 5503–5513 (2010)
  * Carbone et al. [2023] Carbone, M.R., Meng, F., Vorwerk, C., Maurer, B., Peschel, F., Qu, X., Stavitski, E., Draxl, C., Vinson, J., Lu, D.: Lightshow: a Python package for generating computational x-ray absorption spectroscopy input files. Journal of Open Source Software 8(87), 5182 (2023) https://doi.org/10.21105/joss.05182
  * Pedregosa et al. [2011] Pedregosa, F., Michel, V., Grisel OLIVIERGRISEL, O., Blondel, M., Prettenhofer, P., Weiss, R., Vanderplas, J., Cournapeau, D., Pedregosa, F., Varoquaux, G., Gramfort, A., Thirion, B., Grisel, O., Dubourg, V., Passos, A., Brucher, M., Perrot andÉdouardand, M., Duchesnay, a., Duchesnay EDOUARDDUCHESNAY, F.: Scikit-learn: Machine Learning in Python Gaël Varoquaux Bertrand Thirion Vincent Dubourg Alexandre Passos PEDREGOSA, VAROQUAUX, GRAMFORT ET AL. Matthieu Perrot. Journal of Machine Learning Research 12, 2825–2830 (2011)
  * Virtanen et al. [2020] Virtanen, P., Gommers, R., Oliphant, T.E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., Walt, S.J., Brett, M., Wilson, J., Millman, K.J., Mayorov, N., Nelson, A.R.J., Jones, E., Kern, R., Larson, E., Carey, C.J., Polat, .I., Feng, Y., Moore, E.W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E.A., Harris, C.R., Archibald, A.M., Ribeiro, A.H., Pedregosa, F., Mulbregt, P., SciPy 1.0 Contributors: SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods 17, 261–272 (2020) https://doi.org/10.1038/s41592-019-0686-2