EGraFFBench: Evaluation of Equivariant Graph Neural Network Force Fields
  for Atomistic Simulations
Equivariant graph neural networks force fields (EGraFFs) have shown great
promise in modelling complex interactions in atomic systems by exploiting the
graphs' inherent symmetries. Recent works have led to a surge in the
development of novel architectures that incorporate equivariance-based
inductive biases alongside architectural innovations like graph transformers
and message passing to model atomic interactions. However, thorough evaluations
of these deploying EGraFFs for the downstream task of real-world atomistic
simulations, is lacking. To this end, here we perform a systematic benchmarking
of 6 EGraFF algorithms (NequIP, Allegro, BOTNet, MACE, Equiformer, TorchMDNet),
with the aim of understanding their capabilities and limitations for realistic
atomistic simulations. In addition to our thorough evaluation and analysis on
eight existing datasets based on the benchmarking literature, we release two
new benchmark datasets, propose four new metrics, and three challenging tasks.
The new datasets and tasks evaluate the performance of EGraFF to
out-of-distribution data, in terms of different crystal structures,
temperatures, and new molecules. Interestingly, evaluation of the EGraFF models
based on dynamic simulations reveals that having a lower error on energy or
force does not guarantee stable or reliable simulation or faithful replication
of the atomic structures. Moreover, we find that no model clearly outperforms
other models on all datasets and tasks. Importantly, we show that the
performance of all the models on out-of-distribution datasets is unreliable,
pointing to the need for the development of a foundation model for force fields
that can be used in real-world simulations. In summary, this work establishes a
rigorous framework for evaluating machine learning force fields in the context
of atomic simulations and points to open research challenges within this
domain.
Vaibhav Bihani, Utkarsh Pratiush, Sajid Mannan, Tao Du, Zhimin Chen, Santiago Miret, Matthieu Micoulaut, Morten M Smedskjaer, Sayan Ranu, N M Anoop Krishnan
# EGraFFBench: Evaluation of Equivariant Graph Neural Network Force Fields for Atomistic Simulations

###### Abstract

Equivariant graph neural networks force fields (EGraFFs) have shown great promise in modeling complex interactions in atomic systems by exploiting the graphs' inherent symmetries. Recent works have led to a surge in the development of novel architectures that incorporate equivariance-based inductive biases alongside architectural innovations like graph transformers and message passing to model atomic interactions. However, thorough evaluations of these deploying EGraFFs for the downstream task of real-world atomistic simulations, are lacking. To this end, here we perform a systematic benchmarking of 6 EGraFF algorithms (NequiP, Allegro, BOTNet, MACE, Equiformer, TorchMDNet), to understand their capabilities and limitations for realistic atomistic simulations. In addition to our thorough evaluation and analysis of eight existing datasets based on the benchmarking literature, we release two new benchmark datasets, propose four new metrics, and three challenging tasks. The new datasets and tasks evaluate the performance of EGraFF to out-of-distribution data, in terms of different crystal structures, temperatures, and new molecules. Interestingly, evaluation of the EGraFF models based on dynamic simulations reveals that having a
lower error on energy or force does not guarantee stable or reliable simulation or faithful replication of the atomic structures. Moreover, no model clearly outperforms other models on all datasets and tasks. Importantly, we show that the performance of all the models on out-of-distribution datasets is unreliable, pointing to the need to develop a foundation model for force fields that can be used in real-world simulations. In summary, this work establishes a rigorous framework for evaluating machine learning force fields in the context of atomic simulations and points to open research challenges within this domain.

## 1 Introduction

Graph neural networks (Gnns) have emerged as powerful tools for learning representations of graph-structured data, enabling breakthroughs in various domains such as social networks, mechanics, drug discovery, and natural language processing (Perozzi et al., 2014; Wu et al., 2020; Zhang and Chen, 2018; Stokes et al., 2020; Zhou et al., 2020; Miret et al., 2023; Lee et al., 2023). In the field of atomistic simulations, Gnn force fields have shown significant promise in capturing complex interatomic interactions and accurately predicting the potential energy surfaces of atomic systems (Park et al., 2021; Sanchez-Gonzalez et al., 2020; Schutt et al., 2021; Qiao et al., 2021). These force fields can, in turn, be used to study the dynamics of atomic systems--that is, how the atomic systems evolve with respect to time--enabling several downstream applications such as drug discovery, protein folding, stable structures of materials, and battery materials with targeted diffusion properties.

Recent work has shown that Gnn force fields can be further enhanced and made data-efficient by enforcing additional inductive biases, in terms of equivariance, leveraging the underlying symmetry of the atomic structures. This family of Gnns, hereafter referred to as equivariant graph neural network force fields (EGraFFs), have demonstrated their capability to model symmetries inherent in atomic systems, resulting in superior performance in comparison to other machine-learned force fields. This is achieved by explicitly accounting for symmetry operations, such as rotations and translations, and ensuring that the learned representations in EGraFFs are consistent under these transformations.

Traditionally, EGraFFs are trained on the forces and energies based on first principle simulations data, such as density functional theory. Recently work has shown that low training or test error does not guarantee the performance of the EGraFFs for the downstream task involving atomistic or molecular dynamics (MD) simulations (Fu et al., 2023). Specifically, EGraFFs can suffer from several major issues such as (i) unstable trajectory (the simulation suddenly explodes/becomes unstable due to high local forces), (ii) poor structure (the structure of the atomic system including the coordination, bond angles, bond lengths is not captured properly), (iii) poor generalization to out-of-distribution datasets including simulations at different temperatures or pressures of the same system, simulations of different structures having the same chemical composition--for example, crystalline (ordered) and glassy (disordered) states of the same system, or simulations of different compositions having the same chemical components--for example, Li\({}_{4}\)P\({}_{2}\)S\({}_{6}\) and Li\({}_{7}\)P\({}_{3}\)S\({}_{11}\). Note that these are realistic tasks for which a force field that is well-trained on one system can generalize to other similar systems. As such, an extensive evaluation and comparison of EGraFFs is needed, which requires standardized datasets, well-defined metrics, and comprehensive benchmarking, that capture the diversity and complexity of atomic systems.

An initial effort to capture the performance of machine-learned force fields was carried out (Fu et al., 2023). In this work, the authors focused on existing datasets and some metrics, such as radial distribution functions and diffusion constants of atomic systems. However, the work did not cover the wide range of EGraFFs that has been newly proposed, many of which have shown superior performance on common tasks. Moreover, the metrics in Fu et al. (2023) were limited to stability, mean absolute error of forces radial distribution function, and diffusivity. While useful, these metrics either do not capture the variations during the dynamic simulation (e.g., how the force or energy error evolves during simulation) or require long simulations (such as diffusion constants, which requires many steps to reach the diffusive regime). Further, the work does not propose any novel tasks that can serve as a benchmark for the community developing new force fields.

With the increasing interest in EGraFFs for atomic simulations, we aim to address the gap in benchmarking by performing a rigorous evaluation of the quality of simulations obtained using modern EGraFF force fields. To this extent, we evaluate 6 EGraFFs on 10 datasets, including two new challenging datasets that we contribute, and propose new metrics based on real-world simulations. By employing a diverse set of atomic systems and benchmarking metrics, we aim to objectively and rigorously assess the capabilities and limitations of EGraFFs. The main contributions of this research paper are as follows:

* **EGraFFs:** We present a benchmarking package to evaluate \(6\) EGraFFs for atomistic simulations. As a byproduct of this benchmarking study, we release a well-curated codebase of the prominent Equivariant Gnnforce fields in the literature enabling easier and streamlined access to relevant modeling pipelines
* **Challenging benchmark datasets:** We present \(10\) datasets, including two new datasets, namely GeTe and LiPS20. The datasets cover a wide range of atomic systems, from small molecules to bulk systems. The datasets capture several scenarios, such as compounds with the same elements but different chemical compositions, the same composition with different crystal structures, and the same structure at different temperatures. This includes complex scenarios such as melting trajectories of crystals.
* **Challenging downstream tasks:** We propose several challenging downstream tasks that evaluate the ability of EGraFFs to model the out-of-distribution datasets described earlier.
* **Improved metrics:** We propose additional metrics that evaluate the quality of the atomistic simulations regarding the structure and dynamics with respect to the ground truth.

## 2 Preliminaries

Every material consists of atoms that interact with each other based on the different types of bondings (e.g., covalent and ionic). These bonds are approximated by force fields that model the atomic interactions. Here, we briefly describe atomistic simulations and the equivariant Gnns used for modeling these systems.

### Atomistic simulation

Consider a set of \(N\) atoms represented by a point cloud corresponding to their position vectors \((r_{1},r_{2},\ldots,r_{N})\) and their types \(\omega_{i}\). Specifically, the potential energy of a system can be written as the summation of one-body \(U(r_{i})\), two-body \(U(r_{i},r_{j})\), three-body \(U(r_{i},r_{j},r_{k})\), up to \(N\)-body interaction terms as

\[U=\sum_{i=1}^{N}U(r_{i})+\sum_{\begin{subarray}{c}i,j=1;\\ i\neq j\end{subarray}}^{N}U(r_{i},r_{j})+\sum_{\begin{subarray}{c}i,j,k=1;\\ i\neq j\neq k\end{subarray}}^{N}U(r_{i},r_{j},r_{k})+\cdots \tag{1}\]

Since the exact computation of this potential energy is challenging, they are approximated using empirical force fields that learn the effective potential energy surface as a function of two-, three-, or four-body interactions. In atomistic simulations, these force fields are used to obtain the system's energy. The forces on each particle are then obtained as \(F_{i}=-\partial U/\partial r_{i}\). The acceleration of each atom is obtained from these forces as \(F_{i}/m_{i}\) where \(m_{i}\) is the mass of each atom. Accordingly, the updated position is computed by numerically integrating the equations of motion using a symplectic integrator. These steps are repeated to study the dynamics of atomic systems.

### Equivariant Gnn force fields (EGraFF)

Gnns are widely used to model the force field due to the topological similarity with atomic systems. Specifically, nodes are considered atoms, the edges represent interactions/bonds, and the energy or force is predicted as the output at the node or edge levels. Equivariant Gnns employ a message passing scheme that is equivariant to rotations, that is, \(G(Rx)=RG(x)\), where \(R\) is a rotation and \(G\) is an equivariant transformation (see Fig.1). This enables a rich representation of atomic environments equivariant to rotation. Notably, while the energy of an atomic system is invariant to rotation (that is, a molecule before and after rotation would have the same energy), the force is equivariant to rotation (that is, the forces experienced by the molecules due to the interactions also get rotated when the molecule is rotated).

## 3 Models Studied

All EGraFFs employed in this work rely on equivariance in the graph structure. All models use a one-hot encoding of the atomic numbers \(Z_{i}\) as the node input and the position vector \(r_{i}\) as a node or edge input. Equivariance in these models is ensured by the use of spherical harmonics along with radial basis functions. The convolution or message-passing implementation differs from model to model. Further hyperparameters details for all models are tabulated in App. A.10

**NeuPIP [Batzner et al., 2022]**, based on the tensor field networks, employs a series of self-interaction, convolution, and concatenation with the neighboring atoms. The convolution filter \(S_{m}^{l}(\vec{r}_{ij})=R(|\vec{r}_{ij}|)\times Y_{m}^{l}(\vec{r}_{ij}/|\vec{r} _{ij}|)\) represented as a product of radial basis function \(R\) and spherical harmonics \(Y_{m}^{l}\) ensures equivariance. This was the first EGraFF proposed for atomistic simulations based on spherical harmonics.

Figure 1: Equivariant transformation \(G\) on a molecule under rotation \(R\).
**Allegro Musaelian et al.**[**2022**]** merges the precision of recent equivariant Gnns with stringent locality, without message passing. Its inherent local characteristic enhances its scalability for potentially more extensive systems. In contrast to other models, Allegro predicts the energy as a function of the final edge embedding rather than the node embeddings. All the pairwise energies are summed to obtain the total energy of the system. Allegro features remarkable adaptability to data outside the training distribution, consistently surpassing other force fields in this aspect, especially those employing body-ordered strategies.

**BOTNet**Batatia et al. [2022a] is a refined body-ordered adaptation of NequIP. While maintaining the two-body interactions of NequIP in each layer, it increments the body order by one with every iteration of message passing. Unlike NequIP, BOTNet uses non-linearities in the update step.

**MACE**Batatia et al. [2022b] offers efficient equivariant messages with high body order computation. Due to the augmented body order of the messages, merely two message-passing iterations suffice to attain notable accuracy. This contrasts with the usual five or six iterations observed in other Gnns, rendering MACE both scalable and amenable to parallelization.

**TorchMDNet**Tholke and Fabritiis [2022] introduces a transformer-based Gnn architecture, utilizing a modified multi-head attention mechanism. This modification expands the traditional dot-product attention to integrate edge data, which can enhance the learning of interatomic interactions.

**Equformer**[Liao and Smidt, 2023] is a transformer-based GNN architecture, introducing a new attention mechanism named 'equivariant graph attention'.This mechanism equips conventional attention used in the transformers with equivariance.

**PaiNN**[Schutt et al., 2021] is a polarizable atom interaction neural network consisting of equivariant message passing architecture that takes into account the varying polarizability of atoms in different chemical environments, allowing for a more realistic representation of molecular behavior.

**DimeNet++**[Gasteiger et al., 2020] is a directional message passing neural network where each rotationally equivariant message is associated with a direction in coordinate space.

## 4 Benchmarking Evaluation

In this section, we benchmark the above-mentioned architectures and distill the insights generated. The evaluation environment is detailed in App. A.8.

### Datasets

Since the present work focuses on evaluating EGraFFs for molecular dynamics (MD) simulations, we consider only datasets with included time dynamics--i.e., all the datasets represent the dynamics of atom (see Fig. 2). We consider a total of 10 datasets (see Tab. 10 and App. A.1).

**MD17** is a widely used Batzner et al. [2022], Liao and Smidt [2023], Batatia et al. [2022a,b], Tholke and Fabritiis [2022], Fu et al. [2023] dataset for benchmarking ML force fields. It was proposed by Chmiela et al. [2017] and constitutes a set of small organic molecules, including benzene, toluene, naphthalene, ethanol, uracil, and aspirin, with energy and forces generated by ab initio MD simulations (AIMD). Here, we select four molecules, namely aspirin, ethanol, naphthalene, and salicylic acid, to cover a range of chemical structures and topology. Further, zero-shot evaluation is performed on benzene. We train the models on 950 configurations and validate them on 50.

**3BPA** contains a large flexible drug-like organic molecule 3-(benzyloxy)pyridin-2-amine (3BPA) sampled from different temperature MD trajectories Kovacs et al. [2021]. It has three consecutive rotatable bonds leading to a complex dihedral potential energy surface with many local minima, making it challenging to approximate using classical or ML force fields. The models can be trained either on 300 K snapshots or on mixed temperature snapshots sampled from 300 K, 600 K, and 1200 K. In the following experiments, we train models on 500 configurations sampled at 300 K

Figure 2: Visualisation of datasets. (a) GeTe\({}_{4}\), (b) LiPS20, (c) 3BPA, (d) Acetylacetone, (e) MD17.
and test 1669 configurations sampled at 600 K.

**LiPS** consists of lithium, phosphorous, and sulfur (\(Li_{6.75}P_{3}S_{11}\)), which is used in similar benchmarking analysis Fu et al. (2023), as a representative system for the MD simulations to study kinetic properties in materials. Note that LiPS is a crystalline (ordered structure) that can potentially be used in battery development. We have adopted this dataset from (Batzner et al., 2022)and benchmarked all models for their force and energy errors. The training and testing datasets have 19000 and 1000 configurations, respectively.

**Acetylacetone (AcAc)** The dataset was generated by conducting MD simulations at both 300K and 600K using a Langevin thermostat(Batatia et al., 2022). The uniqueness of this dataset stems from the varying simulation temperatures and the range of sampled dihedral angles. While the training set restricts sampling to dihedral angles below 30\({}^{\circ}\), our models are tested on angles extending up to 180\({}^{\circ}\). The model must effectively generalize on the Potential Energy Surface (PES) for accurate generalization at these higher angles. This challenge presents an excellent opportunity for benchmarking GNNs. The dataset consists of 500 training configurations and 650 testing configurations.

**GeTe** is a new dataset generated by a Car-Parrinello MD (CPMD) simulations Hutter (2012) of Ge and Te atoms, which builds on a density functional theory (DFT) based calculation of the interatomic forces, prior to a classical integration of the equations of motions. It consists of 200 atoms, of which 40 are Ge and 160 are Te, i.e., corresponding to the composition GeTe\({}_{4}\) whose structural properties have been investigated in detail and reproduce a certain number of experimental data in the liquid and amorphous phase from neutron/X-ray scattering Micoulaut et al. (2014); Guanasekera et al. (2014) and Mossbauer spectroscopy Micoulaut et al. (2014). As GeTe belongs to the promising class of phase-change materials Zhang et al. (2019), it is challenging to simulate using classical force fields because of the increased accessibility in terms of time and size. Thus, an accurate force field is essential to understand the structural changes in GeTe during the crystalline to disordered phase transitions. Here, our dataset consists of 1,500 structures in training, 300 in test, and 300 in validation.

**LiPS20** is a new dataset generated from AIMD simulations of a series of systems containing Li, P, and S elements, including both the crystalline and disordered structures of elementary substances and compounds, such as Li, P, S, Li\({}_{2}\)P\({}_{2}\)S\({}_{6}\), \(\beta\)-Li\({}_{3}\)PS\({}_{4}\), \(\gamma\)-Li\({}_{3}\)PS\({}_{4}\), and \(x\)Li\({}_{2}\)S-(\(100-x\))P\({}_{2}\)S\({}_{5}\) (\(x=67,70,75,\) and \(80\)) glasses using the CP2K package Kuhne et al. (2020). Details of dataset generation, structures, and compositions in this dataset are given in App. A.2.

### Evaluation metrics

Ideally, once trained, the forward simulations by EGraFFs should be close to the ground truth (first principle simulations) both in terms of the atomic structure and dynamics. To this extent, we propose four metrics. Note that these metrics are evaluated based on the forward simulation, starting from an arbitrary structure for \(n\) steps employing the force fields; a task for which it is not explicitly trained. All the forward simulations were performed using the Atomic Simulation Environment (ASE) package (Larsen et al., 2017). The simulations were conducted in the canonical (\(NVT\)) ensemble, where the temperature and timesteps were set in accordance with the sampling conditions specified in the respective datasets. See details in App. A.3

#### 4.2.1 Structure metrics

We propose two metrics to evaluate the proximity of structures predicted by the EGraFF to the ground truth.

**Wright's Factor (WF)**, \(R_{\chi}\)**: Grimley et al. (1990) represents the relative difference between the radial distribution function (RDF) of the ground truth atomic structure (\(g_{ref}(r)\)) and the structure obtained from the atomistic simulations employing the EGraFFs (\(g(r)\)) as

\[R_{\chi}=\left[\frac{\sum_{i=1}^{n}\left(g(r)-g_{\mathrm{ref}}(r)\right)^{2}}{ \sum_{i=1}^{n}\left(g_{\mathrm{ref}}(r)\right)^{2}}\right] \tag{2}\]

RDF essentially represents the local time-averaged density of atoms at a distance \(r\) from a central atom (see App. A.4). Hence, it captures the structure simulated by a force field concisely and one-dimensionally. A force field is considered acceptable if it can provide a WF less than 9% for bulk systems Bauchy (2014).

**Jensen-Shannon Divergence(JSD) of radial distribution function:** Jensen-Shannon Divergence (JSD) Cover and Thomas (1991), Shannon (1948) is a useful tool for quantifying the difference or similarity between two probability distributions in a way that overcomes some of the limitations of the KL DivergenceKullback and Leibler (1951). Since the RDF is essentially a distribution of the atomic density, JSD between two predicted RDF and ground truth RDF can be computed as:

\[\text{JSD}(g(r)\parallel g_{ref}(r))=\frac{1}{2}\left(\text{KL}(g(r)\parallel \hat{g}(r))+\text{KL}(g_{ref}(r)\parallel\hat{g}(r))\right) \tag{3}\]

where \(\hat{g}(r)=1/2(g(r)+g_{ref}(r))\) is the mean of the predicted and ground-truth RDFs. (see App. A.4)
#### 4.2.2 Dynamics metrics

We monitor the energy and force error over the forward simulation trajectory to evaluate how close the predicted dynamics are to the ground truth. Specifically, we use the following metrics, namely, **energy violation error, EV(t)**, and **force violation error, FV(t)**, defined as:

\[EV(t)=\frac{(\hat{E}(t)-E(t))^{2}}{\hat{E}(t)^{2}+E(t)^{2}},\text{and}\quad FV(t)= \frac{\|\mathcal{\hat{F}}(t)-\mathcal{F}(t)\|_{2}}{\left(\|\mathcal{\hat{F}}(t )\|_{2}+\|\mathcal{F}(t)\|_{2}\right)} \tag{4}\]

where \(\hat{E}(t)\) and \(E(t)\) are the predicted and ground truth energies respectively and \(\mathcal{\hat{F}}(t)\) and \(\mathcal{F}(t)\) are the predicted and ground truth forces. Note that this metric ensures that the energy and the force violation errors are bounded between 0 and 1, with 0 representing exact agreement with the ground truth and 1 representing no agreement. Further, we compute the geometric mean of \(EV(t)\) and \(FV(t)\) over the trajectory to represent the cumulative \(EV\) and \(FV\).

### Results

#### 4.3.1 Energy and Forces

To evaluate the performance of the trained models on different datasets, we first compute the mean absolute error in predicting the energy and force (see Table 1). First, we observe that no single model consistently outperforms others for all datasets, highlighting the dataset-specific nature of the models. TorchMDNet model has notably lower energy error than other models for most datasets, while NguiIP has minimum force error on majority of datasets with low energy error. On bulk systems such as LiPS and LiPS20, MACE and BOTNet show the lowest energy error. Interestingly, GeTe, the largest dataset in terms of the number of atoms, exhibits significant energy errors across all models, with the Equipment having the lowest energy error. Equipment also exhibits lower force error for datasets like Acetylacetone, 3BPA, and MD17, but suffers high force error on GeTe, LiPS, and LiPS20. Overall, NguiIP seems to perform well in terms of both energy and force errors for several datasets. It is also interesting to note that the models exhibiting low energy error often exhibit high force error, suggesting that the gradient of energy is not captured well by these models. This will potentially lead to poor simulations as the updated positions are computed directly from the forces.

#### 4.3.2 Forward simulations

To evaluate the ability of the trained models to simulate realistic structures and dynamics, we perform MD simulations using the trained models, which are compared with ground truth simulations, both employing the same initial configuration. For each model, five forward simulations of 1000 timesteps are performed on each dataset. Tables 2 and 3 show the JSD and WF, and EV and FV, respectively, of the trained models on the datasets (see App. A.5, A.6 and A.7 for figures). Both in terms of JSD and WF, we observe that NguiIP performs better on most datasets. Interestingly, even on datasets where other models have lower MAE on energy and force error, NguiIP performs better in capturing the atomic structure. Altogether, we observe that NguiIP followed by TorchMDNet performs best in capturing the atomic structure for most datasets. We now evaluate the models' EV and FV during the forward simulation. Interestingly, we observe that NguiIP and Allegro exhibit the least FV for most datasets, while MACE and BOTNet perform better in terms of EV. Interestingly, TorchMDNet, despite having the lowest MAE on energy for most datasets, does not exhibit low EV, indicating that having low MAE during model development does not guarantee low energy error during MD simulation.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**NopicIT**} & \multicolumn{2}{c}{**Alllegro**} & \multicolumn{2}{c}{**BOTNET**} & \multicolumn{2}{c}{**SIXEC**} & \multicolumn{2}{c}{**Geulptororie**} & \multicolumn{2}{c}{**TotchMDNet**} & \multicolumn{2}{c}{**FAN**} & \multicolumn{2}{c}{**DineNet\(\star\)**} \\  & **R** & **F** & **E** & **F** & **E** & **F** & **E** & **F** & **E** & **F** & **E** & **F** & **E** & **F** & **F** & **F** \\ \hline
**Aerojunction** & 1.38 & 4.59 & **1002** & 44.4 & 2.0 & 1.0 & 2.0 & 5.0 & 4.0 & 4.0 & **2.0** & 3.0 & 1.0 & 5.0 & 42.7 & 102.0 & 15.28 \\
**3BPA** & 3.25 & **3.26** & **4.13** & 10.0 & 5.0 & 14.0 & 4.0 & 12.0 & 6.0 & **2.0** & **3.0** & **3.0** & 1.0 & 38.67 & 40.41 & 396.74 & 46.72 \\
**Aspita** & 6.84 & 13.89 & **2007** & 9.17 & 9.9 & 14.0 & 8.5 & 14.0 & 6.5 & 15.0 & 5.3 & 5.3 & 5.3 & 5.9 & 44.9 & 12.41 & 133.34 & 22.07 \\
**Emanuel** & 2.57 & 7.49 & **234** & 5.01 & 2.6 & 6.0 & 6.0 & 2.6 & **3.9** & 2.6 & 6.9 & 9.3 & 2.6 & 5.9 & 7.7 & 11.81 & 499.58 & 17.19 \\
**Sophimation** & 5.70 & 6.20 & 5.44 & 2.64 & 6.57 & 6.5 & 6.0 & 6.8 & **19.5** & 5.8 & 7.0 & **10.5** & 4.03 & 10.56 & 4.01 & 175.04 & 19.65 \\
**Safe Aid** & 5.78 & 8.42 & 5.76 & **6.30** & 5.6 & 5.6 & 1021 & 5.34 & **1.24** & 5.2 & 12.9 & 6.5 & 7.9 & 24.5 & 11.2 & 169.18 & 25.48 \\
**LAPS** & 165.43 & 15.00 & 34.15 & **12.05** & **280.0** & 13.0 & 5.0 & 32.4 & **23.4** & 5.2 & 11.9 & 6.5 & 7.9 & 24.5 & 11.2 & 169.18 & 25.48 \\
**LAPS20** & 165.80 & **3.04** & 31.75 & **32.5** & **280.0** & **13.0** & 30.9 & 15.0 & 33.7 & 51.0 & 6.0 & 6.0 & 128.80 & 112.43 & 55.22 & 42.23 \\
**LAPS20** & 265.80 & **3.04** & 33.17 & 3.14 & 5.49 & **5.90** & **140.0** & 4.84 & 7224.9 & 5.73 & 6.24 & 0.7 & 51.9 & \\
**GeTe** & 1780.951 & **1144.00** & 10004.23 & 250.43 & 250.40 & 250.70 & 247.00 & **6003.7** & 363.17 & 2613.0 & 371.01 & 844.28 & 330.65 & 51704.65 & 222.39 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Energy (E) and force (F) mean absolute error in meV and meV/A, respectively, for the trained models on different datasets. Darker colors represent the better-performing models. We use shades of green and blue color for energy and force, respectively.
#### 4.3.3 Training and inference time

Table 4 shows different models' training and inference time. MACE and TorchMDNet have the lowest per epoch training time. The total training time is higher for transformer models TorchMDNet and Equiformer because of the larger number of epochs required for training. Although NequiIP and Allegro require more time per epoch, they get trained quickly in fewer epochs. LiPS dataset, having the largest dataset size in training of around 20000, has the largest per epoch training time. Since MD simulations are generally performed on CPUs, we report inference time as a mean over five simulations for 1000 steps performed on a CPU. TorchMDNet is significantly fast on all the datasets while Allegro and MACE show competitive performance. A visual analysis of the models on these metrics are given in App. A.7.

### Challenging tasks on EGraFF

#### 4.4.1 Generalizability to unseen structures

The first task focuses on evaluating the models on an unseen small molecule structure. To this extent, we test the models, trained on four molecules of the MD17 dataset (aspirin, ethanol, naphthalene, and salicylic acid), on the benzene molecule, an unseen molecule from the MD17 dataset. Note that the benzene molecule has a cyclic ring structure. Aspirin and Salicylic acid contain one ring, naphthalene is polycyclic with two rings, while ethanol has a chain structure with no rings. Table 5 shows the EV and FV and Table 6 shows the corresponding JSD and WF. We observe that all the models suffer very high errors in force and energy. Equiformer trained on ethanol and salicylic acid exhibits unstable simulation after the first few steps. Interestingly, non-cyclic ethanol models perform better than aspirin and salicylic acid, although the latter structures are more similar to benzene. Similarly, the model trained on polycyclic Naphthalene performs better than other models. Altogether, we observe that despite having the same chemical elements, models trained on one small molecule do not generalize to an unseen molecule with a different structure.

#### 4.4.2 Generalizability to higher temperatures

At higher temperatures, the sampling region in the energy landscape widens; hence, the configurations obtained at higher temperatures come from a broader distribution of structural configurations. In the 3BPA molecule, at 300K, only the stable dihedral angle configurations are present, while at 600K, all configurations are sampled. Here, we

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**NequiIP**} & \multicolumn{2}{c}{**Alllegro**} & \multicolumn{2}{c}{**BOUTNet**} & \multicolumn{2}{c}{**MACE**} & \multicolumn{2}{c}{**Equiformer**} & \multicolumn{2}{c}{**TorchMDNet**} & \multicolumn{2}{c}{**Pash**} & \multicolumn{2}{c}{**DimeNET+**} \\  & **JSD** & **WF** & **JSD** & **WF** & **JSD** & **WF** & **JSD** & **WF** & **JSD** & **WF** & **JSD** & **WF** & **JSD** & **WF** & **JSD** & **WF** \\ \hline
**Acrylenet** & 28.24 & 24.55 & 29.63 & 22.71 & 30.61 & 26.04 & 31.07 & 29.30 & 29.36 & 21.28 & 29.34 & 22.49 & 28.99 & 22.33 & 6.00 & 143.36 \\
**3BPA** & **0.82** & 6.12 & 11.9 & 7.98 & 1.07 & 7.13 & 99.88 & 3.09 & 9.74 & 7.44 & 0.87 & 7.31 & 1.39 & 6.65 & 8.87 & 89.97 \\
**Aspirin** & 0.133 & 30.65 & **0.108** & 23.29 & 0.122 & 27.36 & 0.111 & 18.92 & 0.120 & 23.58 & 0.131 & 23.99 & 10.09 & 4.01 & 0.31 & 167.48 \\
**Ethanol** & 0.526 & 18.34 & 0.159 & **0.580** & **0.500** & **15.57** & 0.944 & 19.33 & 0.549 & 23.48 & 0.464 & 17.70 & 0.78 & **0.42** & 37.55 & 20.57 \\
**Napirin** & 0.089 & 20.96 & 0.082 & 19.44 & 0.031 & 24.65 & 0.095 & 22.89 & 0.090 & 26.72 & 0.081 & 19.25 & 0.08 & 2.62 & 0.23 & 130.40 \\
**Salicylic Acid** & 0.077 & 16.95 & 0.124 & 27.58 & 0.076 & 14.55 & 0.097 & 19.55 & 0.092 & 14.17 & 0.07 & 16.12 & 0.08 & 6.31 & 0.00 & 20.85 \\
**LAPS** & 0.00 & 3.899 & 0.0 & 3.57 & 0.0 & 3.93 & 0.3 & 3.66 & 0.0 & 1.97 & 0.0 & 1.49 & 0.0 & 0.51 & 0.0 & 28.55 \\
**LiPS20** & 0.001 & 14.92 & 0.001 & 18.32 & 0.001 & 17.08 & 0.001 & 17.70 & 2.00 & 0.006 & 41.70 & - & 0.006 & 41.70 & - & 0.007 \\
**Gerfc** & 0.00 & 27.8 & 0.0 & 2026.0 & 0.0 & 20.3 & 0.0 & 0.0 & 20.0 & - & - & 0.00 & 2.80 & 0.0 & 2.29 & 0.0 & 16.77 \\ \hline \hline \end{tabular}
\end{table}
Table 2: JSD and WF for six EGraFFs on all the datasets. The values are computed as the average of five forward simulations for 1000 timesteps on each dataset with different initial conditions.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**NequiIP**} & \multicolumn{2}{c}{**Alllegro**} & \multicolumn{2}{c}{**BOUTNet**} & \multicolumn{2}{c}{**MACE**} & \multicolumn{2}{c}{**Equiformer**} & \multicolumn{2}{c}{**TorchMDNet**} & \multicolumn{2}{c}{**Pash**} & \multicolumn{2}{c}{**DimeNET+**} \\ \hline
**Acrylenet** & 28.09 & 0.390 & 0.50 & 0.710 & 0.737 & 0.715 & 0.813 & 0.715 & 0.810 & 0.711 & 0.830 & 0.715 & 0.820 & 17.15 & 0.820 & 17.87 & 0.205 \\
**Slightlight** & 0.931 & 0.961 & 0.972 & 0.941 & 0.931 & 0.931 & 0.921 & 0.901 & 0.941 & 0.930 & 0.920 & 0.920 & 0.920 & 0.920 & 0.920 & 0.920 \\
**3BPA** & 0.800 & 0.711 & 0.729 & 0.710 & 0.710 & 0.711 & 0.710 & 0.700 & 0.803 & 0.720 & 0.854 & 0.700 & 0.803 & 0.710 & 19.22 & 0.230 \\
**Aspirin** & 0.204 & 0.202 & 0.027 & 0.023 & 0.023 & 0.027 & 0.023 & 0.027 & 0.020 & 0.330 & 0.027 & 0.300 & 0.027 & 0.440 & 0.031 & 0.300 & 0.020 \\
**Aspirin** & 0.351 & 0.081 & 0.950 & 0.813 & 0.857 & 0.878 & 0.048 & 0.855 & 0.830 & 0.850 & 0.850 & 0.850 & 0.850 & 0.850 & 0.850 \\
**Ethanol** & 0.257 & 0.011 & 0.741 & 0.729 & 0.723 & 0.723 & 0.726 & 0.015 & 0.865 & 0.867 & 0.860 & 0.860 & 0.860 & 0.860 & 0.860 \\
**Napirin** & 0.24 & 0.45 & 0.238 & 0.000 & 1.534 & 0.899 & 0.600 & 0.534 & 0.860 & 0.860 & 0.860 & 0.860 & 0.860 & 0.860 & 0.860 \\
**Slightlight +** & 0.258 & 0.258 & 0.268 & 0.002 & 0.001 & 0.001 & 0.949 & 0.600 & 0.530 & 0.860 & 0.860 & 0.860 & 0.860 & 0.860 & 0.860 & 0.860 \\
**Slight +** & 2.155 & 0.455 & **0.095** & **0.004** & **0.04** & **0.02** & **0.02** & **0.04** & **0.00** & **0.00** & **0.00** & **0.00** & **0.00** & **0.00** & **0.00** & **0.00** & **0.00** \\
**LAPS** & 0.008 & 0.008 & 0.008 & 0.008 & 0.004 & 0
evaluate the model trained at lower temperatures for simulations at higher temperatures. Table 7 shows the obtained mean energy and force violation of the forward simulation trajectory, and Table 8 shows the corresponding JSD and WF. We observe that the models can reasonably capture the behavior, both structure and dynamics, at higher temperatures.

#### 4.4.3 Out of distribution tasks on the LIPS20 dataset

**Unseen crystalline structures:** Crystal structures are stable low-energy structures with inherent symmetries and periodicity. Predicting their energy accurately is an extremely challenging task and a cornerstone in materials discovery. Here, we train the models on liquid (disordered) structures and test them on the out-of-distribution crystalline structures to evaluate their generalizability capabilities. Table 9 shows that BOTNet performs appreciably well with almost the same energy and force error on crystal structures as the obtained training error. Both the transformer models have poor performance on the LiPS20 system, in terms of both the training and testing datasets. TorchMDNet has significantly high energy and force errors, whereas Equiformer exhibits instability during the forward simulation.

**Generalizability to unseen composition:** The LiPS20 dataset consists of 20 different compositions with varying system sizes and cell geometries (see App. A.2). In Tables (a)a(a) and (b)b, we show the results on the test structures that are not present in the training datasets. The test dataset consists of system sizes up to 260 atoms, while the models were trained on system sizes with \(<\) 100 atoms. It tests the models' generalization as well as inductive capability. We observe that MACE and BOTNet have the lowest mean energy, force violation, and low WF. NequIP and Allegro have significantly higher test errors.

## 5 Concluding Insights

In this work, we present EGrAFBench, a benchmarking suite for evaluating machine-learned force fields. The key insights drawn from the extensive evaluation are as follows.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{NequIP} & \multicolumn{2}{c}{AllLegro} & \multicolumn{2}{c}{BOTNet} & \multicolumn{2}{c}{MACE} & \multicolumn{2}{c}{Equiformer} & \multicolumn{2}{c}{TorchMDNet} \\  & JSD & WF & JSD & WF & JSD & WF & JSD & WF & JSD & WF & JSD & WF \\ \hline Aspirin & 2368483 & 73.801 & 573090 & **61.518** & **311916** & 62.842 & 47336 & 89.692 & 482522 & 75.081 & 494482 & 76.828 \\ Ethanol & **609735** & 63.01 & 11.3609 & 5.1601 & 11.08856 & 57.181 & 108529 & **61.852** & - & - & 116381 & 65.746 \\ Neuphatence & **32082** & 65.799 & **330412** & 51.018 & 673988 & **22.884** & 821416 & 31.497 & 365549 & 65.117 & 475078 & 110.906 \\ Salicylic acid & 495088 & 70.401 & 525441 & **50.738** & 130028 & 62.014 & 1340236 & 61.483 & - & - & **139295** & 71.778 \\ \hline \hline \end{tabular}
\end{table}
Table 6: JSD and WF over simulation trajectory of benzene molecule using models trained on aspirin, ethanol, napthalene, and salicylic acid.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{NequIP} & \multicolumn{2}{c}{AllLegro} & \multicolumn{2}{c}{BOTNet} & \multicolumn{2}{c}{MACE} & \multicolumn{2}{c}{Equiformer} & \multicolumn{2}{c}{TorchMDNet} \\  & **T** & **I** & **T** & **I** & **T** & **I** & **T** & **I** & **T** & **I** & **T** & **I** \\ \hline
**Acetylacetone** & 0.66 & 3.18 & 0.17 & 1.94 & 0.11 & 1.90 & **0.04** & 2.66 & 0.52 & 9.98 & **0.11** & **1.79** \\
**3BPA** & 1.07 & 7.07 & 1.80 & 4.92 & 0.12 & 4.46 & **0.06** & **4.18** & 0.68 & 19.25 & 0.13 & 4.83 \\
**Asppirin** & 5.23 & 2.93 & 1.61 & 1.68 & 0.21 & 1.76 & 0.14 & 2.45 & 0.85 & 13.04 & 0.15 & **1.41** \\
**Ethanol** & 5.49 & 2.05 & 1.62 & 0.68 & 5.03 & 1.07 & 1.15 & 1.28 & 0.81 & 5.70 & **0.14** & 0.80 \\
**Naphthalene** & 5.26 & 3.75 & 2.11 & 1.07 & 13.47 & 12.27 & 4.72 & 2.28 & 2.08 & 14.67 & **0.14** & 1.37 \\
**Salicylic Acid** & 5.24 & 3.30 & 1.61 & 0.87 & 11.68 & 1.26 & 3.858 & 2.29 & 0.82 & 9.79 & **0.14** & 1.17 \\
**LiPS** & 89.91 & 35.83 & 20.89 & 13.91 & 4.82 & 10.29 & **3.61** & **6.52** & 18.51 & 46.34 & 3.18 & 6.95 \\
**LiPS20** & 2.78 & 25.51 & 0.76 & 11.42 & 0.36 & 15.187 & 0.18 & 6.75 & 1.86 & 56.59 & 0.21 & **5.12** \\
**GeTe** & 7.22 & 105.62 & **4.49** & 220.43 & 2.07 & 78.2 & 0.58 & 26.75 & 9.33 & 143.91 & **1.55** & 21.67 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Training time (T) per epoch and inference time (I) in \(minutes/epoch\) and \(minutes\), respectively, for the trained models on all the datasets. Inference time is the mean over 5 forward simulations of 1000 steps on the CPU.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{NequIP} & \multicolumn{2}{c}{AllLegro} & \multicolumn{2}{c}{BOTNet} & \multicolumn{2}{c}{MACE} & \multicolumn{2}{c}{Equiformer} & \multicolumn{2}{c}{TorchMDNet} \\  & E & F & E & F & E & F & E & F & E & F & E & F \\ \hline Aspirin & 22650 & 0.762 & 22676 & 0.765 & 21880 & 0.760 & 21881 & 0.766 & 47027.742 & 0.769 & 46864 & 0.765 \\ Ethanol & (11.62) & (0.060) & (0.311) & (0.070) & (6.874) & (0.061) & (21.11) & (0.061) & (3.88) & (0.065) & (184.678) & (0.058) \\ Ethanol & 6154.4 & 0.740 & 62242 & **0.011** & **55005** & 0.935 & 5863.2 & 0.921 & - & - & 20262 & 0.742 \\
0.402 & (0.042) & (0.056) & (12.501) & (0.040) & (13.725) & (0.016) & (0.338) & (0.022) & - & - & (19.401) & (0.052) \\
0.402 & (0.056) & (12.501) & (0.040) & (13.725) & (0.016) & (0.338) & (0.022) & - & - & (19.401) & (0.052) \\
0.481 & (16.41) & (0.067) & (0.070) & (3.32) & (0.008) & (0.324) & (0.012) & (6.069) & (0.061) & - & (0.057) \\
0.481 & (1.062) & (0.061) & (1.082) & **10.53** & **1.205** & 0.925 & 0.965 & - & - & 35947 & 0.769 \\
0.308 & (0.067) & (0.314) & (0.076) & (10.309) & (0.005) & (0.310) & (0.007) & - & (2.907) & (0.087) \\ \hline \hline \end{tabular}
\end{table}
Table 5: EV (E) and FV (F) on the forward simulation of benzene molecule by the models trained on aspirin, ethanol, napthathalene, and salicylic acid.
1. **Dataset matters:** There was no single model that was performing best on all the datasets and all the metrics. Thus, the selection of the model depends highly on the nature of the atomic system, whether it is a small molecule or a bulk system, for instance.
2. **Structure is important:** Low force or energy error during model development does not guarantee faithful reproduction of the atomic structure. Conversely, models with higher energy or force error may provide reasonable structures. Accordingly, downstream evaluation of atomic structures using structural metrics is important in choosing the appropriate model.
3. **Stability during dynamics:** Models exhibiting low energy or force errors during the model development on static configurations do not guarantee low errors during forward simulation. Thus, the energy and force violations during molecular dynamics should be evaluated separately to understand the stability of the simulation.
4. **Out-of-distribution is still challenging:** Discovery of novel materials relies on identifying hitherto unknown configurations with low energy. We observe that the models still do not perform reliably on out-of-distribution datasets, leaving an open challenge in materials modeling.
5. **Fast to train and fast on inference:** We observe that some models are fast on training, while others are fast on inference. For instance, TorchMDNet is slow to train but fast on inference. While MACE is fast both on training and inference, it does not give the best results in terms of structure or dynamics. Thus, in cases where larger simulations are required, the appropriate model that balances the training/inference time and accuracy may be chosen.

**Limitations and future work:** Our research clearly points to developing a foundation model trained on large datasets. Further, improved training strategies that (i) ensure the learning of gradients of energies and forces, (ii) take into account the dynamics during simulations, and (iii) reproduce the structure faithfully need to be developed. This suggests moving away from the traditional training approach only on energy and forces and rather focusing on the system's dynamics. Further strategies combining experimentally observed structures and simulated dynamics can be devised through experiment-simulation fusion to develop reliable force fields that are faithful to both experiments and simulations.Another interesting aspect is the empirical evaluation of which particular architectural feature of a model helps in giving a superior performance for a given dataset or system (defined by the type of bonding, number of atoms, crystalline vs disordered, etc.). Such a detailed analysis can be a guide to designing improved architecture while also providing thumb rules toward the use of an appropriate architecture for a given system.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{NEGUP} & \multicolumn{2}{c}{AllEGOR} & \multicolumn{2}{c}{BOTNET} & \multicolumn{2}{c}{MACE} & \multicolumn{2}{c}{Geupformer} & \multicolumn{2}{c}{TongMDNet} \\  & E & F & E & F & E & F & E & F & E & F & E & F \\ \hline Acetylacetone & 300K & 0.959 & 0.7062 & 0.817 & 0.710 & 0.924 & 0.7131 & **0.893** & 0.706 & **0.810** & 0.7113 & 0.836 & 0.7128 \\  & 600K & 1.888 & 0.714 & 1.912 & 0.7137 & 1.898 & 0.7140 & 2.715 & 0.412 & 0.710 & 0.7137 & 1.996 & 0.7129 \\
3BPA & 300K & 0.539 & 0.7106 & 3.708 & 0.7102 & **0.807** & 0.7109 & 0.759 & **0.709** & 0.803 & 0.814 & 0.7097 \\  & 600K & 1.880 & 0.708 & 1.603 & 0.7082 & 1.607 & 0.7102 & 1.241 & **0.708** & 1.319 & 0.7104 & 0.7120 & 0.7121 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Geometric mean of energy (\(\times 10^{-5}\)) and force violation at 300 K and 600 K using model trained at 300 K using model trained at 300 K for acetylacetone and 3BPA dataset.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{NEGUP} & \multicolumn{2}{c}{AllEGOR} & \multicolumn{2}{c}{BOTNET} & \multicolumn{2}{c}{MACE} & \multicolumn{2}{c}{Geupformer} & \multicolumn{2}{c}{TongMDNet} \\  & E & F & E & F & E & F & E & F & E & F & E & F \\ \hline Acetylacetone & 300K & 0.959 & 0.7062 & 0.817 & 0.710 & 0.924 & 0.7131 & **0.893** & 0.706 & **0.810** & 0.7113 & 0.836 & 0.7128 \\  & 600K & 1.888 & 0.714 & 1.912 & 0.7137 & 1.898 & 0.7140 & 2.715 & 0.412 & 0.7127 & 1.70 & 0.7137 & 1.996 & 0.7129 \\  & 300K & 0.539 & 0.7106 & 3.708 & 0.7102 & **0.807** & 0.7109 & 0.759 & **0.709** & 0.803 & 0.814 & 0.7097 \\  & 600K & 1.880 & 0.7085 & 1.603 & 0.7082 & 1.607 & 0.7102 & 1.241 & **0.708** & 1.319 & 0.7104 & 0.7100 & 0.7121 \\ \hline \hline \end{tabular}
\end{table}
Table 8: JSD and WF at 300 K and 600 K using the model trained at 300 K for acetylacetone and 3BPA.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & \multicolumn{2}{c}{NEGUP} & \multicolumn{2}{c}{AllEGOR} & \multicolumn{2}{c}{BOTNET} & \multicolumn{2}{c}{MACE} & \multicolumn{2}{c}{Geupformer} & \multicolumn{2}{c}{TongMDNet} \\  & E & F & E & F & E & F & E & F & E & F & E \\ \hline Acetylacetone & 300K & 0.959 & 0.7062 & 0.817 & 0.710 & 0.924 & 0.7131 & **0.893** & 0.706 & **0.810** & 0.7113 & 0.836 & 0.7128 \\  & 600K & 1.888 & 0.714 & 1.912 & 0.7137 & 1.898 & 0.7140 & 2.715 & 0.4127 & 0.710 & 0.7137 & 1.996 & 0.7129 \\  & 300K & 0.539 & 0.7106 & 3.708 & 0.7102 & **0.807** & 0.7109 & 0.759 & **0.709** & 0.803 & 0.814 & 0.7097 \\  & 600K & 1.880 & 0.7085 & 1.603 & 0.7082 & 1.607 & 0.7102 & 1.241 & **0.708** & 1.319 & 0.7104 & 0.7100 & 0.7121 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Geometric mean of energy (\(\times 10^{-5}\)) and force violation at 300 K and 600 K using model trained at 300 K and 600 K using model trained at 300 K for acetylacetone and 3BPA.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{2}{c}{NEGUP} & \multicolumn{2}{c}{AllEGOR} & \multicolumn{2}{c}{BOTNET} & \multicolumn{2}{c}{MACE} & \multicolumn{2}{c}{TongMDNet} \\ \hline Train structures & E & 0.4500 & 0.2736 & 0.2797 & 0.4175 & 0.5106 & 0.747 & 0.706 \\  & F & **0.7021** & 0.725 & **0.720** & **0.503** & **0.704** & **0.703** \\ Crystal structures & E & 10.854 & 10.725 & **0.720** & **0.503** & **0.704** & **0.705** \\  & F & 10.820 & 0.725 & **0.722** & **0.836** \\ Test structures & E & 15.549,338 & 14.0035 & 12.731 & **0.990** & **0.813** \\  & F & 0.738 & 0.729 & 0.729 & **0.723** & **0.992** & 0.902** \\ \hline \hline \end{tabular}
\end{table}
Table 9: LiPS20 test on train structures, unseen crystalline structures, and test structures: (a) Energy and Force violation and (b) JSD and WF.