<EMAIL_ADDRESS>National Institute of Standards and Technology,
Gaithersburg, 20899, MD, USA<EMAIL_ADDRESS>National Institute of
Standards and Technology, Gaithersburg, 20899, MD, USA.

# Graph Neural Network Predictions of Metal Organic Framework CO2 Adsorption
Properties

Kamal Choudhary  Materials Science and Engineering Division, National
Institute of Standards and Technology, Gaithersburg, 20899, MD, USA.  Taner
Yildirim  NIST Center for Neutron Research, National Institute of Standards
and Technology, Gaithersburg, 20899, MD, USA.  Daniel W. Siderius  Chemical
Sciences Division, National Institute of Standards and Technology,
Gaithersburg, 20899, MD, USA.  A. Gilad Kusne  Materials Measurement Science
Division, National Institute of Standards and Technology, Gaithersburg, 20899,
MD, USA.  Austin McDannald  Materials Measurement Science Division, National
Institute of Standards and Technology, Gaithersburg, 20899, MD, USA.  Diana L.
Ortiz-Montalvo  Materials Measurement Science Division, National Institute of
Standards and Technology, Gaithersburg, 20899, MD, USA.

###### Abstract

The increasing CO2 level is a critical concern and suitable materials are
needed to capture such gases from the environment. While experimental and
conventional computational methods are useful in finding such materials, they
are usually slow and there is a need to expedite such processes. We use
Atomistic Line Graph Neural Network (ALIGNN) method to predict CO2 adsorption
in metal organic frameworks (MOF), which are known for their high functional
tunability. We train ALIGNN models for hypothetical MOF (hMOF) database with
137953 MOFs with grand canonical Monte Carlo (GCMC) based CO2 adsorption
isotherms. We develop high accuracy and fast models for pre-screening
applications. We apply the trained model on CoREMOF database and
computationally rank them for experimental synthesis. In addition to the CO2
adsorption isotherm, we also train models for electronic bandgaps, surface
area, void fraction, lowest cavity diameter, and pore limiting diameter, and
illustrate the strength and limitation of such graph neural network models.
For a few candidate MOFs we carry out GCMC calculations to evaluate the deep-
learning (DL) predictions.

###### keywords:

_Machine learning_ , Metal organic framework, CO2 absorption.

##  1 Introduction

The CO2 concentration in air is increasing at an alarming rate which could
escalate global temperature by 1.5 °C by 2100 [1]. CO2 capture from pre-
combustion plants is an effective way to reduce CO2 level. Typical materials
used in this process are zeolites, calcium chabazite, hydrotalcite, and
activated carbon [2], but they usually suffer from low CO2 working capacities,
low tunability, and limited material diversity [3].

Metal organic frameworks (MOFs) are a class of porous materials with inorganic
clusters and organic building blocks, which can be repeated to build an
extended 3D structure. MOFs provide high thermal stabilities (as high as 500
°C), a wide range of porosities (0.3–0.9), large surface areas (>8000 m2g-1),
low densities (0.2 g cm-3), a wide range of pore sizes (3–100 Å), active sites
that can be used for site-specific adsorption and catalysis, and most
importantly a wide range of chemical tunability unlike conventional porous
materials [4]. Hence, MOFs are considered as one of the top ten materials for
emerging technologies by the International Union of Pure and Applied Chemistry
(IUPAC) [5].

In the past, there have been several efforts to use conventional computational
methods such as grand canonical Monte Carlo (GCMC) to develop CO2 isotherm
databases for thousands of hypothetical as well as experimentally realizable
MOFs [6, 7, 8, 9, 10, 11, 12, 13, 14]. However, such computational methods are
still time consuming, especially to screen thousands to millions of MOF
candidates. One of the most obvious ways to tackle this issue is using machine
learning (ML) methods, where ML methods act as a pre-screening tool for
conventional computational methods, which then can be used as a screening tool
for experimental designs [15, 16, 17, 18, 19, 20, 21, 20].

The key precursors for applying ML methods for CO2 absorption in MOF are : 1)
large datasets of experimental or computational CO2 isotherms and other MOF
properties, 2) an ML framework that can accurately represent arbitrary MOF
crystals and chemical structures. Fortunately, due to materials genome
initiative (MGI) [22] and similar initiatives there has been a rapid
development for generating large databases of MOFs such as: Computation-Ready,
Experimental (CoREMOF), hypothetical (hMOF), and quantum (QMOF) [7, 8, 9, 10,
11, 12, 13, 14]. These databases are based on computational methods such as
density functional theory (DFT) or GCMC and the results are compared to
experiments wherever applicable [23]. They contain exact coordinates for MOF
atomic structures and their properties.

Machine learning has been already used for some of the atomistic property
predictions of MOFs [4, 24, 25, 26, 19, 21, 27], but the application of multi-
output ML methods especially for pressure dependent CO2 adsorption has not
been reported earlier to the best of our knowledge. Also, previous ML models
are based on cumbersome handcrafted features which can be superseded with deep
learning based methods [28]. Graph neural network (GNN)/deep learning (DL)
methods such as Crystal Graph Convolutional Neural Networks (CGCNN) [29] and
Atomistic Line Graph Neural Network (ALIGNN) [30] can be used to represent
arbitrary chemistry and atomic structure with respect to interatomic bonds and
angles and can utilize the full power of deep neural network framework.

In this work, we apply the ALIGNN model for predicting properties in hMOF and
QMOF databases which can be used for accelerated MOF design. We train models
for technologically important quantities such as CO2 adsorption isotherms at
different pressures, electronic bandgap, lowest cavity diameter (LCD), pore
limiting diameter (PLD), void fraction, volumetric, and gravimetric surface
areas. After training the models, we apply them to more CoREMOF database to
screen for potential MOF candidates for CO2 capture. We compare the model
predictions with a few experimental measurement results to check accuracy.
Additionally, we identify some of the key strength and challenges of the DL
framework.

##  2 Methods

This study uses the QMOF dataset [14] and the hMOF database [8] to train the
DL model. The trained DL model is applied to the CoreMOF database [7] to
computationally rank the MOFs based on max CO2 adsorption.

The QMOF dataset consists of 20425 MOFs that Rosen et al. [14] generated using
a subset of the CoRE-MOF and Cambridge Structural Database (CSD) [31] and made
them suitable for density functional theory (DFT) calculations. The DFT
calculations [32] were carried out at PBE-D3(BJ) level with Vienna ab initio
Simulation Package (VASP) [33, 34].

The hMOF database consists of 137953 MOF structures which were generated using
102 building blocks from Wilmer et al. [9]. Details about the generation of
the hMOF database can be found in Wilmer et al. [10]. Briefly, the Grand
canonical Monte-Carlo (GCMC) calculations were performed using the RASPA code
[35]. GCMC requires reliable force-fields for simulation. Interaction energies
between non-bonded atoms were computed through the Lennard-Jones (LJ) and
Coulomb potentials. LJ parameters for the framework atoms were taken from the
Universal Force Field (UFF) [36]. Partial charges and LJ parameters for CO2
were taken from the TraPPE force field [37]. Each GCMC calculation used a 1000
cycle equilibration period followed by a 1000 cycle production run. A cycle
consists of n Monte Carlo steps; where n is equal to the number of molecules
(which fluctuates during a GCMC simulation). All simulations included random
insertion, deletion, rotation, and translation moves of molecules with equal
probabilities.

Unlike the previous database where the MOF structures have been generated
using computer algorithms, Computation-Ready, Experimental Metal–Organic
Framework (CoREMOF) database contains materials that are derived from
synthesized materials following the synthesis and data-refining protocols to
provide closer to reality structures. CoREMOF database contains more than
14000 MOF structures [7]. There is no direct overlap between hMOF and CoREMOF
database.

Graph neural networks are trained with atomistic line graph neural network
(ALIGNN) [30] which is available at https://github.com/usnistgov/alignn.
ALIGNN has been used to train more than 50 properties of solids and molecules
with high accuracy. A MOF structure is represented as a graph using atomic
elements as nodes, and atomic bonds as edges. Each node in the atomistic graph
is assigned 9 input node features based on its atomic species:
electronegativity, group number, covalent radius, valence electrons, first
ionization energy, electron affinity, block and atomic volume. The inter-
atomic bond distances are used as edge features with radial basis function
upto 8 Å cut-off. We use a periodic 12-nearest-neighbor graph construction.
This atomistic graph is then used for constructing corresponding line graph
using interatomic bond-distances as nodes and bond-angles as edge features.
ALIGNN uses Edge-gated graph convolution for updating nodes as well as edge
features. One ALIGNN layer composes an edge-gated graph convolution on the
bond graph with an edge-gated graph convolution on the line graph. The line
graph convolution produces bond messages that are propagated to the atomistic
graph, which further updates the bond features in combination with atom
features. The ALIGNN model is implemented in PyTorch [38] and deep graph
library (DGL) [39].

For regression targets we minimize the mean squared error (MSE) loss for 50
epochs using the AdamW optimizer with normalized weight decay of 10-5 and a
batch size of 64.The learning rate is scheduled according to the one-cycle
policy with a maximum learning rate of 0.001. We use 80 initial bond radial
basis function (RBF) features, and 40 initial bond angle RBF features. The
atom, bond, and bond angle feature embedding layers produce 64-dimensional
inputs to the graph convolution layers. The main body of the network consists
of 4 ALIGNN layers and 4 graph convolution (GCN) layers, each with hidden
dimension 256 nodes. The final atom representations are reduced by atom-wise
average pooling and mapped to regression or classification outputs by a single
linear layer.

We used National Institute of Standards and Technology (NIST)’s Nisaba cluster
to train all ALIGNN models [32]. Each model is trained on a single Tesla V100
SXM2 32 gigabyte Graphics processing unit (GPU), with 8 Intel Xeon E5-2698 v4
CPU cores for concurrently fetching and preprocessing batches of data during
training. We use 80 %:10 %: 10 % splits. The 10 % test data is never used
during training procedures. Except for the pressure dependent CO2 adsorption
data, which is a multi-output model, all other models are single output
regression.

##  3 Results and discussion

We apply ALIGNN model to predict MOF properties in QMOF and hMOF databases.
QMOF is developed using quantum density functional theory method while hMOF is
based on classical force field guided GCMC calculations. The QMOF and hMOF
materials are composed of some of the combination of the following 9 elements:
C, H, O, N, Zn, Cu, Zr, F, Cl, and Br. These datasets are kept in the jarvis-
tools package https://jarvis-tools.readthedocs.io/en/master/databases.html
[40] in a format to easily apply DL methods. In this section, we will first
visualize the data distribution and then discuss the ALIGNN application on
these datasets.

###  3.1 Application to QMOF database

Figure 1: Data visualization of QMOF electronic bandgaps and GNN predictions
on the test set. a) Electronic bandgap data distribution in QMOF bandgap
distribution database, b) ALIGNN based deep-learning (DL) predictions on 10 %
test set.

Before applying the DL model on QMOF we visualize the properties in QMOF
database with about 20425 MOFs. One of the unique features of QMOF is
electronic bandgap data which cannot be obtained from classical force field
calculations such as in hMOF. Although the focus of the work is on CO2
adsorption, we note that MOFs can be viewed as photocatalysts and tailoring of
bandgaps can provide dual technological benefits of CO2 reduction as well as
solar power generation [41, 42]. We visualize the bandgap data distribution in
Fig. 1a. We observe a high peak around 1 eV and 3 eV with data range up to 6
eV.

The developers of QMOF applied several machine-learning models for predicting
the bandgaps. They compared well-known machine learning model such as Sine
Coulomb matrix, Orbital field matrix [43, 44], Smooth overlap of atomic
orbitals (SOAP) [45], and crystal graph convolution neural network (CGCNN)
[29] to develop a model for predicting the bandgap of MOFs. They found that
the best model was achieved with CGCNN with a mean absolute error (MAE) of
0.274 eV for bandgap prediction. Following a similar training strategy, we
trained ALIGNN model on this dataset and find the MAE to be 0.20 eV, which is
nearly 27 % improvement in accuracy. A parity plot for the actual DFT data and
ALIGNN based predictions on 10 % test set is shown in Fig. 1b. For a perfect
agreement, all the data-points would fall upon $y=x$ line, which the data
points in Fig. 1b closely resemble. ALIGNN has already been shown to
outperform CGCNN and other models for other solid-state and molecular dataset
and the previous results support the findings [30].

###  3.2 Application to hMOF database

Figure 2: Workflow used in training and applying graph neural network models
for several properties in hMOF database.

Next we apply the ALIGNN model for training properties in the hMOF databases.
The list of properties include: gravimetric surface area ( m2g-1) , volumetric
surface area (m2cm-3), largest cavity diameter (LCD), pore limiting diameter
(PLD), void fraction, and pressure dependent CO2 absorption. All the models
except the CO2 isotherm are single output models. The adsorption isotherm
predictions has 5 outputs representing CO2 absorption at 5 different pressures
0.01, 0.05, 0.1, 0.5, and 2.5 bar. In addition to the multi-output model, we
also train models for minimum (0.01 bar) and maximum (2.5 bar) pressures
(under consideration) to check whether having more data as targets (five
versus one) helps improve the model predictions.

Figure 3: Data distribution visualization of several properties in the hMOF
database. a) density, b) gravimetric surface area, c) volumetric surface area,
d) void fraction, e) LCD, f) PLD, e) minimum value of CO2 dsorption among 5
pressure values (i.e. adsorption at 0.0 1bar), and f) maximum value of CO2
adsorption (i.e. adsorption at 2.5 bar) among 5 pressure values. Figure 4: GNN
predictions and GCMC actual value comparisons of several properties on 10 %
held test set (13796 MOFs) in the hMOF dataset. a) gravimetric surface area,
b) volumetric surface area, c) void fraction, d) largest cavity diameter, e)
pore limiting diameter, f) adsorption at five different pressures, g)
adsorption at 0.01 bar, and h) adsorption at 2.5 bar,

In Fig. 2 we show the training process for hMOF which is described below.
Before applying any DL, we first visualize the distribution of the above
mentioned properties for 137953 MOFs in Fig. 3. As observed in Fig. 3a, MOFs
in hMOF have low bulk densities with a peak around 0.7 g cm-3, which is much
lower than usual solids such as silicon (2.3 g cm-3) and aluminum (2.7 g
cm-3). The gravimetric and volumetric surface areas in Fig. 3b and Fig. 3c
varies up to 6000 m2 g-1 and 3000 m2 cm-3, respectively. The gravimetric
surface area is almost uniformly distributed across the range while the
volumetric surface area peaks around 2000 m2 cm-3. We note that high
volumetric surface area doesn’t necessarily imply high gravimetric surface
area and vice versa. The void fraction in Fig. 3d shows a peak around 0.8
suggesting high porosity of MOFs. The largest cavity diameters (LCD) and pore
limiting diameters (PLD) are other critical parameters for adsorption. Both of
them vary up to 20 Å with a peak around 7 Å (as shown in Fig. 3e and Fig. 3f).
As mentioned above, hMOF contains CO2 adsorption at 5 different and
monotonically increasing pressure values, so we show the CO2 adsorption at
0.01 bar and 2.5 bar pressures in Fig. 3g and Fig. 3h, respectively.

We note that some of these quantities (e.g. PLD, LCD, and void fraction) can
be easily calculated for the whole dataset of potential MOFs using
conventional computational methods such as Zeo++ [46]. Using DL based methods
to predict these properties will elucidate the strengths and weaknesses of the
DL models. However, predicting adsorption isotherms can be prohibitively
expensive for the whole dataset by traditional means (e.g. GCMC). Therefore
the DL methods can be useful as a surrogate model, with the usefulness
characterized by the appropriate performance metrics.

The parity plots for predictions on the test dataset by the ALIGNN models for
different properties is shown in Fig 4. For an ideal model the DL and actual
target values should line on $y=x$ straight line. Corresponding performance
metric for the trained models are charted in Table. 1. In addition to the
performance metric such as the mean absolute error (MAE), we also provide the
mean absolute deviation (MAD) which serves as the baseline for the model
predictions. Usually, a good machine learning model has MAE five times lower
than its MAD. The MAD:MAE ratio also serves as a good comparison metric across
different properties. From Table 1, we observe that most of our models have
MAD:MAE greater than 5, with highest value for gravimetric surface area and
the minimum value for the pore limiting diameter. Same behavior can be
observed from the parity plots.

Comparing Fig. 4a and Fig. 4b, we observe that gravimetric surface area is
easier to predict than the volumetric surface area. Currently we don’t have a
good explanation for this behavior. The void fraction is easier to predict as
well as shown in Fig. 4c. The LCD and PLD parity plots in Fig, 4d and Fig. 4e
suggest they are much difficult to predict. This shows one of the limitations
of the ALIGNN model where it cannot capture long range order based properties.
In Fig. 4f, we show the DL predicted data against the actual GCMC data for all
the pressures in one plot. Similar plots for the minimum and maximum pressures
are shown in Fig. 4 g and 4h. Interestingly, the full-isotherm data based
model has lower overall MAE as well as lower MAD:MAE in comparison with the
maximum pressure adsorption model. This result suggests that training multi-
output model could be advantageous for adsorption data especially at higher
pressure values. The colorbar in the plots represent void fraction with the
exception of Fig. 4c (surface area is represented). We observe that most of
the deviations from $y=x$ line in the test set occur on high void fraction
MOFs (such as upper yellow part of Fig. 4b, 4d, 4e and 4h). This can be due to
the k-nearest neighbor and relatively shorter cut-off (8 Å) strategies for
graph construction in ALIGNN models. Larger k-nearest neighbors and cut-off
impose higher computational expense, a challenge for ALIGNN.

In addition to the parity plots, we also analyze the best and worst isotherm
predictions using the multi-output isotherm predictor model. We sort the test
data based predictions with respect to MAE and show top 5 results in Fig. 5a
to Fig. 5e and worst values from Fig. 5f to Fig. 5j. Clearly, the best
predictions are on par with the target data. For the worst predictions, the
multi-output predictions are higher in some cases and lower in others compared
to actual GCMC data implying non-uniform trends.

Figure 5: Comparison of actual GCMC and ALIGNN predictions of CO2 adsorption at different pressures values. The model was tested on the 10 % held set of 13796 materials, out of which 5 best and 5 worst predictions are shown. a-e) 5 best predictions, f-j) 5 worst predictions. Table 1: ALIGNN performance on hMOF properties interms of mean absolute error (MAE) amd mean absolute deviation (MAD). Property | Unit | MAD | MAE | MAD:MAE
---|---|---|---|---
Grav. surface area | m2 g-1 | 1430.82 | 96.06 | 14.90
Vol. surface area | m2 cm-3 | 561.44 | 109.84 | 5.11
Void fraction | No unit | 0.16 | 0.018 | 8.89
LCD | Å | 3.44 | 0.76 | 4.53
PLD | Å | 3.55 | 0.92 | 3.86
All adsp | mol kg-1 | 1.70 | 0.19 | 8.95
Min adsp | mol kg-1 | 0.12 | 0.04 | 3.0
Max adsp | mol kg-1 | 2.16 | 0.48 | 4.50

To investigate the reason why the DL model works well for a certain class of
materials, we carried out several correlation analyses between the errors of
multi-output isotherm predictor model and materials properties. We calculate
the mean of absolute errors between actual and DL predictor values. From Table
2, we find the minimum and maximum CO2 intake and density of the MOFs relate
the most to the errors in the DL predictions. Note that there are relatively
few examples of low-density materials, hence DL is less likely to generalize
well for these set of materials.

Table 2: Pearson and Spearman correlation between mean of mean absolute errors of the CO2 absorptions on test set and physical properties. Property | Unit | Pearson | Spearman
---|---|---|---
Density | g m-3 | 0.04 | 0.22
Grav. surface area | m2 g-1 | -0.-07 | -0.10
Vol. surface area | m2 cm-3 | -0.-04 | -0.10
Void fraction | No unit | -0.02 | -0.2
N atoms | No unit | 0.1 | 0.2
PLD | Å | -0.07 | -0.23
LCD | Å | -0.05 | -0.21
Min adsp | mol kg-1 | 0.33 | 0.56
Max adsp | mol kg-1 | 0.27 | 0.40

Next, we evaluate the accuracy of the DL model based predictions by comparing
them with a few experimental and GCMC calculation results. Over the years we
have experimentally characterized a large number of MOFs using same
measurement protocol and very accurate high-pressure sievert apparatus[47]. We
compare the experimental CO2 adsorption at 2.5 bar for some of these
previously well characterized MOFs [48, 47, 49, 50] with DL predictions in
Fig. 6. We observe excellent comparison between DL and experimental results.
We find that the mean absolute difference for DL and experimental values for
CO2 adsorption and surface area are 1.54 molkg-1 and 399.75 m2g-1 which are
approximately 3 and 4 times higher than MAE of respective models in Table. 1.
The above results suggest that the DL based model predictions can be useful
for guiding experimental measurements.

Moreover, we apply the DL model for predicting maximum CO2 adsorption trained
on hMOF database to screen high CO2 adsobing MOFs in the CoREMOF database Note
that there is almost no overlap between the MOFs in these two databases. The
hMOF database is based on hypothetical structures while CoreMOF uses
experimental structures which have been processed following several protocols.
DL prediction for a MOF takes less than a few seconds. We rank the MOFs and
provide the DL based predictions for MOFs in the supplementary information.
Some of the high rank materials predicted by our DL model are: ATEYUV (10.7
mol.kg-1), WEHJUQ (10.2 mol.kg-1), ATEYOP (9.94 mol.kg-1), HOJLID (9.86
mol.kg-1), and MINCUJ (9.81 mol.kg-1) , shown in Fig. 7 and Fig. 8. The
crystal structure visualization for these MOFs as well as other screened
materials are available at https://mof.tech.northwestern.edu

Figure 6: Comparison of experimental measurements and ALIGNN (DL) predictions
for a few previously characterized MOFs from literature. a) CO2 adsorption ,
b) gravimetric surface area. Figure 7: ALIGNN predicted some examples of the
high-CO2 adsorbing MOFs from CoREMOF DB. Complete list is provided in the
supplementary information. a) ATEYUV, b) WEHJUQ, c) ATEYOP , d) HOJLID , and
e) MINCUJ. The crystal structure visualization for these MOFs as well as other
screened materials are available at https://mof.tech.northwestern.edu. Figure
8: Comparison of GCMC and ALIGNN (DL) predictions for top 10 screened
candidates from CoREMOF DB. a) CO2 adsorption , b) gravimetric surface area.

In order to check the DL predictions, we carried out accurate GCMC simulations
and compare the results with DL predictions. In Fig. 8a, the comparison of
GCMC vs DL predictions for CO2 on top 10 screened candidates shows the DL
method can indeed predict high CO2 adsorption MOFs. The GCMC data are obtained
with the similar computation set-up as of hMOF database but with larger cutoff
(up to 18 Å) and atomic charges of MOFs. The partial atomic charges of the
framework are obtained by message passing neural networks model developed in
Ref.[51] and included in the CO2 adsorption simulations. Although the GCMC
values do not perfectly align with the DL data, the trends are noteworthy. We
note that nine out of ten predicted structures give a CO2 adsorption above 10
$mol/kg$ at 2.5 bar, which is quite high compared to all other experimentally
well known MOFs. Fig. 8a clearly suggests that DL can predict high performing
MOFs which are also validated with the GCMC calculations. We find that the
mean absolute difference between the GCMC and DL values for CO2 adsorption at
2.5 bar for these candidates is 4.2 molkg-1 which is almost 9 times higher
than that of MAE reported in Table 1. Next we compare the DL and GCMC bases
surface area predictions for these top 10 screened MOFs. We notice that the
trends in the predictions (as shown in Fig. 8b) are much better than that of
CO2 adsorption. We find the mean absolute difference between these values as
697.2 m2g-1 which is 7 times higher than MAE of the DL model obtained for
gravimetric surface area in Table. 1. Experimental synthesis and
characterizations of these is beyond the scope of the present work and will be
carried out in future. The complete list of MOFs during the above screening
procedure is provided in the supplementary information. The trained ALIGNN
(DL) models for the properties trained above are also distributed in the
ALIGNN GitHub repo as pre-trained models for making quick predictions on
arbitrary MOFs.

##  4 Conclusion

In this article, we have applied atomistic line graph neural network (ALIGNN)
to predict several properties such as volumetric and gravimetric surface area,
void fraction, maximum and minimum CO2 adsorption, adsorption isotherms at
several pressures and electronic bandgaps in hMOF and QMOF datasets. High
accuracy models are obtained for surface area and CO2 adsorption that can be
used for pre-screening applications. We have applied the trained model to pre-
screening high-performance MOFs in the CoREMOF database and predicted several
promising MOF for CO2 capture. Moreover, we carry out GCMC calculations for a
few screened candidates and compare them with DL predictions showing promising
trends. This study demonstrates some of the key strength for using GNN over
usual conventional methods. We also discuss some of the weaknesses of the
model that motivate further development for the GNN model. Our work also
establishes that GNN models can not only be successfully applied for molecules
and solids but MOF structures as well.

## acknowledgements

K.C. thanks National Institute of Standards and Technology for computational
and funding support.

## conflict of interest

The authors declare no conflict of interest.

## Supporting Information

Supporting Information is available from the Wiley Online Library or from the
author. A list of DL based screened MOF candidates in provided in the
supporting information.

## References

  * Kirtman et al. [2013] Kirtman B, Power SB, Adedoyin AJ, Boer GJ, Bojariu R, Camilloni I, et al.  Near-term climate change: projections and predictability.  Cambridge University Press; 2013.
  * Lee et al. [2011] Lee A, Xiao G, Xiao P, Joshi K, Singh R, Webley PA.  High temperature adsorption materials and their performance for pre-combustion capture of carbon dioxide.  Energy Procedia 2011;4:1199–1206.
  * Stocker et al. [2017] Stocker K, Ellersdorfer M, Lehner M, Raith JG.  Characterization and utilization of natural zeolites in technical applications.  BHM Berg-und Hüttenmännische Monatshefte 2017;162(4):142–147.
  * Altintas et al. [2021] Altintas C, Altundal OF, Keskin S, Yildirim R.  Machine Learning Meets with Metal Organic Frameworks for Gas Storage and Separation.  Journal of Chemical Information and Modeling 2021;61(5):2131–2146.
  * Gomollón-Bel [2019] Gomollón-Bel F.  Ten Chemical Innovations That Will Change Our World: IUPAC identifies emerging technologies in Chemistry with potential to make our planet more sustainable.  Chemistry International 2019;41(2):12–17.
  * Altintas et al. [2018] Altintas C, Avci G, Daglar H, Nemati Vesali Azar A, Velioglu S, Erucar I, et al.  Database for CO2 separation performances of MOFs based on computational materials screening.  ACS applied materials & interfaces 2018;10(20):17257–17268.
  * Chung et al. [2019] Chung YG, Haldoupis E, Bucior BJ, Haranczyk M, Lee S, Zhang H, et al.  Advances, updates, and analytics for the computation-ready, experimental metal–organic framework database: CoRE MOF 2019.  Journal of Chemical & Engineering Data 2019;64(12):5985–5998.
  * Bobbitt et al. [2016] Bobbitt NS, Chen J, Snurr RQ.  High-throughput screening of metal–organic frameworks for hydrogen storage at cryogenic temperature.  The Journal of Physical Chemistry C 2016;120(48):27328–27341.
  * Wilmer et al. [2012a] Wilmer CE, Leaf M, Lee CY, Farha OK, Hauser BG, Hupp JT, et al.  Large-scale screening of hypothetical metal–organic frameworks.  Nature chemistry 2012;4(2):83–89.
  * Wilmer et al. [2012b] Wilmer CE, Farha OK, Bae YS, Hupp JT, Snurr RQ.  Structure–property relationships of porous materials for carbon dioxide separation and capture.  Energy & Environmental Science 2012;5(12):9849–9856.
  * Sikora et al. [2012] Sikora BJ, Wilmer CE, Greenfield ML, Snurr RQ.  Thermodynamic analysis of Xe/Kr selectivity in over 137000 hypothetical metal–organic frameworks.  Chemical Science 2012;3(7):2217–2223.
  * Colón et al. [2017] Colón YJ, Gomez-Gualdron DA, Snurr RQ.  Topologically guided, automated construction of metal–organic frameworks and their evaluation for energy-related applications.  Crystal Growth & Design 2017;17(11):5801–5810.
  * Gómez-Gualdrón et al. [2016] Gómez-Gualdrón DA, Colón YJ, Zhang X, Wang TC, Chen YS, Hupp JT, et al.  Evaluating topologically diverse metal–organic frameworks for cryo-adsorbed hydrogen storage.  Energy & Environmental Science 2016;9(10):3279–3289.
  * Rosen et al. [2021] Rosen AS, Iyer SM, Ray D, Yao Z, Aspuru-Guzik A, Gagliardi L, et al.  Machine learning the quantum-chemical properties of metal–organic frameworks for accelerated materials discovery.  Matter 2021;4(5):1578–1597.
  * Vasudevan et al. [2019] Vasudevan RK, Choudhary K, Mehta A, Smith R, Kusne G, Tavazza F, et al.  Materials science in the artificial intelligence age: high-throughput library generation, machine learning, and a pathway from correlations to the underpinning physics.  MRS communications 2019;9(3):821–838.
  * Schleder et al. [2019] Schleder GR, Padilha AC, Acosta CM, Costa M, Fazzio A.  From DFT to machine learning: recent approaches to materials science–a review.  Journal of Physics: Materials 2019;2(3):032001.
  * Schmidt et al. [2019] Schmidt J, Marques MR, Botti S, Marques MA.  Recent advances and applications of machine learning in solid-state materials science.  npj Computational Materials 2019;5(1):1–36.
  * Ohno and Mukae [2016] Ohno H, Mukae Y.  Machine learning approach for prediction and search: application to methane storage in a metal–organic framework.  The Journal of Physical Chemistry C 2016;120(42):23963–23968.
  * Fernandez et al. [2014] Fernandez M, Boyd PG, Daff TD, Aghaji MZ, Woo TK.  Rapid and accurate machine learning recognition of high performing metal organic frameworks for CO2 capture.  The journal of physical chemistry letters 2014;5(17):3056–3060.
  * Jablonka et al. [2020] Jablonka KM, Ongari D, Moosavi SM, Smit B.  Big-data science in porous materials: materials genomics and machine learning.  Chemical reviews 2020;120(16):8066–8129.
  * Chong et al. [2020] Chong S, Lee S, Kim B, Kim J.  Applications of machine learning in metal-organic frameworks.  Coordination Chemistry Reviews 2020;423:213487.
  * de Pablo et al. [2019] de Pablo JJ, Jackson NE, Webb MA, Chen LQ, Moore JE, Morgan D, et al.  New frontiers for the materials genome initiative.  npj Computational Materials 2019;5(1):1–23.
  * Sun et al. [2014] Sun W, Lin LC, Peng X, Smit B.  Computational screening of porous metal-organic frameworks and zeolites for the removal of SO2 and NOx from flue gases.  AIChE Journal 2014;60(6):2314–2323.
  * Anderson et al. [2018] Anderson R, Rodgers J, Argueta E, Biong A, Gómez-Gualdrón DA.  Role of pore chemistry and topology in the CO2 capture capabilities of MOFs: from molecular simulation to machine learning.  Chemistry of Materials 2018;30(18):6325–6337.
  * Guda et al. [2020] Guda A, Guda S, Martini A, Bugaev A, Soldatov M, Soldatov A, et al.  Machine learning approaches to XANES spectra for quantitative 3D structural determination: The case of CO2 adsorption on CPO-27-Ni MOF.  Radiation Physics and Chemistry 2020;175:108430.
  * Burner et al. [2020] Burner J, Schwiedrzik L, Krykunov M, Luo J, Boyd PG, Woo TK.  High-Performing Deep Learning Regression Models for Predicting Low-Pressure CO2 Adsorption Properties of Metal–Organic Frameworks.  The Journal of Physical Chemistry C 2020;124(51):27996–28005.
  * Yao et al. [2021] Yao Z, Sánchez-Lengeling B, Bobbitt NS, Bucior BJ, Kumar SGH, Collins SP, et al.  Inverse design of nanoporous crystalline reticular materials with deep generative models.  Nature Machine Intelligence 2021;3(1):76–86.
  * Kearnes et al. [2016] Kearnes S, McCloskey K, Berndl M, Pande V, Riley P.  Molecular graph convolutions: moving beyond fingerprints.  Journal of computer-aided molecular design 2016;30(8):595–608.
  * Xie and Grossman [2018] Xie T, Grossman JC.  Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties.  Physical review letters 2018;120(14):145301.
  * Choudhary and DeCost [2021] Choudhary K, DeCost B.  Atomistic Line Graph Neural Network for improved materials property predictions.  npj Computational Materials 2021;7(1):1–8.
  * Moghadam et al. [2017] Moghadam PZ, Li A, Wiggin SB, Tao A, Maloney AG, Wood PA, et al.  Development of a Cambridge Structural Database subset: a collection of metal–organic frameworks for past, present, and future.  Chemistry of Materials 2017;29(7):2618–2625.
  * NIST disclaimer [2021] NIST disclaimer, NIST, editor, Certain commercial equipment, instruments, or materials are identified in this paper in order to specify the experimental procedure adequately. Such identification is not intended to imply recommendation or endorsement by NIST, nor is it intended to imply that the materials or equipment identified are necessarily the best available for the purpose.  NIST; 2021.
  * Kresse and Furthmüller [1996] Kresse G, Furthmüller J.  Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set.  Physical review B 1996;54(16):11169.
  * Kresse and Joubert [1999] Kresse G, Joubert D.  From ultrasoft pseudopotentials to the projector augmented-wave method.  Physical review b 1999;59(3):1758.
  * Dubbeldam et al. [2016] Dubbeldam D, Calero S, Ellis DE, Snurr RQ.  RASPA: molecular simulation software for adsorption and diffusion in flexible nanoporous materials.  Molecular Simulation 2016;42(2):81–101.
  * Rappé et al. [1992] Rappé AK, Casewit CJ, Colwell K, Goddard III WA, Skiff WM.  UFF, a full periodic table force field for molecular mechanics and molecular dynamics simulations.  Journal of the American chemical society 1992;114(25):10024–10035.
  * Potoff and Siepmann [2001] Potoff JJ, Siepmann JI.  Vapor–liquid equilibria of mixtures containing alkanes, carbon dioxide, and nitrogen.  AIChE journal 2001;47(7):1676–1682.
  * Paszke et al. [2019] Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems 2019;32:8026–8037.
  * Wang et al. [2019] Wang M, Yu L, Zheng D, Gan Q, Gai Y, Ye Z, et al.  Deep Graph Library: Towards Efficient and Scalable Deep Learning on Graphs.  arXiv e-prints 2019;.
  * Choudhary et al. [2020] Choudhary K, Garrity KF, Reid AC, DeCost B, Biacchi AJ, Walker ARH, et al.  The joint automated repository for various integrated simulations (JARVIS) for data-driven materials design.  npj Computational Materials 2020;6(1):1–13.
  * Lee et al. [2014] Lee DY, Shinde DV, Yoon SJ, Cho KN, Lee W, Shrestha NK, et al.  Cu-based metal–organic frameworks for photovoltaic application.  The Journal of Physical Chemistry C 2014;118(30):16328–16334.
  * Guo et al. [2021] Guo X, Liu L, Xiao Y, Qi Y, Duan C, Zhang F.  Band gap engineering of metal-organic frameworks for solar fuel productions.  Coordination Chemistry Reviews 2021;435:213785.
  * Rupp et al. [2012] Rupp M, Tkatchenko A, Müller KR, Von Lilienfeld OA.  Fast and accurate modeling of molecular atomization energies with machine learning.  Physical review letters 2012;108(5):058301.
  * Bartók et al. [2013] Bartók AP, Kondor R, Csányi G.  On representing chemical environments.  Physical Review B 2013;87(18):184115.
  * De et al. [2016] De S, Bartók AP, Csányi G, Ceriotti M.  Comparing molecules and solids across structural and alchemical space.  Physical Chemistry Chemical Physics 2016;18(20):13754–13769.
  * Willems et al. [2012] Willems TF, Rycroft CH, Kazi M, Meza JC, Haranczyk M.  Algorithms and tools for high-throughput geometry-based analysis of crystalline porous materials.  Microporous and Mesoporous Materials 2012;149(1):134–141.
  * Peng et al. [2013] Peng Y, Krungleviciute V, Eryazici I, Hupp JT, Farha OK, Yildirim T.  Methane Storage in Metal-Organic Frameworks: Current records, surprising findings,and challanges.  Journal of American Chemical Society 2013;135(32):11887–11894.
  * Simmons et al. [2011] Simmons JM, Wu H, Wei Z, Yildirim T.  Carbon capture in metal-organic frameworks; a comparative study.  Energy &Environmental Science 2011;4(6):2177–2185.
  * Wang et al. [2021] Wang B, Zhang X, Huang H, Zhang Z, Yildirim T, Zhou W, et al.  A microporous aluminum-based metal-organic framework for high methane, hydrogen, and carbon dioxide storage.  Nano Research 2021;14(2):507–511.
  * Bucior et al. [2019] Bucior BJ, Bobbitt NS, Islamoglu T, Goswami S, Gopalan A, Yildirim T, et al.  Energy-based descriptors to rapidly predict hydrogen storage in metal–organic frameworks.  Molecular Systems Design & Engineering 2019;4(1):162–174.
  * Raza et al. [2020] Raza A, Sturluson A, Simon CM, Fern X.  Message Passing Neural Networks for Partial Charge Assignment to Metal-Organic Frameworks.  Journal of Physical Chemistry C 2020;124(35):19070–19082.


# Hybrid-SD ($\text{H}_{\text{SD}}$) : A new hybrid evaluation metric for
automatic speech recognition tasks

###### Abstract

Many studies have examined the shortcomings of word error rate (WER) as an
evaluation metric for automatic speech recognition (ASR) systems, particularly
when used for spoken language understanding tasks such as intent recognition
and dialogue systems. In this paper, we propose Hybrid-SD
($\text{H}_{\text{SD}}$), a new hybrid evaluation metric for ASR systems that
takes into account both semantic correctness and error rate. To generate
sentence dissimilarity scores (SD), we built a fast and lightweight SNanoBERT
model using distillation techniques. Our experiments show that the SNanoBERT
model is 25.9x smaller and 38.8x faster than SRoBERTa while achieving
comparable results on well-known benchmarks. Hence, making it suitable for
deploying with ASR models on edge devices. We also show that
$\text{H}_{\text{SD}}$ correlates more strongly with downstream tasks such as
intent recognition and named-entity recognition (NER).

Index Terms—  WER, evaluation metrics, sentence dissimilarity, automatic
speech recognition.

##  1 INTRODUCTION

WER is the de-facto evaluation metric used by the speech community for
assessing the performance of ASR models. WER is calculated from a literal
word-level comparison of ground-truth with the hypothesis. There is an
increasing interest among the speech community to consider an alternate
evaluation metric[1, 2], since WER does not capture the semantic correctness
of the hypothesis generated. Consider the ground-truth as ‘I am going to die’,
and the two hypotheses from ASR be ‘I m going to die’ and ‘I am going to
live’. WER is same for both the sentences, but we can clearly see that the
former hypothesis makes more sense than the latter which completely negates
the meaning of the ground-truth. Even though, WER is flawed and is an
imperfect metric for evaluation, several works tend to use WER as a metric
because of the following reasons : (1) existing speech recognition models are
benchmarked with WER as the metric, (2) an improved alternative metric does
not exist to replace WER.

Over the years, many works have attempted to address the flaws of WER by
considering weighted WER or by adopting information retrieval approaches as
mentioned in [3, 4, 5]. But all of these metrics are based on word level
comparison, has not considered the semantic-level correctness of the
sentences. Meanwhile many prior works based on the pre-trained transformer [6]
based language models, such as BERT [7], RoBERTa [8] showed promising results
in various downstream tasks such as question answering, and sentiment
analysis. Recently, [9, 10] proposed an alternative metric named as semantic
distance for ASR tasks based on semantic correctness using Siamese-BERT [11]
models. Our extensive analysis of this metric on ASR hypotheses shows that
using a metric just on semantic correctness is also not an optimal evaluation
metric. Moreover, these models with billions of parameters are often large and
cause a large memory overhead and latency when deployed on edge and mobile
devices to complete the evaluation process.

In this work, we use our proposed evaluation metric to benchmark the model’s
performance. We first focus on building a lighter model to get the sentence
dissimilarity (SD) score. We aim to reduce the memory footprint of the model
to the minimum, so that it will be suitable for edge and mobile based real-
world applications. We use the knowledge distillation [12] approach and
created a lighter siamese-BERT model, named as SNanoBERT to get the SD score.
We observe that this measure is biased towards the words which contributes
more to the meaning of the sentence, and sometimes fails to give a proper
weightage to the remaining words. Hence, we propose a new weighted evaluation
metric Hybrid-SD ($\text{H}_{\text{SD}}$), which is a weighted combination of
SD score and non-keyword error rate. We use the SNanoBERT model to extract the
keywords from the ground-truth, which is crucial for our hybrid evaluation
criteria. We show that the proposed alternative evaluation metric leverages
the benefits of considering both semantic correctness and word based errors.

This paper is organized as follows. In Section II, we discuss the proposed
model to generate the SD score. In section III, we discuss the proposed hybrid
evaluation metric. In Section IV, we explain the experimental setting and the
results and finally, concluding remarks are presented in Section V.

##  2 Sentence dissimilarity score

In this section, we discuss about how to generate the SD score and extract
keywords from our proposed SNanoBERT model.

###  2.1 SNanoBERT model

Our goal is to create a fast and light model for generating SD scores,
allowing to deploy the model on resource-constrained devices and use it during
real-time evaluation. Siamese models outperform other models in semantic
textual similarity (STS) tasks, in which two models connected by a loss
function are fed different inputs. To compute the similarity between fixed
sized embedding vectors generated by both models, the cosine similarity
measure is used. Although the SRoBERTa model achieves cutting-edge performance
on feature extraction tasks, its large size (500MB) renders it unsuitable for
deployment on edge devices. To overcome this limitation, we use SRoBERTa as a
teacher model and a lighter model as a student model to perform aggressive
task-specific knowledge distillation.

Table 1: Comparison of SD of GT and hypothesis by inducing an error on each word. WER is 0.33 for all the samples. Selected word | GT : You are late | SD
---|---|---
you | H : yu are late | 0.16
are | H : you r late | 0.20
late | H : you are leite | 0.88

We use a SRoBERTa [11] model fine tuned on the STSB [13] dataset as the
teacher model, and our custom model with 4 layers (transformer blocks), 128
hidden dimensions and 4 attention heads as the student model. We carry out
distillation with the task of generating embeddings using the AllNLI [14, 15]
and English Wikipedia datasets and use the STSB to evaluate our student model.
As the student model has fewer hidden dimensions (768 of the teacher vs 128)
we perform dimensionality reduction using principal component analysis (PCA).
Our smaller model architecture, named as nanoBERT, uses a word piece tokenizer
during the pre-processing stage, that learns to tokenize a corpus into “word
pieces”, starting from an initial alphabet containing all the characters in
the corpus. The word piece tokenizer then adds tokens with the highest score
to its vocabulary until a suitable size is reached. These word pieces allows
the model to accurately score similar sentences that share same tokens. We
employ the nanoBERT model in the siamese configuration to get SNanoBERT,
followed by a mean pooling of the sentence embedding tensors to derive fixed
size vectors for sentence pairs. We train our model using mean squared error
loss function and our model is distilled at a learning rate of 1e-4. Further,
we use cosine similarity to compute the SD score between two sentences $S_{1}$
and $S_{2}$ as:

$\displaystyle
SD~{}(S_{1},S_{2})=1-\frac{e_{S_{1}}~{}.~{}e_{S_{2}}}{||e_{S_{1}}||~{}||e_{S_{2}}||}$
(1)

where $e_{S_{i}}$ denotes the respective embedding vector generated from our
model for a sentence $S_{i}$. SD is confined between 0 and 1, where lower
scores indicate stronger semantic similarity, and vice versa.

###  2.2 Comparison between SD and WER

In this section, we examine how WER and SD values differ from one another with
some simple examples, and discuss the benefits of SD over WER. We also see
that SD better captures minor syntactic errors that do not affect the true
meaning of the sentence than WER.

When subwords are present in the sentence, the SD score improves because of
the word piece tokenizer used. Suppose GT denotes the ground-truth and H1 and
H2 are two hypotheses from the ASR model, i.e.,GT = ‘Smoking’, H1 = ‘Smoke’,
and H2 = ‘Something’. Then $\text{SD}(\text{GT},\text{H1})$ = 0.09, which is
less than $\text{SD}(\text{GT},\text{H2})$ = 0.85. But if we use WER, then
$\text{WER}(\text{GT},\text{H1})$ = $\text{WER}(\text{GT},\text{H2})$ = 1.

When a word splits into multiple words, WER considers it a completely
incorrect prediction. Despite the fact that we can correctly interpret the
actual word, WER completely dismisses this case with a higher value, whereas
SD assigns a smaller value. For e.g., Suppose GT = ‘seatbelt’, H1 = ‘seat
belt’, then $\text{SD}(\text{GT},\text{H1})$ = 0.12, whereas
$\text{WER}(\text{GT},\text{H1})$ = 2.

Next, we generate hypotheses by inducing mistakes to each word in order to
understand how the SD values vary with respect to the type of words present in
a sentence. We can see from the Table 1 that the SD score increment is much
lower when words like articles and prepositions are incorrect. Meanwhile, when
the keywords are incorrect, as in row 3, the SD scores increase significantly
because they add more value to the meaning of the sentence. Despite the fact
that WER is fixed, we can see that SD assigns different scores to sentence
pairs based on the errors made. As a result, we can conclude that SD scores
outperform WER by learning not to penalise the model excessively for errors
that do not change the true meaning of the sentence.

Table 2: Spearman rank correlation between the cosine similarity of sentence
embeddings and the gold labels for various STS tasks using teacher and our
student models. All models use the siamese network architecture

Model | # Layers | # Hidden | # Attention | Size | Latency | STS-12 | STS-13 | STS-14 | STS-15 | STS-16 | SICK-R | STS-B | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---
|  | Dim | Heads | (MB) | (ms/ |  |  |  |  |  |  |  |
|  |  |  |  | sample) |  |  |  |  |  |  |  |
SRoBERTa | 12 | 768 | 12 | 499.0 | 129.32 | 0.787 | 0.901 | 0.882 | 0.9 | 0.848 | 0.797 | 0.871 | 0.855
SRoBERTa [1,4,7,10] | 4 | 768 | 12 | 271.8 | 41.12 | 0.751 | 0.886 | 0.86 | 0.892 | 0.836 | 0.793 | 0.867 | 0.840
STinyBERT | 4 | 312 | 12 | 57.4 | 6.37 | 0.793 | 0.862 | 0.834 | 0.88 | 0.804 | 0.784 | 0.824 | 0.825
SNanoBERT | 4 | 128 | 4 | 19.2 | 3.33 | 0.732 | 0.851 | 0.807 | 0.866 | 0.801 | 0.761 | 0.81 | 0.804
SPicoBERT | 1 | 64 | 1 | 8.3 | 0.65 | 0.772 | 0.835 | 0.783 | 0.849 | 0.771 | 0.756 | 0.778 | 0.792

###  2.3 Keyword extraction using SNanoBERT

Keyword extraction algorithms are used to extract the most relevant keywords
or key-phrases from text input. We aim to interpret the affect of keywords and
non-keywords in the sentence on the SD score. To get a quick overview of a
text content, it can be helpful to extract keywords that concisely reflect its
semantic context. Our aim is to extract the keywords using the same SNanoBERT
model, instead of using a separate model like keyBERT [16] for keywords. Our
keyword extraction approach is as follows:

$\displaystyle S=$
$\displaystyle\\{x:x=SD~{}(GT,w_{i}),~{}\forall~{}w_{i}\in~{}W\\}\ $
$\displaystyle K=$
$\displaystyle\\{w_{i}:MinMax~{}(S~{}[w_{i}])~{}<~{}\gamma,~{}\forall~{}w_{i}\in~{}W\\}$

where $W$ is the set of words in GT after stopword filtering. We calculate the
SD score for GT and each word in set $W$, and form the set $S$. Then, we
normalise the scores in $S$ by using Min-Max scaling, and construct the set
$K$ by thresholding using $\gamma$ parameter. Next, the rest of the words in
GT are added to a set named $NK$. Thus, we create two sets of words from GT
namely: $K$ and $NK$ containing keywords and non-keywords respectively. Let
$N_{nk}$ denotes the number of non-keywords present in GT, i.e.,
$N_{nk}=|NK|$. $N_{wk}$ and $N_{wnk}$ be the number of wrongly recognized
keywords and non-keywords respectively in the hypothesis. To compute $N_{wk}$
and $N_{wnk}$ from hypothesis, we use an edit-distance based alignment between
GT and hypothesis. Finally, we calculate the non-keyword error rate (NKER) as
the ratio of $N_{wnk}$ to $N_{nk}$.

##  3 A hybrid evaluation metric

We see that the SD score is biased towards keywords that contribute more to
the meaning of the sentence, and that it sometimes fails to give the remaining
words an appropriate weightage. In this section, we propose a new hybrid
evaluation metric that combines the SD score and NKER.

We begin with an example by demonstrating the limitations of SD. Consider GT
as “This is your captain speaking”, and let H1 be “Ths s ur captain speaking”
and H2 be “This is your kepten speaking”. Then
$\text{SD}(\text{GT},\text{H1})$ = 0.22 and $\text{SD}(\text{GT},\text{H2})$ =
0.48 . The SNanoBERT model extracted the keywords ‘captain’ and ‘speaking.’ H1
has all of the keywords correct, but all of the non-keywords are incorrect.
H2, all of the words except ”captain” are correct. As a result, it is natural
to select H2 as the best hypothesis with the fewest errors. However, the SD of
H1 is still lower than that of H2, because the presence of ‘captain’ preserves
the meaning of the sentence and thus overrides the errors in the other non-
keywords. Despite the fact that H2 has fewer errors, we tend to favour H1. SD
appears to be a better metric for tasks where keywords are important. The
solution is to arrive at a metric that considers both semantic correctness and
the number of word errors.

We propose a hybrid evaluation metric which is a weighted combination of SD
and NKER, and is calculated as follows:

$\displaystyle H_{SD}$ $\displaystyle=\alpha_{1}*SD~{}+~{}\alpha_{2}*NKER$ (2)
$\displaystyle\alpha_{1}$
$\displaystyle=\frac{N_{wk}*p}{N_{wnk}+1},~{}~{}\alpha_{2}=\frac{N_{wnk}}{N_{wk}*p+1}$
(3)

where the first term is used to monitor semantic correctness by prioritising
keywords, and the the second term monitors the non-keywords. We know that when
all the keywords are correct, then SD will be lower. In such cases, we must
track the non-keywords in the hypothesis. Similarly, NKER is given a lower
importance when there are no incorrect non-keywords in the hypothesis. We use
$p$, an important tunable hyperparameter that regulates the weightage a
particular keyword has in terms of non-keywords. We calculate the weighting
coefficients $\alpha_{1}$ and $\alpha_{2}$ for SD and NKER based on the number
of wrong keywords and wrong non-keywords present in the hypothesis.

##  4 EXPERIMENTAL RESULTS

We conducted extensive experiments to investigate the efficacy of our proposed
$\text{H}_{\text{SD}}$, and the results are presented in this section.
Throughout our experiments, we kept the parameters $\gamma$ and $p$ at 0.4 and
2, respectively.

###  4.1 SNanoBERT

We benchmark our SNanoBERT model as well as the teacher model (SRoBERTa) on a
collection of sentence similarity benchmarks such as STS-12[17], STS-13[18],
STS-14[19], STS-15[20], STS-16[21], SICK-R[22], STS-B[13] and have tabulated
the spearman correlation coefficient between the cosine-similarity of the
sentence embeddings and the gold labels as shown in Table 2. We created four
distilled siamese models SRoBERTa[1,4,7,10], STinyBERT, SNanoBERT, and
SPicoBERT whose sizes ranges from 271.8 MB to 8.3MB. SNanoBERT achieves an
average correlation coefficient of 0.80, which is only 0.05 lower than
SRoBERTa while being 25.9x smaller and 38.8x faster than SRoBERTa. Although
SPicoBERT achieves results similar to SNanoBERT, we observed that it does not
perform as well in real-world testing. Therefore, we select SNanoBERT which
achieves comparable performance to the teacher SRoBERTa model, meanwhile
significantly reducing the model size and latency, making it suitable for
deploying on edge devices.

###  4.2 Correlation between $\text{H}_{\text{SD}}$ and WER

We begin by examining the relationship between the proposed
$\text{H}_{\text{SD}}$ metric and WER using 1500 text-to-speech generated
samples from 6 different speakers. We use an end-to-end acoustic model similar
to the DeepSpeech2 [23] model. The input speech samples are divided into
frames of 32 milliseconds with 50% overlap. From each frame, 80-dimensional
log-mel features are extracted, stacked together to form the input to the
model. The scatter-plot between $\text{H}_{\text{SD}}$ and WER shows that they
are positively correlated with a Pearson correlation coefficient of 0.78. It
is also observed that as WER increases, the $\text{H}_{\text{SD}}$ values for
a given WER value is more widely distributed.

Table 3: Results from intent recognition and NER: WER, $\text{H}_{\text{SD}}$, intent accuracy and named-entity error rate for benchmark dataset, set A and set B.  Dataset |  | WER
---
(%)
$\text{H}_{\text{SD}}$ |  | Intent
---
accuracy
(%)
| Entity
---
error rate
(%)
ATIS test set | - | - | 98.32 | 15.2
Set A | 27 | 3.48 | 84.43 | 64.43
Set B | 27 | 1.60 | 97.53 | 36.02

###  4.3 Intent recognition and NER

To demonstrate the effectiveness of our metric on various downstream tasks, we
evaluate it for intent recognition and named-entity recognition (NER) tasks.
We use the ATIS [24] training and test set, which contain 4,478 and 893
sentences with 26 intent categories, respectively. For evaluating our
$\text{H}_{\text{SD}}$ metric, we generated two hypothesis sets from ATIS test
set. Set A is made by inducing errors in the keywords by removing vowels.
Similarly, set B is produced by inducing errors in non-keywords. The WER for
both sets is the same.

Our goal is to investigate the connection between our $\text{H}_{\text{SD}}$
metric and intent accuracy. We use the existing jointBERT [25] model trained
on the ATIS and Snips [26] data sets for intent recognition. Intents are
extracted from set A and set B, and intent accuracy is calculated. Table 3
shows that set A has a higher $\text{H}_{\text{SD}}$ metric and a lower intent
accuracy due to incorrectly recognized keywords in the hypothesis. Set B, on
the other hand, reports a lower $\text{H}_{\text{SD}}$ value with higher
intent accuracy. Even though the WER is the same for both sets, we can see
that the $\text{H}_{\text{SD}}$ metric value varies, as does the intent
accuracy. Thus we can conclude that the $\text{H}_{\text{SD}}$ metric is an
effective evaluation metric for intent recognition tasks.

Following that, we investigate the relationship between the
$\text{H}_{\text{SD}}$ metric and the named-entity error rate. For NER, we use
Spacy [27], an open-source software library. Table 3 shows that with a fixed
WER, the $\text{H}_{\text{SD}}$ metric and entity error rate are higher for
set A because the keywords are misspelt. The $\text{H}_{\text{SD}}$ metric is
lower for set B, as is the entity error rate, because non-keywords are
misspelt here. As a result, our proposed metric can be used as a more
effective evaluation metric for NER tasks.

Table 4: Comparison of $\text{H}_{\text{SD}}$ for selected example sentence with fixed WER. Examples | WER | $\text{H}_{\text{SD}}$
---|---|---
GT : The flight is about to land |  |
H1 : The fite is about to lamt | 0.33 | 3.42
H2 : Te flight s about to land | 0.33 | 1.00
GT : Data set needs to be cleaned |  |
H1 : Dase needs to be cleaned | 0.33 | 0.31
H2 : Dataset needs to be cleaned | 0.33 | 0.14
GT : Whomsoever it is concerned |  |
H1 : hm so er it is concerned | 0.75 | 0.78
H2 : whom so ever it is concerned | 0.75 | 0.37

###  4.4 Examples : $\text{H}_{\text{SD}}$ vs WER

We will take some examples as shown in Table 4 and explain the detailed
working of $\text{H}_{\text{SD}}$ and WER. Consider two hypothesis generated
from ASR models as H1 and H2. For all the examples, WER is same for both the
sentences since the number of errors are the same. In the first example, WER
fails to track the improvement of “fite” and “lamt” to “flight” and “land”,
due to the deterioration of “is” and “the” to “s” and “te” at the same time.
Since “flight” and “land” offer more contextual information than “is” and
“the”, $\text{H}_{\text{SD}}$ decreases significantly for H2. In the second
and third example, H2 has a lower $\text{H}_{\text{SD}}$ value because of the
word-piece tokenizer’s ability to capture subwords. Revisiting the example in
Section 3, our $\text{H}_{\text{SD}}$ gives the following results
$\text{H}_{\text{SD}}(\text{GT},\text{H1})$ = 0.97 and
$\text{H}_{\text{SD}}(\text{GT},\text{H2})$ = 0.42. In contrast to SD,
$\text{H}_{\text{SD}}$ favours H2 by taking into account both semantic
correctness and error rate. Hence, making $\text{H}_{\text{SD}}$ well suited
for tracking ASR performance.

##  5 SUMMARY AND CONCLUSIONS

In this work, we built a fast and lightweight SNanoBERT model using
distillation to generate SD score. Our experiments show that the SNanoBERT
model has similar performance to the teacher SRoBERTa model for STS tasks
while being 25.9x smaller and 38.8x faster. Hence, making it suitable for
deploying with ASR models on edge devices. Motivated by the shortcomings of
WER and SD, we propose a new hybrid evaluation metric for ASR systems,
$\text{H}_{\text{SD}}$, which combines the SD and non-keyword error rate. We
demonstrate how the new metric benefits from both semantic correctness and
error rates. We show that $\text{H}_{\text{SD}}$ correlates more strongly with
intent recognition and NER tasks.

## References

  * [1] He, X., Deng, L. and Acero, A.,“ Why word error rate is not a good metric for speech recognizer training for the speech translation task?”, In 2011 IEEE Int. Conf. on Acoustics, Speech and Signal Process. (ICASSP) (pp. 5632-5635).
  * [2] Favre, B., Cheung, K., Kazemian, S., Lee, A., Liu, Y., Munteanu, C., Zeller, F.,“ Automatic human utility evaluation of ASR systems: Does WER really predict performance?”, INTERSPEECH (pp. 3463-3467), 2013.
  * [3] McCowan, I. A., Moore, D., Dines, J., Gatica-Perez, D., Flynn, M., Wellner, P., Bourlard, H.,“ On the use of information retrieval measures for speech recognition evaluation”, IDIAP, (2004).
  * [4] Morris, A. C., Maier, V., Green, P., “From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition,”. In Eighth Int. Conf. on Spoken Language Process.,(2004).
  * [5] J. Makhoul, F. Kubala, R. Schwartz, R. Weischedelet al., “Performance measures for information extraction”, in Proceedings of DARPA broadcast news workshop, Herndon, VA, 1999, pp. 249–252.
  * [6] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N.Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in NeurIPS, 2017.
  * [7] Devlin, J., Chang, M.W., Lee, K. and Toutanova, K.,“Bert: Pre-training of deep bidirectional transformers for language understanding”, arXiv preprint arXiv:1810.04805, 2018.
  * [8] Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V,“ Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.
  * [9] Kim, Suyoun, et al. “Semantic Distance: A New Metric For ASR Performance Analysis Towards Spoken Language Understanding”, arXiv preprint arXiv:2104.02138, 2021.
  * [10] Suyoun, K. et al. “Evaluating user perception of speech recognition system quality with semantic distance metric,” arXiv preprint arXiv:2110.05376, 2021.
  * [11] Reimers, N. and Gurevych, I., “Sentence-bert: Sentence embeddings using siamese bert-networks”, arXiv preprint arXiv:1908.10084, 2019.
  * [12] Hinton, G.E., Vinyals, O. and Dean, J. “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531 2, 2015.
  * [13] Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I. and Specia, L.,“ Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation”. arXiv preprint arXiv:1708.00055,2017.
  * [14] Adina, W., Nangia, N., and Bowman, S. “A broad-coverage challenge corpus for sentence understanding through inference.” arXiv preprint arXiv:1704.05426 (2017).
  * [15] Bowman, S.R., Angeli, G., Potts, C. and Manning, C.D., “A large annotated corpus for learning natural language inference”, arXiv preprint arXiv:1508.05326, 2015.
  * [16] Maarten Grootendorst, Keybert: Minimal key-word extraction with bert, 2020. https://zenodo.org/record/4461265.Y1pVfSbhWDQ
  * [17] Agirre, E et al.,“ SemEval-2012,” Joint Conference on Lexical and Computational Semantics, SemEval’16.
  * [18] Agirre, E et al.,“ SEM 2013 sharedtask: Semantic Textual Similarity”, In Second joint conf. on lexical and computational semantics, (pp. 32-43),2013.
  * [19] Agirre, E et al. ,“SemEval-2014 ”, Int. Workshop on Semantic Evaluation(SemEval 2014).
  * [20] Agirre, E et al. ,“ SemEval-2015 ,”Int. Workshop on Semantic Evaluation, 2015.
  * [21] Agirre, E et al. ,“SemEval-2016”, Int. Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016.
  * [22] Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R. and Zamparelli, R.,“ A SICK cure for the evaluation of compositional distributional semantic models”. Int. Conf. on Language Resources and Evaluation,2014.
  * [23] D. Amodei et al.,“Deep speech 2: End-to-end speech recognition in english and mandarin,” Int. Conf. on Machine Learning, 2016.
  * [24] G. Tur, D. Hakkani-Tür and L. Heck, “What is left to be understood in ATIS?,” IEEE Spoken Language Technology Workshop, 2010.
  * [25] Chen, Q., Zhuo, Z. and Wang, W., “Bert for joint intent classification and slot filling”, arXiv preprint arXiv:1902.10909, 2019.
  * [26] Coucke, A., Saade, A., Ball, A., Bluche, T., Caulier, A., Leroy, D., Dureau, J.,“Snips voice platform: an embedded spoken language understanding system for private by design voice interfaces”, arXiv preprint arXiv:1805.10190, (2018).
  * [27] M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd,“spaCy:Industrial-strength Natural Language Processing in Python,”2020,[Online].