# On Measuring Calibration of Discrete Probabilistic Neural Networks

Spencer Young  Delicious AIUnited States<EMAIL_ADDRESS>and
Porter Jenkins  Brigham Young UniversityUnited States<EMAIL_ADDRESS>
###### Abstract.

As machine learning systems become increasingly integrated into real-world
applications, accurately representing uncertainty is crucial for enhancing
their safety, robustness, and reliability. Training neural networks to fit
high-dimensional probability distributions via maximum likelihood has become
an effective method for uncertainty quantification. However, such models often
exhibit poor calibration, leading to overconfident predictions. Traditional
metrics like Expected Calibration Error (ECE) and Negative Log Likelihood
(NLL) have limitations, including biases and parametric assumptions. This
paper proposes a new approach using conditional kernel mean embeddings to
measure calibration discrepancies without these biases and assumptions.
Preliminary experiments on synthetic data demonstrate the method’s potential,
with future work planned for more complex applications.

##  1\. Introduction

As more machine learning technologies are deployed into the real world,
increasing attention is being paid to the ability of these systems to
accurately represent their uncertainty. Faithful uncertainty quantification
improves the safety, robustness, and reliability an ML system in the face of a
noisy environment.

While a variety of methods exist for building uncertainty-aware ML systems, in
recent years training neural networks to fit high dimensional probability
distributions via maximum likelihood has emerged as an effective strategy. For
example, a simple neural network classifier trained with cross entropy loss
can also be viewed as one trained to maximize the likelihood of a categorical
distribution over the output classes. In the regression setting, one can
parameterize the mean and variance of a Gaussian distribution (Nix and
Weigend, 1994; Stirn et al., 2023; Immer et al., 2024) as the outputs of a
neural network,
$\left[\hat{\mu}_{i},\hat{\sigma}_{i}^{2}\right]^{T}=\mathbf{f}_{\mathbf{\Theta}}(\textbf{x}_{i})$
and maximize the likelihood of the regression targets. In the case of
generative models, techniques such Variational Autoencoders (VAEs),
Normalizing Flows and Diffusion models all maximize an exact or lower-bound
estimate of the likelihood of the observed data. In the general, these models
output a predictive probability distribution over the targets.

However, modern neural networks trained under this maximum likelihood regime
have been shown to be poorly calibrated, resulting in over-confident
predictive distributions (Guo et al., 2017). This is in large part due to
overparameterization, as well as underspecification by a finite training
dataset. Proper calibration broadly means that for a given event, a model’s
predicted probability of the event occurring should align exactly with the
event’s true likelihood. The notion of calibration was first defined in the
classification setting by (Dawid, 1982) and Expected Calibration Error (ECE)
(Naeini et al., 2015) has emerged as a common measure of calibration. ECE has
since been extended to the multi-class setting (Nixon et al., 2019) and to
regression problems (Gneiting et al., 2007; Kuleshov et al., 2018).
Additionally, negative log likelihood (NLL) has been used as an alternative
measurement due its properties as a proper scoring rule (Lakshminarayanan et
al., 2017). Despite a general understanding of how a calibrated model should
behave, no consensus yet exists for how to best quantify a model’s calibration
(Błasiok et al., 2023).

In fact, many of these measurements have theoretical shortcomings. For
example, NLL requires 1) specific parametric assumptions about the target 2)
its minimum isn’t 0 for a perfectly specified model and can’t be viewed as a
distance from calibration, and 3) can be difficult to compute for certain
classes of models. Moreover, in this paper we show that ECE has a number of
shortcomings including bias in favor of continuous (vs. discrete)
distributions, and an implicit assumption of the uniformity of the probability
integral transform (PIT).

To address these issues, we adopt the theoretical framework of (Błasiok et
al., 2023) to rigorously define the calibration of a probabilistic regressor.
We show how recent theoretical advances (Park and Muandet, 2020) in
conditional kernel mean embeddings can be used to measure the discrepancy
between learned and empirical conditional distributions of neural regression
models. A major advantage of this approach is that it removes the biases
present when comparing both discrete and continuous outputs via ECE.
Additionally, because this approach compares samples from a probabilistic
model with the ground truth samples from the test set, it removes the need to
make rigid parametric assumptions, and can be applied to models where the
likelihood is difficult or impossible to evaluate. We illustrate the behavior
and effectiveness of this method through a series of preliminary experiments
on a synthetic data where the data generating process is known. Future work
will look at this technique in more complicated, higher dimensional problems
such as image regression or generative modelling.

##  2\. Preliminaries

Suppose we are given an unknown, input-conditional data-generating
distribution $Y|X$, where $X\in\mathcal{X}$ and $Y\in\mathcal{Y}$. We denote
$\mathcal{P}_{Y|X}$ to be the space of all possible $X$-conditional
distributions over $Y$.

We assume access to a probabilistic model
$f:\mathcal{X}\rightarrow\mathcal{P}_{Y|X}$, as well as $T$ draws from $Y|X$,
$\\{(x_{t},y_{t})\\}_{t=1}^{T}$, that have been withheld from model training
and can thus be used for evaluation.

We define calibration as follows:

###### Definition 0.

A probabilistic model $f:\mathcal{X}\rightarrow\mathcal{P}_{Y|X}$ is
calibrated if, for all $X\in\mathcal{X}$, we have $Y|X\sim f(X)$.

Inspired by (Błasiok et al., 2023), we choose to quantify calibration by
measuring distance to the closest calibrated model. Let
$\mathscr{F}_{(\mathcal{X},\mathcal{Y})}$ be the space of all probabilistic
models mapping $X\in\mathcal{X}$ to $\mathcal{P}_{Y|X}$. Let
$\mathscr{C}_{Y|X}:=\\{g\nonscript\,|\allowbreak\nonscript\,\mathopen{}g\in\mathscr{F}_{(\mathcal{X},\mathcal{Y})},\;Y|X\sim
g(X)\;\forall X\in\mathcal{X}\\}$ be the set of all calibrated models
describing $Y|X$. Given a model $f$, its “distance from calibration”
(hereafter called calibration error) is defined as

(1) $\text{CE}_{Y|X}(f)=\inf_{g\in\mathscr{C}_{Y|X}}\
\operatorname*{\mathbb{E}}_{Y|X}\bigl{[}d(f,g)\bigr{]}$

where $d$ denotes an arbitrary distance measure for the metric space of all
models.

##  3\. Measuring Calibration

Despite the usefulness of Equation 1 for theoretically treating calibration,
in practice, computing the calibration error is intractable, seeing as:

  1. (1)

We do not have access to $\mathscr{C}_{Y|X}$.

  2. (2)

We only have access to a finite number of draws from $Y|X$.

  3. (3)

Distance between $f$ and any $g$ is ill-defined in the space of all possible
models.

  4. (4)

We are not guaranteed that $\mathscr{C}_{Y|X}$ is a convex set.

Thus, estimation of $\text{CE}_{Y|X}(f)$ is the common practice.

Figure 1. We simulate four data generating processes for $Y|X$ with identical
first and second moments: (1) $\mathcal{N}(X,X)$, (2) $\texttt{Poisson}(X)$,
(3) $\texttt{DoublePoisson}(X,1)$ (see (Efron, 1986) for a definition), and
(4)
$\texttt{NegativeBinomial}(\lceil\frac{X^{2}}{\varepsilon}\rceil,1-{\varepsilon})$,
where ($\varepsilon$<< 1). For each dataset, we form a perfectly specified
probabilistic model and visualize its mean fit (along with 95% centered
credible intervals shaded in gray) in row 1. We also show the histogram of the
model’s Probability Integral Transform (PIT) in row 2 and reliability diagrams
plotting $\hat{p}_{j}$ against $p_{j}$ (as outlined in (Kuleshov et al.,
2018)) in row 3. Overlaid on each reliability diagram is the model’s expected
calibration error as defined in Equation 2 (with $\alpha=1$). In the final
row, we display the Maximum Conditional Mean Discrepancy (MCMD) between
samples drawn from the posterior predictive distribution and the ground truth
dataset. MCMD values are evaluated on a grid spanning the domain of $X$, with
the mean displayed near the top of each plot. Though the commonly-used PIT
histograms, reliability diagrams, and ECE values identify a well-specified
Gaussian model for the first data generating process, they fail to capture
probabilistic calibration for discrete targets. Meanwhile, our proposed
approach for measuring calibration via the MCMD correctly indicates that each
model is (with some sampling error) perfectly calibrated.

###  3.1. Expected Calibration Error

One estimate of $\text{CE}_{Y|X}(f)$ that is commonly used to measure the
calibration of probabilistic regression models is the expected calibration
error, defined by (Kuleshov et al., 2018) as

(2)
$\text{ECE}(F_{1},y_{1},...,F_{T},y_{T})=\sum_{j=1}^{m}w_{j}\cdot\left|p_{j}-\hat{p}_{j}\right|^{\alpha}$

where we select $m$ confidence levels $0\leq p_{1}<p_{2}<...<p_{m}\leq 1$, and
for each $p_{j}$ we compute the empirical quantity (with $F_{t}$ denoting a
model’s predicted CDF for target $y_{t}$)

(3) $\hat{p}_{j}=\mathbb{E}_{t}[\mathbbm{1}\\{F_{t}(y_{t}|x_{t})\leq
p_{j}\\}]=\frac{\left|\\{y_{t}|F_{t}(y_{t})\leq p_{j},t=1,...,T\right|}{T}$

This metric can be viewed as a direct application of the probability integral
transform (PIT; also known as universality of the uniform), since it is simply
an empirical estimate of the distance (Wasserstein when $\alpha=1$, Cramér–von
Mises when $\alpha=2$) between realizations of the PIT
$Z_{t}=F_{t}(y_{t}|x_{t})$ and $\texttt{Uniform}(0,1)$ (Dheur and Taieb,
2023). Interpreted within the framework of Equation 1, the ECE assumes that
the closest $g\in\mathscr{C}_{Y|X}$ is the model whose PIT is
$\text{Uniform}(0,1)$, and chooses the distance between the probability
integral transforms of $f$ and $g$ for its distance measure $d$ . We note that
(Hamill, 2001) has already identified the pitfalls of assuming that uniformity
of the PIT is a sufficient condition for an ideal model.

When the assumptions of the Probability Integral Transform hold (namely, that
$Y|X$ is a random variable with a continuous CDF), the expected calibration
error can differentiate between well-calibrated and miscalibrated
distributions. However, we note that in the discrete setting, $Y|X$ (and by
extension, its CDF) is not continuous. In fact, a discrete CDF has a countable
range, thus its PIT realizations can never be distributed as
$\texttt{Uniform}(0,1)$. This provides reason to doubt the applicability of
the expected calibration error for count regression. We illustrate this point
empirically in Figure 1, in which perfectly specified count models fail to
achieve zero (or approximately zero) ECE.

###  3.2. Maximum Conditional Mean Discrepancy

In the typical probabilistic machine learning setup, we are only given a
finite number of draws from $Y|X$ to use for evaluation. This introduces great
difficulty in estimating $CE_{Y|X}$, especially since, as pointed out in
(Błasiok et al., 2023), we require multiple examples with the exact same
conditioning input $x$ to estimate $Y|X=x$ with any degree of confidence. In
most domains, the probability of this occurrence across multiple inputs is
essentially zero.

One reasonable assumption to make estimation of $Y|X=x$ more tractable is that
similar points $x$ should yield similar conditional distributions $Y|X=x$.
Thus, $Y|X=x$ can be approximated by aggregating $(x,y)$ samples with similar
$x$ values. This process is formalized using kernel functions.

A kernel function $k:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$ is
commonly used as a method to measure similarity between two point, where a
kernel $k$ that is symmetric, continuous, and positive semi-definite,
$k(\mathbf{x},\mathbf{x^{\prime}})$ represents an inner product
$\langle\mathbf{\phi}(\mathbf{x}),\mathbf{\phi}(\mathbf{x^{\prime}})\rangle_{\mathcal{V}}$
in some (possibly higher-dimensional) feature space $\mathcal{V}$.

Kernel mean embeddings (Muandet et al., 2017) use a special class of kernels
called characteristic kernels to represent a probability distribution $p(z)$
in an infinite-dimensional reproducing kernel Hilbert space (RKHS) directly
from data points (Muandet et al., 2017). One application of this technique is
the maximum mean discrepancy (MMD), which estimates the distance between two
distributions by computing their RKHS inner product via the kernel trick. The
MMD was extended to the conditional setting in (Muandet et al., 2017),
resulting in the Maximum Conditional Mean Discrepancy (MCMD). More recently,
(Park and Muandet, 2020) derived a measure-theoretic analytical solution that
reduces the stringent assumptions of previous results. The MCMD applies
kernels in both the conditioning/input space and the output space, providing
us with an estimate of the distance between two conditional distributions
$Y|X$ and $Y^{\prime}|X^{\prime}$ along a grid of conditioning values.
Advantageously, (Park and Muandet, 2020) showed how to compute this measure in
closed form.

####  3.2.1. Computing MCMD

Given a set of test samples, $\\{(x_{i},y_{i}\\})_{i=1}^{n}$ drawn from a
conditional distribution $P_{Y|X}$, and a set of samples from a model to be
evaluated, $\\{(x_{i},y^{\prime}_{i}\\})_{i=1}^{n}$, drawn from
$P_{Y^{\prime}|X}$ we use the use the following closed-form solution for the
squared MCMD (Park and Muandet, 2020):

$\displaystyle\hat{M}^{2}(P_{Y|X},P_{Y^{\prime}|X})=\textbf{k}^{T}_{Y}(\cdot)\textbf{W}_{Y}\textbf{K}_{X}\textbf{W}_{Y}^{T}\textbf{k}_{y}(\cdot)-$
(4) $\displaystyle
2\textbf{k}_{Y}^{T}\textbf{W}_{Y}\textbf{K}_{XX^{\prime}}\textbf{W}_{Y}^{T}\textbf{k}_{Y^{\prime}}+\textbf{k}_{Y^{\prime}}^{T}\textbf{W}_{Y^{\prime}}\textbf{K}_{X^{\prime}}\textbf{W}_{Y^{\prime}}^{T}\textbf{k}_{Y^{\prime}}(\cdot)$

Where $\textbf{K}_{X}^{(ij)}=k_{\mathcal{X}}(x_{i},x_{j})$,
$\textbf{K}_{X^{\prime}}^{(ij)}=k_{\mathcal{X^{\prime}}}(x^{\prime}_{i},x^{\prime}_{j})$,
$\textbf{K}_{XX^{\prime}}^{(ij)}=k_{\mathcal{X}}(x_{i},x^{\prime}_{j})$,
$\textbf{K}_{Y^{\prime}}^{(ij)}=k_{\mathcal{X}}(y^{\prime}_{i},y^{\prime}_{j})$,
$\textbf{k}_{Y^{\prime}}(\cdot)$ =
$(k_{Y}(y^{\prime}_{1},\cdot),...,k_{Y}(y^{\prime}_{m},\cdot))^{T}$,
$\textbf{W}_{Y}=(\textbf{K}_{Y}+n\lambda\textbf{I}_{n})^{-1}$ and
$\textbf{W}_{Y^{\prime}}=(\textbf{K}_{Y^{\prime}}+m\lambda^{\prime}\textbf{I}_{m})^{-1}$

To use the MCMD to measure the calibration of a model $f$, we apply the
following procedure:

  1. (1)

For $t$ in $[1..T]$, draw $m$ samples from $f(x_{t})$. This yields a set of
$mT$ samples from the model’s posterior predictive distribution over the
evaluation dataset.

  2. (2)

Across a grid of uniformly spaced values in
$[\min_{t}(x_{t}),\max_{t}(x_{t})]$, compute the MCMD between
$(x_{t},y_{t})_{t=1}^{T}$ and $\\{(x_{t},y_{t_{i}})_{i=1}^{m}\\}_{t=1}^{T}$.

  3. (3)

Calculate the mean of the computed MCMD values.

We note that if we define $g$ to be nature’s model that generated $Y|X$, and
use the MCMD as our distance metric $d$, this proposed procedure (taking the
mean/expectation of MCMD values across a collection of inputs/outputs) is
essentially a direct estimate of $CE_{Y|X}$.

##  4\. Experiments

We first illustrate the behavior of the MCMD on a variety of synthetically-
generated conditional distributions. We then provide a case study showing how
MCMD can be used to effectively evaluate the calibration of a trained
probabilistic regression model. Unless otherwise specified in this section, we
use $(x,y)$ and $(x^{\prime},y^{\prime})$ to denote sets of draws from two
conditional distributions to be compared, i.e. $Y|X$ vs.
$Y^{\prime}|X^{\prime}$. We refer to the mean and standard deviation of these
conditional distributions as $\mu$ or $\mu^{\prime}$ and $\sigma$ or
$\sigma^{\prime}$, respectively.

Figure 2. MCMD plots for homoscedastic ground-truth data compared against
various conditional distributions. The mean and standard deviation of the
ground-truth data generating process are denoted $\mu$ and $\sigma$,
respectively, while the mean and standard deviation of the distribution being
compared to the ground truth are written $\mu^{\prime}$ and $\sigma^{\prime}$.
The MCMD exhibits the ability to delineate samples from different
distributions, regardless of how the distributions are misaligned (different
means, different variances, or some combination of the two). A few desirable
behaviors include 1) penalizing poor mean fit more harshly when the requisite
variance is not increased to explain the mismatch (compare columns 4 and 5);
2) penalizing poorly-fit uncertainty even when the means are aligned (see
columns 2 and 3); and 3) identifying specific areas in the input where the
distributional discrepancy is largest (see column 6). Figure 3. MCMD plots for
heteroscedastic ground-truth data compared against various conditional
distributions. The mean and standard deviation of the ground-truth data
generating process are denoted $\mu$ and $\sigma$, respectively, while the
mean and standard deviation of the distribution being compared to the ground
truth are written $\mu^{\prime}$ and $\sigma^{\prime}$. Just as in the
homoscedastic case, the MCMD quantifies all forms of distributional
misalignment in a way that matches with intuition.

###  4.1. General Behavior

To investigate the behavior of the MCMD, we show how it estimates the distance
between distributions in a variety of settings. We simulate both homoscedastic
and heteroscedastic data generating processes, sampling 1000 $(x,y)$ pairs
from each. For both processes, we then draw 500 $(x^{\prime},y^{\prime})$
pairs from 1) the same distribution; 2) a distribution with the same mean but
smaller variance; 3) a distribution with the same mean but higher variance; 4)
a distribution with a different mean and smaller variance; 5) a distribution
with a different mean and higher variance; and 6) a distribution with an
entirely different conditional relationship / shape. Pairing each of (1-6)
with the original distribution, we then compute and plot the MCMD between
$(x,y)$ and $(x^{\prime},y^{\prime})$ across a uniform grid spanning the input
space. In addition to the MCMD plot, we report the mean value of the MCMD. We
compute the MCMD with the regularization parameter $\lambda=0.1$. To measure
similarity in both $x$ and $y$, we employ the popular radial basis function
(RBF) kernel

(5)
$k_{\gamma}(\mathbf{x},\mathbf{x^{\prime}})=\exp{(-\gamma||\mathbf{x}-\mathbf{x^{\prime}}||^{2})}$

with $\gamma=0.5$. Results for the homoscedastic dataset are visualized in
Figure 2, while the heteroscedastic case is depicted in Figure 3. Details
about the exact data-generating processes used to produce these figures are
provided in Appendix A.

##### Results

Overall, the MCMD appears to behave quite intuitively, with higher values
consistently corresponding to areas where the distributions vary from each
other more in both mean and standard deviation. Beyond the “eye test”, we
specifically highlight a few key desirable characteristics of this metric.
First, we see from columns 4 and 5 of Figures 2 and 3 that poor mean fit is
penalized more harshly by MCMD when the corresponding variance is low. This
indicates that the MCMD requires high variance to explain misalignment of
means, which is arguably the correct stipulation to enforce. Additionally,
from columns 2 and 3 of the figures, we observe that the MCMD recognizes the
case where the means match but the spread of two distributions is different.
This suggests the MCMD is robust at identifying various forms of potential
miscalibration between a predictive distribution and the ground truth.
Finally, we note that unlike other common calibration metrics, the MCMD can
yield both a summary statistic (the mean MCMD) indicating overall calibration
and provide a granular notion of which range of inputs a model is most
miscalibrated on (since it is evaluated over a grid) – valuable feedback that
can assist in selecting further training procedures.

###  4.2. Evaluating Calibration with MCMD

Figure 4. Visualization of the probabilistic fit of 3 deep neural networks on
a synthetic dataset with discrete regression targets. In the first row, we
plot the mean and centered 95% credible interval for each model on the test
split of the dataset, along with the mean absolute error (MAE) and NLL. In the
second row, we depict the MCMD between the ground-truth and predictive
distributions, along with the mean MCMD across the evaluation grid. In
aggregate, the MCMD correctly identifies the Gaussian DNN as the best model in
terms of probabilistic fit, with a mean MCMD of almost zero. We also observe
that for the generally miscalibrated Poisson DNN and NB DNN models, the MCMD
is highest over the inputs in $[\pi,2\pi]$. This corresponds to the worst
regions of probabilistic fit for these models, since they lack the flexibility
to represent the under-dispersion at these counts. This toy example
demonstrates the usefulness of MCMD for quantifying model calibration.

Since we are proposing the use of MCMD to measure the quality of posterior
predictive distributions, we run a toy experiment to illustrate how this
metric might be used to identify calibrated vs. miscalibrated models.

We simulate a discrete dataset that exhibits both under-dispersion (variance
lower than the count) and over-dispersion (variance higher than the count). We
then train a small multi-layer perceptron (MLP) to output the parameters of a
Poisson, Negative Binomial, or Gaussian distribution using the appropriate
negative log likelihood (NLL) loss. The resultant models’ predictive
distributions over the test split of the synthetic dataset are visualized in
row 1 of Figure 4, with the respective mean absolute error (MAE) and NLL of
the model overlaid on the plot. To measure the MCMD between our $T$ ground-
truth test samples $(X,Y)$ and the model’s predictive distribution, we form
$(X^{\prime},Y^{\prime})$ in the following way: For each test point $y_{t}$,
we draw 100 samples from $f(x_{t})$, the model’s $x_{t}$-conditional
predictive distribution. This yields $100T$ total samples from
$(X^{\prime},Y^{\prime})$, where for a given $t$ we have corresponding draws
$(x_{t},y^{\prime}_{t_{i}}\sim f(x_{t}))_{i=1}^{100}$. We then compute the
MCMD over a uniformly-spaced grid of $x$ values, plotting the resultant scores
in row 2 of Figure 4 (with the mean MCMD indicated in each respective plot).
As in Section 4.1, we set $\lambda=0.1$ and use the RBF kernel with
$\gamma=0.5$. Training and architecture details for each MLP, along with the
generating process for the synthetic dataset, can be found in Appendix B.

##### Results

Since the Poisson and Negative Binomial (NB) distributions do not have the
flexibility to model under-dispersion, they struggle most in the $[\pi,2\pi]$
region of the input (where the requisite regression targets have very low
variance). Accordingly, this is where the MCMD identifies the highest
discrepancy between $(X,Y)$ and $(X^{\prime},Y^{\prime})$. Meanwhile, the
Gaussian model achieves both a good mean fit and calibrated uncertainty
predictions. This is supported by its near-zero MCMD values. Using the mean
MCMD as a ranking statistic, we can (rightly) conclude that the Gaussian DNN
achieves the best probabilistic fit. We note that this fact is obvious in our
univariate setup where the predictive distributions can be easily visualized,
but in higher dimensions where we cannot depict the exact learned distribution
of the model, a metric such as the mean MCMD provides much greater value in
model selection. We also note that though the NLL suggests a similar ranking
of models as the mean MCMD in this scenario, its undetermined lower bound for
a specific dataset (as mentioned in (Błasiok et al., 2023)) makes it difficult
to quantify how close to perfectly calibrated each model is, since the NLL
cannot be easily interpreted as an estimate of $CE_{Y|X}$.

##  5\. Conclusion

Overall, we see that significant issues exist with using ECE to evaluate the
quality of uncertainty quantification for certain classes of probabilistic
neural networks. We propose to use MCMD to remedy this issue, and show that it
provides a promising new alternative to likelihood-based methods and ECE for
measuring the calibration of models. MCMD is advantageous because it is
distribution-free and therefore removes biases that arise from the choice of
parametric assumptions; additionally it only requires samples from a
probabilistic model and can therefore be used with models where the likelihood
cannot be evaluated.

###  5.1. Limitations and Future Work

This paper is meant to shed light on problems of existing approaches and to
present preliminary evidence that MCMD provides a useful evaluation framework
for assessing model calibration. There are a number of open questions related
to implementation we leave for future work. First, all of our experiments are
on simple toy problems. Future work should explore extending this idea to
higher dimensional problems such as image regression or language modeling. It
will likely be necessary to first use a function to embed the raw data into
lower dimensional output akin to Inception Score (Salimans et al., 2016) or
FID score (Heusel et al., 2017). Additionally, further work should study the
choice of kernel, as well as the impact of hyperparameters and sample size.

## References

  * (1)
  * Błasiok et al. (2023) Jarosław Błasiok, Parikshit Gopalan, Lunjia Hu, and Preetum Nakkiran. 2023.  A unifying theory of distance from calibration. In _Proceedings of the 55th Annual ACM Symposium on Theory of Computing_. 1727–1740.
  * Dawid (1982) AP Dawid. 1982.  Objective probability forecasts.  _University College London, Dept. of Statistical Science. Research Report_ 14 (1982).
  * Dheur and Taieb (2023) Victor Dheur and Souhaib Ben Taieb. 2023.  A large-scale study of probabilistic calibration in neural network regression. In _International Conference on Machine Learning_. PMLR, 7813–7836.
  * Efron (1986) Bradley Efron. 1986.  Double exponential families and their use in generalized linear regression.  _J. Amer. Statist. Assoc._ 81, 395 (1986), 709–721.
  * Gneiting et al. (2007) Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. 2007.  Probabilistic forecasts, calibration and sharpness.  _Journal of the Royal Statistical Society Series B: Statistical Methodology_ 69, 2 (2007), 243–268.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017.  On calibration of modern neural networks. In _International conference on machine learning_. PMLR, 1321–1330.
  * Hamill (2001) Thomas M Hamill. 2001.  Interpretation of rank histograms for verifying ensemble forecasts.  _Monthly Weather Review_ 129, 3 (2001), 550–560.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ 30 (2017).
  * Hill (2011) Theodore Hill. 2011.  Conflations of probability distributions.  _Trans. Amer. Math. Soc._ 363, 6 (2011), 3351–3372.
  * Immer et al. (2024) Alexander Immer, Emanuele Palumbo, Alexander Marx, and Julia Vogt. 2024.  Effective Bayesian heteroscedastic regression with deep neural networks.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Kuleshov et al. (2018) Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. 2018.  Accurate uncertainties for deep learning using calibrated regression. In _International conference on machine learning_. PMLR, 2796–2804.
  * Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _Advances in neural information processing systems_ 30 (2017).
  * Loshchilov and Hutter (2017a) Ilya Loshchilov and Frank Hutter. 2017a.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ (2017).
  * Loshchilov and Hutter (2017b) Ilya Loshchilov and Frank Hutter. 2017b.  SGDR: Stochastic Gradient Descent with Warm Restarts.  arXiv:1608.03983 [cs.LG]
  * Muandet et al. (2017) Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, Bernhard Schölkopf, et al. 2017\.  Kernel mean embedding of distributions: A review and beyond.  _Foundations and Trends® in Machine Learning_ 10, 1-2 (2017), 1–141.
  * Naeini et al. (2015) Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015.  Obtaining well calibrated probabilities using bayesian binning. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 29.
  * Nix and Weigend (1994) David A Nix and Andreas S Weigend. 1994.  Estimating the mean and variance of the target probability distribution. In _Proceedings of 1994 ieee international conference on neural networks (ICNN’94)_ , Vol. 1. IEEE, 55–60.
  * Nixon et al. (2019) Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. 2019.  Measuring calibration in deep learning.. In _CVPR workshops_ , Vol. 2.
  * Park and Muandet (2020) Junhyung Park and Krikamol Muandet. 2020.  A measure-theoretic approach to kernel conditional mean embeddings.  _Advances in neural information processing systems_ 33 (2020), 21247–21259.
  * Salimans et al. (2016) Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016.  Improved techniques for training gans.  _Advances in neural information processing systems_ 29 (2016).
  * Stirn et al. (2023) Andrew Stirn, Harm Wessels, Megan Schertzer, Laura Pereira, Neville Sanjana, and David Knowles. 2023.  Faithful heteroscedastic regression with neural networks. In _International Conference on Artificial Intelligence and Statistics_. PMLR, 5593–5613.

##  Appendix A Experimental Details: General Behavior

Here we provide the data-generating processes used in the experiments detailed
in Section 4.1.

###  A.1. Homoscedastic Datasets

To create the “ground-truth” homoscedastic distribution $(x,y)$ used in these
experiments, we first sampled $x\sim\texttt{Uniform}[0,2\pi]$. We then drew
our targets $y|x\sim\mathcal{N}(\cos{(x)},\frac{1}{2})$. When forming
distributions $(x^{\prime},y^{\prime})$ to compare against with MCMD, we also
drew $x\sim\texttt{Uniform}[0,2\pi]$. We then sampled
$y|x\sim\mathcal{N}(\mu^{\prime}(x),\sigma^{\prime})$, where:

\- When $\mu^{\prime}\neq\mu$, we fixed $\mu^{\prime}(x)=\mu(x)-2$

\- When $\sigma^{\prime}<\sigma$, we fixed $\sigma^{\prime}=\frac{1}{10}$

\- When $\sigma^{\prime}>\sigma$, we fixed $\sigma^{\prime}=\frac{3}{2}$

\- When a different shape was desired, we let $\mu^{\prime}(x)=x-5$ and
$\sigma^{\prime}=\frac{1}{2}$

###  A.2. Heteroscedastic Datasets

To create the “ground-truth” heteroscedastic distribution $(x,y)$ used in
these experiments, we first sampled $x\sim\texttt{Uniform}[-\pi,\pi]$. We then
drew our targets
$y|x\sim\mathcal{N}\left(\sin{(x)},\sqrt{\max{\\{x,\frac{1}{5}\\}}}\right)$.
When forming distributions $(x^{\prime},y^{\prime})$ to compare against with
MCMD, we also drew $x\sim\texttt{Uniform}[-\pi,\pi]$. We then sampled
$y|x\sim\mathcal{N}(\mu^{\prime}(x),\sigma^{\prime}(x))$, where:

\- When $\mu^{\prime}\neq\mu$, we fixed $\mu^{\prime}(x)=\mu(x)-3$

\- When $\sigma^{\prime}<\sigma$, we fixed
$\sigma^{\prime}(x)=\frac{\sigma(x)}{4}$

\- When $\sigma^{\prime}>\sigma$, we fixed $\sigma^{\prime}(x)=2\sigma(x)$

\- When a different shape was desired, we let $\mu^{\prime}(x)=x-5$ and
$\sigma^{\prime}(x)=\frac{1}{2}$

##  Appendix B Experimental Details: Evaluating Calibration with MCMD

The synthetic dataset used in Section 4.2 is generated with the following
procedure: First, we sample $x$ from a uniform distribution,
$x\sim\texttt{Uniform}(0,2\pi)$. Next, we draw an initial proposal for $y$
from a conflation (Hill, 2011) of five identical Poissons, each with rate
parameterized by $\lambda(x)=10\sin(x)+10$. Finally, we scale $y$ by $-1$ and
shift it by $+30$.

Each MLP trained for this experiment had an identical base architecture, with
layers of width [128, 128, 128, 64]. For each method, the output dimension was
either 1 or 2, based on the number of parameters needed to define the
distribution being regressed ($d=2$ for the Gaussian DNN and NB DNN, $d=1$ for
the Poisson DNN). Networks were trained for 200 epochs on a CPU with a batch
size of 32 using the AdamW optimizer (Loshchilov and Hutter, 2017a). The
initial learning rate was set to $10^{-3}$ and annealed to 0 with a cosine
schedule (Loshchilov and Hutter, 2017b), and weight decay was set to
$10^{-5}$.


# Quantum buckling in metal-organic framework materials

R. Matthias Geilhufe  Nordita, KTH Royal Institute of Technology and Stockholm
University, Roslagstullsbacken 23, 10691 Stockholm, Sweden
<EMAIL_ADDRESS>
###### Abstract

Metal organic frameworks are porous materials composed of metal ions or
clusters coordinated by organic molecules. As a response to applied uniaxial
pressure, molecules of straight shape in the framework start to buckle. Under
sufficiently low temperatures, this buckling is of quantum nature, described
by a superposition of degenerate buckling states. Buckling states of adjacent
molecules couple in a transverse Ising type behavior. On the example of the
metal organic framework topology MOF-5 we derive the phase diagram under
applied strain, showing a normal, a parabuckling, and a ferrobuckling phase.
At zero temperature, quantum phase transitions between the three phases can be
induced by strain. This novel type of order opens a new path towards strain
induced quantum phases.

Under sufficient axial load, a column responds by a sudden deformation -
buckling. The deformation corresponds to a classical solution minimizing the
action. In the nanoscale, the electrostatic control of buckling was recently
realized, giving rise to buckling bits for nanomechanical computation Erbil
_et al._ (2020). Decreasing the column size even further, quantum effects
become dominant, allowing for the tunneling between degenerate buckling
states. Recently, this line of thought has initiated research in realizing
mechanical qubits Savel’ev  _et al._ (2006, 2007). Prominent designs are
proposed, e.g., based on carbon nanotubes Pistolesi  _et al._ (2021).
Realizing entanglement between various adjacent mechanical qubits has remained
an open question.

Metal organic framework materials are compounds built of metal ions or
clusters coordinated by organic ligands. After the first MOFs were realized in
the late 1990s Li  _et al._ (1999), more than 90,000 stable structures have
been synthesized and characterized to date Moosavi  _et al._ (2020). They are
intensively discussed in the context of gas sorption and storage, catalysis,
electronic devices, etc Yuan  _et al._ (2018); Jiao  _et al._ (2019); Furukawa
_et al._ (2013). Mechanical properties and flexibility of MOFs are summarized
in Ref. Schneemann  _et al._ (2014).

In the present paper it is shown how uniaxial pressure in MOFs can be used to
induce quantum buckling of ligand molecules. Interestingly, the buckling of
individual molecules is not independent, but interacts similarly to a
transverse field Ising model. We motivate the model on the example of
molecular buckling in the system MOF-5. Depending on applied uniaxial
pressure, the system undergoes two quantum phase transitions: first, from a
normal phase to a parabuckling phase; second: from a parabuckling phase to a
ferrobuckling phase. In the parabuckling phase, elementary excitations exhibit
a gap (mass) of $\Delta=4\sqrt{t(t-4J)}$. At the parabuckling-ferrobuckling
phase transition ($t=4J$) the gap closes. The parabuckling-ferrobuckling
quantum critical point might give rise to novel types of fluctuation-induced
order, such as strain-controlled superconductivity.

Figure 1: Buckling of BDC molecules in MOF-5. (a) shows the crystal structure
of MOF-5. Zn4O clusters are coordinated by BDC molecules, forming a periodic
3D cubic network. BDC Molecules can be buckled by uniaxial pressure. (b) Total
energy $V$ versus buckling of BDC molecules for various strain values. $V_{g}$
denotes the equilibrium total energy. Under strain, the total energy follows
the form of a double-well potential. (c) Barrier height $\delta V$ of the
double-well potential against uniaxial strain along the Cartesian $z$-axis.

MOF-5 describes a cubic framework topology Li  _et al._ (1999) with the sum
formula Zn4O(BDC)3. BDC is an abbreviation for 1,4-benzenedicarboxylate (Fig.
1(a)). The structure is open, with a spacing of 12.94 Å between the centers of
adjacent Zn4O clusters. MOF-5 is extremely soft with a bulk modulus of 15.37
GPa Yang  _et al._ (2010). Applying strain along one of the Cartesian axes,
the bond lengths within the Zn4O clusters and BDC molecules are squeezed up to
the point where the structure responds by a buckling of the molecules. To
approximate the effect, we monitor the change in total energy upon buckling,
for individual BDC molecules bound to Zn atoms and applied strain along the
Cartesian $z$-direction. Calculations based on density functional theory were
performed using VASP Kresse and Furthmüller (1996). The exchange-correlation
functional was approximated by the MetaGGA functional SCAN Sun  _et al._
(2015) in combination with the rVV10 Van-der-Waals corrections Sabatini  _et
al._ (2013); Peng  _et al._ (2016). The energy cutoff was set to 700 eV.
Zn-$d$-electrons were treated with a Hubbard-U correction of 2 eV Dudarev  _et
al._ (1998).

The buckling potential for various strain strength is shown in Fig. 1(b). By
applying strain, the total energy increases and the energy profile forms a
double-well potential. The potential has two degenerate minima, the left- and
right-buckled state. To lowest order, the potential can be described by a
fourth order polynomial,

$V(b)=V_{0}+a(b^{2}-b_{0}^{2})^{2}.$ (1)

$a$ is an overall scaling factor and $b_{0}$ the potential minimum or
classical solution. $V_{0}$ is the characteristic energy. It coincides with
the potential minimum for the zero-strain case ($b_{0}=0$). Note, that all
three parameters are strain dependent (see Table 1).

|  $V_{0}$ [eV] |  $a$ [eVÅ-4] |  $b_{0}$ [Å]
---|---|---|---
1% | -171.70 | 0.35 | 0.19
2% | -171.61 | 0.40 | 0.54
3% | -171.49 | 0.45 | 0.69
4% | -171.33 | 0.51 | 0.81
Table 1: Fitting parameters for the double-well potential. The table shows the
strain dependence of the characteristic energy $V_{0}$, the scaling factor
$a$, and the classical solution or potential minimum $b_{0}$.

In the following, we derive the quantum buckling Hamiltonian of strained
MOF-5. Assume a sufficiently high potential barrier. Then, the molecule only
buckles in one potential well, e.g., the right-buckled solution. The
corresponding energy is $E_{g}$ and can be approximated as follows. Around the
classical solution $b\approx b_{0}$, the potential is harmonic with
$V(b_{0}+\delta b)\approx V_{0}+4ab_{0}^{2}\,\delta b^{2}$, i.e., a quantum
harmonic oscillator. The ground state energy is given by

$E_{g}\approx V_{0}+E_{0}=V_{0}+\hbar\sqrt{\frac{2ab_{0}^{2}}{m}}.$ (2)

The wave function $\psi_{r}$ is normalized as $\left|\psi_{r}\right|^{2}=1$
over the right potential well. A similar construction is done for the wave
function $\psi_{l}$ describing the buckling in the left potential well. As the
barrier between both wells is finite, tunneling is allowed. We describe the
effective Hamiltonian incorporating the tunneling with tunneling strength $t$
by

$\mathrm{h}=\left(\mathrm{a}_{l}^{\dagger},\mathrm{a}_{r}^{\dagger}\right)\left(\begin{array}[]{cc}E_{g}&t\\\
t&E_{g}\end{array}\right)\left(\begin{array}[]{c}\mathrm{a}_{l}\\\
\mathrm{a}_{r}\end{array}\right).$ (3)

Here, $\mathrm{a}_{l}^{\dagger}$ and $\mathrm{a}_{r}^{\dagger}$
($\mathrm{a}_{l}$ and $\mathrm{a}_{r}$) are creation (annihilation operators)
for the left-buckled and right-buckled solution, respectively. The energy
levels correspond to the symmetric and antisymmetric solution of the system,

$E_{\pm}=E_{g}\pm
t,\quad\mathrm{\psi}_{\pm}=\frac{1}{\sqrt{2}}\left(\mathrm{a}_{l}\pm\mathrm{a}_{r}\right).$
(4)

Following Ref Savel’ev  _et al._ (2006), we can approximate $t$ as follows

$t=\frac{2\hbar}{\pi}\sqrt{\frac{2ab_{0}^{2}}{m}}\exp\left[-\frac{\pi}{2\hbar}\sqrt{\frac{m}{2ab_{0}^{2}}}(\delta
V-E_{0})\right].$ (5)

Figure 2: Exchange interaction of buckled linker molecules. (a) shows the
molecular arrangement of BDC linker molecules in MOF-5 in the
$\boldsymbol{a}-\boldsymbol{b}$ plane. Molecules form the pattern of a square
lattice. (b) Shows the energy difference between ferro- and antiferro-
buckling.

So far, we focussed on the buckling of a single molecule. However, MOF-5 is a
lattice periodic framework. As a result, the buckling of adjacent molecules
$i$ and $j$ is coupled, with the coupling strength $J_{ij}$. As shown in Fig.
2, for $J_{ij}>0$ ($J_{ij}<0$) the buckling in the same (opposite) buckling
state is energetically preferred. Merging this result with the single-molecule
buckling Hamiltonian of (3), we can write down the effective quantum buckling
Hamiltonian for the MOF in the following way (we neglect the constant energy
shift $E_{0}$),

$\mathrm{H}=-t\sum_{i}\mathrm{\sigma}^{1}_{i}-\sum_{ij}J_{ij}\mathrm{\sigma}^{3}_{i}\mathrm{\sigma}^{3}_{j}.$
(6)

Equation (6) is the mechanical buckling version of the well-studied transverse
field Ising model. The transverse field Ising model has been applied
intensively to study order-disorder ferroelectrics, simple ferromagnets,
simple Jahn-Teller systems and more Stinchcombe (1973); Suzuki  _et al._
(2012). In the following we summarize a few key results and their
interpretation for quantum buckling.

Figure 3: (a) Phase diagram of strained MOF-5. If the single-well ground state
energy is higher than the barrier height, no buckling occurs (normal phase).
For large enough barrier height and dominating tunneling $t>4J$ molecules are
in a superposition of left and right buckled states with no long-range order
(parabuckling). For dominating exchange energy $4J>t$, buckled molecules form
longrange order. For MOF-5, the expected transition temperature for 3% strain
is $\approx 10$ K. (b) Specific heat close to the quantum phase transition
between para- and ferrobuckling. In the ferrobuckling regime, the specific
heat contribution of the collective buckling state exhibits a discontinuity at
the transition temperature.

The zero temperature phase diagram of strained MOF-5 is shown in Fig. 3.
Depending on the strain strength, three phases are present. For sufficiently
weak strain, the double-well potential barrier is lower than the lowest lying
level of the single-well solution ($\delta V<E_{0}$). As a result, the quantum
state of the molecule does not experience the presence of two distinct minima
and the material remains in a normal state, with no buckling present. On the
opposite, for sufficiently large strain and dominating exchange
$\left|J_{ij}\right|\gg t$, the MOF will show ordered buckling. The simplest
order possible would be the ferrobuckling state ($J_{ij}>0$), with all
molecules occupying buckling states pointing in the same direction, i.e.,
$\left<\sigma_{i}^{z}\right>\approx 1$, $\left<\sigma_{i}^{x}\right>\approx
0$. Between the normal and the ferrobuckling phase is the parabuckling phase.
Here, the energy of the single-well solutions lies below the potential barrier
($\delta V>E_{0}$). Also, the tunneling term for each molecule dominates the
exchange between adjacent molecules ($t>4J$),
$\left<\sigma_{i}^{z}\right>\approx 0$, $\left<\sigma_{i}^{x}\right>\approx
1$.

In the mean-field approximation, we write (6) in terms of single-site
contributions Suzuki  _et al._ (2012),

$\mathrm{H}\approx-\sum_{i}\boldsymbol{\mathrm{h}}_{i}\cdot\boldsymbol{\mathrm{\sigma}}_{i},\qquad\boldsymbol{\mathrm{h}}_{i}=t\hat{\boldsymbol{x}}+\sum_{j}J_{ij}\left<\mathrm{\sigma}^{3}_{j}\right>\hat{\boldsymbol{z}}.$
(7)

Assuming weak variations in the buckling state
$\left<\sigma_{j}^{3}\right>\approx\left<\sigma^{3}\right>$ and nearest-
neighbor approximation, each single-site contribution
$\boldsymbol{\mathrm{h}}_{i}\cdot\boldsymbol{\mathrm{\sigma}}_{i}$ has
eigenvalues
$h_{\pm}=\pm\sqrt{t^{2}+\left(4J\left<\sigma^{3}\right>\right)^{2}}=\pm\left|\boldsymbol{h}\right|$.
As a result, we obtain for the buckling per site
$\left<\boldsymbol{\sigma}\right>=\frac{\boldsymbol{h}}{h}\tanh\left(\beta
h\right)$, with $\beta=(k_{b}T)^{-1}$. From the expectation value
$\left<\sigma^{3}\right>$ we can deduce the critical temperature given by

$\tanh\left(\beta_{c}t\right)=\frac{t}{4J}\quad\xrightarrow[]{J\gg t}\quad
k_{B}T_{c}\approx 4J.$ (8)

To estimate the exchange $J$ for MOF-5, we performed DFTtotal energy
calculations on a pair of molecules for the two configurations shown in Fig.
2. The exchange $J$ is then obtained from the energy difference of the ferro-
and antiferrobuckled configurations. The result was a fairly constant value of
$\approx 0.25$ meV in the low strain regime. Hence, according to equation (8),
the transition temperature is $\approx 10$ K. We note that we focus on the
regime of sufficiently small strain which justifies the nearest neighbor
approximation. A more detailed investigation of the strength of nearest
neighbor, second nearest neighbor, and four-$\sigma^{3}$ interactions would be
necessary, but is outside the scope of this paper. In particular, the latter
is expected to become significant for large strain. Such an extended four-spin
transverse field Ising model is in close connection to the 8-vertex model,
which has been intensively discussed Baxter (1971, 2016). Such a model also
captures the transition between ordered and glass states.

Besides the transition temperature, mean-field approximation allows to
estimate the ensemble average of the enegry, given by
$\left<E\right>=-h\tanh\left(\beta h\right).$ We numerically evaluate the
specific heat contribution due to collective buckling as
$C=\frac{\partial\left<E\right>}{\partial T}$, taking into account the
implicit temperature dependence of $\left<\sigma^{3}\right>$. We plot $C/T$
per mol in Fig 3 (b) for a strain close to the quantum critical point between
para- and ferrobuckling. In the parabuckling regime (dotted blue line, 1.9 %
strain) the specific heat is a smooth function in temperature. In contrast, in
the ferrobuckling regime (solid orange line, 2 % strain), a phase transition
at a temparature of $\approx 6.15$ K leads to a discontinuity in the specific
heat. Hence, a strain-dependent measurement of the specific heat at low
temperatures could provide experimental evidence for the quantumbuckling
phases.

Collective excitations of the ordered buckling states emerge similarly to
magnons in magnetically ordered systems. The buckleon excitations of the
parabuckling phase can be estimated by evaluating the Heisenberg equation of
motion to calculate $\mathrm{\ddot{\sigma}}^{z}_{i}$. Using the random phase
approximation
$\mathrm{\sigma}_{i}^{x}\mathrm{\sigma}_{j}^{z}\approx\left<\mathrm{\sigma}_{i}^{x}\right>\mathrm{\sigma}_{j}^{z}+\mathrm{\sigma}_{i}^{x}\left<\mathrm{\sigma}_{j}^{z}\right>$
Suzuki  _et al._ (2012), we obtain for the effective square lattice of MOF-5

$\mathrm{\ddot{\sigma}}_{i}^{z}+\omega_{\boldsymbol{q}}^{2}\mathrm{\sigma}_{i}^{z}=0,\quad\omega_{\boldsymbol{q}}=\pm\frac{2t}{\hbar}\sqrt{1-\frac{2J}{t}\left[\cos{q_{x}}+\cos{q_{y}}\right]}.$
(9)

At $\boldsymbol{q}=0$, the spectrum has a gap $\Delta=4\sqrt{t(t-4J)}$. As a
result, interactions mediated by buckleons in the parabuckling phase are
exponentially decaying and short ranged. At the quantum phase transition
between ferro and parabuckling phase, $t\approx 4J$, the gap closes, leading
to significantly enhanced interactions due to strong fluctuations.

For example, in quantum ferroelectrics, fluctuations at the quantum critical
point between the para and ferroelectric phase enhance superconductivity
Rowley  _et al._ (2014); Edge  _et al._ (2015). In contrast to the
ferroelectric counterparts, the quantum parabuckling-ferrobuckling phase
transition can be induced straightforwardly by uniaxial pressure (the
calculated bulk modulus of MOF-5 is 15.37 GPa Yang  _et al._ (2010), whereas
the bulk modulus of SrTiO3 is 172.1 GPa Beattie and Samara (1971)). Modifying
the formalism of Edge et al. Edge  _et al._ (2015) and the strong-coupling
theory of McMillan McMillan (1968) to the present case of electron-buckleon
mediated coupling, the superconducting coupling strength goes as

$\displaystyle\lambda$
$\displaystyle=\int_{0}^{\infty}\alpha^{2}(\omega)F(\omega)\frac{d\omega}{\omega}$
(10)
$\displaystyle\approx\alpha^{2}\int\mathrm{d}\boldsymbol{q}\,\frac{1}{\frac{2t}{\hbar}\sqrt{1-\frac{2J}{t}\left[\cos{q_{x}}+\cos{q_{y}}\right]}}.$
(11)

Here, we assumed a constant coupling constant
$\alpha^{2}(\omega)\approx\alpha^{2}$ and a spectral density
$F(\omega)=\int\mathrm{d}\boldsymbol{q}\,\delta(\omega-\omega_{\boldsymbol{q}})$.
Hence, at the ferrobuckling transition where $\frac{4J}{t}\approx 1$ and
$\omega_{\boldsymbol{q}=0}\rightarrow 0$, the superconducting interaction
strength is significantly enhanced. We note that MOF-5 is an insulator with a
gap of $\approx 3.4$ eV Yang  _et al._ (2010). Hence, to observe
superconductivity, the material would have to be doped accordingly. Currently,
only very few examples of superconductivity in MOFs are known Takenaka  _et
al._ (2021); Huang  _et al._ (2018). These examples are most-likely based on
strong correlations, similar to high-$T_{c}$ superconductors. We believe, the
emergence of order due to quantum critical buckling is a different and more
general concept in MOFs. This concept goes beyond MOF-5 and superconductivity.

In summary, we showed that applying uniaxial pressure to MOFs can lead to
buckling of the organic linker molecules. At low temperatures the buckling of
individual molecules is of quantum nature, where the molecule is in a
superposition of left- and right-buckled states. Also, the buckling of
adjacent molecules is not independent but weakly coupled. As a result, the
material can undergo a phase transition into a collective para- and
ferrobuckling state. MOFs are soft. Therefore, the tuning of the quantum
buckling phases by uniaxial pressure can be achieved straightforwardly. The
emergence of a phase transition at low temperatures should be seen in the
specific heat of the material. For MOF-5 and a strain of $\approx 2-5~{}\%$,
we expect the phase transition into a ferrobuckling phase to take place at
$\approx 10$ K. By slowly decreasing the uniaxial pressure to $<2~{}\%$ a
quantum phase transition to a parabuckling phase is expected. The quantum
critical point between the two phases might be a prominent experimental
platform to investigate novel types of fluctuation induced order.

Acknowledgment. We are grateful to Marc Suñé Simon, Jérémy Vachier, Anthony
Bonfils, Prabal Negi, Bart Olsthoorn, Vladimir Juričić, and Alexander V.
Balatsky for inspiring discussions. We acknowledge support from research
funding granted to A. V. Balatsky, i.e., VILLUM FONDEN via the Centre of
Excellence for Dirac Materials (Grant No. 11744), the European Research
Council under the European Union Seventh Framework ERS-2018-SYG 810451 HERO,
the Knut and Alice Wallenberg Foundation KAW 2018.0104. Computational
resources were provided by the Swedish National Infrastructure for Computing
(SNIC) via the High Performance Computing Centre North (HPC2N) and the Uppsala
Multidisciplinary Centre for Advanced Computational Science (UPPMAX).

## References

  * Erbil  _et al._ (2020) S. O. Erbil, U. Hatipoglu, C. Yanik, M. Ghavami, A. B. Ari, M. Yuksel, and M. S. Hanay, Phys. Rev. Lett. 124, 046101 (2020).
  * Savel’ev  _et al._ (2006) S. Savel’ev, X. Hu, and F. Nori, New Journal of Physics 8, 105 (2006).
  * Savel’ev  _et al._ (2007) S. Savel’ev, A. Rakhmanov, X. Hu, A. Kasumov, and F. Nori, Physical Review B 75, 165417 (2007).
  * Pistolesi  _et al._ (2021) F. Pistolesi, A. N. Cleland, and A. Bachtold, Phys. Rev. X 11, 031027 (2021).
  * Li  _et al._ (1999) H. Li, M. Eddaoudi, M. O’Keeffe, and O. M. Yaghi, nature 402, 276 (1999).
  * Moosavi  _et al._ (2020) S. M. Moosavi, A. Nandy, K. M. Jablonka, D. Ongari, J. P. Janet, P. G. Boyd, Y. Lee, B. Smit, and H. J. Kulik, Nature communications 11, 1 (2020).
  * Yuan  _et al._ (2018) S. Yuan, L. Feng, K. Wang, J. Pang, M. Bosch, C. Lollar, Y. Sun, J. Qin, X. Yang, P. Zhang, Q. Wang, L. Zou, Y. Zhang, L. Zhang, Y. Fang, J. Li, and H.-C. Zhou, Advanced Materials 30, 1704303 (2018), https://onlinelibrary.wiley.com/doi/pdf/10.1002/adma.201704303 .
  * Jiao  _et al._ (2019) L. Jiao, J. Y. R. Seow, W. S. Skinner, Z. U. Wang, and H.-L. Jiang, Materials Today 27, 43 (2019).
  * Furukawa  _et al._ (2013) H. Furukawa, K. E. Cordova, M. O’Keeffe, and O. M. Yaghi, Science 341, 1230444 (2013).
  * Schneemann  _et al._ (2014) A. Schneemann, V. Bon, I. Schwedler, I. Senkovska, S. Kaskel, and R. A. Fischer, Chemical Society Reviews 43, 6062 (2014).
  * Yang  _et al._ (2010) L.-M. Yang, P. Vajeeston, P. Ravindran, H. Fjellvag, and M. Tilset, Inorganic chemistry 49, 10283 (2010).
  * Kresse and Furthmüller (1996) G. Kresse and J. Furthmüller, Physical Review B 54, 11169 (1996).
  * Sun  _et al._ (2015) J. Sun, A. Ruzsinszky, and J. P. Perdew, Phys. Rev. Lett. 115, 036402 (2015).
  * Sabatini  _et al._ (2013) R. Sabatini, T. Gorni, and S. de Gironcoli, Phys. Rev. B 87, 041108 (2013).
  * Peng  _et al._ (2016) H. Peng, Z.-H. Yang, J. P. Perdew, and J. Sun, Phys. Rev. X 6, 041005 (2016).
  * Dudarev  _et al._ (1998) S. L. Dudarev, G. A. Botton, S. Y. Savrasov, C. J. Humphreys, and A. P. Sutton, Phys. Rev. B 57, 1505 (1998).
  * Stinchcombe (1973) R. Stinchcombe, Journal of Physics C: Solid State Physics 6, 2459 (1973).
  * Suzuki  _et al._ (2012) S. Suzuki, J.-i. Inoue, and B. K. Chakrabarti,  _Quantum Ising phases and transitions in transverse Ising models_ , Vol. 862 (Springer, 2012).
  * Baxter (1971) R. J. Baxter, Physical Review Letters 26, 832 (1971).
  * Baxter (2016) R. J. Baxter,  _Exactly solved models in statistical mechanics_ (Elsevier, 2016).
  * Rowley  _et al._ (2014) S. Rowley, L. Spalek, R. Smith, M. Dean, M. Itoh, J. Scott, G. Lonzarich, and S. Saxena, Nature Physics 10, 367 (2014).
  * Edge  _et al._ (2015) J. M. Edge, Y. Kedem, U. Aschauer, N. A. Spaldin, and A. V. Balatsky, Phys. Rev. Lett. 115, 247002 (2015).
  * Beattie and Samara (1971) A. Beattie and G. Samara, Journal of Applied Physics 42, 2376 (1971).
  * McMillan (1968) W. McMillan, Physical Review 167, 331 (1968).
  * Takenaka  _et al._ (2021) T. Takenaka, K. Ishihara, M. Roppongi, Y. Miao, Y. Mizukami, T. Makita, J. Tsurumi, S. Watanabe, J. Takeya, M. Yamashita, _et al._ , Science Advances 7, eabf3996 (2021).
  * Huang  _et al._ (2018) X. Huang, S. Zhang, L. Liu, L. Yu, G. Chen, W. Xu, and D. Zhu, Angewandte Chemie International Edition 57, 146 (2018).


Comment on ‘Fickian Non-Gaussian Diffusion in Glass-Forming Liquids’.

A recent Letter Rusciano2022  examined the statistics of individual particles
displacements $\Delta x(t)$ over time $t$ in two-dimensional glass-formers and
concluded that the corresponding probability distribution $G_{s}(\Delta x,t)$,
called the van-Hove distribution, is non-Gaussian in a time regime where the
mean-squared displacement (MSD) is Fickian, $\langle\Delta
x^{2}(t)\rangle\propto D_{s}t$, where $D_{s}$ is the self-diffusion constant.
If this analysis were correct, glass-formers would be ‘Fickian non-Gaussian’
materials FNG ; miotto2021 .

Here, we clarify that the multiple lengthscales and timescales reported in
Rusciano2022  have either been characterized before, or are not well-defined.
This leads us to dispute the conclusions that glass-formers display Fickian
non-Gaussian behaviour and that this analogy fruitfully addresses the central
questions regarding the nature of dynamic heterogeneity in these systems.

Let us first recall that the main features of self-diffusion in supercooled
liquids are explained by invoking two characteristic timescales Berthier2005 ;
Chaudhuri2007a ; Chaudhuri2007b ; hedges2007 . The self-diffusion coefficient
$D_{s}$ controls the first one, $\tau_{D}=\sigma^{2}/D_{s}$, where $\sigma$ is
the particle size. The second one is the structural relaxation time
$\tau_{\alpha}$ determined from usual time correlations, such as the self-
intermediate scattering function. The adimensional ratio
$X=\tau_{\alpha}/\tau_{D}$ plays a special role. It controls both the amount
of decoupling $X(T)\sim D_{s}\tau_{\alpha}$ (akin to violations of the Stokes-
Einstein relation Berthier2004 )), and the Fickian lengthscale
$\ell_{F}\propto\sqrt{X}$ Berthier2005 . These known results Berthier2005 ;
Berthier2004 ; Szamel2006 ; Chaudhuri2007a ; Chaudhuri2007b ; hedges2007 ;
jorg2014  paint a picture that is inconsistent with several  conclusions
reported in Rusciano2022 ; miotto2021  as we now show.

Let us start with the van Hove distribution. It was found in Szamel2006  that
$G_{s}(\Delta x,t)$ approaches a Gaussian distribution only for times much
longer than $\tau_{D}$, a result rediscovered in Rusciano2022  with equivalent
tools. However, the non-Gaussian parameter $\alpha_{2}(t)$ used in
Rusciano2022  to reveal Gaussianity decays  as a power law at large times.
Hence, the gradual emergence of Gaussian behaviour from $\alpha_{2}(t)$ is a
scale-free process and there is no characteristic timescale after which self-
diffusion is Gaussian, although of course empirically $\alpha_{2}(t)$ will be
small when $\langle\Delta x^{2}\rangle\gg\ell_{F}^{2}$. It was found
numerically Berthier2004  and explained theoretically Berthier2005  that it is
the lengthscale $\ell_{F}$ that controls  the crossover in the wavevector
dependence of the self-intermediate scattering function, a result  ignored in
Rusciano2022 .

For times $t<\tau_{\alpha}$, $G_{s}(\Delta x,t)$ is characterized by a
Gaussian core at small $\Delta x$ and a nearly exponential tail
$G_{s}\propto\exp(-|\Delta x|/\lambda)$ at large $\Delta x$ Chaudhuri2007a .
Refs. Rusciano2022 ; miotto2021  discuss the existence and possible
universality of a power law description of the time evolution of the
exponential tail, $\lambda(t)\sim t^{\alpha}$. As noticed in Chaudhuri2007a ;
Chaudhuri2007b ; barkai2020 , the exponential tail is generically explained by
a large deviation argument, but asymptotic convergence is so slow that the
actual value of $\lambda$ depends on the fitted range  (see Chaudhuri2007b
for an explicit test and barkai2020  for analytic results suggesting that
$\alpha=0$), which may explain reported discrepancies Rusciano2022 ;
miotto2021 . More fundamentally $\lambda(t)$ is  difficult to measure, and
$\alpha$ is not a novel characteristic exponent.

Linear behaviour of the MSD is visually detected Rusciano2022  in log-log
representations after a time $\tau_{D}$ which  grows more slowly than
$\tau_{\alpha}$ at low temperature, but the approach to linearity is algebraic
jeppe . This power law approach to Fickian behaviour is again scale free and
no characteristic timescale controls the emergence of Fickian behaviour in
$\langle\Delta x^{2}(t)\rangle$; in particular $\tau_{D}$ does not play this
role.

Even though glass-formers may appear empirically close to Fickian non-Gaussian
materials, there is no characteristic timescales or lengthscales controlling
the approach to either Fickian and Gaussian dynamics, and the existence of a
Fickian non-Gaussian regime cannot be decided. Instead, the salient features
of self-diffusion, including algebraic approach to Fickian and Gaussian
behaviours as well as nearly exponential van Hove distributions, are
analytically captured by (effective) non-interacting continuous time random
walk models Berthier2005 ; Chaudhuri2007a ; Chaudhuri2007b ; hedges2007 ;
jorg2014  based on the only two important and well-defined characteristic
timescales $\tau_{D}$ and $\tau_{\alpha}$. The multiple time and length scales
determined empirically in Rusciano2022 ; miotto2021  are either related to
those, or conceptually ill-defined.

The complexity of glass-formers is that the timescales $\tau_{D}$ and
$\tau_{\alpha}$ emerge from many-body interactions (disorder is self-induced)
and have non-trivial temperature dependencies which are not fully understood,
but from which the very rich statistics of single particle displacement
naturally follows.  The behaviour of supercooled liquids is very different
from several Fickian non Gaussian materials, which are described by
interesting, but quite different, models prx2017 .

We end by noting that the use of two-dimensional simulations to study the
statistics of particle displacements in glass-formers is profoundly influenced
at all timescales by Mermin-Wagner fluctuations Flenner2015 ; Flenner2019 ,
which presumably adds to the profusion of timescales reported in Rusciano2022
.

L. Berthier1,2, E. Flenner3, G. Szamel3

1Laboratoire Charles Coulomb (L2C), Université de Montpellier, CNRS, 34095
Montpellier, France

2Yusuf Hamied Department of Chemistry, University of Cambridge, Lensfield
Road, Cambridge CB2 1EW, United Kingdom

3Department of Chemistry, Colorado State University, Fort Collins Colorado
80523, USA

## References

  * (1) F. Rusciano, R. Pastore, and F. Greco, Phys. Rev. Lett. 128, 168001 (2022).
  * (2) B. Wang, S. M. Anthony, S. C. Bae, and S. Granick, Proc. Natl. Acad. Sci. USA 106, 15160 (2009).
  * (3) J. M. Miotto, S. Pigolotti, A. V. Chechkin, and S. Roldan-Vargas, Phys. Rev. X 11, 031002 (2021).
  * (4) L. Berthier, D. Chandler, and J.P. Garrahan, Europhys. Lett. 69, 320 (2005).
  * (5) P. Chaudhuri, L. Berthier, and W. Kob, Phys. Rev. Lett. 99, 060604 (2007).
  * (6) P. Chaudhuri, Y. Gao, L. Berthier, M. Kilfoil, and W. Kob, J. Phys.: Condens. Matter 20, 244126 (2008).
  * (7) L. O. Hedges, L. Maibaum, D. Chandler, and J. P. Garrahan, J. Chem. Phys. 127, 211101 (2007).
  * (8) L. Berthier, Phys. Rev. E 69, 020201(R) (2004).
  * (9) G. Szamel and E. Flenner, Phys. Rev. E 73, 011504 (2006).
  * (10) J. Helfferich, F. Ziebert, S. Frey, H. Meyer, J. Farago, A. Blumen, and J. Baschnagel, Phys. Rev. E 89, 042604 (2014).
  * (11) E. Barkai and S. Burov, Phys. Rev. Lett. 124, 060603 (2020).
  * (12) T. B. Schroder and J. C. Dyre, J. Chem. Phys. 152, 141101 (2020).
  * (13) E. Flenner and G. Szamel, Nature Comm. 6, 7392 (2015).
  * (14) E. Flenner and G. Szamel, Proc. Natl. Acad. Sci. U.S.A. 116, 2015 (2019).
  * (15) A. V. Chechkin, F. Seno, R. Metzler, and I. M. Sokolov, Phys. Rev. X 7, 021002 (2017).


give five more, for a total of nine. That the coupler curve contains a given
point in ${\mathbb{R}}^{2}$ is a single, simple condition on the space of
four-bar mechanisms. Thus we expect there are only finitely many mechanisms
whose coupler curve contains nine given general points. Alt [2] recognized
this, and his nine-point synthesis problem asks for the mechanisms whose
coupler curve contains a given nine points.

Identifying ${\mathbb{R}}^{2}$ with ${\mathbb{C}}$, we represent the bars as
complex numbers. Complexifying gives a useful formulation of Alt’s problem in
isotropic coordinates—this is described in [60]. Solutions to these equations
for nine general points were computed using homotopy continuation in [60],
finding 8652 solutions.

In [36], this computation was repeated and a soft certificate was computed to
certify that the 8652 computed solutions. While 8652 is almost surely the
number of solutions, these computations only show that it is a lower bound,
and a proof of the number 8652 remains elusive. Further evidence for the
number 8652 was found in [34], but that result also only implies that 8652 is
a lower bound.

In this formulation, solutions come in pairs due to relabeling—swapping labels
of the bars incident to the base and to the apex of the triangle results in
another solution and gives the same four-bar mechanism. Classically Roberts
[68] and Chebyshev [15] (see [91] for a discussion) show that there are three
mechanisms—called Robert’s Cognates—with the same coupler curve. Consequently,
the Galois group of this formulation of Alt’s problem is imprimitive as it
preserves the six solutions which give the same coupler curve. We also see
that, assuming the number 8652 is correct, that there are 4326 four-bar
mechanisms whose coupler curve contains nine given points, and 1442 distinct
coupler curves.

Since label swapping may be done independently on each cognate, the Galois
group $G$ of the six solutions with given coupler curve is a subgroup of
${\mathbb{Z}}/2{\mathbb{Z}}\wr S_{3}=({\mathbb{Z}}/2{\mathbb{Z}})^{3}\rtimes
S_{3}$—this assumes that the cognates have symmetry $S_{3}$. Consequently, the
Galois group of this formulation is a subgroup of $G\wr S_{1432}$. To the best
of our knowledge, the Galois group of this problem has not been determined.

## References

  * [1] E. L. Allgower and K. Georg, _Introduction to numerical continuation methods_ , Classics in Applied Mathematics, vol. 45, SIAM, 2003.
  * [2] H. Alt, _Über die erzeugung gegebener ebener kurven mit hilfe des gelenkviereckes_ , Zeitschrift für Angewandte Mathematik und Mechanik 3 (1923), no. 1, 13–19.
  * [3] C. Améndola, J. Lindberg, and J. I. Rodriguez, _Solving parameterized polynomial systems with decomposable projections_ , 2021, arXiv:1612.08807.
  * [4] B. Banwait, F. Fité, and D. Loughran, _Del Pezzo surfaces over finite fields and their Frobenius traces_ , Math. Proc. Cambridge Philos. Soc. 167 (2019), no. 1, 35–60.
  * [5] D.J. Bates, J.D. Hauenstein, A.J. Sommese, and C.W. Wampler, _Bertini: Software for numerical algebraic geometry_ , Available at http://www.nd.edu/~sommese/bertini.
  * [6] C. Beltrán and A. Leykin, _Certified numerical homotopy tracking_ , Exp. Math. 21 (2012), no. 1, 69–83.
  * [7] by same author, _Robust certified numerical homotopy tracking_ , Found. Comput. Math. 13 (2013), no. 2, 253–295.
  * [8] D. N. Bernstein, _The number of roots of a system of equations_ , Funct. Anal. Appl. 9 (1975), 183–185.
  * [9] P. Breiding, K. Rose, and S. Timme, _Certifying zeros of polynomial systems using interval arithmetic_ , 2020, arXiv:2011.05000.
  * [10] P. Breiding and S. Timme, _HomotopyContinuation.jl: A Package for Homotopy Continuation in Julia_ , International Congress on Mathematical Software, Springer, 2018, pp. 458–465.
  * [11] C.J. Brooks, A. Martín del Campo, and F. Sottile, _Galois groups of Schubert problems of lines are at least alternating_ , Trans. Amer. Math. Soc. 367 (2015), no. 6, 4183–4206.
  * [12] T. Brysiewicz, J. I. Rodriguez, F. Sottile, and T. Yahl, _Solving decomposable sparse systems_ , Numerical Algorithms (2020).
  * [13] by same author, _Decomposable sparse polynomial systems_ , Journal of Software for Algebra and Geometry (2021).
  * [14] A. Cayley, _On the triple tangent planes of surfaces of third order_ , The Cambridge and Dublin Mathematical Journal 4 (1849), 118–138.
  * [15] P. Chebyshev, _Les plus simple systémes de tiges articulées_ , Oeuvres de P.L. Tchebychef, Tome II (A. Markoff and N. Sonin, eds.), l’académie impériale des sciences, St. Petersburg, 1907, pp. 271–281.
  * [16] O. Debarre and L. Manivel, _Sur la variété des espaces linéaires contenus dans une intersection complète_ , Mathematische Annalen 312 (1998), 549–574.
  * [17] W. Decker, G.-M. Greuel, G. Pfister, and H. Schönemann, _Singular 4-0-3 — A computer algebra system for polynomial computations_, http://www.singular.uni-kl.de, 2016.
  * [18] H. D’Souza, _On the monodromy group of everywhere tangent lines to the octic surface in ${\bf P}^{3}$_, Proc. Amer. Math. Soc. 104 (1988), no. 4, 1010–1013.
  * [19] T. Duff, V. Korotynskiy, T. Pajdla, and M. Regan, _Galois/monodromy groups for decomposing minimal problems in 3D reconstruction_ , 2021, arXiv:2105.04460.
  * [20] Timothy Duff, Kathlen Kohn, Anton Leykin, and Tomas Pajdla, _Plmp – point-line minimal problems in complete multi-view visibility_ , Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019\.
  * [21] T. Ekedahl, _An effective version of Hilbert’s irreducibility theorem_ , Séminaire de Théorie des Nombres, Paris 1988–1989, Progr. Math., vol. 91, Birkhäuser, Boston, MA, 1990, pp. 241–249.
  * [22] A. Eremenko and A. Gabrielov, _Rational functions with real critical points and the B. and M. Shapiro conjecture in real enumerative geometry_ , Ann. of Math. (2) 155 (2002), no. 1, 105–129.
  * [23] A. Eremenko, A. Gabrielov, M. Shapiro, and A. Vainshtein, _Rational functions and real Schubert calculus_ , Proc. Amer. Math. Soc. 134 (2006), no. 4, 949–957 (electronic).
  * [24] A. Esterov, _Galois theory for general systems of polynomial equations_ , Compos. Math. 155 (2019), no. 2, 229–245.
  * [25] A. Esterov and L. Lang, _Sparse polynomial equations and other enumerative problems whose Galois groups are wreath products_ , 2019, arXiv:1812.07912.
  * [26] G. Ewald, _Combinatorial convexity and algebraic geometry_ , Graduate Texts in Mathematics, vol. 168, Springer-Verlag, New York, 1996.
  * [27] J.-C. Faugère, _A new efficient algorithm for computing Gröbner bases without reduction to zero $(F_{5})$_, Proceedings of the 2002 International Symposium on Symbolic and Algebraic Computation, ACM, New York, 2002, pp. 75–83.
  * [28] Wm. Fulton, _Young tableaux_ , Cambridge University Press, Cambridge, 1997.
  * [29] The GAP Group, _GAP – Groups, Algorithms, and Programming, Version 4.11.1_ , 2021.
  * [30] D.R. Grayson and M.E. Stillman, _Macaulay2, a software system for research in algebraic geometry_ , Available at http://www.math.uiuc.edu/Macaulay2/.
  * [31] J. Harris, _Galois groups of enumerative problems_ , Duke Math. Journal 46 (1979), no. 4, 685–724.
  * [32] S. Hashimoto and B. Kadets, _38406501359372282063949 and all that: Monodromy of Fano problems_ , International Mathematics Research Notices (2020), rnaa275, arxiv:2002.04580.
  * [33] A. Hatcher, _Algebraic topology_ , Cambridge University Press, Cambridge, 2002\.
  * [34] J. Hauenstein and M. Helmer, _Probabilistic saturations and Alt’s problem_ , 2019, arxiv:1908.06020.
  * [35] J.D. Hauenstein, J.I. Rodriguez, and F. Sottile, _Numerical Computation of Galois Groups_ , Found. Comput. Math. 18 (2018), no. 4, 867–890.
  * [36] J.D. Hauenstein and F. Sottile, _Algorithm 921: alphaCertified: Certifying solutions to polynomial systems_ , ACM Trans. Math. Softw. 38 (2012), no. 4, 28.
  * [37] N. Hein, C.J. Hillar, A. Martín del Campo, F. Sottile, and Z. Teitler, _The monotone secant conjecture in the real Schubert calculus_ , Exp. Math. 24 (2015), no. 3, 261–269.
  * [38] N. Hein, F. Sottile, and I. Zelenko, _A congruence modulo four in real Schubert calculus_ , J. Reine Angew. Math. 714 (2016), 151–174.
  * [39] C. Hermite, _Sur les fonctions algébriques_ , CR Acad. Sci.(Paris) 32 (1851), 458–461.
  * [40] D. Hilbert, _Mathematical problems_ , Bull. Amer. Math. Soc. 8 (1902), no. 10, 437–479.
  * [41] B. Huber, F. Sottile, and B. Sturmfels, _Numerical Schubert calculus_ , Journal of Symbolic Computation 26 (1998), no. 6, 767–788.
  * [42] B. Huber and B. Sturmfels, _A polyhedral method for solving sparse polynomial systems_ , Math. Comp. 64 (1995), no. 212, 1541–1555.
  * [43] C. Jordan, _Traité des Substitutions et des Équations algébriques_ , Gauthier-Villars, Paris, 1870.
  * [44] S.L. Kleiman, _The transversality of a general translate_ , Compositio Math. 28 (1974), 287–297.
  * [45] S.L. Kleiman and D. Laksov, _Schubert calculus_ , Amer. Math. Monthly 79 (1972), 1061–1082.
  * [46] R. Krawczyk, _Newton-Algorithmen zur Bestimmung von Nullstellen mit Fehlerschranken_ , Computing (Arch. Elektron. Rechnen) 4 (1969), 187–201.
  * [47] A. Kukelova, _Algebraic methods in computer vision_ , Ph.D. thesis, Czech Technical University in Prague, 2013.
  * [48] A.G. Kušnirenko, _Newton polyhedra and Bezout’s theorem_ , Funkcional. Anal. i Priložen. 10 (1976), no. 3, 82–83.
  * [49] by same author, _Polyèdres de Newton et nombres de Milnor_ , Invent. Math. 32 (1976), no. 1, 1–31.
  * [50] S. Lang, _Algebra_ , Graduate Texts in Mathematics, vol. 211, Springer-Verlag, New York, 2002.
  * [51] J. Levinson and K. Purbhoo, _A topological proof of the Shapiro-Shapiro conjecture_ , 2019, ArXiv.org/1907.11924.
  * [52] A. Leykin, _Numerical Algebraic Geometry_ , The Journal of Software for Algebra and Geometry 3 (2011), 5–10, Available at http://j-sag.org/volume3.html.
  * [53] A. Leykin, J.I. Rodriguez, and F. Sottile, _Trace test_ , Arnold Mathematical Journal 4 (2018), no. 1, 113–125.
  * [54] A. Leykin and F. Sottile, _Galois groups of Schubert problems via homotopy computation_ , Math. Comp. 78 (2009), no. 267, 1749–1765.
  * [55] T.Y. Li, T. Sauer, and J.A. Yorke, _The cheater’s homotopy: an efficient procedure for solving systems of polynomial equations_ , SIAM Journal on Numerical Analysis 26 (1989), no. 5, 1241–1251.
  * [56] A. Martín del Campo and F. Sottile, _Experimentation in the Schubert calculus_ , Schubert Calculus, Osaka 2012 (H. Naruse, T. Ikeda, M. Masuda, and T. Tanisaki, eds.), Advanced Studies in Pure Mathematics, vol. 71, Mathematical Society of Japan, 2016, pp. 295–336.
  * [57] A. Martín del Campo, F. Sottile, and R. Williams, _Classification of Schubert Galois groups in ${G}r(4,9)$_, arXiv.org/1902.06809, 2019.
  * [58] A. Morgan, _Solving polynomial systems using continuation for engineering and scientific problems_ , Classics in Applied Mathematics, vol. 57, SIAM, 2009\.
  * [59] A.P. Morgan and A.J. Sommese, _Coefficient-parameter polynomial continuation_ , Appl. Math. Comput. 29 (1989), no. 2, part II, 123–160.
  * [60] A.P. Morgan, A.J. Sommese, and C.W. Wampler, _Complete solution of the nine-point path synthesis problem for four-bar linkages_ , ASME J. Mech. Des. 114 (1992), no. 1, 153–159.
  * [61] E. Mukhin and V. Tarasov, _Lower bounds for numbers of real solutions in problems of Schubert calculus_ , Acta Math. 217 (2016), no. 1, 177–193.
  * [62] E. Mukhin, V. Tarasov, and A. Varchenko, _The B. and M. Shapiro conjecture in real algebraic geometry and the Bethe ansatz_ , Ann. of Math. (2) 170 (2009), no. 2, 863–881.
  * [63] by same author, _Schubert calculus and representations of the general linear group_ , J. Amer. Math. Soc. 22 (2009), no. 4, 909–940.
  * [64] J.R. Munkres, _Topology: a first course_ , Prentice-Hall, Inc., Englewood Cliffs, N.J., 1975.
  * [65] D. Nistér, _An efficient solution to the five-point relative pose problem_ , IEEE Transactions on Pattern Analysis and Machine Intelligence 26 (2004), no. 6, 756–770.
  * [66] G.P. Pirola and E. Schlesinger, _Monodromy of projective curves_ , J. Algebraic Geom. 14 (2005), no. 4, 623–642.
  * [67] K. Purbhoo, _Reality and transversality for Schubert calculus in ${\rm OG}(n,2n+1)$_, Math. Res. Lett. 17 (2010), no. 6, 1041–1046.
  * [68] S. Roberts, _On three-bar motion in plane space_ , Proceedings London Mathematical Society (1875), 14–23.
  * [69] J. Ruffo, Y. Sivan, E. Soprunova, and F. Sottile, _Experimentation and conjectures in the real Schubert calculus for flag manifolds_ , Experiment. Math. 15 (2006), no. 2, 199–221.
  * [70] G. Salmon, _On the triple tangent planes to a surface of third order_ , The Cambridge and Dublin Mathematical Journal 4 (1849), 252–260.
  * [71] L. Schläfli, _An attempt to determine the twenty-seven lines upon a surface of the third order and to divide such surfaces into species in reference to the reality of the lines upon the surface_ , Quarterly Journal of Math. 2 (1858), 55–65, 110–121.
  * [72] H. Schubert, _Kalkül der abzählenden Geometrie_ , Springer-Verlag, Berlin-New York, 1979, Reprint of the 1879 original, With an introduction by Steven L. Kleiman.
  * [73] S. Smale, _Newton’s method estimates from data at one point_ , The merging of disciplines: new directions in pure, applied, and computational mathematics, Springer, New York, 1986, pp. 185–196.
  * [74] A.J. Sommese, J. Verschelde, and C.W. Wampler, _Numerical decomposition of the solution sets of polynomial systems into irreducible components_ , SIAM J. Numer. Anal. 38 (2001), no. 6, 2022–2046.
  * [75] by same author, _Using monodromy to decompose solution sets of polynomial systems into irreducible components_ , Applications of algebraic geometry to coding theory, physics and computation (Eilat, 2001), NATO Sci. Ser. II Math. Phys. Chem., vol. 36, Kluwer Acad. Publ., Dordrecht, 2001, pp. 297–315.
  * [76] by same author, _Symmetric functions applied to decomposing solution sets of polynomial systems_ , SIAM J. Numer. Anal. 40 (2002), no. 6, 2026–2046.
  * [77] by same author, _Introduction to numerical algebraic geometry_ , Solving polynomial equations, Algorithms Comput. Math., vol. 14, Springer, Berlin, 2005, pp. 301–335.
  * [78] A.J. Sommese and C.W. Wampler, _Numerical algebraic geometry_ , The mathematics of numerical analysis (Park City, UT, 1995), Lectures in Appl. Math., vol. 32, Amer. Math. Soc., Providence, RI, 1996, pp. 749–763.
  * [79] by same author, _The numerical solution of systems of polynomials arising in engineering and science_ , World Scientific Publishing Co. Pte. Ltd., Hackensack, NJ, 2005.
  * [80] F. Sottile, _Real Schubert calculus: polynomial systems and a conjecture of Shapiro and Shapiro_ , Experiment. Math. 9 (2000), no. 2, 161–182.
  * [81] by same author, _Real solutions to equations from geometry_ , University Lecture Series, vol. 57, American Mathematical Society, Providence, RI, 2011.
  * [82] F. Sottile and J. White, _Double transitivity of Galois groups in Schubert calculus of Grassmannians_ , Algebr. Geom. 2 (2015), no. 4, 422–445.
  * [83] F. Sottile, J. White, and R. Williams, _Galois groups in simple Schubert problems are at least alternating_ , in preparation.
  * [84] F. Sottile, R. Williams, and L. Ying, _Galois groups of composed Schubert problems_ , Facets of Algebraic Geometry: A Collection in Honor of William Fulton’s 80th Birthday (P. Aluffi, D. Anderson, M. Hering, M. Mustaţă, and S & Payne, eds.), London Mathematical Society Lecture Note Series, Cambridge University Press, 2022, p. ????
  * [85] R. Steffens and T. Theobald, _Mixed volume techniques for embeddings of Laman graphs_ , Comput. Geom. 43 (2010), no. 2, 84–93.
  * [86] P. Stevenhagen and H.W. Lenstra, Jr., _Chebotarëv and his density theorem_ , Math. Intelligencer 18 (1996), no. 2, 26–37.
  * [87] N. Tschebotareff, _Die Bestimmung der Dichtigkeit einer Menge von Primzahlen, welche zu einer gegebenen Substitutionsklasse gehören_ , Math. Ann. 95 (1926), no. 1, 191–228.
  * [88] R. Vakil, _A geometric Littlewood-Richardson rule_ , Ann. of Math. (2) 164 (2006), no. 2, 371–421, Appendix A written with A. Knutson.
  * [89] by same author, _Schubert induction_ , Ann. of Math. (2) 164 (2006), no. 2, 489–512.
  * [90] J. Verschelde, _Algorithm 795: PHCpack: A general-purpose solver for polynomial systems by homotopy continuation_ , ACM Transactions on Mathematical Software 25 (1999), no. 2, 251–276.
  * [91] E. Verstraten, _Cognate linkages the roberts–chebyshev theorem_ , Explorations in the History of Machines and Mechanisms (Dordrecht) (T. Koetsier and M. Ceccarelli, eds.), Springer Netherlands, 2012, pp. 505–519.
  * [92] R. Williams, _Restrictions on Galois groups of Schubert problems_ , Ph.D. thesis, Texas A&M Univeristy, 2017.
  * [93] O. Zariski, _A theorem on the Poincaré group of an algebraic hypersurface_ , Annals of Mathematics 38 (1937), no. 1, 131–141.