# Multi-fidelity Bayesian Neural Networks: Algorithms and Applications

Xuhui Meng  Hessam Babaee  George Em Karniadakis 111Corresponding author:
<EMAIL_ADDRESS>(George Em Karniadakis). Division of Applied
Mathematics, Brown University, Providence, RI 02906, USA  Mechanical
Engineering and Materials Science, University of Pittsburgh, Pittsburgh, PA
15260, USA  Pacific Northwest National Laboratory, Richland, WA 99354, USA

###### Abstract

We propose a new class of Bayesian neural networks (BNNs) that can be trained
using noisy data of variable fidelity, and we apply them to learn function
approximations as well as to solve inverse problems based on partial
differential equations (PDEs). These multi-fidelity BNNs consist of three
neural networks: The first is a fully connected neural network, which is
trained following the maximum a posteriori probability (MAP) method to fit the
low-fidelity data; the second is a Bayesian neural network employed to capture
the cross-correlation with uncertainty quantification between the low- and
high-fidelity data; and the last one is the physics-informed neural network,
which encodes the physical laws described by PDEs. For the training of the
last two neural networks, we first employ the mean-field variational inference
(VI) to maximize the evidence lower bound (ELBO) to obtain informative prior
distributions for the hyperparameters in the BNNs, and subsequently we use the
Hamiltonian Monte Carlo method to estimate accurately the posterior
distributions for the corresponding hyperparameters. We demonstrate the
accuracy of the present method using synthetic data as well as real
measurements. Specifically, we first approximate a one- and four-dimensional
function, and then infer the reaction rates in one- and two-dimensional
diffusion-reaction systems. Moreover, we infer the sea surface temperature
(SST) in the Massachusetts and Cape Cod Bays using satellite images and in-
situ measurements. Taken together, our results demonstrate that the present
method can capture both linear and nonlinear correlation between the low- and
high-fideilty data adaptively, identify unknown parameters in PDEs, and
quantify uncertainties in predictions, given a few scattered noisy high-
fidelity data. Finally, we demonstrate that we can effectively and efficiently
reduce the uncertainties and hence enhance the prediction accuracy with an
active learning approach, using as examples a specific one-dimensional
function approximation and an inverse PDE problem.

###### keywords:

auto-regressive scheme , nonlinear correlation , physics-informed neural
networks , Hamiltonian Monte Carlo , uncertainty quantification , active
learning , satellite data

††journal: Journal Name

##  1 Introduction

In many applications in science and engineering the accuracy of predictions by
computational models may vary, depending on the the available multi-fidelity
data [1, 2]. Typically, increased fidelity of the data sources is associated
with higher acquisition cost. Expensive computations may include tasks such as
uncertainty quantification, optimization, inverse problems and sensitivity
analysis, in which a computational model must be evaluated in an outer loop
[3, 4, 5, 6]. For these systems, various data sources for obtaining
predictions may be available such as fine-grid/coarse-grid numerical models or
reduced order models. In other applications, the data may come both from both
a numerical model and experimental measurements. In the latter case, the data
may be obtained from different types of instruments; for example, satellite
images and in-situ measurements can provide data sources for inferring sea
surface temperature (SST). In a traditional _single data source_ approach, a
decision has to be made between using low-fidelity predictions with dense
samples, which can yield a high-resolution input-output map versus sparser
high-fidelity predictions, since they are costly to acquire.

Multi-fidelity techniques provide a _data multi-source_ paradigm, in which
_all_ sources of data with variable fidelity are utilized to enhance
predictions. Auto-regressive Gaussian processes regression (GPR) is one of the
most commonly used Bayesian techniques for multi-fidelity modeling [7, 8],
yielding predictions of the mean values as well as estimates of uncertainty.
Predicting uncertainty is critical in many applications, especially for cost-
effective allocation of limited data acquisition resources. The fact that GPR
multi-fidelity models provide uncertainty quantification is one of the main
reasons of the popularity of these techniques; see, for example [9], where the
GPR multi-fidelity model for SST is built by combining SST satellite
measurements (low-fidelity) with in-situ measurements (high-fidelity). The GPR
uncertainty maps obtaoned could then be utilized for guiding future in-situ
measurements using an appropriate acquisition function [10].

One of the challenges of GPR multi-fidelity models is that the computational
complexity of training multi-fidelity GPR scales as $\mathcal{O}(N^{3})$,
where $N$ is the total number of high-fidelity and low-fidelity training
points. As a result, the utility of GPR is largely limited to problems with
low dimensional input space corresponding to a small number of training
points. Moreover, GPR multi-fidelity models seek to exploit linear
correlations between various levels of fidelity. Although an extension of the
GPR multi-fidelity model to extract nonlinear correlations has been developed
in [11], this comes at a cost of adding one more dimension to the input space,
and, therefore, it requires larger datasets for training. As a remedy to these
limitations, recently, a multi-fidelity scheme based on a composite neural
network (NN) was presented [12], in which both linear and nonlinear
correlations between low-fidelity and high-fidelity data sets are exploited.
The minibatch training of the NN enables the application of this technique to
large training datasets in a scalable manner. Furthermore, the composite
neural network (NN) developed in [12] has also been successfully applied to
solve inverse PDE problems based on the idea of physics-informed neural
networks [13]. However, the existing multi-fidelity model using the composite
NN does not predict uncertainty, which is a byproduct of Bayesian multi-
fidelity models. This is the issue we address in the current work.

Specifically, in this paper we present a multi-fidelity scheme based on
Bayesian neural networks. The novelty of the framework we propose is three-
fold: (i) it obtains good uncertainty estimates for the quantity of interest,
e.g., high-fidelity predictions, unknown parameters in PDEs, etc.; (ii) it
exploits both linear and nonlinear correlations between low-fidelity and high-
fidelity data adaptively; and (iii) it is scalable with respect to the size of
the training datasets. The paper is organized as follows: we first present the
details of the proposed multi-fidelity approach in Sec. 2, while the results
including function approximations, inverse PDE problems, and active learning
are included in Sec. 3; a summary of the present study is given in Sec. 4.

##  2 Methodology

###  2.1 Multi-fidelity Bayesian neural networks

Figure 1:  Schematic of multi-fidelity Bayesian neural networks. The orange
panel on the left represents the deep neural network (DNN) for low-fidelity
data, the blue panel represents the Bayesian neural network (BNN) for the
cross-correlation, and the green panel is the physics-informed part.
$P(\bm{\theta})$ is the prior distribution for hyperparameters in BNNs and the
unknowns in PDEs, $P(\mathcal{D}|\bm{\theta})$ represents the likelihood of
observations (e.g., $u,b,f$), $P(\bm{\theta}|\mathcal{D})$ is the posterior,
and $P(\mathcal{D})$ denotes the marginal likelihood. $\bm{x}_{L}/\bm{x}_{H}$:
locations of the low-/high-fidelity training data,
$\tilde{u}_{L}$/$\tilde{u}_{H}$: low- and high-fidelity predictions,
$\tilde{u}_{L}(\bm{x}_{H})$: low-fidelity predictions at $\bm{x}_{H}$, $\phi$:
activation function, which is the hyperbolic tangent function in this study.
MAP: maximum a posteriori probability, HMC: Hamiltonian Monte Carlo.

As shown in Fig. 1, the multi-fidelity Bayesian neural network is composed of
three different neural networks: the first is a deep neural network (DNN) to
approximate the low-fidelity data, while the second one is a Bayesian neural
network (BNN) for learning the correlation (with uncertainty quantification)
between the low- and high-fidelity data. The last one is the physics-informed
neural network (PINN), which is employed to encode the physical laws
represented by partial differential equations (PDEs) [13, 14].

Considering that the low-fidelity data are generally plentiful, the
computational cost is prohibitive if a BNN is used here. To achieve high
computational efficiency, we therefore employ the maximum a posteriori
probability estimate (MAP) to train the first neural network (NN), which is
the same as in [12]. Each hyperparameter in the first neural network now is a
point estimate rather than a distribution. The loss function for the MAP is
expressed as

$\displaystyle\mathcal{L}_{L}=\frac{1}{N_{L}}|\bm{u}_{L}-\tilde{\bm{u}}_{L}|^{2}+\alpha|\bm{w}_{L}|^{2},$
(1)

where $N_{L}$ is the number of low-fidelity training data, $\bm{u}_{L}$ and
$\tilde{\bm{u}}_{L}$ are low-fidelity training data and predictions,
respectively, the second term is the $\mathcal{L}_{2}$ regularization with
$\bm{w}_{L}$ denoting the weights in the first DNN, with $\alpha$ the weight
of the regularization. Note that the $\mathcal{L}_{2}$ regularization here is
capable of reducing the overfitting caused by the measurement noise as well as
the DNN itself.

We then employ the Bayesian neural network [15] (Fig. 1), which takes
$(\bm{x}_{H},\tilde{\bm{u}}(\bm{x}_{H}))$ as inputs to quantify the
uncertainties in predicting the correlations between the low- and high-
fidelity data. In particular, the Hamiltonian Monte Carlo (HMC) method [16],
which is frequently used as ground truth for posterior estimation, is adopted
here to estimate the posterior distributions in BNN [17]. In general, the
estimation of posterior distributions for hyperparameters in the BNN is
computationally prohibitive for problems with big data using HMC [14].
However, the high-fidelity data are generally scarce due to high acquisition
cost associated with high-fidelity data. Therefore, we can exploit the BNN to
learn the correlation between the low- and high-fidelity data with uncertainty
quantification given a small amount of high-fidelity observations at low cost.
Details for the training of BNNs for function approximation as well as inverse
PDE problems will be elaborated below. Note that all variables without
subscripts in this study denote the measurements/predictions at the high-
fidelity level (as well as variables with subscript “H”).

We assume that for function approximation, the high-fidelity datasets are
noisy scattered measurements on $u$, for example from sensors, i.e.,
$\mathcal{D}=\mathcal{D}_{u}$. In addition, we assume that the measurements
are independent Gaussian distributions, i.e.,

$\displaystyle{u}^{(i)}=\bar{u}(\bm{x}_{u}^{(i)})+\epsilon_{u}^{(i)},\quad
i=1,2...N_{u},$ (2)

where $\bar{u}(\bm{x}_{u}^{(i)})$ and $\epsilon_{u}^{(i)}$ are the exact value
and the independent Gaussian noise with zero mean. Furthermore, the noise
scale of each sensor is assumed to be known, i.e., the standard deviations of
$\epsilon_{u}^{(i)}$ is known to be $\sigma_{u}^{(i)}$.

We also consider inverse PDE problems given by,

$\displaystyle\mathcal{N}_{\bm{x}}(u;\bm{\lambda})$
$\displaystyle=f,\quad\bm{x}\in R^{D},$ (3)
$\displaystyle\mathcal{B}_{\bm{x}}(u;\bm{\lambda})$
$\displaystyle=b,\quad\bm{x}\in\Gamma,$

where $\mathcal{N}_{\bm{x}}$ is a general differential operator, $D$
represents the number of dimension for a physical domain, $u=u(\bm{x})$ is the
solution of the PDE, and $\bm{\lambda}$ is a vector of unknown parameters in
the PDE, $f=f(\bm{x})$ is the forcing term, $\mathcal{B}_{\bm{x}}$ is the
boundary condition operator, and $\Gamma$ represents the domain boundary. In
addition to $u$, we also have noisy scattered measurements on $f$ and $b$ from
sensors. The entire high-fidelity dataset $\mathcal{D}$ can then be expressed
as

$\displaystyle\mathcal{D}$
$\displaystyle=\mathcal{D}_{u}\cup\mathcal{D}_{f}\cup\mathcal{D}_{b},$ (4)

where $\leavevmode\nobreak\
\mathcal{D}_{f}=\\{(\bm{x}_{f}^{(i)},\leavevmode\nobreak\
{f}^{(i)})\\}_{i=1}^{N_{f}}$, and
$\mathcal{D}_{b}=\\{(\bm{x}_{b}^{(i)},\leavevmode\nobreak\
{b}^{(i)})\\}_{i=1}^{N_{b}}$. Similarly, we also assume that the measurements
on $f$ and $b$ are independent Gaussian distributions, i.e.,

$\displaystyle{f}^{(i)}$
$\displaystyle=\bar{f}(\bm{x}_{f}^{(i)})+\epsilon_{f}^{(i)},\quad
i=1,2...N_{f},$ (5) $\displaystyle{b}^{(i)}$
$\displaystyle=\bar{b}(\bm{x}_{b}^{(i)})+\epsilon_{b}^{(i)},\quad
i=1,2...N_{b},$

where $\bar{f}$ and $\bar{b}$ are the exact values for $f$ and $b$,
respectively, and $\epsilon_{f}^{(i)}$ and $\epsilon_{b}^{(i)}$ are
independent Gaussian noises with zero mean and known standard deviations,
i.e., $\sigma_{f}^{(i)}$ and $\sigma_{b}^{(i)}$, respectively.

In the Bayesian framework, we parametrize $u$ with a surrogate model
$\tilde{u}(\bm{x}_{u},\tilde{{u}}_{L};\bm{\theta})$, where $\bm{\theta}$
represents the hyperparameters in the surrogate model (i.e., BNN) and/or
unknown parameters in the PDE with a prior distribution $P(\bm{\theta})$.
Consequently, $f$ and $b$ can then be represented by:

$\displaystyle\tilde{f}(\bm{x}_{f};\bm{\theta})=\mathcal{N}_{\bm{x}}(\tilde{u}(\bm{x}_{f},\tilde{{u}}_{L};\bm{\theta});\bm{\lambda}),\leavevmode\nobreak\
\tilde{b}(\bm{x};\bm{\theta})=\mathcal{B}_{\bm{x}_{b}}(\tilde{u}(\bm{x}_{b},\tilde{{u}}_{L};\bm{\theta});\bm{\lambda}).$
(6)

The likelihood for $\mathcal{D}$ can be computed as:

$\displaystyle P(\mathcal{D}|\bm{\theta})$
$\displaystyle=P(\mathcal{D}_{u}|\bm{\theta})P(\mathcal{D}_{f}|\bm{\theta})P(\mathcal{D}_{b}|\bm{\theta}),$
(7) $\displaystyle P(\mathcal{D}_{u}|\bm{\theta})$
$\displaystyle=\prod_{i=1}^{N_{u}}\frac{1}{\sqrt{2\pi{\sigma_{u}^{(i)}}^{2}}}\exp\left(-\frac{(\tilde{u}(\bm{x}_{u}^{(i)};\bm{\theta})-\bar{u}^{(i)})^{2}}{2{\sigma_{u}^{(i)}}^{2}}\right),$
$\displaystyle P(\mathcal{D}_{f}|\bm{\theta})$
$\displaystyle=\prod_{i=1}^{N_{f}}\frac{1}{\sqrt{2\pi{\sigma_{f}^{(i)}}^{2}}}\exp\left(-\frac{(\tilde{f}(\bm{x}_{f}^{(i)};\bm{\theta})-\bar{f}^{(i)})^{2}}{2{\sigma_{f}^{(i)}}^{2}}\right),$
$\displaystyle P(\mathcal{D}_{b}|\bm{\theta})$
$\displaystyle=\prod_{i=1}^{N_{b}}\frac{1}{\sqrt{2\pi{\sigma_{b}^{(i)}}^{2}}}\exp\left(-\frac{(\tilde{b}(\bm{x}_{b}^{(i)};\bm{\theta})-\bar{b}^{(i)})^{2}}{2{\sigma_{b}^{(i)}}^{2}}\right).$

We can then obtain the posterior based on Bayes’ rule as:

$\displaystyle
P(\bm{\theta}|\mathcal{D})=\frac{P(\mathcal{D}|\bm{\theta})P(\bm{\theta})}{P(\mathcal{D})}\simeq
P(\mathcal{D}|\bm{\theta})P(\bm{\theta}),$ (8)

in which “$\simeq$” represents equality up to a constant. Generally,
$P(\mathcal{D})$ is analytically intractable. Here, we employ HMC to sample
from the unnormalized $P(\bm{\theta}|\mathcal{D})$ [14, 16]. Then, we can
obtain predictions on $u$ at any $\bm{x}$ (i.e.,
$\\{\tilde{u}^{(i)}(\bm{x})\\}^{M}_{i=1}$) based on the posterior samples
(i.e., $\\{{\bm{\theta}}^{(i)}\\}_{i=1}^{M}$) from the HMC. Finally, we can
compute the mean as well as the standard deviation for the predictions since
the former represents the predictions on $u(\bm{x})$ while the latter
quantifies the uncertainty. Note that we can also obtain means and standard
deviations for the unknown parameters in the PDEs, i.e., $\bm{\lambda}$, based
on the posterior samples in a similar way for $u$.

We note that by $\bm{\lambda}$ we denote a set of unknown constants in the
present study. However, in general, $\bm{\lambda}$ can represent an unknown
field in this framework and can be represented by another neural network. The
interested readers can refer to [14] for more details.

###  2.2 Priors in BNNs

The surrogate model for $u$, i.e., BNN, is a fully-connected neural network
with $L\geq 1$ hidden layers. The input of the neural network is the
concatenation of $\bm{x_{H}}$ and the low-fidelity prediction
$\tilde{\bm{u}}_{L}(\bm{x}_{H})$. The output of the $l$-th hidden layer
$\bm{z}_{l}\in\mathbb{R}^{N_{l}}$ ($l=1,2...N$) is then denoted as

$\displaystyle\bm{z}_{l}$
$\displaystyle=\phi(\bm{w}_{l-1}\bm{z}_{l-1}+\bm{b}_{l-1}),\quad l=1,2...L,$
(9) $\displaystyle\tilde{u}$ $\displaystyle=\bm{w}_{L}\bm{z}_{L}+\bm{b}_{L},$

where $\bm{w}_{l}\in\mathbb{R}^{N_{l+1}\times N_{l}}$ are the weight matrices,
$b_{l}\in\mathbb{R}^{N_{l+1}}$ are the bias vectors, $\phi$ is the nonlinear
activation function, which is the hyperbolic tangent function in the present
study, and
$\bm{z}_{0}=(\bm{x}_{H},\tilde{\bm{u}}_{L}(\bm{x}_{H})),\leavevmode\nobreak\
N_{0}=N_{x_{H}}+N_{\tilde{\bm{u}}_{L}(x_{H})}$, and $N_{L+1}=1$. When using a
neural network as a surrogate model, the unknown parameters $\bm{\theta}$ are
the concatenation of all the weight matrices and bias vectors.

In applications of Bayesian neural networks, the independent Gaussian
distribution with zero mean for each component of $\bm{\theta}$ is a commonly
used prior. In addition, the entries of $\bm{w}_{l}$ and $\bm{b}_{l}$ have the
variances $\sigma_{w,l}$ and $\sigma_{b,l}$, respectively, for $l=0,1...L$
[15, 16]. In this case, it can be shown that the prior of the function
$\tilde{u}(\bm{x})$ is actually a Gaussian process as the width of hidden
layers goes to infinity with $\sqrt{N_{l}}\sigma_{w,l}$ fixed for $l=1,2...L$
[15, 18, 19]. In the present study, a BNN with finite width is employed, and
we also set the Gaussian distribution as prior for each hyperparameter. The
priors for hyperparameters in the same layer are identical. In particular,
$\sigma_{w,l}=\sigma/\sqrt{N}_{l}$, and $\sigma_{b,l}=1$, where $l=0,1...L$,
suggesting that the prior distribution is determined by $\sigma$ here.

A good prior is crucial in Bayesian inference, especially for problems with
small datasets, e.g., the high-fidelity training data in multi-fidelity
modelings. To obtain a proper prior, here we employ the mean-field variational
inference (VI) to estimate $\sigma$ by maximizing the marginal likelihood. In
particular, the posterior density of the unknown parameter vector
$\bm{\theta}=(\theta_{1},\theta_{2}...\theta_{d_{\bm{\theta}}})$, i.e.,
$P(\bm{\theta}|\mathcal{D})$, is approximated by another density function
$Q(\bm{\theta};\bm{\zeta})$ parameterized by $\bm{\zeta}$ in the variational
inference. A common choice in deep learning is the mean-field Gaussian
approximation [17], i.e., $Q$ is a factorizable Gaussian distribution as
follows:

$Q(\bm{\theta};\bm{\zeta})=\prod_{i=1}^{d_{\bm{\theta}}}q(\theta_{i};\zeta_{\mu,i},\zeta_{\rho,i}),$
(10)

where $\bm{\zeta}=(\bm{\zeta}_{\mu},\bm{\zeta}_{\rho})$,
$\bm{\zeta}_{\mu}=(\zeta_{\mu,1},\zeta_{\mu,2}...\zeta_{\mu,d_{\bm{\theta}}})$,
$\bm{\zeta}_{\rho}=(\zeta_{\rho,1},\zeta_{\rho,2}...\zeta_{\rho,d_{\bm{\theta}}})$,
and $q(\theta_{i};\zeta_{\mu,i},\zeta_{\rho,i})$ is the density of the one-
dimensional Gaussian distribution with mean $\zeta_{\mu,i}$ and standard
deviation $\log(1+\exp(\zeta_{\rho,i}))$. In this approach, we can tune
$\bm{\zeta}$ by maximizing the evidence lower bound (ELBO) as [20]:

$\displaystyle\arg\max_{\bm{\zeta},\sigma}\mathcal{L}_{ELBO}$
$\displaystyle=\log
P(\mathcal{D})-D_{KL}\left[Q(\bm{\theta};\bm{\zeta})||P(\bm{\theta}|\mathcal{D})\right]$
$\displaystyle\simeq\mathbb{E}_{\bm{\theta}\thicksim Q}[\log
Q(\bm{\bm{\theta}};\bm{\zeta})-\log P(\bm{\theta})-\log
P(\mathcal{D}|\bm{\theta})],$

which is equivalent to maximizing the marginal likelihood as well as
minimizing the Kullback-Leibler divergence, $D_{KL}$, between
$Q(\bm{\theta};\bm{\zeta})$ and $P(\bm{\theta}|\mathcal{D})$, i.e.,
$D_{KL}\left[Q(\bm{\theta};\bm{\zeta})||P(\bm{\theta}|\mathcal{D})\right]$. In
the present study, we employ the Adam optimizer to obtain the optimal
$\bm{\zeta}$ as well as $\sigma$.

The mean-field VI is easy to implement and it is scalable to big data problems
by using minibatch training. However, the estimation of posterior distribution
(i.e., $Q(\bm{\theta};\bm{\zeta})$) is not quite accurate since the mean-field
VI is employed [14, 17]. To overcome this drawback, we then employ HMC to
estimate the posterior distribution based on the prior provided by the VI.
Furthermore, the stochastic HMC can also be used to handle big data in high-
dimensional problems. The main steps of the present multi-fidelity Bayesian
neural network can be found in Algorithm 1.

Algorithm 1 Multi-fidelity Bayesian neural network (MBNN)

Low- and high-fidelity noisy measurements and/or PDEs.

$\bullet$ Train the first DNN to fit the low-fidelity data using MAP;

$\bullet$ Train the BNN with high-fidelity training data using mean-fidelity
VI to get a proper prior, i.e., $\sigma$;

$\bullet$ Employ the HMC (or stochastic HMC) to get posterior samples for the
hyperparameters in BNN with the priors provided by the mean-fidelity VI;

$\bullet$ Get predictions for $\\{\tilde{u}^{(i)}(\bm{x})\\}^{M}_{i=1}$) based
on the posterior samples from the previous step.

##  3 Results

We test the performance of the proposed multi-fidelity BNN for diverse
applications, including function approximations as well as inverse PDE
problems. In addition, we conduct active learning to improve the prediction
accuracy based on the predicted uncertainty. For the training of the DNN, we
employ the Adam optimizer with an initial learning rate $10^{-3}$, while the
number of training steps is set to 50,000 in each case. The weight $\alpha$
for the $\mathcal{L}_{2}$ regularization is set to $\sigma^{2}_{u_{L}}/N_{L}$,
where $\sigma^{2}_{u_{L}}$ and $N_{L}$ are the variance of the noise and the
number for the low-fidelity data, respectively. As for the BNN, we first use
the Adam optimizer with an initial learning rate $10^{-3}$ to train it for
200,000 steps, and then we employ the HMC with adaptive time step for
posterior estimation [14] with the prior (i.e., $\sigma$) provided by the VI.
The number of burn-in steps is set to 10,000, the initial time step is set as
$\delta_{t}=0.1$, and the leapfrog step is $L=50\delta_{t}$. Finally, we
employ 1,000 posterior samples, i.e., $M=1,000$, for the computations of the
predicted means and standard deviations in each case.

###  3.1 Function approximation

To further explain the method, in this section, we consider approximations of
a one-dimensional (1D) and a four-dimensional (4D) function using bi-fidelity
data. Both linear and nonlinear correlations between the low- and high-
fidelity data are tested.

####  3.1.1 1D function approximation

The low- and high-fidelity data are generated using the following two
functions [11]:

$\displaystyle u_{L}$ $\displaystyle=\sin(8x),\leavevmode\nobreak\ x\in[0,1],$
(11) $\displaystyle u_{H}$ $\displaystyle=(x-\sqrt{2})u^{2}_{L},$ (12)

where $u_{L}$ and $u_{H}$ represent the low- and high-fidelity functions,
respectively. In addition, we assume that the high-fidelity measurements are
noisy. As for the low-fidelity data, we consider two different sources, i.e.,
(1) “simulation” data with no noise, and (2) noisy “measurements”. Here, all
the noises are assumed to be Gaussian with zero mean. Specifically, we test
two different cases: (1) $\epsilon_{L}=0$,
$\epsilon_{H}\thicksim\mathcal{N}(0,0.01^{2})$; and (2)
$\epsilon_{L}\thicksim\mathcal{N}(0,0.05^{2})$,
$\epsilon_{H}\thicksim\mathcal{N}(0,0.01^{2})$, where $\epsilon_{L}$ and
$\epsilon_{H}$ denote the noise of the low- and high-fidelity data,
respectively. For both cases, we employ 100 uniformly distributed data at the
low-fidelity level and 14 high-fidelity measurements, which is the same setup
as in [11], as displayed in Fig. 2.

Figure 2:  1D function approximation for cases with different noise levels:
Predicted means and uncertainties. (a) Multi-fidelity modeling with
$\epsilon_{L}\thicksim 0$, $\sigma=1.2$. (b) Multi-fidelity modeling with
$\epsilon_{L}\thicksim\mathcal{N}(0,0.05^{2})$, $\sigma=1.4$. (c) Single-
fidelity modeling, $\sigma=6.5$. Blue circles: high-fidelity training data. 2
std: two standard deviations.

In the DNN we employ 2 hidden layers with 20 neurons per layer, while in the
BNN we use 1 hidden layer with 50 neurons. The results from the multi-fidelity
modeling are illustrated in Fig. 2. As for the first case (Fig. 2), the multi-
fidelity predictions are reasonable: (1) the predicted means are in good
agreement with the exact solutions, and (2) the errors between the predicted
means and the exact solutions are mostly bounded by the two standard
deviations in the entire domain. Similar results can also be observed in Fig.
2 as we perturb the low-fidelity training data with the Gaussian noise, i.e.,
$\mathcal{N}(0,0.05^{2})$. However, the predicted uncertainties for the case
in Fig. 2 are generally larger than those in Fig. 2, which can be attributed
to the less accurate predictions for the low-fidelity data. To demonstrate the
effectiveness of the multi-fidelity modeling, we further present the results
using single-fidelity modeling, i.e., we train the same BNN employed in the
multi-fidelity modeling using the high-fidelity data only. As displayed in
Fig. 2, the multi-fidelity modeling can significantly enhance the prediction
accuracy compared to the single-fidelity modeling. We note that the results
from single-fidelity modeling may be improved by using different architectures
for BNN. Here, we employ the same BNN in both single- and multi-fidelity
modeling for a fair comparison.

We observe that the priors (i.e., $\sigma$) for the multi-fidelity cases
(Figs. 2 and 2) are slightly different, while they are quite different from
the prior in the single-fidelity case (Fig. 2). The difference in the priors
of the multi-fidelity cases is attributed to the differences in low-fidelity
predictions at $\bm{x}_{H}$, i.e., $\tilde{\bm{u}}_{L}(\bm{x}_{H})$. In
addition, the input and output in the single-fidelity modeling are
$\bm{x}_{H}$ and $\bm{u}_{H}(\bm{x}_{H})$, respectively, which is the same as
in [14]. Considering that different inputs are employed in the multi-fidelity
modeling (Figs. 2-2) and single-fidelity modeling (Fig. 2), the priors, i.e.,
$\sigma$, in these cases are also quite different.

We proceed to study the case in which the measurement noise for the low-
fidelity data is much larger, i.e.,
$\epsilon_{L}\thicksim\mathcal{N}(0,0.3^{2})$. We first test the case with 100
low-fidelity data that are uniformly distributed in $x\in[0,1]$. The
parameters (e.g., optimizer, learning rate, etc.) and architectures for both
neural networks are kept the same as the previous case. The low- and high-
fidelity predictions are illustrated in Fig. 3; the high-fidelity predictions
are not as good as the results in Fig. 2, which can be attributed to the
inaccurate predictions for the low-fidelity profile (left in Fig. 3). To
enhance the prediction accuracy, we then increase the number of low-fidelity
training data, to 1,000, and 1,500. As we can see in the second and third
columns of Fig. 3, the low-fidelity predictions become more accurate.
Furthermore, the accuracy for the multi-fidelity modelings (i.e., middle and
right panels of Fig. 3) are also enhanced as compared to the first left in
Fig. 3. In particular, the high-fidelity predictions are quite similar as the
results in Fig. 2 in which the low-fidelity data are noise free. It is also
interesting to find that the prior (i.e., $\sigma$) for the case with 1,500
low-fidelity training data is the same as in Fig. 2, which indicates that the
low-fidelity predictions at $\bm{x}_{H}$ (i.e.,
$\tilde{\bm{u}}_{L}(\bm{x}_{H})$) are quite similar in these two cases.

All the above results indicate that accurate low-fidelity predictions can
enhance the accuracy for multi-fidelity modeling. A possible way to improve
the accuracy for low-fidelity predictions is to increase the number of low-
fidelity training data as shown in Fig. 3.

Figure 3:  Multi-fidelity modeling for 1D function approximations with
$\epsilon_{L}\thicksim\mathcal{N}(0,0.3^{2})$: Different numbers of low-
fidelity training data, i.e., 100, 1,000, and 1,500 (from left to right). (a)
Predicted high-fidelity profiles for different low-fidelity points. From left
to right: 100 ($\sigma=2.2$), 1,000 ($\sigma=2.2$), and 1,500 ($\sigma=1.2$).
(b) Predicted low-fidelity profiles for different numbers of low-fidelity
training data. From left to right: 100, 1,000, and 1,500.

We further consider a more practical case, in which both the low- and high-
fidelity data are from sensors but with different accuracy. In addition, we
assume that there exists a bias between the low- and high-fidelity data, i.e.,

$\displaystyle u_{L}$ $\displaystyle=(x-\sqrt{2})\sin^{2}(8\pi
x)+x-2,\leavevmode\nobreak\ x\in[0,1],$ (13) $\displaystyle u_{H}$
$\displaystyle=u_{L}-x+2.$ (14)

We test two different noise levels for the low-fidelity data, i.e.,
$\epsilon_{L}\thicksim\mathcal{N}(0,0.05^{2})$ and
$\epsilon_{L}\thicksim\mathcal{N}(0,0.3^{2})$. We employ 100 uniformly
distributed low-fidelity training data for the case with
$\epsilon_{L}\thicksim\mathcal{N}(0,0.05^{2})$, and 1,000 uniformly
distributed low-fidelity training data for the case with
$\epsilon_{L}\thicksim\mathcal{N}(0,0.3^{2})$; the high-fidelity training data
are kept the same as in the previous cases. We see in Fig. 4 that: (1) the
predicted means for high-fidelity profiles are in good agreement with the
exact solutions for both cases, and (2) the computational errors between the
predicted means and exact solutions for $x\in[0,1]$ are mostly bounded by the
two standard deviations.

Figure 4:  Multi-fidelity modeling for 1D function approximation. (a)
Predicted low-fidelity profile for
$\epsilon_{L}\thicksim\mathcal{N}(0,0.05^{2})$. (b) Predicted low-fidelity
profile for $\epsilon_{L}\thicksim\mathcal{N}(0,0.3^{2})$. (c) Predicted means
and uncertainties for high-fidelity profiles with
$\epsilon_{L}\thicksim\mathcal{N}(0,0.05^{2})$, $\sigma=3.1$. (d) Predicted
means and uncertainties for high-fidelity profiles with
$\epsilon_{L}\thicksim\mathcal{N}(0,0.3^{2})$, $\sigma=3.5$.

####  3.1.2 4D function approximation

We now consider a four-dimensional function approximation problem using multi-
fidelity noisy data. The multi-fidelity training data are generated from the
following functions:

$\displaystyle u_{H}$
$\displaystyle=\frac{1}{2}\left[0.1\exp(x_{1}+x_{2})-x_{4}\sin(12\pi
x_{3})+x_{3}\right],$ (15) $\displaystyle u_{L}$ $\displaystyle=1.2u_{H}-0.5,$
(16)

where $x_{i}\in(0,1],\leavevmode\nobreak\ i=1,2,3,4$. Similarly, we also
consider two different sources of low-fidelity data, i.e., (1) “simulation”
data with no noise, and (2) “measurements” with Gaussian noise, i.e.,
$\mathcal{N}(0,0.05^{2})$. In addition, the measurement errors for the high-
fidelity data are still Gaussian but with a smaller variance, i.e.,
$\mathcal{N}(0,0.01^{2})$. In both cases, we employ 25,000 and 150 randomly
sampled low- and high-fidelity data for training at the low- and high-fidelity
levels, respectively. In addition, the DNN employed for approximating the low-
fidelity data has 2 hidden layers with 50 neurons per layer, while for the
BNNs, we employ 1 hidden layers with 50 neurons. Similarly, the same BNN is
employed in the single-fidelity modeling for the purpose of comparison. For
validation, we present the high-fidelity predictions at 100,000 random
locations. As shown in Fig. 5, the predicted means from the multi-fidelity
BNNs for both cases (Figs. 5-5) are much better than the single-fidelity
modeling (Fig. 5).

Figure 5:  4D function approximation. (a) Predicted high-fidelity data at
100,000 random testing locations with $\epsilon_{L}\thicksim 0$, $\sigma=4.2$.
(b) Predicted high-fidelity data at 100,000 random testing locations with
$\epsilon_{L}\thicksim\mathcal{N}(0,0.05^{2})$, $\sigma=4$. (c) Single-
fidelity predictions at 100,000 random testing locations, $\sigma=7.5$.

To evaluate the uncertainty in the test cases, here, we employ the following
metric, the prediction interval coverage probability (PICP) [17]:

$\displaystyle\frac{1}{N}\sum^{N}_{n=1}c_{i},$ (17)

where $N$ is the total number of predictions, and $c_{i}=1$ if
$u(\bm{x}_{n})\in[\tilde{u}_{low},\tilde{u}_{high}]$, otherwise $C_{i}=0$. In
addition, $u(\bm{x}_{n})$ is the exact value at $\bm{x}_{n}$,
$\tilde{u}_{low}=\tilde{u}-2\tilde{u}_{std}$, and
$\tilde{u}_{high}=\tilde{u}+2\tilde{u}_{std}$, where $\tilde{u}$ and
$\tilde{u}_{std}$ denote the predicted mean and standard deviation for
$\tilde{u}$. Ideally, PICP should be close to 1. As displayed in Table 1, the
PICPs for both multi-fidelity cases are greater than $90\%$, which means most
of the errors in the multi-fidelity modeling are bounded by two standard
deviations, and the PICPs from the multi-fidelity modeling are much better
than in the single-fidelity modeling, which demonstrates the effectiveness of
the present multi-fidelity approach.

Table 1:  4D function approximation with different noise scales: prediction interval coverage probability (PICP). SF: single-fidelity modeling, MF: multi-fidelity modeling.  | MF, $\epsilon_{L}=0$ | MF, $\epsilon_{L}\thicksim\mathcal{N}(0,0.05^{2})$ | SF
---|---|---|---
PICP | $98.1\%$ | $92.0\%$ | $29.8\%$

####  3.1.3 Multi-fidelity modeling of sea surface temperature (SST)

Figure 6:  Training data for sea surface temperature approximation: (a) Low-
fidelity training data: MODIS. (b) High-fidelity data: MWRA. The symbols
represent MWRA stations. Black circles: Training data, Red Crosses: Testing
data.

We now apply the present model to approximate the sea surface temperature
(SST) in the Massachusetts and Cape Cod Bays. Two sources of data are employed
here: (1) Low-fidelity: The MODerate-resolution Imaging Spectroradiometer
(MODIS) Terra on board NASA satellite with the spatial resolution $4\times
4\leavevmode\nobreak\ km$ [21], and (2) High-fidelity (in situ): The
Massachusetts Water Resource Authority (MWRA) [22]. In particular, stations
for temperature which are 1$m$ below the sea surface are used as an estimate
for seawater surface temperature. More details on the data can be found in
[9]. The low- and high-fidelity data are for September 8th, 2015. All the MWRA
measurements of different stations are assumed to be taken simultaneously. The
locations of MWRA stations are shown in Fig. 7, where 14 measurements are
available. Twelve of them are employed as the training data (black circles in
Fig. 7), while the others ($x_{A}:(-70.86,42.42)$, and $x_{B}:(-70.23,41.91)$)
(red crosses in Fig. 7) are used to validate the predictions.

In multi-fidelity modeling, we employ 2 hidden layers with 40 neurons per
layer for the low-fidelity NN, and only 1 hidden layer with 50 neurons in the
BNN. The predicted temperature field and the uncertainty map using BNN are
shown in Fig. 7. It is clear that the uncertainty is near zero close to MWRA
stations (high-fidelity measurements) while the uncertainty is much higher in
the north east of the map, where no MWRA in situ measurements are available.
We also show similar results obtained from GPR multi-fidelity model and
single-fidelity BNNs in Figs. 7 and 7, respectively. Both BNNs and GPR multi-
fidelity models show finer structure SST variations as they both utilize the
high resolution (in comparison with scarce MWRA measurements) satellite data,
while the single-fidelity BNN captures the large structures in the SST as it
only utilizes scarce (12) MWRA measurements. Furthermore, we display the
predicted SSTs at the two testing locations, i.e., $x_{A}$ and $x_{B}$, in
Table 2, as well as the results from single-fidelity modeling. We observe that
the predicted means at the two testing locations from these two methods have
comparable accuracy. However, the multi-fidelity modeling provides more
accurate prediction for $x_{A}$ than the single-fidelity modeling. While this
demonstration shows that BNN performs comparably to GPR multi-fidelity, it is
important to note that training the GPR multi-fidelity for large datasets, for
example annual space-time SST models, can be computationally prohibitive,
whereas training the BNN can be done in a scalable manner for large training
data.

Figure 7:  Approximation of sea surface temperature from multi-fidelity data: (a) Predicted means and uncertainties from multi-fidelity BNNs with $\sigma=1.8$. (b) Predicted means and uncertainties from multi-fidelity GPR. (c) Predicted means and uncertainties from single-fidelity BNNs with $\sigma=1.8$.  Table 2:  Single- and multi-fidelity predictions for the temperature at the testing locations (mean $\pm$ one standard deviation). MF-BNNs: multi-fidelity BNNs, MF-GP: multi-fidelity GP, SF-BNNs: single-fidelity BNNs. | MWRA | MF-BNNs | MF-GP | SF-BNNs
---|---|---|---|---
$x_{A}$ | 19.72 | 20.12 $\pm$ 0.27 | 20.10 $\pm$ 0.38 | 19.13 $\pm$ 0.18
$x_{B}$ | 21.27 | 20.52 $\pm$ 1.01 | 20.84 $\pm$ 0.46 | 21.94 $\pm$ 0.81

###  3.2 Data-driven solutions for inverse PDE problems

Next, we employ PINNs attached to BNN (see Fig. 1) to test our method using
PDE-based problems.

####  3.2.1 1D inverse PDE problem

We first consider the following 1D nonlinear problem, which is described by

$\displaystyle\frac{1}{192\pi^{2}}u_{xx}-\frac{k}{24\pi}uu_{x}$
$\displaystyle=f(x),\leavevmode\nobreak\ x\in[0,1],$ (18)

where $k=1$ is a constant. The exact solution to this problem is expressed as
$u(x)=(x-\sqrt{2})\sin^{2}(8\pi x)$, and $f(x)$ can be derived from Eq. (18).
Here, we assume that $k$ is unknown, and the objective is to identify the
value of $k$ given a limited number of measurements for $u(x)$ and $f(x)$.

As for measurements, we assume that we have high-fidelity sensors for $u(x)$
and $f(x)$, respectively. In particular, we have the same number of sensors
for both $u$ and $f$, i.e., 10, which are randomly distributed in the
computational domain, and we have two additional sensors for $u(x)$, which are
at the two boundaries to provide Dirichlet boundary conditions. Finally, we
assume that the measurement errors for $u$ and $f$ follow Gaussian
distributions, i.e., $\epsilon_{u}\thicksim\mathcal{N}(0,0.01^{2})$, and
$\epsilon_{f}\thicksim\mathcal{N}(0,0.01^{2})$.

Figure 8:  1D inverse PDE problem. (a) Predicted means and uncertainties for $u$ and $f$ using multi-fidelity data: $\epsilon_{u_{L}}=0$, and $\sigma=1.4$. (b) Predicted means and uncertainties for $u$ and $f$ using multi-fidelity data: $\epsilon_{u_{L}}\thicksim\mathcal{N}(0,0.05^{2})$), and $\sigma=1.4$. (c) Predicted means and uncertainties for $u$ and $f$ using high-fidelity data only with $\sigma=4.6$. Blue circles: high-fidelity training data.  |  MF, $\epsilon_{u_{L}}=0$ |  MF, $\epsilon_{u_{L}}\thicksim\mathcal{N}(0,0.05^{2})$ | SF
---|---|---|---
Mean | 1.006 | 1.031 | 4.128
Std | $4.20\times 10^{-2}$ | $5.08\times 10^{-2}$ | $1.63\times 10^{-1}$
Table 3:  1D inverse PDE problem: Predicted mean and standard deviation for
$k$, where $k=1$ is the exact solution. MF, $\epsilon_{u_{L}}=0$: multi-
fidelity modeling with $\epsilon_{u_{L}}=0$; MF,
$\epsilon_{u_{L}}\thicksim\mathcal{N}(0,0.05^{2})$: multi-fidelity modeling
with $\epsilon_{u_{L}}\thicksim\mathcal{N}(0,0.05^{2})$; SF: single-fidelity
modeling.

To obtain the low-fidelity training data, we can either solve Eq. (18)
numerically or analytically with a guessed $k$ and $f$ based on our prior
knowledge, which is similar to the work in [12], or directly measure $u$ using
low-fidelity sensors. Here we simply assume that the exact low-fidelity
function is $u_{L}(x)=\sin(8\pi x)$, which is the same as in Sec. 3.1.1.
Similarly, we consider the following two different low-fidelity data: (1)
simulation data with $\epsilon_{u_{L}}=0$, and (2) measurements with Gaussian
noise, $\epsilon_{u_{L}}\thicksim\mathcal{N}(0,0.05^{2})$. In both cases, 500
uniformly distributed low-fidelity samples are employed. In the DNN for the
low-fidelity data, we employ 2 hidden layers with 20 neurons per layers, while
in the BNN we use 1 hidden layer with 50 neurons. To obtain a proper prior
distribution for $k$, we can either follow the same way for obtaining the
priors of the hyperparameters in BNNs, i.e., maximizing the ELBO using VI, or
set it directly based on the prior physical knowledge. Here we employ the
latter approach for the prior of $k$ for simplicity, i.e., a standard normal
distribution here. The results from the multi-fidelity modeling are
illustrated in Figs. 8-8. As shown, the predicted means for $u$ and $f$ agree
with the exact solutions well for both cases, which can also be reflected by
the small predicted uncertainties. In addition, the predicted uncertainties
for both $u$ and $f$ for the case in Fig. 8 are larger than those in Fig. 8,
which is reasonable due to the noise scale for the low-fidelity data in the
former case, which is larger than in the latter case. Note that similar
results have also been observed in Sec. 3.1.1, and will not be discussed in
detail here. Furthermore, we present the results from single-fidelity
modeling, i.e., using the same BNNs and train them with the high-fidelity
training data only, which is the same as in the Bayesian physics-informed
neural networks (B-PINN) [14]. As shown in Fig. 8, the multi-fidelity results
are more accurate than the single-fidelity solutions.

Finally, the predicted $k$ for the three test cases in Fig. 8 are displayed in
Table 3, which demonstrates: (1) the predicted means in both multi-fidelity
cases are in good agreement with the exact solutions, (2) the computational
error increases with the increasing noise scale in low-fidelity data, which is
consistent with the results in Figs. 8-8, (3) the computational errors in both
multi-fidelity cases have the similar order as the noise scale, and they are
all bounded by the one standard deviation, and (4) the multi-fidelity modeling
outperforms the single-fidelity modeling with the aid of low-fidelity data.

####  3.2.2 2D inverse PDE problem

We proceed to consider a 2D steady-state diffusion-reaction system as

$\displaystyle\lambda\left(\partial^{2}_{x}u+\partial^{2}_{y}u\right)-ku^{2}=f,\leavevmode\nobreak\
x,\leavevmode\nobreak\ y\leavevmode\nobreak\ \in[-1,1],$ (19)

where $u$ represents the concentration, $\lambda=0.01$ is the diffusion
coefficient, $k$ is the reaction rate, and $f$ is the source term. The exact
solution to Eq. (19) is $u=\sin(2\pi x)\sin(2\pi y)$, while $f$ can be derived
from the exact solution for Eq. (19). Here, we assume that we have a limited
number of high-fidelity sensors of $u$ and $f$ rather than exact expressions
for them. Similarly, we also assume that $k$ is an unknown constant. The
objective is then to predict $u$ and $f$ in the entire computational domain as
well as to identify $k$.

Similarly, we assume that we have lots of low-fidelity observations for $u$,
which are generated from $u_{L}=0.8u+0.2$ here. It is worth mentioning that
the low-fidelity data can be obtained from numerical simulations with a
guessed $f$ or measurements with lower accuracy, which are similar as the
assumption in Sec. 3.2.1. Therefore, we again consider two scenarios for the
low-fidelity data, i.e., (1) simulation data with no noise, and (2) low-
fidelity measurements with Gaussian noise
$\epsilon_{u_{L}}\thicksim\mathcal{N}(0,0.05^{2})$. In both cases, we assume
that the noise for the high-fidelity measurements is the same, i.e.,
$\epsilon_{u}\thicksim\mathcal{N}(0,0.01^{2})$, and
$\epsilon_{f}\thicksim\mathcal{N}(0,0.01^{2})$.

Figure 9:  2D inverse problem: (a) Predicted means and uncertainties for $u$
and $f$ using multi-fidelity data: $\epsilon_{L}\thicksim 0$, $\sigma=0.7$.
(b) Predicted means and uncertainties for $u$ and $f$ using multi-fidelity
data: $\epsilon_{L}\thicksim\mathcal{(}0,0.05^{2})$, $\sigma=2.8$. (c)
Predicted means and uncertainties for $u$ and $f$ using high-fidelity data:
$\sigma=2.3$. Magenta cross: high-fidelity training data for $u$, Magenta
circle: high-fidelity training data for $f$.

In these two cases, 6,000 randomly selected low-fidelity data are employed. In
addition, 20 high-fidelity measurements for $u$ at each boundary are used to
provide boundary conditions. Furthermore, we assume that 10 and 20 sensors for
$u$ and $f$ are available, which are randomly located inside the computational
domain. In the DNN for the low-fidelity data we employ 2 hidden layers with 40
neurons per layer, while for the BNN we use 1 hidden layer with 50 neurons.
The prior distribution for each hyperparameter is obtained by maximizing the
ELBO in the VI, and a standard normal distribution is again utilized as the
prior for $k$, which is the same as in Sec. 3.2.1. As before, we also test the
performance of the single-fidelity modeling, i.e., we employ the same BNN
using the high-fidelity data only.

The predicted means and uncertainties for $u$ and $f$ using the multi-fidelity
modelings are illustrated in Figs. 9-9. As shown, the computational errors for
$u$ and $f$ in both cases are mostly bounded by the two standard deviations.
Furthermore, we present the results from single-fidelity modeling in Fig. 9.
We see that the computational errors are about one order larger than the
results from multi-fidelity modelings. Finally, the predicted values of $k$
are displayed in Table 4; we observe that (1) the computational errors in both
multi-fidelity modelings are bounded by the one standard deviation, (2) the
computational errors increase with the noise scale in the low-fidelity data,
consistent with the predicted uncertainties, and (3) the results from multi-
fidelity modeling are more accurate than those from single-fidelity modeling.

|  MF, $\epsilon_{u_{L}}=0$ |  MF, $\epsilon_{u_{L}}\thicksim\mathcal{N}(0,0.05^{2})$ | SF
---|---|---|---
Mean | 0.981 | 1.058 | 2.287
Std | $3.02\times 10^{-2}$ | $5.43\times 10^{-2}$ | $1.915\times 10^{-1}$
Table 4:  2D inverse PDE problem: Predicted mean and standard deviation for
$k$, where $k=1$ is the exact solution. MF, $\epsilon_{u_{L}}=0$: multi-
fidelity modeling with $\epsilon_{u_{L}}=0$; MF,
$\epsilon_{u_{L}}\thicksim\mathcal{N}(0,0.05^{2})$: multi-fidelity modeling
with $\epsilon_{u_{L}}\thicksim\mathcal{N}(0,0.05^{2})$; SF: single-fidelity
modeling.

###  3.3 Active learning for reducing uncertainty

####  3.3.1 Function approximation

The low- and high-fidelity data are generated using the same functions as in
Sec. 3.1.1. In addition, we also consider that we have two types of low-
fidelity data, i.e., (1) simulation data with no noise, and (2) measurements
with large noise, $\epsilon_{L}\thicksim\mathcal{N}(0,0.05^{2})$. In both
cases, we again employ 100 uniformly distributed data at the low level for
training. As for the high-fidelity training samples, the noise is also assumed
to be Gaussian, i.e., $\epsilon_{H}\thicksim\mathcal{N}(0,0.01^{2})$. We
assume that we only have 10 random high-fidelity measurements at the
beginning. The architectures for the NNs are the same as used in Sec. 3.2.1,
i.e., 2 hidden layers with 20 neurons per layer for the low-fidelity DNN, and
1 hidden layer with 50 neurons for the BNN. Furthermore, the priors for the
aforementioned cases are $\sigma=1.2$ and 1.4, respectively.

Figure 10:  Active learning for function approximation. (a) Predicted means
and uncertainties for high-fidelity profiles using 10 initial high-fidelity
data with $\epsilon_{L}\thicksim 0$. (b) Predicted means and uncertainties for
high-fidelity profiles using 10 initial high-fidelity data with
$\epsilon_{L}\thicksim\mathcal{N}(0,0.05^{2})$. (c) Predicted means and
uncertainties for high-fidelity profiles with 12 high-fidelity data with
$\epsilon_{L}\thicksim 0$. (d) Predicted means and uncertainties for high-
fidelity profiles with 13 high-fidelity data with
$\epsilon_{L}\thicksim\mathcal{N}(0,0.05^{2})$. Blue circle: initial high-
fidelity training data, Blue cross: added high-fidelity observations.

The predicted high-fidelity profiles are illustrated in Figs. 10-10. As shown,
the predicted means agree well with the exact solution for $x<0.44$. However,
the discrepancy becomes larger for $x>0.44$ since we only have 2 training data
in this region; the predicted uncertainty is larger for $x>0.44$ than in the
region of $x<0.44$. To enhance the predicted accuracy, we can add more noisy
high-fidelity training data. In the present study, we employ the same strategy
to acquire more high-fidelity data as in [23], i.e., adding a new observation
$x^{*}$ for $u_{H}$ at the location where the posterior variance is maximized.
Finally, the iterations of adding high-fidelity measurements terminate as the
predicted variance is smaller than $0.05^{2}$. As shown in Figs. 10-10, the
predicted high-fidelity profiles are significantly improved after more high-
fidelity observations are added in both cases.

####  3.3.2 Inverse PDE problem

Here we employ the same one-dimensional diffusion-reaction system in Sec.
3.2.1 to demonstrate the concept of the active learning for inverse PDE
problem. We assume that we have 3 and 10 high-fidelity measurements for $u$
and $f$ as the initial training set, respectively. In addition, both
measurements are randomly distributed in $x\in[0,1]$, as shown in Fig. 11. The
measurement errors for $u$ and $f$ are also assumed to be Gaussian, i.e.,
$\epsilon_{u}\thicksim\mathcal{N}(0,0.01^{2})$ and
$\epsilon_{f}\thicksim\mathcal{N}(0,0.01^{2})$. We also assume that we are
able to obtain the low-fidelity solution for $u$ with a guessed $f$, which is
the same as the one used in Sec. 3.2.1, i.e., $u_{L}(x)=\sin(8\pi x)$. The
low-fidelity data here are assumed to be from simulations, which are noise
free. In addition, 100 uniformly distributed samples are employed as the low-
fidelity training data. The architectures for the neural networks utilized
here are the same as in Sec. 3.2.1. In addition, the standard deviation for
the prior distribution is set as $\sigma=1.4$.

We start with the initial training set to infer the reaction rate as well as
reconstruct the profiles of $u$ and $f$. As shown in Fig. 11 (first column),
neither the predicted $u$ or $f$ are satisfactory, which can be reflected by
the large predicted uncertainties. To reduce the uncertainty and enhance the
accuracy, we then keep adding more high-fidelity noisy samples for both $u$
and $f$ using the same strategy as in Sec. 3.3.1. As displayed in Figs. 11,
the discrepancy between the predicted $u$ and $f$ and the exact solutions
become smaller as more training samples are used.

To demonstrate the effect of the training samples on the prediction accuracy
quantitatively, we compute the error (i.e., $E$) for $u$ and $f$ as

$\displaystyle
E=\frac{1}{N}\sqrt{\sum^{N}_{i=1}|\psi_{i}-\tilde{\psi}_{i}|^{2}},$ (20)

where $N=1,000$ is the number of predictions, and $\psi$ ($\psi=u/f$) and
$\tilde{\psi}$ represent the exact and predicted results, respectively. As
observed in Fig. 11, the computational error decreases with the increasing
number of iterations. The prediction accuracy for $k$ also improves as the
predicted $u$ and $f$ become more accurate as expected (Fig. 11).

Figure 11:  Active learning for 1D inverse diffusion-reaction system. (a)
Predicted $u$ and $f$ for $N=1$, 2, and 5. Blue circle: initial training data;
Blue cross: added training points based on the uncertainty. (b) Errors for $u$
and $f$ for different iterations. (c) Predicted $k$ at different iterations.
$N$ represents the number of iterations. Red triangle: predicted mean; error
bar: $\pm 2\sigma$ (i.e., two standard deviations).

##  4 Summary

We presented a multi-fidelity Bayesian neural network (MBNN), which is capable
of assimilating a large set of low-fidelity data and scarce high-fidelity data
for function approximation as well as for inverse PDE problems and enhancing
the prediction accuracy. In particular, MBNN consists of three neural
networks: the first one is a deep neural network (DNN) to fit the low-fidelity
data, while the second one is a Bayesian neural network (BNN) for capturing
the correlation between the low- and high-fidelity data, and the last one is
used to encode the physical laws. The first DNN is trained using the maximum a
posterior probability (MAP), which is able to handle big data. As for the BNN,
we first utilize the mean-field variational inference (VI) to obtain
informative prior distributions for the hyperparameters, and then we employ
the Hamiltonian Monte Carlo method to sample from the posterior distributions.
The low-fidelity data used in the present work can be either simulation data
or noisy measurements with low accuracy. In addition, the high-fidelity data
are noisy measurements with higher accuracy.

We demonstrated two major advantages of the proposed multi-fidelity BNN: (1)
The present approach is capable of capturing both linear and nonlinear
correlations between the low- and high-fidelity data adaptively; and (2) it
can quantify both aleatoric uncertainty associated with noisy data and
epistemic uncertainty associated with unknown parameters.

The multi-fidelity BNN is first applied for function approximations, including
one- and four-dimensional functions, and a two-dimensional sea surface
temperature case. The results show that the present method is capable of
capturing both linear and nonlinear correlation between the low- and high-
fidelity data adaptively, and can provide reasonable uncertainties in
predictions. We further employ the multi-fidelity BNN to identify the unknown
parameters in both one- and two-dimensional diffusion-reaction systems given a
small set of noisy high-fidelity measurements. The results demonstrated that
the present method can estimate the unknown parameter as well as reconstruct
the solution with high accuracy. In addition, the multi-fidelity BNN
outperforms the single-fidelity modeling in all test cases. We then performed
active learning for both a function approximation case as well as an inverse
PDE problem, i.e., adding more high-fidelity training data at the location
where the uncertainty is maximum. The results indicated that the active
learning approach can reduce the uncertainties efficiently and effectively.
Finally, we would like to point out that the present method can be easily
extended to big data problems since (1) the mean-field VI used to provide
priors can handle big data using the minibatch training, and (2) the
stochastic Hamiltonian Monte Carlo [24, 25, 26] can be used to replace the
Hamiltonian Monte Carlo of the present work to enable the minibatch training
for posterior sampling.

## Acknowledgement

This work was supported by the PhILMS grant DE-SC0019453, OSD/AFOSR MURI grant
FA9550-20-1-0358, and the NIH grant U01 HL142518.

## References

  * [1] B. Peherstorfer, K. Willcox, and M. Gunzburger.  Survey of multifidelity methods in uncertainty propagation, inference, and optimization.  SIAM Review, 60(3):550–591, 2019/04/26 2018.
  * [2] M. G. Fernández-Godino, C. Y. Park, N. H. Kim, and R. T Haftka.  Review of multi-fidelity models.  arXiv preprint arXiv:1609.07196, 2016.
  * [3] A. I. J. Forrester, A. Sóbester, and A. J. Keane.  Multi-fidelity optimization via surrogate modelling.  Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 463(2088):3251–3269, 2007.
  * [4] H. Babaee, S. Acharya, and X. Wan.  Optimization of forcing parameters of film cooling effectiveness.  Journal of Turbomachinery, 136(6):061016–061016, 11 2013.
  * [5] P. Perdikaris, D. Venturi, J. O. Royset, and G. E. Karniadakis.  Multi-fidelity modelling via recursive co-kriging and gaussian–markov random fields, 2015.
  * [6] P. Perdikaris and G. E. Karniadakis.  Model inversion via multi-fidelity bayesian optimization: a new paradigm for parameter estimation in haemodynamics, and beyond.  Journal of The Royal Society Interface, 13(118), 2016.
  * [7] M. C. Kennedy and A. O’Hagan.  Predicting the output from a complex computer code when fast approximations are available.  Biometrika, 87(1):1–13, 2000.
  * [8] H. Babaee, P. Perdikaris, C. Chryssostomidis, and G. E. Karniadakis.  Multi-fidelity modelling of mixed convection based on experimental correlations and numerical simulations.  Journal of Fluid Mechanics, 809:895–917, 12 2016.
  * [9] H. Babaee, C. Bastidas, M. DeFilippo, C. Chryssostomidis, and G. E. Karniadakis.  A multi-fidelity framework and uncertainty quantification for sea surface temperature in the massachusetts and cape cod bays.  Earth and Space Science, n/a(n/a):e458, 2020/02/17 2020.
  * [10] R. Martinez-Cantin, N. de Freitas, E. Brochu, J. Castellanos, and A. Doucet.  A bayesian exploration-exploitation approach for optimal online sensing and planning with a visually guided mobile robot.  Autonomous Robots, 27(2):93–103, 2009.
  * [11] P. Perdikaris, M. Raissi, A. Damianou, N. D. Lawrence, and G. E. Karniadakis.  Nonlinear information fusion algorithms for data-efficient multi-fidelity modelling.  Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 473(2198), 2017.
  * [12] X. H. Meng and G. E. Karniadakis.  A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems.  Journal of Computational Physics, 401:109020, 2020.
  * [13] M. Raissi, P. Perdikaris, and G. E. Karniadakis.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational Physics, 378:686–707, 2019.
  * [14] L. Yang, X. H. Meng, and G. E. Karniadakis.  B-PINNs: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data.  arXiv preprint arXiv:2003.06097, 2020.
  * [15] R. M Neal.  Bayesian learning for neural networks, volume 118.  Springer Science & Business Media, 2012.
  * [16] R. M Neal et al.  MCMC using Hamiltonian dynamics.  Handbook of Markov chain Monte Carlo, 2(11):2, 2011.
  * [17] J. Yao, W. W. Pan, S. Ghosh, and F. Doshi-Velez.  Quality of uncertainty quantification for Bayesian neural network inference.  arXiv preprint arXiv:1906.09686, 2019.
  * [18] J. Lee, Y. Bahri, R. Novak, et al.  Deep neural networks as Gaussian processes.  arXiv preprint arXiv:1711.00165, 2017.
  * [19] G. F. Pang, L. Yang, and G. E. Karniadakis.  Neural-net-induced Gaussian process regression for function approximation and PDE solution.  Journal of Computational Physics, 384:270–288, 2019.
  * [20] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra.  Weight uncertainty in neural networks.  arXiv preprint arXiv:1505.05424, 2015.
  * [21] P. J. Werdell, B. A. Franz, S. W. Bailey, et al.  Generalized ocean color inversion model for retrieving marine inherent optical properties.  Applied optics, 52(10):2019–2037, 2013.
  * [22] C. Werme and C. Hunt.  Outfall monitoring overview: Massachusetts water resources authority.  Report ENQUAD, 13:97, 2004.
  * [23] M. Raissi, P. Perdikaris, and G. E. Karniadakis.  Inferring solutions of differential equations using noisy multi-fidelity data.  Journal of Computational Physics, 335:736–746, 2017.
  * [24] T. Chen, E. Fox, and C. Guestrin.  Stochastic gradient Hamiltonian Monte Carlo.  In International Conference on Machine Learning, pages 1683–1691, 2014.
  * [25] N. Ding, Y. H. Fang, R. Babbush, et al.  Bayesian sampling using stochastic gradient thermostats.  In Advances in Neural Information Processing Systems, pages 3203–3211, 2014.
  * [26] Y. A. Ma, T. Q. Chen, and E. Fox.  A complete recipe for stochastic gradient MCMC.  In Advances in Neural Information Processing Systems, pages 2917–2925, 2015.


# Enhanced Hallucination Detection in Neural Machine Translation through
Simple Detector Aggregation

Anas Himmi1 Guillaume Staerman2 Marine Picot3
Pierre Colombo1,4 Nuno M. Guerreiro1,5,6,7
1MICS, CentraleSupélec, Universite Paris-Saclay, Paris, France,
2Universite Paris-Saclay, Inria, CEA, Palaiseau, France,
3digeiz, Paris, France, 4Equall, Paris, France, 5Instituto de
Telecomunicações, Lisbon, Portugal
6Unbabel, Lisbon, Portugal, 7Instuto Superior Técnico, University of Lisbon,
Portugal

###### Abstract

Hallucinated translations pose significant threats and safety concerns when it
comes to practical deployment of machine translation systems. Previous
research works have identified that detectors exhibit complementary
performance — different detectors excel at detecting different types of
hallucinations. In this paper, we propose to address the limitations of
individual detectors by combining them and introducing a straightforward
method for aggregating multiple detectors. Our results demonstrate the
efficacy of our aggregated detector, providing a promising step towards
evermore reliable machine translation systems.

##  1 Introduction

Neural Machine Translation (NMT) has become the dominant methodology for real-
world machine translation applications and production systems. As these
systems are deployed in-the-wild for real-world usage, it is ever more
important to ensure that they are highly reliable. While NMT systems are known
to suffer from various pathologies (Koehn and Knowles, 2017), the most severe
among them is the generation of translations that are detached from the source
content, typically known as hallucinations (Raunak et al., 2021; Guerreiro et
al., 2022b). Although rare, particularly in high-resource settings, these
translations can have dramatic impact on user trust Perez et al. (2022). As
such, researchers have worked on (i) methods to reduce hallucinations either
during training-time or even inference time (Xiao and Wang, 2021; Guerreiro et
al., 2022b; Dale et al., 2022; Sennrich et al., 2024), and alternatively, (ii)
the development of highly effective on-the-fly hallucination detectors
(Guerreiro et al., 2022b, a; Dale et al., 2022) to flag these translations
before they reach end-users. In this paper, we will focus on the latter.

One immediate way to approach the problem of hallucination detection is to
explore high-quality external models that can serve as proxies to measure
detachment from the source content, e.g., quality estimation (QE) models such
as CometKiwi (Rei et al., 2022), or cross-lingual sentence similarity models
like LASER (Artetxe and Schwenk, 2019) and LaBSE (Feng et al., 2022).
Intuitively, extremely low-quality translations or translations that are very
dissimilar from the source are more likely to be hallucinations. And, indeed,
these detectors can perform very effectively as hallucination detectors
(Guerreiro et al., 2022b; Dale et al., 2022). Alternatively, another effective
approach is to leverage internal model features such as attention maps and
sequence log-probability (Guerreiro et al., 2022b, a; Dale et al., 2022). The
assumption here is that when translation models generate hallucinations, they
may reveal anomalous internal patterns that can be highly predictive and
useful for detection, e.g., lack of contribution from the source sentence
tokens to the generation of the translation (Ferrando et al., 2022). Most
importantly, different detectors exhibit complementary properties. For
instance, oscillatory hallucinations — translations with anomalous repetitions
of phrases or $n$-grams (Raunak et al., 2021) — are readily identified by
CometKiwi, while detectors based on low source contribution or sentence
dissimilarity struggle in this regard. Therefore, there is an inherent trade-
off stemming from the diverse anomalies different detectors excel at.

In this paper, we address this trade-off by proposing a simple yet highly
effective method to aggregate different detectors to leverage their
complementary strengths. Through experimentation in the two most widely used
hallucination detection benchmarks, we show that our method consistently
improves detection performance.

Our key contributions are can be summarized as follows:

  * •

We propose STARE, an unsupervised Simple deTectors AggREgation method that
achieves state-of-the-art performance well on two hallucination detection
benchmarks.

  * •

We demonstrate that our consolidated detector can outperform single-based
detectors with as much as aggregating two complementary detectors.
Interestingly, our results suggest that internal detectors, which typically
lag behind external detectors, can be combined in such a way that they
outperform the latter.

We release our code and scores to support future research and ensure
reproducibility.111Code is available here:
https://github.com/AnasHimmi/Hallucination-Detection-Score-Aggregation.

##  2 Detectors Aggregation Method

###  2.1 Problem Statement

#### Preliminaries.

Consider a vocabulary $\Omega$ and let $(X,Y)$ be a random variable taking
values in $\mathcal{X}\times\mathcal{Y}$, where $\mathcal{X}\subseteq\Omega$
represents translations and $\mathcal{Y}=\\{0,1\\}$ denotes labels indicating
whether a translation is a hallucination ($Y=1$) or not ($Y=0$). The joint
probability distribution of $(X,Y)$ is $P_{XY}$.

#### Hallucination detection.

The goal of hallucination detection is to classify a given translation $x\in
X$ as either an expected translation from the distribution $P_{X|Y=0}$ or as a
hallucination from $P_{X|Y=1}$. This classification is achieved by a binary
decision function $g:X\rightarrow{0,1}$, which applies a threshold
$\gamma\in\mathbb{R}$ to a hallucination score function
$s:X\rightarrow\mathbb{R}$. The decision function is defined as:

$g(x)=\left\\{\begin{array}[]{ll}1&\text{if }s(x)>\gamma,\\\
0&\text{otherwise}.\end{array}\right.$

The objective is to create an hallucination score function $s$ that
effectively distinguishes hallucinated translations from other translations.

#### Aggregation.

Assume that we have several hallucination score detectors222We use the
notation $\\{s_{k}\\}_{k=1}^{K}$ to represent a set consisting of $K$
hallucination detectors, where each $s_{k}$ is a function mapping from
$\mathcal{X}$ to $\mathbb{R}$.. When evaluating a specific translation
$x^{\prime}$, our goal is to combine the scores from the single detectors into
a single, more reliable score that outperforms any of the individual detectors
alone. Formally, this aggregation method, denoted as $\operatorname{Agg}$, is
defined as follows:

$\displaystyle\operatorname{Agg}:\quad\quad\quad\quad\quad\mathbb{R}^{K}$
$\displaystyle\rightarrow\mathbb{R}$
$\displaystyle\\{s_{k}(x^{\prime})\\}_{k=1}^{K}$
$\displaystyle\rightarrow\operatorname{Agg}\bigg{(}\\{s_{k}\\}_{k=1}^{K}\bigg{)}.$

###  2.2 Proposed Aggregation Method

We start with the assumption that we have access to $K$ hallucination scores
and aim to construct an improved hallucination detector using these scores.
The primary challenge in aggregating these scores arises from the fact that
they are generated in an unconstrained setting, meaning that each score may be
measured on a different scale. Consequently, the initial step is to devise a
method for standardizing these scores to enable their aggregation. The
standardization weights we propose, $w_{k}$, are specific to each detection
score. Using the min-max normalization, they are designed based on the whole
training dataset $\mathcal{D}_{n}=\\{x_{1},\ldots,x_{n}\\}$. Formally:

$w_{k}=\frac{s_{k}(x^{\prime})-\underset{z\in\mathcal{D}_{n}}{\mathrm{min}}~{}s_{k}(z)}{\underset{z\in\mathcal{D}_{n}}{\mathrm{max}}~{}s_{k}(z)-\underset{z\in\mathcal{D}_{n}}{\mathrm{min}}~{}s_{k}(z)}.$

Given these weights, we build a hallucination detector based on a weighted
averaged of the score $s_{k}$ relying upon the previous “normalization
weights”:

$\operatorname{Agg}(x^{\prime})=\sum_{k=1}^{K}w_{k}s_{k}(x^{\prime}).$ (1)

We denote this method as STARE.

Detector | AUROC $\uparrow$ | FPR $\downarrow$
---|---|---
Individual Detectors
External
COMET-QE | 70.15 0000 | 57.24 0000
CometKiwi | 86.96 0000 | 35.15 0000
LaBSE |  91.72 |  26.86
1-3[.4pt/2pt] Model-based |  |
Seq-Logprob | 83.40 0000 | 58.99 0000
ALTI+ | 84.24 0000 | 66.19 0000
Wass-Combo |  87.02 0000 |  48.38 0000
Aggregated Detectors
External Only (gap to best single External)
Isolation Forest | 92.61 $\uparrow$0.89 | 19.08 $\downarrow$7.78
Max-Norm | 92.43 $\uparrow$0.71 | 22.09 $\downarrow$4.77
STARE | 93.32 $\uparrow$1.60 | 20.67 $\downarrow$6.19
1-3[.4pt/2pt] Model-based Only (gap to best single Model-based) |  |
Isolation Forest | 88.19 $\uparrow$1.17 | 36.63 $\downarrow$11.8
Max-Norm | 83.81 $\downarrow$3.21 | 62.94 $\uparrow$14.6
STARE | 89.07 $\uparrow$2.05 | 42.50 $\downarrow$5.88
1-3[.4pt/2pt] All (gap to best overall) |  |
Isolation Forest | 92.84 $\uparrow$1.12 | 23.90 $\downarrow$2.96
Max-Norm | 91.60 $\downarrow$0.12 | 26.38 $\downarrow$0.48
STARE |  94.12 $\uparrow$2.40 |  17.06 $\downarrow$9.80

(a) Results on LfaN-Hall.

Detector | AUROC $\uparrow$ | FPR $\downarrow$
---|---|---
Individual Detectors
External
COMET-QE | 82.22 0000 | 47.40 0000
LASER | 81.11 0000 | 47.04 0000
XNLI | 82.44 0000 |  33.20 0000
LaBSE |  88.77 | 34.96
1-3[.4pt/2pt] Model-based |  |
Seq-Logprob |  86.72 0000 |  28.86 0000
ALTI+ | 82.26 0000 | 58.40 0000
Wass-Combo | 64.82 0000 | 84.62 0000
Aggregation Detectors
External Only (gap to best single External)
Isolation Forest | 71.35 $\downarrow$17.4 | 57.75 $\uparrow$22.8
Max-Norm | 88.57 $\uparrow$0.48 | 32.59 $\downarrow$2.86
STARE | 89.76 $\uparrow$0.99 | 32.74 $\downarrow$2.22
1-3[.4pt/2pt] Model-based Only (gap to best single Model-based) |  |
Isolation Forest | 75.35 $\downarrow$11.4 | 69.71 $\uparrow$40.9
Max-Norm | 67.70 $\downarrow$17.3 | 83.83 $\uparrow$53.1
STARE | 89.92 $\uparrow$3.20 | 30.37 $\uparrow$1.51
1-3[.4pt/2pt] All (gap to best overall) |  |
Isolation Forest | 76.25 $\downarrow$12.5 | 56.28 $\uparrow$21.3
Max-Norm | 80.67 $\downarrow$7.01 | 41.52 $\uparrow$1.91
STARE |  91.18 $\uparrow$2.41 |  28.85 $\downarrow$6.11

(b) Results on HalOmi. (c) Performance, according to AUROC and FPR, of all
single detectors available and aggregation methods via combination of external
detectors, model-based detectors, or both simultaneously. We represent with
the best overall single detector and underline the best detectors for each
class, according to our primary metric AUROC.

##  3 Experimental Setup

###  3.1 Datasets

In our experiments, we utilize the human-annotated datasets released in
Guerreiro et al. (2022b) and Dale et al. (2023). Both datasets include
detection scores — both for internal and external detectors — for each
individual translation:

####  LfaN-Hall.

A dataset of 3415 translations for WMT18 German$\rightarrow$English news
translation data (Bojar et al., 2018) with annotations on critical errors and
hallucinations (Guerreiro et al., 2022b). This dataset contains a mixture of
oscillatory hallucinations and fluent but detached hallucinations. We provide
examples of such translations in Appendix A. For each translation, there are
six different detector scores: three are from external models (scores from
COMET-QE and CometKiwi, two quality estimation models, and sentence similarity
from LaBSE, a cross-lingual embedding model), and three are from internal
methods (length-normalized sequence log-probability, Seq-Logprob; contribution
of the source sentence for the generated translation according to ALTI+
(Ferrando et al., 2022), and Wass-Combo, an Optimal Transport inspired method
that relies on the aggregation of attention maps).

####  HalOmi.

A dataset with human-annotated hallucination in various translation
directions. We test translations into and out of English, pairing English with
five other languages — Arabic, German, Russian, Spanish, and Chinese,
consisting of over 3000 sentences across the ten different language pairs.
Importantly, this dataset has two important properties that differ from LfaN-
Hall: (i) it has a much bigger proportion of fluent but detached
hallucinations (oscillatory hallucinations were not considered as a separate
category), and (ii) nearly 35$\%$ of the translations are deemed
hallucinations, as opposed to about 8$\%$ for LfaN-Hall.333Given the rarity of
hallucinations in practical translation scenarios (Guerreiro et al., 2023),
LfaN-Hall offers a more realistic simulation of detection performance. For
each translation, there are seven different detection scores: the same
internal detection scores as LfaN-Hall, and four different detector scores:
COMET-QE, LASER, XNLI and LaBSE.

We provide more details on both datasets in Appendix A.

#### Aggregation Baselines.

The closest related work is Darrin et al. (2023b) on out-of-distribution
detection methods, using an Isolation Forest (IF; Liu et al., 2008) for per-
class anomaly scores. We adapt their method, employing a single Isolation
Forest, and designate it as our baseline. Alternatively, we also consider a
different way to use the individual scores and normalization weights in
Equation 1: instead of performing a sum over the weighted scores, we take the
maximum score. We denote this baseline as Max-Norm.

#### Evaluation method.

Following Guerreiro et al. (2022a), we report Area Under the Receiver
Operating Characteristic curve (AUROC) as our primary metric, and False
Positive Rate at 90% True Positive Rate (FPR@90TPR) as a secondary metric.

#### Implementation details.

For LfaN-Hall, we normalize the metrics by leveraging the held-out set
released with the dataset consisting of 100,000 non-annotated in-domain
scores. In the case of HalOmi, however, no held-out set was released. As such,
we rely on sampling random splits that consist of 10% of the dataset for
calibration. We repeat the process 10 different times. We report average
scores over those different runs. We also report the performance variance in
the Appendix.

###  3.2 Performances Analysis

Results on hallucination detection performance on LfaN-Hall and HaloMNI are
reported in Section 2.2.

#### Global Analysis.

STARE aggregation method consistently outperforms (i) single detectors’
performance, and (ii) other aggregation baselines. Moreover, we find that the
combination of all detectors — both model-based and external-based detectors —
yields the best overall results, improving over the STARE method based on
either internal or external models only. Importantly, these trends, contrary
to other alternative aggregation strategies, hold across both datasets.

#### Aggregation of External Detectors.

STARE demonstrates robust performance when aggregating external detectors on
both LfaN-Hall and HalOmi: improvements in AUROC (over a point) and in FPR
(between two to six points). Interestingly, we also observe that the best
overall performance obtained exclusively with external models lags behind that
of the overall aggregation. This suggests that internal models features —
directly obtained via the generation process — contribute with complementary
information to that captured by external models.

#### Aggregation of Internal Detectors.

Aggregation of internal detectors, can achieve higher AUROC scores than the
best single external detector on HalOmi. This results highlights how model-
based features — such as attention and sequence log-probability — that are
readily and efficiently obtained as a by-product of the generation can, when
aggregated effectively, outperform more computationally expensive external
solutions.

###  3.3 Ablation Studies

In this section, our focus is two-fold: (i) exploring optimal selections of
detectors, and (ii) understanding the relevance of the reference set’s size.

#### Optimal Choice of detectors.

We report the performance of the optimal combination of $N$-detectors on both
datasets in Section 3.3.444We report the optimal combinations in Appendix C.
We note that including all detectors yields comparable performance to the best
mix of detectors. Interestingly, aggregation always brings improvement, even
when only combining two detectors. As expected, the best mixture of detectors
leverages information from different signals: contribution of source
contribution, low-quality translations, and dissimilarity between source and
translation.

| LfaN-Hall | HalOmi
---|---|---
1-5[.4pt/2pt] $N$ | AUROC | FPR@90 | AUROC | FPR@90
LaBSE | 91.72 | 26.86 | 88.77 | 34.96
1-5[.4pt/2pt] 2 | 93.32 | 20.67 | 90.40 | 27.52
3 | 94.11 | 17.27 | 90.61 | 27.24
4 | 94.45 | 13.69 | 91.09 | 26.91
5 | 94.12 | 17.06 | 91.25 | 28.48
6 | — | — | 91.40 | 27.93
1-5[.4pt/2pt] STARE | 94.12 | 17.06 | 91.18 | 28.85

Table 1: Ablation Study on the Optimal Choice of Detectors when using STARE.

#### Impact of the size of the references set.

The calibration of scores relies on a reference set. Here, we examine the
impact of the calibration set size on performance, by ablating on the held-out
set LfaN-Hall, which comprises of 100k sentences. Figure 1 shows that the
Isolation Forest requires a larger calibration set to achieve similar
performance. This phenomenon might explain the drop in performance observed on
HalOmi (Table 2.2). Interestingly, the performance improvement for STARE,
particularly in FPR, plateaus when the reference set exceeds 1,000 samples,
which suggests that STARE can adapt to different domains with a rather small
reference set.

Figure 1: Impact of reference set size on LfaN-Hall.

##  4 Conclusion & Future Perspectives

We propose a simple aggregation method to combine hallucination detectors to
exploit complementary benefits from each individual detector. We show that our
method can bring consistent improvements over previous detection approaches in
two human-annotated datasets across different language pairs. We are also
releasing our code and detection scores to support future research on this
topic.

##  5 Limitations

Our methods are evaluated in a limited setup due to the limited availability
of translation datasets with annotation of hallucinations. Moreover, in this
study, we have not yet studied compute-optimal aggregation of detectors — we
assume that we already have access to multiple different detection scores.

##  6 Acknowledgements

Training compute is obtained on the Jean Zay supercomputer operated by GENCI
IDRIS through compute grant 2023-AD011014668R1, AD010614770 as well as on
Adastra through project c1615122, cad15031, cad14770 .

## References

  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.  _Transactions of the Association for Computational Linguistics_ , 7:597–610.
  * Bojar et al. (2018) Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018.  Findings of the 2018 conference on machine translation (WMT18).  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 272–303, Belgium, Brussels. Association for Computational Linguistics.
  * Colombo et al. (2022) Pierre Colombo, Eduardo Dadalto Câmara Gomes, Guillaume Staerman, Nathan Noiry, and Pablo Piantanida. 2022.  Beyond mahalanobis distance for textual ood detection.  In _NeurIPS 2022_.
  * Colombo et al. (2023) Pierre Colombo, Marine Picot, Nathan Noiry, Guillaume Staerman, and Pablo Piantanida. 2023.  Toward stronger textual attack detectors.  _Findings EMNLP 2023_.
  * Dale et al. (2022) David Dale, Elena Voita, Loïc Barrault, and Marta R Costa-jussà. 2022.  Detecting and mitigating hallucinations in machine translation: Model internal workings alone do well, sentence similarity even better.  _arXiv preprint arXiv:2212.08597_.
  * Dale et al. (2023) David Dale, Elena Voita, Janice Lam, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Loïc Barrault, and Marta R Costa-jussà. 2023.  Halomi: A manually annotated benchmark for multilingual hallucination and omission detection in machine translation.  _arXiv preprint arXiv:2305.11746_.
  * Darrin et al. (2023a) Maxime Darrin, Pablo Piantanida, and Pierre Colombo. 2023a.  Rainproof: An umbrella to shield text generators from out-of-distribution data.  _EMNLP 2023_.
  * Darrin et al. (2023b) Maxime Darrin, Guillaume Staerman, Eduardo Dadalto Câmara Gomes, Jackie CK Cheung, Pablo Piantanida, and Pierre Colombo. 2023b.  Unsupervised layer-wise score aggregation for textual ood detection.  _arXiv preprint arXiv:2302.09852_.
  * Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022.  Language-agnostic BERT sentence embedding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 878–891, Dublin, Ireland. Association for Computational Linguistics.
  * Ferrando et al. (2022) Javier Ferrando, Gerard I. Gállego, Belen Alastruey, Carlos Escolano, and Marta R. Costa-jussà. 2022.  Towards opening the black box of neural machine translation: Source and target interpretations of the transformer.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8756–8769, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Guerreiro et al. (2023) Nuno M. Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023.  Hallucinations in Large Multilingual Translation Models.  _Transactions of the Association for Computational Linguistics_ , 11:1500–1517.
  * Guerreiro et al. (2022a) Nuno M Guerreiro, Pierre Colombo, Pablo Piantanida, and André FT Martins. 2022a.  Optimal transport for unsupervised hallucination detection in neural machine translation.  _arXiv preprint arXiv:2212.09631_.
  * Guerreiro et al. (2022b) Nuno M Guerreiro, Elena Voita, and André FT Martins. 2022b.  Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation.  _arXiv preprint arXiv:2208.05309_.
  * Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017.  Six challenges for neural machine translation.  In _Proceedings of the First Workshop on Neural Machine Translation_ , pages 28–39, Vancouver. Association for Computational Linguistics.
  * Liu et al. (2008) Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2008.  Isolation forest.  In _2008 eighth ieee international conference on data mining_ , pages 413–422. IEEE.
  * NLLB Team et al. (2022) NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022.  No language left behind: Scaling human-centered machine translation.
  * Perez et al. (2022) Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022.  Red teaming language models with language models.  _arXiv preprint arXiv:2202.03286_.
  * Picot et al. (2023a) Marine Picot, Federica Granese, Guillaume Staerman, Marco Romanelli, Francisco Messina, Pablo Piantanida, and Pierre Colombo. 2023a.  A halfspace-mass depth-based method for adversarial attack detection.  _TMLR 2023_.
  * Picot et al. (2023b) Marine Picot, Nathan Noiry, Pablo Piantanida, and Pierre Colombo. 2023b.  Adversarial attack detection under realistic constraints.
  * Picot et al. (2023c) Marine Picot, Guillaume Staerman, Federica Granese, Nathan Noiry, Francisco Messina, Pablo Piantanida, and Pierre Colombo. 2023c.  A simple unsupervised data depth-based method to detect adversarial images.
  * Raunak et al. (2021) Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021.  The curious case of hallucinations in neural machine translation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1172–1183, Online. Association for Computational Linguistics.
  * Rei et al. (2022) Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022.  CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Sennrich et al. (2024) Rico Sennrich, Jannis Vamvas, and Alireza Mohammadshahi. 2024.  Mitigating hallucinations and off-target machine translation with source-contrastive and language-contrastive decoding.
  * Staerman et al. (2021) Guillaume Staerman, Pavlo Mozharovskyi, Pierre Colombo, Stéphan Clémençon, and Florence d’Alché Buc. 2021.  A pseudo-metric between probability distributions based on depth-trimmed regions.  _TMLR 2024_ , pages arXiv–2103.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Xiao and Wang (2021) Yijun Xiao and William Yang Wang. 2021.  On hallucination and predictive uncertainty in conditional language generation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2734–2744, Online. Association for Computational Linguistics.

##  Appendix A Model and Data Details

###  A.1 LfaN-Hall dataset

#### NMT Model.

The model used in Guerreiro et al. (2022b) is a Transformer base model Vaswani
et al. (2017) (hidden size of 512, feedforward size of 2048, 6 encoder and 6
decoder layers, 8 attention heads). The model has approximately 77M
parameters. It was trained on WMT18 de-en data: the authors randomly choose
2/3 of the dataset for training and use the remaining 1/3 as a held-out set
for analysis. We use a section of that same held-out set in this work.

#### Dataset Stats.

The dataset consists of 3415 translations from WMT18 de-en data. Overall,
there are 218 translations annotated as detached hallucinations (fully and
strongly detached — see more details in Guerreiro et al. (2022b)), and 86 as
oscillatory hallucinations.555Some strongly detached hallucinations have also
been annotated as oscillatory hallucinations. In these cases, we follow
Guerreiro et al. (2022a) and consider them to be oscillatory. The other
translations are either incorrect (1073) or correct (2048). We show examples
of hallucinations for each category in Table 3.666All data used in this paper
is licensed under a MIT License.

Detector | AUROC $\uparrow$ | FPR@90TPR $\downarrow$
---|---|---
Individual Detectors
External
COMET-QE | 82.22 $\pm$ 0.28 | 47.40 $\pm$ 0.82
LASER | 81.11 $\pm$ 0.21 | 47.04 $\pm$ 0.78
XNLI | 82.44 $\pm$ 0.18 | 33.20 $\pm$ 0.63
LaBSE | 88.77 $\pm$ 0.21 | 34.96 $\pm$ 0.72
1-3[.4pt/2pt] Model-based |  |
Seq-Logprob | 86.72 $\pm$ 0.22 | 28.86 $\pm$ 0.64
ALTI+ | 82.26 $\pm$ 0.28 | 58.40 $\pm$ 0.54
Wass-Combo | 64.82 $\pm$ 0.20 | 84.62 $\pm$ 0.52
Aggregated Detectors
External Only
Isolation Forest | 71.35 $\pm$ 1.62 | 57.75 $\pm$ 4.55
Max-Norm | 88.57 $\pm$ 0.38 | 32.59 $\pm$ 0.60
STARE | 89.76 $\pm$ 0.19 | 32.74 $\pm$ 0.50
1-3[.4pt/2pt] Model-based Only |  |
Isolation Forest | 75.35 $\pm$ 2.32 | 69.71 $\pm$ 5.01
Max-Norm | 67.70 $\pm$ 1.31 | 83.83 $\pm$ 1.40
STARE | 89.92 $\pm$ 0.20 | 30.37 $\pm$ 1.84
1-3[.4pt/2pt] All |  |
Isolation Forest | 76.25 $\pm$ 2.16 | 56.28 $\pm$ 6.29
Max-Norm | 80.67 $\pm$ 1.37 | 41.52 $\pm$ 5.87
STARE | 91.18 $\pm$ 0.20 | 28.85 $\pm$ 0.89

Table 2: Performance of individual and aggregated hallucination detectors on
the HalOmi dataset, including average performance and standard deviations
across ten different calibration sets.

Category |  Source Sentence |  Reference Translation |  Hallucination
---|---|---|---
Oscillatory |  Als Maß hierfür wird meist der sogenannte Pearl Index benutzt (so benannt nach einem Statistiker, der diese Berechnungsformel einführte). |  As a measure of this, the so-called Pearl Index is usually used (so named after a statistician who introduced this calculation formula). |  The term "Pearl Index" refers to the term "Pearl Index" (or "Pearl Index") used to refer to the term "Pearl Index" (or "Pearl Index").
Strongly Detached |  Fraktion der Grünen / Freie Europäische Allianz |  The Group of the Greens/European Free Alliance |  Independence and Democracy Group (includes 10 UKIP MEPs and one independent MEP from Ireland)
Fully Detached |  Die Zimmer beziehen, die Fenster mit Aussicht öffnen, tief durchatmen, staunen. |  Head up to the rooms, open up the windows and savour the view, breathe deeply, marvel. |  The staff were very friendly and helpful.
Table 3: Examples of hallucination types. Hallucinated content is shown
shaded.

###  A.2 HalOmi dataset

#### NMT model.

Translations on this dataset come from 600M distilled NLLB model (NLLB Team et
al., 2022).

##  Appendix B Variance of performance on the HalOmi dataset

We report in Table A.1 the average performance as well as the standard
deviation across the different ten runs on different calibration sets. Despite
variance between different runs, the STARE aggregation method consistently
outperforms individual detectors and other aggregation techniques.

##  Appendix C Optimal Combination of Detectors via STARE

####  LfaN-Hall.

The optimal set of detectors for various values of $N$ is:

  * •

for $N=1$: LaBSE

  * •

for $N=2$: CometKiwi, LaBSE

  * •

for $N=3$: Wass_Combo, CometKiwi, LaBSE

  * •

for $N=4$: ALTI+, Wass_Combo, CometKiwi, LaBSE

  * •

for $N=5$: ALTI+, SeqLogprob, Wass_Combo, CometKiwi, LaBSE

####  HalOmi.

The optimal set of detectors for various values of $N$ is:

  * •

for $N=2$: LaBSE, SeqLogprob

  * •

for $N=3$: LaBSE, SeqLogprob, Wass-Combo

  * •

for $N=4$: LaBSE, SeqLogprob, XNLI, COMET-QE

  * •

for $N=5$: LaBSE, SeqLogprob, XNLI, COMET-QE, ALTI+

  * •

for $N=6$: LaBSE, Log Loss, XNLI, COMET-QE, ALTI+, Wass-Combo

  * •

for $N=7$: LaBSE, SeqLogprob, XNLI, COMET-QE, ALTI+, Laser, Wass-Combo

##  Appendix D Future Work & Perspectives

In the future, we would like to explore more anomaly detection methods to
improve the aggregation quality. Specifically, we would like to test
Information Projections Picot et al. (2023c); Darrin et al. (2023a) and data
depths Picot et al. (2023b, a); Colombo et al. (2022); Staerman et al. (2021);
Colombo et al. (2023).