# Errors-in-Variables for deep learning:
rethinking aleatoric uncertainty

Jörg Martin$\ast$, Clemens Elster$\ast$

###### Abstract

We present a Bayesian treatment for deep regression using an Errors-in-
Variables model which accounts for the uncertainty associated with the input
to the employed neural network. It is shown how the treatment can be combined
with already existing approaches for uncertainty quantification that are based
on variational inference. Our approach yields a decomposition of the
predictive uncertainty into an aleatoric and epistemic part that is more
complete and, in many cases, more consistent from a statistical perspective.
We illustrate and discuss the approach along various toy and real world
examples.

##  1 Introduction

00footnotetext: Physikalisch-Technische Bundesanstalt, Abbestr. 2, 10587
Berlin

In this work we will focus on regression problems that describe the supervised
training of neural networks with normally distributed data. For motivational
purposes let us first consider the following, quite common, model that links
input data $x$ to outputs $y$ :

$\displaystyle y=f_{\theta}(x)+\varepsilon_{y}\,,$ (1)

where $f_{\theta}$ is a neural network with parameters $\theta$ and where
$\varepsilon_{y}\sim\mathcal{N}(0,\sigma_{y}^{2})$ is some normally
distributed noise111The index $y$ should not be misunderstood as
$\varepsilon_{y}$ or $\sigma_{y}$ being $y$ dependent, but is supposed to
indicate that the noise corresponds to the output. The same is true for
$\varepsilon_{x}$ and $\sigma_{x}$, which we introduce in (2) below. that
disturbs the true label (i.e., value of the regression function)
$f_{\theta}(x)$ corresponding to $x$. Learning the neural network means to
infer a value for $\theta$ from pairs $(x,y)$ contained in a training set
$\mathcal{D}$. The trained neural network can then be used to make
predictions. In real-world applications, especially in those where reliability
and safety are crucial [20, 19, 22], it is valuable, if not indispensible, to
know the uncertainty behind a prediction.

Uncertainty in machine learning is usually categorized by two different terms.
_Epistemic_ uncertainty arises from the uncertainty about the trained model,
that is about $\theta$ in the notation of (1). In a Bayesian approach, as in
this work, this uncertainty is described by the posterior
$\pi(\theta|\mathcal{D})$, which is the distribution of $\theta$ conditional
on the data $\mathcal{D}$ [1, 8, 16]. This type of uncertainty vanishes as the
number of observations tends to infinity, as follows for instance from the
Bernstein-von Mises theorem. _Aleatoric_ uncertainty, on the other hand,
describes an uncertainty that is inherent to the data and cannot be diminished
even with an infinite training set. In the context of regression this would
correspond to a noise such as $\varepsilon_{y}$ in (1) and might be measured,
for instance, by $\sigma_{y}$.

A neural network trained for the regression model (1) can be used to make a
prediction for the output $y^{*}$ given some (further) input $x^{*}$. In a
Bayesian treatment, this is done in terms of the posterior predictive
distribution. For example, mean and standard deviation of that distribution
can be taken as an estimate for $y^{*}$ and its associated uncertainty. Both
the epistemic and the aleatoric part of the uncertainty enter the posterior
predictive distribution and will affect the uncertainty quantification – the
epistemic part through the uncertainty in the regression function
$f_{\theta}(x^{*})$, and the aleatoric part due to random noise contained in
$y^{*}$.

In deep learning, the aleatoric uncertainty expressed through $\sigma_{y}$ is
often given a greater importance and is regarded as measuring inherent,
irreducible label ambiguity [5, 11, 13, 14]. However, such a point of view
does no longer apply in those regression problems where the goal is the
prediction of the regression function $f_{\theta}(x^{*})$ at some $x^{*}$. In
this case only the epistemic uncertainty about the network’s parameter
$\theta$ remains which, in principle, could become arbitrarily small as the
training set grows. Furthermore, typically $x^{*}=\zeta^{*}+\varepsilon_{x}$
contains some noise $\varepsilon_{x}$ as well, and one is actually interested
in the value of $f_{\theta}(\zeta^{\ast})$. The uncertainty caused by the fact
that $x^{*}$, and not $\zeta^{*}$, is observed constitutes an aleatoric part
of the uncertainty for the prediction of $f_{\theta}(\zeta^{*})$ which is
neither covered by $\sigma_{y}$ nor by the uncertainty of $\theta$.

We here attempt to give a more consistent view on these issues and, arguably,
a more accurate depiction. In many cases it is a too crude assumption to
presume that while $y$ is deranged by noise, $x$ is not. Standard estimation
procedures such as maximum likelihood or nonlinear least-squares become biased
for a regression model of the form (1) when $x$ is observed with noise [7].
Furthermore, the separate quantification of the aleatoric part of the
uncertainty due to the noise of the input is not possible in a model like (1)
since the variance $\sigma_{y}^{2}$ also accounts for noise in the output.
Following the idea of Errors-in-Variables (EiV), a quite classical concept in
statistics [7], we try to resolve these issues by changing (1) to

$\displaystyle\begin{aligned} x&=\zeta+\varepsilon_{x}\,,\qquad
y=f_{\theta}(\zeta)+\varepsilon_{y}\,,\end{aligned}$ (2)

where $\zeta$ denotes the true, but unknown input value. Besides allowing for
noisy input data, the model (2) enables a treatment of aleatoric uncertainty
that is more coherent from a statistical perspective. We will refer to (1)
instead as the “non-EiV” model throughout this work. An illustration of both
approaches can be found in the appendix.

The uncertainty that arises in the EiV model from estimating $\zeta$ given $x$
is an aleatoric uncertainty that has to be taken into account whenever
predicting $f_{\theta}(\zeta)$. It explains why there can be an uncertainty of
the prediction even if there is no epistemic uncertainty left, without forcing
this role on the output noise. We will approach (2) from a Bayesian point of
view which is a consistent way for describing uncertainty but poses the
challenge to sample from a high dimensional posterior on $\theta$. To overcome
this obstacle, we build on the idea to use variational inference, as e.g. in
[1, 8, 9, 16]. In fact our approach is agnostic to the specific form of the
variational distribution and could be used to extend any of these works. For
convenience the results in Section 3 below were produced with the variational
distribution induced by Monte Carlo dropout [8], but let us stress that there
is no profound reason for using this approach in this work other than its ease
of implementation. In fact, studying how well different methods perform under
EiV could be an exciting outlook for future work.

#### Existing work and structure of the article

Errors-in-Variables is a statistical concept that is around since decades
[10]. The usage of this concept for the learning of a neural networks was
proposed before in [23, 24]. In contrast to this article, those works consider
a non-Bayesian framework without the quantification of uncertainties. The idea
of fixing the ratio between $\sigma_{x}$ and $\sigma_{y}$ in EiV, which we
propose in Section 2.3 below, is also quite classical [7] and occasionally
named after Deming [4]. Variational inference is another well-established tool
from statistics that is behind much of the deep learning literature on
uncertainty quantification [1, 8, 9, 16, 6, 25]. A concept building on
variational inference that has some particular resemblance to this work is the
one of variational autoencoders [17]: think of $\zeta$ in (2) to be the
“latent variable”. However variational autoencoders differ in their philosophy
and their loss function and are therefore not directly comparable to our
approach, although combining both ideas might be a way to enhance the model in
(2). This article is structured as follows. In Section 2 we will discuss the
generic approach presented in this work, give a proposal for the underlying
priors and discuss some details useful for implementation, such as how to link
$\sigma_{x}$ and $\sigma_{y}$ via a Deming factor $\delta$ to ensure
identifiability and prevent overfitting. In Section 3 we discuss the results
of some experiments. The focus will be laid on toy models with known ground
truth to study the behavior of the EiV model in greater depth.

##  2 An EiV model for deep learning

In this section we will introduce an Errors-in-variables (EiV) model for deep
learning, discuss the concept of aleatoric and epistemic uncertainty in the
light of this approach and propose a few algorithmic details for training.

###  2.1 The generic model

Suppose we have $N$ datapoints
$\mathcal{D}=\\{(x_{1},y_{1}),\ldots,(x_{N},y_{N})\\}\subseteq\mathbb{R}^{n_{X}}\times\mathbb{R}^{n_{Y}}$
that we model by

$\displaystyle\begin{aligned} x_{i}=\zeta_{i}+\varepsilon_{x,i}\,,\qquad
y_{i}=f_{\theta}(\zeta_{i})+\varepsilon_{y,i}\end{aligned}$ (3)

with normally distributed
$\varepsilon_{x,i}\sim\mathcal{N}(0,\sigma_{x}^{2}I_{n_{X}\times
n_{X}}),\,\varepsilon_{y,i}\sim\mathcal{N}(0,\sigma_{y}^{2}I_{n_{Y}\times
n_{Y}})$ and where the parameters
$\theta\in\mathbb{R}^{p},\sigma_{x},\sigma_{y}>0$ and
$\zeta_{1},\ldots,\zeta_{N}\in\mathbb{R}^{n_{X}}$ are unknown. The function
$f_{\theta}$ will be a neural network. The $\zeta_{i}$ ought to be considered
as the true, but unknown, inputs we would like to feed to $f_{\theta}$. Given
$\boldsymbol{\zeta}=(\zeta_{1},\ldots,\zeta_{N}),\,\theta$ and
$\boldsymbol{\sigma}^{2}=(\sigma_{x}^{2},\sigma_{y}^{2})$, the likelihood for
the data $\mathcal{D}$ under (3) is

$\displaystyle
p(\mathcal{D}|\theta,\boldsymbol{\zeta},\boldsymbol{\sigma}^{2})=\prod_{i=1}^{N}p(x_{i}|\zeta_{i},\sigma_{x}^{2})\cdot
p(y_{i}|\theta,\zeta_{i},\sigma_{y}^{2})$ (4)

with
$p(x_{i}|\zeta_{i},\sigma_{x}^{2})=\mathcal{N}(x_{i}|\zeta_{i},\sigma_{x}^{2})$
and
$p(y_{i}|\theta,\zeta_{i},\sigma_{y}^{2})=\mathcal{N}(y_{i}|f_{\theta}(\zeta_{i}),\sigma_{y}^{2})$.
While $\theta$ will be considered as the parameter of interest, the
$\boldsymbol{\zeta}$ will be considered as nuisance parameters and the
variances in $\boldsymbol{\sigma}^{2}$ as hyperparameters that will be
finetuned, learned or chosen from experience. Fixing a prior
$\pi(\theta,\boldsymbol{\zeta})=\pi(\theta)\pi(\boldsymbol{\zeta})=\pi(\theta)\prod_{i=1}^{N}\pi(\zeta_{i})$
we have, by Bayes’ theorem,

$\displaystyle\pi(\theta|\mathcal{D},\boldsymbol{\sigma}^{2})=\frac{\pi(\theta)\pi(\mathcal{D}|\theta,\boldsymbol{\sigma}^{2})}{\pi(\mathcal{D}|\boldsymbol{\sigma}^{2})}\,,$
(5)

where
$\pi(\mathcal{D}|\boldsymbol{\sigma}^{2})=\int\mathrm{d}\theta\,\pi(\theta)\pi(\mathcal{D}|\theta,\boldsymbol{\sigma}^{2})$
and, due to (4) and once more Bayes’ theorem,

$\displaystyle\begin{aligned}
\pi(\mathcal{D}|\theta,\boldsymbol{\sigma}^{2})&=\int\mathrm{d}\boldsymbol{\zeta}\,\pi(\boldsymbol{\zeta})\,p(\mathcal{D}|\theta,\boldsymbol{\zeta},\boldsymbol{\sigma}^{2})=\prod_{i=1}^{N}\pi(x_{i}|\sigma_{x}^{2})\int\mathrm{d}\zeta_{i}\,\pi(\zeta_{i}|x_{i},\sigma_{x}^{2})p(y_{i}|\zeta_{i},\theta,\sigma_{y}^{2})\end{aligned}$
(6)

with
$\pi(x_{i}|\sigma_{x}^{2})=\int\mathrm{d}\zeta_{i}\pi(\zeta_{i})p(x_{i}|\zeta_{i},\sigma_{x}^{2})$.
As we do not expect $\pi(\theta|\mathcal{D},\boldsymbol{\sigma}^{2})$ to be
feasible we approximate it via a variational distribution

$\displaystyle\pi(\theta|\mathcal{D},\boldsymbol{\sigma}^{2})\approx
q_{\phi}(\theta)$ (7)

with variational parameter $\phi$. In variational inference the distance in
(7) is measured by the Kullback-Leibler divergence
$D_{\mathrm{KL}}(q_{\phi}(\theta)\|\pi(\theta|\mathcal{D},\boldsymbol{\sigma}^{2}))$.
Using that the divergence is non-negative, (5) and (6) we obtain the following
evidence lower bound (ELBO) for the data evidence
$\log\pi(\mathcal{D}|\boldsymbol{\sigma}^{2})$

$\displaystyle\begin{aligned}
\mathrm{ELBO}=\sum_{i=1}^{N}\int\mathrm{d}\theta\,q_{\phi}(\theta)\log\left(\int\mathrm{d}\zeta_{i}\pi(\zeta_{i}|x_{i},\sigma_{x}^{2})p(y_{i}|\theta,\zeta_{i},\sigma_{y}^{2})\right)\\\
-D_{\mathrm{KL}}(q_{\phi}(\theta)\|\pi(\theta))+\sum_{i=1}^{N}\log\pi(x_{i}|\sigma_{x}^{2})\leq\log\pi(\mathcal{D}|\boldsymbol{\sigma}^{2})\,.\end{aligned}$
(8)

Setting $\mathcal{L}(\phi,\boldsymbol{\sigma}^{2})=-\mathrm{ELBO}$ we then
want to find $\phi,\sigma_{x}^{2},\sigma_{y}^{2}$ so that

$\displaystyle\phi,(\sigma_{x}^{2},\sigma_{y}^{2})=\operatorname*{\mathrm{arg}\min}_{\phi,\,\boldsymbol{\sigma}^{2}}\mathcal{L}(\phi;\boldsymbol{\sigma}^{2})\,.$
(9)

In optimizing $\phi$ and the hyperparameters $\boldsymbol{\sigma}^{2}$ at the
same time we here follow a common scheme, cf. [17]. To solve (9), we use
backpropagation on the following Monte-Carlo object

$\displaystyle\begin{aligned}
\mathcal{L}^{\mathrm{M.C.}}(\phi;\boldsymbol{\sigma}^{2})&:=-\frac{1}{M}\sum_{m=1}^{M}\log\left(\frac{1}{L}\sum_{l=1}^{L}p(y_{i_{m}}|\theta_{m},\zeta_{i_{m},l},\sigma_{y}^{2})\right)\\\
&+\frac{1}{N}D_{\mathrm{KL}}(q_{\phi}(\theta)\|\pi(\theta))-\frac{1}{M}\sum_{m=1}^{M}\log\pi(x_{i_{m}}|\sigma_{x}^{2})\,,\end{aligned}$
(10)

where the minibatches
$\\{(x_{i_{1}},y_{i_{1}}),\ldots,(x_{i_{M}},y_{i_{M}}))\\}$, the $M$ samples
$\theta_{m}\sim q_{\phi}(\theta_{m})$ and $M\cdot L$ samples
$\zeta_{i_{m},l}\sim\pi(\zeta_{i_{m},l}|x_{i},\sigma_{x}^{2})$ will be drawn
anew in each optimization step. Note that the way $\theta_{m}$ is sampled in
(10) differs slightly from the way this is done in approaches such as [8, 9,
16] as $\theta_{m}$ will be identical for all inputs $\zeta_{i_{m},l}$ with
the same $m$.222For the approach from [16] this also means that Gaussian
dropout of “type A” has to be used. For some algorithms on the other hand,
such as Bayes by Backprop [1], taking in (10) a single Monte Carlo sample
$\theta_{m}=\theta$ for all $m$ instead might allow for a more efficient
implementation. The loss function in (8) and (10) is the backbone of the EiV
algorithm. To make the first term in (10) numerically stable, and suitable for
backpropagation, the common ”logsumexp” function can be used. Before we can
use (10) for training however, we still have to fix the prior distributions
and the variational distribution $q_{\phi}$.

###  2.2 Choosing $\pi(\theta),\pi(\zeta_{i})$ and $q_{\phi}$

Choosing $\pi(\theta)$ is rather standard in the literature on Bayesian neural
networks and complies with choosing a regularization for $\theta$, cf. [1]. We
will here follow the common choice of a centered normal distribution
$\pi(\theta)=\mathcal{N}(\theta|0,\lambda_{\theta}^{2})$. For Bernoulli
dropout [8] with rate $p$ the term
$\frac{1}{N}D_{\mathrm{KL}}(q_{\phi}(\theta)\|\pi(\theta))$ in (10) then
equals, up to a constant, $\frac{(1-p)}{2N\lambda_{\theta}^{2}}|\theta|^{2}$.

The prior $\pi(\zeta_{i})$ is more particular to the EiV approach. If we
consider for instance images as the inputs to our neural network a natural
choice would be to consider a distribution whose bulk covers the pixel range.
For the choice $\pi(\zeta_{i})=\mathcal{N}(\zeta_{i}|0,\lambda_{\zeta}^{2})$
we obtain

$\displaystyle\pi(\zeta_{i}|x_{i},\sigma_{x}^{2})$
$\displaystyle=\mathcal{N}\Big{(}\zeta_{i}\Big{|}(1+\frac{\sigma_{x}^{2}}{\lambda_{\zeta}^{2}})^{-1}x_{i},(1+\frac{\sigma_{x}^{2}}{\lambda_{\zeta}^{2}})^{-1}\sigma_{x}^{2}\Big{)}\,,\,\,\pi(x_{i}|\sigma_{x}^{2})$
$\displaystyle=\mathcal{N}(x_{i}|0,\sigma_{x}^{2}+\lambda_{\zeta}^{2})\,.$
(11)

In Section 3 we will use the constant, improper, prior that arises for
$\lambda_{\zeta}\rightarrow\infty$. In this limit the marginal
$\pi(x_{i}|\sigma_{x}^{2})$ is obviously ill-defined. However, this is not
really problem, since

  * •

$\pi(\zeta_{i}|x_{i},\sigma_{x}^{2})\overset{\lambda_{\zeta}\rightarrow\infty}{\longrightarrow}\mathcal{N}\left(\zeta_{i}|x_{i},\sigma_{x}^{2}\right)$
is well-defined and can be sampled from.

  * •

The divergence of $\prod_{i}\pi(x_{i}|\sigma_{x}^{2})$ that appears in (6)
cancels out in (5), so that there is a proper posterior
$\pi(\theta|\mathcal{D},\boldsymbol{\sigma}^{2})$ in the limit.

  * •

The only non-convergent term of $-\mathcal{L}(\phi,\sigma^{2})$ for
$\lambda_{\zeta}\,{\rightarrow}\,\infty$ is given by
$\frac{1}{N}\sum_{i=1}^{N}\log\pi(x_{i}|\sigma_{x}^{2})=-\frac{1}{N}\sum_{i=1}^{N}\frac{x_{i}^{2}}{2(\sigma_{x}^{2}+\lambda_{\zeta}^{2})}-\frac{1}{2}\log(2\pi(1+\frac{\sigma_{x}^{2}}{\lambda_{\zeta}}))-\log\lambda_{\zeta}$
or, more precisely, by its term $-\log\lambda_{\zeta}$ that can be dropped as
it does not depend on $\phi$ or $\boldsymbol{\sigma}^{2}$.

The choice of $q_{\phi}$ is only limited by three requirements. First of all,
we need to be able to sample from $q_{\phi}$. Secondly, we need an expression
for the regularizer $D_{\mathrm{KL}}(q_{\phi}(\theta)\|\pi(\theta))$ \- either
explicitly or via Monte Carlo Sampling [17]. Finally, we have to be able to
optimize the arising loss function $\mathcal{L}(\phi,\boldsymbol{\sigma}^{2})$
w.r.t. $\phi$ for example via the reparametrization trick [17]. Merely for
convenience we will restrict ourselve in this work to the popular choice of
Monte Carlo dropout [8] where $q_{\phi}$ arises from dropping randomly nodes
of the network with some probability and where $\phi$ simply coincides with
the network parameters. However, let us emphasize that the algorithm described
in this work is by no means restricted to this particular choice but is usable
for any $q_{\phi}$ for which the above applies.

###  2.3 Algorithmic details

In total our algorithm, that implements the optimization (9), is optimizing
three objects $\phi,\sigma_{x}$ and $\sigma_{y}$. While $\phi$ and
$\sigma_{y}$ can be optimized as in the corresponding non-EiV model, the
optimization w.r.t $\sigma_{x}$ needs some discussion. Using (11) this
optimization can be implemented via the reparametrization trick and is
influenced by the chosen $\lambda_{\zeta}$. If we let
$\lambda_{\zeta}\rightarrow\infty$ as described in Subsection 2.2 we found,
for the examples from Section 3, that the optimization (9) often led to
vanishing $\sigma_{x}$.333At least when looking at the maximum likelihood
solution of the EiV model (3) this behavior appears sensible as the likelihood
becomes infinite for $\sigma_{x}=0$ and $\zeta_{i}=x_{i}$ if there are no
”duplicate” $x_{i}$ with different labels in the data. To prevent this from
happening, we here propose, at least for large or infinite $\lambda_{\zeta}$
to use the Deming regression [4], that is

$\displaystyle\sigma_{x}=\delta\cdot\sigma_{y}\,.$ (12)

We will dub in this work $\delta$ the _Deming factor_ and keep it fixed during
training. By choosing $\delta$ the impact of the input noise can be
controlled. For data that is normalized in both, input and output, choosing
$\delta=1$ might be a reasonable choice when one expects input and output to
be noisy whereas for $\delta\searrow 0$ the impact of the input noise in the
EiV model is diminished and the non-EiV model is recovered. A strategy to
estimate the best $\delta$ for a given dataset would be the content of a work
on its own; nevertheless we will give a small discussion in Section 3 below.
Practically, (12) means that when learning
$\boldsymbol{\sigma}^{2}=(\sigma_{x}^{2},\sigma_{y}^{2})$ we only have to
learn one of its components. In this work we learn $\sigma_{y}^{2}$ and
compute $\sigma_{x}^{2}$ from (12).

When training $\phi$ and $\sigma_{y}^{2}$ using (12) we found the following
splitting effective in producing better results on both parameters: split the
number of training epochs $n_{\mathrm{train}}$ in three parts
$n_{\mathrm{train}}=n_{\mbox{\scriptsize only $\phi$}}+n_{\mbox{\scriptsize
increase $\delta$}}+n_{\mbox{\scriptsize fine-tune}}$. Starting from some
initial values for $\phi$ and $\sigma_{y}^{2}$ and with Deming factor
$\tilde{\delta}=0$ only update $\phi$ via (10) for $n_{\mbox{\scriptsize only
$\phi$}}$ epochs. Thereafter, for $n_{\mbox{\scriptsize increase $\delta$}}$
epochs, starting from 0, increase $\tilde{\delta}$ before each epoch by
$\delta/n_{\mbox{\scriptsize increase $\delta$}}$ and then train $\phi$ and
$\sigma_{y}^{2}$. Finally, train $\phi$ and $\sigma_{y}^{2}$ using the full
Deming factor $\tilde{\delta}=\delta$ for $n_{\mbox{\scriptsize fine-tune}}$
epochs. The full algorithm is summarized in the appendix.

###  2.4 A new view on aleatoric uncertainty

Once the neural network is trained the learned $\phi$ and
$\boldsymbol{\sigma}^{2}$ can be used to obtain uncertainties and predictions
for a new $x^{\ast}$. Namely, we draw $\zeta^{\ast}$ from444Note that we
understand $\zeta^{\ast}$ as independent of the $\zeta_{i}$ that generated the
$x_{i}$ and $y_{i}$ in $\mathcal{D}$, which is why we can drop the
conditioning on $\mathcal{D}$.
$\pi(\zeta^{\ast}|x^{\ast},\sigma_{x}^{2})=\pi(\zeta^{\ast}|x^{\ast},\mathcal{D},\sigma_{x}^{2})$
and $\theta$ from $q_{\phi}(\theta)$ and use the distribution of the
regression curve $f_{\theta}(\zeta^{\ast})$. To express predictions and
uncertainties, we can then either use quantiles or moments. For the latter we
set

$\displaystyle
m(x^{\ast})=\mathbb{E}_{\begin{subarray}{c}\theta\sim\pi(\theta|\mathcal{D},\boldsymbol{\sigma}^{2}),\,\\\
\zeta^{\ast}\sim\pi(\zeta^{\ast}|x^{\ast},\sigma_{x}^{2})\end{subarray}}[f_{\theta}(\zeta^{\ast})],\,u(x^{\ast})^{2}=\mathrm{Var}_{\begin{subarray}{c}\theta\sim\pi(\theta|\mathcal{D},\boldsymbol{\sigma}^{2}),\,\\\
\zeta^{\ast}\sim\pi(\zeta^{\ast}|x^{\ast},\sigma_{x}^{2})\end{subarray}}[f_{\theta}(\zeta^{\ast})]\,,$
(13)

where the uncertainty $u(x^{\ast})$ can be split into (law of total variance)

$\displaystyle\begin{aligned}
u(x^{\ast})^{2}&=\mathbb{E}_{\zeta^{\ast}\sim\pi(\zeta^{\ast}|x^{\ast},\sigma_{x}^{2})}(\mathrm{Var}_{\theta\sim\pi(\theta|\mathcal{D},\boldsymbol{\sigma}^{2})}[f_{\theta}(\zeta^{\ast})])\qquad\mbox{(epistemic)}\\\
&+\mathrm{Var}_{\zeta^{\ast}\sim\pi(\zeta^{\ast}|x^{\ast},\sigma_{x}^{2})}(\mathbb{E}_{\theta\sim\pi(\theta|\mathcal{D},\boldsymbol{\sigma}^{2})}[f_{\theta}(\zeta^{\ast})])\,.\qquad\mbox{(aleatoric)}\end{aligned}$
(14)

The detailed algorithm on how to compute $m(x^{\ast})$ und $u(x^{\ast})$ is
contained in the appendix.

The “classical” treatment of epistemic and aleatoric uncertainty, based on the
posterior predicive and discussed in Section 1, can be easily combined with
the above. For the posterior predictive
$\pi(y^{\ast}|x^{\ast},\mathcal{D},\boldsymbol{\sigma}^{2})$, we draw for each
sample $f_{\theta}(\zeta^{\ast})$ labels $y^{\ast}$ from
$\mathcal{N}(f_{\theta}(\zeta^{\ast}),\sigma_{y}^{2})$. The variance of this
distribution can then be split into

$\displaystyle\mathrm{Var}_{y^{\ast}\sim\pi(y^{\ast}|x^{\ast},\mathcal{D},\boldsymbol{\sigma}^{2})}[y^{\ast}]=u(x^{\ast})^{2}+\sigma_{y}^{2}$
(15)

and is therefore simply augmented by an extra term $\sigma_{y}^{2}$. For
$\sigma_{x}\searrow 0$ the aleatoric part of $u(x^{\ast})$ in (14) vanishes,
$u(x^{\ast})$ coincides with the epistemic uncertainty and (15) morphs into
the “classical” split in aleatoric and epistemic uncertainty. The usage of the
aleatoric part $\sigma_{y}^{2}$ that appears in (15) as an uncertainty is only
justified if one is really interested in the uncertainty of a noise perturbed
label $y^{\ast}$ given $x^{\ast}$. If one is actually interested in the ground
truth predicted by $f_{\theta}(\zeta^{\ast})$ (EiV) or $f_{\theta}(x^{\ast})$
(non-EiV) than it is inappropriate to take the second term in (15) into
account.

By introducing an additional aleatoric uncertainty in (14) the EiV model
introduces an aleatoric uncertainty that is still consistent with statistics
in such cases. In fact as we will see in Figure 1 below, for increasing Deming
factor $\delta$ part of the uncertainty previously captured by $\sigma_{y}$
gets transferred to the input uncertainty that is captured by $\sigma_{x}$.
The second term of (14) not only provides an aleatoric uncertainty that is
still available if considering the ground truth predicted by $f_{\theta}$, but
in addition exploits a source of uncertainty that is not present in the non-
EiV model and thus gives a more complete description of reality. Finally, by
distinguishing between $\zeta^{\ast}$ and $x^{\ast}$ the EiV model introduces
a new concept of prediction, cf. (13), that differs from the non-EiV model.

##  3 Experiments

For evaluating the model introduced in Section 2 we used four datasets, where
two were generated with a known ground truth and two are real world examples.
To keep the setup as lucid as possible and to avoid anisotropic issues, we
here only consider datasets with 1-dimensional labels $y$. For all examples
below, we used the $q_{\phi}$ induced by Bernoulli dropout [8] with a dropout
rate of 0.5, an Adam optimizer [15] and a learning rate of $10^{-3}$ that was
reduced to $10^{-4}$ for the last 50 epochs. Training both, EiV and non-EiV,
alongside each other with the same $q_{\phi}$ allows us to judge the quality
of our EiV model and serves as a consistency check as we expect the EiV model
to behave like the non-EiV model for small Deming factors, $\delta\searrow 0$.
Due to the multiple sampling of $\zeta$ in (10) training an EiV model requires
more time for the same number of epochs. We used $L=5$ and observed an
increase of the training time by a factor of about $2$. Using a single Tesla
K80 GPU a training run of both models took a few minutes for the examples
below, with exception of the multinomial example where the training time was
around 1/2 h (non-EiV) and 1 h (EiV). All experiments were implemented in the
Python library PyTorch [21] (BSD license).

Table 1: Results of the non-EiV model and the EiV model for those datasets from Section 3 with known ground truth, varying $\tilde{\sigma}_{x}$ and Deming factors $\delta=0.15$ (for $\tilde{\sigma}_{x}$=0.05), $0.20$ (for $\tilde{\sigma}_{x}$=0.07) and $0.30$ (for $\tilde{\sigma}_{x}$=0.10). The RMSE is w.r.t. noisy data (and not the ground truth). The second column denotes the coverage of the ground truth by the uncertainty times $1.96$. All results were averaged over 20 different training runs and different random seeds. The displayed digits are certain w.r.t. the corresponding standard errors. | data | Mexican hat | multinomial
---|---|---|---
| $\sigma_{x}$ | 0.05 | 0.07 | 0.10 | 0.05 | 0.07 | 0.10
RMSE | EiV | 0.34 | 0.35 | 0.38 | 0.67 | 0.76 | 0.85
non-EiV | 0.34 | 0.36 | 0.38 | 0.63 | 0.71 | 0.80
coverage | EiV | 0.97 | 0.93 | 0.91 | 0.96 | 0.92 | 0.79
non-EiV | 0.90 | 0.82 | 0.72 | 0.80 | 0.63 | 0.42
Figure 1: The learned $\sigma_{y}$ during training for the non-EiV model and
the EiV model with varying Deming factor $\delta$ for data around a Mexican
hat curve and 20 training runs. The plotted areas depict one standard error
around the average. The $\tilde{\sigma}_{y}$ used for generating the data is
marked by the blue, dashed line. A larger Deming factor lets the final
$\sigma_{y}$ decrease by explaining more of the noise via input uncertainty.
For $\delta\searrow 0$ EiV and non-EiV coincide.

First, we consider the problem of fitting a one-dimensional “Mexican hat”
$g(\zeta)=(1-(4\zeta)^{2})e^{-\frac{1}{2}(4\zeta)^{2}}$, compare the blue line
in Figure 2(a). We drew 300 values for $\zeta$ uniformly between $-1$ and 1,
computed the corresponding $g(\zeta)$ and disturbed $\zeta$ and $g(\zeta)$ by
Gaussian noise with standard deviations $\tilde{\sigma}_{x}=0.05,0.07,0.10$
and $\tilde{\sigma}_{y}=0.30$. For each choice of $\tilde{\sigma}_{x}$ and
various choices of the Deming factor $\delta$ (cf. Figure 1) a fully connected
neural network with 3 hidden layers (50, 100 and 50 neurons) was trained, both
with and without EiV. With the notation of Section 2 we used
$n_{\mathrm{train}}=1000$, a batch size of 25, a regularization of
$\mbox{\small$(2N\lambda_{\theta}^{2})^{-1}$}=10^{-7}$, $n_{\mbox{\scriptsize
only $\phi$}}=300$ and, for the EiV models, $n_{\mbox{\scriptsize increase
$\delta$}}=20$.
Figure 1 shows the evolution of $\sigma_{y}$ during training for the non-EiV
model and the EiV model for all studied Deming factors and
$\tilde{\sigma}_{x}=0.07$ for 20 training runs. For small $\delta$ the
evolution of $\sigma_{y}$ for the EiV model and the non-EiV are essentially
identical. For increasing $\delta$ the learned value of $\sigma_{y}$ decreases
as $\sigma_{x}=\delta\cdot\sigma_{y}$ is more and more used to capture the
noise of the data. However, for all chosen values we see that after training
we obtain $\sigma_{y}>\tilde{\sigma}_{y}$ (dotted, blue), which is natural as
we expect the learned $\sigma_{y}$ to partially include the model misfit. We
thus expect the optimal choice of $\delta$ to be slightly below the quotient
$\frac{\tilde{\sigma}_{x}}{\tilde{\sigma}_{y}}$.

(a) Evaluation on $\zeta$

(b) Evaluation on $x\sim p(x|\zeta,\sigma_{x}^{2})$

Figure 2: Predictions and uncertainties of a non-EiV model (black) and an EiV
model (red, $\delta=0.2$), trained on data around a Mexican hat (blue) that
was disturbed by noise with $\tilde{\sigma}_{x}=0.07,\tilde{\sigma}_{y}=0.3$.
On the left the models are evaluated on “true” $\zeta$ and on the right on
noisy $x\sim p(x|\zeta,\tilde{\sigma}_{x}^{2})$. Predictions and uncertainties
are computed as described in Section 3.

Figure 2 shows the prediction of an EiV model with $\delta=0.2$ (slightly
below $\frac{\tilde{\sigma}_{x}}{\tilde{\sigma}_{y}}$) and a non-EiV model,
both trained on input data with $\tilde{\sigma}_{x}=0.07$. Considering errors
in the input has the interesting consequence that we can compare two kind of
predictions, which are shown in Figure 2(a) and 2(b). In both Figures the blue
curve shows the ground truth $g(\zeta)$. Figure 2(a) compares
$\mathbb{E}_{\theta\sim q_{\phi}(\theta)}[f_{\theta}(\zeta)]$ of the EiV model
(red) and the non-EiV model (black). The red and black areas mark 1.96 times
the standard deviaton of $f_{\theta}(\zeta)$ when drawing $\theta$ from
$q_{\phi}(\theta)$. In practice, however, $\zeta$ is unknown and all we see is
a noisy $x\sim p(x|\zeta,\tilde{\sigma}_{x}^{2})$. This case is depicted in
Figure 2(b). For each $\zeta$ we drew a $x\sim
p(x|\zeta,\tilde{\sigma}_{x}^{2})$ and then computed $\mathbb{E}_{\theta\sim
q_{\phi}(\theta)}[f_{\theta}(x)]$ for the non-EiV model and $m(x)$ as in (13)
for the EiV model. The black area depicts once more 1.96 times the epistemic
uncertainty, while the red one now represents 1.96 times $u(x)$ of (13).

Figure 2(a) shows that the epistemic uncertainty of the EiV model and non-EiV
evaluated on $\zeta$ are pretty comparable with only slight differences. In
Figure 2(b) we observe larger uncertainties for the EiV model. Due to the
random nature of the plot it is however hard to perceive if these larger
uncertainties are necessary for a better coverage of the ground truth. The
plot in Figure 3 reveals that this is indeed the case. It shows for each
$\zeta$ the coverage, under repeated sampling of
$x\sim\pi(\zeta|x,\tilde{\sigma}_{x}^{2})$, of the ground truth $g(\zeta)$ by
the intervals $m(x)\pm 1.96\,u(x)$ for the EiV model (red) and
$\mathbb{E}_{\theta\sim q_{\phi}(\theta)}[f_{\theta}(x)]\pm
1.96\,\mathrm{Var}_{\theta\sim q_{\phi}(\theta)}(f_{\theta}(x))^{1/2}$ for the
non-EiV model (black). The dots and bars in Figure 3 mark the mean of the
coverage and its standard error under 20 different training runs and random
seeds. The dashed, blue line marks $0.95$. We see that with the exception of
the peak of the Mexican hat, which both models fail to capture correctly, the
EiV model has a substantially better coverage of the ground truth. The middle
column of Table 1 summarizes the coverage for all considered choices of
$\tilde{\sigma}_{x}$. Table 1 also shows the root mean squared error (RMSE) of
the prediction $m(x)$ (EiV), as in (13), and of $\mathbb{E}_{\theta\sim
q_{\phi}(\theta)}[f_{\theta}(x)]$ (non-EiV) w.r.t. the noisy data. Note that
while the RMSE is almost identical for EiV and non-EiV the coverage is
substantially improved by considering Errors-in-Variables. The results in
Table 1 were averaged over 20 training runs and are specified with those
digits that are certain under the corresponding standard error.

The right column “multinomial” of Table 1 shows the according results for
another training set with $10^{5}$ datapoints created with a 5D polynomial,
with random coefficients, that was multiplied by $e^{-\sin(5|\zeta|^{2})}$. We
used the same choices of $\tilde{\sigma}_{x}$, $\tilde{\sigma}_{y}$ and
$\delta$ as above and a fully connected neural networks with three hidden
layers (500, 300 and 100 neurons) that was trained for 350 epochs with a batch
size of 200 and the regularization
$\mbox{\small$(2N\lambda_{\theta}^{2})^{-1}$}=10^{-6}$. Figure 4 shows, as in
Figure 2(b), the predictions and uncertainties for $x\sim
p(x|\zeta,\tilde{\sigma}_{x}^{2})$ drawn for $\zeta$ along one axis. The
results of Table 1, produced with 20 training runs and $10^{3}$ test
datapoints, show once more that the EiV model produces a similar, only
slightly increased, RMSE compared to the non-EiV model but a substantially
better coverage with its uncertainties.

Figure 3: The coverage of the ground truth by the uncertainty (times 1.96) for
the EiV model (red-orange, $\delta=0.2$) and the non-EiV model (black-gray) as
in Figure 2(b) for the Mexican hat with $\tilde{\sigma}_{x}=0.07$, under
repeated sampling of $x\sim p(x|\zeta,\tilde{\sigma}_{x}^{2})$. The plot shows
the averages (dots) and its standard errors (bars) over 20 training runs with
different random seeds. The dashed, blue line marks $0.95$.

Figure 4: Predictions and uncertainties of a non-EiV model (black) and an EiV
model (red, $\delta=0.2$) for data following a modulated 5D polynomial (blue)
with $\tilde{\sigma}_{x}=0.07,\tilde{\sigma}_{y}=0.30$ along one axis when
evaluating the model on $x\sim p(x|\zeta,\tilde{\sigma}_{x}^{2})$. Predictions
and uncertainties are computed as described in Section 3.

While examples with a known ground truth are useful to judge quantities such
as the epistemic uncertainty or (14), in practice the noise-free ground truth
will usually be unknown, even for the training data. As two final examples we
will therefore study two real world examples, where no ground truth is
available and which are available as open datasets. We consider the prediction
of the quality of Portuguese red ”Vinho Verde” [2], that contains for each of
the 1599 studied wines a quantity quantifying the quality and 11 chemical
properties, as well as the Boston Housing dataset [12]. For the first dataset
all concentrations were transformed by a logarithm and the inputs and output
were normalized to have mean 0 and standard deviation 1. A subset of 20% was
split off for testing purposes. On the remaining a fully connected neural
network with three hidden layers (with 200, 100 and 50 neurons) was trained
for 400 epochs with a batch size of 16,
$\mbox{\small$(2N\lambda_{\theta}^{2})^{-1}$}=10^{-9}$ and various choices of
Deming factors $\delta$. On the second dataset, the disputed [3] 12th column
was removed, inputs and output were normalized and a subset of 20% was split
off for testing. On the remainder a fully connected neural network with three
hidden layers (200, 100 and 50 neurons) was trained for 1000 epochs with a
batch size of 16 and a strong regularization
$\mbox{\small$(2N\lambda_{\theta}^{2})^{-1}$}=10^{-2}$ (to prevent
overfitting). Figure 5 shows the average of the RMSE, together with its
standard error, on the test set depending on the used $\delta$ for 20 training
runs with different randoms seeds for both datasets. For the non-EiV model the
area of one standard error around the mean is depicted in gray. We saw already
in Table 1 that the RMSE cannot be used as a sole measure to judge the quality
of the used model: a slightly worse RMSE can come with a much better
uncertainty coverage. However, studying the RMSE can help to rule out too
large values for $\delta$ that lead to a bad fit and overprudent small
$\delta$. From Figure 5(a) a good choice for the wine quality dataset might be
$\delta=0.6$ with a relatively small RMSE. The “symmetric” choice $\delta=1$
(with $\sigma_{x}=\sigma_{y}$) on the other hand yields only a slightly worse
RMSE as the non-EiV model. In fact, as we can see in Figure 6(a) the errors
that both models make for $\delta=1$ are almost identical. The uncertainty, on
the other hand, depicted in Figure 6(b), is increased for the EiV model. This
is identical to the behavior observed for the toy examples in Figure 2(b) and
4. As we have no ground truth available for this dataset a coverage as in
Table 1 cannot be computed.

(a) Wine quality

(b) Boston Housing

Figure 5: Dependency of the RMSE of the EiV model on the Deming factor
$\delta$ from (12) for the Wine quality and Boston Housing dataset for 20
training runs and different random seeds. The plot shows the means (dots) and
its standard errors (bars). For the non-EiV model the range of the standard
error around the mean is depicted by the gray area.

(a) errors

(b) uncertainties

Figure 6: The deviation of the prediction from the label $y$ (left) and the
uncertainty (right) for test datapoints from the wine quality dataset [2] for
the EiV model ($y$-axis, $\delta=1$) and the non-EiV model ($x$-axis). The
dashed, black lines show the diagonals.

##  4 Conclusion and discussion

We showed how Errors-in-Variables (EiV), a classical concept from statistics,
can be combined with existing Bayesian methods for uncertainty quantification
in deep regression. This not only allows to treat the input of the network as
equipped with an uncertainty but also gives a notion of aleatoric uncertainty
that is, in many cases, more coherent with statistics. We found this method to
give predictions of a similar quality as the non-EiV method but with an
increased uncertainty. For examples with known ground truth and noisy input
this increased uncertainty was observed to be necessary to achieve a better
coverage of the ground truth.

The aim of this work is to propose a comparably simple and effective way to
treat uncertain input in deep learning, and thereby to allow reconsidering the
concept of aleatoric uncertainty. In cases where safety is relevant, e.g. in
medical applications, the question whether to doubt or trust a prediction can
easily get an ethical dimension. For this reason let us emphasize that,
hitherto, all existing uncertainty methods for neural networks, including the
one presented in this work, are approximations. Using them blindly might lead
to overconfident trust in predictions and thereby to negative individual or
societal consequences. A sustainable usage of these methods will require
wariness and future research to understand their limits and possible
improvements. Concerning this work let us recall some aspects that we
encourage to be studied in future work. For instance, to keep the setup simple
we restricted ourselves to the variational inference based on Bernoulli
dropout. Analyzing how different approaches will impact or perform under
Errors-in-Variables and examining a larger variety of datasets would make a
natural follow-up study to this article. The same is true for an enhancement
of the distributions involved in (2), which could for instance involve an
anisotropic or heteroscedastic adaption or the usage of a learnable push-
forward mapping. Finally, our model could further be enhanced by a criterion
that allows for a good choice of the Deming factor from (12) or by an approach
that replaces the latter by something more sophisticated.

## References

  * [1] Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural network.  In International Conference on Machine Learning (2015), PMLR, pp. 1613–1622.
  * [2] Cortez, P., Cerdeira, A., Almeida, F., Matos, T., and Reis, J. Modeling wine preferences by data mining from physicochemical properties.  Decision support systems 47, 4 (2009), 547–553.
  * [3] Crangle, C. Inadvertent racism in popular data science data set.  available at https://www.linkedin.com/pulse/its-time-retire-boston-housing-dataset-colleen-e-crangle.  Accessed: 2021-05-17.
  * [4] Deming, W. E. Statistical adjustment of data., 1943.  Wiley.
  * [5] Depeweg, S., Hernandez-Lobato, J.-M., Doshi-Velez, F., and Udluft, S. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning.  In International Conference on Machine Learning (2018), PMLR, pp. 1184–1193.
  * [6] Duvenaud, D., Maclaurin, D., and Adams, R. Early stopping as nonparametric variational inference.  In Artificial Intelligence and Statistics (2016), PMLR, pp. 1070–1077.
  * [7] Fuller, W. A. Measurement error models, vol. 305.  John Wiley & Sons, 2009.
  * [8] Gal, Y., and Ghahramani, Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In international conference on machine learning (2016), PMLR, pp. 1050–1059.
  * [9] Gal, Y., Hron, J., and Kendall, A. Concrete dropout.  arXiv preprint arXiv:1705.07832 (2017).
  * [10] Gillard, J. An historical overview of linear regression with errors in both variables.  Math. School, Cardiff Univ., Wales, UK, Tech. Rep (2006).
  * [11] Gustafsson, F. K., Danelljan, M., and Schon, T. B. Evaluating scalable bayesian deep learning methods for robust computer vision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2020), pp. 318–319.
  * [12] Harrison Jr, D., and Rubinfeld, D. L. Hedonic housing prices and the demand for clean air.  Journal of environmental economics and management 5, 1 (1978), 81–102.
  * [13] Hüllermeier, E., and Waegeman, W. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.  Machine Learning 110, 3 (2021), 457–506.
  * [14] Kendall, A., and Gal, Y. What uncertainties do we need in bayesian deep learning for computer vision?  arXiv preprint arXiv:1703.04977 (2017).
  * [15] Kingma, D. P., and Ba, J. Adam: A method for stochastic optimization.  In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015).
  * [16] Kingma, D. P., Salimans, T., and Welling, M. Variational dropout and the local reparameterization trick.  arXiv preprint arXiv:1506.02557 (2015).
  * [17] Kingma, D. P., and Welling, M. Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114 (2013).
  * [18] LeNail, A. Nn-svg: Publication-ready neural network architecture schematics.  Journal of Open Source Software 4, 33 (2019), 747.
  * [19] Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., Van Der Laak, J. A., Van Ginneken, B., and Sánchez, C. I. A survey on deep learning in medical image analysis.  Medical image analysis 42 (2017), 60–88.
  * [20] McAllister, R., Gal, Y., Kendall, A., Van Der Wilk, M., Shah, A., Cipolla, R., and Weller, A. Concrete problems for autonomous vehicle safety: Advantages of bayesian deep learning.  International Joint Conferences on Artificial Intelligence, Inc.
  * [21] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in pytorch.  In NIPS-W (2017).
  * [22] Sünderhauf, N., Brock, O., Scheirer, W., Hadsell, R., Fox, D., Leitner, J., Upcroft, B., Abbeel, P., Burgard, W., Milford, M., et al. The limits and potentials of deep learning for robotics.  The International Journal of Robotics Research 37, 4-5 (2018), 405–420.
  * [23] Van Gorp, J., Schoukens, J., and Pintelon, R. The errors-in-variables cost function for learning neural networks with noisy inputs.  Intelligent Engineering through Artificial Neural Networks 8 (1998), 141–146.
  * [24] Van Gorp, J., Schoukens, J., and Pintelon, R. Learning neural networks with noisy inputs using the errors-in-variables approach.  IEEE Transactions on Neural Networks 11, 2 (2000), 402–414.
  * [25] Zhang, G., Sun, S., Duvenaud, D., and Grosse, R. Noisy natural gradient as variational inference.  In International Conference on Machine Learning (2018), PMLR, pp. 5852–5861.

##  5 Appendix

Data: Training data $\mathcal{D}=\\{(x_{1},y_{1}),\ldots,(x_{N},y_{N})\\}$

Result: $\phi$ and $\boldsymbol{\sigma}^{2}$

Fix $\pi(\zeta)$, $\pi(\theta)$, $\delta$;

Choose initial $\phi$ and $\sigma_{y}$;

Fix $n_{\mathrm{train}}=n_{\mbox{\scriptsize$\mathrm{only}$
$\phi$}}+n_{\mbox{\scriptsize$\mathrm{increase}$
$\delta$}}+n_{\mbox{\scriptsize$\mathrm{fine{-}tune}$}}$;

for  _$j$ in $1,...,n_{\mbox{\scriptsize$\mathrm{train}$}}$_ do

if  _$j\leq n_{\mbox{\scriptsize$\mathrm{only}$ $\phi$}}$_ then

$\tilde{\delta}\leftarrow 0$;

else if  _$j\leq n_{\mbox{\scriptsize$\mathrm{only}$
$\phi$}}+n_{\mbox{\scriptsize$\mathrm{increase}$ $\delta$}}$_ then

$\tilde{\delta}\leftarrow\tilde{\delta}+\delta/n_{\mbox{\scriptsize$\mathrm{increase}$
$\delta$}}$;

else

$\tilde{\delta}\leftarrow\delta$;

end if

_ _for  _minibatches $(x_{i_{1}},y_{i_{1}}),\ldots,(x_{i_{M}},y_{i_{M}})$ from
$\mathcal{D}$;_

 __ do

Compute $\sigma_{x}=\tilde{\delta}\cdot\sigma_{y}$;

Draw $\zeta_{i_{m},1},\ldots,\zeta_{i_{m},L}$ from
$\pi(\zeta_{i_{m,l}}|x_{i_{m}},\sigma_{x}^{2})$ for each $m$;

Draw $\theta_{1},\ldots,\theta_{M}$ from $q_{\phi}(\theta_{m})$;

Feedforward $(\zeta_{i_{m},l})_{m\leq M,l\leq L}$ through $f_{\theta}$;

Compute $\mathcal{L}^{\mathrm{M.C.}}(\phi,\boldsymbol{\sigma}^{2})$ as in
(10);

if  _$j\leq n_{\mbox{\scriptsize$\mathrm{only}$ $\phi$}}$_ then

Update $\phi$ via
$\nabla_{\phi}\mathcal{L}^{\mathrm{M.C.}}(\phi,\boldsymbol{\sigma}^{2})$;

else

Update $\phi,\sigma_{y}^{2}$ via
$\nabla_{\phi,\sigma_{y}^{2}}\mathcal{L}^{\mathrm{M.C.}}(\phi,\boldsymbol{\sigma}^{2})$;

end if

end for

end for

Algorithm 1 Training a neural network with EiV as described in Section 2. The
algorithm uses the Deming assumption (12).

Data: $x^{\ast}$ and $\phi,\sigma_{x}$

Result: Approximation to $m(x^{\ast})$ and $u(x^{\ast})$

Sample $\zeta_{1}^{\ast},\ldots,\zeta_{L}^{\ast}$ from
$\pi(\zeta^{\ast}_{l}|x^{\ast},\sigma_{x}^{2})$;

Sample $\theta_{1},\ldots,\theta_{K}$ from $q_{\phi}(\theta_{k})$;

for  _$k$ in $1,\ldots,K$_ do

Feed-forward $m_{lk}=f_{\theta_{k}}(\zeta_{l}^{\ast})$ for $l$ in
$1,\ldots,L$;

end for

Set $m(x^{\ast})=\frac{1}{LK}\sum_{l,k}m_{lk}$ ;

Set $u(x^{\ast})=\frac{1}{LK-1}\sqrt{\sum_{l,k}(m_{lk}-m(x^{\ast}))^{2}}$ ;

Algorithm 2 Computing the prediction $m(x^{\ast})$ and the corresponding
uncertainty $u(x^{\ast})$ for an input $x^{\ast}$ of an EiV model as described
in Section 2.4 and using the approximation (7). Figure 7: Illustration of the
Errors-in-Variables (EiV) approach presented in this work and the method
without EiV (“non-EiV”). The EiV model introduces an additional uncertainty to
the input of the network that is aleatoric and, in contrast to the one linked
with $y$, in many cases more coherent with the classical statistical view on
uncertainty, cf. Section 1. The network illustration was created using [18].


# MIMO-GAN: Generative MIMO Channel Modeling

Tribhuvanesh Orekondy1, Arash Behboodi1, Joseph B. Soriaga2 Qualcomm AI
Research is an initiative of Qualcomm Technologies, Inc. 1Qualcomm
Technologies Netherlands B.V.  2Qualcomm Technologies, Inc.  Qualcomm AI
Research

###### Abstract

We propose generative channel modeling to learn statistical channel models
from channel input-output measurements. Generative channel models can learn
more complicated distributions and represent the field data more faithfully.
They are tractable and easy to sample from, which can potentially speed up the
simulation rounds. To achieve this, we leverage advances in generative
adversarial network (GAN), which helps us learn an implicit distribution over
stochastic MIMO channels from observed measurements. In particular, our
approach MIMO-GAN implicitly models the wireless channel as a distribution of
time-domain band-limited impulse responses. We evaluate MIMO-GAN on 3GPP TDL
MIMO channels and observe high-consistency in capturing power, delay and
spatial correlation statistics of the underlying channel. In particular, we
observe MIMO-GAN achieve errors of under 3.57 ns average delay and -18.7 dB
power.

##  I Introduction

Statistical channel models, from Rayleigh and Rician fading models to 3GPP
models like TDL and CDL [1], are widely used for benchmarking in wireless
communication. These models use distributions that are easy to sample and
therefore convenient for rapid benchmarking. They target generic scenarios
relying on certain abstractions and simplifications. For instance, TDL channel
models utilize pre-defined delay profiles with Rayleigh or Rician stochastic
channel gains. Rayleigh fading model or Jakes model assume large number of
arriving paths uniformly distributed in the space.

There are a few reasons for thinking differently about channel modeling.
First, these models do not accurately capture the field data distribution.
This mismatch would potentially incur performance loss if the communication
system design is based on these models. This effect would become more
pronounced as we try to leverage the power of machine learning based domain
adaptation. More accurate models are crucial for demonstrating the gain
expected from machine learning solutions. Second, it is cumbersome to build a
channel model from field data measured in an environment. The current methods
operate based on selecting a channel model and fitting the parameters to the
measured data. It is desirable to unify the model selection and parameter
fitting steps and learn both in a systematic way.

Recent advances in field of generative modeling made it possible to model more
complicated data distributions and yet being able to flexibly sample from
them. In this work, we propose a method that addresses the above issues based
on generative modeling. The challenge is to design a generative model that
captures well the main features of the communication channel such as delay
profile and spatial correlation.

We propose MIMO-GAN, a GAN based model to directly learn the distribution of
channel impulse response (CIR), denoted by ${\bm{h}}$, only based on the
transmit and receive signal pairs $({\bm{x}},{\bm{y}})$. The generator models
the distribution of channel taps. One challenge is that the ground truth CIR
is not given and therefore cannot be used to train the discriminator.
Therefore, we can only train the discriminator on features extracted from
$({\bm{x}},{\bm{y}})$. For a generated CIR from the model and knowing transmit
signal ${\bm{x}}$, we assume that the channel is linear and generate the
receive signal using simple convolution. To learn specific features of CIR, we
first relate them to some receive signal features and train the discriminator
on them. For example, the receive spatial correlation of the MIMO channel is
learned by using ${\bm{y}}^{H}{\bm{y}}$ as input to the discriminator. We will
comment more on the details of this design.

The resulting generative model can learn more complex distribution, and it is
easier to sample from. Using neural networks to model the channel brings
additional potential benefits. First, the neural implementation of the model
and its parallel computation, and ease of sampling can bring simulation speed-
ups. With growing dimensionality of the channel, as in massive MIMO systems,
or with increasing complexity of the modeling as in mmWave and Terraherz
applications, the simulation phase becomes more time consuming. Although the
conventional simulators can be accelerated with parallelized implementatinos,
neural models are inherently parallelizable and can reduce considerably the
benchmarking rounds. Second, these models are differentiable and can be used
in end-to-end designs of communication systems where it is required to back-
propagate through the channel. Third, although the linearity assumption is
used in our current design, we can use a non-linear mapping from
$({\bm{x}},{\bm{h}})$ to ${\bm{y}}$ and learn non-linear effects caused by
hardware impairments. Finally, we can mention that generative models can be
used for better estimation and prediction of channel parameters similar to [2]
and follow up works.

The main contributions of the paper are as follows. We propose a GAN-based
model for statistical modeling of MIMO channels, that implicitly learns the
distribution of band-limited time-domain impulse responses using input-output
measurements. We show how to design the discriminator to learn spatial
correlation for MIMO channels. By evaluating this approach on a dataset
generated from 3GPP standard model, we demonstrate the consistency of MIMO-GAN
with the underlying channel across a range of metrics e.g., delay profiles,
spatial correlations.

This paper is organized as follows. We review the related works in Section II.
In Section III, we present the main approach and architecture details. The
numerical results are presented in Section IV.

##  II Related Work

A long line of literature tackles modelling the relation between wireless
input-output signals and understanding influence of various factors (e.g.,
carrier frequency, relative movement) on communication; see the classic work
of [3] for an example. Such works predominantly require significant domain
expertise to construct mathematical channel models and is also further
accompanied by extensive field measurements to identify parameters for these
models.

Recently, there has been a focus on data-driven based approaches towards
modelling channels, wherein the wireless channel model is represented by a
deep neural network (DNN). DNN-based channel models are appealing to rapidly
build channel models that capture the data distribution better. They further
enable differentiable computations and end-to-end learning of transceiver
algorithms. Existing approaches show promising results achieved by DNN-based
channel models, such as estimating the gradient of the channel using policy
gradient [4], learning conditional output distributions of the channel over
QAM symbols [5, 6, 7], modelling AWGN, Rayleigh fading channels [5] and
frequency-selective channels [8] and in some cases, also extending to
collected measurement data [9]. Common to these approaches is leveraging
advances in unsupervised learning, such as by using generative adversarial
networks (GANs) [10] or variational autoencoders (VAEs) [11] to represent the
channel model.

Although existing DNN-based channel model approaches have shown to be
successful in certain scenarios, they are limited by some shortcomings which
we address in this paper. First, to the best of our knowledge, the existing
methods are limited mostly to SISO scenarios. Second, it is unclear whether
the approaches also extend to simulating complex channels (e.g., frequency-
selective MIMO channels) as experimentally the approaches are largely
evaluated in relatively simple scenarios (e.g., AWGN). We tackle this by
proposing an approach that we show effective on 3GPP 4$\times$4 frequency-
selective TDL MIMO channels.

Next, unlike other statistical channel models, existing approaches model the
conditional distribution specific to a particular setup (for example limited
to QAM symbol communication), which restricts their usage as a general purpose
simulation tool (e.g., for different constellations, modulation and MIMO
dimensions). To address generalizing to novel inference scenarios, we propose
learning the MIMO channel CIR instead of general input-output mapping.

##  III Approach

In this section, we begin by introducing the system model in subsection III-A
and then describe in detail our proposed approach MIMO-GAN in subsection
III-B.

###  III-A System Model

Consider a static MIMO channel with its discrete impulse response
${\mathbf{H}}(\tau)\in\mathbb{C}^{N_{R}\times N_{T}}$ at a fixed sampling
frequency given by:

$\displaystyle{\mathbf{H}}(\tau)$
$\displaystyle=\begin{pmatrix}h_{1,1}(\tau)&h_{1,2}(\tau)&\cdots&h_{1,N_{T}}(\tau)\\\
h_{2,1}(\tau)&h_{2,2}(\tau)&\cdots&h_{2,N_{T}}(\tau)\\\
\vdots&\vdots&\ddots&\vdots\\\
h_{N_{R},1}(\tau)&h_{N_{R},2}(\tau)&\cdots&h_{N_{R},N_{T}}(\tau)\end{pmatrix}$
(1)

where $\tau=0,\dots,L-1$, and $L$ is the number of taps. The transmit signal
at sample $n$ with the sample sampling frequency is given by
${\mathbf{x}}[n]$. The receive signal is given by the discrete convolution:

$\displaystyle{\mathbf{y}}$ $\displaystyle={\mathbf{H}}*{\mathbf{x}}.$ (2)

We choose $T$ samples from the input and output of the channel. $T$ is chosen
large enough to cover the communication duration. Note that the sampling
frequency choice determines the maximum bandwidth we can simulate using the
proposed generative model. We denote the transmit signal from $j$’th antenna
as ${\bm{x}}_{j}\in{\mathbb{C}}^{T}$ and the receive signal from $i$’th
antenna as ${\bm{y}}_{i}\in{\mathbb{C}}^{T}$.

###  III-B Proposed Model: MIMO-GAN

Having presented the system model in the previous section, we now elaborate on
our approach by first presenting the key idea and then the details of our GAN-
based approach.

Figure 1: MIMO-GAN. Approach overview.

Motivation and Key Idea. A recent line of work [9] has demonstrated reasonable
success by explicitly learning the input-output relation using deep neural
networks (DNN). Specifically, learning the relation is achieved by modeling
$f$ as a DNN $f_{\theta}({\mathbf{x}},{\bm{z}})$ where the parameters $\theta$
are optimized to capture the channel effects and ${\bm{z}}$ is used to
introduce stochasticity. While they have demonstrated promising results, we
find explicitly learning the input-output relation leads to three
shortcomings: (a) it is unclear how $f_{\theta}$ (or generally the channel
model) generalizes to certain configuration changes e.g., increasing the CP
length and thereby the input length; such a configuration change requires
$f_{\theta}$ to be retrained; (b) the channel effects by default are not
interpretable; and (c) at inference time we want the model to generalize to
any arbitrary input ${\mathbf{x}}\in{\mathcal{X}}$. However, in practise it is
challenging with only a limited set of training data to capture the diversity
over the entire input space ${\mathbf{x}}\in{\mathcal{X}}$ (lengthy waveforms
in our case). This results in encountering out-of-distribution (OOD) data at
inference time with a high probability, which has been shown by many works
[10, 12] to result in erratic outputs.

The key idea in our work is to tackle these shortcomings by implicitly
learning the input-output relation. Instead of modelling $f$ (the explicit
input-output relation), we implicitly model the distribution of stochastic
channel effects $p({\mathbf{H}})$ using a generative model $h_{\theta}$. At
inference time, we can sample instances of channel response filters
${\mathbf{H}}$ from the generative model and apply its effect on the input
signal. Since the channel filter is stochastically generated independent of
the input, the learnt model has the benefit of being invariant to certain
input-encoding factors and notably the duration of the input waveform.

We now provide the necessary background of Generative Adversarial Networks
[13], a generative model which helps us capture stochastic channel effects.
Following the background, we elaborate on the specifics of the MIMO-GAN
approach.

Background: Generative Adversarial Networks (GAN). Since our approach builds
on top of building DNN-based generative models using the GAN framework [13],
we provide the background necessary for the elaboration that follows. The
objective in GAN is to train a DNN model that captures the distribution
$p({\mathbf{x}})$ of some observed data $\\{{\bm{x}}_{i}\\}_{i=1}^{N}$,
allowing one to sample new datapoints. This is achieved by leveraging two ML
models (typically DNNs): (i) a generator $G:{\bm{z}}\mapsto{\bm{x}}$ which
transforms a random noise vector (drawn from $d$-dim isotropic Gaussian
distribution) into a sample of the distribution; and (ii) a discriminator
$D:{\bm{x}}\mapsto[0,1]$ which predicts whether a given sample was generated
by $G$ (i.e., considered ‘fake’) or if it is a sample from the empirical
distribution (i.e., considered ‘real’). The two networks $G$ and $D$ are
trained simultaneously using a min-max objective:

$\displaystyle\min_{G}\max_{D}\;$ $\displaystyle\mathbb{E}_{{\bm{x}}\sim
p_{\text{data}}}[\log D({\bm{x}})]-\mathbb{E}_{{\bm{z}}\sim
p({\bm{z}})}\left[\log\left(D(G({\bm{z}}))\right)\right]$ (3)

While this formulation has proven successful, it typically suffers from
training instabilities (which we also observed in our experiments) due to
large variance in gradients during optimization. To tackle this shortcoming, a
recent line of work [14] propose enforcing a Lipschitz constraint, such as by
using a gradient penalty term [15]:

$\displaystyle\min_{G}\max_{D}\;$ $\displaystyle\mathbb{E}_{{\bm{x}}\sim
p_{\text{data}}}[D({\bm{x}})]-\mathbb{E}_{{\bm{z}}\sim
p({\bm{z}})}\left[\left(D(G({\bm{z}}))\right)\right]$
$\displaystyle-\lambda\mathbb{E}_{\tilde{{\bm{x}}}}\left[\left(||\nabla_{\tilde{{\bm{x}}}}D(\tilde{{\bm{x}}})||_{2}-1\right)^{2}\right]$
(4)

where $\tilde{{\bm{x}}}$ in the last term (gradient penalty) corresponds
uniformly sampled interpolates between real ${\bm{x}}\sim p_{\text{data}}$ and
generated $\hat{{\bm{x}}}\sim G(D({\bm{z}}))$ samples. In this formulation,
$D:{\bm{x}}\mapsto\mathbb{R}$ is a value function and no longer maps inputs to
probability scores. Note that although $D$ is more precisely a ‘critic’ in the
WGAN formulation, we refer to it as a ‘discriminator’ to remain consistent
with GAN literature.

MIMO-GAN: Overview. The core idea of our approach is to implicitly model the
distribution over band-limited impulse responses $p(h)$, which captures the
stochastic channel effects. It is implicit as we leverage only input-output
measurements during training. As detailed in Figure 1, MIMO-GAN primarily
involves two DNNs: a generator $G$ and discriminator $D$. The generator $G$ is
at the core of learnt channel $h_{\theta}$ (orange box of Figure 1). To
perform a simulation, we (a) sample an instance of MIMO channel
${\mathbf{H}}(\tau)$ from the generator $G$; and (b) convolve the input signal
${\bm{x}}$ with sampled instance. We now take a closer look at each $G$ and
$D$, and conclude the section with details on training.

Generative Channel Model. The backbone to the generative channel model
$h_{\theta}$ is a DNN generator model $G$ that transforms a noise vector
(${\bm{z}}\sim{\mathcal{N}}(\bm{0},{\bm{I}})$) into a sample from the
distribution of interest ($p(h)$ in our case). Specifically, the generator
produces channel instances conditioned on the spatial co-ordinates of the tx-
rx antenna $(i,j)$. The conditional co-ordinates, which are given as 1-hot
encodings, are embedded into a $D$-dim space (set to 4 in our experiments)
using a linear layer. The conditional generation of channels based on embedded
link-level information is particularly beneficial for three reasons. First,
since parameters of $G$ are shared for generation channels ${\bm{h}}_{i,j}$
across all links $(i,j)$, the network is encouraged to learn and exploit
correlations between the links. Second, we have parameter efficiency since
size of $G$’s parameters can remain constant in spite of large antenna arrays.
Third, we could leverage the same network for new configurations e.g., 8x8
(assuming critical spacing) can be used in e.g., 4x4, 8x2 setups. To sample
and construct a MIMO channel ${\mathbf{H}}$, we parallelize iterations over
the spatial co-ordinates $(i,j)$ and build up the channel tensor (as shown in
Equation 1). With the sampled ${\mathbf{H}}$ and by treating the channel as an
LTI system, we evaluate the output as Equation 2.

Discriminator. We leverage a discriminator DNN $D$ to encourage the generative
channel model $h_{\theta}$, given some input ${\mathbf{x}}$, to produce
realistic channel samples $\hat{{\mathbf{y}}}\in{\mathbb{C}}^{N_{R}\times T}$
consistent with the GT channel ${\mathbf{y}}\in{\mathbb{C}}^{N_{R}\times T}$
for a fixed input ${\mathbf{x}}\in{\mathbb{C}}^{N_{T}\times T}$. The goal of
the discriminator is to weed out ‘fake’ generated samples from ‘real’ GT
samples. Similar to the generative channel model $h_{\theta}$, we also provide
conditional information to the discriminator in the form of embeddings of
receive co-ordinates $embed(i)$ via a linear layer. As a result, the
discriminator discriminates between real and fake samples of output waveforms
${\bm{y}},\hat{{\bm{y}}}$ conditioned on the receive co-ordinate $i$. The
spatially-conditioned discriminator, similar to the conditional channel
generation, benefits from evaluating in different MIMO configurations and the
numbers of parameters remaining constant in different MIMO configurations. To
further complement the local (rx-specific) information observed in the
samples, we additionally introduce global information (across all rx antennas)
in the form of rx-side spatial correlations
${\bm{y}}^{H}{\bm{y}}\in{\mathbb{C}}^{N_{R}\times N_{R}}$.

Training. We extend the original WGAN-GP objective (Equation 4) to implicitly
learn the distribution over channels ${\mathbf{H}}\sim G({\bm{z}})$. However,
since our goal is to learn this only using input-output measurements, we
rewrite the objective in terms of the channel outputs:

$\displaystyle\min_{G}\max_{D}\;$ $\displaystyle\mathbb{E}_{{\bm{x}}\sim
p_{\text{data}}}[D({\bm{y}},{\bm{y}}^{H}{\bm{y}})]-\mathbb{E}_{{\bm{z}}\sim
p({\bm{z}})}\left[\left(D(\hat{{\bm{y}}},\hat{{\bm{y}}}^{H}\hat{{\bm{y}}})\right)\right]$
$\displaystyle-\lambda\mathbb{E}_{\tilde{{\bm{y}}}}\left[\left(||\nabla_{\tilde{{\bm{y}}}}D(\tilde{{\bm{y}}},\tilde{{\bm{y}}}^{H}\tilde{{\bm{y}}})||_{2}-1\right)^{2}\right]$
(5)

where $\hat{{\bm{y}}}={\bm{x}}*{\mathbf{H}}$. Note that unlike a typical GAN
where the discriminator directly critiques the generator’s outputs
(${\mathbf{H}}$ in this case), instead critiques the output of the channels
$\hat{{\bm{y}}}$.

Implementation Details. We model $G$ and $D$ as multi-layer perceptrons (MLPs)
with ReLU activations, both two hidden layers of 100 units. We train both
networks simultaneously using Adam optimizer with a learning rate of
2$\times$10-4 for 500 epochs. For every training iteration of $G$, we make 25
iterations of the discriminator (critic) network $D$. We use a gradient
penalty weight $\lambda$ set to 10.

##  IV Evaluation

###  IV-A Dataset

To train and evaluate our generative channel model MIMO-GAN, we collect
synthetic input-output measurements from a reference GT channel. We now
elaborate on the GT channel and details on measurements.

Ground-truth Reference Channel. We use as reference the 3GPP TDL channels [1]
with a long delay spread (300 ns) and specifically TDL-A and TDL-B. The TDL
channels captures properties of noiseless channels with multipath fading
demonstrating stochastic gains across a set of 23 paths. For the MIMO setting,
we consider co-polar uniform linear array antennas with $N_{T}$ and $N_{R}$
elements and ‘Medium-A’ correlation. In our experiments, we primarily consider
$N_{T}$=4 and $N_{R}$=4. We assume that the channel is stationary (by
configuring the doppler shift to 0 Hz in simulations). In particular, we use
the Matlab implementation of the TDL channels to simulate input-output
measurements.

Channel Input-Output Measurements. Using the reference GT TDL-A and TDL-B
channels, we collect 60K input-output measurements at a bandwidth of 30.72
MHz. The input signal per transmit antenna is fixed to a digital impulse
signal with unit power. We consider two approaches to transmit the input: (i)
by transmitting the digital impulse simultaneously across all transmit
antennas; and (ii) by transmitting the impulse sequentially over the transmit
antennas. By default, we use the latter approach of sequential transmissions,
which as we show in the next section, allows us to faithfully capture
transmit-side spatial correlations. We sample the channel for a duration of
4.17 $\mu$s (thereby collecting 128 samples). We find the sampling duration is
sufficient to capture the last multipath component in our evaluation setting
(2.89 $\mu$s for TDL-A and 1.43 $\mu$s for TDL-B). As a result of this effort,
we collect 60K input-output measurements, out of which we use 60% for training
and reserve 20% each for validation and testing.

###  IV-B Results

In this section we walk through the evaluation of our MIMO-GAN approach
trained and evaluated on 3GPP TDL MIMO channels. Specifically, we evaluate the
MIMO-GAN for the TDL-A and TDL-B channel models and compare the statistics
with the underlying ground-truth channel.

TABLE I: Power and delay statistics of MIMO-GAN and ground-truth (GT) channels. |  | Total Power (dB) | Average Delay ($\mu$s) | RMS Delay Spread ($\mu$s)
---|---|---|---|---
TDL-A | MIMO-GAN | 4.648 | 0.2643 | 0.2862
GT | 4.628 | 0.2641 | 0.2897
MAE | -18.73 | 2.36$\times$10-4 | 3.57$\times$10-3
TDL-B | MIMO-GAN | 4.735 | 0.2276 | 0.2954
GT | 4.688 | 0.2285 | 0.2987
MAE | -14.95 | 9.05$\times$10-4 | 3.37$\times$10-3
TABLE II: MAEs of Power and Delay statistics comparing unconditioned (‘✗’) and conditioned (‘✓’) generation/discrimination of channels. Last row corresponds to MIMO-GAN. In each column, we represent the best performance in bold.  |  |  | TDL-A |  | TDL-B
---|---|---|---|---|---
Cond. G | Cond. D |  | Total power (dB) | Avg. Delay (ns) | RMS Spread (ns) |  | Total power (dB) | Avg. Delay (ns) | RMS Spread (ns)
✗ | ✗ |  | -11.47 | 49.00 | 82.00 |  | -0.13 | 105.20 | 152.8
✗ | ✓ |  | -11.67 | 38.10 | 68.00 |  | -6.03 | 82.94 | 133.4
✓ | ✗ |  | -12.25 | 3.76 | 7.93 |  | -11.74 | 18.72 | 23.01
✓ | ✓ |  | -18.73 | 0.24 | 3.57 |  | -14.95 | 0.90 | 3.37

Figure 2: Power Density Profile (TDL-A and TDL-B) Figure 3: Samples of
channels $h_{ij}$ drawn from MIMO-GAN.

Figure 4: Power profile and Spectral density (TDL-A) in 4$\times$4 MIMO
configuration (shown here for second and fourth antenna elements).

Power Delay Profile. We capture and compare the statistical parameters over a
set of 12K impulse responses generated using MIMO-GAN and compare it with the
parameters of the ground truth channel. Specifically, we compare the
statistical parameters defined in the ITU-R P.1407-8 [16] standards by using a
power cutoff value of 20 dB below peak value.

In Table I, we present the power delay statistics of our approach and ground-
truth channels. Across all metrics, we find that our approach can closely
capture the GT statistics. For instance, on TDL-A, we observe MIMO-GAN is
within -18.69 dB mean absolute error (MAE) of power compared to the GT channel
and within 3.57 ns of average delay.

Furthermore, in Figure 2, we display the power delay profile of the impulse
responses of both the GT channel and MIMO-GAN. Remarkably, MIMO-GAN implicitly
captures the impulse response of the learnt channel with reasonable accuracy;
we observe mean absolute errors of -21.12 dB (TDL-A) and -20.69 dB (TDL-B). We
find these results also extend when observing across all receive antenna
elements, as shown in Figure 4. As a result of implicitly modelling channe
distribution using MIMO-GAN, we can also leverage the generator to sample
channel instances, as shown in Figure 3.

Conditioned vs. Unconditioned $G$ and $D$. In the previous section, we
highlighted the consistency of MIMO-GAN with the underlying TDL-A and TDL-B
channels when observing the power and delay statistics. Now, we evaluate a key
aspect of MIMO-GAN: learning the conditional distribution of the channel,
where the channel generation is conditioned on the spatial co-ordinates of the
antenna element. MIMO-GAN achieves this re-using the same network $G$ and $D$,
but by additionally conditioning the network on spatial co-ordinates of tx-rx
antennas $(i,j)$. Now, we contrast the results by further examining an
unconditional architecture, where: (a) Generator $G$ generates the entire
channel tensor ${\mathbf{H}}\in\mathbb{C}^{N_{R}\times N_{T}\times L}$ (as
opposed to previously generating ${\bm{h}}_{ij}\in\mathbb{C}^{L}$); and (b)
Discriminator $D$ discriminating the multiple-output signal
${\mathbf{y}}\in{\mathbb{C}}^{N_{R}\times T}$ (as opposed to previously
discriminating ${\bm{y}}_{i}\in\mathbb{C}^{T}$. Such an unconditional
architecture is reminiscent of existing generative channel models approaches
which have been primarily studied in a SISO setup. Note that the unconditioned
architectures introduces a significant increase in the number of trainable
parameters (172K $\rightarrow$ 865K).

In Table II, we compare the results of MIMO-GAN (last row of table) with
conditioned $G$ and $D$ with unconditioned counter-parts (first three rows of
table). Remarkably, we find that the conditional architecture results in
performance improvements across all metrics in both TDL-A and TDL-B. For
instance, in TDL-A, we find an improvement of -12.37 dB (-11.47 dB vs. -18.73
dB) in total power and a 203$\times$ (49 ns vs. 0.24 ns) improvement in
capturing average delay statistics. We additionally investigate the influence
of individually conditioning $G$ and $D$, as shown by the second and third
rows of Table II. Here, we find conditional generation have a more significant
impact on improving performance than conditional discrimination. For instance,
in TDL-A, conditional generation reduces MAE of average delays by 45.24 ns
(92.3% decrease) whereas conditional discrimination reduces MAE only by 10.9
ns (22.2$ decrease). We attribute the performance improvements to the networks
being able to more efficiently exploit correlations of channels across various
links.

(a) Transmit-side correlations
$R_{TX}=\mathbb{E}[{\mathbf{H}}^{H}{\mathbf{H}}]$

(b) Receive-side correlations
$R_{RX}=\mathbb{E}[{\mathbf{H}}{\mathbf{H}}^{H}]$

Figure 5: Antenna Spatial Correlations in our 4$\times$4 MIMO setup.  TABLE III: Mean absolute errors between the GT and predicted antenna spatial correlations. ‘GM’ indicates discriminating additionally on the receive-side gram matrices ${\mathbf{y}}^{H}{\mathbf{y}}$ (Equation 5). ‘SQ’ indicates transmitting the digital impulse sequentially along the transmit antennas. In each column, we represent the best performance in bold and second-best in italics.  |  |  | TDL-A |  | TDL-B
---|---|---|---|---|---
GM | SQ |  | $R_{TX}$ | $R_{RX}$ |  | $R_{TX}$ | $R_{RX}$
✗ | ✗ |  | 0.144 | 0.171 |  | 0.139 | 0.143
✓ | ✗ |  | 0.204 | 0.061 |  | 0.154 | 0.056
✗ | ✓ |  | 0.071 | 0.177 |  | 0.067 | 0.164
✓ | ✓ |  | 0.059 | 0.057 |  | 0.058 | 0.063

Spatial Correlations. For MIMO evaluation, we also consider the transmit- and
receive-sided spatial correlations of responses between antenna elements. We
display the correlations across top 10 taps for TDL-A channels in Figure 5.
Overall, we observe promising results with mean absolute errors between the GT
and predicted spatial correlations: 0.057 (TDL-A) and 0.063 (TDL-B).

We find two steps especially crucial to successfully capture the MIMO spatial
correlations, as shown in Table III. First, to effectively capture the
receive-side correlations in the generated MIMO channels, we find it crucial
to additionally condition the discriminator on the gram matrices
(${\mathbf{y}}^{H}{\mathbf{y}}$) of output measurements. We find that
including this conditioning (second row in Table III) reduces the errors by
60-64%. Second, to capture the transmit-side correlations, we observe it is
essential to sequentially transmit the inputs over each transmit antenna.
Without this step, the receiver observes only the superposition of the
impulses and is thereby ill-posed to learn channel conditions at the transmit
side. As a result of sequential input transmissions (third row in Table III),
we can similarly reduce the transmit correlation errors by 51%. Overall, with
the above steps, we find that MIMO-GAN additionally captures transmit- and
receive-side antenna spatial correlations with mean absolute errors of
0.057-0.063 (final row in Table III).

TABLE IV: Simulation times. Speed-ups over GT TDL-A implementation. 1$\times$1
---
| Batched | GPU | simulation time (ms) | speed up
GT | ✗ | ✗ | 9.67 $\pm$ 55.69 | 1.00
MIMO-GAN | ✗ | ✗ | 0.53 $\pm$ 0.12 | 18.25
✓ | ✗ | 0.11 $\pm$ 0.01 | 87.91
✗ | ✓ | 0.99 $\pm$ 0.08 | 9.77
✓ | ✓ | 0.02 $\pm$ 0 | 483.50
2$\times$2
| Batched | GPU | simulation time (ms) | speed up
GT | ✗ | ✗ | 10.37 $\pm$ 54.39 | 1.00
MIMO-GAN | ✗ | ✗ | 1.40 $\pm$ 0.27 | 7.41
✓ | ✗ | 0.21 $\pm$ 0.02 | 49.38
✗ | ✓ | 2.23 $\pm$ 0.06 | 4.65
✓ | ✓ | 0.14 $\pm$ 0.01 | 74.07
4$\times$4
| Batched | GPU | simulation time (ms) | speed up
GT | ✗ | ✗ | 12.49 $\pm$ 56.07 | 1.00
MIMO-GAN | ✗ | ✗ | 3.91 $\pm$ 0.62 | 3.19
✓ | ✗ | 0.43 $\pm$ 0.04 | 29.05
✗ | ✓ | 6.10 $\pm$ 0.16 | 2.05
✓ | ✓ | 0.23 $\pm$ 0.01 | 54.30

Simulation Times. A side-benefit provided by neural surrogates for channel
model, such as by MIMO-GAN, is enabling rapid simulations. Such rapid
simulations makes MIMO-GAN especially appealing for end-to-end training
approaches [17]; such approaches typically require hundreds of thousands of
forward (simulations) and backward passes through the channel model. We
present the simulation times of TDL-A using MIMO-GAN in Table IV and contrast
it with a publicly available implementation (Matlab in this case). The table
reports the mean and standard deviations of 2048 simulations of a time-
invariant TDL-A channel. Overall (last rows in the table), we observe that
MIMO-GAN simulations are significantly faster, with speed-ups of
54.3-483$\times$. We attribute these speed-ups to a combination of: (a)
batching (independently leading to 29-88$\times$ speed-ups), as multiple
simulations (which in our case is a simple forward pass) can efficiently
parallelized; and (b) GPU usage (independently leading to 2-10$\times$ speed-
ups), as the effectiveness of parallelized operations can be further amplified
using GPU cores.

###  IV-C Discussion

In this paper, we proposed MIMO-GAN as a data-driven neural surrogate for
channel models. They key idea of MIMO-GAN is to implicitly learn the channel
distribution $p({\mathbf{H}})$ from input-output measurements. We discussed in
subsection IV-B that in addition to faithfully capturing the power delay
profile of the underlying channel, MIMO-GAN also offers many other benefits
such as capturing the MIMO spatial correlations and rapid simulation times.

We envision MIMO-GAN, or more generally neural channel models, are appealing
for many applications in wireless communications. We highlight two such
applications. First, neural channel models allow end-to-end learning [18, 19]
of the physical layer. In this case, parameters and operational configurations
on both the transmitter and receiver can be optimized in response to the
channel conditions experienced by the transceiver. Second, given that MIMO-GAN
depends on input-output measurement data (which can be collected on-device
with relative ease), the model can be rapidly built on-device and further
fine-tuned based on new measurement data. Having access to such a neural
channel model on-device opens up many possibilities, such as performing
inference [20] to obtain a compact channel representation.

While we found promising results with MIMO-GAN, our findings also highlights
many interesting open challenges. One such challenge is additionally modelling
time-variation. Although MIMO-GAN can be fine-tuned periodically to obtain
instantaneous channel representations, there is scope for a more principled
approach to model time-variation. Advances in neural stochastic processes [21]
might hold the key to model time-variations in statistical channels. Another
challenge is extending the approach to massive MIMO systems. While MIMO-GAN
provides the benefit of parameter efficiency in such systems, it is hindered
by computation complexity due to conditional sequential generation of
channels. We believe significant speed-ups, as to what might be needed for
massive MIMO, can be obtained by efficient parallelized implementation of the
generative process.

##  V Conclusion

In this paper, we proposed ‘MIMO-GAN’ towards building frequency-selective
MIMO channel models based on input-output measurements. The key benefits to
our approach, as a result of implicitly learning the channel distribution, are
interpretability (since instantaneous channels are visible) and being able to
generalize to configuration changes (e.g., sampling the channel for variable
durations). Furthermore, Based on evaluations on 3GPP TDL channels, we found
MIMO-GAN consistently captures the power delay statistics and additionally
spatial correlations between antennas.

## References

  * [1] ETSI 3rd Generation Partnership Project (3GPP), “Study on channel model for frequencies from 0.5 to 100 ghz (version 16.1.0),” _3GPP TR 38.901, Jun, Tech. Rep._ , 2020.
  * [2] A. Bora, A. Jalal, E. Price, and A. G. Dimakis, “Compressed Sensing using Generative Models,” in _PMLR_ , Jul. 2017, pp. 537–546. [Online]. Available: http://proceedings.mlr.press/v70/bora17a.html
  * [3] W. C. Jakes and D. C. Cox, _Microwave mobile communications_. Wiley-IEEE press, 1994.
  * [4] F. A. Aoudia and J. Hoydis, “End-to-End Learning of Communications Systems Without a Channel Model,” in _2018 52nd Asilomar Conference on Signals, Systems, and Computers_ , Oct. 2018, pp. 298–303.
  * [5] T. J. O’Shea, T. Roy, and N. West, “Approximating the void: Learning stochastic channel models from observation with variational generative adversarial networks,” in _ICNC_ , 2019.
  * [6] H. Ye, G. Y. Li, B.-H. F. Juang, and K. Sivanesan, “Channel agnostic end-to-end learning based communication systems with conditional gan,” in _IEEE Globecom Workshops_ , 2018.
  * [7] Y. Yang, Y. Li, W. Zhang, F. Qin, P. Zhu, and C.-X. Wang, “Generative-adversarial-network-based wireless channel modeling: Challenges and opportunities,” _IEEE Communications Magazine_ , 2019.
  * [8] H. Ye, L. Liang, G. Y. Li, and B.-H. Juang, “Deep learning-based end-to-end wireless communication systems with conditional gans as unknown channels,” _IEEE Transactions on Wireless Communications_ , 2020.
  * [9] S. Dörner, M. Henninger, S. Cammerer, and S. ten Brink, “Wgan-based autoencoder training over-the-air,” in _IEEE International Workshop on Signal Processing Advances in Wireless Communications_ , 2020.
  * [10] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in _ICLR_ , 2015.
  * [11] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [12] M. Hein, M. Andriushchenko, and J. Bitterwolf, “Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem,” in _CVPR_ , 2019.
  * [13] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _NeurIPS_ , 2014.
  * [14] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in _ICML_ , 2017.
  * [15] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, “Improved training of wasserstein gans,” in _NeurIPS_ , 2017.
  * [16] R. S. of International Telecommunication Union (ITU-R), “Multipath propagation and parameterization of its characteristics,” _Recommendation ITU-R P.1407-8_ , 2021.
  * [17] V. Raj and S. Kalyani, “Backpropagating through the air: Deep learning at physical layer without channel models,” _IEEE Communications Letters_ , 2018\.
  * [18] T. O’shea and J. Hoydis, “An introduction to deep learning for the physical layer,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 3, no. 4, pp. 563–575, 2017.
  * [19] S. Dörner, S. Cammerer, J. Hoydis, and S. Ten Brink, “Deep learning based communication over the air,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 12, no. 1, pp. 132–143, 2017.
  * [20] J. Donahue, P. Krähenbühl, and T. Darrell, “Adversarial feature learning,” in _ICLR_ , 2017.
  * [21] R. Deng, B. Chang, M. A. Brubaker, G. Mori, and A. Lehrmann, “Modeling continuous stochastic processes with dynamic normalizing flows,” _NeurIPS_ , 2020.

  *[GAN]: generative adversarial network
  *[CIR]: channel impulse response