# Dynamical uncertainty propagation with noisy quantum parameters

Mogens Dalgaard  Department of Physics and Astronomy, Aarhus University, Ny
Munkegade 120, 8000 Arhus C, Denmark  Carrie A. Weidner  Department of Physics
and Astronomy, Aarhus University, Ny Munkegade 120, 8000 Arhus C, Denmark
Felix Motzoi<EMAIL_ADDRESS>Forschungszentrum Jülich, Institute of
Quantum Control (PGI-8), D-52425 Jülich, Germany

###### Abstract

Many quantum technologies rely on high-precision dynamics, which raises the
question of how these are influenced by the experimental uncertainties that
are always present in real-life settings. A standard approach in the
literature to assess this is Monte Carlo sampling, which suffers from two
major drawbacks. First, it is computationally expensive. Second, it does not
reveal the effect that each individual uncertainty parameter has on the state
of the system. In this work, we evade both these drawbacks by incorporating
propagation of uncertainty directly into simulations of quantum dynamics,
thereby obtaining a method that is faster than Monte Carlo simulations and
directly provides information on how each uncertainty parameter influence the
system dynamics. Additionally, we compare our method to experimental results
obtained using the IBM quantum computers.

Technologies based on quantum information may lead to groundbreaking progress
within numerical search Grover (1996), cryptography Shor (1999), simulation
Barreiro  _et al._ (2011); Bloch  _et al._ (2012), optimization Das and
Chakrabarti (2008); McGeoch (2014), and machine learning Schuld  _et al._
(2015); Biamonte  _et al._ (2017). Realization of these technologies requires
improvements in our ability to measure, design, build, and realistically model
subparts of these quantum systems Acín  _et al._ (2017). The latter
necessitates the incorporation of the experimental uncertainties that are
always present in real quantum systems into numerical simulations of such
systems.

Propagation of uncertainty (or error) is a standard tool for understanding how
experimental uncertainties transform in calculations Taylor (1997). In this
work we incorporate propagation of uncertainty into quantum dynamical
simulation and characterization by drawing on methods from quantum control
theory Khaneja  _et al._ (2005); Motzoi  _et al._ (2011); De Fouquieres  _et
al._ (2011); Dalgaard  _et al._ (2020). In doing so we demonstrate a
significant speed up relative to Monte Carlo sampling, which is the current
standard in the literature (see, e.g., Refs. Langrock and DiVincenzo (2020);
Sørensen  _et al._ (2020); Ge  _et al._ (2020); Vieira and Rigolin (2020);
Song  _et al._ (2021); Estarellas  _et al._ (2020); Coden  _et al._ (2021);
Cabedo  _et al._ (2021); Kiely and Campbell (2021); D’Angelis  _et al._
(2020); Calderon-Vargas  _et al._ (2021); Wu  _et al._ (2021); Giorgi  _et
al._ (2020); Hu  _et al._ (2020) for a series of recent papers).

Moreover, our method allows for an in-depth analysis of how experimental
uncertainties independently or cooperatively influence the dynamical
observables in the system. This is valuable for investigating the consistency
between models and experimental results, as well as the characterization of
noise, uncertainty levels, and the influence of different experimental
parameters. As one example, within circuit QED we have, in recent years, seen
improvements in one- and two-qubit gate operations due to better understanding
and calibration of the experimental systems Sheldon  _et al._ (2016); Theis
_et al._ (2018); Patterson  _et al._ (2019); Magesan and Gambetta (2020);
Sundaresan  _et al._ (2020). Careful study of these systems’ uncertainties and
their influence on quantum dynamics is necessary to push this frontier even
further.

The methods proposed in this paper could also help in the construction of
composite control sequences that are specifically designed to perform robustly
against fluctuations in the system parameters Wimperis (1994); Cummins  _et
al._ (2003); Vandersypen and Chuang (2005); Möttönen  _et al._ (2006), as well
as enhancing quantum control and optimization protocols Khaneja  _et al._
(2005); Borneman  _et al._ (2010); Skinner  _et al._ (2011); Motzoi  _et al._
(2016); Platzer  _et al._ (2010); Sørensen  _et al._ (2020); Ge  _et al._
(2020). That is, the sensitivity to parameter uncertainties can be modified as
needed for tomographic reconstructions D’Ariano and Presti (2001); Dankert
_et al._ (2009), system identification Mabuchi (1996); Schirmer and Oi (2010);
Burgarth and Yuasa (2012); Zhang and Sarovar (2014); Guţă and Yamamoto (2015),
and quantum metrology Braunstein and Caves (1994); Giovannetti  _et al._
(2004, 2011). In addition, the presented method could also aid physicists in
designing quantum computation architectures Koch  _et al._ (2007); Rigetti and
Devoret (2010); Goerz  _et al._ (2017); Brooks  _et al._ (2013); Menke  _et
al._ (2021); Nguyen  _et al._ (2019). These need to perform stably under
smaller fluctuations in the system parameters, and as such, a detailed
analysis of their performance under realistic and expected uncertainties could
help assess the robustness of a given architecture. We term our approach
Quantum Uncertainty Propagation In Dynamics (QUPID).

This paper is organized as follows: in Section I we introduce QUPID, in
Section II we compare to Monte Carlo simulations, and in Section III we
compare to experimental results. Further, in Section IV we investigate the
scaling of QUPID and finally, in Section V, we conclude the paper.

Figure 1: Quantum uncertainty propagation when applied to a Gaussian pulse.
(a) The uncertain Gaussian pulse (solid, see main text for details) and the
result of Monte Carlo sampling (dashed line, with $2\sigma$ confidence
interval shaded in red). (b) Comparison between propagation of uncertainty and
Monte Carlo sampling, zoomed in on later times. Here, the Monte Carlo mean
(solid line) and confidence interval (shading) are shown in red, and the mean
and confidence intervals derived from our method are shown with the black
dotted and dashed lines, respectively, showing excellent agreement relative to
the Monte Carlo case. The mean for the uncertainty-free case is shown by the
green dash-dotted line (c) The relative error contribution as a function of
time, i.e., the contribution to the shift from the uncertainty-less case
resulting from uncertainties in the maximum drive amplitude
$\Omega_{\mathrm{max}}$ (blue, solid) and detuning $\Delta$ (red, dashed).

##  I Theory

Let $Z(t)$ denote a function that depends on the evolution of the system, such
as the expectation value of a Hermitian operator $Z(t)=\braket{\hat{Z}}$, or
the Haar-averaged gate fidelity
$Z(t)=\int_{\psi}\bra{\psi}U(t)^{\dagger}U_{target}\ket{\psi}d\psi$. Now let
the Hamiltonian operator and other system evolution variables depend on a set
of normally distributed parameters
$\boldsymbol{\theta}=(\theta_{1},\theta_{2},\ldots,\theta_{M})$ with mean
$\bar{\boldsymbol{\theta}}=(\bar{\theta_{1}},\bar{\theta_{2}},\ldots,\bar{\theta_{M}})$
and covariances
$\sigma_{i,j}=\mathbb{E}[(\theta_{i}-\bar{\theta_{i}})(\theta_{j}-\bar{\theta_{j}})]$,
which we assume are relatively small 111Note that the expression we derive for
the mean works for any symmetric distribution.. Here
$\sigma_{j}^{2}=\sigma_{j,j}$ denotes the variance. Thus, $Z$ is a function of
the parameters $\boldsymbol{\theta}$, which we may Taylor expand around the
mean values $\bar{\boldsymbol{\theta}}$,

$\displaystyle Z(\boldsymbol{\theta},t)$ $\displaystyle\simeq
Z(\bar{\boldsymbol{\theta}},t)+\sum_{j}\frac{\partial
Z(\bar{\boldsymbol{\theta}},t)}{\partial\theta_{j}}(\theta_{j}-\bar{\theta}_{j})$
$\displaystyle+\frac{1}{2}\sum_{i,j}\frac{\partial^{2}Z(\bar{\boldsymbol{\theta}},t)}{\partial\theta_{i}\partial\theta_{j}}(\theta_{i}-\bar{\theta}_{i})(\theta_{j}-\bar{\theta}_{j}),$
(1)

where truncating the Taylor expansion to second order is justified by assuming
small covariances. Evaluating the mean value reveals

$\displaystyle\mathbb{E}[Z(\boldsymbol{\theta},t)]$ $\displaystyle\simeq
Z(\boldsymbol{\bar{\theta}},t)+\frac{1}{2}\sum_{i,j}\frac{\partial^{2}Z(\bar{\boldsymbol{\theta}},t)}{\partial\theta_{i}\partial\theta_{j}}\sigma_{i,j}.$
(2)

From Eq. (2) we can see directly how uncertainties in the system dynamics may
lead to shifts away from the expected value $Z(\boldsymbol{\bar{\theta}},t)$
at different times. For example, this term could cause a reduction in the
predicted average gate fidelity. This is especially relevant to analytical
protocols for parameter estimation, where stochasticity in the parameter may
itself modify the mean estimation. Moreover, the intuitive approach of
assessing robustness via first derivatives will fail to capture the expected
loss in average fidelity, as its effect will simply average to zero. In the
case that the uncertainties are statistically independent ($\sigma_{i,j}=0$ if
$i\neq j$), we can further see that the individual parameter uncertainties are
additive, allowing them to be calculated individually by, e.g., Monte Carlo
simulations, or more efficiently, as we show below, by propagation of
uncertainty through the dynamics.

A similar calculation for the variance (see Supplemental Material) yields

$\displaystyle\text{Var}[Z(\boldsymbol{\theta},t)]$
$\displaystyle\simeq\sum_{i,j}\frac{\partial
Z(\bar{\boldsymbol{\theta}},t)}{\partial\theta_{i}}\frac{\partial
Z(\bar{\boldsymbol{\theta}},t)}{\partial\theta_{j}}\sigma_{i,j}$
$\displaystyle+\frac{1}{2}\sum_{i,j,k,l}\frac{\partial^{2}Z(\bar{\boldsymbol{\theta}},t)}{\partial\theta_{i}\partial\theta_{j}}\frac{\partial^{2}Z(\bar{\boldsymbol{\theta}},t)}{\partial\theta_{k}\partial\theta_{l}}\sigma_{i,k}\sigma_{j,l}.$
(3)

We now sketch how to obtain the derivatives that appear in Eq. (2) and (3)
through analysis of the dynamics.

In order to be able to calculate the dynamical uncertainties in system
observables with the above equations, it is necessary to be able to
efficiently propagate the equations of motion for the derivatives of these
observables. To do this, we draw on established methods from the field of
quantum optimal control theory Khaneja  _et al._ (2005); De Fouquieres  _et
al._ (2011); Motzoi  _et al._ (2011); Dalgaard  _et al._ (2020). We have so
far tacitly assumed that $Z$ was a functional of the current state of the
system $\chi(t)$, which solves the dynamic equation of motion
$\frac{\partial}{\partial t}\chi(t)=A(t)\chi(t)$. This could be, e.g., the
Schrödinger equation $A(t)=-iH(t)$ with $\hbar=1$ and $\chi$ being a single
quantum state, a unitary operator, or a collection of states, or a density
matrix with evolution governed by the von Neumann or the Lindblad master
equation. We elaborate further on this in the Supplemental Material.

We discretize the time evolution for a total duration $T$ in $N$ equidistant
steps $\Delta t$ such that $\Delta t=T/N$. This allows us to numerically solve
the equation of motion using the time evolution operator
$\chi_{n+1}=U_{n}\chi_{n}$, where $\chi(t_{n})=\chi_{n}$ and
$U_{n}=e^{A(t_{n}+\Delta t/2)}$ is the mid-point interpolation of the
truncated Magnus series Blanes  _et al._ (2009), leading to a global second
order-error in $\chi(T)$ that can be made arbitrarily small by suitable choice
of $\Delta t$. The derivatives of $Z$ ultimately depend on the derivative of
$\chi$, which we may obtain by using the chain rule

$\displaystyle\frac{\partial}{\partial\theta_{j}}\chi_{n+1}=\Big{(}\frac{\partial}{\partial\theta_{j}}U_{n}\Big{)}\chi_{n}+U_{n}\frac{\partial}{\partial\theta_{j}}\chi_{n},$
(4)

and similarly we obtain

$\displaystyle\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}\chi_{n+1}$
$\displaystyle=\Big{(}\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}U_{n}\Big{)}\chi_{n}+\Big{(}\frac{\partial}{\partial\theta_{j}}U_{n}\Big{)}\frac{\partial}{\partial\theta_{i}}\chi_{n}$
$\displaystyle+\Big{(}\frac{\partial}{\partial\theta_{i}}U_{n}\Big{)}\frac{\partial}{\partial\theta_{j}}\chi_{n}+U_{n}\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}\chi_{n}.$
(5)

To calculate the derivatives for each time slice one should solve Wilcox
(1967)

$\frac{d}{d\eta}e^{\chi(\eta)}=\int_{0}^{1}e^{\alpha\chi(\eta)}\frac{d\chi(\eta)}{d\eta}e^{(1-\alpha)\chi(\eta)}d\alpha.$
(6)

which has a known solution in terms of the eigenvalues of $\chi$ De Fouquieres
_et al._ (2011); Machnes  _et al._ (2011); Dalgaard  _et al._ (2020). Further
steps in the derivation are given in the Supplemental Material.

Figure 2: Comparison of QUPID to experimental results using the IBM Armonk
qubit showing how the uncertainty changes over time for a Gaussian (a) and BB1
(b) $\pi$ pulse and a Gaussian (c) and BB1 (d) $\pi/2$ pulse. The black points
(error bars) denote the mean (twice the standard deviation of the mean) for
$50$ experimental runs, where for each run, we sample the amplitude from a
normal distribution of width $\epsilon=0.05$. The blue points and error bars
were generated from data taken with a constant amplitude, i.e., $\epsilon=0$.
In both cases, each experimental run outputs the average results over $1024$
measurements. The theoretical BB1 uncertainty goes to zero at the endpoints
since the BB1 pulses are robust up to sixth-order error in amplitude, but
otherwise we obtain excellent agreement between theory and experiment when the
amplitude uncertainty errors dominate over other errors in the system, e.g.,
measurement errors. The inset in (c) compares the predictions of the final
expectation value $\braket{Z}$ to experimental data with respect to changes in
the amplitude error $\epsilon$ for a Gaussian $\pi/2$ pulse. Plots (a) and (c)
zoom in on the first Gaussian pulse of the BB1 sequence (marked with a
vertical dashed line); some data points have been omitted in the BB1 plots for
clarity.

##  II Comparison to Monte Carlo simulations

We consider here a two-level Hamiltonian given by ($\hbar=1$)

$\displaystyle H(t)=\Delta\hat{Z}+\Omega_{\max}u(t)\hat{X},$ (7)

with detuning $\Delta=0$, maximum amplitude $\Omega_{\max}$, Pauli operators
$\hat{X}$ and $\hat{Z}$, and dimensionless control function $-1\leq
u(t)\leq+1$. We consider a state transfer $\ket{0}\rightarrow\ket{1}$ where
the dynamical evolution is subject to uncertainties in the system parameters
$\Omega_{\max}$ and $\Delta$. The transfer itself is induced by Gaussian
$\pi$-pulse (Fig. 1(a)). We time-evolve the system for a duration
$T=6/\Omega_{\max}$ with independent uncertainties
$\sigma(\Omega_{\max})/\Omega_{\max}=\sigma(\Delta)/\Omega_{\max}=3.0\times
10^{-2}$ using 10,000 Monte Carlo simulations. We calculate the average
trajectory over the Pauli observable $\braket{\hat{Z}}$ and define the
confidence interval as twice the standard deviation $\pm
2\sigma(\braket{\hat{Z}})$ from the average trajectory, which we truncate
between $\pm 1$, as this is a physical limitation on $\braket{\hat{Z}}$ (Fig.
1(a)). We now compare the predictions of QUPID, Eqs. (2) and (3), to the Monte
Carlo simulations by zooming in on the end of the trajectory in Fig. 1(b).
Here we see how uncertainties in the system parameters induce a shift in the
mean value away from the ideal, uncertainty-less case $\braket{\hat{Z}}=-1$.
In addition, we see an excellent agreement between the Monte Carlo simulations
and uncertainty propagation predictions. On a standard laptop computer, the
Monte Carlo simulations take $\approx 30$ minutes, whereas QUPID takes
$\approx 0.2$ seconds, thereby providing a significant speed up.

An additional benefit of QUPID over Monte Carlo methods is that Eq. (2) allows
us to directly track how each uncertainty parameter influences the system
dynamics. To this end, we may define the error contribution from Eq. (2) for
each parameter as
$\frac{1}{2}\frac{\partial^{2}\braket{\hat{Z}}}{\partial\theta^{2}}\sigma(\theta)^{2}$
with $\theta=\Omega_{\max},\Delta$. We plot in Fig. 1(c) the error
contribution for each parameter along the trajectory. Interestingly, we see
that around half-way through the trajectory, the error contributions from the
amplitude and detuning are opposite in sign and thus cancel each other out to
some extent; such fortuitous cancellations can be analyzed and exploited when
designing pulse sequences using our method. We also see at the end of the
trajectory that the two error contributions converge on different specific
values, i.e. uncertainty in the amplitude and detuning constitute respectively
around $59\%$ and $41\%$ of the total error. This analysis is useful in
determining, for example, which parameters are most sensitive in an experiment
and thus where efforts must be concentrated when improving the precision of
experimental systems.

##  III Experimental comparison

We compare the predictions of QUPID to a single-qubit experiment via pulse-
level control of IBM’s _ibmq_armonk_ single-qubit system accessible via Qiskit
pulse McKay  _et al._ (2018); Alexander  _et al._ (2020a, b). Such pulse-level
control has previously been used to optimize the fidelity of cross-resonance
gates Sheldon  _et al._ (2016), the CV-gate Oomura  _et al._ (2021), and to
build a quantum compiler implementing basis gates and qudit operations Gokhale
_et al._ (2020).

First, we determine Gaussian and composite BB1 Wimperis (1994) pulse sequences
that drive $\pi/2$ (Hadamard) and $\pi$ (bit-flip) transitions. After
calibration (see Supplemental Material), we artificially add amplitude noise
to each of the aforementioned pulse sequences. The amplitude noise is sampled
from a Gaussian distribution centered around the optimal amplitudes with a
width of $\epsilon=\sigma(\Omega_{\mathrm{max}})/\Omega_{\mathrm{max}}$ for
$\Theta=\pi/2$ and $\pi$ 222For the qubit drive times considered here, the
intrinsic qubit errors, e.g. due to relaxation, are negligible, so this system
as designed is a suitable playground for artificially-inserted uncertainties
that can be used to test our theory.. We compare the predictions of our theory
to the experimental results in Fig. 2 with $\epsilon=0.05$ for Gaussian and
BB1 $\pi$ and $\pi/2$ pulses. The data and simulation match well throughout
the duration of the pulses for both cases, although our theory cannot fully
account for measured BB1 errors at the final time due to discrimination/shot
noise (see blue error bars) being larger than the BB1 final error, which has
intrinsic robustness up to sixth order in amplitude error Wimperis (1994);
Vandersypen and Chuang (2005). To determine how the error scales with
$\epsilon$, we compare the experimental data and theoretical predictions for
the final expectation value $\braket{\hat{Z}}$ of a Gaussian $\pi/2$ pulse in
the inset of Fig. 2(c), where we again obtain excellent agreement between
theory and experiment. Thus, when compared with experimental data, our theory
can determine at what point various experimental uncertainties dominate the
error, which can be useful for experimental debugging, theoretical model-
building, or designing new systems. Moreover, by calibrating the exact value
of $\epsilon$ in the model, one can readily use Eq. (2) to remove the
correction term coming from uncertainty and thereby estimate the true model
parameters, e.g. for Hamiltonian learning or quantum sensing.

Figure 3: Simulation results with the spin-star system (top right inset). The
lines depict the expected error of Monte Carlo simulations as a function of
the number of fidelity evaluations for 1 (blue solid), 10 (orange dashed), 20
(green, dash-dotted), and 30 (red dotted) parameters. This is compared to
QUPID for 1 (blue circle), 10 (orange, down triangle), 20 (green, up triangle)
and 30 (red square) parameter(s) chosen at random, where the computational
expense has been evaluated with respect to a finite difference scheme. The
horizontal arrows mark the speed improvement in an equal-accuracy comparison.

##  IV Scaling and speed up

Here, we consider how QUPID scales with the number of parameters compared to
Monte Carlo simulations. In the case of independent uncertainties, estimating
the mean via Eq. (2) scales linearly with the number of parameters, while it
scales quadratically in the more general case of dependent uncertainties. If
we compare to Monte Carlo simulations, the averaging error over $N$ samples is
given by the standard deviation of the mean $\bar{\sigma}=\sigma/\sqrt{N}$,
leading to a $N^{-1/2}$ scaling. We may consider the scaling of the standard
deviation by considering the square root of Eq. (3). In the general case of
dependent uncertainties, Eq. (3) has a fourth order scaling with the number of
parameters, which implies that the standard deviation would scale
quadratically. In the case of independent parameters, this reduces to a linear
scaling. In other words, the scaling of the two approaches with respect to the
number of uncertainty parameters remains, in principle, the same.

However, in practice, many of the mixed derivatives in Eq. (3) tend to be
zero, leading to an effective sub-linear and sub-quadratic scaling of Monte
Carlo simulations with independent and dependent uncertainties, respectively.
For this reason, we should compare the precision of the two methods over a
wide range of uncertainty parameters. We consider the spin-star system
illustrated in the top right of Fig. 3. The system consists of 5 qubits with
interactions between the nearest and the next-nearest spins, where we model an
external, global $X$-control for all qubits. The Hamiltonian for the spin-star
system is given by

$\displaystyle H$ $\displaystyle=u(t)\sum_{j}\hat{X}_{j}$
$\displaystyle-\sum_{\braket{i,j}}\big{(}J_{x}^{(i,j)}\hat{X}_{i}\hat{X}_{j}+J_{y}^{(i,j)}\hat{Y}_{i}\hat{Y}_{j}+J_{x}^{(i,j)}\hat{Z}_{i}\hat{Z}_{j}\big{)}$
$\displaystyle-\sum_{\langle\langle
i,j\rangle\rangle}\big{(}g_{x}^{(i,j)}\hat{X}_{i}\hat{X}_{j}+g_{y}^{(i,j)}\hat{Y}_{i}\hat{Y}_{j}+g_{x}^{(i,j)}\hat{Z}_{i}\hat{Z}_{j}\big{)},$
(8)

where $\braket{,}$ denotes the sum over neighboring qubits, while
$\braket{\braket{,}}$ denotes next-neighboring qubits. $J=J_{k}^{(i,j)}$
denotes the nearest coupling and we choose the next-nearest coupling
$g=g_{k}^{(i,j)}=J/10$ for $k=x,y,z$ and all indices $(i,j)$. With this model
we have $30$ different parameters. We start in the initial state
$\ket{\psi_{0}}=\ket{0,0,0,0,0}$ and evolve with a control pulse of the form
$u(t)=\phi_{1}(t)+\phi_{3}(t)-\phi_{5}(t)$, where $\phi_{n}(t)=\sin(n\pi t/T)$
for a total duration $TJ=10$. In the absence of uncertainties this leads to a
final state $\ket{\tilde{\psi}}=\ket{\psi(T)}$ that populates a wide range of
different Fock states in the Hilbert space.

We investigate how uncertainties in the system parameters induces population
in states other than $\ket{\tilde{\psi}}$, which can be described by the
fidelity $F=\braket{P}$, with $P=\ket{\tilde{\psi}}\bra{\tilde{\psi}}$
denoting the projector onto the expected final state. Again, we assume
independent normally-distributed uncertainties, with relative amplitude
$\sigma(J)/J=\sigma(g)/g=0.01$, which we model in a random subset of the $30$
system parameters. This generally leads to a population leakage away from the
uncertainty-less case with an error $1-F$ in the range from $10^{-2}$ to
$10^{-4}$ depending on the number of parameters.

We model uncertainties in $1$, $10$, $20$, and $30$ parameters chosen at
random, where we perform 100,000 Monte Carlo simulations for each choice. To
understand the scaling of the Monte Carlo method, we compare the difference
between the fidelity average over the 100,000 samples ($F_{\text{true}}$)
versus only averaging over a subset of the simulations ranging from $1$ to
10,000 ($F_{\text{approx}}$).

For each given subset, we draw a random number of fidelities from the entire
set of simulated fidelities. This procedure is repeated 10,000 times in order
to calculate the mean estimation error i.e., average over
$|F_{\text{true}}-F_{\text{approx}}|$. The results are depicted in Fig. 3 for
different numbers of uncertainty parameters. Note the lines scale as
$N^{-1/2}$, consistent with the standard deviation of the mean, but at
different heights due to different standard deviations over the fidelity
distributions.

We may compare the performance of QUPID to Monte Carlo simulations in both
accuracy and computational expense. The latter is achieved by comparing the
number of fidelity evaluations required to calculate the derivatives using a
finite difference scheme (see Supplemental Material). Since evaluating the
fidelities is by far the most expensive step in each approach, this gives an
accurate comparison. We plot the results of QUPID in Fig. 3 as individual data
points.

We may read the speed up of QUPID relative to Monte Carlo simulations in an
equal-accuracy comparison by considering the distance from the individual data
points in Fig. 3 to the lines along the first axis. For clarity we have
indicated the factor of speed up with arrows on the figure, which ranges from
$6818$ for one parameter to $109$ for $30$ parameters. Note that the relative
speed up grows smaller with the number of parameters due to the sub-linear
scaling of Monte Carlo. This implies that for a large enough number of
uncertainty parameters, Monte Carlo would be preferable. However, we have
still demonstrated significant improvements with up to 30 parameters, which
would comprise the vast majority of cases typically considered in the
literature Langrock and DiVincenzo (2020); Vieira and Rigolin (2020); Song
_et al._ (2021); Estarellas  _et al._ (2020); Coden  _et al._ (2021); Cabedo
_et al._ (2021); Kiely and Campbell (2021); D’Angelis  _et al._ (2020);
Calderon-Vargas  _et al._ (2021); Wu  _et al._ (2021); Giorgi  _et al._
(2020); Hu  _et al._ (2020).

##  V Outlook

In this work, we have demonstrated QUPID, a method for analytically
incorporating and propagating uncertainties during quantum dynamical
evolution. For a reasonable number of parameters (at least up to 30), this
method performs dramatically better than the standard, cutting-edge methods
that are commonly used in the literature, namely Monte Carlo sampling. This
analytical form has the additional advantages of (a) being able to separate
the role of different error sources and (b) understanding how these
uncertainties develop over time. This can be useful for theoretical protocol
improvement but also for experimental design and debugging. That is, with an
understanding of the dominant uncertainty sources in a given protocol,
experimentalists can better understand the level of precision that must be
reached in their system and which parameters are most likely to cause
imprecisions. Likewise, these analytical forms allow the design of protocols
(e.g. pulse sequences) that may either decrease or increase the effect of
parameter uncertainty, which is useful for increased robustness or improved
sensing, respectively.

###### Acknowledgements.

We acknowledge the use of IBM Quantum services for this work. The views
expressed are those of the authors, and do not reflect the official policy or
position of IBM or the IBM Quantum team. This work was funded by the Carlsberg
Foundation and by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under Germany’s Excellence Strategy – Cluster of Excellence Matter
and Light for Quantum Computing (ML4Q) EXC 2004/1 – 390534769. The numerical
results presented in this work were obtained at the Centre for Scientific
Computing, Aarhus, phys.au.dk/forskning/cscaa.

## Supplementary Material

### Calculation of variance

We start by evaluating the general expression of the variance based on the
results derived in the main text

$\displaystyle\text{Var}[Z(\boldsymbol{\theta})]$
$\displaystyle=\mathbb{E}\Big{[}\big{(}Z(\boldsymbol{\theta},t)-\mathbb{E}[Z(\boldsymbol{\theta},t)]\big{)}^{2}\Big{]}$
$\displaystyle=\mathbb{E}\Big{[}\sum_{i,j}\frac{\partial
Z(\boldsymbol{\bar{\boldsymbol{\theta}}},t)}{\partial\theta_{i}}\frac{\partial
Z(\boldsymbol{\bar{\boldsymbol{\theta}}},t)}{\partial\theta_{j}}(\theta_{i}-\mu_{i})(\theta_{j}-\mu_{j})$
$\displaystyle+\sum_{i,j,k}\frac{\partial
Z(\boldsymbol{\bar{\boldsymbol{\theta}}},t)}{\partial\theta_{k}}\frac{\partial^{2}Z\boldsymbol{\bar{\boldsymbol{\theta}}},t)}{\partial\theta_{i}\partial\theta_{j}}(\theta_{k}-\mu_{k})(\theta_{i}-\mu_{i})(\theta_{j}-\mu_{j})$
$\displaystyle-\sum_{i,j,k}\frac{\partial
Z(\boldsymbol{\bar{\boldsymbol{\theta}}},t)}{\partial\theta_{k}}\frac{\partial^{2}Z(\boldsymbol{\bar{\boldsymbol{\theta}}},t)}{\partial\theta_{i}\partial\theta_{j}}\sigma_{i,j}(\theta_{k}-\mu_{k})$
$\displaystyle+\frac{1}{4}\sum_{i,j,k,l}\frac{\partial^{2}Z(\boldsymbol{\bar{\boldsymbol{\theta}}},t)}{\partial\theta_{i}\partial\theta_{j}}\frac{\partial^{2}Z(\boldsymbol{\bar{\boldsymbol{\theta}}},t)}{\partial\theta_{k}\partial\theta_{l}}(\theta_{i}-\mu_{i})(\theta_{j}-\mu_{j})(\theta_{k}-\mu_{k})(\theta_{l}-\mu_{l})$
$\displaystyle-\frac{1}{2}\sum_{i,j,k,l}\frac{\partial^{2}Z(\boldsymbol{\bar{\boldsymbol{\theta}}},t)}{\partial\theta_{i}\partial\theta_{j}}\frac{\partial^{2}Z(\boldsymbol{\bar{\boldsymbol{\theta}}},t)}{\partial\theta_{k}\partial\theta_{l}}\sigma_{k,l}(\theta_{i}-\mu_{i})(\theta_{j}-\mu_{j})+\frac{1}{4}\sum_{i,j,k,l}\frac{\partial^{2}Z(\boldsymbol{\bar{\boldsymbol{\theta}}},t)}{\partial\theta_{i}\partial\theta_{j}}\frac{\partial^{2}Z(\boldsymbol{\bar{\boldsymbol{\theta}}},t)}{\partial\theta_{k}\partial\theta_{l}}\sigma_{i,j}\sigma_{k,l}\Big{]}.$
(9)

In order to evaluate the above we use Wick’s probability theorem Wick (1950)
(also known as Isserlis’ theorem Isserlis (1918)), which states that the mean
of product of $n$ zero-mean normal distributed variables $\\{Y_{j}\\}_{j}^{n}$
is given by

$\displaystyle\mathbb{E}\Big{[}Y_{1}Y_{2}\ldots Y_{n}\Big{]}=\sum_{p\in
P_{n}^{2}}\prod_{(i,j)\in p}\sigma_{i,j}.$ (10)

Here the sum is over all distinct partitions $P_{n}^{2}$ of
$\\{Y_{j}\\}_{j}^{n}$ into pairs. Since this cannot be done for an odd number
$n$, the above gives zero for all odd $n$. From this we e.g. have

$\displaystyle\mathbb{E}\Big{[}Y_{1}Y_{2}Y_{3}\Big{]}=0;\quad\mathbb{E}\Big{[}Y_{1}Y_{2}Y_{3}Y_{4}\Big{]}=\sigma_{1,2}\sigma_{3,4}+\sigma_{1,3}\sigma_{2,4}+\sigma_{1,4}\sigma_{2,3}.$
(11)

Applying Wick’s probability theorem to Eq. (9) and using that the order of
differentiation does not matter we obtain the experssion for the variance as
stated in the main text.

### Numerical calculation of derivatives

In this work we seek to solve the equation of motion for a quantum system on
the form

$\displaystyle\frac{\partial}{\partial t}\chi(t)=A(t)\chi(t),$ (12)

which could be the Schrödinger equation with $A(t)=-iH(t)$ and $\chi(t)$ being
a single state $\chi(t)=\ket{\psi(t)}$, a unitary $\chi(t)=U(t)$, or a
collection of $K$ states with the $n$th column of $\chi$ being the $n$th state
$\chi(t)=\big{[}\ket{\psi_{1}(t)},\ket{\psi_{2}(t)},\ldots,\ket{\psi_{K}(t)}\big{]}$.
Eq. (12) could also be the von Neumann equation or the master equation on the
Linblad form, where $A(t)=\mathbb{L}(t)$ denotes the Liouvillian and
$\chi=\ket{\rho}$ the vectorized form of the density matrix i.e.
$\ket{\rho}_{n+(m-1)d}=\rho_{n,m}$, with $d$ denoting the Hilbert space
dimension Navarrete-Benlloch (2015). In either case we may numerically solve
the equation of motion by propagating the state in sufficiently small time-
steps $\Delta t$ via the time evolution operator elaborated on in the main
text.

In order to evaluate the derivatives, we assume the time propagator on the
form $A(t)=A_{0}(t)+\sum_{n}u_{n}A_{n}(t)$, where $u_{n}$ is assumed a
constant with time. This could either be a system-parameter $u_{n}=\theta_{n}$
or a function of the system parameters. In the latter case we may evaluate the
derivative via the chain rule
$\frac{\partial}{\partial\theta_{j}}=\frac{\partial
u_{n}}{\partial\theta_{n}}\frac{\partial}{\partial u_{n}}$. Here both the
first De Fouquieres  _et al._ (2011); Machnes  _et al._ (2011) and second
order derivatives Dalgaard  _et al._ (2020) has previously been calculated in
literature but the results are reproduced here for completeness. Note this
approach also allow us model uncertainties in the initial state of the system
$\chi_{0}=\chi(t_{0})$, which could e.g. stem from initialization errors. Note
the following works for Hermitian dynamics such as the Schrödinger equation
$A=-iH$. For non-Hermitian dynamics the second order derivative is calculable
by other means Goodwin and Kuprov (2016). In the following we assume, for
simplicity, that $A=-iH$.

The derivative of the time evolution operator may be evaluated in the
eigenbasis of $H_{n}=H(t_{n}+\Delta t/2)$ De Fouquieres  _et al._ (2011);
Machnes  _et al._ (2011)

$\displaystyle\Braket{h}{\frac{\partial
U_{n}}{\partial\theta_{j}}}{k}=\Braket{h}{\frac{\partial
H_{n}}{\partial\theta_{j}}}{k}I(h,k).$ (13)

Here $\ket{h}$ and $\ket{k}$ denote eigenstates of $H_{n}$. We have further
defined

$\displaystyle I(h,k)=\begin{cases}-i\Delta te^{-iE_{h}\Delta t},&\text{ if
}E_{h}=E_{k}\\\ \frac{e^{-iE_{h}\Delta t}-e^{-iE_{k}\Delta
t}}{E_{h}-E_{k}},&\text{ if }E_{h}\neq E_{k}.\end{cases}$ (14)

Here $E_{h}$ denotes the eigenenergy of $\ket{h}$. In Ref. Dalgaard  _et al._
(2020) the second order derivative was also calculated

$\displaystyle\Braket{h}{\frac{\partial
U_{n}}{\partial\theta_{i}\partial\theta_{j}}}{k}=$
$\displaystyle\sum_{l}\Big{(}\frac{\partial
H_{n}^{(h,l)}}{\partial\theta_{i}}\frac{\partial
H_{n}^{(l,k)}}{\partial\theta_{j}}+\frac{\partial
H_{n}^{(h,l)}}{\partial\theta_{j}}\frac{\partial
H_{n}^{(l,k)}}{\partial\theta_{i}}\Big{)}\mathcal{I}(k,l,h),$ (15)

where we use the notation $\frac{\partial
H_{n}^{(h,l)}}{\partial\theta_{i}}=\braket{h}{\frac{\partial
H_{n}}{\partial\theta_{i}}}{l}$. Here $\mathcal{I}(k,l,h)$ can be expressed
through the already calculated elements $I(h,k)$, see Eq. (B9) in Ref.
Dalgaard  _et al._ (2020).

The method outlined here requires propagating a single state, if we assume $M$
system-parameters, we also need $M$ first order derivatives, and $M^{2}$
second order derivatives. However, since the order of differentiation does not
matter, this reduces to $\frac{1}{2}M(M-1)$ second order derivatives. In most
applications, this is significantly lower than Monte Carlo sampling, which
typically requires several hundreds if not thousands of samples.

To concretize these ideas, we demonstrate how to calculate the derivatives of
an expectation value of a Hermetian operator $\hat{A}$ with respect to a
single state $Z(t)=\braket{\psi(t)}{\hat{A}}{\psi(t)}$ as also treated in the
main text. This has also previously been calculated in e.g. Ref. Khaneja  _et
al._ (2005).

$\displaystyle\frac{\partial}{\partial\theta_{j}}Z(t)$
$\displaystyle=\Braket{\frac{\partial}{\partial\theta_{j}}\psi(t)}{\hat{A}}{\psi(t)}+\Braket{\psi(t)}{\hat{A}}{\frac{\partial}{\partial\theta_{j}}\psi(t)}+\Braket{\psi(t)}{\frac{\partial\hat{A}}{\partial\theta_{j}}}{\psi(t)}$
$\displaystyle=2\mathbb{R}\Bigg{[}\Braket{\psi(t)}{\hat{A}}{\frac{\partial}{\partial\theta_{j}}\psi(t)}\Bigg{]}+\Braket{\psi(t)}{\frac{\partial\hat{A}}{\partial\theta_{j}}}{\psi(t)}.$
(16)

Where we in the above equation, have taken into account that $\hat{A}$ itself
may also be dependent on the parameters, for instance, the case where
$\hat{A}=H$ is the expected energy of the system. The second derivative reads

$\displaystyle\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}Z(t)$
$\displaystyle=2\mathbb{R}\Bigg{[}\Braket{\frac{\partial}{\partial\theta_{i}}\psi(t)}{\hat{A}}{\frac{\partial}{\partial\theta_{j}}\psi(t)}+\Braket{\psi(t)}{\hat{A}}{\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}\psi(t)}+\Braket{\psi(t)}{\frac{\partial\hat{A}}{\partial\theta_{j}}}{\frac{\partial}{\partial\theta_{i}}\psi(t)}\Bigg{]}$
$\displaystyle+\Braket{\psi(t)}{\frac{\partial^{2}\hat{A}}{\partial\theta_{i}\partial\theta_{j}}}{\psi(t)}.$
(17)

As an alternative to calculating the derivatives exactly, we also demonstrate
how they may be estimated using finite differences Fornberg (1988). The first
derivative using a central difference scheme is given by

$\displaystyle\frac{\partial
Z(\theta_{j})}{\partial\theta_{j}}=\frac{Z(\theta_{j}+\epsilon)-Z(\theta_{j}-\epsilon)}{2\epsilon}+\mathcal{O}(\epsilon^{2}),$
(18)

where we have used the shorhand notation
$Z(\boldsymbol{\theta},t)=Z(\theta_{1},\theta_{2},\ldots\theta_{j},\ldots\theta_{M},t)=Z(\theta_{j})$.
Note, the error of the approximation in Eq. (18) can be made negligibly small
by a sufficiently small selection of $\epsilon$, but not too small to induce
round-off errors. In a similar manner, the second order derivative reads

$\displaystyle\frac{\partial^{2}Z(\theta_{j})}{\partial\theta_{j}^{2}}=\frac{Z(\theta_{j}+\epsilon)-2Z(\theta_{j})+Z(\theta_{j}-\epsilon)}{\epsilon^{2}}+\mathcal{O}(\epsilon^{2}).$
(19)

Note that Eq. (18) and (19) requires $1+2M$ function evaluations, where $M$
again denotes the number of system-parameters. That is, for applications where
we do not model covariances (i.e. assume independent parameters), and only
need the mean value, our method scales linearly with the number of function
evaluations.

If we also needed the mixed derivatives, a finite difference scheme is given
by

$\displaystyle\frac{\partial^{2}Z(\theta_{j},\theta_{i})}{\partial\theta_{j}\partial\theta_{i}}=\frac{1}{\epsilon^{2}}\Big{[}Z(\theta_{j}+\epsilon,\theta_{i}+\epsilon)-Z(\theta_{j}+\epsilon,\theta_{i})-Z(\theta_{j},\theta_{i}+\epsilon)$
$\displaystyle+2Z(\theta_{j},\theta_{i})-Z(\theta_{j}-\epsilon,\theta_{i})-Z(\theta_{j},\theta_{i}-\epsilon)+Z(\theta_{j}-\epsilon,\theta_{i}-\epsilon)\Big{]}+\mathcal{O}(\epsilon^{2}).$
(20)

Note, this only requires two additional function evaluations for each of the
$\frac{1}{2}M(M-1)$ possible combinations. This means that the overall number
of evaluations becomes $1+M+M^{2}$. Note that although finite difference
schemes are often easier to implement, they are not as numerically stable as
calculating the exact derivatives, which does not depend on a choice of
$\epsilon$. Note in the main text, Section IV, we compare calculating the mean
with independent parameters, and so we use a scaling of $1+2M$.

### Experimental details

The details of the experiment are as follows: First, we initialize the qubit
in the $\ket{0}$ state. We then drive the qubit on resonance with a series of
Gaussian pulses of constant width (the pulse is truncated at $\pm 6\sigma$,
where $\sigma=75$ ns) and varying amplitude, which effectively realizes the
two level Hamiltonian in the main text with $\Delta=0$ and varying
$\Omega_{\mathrm{max}}$. The qubit is then measured and a discriminator is
applied such that we obtain a zero or one depending on if the qubit was in the
ground or excited state, respectively. This experiment, and all other
experimental runs reported on in this manuscript, is repeated $1024$ times to
obtain statistics, and the resulting curve is fit to a sinusoid and scaled
between $\braket{z}=\pm 1$, which allows us to maintain consistency with
theory in the face of experimental systematics that result in finite contrast.
From this, we can determine the pulse amplitudes $A_{\Theta}$ required to
obtain a given $X$ rotation $\Theta$. This allows us to calibrate the system.

The Gaussian pulses found above can also be used to form composite BB1 pulse
sequences Wimperis (1994), which take the form

$R_{\mathrm{BB1}}(\Theta)=R_{\pi,\phi(\Theta)}R_{2\pi,3\phi(\Theta)}R_{\pi,\phi(\Theta)}R_{\theta,0}$
(21)

where $R_{\theta,\phi}$ defines a Gaussian pulse with a phase $\phi$ that
results in a rotation $\theta$. The optimal phase $\phi$ for a desired final
rotation $\Theta$ is given by $\phi(\Theta)=\cos^{-1}{(-\Theta/4\pi)}$
Wimperis (1994).

## References

  * Grover (1996) L. K. Grover, in  _Proceedings of the twenty-eighth annual ACM symposium on theory of computing_ (1996) pp. 212–219.
  * Shor (1999) P. W. Shor, SIAM Review 41, 303 (1999).
  * Barreiro  _et al._ (2011) J. T. Barreiro, M. Müller, P. Schindler, D. Nigg, T. Monz, M. Chwalla, M. Hennrich, C. F. Roos, P. Zoller, and R. Blatt, Nature 470, 486 (2011).
  * Bloch  _et al._ (2012) I. Bloch, J. Dalibard, and S. Nascimbene, Nature Physics 8, 267 (2012).
  * Das and Chakrabarti (2008) A. Das and B. K. Chakrabarti, Reviews of Modern Physics 80, 1061 (2008).
  * McGeoch (2014) C. C. McGeoch, Synthesis Lectures on Quantum Computing 5, 1 (2014).
  * Schuld  _et al._ (2015) M. Schuld, I. Sinayskiy, and F. Petruccione, Contemporary Physics 56, 172 (2015).
  * Biamonte  _et al._ (2017) J. Biamonte, P. Wittek, N. Pancotti, P. Rebentrost, N. Wiebe, and S. Lloyd, Nature 549, 195 (2017).
  * Acín  _et al._ (2017) A. Acín, I. Bloch, H. Buhrman, T. Calarco, C. Eichler, J. Eisert, D. Esteve, N. Gisin, S. J. Glaser, F. Jelezko, _et al._ , arXiv preprint arXiv:1712.03773 (2017).
  * Taylor (1997) J. Taylor,  _Introduction to Error Analysis, the Study of Uncertainties in Physical Measurements_ (University Science Books, 1997).
  * Khaneja  _et al._ (2005) N. Khaneja, T. Reiss, C. Kehlet, T. Schulte-Herbrüggen, and S. J. Glaser, Journal of magnetic resonance 172, 296 (2005).
  * Motzoi  _et al._ (2011) F. Motzoi, J. M. Gambetta, S. Merkel, and F. Wilhelm, Phys. Rev. A 84, 022307 (2011).
  * De Fouquieres  _et al._ (2011) P. De Fouquieres, S. Schirmer, S. Glaser, and I. Kuprov, Journal of Magnetic Resonance 212, 412 (2011).
  * Dalgaard  _et al._ (2020) M. Dalgaard, F. Motzoi, J. H. M. Jensen, and J. Sherson, Physical Review A 102, 042612 (2020).
  * Langrock and DiVincenzo (2020) V. Langrock and D. P. DiVincenzo, arXiv preprint arXiv:2012.09517 (2020).
  * Sørensen  _et al._ (2020) J. J. Sørensen, J. S. Nyemann, F. Motzoi, J. Sherson, and T. Vosegaard, The Journal of Chemical Physics 152, 054104 (2020).
  * Ge  _et al._ (2020) X. Ge, H. Ding, H. Rabitz, and R.-B. Wu, Physical Review A 101, 052317 (2020).
  * Vieira and Rigolin (2020) R. Vieira and G. Rigolin, Physics Letters A 384, 126536 (2020).
  * Song  _et al._ (2021) W. Song, J. Ryu, K. Baek, and J. Bang, Journal of the Korean Physical Society , 1 (2021).
  * Estarellas  _et al._ (2020) M. P. Estarellas, T. P. Spiller, and I. D’Amico, in  _Journal of Physics: Conference Series_ , Vol. 1638 (IOP Publishing, 2020) p. 012013.
  * Coden  _et al._ (2021) D. A. Coden, S. S. Gómez, A. Ferrón, and O. Osenda, Physics Letters A 387, 127009 (2021).
  * Cabedo  _et al._ (2021) J. Cabedo, J. Claramunt, and A. Celi, arXiv preprint arXiv:2101.08253 (2021).
  * Kiely and Campbell (2021) A. Kiely and S. Campbell, arXiv preprint arXiv:2101.04995 (2021).
  * D’Angelis  _et al._ (2020) F. M. D’Angelis, F. A. Pinheiro, D. Guéry-Odelin, S. Longhi, and F. Impens, Physical Review Research 2, 033475 (2020).
  * Calderon-Vargas  _et al._ (2021) F. A. Calderon-Vargas, E. Barnes, and S. E. Economou, arXiv preprint arXiv:2101.11592 (2021).
  * Wu  _et al._ (2021) J.-L. Wu, Y. Wang, J.-X. Han, S.-L. Su, Y. Xia, Y. Jiang, and J. Song, Physical Review A 103, 012601 (2021).
  * Giorgi  _et al._ (2020) G. L. Giorgi, S. Lorenzo, and S. Longhi, in  _Photonics_ , Vol. 7 (Multidisciplinary Digital Publishing Institute, 2020) p. 18.
  * Hu  _et al._ (2020) S. Hu, Y. Ke, and C. Lee, Physical Review A 101, 052323 (2020).
  * Sheldon  _et al._ (2016) S. Sheldon, E. Magesan, J. M. Chow, and J. M. Gambetta, Physical Review A 93, 060302 (2016).
  * Theis  _et al._ (2018) L. Theis, F. Motzoi, S. Machnes, and F. Wilhelm, EPL (Europhysics Letters) 123, 60001 (2018).
  * Patterson  _et al._ (2019) A. Patterson, J. Rahamim, T. Tsunoda, P. Spring, S. Jebari, K. Ratter, M. Mergenthaler, G. Tancredi, B. Vlastakis, M. Esposito, _et al._ , Physical Review Applied 12, 064013 (2019).
  * Magesan and Gambetta (2020) E. Magesan and J. M. Gambetta, Physical Review A 101, 052308 (2020).
  * Sundaresan  _et al._ (2020) N. Sundaresan, I. Lauer, E. Pritchett, E. Magesan, P. Jurcevic, and J. M. Gambetta, arXiv preprint arXiv:2007.02925 (2020).
  * Wimperis (1994) S. Wimperis, Journal of Magnetic Resonance, Series A 109, 221 (1994).
  * Cummins  _et al._ (2003) H. K. Cummins, G. Llewellyn, and J. A. Jones, Physical Review A 67, 042308 (2003).
  * Vandersypen and Chuang (2005) L. M. Vandersypen and I. L. Chuang, Reviews of Modern Physics 76, 1037 (2005).
  * Möttönen  _et al._ (2006) M. Möttönen, R. de Sousa, J. Zhang, and K. B. Whaley, Physical Review A 73, 022332 (2006).
  * Borneman  _et al._ (2010) T. W. Borneman, M. D. Hürlimann, and D. G. Cory, Journal of Magnetic Resonance 207, 220 (2010).
  * Skinner  _et al._ (2011) T. E. Skinner, M. Braun, K. Woelk, N. I. Gershenzon, and S. J. Glaser, Journal of Magnetic Resonance 209, 282 (2011).
  * Motzoi  _et al._ (2016) F. Motzoi, E. Halperin, X. Wang, K. B. Whaley, and S. Schirmer, Physical Review A 94, 032313 (2016).
  * Platzer  _et al._ (2010) F. Platzer, F. Mintert, and A. Buchleitner, Physical review letters 105, 020501 (2010).
  * D’Ariano and Presti (2001) G. D’Ariano and P. L. Presti, Physical review letters 86, 4195 (2001).
  * Dankert  _et al._ (2009) C. Dankert, R. Cleve, J. Emerson, and E. Livine, Physical Review A 80, 012304 (2009).
  * Mabuchi (1996) H. Mabuchi, Quantum and Semiclassical Optics: Journal of the European Optical Society Part B 8, 1103 (1996).
  * Schirmer and Oi (2010) S. Schirmer and D. Oi, Laser Physics 20, 1203 (2010).
  * Burgarth and Yuasa (2012) D. Burgarth and K. Yuasa, Physical Review Letters 108, 080502 (2012).
  * Zhang and Sarovar (2014) J. Zhang and M. Sarovar, Physical review letters 113, 080401 (2014).
  * Guţă and Yamamoto (2015) M. Guţă and N. Yamamoto, IEEE Transactions on Automatic Control 61, 921 (2015).
  * Braunstein and Caves (1994) S. L. Braunstein and C. M. Caves, Physical Review Letters 72, 3439 (1994).
  * Giovannetti  _et al._ (2004) V. Giovannetti, S. Lloyd, and L. Maccone, Science 306, 1330 (2004).
  * Giovannetti  _et al._ (2011) V. Giovannetti, S. Lloyd, and L. Maccone, Nature photonics 5, 222 (2011).
  * Koch  _et al._ (2007) J. Koch, M. Y. Terri, J. Gambetta, A. A. Houck, D. Schuster, J. Majer, A. Blais, M. H. Devoret, S. M. Girvin, and R. J. Schoelkopf, Physical Review A 76, 042319 (2007).
  * Rigetti and Devoret (2010) C. Rigetti and M. Devoret, Physical Review B 81, 134507 (2010).
  * Goerz  _et al._ (2017) M. H. Goerz, F. Motzoi, K. B. Whaley, and C. P. Koch, npj Quantum Information 3, 1 (2017).
  * Brooks  _et al._ (2013) P. Brooks, A. Kitaev, and J. Preskill, Physical Review A 87, 052306 (2013).
  * Menke  _et al._ (2021) T. Menke, F. Häse, S. Gustavsson, A. J. Kerman, W. D. Oliver, and A. Aspuru-Guzik, npj Quantum Information 7, 1 (2021).
  * Nguyen  _et al._ (2019) L. B. Nguyen, Y.-H. Lin, A. Somoroff, R. Mencia, N. Grabon, and V. E. Manucharyan, Physical Review X 9, 041041 (2019).
  * Note (1) Note that the expression we derive for the mean works for any symmetric distribution.
  * Blanes  _et al._ (2009) S. Blanes, F. Casas, J.-A. Oteo, and J. Ros, Physics reports 470, 151 (2009).
  * Wilcox (1967) R. M. Wilcox, Journal of Mathematical Physics 8, 962 (1967).
  * Machnes  _et al._ (2011) S. Machnes, U. Sander, S. J. Glaser, P. De Fouquières, A. Gruslys, S. Schirmer, and T. Schulte-Herbrüggen, Physical Review A 84, 022305 (2011).
  * McKay  _et al._ (2018) D. C. McKay _et al._ , “Qiskit backend specifications for OpenQASM and OpenPulse experiments,” (2018), arXiv:1809.03452v1 .
  * Alexander  _et al._ (2020a) T. Alexander, N. Kanazawa, D. J. Egger, L. Capelluto, C. J. Wood, A. Javadi-Abhari, and D. C. McKay, Quantum Science and Technology 5, 044006 (2020a).
  * Alexander  _et al._ (2020b) T. Alexander, N. Kanazawa, D. J. Egger, L. Capelluto, C. J. Wood, A. Javadi-Abhari, and D. C McKay, Quantum Science and Technology 5, 044006 (2020b).
  * Oomura  _et al._ (2021) S. Oomura, T. Satoh, M. Sugawara, and N. Yamamoto, “Design and application of high-speed and high-precision CV gate on IBM Q OpenPulse system,” (2021), arXiv:2102.06117 [quant-ph] .
  * Gokhale  _et al._ (2020) P. Gokhale, A. Javadi-Abhari, N. Earnest, Y. Shi, and F. T. Chong, in  _2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_ (IEEE, Athens, Greece, 2020) pp. 186–200.
  * Note (2) For the qubit drive times considered here, the intrinsic qubit errors, e.g. due to relaxation, are negligible, so this system as designed is a suitable playground for artificially-inserted uncertainties that can be used to test our theory.
  * Wick (1950) G.-C. Wick, Physical Review 80, 268 (1950).
  * Isserlis (1918) L. Isserlis, Biometrika 12, 134 (1918).
  * Navarrete-Benlloch (2015) C. Navarrete-Benlloch, arXiv preprint arXiv:1504.05266 (2015).
  * Goodwin and Kuprov (2016) D. Goodwin and I. Kuprov, The Journal of chemical physics 144, 204107 (2016).
  * Fornberg (1988) B. Fornberg, Mathematics of computation 51, 699 (1988).


# On Monocular Depth Estimation and Uncertainty Quantification using
Classification Approaches for Regression

###### Abstract

Monocular depth is important in many tasks, such as 3D reconstruction and
autonomous driving. Deep learning based models achieve state-of-the-art
performance in this field. A set of novel approaches for estimating monocular
depth consists of transforming the regression task into a classification one.
However, there is a lack of detailed descriptions and comparisons for
Classification Approaches for Regression (CAR) in the community and no in-
depth exploration of their potential for uncertainty estimation. To this end,
this paper will introduce a taxonomy and summary of CAR approaches, a new
uncertainty estimation solution for CAR, and a set of experiments on depth
accuracy and uncertainty quantification for CAR-based models on KITTI dataset.
The experiments reflect the differences in the portability of various CAR
methods on two backbones. Meanwhile, the newly proposed method for uncertainty
estimation can outperform the ensembling method with only one forward
propagation.

Index Terms—  Depth estimation, Uncertainty Estimation

††We acknowledge the support of the Saclay-IA computing platform.

##  1 Introduction

In machine learning, regression tasks predict a continuous output based on a
given input. Yet, if the ground truth (prediction target) is within a specific
range, e.g., in the case of age estimation [1], one can quantize the ground
truth and cast regression into a classification problem. We refer to these
techniques as Classification Approaches for Regression (CAR), and in this
paper we explore CAR techniques applied on monocular depth estimation.

Monocular depth estimation (MDE), which is an ill-posed problem [2], consists
in predicting the scene depth given only an RGB image as the input. Deep
Neural Networks (DNNs) learn the mapping between the single RGB images and
their corresponding depth maps to solve MDE, and show good performance on
indoor and outdoor benchmarks [3, 4].

Classification Approaches for Regression (CAR) [5, 6, 7, 8, 9, 10] have
emerged recently in the spotlight among MDE algorithms. The core idea is to
transfer regression to a classification problem using quantization (or
discretization) strategies. The classification models can natively provide the
confidence for prediction results, which also has the potential to improve the
prediction accuracy [8]. DNNs are prone to two kinds of uncertainty: aleatoric
uncertainty and epistemic uncertainty [11]. It is crucial to study the
uncertainty of DNNs if we want to rely on their predictions. Some works
proposed to estimate the uncertainty of MDE DNNs by using an auxiliary network
[12, 13], or ensembling [14]. Here we gain access to the uncertainty directly
using the CAR DNN.

This work will investigate and show the complete picture of the CAR MDE
methods. The contributions are as follows:
1\. We systematically summarize and formalize the all major CAR MDE mechanisms
to the best of our knowledge;
2\. We implement these mechanisms on top of two different backbones, comparing
depth prediction and uncertainty quality on various evaluation metrics;
3\. We propose a new, effective uncertainty estimation method named
Expectation of Distance for CAR MDE models.

##  2 Overview of Classification Approaches for Regression (CAR)

###  2.1 Taxonomy of CAR MDE

| Discretization |
---|---|---
| Fully Handcrafted | Adaptive |
| Handcrafted | Handcrafted | Handcrafted |
CAR-MDEs | \+ One-hot | \+ Ordinal | \+ Smooth |  Post processing
|  KL divergence/ Weighted CE loss | - | - | SORN [8] | - | Argmax
| - | - | - | Cao et al. [5] ([15]) | - |  Soft weighted sum
| CE loss |  | Li et al. [6] ([16, 17])
---
- | - | -
| Multiple BCE loss | - | - | Yang et al. [9] | -
| Regression loss | DS-SIDENet [18] | - | - |  | Adabins [10]
---
Loss function |  | Ordinal
---
Regression loss
- |  | DORN [7]
---
([19, 20])
- | - | Ordinal sum

Table 1: Summary on CAR-MDE solutions. In parentheses are methods that use the
corresponding schemes as part of their solutions. The categories under
Discretization, Loss function and Post processing are detailed in Sec.2.

To better unify the terms and make it easier to grasp the differences in
contributions of CAR strategies, we propose to decompose the CAR problems into
three key components: discretization, loss function and post-processing. Table
1 offers an overview of the specific strategies used in the previous works.
The details are provided in the following sections.

The contributions of previous CAR based MDE solutions fall in two main groups:
1. novel strategies in the three components mentioned above [5, 6, 7, 8, 9,
10]; 2. architecture and/or loss modifications based on the previous
strategies [16, 19, 17, 20, 15, 18]. In most papers, CAR can improve model
accuracy, making it outperform its regression version [5, 6, 7, 8, 9, 10, 16],
or may improve model performance as part of multi-task learning [18, 17].

####  2.1.1 General notations

Let us first consider a monocular depth dataset
$D=\\{(\mathbf{x}_{i},\mathbf{d}_{i})\\}_{i}$, where
$\mathbf{x}_{i}\in{\mathbb{R}^{3\times H\times W}}$, and
$\mathbf{d}_{i}\in{(\mathbb{R}^{+})^{H\times W}}$ represents the ground truth
depth $\mathbf{d}_{i}$ for the image $\mathbf{x}_{i}$. We denote
$\\{d_{i,j}\\}_{j}^{N}$ all the pixel values in $\mathbf{d}_{i}$, where $N$ is
the number of pixels with valid ground truth. $a,b$ are two real values
representing the minimum and maximum depth value for the dataset.

For CAR strategies, we denote $K$ the number of classes, which represents the
level of discretization. Additionally, to simplify the notations, we use
$\log$ as the logarithm with base $e$ for all papers except for [16, 18],
where the $\log$ refers to the logarithm with base $10$. We denote
$f_{\theta_{1}}$ the DNN with parameters $\theta_{1}$. Given $\mathbf{x}_{i}$,
the prediction of $f_{\theta_{1}}$:

$\hat{\mathbf{y}}_{i}=\frac{e^{f_{\theta_{1}}(\mathbf{x}_{i})}}{\sum^{c-1}_{p=0}e^{[f_{\theta_{1}}(\mathbf{x}_{i})]_{p}}}$
(1)

where $f_{\theta_{1}}(\mathbf{x}_{i})\in\mathbb{R}^{c\times H\times W}$ is the
logit map and $[f_{\theta_{1}}(\mathbf{x}_{i})]_{p}$ its $p$-th coefficient,
and $\hat{\mathbf{y}}_{i}$ is the Softmax output. The number of its channels
$c=K$ by default, otherwise equals to the specific settings as in DORN [7]
($2K$) and Adabins [10] (128).

####  2.1.2 Discretization: Fully Handcrafted

The discretization function will output two components given $\mathbf{d}$: a
depth table $\bar{\mathbf{d}}=\\{\bar{d_{p}}\\}^{K}_{p}\in\mathbb{R}^{K}$
where the possible discrete depth values are set ordinally, and an indicator
map equivalent to a classification map
$\mathbf{y}_{i}=\\{{\\{{y_{i,j,p}}\\}_{p}^{K}}\\}_{j}^{N}\in{(\mathbb{R}^{+})}^{K\times
H\times W}$ which points for each pixel the closest discrete depth value. This
closest depth value can be considered as a class, leading to a classification
task. Both $\bar{\mathbf{d}}$ and $\mathbf{y}_{i}$ are handcrafted, and the
goal becomes the learning of $\mathbf{y}_{i}$.

Handcrafted $\bar{\mathbf{d}}$: $\bar{\mathbf{d}}$ contains $K$ values
representing the centers of intervals
$[\bar{\bar{d}}_{0},\bar{\bar{d}}_{1}[,..,[\bar{\bar{d}}_{K-1},\bar{\bar{d}}_{K}[$
with an interval width $q$:

$\displaystyle\bar{\mathbf{d}}=\\{\bar{d_{p}}\\}^{K}_{p}=\\{(\bar{\bar{d}}_{0}+\bar{\bar{d}}_{1})/2,...,(\bar{\bar{d}}_{K-1}+\bar{\bar{d}}_{K})/2\\}$
(2) $\displaystyle\bar{\bar{d}}_{k}=\log a+k\cdot q,k\in[0,K],q=(\log b-\log
a)/K$

Handcrafted $\bar{\mathbf{d}}$ \+ One-hot $\mathbf{y}_{i}$: Given
$\bar{\mathbf{d}}$, constructing $\mathbf{y}_{i}$ is done using one-hot
encoding, as applied in [6, 16, 17, 18]:

$\displaystyle\mathbf{y}^{[\text{onehot}]}_{i,j}=\begin{bmatrix}y_{i,j,0}&\ldots&y_{i,j,k}&\ldots&y_{i,j,K-1}\end{bmatrix}\in\mathbb{R}^{K}$
(3) $\displaystyle\mbox{with }y_{i,j,k}=1,\text{ if
}k=\lfloor(\log(d_{i,j}/a))/q\rceil\text{, and }0\text{ otherwise.}$

in which, $\lfloor.\rceil$ is a rounding operator, $q$ is defined in Eq. 2.

Handcrafted $\bar{\mathbf{d}}_{i}$ \+ Ordinal $\mathbf{y}_{i}$: Furthermore,
there are several variants of Eq 3. Ordinal properties can be applied on it as
presented in [7] and the followed works [19, 20]:

$\displaystyle\mathbf{y}^{[\text{ordi}]}_{i,j}=\begin{bmatrix}y_{i,j,0}&\ldots&y_{i,j,k}&\ldots&y_{i,j,K-1}\end{bmatrix}\in\mathbb{R}^{K}$
(4) $\displaystyle\mbox{with }y_{i,j,k}=1,\text{ if
}k\leq\lfloor(\log(d_{i,j}/a))/q\rceil\text{, and }0\text{ otherwise.}$

Handcrafted $\bar{\mathbf{d}}_{i}$ \+ Smooth $\mathbf{y}_{i}$: It is also
possible to predict a smooth discrete map from the initial discrete map
$\mathbf{y}^{[\text{onehot}]}_{i,j}$. The indicator in the classification map
is softened by applying on $\bar{\mathbf{d}}$ a Gaussian kernel, as to predict
distance within a coarser range. The smooth $\mathbf{y}_{i}$ are defined by:

$\displaystyle\mathbf{y}{{}^{[\text{smo1}]}_{i,j}}=e^{-\gamma||\log(d_{i,j})-\bar{\mathbf{d}}||^{2}}$
(5)
$\displaystyle\mathbf{y}{{}^{[\text{smo2}]}_{i,j}}=\frac{e^{-\gamma||\log(d_{i,j})-\bar{\mathbf{d}}||^{2}}}{\sum_{p=0}^{K-1}{e^{-\gamma||\log(d_{i,j})-{\bar{d}}_{p}||^{2}}}}$
(6)

where $\gamma$ is a hyperparameter which can be regarded as the scale of the
discrete distribution (the smaller $\gamma$, the flatter the label
distribution in $\mathbf{y}{{}_{i,j}}$). Specifically, Yang et al. [9] use Eq
5 as the unnormalized soft target labels, while SORN [8] applies the
normalized version in Eq 6. Moreover, Cao et al. [5] introduce a $K\times K$
symmetric “information gain” matrix $H$ in their loss function with elements
$H(k,\mathbf{p})=e^{-\gamma||k-\mathbf{p}||^{2}}$, where
$\mathbf{p}=[0,...,K-1]$ and $k$ is the discrete ground truth index as defined
in Eq 3. In this case:

$\mathbf{y}^{\text{[smo3]}}_{i,j}=e^{-\gamma||k-\mathbf{p}||^{2}}=e^{-\gamma\cdot
q^{-2}||\log(d_{i,j})-(\bar{\mathbf{d}}-0.5q)||^{2}}$ (7)

Since $q$ is a constant, this strategy can be regarded as being equivalent to
the $\mathbf{y}^{\text{[smo1]}}_{i,j}$ in Eq 5.

####  2.1.3 Discretization: Adaptive

In the absence of the handcrafted depth table or the classification map, one
may also implicitly train both of them using a regression loss as in Adabins
[10]. Thus the goal of the DNN is changed from fitting the handcrafted
classification maps to fitting the continuous ground truth depth, while still
following the principle of building depth tables and classification maps.

In Adabins [10], the depth table is implicitly trained along with the
classification map using a non-linear block $g_{\theta_{2}}$ with $\theta_{2}$
the parameters of $g$, which is a mini ViT [10]. In this case,
$g_{\theta_{2}}$ is set on top of the backbone $f_{\theta_{1}}$, and it will
output $\hat{\bar{\mathbf{d}}}_{i}^{[\text{ada}]}$ and
$\hat{\mathbf{y}}_{i}^{[\text{ada}]}$ given $f_{\theta_{1}}(\mathbf{x}_{i})$:

$\displaystyle\hat{\bar{\mathbf{d}}}^{[\text{ada}]}_{i}=\\{a+(b-a)(\sum_{s=0}^{p}{\hat{\bar{d}}^{\text{[ada]}}_{i,s}})\\}_{p=0}^{K-1},\hat{\mathbf{y}}_{i,j}^{[\text{ada}]}=\frac{e^{\mathbf{l}_{i,j}}}{\sum^{K-1}_{p=0}e^{l_{i,j,p}}}$
(8) $\displaystyle\text{with
}\\{\hat{\bar{d}}^{\text{[ada]}}_{i}\\}_{p}^{K},\mathbf{l}_{i}=g_{\theta_{2}}(f_{\theta_{1}}(\mathbf{x}_{i}))$

where, $\hat{\mathbf{y}}_{i,j}^{[\text{ada}]}$ is the product of a Softmax
function, and $\hat{\bar{\mathbf{d}}}_{i}^{[\text{ada}]}$ is a cumulative
summation output followed by a normalization operation which is included in
$g_{\theta_{2}}$. Since $\hat{\bar{\mathbf{d}}}_{i}^{[\text{ada}]}$ is a
product of $g_{\theta_{2}}$ taking $f_{\theta_{1}}(\mathbf{x}_{i})$, for each
$\mathbf{x}_{i}$, not only a unique classification map but also a unique depth
table will be provided.

####  2.1.4 Loss function

Based on the previous discretization strategies, we introduce here the loss
function design. Models should fit their output to the designed $\mathbf{y}$
or $\mathbf{d}$. For brevity, we define first the total loss
$L_{\text{total}}=\sum_{i}\sum_{j=0}^{N-1}L_{{i,j}}$ where $L_{i,j}$ is the
loss for pixel $j$, on the $i$-th data. We just define $L_{i,j}$ in the
following sections for simplicity.

Cross entropy (CE) loss: is a straightforward solution given an one-hot
classification map:
${L_{i,j}^{{[\text{CE}]}}}(\theta_{1})=-({{\mathbf{y}^{[\text{onehot}]}_{i,j}}\log\hat{\mathbf{y}}_{i,j}})$.

Ordinal regression loss: is essentially an implicit ordinal selection plus a
multiple binary cross entropy (BCE) loss. Instead of directly using
$\hat{\mathbf{y}}_{i,j}$, it requires to do an ordinal selection on the logit
map $f_{\theta_{1}}(\mathbf{x}_{i})$ with $c=2K$ to $c=K$ as the predicted
classification map, then to apply a Multiple-BCE loss on it:

$\displaystyle{L_{i,j}^{{[\text{ordi}]}}}(\theta_{1})\text{=}-[{{\mathbf{y}^{[\text{ordi}]}_{i,j}}\log\hat{\mathbf{y}}_{i,j}^{{[\text{ordi}]}}+(1-{\mathbf{y}^{[\text{ordi}]}_{i,j}})\log(1-\hat{\mathbf{y}}_{i,j}^{{[\text{ordi}]}})}]$
(9) $\displaystyle\text{with
}\hat{\mathbf{y}}_{i,j}^{{[\text{ordi}]}}=\frac{e^{[f_{\theta_{1}}(\mathbf{x}_{i})]_{2p+1}}}{e^{[f_{\theta_{1}}(\mathbf{x}_{i})]_{2p+1}}+e^{[f_{\theta_{1}}(\mathbf{x}_{i})]_{2p}}}$

where $2p+1$ and $2p$ represent the indices of the coefficient.

Weighted CE loss: is applied when the target vector is a soft discrete
distribution. The CE loss turns to be equal to:
${L_{i,j}^{{[\text{WCE}]}}}(\theta_{1})=-({\mathbf{y}{{}^{[\text{smo2}]}_{i,j}}\log\hat{\mathbf{y}}_{i,j}})$,
and it has the same form as the Kullback-Leibler divergence loss.

Multiple BCE loss: is another solution when the target is a soft discrete
distribution. Yang et al. [9] apply BCE loss on every class value in
$\mathbf{y}^{[\text{smo1}]}_{i,j}$ defined in Eq. 5.

Regression losses: DS-SIDENet [18] applies CAR with a smooth L1 loss [21] to
fit the one-hot classification map target. Conversely, Adabins [10] combines
the per-image adaptive depth table $\hat{\bar{\mathbf{d}}}^{[\text{ada}]}_{i}$
and $\hat{\mathbf{y}}_{i,j}^{[\text{ada}]}$ defined in Eq. 8 as the predicted
depth, then applies a Scale-Invariant loss [22].

####  2.1.5 Post-processing

Post-processing aims to restore the discrete predicted labels to continuous
depth values. In the following equations, we use the power function in base
$e$, see Sec. 2.1.1.

Ordinal sum: For DORN [7], the continuous depth is restored from the sum of
the output Sigmoid labels which are higher than or equal to 0.5:

$\hat{d}_{i,j}=\text{exp}\\{{\log(a)+q\cdot[\sum_{p=0}^{K-1}\mathbbm{1}\\{\hat{y}^{\text{[ordi]}}_{i,j,p}\geq
0.5\\}+0.5]}\\}$ (10)

where $q$ and $\hat{y}^{\text{[ordi]}}_{i,j,p}$ are defined in Eq. 2 and Eq. 9
respectively.

Soft weighted sum: is a solution that may applied on both handcrafted or
learned depth tables. It sums the Hadamard product between the depth table and
the classification map:

$\displaystyle\hat{d}_{i,j}=\text{exp}\\{\sum_{p=0}^{K-1}{\bar{d}_{p}\cdot\hat{y}_{i,j,p}}\\}$
(11)

Note that essentially AdaBins [10] also follows this pattern.

Argmax: The authors of SORN [8] claim that Argmax outperforms Soft weighted
sum in their case:

$\hat{d}_{i,j}=\text{exp}\\{\log(a)+q\cdot[\underset{p}{\mathrm{argmax}}\,(\\{\hat{y}_{i,j,p}\\}^{K}_{p})+0.5]\\}$
(12)

where $q$ is defined in Eq. 2.

###  2.2 Uncertainty estimation of CAR MDE

In this section, we will discuss the previous works on uncertainty estimation
for CAR MDE, the difficulty of this problem and our proposed approaches on
estimating CAR uncertainty.

The ground truth uncertainty or the Oracle should be the model’s prediction
error. The previous works on MDE uncertainty estimation [13, 12] mainly use
the principle of learning the prediction error [11]. Meanwhile, the Variance
among the point estimations given by MC-Dropout [23] and Deep Ensembles [14]
can also be applied for this task. Unlike the previous works, the likelihoods
of the predicted class (the quantified depth value) given the input data
provided by CARs can offer another possibility to estimate the uncertainty
mentioned in the previous works but rarely discussed. Yang et al. [9] suggest
to use Shannon Entropy (S-Entr) [24] among the output Softmax classification
map: $-\sum_{p=0}^{K-1}\hat{{y}}_{i,j,p}\log{\hat{{y}}_{i,j,p}}$ Moreover,
they showed cases where the depth is well predicted, yet the entropy is high,
leading to an under-confident uncertainty score. Other strategies such as
1-Maximum Class Probability (1-MCP) can also be regarded as the uncertainty:
$1-\max_{p}(\\{\hat{{y}}_{i,j,p}\\}_{p})$ These are typical solutions used in
classification tasks, and we argue that they will be suitable in case of using
Argmax in post-processing for CAR problems. Widely used soft weighted sum (see
Table 1) makes the property of CAR special: not only the classification map
but also the depth table should be taken into account in the final result as
shown in Eq. 11.

Following these remarks, we propose a new solution for CAR uncertainty. We
first note that the previously mentioned methods lack consideration of the
depth table, and further its relationship to the classification map. Hence, we
define as CAR uncertainty metric: the Expectation of Distance (E-Dist) between
the quantified depth values (either handcrafted logarithm depth table
$\bar{\mathbf{d}}$ (Eq. 2) or the learned one
$\hat{\bar{\mathbf{d}}}^{\text{[ada]}}_{i}$ (Eq. 8)) and the final predicted
depth $\hat{\mathbf{d}}_{i}$ (Eq. 11, 12) :

$\text{E-Dist}=\sum_{p=0}^{K-1}{\hat{{y}}_{i,j,p}\cdot(e^{\bar{d}_{p}}-\hat{d}_{i,j})^{2}}\hskip
4.0pt\text{or}\hskip
4.0pt\text{E-Dist}=\sum_{p=0}^{K-1}{\hat{{y}}_{i,j,p}\cdot(\hat{\bar{d}}^{\text{[ada]}}_{i,p}-\hat{d}_{i,j})^{2}}$
(13)

Additionally, to our knowledge, no previous works discuss the uncertainty of
ordinal regression model [7]. According to its CAR strategy, only values
greater than or equal to 0.5 in its classification map will be considered in
the final depth calculation, thus we argue that the uncertainty comes from
this part. The modified E-Dist for ordinal regression is as below: We propose
to discretize the depth prediction $\hat{\mathbf{d}}_{i}$ (Eq. 10) using Eq.
4, that we denote as ${\mathbf{y}^{\prime[\text{ordi}]}_{i,j}}$. Then we
calculate the distance between ${\mathbf{y}^{\prime[\text{ordi}]}_{i,j}}$ and
${\hat{\mathbf{y}}^{[\text{ordi}]}_{i,j}}$ (defined in Eq.9) weighted by the
depth table and only consider the part with
$\hat{\mathbf{y}}^{\text{[ordi]}}_{i,j}\geq 0.5$:

$\sum_{p=0}^{K-1}{e^{\bar{d}_{p}}\cdot({y^{\prime[\text{ordi}]}_{i,j,p}}-{\hat{y}^{[\text{ordi}]}_{i,j,p}})^{2}}\cdot\mathbbm{1}\\{\hat{y}^{\text{[ordi]}}_{i,j,p}\geq
0.5\\}$ (14)

##  3 Experiments

Backbones | BTS | FCN |  K
---|---|---|---
Metrics |  $\delta$1$\uparrow$ |  $\delta$2$\uparrow$ |  $\delta$3$\uparrow$ |  | Abs
---
Rel$\downarrow$
| Sq
---
Rel$\downarrow$
RMSE$\downarrow$ |  | RMSE
---
log$\downarrow$
log10$\downarrow$ |  $\delta$1$\uparrow$ |  $\delta$2$\uparrow$ |  $\delta$3$\uparrow$ |  | Abs
---
Rel$\downarrow$
| Sq
---
Rel$\downarrow$
RMSE$\downarrow$ |  | RMSE
---
log$\downarrow$
log10$\downarrow$
DORN [7] | 0.952 | 0.992 | 0.998 | 0.069 | 0.267 | 2.802 | 0.103 | 0.029 | 0.940 | 0.990 | 0.998 | 0.076 | 0.292 | 2.962 | 0.113 | 0.033 | 80
Cao et al. [5] | 0.945 | 0.992 | 0.998 | 0.077 | 0.292 | 2.988 | 0.111 | 0.034 | 0.928 | 0.989 | 0.998 | 0.084 | 0.344 | 3.223 | 0.122 | 0.036 | 50
Li et al. [6] | 0.950 | 0.990 | 0.998 | 0.070 | 0.287 | 2.928 | 0.106 | 0.030 | 0.940 | 0.988 | 0.997 | 0.075 | 0.314 | 3.190 | 0.116 | 0.033 | 150 [16]
SORN [8] | 0.947 | 0.992 | 0.998 | 0.071 | 0.290 | 2.929 | 0.107 | 0.031 | 0.863 | 0.976 | 0.995 | 0.119 | 0.563 | 3.938 | 0.163 | 0.051 | 120
Yang et al. [9] | 0.951 | 0.991 | 0.998 | 0.065 | 0.276 | 2.897 | 0.103 | 0.029 | 0.940 | 0.989 | 0.997 | 0.072 | 0.302 | 3.096 | 0.113 | 0.032 | 128
DS-SIDE [18] | 0.950 | 0.991 | 0.998 | 0.071 | 0.275 | 2.886 | 0.106 | 0.032 | 0.931 | 0.990 | 0.998 | 0.079 | 0.331 | 3.353 | 0.119 | 0.035 | 80
Adabins [10] | 0.935 | 0.990 | 0.998 | 0.078 | 0.347 | 3.143 | 0.114 | 0.033 | 0.937 | 0.991 | 0.998 | 0.079 | 0.331 | 3.027 | 0.113 | 0.033 | 256
DORN [7] | 0.952 | 0.992 | 0.998 | 0.069 | 0.267 | 2.802 | 0.103 | 0.029 | 0.940 | 0.990 | 0.998 | 0.076 | 0.292 | 2.962 | 0.113 | 0.033 | 80
Cao et al. [5] | 0.953 | 0.991 | 0.998 | 0.066 | 0.268 | 2.857 | 0.103 | 0.029 | 0.934 | 0.989 | 0.997 | 0.076 | 0.319 | 3.124 | 0.117 | 0.033 | 80
Li et al. [6] | 0.949 | 0.990 | 0.997 | 0.087 | 0.305 | 2.982 | 0.116 | 0.037 | 0.933 | 0.988 | 0.997 | 0.096 | 0.350 | 3.157 | 0.126 | 0.040 | 80
SORN [8] | 0.949 | 0.993 | 0.998 | 0.072 | 0.283 | 2.902 | 0.106 | 0.031 | 0.863 | 0.976 | 0.995 | 0.122 | 0.573 | 3.950 | 0.165 | 0.052 | 80
Yang et al. [9] | 0.948 | 0.991 | 0.998 | 0.070 | 0.284 | 2.973 | 0.107 | 0.031 | 0.940 | 0.990 | 0.997 | 0.076 | 0.308 | 3.067 | 0.115 | 0.034 | 80
DS-SIDE [18] | 0.950 | 0.991 | 0.998 | 0.071 | 0.275 | 2.886 | 0.106 | 0.032 | 0.931 | 0.990 | 0.998 | 0.079 | 0.331 | 3.353 | 0.119 | 0.035 | 80
Adabins [10] | 0.933 | 0.989 | 0.998 | 0.079 | 0.357 | 3.203 | 0.116 | 0.033 | 0.937 | 0.990 | 0.998 | 0.076 | 0.318 | 3.062 | 0.112 | 0.032 | 80
Org | 0.955 | 0.993 | 0.998 | 0.060 | 0.249 | 2.798 | 0.096 | 0.027 | 0.944 | 0.992 | 0.998 | 0.069 | 0.275 | 2.938 | 0.107 | 0.030 | 1
MC-Dropout [23] | 0.941 | 0.992 | 0.998 | 0.083 | 0.308 | 2.910 | 0.114 | 0.035 | 0.918 | 0.984 | 0.996 | 0.085 | 0.369 | 3.157 | 0.125 | 0.036 | 1
Deep Ensembles [14] | 0.957 | 0.993 | 0.999 | 0.059 | 0.233 | 2.688 | 0.093 | 0.026 | 0.946 | 0.992 | 0.998 | 0.068 | 0.269 | 2.923 | 0.106 | 0.030 | 1

(a)

Backbones | BTS | FCN |  K
---|---|---|---
Metrics | AUSE RMSE$\downarrow$ | AUSE AbsRel$\downarrow$ | AUSE RMSE$\downarrow$ | AUSE AbsRel$\downarrow$
Methods | 1-MCP | S-Entr | E-Dist | 1-MCP | S-Entr | E-Dist | 1-MCP | S-Entr | E-Dist | 1-MCP | S-Entr | E-Dist
Cao et al. [5] | 0.542 | 0.770 | 0.133 | 0.382 | 0.424 | 0.411 | 0.532 | 0.701 | 0.127 | 0.354 | 0.375 | 0.375 | 50
Li et al. [6] | 0.174 | 0.153 | 0.187 | 0.276 | 0.262 | 0.409 | 0.137 | 0.138 | 0.132 | 0.241 | 0.235 | 0.259 | 150
SORN [8] | 1.371 | 1.394 | 0.157 | 0.939 | 0.982 | 0.427 | 1.244 | 1.283 | 0.170 | 0.754 | 0.755 | 0.451 | 120
Yang et al. [9] | 0.141 | 0.145 | 0.094 | 0.232 | 0.219 | 0.256 | 0.142 | 0.161 | 0.111 | 0.225 | 0.226 | 0.247 | 128
DS-SIDE [18] | 0.698 | 0.806 | 0.293 | 0.525 | 0.544 | 0.397 | 1.212 | 1.331 | 0.995 | 0.630 | 0.722 | 0.484 | 80
Adabins [10] | 0.855 | 0.827 | 0.179 | 0.536 | 0.527 | 0.377 | 0.984 | 0.945 | 0.191 | 0.608 | 0.589 | 0.398 | 256
DORN [7] | 0.188 | 0.158 | 0.128 | 0.530 | 0.430 | 0.303 | 0.202 | 0.165 | 0.135 | 0.593 | 0.445 | 0.283 | 80
Cao et al. [5] | 0.371 | 0.476 | 0.119 | 0.323 | 0.329 | 0.356 | 0.349 | 0.393 | 0.117 | 0.284 | 0.265 | 0.308 | 80
Li et al. [6] | 0.206 | 0.178 | 0.181 | 0.355 | 0.348 | 0.449 | 0.170 | 0.163 | 0.124 | 0.318 | 0.310 | 0.333 | 80
SORN [8] | 1.367 | 1.390 | 0.157 | 0.900 | 0.941 | 0.444 | 1.228 | 1.275 | 0.175 | 0.725 | 0.737 | 0.473 | 80
Yang et al. [9] | 0.194 | 0.179 | 0.099 | 0.273 | 0.259 | 0.271 | 0.156 | 0.169 | 0.104 | 0.258 | 0.252 | 0.274 | 80
DS-SIDE [18] | 0.698 | 0.806 | 0.293 | 0.525 | 0.544 | 0.397 | 1.212 | 1.331 | 0.995 | 0.630 | 0.722 | 0.484 | 80
Adabins [10] | 0.823 | 0.683 | 0.181 | 0.499 | 0.450 | 0.360 | 0.775 | 0.710 | 0.234 | 0.502 | 0.478 | 0.391 | 80
DORN [7] | 0.188 | 0.158 | 0.128 | 0.530 | 0.430 | 0.303 | 0.202 | 0.165 | 0.135 | 0.593 | 0.445 | 0.283 | 80
MC-Dropout [23] | 0.460 (Variance) | 0.501 (Variance) | 0.322 (Variance) | 0.456 (Variance) | 1
Deep Ensembles [14] | 0.165 (Variance) | 0.261 (Variance) | 0.184 (Variance) | 0.290 (Variance) | 1

(b)

Table 2:  The best/second-best values are highlighted in dark/light blue. The
results from regression-based models are provided as reference (lower rows),
and we only highlight the CAR-based results. (a) Depth uncertainty
evaluations. MC-Dropout and Deep Ensembles will only provide the uncertainty
with one method. (b) Depth accuracy evaluations. Org: the original BTS [25]
model and the regression version applied on FCN [26] model.  Fig. 1:
Experiment pipeline. Three heads will be applied on the backbone: I. original
regression version; II: MDE with handcrafted discretization; III: MDE with
adaptive discretization through a mini ViT module [10].

In this section, we fill in the missing comparisons of the previous works.
Meanwhile, our experiments provide an extensive analysis of CAR MDE
uncertainty estimation. While it is not trivial to propose a model-agnostic
approach, the ensuing discussion establishes some important guidelines about
performing this task on CAR models.

###  3.1 Experiment settings

All the experiments are based on Eigen-split [22] KITTI dataset [3]. We
followed the original settings in the corresponding papers for CAR strategies
and applied them on a regression-based and a classification-based backbone
respectively. In addition, we added experiments with $K=80$ to the methods
with originally different choices for $K$ for better comparison. Fig 1
illustrates the experiment pipeline.
Reg.-based backbone: We use BTS-DenseNet161 [25, 27]. Same BTS training
settings are applied for all the methods.
Cla.-based backbone: We choose FCN-ResNet101 [26, 28]. FCN is originally
designed for semantic segmentation, thus it is suitable for CAR methods. For
the one-channel regression version (org), followed BTS, we apply a Sigmoid on
the top and multiply the output by $b$.
Evaluation matrices: We use the same matrices first introduced in [22] and
used in many subsequent works for depth performance. For uncertainty
estimation, we use the area under sparsification error curve (AUSE), as in [9,
12, 13]. 1% of pixels are removed each time and we calculate RMSE and AbsRel
for the rest. Uncertainty estimation methods we used are introduced in Sec.
2.2. We will compare the CAR MDE uncertainty with widely used MC-Dropout [23]
(with 8 forward passes) and Deep Ensembles [14] (with 3 models).
Training time consumption: We use one NVIDIA Titan RTX to count the average
time consumption on Forward+Backward passes for one image for all CAR methods
with $K=80$ as well as the original regression method and the Deep Ensembles
[14] using the same training settings.

###  3.2 Performance and discussions

Table 1(a) and Table 1(b) provide depth and uncertainty results.
Depth: We find that all CAR MDE methods are portable, but training directly
with the settings of the original backbones degrades performance. We discover
that the Adabins, DS-SIDE and SORN [10, 18, 8] based models are more sensitive
to the selected backbone than the other ones. Despite the influence of the
backbones, we also consider that the training settings for the original
Adabins are more different from the ones of BTS. This difference may cause
Adabins to produce worse performance after porting. DORN-based model [7]
achieves the best result among CAR DNNs, which confirms the effectiveness of
ordinal constraints.
Uncertainty: Our proposed E-Dist shows good and robust performance in most
cases given a CAR MDE method. Among the CAR strategies, we found that the
uncertainty quality is related to the sharpness of the labeling during
discretization, and also to the loss function. Li et al. [6], Yang et al. [9]
and Cao et al. [5] based DNNs perform better for the uncertainty. Li et al.
[6] model has one-hot encoded labels in the classification map which leads to
the sharpest label distribution. Yang et al. [9] model has $\gamma=15$ in Eq.
5 and we can also have $\gamma\cdot q^{-2}=65$ in Eq. 7 for Cao et al. [5]
model. This big coefficient can sharpen the label distribution. Conversely, in
SORN [8] the $\gamma$ in Eq. 6 is much smaller, which results in the evener
distributed labels, and we consider this is the main cause of its worse
performance. Yang et al. [9] based model outperforms the others, which
indicates that the Multi-BCE loss is more suitable for uncertainty estimation,
which is similar to the one-versus-all strategy [29].
Choices of $K$: According to two sets of results separated by $K$, the
performance rankings are consistent: the depth precision and the uncertainty
quality increase along $K$, and $K$ has biggest impact on Cao et al. [5] based
model.
Time efficiency: According to Table. 3, we argue that the CAR strategy will
slightly slow down the training, especially for the ones requiring label
smoothing in discretization [9, 5, 7]. However, Deep Ensembles [14] with only
three models still require the most training time.

Time consumption (ms)
---
| CAR
---
Solutions
DORN [7] |  | Cao
---
et al. [5]
Li et al. [6] | SORN [8] |  | Yang
---
et al. [9]
DS-SIDE [18] | Adabins [10] | Org |  | Deep
---
Ensembles [14]
BTS | 610.96 | 509.18 | 431.32 | 444.66 | 613.98 | 430.14 | 421.06 | 378.98 | 1136.94
FCN | 735.38 | 614.52 | 538.92 | 556.84 | 722.80 | 540.70 | 588.26 | 517.66 | 1552.98

Table 3: Time consumption on Forward+Backward passes for one image.

##  4 Conclusion

In this paper, we summarize the CAR MDE methods in detail along three key
components, and conduct experiments on their portability and performance,
including both depth and uncertainty quality. In the future, we will try to
apply CAR strategy and E-Dist uncertainty estimator on more tasks.

## References

  * [1] Gil Levi and Tal Hassner,  “Age and gender classification using convolutional neural networks,”  in CVPR workshops, 2015.
  * [2] Pawan Sinha and Edward Adelson,  “Recovering reflectance and illumination in a world of painted polyhedra,”  in ICCV, 1993.
  * [3] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger,  “Sparsity invariant cnns,”  in 3DV, 2017.
  * [4] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus,  “Indoor segmentation and support inference from rgbd images,”  in ECCV, 2012.
  * [5] Yuanzhouhan Cao, Zifeng Wu, and Chunhua Shen,  “Estimating depth from monocular images as classification using deep fully convolutional residual networks,”  TCSVT, 2017.
  * [6] Bo Li, Yuchao Dai, and Mingyi He,  “Monocular depth estimation with hierarchical fusion of dilated cnns and soft-weighted-sum inference,”  Pattern Recognition, 2018.
  * [7] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao,  “Deep ordinal regression network for monocular depth estimation,”  in CVPR, 2018.
  * [8] Raul Diaz and Amit Marathe,  “Soft labels for ordinal regression,”  in CVPR, 2019.
  * [9] Gengshan Yang, Peiyun Hu, and Deva Ramanan,  “Inferring distributions over depth from a single image,”  in IROS, 2019.
  * [10] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka,  “Adabins: Depth estimation using adaptive bins,”  in CVPR, 2021.
  * [11] Alex Kendall and Yarin Gal,  “What uncertainties do we need in bayesian deep learning for computer vision?,”  NIPS, 2017.
  * [12] Matteo Poggi, Filippo Aleotti, Fabio Tosi, and Stefano Mattoccia,  “On the uncertainty of self-supervised monocular depth estimation,”  in CVPR, 2020.
  * [13] Xuanlong Yu, Gianni Franchi, and Emanuel Aldea,  “Slurp: Side learning uncertainty for regression problems,”  in BMVC, 2021.
  * [14] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell,  “Simple and scalable predictive uncertainty estimation using deep ensembles,”  in NIPS, 2017.
  * [15] Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan,  “Enforcing geometric constraints of virtual normal for depth prediction,”  in ICCV, 2019.
  * [16] Ruibo Li, Ke Xian, Chunhua Shen, Zhiguo Cao, Hao Lu, and Lingxiao Hang,  “Deep attention-based classification network for robust depth prediction,”  in ACCV, 2018.
  * [17] Lukas Liebel and Marco Körner,  “Multidepth: Single-image depth estimation via multi-task regression and classification,”  in ITSC, 2019.
  * [18] Haoyu Ren, Mostafa El-Khamy, and Jungwon Lee,  “Deep robust single image depth estimation neural network using scene understanding.,”  in CVPR Workshops, 2019.
  * [19] Chen-Chou Lo and Patrick Vandewalle,  “Depth estimation from monocular images and sparse radar using deep ordinal regression network,”  in ICIP, 2021.
  * [20] Minh Hieu Phan, Son Lam Phung, and Abdesselam Bouzerdoum,  “Ordinal depth classification using region-based self-attention,”  in ICPR, 2021.
  * [21] Jia-Ren Chang and Yong-Sheng Chen,  “Pyramid stereo matching network,”  in CVPR, 2018.
  * [22] David Eigen, Christian Puhrsch, and Rob Fergus,  “Depth map prediction from a single image using a multi-scale deep network,”  in NIPS, 2014.
  * [23] Yarin Gal and Zoubin Ghahramani,  “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,”  in ICML, 2016.
  * [24] Claude Elwood Shannon,  “A mathematical theory of communication,”  ACM SIGMOBILE MC2R, 2001.
  * [25] Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Il Hong Suh,  “From big to small: Multi-scale local planar guidance for monocular depth estimation,”  arXiv preprint arXiv:1907.10326, 2019.
  * [26] Jonathan Long, Evan Shelhamer, and Trevor Darrell,  “Fully convolutional networks for semantic segmentation,”  in CVPR, 2015.
  * [27] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger,  “Densely connected convolutional networks,”  in CVPR, 2017.
  * [28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in CVPR, 2016.
  * [29] Gianni Franchi, Andrei Bursuc, Emanuel Aldea, Severine Dubuisson, and Isabelle Bloch,  “One versus all for deep neural network incertitude (ovnni) quantification,”  IEEE Access, 2021.

On Monocular Depth Estimation and Uncertainty Quantification using
Classification Approaches for Regression
—— Supplementary Material ——

(a) Input image

(b) Ground truth depth

(c) Predicted depth

(d) Variance uncertainty

(e) Predicted depth

(f) 1-MCP uncertainty

(g) S-Entr uncertainty

(h) E-Dist uncertainty

(i) Predicted depth

(j) 1-MCP uncertainty

(k) S-Entr uncertainty

(l) E-Dist uncertainty

(m) Predicted depth

(n) 1-MCP uncertainty

(o) S-Entr uncertainty

(p) E-Dist uncertainty

Fig. 2: Illustrations of predicted depth and uncertainty for the selected
strategies applied on KITTI dataset. For both depth and the uncertainty, the
brighter the pixel is, the higher depth/uncertainty value is. The figures are
arranged as follows:
Input image and ground truth depth: (a) (b). For the different strategies:
Deep Ensembles [14]: (c) (d); Adabins based [10]: (e) - (h); Dorn based [7]:
(i) - (l); Yang et al. based [9]: (m) - (p). All theses strategies are based
on FCN-ResNet101 [26, 28] backbone. The different uncertainty outputs are
given by the solutions introduced in Sec. 2.2 in the main paper.

## Overview

### Notations

Table. 4 lists some of the notations we denote and use in the Sec. 2 in the
main paper.

Types | Notations | Meanings
---|---|---
sub/super- scripts | $i$ | subscript for image/depth index
$j$ | subscript for pixel index
$p$ | subscript for channel index
${[\text{method name}]}$ | superscript for indicating different methods
common capital letters | $N$ | number of pixels with valid ground truth
$K$ | number of bins, the level of discretization
some other notations | $\mathbf{d}_{i}=\\{d_{i,j}\\}_{j}^{N}$ | ground truth depth values
$\bar{\mathbf{d}}=\\{\bar{d}_{p}\\}_{p}^{K}$ | handcrafted logarithm depth table
$\hat{\bar{\mathbf{d}}}_{i}=\\{\hat{\bar{d}}_{p}\\}_{p}^{K}$ | learned (adaptive) depth table for a given image
$q$ | width between two side-by-side bins in the depth table
$f_{\theta_{1}}(\mathbf{x}_{i})$ | output logits of the DNN $f_{\theta_{1}}$
$\hat{{\mathbf{y}}}_{i}=\\{\\{\hat{{y}}_{i,j,p}\\}_{p}^{K}\\}_{j}^{N}$ | softmax output of the DNN $f_{\theta_{1}}$

Table 4: Reminder for the notations

### Original settings for CAR strategies on KITTI experiment

Table. 5 lists the original model backbone choices and experiment settings for
different CAR strategies on KITTI experiment. For the coefficient used for
label smoothing in Cao et al. [5], according to our discussions in the end of
Sec. 2.1.2, we can transfer the original coefficient (0.5) to 65 as we report
in the table using $\gamma\cdot q^{-2}=0.5*(\log 80/50)^{-2}=65$.

As we can see, the previous works collected in Table. 1 in the main paper lack
a full comparison, and the network structures and hyperparameters they use are
also different.

### Complementary to the loss function descriptions

Multiple BCE loss: is another solution when the target is a soft discrete
distribution. Yang et al. [9] apply BCE loss on every class value in
$\mathbf{y}^{[\text{smo1}]}_{i,j}$ defined in Eq. 5. The loss function is
similar to Eq. 9:

$\displaystyle{L_{i,j}^{{[\text{MBCE}]}}}(\theta_{1})\text{=}-[{{\mathbf{y}^{[\text{smo1}]}_{i,j}}\log\delta([f_{\theta_{1}}(\mathbf{x}_{i})]_{j})\text{+}(1\text{-}{\mathbf{y}^{[\text{smo1}]}_{i,j}})\log(1\text{-}\delta([f_{\theta_{1}}(\mathbf{x}_{i})]_{j})}]$
(15)

Regression loss (Smooth L1 loss): DS-SIDENet [18] applies CAR with a smooth L1
loss [21] to fit the one-hot classification map target:

$\displaystyle{L^{[\text{smoL1}]}_{i,j}}(\theta_{1})=\left\\{\begin{array}[]{rl}0.5(\hat{d}_{i,j}-k)^{2}&{\text{if
}|\hat{d}_{i,j}-k|<1}\\\
|\hat{d}_{i,j}-k|-0.5&{\text{otherwise}}\end{array}\right.$ (18)
$\displaystyle\text{with
}\hat{d}_{i,j}=\sum_{p=0}^{K-1}\hat{{y}}_{i,j,p}\cdot(p+1)$

where $k$ is defined in Eq. 3.

Regression loss (Scale-Invariant loss): is applied along with a post-
processing to produce the continuous depth. Adabins [10] uses the per-image
adaptive depth table $\hat{\bar{\mathbf{d}}}^{[\text{ada}]}_{i}$ defined in
Eq. 8 instead of the fixed $\bar{\mathbf{d}}$, then applies a Scale-Invariant
loss [22]:

$\displaystyle{L^{[\text{SI}]}_{i}}(\theta_{1},\theta_{2})=\omega\sqrt{\frac{1}{N}\sum_{j=0}^{N}{{h^{2}_{i,j}}}-\frac{\lambda}{N^{2}}(\sum_{j=0}^{N}{h_{i,j}})^{2}}$
(19) $\displaystyle\text{with
}h_{i,j}=\log\hat{d}^{[\text{ada}]}_{i,j}-\log{d}_{i,j};\hskip
5.0pt\hat{d}^{[\text{ada}]}_{i,j}=\sum_{p=0}^{K-1}\hat{{y}}^{[\text{ada}]}_{i,j,p}\cdot{\hat{\bar{{d}}}^{[\text{ada}]}_{i,p}}$

where $\omega$ and $\lambda$ are hyper-parameters.

| CAR MDE
---
solutions
| Encoder
---
Backbone
| Choice
---
of $K$
| Param.
---
smooth
| Uncertainty
---
eval.?
| Comp. with
---
other CAR?
| Comp. with
---
Reg?
DORN [7] | ResNet-101 | 80 | - | ✗ | ✓ | ✓
Cao et al. [5] | ResNet-101 | 50 | 65 | ✗ | ✗ | ✓
Li et al. [6] | ResNet-152 | 50 (150 [16]) | - | ✗ | ✗ | ✓
SORN [8] | Xception | 120 | 1 | ✗ | ✗ | ✓
Yang et al. [9] | ResNet-50 | 128 | 15 | ✓ | ✓ | ✓
DS-SIDE [18] | Self-made | 80 | - | ✗ | ✓ | ✓
Adabins [10] | EfficientNet | 256 | - | ✗ | ✓ | ✓

Table 5: Summary on KITTI experiment settings for typical CAR strategies.
Param. smooth: Coefficient used for label smoothing. Uncertainty eval.:
Whether this work evaluates the uncertainty and compared with the other works.
Comp. with other CAR: Whether this work compared their proposed CAR strategy
with the other CAR methods. Comp. with Reg.: Whether this work compared the
CAR with the regression version of its model.

## Experiments

### Evaluation matrices

For the depth evaluation, we use the same matrices first introduced in [22]
and used in many subsequent works. We list them as follows:
1\. RMSE:
$\sqrt{\frac{1}{|D|}\sum_{\mathbf{d}_{i}\in{|D|}}||\hat{\mathbf{d}}_{i}-\mathbf{d}_{i}||^{2}}$;
2. Absrel:
$\frac{1}{|D|}\sum_{\mathbf{d}_{i}\in{D}}|\hat{\mathbf{d}}_{i}-\mathbf{d}_{i}|/\mathbf{d}_{i}$;
3. Threshold dk: Inlier metrics, $k$ in $dk$ indicates the power of the
threshold ($t$), we take $t=1.25$ and $dk=\frac{|A|}{|D|},\text{where
}A=\left\\{\mathbf{x}_{i},\text{such that
}\delta_{i}=\max(\frac{\hat{\mathbf{d}}_{i}}{\mathbf{d}_{i}},\frac{\mathbf{d}_{i}}{\hat{\mathbf{d}}_{i}})\text{
and }\delta_{i}<t^{k}\right\\}$; 4. SqRel:
$\frac{1}{|D|}\sum_{d_{i}\in{D}}{||\hat{\mathbf{d}}_{i}-\mathbf{d}_{i}||^{2}/\mathbf{d}_{i}}$;
5\. RMSElog:
$\sqrt{\frac{1}{|D|}\sum_{d_{i}\in{|D|}}{||\log\hat{\mathbf{d}}_{i}-\log\mathbf{d}_{i}||^{2}}}$;
6. log10:
$\frac{1}{|D|}\sum_{\mathbf{d}_{i}\in{|D|}}{|\log_{10}\hat{\mathbf{d}}_{i}-\log_{10}\mathbf{d}_{i}|}$.

### Illustrations

Fig. 2 shows some illustrations for predicted depth as well as the predicted
uncertainty given by different uncertainty estimation strategies.