# Predictive Coding, Precision and Natural Gradients

Andre Ofner  Raihan Kabir Ratul  Suhita Ghosh  Sebastian Stober

###### Abstract

There is an increasing convergence between biologically plausible
computational models of inference and learning with local update rules and the
global gradient-based optimization of neural network models employed in
machine learning. One particularly exciting connection is the correspondence
between the locally informed optimization in predictive coding networks and
the error backpropagation algorithm that is used to train state-of-the-art
deep artificial neural networks. Here we focus on the related, but still
largely under-explored connection between precision weighting in predictive
coding networks and the Natural Gradient Descent algorithm for deep neural
networks. Precision-weighted predictive coding is an interesting candidate for
scaling up uncertainty-aware optimization – particularly for models with large
parameter spaces – due to its distributed nature of the optimization process
and the underlying local approximation of the Fisher information metric, the
adaptive learning rate that is central to Natural Gradient Descent. Here, we
show that hierarchical predictive coding networks with learnable precision
indeed are able to solve various supervised and unsupervised learning tasks
with performance comparable to global backpropagation with natural gradients
and outperform their classical gradient descent counterpart on tasks where
high amounts of noise are embedded in data or label inputs. When applied to
unsupervised auto-encoding of image inputs, the deterministic network produces
hierarchically organized and disentangled embeddings, hinting at the close
connections between predictive coding and hierarchical variational inference.

##  1 Introduction

Predictive coding is a theory of brain function originating from cognitive
neuroscience. It casts the brain’s main function as the minimization of
prediction errors that are generated with respect to a generative model of the
world [1, 2]. Predictive coding, jointly with related process theories that
revolve around Bayesian approaches to brain function, such as active
inference, is highly influential in its field of origin, cognitive and
computational neuroscience [3]. A core aspect of predictive coding models is
their biological plausibility, which is backed up by a plethora of
neurophysiological evidence. Predictive coding is a highly interesting
candidate for the design of machine learning systems, e.g. due to the
uncertainty-aware and hierarchically organized structure of predictive coding
models.

Despite this, the influence of the predictive coding theory on machine
learning and artificial intelligence, however, is less significant at the time
of writing. There are many conceptual similarities between the state-of-the-
art in machine learning, namely deep neural networks and the error back-
propagation algorithm, and predictive coding models. Work that aims at
directly connecting these fields, however, is still relatively sparse [4, 5,
6, 7, 8]. Recently, it has been suggested that gradient based predictive
coding, when it includes precision estimations, directly implements a form of
Natural Gradient Descent, i.e. second order, uncertainty-aware optimization of
the network’s parameters [4]. Within the domain of Deep Learning, Natural
Gradient Descent (NGD) is a highly attractive optimization method, due to the
speed and stability of convergence [9, 10]. In practice, however, NGD is
oftentimes too computationally expensive and substantial effort has been
devoted to methods that approximate the underlying second order statistics
[11, 12].

Still, despite these interesting connections between predictive coding and
uncertainty-aware optimization of artificial neural networks, there is a lack
of models that actually implement precision weighting in hierarchical models
that scale to complex spatial and temporal inputs, such as they are common in
deep learning.

Here we will address this lack of precision-aware predictive coding models and
will try to clarify core conceptual questions that are still open with respect
to top-down versus bottom-up processing with precision. Our method offers an
attractive approach to uncertainty-aware optimization due to its distributed
nature that is still directly compatible with the current methodology of
training deep neural networks with (global) error backpropagation.

##  2 Related work

Existing work on predictive coding models in the domain of machine learning,
and deep learning in particular, can roughly be divided into probabilistic,
i.e. with explicit modeling of distributions or explicit sampling strategies,
and deterministic models, without a stochastic part. Models implementing
deterministic variants typically implement only selected concepts of the
predictive coding theory, such as hierarchical error prediction or predictive
representations for efficient spatio-temporal representation learning [13, 14,
6, 8]. Predictive coding models with probabilistic structure have found
application in the domains of robotics, sensory and brain signal prediction
[15, 7]. Other models explicitly compare deterministic and probabilistic
implementations to predictive coding [16]. Gradient based predictive coding,
the approach we will use here, is largely biologically plausible as all
optimization can be described with Hebbian updates rules [5, 17]. Gradient
based predictive coding is deterministic based on the nature of its updates
without explicit sampling. It does, however, allow to estimate the uncertainty
of the prediction errors that drive the optimization of activities and weights
in the network. It’s these uncertainties that we will focus on in this work.
Previous research revolving around gradient based predictive coding has
illuminated its close connection to the error backpropagation algorithm, whose
gradients have been shown to be directly approximated by discriminative
predictive coding [4]. Predictive coding models have also been compared to
other popular inference methods, such as the Kalman filter or variational
inference in general [2]. While Kalman Filters have already been shown to
implement a form of Natural Gradient Descent, the connection between natural
gradients to predictive coding networks is yet to be explored in detail [18].

Second order optimization for deep neural networks is typically addressed by
systems that approximate the second order statistics of the model with simpler
and less computationally expensive approaches. Adam and RMSProp approximate
the second order derivatives by computing the variance of the gradient [19].
Other approaches compute approximate or exact second order derivatives
directly, e.g. [12] and [11].

##  3 Precision and Natural Gradient Descent

Estimating model parameters from noisy data samples is challenging. Since
predictive coding networks possess multiple layers operating in parallel, each
layer needs to be able to quantify the expected variance (or inverse
precision) of neighboring activities. In predictive coding, this task is
solved via the propagation of precision-weighted prediction errors. These
weighted prediction errors measure the difference between the expected and
observed precision of the measured activity. As we will see later, these
precision-weighted prediction errors have a direct connection to the Fisher
information, a central quantity in Natural Gradient Descent.

In information geometry, the quality of a parameter estimation rises and falls
proportionally with the variance, or sharpness of the underlying probability
density function (PDF). We refer to the gradient of the log-likelihood
function $L$ as its score:

$u(\theta)=\frac{\partial}{\partial\theta}[\ln L(\theta)]$ (1)

This gradient, or first derivative, is zero at maximum and minimum points of
the curve and can be used to find maximum likelihood estimates of the
network’s parameters $\theta$:

$u(\hat{\theta})=0$ (2)

Under the regularity condition that the expectation of the score at the
maximum likelihood (ML) estimate is zero, the Fisher Information $I(\theta)$
measures the variance of the score and thus the sharpness (or second
derivative) of the log likelihood function:

$I(\theta)=-\operatorname{var}[u(\theta)]=-E\left[\frac{\partial^{2}\ln[L(\theta)]}{\partial\theta^{2}}\right]$
(3)

A log likelihood function with a low amount of variance will have a sharp
curve and high Fisher information, while high variance is reflected in low
Fisher information. This means that, when we have access to the Fisher
information of the log likelihood function, we can assess the accuracy of our
estimation with respect to the function’s parameters.

This property of the Fisher Information is used in Natural Gradient Descent as
an adaptive learning rate during parameter updating. In practice, however,
Natural Gradient Descent is computationally feasible only for models with
relatively low complexity, since it’s generally infeasible to compute and
store the matrix for large parameter spaces. Furthermore, a direct computation
of the Fisher information in the layers of a neural network is deemed not
biologically plausible.

More intuitively speaking, we can characterize precision as a measure of the
repeatability of a measurement. If the network makes precise measurements of a
constant quantity, the inferred value will be the same between repetitions and
the likelihood function will be sharp with variance close to zero. Precision
decreases with increasing amount of noise, either in the measured quantity or
inherent in the measuring sensor (c.f. input noise and parameter dropout noise
in deep neural networks).

Figure 1: Precision and accuracy on a simple toy example.

##  4 Method

Our suggested predictive coding network is implemented via stacked and only
locally connected network modules that are trained via standard gradient
descent and backpropagation of the relevant prediction errors. This allows to
effortlessly merge exact inference within the modules via backpropagation
(through potentially very deep and nonlinear functions) to the distributed,
uncertainty aware and bi-directional inference of the PC network. When each
module contains a single layer, the network has a structure that is equivalent
to the models discussed in [17] and [2], i.e. simplifies to gradient based
predictive coding.

Figure 2: Predictive coding model with three layers.

However, in contrast to these networks, our network actually learns the
precision embedded in its hierarchical structure and does not simply set them
to a constant value. We refer to this sort of implementation as ”local
backpropagation” and hope that it accelerates the integration of concepts from
predictive coding into the established methodology of training deep neural
networks. Figure 2 shows an overview of the predictive coding network with
precision estimation, that is employed in this work and will be discussed in
detail in later sections.

###  4.1 Precision estimation from local prediction errors

Figure 3: Directly and indirectly observable sources of uncertainty in
gradient based predictive coding.

Figure 3 shows an overview of the sources of uncertainty that arise when
estimating the precision of observed prediction errors in each layer of the
network. When all inherent timescales in the network are the same, i.e. all
learned quantities are updated at the same frequency, then only the prediction
error is directly observable. However, with increasing separation of
timescales (e.g. weights update slower than activities), the remaining sources
of uncertainty can be observed more directly. While some choice of timescales
is inherent to all neural network models, here we specifically look at their
dynamic modulation.

These timescales can either be included explicitly into the model (e.g. via
learning rate priors or fixed amount of update steps for each variable). The
timescales can also be encoded implicitly, e.g. through the adaptation of
learning rates (e.g. high learning rate is similar to making more updates).
This hints at the aspect of learned (dynamic) timescales in precision weighted
predictive coding. For example, the estimated precision in a model with rapid
weight updates looks vastly different to a model with slow weights learning
rate. Since the learning rate is directly dependent on the estimated
precision, this results in a feedback loop between currently inferred states,
network parameters and learning rates.

Following the notation of [2] and [4], the Free Energy can be defined as:

$\mathcal{F}=\sum_{l=1}^{L}\Sigma_{l}^{-1}\epsilon_{l}^{2}+\ln
2\pi\Sigma_{l}^{-1}$ (4)

with prediction errors $\epsilon_{l}$ between mean activity $\mu_{l}$ and
predicted mean activity $\hat{\mu}_{l}$ at layer $l$:

$\epsilon_{l}=\mu_{l}-f_{l}\left(\theta_{l+1},\mu_{l+1}\right)$ (5)

The dynamics of the mean activity $\mu_{l}$ and synaptic weights $\theta_{l}$
in layer l compute as:

$\displaystyle\frac{d\mu}{dt}=-\frac{\partial\mathcal{F}}{\partial\mu}=\Sigma_{l+1}^{-1}\epsilon_{o}\frac{\partial
f}{\partial\mu}\theta^{T}-\Sigma_{l}^{-1}\epsilon_{x}$ (6)
$\displaystyle\frac{d\theta_{l}}{dt}=\frac{\partial\mathcal{F}}{\partial\theta_{l}}=-\Sigma_{l+1}^{-1}\epsilon_{o}\frac{\partial
f}{\partial\theta_{l}}\mu^{T}$
$\displaystyle\frac{d\theta_{l+1}}{dt}=\frac{\partial\mathcal{F}}{\partial\theta_{l+1}}=-\Sigma_{l}^{-1}\epsilon_{x}\frac{\partial
g}{\partial\theta_{l+1}}\bar{\mu}^{T}$

with

$\epsilon_{o}=o-f\left(\mu,\theta_{l}\right)\text{ and
}\epsilon_{x}=\mu-g\left(\bar{\mu},\theta_{l+1}\right)$ (7)

The layer’s precision $\Sigma_{l}^{-1}$ of layer $i$ is the inverse variance,
which is computed as:

$\displaystyle\frac{d\Sigma}{dt}=-\frac{\partial\mathcal{F}}{\partial\Sigma}=\Sigma_{l}^{-T}\epsilon_{l}\epsilon_{l}^{T}\Sigma_{l}^{-T}-\Sigma_{l}$
(8)
$\displaystyle=\Sigma_{l}^{-1}\epsilon_{l}\epsilon_{l}^{T}\Sigma_{l}^{-T}-\Sigma_{l}$
$\displaystyle=\tilde{\epsilon}_{l}\tilde{\epsilon}_{l}^{T}-\Sigma_{l}$

where $\tilde{\epsilon}_{l}$ are the precision weighted prediction errors:

$\tilde{\epsilon}_{l}=\sum_{l}^{-1}\epsilon_{l}$ (9)

Update rules for the layer’s activities $v_{i}$ and weights $\theta_{i}$:

$v^{t+1}=v^{t}+\eta\Sigma^{-1}\frac{\partial\tilde{\mathcal{F}}}{\partial
v_{i}}$ (10)
${\theta^{t+1}=\theta^{t}+\eta\Sigma^{-1}\frac{\partial\tilde{\mathcal{F}}}{\partial\theta_{i}}}$
(11)

The Fisher information of the layer’s activity is equal to the estimated
precision error precision at that layer [2]:

$\displaystyle\mathcal{G}\left(\mathcal{F},\mu_{l}\right)$
$\displaystyle=\mathbb{E}\left[\frac{\partial^{2}}{\partial\mu_{l}^{2}}\mathcal{F}\right]$
(12)
$\displaystyle=\mathbb{E}\left[\frac{\partial^{2}}{\partial\mu_{l}^{2}}\left(\left(\mu_{l}-f\left(\theta_{l}\mu_{l+1}\right)\right)^{T}\Sigma_{l}\left(\mu_{l}-f\left(\theta_{l}\mu_{l+1}\right)\right)\right)\right]$
$\displaystyle=\mathbb{E}\left[\frac{\partial^{2}}{\partial\mu_{l}^{2}}\left(\mu_{l}\Sigma^{-1}\mu_{l}\right)\right]$
$\displaystyle=\mathbb{E}\left[\Sigma_{l}^{-1}\right]$
$\displaystyle=\Sigma^{-1}$

Similarly, the Fisher information with respect to the weights can be expressed
as the layer’s estimated precision of prediction error, scaled by the variance
of activities at the next higher layer. A slightly longer derivation of this
connection can be found in appendix A.

$\displaystyle\mathcal{G}\left(\mathcal{F},\theta_{l}\right)$
$\displaystyle=\mathbb{E}\left[\frac{\partial^{2}}{\partial\theta_{l}^{2}}\mathcal{F}\right]$
(13)
$\displaystyle=\mathbb{E}\left[\frac{\partial^{2}}{\partial\theta_{l}^{2}}\left(\left(\mu_{l}-f\left(\theta_{l}\mu_{l+1}\right)\right)\Sigma_{l}\left(\mu_{l}-f\left(\theta_{l}\mu_{l+1}\right)\right)\right)\right]$
$\displaystyle\left.=\mathbb{E}\left[\frac{\partial^{2}}{\partial\theta_{l}^{2}}\left(\theta_{l}\mu_{l+1}\right)^{T}\Sigma_{l}\theta_{l}\mu_{l+1}\right)\right]$
$\displaystyle=\Sigma_{l}^{-1}\mathbb{E}\left[\mu_{l+1}\mu_{l+1}^{T}\right]$
$\displaystyle=\Sigma_{l}^{-1}\mathbb{V}\left[\mu_{l+1}\right]$

###  4.2 Precision estimation with fixed predictions or inputs

The exact role of bottom-up and top-down propagation of precision is a largely
under-explored topic in predictive coding. The often mentioned aspect of
precision propagation within PC networks stands in contrast to the actual
precision updates that are intrinsic to each layer, without the need to
explicitly exchange information about expected precision between layers. We
hope that our model clarifies some of these issues. The inferred variance in
each layer depends on the error on two constantly changing input sources: The
posterior activities inferred from a stream of bottom-up prediction errors and
the top-down predictions from the next higher layer.

In a model with fixed weights and posterior activities in hidden layers one
can iteratively sample multiple predictions of the input per data point, or
sample multiple observations of the same data point (e.g. the same input image
changed by additive noise). We can then update the variance estimate until
convergence and only then progress to the next activity update. Comparison of
such different types of inference on variance with frozen model weights is
discussed in section 5.1. In contrast, when model parameters and activities
are updated during variance estimation, the precision-weighted prediction
errors naturally scale the learning rates of updates for each variable
individually. Experiments conducted with these models are presented in
sections 5.1.2 and . Since the learning rates directly affect the magnitude of
change over time for each parameter, this resembles implicitly learned
timescales in predictive coding models.

##  5 Results

###  5.1 Variance estimation

####  5.1.1 Unsupervised Predictive Coding

Experimental setup:

  1. 1.

Images are auto-encoded by repeatedly running forward and backward passes
through the entire network. Still, information is updated only locally between
layers.

  2. 2.

We can compare the inference process to an autoencoder, where the activity
units of the deepest layer play the role of the latent embedding. Decoding in
autoencoders can be compared to the backward prediction pass in predictive
coding. Similarly, the encoding process can be equated to the forward
propagation of prediction errors.

  3. 3.

We evaluate the unsupervised model with respect to reconstruction errors and
the level of disentanglement in the deepest layer.

  4. 4.

The PC network resembles a hierarchical autoencoder, since we can decode the
activities (i.e. latent states) in any of the hidden layers, too. We expect
the information in these layers to be less disentangled.

(a) Autoencoded digits of the MNIST dataset.

(b) Learned embedding visualization (t-SNE) on the MNIST test set.

Figure 4: Visualized reconstructions and embeddings after training for 10
epochs on the MNIST dataset.

Despite there being no explicit sampling step, the embeddings for MNIST show a
clear disentanglement when analyzed with t-SNE, hinting at the close
connections between predictive coding and variational inference.

####  5.1.2 Variance estimation with noisy data and weights

Experimental setup:

  1. 1.

controlled noise on data with mean zero and variance of ten

  2. 2.

controlled noise on weights using dropout at test time (MC dropout)

  3. 3.

uncertainty is updated in all layers in parallel, only the input layer is
evaluated

(a)

(b)

Figure 5: Estimating variance during the prediction of constant inputs with
zero mean and additive Gaussian noise with variance 10. Shown are inferred
variance values with respect to either fixed inputs (a) or predictions (b)
during variance updating. Both variants successfully infer the observed
variance, despite the noisiness (30% dropout) of the top-down prediction.

For fixed inputs or predictions, the inferred variance values converge
quickly: Variance learning rate 0.1, 10-100 updates. Variance estimates
without fixed inputs or predictions still infer the true data noise for inputs
with constant mean.

###  5.2 Weights learning with and without precision weighting

Experimental setup:

  1. 1.

single layer PC model with activity set to be the input and top-down
prediction computed via targets and weights

  2. 2.

the network is trained to infer the precision of prediction error and the
correct weights at the same time

  3. 3.

single unit input: controlled noise on data with mean 5 and variance of 2

  4. 4.

we test two conditions: controlled noise on target (mean 2,5 and variance 2)
and without target noise

  5. 5.

when correctly learned, the inferred weight value should be 2 and the variance
should be 2 without target noise and 4 with target noise

The learned precision of prediction error accounts for uncertainty from
(bottom-up) activity and (top-down) prediction. Note that the (top-down)
prediction splits up into fluctuations caused by the activity at the higher
layer and fluctuations from the weights updates.

Figure 6: Weights learning with and without precision weighted prediction
errors. Observed inputs have a constant variance of 2. The network trained
with precision weighted errors more accurately infers the observed variance.

Figure 7: Weights learning with and without precision weighted prediction
errors. Observed inputs and targets have a constant variance of 2. In this
setting, the learned precision needs to integrate observed variances from two
sources: the bottom-up inputs (from the noisy data) and the top-down predicted
input (from the noisy target). Again, only the weights updates that include
precision weighting infer the correct total variance value of 4. During
training without precision weighting, the standard deviation of both inferred
weights and precision values is significantly higher.

###  5.3 Noisy MNIST dataset

####  5.3.1 MNIST classification

Model | Input noise | Target noise | Hidden weights | Prediction Dropout | Accuracy
---|---|---|---|---|---
PC-SGD | 0. | 0. | 256 | 0.3 | 0.90
PC-Adadelta | 0. | 0. | 256 | 0.3 | 0.93

####  5.3.2 MNIST reconstruction

(a) Comparison of prior predictions from the target label and posterior
predictions after 10 updates of hidden activities using the prediction error
on the true input.

(b) Estimated variance in the input layer. True variance is zero since no
additive noise was added to the original MNIST dataset.

Figure 8: Estimating variance during the prediction of constant inputs with
zero mean and additive Gaussian noise with variance 10. Shown are inferred
variance values with respect to either fixed inputs a) or predictions b)
during variance updating. Both variants successfully infer the observed
variance, despite the noisiness (30% dropout) of the top-down prediction.

## References

  * [1] Karl Friston and Stefan Kiebel  “Predictive coding under the free-energy principle”  In _Philosophical transactions of the Royal Society B: Biological sciences_ 364.1521 The Royal Society London, 2009, pp. 1211–1221
  * [2] Beren Millidge, Anil Seth and Christopher L Buckley  “Predictive Coding: a Theoretical and Experimental Review”  In _arXiv preprint arXiv:2107.12979_ , 2021
  * [3] Karl Friston et al.  “Active inference: a process theory”  In _Neural computation_ 29.1 MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info …, 2017, pp. 1–49
  * [4] Beren Millidge, Alexander Tschantz and Christopher L Buckley  “Predictive coding approximates backprop along arbitrary computation graphs”  In _arXiv preprint arXiv:2006.04182_ , 2020
  * [5] Beren Millidge, Alexander Tschantz, Anil Seth and Christopher Buckley  “Neural Kalman Filtering”  In _arXiv preprint arXiv:2102.10021_ , 2021
  * [6] Roshan Prakash Rane et al.  “PredNet and Predictive Coding: A Critical Review”  In _Proceedings of the 2020 International Conference on Multimedia Retrieval_ , 2020, pp. 233–241
  * [7] Ahmadreza Ahmadi and Jun Tani  “A novel predictive-coding-inspired variational RNN model for online prediction and recognition”  In _Neural computation_ 31.11 MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info …, 2019, pp. 2025–2074
  * [8] Aaron van den Oord, Yazhe Li and Oriol Vinyals  “Representation learning with contrastive predictive coding”  In _arXiv preprint arXiv:1807.03748_ , 2018
  * [9] Magnus Rattray, David Saad and Shun-ichi Amari  “Natural gradient descent for on-line learning”  In _Physical review letters_ 81.24 APS, 1998, pp. 5461
  * [10] Frederik Kunstner, Lukas Balles and Philipp Hennig  “Limitations of the empirical Fisher approximation for natural gradient descent”  In _arXiv preprint arXiv:1905.12558_ , 2019
  * [11] Zhewei Yao et al.  “ADAHESSIAN: An adaptive second order optimizer for machine learning”  In _arXiv preprint arXiv:2006.00719_ , 2020
  * [12] Matthew D Zeiler  “Adadelta: an adaptive learning rate method”  In _arXiv preprint arXiv:1212.5701_ , 2012
  * [13] Kuan Han et al.  “Deep predictive coding network with local recurrent processing for object recognition”  In _arXiv preprint arXiv:1805.07526_ , 2018
  * [14] William Lotter, Gabriel Kreiman and David Cox  “Deep predictive coding networks for video prediction and unsupervised learning”  In _arXiv preprint arXiv:1605.08104_ , 2016
  * [15] André Ofner and Sebastian Stober  “Balancing Active Inference and Active Learning with Deep Variational Predictive Coding for EEG”  In _2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_ , 2020, pp. 3839–3844  IEEE
  * [16] Ahmadreza Ahmadi and Jun Tani  “Bridging the gap between probabilistic and deterministic models: a simulation study on a variational bayes predictive coding recurrent neural network model”  In _International conference on neural information processing_ , 2017, pp. 760–769  Springer
  * [17] Rafal Bogacz  “A tutorial on the free-energy framework for modelling perception and learning”  In _Journal of mathematical psychology_ 76 Elsevier, 2017, pp. 198–211
  * [18] Yann Ollivier  “Online natural gradient as a Kalman filter”  In _Electronic Journal of Statistics_ 12.2 Institute of Mathematical StatisticsBernoulli Society, 2018, pp. 2930–2961
  * [19] Diederik P Kingma and Jimmy Ba  “Adam: A method for stochastic optimization”  In _arXiv preprint arXiv:1412.6980_ , 2014

##  Appendix A Appendix A

###  A.1 Fisher information with respect to the Free Energy and model weights

$\displaystyle\mathcal{F}=D_{\mathrm{KL}}[q(x\mid o;\phi)\|p(o,x;\theta)]$
(14) $\displaystyle=\underbrace{\mathbb{E}_{q(x\mid o;\phi)}[\ln q(x\mid
o;\phi)]}_{\text{Entropy }}-\underbrace{\mathbb{E}_{q(x\mid o;\phi)}[\ln
p(o,x;\theta)]}_{\text{Energy }}$
$\displaystyle=-\underbrace{\mathbb{E}_{q(x\mid o;\phi)}[\ln
p(o,x;\theta)]}_{\text{Energy }}$

Where the entropy of the dirac-delta distribution is 0 and only the energy
term is non-zero.

The variational free energy is the sum of the prediction errors at each layer:

$\mathcal{F}=\sum_{l=1}^{L}\Sigma_{l}^{-1}\epsilon_{l}^{2}+\ln
2\pi\Sigma_{l}\\\ $ (15)

with errors $\epsilon_{l}=\mu_{l}-f_{l}\left(\theta_{l+1},\mu_{l+1}\right)$.

The Fisher information with respect to the weights of the model is defined as
the expected value of the second derivative of the variational free energy:

$\displaystyle(\sum_{l=1}^{L}\Sigma_{l}^{-1}\epsilon_{l}^{2}+\ln
2\pi\Sigma_{l})$
$\displaystyle=\mathbb{E}\left[\frac{\partial^{2}}{\partial\theta_{l}^{2}}\mathcal{F}\right]$
(16)
$\displaystyle=\mathbb{E}\left[\frac{\partial^{2}}{\partial\theta_{l}^{2}}(\sum_{l=1}^{L}\Sigma_{l}^{-1}\epsilon_{l}^{2}+\ln
2\pi\Sigma_{l})\right]$
$\displaystyle=\mathbb{E}\left[\frac{\partial^{2}}{\partial\theta_{l}^{2}}(\sum_{l=1}^{L}\Sigma_{l}^{-1}\epsilon_{l}^{2})\right]$
$\displaystyle=\mathbb{E}\left[\frac{\partial^{2}}{\partial\theta_{l}^{2}}\sum_{l=2}^{L}\Sigma_{l-1}^{-1}(f_{l-1}\left(\theta_{l},\mu_{l}\right))^{2}\right]$
$\displaystyle=\mathbb{E}\left[\frac{\partial^{2}}{\partial\theta_{l}^{2}}\sum_{l=2}^{L}\Sigma_{l-1}^{-1}f_{l-1}\left(\theta_{l},\mu_{l}\right)f_{l-1}\left(\theta_{l},\mu_{l}\right)\right]$
$\displaystyle=\mathbb{E}\left[\frac{\partial^{2}}{\partial\theta_{l}^{2}}\sum_{l=2}^{L}\Sigma_{l-1}^{-1}(\theta_{l}\mu_{l})(\theta_{l}\mu_{l})\right]$
$\displaystyle=\mathbb{E}\left[\frac{\partial^{2}}{\partial\theta_{l}^{2}}\sum_{l=2}^{L}\Sigma_{l-1}^{-1}\theta_{l}^{2}\mu_{l}^{2}\right]$
$\displaystyle=\sum_{l=1}^{L}\Sigma_{l}^{-1}\mathbb{E}\left[\mu_{l+1}^{2}\right]$
$\displaystyle=\sum_{l=1}^{L}\Sigma_{l}^{-1}\mathbb{V}\left[\mu_{l+1}\right]$


1260 Research  Scale-aware insertion of virtual objects into videos. Top:
Input frames with extracted dimension and optimized size information of
objects in videos (‘G’ for the ground-truth sizes, and ‘E’ for the estimated
sizes). The red lines indicate the extracted plausible dimensions of the
objects. Bottom: Output frames with different objects inserted with proper
sizes.

# Scale-aware Insertion of Virtual Objects in Monocular Videos

Songhai Zhang
Tsinghua University
BNRist  e-mail<EMAIL_ADDRESS>Xiangli Li
Tsinghua University<EMAIL_ADDRESS>Yingtian Liu
Tsinghua University<EMAIL_ADDRESS>Hongbo Fu
City University of Hong Kong  e-mail<EMAIL_ADDRESS>
###### Abstract

In this paper, we propose a scale-aware method for inserting virtual objects
with proper sizes into monocular videos. To tackle the scale ambiguity problem
of geometry recovery from monocular videos, we estimate the global scale
objects in a video with a Bayesian approach incorporating the size priors of
objects, where the scene objects sizes should strictly conform to the same
global scale and the possibilities of global scales are maximized according to
the size distribution of object categories. To do so, we propose a dataset of
sizes of object categories: Metric-Tree, a hierarchical representation of
sizes of more than 900 object categories with the corresponding images. To
handle the incompleteness of objects recovered from videos, we propose a novel
scale estimation method that extracts plausible dimensions of objects for
scale optimization. Experiments have shown that our method for scale
estimation performs better than the state-of-the-art methods, and has
considerable validity and robustness for different video scenes. Metric-Tree
has been made available at: https://metric-tree.github.io

Computing methodologiesComputer graphicsGraphics systems and interfaces

##  1 Introduction

As one of most important research problems in computer graphics and VR/AR,
inserting virtual objects into real video scenes has enormous applications for
both individual users and mass video owners. For example, automatic insertion
of virtual objects into mass videos brings advertisers and content owners new
opportunities by monetizing their video assets. Seamless merging of virtual
objects into videos should take into account many aspects, including scene
geometry recovery[27, 4], illumination recovery[15], rendering[15], and an
inserted object’s position[40] and sizing[35]. However, most of the videos
capturing real-world scenes are captured by monocular cameras often without
any recording camera parameters, and thus existing geometry recovering methods
[27, 4] often fail to recover the actual scene metrics of such videos. This is
known as the scale ambiguity problem.

Figure 1: The miniature brings fake illusion to people on the size of a scene.

We observe that the perception of an actual size of a scene in an image mainly
depends on the knowledge of the sizes of objects in the scene[14] (see Fig.
1). There are certain regular sizes of many objects in human and natural
environments, such as the fixed size of A4 papers, limited size choices for
beds, and a limited size range for chairs due to their use for human users.
Such sizes following the customs of humans and size distribution statistics of
natural objects, bring the background knowledge for size perception and thus
make it possible to automatically estimate the actual sizes of scenes in
monocular videos.  This motivates us to design a two-stage system to estimate
the scaling factor between a physical (or actual) size and the size of a 3D
scene recovered from a monocular video. Our approach first extracts the
plausible dimensions of objects from semantic segmentation of a 3D point cloud
reconstructed from an input monocular video and then optimizes the scaling
factor by incorporating the actual size distribution of object categories. In
this way, the actual scene size in a monocular video can be calculated and
virtual objects can be inserted with proper sizes.

Some pioneer works[35, 25] have shown the effectiveness of such hints in scale
estimation in scene recovery from monocular videos, and discovered at least
two difficulties, making the problem non-trivial:

  * •

Objects might be partially detected or partially visible, making their size
estimates inaccurate.

  * •

The scale estimation decreases in accuracy due to intra-class variations in
size, or even fails if no object is detected.

We propose the following novel strategies to cope with these two issues, and
make scale-aware insertion of virtual objects more automatic.

Limited capturing views often cause incompleteness or inaccuracy of object
geometry recovery by sparse or dense structure from motion (SFM) methods, so
that the three dimensions of a 3D object detected in a video may not be all
plausible to depict the size of this object. Observing the spatial features of
incomplete objects, we extract plausible dimensions of objects from semantic
segmentation of the point cloud. A key observation is that all the lengths in
the recovered geometry divided by their physical sizes should strictly conform
to a global scaling factor. As the variance in sizes of object categories, we
optimize the scaling factor by maximizing the likelihood of lengths of
extracted dimensions divided by their physical sizes according to the size
distributions of objects of these dimensions.

The richness of object categories and the accuracy of their size distributions
are crucial to estimate the scaling factor. The existing works take advantage
of size priors by indicating the heights of several object categories, such as
bottles [35]. In order to put it in application, we collect the physical size
prior of different semantic objects from Amazon and Wikipedia, and build
_Metric-Tree_ , a hierarchical representation of sizes of objects according to
their category trees. Metric-Tree has five hierarchical levels and more than
900 object categories as leaf nodes, covering the furniture, car, electric
appliance, person, and so on. For each category, we build a Gaussian Mixture
Model (GMM) model on 3D sizes of the associated objects according to the
collected size data.

The results of different interior and outdoor scenes show that although every
object category contains variance in size, the scale estimation error rapidly
decreases along with the number of objects involved in scale optimization. The
scale estimation error can be reached less than 8% in average in the Kitti
dataset for outdoor videos and our captured indoor-scene videos in daily
lives.

In summary, our work has two contributions:

  * •

We propose a large-scale dataset of sizes of object categories: Metric-Tree, a
hierarchical representation of sizes of more than 900 object categories as
well as the corresponding images, connecting the appearance and the size of
each object category.

  * •

We propose a novel scale estimation method that extracts plausible dimensions
of objects for scaling factor optimization to alleviate the incompleteness of
geometry recovery from monocular videos.

##  2 Related work

###  2.1 Object insertion in videos

Synthesizing realistic images or videos by merging virtual objects into real
world scenes seamlessly is one of the longstanding goals of computer graphics
and one of the main applications in AR/VR. Although there is still no
completely automatic solution as far as we know, different aspects of relative
research have been taken out to make this process more intelligent and thus
automatic. The underlying geometry of scenes in videos can be recovered by
structure from motion (SFM) and visual simultaneous localization and mapping
(vSLAM) techniques[27, 4]. Capturing, estimating and rendering with scene
illumination are summarized by [15]. Context-based recommendation for object
insertion in visual scenes is a relatively new topic, and there have been some
pioneer works on recommendation in image [40] by modeling the joint
probability distribution of object categories, and object recommendation
systems by neural networks [37]. For automatic insertion of virtual objects
into monocular videos, the size of inserted objects is a critical factor to
influence the photo-realistic effects of the resulting videos. However, the
scale ambiguity problem for monocular videos has been largely unexplored and
is the focus of our work.

###  2.2 Scale estimation

Due to the classical problem of scale ambiguity in the 3D reconstruction from
monocular videos, at least one scale-related piece of knowledge needs to be
introduced to recover the actual size of the overall scene. Some methods
combine sensors, such as inertial measurement units (IMUs) [23, 21, 9, 11,
7]and LiDAR [3, 39] into SLAM systems to estimate the unknown scaling factor.
Other methods incorporate camera setup information as priors into SLAM
systems, such as camera height in a car based on the ground plane assumption
[33, 34, 45, 10, 44] and the information of the camera’s offset relative to a
vehicle’s axis of rotation when the vehicle turns [30]. Those methods with
additional sensors or camera setup information achieve impressive
results,whereas they do not meet our need for addressing the scale estimation
problem for monocular videos in absence of the camera parameters.

In the autonomous driving area, a large number of visual odometry systems
incorporate semantic information by object tracking or instance segmentation
to address the scale drift problem [2, 13, 36, 18], where the basic idea is to
find semantic-level feature correspondences among key frames and combine them
with feature matching into bundle adjustment. Their goal is to alleviate the
scale drift same as the loop closure, and our method is to estimate the actual
size of a scene captured in a monocular video.

Ku et al. [16] proposed a 3D object detection method from an image, and derive
3D bounding boxes of three object categories including car, pedestrian and
cyclist, in actual sizes. To achieve this, they take advantage of the LiDAR
data in training to learn the shape and scale information. The strong prior
knowledge of point clouds of scenes provides pretty good estimation on object
sizes but also limits the application of their approach to a wider range of
scenes. Our method makes use of a new dataset of sizes of object categories
and their corresponding images for instance segmentation and scaling factor
estimation. Our dataset is much easier for expansion than point clouds to cope
with many more new types of objects.

Sucar et al. [35] present a pioneer work on scale estimation from monocular
videos,  and their approach is the closest to ours. They use the YOLO v2
network for the object recognition task and project an object’s bounding box
in an image frame into a 3D scene to calculate the object’s height and thus
estimate the scaling factor based on the assumed height distribution. They
experimentally demonstrate the feasibility of their method under ideal
conditions, but their performance on real object size distributions has not
been evaluated. One shortcoming with their approach is its prescription for
the vertical orientation of the scene. We propose a more advanced method for
plausible dimension extraction of objects and incorporate a size dataset of
object categories for scaling factor optimization, thus achieving more
accurate results (see Section 6.1).

###  2.3 Datasets of object sizes

Some existing works have collected size data for sizing the 3D shapes in a 3D
collection. Shao at el. [31] proposed a method for transferring physical scale
attributes between web pages and 3D Shapes. They leverage the text and image
information to connect web pages and shapes. The former is used for matching
web page text with object text to build a direct connection. The latter relies
on visual similarity to construct a joint embedding space between images and
shapes. Finally, scale attributes can be transferred between the closest pairs
in the embedding space. Savva at el. [28] proposed a probabilistic graphical
model for sizing the collections of 3D shapes. They also collect 3,099
furniture items in 55 categories and transfer the sizes to other 3D shapes by
maximizing the probability distribution with size priors of object categories
and the scale consistency of co-occurrences of objects in 3D scenes in virtual
scenes. Savva at el. [29] further connected more physical attributes of
objects with 3D shapes including weight, static support and attachment
surfaces. We focus on the sizes and appearances of object categories in the
image space, and collect a much larger dataset involving more than 900
categories with sufficient samples in each category to support scene size
understanding. Some professional websites, such as
dimensions.guide111https://www.dimensions.guide also provide sizes of everyday
objects and spaces that make up the world. These websites mainly serve
industrial design, and have only a few typical samples for each category.

Figure 2: The pipeline of our method.

##  3 Overview

Our system for scale-aware virtual object insertion into monocular videos
follows the general pipeline of virtual object insertion, as shown in Fig. 2.
It begins with reconstructing a 3D scene represented as a point cloud from an
input monocular video [1]. We then perform instance segmentation of the point
cloud by fusing per-frame segmentation. The key step is scale estimation
between the point cloud and the actual scene (Section 5). In this step, due to
the incompetence of objects in the point cloud, we extract plausible
dimensions of objects and optimize the scaling factor by incorporating with
the priors of size distributions of the corresponding object categories. We
assume the actual sizes of virtual objects are known, so that they can be
inserted with proper sizes by multiplying the estimated scaling factor. Our
proposed dataset Metric-Tree (Section 4) provides not only the strong priors
of size distributions to scaling factor optimization, but also provides image
samples for semantic segmentation.

Figure 3: The structure of Metric Tree. The hollow dots are expanded nodes and
the solid dots are folded nodes.

##  4 Metric-Tree: A Repository of Object Sizes

As mentioned above, the richness of object categories and the accuracy of
their size distributions are crucial to estimate the scaling factor. However
there is no open source repository of object dimensions, and thus we have to
collect abundant sizes of objects for constructing distributions of sizes. Our
approach is based on three key facts and assumptions:

  * •

In the real world, the dimensions of most objects are fixed, or with a small
range of changes in some dimensions.

  * •

The dimensions of objects’ minimum bounding boxes are consistent and very
close to physical sizes.

  * •

The sizes of objects in a real scene are interrelated, and the sizes of
objects in the same scene should be consistent.

Webpage Data. Like [31], we collect the sizes of objects with their images and
texts through the Internet. We crawl the Amazon websites to extract the
physical sizes of object categories (e.g., ”table”, ”chair”, etc.). Besides,
for the other objects that do not appear in Amazon (like ”car”, ”hydrant”,
”person”), we crawl from Wikipedia, Crate&Barrel, car websites, etc. for
gathering their sizes and corresponding images. We finally get about 10,000
raw categories, among which there are about 80,000 raw items with category
annotation, and images (from the source websites we crawled, and most of them
are with white background). After removing categories without any size
information and merging the similar categories, we build a dataset of object
sizes with a five-level tree structure based on WordNet [22] with about 900
categories in the leaf nodes.

Figure 4: Size distribution of severeal typical object categories.

Metric-Tree. As shown in Fig. 3 (which shows the part of Metric-Tree, due to
space limitations), Metric-Tree is a tree structure, with each node being an
object category. Each node has two components: a set of dimension data and a
dataset of images of the corresponding object category. We organize the size
data based on WordNet by generating a tree structure and attaching the
dimensions and corresponding images to leaf nodes. For categories not included
in WordNet, we use a multi-person proofreading method to insert them into the
data structure. For inner nodes, the dimension data set and image data set are
all defined as the aggregate of all their children. For each category, we
build a Gaussian Mixture Model (GMM) for 3D sizes of the objects as a size
distribution according to its dimension data set.

We also use BASNet [24] to perform foregound segmentation  to extract
foreground objects.

As the state-of-the-art fine-grained classification methods still do not work
well, we retrain Mask R-CNN[12, 38] as our instance segmentation network. We
select 43 object categories of segmentation according to the following rules:

  * •

The divergence of size distribution of object categories is low. Fig. 4
illustrates the size distributions on height of several typical categories
with different divergence.

  * •

The number of samples in the dimension data set is enough to depict the size
distribution.

  * •

There are additional training data in other famous datasets.

The traing set of the images is the combination of image datasets of
corresponding object categories and correponding subsets with the label of
object categories in the COCO dataset[19], ADE20K[43, 42] and Open Images[17].

##  5 Scale estimation with plausible dimensions

###  5.1 Instance segmentation on point clouds

Figure 5: Instance segmentation results from Mask R-CNN.

We use the open source OpenSfM [1] system integrated with the PatchMatch [32]
to reconstruct 3D scenes, and use Mask R-CNN[12, 38] retrained by our
collected image data(Section 4) to perform instance segmentation(Fig. 5).
Although some existing methods provide instance-level point cloud
segmentation, such as MaskFusion[26], given that it uses RGB-D inputs, we
cannot ensure its validity with RGB inputs, so we clarify our approach below.

After 3D reconstruction and instance segmentation on frames
$\\{F_{i}\\}_{i=1}^{N}$, for frame $F_{i}$, we get the point cloud $S_{i}$,
the camera pose $C_{i}$ and the segmentation results
$\\{O_{ij}\\}_{j=1}^{M_{i}}$, where $M_{i}$ is the number of recognized
objects and $O_{ij}$ includes the class $c_{ij}$ of the $j$-th object $o_{ij}$
and the pixel-level mask $m_{ij}$(see Fig 5 for example). We map the 2D
instance segmentation results to the point cloud. Each point is labeled
according to the 2D instance where its projection is located. Given the point
cloud $S_{i}$ of $i$-th frame and segmentation results
$\\{O_{ij}\\}_{j=1}^{M_{i}}$, the reconstructed point cloud $s_{ij}$ of object
$o_{ij}$ in this frame is

$s_{ij}=\\{\ p\ |\ p\ \in S_{i}\ \wedge\ \hat{p}\in m_{ij}\\},$ (1)

where $\hat{p}$ denotes the projection of $p$ in frame $i$.

The next step is to merge point clouds of the same object that are split in
different frames to get the complete point cloud of that object. For a real
object $A$ in the scene, let
$O_{{i_{1}}{j_{1}}},\dots,O_{{i_{N_{A}}}{j_{N_{A}}}}$ denote the recognition
of $A$ in different frames, where $N_{A}$ is the times $A$ appears. Then the
merged point cloud of $A$ is

$S_{A}=\cup_{k=1}^{N_{A}}\ s_{{i_{k}}{j_{k}}}.$ (2)

To solve the problem of correspondence of instances of the same object in
multiple frames, we propose an incremental point-cloud merging method that
recovers all real objects in the scene by merging point clouds frame by frame.
In the following discussion, without loss of generality, we consider only the
case where objects are of the same class, otherwise we can split the point
clouds by classes first, since point clouds with different classes obviously
do not correspond to the same objects.

Let $U=\\{u_{k}\\}_{k=1}^{K}$ denote the intermediate point cloud during
merging process and $K$ denote the number of objects we have obtained.
Initially, $U$ is empty. If the previous $g$ frames have been merged, then we
consider the relationship between the current $U$ and the reconstructed point
cloud $S_{g+1}$ of the $g+1$ frame: some of the point clouds in $S_{g+1}$
correspond to objects already in $U$ and some of them are new objects (the
subscript $g+1$ is omitted following for brevity). Here we define the distance
between point cloud $A=\\{a_{i}\\}_{i=1}^{M},B=\\{b_{i}\\}_{i=1}^{N}$ (without
loss of generality, let $M\leq N$) as

$D(A,B)=\frac{1}{M}\sum_{i=1}^{M}\min_{\forall b_{j}\in B}\lVert
a_{i}b_{j}\rVert.$ (3)

where $M=|S|$ denotes the number of instances recognized in frame $g+1$. We
calculate the distances of $u_{k}\in U$ and $s_{j}\in S$ based on above
distance definition. Then we greedily look for the point cloud pair
$(u_{k},s_{j})$ with the smallest distance and mark them as the same object
until the distance $D(u_{k},s_{j})$ exceeds a predefined threshold or one of
$U$ or $S$ has completed the match. After repeating this process frame by
frame, we complete the point cloud merging(see 1 for pseudocode). The merged
point cloud may have some noise resulting from incorrect feature point
reconstructions or inaccurate merge, so we can obtain the main part of each
object after further de-outlier operations. Here KNN[5] and the isolated
forest method[20] are used.

Figure 6: 4 bottles are recovered incompletely due to blocking in the scene
shown in Fig 5. Dimension extraction is required to obtain plausible
dimensions.  Algorithm 1 Incremental point cloud merging

1:point clouds from all frames $S=\\{s_{i}\\}_{i=1}^{N}$

2:object point clouds $U=\\{u_{k}\\}$

3:$U\leftarrow$ empty list

4:for $s_{i}\in S$ do

5: $U\leftarrow$ MergePointCloud($U,s_{i}$)

6:end for

7:return $U$

8:

9:function MergePointCloud($U,s$)

10: $m\leftarrow|U|$, $n\leftarrow|s|$

11: $\hat{U}\leftarrow U$, $\hat{s}\leftarrow s$

12: $U\leftarrow$ empty list

13: while $\hat{U}\ \textit{not empty}\ \textbf{and}\ \hat{s}\ \textit{not
empty}$ do

14: $u_{i},s_{j}\leftarrow\operatorname*{arg\,min}_{\forall u_{i}\ \in\
\hat{U},\ s_{j}\ \in\ \hat{s}}D(u_{i},s_{j})$

15: if $D(u_{i},s_{j})<\ \textbf{threshold}$ then

16: break

17: end if

18: $\hat{U}=\hat{U}\setminus u_{i},\ \hat{s}=\hat{s}\setminus s_{j}$

19: $U=U\cup\\{(u_{i}\cup s_{j})\\}$

20: end while

21: $U=U\cup\hat{s}$

22: return $U$

23:end function

###  5.2 Dimension extraction for 3D objects

Since in practice we mostly use the length, width and height of an object,
also known as dimensions, to describe its size. To determine the orientations
of the object, We first use the camera pose to determine the orientation of
the bottom. In order to reduce the uncertainty of the orientation estimation,
we make the following assumptions about the camera and the scene: the roll of
the camera is zero, and the object is placed on a flat, horizontal surface.
Based on these assumptions, we can constrain the direction of the bottom
surface of the object. Let $\\{\vec{r_{i}}\\}_{i=1}^{N}$ denote the $x$ axis
of the camera in all frames, as well as the right directions. Then the normal
vector $\vec{n}$ of the desired horizontal plane is the solution of this
least-squares minimization problem by denoting $R=[r_{1}r_{2}\dots
r_{N}]^{T}$:

$\vec{n}\ =\ \operatorname*{arg\,min}_{\vec{x}\ \in\ \mathbb{R}^{3},\
\lVert\vec{x}\rVert=1}\ \lVert\ R\vec{x}\ \rVert.$ (4)

It is easy to know that $\vec{n}$ is the unit eigenvector corresponding to the
minimum eigenvalue of $R^{T}R$. After the bottom of the object is identified,
the problem is reduced to a 2D point cloud dimension calculation, which can be
solved by minimum bounding box method.

We have found that the dimension extraction of objects is susceptible to the
results of reconstruction and segmentation, with the corresponding dimension
of an object being inaccurate when the local point cloud is relatively sparse
and incorrectly segmentation occurs. We therefore introduce dimension
confidence (also called reliability) to measure the results and to provide
guidance for subsequent scale optimization. A reliable reconstructed point
cloud should have a similar density in all regions, so we estimate the
reliability of the dimension computation based on this assumption.

We divide the 3D bounding box where the object locates into $8\times 8\times
8$ space grids and count the points in each grid. Let
$\mathbb{G}=\\{G_{(x,y,z)}\\}_{x,y,z=1}^{x,y,z=8}$ denote these grids and
$N(G)$ denote the number of points $G$ contains where $x,y,z$ correspond to
the direction of length, width and height, respectively. We define the global
density of the point cloud as follows:

$\rho_{g}=\frac{\sum_{G\in M}N(G)}{|M|}\ ,\\\ where\ M=\\{G|G\in\mathbb{G}\ ,\
N(G)>0\\}$ (5)

and similarly define the density of point clouds on both sides of the length
direction as follows:

$\displaystyle\rho_{x_{head}}=\frac{\sum_{G\in
M_{x_{head}}}N(G)}{|M_{x_{head}}|}\ ,$
$\displaystyle\rho_{x_{tail}}=\frac{\sum_{G\in
M_{x_{tail}}}N(G)}{|M_{x_{tail}}|}\ ,$ (6) $\displaystyle where$
$\displaystyle\ M_{x_{head}}=\\{G_{(x,y,z)}|G_{(x,y,z)}\in M,x=1\\}\ ,$
$\displaystyle\ M_{x_{tail}}=\\{G_{(x,y,z)}|G_{(x,y,z)}\in M,x=8\\}.$

In this way we can determine the reliability of the beginning and end of the
object in that direction, that is, the reliability of the dimension. The
confidence (see Figure 7 for  an example) of this dimension is defined as
(similarly for other dimensions):

$\eta_{x}=\frac{1}{\rho_{g}}\sqrt{\rho_{x_{head}}\cdot\rho_{x_{tail}}}.$ (7)

When the confidence falls below a certain threshold, we assume that the
results of this dimension are unreliable and accordingly do not use the
distribution of this object in three dimensions in the scale optimization
process, but instead degrade it to a two-dimensional distribution.

Figure 7:  Top: an example scene.  Bottom: the dimensions and confidence of
different objects. The first row contains apple, notebook, and bottle. The
second row contains laptop, mouse, and table. The data are annotated in the
form of [dimension]([confidence]). Unreliable dimensions are marked red if we
take 0.7 as the threshold.

(a) 0 degree (b) 30 degrees

(c) 45 degrees

Cases |  $\eta_{x}$ |  $\eta_{y}$ |  $\eta_{z}$
---|---|---|---
a |  0.912 |  1.077 |  0.926
b |  0.885 |  1.001 |  0.543
c |  0.593 |  1.045 |  0.291

Figure 8: Top:  A bottle is placed at three different angles relative to the
vertical direction. The three cases are (a) 0 degree, (b) 30 degree s and (c)
45 degree s, respectively. Bottom: confidence of three dimensions in the three
cases.

Usually, objects are placed upright on a horizontal plane. In such cases, our
dimension extraction is accurate and efficient. when objects are tilted, the
estimated height of these objects can be smaller and the length and width can
be larger accordingly, and however the confidence on these dimensions will
also go down (see Figure 8 for example), so that these dimensions tend not to
be selected. Even there are few such dimensions extracted with errors, the
accuracy of our algorithm will not decrease significantly, since we use all
the objects in the scene to optimize the scaling factor.

###  5.3 Scaling factor optimization

The reconstructed point cloud is similar with the real scene, and there is a
proportional coefficient $s$ between them. For a distance estimated by the
point cloud $l$, let $l^{*}$ denote the real distance and they have the
following relationship:

$s\ =\ \frac{l^{*}}{l}.$ (8)

We call $s$ as the scaling factor. The goal of scale optimization is to find
the most likely scaling factor $s^{*}$ based on the statistical distribution
of object sizes. Let $\\{m_{i}\\}_{i=1}^{N}$ denote the objects that appear in
the scene, where $m_{i}$ has its dimensions as

$L_{i}=(w_{i},l_{i},h_{i}),\ i=1,2,\dots,N$ (9)

Let $\\{\varphi_{i}(L)_{i=1}^{N}\\}$ be the obtained size prior, each
described by a GMM model. Assuming that the size of each object is independent
of each other, with Bayesian Rule, we have

$\displaystyle P(s|m_{1},m_{2},\dots,m_{n})$ (10) $\displaystyle\propto
P(s|m_{1},m_{2},\dots,m_{n-1})\ P(m_{n}|s)$
$\displaystyle\propto\prod_{i=1}^{N}\ P(m_{i}|s)$
$\displaystyle\propto\prod_{i=1}^{N}\ \varphi_{i}(sL_{i}).$

Thus the best scaling factor $s^{*}$ is

$s^{*}=\operatorname*{arg\,max}_{\forall s\in\mathbb{R^{+}}}\ \prod_{i=1}^{N}\
\varphi_{i}(sL_{i}).$ (11)

Since it is difficult to find mathematically precise optimal values for the
above problems, we estimate the above probabilities on a series of discrete
candidate values $s\in[s_{min},s_{min}+\Delta s,\dots,s_{max}]$, and select
$\hat{s}$ with the maximum probability as the estimated scaling factor. In the
course of the experiment, it was found that usually $\prod_{i=1}^{N}\
\varphi_{i}(sL_{i})$ is well convex near the maximum value, thus almost
ensuring that $\lvert\hat{s}-s^{*}\rvert<\Delta s$.

In the actual optimization, because some objects do not have high confidence
in a certain dimension, or the certain dimension of some category of objects
is insignificant or hard to collect, in such cases we do not use all three
dimensions, but rather prescribe which dimensions of a certain type of objects
to use as a priori distribution (e.g., length-width distribution for keyboard,
height distribution for human, etc.), that is, determining $\varphi_{i}$, and
consider the calculated confidence to decide which dimensions of objects to
use, that is, determining $L_{i}$.

##  6 Results and discussion

In this section, we conduct several experiments including comparisons to the
existing techniques to verify the effectiveness of our method. We also present
virtual object insertion results as well as qualitative evaluation by a user
study. Finally, we perform additional experiments to show the necessity of
dimension extraction and the importance of richness of size priors to the
accuracy of scaling factor optimization.

Figure 9: Inserted object s with their physical sizes.

Figure 10: Synthesis results of insertion of virtual objects in Fig.9. Rows 1
and 3 are input videos and Rows 2 and 4 are output videos. Figure 11: The
results of user study.

###  6.1 Comparison on scene size estimation

We choose [35] as our baseline, since the goal of this work is similar to
ours, which is to calculate the global scale of the scene. We compare the
performance of scale estimation by these two approaches on two datasets: the
Kitti raw data[8] as an outdoor dataset and our captured indoor video dataset.

Datasets. To demonstrate the robustness and superiority of our method, we use
both an indoor dataset and an outdoor dataset for comparison. For the indoor
dataset, we captured 5 indoor videos including sitting room, kitchen, toilet,
bedroom and study and measured the sizes of objects in these videos as a
validation set. For the outdoor dataset, We choose 18 video sequences from the
Kitti raw Data supported by Geiger et al. [8] according to the segmentation
results. This outdoor dataset contains processed color stereo sequences and 3D
Velodyne point clouds, with the known camera parameters and 3D object tracklet
labels.

Comparison. For fair comparison, we use Mask RCNN as the 2D object detector
and OpenSfM as the reconstruction module for our approach and [35]. We
calculate the accuracy of the global scale by using the relative error of the
estimated object sizes and the physical object sizes for validation. In our
experiments, we use millimeter as the unit. Since the method in [35] can only
estimate the heights of objects, we use the mean of the relative error between
the estimated object height and the ground-truth height over all objects as
the error metric. The comparison results in Table 1 show that the relative
error of our algorithm is nearly 10% lower than the baseline, reflecting the
robustness and superiority of our method.

Processing timings. We tested the performance of our method on a PC with
i7-6850k CPU and Nvidia 1080 Ti, 32G RAM. In our experiments, video frames
range from 100 to 400, and the processing time of OpenSfM is relative to the
square of the number of frames. For example, if there are about 100 video
frames, OpenSfM takes about 36 mins. The processing time of other modules is
relatively fixed. Specifically, instance segmentation and point cloud
segmentation take about 0.3s and 0.2s per frame, respectively. Dimension
extraction takes about 0.02s, and the optimization step takes about 0.004s for
an entire video. Therefore, a real-time vSLAM and a lightweight instance
segmentation implementation would greatly improve the overall performance.

Table 1: Performance comparison on video datasets. Method | Amount | Sucar et al. | Ours
---|---|---|---
|  | Err | Std | Err | Std
Indoor | 5 | 0.168 | 0.049 | 0.072 | 0.025
Outdoor | 18 | 0.163 | 0.099 | 0.074 | 0.034

###  6.2 Virtual object insertion results

We present 10 synthesis virtual object insertion results for indoor and
outdoor scenes with automatic determination of the sizes of the inserted
objects, shown in Fig. Scale-aware Insertion of Virtual Objects in Monocular
Videos and 10. The corresponding videos can be found in the supplementary
materials. Note that currently we manually choose the position in the scenes.

We conduct a user study to evaluate the synthesis results qualitatively. We
sample 30 frames from the above synthesis results (3 for each) containing the
inserted virtual objects from near, medium and far distant for outdoor videos
and from different views for indoor videos. Fig. 10 shows a few representative
frames.

For each frame, we also generate the synthesis results for the same virtual
objects at the same positions with only the difference in size with the
interval [-30%,-20%,-15%,-10%,-5%,0%,5%,10%,15%,20%,30%] relative to our
estimated sizes. Then 27 subjects were invited to judge whether or not the
inserted objects in the synthesis results are reasonable in size by answering
Yes or No. All the subjects were university students above the age of 20.

Figure 11 illustrates the average scores of all scenes from all participants.
The horizontal axis corresponds to the the difference in size relative to our
estimated sizes, and the vertical axis is the averaged score (1 for YES) for
all participants.

The peak of the curve is 0.65, showing that human size judgments have
significant variation, and can be influenced by context and familiarity[6].
Despite the variation, there are still some useful conclusions from the curve.
First, our score is above 0.5 showing that more than half participants agree
with our results. And the peak is on the -0.05 of X-axis, which means that the
ground truth of size appears at 0.95 times our estimated sizes, showing that
the scale error of our method is around 5%.

###  6.3 Discussion

With/without dimension extraction. There is a simple strategy by sending all
the three dimensions of segmented objects in point clouds to scaling factor
optimization in order to provide more hints on sizes. This is helpful
apparently only for relatively good 3D reconstruction results from carefully
captured videos, for example, with the camera moving around a center of the
scene. However, most of the captured videos involve more complicated camera
movement, causing only partial geometry recovered from the videos. We did an
experiment on the same scenes in scaling factor optimization with the
extracted dimensions or directly with the dimensions of bounding boxes. The
results show that the relative error of the scaling factor increases from 7%
to 30%.

Figure 12: The relation between extracted dimensions and scale accuracy.

Limitations. Our method estimates actual object sizes in the process of 3D
reconstruction, and thus would fail when OpenSfM fails, e.g., due to static
views or highly dynamic scenes. We assume that the intrinsic parameters of the
camera are fixed and the roll of the camera is zero, which are common cases in
real life, so that scene object sizes should strictly conform to the same
global scale. Our method can cope with scenes with some objects that are not
upright, but will fail if a scene is totally in a mess.

Relation between scale accuracy and the number of extracted dimensions. We
show some experiment results to illustrate how the estimated accuracy of scale
is related to the number of extracted dimensions of objects incorporated in
scale optimization. Fig. 12 shows the relation between extracted dimensions
and scale accuracy of 23 scenes. In this experiment, we gradually decrease the
number to 3 of extracted dimensions for simulation and illustrated as scatter
plot. The figure shows the rapidly descending trends of scale accuracy along
with the number of extracted dimensions.

We did a simulated experiment on the Structure3D dataset [41]. This dataset
contains 3,500 house designs and over 343 thousand instances in 24 categories.
The size distributions of all 24 categories are calculated by instance size
statistics. We select different numbers $N$ of objects randomly from each
house,  then disturb the sizes of selected objects in no more than a relative
error $R$, and  finally estimate the average of scale of each house. Fig. 13
is drawn with $N$ in range of $[1-10,20,50]$ and $R$ in range of
$[0,3\%,6\%,9\%,12\%,15\%]$. The results provide us several interesting
observations, which can guide the choice according to the performance of
semantic segmentation and 3D reconstruction:

  * •

The scale error decreased rapidly when the number of recognized objects is
below 10.

  * •

The accuracy of bounding box of recovered objects in scenes is a critical
factor to scale estimation, so that the extraction of plausible dimensions is
quite important to the incomplete and inaccurate recovered geometry.

  * •

In the case of the same number of recognized objects, our scale accuracy in
inaccurate geometry can compete the simulated results depending on the precise
size distribution, showing that a fine-grained classification method is
preferred to indicate a more precise size distribution of objects.

Figure 13: The scale accuracy in simulation data.

##  7 Conclusion and future work

With the aid of the object size distribution in Metric-Tree, we are able to
make relatively accurate scale estimates of the scenes in monocular videos
without other size inputs, resulting in the scale-aware object insertion. And
the experiments on real scenes show that our method is a significant
improvement on the scale estimation problem relative to similar previous work.
Besides, the user study indicates that such virtual object insertion results
are consistent with users’ perceptions of scales.

Metric-Tree with a large physical size information as priors can be applied to
more and more visual fields, including scale drift correction in automatic
driving drift, layout scheme optimization in 3D scene synthesis, object pose
optimization for object pose estimation. The collected images are helpful to
fine grained classification, and conversely, the scale estimation also needs a
fine grained classification method to indicating more precise size
distribution of the objects.

###### Acknowledgements.

The authors would like to thank all reviewers for their thoughtful comments.
This work was supported by the National Key Technology R&D Program (Project
Number 2017YFB1002604), the National Natural Science Foundation of China
(Project Numbers 61521002, 61772298), Research Grant of Beijing Higher
Institution Engineering Research Center, and Tsinghua–Tencent Joint Laboratory
for Internet Innovation Technology.

## References

  * [1] M. Adorjan.  opensfm ein kollaboratives structure-from-motion system.  betreuer/in (nen): M. wimmer, m. birsak, 2016.
  * [2] T. Botterill, S. Mills, and R. Green.  Correcting scale drift by object recognition in single-camera slam.  IEEE Transactions on Cybernetics, 43(6):1767–1780, 2013.
  * [3] C. Brenneke, O. Wulf, and B. Wagner.  Using 3d laser range data for slam in outdoor environments.  In Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003)(Cat. No. 03CH37453), vol. 1, pp. 188–193. IEEE, 2003.
  * [4] G. Chahine and C. Pradalier.  Survey of monocular slam algorithms in natural environments.  In 2018 15th Conference on Computer and Robot Vision (CRV), pp. 345–352. IEEE, 2018.
  * [5] T. Cover and P. Hart.  Nearest neighbor pattern classification.  IEEE transactions on information theory, 13(1):21–27, 1967.
  * [6] J. Fredebon.  The role of instructions and familiar size in absolute judgments of size and distance.  Percept & Psychophys, 51(4):344–354, 1992.
  * [7] V. Garro, G. Pintore, F. Ganovelli, E. Gobbetti, and R. Scopigno.  Fast metric acquisition with mobile devices.  In VMV, pp. 29–36, 2016.
  * [8] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun.  Vision meets robotics: The kitti dataset.  The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  * [9] V. Grabe, H. H. Bülthoff, and P. R. Giordano.  A comparison of scale estimation schemes for a quadrotor uav based on optical flow and imu measurements.  In 2013 IEEE/RSJ international conference on intelligent robots and systems, pp. 5193–5200. IEEE, 2013.
  * [10] J. Gräter, T. Schwarze, and M. Lauer.  Robust scale estimation for monocular visual odometry using structure from motion and vanishing points.  In 2015 IEEE Intelligent Vehicles Symposium (IV), pp. 475–480. IEEE, 2015.
  * [11] C. Ham, S. Lucey, and S. Singh.  Hand waving away scale.  In European conference on computer vision, pp. 279–293. Springer, 2014.
  * [12] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pp. 2961–2969, 2017.
  * [13] S. Hilsenbeck, A. Möller, R. Huitl, G. Schroth, M. Kranz, and E. Steinbach.  Scale-preserving long-term visual odometry for indoor navigation.  In 2012 International Conference on Indoor Positioning and Indoor Navigation (IPIN), pp. 1–10, 2012.
  * [14] T. Konkle and A. Oliva.  A real-world size organization of object responses in occipitotemporal cortex.  Neuron, 74(6):1114–1124, 2012.
  * [15] J. Kronander, F. Banterle, A. Gardner, E. Miandji, and J. Unger.  Photorealistic rendering of mixed reality scenes.  Computer Graphics Forum, 34:643–665, 05 2015. doi: 10 . 1111/cgf . 12591
  * [16] J. Ku, A. D. Pon, and S. L. Waslander.  Monocular 3d object detection leveraging accurate proposals and shape reconstruction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11867–11876, 2019.
  * [17] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, T. Duerig, et al.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  arXiv preprint arXiv:1811.00982, 2018.
  * [18] K.-N. Lianos, J. L. Schonberger, M. Pollefeys, and T. Sattler.  Vso: Visual semantic odometry.  In Proceedings of the European conference on computer vision (ECCV), pp. 234–250, 2018.
  * [19] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pp. 740–755. Springer, 2014.
  * [20] F. T. Liu, K. M. Ting, and Z.-H. Zhou.  Isolation forest.  In 2008 Eighth IEEE International Conference on Data Mining, pp. 413–422. IEEE, 2008.
  * [21] A. Martinelli.  Vision and imu data fusion: Closed-form solutions for attitude, speed, absolute scale, and bias determination.  IEEE Transactions on Robotics, 28(1):44–60, 2011.
  * [22] G. A. Miller.  Wordnet: a lexical database for english.  Communications of the ACM, 38(11):39–41, 1995.
  * [23] G. Nützi, S. Weiss, D. Scaramuzza, and R. Siegwart.  Fusion of imu and vision for absolute scale estimation in monocular slam.  Journal of intelligent & robotic systems, 61(1-4):287–299, 2011\.
  * [24] X. Qin, Z. Zhang, C. Huang, C. Gao, M. Dehghan, and M. Jagersand.  Basnet: Boundary-aware salient object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7479–7489, 2019.
  * [25] D. Rukhovich, D. Mouritzen, R. Kaestner, M. Rufli, and A. Velizhev.  Estimation of absolute scale in monocular slam using synthetic data.  In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 803–812, 2019.
  * [26] M. Runz, M. Buffier, and L. Agapito.  Maskfusion: Real-time recognition, tracking and reconstruction of multiple moving objects.  In 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 10–20. IEEE, 2018.
  * [27] M. R. U. Saputra, A. Markham, and N. Trigoni.  Visual slam and structure from motion in dynamic environments: A survey.  ACM Computing Surveys (CSUR), 51(2):1–36, 2018.
  * [28] M. Savva, A. X. Chang, G. Bernstein, C. D. Manning, and P. Hanrahan.  On being the right scale: Sizing large collections of 3d models.  In SIGGRAPH Asia 2014 Indoor Scene Understanding Where Graphics Meets Vision, pp. 1–11. 2014.
  * [29] M. Savva, A. X. Chang, and P. Hanrahan.  Semantically-enriched 3d models for common-sense knowledge.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 24–31, 2015.
  * [30] D. Scaramuzza, F. Fraundorfer, M. Pollefeys, and R. Siegwart.  Absolute scale in structure from motion from a single vehicle mounted camera by exploiting nonholonomic constraints.  In 2009 IEEE 12th International Conference on Computer Vision, pp. 1413–1419. IEEE, 2009.
  * [31] L. Shao, A. X. Chang, H. Su, M. Savva, and L. Guibas.  Cross-modal attribute transfer for rescaling 3d models.  In 2017 International Conference on 3D Vision (3DV), pp. 640–648. IEEE, 2017.
  * [32] S. Shen.  Accurate multiple view 3d reconstruction using patch-based stereo for large-scale scenes.  IEEE transactions on image processing, 22(5):1901–1914, 2013.
  * [33] S. Song and M. Chandraker.  Robust scale estimation in real-time monocular sfm for autonomous driving.  In 2014 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1566–1573, 2014.
  * [34] S. Song, M. Chandraker, and C. C. Guest.  High accuracy monocular sfm and scale correction for autonomous driving.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(4):730–743, 2016.
  * [35] E. Sucar and J.-B. Hayet.  Probabilistic global scale estimation for monoslam based on generic object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 48–56, 2017.
  * [36] E. Sucar and J.-B. Hayet.  Bayesian scale estimation for monocular slam based on generic object detection for correcting scale drift.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1–7. IEEE, 2018.
  * [37] K. Wang, Y.-A. Lin, B. Weissmann, M. Savva, A. X. Chang, and D. Ritchie.  Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks.  ACM Transactions on Graphics (TOG), 38(4):1–15, 2019.
  * [38] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [39] J. Zhang and S. Singh.  Visual-lidar odometry and mapping: Low-drift, robust, and fast.  In 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 2174–2181. IEEE, 2015.
  * [40] S.-H. Zhang, Z.-P. Zhou, B. Liu, X. Dong, and P. Hall.  What and where: A context-based recommendation system for object insertion.  Computational Visual Media, 6:79–93, 2020.
  * [41] J. Zheng, J. Zhang, J. Li, R. Tang, S. Gao, and Z. Zhou.  Structured3d: A large photo-realistic dataset for structured 3d modeling.  arXiv preprint arXiv:1908.00222, 2019.
  * [42] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba.  Scene parsing through ade20k dataset.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633–641, 2017.
  * [43] B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba.  Semantic understanding of scenes through the ade20k dataset.  International Journal of Computer Vision, 127(3):302–321, 2019\.
  * [44] D. Zhou, Y. Dai, and H. Li.  Reliable scale estimation and correction for monocular visual odometry.  In 2016 IEEE Intelligent Vehicles Symposium (IV), pp. 490–495. IEEE, 2016.
  * [45] D. Zhou, Y. Dai, and H. Li.  Ground-plane-based absolute scale estimation for monocular visual odometry.  IEEE Transactions on Intelligent Transportation Systems, 21(2):791–802, 2019.