Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights. Recent work developing this class of methods has explored ever richer parameterizations of the approximate posterior in the hope of improving performance. In contrast, here we share a curious experimental finding that suggests instead restricting the variational distribution to a more compact parameterization. For a variety of deep Bayesian neural networks trained using Gaussian mean-field variational inference, we find that the posterior standard deviations consistently exhibit strong low-rank structure after convergence. This means that by decomposing these variational parameters into a low-rank factorization, we can make our variational approximation more compact without decreasing the models' performance. Furthermore, we find that such factorized parameterizations improve the signal-to-noise ratio of stochastic gradient estimates of the variational lower bound, resulting in faster convergence. Bayesian Neural Networks (MacKay, 1992; Neal, 1993) explicitly represent their parameteruncertainty by forming a posterior distribution over model parameters, instead of relying on a single point estimate for making predictions, as is done in traditional deep learning. Besides offering improved predictive performance over single models, Bayesian neural networks are also more robust to hard examples (Raftery et al., 2005) , have better calibration of predictive uncertainty and thus can be used for out-of-domain detection or other risk-sensitive applications (Ovadia et al., 2019) . Variational inference (Peterson, 1987; Hinton and Van Camp, 1993 ) is a popular class of methods for approximating the posterior distribution p(w|x, y), since the exact Bayes' rule is often intractable to compute for models of practical interest. This class of methods specifies a distribution q θ (w) of given parametric or functional form as the posterior approximation, and optimizes the approximation by solving an optimization problem. In particular, we minimize the negative Evidence Lower Bound (negative ELBO) approximated by samples from the posterior: by differentiating with respect to the variational parameters θ (Salimans et al., 2013; Kingma and Welling, 2013) . In Gaussian Mean Field Variational Inference (GMFVI) (Blei et al., 2017; Blundell et al., 2015) , we choose the variational approximation to be a fully factorized Gaussian distribution: q(w ij ), with q(w ij ) = N (µ ij , σ where W ∈ R m×n is a weight matrix of a single network layer and i and j are the row and column indices in this weight matrix. In practice, we often represent the posterior standard deviation parameters σ ij in the form of a matrix A ∈ R m×n + . With this notation, we have the relationship Σ q = diag(vec(A 2 )) where the elementwise-squared A is vectorized by stacking its columns, and then expanded as a diagonal matrix into R mn×mn + . While Gaussian Mean-Field posteriors are considered to be one of the simplest types of variational approximations, with some known limitations (Giordano et al., 2018) , they scale to comparatively large models and generally provide competitive performance (Ovadia et al., 2019) . However, when compared to deterministic neural networks, GMFVI doubles the number of parameters and is often harder to train due to the increased noise in stochastic gradient estimates. Beyond fully factorized mean-field, recent research in variational inference has explored richer parameterizations of the approximate posterior in order to improve the performance of Bayesian neural networks (see Appendix A and Figure 3 ). For instance, various structures of Gaussian posteriors have been proposed, with per layer block-structured covariances (Louizos and Welling, 2016; Sun et al., 2017; Zhang et al., 2017) , full covariances (Barber and Bishop, 1998) with different parametrizations (Seeger, 2000) , up to more flexible approximate posteriors using normalizing flows (Rezende and Mohamed, 2015) and extensions thereof (Louizos and Welling, 2017 ). In contrast, here we study a simpler, more compactly parameterized mean-field variational posterior which ties variational parameters in the already diagonal covariance matrix. We show that such a posterior approximation can also work well for a variety of models. In particular we find that: • Converged posterior standard deviations under GMFVI consistently display strong low-rank structure. This means that by decomposing these variational parameters into a low-rank factorization, we can make our variational approximation more compact without decreasing our model's performance. • Factorized parameterizations of posterior standard deviations improve the signal-to-noise ratio of stochastic gradient estimates, and thus not only reduce the number of parameters compared to standard GMFVI, but also can lead to faster convergence. In this work we have shown that Bayesian Neural Networks trained with standard Gaussian meanfield variational inference exhibit posterior standard deviation matrices that can be approximated with little information loss by a low-rank decomposition. This suggests that richer parameterizations of the variational posterior may not always be needed, and that compact parameterizations can also work well. We used this insight to propose a simple, yet effective variational posterior parametrization, which speeds up training and reduces the number of variational parameters without degrading predictive performance on three different model types. In future work, we hope to scale up variational inference with compactly parameterized approximate posteriors to much larger models and more complex problems. For mean-field variational inference to work well in that setting several challenges will likely need to be addressed (Osawa et al., 2019) ; improving the signal-to-noise ratio of ELBO gradients using our compact variational parameterizations may provide a piece of the puzzle. 1. Explained variance for the rank k approximation is calculated as γ , where g b is the gradient value for a single parameter. The expectation E and variance V ar of the gradient values g b are calculated over a window of last 10 batches.
Mean field VB uses twice as many parameters; we tie variance parameters in mean field VB without any loss in ELBO, gaining speed and lower variance gradients.