Variational Bayesian neural networks (BNN) perform variational inference over weights, but it is difficult to specify meaningful priors and approximating posteriors in a high-dimensional weight space. We introduce functional variational Bayesian neural networks (fBNNs), which maximize an Evidence Lower BOund (ELBO) defined directly on stochastic processes, i.e. distributions over functions. We prove that the KL divergence between stochastic processes is equal to the supremum of marginal KL divergences over all finite sets of inputs. Based on this, we introduce a practical training objective which approximates the functional ELBO using finite measurement sets and the spectral Stein gradient estimator. With fBNNs, we can specify priors which entail rich structure, including Gaussian processes and implicit stochastic processes. Empirically, we find that fBNNs extrapolate well using various structured priors, provide reliable uncertainty estimates, and can scale to large datasets.
We perform functional variational inference on the stochastic processes defined by Bayesian neural networks. Fitting of variational Bayesian Neural Network approximations in functional form and considering matching to a stochastic process prior implicitly via samples. Presents a novel ELBO objective for training BNNs which allows for more meaningful priors to be encoded in the model rather than the less informative weight priors features in the literature. Presents a new variational inference algorithm for Bayesian neural network models where the prior is specified functionally rather than via a prior over weights. 