Particle-based inference algorithm is a promising method to efficiently generate samples for an intractable target distribution by iteratively updating a set of particles. As a noticeable example, Stein variational gradient descent (SVGD) provides a deterministic and computationally efficient update, but it is known to underestimate the variance in high dimensions, the mechanism of which is poorly understood. In this work we explore a connection between SVGD and MMD-based inference algorithm via Stein's lemma. By comparing the two update rules, we identify the source of bias in SVGD as a combination of high variance and deterministic bias, and empirically demonstrate that the removal of either factors leads to accurate estimation of the variance. In addition, for learning high-dimensional Gaussian target, we analytically derive the converged variance for both algorithms, and confirm that only SVGD suffers from the "curse of dimensionality". The Stein Variational Gradient Descent (SVGD) (Liu and Wang, 2016 ) is a deterministic particle-based inference algorithm that iteratively transports the particles by the functional gradient in the reproducing kernel Hilbert space (RKHS) of KL-divergence, which takes the form of a kernelized Stein's operator. In contrast to the empirical successes (Liu et al., 2017; Haarnoja et al., 2017; Kim et al., 2018) , very few convergence guarantees have been established for SVGD except for the mean-field regime (Liu and Wang, 2018; Lu et al., 2019) . Moreover, it has been observed that the variance estimated by SVGD scales inversely with the dimensionality of the problem. This is a highly undesirable property for two reasons: 1) underestimating the variance leads to failures of explaining the uncertainty of model predictions; 2) modern inference problems are usually high-dimensional. For example, Bayesian neural networks (MacKay, 1992) could be more than millions of dimensions. We study the algorithmic bias of SVGD that leads to the variance underestimation in high dimensions. We construct another kernel-based inference algorithm termed MMDdescent, which closely resembles SVGD but estimate the variance accurately. By comparing their updates, we identify the cause of variance collapse in SVGD as a combination of high variance due to Stein's lemma, and deterministic bias, i.e. the inability to resample particles. We empirically verify that removing either of these two factors, while computationally expensive, leads to accurate variance estimation. Then, under mild assumptions, we derive the equilibrium variance of SVGD and MMD-descent in matching high-dimensional Gaussians, and confirm that variance estimated by SVGD scales inversely with the dimensionality.
Analyze the underlying mechanisms of variance collapse of SVGD in high dimensions.