A deep generative model is a powerful method of learning a data distribution, which has achieved tremendous success in numerous scenarios. However, it is nontrivial for a single generative model to faithfully capture the distributions of the complex data such as images with complicate structures. In this paper, we propose a novel approach of cascaded boosting for boosting generative models, where meta-models (i.e., weak learners) are cascaded together to produce a stronger model. Any hidden variable meta-model can be leveraged as long as it can support the likelihood evaluation. We derive a decomposable variational lower bound of the boosted model, which allows each meta-model to be trained separately and greedily. We can further improve the learning power of the generative models by combing our cascaded boosting framework with the multiplicative boosting framework. The past decade has witnessed tremendous success in the field of deep generative models (DGMs) in both unsupervised learning (Goodfellow et al., 2014; Kingma & Welling, 2013; Radford et al., 2015) and semi-supervised learning (Abbasnejad et al., 2017; Kingma et al., 2014; Li et al., 2018) paradigms. DGMs learn the data distribution by combining the scalability of deep learning with the generality of probabilistic reasoning. However, it is not easy for a single parametric model to learn a complex distribution, since the upper limit of a model's ability is determined by its fixed structure. If a model with low capacity was adopted, the model would be likely to have a poor performance. Straightforwardly increasing the model capacity (e.g., including more layers or more neurons) is likely to cause serious challenges, such as vanishing gradient problem (Hochreiter et al., 2001 ) and exploding gradient problem (Grosse, 2017 ). An alternative approach is to integrate multiple weak models to achieve a strong one. The early success was made on mixture models (Dempster et al., 1977; Figueiredo & Jain, 2002; Xu & Jordan, 1996) and product-of-experts (Hinton, 1999; . However, the weak models in such work are typically shallow models with very limited capacity. Recent success has been made on boosting generative models, where a set of meta-models (i.e., weak learners) are combined to construct a stronger model. In particular, Grover & Ermon (2018) propose a method of multiplicative boosting, which takes the geometric average of the meta-model distributions, with each assigned an exponentiated weight. This boosting method improves performance on density estimation and sample generation, compared to a single meta-model. However, the boosted model has an explicit partition function, which requires importance sampling (Rubinstein & Kroese, 2016) for an estimation. In general, sampling from the boosted model is conducted based on Markov chain Monte Carlo (MCMC) method (Hastings, 1970) . As a result, it requires a high time complexity of likelihood evaluation and sample generation. Rosset & Segal (2003) propose another method of additive boosting, which takes the weighted arithmetic mean of meta-models' distributions. This method can sample fast, but the improvement of performance on density estimation is not comparable to the multiplicative boosting, since additive boosting requires that the expected log-likelihood and likelihood of the current meta-model are better-or-equal than those of the previous boosted model (Grover & Ermon, 2018) , which is difficult to satisfy. In summary, it is nontrivial for both of the previous boosting methods to balance well between improving the learning power and keeping the efficiency of sampling and density estimation. To address the aforementioned issues, we propose a novel boosting framework, called cascaded boosting, where meta-models are connected in cascade. The framework is inspired by the greedy layer-wise training algorithm of DBNs (Deep Belief Networks) (Bengio et al., 2007; Hinton et al., 2006) , where an ensemble of RBMs (Restricted Boltzmann Machines) (Smolensky, 1986) are converted to a stronger model. We propose a decomposable variational lower bound, which reveals the principle behind the greedy layer-wise training algorithm. The decomposition allows us to incorporate any hidden variable meta-model, as long as it supports likelihood evaluation, and train these meta-models separately and greedily, yielding a deep boosted model. Finally, We demonstrate that our boosting framework can be integrated with the multiplicative boosting framework (Grover & Ermon, 2018) , yielding a hybrid boosting with an improved learning power of generative models. To summary, we make the following contributions: • We propose a boosting framework to boost generative models, where meta-models are cascaded together to produce a stronger model. • We give a decomposable variational lower bound of the boosted model, which reveals the principle behind the greedy layer-wise training algorithm. • We finally demonstrate that our boosting framework can be extended to a hybrid model by integrating it with the multiplicative boosting models, which further improves the learning power of generative models. We propose a framework for boosting generative models by cascading meta-models. Any hidden variable meta-model can be incorporated, as long as it supports likelihood evaluation. The decomposable lower bound allows us to train meta-models separately and greedily. Our cascaded boosting can be integrated with the multiplicative boosting. In our experiments, we first validate that the non-decreasing property of the decomposable variational lower bound (Equation 5) holds in practice, and next further promote the performance of some advanced models, which represent state-ofthe-art methods. Then, we show that our cascaded boosting has better performance of improving models' learning power, compared with naively increasing model capacity. Finally, we compare different generative boosting methods, validating the ability of the hybrid boosting in further improving learning power of generative models. A PROOF OF THEOREM 1 Proof. Using q(h 1 , · · · , h k−1 |x) as the approximate posterior, we have a variational lower bound Thus, the lower bound is equal to: Thus, Take the expection with respect to dataset D, we have B PROOF OF THEOREM 3 AND THEOREM 4 where n is the number of meta-models. Since we can omit the subscript, thereby writing q i as q. For any j ∈ [k + 1, n] ∩ Z and any m k+1 , m k+2 , · · · , m j , given i (k + 1 ≤ i ≤ j), we have Let q(h k , · · · , h j−1 |h k−1 ) be the approximate posterior of p j (h k−1 , · · · , h j−1 ), according to Theorem 1, we have we have
Propose an approach for boosting generative models by cascading hidden variable models This paper proposed a novel approach of cascaded boosting for boosting generative models which allows each each meta-model to be trained separately and greedily.