Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We train \(q\) with variational inference, using an invertible \(h\) to enable efficient estimation of the variational lower bound on the posterior \(p(\mathbf{\theta}|\mathcal{D})\) via sampling.

The evidence to be evaluated is:Broadly speaking, existing methods either 1) use Markov chain Monte Carlo (Welling & Teh, 2011; Neal, 1996), or 2) directly learn an approximate posterior distribution using (stochastic) variational inference (Graves, 2011; Gal & Ghahramani, 2016; Salimans et al., 2015; Blundell et al., 2015), expectation propagation (Hernandez-Lobato & Adams, 2015; Soudry et al., 2014), or \(\alpha\)-divergences (Li & Gal, 2017). We focus here on the most popular approach: variational inference. Notable recent work in this area includes Gal & Ghahramani (2016), who interprets the popular dropout (Srivastava et al., 2014) algorithm as a variational inference method ("MC dropout"). This has the advantages of being simple to implement and allowing cheap samples from \(q(\mathbf{\theta})\).

Because \(\mathbf{z}\) follows a complicated distribution, this forms a highly flexible approximate posterior: \(q(\mathbf{\theta})=\int q(\mathbf{\theta}|\mathbf{z})q(\mathbf{z})\mathrm{d}\mathbf{z}\). However, this approach also requires them to introduce an auxiliary inference network to approximate \(q(\mathbf{z}|\mathbf{\theta})\) in order to estimate the entropy term of the variational lower bound, resulting in lower bound _on the variational lower bound_.

Finally, the variational autoencoder (VAE) (Jimenez Rezende et al., 2014; Kingma & Welling, 2013) family of generative models is likely the best known application of variational inference in DNNs, but note that the VAE is _not_ a Bayesian DNN in our sense. VAEs approximate the posterior over latent variables, given a datapoint; Bayesian DNNs approximate the posterior over model parameters, given a dataset.

DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) (Kingma and Welling, 2013; Jimenez Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014; Goodfellow, 2017). We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016).

We now describe how variational inference is applied to Bayesian deep nets (Section 3.1), and how we compose the methods described in Sections 2.2 and 2.3 to produce Bayesian hypernets (Section 3.2). In variational inference, the goal is to maximize a lower bound on the marginal log-likelihood of the data, \(\log p(\mathcal{D})\) under some statistical model.

Let \(\mathbf{\theta}\in\mathbb{R}^{D}\) be parameters given the Bayesian treatment as random variables, \(\mathcal{D}\) a training set of observed data, and \(q(\mathbf{\theta})\) a learned approximation to the true posterior \(p(\mathbf{\theta}|\mathcal{D})\).

\[\log p(\mathcal{D}) =\text{KL}(q(\mathbf{\theta})\|p(\mathbf{\theta}|\mathcal{D}))+\mathbb{E} _{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{\theta})-\log q(\mathbf{\theta})] \tag{1}\] \[\geq\mathbb{E}_{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{ \theta})-\log q(\mathbf{\theta})]\,. \tag{2}\]

Computing the expectation in (2) is generally intractable for deep nets, but can be estimated by Monte Carlo sampling.

Here we approximate the posterior distribution \(p(\mathbf{\theta}|\mathcal{D})\) using the approximate posterior \(q(\mathbf{\theta})\). We further use \(S\) Monte Carlo samples to approximate the integral.

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\).

\[q(\mathbf{\theta})= \int q(\mathbf{\theta};h(\mathbf{\epsilon}))q(\mathbf{\epsilon})d\mathbf{ \epsilon}\,. \tag{5}\] To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\). As discussed in Section 2.3, a number of techniques have been developed for efficiently training such invertible DDGNs. In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of \(h\), as in generative modeling applications, e.g., Dinh et al. (2016)); and this also means that we can use a lower-dimensional \(\mathbf{\epsilon}\) to generate samples along a submanifold of the entire parameter space, as detailed below.

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations.

In this paper, we employ weight normalization in the primary network (7), treating (only) the scaling factors \(\mathbf{g}\) as random variables. We choose an isotropic Gaussian prior for \(\mathbf{g}\): \(p(\mathbf{g})=\mathcal{N}(\mathbf{g};\mathbf{0},\lambda\mathbf{I})\), which results in an \(L_{2}\) weight-decay penalty on \(\mathbf{g}\), or, equivalently, \(\mathbf{w}=\mathbf{g}\frac{\mathbf{v}}{||\mathbf{v}||_{2}}\). Our objective and lower bound are then: \[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim: "We train q with variational inference, using an invertible h to enable efficient estimation of the variational lower bound on the posterior p(θ|D) via sampling."

Analysis:
The evidence strongly supports this claim through multiple components:
1. The text explicitly shows how variational inference is used to train q(θ)
2. Equations (6) and (10) demonstrate how the invertible h enables efficient computation of q(θ) using the change of variables formula
3. The mathematical derivation shows how this allows estimation of the variational lower bound via sampling
4. The evidence provides detailed technical explanation of how invertible DDGNs (h) are used for efficient sampling and training

The claim is precise, technically accurate, and directly supported by the mathematical formulations and explanations provided. It doesn't overextend beyond what the evidence demonstrates.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We train \(q\) with variational inference, using an invertible \(h\) to enable efficient estimation of the variational lower bound on the posterior \(p(\mathbf{\theta}|\mathcal{D})\) via sampling.

The review comment to be evaluated is:

This paper presents Bayesian Hypernetworks; variational Bayesian neural networks where the variational posterior over the weights is governed by a hyper network that implements a normalizing flow (NF) such as RealNVP and IAF. As directly outputting the weight matrix with a hyper network is computationally expensive the authors instead propose to utilize weight normalisation on the weights and then use the hyper network to output scalar scaling variables for each hidden unit, similarly to what was done at [1]. The main difference with this prior work is that [1] consider these NF scaling variables as auxiliary random variables to a mean field Gaussian distribution over the weights whereas this paper attempts to posit a distribution directly on the weights via the NF. This avoids the nested variational approximation and auxiliary models of [1], which can potentially yield a tighter bound. The proposed method is evaluated on extensive experiments.

This paper seems like a plausible idea with extensive experiments but the similarity with [1] make it an incremental contribution and, furthermore, it seems that it has a technical issue with what is explained at Section 3.3. More specifically, if you generate the parameters \theta according to Eq. 7 and posit a prior over \theta then you will have a problematic variational bound as there will be a KL divergence, KL(q(\theta) || p(\theta)), with distributions of different support (since q(\theta) is defined only along the directions spanned by u), which is infinite. For the KL to be valid you will need to posit a prior distribution over `g`, p(g), and then consider KL(q(g) || p(g)), with q(g) being given by the NF. From the experiment paragraph at page 5 though I deduct that you instead employ “an isotropic standard normal prior over the weights”, i.e. \theta, thus I believe that you indeed have a problematic bound. How do you actually compute logq(\theta) when you employ the parametrisation discussed at 3.3? Did you use that parametrisation in every experiment?

Other than that, I believe that it would be interesting to experiment with a `full` hyper network, i.e. generating directly the entire parameter vector \theta, e.g. at the toy regression experiment where the dimensionality is small. This would then better illustrate the tradeoffs you make when you reduce the flexibility of the hyper-network to just outputting the row scaling variables and the effect this has at the posterior approximation.

Typos:
(1) Page 3, 3.1.1 log(\theta) -> logp(\theta).
(2) Eq. 6, it needs to be |det \frac{\partial h(\epsilon)}{\partial \epsilon}|^{-1} or |det \frac{\partial h^{-1}(\theta)}{\partial \theta}| for a valid change of variables formula.

[1] Louizos & Welling, Multiplicative Normalizing Flows for Variational Bayesian Neural Networks.

The evidence to be evaluated is:Broadly speaking, existing methods either 1) use Markov chain Monte Carlo (Welling & Teh, 2011; Neal, 1996), or 2) directly learn an approximate posterior distribution using (stochastic) variational inference (Graves, 2011; Gal & Ghahramani, 2016; Salimans et al., 2015; Blundell et al., 2015), expectation propagation (Hernandez-Lobato & Adams, 2015; Soudry et al., 2014), or \(\alpha\)-divergences (Li & Gal, 2017). We focus here on the most popular approach: variational inference. Notable recent work in this area includes Gal & Ghahramani (2016), who interprets the popular dropout (Srivastava et al., 2014) algorithm as a variational inference method ("MC dropout"). This has the advantages of being simple to implement and allowing cheap samples from \(q(\mathbf{\theta})\).

Because \(\mathbf{z}\) follows a complicated distribution, this forms a highly flexible approximate posterior: \(q(\mathbf{\theta})=\int q(\mathbf{\theta}|\mathbf{z})q(\mathbf{z})\mathrm{d}\mathbf{z}\). However, this approach also requires them to introduce an auxiliary inference network to approximate \(q(\mathbf{z}|\mathbf{\theta})\) in order to estimate the entropy term of the variational lower bound, resulting in lower bound _on the variational lower bound_.

Finally, the variational autoencoder (VAE) (Jimenez Rezende et al., 2014; Kingma & Welling, 2013) family of generative models is likely the best known application of variational inference in DNNs, but note that the VAE is _not_ a Bayesian DNN in our sense. VAEs approximate the posterior over latent variables, given a datapoint; Bayesian DNNs approximate the posterior over model parameters, given a dataset.

DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) (Kingma and Welling, 2013; Jimenez Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014; Goodfellow, 2017). We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016).

We now describe how variational inference is applied to Bayesian deep nets (Section 3.1), and how we compose the methods described in Sections 2.2 and 2.3 to produce Bayesian hypernets (Section 3.2). In variational inference, the goal is to maximize a lower bound on the marginal log-likelihood of the data, \(\log p(\mathcal{D})\) under some statistical model.

Let \(\mathbf{\theta}\in\mathbb{R}^{D}\) be parameters given the Bayesian treatment as random variables, \(\mathcal{D}\) a training set of observed data, and \(q(\mathbf{\theta})\) a learned approximation to the true posterior \(p(\mathbf{\theta}|\mathcal{D})\).

\[\log p(\mathcal{D}) =\text{KL}(q(\mathbf{\theta})\|p(\mathbf{\theta}|\mathcal{D}))+\mathbb{E} _{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{\theta})-\log q(\mathbf{\theta})] \tag{1}\] \[\geq\mathbb{E}_{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{ \theta})-\log q(\mathbf{\theta})]\,. \tag{2}\]

Computing the expectation in (2) is generally intractable for deep nets, but can be estimated by Monte Carlo sampling.

Here we approximate the posterior distribution \(p(\mathbf{\theta}|\mathcal{D})\) using the approximate posterior \(q(\mathbf{\theta})\). We further use \(S\) Monte Carlo samples to approximate the integral.

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\).

\[q(\mathbf{\theta})= \int q(\mathbf{\theta};h(\mathbf{\epsilon}))q(\mathbf{\epsilon})d\mathbf{ \epsilon}\,. \tag{5}\] To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\). As discussed in Section 2.3, a number of techniques have been developed for efficiently training such invertible DDGNs. In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of \(h\), as in generative modeling applications, e.g., Dinh et al. (2016)); and this also means that we can use a lower-dimensional \(\mathbf{\epsilon}\) to generate samples along a submanifold of the entire parameter space, as detailed below.

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations.

In this paper, we employ weight normalization in the primary network (7), treating (only) the scaling factors \(\mathbf{g}\) as random variables. We choose an isotropic Gaussian prior for \(\mathbf{g}\): \(p(\mathbf{g})=\mathcal{N}(\mathbf{g};\mathbf{0},\lambda\mathbf{I})\), which results in an \(L_{2}\) weight-decay penalty on \(\mathbf{g}\), or, equivalently, \(\mathbf{w}=\mathbf{g}\frac{\mathbf{v}}{||\mathbf{v}||_{2}}\). Our objective and lower bound are then: \[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "We train q with variational inference, using an invertible h to enable efficient estimation of the variational lower bound on the posterior p(θ|D) via sampling."

Key analysis:
1. The evidence clearly shows the use of variational inference and an invertible hypernetwork h for training q(θ).
2. The equations (particularly 6 and 10) demonstrate how the invertible h enables efficient computation of q(θ) through the change of variables formula.
3. However, the reviewer points out a significant technical issue with the KL divergence computation when using the parametrization discussed in Section 3.3, suggesting the bound may be problematic.
4. The reviewer notes that for a valid KL divergence, they need to consider KL(q(g) || p(g)) rather than KL(q(θ) || p(θ)).

Given these points, I would rate this claim as moderately overstated. While the basic mechanism is correct, the claim doesn't acknowledge the technical limitations regarding the support of the distributions in the KL divergence computation that the reviewer highlighted.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We train \(q\) with variational inference, using an invertible \(h\) to enable efficient estimation of the variational lower bound on the posterior \(p(\mathbf{\theta}|\mathcal{D})\) via sampling.

The review comment to be evaluated is:

This paper proposes Bayesian hypernetworks to carry out Bayesian learning of deep networks. The idea is to construct a generative model capable of approximating the posterior distribution over the parameters of deep networks. I think that the paper is well written and easy to follow.

I like the idea of constructing general approximation strategies for complex posterior distribution and the proposed approach inherits all the scalability properties of modern deep learning techniques. In this respect, I think that the paper tackles a timely topic and is interesting to read.

It is not entirely clear to me why the Authors name their proposal Bayesian hypernetworks. This seems to suggest that also the hypernetwork is infered using Bayesian inference, but if I understand correctly this is not the case.

I have some comments on novelty and realization of the experiments. In the positioning of the work in the literature, the Authors point out that hypernetworks have been proposed before, so it is not clear what is the actual novelty in the proposal. Is it the use of Real NVPs and IAFs as hypernetworks? These methods have been already proposed and extensively studied in the literature, and even if they have been adapted to be hypernetworks here, I believe that the novelty is fairly limited.

The experimental part is interesting as it explores a number of learning scenarios. However, I think that it would have been useful to add comparisons with standard variational inference (e.g., Graves, 2011) for deep networks to substantiate the claims that this approach underestimates uncertainty. I believe that this would strengthen the comparative evaluation.

I think the paper would have made a stronger case by including other approaches to approximate posteriors using generative models. For example, the variational Gaussian process paper sounds like an ideal method to include here.

[1] D. Tran, R. Ranganath, and D. M. Blei. Variational Gaussian process. arXiv preprint arXiv:1511.06499, 2015.

The evidence to be evaluated is:Broadly speaking, existing methods either 1) use Markov chain Monte Carlo (Welling & Teh, 2011; Neal, 1996), or 2) directly learn an approximate posterior distribution using (stochastic) variational inference (Graves, 2011; Gal & Ghahramani, 2016; Salimans et al., 2015; Blundell et al., 2015), expectation propagation (Hernandez-Lobato & Adams, 2015; Soudry et al., 2014), or \(\alpha\)-divergences (Li & Gal, 2017). We focus here on the most popular approach: variational inference. Notable recent work in this area includes Gal & Ghahramani (2016), who interprets the popular dropout (Srivastava et al., 2014) algorithm as a variational inference method ("MC dropout"). This has the advantages of being simple to implement and allowing cheap samples from \(q(\mathbf{\theta})\).

Because \(\mathbf{z}\) follows a complicated distribution, this forms a highly flexible approximate posterior: \(q(\mathbf{\theta})=\int q(\mathbf{\theta}|\mathbf{z})q(\mathbf{z})\mathrm{d}\mathbf{z}\). However, this approach also requires them to introduce an auxiliary inference network to approximate \(q(\mathbf{z}|\mathbf{\theta})\) in order to estimate the entropy term of the variational lower bound, resulting in lower bound _on the variational lower bound_.

Finally, the variational autoencoder (VAE) (Jimenez Rezende et al., 2014; Kingma & Welling, 2013) family of generative models is likely the best known application of variational inference in DNNs, but note that the VAE is _not_ a Bayesian DNN in our sense. VAEs approximate the posterior over latent variables, given a datapoint; Bayesian DNNs approximate the posterior over model parameters, given a dataset.

DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) (Kingma and Welling, 2013; Jimenez Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014; Goodfellow, 2017). We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016).

We now describe how variational inference is applied to Bayesian deep nets (Section 3.1), and how we compose the methods described in Sections 2.2 and 2.3 to produce Bayesian hypernets (Section 3.2). In variational inference, the goal is to maximize a lower bound on the marginal log-likelihood of the data, \(\log p(\mathcal{D})\) under some statistical model.

Let \(\mathbf{\theta}\in\mathbb{R}^{D}\) be parameters given the Bayesian treatment as random variables, \(\mathcal{D}\) a training set of observed data, and \(q(\mathbf{\theta})\) a learned approximation to the true posterior \(p(\mathbf{\theta}|\mathcal{D})\).

\[\log p(\mathcal{D}) =\text{KL}(q(\mathbf{\theta})\|p(\mathbf{\theta}|\mathcal{D}))+\mathbb{E} _{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{\theta})-\log q(\mathbf{\theta})] \tag{1}\] \[\geq\mathbb{E}_{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{ \theta})-\log q(\mathbf{\theta})]\,. \tag{2}\]

Computing the expectation in (2) is generally intractable for deep nets, but can be estimated by Monte Carlo sampling.

Here we approximate the posterior distribution \(p(\mathbf{\theta}|\mathcal{D})\) using the approximate posterior \(q(\mathbf{\theta})\). We further use \(S\) Monte Carlo samples to approximate the integral.

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\).

\[q(\mathbf{\theta})= \int q(\mathbf{\theta};h(\mathbf{\epsilon}))q(\mathbf{\epsilon})d\mathbf{ \epsilon}\,. \tag{5}\] To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\). As discussed in Section 2.3, a number of techniques have been developed for efficiently training such invertible DDGNs. In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of \(h\), as in generative modeling applications, e.g., Dinh et al. (2016)); and this also means that we can use a lower-dimensional \(\mathbf{\epsilon}\) to generate samples along a submanifold of the entire parameter space, as detailed below.

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations.

In this paper, we employ weight normalization in the primary network (7), treating (only) the scaling factors \(\mathbf{g}\) as random variables. We choose an isotropic Gaussian prior for \(\mathbf{g}\): \(p(\mathbf{g})=\mathcal{N}(\mathbf{g};\mathbf{0},\lambda\mathbf{I})\), which results in an \(L_{2}\) weight-decay penalty on \(\mathbf{g}\), or, equivalently, \(\mathbf{w}=\mathbf{g}\frac{\mathbf{v}}{||\mathbf{v}||_{2}}\). Our objective and lower bound are then: \[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim "We train q with variational inference, using an invertible h to enable efficient estimation of the variational lower bound on the posterior p(θ|D) via sampling."

1. Summary of evaluation:
The claim is well-stated and accurately reflects the evidence provided. The paper thoroughly demonstrates:
- The use of variational inference for training q (equations 1-2)
- The employment of an invertible hypernetwork h (equations 5-6)
- How this enables efficient estimation through sampling (equation 10)
The mathematical formulation and derivations are clearly presented, showing how the invertible transform allows direct computation of q(θ) using the change of variables formula. The claim's technical aspects are precisely grounded in the evidence without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We train \(q\) with variational inference, using an invertible \(h\) to enable efficient estimation of the variational lower bound on the posterior \(p(\mathbf{\theta}|\mathcal{D})\) via sampling.

The review comment to be evaluated is:

* Edit: I increased my rating to 6. The authors fixed the first error I pointed out below. Regarding the second point: I still think it is possible to take a limit of sigma -> 0 in MNF, which makes the methods very similar.

The authors propose a new method of defining approximate posteriors for use in Bayesian neural networks. The idea of using hypernetworks for Bayesian inference is compelling, and the authors show some promising first results. I see two issues, and would be willing to increase my rating if these were sufficiently addressed.

- The paper says it uses an "isotropic standard normal prior on the weights of the network". However, the stochastic part of the generated weights (i.e. the scales) is of a lower dimension than the weights. It seems to me this means that the KL divergence between prior and posterior is undefined, or infinite, as the posterior is only defined on a sub-manifold. What exactly is the loss term that is added to the training objective? And how is this justified?

- The instantiation of Bayesian hypernetworks that is used in experiments seems to be a special case of the method of multiplicative normalizing flows as proposed by Louizos and Welling and discussed in this paper. If the variances / sigmas are zero in the latter method, their approximation seems functionally equivalent to Bayesian hypernetworks (though with different parameterization). Is my understanding correct? If so, the novelty of the proposed method is limited.

The evidence to be evaluated is:Broadly speaking, existing methods either 1) use Markov chain Monte Carlo (Welling & Teh, 2011; Neal, 1996), or 2) directly learn an approximate posterior distribution using (stochastic) variational inference (Graves, 2011; Gal & Ghahramani, 2016; Salimans et al., 2015; Blundell et al., 2015), expectation propagation (Hernandez-Lobato & Adams, 2015; Soudry et al., 2014), or \(\alpha\)-divergences (Li & Gal, 2017). We focus here on the most popular approach: variational inference. Notable recent work in this area includes Gal & Ghahramani (2016), who interprets the popular dropout (Srivastava et al., 2014) algorithm as a variational inference method ("MC dropout"). This has the advantages of being simple to implement and allowing cheap samples from \(q(\mathbf{\theta})\).

Because \(\mathbf{z}\) follows a complicated distribution, this forms a highly flexible approximate posterior: \(q(\mathbf{\theta})=\int q(\mathbf{\theta}|\mathbf{z})q(\mathbf{z})\mathrm{d}\mathbf{z}\). However, this approach also requires them to introduce an auxiliary inference network to approximate \(q(\mathbf{z}|\mathbf{\theta})\) in order to estimate the entropy term of the variational lower bound, resulting in lower bound _on the variational lower bound_.

Finally, the variational autoencoder (VAE) (Jimenez Rezende et al., 2014; Kingma & Welling, 2013) family of generative models is likely the best known application of variational inference in DNNs, but note that the VAE is _not_ a Bayesian DNN in our sense. VAEs approximate the posterior over latent variables, given a datapoint; Bayesian DNNs approximate the posterior over model parameters, given a dataset.

DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) (Kingma and Welling, 2013; Jimenez Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014; Goodfellow, 2017). We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016).

We now describe how variational inference is applied to Bayesian deep nets (Section 3.1), and how we compose the methods described in Sections 2.2 and 2.3 to produce Bayesian hypernets (Section 3.2). In variational inference, the goal is to maximize a lower bound on the marginal log-likelihood of the data, \(\log p(\mathcal{D})\) under some statistical model.

Let \(\mathbf{\theta}\in\mathbb{R}^{D}\) be parameters given the Bayesian treatment as random variables, \(\mathcal{D}\) a training set of observed data, and \(q(\mathbf{\theta})\) a learned approximation to the true posterior \(p(\mathbf{\theta}|\mathcal{D})\).

\[\log p(\mathcal{D}) =\text{KL}(q(\mathbf{\theta})\|p(\mathbf{\theta}|\mathcal{D}))+\mathbb{E} _{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{\theta})-\log q(\mathbf{\theta})] \tag{1}\] \[\geq\mathbb{E}_{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{ \theta})-\log q(\mathbf{\theta})]\,. \tag{2}\]

Computing the expectation in (2) is generally intractable for deep nets, but can be estimated by Monte Carlo sampling.

Here we approximate the posterior distribution \(p(\mathbf{\theta}|\mathcal{D})\) using the approximate posterior \(q(\mathbf{\theta})\). We further use \(S\) Monte Carlo samples to approximate the integral.

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\).

\[q(\mathbf{\theta})= \int q(\mathbf{\theta};h(\mathbf{\epsilon}))q(\mathbf{\epsilon})d\mathbf{ \epsilon}\,. \tag{5}\] To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\). As discussed in Section 2.3, a number of techniques have been developed for efficiently training such invertible DDGNs. In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of \(h\), as in generative modeling applications, e.g., Dinh et al. (2016)); and this also means that we can use a lower-dimensional \(\mathbf{\epsilon}\) to generate samples along a submanifold of the entire parameter space, as detailed below.

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations.

In this paper, we employ weight normalization in the primary network (7), treating (only) the scaling factors \(\mathbf{g}\) as random variables. We choose an isotropic Gaussian prior for \(\mathbf{g}\): \(p(\mathbf{g})=\mathcal{N}(\mathbf{g};\mathbf{0},\lambda\mathbf{I})\), which results in an \(L_{2}\) weight-decay penalty on \(\mathbf{g}\), or, equivalently, \(\mathbf{w}=\mathbf{g}\frac{\mathbf{v}}{||\mathbf{v}||_{2}}\). Our objective and lower bound are then: \[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
1. Summary evaluation:
The claim about training q with variational inference using an invertible h is well-supported by the evidence. The paper provides detailed mathematical derivations showing how the invertible hypernetwork h enables efficient estimation of the variational lower bound (equations 8-10). The evidence extensively covers the methodology using change of variables formula (equation 6) and explains how invertible DDGNs like RealNVP and IAF are employed. While the reviewer raises concerns about dimensionality and similarity to other methods, these don't directly contradict the specific technical claim being evaluated about the training procedure.

<score>1</score>

Justification:
1. The claim makes a precise technical statement about the training methodology that is thoroughly documented in the evidence through:
- Explicit mathematical formulation of the variational inference approach (equations 1-2)
- Detailed explanation of how invertible h enables efficient computation through change of variables (equation 6)
- Specific implementation details using established methods (RealNVP and IAF)

2. The reviewer's concerns about:
- Dimensionality issues with KL divergence
- Similarity to multiplicative normalizing flows
While valid broader concerns, don't impact the accuracy of this specific claim about the training procedure.

3. The strength of evidence is high:
- Complete mathematical derivation provided
- Clear connection to existing literature and methods
- Specific technical details about implementation

The claim accurately reflects what is demonstrated in the evidence without overstatement or exaggeration.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The keyinsight for building such a model is the use of an invertible hypernet, which enables Monte Carlo estimation of the entropy term \(-\log q(\mathbf{\theta})\) in the variational inference training objective.

The evidence to be evaluated is:Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) (Goodfellow et al., 2016) as a generative model of the primary net parameters.

We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016). Training these models uses the change of variables formula, which involves computing the log-determinant of the inverse Jacobian of the generator network.

Let \(\mathbf{\theta}\in\mathbb{R}^{D}\) be parameters given the Bayesian treatment as random variables, \(\mathcal{D}\) a training set of observed data, and \(q(\mathbf{\theta})\) a learned approximation to the true posterior \(p(\mathbf{\theta}|\mathcal{D})\).

\[\log p(\mathcal{D}) =\text{KL}(q(\mathbf{\theta})\|p(\mathbf{\theta}|\mathcal{D}))+\mathbb{E} _{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{\theta})-\log q(\mathbf{\theta})] \tag{1}\] \[\geq\mathbb{E}_{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{ \theta})-\log q(\mathbf{\theta})]\,. \tag{2}\]

Computing the expectation in (2) is generally intractable for deep nets, but can be estimated by Monte Carlo sampling.

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\).

However, while DDGNs are convenient to sample from, computing the entropy term (\(\mathbb{E}_{q}[-\log q(\mathbf{\theta})]\)) of the ELBO additionally requires evaluating the likelihood of generated samples, and most popular DDGNs (such as VAEs and GANs) do not provide a convenient way of doing so.5

To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\).

In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016).

We only output the scaling factors \(g\) from the hypernet, and learn a maximum likelihood estimate of \(\mathbf{v}\).7 This allows us to overcome the computational limitations of naively-parametrized BHNs noted by Louizos & Welling (2017), since computation now scales linearly, instead of quadratically, in the number of primary net units. Using this parametrization restricts the family of approximate posteriors, but still allows for a high degree of multimodality and dependence between the parameters.

For the hypernet architecture, we try both RealNVP (Dinh et al., 2016) and IAF(Kingma et al., 2016) with MADE(Germain et al., 2015), with 1-layer ReLU-MLP coupling functions with 200 hidden units (each).

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations.

In this paper, we employ weight normalization in the primary network (7), treating (only) the scaling factors \(\mathbf{g}\) as random variables. We choose an isotropic Gaussian prior for \(\mathbf{g}\): \(p(\mathbf{g})=\mathcal{N}(\mathbf{g};\mathbf{0},\lambda\mathbf{I})\), which results in an \(L_{2}\) weight-decay penalty on \(\mathbf{g}\), or, equivalently, \(\mathbf{w}=\mathbf{g}\frac{\mathbf{v}}{||\mathbf{v}||_{2}}\). Our objective and lower bound are then: \[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim based on the provided evidence:

The claim states that the key insight is using an invertible hypernet to enable Monte Carlo estimation of the entropy term. The evidence strongly supports this by:

1. Demonstrating mathematically (via equation 6) how the invertible nature allows computing q(θ) using the change of variables formula
2. Explicitly stating that non-invertible DDGNs (like VAEs and GANs) cannot conveniently compute the entropy term
3. Showing in equations 8-10 how this enables Monte Carlo estimation of the full variational objective

The claim is well-stated (score of 1) as it accurately captures a key technical innovation that is thoroughly substantiated in the mathematical derivations and explanations provided.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The keyinsight for building such a model is the use of an invertible hypernet, which enables Monte Carlo estimation of the entropy term \(-\log q(\mathbf{\theta})\) in the variational inference training objective.

The review comment to be evaluated is:

This paper presents Bayesian Hypernetworks; variational Bayesian neural networks where the variational posterior over the weights is governed by a hyper network that implements a normalizing flow (NF) such as RealNVP and IAF. As directly outputting the weight matrix with a hyper network is computationally expensive the authors instead propose to utilize weight normalisation on the weights and then use the hyper network to output scalar scaling variables for each hidden unit, similarly to what was done at [1]. The main difference with this prior work is that [1] consider these NF scaling variables as auxiliary random variables to a mean field Gaussian distribution over the weights whereas this paper attempts to posit a distribution directly on the weights via the NF. This avoids the nested variational approximation and auxiliary models of [1], which can potentially yield a tighter bound. The proposed method is evaluated on extensive experiments.

This paper seems like a plausible idea with extensive experiments but the similarity with [1] make it an incremental contribution and, furthermore, it seems that it has a technical issue with what is explained at Section 3.3. More specifically, if you generate the parameters \theta according to Eq. 7 and posit a prior over \theta then you will have a problematic variational bound as there will be a KL divergence, KL(q(\theta) || p(\theta)), with distributions of different support (since q(\theta) is defined only along the directions spanned by u), which is infinite. For the KL to be valid you will need to posit a prior distribution over `g`, p(g), and then consider KL(q(g) || p(g)), with q(g) being given by the NF. From the experiment paragraph at page 5 though I deduct that you instead employ “an isotropic standard normal prior over the weights”, i.e. \theta, thus I believe that you indeed have a problematic bound. How do you actually compute logq(\theta) when you employ the parametrisation discussed at 3.3? Did you use that parametrisation in every experiment?

Other than that, I believe that it would be interesting to experiment with a `full` hyper network, i.e. generating directly the entire parameter vector \theta, e.g. at the toy regression experiment where the dimensionality is small. This would then better illustrate the tradeoffs you make when you reduce the flexibility of the hyper-network to just outputting the row scaling variables and the effect this has at the posterior approximation.

Typos:
(1) Page 3, 3.1.1 log(\theta) -> logp(\theta).
(2) Eq. 6, it needs to be |det \frac{\partial h(\epsilon)}{\partial \epsilon}|^{-1} or |det \frac{\partial h^{-1}(\theta)}{\partial \theta}| for a valid change of variables formula.

[1] Louizos & Welling, Multiplicative Normalizing Flows for Variational Bayesian Neural Networks.

The evidence to be evaluated is:Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) (Goodfellow et al., 2016) as a generative model of the primary net parameters.

We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016). Training these models uses the change of variables formula, which involves computing the log-determinant of the inverse Jacobian of the generator network.

Let \(\mathbf{\theta}\in\mathbb{R}^{D}\) be parameters given the Bayesian treatment as random variables, \(\mathcal{D}\) a training set of observed data, and \(q(\mathbf{\theta})\) a learned approximation to the true posterior \(p(\mathbf{\theta}|\mathcal{D})\).

\[\log p(\mathcal{D}) =\text{KL}(q(\mathbf{\theta})\|p(\mathbf{\theta}|\mathcal{D}))+\mathbb{E} _{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{\theta})-\log q(\mathbf{\theta})] \tag{1}\] \[\geq\mathbb{E}_{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{ \theta})-\log q(\mathbf{\theta})]\,. \tag{2}\]

Computing the expectation in (2) is generally intractable for deep nets, but can be estimated by Monte Carlo sampling.

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\).

However, while DDGNs are convenient to sample from, computing the entropy term (\(\mathbb{E}_{q}[-\log q(\mathbf{\theta})]\)) of the ELBO additionally requires evaluating the likelihood of generated samples, and most popular DDGNs (such as VAEs and GANs) do not provide a convenient way of doing so.5

To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\).

In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016).

We only output the scaling factors \(g\) from the hypernet, and learn a maximum likelihood estimate of \(\mathbf{v}\).7 This allows us to overcome the computational limitations of naively-parametrized BHNs noted by Louizos & Welling (2017), since computation now scales linearly, instead of quadratically, in the number of primary net units. Using this parametrization restricts the family of approximate posteriors, but still allows for a high degree of multimodality and dependence between the parameters.

For the hypernet architecture, we try both RealNVP (Dinh et al., 2016) and IAF(Kingma et al., 2016) with MADE(Germain et al., 2015), with 1-layer ReLU-MLP coupling functions with 200 hidden units (each).

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations.

In this paper, we employ weight normalization in the primary network (7), treating (only) the scaling factors \(\mathbf{g}\) as random variables. We choose an isotropic Gaussian prior for \(\mathbf{g}\): \(p(\mathbf{g})=\mathcal{N}(\mathbf{g};\mathbf{0},\lambda\mathbf{I})\), which results in an \(L_{2}\) weight-decay penalty on \(\mathbf{g}\), or, equivalently, \(\mathbf{w}=\mathbf{g}\frac{\mathbf{v}}{||\mathbf{v}||_{2}}\). Our objective and lower bound are then: \[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "The key insight for building such a model is the use of an invertible hypernet, which enables Monte Carlo estimation of the entropy term -log q(θ) in the variational inference training objective."

Analysis:
The evidence strongly supports this claim through:
1. The paper explicitly shows how invertibility enables computing q(θ) using the change of variables formula (Equation 6)
2. The review acknowledges this technical contribution while pointing out similarities to prior work [1]
3. The evidence demonstrates how this enables Monte Carlo estimation through equations 8-10
4. The reviewer's technical concern about the KL divergence support doesn't invalidate this specific claim about entropy estimation

While there are some technical issues raised by the reviewer, they don't directly contradict this specific claim about the role of invertibility in enabling entropy term estimation. The claim carefully stays within what's demonstrated by the mathematical derivations.

<score>1</score>

The claim is well-stated, accurately reflecting the technical contribution without overstating its significance. The evidence directly shows how invertibility enables computing the entropy term through the change of variables formula, which is essential for the Monte Carlo estimation approach described in the methodology.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The keyinsight for building such a model is the use of an invertible hypernet, which enables Monte Carlo estimation of the entropy term \(-\log q(\mathbf{\theta})\) in the variational inference training objective.

The review comment to be evaluated is:

This paper proposes Bayesian hypernetworks to carry out Bayesian learning of deep networks. The idea is to construct a generative model capable of approximating the posterior distribution over the parameters of deep networks. I think that the paper is well written and easy to follow.

I like the idea of constructing general approximation strategies for complex posterior distribution and the proposed approach inherits all the scalability properties of modern deep learning techniques. In this respect, I think that the paper tackles a timely topic and is interesting to read.

It is not entirely clear to me why the Authors name their proposal Bayesian hypernetworks. This seems to suggest that also the hypernetwork is infered using Bayesian inference, but if I understand correctly this is not the case.

I have some comments on novelty and realization of the experiments. In the positioning of the work in the literature, the Authors point out that hypernetworks have been proposed before, so it is not clear what is the actual novelty in the proposal. Is it the use of Real NVPs and IAFs as hypernetworks? These methods have been already proposed and extensively studied in the literature, and even if they have been adapted to be hypernetworks here, I believe that the novelty is fairly limited.

The experimental part is interesting as it explores a number of learning scenarios. However, I think that it would have been useful to add comparisons with standard variational inference (e.g., Graves, 2011) for deep networks to substantiate the claims that this approach underestimates uncertainty. I believe that this would strengthen the comparative evaluation.

I think the paper would have made a stronger case by including other approaches to approximate posteriors using generative models. For example, the variational Gaussian process paper sounds like an ideal method to include here.

[1] D. Tran, R. Ranganath, and D. M. Blei. Variational Gaussian process. arXiv preprint arXiv:1511.06499, 2015.

The evidence to be evaluated is:Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) (Goodfellow et al., 2016) as a generative model of the primary net parameters.

We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016). Training these models uses the change of variables formula, which involves computing the log-determinant of the inverse Jacobian of the generator network.

Let \(\mathbf{\theta}\in\mathbb{R}^{D}\) be parameters given the Bayesian treatment as random variables, \(\mathcal{D}\) a training set of observed data, and \(q(\mathbf{\theta})\) a learned approximation to the true posterior \(p(\mathbf{\theta}|\mathcal{D})\).

\[\log p(\mathcal{D}) =\text{KL}(q(\mathbf{\theta})\|p(\mathbf{\theta}|\mathcal{D}))+\mathbb{E} _{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{\theta})-\log q(\mathbf{\theta})] \tag{1}\] \[\geq\mathbb{E}_{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{ \theta})-\log q(\mathbf{\theta})]\,. \tag{2}\]

Computing the expectation in (2) is generally intractable for deep nets, but can be estimated by Monte Carlo sampling.

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\).

However, while DDGNs are convenient to sample from, computing the entropy term (\(\mathbb{E}_{q}[-\log q(\mathbf{\theta})]\)) of the ELBO additionally requires evaluating the likelihood of generated samples, and most popular DDGNs (such as VAEs and GANs) do not provide a convenient way of doing so.5

To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\).

In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016).

We only output the scaling factors \(g\) from the hypernet, and learn a maximum likelihood estimate of \(\mathbf{v}\).7 This allows us to overcome the computational limitations of naively-parametrized BHNs noted by Louizos & Welling (2017), since computation now scales linearly, instead of quadratically, in the number of primary net units. Using this parametrization restricts the family of approximate posteriors, but still allows for a high degree of multimodality and dependence between the parameters.

For the hypernet architecture, we try both RealNVP (Dinh et al., 2016) and IAF(Kingma et al., 2016) with MADE(Germain et al., 2015), with 1-layer ReLU-MLP coupling functions with 200 hidden units (each).

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations.

In this paper, we employ weight normalization in the primary network (7), treating (only) the scaling factors \(\mathbf{g}\) as random variables. We choose an isotropic Gaussian prior for \(\mathbf{g}\): \(p(\mathbf{g})=\mathcal{N}(\mathbf{g};\mathbf{0},\lambda\mathbf{I})\), which results in an \(L_{2}\) weight-decay penalty on \(\mathbf{g}\), or, equivalently, \(\mathbf{w}=\mathbf{g}\frac{\mathbf{v}}{||\mathbf{v}||_{2}}\). Our objective and lower bound are then: \[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "The key insight for building such a model is the use of an invertible hypernet, which enables Monte Carlo estimation of the entropy term -log q(θ) in the variational inference training objective."

Analysis:
The evidence strongly supports this claim through:
1. Clear mathematical demonstration showing how invertibility allows computing q(θ) using the change of variables formula (equation 6)
2. Explicit explanation that most DDGNs don't provide a convenient way to evaluate the likelihood of generated samples
3. Detailed derivation showing how the invertible architecture enables computing the ELBO and its derivatives (equations 8-10)
4. Implementation details confirming the use of invertible architectures (RealNVP and IAF)

The reviewer comments don't dispute this technical aspect of the work. The claim accurately represents the mathematical foundation without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The keyinsight for building such a model is the use of an invertible hypernet, which enables Monte Carlo estimation of the entropy term \(-\log q(\mathbf{\theta})\) in the variational inference training objective.

The review comment to be evaluated is:

* Edit: I increased my rating to 6. The authors fixed the first error I pointed out below. Regarding the second point: I still think it is possible to take a limit of sigma -> 0 in MNF, which makes the methods very similar.

The authors propose a new method of defining approximate posteriors for use in Bayesian neural networks. The idea of using hypernetworks for Bayesian inference is compelling, and the authors show some promising first results. I see two issues, and would be willing to increase my rating if these were sufficiently addressed.

- The paper says it uses an "isotropic standard normal prior on the weights of the network". However, the stochastic part of the generated weights (i.e. the scales) is of a lower dimension than the weights. It seems to me this means that the KL divergence between prior and posterior is undefined, or infinite, as the posterior is only defined on a sub-manifold. What exactly is the loss term that is added to the training objective? And how is this justified?

- The instantiation of Bayesian hypernetworks that is used in experiments seems to be a special case of the method of multiplicative normalizing flows as proposed by Louizos and Welling and discussed in this paper. If the variances / sigmas are zero in the latter method, their approximation seems functionally equivalent to Bayesian hypernetworks (though with different parameterization). Is my understanding correct? If so, the novelty of the proposed method is limited.

The evidence to be evaluated is:Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) (Goodfellow et al., 2016) as a generative model of the primary net parameters.

We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016). Training these models uses the change of variables formula, which involves computing the log-determinant of the inverse Jacobian of the generator network.

Let \(\mathbf{\theta}\in\mathbb{R}^{D}\) be parameters given the Bayesian treatment as random variables, \(\mathcal{D}\) a training set of observed data, and \(q(\mathbf{\theta})\) a learned approximation to the true posterior \(p(\mathbf{\theta}|\mathcal{D})\).

\[\log p(\mathcal{D}) =\text{KL}(q(\mathbf{\theta})\|p(\mathbf{\theta}|\mathcal{D}))+\mathbb{E} _{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{\theta})-\log q(\mathbf{\theta})] \tag{1}\] \[\geq\mathbb{E}_{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{ \theta})-\log q(\mathbf{\theta})]\,. \tag{2}\]

Computing the expectation in (2) is generally intractable for deep nets, but can be estimated by Monte Carlo sampling.

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\).

However, while DDGNs are convenient to sample from, computing the entropy term (\(\mathbb{E}_{q}[-\log q(\mathbf{\theta})]\)) of the ELBO additionally requires evaluating the likelihood of generated samples, and most popular DDGNs (such as VAEs and GANs) do not provide a convenient way of doing so.5

To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\).

In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016).

We only output the scaling factors \(g\) from the hypernet, and learn a maximum likelihood estimate of \(\mathbf{v}\).7 This allows us to overcome the computational limitations of naively-parametrized BHNs noted by Louizos & Welling (2017), since computation now scales linearly, instead of quadratically, in the number of primary net units. Using this parametrization restricts the family of approximate posteriors, but still allows for a high degree of multimodality and dependence between the parameters.

For the hypernet architecture, we try both RealNVP (Dinh et al., 2016) and IAF(Kingma et al., 2016) with MADE(Germain et al., 2015), with 1-layer ReLU-MLP coupling functions with 200 hidden units (each).

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations.

In this paper, we employ weight normalization in the primary network (7), treating (only) the scaling factors \(\mathbf{g}\) as random variables. We choose an isotropic Gaussian prior for \(\mathbf{g}\): \(p(\mathbf{g})=\mathcal{N}(\mathbf{g};\mathbf{0},\lambda\mathbf{I})\), which results in an \(L_{2}\) weight-decay penalty on \(\mathbf{g}\), or, equivalently, \(\mathbf{w}=\mathbf{g}\frac{\mathbf{v}}{||\mathbf{v}||_{2}}\). Our objective and lower bound are then: \[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate this claim:

1. Summary of evaluation:
The claim states that "the key insight is the use of an invertible hypernet, which enables Monte Carlo estimation of the entropy term." The evidence strongly supports this through detailed mathematical derivations showing how the invertible hypernetwork allows computation of q(θ) using the change of variables formula (equation 6). However, based on the reviewer's comments, this may be similar to existing methods (multiplicative normalizing flows) when certain parameters approach zero, suggesting the novelty might be overstated. Additionally, there appear to be some technical concerns about the KL divergence calculation that weren't fully addressed in the evidence.

<score>3</score>

2. Justification:
- While the evidence demonstrates that the invertible hypernetwork does enable Monte Carlo estimation through the change of variables formula, calling it "the key insight" may overstate its uniqueness given the reviewer's point about similarity to existing methods.
- The mathematical formulation in equations 1-2 and 6-10 supports the technical validity of using invertible hypernets for the stated purpose.
- The reviewer raises valid concerns about potential equivalence to prior work (multiplicative normalizing flows) and technical issues with KL divergence calculations that aren't fully addressed in the evidence.
- The claim would be better stated if it acknowledged these limitations or positioned the insight as an effective approach rather than "the key insight."

The medium score (3) reflects that while the technical claim is supported by evidence, its presentation as a key novel insight appears somewhat overstated given the context from the review.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In contrast to most methods for Bayesian deep learning, Bayesian hypernets can represent a complex multimodal approximate posterior with correlations between parameters, while enabling cheap iid sampling of \(q(\mathbf{\theta})\).

The evidence to be evaluated is:Broadly speaking, existing methods either 1) use Markov chain Monte Carlo (Welling & Teh, 2011; Neal, 1996), or 2) directly learn an approximate posterior distribution using (stochastic) variational inference (Graves, 2011; Gal & Ghahramani, 2016; Salimans et al., 2015; Blundell et al., 2015), expectation propagation (Hernandez-Lobato & Adams, 2015; Soudry et al., 2014), or \(\alpha\)-divergences (Li & Gal, 2017).

Kingma et al. (2015) emulates Gaussian dropout, but yields a unimodal approximate posterior, and does not allow arbitrary dependencies between the parameters. The other important points of reference for our work are Bayes by Backprop (BbB) (Blundell et al., 2015), and multiplicative normalizing flows (Louizos & Welling, 2017). Bayes by Backprop can be viewed as a special instance of a Bayesian hypernet, where the hypernetwork only performs an element-wise scale and shift of the input noise (yielding a factorial Gaussian distribution).

Instead, in their work, they use a hypernet to generate scaling factors, \(\mathbf{z}\) on the means \(\mathbf{\mu}\) of a factorial Gaussian distribution. Because \(\mathbf{z}\) follows a complicated distribution, this forms a highly flexible approximate posterior: \(q(\mathbf{\theta})=\int q(\mathbf{\theta}|\mathbf{z})q(\mathbf{z})\mathrm{d}\mathbf{z}\).

A hypernetwork (Ha et al., 2017; Brabandere et al., 2016; Bertinetto et al., 2016) is a neural net that outputs parameters of another neural net (the "primary network").3

Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) (Goodfellow et al., 2016) as a generative model of the primary net parameters. DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) (Kingma and Welling, 2013; Jimenez Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014; Goodfellow, 2017). We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016). Training these models uses the change of variables formula, which involves computing the log-determinant of the inverse Jacobian of the generator network. This computation involves a potentially costly matrix determinant, and these works propose innovative architectures which reduce the cost of this operation but can still express complicated deformations, which are referred to as "normalizing flows".

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\). Since BHNs are both trained and evaluated via samples of \(q(\mathbf{\theta})\), expressing \(q(\mathbf{\theta})\) as a generative model is a natural strategy.

\[q(\mathbf{\theta})= \int q(\mathbf{\theta};h(\mathbf{\epsilon}))q(\mathbf{\epsilon})d\mathbf{ \epsilon}\,. \tag{5}\] To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\).

In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of \(h\), as in generative modeling applications, e.g., Dinh et al. (2016)); and this also means that we can use a lower-dimensional \(\mathbf{\epsilon}\) to generate samples along a submanifold of the entire parameter space, as detailed below.

This allows us to overcome the computational limitations of naively-parametrized BHNs noted by Louizos & Welling (2017), since computation now scales linearly, instead of quadratically, in the number of primary net units. Using this parametrization restricts the family of approximate posteriors, but still allows for a high degree of multimodality and dependence between the parameters.

Next, we demonstrate the distinctive ability of Bayesian hypernets to learn multi-modal, dependent distributions. Figure 6 (appendix) shows that BHNs do learn approximate posteriors with dependence between different parameters, as measured by the Pearson correlation coefficient. Meanwhile, Figure 2 shows that BHNs are capable of learning multimodal posteriors. For this experiment, we trained an over-parametrized linear (primary) network: \(\hat{y}=a\cdot b\cdot x\) on a dataset generated as \(y=x+\epsilon\), and the BHN learns capture both the modes of \(a=b=1\) and \(a=b=-1\).

furthermore, increasing the flexibility of the posterior by adding more coupling layers improves performance, especially compared with models with 0 coupling layers, which cannot model dependencies between the parameters.

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations. We present a method of parametrizing BHNs which allows them to scale successfully to real world tasks, and show that BHNs can offer significant benefits over simpler methods for Bayesian deep learning.

In this paper, we employ weight normalization in the primary network (7), treating (only) the scaling factors \(\mathbf{g}\) as random variables. We choose an isotropic Gaussian prior for \(\mathbf{g}\): \(p(\mathbf{g})=\mathcal{N}(\mathbf{g};\mathbf{0},\lambda\mathbf{I})\), which results in an \(L_{2}\) weight-decay penalty on \(\mathbf{g}\), or, equivalently, \(\mathbf{w}=\mathbf{g}\frac{\mathbf{v}}{||\mathbf{v}||_{2}}\). Our objective and lower bound are then: \[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input: Figure 2: Learning the identity function with an overparametrized network: ŷ = a · b · x. This parametrization results in symmetries shown by the dashed red lines, and the Bayesian hypernetwork assigns significant probability mass to both modes of the posterior (a = b = 1 and a = b = −1).

Input: Figure 6: Histogram of Pearson correlation coefficient p-values (left) and a scatter matrix (right) of samples from a hypernet approximate posterior. We see that the hypernet posterior includes correlations between different parameters. Many of the p-values of the Pearson correlation test are below .05.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim about Bayesian hypernets' ability to represent complex multimodal approximate posteriors and enable cheap sampling.

The evidence strongly supports this claim through:
1. Theoretical framework showing how BHNs use invertible DDGNs to transform simple noise into complex distributions (equations 5-6)
2. Empirical demonstration of multimodality in Figure 2, showing BHNs capturing both modes (a=b=1 and a=b=-1)
3. Evidence of parameter correlations in Figure 6
4. Computational efficiency demonstrated through linear scaling
5. Direct comparisons to other methods showing BHNs' advantages over simpler variational approximations

The claim appears to be carefully worded and accurately reflects both the theoretical foundations and empirical results without exaggeration.

<score>1</score>

Justification:
- The claim's two main assertions (complex multimodal posterior + cheap sampling) are directly supported by mathematical formulation and experimental results
- Limitations are acknowledged throughout (e.g., "restricts the family of approximate posteriors")
- The evidence demonstrates both capabilities rigorously through theory and experiments
- The comparative statements are measured and supported by specific references to other methods
- No overreaching generalizations or unsupported statements are made