Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

With this in mind, we propose learning an extremely flexible and powerful posterior, parametrized by a DNN \(h\), which we refer to as a Bayesian hypernetwork in reference to Ha et al. (2017).

The evidence to be evaluated is:The other important points of reference for our work are Bayes by Backprop (BbB) (Blundell et al., 2015), and multiplicative normalizing flows (Louizos & Welling, 2017). Bayes by Backprop can be viewed as a special instance of a Bayesian hypernet, where the hypernetwork only performs an element-wise scale and shift of the input noise (yielding a factorial Gaussian distribution). More similar is the work of Louizos & Welling (2017), who propose and dismiss BHNs due to the issues of scaling BHNs to large primary networks, which we address in Section 3.3.2

The idea is also explored by Shi et al. (2017), who likewise reject it in favor of their implicit approach which estimates the KL-divergence using a classifier.

A hypernetwork (Ha et al., 2017; Brabandere et al., 2016; Bertinetto et al., 2016) is a neural net that outputs parameters of another neural net (the "primary network").3 The hypernet and primary net together form a single model which is trained by backpropagation.

Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) (Goodfellow et al., 2016) as a generative model of the primary net parameters. DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) (Kingma and Welling, 2013; Jimenez Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014; Goodfellow, 2017). We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016).

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\). Since BHNs are both trained and evaluated via samples of \(q(\mathbf{\theta})\), expressing \(q(\mathbf{\theta})\) as a generative model is a natural strategy.

\[q(\mathbf{\theta})= \int q(\mathbf{\theta};h(\mathbf{\epsilon}))q(\mathbf{\epsilon})d\mathbf{ \epsilon}\,. \tag{5}\] To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\). As discussed in Section 2.3, a number of techniques have been developed for efficiently training such invertible DDGNs. In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of \(h\), as in generative modeling applications, e.g., Dinh et al. (2016)); and this also means that we can use a lower-dimensional \(\mathbf{\epsilon}\) to generate samples along a submanifold of the entire parameter space, as detailed below. In order to scale BHNs to large primary networks, we use the weight normalization reparametrization (Salimans & Kingma, 2016)6:

We only output the scaling factors \(g\) from the hypernet, and learn a maximum likelihood estimate of \(\mathbf{v}\).7 This allows us to overcome the computational limitations of naively-parametrized BHNs noted by Louizos & Welling (2017), since computation now scales linearly, instead of quadratically, in the number of primary net units. Using this parametrization restricts the family of approximate posteriors, but still allows for a high degree of multimodality and dependence between the parameters.

Next, we demonstrate the distinctive ability of Bayesian hypernets to learn multi-modal, dependent distributions. Figure 6 (appendix) shows that BHNs do learn approximate posteriors with dependence between different parameters, as measured by the Pearson correlation coefficient. Meanwhile, Figure 2 shows that BHNs are capable of learning multimodal posteriors.

In our experiments, we find that BHNs perform on par with dropout on full datasets of MNIST and CIFAR10; furthermore, increasing the flexibility of the posterior by adding more coupling layers improves performance, especially compared with models with 0 coupling layers, which cannot model dependencies between the parameters.

In these MNIST experiments, we use MLPs with 2 hidden layers of 800 or 1200 hidden units each. For CIFAR10, we train a convolutional neural net (CNN) with 4 hidden layers of \([64,64,128,128]\) channels, \(2\times 2\) max pooling after the second and the fourth layers, filter size of \(3\), and a single fully connected layer of \(512\) units.

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations. We present a method of parametrizing BHNs which allows them to scale successfully to real world tasks, and show that BHNs can offer significant benefits over simpler methods for Bayesian deep learning.

In this paper, we employ weight normalization in the primary network (7), treating (only) the scaling factors \(\mathbf{g}\) as random variables. We choose an isotropic Gaussian prior for \(\mathbf{g}\): \(p(\mathbf{g})=\mathcal{N}(\mathbf{g};\mathbf{0},\lambda\mathbf{I})\), which results in an \(L_{2}\) weight-decay penalty on \(\mathbf{g}\), or, equivalently, \(\mathbf{w}=\mathbf{g}\frac{\mathbf{v}}{||\mathbf{v}||_{2}}\). Our objective and lower bound are then: \[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input: Figure 2: Learning the identity function with an overparametrized network: ŷ = a · b · x. This parametrization results in symmetries shown by the dashed red lines, and the Bayesian hypernetwork assigns significant probability mass to both modes of the posterior (a = b = 1 and a = b = −1).

Input: Figure 6: Histogram of Pearson correlation coefficient p-values (left) and a scatter matrix (right) of samples from a hypernet approximate posterior. We see that the hypernet posterior includes correlations between different parameters. Many of the p-values of the Pearson correlation test are below .05.

Input: Figure 3: Box plot of performance across 10 trials. Bayesian hypernets (BHNs) with inverse autoregressive flows (IAF) consistently outperform the other methods.

Input: Figure 4: Active learning: Bayesian hypernets outperform other approaches after sufficient acquisitions when warm-starting (left), for both random acquisition function (top) and BALD acquisition function (bottom). Warm-starting improves stability for all methods, but hurts performance for other approaches, compared with randomly re-initializing parameters as in Gal et al. (2017) (right). We also note that the baseline model (no dropout) is competitive with MCdropout, and outperforms the Dropout baseline used by (Gal et al., 2017).9 These curves are the average of three experiments.

Input: Figure 1: Illustration of BHNs (second and third) and a traditional non-Bayesian DNN (first) on the toy problem from Blundell et al. (2015). In the second subplot, we place a prior on the scaling factor g and infer the posterior distribution using a BHN, while in the third subplot the hypernet is used to generate the whole weight matrices of the primary net. Each shaded region represents half a standard deviation in the posterior on the predictive mean. The red crosses are 50 examples from the training dataset.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
1. Based on the evidence, the claim appears to be well-stated (score of 1). The evidence thoroughly demonstrates that the proposed Bayesian hypernetwork is indeed flexible and powerful through:
- Mathematical formulation showing its capability to transform random noise into complex parameter distributions
- Experimental results showing it can learn multi-modal posteriors (Figure 2)
- Evidence of capturing parameter dependencies (Figure 6)
- Competitive performance on real datasets like MNIST and CIFAR10
- Successful scaling to large networks through weight normalization
- Demonstration of superior performance in active learning scenarios (Figure 4)

The claim's characterization as "extremely flexible and powerful" is supported by these comprehensive theoretical foundations and empirical results.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

With this in mind, we propose learning an extremely flexible and powerful posterior, parametrized by a DNN \(h\), which we refer to as a Bayesian hypernetwork in reference to Ha et al. (2017).

The review comment to be evaluated is:

This paper presents Bayesian Hypernetworks; variational Bayesian neural networks where the variational posterior over the weights is governed by a hyper network that implements a normalizing flow (NF) such as RealNVP and IAF. As directly outputting the weight matrix with a hyper network is computationally expensive the authors instead propose to utilize weight normalisation on the weights and then use the hyper network to output scalar scaling variables for each hidden unit, similarly to what was done at [1]. The main difference with this prior work is that [1] consider these NF scaling variables as auxiliary random variables to a mean field Gaussian distribution over the weights whereas this paper attempts to posit a distribution directly on the weights via the NF. This avoids the nested variational approximation and auxiliary models of [1], which can potentially yield a tighter bound. The proposed method is evaluated on extensive experiments.

This paper seems like a plausible idea with extensive experiments but the similarity with [1] make it an incremental contribution and, furthermore, it seems that it has a technical issue with what is explained at Section 3.3. More specifically, if you generate the parameters \theta according to Eq. 7 and posit a prior over \theta then you will have a problematic variational bound as there will be a KL divergence, KL(q(\theta) || p(\theta)), with distributions of different support (since q(\theta) is defined only along the directions spanned by u), which is infinite. For the KL to be valid you will need to posit a prior distribution over `g`, p(g), and then consider KL(q(g) || p(g)), with q(g) being given by the NF. From the experiment paragraph at page 5 though I deduct that you instead employ “an isotropic standard normal prior over the weights”, i.e. \theta, thus I believe that you indeed have a problematic bound. How do you actually compute logq(\theta) when you employ the parametrisation discussed at 3.3? Did you use that parametrisation in every experiment?

Other than that, I believe that it would be interesting to experiment with a `full` hyper network, i.e. generating directly the entire parameter vector \theta, e.g. at the toy regression experiment where the dimensionality is small. This would then better illustrate the tradeoffs you make when you reduce the flexibility of the hyper-network to just outputting the row scaling variables and the effect this has at the posterior approximation.

Typos:
(1) Page 3, 3.1.1 log(\theta) -> logp(\theta).
(2) Eq. 6, it needs to be |det \frac{\partial h(\epsilon)}{\partial \epsilon}|^{-1} or |det \frac{\partial h^{-1}(\theta)}{\partial \theta}| for a valid change of variables formula.

[1] Louizos & Welling, Multiplicative Normalizing Flows for Variational Bayesian Neural Networks.

The evidence to be evaluated is:The other important points of reference for our work are Bayes by Backprop (BbB) (Blundell et al., 2015), and multiplicative normalizing flows (Louizos & Welling, 2017). Bayes by Backprop can be viewed as a special instance of a Bayesian hypernet, where the hypernetwork only performs an element-wise scale and shift of the input noise (yielding a factorial Gaussian distribution). More similar is the work of Louizos & Welling (2017), who propose and dismiss BHNs due to the issues of scaling BHNs to large primary networks, which we address in Section 3.3.2

The idea is also explored by Shi et al. (2017), who likewise reject it in favor of their implicit approach which estimates the KL-divergence using a classifier.

A hypernetwork (Ha et al., 2017; Brabandere et al., 2016; Bertinetto et al., 2016) is a neural net that outputs parameters of another neural net (the "primary network").3 The hypernet and primary net together form a single model which is trained by backpropagation.

Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) (Goodfellow et al., 2016) as a generative model of the primary net parameters. DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) (Kingma and Welling, 2013; Jimenez Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014; Goodfellow, 2017). We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016).

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\). Since BHNs are both trained and evaluated via samples of \(q(\mathbf{\theta})\), expressing \(q(\mathbf{\theta})\) as a generative model is a natural strategy.

\[q(\mathbf{\theta})= \int q(\mathbf{\theta};h(\mathbf{\epsilon}))q(\mathbf{\epsilon})d\mathbf{ \epsilon}\,. \tag{5}\] To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\). As discussed in Section 2.3, a number of techniques have been developed for efficiently training such invertible DDGNs. In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of \(h\), as in generative modeling applications, e.g., Dinh et al. (2016)); and this also means that we can use a lower-dimensional \(\mathbf{\epsilon}\) to generate samples along a submanifold of the entire parameter space, as detailed below. In order to scale BHNs to large primary networks, we use the weight normalization reparametrization (Salimans & Kingma, 2016)6:

We only output the scaling factors \(g\) from the hypernet, and learn a maximum likelihood estimate of \(\mathbf{v}\).7 This allows us to overcome the computational limitations of naively-parametrized BHNs noted by Louizos & Welling (2017), since computation now scales linearly, instead of quadratically, in the number of primary net units. Using this parametrization restricts the family of approximate posteriors, but still allows for a high degree of multimodality and dependence between the parameters.

Next, we demonstrate the distinctive ability of Bayesian hypernets to learn multi-modal, dependent distributions. Figure 6 (appendix) shows that BHNs do learn approximate posteriors with dependence between different parameters, as measured by the Pearson correlation coefficient. Meanwhile, Figure 2 shows that BHNs are capable of learning multimodal posteriors.

In our experiments, we find that BHNs perform on par with dropout on full datasets of MNIST and CIFAR10; furthermore, increasing the flexibility of the posterior by adding more coupling layers improves performance, especially compared with models with 0 coupling layers, which cannot model dependencies between the parameters.

In these MNIST experiments, we use MLPs with 2 hidden layers of 800 or 1200 hidden units each. For CIFAR10, we train a convolutional neural net (CNN) with 4 hidden layers of \([64,64,128,128]\) channels, \(2\times 2\) max pooling after the second and the fourth layers, filter size of \(3\), and a single fully connected layer of \(512\) units.

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations. We present a method of parametrizing BHNs which allows them to scale successfully to real world tasks, and show that BHNs can offer significant benefits over simpler methods for Bayesian deep learning.

In this paper, we employ weight normalization in the primary network (7), treating (only) the scaling factors \(\mathbf{g}\) as random variables. We choose an isotropic Gaussian prior for \(\mathbf{g}\): \(p(\mathbf{g})=\mathcal{N}(\mathbf{g};\mathbf{0},\lambda\mathbf{I})\), which results in an \(L_{2}\) weight-decay penalty on \(\mathbf{g}\), or, equivalently, \(\mathbf{w}=\mathbf{g}\frac{\mathbf{v}}{||\mathbf{v}||_{2}}\). Our objective and lower bound are then: \[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input: Figure 2: Learning the identity function with an overparametrized network: ŷ = a · b · x. This parametrization results in symmetries shown by the dashed red lines, and the Bayesian hypernetwork assigns significant probability mass to both modes of the posterior (a = b = 1 and a = b = −1).

Input: Figure 6: Histogram of Pearson correlation coefficient p-values (left) and a scatter matrix (right) of samples from a hypernet approximate posterior. We see that the hypernet posterior includes correlations between different parameters. Many of the p-values of the Pearson correlation test are below .05.

Input: Figure 3: Box plot of performance across 10 trials. Bayesian hypernets (BHNs) with inverse autoregressive flows (IAF) consistently outperform the other methods.

Input: Figure 4: Active learning: Bayesian hypernets outperform other approaches after sufficient acquisitions when warm-starting (left), for both random acquisition function (top) and BALD acquisition function (bottom). Warm-starting improves stability for all methods, but hurts performance for other approaches, compared with randomly re-initializing parameters as in Gal et al. (2017) (right). We also note that the baseline model (no dropout) is competitive with MCdropout, and outperforms the Dropout baseline used by (Gal et al., 2017).9 These curves are the average of three experiments.

Input: Figure 1: Illustration of BHNs (second and third) and a traditional non-Bayesian DNN (first) on the toy problem from Blundell et al. (2015). In the second subplot, we place a prior on the scaling factor g and infer the posterior distribution using a BHN, while in the third subplot the hypernet is used to generate the whole weight matrices of the primary net. Each shaded region represents half a standard deviation in the posterior on the predictive mean. The red crosses are 50 examples from the training dataset.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that it is somewhat overstated. While the paper does present a Bayesian hypernetwork that shows flexibility in modeling multimodal and dependent distributions (as shown in Figures 2 and 6), calling it "extremely flexible and powerful" is an exaggeration based on the available evidence. The reviewer points out technical issues with the KL divergence bound and notes the similarity to previous work, suggesting incremental rather than revolutionary advancement. The experimental results show comparable but not clearly superior performance to existing methods like dropout on standard benchmarks.

<score>3</score>

Justification:
1. The statement overstates the flexibility and power without sufficient comparative evidence. While results show multimodality and parameter dependencies, the term "extremely" is not justified.

2. The reviewer highlights technical issues regarding the KL divergence bound and support of distributions, indicating limitations not acknowledged in the claim.

3. The experimental evidence shows:
- Comparable performance to dropout on MNIST and CIFAR10
- Some advantages in specific scenarios like active learning
- Restricted flexibility due to the weight normalization parameterization used for scaling

4. The direct citation showing limitation: "Using this parametrization restricts the family of approximate posteriors, but still allows for a high degree of multimodality and dependence between the parameters."

5. More measured language about capabilities and limitations would better reflect the evidence presented.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

With this in mind, we propose learning an extremely flexible and powerful posterior, parametrized by a DNN \(h\), which we refer to as a Bayesian hypernetwork in reference to Ha et al. (2017).

The review comment to be evaluated is:

This paper proposes Bayesian hypernetworks to carry out Bayesian learning of deep networks. The idea is to construct a generative model capable of approximating the posterior distribution over the parameters of deep networks. I think that the paper is well written and easy to follow.

I like the idea of constructing general approximation strategies for complex posterior distribution and the proposed approach inherits all the scalability properties of modern deep learning techniques. In this respect, I think that the paper tackles a timely topic and is interesting to read.

It is not entirely clear to me why the Authors name their proposal Bayesian hypernetworks. This seems to suggest that also the hypernetwork is infered using Bayesian inference, but if I understand correctly this is not the case.

I have some comments on novelty and realization of the experiments. In the positioning of the work in the literature, the Authors point out that hypernetworks have been proposed before, so it is not clear what is the actual novelty in the proposal. Is it the use of Real NVPs and IAFs as hypernetworks? These methods have been already proposed and extensively studied in the literature, and even if they have been adapted to be hypernetworks here, I believe that the novelty is fairly limited.

The experimental part is interesting as it explores a number of learning scenarios. However, I think that it would have been useful to add comparisons with standard variational inference (e.g., Graves, 2011) for deep networks to substantiate the claims that this approach underestimates uncertainty. I believe that this would strengthen the comparative evaluation.

I think the paper would have made a stronger case by including other approaches to approximate posteriors using generative models. For example, the variational Gaussian process paper sounds like an ideal method to include here.

[1] D. Tran, R. Ranganath, and D. M. Blei. Variational Gaussian process. arXiv preprint arXiv:1511.06499, 2015.

The evidence to be evaluated is:The other important points of reference for our work are Bayes by Backprop (BbB) (Blundell et al., 2015), and multiplicative normalizing flows (Louizos & Welling, 2017). Bayes by Backprop can be viewed as a special instance of a Bayesian hypernet, where the hypernetwork only performs an element-wise scale and shift of the input noise (yielding a factorial Gaussian distribution). More similar is the work of Louizos & Welling (2017), who propose and dismiss BHNs due to the issues of scaling BHNs to large primary networks, which we address in Section 3.3.2

The idea is also explored by Shi et al. (2017), who likewise reject it in favor of their implicit approach which estimates the KL-divergence using a classifier.

A hypernetwork (Ha et al., 2017; Brabandere et al., 2016; Bertinetto et al., 2016) is a neural net that outputs parameters of another neural net (the "primary network").3 The hypernet and primary net together form a single model which is trained by backpropagation.

Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) (Goodfellow et al., 2016) as a generative model of the primary net parameters. DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) (Kingma and Welling, 2013; Jimenez Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014; Goodfellow, 2017). We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016).

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\). Since BHNs are both trained and evaluated via samples of \(q(\mathbf{\theta})\), expressing \(q(\mathbf{\theta})\) as a generative model is a natural strategy.

\[q(\mathbf{\theta})= \int q(\mathbf{\theta};h(\mathbf{\epsilon}))q(\mathbf{\epsilon})d\mathbf{ \epsilon}\,. \tag{5}\] To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\). As discussed in Section 2.3, a number of techniques have been developed for efficiently training such invertible DDGNs. In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of \(h\), as in generative modeling applications, e.g., Dinh et al. (2016)); and this also means that we can use a lower-dimensional \(\mathbf{\epsilon}\) to generate samples along a submanifold of the entire parameter space, as detailed below. In order to scale BHNs to large primary networks, we use the weight normalization reparametrization (Salimans & Kingma, 2016)6:

We only output the scaling factors \(g\) from the hypernet, and learn a maximum likelihood estimate of \(\mathbf{v}\).7 This allows us to overcome the computational limitations of naively-parametrized BHNs noted by Louizos & Welling (2017), since computation now scales linearly, instead of quadratically, in the number of primary net units. Using this parametrization restricts the family of approximate posteriors, but still allows for a high degree of multimodality and dependence between the parameters.

Next, we demonstrate the distinctive ability of Bayesian hypernets to learn multi-modal, dependent distributions. Figure 6 (appendix) shows that BHNs do learn approximate posteriors with dependence between different parameters, as measured by the Pearson correlation coefficient. Meanwhile, Figure 2 shows that BHNs are capable of learning multimodal posteriors.

In our experiments, we find that BHNs perform on par with dropout on full datasets of MNIST and CIFAR10; furthermore, increasing the flexibility of the posterior by adding more coupling layers improves performance, especially compared with models with 0 coupling layers, which cannot model dependencies between the parameters.

In these MNIST experiments, we use MLPs with 2 hidden layers of 800 or 1200 hidden units each. For CIFAR10, we train a convolutional neural net (CNN) with 4 hidden layers of \([64,64,128,128]\) channels, \(2\times 2\) max pooling after the second and the fourth layers, filter size of \(3\), and a single fully connected layer of \(512\) units.

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations. We present a method of parametrizing BHNs which allows them to scale successfully to real world tasks, and show that BHNs can offer significant benefits over simpler methods for Bayesian deep learning.

In this paper, we employ weight normalization in the primary network (7), treating (only) the scaling factors \(\mathbf{g}\) as random variables. We choose an isotropic Gaussian prior for \(\mathbf{g}\): \(p(\mathbf{g})=\mathcal{N}(\mathbf{g};\mathbf{0},\lambda\mathbf{I})\), which results in an \(L_{2}\) weight-decay penalty on \(\mathbf{g}\), or, equivalently, \(\mathbf{w}=\mathbf{g}\frac{\mathbf{v}}{||\mathbf{v}||_{2}}\). Our objective and lower bound are then: \[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input: Figure 2: Learning the identity function with an overparametrized network: ŷ = a · b · x. This parametrization results in symmetries shown by the dashed red lines, and the Bayesian hypernetwork assigns significant probability mass to both modes of the posterior (a = b = 1 and a = b = −1).

Input: Figure 6: Histogram of Pearson correlation coefficient p-values (left) and a scatter matrix (right) of samples from a hypernet approximate posterior. We see that the hypernet posterior includes correlations between different parameters. Many of the p-values of the Pearson correlation test are below .05.

Input: Figure 3: Box plot of performance across 10 trials. Bayesian hypernets (BHNs) with inverse autoregressive flows (IAF) consistently outperform the other methods.

Input: Figure 4: Active learning: Bayesian hypernets outperform other approaches after sufficient acquisitions when warm-starting (left), for both random acquisition function (top) and BALD acquisition function (bottom). Warm-starting improves stability for all methods, but hurts performance for other approaches, compared with randomly re-initializing parameters as in Gal et al. (2017) (right). We also note that the baseline model (no dropout) is competitive with MCdropout, and outperforms the Dropout baseline used by (Gal et al., 2017).9 These curves are the average of three experiments.

Input: Figure 1: Illustration of BHNs (second and third) and a traditional non-Bayesian DNN (first) on the toy problem from Blundell et al. (2015). In the second subplot, we place a prior on the scaling factor g and infer the posterior distribution using a BHN, while in the third subplot the hypernet is used to generate the whole weight matrices of the primary net. Each shaded region represents half a standard deviation in the posterior on the predictive mean. The red crosses are 50 examples from the training dataset.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "With this in mind, we propose learning an extremely flexible and powerful posterior, parametrized by a DNN (h), which we refer to as a Bayesian hypernetwork in reference to Ha et al. (2017)."

Summary:
The evidence substantially supports this claim, though with some minor overstatement. The paper demonstrates the flexibility through:
- Ability to learn multimodal distributions (Figure 2)
- Capturing parameter dependencies (Figure 6)
- Competitive performance on real tasks (MNIST, CIFAR10)
- Successful scaling to large networks

However, the term "extremely flexible and powerful" is somewhat overstated, as noted in the review comments about limited novelty and lack of comparisons to some alternative methods. The approach builds incrementally on existing techniques rather than representing an extreme advancement.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

With this in mind, we propose learning an extremely flexible and powerful posterior, parametrized by a DNN \(h\), which we refer to as a Bayesian hypernetwork in reference to Ha et al. (2017).

The review comment to be evaluated is:

* Edit: I increased my rating to 6. The authors fixed the first error I pointed out below. Regarding the second point: I still think it is possible to take a limit of sigma -> 0 in MNF, which makes the methods very similar.

The authors propose a new method of defining approximate posteriors for use in Bayesian neural networks. The idea of using hypernetworks for Bayesian inference is compelling, and the authors show some promising first results. I see two issues, and would be willing to increase my rating if these were sufficiently addressed.

- The paper says it uses an "isotropic standard normal prior on the weights of the network". However, the stochastic part of the generated weights (i.e. the scales) is of a lower dimension than the weights. It seems to me this means that the KL divergence between prior and posterior is undefined, or infinite, as the posterior is only defined on a sub-manifold. What exactly is the loss term that is added to the training objective? And how is this justified?

- The instantiation of Bayesian hypernetworks that is used in experiments seems to be a special case of the method of multiplicative normalizing flows as proposed by Louizos and Welling and discussed in this paper. If the variances / sigmas are zero in the latter method, their approximation seems functionally equivalent to Bayesian hypernetworks (though with different parameterization). Is my understanding correct? If so, the novelty of the proposed method is limited.

The evidence to be evaluated is:The other important points of reference for our work are Bayes by Backprop (BbB) (Blundell et al., 2015), and multiplicative normalizing flows (Louizos & Welling, 2017). Bayes by Backprop can be viewed as a special instance of a Bayesian hypernet, where the hypernetwork only performs an element-wise scale and shift of the input noise (yielding a factorial Gaussian distribution). More similar is the work of Louizos & Welling (2017), who propose and dismiss BHNs due to the issues of scaling BHNs to large primary networks, which we address in Section 3.3.2

The idea is also explored by Shi et al. (2017), who likewise reject it in favor of their implicit approach which estimates the KL-divergence using a classifier.

A hypernetwork (Ha et al., 2017; Brabandere et al., 2016; Bertinetto et al., 2016) is a neural net that outputs parameters of another neural net (the "primary network").3 The hypernet and primary net together form a single model which is trained by backpropagation.

Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) (Goodfellow et al., 2016) as a generative model of the primary net parameters. DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) (Kingma and Welling, 2013; Jimenez Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014; Goodfellow, 2017). We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016).

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\). Since BHNs are both trained and evaluated via samples of \(q(\mathbf{\theta})\), expressing \(q(\mathbf{\theta})\) as a generative model is a natural strategy.

\[q(\mathbf{\theta})= \int q(\mathbf{\theta};h(\mathbf{\epsilon}))q(\mathbf{\epsilon})d\mathbf{ \epsilon}\,. \tag{5}\] To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\). As discussed in Section 2.3, a number of techniques have been developed for efficiently training such invertible DDGNs. In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of \(h\), as in generative modeling applications, e.g., Dinh et al. (2016)); and this also means that we can use a lower-dimensional \(\mathbf{\epsilon}\) to generate samples along a submanifold of the entire parameter space, as detailed below. In order to scale BHNs to large primary networks, we use the weight normalization reparametrization (Salimans & Kingma, 2016)6:

We only output the scaling factors \(g\) from the hypernet, and learn a maximum likelihood estimate of \(\mathbf{v}\).7 This allows us to overcome the computational limitations of naively-parametrized BHNs noted by Louizos & Welling (2017), since computation now scales linearly, instead of quadratically, in the number of primary net units. Using this parametrization restricts the family of approximate posteriors, but still allows for a high degree of multimodality and dependence between the parameters.

Next, we demonstrate the distinctive ability of Bayesian hypernets to learn multi-modal, dependent distributions. Figure 6 (appendix) shows that BHNs do learn approximate posteriors with dependence between different parameters, as measured by the Pearson correlation coefficient. Meanwhile, Figure 2 shows that BHNs are capable of learning multimodal posteriors.

In our experiments, we find that BHNs perform on par with dropout on full datasets of MNIST and CIFAR10; furthermore, increasing the flexibility of the posterior by adding more coupling layers improves performance, especially compared with models with 0 coupling layers, which cannot model dependencies between the parameters.

In these MNIST experiments, we use MLPs with 2 hidden layers of 800 or 1200 hidden units each. For CIFAR10, we train a convolutional neural net (CNN) with 4 hidden layers of \([64,64,128,128]\) channels, \(2\times 2\) max pooling after the second and the fourth layers, filter size of \(3\), and a single fully connected layer of \(512\) units.

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations. We present a method of parametrizing BHNs which allows them to scale successfully to real world tasks, and show that BHNs can offer significant benefits over simpler methods for Bayesian deep learning.

In this paper, we employ weight normalization in the primary network (7), treating (only) the scaling factors \(\mathbf{g}\) as random variables. We choose an isotropic Gaussian prior for \(\mathbf{g}\): \(p(\mathbf{g})=\mathcal{N}(\mathbf{g};\mathbf{0},\lambda\mathbf{I})\), which results in an \(L_{2}\) weight-decay penalty on \(\mathbf{g}\), or, equivalently, \(\mathbf{w}=\mathbf{g}\frac{\mathbf{v}}{||\mathbf{v}||_{2}}\). Our objective and lower bound are then: \[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input: Figure 2: Learning the identity function with an overparametrized network: ŷ = a · b · x. This parametrization results in symmetries shown by the dashed red lines, and the Bayesian hypernetwork assigns significant probability mass to both modes of the posterior (a = b = 1 and a = b = −1).

Input: Figure 6: Histogram of Pearson correlation coefficient p-values (left) and a scatter matrix (right) of samples from a hypernet approximate posterior. We see that the hypernet posterior includes correlations between different parameters. Many of the p-values of the Pearson correlation test are below .05.

Input: Figure 3: Box plot of performance across 10 trials. Bayesian hypernets (BHNs) with inverse autoregressive flows (IAF) consistently outperform the other methods.

Input: Figure 4: Active learning: Bayesian hypernets outperform other approaches after sufficient acquisitions when warm-starting (left), for both random acquisition function (top) and BALD acquisition function (bottom). Warm-starting improves stability for all methods, but hurts performance for other approaches, compared with randomly re-initializing parameters as in Gal et al. (2017) (right). We also note that the baseline model (no dropout) is competitive with MCdropout, and outperforms the Dropout baseline used by (Gal et al., 2017).9 These curves are the average of three experiments.

Input: Figure 1: Illustration of BHNs (second and third) and a traditional non-Bayesian DNN (first) on the toy problem from Blundell et al. (2015). In the second subplot, we place a prior on the scaling factor g and infer the posterior distribution using a BHN, while in the third subplot the hypernet is used to generate the whole weight matrices of the primary net. Each shaded region represents half a standard deviation in the posterior on the predictive mean. The red crosses are 50 examples from the training dataset.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
The claim about proposing "an extremely flexible and powerful posterior" via Bayesian hypernetworks requires careful assessment against the evidence:

Key supporting evidence:
- The method can learn multi-modal, dependent distributions (shown in Figures 2 and 6)
- BHNs perform on par with dropout on MNIST/CIFAR10 datasets
- They demonstrate successful scaling to real-world tasks
- The reviewer acknowledges the method is "compelling" with "promising results"

However, there are important caveats:
- The reviewer points out similarity to existing methods (multiplicative normalizing flows)
- The flexibility comes with computational costs that required specific parameterization to address
- Performance improvements are modest compared to baselines in some cases

While the approach shows merit in terms of flexibility and capability, describing it as "extremely flexible and powerful" appears somewhat overstated given the evidence and reviewer comments. A more measured description would better reflect the demonstrated capabilities.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

A Bayesian hypernetwork (BHN) takes random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\) as input and outputs a sample from the approximate posterior \(q(\mathbf{\theta})\) for another DNN of interest (the "primary network").

The evidence to be evaluated is:Broadly speaking, existing methods either 1) use Markov chain Monte Carlo (Welling & Teh, 2011; Neal, 1996), or 2) directly learn an approximate posterior distribution using (stochastic) variational inference (Graves, 2011; Gal & Ghahramani, 2016; Salimans et al., 2015; Blundell et al., 2015), expectation propagation (Hernandez-Lobato & Adams, 2015; Soudry et al., 2014), or \(\alpha\)-divergences (Li & Gal, 2017). We focus here on the most popular approach: variational inference.

The other important points of reference for our work are Bayes by Backprop (BbB) (Blundell et al., 2015), and multiplicative normalizing flows (Louizos & Welling, 2017). Bayes by Backprop can be viewed as a special instance of a Bayesian hypernet, where the hypernetwork only performs an element-wise scale and shift of the input noise (yielding a factorial Gaussian distribution).

A hypernetwork (Ha et al., 2017; Brabandere et al., 2016; Bertinetto et al., 2016) is a neural net that outputs parameters of another neural net (the "primary network").3 The hypernet and primary net together form a single model which is trained by backpropagation.

Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) (Goodfellow et al., 2016) as a generative model of the primary net parameters. DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) (Kingma and Welling, 2013; Jimenez Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014; Goodfellow, 2017). We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016).

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\). Since BHNs are both trained and evaluated via samples of \(q(\mathbf{\theta})\), expressing \(q(\mathbf{\theta})\) as a generative model is a natural strategy.

\[q(\mathbf{\theta})= \int q(\mathbf{\theta};h(\mathbf{\epsilon}))q(\mathbf{\epsilon})d\mathbf{ \epsilon}\,. \tag{5}\] To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\). As discussed in Section 2.3, a number of techniques have been developed for efficiently training such invertible DDGNs. In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of \(h\), as in generative modeling applications, e.g., Dinh et al. (2016)); and this also means that we can use a lower-dimensional \(\mathbf{\epsilon}\) to generate samples along a submanifold of the entire parameter space, as detailed below.

\[\mathbf{\theta}_{j}=g\,\mathbf{u}\,,\quad\mathbf{u}:=\frac{\mathbf{v}}{\|\mathbf{ v}\|_{2}}\,,\quad g\in\mathbb{R}\,, \tag{7}\] where \(\mathbf{\theta}_{j}\) are the input weights associated with a single unit \(j\) in the primary network. We only output the scaling factors \(g\) from the hypernet, and learn a maximum likelihood estimate of \(\mathbf{v}\).7

Next, we demonstrate the distinctive ability of Bayesian hypernets to learn multi-modal, dependent distributions. Figure 6 (appendix) shows that BHNs do learn approximate posteriors with dependence between different parameters, as measured by the Pearson correlation coefficient. Meanwhile, Figure 2 shows that BHNs are capable of learning multimodal posteriors. For this experiment, we trained an over-parametrized linear (primary) network: \(\hat{y}=a\cdot b\cdot x\) on a dataset generated as \(y=x+\epsilon\), and the BHN learns capture both the modes of \(a=b=1\) and \(a=b=-1\).

furthermore, increasing the flexibility of the posterior by adding more coupling layers improves performance, especially compared with models with 0 coupling layers, which cannot model dependencies between the parameters.

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations. We present a method of parametrizing BHNs which allows them to scale successfully to real world tasks, and show that BHNs can offer significant benefits over simpler methods for Bayesian deep learning.

\[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input: Figure 2: Learning the identity function with an overparametrized network: ŷ = a · b · x. This parametrization results in symmetries shown by the dashed red lines, and the Bayesian hypernetwork assigns significant probability mass to both modes of the posterior (a = b = 1 and a = b = −1).

Input: Figure 6: Histogram of Pearson correlation coefficient p-values (left) and a scatter matrix (right) of samples from a hypernet approximate posterior. We see that the hypernet posterior includes correlations between different parameters. Many of the p-values of the Pearson correlation test are below .05.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, I find this claim to be well-stated (accurately reflects the evidence). The evidence thoroughly supports the fundamental mechanism described in the claim through mathematical formulations, implementation details, and empirical demonstrations. The paper provides:
- Explicit mathematical formulation showing how BHN transforms noise ε~N(0,I) to posterior samples via equations (5) and (6)
- Practical implementation details using invertible architectures (RealNVP, IAF)
- Empirical validation showing BHNs can learn complex multimodal posteriors (Figure 2) and parameter dependencies (Figure 6)
The claim makes no exaggerated statements and accurately represents the core functionality backed by comprehensive evidence.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

A Bayesian hypernetwork (BHN) takes random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\) as input and outputs a sample from the approximate posterior \(q(\mathbf{\theta})\) for another DNN of interest (the "primary network").

The review comment to be evaluated is:

This paper presents Bayesian Hypernetworks; variational Bayesian neural networks where the variational posterior over the weights is governed by a hyper network that implements a normalizing flow (NF) such as RealNVP and IAF. As directly outputting the weight matrix with a hyper network is computationally expensive the authors instead propose to utilize weight normalisation on the weights and then use the hyper network to output scalar scaling variables for each hidden unit, similarly to what was done at [1]. The main difference with this prior work is that [1] consider these NF scaling variables as auxiliary random variables to a mean field Gaussian distribution over the weights whereas this paper attempts to posit a distribution directly on the weights via the NF. This avoids the nested variational approximation and auxiliary models of [1], which can potentially yield a tighter bound. The proposed method is evaluated on extensive experiments.

This paper seems like a plausible idea with extensive experiments but the similarity with [1] make it an incremental contribution and, furthermore, it seems that it has a technical issue with what is explained at Section 3.3. More specifically, if you generate the parameters \theta according to Eq. 7 and posit a prior over \theta then you will have a problematic variational bound as there will be a KL divergence, KL(q(\theta) || p(\theta)), with distributions of different support (since q(\theta) is defined only along the directions spanned by u), which is infinite. For the KL to be valid you will need to posit a prior distribution over `g`, p(g), and then consider KL(q(g) || p(g)), with q(g) being given by the NF. From the experiment paragraph at page 5 though I deduct that you instead employ “an isotropic standard normal prior over the weights”, i.e. \theta, thus I believe that you indeed have a problematic bound. How do you actually compute logq(\theta) when you employ the parametrisation discussed at 3.3? Did you use that parametrisation in every experiment?

Other than that, I believe that it would be interesting to experiment with a `full` hyper network, i.e. generating directly the entire parameter vector \theta, e.g. at the toy regression experiment where the dimensionality is small. This would then better illustrate the tradeoffs you make when you reduce the flexibility of the hyper-network to just outputting the row scaling variables and the effect this has at the posterior approximation.

Typos:
(1) Page 3, 3.1.1 log(\theta) -> logp(\theta).
(2) Eq. 6, it needs to be |det \frac{\partial h(\epsilon)}{\partial \epsilon}|^{-1} or |det \frac{\partial h^{-1}(\theta)}{\partial \theta}| for a valid change of variables formula.

[1] Louizos & Welling, Multiplicative Normalizing Flows for Variational Bayesian Neural Networks.

The evidence to be evaluated is:Broadly speaking, existing methods either 1) use Markov chain Monte Carlo (Welling & Teh, 2011; Neal, 1996), or 2) directly learn an approximate posterior distribution using (stochastic) variational inference (Graves, 2011; Gal & Ghahramani, 2016; Salimans et al., 2015; Blundell et al., 2015), expectation propagation (Hernandez-Lobato & Adams, 2015; Soudry et al., 2014), or \(\alpha\)-divergences (Li & Gal, 2017). We focus here on the most popular approach: variational inference.

The other important points of reference for our work are Bayes by Backprop (BbB) (Blundell et al., 2015), and multiplicative normalizing flows (Louizos & Welling, 2017). Bayes by Backprop can be viewed as a special instance of a Bayesian hypernet, where the hypernetwork only performs an element-wise scale and shift of the input noise (yielding a factorial Gaussian distribution).

A hypernetwork (Ha et al., 2017; Brabandere et al., 2016; Bertinetto et al., 2016) is a neural net that outputs parameters of another neural net (the "primary network").3 The hypernet and primary net together form a single model which is trained by backpropagation.

Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) (Goodfellow et al., 2016) as a generative model of the primary net parameters. DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) (Kingma and Welling, 2013; Jimenez Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014; Goodfellow, 2017). We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016).

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\). Since BHNs are both trained and evaluated via samples of \(q(\mathbf{\theta})\), expressing \(q(\mathbf{\theta})\) as a generative model is a natural strategy.

\[q(\mathbf{\theta})= \int q(\mathbf{\theta};h(\mathbf{\epsilon}))q(\mathbf{\epsilon})d\mathbf{ \epsilon}\,. \tag{5}\] To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\). As discussed in Section 2.3, a number of techniques have been developed for efficiently training such invertible DDGNs. In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of \(h\), as in generative modeling applications, e.g., Dinh et al. (2016)); and this also means that we can use a lower-dimensional \(\mathbf{\epsilon}\) to generate samples along a submanifold of the entire parameter space, as detailed below.

\[\mathbf{\theta}_{j}=g\,\mathbf{u}\,,\quad\mathbf{u}:=\frac{\mathbf{v}}{\|\mathbf{ v}\|_{2}}\,,\quad g\in\mathbb{R}\,, \tag{7}\] where \(\mathbf{\theta}_{j}\) are the input weights associated with a single unit \(j\) in the primary network. We only output the scaling factors \(g\) from the hypernet, and learn a maximum likelihood estimate of \(\mathbf{v}\).7

Next, we demonstrate the distinctive ability of Bayesian hypernets to learn multi-modal, dependent distributions. Figure 6 (appendix) shows that BHNs do learn approximate posteriors with dependence between different parameters, as measured by the Pearson correlation coefficient. Meanwhile, Figure 2 shows that BHNs are capable of learning multimodal posteriors. For this experiment, we trained an over-parametrized linear (primary) network: \(\hat{y}=a\cdot b\cdot x\) on a dataset generated as \(y=x+\epsilon\), and the BHN learns capture both the modes of \(a=b=1\) and \(a=b=-1\).

furthermore, increasing the flexibility of the posterior by adding more coupling layers improves performance, especially compared with models with 0 coupling layers, which cannot model dependencies between the parameters.

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations. We present a method of parametrizing BHNs which allows them to scale successfully to real world tasks, and show that BHNs can offer significant benefits over simpler methods for Bayesian deep learning.

\[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input: Figure 2: Learning the identity function with an overparametrized network: ŷ = a · b · x. This parametrization results in symmetries shown by the dashed red lines, and the Bayesian hypernetwork assigns significant probability mass to both modes of the posterior (a = b = 1 and a = b = −1).

Input: Figure 6: Histogram of Pearson correlation coefficient p-values (left) and a scatter matrix (right) of samples from a hypernet approximate posterior. We see that the hypernet posterior includes correlations between different parameters. Many of the p-values of the Pearson correlation test are below .05.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I assess that the claim is somewhat overstated. While it accurately states that BHN takes random noise as input and outputs samples from an approximate posterior, it oversimplifies several important technical aspects:

1. The evidence shows BHNs specifically use an invertible DDGN (h) to transform the noise, which is crucial for computing q(θ) using the change of variables formula
2. Based on the reviewer's comments, there are technical issues with the support of distributions when using the parametrization in Section 3.3
3. The claim doesn't mention that BHN can output parameters along a submanifold rather than the entire parameter space, which is an important qualification noted in the evidence

A more precise statement would include these technical details and limitations.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

A Bayesian hypernetwork (BHN) takes random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\) as input and outputs a sample from the approximate posterior \(q(\mathbf{\theta})\) for another DNN of interest (the "primary network").

The review comment to be evaluated is:

This paper proposes Bayesian hypernetworks to carry out Bayesian learning of deep networks. The idea is to construct a generative model capable of approximating the posterior distribution over the parameters of deep networks. I think that the paper is well written and easy to follow.

I like the idea of constructing general approximation strategies for complex posterior distribution and the proposed approach inherits all the scalability properties of modern deep learning techniques. In this respect, I think that the paper tackles a timely topic and is interesting to read.

It is not entirely clear to me why the Authors name their proposal Bayesian hypernetworks. This seems to suggest that also the hypernetwork is infered using Bayesian inference, but if I understand correctly this is not the case.

I have some comments on novelty and realization of the experiments. In the positioning of the work in the literature, the Authors point out that hypernetworks have been proposed before, so it is not clear what is the actual novelty in the proposal. Is it the use of Real NVPs and IAFs as hypernetworks? These methods have been already proposed and extensively studied in the literature, and even if they have been adapted to be hypernetworks here, I believe that the novelty is fairly limited.

The experimental part is interesting as it explores a number of learning scenarios. However, I think that it would have been useful to add comparisons with standard variational inference (e.g., Graves, 2011) for deep networks to substantiate the claims that this approach underestimates uncertainty. I believe that this would strengthen the comparative evaluation.

I think the paper would have made a stronger case by including other approaches to approximate posteriors using generative models. For example, the variational Gaussian process paper sounds like an ideal method to include here.

[1] D. Tran, R. Ranganath, and D. M. Blei. Variational Gaussian process. arXiv preprint arXiv:1511.06499, 2015.

The evidence to be evaluated is:Broadly speaking, existing methods either 1) use Markov chain Monte Carlo (Welling & Teh, 2011; Neal, 1996), or 2) directly learn an approximate posterior distribution using (stochastic) variational inference (Graves, 2011; Gal & Ghahramani, 2016; Salimans et al., 2015; Blundell et al., 2015), expectation propagation (Hernandez-Lobato & Adams, 2015; Soudry et al., 2014), or \(\alpha\)-divergences (Li & Gal, 2017). We focus here on the most popular approach: variational inference.

The other important points of reference for our work are Bayes by Backprop (BbB) (Blundell et al., 2015), and multiplicative normalizing flows (Louizos & Welling, 2017). Bayes by Backprop can be viewed as a special instance of a Bayesian hypernet, where the hypernetwork only performs an element-wise scale and shift of the input noise (yielding a factorial Gaussian distribution).

A hypernetwork (Ha et al., 2017; Brabandere et al., 2016; Bertinetto et al., 2016) is a neural net that outputs parameters of another neural net (the "primary network").3 The hypernet and primary net together form a single model which is trained by backpropagation.

Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) (Goodfellow et al., 2016) as a generative model of the primary net parameters. DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) (Kingma and Welling, 2013; Jimenez Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014; Goodfellow, 2017). We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016).

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\). Since BHNs are both trained and evaluated via samples of \(q(\mathbf{\theta})\), expressing \(q(\mathbf{\theta})\) as a generative model is a natural strategy.

\[q(\mathbf{\theta})= \int q(\mathbf{\theta};h(\mathbf{\epsilon}))q(\mathbf{\epsilon})d\mathbf{ \epsilon}\,. \tag{5}\] To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\). As discussed in Section 2.3, a number of techniques have been developed for efficiently training such invertible DDGNs. In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of \(h\), as in generative modeling applications, e.g., Dinh et al. (2016)); and this also means that we can use a lower-dimensional \(\mathbf{\epsilon}\) to generate samples along a submanifold of the entire parameter space, as detailed below.

\[\mathbf{\theta}_{j}=g\,\mathbf{u}\,,\quad\mathbf{u}:=\frac{\mathbf{v}}{\|\mathbf{ v}\|_{2}}\,,\quad g\in\mathbb{R}\,, \tag{7}\] where \(\mathbf{\theta}_{j}\) are the input weights associated with a single unit \(j\) in the primary network. We only output the scaling factors \(g\) from the hypernet, and learn a maximum likelihood estimate of \(\mathbf{v}\).7

Next, we demonstrate the distinctive ability of Bayesian hypernets to learn multi-modal, dependent distributions. Figure 6 (appendix) shows that BHNs do learn approximate posteriors with dependence between different parameters, as measured by the Pearson correlation coefficient. Meanwhile, Figure 2 shows that BHNs are capable of learning multimodal posteriors. For this experiment, we trained an over-parametrized linear (primary) network: \(\hat{y}=a\cdot b\cdot x\) on a dataset generated as \(y=x+\epsilon\), and the BHN learns capture both the modes of \(a=b=1\) and \(a=b=-1\).

furthermore, increasing the flexibility of the posterior by adding more coupling layers improves performance, especially compared with models with 0 coupling layers, which cannot model dependencies between the parameters.

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations. We present a method of parametrizing BHNs which allows them to scale successfully to real world tasks, and show that BHNs can offer significant benefits over simpler methods for Bayesian deep learning.

\[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input: Figure 2: Learning the identity function with an overparametrized network: ŷ = a · b · x. This parametrization results in symmetries shown by the dashed red lines, and the Bayesian hypernetwork assigns significant probability mass to both modes of the posterior (a = b = 1 and a = b = −1).

Input: Figure 6: Histogram of Pearson correlation coefficient p-values (left) and a scatter matrix (right) of samples from a hypernet approximate posterior. We see that the hypernet posterior includes correlations between different parameters. Many of the p-values of the Pearson correlation test are below .05.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll assess the claim: "A Bayesian hypernetwork (BHN) takes random noise ε~N(0,I) as input and outputs a sample from the approximate posterior q(θ) for another DNN of interest (the 'primary network')."

This claim appears to be well-stated and accurately reflects the evidence without exaggeration. The evidence directly supports this in multiple ways:

1. The paper explicitly states "BHNs express a flexible q(θ) by using a DDGN...to transform random noise ε~N(0,I_D) into independent samples from q(θ)"

2. The evidence demonstrates this mechanism through equations (5) and (6), showing how the transformation occurs

3. Figures 2 and 6 provide empirical validation by showing actual samples generated through this process

4. The reviewer comments don't dispute this fundamental mechanical description

The claim is a straightforward technical description of the BHN's basic operation, without overstating capabilities or making unsupported generalizations.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

A Bayesian hypernetwork (BHN) takes random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\) as input and outputs a sample from the approximate posterior \(q(\mathbf{\theta})\) for another DNN of interest (the "primary network").

The review comment to be evaluated is:

* Edit: I increased my rating to 6. The authors fixed the first error I pointed out below. Regarding the second point: I still think it is possible to take a limit of sigma -> 0 in MNF, which makes the methods very similar.

The authors propose a new method of defining approximate posteriors for use in Bayesian neural networks. The idea of using hypernetworks for Bayesian inference is compelling, and the authors show some promising first results. I see two issues, and would be willing to increase my rating if these were sufficiently addressed.

- The paper says it uses an "isotropic standard normal prior on the weights of the network". However, the stochastic part of the generated weights (i.e. the scales) is of a lower dimension than the weights. It seems to me this means that the KL divergence between prior and posterior is undefined, or infinite, as the posterior is only defined on a sub-manifold. What exactly is the loss term that is added to the training objective? And how is this justified?

- The instantiation of Bayesian hypernetworks that is used in experiments seems to be a special case of the method of multiplicative normalizing flows as proposed by Louizos and Welling and discussed in this paper. If the variances / sigmas are zero in the latter method, their approximation seems functionally equivalent to Bayesian hypernetworks (though with different parameterization). Is my understanding correct? If so, the novelty of the proposed method is limited.

The evidence to be evaluated is:Broadly speaking, existing methods either 1) use Markov chain Monte Carlo (Welling & Teh, 2011; Neal, 1996), or 2) directly learn an approximate posterior distribution using (stochastic) variational inference (Graves, 2011; Gal & Ghahramani, 2016; Salimans et al., 2015; Blundell et al., 2015), expectation propagation (Hernandez-Lobato & Adams, 2015; Soudry et al., 2014), or \(\alpha\)-divergences (Li & Gal, 2017). We focus here on the most popular approach: variational inference.

The other important points of reference for our work are Bayes by Backprop (BbB) (Blundell et al., 2015), and multiplicative normalizing flows (Louizos & Welling, 2017). Bayes by Backprop can be viewed as a special instance of a Bayesian hypernet, where the hypernetwork only performs an element-wise scale and shift of the input noise (yielding a factorial Gaussian distribution).

A hypernetwork (Ha et al., 2017; Brabandere et al., 2016; Bertinetto et al., 2016) is a neural net that outputs parameters of another neural net (the "primary network").3 The hypernet and primary net together form a single model which is trained by backpropagation.

Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) (Goodfellow et al., 2016) as a generative model of the primary net parameters. DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) (Kingma and Welling, 2013; Jimenez Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014; Goodfellow, 2017). We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016).

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\). Since BHNs are both trained and evaluated via samples of \(q(\mathbf{\theta})\), expressing \(q(\mathbf{\theta})\) as a generative model is a natural strategy.

\[q(\mathbf{\theta})= \int q(\mathbf{\theta};h(\mathbf{\epsilon}))q(\mathbf{\epsilon})d\mathbf{ \epsilon}\,. \tag{5}\] To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\). As discussed in Section 2.3, a number of techniques have been developed for efficiently training such invertible DDGNs. In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of \(h\), as in generative modeling applications, e.g., Dinh et al. (2016)); and this also means that we can use a lower-dimensional \(\mathbf{\epsilon}\) to generate samples along a submanifold of the entire parameter space, as detailed below.

\[\mathbf{\theta}_{j}=g\,\mathbf{u}\,,\quad\mathbf{u}:=\frac{\mathbf{v}}{\|\mathbf{ v}\|_{2}}\,,\quad g\in\mathbb{R}\,, \tag{7}\] where \(\mathbf{\theta}_{j}\) are the input weights associated with a single unit \(j\) in the primary network. We only output the scaling factors \(g\) from the hypernet, and learn a maximum likelihood estimate of \(\mathbf{v}\).7

Next, we demonstrate the distinctive ability of Bayesian hypernets to learn multi-modal, dependent distributions. Figure 6 (appendix) shows that BHNs do learn approximate posteriors with dependence between different parameters, as measured by the Pearson correlation coefficient. Meanwhile, Figure 2 shows that BHNs are capable of learning multimodal posteriors. For this experiment, we trained an over-parametrized linear (primary) network: \(\hat{y}=a\cdot b\cdot x\) on a dataset generated as \(y=x+\epsilon\), and the BHN learns capture both the modes of \(a=b=1\) and \(a=b=-1\).

furthermore, increasing the flexibility of the posterior by adding more coupling layers improves performance, especially compared with models with 0 coupling layers, which cannot model dependencies between the parameters.

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations. We present a method of parametrizing BHNs which allows them to scale successfully to real world tasks, and show that BHNs can offer significant benefits over simpler methods for Bayesian deep learning.

\[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input: Figure 2: Learning the identity function with an overparametrized network: ŷ = a · b · x. This parametrization results in symmetries shown by the dashed red lines, and the Bayesian hypernetwork assigns significant probability mass to both modes of the posterior (a = b = 1 and a = b = −1).

Input: Figure 6: Histogram of Pearson correlation coefficient p-values (left) and a scatter matrix (right) of samples from a hypernet approximate posterior. We see that the hypernet posterior includes correlations between different parameters. Many of the p-values of the Pearson correlation test are below .05.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim about Bayesian hypernetworks based on the provided evidence.

1. Summary Evaluation:
The claim is well-stated and accurately reflects the evidence. The evidence thoroughly demonstrates that BHNs do take random noise ε~N(0,I) as input and output samples from the approximate posterior q(θ). This is supported by:
- Explicit mathematical formulation in equations (5) and (6)
- Demonstration of multimodal posterior distributions in Figure 2
- Empirical validation through correlation analysis in Figure 6
- Detailed theoretical framework explaining the transformation process

The claim is conservative in its statement, focusing on the core functional aspect without overstating capabilities or advantages.

<score>1</score>

Justification:
1. Direct mathematical support: The evidence provides formal equations describing exactly how BHNs transform noise into posterior samples
2. Visual confirmation: Figures 2 and 6 demonstrate the claimed functionality
3. Measured scope: The claim sticks to basic operational description without exaggerating capabilities
4. Theoretical grounding: Evidence thoroughly explains the underlying mechanisms
5. The reviewer's comments don't dispute this basic functional claim, instead focusing on other aspects like novelty and implementation details