Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

3) image to image: generating images with the GAN image generator, or retrieving images, given the variational posterior produced by the VHE image encoder fed with a natural image, with the similarity between a generated image and the input one measurable by the GAN discriminator; and 4) noise to text-image pair: generating a document using the VHE text decoder and a corresponding image by feeding the latent variable of the VHE text decoder into the GAN image generator.

The evidence to be evaluated is:Integrating VHE and GAN, the proposed VHEGAN, as shown in Fig. 1 (c), tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document, under the Poisson GBN text decoder, be the distribution of the random noise fed into the GAN image generator; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder; and the GAN discriminator tries to discriminate between that selected image and the generated one.

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN.

Thus, we combine GAN and VHE in a novel way via a shared latent space, to further explore the relationships between two different modes for ZSL.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator. More specifically, we modify the GAN objective function as \[\min_{G}\max_{D}L_{\text{GAN}}(D,G)=\min_{G}\max_{D}\mathbb{E}_{\mathbf{x}\sim p_ {data}(\mathbf{x})}\left\{\ln D(\mathbf{x})-\mathbb{E}_{\mathbf{\theta}\sim q_{\mathbf{\Omega} }(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln D(G(\mathbf{\theta}))\right]\right\} \tag{6}\] where \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) is defined as in (2).

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities.

Note we choose the GAN generator and discriminator to be the same as those of deep convolutional GAN of Radford et al. (2016). Other more recent GANs, such as Wasserstein GAN (Arjovsky et al., 2017), Wasserstein GAN with gradient penalty (Gulrajani et al., 2017), spectral norm GAN (Miyato et al., 2018), and progressive growing GAN (Karras et al., 2018), may be combined with the VHE to construct even more expressive VHEGANs.

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7).

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning.

Both these two approaches are two-step models with the additional textual feature extraction realized by LSTM and "shallow" LDA, respectively, whereas the VHEGAN can realize joint optimization with hybrid SGMCMC/VHE/SGD algorithm.

To solve the problem of complete random generate process in GAN, conditional GANs (CGAN) (Mirza & Osindero, 2014; Reed et al., 2016) are alsodeveloped, where the conditions are labels, or attributes learned as point estimations separately with GAN, while VHEGAN learns a distributed condition jointly with a VAE model. Moreover, VHEGAN is similar with BiGAN (Donahue et al., 2017) and ALI (Dumoulin et al., 2017) since all of them learn a encoder that maps data into a latent space in GAN. Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

The 4096 dimensional image features, \(f(\mathbf{x})\) in (3), from the fc1 layer of the pre-trained VGG16 network (Simonyan & Zisserman, 2015) are fed into the VHE image encoder, and the size of each generated RGB image is set as \(64\times 64\).

except for the proposed VHEGAN that aims to find a shared semantically meaningful latent space between the image and text modalities, none of the other methods have generative models for both modalities, regardless of whether they learn a classifier or a distance metric in a latent space for ZSL.

Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images.

We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images). We repeat the text-to-image generation experiment on Flower to generate random images given textual captions for three different unseen classes. As shown in Fig. 1(b), VHEGAN successfully generates flower images that semantically match their corresponding textual descriptions and visually resemble their corresponding example real images.

Giving the textural description \(\mathbf{t}\) of an unseen class as the text query, we retrieve the top five images ranked by the cosine distances between \(\mathbf{t}\) and \(\hat{\mathbf{t}}_{i}\). Shown in Fig. 3 are two example image retrieval results, which suggest that the retrieved images are semantically related to their text queries in colors, shapes, and locations.

With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image.

After training a 3-layer VHEGAN, following the data generation process of the VHE text decoder (\(i.e.\), the Poisson GBN), given \(\{\mathbf{\Phi}^{(l)}\}_{l=1}^{L}\) and \(\mathbf{r}\), we first generate \(\mathbf{\theta}^{(L)}\sim\text{Gam}\left(\mathbf{r},1/s^{L+1}\right)\) and then downward propagate it through the Poisson GBN as in (1) to calculate the Poisson rates for all words using \(\mathbf{\Phi}^{(1)}\mathbf{\theta}^{(1)}\). Given a random draw, the concatenation of \(\{\mathbf{\theta}^{(l)}\}_{l=1}^{L}\) is fed into the GAN image generator to generate a corresponding image. Shown in Fig. 6 are four random draws, for each of which we show its top seven words and its generated image, whose relationships are clearly interpretable, suggesting that VHEGAN is able to recode the key information of both modalities and the relationships between them. **Image to image generation:** We note for the VHEGAN, its VHE image encoder and GAN component together can also be viewed as an "autoencoding" GAN for images. Different from Dumoulin et al. (2017) and Donahue et al. (2017), the encoded latent space is affected not only by the GAN for the image modality, but also by the Poisson GBN for the text modailty. In Fig. 7, we present the real images \(\mathbf{x}\) and their VHEGANregenerations \(G\left(q_{\boldsymbol{\Omega}}\left(\boldsymbol{\theta}\,|\,\boldsymbol{\Phi}, \boldsymbol{x}\right)\right)\), which suggest that the VHEGAN can use its GAN image generator to regenerate random images that more or less resemble the original real image fed into the VHE image encoder.

For each column in Fig. 7(b), conditioning on a textual description, we show the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all layers, respectively, as the latent space fed into the GAN image generator.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space. With a stochastic-gradient MCMC algorithm to sample the global parameters of the VHE text decoder, a reparameterized Weibull distribution based variational encoder to approximate the posterior distribution of the local parameters of the VHE image encoder, and stochastic gradient descent (SGD) to estimate the parameters of the GAN image generator and discriminator, we develop a SG-MCMC/VHE/SGD hybrid inference algorithm to jointly train the components of VHEGAN. Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

We give a detailed hybrid SGMCMC/SGD algorithm for VHEGAN in this section, to realize learning end-to-end.

In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively.

\(\mathbf{x}\rightarrow\mathbf{h}^{(l)},l=1,2,3\)\(\mathbf{x}\rightarrow\mathbf{h}^{(1)}\)\(1\times\) { fully-connected layer with 256 units and Leaky-RELU activation.}\(\mathbf{h}^{(1)}\rightarrow\mathbf{h}^{(2)}\)\(1\times\) { fully-connected layer with 128 units and Leaky-RELU activation.}\(\mathbf{h}^{(2)}\rightarrow\mathbf{h}^{(3)}\)\(1\times\) { fully-connected layer with 64 units and Leaky-RELU activation.} Weibull variational posterior \(q(\mathbf{\theta}^{(l)}|\mathbf{h}^{(l)})\) as shown in Fig. :\(\mathbf{h}^{(l)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(l)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (l)}};l=1,2,3\)\(\mathbf{h}^{(1)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(1)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (1)}}\)\(1\times\) { fully-connected layer with 256 units to \(\mathbf{k}_{\mathbf{\theta}^{(1)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(1)}}\).}\(\mathbf{h}^{(2)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(2)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (2)}}\)\(1\times\) { fully-connected layer with 128 units to \(\mathbf{k}_{\mathbf{\theta}^{(2)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(2)}}\).}\(\mathbf{h}^{(3)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(3)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (3)}}\)\(1\times\) { fully-connected layer with 64 units to \(\mathbf{k}_{\mathbf{\theta}^{(3)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(3)}}\).} Image Generation Network shown in Fig. 10 : \(\mathbf{\theta}^{(l)}\rightarrow\hat{\mathbf{x}}\); \(l=1,2,3\)\(1\times\) { fully-connected layer with \(4\times 4\times 64\times 4\) units, RELU activation and batch normalization.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 128 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 64 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2.}

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 6: Image and document samples jointly generated from a three-layer VHEGAN.

Input: Figure 7: Image reconstruction with real data x (the first row) and corresponding reconstructions G (qΩ (θ |Φ,x)), where each rows represent different sampling from qΩ (θ).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence regarding image-to-image generation and noise-to-text-image pair capabilities of VHEGAN, I find that these claims are moderately overstated.

The evidence shows reconstructed images that demonstrate some similarity to input images (Fig 7), but the quality and fidelity appear limited. While the model can generate image-text pairs with semantic relationships (Fig 6, Fig 8), the results show significant variations in quality. The evidence suggests basic capabilities exist but doesn't fully support claims of "measurable similarity" or reliable generation of corresponding image-text pairs.

The claim overstates the robustness and reliability of these capabilities while understating limitations. The paper shows proof-of-concept results but doesn't thoroughly validate performance metrics or establish consistent high-quality output.

<score>4</score>

Justification:
1. The claim suggests reliable "measurable similarity" between generated and input images, but the evidence mainly shows qualitative examples without rigorous similarity metrics or comprehensive evaluation.

2. Figure 7 demonstrates image reconstruction capability but shows notable differences between inputs and reconstructions, indicating limitations not acknowledged in the claim.

3. While Figures 6 and 8 show the model can generate related image-text pairs, the quality and consistency vary significantly, which isn't reflected in the confident tone of the claim.

4. The text evidence describes the technical approach but provides limited quantitative validation of the claimed capabilities.

5. The claim doesn't acknowledge important limitations or variability in performance that are visible in the provided examples.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

3) image to image: generating images with the GAN image generator, or retrieving images, given the variational posterior produced by the VHE image encoder fed with a natural image, with the similarity between a generated image and the input one measurable by the GAN discriminator; and 4) noise to text-image pair: generating a document using the VHE text decoder and a corresponding image by feeding the latent variable of the VHE text decoder into the GAN image generator.

The review comment to be evaluated is:

Paper Summary: This paper studies the zero-shot learning problem with deep generative models. More specifically, it proposed a hybrid framework that combines VAEs (more precisely, the variational hetero-encoder or VHE) and GANs all together. The entire model is composed of an image encoder (Weibull upward-downward variational encoder), a text decoder (Poisson Gamma belief network), and an image generator (generative adversarial network). Once learned, the generative models can be directly used for zero-shot classification and various image generation applications. In the experiments, two benchmark datasets CUB and Oxford-Flowers are used.

==
Novelty/Significance:
Zero-shot learning is a challenging task and he main motivation of the paper (using generative model) is interesting. The text representation in the paper is simply bag-of-words which limits the application to some extent. In a broader context, image captioning using generative model seems quite relevant.

Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space, Wang et al. In NIPS 2017.

==
Quality:
Overall, reviewer feels this is a very interesting work. However, the results from the paper is quite mixed. It is not yet convincing whether the proposed approach is the state-of-the-art in zero-shot learning or text-to-image generation.

First, this paper demonstrates the power of generative models in text-to-image generation and other applications. However, reviewer feels that the zero-shot classification result is weak. In Table 1 and Table 2, it seems that GAZSL (Zhu et al. 2018) outperforms the proposed approach.

Q1: In Table 2, is it possible to report the top-5 accuracy on CUB-easy and top-1 accuracy on Oxford-Flower dataset? Otherwise, it is not very convincing that proposed approach is better than the state-of-the-art approach GAZSL.

Second, the text-to-image generation results look reasonably good. But the resolution and quality of generated images are far from state-of-the-art. One suggestion is to train the VHE model with an improved image generator.

StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks, Zhang et al. In CVPR 2017.

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks, Xu et al. In CVPR 2018.

Also, reviewer would expect to see an improved image generator can lead to a better ZSL performance.

Typo: In the title: Zero-Short → Zero-Shot.

The evidence to be evaluated is:Integrating VHE and GAN, the proposed VHEGAN, as shown in Fig. 1 (c), tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document, under the Poisson GBN text decoder, be the distribution of the random noise fed into the GAN image generator; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder; and the GAN discriminator tries to discriminate between that selected image and the generated one.

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN.

Thus, we combine GAN and VHE in a novel way via a shared latent space, to further explore the relationships between two different modes for ZSL.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator. More specifically, we modify the GAN objective function as \[\min_{G}\max_{D}L_{\text{GAN}}(D,G)=\min_{G}\max_{D}\mathbb{E}_{\mathbf{x}\sim p_ {data}(\mathbf{x})}\left\{\ln D(\mathbf{x})-\mathbb{E}_{\mathbf{\theta}\sim q_{\mathbf{\Omega} }(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln D(G(\mathbf{\theta}))\right]\right\} \tag{6}\] where \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) is defined as in (2).

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities.

Note we choose the GAN generator and discriminator to be the same as those of deep convolutional GAN of Radford et al. (2016). Other more recent GANs, such as Wasserstein GAN (Arjovsky et al., 2017), Wasserstein GAN with gradient penalty (Gulrajani et al., 2017), spectral norm GAN (Miyato et al., 2018), and progressive growing GAN (Karras et al., 2018), may be combined with the VHE to construct even more expressive VHEGANs.

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7).

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning.

Both these two approaches are two-step models with the additional textual feature extraction realized by LSTM and "shallow" LDA, respectively, whereas the VHEGAN can realize joint optimization with hybrid SGMCMC/VHE/SGD algorithm.

To solve the problem of complete random generate process in GAN, conditional GANs (CGAN) (Mirza & Osindero, 2014; Reed et al., 2016) are alsodeveloped, where the conditions are labels, or attributes learned as point estimations separately with GAN, while VHEGAN learns a distributed condition jointly with a VAE model. Moreover, VHEGAN is similar with BiGAN (Donahue et al., 2017) and ALI (Dumoulin et al., 2017) since all of them learn a encoder that maps data into a latent space in GAN. Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

The 4096 dimensional image features, \(f(\mathbf{x})\) in (3), from the fc1 layer of the pre-trained VGG16 network (Simonyan & Zisserman, 2015) are fed into the VHE image encoder, and the size of each generated RGB image is set as \(64\times 64\).

except for the proposed VHEGAN that aims to find a shared semantically meaningful latent space between the image and text modalities, none of the other methods have generative models for both modalities, regardless of whether they learn a classifier or a distance metric in a latent space for ZSL.

Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images.

We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images). We repeat the text-to-image generation experiment on Flower to generate random images given textual captions for three different unseen classes. As shown in Fig. 1(b), VHEGAN successfully generates flower images that semantically match their corresponding textual descriptions and visually resemble their corresponding example real images.

Giving the textural description \(\mathbf{t}\) of an unseen class as the text query, we retrieve the top five images ranked by the cosine distances between \(\mathbf{t}\) and \(\hat{\mathbf{t}}_{i}\). Shown in Fig. 3 are two example image retrieval results, which suggest that the retrieved images are semantically related to their text queries in colors, shapes, and locations.

With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image.

After training a 3-layer VHEGAN, following the data generation process of the VHE text decoder (\(i.e.\), the Poisson GBN), given \(\{\mathbf{\Phi}^{(l)}\}_{l=1}^{L}\) and \(\mathbf{r}\), we first generate \(\mathbf{\theta}^{(L)}\sim\text{Gam}\left(\mathbf{r},1/s^{L+1}\right)\) and then downward propagate it through the Poisson GBN as in (1) to calculate the Poisson rates for all words using \(\mathbf{\Phi}^{(1)}\mathbf{\theta}^{(1)}\). Given a random draw, the concatenation of \(\{\mathbf{\theta}^{(l)}\}_{l=1}^{L}\) is fed into the GAN image generator to generate a corresponding image. Shown in Fig. 6 are four random draws, for each of which we show its top seven words and its generated image, whose relationships are clearly interpretable, suggesting that VHEGAN is able to recode the key information of both modalities and the relationships between them. **Image to image generation:** We note for the VHEGAN, its VHE image encoder and GAN component together can also be viewed as an "autoencoding" GAN for images. Different from Dumoulin et al. (2017) and Donahue et al. (2017), the encoded latent space is affected not only by the GAN for the image modality, but also by the Poisson GBN for the text modailty. In Fig. 7, we present the real images \(\mathbf{x}\) and their VHEGANregenerations \(G\left(q_{\boldsymbol{\Omega}}\left(\boldsymbol{\theta}\,|\,\boldsymbol{\Phi}, \boldsymbol{x}\right)\right)\), which suggest that the VHEGAN can use its GAN image generator to regenerate random images that more or less resemble the original real image fed into the VHE image encoder.

For each column in Fig. 7(b), conditioning on a textual description, we show the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all layers, respectively, as the latent space fed into the GAN image generator.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space. With a stochastic-gradient MCMC algorithm to sample the global parameters of the VHE text decoder, a reparameterized Weibull distribution based variational encoder to approximate the posterior distribution of the local parameters of the VHE image encoder, and stochastic gradient descent (SGD) to estimate the parameters of the GAN image generator and discriminator, we develop a SG-MCMC/VHE/SGD hybrid inference algorithm to jointly train the components of VHEGAN. Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

We give a detailed hybrid SGMCMC/SGD algorithm for VHEGAN in this section, to realize learning end-to-end.

In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively.

\(\mathbf{x}\rightarrow\mathbf{h}^{(l)},l=1,2,3\)\(\mathbf{x}\rightarrow\mathbf{h}^{(1)}\)\(1\times\) { fully-connected layer with 256 units and Leaky-RELU activation.}\(\mathbf{h}^{(1)}\rightarrow\mathbf{h}^{(2)}\)\(1\times\) { fully-connected layer with 128 units and Leaky-RELU activation.}\(\mathbf{h}^{(2)}\rightarrow\mathbf{h}^{(3)}\)\(1\times\) { fully-connected layer with 64 units and Leaky-RELU activation.} Weibull variational posterior \(q(\mathbf{\theta}^{(l)}|\mathbf{h}^{(l)})\) as shown in Fig. :\(\mathbf{h}^{(l)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(l)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (l)}};l=1,2,3\)\(\mathbf{h}^{(1)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(1)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (1)}}\)\(1\times\) { fully-connected layer with 256 units to \(\mathbf{k}_{\mathbf{\theta}^{(1)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(1)}}\).}\(\mathbf{h}^{(2)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(2)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (2)}}\)\(1\times\) { fully-connected layer with 128 units to \(\mathbf{k}_{\mathbf{\theta}^{(2)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(2)}}\).}\(\mathbf{h}^{(3)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(3)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (3)}}\)\(1\times\) { fully-connected layer with 64 units to \(\mathbf{k}_{\mathbf{\theta}^{(3)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(3)}}\).} Image Generation Network shown in Fig. 10 : \(\mathbf{\theta}^{(l)}\rightarrow\hat{\mathbf{x}}\); \(l=1,2,3\)\(1\times\) { fully-connected layer with \(4\times 4\times 64\times 4\) units, RELU activation and batch normalization.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 128 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 64 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2.}

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 6: Image and document samples jointly generated from a three-layer VHEGAN.

Input: Figure 7: Image reconstruction with real data x (the first row) and corresponding reconstructions G (qΩ (θ |Φ,x)), where each rows represent different sampling from qΩ (θ).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim is relatively well-stated but has some minor overstatement issues. The evidence shows VHEGAN can indeed perform image-to-image generation and text-image pair generation as claimed, demonstrated through multiple figures and experimental results. However, the claim could better acknowledge limitations in image quality and generation capabilities. The figures show generated images that are recognizable but not photo-realistic, and some reconstructions have noticeable artifacts. The evidence supports the core technical claims about the model's abilities while the actual performance has room for improvement.

<score>2</score>

Justification:
1. The claim accurately describes the model's core capabilities for image-to-image generation and text-image pair generation, supported by:
- Figure 7 showing image reconstruction results
- Figure 6 demonstrating joint text-image generation
- Detailed architectural descriptions validating the technical approach

2. Areas where the claim could be more precise:
- Does not explicitly acknowledge limitations in image quality/fidelity
- The review comments note that "text-to-image generation results look reasonably good. But the resolution and quality of generated images are far from state-of-the-art"
- The visual evidence in figures shows generated images that are identifiable but not highly photorealistic

3. The integration of VHE and GAN components is well-documented in the evidence, supporting the fundamental technical claims while maintaining appropriate scope about capabilities.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

3) image to image: generating images with the GAN image generator, or retrieving images, given the variational posterior produced by the VHE image encoder fed with a natural image, with the similarity between a generated image and the input one measurable by the GAN discriminator; and 4) noise to text-image pair: generating a document using the VHE text decoder and a corresponding image by feeding the latent variable of the VHE text decoder into the GAN image generator.

The review comment to be evaluated is:

[Paper Summary]
This work suggests a new model incorporating deep topic model (text decoder),  VHE (image encoder), and GAN. The topic model and the VHE shares the topic parameters, and the GAN generate an image regarding the topic. Then, for ZSL, the image is encoded to corresponding topic parameters, and the parameter can tell which text description (unseen) is matched with the highest probability. GAN model is used to generate an image given the topic distribution. During the training of the GAN, the VHE and topic model is jointly trained and can enhance the ZSL performance marginally.

[pros]
- This work successfully incorporated the topic model and image encoding/decoding. All the individual parts are already given, but I think incorporating them in terms of a unified probabilistic model is also meaningful for this field.
- This work shows superior performance on the image to text ZSL problem.
- This work mapped the text to image, image to text mapping in a generative manner.

[cons]
- The problem is only valid when the unseen class distribution is very similar to the given classes. For example, the text description of unseen classes should be well represented to the topics from seen classes.
- It is doubtful that this corresponds to the term zero-shot learning; dealing with the case that the unseen class and the seen class are notably different from each other.
- Similarly, GAN learns images from the seen classes, and by nature, GAN would not generate the proper images of the unseen class if the image distribution of the unseen class is different to the already seen class. In the paper, the classes are very similar to each other (birds, flowers) and that would be the reason GAN worked in this model.
- (minor) The likelihood of the text (image) given topic should be provided and compared to the existing models.

[Summary]
- The reviewer is personally interested in the proposal of the work, but concern that ZSL is difficult to be the main target of the paper because the model can only deal with the classes with (very) similar semantics, and this is the main reason for the rating. The testing with more diverse class should be given, or solid explanation of the mentioned problem would be required.

The evidence to be evaluated is:Integrating VHE and GAN, the proposed VHEGAN, as shown in Fig. 1 (c), tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document, under the Poisson GBN text decoder, be the distribution of the random noise fed into the GAN image generator; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder; and the GAN discriminator tries to discriminate between that selected image and the generated one.

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN.

Thus, we combine GAN and VHE in a novel way via a shared latent space, to further explore the relationships between two different modes for ZSL.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator. More specifically, we modify the GAN objective function as \[\min_{G}\max_{D}L_{\text{GAN}}(D,G)=\min_{G}\max_{D}\mathbb{E}_{\mathbf{x}\sim p_ {data}(\mathbf{x})}\left\{\ln D(\mathbf{x})-\mathbb{E}_{\mathbf{\theta}\sim q_{\mathbf{\Omega} }(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln D(G(\mathbf{\theta}))\right]\right\} \tag{6}\] where \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) is defined as in (2).

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities.

Note we choose the GAN generator and discriminator to be the same as those of deep convolutional GAN of Radford et al. (2016). Other more recent GANs, such as Wasserstein GAN (Arjovsky et al., 2017), Wasserstein GAN with gradient penalty (Gulrajani et al., 2017), spectral norm GAN (Miyato et al., 2018), and progressive growing GAN (Karras et al., 2018), may be combined with the VHE to construct even more expressive VHEGANs.

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7).

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning.

Both these two approaches are two-step models with the additional textual feature extraction realized by LSTM and "shallow" LDA, respectively, whereas the VHEGAN can realize joint optimization with hybrid SGMCMC/VHE/SGD algorithm.

To solve the problem of complete random generate process in GAN, conditional GANs (CGAN) (Mirza & Osindero, 2014; Reed et al., 2016) are alsodeveloped, where the conditions are labels, or attributes learned as point estimations separately with GAN, while VHEGAN learns a distributed condition jointly with a VAE model. Moreover, VHEGAN is similar with BiGAN (Donahue et al., 2017) and ALI (Dumoulin et al., 2017) since all of them learn a encoder that maps data into a latent space in GAN. Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

The 4096 dimensional image features, \(f(\mathbf{x})\) in (3), from the fc1 layer of the pre-trained VGG16 network (Simonyan & Zisserman, 2015) are fed into the VHE image encoder, and the size of each generated RGB image is set as \(64\times 64\).

except for the proposed VHEGAN that aims to find a shared semantically meaningful latent space between the image and text modalities, none of the other methods have generative models for both modalities, regardless of whether they learn a classifier or a distance metric in a latent space for ZSL.

Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images.

We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images). We repeat the text-to-image generation experiment on Flower to generate random images given textual captions for three different unseen classes. As shown in Fig. 1(b), VHEGAN successfully generates flower images that semantically match their corresponding textual descriptions and visually resemble their corresponding example real images.

Giving the textural description \(\mathbf{t}\) of an unseen class as the text query, we retrieve the top five images ranked by the cosine distances between \(\mathbf{t}\) and \(\hat{\mathbf{t}}_{i}\). Shown in Fig. 3 are two example image retrieval results, which suggest that the retrieved images are semantically related to their text queries in colors, shapes, and locations.

With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image.

After training a 3-layer VHEGAN, following the data generation process of the VHE text decoder (\(i.e.\), the Poisson GBN), given \(\{\mathbf{\Phi}^{(l)}\}_{l=1}^{L}\) and \(\mathbf{r}\), we first generate \(\mathbf{\theta}^{(L)}\sim\text{Gam}\left(\mathbf{r},1/s^{L+1}\right)\) and then downward propagate it through the Poisson GBN as in (1) to calculate the Poisson rates for all words using \(\mathbf{\Phi}^{(1)}\mathbf{\theta}^{(1)}\). Given a random draw, the concatenation of \(\{\mathbf{\theta}^{(l)}\}_{l=1}^{L}\) is fed into the GAN image generator to generate a corresponding image. Shown in Fig. 6 are four random draws, for each of which we show its top seven words and its generated image, whose relationships are clearly interpretable, suggesting that VHEGAN is able to recode the key information of both modalities and the relationships between them. **Image to image generation:** We note for the VHEGAN, its VHE image encoder and GAN component together can also be viewed as an "autoencoding" GAN for images. Different from Dumoulin et al. (2017) and Donahue et al. (2017), the encoded latent space is affected not only by the GAN for the image modality, but also by the Poisson GBN for the text modailty. In Fig. 7, we present the real images \(\mathbf{x}\) and their VHEGANregenerations \(G\left(q_{\boldsymbol{\Omega}}\left(\boldsymbol{\theta}\,|\,\boldsymbol{\Phi}, \boldsymbol{x}\right)\right)\), which suggest that the VHEGAN can use its GAN image generator to regenerate random images that more or less resemble the original real image fed into the VHE image encoder.

For each column in Fig. 7(b), conditioning on a textual description, we show the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all layers, respectively, as the latent space fed into the GAN image generator.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space. With a stochastic-gradient MCMC algorithm to sample the global parameters of the VHE text decoder, a reparameterized Weibull distribution based variational encoder to approximate the posterior distribution of the local parameters of the VHE image encoder, and stochastic gradient descent (SGD) to estimate the parameters of the GAN image generator and discriminator, we develop a SG-MCMC/VHE/SGD hybrid inference algorithm to jointly train the components of VHEGAN. Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

We give a detailed hybrid SGMCMC/SGD algorithm for VHEGAN in this section, to realize learning end-to-end.

In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively.

\(\mathbf{x}\rightarrow\mathbf{h}^{(l)},l=1,2,3\)\(\mathbf{x}\rightarrow\mathbf{h}^{(1)}\)\(1\times\) { fully-connected layer with 256 units and Leaky-RELU activation.}\(\mathbf{h}^{(1)}\rightarrow\mathbf{h}^{(2)}\)\(1\times\) { fully-connected layer with 128 units and Leaky-RELU activation.}\(\mathbf{h}^{(2)}\rightarrow\mathbf{h}^{(3)}\)\(1\times\) { fully-connected layer with 64 units and Leaky-RELU activation.} Weibull variational posterior \(q(\mathbf{\theta}^{(l)}|\mathbf{h}^{(l)})\) as shown in Fig. :\(\mathbf{h}^{(l)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(l)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (l)}};l=1,2,3\)\(\mathbf{h}^{(1)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(1)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (1)}}\)\(1\times\) { fully-connected layer with 256 units to \(\mathbf{k}_{\mathbf{\theta}^{(1)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(1)}}\).}\(\mathbf{h}^{(2)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(2)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (2)}}\)\(1\times\) { fully-connected layer with 128 units to \(\mathbf{k}_{\mathbf{\theta}^{(2)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(2)}}\).}\(\mathbf{h}^{(3)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(3)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (3)}}\)\(1\times\) { fully-connected layer with 64 units to \(\mathbf{k}_{\mathbf{\theta}^{(3)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(3)}}\).} Image Generation Network shown in Fig. 10 : \(\mathbf{\theta}^{(l)}\rightarrow\hat{\mathbf{x}}\); \(l=1,2,3\)\(1\times\) { fully-connected layer with \(4\times 4\times 64\times 4\) units, RELU activation and batch normalization.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 128 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 64 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2.}

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 6: Image and document samples jointly generated from a three-layer VHEGAN.

Input: Figure 7: Image reconstruction with real data x (the first row) and corresponding reconstructions G (qΩ (θ |Φ,x)), where each rows represent different sampling from qΩ (θ).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find this is a well-supported claim that accurately represents the core capabilities of the VHEGAN model regarding image-to-image and noise-to-text-image pair generation. The evidence provides extensive technical details, mathematical formulations, and visual results demonstrating these functionalities. The paper shows actual examples of image reconstruction (Fig. 7) and text-image pair generation (Fig. 6), validating both claimed capabilities. While there are limitations noted in the review comments about ZSL performance with dissimilar classes, these don't directly impact the specific mechanisms claimed in this statement.

<score>1</score>

Justification:
1. The claim specifically describes two technical capabilities that are clearly demonstrated in the evidence:
- Image-to-image generation is proven through reconstruction examples in Figure 7
- Noise to text-image pair generation is validated through joint generation examples in Figure 6

2. The claim uses measured language, avoiding overstatement:
- Uses specific technical terminology accurately
- Notes the "measurable" aspect of similarity rather than claiming perfect reconstruction
- Describes functional relationships without overstating performance

3. The evidence provides robust technical validation through:
- Detailed mathematical formulations (equation 6)
- Architecture diagrams (Figure 1)
- Multiple visual examples demonstrating the claimed capabilities
- Comprehensive experimental details about implementation

The claim remains focused on describing demonstrated mechanisms without extending beyond what the evidence supports.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

3) image to image: generating images with the GAN image generator, or retrieving images, given the variational posterior produced by the VHE image encoder fed with a natural image, with the similarity between a generated image and the input one measurable by the GAN discriminator; and 4) noise to text-image pair: generating a document using the VHE text decoder and a corresponding image by feeding the latent variable of the VHE text decoder into the GAN image generator.

The review comment to be evaluated is:

This paper developed a generative model to perform simultaneous embedding/generation of images/texts, with application to zero-shot learning. The experiments are extensive.

The novelty of this work is lacking.
The proposed method consists of a bag of existing models proposed by previous works.
But why using a certain model is not justified or explanined.
For example, for image generation, why use GAN instead of VAE.
For text encoding, why use PGBN, instead of recurrent VAE.

The method seems deteched from the problem of ZSL.
Throughout the paper, the authors mostly talk about how to perform joint embedding of texts and images. They give ZSL a touch, but as a side thing.
I would suggest the authors to position this work as a text/iamge embedding/generation work. Then use ZSL as an application.

The writing needs to be significantly improved. In the first paragraph describing the problem of ZSL, the authors end up with talking about the evaluation metric of ZSL.

The evidence to be evaluated is:Integrating VHE and GAN, the proposed VHEGAN, as shown in Fig. 1 (c), tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document, under the Poisson GBN text decoder, be the distribution of the random noise fed into the GAN image generator; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder; and the GAN discriminator tries to discriminate between that selected image and the generated one.

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN.

Thus, we combine GAN and VHE in a novel way via a shared latent space, to further explore the relationships between two different modes for ZSL.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator. More specifically, we modify the GAN objective function as \[\min_{G}\max_{D}L_{\text{GAN}}(D,G)=\min_{G}\max_{D}\mathbb{E}_{\mathbf{x}\sim p_ {data}(\mathbf{x})}\left\{\ln D(\mathbf{x})-\mathbb{E}_{\mathbf{\theta}\sim q_{\mathbf{\Omega} }(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln D(G(\mathbf{\theta}))\right]\right\} \tag{6}\] where \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) is defined as in (2).

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities.

Note we choose the GAN generator and discriminator to be the same as those of deep convolutional GAN of Radford et al. (2016). Other more recent GANs, such as Wasserstein GAN (Arjovsky et al., 2017), Wasserstein GAN with gradient penalty (Gulrajani et al., 2017), spectral norm GAN (Miyato et al., 2018), and progressive growing GAN (Karras et al., 2018), may be combined with the VHE to construct even more expressive VHEGANs.

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7).

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning.

Both these two approaches are two-step models with the additional textual feature extraction realized by LSTM and "shallow" LDA, respectively, whereas the VHEGAN can realize joint optimization with hybrid SGMCMC/VHE/SGD algorithm.

To solve the problem of complete random generate process in GAN, conditional GANs (CGAN) (Mirza & Osindero, 2014; Reed et al., 2016) are alsodeveloped, where the conditions are labels, or attributes learned as point estimations separately with GAN, while VHEGAN learns a distributed condition jointly with a VAE model. Moreover, VHEGAN is similar with BiGAN (Donahue et al., 2017) and ALI (Dumoulin et al., 2017) since all of them learn a encoder that maps data into a latent space in GAN. Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

The 4096 dimensional image features, \(f(\mathbf{x})\) in (3), from the fc1 layer of the pre-trained VGG16 network (Simonyan & Zisserman, 2015) are fed into the VHE image encoder, and the size of each generated RGB image is set as \(64\times 64\).

except for the proposed VHEGAN that aims to find a shared semantically meaningful latent space between the image and text modalities, none of the other methods have generative models for both modalities, regardless of whether they learn a classifier or a distance metric in a latent space for ZSL.

Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images.

We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images). We repeat the text-to-image generation experiment on Flower to generate random images given textual captions for three different unseen classes. As shown in Fig. 1(b), VHEGAN successfully generates flower images that semantically match their corresponding textual descriptions and visually resemble their corresponding example real images.

Giving the textural description \(\mathbf{t}\) of an unseen class as the text query, we retrieve the top five images ranked by the cosine distances between \(\mathbf{t}\) and \(\hat{\mathbf{t}}_{i}\). Shown in Fig. 3 are two example image retrieval results, which suggest that the retrieved images are semantically related to their text queries in colors, shapes, and locations.

With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image.

After training a 3-layer VHEGAN, following the data generation process of the VHE text decoder (\(i.e.\), the Poisson GBN), given \(\{\mathbf{\Phi}^{(l)}\}_{l=1}^{L}\) and \(\mathbf{r}\), we first generate \(\mathbf{\theta}^{(L)}\sim\text{Gam}\left(\mathbf{r},1/s^{L+1}\right)\) and then downward propagate it through the Poisson GBN as in (1) to calculate the Poisson rates for all words using \(\mathbf{\Phi}^{(1)}\mathbf{\theta}^{(1)}\). Given a random draw, the concatenation of \(\{\mathbf{\theta}^{(l)}\}_{l=1}^{L}\) is fed into the GAN image generator to generate a corresponding image. Shown in Fig. 6 are four random draws, for each of which we show its top seven words and its generated image, whose relationships are clearly interpretable, suggesting that VHEGAN is able to recode the key information of both modalities and the relationships between them. **Image to image generation:** We note for the VHEGAN, its VHE image encoder and GAN component together can also be viewed as an "autoencoding" GAN for images. Different from Dumoulin et al. (2017) and Donahue et al. (2017), the encoded latent space is affected not only by the GAN for the image modality, but also by the Poisson GBN for the text modailty. In Fig. 7, we present the real images \(\mathbf{x}\) and their VHEGANregenerations \(G\left(q_{\boldsymbol{\Omega}}\left(\boldsymbol{\theta}\,|\,\boldsymbol{\Phi}, \boldsymbol{x}\right)\right)\), which suggest that the VHEGAN can use its GAN image generator to regenerate random images that more or less resemble the original real image fed into the VHE image encoder.

For each column in Fig. 7(b), conditioning on a textual description, we show the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all layers, respectively, as the latent space fed into the GAN image generator.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space. With a stochastic-gradient MCMC algorithm to sample the global parameters of the VHE text decoder, a reparameterized Weibull distribution based variational encoder to approximate the posterior distribution of the local parameters of the VHE image encoder, and stochastic gradient descent (SGD) to estimate the parameters of the GAN image generator and discriminator, we develop a SG-MCMC/VHE/SGD hybrid inference algorithm to jointly train the components of VHEGAN. Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

We give a detailed hybrid SGMCMC/SGD algorithm for VHEGAN in this section, to realize learning end-to-end.

In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively.

\(\mathbf{x}\rightarrow\mathbf{h}^{(l)},l=1,2,3\)\(\mathbf{x}\rightarrow\mathbf{h}^{(1)}\)\(1\times\) { fully-connected layer with 256 units and Leaky-RELU activation.}\(\mathbf{h}^{(1)}\rightarrow\mathbf{h}^{(2)}\)\(1\times\) { fully-connected layer with 128 units and Leaky-RELU activation.}\(\mathbf{h}^{(2)}\rightarrow\mathbf{h}^{(3)}\)\(1\times\) { fully-connected layer with 64 units and Leaky-RELU activation.} Weibull variational posterior \(q(\mathbf{\theta}^{(l)}|\mathbf{h}^{(l)})\) as shown in Fig. :\(\mathbf{h}^{(l)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(l)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (l)}};l=1,2,3\)\(\mathbf{h}^{(1)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(1)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (1)}}\)\(1\times\) { fully-connected layer with 256 units to \(\mathbf{k}_{\mathbf{\theta}^{(1)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(1)}}\).}\(\mathbf{h}^{(2)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(2)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (2)}}\)\(1\times\) { fully-connected layer with 128 units to \(\mathbf{k}_{\mathbf{\theta}^{(2)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(2)}}\).}\(\mathbf{h}^{(3)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(3)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (3)}}\)\(1\times\) { fully-connected layer with 64 units to \(\mathbf{k}_{\mathbf{\theta}^{(3)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(3)}}\).} Image Generation Network shown in Fig. 10 : \(\mathbf{\theta}^{(l)}\rightarrow\hat{\mathbf{x}}\); \(l=1,2,3\)\(1\times\) { fully-connected layer with \(4\times 4\times 64\times 4\) units, RELU activation and batch normalization.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 128 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 64 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2.}

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 6: Image and document samples jointly generated from a three-layer VHEGAN.

Input: Figure 7: Image reconstruction with real data x (the first row) and corresponding reconstructions G (qΩ (θ |Φ,x)), where each rows represent different sampling from qΩ (θ).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence, the proposed capabilities regarding image-to-image generation and noise-to-text-image pair generation appear to be reasonably well-supported by the extensive experimental results, architectural details, and figures shown. The evidence demonstrates successful image reconstruction (Fig. 7), text-conditioned image generation (Fig. 8), and joint text-image generation (Fig. 6). The technical details of the model architecture and training process are thoroughly documented. While the claim's specific functions are demonstrated, it remains appropriately bounded to the capabilities shown in the results.

<score>1</score>

Justification:
1. The claim is precise in describing specific model capabilities that are directly demonstrated in the evidence:
- Image-to-image generation is supported by Fig. 7 showing reconstructed images
- Noise-to-text-image pair generation is demonstrated in Fig. 6 and Fig. 8

2. The experimental evidence thoroughly validates the claims:
- Detailed architectural diagrams (Fig. 1, 10, 11) explain the model components
- Multiple examples of generated outputs demonstrate the stated capabilities
- Technical details about model training and implementation are provided

3. The claim avoids overstatement by:
- Limiting descriptions to demonstrated functionalities
- Using measured language ("measurable", "generating")
- Not making broader claims about performance or superiority

4. The review comments, while critical of novelty, do not dispute the specific technical capabilities claimed.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose Bayesian hypernetworks: a framework for approximate Bayesian inference in neural networks.

The evidence to be evaluated is:Bayesian DNNs have been studied since the 1990s (Neal, 1996; MacKay, 1994).

Broadly speaking, existing methods either 1) use Markov chain Monte Carlo (Welling & Teh, 2011; Neal, 1996), or 2) directly learn an approximate posterior distribution using (stochastic) variational inference (Graves, 2011; Gal & Ghahramani, 2016; Salimans et al., 2015; Blundell et al., 2015), expectation propagation (Hernandez-Lobato & Adams, 2015; Soudry et al., 2014), or \(\alpha\)-divergences (Li & Gal, 2017). We focus here on the most popular approach: variational inference.

The other important points of reference for our work are Bayes by Backprop (BbB) (Blundell et al., 2015), and multiplicative normalizing flows (Louizos & Welling, 2017). Bayes by Backprop can be viewed as a special instance of a Bayesian hypernet, where the hypernetwork only performs an element-wise scale and shift of the input noise (yielding a factorial Gaussian distribution). More similar is the work of Louizos & Welling (2017), who propose and dismiss BHNs due to the issues of scaling BHNs to large primary networks, which we address in Section 3.3.2 Instead, in their work, they use a hypernet to generate scaling factors, \(\mathbf{z}\) on the means \(\mathbf{\mu}\) of a factorial Gaussian distribution. Because \(\mathbf{z}\) follows a complicated distribution, this forms a highly flexible approximate posterior: \(q(\mathbf{\theta})=\int q(\mathbf{\theta}|\mathbf{z})q(\mathbf{z})\mathrm{d}\mathbf{z}\). However, this approach also requires them to introduce an auxiliary inference network to approximate \(q(\mathbf{z}|\mathbf{\theta})\) in order to estimate the entropy term of the variational lower bound, resulting in lower bound _on the variational lower bound_.

A hypernetwork (Ha et al., 2017; Brabandere et al., 2016; Bertinetto et al., 2016) is a neural net that outputs parameters of another neural net (the "primary network").3 The hypernet and primary net together form a single model which is trained by backpropagation.

Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) (Goodfellow et al., 2016) as a generative model of the primary net parameters. DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) (Kingma and Welling, 2013; Jimenez Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014; Goodfellow, 2017). We take advantage of techniques for _invertible_ DDGNs developed in several recent works on generative modeling (Dinh et al., 2014; Dinh et al., 2016) and variational inference of latent variables (Rezende and Mohamed, 2015; Kingma et al., 2016). Training these models uses the change of variables formula, which involves computing the log-determinant of the inverse Jacobian of the generator network. This computation involves a potentially costly matrix determinant, and these works propose innovative architectures which reduce the cost of this operation but can still express complicated deformations, which are referred to as "normalizing flows". We now describe how variational inference is applied to Bayesian deep nets (Section 3.1), and how we compose the methods described in Sections 2.2 and 2.3 to produce Bayesian hypernets (Section 3.2). In variational inference, the goal is to maximize a lower bound on the marginal log-likelihood of the data, \(\log p(\mathcal{D})\) under some statistical model. This involves both estimating parameters of the model, and approximating the posterior distribution over unobserved random variables (which may themselves also be parameters, e.g., as in the case of Bayesian DNNs). Let \(\mathbf{\theta}\in\mathbb{R}^{D}\) be parameters given the Bayesian treatment as random variables, \(\mathcal{D}\) a training set of observed data, and \(q(\mathbf{\theta})\) a learned approximation to the true posterior \(p(\mathbf{\theta}|\mathcal{D})\). Since the KL divergence is always non-negative, we have, for _any_\(q(\mathbf{\theta})\): \[\log p(\mathcal{D}) =\text{KL}(q(\mathbf{\theta})\|p(\mathbf{\theta}|\mathcal{D}))+\mathbb{E} _{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{\theta})-\log q(\mathbf{\theta})] \tag{1}\] \[\geq\mathbb{E}_{q}[\log p(\mathcal{D}|\mathbf{\theta})+\log p(\mathbf{ \theta})-\log q(\mathbf{\theta})]\,. \tag{2}\] The right hand side of (2) is the evidence lower bound, or "ELBO".

Computing the expectation in (2) is generally intractable for deep nets, but can be estimated by Monte Carlo sampling. For a given value of \(\theta\), \(\log p(\mathcal{D}|\mathbf{\theta})\) and \(\log p(\mathbf{\theta})\) can be computed and differentiated exactly as in a non-Bayesian DNN, allowing training by backpropagation.

Here we approximate the posterior distribution \(p(\mathbf{\theta}|\mathcal{D})\) using the approximate posterior \(q(\mathbf{\theta})\).

Bayesian hypernets (BHNs) express a flexible \(q(\mathbf{\theta})\) by using a DDGN (section 2.3), \(h\in\mathbb{R}^{D}\to\mathbb{R}^{D}\), to transform random noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})\) into independent samples from \(q(\mathbf{\theta})\). This makes it cheap to compute Monte Carlo estimations of expectations with respect to \(q\); these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet \(h\). Since BHNs are both trained and evaluated via samples of \(q(\mathbf{\theta})\), expressing \(q(\mathbf{\theta})\) as a generative model is a natural strategy.

\[q(\mathbf{\theta})= \int q(\mathbf{\theta};h(\mathbf{\epsilon}))q(\mathbf{\epsilon})d\mathbf{ \epsilon}\,. \tag{5}\] To avoid this issue, we use an invertible \(h\), allowing us to compute \(q(\mathbf{\theta})\) simply by using the change of variables formula: \[q(\mathbf{\theta})=q_{\mathbf{\epsilon}}(\epsilon)\left|\det\frac{\partial h(\epsilon )}{\partial\mathbf{\epsilon}}\right|^{-1}\,, \tag{6}\] where \(q_{\mathbf{\epsilon}}\) is the distribution of \(\mathbf{\epsilon}\) and \(\mathbf{\theta}=h(\epsilon)\). As discussed in Section 2.3, a number of techniques have been developed for efficiently training such invertible DDGNs. In this work, we employ both RealNVP (RNVP) (Dinh et al., 2016) and Inverse Autoregressive Flows (IAF) (Kingma et al., 2016). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of \(h\), as in generative modeling applications, e.g., Dinh et al. (2016)); and this also means that we can use a lower-dimensional \(\mathbf{\epsilon}\) to generate samples along a submanifold of the entire parameter space, as detailed below. In order to scale BHNs to large primary networks, we use the weight normalization reparametrization (Salimans & Kingma, 2016)6:

This allows us to overcome the computational limitations of naively-parametrized BHNs noted by Louizos & Welling (2017), since computation now scales linearly, instead of quadratically, in the number of primary net units. Using this parametrization restricts the family of approximate posteriors, but still allows for a high degree of multimodality and dependence between the parameters. We also employ weight normalization within the hypernet, and found this stabilizes training dramatically. Initialization plays an important role as well; we recommend initializing the hypernet weights to small values to limit the impact of noise at the beginning of training. We also find clipping the outputs of the softmax to be within \((0.001,0.999)\) critical for numerical stability.

Parameter uncertainty also has regularization benefits: integrating over the posterior creates an implicit ensemble. Intuitively, when the most likely hypothesis predicts "A", but the posterior places more total mass on hypotheses predicting "B", we prefer predicting "B". By improving our estimate of the posterior, we more accurately weigh the evidence for different hypotheses.

We use an isotropic standard normal prior on the scaling factors (\(g\)) of the weights of the network.

We first demonstrate the behavior of the network on the toy 1D-regression problem from Blundell et al. (2015) in Figure 1.

We also use this experiment to evaluate the effects of our proposal for scaling BHNs via the weight norm parametrization (Section 3.3) by comparing with a model which generates the full set of parameters, and find that the two models produce very similar results, suggesting that our proposed method strikes a good balance between scalability and expressiveness. Next, we demonstrate the distinctive ability of Bayesian hypernets to learn multi-modal, dependent distributions. Figure 6 (appendix) shows that BHNs do learn approximate posteriors with dependence between different parameters, as measured by the Pearson correlation coefficient. Meanwhile, Figure 2 shows that BHNs are capable of learning multimodal posteriors. For this experiment, we trained an over-parametrized linear (primary) network: \(\hat{y}=a\cdot b\cdot x\) on a dataset generated as \(y=x+\epsilon\), and the BHN learns capture both the modes of \(a=b=1\) and \(a=b=-1\). We now show that BHNs act as a regularizer, outperforming dropout and traditional mean field (BbB).

In our experiments, we find that BHNs perform on par with dropout on full datasets of MNIST and CIFAR10; furthermore, increasing the flexibility of the posterior by adding more coupling layers improves performance, especially compared with models with 0 coupling layers, which cannot model dependencies between the parameters.

In these MNIST experiments, we use MLPs with 2 hidden layers of 800 or 1200 hidden units each. For CIFAR10, we train a convolutional neural net (CNN) with 4 hidden layers of \([64,64,128,128]\) channels, \(2\times 2\) max pooling after the second and the fourth layers, filter size of \(3\), and a single fully connected layer of \(512\) units.

We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations. We present a method of parametrizing BHNs which allows them to scale successfully to real world tasks, and show that BHNs can offer significant benefits over simpler methods for Bayesian deep learning.

In this paper, we employ weight normalization in the primary network (7), treating (only) the scaling factors \(\mathbf{g}\) as random variables. We choose an isotropic Gaussian prior for \(\mathbf{g}\): \(p(\mathbf{g})=\mathcal{N}(\mathbf{g};\mathbf{0},\lambda\mathbf{I})\), which results in an \(L_{2}\) weight-decay penalty on \(\mathbf{g}\), or, equivalently, \(\mathbf{w}=\mathbf{g}\frac{\mathbf{v}}{||\mathbf{v}||_{2}}\). Our objective and lower bound are then: \[\log p(\mathcal{D};\mathbf{v},\mathbf{b}) =\log\int_{\mathbf{g}}p(\mathcal{D}|\mathbf{g};\mathbf{v},\mathbf{ b})p(\mathbf{g})d\mathbf{g} \tag{8}\] \[\geq\mathbb{E}_{q(\mathbf{g})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{g})]\] (9) \[\geq\mathbb{E}_{\mathbf{\epsilon}\sim q_{\mathbf{\epsilon}}(\mathbf{ \epsilon}),\mathbf{g}=h_{\phi}(\mathbf{\epsilon})}[\log p(\mathcal{D}|\mathbf{g}; \mathbf{v},\mathbf{b})+\log p(\mathbf{g})-\log q(\mathbf{\epsilon})+\log\left| \det\frac{\partial h_{\phi}(\mathbf{\epsilon})}{\partial\mathbf{\epsilon}}\right|] \tag{10}\] where \(\mathbf{v}\) and \(\mathbf{b}\) are the direction and bias parameters of the primary net, and \(\phi\) is the parameters of the hypernetwork. We optimize this bound with respect to \(\{\mathbf{v},\mathbf{b},\phi\}\) during training.

Input: Figure 2: Learning the identity function with an overparametrized network: ŷ = a · b · x. This parametrization results in symmetries shown by the dashed red lines, and the Bayesian hypernetwork assigns significant probability mass to both modes of the posterior (a = b = 1 and a = b = −1).

Input: Figure 6: Histogram of Pearson correlation coefficient p-values (left) and a scatter matrix (right) of samples from a hypernet approximate posterior. We see that the hypernet posterior includes correlations between different parameters. Many of the p-values of the Pearson correlation test are below .05.

Input: Figure 1: Illustration of BHNs (second and third) and a traditional non-Bayesian DNN (first) on the toy problem from Blundell et al. (2015). In the second subplot, we place a prior on the scaling factor g and infer the posterior distribution using a BHN, while in the third subplot the hypernet is used to generate the whole weight matrices of the primary net. Each shaded region represents half a standard deviation in the posterior on the predictive mean. The red crosses are 50 examples from the training dataset.

Input: Figure 3: Box plot of performance across 10 trials. Bayesian hypernets (BHNs) with inverse autoregressive flows (IAF) consistently outperform the other methods.

Input: Figure 5: Adversary detection: Horizontal axis is the step size of the FGS algorithm. While accuracy drops when more perturbation is added to the data (left), uncertainty measures also increase (first row). In particular, the BALD and Mean STD scores, which measure epistemic uncertainty, are strongly increasing for BHNs, but not for dropout. The second row and third row plots show results for adversary detection and error detection (respectively) in terms of the AUC of ROC (y-axis) with increasing perturbation along the x-axis. Gradient direction is estimated with one Monte Carlo sample of the weights/dropout mask.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, I'll evaluate the claim "We propose Bayesian hypernetworks: a framework for approximate Bayesian inference in neural networks."

This claim appears to be well-stated for several reasons:
1. The evidence demonstrates thorough theoretical development of Bayesian hypernetworks (BHNs) with detailed mathematical formulations
2. The authors provide comprehensive empirical validation through experiments on toy problems and real datasets
3. The results show BHNs can effectively model multimodal posteriors and parameter dependencies
4. The authors acknowledge limitations and build upon existing work, positioning BHNs appropriately within the field
5. The experiments demonstrate practical benefits over baseline methods while being transparent about implementation details

The claim modestly introduces the framework without overstating its capabilities or impact.

<score>1</score>