Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

For these reasons, to address the challenges of text-based ZSL, we first introduce a variational hetero-encoder (VHE) that encodes an image to decode its _BOW_ textual description.

The review comment to be evaluated is:

Paper Summary: This paper studies the zero-shot learning problem with deep generative models. More specifically, it proposed a hybrid framework that combines VAEs (more precisely, the variational hetero-encoder or VHE) and GANs all together. The entire model is composed of an image encoder (Weibull upward-downward variational encoder), a text decoder (Poisson Gamma belief network), and an image generator (generative adversarial network). Once learned, the generative models can be directly used for zero-shot classification and various image generation applications. In the experiments, two benchmark datasets CUB and Oxford-Flowers are used.

==
Novelty/Significance:
Zero-shot learning is a challenging task and he main motivation of the paper (using generative model) is interesting. The text representation in the paper is simply bag-of-words which limits the application to some extent. In a broader context, image captioning using generative model seems quite relevant.

Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space, Wang et al. In NIPS 2017.

==
Quality:
Overall, reviewer feels this is a very interesting work. However, the results from the paper is quite mixed. It is not yet convincing whether the proposed approach is the state-of-the-art in zero-shot learning or text-to-image generation.

First, this paper demonstrates the power of generative models in text-to-image generation and other applications. However, reviewer feels that the zero-shot classification result is weak. In Table 1 and Table 2, it seems that GAZSL (Zhu et al. 2018) outperforms the proposed approach.

Q1: In Table 2, is it possible to report the top-5 accuracy on CUB-easy and top-1 accuracy on Oxford-Flower dataset? Otherwise, it is not very convincing that proposed approach is better than the state-of-the-art approach GAZSL.

Second, the text-to-image generation results look reasonably good. But the resolution and quality of generated images are far from state-of-the-art. One suggestion is to train the VHE model with an improved image generator.

StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks, Zhang et al. In CVPR 2017.

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks, Xu et al. In CVPR 2018.

Also, reviewer would expect to see an improved image generator can lead to a better ZSL performance.

Typo: In the title: Zero-Short → Zero-Shot.

The evidence to be evaluated is:The proposed VHE uses the Poisson gamma belief network (GBN) of Zhou et al. (2015) as its text decoder, whose output is a bag-of-words count vector, as shown in Fig. 1 (a), and a Weibull distribution based deterministic-upward-stochastic-downward inference network (Zhang et al., 2018) as its image encoder, whose input is a natural image, as shown in Fig. 1 (b).

_Two methods are popular in representing texts: word embedding (Bengio et al., 2003; Mikolov et al., 2013) and BOW count vectors. Considering the need of semantically relate a class of images to the key words in the corresponding class-specific textual description, we represent the text descriptions as BOW count vectors as \(\left\{\mathbf{t}_{c}\in\mathbb{Z}^{K_{0}}\right\}_{c=1}^{S+U}\), where \(\mathbb{Z}=\{0,1,\cdots\}\) and \(K_{0}\) is the vocabulary size. To extract hierarchical latent representation from these high-dimensional sparse count vectors \(\mathbf{t}_{c}\), we choose the Poisson GBN (Zhou et al., 2015), which can be considered as a member of deep exponential families (Ranganath et al., 2015). It consists of multiple gamma distributed stochastic hidden layers, generalizing the "shallow" Poisson factor analysis (Zhou et al., 2012; Zhou and Carin, 2015) that is closely related to latent Dirichlet allocation (LDA) (Blei et al., 2003) into a deep setting. More specifically, modeling \(\mathbf{t}_{c}\) under the Poisson likelihood, its generative model with \(L\) hidden layers, from top to bottom, is expressed as \[\mathbf{\theta}_{c}^{(L)} \sim\text{Gam}\left(\mathbf{r},1/s_{c}^{(L+1)}\right),\,\ldots,\mathbf{ \theta}_{c}^{(l)}\sim\text{Gam}\left(\mathbf{\Phi}^{(l+1)}\mathbf{\theta}_{c}^{(l+1)},1/s_{c}^{(l+1)}\right),\,\ldots,\] \[\mathbf{\theta}_{c}^{(1)} \sim\text{Gam}\left(\mathbf{\Phi}^{(2)}\mathbf{\theta}_{c}^{(2)},1/s_{c} ^{(2)}\right),\,\mathbf{t}_{c}\!\sim\!\text{Pois}\left(\mathbf{\Phi}^{(1)}\mathbf{\theta}_ {c}^{(1)}\right), \tag{1}\] where the hidden units \(\mathbf{\theta}_{c}^{(l)}\in\mathbb{R}_{+}^{K_{l}}\) of layer \(l\) are factorized under the gamma likelihood into the product of the factor loading \(\mathbf{\Phi}^{(l)}\in\mathbb{R}_{+}^{K_{l-1}\times K_{l}}\) and hidden units of the next layer, \(\mathbb{R}_{+}=\{x,x\geq 0\}\), \(s_{c}^{(l)}>0\), and \(K_{l}\) is the number of topics (factors) of layer \(l\).

Choosing a deep generative model as the text decoder distinguishes VHE from Gomez et al. (2017) that uses LDA (Blei et al., 2003), a "shallow" probabilistic model, and Ba et al. (2015) that uses non-probabilistic black-box DNNs to learn class-specific document features.

In this paper, we leverage WUDVE as the variational inference network to approximate the true posterior distribution \(p(\mathbf{\theta}\,|\,\mathbf{t},\mathbf{\Phi})\), where \(\mathbf{\theta}=\{\mathbf{\theta}^{(l)}\}_{1,L}\) denotes the set of local parameters, but change the input of WUDVE from the bag-of-words vector \(\mathbf{t}\) into an image \(\mathbf{x}\) associated with \(\mathbf{t}\). More specifically, as illustrated in Fig. 1b, we construct it as

Given \(\mathbf{\Phi}\) of the Poisson GBN, to approximate the posterior distribution of the local parameters \(\mathbf{\theta}\) given text description \(\mathbf{t}\), we use WVDVE \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\), whose input is an image \(\mathbf{x}\), and learn its parameters \(\mathbf{\Omega}\) by maximizing the evidence lower bound (ELBO) as \[\max_{\mathbf{\Omega}}L_{\text{VHE}}(\mathbf{\Omega})=\mathbb{E}_{(\mathbf{x},\mathbf{t}) \sim p_{data}(\mathbf{x},\mathbf{t})}\left\{\mathbb{E}_{q_{\mathbf{\Omega}}(\mathbf{\theta} \,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln p\left(\mathbf{t}\,|\,\mathbf{\Phi}^{(1)},\mathbf{\theta }^{(1)}\right)-\ln\frac{q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}{p( \mathbf{\theta}\,|\,\mathbf{\Phi})}\right]\right\}, \tag{4}\]

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN. _Both VAE and GAN are widely used for image generation, but we find in experiments that VAE is not expressive enough for mid/high-resolution RGB images (64 \(\times\) 64 resolution in our task) to clearly help the learning of VHE, whereas GAN does better for this purpose. Thus, we combine GAN and VHE in a novel way via a shared latent space, to further explore the relationships between two different modes for ZSL.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator.

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities.

The network architecture for the overall model shown in Fig. 1 is described in detail in the Appendix.

The proposed VHE by itself and VHEGAN can both be used for ZSL in the same way: with the Poisson GBN based VHE text decoder and the WUDVE based VHE image encoder, to perform zero-shot classification for a test image \(\mathbf{x}\) from an unseen class, we predict its class label as

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7).

To remedy these drawbacks, text-based ZSL have been developed using easily accessible textural descriptions such as Wikipedia. Along this line, (Elloseiny et al., 2013; 2017; Ba et al., 2015; Qiao et al., 2016) learn explicit visual classifiers conditioned on the textual description with seen classes, further transferred to the unseen ones.

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning. In Srivastava & Salakhutdinov (2012, 2014), a deep Boltzmann machine is developed to model the joint generation of image feature and BOW tags.

Besides, VHEGAN is able to realize multi-directed transformation, where ZSL can be seen as predicting the BOW textual features from the images.

VHE builds a variational posterior conditioned on images to approximate Gamma latent variables to generate the BOW vector of documents.

Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

The 4096 dimensional image features, \(f(\mathbf{x})\) in (3), from the fc1 layer of the pre-trained VGG16 network (Simonyan & Zisserman, 2015) are fed into the VHE image encoder, and the size of each generated RGB image is set as \(64\times 64\).

For VHEs and VHEGANs with different network structures, Table 1 shows that given the same structure on the text decoder and image encoder, each VHEGAN consistently outperforms its VHE counterpart, suggesting the advantage of a joint generation of both the image and text modalities.

Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images.

We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images).

**Image retrieval given a text query:** For image \(\mathbf{x}_{i}\), via the text generator and image encoder, we generate a bag-of-words vector \(\hat{\mathbf{t}}_{i}\) using Poisson mean \(\mathbf{\Phi}^{(1)}\mathbf{\theta}^{(1)}\) as \[\hat{\mathbf{t}}_{i}\,|\,\mathbf{\theta}_{i}\sim p(\mathbf{t}\,|\,\mathbf{\Phi},\mathbf{\theta}_{ i}),\,\mathbf{\theta}_{i}\,|\,\mathbf{x}_{i}\sim q_{\Omega}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x} _{i}). \tag{9}\]

The text-ZSL accuracies in Table 1 have already illustrated the effectiveness of VHEGAN in retrieving relevant textual descriptions given an image. With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space.

Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

We give a detailed hybrid SGMCMC/SGD algorithm for VHEGAN in this section, to realize learning end-to-end.

In Fig. 1, we give a simple model structure of VHEGAN. In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively.

\(\mathbf{x}\rightarrow\mathbf{h}^{(l)},l=1,2,3\)\(\mathbf{x}\rightarrow\mathbf{h}^{(1)}\)\(1\times\) { fully-connected layer with 256 units and Leaky-RELU activation.}\(\mathbf{h}^{(1)}\rightarrow\mathbf{h}^{(2)}\)\(1\times\) { fully-connected layer with 128 units and Leaky-RELU activation.}\(\mathbf{h}^{(2)}\rightarrow\mathbf{h}^{(3)}\)\(1\times\) { fully-connected layer with 64 units and Leaky-RELU activation.} Weibull variational posterior \(q(\mathbf{\theta}^{(l)}|\mathbf{h}^{(l)})\) as shown in Fig. :\(\mathbf{h}^{(l)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(l)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (l)}};l=1,2,3\)\(\mathbf{h}^{(1)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(1)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (1)}}\)\(1\times\) { fully-connected layer with 256 units to \(\mathbf{k}_{\mathbf{\theta}^{(1)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(1)}}\).}\(\mathbf{h}^{(2)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(2)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (2)}}\)\(1\times\) { fully-connected layer with 128 units to \(\mathbf{k}_{\mathbf{\theta}^{(2)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(2)}}\).}\(\mathbf{h}^{(3)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(3)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (3)}}\)\(1\times\) { fully-connected layer with 64 units to \(\mathbf{k}_{\mathbf{\theta}^{(3)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(3)}}\).} Image Generation Network shown in Fig. 10 : \(\mathbf{\theta}^{(l)}\rightarrow\hat{\mathbf{x}}\); \(l=1,2,3\)\(1\times\) { fully-connected layer with \(4\times 4\times 64\times 4\) units, RELU activation and batch normalization.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 128 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 64 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2.}

Input: Table 1: Accuracy (%) of zero-shot classification on CUB2011-hard, CUB2011-easy, and Flower Datasets. Note that some of them are attribute-based methods but applicable in our setting by replacing attribute vectors with text features (labeled by ∗), as discussed in Elhoseiny et al. (2017b). ZSLPP and GAZSL use a welldefined part detection features for images. All results on CUB2011 and the results of WAC on Flower come from Elhoseiny et al. (2017b); Qiao et al. (2016); Elhoseiny et al. (2017a), with all the others obtained by running the code provided by the original authors.

Input: Figure 9: The architecture of WUDVE in VHE and VHEGAN.

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 2: Class specific image generation based on the textual descriptions of unseen classes. (a) CUB2011easy and Flower data, where the first row shows real samples, while the others generated images; (b) Flower data, where the top row shows the representative textual captions and two real images of the same class.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that it is moderately overstated. While the evidence shows that VHE does encode images to generate BOW text descriptions through a Weibull-based encoder and Poisson GBN decoder architecture, the claim doesn't acknowledge key limitations. The review points out that the text representation being limited to BOW restricts applications. Additionally, the results are described as "mixed" and not clearly state-of-the-art. The model structure and mathematical framework are well-documented, but the claim would be better stated with appropriate caveats about the representational limitations and performance uncertainties.

<score>3</score>

Justification:
1. The evidence confirms the technical description of VHE's architecture and capability to encode images for BOW text generation
2. The reviewer highlights limitations of BOW representation that aren't acknowledged in the claim
3. Results are described as "mixed" in the review, suggesting the claim should be more measured
4. The mathematical foundations and model structure are well-supported by detailed equations and figures
5. The claim would be more accurate if it noted the specific text representation approach (BOW) and its limitations


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

For these reasons, to address the challenges of text-based ZSL, we first introduce a variational hetero-encoder (VHE) that encodes an image to decode its _BOW_ textual description.

The review comment to be evaluated is:

[Paper Summary]
This work suggests a new model incorporating deep topic model (text decoder),  VHE (image encoder), and GAN. The topic model and the VHE shares the topic parameters, and the GAN generate an image regarding the topic. Then, for ZSL, the image is encoded to corresponding topic parameters, and the parameter can tell which text description (unseen) is matched with the highest probability. GAN model is used to generate an image given the topic distribution. During the training of the GAN, the VHE and topic model is jointly trained and can enhance the ZSL performance marginally.

[pros]
- This work successfully incorporated the topic model and image encoding/decoding. All the individual parts are already given, but I think incorporating them in terms of a unified probabilistic model is also meaningful for this field.
- This work shows superior performance on the image to text ZSL problem.
- This work mapped the text to image, image to text mapping in a generative manner.

[cons]
- The problem is only valid when the unseen class distribution is very similar to the given classes. For example, the text description of unseen classes should be well represented to the topics from seen classes.
- It is doubtful that this corresponds to the term zero-shot learning; dealing with the case that the unseen class and the seen class are notably different from each other.
- Similarly, GAN learns images from the seen classes, and by nature, GAN would not generate the proper images of the unseen class if the image distribution of the unseen class is different to the already seen class. In the paper, the classes are very similar to each other (birds, flowers) and that would be the reason GAN worked in this model.
- (minor) The likelihood of the text (image) given topic should be provided and compared to the existing models.

[Summary]
- The reviewer is personally interested in the proposal of the work, but concern that ZSL is difficult to be the main target of the paper because the model can only deal with the classes with (very) similar semantics, and this is the main reason for the rating. The testing with more diverse class should be given, or solid explanation of the mentioned problem would be required.

The evidence to be evaluated is:The proposed VHE uses the Poisson gamma belief network (GBN) of Zhou et al. (2015) as its text decoder, whose output is a bag-of-words count vector, as shown in Fig. 1 (a), and a Weibull distribution based deterministic-upward-stochastic-downward inference network (Zhang et al., 2018) as its image encoder, whose input is a natural image, as shown in Fig. 1 (b).

_Two methods are popular in representing texts: word embedding (Bengio et al., 2003; Mikolov et al., 2013) and BOW count vectors. Considering the need of semantically relate a class of images to the key words in the corresponding class-specific textual description, we represent the text descriptions as BOW count vectors as \(\left\{\mathbf{t}_{c}\in\mathbb{Z}^{K_{0}}\right\}_{c=1}^{S+U}\), where \(\mathbb{Z}=\{0,1,\cdots\}\) and \(K_{0}\) is the vocabulary size. To extract hierarchical latent representation from these high-dimensional sparse count vectors \(\mathbf{t}_{c}\), we choose the Poisson GBN (Zhou et al., 2015), which can be considered as a member of deep exponential families (Ranganath et al., 2015). It consists of multiple gamma distributed stochastic hidden layers, generalizing the "shallow" Poisson factor analysis (Zhou et al., 2012; Zhou and Carin, 2015) that is closely related to latent Dirichlet allocation (LDA) (Blei et al., 2003) into a deep setting. More specifically, modeling \(\mathbf{t}_{c}\) under the Poisson likelihood, its generative model with \(L\) hidden layers, from top to bottom, is expressed as \[\mathbf{\theta}_{c}^{(L)} \sim\text{Gam}\left(\mathbf{r},1/s_{c}^{(L+1)}\right),\,\ldots,\mathbf{ \theta}_{c}^{(l)}\sim\text{Gam}\left(\mathbf{\Phi}^{(l+1)}\mathbf{\theta}_{c}^{(l+1)},1/s_{c}^{(l+1)}\right),\,\ldots,\] \[\mathbf{\theta}_{c}^{(1)} \sim\text{Gam}\left(\mathbf{\Phi}^{(2)}\mathbf{\theta}_{c}^{(2)},1/s_{c} ^{(2)}\right),\,\mathbf{t}_{c}\!\sim\!\text{Pois}\left(\mathbf{\Phi}^{(1)}\mathbf{\theta}_ {c}^{(1)}\right), \tag{1}\] where the hidden units \(\mathbf{\theta}_{c}^{(l)}\in\mathbb{R}_{+}^{K_{l}}\) of layer \(l\) are factorized under the gamma likelihood into the product of the factor loading \(\mathbf{\Phi}^{(l)}\in\mathbb{R}_{+}^{K_{l-1}\times K_{l}}\) and hidden units of the next layer, \(\mathbb{R}_{+}=\{x,x\geq 0\}\), \(s_{c}^{(l)}>0\), and \(K_{l}\) is the number of topics (factors) of layer \(l\).

Choosing a deep generative model as the text decoder distinguishes VHE from Gomez et al. (2017) that uses LDA (Blei et al., 2003), a "shallow" probabilistic model, and Ba et al. (2015) that uses non-probabilistic black-box DNNs to learn class-specific document features.

In this paper, we leverage WUDVE as the variational inference network to approximate the true posterior distribution \(p(\mathbf{\theta}\,|\,\mathbf{t},\mathbf{\Phi})\), where \(\mathbf{\theta}=\{\mathbf{\theta}^{(l)}\}_{1,L}\) denotes the set of local parameters, but change the input of WUDVE from the bag-of-words vector \(\mathbf{t}\) into an image \(\mathbf{x}\) associated with \(\mathbf{t}\). More specifically, as illustrated in Fig. 1b, we construct it as

Given \(\mathbf{\Phi}\) of the Poisson GBN, to approximate the posterior distribution of the local parameters \(\mathbf{\theta}\) given text description \(\mathbf{t}\), we use WVDVE \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\), whose input is an image \(\mathbf{x}\), and learn its parameters \(\mathbf{\Omega}\) by maximizing the evidence lower bound (ELBO) as \[\max_{\mathbf{\Omega}}L_{\text{VHE}}(\mathbf{\Omega})=\mathbb{E}_{(\mathbf{x},\mathbf{t}) \sim p_{data}(\mathbf{x},\mathbf{t})}\left\{\mathbb{E}_{q_{\mathbf{\Omega}}(\mathbf{\theta} \,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln p\left(\mathbf{t}\,|\,\mathbf{\Phi}^{(1)},\mathbf{\theta }^{(1)}\right)-\ln\frac{q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}{p( \mathbf{\theta}\,|\,\mathbf{\Phi})}\right]\right\}, \tag{4}\]

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN. _Both VAE and GAN are widely used for image generation, but we find in experiments that VAE is not expressive enough for mid/high-resolution RGB images (64 \(\times\) 64 resolution in our task) to clearly help the learning of VHE, whereas GAN does better for this purpose. Thus, we combine GAN and VHE in a novel way via a shared latent space, to further explore the relationships between two different modes for ZSL.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator.

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities.

The network architecture for the overall model shown in Fig. 1 is described in detail in the Appendix.

The proposed VHE by itself and VHEGAN can both be used for ZSL in the same way: with the Poisson GBN based VHE text decoder and the WUDVE based VHE image encoder, to perform zero-shot classification for a test image \(\mathbf{x}\) from an unseen class, we predict its class label as

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7).

To remedy these drawbacks, text-based ZSL have been developed using easily accessible textural descriptions such as Wikipedia. Along this line, (Elloseiny et al., 2013; 2017; Ba et al., 2015; Qiao et al., 2016) learn explicit visual classifiers conditioned on the textual description with seen classes, further transferred to the unseen ones.

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning. In Srivastava & Salakhutdinov (2012, 2014), a deep Boltzmann machine is developed to model the joint generation of image feature and BOW tags.

Besides, VHEGAN is able to realize multi-directed transformation, where ZSL can be seen as predicting the BOW textual features from the images.

VHE builds a variational posterior conditioned on images to approximate Gamma latent variables to generate the BOW vector of documents.

Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

The 4096 dimensional image features, \(f(\mathbf{x})\) in (3), from the fc1 layer of the pre-trained VGG16 network (Simonyan & Zisserman, 2015) are fed into the VHE image encoder, and the size of each generated RGB image is set as \(64\times 64\).

For VHEs and VHEGANs with different network structures, Table 1 shows that given the same structure on the text decoder and image encoder, each VHEGAN consistently outperforms its VHE counterpart, suggesting the advantage of a joint generation of both the image and text modalities.

Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images.

We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images).

**Image retrieval given a text query:** For image \(\mathbf{x}_{i}\), via the text generator and image encoder, we generate a bag-of-words vector \(\hat{\mathbf{t}}_{i}\) using Poisson mean \(\mathbf{\Phi}^{(1)}\mathbf{\theta}^{(1)}\) as \[\hat{\mathbf{t}}_{i}\,|\,\mathbf{\theta}_{i}\sim p(\mathbf{t}\,|\,\mathbf{\Phi},\mathbf{\theta}_{ i}),\,\mathbf{\theta}_{i}\,|\,\mathbf{x}_{i}\sim q_{\Omega}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x} _{i}). \tag{9}\]

The text-ZSL accuracies in Table 1 have already illustrated the effectiveness of VHEGAN in retrieving relevant textual descriptions given an image. With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space.

Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

We give a detailed hybrid SGMCMC/SGD algorithm for VHEGAN in this section, to realize learning end-to-end.

In Fig. 1, we give a simple model structure of VHEGAN. In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively.

\(\mathbf{x}\rightarrow\mathbf{h}^{(l)},l=1,2,3\)\(\mathbf{x}\rightarrow\mathbf{h}^{(1)}\)\(1\times\) { fully-connected layer with 256 units and Leaky-RELU activation.}\(\mathbf{h}^{(1)}\rightarrow\mathbf{h}^{(2)}\)\(1\times\) { fully-connected layer with 128 units and Leaky-RELU activation.}\(\mathbf{h}^{(2)}\rightarrow\mathbf{h}^{(3)}\)\(1\times\) { fully-connected layer with 64 units and Leaky-RELU activation.} Weibull variational posterior \(q(\mathbf{\theta}^{(l)}|\mathbf{h}^{(l)})\) as shown in Fig. :\(\mathbf{h}^{(l)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(l)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (l)}};l=1,2,3\)\(\mathbf{h}^{(1)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(1)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (1)}}\)\(1\times\) { fully-connected layer with 256 units to \(\mathbf{k}_{\mathbf{\theta}^{(1)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(1)}}\).}\(\mathbf{h}^{(2)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(2)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (2)}}\)\(1\times\) { fully-connected layer with 128 units to \(\mathbf{k}_{\mathbf{\theta}^{(2)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(2)}}\).}\(\mathbf{h}^{(3)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(3)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (3)}}\)\(1\times\) { fully-connected layer with 64 units to \(\mathbf{k}_{\mathbf{\theta}^{(3)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(3)}}\).} Image Generation Network shown in Fig. 10 : \(\mathbf{\theta}^{(l)}\rightarrow\hat{\mathbf{x}}\); \(l=1,2,3\)\(1\times\) { fully-connected layer with \(4\times 4\times 64\times 4\) units, RELU activation and batch normalization.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 128 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 64 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2.}

Input: Table 1: Accuracy (%) of zero-shot classification on CUB2011-hard, CUB2011-easy, and Flower Datasets. Note that some of them are attribute-based methods but applicable in our setting by replacing attribute vectors with text features (labeled by ∗), as discussed in Elhoseiny et al. (2017b). ZSLPP and GAZSL use a welldefined part detection features for images. All results on CUB2011 and the results of WAC on Flower come from Elhoseiny et al. (2017b); Qiao et al. (2016); Elhoseiny et al. (2017a), with all the others obtained by running the code provided by the original authors.

Input: Figure 9: The architecture of WUDVE in VHE and VHEGAN.

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 2: Class specific image generation based on the textual descriptions of unseen classes. (a) CUB2011easy and Flower data, where the first row shows real samples, while the others generated images; (b) Flower data, where the top row shows the representative textual captions and two real images of the same class.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "we first introduce a variational hetero-encoder (VHE) that encodes an image to decode its BOW textual description" appears well-stated and accurately reflects what is presented in the evidence. The paper provides detailed mathematical formulations, model architectures, and experimental results demonstrating how VHE encodes images and decodes textual descriptions. The evidence shows thorough technical details of the VHE architecture, including the Poisson GBN text decoder and WUDVE image encoder components. The review comments, while raising some limitations about ZSL performance, don't dispute the fundamental claim about the VHE model's core functionality.

<score>1</score>

The justification for this score:
1. The technical details provided fully support the claim's specific statement about the VHE architecture
2. The model diagrams and equations clearly illustrate the image encoding and text decoding process
3. The claim doesn't overstate capabilities or results - it simply describes what the model does
4. Even the critical review comments acknowledge the basic model architecture works as claimed
5. The evidence shows working implementations and results that validate the basic claim


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

For these reasons, to address the challenges of text-based ZSL, we first introduce a variational hetero-encoder (VHE) that encodes an image to decode its _BOW_ textual description.

The review comment to be evaluated is:

This paper developed a generative model to perform simultaneous embedding/generation of images/texts, with application to zero-shot learning. The experiments are extensive.

The novelty of this work is lacking.
The proposed method consists of a bag of existing models proposed by previous works.
But why using a certain model is not justified or explanined.
For example, for image generation, why use GAN instead of VAE.
For text encoding, why use PGBN, instead of recurrent VAE.

The method seems deteched from the problem of ZSL.
Throughout the paper, the authors mostly talk about how to perform joint embedding of texts and images. They give ZSL a touch, but as a side thing.
I would suggest the authors to position this work as a text/iamge embedding/generation work. Then use ZSL as an application.

The writing needs to be significantly improved. In the first paragraph describing the problem of ZSL, the authors end up with talking about the evaluation metric of ZSL.

The evidence to be evaluated is:The proposed VHE uses the Poisson gamma belief network (GBN) of Zhou et al. (2015) as its text decoder, whose output is a bag-of-words count vector, as shown in Fig. 1 (a), and a Weibull distribution based deterministic-upward-stochastic-downward inference network (Zhang et al., 2018) as its image encoder, whose input is a natural image, as shown in Fig. 1 (b).

_Two methods are popular in representing texts: word embedding (Bengio et al., 2003; Mikolov et al., 2013) and BOW count vectors. Considering the need of semantically relate a class of images to the key words in the corresponding class-specific textual description, we represent the text descriptions as BOW count vectors as \(\left\{\mathbf{t}_{c}\in\mathbb{Z}^{K_{0}}\right\}_{c=1}^{S+U}\), where \(\mathbb{Z}=\{0,1,\cdots\}\) and \(K_{0}\) is the vocabulary size. To extract hierarchical latent representation from these high-dimensional sparse count vectors \(\mathbf{t}_{c}\), we choose the Poisson GBN (Zhou et al., 2015), which can be considered as a member of deep exponential families (Ranganath et al., 2015). It consists of multiple gamma distributed stochastic hidden layers, generalizing the "shallow" Poisson factor analysis (Zhou et al., 2012; Zhou and Carin, 2015) that is closely related to latent Dirichlet allocation (LDA) (Blei et al., 2003) into a deep setting. More specifically, modeling \(\mathbf{t}_{c}\) under the Poisson likelihood, its generative model with \(L\) hidden layers, from top to bottom, is expressed as \[\mathbf{\theta}_{c}^{(L)} \sim\text{Gam}\left(\mathbf{r},1/s_{c}^{(L+1)}\right),\,\ldots,\mathbf{ \theta}_{c}^{(l)}\sim\text{Gam}\left(\mathbf{\Phi}^{(l+1)}\mathbf{\theta}_{c}^{(l+1)},1/s_{c}^{(l+1)}\right),\,\ldots,\] \[\mathbf{\theta}_{c}^{(1)} \sim\text{Gam}\left(\mathbf{\Phi}^{(2)}\mathbf{\theta}_{c}^{(2)},1/s_{c} ^{(2)}\right),\,\mathbf{t}_{c}\!\sim\!\text{Pois}\left(\mathbf{\Phi}^{(1)}\mathbf{\theta}_ {c}^{(1)}\right), \tag{1}\] where the hidden units \(\mathbf{\theta}_{c}^{(l)}\in\mathbb{R}_{+}^{K_{l}}\) of layer \(l\) are factorized under the gamma likelihood into the product of the factor loading \(\mathbf{\Phi}^{(l)}\in\mathbb{R}_{+}^{K_{l-1}\times K_{l}}\) and hidden units of the next layer, \(\mathbb{R}_{+}=\{x,x\geq 0\}\), \(s_{c}^{(l)}>0\), and \(K_{l}\) is the number of topics (factors) of layer \(l\).

Choosing a deep generative model as the text decoder distinguishes VHE from Gomez et al. (2017) that uses LDA (Blei et al., 2003), a "shallow" probabilistic model, and Ba et al. (2015) that uses non-probabilistic black-box DNNs to learn class-specific document features.

In this paper, we leverage WUDVE as the variational inference network to approximate the true posterior distribution \(p(\mathbf{\theta}\,|\,\mathbf{t},\mathbf{\Phi})\), where \(\mathbf{\theta}=\{\mathbf{\theta}^{(l)}\}_{1,L}\) denotes the set of local parameters, but change the input of WUDVE from the bag-of-words vector \(\mathbf{t}\) into an image \(\mathbf{x}\) associated with \(\mathbf{t}\). More specifically, as illustrated in Fig. 1b, we construct it as

Given \(\mathbf{\Phi}\) of the Poisson GBN, to approximate the posterior distribution of the local parameters \(\mathbf{\theta}\) given text description \(\mathbf{t}\), we use WVDVE \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\), whose input is an image \(\mathbf{x}\), and learn its parameters \(\mathbf{\Omega}\) by maximizing the evidence lower bound (ELBO) as \[\max_{\mathbf{\Omega}}L_{\text{VHE}}(\mathbf{\Omega})=\mathbb{E}_{(\mathbf{x},\mathbf{t}) \sim p_{data}(\mathbf{x},\mathbf{t})}\left\{\mathbb{E}_{q_{\mathbf{\Omega}}(\mathbf{\theta} \,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln p\left(\mathbf{t}\,|\,\mathbf{\Phi}^{(1)},\mathbf{\theta }^{(1)}\right)-\ln\frac{q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}{p( \mathbf{\theta}\,|\,\mathbf{\Phi})}\right]\right\}, \tag{4}\]

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN. _Both VAE and GAN are widely used for image generation, but we find in experiments that VAE is not expressive enough for mid/high-resolution RGB images (64 \(\times\) 64 resolution in our task) to clearly help the learning of VHE, whereas GAN does better for this purpose. Thus, we combine GAN and VHE in a novel way via a shared latent space, to further explore the relationships between two different modes for ZSL.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator.

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities.

The network architecture for the overall model shown in Fig. 1 is described in detail in the Appendix.

The proposed VHE by itself and VHEGAN can both be used for ZSL in the same way: with the Poisson GBN based VHE text decoder and the WUDVE based VHE image encoder, to perform zero-shot classification for a test image \(\mathbf{x}\) from an unseen class, we predict its class label as

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7).

To remedy these drawbacks, text-based ZSL have been developed using easily accessible textural descriptions such as Wikipedia. Along this line, (Elloseiny et al., 2013; 2017; Ba et al., 2015; Qiao et al., 2016) learn explicit visual classifiers conditioned on the textual description with seen classes, further transferred to the unseen ones.

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning. In Srivastava & Salakhutdinov (2012, 2014), a deep Boltzmann machine is developed to model the joint generation of image feature and BOW tags.

Besides, VHEGAN is able to realize multi-directed transformation, where ZSL can be seen as predicting the BOW textual features from the images.

VHE builds a variational posterior conditioned on images to approximate Gamma latent variables to generate the BOW vector of documents.

Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

The 4096 dimensional image features, \(f(\mathbf{x})\) in (3), from the fc1 layer of the pre-trained VGG16 network (Simonyan & Zisserman, 2015) are fed into the VHE image encoder, and the size of each generated RGB image is set as \(64\times 64\).

For VHEs and VHEGANs with different network structures, Table 1 shows that given the same structure on the text decoder and image encoder, each VHEGAN consistently outperforms its VHE counterpart, suggesting the advantage of a joint generation of both the image and text modalities.

Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images.

We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images).

**Image retrieval given a text query:** For image \(\mathbf{x}_{i}\), via the text generator and image encoder, we generate a bag-of-words vector \(\hat{\mathbf{t}}_{i}\) using Poisson mean \(\mathbf{\Phi}^{(1)}\mathbf{\theta}^{(1)}\) as \[\hat{\mathbf{t}}_{i}\,|\,\mathbf{\theta}_{i}\sim p(\mathbf{t}\,|\,\mathbf{\Phi},\mathbf{\theta}_{ i}),\,\mathbf{\theta}_{i}\,|\,\mathbf{x}_{i}\sim q_{\Omega}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x} _{i}). \tag{9}\]

The text-ZSL accuracies in Table 1 have already illustrated the effectiveness of VHEGAN in retrieving relevant textual descriptions given an image. With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space.

Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

We give a detailed hybrid SGMCMC/SGD algorithm for VHEGAN in this section, to realize learning end-to-end.

In Fig. 1, we give a simple model structure of VHEGAN. In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively.

\(\mathbf{x}\rightarrow\mathbf{h}^{(l)},l=1,2,3\)\(\mathbf{x}\rightarrow\mathbf{h}^{(1)}\)\(1\times\) { fully-connected layer with 256 units and Leaky-RELU activation.}\(\mathbf{h}^{(1)}\rightarrow\mathbf{h}^{(2)}\)\(1\times\) { fully-connected layer with 128 units and Leaky-RELU activation.}\(\mathbf{h}^{(2)}\rightarrow\mathbf{h}^{(3)}\)\(1\times\) { fully-connected layer with 64 units and Leaky-RELU activation.} Weibull variational posterior \(q(\mathbf{\theta}^{(l)}|\mathbf{h}^{(l)})\) as shown in Fig. :\(\mathbf{h}^{(l)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(l)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (l)}};l=1,2,3\)\(\mathbf{h}^{(1)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(1)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (1)}}\)\(1\times\) { fully-connected layer with 256 units to \(\mathbf{k}_{\mathbf{\theta}^{(1)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(1)}}\).}\(\mathbf{h}^{(2)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(2)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (2)}}\)\(1\times\) { fully-connected layer with 128 units to \(\mathbf{k}_{\mathbf{\theta}^{(2)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(2)}}\).}\(\mathbf{h}^{(3)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(3)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (3)}}\)\(1\times\) { fully-connected layer with 64 units to \(\mathbf{k}_{\mathbf{\theta}^{(3)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(3)}}\).} Image Generation Network shown in Fig. 10 : \(\mathbf{\theta}^{(l)}\rightarrow\hat{\mathbf{x}}\); \(l=1,2,3\)\(1\times\) { fully-connected layer with \(4\times 4\times 64\times 4\) units, RELU activation and batch normalization.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 128 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 64 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2.}

Input: Table 1: Accuracy (%) of zero-shot classification on CUB2011-hard, CUB2011-easy, and Flower Datasets. Note that some of them are attribute-based methods but applicable in our setting by replacing attribute vectors with text features (labeled by ∗), as discussed in Elhoseiny et al. (2017b). ZSLPP and GAZSL use a welldefined part detection features for images. All results on CUB2011 and the results of WAC on Flower come from Elhoseiny et al. (2017b); Qiao et al. (2016); Elhoseiny et al. (2017a), with all the others obtained by running the code provided by the original authors.

Input: Figure 9: The architecture of WUDVE in VHE and VHEGAN.

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 2: Class specific image generation based on the textual descriptions of unseen classes. (a) CUB2011easy and Flower data, where the first row shows real samples, while the others generated images; (b) Flower data, where the top row shows the representative textual captions and two real images of the same class.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence, the claim appears relatively well-stated but has some minor overstatement. The evidence shows they do introduce a VHE that encodes images to decode BOW text descriptions, supported by detailed mathematical formulations, architecture diagrams, and experimental results. However, saying they "first introduce" this concept may be slightly overstated, as the reviewer notes the method combines existing models. The technical details and results validate the core claim about the model's functionality, but the novelty aspect could be more carefully qualified.

<score>2</score>

Justification:
1. The evidence thoroughly documents the VHE architecture and demonstrates its ability to encode images and decode text descriptions through:
- Detailed mathematical formulations of the model components
- Clear architectural diagrams showing the encoder-decoder structure
- Experimental results showing the model's performance
- Extensive technical documentation of the implementation

2. Areas preventing a score of 1:
- The reviewer points out that "The novelty of this work is lacking" and it uses "a bag of existing models"
- The claim of "first introduce" may overstate the novelty since the approach builds on existing methods
- The justification for specific model choices (e.g., GAN vs VAE) could be better grounded according to reviewer

3. The core technical claim about the model's functionality is well-supported by evidence, making this closer to well-stated than overstated, hence a score of 2.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To further explore tighter relationships between images and texts to improve the performance of VHE for ZSL and add the capability to generate random images conditioning on an image, a document, or random noise,_ we further propose **V**ariational **H**etero-**E**ncoder randomized **G**enerative **A**dversarial **N**et (VHEGAN) that exploits the synergy between VHE and GAN.

The evidence to be evaluated is:Integrating VHE and GAN, the proposed VHEGAN, as shown in Fig. 1 (c), tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document, under the Poisson GBN text decoder, be the distribution of the random noise fed into the GAN image generator; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder; and the GAN discriminator tries to discriminate between that selected image and the generated one. All the three components that are structurally coupled with each other will be jointly optimized.

In this paper, we leverage WUDVE as the variational inference network to approximate the true posterior distribution \(p(\mathbf{\theta}\,|\,\mathbf{t},\mathbf{\Phi})\), where \(\mathbf{\theta}=\{\mathbf{\theta}^{(l)}\}_{1,L}\) denotes the set of local parameters, but change the input of WUDVE from the bag-of-words vector \(\mathbf{t}\) into an image \(\mathbf{x}\) associated with \(\mathbf{t}\).

Given \(\mathbf{\Phi}\) of the Poisson GBN, to approximate the posterior distribution of the local parameters \(\mathbf{\theta}\) given text description \(\mathbf{t}\), we use WVDVE \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\), whose input is an image \(\mathbf{x}\), and learn its parameters \(\mathbf{\Omega}\) by maximizing the evidence lower bound (ELBO) as \[\max_{\mathbf{\Omega}}L_{\text{VHE}}(\mathbf{\Omega})=\mathbb{E}_{(\mathbf{x},\mathbf{t}) \sim p_{data}(\mathbf{x},\mathbf{t})}\left\{\mathbb{E}_{q_{\mathbf{\Omega}}(\mathbf{\theta} \,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln p\left(\mathbf{t}\,|\,\mathbf{\Phi}^{(1)},\mathbf{\theta }^{(1)}\right)-\ln\frac{q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}{p( \mathbf{\theta}\,|\,\mathbf{\Phi})}\right]\right\}, \tag{4}\] where \(p(\mathbf{\theta}\,|\,\mathbf{\Phi})=\left[\prod_{l=1}^{L-1}p(\mathbf{\theta}^{(l)}\,|\, \mathbf{\Phi}^{(l+1)},\mathbf{\theta}^{(l+1)},s^{(l+1)})\right]p(\mathbf{\theta}^{(L)}\,| \,\mathbf{r},s^{L+1})\) is the prior distribution. We refer to Zhang et al. (2018) for more details on how WUDVE can be combined with TLASGR-MCMC (Cong et al., 2017) to develop Weibull hybrid autoencoding inference (WHAI), a hybrid SG-MCMC/variational inference that represents the posterior of \(\mathbf{\Phi}\) using MCMC samples, and the posterior of \(\mathbf{\theta}\) given a sample of \(\mathbf{\Phi}\) using WUDVE. To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN. _Both VAE and GAN are widely used for image generation, but we find in experiments that VAE is not expressive enough for mid/high-resolution RGB images (64 \(\times\) 64 resolution in our task) to clearly help the learning of VHE, whereas GAN does better for this purpose. Thus, we combine GAN and VHE in a novel way via a shared latent space, to further explore the relationships between two different modes for ZSL.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator.

To jointly optimize the VHE and GAN components of VHEGAN, we merge the expectations in (4) and (6) to define its loss function as \[\min_{G,\mathbf{\theta}}\max_{\mathbf{x},\mathbf{\theta}}\mathbb{E}_{(\mathbf{x},\mathbf{\theta}) \sim p_{data}(\mathbf{x},\mathbf{t})}\left\{\mathbb{E}_{q_{\mathbf{\Omega}}(\mathbf{\theta}\,| \,\mathbf{\Phi},\mathbf{x})}\left[-\ln p\left(\mathbf{t}\,|\,\mathbf{\Phi}^{(1)},\mathbf{\theta}^{( 1)}\right)+\ln\frac{q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}{p(\mathbf{ \theta}\,|\,\mathbf{\Phi})}-\ln D(G(\mathbf{\theta}))\right]+\ln D(\mathbf{x})\right\}. \tag{7}\] It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities.

What's more, by the aid of TLASGR-MCMC of Cong et al. (2017), we can collect posterior MCMC samples for the global parameters \(\mathbf{\Phi}\) of the Poisson GBN, and develop a hybrid SG-MCMC/VHE/SGD end-to-end learning algorithm (Zhang et al., 2018) for the parameters of VHEGAN, as detailedly described in Algorithm 1 in the Appendix.

Other more recent GANs, such as Wasserstein GAN (Arjovsky et al., 2017), Wasserstein GAN with gradient penalty (Gulrajani et al., 2017), spectral norm GAN (Miyato et al., 2018), and progressive growing GAN (Karras et al., 2018), may be combined with the VHE to construct even more expressive VHEGANs.

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7).

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning.

Both these two approaches are two-step models with the additional textual feature extraction realized by LSTM and "shallow" LDA, respectively, whereas the VHEGAN can realize joint optimization with hybrid SGMCMC/VHE/SGD algorithm. Besides, VHEGAN is able to realize multi-directed transformation, where ZSL can be seen as predicting the BOW textual features from the images.

VHE builds a variational posterior conditioned on images to approximate Gamma latent variables to generate the BOW vector of documents. To solve the problem of complete random generate process in GAN, conditional GANs (CGAN) (Mirza & Osindero, 2014; Reed et al., 2016) are alsodeveloped, where the conditions are labels, or attributes learned as point estimations separately with GAN, while VHEGAN learns a distributed condition jointly with a VAE model. Moreover, VHEGAN is similar with BiGAN (Donahue et al., 2017) and ALI (Dumoulin et al., 2017) since all of them learn a encoder that maps data into a latent space in GAN. Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

The 4096 dimensional image features, \(f(\mathbf{x})\) in (3), from the fc1 layer of the pre-trained VGG16 network (Simonyan & Zisserman, 2015) are fed into the VHE image encoder, and the size of each generated RGB image is set as \(64\times 64\).

except for the proposed VHEGAN that aims to find a shared semantically meaningful latent space between the image and text modalities, none of the other methods have generative models for both modalities, regardless of whether they learn a classifier or a distance metric in a latent space for ZSL. Table 1 shows that VHEGAN-layer3 clearly outperforms the state-of-the-art in terms of Top-1 accuracy on both the CUB2011-hard and Flower text-ZSL tasks, and is comparable to the second best on CUB2011-easy (it is also the best among all methods that have reported their Top-5 accuracies on CUB2011-easy).

For VHEs and VHEGANs with different network structures, Table 1 shows that given the same structure on the text decoder and image encoder, each VHEGAN consistently outperforms its VHE counterpart, suggesting the advantage of a joint generation of both the image and text modalities. It also shows that both VHEGAN and VHE have a clear improving trend as the Poisson GBN becomes deeper, suggesting the advantage of having a deep hierarchical representation for text generation.

In addition to quantitative evaluation on text-ZSL tasks, below we provide qualitative analysis to demonstrate the ability of VHEGAN in jointly modeling both the text and image modalities.

Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images.

We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images). We repeat the text-to-image generation experiment on Flower to generate random images given textual captions for three different unseen classes. As shown in Fig. 1(b), VHEGAN successfully generates flower images that semantically match their corresponding textual descriptions and visually resemble their corresponding example real images.

The text-ZSL accuracies in Table 1 have already illustrated the effectiveness of VHEGAN in retrieving relevant textual descriptions given an image. With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image. **Latent space interpolation:** In order to understand the learned manifold in the latent space, given two texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), after drawing \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) from their own conditional posteriors using Gibbs sampling, we interpolate between \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) and use these interpolated latent variables to generate both the images via the GAN image generator and texts via the VHE text decoder. We show the results in Fig. 5, where the first and last columns show the true texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), and the images generated from \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\), respectively, while the second to third columns show the generated texts and images from the interpolated \(\mathbf{\theta}\)'s. The strong correspondences between the generated images and texts and smooth changes between adjacent columns suggest that the VHEGAN latent space is semantically meaningful for both the image and text modalities. **Random text-image pairs generation:** Below we show how to generate data samples that contain both modalities. After training a 3-layer VHEGAN, following the data generation process of the VHE text decoder (\(i.e.\), the Poisson GBN), given \(\{\mathbf{\Phi}^{(l)}\}_{l=1}^{L}\) and \(\mathbf{r}\), we first generate \(\mathbf{\theta}^{(L)}\sim\text{Gam}\left(\mathbf{r},1/s^{L+1}\right)\) and then downward propagate it through the Poisson GBN as in (1) to calculate the Poisson rates for all words using \(\mathbf{\Phi}^{(1)}\mathbf{\theta}^{(1)}\). Given a random draw, the concatenation of \(\{\mathbf{\theta}^{(l)}\}_{l=1}^{L}\) is fed into the GAN image generator to generate a corresponding image. Shown in Fig. 6 are four random draws, for each of which we show its top seven words and its generated image, whose relationships are clearly interpretable, suggesting that VHEGAN is able to recode the key information of both modalities and the relationships between them.

We note for the VHEGAN, its VHE image encoder and GAN component together can also be viewed as an "autoencoding" GAN for images. Different from Dumoulin et al. (2017) and Donahue et al. (2017), the encoded latent space is affected not only by the GAN for the image modality, but also by the Poisson GBN for the text modailty. In Fig. 7, we present the real images \(\mathbf{x}\) and their VHEGANregenerations \(G\left(q_{\boldsymbol{\Omega}}\left(\boldsymbol{\theta}\,|\,\boldsymbol{\Phi}, \boldsymbol{x}\right)\right)\), which suggest that the VHEGAN can use its GAN image generator to regenerate random images that more or less resemble the original real image fed into the VHE image encoder.

For each column in Fig. 7(b), conditioning on a textual description, we show the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all layers, respectively, as the latent space fed into the GAN image generator.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space. With a stochastic-gradient MCMC algorithm to sample the global parameters of the VHE text decoder, a reparameterized Weibull distribution based variational encoder to approximate the posterior distribution of the local parameters of the VHE image encoder, and stochastic gradient descent (SGD) to estimate the parameters of the GAN image generator and discriminator, we develop a SG-MCMC/VHE/SGD hybrid inference algorithm to jointly train the components of VHEGAN. Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

We give a detailed hybrid SGMCMC/SGD algorithm for VHEGAN in this section, to realize learning end-to-end.

In Fig. 1, we give a simple model structure of VHEGAN. In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively.

Image encoder \(\mathbf{x}\rightarrow\mathbf{h}^{(l)},l=1,2,3\)\(\mathbf{x}\rightarrow\mathbf{h}^{(1)}\)\(1\times\) { fully-connected layer with 256 units and Leaky-RELU activation.}\(\mathbf{h}^{(1)}\rightarrow\mathbf{h}^{(2)}\)\(1\times\) { fully-connected layer with 128 units and Leaky-RELU activation.}\(\mathbf{h}^{(2)}\rightarrow\mathbf{h}^{(3)}\)\(1\times\) { fully-connected layer with 64 units and Leaky-RELU activation.} Weibull variational posterior \(q(\mathbf{\theta}^{(l)}|\mathbf{h}^{(l)})\) as shown in Fig. :\(\mathbf{h}^{(l)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(l)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (l)}};l=1,2,3\)\(\mathbf{h}^{(1)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(1)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (1)}}\)\(1\times\) { fully-connected layer with 256 units to \(\mathbf{k}_{\mathbf{\theta}^{(1)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(1)}}\).}\(\mathbf{h}^{(2)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(2)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (2)}}\)\(1\times\) { fully-connected layer with 128 units to \(\mathbf{k}_{\mathbf{\theta}^{(2)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(2)}}\).}\(\mathbf{h}^{(3)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(3)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (3)}}\)\(1\times\) { fully-connected layer with 64 units to \(\mathbf{k}_{\mathbf{\theta}^{(3)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(3)}}\).} Image Generation Network shown in Fig. 10 : \(\mathbf{\theta}^{(l)}\rightarrow\hat{\mathbf{x}}\); \(l=1,2,3\)\(1\times\) { fully-connected layer with \(4\times 4\times 64\times 4\) units, RELU activation and batch normalization.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 128 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 64 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2.}

Input: Table 1: Accuracy (%) of zero-shot classification on CUB2011-hard, CUB2011-easy, and Flower Datasets. Note that some of them are attribute-based methods but applicable in our setting by replacing attribute vectors with text features (labeled by ∗), as discussed in Elhoseiny et al. (2017b). ZSLPP and GAZSL use a welldefined part detection features for images. All results on CUB2011 and the results of WAC on Flower come from Elhoseiny et al. (2017b); Qiao et al. (2016); Elhoseiny et al. (2017a), with all the others obtained by running the code provided by the original authors.

Input: Figure 9: The architecture of WUDVE in VHE and VHEGAN.

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 2: Class specific image generation based on the textual descriptions of unseen classes. (a) CUB2011easy and Flower data, where the first row shows real samples, while the others generated images; (b) Flower data, where the top row shows the representative textual captions and two real images of the same class.

Input: Figure 6: Image and document samples jointly generated from a three-layer VHEGAN.

Input: Figure 7: Image reconstruction with real data x (the first row) and corresponding reconstructions G (qΩ (θ |Φ,x)), where each rows represent different sampling from qΩ (θ).

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that it is relatively well-stated (2) based on the following evaluation:

The claim states that VHEGAN builds on VHE to improve performance and add generative capabilities by exploiting synergies with GAN. The evidence strongly supports this through:

1. Detailed mathematical formulations and architecture descriptions showing how VHE and GAN are integrated
2. Quantitative results in Table 1 demonstrating improved ZSL performance
3. Qualitative results showing successful text-to-image and image-to-text generation capabilities
4. Clear explanations of how the components work together through shared latent space

While somewhat technical in nature, the claim avoids overstatement by:
- Being specific about what capabilities are added
- Grounding claims in concrete evidence
- Acknowledging limitations and potential improvements

The claim leans toward being well-stated but includes some moderate technical language that could be simplified, hence a score of 2.

<score>2</score>