Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

After training with a hybrid stochastic-gradient MCMC/variational inference/stochastic gradient descent inference algorithm, VHEGAN can be used in a variety of settings, such as text generation/retrieval conditioning on an image, image generation/retrieval conditioning on a document/image, and generation of text-image pairs.

The review comment to be evaluated is:

This paper developed a generative model to perform simultaneous embedding/generation of images/texts, with application to zero-shot learning. The experiments are extensive.

The novelty of this work is lacking.
The proposed method consists of a bag of existing models proposed by previous works.
But why using a certain model is not justified or explanined.
For example, for image generation, why use GAN instead of VAE.
For text encoding, why use PGBN, instead of recurrent VAE.

The method seems deteched from the problem of ZSL.
Throughout the paper, the authors mostly talk about how to perform joint embedding of texts and images. They give ZSL a touch, but as a side thing.
I would suggest the authors to position this work as a text/iamge embedding/generation work. Then use ZSL as an application.

The writing needs to be significantly improved. In the first paragraph describing the problem of ZSL, the authors end up with talking about the evaluation metric of ZSL.

The evidence to be evaluated is:Integrating VHE and GAN, the proposed VHEGAN, as shown in Fig. 1 (c), tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document, under the Poisson GBN text decoder, be the distribution of the random noise fed into the GAN image generator; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder; and the GAN discriminator tries to discriminate between that selected image and the generated one. All the three components that are structurally coupled with each other will be jointly optimized.

Note as shown in Cong et al. (2017), the Poisson GBN can also be represented as deep LDA, an alternative representation that facilitates the derivation of a scalable stochastic-gradient MCMC algorithm that is well suited for deep topic modeling.

The Poisson GBN is equipped with both a closed-form upward-downward Gibbs sampler that is suitable for a moderate-sized corpus (Zhou et al., 2016), and a topic-layer-adaptive stochastic gradient Riemannian (TLASGR) MCMC that is scalable to a big corpus (Cong et al., 2017).

In this paper, we leverage WUDVE as the variational inference network to approximate the true posterior distribution \(p(\mathbf{\theta}\,|\,\mathbf{t},\mathbf{\Phi})\), where \(\mathbf{\theta}=\{\mathbf{\theta}^{(l)}\}_{1,L}\) denotes the set of local parameters, but change the input of WUDVE from the bag-of-words vector \(\mathbf{t}\) into an image \(\mathbf{x}\) associated with \(\mathbf{t}\).

We refer to Zhang et al. (2018) for more details on how WUDVE can be combined with TLASGR-MCMC (Cong et al., 2017) to develop Weibull hybrid autoencoding inference (WHAI), a hybrid SG-MCMC/variational inference that represents the posterior of \(\mathbf{\Phi}\) using MCMC samples, and the posterior of \(\mathbf{\theta}\) given a sample of \(\mathbf{\Phi}\) using WUDVE. To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN.

What's more, by the aid of TLASGR-MCMC of Cong et al. (2017), we can collect posterior MCMC samples for the global parameters \(\mathbf{\Phi}\) of the Poisson GBN, and develop a hybrid SG-MCMC/VHE/SGD end-to-end learning algorithm (Zhang et al., 2018) for the parameters of VHEGAN, as detailedly described in Algorithm 1 in the Appendix.

Other more recent GANs, such as Wasserstein GAN (Arjovsky et al., 2017), Wasserstein GAN with gradient penalty (Gulrajani et al., 2017), spectral norm GAN (Miyato et al., 2018), and progressive growing GAN (Karras et al., 2018), may be combined with the VHE to construct even more expressive VHEGANs.

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7).

While neither the VHE or VHEGAN are equipped with class-specific classifiers, they can be directly applied for GSZL by simply changing \(\arg\max\) in (8) from \(c\in\{S+1,\dots,S+U\}\) to \(c\in\{1,\dots,S+U\}\).

To remedy these drawbacks, text-based ZSL have been developed using easily accessible textural descriptions such as Wikipedia. Along this line, (Elloseiny et al., 2013; 2017; Ba et al., 2015; Qiao et al., 2016) learn explicit visual classifiers conditioned on the textual description with seen classes, further transferred to the unseen ones.

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning.

Both these two approaches are two-step models with the additional textual feature extraction realized by LSTM and "shallow" LDA, respectively, whereas the VHEGAN can realize joint optimization with hybrid SGMCMC/VHE/SGD algorithm. Besides, VHEGAN is able to realize multi-directed transformation, where ZSL can be seen as predicting the BOW textual features from the images.

To solve the problem of complete random generate process in GAN, conditional GANs (CGAN) (Mirza & Osindero, 2014; Reed et al., 2016) are alsodeveloped, where the conditions are labels, or attributes learned as point estimations separately with GAN, while VHEGAN learns a distributed condition jointly with a VAE model.

Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images. Given example encyclopedia documents for the unseen classes, we follow this approach to generate random images. We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images). We repeat the text-to-image generation experiment on Flower to generate random images given textual captions for three different unseen classes. As shown in Fig. 1(b), VHEGAN successfully generates flower images that semantically match their corresponding textual descriptions and visually resemble their corresponding example real images.

Giving the textural description \(\mathbf{t}\) of an unseen class as the text query, we retrieve the top five images ranked by the cosine distances between \(\mathbf{t}\) and \(\hat{\mathbf{t}}_{i}\). Shown in Fig. 3 are two example image retrieval results, which suggest that the retrieved images are semantically related to their text queries in colors, shapes, and locations.

The text-ZSL accuracies in Table 1 have already illustrated the effectiveness of VHEGAN in retrieving relevant textual descriptions given an image. With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively.

**Latent space interpolation:** In order to understand the learned manifold in the latent space, given two texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), after drawing \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) from their own conditional posteriors using Gibbs sampling, we interpolate between \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) and use these interpolated latent variables to generate both the images via the GAN image generator and texts via the VHE text decoder. We show the results in Fig. 5, where the first and last columns show the true texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), and the images generated from \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\), respectively, while the second to third columns show the generated texts and images from the interpolated \(\mathbf{\theta}\)'s. The strong correspondences between the generated images and texts and smooth changes between adjacent columns suggest that the VHEGAN latent space is semantically meaningful for both the image and text modalities. **Random text-image pairs generation:** Below we show how to generate data samples that contain both modalities. After training a 3-layer VHEGAN, following the data generation process of the VHE text decoder (\(i.e.\), the Poisson GBN), given \(\{\mathbf{\Phi}^{(l)}\}_{l=1}^{L}\) and \(\mathbf{r}\), we first generate \(\mathbf{\theta}^{(L)}\sim\text{Gam}\left(\mathbf{r},1/s^{L+1}\right)\) and then downward propagate it through the Poisson GBN as in (1) to calculate the Poisson rates for all words using \(\mathbf{\Phi}^{(1)}\mathbf{\theta}^{(1)}\). Given a random draw, the concatenation of \(\{\mathbf{\theta}^{(l)}\}_{l=1}^{L}\) is fed into the GAN image generator to generate a corresponding image. Shown in Fig. 6 are four random draws, for each of which we show its top seven words and its generated image, whose relationships are clearly interpretable, suggesting that VHEGAN is able to recode the key information of both modalities and the relationships between them. **Image to image generation:** We note for the VHEGAN, its VHE image encoder and GAN component together can also be viewed as an "autoencoding" GAN for images. Different from Dumoulin et al. (2017) and Donahue et al. (2017), the encoded latent space is affected not only by the GAN for the image modality, but also by the Poisson GBN for the text modailty. In Fig. 7, we present the real images \(\mathbf{x}\) and their VHEGANregenerations \(G\left(q_{\boldsymbol{\Omega}}\left(\boldsymbol{\theta}\,|\,\boldsymbol{\Phi}, \boldsymbol{x}\right)\right)\), which suggest that the VHEGAN can use its GAN image generator to regenerate random images that more or less resemble the original real image fed into the VHE image encoder.

For each column in Fig. 7(b), conditioning on a textual description, we show the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all layers, respectively, as the latent space fed into the GAN image generator.

Examining Fig. 7(b) suggests that different hidden layers concentrate on somewhat different visual information and combining them lead to the best visual quality, a reason that by default all hidden layers are concatenated to fed into the GAN image generator. To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model.

With a stochastic-gradient MCMC algorithm to sample the global parameters of the VHE text decoder, a reparameterized Weibull distribution based variational encoder to approximate the posterior distribution of the local parameters of the VHE image encoder, and stochastic gradient descent (SGD) to estimate the parameters of the GAN image generator and discriminator, we develop a SG-MCMC/VHE/SGD hybrid inference algorithm to jointly train the components of VHEGAN. Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

We give a detailed hybrid SGMCMC/SGD algorithm for VHEGAN in this section, to realize learning end-to-end.

The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively.

Input: Table 1: Accuracy (%) of zero-shot classification on CUB2011-hard, CUB2011-easy, and Flower Datasets. Note that some of them are attribute-based methods but applicable in our setting by replacing attribute vectors with text features (labeled by ∗), as discussed in Elhoseiny et al. (2017b). ZSLPP and GAZSL use a welldefined part detection features for images. All results on CUB2011 and the results of WAC on Flower come from Elhoseiny et al. (2017b); Qiao et al. (2016); Elhoseiny et al. (2017a), with all the others obtained by running the code provided by the original authors.

Input: Figure 6: Image and document samples jointly generated from a three-layer VHEGAN.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the claim is moderately overstated. While VHEGAN does demonstrate capabilities in text-image generation and retrieval as shown in various experiments (Fig. 6 showing joint text-image generation, and other examples of text-to-image and image-to-text transformations), the claim overstates the ease and effectiveness across "a variety of settings." The evidence shows that performance varies across tasks and the model has certain limitations. The review comments also highlight that the method combines existing models without clear justification for architectural choices.

<score>3</score>

Justification:
1. The evidence demonstrates VHEGAN can perform:
- Text-to-image generation (shown in experiments with CUB2011 and Flower datasets)
- Image-to-text generation (Fig. 4 shows keyword generation)
- Image retrieval from text queries (Fig. 3)
- Joint text-image pair generation (Fig. 6)

2. However, the claim overstates by:
- Not acknowledging limitations or varying performance across tasks
- Using broad phrasing "can be used in a variety of settings" without qualifying effectiveness
- Not mentioning that some capabilities require additional steps (e.g., using Gibbs sampling for text-to-image generation)
- As noted in the review, architectural choices aren't well justified

3. The evidence does support basic functionality but not as comprehensively as the claim suggests. A more measured statement would specify the demonstrated capabilities while acknowledging limitations.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

1) text to image: generating images with the GAN image generator, or retrieving images, given the conditional posterior of a document under the VHE text decoder;

The evidence to be evaluated is:Integrating VHE and GAN, the proposed VHEGAN, as shown in Fig. 1 (c), tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document, under the Poisson GBN text decoder, be the distribution of the random noise fed into the GAN image generator; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder; and the GAN discriminator tries to discriminate between that selected image and the generated one. All the three components that are structurally coupled with each other will be jointly optimized.

Choosing a deep generative model as the text decoder distinguishes VHE from Gomez et al. (2017) that uses LDA (Blei et al., 2003), a "shallow" probabilistic model, and Ba et al. (2015) that uses non-probabilistic black-box DNNs to learn class-specific document features.

In this paper, we leverage WUDVE as the variational inference network to approximate the true posterior distribution \(p(\mathbf{\theta}\,|\,\mathbf{t},\mathbf{\Phi})\), where \(\mathbf{\theta}=\{\mathbf{\theta}^{(l)}\}_{1,L}\) denotes the set of local parameters, but change the input of WUDVE from the bag-of-words vector \(\mathbf{t}\) into an image \(\mathbf{x}\) associated with \(\mathbf{t}\).

where, with \(f(\mathbf{x})\) representing a feature vector deterministically projected from \(\mathbf{x}\) via an off-the-shelf pre-trained deep neural network \(f(\mathbf{\cdot})\), we define

Given \(\mathbf{\Phi}\) of the Poisson GBN, to approximate the posterior distribution of the local parameters \(\mathbf{\theta}\) given text description \(\mathbf{t}\), we use WVDVE \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\), whose input is an image \(\mathbf{x}\), and learn its parameters \(\mathbf{\Omega}\) by maximizing the evidence lower bound (ELBO) as \[\max_{\mathbf{\Omega}}L_{\text{VHE}}(\mathbf{\Omega})=\mathbb{E}_{(\mathbf{x},\mathbf{t}) \sim p_{data}(\mathbf{x},\mathbf{t})}\left\{\mathbb{E}_{q_{\mathbf{\Omega}}(\mathbf{\theta} \,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln p\left(\mathbf{t}\,|\,\mathbf{\Phi}^{(1)},\mathbf{\theta }^{(1)}\right)-\ln\frac{q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}{p( \mathbf{\theta}\,|\,\mathbf{\Phi})}\right]\right\}, \tag{4}\]

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN. _Both VAE and GAN are widely used for image generation, but we find in experiments that VAE is not expressive enough for mid/high-resolution RGB images (64 \(\times\) 64 resolution in our task) to clearly help the learning of VHE, whereas GAN does better for this purpose. Thus, we combine GAN and VHE in a novel way via a shared latent space, to further explore the relationships between two different modes for ZSL.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator.

\[\min_{G}\max_{D}L_{\text{GAN}}(D,G)=\min_{G}\max_{D}\mathbb{E}_{\mathbf{x}\sim p_ {data}(\mathbf{x})}\left\{\ln D(\mathbf{x})-\mathbb{E}_{\mathbf{\theta}\sim q_{\mathbf{\Omega} }(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln D(G(\mathbf{\theta}))\right]\right\} \tag{6}\]

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities.

Note we choose the GAN generator and discriminator to be the same as those of deep convolutional GAN of Radford et al. (2016). Other more recent GANs, such as Wasserstein GAN (Arjovsky et al., 2017), Wasserstein GAN with gradient penalty (Gulrajani et al., 2017), spectral norm GAN (Miyato et al., 2018), and progressive growing GAN (Karras et al., 2018), may be combined with the VHE to construct even more expressive VHEGANs.

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7).

To remedy these drawbacks, text-based ZSL have been developed using easily accessible textural descriptions such as Wikipedia.

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning.

In Reed et al. (2016) and Gomez et al. (2017), a convolutional GAN and a CNN model are used to focus on describing image from text and translating image into textual features in one direction, respectively. Both these two approaches are two-step models with the additional textual feature extraction realized by LSTM and "shallow" LDA, respectively, whereas the VHEGAN can realize joint optimization with hybrid SGMCMC/VHE/SGD algorithm. Besides, VHEGAN is able to realize multi-directed transformation, where ZSL can be seen as predicting the BOW textual features from the images.

To solve the problem of complete random generate process in GAN, conditional GANs (CGAN) (Mirza & Osindero, 2014; Reed et al., 2016) are alsodeveloped, where the conditions are labels, or attributes learned as point estimations separately with GAN, while VHEGAN learns a distributed condition jointly with a VAE model.

Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

We make comparison between a variety of methods that are suitable for text-ZSL, as summarized in Table 1, using Top-1 accuracy (some methods also provide Top-5 accuracy on CUB2011-easy) for the unseen classes.

except for the proposed VHEGAN that aims to find a shared semantically meaningful latent space between the image and text modalities, none of the other methods have generative models for both modalities, regardless of whether they learn a classifier or a distance metric in a latent space for ZSL.

For VHEs and VHEGANs with different network structures, Table 1 shows that given the same structure on the text decoder and image encoder, each VHEGAN consistently outperforms its VHE counterpart, suggesting the advantage of a joint generation of both the image and text modalities.

**Text to image generation:** Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images. Given example encyclopedia documents for the unseen classes, we follow this approach to generate random images. We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images). We repeat the text-to-image generation experiment on Flower to generate random images given textual captions for three different unseen classes. As shown in Fig. 1(b), VHEGAN successfully generates flower images that semantically match their corresponding textual descriptions and visually resemble their corresponding example real images.

Giving the textural description \(\mathbf{t}\) of an unseen class as the text query, we retrieve the top five images ranked by the cosine distances between \(\mathbf{t}\) and \(\hat{\mathbf{t}}_{i}\).

With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image. **Latent space interpolation:** In order to understand the learned manifold in the latent space, given two texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), after drawing \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) from their own conditional posteriors using Gibbs sampling, we interpolate between \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) and use these interpolated latent variables to generate both the images via the GAN image generator and texts via the VHE text decoder. We show the results in Fig. 5, where the first and last columns show the true texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), and the images generated from \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\), respectively, while the second to third columns show the generated texts and images from the interpolated \(\mathbf{\theta}\)'s. The strong correspondences between the generated images and texts and smooth changes between adjacent columns suggest that the VHEGAN latent space is semantically meaningful for both the image and text modalities.

Given a random draw, the concatenation of \(\{\mathbf{\theta}^{(l)}\}_{l=1}^{L}\) is fed into the GAN image generator to generate a corresponding image.

In Fig. 7, we present the real images \(\mathbf{x}\) and their VHEGANregenerations \(G\left(q_{\boldsymbol{\Omega}}\left(\boldsymbol{\theta}\,|\,\boldsymbol{\Phi}, \boldsymbol{x}\right)\right)\), which suggest that the VHEGAN can use its GAN image generator to regenerate random images that more or less resemble the original real image fed into the VHE image encoder.

For each column in Fig. 7(b), conditioning on a textual description, we show the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all layers, respectively, as the latent space fed into the GAN image generator.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space. With a stochastic-gradient MCMC algorithm to sample the global parameters of the VHE text decoder, a reparameterized Weibull distribution based variational encoder to approximate the posterior distribution of the local parameters of the VHE image encoder, and stochastic gradient descent (SGD) to estimate the parameters of the GAN image generator and discriminator, we develop a SG-MCMC/VHE/SGD hybrid inference algorithm to jointly train the components of VHEGAN. Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

Generative adversarial nets.

Improved training of Wasserstein GANs.

We give a detailed hybrid SGMCMC/SGD algorithm for VHEGAN in this section, to realize learning end-to-end.

In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively.

Image Generation Network shown in Fig. 10 : \(\mathbf{\theta}^{(l)}\rightarrow\hat{\mathbf{x}}\); \(l=1,2,3\)\(1\times\) { fully-connected layer with \(4\times 4\times 64\times 4\) units, RELU activation and batch normalization.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 128 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 64 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2.}

Input: Table 1: Accuracy (%) of zero-shot classification on CUB2011-hard, CUB2011-easy, and Flower Datasets. Note that some of them are attribute-based methods but applicable in our setting by replacing attribute vectors with text features (labeled by ∗), as discussed in Elhoseiny et al. (2017b). ZSLPP and GAZSL use a welldefined part detection features for images. All results on CUB2011 and the results of WAC on Flower come from Elhoseiny et al. (2017b); Qiao et al. (2016); Elhoseiny et al. (2017a), with all the others obtained by running the code provided by the original authors.

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 2: Class specific image generation based on the textual descriptions of unseen classes. (a) CUB2011easy and Flower data, where the first row shows real samples, while the others generated images; (b) Flower data, where the top row shows the representative textual captions and two real images of the same class.

Input: Figure 6: Image and document samples jointly generated from a three-layer VHEGAN.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence carefully, I assess that the claim about text-to-image generation capabilities is moderately well-stated but has some overstatement. The evidence shows VHEGAN can indeed generate images from textual descriptions and retrieve images, demonstrated through multiple examples in figures showing reasonable image generation results. However, the text admits limitations, noting that "VAE is not expressive enough for mid/high-resolution RGB images" and the generated images, while semantically relevant, don't perfectly match real images. The model requires specific conditions and preprocessing steps that aren't mentioned in the simplified claim.

<score>2</score>

Justification:
1. The evidence supports the core capability claimed, showing both generation and retrieval functionality. Figures 1a/b demonstrate text-to-image generation for unseen classes.

2. The paper is transparent about limitations and requirements:
- Limited to 64x64 resolution images
- Requires preprocessing and specific model architectures
- Uses conditional posteriors derived from training data
- Results show semantic similarity but not perfect reproduction

3. The claim could be more precise by:
- Specifying resolution limitations
- Noting the need for training data/preprocessing
- Acknowledging the semantic rather than exact nature of generated images

The claim is relatively well-stated but would benefit from these qualifications, hence a score of 2.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

1) text to image: generating images with the GAN image generator, or retrieving images, given the conditional posterior of a document under the VHE text decoder;

The review comment to be evaluated is:

Paper Summary: This paper studies the zero-shot learning problem with deep generative models. More specifically, it proposed a hybrid framework that combines VAEs (more precisely, the variational hetero-encoder or VHE) and GANs all together. The entire model is composed of an image encoder (Weibull upward-downward variational encoder), a text decoder (Poisson Gamma belief network), and an image generator (generative adversarial network). Once learned, the generative models can be directly used for zero-shot classification and various image generation applications. In the experiments, two benchmark datasets CUB and Oxford-Flowers are used.

==
Novelty/Significance:
Zero-shot learning is a challenging task and he main motivation of the paper (using generative model) is interesting. The text representation in the paper is simply bag-of-words which limits the application to some extent. In a broader context, image captioning using generative model seems quite relevant.

Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space, Wang et al. In NIPS 2017.

==
Quality:
Overall, reviewer feels this is a very interesting work. However, the results from the paper is quite mixed. It is not yet convincing whether the proposed approach is the state-of-the-art in zero-shot learning or text-to-image generation.

First, this paper demonstrates the power of generative models in text-to-image generation and other applications. However, reviewer feels that the zero-shot classification result is weak. In Table 1 and Table 2, it seems that GAZSL (Zhu et al. 2018) outperforms the proposed approach.

Q1: In Table 2, is it possible to report the top-5 accuracy on CUB-easy and top-1 accuracy on Oxford-Flower dataset? Otherwise, it is not very convincing that proposed approach is better than the state-of-the-art approach GAZSL.

Second, the text-to-image generation results look reasonably good. But the resolution and quality of generated images are far from state-of-the-art. One suggestion is to train the VHE model with an improved image generator.

StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks, Zhang et al. In CVPR 2017.

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks, Xu et al. In CVPR 2018.

Also, reviewer would expect to see an improved image generator can lead to a better ZSL performance.

Typo: In the title: Zero-Short → Zero-Shot.

The evidence to be evaluated is:Integrating VHE and GAN, the proposed VHEGAN, as shown in Fig. 1 (c), tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document, under the Poisson GBN text decoder, be the distribution of the random noise fed into the GAN image generator; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder; and the GAN discriminator tries to discriminate between that selected image and the generated one. All the three components that are structurally coupled with each other will be jointly optimized.

Choosing a deep generative model as the text decoder distinguishes VHE from Gomez et al. (2017) that uses LDA (Blei et al., 2003), a "shallow" probabilistic model, and Ba et al. (2015) that uses non-probabilistic black-box DNNs to learn class-specific document features.

In this paper, we leverage WUDVE as the variational inference network to approximate the true posterior distribution \(p(\mathbf{\theta}\,|\,\mathbf{t},\mathbf{\Phi})\), where \(\mathbf{\theta}=\{\mathbf{\theta}^{(l)}\}_{1,L}\) denotes the set of local parameters, but change the input of WUDVE from the bag-of-words vector \(\mathbf{t}\) into an image \(\mathbf{x}\) associated with \(\mathbf{t}\).

where, with \(f(\mathbf{x})\) representing a feature vector deterministically projected from \(\mathbf{x}\) via an off-the-shelf pre-trained deep neural network \(f(\mathbf{\cdot})\), we define

Given \(\mathbf{\Phi}\) of the Poisson GBN, to approximate the posterior distribution of the local parameters \(\mathbf{\theta}\) given text description \(\mathbf{t}\), we use WVDVE \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\), whose input is an image \(\mathbf{x}\), and learn its parameters \(\mathbf{\Omega}\) by maximizing the evidence lower bound (ELBO) as \[\max_{\mathbf{\Omega}}L_{\text{VHE}}(\mathbf{\Omega})=\mathbb{E}_{(\mathbf{x},\mathbf{t}) \sim p_{data}(\mathbf{x},\mathbf{t})}\left\{\mathbb{E}_{q_{\mathbf{\Omega}}(\mathbf{\theta} \,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln p\left(\mathbf{t}\,|\,\mathbf{\Phi}^{(1)},\mathbf{\theta }^{(1)}\right)-\ln\frac{q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}{p( \mathbf{\theta}\,|\,\mathbf{\Phi})}\right]\right\}, \tag{4}\]

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN. _Both VAE and GAN are widely used for image generation, but we find in experiments that VAE is not expressive enough for mid/high-resolution RGB images (64 \(\times\) 64 resolution in our task) to clearly help the learning of VHE, whereas GAN does better for this purpose. Thus, we combine GAN and VHE in a novel way via a shared latent space, to further explore the relationships between two different modes for ZSL.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator.

\[\min_{G}\max_{D}L_{\text{GAN}}(D,G)=\min_{G}\max_{D}\mathbb{E}_{\mathbf{x}\sim p_ {data}(\mathbf{x})}\left\{\ln D(\mathbf{x})-\mathbb{E}_{\mathbf{\theta}\sim q_{\mathbf{\Omega} }(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln D(G(\mathbf{\theta}))\right]\right\} \tag{6}\]

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities.

Note we choose the GAN generator and discriminator to be the same as those of deep convolutional GAN of Radford et al. (2016). Other more recent GANs, such as Wasserstein GAN (Arjovsky et al., 2017), Wasserstein GAN with gradient penalty (Gulrajani et al., 2017), spectral norm GAN (Miyato et al., 2018), and progressive growing GAN (Karras et al., 2018), may be combined with the VHE to construct even more expressive VHEGANs.

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7).

To remedy these drawbacks, text-based ZSL have been developed using easily accessible textural descriptions such as Wikipedia.

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning.

In Reed et al. (2016) and Gomez et al. (2017), a convolutional GAN and a CNN model are used to focus on describing image from text and translating image into textual features in one direction, respectively. Both these two approaches are two-step models with the additional textual feature extraction realized by LSTM and "shallow" LDA, respectively, whereas the VHEGAN can realize joint optimization with hybrid SGMCMC/VHE/SGD algorithm. Besides, VHEGAN is able to realize multi-directed transformation, where ZSL can be seen as predicting the BOW textual features from the images.

To solve the problem of complete random generate process in GAN, conditional GANs (CGAN) (Mirza & Osindero, 2014; Reed et al., 2016) are alsodeveloped, where the conditions are labels, or attributes learned as point estimations separately with GAN, while VHEGAN learns a distributed condition jointly with a VAE model.

Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

We make comparison between a variety of methods that are suitable for text-ZSL, as summarized in Table 1, using Top-1 accuracy (some methods also provide Top-5 accuracy on CUB2011-easy) for the unseen classes.

except for the proposed VHEGAN that aims to find a shared semantically meaningful latent space between the image and text modalities, none of the other methods have generative models for both modalities, regardless of whether they learn a classifier or a distance metric in a latent space for ZSL.

For VHEs and VHEGANs with different network structures, Table 1 shows that given the same structure on the text decoder and image encoder, each VHEGAN consistently outperforms its VHE counterpart, suggesting the advantage of a joint generation of both the image and text modalities.

**Text to image generation:** Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images. Given example encyclopedia documents for the unseen classes, we follow this approach to generate random images. We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images). We repeat the text-to-image generation experiment on Flower to generate random images given textual captions for three different unseen classes. As shown in Fig. 1(b), VHEGAN successfully generates flower images that semantically match their corresponding textual descriptions and visually resemble their corresponding example real images.

Giving the textural description \(\mathbf{t}\) of an unseen class as the text query, we retrieve the top five images ranked by the cosine distances between \(\mathbf{t}\) and \(\hat{\mathbf{t}}_{i}\).

With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image. **Latent space interpolation:** In order to understand the learned manifold in the latent space, given two texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), after drawing \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) from their own conditional posteriors using Gibbs sampling, we interpolate between \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) and use these interpolated latent variables to generate both the images via the GAN image generator and texts via the VHE text decoder. We show the results in Fig. 5, where the first and last columns show the true texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), and the images generated from \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\), respectively, while the second to third columns show the generated texts and images from the interpolated \(\mathbf{\theta}\)'s. The strong correspondences between the generated images and texts and smooth changes between adjacent columns suggest that the VHEGAN latent space is semantically meaningful for both the image and text modalities.

Given a random draw, the concatenation of \(\{\mathbf{\theta}^{(l)}\}_{l=1}^{L}\) is fed into the GAN image generator to generate a corresponding image.

In Fig. 7, we present the real images \(\mathbf{x}\) and their VHEGANregenerations \(G\left(q_{\boldsymbol{\Omega}}\left(\boldsymbol{\theta}\,|\,\boldsymbol{\Phi}, \boldsymbol{x}\right)\right)\), which suggest that the VHEGAN can use its GAN image generator to regenerate random images that more or less resemble the original real image fed into the VHE image encoder.

For each column in Fig. 7(b), conditioning on a textual description, we show the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all layers, respectively, as the latent space fed into the GAN image generator.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space. With a stochastic-gradient MCMC algorithm to sample the global parameters of the VHE text decoder, a reparameterized Weibull distribution based variational encoder to approximate the posterior distribution of the local parameters of the VHE image encoder, and stochastic gradient descent (SGD) to estimate the parameters of the GAN image generator and discriminator, we develop a SG-MCMC/VHE/SGD hybrid inference algorithm to jointly train the components of VHEGAN. Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

Generative adversarial nets.

Improved training of Wasserstein GANs.

We give a detailed hybrid SGMCMC/SGD algorithm for VHEGAN in this section, to realize learning end-to-end.

In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively.

Image Generation Network shown in Fig. 10 : \(\mathbf{\theta}^{(l)}\rightarrow\hat{\mathbf{x}}\); \(l=1,2,3\)\(1\times\) { fully-connected layer with \(4\times 4\times 64\times 4\) units, RELU activation and batch normalization.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 128 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 64 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2.}

Input: Table 1: Accuracy (%) of zero-shot classification on CUB2011-hard, CUB2011-easy, and Flower Datasets. Note that some of them are attribute-based methods but applicable in our setting by replacing attribute vectors with text features (labeled by ∗), as discussed in Elhoseiny et al. (2017b). ZSLPP and GAZSL use a welldefined part detection features for images. All results on CUB2011 and the results of WAC on Flower come from Elhoseiny et al. (2017b); Qiao et al. (2016); Elhoseiny et al. (2017a), with all the others obtained by running the code provided by the original authors.

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 2: Class specific image generation based on the textual descriptions of unseen classes. (a) CUB2011easy and Flower data, where the first row shows real samples, while the others generated images; (b) Flower data, where the top row shows the representative textual captions and two real images of the same class.

Input: Figure 6: Image and document samples jointly generated from a three-layer VHEGAN.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, I'll evaluate the claim about text-to-image generation capabilities:

Summary: The evidence shows that VHEGAN can generate images from textual descriptions through multiple approaches (GAN generation and image retrieval), with demonstrated results on CUB and Oxford-Flowers datasets. However, the review comments note that "the resolution and quality of generated images are far from state-of-the-art" and suggest using improved image generators. The paper itself acknowledges limitations by noting that other more recent GANs could be used to construct "even more expressive VHEGANs." The results show the model can generate semantically relevant images, but with clear quality limitations.

The claim appears to be factually accurate but somewhat understated, as it doesn't oversell the capabilities and simply states the functional abilities without making grandiose claims about performance or quality. The claim is well-grounded in the presented evidence and acknowledges the technical approach without overstating its effectiveness.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

1) text to image: generating images with the GAN image generator, or retrieving images, given the conditional posterior of a document under the VHE text decoder;

The review comment to be evaluated is:

[Paper Summary]
This work suggests a new model incorporating deep topic model (text decoder),  VHE (image encoder), and GAN. The topic model and the VHE shares the topic parameters, and the GAN generate an image regarding the topic. Then, for ZSL, the image is encoded to corresponding topic parameters, and the parameter can tell which text description (unseen) is matched with the highest probability. GAN model is used to generate an image given the topic distribution. During the training of the GAN, the VHE and topic model is jointly trained and can enhance the ZSL performance marginally.

[pros]
- This work successfully incorporated the topic model and image encoding/decoding. All the individual parts are already given, but I think incorporating them in terms of a unified probabilistic model is also meaningful for this field.
- This work shows superior performance on the image to text ZSL problem.
- This work mapped the text to image, image to text mapping in a generative manner.

[cons]
- The problem is only valid when the unseen class distribution is very similar to the given classes. For example, the text description of unseen classes should be well represented to the topics from seen classes.
- It is doubtful that this corresponds to the term zero-shot learning; dealing with the case that the unseen class and the seen class are notably different from each other.
- Similarly, GAN learns images from the seen classes, and by nature, GAN would not generate the proper images of the unseen class if the image distribution of the unseen class is different to the already seen class. In the paper, the classes are very similar to each other (birds, flowers) and that would be the reason GAN worked in this model.
- (minor) The likelihood of the text (image) given topic should be provided and compared to the existing models.

[Summary]
- The reviewer is personally interested in the proposal of the work, but concern that ZSL is difficult to be the main target of the paper because the model can only deal with the classes with (very) similar semantics, and this is the main reason for the rating. The testing with more diverse class should be given, or solid explanation of the mentioned problem would be required.

The evidence to be evaluated is:Integrating VHE and GAN, the proposed VHEGAN, as shown in Fig. 1 (c), tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document, under the Poisson GBN text decoder, be the distribution of the random noise fed into the GAN image generator; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder; and the GAN discriminator tries to discriminate between that selected image and the generated one. All the three components that are structurally coupled with each other will be jointly optimized.

Choosing a deep generative model as the text decoder distinguishes VHE from Gomez et al. (2017) that uses LDA (Blei et al., 2003), a "shallow" probabilistic model, and Ba et al. (2015) that uses non-probabilistic black-box DNNs to learn class-specific document features.

In this paper, we leverage WUDVE as the variational inference network to approximate the true posterior distribution \(p(\mathbf{\theta}\,|\,\mathbf{t},\mathbf{\Phi})\), where \(\mathbf{\theta}=\{\mathbf{\theta}^{(l)}\}_{1,L}\) denotes the set of local parameters, but change the input of WUDVE from the bag-of-words vector \(\mathbf{t}\) into an image \(\mathbf{x}\) associated with \(\mathbf{t}\).

where, with \(f(\mathbf{x})\) representing a feature vector deterministically projected from \(\mathbf{x}\) via an off-the-shelf pre-trained deep neural network \(f(\mathbf{\cdot})\), we define

Given \(\mathbf{\Phi}\) of the Poisson GBN, to approximate the posterior distribution of the local parameters \(\mathbf{\theta}\) given text description \(\mathbf{t}\), we use WVDVE \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\), whose input is an image \(\mathbf{x}\), and learn its parameters \(\mathbf{\Omega}\) by maximizing the evidence lower bound (ELBO) as \[\max_{\mathbf{\Omega}}L_{\text{VHE}}(\mathbf{\Omega})=\mathbb{E}_{(\mathbf{x},\mathbf{t}) \sim p_{data}(\mathbf{x},\mathbf{t})}\left\{\mathbb{E}_{q_{\mathbf{\Omega}}(\mathbf{\theta} \,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln p\left(\mathbf{t}\,|\,\mathbf{\Phi}^{(1)},\mathbf{\theta }^{(1)}\right)-\ln\frac{q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}{p( \mathbf{\theta}\,|\,\mathbf{\Phi})}\right]\right\}, \tag{4}\]

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN. _Both VAE and GAN are widely used for image generation, but we find in experiments that VAE is not expressive enough for mid/high-resolution RGB images (64 \(\times\) 64 resolution in our task) to clearly help the learning of VHE, whereas GAN does better for this purpose. Thus, we combine GAN and VHE in a novel way via a shared latent space, to further explore the relationships between two different modes for ZSL.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator.

\[\min_{G}\max_{D}L_{\text{GAN}}(D,G)=\min_{G}\max_{D}\mathbb{E}_{\mathbf{x}\sim p_ {data}(\mathbf{x})}\left\{\ln D(\mathbf{x})-\mathbb{E}_{\mathbf{\theta}\sim q_{\mathbf{\Omega} }(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln D(G(\mathbf{\theta}))\right]\right\} \tag{6}\]

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities.

Note we choose the GAN generator and discriminator to be the same as those of deep convolutional GAN of Radford et al. (2016). Other more recent GANs, such as Wasserstein GAN (Arjovsky et al., 2017), Wasserstein GAN with gradient penalty (Gulrajani et al., 2017), spectral norm GAN (Miyato et al., 2018), and progressive growing GAN (Karras et al., 2018), may be combined with the VHE to construct even more expressive VHEGANs.

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7).

To remedy these drawbacks, text-based ZSL have been developed using easily accessible textural descriptions such as Wikipedia.

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning.

In Reed et al. (2016) and Gomez et al. (2017), a convolutional GAN and a CNN model are used to focus on describing image from text and translating image into textual features in one direction, respectively. Both these two approaches are two-step models with the additional textual feature extraction realized by LSTM and "shallow" LDA, respectively, whereas the VHEGAN can realize joint optimization with hybrid SGMCMC/VHE/SGD algorithm. Besides, VHEGAN is able to realize multi-directed transformation, where ZSL can be seen as predicting the BOW textual features from the images.

To solve the problem of complete random generate process in GAN, conditional GANs (CGAN) (Mirza & Osindero, 2014; Reed et al., 2016) are alsodeveloped, where the conditions are labels, or attributes learned as point estimations separately with GAN, while VHEGAN learns a distributed condition jointly with a VAE model.

Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

We make comparison between a variety of methods that are suitable for text-ZSL, as summarized in Table 1, using Top-1 accuracy (some methods also provide Top-5 accuracy on CUB2011-easy) for the unseen classes.

except for the proposed VHEGAN that aims to find a shared semantically meaningful latent space between the image and text modalities, none of the other methods have generative models for both modalities, regardless of whether they learn a classifier or a distance metric in a latent space for ZSL.

For VHEs and VHEGANs with different network structures, Table 1 shows that given the same structure on the text decoder and image encoder, each VHEGAN consistently outperforms its VHE counterpart, suggesting the advantage of a joint generation of both the image and text modalities.

**Text to image generation:** Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images. Given example encyclopedia documents for the unseen classes, we follow this approach to generate random images. We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images). We repeat the text-to-image generation experiment on Flower to generate random images given textual captions for three different unseen classes. As shown in Fig. 1(b), VHEGAN successfully generates flower images that semantically match their corresponding textual descriptions and visually resemble their corresponding example real images.

Giving the textural description \(\mathbf{t}\) of an unseen class as the text query, we retrieve the top five images ranked by the cosine distances between \(\mathbf{t}\) and \(\hat{\mathbf{t}}_{i}\).

With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image. **Latent space interpolation:** In order to understand the learned manifold in the latent space, given two texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), after drawing \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) from their own conditional posteriors using Gibbs sampling, we interpolate between \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) and use these interpolated latent variables to generate both the images via the GAN image generator and texts via the VHE text decoder. We show the results in Fig. 5, where the first and last columns show the true texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), and the images generated from \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\), respectively, while the second to third columns show the generated texts and images from the interpolated \(\mathbf{\theta}\)'s. The strong correspondences between the generated images and texts and smooth changes between adjacent columns suggest that the VHEGAN latent space is semantically meaningful for both the image and text modalities.

Given a random draw, the concatenation of \(\{\mathbf{\theta}^{(l)}\}_{l=1}^{L}\) is fed into the GAN image generator to generate a corresponding image.

In Fig. 7, we present the real images \(\mathbf{x}\) and their VHEGANregenerations \(G\left(q_{\boldsymbol{\Omega}}\left(\boldsymbol{\theta}\,|\,\boldsymbol{\Phi}, \boldsymbol{x}\right)\right)\), which suggest that the VHEGAN can use its GAN image generator to regenerate random images that more or less resemble the original real image fed into the VHE image encoder.

For each column in Fig. 7(b), conditioning on a textual description, we show the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all layers, respectively, as the latent space fed into the GAN image generator.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space. With a stochastic-gradient MCMC algorithm to sample the global parameters of the VHE text decoder, a reparameterized Weibull distribution based variational encoder to approximate the posterior distribution of the local parameters of the VHE image encoder, and stochastic gradient descent (SGD) to estimate the parameters of the GAN image generator and discriminator, we develop a SG-MCMC/VHE/SGD hybrid inference algorithm to jointly train the components of VHEGAN. Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

Generative adversarial nets.

Improved training of Wasserstein GANs.

We give a detailed hybrid SGMCMC/SGD algorithm for VHEGAN in this section, to realize learning end-to-end.

In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively.

Image Generation Network shown in Fig. 10 : \(\mathbf{\theta}^{(l)}\rightarrow\hat{\mathbf{x}}\); \(l=1,2,3\)\(1\times\) { fully-connected layer with \(4\times 4\times 64\times 4\) units, RELU activation and batch normalization.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 128 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 64 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2.}

Input: Table 1: Accuracy (%) of zero-shot classification on CUB2011-hard, CUB2011-easy, and Flower Datasets. Note that some of them are attribute-based methods but applicable in our setting by replacing attribute vectors with text features (labeled by ∗), as discussed in Elhoseiny et al. (2017b). ZSLPP and GAZSL use a welldefined part detection features for images. All results on CUB2011 and the results of WAC on Flower come from Elhoseiny et al. (2017b); Qiao et al. (2016); Elhoseiny et al. (2017a), with all the others obtained by running the code provided by the original authors.

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 2: Class specific image generation based on the textual descriptions of unseen classes. (a) CUB2011easy and Flower data, where the first row shows real samples, while the others generated images; (b) Flower data, where the top row shows the representative textual captions and two real images of the same class.

Input: Figure 6: Image and document samples jointly generated from a three-layer VHEGAN.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find that the claim about text-to-image capabilities is well-supported but slightly overstated. While VHEGAN demonstrates text-to-image generation and retrieval abilities, there are some important limitations:

1. The system's performance is heavily dependent on semantic similarity between seen and unseen classes, as noted in the review comments.
2. The quality of generated images appears moderate rather than high-fidelity.
3. The paper acknowledges limitations by mentioning that "VAE is not expressive enough for mid/high-resolution RGB images" and suggests that newer GAN architectures could improve results.
4. The evidence shows successful text-to-image generation but primarily on limited domains (birds, flowers) with similar visual characteristics.

The claim is functionally accurate but could benefit from acknowledging these constraints.

<score>2</score>

Justification:
The claim reasonably describes the model's capabilities but would be more well-stated if it acknowledged:
1. Limited scope of successful generation to visually similar classes
2. Resolution/quality constraints
3. Dependence on semantic similarity between seen/unseen classes

The evidence from figures and experimental results supports that the system can indeed perform the claimed tasks, even if with limitations, making this a relatively well-stated claim with minor overstatement.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

1) text to image: generating images with the GAN image generator, or retrieving images, given the conditional posterior of a document under the VHE text decoder;

The review comment to be evaluated is:

This paper developed a generative model to perform simultaneous embedding/generation of images/texts, with application to zero-shot learning. The experiments are extensive.

The novelty of this work is lacking.
The proposed method consists of a bag of existing models proposed by previous works.
But why using a certain model is not justified or explanined.
For example, for image generation, why use GAN instead of VAE.
For text encoding, why use PGBN, instead of recurrent VAE.

The method seems deteched from the problem of ZSL.
Throughout the paper, the authors mostly talk about how to perform joint embedding of texts and images. They give ZSL a touch, but as a side thing.
I would suggest the authors to position this work as a text/iamge embedding/generation work. Then use ZSL as an application.

The writing needs to be significantly improved. In the first paragraph describing the problem of ZSL, the authors end up with talking about the evaluation metric of ZSL.

The evidence to be evaluated is:Integrating VHE and GAN, the proposed VHEGAN, as shown in Fig. 1 (c), tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document, under the Poisson GBN text decoder, be the distribution of the random noise fed into the GAN image generator; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder; and the GAN discriminator tries to discriminate between that selected image and the generated one. All the three components that are structurally coupled with each other will be jointly optimized.

Choosing a deep generative model as the text decoder distinguishes VHE from Gomez et al. (2017) that uses LDA (Blei et al., 2003), a "shallow" probabilistic model, and Ba et al. (2015) that uses non-probabilistic black-box DNNs to learn class-specific document features.

In this paper, we leverage WUDVE as the variational inference network to approximate the true posterior distribution \(p(\mathbf{\theta}\,|\,\mathbf{t},\mathbf{\Phi})\), where \(\mathbf{\theta}=\{\mathbf{\theta}^{(l)}\}_{1,L}\) denotes the set of local parameters, but change the input of WUDVE from the bag-of-words vector \(\mathbf{t}\) into an image \(\mathbf{x}\) associated with \(\mathbf{t}\).

where, with \(f(\mathbf{x})\) representing a feature vector deterministically projected from \(\mathbf{x}\) via an off-the-shelf pre-trained deep neural network \(f(\mathbf{\cdot})\), we define

Given \(\mathbf{\Phi}\) of the Poisson GBN, to approximate the posterior distribution of the local parameters \(\mathbf{\theta}\) given text description \(\mathbf{t}\), we use WVDVE \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\), whose input is an image \(\mathbf{x}\), and learn its parameters \(\mathbf{\Omega}\) by maximizing the evidence lower bound (ELBO) as \[\max_{\mathbf{\Omega}}L_{\text{VHE}}(\mathbf{\Omega})=\mathbb{E}_{(\mathbf{x},\mathbf{t}) \sim p_{data}(\mathbf{x},\mathbf{t})}\left\{\mathbb{E}_{q_{\mathbf{\Omega}}(\mathbf{\theta} \,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln p\left(\mathbf{t}\,|\,\mathbf{\Phi}^{(1)},\mathbf{\theta }^{(1)}\right)-\ln\frac{q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}{p( \mathbf{\theta}\,|\,\mathbf{\Phi})}\right]\right\}, \tag{4}\]

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN. _Both VAE and GAN are widely used for image generation, but we find in experiments that VAE is not expressive enough for mid/high-resolution RGB images (64 \(\times\) 64 resolution in our task) to clearly help the learning of VHE, whereas GAN does better for this purpose. Thus, we combine GAN and VHE in a novel way via a shared latent space, to further explore the relationships between two different modes for ZSL.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator.

\[\min_{G}\max_{D}L_{\text{GAN}}(D,G)=\min_{G}\max_{D}\mathbb{E}_{\mathbf{x}\sim p_ {data}(\mathbf{x})}\left\{\ln D(\mathbf{x})-\mathbb{E}_{\mathbf{\theta}\sim q_{\mathbf{\Omega} }(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln D(G(\mathbf{\theta}))\right]\right\} \tag{6}\]

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities.

Note we choose the GAN generator and discriminator to be the same as those of deep convolutional GAN of Radford et al. (2016). Other more recent GANs, such as Wasserstein GAN (Arjovsky et al., 2017), Wasserstein GAN with gradient penalty (Gulrajani et al., 2017), spectral norm GAN (Miyato et al., 2018), and progressive growing GAN (Karras et al., 2018), may be combined with the VHE to construct even more expressive VHEGANs.

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7).

To remedy these drawbacks, text-based ZSL have been developed using easily accessible textural descriptions such as Wikipedia.

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning.

In Reed et al. (2016) and Gomez et al. (2017), a convolutional GAN and a CNN model are used to focus on describing image from text and translating image into textual features in one direction, respectively. Both these two approaches are two-step models with the additional textual feature extraction realized by LSTM and "shallow" LDA, respectively, whereas the VHEGAN can realize joint optimization with hybrid SGMCMC/VHE/SGD algorithm. Besides, VHEGAN is able to realize multi-directed transformation, where ZSL can be seen as predicting the BOW textual features from the images.

To solve the problem of complete random generate process in GAN, conditional GANs (CGAN) (Mirza & Osindero, 2014; Reed et al., 2016) are alsodeveloped, where the conditions are labels, or attributes learned as point estimations separately with GAN, while VHEGAN learns a distributed condition jointly with a VAE model.

Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

We make comparison between a variety of methods that are suitable for text-ZSL, as summarized in Table 1, using Top-1 accuracy (some methods also provide Top-5 accuracy on CUB2011-easy) for the unseen classes.

except for the proposed VHEGAN that aims to find a shared semantically meaningful latent space between the image and text modalities, none of the other methods have generative models for both modalities, regardless of whether they learn a classifier or a distance metric in a latent space for ZSL.

For VHEs and VHEGANs with different network structures, Table 1 shows that given the same structure on the text decoder and image encoder, each VHEGAN consistently outperforms its VHE counterpart, suggesting the advantage of a joint generation of both the image and text modalities.

**Text to image generation:** Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images. Given example encyclopedia documents for the unseen classes, we follow this approach to generate random images. We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images). We repeat the text-to-image generation experiment on Flower to generate random images given textual captions for three different unseen classes. As shown in Fig. 1(b), VHEGAN successfully generates flower images that semantically match their corresponding textual descriptions and visually resemble their corresponding example real images.

Giving the textural description \(\mathbf{t}\) of an unseen class as the text query, we retrieve the top five images ranked by the cosine distances between \(\mathbf{t}\) and \(\hat{\mathbf{t}}_{i}\).

With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image. **Latent space interpolation:** In order to understand the learned manifold in the latent space, given two texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), after drawing \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) from their own conditional posteriors using Gibbs sampling, we interpolate between \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) and use these interpolated latent variables to generate both the images via the GAN image generator and texts via the VHE text decoder. We show the results in Fig. 5, where the first and last columns show the true texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), and the images generated from \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\), respectively, while the second to third columns show the generated texts and images from the interpolated \(\mathbf{\theta}\)'s. The strong correspondences between the generated images and texts and smooth changes between adjacent columns suggest that the VHEGAN latent space is semantically meaningful for both the image and text modalities.

Given a random draw, the concatenation of \(\{\mathbf{\theta}^{(l)}\}_{l=1}^{L}\) is fed into the GAN image generator to generate a corresponding image.

In Fig. 7, we present the real images \(\mathbf{x}\) and their VHEGANregenerations \(G\left(q_{\boldsymbol{\Omega}}\left(\boldsymbol{\theta}\,|\,\boldsymbol{\Phi}, \boldsymbol{x}\right)\right)\), which suggest that the VHEGAN can use its GAN image generator to regenerate random images that more or less resemble the original real image fed into the VHE image encoder.

For each column in Fig. 7(b), conditioning on a textual description, we show the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all layers, respectively, as the latent space fed into the GAN image generator.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space. With a stochastic-gradient MCMC algorithm to sample the global parameters of the VHE text decoder, a reparameterized Weibull distribution based variational encoder to approximate the posterior distribution of the local parameters of the VHE image encoder, and stochastic gradient descent (SGD) to estimate the parameters of the GAN image generator and discriminator, we develop a SG-MCMC/VHE/SGD hybrid inference algorithm to jointly train the components of VHEGAN. Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

Generative adversarial nets.

Improved training of Wasserstein GANs.

We give a detailed hybrid SGMCMC/SGD algorithm for VHEGAN in this section, to realize learning end-to-end.

In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively.

Image Generation Network shown in Fig. 10 : \(\mathbf{\theta}^{(l)}\rightarrow\hat{\mathbf{x}}\); \(l=1,2,3\)\(1\times\) { fully-connected layer with \(4\times 4\times 64\times 4\) units, RELU activation and batch normalization.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 128 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 64 feature maps with \(5\times 5\) kernels, stride 2, RELU activation.}\(1\times\) { deconv2d 256 feature maps with \(5\times 5\) kernels, stride 2.}

Input: Table 1: Accuracy (%) of zero-shot classification on CUB2011-hard, CUB2011-easy, and Flower Datasets. Note that some of them are attribute-based methods but applicable in our setting by replacing attribute vectors with text features (labeled by ∗), as discussed in Elhoseiny et al. (2017b). ZSLPP and GAZSL use a welldefined part detection features for images. All results on CUB2011 and the results of WAC on Flower come from Elhoseiny et al. (2017b); Qiao et al. (2016); Elhoseiny et al. (2017a), with all the others obtained by running the code provided by the original authors.

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 2: Class specific image generation based on the textual descriptions of unseen classes. (a) CUB2011easy and Flower data, where the first row shows real samples, while the others generated images; (b) Flower data, where the top row shows the representative textual captions and two real images of the same class.

Input: Figure 6: Image and document samples jointly generated from a three-layer VHEGAN.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim about text-to-image generation capabilities appears to be well-stated and accurately reflects what the model can do, as demonstrated by multiple examples and experimental results. The evidence shows:

1. Clear demonstrations of text-to-image generation through both GAN generation and image retrieval
2. Multiple figures showing successful image generation from textual descriptions for unseen classes
3. Detailed technical explanations of how the text-to-image generation process works
4. Validation through both qualitative examples and quantitative results
5. Appropriate acknowledgment of limitations and model choices

The review comment raises concerns about novelty but not about the specific capabilities claimed. The primary claim stays within the bounds of what is demonstrated by the evidence without overstating the results.

<score>1</score>

Justification:
- The claim is precise and technically accurate about the dual approaches (GAN generation and retrieval)
- Multiple figures (Fig 1, 2, 6, etc.) provide clear evidence of the text-to-image generation capabilities
- The method's limitations and design choices are explicitly discussed (e.g., choice of GAN vs VAE)
- The technical details provided match the level of capability being claimed
- No exaggeration of results or unsubstantiated claims are made