Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To extract and relate visual and linguistic concepts from images and textual descriptions for text-based zero-shot learning (ZSL), we develop variational hetero-encoder (VHE) that decodes text via a deep probabilistic topic model, the variational posterior of whose local latent variables is encoded from an image via a Weibull distribution based inference network.

The evidence to be evaluated is:The proposed VHE uses the Poisson gamma belief network (GBN) of Zhou et al. (2015) as its text decoder, whose output is a bag-of-words count vector, as shown in Fig. 1 (a), and a Weibull distribution based deterministic-upward-stochastic-downward inference network (Zhang et al., 2018) as its image encoder, whose input is a natural image, as shown in Fig. 1 (b). Integrating VHE and GAN, the proposed VHEGAN, as shown in Fig. 1 (c), tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document, under the Poisson GBN text decoder, be the distribution of the random noise fed into the GAN image generator; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder; and the GAN discriminator tries to discriminate between that selected image and the generated one.

To extract hierarchical latent representation from these high-dimensional sparse count vectors \(\mathbf{t}_{c}\), we choose the Poisson GBN (Zhou et al., 2015), which can be considered as a member of deep exponential families (Ranganath et al., 2015). It consists of multiple gamma distributed stochastic hidden layers, generalizing the "shallow" Poisson factor analysis (Zhou et al., 2012; Zhou and Carin, 2015) that is closely related to latent Dirichlet allocation (LDA) (Blei et al., 2003) into a deep setting. More specifically, modeling \(\mathbf{t}_{c}\) under the Poisson likelihood, its generative model with \(L\) hidden layers, from top to bottom, is expressed as \[\mathbf{\theta}_{c}^{(L)} \sim\text{Gam}\left(\mathbf{r},1/s_{c}^{(L+1)}\right),\,\ldots,\mathbf{ \theta}_{c}^{(l)}\sim\text{Gam}\left(\mathbf{\Phi}^{(l+1)}\mathbf{\theta}_{c}^{(l+1)},1/s_{c}^{(l+1)}\right),\,\ldots,\] \[\mathbf{\theta}_{c}^{(1)} \sim\text{Gam}\left(\mathbf{\Phi}^{(2)}\mathbf{\theta}_{c}^{(2)},1/s_{c} ^{(2)}\right),\,\mathbf{t}_{c}\!\sim\!\text{Pois}\left(\mathbf{\Phi}^{(1)}\mathbf{\theta}_ {c}^{(1)}\right), \tag{1}\] where the hidden units \(\mathbf{\theta}_{c}^{(l)}\in\mathbb{R}_{+}^{K_{l}}\) of layer \(l\) are factorized under the gamma likelihood into the product of the factor loading \(\mathbf{\Phi}^{(l)}\in\mathbb{R}_{+}^{K_{l-1}\times K_{l}}\) and hidden units of the next layer, \(\mathbb{R}_{+}=\{x,x\geq 0\}\), \(s_{c}^{(l)}>0\), and \(K_{l}\) is the number of topics (factors) of layer \(l\).

Choosing a deep generative model as the text decoder distinguishes VHE from Gomez et al. (2017) that uses LDA (Blei et al., 2003), a "shallow" probabilistic model, and Ba et al. (2015) that uses non-probabilistic black-box DNNs to learn class-specific document features.

To avoid using iterations to sample the latent representation of a testing document from its posterior, Zhang et al. (2018) develop Weibull upward-downward variational encoder (WUDVE) that directly projects an observed bag-of-words vector into its posterior distribution given a random sample of the global parameters, leading to extremely fast inference for out-of-sample prediction.

In this paper, we leverage WUDVE as the variational inference network to approximate the true posterior distribution \(p(\mathbf{\theta}\,|\,\mathbf{t},\mathbf{\Phi})\), where \(\mathbf{\theta}=\{\mathbf{\theta}^{(l)}\}_{1,L}\) denotes the set of local parameters, but change the input of WUDVE from the bag-of-words vector \(\mathbf{t}\) into an image \(\mathbf{x}\) associated with \(\mathbf{t}\).

where, with \(f(\mathbf{x})\) representing a feature vector deterministically projected from \(\mathbf{x}\) via an off-the-shelf pre-trained deep neural network \(f(\mathbf{\cdot})\), we define \[q_{\mathbf{\Omega}}(\mathbf{\theta}^{(l)}\,|\,\mathbf{\Phi}^{(l+1)},\mathbf{ \theta}^{(l+1)},\mathbf{x}) =\text{Weibull}(\mathbf{k}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))+\mathbf{\Phi}^{(l +1)}\mathbf{\theta}^{(l+1)},\mathbf{\lambda}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))),\] \[q_{\mathbf{\Omega}}(\mathbf{\theta}^{(L)}\,|\,\mathbf{r},\mathbf{x}) =\text{Weibull}(\mathbf{k}_{\mathbf{\Omega}}^{(L)}(f(\mathbf{x}))+\mathbf{r}, \mathbf{\lambda}_{\mathbf{\Omega}}^{(L)}(f(\mathbf{x}))), \tag{3}\]where \(\mathbf{k}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))\in\mathbb{R}^{K_{l}}\) and \(\mathbf{\lambda}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))\in\mathbb{R}^{K_{l}}\) are deterministically transformed from \(f(\mathbf{x})\). We use \(\mathbf{\Omega}\) to denote all the parameters of the inference network.

Given \(\mathbf{\Phi}\) of the Poisson GBN, to approximate the posterior distribution of the local parameters \(\mathbf{\theta}\) given text description \(\mathbf{t}\), we use WVDVE \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\), whose input is an image \(\mathbf{x}\), and learn its parameters \(\mathbf{\Omega}\) by maximizing the evidence lower bound (ELBO) as \[\max_{\mathbf{\Omega}}L_{\text{VHE}}(\mathbf{\Omega})=\mathbb{E}_{(\mathbf{x},\mathbf{t}) \sim p_{data}(\mathbf{x},\mathbf{t})}\left\{\mathbb{E}_{q_{\mathbf{\Omega}}(\mathbf{\theta} \,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln p\left(\mathbf{t}\,|\,\mathbf{\Phi}^{(1)},\mathbf{\theta }^{(1)}\right)-\ln\frac{q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}{p( \mathbf{\theta}\,|\,\mathbf{\Phi})}\right]\right\}, \tag{4}\]

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator. More specifically, we modify the GAN objective function as \[\min_{G}\max_{D}L_{\text{GAN}}(D,G)=\min_{G}\max_{D}\mathbb{E}_{\mathbf{x}\sim p_ {data}(\mathbf{x})}\left\{\ln D(\mathbf{x})-\mathbb{E}_{\mathbf{\theta}\sim q_{\mathbf{\Omega} }(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln D(G(\mathbf{\theta}))\right]\right\} \tag{6}\] where \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) is defined as in (2).

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities. For inference, since the Weibull distribution \(\theta\sim\text{Weibull}(k,\lambda)\) can be reparameterized as \(\theta=\lambda(-\ln(1-\epsilon))^{1/k}\), where \(\epsilon\sim\text{Uniform}(0,1)\), it is convenient to calculate the gradients of (7) with respect to \(\{\mathbf{\Omega},G,D\}\) by back-propagation (BP) using reparameterized random samples from \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\).

The proposed VHE by itself and VHEGAN can both be used for ZSL in the same way: with the Poisson GBN based VHE text decoder and the WUDVE based VHE image encoder, to perform zero-shot classification for a test image \(\mathbf{x}\) from an unseen class, we predict its class label as

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7). Conventional ZSL methods often make a restrictive assumption that a test example only comes from one of the unseen classes. A more challenging setting where a test example can come from either a seen class or an unseen one is known as generalized ZSL (GZSL) (Socher et al., 2013; Verma et al., 2018; Chao et al., 2016).

To remedy these drawbacks, text-based ZSL have been developed using easily accessible textural descriptions such as Wikipedia.

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning. In Srivastava & Salakhutdinov (2012, 2014), a deep Boltzmann machine is developed to model the joint generation of image feature and BOW tags. Moving beyond binary hidden units, Wang et al. (2018) propose a multimodal Poisson gamma belief network, similar to the document decoder of VHEGAN.

Both these two approaches are two-step models with the additional textual feature extraction realized by LSTM and "shallow" LDA, respectively, whereas the VHEGAN can realize joint optimization with hybrid SGMCMC/VHE/SGD algorithm.

VHE builds a variational posterior conditioned on images to approximate Gamma latent variables to generate the BOW vector of documents. To solve the problem of complete random generate process in GAN, conditional GANs (CGAN) (Mirza & Osindero, 2014; Reed et al., 2016) are alsodeveloped, where the conditions are labels, or attributes learned as point estimations separately with GAN, while VHEGAN learns a distributed condition jointly with a VAE model.

Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

The 4096 dimensional image features, \(f(\mathbf{x})\) in (3), from the fc1 layer of the pre-trained VGG16 network (Simonyan & Zisserman, 2015) are fed into the VHE image encoder, and the size of each generated RGB image is set as \(64\times 64\).

except for the proposed VHEGAN that aims to find a shared semantically meaningful latent space between the image and text modalities, none of the other methods have generative models for both modalities, regardless of whether they learn a classifier or a distance metric in a latent space for ZSL.

Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images.

We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images).

As shown in Fig. 1(b), VHEGAN successfully generates flower images that semantically match their corresponding textual descriptions and visually resemble their corresponding example real images. **Image retrieval given a text query:** For image \(\mathbf{x}_{i}\), via the text generator and image encoder, we generate a bag-of-words vector \(\hat{\mathbf{t}}_{i}\) using Poisson mean \(\mathbf{\Phi}^{(1)}\mathbf{\theta}^{(1)}\) as \[\hat{\mathbf{t}}_{i}\,|\,\mathbf{\theta}_{i}\sim p(\mathbf{t}\,|\,\mathbf{\Phi},\mathbf{\theta}_{ i}),\,\mathbf{\theta}_{i}\,|\,\mathbf{x}_{i}\sim q_{\Omega}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x} _{i}). \tag{9}\] Giving the textural description \(\mathbf{t}\) of an unseen class as the text query, we retrieve the top five images ranked by the cosine distances between \(\mathbf{t}\) and \(\hat{\mathbf{t}}_{i}\).

With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image. **Latent space interpolation:** In order to understand the learned manifold in the latent space, given two texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), after drawing \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) from their own conditional posteriors using Gibbs sampling, we interpolate between \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) and use these interpolated latent variables to generate both the images via the GAN image generator and texts via the VHE text decoder. We show the results in Fig. 5, where the first and last columns show the true texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), and the images generated from \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\), respectively, while the second to third columns show the generated texts and images from the interpolated \(\mathbf{\theta}\)'s. The strong correspondences between the generated images and texts and smooth changes between adjacent columns suggest that the VHEGAN latent space is semantically meaningful for both the image and text modalities.

Given a random draw, the concatenation of \(\{\mathbf{\theta}^{(l)}\}_{l=1}^{L}\) is fed into the GAN image generator to generate a corresponding image. Shown in Fig. 6 are four random draws, for each of which we show its top seven words and its generated image, whose relationships are clearly interpretable, suggesting that VHEGAN is able to recode the key information of both modalities and the relationships between them.

We note for the VHEGAN, its VHE image encoder and GAN component together can also be viewed as an "autoencoding" GAN for images. Different from Dumoulin et al. (2017) and Donahue et al. (2017), the encoded latent space is affected not only by the GAN for the image modality, but also by the Poisson GBN for the text modailty.

The inferred topics at different layers and the inferred connection weights between the topics of adjacent layers are found to be highly interpretable. In particular, we can understand the meaning of each topic by projecting it back to the original data space via \(\left[\prod_{t=1}^{l-1}\boldsymbol{\Phi}^{(t)}\right]\boldsymbol{\phi}_{k}^{(l)}\).

The semantic meaning of each topic and the connection weights between the topics of adjacent layers are highly interpretable, where the topics describe very specific flower characteristics, such as colors, shapes, and parts, at the bottom layer, and become increasingly more general when moving upwards.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space. With a stochastic-gradient MCMC algorithm to sample the global parameters of the VHE text decoder, a reparameterized Weibull distribution based variational encoder to approximate the posterior distribution of the local parameters of the VHE image encoder, and stochastic gradient descent (SGD) to estimate the parameters of the GAN image generator and discriminator, we develop a SG-MCMC/VHE/SGD hybrid inference algorithm to jointly train the components of VHEGAN. Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

* Blei et al. (2017) David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians.

* Changpinyo et al. (2016) Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning.

Self-supervised learning of visual features through embedding images into text topic spaces.

In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively. For Poission GBN, only the number of topics in different layers should be set. In our experiment's setting, we set the number of topics from the first to the third layer to 256, 128 and 64, respectively.

\(\mathbf{x}\rightarrow\mathbf{h}^{(l)},l=1,2,3\)\(\mathbf{x}\rightarrow\mathbf{h}^{(1)}\)\(1\times\) { fully-connected layer with 256 units and Leaky-RELU activation.}\(\mathbf{h}^{(1)}\rightarrow\mathbf{h}^{(2)}\)\(1\times\) { fully-connected layer with 128 units and Leaky-RELU activation.}\(\mathbf{h}^{(2)}\rightarrow\mathbf{h}^{(3)}\)\(1\times\) { fully-connected layer with 64 units and Leaky-RELU activation.} Weibull variational posterior \(q(\mathbf{\theta}^{(l)}|\mathbf{h}^{(l)})\) as shown in Fig. :\(\mathbf{h}^{(l)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(l)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (l)}};l=1,2,3\)\(\mathbf{h}^{(1)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(1)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (1)}}\)\(1\times\) { fully-connected layer with 256 units to \(\mathbf{k}_{\mathbf{\theta}^{(1)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(1)}}\).}\(\mathbf{h}^{(2)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(2)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (2)}}\)\(1\times\) { fully-connected layer with 128 units to \(\mathbf{k}_{\mathbf{\theta}^{(2)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(2)}}\).}\(\mathbf{h}^{(3)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(3)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (3)}}\)\(1\times\) { fully-connected layer with 64 units to \(\mathbf{k}_{\mathbf{\theta}^{(3)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(3)}}\).}

Input: Figure 9: The architecture of WUDVE in VHE and VHEGAN.

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 7: Image reconstruction with real data x (the first row) and corresponding reconstructions G (qΩ (θ |Φ,x)), where each rows represent different sampling from qΩ (θ).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the claim is generally well-stated and accurately reflects the technical details presented in the evidence. The paper clearly demonstrates:

1. The development of VHE that uses a Weibull distribution-based inference network to encode images
2. The decoding of text through a deep probabilistic topic model (Poisson GBN)
3. The relationship between visual and linguistic concepts through the shared latent space

The evidence provides extensive mathematical formulations, architectural details, and experimental results that support each component mentioned in the claim. The wording is precise and appropriately technical without overstating capabilities or results.

<score>1</score>

Justification:
- The claim precisely describes the technical approach without exaggeration
- Mathematical formulations in equations (1)-(9) directly support the claimed mechanisms
- Experimental results and figures (especially Fig. 1) demonstrate the described image-text relationships
- The model architecture details in the appendix validate the claimed components
- No unsupported assertions or overgeneralized statements are made


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To extract and relate visual and linguistic concepts from images and textual descriptions for text-based zero-shot learning (ZSL), we develop variational hetero-encoder (VHE) that decodes text via a deep probabilistic topic model, the variational posterior of whose local latent variables is encoded from an image via a Weibull distribution based inference network.

The review comment to be evaluated is:

Paper Summary: This paper studies the zero-shot learning problem with deep generative models. More specifically, it proposed a hybrid framework that combines VAEs (more precisely, the variational hetero-encoder or VHE) and GANs all together. The entire model is composed of an image encoder (Weibull upward-downward variational encoder), a text decoder (Poisson Gamma belief network), and an image generator (generative adversarial network). Once learned, the generative models can be directly used for zero-shot classification and various image generation applications. In the experiments, two benchmark datasets CUB and Oxford-Flowers are used.

==
Novelty/Significance:
Zero-shot learning is a challenging task and he main motivation of the paper (using generative model) is interesting. The text representation in the paper is simply bag-of-words which limits the application to some extent. In a broader context, image captioning using generative model seems quite relevant.

Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space, Wang et al. In NIPS 2017.

==
Quality:
Overall, reviewer feels this is a very interesting work. However, the results from the paper is quite mixed. It is not yet convincing whether the proposed approach is the state-of-the-art in zero-shot learning or text-to-image generation.

First, this paper demonstrates the power of generative models in text-to-image generation and other applications. However, reviewer feels that the zero-shot classification result is weak. In Table 1 and Table 2, it seems that GAZSL (Zhu et al. 2018) outperforms the proposed approach.

Q1: In Table 2, is it possible to report the top-5 accuracy on CUB-easy and top-1 accuracy on Oxford-Flower dataset? Otherwise, it is not very convincing that proposed approach is better than the state-of-the-art approach GAZSL.

Second, the text-to-image generation results look reasonably good. But the resolution and quality of generated images are far from state-of-the-art. One suggestion is to train the VHE model with an improved image generator.

StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks, Zhang et al. In CVPR 2017.

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks, Xu et al. In CVPR 2018.

Also, reviewer would expect to see an improved image generator can lead to a better ZSL performance.

Typo: In the title: Zero-Short → Zero-Shot.

The evidence to be evaluated is:The proposed VHE uses the Poisson gamma belief network (GBN) of Zhou et al. (2015) as its text decoder, whose output is a bag-of-words count vector, as shown in Fig. 1 (a), and a Weibull distribution based deterministic-upward-stochastic-downward inference network (Zhang et al., 2018) as its image encoder, whose input is a natural image, as shown in Fig. 1 (b). Integrating VHE and GAN, the proposed VHEGAN, as shown in Fig. 1 (c), tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document, under the Poisson GBN text decoder, be the distribution of the random noise fed into the GAN image generator; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder; and the GAN discriminator tries to discriminate between that selected image and the generated one.

To extract hierarchical latent representation from these high-dimensional sparse count vectors \(\mathbf{t}_{c}\), we choose the Poisson GBN (Zhou et al., 2015), which can be considered as a member of deep exponential families (Ranganath et al., 2015). It consists of multiple gamma distributed stochastic hidden layers, generalizing the "shallow" Poisson factor analysis (Zhou et al., 2012; Zhou and Carin, 2015) that is closely related to latent Dirichlet allocation (LDA) (Blei et al., 2003) into a deep setting. More specifically, modeling \(\mathbf{t}_{c}\) under the Poisson likelihood, its generative model with \(L\) hidden layers, from top to bottom, is expressed as \[\mathbf{\theta}_{c}^{(L)} \sim\text{Gam}\left(\mathbf{r},1/s_{c}^{(L+1)}\right),\,\ldots,\mathbf{ \theta}_{c}^{(l)}\sim\text{Gam}\left(\mathbf{\Phi}^{(l+1)}\mathbf{\theta}_{c}^{(l+1)},1/s_{c}^{(l+1)}\right),\,\ldots,\] \[\mathbf{\theta}_{c}^{(1)} \sim\text{Gam}\left(\mathbf{\Phi}^{(2)}\mathbf{\theta}_{c}^{(2)},1/s_{c} ^{(2)}\right),\,\mathbf{t}_{c}\!\sim\!\text{Pois}\left(\mathbf{\Phi}^{(1)}\mathbf{\theta}_ {c}^{(1)}\right), \tag{1}\] where the hidden units \(\mathbf{\theta}_{c}^{(l)}\in\mathbb{R}_{+}^{K_{l}}\) of layer \(l\) are factorized under the gamma likelihood into the product of the factor loading \(\mathbf{\Phi}^{(l)}\in\mathbb{R}_{+}^{K_{l-1}\times K_{l}}\) and hidden units of the next layer, \(\mathbb{R}_{+}=\{x,x\geq 0\}\), \(s_{c}^{(l)}>0\), and \(K_{l}\) is the number of topics (factors) of layer \(l\).

Choosing a deep generative model as the text decoder distinguishes VHE from Gomez et al. (2017) that uses LDA (Blei et al., 2003), a "shallow" probabilistic model, and Ba et al. (2015) that uses non-probabilistic black-box DNNs to learn class-specific document features.

To avoid using iterations to sample the latent representation of a testing document from its posterior, Zhang et al. (2018) develop Weibull upward-downward variational encoder (WUDVE) that directly projects an observed bag-of-words vector into its posterior distribution given a random sample of the global parameters, leading to extremely fast inference for out-of-sample prediction.

In this paper, we leverage WUDVE as the variational inference network to approximate the true posterior distribution \(p(\mathbf{\theta}\,|\,\mathbf{t},\mathbf{\Phi})\), where \(\mathbf{\theta}=\{\mathbf{\theta}^{(l)}\}_{1,L}\) denotes the set of local parameters, but change the input of WUDVE from the bag-of-words vector \(\mathbf{t}\) into an image \(\mathbf{x}\) associated with \(\mathbf{t}\).

where, with \(f(\mathbf{x})\) representing a feature vector deterministically projected from \(\mathbf{x}\) via an off-the-shelf pre-trained deep neural network \(f(\mathbf{\cdot})\), we define \[q_{\mathbf{\Omega}}(\mathbf{\theta}^{(l)}\,|\,\mathbf{\Phi}^{(l+1)},\mathbf{ \theta}^{(l+1)},\mathbf{x}) =\text{Weibull}(\mathbf{k}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))+\mathbf{\Phi}^{(l +1)}\mathbf{\theta}^{(l+1)},\mathbf{\lambda}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))),\] \[q_{\mathbf{\Omega}}(\mathbf{\theta}^{(L)}\,|\,\mathbf{r},\mathbf{x}) =\text{Weibull}(\mathbf{k}_{\mathbf{\Omega}}^{(L)}(f(\mathbf{x}))+\mathbf{r}, \mathbf{\lambda}_{\mathbf{\Omega}}^{(L)}(f(\mathbf{x}))), \tag{3}\]where \(\mathbf{k}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))\in\mathbb{R}^{K_{l}}\) and \(\mathbf{\lambda}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))\in\mathbb{R}^{K_{l}}\) are deterministically transformed from \(f(\mathbf{x})\). We use \(\mathbf{\Omega}\) to denote all the parameters of the inference network.

Given \(\mathbf{\Phi}\) of the Poisson GBN, to approximate the posterior distribution of the local parameters \(\mathbf{\theta}\) given text description \(\mathbf{t}\), we use WVDVE \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\), whose input is an image \(\mathbf{x}\), and learn its parameters \(\mathbf{\Omega}\) by maximizing the evidence lower bound (ELBO) as \[\max_{\mathbf{\Omega}}L_{\text{VHE}}(\mathbf{\Omega})=\mathbb{E}_{(\mathbf{x},\mathbf{t}) \sim p_{data}(\mathbf{x},\mathbf{t})}\left\{\mathbb{E}_{q_{\mathbf{\Omega}}(\mathbf{\theta} \,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln p\left(\mathbf{t}\,|\,\mathbf{\Phi}^{(1)},\mathbf{\theta }^{(1)}\right)-\ln\frac{q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}{p( \mathbf{\theta}\,|\,\mathbf{\Phi})}\right]\right\}, \tag{4}\]

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator. More specifically, we modify the GAN objective function as \[\min_{G}\max_{D}L_{\text{GAN}}(D,G)=\min_{G}\max_{D}\mathbb{E}_{\mathbf{x}\sim p_ {data}(\mathbf{x})}\left\{\ln D(\mathbf{x})-\mathbb{E}_{\mathbf{\theta}\sim q_{\mathbf{\Omega} }(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln D(G(\mathbf{\theta}))\right]\right\} \tag{6}\] where \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) is defined as in (2).

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities. For inference, since the Weibull distribution \(\theta\sim\text{Weibull}(k,\lambda)\) can be reparameterized as \(\theta=\lambda(-\ln(1-\epsilon))^{1/k}\), where \(\epsilon\sim\text{Uniform}(0,1)\), it is convenient to calculate the gradients of (7) with respect to \(\{\mathbf{\Omega},G,D\}\) by back-propagation (BP) using reparameterized random samples from \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\).

The proposed VHE by itself and VHEGAN can both be used for ZSL in the same way: with the Poisson GBN based VHE text decoder and the WUDVE based VHE image encoder, to perform zero-shot classification for a test image \(\mathbf{x}\) from an unseen class, we predict its class label as

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7). Conventional ZSL methods often make a restrictive assumption that a test example only comes from one of the unseen classes. A more challenging setting where a test example can come from either a seen class or an unseen one is known as generalized ZSL (GZSL) (Socher et al., 2013; Verma et al., 2018; Chao et al., 2016).

To remedy these drawbacks, text-based ZSL have been developed using easily accessible textural descriptions such as Wikipedia.

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning. In Srivastava & Salakhutdinov (2012, 2014), a deep Boltzmann machine is developed to model the joint generation of image feature and BOW tags. Moving beyond binary hidden units, Wang et al. (2018) propose a multimodal Poisson gamma belief network, similar to the document decoder of VHEGAN.

Both these two approaches are two-step models with the additional textual feature extraction realized by LSTM and "shallow" LDA, respectively, whereas the VHEGAN can realize joint optimization with hybrid SGMCMC/VHE/SGD algorithm.

VHE builds a variational posterior conditioned on images to approximate Gamma latent variables to generate the BOW vector of documents. To solve the problem of complete random generate process in GAN, conditional GANs (CGAN) (Mirza & Osindero, 2014; Reed et al., 2016) are alsodeveloped, where the conditions are labels, or attributes learned as point estimations separately with GAN, while VHEGAN learns a distributed condition jointly with a VAE model.

Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

The 4096 dimensional image features, \(f(\mathbf{x})\) in (3), from the fc1 layer of the pre-trained VGG16 network (Simonyan & Zisserman, 2015) are fed into the VHE image encoder, and the size of each generated RGB image is set as \(64\times 64\).

except for the proposed VHEGAN that aims to find a shared semantically meaningful latent space between the image and text modalities, none of the other methods have generative models for both modalities, regardless of whether they learn a classifier or a distance metric in a latent space for ZSL.

Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images.

We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images).

As shown in Fig. 1(b), VHEGAN successfully generates flower images that semantically match their corresponding textual descriptions and visually resemble their corresponding example real images. **Image retrieval given a text query:** For image \(\mathbf{x}_{i}\), via the text generator and image encoder, we generate a bag-of-words vector \(\hat{\mathbf{t}}_{i}\) using Poisson mean \(\mathbf{\Phi}^{(1)}\mathbf{\theta}^{(1)}\) as \[\hat{\mathbf{t}}_{i}\,|\,\mathbf{\theta}_{i}\sim p(\mathbf{t}\,|\,\mathbf{\Phi},\mathbf{\theta}_{ i}),\,\mathbf{\theta}_{i}\,|\,\mathbf{x}_{i}\sim q_{\Omega}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x} _{i}). \tag{9}\] Giving the textural description \(\mathbf{t}\) of an unseen class as the text query, we retrieve the top five images ranked by the cosine distances between \(\mathbf{t}\) and \(\hat{\mathbf{t}}_{i}\).

With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image. **Latent space interpolation:** In order to understand the learned manifold in the latent space, given two texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), after drawing \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) from their own conditional posteriors using Gibbs sampling, we interpolate between \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) and use these interpolated latent variables to generate both the images via the GAN image generator and texts via the VHE text decoder. We show the results in Fig. 5, where the first and last columns show the true texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), and the images generated from \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\), respectively, while the second to third columns show the generated texts and images from the interpolated \(\mathbf{\theta}\)'s. The strong correspondences between the generated images and texts and smooth changes between adjacent columns suggest that the VHEGAN latent space is semantically meaningful for both the image and text modalities.

Given a random draw, the concatenation of \(\{\mathbf{\theta}^{(l)}\}_{l=1}^{L}\) is fed into the GAN image generator to generate a corresponding image. Shown in Fig. 6 are four random draws, for each of which we show its top seven words and its generated image, whose relationships are clearly interpretable, suggesting that VHEGAN is able to recode the key information of both modalities and the relationships between them.

We note for the VHEGAN, its VHE image encoder and GAN component together can also be viewed as an "autoencoding" GAN for images. Different from Dumoulin et al. (2017) and Donahue et al. (2017), the encoded latent space is affected not only by the GAN for the image modality, but also by the Poisson GBN for the text modailty.

The inferred topics at different layers and the inferred connection weights between the topics of adjacent layers are found to be highly interpretable. In particular, we can understand the meaning of each topic by projecting it back to the original data space via \(\left[\prod_{t=1}^{l-1}\boldsymbol{\Phi}^{(t)}\right]\boldsymbol{\phi}_{k}^{(l)}\).

The semantic meaning of each topic and the connection weights between the topics of adjacent layers are highly interpretable, where the topics describe very specific flower characteristics, such as colors, shapes, and parts, at the bottom layer, and become increasingly more general when moving upwards.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space. With a stochastic-gradient MCMC algorithm to sample the global parameters of the VHE text decoder, a reparameterized Weibull distribution based variational encoder to approximate the posterior distribution of the local parameters of the VHE image encoder, and stochastic gradient descent (SGD) to estimate the parameters of the GAN image generator and discriminator, we develop a SG-MCMC/VHE/SGD hybrid inference algorithm to jointly train the components of VHEGAN. Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

* Blei et al. (2017) David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians.

* Changpinyo et al. (2016) Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning.

Self-supervised learning of visual features through embedding images into text topic spaces.

In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively. For Poission GBN, only the number of topics in different layers should be set. In our experiment's setting, we set the number of topics from the first to the third layer to 256, 128 and 64, respectively.

\(\mathbf{x}\rightarrow\mathbf{h}^{(l)},l=1,2,3\)\(\mathbf{x}\rightarrow\mathbf{h}^{(1)}\)\(1\times\) { fully-connected layer with 256 units and Leaky-RELU activation.}\(\mathbf{h}^{(1)}\rightarrow\mathbf{h}^{(2)}\)\(1\times\) { fully-connected layer with 128 units and Leaky-RELU activation.}\(\mathbf{h}^{(2)}\rightarrow\mathbf{h}^{(3)}\)\(1\times\) { fully-connected layer with 64 units and Leaky-RELU activation.} Weibull variational posterior \(q(\mathbf{\theta}^{(l)}|\mathbf{h}^{(l)})\) as shown in Fig. :\(\mathbf{h}^{(l)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(l)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (l)}};l=1,2,3\)\(\mathbf{h}^{(1)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(1)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (1)}}\)\(1\times\) { fully-connected layer with 256 units to \(\mathbf{k}_{\mathbf{\theta}^{(1)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(1)}}\).}\(\mathbf{h}^{(2)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(2)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (2)}}\)\(1\times\) { fully-connected layer with 128 units to \(\mathbf{k}_{\mathbf{\theta}^{(2)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(2)}}\).}\(\mathbf{h}^{(3)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(3)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (3)}}\)\(1\times\) { fully-connected layer with 64 units to \(\mathbf{k}_{\mathbf{\theta}^{(3)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(3)}}\).}

Input: Figure 9: The architecture of WUDVE in VHE and VHEGAN.

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 7: Image reconstruction with real data x (the first row) and corresponding reconstructions G (qΩ (θ |Φ,x)), where each rows represent different sampling from qΩ (θ).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the claim reasonably describes the key technical aspects of the VHE model without major exaggeration. The evidence demonstrates that:

1. The model does use a Weibull distribution-based inference network for image encoding, as claimed
2. The text decoding is handled by a deep probabilistic topic model (Poisson GBN)
3. The technical details regarding extraction and relation of visual-linguistic concepts are supported by extensive mathematical formulations and experimental results showing the model's ability to learn meaningful representations across modalities

While technical, the claim accurately reflects the model's architecture and capabilities without overstating its performance or novelty.

<score>1</score>

The claim is well-stated, avoiding hyperbole while precisely describing the key technical contributions with appropriate technical terminology supported by the evidence.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To extract and relate visual and linguistic concepts from images and textual descriptions for text-based zero-shot learning (ZSL), we develop variational hetero-encoder (VHE) that decodes text via a deep probabilistic topic model, the variational posterior of whose local latent variables is encoded from an image via a Weibull distribution based inference network.

The review comment to be evaluated is:

[Paper Summary]
This work suggests a new model incorporating deep topic model (text decoder),  VHE (image encoder), and GAN. The topic model and the VHE shares the topic parameters, and the GAN generate an image regarding the topic. Then, for ZSL, the image is encoded to corresponding topic parameters, and the parameter can tell which text description (unseen) is matched with the highest probability. GAN model is used to generate an image given the topic distribution. During the training of the GAN, the VHE and topic model is jointly trained and can enhance the ZSL performance marginally.

[pros]
- This work successfully incorporated the topic model and image encoding/decoding. All the individual parts are already given, but I think incorporating them in terms of a unified probabilistic model is also meaningful for this field.
- This work shows superior performance on the image to text ZSL problem.
- This work mapped the text to image, image to text mapping in a generative manner.

[cons]
- The problem is only valid when the unseen class distribution is very similar to the given classes. For example, the text description of unseen classes should be well represented to the topics from seen classes.
- It is doubtful that this corresponds to the term zero-shot learning; dealing with the case that the unseen class and the seen class are notably different from each other.
- Similarly, GAN learns images from the seen classes, and by nature, GAN would not generate the proper images of the unseen class if the image distribution of the unseen class is different to the already seen class. In the paper, the classes are very similar to each other (birds, flowers) and that would be the reason GAN worked in this model.
- (minor) The likelihood of the text (image) given topic should be provided and compared to the existing models.

[Summary]
- The reviewer is personally interested in the proposal of the work, but concern that ZSL is difficult to be the main target of the paper because the model can only deal with the classes with (very) similar semantics, and this is the main reason for the rating. The testing with more diverse class should be given, or solid explanation of the mentioned problem would be required.

The evidence to be evaluated is:The proposed VHE uses the Poisson gamma belief network (GBN) of Zhou et al. (2015) as its text decoder, whose output is a bag-of-words count vector, as shown in Fig. 1 (a), and a Weibull distribution based deterministic-upward-stochastic-downward inference network (Zhang et al., 2018) as its image encoder, whose input is a natural image, as shown in Fig. 1 (b). Integrating VHE and GAN, the proposed VHEGAN, as shown in Fig. 1 (c), tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document, under the Poisson GBN text decoder, be the distribution of the random noise fed into the GAN image generator; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder; and the GAN discriminator tries to discriminate between that selected image and the generated one.

To extract hierarchical latent representation from these high-dimensional sparse count vectors \(\mathbf{t}_{c}\), we choose the Poisson GBN (Zhou et al., 2015), which can be considered as a member of deep exponential families (Ranganath et al., 2015). It consists of multiple gamma distributed stochastic hidden layers, generalizing the "shallow" Poisson factor analysis (Zhou et al., 2012; Zhou and Carin, 2015) that is closely related to latent Dirichlet allocation (LDA) (Blei et al., 2003) into a deep setting. More specifically, modeling \(\mathbf{t}_{c}\) under the Poisson likelihood, its generative model with \(L\) hidden layers, from top to bottom, is expressed as \[\mathbf{\theta}_{c}^{(L)} \sim\text{Gam}\left(\mathbf{r},1/s_{c}^{(L+1)}\right),\,\ldots,\mathbf{ \theta}_{c}^{(l)}\sim\text{Gam}\left(\mathbf{\Phi}^{(l+1)}\mathbf{\theta}_{c}^{(l+1)},1/s_{c}^{(l+1)}\right),\,\ldots,\] \[\mathbf{\theta}_{c}^{(1)} \sim\text{Gam}\left(\mathbf{\Phi}^{(2)}\mathbf{\theta}_{c}^{(2)},1/s_{c} ^{(2)}\right),\,\mathbf{t}_{c}\!\sim\!\text{Pois}\left(\mathbf{\Phi}^{(1)}\mathbf{\theta}_ {c}^{(1)}\right), \tag{1}\] where the hidden units \(\mathbf{\theta}_{c}^{(l)}\in\mathbb{R}_{+}^{K_{l}}\) of layer \(l\) are factorized under the gamma likelihood into the product of the factor loading \(\mathbf{\Phi}^{(l)}\in\mathbb{R}_{+}^{K_{l-1}\times K_{l}}\) and hidden units of the next layer, \(\mathbb{R}_{+}=\{x,x\geq 0\}\), \(s_{c}^{(l)}>0\), and \(K_{l}\) is the number of topics (factors) of layer \(l\).

Choosing a deep generative model as the text decoder distinguishes VHE from Gomez et al. (2017) that uses LDA (Blei et al., 2003), a "shallow" probabilistic model, and Ba et al. (2015) that uses non-probabilistic black-box DNNs to learn class-specific document features.

To avoid using iterations to sample the latent representation of a testing document from its posterior, Zhang et al. (2018) develop Weibull upward-downward variational encoder (WUDVE) that directly projects an observed bag-of-words vector into its posterior distribution given a random sample of the global parameters, leading to extremely fast inference for out-of-sample prediction.

In this paper, we leverage WUDVE as the variational inference network to approximate the true posterior distribution \(p(\mathbf{\theta}\,|\,\mathbf{t},\mathbf{\Phi})\), where \(\mathbf{\theta}=\{\mathbf{\theta}^{(l)}\}_{1,L}\) denotes the set of local parameters, but change the input of WUDVE from the bag-of-words vector \(\mathbf{t}\) into an image \(\mathbf{x}\) associated with \(\mathbf{t}\).

where, with \(f(\mathbf{x})\) representing a feature vector deterministically projected from \(\mathbf{x}\) via an off-the-shelf pre-trained deep neural network \(f(\mathbf{\cdot})\), we define \[q_{\mathbf{\Omega}}(\mathbf{\theta}^{(l)}\,|\,\mathbf{\Phi}^{(l+1)},\mathbf{ \theta}^{(l+1)},\mathbf{x}) =\text{Weibull}(\mathbf{k}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))+\mathbf{\Phi}^{(l +1)}\mathbf{\theta}^{(l+1)},\mathbf{\lambda}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))),\] \[q_{\mathbf{\Omega}}(\mathbf{\theta}^{(L)}\,|\,\mathbf{r},\mathbf{x}) =\text{Weibull}(\mathbf{k}_{\mathbf{\Omega}}^{(L)}(f(\mathbf{x}))+\mathbf{r}, \mathbf{\lambda}_{\mathbf{\Omega}}^{(L)}(f(\mathbf{x}))), \tag{3}\]where \(\mathbf{k}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))\in\mathbb{R}^{K_{l}}\) and \(\mathbf{\lambda}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))\in\mathbb{R}^{K_{l}}\) are deterministically transformed from \(f(\mathbf{x})\). We use \(\mathbf{\Omega}\) to denote all the parameters of the inference network.

Given \(\mathbf{\Phi}\) of the Poisson GBN, to approximate the posterior distribution of the local parameters \(\mathbf{\theta}\) given text description \(\mathbf{t}\), we use WVDVE \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\), whose input is an image \(\mathbf{x}\), and learn its parameters \(\mathbf{\Omega}\) by maximizing the evidence lower bound (ELBO) as \[\max_{\mathbf{\Omega}}L_{\text{VHE}}(\mathbf{\Omega})=\mathbb{E}_{(\mathbf{x},\mathbf{t}) \sim p_{data}(\mathbf{x},\mathbf{t})}\left\{\mathbb{E}_{q_{\mathbf{\Omega}}(\mathbf{\theta} \,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln p\left(\mathbf{t}\,|\,\mathbf{\Phi}^{(1)},\mathbf{\theta }^{(1)}\right)-\ln\frac{q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}{p( \mathbf{\theta}\,|\,\mathbf{\Phi})}\right]\right\}, \tag{4}\]

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator. More specifically, we modify the GAN objective function as \[\min_{G}\max_{D}L_{\text{GAN}}(D,G)=\min_{G}\max_{D}\mathbb{E}_{\mathbf{x}\sim p_ {data}(\mathbf{x})}\left\{\ln D(\mathbf{x})-\mathbb{E}_{\mathbf{\theta}\sim q_{\mathbf{\Omega} }(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln D(G(\mathbf{\theta}))\right]\right\} \tag{6}\] where \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) is defined as in (2).

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities. For inference, since the Weibull distribution \(\theta\sim\text{Weibull}(k,\lambda)\) can be reparameterized as \(\theta=\lambda(-\ln(1-\epsilon))^{1/k}\), where \(\epsilon\sim\text{Uniform}(0,1)\), it is convenient to calculate the gradients of (7) with respect to \(\{\mathbf{\Omega},G,D\}\) by back-propagation (BP) using reparameterized random samples from \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\).

The proposed VHE by itself and VHEGAN can both be used for ZSL in the same way: with the Poisson GBN based VHE text decoder and the WUDVE based VHE image encoder, to perform zero-shot classification for a test image \(\mathbf{x}\) from an unseen class, we predict its class label as

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7). Conventional ZSL methods often make a restrictive assumption that a test example only comes from one of the unseen classes. A more challenging setting where a test example can come from either a seen class or an unseen one is known as generalized ZSL (GZSL) (Socher et al., 2013; Verma et al., 2018; Chao et al., 2016).

To remedy these drawbacks, text-based ZSL have been developed using easily accessible textural descriptions such as Wikipedia.

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning. In Srivastava & Salakhutdinov (2012, 2014), a deep Boltzmann machine is developed to model the joint generation of image feature and BOW tags. Moving beyond binary hidden units, Wang et al. (2018) propose a multimodal Poisson gamma belief network, similar to the document decoder of VHEGAN.

Both these two approaches are two-step models with the additional textual feature extraction realized by LSTM and "shallow" LDA, respectively, whereas the VHEGAN can realize joint optimization with hybrid SGMCMC/VHE/SGD algorithm.

VHE builds a variational posterior conditioned on images to approximate Gamma latent variables to generate the BOW vector of documents. To solve the problem of complete random generate process in GAN, conditional GANs (CGAN) (Mirza & Osindero, 2014; Reed et al., 2016) are alsodeveloped, where the conditions are labels, or attributes learned as point estimations separately with GAN, while VHEGAN learns a distributed condition jointly with a VAE model.

Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

The 4096 dimensional image features, \(f(\mathbf{x})\) in (3), from the fc1 layer of the pre-trained VGG16 network (Simonyan & Zisserman, 2015) are fed into the VHE image encoder, and the size of each generated RGB image is set as \(64\times 64\).

except for the proposed VHEGAN that aims to find a shared semantically meaningful latent space between the image and text modalities, none of the other methods have generative models for both modalities, regardless of whether they learn a classifier or a distance metric in a latent space for ZSL.

Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images.

We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images).

As shown in Fig. 1(b), VHEGAN successfully generates flower images that semantically match their corresponding textual descriptions and visually resemble their corresponding example real images. **Image retrieval given a text query:** For image \(\mathbf{x}_{i}\), via the text generator and image encoder, we generate a bag-of-words vector \(\hat{\mathbf{t}}_{i}\) using Poisson mean \(\mathbf{\Phi}^{(1)}\mathbf{\theta}^{(1)}\) as \[\hat{\mathbf{t}}_{i}\,|\,\mathbf{\theta}_{i}\sim p(\mathbf{t}\,|\,\mathbf{\Phi},\mathbf{\theta}_{ i}),\,\mathbf{\theta}_{i}\,|\,\mathbf{x}_{i}\sim q_{\Omega}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x} _{i}). \tag{9}\] Giving the textural description \(\mathbf{t}\) of an unseen class as the text query, we retrieve the top five images ranked by the cosine distances between \(\mathbf{t}\) and \(\hat{\mathbf{t}}_{i}\).

With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image. **Latent space interpolation:** In order to understand the learned manifold in the latent space, given two texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), after drawing \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) from their own conditional posteriors using Gibbs sampling, we interpolate between \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) and use these interpolated latent variables to generate both the images via the GAN image generator and texts via the VHE text decoder. We show the results in Fig. 5, where the first and last columns show the true texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), and the images generated from \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\), respectively, while the second to third columns show the generated texts and images from the interpolated \(\mathbf{\theta}\)'s. The strong correspondences between the generated images and texts and smooth changes between adjacent columns suggest that the VHEGAN latent space is semantically meaningful for both the image and text modalities.

Given a random draw, the concatenation of \(\{\mathbf{\theta}^{(l)}\}_{l=1}^{L}\) is fed into the GAN image generator to generate a corresponding image. Shown in Fig. 6 are four random draws, for each of which we show its top seven words and its generated image, whose relationships are clearly interpretable, suggesting that VHEGAN is able to recode the key information of both modalities and the relationships between them.

We note for the VHEGAN, its VHE image encoder and GAN component together can also be viewed as an "autoencoding" GAN for images. Different from Dumoulin et al. (2017) and Donahue et al. (2017), the encoded latent space is affected not only by the GAN for the image modality, but also by the Poisson GBN for the text modailty.

The inferred topics at different layers and the inferred connection weights between the topics of adjacent layers are found to be highly interpretable. In particular, we can understand the meaning of each topic by projecting it back to the original data space via \(\left[\prod_{t=1}^{l-1}\boldsymbol{\Phi}^{(t)}\right]\boldsymbol{\phi}_{k}^{(l)}\).

The semantic meaning of each topic and the connection weights between the topics of adjacent layers are highly interpretable, where the topics describe very specific flower characteristics, such as colors, shapes, and parts, at the bottom layer, and become increasingly more general when moving upwards.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space. With a stochastic-gradient MCMC algorithm to sample the global parameters of the VHE text decoder, a reparameterized Weibull distribution based variational encoder to approximate the posterior distribution of the local parameters of the VHE image encoder, and stochastic gradient descent (SGD) to estimate the parameters of the GAN image generator and discriminator, we develop a SG-MCMC/VHE/SGD hybrid inference algorithm to jointly train the components of VHEGAN. Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

* Blei et al. (2017) David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians.

* Changpinyo et al. (2016) Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning.

Self-supervised learning of visual features through embedding images into text topic spaces.

In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively. For Poission GBN, only the number of topics in different layers should be set. In our experiment's setting, we set the number of topics from the first to the third layer to 256, 128 and 64, respectively.

\(\mathbf{x}\rightarrow\mathbf{h}^{(l)},l=1,2,3\)\(\mathbf{x}\rightarrow\mathbf{h}^{(1)}\)\(1\times\) { fully-connected layer with 256 units and Leaky-RELU activation.}\(\mathbf{h}^{(1)}\rightarrow\mathbf{h}^{(2)}\)\(1\times\) { fully-connected layer with 128 units and Leaky-RELU activation.}\(\mathbf{h}^{(2)}\rightarrow\mathbf{h}^{(3)}\)\(1\times\) { fully-connected layer with 64 units and Leaky-RELU activation.} Weibull variational posterior \(q(\mathbf{\theta}^{(l)}|\mathbf{h}^{(l)})\) as shown in Fig. :\(\mathbf{h}^{(l)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(l)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (l)}};l=1,2,3\)\(\mathbf{h}^{(1)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(1)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (1)}}\)\(1\times\) { fully-connected layer with 256 units to \(\mathbf{k}_{\mathbf{\theta}^{(1)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(1)}}\).}\(\mathbf{h}^{(2)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(2)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (2)}}\)\(1\times\) { fully-connected layer with 128 units to \(\mathbf{k}_{\mathbf{\theta}^{(2)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(2)}}\).}\(\mathbf{h}^{(3)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(3)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (3)}}\)\(1\times\) { fully-connected layer with 64 units to \(\mathbf{k}_{\mathbf{\theta}^{(3)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(3)}}\).}

Input: Figure 9: The architecture of WUDVE in VHE and VHEGAN.

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 7: Image reconstruction with real data x (the first row) and corresponding reconstructions G (qΩ (θ |Φ,x)), where each rows represent different sampling from qΩ (θ).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim provides a technically accurate but somewhat overstated description of VHE's approach. While the evidence confirms that VHE uses a deep probabilistic topic model (Poisson GBN) for text decoding and Weibull distribution-based inference network for image encoding, the broader claim about "extracting and relating visual and linguistic concepts" oversimplifies the complex architecture and multiple components involved. The review comment rightfully points out limitations regarding zero-shot learning performance being dependent on similarity between seen/unseen classes. The evidence shows this is achieved through a sophisticated combination of VHE, GAN, and joint training rather than just the two components mentioned in the claim.

<score>3</score>

Justification:

1. The claim is technically correct about the core components but oversimplifies by:
- Not mentioning the critical role of GAN in relating visual-linguistic concepts
- Omitting important details about joint training and optimization
- Not acknowledging limitations in zero-shot learning capabilities

2. Evidence supporting score:
- The paper demonstrates more complex architecture than claimed: "Integrating VHE and GAN, the proposed VHEGAN, tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator..."
- Reviewer notes important limitations: "The problem is only valid when the unseen class distribution is very similar to the given classes."
- Success depends on multiple components working together: "VHEGAN learns a distributed condition jointly with a VAE model"

3. The claim would be more well-stated if it:
- Acknowledged the full system architecture
- Mentioned limitations regarding class similarity requirements
- Described the joint training approach
- Specified the role of GAN in bridging modalities


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To extract and relate visual and linguistic concepts from images and textual descriptions for text-based zero-shot learning (ZSL), we develop variational hetero-encoder (VHE) that decodes text via a deep probabilistic topic model, the variational posterior of whose local latent variables is encoded from an image via a Weibull distribution based inference network.

The review comment to be evaluated is:

This paper developed a generative model to perform simultaneous embedding/generation of images/texts, with application to zero-shot learning. The experiments are extensive.

The novelty of this work is lacking.
The proposed method consists of a bag of existing models proposed by previous works.
But why using a certain model is not justified or explanined.
For example, for image generation, why use GAN instead of VAE.
For text encoding, why use PGBN, instead of recurrent VAE.

The method seems deteched from the problem of ZSL.
Throughout the paper, the authors mostly talk about how to perform joint embedding of texts and images. They give ZSL a touch, but as a side thing.
I would suggest the authors to position this work as a text/iamge embedding/generation work. Then use ZSL as an application.

The writing needs to be significantly improved. In the first paragraph describing the problem of ZSL, the authors end up with talking about the evaluation metric of ZSL.

The evidence to be evaluated is:The proposed VHE uses the Poisson gamma belief network (GBN) of Zhou et al. (2015) as its text decoder, whose output is a bag-of-words count vector, as shown in Fig. 1 (a), and a Weibull distribution based deterministic-upward-stochastic-downward inference network (Zhang et al., 2018) as its image encoder, whose input is a natural image, as shown in Fig. 1 (b). Integrating VHE and GAN, the proposed VHEGAN, as shown in Fig. 1 (c), tightly couples the VHE text generator, VHE image encoder, and GAN image generator and discriminator by imposing the variational posterior of the bag-of-words vector of a document, under the Poisson GBN text decoder, be the distribution of the random noise fed into the GAN image generator; the variational posterior is encoded from an image associated with that document via the Weibull distribution based image encoder; and the GAN discriminator tries to discriminate between that selected image and the generated one.

To extract hierarchical latent representation from these high-dimensional sparse count vectors \(\mathbf{t}_{c}\), we choose the Poisson GBN (Zhou et al., 2015), which can be considered as a member of deep exponential families (Ranganath et al., 2015). It consists of multiple gamma distributed stochastic hidden layers, generalizing the "shallow" Poisson factor analysis (Zhou et al., 2012; Zhou and Carin, 2015) that is closely related to latent Dirichlet allocation (LDA) (Blei et al., 2003) into a deep setting. More specifically, modeling \(\mathbf{t}_{c}\) under the Poisson likelihood, its generative model with \(L\) hidden layers, from top to bottom, is expressed as \[\mathbf{\theta}_{c}^{(L)} \sim\text{Gam}\left(\mathbf{r},1/s_{c}^{(L+1)}\right),\,\ldots,\mathbf{ \theta}_{c}^{(l)}\sim\text{Gam}\left(\mathbf{\Phi}^{(l+1)}\mathbf{\theta}_{c}^{(l+1)},1/s_{c}^{(l+1)}\right),\,\ldots,\] \[\mathbf{\theta}_{c}^{(1)} \sim\text{Gam}\left(\mathbf{\Phi}^{(2)}\mathbf{\theta}_{c}^{(2)},1/s_{c} ^{(2)}\right),\,\mathbf{t}_{c}\!\sim\!\text{Pois}\left(\mathbf{\Phi}^{(1)}\mathbf{\theta}_ {c}^{(1)}\right), \tag{1}\] where the hidden units \(\mathbf{\theta}_{c}^{(l)}\in\mathbb{R}_{+}^{K_{l}}\) of layer \(l\) are factorized under the gamma likelihood into the product of the factor loading \(\mathbf{\Phi}^{(l)}\in\mathbb{R}_{+}^{K_{l-1}\times K_{l}}\) and hidden units of the next layer, \(\mathbb{R}_{+}=\{x,x\geq 0\}\), \(s_{c}^{(l)}>0\), and \(K_{l}\) is the number of topics (factors) of layer \(l\).

Choosing a deep generative model as the text decoder distinguishes VHE from Gomez et al. (2017) that uses LDA (Blei et al., 2003), a "shallow" probabilistic model, and Ba et al. (2015) that uses non-probabilistic black-box DNNs to learn class-specific document features.

To avoid using iterations to sample the latent representation of a testing document from its posterior, Zhang et al. (2018) develop Weibull upward-downward variational encoder (WUDVE) that directly projects an observed bag-of-words vector into its posterior distribution given a random sample of the global parameters, leading to extremely fast inference for out-of-sample prediction.

In this paper, we leverage WUDVE as the variational inference network to approximate the true posterior distribution \(p(\mathbf{\theta}\,|\,\mathbf{t},\mathbf{\Phi})\), where \(\mathbf{\theta}=\{\mathbf{\theta}^{(l)}\}_{1,L}\) denotes the set of local parameters, but change the input of WUDVE from the bag-of-words vector \(\mathbf{t}\) into an image \(\mathbf{x}\) associated with \(\mathbf{t}\).

where, with \(f(\mathbf{x})\) representing a feature vector deterministically projected from \(\mathbf{x}\) via an off-the-shelf pre-trained deep neural network \(f(\mathbf{\cdot})\), we define \[q_{\mathbf{\Omega}}(\mathbf{\theta}^{(l)}\,|\,\mathbf{\Phi}^{(l+1)},\mathbf{ \theta}^{(l+1)},\mathbf{x}) =\text{Weibull}(\mathbf{k}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))+\mathbf{\Phi}^{(l +1)}\mathbf{\theta}^{(l+1)},\mathbf{\lambda}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))),\] \[q_{\mathbf{\Omega}}(\mathbf{\theta}^{(L)}\,|\,\mathbf{r},\mathbf{x}) =\text{Weibull}(\mathbf{k}_{\mathbf{\Omega}}^{(L)}(f(\mathbf{x}))+\mathbf{r}, \mathbf{\lambda}_{\mathbf{\Omega}}^{(L)}(f(\mathbf{x}))), \tag{3}\]where \(\mathbf{k}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))\in\mathbb{R}^{K_{l}}\) and \(\mathbf{\lambda}_{\mathbf{\Omega}}^{(l)}(f(\mathbf{x}))\in\mathbb{R}^{K_{l}}\) are deterministically transformed from \(f(\mathbf{x})\). We use \(\mathbf{\Omega}\) to denote all the parameters of the inference network.

Given \(\mathbf{\Phi}\) of the Poisson GBN, to approximate the posterior distribution of the local parameters \(\mathbf{\theta}\) given text description \(\mathbf{t}\), we use WVDVE \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\), whose input is an image \(\mathbf{x}\), and learn its parameters \(\mathbf{\Omega}\) by maximizing the evidence lower bound (ELBO) as \[\max_{\mathbf{\Omega}}L_{\text{VHE}}(\mathbf{\Omega})=\mathbb{E}_{(\mathbf{x},\mathbf{t}) \sim p_{data}(\mathbf{x},\mathbf{t})}\left\{\mathbb{E}_{q_{\mathbf{\Omega}}(\mathbf{\theta} \,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln p\left(\mathbf{t}\,|\,\mathbf{\Phi}^{(1)},\mathbf{\theta }^{(1)}\right)-\ln\frac{q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}{p( \mathbf{\theta}\,|\,\mathbf{\Phi})}\right]\right\}, \tag{4}\]

To further improve the proposed VHE, below we show how to integrate it with a modified GAN to construct VHEGAN.

In this paper, to exploit the synergy between variational inference and generative adversarial learning to achieve improved performance for both VHE and GAN, we use the variational posterior of VHE given an image, which is the one fed into the GAN image discriminator, as the random noise distribution of the GAN image generator. More specifically, we modify the GAN objective function as \[\min_{G}\max_{D}L_{\text{GAN}}(D,G)=\min_{G}\max_{D}\mathbb{E}_{\mathbf{x}\sim p_ {data}(\mathbf{x})}\left\{\ln D(\mathbf{x})-\mathbb{E}_{\mathbf{\theta}\sim q_{\mathbf{\Omega} }(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})}\left[\ln D(G(\mathbf{\theta}))\right]\right\} \tag{6}\] where \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) is defined as in (2).

It is important to note that the update of the VHE image encoder parameters \(\mathbf{\Omega}\) is related to not only the ELBO of VHE, but also a modified GAN mini-max objective function, forcing the variational posterior \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\) to serve as a bridge between the image and text modalities. For inference, since the Weibull distribution \(\theta\sim\text{Weibull}(k,\lambda)\) can be reparameterized as \(\theta=\lambda(-\ln(1-\epsilon))^{1/k}\), where \(\epsilon\sim\text{Uniform}(0,1)\), it is convenient to calculate the gradients of (7) with respect to \(\{\mathbf{\Omega},G,D\}\) by back-propagation (BP) using reparameterized random samples from \(q_{\mathbf{\Omega}}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x})\).

The proposed VHE by itself and VHEGAN can both be used for ZSL in the same way: with the Poisson GBN based VHE text decoder and the WUDVE based VHE image encoder, to perform zero-shot classification for a test image \(\mathbf{x}\) from an unseen class, we predict its class label as

Note that VHEGAN differs from VHE in that the optimization for the image encoder parameters \(\mathbf{\Omega}\) is also influenced by the modified GAN min-max objective function, as shown in (7). Conventional ZSL methods often make a restrictive assumption that a test example only comes from one of the unseen classes. A more challenging setting where a test example can come from either a seen class or an unseen one is known as generalized ZSL (GZSL) (Socher et al., 2013; Verma et al., 2018; Chao et al., 2016).

To remedy these drawbacks, text-based ZSL have been developed using easily accessible textural descriptions such as Wikipedia.

From the view of the model, the proposed VHEGAN is able to generate not only the visual images but also the documents with a shared probabilistic space, which can be also seen as a multimodal learning. In Srivastava & Salakhutdinov (2012, 2014), a deep Boltzmann machine is developed to model the joint generation of image feature and BOW tags. Moving beyond binary hidden units, Wang et al. (2018) propose a multimodal Poisson gamma belief network, similar to the document decoder of VHEGAN.

Both these two approaches are two-step models with the additional textual feature extraction realized by LSTM and "shallow" LDA, respectively, whereas the VHEGAN can realize joint optimization with hybrid SGMCMC/VHE/SGD algorithm.

VHE builds a variational posterior conditioned on images to approximate Gamma latent variables to generate the BOW vector of documents. To solve the problem of complete random generate process in GAN, conditional GANs (CGAN) (Mirza & Osindero, 2014; Reed et al., 2016) are alsodeveloped, where the conditions are labels, or attributes learned as point estimations separately with GAN, while VHEGAN learns a distributed condition jointly with a VAE model.

Different from them that only use discriminator as the similarity measurement, VHEGAN fuse VHE for documents and GAN for images, making itself realize not only an image-to-image "autoencoder" like BiGAN and ALI, but also an image-to-text and text-to-image "hetero-encoder".

The 4096 dimensional image features, \(f(\mathbf{x})\) in (3), from the fc1 layer of the pre-trained VGG16 network (Simonyan & Zisserman, 2015) are fed into the VHE image encoder, and the size of each generated RGB image is set as \(64\times 64\).

except for the proposed VHEGAN that aims to find a shared semantically meaningful latent space between the image and text modalities, none of the other methods have generative models for both modalities, regardless of whether they learn a classifier or a distance metric in a latent space for ZSL.

Although VHEGAN does not have a text encoder to directly project a document to the shared latent space, given a document and a set of topics inferred during training, we use the upward-downward Gibbs sampler of Zhou et al. (2016) to draw \(\{\mathbf{\theta}^{(l)}\}_{l=1,L}\) from its conditional posterior under the Poisson GBN, which are then fed into the GAN image generator to generate random images.

We train VHEGAN on CUB2011-easy and then perform image generation given the textual descriptions of four different unseen classes. Comparing the generated images, as shown in the third to fifth rows of Fig. 1(a), with their corresponding texts and example real images, as shown in the first and second rows of Fig. 1(a), respectively, clearly suggest that the proposed VHEGAN successfully transfers the learned knowledge of the seen classes to the unseen ones, generating images semantically (visually) similar to their corresponding textual descriptions (real images).

As shown in Fig. 1(b), VHEGAN successfully generates flower images that semantically match their corresponding textual descriptions and visually resemble their corresponding example real images. **Image retrieval given a text query:** For image \(\mathbf{x}_{i}\), via the text generator and image encoder, we generate a bag-of-words vector \(\hat{\mathbf{t}}_{i}\) using Poisson mean \(\mathbf{\Phi}^{(1)}\mathbf{\theta}^{(1)}\) as \[\hat{\mathbf{t}}_{i}\,|\,\mathbf{\theta}_{i}\sim p(\mathbf{t}\,|\,\mathbf{\Phi},\mathbf{\theta}_{ i}),\,\mathbf{\theta}_{i}\,|\,\mathbf{x}_{i}\sim q_{\Omega}(\mathbf{\theta}\,|\,\mathbf{\Phi},\mathbf{x} _{i}). \tag{9}\] Giving the textural description \(\mathbf{t}\) of an unseen class as the text query, we retrieve the top five images ranked by the cosine distances between \(\mathbf{t}\) and \(\hat{\mathbf{t}}_{i}\).

With (9), we may generate a textual description given an image, as shown in Fig. 4, where the true and generated key words are displayed on the left and right of the input image, respectively. It is clear that VHEGAN successfully captures the flower colors, shapes, and locations to impute relevant key words given an input image. **Latent space interpolation:** In order to understand the learned manifold in the latent space, given two texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), after drawing \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) from their own conditional posteriors using Gibbs sampling, we interpolate between \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\) and use these interpolated latent variables to generate both the images via the GAN image generator and texts via the VHE text decoder. We show the results in Fig. 5, where the first and last columns show the true texts \(\mathbf{t}_{1}\) and \(\mathbf{t}_{2}\), and the images generated from \(\mathbf{\theta}_{1}\) and \(\mathbf{\theta}_{2}\), respectively, while the second to third columns show the generated texts and images from the interpolated \(\mathbf{\theta}\)'s. The strong correspondences between the generated images and texts and smooth changes between adjacent columns suggest that the VHEGAN latent space is semantically meaningful for both the image and text modalities.

Given a random draw, the concatenation of \(\{\mathbf{\theta}^{(l)}\}_{l=1}^{L}\) is fed into the GAN image generator to generate a corresponding image. Shown in Fig. 6 are four random draws, for each of which we show its top seven words and its generated image, whose relationships are clearly interpretable, suggesting that VHEGAN is able to recode the key information of both modalities and the relationships between them.

We note for the VHEGAN, its VHE image encoder and GAN component together can also be viewed as an "autoencoding" GAN for images. Different from Dumoulin et al. (2017) and Donahue et al. (2017), the encoded latent space is affected not only by the GAN for the image modality, but also by the Poisson GBN for the text modailty.

The inferred topics at different layers and the inferred connection weights between the topics of adjacent layers are found to be highly interpretable. In particular, we can understand the meaning of each topic by projecting it back to the original data space via \(\left[\prod_{t=1}^{l-1}\boldsymbol{\Phi}^{(t)}\right]\boldsymbol{\phi}_{k}^{(l)}\).

The semantic meaning of each topic and the connection weights between the topics of adjacent layers are highly interpretable, where the topics describe very specific flower characteristics, such as colors, shapes, and parts, at the bottom layer, and become increasingly more general when moving upwards.

To exploit and explore the relationships between the images and texts for text-based zero-shot learning (ZSL), we propose variational hetero-encoder (VHE) that encodes an image, via a Weibull distribution based inference network, to decode its textual description, via a deep probabilistic topic model. We introduce VHE randomized generative adversarial network (VHEGAN) to further strengthen the interaction between the image and text modalities in their shared latent space. With a stochastic-gradient MCMC algorithm to sample the global parameters of the VHE text decoder, a reparameterized Weibull distribution based variational encoder to approximate the posterior distribution of the local parameters of the VHE image encoder, and stochastic gradient descent (SGD) to estimate the parameters of the GAN image generator and discriminator, we develop a SG-MCMC/VHE/SGD hybrid inference algorithm to jointly train the components of VHEGAN. Both quantitative results on ZSL and generalized ZSL task and qualitative analysis suggest that VHEGAN is able to infer the parameters of probabilistic deep neural networks to extract and relate visual and linguistic concepts from the training data, and use the inferred networks to perform a wide variety of conditional or unconditional image/text generation/retrieval tasks.

* Blei et al. (2017) David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians.

* Changpinyo et al. (2016) Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning.

Self-supervised learning of visual features through embedding images into text topic spaces.

In this appendix section, we give more detailed model architecture for all of three parts (image encoder named WUDVE, image decoder named Poission GBN, and image decoder GAN) of VHEGAN. The input of the VHEGAN is abbreviated as \(\{\mathbf{x},\mathbf{t}\}\) pair, which represents images and texts, respectively. For Poission GBN, only the number of topics in different layers should be set. In our experiment's setting, we set the number of topics from the first to the third layer to 256, 128 and 64, respectively.

\(\mathbf{x}\rightarrow\mathbf{h}^{(l)},l=1,2,3\)\(\mathbf{x}\rightarrow\mathbf{h}^{(1)}\)\(1\times\) { fully-connected layer with 256 units and Leaky-RELU activation.}\(\mathbf{h}^{(1)}\rightarrow\mathbf{h}^{(2)}\)\(1\times\) { fully-connected layer with 128 units and Leaky-RELU activation.}\(\mathbf{h}^{(2)}\rightarrow\mathbf{h}^{(3)}\)\(1\times\) { fully-connected layer with 64 units and Leaky-RELU activation.} Weibull variational posterior \(q(\mathbf{\theta}^{(l)}|\mathbf{h}^{(l)})\) as shown in Fig. :\(\mathbf{h}^{(l)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(l)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (l)}};l=1,2,3\)\(\mathbf{h}^{(1)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(1)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (1)}}\)\(1\times\) { fully-connected layer with 256 units to \(\mathbf{k}_{\mathbf{\theta}^{(1)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(1)}}\).}\(\mathbf{h}^{(2)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(2)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (2)}}\)\(1\times\) { fully-connected layer with 128 units to \(\mathbf{k}_{\mathbf{\theta}^{(2)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(2)}}\).}\(\mathbf{h}^{(3)}\rightarrow\mathbf{k}_{\mathbf{\theta}^{(3)}},\mathbf{\lambda}_{\mathbf{\theta}^{ (3)}}\)\(1\times\) { fully-connected layer with 64 units to \(\mathbf{k}_{\mathbf{\theta}^{(3)}}\) and \(\mathbf{\lambda}_{\mathbf{\theta}^{(3)}}\).}

Input: Figure 9: The architecture of WUDVE in VHE and VHEGAN.

Input: Figure 10: Generative model in GAN of VHEGAN

Input: Figure 1: The overall model architecture of variational hetero-encoder (VHE), consisting of (a) and (b), and VHE randomized GAN (VHEGAN), consisting of (a), (b), and (c), where (a) is the VHE text decoder, a three-hidden-layer Poisson gamma belief network, (b) is the VHE image encoder, a Weibull distribution based deterministic-upward (denoted with orange arrows) and stochastic-downward (denoted with blue arrows) variational encoder, and (c) is the GAN image generator and discriminator of VHEGAN.

Input: Figure 11: Discriminant model in GAN of VHEGAN

Input: Figure 8: (a) is an example of learned hierarchical topics; (b) is the generated images, from the first to fourth rows, using the third, second, first, and the concatenation of all hidden layers fed into GAN, respectively. Each one in the bottom row is a real image belonging to the corresponding class of that column.

Input: Figure 7: Image reconstruction with real data x (the first row) and corresponding reconstructions G (qΩ (θ |Φ,x)), where each rows represent different sampling from qΩ (θ).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that while the core technical description of VHE's architecture and functionality is accurate, the claim's presentation could be more precise. The evidence thoroughly documents the model's components, mathematical foundations, and implementation details. However, the initial claim is densely packed and could better emphasize the key innovations and purpose.

The evidence shows the paper successfully implements a variational hetero-encoder using Weibull distribution and Poisson GBN for text-image learning, but the initial claim tries to compress too much technical detail into a single sentence, making it harder to grasp the main contribution. A more focused presentation of the core concept would be clearer.

<score>2</score>

Justification:
1. The evidence substantiates the technical claims about model architecture and methodology
2. The mathematical formulations and experimental results support the described approach
3. The reviewer comments note that while experiments are extensive, the writing needs improvement
4. The dense technical phrasing in the claim, while accurate, could be more accessible
5. The claim would benefit from better emphasizing the key contributions rather than trying to include all technical details in one sentence