Auto-Embedding Generative Adversarial Networks for High Resolution Image Synthesis
Yong Guo Qi Chen Jian Chen Qingyao Wu Qinfeng Shi Mingkui Tan

	
	
	
	
	
	Auto-Embedding Generative Adversarial Networks for High Resolution Image Synthesis
    
		Yong Guo^*^* Authors contributed equally.,
		Qi Chen^*,
		Jian Chen^*,
		Qingyao Wu,
		Qinfeng Shi,
		and Mingkui Tan^†^† Corresponding author.
		
		
	
    December 30, 2023
==============================================================================================================================================================


		Generating images via generative adversarial network (GAN) has attracted much attention recently. However, most of the existing GAN-based methods can only produce low resolution images of limited quality.
		Directly generating high resolution images using GANs is nontrivial, and often produces problematic images with incomplete objects.
		To address this issue, we develop a novel GAN called Auto-Embedding Generative Adversarial Network (AEGAN), which simultaneously encodes the global structure features and captures the 
		fine-grained details. In our network, we use an autoencoder to learn the intrinsic high-level structure of real images and design a novel denoiser network to 
		provide photo-realistic details for the generated images.
		In the experiments, we are able to produce 512 × 512 images of promising quality directly from the input noise. 
		The resultant images exhibit better perceptual photo-realism, i.e., with sharper structure and richer details, than other baselines on several datasets, including Oxford-102 Flowers, Caltech-UCSD Birds (CUB), High-Quality Large-scale CelebFaces Attributes (CelebA-HQ), Large-scale Scene Understanding (LSUN) and ImageNet.
	
	
	
		Generative models, adversarial learning, low-dimensional embedding, autoencoder.
	
	
	
	

§ INTRODUCTION

	
	Building a generative model that produces photo-realistic images of high resolution has been a challenging problem in the field of computer vision. 
	Compared to low resolution images, the high resolution images of promising quality often provide richer information and benefit the training of deep neural networks (DNNs) in many real-world applications,
	such as texture synthesis <cit.>, super-resolution <cit.> and attribute editing <cit.>.
	However, how to produce high-quality data when increasing the image resolution still remains an open question.
	
	Recently, generative adversarial networks (GANs) <cit.> have achieved great success and become the workhorse of many challenging tasks, including
	image generation <cit.>
	video prediction <cit.> and image translation <cit.>. Typically, GANs learn to generate data by playing a two-player game: a generator attempts to produce samples from a simple prior distribution (e.g., Gaussian distribution), while a discriminator acts as a judge to distinguish the generated data from the real one.
	
	When generating images directly from the prior distribution, 
	the quality of the images generated
	by most of the existing models can be quite limited, especially when synthesizing very high resolution images. To be specific, deep generative models often produce meaningless images that may contain multiple distorted regions with blurred structure <cit.>. To illustrate this issue, in Fig. <ref>, 
	we compare generated samples of different resolutions produced by the well-known model DCGAN <cit.> and by our proposed method. 
	From Fig. <ref> top, the low resolution images of size 64 × 64 and 128 × 128 preserve clear and complete object structure, including stamen, petals, etc. However, for the higher resolutions of 256 × 256 or 512 × 512, DCGAN fails to produce meaningful samples and yields very poor results compared to low resolutions. 
	Regarding this issue, it is necessary and important to explore a new method to improve the performance of generative models when increasing the image resolution.
	
	
	
	The difficulties of generating high resolution images are generally attributed to two reasons. 
	The first is that it is hard to directly learn a mapping between the prior distribution (i.e., input noise) 
	and the distribution of high-dimensional real data, due to the large distribution gap.
	Since high-dimensional data often lie on some low-dimensional manifold <cit.>,
	we use low-dimensional embedding to uncover the image's
	structural information, which acts as a bridge to connect the prior distribution with the distribution of real data. This embedding is extracted by an autoencoder that is able to reconstruct the real images while preserving clear photo-structure.
	More critically, matching a low-dimensional embedding representation can significantly ease the training of GANs compared to learning high resolution images directly.
	
	The second reason regarding this issue is that there is no additional knowledge, such as label or semantic information obtained from real data, to help the model training.
	Note that the prior distribution is usually very simple, e.g., standard Gaussian distributions or uniform distributions <cit.>, and is independent of the real data, which may lose the original semantic information.
	In practice, GANs can easily lose the primary characteristics of data 
	that are used by humans in image recognition,
	resulting in meaningless images (see results in Section <ref>). 
	On the contrary, the extracted embedding, which contains the spatial structural information, is able to act as a kind of additional knowledge to improve the training.
	
	By translating the input noise into an embedding and decoding it into a corresponding synthetic image, however,
	the model may introduce many noisy artifacts.
	To alleviate this issue, we further develop an adversarial denoiser network to enhance the photo-realism of the generated images. This denoiser model takes synthetic images as input and forces them to be perceptually indistinguishable from the real images in terms of texture details. In practice, we feed the generated images into the denoiser model to remove these artifacts and provide photo-realistic details.
	
	Based on the above intuitions, we propose the Auto-Embedding Generative Adversarial Networks (AEGANs) for high resolution image synthesis. 
	Unlike the considered baselines, AEGAN is able to consistently generate perceptually promising images at different resolutions (see Fig. <ref> bottom).
	Moreover, the proposed method significantly outperforms the alternative approaches and produces photo-realistic 512 × 512 images, which simultaneously capture the high-level photo-structure and preserve the low-level details.
	
	In this paper, we make the following contributions:
	
		
  * We devise a novel Auto-Embedding Generative Adversarial Net (AEGAN) 
		that generates
		high resolution images by learning a latent embedding extracted from an autoencoder. 
		AEGAN exploits the high-level photo-structure and acts as a bridge to connect the distributions of the input noise and real data. 
		As a result, the proposed method can produce much more meaningful images with clear structure and rich details.
		
  * We develop a new denoiser network that uses
		an encoder-decoder network as the generator to remove artifacts and provide 
		photo-realistic details in the generated images.
		
  * The proposed method is able to produce high resolution images with promising quality. For example, when the desired resolution is 512 × 512, the generated images are of much higher quality than those obtained using the considered methods.
	
	
	

§ RELATED WORK

	
	

 §.§ Generative Adversarial Networks

	Recently, Generative Adversarial Networks (GANs) <cit.> have shown promising performance for generating natural images,
		such as DCGAN <cit.>, WGAN <cit.>, WGAN-GP <cit.>, LAPGAN <cit.>, etc.
		More recently, Chen propose a modification of generators in GANs, called Self-Modulation framework <cit.> that improves the performance of the generated images 
		by modulating the architectural features of GAN generators.
		Moreover, GANs have also been applied to range of other interesting applications, such as text to image synthesis <cit.>, super-resolution <cit.>, image inpainting <cit.> and so on.
	
	
	
	

 §.§ High-Resolution Image Generation

	Generating high resolution images has gained much attention in the last few years in light of the advances in deep learning.
		To achieve this, one can compare the distribution divergences of the real and the generated data in a low-dimensional data space, such as StackGAN <cit.>, StackGAN-v2 <cit.>, AGE <cit.> and AttnGAN <cit.>.
	
	
	
		In StackGAN, the low resolution image obtained from Stage-I and the conditional text are used to produce high resolution image through Stage-II GAN, just as is done in super resolution. 
		
			However, for unconditional image generation, the conditional text is not necessary and can be discarded. Moreover, the Stage-II GAN is trained by minimizing the JS distance between real and fake data. Unlike StackGANs,
			in addition to minimizing the JS divergence,
			we seek to increase the fidelity of images while preserving the image content by introducing a pixel-wise loss.
		
	
	
		StackGAN-v2 is an extension of StackGAN that uses a cascade mechanism to generate the image in gradually growing resolution. Based on StackGAN-v2, AttnGAN introduces the attention mechanism to improve the visual quality of the generated images.
		AGE uses an autoencoder to project both the real data and the generated data into a latent space. To compare the distribution divergence, AGE aligns a simple prior distribution in the latent space and the data distribution in the image space.
		More recently, Zhang design a SAGAN model <cit.> that incorporates the self-attention mechanism into the generator and discriminator. Based SAGAN model,
		Brock propose a BigGAN model <cit.> that improves the performance using a suite of tricks for training.
	
	

 §.§ Autoencoder GAN

	Autoencoders have been widely used in GANs and a plethora of attempts have been made to improve the training. For example, the adversarial autoencoder (AAE) leverages an autoencoder to match the distribution of latent embedding with the prior.
		Boundary-equilibrium GANs <cit.> and energy-based GANs <cit.> use an autoencoder as the discriminator to stabilize the training. Warde-Farley et al. <cit.> combine an autoencoder loss with the GAN loss by matching high-level feature similarity. 
		CycleGANs <cit.> and DualGANs <cit.>
		construct a bidirectional loss with autoencoder and GAN for data translation between two domains.
		The methods described in <cit.>, which are based on autoencoders, combine a Variational Autoencoder (VAE) with GAN
		using variational inference to solve the intractability of the marginal likelihood in GAN.
		Plug and play generative networks (PPGN) <cit.> combines an autoencoder loss with a GAN loss and a classification loss. 
	
	
	
		Very recently, Karras <cit.> have proposed a new progressively-growing training method that gradually adds one more block per stage in both the generator and discriminator networks. In this way, it can significantly improve the synthetic image quality.
		However, this training method takes a long time to converge (i.e., more than two days), while DCGANs and
		our proposed AEGANs only require several hours for training.
		Unlike other methods, in this paper, we use an autoencoder to transform high-dimensional data to a low-dimensional embedding in the latent space. The extracted embedding contains rich structural information of real images and aids the training to produce images of promising quality.
	
	
	
	

§ THE PROPOSED METHOD

	
	
		Considering that high-dimensional data often lie on some low-dimensional manifold, we seek to exploit a low-dimensional embedding extracted from large training data to improve the performance of GANs.
	
	To this end,
	we present a novel generative method called Auto-Embedding Generative Adversarial Network (AEGAN). 
	Corresponding to the overall structure in Fig. <ref>, the detailed algorithm is shown in Algorithm <ref>.
	
	We propose to train GANs by matching the distribution of the low-dimensional embeddings on a latent manifold (see Step 1) instead of the distributions of pixel-level images, 
	
		which is very different from the existing GANs methods.
	
	Specifically, we use the embeddings extracted from an autoencoder to bridge the distribution gap between the input noise and real data. 
	The embeddings often contain rich structural information and help to generalize well with high resolution images.
	After using the autoencoder to extract the low-dimensional embedding, we learn a GAN in the embedding space to effectively exploit the structural information. 
	Moreover, we devise a denoiser network to remove artifacts/noises and refine the photo-realistic details (see Step 2).
	During training, we can use a fine-tuning strategy (which, however, is optional) to further improve the performance. 
	As shown in Fig. <ref>, the whole scheme mainly includes three important components, namely an autoencoder, a generative module, and a denoiser network.
	In this section, we will depict each part of the method in the following subsections.
	
	

 §.§ Learning Embedding by Autoencoder

	In this paper, we seek to bridge the distribution gap between the input noise and the real images using a latent embedding extracted from an autoencoder.
	The autoencoder contains an encoder H which maps the high resolution images into a low-dimensional embedding and a decoder F which translates the latent embedding back to high resolution images. Here, the encoder is a fully convolutional network that extracts the high-level features of the data, while the decoder is a fully deconvolutional model that effectively recovers the high resolution images. Given a collection of n real samples {_i}_i=1^n, we minimize the following reconstruction loss:
	
    L_AE(θ_F, θ_H) = 1/n∑_i=1^n  F(H(_i)) - _i _1,

	where θ_H and θ_F denote the parameters of the encoder H and the decoder F, respectively.
	Here, we use L1 loss instead of the L2 loss to better capture the high-level and salient features <cit.>. This feature becomes the embedding which is able to uncover the primary characteristics of real data, e.g. the structural information or image style.
	In this way, the learned latent embedding has the potential to produce meaningful images.
	
	There are two advantages of extracting the embedding from data using an autoencoder.
	Firstly, the autoencoder extracts the high-level features which effectively preserve the primary data characteristics, e.g., structural information <cit.>, to reconstruct the original images.
	It is helpful to train a generator on the extracted features to produce meaningful samples (see results in Section <ref>).
	In practice, we set the embedding to 32 × 32 with 64 channels to represent the 512 × 512 RGB images. In this sense, the dimensionality has been largely reduced by over 10 fold
	in terms of the number of pixels.
	Secondly, the generator only needs to learn a mapping from the input noise z ∈ℝ^100 to the extracted low-dimensional embedding rather than the high-dimensional images, which greatly facilitates
	the training of deep generative models.
	
	
	
	

 §.§ Adversarial Embedding Generator

	With the extracted embedding which contains image structural information, we seek to exploit it to improve the training of GANs.
	We define a generative model to match the meaningful embedding extracted from real data.
	The objective function of the generative model is as follows:
	
    L_E(θ_G_E, θ_D_E)    = 1/n∑_i=1^n log D_E(H(_i)) 
        + 1/n∑_j=1^n log (1 - D_E(G_E(_j))) ,

	where H() denotes the extracted embedding from real images and  denotes the input noise sampled from a prior distribution.
	We optimize G_E and D_E in an alternating manner by solving the minimax problem:
	
    min_θ_G_Emax_θ_D_E  L_E(θ_G_E, θ_D_E).

	
	During training, we fix the parameters of autoencoder and train the GAN model to match the distribution of embeddings in the latent space.
	Note that we share the decoder during both extracting the high-level feature in the autoencoder and recovering the synthetic high resolution images from the generated fake embeddings.
	Ideally, with a well-trained model, we can map a noise vector to obtain a latent embedding with photo-structure information, which can be then decoded into a meaningful high resolution image.
	
	

 §.§ Adversarial Denoiser Network

	We observe that the decoded images often encounter visual noisy artifacts after going through the pipeline of the GAN and the decoder (see Figs. <ref> and <ref>). An inevitable problem is how to remove these artifacts and provide photo-realism in the generated images.
	We thus develop a denoiser network to address this issue.
	
	Actually, refining images using adversarial training has been developed in <cit.>, which uses a pixel-level method, i.e., a chain of convolutional layers without striding or pooling to refine the blurring regions.
	However, when producing high resolution images, the synthetic images often contain severe visual artifacts which are hard to be refined using such a pixel-level method without scaling.
	To address this issue, we develop a new denoiser model ϕ that contains multiple layers of convolution and deconvolution operators. The structure is shown in Fig. <ref> (bottom).
	
	The denoiser network ϕ follows the encoder-decoder design.
	It is worth mentioning that the stride operation of convolution can effectively extract the primary features of the data and discard pixel-level noises <cit.>.
	To improve the fidelity of the synthetic images, we introduce an adversarial loss to provide 
	photo-realistic details
	in the generated images.
	We implement the D_R as a convolutional network which outputs the probability of samples being positive.
	In addition to denoising artifacts from the generated images, we also have to preserve the content of generated images. To this end, a pixel-wise loss is taken into account.
	As a result, we can simultaneously preserve the image content and increase the perceptual fidelity of the images. The training objective becomes
	
    L_ϕ(θ_ϕ, θ_D_R, θ_G_E, θ_F)     =1/n∑_i=1^n log D_R(H(_i)) 
        +1/n∑_j=1^n log (1  - D_R(ϕ(_j)))  +λϕ(_j)  -_j _1,

	where = F(G_E()) indicates the synthetic images before feeding them into the denoiser network.
	During training, we fix G_E and θ_F and only update the adversarial denoiser network.
	Similar to the embedding generative model, we also train the denoiser model by solving the minimax problem:
	
    min_θ_ϕmax_θ_D_R  L_ϕ(θ_ϕ, θ_D_R, θ_G_E, θ_F).

	
	With the denoiser network, the noisy artifacts can be effectively removed from the generated images. As a result, the proposed method is able to produce promising images with high perceptual fidelity.
	
	
	

 §.§ Training and Inference Method

	The proposed method consists of three components, namely the autoencoder, the embedding generative model and the denoiser network. To effectively train the whole model, we adopt a step-wise strategy to train all the components and then conduct end-to-end fine-tuning to obtain a new unified model. The training procedure is shown in Algorithm <ref>.
	
	We divide the training of the proposed AEGAN into three steps.
	In the first step, we train the autoencoder by minimizing the reconstruction loss to extract the low-dimensional embedding. The learned embedding is able to capture the image structural information and effectively recover the high resolution images.
	In the second step, by fixing the autoencoder model, we train an embedding generative model and a denoiser network in an alternating manner. In the proposed model, the autoencoder acts as a bridge that connects all the three components.
	
	However, when one of the components fails, there is no way to correct it. To ensure the final performance and obtain a new unified model, in the last step, we conduct end-to-end training to fine-tune the whole model by optimizing the following joint loss:
	
    min_𝒫max_𝒬  L_AE(θ_F, θ_H)  + L_E(θ_G_E, θ_D_E)  + L_ϕ(θ_ϕ, θ_D_R, θ_G_E, θ_F),

	where 𝒫 ={θ_G_E, θ_F, θ_ϕ, θ_H} 
	and 𝒬={θ_D_E, θ_D_R}.
	In each iteration of the final step, we first update the discriminators D_E and D_R using the gradients propagated from L_E and L_ϕ, respectively.
	We then
	backpropagate the signals of each loss through the model and the gradient for each component will accumulate.
	The fine-tuning step ensures the coherence of the whole model and is able to achieve slightly better performance. 
	In practice, we observe that failure of the components rarely occurs. Thus, the fine-tuning step is optional.
	Compared to directly optimizing the objective in Eqn. (<ref>), each separately trained model can be viewed as a good initialization and can be used to accelerate the training to obtain a new unified model.
	
	For inference, 
	we sample a vector ∈^100 from a prior distribution to produce a latent embedding. 
	We then feed the latent embedding into the decoder F to produce a corresponding high resolution image.
	Finally, we take the generated images into the denoiser model. Through the pipeline of matching the random noise to a high resolution image, AEGAN is able to generate 512 × 512 RGB images with promising quality in terms of both quantitative and qualitative results. 
	
	
	
	
	
	

 §.§ Model Architecture

	
		We build the embedding generator G_E and the decoder F with a stack of up-sampling blocks (i.e., a strided deconvolutional layer followed by a residual blocks <cit.>). The number of blocks is determined by the upscaling factor of the feature maps.
		In the experiment, we add 3 and 4 up-sampling blocks in G_E and F, respectively. 
		For the encoder H, we build the model with a stack of down-sampling blocks to encode RGB images to the low-dimensional embeddings. Each down-sampling block contains a strided deconvolutional layer followed by a residual blocks <cit.>. Similar to the design of the decoder F, we also add 4 down-sampling blocks in H.
		For the discriminators D_R and D_E, we build the model with a stack of strided convolution layers to down-sample the input images or embeddings.
		We add a Batch Normalization <cit.> and a ReLU <cit.> layer behind each convolutional layer.
		The denoiser network ϕ follows the encoder-decoder design. The input generated images are fed into several down-sampling modules (i.e., a strided convolutional layer followed by a Batch Normalization and a LeakyReLU <cit.> layer) until it has a size of 512. A series of up-sampling modules (i.e., strided deconvolutional layer followed by a Batch Normalization and a ReLU layer) are then used to generate 512×512 RGB high-resolution images. 
		The strides of all the strided convolutional and deconvolution layer are set to 2 and 1/2, respectively.
	
	
	

 §.§ Implementation Details

	We follow the experimental settings used in <cit.> to train the proposed AEGAN in PyTorch.
	There is no preprocessing or data augmentation of the training samples other than resizing all the training images to 512 × 512 RGB images.
	In all the experiments, we set the embedding to 32 × 32 with 64 channels to represent the 512 × 512 RGB images.
	The weights are initialized from a normal distribution with zero-mean and standard deviation of 0.02.
	The hyperparameter λ in Eqn. (<ref>) is set to λ=100. We use ReLU activation in the upsamplers, such as the embedding generator in GANs and the decoder in the autoencoder, while using LeakyReLU in the downsamplers including discriminators in GANs and the encoder in the autoencoder. The slope of the leak in LeakyReLU is set to 0.2.
	In the training, we use Adam <cit.> with β_1=0.9 to update the model parameters.
	In the first training step, we use a fixed learning rate of 10^-5 to train the autoencoder.
	For the second step, we follow the same settings used in DCGAN <cit.> and set the learning rate to 0.0002. At the final step, we adjust the learning rate to 10^-7 for end-to-end fine-tuning. During training, we set the minibatch size to 16. For the three steps in Algorithm <ref>, we train the corresponding models with 100 epochs, 200 epochs and 100 epochs, respectively.
	
	

§ EXPERIMENTS

	In the experiments, we focus on generating high resolution 
	images of 512 × 512.
	To evaluate the performance of the proposed method, several generative models are adopted for comparison, including DCGAN <cit.>,
	WGAN-GP <cit.>,
	AGE <cit.>, 
	StackGAN <cit.>,
	StackGAN-v2 <cit.>
	and Progressive GAN <cit.>.

	
	For the comparison methods, we use the official source codes and the original settings specified in their papers. Specifically, we train DCGAN, WGAN-GP, AGE and StackGAN-v2 for 200, 3906, 150 and 600 epochs, respectively, following the default settings in the original papers.
		We train StackGAN for 220 epochs, i.e., 120 for Stage-I and 100 for Stage-II. For Progressive GAN, we start with 4 × 4 resolution and train the model at each resolution using 1200k images, i.e., 600k for fade-in and 600k for stabilizing.
	
	For convenience, we organize the experiments as follows. Firstly, we introduce some details about the benchmark datasets and the evaluation metrics in Section <ref>. Then, we compare the performance of our method with several baselines on five benchmark datasets and show both quantitative and qualitative comparisons in Section <ref> and <ref>. 
	
	
	
	
	
	

 §.§ Datasets and Evaluation Metrics

	We evaluate the proposed method using a wide variety of benchmark datasets, including Oxford-102 Flowers <cit.>, Caltech-UCSD Birds (CUB) <cit.>, High-Quality Large-scale CelebFaces Attributes (CelebA-HQ) <cit.>, Large-scale Scene Understanding (LSUN) <cit.> and ImageNet <cit.>.
	Oxford-102 Flowers contains 8189 images of flowers from 102 fine-grained classes and CUB contains 200 bird species with 11,788 high resolution images in total. 
	The images in Oxford-102 Flowers have resolutions of over 500 × 500 while CUB contains images with resolution ranging from 300 to 500.
	The CelebA-HQ dataset,which is generated from the original CelebA dataset <cit.>, contains 30K celebrity face images at 1024 ×1024 resolution.  
	In our experiments, we resize the training samples to 512 × 512 resolution.
	LSUN contains approximately one million images at the resolutions ranging from 300 to 500. 
	ImageNet contains 1,000 classes and 1.28 million images with the resolutions  ranging from 200 to 700 (the average image resolution is 469×387).  Due to the setting of unconditional image generation, we train all the methods on each category separately.
	For each dataset, we resize all the images to 512 × 512 resolution during training.
	
	For quantitative evaluation, we use Frechet Inception Distance (FID) <cit.>, Inception Score <cit.> and MS-SSIM <cit.> to evaluate the generated samples.
	FID is a widely used metric for implicit generative models, as it correlates well with the visual quality of generated samples.
	Inception Score can be used to measure both image quality and image diversity over a large number of samples. In general, larger Inception Score value indicates the better performance. MS-SSIM measures the diversity of the generated samples and the resulting values range from 0.0 to 1.0. Higher MS-SSIM values correspond to perceptually more similar images. 
	In the experiments, we use 10,000 images to calculate these scores. For Inception Score, 10 splits are used to compute the standard deviations.
	
	
	
	
	
	

 §.§ Quantitative Comparisons

	
	
	
	
	
	
	
		In this section, we compare the performance of different methods on 5 benchmark datasets in terms of FID <cit.>, Inception Score <cit.> and MS-SSIM <cit.>.
		The results are shown in Table <ref>.
	
	
	
		On the Oxford-102 and CUB datasets, 
		the proposed AEGAN model obtains slightly worse results than Progressive GAN <cit.> in terms of MS-SSIM but achieves the best FID and Inception Score.
		On the CelebA-HQ dataset, AEGAN still obtains comparable performance with the strong baseline Progressive GAN. 
		These results demonstrate that the proposed method is able to produce images of promising quality while maintaining a large diversity. 
	
	
	For the more challenging datasets LSUN and ImageNet, we train AEGAN and the considered baseline methods on each category separately. On LSUN, we compare the results in 3 categories, including Bedroom, Classroom and Conference. However, on ImageNet (containing 1000 categories in total), training 1000 models is infeasible and impractical. For convenience, we choose two of them, i.e., Promontory and Volcano, to evaluate the performance of our proposed method. From Table <ref>, AEGAN outperforms the considered baseline methods in most categories in terms of FID, Inception score and MS-SSIM.
		These results demonstrate the superiority of the proposed method in high resolution image synthesis.
	
	
	

 §.§ Qualitative Comparisons

	
		In this section, we compare the visual quality of the images generated by the proposed AEGAN and several baseline methods on 5 benchmark datasets, including Oxford-102, CUB, CelebA-HQ, LSUN and ImageNet.
	
	
	
		Figs. <ref> and <ref> show that, when producing high resolution images of 512 × 512, DCGAN tends to produce colorful but meaningless images that contain many isolated regions. 
		WGAN-GP <cit.> and AGE <cit.> are able to capture the rough structure and produce meaningful images. However, the generated samples appear very blurred and lack photo-realistic details. 
		Note that StackGAN is originally devised to generate images by taking the input text as a condition. 
		In this experiment, since there is no additional information acting as the condition,
		the transition of StackGAN from the first stage to the second one will be incoherent. As a result, the model fails to effectively capture the style and texture information.
		Compared to Progressive GAN <cit.>, AEGAN is able to produce promising images with sharper object structure and richer details than the other methods.
	
	
	
		On CelebA-HQ, we also compare some detailed regions of the generated images in Fig. <ref>. Specifically, we compare the detailed structure and texture in the generated face images, such as eyes, nose, and mouth. 
		The top row shows 
			the images generated by the five considered baselines and their detailed regions,
			and the bottom row shows a set of samples generated by AEGAN.
		Compared to the considered baselines, 
		AEGAN is able to produce promising images with sharper facial structure and finer details.
	
	
	
		When generating more complex images on LSUN and ImageNet, AEGAN is able to produce meaningful images of better perceptual quality compared to the baseline methods. Figs. <ref> and <ref> show that none of the baseline methods except Progressive GAN, is able to produce visually acceptable scene images. 
		The images generated by Progressive GAN also lack
		some structural and textural information. However, most of the images generated by AEGAN capture the salient features of the specific category and 
		obtain objects with complete and sharp structure,
		e.g. the structure of beds and the shape of desks. This is attributed to the extracted high-level embedding. The images generated by AEGAN are more photo-realistic than the images generated by the considered baselines in all categories.
		These results demonstrate the effectiveness of the proposed AEGAN in producing high resolution images.
	
	
	
	
	

§ FURTHER EXPERIMENTS

	
	In this section, we conduct further analyses and discussions of our proposed method.
		In Section <ref>, we perform an ablation study of the end-to-end fine-tuning step.
		In section <ref>, we discuss the effect of our proposed denoiser networks.
		In Section <ref>, we investigate the effect of embedding dimension on the performance of AEGAN.
		In Section <ref>, we conduct an experiment of latent space interpolation.
	
	Finally, we compare the training time of our AEGAN to baselines in Section <ref>.
	
	

 §.§ Ablation Study on End-to-end Fine-tuning

	In this section, we investigate the effect of the final fine-tuning step (the third step in Algorithm <ref>) in generating high resolution images by ablation study. The experimental results are shown in Table <ref>.
	From Table <ref>, the model with fine-tuning achieves slightly better performance than the model without fine-tuning.
	
	
		We note that there is a risk of model failure in the first two steps. Fortunately, the end-to-end fine-tuning is able to effectively address this issue, thereby producing 
		a new unified model with better performance. To verify this, we show some examples of a failure case in Fig. <ref>. However, the failure rarely occurs in practice. The end-to-end fine-tuning helps to guarantee the performance of our method.
	
	
	

 §.§ Effect of Denoiser

	
		In this section, we investigate the effect of the denoiser module on the performance of high resolution images synthesis. To this end, we add the denoiser module to all the considered baselines and compare the results obtained using these
		methods with or without the denoiser module. The experimental results are shown in Table <ref>.
		From Table <ref>, the denoiser module is able to improve the performance of different methods in terms of both FID and Inception Score, demonstrating the effectiveness of the proposed denoiser network. 
		Moreover, AEGAN consistently outperforms the baseline methods with or without the denoiser network. 
	
	
	
	
	

 §.§ Effect of Embedding Dimension

	
	In this section, we investigate the effect of embedding dimension on the performance of AEGAN.
	Actually, the dimension of the extracted embedding indicates the representation ability of latent space. 
	Although the representation ability will become more powerful with the increase of dimension, the computational complexity will increase accordingly.
	Therefore, a smaller dimension is preferable when the representation ability of the embedding is sufficient. 
	Here, we test smaller embedding dimensions of 8 × 8 with 64 channels in the autoencoder.
	The results are shown in Fig. <ref>.
	
	Fig. <ref> shows that, when we map the input noise to such a small embedding, the final decoded images contain much more visual artifacts than the standard setting in previous experiments, i.e., 32 × 32 with 64 channels.
	These results demonstrate that to synthesize a high resolution image, we must use a sufficiently large embedding space to represent the information of style and structure.
	
	
	
	

 §.§ Results of Latent Space Interpolation

	In this section, we conduct an experiment in which we investigate the landscape of the latent space. Following the settings in <cit.>, we conduct linear interpolations between two data points in the latent space and feed them into the generative models.
	The generated samples are shown in Fig. <ref>.
	In Fig. <ref>, 
	there are no sharp transitions and the generated images change smoothly.
	These results demonstrate that the proposed AEGAN generalizes well to unseen data
	rather than simply memorizing the training samples. 
	
	

 §.§ Comparison of Training Time

	
		In this section, we compare the training time of the considered methods. We train all the methods except Progressive GAN on single TITAN X Pascal GPU. For Progressive GAN, we train the model on two GPUs with a batch size of 32 for each of GPU (i.e., 64 in total).
		The training times and the corresponding FID scores are shown in Fig. <ref>.
	
	
	
		From Fig. <ref>, the proposed AEGAN achieves the best FID score of 65.41 and only requires 17 hours for training, which is much more efficient than WGAN-GP and Progressive GAN. These results demonstrate the effectiveness of the proposed method in terms of both image generation performance and training efficiency.
	
	
	
	
	

§ CONCLUSION

	In this paper, we have proposed a novel scheme for high resolution image synthesis. In contrast to traditional GAN methods, 
	we use low-dimensional embedding to bridge the distribution gap between input noise and real data.
	Furthermore, we also devise a denoiser network that removes the noisy artifacts and provides low-level photo-realistic details in the generated images. The proposed method produces images with 
	sharp structures and rich photo-realistic details,
	and significantly outperforms the considered generative models.
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	10
		
		arjovsky2017wasserstein
		Martin Arjovsky, Soumith Chintala, and Léon Bottou.
		Wasserstein GAN.
		International Conference on Machine Learning, 2017.
		
		berthelot2017began
		David Berthelot, Tom Schumm, and Luke Metz.
		BEGAN: Boundary Equilibrium Generative Adversarial
		Networks.
		arXiv preprint arXiv:1703.10717, 2017.
		
		chen2018low
		Yudong Chen, Zhihui Lai, WaiKeung Wong, Linlin Shen, and Qinghua Hu.
		Low-rank Linear Embedding for Image Recognition.
		IEEE Transactions on Multimedia, 2018.
		
		denton2015deep
		Emily L Denton, Soumith Chintala, Rob Fergus, et al.
		Deep Generative Image Models using an Laplacian Pyramid
		of Adversarial Networks.
		In Advances in neural information processing systems, pages
		1486–1494, 2015.
		
		dolhansky2018eye
		Brian Dolhansky and Cristian Canton Ferrer.
		Eye In-painting with Exemplar Generative Adversarial Networks.
		In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pages
		7902–7911, 2018.
		
		goodfellow2014generative
		Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
		Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
		Generative Adversarial Nets.
		In Advances in neural information processing systems, pages
		2672–2680, 2014.
		
		gulrajani2017improved
		Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron
		Courville.
		Improved Training of Wasserstein GANs.
		Advances in Neural Information Processing Systems, 2017.
		
		he2016deep
		Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
		Deep Residual Learning for Image Recognition.
		In Proceedings of the IEEE Conference on Computer Vision and
			Pattern Recognition, pages 770–778, 2016.
		
		he2017arbitrary
		Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen.
		Arbitrary Facial Attribute Editing: Only Change What You Want.
		arXiv preprint arXiv:1711.10678, 2017.
		
		isola2016image
		Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.
		Image-to-image Translation with Conditional Adversarial
		Networks.
		Proceedings of the IEEE Conference on Computer Vision and
			Pattern Recognition, 2017.
		
		jetchev2016texture
		Nikolay Jetchev, Urs Bergmann, and Roland Vollgraf.
		Texture Synthesis with Spatial Generative Adversarial
		Networks.
		NIPS Workshop on Adversarial Training, 2016.
		
		karras2017progressive
		Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.
		Progressive Growing of GANs for Improved Quality,
		Stability, and Variation.
		International Conference on Learning Representations, 2018.
		
		kingma2014adam
		Diederik Kingma and Jimmy Ba.
		Adam: A method for stochastic optimization.
		International Conference on Learning Representations, 2015.
		
		larsen2015autoencoding
		Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and
		Ole Winther.
		Autoencoding beyond Pixels using a Learned Similarity
		Metric.
		International Conference on Machine Learning, 2016.
		
		ledig2016photo
		Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew
		Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz,
		Zehan Wang, et al.
		Photo-realistic Single Image Super-resolution using a
		Generative Adversarial Network.
		IEEE Conference on Computer Vision and Pattern Recognition
			(CVPR), 2017.
		
		ledig2017photo
		Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew
		Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz,
		Zehan Wang, et al.
		Photo-realistic Single Image Super-resolution using a Generative
		Adversarial Network.
		In CVPR, volume 2, page 4, 2017.
		
		lee2017photo
		Hui-Jin Lee, Ki-Sang Hong, Henry Kang, and Seungyong Lee.
		Photo Aesthetics analysis via DCNN Feature Encoding.
		IEEE Transactions on Multimedia, 19(8):1921–1932, 2017.
		
		liu2015faceattributes
		Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.
		Deep Learning Face Attributes in the Wild.
		In Proceedings of International Conference on Computer Vision
			(ICCV), 2015.
		
		mao2016image
		Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang.
		Image Restoration using Very Deep Convolutional
		Encoder-decoder Networks with Symmetric Skip Connections.
		In Advances in Neural Information Processing Systems, pages
		2802–2810, 2016.
		
		mathieu2015deep
		Michael Mathieu, Camille Couprie, and Yann LeCun.
		Deep Multi-scale Video Prediction beyond Mean Square
		Error.
		International Conference on Learning Representations, 2016.
		
		ndjiki2011depth
		Patrick Ndjiki-Nya, Martin Koppel, Dimitar Doshkov, Haricharan Lakshman,
		Philipp Merkle, Karsten Muller, and Thomas Wiegand.
		Depth Image-based Rendering with Advanced Texture Synthesis for 3D Video.
		IEEE Transactions on Multimedia, 13(3):453–465, 2011.
		
		nilsback2008automated
		Maria-Elena Nilsback and Andrew Zisserman.
		Automated Flower Classification over a Large Number of
		Classes.
		In Computer Vision, Graphics & Image Processing, 2008.
			ICVGIP'08. Sixth Indian Conference on, pages 722–729. IEEE, 2008.
		
		odena2016conditional
		Augustus Odena, Christopher Olah, and Jonathon Shlens.
		Conditional Image Synthesis with Auxiliary Classifier
		GANs.
		International Conference on Machine Learning, 2017.
		
		radford2015unsupervised
		Alec Radford, Luke Metz, and Soumith Chintala.
		Unsupervised Representation Learning with Deep Convolutional
		Generative Adversarial Networks.
		International Conference on Learning Representations, 2016.
		
		ranzato2014video
		MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan
		Collobert, and Sumit Chopra.
		Video (language) Modeling: a Baseline for Generative Models
		of Natural Videos.
		arXiv preprint arXiv:1412.6604, 2014.
		
		reed2016generative
		Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and
		Honglak Lee.
		Generative Adversarial Text to Image Synthesis.
		In Proceedings of The 33rd International Conference on Machine
			Learning, volume 3, 2016.
		
		rosca2017variational
		Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed.
		Variational Approaches for Auto-Encoding Generative
		Adversarial Networks.
		arXiv preprint arXiv:1706.04987, 2017.
		
		roweis2000nonlinear
		Sam T Roweis and Lawrence K Saul.
		Nonlinear Dimensionality Reduction by Locally Linear Embedding.
		science, 290(5500):2323–2326, 2000.
		
		russakovsky2015imagenet
		Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma,
		Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.
		Imagenet Large Scale Visual Recognition Challenge.
		International Journal of Computer Vision, 115(3):211–252,
		2015.
		
		salakhutdinov2007learning
		Ruslan Salakhutdinov and Geoff Hinton.
		Learning a Nonlinear Embedding by Preserving Class
		Neighbourhood Structure.
		In Artificial Intelligence and Statistics, pages 412–419,
		2007.
		
		shi2016real
		Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken,
		Rob Bishop, Daniel Rueckert, and Zehan Wang.
		Real-time Single Image and Video Super-resolution using an
		Efficient Sub-pixel Convolutional Neural Network.
		In Proceedings of the IEEE Conference on Computer Vision and
			Pattern Recognition, pages 1874–1883, 2016.
		
		shrivastava2016learning
		Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and
		Russ Webb.
		Learning from Simulated and Unsupervised Images through
		Adversarial Training.
		Proceedings of the IEEE Conference on Computer Vision and
			Pattern Recognition, 2017.
		
		sun2016hems
		Yipeng Sun, Xiaoming Tao, Yang Li, Linhao Dong, and Jianhua Lu.
		Hems: Hierarchical Exemplar-based Matching-synthesis for Object-aware Image Reconstruction.
		IEEE Transactions on Multimedia, 18(2):171–181, 2016.
		Tao18attngan
			Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, Xiaodong He, Tao Xu,
			and Pengchuan Zhang.
			Attngan: Fine-grained Text to Image Generation with Attentional Generative Adversarial Networks.
			2018.
		brock2018large
			Andrew Brock, Jeff Donahue, and Karen Simonyan.
			Large Scale GAN Training for High Fidelity Natural Image Synthesis.
			arXiv preprint arXiv:1809.11096, 2018.
		chen2018self
			Ting Chen, Mario Lucic, Neil Houlsby, and Sylvain Gelly.
			On self Modulation for Generative Adversarial Networks.
			arXiv preprint arXiv:1810.01365, 2018.
		heusel2017gans
			Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp
			Hochreiter.
			Gans Trained by a Two Time-scale Update Rule Converge to a Local Nash Equilibrium.
			In Advances in Neural Information Processing Systems, pages
			6626–6637, 2017.
		nguyen2016plug
			Anh Nguyen, Jason Yosinski, Yoshua Bengio, Alexey Dosovitskiy, and Jeff Clune.
			Plug & Play Generative Networks: Conditional Iterative
			Generation of Images in Latent Space.
			Proceedings of the IEEE Conference on Computer Vision and
				Pattern Recognition, 2017.
		zhang2018self
			Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.
			Self-attention Generative Adversarial Networks.
			arXiv preprint arXiv:1805.08318, 2018.
		Han17stackgan2
			Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang,
			and Dimitris Metaxas.
			Stackgan++: Realistic Image Synthesis with Stacked Generative
			Adversarial Networks.
			arXiv: 1710.10916, 2017.
		
		tenenbaum2000global
		Joshua B Tenenbaum, Vin De Silva, and John C Langford.
		A Global Geometric Framework for Nonlinear Dimensionality Reduction.
		science, 290(5500):2319–2323, 2000.
		
		ulyanov2016texture
		Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor S Lempitsky.
		Texture Networks: Feed-forward Synthesis of Textures and
		Stylized Images.
		In International Conference on Machine Learning, pages
		1349–1357, 2016.
		
		ulyanov2017adversarial
		Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.
		Adversarial Generator-Encoder Networks.
		Association for the Advance of Artificial Intelligence, 2018.
		
		cao2018adversarial
		Cao, Jiezhang and Guo, Yong and Wu, Qingyao and Shen, Chunhua and Tan, Mingkui.
		Adversarial Learning with Local Coordinate Coding.
		International Conference on Machine Learning, 2018.	
		
		van2009learning
		Laurens Van Der Maaten.
		Learning a Parametric Embedding by Preserving Local
		Structure.
		RBM, 500(500):26, 2009.
		
		wah2011caltech
		Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge
		Belongie.
		The caltech-UCSD Birds-200-2011 Dataset.
		2011.
		
		wang2018esrgan
		Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change
		Loy, Yu Qiao, and Xiaoou Tang.
		ESRGAN: Enhanced Super-resolution Generative Adversarial Networks.
		arXiv preprint arXiv:1809.00219, 2018.
		
		wang2003multiscale
		Zhou Wang, Eero P Simoncelli, and Alan C Bovik.
		Multiscale Structural Similarity for Image Quality Assessment.
		In Signals, Systems and Computers, 2004. Conference Record of
			the Thirty-Seventh Asilomar Conference on, volume 2, pages 1398–1402. Ieee,
		2003.
		
		warde2016improving
		David Warde-Farley and Yoshua Bengio.
		Improving Generative Adversarial Networks with Denoising
		Feature Matching.
		International Conference on Learning Representations, 2017.
		
		yang2017high
		Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li.
		High-resolution Image Inpainting using Multi-scale Neural Patch Synthesis.
		In The IEEE Conference on Computer Vision and Pattern
			Recognition (CVPR), volume 1, page 3, 2017.
		
		yang2013self
		Min-Chun Yang and Yu-Chiang Frank Wang.
		A Self-learning Approach to Single Image Super-resolution.
		IEEE Transactions on multimedia, 15(3):498–508, 2013.
		
		yi2017dualgan
		Zili Yi, Hao Zhang, Ping Tan Gong, et al.
		DualGAN: Unsupervised Dual Learning for Image-to-image Translation.
		International Conference on Computer Vision, 2017.
		
		yin2008extracting
		Wotao Yin and Yin Zhang.
		Extracting Salient Features from Less Data via
		L1-minimization.
		SIAG/OPT Views-and-News, 19(1):11–19, 2008.
		
		yu15lsun
		Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao.
		Lsun: Construction of a Large-scale Image Dataset using Deep Learning
		with humans in the loop.
		arXiv preprint arXiv:1506.03365, 2015.
		
		yu2018generative
		Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.
		Generative Image Inpainting with Contextual Attention.
		arXiv preprint, 2018.
		
		yu2018super
		Xin Yu, Basura Fernando, Richard Hartley, and Fatih Porikli.
		Super-resolving Very Low-resolution Face Images with Supplementary Attributes.
		In Proceedings of the IEEE Conference on Computer Vision and
			Pattern Recognition, pages 908–917, 2018.
		
		zhang2016stackgan
		Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Xiaogang Wang,
		and Dimitris Metaxas.
		StackGAN: Text to Photo-realistic Image Synthesis with
		Stacked Generative Adversarial Networks.
		International Conference on Computer Vision, 2017.
		
		zhao2016energy
		Junbo Zhao, Michael Mathieu, and Yann LeCun.
		Energy-based Generative Adversarial Network.
		International Conference on Learning Representations, 2017.
		
		zhu2017unpaired
		Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.
		Unpaired Image-to-Image Translation using
		Cycle-Consistent Adversarial Networks.
		International Conference on Computer Vision, 2017.
		
		ioffe2015batch
		Ioffe, Sergey and Szegedy, Christian.
		Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.
		International Conference on Machine Learning, 2015.
		
		nair2010rectified
		Nair, Vinod and Hinton, Geoffrey E.
		Rectified Linear Units Improve Restricted Boltzmann Machines.
		International Conference on Machine Learning, 2010.
		
		maas2013rectifier
		Maas, Andrew L and Hannun, Awni Y and Ng, Andrew Y.
		Rectifier Nonlinearities Improve Neural Network Acoustic Models.
		International Conference on Machine Learning, 2013.	
		
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
