The unconditional generation of high fidelity images is a longstanding benchmark
 for testing the performance of image decoders. Autoregressive image models
 have been able to generate small images unconditionally, but the extension of
 these methods to large images where fidelity can be more readily assessed has
 remained an open problem. Among the major challenges are the capacity to encode
 the vast previous context and the sheer difficulty of learning a distribution that
 preserves both global semantic coherence and exactness of detail. To address the
 former challenge, we propose the Subscale Pixel Network (SPN), a conditional
 decoder architecture that generates an image as a sequence of image slices of equal
 size. The SPN compactly captures image-wide spatial dependencies and requires a
 fraction of the memory and the computation. To address the latter challenge, we
 propose to use multidimensional upscaling to grow an image in both size and depth
 via intermediate stages corresponding to distinct SPNs. We evaluate SPNs on the
 unconditional generation of CelebAHQ of size 256 and of ImageNet from size 32
 to 128. We achieve state-of-the-art likelihood results in multiple settings, set up
 new benchmark results in previously unexplored settings and are able to generate
 very high fidelity large scale samples on the basis of both datasets. A successful generative model has two core aspects: it produces targets that have high fidelity and it generalizes well on held-out data. Autoregressive (AR) models trained by conventional maximum likelihood estimation (MLE) have produced superior scores on held-out data across a wide range of domains such as text BID16 BID18 , audio BID13 , images and videos BID4 . These scores are a measure of the models' ability to generalize in that setting. From the perspective of sample fidelity, the outputs generated by AR models have also achieved state-of-the-art fidelity in many of the aforementioned domains with one notable exception. In the domain of unconditional large-scale image generation, AR samples have yet to manifest long-range structure and semantic coherence.One source of difficulties impeding high-fidelity image generation is the multi-faceted relationship between the MLE scores achieved by a model and the model's sample fidelity. On the one hand, MLE is a well-defined measure as improvements in held-out scores generally produce improvements in the visual fidelity of the samples. On the other hand, as opposed to for example adversarial methods BID0 , MLE forces the model to support the entire empirical distribution. This guarantees the model's ability to generalize at the cost of allotting capacity to parts of the distribution that are irrelevant to fidelity. A second source of difficulties arises from the high dimensionality of large images. A 256 × 256 × 3 image has a total of 196,608 positions that need to be architecturally connected in order to learn dependencies among them; the representations at each position require sufficient capacity to express their respective surrounding contexts. These requirements translate to large amounts of memory and computation. Figure 1: A representation of Multidimensional Upscaling. Left: depth upscaling is applied to a generated 3-bit 256 × 256 RGB subimage from CelebAHQ to map it to a full 8-bit 256 × 256 RGB image. Right: size upscaling followed by depth upscaling are applied to a generated 3-bit 32 × 32 RGB subimage from ImageNet to map it to the target resolution of the 8-bit 128 × 128 RGB image. We stress that the rightmost column of both figures are true unconditional samples from our model at full 8bit depth.These difficulties notwithstanding, we aim to learn the full distribution over 8-bit RGB images of size up to 256 × 256 well enough so that the samples have high fidelity. We aim to guide the model to focus first on visually more salient bits of the distribution and later on the visually less salient bits. We identify two visually salient subsets of the distribution: first, the subset determined by sub-images ("slices") of smaller size (e.g. 32 × 32) sub-sampled at all positions from the original image; and secondly, the subset determined by the few (e.g. 3) most significant bits of each RGB channel in the image. We use Multidimensional Upscaling to map from one subset of the distribution to the other one by upscaling images in size or in depth. For example, the generation of a 128 × 128 8-bit RGB image proceeds by first upscaling it in size from a 32 × 32 3-bit RGB image to a 128 × 128 3-bit RGB image; we then upscale the resulting image in depth to the original resolution of the 128 × 128 8-bit RGB image. We thus train three networks: (a) a decoder on the small size, low depth image slices subsampled at every n pixels from the original image with the desired target resolution; (b) a size-upscaling decoder that generates the large size, low depth image conditioned on the small size, low depth image; and (c) a depth-upscaling decoder that generates the large size, high depth image conditioned on the large size, low depth image. Figure 1 illustrates this process.To address the latter difficulties that ensue in the training of decoders (b) and (c), we develop the Subscale Pixel Network (SPN) architecture. The SPN divides an image of size N ×N into sub-images of size N S × N S sliced out at interleaving positions (see FIG1 ), which implicitly also captures a form of size upscaling. The N × N image is generated one slice at a time conditioned on previously generated slices in a way that encodes a rich spatial structure. SPN consists of two networks, a conditioning network that embeds previous slices and a decoder proper that predicts a single target slice given the context embedding. The decoding part of the SPN acts over image slices with the same spatial structure and it can share weights for all of them. The SPN is an independent image decoder with an implicit size upscaling mechanism, but it can also be used as an explicit size upscaling network by initializing the first slice of the SPN input at sampling time with one generated separately during step (a).We extensively evaluate the performance of SPN and the size and depth upscaling methods both quantitatively and from a fidelity perspective on two unconditional image generation benchmarks, CelebAHQ-256 and ImageNet of various sizes up to 256. From a MLE scores perspective, we compare with previous work to obtain state-of-the-art results on CelebAHQ-256, both at full 8-bit resolution and at the reduced 5-bit resolution BID7 , and on ImageNet-64. We also establish MLE baselines for ImageNet-128 and ImageNet-256. From a sample fidelity perspective, we show the strong benefits of multidimensional upscaling as well as the benefits of the SPN. We produce CelebAHQ-256 samples (at full 8-bit resolution) that are of similar visual fidelity to those produced with methods such as GANs that lack however an intrinsic measure of generalization BID10 BID6 . We also produce some of the first successful samples on unconditional ImageNet-128 (also at 8-bit) showing again the striking impact of the SPN and of multidimensional upscaling on sample quality and setting a fidelity baseline for future methods. The problem of whether it is possible to learn the distribution of complex natural images and attain high sample fidelity has been a long-standing one in the tradition of generative models. The SPN and Multidimensional Upscaling model that we introduce accomplishes a large step towards solving this problem, by attaining both state-of-the-art MLE scores on large-scale images from complex domains such as CelebAHQ-256 and ImageNet-128 and by being able to generate high fidelity full 8-bit samples from the resulting learnt distributions without alterations to the sampling process (via e.g. heavy modifications of the temperature of the output distribution). The generated samples show an unprecedented amount of semantic coherence and exactness of details even at the large scale size of full 8-bit 128 × 128 and 256 × 256 images.
We show that autoregressive models can generate high fidelity images.  An architecture utilizing decoder, size-upscaling decoder, and depth-upscaling decoder components to tackle the problem of learning long-range dependencies in images in order to obtain high fidelity images. This paper addresses the problem of generation to high fidelity images, successfully showing convincing Imagenet samples with 128x128 resolution for a likelihood density model.