\begin{table}[!ht]
    \centering
    \begin{tabular}{l|l|c}
    \toprule
    \textbf{Arch} & \textbf{Model} & \textbf{FID $\downarrow$} \\
    \hline
    {\multirow{2}{*}{Diffusion}}
    & Diffusion StyleGAN2~\cite{wang2022diffusiongan} & 3.17 \\
    & Diffusion ProjectedGAN~\cite{wang2022diffusiongan} & 1.85 \\
    \hline
    {\multirow{3}{*}{Convs}}
    & SWAGAN-Bi~\cite{gal2021swagan} & 4.97 \\
    & StyleGAN2~\cite{Karras2020AnalyzingAI} & 3.86 \\
    & Projected GAN~\cite{Sauer2021NEURIPS} & 1.59 \\
    \hline
    {\multirow{4}{*}{Transformer}}
    & TransGAN~\cite{Jiang2021TransGANTP} & 8.94\\
    & Unleashing~\cite{bond2021unleashing} & 4.07\\
    & StyleSwin~\cite{zhang2022styleswin} & 2.95 \\
    & StyleNAT(ours) &  3.40\\
    \bottomrule
    \end{tabular}
    \caption{FID50k results for LSUN-Church Outdoor (256) dataset.}
    \label{tab:church}
\end{table}
\author{
    Steven Walton\textsuperscript{1},
    Ali Hassani\textsuperscript{1},
    Xingqian Xu\textsuperscript{1},
    Zhangyang Wang\textsuperscript{2,3},
    Humphrey Shi\textsuperscript{1,3} \\
{\small \textsuperscript{1}SHI Lab @ U of Oregon \& UIUC, \textsuperscript{2}UT Austin, \textsuperscript{3}Picsart AI Research (PAIR)}\\
{\small \textbf{\url{https://github.com/SHI-Labs/StyleNAT}}}
}
\twocolumn[{
\renewcommand\twocolumn[1][]{#1}
\maketitle

Adversarially trained generative image modeling has been long dominated by Convolutional Neural Networks (CNNs). 
More recently, however, there have been strides in shifting towards new architectures, such as Transformer-based GANs~\cite{zhang2022styleswin,zhao2021improved} and Diffusion-based GANs~\cite{wang2022diffusiongan}.
In addition to this, other generative modeling methods, such as pure Diffusion~\cite{Dhariwal2021DiffusionMB,Song2021DenoisingDI} and Variational Autoencoders~\cite{Child2021VeryDV,hazami2022efficient}, are catching up to the standards set by GANs.
Diffusion and VAEs have some advantages in that they are able to approximate the densities of the training distribution, and thus can incorporate more features and can have a higher recall.
But these methods also have a major disadvantage in that they need to process substantially more information, compared to GANs, and thus can be quite computationally expensive and difficult to train.

\subsection{Attention-based Models}
The Transformer~\cite{vaswani2017attention} is arguably one of the most prevalent architectures in language processing. The architecture simply consists of linear projections and dot product attention, along with normalization layers and skip connections. Other than large language models directly using this architecture~\cite{radford2018improving,devlin2019bert}, the work inspired research into models built with dot product self attention~\cite{parmar2018image,ramachandran2019stand}. Later in 2020, Vision Transformer~\cite{dosovitskiy2020image} applied a plain transformer encoder to image classification, which was outperformed existing CNNs at large scale classification. 
This inspired many researchers to study Transformers as direct competitors to CNNs in different settings~\cite{touvron2020training,touvron2021going,hassani2021escaping} and across different vision tasks~\cite{li2022exploring,he2022masked}.

\subsubsection{Neighborhood Attention Transformer}
Neighborhood Attention (NA)~\cite{hassani2022neighborhood} was proposed as a direct restriction of self attention to local windows.
The key difference between NA and WSA is that the restriction is pixel-wise, leading to each pixel attending to only its nearest-neighboring pixels.
The resulting attention spans would be in theory similar to how convolutions apply weights, with the exception of how cornering pixels are handled.
Compared to WSA, in addition to introducing locality~\cite{liu2021swin}, NA also maintains translational equivariance.
NA also approaches self attention itself as its window size grows, and unlike WSA, would not need pixel shifts as it is a dynamic operation.
Similar operations, in which self attention is restricted in a token-wise manner, had been investigated prior to this work~\cite{ramachandran2019stand}, but were less actively studied due to implementation difficulties~\cite{ramachandran2019stand,vaswani2021scaling,liu2021swin}.
To that end, Neighborhood Attention Extension (NATTEN)~\cite{hassani2022neighborhood} was created as an extension to PyTorch, with efficient CUDA kernels, which allow NA to run even faster than WSA, while using less memory.
The model built with NA, Neighborhood Attention Transformer (NAT)~\cite{hassani2022neighborhood}, was shown to have superior classification performance compared to Swin, and competitive object detection, instance segmentation, and semantic segmentation performance.

\subsubsection{Dilated Neighborhood Attention Transformer}
Dilated Neighborhood Attention Transformer (DiNAT)~\cite{hassani2022dilated} followed NAT~\cite{hassani2022neighborhood} by extending NA to Dilated NA (DiNA), which allows models to use extended receptive fields
and capture more global context, all with no additional computational cost.
DiNA was added to the existing NATTEN package and the underlying CUDA kernels, which allowed the easy utilization of this attention pattern.
By simply stacking NA and DiNA layers in the same manner that Swin stacks WSA and SWSA, DiNAT models significantly outperformed Swin Transformer~\cite{liu2021swin}, as well as NAT~\cite{hassani2022neighborhood} across multiple vision tasks, especially downstream tasks. 
DiNAT also outperformed Swin's convolutional competitor, ConvNeXt~\cite{liu2022convnet}, in object detection, instance segmentation, and semantic segmentation.

\subsubsection{StyleGAN}
Karras et al. first introduced the idea of using progressive generating using convolutions where the one would begin with a small latent representation, process each layer, and then scale up and repeat the process until the desired image size is achieved~\cite{Karras2018ProgressiveGO}.
This architecture has major advantages in that it reduces the size of the latent representation, and thus the complexity of data that the model needs to learn.
This comes with the obvious drawbacks that we make a trade-off of diversity for fidelity and computational simplicity.
Karras et al. later produced several papers that built on this idea~\cite{karras2019a,Karras2020AnalyzingAI,Karras2020TrainingGA,Karras2021AliasFreeGA}, improving the progressive network, but also introducing a sub-network, called the ``style network'', that allows one to control the latent representations at each of these levels.
This innovation dramatically increased the usability of this type of network architecture because it allows for simpler feature interpolation, which is often difficult in implicit density models.
Because of the high fidelity, flexibility, and the ability to perform feature interpolation, this style of network has become the de facto choice for many researchers~\cite{Abdal2021StyleFlowAE,humayun2022magnet,humayun2022polarity,karnewar2020msg}. 
\subsubsection{StyleSwin}
StyleSwin~\cite{zhang2022styleswin} introduced Swin based transformers into the StyleGAN framework, beating other transformer based models, but still not achieving any state of the art results.
This work did show the power of Swin and how localized attention mechanisms are able to out perform many CNN based architectures and did extend the state of the art for style based networks.
StyleSwin achieved this not only by replacing the CNN layers with Swin layers, but also partitioned the transformer heads into two, so that each half could operate with a different kernel.
This was key to improving their generative quality, allowing the framework to increase flexibility and pay attention to distinct parts of the scene at the same time. 
StyleSwin also included a wavelet based discriminator, as was introduced in SWAGAN~\cite{gal2021swagan} and also used by StyleGAN3~\cite{Karras2021AliasFreeGA}, which all show improvements for addressing high-frequency content that causes artifacts.
StyleSwin uses a constant window size of 8 for all latents larger than the initial $4\times4$, wherein they use a window size of 4.
We follow a similar structure except replace the initial layer with MHSA for efficiency. 

To accurately judge the quality and usability of StyleNAT we investigate the performance on FFHQ (\cref{sec:ffhq}) and LSUN Churches (\cref{sec:church}). 
We also perform an ablation study compared to StyleSwin (\cref{sec:ablation}) to demonstrate the effects of the improvements we have made.
These datasets are common evaluation benchmarks that can help us understand how the generator works on more regular data as well as irregular data.
All arguments for experiments are saved alongside the checkpoints for verification. 
For best comparisons we keep hyper-parameters similar to StyleSwin, unless otherwise specified.
This means that we use the Two Time-Scale Update Rule (TTUR)~\cite{Heusel2017GANsTB} training method, have a discriminator learning rate of $2\times 10^{-4}$, use Balanced Consistency Regularization (bCR)~\cite{Zhao2021ImprovedCR}, r1 regularization~\cite{Mescheder2018WhichTM}, and a learning rate decay (specified per dataset).

Our second experiment is with LSUN~\cite{yu2015lsun} Church.
This dataset is uni-modal, as all images are of churches, but is less structural than FFHQ where images have much higher variance from one another.
There is a higher variability in designs as some churches are quite simple and others are complex and have other scene information like people in the foreground/background.
This task demonstrates the ability to perform on diverse and complicated distributions of images.
We found that with just the two partition design that our models would converge to a FID of around 20.
This shows that the simple two partition does not incorporate enough information to build the more complex structures of this distribution.
Introducing the four head partition we found that we could achieve a significant improvement in performance, and this dataset represented the major motivation to expanding the capabilities of our architecture. 
Looking across generative literature it is clear that there is a pattern that models perform better on more structured data or less structured data.
We show our results in~\cref{tab:church}, but note that not all models from~\cref{table:ffhq} also tested on Church.
Computational complexity and throughput will be identical to those of the FFHQ-256 cases as the image sizes are the same.
The number of partitions has almost no affect on throughput, slight due to lack of CUDA optimization, and does not affect MACs.
{"value": "3.17", "type": "Result", "task": "Image Generation", "metric": "FID", "training data/set": "LSUN Church Outdoor", "test data/set": "LSUN Church Outdoor", "experimental settings": {"resolution": "256"}, "model": "Diffusion StyleGAN2", "model settings": {"Arch": "Diffusion"}} {"value": "1.85", "type": "Result", "task": "Image Generation", "metric": "FID", "training data/set": "LSUN Church Outdoor", "test data/set": "LSUN Church Outdoor", "experimental settings": {"resolution": "256"}, "model": "Diffusion ProjectedGAN", "model settings": {"Arch": "Diffusion"}} {"value": "4.97", "type": "Result", "task": "Image Generation", "metric": "FID", "training data/set": "LSUN Church Outdoor", "test data/set": "LSUN Church Outdoor", "experimental settings": {"resolution": "256"}, "model": "SWAGAN-Bi", "model settings": {"Arch": "Convs"}} {"value": "3.86", "type": "Result", "task": "Image Generation", "metric": "FID", "training data/set": "LSUN Church Outdoor", "test data/set": "LSUN Church Outdoor", "experimental settings": {"resolution": "256"}, "model": "StyleGAN2", "model settings": {"Arch": "Convs"}} {"value": "1.59", "type": "Result", "task": "Image Generation", "metric": "FID", "training data/set": "LSUN Church Outdoor", "test data/set": "LSUN Church Outdoor", "experimental settings": {"resolution": "256"}, "model": "Projected GAN", "model settings": {"Arch": "Convs"}} {"value": "8.94", "type": "Result", "task": "Image Generation", "metric": "FID", "training data/set": "LSUN Church Outdoor", "test data/set": "LSUN Church Outdoor", "experimental settings": {"resolution": "256"}, "model": "TransGAN", "model settings": {"Arch": "Transformer"}} {"value": "4.07", "type": "Result", "task": "Image Generation", "metric": "FID", "training data/set": "LSUN Church Outdoor", "test data/set": "LSUN Church Outdoor", "experimental settings": {"resolution": "256"}, "model": "Unleashing", "model settings": {"Arch": "Transformer"}} {"value": "2.95", "type": "Result", "task": "Image Generation", "metric": "FID", "training data/set": "LSUN Church Outdoor", "test data/set": "LSUN Church Outdoor", "experimental settings": {"resolution": "256"}, "model": "StyleSwin", "model settings": {"Arch": "Transformer"}} {"value": "3.40", "type": "Result", "task": "Image Generation", "metric": "FID", "training data/set": "LSUN Church Outdoor", "test data/set": "LSUN Church Outdoor", "experimental settings": {"resolution": "256"}, "model": "StyleNAT", "model settings": {"Arch": "Transformer"}}