network. Each sub-layer uses residual connections around each of the sub-layers, followed by layer normalization. The decoder has an additional third sub-layer, which performs multi-head attention over the output of the encoder stack.

Since, attention mechanism is at the core of this generator, the decoder can attend over all positions of input sequence. Computations over a sequence can be parallelized in this case and hence it is faster in performance. The modifications made to the computing units of RNN mentioned in [§\[sec:bg-gen-rnn\]]{} which use parameters specific to control attributes such as style, dialog act etc have not been explored with the transformers architecture.

Pre-trained models
------------------

Recently pre-trained conditional language models are used for text generation like GPT [@radford2018improving], GPT2 [@radford2019language], XLNet [@yang2019xlnet], etc. Several works have fine-tuned the pre-trained models for downstream controllable text generation tasks [@sudhakar2019transforming; @dinan2018wizard; @urbanek2019learning]. The language modeling aspects of generation like fluency and grammaticality are already learnt if pre-trained models are used.

These models are hard to fine-tune for sequence-to-sequence tasks such as machine translation, abstractive summarization etc. BART [@lewis2019bart] is a denoising autoencoder built with a sequence-to-sequence model and is particularly effective when fine tuned for text generation. Alternatively, T5 [@raffel2019exploring] treats every 