Romantic-Computing

Honors Thesis

by

Elizabeth Horishny
Faculty Advisor: Dr. Simona Doboli

April 2022

##  1 Abstract

In this paper we compare various text generation models’ ability to write
poetry in the style of early English Romanticism. These models include:
Character-Level Recurrent Neural Networks with Long Short-Term Memory, Hugging
Face’s GPT-2, OpenAI’s GPT-3, and EleutherAI’s GPT-NEO. Quality was measured
based syllable count and coherence with the automatic evaluation metric GRUEN.
Character-Level Recurrent Neural Networks performed far worse compared to
transformer models. And, as parameter-size increased, the quality of
transformer models’ poems improved. These models are typically not compared in
a creative context, and we are happy to contribute.

##  2 Introduction

In this report, we will briefly introduce language models, and introduce some
machine learning concepts. This will be followed by related work in the field.
Consequently, we will discuss our data set in detail. Then, we will explain
our implementations across each model. Afterwards, we will showcase poetry and
make generalizations about how each model performed. Finally, we will state
the conclusions made based on an analysis of our results. An appendix with
additional poems, charts, and tables will be included at the end.

###  2.1 Background

Recurrent neural networks and transformer models are compared in the domains
of translation, question-answering, and reading comprehension.[1] However,
less work has been done analyzing the creative, or artistic, abilities of text
generation models. This report compares several neural network implementations
of language models. Namely, character-level recurrent neural networks with
Long-Short Term Memory and pre-trained Transformer models.
By definition, a language model is a statistical probability distribution over
a set of words. Given a sequence of tokens $w$ of length $m$, a language model
assigns a probability $P(w_{1},...,w_{m})$ to each token. At time-step $i$, a
token is chosen based on the language model’s probability distribution
$\prod_{i=1}^{m-1}P(w_{i}|w_{1},...,w_{i-1})$. In contrast with conventional
neural networks, recurrent neural networks (RNN) contain cycles within the
network which allow reference of previous outputs. In other words, RNN allow
neural networks to remember. Unfortunately, on its own, a RNN’s memory, or
context, becomes over-saturated with information. Over time, output recycled
back into the hidden layer will either explode or become hidden. This problem
is commonly referred to as the ’vanishing gradient’ problem.
Long short-term memory, or LSTM, is one of many ’attention-mechanisms’ built
to prevent this problem. During each time-step, a hidden ‘short-term’ state is
composed, retaining information which could have been lost. Essentially, at
each time-step or token (think each word in a sentence) an LSTM unit will
compute what should be remembered, and consequently, what should be forgotten.
This output is sent back as input, alongside the subsequent input token,
during the next time-step.
Unfortunately, recurrent neural network architectures are still privy to the
vanishing gradient problem. Especially when tasked with long texts.
Furthermore, because recurrent neural networks process information token-by-
token, they cannot process text concurrently. This makes RNNs extremely slow
to train. On the other hand, transformers, an alternative approach to
generating text, analyze all tokens simultaneously. More accurately, each
token is processed through an alignment score which determines what
information is relevant in a given moment. Alignment scores are determined in
a few primary ways. The most popular methods are additive or concat[2],
location-based, general and dot-product[3].

Figure 1: Attention-mechanism visualization. Transformers differentiate the
pronoun ’it’ based on context.

The era of Early English Romanticism was chosen because of it’s multitude of
quantifiable characteristics. Firstly, most poems follow a quatrain rhyme
scheme. In other words, every four lines end in one of the following formats:
AABB, ABAB, or ABBA. Additionally, rhyming line pairs will match in meter. In
other words, when two lines share meter when the lines have the same number of
syllables. Meter also encompasses the arrangement of stressed and unstressed
syllables. But, this report only considers syllable count. Furthermore, many
reoccurring themes are expressed within these poems. For instance, themes of
nature, love, and night-time are abundant. With similar themes, we expect a
specialized vocabulary capable of encapsulating Romanticism-era imagery.
Thereinby, poem-quality across all models will be based on general coherence
in addition to conforming to this poetic structure.

##  3 Related Work

There have been many creative approaches to generating poetry using recurrent
neural networks.
Deep-speare is a word-level recurrent neural network composed of 3 models, all
of which have a unique purpose: a Language Model, a Pentameter Model, and a
Rhyme Model. The Language Model builds on a LSTM encoder-decoder Model. Here,
each word is encoded with orthographic information. The Pentameter Model is of
an encoder-decoder architecture with attention as well. Using output from the
Lanugage Model, the Pentameter Model suggests 10 candidate lines following
accurate iambic pentameter. Finally, the Rhyme Model is trained to distinguish
rhyming word pairs. In short, researchers found that their network was very
good at grasping meter and rhyme. The surveyed layman population could not
distinguish human written poems from generated ones. However, experts easily
distinguished human poems from nonhuman ones. The network’s focus on form and
rhyme hurt its readability and emotion. To quote the paper, ”real poets will
break rules of form to create other effects.”[4]
In Generate and Revise[5], researchers implemented a Reinforcement Learning
approach to text generation. The intention of Generate and Revise is to
generate poetry in such a way that mimics a human’s revision process. Like
Deep-speare, this system is composed of three neural network models- the
Generator, Detector, and Prompter. Firstly, the Generator Model will compose a
quatrain. Then, the Detector Model will seek out words that should be revised
in order to fit the encoded rhyme scheme. Finally, the Prompter Model will
generator a replacement word. The poem is then cycled between the Detector and
Prompter Models. Results largely satisfied the intended quatrain rhyme scheme.
In terms of coherence, results were measured against automatic text evaluation
metrics BLEU and ROUGE.
The Recurrent Neural Network Poem Generator (RNNPG) approach was designed by
Zhang et al.[6] Here, poems were designed to fit a Chinese 5-character
quatrain style. Each line is meant to follow different guidelines in terms of
meter. Therefore, each token was encoded with its line number. Finally, poems
were analyzed by humans in addition to automatic evaluator BLEU. Human ratings
judged fluency, coherence, meaning, poetic-ness, and rank. Overall, human
poems always scored slightly better than RNNPG poems in BLEU and human
ratings.

##  4 Data Description

Lord Byron and Percy Bysshe Shelley are noted as preeminent poets in early
English Romanticism. For our data set, we used the entire bibliography of
poets Percy Bysshe Shelley and Lord Byron, as listed by poetry catalog website
allpoetry.com.[7] Using python package Beautiful Soup[8], all poems were
scraped from allpoetry.com and into a single text file. In total, 628 poems
were scraped with a fairly even split with 334 belonging to Percy Bysshe
Shelley and 294 belonging to Lord Byron. In total, this text file was
comprised of 80,409 lines of poetry, 338,387 words, 46,926 unique words, and
5,356,670 bytes. All models were either fully-trained or fine-tuned on this
data set. Technically, our character-level recurrent neural network models
were the only properly trained models. Most other models belonged to pre-
trained transformer models and were only fine-tuned, or not tuned at all.

##  5 Methods

###  5.1 Character-level Recurrent Neural Network with LSTM

The first models we trained were a character-level recurrent neural network
with long short-term memory layers. We chose to generate at the character-
level instead of the word-level in order to compromise with the relatively
small data set. The model was built in Python using PyTorch nn. Across all
models, the learning rate was 0.02, the chunk length was 200, and the batch
size was 100. Otherwise hidden-layer size varied between 100-200 neurons, and
2 or 3 LSTM units were included. Finally, training was determined based on
perceived improvement during the training process. This varied between
6,000-24,000 epochs. Code was largely taken from https://github.com/spro/char-
rnn.pytorch[9].

###  5.2 GPT-2

The first Transformer model GPT-2, or Generative Pre-trained Transformer was
released in 2011 by OpenAI. OpenAI’s GPT-2 Model is an evolved version of
their previous GPT model- primarily due to its increase in parameters (117
million to 1.5 billion). GPT-2 was also trained on a much larger data set,
utilizing a subset of Common Crawl- a web scraping archive.[10] Although a
massive innovation, fine-tuning still required the acquisition of data sets.
And, the use of a training set comes with the implications of faulty
generations.
GPT-2 Experiments were conducted using Hugging Face’s transformers library
which uses a byte-lever version of Byte Pair Encoding and a vocabulary size of
50,257. The model weights were initialized from Hugging Face’s GPT-2
weights[1]. The batch-size and sequence-length was 64. Code was primarily
based off of a Kaggle tutorial by user Michaelarman[11].
For our first model, we used the Hugging Face transformer library with
PyTorch’s DataLoader Class, for forming training and testing data sets as well
as batching data. Fine-tuning was conducted using Hugging Face libraries for
10 epochs. After 10 epochs, the model began to over-fit (validation score
decreased while perplexity increased).
Another GPT-2 model used the Hugging Face transformer library in tandem with
Fast.ai’s DataLoader and Loader classes. Fast.ai’s DataLoader structures data,
like PyTorch’s DataLoader. But, it also automatically feeds data into the
model as batches, and creates validation sets for training and testing.
Furthermore, Fast.ai’s Learner class was used for fine-tuning. The Learner
class is supplied with Fast.ai’s custom loss functions. CrossEntropyLossFlat()
is the loss function we used to train for a total of 48 epochs. After 48
epochs, perplexity stopped significantly decreasing.
To recapitulate, both models used Hugging Face’s GPT-2, but were fine-tuned
using different methods.

###  5.3 GPT-3

OpenAI’s GPT-3 uses the same foundation as GPT-2, but the number of parameters
is increased from 1 billion to 175 billion. Essentially, GPT-3 is GPT-2, but
scaled up by 10. See Table 1 to see how they compare.
GPT-3’s training data primarily consisted of a larger section of the Common
Crawl data set, which amounts to a trillion words. With this enormous dataset,
GPT-3 completely eliminates the need for fine-tuning. As a matter of fact,
GPT-3 gave rise to zero-shot learners. Whereas training or fine-tuning primes
a model with examples, GPT-3 only needs a prompt. See Figure 2 for more
examples of zero-shot prompts. And, although GPT-3 works with only a prompt,
GPT-3 has been recorded to answer with higher accuracy given a few examples
(few-shot learning).[12]

Figure 2: Examples of zero-shot, one-shot and few-shot, contrasted with
traditional fine-tuning.[12]

GPT-3 has yet to be fully released, but poems can be generated after applying
for an API-key from https://beta.openai.com/. This API-key was downloaded and
used.

###  5.4 GPT-NEO

GPT-3 code has yet to be fully released to the public. In an attempt to supply
an open-source few-shot learner, EluetherAI released Generative Pre-Trained
Transformer NEO. It scales larger than GPT-2 in terms of parameter-size and
context-size, but smaller than GPT-3.[13] Code was based off of an aitextgen
tutorial.[14]

| GPT-2 | GPT-NEO | GPT-3
---|---|---|---
Vocabulary | 50,257 | 50,257 | 50,257
Layers | 48 | n/a | 96
Batch-size | 512 | n/a | 3.2M
Context-size | 1024 | 2048 | 2048
Parameters | 1.5B | 2.7B | 175B
Table 1: Comparison of Transformer models.

###  5.5 GRUEN

GRUEN, which loosely stands for Grammaticality, non- Redundancy, focUs,
structure and coherENce, is an automatic evaluation metric for generated text.
It measures grammatical correctness, the degree of focus across all sentences,
whether each sentence flows from one to the next, and a text’s redundancy or
repetitiveness. GRUEN outperforms BLEU and ROUGE in terms of accessing
accuracy.[15] We will be using GRUEN to check the legibility of our generated
texts.

###  5.6 Meter

In order to quantify poetic meter, we wrote a Python script to count syllables
in each line. Then, we calculated the standard deviation to see how close the
syllables carried across a poem per line. In Byron and Shelley’s poems,
rhyming line pairs matched in syllable-length as well as rhyme. So, if a poem
had the same number of syllables across all lines, the standard deviation
would be zero, implying accurate use of meter. But, arose the reoccurring
issues of ’interrupted’ and ’feeble beginnings’ generations. Interrupted
generations refer to poems which could have finished a line complying to the
meter scheme, but were cut short by the max-length generation parameter. This
case was determined when the final line was 4 or more syllables lacking from
the previous line. In an interrupted generation, the last line was not
counted. Furthermore, a feeble beginning occurred when the prompt was
interpreted as the ending of a previous quatrain, rather than the beginning of
a new one. In this case, if the first line was 4 or more syllables less than
the previous line, it was discarded from the meter metric.

###  5.7 Generation parameters

For all models, generations were kept relatively short at a max-length of 300
characters or 75 words (length token differed per model). The intention was to
produce a little more than a quatrain’s worth of text. With at-most 300
characters, or 75 words, our results quantify how models compare in short-term
generations. For Character-level RNN, GPT-2, and GPT-NEO generations, we
settled on a temperature of 0.7 with the intention of producing creative, but
not gibberish, text. For prompts, we used the starter sequences: ’Roses are
red, ’ and ’Dearest audience, ’. The last two characters, a space and a comma,
were essential to guarantee Char-RNN models wouldn’t add additional characters
to the prompt, resulting in a non-english word.
GPT-3 was treated differently during generation because, rather than starter
sequences, GPT-3 is a few-shot and zero-shot learner (Figure 2. A generation
parameter we could have tweaked was generation. Temperature controls how
’random’ or expected a generation will become. The closer to 0 the more
’conservative’ a generation will be. For a wider breadth of results, we used
temperatures ranging from 0.7 to 1.5. Since GPT-3 accepts Zero-shot prompts
rather than a beginner sequence, we queried the model with prompts such as:
Write a poem in the style of Percy Bysshe Shelley. Start with ’Dearest
audience, ’, Write a poem in the style of English romantic movement, Start
with: ’Roses are red’. All other parameters were default as supplied by
OpenAI.

Figure 3: Distribution of Prompts for the experiment

##  6 Results

###  6.1 Character-Level Recurrent Neural Network

Overall, there was no noticeable difference between RNN models, regardless of
the number of LSTM layers, or the size of hidden layers. Therefore, all
character-level RNN results will be considered analogously. Overall,
generations mostly conformed to an English vocabulary. However, generations
often lacked in semantic meaning. Largely, generations used flowery language
to illicit an enigmatic, vaguely romantic, theme. Additionally, hints of a
rhyme scheme would peek out in the occasional poem (in Figure 4\- die/high,
and perhaps wind if pronounced wīnd), but rhymes appeared more accidental than
premeditated. See Figure 12 to see how all RNN models differed with regards to
Meter and GRUEN.

Dearest audience, mad shadows,
Who lose like that gentle heart, and will die
That upon the burning vapoured in high,
Wrought out of his sea and are his und with power,
And in the sea of sunless land to wear
The driver, at last displayed the winds.
The rapid that oneselv

Figure 4: Char-RNN. 2 LSTM layers, hidden size 100.

###  6.2 GPT-2

Originally, we set out to compare two different versions of Hugging Face’s
GPT-2 model, with minute differences in fine tuning. However, after collecting
results, there was not a clear difference in quality between the pure Hugging
Face model and the Hugging Face + Fast.ai hybrid. Furthermore, the jump in
text quality is immense compared to the Character-Level Recurrent Neural
Network. Average GRUEN went up by 200%, and standard meter went down about
35%. Again, rhyme schemes were inconsistently followed. But, instances of
rhyming did occur (see Figure 8). Refer to Table 3 to see how the two models
compare between meter and GRUEN metrics.

Roses are red,
When the mountains are blue;
When the stars are purple,
And when the clouds are quivering,
And when the wind is wild;
We feel that our bodies are a thing
Immensely fixed on sensation,
And that, whatever we feel,
Might be already.

Figure 5: GPT-2 Pure Hugging Face model under prompt: ’Roses are Red, ’.

###  6.3 GPT-NEO

GPT-NEO’s generations were majorly improved in comparison to the GPT-2 models.
Interestingly, this model performed the best on GRUEN. Meter was also
marginally better than GPT-2. The most impressive quality of GPT-NEO, is
within its long-term poems with a consistent theme from beginning to end.
Although, this wasn’t fully explored do to this experiments focus on short
generations. See Figure 9 for an example of a longer GPT-NEO poem.

Dearest audience!
She was a child of Nature;
Her very bloom was of the air,
And her very sparkles of the eye;
And she was the first of those,
Who saw the sun rise

Figure 6: GPT-NEO under prompt ’Dearest audience, ’

###  6.4 GPT-3

Finally, GPT-3 was the most impressive model. Its ability to understand zero-
shot prompts can outperform a person tasked with the same problem. GPT-3 had
the best understanding of meter, doing twice as good as Char-RNN. The only
downside of GPT-3 are the bland, or conservative, generations. The choice in
increasing temperature for this model was chosen in order to deter this, but
”boring” generations still arose. See Figures 10 and 11) for a few examples of
GPT-3’s worst poems.
Additionally, with regard to performing worse than GPT-NEO on GRUEN: Although
GPT-NEO had a higher score, from human analysis, poems generated by GPT-NEO
were no more legible than those generated by GPT-3. This is perhaps because
GPT-3 followed a popular ”Roses are red” poem structure that is recognizable
by most people, but technically grammatically incorrect.

Dearest audience,
I welcome you all
To my dark and dreary play
Which I have written
In hopes that it will
Bring you some small measure
Of the joy that I have found
In its crafting.
Please do not be alarmed
By the blood and gore
Which stains

Figure 7: GPT-3 under prompt: Write a poem in the style of Percy Bysshe Shelley. Start with: ’Dearest audience, ’ | CHAR-RNN | GPT-2 | GPT-NEO | GPT-3
---|---|---|---|---
Mean GRUEN | 0.18 | 0.36 | .48 | .39
Median GRUEN | 0.18 | 0.39 | .53 | .39
Mean Meter $\sigma$ | 3.06 | 2.39 | 2.18 | 1.23
Median Meter $\sigma$ | 2.85 | 2.08 | 1.76 | 1.15
Table 2: Averages of GRUEN and Meter across all models.

##  7 Conclusions

Across all models, some semblance of poetic form was learned. Only GPT-2 would
occasionally generate text formatted in a block of text (no new lines).
Furthermore, all models picked up on the importance of matching lines
syllabic-ally. This was less apparent with the character-level recurrent
neural network models, however, more research is needed. Character-level RNN’s
were trained on far less data than any of the transformer models, since the
transformer models were pre-trained. Perhaps if more training data was added,
character-level RNN’s would improve better in terms of meter. Same goes for
GRUEN; character-level RNN’s performed far poorer in GRUEN than any
transformer models, but may have scaled better if their training vocabulary
was increased.
Additionally, very few training parameters were tweaked. And, although altered
parameters did not result in a noticeable change, the breadth of combinations
of a recurrent neural network’s architecture is immense. The same thought
holds for generation parameters. It’s possible that a different number of
layers, length of training, and temperature could have resulted in entirely
different GRUEN and meter analytics.
To recapitulate, all models learned some poetic form from the Lord Byron and
Percy Bysshe Shelley data set. The importance of meter, or syllable-count, was
also encapsulated by all models. There was no significant presence of rhyme
within the results, but all models showed a capability of making two line
pairs rhyme. In terms of coherence, character-level recurrent neural networks
performed the worst, with poems which rarely showcased meaning. On the other
hand, all transformer models were capable of dictating a poem with consistent
meaning from beginning to end. As parameter size increased, GRUEN and
perceived meaning did as well.

##  Appendix A Appendix with Additional Poems, Tables, and Charts

Dearest Audience, to my longing sight
Is more than thou canst be with me -
Farewell! and though the dream now hath wing’d me,
I’ll not sleep–and cannot leave to thee.
My life is Lara–when

Figure 8: GPT-2 Hugging Face and FastAI hybrid model under prompt ’Dearest
Audience, ’.

I think, in the end, it is better to die,
And live on the sea, and be what you are–
But die, and be what you wish to be.
It is a sad reality–but true–
That death is what makes us human,
And that we are the consequence.
But in the end, life is better than death–
And that the consequence is life’s loss.
I do not pretend to say what’s best,
But merely that the world is hell.
I say that life is, and ever was,
The last of human perfection.
But I would not pretend to say it–
To soothe my penitents’ fears,
That life could not be happiness;

Figure 9: GPT-NEO under no prompt.

Dearest audience,
I love you all so much,
I wish I could take you with me
On all my journeys.
You are the stars that light up my sky,
The sun that warms my soul,
The moon that guides my way.
You are everything to me,

Figure 10: GPT-3 under prompt: Write a poem in the style of the English
romantic movement. prompt: ’Dearest audience, ’

Roses are red,
Violets are blue,
I love you,
And I always will too.

Figure 11: GPT-3 under prompt: Write a poem in the style of Percy Bysshe
Shelley. Start with: ’Roses are red, ’

Figure 12: Meter and GRUEN metrics across all RNN models. Little differentiation found between models. | GPT-2 | GPT-2 HF | GPT-2 HF+FAI
---|---|---|---
Mean GRUEN | 0.36 | 0.33 | 0.38
Median GRUEN | 0.39 | 0.38 | 0.4
Mean Meter $sigma$ | 2.39 | 2.4 | 2.37
Median Meter $sigma$ | 2.08 | 2.18 | 2.05
Table 3: Little quantifiable difference between GPT-2 models.

## References

  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * [2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  URL https://arxiv.org/abs/1409.0473.
  * [3] Minh-Thang Luong, Hieu Pham, and Christopher D. Manning.  Effective approaches to attention-based neural machine translation.  URL https://arxiv.org/abs/1508.04025.
  * Lau et al. [2018] Jey Han Lau, Trevor Cohn, Timothy Baldwin, Julian Brooke, and Adam Hammond.  Deep-speare: A joint neural model of poetic language, meter and rhyme.  _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018.  doi: 10.18653/v1/p18-1181.
  * Zugarini et al. [2021] Andrea Zugarini, Luca Pasqualini, Stefano Melacci, and Marco Maggini.  Generate and revise: Reinforcement learning in neural poetry.  _2021 International Joint Conference on Neural Networks (IJCNN)_ , 2021.  doi: 10.1109/ijcnn52387.2021.9533573.
  * Zhang and Lapata [2014] Xingxing Zhang and Mirella Lapata.  Chinese poetry generation with recurrent neural networks.  _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2014.  doi: 10.3115/v1/d14-1074.
  * [7] Welcome to poetry.com.  URL https://www.poetry.com/.
  * [8] Beautiful soup documentation¶.  URL https://beautiful-soup-4.readthedocs.io/en/latest/.
  * [9] Spro.  Spro/char-rnn.pytorch: Pytorch implementation of char-rnn (character-level language model).  URL https://github.com/spro/char-rnn.pytorch.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA_ , pages 6000–6010, 2017.  URL http://papers.nips.cc/paper/7181-attention-is-all-you-need.
  * Michaelarman [2021] Michaelarman.  Poem generation with transformers, Feb 2021.  URL https://www.kaggle.com/code/michaelarman/poem-generation-with-transformers.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.  URL https://arxiv.org/abs/2005.14165.
  * Black et al. [2021] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021.  URL https://doi.org/10.5281/zenodo.5297715.  If you use this software, please cite it using these metadata.
  * [14] Max Woolf.  Colaboratory notebooks¶.  URL https://docs.aitextgen.io/tutorials/colab/.
  * Zhu and Bhat [2020] Wanzheng Zhu and Suma Bhat.  Gruen for evaluating linguistic quality of generated text.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 94–108, 2020.