# Machine Learning Model Sizes and the Parameter Gap

Pablo Villalobos1, Jaime Sevilla12, Tamay Besiroglu13, Lennart Heim14, Anson
Ho1, Marius Hobbhahn 15

###### Abstract

††1Epoch, 2University of Aberdeen, 3MIT Computer Science & Artificial
Intelligence Laboratory, 4Centre for the Governance of AI, 5University of
Tübingen

We study trends in model size of notable machine learning systems over time
using a curated dataset. From 1950 to 2018, model size in language models
increased steadily by seven orders of magnitude. The trend then accelerated,
with model size increasing by another five orders of magnitude in just 4 years
from 2018 to 2022. Vision models grew at a more constant pace, totaling 7
orders of magnitude of growth between 1950 and 2022.

We also identify that, since 2020, there have been many language models below
20B parameters, many models above 70B parameters, but a scarcity of models in
the 20-70B parameter range. We refer to that scarcity as the _parameter gap_.

We provide some stylized facts about the parameter gap and propose a few
hypotheses to explain it. The explanations we favor are: (a) increasing model
size beyond 20B parameters requires adopting different parallelism techniques,
which makes mid-sized models less cost-effective, (b) GPT-3 was one order of
magnitude larger than previous language models, and researchers afterwards
primarily experimented with bigger models to outperform it. While these
dynamics likely exist, and we believe they play some role in generating the
gap, we don’t have high confidence that there are no other, more important
dynamics at play.

##  I Introduction

Over the past decades, we have witnessed a great increase in the size of
machine learning models, measured in the number of free parameters that have
to be fit to the data. Model size has become especially important as we have
improved our understanding of scaling laws for language models, which govern
how increases in model size and training data produce better performance [1,
2].

This implies that understanding trends in model size is relevant for
understanding trends in performance. As an example, it is conceivable that
scaling models beyond trillions of parameters proves to be a considerable
challenge (for example, due to the lack of sufficient onboard memory in which
to store the model weights). This would cause slower scaling of ML models
going forward.

In this article, we study trends in the model sizes of milestone ML models
from the 1950s to 2022. In section 2, we analyze the trends we have identified
in the data. We find a significant speedup in model size growth starting from
2018, plus the apparition of a new trend of very big models starting from
2020. Section 3 provides a detailed discussion of the _parameter gap_ – the
striking scarcity of models of size between 20B and 70B parameters across all
publication dates. We establish that the gap is unlikely to be the result of
chance, and discuss five hypotheses to explain it. While we have been able to
falsify some of the hypotheses, we do not have a definitive explanation.

###  I-A Previous work

To our knowledge, there are no previous studies on trends in model size of
published machine learning systems across architectures. There have been
previous efforts on gathering model size data [3]. Similar analyses have been
performed on compute utilization [4], as well as performance improvement over
time [5]. In addition, the scaling literature contains many detailed reports
on the properties of concrete architectures at different model sizes [1, 2].
However, none of these address the study of trends in published model size
across different architectures.

Figure 1: Full dataset, separated by category. The cluster of large models is
indicated by a red circle.

###  I-B Our dataset

Figure 2: Log-linear regressions. Left: transition period between the pre-2018
and post-2018 trends. Right: Post-2018 trends, split by size

In previous work, we have compiled a dataset of the parameter counts of 238
notable ML models [6], chosen based on a series of inclusion criteria.
Specifically, we include models which satisfy at least one of the following:

  * •

Have more than 1,000 citations

  * •

Have widely accepted historical importance

  * •

Increased the state-of-the-art performance in a relevant benchmark or task

  * •

Were deployed in an important context, such as a widely-used search engine or
translation software.

Even if the total parameter count of a model is not specified in the
accompanying paper, it is usually straightforward to determine this based on
the model architecture. This is in contrast to calculating other metrics like
training compute, where there is more uncertainty in the estimates.

Each model in our dataset is categorized into one of four domains according to
the task it is designed to solve. The domains are Vision, Language, Games and
Other111The Other category compiles other domains like drawing, speech
recognition or multimodal models. A domain is aggregated into Other when we
have less than 15 systems of that domain in the dataset.. The full dataset can
be seen in Figure 1. The data covers the period from the 1950s to 2022, but
80% of the points are concentrated in the period after 2010, so it can only
provide reliable information on the last decade of ML history.

Period | Data | Scale (start to end) | Slope | Doubling time | R2
---|---|---|---|---|---
| 1952 to 2018
---
| $n=109$
---
1e+01 to 3e+7 params |  | 0.1 OOMs/year
---
[0.1; 0.1; 0.1]
| 39.1 months
---
[36.4; 39.7; 40.7]
| 0.62
---
| 2018 to 2022
---
(single trend)
| $n=129$
---
3e+7 to 2e+12 params |  | 0.9 OOMs/year
---
[0.9; 0.9; 1.0]
| 4.2 months
---
[3.5; 4.0; 4.3]
| 0.31
---
| 2018 to 2022
---
(above gap)
| $n=27$
---
7e+10 to 2e+12 params |  | 0 OOMs/year
---
[-0.4; -0.1; 0.2]
| 209 months
---
[-52.5; -14.2; 52.0]
| 0.00
---
| 2018 to 2022
---
(below gap)
| $n=102$
---
3e+7 to 2e+10 params |  | 0.5 OOMs/year
---
[0.4; 0.5; 0.5]
| 8 months
---
[7.0; 8.0; 9.8]
| 0.25
---
TABLE I: Summary of our main results. Around 2018 there was a general increase
in growth. This can be split into the previous trend increasing its growth
rate, and a separate cluster of very large models appearing on top.

##  II Trends in model size

The most salient feature is the recent increase in the rate of growth of model
size, starting from mid-2018222There is one Language system with around 1e+10
parameters from 2014, Word2Vec(large), which we treat as an outlier since it’s
a word embedding and thus has very different characteristics than most other
systems. The growth rate before 2018 was 0.1 OOMs/year (order of magnitudes
per year), whereas after 2018 the growth rate is 0.9 OOMs/year if we assume a
single exponential trend (see Figure 2).

However, we think this might be better explained as two different trends. It
is visually apparent in Figure 1 that in the 2018-2022 period the models can
be divided into two clusters: those above 70B parameters and those below 20B
parameters. The separation between these clusters is the “parameter gap” that
we mentioned earlier. However, doing this separation runs the risk of
overfitting, given the small sample size and that we are splitting the data
across time and model size into three different groups.

We can think of the models below the gap as mostly continuing the previous
trend, whereas the models above the gap constitute a separate new trend of big
models. With this distinction, the growth rate for models below the gap is
around 0.5 OOMs/year, while for those above the gap there’s no identifiable
trend333Fitting a trendline produces an R2 of 0.00.. A summary of all the
regressions can be found in Table I.

If we look at the trends by category (see Figure 3), all of them were growing
at 0.1 OOMs/year before 2018, but there are some differences after that. The
Games category seems to have been less prominent in recent years, slowing down
its growth rate. Meanwhile, the Language and Other categories are leading the
recent growth spurt, even after we remove the outliers above the parameter
gap. The Vision category sits in the middle of these two extremes, growing at
a moderate pace.

Figure 3: Below-gap trends by category.

Above the gap, we only see Language and Other systems. The Language systems
appear to be decreasing in size over time, but this is not really a
significant trend and a few early big Mixture-of-Experts (MoE) systems might
be causing this apparent decline444Since MoE models can be made much bigger
than dense models. Concretely, Switch [7] and GShard [8] are examples of early
big MoE models that might distort the trend. Indeed, if we remove them the
trendline looks completely flat for Language systems..

It is perhaps surprising that this acceleration took place around 2018, since
previous analyses on training compute trends found a similar increase in 2015
[9]. This discrepancy might reflect the recent increased interest in natural
language processing relative to other domains and the apparition of large
language models, which have a relatively high size-to-compute ratio compared
to other systems.

##  III The parameter gap

The parameter gap (Figure 4) was first observed by Jones 2021 [10], who
attributed it to differences in the computing infrastructure used to train the
models. Here we will build on his observations to clarify the significance and
nature of the gap. In section 3.1 we will establish that a gap of this size is
unlikely to have been produced by chance. In section 3.2 we add some further
context that we have found valuable to clarify possible explanations, and in
section 3.3 we present our best hypotheses to explain the gap.

Figure 4: The parameter gap, from 2+10 to 7e+10 parameters

###  III-A Statistical analysis

Before attempting to explain this gap, we performed a simple analysis to try
to establish the likelihood that this gap is due to random chance. We model
the model size as an exponential trend with lognormal noise.

$\log(s)=\beta t+\epsilon$

where $t$ is the year, $s$ is the model size and $\epsilon\sim
N(0,\sigma^{2})$.

After fitting the trend slope ($\beta$) and the noise variance($\sigma^{2}$)
to the observed data, we ran a Monte Carlo simulation of the model. In each
simulation we sample 238 points from these distributions on the same dates as
our observed data and calculate the maximum observed gap between two groups of
at least 10 points. The resulting gap size distribution after 10,000
simulations, together with the observed gap, can be seen in Figure 5.

Figure 5: Distribution of the widest parameter gap (in OOMs) for 10,000
simulations and observed gap size.

The observed gap is extremely unlikely according to our simple model – the
probability of observing a gap as large as the one present is on the order of
$10^{-5}$. This strongly suggests that the gap is not simply due to random
chance.

Figure 6: Distribution of maximum parameter gap observed in 10,000
simulations, fitting parameters only to models in the Other domain (left) or
Language domain (right).

###  III-B Further context

We list now some facts that might be useful to better understand the gap.

  1. 1.

The region in the gap is not completely empty. We are aware of some models
that lie within the gap, but which are not included in our datasets555This
includes Anthropic’s 52B model, the 39B version of HyperCLOVA, the 68B version
of FLAN, and the 62B version of PaLM.. Some of these are smaller versions of
larger models trained to test scaling laws, which we do not include because we
only include the largest model from each paper. However, in our view, this
does not explain why there are no publications whose biggest model falls into
that range.

We also found a single model, Anthropic’s 52B-parameter language model [11],
which was not included in the dataset for not satisfying the inclusion
criteria. Nevertheless, we think the relative scarcity of models in this range
is still surprising, even if some models do lie within.

  2. 2.

It is mostly a gap in language models. Since only models in the Language and
Other categories have reached the requisite size to be above the gap, we can
restrict our analysis to these two domains. The same statistical analysis of
the previous section, when performed only on systems in the Other domain,
shows that the gap can be explained by random noise, whereas for systems in
the Language domains the gap is very unlikely: The probability of observing
this gap conditional on our model being correct is 0.23 for systems in the
Other domain and 0.02 for those in the Language domain666There are 71 language
models in total, of which 12 are above the gap.. (see Figure 6). All of this
suggests that we focus our analysis on language models.

  3. 3.

The gap shrinks over time777Concretely, it initially spanned a full order of
magnitude, then was reduced to 0.91 OOMs by FLAN in 2021, then to 0.61 OOMs by
Chinchilla, and finally to 0.54 OOMs by GPT-NeoX-20B in 2022.. It seems likely
that within the next couple of years it will disappear, as other domains catch
up in model size and hardware and economic conditions change, and as models
are trained for a wider range of purposes.

  4. 4.

The first language model to surpass the gap was GPT-3, which when released was
10 times bigger than the largest previous language models. There are around 4
jumps of this size for language models in our dataset888The other jumps were
due to the release of Word2Vec, Hiero and IBM Watson. Of these, only Word2Vec
and GPT-3 happened after 2010.

  5. 5.

The parameter gap probably reflects a divide between expensive models and
cheaper models. We should expect the parameter gap to reflect a proportional
gap in training compute999Model size and training cost are not in general
directly proportional, since the size of the training dataset also influences
the amount of compute spent on training. However, increased understanding of
scaling laws since 2020 has compelled researchers to use a nearly fixed ratio
of model and dataset size.. This is not true in general, but we do see a small
gap if we focus only on language models (see Figure 7).

Furthermore, if we plot training compute and model size, we observe two
distinct clusters after GPT-3101010Here we ignore Mixture of Experts (MoE)
models because their scaling properties are very different from dense models.,
whose centers vary both across compute and across size (see Figure 7)111111We
do not have estimates of the training compute for all the models, so this plot
only includes a subset of the models.. So we can probably say that the
parameter gap reflects a divide between expensive models and cheaper models.

Figure 7: Left: Gap in training compute for language models, corresponding to
the gap in parameters. Right: Training compute and model size of language
models, showing two clusters of models after GPT-3.

  6. 6.

Models above the gap are almost exclusively language models, while those below
it span all domains. Models above the gap are overwhelmingly notable for
pushing the state of the art in language generation either in English or
Chinese, whereas models below the gap are much more varied in their domains
and contributions (see Table II in the appendix).

###  III-C Possible explanations

We present some hypotheses that try to explain the relative scarcity of models
in the 20B-70B range. These theories can be classified into two groups:

  1. 1.

A “fundamental threshold” group of theories. These state that there is
something specific about that size range that causes some tendency to avoid
models within it. An example of this kind of theory is “training models at
that scale is significantly less cost-effective than other scales”.

  2. 2.

An “external effects” group of theories that state that there is nothing
particular about that range, and instead there are outside incentives like
competitive pressure or different access to compute which produce this divide.

In addition, there’s the possibility that there is not really a gap and
instead our dataset is missing the relevant models. While this is somewhat
plausible, we could not find many systems within that range outside our
dataset.

####  III-C1 Fundamental threshold theories

The fundamental threshold theories posit that there is some transition around
the size of the gap where previous scaling strategies give way to new ones, in
a way that training a model at the intermediate regime is not as cost-
effective as training one on either side of the gap.

Cloud vs bespoke infrastructure: The first hypothesis from Derek Jones, was
that 50B parameters is the size at which it becomes more cost-effective to
employ bespoke computing infrastructure (supercomputers)121212Jones did not
clarify exactly what he meant by bespoke infrastructure, but the way we
understand it it means hardware which is in general not easily available for
rental and that might have been configured specifically for the task at hand.
An example might be BigScience’s language model being trained on the Jean Zay
supercomputer, which is only available for academic research and has a team of
engineers ready to help users., rather than generic cloud hardware.

It’s true that some of the bigger models have been trained on supercomputers,
in particular BigScience’s language model as well as some of the Chinese
models. However, we are not aware of other models that were trained in
supercomputers and in general, this does not seem to be the case. Most of the
systems above and below the gap were trained on TPUs and NVIDIA GPUs which are
commercially available in cloud services. See Table II in the appendix for a
full list.

Training parallelism threshold: GPUs have a limited amount of memory, 32GB for
NVIDIA V100, 40GB or 80GB for NVIDIA A100. Using a standard Adam optimizer
with no model parallelism or optimizer state sharding, this amount of memory
can fit around 1-2B parameters for training [12], since in addition to the
parameters themselves the optimization process imposes additional memory
overhead.

For this reason, models above that size usually use either pipeline
parallelism or tensor parallelism to split the training onto multiple GPUs.
Since a single computing node usually contains 8 or 16 GPUs131313For example,
NVIDIA’s DGX node contains 8 GPUs, whereas their DGX-2 node contains 16 GPUs.,
using this parallelism it is possible to train models up to around 20B
parameters within a single node [12, 13].

To train bigger models, we know of two techniques: taking advantage of non-GPU
memory to fit larger models in one node, and splitting the model across nodes.
As far as we know, the first option incurs a significant decrease in the
hardware utilization rate that can be attained. However, the second option
requires significant code refactoring and engineering expertise, since this
splitting requires significant communication between nodes, which are usually
connected by a slower network than within-node GPUs. Moreover, once past this
multi-node splitting threshold, training becomes more efficient with larger
models [12].

So, projects passing this threshold must pay a fixed cost in engineering
effort for training models bigger than 20B parameters. This implies that the
marginal cost of going from 20B to, for example, 40B parameters might be
higher than going from 40B to 60B parameters, so the total cost per parameter
is much higher in the region immediately after the threshold.

We are quite confident from our exploration of the scaling literature that
this threshold exists and it imposes a fixed cost, but it is not clear whether
such a cost is high enough to explain the gap. In our correspondence with
practitioners, this threshold has not been explicitly mentioned as a major
factor in their scaling decisions, which suggests that it is not that
important in practice.

Inference parallelism threshold: Another possibility is that the parallelism
threshold is more relevant during inference. Without needing to store
optimizer data, a single GPU can fit a 20B size model, but not a 40B size one.
In correspondence, EleutherAI stated this as a reason for limiting their model
size to 20B.

Deployment usually consumes much more compute than training: Patterson et al.
find that around 90% of total ML compute is spent on inference [14], so it
might make sense to optimize models for efficiency during inference more than
training. However, from our understanding parallelizing inference poses less
complex engineering challenges than training, since information only flows
forward so pipeline parallelism can be used much more efficiently, reducing
the need for other scaling techniques141414We are not practitioners so we are
not entirely sure about this, but basically during training the need to do a
forward pass of activations and a backward pass of gradients leads to less
utilization when using pipeline parallelism. However, if there is only a
forward pass pipeline parallelism should be able to achieve high hardware
utilization.. So the cost of passing this threshold seems much lower than for
training.

In our reading of the literature, we found quite a few papers that provide
details on their training infrastructure setup, whereas the inference
infrastructure setup is not usually mentioned. This provides some evidence
that inference is not seen as important from an engineering standpoint.

For collectives such as EleutherAI whose goal is increasing access to cutting-
edge language models, rather than commercial viability, reducing the number of
GPUs needed during inference might be an important goal in itself. On the
other hand, OPT-175B was also released to democratize access to big language
models, yet it employs 16 V100 GPUs during deployment [15]. BigScience also
has democratization as their goal, but they are training a 176B model which
will also require parallelization during inference. This indicates that
inference parallelization might not be such an important factor, even for
actors motivated by democratization.

####  III-C2 External effects theories

The external effects theories attempt to explain the gap by using arguments
unrelated to the technical details of the 20B-70B parameter range, such as
historical contingencies, economic or research incentives, and access to
hardware.

Compute divide + scaling laws: Due to scaling laws, recent language models
have been trained following a fixed ratio of training compute to model size
[2]. So perhaps the reason for the gap lies in the ‘compute divide’ [16]: some
players like Google or OpenAI have access to enormous computational resources,
whereas academic laboratories or other researchers usually have fewer
resources.

Under this hypothesis, large corporations and research laboratories could
train bigger models and other researchers would be forced to work with smaller
models. However, a significant number of models below the gap were produced by
top research laboratories, and some models above the gap have been produced by
other players151515Examples of systems below the gap made by big players
include Retro-7B, DeepNet and BigSSL. Systems above the gap are almost
completely produced by corporations, but BigScience is training a 176B model
which when completed will be above the gap. See Table II in the appendix for a
list of models and the organization that created them.. Thus, even if this
divide exists and prevents less resourceful actors from training big models,
it does not explain why no models trained by big corporations are present
within the gap.

Initial discontinuity + Record-setting models: Since GPT-3 demonstrated the
robustness of scaling, any project that wants to compete with it and push the
state of the art in language generation must either be bigger or leverage new
insights (for example, Chinchilla’s compute-optimal training). However, when
GPT-3 was released it was one order of magnitude bigger than any previous
language model. So its release caused a discontinuity in model size that was
preserved by the incentives to outperform GPT-3.

Under this hypothesis, the models below the gap can not be directly competing
with those above, perhaps because they are specialized in more niche tasks
(like non-English languages), or because the researchers have few incentives
to outcompete existing models (this might be the case for research collectives
like BigScience or EleutherAI).

Indeed, as can be seen in Table II, most models above the gap are notable for
having pushed the state of the art in English or Chinese language generation,
whereas below the gap we find other tasks such as language understanding and
Arabic and Japanese language generation. We also find more papers whose main
contributions are algorithmic improvements or which were produced by
independent authors or research collectives.

A problem with this hypothesis is that it predicts gaps that are not present:
if this hypothesis was true we would expect to see a gap in our dataset every
time there is a significant jump in model size. For example, when Word2vec was
released it was an order of magnitude bigger than previous language models,
but no gap appeared afterwards. Some peculiarities of the 2020 jump that might
explain the presence of a gap are the increased importance of scaling laws and
the fact that this time the jump represented a significant increase in
training cost.

####  III-C3 Comparing hypotheses

While we do not have evidence to establish whether our hypothesis fully
explain the gap, we have outlined the reasons to reject some of them, as shown
in the previous section. All in all, the hypotheses that we can’t rule out are
the training and inference parallelism threshold and record-setting models.

More concretely, while the parallelism thresholds are well documented, we are
uncertain about the fixed cost imposed by such thresholds and whether this
cost is high enough to explain the gap. Especially for inference, where the
cost of passing the threshold seems lower.

As for the record-setting models hypothesis, we have confirmed that most
models below the gap released after GPT-3 do not seem to compete directly with
big models on language generation performance, but instead make progress in
niche tasks or introduce innovations that do not include raising the SOTA in
language generation benchmarks (again, see Table II).

This makes the record-setting models hypothesis the one best supported by the
evidence so far, but we still are not confident about its ability to fully
account for the gap. Of course, these hypotheses are not mutually exclusive,
and it is likely that all of them play some role in generating the gap.
Establishing the relative importance of each of them is beyond our scope at
this point.

##  IV Conclusion

In this article, we explored the model size of some notable machine learning
systems and presented and analyzed some trends that this metric has been
following over the years.

While this data is somewhat noisy and hard to reason about (for example, due
to architectural differences between models), we believe we can draw from this
data some conclusions about the dynamics of progress in ML from it:

  * •

While language and recommendation models have driven a huge increase in size
over the last two years, other domains have seen much more moderate increases.

  * •

If we restrict ourselves to the models above the gap, there are no signs of
growth in model size in the past few years, so the overall trend will likely
catch up in the next few years.

In addition, we can generate some predictions from our hypotheses on the
parameter gap:

  * •

Hardware limitations might cause discontinuities or bottlenecks in scaling by
limiting model size rather than compute. For example, 3,072 GPUs can be used
to train a trillion parameter Transformer [12]. Summit, the 4th top
supercomputer as of July 2022, has 27,648 GPUs. So training dense language
models beyond 10T parameters will likely require using non-GPU memory, bigger
supercomputers (which take time to build), or communication between
supercomputers.

  * •

After a sudden jump in the largest model size used for a given task or
benchmark, we should expect to stop seeing smaller notable models competing in
the same tasks or benchmarks, at least in domains with robust scaling laws.
This implies that a single well-funded actor can suddenly change the dynamics
of AI research. In particular, by (a) causing less resourceful actors to stop
working on a given task or benchmark since the well-funded actor jumped to too
large a model size to be worth trying to compete with, (b) causing other well
funded actors to increase their spending if they want to remain competitive.

## Acknowledgments

Many thanks to Derek Jones who spotted the parameter gap, as well as Stella
Biderman, Peter Wildeford, Yu Zhang, Furu Wei, Stephen Roller, Laurent Sifre
and Jared Kaplan for answering our questions. We also thank Ben Cottier,
Michael Ashcroft and Michael Aird for their reviews and comments on earlier
drafts of this paper.

## References

  * Kaplan et al. [2020] J. Kaplan _et al._ , “Scaling Laws for Neural Language Models,” 2020, _eprint: 2001.08361.
  * Hoffmann et al. [2022] J. Hoffmann _et al._ , “Training compute-optimal large language models,” 2022\.
  * Talat et al. [2022] Z. Talat _et al._ , “You reap what you sow: On the challenges of bias evaluation under multilingual settings,” in _Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models_. virtual+Dublin: Association for Computational Linguistics, May 2022, pp. 26–41.
  * Amodei and Hernandez [2018] D. Amodei and D. Hernandez, “ _AI and Compute_ ,” May 2018, published: OpenAI Blog.
  * [5] “Papers with code - the latest in machine learning.”
  * Sevilla et al. [2021] J. Sevilla, P. Villalobos, and J.F. Cerón, “\emphParameter Counts in Machine Learning,” Jun. 2021, published: Alignment Forum (blog).
  * Fedus et al. [2021] W. Fedus, B. Zoph, and N. Shazeer, “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity,” Jan. 2021.
  * Lepikhin et al. [2020] D. Lepikhin _et al._ , “Gshard: Scaling giant models with conditional computation and automatic sharding,” 2020.
  * Sevilla et al. [2022] J. Sevilla _et al._ , “Compute trends across three eras of machine learning,” https://epochai.org/blog/compute-trends, 2022, accessed: 2022-06-09.
  * Jones [2022] D. Jones, “Growth in flops used to train ml models,” 2022.
  * Askell et al. [2021] A. Askell _et al._ , “A general language assistant as a laboratory for alignment,” 2021.
  * Narayanan et al. [2021] D. Narayanan _et al._ , “Efficient large-scale language model training on gpu clusters using megatron-lm,” 2021.
  * Rajbhandari et al. [2021] S. Rajbhandari _et al._ , “Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning,” 2021.
  * Patterson et al. [2021] D. Patterson _et al._ , “Carbon emissions and large neural network training,” 2021.
  * Zhang et al. [2022] S. Zhang _et al._ , “Opt: Open pre-trained transformer language models,” 2022\.
  * Ahmed and Wahed [2020] N. Ahmed and M. Wahed, “The de-democratization of ai: Deep learning and the compute divide in artificial intelligence research,” 2020.
  * Hendrycks et al. [2021] D. Hendrycks _et al._ , “Measuring mathematical problem solving with the math dataset,” 2021.
  * Brown et al. [2020] T.B. Brown _et al._ , “Language Models are Few-Shot Learners,” 2020, _eprint: 2005.14165.
  * Jones [2021] A.L. Jones, “Scaling Scaling Laws with Board Games,” _arXiv preprint arXiv:2104.03113_ , pp. 1–8, 2021.
  * Thompson et al. [2020] N.C. Thompson _et al._ , “The Computational Limits of Deep Learning,” 2020, _eprint: 2007.05558.
  * Sharir et al. [2020] O. Sharir, B. Peleg, and Y. Shoham, “The Cost of Training NLP Models: A Concise Overview,” 2020, _eprint: 2004.08900.
  * Naumov et al. [2019] M. Naumov _et al._ , “Deep Learning Recommendation Model for Personalization and Recommendation Systems,” May 2019.
  * Shoeybi et al. [2019] M. Shoeybi _et al._ , “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism,” Sep. 2019.
  * Rosset [2020] C. Rosset, “Turing-NLG: A 17-billion-parameter language model by Microsoft,” Feb. 2020.
  * Zhang et al. [2020] Z. Zhang _et al._ , “CPM: A Large-scale Generative Chinese Pre-trained Language Model,” Dec. 2020.
  * Antoun et al. [2020] W. Antoun, F. Baly, and H. Hajj, “AraGPT2: Pre-Trained Transformer for Arabic Language Generation,” Dec. 2020\.
  * Pham et al. [2020] H. Pham _et al._ , “Meta Pseudo Labels,” Mar. 2020.
  * Mudigere et al. [2021] D. Mudigere _et al._ , “Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models,” Apr. 2021.
  * So et al. [2021] D.R. So _et al._ , “Primer: Searching for Efficient Transformers for Language Modeling,” Sep. 2021.
  * [30] “GPT-Neo,” Eleuther AI.
  * Zeng et al. [2021] W. Zeng _et al._ , “PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation,” Apr. 2021.
  * Komatsuzaki [2021] A. Komatsuzaki, “GPT-J-6B: 6B JAX-Based Transformer,” Jun. 2021.
  * [33] “Naver Corporation,” Naver Corporation.
  * [34] “Baidu Research,” Baidu Research.
  * Hsu et al. [2021] W.N. Hsu _et al._ , “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,” Jun. 2021.
  * [36] O. Lieber _et al._ , “Announcing AI21 Studio and Jurassic-1 Language Models.”
  * Lin et al. [2021] J. Lin _et al._ , “M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining,” Oct. 2021.
  * Alvi and Kharya [2021] A. Alvi and P. Kharya, “Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, the World’s Largest and Most Powerful Generative Language Model,” Oct. 2021\.
  * Wu et al. [2021] S. Wu _et al._ , “Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning,” Oct. 2021.
  * [40] J. Rae, G. Irving, and L. Weidinger, “Language modelling at scale.”
  * Lan et al. [2019] Z. Lan _et al._ , “ALBERT: A Lite BERT for Self-supervised Learning of Language Representations,” Sep. 2019.
  * Thoppilan et al. [2022] R. Thoppilan _et al._ , “LaMDA: Language Models for Dialog Applications,” Jan. 2022.
  * Leahy [2022] C. Leahy, “Announcing GPT-NeoX-20B,” Feb. 2022\.
  * Chowdhery et al. [2022] A. Chowdhery _et al._ , “Palm: Scaling language modeling with pathways,” 2022\.
  * Alayrac et al. [2022] J.B. Alayrac _et al._ , “Flamingo: a visual language model for few-shot learning,” 2022.

## Appendix: list of models

Above gap
---
System | Organization | Hardware type | Number of chips | Notability
GPT-3 | OpenAI | V100 |  | Language generation SOTA
FLAN | Google | TPUv3 | 128 | Language generation SOTA
ERNIE 3.0 Titan | Baidu | V100 + Ascend 910 |  | Language generation SOTA (Chinese)
Efficient Large-Scale Training on GPU Clusters | Nvidia | A100 | 3072 | Scaling
Jurassic-1 | AI21 | GPU | 800 | Language generation SOTA
Yuan 1.0 | Inspur | GPU (likely V100) | 2128 | Language generation SOTA (Chinese)
Turing-NLG 530B | Microsoft | A100 |  $\sim$8000 | Language generation SOTA
Gopher | Google | TPUv3 | 4096 | Language generation SOTA
LaMDA | Google | TPUv3 | 1024 | Dialog
PaLM | Google | TPUv4 | 6144 | Language generation SOTA
Flamingo | Google | TPUv4 | 1536 | Multimodal text + image understanding SOTA
OPT-175B | Meta | A100 | 992 | Efficiency, democratization
Pangu-$\alpha$ | Pangu-$\alpha$ team | Ascend 910 | 2048 | Language generation SOTA (Chinese)
HyperCLOVA | Naver | A100 | 1024 | Language generation SOTA (Korean)
Below gap
---
System | Org | Hardware type | Number of chips | Notability
DeepNet | Microsoft |  |  | Algorithmic innovation
Retro-7B | DeepMind | TPUv3 |  | Algorithmic innovation
GPT-J-6B | EleutherAI | TPUv3 |  | Produced by research collective
Seq2seq | Facebook | GPUs |  | Dialog SOTA
BigSSL | Google | TPUv3 | 1024 | Speech recognition SOTA
Turing NLG 17B161616This model was released before GPT-3. | Microsoft | V100 | 4096 | Language generation SOTA
DeBERTa | Microsoft | V100 | 4096 | Language understanding SOTA
Japanese dialog transformers | NTT | V100 16GB | 400 | Dialog (Japanese)
Primer | Google | TPUv4171717This is the hardware used for the biggest model, other models used TPUv2, TPUv3 and V100 GPUs | 512 | Algorithmic innovation
AraGPT2 | American University of Beirut | TPUv3 | 128 | Language generation (Arabic)
T0-XXL | BigScience | TPUv3 | 512 | Improved training methodology
XLM-R | Facebook |  |  | Language understanding SOTA
Codex | OpenAI |  |  | Code generation SOTA
ERNIE 3.0 | Baidu | V100 | 384 | Language understanding SOTA (Chinese)
GPT-NeoX-20B | EleutherAI | A100 40GB | 96 | Produced by research collective
TABLE II: List of post-GPT-3 language models above and below the gap, together
with the organization which produced them, the hardware used for training, and
the reason why they are notable according to our criteria. This last item was
quickly and subjectively evaluated from quick glances at the paper abstracts,
so it’s not a complete characterization of the merits of each paper.