# The Ecological Footprint of Neural Machine Translation Systems

Dimitar Shterionov111Corresponding author Eva Vanmassenhove
Department of Cognitive Science
And Artificial Intelligence
Tilburg University
{d.shterionov<EMAIL_ADDRESS>
(March 2021)

###### Abstract

Over the past decade, deep learning (DL) has led to significant advancements
in various fields of artificial intelligence, including machine translation
(MT). These advancements would not be possible without the ever-growing
volumes of data and the hardware that allows large DL models to be trained
efficiently. Due to the large amount of computing cores as well as dedicated
memory, graphics processing units (GPUs) are a more effective hardware
solution for training and inference with DL models than central processing
units (CPUs). However, the former is very power demanding. The electrical
power consumption has economical as well as ecological implications.

This chapter focuses on the ecological footprint of neural MT systems. It
starts from the power drain during the training of and the inference with
neural MT models and moves towards the environment impact, in terms of carbon
dioxide emissions. Different architectures (RNN and Transformer) and different
GPUs (consumer-grate NVidia 1080Ti and workstation-grade NVidia P100) are
compared. Then, the overall CO2 offload is calculated for Ireland and the
Netherlands. The NMT models and their ecological impact are compared to common
household appliances to draw a more clear picture.

The last part of this chapter analyses quantization, a technique for reducing
the size and complexity of models, as a way to reduce power consumption. As
quantized models can run on CPUs, they present a power-efficient inference
solution without depending on a GPU.

Keywords— Neural machine translation, Power consumption, Carbon dioxide
emissions, GPU comparison, Quantization, LSTM, Transformer, Europarl

##  1 Introduction

Over the past decade, Deep Learning (DL) techniques took the world by storm
and their application led to state-of-the-art results in various fields. The
same holds for the field of Machine Translation (MT), where the latest AI boom
led the way for the emergence of a new paradigm _Neural Machine Translation_
(NMT). Since most of the foundational techniques used in current applications
of AI were developed before the turn of the century, the main triggers of the
boom were innovations in general purpose GPU computing as well as hardware
advancements that facilitate much more efficient training and inference. In
spite of the improvements in terms of efficiency, GPUs are more power
demanding than their CPU predecessors and as such, they have a considerably
higher environmental impact.

In this chapter, a brief overview of the most recent paradigms is presented
along with a more in depth introduction to the core processing technology
involved in NMT (GPUs as opposed to CPUs). We elaborate on the exponential
growth of models, the trend of ‘big data’ and their relation to model
performance. The related work discusses pioneering and recent papers on Green
AI for Natural Language Processing (NLP) as well as tools to quantify the
environmental impact of AI.

As a case study and to outline the realistic dimensions of power consumption
and environmental footprint of NMT, we train 16 NMT models and use them for
translation, while collecting power readings from the corresponding GPUs.
Using the collected measurements we compare different NMT architectures (Long
Short-Term Memory (LSTM) and Transformer), different GPUs (NVidia GTX 1080Ti
and NVidia Tesla P100) as well as the environmental footprint of training and
translation with these models to other, commonly-used devices.

Together with the research and analytical contributions, this chapter also
aims to motivate researchers to devote time, efforts and investments in
developing more ecological solutions for MT.

##  2 The Technological Shift(s) in MT

Machine translation (MT), the task of automatically translating text in one
language, into text in another language using a computer system, has become an
indispensable tool for professional translators (to assist in the translation
workflow), for commercial users, e.g. e-commerce companies (to make their
content quickly available in multiple languages), to every-day users (to
access information unrestricted by the language in which it is produced).
Since its inception in the late 1950s, MT has undergone many shifts, the
latest of which, neural machine translation (NMT) imposes a requirement for
hardware that can facilitate efficient training and inference with NMT models.
Most commonly used hardware are graphics processing units (GPUs) that support
embarrassingly parallel computations due to the large amounts of processing
cores and dedicated memory.222There are other hardware and software that are
specifically developed for AI accelerated computing, e.g. tensor processing
units (https://cloud.google.com/tpu/docs/tpus). However, as the most commonly
used and easily accessible such devices are GPUs, our work focuses on the
power considerations and environmental footprint of GPUs.

###  2.1 From rule-based to neural MT

In the early days of machine translation (MT), rule-based MT (RBMT) systems
were built around dictionaries and human-crafted rules to convert a source
sentence into its equivalent in the target language. Such systems were heavily
dependant on the efforts and skills of linguists. Developing a rule-based
system for a new domain or a new language pair was (an still is) a cumbersome,
time-consuming task that requires extensive linguistic expertise (Arnold et
al., 1994). However, from a computational point of view, it is an inexpensive
task – using an RBMT system does not require substantial computational
resources.

In the 1980s, researchers attempted overcoming some of the shortcomings of
RBMT when dealing with languages that differed substantially structurally
(e.g. English vs Japanese). Focusing specifically on collocations, examples
could be used for transfer when rules and trees failed (Nagao, 1984). These
hybrid approaches evolved by the end of the decade into a more example-
centered approach (example-based MT (EBMT)), where patterns would be retrieved
from existing corpora and adapted using hand-written rules (Hutchins, 2005b).
The idea of using patterns extracted from corpora culminated when, in the
early 90s, a group of researchers at IBM created an MT system relying solely
on statistical information extracted from corpora (Hutchins, 2005a).

The generation of corpus-based MT systems rely on data and statistical models
to derive word-, phrase- or segment-level translations eliminating the need
for complex semantic and/or syntactic rules. As such, the core mechanism of MT
systems shifted from human expertise in linguistics to machine learning
techniques. This shift entailed other important changes for MT related to the
development time and the computational resources required for training333As a
matter of fact, the technical advances in terms of computational resources
facilitated developments in the field of corpus-based MT. The paradigm shift
furthermore coincided with the late 90s and early 2000s growth in terms of
direct applications for MT and localisation.. Furthermore, this group of MT
paradigms that learn automatic translation models from large amounts of
parallel and monolingual data reached – for in-domain translations and given
enough available training data – a better overall translation quality than
that achieved by earlier RBMT systems.

Up until about 2016, Phrase-Based Statistical MT (PB-SMT) was the dominant
corpus-based paradigm (especially after Google Translate made the switch from
RBMT to SMT) (Bentivogli et al., 2016). Currently, most state-of-the-art
results for MT are achieved using neural approaches, i.e. models based on
artificial neural networks and most prominently recurrent neural networks
(RNNs) (Sutskever et al., 2014; Bahdanau et al., 2015; Cho et al., 2014) and
Transformer architectures (Vaswani et al., 2017). RNNs, as the name suggests,
feed their output as input, along with new input. This enables RNNs to
compress sequences (of tokens, e.g. words) of arbitrary length into a fixed-
size representation. This representation can then be used to initiate a
decoding process where one token is generated at a time (again using a
recurrent network) conditioned on the previously generated tokens and the
encoded representation of the input until a certain condition is met. In the
context of MT, this generation process typically continues until the end-of-
sentence token is generated. To mitigate issues related to long-distance
dependencies, LSTM units (Zaremba et al., 2014) or gated recurrent units (Cho
et al., 2014) are typically used instead of simple RNNs. In addition, to
further improve the relation between encoder and decoder, an attention
mechanism is added (Luong et al., 2015). The attention mechanism learns to
associate different weights based on the importance of individual input
tokens. It has been shown that attention-based models significantly outperform
those that do not employ attention. In contrast to NMT using RNNs, Transformer
does not employ recurrence; it uses self attention, i.e. an attention
mechanism that indicates the importance of a token with respect to the other
input tokens. Within a self-attention mechanism the positional information is
lost. As such, Transformer employs a positional encoding that learns the
positional information. These operations can be performed in parallel for
different tokens, which allows the training process of a Transformer model to
be parallelised, unlike RNNs, where operations are performed sequentially,
making Transformer a more efficient architecture.

For a more complete overview and description of the history of Machine
Translation, we refer to the work of Hutchins (2005b), Kenny (2005), Poibeau
(2017) and Koehn (2020).

###  2.2 Why GPUs?

Neural MT (NMT) evolved very quickly, replacing PB-SMT in both academia and
industry. Aside from the paradigm shift, NMT imposed another big change within
the field of MT – that of the core processing technology. In particular,
employing graphics processing units (GPUs) instead of central processing units
(CPUs). Training neural networks (NNs) revolves around the manipulation of
large matrices. A general-purpose, high-performance CPU typically contains 16
to 32, high-frequency cores that can operate in parallel. CPUs are designed
for general purpose, sequential operations. However, the sizes of matrices
involved in training NNs are far beyond the processing and memory capacities
of CPUs; processes that can (theoretically) be parallelised need to be
serialized and conducted in a sequence, leading to rather high processing
times. To perform all matrix-operations efficiently a parallel-processing
framework is more suitable. Graphics processing units (GPUs) are designed to
render and update graphics. They encapsulate thousands of cores with tens of
thousands of threads. With their large dedicated memory GPUs can host both an
NN model and training examples reducing the required memory transfer. The
large amount of processing cores that can operate in parallel and the
dedicated memory makes GPUs a much more effective option for training NN
models (Raina et al., 2009), including NMT models. See Figure 1 for a visual
comparison between a CPU and a GPU in terms of cache, control and processing
units.

Figure 1: Visual comparison between a CPU and a GPU in terms of cache, control
and processing units. Source: https://fabiensanglard.net/cuda/

In GPUs, however, many more transistors are dedicated to data processing,
rather than to caching and control flow as in CPUs (Raina et al., 2009). GPUs
are in fact much more power demanding than CPUs leading to two types of
considerations: (i) first, the physical power requirements towards a data
center or a workstation dedicated to training NN models, and (ii) second, the
ecological concern related to the production and consumption of electricity to
power up and sustain the training of NN, including NMT, models. In this
chapter we are going to focus on the second question. We will present actual
power and thermal indicators measured during the training of different NMT
models and align them with ecological as well as economical markers in Section
5. Then, in Section 6, we will discuss two approaches to reduce the GPU power
consumption – distribution and parallelisation, and quantization.

###  2.3 The more, the better?

Since statistical models (both traditional and deep learning) took over in the
field of Natural Language Processing (NLP), datasets and models grew bigger.
Especially since 2018 when BERT (Devlin et al., 2019) and its successors (a.o.
GPT 2 (Radford et al., 2019), GPT-3 (Brown et al., 2020) and Turing NLG
(Microsoft, 2020)) appeared, the size of language models and the amount of
parameters grew exponentially. Since the relation between a model’s
performance and its complexity is at best logarithmic (Schwartz et al., 2020),
exponentially larger models are being trained for often small gains in
performance. This exponential growth is illustrated well by the Switch-C, the
current largest language model introduced in 2021 with a capacity of 1.6
trillion parameters. For comparison, one of its recent predecessors GPT-3,
currently the third444The second largest model is the GShard (Lepikhin et al.,
2020), introduced in September 2020 which had a capacity of 600 billian
parameters. largest model, introduced in June 2020 had a capacity of ‘only’
175 billion parameters.555https://analyticsindiamag.com/open-ai-
gpt-3-language-model/ Similarly, a blogpost by OpenAI (Open AI, 2018)
demonstrated how the compute grew by more than 300.000x. This corresponds to a
doubling every 3.4 months 666For comparison, Moore’s Law forecasted a doubling
every 2 years for the number of transistors in a dense integrated circuit
(Open AI, 2018)

Many studies have investigated the performance of NMT in terms of adequacy,
fluency, errors, data requirements, impact on the translation workflow and
language service providers, bias. The efficiency and energy impact of
developing new Neural MT (NMT), however, has not yet received the necessary
attention. In Section 3 we present related work in order to properly position
our research in current literature.

##  3 Related work

The related work is divided into three subsections: Section 3.1, Section 3.2
and Section 3.3. In Section 3.1, we cover the related research papers. Since
there are a few more practical tools that have been suggested and constructed
in order to measure the environmental and financial cost, we cover the main
tools in Section 3.2. Finally, in Section 3.3, we mention some recent
initiatives related to sustainable NLP.

###  3.1 Research

Recent work (Strubell et al., 2019; Schwartz et al., 2020) brought to the
attention of the NLP community the environmental (carbon footprint) and
financial (hardware and electricity or cloud compute time) cost of training
‘deep’ NLP models. In Strubel et al. (2019) the energy consumption (in
kilowatts) of different state-of-the-art NLP models is estimated. With this
information, the carbon emissions and electricity costs of the models can be
approximated. From their experiments regarding the cost of training, it
results that training BERT (Devlin et al., 2019) is comparable to a trans-
American flight in terms of carbon emissions. They also quantified the cost of
development by studying logs of a multi-task NLP model (Strubell et al., 2018)
that received the Best Long Paper award at EMNLP 2018. From the estimated
development costs, it resulted that the most problematic aspect in terms of
cost is the tuning process and the full development cycle (due to
hyperparameter grid searches)777During tuning the model is trained from an
already existing checkpoint, typically using new data.,888In the development
cycle, different versions of the model are trained or tuned and evaluated.
Each of those differ in terms of hyperparameter values. and not the training
process of a single model. They conclude their work with three recommendations
for NLP research which stress the importance of: (i) reporting the time
required for (re)training and the hyperparameters’ sensitivity, (ii) the need
for equitable access to computational resources in academia, and (iii) the
development of efficient hardware and models.

Similar to Strubell et al. (2019), the work by Schwartz et al. (2020)
advocates for ‘Green AI’, which is defined as “AI research that is more
environmentally friendly and inclusive” (Schwartz et al., 2020) and is
directly opposed to environmentally unfriendly, expensive and thus exclusive
‘Red AI’. Although the ‘Red AI’ trend has led to significant improvements for
a variety of AI tasks, Schwartz et al. (2020) stress that there should also be
room for other types of contributions that are greener, less expensive and
that allow young researcher and undergraduates to experiment, research and
have the ability to publish high-quality work at top conferences. The trend of
so-called ‘Red AI’, where massive models are trained using huge amount of
resources, can almost be seen as a type of ‘buying’ stronger results,
especially given that the relation between the complexity of a model and its
performance is at best logarithmic implying that exponentially larger model
are required for linear gains. Nevertheless, their analysis of the trends in
AI based on papers from top conferences such as ACL999https://acl2018.org and
NeurIPS 101010https://nips.cc/Conferences/2018 reveals that there is a strong
tendency within the field to focus merely on the accuracy (or performance) of
the proposed models with very few papers even mentioning other measures such
as speed, model size or efficiency. They propose making the efficiency of
models a key criterion aside (or integrated with) commonly used metrics. There
are multiple ways to measure efficiency111111For a more detailed overview of
measures we refer to the paper itself., floating point operations (FPO) being
the one advocated for by Schwartz et al. (2020). FPO is a metric that has
occasionally been used to determine the energy footprint of models (Molchanov
et al., 2016; Vaswani et al., 2017; Gordon et al., 2018; Veniat and Denoyer,
2018) as it estimates “the work performed by a computation process” (Schwartz
et al., 2020) based on two operations ‘ADD’ and ‘MUL’. They furthermore
advocate for reporting a baseline that promotes data-efficient approaches by
plotting accuracy as a function of “computational cost and of training set
size” (Schwartz et al., 2020). Aside from the environmental impact, both
recent papers stress also the importance of making research more inclusive and
accessible and thus advise reviewers of journals and conferences to recognize
contributions to the field by valuing certain contributions (e.g. based on
efficiency) even when they do not beat state-of-the-art results.

While in the context of a different topic, that of NLP leaderboards, in its
critique on how the community’s focus has shifted towards accuracy rather than
applicability, the work of Ethayarajh and Jurafsky (2020) points out the
importance of efficiency, fairness, robustness and other desiderata on
applicability of NLP models in practice. In fact, in a commercial setting
where revenue is the main criterion of success, efficiency (i.e. of training /
updating models, or of inference) is often more important than accuracy. See,
for example, the work of Shterionov et al. (2019) which compares several
systems for quality estimation of MT based on performance metrics, business
metrics and efficiency (training and inference times). This publication stems
from an academic-industry collaboration, based on commercial data and use-
cases.

Another recent, yet heavily debated121212Among others due to the fact that it
has been considered the centerpiece that led to Google firing leading A.I.
ethics researcher Dr. Timnit Gebru, and subsequently Dr. Margaret Mitchell,
two of the authors of the paper., paper by Bender et al. (2021), focuses on a
broader range of problems related to current trends in A.I. and the risks
associated with these trends. Their paper goes beyond merely discussing the
environmental impact of large models as it covers various concerns related
specifically to the unfathomable nature of large data sets and its potential
consequences (related to e.g. bias, derogatory associations, stereotyping),
including for downstream tasks. The section that focuses specifically on
environmental and financial cost mainly summarizes findings of previous papers
(e.g. Strubell et al. (2019) and Schwartz et al. (2020)) as well as some
recent tools and benchmarks. They also provide a table with an overview of the
12 most recent, largest language models along with their number of parameters
and the data set sizes, which illustrates well how these massive language
models have grown exponentially over the last two years. The paper furthermore
points out that the majority of these technologies are constructed for the
already privileged part of society while marginalized populations are the ones
more likely to experience environmental racism (Barsh, 1990; Pulido, 2016;
Bender et al., 2021).

Both Lacoste et al. (2019) and Henderson et al. (2020) provide tools (see
Section 3.2 for more details) to help researchers report carbon emissions and
energy consumption. Aside from the tools themselves, both papers also provide
an explanation of certain factors (such as server location, energy grid,
hardware and the run-time of training procedures) and their impact on the
emissions as well as more practical mitigation guidelines for researchers. The
recommendations and best practices listed in Lacoste et al. (2019) center
around transparency, the choice of cloud providers, the location of the data
center, a reduction of wasted resources and the choice of hardware. Henderson
et al. (2020) provides additional mitigation and reporting strategies
highlighting how both industry and academia can improve their environmental
impact in terms of carbon and energy efficiency. Their mitigation strategies
are an extension of those provided in Lacoste et al. (2019) and include:
ensuring the reproducibility of experiments (and thus avoiding replication
difficulties), focusing more on energy-efficiency and energy-performance
trade-offs, reducing overheads for utilizing efficient algorithms and
selecting efficient test environments.

###  3.2 Tools

Quantifying the environmental impact of AI technology is a difficult task that
might have prevented researchers from reporting energy consumption and/or
carbon emissions. To overcome this problem and to make this information more
easy to report and calculate, Lacoste et al. (2019) released an online
tool.131313https://mlco2.github.io/impact Given the run-time, hardware and
cloud provider, the emission calculator estimates the raw carbon emissions and
the approximate offset carbon emissions of your research. The cloud provider
is of importance since the emissions incurred during training depend on the
energy grid and the location of the training server. The main goal of the tool
is to push researchers for transparency within the field of Machine Learning
by publishing the ‘Machine Learning Emissions Calculator’ results in their
publications.

More recently, and similar to Lacoste et al. (2019), Henderson et al. (2020)
hypothesized that the complexity of collecting and estimating energy and
carbon metrics has been one of the main bottlenecks for researchers. To
encourage researchers to include such metrics in their work, they present the
‘experiment-impact-tracker’.141414https://github.com/Breakend/experiment-
impact-tracker Their framework aims to facilitate “consistent, easy and more
accurate reporting of energy, compute and carbon impacts of ML systems”
(Henderson et al., 2020). The authors claim that the assumptions made by
previous estimation methods (Lacoste et al., 2019; Schwartz et al., 2020) lead
to to significant inaccuracies, particularly for experiments relying heavily
on both GPUs and CPUs.

###  3.3 Workshop

SustaiNLP151515https://sites.google.com/view/sustainlp2020: The first
SustaiNLP workshop was held in November, 2020 and (virtually) co-located with
EMNLP2020.161616https://2020.emnlp.org/ It specifically focused on efficiency,
by encouraging researchers to design solutions that are more simple yet
competitive with the state-of-the-art, and justifiability, by stimulating
researchers to provide justifications for models and in this way encouraging
for more novel and creative designs.

##  4 Case Study: Empirical evaluation of MT systems

In the following three sections we will try to give a realistic image of the
power consumption and environmental footprint, in terms of carbon emissions,
related to the usage of GPUs for training and translating with NMT models. To
do that, we train multiple NMT models, using both LSTM and Transformer
architectures on different GPUs. We record the power consumption for each GPU
during training as well as during translation. In Section 5 we analyse the
results and in Section 6 we present one possible strategy to reduce power
consumption at inference time, i.e. quantization – the process of
approximating a neural network’s parameters by reducing their precision, thus
reducing the size of a model.

###  4.1 Hardware setup

To assess the energy consumption during the training and translation processes
of NMT we trained different NMT models from scratch, i.e. without any
pretraining nor relying on additional models (e.g. BERT) on two different
workstations, equipped with different GPUs. The GPUs we had at our disposal
are four NVidia GeForce 1080Ti with 11GB of vRAM and 3 NVidia Tesla P100 with
16GB or vRAM. These units differ not only in terms of technical
specifications, but also in their purpose of use – the 1080Ti is a user-class
GPU, developed for desktop machine with active cooling; the P100 is designed
for workstations that operate continuously and does not support active
cooling.

In Table 1, we compare the two GPU types in terms of their specifications.

| 1080Ti | P100
---|---|---
CUDA cores | 3584 | 3584
vRAM | 11 GB | 16 GB
Core clock speed | 1481 MHz | 1190 MHz
Boost clock | 1600 MHz | 1329 MHz
Transistor count | 11,800 million | 15,300 million
Manufacturing process technology | 16 nm | 16 nm
Power consumption (TDP) | 250 Watt | 250 Watt
Maximum GPU temperature |  91 ∘C |  85 ∘C
Floating-point performance | 11,340 gflops | 10,609 gflops
Type | Desktop | Workstation
Table 1: An overview of the GPU specifications of 1080Ti and P100.

The rest of the configurations are as follows: (i) for the 1080Ti desktop
workstation – an Intel(R) Core(TM) i7-7820X CPU @ 3.60GHz, RAM: 64GB (512MB
block) (ii) for the P100 workstation – CPU: Intel(R) Xeon(R) Gold 6128 CPU @
3.40GHz, RAM: 196 GB (1GB block).

In this work we focus on assessing the power consumption related to the
utilization of GPUs. As such, in this chapter we will not provide metrics
related to the CPU utilization and the corresponding power consumption. This
decision is motivated by the fact that GPUs are the main processing unit for
NMT models.

###  4.2 Machine Translation Systems

We experimented with the current state-of-the-art data-driven paradigms, RNN
(LSTM) and Transformer. We used data from the Europarl corpus (Koehn, 2005)
for two language pairs, English–French and English–Spanish in both direction
(EN$\rightarrow$FR, FR$\rightarrow$EN, EN$\rightarrow$ES and
ES$\rightarrow$EN). Our data is summarised in Table 2. For reproducibility
reasons, the model specifications, data, scripts and our results are made
publicly available for comparison on https://github.com/dimitarsh1/NMT-
EcoFootprint.git.

Lang. pair | Train | Test | Dev
---|---|---|---
EN-FR/FR-EN | 1,467,489 | 499,487 | 7,723
EN-ES/ES-EN | 1,472,203 | 459,633 | 5,734
Table 2: Number of parallel sentences for the training, testing and
development sets.

We ought to note that our test set is atypically large. One of the main
reasons for these experiments is that it can give us measures over a larger
period of time enabling us to draw more stable conclusions.

For the RNN and Transformer systems we used OpenNMT-
py.171717https://opennmt.net/OpenNMT-py/ The systems were trained for maximum
150K steps, saving an intermediate model every 5000 steps or until reaching
convergence according to an early stopping criteria of no improvements of the
perplexity (scored on the development set) for 5 intermediate models. The
options we used for the neural systems are:

  * •

RNN: size: 512, RNN type: bidirectional LSTM, number of layers of the encoder
and of the decoder: 4, attention type: MLP, dropout: 0.2, batch size: 128,
learning optimizer: Adam (Kingma and Ba, 2014) and learning rate: 0.0001.

  * •

Transformer: number of layers: 6, size: 512, transformer_ff: 2048, number of
heads: 8, dropout: 0.1, batch size: 4096, batch type: tokens, learning
optimizer Adam with beta${}_{2}=0.998$, learning rate: 2.

All NMT systems have the learning rate decay enabled and their training is
distributed over 4 nVidia 1080Ti GPUs. The selected settings for the RNN
systems are optimal according to Britz et al. (2017); for the Transformer we
use the settings suggested by the OpenNMT
community181818http://opennmt.net/OpenNMT-py/FAQ.html as the optimal ones that
lead to quality on par with the original Transformer work (Vaswani et al.,
2017).

For training, testing and validation of the systems we used the same data. To
build the vocabularies for the NMT systems we used sub-word units, allowing
NMT to be more creative. Using sub-word units also mitigates to a certain
extent the out of vocabulary problem. To compute the sub-word units we used
BPE with 50,000 merging operations for all our data sets. Separate subword
vocabularies were used for every language. In Table 3 we present the
vocabulary sizes of the data used to train our PB-SMT and NMT systems.

| no BPE | with BPE
---|---|---
Lang. pair | EN | FR/ES | EN | FR/ES
EN-FR/FR-EN | 113,132 | 131,104 | 47,628 | 48,459
EN-ES/ES-EN | 113,692 | 168,195 | 47,639 | 49,283
Table 3: Vocabulary sizes. For completeness we also present the vocabulary
size without BPE, i.e. the number of unique words in the corpora.

The quality of our MT systems is evaluated on the test set using standard
evaluation metrics – BLEU (Papineni et al., 2002) (as implemented in SacreBLEU
(Post, 2018) and TER (Snover et al., 2006) (as implemented in MultEval (Clark
et al., 2011)). Our evaluation scores are presented in Table 4.

| English as source | English as target
---|---|---
|  |  EN$\rightarrow$FR |  EN$\rightarrow$ES |  FR$\rightarrow$EN |  ES$\rightarrow$EN
| System |  BLEU$\uparrow$ |  TER$\downarrow$ |  BLEU$\uparrow$ |  TER$\downarrow$ |  BLEU$\uparrow$ |  TER$\downarrow$ |  BLEU$\uparrow$ |  TER$\downarrow$
1080Ti | LSTM | 34.2 | 50.9 | 38.2 | 45.3 | 34.6 | 48.2 | 38.1 | 44.7
TRANS | 37.2 | 48.7 | 40.9 | 43.4 | 37.0 | 46.4 | 41.3 | 41.4
P100 | LSTM | 34.1 | 50.7 | 37.3 | 47 | 34.9 | 48 | 38.5 | 44.4
TRANS | 37.4 | 48.4 | 40.9 | 43.3 | 37.3 | 47 | 41.6 | 42.5
Table 4: Quality evaluation scores for our MT systems. TRANS denotes
Transformer systems.

We computed pairwise statistical significance using bootstrap resampling
(Koehn, 2004) and a 95% confidence interval. The results shown in Table 4 are
all statistically significant based on 1000 iterations and samples of 100
sentences. All metrics show the same performance trends for all language
pairs: Transformer (TRANS) outperforms all other systems, followed by PB-SMT,
and LSTM.

###  4.3 GPU power consumption

To measure the consumption of power, memory and core utilisation, as well as
the heat generated during the training and use of an NMT model, several tools
can be exploited, e.g. `experiment-impact-
tracker`191919https://github.com/Breakend/experiment-impact-tracker(Henderson
et al., 2020), “Weights and Biases”202020https://wandb.ai, as well as NVidia’s
device monitoring: `nvidia-smi dmon`. We decided to stick to the mainstream
NVIDIA System Management Interface program (`nvidia-smi`)
212121https://developer.download.nvidia.com/compute/DCGM/docs/nvidia-
smi-367.38.pdf which does not require additional installation or setup. It is
a tool by NVidia for monitoring and management of major lines of their GPUs.
The `nvidia-smi` is cross-platform and supports all standard NVidia driver-
supported Linux distributions as well as 64bit versions of the Windows
operating system. In this work we used the `nvidia-smi dmon` command to
monitor all GPUs during the training and inference processes. This command
displays one line of monitoring data per monitoring cycle. The default range
of metrics includes power usage (or power draw – the last measured power draw
for the entire board reported in watts), temperature, SM clocks, memory clocks
and utilization values for SM, memory, encoder and decoder. By default we
monitor all GPUs during the training process as training is distributed over
all four 1080Ti or three P100 GPUs. However, at inference time we use only one
GPU. As such, we only monitor and report values for that specific GPU during
inference.

We ought to note that the `experiment-impact-tracker`, which internally
invokes `nvidia-smi`, is a much more elaborate tool, designed to ease the
collection and analysis of data. However, it utilizes Intel’s RAPL
interface.222222Intel’s Running Average Power Limit or _RAPL_(Intel, 2009)
interface exposes power meters and power limits. It also allows power limits
on the CPU and the DRAM to be set. As part of the Linux kernel, RAPL is
read/write protected to normal users and the information it generates is
readable by super users only. As such, the `experiment-impact-tracker`, which
utilizes RAPL would not collect any metrics from the CPU. We did not find any
work-around but to disable this functionality. That, in turn, resulted in only
collecting metrics from `nvidia-smi` and therefore we did not employ the tool.
However, along with the resource monitoring, this tool can compute the CO2
emissions based on the compute time and the energy consumed.

The CO2 emissions generated by each experiment are computed based on Equation
1 (Strubell et al., 2019; Henderson et al., 2020).

$CO2_{e}missions=\frac{PUE*kWh*I^{CO2}}{1000}$ (1)

where $PUE$, which stands for Power Usage Effectiveness, defines how
efficiently data centres use energy, i.e. it accounts for the additional
energy required to support the compute infrastructure; $kWh$ are the total
kilowatt hours consumed; $I^{CO2}$ is the CO2 intensity. The $PUE$ and
$I^{CO2}$ values vary greatly and depend on a large set of factors. In our
computations, similar to (Henderson et al., 2020) we use averages reported on
a global or national level. In particular, we use the global average $PUE$
value reported by Ascierto and Lawrence (2020) of $1.59$.232323It is worth
noting that the average $PUE$ value follows a descending trend until 2018 when
the average $PUE$ is $1.58$ (as reported in the Uptime Institute global data
center survey for 2018) and used in the experiment-impact-tracker tool.
However, in 2019 the global average $PUE$ is $1.67$, surpassing 2013; in 2020,
the value is still high – $1.59$ (see p. 10 of the 2020 survey (Ascierto and
Lawrence, 2020)).

The $kWh$ are computed as the sum of all power (in watts) drawn per second per
GPU which is then divided by $3.600.000$. We ought to note that during
training on the P100 workstation and translation (both quantized and not
quantized) on both workstations we collected the readings of `nvidia-smi`
every second; for the training on the 1080Ti workstation, the readings were
collected every five seconds. In order to make the comparison more realistic,
for the latter case we interpolated the missing values instead of simply
averaging for every 5-second interval.242424We used SciPy’s interpolate
https://docs.scipy.org/doc/scipy/reference/interpolate.html

The carbon intensity is a measure of how much CO2 emissions are produced per
kilowatt hour of electricity consumed.252525https://carbonintensity.org.uk/.
It is a constantly-changing value and as such we consider an average collected
over the first half of 2020. The electricityMap
project262626https://www.electricitymap.org/ (Tranberg et al., 2019) collects
and distributes elaborate information related to electricity production and
consumption, including carbon intensity per country and per specific
timestamp. Since our research is conducted for academic purposes
‘electricityMap’ gave us access to historical data from which we calculated
the mean CO2 intensity and standard deviation for both Ireland (IE) and the
Netherlands (NL), where our workstations are located (the 1080Ti workstation
is located in Ireland, the P100 workstation – in the Netherlands. The values
are as follows: IE: 229.8718  $\pm$ 77.4026; NL: 399.3685  $\pm$ 31.9251.

##  5 Power consumption and CO2 footprint

#### Train and translation times

We first present the run times for each experiment and the training step at
which the training was terminated (using early stopping). The run times are
shown in Table 5. Although we are using the exact same data, software,
hyperparameter values272727which are not dependent on the number of GPUs and
random seeds, the training processes on the different hardware deviate due to
differences in the hardware, number of GPUs and the NVidia
driver.282828https://pytorch.org/docs/stable/notes/randomness.html

| 1080Ti | P100
---|---|---
System | Elapsed | # steps | time/ | Elapsed | # steps | time/
| time (h) | x 1 000 | step | time (h) | x 1 000 | step
LSTM | EN-FR | 25.08 | 160 | 0.16 | 18.83 | 145 | 0.13
EN-ES | 28.41 | 180 | 0.16 | 16.66 | 130 | 0.13
FR-EN | 23.51 | 145 | 0.16 | 13.95 | 105 | 0.13
ES-EN | 24.38 | 145 | 0.17 | 19.21 | 145 | 0.13
TRANS | EN-FR | 5.22 | 14.5 | 0.36 | 5.06 | 11 | 0.46
EN-ES | 6.6 | 19.5 | 0.34 | 6.06 | 13 | 0.47
FR-EN | 6.15 | 17.5 | 0.35 | 4.85 | 11 | 0.44
ES-EN | 6.36 | 19 | 0.33 | 6.2 | 13 | 0.48
Table 5: Train time in hours, number of steps and average train time for one
step.

The values in Table 5 indicate a larger train time for the LSTM models
compared to the the TRANS models, for both the 1080Ti and the more performant
P100. We furthermore observe an overall larger train time for the 1080Ti.
However, we ought to note that the average time per step for the TRANS models
is larger for the P100. We associate these differences with the number of GPUs
on which we trained the models – 4 for the 1080Ti and 3 for the P100.

In Table 6 we present the elapsed time during translation of our test set.
Recall that the test set is intentionally large. The translation is conducted
on a single GPU and no file is translated by two models at the same time in
order to avoid any I/O delays.

System | 1080Ti | P100
---|---|---
LSTM | EN-FR | 1.52 | 1.84
EN-ES | 1.38 | 1.69
FR-EN | 1.48 | 1.79
ES-EN | 1.34 | 1.62
TRANS | EN-FR | 2.63 | 3.01
EN-ES | 2.48 | 2.80
FR-EN | 2.47 | 3.18
ES-EN | 2.45 | 2.69
Table 6: Translation time in hours.

Contrary to what is ought to be expected, from the experiments it can be
observed that the translation times on the 1080Ti machine are constantly lower
than the ones on the P100.

#### Power consumption and CO2 emissions.

Below we summarise the measurements collected through the `nvidia-smi` tool
and analyse the ecological footprint in terms of CO2 emissions. In Table 7 and
Table 8 we present the power consumption and carbon dioxide emissions at train
and at translation time, respectively.

| 1080Ti | P100
---|---|---
| Elapsed | Avg. |  | CO2 | Elapsed | Avg. |  | CO2
System | time | power | kWh | (kg) | time | power | kWh | (kg)
| (h) | (W) |  |  | (h) | (W) |  |
LSTM | EN-FR | 25.08 | 142.05 | 14.07 |  5.14  $\pm$ 1.73 | 18.83 | 115.09 | 6.33 |  4.02  $\pm$ 0.32
EN-ES | 28.41 | 140.88 | 15.79 |  5.77  $\pm$ 1.94 | 16.66 | 113.99 | 5.54 |  3.52  $\pm$ 0.28
FR-EN | 23.51 | 141.85 | 13.15 |  4.81  $\pm$ 1.62 | 13.95 | 113.48 | 4.63 |  2.94  $\pm$ 0.24
ES-EN | 24.38 | 139.90 | 13.44 |  4.91  $\pm$ 1.65 | 19.21 | 113.91 | 6.37 |  4.04  $\pm$ 0.32
TRANS | EN-FR | 5.22 | 176.70 | 3.64 |  1.33  $\pm$ 0.45 | 5.06 | 153.47 | 2.27 |  1.44  $\pm$ 0.12
EN-ES | 6.60 | 176.54 | 4.60 |  1.68  $\pm$ 0.56 | 6.06 | 152.08 | 2.69 |  1.71  $\pm$ 0.14
FR-EN | 6.15 | 176.64 | 4.29 |  1.56  $\pm$ 0.53 | 4.85 | 151.43 | 2.15 |  1.37  $\pm$ 0.11
ES-EN | 6.36 | 179.48 | 4.50 |  1.64  $\pm$ 0.55 | 6.20 | 151.59 | 2.74 |  1.74  $\pm$ 0.14
Table 7: Run-time, power draw and CO2 emissions (kg) at train time.  | 1080Ti | P100
---|---|---
System | Elapsed | Average | kWh | CO2 | Elapsed | Average | kWh | CO2
| time (h) | power (w) |  | (kg) | time (h) | power (w) |  | (kg)
LSTM | EN-FR | 1.52 | 157.80 | 0.22 |  0.08  $\pm$ 0.03 | 1.84 | 90.50 | 0.16 |  0.10  $\pm$ 0.01
EN-ES | 1.38 | 158.51 | 0.20 |  0.07  $\pm$ 0.02 | 1.69 | 89.06 | 0.15 |  0.10  $\pm$ 0.01
FR-EN | 1.34 | 153.43 | 0.19 |  0.07  $\pm$ 0.02 | 1.79 | 93.14 | 0.16 |  0.10  $\pm$ 0.01
ES-EN | 1.48 | 154.98 | 0.21 |  0.08  $\pm$ 0.03 | 1.62 | 89.35 | 0.14 |  0.09  $\pm$ 0.01
TRANS | EN-FR | 2.63 | 188.75 | 0.45 |  0.16  $\pm$ 0.06 | 3.01 | 104.52 | 0.31 |  0.20  $\pm$ 0.02
EN-ES | 2.48 | 170.02 | 0.38 |  0.14  $\pm$ 0.05 | 2.80 | 102.71 | 0.28 |  0.18  $\pm$ 0.01
FR-EN | 2.47 | 193.34 | 0.47 |  0.17  $\pm$ 0.06 | 3.18 | 100.93 | 0.31 |  0.20  $\pm$ 0.02
ES-EN | 2.45 | 175.60 | 0.42 |  0.15  $\pm$ 0.05 | 2.69 | 104.35 | 0.28 |  0.18  $\pm$ 0.01
Table 8: Run-time, average power draw and CO2 emissions (kg) at translation
time.

The average power draw, as shown in Table 7 and Table 8, is computed as the
sum of all readings (in watt) for all GPUs divided by the number of readings.
That is, it is a value that does not take into account the number of GPUs.
Comparing the average power draw of LSTM and Transformer (TRANS) models, we
observe that for both workstations (both types of GPUs), LSTMs consume less
power (at a time) than TRANS models, during both training and translation.
However, due to the excessively larger train time of LSTM models, their
overall power consumption (reported as kWh) is much larger than that of TRANS
models. As such, in our experiments they lead to a larger CO2 emission, as
computed by Equation 1 for all 4 1080Ti or 3 P100 GPUs: in the case of the
1080Ti, between 2.99 times (4.91 kg to 1.64 kg) for the ES-EN language pair
and 3.86 times (5.14 kg to 1.64 kg) for EN-FR; in the case of the P100,
between 2.06 times (4.63 kg to 2.15 kg) for FR-EN and 2.79 times (6.33 kb to
2.27 kg) for EN-FR. These results indicate that _for the same data and optimal
hyperparameters, 292929As recommended in the literature. LSTMs have a larger
ecological footprint at train time._ However, at translation time the
observations are reversed – as suggested by the lower average power
consumption for LSTM models and the larger translation time for TRANS models,
the CO2 emission regarded to LSTM models is two times lower than that of TRANS
models (consistent over all language pairs and GPU types). This would imply
that after a certain time of usage, our LSTM models are “greener” than the
TRANS models. In particular, for the 1080Ti GPUs, our LSTMs would become
“greener” after 10 to 40 days and for the P100 – after 9 and 12
days.303030This is the time that would be required for the two types of
architectures to consume the same power (in watt) including training (on the
same data) and translation. Since LSTM consumes less power at translation time
per time unit (e.g. second), any further operation of both LSTM and TRANS
models would lead to an overall lower energy consumption by LSTM models.

When we compare the two types of GPUs, we notice that the P100 is much less
power-demanding than the 1080Ti with an average power draw between 113.91 (ES-
EN LSTM) and 115.09 (EN-FR LSTM) versus between 139.9 (ES-EN LSTM) and 142.05
(EN-FR LSTM), and between 151.43 (FR-EN TRANS) and 153.47 (EN-FR TRANS) versus
between 176.54 (EN-ES TRANS) and 179.48 (FR-EN TRANS).313131At train time.
This is also reflected in the overall power consumption to which a large
contributing factor is train time which is much smaller with the P100
workstation. That is, at train time, the power consumption for the P100 GPU
workstation is almost three times smaller than that of the 1080Ti machine for
LSTM models and around two times smaller for TRANS models. At translation
time, while still leading to a lower power consumption, mainly because of the
lower average power draw, the differences are much smaller.

A special attention needs to be paid at the impact of the national carbon
intensity. For Ireland it is much lower than that of the Netherlands. These
differences have a substantial impact on the carbon emissions – the TRANS
models trained on the P100 workstation in the Netherlands have a larger
footprint than in the 1080Ti case in three out of the four cases (EN-FR, EN-ES
and ES-EN); at translation time for all models running on the P100 machine,
the footprint is larger than in the case of the 1080Ti.

Analysing the data from Tranberg et al. (2019) we see that the power for
Ireland originates primarily from renewable sources while that is not the case
for the Netherlands. Figure 2 illustrates this for the period between
01/01/2020 and 01/06/2020 on a monthly basis.

Figure 2: Power origin distribution.

#### The impact

Comparing different types of workstations, different NMT architectures and
language pairs as well as the regional factors related to carbon emissions
gives us an understanding of the conditions under which a certain amount of
electricity is consumed and the factors that we should consider to optimize
our NMT-related ecological footprint. To put matters into perspective, we
compare our models and the related measurements to common everyday devices. We
assess (i) the power draw and (ii) the carbon emissions. For the former, we
collect the power consumption as indicated by _Caruna_
323232https://www.caruna.fi, a Finnish electricity distribution company; for
the latter, we gather data from _Carbon Footprint_
333333https://www.carbonfootprint.com, a company that specialises in carbon
emission assessments, LifeCycle Analysis, environmental strategy/planing,
etc., and indicates an estimated times or hours of use of common devices.

Figure 3: Average power draw.

Figure 4: Estimated yearly CO2 emission in kg.

Figure 4 compares the average power draw of the workstations during training
and translation for the different models (marked in light blue)343434These are
the same values indicated in Table 7 and Table 8. to common devices (marked in
light orange). The common devices are typical household appliances; we have
excluded larger, more power demanding devices such as electric shower (8 750
W) since our range, for practical reasons, spans to maximum 1 200 W. From
Figure 4, it is clear that the GPUs are less power demanding than some common
household appliances. That is, at a given moment in time, a GPU board draws
fewer watts of electricity than some common household appliances (e.g. a
microwave, an electric mower, etc.).

However, the usage (or utilisation) time of a GPU is much higher than that of
a microwave and as such the power consumption (computed in kWh) as well as the
carbon emissions are significantly different. When computing the carbon
emissions of a device for a certain period of time, one must take into account
the utilisation rate of that device for the given time period, i.e. the number
of hours or times the device is used within the considered time interval to
the time it is idle. According to (Nadjaran Toosi et al., 2017) and (Doyle and
Bashroush, 2020), in data centers, the average utilization rate ranges between
20% and 40%. However, this is not an ideal scenario for a GPU workstation as
the return on investment would be low due to the high costs of GPUs.353535A
modern workstation with 3 x Nvidia Tesla V100 costs approximately €30 000; a
workstation with 4 x Nvidia RTX3060 is approximately €7 000. As an ideal and
rather extreme scenario, we assume that in an industry environment a GPU
workstation is utilised 100%. This, in turn, will indicate the maximum amount
of CO2 emissions if GPU workstations of the types we considered are used to
train and translate with our models. To compute the impact, in terms of CO2
emissions, we first estimate the carbon emission per unit of time the product
of power draw (in watts), utilisation rate373737Values are based on data from
_Carbon Footprint_ 363636https://www.carbonfootprint.com and carbon intensity
(for both Ireland and the Netherlands) and second we extrapolate the CO2
emissions over a year based on the recorded values for the given time period.
Our estimates are shown in Figure 4.

Here, however, we ought to note that while at train time each workstation
utilizes all of its GPUs, that is not the case for the translation. As such,
we advise the reader to consider these as two different types of measurements
– one where all GPUs are utilized (train time) and one where only one GPU is
used (translation time).

From Figure 4 we observe that in one year, at 100% utilization, a GPU
workstation that is used to train simple models, such as ours, could produce
up to $2.500$ kg of CO2.383838One can also multiply the CO2 emissions at
translation time by 4 for the 1080Ti or by 3 for the P100 GPUs and get an
indication about how much CO2 emissions would be generated if a workstation
(with 4 or 3 GPUs, as in our case) is utilized 100% at translation time. That
is, either all GPUs translate using the same model in parallel, or different
models are used at the same time for different translation jobs. This is
approximately equal to the CO2 emissions from the electricity consumption of
two small households in the UK.393939According to data from
https://www.carbonindependent.org.

##  6 Optimizing at inference time through model quantization

In an academic environment, the main use of MT is as an experimentation ground
for research. Thus, it is often the case that many NMT models are trained and
evaluated on a small test set. However, within an industry environment, the
main purpose of an NMT model is to be used to translate as much new content as
possible before updating or rebuilding it. From the evaluation summarised in
Section 5 we see that, at translation time the P100 workstation uses less
energy than a 1080Ti workstation (see Table 8); furthermore at translation
time the P100 workstation uses less energy than both workstations at train
time (see Table 7). And while this is an indication that investing in a
higher-end GPU workstation is better (in terms of electricity consumption and
carbon emissions) we ought to investigate other means to optimize both power
consumption and inference time. With that regards, in this section we
investigate how quantizing the model can improve both energy consumption,
inference time as well as what the impact is on the translation quality.

#### Quantization

Deep neural networks contain a huge amount of parameters (biases, weights)
that are adapted during training to reduce the loss. These are typically
stored as 32-bit floating point numbers. In every forward pass all these
parameters are involved in computing the output of the network. The large
precision requires more memory and processing power than, e.g. integer
numbers. Quantization is the process of approximating a neural network’s
parameters by reducing their precision. A quantized model executes some or all
of the operations on tensors with integers rather than floating point
values.404040https://pytorch.org/docs/stable/quantization.html Quantization is
a term that encapsulates a broad range of approaches to the aforementioned
process: binary quantization (Courbariaux and Bengio, 2016), ternary (Lin et
al., 2016; Li and Liu, 2016), uniform (Jacob et al., 2018) and learned (Zhang
et al., 2018), to mention a few. The benefits of quantization are a reduced
model size and the option to use more high performance vectorized operations,
as well as, the efficient use of other hardware platforms414141For example,
the $2^{nd}$ generation Intel® Xeon® Scale processors incorporate an INT8 data
type acceleration (Intel® DL Boost Vector Neural Network Instructions (VNNI)
(Evarist, 2018)), specifically designed to accelerate neural network-related
computations (Rodriguez et al., 2018).. However, more efficient quantized
models may suffer from worse inference quality. In the field of NMT Bhandare
et al. (2019) quantize trained Transformer models to a lower precision (8-bit
integers) for inference on Intel® CPUs. They investigate three approaches to
quantize the weights of a Transformer model and achieve a drop in performance
of only 0.35 to 0.421 BLEU points (from 27.68 originally). Prato et al. (2020)
investigate a uniform quantization for Transfromer, quantizing matrix
multiplications and divisions (if both the numerator and denominator are
second or higher rank tensors), i.e. all operations that could improve the
inference speed. Their 6-bit quantized EN-DE Transformer base model is more
than 5 times smaller than the baseline and achieves higher BLEU scores. For
EN-FR, the 8-bit quantized model which achieves the highest performance is
almost 4 times smaller than the baseline. With the exception of the 4-bit
fully quantized models and the naive approach, all the rest show a significant
reduction in model size with almost no loss in translation quality (in terms
of BLEU).

The promising results from the aforementioned works motivated us to
investigate the power consumption of quantized versions of our models running
on a GPU. We used the `CTranslate2` tool of OpenNMT.
424242https://github.com/OpenNMT/CTranslate2 As noted in the repository
“CTranslate2 is a fast inference engine for OpenNMT-py and OpenNMT-tf models
supporting both CPU and GPU execution. The goal is to provide comprehensive
inference features and be the most efficient and cost-effective solution to
deploy standard neural machine translation systems such as Transformer
models.”

#### Quality of quantized Transformer models

We quantized our Transformer models to INT8 and INT16 and translated our test
sets on the 1080Ti and P100 workstations. On each workstation, we quantized
the models that were originally trained on that machine. After translation, we
scored BLEU and TER the same way as with our normal models. The quality
results are summarised in Table 9.

|  | EN-FR | EN-ES | FR-EN | ES-EN
---|---|---|---|---|---
GPU | Prec. |  BLEU$\uparrow$ |  TER$\downarrow$ |  BLEU$\uparrow$ |  TER$\downarrow$ |  BLEU$\uparrow$ |  TER$\downarrow$ |  BLEU$\uparrow$ |  TER$\downarrow$
1080Ti | FP32 | 37.2 | 48.7 | 40.9 | 43.4 | 37.0 | 46.4 | 41.3 | 41.4
INT16 | 36.4 | 49.5 | 40.6 | 44.1 | 36.6 | 47.3 | 40.9 | 44.0
INT8 | 36.3 | 49.5 | 40.5 | 44.1 | 36.6 | 47.3 | 40.8 | 44.1
P100 | FP32 | 37.4 | 48.4 | 40.9 | 43.3 | 37.3 | 47 | 41.6 | 42.5
INT16 | 36.4 | 49.1 | 40.7 | 44.0 | 36.5 | 50.6 | 40.5 | 43.8
INT8 | 36.4 | 49.2 | 40.7 | 44.0 | 36.5 | 50.4 | 40.5 | 43.8
Table 9: Evaluation metrics for quantized and baseline Transformer models.
FP32 stands for 32-bit floating point; INT16 or INT8 stand for 16-bit or 8-bit
integer. FP32 is the default, non-quantised model (See Table 4).

#### Energy considerations for quantized Transformer models

We measured the power draw during the translation process with our quantized
models and computed the consumed kWh as well as CO2 emissions per model and
per region. Our results are summarised in Table 10.

| 1080Ti | P100
---|---|---
| Elapsed | Avg. |  | CO2 | Elapsed | Avg. |  | CO2
System | time | power | kWh | (kg) | time | power | kWh | (kg)
| (h) | (W) |  |  | (h) | (W) |  |
INT16 | EN-FR | 5.17 | 130.61 | 0.13 |  0.05 $\pm$ 0.02 | 0.79 | 81.54 | 0.01 |  0.01 $\pm$0.00
INT8 | EN-FR | 4.66 | 115.99 | 0.11 |  0.04 $\pm$ 0.01 | 2.16 | 49.06 | 0.02 |  0.01 $\pm$0.00
INT16 | EN-ES | 4.45 | 158.96 | 0.14 |  0.05 $\pm$ 0.02 | 0.99 | 65.4 | 0.01 |  0.01 $\pm$0.00
INT8 | EN-ES | 4.15 | 124.40 | 0.10 |  0.04 $\pm$ 0.01 | 1.00 | 68.01 | 0.01 |  0.01 $\pm$0.00
INT16 | FR-EN | 4.57 | 139.38 | 0.13 |  0.05 $\pm$ 0.02 | 1.28 | 67.57 | 0.02 |  0.01 $\pm$0.00
INT8 | FR-EN | 4.39 | 107.87 | 0.09 |  0.03 $\pm$ 0.01 | 1.29 | 68.39 | 0.02 |  0.01 $\pm$0.00
INT16 | ES-EN | 4.45 | 131.66 | 0.12 |  0.04 $\pm$ 0.01 | 1.02 | 68.33 | 0.01 |  0.01 $\pm$0.00
INT8 | ES-EN | 4.03 | 117.48 | 0.09 |  0.03 $\pm$ 0.01 | 1.04 | 67.25 | 0.01 |  0.01 $\pm$0.00
Table 10: Run-time, average power draw and CO2 emissions (kg) at translation
time for quantized models.

Comparing these results to the ones in Table 8 (TRANS) we first notice the
increased translation time for the quantized models running on the 1080Ti
machine – from 2.45 to 4.03 (INT8) and 4.45 (INT16) hours for the fastest ES-
EN and from 2.63 to 5.17 (INT16) and 4.66 (INT8) for the slowest EN-FR.
However, due to the lower power draw the overall energy consumption as well as
the CO2 emissions at translation time with these models (on the 1080Ti
workstation) is still lower than for the non-quantized models (on the same
workstation).

When comparing the performance of these models on the P100 workstation we
notice a much lower translation time over all models, even if the difference
between the EN-FR / INT16 model and the non-quantized EN-FR Transformer model
is not so drastic. At the same time the power draw for all models is lower
than for their non-quantized version, leading to a very low electricity
consumption and low carbon emissions.

We ought to note that the time for quantization, i.e. the process of
converting a non-quantized Transformer model into a quantized one is very low
(between 6 and 12 seconds on the P100 workstation). Furthermore, quantization
is very much suitable for inference on CPU. Based on the results in Table 9
and Table 10, the low quantization time and the fact that quantized models can
easily be run on CPU, we would recommend quantized models at translation time
for large-scale translation projects in the pursuit of greener MT.

##  7 Conclusions and future work

In the era of deep learning, neural models are continuously pushing the
boundaries in NLP, MT including. The ever growing volumes of data and the
advanced, larger models keep delivering new state-of-the-art results. A
facilitator for these results are the innovations in general purpose GPU
computing, as well as in the hardware itself, i.e. GPUs. The embarrassingly
parallel processing required for deep learning models is easily distributed on
the thousands of processing cores on a GPUs, making training and inference
with such models much more efficient than on CPUs. However, GPUs are much more
power demanding and as such have a higher environmental impact. In this
chapter we discussed considerations related to the power consumption and
ecological footprint, in terms of carbon emissions associated with the
training and the inference with MT models.

After briefly presenting the evolution of MT and the shift to GPUs as the core
processing technology of (N)MT, we discuss the related work addressing the
issues of power consumption and environmental footprint of computational
models. We acknowledge that the work of colleague researchers and
practitioners, such as Strubell et al. (2019) and Schwartz et al. (2020),
among others, raises awareness about the environmental footprint that deep
learning models have, and we would like to join them in their appeal towards
“greener” AI. This could be achieved through optimizing models through
quantization, as discussed in Section 6, but also, through reusability,
smarter data selection, knowledge distillation and others.

To outline the realistic dimensions of power consumption and environmental
footprint of NMT, we analysed a number of NMT models, running (training or
inference) on two types of GPUs – a consumer GPU card (NVidia GTX 1080Ti),
designed to work on a desktop machine and a workstation GPU (NVidia Tesla
P100) developed for heavy loads of graphics or neural computing. We reported
results for training both LSTM and Transformer models for English-French and
English-Spanish (and vice-versa) language pairs on data from the Europarl
corpus. These models were trained on approximately 1.5M parallel sentences and
used to translate large test sets that include approximately 500.000 sentences
each. Our results and analysis show that, while a Transformer model is much
faster and as such much more power efficient than an LSTM at train time, at
translation time Transformer models are lagging behind LSTM models, in terms
of power consumption, speed as well as carbon emissions. We also note that
using the more expensive P100 is preferable in almost every case. An exception
is the slightly higher translation time, which however, comes with the benefit
of a largely reduced power consumption.

Additionally, we also note the impact of electricity sources on the carbon
emissions by investigating two different countries, each of which has a
different distribution between fossil and renewable energy sources – Ireland
with a larger portion of renewable energy and the Netherlands with a larger
portion of fossil sources.

Together with the aforementioned contributions, we also aim to motivate
researchers to devote time, efforts and investments in developing more
ecological solutions. We are already looking into model reusability, data
selection and filtering, multi-objective optimization of hyperparameters and
other approaches that reduce the environmental footprint of NMT.

## Carbon Impact Statement

This work contributed 50.77 $\pm$11.37kg of CO2eq to the atmosphere and used
111.55 kWh of electricity.

## Acknowledgements

We would like to thank electicityMap.org for their responsiveness to our
queries and providing us with valuable information.

## References

  * Arnold et al. (1994) Douglas Arnold, Lorna Balkan, Siety Meijer, R Lee Humphreys, and Louisa Sadler. 1994\.  An introductory guide.  _Blackwells-NCC, London_.
  * Ascierto and Lawrence (2020) Rhonda Ascierto and Andy Lawrence. 2020.  Uptime institute global data center survey 2020.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015)_ , San Diego, CA, USA.  15pp.
  * Barsh (1990) Russel Barsh. 1990.  Indigenous peoples, racism and the environment.  _Meanjin_ , 49(4):723–731.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency; Association for Computing Machinery: New York, NY, USA_.
  * Bentivogli et al. (2016) Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016.  Neural versus Phrase-Based Machine Translation Quality: a Case Study.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 257–267, Austin, Texas.
  * Bhandare et al. (2019) Aishwarya Bhandare, Vamsi Sripathi, Deepthi Karkada, Vivek Menon, Sun Choi, Kushal Datta, and Vikram Saletore. 2019.  Efficient 8-bit quantization of transformer neural machine language translation model.  _CoRR_ , abs/1906.00532.
  * Britz et al. (2017) Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc Le. 2017.  Massive Exploration of Neural Machine Translation Architectures.  In _Proceedings of the Association for Computational Linguistics (ACL)_ , pages 1442–1451, Vancouver, Canada.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_ , pages 1724–1734, Doha, Qatar.
  * Clark et al. (2011) Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011.  Better hypothesis testing for statistical machine translation: Controlling for optimizer instability.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 176–181, Portland, Oregon, USA. Association for Computational Linguistics.
  * Courbariaux and Bengio (2016) Matthieu Courbariaux and Yoshua Bengio. 2016.  Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1.  _CoRR_ , abs/1602.02830.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (NAACL-HLT 2019)_ , volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, USA.
  * Doyle and Bashroush (2020) J. Doyle and R. Bashroush. 2020.  Case studies for achieving a return on investment with a hardware refresh in organizations with small data centers.  _IEEE Transactions on Sustainable Computing_ , pages 1–1.
  * Ethayarajh and Jurafsky (2020) Kawin Ethayarajh and Dan Jurafsky. 2020.  Utility is in the eye of the user: A critique of NLP leaderboards.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 4846–4853. Association for Computational Linguistics.
  * Evarist (2018) Fomenko Evarist. 2018.  Understanding the new vector neural network instructions.
  * Gordon et al. (2018) Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. 2018.  Morphnet: Fast & simple resource-constrained structure learning of deep networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1586–1595.
  * Henderson et al. (2020) Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. 2020.  Towards the systematic reporting of the energy and carbon footprints of machine learning.  _Journal of Machine Learning Research_ , 21(248):1–43.
  * Hutchins (2005a) John Hutchins. 2005a.  The history of machine translation in a nutshell.  _Retrieved December_ , 20(2009):1–1.
  * Hutchins (2005b) John Hutchins. 2005b.  Towards a definition of example-based machine translation.  In _Machine Translation Summit X, Second Workshop on Example-Based Machine Translation_ , pages 63–70. Citeseer.
  * Intel (2009) IA Intel. 2009.  Intel architecture software developer’s manual volume 3: System programming guide.
  * Jacob et al. (2018) Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_ , pages 2704–2713. IEEE Computer Society.
  * Kenny (2005) Dorothy Kenny. 2005.  Parallel corpora and translation studies: old questions, new perspectives? reporting that in gepcolt: a case study.  _Meaningful texts: The extraction of semantic information from monolingual and multilingual corpora_ , pages 154–165.
  * Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980.
  * Koehn (2004) Philipp Koehn. 2004.  Statistical significance tests for machine translation evaluation.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP2004)_ , pages 388–395, Barcelona, Spain.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A Parallel Corpus for Statistical Machine Translation.  In _Proceedings of The Tenth Machine Translation Summit (MT Summit 2005)_ , pages 79–86, Phuket, Thailand.
  * Koehn (2020) Philipp Koehn. 2020.  _Neural machine translation_.  Cambridge University Press.
  * Lacoste et al. (2019) Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019\.  Quantifying the carbon emissions of machine learning.  _arXiv preprint arXiv:1910.09700_.
  * Lepikhin et al. (2020) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020.  Gshard: Scaling giant models with conditional computation and automatic sharding.  _arXiv preprint arXiv:2006.16668_.
  * Li and Liu (2016) Fengfu Li and Bin Liu. 2016.  Ternary weight networks.  _CoRR_ , abs/1605.04711.
  * Lin et al. (2016) Zhouhan Lin, Matthieu Courbariaux, Roland Memisevic, and Yoshua Bengio. 2016.  Neural networks with few multiplications.  In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
  * Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015.  Effective approaches to attention-based neural machine translation.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_ , pages 1412–1421. The Association for Computational Linguistics.
  * Microsoft (2020) Microsoft. 2020.  Turing-nlg: A 17 billion parameter language model by microsoft.
  * Molchanov et al. (2016) Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2016.  Pruning convolutional neural networks for resource efficient inference.  _arXiv preprint arXiv:1611.06440_.
  * Nadjaran Toosi et al. (2017) Adel Nadjaran Toosi, Chenhao Qu, Marcos Dias de Assunção, and Rajkumar Buyya. 2017.  Renewable-aware geographical load balancing of web applications for sustainable data centers.  _Journal of Network and Computer Applications_ , 83:155–168.
  * Nagao (1984) Makoto Nagao. 1984.  A framework of a mechanical translation between japanese and english by analogy principle.  In _Proc. of the International NATO Symposium on Artificial and Human Intelligence_ , page 173–180, USA. Elsevier North-Holland, Inc.
  * Open AI (2018) Dario Amodei Danny Hernandez Open AI. 2018.  Ai and compute.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: A Method for Automatic Evaluation of Machine Translation.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL 2002)_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Poibeau (2017) Thierry Poibeau. 2017.  _Machine translation_.  MIT Press.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Belgium, Brussels. Association for Computational Linguistics.
  * Prato et al. (2020) Gabriele Prato, Ella Charlaix, and Mehdi Rezagholizadeh. 2020.  Fully quantized transformer for machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1–14, Online. Association for Computational Linguistics.
  * Pulido (2016) Laura Pulido. 2016.  Flint, environmental racism, and racial capitalism.  _Capitalism Nature Socialism_ , 27(3):1–16.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raina et al. (2009) Rajat Raina, Anand Madhavan, and Andrew Y. Ng. 2009.  Large-scale deep unsupervised learning using graphics processors.  In _Proceedings of the 26th Annual International Conference on Machine Learning_ , ICML ’09, page 873–880.
  * Rodriguez et al. (2018) Andres Rodriguez, Eden Segal, Etay Meiri, Evarist Fomenko, Y Jim Kim, Haihao Shen, and Barukh Ziv. 2018.  Lower numerical precision deep learning inference and training.  _Intel White Paper_ , 3:1–19.
  * Schwartz et al. (2020) Roy Schwartz, Jesse Dodge, N. A. Smith, and Oren Etzioni. 2020.  Green ai.  _Communications of the ACM_ , 63:54 – 63.
  * Shterionov et al. (2019) Dimitar Shterionov, Félix Do Carmo, Joss Moorkens, Eric Paquin, Dag Schmidtke, Declan Groves, and Andy Way. 2019.  When less is more in neural quality estimation of machine translation. an industry case study.  In _Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks_ , pages 228–235, Dublin, Ireland. European Association for Machine Translation.
  * Snover et al. (2006) Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006.  A study of translation edit rate with targeted human annotation.  In _Proceedings of the 7th Conference of the Association for Machine Translation of the Americas (AMTA 2006). Visions for the Future of Machine Translation_ , pages 223–231, Cambridge, Massachusetts, USA.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  * Strubell et al. (2018) Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018\.  Linguistically-informed self-attention for semantic role labeling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5027–5038, Brussels, Belgium. Association for Computational Linguistics.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.  Sequence to sequence learning with neural networks.  In _Proceedings of Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems_ , pages 3104–3112, Montreal, Quebec, Canada.
  * Tranberg et al. (2019) Bo Tranberg, Olivier Corradi, Bruno Lajoie, Thomas Gibon, Iain Staffell, and Gorm Bruun Andresen. 2019.  Real-time carbon accounting method for the european electricity markets.  _Energy Strategy Reviews_ , 26:100367.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 5998–6008. Curran Associates, Inc., Red Hook, NY, USA.
  * Veniat and Denoyer (2018) Tom Veniat and Ludovic Denoyer. 2018.  Learning time/memory-efficient deep architectures with budgeted super networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3492–3500.
  * Zaremba et al. (2014) Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014.  Recurrent neural network regularization.  _CoRR_ , abs/1409.2329.
  * Zhang et al. (2018) Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. 2018.  Lq-nets: Learned quantization for highly accurate and compact deep neural networks.  In _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VIII_ , volume 11212 of _Lecture Notes in Computer Science_ , pages 373–390. Springer.