# Uncertainty in Contrastive Learning: On the Predictability of Downstream
Performance

Shervin Ardeshir
Netflix
<EMAIL_ADDRESS>
&Navid Azizan
Massachusetts Institute of Technology
<EMAIL_ADDRESS>

###### Abstract

The superior performance of some of today’s state-of-the-art deep learning
models is to some extent owed to extensive (self-)supervised _contrastive_
pretraining on large-scale datasets. In contrastive learning, the network is
presented with pairs of positive (similar) and negative (dissimilar)
datapoints and is trained to find an embedding vector for each datapoint,
i.e., a representation, which can be further fine-tuned for various downstream
tasks. In order to safely deploy these models in critical decision-making
systems, it is crucial to equip them with a measure of their _uncertainty_ or
_reliability_. However, due to the pairwise nature of training a contrastive
model, and the lack of absolute labels on the output (an abstract embedding
vector), adapting conventional uncertainty estimation techniques to such
models is non-trivial. In this work, we study whether the uncertainty of such
a representation can be quantified for a single datapoint in a meaningful way.
In other words, we explore if the downstream performance on a given datapoint
is predictable, directly from its pre-trained embedding. We show that this
goal can be achieved by directly estimating the distribution of the training
data in the embedding space and accounting for the local consistency of the
representations. Our experiments show that this notion of uncertainty for an
embedding vector often strongly correlates with its downstream accuracy.

##  1 Introduction

Uncertainty estimation is an imperative for safe deployment of deep neural
networks in critical decision-making systems. While deep learning approaches
are capable of finding useful representations that have demonstrably enabled
breakthroughs in a wide variety of tasks, one cannot wishfully assume that
their predictions will always be accurate when queried on various inputs.
There have been many examples of these systems making wrong predictions, which
in some cases have led to fatal accidents (NTS, 2017; Varshney and Alemzadeh,
2017) and unacceptable errors (Guynn, 2015). Many such failures may be
prevented if the system could supplement its predictions with a level of
uncertainty or confidence in those predictions (Dietterich, 2017), which is
crucial for building societal trust in such systems. Besides safety,
uncertainty estimation is also needed as a part of certain learning algorithms
(Hüllermeier and Waegeman, 2021), e.g., for uncertainty reduction in active
learning (Aggarwal et al., 2014).

To that end, a large body of work on uncertainty estimation for deep neural
networks has emerged over the past few years (see, e.g., (Blundell et al.,
2015; Gal and Ghahramani, 2016; Lakshminarayanan et al., 2017; Loquercio et
al., 2020; Sharma et al., 2021; Osband et al., 2021)). While there are many
sources of uncertainty, in Bayesian modeling, they are often categorized into
two types: _aleatoric_ and _epistemic_ 111Aleatoric uncertainty relates to
chance (Latin: _alea_ $\leftrightarrow$ dice) and epistemic uncertainty
relates to knowledge (ancient Greek: _episteme_ $\leftrightarrow$ knowledge)
Osband et al. (2021) (Kiureghian and Ditlevsen, 2009). Distinguishing between
these two types of uncertainties in deep learning has been recently advocated
for in the literature (Kendall and Gal, 2017). In particular, aleatoric
uncertainty refers to the noise inherent in the observations, while epistemic
uncertainty captures the uncertainty in the model. Aleatoric uncertainty
cannot be reduced even if more data is collected (e.g., sensor noise), while
epistemic uncertainty, which accounts for the model’s ignorance, can, in
principle, be reduced with more data (Hüllermeier and Waegeman, 2021; Kendall
and Gal, 2017).

Contrastive learning is a powerful approach to representation learning, which
is responsible for the success of many state-of-the-art deep neural networks
(Chen et al., 2020; Khosla et al., 2020; Wu et al., 2018; Hénaff et al., 2020;
Oord et al., 2018; Tian et al., 2020; Hjelm et al., 2018; He et al., 2020).
During a typical training regime of a contrastive model, pairs of datapoints
are provided as positives or negatives; the contrastive objective then aims to
find a data representation in which the positive pairs “attract,” (i.e., fall
close to each other with an appropriate notion of distance) and the negative
pairs “repulse” each other in the embedding space. This approach tends to
yield rich and robust representations of the data, which may be further used
or fine-tuned for downstream tasks. Depending on the availability of labeled
data, the contrastive pre-training phase may be performed in a supervised
(Khosla et al., 2020) or self-supervised (Chen et al., 2020) fashion.

Despite the recent developments in uncertainty quantification, the majority of
the literature has been focused on supervised settings, in which a single
input is mapped to an absolute target value. Such approaches are not readily
applicable to contrastive models, as the model’s prediction on a single
datapoint is an abstract embedding vector. Nonetheless, such approaches are
applicable for measuring the (un)certainty of the model on a pair of
datapoints, treating contrastive models as binary classifiers. To that end, a
Bayesian metric learning framework was proposed by Wang and Tan (2017) to
construct a robust estimation of the distance, given a pair of datapoints.
Measuring the uncertainty of a metric learning model, given a pair of
datapoints, was further explored by Qian et al. (2018). A relatively close
work to ours is that of Oh et al. (2018), in which the aleatoric embedding
uncertainty is evaluated in terms of instance retrieval, which is the
objective the model is directly trained for. Unlike these works, we aim to
measure the uncertainty/reliability of the embedding for a single datapoint,
in such a way that is predictive of its _downstream performance_. The closest
works to ours are recent works of Zhang et al. (2021); Wu and Goodman (2020),
both of which tackle the problem of uncertainty estimation in the case of a
contrastive objective. The key difference between our work and theirs is two
folds: First, unlike Zhang et al. (2021); Wu and Goodman (2020), in our setup,
a pre-trained model is given to us as a black box, for which the uncertainty
of a datapoint should be estimated. In the aforementioned work, the model is
being pre-trained using a modified objective function, to incorporate the
notion of uncertainty. Our work, on the contrary, is a post-processing step on
a pre-trained black-box contrastive model. Second, in our work, the notion of
certainty for a datapoint is defined as its downstream performance, as opposed
to the work of Zhang et al. (2021); Wu and Goodman (2020), in which a small
variance of the embedding is indicative of certainty.

Figure 1: We are given a contrastive model $f$, which is pre-trained in a
supervised or self-supervised manner on a training dataset. Given a test image
$x$, we measure the reliability of its resulting embedding $f(x)$ using
notions such as embedding-variance (given different augmentations), and the
distribution of the training data embeddings, to quantify this reliability. We
show this reliability is not only capable of detecting out of distribution
samples (epistemic uncertainty), but also predictive of the performance on a
datapoint in a downstream task.

In this paper, we explore the possibility of capturing the reliability of an
embedding resulting from a (black box) pre-trained contrastive model, in terms
of predicting downstream performance. Figure 1 shows an overview of our setup.
A back-box model $f$ is pre-trained with a contrastive objective, on a
training dataset, resulting in an embedding vector representing each
datapoint. The goal is to study whether there are any notions of uncertainty
for an embedding vector which is indicative of its reliability, i.e. how it
would later perform downstream. Given the non-triviality of predicting
downstream performance solely from a pre-trained embedding vector, we explore
the possibility of such prediction using a few intuitive measures. To that
end, given an input and a pre-trained model, we measure the reliability of the
resulting embedding in three aspects: (1) How certain the model is about the
location of an embedding vector. This is computed by introducing variations to
the input datapoint and measuring variations in its embedding vector. (2) How
familiar the model is with that area of the embedding space. In other words,
has the model seen training examples with similar embeddings. This notion is
computed by directly estimating the distribution of the embedding vectors of
the training data. (3) How well does the model perform in that region of the
embedding space. This is measured by calculating the local retrieval
performance of the model. We study whether these intuitive notions
meaningfully correlate with the downstream performance on a given input.

##  2 Framework and Proposed Method

In this section, we describe our framework and how our various notions of
uncertainty/reliability are constructed. Let us consider a model
$f:\mathbb{R}^{n}\to\mathcal{S}^{m-1}$, which maps an $n$-dimensional input
datapoint (e.g., an image) $x$ to the $\ell_{2}$-normalized $m$-dimensional
feature vector $f(x)$ (on the unit hypersphere). Given the model $f$, we aim
to measure the reliability of the embedding vector $f(x)$ for any given input
$x$. As discussed earlier, we do so based on quantifying the uncertainty in
the location of the point in the embedding space as well as the consistency of
the model’s prediction in that region.

###  2.1 Per-Sample Feature Variation: $\delta$

The first notion of uncertainty that we define aims to capture how certain the
model is about the location of an embedding vector. Given a set of
variations/transformations222These are often referred to as data
“augmentation” techniques because they are used for augmenting the dataset.
Geometric transformations, flipping, color modification, cropping, rotation,
noise injection and random erasing are among the common ones (Shorten and
Khoshgoftaar, 2019). $\\{T_{1},T_{2},\dots,T_{l}\\}$ used in the training of
the contrastive model, we measure the variation across
$\\{z_{1},z_{2},\dots,z_{l}\\}$, where $z_{i}=f(T_{i}(x))$ is the embedding
vector corresponding to the $i$-th transformation of the input $x$. More
specifically, we define $\delta({x})$ as the sum of the variances for
different dimensions, i.e., the trace of the sample covariance matrix for the
observation vectors $\\{z_{1},z_{2},\dots,z_{l}\\}$. Our experiments show that
this simple quantity often meaningfully predicts the reliability of the
embedding, as measured by the performance in a downstream task. An important
characteristic of this metric is that it does not require access to the
training data and would work on any black-box model. Note that the underlying
assumption here is that the downstream task is invariant to the pre-training
data transformations (augmentations).

###  2.2 Embedding Distribution Estimation

This notion is based on estimating the distribution of the embedding, which we
refer to as $p_{\mathrm{emb}}$. This probability distribution captures the two
key features of density and consistency over the unit hypersphere embedding
space.

| Model |  Training Inputs |  Training Labels |  Downstream Classifier
---|---|---|---|---
Per-Sample Feature Variation: $\delta$ (2.1) | ✓ | $\times$ | $\times$ | $\times$
Embedding Density: $p_{\mathrm{emb}}$ (2.2.1) | ✓ | ✓ | $\times$ | $\times$
Embedding Consistency: $p_{\text{emb-ens}}$ (2.2.2) | ✓ | ✓ | $\times$ | $\times$
Ensembled Embedding Density: $p^{k,\tau}_{\mathrm{emb}}$ (2.3) | ✓ | ✓ | ✓ | $\times$
Ensembled Embedding Consistency: $p^{k,\tau}_{\text{emb-ens}}$ (2.3) | ✓ | ✓ | ✓ | $\times$
Entropy (3.1) | ✓ | ✓ | ✓ | ✓
Max Score (3.1) | ✓ | ✓ | ✓ | ✓
Table 1: Our different uncertainty measures make different assumptions on
access to the model and the training data. A ✓ indicates requiring access.
Note that entropy and max score require access to the downstream classifier
and are thus not applicable in our setting.

####  2.2.1 Density: $p_{\mathrm{emb}}$

The density of the embedding space at a point $z$ would intuitively capture
how much data has the model observed around $z$ during training, which is the
transformation of the training data distribution under $f$. To estimate this
distribution, we fit a Gaussian333To be precise, one has to use a
Fisher–Bingham (or Kent) distribution (Jupp and Mardia, 2009; Kent, 1982) over
the $(m-1)$-sphere, which is the analogue of a Gaussian on hypersphere.
mixture model (GMM) to the $m$-dimensional embeddings of the training data.
Computing this density function requires access to the pre-trained model and
an _unsupervised_ training dataset, i.e., only the input datapoints and not
the labels.

####  2.2.2 Consistency: $p^{k,\tau}_{\mathrm{emb}}$

The consistency of the model at $z$ measures whether the training datapoints
mapped closest to $z$ have consistent labels. This notion would capture how
accurate the model is at $z$, based on the fact that a more accurate
contrastive model should have a more pure local correspondence. Note that
unlike the density-only distribution mentioned above, estimating this
distribution requires access to both training data and training labels
(correspondences). For each training datapoint, we calculate the fraction of
its $k$ nearest neighbors ($k$-NN) in the embedding space whose class labels
are consistent with that datapoint. We then filter out the datapoints based on
their $k$-NN accuracy with a threshold $\tau$, and fit a Gaussian mixture
model to the datapoints whose $k$-NN consistency is above the threshold
$\tau$. We denote this distribution by $p^{k,\tau}_{\mathrm{emb}}$. This
notion would require access to the model and a _supervised_ training dataset,
and is thus only applicable to the supervised contrastive learning setup
(Khosla et al., 2020).

It is worth noting that setting the threshold $\tau$ to zero yields
$p^{k,0}_{\mathrm{emb}}(\cdot)=p_{\mathrm{emb}}(\cdot)$, which would solely
capture the density of each datapoint in the training data.

Intuitively, the two notions defined above could lead to the following
scenarios:

  * •

High $p_{\mathrm{emb}}(z)$ and high $p_{\mathrm{emb}}^{k,\tau}(z)$: The model
has seen many consistent examples like $z$ (low uncertainty).

  * •

High $p_{\mathrm{emb}}(z)$ and low $p^{k,\tau}_{\mathrm{emb}}(z)$: The model
has seen samples similar to $z$ during training, but has not been consistent
for them. This could be due to the fact that these are hard examples, thus
implying low epistemic uncertainty but high aleatoric uncertainty.

  * •

Low $p_{\mathrm{emb}}(z)$: The model has not seen samples similar to $z$. This
implies the sample is likely out-of-distribution with respect to the training
set, and thus has a high epistemic uncertainty.

###  2.3 Per-Sample Feature Variance + Embedding Distribution:
$p^{k,\tau}_{\text{emb-ens}}$

One could also combine the two notions of per-sample variance and embedding
distribution, which has the interpretation of a stochastic embedding (Wang and
Isola, 2020). More specifically, we have an ensemble of probabilities through
the $l$ transformations $\\{T_{1},T_{2},...,T_{l}\\}$, and using the law of
total probability we have

$p^{k,\tau}_{\text{emb-
ens}}=\sum_{i=1}^{l}p^{k,\tau}_{\mathrm{emb}}(f(T_{i}(x)))p(T_{i})=\frac{1}{l}\sum_{i=1}^{l}p^{k,\tau}_{\mathrm{emb}}(f(T_{i}(x))).$

The measures mentioned above have different requirements, ranging from access
to the black-box model only (feature-variation measure), to requiring access
to a fully supervised training dataset (consistency measure). Table 1
summarizes the requirements for each measure. Note that the last two measures
(entropy and max score), which are explained in Section 3, require the full
observation of the downstream task and are solely defined as a baseline.

##  3 Experimental Results

We pre-train self-supervised (SimCLR) (Chen et al., 2020) and supervised
(SupCon) (Khosla et al., 2020) contrastive models with ResNet18 (He et al.,
2016) backbones, and on the training set of CIFAR10 or CIFAR100 (Krizhevsky et
al., 2009) datasets. We then perform inference on their test sets, alongside
test sets of CUBS2011 (Wah et al., 2011) and SVHN (Netzer et al., 2011) as
other out-of-distribution datasets. We follow the pre-training and linear
fine-tuning protocols in accordance to Khosla et al. (2020).

|  | Aleatoric | Epistemic | Overall
---|---|---|---|---
In-distribution | Correct | 0 | 0 | 0
Incorrect | 1 | 0 | 1
Out-of-distribution |  | - | 1 | 1
Table 2: This table shows how the ground-truth labels for evaluating each of
our notions of uncertainty are defined based on in- vs. out-of-distribution
and the correctness of the downstream classifier. Aleatoric uncertainty
evaluates whether the measure correlates with datapoint difficulty (downstream
failure) on the in-distribution samples. Epistemic uncertainty measures
whether unfamiliar (out-of-distribution) samples are distinguishable from
familiar (in-distribution) datapoints. Overall uncertainty measures the
retrieval of correctly classified in-distribution datapoints.

###  3.1 Uncertainty Measures

As discussed earlier, our different uncertainty measures make different
assumptions about access to the data and models, according to which we
categorize them into several groups.

Pre-trained model only. The feature-variation measure ($\delta$), described in
Section 2.1, only requires access to the pre-trained model. Our quantitative
results indicate that this very simple approach (directly applicable at
inference time) already allows for measuring uncertainty in many scenarios.

Pre-trained model + unsupervised training data. $p_{\mathrm{emb}}$ and
$p_{\text{emb-ens}}$ (described in Section 2.2) would require access to the
model and the training dataset without supervision (i.e., training images
only), making them applicable to both self-supervised and supervised setups.
The training data is only used for a single forward pass.

Pre-trained model + supervised training data. $p^{k,\tau}_{\mathrm{emb}}$ and
$p^{k,\tau}_{\text{emb-ens}}$ require access to the labeled training dataset,
as they incorporate the consistency notion mentioned in Section 2.2.2. Thus,
these measures are only applicable to the supervised contrastive (Khosla et
al., 2020) setup. In our experiments, we define consistency as having top 1%
$k$-NN accuracy threshold of 50%. We also study the impact of $k$ (the number
of neighbors) and $\tau$ (the threshold) on different types of uncertainty
estimation metrics in Section 3.3.

Fine-tuned model. The following two measures are not computable in our
scenario, as they are only measurable _after_ a downstream classifier is fine-
tuned on the pre-trained features. Thus, our approach would not be comparable
to these measures. Regardless, we report the measurements to put our
quantitative measurements in context. Our experiments indicate that in some
scenarios, our measures achieve competitive, or sometimes even slightly
better, performance compared with these measures.
Entropy: Entropy of a classifier is often used as a measure of uncertainty. We
measure the entropy of the downstream fine-tuned classifier on each sample and
use that as a measure of certainty.
Max score: The maximum score (confidence) of the downstream classifier is used
as a measure of certainty.

A summary of the requirements for each of the measures is provided in Table 1.

Setup | Dataset | $\delta$ | $p_{\mathrm{emb}}$ | $p_{\text{emb-ens}}$ | $p^{k,\tau}_{\mathrm{emb}}$ | $p^{k,\tau}_{\text{emb-ens}}$ | Entropy | Max score
---|---|---|---|---|---|---|---|---
SimCLR | CIFAR10 | 0.652 | 0.702 $\pm$ 0.023 | 0.719 $\pm$ 0.004 | - | - | 0.883 | 0.836
SimCLR | CIFAR100 | 0.647 | 0.559 $\pm$ 0.009 | 0.564 $\pm$ 0.017 | - | - | 0.816 | 0.761
SupCon | CIFAR10 | 0.830 | 0.805 $\pm$ 0.025 | 0.858 $\pm$ 0.004 | 0.808 $\pm$ 0.026 | 0.862 $\pm$ 0.002 | 0.916 | 0.892
SupCon | CIFAR100 | 0.766 | 0.735 $\pm$ 0.017 | 0.764 $\pm$ 0.004 | 0.720 $\pm$ 0.017 | 0.743 $\pm$ 0.002 | 0.879 | 0.852

Table 3: Aleatoric uncertainty (quantified based on AUROC as described in
3.2.1). Predicting performance on each datapoint in the downstream task of
image classification. As it can be observed, all of our notions meaningfully
capture sample difficulty as measured by correctness on in-distribution
datapoints.

Setup | In-dist | Out-of-dist | $\delta$ | $p_{\mathrm{emb}}$ | $p_{\text{emb-ens}}$ | $p^{k,\tau}_{\mathrm{emb}}$ | $p^{k,\tau}_{\text{emb-ens}}$ | Entropy | Max score
---|---|---|---|---|---|---|---|---|---
SimCLR | CIFAR10 | CUBS2011 | 0.766 | 0.59 $\pm$ 0.004 | 0.602 $\pm$ 0.074 | - | - | 0.689 | 0.745
SimCLR | CIFAR10 | SVHN | 0.393 | 0.960 $\pm$ 0.000 | 0.975 $\pm$ 0.018 | - | - | 0.890 | 0.918
SimCLR | CIFAR10 | CIFAR100 | 0.645 | 0.773 $\pm$ 0.003 | 0.793 $\pm$ 0.035 | - | - | 0.851 | 0.858
SimCLR | CIFAR100 | CUBS2011 | 0.783 | 0.598 $\pm$ 0.0032 | 0.608 $\pm$ 0.020 | - | - | 0.775 | 0.783
SimCLR | CIFAR100 | SVHN | 0.365 | 0.810 $\pm$ 0.002 | 0.846 $\pm$ 0.022 | - | - | 0.761 | 0.789
SimCLR | CIFAR100 | CIFAR10 | 0.610 | 0.515 $\pm$ 0.0023 | 0.516 $\pm$ 0.010 | - | - | 0.692 | 0.670
SupCon | CIFAR10 | CUBS2011 | 0.580 | 0.644 $\pm$ 0.005 | 0.660 $\pm$ 0.031 | 0.640 $\pm$ 0.006 | 0.655 $\pm$ 0.022 | 0.671 | 0.690
SupCon | CIFAR10 | SVHN | 0.548 | 0.977 $\pm$ 0.003 | 0.995 $\pm$ 0.000 | 0.976 $\pm$ 0.003 | 0.995 $\pm$ 0.001 | 0.962 | 0.964
SupCon | CIFAR10 | CIFAR100 | 0.765 | 0.878 $\pm$ 0.003 | 0.918 $\pm$ 0.002 | 0.877 $\pm$ 0.003 | 0.916 $\pm$ 0.002 | 0.903 | 0.900
SupCon | CIFAR100 | CUBS2011 | 0.853 | 0.727 $\pm$ 0.006 | 0.762 $\pm$ 0.084 | 0.718 $\pm$ 0.004 | 0.747 $\pm$ 0.026 | 0.877 | 0.884
SupCon | CIFAR100 | SVHN | 0.546 | 0.904 $\pm$ 0.003 | 0.940 $\pm$ 0.015 | 0.867 $\pm$ 0.002 | 0.903 $\pm$ 0.017 | 0.845 | 0.852
SupCon | CIFAR100 | CIFAR10 | 0.720 | 0.667 $\pm$ 0.005 | 0.689 $\pm$ 0.016 | 0.644 $\pm$ 0.002 | 0.661 $\pm$ 0.028 | 0.739 | 0.723

Table 4: Epistemic uncertainty (quantified based on AUROC as described in
3.2.2) is evaluated by measuring performance in terms of out-of-distribution
detection. As expected, $p_{\mathrm{emb}}$, which aims to directly estimate
the embedding distribution, outperforms other measures in most instances.

Setup | In-dist | Out-of-dist | $\delta$ | $p_{\mathrm{emb}}$ | $p_{\text{emb-ens}}$ | $p^{k,\tau}_{\mathrm{emb}}$ | $p^{k,\tau}_{\text{emb-ens}}$ | Entropy | Max score
---|---|---|---|---|---|---|---|---|---
SimCLR | CIFAR10 | CUBS2011 | 0.768 | 0.627 $\pm$ 0.006 | 0.640 $\pm$ 0.071 | - | - | 0.765 | 0.802
SimCLR | CIFAR10 | SVHN | 0.424 | 0.956 $\pm$ 0.001 | 0.970 $\pm$ 0.017 | - | - | 0.926 | 0.945
SimCLR | CIFAR10 | CIFAR100 | 0.666 | 0.785 $\pm$ 0.003 | 0.805 $\pm$ 0.036 | - | - | 0.896 | 0.894
SimCLR | CIFAR100 | CUBS2011 | 0.781 | 0.600 $\pm$ 0.006 | 0.609 $\pm$ 0.022 | - | - | 0.862 | 0.848
SimCLR | CIFAR100 | SVHN | 0.448 | 0.798 $\pm$ 0.005 | 0.829 $\pm$ 0.019 | - | - | 0.865 | 0.875
SimCLR | CIFAR100 | CIFAR10 | 0.668 | 0.539 $\pm$ 0.003 | 0.541 $\pm$ 0.010 | - | - | 0.810 | 0.768
SupCon | CIFAR10 | CUBS2011 | 0.616 | 0.675 $\pm$ 0.004 | 0.697 $\pm$ 0.028 | 0.671 $\pm$ 0.004 | 0.692 $\pm$ 0.021 | 0.713 | 0.729
SupCon | CIFAR10 | SVHN | 0.572 | 0.979 $\pm$ 0.003 | 0.995 $\pm$ 0.000 | 0.978 $\pm$ 0.003 | 0.995 $\pm$ 0.001 | 0.975 | 0.976
SupCon | CIFAR10 | CIFAR100 | 0.788 | 0.891 $\pm$ 0.003 | 0.931 $\pm$ 0.002 | 0.890 $\pm$ 0.003 | 0.930 $\pm$ 0.002 | 0.927 | 0.922
SupCon | CIFAR100 | CUBS2011 | 0.877 | 0.773 $\pm$ 0.012 | 0.811 $\pm$ 0.061 | 0.760 $\pm$ 0.009 | 0.790 $\pm$ 0.017 | 0.935 | 0.933
SupCon | CIFAR100 | SVHN | 0.645 | 0.916 $\pm$ 0.005 | 0.947 $\pm$ 0.009 | 0.8820 $\pm$ 0.005 | 0.912 $\pm$ 0.011 | 0.926 | 0.926
SupCon | CIFAR100 | CIFAR10 | 0.794 | 0.727 $\pm$ 0.010 | 0.756 $\pm$ 0.014 | 0.703 $\pm$ 0.007 | 0.725 $\pm$ 0.025 | 0.854 | 0.828

Table 5: Overall uncertainty (quantified based on AUROC as described in
3.2.3). Similar to epistemic uncertainty, $p_{\mathrm{emb}}$ seem to often
yield better quantitative metrics, especially in scenarios in which the
feature variation measure seems to fail, which shows the two notions are
complementary.

###  3.2 Evaluation

We evaluate different notions of uncertainty to cover different aspects of
downstream predictability. To evaluate the different metrics, we treat them as
retrieval instances and compute their AUROC (Area Under the Receiver Operating
Characteristic curve). The ground-truth label of the retrieval instance could
be derived as a function of the downstream accuracy of a datapoint and whether
the model has been exposed to the datapoint’s semantic class during pre-
training. In what follows, we discuss the details of this evaluation for each
uncertainty notion. Table 2 summarizes what each uncertainty notion is
capturing.

Figure 2: The effect of number of GMM components on our aleatoric uncertainty
measures. We generally observe improvements with larger $n_{\text{comp}}$.
However, for $n_{\text{comp}}>20$, the improvements seem rather marginal and
the results seem stable across all models and datasets.

####  3.2.1 Aleatoric Uncertainty

Aleatoric uncertainty is often defined as the “noise inherent in the data,”
which leads to difficulty of understanding a sample datapoint. We use
downstream performance of a datapoint as a proxy for measuring its difficulty.
To quantify such notion, we evaluate our proposed uncertainty measures on in-
distribution test-set datapoints, and in terms of their capability in
retrieving samples which are correctly classified in a downstream linear
classifier. Table 3 shows this metric for our different uncertainty measures.

####  3.2.2 Epistemic Uncertainty

We evaluate our uncertainty estimation measures on images from the in-
distribution (pre-training dataset) and an out-of-distribution dataset and
quantify their performance in terms of retrieving the in-distribution
embeddings. In other words, a model pre-trained (supervised or self-
supervised) on the training set of dataset A is fed test datapoints from
datasets A and B. Then, the effectiveness of the uncertainty measures are
evaluated in terms of distinguishing datapoints of dataset A from those of B.
Table 4 contains the performance of our different measures on this task. It
can be observed that in most cases, $p_{\mathrm{emb}}$ has the best
performance, whose definition is also more consistent with out-of-distribution
detection tasks, as it directly estimates the embedding distribution that
comes from the training data. Another observation would be the failure of the
feature variation measure in detecting out-of-distribution samples of SVHN in
the self-supervised setups. We hypothesize this could be due to the fact that
SVHN is a less diverse dataset, which results in its images being mapped close
to one another in a CIFAR10 or CIFAR100 pre-trained model. As a result,
feature variation would not be a good notion for distinguishing such samples.
On the other hand, the probability-based measures result in very high AUROC
scores, alluding that these measures capture complementary notions of
reliability. Another explanation could be that measuring the effect of data
transformation could be interpreted as mainly a notion of data uncertainty.
This would suggest that this measure is a better fit for aleatoric uncertainty
estimation.

Figure 3: The effect of number of GMM components on different epistemic
uncertainty measures. The titles specify the in-distribution (pre-training)
datasets vs. the out-of-distribiution datasets. Similar to the other types of
uncertainty, we generally observe improvements with larger $n_{\text{comp}}$.
However the improvements are rather marginal for higher $n_{\text{comp}}$, and
the results seem stable across all models and datasets.

On another note, we observe that our probability measures have lower
discriminative power distinguishing between CIFAR100 and CIFAR10, as opposed
to between CIFAR examples and non-CIFAR examples. This observation is
consistent across both datasets, and across both supervised (SupCon) and self-
supervised (SimCLR) setups. Also, detecting CIFAR10 samples as out-of-
distribution, given a CIFAR100 pre-trained $p_{\mathrm{emb}}$ model, is
noticeably more difficult than distinguishing CIFAR100 samples using a CIFAR10
pre-trained $p_{\mathrm{emb}}$.

Here we followed standard practice for evaluating epistemic uncertainty.
However, we argue that the assumption of all the datapoints in the test set of
dataset A being “in-distribution” to the model may not necessarily hold. To
address that, we define the following alternative, which captures such
nuances.

####  3.2.3 Overall Uncertainty

Here we introduce a hybrid definition of uncertainty, taking into account both
aleatoric and epistemic uncertainties. Given a model trained on dataset A, we
evaluate its uncertainty measures on datasets A and B. We then evaluate how
well the uncertainty measure retrieves datapoints in A which are correctly
classified by the downstream classifier. In other words, the model should not
be certain about all the datapoints in A, but only the ones that are going to
be correctly classified downstream. The quantitative measures using this
metric are reported in Table 5. Comparing values in this table with their
corresponding values in the epistemic uncertainty evaluation (Table 4), we
generally observe higher values across all measures.

###  3.3 Ablation Study

In this section, we analyze the effect of different parameters on the
performance of our approach.

Figure 4: Observing the effect of $k$ (number of neighbors) and $\tau$
(threshold) in the performance of $p_{\mathrm{emb}}^{k,\tau}$ shows a
diminishing return on using higher accuracy thresholds, as it leads to pruning
a large number of datapoints. For epistemic uncertainty (center), we observe
the peak of performance to be at $\tau=0$, which corresponds to directly
estimating embedding distribution of all of the training set. This is
consistent with the fact that epistemic uncertainty is evaluated by detecting
out-of-distribution samples. On the other hand, for aleatoric and overall
uncertainties, a marginal improvement is achieved at higher thresholds, as the
consistency notion does contribute to better downstream performance
prediction. Figure 5: The effect of number of transformations on aleatoric
uncertainty estimation. Unlike other variations of uncertainty, we observe no
improvement with more than 2 augmentations in most cases.

Effect of number of GMM components. We evaluate the effect of the number of
GMM components ($n_{\text{comp}}$), by evaluating the metrics, while sweeping
$n_{\text{comp}}$ from 2 to 150 components. Figures 2 and 3 show the aleatoric
and epistemic uncertainty measures (y-axis), respectively, using different
number of components (x-axis). Interestingly, except for extremely small
values ($n_{\text{comp}}<10$), we observe relatively stable performance across
all setups. This observation seems to be consistent across both supervised and
self-supervised setups. The overall uncertainty has a very similar trend as
the epistemic uncertainty. For the sake of brevity, we relegated the result to
the Appendix.

Figure 6: The effect of number of transformations on epistemic uncertainty
estimation. As expected, combining likelihoods resulting from more
augmentations result in better out-of-distribution detection.

Effect of threshold and number of nearest neighbors. Higher thresholds and
number of nearest neighbors would result in maintaining highly consistent
points, at the cost of losing information (having less remaining datapoints
for estimating the GMMs). The diminishing returns of such parameters can be
seen in Figure 4. It can be observed that peak-performance for epistemic
uncertainty (middle), is at threshold of 0 ($p_{\mathrm{emb}}$), which is
consistent with its definition of directly estimating the training data (in-
distribution likelihood). On the contrary, consistency does improve the
estimation of aleatoric uncertainty, as the peak of the distribution does
occur at ($\tau=0.4$, $k/N=0.025$), which means that a consistent datapoint is
defined as one whose top 2.5% of nearest neighbors are more than 40%
consistent with its semantic label. Overall, we observe the effect of
consistency to be marginal in our experiments. We hypothesize that such
behavior could be due to a highly accurate pre-training, resulting in a high
correlation between consistency and density. This correlation could be
measured at larger scale and across different setups and datasets (with
different accuracies) to validate this hypothesis.

Effect of number of transformations. Figure 5 shows the effect of number of
augmentations on aleatoric uncertainty estimation. It can be observed that
after 2 augmentations, the results are relatively stable. On the other hand,
for epistemic uncertainty, shown in Figure 6, more improvement (yet marginal)
could be achieved with more augmentations. The overall uncertainty has a very
similar behavior to the epistemic uncertainty, shown in Figure 7.

##  4 Conclusion

In this paper, we explored the possibility of estimating a
reliability/uncertainty measure for the abstract embeddings of contrastive
models. We show that our uncertainty measures not only are able to
meaningfully detect out-of-distribution samples but also are predictive of
performance in downstream tasks. We believe that having such notion of
reliability/uncertainty can particularly be insightful, e.g., for deciding
between different options of pre-trained models, or for deciding on specific
sample weighting policies in downstream fine-tuning tasks.

## References

  * NTS [2017] Collision between a car operating with automated vehicle control systems and a tractor-semitrailer truck near Williston, Florida, May 7, 2016. highway accident report NTSB/HAR-17/02.  Technical report, National Transportation Safety Board, 2017.
  * Varshney and Alemzadeh [2017] Kush R Varshney and Homa Alemzadeh.  On the safety of machine learning: Cyber-physical systems, decision sciences, and data products.  _Big data_ , 5(3):246–255, 2017.
  * Guynn [2015] Jessica Guynn.  Google photos labeled black people’gorillas’.  _USA Today_ , 1, 2015.
  * Dietterich [2017] Thomas G Dietterich.  Steps toward robust artificial intelligence.  _AI Magazine_ , 38(3):3–24, 2017.
  * Hüllermeier and Waegeman [2021] Eyke Hüllermeier and Willem Waegeman.  Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.  _Machine Learning_ , 110(3):457–506, 2021.
  * Aggarwal et al. [2014] Charu C Aggarwal, Xiangnan Kong, Quanquan Gu, Jiawei Han, and S Yu Philip.  Active learning: A survey.  In _Data Classification: Algorithms and Applications_ , pages 571–605. CRC Press, 2014.
  * Blundell et al. [2015] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight uncertainty in neural network.  In _International Conference on Machine Learning_ , pages 1613–1622. PMLR, 2015.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , pages 1050–1059. PMLR, 2016.
  * Lakshminarayanan et al. [2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Loquercio et al. [2020] Antonio Loquercio, Mattia Segu, and Davide Scaramuzza.  A general framework for uncertainty estimation in deep learning.  _IEEE Robotics and Automation Letters_ , 5(2):3153–3160, 2020.
  * Sharma et al. [2021] Apoorva Sharma, Navid Azizan, and Marco Pavone.  Sketching curvature for efficient out-of-distribution detection for deep neural networks.  In _Conference on Uncertainty in Artificial Intelligence (UAI)_ , 2021\.
  * Osband et al. [2021] Ian Osband, Zheng Wen, Mohammad Asghari, Morteza Ibrahimi, Xiyuan Lu, and Benjamin Van Roy.  Epistemic neural networks.  _arXiv preprint arXiv:2107.08924_ , 2021.
  * Kiureghian and Ditlevsen [2009] Armen Der Kiureghian and Ove Ditlevsen.  Aleatory or epistemic? does it matter?  _Structural Safety_ , 31(2):105–112, 2009.  ISSN 0167-4730.
  * Kendall and Gal [2017] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  _Advances in Neural Information Processing Systems_ , 30:5574–5584, 2017.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020.
  * Khosla et al. [2020] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Wu et al. [2018] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3733–3742, 2018.
  * Hénaff et al. [2020] Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord.  Data-efficient image recognition with contrastive predictive coding.  In _International Conference on Machine Learning_ , pages 4182–4192. PMLR, 2020.
  * Oord et al. [2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Tian et al. [2020] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16_ , pages 776–794. Springer, 2020.
  * Hjelm et al. [2018] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio.  Learning deep representations by mutual information estimation and maximization.  In _International Conference on Learning Representations_ , 2018.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9729–9738, 2020.
  * Wang and Tan [2017] Dong Wang and Xiaoyang Tan.  Robust distance metric learning via bayesian inference.  _IEEE Transactions on Image Processing_ , 27(3):1542–1553, 2017.
  * Qian et al. [2018] Qi Qian, Jiasheng Tang, Hao Li, Shenghuo Zhu, and Rong Jin.  Large-scale distance metric learning with uncertainty.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8542–8550, 2018.
  * Oh et al. [2018] Seong Joon Oh, Kevin P Murphy, Jiyan Pan, Joseph Roth, Florian Schroff, and Andrew C Gallagher.  Modeling uncertainty with hedged instance embeddings.  In _International Conference on Learning Representations_ , 2018.
  * Zhang et al. [2021] Oliver Zhang, Mike Wu, Jasmine Bayrooti, and Noah Goodman.  Temperature as uncertainty in contrastive learning.  _arXiv preprint arXiv:2110.04403_ , 2021.
  * Wu and Goodman [2020] Mike Wu and Noah Goodman.  A simple framework for uncertainty in contrastive learning.  _arXiv preprint arXiv:2010.02038_ , 2020.
  * Shorten and Khoshgoftaar [2019] Connor Shorten and Taghi M Khoshgoftaar.  A survey on image data augmentation for deep learning.  _Journal of Big Data_ , 6(1):1–48, 2019.
  * Jupp and Mardia [2009] Peter E Jupp and Kanti V Mardia.  _Directional statistics_.  John Wiley & Sons, 2009.
  * Kent [1982] John T Kent.  The fisher-bingham distribution on the sphere.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 44(1):71–80, 1982.
  * Wang and Isola [2020] Tongzhou Wang and Phillip Isola.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  In _International Conference on Machine Learning_ , pages 9929–9939. PMLR, 2020.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Wah et al. [2011] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * Netzer et al. [2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.

##  Appendix A Appendix

Effect of number of transformations (for overall uncertainty)

As mentioned in Section 3.3, we provide the effect of number of data
transformations (image augmentations) on the overall uncertainty estimation,
shown in Figure 7. We observe the trends to be very similar to those of
epistemic uncertainty. We observe a monotonically increasing performance as a
function of the number of transformations. However, in many scenarios such as
SupCon on CIFAR10 vs. SVHN, and CIFAR100 vs. CUBS2011, the amount of
improvement becomes marginal beyond 2 augmentations.

Figure 7: The effect of number of transformations on overall uncertainty.
Similar as in the case of epistemic uncertainty, we observe an increase in the
performance with more transformations.

Effect of number of GMM components (for overall uncertainty)

Again, related to Section 3.3, we provide the effect of number of components
on the overall uncertainty estimation, shown in Figure 8. We observe very
similar trends compared to epistemic uncertainty, and we find the performance
to be rather stable beyond $n_{\text{comp}}=30$.

Figure 8: The effect of number of components in the GMM on different overall
uncertainty measures. The titles specify the in-distribution (pre-training)
datasets vs. the out-of-distribiution datasets. Similar to the other types of
uncertainty, we generally observe improvements with larger $n_{\text{comp}}$.
However the improvements are rather marginal for higher $n_{\text{comp}}$, and
the results seem stable across all models and datasets.


Department of Computer Science
TDT4173 - Assignment 1

Time Series

Long Short-term Memory RNN

_Group:_

Group 8 - Time Series

_Authors:_

Adrian Kjærran (adriankj)
Erling Stray Bugge (erlinsb)
Christian Bakke Vennerød (christbv)

August 27, 2024

###### Abstract

This paper is the first assignment in the course Machine Learning TDT4173,
fall 2020. The project was conducted by reading through some of the latest
developments within time-series forecasting methods in the scientific
community over the past 5 years. The result is a summary of the essential
aspects.

In this paper, we introduce an LSTM cell’s architecture and explain how
different components go together to alter the cell’s memory and predict the
output. Also, the paper provides the necessary formulas and foundations to
calculate a forward iteration through an LSTM. Then, the paper refers to
applications and research that emphasize the strengths and opportunities of
LSTMs, shown within the time-series domain and the natural language processing
(NLP) domain. Finally, alternative statistical methods for time series
predictions are highlighted, where the paper look into ARIMA and exponential
smoothing. Nevertheless, as LSTMs can be viewed as a complex architecture, the
paper assumes that the reader has a general knowledge of machine learning
aspects, such as neural networks, activation functions, overfitting,
underfitting, metrics and bias.

###### Contents

  1. 1 Introduction
  2. 2 Foundations
    1. 2.1 Recurrent Neural Networks
    2. 2.2 The Vanishing Gradient Problem
    3. 2.3 The Idea behind LSTMs
    4. 2.4 Intuition behind LSTM
    5. 2.5 Components and Calculations
    6. 2.6 Pseudocode
    7. 2.7 Evaluation
  3. 3 Current Applications
    1. 3.1 LSTMs used in forecasting solar power production
    2. 3.2 LSTMs used in Natural Language Processing
  4. 4 Alternative methods
    1. 4.1 ARIMA and its components
    2. 4.2 Exponential Smoothing
    3. 4.3 Discussion and Comparison of LSTM with Statistical Methods
  5. 5 Conclusions and Further Work

###### List of Figures

  1. 1 LSTM placed in the Machine Learning taxonomy.
  2. 2 Unfolded representation of an RNN.
  3. 3 The Architecture of a LSTM Cell
  4. 4 Choose n at the point where RMSE starts to rise [Gensler et al., 2016].

###### List of Tables

  1. 1 Test-results for power generation forecasting. Values closer to 0 the better. [Gensler et al., 2016].
  2. 2 The results of previous methods, human performance and ELMo on two different NLP benchmarks. 100 is a perfect score.

##  1 Introduction

A time series is a sequence of historical measurements of an observable
variable at equal time intervals. Time series data often exhibit patterns such
as trend, seasonality and cycles: A time series has a trend if there is a
long-term increase or decrease in the data, it is seasonal if the data is
affected seasonally at a known and fixed frequency, e.g. time of the year or
time of the day, and it is cyclical if the data either rises or falls
according to an unfixed frequency, e.g. how the economy (GDP) oscillates
around the long-term growth trend [Nau, 2020]. Such patterns, as described
above, can be exploited by a forecasting-model to infer dependencies between
the past and the present, to predict the future [Hyndman and Athanasopoulos,
2018].

Over the past 50+ years, many statistical, structured models have yielded the
best predictions on time series data [Hyndman and Athanasopoulos, 2018]. Such
structured models, e.g. ARIMA, try to prescribe the data generating process
[Barker, 2020]. However, with the emergence of new machine learning models,
increasingly complex data, and more computing power, the structured,
statistical models are now challenged by the power of artificial intelligence
[Bontempi et al., 2013].

One subfield of artificial intelligence is machine learning, where models
automatically improve through experience [Goodfellow et al., 2016]. A class of
machine learning methods designed to exhibit temporal behaviour is recurrent
neural networks (RNN). An RNN is a standard feedforward neural network, but
with a feedback loop, facilitating the network to handle temporal information
or dynamic patterns, e.g. time series, where the value at a given time often
depends on past values [Alanis and Sanchez, 2017]. One architecture building
on the RNN architecture is LSTM, the main topic of this paper. An LSTM is an
RNN, but with long short-term memory cells. The LSTM cell contains numerous
internal weights and is particularly designed to capture and retain essential
information from previous outputs.

In order for an LSTM to function properly, the weights of the internal LSTM
cells must be trained. The most important aspects of the training process can
be described in three steps: Firstly, the internal weights are initialized in
a random configuration. Secondly, a batch of training-data is fed to the
network, and an error-metric compares the output from the network and ground
truth. Finally, the error is propagated back through the network, adjusting
the weights within the LSTM cells through a process called back-propagation.
Then this is repeated for each batch from the training data. Hence, during
training, the network learns the patterns of the data by adjusting its weights
towards the known correct outputs; this makes LSTM a supervised learning
architecture [Hochreiter and Schmidhuber, 1997]. It is worth mentioning
however, that LSTMs can be used on unsupervised problems too; for instance, in
sequence modelling tasks such as NLP, target outputs can be generated by
shifting the input one or more time-steps [Nordanger, 2020], despite LSTM
being a predominantly supervised algorithm. Figure 1 displays how LSTM belongs
under the RNN branch in the deep-learning part of supervised learning.

Figure 1: LSTM placed in the Machine Learning taxonomy.

##  2 Foundations

###  2.1 Recurrent Neural Networks

LSTMs are recurrent neural networks, where the neurons are replaced by LSTMs
cells. Therefore, to understand LSTMs we must understand the emergence and
architecture of recurrent neural networks. The motivation behind RNNs is to
tackle the problem of a standard neural network’s limited memory about
previous data. Standard neural network’s memory is limited to their learned
weight from training, and during prediction, they only utilize information
about the current input $p(y_{t}\mid x_{t})$. With exploiting knowledge of
previously seen data, this becomes a difficult task. To solve this, the
neurons in RNNs recurs a hidden variable $h_{t}$ which contains information
that later iteration through the network can benefit utilize $p(y_{t}\mid
x_{t},h_{t-1})$; hence their name recurrent neural networks [Zhang et al.,
2020]. This change of architecture allows information to persist within the
network and makes RNNs more suitable for predicting the outcome of a time-
series compared to traditional neural networks.

Figure 2: Unfolded representation of an RNN.

###  2.2 The Vanishing Gradient Problem

In theory, the hidden state $h_{t}$, in each RNN cell can keep track of
information for an arbitrary amount of time. However, as the hidden state
$h_{t}$ is updated at each time step, i.e. $h_{t}=f(x_{t},h_{t-1})$, the time
dependency often causes long-term dependencies, e.g 100 time steps, to be
ignored during training [Le and Zuidema, 2016]. Neural networks are trained
through the back-propagating of an error measure through the network, and
adjusting the weights of the matrices and biases such that the error is
reduced for future iterations. One problem that emerges when the network
becomes deeper is that the gradients tend to explode or vanish. This is
because the gradients, i.e. the change of weights, are computed based on the
cell’s contribution to the output. The problem greatly intensifies with RNNs,
since gradients are not only calculated in the ”depth”-dimension, but also in
the ”time”-dimension [Guo, 2013]. This problem is called the vanishing
gradient problem, and it makes it difficult for standard RNNs to learn long-
term dependencies in the data.

###  2.3 The Idea behind LSTMs

LSTMs were introduced to handle the vanishing gradient problem of the
traditional RNNs. An LSTMs architecture is similar to an RNN, but the
recurrent cells are replaced with LSTM cells [Hochreiter and Schmidhuber,
1997] which are constructed to retain information for extended periods.
Similar to the cells in standard RNNs, the LSTM cell recurs the previous
output, but also, it keeps track of an internal cell state $c_{t}$, which is a
vector serving as long-term memory. Hence, the LSTMs have access to both
short-term memory, i.e. the hidden state $h_{t}$, and long-term memory, i.e.
the cell state $c_{t}$, resulting in the name: long short-term memory cells
[Hochreiter and Schmidhuber, 1997]. The cell state can metaphorically be
viewed as a conveyor belt that passes information down the line, to be used in
later calculations. The content of this cell state vector is carefully altered
through operations by gates inside the LSTM cell, which lets essential
information and gradients to flow unchanged.

###  2.4 Intuition behind LSTM

The main idea of the LSTM cell is to regulate the updates of the long-term
memory (cell state), such that information and gradients (for training) can
flow unchanged between iterations.

To give an intuition, imagine an LSTM that is to predict the next scene in a
movie with two main characters, Alice and Bob. The core idea would be to let
the movie scenes without Alice leave the Alice-related information completely
unchanged. Through careful regulation, we could display half-an-hour of scenes
with Bob without this interfering with the cell state’s weights which saves
information about Alice. To carefully regulate the cell-state, the LSTM uses
the three yellow gates in Figure 3.

Forget gate – Regulates forgetting of previous information. If a scene ends,
we want to forget all the information associated with the scene, but still
remember information about Alice and Bob

Input gate – Controls the relevance and saving of new information. When a new
scene is introduced, the input gate ”writes” to memory the critical
information to remember about the scene.

Output gate – Joins the long-term information with the short-term information
to give a better prediction about the next hidden unit/output.

These precise operations let the LSTM cell keep track of the most relevant
information, update the cell state in a precise way, and choose what specific
information to pay attention to when predicting new outputs.

###  2.5 Components and Calculations

To understand how LSTMs work, we must thoroughly understand the components of
the LSTM cell. The cell has three internal gates, namely the forget gate, the
input gate, and the output gate, see figure 3. These gates control the
internal cell state $c_{t}$, and the hidden state $h_{t}$. In figure 3, we
observe in the yellow boxes how the input vector, $x_{t[n-1]}$, and the
previous hidden state, $h_{t-1[m,1]}$, are multiplied respectively with four
different matrices, $W_{j}{[m,m+n]}$, and added respectively with four
different biases, $b_{j}{[m,1]}$. The outputs of these gates are then used to
both update the cell state to form $c_{t[m-1]}$, and to calculate the new
hidden state $h_{t[m,1]}$. Here, the notation within the brackets, i.e.
$[m,1]$ and $[m,m+n]$, are the dimensions of the matrices. Furthermore, the
”cross-sign” in the gates are a dot product, and the ”plus-sign” addition.

Figure 3: The Architecture of a LSTM Cell

The four combinations of a matrix $W$, a bias $b$, and an activation function,
that is either the sigmoid or the tanh, can be view as a one-layer neural
network inside of the cell. We define each of these one-layer neural networks
as functional units, and their general formula is shown equation 1.

$OutFunctionalUnit_{[m,1]}=ActivationFunction(W_{[m,m+n]}*[h_{[m,1]},x_{[n,1]}]+b_{[m,1]})$
(1)

Forget Gate

The forget gate decides what information and how much information to erase
from the cell state. The information in the cell state $c_{t-1}$ is altered
through an elementwise multiplication of the forget gate’s output $f_{t}$ 2,
see equation 3 and Figure 3. The forget gate’s activation function is the
sigmoid function; this ensures that the output vector from the gate yields
continuous values between 1 (keep) and 0 (forget). The subscript ‘f’ on the
matrix and bias in equation 2 indicates their belongingness to the forget
gate.

$f_{t}=\sigma(W_{f}\cdot[h_{t-1},x_{t}]+b_{f})$ (2) $c_{f}=c_{t-1}\times
f_{t}$ (3)

Input Gate

The next important gate is the input gate. This gate uses two functional
units. The first functional unit use a tanh activation which output values
between -1 and 1 and decide the change of the cell state $\tilde{c}_{t}$. The
latter use a sigmoid activation function, and it is responsible for the
magnitude of the change $m_{t}$. After multiplying the results of these two
functional units, the input gate adds the result to the cell state, equation
6, and this completes the update of the cell state $c_{t}$

$\tilde{c}_{t}=tanh(W_{c}\cdot[h_{t-1},x_{t}]+b_{c})$ (4)
$m_{t}=\sigma(W_{m}\cdot[h_{t-i},x_{t}]+b_{m})$ (5)
$c_{t}=c_{f}+\tilde{c}_{t}*m_{t}$ (6)

Output Gate

The output gate uses a trained matrix $W_{o}$ and bias $b_{o}$, to fetch
relevant information from the current input and previous output, equation 7.
This information is combined with the newly adjusted cell state $c_{t}$ to
predict the next output, $h_{t}$, equation 8. This output recurs such that the
next iteration can use it. If there are multiple layers, this output is also
used as input in the next layer, if not this output is the prediction.

$o_{t}=\sigma(W_{o}[h_{t-1},x_{t}]+b_{o})$ (7) $h_{t}=o_{t}*tanh(c_{t})$ (8)

###  2.6 Pseudocode

Algorithm 1 One iteration through an LSTM cell

procedure Iteration($c_{t-1},h_{t-1},x_{t}$)

$c_{t}\leftarrow$ Element-wise multiply $c_{t-1}$ with output from forget gate
$f_{t}$ $\triangleright$ Equation (3)

$c_{t}\mathrel{{+}{=}}$ The output from the input gate
$(\tilde{c}_{t}*m_{t}$)$\triangleright$ Equation (6)

$h_{t}\leftarrow$ Combination of the cell state $c_{t}$, input $x_{t}$ and
previous output $h_{t-1}$ $\triangleright$ Equations (7,8)

return $(c_{t},h_{t})$

end procedure

###  2.7 Evaluation

To evaluate LSTMs performance, it is common to use error metrics such as root
mean squared error (RMSE) and mean absolute error (MAE) [Bouktif et al.,
2018]. In general, When evaluating neural networks, it is important to
evaluate the model on test data that the algorithm has not seen before. This
is especially important in neural networks because their ability to learn
complex structures makes them vulnerable to overfitting. In Section 3 we will
use these error metrics to discuss LSTMs performance.

##  3 Current Applications

Due to the inherent ability of LSTMs to handle temporal data, variants of
LSTMs architectures can be used in various types of applications, in many
different domains. This section focus on applications within the time-series
domain and the natural language processing (NLP) domain.

###  3.1 LSTMs used in forecasting solar power production

Being able to forecast future power generation for a power plant can, among
other things, help to minimize operational costs, and more efficiently plan
for maintenance and improvements Zuzuki [2015]. As the power generation for
solar power plants is highly dependent on the current weather situation, the
resulting prediction is influenced by the performance of the weather
prediction, as well as the performance of the power-generation prediction.
Gensler et al. 2016 focused only on the last part, where they used the weather
forecasts as a given input and utilized LSTMs to predict power generation.

Figure 4: Choose n at the point where RMSE starts to rise [Gensler et al.,
2016].

####  3.1.1 How LSTM was implemented

The authors implemented both an Auto-Encoder LSTM and a traditional LSTM
network. We will in this section, however, only focus on the latter one. The
network consisted of four LSTM layers stacked on top of each other and an
output layer with Rectified Linear Unit (ReLU) as an activation function for
the predictions [Gensler et al., 2016]. The dataset was aggregated on 990 days
of weather data in 3-hour resolutions from 21 different power plants. The LSTM
network was trained on $n=2$ previous time-steps to predict a new value, see
figure 4 for an explanation of why $n=2$ was chosen. This means that the
memory cells of the LSTMs were trained by trying to predict $y_{t}$ based on
the two past values, $\\{y_{t-1},y_{t-2}\\}$, then predicting $y_{t+1}$ based
on $\\{y_{t-1},y_{t-2}\\}$ and so on. However, as the model’s performance was
based on its ability to predict 24h and 48h into the future, a recursive
strategy was implemented to turn one-step predictions into a multi-step
prediction. A physical forecasting model, P-PVFM, was also implemented as a
reference model. The results are summarized in table 1. We see from the Root
Mean Squared Error (RMSE) and Mean Average Error (MAE) that the LSTM model
prove superior to the reference model. Furthermore, whereas the reference
model overestimates the power generation (positive bias), the LSTM model
underestimates the power output (negative bias), although to a lower extent
[Gensler et al., 2016].

| P-PVFM | LSTM |
---|---|---|---
Avg. RMSE | 0.1086 | 0.0724 |
Avg. MAE | 0.0560 | 0.0368 |
Avg. BIAS | 0.0399 | -0.0073 |
Table 1: Test-results for power generation forecasting. Values closer to 0 the
better. [Gensler et al., 2016].

####  3.1.2 Why LSTMs works well in this scenario

Physical models often require specific knowledge of the power plant and are
more based on parameters such as solar curves, wind turbine dimensions etc.
Whereas physical models can be highly accurate and offer a high degree of
interpretability, they are often too dependent upon the precision of
measurements. Hence, as statistical- or machine learning models do not try to
explain the underlying process directly, but instead try to find a function
describing the relationship between weather data (input) and power generation
(output), they are more flexible and less affected by measurements errors
[Gensler et al., 2016]. Furthermore, as the dataset included dates and
contained a large number of features, both numerical and categorical
(snowfall, rain etc.), the LSTM work even better as they can benefit from
cross-series training, i.e. learn dependencies across different time-series,
and thus learn not just the trend, but also seasonal patterns and level. This
is also emphasized by Barker [2020], who highlights why global, unstructured
models, e.g. LSTMs, can be better than structured models, e.g. ARIMA, at
capturing elements of trend, levels, and seasonality on more complex data.

###  3.2 LSTMs used in Natural Language Processing

LSTMs can be used in a variety of NLP applications, such as language
modelling, where a probability distribution over words are learned. This is
because the architecture of LSTMs is well suited for sequence modelling; in
the same way as predicting $y_{t}$ based on a time series sequence
$Y={y_{t-N},\dots,y_{t-1}}$, one can use the previous words in a sentence
$W=\\{w_{t-N},\dots,w_{t-1}\\}$ to predict $w_{t}$.

####  3.2.1 How LSTM was implemented

One example of LSTMs used in NLP is with ELMo (Embeddings from Language
Models) from 2018 [Peters et al., 2018]. The full details are beyond the scope
of this paper. Nevertheless, the model uses bidirectional LSTMs to produce
context-sensitive embeddings. Whereas language models is a probability
distribution over words, an embedding is a floating-point vector-
representation of words. The higher the vectors’ dimensions in the embedding,
the more fine-grained relationships between words can be captured. Two key
advantages of embeddings are that one can use cosine-similarity to find
related words (synonyms) and use vector operations on words [Mikolov et al.,
2013]. The results are shown in Table 2, where SQuAD and NER are benchmarks
for question answering and named entity recognition (classifying a word in a
sequence to an entity), respectively. Note that newer models have surpassed
the human benchmark on the SQuAD dataset [Rajpurkar, 2020], and F1 is a metric
which uses the harmonic weighting between precision and recall.

| Previous methods | Humans | ELMo |
---|---|---|---|---
SQuAD 1.1 (F1) | 84.4 | 91.2 | 87.4 |
NER (F1) | 91.9 | 96.9 | 92.2 |
Table 2: The results of previous methods, human performance and ELMo on two
different NLP benchmarks. 100 is a perfect score.

####  3.2.2 Why LSTMs works well in this scenario

One of the reasons LSTMs worked well in this scenario is their ability to be
bidirectional. ELMo used two separate forward and backwards 2-layered LSTMs
(bidirectional LSTM). In contrast to the forward-LSTM in ELMo, who looks at
the preceding words $\\{w_{1},\dots,w_{k-1}\\}$ in a sequence to find the
probability of $w_{k}$, a backward-LSTM looks at the succeeding words
$\\{w_{k+1},\dots,w_{N}\\}$ to predict $w_{k}$ [Peters et al., 2018]. One
advantage of this is that it enabled ELMo to learn context-sensitive
embeddings [Peters et al., 2018], meaning that a word such as ”bank” will have
a different vector representation if it is used in the context of a ”river
bank”, versus in the context of ”Bank of America”. This is one of the reasons
why ELMo could learn more patterns and relationships in the language it is
trained on and thus increased its performance.

##  4 Alternative methods

As described in the applications section, LSTM-architectures can be used in
various types of forecasting applications. It is, however, important to
emphasize that the type of data available and the type of problem heavily
influence the right choice of method. This section covers two alternatives,
statistical methods that have been widely used in the past 50+ years: ARIMA
models and exponential smoothing models. After giving the foundations of these
methods, we compare different strengths and weaknesses of the more statistical
methods to LSTMs.

###  4.1 ARIMA and its components

ARIMA is an acronym for Autoregressive Integrated Moving Average and is a
general class of linear forecasting models In fact, more known methods such as
linear regression, random-walk and exponential smoothing are all special cases
of ARIMA models Hyndman and Athanasopoulos [2018]. ARIMA models can work on
both seasonal and non-seasonal data. However, we will in this section only
focus on non-seasonal ARIMA models and non-seasonal data, which consist of
three components: Autoregressive models (AR), Integrated (I), and Moving
Average models (MA) Hyndman and Athanasopoulos [2018].

Firstly, Autoregressive models (AR) attempt to predict a future variable,
$y_{t}$, by creating a linear combination of past values of the variable the
model aims to predict $y_{t+1}$.

$y_{t}=c+\phi_{1}y_{t-1}+\phi_{2}y_{t-2}+\dots+\phi_{p}y_{t-p}+\epsilon_{t}$
(9)

Here, $\epsilon_{t}$ is an error term, e.g. white noise, $\phi_{j}$ is the
variables to be fitted, $j\in(1,p)$, and $c$ is the average of the period
change (long-term drift). If $c$ is positive, $y_{t}$ will tend to drift
upwards, whereas if $c$ is negative, $y_{t}$ will tend to drift downwards
[Hyndman and Athanasopoulos, 2018].

The Integrated component (I) is a component that tries to reduce the
seasonality and trend in a time series. A stationary time series means that
the time series has a constant mean and exhibits neither trend nor seasonality
[Nau, 2020]. This implies that the statistical properties are constant over
time, i.e. a constant autocorrelation, meaning that the correlations between
all two adjacent data points in the time series are constant. Through
differencing a given number of times, a non-stationary time series can often
be made stationary. Hence, $y^{\prime}$ represents the observed variable
differenced $d$ number of times, i.e. $(y^{\prime}_{t}=y_{t}-y_{t-1})^{d}$.

The last component, the Moving Average (MA) model, use a linear combination of
the past $q$ forecast errors to predict $y_{t}$.

$y_{t}=c+\epsilon_{t}+\theta_{1}\epsilon_{t-1}+\dots+\theta_{q}\epsilon_{t-q}$
(10)

Here, $\epsilon_{t}$ is an error term, and c is the average of the long-term
drift. $\theta_{t}$ can be estimated through maximum likelihood estimation
Hyndman and Athanasopoulos [2018].

Hence, now that we have outlined the three components making up ARIMA, we can
for a given time-series difference a given number of times to make it
stationary, and combine equation (9) and equation (10), to form
$ARIMA(p,d,q)$, where $p$ represents the order of the autoregressive
component, $d$ represents the degree of differencing, and $q$ is the order of
the moving average component. That is,

$y^{\prime}_{t}=c+\phi_{1}y^{\prime}_{t-1}+\dots+\phi_{p}y^{\prime}_{t-p}+\theta_{1}\epsilon_{t-1}+\dots+\theta_{q}\epsilon_{t-q}+\epsilon_{t}$
(11)

###  4.2 Exponential Smoothing

Many Exponential Smoothing (ES) models are special cases of ARIMA models, and
made up of an exponentially decaying autocorrelation forecast [Hyndman and
Athanasopoulos, 2018]. Furthermore, there are also parameters for adjusting
with respect to both trend and seasonality. These adjustments will be made
either linearly or multiplicatively, depending on the type of trend and
seasonality observed in the time series. This subsection will describe the
components which make up the linear ES-model, in addition, to describe how
these parameters can be used to make both a linear and a non-linear ES-model.

####  4.2.1 Simple Exponential Smoothing

The simplest form for an exponential smoothing algorithm is a weighted average
of the past observations, where the weights have a decreasing magnitude as we
move away in time from the event we try to predict. We use the parameter
$\alpha$ to denote this decay and $h$ as a step length. As a result,
$\hat{y}_{t+1|t}$ is predicted in the following manner [Hyndman and
Athanasopoulos, 2018]:

$\begin{split}&\textrm{Forecast Equation: }\hat{y}_{t+h|t}=l_{t}\\\
&\textrm{Smoothing Equation: }l_{t}=\alpha y_{t}+(1-\alpha)l_{t-1}\end{split}$
(12)

A higher value of $\alpha$ will result in more weight on the most recent
observations.

####  4.2.2 Modeling trends with Exponential Smoothing

Time series often include trends. In order to model this in the ES algorithm,
a method known as Holt’s linear method is used [Hyndman and Athanasopoulos,
2018]. There are here two components, which makes up the forecasted value, a
level equation and a trend equation.

$\begin{split}&\textrm{Forecast Equation: }\hat{y}_{t+h}=l_{t}+hb_{t}\\\
&\textrm{Level Equation: }l_{t}=\alpha y+(1-\alpha)(l_{t-1}+b_{t-1})\\\
&\textrm{Trend Equation:
}b_{t}=\beta(l_{t}-l_{t-1})+(1-\beta)b_{t-1}\end{split}$ (13)

Now, the level equation is based on a weighted average between the last trend
parameter, $b_{t-1}$, and the last observed value of $y_{t}$. The $\beta$ will
be a smoothing parameter for the trend and will have a value in the range
$[0,1]$. Depending on how much one desire to weight the new level differences
as a trend or used the past calculation of the trend.

The equations above now show a forecasting expanding to infinity for large
values of h. Therefore, it is relevant to find methods for limiting the growth
of $\hat{y}$, when h takes a significant value. This is known as damping.
Mathematically, we can swap $h$ for $(\phi+\phi^{2}+...+\phi^{h})$, and
multiply $b_{t}$ with $\phi$ in the trend equation. $\phi$ will be in the
range $<0,1>$.

####  4.2.3 Modeling Seasonality with Exponential Smoothing

For modelling seasonality with ES-models, a third smoothing parameter is
required, in addition to a value to indicate the cyclical fluctuations in the
seasonality. The smoothing parameter is $\gamma$, and the cycles have a
duration of m with respect to the duration of a time unit. The forecast is
written [Hyndman and Athanasopoulos, 2018]:

$\begin{split}&\textrm{Forecast Equation:
}\hat{y}_{t+h}=l_{t}+hb_{t}+s_{t+h-m(k+1)}\\\ &\textrm{Level Equation:
}l_{t}=\alpha(y_{t}-s_{t-1})+(1-\alpha)(l_{t-1}+b_{t-1})\\\ &\textrm{Seasonal
Equation: }s_{t}=\gamma(y_{t}-l_{t-1}-b_{t-1})+(1-\gamma)s_{t-m}\end{split}$
(14)

The parameter $k$ is the integer part of $(h-1)/m$, hence, indicating the
number of whole cycle lengths we forecast in the future.

###  4.3 Discussion and Comparison of LSTM with Statistical Methods

When reading papers comparing the statistical approaches to forecasting with
those of deep learning, it is obvious that the results are somewhat diverging.
Papers are describing that advanced machine learning models are capable of
extracting highly complex relationships in the data, and utilize these to have
superior forecasting compared to ARIMA and ES [Sima Siami-Namini, 2019]. On
the other hand, some papers have found that the machine learning models fit
the training data and perform poorly in testing, due to over-fitting [Spyros
Makridakis, 2018].

First of all, the article [Spyros Makridakis, 2018] concludes that an LSTM
approach for forecasting single and multiple steps of a time series, based on
univariate data, is a problem best solved using an ES or ARIMA model. They
have found that the LSTM is capable of learning the relationships of the
training data, but lacks the capability to predict new values. In other words,
the LSTM will over-fit the training data. They credit the ARIMA model for
restricting complexity, and therefore reducing the generalization error in the
testing phase. Furthermore, the article [Spyros Makridakis, 2018] discusses
how machine learning algorithms need more insight into the objective of the
time series to make accurate predictions.

Despite what has been described in the paragraph above, some papers strongly
argue that machine learning models, such as the LSTM, are superior to
statistical approaches [Sima Siami-Namini, 2019]. The research they have
conducted strongly indicate that the LSTM outperforms the ARIMA model.
Furthermore, they have made the study based on some of the most complex graphs
available - stock market graphs. Hence, this strongly advocates for the
superiority of the LSTM. The core of their argumentation for the superiority
of the LSTM is the gradient descent optimization algorithm.

Combining the studies we have discussed so far, it is evident that there are
diverging answers to whether some of the statistical approaches to time series
forecasting are better or worse than an LSTM approach. It is hard to arrive at
a precise conclusion. Therefore, we will discuss some of the issues we have
seen in the papers, and give some guidance for choosing a time series
forecasting model.

When constructing models as complex as both the ARIMA, ES and LSTM, the
hyperparameters are paramount to ensure a model which can capture and
communicate the relationships in the fluctuations of the time series data.
Since some of the algorithms had standardized hyperparameters in their
construction, it is possible to argue for the fact that they were not able to
forecast at their best.

When reading the research papers, it is evident that it is not a standardized
data set which has been used, but many different ones. Hence, the quantitative
assessment of the different models’ time series forecasting abilities is not
that comparable across the research papers.

Therefore, we can conclude that there are options to the LSTM models for time
series forecasting. These models can be made linear and are in many cases
great at doing time series forecasting. As a result, one should not completely
disregard the more statistical processes of time series analysis, but use
these as a reference and tool, since they are a lot less computationally
demanding. Furthermore, we believe that as the world progresses and technology
with it, we will discover a lot more complex and precise models for time
series analysis, as stated in [Spyros Makridakis, 2018]. This is evident in
the democratization process of computer power and the availability of open-
source code.

##  5 Conclusions and Further Work

This paper explains and presents LSTM cells and tie them to the time series
domain. We have seen how LSTMs differ from traditional RNNs through their
ability to deal with the vanishing gradient problem. LSTMs are also compared
to more structured, traditional statistical models such as ARIMA, where we
highlight the discussion around the applicability of LSTM compared to the
traditional statistical approaches. Later on, different applications of the
LSTM architecture are also presented, where we find that it achieves great
results on predicting future power generation. Furthermore, we have seen how
and why LSTMs can be applied to tasks belonging to other domains, such as NLP,
and on their ability to create robust embeddings from language models.

Looking at further work, several improvements can increase the performance of
LSTMs architectures. One increasingly popular implementation is deep-LSTM
(DLSTM), where multiple LSTM cells are layer-wised stacked on top of each
other to build more complex realizations of the data. Others are through using
the concept of self-attention, outlined by [Vaswani et al., 2018], where LSTM
architectures can be adapted to focus more on relevant parts for a given data
[Lim and Stefan, 2020]. Furthermore, methods such as BERT, a state-of-the-art
encoder architecture released by Google in 2018, which builds upon the
encoder-decoder thought from LSTM, can be used [Devlin et al., 2019].
Furthermore, many statistical, structured forecasting methods can also be used
in combination with neural networks to yield great results on time series
data.

As the world becomes more data-driven than ever before, we believe that with
increasing computing power, and more sharing of knowledge through the open-
source community, machine learning models building on LSTM architectures will
both become more widespread and accurate. We also want to emphasize that deep
learning is not necessarily the best solution; classical statistical methods
such as ARIMA can perform at least as good or even better depending on the
dataset used or the available resources. We also want to emphasize the
challenge of interpreting LSTM models. Increased depth and complexity of
machine learning models can make it even harder to explain why models do as
they do, which can be a severe threat under certain circumstances. To
conclude, the choice of time series model depends on the data available and
the data scientist’s knowledge about structures in the data. Under the right
conditions, we have seen that the power of LSTMs and deep learning can result
in groundbreaking results.

## References

  * Alanis and Sanchez [2017] Alma Y. Alanis and Edgar N. Sanchez.  Chapter 2 - mathematical preliminaries.  In Alma Y. Alanis and Edgar N. Sanchez, editors, _Discrete-Time Neural Observers_ , pages 9 – 22. Academic Press, 2017.  doi: https://doi.org/10.1016/B978-0-12-810543-6.00002-2.
  * Barker [2020] J. Barker.  Machine learning in M4: What makes a good unstructured model?  _International Journal of Forecasting_ , 36:150–155, 2020\.
  * Bontempi et al. [2013] G. Bontempi, Y. L. Borgne, and S. B. Taieb.  Machine Learning Strategies for Time Series Forecasting.  _Lecture Notes in Business Information Processing_ , 2013.
  * Bouktif et al. [2018] Salah Bouktif, Ali Fiaz, Ali Ouni, and Mohamed Serhani.  Optimal deep learning lstm model for electric load forecasting using feature selection and genetic algorithm: Comparison with machine learning approaches †.  _Energies_ , 11:1636, 06 2018.  doi: 10.3390/en11071636.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * Gensler et al. [2016] A. Gensler, J. Henze, B. Sick, and N. Raabe.  Deep Learning for solar power forecasting - An approach using AutoEncoder and LSTM Neural Networks.  _IEEE International Conference on Systems, Man and Cybernetics (SMC)_ , 2016.
  * Goodfellow et al. [2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  _Deep Learning Book_.  2016\.  https://deeplearningbook.org. Accessed on 12.10.2020.
  * Guo [2013] Jiang Guo.  Backpropagation through time.  2013\.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9:1735–80, 12 1997.  doi: 10.1162/neco.1997.9.8.1735.
  * Hyndman and Athanasopoulos [2018] R.J. Hyndman and G. Athanasopoulos.  _Forecasting: principles and practice, 2nd. edition_.  2018\.  https://OTexts.com/fpp2. Accessed on 09.10.2020.
  * Le and Zuidema [2016] Phong Le and Willem Zuidema.  Quantifying the vanishing gradient and long distance dependency problem in recursive neural networks and recursive lstms, 2016.
  * Lim and Stefan [2020] Bryan Lim and Zohren Stefan.  Time series forecasting with deep learning: A survey.  2020\.
  * Mikolov et al. [2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In _Advances in Neural Information Processing Systems 26_ , pages 3111–3119. 2013.  URL http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf.
  * Nau [2020] Robert Nau.  Statistical forecasting: Notes on regression and time series analysis, 2020.
  * Nordanger [2020] Ole P. Nordanger.  Blackboard. discussion forum under time-series - ”contextualize lstm”.  2020\.
  * Peters et al. [2018] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer.  Deep Contextualized word representations.  _Allen Institute for Artificial Intelligence_ , 2018.
  * Rajpurkar [2020] P Rajpurkar.  Stanford question answering dataset (squad) 2.0 - leaderboard, 2020.  URL https://rajpurkar.github.io/SQuAD-explorer/.
  * Sima Siami-Namini [2019] Akbar Siami Namin Sima Siami-Namini, Neda Tavakoli.  A comparative analysis of forecasting financial time series using arima, lstm, and bilstm.  2019\.  https://arxiv.org/pdf/1911.09512.pdf.
  * Spyros Makridakis [2018] Vassilios Assimakopoulos Spyros Makridakis, Evangelos Spiliotis.  Statistical and machine learning forecasting methods: Concerns and ways forward.  2018\.  https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0194889.
  * Vaswani et al. [2018] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  2018\.
  * Zhang et al. [2020] Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola.  _Dive into Deep Learning_.  GitHub, 2020.  https://d2l.ai.
  * Zuzuki [2015] A. Zuzuki.  Forecaster introduction, 2015.  URL https://www.dnvgl.com/services/forecaster-introduction-3848.  [Online; accessed 17. september 2020].