# Differential Entropy Rate Characterisations of Long Range Dependent
Processes

Andrew Feutrill, and Matthew Roughan

###### Abstract

A quantity of interest to characterise continuous-valued stochastic processes
is the differential entropy rate. The rate of convergence of many properties
of long range dependent (LRD) processes is slower than might be expected,
based on the intuition for conventional processes, _e.g. Markov processes_. Is
this also true of the entropy rate?

In this paper we consider the properties of the differential entropy rate of
stochastic processes that have an autocorrelation function that decays as a
power law. We show that power law decaying processes with similar
autocorrelation and spectral density functions, Fractional Gaussian Noise and
ARFIMA(0,d,0), have different entropic properties, particularly for negatively
correlated parameterisations. Then we provide an equivalence between the
mutual information between past and future and the differential excess entropy
for stationary Gaussian processes, showing the finiteness of this quantity is
the boundary between long and short range dependence. Finally, we analyse the
convergence of the conditional entropy to the differential entropy rate and
show that for short range dependence that the rate of convergence is of the
order $O(n^{-1})$, but it is slower for long range dependent processes and
depends on the Hurst parameter.

##  1 Introduction

The entropy rate of discrete time stochastic processes has been studied as a
measure of the average uncertainty. Most investigations of this type have
focussed on processes whose correlations decay quickly, and hence the
dependence on past observations disappears rapidly. However, many real
processes from a variety of contexts, i.e., data networks (e.g., Leland et al.
(1993); Willinger et al. (1997, 1995)), climate (e.g., Varotsos and Kirk-
Davidoff (2006)), hydrology (e.g., Beran (1994); Lawrance and Kottegoda
(1977)), and economics (e.g., Willinger et al. (1999); Cont (2005)), have been
shown to exhibit long range dependence, meaning correlations exist between
past and future observations that cannot be ignored at any time lag.

Information and coding theory have had profoundly important uses in signal
processing and communication. Noise processes are an important part of this
story. However, in most works, for instance on designing optimal codes on
noisy channels, the noise processes are presumed to be short range dependent.
However, as far back as 1965, Mandelbrot (1965) showed that some noise
processes are also long-range dependent (though the terminology was still
developing).

Recent work has investigated an information theoretic characterisation of long
range and short range processes (e.g. Li (2006); Chavez (2016); Ding and Xiang
(2016)), using the finiteness of mutual information between past and future.
We aim to clarify this characterisation and investigate its implications.

This paper calculates the differential entropy rate for the two most common
stationary Gaussian Long Range Dependent (LRD) processes: Fractional Gaussian
Noise (FGN) and the Auto-Regressive Fractionally-Integrated Moving Average
(ARFIMA) process. We start by deriving the entropy rate for these processes,
and show that they both have negative poles as the processes tend towards
strong long-range correlations, but that their behaviour when anti-correlated
is surprisingly different: FGN has a pole similar to that for positive
correlations, but ARFIMA does not. This contradicts common intuition based on
their similar spectral densities that FGN and ARFIMA(0,d,0) are close to
equivalent.

We also investigate the links between the two information measures: excess
entropy and the mutual information between past and future processes, and
compare these to the differential entropy rate. We show that the differential
entropy rate definition for excess entropy is equivalent to the mutual
information between past and future for continuous valued discrete time
Gaussian processes, and hence that excess entropy is infinite for all long
range dependent Gaussian processes.

Finally, estimators, such as the sample mean, applied to LRD processes have
been shown to have slow convergence rates, which can lead to a larger than
expected uncertainty when investigating these processes. We ask, “Does this
behaviour apply to estimators of entropy?” and “What is the impact of the
degree of positive or negative correlations on the entropy rate?” We show that
while the convergence rate of the conditional entropy of short range dependent
Gaussian processes is in the order of $n^{-1}$, the rate of convergence for
LRD Gaussian processes is slower. Although this parallels many of the other
results for LRD processes, the actual rate of convergence is different at
$O\Big{(}(1-2H)^{2}\log(n)/n\Big{)}$, where $H$ is the Hurst parameter. As
$H\rightarrow 1$ we can see that this convergence is at its worst, though
$\mathcal{H}$ only appears as a factor not in the exponent.

##  2 Background

###  2.1 Long range dependence

LRD refers to a process where correlations decay slower over time such that
the future is non-trivially dependent on the past no matter how far forward we
proceed. A sample path of an LRD fractional Gaussian noise process, with
$H=0.8$, is shown in Figure 1. As is typical for LRD processes, there is the
appearance of long periods of upwards and downwards “trends”, even though the
process is stationary.

LRD can be defined in two equivalent ways, via the autocorrelation or spectral
density. These definitions are related through the Fourier Transform
(Brockwell and Davis, 1986, pg. 117), and are equivalent via the Kolmogorov
Isomorphism Theorem (Bingham (2012)). The following statement defines the
concept of long range dependence in terms of its autocorrelation function. For
reference, the autocovariance function,
$\gamma(k)=\mathds{E}[\left(X_{n+k}-\mu\right)\left(X_{n}-\mu\right)]$ for a
process with mean $\mu$, and autocorrelation coefficient for $k$ lags,
$\rho(k)$, are related by $\rho(k)=\gamma(k)/\gamma(0)=\gamma(k)/\sigma^{2}$,
where $\sigma^{2}$ is the variance of the stochastic process. Note we are
using the autocorrelation coefficient and the the autocorrelation function,
$\mathds{E}[X_{n+k}X_{n}]$, this is consistent with many works in the analysis
of LRD processes.

######  Definition 2.1.

Let $\\{X_{n}\\}_{n\in\mathbb{N}}$ be a stationary process. If there exists
$\alpha\in(0,1)$, and $c_{\gamma}>0$, such that the auto-covariance
$\gamma(k)$ satisfies

$\displaystyle\lim\limits_{k\rightarrow\infty}\frac{\gamma(k)}{c_{\gamma}k^{-\alpha}}=1,$

then we say that the process is long range dependent.

The equivalent definition in the frequency domain considers the limit of the
spectral density near the origin.

######  Definition 2.2.

Let $\\{X_{n}\\}_{n\in\mathbb{N}}$ be a stationary process. If there exists
$\beta\in(0,1)$, and $c_{f}>0$, such that the spectral density $f(\lambda)$
satisfies

$\displaystyle\lim\limits_{\lambda\rightarrow
0}\frac{f(\lambda)}{c_{f}|\lambda|^{-\beta}}=1,$

then we say that the process is long range dependent.

The concept of LRD is often characterised by the Hurst parameter, $H$, which
measures the strength of the correlations between the past and present of a
stochastic process. It is perhaps unfortunate that $H$ is used as standard
notation both for the Hurst parameter, and for Shannon entropy, but we shall
side-step that issue here as we are mainly concerned with entropy rates, which
we will designate with lower-case $h$.

Hurst (1951) developed the parameter when he was measuring flows in the Nile
River. There is a relationship between $H$ and the $\alpha$ in Definition 2.1,
namely

$H=1-\alpha/2.$

LRD processes have $\alpha\in(0,1)$ and hence have $H\in(0.5,1)$. The
parameter takes on values between 0 and 1, with $H>\nicefrac{{1}}{{2}}$
representing a positively correlated process and $H<\nicefrac{{1}}{{2}}$
representing a negatively correlated process and $H=\nicefrac{{1}}{{2}}$ being
a short range correlated process, such as white Gaussian noise.

Figure 1: Sample path of Fractional Gaussian Noise with Hurst parameter,
$H=0.8$. We can see that the high correlations lead to the appearance of
longer trends than we would expect for an independent and identically
distributed process.

Another property of note, that has been used as the definition of LRD
processes, is the (un)summability of the autocorrelation function. For a LRD
processes the sum of the autocovariances diverges, i.e.,
$\sum_{k=1}^{\infty}\gamma(k)\rightarrow\infty$, whereas for SRD processes
this is finite (Beran (1994)). Then we can interpret LRD as having such strong
correlations that the autocovariance values decay such that the distant past
still influences the future. The intuition for non-LRD processes is that the
autocovariance function decays exponentially, or quicker, and in many cases
this means, for analysis, we can ignore correlations beyond some short lag.

The negatively correlated processes, $H<\nicefrac{{1}}{{2}}$, have not
received as much consideration at the SRD and LRD cases but we include them in
the analysis here. These are similar to short range dependent processes, such
as white Gaussian noise, as they still have a summable autocorrelation
function (see Beran (1994); Gefferth et al. (2003)). In fact their structure
enforces that $\sum_{k=1}^{\infty}\gamma(k)=0$. This is quite a strict and
surprising property, and hence Gefferth et al. (2003) called these processes
constrained short range dependent (CSRD).

We will be working with Gaussian processes in this paper.

######  Definition 2.3.

A stochastic process is a Gaussian process if and only if each finite
collection of random variables from the process has a multivariate Gaussian
distribution.

This definition applies to both discrete and continuous time processes, though
here we are principally interested in the former. Gaussian processes are
completely characterised by their second order statistics (the mean and
autocovariance function) (Box et al. (2015)), which makes them the primary
type of stochastic model used in this context.

###  2.2 Entropy rate

As we are considering continuous random variables in this paper, we will be
considering differential entropy, which is a continuous extension of Shannon
entropy for discrete random variables. In this paper, we will be using the
natural logarithm in all of the definitions, and hence the units of entropy
that we will be working with are nats. We include standard definitions in
order that all notation be precisely defined.

######  Definition 2.4.

The differential entropy, $h(X)$, of a continuous random variable with
probability density function, $f(x)$, is defined as,

$\displaystyle h(X)=-\int_{\Omega}f(x)\log f(x)\,dx,$

where $\Omega$ is the support of the random variable.

Differential entropy has some important properties which are different from
Shannon entropy. For example, differential entropy can be negative, or even
diverge to $-\infty$, which we can see by considering the Dirac delta
function, $\delta(x)$. The delta, i.e., the unit impulse, is defined by the
properties $\delta(x)=0$ for $x\neq 0$ and
$\int_{-\infty}^{\infty}\delta(x)\,dx=1$. The Dirac delta can be thought of in
terms of probability as a completely determined point in time, that is, a
function possessing no uncertainty. It can be constructed as the limit of
rectangular pulses of constant area 1 as their width decreases, and hence we
can calculate the entropy of the delta as

$\displaystyle h(X)$
$\displaystyle=-\int_{-a}^{a}\frac{1}{2a}\log\left(\frac{1}{2a}\right)dx,$
$\displaystyle=\log(2a),$

which tends to $-\infty$ as $a\rightarrow 0$.

The intuition for $h(X)=-\infty$ from Cover and Thomas (2006) is that the
number of bits (note we are working in nats) on average required to fix a
random variable, $X$ to n-bit accuracy is $h(X)+n$. Meaning $h(X)=-\infty$,
can be read as requiring $n-\infty$ bits, or that we can describe the random
variable arbitrarily accurately without using any bits.

We will see the same type of asymptotic behaviour for LRD processes as
$H\rightarrow 1$. Effectively, in the limit the correlations in the process
straight-jacket it, such that the future is completely determined by the past,
and so the incremental uncertainty in the process is the same as that of the
delta.

The differential entropy can be extended into the multivariate case and hence
to stochastic processes using the joint entropy for a collection of random
variables.

######  Definition 2.5.

The joint differential entropy of a collection of random variables
$X_{1},X_{2},...,X_{n}$, with density $f(x_{1},x_{2},...,x_{n})=f(\mathbf{x})$
is defined as,

$\displaystyle h(X_{1},X_{2},...,X_{n})=-\int_{\Omega}f(\mathbf{x})\log
f(\mathbf{x})\,d\mathbf{x},$

where $\Omega=\Omega_{1}\times\Omega_{2}\times...\times\Omega_{n}$ is the
support of the random variables.

Similarly, the conditional differential entropy can be defined for a random
variable, given knowledge of other variables.

######  Definition 2.6.

The conditional differential entropy of a random variable, $X_{n}$, given a
collection of random variables $X_{1},X_{2},...,X_{n-1}$, with a joint density
$f(x_{1},x_{2},...,x_{n})=f(\mathbf{x})$, is defined as,

$\displaystyle h(X_{n}|X_{n-1},...,X_{1})=-\int_{\Omega}f(\mathbf{x})\log
f(x_{n}|x_{n-1},...,x_{1})d\mathbf{x},$

where $\Omega=\Omega_{1}\times\Omega_{2}\times...\times\Omega_{n}$ is the
support of the random variables.

Finally, we define the concept of differential entropy rate, which can be
thought of as the average amount of new information from each sample of a
random variable in a discrete time process.

######  Definition 2.7.

Where the limit exists, the differential entropy rate of a stochastic process
$\raisebox{0.0pt}{$\chi$}=\\{X_{i}\\}_{i\in\mathbb{N}}$ is defined to be,

$\displaystyle h(\raisebox{0.0pt}{$\chi$})$
$\displaystyle=\lim\limits_{n\rightarrow\infty}\frac{h(X_{1},...,X_{n})}{n}.$

An example of a process which is non-stationary but has a differential entropy
rate is the Gaussian walk, $S_{n}$. This is defined as the process of sums of
i.i.d. normally distributed random variables, i.e.,
$S_{n}=\sum_{i=1}^{n}X_{i}$, where $X_{i}\sim\mathcal{N}(0,\sigma^{2})$. The
process has mean 0 for all $n$, however it is non-stationary as the variance
depends on $n$ as,

$\displaystyle\mbox{Var}(S_{n})=\mbox{Var}\left(\sum_{i=1}^{n}X_{i}\right)=n\sigma^{2}.$

However, the entropy rate converges and is equal to

$\displaystyle\lim\limits_{n\rightarrow\infty}\frac{h(S_{1},...,S_{n})}{n}$
$\displaystyle=\lim\limits_{n\rightarrow\infty}\frac{\sum_{i=1}^{n}h(S_{i}|S_{i-1},\ldots,S_{1})}{n},$
$\displaystyle=\lim\limits_{n\rightarrow\infty}\frac{nh(X_{i})}{n},$
$\displaystyle=h(X_{i}),$

as each random variable $X_{i}$ is independent.

An alternative characterisation of the differential entropy rate is given by
the following theorem for stationary processes. This was developed for the
Shannon entropy of discrete processes (Cover and Thomas, 2006, Theorem 4.2.1),
however this has been extended to differential entropy (Cover and Thomas,
2006, pg. 416).

######  Theorem 2.1 (Theorem 4.2.1 Cover and Thomas (2006)).

For a stationary stochastic process,
$\raisebox{0.0pt}{$\chi$}=\\{X_{i}\\}_{i\in\mathbb{N}}$, the entropy rate is
equal to,

$\displaystyle h(\raisebox{0.0pt}{$\chi$})$
$\displaystyle=\lim\limits_{n\rightarrow\infty}h(X_{n}|X_{n-1},...,X_{1}).$

The second equivalent definition is useful because it will allow us to analyse
the convergence rates of conditional entropy to the differential entropy rate,
which is important in estimation of entropy rates.

##  3 Entropy rate function for Fractional Gaussian Noise

We want to understand the effect of memory on the entropic properties of a
stochastic process. We start with the entropy rate characterisation for
Gaussian processes originally derived by Kolmogorov ((Ihara, 1993, pg. 76))

$h(\raisebox{0.0pt}{$\chi$})=\frac{1}{2}\log(2\pi
e)+\frac{1}{4\pi}\int_{-\pi}^{\pi}\log(2\pi f(\lambda))\,d\lambda,$ (1)

where $f(\lambda)$ is the spectral density, i.e., the Fourier transform of the
autocovariance function for a mean zero process.

We’ll begin by investigating the spectral density of Fractional Gaussian Noise
(FGN), which is given by Beran (1994) as

$f(\lambda)=2c_{f}(1-\cos\lambda)\sum_{j=-\infty}^{\infty}|2\pi
j+\lambda|^{-2H-1},$ (2)

where, $c_{f}=\frac{\sigma^{2}}{2\pi}\sin(\pi H)\Gamma(2H+1)$, $H$ is the
Hurst parameter, and $\sigma^{2}$ is the variance of the process.

This spectral density is difficult to analyse as it has an infinite sum of
absolute values. In particular, when we apply the entropy rate
characterisation 1, as it involves taking a logarithm of a sum, making
analytical calculation prohibitively difficult. However, we can still use this
expression to derive some properties of the entropy rate of FGN processes.

###  3.1 Comparison of approximate and analytical spectral density for entropy
rate calculation

Substituting the spectral density of FGN (2) into the second term in the
entropy rate expression (1) we get

$\displaystyle\int_{-\pi}^{\pi}$ $\displaystyle\log\big{(}2\pi
f(\lambda)\big{)}\,d\lambda$ $\displaystyle=2\pi\log(4\pi
c_{f})+\int_{-\pi}^{\pi}\log(1-\cos\lambda)d\lambda+\int_{-\pi}^{\pi}\log\left(\sum_{j=-\infty}^{\infty}|2\pi
j+\lambda|^{-2H-1}\right)d\lambda,$ $\displaystyle=2\pi\log(4\pi
c_{f})-2\pi\log 2+\int_{-\pi}^{\pi}\log\left(\sum_{j=-\infty}^{\infty}|2\pi
j+\lambda|^{-2H-1}\right)d\lambda.$

The last term is finite for all $H\in(0,1)$, since the singularity that exists
when $\lambda=j=0$ in the absolute value term is integrable. This is important
as we can then see that this does not affect the asymptotic behaviour of FGN
processes. The resulting entropy rate is

$\displaystyle h(\raisebox{0.0pt}{$\chi$})=\frac{1}{2}$
$\displaystyle\log(2\pi e)+\frac{1}{2}\log\left(2\sigma^{2}\sin(\pi
H)\Gamma(2H+1)\right)$
$\displaystyle+\frac{1}{4\pi}\int_{-\pi}^{\pi}\log\left(\sum_{j=-\infty}^{\infty}|2\pi
j+\lambda|^{-2H-1}\right)d\lambda.$

We calculate $h(\raisebox{0.0pt}{$\chi$})$ using numerical integration via
Python’s Scipy library (Virtanen et al. (2020)). We plot the differential
entropy rate of Fractional Gaussian Noise as a function of the Hurst
parameter, H, in Figure 2. The plot shows the impact of the variance on
entropy rate calculation, and hence that the entropy rate of Fractional
Gaussian Noise has a large dependence on the variance, _i.e._ the second order
properties. Each unit increase in variance has a smaller effect on the value
of the differential entropy, due to the $\log(\sigma^{2})$ term.

The spectral density expression is quite cumbersome to work with and an
approximation is often used, which is accurate at low frequencies (Beran,
1994, pg. 53). It is derived from a Taylor series expansion of the spectral
density and is given by,

$\displaystyle f(\lambda)\approx c_{f}|\lambda|^{1-2H}.$

We can obtain a closed form for the entropy rate if we substitute this
approximation into the integral in the entropy rate expression (1) to get

$\displaystyle\int_{-\pi}^{\pi}\log(2\pi f(\lambda))\,d\lambda$
$\displaystyle=\int_{-\pi}^{\pi}\log\left(2\pi
c_{f}\right)d\lambda+\int_{-\pi}^{\pi}\log\left(|\lambda|^{1-2H}\right)\,d\lambda,$
$\displaystyle=2\pi\log\left(2\pi
c_{f}\right)+2\left(1-2H\right)\int_{0}^{\pi}\log\left(\lambda\right)\,d\lambda,$
$\displaystyle=2\pi\log\left(2\pi
c_{f}\right)+2\left(1-2H\right)\left(\pi\log\pi-\pi\right).$

Note that there is a singularity at the origin of the spectral density of LRD
processes. However, the integral is still well defined and finite in this
case. Therefore the entropy rate approximation is,

$\displaystyle\tilde{h}(\raisebox{0.0pt}{$\chi$})=\frac{1}{2}\log(2\pi
e)+\frac{1}{2}\log 2\pi c_{f}+(1-2H)(\log\pi-1),$

which differs from the exact formulation only in the last term.

Figure 2: Entropy rate of Fractional Gaussian Noise as a function of the Hurst
Parameter. The maximum is at $H=0.5$, where the process is white Gaussian
noise. As $H\rightarrow 0$ or $1$, the function tends towards $-\infty$, as
the strength of the negative or positive correlations increase. The impact of
changing variance decreases as the variance increase, due to the
$\log(\sigma^{2})$ term. Figure 3: Comparison of the numerically integrated
spectral density and the spectral density approximation. The approximation is
relatively good for $H\geq\nicefrac{{1}}{{2}}$ but an underestimate for
$H\leq\nicefrac{{1}}{{2}}$.

Figure 3 shows the entropy rate and its approximation. We can see that the
entropy rate approximation is very good for the positively correlated cases
$H\geq 0.5$ and at the limits around $H=0$ or $1$. However for moderately,
negatively-correlated processes the approximation is a noticeable
underestimate of the entropy rate.

###  3.2 Properties of Entropy rate for Fractional Gaussian Noise

Figure 3 shows some interesting properties

  * •

The entropy rate as a function of $H$ is not symmetric. Negatively correlated
processes seem to have higher uncertainty the same distance from $H=0.5$.

  * •

The entropy rate asymptotically tends to $-\infty$ as $H\rightarrow$ 0 or 1.

  * •

The maximum entropy rate occurs at 0.5. Indicating that the maximum entropy
occurs for white Gaussian noise.

We explain how these properties emerge below.

####  3.2.1 Asymptotic behaviour

######  Theorem 3.1.

The approximate differential entropy rate of Fractional Gaussian Noise,
$\tilde{h}(\raisebox{0.0pt}{$\chi$})\rightarrow-\infty$ as $H\rightarrow 0$ or
$1$.

###### Proof.

When $H\rightarrow 0$ or $1$, the term $c_{f}\rightarrow 0$, as the gamma
function terms are finite, however the trigonometric terms tend to 0 as $H$
tends to an integer value. Hence, asymptotically the approximate entropy rate
expression is dominated by $\log c_{f}\rightarrow-\infty$, as
$c_{f}\rightarrow 0$. ∎

######  Remark.

Note that the approximation works well in the limits $\mathcal{H}\rightarrow
0$ or 1, and so the theorem describes the asymptotic behaviour of entropy rate
well. Moreover, the theorem lines up with the intuition for an LRD process. As
we move closer to either perfectly positively or negatively correlated, the
process becomes “less uncertain”, _i.e._ , we have less entropy on average.
When the uncertainty disappears, by viewing the entire past we can accurately
infer the current value. It’s important to reiterate that the differential
entropy can be $-\infty$, which can be interpreted as least uncertainty for a
process.

####  3.2.2 Maximum

We want to understand the maximum of differential entropy rate, as a function
of the Hurst parameter. This will provide an understanding of which parameter
choices represent the highest uncertainty. We differentiate the entropy rate,
with respect to $H$ and then solve for $H$ when the derivative equals zero.
Here we need to apply this to the exact formula because the approximation
distorts the location of the maximum. Therefore, dropping constant terms, we
get

$\displaystyle\frac{dh}{dH}$
$\displaystyle=\frac{1}{2}\frac{d}{dH}\log\left(\sigma^{2}\sin(\pi
H)\Gamma(2H+1)\right)$
$\displaystyle\qquad+\frac{1}{4\pi}\frac{d}{dH}\int_{-\pi}^{\pi}\log\left(\sum_{j=-\infty}^{\infty}|2\pi
j+\lambda|^{-2H-1}\right)d\lambda$
$\displaystyle=\frac{1}{2}\frac{d}{dH}\log\big{(}\sin(\pi
H)\big{)}+\frac{1}{2}\frac{d}{dH}\log\big{(}\Gamma(2H+1)\big{)}$
$\displaystyle\qquad-\frac{1}{2\pi}\int_{-\pi}^{\pi}\frac{\sum_{j}\log(|2\pi
j+\lambda|)|2\pi j+\lambda|^{-2H-1}}{\sum_{j}|2\pi
j+\lambda|^{-2H-1}}d\lambda$ $\displaystyle=\frac{\pi}{2}\cot(\pi
H)+\psi(2H+1)-\frac{1}{2\pi}\int_{-\pi}^{\pi}\frac{\sum_{j}\log(|2\pi
j+\lambda|)|2\pi j+\lambda|^{-2H-1}}{\sum_{j}|2\pi
j+\lambda|^{-2H-1}}d\lambda.$

where $\psi(z)=\Gamma^{\prime}(z)/\Gamma(z)$ is the digamma function.

Then we set this expression to zero, and solve for $\mathcal{H}$. This is a
transcendental equation with no closed form. We solve it numerically using
Python’s SciPy package (Virtanen et al. (2020)), which yields
$\mathcal{H}\approx 0.500$. Therefore we conclude that the maximum entropy
rate, using the exact spectral density, is at $\mathcal{H}=0.5$, which aligns
with the idea that a SRD process has more uncertainty than any equivalent LRD
process.

Note that from the solution of the spectral density approximation is $H\approx
0.516$. So although using the spectral density approximation is acceptable for
many purposes, it can lead to false conclusions about the properties of the
differential entropy rate.

##  4 Entropy rate function for ARFIMA(p,d,q)

We consider the differential entropy rate function of a related process to
Fractional Gaussian Noise, which is ARFIMA(p,d,q), the fractional extension of
the ARIMA (Autoregressive Integrated Moving Average) processes, by extending
to non-integer differencing parameters, $d$ (see Hosking (1981); Granger and
Joyeux (1980)). FGN and ARFIMA(0,d,0) are commonly used stationary LRD
processes for modelling real phenomena, and in particular FGN and
ARFIMA(0,d,0) have very similar properties in the time and frequency domains.
Additionally, these processes have been linked by by limit of their
autocorrelation coefficients, $\rho(k):=\gamma(k)/\sigma^{2}$, under
aggregation and rescaling (see Gefferth et al. (2003)). However, ARFIMA
processes do differ from FGN in that you could change the fixed point, _i.e._
, alter the eventual limit under aggregation and rescaling, with the addition
of additive noise (Veitch et al. (2013)), _i.e._ , this class is less robust
to the addition of noise. Hence, there may be some differences in behaviour
when looking through an entropic lens.

Before we define an ARFIMA(p,d,q) process, we define two polynomials that are
required for the ARFIMA definition. These are polynomials of the lag operator,
$L$, where $LX_{n}=X_{n-1}$, defined by

$\displaystyle\phi(x)$ $\displaystyle=1-\sum_{j=1}^{p}\phi_{j}x^{j},\text{for
coefficients }\phi_{j}\text{ and }p\in\mathbb{Z^{+}},$ $\displaystyle\psi(x)$
$\displaystyle=1+\sum_{j=1}^{q}\psi_{j}x^{j},\text{for coefficients
}\psi_{j}\text{ and }q\in\mathbb{Z^{+}}.$

Now we can define an ARFIMA(p,d,q) process,

######  Definition 4.1.

For a stationary stochastic process $\\{X_{i}\\}_{i\in\mathbb{Z}}$, such that

$\displaystyle\phi(L)(1-L)^{d}X_{i}$ $\displaystyle=\psi(L)\epsilon_{i},$

for some $-\nicefrac{{1}}{{2}}<d<\nicefrac{{1}}{{2}}$ and $\epsilon_{i}$, a
zero mean normally distributed random variable with variance
$\sigma_{\epsilon}^{2}$. $\\{X_{i}\\}_{i\in\mathbb{Z}}$ is called a
ARFIMA$(p,d,q)$ process.

There is a connection between the differencing parameter, $d$, and the Hurst
parameter, $H$, for these processes

An ARFIMA(0,d,0) process is a special case of an ARFIMA(p,d,q), where
$\phi(x)=\psi(x)=1$, that is there is no lag on the noise, $\epsilon$, and all
the auto-regressive lags on the previous values come from the differencing
operator.

The spectral density of an ARFIMA(p,d,q) is given by Beran (1994) as

$\displaystyle f(\lambda)$
$\displaystyle=\frac{\sigma_{\epsilon}^{2}|\psi(e^{i\lambda})|^{2}}{2\pi|\phi(e^{i\lambda})|^{2}}|1-e^{i\lambda}|^{-2d}.$
(3)

The following theorem from Hosking (1981) (and in Beran (1994)) gives infinite
autoregressive and moving average representations for ARFIMA(0,d,0) processes.

######  Theorem 4.1 (Proposition 2.2 Beran (1994)).

Let $X_{n}$ be a fractional ARIMA(0,d,0) process with
-$\frac{1}{2}<d<\frac{1}{2}$ . Then
(i) the following infinite autoregressive representation holds:

$\displaystyle\sum_{k=0}^{\infty}\pi_{k}X_{n-k}=\epsilon_{n},$

where $\epsilon_{n}(n=1,2,...)$ are independent identically distributed random
variables and

$\displaystyle\pi_{k}=\frac{\Gamma(k-d)}{\Gamma(k+1)\Gamma(-d)}.$

For $k\rightarrow\infty$ we have,

$\displaystyle\pi_{k}\sim\frac{1}{\Gamma(-d)}k^{-d-1}.$

(ii) The following infinite moving average representation holds:

$\displaystyle X_{n}=\sum_{k=0}^{\infty}a_{k}\epsilon_{n-k}$

where $\epsilon_{n}(n=1,2,...)$ are independent identically distributed random
variables and

$\displaystyle a_{k}=\frac{\Gamma(k+d)}{\Gamma(k+1)\Gamma(d)}.$

For $k\rightarrow\infty$ we have

$\displaystyle a_{k}\sim\frac{1}{\Gamma(d)}k^{d-1}.$

We will express an entropy rate characterisation for ARMA processes in terms
of its innovation process variance, from Ihara (1993), and show that this can
be extended to ARFIMA(0,d,0) and ARFIMA(p,d,q) processes. Then we will use the
result to characterise the entropy rate of an ARFIMA(0,d,0) process in terms
of its process variance.

######  Theorem 4.2 (Proposition 2.2 Ihara (1993)).

The entropy rate of an ARMA(p,q) process is given by,
$h(\raisebox{0.0pt}{$\chi$})=\frac{1}{2}\log(2\pi e\sigma_{\epsilon}^{2})$.

Now, we state our extension to ARFIMA(0,d,0) and present a proof based on
Ihara’s proof of Theorem 4.2

######  Theorem 4.3.

The entropy rate of an ARFIMA(0,d,0) process is given by,
$h(\raisebox{0.0pt}{$\chi$})=\frac{1}{2}\log(2\pi e\sigma_{\epsilon}^{2})$.

###### Proof.

First we calculate $\int_{-\pi}^{\pi}\log(2\pi f(\lambda))d\lambda$, using the
spectral density of an ARFIMA process given in (3)

$\displaystyle\int_{-\pi}^{\pi}\log(2\pi f(\lambda))d\lambda$
$\displaystyle=\int_{-\pi}^{\pi}\log(\sigma_{\epsilon}^{2}|1-e^{i\lambda}|^{1-2H})d\lambda,$
$\displaystyle=\int_{-\pi}^{\pi}\log(\sigma_{\epsilon}^{2})d\lambda+(1-2H)\int_{-\pi}^{\pi}\log|1-e^{i\lambda}|d\lambda.$

Now we transform the elements in the last term using their trigonometric
representation,

$\displaystyle|1-e^{i\lambda}|$
$\displaystyle=|1-\cos(\lambda)-i\sin(\lambda)|,$
$\displaystyle=\sqrt{(1-\cos(\lambda))^{2}+\sin^{2}(\lambda)},$
$\displaystyle=\sqrt{2-2\cos(\lambda)},$
$\displaystyle=\sqrt{4\sin^{2}\left(\frac{\lambda}{2}\right)},$
$\displaystyle=2|\sin\left(\frac{\lambda}{2}\right)|.$

This makes the integral of the log spectral density,

$\displaystyle\int_{-\pi}^{\pi}\log|1-e^{i\lambda}|d\lambda$
$\displaystyle=2\int_{0}^{\pi}\log\left(2\sin\left(\frac{\lambda}{2}\right)\right)d\lambda,$
$\displaystyle=2\int_{0}^{\pi}\log(2)d\lambda+2\int_{0}^{\pi}\log\left(\sin\left(\frac{\lambda}{2}\right)\right)d\lambda,$

We substitute $y=\lambda/2$,

$\displaystyle\int_{-\pi}^{\pi}\log|1-e^{i\lambda}|d\lambda$
$\displaystyle=2\pi\log(2)+2\int_{0}^{\frac{\pi}{2}}\log(\sin y)2dy,$
$\displaystyle=2\pi\log(2)+4\left(-\frac{\pi}{2}\log(2)\right),$
$\displaystyle=0.$

Where the equality $\int_{0}^{\frac{\lambda}{2}}\log(\sin
y)dy=-\frac{\pi}{2}\log(2)$ is given by Koyama and Kurokawa (2005).

So the last term of the spectral density vanishes, and

$\displaystyle\int_{-\pi}^{\pi}\log(2\pi f(\lambda))d\lambda$
$\displaystyle=\int_{-\pi}^{\pi}\log(\sigma_{\epsilon}^{2})d\lambda+(1-2H)\int_{-\pi}^{\pi}\log|1-e^{i\lambda}|d\lambda,$
$\displaystyle=2\pi\log(\sigma_{\epsilon}^{2}).$

Using Kolmogorov’s entropy rate expression, the entropy rate is therefore,

$\displaystyle h(\raisebox{0.0pt}{$\chi$})$
$\displaystyle=\frac{1}{2}\log(2\pi
e)+\frac{1}{4\pi}\left(2\pi\log(\sigma_{\epsilon}^{2})\right),$
$\displaystyle=\frac{1}{2}\log(2\pi e\sigma_{\epsilon}^{2}).$

∎

######  Remark.

This can be shown also using the infinite autoregressive expression above,
$X_{n}=\epsilon_{n}-\sum_{k=1}^{\infty}\pi_{k}X_{n-k}$, and substituting into
the conditional entropy rate for stationary processes,
$h(\raisebox{0.0pt}{$\chi$})=\lim\limits_{n\rightarrow\infty}h(X_{n}|X_{n}-1,...,X_{0})$.
Then we can remove the conditioning from the entropy rate calculation,

$\displaystyle
h(\raisebox{0.0pt}{$\chi$})=\lim\limits_{n\rightarrow\infty}h\left(\epsilon_{n}-\sum_{k=1}^{\infty}\pi_{k}X_{n-k}\bigg{|}X_{n}-1,...,X_{0}\right)=\lim\limits_{n\rightarrow\infty}h(\epsilon_{n}).$

Which then implies that $h(\raisebox{0.0pt}{$\chi$})=\frac{1}{2}\log(2\pi
e\sigma_{\epsilon}^{2})$, _i.e._ , the entropy rate of the process depends
only on the entropy introduced at each step by the innovations. Therefore, we
conclude that the entropy rate of an ARFIMA(0,d,0) process depends on the
innovation variance and not $H$.

We can generalise to ARFIMA(p,d,q) processes by adding an additional
condition, the invertibility of the moving average polynomial. This is an
extremely common condition applied in the theory of autoregressive-moving
average, _i.e._ , ARMA(p,q) processes. The condition implies that all roots of
the moving average polynomial lie outside of the unit circle, and similarly
the stationarity condition of the process ensures that all roots of the
autoregressive polynomial lie outside of the unit circle (Box et al. (2015)).
We will use these conditions on the ARFIMA processes to analyse their
properties.

######  Theorem 4.4.

The entropy rate of a stationary ARFIMA(p,d,q) process with invertible moving
average polynomial is given by,
$h(\raisebox{0.0pt}{$\chi$})=\frac{1}{2}\log(2\pi e\sigma_{\epsilon}^{2})$.

###### Proof.

Since ARFIMA(p,d,q) processes are stationary and invertible, this implies that
the polynomials $\phi(x)$ and $\psi(x)$, have roots outside of the unit
circle, i.e. each root $z\in\mathbb{C}$ is such that $|z|>1$. By the
Fundamental Theorem of Algebra, both the autoregressive and moving average
polynomials can be factored into linear factors. As the constant terms are 1,
this implies the polynomials can be factored as
$\phi(x)=\prod_{i=1}^{p}(1-a_{i}e^{i\lambda})$ and
$\psi(x)=\prod_{i=i}^{q}(1-b_{i}e^{i\lambda})$, where
$|a_{i}|,|b_{i}|<1,\forall i$. Recall from equation (3)that the spectral
density is given by,

$\displaystyle f(\lambda)$
$\displaystyle=\frac{\sigma_{\epsilon}^{2}}{2\pi}|1-e^{i\lambda}|^{-2d}\frac{|\psi(e^{i\lambda})|^{2}}{|\phi(e^{i\lambda})|^{2}}.$

Hence,

$\displaystyle\log((2\pi f(\lambda)))$
$\displaystyle=\log(\sigma_{\epsilon}^{2})-2d\log|1-e^{i\lambda}|$
$\displaystyle\qquad+2\log\left|\prod_{j=1}^{q}(1-a_{j}e^{i\lambda})\right|-2\log\left|\prod_{j=1}^{p}(1-b_{j}e^{i\lambda})\right|,$
$\displaystyle=\log(\sigma_{\epsilon}^{2})-2d\log|1-e^{i\lambda}|$
$\displaystyle\qquad+\sum_{j=1}^{q}2\log|1-a_{j}e^{i\lambda}|-\sum_{j=1}^{p}2\log|1-b_{j}e^{i\lambda}|.$

Now we calculate the integral of the log spectral density,

$\displaystyle\int_{-\pi}^{\pi}\log(2\pi f(\lambda))d\lambda$
$\displaystyle=\int_{-\pi}^{\pi}\log(\sigma_{\epsilon}^{2})d\lambda-\int_{-\pi}^{\pi}2d\log|1-e^{i\lambda}|d\lambda$
$\displaystyle\qquad+\sum_{j=1}^{q}2\int_{-\pi}^{\pi}\log|1-a_{j}e^{i\lambda}|d\lambda-\sum_{j=1}^{p}2\int_{-\pi}^{\pi}\log|1-b_{j}e^{i\lambda}|d\lambda,$
$\displaystyle=2\pi\log(\sigma_{\epsilon}^{2}).$

Where the third equality is given as all the integrals of
$\log|1-ae^{i\lambda}|$ over $[-\pi,\pi]$ vanish for $|a|\leq 1$ (Shiu and
Shakarchi (2004)).

We substitute this expression into Kolmogorov’s entropy rate expression for
Gaussian processes.

$\displaystyle h(\raisebox{0.0pt}{$\chi$})$
$\displaystyle=\frac{1}{2}\log(2\pi
e)+\frac{1}{4\pi}(2\pi\log(\sigma_{\epsilon}^{2})),$
$\displaystyle=\frac{1}{2}\log(2\pi e\sigma_{\epsilon}^{2}).$

∎

This result leads to the following corollary, which can finalise the
discussion of the differential entropy rate in terms of innovation variance
for the classes of AR, MA, ARMA processes. This is relevant as the definition
in terms of the innovation variance is the perspective that is commonly used
in the time series literature, when modelling real world processes.

######  Corollary 4.4.1.

The differential entropy rate of stationary AR(p), invertible MA(q) and,
stationary and invertible ARMA(p,q) processes is
$h(\raisebox{0.0pt}{$\chi$})=\frac{1}{2}\log(2\pi e\sigma_{\epsilon}^{2})$.

Hence, for these models the entropy rate can be calculated in terms of the
variance of its innovations. However we want to compare the entropy rates, as
a function of their Hurst parameter, between ARFIMA(0,d,0) and FGN, so we want
to fix the variance of process itself, $\sigma^{2}$. We will use the
autocovariance function of ARFIMA(0,d,0), from Beran (1994),

$\displaystyle\gamma(k)$
$\displaystyle=\sigma_{\epsilon}^{2}\frac{(-1)^{k}\Gamma(1-2d)}{\Gamma(k-d+1)\Gamma(1-k-d)}.$

Note that $\gamma(0)=\sigma^{2}$,

$\displaystyle\sigma^{2}=\gamma(0)$
$\displaystyle=\sigma_{\epsilon}^{2}\frac{\Gamma(1-2d)}{\Gamma(1-d)^{2}},$

and hence,

$\displaystyle\sigma_{\epsilon}^{2}$
$\displaystyle=\sigma^{2}\frac{\Gamma(1-d)^{2}}{\Gamma(1-2d)}.$

This leads to the following characterisation of ARFIMA(0,d,0) processes in
terms of the Hurst parameter, $H$, noting that $d=H-\nicefrac{{1}}{{2}}$.

######  Corollary 4.4.2.

The entropy rate of an ARFIMA(0,d,0) process for a fixed process variance,
$\sigma^{2}$, is given by,

$\displaystyle h(\raisebox{0.0pt}{$\chi$})=\frac{1}{2}\log(2\pi
e\sigma^{2})+\log\left(\Gamma\left(\frac{3}{2}-\mathcal{H}\right)\right)-\frac{1}{2}\log\bigg{(}\Gamma(2-2\mathcal{H})\bigg{)}.$
(4)

###### Proof.

By Theorem 4.3 and from the characterisation of $\sigma_{\epsilon}^{2}$ above,

$\displaystyle h(\raisebox{0.0pt}{$\chi$})$
$\displaystyle=\frac{1}{2}\log\left(2\pi
e\sigma^{2}\frac{\Gamma(1-d)^{2}}{\Gamma(1-2d)}\right),$
$\displaystyle=\frac{1}{2}\log(2\pi
e\sigma^{2})+\log(\Gamma(1-d))-\frac{1}{2}\log(\Gamma(1-2d)),$
$\displaystyle=\frac{1}{2}\log(2\pi
e\sigma^{2})+\log\left(\Gamma\left(\frac{3}{2}-\mathcal{H}\right)\right)-\frac{1}{2}\log\bigg{(}\Gamma(2-2\mathcal{H})\bigg{)}.$

∎

######  Remark.

The same approach can be used for more general ARFIMA(p,d,q) processes.
However in this case, there is no general closed form for the autocovariance
function, so the variance must be calculated for each process and then
substituted for the innovation variance. Interestingly, this result indicates
that the effect of the changing the process variance is balanced by the effect
of the change in the Hurst parameter, with respect to the innovation variance.
This results in the constant differential entropy rate when considered in
terms of its innovation variance.

Figure 4: The entropy rate of ARFIMA(0,d,0) as a function of the Hurst
parameter, $H$, for variance, $\sigma^{2}=1,2,3,4$. On the positively
correlated side, $H>0.5$, we see a similar asymptotic behaviour to FGN.
However, for negatively correlated processes, the amount of entropy in the
process, stays quite high. We see the maximum of the function at $H=0.5$,
which intuitively shows that the highest uncertainty occurs for the white
Gaussian noise process.

We show the plot of the ARFIMA(0,d,0) entropy rate as a function of the Hurst
parameter, $H$, with process variance, $\sigma^{2}=1,2,3,4$, in Figure 4. The
plot shows some interesting behaviour, particularly when compared to the FGN
entropy rate function in Figure 5. Some of these observed properties are:

  * •

The entropy rate is not symmetric, much less so than FGN. The positively
correlated side has a dramatic drop, however the negatively correlated side
stays relatively high. In order words, there is a demonstrable difference
between FGN and ARFIMA(0,d,0) in the behaviour as CSRD processes.

  * •

The entropy rate asymptotically tends to $-\infty$ as $H\rightarrow$ 1 only.

  * •

The maximum entropy rate occurs at the same point as FGN, $\mathcal{H}=0.5$,
indicating that the maximum entropy occurs for white Gaussian noise.

Figure 5: The comparison of the entropy rate as function of the Hurst
parameter, for both ARFIMA(0,d,0) and FGN processes, with variance 1. It
appears that the ARFIMA(0,d,0) process has an entropy rate which is greater
than or equal to FGN for all values of $H$. The negatively correlated portion
falls away quickly as $H\rightarrow 0$ for FGN but stays relatively high for
the ARFIMA(0,d,0) process. The maximum of the functions coincide at $H=0.5$.

Similar to the previous section, we will prove the asymptotics of the entropy
rate function, and show that the maximum occurs at $H=0.5$.

######  Corollary 4.4.3.

The differential entropy rate of ARFIMA(0,d,0), $h(\raisebox{0.0pt}{$\chi$})$,
tends to negative infinity as $H\rightarrow 1$, for a fixed variance
$\sigma^{2}\in\mathbb{R}$.

###### Proof.

As $H\rightarrow 1$, the term $\Gamma(\frac{3}{2}-H)$ is finite, as well as
the $\frac{1}{2}\log(2\pi e\sigma^{2})$, for a fixed variance
$0<\sigma^{2}<\infty$. Now, as $H\rightarrow 1$, the term
$\Gamma(2-2H)\rightarrow\Gamma(0)$. There exists a singularity for the gamma
function at 0, which diverges to infinity. Which implies that the term
$-\frac{1}{2}\log(\Gamma(2-2H))\rightarrow-\infty$, since
$\Gamma(x)\rightarrow\infty$, as $x\rightarrow 0$. This implies that
$h(\raisebox{0.0pt}{$\chi$})\rightarrow-\infty$, as $H\rightarrow 1$. ∎

######  Remark.

Note that the value of the entropy rate function for an ARFIMA(0,d,0) process
as $H\rightarrow 0$, is

$\displaystyle h(\raisebox{0.0pt}{$\chi$})=\frac{1}{2}\log(2\pi
e\sigma^{2})+\log(\Gamma(\frac{3}{2}))-\frac{1}{2}\log(\Gamma(2))\approx
1.298.$

To complete this section of the analysis, we will consider the maximum of the
entropy rate function of ARFIMA(0,d,0), and conclude which Hurst parameter has
the highest uncertainty, in the sense of maximum differential entropy rate.

######  Theorem 4.5.

The differential entropy rate of ARFIMA(0,d,0) as a function of $H$ attains
the maximum at $H=\nicefrac{{1}}{{2}}$.

###### Proof.

We differentiate the entropy rate function with respect to $H$, and we get

$\displaystyle\frac{dh(\raisebox{0.0pt}{$\chi$})}{dH}$
$\displaystyle=\frac{d}{dH}\left(\frac{1}{2}\log(2\pi
e\sigma^{2})+\log\left(\Gamma(\frac{3}{2}-H)\right)-\frac{1}{2}\log\left(\Gamma(2-2H)\right)\right),$
$\displaystyle=\frac{d}{dH}\left(\log(\Gamma(\frac{3}{2}-H))\right)-\frac{d}{dH}\left(\log(\Gamma(2-2H))\right),$
$\displaystyle=\frac{\Gamma(\frac{3}{2}-H)\psi\left(\frac{3}{2}-H\right)}{\Gamma(\frac{3}{2}-H)}-\frac{\Gamma(2-2H)\psi(2-2H)}{\Gamma(2-2H)},$
$\displaystyle=\psi\left(\frac{3}{2}-H\right)-\psi(2-2H),$

where $\psi(x)$ is the digamma function.

Then we set $\frac{dh(\raisebox{0.0pt}{$\chi$})}{dH}=0$, and solve for $H$.
Since $\psi(x)$ is a monotonically increasing function on $\mathbb{R^{+}}$,
this implies that $\frac{dh(\raisebox{0.0pt}{$\chi$})}{dH}$ has one solution.
Since $\frac{3}{2}-H=2-2H$ only when $H=\nicefrac{{1}}{{2}}$, this implies
that $h(\raisebox{0.0pt}{$\chi$})$ achieves a unique maximum at this point. ∎

This aligns with our intuition, that the highest uncertainty occurs for this
model when it is uncorrelated and equal to white Gaussian noise, as it
simplifies to $X_{n}=\epsilon_{n}$, identical to FGN processes. This explains
why the maxima coincide for the two processes, given the same process
variance, although ARFIMA(0,d,0) appears to higher differential entropy across
the entire parameter range, when not at $H=0.5$. This provides support that
the maximum entropy process for LRD covariance constraints is the ARFIMA
class, which echos previous results in this area such as Burg’s Theorem (Choi
and Cover (1984)), that the AR and ARMA class of processes are the maximum
entropy models given appropriate constraints on the covariances and impulse
responses (see Ihara (1984); Franke (1985)).

We have shown in this section that the behaviour for the ARFIMA(0,d,0) model
differs from that of FGN in the behaviour of their CSRD processes. This is a
surprising discovery and warrants further investigation. Both models, however,
have much less uncertainty as the strength of the positive correlations
increases, as well as a maximum uncertainty occurring for uncorrelated
processes. Hence, we may be able to characterise the behaviour of LRD
processes on the entropy rate as tending to $-\infty$ as the strength of
correlations increases.

In remainder of the paper we look at other information theoretic measures as
way to characterise the behaviour of SRD and LRD processes.

##  5 Mutual Information and Excess Entropy for Long Range Dependent Processes

In this section we continue analysing of the differential entropy rate for
stochastic processes with power-law decaying covariance function. We
investigate the links between the amount of entropy that is accumulated during
the convergence of the conditional entropy to the entropy rate and the amount
of information that is shared between the past and future of a stochastic
process.

We extend the standard notion of mutual information to the special case of
mutual information between past and future, $I_{\operatorname{p-f}}$, which
will measure the amount of information about the infinite future, given
knowledge of the infinite past of stochastic processes.

######  Definition 5.1.

The mutual information for continuous random variables is defined as

$\displaystyle
I(X;Y)=\int_{\mathcal{Y}}\int_{\mathcal{X}}f(x,y)\log\left(\frac{f(x,y)}{f(x)f(y)}\right)dxdy,$

and in particular the mutual information between past and future with $n$
lags, $I^{(n)}$, for a stochastic process $\\{X_{i}\\}_{i\in\mathbb{Z}}$ is
defined as $I(\\{X_{s},s<0\\},\\{X_{s},s\geq n\\})$. The case with $n=0$ is
called the mutual information between past and future,
$I_{\operatorname{p-f}}$, and is of special interest.

We present a theorem from Li (2006), that links the value of
$I_{\operatorname{p-f}}$, and autocovariance function and the Fourier
coefficients of the logarithm of the spectral density function.

######  Theorem 5.1 (From Li (2006)).

Let $\\{X_{i}\\}_{i\in\mathbb{Z}}$ be a stationary Gaussian stochastic
process:

  * •

if the spectral density $f(\lambda)$ is continuous and $f(\lambda)>0$, then
$I_{\operatorname{p-f}}$ is finite if and only if the autocovariance function
satisfies the condition $\sum_{k=-\infty}^{\infty}k\gamma(k)^{2}<\infty$.

  * •

$I_{\operatorname{p-f}}$ is finite if and only if the cepstrum coefficients,
$b_{k}=\frac{1}{2\pi}\int_{-\pi}^{\pi}\log f(\lambda)e^{-ik\lambda}d\lambda$,
satisfy the condition $\sum_{k=1}^{\infty}kb_{k}^{2}<\infty$. In this case,
$I_{p-f}=\frac{1}{2}\sum_{k=1}^{\infty}kb_{k}^{2}$.

######  Remark.

The convergence of the sum $\sum_{k=-\infty}^{\infty}kb_{k}^{2}$ requires that
$\sum_{k=1}^{\infty}kb_{k}^{2}<\infty$ and
$\sum_{k=1}^{\infty}-kb_{-k}^{2}<\infty$ separately.

This theorem gives us a way to classify whether processes have infinite mutual
information between past and future, in this paper we will use this quantity
to analyse convergence towards the entropic rate, in particular for LRD
stochastic processes.

In the next result we make an explicit link between LRD processes and the
finiteness of the mutual information between past and future. This perspective
provides us with a characterisation of LRD processes, these are processes that
“share infinite information from the past to the future”.

######  Theorem 5.2.

The Mutual Information between past and future, $I_{\operatorname{p-f}}$, for
stationary LRD Gaussian processes is infinite.

###### Proof.

To analyse LRD processes we use the spectral density asymptotic representation
around the origin, $f(\lambda)\sim c_{f}|\lambda|^{1-2H}$, as we can see the
divergence by considering the asymptotic behaviour around the singularity at
the origin. Hence, as $b_{k}=\frac{1}{2\pi}\int_{-\pi}^{\pi}\log
f(\lambda)e^{-ik\lambda}d\lambda$, then

$\displaystyle b_{k}$
$\displaystyle\sim\frac{1}{2\pi}\int_{-\pi}^{\pi}\log\left(c_{f}|\lambda|^{1-2H}\right)e^{-ik\lambda}d\lambda,$
$\displaystyle=\frac{\log
c_{f}}{2\pi}\int_{-\pi}^{\pi}e^{-ik\lambda}d\lambda+\frac{1-2H}{2\pi}\int_{-\pi}^{\pi}\log|\lambda|e^{-ik\lambda}d\lambda.$

We split the integral into positive and negative components,

$\displaystyle b_{k}$ $\displaystyle=\frac{2\log c_{f}\sin\left(\pi
k\right)}{2\pi
k}+\frac{1-2H}{2\pi}\int_{0}^{\pi}\log\lambda\left(e^{-ik\lambda}+e^{ik\lambda}\right)d\lambda.$

Since $\sin(\pi k)=0,\forall k\in\mathbb{Z}$, the first term vanishes and we
only need to consider the integral. This can be decomposed into a
trigonometric representation since
$e^{-ik\lambda}+e^{ik\lambda}=2\cos\left(k\lambda\right)$, so

$\displaystyle b_{k}$
$\displaystyle\sim\frac{1-2H}{2\pi}\int_{0}^{\pi}2\log\lambda\cos\left(k\lambda\right)d\lambda.$

Integrating this expression by parts, we get

$\displaystyle b_{k}$
$\displaystyle\sim\frac{1-2H}{2\pi}\left[\frac{\log\lambda\sin\left(k\lambda\right)}{k}\right]_{0}^{\pi}-\int_{0}^{\pi}\frac{\sin\left(k\lambda\right)}{k\lambda}d\lambda.$

To analyse the integral in the second term we use the substitution,
$u=k\lambda$, and therefore

$\displaystyle\int_{0}^{\pi}\frac{\sin\left(k\lambda\right)}{k\lambda}d\lambda$
$\displaystyle=\int_{0}^{k\pi}\frac{\sin\left(u\right)}{u}du.$

Then undoing the u-substitution and noting that Si$(x)=\int_{0}^{x}\frac{\sin
t}{t}dt$, is the Sine integral,

$\displaystyle b_{k}$
$\displaystyle\sim\frac{1-2H}{\pi}\left[\frac{\log\lambda\sin\left(k\lambda\right)-\text{Si}(\lambda)}{k}\right]_{0}^{\pi},$
$\displaystyle=\frac{\left(1-2H\right)\left(-\text{Si}\left(\pi\right)\right)}{\pi
k}.$

The partial sum has the asymptotic form,

$\displaystyle\sum_{k=1}^{m}kb_{k}^{2}$
$\displaystyle\sim\sum_{k=1}^{m}\frac{\left(1-2H\right)^{2}\
Si\left(\pi\right)^{2}}{\pi^{2}k},$
$\displaystyle=\frac{\left(1-2H\right)^{2}\
Si\left(\pi\right)^{2}}{\pi^{2}}\sum_{k=1}^{m}\frac{1}{k},$
$\displaystyle\sim\frac{\left(1-2H\right)^{2}\
Si\left(\pi\right)^{2}}{\pi^{2}}\log m,$

because the harmonic series, $\sum_{k=1}^{m}\frac{1}{k}\sim\log(m)$ as
$m\rightarrow\infty$. Hence, as $m\rightarrow\infty$, the sum,
$\sum_{k=1}^{m}kb_{k}^{2}$, diverges. This implies that the sum,
$\sum_{k=-\infty}^{\infty}kb_{k}^{2}$, diverges and therefore by Theorem 5.1,
$I_{\operatorname{p-f}}$ is infinite. ∎

######  Remark.

We can quite easily show this result with the additional assumptions that the
spectral density, $f(\lambda)$ is positive and continuous. The asymptotic
expression of the autocovariance function,
$\gamma(k)\sim\sigma^{2}c_{\rho}|k|^{-\alpha}$. Hence, considering the sum,
$\sum_{k=1}^{\infty}k\gamma(k)^{2}$, from the first part of Theorem 5.1,

$\displaystyle\sum_{k=1}^{\infty}k\gamma(k)^{2}$
$\displaystyle\sim\sum_{k=1}^{\infty}k(\sigma^{2}c_{\rho}|k|^{-\alpha})^{2}$
$\displaystyle=\sigma^{4}c_{\rho}^{2}\sum_{k=1}^{\infty}k^{1-2\alpha}.$

This sum diverges in the parameter range, $\alpha\in(0,1)$, and hence by
Theorem 5.1, $I_{\operatorname{p-f}}$ is infinite for LRD processes.

There exist many processes that have infinite excess entropy but are not long
range dependent. Some examples are given, including deterministic processes,
in Crutchfield and Feldman (2003).

Crutchfield and Feldman (2003) analysed a quantity named excess entropy,
$\sum_{n=1}^{\infty}\left(H(X_{n}|X_{n-1},\ldots,X_{0})-H(\raisebox{0.0pt}{$\chi$})\right)$,
for the Shannon entropy $H$ and corresponding entropy rate
$H(\raisebox{0.0pt}{$\chi$})$, which has been shown to be equivalent to the
mutual information between past and future. This has been named, with implicit
interpretation, as stored information (e.g., Shaw (1984)), effective measure
complexity (e.g., Grassberger (1986); Lindgren and Nordahl (1988)), predictive
information (e.g., Nemenman (2000)). Importantly, it has been used to measure
the convergence rate of the conditional entropy, based on past observations,
to the entropy rate. We aim to extend this result to differential entropy, and
then the question of classification of LRD processes via the amount of shared
information can be made by the convergence rate to the entropy rate. We extend
the definition of the excess entropy to the case of differential entropy.

######  Definition 5.2.

The differential _excess_ entropy, $E$, of a stochastic process,
$\\{X_{i}\\}_{i\in\mathbb{N}}$, is defined as,

$\displaystyle E$
$\displaystyle=\sum_{n=1}^{\infty}\left(h_{e}(n)-h(\raisebox{0.0pt}{$\chi$})\right),$
$\displaystyle=\lim\limits_{n\rightarrow\infty}\left[h(X_{n},\ldots,X_{0})-nh(\raisebox{0.0pt}{$\chi$})\right].$

where

$\displaystyle h_{e}(n)$
$\displaystyle=h(X_{1},..,X_{n})-h(X_{1},...,X_{n-1}),$
$\displaystyle=h(X_{n}|X_{n-1},...,X_{1}).$

We have the tools available to make an explicit link between the mutual
information between past and future and the excess entropy of a continuous-
valued, discrete-time stochastic process. This is an exact analogue of
Proposition 8 from Crutchfield and Feldman (2003) and has been stated
utilising a different approach by Ding and Xiang (2016).

######  Theorem 5.3.

For a stationary, continuous-valued stochastic process, the mutual information
between past and future, $I_{\operatorname{p-f}}$, is equal to the
differential excess entropy, $E$.

###### Proof.

The mutual information for a process $X$, with a past and future of $n$
observations,

$\displaystyle I[\\{X_{s},-n\leq s<0\\};\\{X_{s},0\leq s\leq n\\}]$
$\displaystyle=h(X_{0},...,X_{n-1})-h(X_{0},...,X_{n-1}|X_{-n},...,X_{-1}),$
$\displaystyle=\sum_{i=0}^{n-1}h(X_{i}|X_{i-1},...,X_{0})-\sum_{i=0}^{n-1}h(X_{i}|X_{i-1},...,X_{-n}),$
$\displaystyle=\sum_{i=0}^{n-1}\bigg{(}h(X_{i}|X_{i-1},...,X_{0})-h(X_{i}|X_{i-1},...,X_{-n})\bigg{)},$

by the chain rule of differential entropy (Cover and Thomas, 2006, pg. 253).
Then we consider the mutual information between past and future, by taking the
limit of the above expression as $n\rightarrow\infty$, which leads to

$\displaystyle I_{\operatorname{p-f}}$
$\displaystyle=\lim\limits_{n\rightarrow\infty}\left[\sum_{i=0}^{n-1}\bigg{(}h(X_{i}|X_{i-1},...,X_{0})-h(X_{i}|X_{i-1},...,X_{-n})\bigg{)}\right],$
$\displaystyle=\lim\limits_{n\rightarrow\infty}\left[\sum_{i=0}^{\infty}\bigg{(}h(X_{i}|X_{i-1},...,X_{0})-h(X_{i}|X_{i-1},...,X_{-n})\bigg{)}\mathbbm{1}_{\\{i\leq
n\\}}\right].$

We define the sequence of measurable functions, $f_{n}(i)$ as

$\displaystyle f_{n}(i)$
$\displaystyle=\bigg{(}h(X_{i}|X_{i-1},...,X_{0})-h(X_{i}|X_{i-1},...,X_{-n})\bigg{)}\mathbbm{1}_{\\{i\leq
n\\}},$

and we define the function, $g(i)$ as

$\displaystyle g(i)$
$\displaystyle=h(X_{i}|X_{i-1},...,X_{0})-h(\raisebox{0.0pt}{$\chi$}).$

We want to show that $|f_{n}(i)|\leq g(i)$ for all $n$ and for all
$i\in\mathbb{N}$. In this case it is equivalent to showing that $f_{n}(i)\leq
g(i)$, since $f_{n}(i)\geq 0$ for all $n,i\in\mathbb{N}$, as the second term
of $f_{n}(i)$ conditions on more random variables, and since conditioning
cannot increase entropy this implies that $h(X_{i}|X_{i-1},...,X_{0})\geq
h(X_{i}|X_{i-1},...,X_{-n})$. We consider two cases, $i\leq n$ and $i>n$,
separately. In the case, $i>n$, we have that $f_{n}(i)=0$, and since $g(i)\geq
0$ for all $i$, this implies that $f_{n}(i)\leq g(i)$. Considering the second
case, $i\leq n$, we have that

$\displaystyle f_{n}(i)$
$\displaystyle=\bigg{(}h(X_{i}|X_{i-1},...,X_{0})-h(X_{i}|X_{i-1},...,X_{-n})\bigg{)},$

and therefore,

$\displaystyle g(i)-f_{n}(i)$
$\displaystyle=h(X_{i}|X_{i-1},...,X_{-n})-h(\raisebox{0.0pt}{$\chi$}).$

Again, since conditioning does not increase entropy and the characterisation
of entropy rate for stationary processes from Theorem 2.1 this implies that
$g(i)-f_{n}(i)\geq 0$ and therefore $g(i)\geq f_{n}(i)$ for all $n,i$ such
that $i\leq n$. Then we can apply the dominated convergence theorem (Durrett,
2010, pg. 26), since
$f_{n}(i)\rightarrow(X_{i}|X_{i-1},...,X_{0})-h(\raisebox{0.0pt}{$\chi$})$
pointwise, this implies that

$\displaystyle I_{\operatorname{p-f}}$
$\displaystyle=\lim\limits_{n\rightarrow\infty}\sum_{i=0}^{\infty}\bigg{(}h(X_{i}|X_{i-1},...,X_{0})-h(X_{i}|X_{i-1},...,X_{-n})\bigg{)}\mathbbm{1}_{\\{i\leq
n\\}},$
$\displaystyle=\sum_{i=0}^{\infty}\lim\limits_{n\rightarrow\infty}\bigg{(}h(X_{i}|X_{i-1},...,X_{0})-h(X_{i}|X_{i-1},...,X_{-n})\bigg{)}\mathbbm{1}_{\\{i\leq
n\\}},$
$\displaystyle=\sum_{i=0}^{\infty}\bigg{(}h(X_{i}|X_{i-1},...,X_{0})-h(\raisebox{0.0pt}{$\chi$})\bigg{)}.$

∎

######  Remark.

This proof is similar to that of Proposition 8 from Crutchfield and Feldman
(2003). However, it is more rigorous since the limit is kept out the front of
the sum while simultaneously applied to the second term in the sum. This
approach using dominated convergence can resolve the issue in their proof.

In Crutchfield and Feldman (2003), they analyse the excess entropy of discrete
random variables to understand the convergence rate of the conditional entropy
to the entropy rate. From this link we can utilise our knowledge of the mutual
information between the past and future to classify the rate of convergence of
the conditional entropy to the entropy rate.

This result and Theorem 5.2, lead to the following corollary which gives us an
approach to understand the entropy rate convergence by conditional entropy, of
Gaussian LRD processes and is a generalisation of a theorem stated for LRD FGN
processes (Ding and Xiang (2016)).

######  Corollary 5.3.1.

The excess entropy of an LRD Gaussian process is infinite.

###### Proof.

This is shown by combining Theorem 5.2 and Theorem 5.3. ∎

We will use this idea in the subsequent sections to analyse the excess
entropy, given its relationship to convergence to entropy rate which is noted
for Shannon entropy rate by Crutchfield and Feldman (2003).

##  6 Excess Entropy for Stationary Gaussian Processes

In this section we investigate the behaviour of the excess entropy, $E$, for
Gaussian processes which have an autocorrelation function which decays as a
power law.

For stationary processes, the terms in the excess entropy will tend to zero as
$n$ increases because

$\displaystyle\lim\limits_{n\rightarrow\infty}h_{e}(n)=\lim\limits_{n\rightarrow\infty}h(X_{n}|X_{n-1},...,X_{1})=h(\raisebox{0.0pt}{$\chi$}).$

However, we will investigate the nature of the convergence using the
conditional entropy to gain some additional insight.

We begin by looking at the behaviour of the individual terms of the excess
entropy series to understand why these terms decay to zero.

Since $h(X_{n},...,X_{1})=\frac{1}{2}\log\left((2\pi e)^{n}|K^{(n)}|\right)$
for a finite collection of random variables of a Gaussian process (Cover and
Thomas, 2006, pg. 416), we have

$\displaystyle h_{e}(n)$
$\displaystyle=h(X_{n},...,X_{1})-h(X_{n-1},...,X_{1}),$
$\displaystyle=\frac{1}{2}\log\left((2\pi
e)^{n}|K^{(n)}|\right)-\frac{1}{2}\log\left((2\pi e)^{n-1}|K^{(n-1)}|\right),$
$\displaystyle=\frac{1}{2}\log\left(2\pi
e\frac{|K^{(n)}|}{|K^{(n-1)}|}\right),$ (5)

where $K^{(n)}$ is the autocovariance matrix of the process $X_{n}$. Note that
$K^{(n)}$ is a Toeplitz matrix of size $n\times n$.

We analyse each element of the infinite series of differential excess entropy
by substituting the characterisation of $h_{e}(n)$ in (5) and utilising the
entropy rate characterisation for Gaussian processes given in (1). This gives

$\displaystyle h_{e}(n)-h(\raisebox{0.0pt}{$\chi$})$
$\displaystyle=\frac{1}{2}\log\left(2\pi
e\frac{|K^{(n)}|}{|K^{(n-1)}|}\right)-\frac{1}{2}\log(2\pi
e)-\frac{1}{4\pi}\int_{-\pi}^{\pi}\log f(\lambda)d\lambda,$
$\displaystyle=\frac{1}{2}\log\frac{|K^{(n)}|}{|K^{(n-1)}|}-\frac{1}{4\pi}\int_{-\pi}^{\pi}\log
f(\lambda)d\lambda.$ (6)

The expression, (6), informs us about the rate of convergence of the
conditional entropy to the entropy rate, and about the general convergence of
the excess entropy. If $\log f(\lambda)$ is integrable, there is a well known
limit theorem by Szegö (see Szegö (1915); Bingham (2012)), given below and
used to evaluate the limit of $\frac{|K^{(n)}|}{|K^{(n-1)}|}$ as
$n\rightarrow\infty$.

First we show that $\log f(\lambda)\in L_{1}[-\pi,\pi]$, for LRD processes and
hence that Szegö’s limit theorem can be applied. We use the asymptotic form to
analyse this case, as the issue with integrability is the singularity that
exists at the origin of the spectral density functions. Therefore, the
integrability of the asymptotic form implies the integrability for LRD
processes.

######  Theorem 6.1.

For a spectral density, $f(\lambda)$ of a LRD stochastic process, $\log
f(\lambda)\in L_{1}[-\pi,\pi]$.

###### Proof.

Using the asymptotic expression for spectral density for LRD processes,
$f(\lambda)\sim c_{f}|\lambda|^{-\beta},\text{ }\beta\in(0,1),\text{
}0<c_{f}<\infty$.
Hence,

$\displaystyle\int_{-\pi}^{\pi}\log(c_{f}|\lambda|^{-\beta})d\lambda$
$\displaystyle=\int_{-\pi}^{\pi}\log
c_{f}d\lambda-\beta\int_{-\pi}^{\pi}\log|\lambda|d\lambda,$
$\displaystyle=2\pi\log c_{f}-2\beta\int_{0}^{\pi}\log\lambda\ d\lambda,$
$\displaystyle=2\pi\log
c_{f}-2\beta\left[\lambda\log\lambda-\lambda\right]_{0}^{\pi},$
$\displaystyle=2\pi\log c_{f}-2\beta\left[\pi\log\pi-\pi\right],$

Since we have that $0\log 0=0$. By the finiteness of all of the terms, this
implies that,

$\displaystyle\int_{-\pi}^{\pi}\log f(\lambda)d\lambda<\infty.$

∎

The following theorem was originally formulated by Szegö (1915), and then
extended to include another term in Szegö (1952). This theorem has a history
of having the conditions generalised, and applications being found in many
areas in functional analysis, statistics and probability (Grenander and Szegö,
1958, pg. 145-228), to calculate functions of eigenvalues of Toeplitz
matrices. The statement of the theorem that we will be using is by Bingham
(2012), which is from the probabilistic perspective and includes the most
recent generalisations in the conditions.

######  Theorem 6.2 (Szegö’s Theorem (Bingham (2012))).

For a sequence of Toeplitz matrices, $\Gamma^{(n)}=[\gamma(|i-j|)]_{i,j}$, of
increasing size $n$, where
$\gamma(n)=\int_{-\pi}^{\pi}f(\lambda)e^{in\lambda}d\lambda$, and $f(\lambda)$
is a spectral density such that $\log f(\lambda)\in L_{1}[-\pi,\pi]$, then
there exists a limit

$\displaystyle\lim\limits_{n\to\infty}|\Gamma^{(n)}|^{\frac{1}{n}}=\exp\left\\{\frac{1}{2\pi}\int_{-\pi}^{\pi}\log
f(\lambda)d\lambda\right\\}.$

We apply this theorem to the ratios of the autocovariance function, $K^{(n)}$,
of the process, $\\{X_{n}\\}_{n\in\mathbb{N}}$. Due to the finiteness of this
limit we can take the $n$th power, for a covariance matrix $K^{(n)}$. Which
has the following form,

$\displaystyle\lim\limits_{n\to\infty}|K^{(n)}|$
$\displaystyle=\exp\left\\{\frac{1}{2\pi}\int_{-\pi}^{\pi}\log
f(\lambda)d\lambda\right\\}^{n},$

taking the $n$th power. Applying this result to the limit of the $n$th term of
the differential excess entropy (6), gives

$\displaystyle\lim\limits_{n\to\infty}h_{e}(n)-h(\raisebox{0.0pt}{$\chi$})$
$\displaystyle=\frac{1}{2}\log\left(\exp\left\\{\frac{1}{2\pi}\int_{-\pi}^{\pi}\log
f(\lambda)d\lambda\right\\}\right)-\frac{1}{4\pi}\int_{-\pi}^{\pi}\log
f(\lambda)d\lambda,$ $\displaystyle=0.$

As $n\to\infty,h_{e}(n)-h(\raisebox{0.0pt}{$\chi$})\to 0$, which implies the
convergence of the conditional entropy, conditioned on the infinite past, to
the entropy rate, equivalent to the result of Theorem 2.1. We have gained some
additional insight that the summands converge to zero in the limit as
$n\rightarrow\infty$ due to the convegence of the ratio of subsequent
covariance matrix determinants. However, this does not tell us anything about
the rate of convergence of the terms or whether $E$ converges at all, but this
approach can be extended to garner such information.

In following section we will use a stronger version of the Szegö theorem, with
an additional term in the limit. This gives an approach to analyse the
convergence properties to the entropy rate that arise from observing the
conditional entropy, conditioned on the past observations.

##  7 Entropy rate convergence for LRD processes

From the previous sections we have gained an understanding of some of the
properties of the entropy rate function for common LRD models. In this
section, we classify the convergence rate of the conditional entropy to the
entropy rate for SRD and LRD processes.

In Section 6 we used Szegö’s theorem to show,

$\displaystyle\lim\limits_{n\to\infty}\frac{|K^{(n)}|}{|K^{(n-1)}|}=\exp\left(\frac{1}{2\pi}\int_{-\pi}^{\pi}\log
f(\lambda)d\lambda\right),$

if $\log f(\lambda)\in L_{1}[-\pi,\pi]$. This provides another perspective to
explain why the conditional entropy converges to the entropy rate. The Strong
Szegö Theorem, provides an additional term to this limit with the same
regularity conditions. The Szegö theorem showed why the convergence occurred
for the conditional entropy, and the strong Szegö theorem we explain the
convergence rate. We will state the version as given in Bingham (2012), which
is more suited to probabilistic analysis.

For ease of notation we define the limit of the Szegö theorem,

$\displaystyle G(\mu):=\exp\left(\frac{1}{2}\int_{-\pi}^{\pi}\log
f(\lambda)d\lambda\right),$

and the partial autocorrelation coefficients, $\alpha_{n}$, as,

$\displaystyle\alpha_{n}=corr(X_{n}-P_{[1,n-1]}X_{n},X_{0}-P_{[1,n-1]}X_{0}),$

where $P_{[1,n-1]}$ is the projection onto the linear space spanned by
$\\{X_{-n},...,X_{-1}\\}$ and the correlation function is defined as,

$\displaystyle
corr(X,Y)=\frac{\mathds{E}[X\bar{Y}]}{\sqrt{\mathds{E}[|X|^{2}]\mathds{E}[|Y|^{2}]}},$

for $X$ and $Y$ zero mean random variables. Note that $P_{[1,n-1]}X_{0}$ is
the best linear predictor of $X_{0}$ given the finite past of length $n$.

The partial autocorrelation function is related to the autocorrelation
function by the removal of the linear dependence on the variables within n
lags. For example, in the case of finite lag processes, such as AR(p), the
partial correlation function is 0 for a lag greater than p. In the case of the
ARFIMA(p,d,q) process, the decay is slower for LRD parametrisations (Inoue
(2002)).

We define the Hardy space $\mathcal{H}^{\frac{1}{2}}$, which is the subspace
of $\ell_{2}$ of sequences $a=(a_{n})$ such that
$||a||^{2}:=\sum_{n}(1+|n|)|a_{n}|^{2}<\infty$, which is a well defined norm
on $\ell_{2}$. We use this to describe the conditions on the strong Szegö
theorem. Also, the cepstral coefficients that were defined in Theorem 5.1 will
be used in the definition of the additional term of the Szegö limit theorem,
reinforcing their connection with the information theoretic perspective of
stochastic processes.

######  Theorem 7.1 (Strong Szegö Theorem (Bingham (2012))).

For a sequence of Toeplitz matrices, $\Gamma^{(n)}=[\gamma(|i-j|)]_{i,j}$, of
increasing size $n$, for covariance function $\gamma(n)$, with associated
spectral density $f(\lambda)$, such that $\log f(\lambda)\in L_{1}[-\pi,\pi]$,
then

$\displaystyle\lim\limits_{n\to\infty}\frac{|\Gamma^{(n)}|}{G(\mu)^{n}}\to
E(\mu),$

where,

$\displaystyle
E(\mu):=\prod_{j=1}^{\infty}(1-|\alpha_{j}|^{2})^{-j}=\exp\left(\sum_{k=1}^{\infty}kb_{k}^{2}\right),$

noting that these expressions may be infinite. The infinite product converges
if and only if

  * •

The strong Szegö condition, $\alpha\in\mathcal{H}^{\frac{1}{2}}$ holds; or,

  * •

The sum of cepstral coefficients
$\sum_{k=1}^{\infty}kb_{k}^{2}\in\mathcal{H}^{\frac{1}{2}}$.

Using this theorem we will be able to characterise the convergence rate of the
short range dependent processes that we have been considering in this paper.

The SRD and CSRD versions of ARFIMA and Fractional Gaussian Noise meet the
conditions of the convergence of the infinite product and sum of Strong Szegö
Theorem. This is shown by Theorem 5.1. For a positive, continuous spectral
density $\sum_{k=1}^{\infty}k\gamma(k)^{2}<\infty$ if and only if
$I_{p-f}<\infty$, which in turn holds if and only if
$\sum_{k=-\infty}^{\infty}|k||b_{k}|^{2}<\infty$. In the proof of Theorem 5.2,
we showed that the boundary of the finiteness and infiniteness of
$\sum_{k=1}^{\infty}kb_{k}^{2}$, coincided with the boundary between SRD and
LRD processes. For example, for ARFIMA and Fractional Gaussian Noise,
$\sum_{k=1}^{\infty}k\gamma(k)^{2}<\infty$ when $H\leq\frac{1}{2}$ and
therefore by Theorem 5.2, $\sum_{k=1}^{\infty}|k||b_{k}|^{2}<\infty$.

One additional note about the conditions of the infinite product and sum in
this theorem, the conditions for $\ell_{1}$ and $H<\frac{1}{2}$, coincide for
sequences that decay as power laws, i.e., $n^{\alpha}$, which is common when
considering the convergence or divergence of sequences of LRD processes.
However, it may be the case that a process may be in $\ell_{1}$ and not in
$\mathcal{H}^{\frac{1}{2}}$, _e.g._ , $a_{n}=\frac{1}{n\log n}$ (Bingham
(2012)).

Before we continue, we require a lemma to prove a theorem about the
convergence rate of the differential entropy rate of SRD and LRD processes,
which gives two different determinant limits for the results of the Szegö
limit theorems.

######  Lemma 7.2.

For all discrete-time stationary Gaussian processes,

$\displaystyle\lim\limits_{n\to\infty}\left(|K^{(n)}|\right)^{\frac{1}{n}}$
$\displaystyle=\lim\limits_{n\to\infty}\frac{|K^{(n)}|}{|K^{(n-1)}|},$

where $K^{(n)}$ is the $n\times n$ autocovariance matrix of the process.

###### Proof.

For discrete-time stationary Gaussian processes,

$\displaystyle\lim\limits_{n\to\infty}\frac{1}{n}h(X_{1},..,X_{n})$
$\displaystyle=\lim\limits_{n\to\infty}h(X_{n}|X_{n-1},...,X_{1}),$

by Theorem 2.1 and the fact that both of the limits exist for stationary
processes. The form of the joint entropy of a multivariate Gaussian random
vector of length $n$ (Cover and Thomas, 2006, pg. 249) is

$\displaystyle h(X_{1},..,X_{n})=\frac{1}{2}\log\left(2\pi e|K^{(n)}|\right).$

Therefore as,

$\displaystyle\lim\limits_{n\to\infty}\frac{1}{n}h(X_{1},..,X_{n})$
$\displaystyle=\lim\limits_{n\to\infty}\frac{1}{n}\left(\frac{1}{2}\log\left((2\pi
e)^{n}|K^{(n)}|\right)\right),$
$\displaystyle=\lim\limits_{n\to\infty}\frac{1}{2}\log\left(2\pi
e|K^{(n)}|^{\frac{1}{n}}\right).$

Equating this for the entropy rate characterised using conditional entropy we
get

$\displaystyle\lim\limits_{n\to\infty}\frac{1}{2}\log\left(2\pi
e|K^{(n)}|^{\frac{1}{n}}\right)$
$\displaystyle=\lim\limits_{n\to\infty}h(X_{n},...,X_{1})-h(X_{n-1},...,X_{1}),$
$\displaystyle=\lim\limits_{n\to\infty}\frac{1}{2}\log\left((2\pi
e)^{n}|K^{(n)}|\right)-\frac{1}{2}\log\left((2\pi e)^{n-1}|K^{(n-1)}|\right),$
$\displaystyle=\lim\limits_{n\to\infty}\frac{1}{2}\log\left((2\pi
e)\frac{|K^{(n)}|}{|K^{(n-1)}|}\right),$

Hence, we conclude that,

$\displaystyle\lim\limits_{n\to\infty}(|K^{(n)}|)^{\frac{1}{n}}$
$\displaystyle=\lim\limits_{n\to\infty}\frac{|K^{(n)}|}{|K^{(n-1)}|}.$

∎

Now we can characterise the convergence rate of the conditional entropy of SRD
ARFIMA and Fractional Gaussian Noise processes to the differential entropy
rate. Then we will show that a slower decay exists in the case of long range
dependence.

######  Theorem 7.3.

For all discrete-time Gaussian SRD processes, such that the autocovariance
function $\gamma\in\ell_{1}$, the convergence rate of the conditional entropy,
$h(X_{n}|X_{n-1},...,X_{1})$, to the differential entropy rate of the process
is $O(n^{-1})$.

###### Proof.

We rearrange the asymptotic expression of Szegö’s strong theorem to get

$\displaystyle\frac{|K^{(n)}|}{G(\mu)^{n}}$ $\displaystyle\sim E(\mu),$
$\displaystyle\implies|K^{(n)}|^{\frac{1}{n}}$ $\displaystyle\sim
G(\mu)E(\mu)^{\frac{1}{n}}.$

Then we use the asymptotic form of the determinant limit from Lemma 7.2,

$\displaystyle\frac{|K^{(n)}|}{|K^{(n-1)}|}$ $\displaystyle\sim
G(\mu)\exp\left(\sum_{k=1}^{\infty}kb_{k}^{2}\right)^{\frac{1}{n}},$
$\displaystyle\implies\frac{|K^{(n)}|}{|K^{(n-1)}|}$ $\displaystyle\sim
G(\mu)\exp\left(\frac{1}{n}\sum_{k=1}^{\infty}kb_{k}^{2}\right).$

From the asymptotic form of the Szegö theorem, Theorem 6.2, the LHS will
converge to $G(\mu)$ as $n$ increases. This implies that the convergence rate
of the conditional entropy is controlled by the term,
$\exp(\frac{1}{n}\sum_{k=1}^{\infty}kb_{k}^{2})$, and its convergence rate to
1. That is, $n\to\infty$

$\displaystyle\left|G(\mu)\exp\left(\frac{1}{n}\sum_{k=1}^{\infty}kb_{k}^{2}\right)-G(\mu)\right|$
$\displaystyle=G(\mu)\left|\exp\left(\frac{1}{n}\sum_{k=1}^{\infty}kb_{k}^{2}\right)-\exp(0)\right|,$
$\displaystyle=G(\mu).$

Since $\frac{1}{n}\sum_{k=1}^{\infty}kb_{k}^{2}\to 0$, as
$n\rightarrow\infty$.

Since $\sum_{k=1}^{\infty}kb_{k}^{2}$ is finite for $H\leq\frac{1}{2}$, as
$\gamma\in\ell_{1}$, this implies that the convergence is at the rate of
$O(n^{-1})$ as

$\displaystyle\left|\exp\left(\frac{1}{n}\sum_{k=1}^{\infty}kb_{k}^{2}\right)-\exp(0)\right|$
$\displaystyle\to 0,$
$\displaystyle\implies\left|\frac{1}{n}\sum_{k=1}^{\infty}kb_{k}^{2}-0\right|$
$\displaystyle\to 0,$ $\displaystyle\text{and,
}\left|\frac{1}{n}\sum_{k=1}^{\infty}kb_{k}^{2}\right|$
$\displaystyle\leq\frac{C}{n},C\in\mathbb{R^{+}},\forall n\in\mathbb{Z^{+}}.$

For $C<\infty$, such that $C\geq\sum_{k=1}^{\infty}kb_{k}^{2}$. ∎

The key part of the proof that allows us to characterise the convergence rate
of the conditional entropy is that the series,
$\sum_{k=1}^{\infty}kb_{k}^{2}$, is finite. By Theorem 5.1, we know that this
is the boundary between LRD and SRD processes. This behaviour intuitively
indicates that a limit would take a longer time to converge in the LRD case,
similar to many estimators for LRD processes. Hence, we expect that the
convergence to the entropy rate from the conditional entropy rate will be
slower for LRD processes. This leads to the following theorem and
characterisation of LRD processes through convergence of conditional entropy.

######  Theorem 7.4.

For all Gaussian LRD processes the convergence rate of the conditional
entropy, $h(X_{n}|X_{n-1},...,X_{1})$, to the differential entropy rate of the
process is

$\displaystyle O(\frac{\log(n^{(1-2H)^{2}})}{n}).$

###### Proof.

Similar to the theorem above, we consider the convergence of the term,
$\exp(\frac{1}{n}\sum_{k=1}^{\infty}kb_{k}^{2})$, to 0, and use the following
expansion of the term,

$\displaystyle\lim\limits_{n\rightarrow\infty}\exp\left(\frac{1}{n}\sum_{k=1}^{\infty}kb_{k}^{2}\right)$
$\displaystyle=\lim\limits_{n\rightarrow\infty}\exp\left(\frac{1}{n}\lim\limits_{m\rightarrow\infty}\sum_{k=1}^{m}kb_{k}^{2}\right),$
$\displaystyle=\lim\limits_{n\rightarrow\infty}\lim\limits_{m\rightarrow\infty}\exp\left(\frac{1}{n}\sum_{k=1}^{m}kb_{k}^{2}\right).$

Where we use the continuity in the exponential function to exchange the limit
in the function. From Theorem 5.2, we showed that the rate of divergence of
the partial sum of $kb_{k}^{2}$ is equal to,

$\displaystyle\sum_{k=1}^{m}kb_{k}^{2}$
$\displaystyle\sim\frac{\left(1-2H\right)^{2}\
Si\left(\pi\right)^{2}}{\pi^{2}}\log m,$
$\displaystyle=\frac{Si\left(\pi\right)^{2}}{\pi^{2}}\log\left(m^{\left(1-2H\right)^{2}}\right).$

Hence we can consider the limits as $n$ and $m$ tend to infinity,

$\displaystyle\lim\limits_{n\rightarrow\infty}\lim\limits_{m\rightarrow\infty}\exp\left(\frac{1}{n}\sum_{k=1}^{m}kb_{k}^{2}\right)$
$\displaystyle\sim\lim\limits_{n\rightarrow\infty}\lim\limits_{m\rightarrow\infty}\exp\left(\frac{1}{n}\frac{Si\left(\pi\right)^{2}}{\pi^{2}}\log\left(m^{\left(1-2H\right)^{2}}\right)\right).$

If we take the $n\rightarrow\infty$ and $m\rightarrow\infty$ such that $m=n$,
then we have

$\displaystyle\lim\limits_{n\rightarrow\infty}\exp\left(\frac{1}{n}\frac{Si\left(\pi\right)^{2}}{\pi^{2}}\log\left(n^{\left(1-2H\right)^{2}}\right)\right)\rightarrow
1.$

Hence,

$\displaystyle\lim\limits_{n\rightarrow\infty}\frac{\log\left(n^{\left(1-2H\right)^{2}}\right)}{n}\rightarrow
0.$

Which gives the rate of convergence of the conditional entropy to entropy
rate. ∎

Therefore, the convergence of the conditional entropy to entropy rate is
slower for LRD processes than SRD processes. This provides an information
theoretic characterisation of LRD by convergence properties, similar to the
covariance function, the sample mean and the parameters of linear predictors
on the infinite past. The rate of convergence to the entropy rate decreases
rapidly as $H\rightarrow 1$, because in $\left(1-2H\right)^{2}$ the influence
of the Hurst parameter is squared. The convergence becomes much quicker as the
Hurst parameter approaches 0.5, and eventually reaches the point where it
converges immediately at $H=0.5$, as
$h(X_{i})=h(\raisebox{0.0pt}{$\chi$}),\forall i\in\mathbb{Z^{+}}$ in the
absence of any correlations.

##  8 Conclusion

In this paper, we are concerned with the behaviour of the differential entropy
rate to understand and characterise the behaviour of LRD and SRD processes.
Analysing two common LRD processes, FGN and ARFIMA(0,d,0), we have shown that
the maximum occurs in the absence of correlations, _i.e._ , $H=0.5$, and the
differential entropy rate tends to the minimum, $-\infty$ as the strength of
positive correlations increase, _i.e._ , as we receive more information from
correlations, the entropy of the process decreases. However, there is very
different behaviour for negatively correlated processes, where ARFIMA(0,d,0)
processes do not tend to $-\infty$ as the strength of the negative
correlations increases. Further research is required to understand this
behaviour for these processes.

In addition, we have made a link, similar to Shannon entropy, between the
mutual information between past and future and excess entropy, meaning that
the amount of shared information between the complete past of future of a
process is the same as the additional information that accrues when converging
to the entropy rate, based on past observations. This leads to a
characterisation of LRD processes, as those having infinite mutual information
between past and future. Using this and Szegö’s limit theorems we then can
classify LRD and SRD processes by their convergence rates, and show that LRD
processes have slower convergence of the conditional entropy, conditioned on
the past, to the entropy rate.

## Acknowledgment

This research was funded by CSIRO’s Data61, the Australian Research Council’s
Centre of Excellence in Mathematical and Statistical Frontiers and Defence
Science and Technology Group. Thanks to Adam Hamilton for his assistance in
providing feedback on this manuscript.

## References

  * Beran (1994) J. Beran.  _Statistics for Long-Memory Processes_.  Chapman & Hall/CRC Monographs on Statistics & Applied Probability. Taylor & Francis, 1994.  ISBN 9780412049019.  URL https://books.google.com.au/books?id=jdzDYWtfPC0C.
  * Bingham (2012) N.H. Bingham.  Szegö’s theorem and its probabilistic descendants.  _Probab. Surveys_ , 9:287–324, 2012.  doi: 10.1214/11-PS178.  URL https://doi.org/10.1214/11-PS178.
  * Box et al. (2015) George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung.  _Time series analysis: forecasting and control_.  John Wiley & Sons, 2015.
  * Brockwell and Davis (1986) Peter J Brockwell and Richard A Davis.  _Time Series: Theory and Methods_.  Springer-Verlag, Berlin, Heidelberg, 1986.  ISBN 0387964061.
  * Chavez (2016) Gordon Chavez.  Conditional and marginal mutual information in gaussian and hyperbolic decay time series.  _Journal of Time Series Analysis_ , 37(6):851–861, 2016.  doi: 10.1111/jtsa.12199.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/jtsa.12199.
  * Choi and Cover (1984) BS Choi and Thomas M Cover.  An information-theoretic proof of Burg’s maximum entropy spectrum.  _Proceedings of the IEEE_ , 72(8):1094–1096, 1984\.
  * Cont (2005) Rama Cont.  _Long range dependence in financial market_ , pages 159–179.  01 2005.  doi: 10.1007/1-84628-048-6˙11.
  * Cover and Thomas (2006) Thomas M. Cover and Joy A. Thomas.  _Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing)_.  Wiley-Interscience, USA, 2006.  ISBN 0471241954.
  * Crutchfield and Feldman (2003) James P. Crutchfield and David P. Feldman.  Regularities unseen, randomness observed: Levels of entropy convergence.  _Chaos: An Interdisciplinary Journal of Nonlinear Science_ , 13(1):25–54, 2003.  doi: 10.1063/1.1530990.  URL https://doi.org/10.1063/1.1530990.
  * Ding and Xiang (2016) Yiming Ding and Xuyan Xiang.  An entropic characterization of long memory stationary process, 2016.  arXiv preprint arXiv:1604.05453.
  * Durrett (2010) Rick Durrett.  _Probability: Theory and Examples_.  Cambridge University Press, New York, NY, USA, 4th edition, 2010.  ISBN 0521765390, 9780521765398.
  * Franke (1985) Jürgen Franke.  Arma processes have maximal entropy among time series with prescribed autocovariances and impulse responses.  _Advances in applied probability_ , pages 810–840, 1985.
  * Gefferth et al. (2003) A Gefferth, D Veitch, I Maricza, S Molnár, and I Ruzsa.  The nature of discrete second-order self-similarity.  _Advances in Applied Probability_ , pages 395–416, 2003.
  * Granger and Joyeux (1980) C. W. J. Granger and Roselyne Joyeux.  An introduction to long-memory time series models and fractional differencing.  _Journal of Time Series Analysis_ , 1(1):15–29, 1980.  doi: https://doi.org/10.1111/j.1467-9892.1980.tb00297.x.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-9892.1980.tb00297.x.
  * Grassberger (1986) Peter Grassberger.  Toward a quantitative theory of self-generated complexity.  _International Journal of Theoretical Physics_ , 25(9):907–938, 1986.
  * Grenander and Szegö (1958) Ulf Grenander and Gabor Szegö.  _Toeplitz forms and their applications_.  Univ of California Press, 1958.
  * Hosking (1981) J. R. M. Hosking.  Fractional differencing.  _Biometrika_ , 68(1):165–176, 04 1981.  ISSN 0006-3444.  doi: 10.1093/biomet/68.1.165.  URL https://doi.org/10.1093/biomet/68.1.165.
  * Hurst (1951) H. E. Hurst.  Long-term storage capacity of reservoirs.  _Trans. Amer. Soc. Civil Eng._ , 116:770–799, 1951.  URL https://ci.nii.ac.jp/naid/10014568424/en/.
  * Ihara (1984) Shunsuke Ihara.  Maximum entropy spectral analysis and arma processes (corresp.).  _IEEE transactions on information theory_ , 30(2):377–380, 1984.
  * Ihara (1993) Shunsuke Ihara.  _Information theory for continuous systems_ , volume 2.  World Scientific, 1993.
  * Inoue (2002) Akihiko Inoue.  Asymptotic behavior for partial autocorrelation functions of fractional arima processes.  _Annals of Applied Probability_ , pages 1471–1491, 2002.
  * Koyama and Kurokawa (2005) Shin-Ya Koyama and Nobushige Kurokawa.  Euler’s integrals and multiple sine functions.  _Proceedings of the American Mathematical Society_ , 133(5):1257–1265, 2005.  ISSN 00029939, 10886826.  URL http://www.jstor.org/stable/4097775.
  * Lawrance and Kottegoda (1977) A. J. Lawrance and N. T. Kottegoda.  Stochastic modelling of riverflow time series.  _Journal of the Royal Statistical Society. Series A (General)_ , 140(1):1–47, 1977.  ISSN 00359238.  URL http://www.jstor.org/stable/2344516.
  * Leland et al. (1993) Will E. Leland, Murad S. Taqqu, Walter Willinger, and Daniel V. Wilson.  On the self-similar nature of ethernet traffic.  In _Conference Proceedings on Communications Architectures, Protocols and Applications_ , SIGCOMM’93, pages 183–193, New York, NY, USA, 1993\. ACM.  ISBN 0-89791-619-0.  doi: 10.1145/166237.166255.  URL http://doi.acm.org/10.1145/166237.166255.
  * Li (2006) Lei M. Li.  Some notes on mutual information between past and future.  _Journal of Time Series Analysis_ , 27(2):309–322, 2006.  doi: 10.1111/j.1467-9892.2005.00469.x.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-9892.2005.00469.x.
  * Lindgren and Nordahl (1988) Kristian Lindgren and Mats G Nordahl.  Complexity measures and cellular automata.  _Complex Systems_ , 2(4):409–440, 1988.
  * Mandelbrot (1965) B. Mandelbrot.  Self-similar error clusters in communication systems and the concept of conditional stationarity.  _IEEE Transactions on Communication Technology_ , 13(1):71–90, 1965.  doi: 10.1109/TCOM.1965.1089090.
  * Nemenman (2000) Ilya Nemenman.  Information theory and learning: A physical approach.  _arXiv preprint physics/0009032_ , 2000.
  * Shaw (1984) Robert Shaw.  _The dripping faucet as a model chaotic system_.  Aerial Pr, 1984.
  * Shiu and Shakarchi (2004) P. Shiu and R. Shakarchi.  Complex analysis.  _The Mathematical Gazette_ , 88(512), 2004.  doi: 10.1017/S0025557200175564.
  * Szegö (1952) Gabor Szegö.  On certain hermitian forms associated with the fourier series of a positive function.  _Comm. Sém. Math. Univ. Lund [Medd. Lunds Univ. Mat. Sem.]_ , (Tome Supplémentaire):228–238, 1952.
  * Szegö (1915) G. Szegö.  Ein grenzwertsatz über die toeplitzschen determinanten einer reellen positiven funktion.  _Mathematische Annalen_ , 76(4):490–503, 1915\.  doi: 10.1007/bf01458220.  URL https://app.dimensions.ai/details/publication/pub.1018555817andhttps://zenodo.org/record/2496405/files/article.pdf.
  * Varotsos and Kirk-Davidoff (2006) Costas Varotsos and Daniel Kirk-Davidoff.  Long-memory processes in ozone and temperature variations at the region 60° S-60° N.  _Atmos. Chem. Phys._ , 6, 09 2006.  doi: 10.5194/acp-6-4093-2006.
  * Veitch et al. (2013) Darryl Veitch, A Gorst-Rasmussen, and A Gefferth.  Why FARIMA models are brittle.  _Fractals_ , 21(02):1350012, 2013.
  * Virtanen et al. (2020) Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ , 17:261–272, 2020.  doi: 10.1038/s41592-019-0686-2.
  * Willinger et al. (1997) W. Willinger, M. S. Taqqu, R. Sherman, and D. V. Wilson.  Self-similarity through high-variability: statistical analysis of ethernet lan traffic at the source level.  _IEEE/ACM Transactions on Networking_ , 5(1):71–86, Feb 1997.  ISSN 1063-6692.  doi: 10.1109/90.554723.
  * Willinger et al. (1995) Walter Willinger, Murad S. Taqqu, Will E. Leland, and Daniel V. Wilson.  Self-similarity in high-speed packet traffic: Analysis and modeling of ethernet traffic measurements.  _Statist. Sci._ , 10(1):67–85, 02 1995.  doi: 10.1214/ss/1177010131.  URL https://doi.org/10.1214/ss/1177010131.
  * Willinger et al. (1999) Walter Willinger, Murad S. Taqqu, and Vadim Teverovsky.  Stock market prices and long-range dependence.  _Finance and Stochastics_ , 3(1):1–13, January 1999.  ISSN 0949-2984, 1432-1122.  doi: 10.1007/s007800050049.  URL http://link.springer.com/10.1007/s007800050049.