Deep Policy Gradient Methods in Commodity Markets

Jonas Rotschi Hanetho

Thesis submitted for the degree of
Master in Informatics: Programming and System Architecture

60 credits

Institute for Informatics

Faculty of mathematics and natural sciences

UNIVERSITY OF OSLO

Autumn 2024

Deep Policy Gradient Methods in Commodity Markets

Jonas Rotschi Hanetho

© 2024 Jonas Rotschi Hanetho
Deep Policy Gradient Methods in Commodity Markets
http://www.duo.uio.no/
Printed: Reprosentralen, University of Oslo

### Acknowledgements

This thesis would not have been possible without my supervisors, Dirk Hesse
and Martin Giese. My sincere thanks are extended to Dirk for his excellent
guidance and mentoring throughout this project and to Martin for his helpful
suggestions and advice. Finally, I would like to thank the Equinor data
science team for insightful discussions and for providing me with the tools
needed to complete this project.

### Abstract

The energy transition has increased the reliance on intermittent energy
sources, destabilizing energy markets and causing unprecedented volatility,
culminating in the global energy crisis of 2021. In addition to harming
producers and consumers, volatile energy markets may jeopardize vital
decarbonization efforts. Traders play an important role in stabilizing markets
by providing liquidity and reducing volatility. Forecasting future returns is
an integral part of any financial trading operation, and several mathematical
and statistical models have been proposed for this purpose. However,
developing such models is non-trivial due to financial markets’ low signal-to-
noise ratios and nonstationary dynamics.

This thesis investigates the effectiveness of deep reinforcement learning
methods in commodities trading. It presents related work and relevant research
in algorithmic trading, deep learning, and reinforcement learning. The thesis
formalizes the commodities trading problem as a continuing discrete-time
stochastic dynamical system. This system employs a novel time-discretization
scheme that is reactive and adaptive to market volatility, providing better
statistical properties for the sub-sampled financial time series. Two policy
gradient algorithms, an actor-based and an actor-critic-based, are proposed
for optimizing a transaction-cost- and risk-sensitive trading agent. The agent
maps historical price observations to market positions through parametric
function approximators utilizing deep neural network architectures,
specifically CNNs and LSTMs.

On average, the deep reinforcement learning models produce an $83$ percent
higher Sharpe ratio than the buy-and-hold baseline when backtested on front-
month natural gas futures from 2017 to 2022. The backtests demonstrate that
the risk tolerance of the deep reinforcement learning agents can be adjusted
using a risk-sensitivity term. The actor-based policy gradient algorithm
performs significantly better than the actor-critic-based algorithm, and the
CNN-based models perform slightly better than those based on the LSTM. The
backtest results indicate the viability of deep reinforcement learning-based
algorithmic trading in volatile commodity markets.

###### List of Figures

  1. 2.1 Time series cross-validation (backtesting) compared to standard cross-validation from [HA18].
  2. 3.1 An illustration of the relationship between the capacity of a function approximator and the generalization error from [GBC16].
  3. 3.2 Feedforward neural network from [FFLb].
  4. 3.3 ReLU activation function from [FFLb].
  5. 3.4 Leaky ReLU activation function from [FFLb].
  6. 3.5 An example of the effect of weight decay with parameter $\lambda$ on a high-dimensional polynomial regression model from [GBC16].
  7. 3.6 3D convolutional layer from [FFLc].
  8. 3.7 Recurrent neural network from [FFLa].
  9. 3.8 LSTM cell from [FFLa].
  10. 4.1 Agent-environment interaction from [SB18].
  11. 7.1 Policy network architecture
  12. 7.2 Q-network architecture
  13. 7.3 Convolutional sequential information layer architecture
  14. 7.4 LSTM sequential information layer architecture
  15. 8.1 The training-validation-test split
  16. 8.2 Cumulative logarithmic trade returns for $\lambda_{\sigma}=0$
  17. 8.3 Boxplot of monthly logarithmic trade returns for $\lambda_{\sigma}=0$
  18. 8.4 Cumulative logarithmic trade returns for $\lambda_{\sigma}=0.01$
  19. 8.5 Boxplot of monthly logarithmic trade returns for $\lambda_{\sigma}=0.01$
  20. 8.6 Cumulative logarithmic trade returns for $\lambda_{\sigma}=0.1$
  21. 8.7 Boxplot of monthly logarithmic trade returns for $\lambda_{\sigma}=0.1$
  22. 8.8 Cumulative logarithmic trade returns for $\lambda_{\sigma}=0.2$
  23. 8.9 Boxplot of monthly logarithmic trade returns for $\lambda_{\sigma}=0.2$

###### List of Tables

  1. 1 Hyperparameters
  2. 2 Backtest results

###### Contents

  1. 1 Introduction
    1. 1.1 Motivation
    2. 1.2 Problem description
    3. 1.3 Thesis Organization
  2. I Background
    1. 2 Algorithmic trading
      1. 2.1 Commodity markets
      2. 2.2 Financial trading
      3. 2.3 Modern portfolio theory
      4. 2.4 Efficient market hypothesis
      5. 2.5 Forecasting
      6. 2.6 Mapping forecasts to market positions
      7. 2.7 Feature engineering
      8. 2.8 Sub-sampling schemes
      9. 2.9 Backtesting
    2. 3 Deep learning
      1. 3.1 Machine learning
        1. 3.1.1 No-free-lunch theorem
        2. 3.1.2 The curse of dimensionality
      2. 3.2 Supervised learning
        1. 3.2.1 Function approximation
      3. 3.3 Artificial neural networks
        1. 3.3.1 Feedforward neural networks
        2. 3.3.2 Parameter initialization
        3. 3.3.3 Gradient-based learning
        4. 3.3.4 Backpropagation
        5. 3.3.5 Activation function
        6. 3.3.6 Regularization
        7. 3.3.7 Batch normalization
        8. 3.3.8 Universal approximation theorem
        9. 3.3.9 Deep neural networks
        10. 3.3.10 Convolutional neural networks
        11. 3.3.11 Recurrent neural networks
    3. 4 Reinforcement learning
      1. 4.1 Introduction
      2. 4.2 Markov decision process
        1. 4.2.1 Infinite Markov decision process
        2. 4.2.2 Partially observable Markov decision process
      3. 4.3 Rewards
      4. 4.4 Value function and policy
      5. 4.5 Function approximation
      6. 4.6 Policy gradient methods
        1. 4.6.1 REINFORCE
        2. 4.6.2 Actor-critic
  3. II Methodology
    1. 5 Problem Setting
      1. 5.1 Assumptions
      2. 5.2 Time Discretization
      3. 5.3 State Space
      4. 5.4 Action Space
      5. 5.5 Reward Function
    2. 6 Reinforcement learning algorithm
      1. 6.1 Immediate reward environment
      2. 6.2 Direct policy gradient
      3. 6.3 Deterministic actor-critic
    3. 7 Network topology
      1. 7.1 Network input
      2. 7.2 Policy network
      3. 7.3 Q-network
      4. 7.4 Sequential information layer
        1. 7.4.1 Convolutional neural network
        2. 7.4.2 Long Short-Term Memory
      5. 7.5 Decision-making layer
      6. 7.6 Network optimization
        1. 7.6.1 Regularization
  4. III Experiments
    1. 8 Experiment and Results
      1. 8.1 Materials and Methods
        1. 8.1.1 Baselines
        2. 8.1.2 Hyperparameters
        3. 8.1.3 Training scheme
        4. 8.1.4 Performance metrics
        5. 8.1.5 Dataset
      2. 8.2 Results
      3. 8.3 Discussion of results
        1. 8.3.1 Risk/reward
        2. 8.3.2 RL models
        3. 8.3.3 Networks
      4. 8.4 Discussion of model
        1. 8.4.1 Environment
        2. 8.4.2 Optimization
        3. 8.4.3 Interpretability and trust
    2. 9 Future work
    3. 10 Conclusion

###  1 Introduction

####  1.1 Motivation

The transition to sustainable energy sources is one of the most critical
challenges facing the world today. By 2050, the European Union aims to become
carbon neutral [eur]. However, rising volatility in energy markets,
culminating in the 2021 global energy crisis, complicates this objective.
Supply and demand forces determine price dynamics, where an ever-increasing
share of supply stems from intermittent renewable energy sources such as wind
and solar power. Increasing reliance on intermittent energy sources leads to
unpredictable energy supply, contributing to volatile energy markets [SRH20].
Already volatile markets are further destabilized by evolutionary traits such
as fear and greed, causing human commodity traders to overreact [Lo04].
Volatile markets are problematic for producers and consumers, and failure to
mitigate these concerns may jeopardize decarbonization targets.

Algorithmic trading agents can stabilize commodity markets by systematically
providing liquidity and aiding price discovery [Isi21, Nar13]. Developing
these methods is non-trivial as financial markets are non-stationary with
complicated dynamics [Tal97]. Machine learning (ML) has emerged as the
preferred method in algorithmic trading due to its ability to learn to solve
complicated tasks by leveraging data [Isi21]. The majority of research on ML-
based algorithmic trading has focused on forecast-based supervised learning
(SL) methods, which tend to ignore non-trivial factors such as transaction
costs, risk, and the additional logic associated with mapping forecasts to
market positions [Fis18]. Reinforcement learning (RL) presents a suitable
alternative to account for these factors. In reinforcement learning,
autonomous agents learn to perform tasks in a time-series environment through
trial and error without human supervision. Around the turn of the millennium,
Moody and his collaborators [MW97, MWLS98, MS01] made several significant
contributions to this field, empirically demonstrating the advantages of
reinforcement learning over supervised learning for algorithmic trading.

In the last decade, the deep learning (DL) revolution has made exceptional
progress in areas such as image classification [HZRS15] and natural language
processing [VSP+17], characterized by complex structures and high signal-to-
noise ratios. The strong representation ability of deep learning methods has
even translated to forecasting low signal-to-noise financial data [XNS15,
HGMS18, MRC18]. In complex, high-dimensional environments, deep reinforcement
learning (deep RL), i.e., integrating deep learning techniques into
reinforcement learning, has yielded impressive results. Noteworthy
contributions include achieving superhuman play in Atari games [MKS+13] and
chess [SHS+17], and training a robot arm to solve the Rubik’s cube [AAC+19]. A
significant breakthrough was achieved in 2016 when the deep reinforcement
learning-based computer program AlphaGo[SHM+16] beat top Go player Lee Sedol.
In addition to learning by reinforcement learning through self-play, AlphaGo
uses supervised learning techniques to learn from a database of historical
games. In 2017, an improved version called AlphaGo Zero[SSS+17], which begins
with random play and relies solely on reinforcement learning, comprehensively
defeated AlphaGo. Deep RL has thus far been primarily studied in the context
of game-playing and robotics, and its potential application to financial
trading remains largely unexplored. Combining the two seems promising, given
the respective successes of reinforcement learning and deep learning in
algorithmic trading and forecasting.

####  1.2 Problem description

This thesis investigates the effectiveness of deep reinforcement learning
methods in commodities trading. It examines previous research in algorithmic
trading, state-of-the-art reinforcement learning, and deep learning
algorithms. The most promising methods are implemented, along with novel
improvements, to create a transaction-cost- and risk-sensitive parameterized
agent directly outputting market positions. The agent is optimized using
reinforcement learning algorithms, while deep learning methods extract
predictive patterns from raw market observations. These methods are evaluated
out-of-sample by backtesting on energy futures.

Machine learning relies on generalizability. A common criticism against
algorithmic trading approaches is their alleged inability to generalize to
“extreme” market conditions [Nar13]. This thesis investigates the performance
of algorithmic trading agents out-of-sample under unprecedented market
conditions caused by the energy crisis during 2021-2022. It will address the
following research questions

  1. 1.

Can the risk of algorithmic trading agents operating in volatile markets be
controlled?

  2. 2.

What reinforcement learning algorithms are suitable for optimizing an
algorithmic training agent in an online, continuous time setting?

  3. 3.

What deep learning architectures are suitable for modeling noisy, non-
stationary financial data?

####  1.3 Thesis Organization

The thesis consists of three parts: the background (part I), the methodology
(part II), and the experiments (part III). The list below provides a brief
outline of the chapters in this thesis:

  * •

Chapter 2: Overview of relevant concepts in algorithmic trading.

  * •

Chapter 3: Overview of relevant machine learning and deep learning concepts.

  * •

Chapter 4: Overview of relevant concepts in reinforcement learning.

  * •

Chapter 5: Formalization of the problem setting.

  * •

Chapter 6: Description of reinforcement learning algorithms.

  * •

Chapter 7: Description of the neural network function approximators.

  * •

Chapter 8: Detailed results from experiments.

  * •

Chapter 9: Suggested future work.

  * •

Chapter 10: Summary of contributions, results, and main conclusions.

##  Part I Background

###  2 Algorithmic trading

A phenomenon commonly described as an arms race has resulted from fierce
competition in financial markets. In this phenomenon, market participants
compete to remain on the right side of information asymmetry, which further
reduces the signal-to-noise ratio and the frequency at which information
arrives and is absorbed by the market [Isi21]. An increase in volatility and
the emergence of a highly sophisticated breed of traders called high-frequency
traders have further complicated already complex market dynamics. In these
developed, modern financial markets, the dynamics are so complex and change at
such a high frequency that humans will have difficulty competing. Indeed,
there is reason to believe that machines already outperform humans in the
world of financial trading. The algorithmic hedge fund Renaissance
Technologies, founded by famed mathematician Jim Simons, is considered the
most successful hedge fund ever. From 1988 to 2018, Renaissance Technologies’
Medallion fund generated 66 percent annualized returns before fees relying
exclusively on algorithmic strategies [Zuc19]. In 2020, it was estimated that
algorithmic trading accounts for around 60-73 percent of U.S. and European
equity trading, up from just 15 percent in 2003 [Int20]. Thus, it is clear
that algorithms already play a significant role in financial markets. Due to
the rapid progress of computing power111Moore’s law states that the number of
transistors in an integrated circuit doubles roughly every two years. relative
to human evolution, this importance will likely only grow.

This chapter provides an overview of this thesis’s subject matter, algorithmic
trading on commodity markets, examines related work, and justifies the
algorithmic trading methods described in part II. Section 2.1 presents a brief
overview of commodity markets and energy futures contracts. Sections 2.2, 2.3,
and 2.4 introduce some basic concepts related to trading financial markets
that are necessary to define and justify a trading agent’s goal of maximizing
risk-adjusted returns. This goal has two sub-goals: forecasting returns and
mapping forecasts to market positions, which are discussed separately in
sections 2.5 and 2.6. Additionally, these sections provide an overview of how
the concepts introduced in the following chapters 3 and 4 can be applied to
algorithmic trading and provide an overview of related research. The sections
2.7 and 2.8 describe how to represent a continuous financial market as
discrete inputs to an algorithmic trading system. To conclude, section 2.9
introduces backtesting, a form of cross-validation used to evaluate
algorithmic trading agents.

####  2.1 Commodity markets

Energy products trade alongside other raw materials and primary products on
commodity markets. The commodity market is an exchange that matches buyers and
sellers of the products offered at the market. Traditionally trading was done
in an open-outcry manner, though now an electronic limit order book is used to
maintain a continuous market. Limit orders specify the direction, quantity,
and acceptable price of a security. Limit orders are compared to existing
orders in the limit order book when they arrive on the market. A trade occurs
at the price set by the first order in the event of an overlap. The exchange
makes money by charging a fee for every trade, usually a small percentage of
the total amount traded.

The basis of energy trade is energy futures, a derivative contract with energy
products as the underlying asset [CGLL19]. Futures contracts are standardized
forward contracts listed on stock exchanges. They are interchangeable, which
improves liquidity. Futures contracts obligate a buyer and seller to transact
a given quantity of the underlying asset at a future date and price. The
quantity, quality, delivery location, and delivery date are all specified in
the contract. Futures contracts are usually identified by their expiration
month. The “front-month” is the nearest expiration date and usually represents
the most liquid market. Natural gas futures expire three business days before
the first calendar day of the delivery month. To avoid physical delivery of
the underlying commodity, the contract holder must sell their holding to the
market before expiry. Therefore, the futures and underlying commodity prices
converge as the delivery date approaches. A futures contract achieves the same
outcome as buying a commodity on the spot market on margin and storing it for
a future date. The relative price of these alternatives is connected as it
presents an arbitrage opportunity. The difference in price between a futures
contract and the spot price of the underlying commodity will therefore depend
on the financing cost, storage cost, and convenience yield of holding the
physical commodity over the futures contract. Physical traders use futures as
a hedge while transporting commodities from producer to consumer. If a trader
wishes to extend the expiry of his futures contract, he can “roll” the
contract by closing the contract about to expire and entering into a contract
with the same terms but a later expiry date [CGLL19]. The “roll yield” is the
difference in price for these two contracts and might be positive or negative.
The exchange clearinghouse uses a margin system with daily settlements between
parties to mitigate counterparty risk [CGLL19].

####  2.2 Financial trading

Financial trading is the act of buying and selling financial assets. Owning a
financial asset is called being long that asset, which will realize a profit
if the asset price increases and suffer a loss if the asset price decreases.
Short-selling refers to borrowing, selling, and then, at a later time,
repurchasing a financial asset and returning it to the lender with the hopes
of profiting from a price drop during the loan term. Short-selling allows
traders to profit from falling prices.

####  2.3 Modern portfolio theory

Harry Markowitz laid the groundwork for what is known as Modern Portfolio
Theory (MPT)[Mar68]. MPT assumes that investors are risk-averse and advocates
maximizing risk-adjusted returns. The Sharpe ratio [Sha98] is the most widely-
used measurement of risk-adjusted return developed by economist William F.
Sharpe. The Sharpe ratio compares excess return with the standard deviation of
investment returns and is defined as

$Sharpe\;ratio=\dfrac{\mathbb{E}[r_{t}-\bar{r}]}{\sqrt{var[r_{t}-\bar{r}]}}\simeq\dfrac{\mathbb{E}[r_{t}]}{\sigma_{r_{t}}}$
(2.1)

where $\mathbb{E}[r_{t}]$ is the expected return over $T$ samples, $\bar{r}$
is the risk-free rate, and $\sigma_{r_{t}}>0$ is the standard deviation of the
portfolio’s excess return. Due to negligible low interest rates, the risk-free
rate is commonly set to $\bar{r}=0$. The philosophy of MPT is that the
investor should be compensated through higher returns for taking on higher
risk. The St. Petersburg paradox222For an explanation of the paradox, see the
article [Pet22]. illustrates why maximizing expected reward in a risk-neutral
manner might not be what an individual wants. Although market participants
have wildly different objectives, this thesis will adopt the MPT philosophy of
assuming investors want to maximize risk-adjusted returns. Hence, the goal of
the trading agent described in this thesis will be to maximize the risk-
adjusted returns represented by the Sharpe ratio. Maximizing future risk-
adjusted returns can be broken down into two sub-goals; forecasting future
returns and mapping the forecast to market positions. However, doing so in
highly efficient and competitive financial markets is non-trivial.

####  2.4 Efficient market hypothesis

Actively trading a market suggests that the trader is dissatisfied with market
returns and believes there is potential for extracting excess returns, or
alpha. Most academic approaches to finance are based on the Efficient Market
Hypothesis (EMH)[Fam70], which states that all available information is fully
reflected in the prices of financial assets at any time. According to the EMH,
a financial market is a stochastic martingale process. As a consequence,
searching for alpha is a futile effort as the expected future return of a non-
dividend paying asset is the present value, regardless of past information,
i.e.,

$\mathbb{E}[R_{t+1}|I_{t}]=R_{t}$ (2.2)

Practitioners and certain parts of academia heavily dispute the EMH.
Behavioral economists reject the idea of rational markets and believe that
human evolutionary traits such as fear and greed distort market participants’
decisions, creating irrational markets. The Adaptive Market Hypothesis
(AMH)[Lo04] reconciles the efficient market hypothesis with behavioral
economics by applying evolution principles (competition, adaptation, and
natural selection) to financial interactions. According to the AHM, what
behavioral economists label irrational behavior is consistent with an
evolutionary model of individuals adapting to a changing environment.
Individuals within the market are continually learning the market dynamics,
and as they do, they adapt their trading strategies, which in turn changes the
dynamics of the market. This loop creates complicated price dynamics. Traders
who adapt quickly to changing dynamics can exploit potential inefficiencies.
Based on the AHM philosophy, this thesis hypothesizes that there are
inefficiencies in financial markets that can be exploited, with the
recognition that such opportunities are limited and challenging to discover.

####  2.5 Forecasting

Unless a person is gambling, betting on the price movements of volatile
financial assets only makes sense if the trader has a reasonable idea of where
the price is moving. Since traders face non-trivial transaction costs, the
expected value of a randomly selected trade is negative. Hence, as described
by the gambler’s ruin, a person gambling on financial markets will eventually
go bankrupt due to the law of large numbers. Forecasting price movements,
i.e., making predictions based on past and present data, is a central
component of any financial trading operation and an active field in academia
and industry. Traditional approaches include fundamental analysis, technical
analysis, or a combination of the two [GD34]. These can be further simplified
into qualitative and quantitative approaches (or a combination). A qualitative
approach, i.e., fundamental analysis, entails evaluating the subjective
aspects of a security [GD34], which falls outside the scope of this thesis.
Quantitative (i.e., technical) traders use past data to make predictions
[GD34]. The focus of this thesis is limited to fully quantitative approaches.

Developing quantitative forecasts for the price series of financial assets is
non-trivial as financial markets are non-stationary with a low signal-to-noise
ratio [Tal97]. Furthermore, modern financial markets are highly competitive
and effective. As a result, easily detectable signals are almost certainly
arbitraged out. Researchers and practitioners use several mathematical and
statistical models to identify predictive signals leading to excess returns.
Classical models include the autoregressive integrated moving average (ARIMA)
and the generalized autoregressive conditional heteroskedasticity (GARCH). The
ARIMA is a linear model and a generalization of the autoregressive moving
average (ARMA) that can be applied to time series with nonstationary mean (but
not variance) [SSS00]. The assumption of constant variance (i.e., volatility)
is not valid for financial markets where volatility is stochastic [Tal97]. The
GARCH is a non-linear model developed to handle stochastic variance by
modeling the error variance as an ARMA model [SSS00]. Although the ARIMA and
GARCH have practical applications, their performance in modeling financial
time series is generally unsatisfactory [XNS15, MRC18].

Over the past 20 years, the availability and affordability of computing power,
storage, and data have lowered the barrier of entry to more advanced
algorithmic methods. As a result, researchers and practitioners have turned
their attention to more complex machine learning methods because of their
ability to identify signals and capture relationships in large datasets.
Initially, there was a flawed belief that the low signal-to-noise ratio leaves
viable only simple forecasts such as those based on low-dimensional ordinary
least squares [Isi21]. With the recent deep learning revolution, deep neural
networks have demonstrated strong representation abilities when modeling time
series data [SVL14]. The Makridakis competition evaluates time series
forecasting methods. In its fifth installment held in 2020, all 50 top-
performing models were based on deep learning architectures [MSA22]. A
considerable amount of recent empirical research suggests that deep learning
models significantly outperform traditional models like the ARIMA and GARCH
when forecasting financial time series [XNS15, MRC18, SNN18, SGO20]. These
results are somewhat puzzling. The risk of overfitting is generally higher for
noisy data like financial data. Moreover, the loss function for DNNs is non-
convex, which makes finding a global minimum impossible. Despite the elevated
overfitting risk and the massive overparameterization of DNNs, they still
demonstrate stellar generalization. Thus, based on recent research, the thesis
will apply deep learning techniques to model financial time series.

A review of deep learning methods in financial time series forecasting [SGO20]
found that LSTMs were the preferred choice in sequence modeling, possibly due
to their ability to remember both long- and short-term dependencies.
Convolutional neural networks are another common choice. CNNs are best known
for their ability to process 2D grids such as images; however, they have shown
a solid ability to model 1D grid time series data. Using historical prices,
Hiransha et al. [HGMS18] tested FFNs, vanilla RNNs, LSTMs, and CNNs on
forecasting next-day stock market returns on the National Stock Exchange (NSE)
of India and the New York Stock Exchange (NYSE). In the experiment, CNNs
outperformed other models, including the LSTM. These deep learning models can
extract generalizable patterns from the price series alone [SGO20].

####  2.6 Mapping forecasts to market positions

Most research on ML in financial markets focuses on forecast-based supervised
learning approaches [Fis18]. These methods tend to ignore how to convert
forecasts into market positions or use some heuristics like the Kelly
criterion to determine optimal position sizing [Isi21]. The forecasts are
usually optimized by minimizing a loss function like the Mean Squared Error
(MSE). An accurate forecast (in the form of a lower MSE) may lead to a more
profitable trader, but this is not always true. Not only does the discovered
signal need adequate predictive power, but it must consistently produce
reliable directional calls. Moreover, the mapping from forecast to market
position needs to consider transaction costs and risk, which is challenging in
a supervised learning framework [MW97]. Neglecting transaction costs can lead
to aggressive trading and overestimation of returns. Neglecting risk can lead
to trading strategies that are not viable in the real world. Maximizing risk-
adjusted returns is only feasible when accounting for transaction costs and
risk. These shortcomings are addressed using reinforcement learning [MW97,
MWLS98]. Using RL, deep neural networks can be trained to output market
positions directly. Moreover, the DNN can be jointly optimized for risk- and
transaction-cost-sensitive returns, thus directly optimizing for the true
goal: maximizing risk-adjusted returns.

Moody and Wu [MW97] and Moody et al. [MWLS98] empirically demonstrated the
advantages of reinforcement learning relative to supervised learning. In
particular, they demonstrated the difficulty of accounting for transaction
costs using a supervised learning framework. A significant contribution is
their model-free policy-based RL algorithm for trading financial instruments
recurrent reinforcement learning (RRL). The name refers to the recursive
mechanism that stores the past action as an internal state of the environment,
allowing the agent to consider transaction costs. The agent outputs market
positions and is limited to a discrete action space $a_{t}\in\\{-1,0,1\\}$,
corresponding to maximally short, no position, and maximally long. At time
$t$, the previous action $a_{t-1}$ is fed into the policy network $f_{\theta}$
along with the external state of the environment $s_{t}$ in order to make the
trade decision, i.e.,

$a_{t}=f_{\theta}(s_{t},a_{t-1})$

where $f_{\theta}$ is a linear function, and the external state is constructed
using the past $8$ returns. The return $r_{t}$ is realized at the end of the
period $(t-1,t]$ and includes the returns resulting from the position
$a_{t-1}$ held through this period minus transaction costs incurred at time
$t$ due to a difference in the new position $a_{t}$ from the old $a_{t-1}$.
Thus, the agent learns the relationship between actions and the external state
of the environment and the internal state.

Moody and Saffel [MS01] compared their actor-based RRL algorithm to the value-
based Q-learning algorithm when applied to financial trading. The algorithms
are tested on two real financial time series; the U.S. dollar/British pound
foreign exchange pair and the S&P 500 stock index. While both perform better
than a buy-and-hold baseline, the RRL algorithm outperforms Q-learning on all
tests. The authors argue that actor-based algorithms are better suited to
immediate reward environments and may be better able to deal with noisy data
and quickly adapt to non-stationary environments. They point out that critic-
based RL suffers from the curse of dimensionality and that when extended to
function approximation, it sometimes fails to converge even in simple MDPs.

Deng et al. [DBK+16] combine Moody’s direct reinforcement learning framework
with a recurrent neural network to introduce feature learning through deep
learning. Another addition is the use of continuous action space. To constrain
actions to the interval $[-1,1]$, the RNN output is mapped to a $\tanh{}$
function. Jiang et al. [JXL17] presents a deterministic policy gradient
algorithm that trades a portfolio of multiple financial instruments. The
policy network is modeled using CNNs and LSTMs, taking each period’s closing,
lowest, and highest prices as input. The DNNs are trained on randomly sampled
mini-batches of experience. These methods account for transaction costs but
not risk. Zhang et al. [ZZW+20] present a deep RL framework for a risk-averse
agent trading a portfolio of instruments using both CNNs and LSTMs. Jin and
El-Saawy [JES16] suggest that adding a risk-term to the reward function that
penalizes the agent for volatility produces a higher Sharpe ratio than
optimizing for the Sharpe ratio directly. Zhang et al. [ZZW+20] apply a
similar risk-term penalty to the reward function.

####  2.7 Feature engineering

Any forecast needs some predictor data, or features, to make predictions.
While ML forecasting is a science, feature engineering is an art and arguably
the most crucial part of the ML process. Feature engineering and selection for
financial forecasting are only limited by imagination. Features range from
traditional technical indicators (e.g., Moving Average Convergence Divergence,
Relative Strength Index) [ZZR20] to more modern deep learning-based techniques
like analyzing social-media sentiment of companies using Natural Language
Processing [ZS10] or using CNNs on satellite images along with weather data to
predict cotton yields [TOdSMJZ20]. Research in feature engineering and
selection is exciting and potentially fruitful but beyond this thesis’s scope.
The most reliable predictor of future prices of a financial instrument tends
to be its past price, at least in the short term [Isi21]. Therefore, in this
thesis, higher-order features are not manually extracted. Instead, only the
price series are analyzed.

####  2.8 Sub-sampling schemes

Separating high- and low-frequency trading can be helpful, as they present
unique challenges. High-frequency trading (HFT) focuses on reducing software
and hardware latency, which may include building a $300 million fiber-optic
cable to reduce transmission time by four milliseconds between exchanges to
gain a competitive advantage [LB14] 333Turns out they forgot that light
travels about $30\%$ slower in glass than in air, and they lost their
competitive advantage to simple line-of-sight microwave networks [LB14].. This
type of trading has little resemblance to the low-frequency trading examined
in this thesis, described in minutes or hours rather than milliseconds.

Technical traders believe that the prices of financial instruments reflect all
relevant information [Tal97]. From this perspective, the market’s complete
order history represents the financial market’s state. This state
representation would scale poorly, with computational and memory requirements
growing linearly with time. Consequently, sub-sampling schemes for periodic
feature extraction are almost universally employed. While sampling information
at fixed intervals is straightforward, there may be more effective methods. As
exchange activity varies throughout the day, sampling at fixed intervals may
lead to oversampling during low-activity periods and undersampling during
high-activity periods. In addition, time-sampled series often exhibit poor
statistical properties, such as non-normal returns, autocorrelation, and
heteroskedasticity [DP18].

The normality of returns assumption underpins several mathematical finance
models, e.g., Modern Portfolio Theory [Mar68], and the Sharpe-ratio [Sha98].
There is, however, too much peaking and fatter tails in the actual observed
distribution for it to be relative to samples from Gaussian populations
[Man97] 444Assuming that the S&P 500 index returns were normally distributed,
the probability of daily returns being below five percent between 1962 and
2004 ($10\;698$ observations) would be approximately $0.0005$. However, it
happened $8$ times [Has07].. Mandelbrot showed in 1963 [Man97] that a Lévy
alpha-stable distribution with infinite variance can approximate returns over
fixed periods. In 1967, Mandelbrot and Taylor [MT67] argued that returns over
a fixed number of transactions may be close to Independent and Identically
Distributed (IID)Gaussian. Several empirical studies have since confirmed this
[Cla73, AG00]. Clark [Cla73] discovered that sampling by volume instead of
transactions exhibits better statistical properties, i.e., closer to IID
Gaussian distribution. Sampling by volume instead of ticks has intuitive
appeal. While tick bars count one transaction of $n$ contracts as one bar, $n$
transactions of one contract count as $n$ bars. Sampling according to
transaction volume might lead to significant sampling frequency variations for
volatile securities. When the price is high, the volume will be lower, and
therefore the number of observations will be lower, and vice versa, even
though the same value might be transacted. Therefore, sampling by the monetary
value transacted, also called dollar bars, may exhibit even better statistical
properties [DP18]. Furthermore, for equities, sampling by monetary value
exchanged makes an algorithm more robust against corporate actions like stock
splits, reverse splits, stock offerings, and buybacks. To maintain a suitable
sampling frequency, the sampling threshold may need to be adjusted if the
total market size changes significantly.

Although periodic feature extraction reduces the number of observations that
must be processed, it scales linearly in computation and memory requirements
per observation. A history cut-off is often employed to represent the state by
only the $n$ most recent observations to tackle this problem. Representing the
state of a partially observable MDP by the $n$ most recent observations is a
common technique used in many reinforcement learning applications. Mnih et al.
[MKS+13] used $4$ stacked observations as input to the DQN agent that achieved
superhuman performance on Atari games to capture the trajectory of moving
objects on the screen. The state of financial markets is also usually
approximated by stacking past observations [JXL17, ZZR20, ZZW+20].

####  2.9 Backtesting

Assessing a machine learning model involves estimating its generalization
error on new data. The most widely used method for estimating generalization
error is Cross-Validation (CV), which assumes that observations are IID and
drawn from a shared underlying data-generating distribution. However, the
price of a financial instrument is a nonstationary time series with an
apparent temporal correlation. Conventional cross-validation ignores this
temporal component and is thus unsuitable for assessing a time series
forecasting model. Instead, backtesting, a form of cross-validation for time
series, is used. Backtesting is a historical simulation of how the model would
have performed should it have been run over a past period. The purpose of
backtesting is the same as for cross-validation; to determine the
generalization error of an ML algorithm.

To better understand backtesting, it is helpful to consider an algorithmic
trading agent’s objective and how it operates to achieve it. The algorithmic
trading process involves the agent receiving information and, based on that
information, executing trades at discrete time steps. These trades are
intended to achieve a specific objective set by the stakeholder, which, in the
philosophy of modern portfolio theory, is maximizing risk-adjusted returns.
Thus, assessing an algorithmic trading agent under the philosophy of modern
portfolio theory entails estimating the risk-adjusted returns resulting from
the agent’s actions. However, when testing an algorithmic trading agent, it
cannot access data ahead of the forecasting period, as that would constitute
information leakage to the agent. For this reason, conventional cross-
validation fails in algorithmic trading.

The most precise way to assess the performance of an algorithmic trading agent
is to deploy it to the market, let it trade with the intended amount of
capital, and observe its performance. However, this task would require
considerable time since low-frequency trading algorithms are typically
assessed over long periods, often several years555There are a couple of
reasons for this; first, there are, on average, 252 trading days per year.
Low-frequency trading algorithms typically make fewer than ten trades per day.
In order to obtain sufficient test samples, the agent must trade for several
years. Second, testing the algorithmic trading agent under various market
conditions is crucial. A successful model in particular market conditions may
be biased towards those conditions and fail to generalize to other market
conditions. . Additionally, any small error would likely result in devastating
losses, making algorithmic trading projects economically unfeasible.

Figure 2.1: Time series cross-validation (backtesting) compared to standard
cross-validation from [HA18].

Backtesting is an alternative to this expensive and time-consuming process
where performance on historical simulations functions as a proxy for
generalization error. Backtesting involves a series of tests that progress
sequentially through time, where every test set consists of a single
observation. At test $n$, the model trains on the training set consisting of
observations prior to the observation in the test set ($i<n$). The forecast
is, therefore, not based on future observations, and there is no leakage from
the test set to the training set. Then the backtest progresses to the
subsequent observation $n+1$, where the training set increases666The test set
can also be a fixed-size FIFO queue. to include observations $i<n+1$. The
backtest progresses until there are no test sets left. The backtesting process
is formalized in algorithm 1.

Algorithm 1 Backtesting

Train the model on the first $k$ observations (of $T$ total observations).

for $i=0,1,...,T-k$ do

Select observation $k+i$ from the test set.

Register trade $a_{k+i}$.

Train the model using observations at times $t\leq k+i$.

end for

Measure performance using registered trades $a_{k},a_{k+1},...,a_{T}$ and the
corresponding prices at time $k,k+1,...,T$.

When conducting historical simulations, knowing what information was available
during the studied period is critical. Agents should not have access to data
beyond the point at which they are located in the backtest, in order to avoid
lookahead bias. Lookahead bias is a research bug of inadvertently using future
data, whereas real-time production trading is free of such “feature”. Data
used in forecasting must be stored point-in-time (PIT), which indicates when
the information was made available. An incorrectly labeled dataset can lead to
lookahead bias.

Backtesting is a flawed procedure as it suffers from lookahead bias by design.
Having experienced the hold-out test sample provides insight into what made
markets rise and fall, a luxury not available before the fact. Thus, only live
trading can be considered genuinely out-of-sample. Another form of lookahead
bias is repeated and excessive backtest optimization leading to information
leakage from the test to the training set. A machine learning model will
enthusiastically grab any lookahead but fail to generalize to live trading.
Furthermore, the backtests performed in this thesis rely on assumptions of
zero market impact, zero slippage, fractional trading, and sufficient
liquidity to execute any trade instantaneously at the quoted price. These
assumptions do not reflect actual market conditions and will lead to
unrealistic high performance in the backtest.

Backtesting should emulate the scientific method, where a hypothesis is
developed, and empirical research is conducted to find evidence of
inconsistencies with the hypothesis. It should be distinct from a research
tool for discovering predictive signals. It should only be conducted after
research has been done. A backtest is not an experiment but a simulation to
see if the model behaves as expected [DP18]. Random historical patterns might
exhibit excellent performance in a backtest. However, it should be viewed
cautiously if no ex-ante logical foundation exists to explain the performance
[AHM19].

###  3 Deep learning

An intelligent agent requires some means of modeling the dynamics of the
system in which it operates. Modeling financial markets is complicated due to
low signal-to-noise ratios and non-stationary dynamics. The dynamics are
highly nonlinear; thus, several traditional statistical modeling approaches
cannot capture the system’s complexity. Moreover, reinforcement learning
requires parameterized function approximators, rendering nonparametric
learners, e.g., support vector machines and random forests, unsuitable.
Additionally, parametric learners are generally preferred when the predictor
data is well-defined[GBC16], such as when forecasting security prices using
historical data. In terms of nonlinear parametric learners, artificial neural
networks comprise the most widely used class. Research suggests that deep
neural networks, such as LSTMs and CNNs, effectively model financial data
[XNS15, HGMS18, MRC18]. Therefore, the algorithmic trading agent proposed in
this thesis will utilize deep neural networks to model commodity markets.

This chapter introduces the fundamental concepts of deep learning relevant to
this thesis, starting with some foundational concepts related to machine
learning 3.1 and supervised learning 3.2. Next, section 3.3 covers artificial
neural networks, central network topologies, and how they are initialized and
optimized to achieve satisfactory results. The concepts presented in this
chapter are presented in the context of supervised learning but will be
extended to the reinforcement learning context in the next chapter (4).

####  3.1 Machine learning

Machine Learning (ML)studies how computers can automatically learn from
experience without being explicitly programmed. A general and comprehensive
introduction to machine learning can be found in “Elements of Statistical
Learning” by Trevor Hastie, Robert Tibshirani, and Jerome Friedman [HTFF09].
Tom Mitchell defined the general learning problem as “A computer program is
said to learn from experience E with respect to some class of tasks T and
performance measure P, if its performance on T, as measured by P, improves
with experience E.” [MM97]. In essence, ML algorithms extract generalizable
predictive patterns from data from some, usually unknown, probability
distribution to build a model about the space. It is an optimization problem
where performance improves through leveraging data. Generalizability relates
to a model’s predictive performance on independent test data and is a crucial
aspect of ML. Models should be capable of transferring learned patterns to
new, previously unobserved samples while maintaining comparable performance.
ML is closely related to statistics in transforming raw data into knowledge.
However, while statistical models are designed for inference, ML models are
designed for prediction. There are three primary ML paradigms; supervised
learning, unsupervised learning, and reinforcement learning.

#####  3.1.1 No-free-lunch theorem

The no-free-lunch theorem states that there exists no single universally
superior ML learning algorithm that applies to all possible datasets.
Fortunately, this does not mean ML research is futile. Instead, domain-
specific knowledge is required to design successful ML models. The no-free-
lunch theorem results only hold when averaged over all possible data-
generating distributions. If the types of data-generating distributions are
restricted to classes with certain similarities, some ML algorithms perform
better on average than others. Instead of trying to develop a universally
superior machine learning algorithm, the focus of ML research should be on
what ML algorithms perform well on specific data-generating distributions.

#####  3.1.2 The curse of dimensionality

The Hughes phenomenon states that, for a fixed number of training examples, as
the number of features increases, the average predictive power for a
classifier will increase before it starts deteriorating. Bellman termed this
phenomenon the curse of dimensionality, which frequently manifests itself in
machine learning [SB18]. One manifestation of the curse is that the sampling
density is proportional to $N^{1/p}$, where $p$ is the dimension of the input
space and $N$ is the sample size. If $N=100$ represents a dense sample for
$p=1$, then $N^{1/10}=100^{10}$ is the required sample size for the same
sampling density with $p=10$ inputs. Therefore, in high dimensions, the
training samples sparsely populate the input space [HTFF09]. In Euclidean
spaces, a non-negative term is added to the distance between points with each
new dimension. Thus, generalizing from training samples becomes more complex,
as it is challenging to say something about a space without relevant training
examples.

####  3.2 Supervised learning

Supervised Learning (SL)is the machine learning paradigm where a labeled
training set of $N\in\mathbb{N}_{+}$ observations
$\tau=\\{x_{i},y_{i}\\}_{i=1}^{N}$ is used to learn a functional dependence
$\mathbf{y}=\hat{f}(\mathbf{x})$ that can predict $\mathbf{y}$ from a
previously unobserved $\mathbf{x}$. Supervised learning includes regression
tasks with numerical targets and classification tasks with categorical
targets. A supervised learning algorithm adjusts the input/output relationship
of $\hat{f}$ in response to the prediction error to the target
$y_{i}-\hat{f}(x_{i})$. The hypothesis is that if the training set is
representative of the population, the model will generalize to previously
unseen examples.

#####  3.2.1 Function approximation

Function approximation, or function estimation, is an instance of supervised
learning that concerns selecting a function among a well-defined class that
underlies the predictive relationship between the input vector $\mathbf{x}$
and output variable $\mathbf{y}$. In most cases,
$\mathbf{x}\in\mathbb{R}^{d}$, where $d\in\mathbb{N}_{+}$, and
$\mathbf{y}\in\mathbb{R}$. Function approximation relies on the assumption
that there exists a function $f(\mathbf{x})$ that describes the approximate
relationship between $(\mathbf{x},\mathbf{y})$. This relationship can be
defined as

$\mathbf{y}=f(\mathbf{x})+\mathbf{\epsilon}$ (3.1)

where $\epsilon$ is some irreducible error that is independent of $\mathbf{x}$
where $\mathbb{E}[\epsilon]=0$ and $Var(\epsilon)=\sigma^{2}_{\epsilon}$. All
departures from a deterministic relationship between $(\mathbf{x},\mathbf{y})$
are captured via the error $\epsilon$. The objective of function approximation
is to approximate the function $f$ with a model $\hat{f}$. In reality, this
means finding the optimal model parameters $\theta$. Ordinary least squares
can estimate linear models’ parameters $\theta$.

###### Bias-variance tradeoff

Bias and variance are two sources of error in model estimates. Bias measures
the in-sample expected deviation between the model estimate and the target and
is defined as

$Bias^{2}(\hat{f}(\mathbf{x}))=[\mathbb{E}[\hat{f}(\mathbf{x})]-f(\mathbf{x})]^{2}$
(3.2)

and is a decreasing function of complexity. Variance measures the variability
in model estimates and is defined as

$Var(\hat{f}(\mathbf{x}))=\mathbb{E}[\hat{f}(\mathbf{x})-\mathbb{E}[\hat{f}(\mathbf{x})]]^{2}$
(3.3)

and is an increasing function of complexity. The out-of-sample mean square
error for model estimates is defined as

$MSE=Bias^{2}(\hat{f})+Var(\hat{f})+Var(\epsilon)$ (3.4)

where the last term $Var(\epsilon)$ is the irreducible noise error due to a
target component $\epsilon$ not predictable by $\mathbf{x}$. The bias can be
made arbitrarily small using a more complex model; however, this will increase
the model variance, or generalization error, when switching from in-sample to
out-of-sample. This is known as the bias-variance tradeoff.

###### Overfitting

A good ML model minimizes the model error, i.e., the training error (bias) and
the generalization error (variance). This is achieved at some optimal
complexity level, dependent on the data and the model. Increasing model
complexity, or capacity, can minimize training error. However, such a model is
unlikely to generalize well. Therefore, the difference between training error
and generalization error will be high. This is known as overfitting and
happens when the complexity of the ML model exceeds that of the underlying
problem. Conversely, underfitting is when the training error is high because
the model’s complexity is lower than that of the underlying problem.

Figure 3.1: An illustration of the relationship between the capacity of a
function approximator and the generalization error from [GBC16].

To minimize model error, the complexity of the ML model, or its inductive
bias, must align with that of the underlying problem. The principle of
parsimony, known as Occam’s razor, states that among competing hypotheses
explaining observations equally well, one should pick the simplest one. This
heuristic is typically applied to ML model selection by selecting the simplest
model from models with comparable performance.

Recent empirical evidence has raised questions about the mathematical
foundations of machine learning. Complex models such as deep neural networks
have been shown to decrease both training error and generalization error with
growing complexity [ZBH+21]. Furthermore, the generalization error keeps
decreasing past the interpolation limit. These surprising results contradict
the bias-variance tradeoff that implies that a machine learning model should
balance over- and underfitting. Belkin et al. [BHMM19] reconciled these
conflicting ideas by introducing a “double descent” curve to the bias-variance
tradeoff curve. This increases performance when increasing the model capacity
beyond the interpolation point.

####  3.3 Artificial neural networks

An Artificial Neural Network (ANN)is a parametric learner fitting nonlinear
models. The network defines a mapping
$h_{\theta}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ where $n$ is the input
dimension, $m$ is the output dimension, and $\theta$ are the network weights.
A neural network has a graph-like topology. It is a collection of nodes
organized in layers like a directed and weighted graph. The nodes of an ANN
are typically separated into layers; the input layer, one or more hidden
layers, and the output layer. Their dimensions depend on the function being
approximated. A multi-layer neural network is called a Deep Neural Network
(DNN).

#####  3.3.1 Feedforward neural networks

A Feedforward Network (FFN), or fully-connected network, defines the
foundational class of neural networks where the connections are a directed
acyclic graph that only allows signals to travel forward in the network. A
feedforward network is a mapping $h_{\theta}$ that is a composition of
multivariate functions $f_{1},f_{2},...,f_{k},g$, where $k$ is the number of
layers in the neural network. It is defined as

$h_{\theta}=g\circ f_{k}\circ...\circ f_{2}\circ f_{1}(x)$ (3.5)

The functions $f_{j}$, $j=1,2,...,k$ represent the network’s hidden layers and
are composed of multivariate functions. The function at layer $j$ is defined
as

$f_{j}(x)=\phi_{j}(\theta_{j}x+b_{j})$ (3.6)

where $\phi_{j}$ is the activation function and $b_{j}$ is the bias at layer
$j$. The activation function is used to add nonlinearity to the network. The
network’s final output layer function $g$ can be tailored to suit the problem
the network is solving, e.g., linear for Gaussian output distribution or
Softmax distribution for categorical output distribution.

Figure 3.2: Feedforward neural network from [FFLb].

#####  3.3.2 Parameter initialization

Neural network learning algorithms are iterative and require some starting
point from which to begin. The initial parameters of the networks can affect
the speed and level of convergence or even whether the model converges at all.
Little is known about weight initialization, a research area that is still in
its infancy. Further complicating the issue: initial parameters favorable for
optimization might be unfavorable for generalization [GBC16]. Developing
heuristics for parameter initialization is, therefore, non-trivial. Randomness
and asymmetry between the network units are desirable properties for the
initial weights [GBC16]. Weights are usually drawn randomly from a Gaussian or
uniform distribution in a neighborhood around zero, while the bias is usually
set to some heuristically chosen constant. Larger initial weights will prevent
signal loss during forward- and backpropagation. However, too large values can
result in exploding values, a problem particularly prevalent in recurrent
neural networks. The initial scale of the weights is usually set to something
like $1/\sqrt{m}$ where $m$ is the number of inputs to the network layer.

Kaiming initialization [HZRS15] is a parameter initialization method that
takes the type of activation function (e.g., Leaky-ReLU) used to add
nonlinearity to the neural network into account. The key idea is that the
initialization method should not exponentially reduce or magnify the magnitude
of input signals. Therefore, each layer is initialized at separate scales
depending on their size. Let $m_{l}\in\mathbb{N}_{+}$ be the size of the
inputs into the layer $l\in\mathbb{N}_{+}$. Kaiming He et al. recommend
initializing weights such that

$\frac{1}{2}m_{l}\textrm{Var}{[\theta_{l}]}=1$

Which corresponds to an initialization scheme of

$w_{l}\sim\mathcal{N}(0,2/m_{l})$

Biases are initialized at $0$.

#####  3.3.3 Gradient-based learning

Neural nets are optimized by adjusting their weights $\theta$ with the help of
objective functions. Let $J(\theta)$ define the differentiable objective
function for a neural network, where $\theta$ are the network weights. The
choice of the objective function and whether it should be minimized or
maximized depends on the problem being solved. For regression tasks, the
objective is usually to minimize some loss function like mean-squared error
(MSE)

$J(\theta)=\dfrac{1}{n}\sum_{i=1}^{n}\left(h_{\theta}(x_{i})-y_{i}\right)^{2}$
(3.7)

Due to neural nets’ nonlinearity, most loss functions are non-convex, meaning
it is impossible to find an analytical solution to $\nabla J(\theta)=0$.
Instead, iterative, gradient-based optimization algorithms are used. There are
no convergence guarantees, but it often finds a satisfactorily low value of
the loss function relatively quickly. Gradient descent-based algorithms adjust
the weights $\theta$ in the direction that minimizes the MSE loss function.
The update rule for parameter weights in gradient descent is defined as

$\theta_{t+1}=\theta_{t}-\alpha\nabla_{\theta}J(\theta_{t})$ (3.8)

where $\alpha>0$ is the learning rate and the gradient $\nabla J(\theta_{t})$
is the partial derivatives of the objective function with respect to each
weight. The learning rate defines the rate at which the weights move in the
direction suggested by the gradient of the objective function. Gradient-based
optimization algorithms, also called first-order optimization algorithms, are
the most common optimization algorithms for neural networks [GBC16].

###### Stochastic gradient descent

Optimization algorithms that process the entire training set simultaneously
are known as batch learning algorithms. Using the average of the entire
training set allows for calculating a more accurate gradient estimate. The
speed at which batch learning converges to a local minima will be faster than
online learning. However, batch learning is not suitable for all problems,
e.g., problems with massive datasets due to the high computational costs of
calculating the full gradient or problems with dynamic probability
distributions.

Instead, Stochastic Gradient Descent (SGD)is often used when optimizing neural
networks. SGD replaces the gradient in conventional gradient descent with a
stochastic approximation. Furthermore, the stochastic approximation is only
calculated on a subset of the data. This reduces the computational costs of
high-dimensional optimization problems. However, the loss is not guaranteed to
decrease when using a stochastic gradient estimate. SGD is often used for
problems with continuous streams of new observations rather than a fixed-size
training set. The update rule for SGD is similar to the one for GD but
replaces the true gradient with a stochastic estimate

$\theta_{t+1}=\theta_{t}-\alpha_{t}\nabla_{\theta}J^{(j)}(\theta_{t})$ (3.9)

where $\nabla_{\theta}J^{(j)}(\theta)$ is the stochastic estimate of the
gradient computed from observation $j$. The total loss is defined as
$J(\theta)=\sum_{j=1}^{N}J^{(j)}(\theta)$ where $N\in\mathbb{N}$ is the total
number of observations. The learning rate at time $t$ is $\alpha_{t}>0$. Due
to the noise introduced by the SGD gradient estimate, gradually decreasing the
learning rate over time is crucial to ensure convergence. Stochastic
approximation theory guarantees convergence to a local optima if $\alpha$
satisfies the conditions $\sum\alpha=\infty$ and $\sum\alpha^{2}<\infty$. It
is common to adjust the learning rate using the following update rule
$\alpha_{t}=(1-\beta)\alpha_{0}+\beta\alpha_{\tau}$, where
$\beta=\dfrac{t}{\tau}$, and the learning rate is kept constant after $\tau$
iterations, i.e., $\forall t\geq\tau$, $\alpha_{t}=\alpha_{\tau}$ [GBC16].

Due to hardware parallelization, simultaneously computing the gradient of $N$
observations will usually be faster than computing each gradient separately
[GBC16]. Neural networks are, therefore, often trained on mini-batches, i.e.,
sets of more than one but less than all observations. Mini-batch learning is
an intermediate approach to fully online learning and batch learning where
weights are updated simultaneously after accumulating gradient information
over a subset of the total observations. In addition to providing better
estimates of the gradient, mini-batches are more computationally efficient
than online learning while still allowing training weights to be adjusted
periodically during training. Therefore, minibatch learning can be used to
learn systems with dynamic probability distributions. Samples of the mini-
batches should be independent and drawn randomly. Drawing ordered batches will
result in biased estimates, especially for data with high temporal
correlation.

Due to noisy gradient estimates, stochastic gradient descent and mini-batches
of small size will exhibit higher variance than conventional gradient descent
during training. The higher variance can be helpful to escape local minima and
find new, better local minima. However, high variance can also lead to
problems such as overshooting and oscillation that can cause the model to fail
to converge. Several extensions have been made to stochastic gradient descent
to circumvent these problems.

###### Adaptive gradient algorithm

The Adaptive Gradient (AdaGrad) is an extension to stochastic gradient descent
introduced in 2011 [DHS11]. It outlines a strategy for adjusting the learning
rate to converge quicker and improving the capability of the optimization
algorithm. A per-parameter learning rate allows AdaGrad to improve performance
on problems with sparse gradients. Learning rates are assigned lower for
parameters with frequently occurring features and higher for parameters with
less frequently occurring features. The AdaGrad update rule is given as

$\theta_{t+1}=\theta_{t}-\dfrac{\alpha}{\sqrt{G_{t}+\epsilon}}g_{t}$ (3.10)

where $g_{t}=\nabla_{\theta}J^{(j)}(\theta_{t})$, and
$G_{t}=\sum_{\tau=1}^{t}{g_{t}g_{t}^{\top}}$, is the outer product of all
previous subgradients. $\epsilon>0$ is a smoothing term to avoid division by
zero. As training proceeds, the squared gradients in the denominator of the
learning rate will continue to grow, resulting in a strictly decreasing
learning rate. As a result, the learning rate will eventually become so small
that the model cannot acquire new information.

###### Root mean square propagation

Root Mean Square Propagation (RMSProp) is an unpublished extension to SGD
developed by Geoffrey Hinton. RMSProp was developed to resolve the problem of
AdaGrad’s diminishing learning rate. Like AdaGrad, it maintains a per-
parameter learning rate. To normalize the gradient, it keeps a moving average
of squared gradients. This normalization decreases the learning rate for more
significant gradients to avoid the exploding gradient problem and increases it
for smaller gradients to avoid the vanishing gradient problem. The RMSProp
update rule is given as

$\theta_{t+1}=\theta_{t}-\dfrac{\alpha}{\sqrt{E[g^{2}]_{t}+\epsilon}}g_{t}$
(3.11)

where $E[g^{2}]_{t}=\beta E[g^{2}]_{t}+(1-\beta)g^{2}_{t}$ where $v_{t}$ is
the exponentially decaying average of squared gradients and $\beta>0$ is a
second learning rate conventionally set to $\beta=0.9$.

###### Adam

The Adam optimization algorithm is an extension of stochastic gradient descent
that has recently seen wide adoption in deep learning. It was introduced in
2015 [KB14] and derives its name from adaptive moment estimation. It utilizes
the Adaptive Gradient (AdaGrad) Algorithm and Root Mean Square Propagation
(RMSProp). Adam only requires first-order gradients and little memory but is
computationally efficient and works well with high-dimensional parameter
spaces. As with AdaGrad and RMSProp, Adam utilizes independent per-parameter
learning rates separately adapted during training. Adam stores a moving
average of gradients $E[g]_{t}=\beta_{1}E[g]_{t}+(1-\beta_{1})g_{t}$ with
learning rate $\beta_{1}>0$. Like RMSProp, Adam also stores a moving average
of squared gradients $E[g^{2}]_{t}$ with learning rate $\beta_{2}>0$. The Adam
update rule is given as

$\theta_{t+1}=\theta_{t}-\dfrac{\alpha}{\sqrt{\hat{E[g^{2}]_{t}}}+\epsilon}\hat{E[g]_{t}}$
(3.12)

where $\hat{E[g^{2}]_{t}}=\dfrac{E[g^{2}]_{t}}{1-\beta_{2}^{t}}$ and
$\hat{E[g]_{t}}=\dfrac{E[g]_{t}}{1-\beta_{1}^{t}}$. The authors recommend
learning rates $\beta_{1}=0.9$, $\beta_{2}=0.999$, as well as
$\epsilon=10^{-8}$. Adam has been shown to outperform other optimizers in a
wide range of non-convex optimization problems. Researchers at Google [ARS+20]
recommend the Adam optimization algorithm for SGD optimization in
reinforcement learning.

#####  3.3.4 Backpropagation

Gradient-based optimization requires a method for computing a function’s
gradient. For neural nets, the gradient of the loss function with respect to
the weights of the network $\nabla_{\theta}J(\theta)$ is usually computed
using the backpropagation algorithm (backprop) introduced in 1986 [RHW85].
Backprop calculates the gradient of the loss function with respect to each
weight in the network. This is done by iterating backward through the network
layers and repeatedly applying the chain rule. The chain rule of calculus is
used when calculating derivatives of functions that are compositions of other
functions with known derivatives. Let $y,z:\mathbb{R}\rightarrow\mathbb{R}$ be
functions defined as $y=g(x)$ and $z=f(g(x))=f(y)$. By the chain rule

$\dfrac{dz}{dx}=\dfrac{dz}{dy}\dfrac{dy}{dx}$ (3.13)

Generalizing further, let $x\in\mathbb{R}^{m},y\in\mathbb{R}^{n}$, and define
mappings $g:\mathbb{R}^{m}\rightarrow\mathbb{R}^{n}$ and
$f:\mathbb{R}^{n}\rightarrow\mathbb{R}$. If $y=g(x)$ and $z=f(y)$, then the
chain rule is

$\dfrac{\partial z}{\partial x_{i}}=\sum_{j}\dfrac{\partial z}{\partial
y_{j}}\dfrac{\partial y_{j}}{\partial x_{i}}$ (3.14)

which can be written in vector notation as

$\nabla_{x}z=\left(\dfrac{\partial y}{\partial x}\right)^{\top}\nabla_{y}z$
(3.15)

where $\dfrac{\partial y}{\partial x}$ is the $n\times m$ Jacobian matrix of
$g$. Backpropagation is often performed on tensors and not vectors. However,
backpropagation with tensors is performed similarly by multiplying Jacobians
by gradients. Backpropagation with tensors can be performed by flattening a
tensor into a vector, performing backprop on the vector, and then reshaping
the vector back into a tensor. Let $X$ and $Y$ be tensors and $Y=g(X)$ and
$z=f(Y)$. The chain rule for tensors is

$\nabla_{x}z=\sum_{j}\left(\nabla_{x}Y_{j}\right)\dfrac{\partial z}{\partial
Y_{j}}$ (3.16)

By recursively applying the chain rule, a scalar’s gradient can be expressed
for any node in the network that produced it. This is done recursively,
starting from the output layer and going back through the layers of the
network to avoid storing subexpressions of the gradient or recomputing them
several times.

#####  3.3.5 Activation function

The activation function $\phi(\xi)$ adds nonlinearity to a neural net. If the
activation function in the hidden layer is linear, then the network is
equivalent to a network without hidden layers since linear functions of linear
functions are themselves linear. The activation function must be
differentiable to compute the gradient. Choosing an appropriate activation
function depends on the specific problem. Sigmoid functions $\sigma$, like the
logistic function, are commonly used, as well as other functions such as the
hyperbolic tangent function $tanh$. The derivative of the logistic function is
close to $0$ except in a small neighborhood around $0$. At each backward step,
the $\delta$ is multiplied by the derivative of the activation function. The
gradient will therefore approach $0$ and thus produce extremely slow learning.
This is known as the vanishing gradient problem. For this reason, the
Rectified Linear Unit (ReLU)is the default recommendation for activation
function in modern deep neural nets [GBC16]. ReLU is a ramp function defined
as $ReLU(x)=\max\\{0,x\\}$. The derivative of the ReLU function is defined as

$ReLU^{\prime}(x)={\begin{cases}0&if\ x<0\\\ 1&if\ x>0\\\ \end{cases}}$ (3.17)

The derivative is undefined for $x=0$, but it has subdifferential $[0,1]$, and
it conventionally takes the value $ReLU^{\prime}(0)=0$ in practical
implementations. Since ReLU is a piecewise linear function, it optimizes well
with gradient-based methods.

Figure 3.3: ReLU activation function from [FFLb].

ReLU suffers from what is known as the dying ReLU problem, where a large
gradient could cause a node’s weights to update such that the node will never
output anything but $0$. Such nodes will not discriminate against any input
and are effectively “dead”. This problem can be caused by unfortunate weight
initialization or a too-high learning rate. Generalizations of the ReLU
function, like the Leaky ReLU (LReLU) activation function, has been proposed
to combat the dying ReLU problem [GBC16]. Leaky ReLU allows a small “leak” for
negative values proportional to some slope coefficient $\alpha$, e.g.,
$\alpha=0.01$, determined before training. This allows small gradients to
travel through inactive nodes. Leaky ReLU will slowly converge even on
randomly initialized weights but can also reduce performance in some
applications [GBC16].

Figure 3.4: Leaky ReLU activation function from [FFLb].

#####  3.3.6 Regularization

Minimization of generalization error is a central objective in machine
learning. The representation capacity of large neural networks, expressed by
the universal approximation theorem (3.3.8), comes at the cost of increased
overfitting risk. Consequently, a critical question in ML is how to design and
train neural networks to achieve the lowest generalization error.
Regularization addresses this question. Regularization is a set of techniques
designed to reduce generalization error, possibly at the expense of training
error.

Regularization of estimators trades increased bias for reduced variance. If
effective, it reduces model variance more than it increases bias. Weight decay
is used to regularize ML loss functions by adding the squared $L^{2}$ norm of
the parameter weights $\Omega(\theta)=\frac{1}{2}||\theta||^{2}_{2}$ as a
regularization term to the loss function

$\tilde{J}(\theta)=J(\theta)+\lambda\Omega(\theta)$ (3.18)

where $\lambda\geq 0$ is a constant weight decay parameter. Increasing
$\lambda$ punishes larger weights harsher. Weight decay creates a tradeoff for
the optimization algorithm between minimizing the loss function $J(\theta)$
and the regularization term $\Omega(\theta)$.

Figure 3.5: An example of the effect of weight decay with parameter $\lambda$
on a high-dimensional polynomial regression model from [GBC16].

Dropout [SHK+14] is another regularization strategy that reduces the risk of
overfitting by randomly eliminating non-output nodes and their connections
during training, preventing units from co-adapting too much. Dropout can be
considered an ensemble method, where an ensemble of “thinned” sub-networks
trains the same underlying base network. It is computationally inexpensive and
only requires setting one parameter $\alpha\in[0,1)$, which is the rate at
which nodes are eliminated.

Early stopping is a common and effective implicit regularization technique
that addresses how many epochs a model should be trained to achieve the lowest
generalization error. The training data is split into training and validation
subsets. The model is iteratively trained on the training set, and at
predefined intervals in the training cycle, the model is tested on the
validation set. The error on the validation set is used as a proxy for the
generalization error. If the performance on the validation set improves, a
copy of the model parameters is stored. If performance worsens, the learning
terminates, and the model parameters are reset to the previous point with the
lowest validation set error. Testing too frequently on the validation set
risks premature termination. Temporary dips in performance are prevalent for
nonlinear models, especially when trained with reinforcement learning
algorithms when the agent explores the state and action space. Additionally,
frequent testing is computationally expensive. On the other hand, infrequent
testing increases the risk of not registering the model parameters near their
performance peak. Early stopping is relatively simple but comes at the cost of
sacrificing parts of the training set to the validation set.

#####  3.3.7 Batch normalization

Deep neural networks are sensitive to initial random weights and
hyperparameters. When updating the network, all weights are updated using a
loss estimate under the false assumption that weights in the prior layers are
fixed. In practice, all layers are updated simultaneously. Therefore, the
optimization step is constantly chasing a moving target. The distribution of
inputs during training is forever changing. This is known as internal
covariate shift, making the network sensitive to initial weights and slowing
down training by requiring lower learning rates.

Batch normalization (batch norm) is a method of adaptive reparametrization
used to train deep networks. It was introduced in 2015 [IS15] to help
stabilize and speed up training deep neural networks by reducing internal
covariate shift. Batch norm normalizes the output distribution to be more
uniform across dimensions by standardizing the activations of each input
variable for each mini-batch. Standardization rescales the data to standard
Gaussian, i.e., zero-mean unit variance. The following transformation is
applied to a mini-batch of activations to standardize it

$\hat{x}^{(k)}_{norm}=\dfrac{x^{(k)}-\mathbb{E}[x^{(k)}]}{\sqrt{Var[x^{(k)}]+\epsilon}}$
(3.19)

where the $\epsilon>0$ is a small number such as $10^{-8}$ added to the
denominator for numerical stability. Normalizing the mean and standard
deviation can, however, reduce the expressiveness of the network [GBC16].
Applying a second transformation step to the mini-batch of normalized
activations restores the expressive power of the network

$\tilde{x}^{(k)}=\gamma\hat{x}^{(k)}_{norm}+\beta$ (3.20)

where $\beta$ and $\gamma$ are learned parameters that adjust the mean and
standard deviation, respectively. This new parameterization is easier to learn
with gradient-based methods. Batch normalization is usually inserted after
fully connected or convolutional layers. It is conventionally inserted into
the layer before activation functions but may also be inserted after.
Batchnorm speeds up learning and reduces the strong dependence on initial
parameters. Additionally, it can have a regularizing effect and sometimes
eliminate the need for dropout.

#####  3.3.8 Universal approximation theorem

In 1989, Cybenko [Cyb89] proved that a feedforward network of arbitrary width
with a sigmoidal activation function and a single hidden layer can approximate
any continuous function. The theorem asserts that given any $f\in
C([0,1]^{n})$777Continuous function on the $n$-dimensional unit cube.,
$\epsilon>0$ and sigmoidal activation function $\phi$, there is a finite sum
of the form

$\hat{f}(x)=\sum^{N}_{i=1}\alpha_{i}\phi(\theta_{i}^{\top}x+b_{i})$ (3.21)

where $\alpha_{i},b_{i}\in\mathbb{R}$ and $\theta_{i}\in\mathbb{R}^{2}$, for
which

$|\hat{f}(x)-f(x)|<\epsilon$ (3.22)

for all $x\in[0,1]^{n}$. Hornik [Hor91] later generalized to include all
squashing activation functions in what is known as the universal approximation
theorem. The theorem establishes that there are no theoretical constraints on
the expressivity of neural networks. However, it does not guarantee that the
training algorithm will be able to learn that function, only that it can be
learned for an extensive enough network.

#####  3.3.9 Deep neural networks

Although a single-layer network, in theory, can represent any continuous
function, it might require the network to be infeasibly large. It may be
easier or even required to approximate more complex functions using networks
of deep topology [SB18]. The class of ML algorithms that use neural nets with
multiple hidden layers is known as Deep Learning (DL). Interestingly, the
universal approximation theorem also applies to networks of bounded width and
arbitrary depth. Lu et al. [LPW+17] showed that for any Lebesgue-integrable
function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ and any $\epsilon>0$, there
exists a fully-connected ReLU network $A$ with width $(n+4)$, such that the
function $F_{A}$ represented by this network satisfies

$\int_{\mathbb{R}^{n}}|f(x)-F_{A}(x)|dx<\epsilon$ (3.23)

i.e., any continuous multivariate function
$f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ can be approximated by a deep ReLU
network with width $d_{m}\leq n+4$.

Poggio et al.[PMR+17] showed that a deep network could have exponentially
better approximation properties than a wide shallow network of the same total
size. Conversely, a network of deep topology can attain the same expressivity
as a larger shallow network. They also show that a deep composition of low-
dimensional functions has a theoretical guarantee, which shallow networks do
not have, that they can resist the curse of dimensionality for a large class
of functions.

Several unique network architectures have been developed for tasks like
computer vision, sequential data, and machine translation. As a result, they
can significantly outperform larger and more deeply layered feedforward
networks. The architecture of neural networks carries an inductive bias, i.e.,
an a priori algorithmic preference. A neural network’s inductive bias must
match that of the problem it is solving to generalize well out-of-sample.

#####  3.3.10 Convolutional neural networks

A Convolutional Neural Network (CNN)is a type of neural network specialized in
processing data with a known, grid-like topology such as time-series data
(1-dimensional) or images (2-dimensional) [GBC16]. Convolutional neural
networks have profoundly impacted fields like computer vision [GBC16] and are
used in several successful deep RL applications [MKS+13, HS15, LHP+15]. A CNN
is a neural net that applies convolution instead of general matrix
multiplication in at least one layer. A convolution is a form of integral
transform defined as the integral of the product of two functions after one is
reflected about the y-axis and shifted

$s(t)=(x*w)(t)=\int x(a)w(t-a)da$ (3.24)

where $x(t)\in\mathbb{R}$ and $w(t)$ is a weighting function.

The convolutional layer takes the input $x$ with its preserved spatial
structure. The weights $w$ are given as filters that always extend the full
depth of the input volume but are smaller than the full input size.
Convolutional neural nets utilize weight sharing by applying the same filter
across the whole input. The filter slides across the input and convolves the
filter with the image. It computes the dot product at every spatial location,
which makes up the activation map, i.e., the output. This can be done using
different filters to produce multiple activation maps. The way the filter
slides across the input can be modified. The stride specifies how many pixels
the filter moves every step. It is common to zero pad the border if the stride
is not compatible with the size of the filter and the input.

Figure 3.6: 3D convolutional layer from [FFLc].

After the convolutional layer, a nonlinear activation function is applied to
the activation map. Convolutional networks may also include pooling layers
after the activation function that reduce the dimension of the data. Pooling
can summarize the feature maps to the subsequent layers by discarding
redundant or irrelevant information. Max pooling is a pooling operation that
reports the maximum output within a patch of the feature map. Increasing the
stride of the convolutional kernel also gives a downsampling effect similar to
pooling.

#####  3.3.11 Recurrent neural networks

A Recurrent Neural Network (RNN)is a type of neural network that allows
connections between nodes to create cycles so that outputs from one node
affect inputs to another. The recurrent structure enables networks to exhibit
temporal dynamic behavior. RNNs scale far better than feedforward networks for
longer sequences and are well-suited to processing sequential data. However,
they can be cumbersome to train as their recurrent structure precludes
parallelization. Furthermore, conventional batch norm is incompatible with
RNNs, as the recurrent part of the network is not considered when computing
the normalization statistic.

Figure 3.7: Recurrent neural network from [FFLa].

RNNs generate a sequence of hidden states $h_{t}$. The hidden states enable
weight sharing that allows the model to generalize over examples of various
lengths. Recurrent neural networks are functions of the previous hidden state
$h_{t-1}$ and the input $x_{t}$ at time $t$. The hidden units in a recurrent
neural network are often defined as a dynamic system $h^{(t)}$ driven by an
external signal $x^{(t)}$

$h^{(t)}=f(h^{(t-1)},x^{(t)};\theta)$ (3.25)

Hidden states $h_{t}$ are utilized by RNNs to summarize problem-relevant
aspects of the past sequence of inputs up to $t$ when forecasting future
states based on previous states. Since the hidden state is a fixed-length
vector, it will be a lossly summary. The forward pass is sequential and cannot
be parallelized. Backprop uses the states computed in the forward pass to
calculate the gradient. The backprop algorithm used on unrolled RNNs is called
backpropagation through time (BPTT). All nodes that contribute to an error
should be adjusted. In addition, for an unrolled RNN, nodes far back in the
calculations should also be adjusted. Truncated backpropagation through time
that only backpropagates for a few backward steps can be used to save
computational resources at the cost of introducing bias.

Every time the gradient backpropagates through a vanilla RNN cell, it is
multiplied by the transpose of the weights. A sequence of vanilla RNN cells
will therefore multiply the gradient with the same factor multiple times. If
$x>1$ then $\lim_{n\rightarrow\infty}x^{n}=\infty$, and if $x<1$ then
$\lim_{n\rightarrow\infty}x^{n}=0$. If the largest singular value of the
weight matrix is $>1$, the gradient will exponentially increase as it
backpropagates through the RNN cells. Conversely, if the largest singular
value is $<1$, the opposite happens, where the gradient will shrink
exponentially. For the gradient of RNNs, this will result in either exploding
or vanishing gradients. This is why vanilla RNNs trained with gradient-based
methods do not perform well, especially when dealing with long-term
dependencies. Bengio et al. [BSF94] present theoretical and experimental
evidence supporting this conclusion. Exploding gradients lead to large updates
that can have a detrimental effect on model performance. The standard solution
is to clip the parameter gradients above a certain threshold. Gradient
clipping can be done element-wise or by the norm over all parameter gradients.
Clipping the gradient norm has an intuitive appeal over elementwise clipping.
Since all gradients are normalized jointly with the same scaling factor, the
gradient still points in the same direction, which is not necessarily the case
for element-wise gradient clipping [GBC16]. Let $\lVert\mathbf{g}\rVert$ be
the norm of the gradient $\mathbf{g}$ and $v>0$ be the norm threshold. If the
norm crosses over the threshold $\lVert\mathbf{g}\rVert>v$, the gradient is
clipped to

$\mathbf{g}\leftarrow\frac{\mathbf{g}^{v}}{\lVert\mathbf{g}\rVert}$ (3.26)

Gradient clipping solves the exploding gradient problem and can improve
performance for reinforcement learning with nonlinear function approximation
[ARS+20]. For vanishing gradients, however, the whole architecture of the
recurrent network needs to be changed. This is currently a hot topic of
research [GBC16].

###### Long short-term memory

Long Short-Term Memory (LSTM)is a form of gated RNN designed to have better
gradient flow properties to solve the problem of vanishing and exploding
gradients. LSTMs were introduced in 1997 [HS97] and are traditionally used in
natural language processing [GBC16]. Recently, LSMT networks have been
successfully applied to financial time series forecasting [SNN18]. Although
new architectures like transformers have impressive natural language
processing and computer vision performance, LSTMs are still considered state-
of-the-art for time series forecasting.

Figure 3.8: LSTM cell from [FFLa].

The LSTM is parameterized by the weight $W\in\mathbb{R}^{n}$, which is
optimized using gradient-based methods. While vanilla RNNs only have one
hidden state, LSTMs maintain two hidden states at every time step. One is
$h_{t}$, similar to the hidden state of vanilla RNNs, and the second is
$c_{t}$, the cell state that gets kept inside the network. The cell state runs
through the LSTM cell with only minor linear interactions. LSTMs are composed
of a cell and four gates which regulate the flow of information to and from
the cell state and hidden state

  * •

Input gate $i$; decides which values in the cell state to update

  * •

Forget gate $f$; decides what to erase from the cell state

  * •

Output gate $o$; decides how much to output to the hidden state

  * •

Gate gate $g$; how much to write to cell, decides how much to write to the
cell state

The output from the gates is defined as

$\begin{vmatrix}i\\\ f\\\ o\\\ g\\\ \end{vmatrix}=\begin{vmatrix}\sigma\\\
\sigma\\\ \sigma\\\ \tanh\\\ \end{vmatrix}W\begin{vmatrix}h_{t-1}\\\ x_{t}\\\
\end{vmatrix}$ (3.27)

where $\sigma$ is the sigmoid activation function. The cell state $c_{t}$ and
hidden state $h_{t}$ are updated according to the following rules

$c_{t}=f\odot c_{t-1}+i\odot g$ (3.28) $h_{t}=o\odot\tanh{(c_{t})}$ (3.29)

When the gradient flows backward in the LSTM, it backpropagates from $c_{t}$
to $c_{t-1}$, and there is only elementwise multiplication by the $f$ gate and
no multiplication with the weights. Since the LSTMs backpropagate from the
last hidden state through the cell states backward, it is only exposed to one
$tanh$ nonlinear activation function. Otherwise, the gradient is relatively
unimpeded. Therefore, LSTMs handle long-term dependencies without the problem
of exploding or vanishing gradients.

###  4 Reinforcement learning

An algorithmic trading agent maps observations of some predictor data to
market positions. This mapping is non-trivial, and as noted by Moody et al.
[MWLS98], accounting for factors such as risk and transaction costs is
difficult in a supervised learning setting. Fortunately, reinforcement
learning provides a convenient framework for optimizing risk- and transaction-
cost-sensitive algorithmic trading agents.

The purpose of this chapter is to introduce the fundamental concepts of
reinforcement learning relevant to this thesis. A more general and
comprehensive introduction to reinforcement learning can be found in
“Reinforcement Learning: An Introduction” by Richard Sutton and Andrew Barto
[SB18]. An overview of deep reinforcement learning may be found in “Deep
Reinforcement Learning” by Aske Plaat [Pla22]. This chapter begins by
introducing reinforcement learning 4.1 and the Markov decision process
framework 4.2, and some foundational reinforcement learning concepts 4.3, 4.4.
Section 4.5 discusses how the concepts introduced in the previous chapter (3)
can be combined with reinforcement learning to generalize over high-
dimensional state spaces. Finally, section 4.6 introduces policy gradient
methods, which allow an agent to optimize a parameterized policy directly.

####  4.1 Introduction

Reinforcement Learning (RL)is the machine learning paradigm that studies how
an intelligent agent can learn to make optimal sequential decisions in a time
series environment under stochastic or delayed feedback. It is based on the
concept of learning optimal behavior to solve complex problems by training in
an environment that incorporates the structure of the problem. The agent
optimizes a policy that maps states to actions through reinforcement signals
from the environment in the form of numerical rewards. The goal of using RL to
adjust the parameters of an agent is to maximize the expected reward generated
due to the agent’s actions. This goal is accomplished through trial and error
exploration of the environment. A key challenge of RL is balancing exploring
uncharted territory and exploiting current knowledge, known as the
exploration-exploitation tradeoff. Although it has been studied for many
years, the exploration-exploitation tradeoff remains unsolved. Each action
must be tried multiple times in stochastic environments to get a reliable
estimate of its expected reward. For environments with non-stationary
dynamics, the agent must continuously explore to learn how the distribution
changes over time. The agent-environment interaction in RL is often modeled as
a Markov decision process.

####  4.2 Markov decision process

A Markov Decision Process (MDP)is a stochastic control process and a classical
formalization of sequential decision-making. A MPD is a tuple
$(\mathcal{S},\mathcal{A},p,\mathcal{R},\gamma)$, where

  * •

$\mathcal{S}$ is a countable non-empty set of states (state space).

  * •

$\mathcal{A}$ is a countable non-empty set of actions (action space)

  * •

$p(s^{\prime}|s,a)=Pr(s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a)$ is the transition
probability matrix.

  * •

$\mathcal{R}\subset\mathbb{R}$ is the set of all possible rewards.

  * •

$\gamma\in[0,1]$ is the discount rate.

The agent interacts with the environment at discrete time steps
$t=0,1,2,3,...$, which are not necessarily fixed intervals of real-time. At
each step $t$, the agent receives a representation of the state of the
environment $s_{t}\in\mathcal{S}$, where $s_{0}\in\mathcal{S}$ is the initial
state drawn from some initial state distribution
$p_{0}\in\Delta(\mathcal{S})$. Based on the state $s_{t}=s$, the agent chooses
one of the available actions in the current state $a_{t}\in A(s)$. After
performing the action $a_{t}$, the agent receives an immediate numerical
reward $r_{t+1}\in\mathcal{R}$ and the subsequent state representation
$s_{t+1}\in\mathcal{S}$. This interaction with a Markov decision process
produces a sequence known as a trajectory:
$s_{0},a_{0},r_{1},s_{1},a_{1},r_{2},s_{2},a_{2},r_{3},...$. This sequence is
finite for episodic tasks (with the termination time usually labeled $T$); for
continuing tasks, it is infinite.

Figure 4.1: Agent-environment interaction from [SB18].

The dynamics of the system can be completely described by the one-step
transition function
$p:\mathcal{S}\times\mathcal{R}\times\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$
that is defined as

$p(s^{\prime},r|s,a)=Pr\\{s_{t}=s^{\prime},r_{t}=r|s_{t-1}=s,a_{t-1}=a\\}$
(4.1)

for all $s,s^{\prime}\in\mathcal{S}$, $r\in\mathcal{R}$, and
$a\in\mathcal{A}(s)$. It defines a probability distribution such that

$\sum_{s\in\mathcal{S}}\sum_{a\in\mathcal{A}(s)}p(s^{\prime},r|s,a)=1$ (4.2)

for all $s\in\mathcal{S}$, and $a\in\mathcal{A}(s)$. Note that the one-step
transition function depends only on the current state $s$ and not previous
states, i.e., the state has the Markov property. Essentially, MDPs are Markov
chains with actions and rewards. The transition probabilities
$p:\mathcal{S}\times\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$ are defined
through the dynamics function $p(s^{\prime},r|s,a)$, as

$p(s^{\prime}|s,a)=Pr(s_{t}=s^{\prime}|s_{t-1}=s,a_{t-1}=a)=\sum_{r\in\mathcal{R}}p(s^{\prime},r|s,a)$
(4.3)

The reward generating function
$r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, is defined through the
dynamics function $p(s^{\prime},r|s,a)$, as

$r(s,a)=\mathbb{E}[r_{t+1}|s_{t}=s,a_{t}=a]=\sum_{r\in\mathcal{R}}r\sum_{s^{\prime}\in\mathcal{S}}p(s^{\prime},r|s,a)$
(4.4)

The reward-generating function determines the expected reward from performing
an action $a$ in a state $s$. In practice, the dynamics of the system
$p(s^{\prime},r|s,a)$ are seldom known a priori but learned through
interacting with the environment.

#####  4.2.1 Infinite Markov decision process

A finite MDP is a Markov decision process with countably finite state space
$|\mathcal{S}|<\infty$, action space $|\mathcal{A}|<\infty$. Finite MDPs can
be described as tabular and solved by dynamic programming algorithms with
convergence guarantees if the state and action space dimensions
$|\mathcal{S}\times\mathcal{A}|$ are not too large. Unfortunately, the
applicability of these methods is severely limited by the assumption that
state-action spaces are countable. These assumptions must be relaxed for MDPs
to have significant real-world applications. Fortunately, the same theory for
finite MDPs also applies to continuous and countably infinite state-action
spaces under function approximation. The system dynamics are then described by
a transition probability function $\mathcal{P}$ instead of a matrix.

#####  4.2.2 Partially observable Markov decision process

Let $s_{t}$ be the environment state and $s^{a}_{t}$ be the agent state. A
Markov decision process assumes full observability of the environment, i.e.,
that $s_{t}=s^{a}_{t}$. A Partially Observable Markov Decision Process
(POMDP)relaxes this assumption and allows for optimal decision-making in
environments that are only partially observable to the agent. Since they are
generalizations of MDPs, all algorithms used for MDPs are compatible with
POMDPs. A POMDP is a tuple
$(\mathcal{S},\mathcal{A},\mathcal{O},\mathcal{P},\mathcal{R},\mathcal{Z},\gamma)$
that extends a MDP with two additional elements

  * •

$\mathcal{O}$ is the observation space.

  * •

$\mathcal{Z}$ is the observation function,
$\mathcal{Z}=Pr(o_{t+1}=o|s_{t+1}=s^{\prime},a_{t}=a)$.

An agent in a POMDP cannot directly observe the state; instead, it receives an
observation $o\in\mathcal{O}$ determined by the observation function
$\mathcal{Z}$. The agent state approximates the environment state
$s^{a}_{t}\approx s_{t}$. However, a single observation $o$ is not a Markovian
state signal. Direct mapping between observation and action is insufficient
for optimal behavior, and a memory of past observations is required. The
history of a POMDP is a sequence of actions and observations
$h_{t}=\\{o_{1},a_{1},...,o_{t},a_{t}\\}$. The agent state can be defined as
the history $s^{a}_{t}=h_{t}$. However, storing and processing the complete
history of every action scales linearly with time, both in memory and
computation. A more scalable alternative is a stateful sequential model like a
recurrent neural network (RNN). In this model, the agent state is represented
by the network $s^{a}_{t}=f_{\theta}(s^{a}_{t-1},o_{t})$.

A state can be split into an agent’s internal state and the environment’s
external state. Anything that cannot be changed arbitrarily by the agent is
considered external and, thus, part of the external environment. On the other
hand, the internal data structures of the agent that the agent can change are
part of the internal environment.

####  4.3 Rewards

The goal of a reinforcement learning agent is to maximize the expected return
$\mathbb{E}[G_{t}]$, where the return $G_{t}$ is defined as the sum of rewards

$G_{t}=r_{t+1}+r_{t+2}+...+r_{T}$ (4.5)

In an episodic setting, where $t=0,...,T$, this goal is trivial to define as
the sequence of rewards is finite. However, some problems, like financial
trading, do not naturally break into subsequences and are known as continuing
problems. For continuing problems, where $T=\infty$ and there is no terminal
state, it is clear that the sum of rewards $G_{t}$ could diverge. Discounting
was introduced to solve the problem of returns growing to infinity. Discounted
returns are defined as

$G_{t}=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k+1}=r_{t+1}+\gamma G_{t+1}$ (4.6)

where $\gamma\in[0,1]$ is the discount rate used to scale future rewards.
Setting $\gamma=0$ suggests that the agent is myopic, i.e., only cares about
immediate rewards. As long as $\gamma<1$ and the reward sequence is bounded,
the discounted return $G_{t}$ is finite. Discounting allows reinforcement
learning to be used in continuing problems.

Reinforcement signals $r_{t+1}$ from the environment can be immediate or
delayed. Games and robot control are typical examples of delayed reward
environments, where an action affects not only the immediate reward but also
the next state and, through that, all subsequent rewards. An example of
delayed reward is when chess players occasionally sacrifice a piece to gain a
positional advantage later in the game. Although sacrificing a piece in
isolation is poor, it can still be optimal long-term. Consequently, temporal
credit assignment is a fundamental challenge in delayed reward environments.
AlphaZero [SHS+17] surpassed human-level play in chess in just 24 hours,
starting from random play, using reinforcement learning. Interestingly,
AlphaZero seems unusually (by human standards) open to material sacrifices for
long-term positional compensation, suggesting that the RL algorithm estimates
delayed reward better than human players. Throughout this thesis, financial
trading is modeled as a stochastic immediate reward environment. This choice
is justified in chapter 5. Therefore, the problem reduces to an associative
reinforcement learning problem, a specific instance of the full reinforcement
learning problem. It requires generalization and trial-and-error exploration
but not temporal credit assignment. The methods presented in this chapter will
only be those relevant in an immediate reward environment. Unless otherwise
stated, the discount rate $\gamma$, a tradeoff between immediate and delayed
rewards, is assumed to be zero, making the agent myopic. As a result, the
return $G_{t}$ in an immediate reward environment is defined as the immediate
reward

$G_{t}=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k+1}=r_{t+1}$ (4.7)

####  4.4 Value function and policy

A stochastic policy is a mapping from states to a probability distribution
over the action space and is defined as

$\pi:\mathcal{S}\rightarrow\Delta(\mathcal{A})$ (4.8)

The stochastic policy is a probability measure
$\pi(a|s)=Pr\\{a_{t}=a|s_{t}=s\\}$, which is the probability that the agent
performs an action $a$, given that the current state is $s$. Stochastic
policies can be advantageous in problems with perceptual aliasing.
Furthermore, it handles the exploration-exploitation tradeoff without hard
coding it. A deterministic policy maps states $\mathcal{S}$ to actions
$\mathcal{A}$, and is defined as

$\mu:\mathcal{S}\rightarrow\mathcal{A}$ (4.9)

RL algorithms determine how policies are adjusted through experience, where
the goal is for the agent to learn an optimal or near-optimal policy that
maximizes returns.

Value-based RL algorithms, like Q-learning and SARSA, estimate a state-value
function or an action-value function and extract the policy from it. The
state-value function $V_{\pi}(s)$ is the expected return when starting in
state $s$ and following policy $\pi$. It is defined $\forall s\in\mathcal{S}$
as

$V_{\pi}(s)=\mathbb{E}_{\pi}[G_{t}|s_{t}=s]$ (4.10)

The action-value function $Q_{\pi}(s,a)$ is the expected return when
performing action $a$ in state $s$ and then following the policy $\pi$. It is
defined $\forall s\in\mathcal{S},a\in\mathcal{A}(s)$ as

$Q_{\pi}(s,a)=\mathbb{E}_{\pi}[G_{t}|s_{t}=s,a_{t}=a]$ (4.11)

An example of a value-based policy is the $\epsilon$-greedy policy, defined as

$\pi(a|s,\epsilon)=\begin{cases}\operatorname*{arg\,max}_{a}{Q_{\pi}(s,a)}&\text{with
probability $1-\epsilon$}\\\ \text{sample random action
}a\sim\mathcal{A}(s)&\text{with probability $\epsilon$}\end{cases}$ (4.12)

where $\epsilon\in[0,1]$ is the exploration rate.

Reinforcement learning algorithms are divided into on-policy and off-policy
algorithms. The same policy that generates the trajectories is being optimized
for on-policy algorithms. In contrast, for off-policy algorithms, the policy
generating trajectories differs from the one being optimized. For off-policy
learning, the exploration can be delegated to an explorative behavioral policy
while the agent optimizes a greedy target policy.

####  4.5 Function approximation

Tabular reinforcement learning methods include model-based methods like
dynamic programming as well as model-free methods like Monte-Carlo and
temporal-difference learning methods (e.g., Q-learning and SARSA).
Unfortunately, tabular methods require discrete state and action spaces, and
due to the curse of dimensionality, these spaces must be relatively small.
Thus, their applicability to real-world problems is limited. Complex
environments like financial trading cannot be represented in discrete states.
Instead, feature vectors represent states in environments where the state
space is too large to enumerate. As most states will probably never be visited
and visiting the same states twice is unlikely, it is necessary to generalize
from previous encounters to states with similar characteristics. This is where
the concepts of function approximation discussed in the previous chapter
(3.2.1) come in. Using function approximation, samples from the desired
function, e.g., the value function, are generalized to approximate the entire
function.

Value-based reinforcement learning algorithms such as Deep Q-Network
(DQN)[MKS+13] and Deep Recurrent Q-Network (DRQN)[HS15] use deep neural
networks as function approximators to generalize over a continuous space that
is optimized using the Q-learning algorithm. DQN and DRQN achieved superhuman
performance in playing Atari games using raw pixels as input into a
convolutional neural network that outputs action-value estimates of future
returns. These value-based algorithms are still limited by discrete action
space and the curse of dimensionality as it has to calculate the Q-value of
every single action. In the games DQN and DRQN are tested on, the agent is
limited to a small discrete set of actions (between 4 and 18). However, for
many applications, a discrete action space is severely limiting. Furthermore,
these algorithms use the naive exploration heuristics $\epsilon$-greedy, which
is not feasible in critical domains. Fortunately, policy gradient methods
bypass these problems entirely.

####  4.6 Policy gradient methods

While value-based reinforcement learning algorithms extract a policy from
action-value estimates, Policy Gradient (PG)methods learn a parameterized
policy and optimize it directly. The policy’s parameter vector is
$\theta\in\mathbb{R}^{d^{\prime}}$, with the policy defined as

$\pi_{\theta}(a|s)=Pr\\{a_{t}=a|s_{t}=s,\theta_{t}=\theta\\}$ (4.13)

Continuous action space is modeled by learning the statistics of the
probability of the action space. A natural policy parameterization in
continuous action spaces is the Gaussian distribution
$a\sim\mathcal{N}(\mu_{\theta}(s),\sigma_{\theta}(s)^{2})$ defined as

$\pi_{\theta}(a|s)=\dfrac{1}{\sigma_{\theta}(s)\sqrt{2\pi}}e^{-\dfrac{(a-\mu_{\theta}(s))^{2}}{2\sigma_{\theta}(s)^{2}}}$
(4.14)

where $\mu_{\theta}(s)\in\mathbb{R}$ and $\sigma_{\theta}(s)\in\mathbb{R_{+}}$
are parametric function approximations of the mean and standard deviation,
respectively. The mean decides the space where the agent will favor actions,
while the standard deviation decides the degree of exploration. It is
important to note that this gives a probability density, not a probability
distribution like the softmax distribution.

For policy gradient methods in the continuous time setting, the goal of
optimizing the policy $\pi_{\theta}$ is to find the parameters $\theta$ that
maximize the average rate of return per time step [SB18]. The performance
measure $J$ for the policy $\pi_{\theta}$ in the continuing setting is defined
in terms of the average rate of reward per time step as

$\displaystyle J(\pi_{\theta})$
$\displaystyle=\int_{\mathcal{S}}d^{\pi}(s)\int_{\mathcal{A}}r(s,a)\pi_{\theta}(a|s)dads$
$\displaystyle=\mathbb{E}_{s\sim d^{\pi},a\sim\pi_{\theta}}[r(s,a)]$ (4.15)

where $d^{\pi}(s)=\lim_{t
\rightarrow\infty}Pr\\{s_{t}=s|a_{0:t}\sim\pi_{\theta}\\}$ is the steady-state
distribution under the policy $\pi_{\theta}$.

Policy optimization aims to find the parameters $\theta$ that maximize the
performance measure $J$. Gradient ascent is used as the optimization algorithm
for the policy. The policy parameter $\theta$ is moved in the direction
suggested by the gradient of $J$ to maximize the return, yielding the
following gradient ascent update

$\theta_{t+1}=\theta_{t}+\alpha\widehat{\nabla_{\theta}J(\pi_{\theta_{t}})}$
(4.16)

where $\alpha$ is the step-size and
$\widehat{\nabla_{\theta}J(\pi_{\theta_{t}})}$ is a stochastic estimate whose
expectation approximates the gradient of $J$ with respect to $\theta$ [SB18].

The policy gradient theorem888For the full proof see chapter 13.6 in [SB18]
for the continuing case provides the following expression for the gradient

$\displaystyle\nabla_{\theta}J(\pi_{\theta})$
$\displaystyle=\int_{\mathcal{S}}d^{\pi}(s)\int_{\mathcal{A}}Q_{\pi}(s,a)\nabla_{\theta}\pi_{\theta}(a|s)dads$
$\displaystyle=\mathbb{E}_{s\sim
d^{\pi},a\sim\pi_{\theta}}[Q_{\pi}(s,a)\nabla_{\theta}\log{\pi_{\theta}(a|s)}]$
(4.17)

Even though the steady-state distribution $d^{\pi}$ depends on the policy
parameters $\theta$, the gradient of the performance measure does not involve
the gradient of $d^{\pi}$, allowing the agent to simulate paths and update the
policy parameter at every step [SB18].

#####  4.6.1 REINFORCE

REINFORCE is an on-policy direct policy optimization algorithm derived using
the policy gradient theorem [SB18]. The algorithm is on-policy. Consequently,
the agent will encounter the states in the proportions specified by the
steady-state distribution. Using the policy gradient theorem, the calculation
of the policy gradient reduces to a simple expectation. The only problem is
estimating the action-value function $Q_{\pi}(s,a)$. REINFORCE solves this
problem by using the sampled return $G_{t}$ as an unbiased estimate of the
action-value function $Q_{\pi}(s_{t},a_{t})$. Observing that the state-value
is equal to the expectation of the sampled return, i.e.,
$\mathds{E}_{\pi}[G_{t}|s_{t},a_{t}]=Q_{\pi}(s_{t},a_{t})$, the following
expression for the policy gradient can be defined

$\displaystyle\nabla_{\theta}J(\pi_{\theta})$ $\displaystyle=\mathbb{E}_{s\sim
d^{\pi},a\sim\pi_{\theta}}[Q_{\pi}(s,a)\nabla_{\theta}\log{\pi_{\theta}(a|s)}]$
$\displaystyle=\mathbb{E}_{s\sim
d^{\pi},a\sim\pi_{\theta}}[G_{t}\nabla_{\theta}\log{\pi_{\theta}(a|s)}]$
(4.18)

This expression can be sampled on each time step t, and its expectation equals
the gradient. The gradient ascent policy parameter update for REINFORCE is
defined as

$\theta_{t+1}=\theta_{t}+\alpha
G_{t}\nabla_{\theta}\log{\pi_{\theta_{t}}(a_{t}|s_{t})}$ (4.19)

where $\alpha$ is the step size. The direction of the gradient is in the
parameter space that increases the probability of repeating action $a_{t}$ on
visits to $s_{t}$ in the future the most [SB18]. The higher the return, the
more the agent wants to repeat that action. The update is inversely
proportional to the action probability to adjust for different frequencies of
visits to states, i.e., some states might be visited often and have an
advantage over less visited states.

While REINFORCE is unbiased and only requires estimating the policy, it might
exhibit high variance due to the high variability of sampled returns (if the
trajectory space is large). High variance leads to unstable learning updates
and slower convergence. Furthermore, the stochastic policy used to estimate
the gradient can be disadvantageous in critical domains such as health care or
finance. Thankfully, both these problems can be solved by a class of policy
gradient methods called actor-critic methods.

#####  4.6.2 Actor-critic

Policy-based reinforcement learning is effective in high-dimensional and
continuous action space, while value-based RL is more sample efficient and
more convenient for online learning. Actor-Critic (AC)methods seek to combine
the best of both worlds where a policy-based actor chooses actions, and the
value-based critic critique those actions. The actor optimizes the policy
parameters using stochastic gradient ascent in the direction suggested by the
critic. The critic’s value function is optimized using stochastic gradient
descent to minimize the loss to the target. This use of a critic introduces
bias since the critique is an approximation of the return and not actual
observed returns like in actor-based algorithms like REINFORCE. There are
numerous actor-critic algorithms like advantage actor-critic (A2C) [SB18],
asynchronous advantage actor-critic (A3C) [MBM+16], and proximal policy
optimization (PPO) [SWD+17], that have exhibited impressive performance in a
variety of applications. These methods rely on stochastic policies and
computing the advantage function. For critical domains such as finance, a
deterministic policy directly optimized by a learned action-value function
might be more appropriate. Fortunately, the policy gradient framework can be
extended to deterministic policies [SLH+14, LHP+15].

The idea behind deterministic actor-critic algorithms is based on Q-learning,
where a network $Q(s,a)$ approximates the return. Q-learning can be extended
to high-dimensional state spaces by defining the Q-network as a function
approximator
$Q_{\phi}(s,a):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$,
parameterized by $\phi\in\mathbb{R}^{b^{\prime}}$. If the Q-network is optimal
($Q_{\phi}^{*}$), finding the optimal action ($a^{*}$) in a small discrete
action space is trivial;
$a^{*}(s)=\operatorname*{arg\,max}_{a}{Q_{\phi}^{*}(s,a)}$. However, the
exhaustive computations required for this process are not feasible in high-
dimensional or continuous action spaces due to the curse of dimensionality.
This problem can be bypassed by learning a deterministic policy
$\mu_{\theta}(s):\mathcal{S}\rightarrow\mathcal{A}$, parameterized by
$\theta\in\mathbb{R}^{d^{\prime}}$, as an approximator to $a(s)$, such that
$\max_{a}{Q_{\phi}(s,a)}\approx Q_{\phi}(s,\mu(s))$.

###### Deterministic policy gradient

Let $\mu_{\theta}:\mathcal{S}\rightarrow\mathcal{A}$ be the deterministic
policy parameterized by $\theta\in\mathbb{R}^{d^{\prime}}$. The performance
measure $J$ for the deterministic policy $\mu_{\theta}$ in the continuous time
average reward setting is defined as

$\displaystyle J(\mu_{\theta})$
$\displaystyle=\int_{\mathcal{S}}d^{\mu}(s)r(s,\mu_{\theta}(s))ds$
$\displaystyle=\mathbb{E}_{s\sim d^{\mu}}[r(s,\mu_{\theta}(s))]$ (4.20)

Initially, there was a belief that the deterministic policy gradient did not
exist; however, it was proven by Silver et al. [SLH+14], which provides the
following expression for the gradient

$\displaystyle\nabla_{\theta}J(\mu_{\theta})$
$\displaystyle=\int_{\mathcal{S}}d^{\mu}(s)\nabla_{\theta}\mu_{\theta}(s)\nabla_{a}Q^{\mu}(s,a)|_{a=\mu_{\theta}(s)}ds$
$\displaystyle=\mathbb{E}_{s\sim
d^{\mu}}[\nabla_{\theta}\mu_{\theta}(s)\nabla_{a}Q^{\mu}(s,a)|_{a=\mu_{\theta}(s)}]$
(4.21)

The deterministic policy gradient theorem holds for both on-policy and off-
policy methods. Deterministic policies only require integrating over the state
space and not both the state and action space like stochastic policies. The
true action-value can be approximated by a parameterized critic, i.e.,
$Q_{\phi}(s,a)\approx Q^{\mu}(s,a)$.

###### Off-policy learning

Learning a deterministic policy in continuous action spaces on-policy will
generally not ensure sufficient exploration and can lead to sub-optimal
solutions. To solve this problem, the deterministic actor-critic algorithm
learns off-policy by introducing an exploration policy $\mu^{\prime}_{\theta}$
defined as

$\mu^{\prime}_{\theta}(s)=\mu_{\theta}(s)+\mathcal{W}$ (4.22)

where $\mathcal{W}$ is sampled noise from a noise-generating function. The
exploration policy $\mu^{\prime}_{\theta}$ explores the environment and
generates trajectories that optimize the target policy $\mu_{\theta}$ and
Q-network $Q_{\phi}$.

###### Q-network optimization

Let $Q_{\phi}(s,a):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ be the
Q-network parameterized by $\phi\in\mathbb{R}^{b^{\prime}}$. The Q-network is
iteratively updated to fit a target defined by the recursive relationship
$y=r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime})$ known as the Bellman
equation [SB18]. The Bellman equation reduces to the immediate reward in an
immediate reward environment, where $\gamma=0$. The goal is to find the
weights $\phi$ that minimize the loss (usually MSE) to the target

$L(Q_{\phi})=\mathbb{E}_{s\sim d^{\mu^{\prime}},a\sim\mu^{\prime},r\sim
E}[(Q_{\phi}(s,a)-y)^{2}]$ (4.23)

where $d^{\mu^{\prime}}$ is the steady-state distribution under the
exploration policy $\mu^{\prime}_{\theta}$, and $E$ is the environment. The
gradient of the loss function with respect to the Q-network parameter weights
$\phi$ is defined as

$\nabla_{\phi}L(Q_{\phi})=\mathbb{E}_{s\sim
d^{\mu^{\prime}},a\sim\mu^{\prime},r\sim
E}[(Q_{\phi}(s,a)-y)\nabla_{\phi}Q_{\phi}(s,a)]$ (4.24)

and is used to calculate the backward pass in the Q-network’s stochastic
gradient descent optimization algorithm.

###### Replay memory

Learning policies and Q-networks with large nonlinear function approximators
is generally considered difficult and unstable and do not come with
convergence guarantees. Another challenge of combining deep neural networks
with reinforcement learning is that most ML optimization algorithms assume
that samples are independent and identically distributed (IID). The IID
assumption is rarely valid for RL agents sequentially exploring the state
space. Furthermore, minibatch learning is advantageous as it efficiently
utilizes hardware optimization. The introduction of replay memory [MKS+13]
addresses these problems and trains large nonlinear function approximators
stably and robustly. A replay memory
$\mathcal{D}=\\{\tau_{t-k+1},\tau_{t-k+2},...,\tau_{t}\\}$ is a finite cache
storing the past $k$ transitions $\tau_{t}=(s_{t},a_{t},r_{t})$. A minibatch
$\mathcal{B}\subseteq\mathcal{D}$ of $|\mathcal{B}|>0$ transitions are
randomly sampled from the replay memory and used to update both the policy and
Q-network.

Randomly sampled batches are ineffective for training recurrent neural
networks, which carry forward hidden states through the mini-batch. Deep
Recurrent Q-Network (DRQN) [HS15] is an extension of DQN for recurrent neural
networks. DRQN uses experience replay like DQN; however, the sampled batches
are in sequential order. The randomly sampled batch
$\mathcal{B}\subseteq\mathcal{D}$ consists of the transitions
$\mathcal{B}=\\{\tau_{i},\tau_{i+1},...,\tau_{i+|\mathcal{B}|-2},\tau_{i+|\mathcal{B}|-1}\\}$,
where $i$ is some random starting point for the batch. The RNNs initial hidden
state is zeroed at the start of the mini-batch update but then carries forward
through the mini-batch.

##  Part II Methodology

###  5 Problem Setting

In reinforcement learning, the agent learns through interaction with the
environment. Thus, developing a model of the environment, in this case, the
commodity market, is necessary to optimize an algorithmic trading agent
through reinforcement signals. Commodities trading involves sequential
decision-making in a stochastic and nonstationary environment to achieve some
objective outlined by the stakeholder. This chapter describes a discrete-time
Markov decision process that models this environment 999Although this thesis
focuses on commodities, the model’s general concepts apply to other financial
markets.. Neither the strong assumption of countable state-actions nor the
assumption of full environment observability can be satisfied. Thus, based on
the previously proposed financial markets dynamical system [ZZR20, Hua18,
LXZ+18], this chapter presents an infinite partially observable MDP for
commodities trading.

####  5.1 Assumptions

Since the model will be tested ex-post by backtesting, described in section
2.9, it is necessary to make a couple of simplifying assumptions about the
markets the agent operates in:

  1. 1.

No slippage, i.e., there is sufficient liquidity in the market to fill any
orders placed by the agent, regardless of size, at the quoted price. In other
words, someone is always willing to take the opposite side of the agent’s
trade. This assumption relates to external factors that may affect the price
between the time the agent is quoted the price and the time the order is
filled101010In reality, prices may significantly change between receiving a
quote and placing an order..

  2. 2.

No market impact, i.e., the money invested by the agent is not significant
enough to move the market. This assumption relates to the agent’s own trades’
impact on the market. The reasonability of this assumption depends on the
depth of the market.

####  5.2 Time Discretization

Financial trading involves continuously reallocating capital in one or more
financial assets. The problem does not naturally break into sub-sequences with
terminal states. Therefore, this MDP is in the continuous-time setting. A
discretization operation is applied to the continuous timeline to study the
reinforcement learning-based algorithmic trading described in this thesis,
discretizing the timeline into steps $t=0,1,2,...$. As described in section
2.8, sampling at fixed time intervals is unsatisfactory in markets where
activity varies throughout the day and exhibits undesirable statistical
properties like non-normality of returns and heteroskedasticity. Instead of
the traditional time-based constant duration $\Delta t$, the observations are
sampled as a function of dollar volume based on the ideas from Mandelbrot and
Taylor [Man97, MT67], and Clark [Cla73] presented in section 2.8. Dollar
volume-based sampling provides better statistical properties for the agent and
can, without human supervision, adapt to changes in market activity.

In practice, observations are sampled by sequentially summing the product of
the volume $v_{i}$ and price $p_{i}$ of every trade in the market and then
sampling a new observation once this sum breaches a predefined threshold
$\delta>0$ before starting the process over again. Define the sum of the total
transacted dollar volume from the past sampled point $k$ to point $i$ as

$\chi_{i}=\sum_{j=k+1}^{i}v_{j}\cdot p_{j}$ (5.1)

where $i\geq k+1$. Once $\chi_{i}$ breaches the threshold, i.e.,
$\chi_{i}>\delta$, the sub-sampling scheme samples the trade at time $i$ as a
new observation, $k=i+1$, and resets the sum of dollar volume $\chi_{i+1}=0$.

Due to the increasing volume in the energy futures markets in recent years,
defining an appropriate threshold $\delta$ is complicated. On the one hand,
the purpose of using this sampling scheme is that the sampling frequency will
deviate throughout the day and weeks depending on the transacted dollar
volume. However, if structural changes in the market significantly alter the
transacted dollar volume over long periods, e.g., three months, it would be
advantageous for the threshold to adjust to that change. A constant threshold
will therefore be unsatisfactory as it would not be reactive to these
structural changes over long periods. A more robust alternative is a threshold
that adjusts itself without human supervision. Therefore, the threshold
$\delta$ is defined using a simple moving average over the daily dollar volume
of the past 90 days, avoiding lookahead bugs. The threshold is tuned using one
parameter, the target number of samples per day, defined as
$tgt\in\mathbb{R}_{+}$. The threshold $\delta$ is defined as

$\delta=\frac{SMA_{90d}(v\cdot p)}{tgt}$ (5.2)

which is the threshold needed to achieve the target number of samples per day
in the past 90 days. The threshold continuously updates as trades occur in the
market. There is no guarantee that the threshold will lead to the desired
amount of samples per day, as it is computed from historical data.
Nonetheless, it does achieve satisfactory results, even in unstable markets.

The time discretization scheme presented in this section represents progress
in the research area from fixed time-interval-based sampling, providing better
statistical properties while being more robust and adaptive to changing market
environments.

####  5.3 State Space

The universe of possible investments is limited to one instrument. The state
space of a financial market includes all market participants and their
attitudes toward the financial instrument. Thus, the state space $\mathcal{S}$
is continuous and partially observable. Representing the environment state
$\mathbf{s}_{t}$ to an algorithmic trading agent is impossible, so it needs to
be approximated by an agent state, i.e.,
$\mathbf{s}^{a}_{t}\approx\mathbf{s}_{t}$. This thesis adopts the philosophy
of technical traders described in section 2.5. It uses past trades,
specifically their price and volume, as observations $\mathbf{o}_{t}$ of the
environment. Let $k\in\mathbb{R}_{+}$ be the number of trades for the
instrument during the period $(t-1,t]$. An observation $\mathbf{o}_{t}$ at
time $t$ is defined as

$\mathbf{o}_{t}=[\mathbf{p}_{t},\mathbf{v}_{t}]$ (5.3)

where

  * •

$\mathbf{p}_{t}\in\mathbb{R}^{k}$ are the prices of all $k$ trades during the
period $(-1,t]$. The opening price is denoted $p_{t}$.

  * •

$\mathbf{v}_{t}\in\mathbb{R}^{k}$ are the volumes of all $k$ trades during the
period $(-1,t]$.

A single observation $\mathbf{o}_{t}$ is not a Markovian state signal, and the
agent state can be defined by the entire history $\mathbf{h}_{t}$. However,
this alternative is not scalable. Section 5.2 introduced the time
discretization scheme for this environment, which is a form of sub-sampling.
However, the computational and memory requirements still grow linearly with
the number of samples, so a history cut-off is also employed. In other words,
the agent will only have access to the past $n\in\mathbb{N}_{+}$ observations
$\mathbf{o}_{t-n+1:t}$. In addition, the recursive mechanism of considering
the past action as a part of the internal state of the environment introduced
by Moody et al. [MWLS98] is adopted to consider transaction costs. The agent
state is formed by concatenating the external state consisting of stacking the
$n$ most recent observations with the internal state consisting of the past
action $a_{t-1}$, i.e.,

$\mathbf{s}^{a}_{t}=\\{\mathbf{o}_{t-n+1:t},a_{t-1} \\}$ (5.4)

The dimension of the agent state vector is $\dim{(\mathbf{s}^{a}_{t})}=2kn+1$.

####  5.4 Action Space

At every time step $t$, the agent can buy or sell the instrument on the
market. The opening price of a period $p_{t}$, the price the agent can buy or
sell the instrument for at time $t$, is the last observed price, i.e., the
closing price of the previous period $(t-1,t]$. The no slippage assumption
from section 5.1 implies that the instrument can be bought or sold in any
quantity for the time $t$ at the opening price of that period $p_{t}$.

Some trading environments allow the agent to output the trade directly; e.g.,
$a_{t}=-5$ corresponds to selling five contracts, or $a_{t}=+10$ corresponds
to purchasing ten contracts. However, despite its intuitive nature, it can be
problematic because the agent must maintain a continuous record of the number
of contracts it holds and the amount of available balance at all times in
order to avoid making irrational decisions such as selling contracts they do
not own or purchasing more contracts than they can afford. Adding this layer
of complexity complicates the learning process. Instead, a more
straightforward approach is to have the agent output its desired position
weight. In this case, a trade is not directly outputted but inferred from the
difference between the agent’s current position and its chosen next position.

At every step $t$, the agent performs an action $a_{t}\in[-1,1]$, representing
the agent’s position weight during the period $(t,t+1]$. The weight represents
the type and size of the position the agent has selected, where

  * •

$a_{t}>0$ indicates a long position, where the agent bets the price will rise
from time $t$ to time $t+1$. The position is proportional to the size of the
weight, where $a_{t}=1$ indicates that the agent is maximally long.

  * •

$a_{t}=0$ indicates no position.

  * •

$a_{t}<0$ indicates a short position, where the agent bets the price will
fall. $a_{t}=-1$ indicates that the agent is maximally short. This thesis
assumes that there is no additional cost or restriction on short-selling.

The trading episode starts and ends (if it ends) with no position, i.e.,
${a}_{0}={a}_{T}=0$.

The weight $a_{t}$ represents a fraction of the total capital available to the
agent at any time. For this problem formulation, it is irrelevant if $a_{t}=1$
represents $\$1$ or $\$100$ million. However, this requires that any fraction
of the financial instrument can be bought and sold. E.g., if the agent has
$\$100$ to trade and wants to take the position $a_{t}=0.5$, i.e., a long
position worth $\$50$, the price might not be a factor of $50$, meaning that
the agent would not get the exact position it selected. The fractional trading
assumption is less reasonable the smaller the amount of capital available to
the agent. On the other hand, the assumptions made in section 5.1 are less
reasonable the higher the amount of capital.

####  5.5 Reward Function

As noted in section 2.6, the goal of an algorithmic trading agent should not
be to minimize forecast loss but to maximize returns, as it is more in line
with the ultimate goal of the trader. Transaction costs represent a non-
trivial expense that must be accounted for to generalize to real-world
markets. Moreover, section 2.3 introduced the philosophy of modern portfolio
theory, which advocates maximizing risk-adjusted returns. An advantage of
reinforcement learning is that the trading agent can be directly optimized to
maximize returns while considering transaction costs and risk. This section