Bellman operator as a contraction mapping
https://runzhe-yang.science/2017-10-04-contraction/
How Does Value-Based Reinforcement Learning Find the Optimal Policy?
2017-10-04
Runzhe Yang
<div><article>
<p>DeepMind researchers claimed state-of-the-art results in the <a href="http://www.arcadelearningenvironment.org">Arcade Learning Environment</a> in their recent ICML paper “<a href="https://deepmind.com/research/publications/distributional-perspective-reinforcement-learning/"><strong><em>A Distributional Perspective on Reinforcement Learning</em></strong></a>”. They investigate a distributional perspective of the value function in the reinforcement learning context and further design an algorithm applying Bellman’s equation to approximately learn value distributions, which results in better policy optimization. The discussion in their paper follows closely the classical analysis of <em>Bellman operator</em> and <em>dynamic programming</em>, which should be familiar to most AI researchers.</p>
<p>In this article, I will introduce the mathematical background of the classical analysis of reinforcement learning, that is, the Bellman operator is essentially a contraction mapping on a complete metric space, and explain how value-based reinforcement learning, e.g. Q-learning, finds the optimal policy. Based on these general ideas, readers will see how straightforward the idea that the DeepMind paper comes up with, and hopefully draw some inspiration from this topological perspective to design future RL algorithms.</p>
<p>For the reader who prefers to systematically learn from books, I strongly recommend you to read the textbook <a href="https://www.amazon.com/Operator-Engineering-Science-Mathematical-Sciences/dp/038795001X"><strong><em>Linear Operator Theory in Engineering and Science</em></strong></a> for acquiring mathematical background, and the book <a href="http://web.mit.edu/dimitrib/www/abstractdp_MIT.html"><strong><em>Abstract Dynamic Programming</em></strong></a> by <a href="http://www.mit.edu/~dimitrib/home.html"><em>Dimitri P. Bertsekas</em></a> to study the connection between contraction mapping and dynamic programming. If you don’t remember Markov decision processes or policy optimization in reinforcement learning, <a href="https://runzhe-yang.science/2017-05-25-TRPO/"><em>my previous blog</em></a> has introduced them briefly.</p>
<ul>
<li><a href="#topology-of-metric-spaces">Topology of Metric Spaces</a> <ul>
<li><a href="#cauchy-sequences-and-completeness">Cauchy Sequences and Completeness</a></li>
<li><a href="#contraction-mapping-theorem">Contraction Mapping Theorem</a></li>
</ul>
</li>
<li><a href="#contractive-dynamic-programming">Contractive Dynamic Programming</a> <ul>
<li><a href="#contraction-property">Contraction Property</a></li>
<li><a href="#monotonicity-property">Monotonicity Property</a></li>
</ul>
</li>
<li><a href="#value-based-reinforcement-learning">Value-Based Reinforcement Learning</a> <ul>
<li><a href="#approximate-value-iteration">Approximate Value Iteration</a></li>
<li><a href="#approximate-policy-iteration">Approximate Policy Iteration</a></li>
<li><a href="#q-learning-asynchronous-value-iteration">Q-Learning: Asynchronous Value Iteration</a></li>
</ul>
</li>
<li><a href="#a-methodology-for-developing-iterative-rl-algorithms">A Methodology for Developing Iterative RL Algorithms</a> <ul>
<li><a href="#learning-from-value-distribution">Learning from Value Distribution</a></li>
<li><a href="#multi-objective-rl">Multi-Objective RL</a></li>
</ul>
</li>
<li><a href="#summary-contraction-value-iteration-and-more">Summary: Contraction, Value Iteration, and More</a></li>
</ul>
<h3>Topology of Metric Spaces</h3>
<p>Literally, the word “topology” means “the study of spatial properties”. The space here could be any abstract space with certain properties. A natural way for us to quickly start the discussion is to observe a metric space.</p>
<p><strong>Definition 1. (Metric Space)</strong> A <em>metric space</em> is an ordered pair \((X, d)\) consists of an <em>underlying set</em> \(X\) and a real-valued function \(d(x, y)\), called <em>metric</em>, defined for \(x, y \in X\) such that for any \(x, y, z \in X\) the following conditions are satisfied:</p>
<ol>
<li>\(d(x, y) \geq 0\)                          [<em>non-negativity</em>]</li>
<li>\(d(x, y) = 0 \Leftrightarrow x = y\)         [<em>identity of indiscernibles</em>]</li>
<li>\(d(x, y) = d(y, x)\)                 [<em>symmetry</em>]</li>
<li>\(d(x, y) \leq d(x, z) + d(z, y)\) [<em>triangle inequality</em>]</li>
</ol>
<p>All these four conditions are in harmony with our intuition of distance. Indeed, the Euclidean distance 
\(d(x, y) = \sqrt{(x_1 - y_1)^2 + (x_2 - y_2)^2 + (x_3 - y_3)^2}\) 
is a valid metric.</p>
<h4>Cauchy Sequences and Completeness</h4>
<p>When we construct a metric space \((X, d)\), it is common to expect our metric space has no “point missing” from it. For example, the space \(\mathbb Q\) of rational numbers, with the metric by the absolute value \(|\cdot|\) of the difference, is not complete. Just consider the rational sequence \(x_{n+1} = \frac{x_n}{2} + \frac{1}{x_n}\), \(x_1 = 1\), whose elements become arbitrarily close to each other as the sequence progresses, but it doesn’t converge to any point in space \(\mathbb Q\). Therefore, the space \(\mathbb Q\) is incomplete, since it obviously misses points.</p>
<p>A sequence like that in our example is called <em>Cauchy sequence</em>. Formally, we define the Cauchy sequence in and the completeness of a metric space \((X, d)\) as follows:</p>
<p><strong>Definition 2. (Cauchy Sequence)</strong> A sequence \(\{x_n\}\) is said to be <em>Cauchy sequence</em> if for each \(\epsilon &gt; 0\) there exists an \(N_\epsilon\) such that \(d(x_n, x_m) \leq \epsilon\) for any \(n, m \geq N_\epsilon\).</p>
<p>
</p>
<p><strong>Definition 3. (Complete Metric Space)</strong> A metric space is said to be <em>complete</em> if each Cauchy sequence is a convergent sequence in it.</p>
<p>The significance of complete metric space is difficult to be overemphasized. In many application, it is easier to show a given sequence is a Cauchy sequence than to show it is convergent. However, if the underlying metric space is complete, to show a given sequence is a Cauchy sequence is sufficient for convergence.</p>
<h4>Contraction Mapping Theorem</h4>
<p>Now we introduce the contraction mapping based on the concept of completeness.</p>
<p><strong>Definition 4. (Contraction)</strong> Let (X, d) be a metric space and \(f: X \rightarrow X\). We say that \(f\) is a <em>contraction</em>, or a <em>contraction mapping</em>, if there is a real number \(k \in [0, 1)\), such that
\[
 d(f(x), f(y)) \leq kd(x,y)
\]
for all x and y in X, where the term \(k\) is called a <em>Lipschitz coefficent</em> for \(f\).</p>
<p>The reason that contractions play an important role in <a href="http://www.wikiwand.com/en/Dynamic_programming">dynamic programming</a> problems lies in the following <a href="http://www.wikiwand.com/en/Banach_fixed-point_theorem">Banach fixed-point theorem</a>.</p>
<p><strong>Theorem 1. (Contraction Mapping Theorem)</strong> Let \((X, d)\) be a complete metric space and let \(f: X \rightarrow X\) be a contraction. Then there is one and only one fixed point 
\(x^*\)
such that 
\[
 f(x^* ) = x^* .
\]
Moreover, if \(x\) is any point in \(X\) and \(f^n(x)\) is inductively defined by \(f^2(x) = f(f(x))\), \(f^3(x) = f(f^2(x)), \dots, f^n(x) = f(f^{n-1}(x))\), then \(f^n(x) \rightarrow x^*\) as \(n \rightarrow \infty\).</p>
<p><strong><em>proof.</em></strong> Let us first prove every sequence \(\{f^n(x)\}\) convergent to a fixed point. Since the metric space \((X, d)\) is complete, we only need to prove every \(\{f^n(x)\}\) is a Cauchy sequence. By the triangle inequality and symmetry of metric, for all \(x, y \in X\),</p>
\[\begin{eqnarray}
 d(x, y) &amp;\leq &amp; d(x, f(x)) + d(f(x), f(y)) + d(f(y), y)\\
 &amp;\leq &amp; d(f(x), x) + kd(x, y) + d(f(y), y)\\
 \Rightarrow 
 d(x, y) &amp;\leq&amp; [d(f(x), x) + d(f(y), y)]/(1-k)
 \end{eqnarray}\]
<p>Consider two points \(f^\ell(x)\), \(f^m(x)\) in the sequence. Their distance is bounded by</p>
\[\begin{eqnarray}
 d(f^\ell(x), f^m(x)) &amp;\leq&amp; [d(f^{\ell+1}(x), f^{\ell}(x)) + d(f^{m+1}(x), f^m(x))]/(1-k)\\
 &amp;\leq &amp; [k^\ell d(f(x), x) + k^m d(f(x), x)]/(1-k)\\
 &amp;= &amp; \frac{k^\ell + k^m}{1-k}d(f(x), x)\\
 \end{eqnarray}\]
<p>since \(k \in [0, 1)\) the distance converges to zero as \(\ell, m \rightarrow \infty\), proving that \(\{f^n(x)\}\) is a Cauchy sequence. Because the space \((X, d)\) is complete, \(\{f^n(x)\}\) converges.</p>
<p>We now show that \(x^* = \lim_{m\rightarrow \infty} f^m(x)\) is a fixed point, and it is unique. Since \(f\) is a contration, by definition it is continous. Therefore 
\[
 f(x^* ) = f(\lim_{m\rightarrow \infty} f^m(x)) = \lim_{m\rightarrow \infty} f(f^m(x)) = \lim_{m\rightarrow \infty} f^m(x) = x^* .
\]
The uniqueness of \(x^*\) can be proved by contradiction. Assume \(x^*\) and \(y^*\) are two distinct fixed points of \(f\). We then get the contradiction
\[
 0 &lt; d(x^* , y^* ) = d(f(x^* ) , f(y^* )) \leq k d(x^* , y^* ) &lt; d(x^* , y^* ).
\]
Hence \(f\) has only one fixed point and \(\{f^n(x)\}\) converges to it for any \(x \in X\). \(\blacksquare\)</p>
<h3>Contractive Dynamic Programming</h3>
<p>The contractive dynamic programming model is introduced by <a href="https://www.jstor.org/stable/2027440">Eric V. Denardo in 1967</a>, which applies contraction theorem to a broad class of optimization problems. It captures two widely shared properties of the optimization problem, <em>contraction</em> and <em>monotonicity properties</em> to develop a high level abstraction of optimization methods.</p>
<p>Some terminology is now introduced. Let \(S\), \(A\) be two sets, which we refer to as a set of “states” and a set of “actions”, respectively. For each \(s \in S\), let \(A_s \subseteq A\) be the nonempty subset of actions that are feasible at state \(s\). We denote by \(\Pi\) the set of all stationary deterministic policies \(\pi: S \rightarrow A\), with \(\pi(s) \in A_s\), for all \(s \in S\).</p>
<p>To introduce return function, let \(V\) be the set of all bouned real valued functions from \(S\) to reals, and let \(H: S\times A \times V \rightarrow \mathbb R\) be a given mapping. One might think \(H(s, a, v)\) as the total payoff for starting at state \(s\) and choosing action \(a\) with the prospect of receiving \(v(s')\) if the transition \((s, a)\rightarrow s'\) is happened. We consider the operator \(\mathcal T_\pi: V \rightarrow V\) defined by 
\[
 (\mathcal T_\pi v)(s) := H(s, \pi(s), v), ~\forall s\in S, v\in V,
\]
and we also consier the operator \(\mathcal T\) defined by 
\[
 (\mathcal T v)(s) := \sup_{a \in A_s}H(s, a, v), ~\forall s\in S, v\in V.
\]
We call \(\mathcal T_\pi\) the “evaluation operator” and \(\mathcal T\) the “optimality operator”. Our goal is to find a policy maximizing our total payoff. Following two properties provide conditions under which our goal can be realized by iteratively applying these two operators.</p>
<h4>Contraction Property</h4>
<p>Let a metric \(d\) on \(V\) is defined by 
\(d(u, v) = \sup_{s \in S} |u(s) - v(s)|\). Using the fact \(V\) is a space of bounded functions, it is easy to show that the metric space \((V, d)\) is complete.</p>
<p><strong>Assumption 1. (Contraction)</strong> For all \(u, v \in V\) and \(\pi \in \Pi\), for some \(\gamma \in [0, 1)\), we have
\[
 d(\mathcal T_{\pi} u, \mathcal T_{\pi} v) \leq \gamma d(u ,v).
\]</p>
<p><strong>Proposition 1.</strong> Suppose the contraction assumption is satisfied. Then \(\mathcal T\) is also a contraction with the same Lipschitz coefficient \(\gamma\).</p>
<p><strong><em>Proof.</em></strong> Consider arbitrary \(u, v\) and we can write \((\mathcal T_\pi u) (s) \leq (\mathcal T_\pi v)(s) + \gamma d(u, v)\) for each \(\pi \in \Pi\) by assumption. For any fixed state \(s\), we take supremum of both side over \(\pi \in \Pi\), we have
\[
 \sup_{\pi\in\Pi} \left\{(\mathcal T_\pi u) (s)\right\} = \sup_{\pi(s) \in A_s}H(s, \pi(s), v) = (\mathcal T u) (s) \leq (\mathcal T v)(s) + \gamma d(u, v)
\]
Reversing the rolee of \(u\) and \(v\), we have \((\mathcal T v) (s) \leq (\mathcal T u)(s) + \gamma d(u, v)\) for any fixed state \(s\). Jointly, we have proved \(|(\mathcal T u)(s) - (\mathcal T v) (s)| \leq \gamma d(u, v)\) for all \(s\in S\). Hence, 
\[
 d(\mathcal T u, \mathcal T v) = \sup_{s\in S} |(\mathcal T u)(s), (\mathcal T v)(s)| \leq \gamma d(u, v).
\]
The optimality operator \(\mathcal T\) is also a contraction with Lipschitz coefficient \(\gamma\). \(\blacksquare\)</p>
<p>The <a href="#contraction-mapping-theorem">Banach fixed-point theorem</a> guarantees that iteratively applying evaluation operator \(\mathcal T_\pi\) to any function \(v \in V\) will converge to a unique function \(v_\pi\) in \(V\) such that 
\[
 v_\pi(s) = H(s, \pi(s), v_\pi), ~~ \text{for each}~ s \in S.
\]
The function \(v_\pi\) is called <em>value fuction</em> of the policy \(\pi\). The <em>optimal value function</em> \(v_{\mathtt{opt}}\) is defined by \(v_{\mathtt{opt}}(s) := \sup_{\pi\in \Pi} v_{\pi}(s)\) for all \(s\in S\). Similarly, iteratively applying optimality operator \(\mathcal T\) to any funciton \(v \in V\) will converge to a unique function \(v^* \in V\) such that
\[
 v^* (s) = \sup_{a\in A_s}H(s, \pi(s), v^* ), ~~\text{for each}~ s \in S.
\]
How should we interpretate this fixed point \(v^*\)? When does \(v^*\) equal \(v_{\mathtt{opt}}\) that we desire? One may have a sense that \(v_{\mathtt{opt}} \geq v^*\) since the following result aways holds.</p>
<p><strong>Lemma 1. (\(v_{\mathtt{opt}} \geq v^*\))</strong> For any \(\epsilon &gt; 0\), there exists a policy \(\pi_\epsilon\) such that for any \(s\), \(H(s, \pi_\epsilon(s), v^* ) \geq \sup_{a\in A_s} H(s, a, v^* ) - \epsilon (1-\gamma)\), and any such \(\pi_\epsilon\) satisfies \(v_{\pi_\epsilon} \geq v^* - \epsilon\).</p>
<p><strong><em>Proof.</em></strong> Note that for any \(v \in V\) and \(\pi\in\Pi\), we have an upper bound</p>
\[\begin{eqnarray}
 d(v_{\pi}, v) &amp; = &amp; \lim_{n\rightarrow \infty}d(\mathcal T_{\pi}^{n} v, v)\\
 &amp; \leq &amp; \sum_{\ell = 1}^{\infty}d(\mathcal T_{\pi}^{\ell} v, \mathcal T_{\pi}^{\ell-1} v) ~~~~\text{[triangle inequality]}\\
 &amp; \leq &amp; \sum_{\ell = 1}^{\infty} \gamma^{\ell - 1} d(\mathcal T_{\pi} v, v)~~~~~~~\text{[contraction assumption]}\\
 &amp; = &amp; \frac{1}{1 - \gamma} d(\mathcal T_\pi v, v).
 \end{eqnarray}\]
<p>Now let \(v = v^*\), and \(\pi = \pi_{\epsilon}\) such that \(H(s, \pi_\epsilon(s), v^* ) \geq \sup_{a\in A_s} H(s, a, v^* ) - \epsilon (1-\gamma)\) for any \(s\), we have \(d(v_{\pi_{\epsilon}}, v^* ) \leq \epsilon\). \(\blacksquare\)</p>
<p>If we can show the reverse inequality, \(v^* \geq v_{\mathtt{opt}}\) , then we conclude \(v^* = v_{\mathtt{opt}}\). However, it does not always hold true. A sufficient condition, <em>monotonicity property</em>, is therefore introduced to make them identical.</p>
<h4>Monotonicity Property</h4>
<p><strong>Assumption 2. (Monotonicity)</strong> For \(u, v \in V\), if \(u \geq v\), then \(\mathcal T_\pi u \geq \mathcal T_\pi v\) for each \(\pi\in \Pi\), where we write \(u \geq v\) if \(u(s) \geq v(s)\) for each state \(s\), and \(u &gt; v\) if \(u \geq v\) and \(u \neq v\).</p>
<p>The monotonicity property was introduced by <a href="http://pubsonline.informs.org/doi/pdf/10.1287/opre.12.4.610">L. G. Mitten in 1964</a>. There are many value functions, including the one in the infinite horizon MDP, have the monotonicity property. The following theorem shows why the contraction and monotonicity properties are important in the contractive dynamic programming model.</p>
<p><strong>Theorem 2.</strong> Suppose the contraction and monotonicity properties hold. Then \(v^* = v_{\mathtt{opt}}\).</p>
<p><strong><em>Proof.</em></strong> Note that \(H(s, \pi(s), v^* ) \leq \sup_{a \in A_s} H(s, a, v^* )\) for any \(s\in S\) and \(\pi \in \Pi\). It means \(\mathcal T_{\pi} v^* \leq \mathcal T v^*\) for all \(\pi\). Repeatedly applying the monotonicity property \(k\) times, we derive \(\mathcal T_{\pi}^k v^* \leq \cdots \leq \mathcal T_{\pi}^2 v^* \leq \mathcal T_{\pi} v^* \leq \mathcal T v^* = v^*\). By contraction property \(\lim_{k\rightarrow \infty}\mathcal T_{\pi}^k v^* = v_{\pi}\). Therefore \(v_{\pi} \leq v^*\) for any \(\pi\) and \(v_{\mathtt{opt}} \leq v^*\). On the other hand, by <a href="#contraction-property">Lemma 1</a> which utilizes the contraction property we have \(v^* - \epsilon \leq v_{\pi_{\epsilon}} \leq v_{\mathtt{opt}}\) for any \(\epsilon &gt; 0\). Jointly,
\[
 v^* - \epsilon \leq v_{\mathtt{opt}} \leq v^*
\]
for any \(\epsilon &gt; 0\). This implies \(v^* = v_{\mathtt{opt}}\) and completes the proof. \(\blacksquare\)</p>
<h3>Value-Based Reinforcement Learning</h3>
<p>Reinforcement learning (RL) approximates the optimal policy of a Markov decision process. This method succeeded in solving many challenging problems in diverse fields, such as robostics, transportation, medicine, and finance. Many RL methods are considered by the classic textbook <a href="http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf"><strong><em>Reinforcement Learning: An Introduction</em></strong></a> by Richard S. Sutton and Andrew G. Barto (We are translating its latest version into Chinese!).</p>
<p>Roughly speaking, the value (function) based reinforcement learning is a large category of RL methods which take advantage of the Bellman equation and approximate the value function to find the optimal policy, such as <em>SARSA</em> and <em>Q-Learning</em>. By contrast, the policy-based reinforcement learning methods directly optimize the quantity of total reward, e.g. REINFORCE, which are more stable but less sample efficient than value-based methods.</p>
<p>The key idea of value-based RL relates two <a href="http://www.wikiwand.com/en/Bellman_equation">Bellman equations</a> of an infinite horizon discounted MDP \(\mathcal M = \langle S, A, \mathcal R, \mathcal P, \gamma \rangle\), where \(\mathcal R(s, a)\) is the immediate reward function, and \(\mathcal P(s'|s, a)\) is the probabilistic transition model, and \(\gamma\) is the discount factor (If you just forget the definition of MDPs, please refer to <a href="https://runzhe-yang.science/2017-05-25-TRPO/"><em>my previous blog</em></a>). The first equation is so-called <em>Bellman expectation equation</em>:
\[
 v_{\pi}(s) = R(s, \pi(s)) + \gamma \sum_{s’ \in S} \mathcal P(s’ | s, \pi(s))\cdot v_{\pi}(s’).
\]
This equation describe the expected total reward for taking the action prescribed by policy \(\pi\). The other equation is called <em>Bellman optimality equation</em>:
\[
 v^* (s) = \max_{a\in A} \left\{R(s, a) + \gamma \sum_{s’ \in S} \mathcal P(s’ | s, a)\cdot v^* (s’)\right\}.
\]</p>
<p>Now we consider two corresponding operators, defined by the following
\[
 (\mathcal T_{\pi}v)(s) := R(s, \pi(s)) + \gamma \sum_{s’ \in S} \mathcal P(s’ | s, \pi(s))\cdot v(s’) ~~~\text{[Bellman evaluation operator]},
\]
and 
\[
 (\mathcal Tv)(s) := \max_{a\in A} \left\{ R(s, a) + \gamma \sum_{s’ \in S} \mathcal P(s’ | s, a)\cdot v(s’) \right\} \text{[Bellman optimality operator]}
\]
for any value function \(v \in V\). Note that \(v_{\pi}\) and \(v^*\) are fixed points of operator \(\mathcal T_{\pi}\) and \(\mathcal T\) respectively. If we can show that \(\mathcal T_{\pi}\) satisfies both contraction and monotonicity properties, according to results we have drawn in <a href="#contractive-dynamic-programming">the last section</a>, \(v^\pi\) and \(v^*\) are unique fixed points that repeatedly applying \(\mathcal T_{\pi}\) and \(\mathcal T\) will converge to respectively, and \(v^* = v_{\mathtt{opt}}\).</p>
<p><strong>Proposition 2.</strong> The contraction and monotonicity properties hold for Bellman evaluation operator \(\mathcal T_{\pi}\) for any stationary policy \(\pi \in \Pi\).</p>
<p><strong><em>Proof.</em></strong> For any value functions \(u, v \in V\), and any policy \(\pi \in \Pi\), we have</p>
\[\begin{eqnarray}
 d(\mathcal T_{\pi} u, \mathcal T_{\pi} v) &amp; = &amp; \sup_{s \in S} \left|\gamma \sum_{s' \in S} \mathcal P(s' | s, \pi(s))\cdot (u(s') - v(s'))\right|\\
 &amp; \leq &amp; \sup_{s \in S} \gamma \sum_{s' \in S} \mathcal P(s' | s, \pi(s))\cdot \left|u(s') - v(s')\right|\\
 &amp; \leq &amp; \sup_{s \in S} \gamma \sup_{s' \in S} \left|u(s') - v(s')\right|\\
 &amp; = &amp; \gamma d(u ,v) 
 \end{eqnarray}\]
<p>Therefore \(\mathcal T_{\pi}\) is a contraction with Lipschitz coefficient \(\gamma\). The contraction property holds. According to <a href="#contraction-property">Proposition 1</a>, the optimality operator \(\mathcal T\) is also an \(\gamma\)-contraction.</p>
<p>Further more, for any value functions \(u \geq v\), we have
\[
 (\mathcal T_{\pi} u)(s) - (\mathcal T_{\pi} v)(s) = \gamma \sum_{s’ \in S} \mathcal P(s’ | s, \pi(s))\cdot (u(s’) - v(s’)) \geq 0
\]
for all \(s \in S\). Hence \(\mathcal T u \geq \mathcal T v\). The monotonicity property holds. \(\blacksquare\)</p>
<h4>Approximate Value Iteration</h4>
<p>As the discussion above indicates, if the model is known, one can design an policy optimization algorithm by generating \(\mathcal T v, \mathcal T^2 v, \dots\) which converges to \(v^*\) and then extracting the corresponding optimal policy. This is exactly the value iteration (VI) method. This algorithm simply starts with any value function guess and repeatedly apply the Bellman optimality operator on it to obtain the optimal value function. Sometimes VI method may be implementable only by approximation, e.g., when the state space \(S\) is extremely large or when the value function is updated by some constant step size. A common solution is depicted as the following <em>approximate value iteration algorithm</em>.</p>
<div><pre><code># Approximate Value Iteration 
Initialize value function v arbitrarily.
Repeat:
 Update v, such that sup|v-T(v)| ≤ δ
until v converges
Extract a policy: π, such that sup|T_π(v) - T(v)| ≤ ϵ
</code></pre></div>
<p>By applying the triangle inequality and the contraction property of \(\mathcal T_{\pi}\) and \(\mathcal T\), we can assert the error bounds for the output value function \(v_{\mathtt{final}}\), which is bounded by \(d(v_{\mathtt{final}}, v_{\mathtt{opt}} ) \leq \frac{\delta}{1 - \gamma}\), and the extracted policy \(\pi_{\mathtt{final}}\), whose value is bounded by \(d(v_{\pi_{\mathtt{final}}}, v_{\mathtt{opt}} ) \leq \frac{\epsilon}{1 - \gamma} + \frac{2\gamma\delta}{(1-\gamma)^2}\) (proof is straightforward, check them yourself :-)). The VI algorithm converges exponentially when \(\epsilon = 0\) and \(\delta = 0\).</p>
<h4>Approximate Policy Iteration</h4>
<p>The VI method is a direct creation based on the fact that the Bellman optimality operator \(\mathcal T\) is a contraction. However, people may think by involving policy evaluation each step one can have a better rate of convergence and a better approximation of optimal policy, though it will incur more calculation each iteration. <strong>This is plausible, but not insignificantly so in theory.</strong> The approximate policy iteration (PI) algorithm is stated as follows:</p>
<div><pre><code># Approximate Policy Iteration
Given the current policy π_k
Repeat:
 Policy evaluation: Compute v such that
 sup|v - (T_π_k)^m_k (v) | ≤ δ
 by appying Bellman evaluation operator T_π
 Policy improvement: Extract a policy π_{k+1} that satisfies
 sup|T_π_{k+1}(v)- T(v)| ≤ ϵ
 π_k ← π_{k+1}
until v converges.
Extract π_{k+1} 
</code></pre></div>
<p>In this algorithm, the parameter \(m_k\) is an positive intergers. Taken the usual choice of \(m_k = 1\), we can show that \(d(v_{\pi_{k+1}}, v_{\mathtt{opt}}) \leq \gamma d(v_{\pi_{k}}, v_{\mathtt{opt}}) + \frac{\epsilon + 2\delta\gamma}{1 - \gamma}\) for each iteration. Therefore, its final error bound is asymptotically proportional to \(1/(1-\gamma)^2\), which indicates PI does not hold an accuracy advantage over VI. And the exact PI converges at an exponential rate which is as same as that of exact VI.</p>
<h4>Q-Learning: Asynchronous Value Iteration</h4>
<p>In the last two section we explained how we do classical model based planing for MDPs from the topological point of view. Many readers by now have a sense that how this stuff and value-based reinforcement learning are connected. But there remains a critical question: Can this explanation is still suitable for those general model free cases? Can we explain those classical reinforcement “learning” algorithm, such as Q-learning, in an analog way?</p>
<p>Intuitively, the answer should be “yes”. Consider the evaluation operator \(\mathcal T_{\pi}\) and optimality operator \(\mathcal T\) defined by
\[
 (\mathcal T_{\pi} q)(s, a) := R(s, a) + \gamma\mathbb E_{\mathcal P, \pi} q(s’, a’)
\]
and
\[
 (\mathcal T q)(s, a) := R(s, a) + \gamma\mathbb E_{\mathcal P} \max_{a’\in A} q(s’, a’)
\]
respectively, over the Q value function space \(\mathbb R^{S\times A}\), with the similar metric as that of \(V\). Following the similar analysis, the contraction and the monotonicity properties hold for them. However, since the model is unknown, we cannot update the Q value function as that in value iteration. We can only approximate and update the Q value <strong>for some certain pair \((s, a)\)</strong> by sampling or temporal difference method each iteration. Is its convergence still guaranteed? The following theorem resolves our doubts.</p>
<p><strong>Definition 5. (Asynchronous Value Iteration)</strong> Conside value function \(v\) as a composition of 
\((v_s)_ {s\in S}\) such that in each iteration,
\[
 v_s^{t+1}(s’) = \begin{cases}
 \mathcal Tv^{t}(s’), &amp; \text{if}~s’ = s\newline
 \mathcal v_s^{t}(s’), &amp; \text{otherwise.}
 \end{cases}
\]
<strong>Theorem 3. (Asynchronous Convergence Theorem)</strong> If each restricted value function \(v_s\) can be update infinite times, and there is a sequence of nonempty subsets \(\{V^k\} \subseteq V\) with \(V^{k+1}\subseteq V^{k}\), \(k = 0, 1, \dots\) such that if \(\{v^k\}\) is any sequence with \(v_k\in V_k\) for all \(k \leq 0\), the \(\{v^k\}\) converges pointwise to \(v^*\). Assume further the following:</p>
<ol>
<li><em>Synchronous Convergence Condition</em>: We have 
\[
 \mathcal T v \in V^{k+1}, \forall v \in V^k
\]</li>
<li><em>Box Condition</em>: For all \(k, V^k\) is a Cartesian product of the form
\[
 V^k = \times_{s\in S} V_s^k 
\]</li>
</ol>
<p>where \(V_{s}^k\) is a set of bounded real-valued functions on state \(s\). Then for every \(v^0 \in V^0\) the sequence \(\{v^t\}\) generated by the asynchronous VI algorithm converges to \(v^*\).</p>
<p><strong><em>Proof.</em></strong> To show that the algorithm converges is equivalent to show that the iterates from \(V^k\) will get in to \(V^{k+1}\) eventually, i.e., for each \(k \leq 0\), there is a time \(t_k\) such that \(v^t\in V^{k}\) for all \(t \geq t_k\). We can prove it by mathematical induction.</p>
<p></p>
<p>When \(k = 0\), the statement is true since \(v^0\in V^0\). Assuming the statement is true for a given \(k\), we will show there is a time \(t_{k+1}\) with the required properties. For each \(s\in S\), let set \(L_{s} = \{t: s^t = s\}\) record the time an update happend on the state \(s\). Let \(t(s)\) be the first element in \(L_{s}\) such that \(t(s) \geq t_{k}\). Then by the synchrnous convergence condition, we have \(\mathcal Tv^{t(s)} \in V^{k+1}\) for all \(s\in S\). In the view of the box condition, it implies \(v^{t+1}(s)\in V_s^{k+1}\) for all \(t \geq t(s)\) for any \(s\in S\). Therefore, let \(t_{k+1} = \max_{s}t(s) + 1\), using the box condition, we have \(v^t \in V^{k+1}\) for all \(t \leq t_{k+1}\). By mathematical induction, the statement holds, and \(V^k\) will get in to \(V^{k+1}\) eventually. Hence, \(\{v^t\}\) converges to \(v^*\). \(\blacksquare\)</p>
<h3>A Methodology for Developing Iterative RL Algorithms</h3>
<p>It is really exciting to see that the convergence and correctness of value-based reinforcement learning algorithm, such as Q-learning, can be interpreted in a topological view. In fact, the contraction mapping theory provides us with a methodology for developing iterative value-based RL algorithm. In a high-level overview, iterative value-based RL algorithms are concerned with the following five essential elements:</p>
<ul>
<li>
<p><strong>Value Space \(V\):</strong>
A common choice of \(V\) is the space of value functions in \(\mathbb R^S\) or the space of Q-value functions in \(\mathbb R^{S\times A}\). There are many other choices such as the space of ordered pair of \((V,Q)\) (Example 2.6.2 in Dimitri’s book), or the space of multi-dimensional value functions.</p>
</li>
<li>
<p><strong>Value Metric \(d\):</strong> It defines the “distance” between two points in the value space. Besides the basic four conditions, the metric \(d\) should ensure a complete metric space \((V, d)\) to validate the Banach’s fixed-point theorem. A compatible selection of value metric will make our analysis much easier.</p>
</li>
<li>
<p><strong>Evaluation Operator \(\mathcal T_{\pi}\):</strong> We have constructed a recursive expression of a certain value point in the value space associated with some policy, e.g., the Bellman expectation equation,
\[
 v_{\pi}(\cdot) = H(\cdot, \pi(\cdot), v_{\pi})
\]
to depict the value of a policy \(v_{\pi}\) as a fixed point we desire. Carefully verify that the contraction and the monotonicity properties hold for \(\mathcal T_{\pi} v:=H(\cdot, \pi(\cdot), v)\).</p>
</li>
<li>
<p><strong>Optimality Operator \(\mathcal T\):</strong> Write down recursive expression of the optimal point in the value space, e.g., the Bellman optimality equation,
\[
 v^* (\cdot) = \sup_{\pi\in \Pi}\mathcal T_{\pi} v^* (\cdot).
\]
One thing important is to make sure the contraction property holds for the optimality operator \(\mathcal T:= \sup_{\pi\in \Pi}\mathcal T_{\pi}v\). Note that when the metric \(d\) is the supremum of the absolute value of the difference, the contraction property of \(\mathcal T\) is automatically satisfied according to <a href="#contraction-property">Proposition 1</a>.</p>
</li>
<li>
<p><strong>Updating Scheme:</strong> It is the most tricky part. For those small-scale problems, the asynchronous value iteration is enough. However, to make our reinforcement learning algorithm practical and scalable, we have to consider much more factors. For example: Should our algorithm be on-policy or off-policy? How do we approximate the value and policies? If it is an online algorithm, how do we trade off exploration and exploitation? All these details will significantly influence the performance of our algorithm on real-world tasks.</p>
</li>
</ul>
<h4>Learning from Value Distribution</h4>
<p></p>
<p>Let’s get back to <a href="https://deepmind.com/research/publications/distributional-perspective-reinforcement-learning/">the DeepMind paper mentioned at the begining</a>, which proposes a new reinforcement learning algorithm from a distributional perspective. This work is actually a good example of developing and analyzing a new reinforcement learning algorithm using the methodology that we summarize above.</p>
<p>In this example, the <strong>value space</strong> consists of all the “value distribution” \(Z(s, a)\). The <strong>value metric</strong> of the value space is defined by \(d(Z_1, Z_2):= \sup_{s, a}d_{w}(Z_1(s,a), Z_2(s,a))\), where \(d_{w}\) is the Wasserstein distance between two random variables. The <strong>evaluation operator</strong> is defined by 
\[
 \mathcal T_{\pi} Z(s, a) \stackrel{D}{:=} R(s, a) + \gamma P^{\pi}Z(s, a),
\]
which is proved to be a contraction in \((\mathcal Z, d)\), where “\(\stackrel{D}{=}\)” means random variables on both sides are drawn from the same distribution. The monotonicity property also holds as they define the order as the order of the magnitude of expectation (\(Z_1 \leq Z_2 \Leftrightarrow \mathbb E Z_1 \leq \mathbb E Z_2\)). The <strong>optimality operator</strong> is defined by \(\mathcal T Z := \sup_{\pi}\mathcal T_{\pi}Z\). However, since the relative order is incompatible with the metric, this optimality operator is NOT a contraction in \((\mathcal Z, d)\). They use approximate asynchronous value iteration as the <strong>updating scheme</strong>, where the value distribution function is approximated by a neural network which outputs a discrete distribution between \([V_\text{MAX}, V_\text{MIN}]\).</p>
<p>If we only consider the space of the first-order moment of the value distribution and the metric
\(d_{\mathbb E}(\mathbb E Z_1, \mathbb E Z_2):= \sup_{s, a} |\mathbb E Z_1(s,a) - \mathbb E Z_2(s,a)|\), both \(\mathbb E\mathcal T_{\pi}\) and \(\mathbb E\mathcal T\) are contractions. And this explains why the C51 algorithm works well. However, as for predicting value distribution, I doubt the correctness of directly using the result of value iteration. In my opinion, a policy iteration algorithm would be a better choice here for maintaining value distributions, or we should correct the value distribution function by repeatedly applying evaluation operator after the optimal policy is found.</p>
<h4>Multi-Objective RL</h4>
<p>Another example which might be helpful for us to understand this methodology is the multi-objective reinforcement learning. And this is also an interesting topic I am recently working on. The value space of this problem should be some multi-dimensional value function. The metric can be extended from that of a normed vector space. If we only want to find one policy resulting a Pareto optimal return, or simply want to find the best policy maximizing the return associated with specific weights, the evaluation operator and the optimality operator are very similar to \(R + \gamma P^{\pi}V\) and \(\sup_{\pi} \{R + \gamma P^{\pi}V\}\) (the \(\sup\{\cdot\}\) is defined by Pareto domination and the order of magnitude of weighted sum, respectively). Designing an updating scheme according to the real problem, we are guaranteed to have a viable algorithm. If we want to find all the policies resulting in a Pareto optimal return, or make the policies controllable, how to represent a collection of policies and write down the corresponding evaluation/optimality operator are very challenging.</p>
<h3>Summary: Contraction, Value Iteration, and More</h3>
<p>In this article, some topological backgrounds of metric spaces and the Banach fixed-point theorem are first introduced. Based on these mathematical foundations, we investigate the contractive dynamic programming, where the contraction property and the monotonicity property together guarantee a unique optimal solution can be achieved exponentially fast. Bring these ideas to the reinforcement learning context, we can explain and analyze value iteration, policy iteration and Q-learning algorithms in a unified framework. It further inspires us to develope a methodology for developing iterative value-based reinforcement algorithm, which requires us to consider <em>value space</em>, <em>value metric</em>, <em>evaluation operator</em>, <em>optimality operator</em> and <em>updating scheme</em>.</p>
<p>The next time someone asks, “how does a value-based RL algorithm converge?”, or “how do people come up with a new RL algorithm?”, we can answer him not first by complex, detailed mathematical deduction, but simply by drawing an intuitive picture and analyzing on those accessible abstractions! Our journey on the topic of topological explanation of value-based reinforcement learning comes to an end here. Hope you enjoyed it!</p>
</article></div>||||I|||| Toggle navigation Runzhe Yang
 * About Me
 * Research
 Papers Talks
 * Music

 How Does Value-Based Reinforcement Learning Find the Optimal Policy?

 A General Explanation from Topological Point of View

 Posted on October 4, 2017

 DeepMind researchers claimed state-of-the-art results in the Arcade Learning Environment in their recent ICML paper “A Distributional Perspective on Reinforcement Learning”. They investigate a distributional perspective of the value function in the reinforcement learning context and further design an algorithm applying Bellman’s equation to approximately learn value distributions, which results in better policy optimization. The discussion in their paper follows closely the classical analysis of Bellman operator and dynamic programming, which should be familiar to most AI researchers.

 In this article, I will introduce the mathematical background of the classical analysis of reinforcement learning, that is, the Bellman operator is essentially a contraction mapping on a complete metric space, and explain how value-based reinforcement learning, e.g. Q-learning, finds the optimal policy. Based on these general ideas, readers will see how straightforward the idea that the DeepMind paper comes up with, and hopefully draw some inspiration from this topological perspective to design future RL algorithms.

 For the reader who prefers to systematically learn from books, I strongly recommend you to read the textbook Linear Operator Theory in Engineering and Science for acquiring mathematical background, and the book Abstract Dynamic Programming by Dimitri P. Bertsekas to study the connection between contraction mapping and dynamic programming. If you don’t remember Markov decision processes or policy optimization in reinforcement learning, my previous blog has introduced them briefly.

 * Topology of Metric Spaces
 + Cauchy Sequences and Completeness
 + Contraction Mapping Theorem
 * Contractive Dynamic Programming
 + Contraction Property
 + Monotonicity Property
 * Value-Based Reinforcement Learning
 + Approximate Value Iteration
 + Approximate Policy Iteration
 + Q-Learning: Asynchronous Value Iteration
 * A Methodology for Developing Iterative RL Algorithms
 + Learning from Value Distribution
 + Multi-Objective RL
 * Summary: Contraction, Value Iteration, and More

 Topology of Metric Spaces

 Literally, the word “topology” means “the study of spatial properties”. The space here could be any abstract space with certain properties. A natural way for us to quickly start the discussion is to observe a metric space.

 Definition 1. (Metric Space) A metric space is an ordered pair \((X, d)\) consists of an underlying set \(X\) and a real-valued function \(d(x, y)\), called metric, defined for \(x, y \in X\) such that for any \(x, y, z \in X\) the following conditions are satisfied:

 1. \(d(x, y) \geq 0\) [non-negativity]
 2. \(d(x, y) = 0 \Leftrightarrow x = y\) [identity of indiscernibles]
 3. \(d(x, y) = d(y, x)\) [symmetry]
 4. \(d(x, y) \leq d(x, z) + d(z, y)\) [triangle inequality]

 All these four conditions are in harmony with our intuition of distance. Indeed, the Euclidean distance \(d(x, y) = \sqrt{(x_1 - y_1)^2 + (x_2 - y_2)^2 + (x_3 - y_3)^2}\) is a valid metric.

 Cauchy Sequences and Completeness

 When we construct a metric space \((X, d)\), it is common to expect our metric space has no “point missing” from it. For example, the space \(\mathbb Q\) of rational numbers, with the metric by the absolute value \(|\cdot|\) of the difference, is not complete. Just consider the rational sequence \(x_{n+1} = \frac{x_n}{2} + \frac{1}{x_n}\), \(x_1 = 1\), whose elements become arbitrarily close to each other as the sequence progresses, but it doesn’t converge to any point in space \(\mathbb Q\). Therefore, the space \(\mathbb Q\) is incomplete, since it obviously misses points.

 A sequence like that in our example is called Cauchy sequence. Formally, we define the Cauchy sequence in and the completeness of a metric space \((X, d)\) as follows:

 Definition 2. (Cauchy Sequence) A sequence \(\{x_n\}\) is said to be Cauchy sequence if for each \(\epsilon > 0\) there exists an \(N_\epsilon\) such that \(d(x_n, x_m) \leq \epsilon\) for any \(n, m \geq N_\epsilon\).

 Definition 3. (Complete Metric Space) A metric space is said to be complete if each Cauchy sequence is a convergent sequence in it.

 The significance of complete metric space is difficult to be overemphasized. In many application, it is easier to show a given sequence is a Cauchy sequence than to show it is convergent. However, if the underlying metric space is complete, to show a given sequence is a Cauchy sequence is sufficient for convergence.

 Contraction Mapping Theorem

 Now we introduce the contraction mapping based on the concept of completeness.

 Definition 4. (Contraction) Let (X, d) be a metric space and \(f: X \rightarrow X\). We say that \(f\) is a contraction, or a contraction mapping, if there is a real number \(k \in [0, 1)\), such that \[ d(f(x), f(y)) \leq kd(x,y) \] for all x and y in X, where the term \(k\) is called a Lipschitz coefficent for \(f\).

 The reason that contractions play an important role in dynamic programming problems lies in the following Banach fixed-point theorem.

 Theorem 1. (Contraction Mapping Theorem) Let \((X, d)\) be a complete metric space and let \(f: X \rightarrow X\) be a contraction. Then there is one and only one fixed point \(x^*\) such that \[ f(x^* ) = x^* . \] Moreover, if \(x\) is any point in \(X\) and \(f^n(x)\) is inductively defined by \(f^2(x) = f(f(x))\), \(f^3(x) = f(f^2(x)), \dots, f^n(x) = f(f^{n-1}(x))\), then \(f^n(x) \rightarrow x^*\) as \(n \rightarrow \infty\).

 proof. Let us first prove every sequence \(\{f^n(x)\}\) convergent to a fixed point. Since the metric space \((X, d)\) is complete, we only need to prove every \(\{f^n(x)\}\) is a Cauchy sequence. By the triangle inequality and symmetry of metric, for all \(x, y \in X\),

 \[\begin{eqnarray} d(x, y) &\leq & d(x, f(x)) + d(f(x), f(y)) + d(f(y), y)\\ &\leq & d(f(x), x) + kd(x, y) + d(f(y), y)\\ \Rightarrow d(x, y) &\leq& [d(f(x), x) + d(f(y), y)]/(1-k) \end{eqnarray}\]

 Consider two points \(f^\ell(x)\), \(f^m(x)\) in the sequence. Their distance is bounded by

 \[\begin{eqnarray} d(f^\ell(x), f^m(x)) &\leq& [d(f^{\ell+1}(x), f^{\ell}(x)) + d(f^{m+1}(x), f^m(x))]/(1-k)\\ &\leq & [k^\ell d(f(x), x) + k^m d(f(x), x)]/(1-k)\\ &= & \frac{k^\ell + k^m}{1-k}d(f(x), x)\\ \end{eqnarray}\]

 since \(k \in [0, 1)\) the distance converges to zero as \(\ell, m \rightarrow \infty\), proving that \(\{f^n(x)\}\) is a Cauchy sequence. Because the space \((X, d)\) is complete, \(\{f^n(x)\}\) converges.

 We now show that \(x^* = \lim_{m\rightarrow \infty} f^m(x)\) is a fixed point, and it is unique. Since \(f\) is a contration, by definition it is continous. Therefore \[ f(x^* ) = f(\lim_{m\rightarrow \infty} f^m(x)) = \lim_{m\rightarrow \infty} f(f^m(x)) = \lim_{m\rightarrow \infty} f^m(x) = x^* . \] The uniqueness of \(x^*\) can be proved by contradiction. Assume \(x^*\) and \(y^*\) are two distinct fixed points of \(f\). We then get the contradiction \[ 0 < d(x^* , y^* ) = d(f(x^* ) , f(y^* )) \leq k d(x^* , y^* ) < d(x^* , y^* ). \] Hence \(f\) has only one fixed point and \(\{f^n(x)\}\) converges to it for any \(x \in X\). \(\blacksquare\)

 Contractive Dynamic Programming

 The contractive dynamic programming model is introduced by Eric V. Denardo in 1967, which applies contraction theorem to a broad class of optimization problems. It captures two widely shared properties of the optimization problem, contraction and monotonicity properties to develop a high level abstraction of optimization methods.

 Some terminology is now introduced. Let \(S\), \(A\) be two sets, which we refer to as a set of “states” and a set of “actions”, respectively. For each \(s \in S\), let \(A_s \subseteq A\) be the nonempty subset of actions that are feasible at state \(s\). We denote by \(\Pi\) the set of all stationary deterministic policies \(\pi: S \rightarrow A\), with \(\pi(s) \in A_s\), for all \(s \in S\).

 To introduce return function, let \(V\) be the set of all bouned real valued functions from \(S\) to reals, and let \(H: S\times A \times V \rightarrow \mathbb R\) be a given mapping. One might think \(H(s, a, v)\) as the total payoff for starting at state \(s\) and choosing action \(a\) with the prospect of receiving \(v(s')\) if the transition \((s, a)\rightarrow s'\) is happened. We consider the operator \(\mathcal T_\pi: V \rightarrow V\) defined by \[ (\mathcal T_\pi v)(s) := H(s, \pi(s), v), ~\forall s\in S, v\in V, \] and we also consier the operator \(\mathcal T\) defined by \[ (\mathcal T v)(s) := \sup_{a \in A_s}H(s, a, v), ~\forall s\in S, v\in V. \] We call \(\mathcal T_\pi\) the “evaluation operator” and \(\mathcal T\) the “optimality operator”. Our goal is to find a policy maximizing our total payoff. Following two properties provide conditions under which our goal can be realized by iteratively applying these two operators.

 Contraction Property

 Let a metric \(d\) on \(V\) is defined by \(d(u, v) = \sup_{s \in S} |u(s) - v(s)|\). Using the fact \(V\) is a space of bounded functions, it is easy to show that the metric space \((V, d)\) is complete.

 Assumption 1. (Contraction) For all \(u, v \in V\) and \(\pi \in \Pi\), for some \(\gamma \in [0, 1)\), we have \[ d(\mathcal T_{\pi} u, \mathcal T_{\pi} v) \leq \gamma d(u ,v). \]

 Proposition 1. Suppose the contraction assumption is satisfied. Then \(\mathcal T\) is also a contraction with the same Lipschitz coefficient \(\gamma\).

 Proof. Consider arbitrary \(u, v\) and we can write \((\mathcal T_\pi u) (s) \leq (\mathcal T_\pi v)(s) + \gamma d(u, v)\) for each \(\pi \in \Pi\) by assumption. For any fixed state \(s\), we take supremum of both side over \(\pi \in \Pi\), we have \[ \sup_{\pi\in\Pi} \left\{(\mathcal T_\pi u) (s)\right\} = \sup_{\pi(s) \in A_s}H(s, \pi(s), v) = (\mathcal T u) (s) \leq (\mathcal T v)(s) + \gamma d(u, v) \] Reversing the rolee of \(u\) and \(v\), we have \((\mathcal T v) (s) \leq (\mathcal T u)(s) + \gamma d(u, v)\) for any fixed state \(s\). Jointly, we have proved \(|(\mathcal T u)(s) - (\mathcal T v) (s)| \leq \gamma d(u, v)\) for all \(s\in S\). Hence, \[ d(\mathcal T u, \mathcal T v) = \sup_{s\in S} |(\mathcal T u)(s), (\mathcal T v)(s)| \leq \gamma d(u, v). \] The optimality operator \(\mathcal T\) is also a contraction with Lipschitz coefficient \(\gamma\). \(\blacksquare\)

 The Banach fixed-point theorem guarantees that iteratively applying evaluation operator \(\mathcal T_\pi\) to any function \(v \in V\) will converge to a unique function \(v_\pi\) in \(V\) such that \[ v_\pi(s) = H(s, \pi(s), v_\pi), ~~ \text{for each}~ s \in S. \] The function \(v_\pi\) is called value fuction of the policy \(\pi\). The optimal value function \(v_{\mathtt{opt}}\) is defined by \(v_{\mathtt{opt}}(s) := \sup_{\pi\in \Pi} v_{\pi}(s)\) for all \(s\in S\). Similarly, iteratively applying optimality operator \(\mathcal T\) to any funciton \(v \in V\) will converge to a unique function \(v^* \in V\) such that \[ v^* (s) = \sup_{a\in A_s}H(s, \pi(s), v^* ), ~~\text{for each}~ s \in S. \] How should we interpretate this fixed point \(v^*\)? When does \(v^*\) equal \(v_{\mathtt{opt}}\) that we desire? One may have a sense that \(v_{\mathtt{opt}} \geq v^*\) since the following result aways holds.

 Lemma 1. (\(v_{\mathtt{opt}} \geq v^*\)) For any \(\epsilon > 0\), there exists a policy \(\pi_\epsilon\) such that for any \(s\), \(H(s, \pi_\epsilon(s), v^* ) \geq \sup_{a\in A_s} H(s, a, v^* ) - \epsilon (1-\gamma)\), and any such \(\pi_\epsilon\) satisfies \(v_{\pi_\epsilon} \geq v^* - \epsilon\).

 Proof. Note that for any \(v \in V\) and \(\pi\in\Pi\), we have an upper bound

 \[\begin{eqnarray} d(v_{\pi}, v) & = & \lim_{n\rightarrow \infty}d(\mathcal T_{\pi}^{n} v, v)\\ & \leq & \sum_{\ell = 1}^{\infty}d(\mathcal T_{\pi}^{\ell} v, \mathcal T_{\pi}^{\ell-1} v) ~~~~\text{[triangle inequality]}\\ & \leq & \sum_{\ell = 1}^{\infty} \gamma^{\ell - 1} d(\mathcal T_{\pi} v, v)~~~~~~~\text{[contraction assumption]}\\ & = & \frac{1}{1 - \gamma} d(\mathcal T_\pi v, v). \end{eqnarray}\]

 Now let \(v = v^*\), and \(\pi = \pi_{\epsilon}\) such that \(H(s, \pi_\epsilon(s), v^* ) \geq \sup_{a\in A_s} H(s, a, v^* ) - \epsilon (1-\gamma)\) for any \(s\), we have \(d(v_{\pi_{\epsilon}}, v^* ) \leq \epsilon\). \(\blacksquare\)

 If we can show the reverse inequality, \(v^* \geq v_{\mathtt{opt}}\) , then we conclude \(v^* = v_{\mathtt{opt}}\). However, it does not always hold true. A sufficient condition, monotonicity property, is therefore introduced to make them identical.

 Monotonicity Property

 Assumption 2. (Monotonicity) For \(u, v \in V\), if \(u \geq v\), then \(\mathcal T_\pi u \geq \mathcal T_\pi v\) for each \(\pi\in \Pi\), where we write \(u \geq v\) if \(u(s) \geq v(s)\) for each state \(s\), and \(u > v\) if \(u \geq v\) and \(u \neq v\).

 The monotonicity property was introduced by L. G. Mitten in 1964. There are many value functions, including the one in the infinite horizon MDP, have the monotonicity property. The following theorem shows why the contraction and monotonicity properties are important in the contractive dynamic programming model.

 Theorem 2. Suppose the contraction and monotonicity properties hold. Then \(v^* = v_{\mathtt{opt}}\).

 Proof. Note that \(H(s, \pi(s), v^* ) \leq \sup_{a \in A_s} H(s, a, v^* )\) for any \(s\in S\) and \(\pi \in \Pi\). It means \(\mathcal T_{\pi} v^* \leq \mathcal T v^*\) for all \(\pi\). Repeatedly applying the monotonicity property \(k\) times, we derive \(\mathcal T_{\pi}^k v^* \leq \cdots \leq \mathcal T_{\pi}^2 v^* \leq \mathcal T_{\pi} v^* \leq \mathcal T v^* = v^*\). By contraction property \(\lim_{k\rightarrow \infty}\mathcal T_{\pi}^k v^* = v_{\pi}\). Therefore \(v_{\pi} \leq v^*\) for any \(\pi\) and \(v_{\mathtt{opt}} \leq v^*\). On the other hand, by Lemma 1 which utilizes the contraction property we have \(v^* - \epsilon \leq v_{\pi_{\epsilon}} \leq v_{\mathtt{opt}}\) for any \(\epsilon > 0\). Jointly, \[ v^* - \epsilon \leq v_{\mathtt{opt}} \leq v^* \] for any \(\epsilon > 0\). This implies \(v^* = v_{\mathtt{opt}}\) and completes the proof. \(\blacksquare\)

 Value-Based Reinforcement Learning

 Reinforcement learning (RL) approximates the optimal policy of a Markov decision process. This method succeeded in solving many challenging problems in diverse fields, such as robostics, transportation, medicine, and finance. Many RL methods are considered by the classic textbook Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto (We are translating its latest version into Chinese!).

 Roughly speaking, the value (function) based reinforcement learning is a large category of RL methods which take advantage of the Bellman equation and approximate the value function to find the optimal policy, such as SARSA and Q-Learning. By contrast, the policy-based reinforcement learning methods directly optimize the quantity of total reward, e.g. REINFORCE, which are more stable but less sample efficient than value-based methods.

 The key idea of value-based RL relates two Bellman equations of an infinite horizon discounted MDP \(\mathcal M = \langle S, A, \mathcal R, \mathcal P, \gamma \rangle\), where \(\mathcal R(s, a)\) is the immediate reward function, and \(\mathcal P(s'|s, a)\) is the probabilistic transition model, and \(\gamma\) is the discount factor (If you just forget the definition of MDPs, please refer to my previous blog). The first equation is so-called Bellman expectation equation: \[ v_{\pi}(s) = R(s, \pi(s)) + \gamma \sum_{s’ \in S} \mathcal P(s’ | s, \pi(s))\cdot v_{\pi}(s’). \] This equation describe the expected total reward for taking the action prescribed by policy \(\pi\). The other equation is called Bellman optimality equation: \[ v^* (s) = \max_{a\in A} \left\{R(s, a) + \gamma \sum_{s’ \in S} \mathcal P(s’ | s, a)\cdot v^* (s’)\right\}. \]

 Now we consider two corresponding operators, defined by the following \[ (\mathcal T_{\pi}v)(s) := R(s, \pi(s)) + \gamma \sum_{s’ \in S} \mathcal P(s’ | s, \pi(s))\cdot v(s’) ~~~\text{[Bellman evaluation operator]}, \] and \[ (\mathcal Tv)(s) := \max_{a\in A} \left\{ R(s, a) + \gamma \sum_{s’ \in S} \mathcal P(s’ | s, a)\cdot v(s’) \right\} \text{[Bellman optimality operator]} \] for any value function \(v \in V\). Note that \(v_{\pi}\) and \(v^*\) are fixed points of operator \(\mathcal T_{\pi}\) and \(\mathcal T\) respectively. If we can show that \(\mathcal T_{\pi}\) satisfies both contraction and monotonicity properties, according to results we have drawn in the last section, \(v^\pi\) and \(v^*\) are unique fixed points that repeatedly applying \(\mathcal T_{\pi}\) and \(\mathcal T\) will converge to respectively, and \(v^* = v_{\mathtt{opt}}\).

 Proposition 2. The contraction and monotonicity properties hold for Bellman evaluation operator \(\mathcal T_{\pi}\) for any stationary policy \(\pi \in \Pi\).

 Proof. For any value functions \(u, v \in V\), and any policy \(\pi \in \Pi\), we have

 \[\begin{eqnarray} d(\mathcal T_{\pi} u, \mathcal T_{\pi} v) & = & \sup_{s \in S} \left|\gamma \sum_{s' \in S} \mathcal P(s' | s, \pi(s))\cdot (u(s') - v(s'))\right|\\ & \leq & \sup_{s \in S} \gamma \sum_{s' \in S} \mathcal P(s' | s, \pi(s))\cdot \left|u(s') - v(s')\right|\\ & \leq & \sup_{s \in S} \gamma \sup_{s' \in S} \left|u(s') - v(s')\right|\\ & = & \gamma d(u ,v) \end{eqnarray}\]

 Therefore \(\mathcal T_{\pi}\) is a contraction with Lipschitz coefficient \(\gamma\). The contraction property holds. According to Proposition 1, the optimality operator \(\mathcal T\) is also an \(\gamma\)-contraction.

 Further more, for any value functions \(u \geq v\), we have \[ (\mathcal T_{\pi} u)(s) - (\mathcal T_{\pi} v)(s) = \gamma \sum_{s’ \in S} \mathcal P(s’ | s, \pi(s))\cdot (u(s’) - v(s’)) \geq 0 \] for all \(s \in S\). Hence \(\mathcal T u \geq \mathcal T v\). The monotonicity property holds. \(\blacksquare\)

 Approximate Value Iteration

 As the discussion above indicates, if the model is known, one can design an policy optimization algorithm by generating \(\mathcal T v, \mathcal T^2 v, \dots\) which converges to \(v^*\) and then extracting the corresponding optimal policy. This is exactly the value iteration (VI) method. This algorithm simply starts with any value function guess and repeatedly apply the Bellman optimality operator on it to obtain the optimal value function. Sometimes VI method may be implementable only by approximation, e.g., when the state space \(S\) is extremely large or when the value function is updated by some constant step size. A common solution is depicted as the following approximate value iteration algorithm.

 # Approximate Value Iteration 
 Initialize value function v arbitrarily.
 Repeat:
 Update v, such that sup|v-T(v)| ≤ δ
 until v converges
 Extract a policy: π, such that sup|T_π(v) - T(v)| ≤ ϵ
 

 By applying the triangle inequality and the contraction property of \(\mathcal T_{\pi}\) and \(\mathcal T\), we can assert the error bounds for the output value function \(v_{\mathtt{final}}\), which is bounded by \(d(v_{\mathtt{final}}, v_{\mathtt{opt}} ) \leq \frac{\delta}{1 - \gamma}\), and the extracted policy \(\pi_{\mathtt{final}}\), whose value is bounded by \(d(v_{\pi_{\mathtt{final}}}, v_{\mathtt{opt}} ) \leq \frac{\epsilon}{1 - \gamma} + \frac{2\gamma\delta}{(1-\gamma)^2}\) (proof is straightforward, check them yourself :-)). The VI algorithm converges exponentially when \(\epsilon = 0\) and \(\delta = 0\).

 Approximate Policy Iteration

 The VI method is a direct creation based on the fact that the Bellman optimality operator \(\mathcal T\) is a contraction. However, people may think by involving policy evaluation each step one can have a better rate of convergence and a better approximation of optimal policy, though it will incur more calculation each iteration. This is plausible, but not insignificantly so in theory. The approximate policy iteration (PI) algorithm is stated as follows:

 # Approximate Policy Iteration
 Given the current policy π_k
 Repeat:
 Policy evaluation: Compute v such that
 sup|v - (T_π_k)^m_k (v) | ≤ δ
 by appying Bellman evaluation operator T_π
 
 Policy improvement: Extract a policy π_{k+1} that satisfies
 sup|T_π_{k+1}(v)- T(v)| ≤ ϵ
 π_k ← π_{k+1}
 until v converges.
 Extract π_{k+1} 
 

 In this algorithm, the parameter \(m_k\) is an positive intergers. Taken the usual choice of \(m_k = 1\), we can show that \(d(v_{\pi_{k+1}}, v_{\mathtt{opt}}) \leq \gamma d(v_{\pi_{k}}, v_{\mathtt{opt}}) + \frac{\epsilon + 2\delta\gamma}{1 - \gamma}\) for each iteration. Therefore, its final error bound is asymptotically proportional to \(1/(1-\gamma)^2\), which indicates PI does not hold an accuracy advantage over VI. And the exact PI converges at an exponential rate which is as same as that of exact VI.

 Q-Learning: Asynchronous Value Iteration

 In the last two section we explained how we do classical model based planing for MDPs from the topological point of view. Many readers by now have a sense that how this stuff and value-based reinforcement learning are connected. But there remains a critical question: Can this explanation is still suitable for those general model free cases? Can we explain those classical reinforcement “learning” algorithm, such as Q-learning, in an analog way?

 Intuitively, the answer should be “yes”. Consider the evaluation operator \(\mathcal T_{\pi}\) and optimality operator \(\mathcal T\) defined by \[ (\mathcal T_{\pi} q)(s, a) := R(s, a) + \gamma\mathbb E_{\mathcal P, \pi} q(s’, a’) \] and \[ (\mathcal T q)(s, a) := R(s, a) + \gamma\mathbb E_{\mathcal P} \max_{a’\in A} q(s’, a’) \] respectively, over the Q value function space \(\mathbb R^{S\times A}\), with the similar metric as that of \(V\). Following the similar analysis, the contraction and the monotonicity properties hold for them. However, since the model is unknown, we cannot update the Q value function as that in value iteration. We can only approximate and update the Q value for some certain pair \((s, a)\) by sampling or temporal difference method each iteration. Is its convergence still guaranteed? The following theorem resolves our doubts.

 Definition 5. (Asynchronous Value Iteration) Conside value function \(v\) as a composition of \((v_s)_ {s\in S}\) such that in each iteration, \[ v_s^{t+1}(s’) = \begin{cases} \mathcal Tv^{t}(s’), & \text{if}~s’ = s\newline \mathcal v_s^{t}(s’), & \text{otherwise.} \end{cases} \] Theorem 3. (Asynchronous Convergence Theorem) If each restricted value function \(v_s\) can be update infinite times, and there is a sequence of nonempty subsets \(\{V^k\} \subseteq V\) with \(V^{k+1}\subseteq V^{k}\), \(k = 0, 1, \dots\) such that if \(\{v^k\}\) is any sequence with \(v_k\in V_k\) for all \(k \leq 0\), the \(\{v^k\}\) converges pointwise to \(v^*\). Assume further the following:

 1. Synchronous Convergence Condition: We have \[ \mathcal T v \in V^{k+1}, \forall v \in V^k \]
 2. Box Condition: For all \(k, V^k\) is a Cartesian product of the form \[ V^k = \times_{s\in S} V_s^k \]

 where \(V_{s}^k\) is a set of bounded real-valued functions on state \(s\). Then for every \(v^0 \in V^0\) the sequence \(\{v^t\}\) generated by the asynchronous VI algorithm converges to \(v^*\).

 Proof. To show that the algorithm converges is equivalent to show that the iterates from \(V^k\) will get in to \(V^{k+1}\) eventually, i.e., for each \(k \leq 0\), there is a time \(t_k\) such that \(v^t\in V^{k}\) for all \(t \geq t_k\). We can prove it by mathematical induction.

 When \(k = 0\), the statement is true since \(v^0\in V^0\). Assuming the statement is true for a given \(k\), we will show there is a time \(t_{k+1}\) with the required properties. For each \(s\in S\), let set \(L_{s} = \{t: s^t = s\}\) record the time an update happend on the state \(s\). Let \(t(s)\) be the first element in \(L_{s}\) such that \(t(s) \geq t_{k}\). Then by the synchrnous convergence condition, we have \(\mathcal Tv^{t(s)} \in V^{k+1}\) for all \(s\in S\). In the view of the box condition, it implies \(v^{t+1}(s)\in V_s^{k+1}\) for all \(t \geq t(s)\) for any \(s\in S\). Therefore, let \(t_{k+1} = \max_{s}t(s) + 1\), using the box condition, we have \(v^t \in V^{k+1}\) for all \(t \leq t_{k+1}\). By mathematical induction, the statement holds, and \(V^k\) will get in to \(V^{k+1}\) eventually. Hence, \(\{v^t\}\) converges to \(v^*\). \(\blacksquare\)

 A Methodology for Developing Iterative RL Algorithms

 It is really exciting to see that the convergence and correctness of value-based reinforcement learning algorithm, such as Q-learning, can be interpreted in a topological view. In fact, the contraction mapping theory provides us with a methodology for developing iterative value-based RL algorithm. In a high-level overview, iterative value-based RL algorithms are concerned with the following five essential elements:

 * Value Space \(V\): A common choice of \(V\) is the space of value functions in \(\mathbb R^S\) or the space of Q-value functions in \(\mathbb R^{S\times A}\). There are many other choices such as the space of ordered pair of \((V,Q)\) (Example 2.6.2 in Dimitri’s book), or the space of multi-dimensional value functions.

 * Value Metric \(d\): It defines the “distance” between two points in the value space. Besides the basic four conditions, the metric \(d\) should ensure a complete metric space \((V, d)\) to validate the Banach’s fixed-point theorem. A compatible selection of value metric will make our analysis much easier.

 * Evaluation Operator \(\mathcal T_{\pi}\): We have constructed a recursive expression of a certain value point in the value space associated with some policy, e.g., the Bellman expectation equation, \[ v_{\pi}(\cdot) = H(\cdot, \pi(\cdot), v_{\pi}) \] to depict the value of a policy \(v_{\pi}\) as a fixed point we desire. Carefully verify that the contraction and the monotonicity properties hold for \(\mathcal T_{\pi} v:=H(\cdot, \pi(\cdot), v)\).

 * Optimality Operator \(\mathcal T\): Write down recursive expression of the optimal point in the value space, e.g., the Bellman optimality equation, \[ v^* (\cdot) = \sup_{\pi\in \Pi}\mathcal T_{\pi} v^* (\cdot). \] One thing important is to make sure the contraction property holds for the optimality operator \(\mathcal T:= \sup_{\pi\in \Pi}\mathcal T_{\pi}v\). Note that when the metric \(d\) is the supremum of the absolute value of the difference, the contraction property of \(\mathcal T\) is automatically satisfied according to Proposition 1.

 * Updating Scheme: It is the most tricky part. For those small-scale problems, the asynchronous value iteration is enough. However, to make our reinforcement learning algorithm practical and scalable, we have to consider much more factors. For example: Should our algorithm be on-policy or off-policy? How do we approximate the value and policies? If it is an online algorithm, how do we trade off exploration and exploitation? All these details will significantly influence the performance of our algorithm on real-world tasks.

 Learning from Value Distribution

 Let’s get back to the DeepMind paper mentioned at the begining, which proposes a new reinforcement learning algorithm from a distributional perspective. This work is actually a good example of developing and analyzing a new reinforcement learning algorithm using the methodology that we summarize above.

 In this example, the value space consists of all the “value distribution” \(Z(s, a)\). The value metric of the value space is defined by \(d(Z_1, Z_2):= \sup_{s, a}d_{w}(Z_1(s,a), Z_2(s,a))\), where \(d_{w}\) is the Wasserstein distance between two random variables. The evaluation operator is defined by \[ \mathcal T_{\pi} Z(s, a) \stackrel{D}{:=} R(s, a) + \gamma P^{\pi}Z(s, a), \] which is proved to be a contraction in \((\mathcal Z, d)\), where “\(\stackrel{D}{=}\)” means random variables on both sides are drawn from the same distribution. The monotonicity property also holds as they define the order as the order of the magnitude of expectation (\(Z_1 \leq Z_2 \Leftrightarrow \mathbb E Z_1 \leq \mathbb E Z_2\)). The optimality operator is defined by \(\mathcal T Z := \sup_{\pi}\mathcal T_{\pi}Z\). However, since the relative order is incompatible with the metric, this optimality operator is NOT a contraction in \((\mathcal Z, d)\). They use approximate asynchronous value iteration as the updating scheme, where the value distribution function is approximated by a neural network which outputs a discrete distribution between \([V_\text{MAX}, V_\text{MIN}]\).

 If we only consider the space of the first-order moment of the value distribution and the metric \(d_{\mathbb E}(\mathbb E Z_1, \mathbb E Z_2):= \sup_{s, a} |\mathbb E Z_1(s,a) - \mathbb E Z_2(s,a)|\), both \(\mathbb E\mathcal T_{\pi}\) and \(\mathbb E\mathcal T\) are contractions. And this explains why the C51 algorithm works well. However, as for predicting value distribution, I doubt the correctness of directly using the result of value iteration. In my opinion, a policy iteration algorithm would be a better choice here for maintaining value distributions, or we should correct the value distribution function by repeatedly applying evaluation operator after the optimal policy is found.

 Multi-Objective RL

 Another example which might be helpful for us to understand this methodology is the multi-objective reinforcement learning. And this is also an interesting topic I am recently working on. The value space of this problem should be some multi-dimensional value function. The metric can be extended from that of a normed vector space. If we only want to find one policy resulting a Pareto optimal return, or simply want to find the best policy maximizing the return associated with specific weights, the evaluation operator and the optimality operator are very similar to \(R + \gamma P^{\pi}V\) and \(\sup_{\pi} \{R + \gamma P^{\pi}V\}\) (the \(\sup\{\cdot\}\) is defined by Pareto domination and the order of magnitude of weighted sum, respectively). Designing an updating scheme according to the real problem, we are guaranteed to have a viable algorithm. If we want to find all the policies resulting in a Pareto optimal return, or make the policies controllable, how to represent a collection of policies and write down the corresponding evaluation/optimality operator are very challenging.

 Summary: Contraction, Value Iteration, and More

 In this article, some topological backgrounds of metric spaces and the Banach fixed-point theorem are first introduced. Based on these mathematical foundations, we investigate the contractive dynamic programming, where the contraction property and the monotonicity property together guarantee a unique optimal solution can be achieved exponentially fast. Bring these ideas to the reinforcement learning context, we can explain and analyze value iteration, policy iteration and Q-learning algorithms in a unified framework. It further inspires us to develope a methodology for developing iterative value-based reinforcement algorithm, which requires us to consider value space, value metric, evaluation operator, optimality operator and updating scheme.

 The next time someone asks, “how does a value-based RL algorithm converge?”, or “how do people come up with a new RL algorithm?”, we can answer him not first by complex, detailed mathematical deduction, but simply by drawing an intuitive picture and analyzing on those accessible abstractions! Our journey on the topic of topological explanation of value-based reinforcement learning comes to an end here. Hope you enjoyed it!

 Banach Fixed-Point Theorem, Reinforcement Learning, Paper Notes
 Share: Twitter Facebook Google+ LinkedIn
 * ← Previous Post
 * Next Post →
 Please enable JavaScript to view the comments powered by Disqus.
 * GitHub
 * Twitter
 * Email me
 * LinkedIn
 * RSS

 Runzhe Yang • 2024 • Runzhe-Yang.Science

 Theme by beautiful-jekyll

DeepMind researchers claimed state-of-the-art results in the Arcade Learning Environment in their recent ICML paper “  A Distributional Perspective on Reinforcement Learning  ”. They investigate a distributional perspective of the value function in the reinforcement learning context and further design an algorithm applying Bellman’s equation to approximately learn value distributions, which results in better policy optimization. The discussion in their paper follows closely the classical analysis of Bellman operator and dynamic programming, which should be familiar to most AI researchers.
In this article, I will introduce the mathematical background of the classical analysis of reinforcement learning, that is, the Bellman operator is essentially a contraction mapping on a complete metric space, and explain how value-based reinforcement learning, e.g. Q-learning, finds the optimal policy. Based on these general ideas, readers will see how straightforward the idea that the DeepMind paper comes up with, and hopefully draw some inspiration from this topological perspective to design future RL algorithms.
For the reader who prefers to systematically learn from books, I strongly recommend you to read the textbook   Linear Operator Theory in Engineering and Science   for acquiring mathematical background, and the book   Abstract Dynamic Programming   by  Dimitri P. Bertsekas  to study the connection between contraction mapping and dynamic programming. If you don’t remember Markov decision processes or policy optimization in reinforcement learning,  my previous blog  has introduced them briefly.
 Topology of Metric Spaces 
 Cauchy Sequences and Completeness 
 Contraction Mapping Theorem 
 Contractive Dynamic Programming 
 Contraction Property 
 Monotonicity Property 
 Value-Based Reinforcement Learning 
 Approximate Value Iteration 
 Approximate Policy Iteration 
 Q-Learning: Asynchronous Value Iteration 
 A Methodology for Developing Iterative RL Algorithms 
 Learning from Value Distribution 
 Multi-Objective RL 
 Summary: Contraction, Value Iteration, and More 
Topology of Metric Spaces
Literally, the word “topology” means “the study of spatial properties”. The space here could be any abstract space with certain properties. A natural way for us to quickly start the discussion is to observe a metric space.
 Definition 1. (Metric Space) A metric space is an ordered pair \((X, d)\) consists of an underlying set \(X\) and a real-valued function \(d(x, y)\), called metric, defined for \(x, y \in X\) such that for any \(x, y, z \in X\) the following conditions are satisfied:
\(d(x, y) \geq 0\)                          [non-negativity]
\(d(x, y) = 0 \Leftrightarrow x = y\)         [identity of indiscernibles]
\(d(x, y) = d(y, x)\)                 [symmetry]
\(d(x, y) \leq d(x, z) + d(z, y)\) [triangle inequality]
All these four conditions are in harmony with our intuition of distance. Indeed, the Euclidean distance 
\(d(x, y) = \sqrt{(x_1 - y_1)^2 + (x_2 - y_2)^2 + (x_3 - y_3)^2}\) 
is a valid metric.
Cauchy Sequences and Completeness
When we construct a metric space \((X, d)\), it is common to expect our metric space has no “point missing” from it. For example, the space \(\mathbb Q\) of rational numbers, with the metric by the absolute value \(|\cdot|\) of the difference, is not complete. Just consider the rational sequence \(x_{n+1} = \frac{x_n}{2} + \frac{1}{x_n}\), \(x_1 = 1\), whose elements become arbitrarily close to each other as the sequence progresses, but it doesn’t converge to any point in space \(\mathbb Q\). Therefore, the space \(\mathbb Q\) is incomplete, since it obviously misses points.
A sequence like that in our example is called Cauchy sequence. Formally, we define the Cauchy sequence in and the completeness of a metric space \((X, d)\) as follows:
 Definition 2. (Cauchy Sequence) A sequence \(\{x_n\}\) is said to be Cauchy sequence if for each \(\epsilon &gt; 0\) there exists an \(N_\epsilon\) such that \(d(x_n, x_m) \leq \epsilon\) for any \(n, m \geq N_\epsilon\).
 Definition 3. (Complete Metric Space) A metric space is said to be complete if each Cauchy sequence is a convergent sequence in it.
The significance of complete metric space is difficult to be overemphasized. In many application, it is easier to show a given sequence is a Cauchy sequence than to show it is convergent. However, if the underlying metric space is complete, to show a given sequence is a Cauchy sequence is sufficient for convergence.
Contraction Mapping Theorem
Now we introduce the contraction mapping based on the concept of completeness.
 Definition 4. (Contraction) Let (X, d) be a metric space and \(f: X \rightarrow X\). We say that \(f\) is a contraction, or a contraction mapping, if there is a real number \(k \in [0, 1)\), such that
\[
 d(f(x), f(y)) \leq kd(x,y)
\]
for all x and y in X, where the term \(k\) is called a Lipschitz coefficent for \(f\).
The reason that contractions play an important role in dynamic programming problems lies in the following Banach fixed-point theorem.
 Theorem 1. (Contraction Mapping Theorem) Let \((X, d)\) be a complete metric space and let \(f: X \rightarrow X\) be a contraction. Then there is one and only one fixed point 
\(x^*\)
such that 
\[
 f(x^* ) = x^* .
\]
Moreover, if \(x\) is any point in \(X\) and \(f^n(x)\) is inductively defined by \(f^2(x) = f(f(x))\), \(f^3(x) = f(f^2(x)), \dots, f^n(x) = f(f^{n-1}(x))\), then \(f^n(x) \rightarrow x^*\) as \(n \rightarrow \infty\).
  proof.  Let us first prove every sequence \(\{f^n(x)\}\) convergent to a fixed point. Since the metric space \((X, d)\) is complete, we only need to prove every \(\{f^n(x)\}\) is a Cauchy sequence. By the triangle inequality and symmetry of metric, for all \(x, y \in X\),
\[\begin{eqnarray}
 d(x, y) &amp;\leq &amp; d(x, f(x)) + d(f(x), f(y)) + d(f(y), y)\\
 &amp;\leq &amp; d(f(x), x) + kd(x, y) + d(f(y), y)\\
 \Rightarrow 
 d(x, y) &amp;\leq&amp; [d(f(x), x) + d(f(y), y)]/(1-k)
 \end{eqnarray}\]
Consider two points \(f^\ell(x)\), \(f^m(x)\) in the sequence. Their distance is bounded by
\[\begin{eqnarray}
 d(f^\ell(x), f^m(x)) &amp;\leq&amp; [d(f^{\ell+1}(x), f^{\ell}(x)) + d(f^{m+1}(x), f^m(x))]/(1-k)\\
 &amp;\leq &amp; [k^\ell d(f(x), x) + k^m d(f(x), x)]/(1-k)\\
 &amp;= &amp; \frac{k^\ell + k^m}{1-k}d(f(x), x)\\
 \end{eqnarray}\]
since \(k \in [0, 1)\) the distance converges to zero as \(\ell, m \rightarrow \infty\), proving that \(\{f^n(x)\}\) is a Cauchy sequence. Because the space \((X, d)\) is complete, \(\{f^n(x)\}\) converges.
We now show that \(x^* = \lim_{m\rightarrow \infty} f^m(x)\) is a fixed point, and it is unique. Since \(f\) is a contration, by definition it is continous. Therefore 
\[
 f(x^* ) = f(\lim_{m\rightarrow \infty} f^m(x)) = \lim_{m\rightarrow \infty} f(f^m(x)) = \lim_{m\rightarrow \infty} f^m(x) = x^* .
\]
The uniqueness of \(x^*\) can be proved by contradiction. Assume \(x^*\) and \(y^*\) are two distinct fixed points of \(f\). We then get the contradiction
\[
 0 &lt; d(x^* , y^* ) = d(f(x^* ) , f(y^* )) \leq k d(x^* , y^* ) &lt; d(x^* , y^* ).
\]
Hence \(f\) has only one fixed point and \(\{f^n(x)\}\) converges to it for any \(x \in X\). \(\blacksquare\)
Contractive Dynamic Programming
The contractive dynamic programming model is introduced by Eric V. Denardo in 1967, which applies contraction theorem to a broad class of optimization problems. It captures two widely shared properties of the optimization problem, contraction and monotonicity properties to develop a high level abstraction of optimization methods.
Some terminology is now introduced. Let \(S\), \(A\) be two sets, which we refer to as a set of “states” and a set of “actions”, respectively. For each \(s \in S\), let \(A_s \subseteq A\) be the nonempty subset of actions that are feasible at state \(s\). We denote by \(\Pi\) the set of all stationary deterministic policies \(\pi: S \rightarrow A\), with \(\pi(s) \in A_s\), for all \(s \in S\).
To introduce return function, let \(V\) be the set of all bouned real valued functions from \(S\) to reals, and let \(H: S\times A \times V \rightarrow \mathbb R\) be a given mapping. One might think \(H(s, a, v)\) as the total payoff for starting at state \(s\) and choosing action \(a\) with the prospect of receiving \(v(s')\) if the transition \((s, a)\rightarrow s'\) is happened. We consider the operator \(\mathcal T_\pi: V \rightarrow V\) defined by 
\[
 (\mathcal T_\pi v)(s) := H(s, \pi(s), v), ~\forall s\in S, v\in V,
\]
and we also consier the operator \(\mathcal T\) defined by 
\[
 (\mathcal T v)(s) := \sup_{a \in A_s}H(s, a, v), ~\forall s\in S, v\in V.
\]
We call \(\mathcal T_\pi\) the “evaluation operator” and \(\mathcal T\) the “optimality operator”. Our goal is to find a policy maximizing our total payoff. Following two properties provide conditions under which our goal can be realized by iteratively applying these two operators.
Contraction Property
Let a metric \(d\) on \(V\) is defined by 
\(d(u, v) = \sup_{s \in S} |u(s) - v(s)|\). Using the fact \(V\) is a space of bounded functions, it is easy to show that the metric space \((V, d)\) is complete.
 Assumption 1. (Contraction) For all \(u, v \in V\) and \(\pi \in \Pi\), for some \(\gamma \in [0, 1)\), we have
\[
 d(\mathcal T_{\pi} u, \mathcal T_{\pi} v) \leq \gamma d(u ,v).
\]
 Proposition 1. Suppose the contraction assumption is satisfied. Then \(\mathcal T\) is also a contraction with the same Lipschitz coefficient \(\gamma\).
  Proof.  Consider arbitrary \(u, v\) and we can write \((\mathcal T_\pi u) (s) \leq (\mathcal T_\pi v)(s) + \gamma d(u, v)\) for each \(\pi \in \Pi\) by assumption. For any fixed state \(s\), we take supremum of both side over \(\pi \in \Pi\), we have
\[
 \sup_{\pi\in\Pi} \left\{(\mathcal T_\pi u) (s)\right\} = \sup_{\pi(s) \in A_s}H(s, \pi(s), v) = (\mathcal T u) (s) \leq (\mathcal T v)(s) + \gamma d(u, v)
\]
Reversing the rolee of \(u\) and \(v\), we have \((\mathcal T v) (s) \leq (\mathcal T u)(s) + \gamma d(u, v)\) for any fixed state \(s\). Jointly, we have proved \(|(\mathcal T u)(s) - (\mathcal T v) (s)| \leq \gamma d(u, v)\) for all \(s\in S\). Hence, 
\[
 d(\mathcal T u, \mathcal T v) = \sup_{s\in S} |(\mathcal T u)(s), (\mathcal T v)(s)| \leq \gamma d(u, v).
\]
The optimality operator \(\mathcal T\) is also a contraction with Lipschitz coefficient \(\gamma\). \(\blacksquare\)
The Banach fixed-point theorem guarantees that iteratively applying evaluation operator \(\mathcal T_\pi\) to any function \(v \in V\) will converge to a unique function \(v_\pi\) in \(V\) such that 
\[
 v_\pi(s) = H(s, \pi(s), v_\pi), ~~ \text{for each}~ s \in S.
\]
The function \(v_\pi\) is called value fuction of the policy \(\pi\). The optimal value function \(v_{\mathtt{opt}}\) is defined by \(v_{\mathtt{opt}}(s) := \sup_{\pi\in \Pi} v_{\pi}(s)\) for all \(s\in S\). Similarly, iteratively applying optimality operator \(\mathcal T\) to any funciton \(v \in V\) will converge to a unique function \(v^* \in V\) such that
\[
 v^* (s) = \sup_{a\in A_s}H(s, \pi(s), v^* ), ~~\text{for each}~ s \in S.
\]
How should we interpretate this fixed point \(v^*\)? When does \(v^*\) equal \(v_{\mathtt{opt}}\) that we desire? One may have a sense that \(v_{\mathtt{opt}} \geq v^*\) since the following result aways holds.
 Lemma 1. (\(v_{\mathtt{opt}} \geq v^*\)) For any \(\epsilon &gt; 0\), there exists a policy \(\pi_\epsilon\) such that for any \(s\), \(H(s, \pi_\epsilon(s), v^* ) \geq \sup_{a\in A_s} H(s, a, v^* ) - \epsilon (1-\gamma)\), and any such \(\pi_\epsilon\) satisfies \(v_{\pi_\epsilon} \geq v^* - \epsilon\).
  Proof.  Note that for any \(v \in V\) and \(\pi\in\Pi\), we have an upper bound
\[\begin{eqnarray}
 d(v_{\pi}, v) &amp; = &amp; \lim_{n\rightarrow \infty}d(\mathcal T_{\pi}^{n} v, v)\\
 &amp; \leq &amp; \sum_{\ell = 1}^{\infty}d(\mathcal T_{\pi}^{\ell} v, \mathcal T_{\pi}^{\ell-1} v) ~~~~\text{[triangle inequality]}\\
 &amp; \leq &amp; \sum_{\ell = 1}^{\infty} \gamma^{\ell - 1} d(\mathcal T_{\pi} v, v)~~~~~~~\text{[contraction assumption]}\\
 &amp; = &amp; \frac{1}{1 - \gamma} d(\mathcal T_\pi v, v).
 \end{eqnarray}\]
Now let \(v = v^*\), and \(\pi = \pi_{\epsilon}\) such that \(H(s, \pi_\epsilon(s), v^* ) \geq \sup_{a\in A_s} H(s, a, v^* ) - \epsilon (1-\gamma)\) for any \(s\), we have \(d(v_{\pi_{\epsilon}}, v^* ) \leq \epsilon\). \(\blacksquare\)
If we can show the reverse inequality, \(v^* \geq v_{\mathtt{opt}}\) , then we conclude \(v^* = v_{\mathtt{opt}}\). However, it does not always hold true. A sufficient condition, monotonicity property, is therefore introduced to make them identical.
Monotonicity Property
 Assumption 2. (Monotonicity) For \(u, v \in V\), if \(u \geq v\), then \(\mathcal T_\pi u \geq \mathcal T_\pi v\) for each \(\pi\in \Pi\), where we write \(u \geq v\) if \(u(s) \geq v(s)\) for each state \(s\), and \(u &gt; v\) if \(u \geq v\) and \(u \neq v\).
The monotonicity property was introduced by L. G. Mitten in 1964. There are many value functions, including the one in the infinite horizon MDP, have the monotonicity property. The following theorem shows why the contraction and monotonicity properties are important in the contractive dynamic programming model.
 Theorem 2. Suppose the contraction and monotonicity properties hold. Then \(v^* = v_{\mathtt{opt}}\).
  Proof.  Note that \(H(s, \pi(s), v^* ) \leq \sup_{a \in A_s} H(s, a, v^* )\) for any \(s\in S\) and \(\pi \in \Pi\). It means \(\mathcal T_{\pi} v^* \leq \mathcal T v^*\) for all \(\pi\). Repeatedly applying the monotonicity property \(k\) times, we derive \(\mathcal T_{\pi}^k v^* \leq \cdots \leq \mathcal T_{\pi}^2 v^* \leq \mathcal T_{\pi} v^* \leq \mathcal T v^* = v^*\). By contraction property \(\lim_{k\rightarrow \infty}\mathcal T_{\pi}^k v^* = v_{\pi}\). Therefore \(v_{\pi} \leq v^*\) for any \(\pi\) and \(v_{\mathtt{opt}} \leq v^*\). On the other hand, by Lemma 1 which utilizes the contraction property we have \(v^* - \epsilon \leq v_{\pi_{\epsilon}} \leq v_{\mathtt{opt}}\) for any \(\epsilon &gt; 0\). Jointly,
\[
 v^* - \epsilon \leq v_{\mathtt{opt}} \leq v^*
\]
for any \(\epsilon &gt; 0\). This implies \(v^* = v_{\mathtt{opt}}\) and completes the proof. \(\blacksquare\)
Value-Based Reinforcement Learning
Reinforcement learning (RL) approximates the optimal policy of a Markov decision process. This method succeeded in solving many challenging problems in diverse fields, such as robostics, transportation, medicine, and finance. Many RL methods are considered by the classic textbook   Reinforcement Learning: An Introduction   by Richard S. Sutton and Andrew G. Barto (We are translating its latest version into Chinese!).
Roughly speaking, the value (function) based reinforcement learning is a large category of RL methods which take advantage of the Bellman equation and approximate the value function to find the optimal policy, such as SARSA and Q-Learning. By contrast, the policy-based reinforcement learning methods directly optimize the quantity of total reward, e.g. REINFORCE, which are more stable but less sample efficient than value-based methods.
The key idea of value-based RL relates two Bellman equations of an infinite horizon discounted MDP \(\mathcal M = \langle S, A, \mathcal R, \mathcal P, \gamma \rangle\), where \(\mathcal R(s, a)\) is the immediate reward function, and \(\mathcal P(s'|s, a)\) is the probabilistic transition model, and \(\gamma\) is the discount factor (If you just forget the definition of MDPs, please refer to  my previous blog ). The first equation is so-called Bellman expectation equation:
\[
 v_{\pi}(s) = R(s, \pi(s)) + \gamma \sum_{s’ \in S} \mathcal P(s’ | s, \pi(s))\cdot v_{\pi}(s’).
\]
This equation describe the expected total reward for taking the action prescribed by policy \(\pi\). The other equation is called Bellman optimality equation:
\[
 v^* (s) = \max_{a\in A} \left\{R(s, a) + \gamma \sum_{s’ \in S} \mathcal P(s’ | s, a)\cdot v^* (s’)\right\}.
\]
Now we consider two corresponding operators, defined by the following
\[
 (\mathcal T_{\pi}v)(s) := R(s, \pi(s)) + \gamma \sum_{s’ \in S} \mathcal P(s’ | s, \pi(s))\cdot v(s’) ~~~\text{[Bellman evaluation operator]},
\]
and 
\[
 (\mathcal Tv)(s) := \max_{a\in A} \left\{ R(s, a) + \gamma \sum_{s’ \in S} \mathcal P(s’ | s, a)\cdot v(s’) \right\} \text{[Bellman optimality operator]}
\]
for any value function \(v \in V\). Note that \(v_{\pi}\) and \(v^*\) are fixed points of operator \(\mathcal T_{\pi}\) and \(\mathcal T\) respectively. If we can show that \(\mathcal T_{\pi}\) satisfies both contraction and monotonicity properties, according to results we have drawn in the last section, \(v^\pi\) and \(v^*\) are unique fixed points that repeatedly applying \(\mathcal T_{\pi}\) and \(\mathcal T\) will converge to respectively, and \(v^* = v_{\mathtt{opt}}\).
 Proposition 2. The contraction and monotonicity properties hold for Bellman evaluation operator \(\mathcal T_{\pi}\) for any stationary policy \(\pi \in \Pi\).
  Proof.  For any value functions \(u, v \in V\), and any policy \(\pi \in \Pi\), we have
\[\begin{eqnarray}
 d(\mathcal T_{\pi} u, \mathcal T_{\pi} v) &amp; = &amp; \sup_{s \in S} \left|\gamma \sum_{s' \in S} \mathcal P(s' | s, \pi(s))\cdot (u(s') - v(s'))\right|\\
 &amp; \leq &amp; \sup_{s \in S} \gamma \sum_{s' \in S} \mathcal P(s' | s, \pi(s))\cdot \left|u(s') - v(s')\right|\\
 &amp; \leq &amp; \sup_{s \in S} \gamma \sup_{s' \in S} \left|u(s') - v(s')\right|\\
 &amp; = &amp; \gamma d(u ,v) 
 \end{eqnarray}\]
Therefore \(\mathcal T_{\pi}\) is a contraction with Lipschitz coefficient \(\gamma\). The contraction property holds. According to Proposition 1, the optimality operator \(\mathcal T\) is also an \(\gamma\)-contraction.
Further more, for any value functions \(u \geq v\), we have
\[
 (\mathcal T_{\pi} u)(s) - (\mathcal T_{\pi} v)(s) = \gamma \sum_{s’ \in S} \mathcal P(s’ | s, \pi(s))\cdot (u(s’) - v(s’)) \geq 0
\]
for all \(s \in S\). Hence \(\mathcal T u \geq \mathcal T v\). The monotonicity property holds. \(\blacksquare\)
Approximate Value Iteration
As the discussion above indicates, if the model is known, one can design an policy optimization algorithm by generating \(\mathcal T v, \mathcal T^2 v, \dots\) which converges to \(v^*\) and then extracting the corresponding optimal policy. This is exactly the value iteration (VI) method. This algorithm simply starts with any value function guess and repeatedly apply the Bellman optimality operator on it to obtain the optimal value function. Sometimes VI method may be implementable only by approximation, e.g., when the state space \(S\) is extremely large or when the value function is updated by some constant step size. A common solution is depicted as the following approximate value iteration algorithm.
  # Approximate Value Iteration 
Initialize value function v arbitrarily.
Repeat:
 Update v, such that sup|v-T(v)| ≤ δ
until v converges
Extract a policy: π, such that sup|T_π(v) - T(v)| ≤ ϵ
  
By applying the triangle inequality and the contraction property of \(\mathcal T_{\pi}\) and \(\mathcal T\), we can assert the error bounds for the output value function \(v_{\mathtt{final}}\), which is bounded by \(d(v_{\mathtt{final}}, v_{\mathtt{opt}} ) \leq \frac{\delta}{1 - \gamma}\), and the extracted policy \(\pi_{\mathtt{final}}\), whose value is bounded by \(d(v_{\pi_{\mathtt{final}}}, v_{\mathtt{opt}} ) \leq \frac{\epsilon}{1 - \gamma} + \frac{2\gamma\delta}{(1-\gamma)^2}\) (proof is straightforward, check them yourself :-)). The VI algorithm converges exponentially when \(\epsilon = 0\) and \(\delta = 0\).
Approximate Policy Iteration
The VI method is a direct creation based on the fact that the Bellman optimality operator \(\mathcal T\) is a contraction. However, people may think by involving policy evaluation each step one can have a better rate of convergence and a better approximation of optimal policy, though it will incur more calculation each iteration. This is plausible, but not insignificantly so in theory. The approximate policy iteration (PI) algorithm is stated as follows:
  # Approximate Policy Iteration
Given the current policy π_k
Repeat:
 Policy evaluation: Compute v such that
 sup|v - (T_π_k)^m_k (v) | ≤ δ
 by appying Bellman evaluation operator T_π
 Policy improvement: Extract a policy π_{k+1} that satisfies
 sup|T_π_{k+1}(v)- T(v)| ≤ ϵ
 π_k ← π_{k+1}
until v converges.
Extract π_{k+1} 
  
In this algorithm, the parameter \(m_k\) is an positive intergers. Taken the usual choice of \(m_k = 1\), we can show that \(d(v_{\pi_{k+1}}, v_{\mathtt{opt}}) \leq \gamma d(v_{\pi_{k}}, v_{\mathtt{opt}}) + \frac{\epsilon + 2\delta\gamma}{1 - \gamma}\) for each iteration. Therefore, its final error bound is asymptotically proportional to \(1/(1-\gamma)^2\), which indicates PI does not hold an accuracy advantage over VI. And the exact PI converges at an exponential rate which is as same as that of exact VI.
Q-Learning: Asynchronous Value Iteration
In the last two section we explained how we do classical model based planing for MDPs from the topological point of view. Many readers by now have a sense that how this stuff and value-based reinforcement learning are connected. But there remains a critical question: Can this explanation is still suitable for those general model free cases? Can we explain those classical reinforcement “learning” algorithm, such as Q-learning, in an analog way?
Intuitively, the answer should be “yes”. Consider the evaluation operator \(\mathcal T_{\pi}\) and optimality operator \(\mathcal T\) defined by
\[
 (\mathcal T_{\pi} q)(s, a) := R(s, a) + \gamma\mathbb E_{\mathcal P, \pi} q(s’, a’)
\]
and
\[
 (\mathcal T q)(s, a) := R(s, a) + \gamma\mathbb E_{\mathcal P} \max_{a’\in A} q(s’, a’)
\]
respectively, over the Q value function space \(\mathbb R^{S\times A}\), with the similar metric as that of \(V\). Following the similar analysis, the contraction and the monotonicity properties hold for them. However, since the model is unknown, we cannot update the Q value function as that in value iteration. We can only approximate and update the Q value for some certain pair \((s, a)\) by sampling or temporal difference method each iteration. Is its convergence still guaranteed? The following theorem resolves our doubts.
 Definition 5. (Asynchronous Value Iteration) Conside value function \(v\) as a composition of 
\((v_s)_ {s\in S}\) such that in each iteration,
\[
 v_s^{t+1}(s’) = \begin{cases}
 \mathcal Tv^{t}(s’), &amp; \text{if}~s’ = s\newline
 \mathcal v_s^{t}(s’), &amp; \text{otherwise.}
 \end{cases}
\]
Theorem 3. (Asynchronous Convergence Theorem) If each restricted value function \(v_s\) can be update infinite times, and there is a sequence of nonempty subsets \(\{V^k\} \subseteq V\) with \(V^{k+1}\subseteq V^{k}\), \(k = 0, 1, \dots\) such that if \(\{v^k\}\) is any sequence with \(v_k\in V_k\) for all \(k \leq 0\), the \(\{v^k\}\) converges pointwise to \(v^*\). Assume further the following:
 Synchronous Convergence Condition: We have 
\[
 \mathcal T v \in V^{k+1}, \forall v \in V^k
\]
 Box Condition: For all \(k, V^k\) is a Cartesian product of the form
\[
 V^k = \times_{s\in S} V_s^k 
\]
where \(V_{s}^k\) is a set of bounded real-valued functions on state \(s\). Then for every \(v^0 \in V^0\) the sequence \(\{v^t\}\) generated by the asynchronous VI algorithm converges to \(v^*\).
  Proof.  To show that the algorithm converges is equivalent to show that the iterates from \(V^k\) will get in to \(V^{k+1}\) eventually, i.e., for each \(k \leq 0\), there is a time \(t_k\) such that \(v^t\in V^{k}\) for all \(t \geq t_k\). We can prove it by mathematical induction.
 
When \(k = 0\), the statement is true since \(v^0\in V^0\). Assuming the statement is true for a given \(k\), we will show there is a time \(t_{k+1}\) with the required properties. For each \(s\in S\), let set \(L_{s} = \{t: s^t = s\}\) record the time an update happend on the state \(s\). Let \(t(s)\) be the first element in \(L_{s}\) such that \(t(s) \geq t_{k}\). Then by the synchrnous convergence condition, we have \(\mathcal Tv^{t(s)} \in V^{k+1}\) for all \(s\in S\). In the view of the box condition, it implies \(v^{t+1}(s)\in V_s^{k+1}\) for all \(t \geq t(s)\) for any \(s\in S\). Therefore, let \(t_{k+1} = \max_{s}t(s) + 1\), using the box condition, we have \(v^t \in V^{k+1}\) for all \(t \leq t_{k+1}\). By mathematical induction, the statement holds, and \(V^k\) will get in to \(V^{k+1}\) eventually. Hence, \(\{v^t\}\) converges to \(v^*\). \(\blacksquare\)
A Methodology for Developing Iterative RL Algorithms
It is really exciting to see that the convergence and correctness of value-based reinforcement learning algorithm, such as Q-learning, can be interpreted in a topological view. In fact, the contraction mapping theory provides us with a methodology for developing iterative value-based RL algorithm. In a high-level overview, iterative value-based RL algorithms are concerned with the following five essential elements:
 Value Space \(V\):
A common choice of \(V\) is the space of value functions in \(\mathbb R^S\) or the space of Q-value functions in \(\mathbb R^{S\times A}\). There are many other choices such as the space of ordered pair of \((V,Q)\) (Example 2.6.2 in Dimitri’s book), or the space of multi-dimensional value functions.
 Value Metric \(d\): It defines the “distance” between two points in the value space. Besides the basic four conditions, the metric \(d\) should ensure a complete metric space \((V, d)\) to validate the Banach’s fixed-point theorem. A compatible selection of value metric will make our analysis much easier.
 Evaluation Operator \(\mathcal T_{\pi}\): We have constructed a recursive expression of a certain value point in the value space associated with some policy, e.g., the Bellman expectation equation,
\[
 v_{\pi}(\cdot) = H(\cdot, \pi(\cdot), v_{\pi})
\]
to depict the value of a policy \(v_{\pi}\) as a fixed point we desire. Carefully verify that the contraction and the monotonicity properties hold for \(\mathcal T_{\pi} v:=H(\cdot, \pi(\cdot), v)\).
 Optimality Operator \(\mathcal T\): Write down recursive expression of the optimal point in the value space, e.g., the Bellman optimality equation,
\[
 v^* (\cdot) = \sup_{\pi\in \Pi}\mathcal T_{\pi} v^* (\cdot).
\]
One thing important is to make sure the contraction property holds for the optimality operator \(\mathcal T:= \sup_{\pi\in \Pi}\mathcal T_{\pi}v\). Note that when the metric \(d\) is the supremum of the absolute value of the difference, the contraction property of \(\mathcal T\) is automatically satisfied according to Proposition 1.
 Updating Scheme: It is the most tricky part. For those small-scale problems, the asynchronous value iteration is enough. However, to make our reinforcement learning algorithm practical and scalable, we have to consider much more factors. For example: Should our algorithm be on-policy or off-policy? How do we approximate the value and policies? If it is an online algorithm, how do we trade off exploration and exploitation? All these details will significantly influence the performance of our algorithm on real-world tasks.
Learning from Value Distribution
 
Let’s get back to the DeepMind paper mentioned at the begining, which proposes a new reinforcement learning algorithm from a distributional perspective. This work is actually a good example of developing and analyzing a new reinforcement learning algorithm using the methodology that we summarize above.
In this example, the value space consists of all the “value distribution” \(Z(s, a)\). The value metric of the value space is defined by \(d(Z_1, Z_2):= \sup_{s, a}d_{w}(Z_1(s,a), Z_2(s,a))\), where \(d_{w}\) is the Wasserstein distance between two random variables. The evaluation operator is defined by 
\[
 \mathcal T_{\pi} Z(s, a) \stackrel{D}{:=} R(s, a) + \gamma P^{\pi}Z(s, a),
\]
which is proved to be a contraction in \((\mathcal Z, d)\), where “\(\stackrel{D}{=}\)” means random variables on both sides are drawn from the same distribution. The monotonicity property also holds as they define the order as the order of the magnitude of expectation (\(Z_1 \leq Z_2 \Leftrightarrow \mathbb E Z_1 \leq \mathbb E Z_2\)). The optimality operator is defined by \(\mathcal T Z := \sup_{\pi}\mathcal T_{\pi}Z\). However, since the relative order is incompatible with the metric, this optimality operator is NOT a contraction in \((\mathcal Z, d)\). They use approximate asynchronous value iteration as the updating scheme, where the value distribution function is approximated by a neural network which outputs a discrete distribution between \([V_\text{MAX}, V_\text{MIN}]\).
If we only consider the space of the first-order moment of the value distribution and the metric
\(d_{\mathbb E}(\mathbb E Z_1, \mathbb E Z_2):= \sup_{s, a} |\mathbb E Z_1(s,a) - \mathbb E Z_2(s,a)|\), both \(\mathbb E\mathcal T_{\pi}\) and \(\mathbb E\mathcal T\) are contractions. And this explains why the C51 algorithm works well. However, as for predicting value distribution, I doubt the correctness of directly using the result of value iteration. In my opinion, a policy iteration algorithm would be a better choice here for maintaining value distributions, or we should correct the value distribution function by repeatedly applying evaluation operator after the optimal policy is found.
Multi-Objective RL
Another example which might be helpful for us to understand this methodology is the multi-objective reinforcement learning. And this is also an interesting topic I am recently working on. The value space of this problem should be some multi-dimensional value function. The metric can be extended from that of a normed vector space. If we only want to find one policy resulting a Pareto optimal return, or simply want to find the best policy maximizing the return associated with specific weights, the evaluation operator and the optimality operator are very similar to \(R + \gamma P^{\pi}V\) and \(\sup_{\pi} \{R + \gamma P^{\pi}V\}\) (the \(\sup\{\cdot\}\) is defined by Pareto domination and the order of magnitude of weighted sum, respectively). Designing an updating scheme according to the real problem, we are guaranteed to have a viable algorithm. If we want to find all the policies resulting in a Pareto optimal return, or make the policies controllable, how to represent a collection of policies and write down the corresponding evaluation/optimality operator are very challenging.
Summary: Contraction, Value Iteration, and More
In this article, some topological backgrounds of metric spaces and the Banach fixed-point theorem are first introduced. Based on these mathematical foundations, we investigate the contractive dynamic programming, where the contraction property and the monotonicity property together guarantee a unique optimal solution can be achieved exponentially fast. Bring these ideas to the reinforcement learning context, we can explain and analyze value iteration, policy iteration and Q-learning algorithms in a unified framework. It further inspires us to develope a methodology for developing iterative value-based reinforcement algorithm, which requires us to consider value space, value metric, evaluation operator, optimality operator and updating scheme.
The next time someone asks, “how does a value-based RL algorithm converge?”, or “how do people come up with a new RL algorithm?”, we can answer him not first by complex, detailed mathematical deduction, but simply by drawing an intuitive picture and analyzing on those accessible abstractions! Our journey on the topic of topological explanation of value-based reinforcement learning comes to an end here. Hope you enjoyed it!
How Does Value-Based Reinforcement Learning Find the Optimal Policy?
https://runzhe-yang.science/2017-10-04-contraction/
<div><div><h2>Metric Spaces, Cauchy Sequence, Contraction mapping and Banach Fixed Point Theorem</h2><div><a href="https://medium.com/@vaibkumr?source=post_page-----ac9f0954e19f--------------------------------"><div><p></p></div></a><a href="https://towardsdatascience.com/?source=post_page-----ac9f0954e19f--------------------------------"><div><p></p></div></a></div></div><div><p>STOP READING THIS, WRONG INTUITIONS HERE.</p><p>Reinforcement learning has achieved remarkable results in playing games like StarCraft (AlphaStar) and Go (AlphaGO). At the core of all these successful projects lies — The Bellman optimality<em> </em>equation for Markov decision processes (MDPs).</p><figure><figcaption>Bellman Optimality Equation</figcaption></figure><p>The Bellman optimality<em> </em>equation is a recursive equation that can be solved using dynamic programming (DP) algorithms to find the optimal value function and the optimal policy. In this article, I will try to explain why the Bellman optimality equation can solve every MDP by providing an optimal policy and perform an easy (hopefully) mathematical analysis of the same.</p><h2>Notations</h2><ul><li><code>S</code> is state space.</li><li><code>V</code> is the value function.</li><li><code>V*</code> is the optimal value function.</li><li><code>V(s)</code> is the value of state <code>s</code>.</li><li><code>π</code> is a policy.</li><li><code>π*</code> is the optimal policy.</li><li><code>π(s)</code> returns the action for state <code>s</code> under the policy <code>π</code>.</li><li><code>P</code> is the transition probability matrix.</li><li><code>A</code> is the action space.</li></ul><h2>Prerequisites</h2><p>Despite my best efforts, the analysis is going to be fairly rigorous and I assume the readers to be familiar with the following perquisites:</p><ol><li>Markov decision process (MDP)</li><li>Bellman Equation and how it’s solved using iterative methods.</li><li>Basics of RL — Value functions, rewards, policies, discount factor etc.</li><li>Linear algebra</li><li>Vector calculus</li></ol><h2>Overview</h2><p>If you have studied RL and MDPs, you must’ve encountered the claim: <strong><em>“For each MDP, there is always at least one policy that is better than or equal to all other policies.”</em></strong> This occurs both in Sutton and Barto’s book as well as in the lecture series by David Silver. Reading/Hearing this makes the claim pretty intuitive, however, I had to dig in deeper and understand this in a more concrete manner. Therefore, in this article, I will mathematically prove the following theorem:</p><blockquote><p><strong>Theorem:</strong> For any finite MDP, there exists an optimal policy π* such that it is better than or equal to every other possible policy π.</p></blockquote><p>Before finding the best policy, we need to understand the ordering of policies. When is one policy (<code>π1</code>) considered better than the other (<code>π2</code>)?</p><p>If the value of a state derived using <code>π1</code> is better than or equal the value of a state derived using <code>π2</code> for every state in the environment, then the policy π1 is said to be better than policy <code>π2</code>. Mathematically, this can be written as follows:</p><figure><figcaption>Comparison of policies</figcaption></figure><p>Now that we know how to compare policies, we need to prove that there always exists a policy that is better than all other policies.</p><p>We are going to prove this using the <strong>Banach fixed point theorem </strong>by showing that the <strong>Bellman optimality operator</strong> is a <strong>contraction</strong> over a <strong>complete metric space</strong> of real numbers with metric <strong>L-infinity norm</strong>. For this, we will first discuss the <strong>fixed point problem </strong>and <strong>complete metric spaces</strong> with respect to the <strong>Cauchy sequence</strong>.</p><p>The above paragraph sounds very intimidating but its actually going to be pretty easy and intuitive once we get past the basic terminologies. We will be discussing everything that is in <strong>bold</strong> in the above paragraph. Let’s follow a bottom-up approach and learn each concept and conquer our fears:</p><h2>1. Fixed point problem</h2><p>I am sure most of us are familiar with the problem of finding the roots of an equation. We solve for <code>x</code> such that a function <code>f(x) = 0</code>. However, in a fixed point problem, we solve for <code>x</code> such that <code>f(x) = x</code>. As the name suggests, <code>x</code> is a fixed point, it does not change even on the application of the function. A fixed point problem can be converted into a problem of finding roots by forming another function <code>g(x) = f(x)-x = 0</code>. In fact, even root finding problems can also be converted back to fixed point problems. However, it’s really easy to solve fixed point problems (for special cases) and that is what makes them incredibly interesting and useful (sans the computational overhead).</p><blockquote><p>To solve a fixed point problem, choose a random starting value of x and repeatedly apply f(x) infinite times. If the <strong>function is convergent</strong> and you’re lucky, you will find the solution.</p></blockquote><p>Mathematically, it’s pretty simple, lets first describe a notation:</p><figure><figcaption>f^n(x) notation</figcaption></figure><p>Now, if the function is convergent then it must converge to some value, say, <code>x*</code>. This value, <code>x*</code> is indeed the solution to the fixed point problem as shown below:</p><p>Let's choose some arbitrary value <code>x0</code> and apply the function <code>f(.)</code> on <code>x0</code> for infinite times to get <code>x*</code>, and then use that to solve the fixed point problem:</p><figure><figcaption>Solving the fixed point problem</figcaption></figure><p>The intuition behind this is pretty simple, if a function has converged at some point then the value of this function at that convergent point will be the convergent point itself. Therefore, the convergent point is the fixed point itself.</p><p>This can also be observed empirically through the notebook below:</p><figure></figure><h2>2. Metric space</h2><p>A metric space is simply a set with a metric defined to measure the distance between any two elements of the set. For example, the Euclidean space is a metric space with distance defined as the Euclidean distance in the set of real numbers. Therefore, a metric space <code>M</code> is represented as <code>(X, d)</code> where <code>X</code> is the set and <code>d</code> is some sort of metric. The metric <code>d</code> must satisfy the following properties:</p><ol><li><strong>Identify</strong> <code>d(x,x) = 0</code></li><li><strong>Non-Negativity</strong> <code>d(x, y) &gt;0</code></li><li><strong>Symmetry</strong>: <code>d(x,y) = d(y,x)</code></li><li><strong>Triangular inequality</strong>: <code>d(x,z) ≤ d(x,y)+d(y,z)</code></li></ol><h2><strong>3. Cauchy sequence</strong></h2><p>For a metric space <code>(X, d)</code> the sequence of elements of the set <code>X</code>, <code>(x1, x2, x3…. xn)</code> is a Cauchy sequence if, for every positive real number <code><em>ε</em></code><em>, </em>there exists an integer <code>N</code> such that the following equation holds:</p><figure><figcaption>Cauchy Sequence</figcaption></figure><p>The mathematical explanation here isn’t very intuitive and needlessly complex. In simple words, a sequence of elements of a metric space is Cauchy if this sequence converges at some point (the distance between them becomes constant). Another great explanation for this is given by <a href="https://brilliant.org/wiki/cauchy-sequences/">this website</a> and goes as follows: <em>“for any small distance, there is a certain index past which any two terms are within that distance of each other, which captures the intuitive idea of the terms becoming close.” </em>The idea of “terms become close” is the basic intuition behind convergence or the limit of the series.</p><h2>4. Complete Metric space</h2><p>A metric space <code>(X, d)</code> is complete if every possible Cauchy sequence of the elements in the set <code>X</code> converges to an element that also belongs to the set <code>X</code>. That is to say, the convergent limit of every Cauchy sequence of the elements of set lies in the set itself. Which is why it’s called “complete”.</p><h2>5. Contraction</h2><p>A function (or operator or mapping) defined on the elements of the metric space <code>(X, d)</code> is a contraction (or contractor) if there exists some constant <code>γ∈[0,1)</code> such that for any two elements of the metric space <code>x1</code> and <code>x2</code>, the following condition holds:</p><figure><figcaption>Contraction mapping</figcaption></figure><p>This means that after applying the mapping <code>f(.)</code> on the elements <code>x1</code> and <code>x2</code>, they got closer to each other by at least a factor <code>γ</code><em>. </em>Also, the smallest value of such a constant <code><em>γ</em></code><em> </em>is called the Lipschitz constant (this is an important constant for generative adversarial networks). Also, if <code><em>γ=</em>1</code>, the mapping is no more a contraction but rather a short mapping. Intuitively, it can be observed that the sequential values of the elements are getting closer after applying the contraction mapping.</p><h2>6. Banach fixed point theorem</h2><p>This is the heart and soul of our proof. Informally, this theorem says that for a complete metric space, the application of a contractor on the elements of the set, again and again, would eventually get us to an optimal, unique value. We know:</p><ol><li>Contractors bring the elements of the set together.</li><li>Applying this contractor, again and again, would get us a sequence. (Cauchy?)</li><li>The Cauchy sequences in a complete metric space always converge to a value that is part of the metric space.</li></ol><p>Formally, this theorem can be formulated as:</p><blockquote><p>Theorem: Let (X, d) be a complete metric space and a function f: X-&gt;X be a contractor then, f has a <strong>unique</strong> fixed point x*∈ X (i.e. f(x*)=x*) such that the sequence f(f(f(…f(x)))) <strong>converges</strong> to x*.</p></blockquote><p>Now, to prove this mathematically, we need to prove both the uniqueness and existence of <code>x*</code>.</p><ol><li><strong>Uniqueness</strong>: We will prove this by contradiction. Let us assume that the convergence value is not unique and <code>x1*</code> and <code>x2*</code> are two values at which the sequence of contractor converges then, we will have:</li></ol><figure><figcaption>x1* and x2* are optimal values, the contractor has converged here, the distance does not change anymore</figcaption></figure><p>Also, note that <code>f</code> is a contractor so it must hold the following property:</p><figure><figcaption>The contractor property</figcaption></figure><p>Now since <code>γ∈[0,1)</code>, it’s impossible to satisfy both equation 1 and 2 simultaneously. Therefore our assumption must be wrong. Hence, by contradiction, <code>x*</code> must be unique.</p><p>2. <strong>Existence: </strong>Now that we’ve proved that <code>x*</code> is unique, we need to prove that <code>x*</code> exists. Let <code>(x1, x2, x3, …. xn)</code> be the sequence formed by repeatedly applying the contractor.</p><figure><figcaption>An element of our metric space</figcaption></figure><p>If we <strong>assume</strong> that the sequence <code>(x1, x2, x3, …. xn)</code> is Cauchy, we know for certain that this sequence will converge to some point, say, <code>x*</code>. Also, since the metric space is complete, this convergent point,<code>x*</code> will belong to the metric space <code>(X,d)</code>. Now, we just need to prove that this sequence is Cauchy. We will do this by taking two elements of the set <code>xn</code> and <code>xm</code> such that <code>m&gt;&gt;n </code>and, <code>m</code> is very large, then by repeatedly applying the triangular inequality property of the metric <code>d</code>, we have:</p><figure><figcaption>Expanding the triangular inequality of the metric d</figcaption></figure><p>Now, since f is a contractor, we know that:</p><figure><figcaption>The repeated contractor inequality</figcaption></figure><p>We can further reduce <code>d(xm, xn)</code> as follows:</p><figure></figure><p>Now, by choosing <code>n</code> to be sufficiently large, we can make the RHS of the above equation less than any positive real number <code><em>ε</em></code><em>. </em>Hence, the sequence <code>(x1, x2, x3, …. xn)</code> is Cauchy and an optimal <code>x*∈X</code> exists. This concludes the proof of the Banach fixed point theorem.</p><h2>Coming back to the Bellman Optimality Equation</h2><p>For the value function, <code>V(s)</code> we define a new operator, the optimal Bellman operator,<code>B</code> which takes in a value function and returns another value function. We define this operator as follows:</p><figure><figcaption>The Bellman Operator</figcaption></figure><p>It can easily be observed, <code>B</code> is a recursive operator. Therefore, this will generate a sequence of value functions. If we can show that <code>B</code> is indeed a contractor for some metric space <code>(X,d)</code> then by the Banach fixed point theorem, we can conclude that the repeated application of the optimal Bellman operator will eventually give a unique optimal value function using which an optimal (best) policy can be derived. Therefore, all our work now reduces to proving that <code>B</code> is a contractor. First, let's define the metric space as follows:</p><p><strong>Metric space</strong> <code>(X,d)</code>: The set <code>X</code> is the set of real numbers defined as follows:</p><figure><figcaption>Value function belongs to the set of real numbers. The X in the bottom shows the set for the complete value function while the one at the top shows the set the value of a single state.</figcaption></figure><p>For the metric, we use the <a href="https://en.wikipedia.org/wiki/L-infinity">L-infinity</a> norm defined as follows:</p><figure><figcaption>L-infinity norm</figcaption></figure><p>According to this metric, the distance between the two value functions will be equal to the highest element-wise absolute difference between the two. Also, for finite MDPs with finite rewards, the value functions will always stay in the real space. It is impossible for the value function to not be in the real space, therefore, this finite space will always be complete.</p><blockquote><p><strong>Theorem</strong>: Bellman operator <code>B</code> is a contraction mapping in the finite space <code>(R, L-infinity)</code></p></blockquote><p>Proof: Let <code>V1</code> and <code>V2</code> be two value functions. Then:</p><figure><figcaption>Proof of B being a contraction</figcaption></figure><ul><li>In the second step above, we introduce inequality by replacing <code>a’</code> by <code>a</code> for the second value function. This is because by replacing its optimal action <code>a’</code> by some other action <code>a</code>, we have reduced its overall value thereby introducing an inequality.</li><li>In the fourth step, we remove the L-infinity norm by taking the max over <code>s’</code> (recall the definition of L-infinity with respect to value functions in our setting)</li><li>In the final step, we remove the sigma because the sum of probabilities is always <code>1</code>.</li></ul><p>Finally, for the Bellman optimality equation, since <code>γ∈[0,1)</code> (let's ignore the possibility of <code>γ=1</code> for now), the Bellman operator is, therefore, a contractor.</p><p>Now we know:</p><ol><li><code>(R, L-infinity)</code> is a complete metric space</li><li>Bellman operator <code>B</code> is a contractor</li></ol><p>Hence, by the Banach fixed point theorem, we conclude that there exists a unique optimal value function <code>V*</code> for every MDP. Using this <code>V*</code>, we can derive the optimal policy <code>π*</code>.</p><p>Hence proved, for any finite MDP, there exists an optimal policy <code>π*</code> such that it is better than or equal to every other possible policy <code>π</code>.</p><p>Now, how to find this optimal policy and value function? One way is to just repeatedly apply the Bellman operator to a random initial value function to get the optimal function. But, this is computationally very expensive and often downright infeasible. Therefore, we use iterative methods like value and policy iteration or temporal difference methods like Q-Learning or SARSA. For more on this, please refer to my blog <a href="https://towardsdatascience.com/reinforcement-learning-temporal-difference-sarsa-q-learning-expected-sarsa-on-python-9fecfda7467e#eccc">Reinforcement learning: Temporal-Difference, SARSA, Q-Learning &amp; Expected SARSA in python</a> or just view these algorithms on my Github: <a href="https://github.com/TimeTraveller-San/RL_from_scratch">RL_from_scratch</a>.</p><h2>Conclusion</h2><p>We learned some basic mathematical tools like metric spaces, complete metric spaces, Cauchy sequences, contraction mapping and the Banach fixed point theorem. Building upon all of this, we mathematically proved the unique optimality of the Bellman optimality equation for every MDP.</p><h2>References and credits</h2><ul><li>I used <a href="https://latex.codecogs.com/eqneditor/editor.php">https://latex.codecogs.com/eqneditor/editor.php</a> to generate images for LaTeX. (I wish medium allowed LaTeX)</li><li>Wikipedia for providing all the formal definitions and theorems.</li><li>The <a href="https://www.cs.cmu.edu/~ninamf/courses/401sp18/prev.shtml">CMU 15–781</a> course</li><li>The <a href="http://therisingsea.org/notes/mast30026/lecture14.pdf">notes</a> and <a href="https://www.youtube.com/watch?v=gsM7PPBx16k">video</a> by Dr Daniel Murfet</li><li><a href="https://www.youtube.com/channel/UCZvMhJ3EaNvpacdlMmm3VKA">The free NPTEL course</a> by Dr Balaraman Ravindran and IIT-Madras</li></ul><h2>Endnotes</h2><p>I have tried to explain everything in as simple words as possible. If I was unclear or if I made some mistakes, please let me know in the responses. Also, there might be some mistakes in the latex equations. Please let me know if you find them. Last but not least, thanks for reading!</p></div></div>||||I||||          Open in app

            Sign up

              Sign in

                  Write

              Sign up

                Sign in

                        Mathematical Analysis of Reinforcement Learning — Bellman Optimality Equation

                        Metric Spaces, Cauchy Sequence, Contraction mapping and Banach Fixed Point Theorem

                                                Vaibhav Kumar

                                          ·

                                          Follow

                                          Published in

                                              Towards Data Science

                                          ·
                                        11 min read
                                          ·
                                        Jan 7, 2020

                                        --

                                      7

                                                    Listen

                                          Share

                      STOP READING THIS, WRONG INTUITIONS HERE.

                      Reinforcement learning has achieved remarkable results in playing games like StarCraft (AlphaStar) and Go (AlphaGO). At the core of all these successful projects lies — The Bellman optimality equation for Markov decision processes (MDPs).

                      Bellman Optimality Equation

                      The Bellman optimality equation is a recursive equation that can be solved using dynamic programming (DP) algorithms to find the optimal value function and the optimal policy. In this article, I will try to explain why the Bellman optimality equation can solve every MDP by providing an optimal policy and perform an easy (hopefully) mathematical analysis of the same.

                      Notations

                        * S is state space.
                        * V is the value function.
                        * V* is the optimal value function.
                        * V(s) is the value of state s.
                        * π is a policy.
                        * π* is the optimal policy.
                        * π(s) returns the action for state s under the policy π.
                        * P is the transition probability matrix.
                        * A is the action space.

                      Prerequisites

                      Despite my best efforts, the analysis is going to be fairly rigorous and I assume the readers to be familiar with the following perquisites:

                       1. Markov decision process (MDP)
                       2. Bellman Equation and how it’s solved using iterative methods.
                       3. Basics of RL — Value functions, rewards, policies, discount factor etc.
                       4. Linear algebra
                       5. Vector calculus

                      Overview

                      If you have studied RL and MDPs, you must’ve encountered the claim: “For each MDP, there is always at least one policy that is better than or equal to all other policies.” This occurs both in Sutton and Barto’s book as well as in the lecture series by David Silver. Reading/Hearing this makes the claim pretty intuitive, however, I had to dig in deeper and understand this in a more concrete manner. Therefore, in this article, I will mathematically prove the following theorem:

                      Theorem: For any finite MDP, there exists an optimal policy π* such that it is better than or equal to every other possible policy π.

                      Before finding the best policy, we need to understand the ordering of policies. When is one policy (π1) considered better than the other (π2)?

                      If the value of a state derived using π1 is better than or equal the value of a state derived using π2 for every state in the environment, then the policy π1 is said to be better than policy π2. Mathematically, this can be written as follows:

                      Comparison of policies

                      Now that we know how to compare policies, we need to prove that there always exists a policy that is better than all other policies.

                      We are going to prove this using the Banach fixed point theorem by showing that the Bellman optimality operator is a contraction over a complete metric space of real numbers with metric L-infinity norm. For this, we will first discuss the fixed point problem and complete metric spaces with respect to the Cauchy sequence.

                      The above paragraph sounds very intimidating but its actually going to be pretty easy and intuitive once we get past the basic terminologies. We will be discussing everything that is in bold in the above paragraph. Let’s follow a bottom-up approach and learn each concept and conquer our fears:

                      1. Fixed point problem

                      I am sure most of us are familiar with the problem of finding the roots of an equation. We solve for x such that a function f(x) = 0. However, in a fixed point problem, we solve for x such that f(x) = x. As the name suggests, x is a fixed point, it does not change even on the application of the function. A fixed point problem can be converted into a problem of finding roots by forming another function g(x) = f(x)-x = 0. In fact, even root finding problems can also be converted back to fixed point problems. However, it’s really easy to solve fixed point problems (for special cases) and that is what makes them incredibly interesting and useful (sans the computational overhead).

                      To solve a fixed point problem, choose a random starting value of x and repeatedly apply f(x) infinite times. If the function is convergent and you’re lucky, you will find the solution.

                      Mathematically, it’s pretty simple, lets first describe a notation:

                      f^n(x) notation

                      Now, if the function is convergent then it must converge to some value, say, x*. This value, x* is indeed the solution to the fixed point problem as shown below:

                      Let's choose some arbitrary value x0 and apply the function f(.) on x0 for infinite times to get x*, and then use that to solve the fixed point problem:

                      Solving the fixed point problem

                      The intuition behind this is pretty simple, if a function has converged at some point then the value of this function at that convergent point will be the convergent point itself. Therefore, the convergent point is the fixed point itself.

                      This can also be observed empirically through the notebook below:

                      2. Metric space

                      A metric space is simply a set with a metric defined to measure the distance between any two elements of the set. For example, the Euclidean space is a metric space with distance defined as the Euclidean distance in the set of real numbers. Therefore, a metric space M is represented as (X, d) where X is the set and d is some sort of metric. The metric d must satisfy the following properties:

                       1. Identify d(x,x) = 0
                       2. Non-Negativity d(x, y) >0
                       3. Symmetry: d(x,y) = d(y,x)
                       4. Triangular inequality: d(x,z) ≤ d(x,y)+d(y,z)

                      3. Cauchy sequence

                      For a metric space (X, d) the sequence of elements of the set X, (x1, x2, x3…. xn) is a Cauchy sequence if, for every positive real number ε, there exists an integer N such that the following equation holds:

                      Cauchy Sequence

                      The mathematical explanation here isn’t very intuitive and needlessly complex. In simple words, a sequence of elements of a metric space is Cauchy if this sequence converges at some point (the distance between them becomes constant). Another great explanation for this is given by this website and goes as follows: “for any small distance, there is a certain index past which any two terms are within that distance of each other, which captures the intuitive idea of the terms becoming close.” The idea of “terms become close” is the basic intuition behind convergence or the limit of the series.

                      4. Complete Metric space

                      A metric space (X, d) is complete if every possible Cauchy sequence of the elements in the set X converges to an element that also belongs to the set X. That is to say, the convergent limit of every Cauchy sequence of the elements of set lies in the set itself. Which is why it’s called “complete”.

                      5. Contraction

                      A function (or operator or mapping) defined on the elements of the metric space (X, d) is a contraction (or contractor) if there exists some constant γ∈[0,1) such that for any two elements of the metric space x1 and x2, the following condition holds:

                      Contraction mapping

                      This means that after applying the mapping f(.) on the elements x1 and x2, they got closer to each other by at least a factor γ. Also, the smallest value of such a constant γ is called the Lipschitz constant (this is an important constant for generative adversarial networks). Also, if γ=1, the mapping is no more a contraction but rather a short mapping. Intuitively, it can be observed that the sequential values of the elements are getting closer after applying the contraction mapping.

                      6. Banach fixed point theorem

                      This is the heart and soul of our proof. Informally, this theorem says that for a complete metric space, the application of a contractor on the elements of the set, again and again, would eventually get us to an optimal, unique value. We know:

                       1. Contractors bring the elements of the set together.
                       2. Applying this contractor, again and again, would get us a sequence. (Cauchy?)
                       3. The Cauchy sequences in a complete metric space always converge to a value that is part of the metric space.

                      Formally, this theorem can be formulated as:

                      Theorem: Let (X, d) be a complete metric space and a function f: X->X be a contractor then, f has a unique fixed point x*∈ X (i.e. f(x*)=x*) such that the sequence f(f(f(…f(x)))) converges to x*.

                      Now, to prove this mathematically, we need to prove both the uniqueness and existence of x*.

                       1. Uniqueness: We will prove this by contradiction. Let us assume that the convergence value is not unique and x1* and x2* are two values at which the sequence of contractor converges then, we will have:

                      x1* and x2* are optimal values, the contractor has converged here, the distance does not change anymore

                      Also, note that f is a contractor so it must hold the following property:

                      The contractor property

                      Now since γ∈[0,1), it’s impossible to satisfy both equation 1 and 2 simultaneously. Therefore our assumption must be wrong. Hence, by contradiction, x* must be unique.

                      2. Existence: Now that we’ve proved that x* is unique, we need to prove that x* exists. Let (x1, x2, x3, …. xn) be the sequence formed by repeatedly applying the contractor.

                      An element of our metric space

                      If we assume that the sequence (x1, x2, x3, …. xn) is Cauchy, we know for certain that this sequence will converge to some point, say, x*. Also, since the metric space is complete, this convergent point,x* will belong to the metric space (X,d). Now, we just need to prove that this sequence is Cauchy. We will do this by taking two elements of the set xn and xm such that m>>n and, m is very large, then by repeatedly applying the triangular inequality property of the metric d, we have:

                      Expanding the triangular inequality of the metric d

                      Now, since f is a contractor, we know that:

                      The repeated contractor inequality

                      We can further reduce d(xm, xn) as follows:

                      Now, by choosing n to be sufficiently large, we can make the RHS of the above equation less than any positive real number ε. Hence, the sequence (x1, x2, x3, …. xn) is Cauchy and an optimal x*∈X exists. This concludes the proof of the Banach fixed point theorem.

                      Coming back to the Bellman Optimality Equation

                      For the value function, V(s) we define a new operator, the optimal Bellman operator,B which takes in a value function and returns another value function. We define this operator as follows:

                      The Bellman Operator

                      It can easily be observed, B is a recursive operator. Therefore, this will generate a sequence of value functions. If we can show that B is indeed a contractor for some metric space (X,d) then by the Banach fixed point theorem, we can conclude that the repeated application of the optimal Bellman operator will eventually give a unique optimal value function using which an optimal (best) policy can be derived. Therefore, all our work now reduces to proving that B is a contractor. First, let's define the metric space as follows:

                      Metric space (X,d): The set X is the set of real numbers defined as follows:

                      Value function belongs to the set of real numbers. The X in the bottom shows the set for the complete value function while the one at the top shows the set the value of a single state.

                      For the metric, we use the L-infinity norm defined as follows:

                      L-infinity norm

                      According to this metric, the distance between the two value functions will be equal to the highest element-wise absolute difference between the two. Also, for finite MDPs with finite rewards, the value functions will always stay in the real space. It is impossible for the value function to not be in the real space, therefore, this finite space will always be complete.

                      Theorem: Bellman operator B is a contraction mapping in the finite space (R, L-infinity)

                      Proof: Let V1 and V2 be two value functions. Then:

                      Proof of B being a contraction

                        * In the second step above, we introduce inequality by replacing a’ by a for the second value function. This is because by replacing its optimal action a’ by some other action a, we have reduced its overall value thereby introducing an inequality.
                        * In the fourth step, we remove the L-infinity norm by taking the max over s’ (recall the definition of L-infinity with respect to value functions in our setting)
                        * In the final step, we remove the sigma because the sum of probabilities is always 1.

                      Finally, for the Bellman optimality equation, since γ∈[0,1) (let's ignore the possibility of γ=1 for now), the Bellman operator is, therefore, a contractor.

                      Now we know:

                       1. (R, L-infinity) is a complete metric space
                       2. Bellman operator B is a contractor

                      Hence, by the Banach fixed point theorem, we conclude that there exists a unique optimal value function V* for every MDP. Using this V*, we can derive the optimal policy π*.

                      Hence proved, for any finite MDP, there exists an optimal policy π* such that it is better than or equal to every other possible policy π.

                      Now, how to find this optimal policy and value function? One way is to just repeatedly apply the Bellman operator to a random initial value function to get the optimal function. But, this is computationally very expensive and often downright infeasible. Therefore, we use iterative methods like value and policy iteration or temporal difference methods like Q-Learning or SARSA. For more on this, please refer to my blog Reinforcement learning: Temporal-Difference, SARSA, Q-Learning & Expected SARSA in python or just view these algorithms on my Github: RL_from_scratch.

                      Conclusion

                      We learned some basic mathematical tools like metric spaces, complete metric spaces, Cauchy sequences, contraction mapping and the Banach fixed point theorem. Building upon all of this, we mathematically proved the unique optimality of the Bellman optimality equation for every MDP.

                      References and credits

                        * I used https://latex.codecogs.com/eqneditor/editor.php to generate images for LaTeX. (I wish medium allowed LaTeX)
                        * Wikipedia for providing all the formal definitions and theorems.
                        * The CMU 15–781 course
                        * The notes and video by Dr Daniel Murfet
                        * The free NPTEL course by Dr Balaraman Ravindran and IIT-Madras

                      Endnotes

                      I have tried to explain everything in as simple words as possible. If I was unclear or if I made some mistakes, please let me know in the responses. Also, there might be some mistakes in the latex equations. Please let me know if you find them. Last but not least, thanks for reading!

                  Machine Learning
                  Reinforcement Learning
                  Artificial Intelligence
                  Mathematics

                        --

                        --

                        7

                    Follow

                    Written by Vaibhav Kumar

                      446 Followers
                      · Writer for

                          Towards Data Science

                    Gradient Wave off Kanagawa

                    Follow

                    Help

                    Status

                    About

                    Careers

                    Press

                    Blog

                    Privacy

                    Terms

                    Text to speech

                    Teams

Open in app
            Sign up
              Sign in
                  Write
              Sign up
                Sign in
                        Mathematical Analysis of Reinforcement Learning — Bellman Optimality Equation
                        Metric Spaces, Cauchy Sequence, Contraction mapping and Banach Fixed Point Theorem
                                                Vaibhav Kumar
                                          ·
                                          Follow
                                          Published in
                                              Towards Data Science
                                          ·
                                        11 min read
                                          ·
                                        Jan 7, 2020
                                        --
                                      7
                                                    Listen
                                          Share
                      STOP READING THIS, WRONG INTUITIONS HERE.
                      Reinforcement learning has achieved remarkable results in playing games like StarCraft (AlphaStar) and Go (AlphaGO). At the core of all these successful projects lies — The Bellman optimality equation for Markov decision processes (MDPs).
                      Bellman Optimality Equation
                      The Bellman optimality equation is a recursive equation that can be solved using dynamic programming (DP) algorithms to find the optimal value function and the optimal policy. In this article, I will try to explain why the Bellman optimality equation can solve every MDP by providing an optimal policy and perform an easy (hopefully) mathematical analysis of the same.
                      Notations
                        * S is state space.
                        * V is the value function.
                        * V* is the optimal value function.
                        * V(s) is the value of state s.
                        * π is a policy.
                        * π* is the optimal policy.
                        * π(s) returns the action for state s under the policy π.
                        * P is the transition probability matrix.
                        * A is the action space.
                      Prerequisites
                      Despite my best efforts, the analysis is going to be fairly rigorous and I assume the readers to be familiar with the following perquisites:
                       1. Markov decision process (MDP)
                       2. Bellman Equation and how it’s solved using iterative methods.
                       3. Basics of RL — Value functions, rewards, policies, discount factor etc.
                       4. Linear algebra
                       5. Vector calculus
                      Overview
                      If you have studied RL and MDPs, you must’ve encountered the claim: “For each MDP, there is always at least one policy that is better than or equal to all other policies.” This occurs both in Sutton and Barto’s book as well as in the lecture series by David Silver. Reading/Hearing this makes the claim pretty intuitive, however, I had to dig in deeper and understand this in a more concrete manner. Therefore, in this article, I will mathematically prove the following theorem:
                      Theorem: For any finite MDP, there exists an optimal policy π* such that it is better than or equal to every other possible policy π.
                      Before finding the best policy, we need to understand the ordering of policies. When is one policy (π1) considered better than the other (π2)?
                      If the value of a state derived using π1 is better than or equal the value of a state derived using π2 for every state in the environment, then the policy π1 is said to be better than policy π2. Mathematically, this can be written as follows:
                      Comparison of policies
                      Now that we know how to compare policies, we need to prove that there always exists a policy that is better than all other policies.
                      We are going to prove this using the Banach fixed point theorem by showing that the Bellman optimality operator is a contraction over a complete metric space of real numbers with metric L-infinity norm. For this, we will first discuss the fixed point problem and complete metric spaces with respect to the Cauchy sequence.
                      The above paragraph sounds very intimidating but its actually going to be pretty easy and intuitive once we get past the basic terminologies. We will be discussing everything that is in bold in the above paragraph. Let’s follow a bottom-up approach and learn each concept and conquer our fears:
                      1. Fixed point problem
                      I am sure most of us are familiar with the problem of finding the roots of an equation. We solve for x such that a function f(x) = 0. However, in a fixed point problem, we solve for x such that f(x) = x. As the name suggests, x is a fixed point, it does not change even on the application of the function. A fixed point problem can be converted into a problem of finding roots by forming another function g(x) = f(x)-x = 0. In fact, even root finding problems can also be converted back to fixed point problems. However, it’s really easy to solve fixed point problems (for special cases) and that is what makes them incredibly interesting and useful (sans the computational overhead).
                      To solve a fixed point problem, choose a random starting value of x and repeatedly apply f(x) infinite times. If the function is convergent and you’re lucky, you will find the solution.
                      Mathematically, it’s pretty simple, lets first describe a notation:
                      f^n(x) notation
                      Now, if the function is convergent then it must converge to some value, say, x*. This value, x* is indeed the solution to the fixed point problem as shown below:
                      Let's choose some arbitrary value x0 and apply the function f(.) on x0 for infinite times to get x*, and then use that to solve the fixed point problem:
                      Solving the fixed point problem
                      The intuition behind this is pretty simple, if a function has converged at some point then the value of this function at that convergent point will be the convergent point itself. Therefore, the convergent point is the fixed point itself.
                      This can also be observed empirically through the notebook below:
                      2. Metric space
                      A metric space is simply a set with a metric defined to measure the distance between any two elements of the set. For example, the Euclidean space is a metric space with distance defined as the Euclidean distance in the set of real numbers. Therefore, a metric space M is represented as (X, d) where X is the set and d is some sort of metric. The metric d must satisfy the following properties:
                       1. Identify d(x,x) = 0
                       2. Non-Negativity d(x, y) >0
                       3. Symmetry: d(x,y) = d(y,x)
                       4. Triangular inequality: d(x,z) ≤ d(x,y)+d(y,z)
                      3. Cauchy sequence
                      For a metric space (X, d) the sequence of elements of the set X, (x1, x2, x3…. xn) is a Cauchy sequence if, for every positive real number ε, there exists an integer N such that the following equation holds:
                      Cauchy Sequence
                      The mathematical explanation here isn’t very intuitive and needlessly complex. In simple words, a sequence of elements of a metric space is Cauchy if this sequence converges at some point (the distance between them becomes constant). Another great explanation for this is given by this website and goes as follows: “for any small distance, there is a certain index past which any two terms are within that distance of each other, which captures the intuitive idea of the terms becoming close.” The idea of “terms become close” is the basic intuition behind convergence or the limit of the series.
                      4. Complete Metric space
                      A metric space (X, d) is complete if every possible Cauchy sequence of the elements in the set X converges to an element that also belongs to the set X. That is to say, the convergent limit of every Cauchy sequence of the elements of set lies in the set itself. Which is why it’s called “complete”.
                      5. Contraction
                      A function (or operator or mapping) defined on the elements of the metric space (X, d) is a contraction (or contractor) if there exists some constant γ∈[0,1) such that for any two elements of the metric space x1 and x2, the following condition holds:
                      Contraction mapping
                      This means that after applying the mapping f(.) on the elements x1 and x2, they got closer to each other by at least a factor γ. Also, the smallest value of such a constant γ is called the Lipschitz constant (this is an important constant for generative adversarial networks). Also, if γ=1, the mapping is no more a contraction but rather a short mapping. Intuitively, it can be observed that the sequential values of the elements are getting closer after applying the contraction mapping.
                      6. Banach fixed point theorem
                      This is the heart and soul of our proof. Informally, this theorem says that for a complete metric space, the application of a contractor on the elements of the set, again and again, would eventually get us to an optimal, unique value. We know:
                       1. Contractors bring the elements of the set together.
                       2. Applying this contractor, again and again, would get us a sequence. (Cauchy?)
                       3. The Cauchy sequences in a complete metric space always converge to a value that is part of the metric space.
                      Formally, this theorem can be formulated as:
                      Theorem: Let (X, d) be a complete metric space and a function f: X->X be a contractor then, f has a unique fixed point x*∈ X (i.e. f(x*)=x*) such that the sequence f(f(f(…f(x)))) converges to x*.
                      Now, to prove this mathematically, we need to prove both the uniqueness and existence of x*.
                       1. Uniqueness: We will prove this by contradiction. Let us assume that the convergence value is not unique and x1* and x2* are two values at which the sequence of contractor converges then, we will have:
                      x1* and x2* are optimal values, the contractor has converged here, the distance does not change anymore
                      Also, note that f is a contractor so it must hold the following property:
                      The contractor property
                      Now since γ∈[0,1), it’s impossible to satisfy both equation 1 and 2 simultaneously. Therefore our assumption must be wrong. Hence, by contradiction, x* must be unique.
                      2. Existence: Now that we’ve proved that x* is unique, we need to prove that x* exists. Let (x1, x2, x3, …. xn) be the sequence formed by repeatedly applying the contractor.
                      An element of our metric space
                      If we assume that the sequence (x1, x2, x3, …. xn) is Cauchy, we know for certain that this sequence will converge to some point, say, x*. Also, since the metric space is complete, this convergent point,x* will belong to the metric space (X,d). Now, we just need to prove that this sequence is Cauchy. We will do this by taking two elements of the set xn and xm such that m>>n and, m is very large, then by repeatedly applying the triangular inequality property of the metric d, we have:
                      Expanding the triangular inequality of the metric d
                      Now, since f is a contractor, we know that:
                      The repeated contractor inequality
                      We can further reduce d(xm, xn) as follows:
                      Now, by choosing n to be sufficiently large, we can make the RHS of the above equation less than any positive real number ε. Hence, the sequence (x1, x2, x3, …. xn) is Cauchy and an optimal x*∈X exists. This concludes the proof of the Banach fixed point theorem.
                      Coming back to the Bellman Optimality Equation
                      For the value function, V(s) we define a new operator, the optimal Bellman operator,B which takes in a value function and returns another value function. We define this operator as follows:
                      The Bellman Operator
                      It can easily be observed, B is a recursive operator. Therefore, this will generate a sequence of value functions. If we can show that B is indeed a contractor for some metric space (X,d) then by the Banach fixed point theorem, we can conclude that the repeated application of the optimal Bellman operator will eventually give a unique optimal value function using which an optimal (best) policy can be derived. Therefore, all our work now reduces to proving that B is a contractor. First, let's define the metric space as follows:
                      Metric space (X,d): The set X is the set of real numbers defined as follows:
                      Value function belongs to the set of real numbers. The X in the bottom shows the set for the complete value function while the one at the top shows the set the value of a single state.
                      For the metric, we use the L-infinity norm defined as follows:
                      L-infinity norm
                      According to this metric, the distance between the two value functions will be equal to the highest element-wise absolute difference between the two. Also, for finite MDPs with finite rewards, the value functions will always stay in the real space. It is impossible for the value function to not be in the real space, therefore, this finite space will always be complete.
                      Theorem: Bellman operator B is a contraction mapping in the finite space (R, L-infinity)
                      Proof: Let V1 and V2 be two value functions. Then:
                      Proof of B being a contraction
                        * In the second step above, we introduce inequality by replacing a’ by a for the second value function. This is because by replacing its optimal action a’ by some other action a, we have reduced its overall value thereby introducing an inequality.
                        * In the fourth step, we remove the L-infinity norm by taking the max over s’ (recall the definition of L-infinity with respect to value functions in our setting)
                        * In the final step, we remove the sigma because the sum of probabilities is always 1.
                      Finally, for the Bellman optimality equation, since γ∈[0,1) (let's ignore the possibility of γ=1 for now), the Bellman operator is, therefore, a contractor.
                      Now we know:
                       1. (R, L-infinity) is a complete metric space
                       2. Bellman operator B is a contractor
                      Hence, by the Banach fixed point theorem, we conclude that there exists a unique optimal value function V* for every MDP. Using this V*, we can derive the optimal policy π*.
                      Hence proved, for any finite MDP, there exists an optimal policy π* such that it is better than or equal to every other possible policy π.
                      Now, how to find this optimal policy and value function? One way is to just repeatedly apply the Bellman operator to a random initial value function to get the optimal function. But, this is computationally very expensive and often downright infeasible. Therefore, we use iterative methods like value and policy iteration or temporal difference methods like Q-Learning or SARSA. For more on this, please refer to my blog Reinforcement learning: Temporal-Difference, SARSA, Q-Learning & Expected SARSA in python or just view these algorithms on my Github: RL_from_scratch.
                      Conclusion
                      We learned some basic mathematical tools like metric spaces, complete metric spaces, Cauchy sequences, contraction mapping and the Banach fixed point theorem. Building upon all of this, we mathematically proved the unique optimality of the Bellman optimality equation for every MDP.
                      References and credits
                        * I used https://latex.codecogs.com/eqneditor/editor.php to generate images for LaTeX. (I wish medium allowed LaTeX)
                        * Wikipedia for providing all the formal definitions and theorems.
                        * The CMU 15–781 course
                        * The notes and video by Dr Daniel Murfet
                        * The free NPTEL course by Dr Balaraman Ravindran and IIT-Madras
                      Endnotes
                      I have tried to explain everything in as simple words as possible. If I was unclear or if I made some mistakes, please let me know in the responses. Also, there might be some mistakes in the latex equations. Please let me know if you find them. Last but not least, thanks for reading!
                  Machine Learning
                  Reinforcement Learning
                  Artificial Intelligence
                  Mathematics
                        --
                        --
                        7
                    Follow
                    Written by Vaibhav Kumar
                      446 Followers
                      · Writer for
                          Towards Data Science
                    Gradient Wave off Kanagawa
                    Follow
                    Help
                    Status
                    About
                    Careers
                    Press
                    Blog
                    Privacy
                    Terms
                    Text to speech
                    Teams
Mathematical Analysis of Reinforcement Learning — Bellman Equation
https://towardsdatascience.com/mathematical-analysis-of-reinforcement-learning-bellman-equation-ac9f0954e19f
https://ai.stackexchange.com/questions/11057/what-is-the-bellman-operator-in-reinforcement-learning
What is the Bellman operator in reinforcement learning?
2021-11-18
Nbro Nbro
<div><div>
<p>The notation I'll be using is from <a href="https://web.archive.org/web/20171118120953/http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/MDP.pdf">two</a> <a href="https://web.archive.org/web/20181127083007/http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/DP.pdf">different</a> lectures by David Silver and is also informed by <a href="http://chercheurs.lille.inria.fr/%7Eghavamza/RL-EC-Lille/Lecture3.pdf">these slides</a>.</p>
<p>The expected Bellman equation is
$$v_\pi(s) = \sum_{a\in \cal{A}} \pi(a|s) \left(\cal{R}_s^a + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^a v_\pi(s')\right) \tag 1$$</p>
<p>If we let
$$\cal{P}_{ss'}^\pi = \sum\limits_{a \in \cal{A}} \pi(a|s)\cal{P}_{ss'}^a \tag 2$$
and
$$\cal{R}_{s}^\pi = \sum\limits_{a \in \cal{A}} \pi(a|s)\cal{R}_{s}^a \tag 3$$
then we can rewrite $(1)$ as</p>
<p>$$v_\pi(s) = \cal{R}_s^\pi + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^\pi v_\pi(s') \tag 4$$</p>
<p>This can be written in matrix form</p>
<p>$$\left.
\begin{bmatrix}
v_\pi(1) \\
\vdots \\
v_\pi(n)
\end{bmatrix}=
\begin{bmatrix}
\cal{R}_1^\pi \\
\vdots \\
\cal{R}_n^\pi
\end{bmatrix}
+\gamma
\begin{bmatrix}
\cal{P}_{11}^\pi &amp; \dots &amp; \cal{P}_{1n}^\pi\\
\vdots &amp; \ddots &amp; \vdots\\
\cal{P}_{n1}^\pi &amp; \dots &amp; \cal{P}_{nn}^\pi
\end{bmatrix}
\begin{bmatrix}
v_\pi(1) \\
\vdots \\
v_\pi(n)
\end{bmatrix}
\right. \tag 5$$</p>
<p>Or, more compactly,</p>
<p>$$v_\pi = \cal{R}^\pi + \gamma \cal{P}^\pi v_\pi \tag 6$$</p>
<p>Notice that both sides of $(6)$ are $n$-dimensional vectors. Here $n=|\cal{S}|$ is the size of the state space. We can then define an operator $\cal{T}^\pi:\mathbb{R}^n\to\mathbb{R}^n$ as</p>
<p>$$\cal{T^\pi}(v) = \cal{R}^\pi + \gamma \cal{P}^\pi v \tag 7$$</p>
<p>for any $v\in \mathbb{R}^n$. This is the expected Bellman operator.</p>
<p>Similarly, you can rewrite the Bellman optimality equation</p>
<p>$$v_*(s) = \max_{a\in\cal{A}} \left(\cal{R}_s^a + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^a v_*(s')\right) \tag 8$$</p>
<p>as the Bellman optimality operator</p>
<p>$$\cal{T^*}(v) = \max_{a\in\cal{A}} \left(\cal{R}^a + \gamma \cal{P}^a v\right) \tag 9$$</p>
<p>The Bellman operators are "operators" in that they are mappings from one point to another within the vector space of state values, $\mathbb{R}^n$.</p>
<p>Rewriting the Bellman equations as operators is useful for proving that certain dynamic programming algorithms (e.g. policy iteration, value iteration) converge to a unique fixed point. This usefulness comes in the form of a body of existing work in operator theory, which allows us to make use of special properties of the Bellman operators.</p>
<p>Specifically, the fact that the Bellman operators are <a href="https://en.wikipedia.org/wiki/Contraction_%28operator_theory%29">contractions</a> gives the useful results that, for any policy $\pi$ and any initial vector $v$,</p>
<p>$$\lim_{k\to\infty}(\cal{T}^\pi)^k v = v_\pi \tag{10}$$</p>
<p>$$\lim_{k\to\infty}(\cal{T}^*)^k v = v_* \tag{11}$$</p>
<p>where $v_\pi$ is the value of policy $\pi$ and $v_*$ is the value of an optimal policy $\pi^*$. The proof is due to the <a href="https://en.wikipedia.org/wiki/Banach_fixed-point_theorem">contraction mapping theorem</a>.</p>
 </div>||||I|||| Stack Exchange Network

 Stack Exchange network consists of 181 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

 Visit Stack Exchange
 Loading…
 1. 
 + Tour Start here for a quick overview of the site
 + Help Center Detailed answers to any questions you might have
 + Meta Discuss the workings and policies of this site
 + About Us Learn more about Stack Overflow the company
 + Business Learn more about our products
 2. 

 3. current community

 + Artificial Intelligence
 help chat
 + Artificial Intelligence Meta

 your communities

 Sign up or log in to customize your list.

 more stack exchange communities

 company blog
 4. 
 5. Log in
 6. Sign up

 Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.

 Sign up to join this community
 Anybody can ask a question
 Anybody can answer
 The best answers are voted up and rise to the top
 1. Home
 2. 
 1. Public
 2. Questions
 3. Tags
 4. Users
 5. Companies
 6. Unanswered
 3. 
 1. Teams
 Stack Overflow for Teams – Start collaborating and sharing organizational knowledge. Create a free Team Why Teams?
 2. Teams
 3. Create free Team

 Teams

 Q&A for work

 Connect and share knowledge within a single location that is structured and easy to search.

 Learn more about Teams

 What is the Bellman operator in reinforcement learning?

 Ask Question
 Asked 3 years, 11 months ago
 Modified 1 year, 2 months ago
 Viewed 12k times
 26
 $\begingroup$

 In mathematics, the word operator can refer to several distinct but related concepts. An operator can be defined as a function between two vector spaces, it can be defined as a function where the domain and the codomain are the same, or it can be defined as a function from functions (which are vectors) to other functions (for example, the differential operator), that is, a high-order function (if you are familiar with functional programming).

 * What is the Bellman operator in reinforcement learning (RL)?
 * Why do we even need it?
 * How is the Bellman operator related to the Bellman equations in RL?
 * reinforcement-learning
 * terminology
 * math
 * bellman-equations
 * bellman-operators
 Share
 Improve this question
 Follow
 edited Nov 18, 2021 at 10:19
 Archana David
 257 2 2 silver badges 9 9 bronze badges
 asked Mar 6, 2019 at 14:07
 nbro nbro
 36.4k 11 11 gold badges 87 87 silver badges 162 162 bronze badges
 $\endgroup$
 2
 * $\begingroup$ A few papers related to this topic are Feature-Based Methods for Large Scale Dynamic Programming (by John N. Tsitsiklis and Benjamin Van Roy, 1996), An Analysis of Temporal-Difference Learning with Function Approximation (by John N. Tsitsiklis and Benjamin Van Roy, 1997) and Least-Squares Policy Iteration (by Michail G. Lagoudakis and Ronald Parr, 2003). $\endgroup$
 – nbro
 Mar 10, 2019 at 22:59
 * $\begingroup$ A few more related papers that I found are Generalized Markov Decision Processes: Dynamic-programming and Reinforcement-learning Algorithms (by Csaba Szepesvári and Michael L. Littman, 1997) and $\epsilon$-MDPs: Learning in Varying Environments (by István Szita, Bálint Takács, András Lörincz, 2002). $\endgroup$
 – nbro
 Mar 14, 2019 at 19:34
 Add a comment |

 1 Answer

 Sorted by: Reset to default
 Highest score (default) Date modified (newest first) Date created (oldest first)
 29
 $\begingroup$

 The notation I'll be using is from two different lectures by David Silver and is also informed by these slides.

 The expected Bellman equation is $$v_\pi(s) = \sum_{a\in \cal{A}} \pi(a|s) \left(\cal{R}_s^a + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^a v_\pi(s')\right) \tag 1$$

 If we let $$\cal{P}_{ss'}^\pi = \sum\limits_{a \in \cal{A}} \pi(a|s)\cal{P}_{ss'}^a \tag 2$$ and $$\cal{R}_{s}^\pi = \sum\limits_{a \in \cal{A}} \pi(a|s)\cal{R}_{s}^a \tag 3$$ then we can rewrite $(1)$ as

 $$v_\pi(s) = \cal{R}_s^\pi + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^\pi v_\pi(s') \tag 4$$

 This can be written in matrix form

 $$\left. \begin{bmatrix} v_\pi(1) \\ \vdots \\ v_\pi(n) \end{bmatrix}= \begin{bmatrix} \cal{R}_1^\pi \\ \vdots \\ \cal{R}_n^\pi \end{bmatrix} +\gamma \begin{bmatrix} \cal{P}_{11}^\pi & \dots & \cal{P}_{1n}^\pi\\ \vdots & \ddots & \vdots\\ \cal{P}_{n1}^\pi & \dots & \cal{P}_{nn}^\pi \end{bmatrix} \begin{bmatrix} v_\pi(1) \\ \vdots \\ v_\pi(n) \end{bmatrix} \right. \tag 5$$

 Or, more compactly,

 $$v_\pi = \cal{R}^\pi + \gamma \cal{P}^\pi v_\pi \tag 6$$

 Notice that both sides of $(6)$ are $n$ -dimensional vectors. Here $n=|\cal{S}|$ is the size of the state space. We can then define an operator $\cal{T}^\pi:\mathbb{R}^n\to\mathbb{R}^n$ as

 $$\cal{T^\pi}(v) = \cal{R}^\pi + \gamma \cal{P}^\pi v \tag 7$$

 for any $v\in \mathbb{R}^n$ . This is the expected Bellman operator.

 Similarly, you can rewrite the Bellman optimality equation

 $$v_*(s) = \max_{a\in\cal{A}} \left(\cal{R}_s^a + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^a v_*(s')\right) \tag 8$$

 as the Bellman optimality operator

 $$\cal{T^*}(v) = \max_{a\in\cal{A}} \left(\cal{R}^a + \gamma \cal{P}^a v\right) \tag 9$$

 The Bellman operators are "operators" in that they are mappings from one point to another within the vector space of state values, $\mathbb{R}^n$ .

 Rewriting the Bellman equations as operators is useful for proving that certain dynamic programming algorithms (e.g. policy iteration, value iteration) converge to a unique fixed point. This usefulness comes in the form of a body of existing work in operator theory, which allows us to make use of special properties of the Bellman operators.

 Specifically, the fact that the Bellman operators are contractions gives the useful results that, for any policy $\pi$ and any initial vector $v$ ,

 $$\lim_{k\to\infty}(\cal{T}^\pi)^k v = v_\pi \tag{10}$$

 $$\lim_{k\to\infty}(\cal{T}^*)^k v = v_* \tag{11}$$

 where $v_\pi$ is the value of policy $\pi$ and $v_*$ is the value of an optimal policy $\pi^*$ . The proof is due to the contraction mapping theorem.

 Share
 Improve this answer
 Follow
 edited Aug 12, 2020 at 17:40
 answered Mar 10, 2019 at 4:57
 Philip Raeisghasem Philip Raeisghasem
 2,008 8 8 silver badges 28 28 bronze badges
 $\endgroup$
 0
 Add a comment |

 You must log in to answer this question.

 Not the answer you're looking for? Browse other questions tagged
 * reinforcement-learning
 * terminology
 * math
 * bellman-equations
 * bellman-operators
 .

 * Featured on Meta
 * Accessibility Update: Colors

 Linked

 22
 Why doesn't Q-learning converge when using function approximation?
 8
 What is the proof that policy evaluation converges to the optimal solution?
 3
 How can we find the value function by solving a system of linear equations?
 4
 What is a generalized MDP?
 3
 What do the terms 'Bellman backup' and 'Bellman error' mean?
 1
 Why and how can the policy and value iteration methods converge to the OPTIMAL point?
 1
 How to derive matrix form of the Bellman operators?
 -2
 Why the optimal Bellman operator of a Q-function can be approximated by a single point

 Related

 3
 Apart from the state and state-action value functions, what are other examples of value functions used in RL?
 3
 Why is the max a non-expansive operator?
 29
 What is the difference between latent and embedding spaces?
 4
 How are afterstate value functions mathematically defined?
 0
 What is the exact difference between distributional semantics and distributed semantics?
 3
 What do the terms 'Bellman backup' and 'Bellman error' mean?
 0
 What are the Calculus books recommended for beginner to advanced researchers in artificial intelligence?
 2
 Are the domains of objective functions in AI always equals to $\mathbb{R}^D$ or subset of it?
 1
 Should I need to interpret the word "metric" in "performance metric" rigorously?
 2
 Are my proofs that the Bellman operators are contractions correct?

 Hot Network Questions

 * (Rev. 2) Command-line Tower of Hanoi game
 * Is Metapost still relevant in the age of Lua etc.?
 * Smallest Bit Rotation
 * If a balloon lies within gun range, why would a 21st century warplane shoot it with a missile, instead of rotary guns?
 * How big of an explosion can you make with a fusion reactor?
 * What exactly is the "clutch" doing on a Shimano M8100 rear derailleur?
 * how to modify a wooden fence to keep nubian goats from extending their heads over and eating dwarf tree on other side
 * Can poor grades in the bachelor's affect working in the academia?
 * How to Draw Such a Graph?
 * What does it mean for two properties of objects to be of the same class (or even identical)?
 * Why is b.pop(0) over 200 times slower than del b[0] for bytearray?
 * Juvenile SF Novel With Conflict between Earth Humans and Colonists
 * Some conjectures about prime gaps
 * What changes to Voyager could have been made?
 * Route from North Scotland to Stockholm with no air travel
 * What is the point of Feinting from a distance?
 * Agile Principles: What does it mean sustainable development?
 * Compute "negative" version of vector of vectors
 * How to adapt a spoke salvaged from a 26" wheel to a 24" one?
 * At what point does combining integrated circuits yield something patentable?
 * Do famous people really have the power to ruin the career of unrelated researchers?
 * What would be the next logical step after crossbows?
 * How would you say "mindless fun" in German
 * First Age "of the Sun" terminology
 more hot questions
 Question feed

 Subscribe to RSS

 Question feed

 To subscribe to this RSS feed, copy and paste this URL into your RSS reader.

 Artificial Intelligence

 * Tour
 * Help
 * Chat
 * Contact
 * Feedback

 Company

 * Stack Overflow
 * Teams
 * Advertising
 * Collectives
 * Talent
 * About
 * Press
 * Legal
 * Privacy Policy
 * Terms of Service
 * Cookie Settings
 * Cookie Policy

 Stack Exchange Network

 * Technology
 * Culture & recreation
 * Life & arts
 * Science
 * Professional
 * Business
 * API
 * Data
 * Blog
 * Facebook
 * Twitter
 * LinkedIn
 * Instagram

 Site design / logo © 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA. rev 2023.2.6.43220

 Your privacy

 By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.

 Accept all cookies Customize settings
The notation I'll be using is from two different lectures by David Silver and is also informed by these slides.
The expected Bellman equation is
$$v_\pi(s) = \sum_{a\in \cal{A}} \pi(a|s) \left(\cal{R}_s^a + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^a v_\pi(s')\right) \tag 1$$
If we let
$$\cal{P}_{ss'}^\pi = \sum\limits_{a \in \cal{A}} \pi(a|s)\cal{P}_{ss'}^a \tag 2$$
and
$$\cal{R}_{s}^\pi = \sum\limits_{a \in \cal{A}} \pi(a|s)\cal{R}_{s}^a \tag 3$$
then we can rewrite $(1)$ as
$$v_\pi(s) = \cal{R}_s^\pi + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^\pi v_\pi(s') \tag 4$$
This can be written in matrix form
$$\left.
\begin{bmatrix}
v_\pi(1) \\
\vdots \\
v_\pi(n)
\end{bmatrix}=
\begin{bmatrix}
\cal{R}_1^\pi \\
\vdots \\
\cal{R}_n^\pi
\end{bmatrix}
+\gamma
\begin{bmatrix}
\cal{P}_{11}^\pi &amp; \dots &amp; \cal{P}_{1n}^\pi\\
\vdots &amp; \ddots &amp; \vdots\\
\cal{P}_{n1}^\pi &amp; \dots &amp; \cal{P}_{nn}^\pi
\end{bmatrix}
\begin{bmatrix}
v_\pi(1) \\
\vdots \\
v_\pi(n)
\end{bmatrix}
\right. \tag 5$$
Or, more compactly,
$$v_\pi = \cal{R}^\pi + \gamma \cal{P}^\pi v_\pi \tag 6$$
Notice that both sides of $(6)$ are $n$-dimensional vectors. Here $n=|\cal{S}|$ is the size of the state space. We can then define an operator $\cal{T}^\pi:\mathbb{R}^n\to\mathbb{R}^n$ as
$$\cal{T^\pi}(v) = \cal{R}^\pi + \gamma \cal{P}^\pi v \tag 7$$
for any $v\in \mathbb{R}^n$. This is the expected Bellman operator.
Similarly, you can rewrite the Bellman optimality equation
$$v_*(s) = \max_{a\in\cal{A}} \left(\cal{R}_s^a + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^a v_*(s')\right) \tag 8$$
as the Bellman optimality operator
$$\cal{T^*}(v) = \max_{a\in\cal{A}} \left(\cal{R}^a + \gamma \cal{P}^a v\right) \tag 9$$
The Bellman operators are "operators" in that they are mappings from one point to another within the vector space of state values, $\mathbb{R}^n$.
Rewriting the Bellman equations as operators is useful for proving that certain dynamic programming algorithms (e.g. policy iteration, value iteration) converge to a unique fixed point. This usefulness comes in the form of a body of existing work in operator theory, which allows us to make use of special properties of the Bellman operators.
Specifically, the fact that the Bellman operators are contractions gives the useful results that, for any policy $\pi$ and any initial vector $v$,
$$\lim_{k\to\infty}(\cal{T}^\pi)^k v = v_\pi \tag{10}$$
$$\lim_{k\to\infty}(\cal{T}^*)^k v = v_* \tag{11}$$
where $v_\pi$ is the value of policy $\pi$ and $v_*$ is the value of an optimal policy $\pi^*$. The proof is due to the contraction mapping theorem.
What is the Bellman operator in reinforcement learning?
https://ai.stackexchange.com/questions/11057/what-is-the-bellman-operator-in-reinforcement-learning
https://en.wikipedia.org/wiki/Contraction_mapping
Contraction mapping - Wikipedia
2001-09-01
None
<div><div>
	<p>From Wikipedia, the free encyclopedia</p>
	</div><div><p> In <a href="/wiki/Mathematics">mathematics</a>, a <b>contraction mapping</b>, or <b>contraction</b> or <b>contractor</b>, on a <a href="/wiki/Metric_space">metric space</a> (<i>M</i>, <i>d</i>) is a <a href="/wiki/Function_(mathematics)">function</a> <i>f</i> from <i>M</i> to itself, with the property that there is some <a href="/wiki/Real_number">real number</a> such that for all <i>x</i> and <i>y</i> in <i>M</i>,
</p><dl><dd></dd></dl>
<p>The smallest such value of <i>k</i> is called the <b>Lipschitz constant</b> of <i>f</i>. Contractive maps are sometimes called <b>Lipschitzian maps</b>. If the above condition is instead satisfied for
<i>k</i> ≤ 1, then the mapping is said to be a <a href="/wiki/Non-expansive_map">non-expansive map</a>.
</p><p>More generally, the idea of a contractive mapping can be defined for maps between metric spaces. Thus, if (<i>M</i>, <i>d</i>) and (<i>N</i>, <i>d'</i>) are two metric spaces, then is a contractive mapping if there is a constant such that
</p>
<dl><dd></dd></dl>
<p>for all <i>x</i> and <i>y</i> in <i>M</i>.
</p><p>Every contraction mapping is <a href="/wiki/Lipschitz_continuous">Lipschitz continuous</a> and hence <a href="/wiki/Uniformly_continuous">uniformly continuous</a> (for a Lipschitz continuous function, the constant <i>k</i> is no longer necessarily less than 1).
</p><p>A contraction mapping has at most one <a href="/wiki/Fixed_point_(mathematics)">fixed point</a>. Moreover, the <a href="/wiki/Banach_fixed-point_theorem">Banach fixed-point theorem</a> states that every contraction mapping on a <a href="/wiki/Empty_set">non-empty</a> <a href="/wiki/Complete_metric_space">complete metric space</a> has a unique fixed point, and that for any <i>x</i> in <i>M</i> the <a href="/wiki/Iterated_function">iterated function</a> sequence <i>x</i>, <i>f</i> (<i>x</i>), <i>f</i> (<i>f</i> (<i>x</i>)), <i>f</i> (<i>f</i> (<i>f</i> (<i>x</i>))), ... converges to the fixed point. This concept is very useful for <a href="/wiki/Iterated_function_systems">iterated function systems</a> where <a href="/wiki/Convergence_proof_techniques#contraction_mapping">contraction mappings are often used</a>. Banach's fixed-point theorem is also applied in proving the existence of solutions of <a href="/wiki/Ordinary_differential_equation">ordinary differential equations</a>, and is used in one proof of the <a href="/wiki/Inverse_function_theorem">inverse function theorem</a>.<sup><a href="#cite_note-shifrin-1">[1]</a></sup>
</p><p>Contraction mappings play an important role in <a href="/wiki/Dynamic_programming">dynamic programming</a> problems.<sup><a href="#cite_note-2">[2]</a></sup><sup><a href="#cite_note-3">[3]</a></sup>
</p>
<h2>Firmly non-expansive mapping[<a href="/w/index.php?title=Contraction_mapping&amp;action=edit&amp;section=1">edit</a>]</h2>
<p>A non-expansive mapping with can be generalized to a <b>firmly non-expansive mapping</b> in a <a href="/wiki/Hilbert_space">Hilbert space</a> if the following holds for all <i>x</i> and <i>y</i> in :
</p>
<dl><dd></dd></dl>
<p>where
</p>
<dl><dd>.</dd></dl>
<p>This is a special case of averaged nonexpansive operators with .<sup><a href="#cite_note-4">[4]</a></sup> A firmly non-expansive mapping is always non-expansive, via the <a href="/wiki/Cauchy%E2%80%93Schwarz_inequality">Cauchy–Schwarz inequality</a>.
</p><p>The class of firmly non-expansive maps is closed under <a href="/wiki/Convex_combination">convex combinations</a>, but not compositions.<sup><a href="#cite_note-:0-5">[5]</a></sup> This class includes <a href="/wiki/Proximal_operator">proximal mappings</a> of proper, convex, lower-semicontinuous functions, hence it also includes orthogonal <a href="/wiki/Projection_(mathematics)">projections</a> onto non-empty closed <a href="/wiki/Convex_set">convex sets</a>. The class of firmly nonexpansive operators is equal to the set of resolvents of maximally <a href="/wiki/Monotonic_function#Monotonicity_in_functional_analysis">monotone operators</a>.<sup><a href="#cite_note-6">[6]</a></sup> Surprisingly, while iterating non-expansive maps has no guarantee to find a fixed point (e.g. multiplication by -1), firm non-expansiveness is sufficient to <a href="/wiki/Convergence_proof_techniques">guarantee global convergence</a> to a fixed point, provided a fixed point exists. More precisely, if , then for any initial point , iterating
</p><p>
</p><p>yields convergence to a fixed point . This convergence might be <a href="/wiki/Weak_convergence_(Hilbert_space)">weak</a> in an infinite-dimensional setting.<sup><a href="#cite_note-:0-5">[5]</a></sup>
</p>
<h2>Subcontraction map[<a href="/w/index.php?title=Contraction_mapping&amp;action=edit&amp;section=2">edit</a>]</h2>
<p>A <b>subcontraction map</b> or <b>subcontractor</b> is a map <i>f</i> on a metric space (<i>M</i>, <i>d</i>) such that
</p>
<dl><dd></dd>
<dd></dd></dl>
<p>If the <a href="/wiki/Image_(mathematics)">image</a> of a subcontractor <i>f</i> is <a href="/wiki/Compact_space">compact</a>, then <i>f</i> has a fixed point.<sup><a href="#cite_note-Gold17-7">[7]</a></sup>
</p>
<h2>Locally convex spaces[<a href="/w/index.php?title=Contraction_mapping&amp;action=edit&amp;section=3">edit</a>]</h2>
<p>In a <a href="/wiki/Locally_convex_space">locally convex space</a> (<i>E</i>, <i>P</i>) with <a href="/wiki/Topological_space">topology</a> given by a set <i>P</i> of <a href="/wiki/Seminorm">seminorms</a>, one can define for any <i>p</i> ∈ <i>P</i> a <i>p</i>-contraction as a map <i>f</i> such that there is some <i>k</i><sub><i>p</i></sub> &lt; 1 such that <i>p</i>(<i>f</i>(<i>x</i>) − <i>f</i>(<i>y</i>)) ≤ <i>k<sub>p</sub> p</i>(<i>x</i> − <i>y</i>). If <i>f</i> is a <i>p</i>-contraction for all <i>p</i> ∈ <i>P</i> and (<i>E</i>, <i>P</i>) is sequentially complete, then <i>f</i> has a fixed point, given as limit of any sequence <i>x</i><sub><i>n</i>+1</sub> = <i>f</i>(<i>x</i><sub><i>n</i></sub>), and if (<i>E</i>, <i>P</i>) is <a href="/wiki/Hausdorff_space">Hausdorff</a>, then the fixed point is unique.<sup><a href="#cite_note-8">[8]</a></sup>
</p>
<h2>See also[<a href="/w/index.php?title=Contraction_mapping&amp;action=edit&amp;section=4">edit</a>]</h2>
<ul><li><a href="/wiki/Short_map">Short map</a></li>
<li><a href="/wiki/Contraction_(operator_theory)">Contraction (operator theory)</a></li>
<li><a href="/wiki/Transformation_(function)">Transformation</a></li></ul>
<h2>References[<a href="/w/index.php?title=Contraction_mapping&amp;action=edit&amp;section=5">edit</a>]</h2>
<div><ol>
<li><b><a href="#cite_ref-shifrin_1-0">^</a></b> <cite>Shifrin, Theodore (2005). <i>Multivariable Mathematics</i>. Wiley. pp. 244–260. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-0-471-52638-4"><bdi>978-0-471-52638-4</bdi></a>.</cite>
</li>
<li><b><a href="#cite_ref-2">^</a></b> <cite>Denardo, Eric V. (1967). "Contraction Mappings in the Theory Underlying Dynamic Programming". <i>SIAM Review</i>. <b>9</b> (2): 165–177. <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/1967SIAMR...9..165D">1967SIAMR...9..165D</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1137%2F1009030">10.1137/1009030</a>.</cite>
</li>
<li><b><a href="#cite_ref-3">^</a></b> <cite><a href="/wiki/Nancy_Stokey">Stokey, Nancy L.</a>; <a href="/wiki/Robert_Lucas_Jr.">Lucas, Robert E.</a> (1989). <a href="https://www.google.com/books/edition/_/BgQ3AwAAQBAJ?hl=en&amp;gbpv=1&amp;pg=PA49"><i>Recursive Methods in Economic Dynamics</i></a>. Cambridge: Harvard University Press. pp. 49–55. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-0-674-75096-8"><bdi>978-0-674-75096-8</bdi></a>.</cite>
</li>
<li><b><a href="#cite_ref-4">^</a></b> <cite>Combettes, Patrick L. (2004). "Solving monotone inclusions via compositions of nonexpansive averaged operators". <i><a href="/w/index.php?title=Optimization_(journal)&amp;action=edit&amp;redlink=1">Optimization</a></i>. <b>53</b> (5–6): 475–504. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1080%2F02331930412331327157">10.1080/02331930412331327157</a>.</cite>
</li>
<li>^ <a href="#cite_ref-:0_5-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-:0_5-1"><sup><i><b>b</b></i></sup></a> <cite>Bauschke, Heinz H. (2017). <i>Convex Analysis and Monotone Operator Theory in Hilbert Spaces</i>. New York: Springer.</cite>
</li>
<li><b><a href="#cite_ref-6">^</a></b> <cite>Combettes, Patrick L. (July 2018). "Monotone operator theory in convex optimization". <i>Mathematical Programming</i>. <b>B170</b>: 177–206. <a href="/wiki/ArXiv_(identifier)">arXiv</a>:<a href="//arxiv.org/abs/1802.02694">1802.02694</a>. <a href="/wiki/Bibcode_(identifier)">Bibcode</a>:<a href="https://ui.adsabs.harvard.edu/abs/2018arXiv180202694C">2018arXiv180202694C</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1007%2Fs10107-018-1303-3">10.1007/s10107-018-1303-3</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:49409638">49409638</a>.</cite>
</li>
<li><b><a href="#cite_ref-Gold17_7-0">^</a></b> <cite>Goldstein, A.A. (1967). <i>Constructive real analysis</i>. Harper’s Series in Modern Mathematics. New York-Evanston-London: Harper and Row. p. 17. <a href="/wiki/Zbl_(identifier)">Zbl</a> <a href="//zbmath.org/?format=complete&amp;q=an:0189.49703">0189.49703</a>.</cite>
</li>
<li><b><a href="#cite_ref-8">^</a></b> <cite>Cain, G. L., Jr.; <a href="/wiki/Zuhair_Nashed">Nashed, M. Z.</a> (1971). <a href="https://doi.org/10.2140%2Fpjm.1971.39.581">"Fixed Points and Stability for a Sum of Two Operators in Locally Convex Spaces"</a>. <i>Pacific Journal of Mathematics</i>. <b>39</b> (3): 581–592. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.2140%2Fpjm.1971.39.581">10.2140/pjm.1971.39.581</a>.</cite>
</li>
</ol></div>
<h2>Further reading[<a href="/w/index.php?title=Contraction_mapping&amp;action=edit&amp;section=6">edit</a>]</h2>
<ul><li><cite>Istratescu, Vasile I. (1981). <i>Fixed Point Theory : An Introduction</i>. Holland: D.Reidel. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-90-277-1224-0"><bdi>978-90-277-1224-0</bdi></a>.</cite> provides an undergraduate level introduction.</li>
<li><cite>Granas, Andrzej; <a href="/wiki/James_Dugundji">Dugundji, James</a> (2003). <i>Fixed Point Theory</i>. New York: Springer-Verlag. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-0-387-00173-9"><bdi>978-0-387-00173-9</bdi></a>.</cite></li>
<li><cite>Kirk, William A.; <a href="/wiki/Brailey_Sims">Sims, Brailey</a> (2001). <i>Handbook of Metric Fixed Point Theory</i>. London: Kluwer Academic. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-0-7923-7073-4"><bdi>978-0-7923-7073-4</bdi></a>.</cite></li>
<li><cite>Naylor, Arch W.; Sell, George R. (1982). <a href="https://books.google.com/books?id=t3SXs4-KrE0C&amp;pg=PA125"><i>Linear Operator Theory in Engineering and Science</i></a>. Applied Mathematical Sciences. Vol. 40 (Second ed.). New York: Springer. pp. 125–134. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-0-387-90748-2"><bdi>978-0-387-90748-2</bdi></a>.</cite></li></ul>
</div>||||I|||| Jump to content
 Toggle sidebar
 Search
 * Create account
 Personal tools
 * Create account
 * Log in

 Pages for logged out editors learn more

 * Talk
 * Contributions
 Navigation
 * Main page
 * Contents
 * Current events
 * Random article
 * About Wikipedia
 * Contact us
 * Donate
 Contribute
 * Help
 * Learn to edit
 * Community portal
 * Recent changes
 * Upload file
 Tools
 * What links here
 * Related changes
 * Upload file
 * Special pages
 * Permanent link
 * Page information
 * Cite this page
 * Wikidata item
 Print/export
 * Download as PDF
 * Printable version
 Languages
 On this Wikipedia the language links are at the top of the page across from the article title. Go to top.

 Contents

 move to sidebar hide
 * (Top)
 * 1 Firmly non-expansive mapping
 * 2 Subcontraction map
 * 3 Locally convex spaces
 * 4 See also
 * 5 References
 * 6 Further reading
 Toggle the table of contents

 Contraction mapping

 16 languages
 * Català
 * Čeština
 * Deutsch
 * Español
 * Français
 * 한국어
 * Bahasa Indonesia
 * Italiano
 * 日本語
 * Polski
 * Русский
 * Српски / srpski
 * Suomi
 * Svenska
 * Українська
 * 中文
 Edit links
 * Article
 * Talk
 English
 * Read
 * Edit
 * View history
 More
 * Read
 * Edit
 * View history
 From Wikipedia, the free encyclopedia

 In mathematics, a contraction mapping, or contraction or contractor, on a metric space (M, d) is a function f from M to itself, with the property that there is some real number such that for all x and y in M,

 The smallest such value of k is called the Lipschitz constant of f. Contractive maps are sometimes called Lipschitzian maps. If the above condition is instead satisfied for k ≤ 1, then the mapping is said to be a non-expansive map.

 More generally, the idea of a contractive mapping can be defined for maps between metric spaces. Thus, if (M, d) and (N, d') are two metric spaces, then is a contractive mapping if there is a constant such that

 for all x and y in M.

 Every contraction mapping is Lipschitz continuous and hence uniformly continuous (for a Lipschitz continuous function, the constant k is no longer necessarily less than 1).

 A contraction mapping has at most one fixed point. Moreover, the Banach fixed-point theorem states that every contraction mapping on a non-empty complete metric space has a unique fixed point, and that for any x in M the iterated function sequence x, f (x), f (f (x)), f (f (f (x))), ... converges to the fixed point. This concept is very useful for iterated function systems where contraction mappings are often used. Banach's fixed-point theorem is also applied in proving the existence of solutions of ordinary differential equations, and is used in one proof of the inverse function theorem.[1]

 Contraction mappings play an important role in dynamic programming problems.[2][3]

 Firmly non-expansive mapping [ edit ]

 A non-expansive mapping with can be generalized to a firmly non-expansive mapping in a Hilbert space if the following holds for all x and y in :

 where

 .

 This is a special case of averaged nonexpansive operators with .[4] A firmly non-expansive mapping is always non-expansive, via the Cauchy–Schwarz inequality.

 The class of firmly non-expansive maps is closed under convex combinations, but not compositions.[5] This class includes proximal mappings of proper, convex, lower-semicontinuous functions, hence it also includes orthogonal projections onto non-empty closed convex sets. The class of firmly nonexpansive operators is equal to the set of resolvents of maximally monotone operators.[6] Surprisingly, while iterating non-expansive maps has no guarantee to find a fixed point (e.g. multiplication by -1), firm non-expansiveness is sufficient to guarantee global convergence to a fixed point, provided a fixed point exists. More precisely, if , then for any initial point , iterating

 yields convergence to a fixed point . This convergence might be weak in an infinite-dimensional setting.[5]

 Subcontraction map [ edit ]

 A subcontraction map or subcontractor is a map f on a metric space (M, d) such that

 If the image of a subcontractor f is compact, then f has a fixed point.[7]

 Locally convex spaces [ edit ]

 In a locally convex space (E, P) with topology given by a set P of seminorms, one can define for any p ∈ P a p-contraction as a map f such that there is some kp < 1 such that p ( f ( x ) − f ( y )) ≤ kp p ( x − y ) . If f is a p-contraction for all p ∈ P and (E, P) is sequentially complete, then f has a fixed point, given as limit of any sequence xn+1 = f(xn), and if (E, P) is Hausdorff, then the fixed point is unique.[8]

 See also [ edit ]

 * Short map
 * Contraction (operator theory)
 * Transformation

 References [ edit ]

 1. ^ Shifrin, Theodore (2005). Multivariable Mathematics. Wiley. pp. 244–260. ISBN 978-0-471-52638-4.
 2. ^ Denardo, Eric V. (1967). "Contraction Mappings in the Theory Underlying Dynamic Programming". SIAM Review. 9 (2): 165–177. Bibcode:1967SIAMR...9..165D. doi:10.1137/1009030.
 3. ^ Stokey, Nancy L.; Lucas, Robert E. (1989). Recursive Methods in Economic Dynamics. Cambridge: Harvard University Press. pp. 49–55. ISBN 978-0-674-75096-8.
 4. ^ Combettes, Patrick L. (2004). "Solving monotone inclusions via compositions of nonexpansive averaged operators". Optimization. 53 (5–6): 475–504. doi:10.1080/02331930412331327157.
 5. ^ a b Bauschke, Heinz H. (2017). Convex Analysis and Monotone Operator Theory in Hilbert Spaces. New York: Springer.
 6. ^ Combettes, Patrick L. (July 2018). "Monotone operator theory in convex optimization". Mathematical Programming. B170: 177–206. arXiv:1802.02694. Bibcode:2018arXiv180202694C. doi:10.1007/s10107-018-1303-3. S2CID 49409638.
 7. ^ Goldstein, A.A. (1967). Constructive real analysis. Harper’s Series in Modern Mathematics. New York-Evanston-London: Harper and Row. p. 17. Zbl 0189.49703.
 8. ^ Cain, G. L., Jr.; Nashed, M. Z. (1971). "Fixed Points and Stability for a Sum of Two Operators in Locally Convex Spaces". Pacific Journal of Mathematics. 39 (3): 581–592. doi:10.2140/pjm.1971.39.581.

 Further reading [ edit ]

 * Istratescu, Vasile I. (1981). Fixed Point Theory : An Introduction. Holland: D.Reidel. ISBN 978-90-277-1224-0. provides an undergraduate level introduction.
 * Granas, Andrzej; Dugundji, James (2003). Fixed Point Theory. New York: Springer-Verlag. ISBN 978-0-387-00173-9.
 * Kirk, William A.; Sims, Brailey (2001). Handbook of Metric Fixed Point Theory. London: Kluwer Academic. ISBN 978-0-7923-7073-4.
 * Naylor, Arch W.; Sell, George R. (1982). Linear Operator Theory in Engineering and Science. Applied Mathematical Sciences. Vol. 40 (Second ed.). New York: Springer. pp. 125–134. ISBN 978-0-387-90748-2.
 Retrieved from "https://en.wikipedia.org/w/index.php?title=Contraction_mapping&oldid=1134814273"
 Categories:
 * Metric geometry
 * Fixed points (mathematics)
 Hidden categories:
 * Articles with short description
 * Short description matches Wikidata
 * This page was last edited on 20 January 2023, at 20:42 (UTC) .
 * Text is available under the Creative Commons Attribution-ShareAlike License 3.0; additional terms may apply. By using this site, you agree to the Terms of Use and Privacy Policy. Wikipedia® is a registered trademark of the Wikimedia Foundation, Inc., a non-profit organization.
 * Privacy policy
 * About Wikipedia
 * Disclaimers
 * Contact Wikipedia
 * Mobile view
 * Developers
 * Statistics
 * Cookie statement
 * 
 * 
From Wikipedia, the free encyclopedia
	   In mathematics, a contraction mapping, or contraction or contractor, on a metric space (M, d) is a function f from M to itself, with the property that there is some real number such that for all x and y in M,
    
The smallest such value of k is called the Lipschitz constant of f. Contractive maps are sometimes called Lipschitzian maps. If the above condition is instead satisfied for
k ≤ 1, then the mapping is said to be a non-expansive map.
 More generally, the idea of a contractive mapping can be defined for maps between metric spaces. Thus, if (M, d) and (N, d') are two metric spaces, then is a contractive mapping if there is a constant such that
   
for all x and y in M.
 Every contraction mapping is Lipschitz continuous and hence uniformly continuous (for a Lipschitz continuous function, the constant k is no longer necessarily less than 1).
 A contraction mapping has at most one fixed point. Moreover, the Banach fixed-point theorem states that every contraction mapping on a non-empty complete metric space has a unique fixed point, and that for any x in M the iterated function sequence x, f (x), f (f (x)), f (f (f (x))), ... converges to the fixed point. This concept is very useful for iterated function systems where contraction mappings are often used. Banach's fixed-point theorem is also applied in proving the existence of solutions of ordinary differential equations, and is used in one proof of the inverse function theorem. [1] 
 Contraction mappings play an important role in dynamic programming problems. [2]   [3] 
Firmly non-expansive mapping[edit]
A non-expansive mapping with can be generalized to a firmly non-expansive mapping in a Hilbert space if the following holds for all x and y in :
   
where
 . 
This is a special case of averaged nonexpansive operators with . [4]  A firmly non-expansive mapping is always non-expansive, via the Cauchy–Schwarz inequality.
 The class of firmly non-expansive maps is closed under convex combinations, but not compositions. [5]  This class includes proximal mappings of proper, convex, lower-semicontinuous functions, hence it also includes orthogonal projections onto non-empty closed convex sets. The class of firmly nonexpansive operators is equal to the set of resolvents of maximally monotone operators. [6]  Surprisingly, while iterating non-expansive maps has no guarantee to find a fixed point (e.g. multiplication by -1), firm non-expansiveness is sufficient to guarantee global convergence to a fixed point, provided a fixed point exists. More precisely, if , then for any initial point , iterating
 
 yields convergence to a fixed point . This convergence might be weak in an infinite-dimensional setting. [5] 
Subcontraction map[edit]
A subcontraction map or subcontractor is a map f on a metric space (M, d) such that
  
  
If the image of a subcontractor f is compact, then f has a fixed point. [7] 
Locally convex spaces[edit]
In a locally convex space (E, P) with topology given by a set P of seminorms, one can define for any p ∈ P a p-contraction as a map f such that there is some k  p  &lt; 1 such that p(f(x) − f(y)) ≤ kp p(x − y). If f is a p-contraction for all p ∈ P and (E, P) is sequentially complete, then f has a fixed point, given as limit of any sequence x  n+1 = f(x  n ), and if (E, P) is Hausdorff, then the fixed point is unique. [8] 
See also[edit]
  Short map 
 Contraction (operator theory) 
 Transformation  
References[edit]
 
  ^  Shifrin, Theodore (2005). Multivariable Mathematics. Wiley. pp. 244–260. ISBN  978-0-471-52638-4 .
  ^  Denardo, Eric V. (1967). "Contraction Mappings in the Theory Underlying Dynamic Programming". SIAM Review. 9 (2): 165–177. Bibcode:1967SIAMR...9..165D. doi:10.1137/1009030.
  ^   Stokey, Nancy L.; Lucas, Robert E. (1989).  Recursive Methods in Economic Dynamics . Cambridge: Harvard University Press. pp. 49–55. ISBN  978-0-674-75096-8 .
  ^  Combettes, Patrick L. (2004). "Solving monotone inclusions via compositions of nonexpansive averaged operators".  Optimization . 53 (5–6): 475–504. doi:10.1080/02331930412331327157.
^    a       b    Bauschke, Heinz H. (2017). Convex Analysis and Monotone Operator Theory in Hilbert Spaces. New York: Springer.
  ^  Combettes, Patrick L. (July 2018). "Monotone operator theory in convex optimization". Mathematical Programming. B170: 177–206. arXiv:1802.02694. Bibcode:2018arXiv180202694C. doi:10.1007/s10107-018-1303-3. S2CID 49409638.
  ^  Goldstein, A.A. (1967). Constructive real analysis. Harper’s Series in Modern Mathematics. New York-Evanston-London: Harper and Row. p. 17. Zbl 0189.49703.
  ^  Cain, G. L., Jr.; Nashed, M. Z. (1971). "Fixed Points and Stability for a Sum of Two Operators in Locally Convex Spaces". Pacific Journal of Mathematics. 39 (3): 581–592. doi:10.2140/pjm.1971.39.581.
 
Further reading[edit]
  Istratescu, Vasile I. (1981). Fixed Point Theory : An Introduction. Holland: D.Reidel. ISBN  978-90-277-1224-0 . provides an undergraduate level introduction.
 Granas, Andrzej; Dugundji, James (2003). Fixed Point Theory. New York: Springer-Verlag. ISBN  978-0-387-00173-9 . 
 Kirk, William A.; Sims, Brailey (2001). Handbook of Metric Fixed Point Theory. London: Kluwer Academic. ISBN  978-0-7923-7073-4 . 
 Naylor, Arch W.; Sell, George R. (1982).  Linear Operator Theory in Engineering and Science . Applied Mathematical Sciences. Vol. 40 (Second ed.). New York: Springer. pp. 125–134. ISBN  978-0-387-90748-2 .
Contraction mapping - Wikipedia
https://en.wikipedia.org/wiki/Contraction_mapping
https://www.analyticsvidhya.com/blog/2021/02/introduction-to-reinforcement-learning-for-beginners/
Introduction to Reinforcement Learning for Beginners
2021-02-21
Prathima Kadari
<div><div>
<p>Reinforcement Learning, seems intriguing, right? Here in this article, we will see what it is and why is it so much talked about these days. This acts as a guide to reinforcement learning for beginners. Reinforcement Learning is definitely one of the evident research areas at present which has a good boom to emerge in the coming future and its popularity is increasing day by day. Lets, get it started.</p>
<p>It is basically the concept where machines can teach themselves depending upon the results of their own actions. Without further delay, let’s start.</p>
<h2><b>What is Reinforcement Learning?</b></h2>
<p>Reinforcement Learning is a part of machine learning. Here, agents are self-trained on reward and punishment mechanisms. It’s about taking the best possible action or path to gain maximum rewards and minimum punishment through observations in a specific situation. It acts as a signal to positive and negative behaviors. Essentially an agent (or several) is built that can perceive and interpret the environment in which is placed, furthermore, it can take actions and interact with it.</p>
<div>
<figure>
<div><p></p><p>Basic Diagram of Reinforcement Learning – KDNuggets</p></div></figure>
</div>
<p>To know the meaning of reinforcement learning, let’s go through the formal definition.</p>
<p>Reinforcement learning, a type of machine learning, in which agents take actions in an environment aimed at maximizing their cumulative rewards – NVIDIA</p>
<p>Reinforcement learning (RL) is based on rewarding desired behaviors or punishing undesired ones. Instead of one input producing one output, the algorithm produces a variety of outputs and is trained to select the right one based on certain variables – Gartner</p>
<p>It is a type of machine learning technique where a computer agent learns to perform a task through repeated trial and error interactions with a dynamic environment. This learning approach enables the agent to make a series of decisions that maximize a reward metric for the task without human intervention and without being explicitly programmed to achieve the task – Mathworks</p>
<p>The above definitions are technically provided by experts in that field however for someone who is starting with reinforcement learning, but these definitions might feel a little bit difficult. As this is a reinforcement learning guide for beginners, let’s create our reinforcement learning definition in an easier way.</p>
<h4><b>Simplified Definition of Reinforcement Learning</b></h4>
<p>Through a series of Trial and Error methods, an agent keeps learning continuously in an interactive environment from its own actions and experiences. The only goal of it is to find a suitable action model which would increase the total cumulative reward of the agent. It learns via interaction and feedback.</p>
<p>Well, that’s the definition of reinforcement learning. Now how we come to this definition, how a machine learns and how it can solve complex problems in the world through reinforcement learning, is something we are going to see further.</p>
<h4><b>Explanation to Reinforcement Learning</b></h4>
<p>How does reinforcement learning work? Well, let me explain with an example.<b><br />
</b></p>
<div>
<figure>
<div><p></p><p>Reinforcement Learning Example – KDNuggets</p></div></figure>
</div>
<h4>Here what do you see?</h4>
<p>You can see a dog and a master. Let’s imagine you are training your dog to get the stick. Each time the dog gets a stick successfully, you offered him a feast (a bone let’s say). Eventually, the dog understands the pattern, that whenever the master throws a stick, it should get it as early as it can to gain a reward (a bone) from a master in a lesser time.</p>
<h2><b>Terminologies used in Reinforcement Learning</b></h2>
<div>
<figure>
<div><p></p><p>Terminologies in RL – Techvidvan</p></div></figure>
</div>
<p><b>Agent</b> – is the sole decision-maker and learner</p>
<p><b>Environment</b> – a physical world where an agent learns and decides the actions to be performed</p>
<p><b>Action</b> – a list of action which an agent can perform</p>
<p><b>State</b> – the current situation of the agent in the environment</p>
<p><b>Reward</b> – For each selected action by agent, the environment gives a reward. It’s usually a scalar value and nothing but feedback from the environment</p>
<p><b>Policy</b> – the agent prepares strategy(decision-making) to map situations to actions.</p>
<p><b>Value Function</b> – The value of state shows up the reward achieved starting from the state until the policy is executed</p>
<p><b>Model</b> – Every RL agent doesn’t use a model of its environment. The agent’s view maps state-action pairs probability distributions over the states</p>
<h2><b> Reinforcement Learning Workflow</b></h2>
<div>
<figure>
<div><p></p><p>Reinforcement Learning Workflow – KDNuggets</p></div></figure>
</div>
<p>– Create the Environment</p>
<p>– Define the reward</p>
<p>– Create the agent <b><br />
</b></p>
<p>– Train and validate the agent</p>
<p>– Deploy the policy</p>
<h2><b>How is reinforcement learning different from supervised learning?</b></h2>
<p>In supervised learning, the model is trained with a training dataset that has a correct answer key. The decision is done on the initial input given as it has all the data that’s required to train the machine. The decisions are independent of each other so each decision is represented through a label. Example: Object Recognition</p>
<div>
<figure>
<div><p></p><p>Difference between Supervised and Reinforcement Learning – purestudy</p></div></figure>
</div>
<p>In reinforcement learning, there isn’t any answer and the reinforcement agent decides what to be done to perform the required task. As the training dataset isn’t available, the agent had to learn from its experience. It’s all about compiling the decisions in a sequential manner. To be said in simpler words, the output relies on the current input state and the next input relies on the output of the previous input. We give labels to the sequence of dependent decisions. Decisions are dependent. Example: Chess Game</p>
<h2><b>Characteristics of Reinforcement Learning</b></h2>
<p>– No supervision, only a real value or reward signal</p>
<p>– Decision making is sequential</p>
<p>– Time plays a major role in reinforcement problems</p>
<p>– Feedback isn’t prompt but delayed</p>
<p>– The following data it receives is determined by the agent’s actions</p>
<h5><b>Reinforcement Learning Algorithms</b></h5>
<p>There are 3 approaches to implement reinforcement learning algorithms</p>
<div>
<figure>
<div><p></p><p>Reinforcement Learning Algorithms – AISummer</p></div></figure>
</div>
<p><b>Value-Based</b> – The main goal of this method is to maximize a value function. Here, an agent through a policy expects a long-term return of the current states.</p>
<p><b>Policy-Based</b> – In policy-based, you enable to come up with a strategy that helps to gain maximum rewards in the future through possible actions performed in each state. Two types of policy-based methods are deterministic and stochastic.</p>
<p><b>Model-Based</b> – In this method, we need to create a virtual model for the agent to help in learning to perform in each specific environment</p>
<h2><b>Types of Reinforcement Learning</b></h2>
<p>There are two types :</p>
<div>
<figure>
<div><p></p><p>Reinforcement Theory Example – Tutorialspoint</p></div></figure>
</div>
<h4><b>1. Positive Reinforcement</b></h4>
<p>Positive reinforcement is defined as when an event, occurs due to specific behavior, increases the strength and frequency of the behavior. It has a positive impact on behavior.</p>
<p>Advantages</p>
<p>– Maximizes the performance of an action</p>
<p>– Sustain change for a longer period</p>
<p>Disadvantage</p>
<p>– Excess reinforcement can lead to an overload of states which would minimize the results.</p>
<h4><b>2. Negative Reinforcement<br />
</b></h4>
<p>Negative Reinforcement is represented as the strengthening of a behavior. In other ways, when a negative condition is barred or avoided, it tries to stop this action in the future.</p>
<h4>Advantages</h4>
<p>– Maximized behavior</p>
<p>– Provide a decent to minimum standard of performance</p>
<h4>Disadvantage</h4>
<p>– It just limits itself enough to meet up a minimum behavior</p>
<h2><b>Widely used models for reinforcement learning</b></h2>
<p>1. <b>Markov Decision Process (MDP’s)</b> – are mathematical frameworks for mapping solutions in RL. The set of parameters that include Set of finite states – S, Set of possible Actions in each state – A, Reward – R, Model – T, Policy – π. The outcome of deploying an action to a state doesn’t depend on previous actions or states but on current action and state.</p>
<div>
<figure>
<div><p></p><p>Markov Decision Process – Geeks4geeks</p></div></figure>
</div>
<p>2. <b>Q Learning</b> – it’s a value-based model free approach for supplying information to intimate which action an agent should perform. It revolves around the notion of updating Q values which shows the value of doing action A in state S. Value update rule is the main aspect of the Q-learning algorithm.</p>
<div>
<figure></figure>
</div>
<h2>QLearning – Freecodecamp</h2>
<h4><b>Practical Applications of reinforcement learning</b></h4>
<p>– Robotics for Industrial Automation</p>
<p>– Text summarization engines, dialogue agents (text, speech), gameplays</p>
<p>– Autonomous Self Driving Cars</p>
<p>– Machine Learning and Data Processing</p>
<p>– Training system which would issue custom instructions and materials with respect to the requirements of students</p>
<p>– AI Toolkits, Manufacturing, Automotive, Healthcare, and Bots</p>
<p>– Aircraft Control and Robot Motion Control</p>
<p>– Building artificial intelligence for computer games</p>
<h2><b>Conclusion</b></h2>
<p>The conclusion for this topic is nothing but helping us to discover which action could yield the highest reward for a longer time. Realistic environments can have partial observability and be non-stationary as well. It isn’t very useful to apply when you have hands-on enough data to solve the problem using supervised learning. The main challenge of this method is that parameters could affect the speed of the learning.</p>
<p>Hope now you got the feel and certain level of the description on Reinforcement Learning. Thanks for your time.</p>
<h2><b>About Me</b></h2>
<p>I am Prathima Kadari, a former embedded engineer, working on leveraging my knowledge and upgrading my skills.</p>
<p>Please feel free to connect with me on <b>https://www.linkedin.com/in/prathima-kadari</b></p>
<p><em><b>The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.</b></em></p>
</div>||||I|||| search
 * Start Here
 + Machine Learning
 + Deep Learning
 + NLP
 * Articles
 + Guides
 + Machine Learning
 + Deep Learning
 + NLP
 + Computer Vision
 + Data Visualization
 + Interview Questions
 + More
 Infographics Jobs Podcasts E-Books For Companies Datahack Summit DSAT Glossary Archive
 Write an Article
 * Courses
 + Certified AI & ML BlackBelt Plus
 + Data Science Immersive Bootcamp
 All Courses
 * Blogathon
 * Data Hour
 * 
 * Write an Article Creators Club
 * Login/Signup
 1. Home
 search
 * Start Here
 + Machine Learning
 + Deep Learning
 + NLP
 * Articles
 + Guides
 + Machine Learning
 + Deep Learning
 + NLP
 + Computer Vision
 + Data Visualization
 + Interview Questions
 + More
 Infographics Jobs Podcasts E-Books For Companies Datahack Summit DSAT Glossary Archive
 Write an Article
 * Courses
 + Certified AI & ML BlackBelt Plus
 + Data Science Immersive Bootcamp
 All Courses
 * Blogathon
 * Data Hour
 * 
 * Write an Article Creators Club
 * Login/Signup

 Introduction to Reinforcement Learning for Beginners

 * Facebook
 * Twitter
 * Linkedin
 Prathima Kadari — Published On February 21, 2021 and Last Modified On February 23rd, 2021
 Beginner Reinforcement Learning

 This article was published as a part of the Data Science Blogathon.

 Introduction

 Reinforcement Learning, seems intriguing, right? Here in this article, we will see what it is and why is it so much talked about these days. This acts as a guide to reinforcement learning for beginners. Reinforcement Learning is definitely one of the evident research areas at present which has a good boom to emerge in the coming future and its popularity is increasing day by day. Lets, get it started.

 It is basically the concept where machines can teach themselves depending upon the results of their own actions. Without further delay, let’s start.

 What is Reinforcement Learning?

 Reinforcement Learning is a part of machine learning. Here, agents are self-trained on reward and punishment mechanisms. It’s about taking the best possible action or path to gain maximum rewards and minimum punishment through observations in a specific situation. It acts as a signal to positive and negative behaviors. Essentially an agent (or several) is built that can perceive and interpret the environment in which is placed, furthermore, it can take actions and interact with it.

 Basic Diagram of Reinforcement Learning – KDNuggets

 To know the meaning of reinforcement learning, let’s go through the formal definition.

 Reinforcement learning, a type of machine learning, in which agents take actions in an environment aimed at maximizing their cumulative rewards – NVIDIA

 Reinforcement learning (RL) is based on rewarding desired behaviors or punishing undesired ones. Instead of one input producing one output, the algorithm produces a variety of outputs and is trained to select the right one based on certain variables – Gartner

 It is a type of machine learning technique where a computer agent learns to perform a task through repeated trial and error interactions with a dynamic environment. This learning approach enables the agent to make a series of decisions that maximize a reward metric for the task without human intervention and without being explicitly programmed to achieve the task – Mathworks

 The above definitions are technically provided by experts in that field however for someone who is starting with reinforcement learning, but these definitions might feel a little bit difficult. As this is a reinforcement learning guide for beginners, let’s create our reinforcement learning definition in an easier way.

 Simplified Definition of Reinforcement Learning

 Through a series of Trial and Error methods, an agent keeps learning continuously in an interactive environment from its own actions and experiences. The only goal of it is to find a suitable action model which would increase the total cumulative reward of the agent. It learns via interaction and feedback.

 Well, that’s the definition of reinforcement learning. Now how we come to this definition, how a machine learns and how it can solve complex problems in the world through reinforcement learning, is something we are going to see further.

 Explanation to Reinforcement Learning

 How does reinforcement learning work? Well, let me explain with an example.

 Reinforcement Learning Example – KDNuggets

 Here what do you see?

 You can see a dog and a master. Let’s imagine you are training your dog to get the stick. Each time the dog gets a stick successfully, you offered him a feast (a bone let’s say). Eventually, the dog understands the pattern, that whenever the master throws a stick, it should get it as early as it can to gain a reward (a bone) from a master in a lesser time.

 Terminologies used in Reinforcement Learning

 Terminologies in RL – Techvidvan

 Agent – is the sole decision-maker and learner

 Environment – a physical world where an agent learns and decides the actions to be performed

 Action – a list of action which an agent can perform

 State – the current situation of the agent in the environment

 Reward – For each selected action by agent, the environment gives a reward. It’s usually a scalar value and nothing but feedback from the environment

 Policy – the agent prepares strategy(decision-making) to map situations to actions.

 Value Function – The value of state shows up the reward achieved starting from the state until the policy is executed

 Model – Every RL agent doesn’t use a model of its environment. The agent’s view maps state-action pairs probability distributions over the states

 Reinforcement Learning Workflow

 Reinforcement Learning Workflow – KDNuggets

 – Create the Environment

 – Define the reward

 – Create the agent

 – Train and validate the agent

 – Deploy the policy

 How is reinforcement learning different from supervised learning?

 In supervised learning, the model is trained with a training dataset that has a correct answer key. The decision is done on the initial input given as it has all the data that’s required to train the machine. The decisions are independent of each other so each decision is represented through a label. Example: Object Recognition

 Difference between Supervised and Reinforcement Learning – purestudy

 In reinforcement learning, there isn’t any answer and the reinforcement agent decides what to be done to perform the required task. As the training dataset isn’t available, the agent had to learn from its experience. It’s all about compiling the decisions in a sequential manner. To be said in simpler words, the output relies on the current input state and the next input relies on the output of the previous input. We give labels to the sequence of dependent decisions. Decisions are dependent. Example: Chess Game

 Characteristics of Reinforcement Learning

 – No supervision, only a real value or reward signal

 – Decision making is sequential

 – Time plays a major role in reinforcement problems

 – Feedback isn’t prompt but delayed

 – The following data it receives is determined by the agent’s actions

 Reinforcement Learning Algorithms

 There are 3 approaches to implement reinforcement learning algorithms

 Reinforcement Learning Algorithms – AISummer

 Value-Based – The main goal of this method is to maximize a value function. Here, an agent through a policy expects a long-term return of the current states.

 Policy-Based – In policy-based, you enable to come up with a strategy that helps to gain maximum rewards in the future through possible actions performed in each state. Two types of policy-based methods are deterministic and stochastic.

 Model-Based – In this method, we need to create a virtual model for the agent to help in learning to perform in each specific environment

 Types of Reinforcement Learning

 There are two types :

 Reinforcement Theory Example – Tutorialspoint

 1. Positive Reinforcement

 Positive reinforcement is defined as when an event, occurs due to specific behavior, increases the strength and frequency of the behavior. It has a positive impact on behavior.

 Advantages

 – Maximizes the performance of an action

 – Sustain change for a longer period

 Disadvantage

 – Excess reinforcement can lead to an overload of states which would minimize the results.

 2. Negative Reinforcement

 Negative Reinforcement is represented as the strengthening of a behavior. In other ways, when a negative condition is barred or avoided, it tries to stop this action in the future.

 Advantages

 – Maximized behavior

 – Provide a decent to minimum standard of performance

 Disadvantage

 – It just limits itself enough to meet up a minimum behavior

 Widely used models for reinforcement learning

 1. Markov Decision Process (MDP’s) – are mathematical frameworks for mapping solutions in RL. The set of parameters that include Set of finite states – S, Set of possible Actions in each state – A, Reward – R, Model – T, Policy – π. The outcome of deploying an action to a state doesn’t depend on previous actions or states but on current action and state.

 Markov Decision Process – Geeks4geeks

 2. Q Learning – it’s a value-based model free approach for supplying information to intimate which action an agent should perform. It revolves around the notion of updating Q values which shows the value of doing action A in state S. Value update rule is the main aspect of the Q-learning algorithm.

 QLearning – Freecodecamp

 Practical Applications of reinforcement learning

 – Robotics for Industrial Automation

 – Text summarization engines, dialogue agents (text, speech), gameplays

 – Autonomous Self Driving Cars

 – Machine Learning and Data Processing

 – Training system which would issue custom instructions and materials with respect to the requirements of students

 – AI Toolkits, Manufacturing, Automotive, Healthcare, and Bots

 – Aircraft Control and Robot Motion Control

 – Building artificial intelligence for computer games

 Conclusion

 The conclusion for this topic is nothing but helping us to discover which action could yield the highest reward for a longer time. Realistic environments can have partial observability and be non-stationary as well. It isn’t very useful to apply when you have hands-on enough data to solve the problem using supervised learning. The main challenge of this method is that parameters could affect the speed of the learning.

 Hope now you got the feel and certain level of the description on Reinforcement Learning. Thanks for your time.

 About Me

 I am Prathima Kadari, a former embedded engineer, working on leveraging my knowledge and upgrading my skills.

 Please feel free to connect with me on https://www.linkedin.com/in/prathima-kadari

 The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.

 Related

 blogathonReinforcement Learning

 Machine Learning

 Basics of Machine Learning

 Machine Learning Basics for a Newbie

 Machine Learning Lifecycle

 6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling

 Importance of Stats and EDA

 Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?

 Understanding Data

 Reading Data Files into Python Different Variable Datatypes

 Probability

 Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability

 Exploring Continuous Variable

 Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable

 Exploring Categorical Variables

 Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables

 Missing Values and Outliers

 Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN

 Central Limit theorem

 Sample and Population Central Limit Theorem Confidence Interval and Margin of Error

 Bivariate Analysis Introduction

 Bivariate Analysis Introduction

 Continuous - Continuous Variables

 Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables

 Continuous Categorical

 Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables

 Categorical Categorical

 Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables

 Multivariate Analysis

 Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL

 Different tasks in Machine Learning

 Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model

 Build Your First Predictive Model

 Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification

 Evaluation Metrics

 Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2

 Preprocessing Data

 Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building

 Linear Models

 Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression

 KNN

 Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python

 Selecting the Right Model

 Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation

 Feature Selection Techniques

 Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R

 Decision Tree

 Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees

 Feature Engineering

 Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools

 NaÃ¯ve Bayes

 Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of NaÃ¯ve Bayes

 Multiclass and Multilabel

 Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification

 Basics of Ensemble Techniques

 Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?

 Advance Ensemble Techniques

 Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python

 Hyperparameter Tuning

 Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt

 Support Vector Machine

 Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R

 Advance Dimensionality Reduction

 Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis

 Unsupervised Machine Learning Methods

 Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models

 Recommendation Engines

 Understand Basics of Recommendation Engine with Case Study

 Improving ML models

 8 Proven Ways for improving the â€œAccuracyâ€_x009d_ of a Machine Learning Model

 Working with Large Datasets

 Introduction to Dask Working with CuML

 Interpretability of Machine Learning Models

 Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values

 Automated Machine Learning

 Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML

 Model Deployment

 Pickle and Joblib Introduction to Model Deployment

 Deploying ML Models

 Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask

 Embedded Devices

 Model Deployment in Android Model Deployment in Iphone

 Related Articles

 * Meta-Reinforcement Learning in Data Science
 * Reinforcement Learning and its Scope in 2022
 * All You Need To Know About Reinforcement Learning For Digital Marketing
 * Reinforcement Learning Techniques Based on Types of Interaction
 * Getting to Grips with Reinforcement Learning via Markov Decision Process
 * ChatGPT: Unlocking the Potential of Artificial Intelligence for Human-Like Conversation

 About the Author

 Prathima Kadari

 Our Top Authors

 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * view more

 Download Analytics Vidhya App for the Latest blog/Article

 Previous Post ML Model Deployment with Webhosting frameworks
 Next Post Sentiment Analysis: Predicting Sentiment Of COVID-19 Tweets

 Leave a Reply Your email address will not be published. Required fields are marked *

 Notify me of follow-up comments by email.

 Notify me of new posts by email.

 Submit Δ

 Top Resources

 How to Read and Write With CSV Files in Python:..

 Harika Bonthu - Aug 21, 2021

 Feature Selection Techniques in Machine Learning

 Aman Gupta - Oct 10, 2020

 Naive Bayes Classifier Explained: Applications and Practice Problems of Naive..

 Sunil Ray - Sep 11, 2017

 Guide to AUC ROC Curve in Machine Learning : What..

 Aniruddha Bhandari - Jun 16, 2020

 ×

 Download App

 * Analytics Vidhya

 About Us Our Team Careers Contact us

 * Data Scientists

 Blog Hackathon Discussions Apply Jobs

 * Companies

 Post Jobs Trainings Hiring Hackathons Advertising

 * Visit us

 © Copyright 2013-2023 Analytics Vidhya.

 Privacy Policy Terms of Use Refund Policy

 Please Signup to Analytics Vidhya

 Sign in with Google
 OR
 Sign in with Email
 Skip
 We use cookies on Analytics Vidhya websites to deliver our services, analyze web traffic, and improve your experience on the site. By using Analytics Vidhya, you agree to our Privacy Policy and Terms of Use . Accept
 Privacy & Cookies Policy
 Close

 Privacy Overview

 This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
 Necessary
 Necessary
 Always Enabled
 Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
 Non-necessary
 Non-necessary
 Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
 SAVE & ACCEPT
 ×
Reinforcement Learning, seems intriguing, right? Here in this article, we will see what it is and why is it so much talked about these days. This acts as a guide to reinforcement learning for beginners. Reinforcement Learning is definitely one of the evident research areas at present which has a good boom to emerge in the coming future and its popularity is increasing day by day. Lets, get it started.
It is basically the concept where machines can teach themselves depending upon the results of their own actions. Without further delay, let’s start.
 What is Reinforcement Learning? 
Reinforcement Learning is a part of machine learning. Here, agents are self-trained on reward and punishment mechanisms. It’s about taking the best possible action or path to gain maximum rewards and minimum punishment through observations in a specific situation. It acts as a signal to positive and negative behaviors. Essentially an agent (or several) is built that can perceive and interpret the environment in which is placed, furthermore, it can take actions and interact with it.
   Basic Diagram of Reinforcement Learning – KDNuggets  
To know the meaning of reinforcement learning, let’s go through the formal definition.
Reinforcement learning, a type of machine learning, in which agents take actions in an environment aimed at maximizing their cumulative rewards – NVIDIA
Reinforcement learning (RL) is based on rewarding desired behaviors or punishing undesired ones. Instead of one input producing one output, the algorithm produces a variety of outputs and is trained to select the right one based on certain variables – Gartner
It is a type of machine learning technique where a computer agent learns to perform a task through repeated trial and error interactions with a dynamic environment. This learning approach enables the agent to make a series of decisions that maximize a reward metric for the task without human intervention and without being explicitly programmed to achieve the task – Mathworks
The above definitions are technically provided by experts in that field however for someone who is starting with reinforcement learning, but these definitions might feel a little bit difficult. As this is a reinforcement learning guide for beginners, let’s create our reinforcement learning definition in an easier way.
 Simplified Definition of Reinforcement Learning 
Through a series of Trial and Error methods, an agent keeps learning continuously in an interactive environment from its own actions and experiences. The only goal of it is to find a suitable action model which would increase the total cumulative reward of the agent. It learns via interaction and feedback.
Well, that’s the definition of reinforcement learning. Now how we come to this definition, how a machine learns and how it can solve complex problems in the world through reinforcement learning, is something we are going to see further.
 Explanation to Reinforcement Learning 
How does reinforcement learning work? Well, let me explain with an example. 
 
   Reinforcement Learning Example – KDNuggets  
Here what do you see?
You can see a dog and a master. Let’s imagine you are training your dog to get the stick. Each time the dog gets a stick successfully, you offered him a feast (a bone let’s say). Eventually, the dog understands the pattern, that whenever the master throws a stick, it should get it as early as it can to gain a reward (a bone) from a master in a lesser time.
 Terminologies used in Reinforcement Learning 
   Terminologies in RL – Techvidvan  
 Agent – is the sole decision-maker and learner
 Environment – a physical world where an agent learns and decides the actions to be performed
 Action – a list of action which an agent can perform
 State – the current situation of the agent in the environment
 Reward – For each selected action by agent, the environment gives a reward. It’s usually a scalar value and nothing but feedback from the environment
 Policy – the agent prepares strategy(decision-making) to map situations to actions.
 Value Function – The value of state shows up the reward achieved starting from the state until the policy is executed
 Model – Every RL agent doesn’t use a model of its environment. The agent’s view maps state-action pairs probability distributions over the states
  Reinforcement Learning Workflow 
   Reinforcement Learning Workflow – KDNuggets  
– Create the Environment
– Define the reward
– Create the agent  
 
– Train and validate the agent
– Deploy the policy
 How is reinforcement learning different from supervised learning? 
In supervised learning, the model is trained with a training dataset that has a correct answer key. The decision is done on the initial input given as it has all the data that’s required to train the machine. The decisions are independent of each other so each decision is represented through a label. Example: Object Recognition
   Difference between Supervised and Reinforcement Learning – purestudy  
In reinforcement learning, there isn’t any answer and the reinforcement agent decides what to be done to perform the required task. As the training dataset isn’t available, the agent had to learn from its experience. It’s all about compiling the decisions in a sequential manner. To be said in simpler words, the output relies on the current input state and the next input relies on the output of the previous input. We give labels to the sequence of dependent decisions. Decisions are dependent. Example: Chess Game
 Characteristics of Reinforcement Learning 
– No supervision, only a real value or reward signal
– Decision making is sequential
– Time plays a major role in reinforcement problems
– Feedback isn’t prompt but delayed
– The following data it receives is determined by the agent’s actions
 Reinforcement Learning Algorithms 
There are 3 approaches to implement reinforcement learning algorithms
   Reinforcement Learning Algorithms – AISummer  
 Value-Based – The main goal of this method is to maximize a value function. Here, an agent through a policy expects a long-term return of the current states.
 Policy-Based – In policy-based, you enable to come up with a strategy that helps to gain maximum rewards in the future through possible actions performed in each state. Two types of policy-based methods are deterministic and stochastic.
 Model-Based – In this method, we need to create a virtual model for the agent to help in learning to perform in each specific environment
 Types of Reinforcement Learning 
There are two types :
   Reinforcement Theory Example – Tutorialspoint  
 1. Positive Reinforcement 
Positive reinforcement is defined as when an event, occurs due to specific behavior, increases the strength and frequency of the behavior. It has a positive impact on behavior.
Advantages
– Maximizes the performance of an action
– Sustain change for a longer period
Disadvantage
– Excess reinforcement can lead to an overload of states which would minimize the results.
 2. Negative Reinforcement
 
Negative Reinforcement is represented as the strengthening of a behavior. In other ways, when a negative condition is barred or avoided, it tries to stop this action in the future.
Advantages
– Maximized behavior
– Provide a decent to minimum standard of performance
Disadvantage
– It just limits itself enough to meet up a minimum behavior
 Widely used models for reinforcement learning 
1. Markov Decision Process (MDP’s) – are mathematical frameworks for mapping solutions in RL. The set of parameters that include Set of finite states – S, Set of possible Actions in each state – A, Reward – R, Model – T, Policy – π. The outcome of deploying an action to a state doesn’t depend on previous actions or states but on current action and state.
   Markov Decision Process – Geeks4geeks  
2. Q Learning – it’s a value-based model free approach for supplying information to intimate which action an agent should perform. It revolves around the notion of updating Q values which shows the value of doing action A in state S. Value update rule is the main aspect of the Q-learning algorithm.
 
QLearning – Freecodecamp
 Practical Applications of reinforcement learning 
– Robotics for Industrial Automation
– Text summarization engines, dialogue agents (text, speech), gameplays
– Autonomous Self Driving Cars
– Machine Learning and Data Processing
– Training system which would issue custom instructions and materials with respect to the requirements of students
– AI Toolkits, Manufacturing, Automotive, Healthcare, and Bots
– Aircraft Control and Robot Motion Control
– Building artificial intelligence for computer games
 Conclusion 
The conclusion for this topic is nothing but helping us to discover which action could yield the highest reward for a longer time. Realistic environments can have partial observability and be non-stationary as well. It isn’t very useful to apply when you have hands-on enough data to solve the problem using supervised learning. The main challenge of this method is that parameters could affect the speed of the learning.
Hope now you got the feel and certain level of the description on Reinforcement Learning. Thanks for your time.
 About Me 
I am Prathima Kadari, a former embedded engineer, working on leveraging my knowledge and upgrading my skills.
Please feel free to connect with me on https://www.linkedin.com/in/prathima-kadari 
  The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.
Introduction to Reinforcement Learning for Beginners
https://www.analyticsvidhya.com/blog/2021/02/introduction-to-reinforcement-learning-for-beginners/
https://blog.floydhub.com/an-introduction-to-q-learning-reinforcement-learning/
An introduction to Q-Learning: Reinforcement Learning
2019-05-15
Sayak Paul
<div><div>
<article>
 <header>
 
 
 <p>Learn about the basic concepts of reinforcement learning and implement a simple RL algorithm called Q-Learning.</p>
 
 <figure>
 
 </figure>
 </header>
 <section>
 <blockquote>Have you ever trained a pet and rewarded it for every correct command you asked for? Do you know that this simple way of rewarding behavior can be modeled in a robot or a software program to make it do useful things? In this article, we are going to step into the world of <strong>reinforcement learning</strong>, another beautiful branch of artificial intelligence, which lets machines learn on their own in a way different from traditional machine learning. Particularly, we will be covering the simplest reinforcement learning algorithm i.e. the Q-Learning algorithm in great detail.</blockquote><blockquote>In the first half of the article, we will be discussing reinforcement learning in general with examples where reinforcement learning is not just desired but also required. We will then study the Q-Learning algorithm along with an implementation in Python using Numpy. <br /></blockquote><blockquote>Let’s start, shall we?</blockquote><h2>The brass tacks: What is reinforcement learning?</h2><figure><figcaption><strong>A rough framework of reinforcement learning</strong></figcaption></figure><p>Throughout our lives, we perform a number of actions to pursue our dreams. Some of them bring us good rewards and others do not. Along the way, we keep exploring different paths and try to figure out which action might lead to better rewards. We work hard towards our dreams utilizing the feedback we get based on our actions to improve our strategies. They help us determine how close we are to achieving our goals. Our mental states change continuously to representing this closeness. </p><p>In that description of how we pursue our goals in daily life, we framed for ourselves a representative analogy of reinforcement learning. Let me summarize the above example reformatting the main points of interest. </p><p>Our reality contains <em>environments </em>in which we perform numerous <em>actions. </em>Sometimes we get good or positive <em>rewards </em>for some of these <em>actions in order to achieve goals</em>. During the entire course of life, our mental and physical <em>states </em>evolve. We <em>strengthen </em>our actions in order to get as many rewards as possible. </p><p>The key entities of interest are - <strong>environment</strong>, <strong>action</strong>, <strong>reward </strong>and <strong>state</strong>. Let’s give ourselves a name as well - we are <strong>agents </strong>in this whole game of life. This whole paradigm of exploring our lives and learning from it through actions, rewards and states establishes the foundation of reinforcement learning. In fact, this is almost how we act in any given circumstance in our lives, isn’t it?</p><p>It turns out that the whole idea of reinforcement learning is pretty empirical in nature. Now, think for a moment, how would we train robots and machines to do the kind of useful tasks we humans do. Be it switching off the television, or moving things around, or organizing bookshelves. Fundamentally, these tasks are not about finding a function mapping inputs to outputs or finding hidden representations within input data. These are a completely different set of tasks and require a different learning paradigm for a computer to be able to perform these tasks.<strong> </strong><a href="https://blog.floydhub.com/best-deep-learning-books-updated-for-2019/#hundred-page-machine-learning"><strong>Andriy Burkov</strong> in his <strong>The Hundred Page Machine Learning Book</strong></a> describes reinforcement learning as:</p><blockquote><em>Reinforcement learning solves a particular kind of problem where decision making is sequential, and the goal is long-term, such as game playing, robotics, resource management, or logistics. </em></blockquote><p>For a robot, an environment is a place where it has been put to use. Remember this robot is itself the agent. For example, a textile factory where a robot is used to move materials from one place to another. We will come to its states, actions, rewards later. The tasks we discussed just now, have a property in common -  <em>these tasks involve an environment and expect the agents to learn from that environment</em>. This is where traditional machine learning fails and hence the need for reinforcement learning. </p><p>So, the headline <a href="https://www.forbes.com/sites/samshead/2018/08/06/elon-musk-openai-bots-dota-2/#70512799494c">AI Bots Join Forces To Beat Top Human Dota 2 Team</a> that shook the gaming world is a direct byproduct of reinforcement learning. Beyond beating humans in game-playing, there are some other marvelous use cases of reinforcement learning:</p><ul><li><a href="https://www.technologyreview.com/s/601045/this-factory-robot-learns-a-new-job-overnight/">Optimizing business processes</a></li><li><a href="https://deepmind.com/blog/deepmind-ai-reduces-google-data-centre-cooling-bill-40/">Minimizing energy costs</a></li><li><a href="https://arxiv.org/pdf/1803.09967">Maximizing revenue shares of a company</a></li></ul><p>and lots <a href="https://www.oreilly.com/ideas/practical-applications-of-reinforcement-learning-in-industry">more</a>. </p><p>In the next section, we will begin by defining the problem which is very relevant in the field of reinforcement learning. After that, we will study its agents, environment, states, actions and rewards. We will then directly proceed towards the Q-Learning algorithm.</p><h2>Recipes for reinforcement learning</h2><p>It is good to have an established overview of the problem that is to be solved using reinforcement learning, Q-Learning in this case. It helps to define the main components of a reinforcement learning solution i.e. agents, environment, actions, rewards and states.</p><h2><strong>Defining the problem statement</strong></h2><p>We are to build a few autonomous robots for a guitar building factory. These robots will help the guitar luthiers by conveying them the necessary guitar parts that they would need in order to craft a guitar. These different parts are located at <em>nine </em>different positions within the factory warehouse. Guitars parts include polished wood stick for the fretboard, polished wood for the guitar body, guitar pickups and so on. The luthiers have prioritized the location that contains body woods to be the topmost. They have provided the priorities for other locations as well which we will look in a moment. These locations within the factory warehouse look like - </p><figure><figcaption><strong>A sample environment</strong></figcaption></figure><p>As we can see there are little obstacles present (represented with smoothed lines) in between the locations. <strong>L6 </strong>is the top-priority location that contains the polished wood for preparing guitar bodies. Now, the task is to <em>enable the robots so that they can find the shortest route from any given location to another location on their own</em>. </p><p>The <strong>agents</strong>, in this case, are the robots. The <strong>environment </strong>is the guitar factory warehouse.</p><h2><strong>The states</strong></h2><p>The states are the locations. The location in which a particular robot is present in the particular instance of time will denote its state. Machines understand numbers rather than letters. So, let’s map the location codes to numbers.</p><figure><figcaption><strong>Locations mapped as states</strong></figcaption></figure><h2><strong>The actions</strong></h2><p>In our example, the actions will be <em>the direct locations</em> that a robot can go to from a particular location. Consider, a robot is at the <strong>L8</strong> location and the direct locations to which it can move are <strong>L5</strong>, <strong>L7 </strong>and <strong>L9</strong>. The below figure may come in handy in order to visualize this.</p><figure><figcaption><strong>Sample actions</strong></figcaption></figure><p><br />As you might have already guessed the set of actions here is nothing but the set of all possible states of the robot. For each location, the set of actions that a robot can take will be different. For example, the set of actions will change if the robot is in L1.</p><h2><strong>The rewards</strong></h2><p>By now, we have the following two sets:</p><ul><li>A set of states: $$S = {0, 1, 2, 3, 4, 5, 6, 7, 8}$$</li><li>A set of actions: $$A = {0, 1, 2, 3, 4, 5, 6, 7, 8}$$</li></ul><p>The rewards, now, will be given to a robot if a location (read it <em>state</em>) is <em>directly </em>reachable from a particular location. Let’s take an example: </p><p>L9 is directly reachable from L8. So, if a robot goes from L8 to L9 and vice-versa, it will be rewarded by 1. If a location is not directly reachable from a particular location, we do not give any reward (a reward of <strong>0</strong>). Yes, the reward is just a number here and nothing else. It enables the robots to make sense of their movements helping them in deciding what locations are directly reachable and what are not. With this cue, we can construct a reward table which contains all the reward values mapping between all the possible states (locations).</p><figure><figcaption><strong>Table of rewards</strong></figcaption></figure><p>In the above table, we have all the possible rewards that a robot can get by moving in between the different states. </p><p>Now comes an interesting decision. Remember from the previous sections that the luthier prioritized L6 to be of the topmost? So, how do we incorporate this fact in the above table? This is done by associating the topmost priority location with a very higher reward than the usual ones. Let’s put <strong>999 </strong>in the cell (<strong>L6</strong>, <strong>L6</strong>):</p><figure><figcaption><strong>Table of rewards with a higher reward for the topmost location</strong></figcaption></figure><p>We have now formally defined all the vital components for the solution we are aiming for the problem discussed above. We will shift gears a bit and study some of the fundamental concepts that prevail in the world of reinforcement learning. We will start with the <em>Bellman Equation</em>.</p><h2><strong>The Bellman Equation</strong></h2><p>Consider the following square of rooms which is analogous to the actual environment from our original problem but without the barriers.</p><figure><figcaption><strong>An empty environment</strong></figcaption></figure><p>Now suppose a robot needs to go to the room, marked in green from its current position (A) using the specified direction.</p><figure><figcaption><strong>Sample environment, agent and directions to proceed</strong></figcaption></figure><p>How can we enable the robot to do this programmatically? One idea would be to introduce some kind of footprint which the robot would be able to follow like below.</p><figure><figcaption><strong>An environment with value footprints</strong></figcaption></figure><p>Here, a constant value is specified in each of the rooms which will come along the robot’s way if it follows the direction specified above. In this way, if it starts at location A, it will be able to scan through this constant value and will move accordingly. But this would only work if the direction is prefixed and the robot always starts at location A. Now, consider the robot starts at the following location:</p><figure><figcaption><strong>An environment with value footprint (contd.)</strong></figcaption></figure><p>The robot now sees footprints in two different directions. It is, therefore, unable to decide which way to go in order to get to the destination (green room). It happens primarily because the robot does not have a way to remember the directions to proceed. So, our job now is to enable the robot with a <em>memory</em>. This is where the Bellman Equation comes into play:</p><p>$$V(s)=\max _{a}\left(R(s, a) + \gamma V\left(s^{\prime}\right)\right)$$</p><p>where,</p><ul><li>s = a particular state (room)</li><li>a = action (moving between the rooms)</li><li>s′ = state to which the robot goes from s</li><li>𝜸 = discount factor (we will get to it in a moment)</li><li>R(s, a) = a reward function which takes a state s and action a and outputs a reward value</li><li>V(s) = value of being in a particular state (the footprint)</li></ul><p>We consider all the possible actions and take the one that yields the maximum value. </p><p>There is one constraint however regarding the value footprint i.e. the room, marked in yellow just below the green room will always have a value of 1 to denote that it is one of the nearest room adjacent to the green room. This is also to ensure that a robot gets a reward when it goes from the yellow room to the green room. Let’s now see how to make sense of the above equation here.</p><p>We will assume a discount factor of <strong>0.9</strong>.</p><figure><figcaption><strong>An environment with value footprint (contd.)</strong></figcaption></figure><p><strong><strong>*</strong> </strong>For this room (read state) what will be V(s)? Let’s put the values into the equation straightly:</p><p>$$V(s)=\max _{a}(0 + 0.9 * 1)=0.9$$</p><p>Here, the robot will not get any reward for going to the state (room) marked in yellow, hence <strong>R(s, a) = 0</strong> here. The robot knows the values of being in the yellow room hence <strong>V(s′)</strong> is 1. Following this for the other states, we should get:</p><figure><figcaption><strong>An environment with some value footprints computed from the Bellman equation</strong></figcaption></figure><p>A couple of things to notice here:</p><ul><li>The <strong>max()</strong> function helps the robot to always choose the state that gives it the maximum value of being in that state.</li><li>The discount factor 𝜸 notifies the robot about how far it is from the destination. This typically specified by the developer of the algorithm that would be instilled in the robot. </li></ul><p>The other states can also be given their respective values in a similar way:</p><figure><figcaption><strong>An environment with all the value footprints computed from the Bellman equation</strong></figcaption></figure><p>The robot now can proceed its way through the green room utilizing these value footprints even if it is dropped at any arbitrary room in the above square. Now, if a robot lands up in the highlighted (sky blue) room, it will still find two options to choose from. But eventually, either of the paths will be good enough for the robot to take because of the way the value footprints are now laid out. </p><p>Notice how the main task of reaching a destination from a particular source got broken down to <em>similar </em>subtasks. It turns out that there is a particular programming paradigm developed especially for solving problems that have <em>repetitive subproblems</em> in them. It is referred to as <a href="http://smo.sogang.ac.kr/doc/bellman.pdf">dynamic programming</a>. It was invented by <a href="https://en.wikipedia.org/wiki/Richard_E._Bellman">Richard Bellman</a> in 1954 who also coined the equation we just studied (hence the name, Bellman Equation). <strong>Note </strong>that this is one of the key equations in the world of reinforcement learning. </p><p>If we think realistically, our surroundings do not always work in the way we expect. There is always a bit of stochasticity involved in it. This applies to a robot as well. Sometimes, it might so happen that the robot’s inner machinery got corrupted. Sometimes, the robot may come across some hindrances on its way which may not be known to it beforehand. Sometimes, even if the robot knows that it needs to take the right turn, it will not. So, how do we introduce this stochasticity in our case? <em>Markov Decision Processes</em>.</p><h2>Modeling stochasticity: Markov Decision Processes</h2><p>Consider the robot is currently in the red room and it needs to go to the green room.</p><figure><figcaption><strong>An environment with an agent</strong></figcaption></figure><p>Let’s now consider, the robot has a slight chance of dysfunctioning and might take the left or right or bottom turn instead of taking the upper turn in order to get to the green room from where it is now (red room). Now, the question is how do we enable the robot to handle this when it is out there in the above environment?</p><figure><figcaption><strong>An environment with an agent (with stochasticity)</strong></figcaption></figure><p>This is a situation where the decision making regarding which turn is to be taken is <em>partly random</em> and <em>partly under the control of the robot</em>. Partly random because we are not sure when exactly the robot might dysfunction and partly under the control of the robot because it is still making a decision of taking a turn on its own and with the help of the program embedded into it. Here is the definition of Markov Decision Processes (collected from Wikipedia): </p><blockquote><em>A Markov decision process (MDP) is a discrete time stochastic control process. It provides a mathematical framework for modeling decision making in situations where outcomes are partly random and partly under the control of a decision maker</em>.</blockquote><p>You may focus only on the highlighted part. We have the exact same situation here in our case. </p><p>We have now introduced ourselves with the concept of <em>partly random</em> and <em>partly controlled decision making</em>. We need to give this concept a mathematical shape (most likely an equation) which then can be taken further.  You might be surprised to see that we can do this with the help of the Bellman Equation with a few minor tweaks. </p><p>Here is the original Bellman Equation, again:</p><p>$$V(s)=\max _{a}\left(R(s, a) + \gamma V\left(s^{\prime}\right)\right)$$</p><p>What needs to be changed in the above equation so that we can introduce some amount of randomness here? As long as we are not sure when the robot might not take the expected turn, we are then also not sure in which room it might end up in which is nothing but the room it moves from its current room. At this point, according to the above equation, we are not sure of s′ which is the next state (room, as we were referring them). But we do know all the probable turns the robot might take! In order to incorporate each of these probabilities into the above equation, we need to associate a probability with each of the turns to quantify that <em>the robot has got x% chance of taking this turn</em>. If we do so, we get:</p><p>$$V(s)=\max _{a}\left(R(s, a) + \gamma \sum{s^{\prime}} P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$</p><p>Taking the new notations step by step:</p><ul><li><strong>P(s, a, s′) </strong>- the probability of moving from room <strong>s</strong> to room <strong>s′</strong> with action <strong>a</strong></li><li>$$\sum_{s^{\prime}} P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)$$ - <a href="https://en.wikipedia.org/wiki/Expected_value">expectation </a>of the situation that the robot incurs randomness</li></ul><p>Notice, everything else from the above two points is exactly the same. Let’s assume the following probabilities are associated with each of the turns the robot might take while being in that red room (to go to the green room).</p><figure><figcaption><strong>An environment with an agent (with probabilities)</strong></figcaption></figure><p>When we associate probabilities to each of these turns, <em>we essentially mean that there is an 80% chance that the robot will take the upper turn</em>. If we put all the required values in our equation, we get:</p><p>$$V(s)=\max _{a} (R(s, a) + \gamma((0.8V(room_{up})) +( 0.1V(room_{down})) + ...))$$</p><p><strong>Note </strong>that the value footprints will now change due to the fact that we are incorporating stochasticity here. But this time, we will not calculate those value footprints. Instead, we will let the robot to figure it out (more on this in a moment). </p><p>Up until this point, we have not considered about rewarding the robot for its action of going into a particular room. We are only rewarding the robot when it gets to the destination. Ideally, there should be a reward for every action the robot takes to help it better assess the quality of its actions. The rewards need not be always the same. But it is much better than having some amount reward for the actions than having no rewards at all. This idea is known as the <em>living penalty</em>. In reality, the rewarding system can be very complex and particularly <a href="https://www.youtube.com/watch?v=0Ey02HT_1Ho">modeling sparse rewards</a> is an active area of research in the domain reinforcement learning. If you would like to give a spin to this topic then following resources might come in handy: </p><ul><li><a href="https://pathak22.github.io/noreward-rl/">Curiosity-driven Exploration by Self-supervised Prediction</a></li><li><a href="https://ai.googleblog.com/2018/10/curiosity-and-procrastination-in.html">Curiosity and Procrastination in Reinforcement Learning</a></li><li><a href="https://ai.googleblog.com/2019/02/learning-to-generalize-from-sparse-and.html">Learning to Generalize from Sparse and Underspecified Rewards</a></li></ul><p>In the next section, we will introduce the notion of the <em>quality </em>of an action rather than looking at the value of going into a particular room (<strong>V(s)</strong>).</p><h2><strong>Transitioning to Q-Learning</strong></h2><p>By now, we have got the following equation which gives us a value of going to a particular state (form now on, we will refer to the rooms as states) taking the stochasticity of the environment into the account:</p><p>$$V(s)=\max {a}\left(R(s, a)  + \gamma \sum{s^{\prime}} P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$</p><p>We have also learned very briefly about the idea of <em>living penalty</em> which deals with associating each move of the robot with a reward. </p><p>Q-Learning poses an idea of assessing the <em>quality of an action </em>that is taken to move to a state rather than determining the possible value of the state (value footprint) being moved to. </p><p>Earlier, we had:</p><figure><figcaption><strong>An environment with an agent (with possible value footprints)</strong></figcaption></figure><p>If we incorporate the idea of assessing the quality of actions for moving to a certain state<strong> s′</strong>.</p><figure><figcaption><strong>An environment with an agent (with quality of actions)</strong></figcaption></figure><p>The robot now has four different states to choose from and along with that, there are four different actions also for the current state it is in. So how do we calculate <strong>Q(s, a) </strong>i.e. the cumulative quality of the possible actions the robot might take? Let’s break down. </p><p>From this equation $$V(s)=\max {a}\left(R(s, a) +  \gamma \sum{s^{\prime}} P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$, if we discard the <strong>max()</strong> function, we get:</p><p>$$R(s, a) + \gamma \sum_{s^{\prime}}\left(P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$</p><p>Essentially, in the equation that produces <strong>V(s)</strong>, we are considering all possible actions and all possible states (from the current state the robot is in) and then we are taking the maximum value caused by taking a certain action. The above equation produces a value footprint is for just one possible action. In fact, we can think of it as the quality of the action:</p><p>$$Q(s, a)=R(s, a) + \gamma \sum_{s^{\prime}}\left(P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$</p><p>Now that we have got an equation to quantify the quality of a particular action we are going to make a little adjustment in the above equation. We can now say that<strong> V(s)</strong> is the <em>maximum of all the possible values of <strong>Q(s, a)</strong></em>. Let’s utilize this fact and replace <em><strong>V(s′)</strong> as a function of <strong>Q()</strong>:</em></p><p>$$Q(s, a)=R(s, a) + \gamma \sum_{s^{\prime}}\left(P\left(s, a, s^{\prime}\right) \max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime}\right)\right)$$</p><p>But why would we do that? </p><p><strong>To ease our calculations</strong>. Because now, we have only one function <strong>Q()</strong> (which is also at the core of the dynamic programming paradigm) to calculate and <em>R(s, a) is a quantified metric</em> which produces rewards of moving to a certain state. The qualities of the actions are called <strong>Q-values</strong>. And from now on, we will refer the value footprints as the Q-values. </p><p>We now have the last piece of the puzzle remaining i.e. <em>temporal difference</em> before we jump to the implementation part and we are going to study that in the next section.</p><h2><strong>The last piece of the puzzle: Temporal difference</strong></h2><p>Recollect the statement from a previous section:</p><blockquote><em>But this time, we will not calculate those value footprints. Instead, we will let the robot to figure it out (more on this in a moment).</em><br /></blockquote><p>Temporal Difference is the component that will help the robot to calculate the Q-values with respect to the changes in the environment over time. Consider our robot is currently in the marked state and it wants to move to the upper state. Note that the robot already knows the Q-value of making the action i.e. moving to the upper state.  </p><figure><figcaption><strong>An environment with an agent</strong></figcaption></figure><p>We know that the environment is stochastic in nature and the reward that the robot will get after moving to the upper state might be different from an earlier observation. So how do we capture this change (read difference)? We recalculate the new <strong>Q(s, a) </strong>with the same formula and subtract the previously known <strong>Q(s, a) </strong>from it.</p><figure></figure><p>The equation that we just derived gives the temporal difference in the Q-values which further helps to capture the random changes that the environment may impose. The new <strong>Q(s, a)</strong> is updated as the following:</p><p>$$Q_{t}(s, a)=Q_{t-1}(s, a)+\alpha T D_{t}(a, s)$$</p><p>where,</p><ul><li>ɑ is the <em>learning rate</em> which controls how quickly the robot adopts to the random changes imposed by the environment</li><li><strong>$$Q_t (s, a)$$</strong> is the current Q-value</li><li><strong>$$Q_{t-1} (s, a)$$ </strong>is the previously recorded Q-value </li></ul><p>If we replace $$TD_t(s, a)$$ with its full-form equation, we should get:</p><p>$$Q_{t}(s, a)=Q_{t-1}(s, a)+\alpha\left(R(s, a)+\gamma \max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime}\right)-Q_{t-1}(s, a)\right)$$</p><p>We now have all the little pieces of Q-Learning together to move forward to its implementation part. Feel free to review the problem statement once which we discussed in the very beginning.</p><h2><strong>Implementing Q-Learning in Python with Numpy</strong></h2><p>If you do not have a local setup, you can run <a href="https://github.com/sayakpaul/FloydHub-Q-Learning-Blog">this notebook</a> directly on FloydHub by just clicking on the below button -</p>
<a href="https://floydhub.com/run?template=https://github.com/sayakpaul/FloydHub-Q-Learning-Blog">
 
</a>
<p>To implement the algorithm, we need to understand the warehouse locations and how that can be mapped to different <em>states</em>. Let’s start by recollecting the sample environment shown earlier:</p><figure><figcaption>A sample environment</figcaption></figure><p>Let’s map each of the above locations in the warehouse to numbers (<em>states</em>). It will ease our calculations. </p><pre><code># Define the states
location_to_state = {
 'L1' : 0,
 'L2' : 1,
 'L3' : 2,
 'L4' : 3,
 'L5' : 4,
 'L6' : 5,
 'L7' : 6,
 'L8' : 7,
 'L9' : 8
}
</code></pre>
<p>We will only be using Numpy as our dependency. So let’s import that aliased as <strong>np:</strong></p><p><code>import numpy as np</code></p>
<p>The next step is to define the <em>actions </em>which as mentioned above represents the transition to the next state: </p><pre><code># Define the actions
actions = [0,1,2,3,4,5,6,7,8]
</code></pre>
<p>Now the reward table:</p><pre><code># Define the rewards
rewards = np.array([[0,1,0,0,0,0,0,0,0],
 [1,0,1,0,1,0,0,0,0],
 [0,1,0,0,0,1,0,0,0],
 [0,0,0,0,0,0,1,0,0],
 [0,1,0,0,0,0,0,1,0],
 [0,0,1,0,0,0,0,0,0],
 [0,0,0,1,0,0,0,1,0],
 [0,0,0,0,1,0,1,0,1],
 [0,0,0,0,0,0,0,1,0]])
</code></pre>
<p>If you understood it correctly, there isn't any real barrier limitation as depicted in the image. For example, the transition L4 to L1 is allowed but the reward will be zero to discourage that path.</p><p>In the above code snippet, we took each of the states and put ones in the respective state that are directly reachable from a certain state. Refer to the reward table once again. The above array construction will be easy to understand then. Note that, we did not consider the top-priority location (<strong>L6</strong>) yet.</p><figure><figcaption><strong>Table of rewards (rewards matrix)</strong></figcaption></figure><p>We would also need an inverse mapping from the states back to original location indicators. It will be clearer when we reach to the utter depths of the algorithm. The following line of code will do this for us.</p><pre><code># Maps indices to locations
state_to_location = dict((state,location) for location,state in location_to_state.items())
</code></pre>
<p>Before we write a helper function which will yield the optimal route for going from one location to another, let’s specify the two parameters of the Q-Learning algorithm - ɑ (learning rate) and 𝜸 (discount factor).</p><pre><code># Initialize parameters
gamma = 0.75 # Discount factor 
alpha = 0.9 # Learning rate 
</code></pre>
<p>We will now define a function <strong>get_optimal_route()</strong> which will:</p><ul><li>Take two arguments:</li><li>starting location in the warehouse and</li><li>end location in the warehouse respectively</li><li>And will return the optimal route for reaching the end location from the starting location in the form of an ordered list containing the letters</li></ul><p>We will start defining the function by initializing the Q-values to be all zeros.</p><pre><code># Initializing Q-Values
Q = np.array(np.zeros([9,9]))
</code></pre>
<p>For convenience, we will copy the rewards matrix <strong>rewards </strong>to a separate variable and will operate on that.</p><pre><code># Copy the rewards matrix to new Matrix
rewards_copy = np.copy(rewards)
</code></pre>
<p><code>[9,9]</code> as we have a total of 9 locations. The function will take a starting location and an ending location. So, let’s set the priority of the ending location to a larger integer like 999 and get the ending state from the location letter (such as <strong>L1</strong>, <strong>L2 </strong>and so on).</p><pre><code># Get the ending state corresponding to the ending location as given
ending_state = location_to_state[end_location]
# With the above information automatically set the priority of the
# given ending state to the highest one
rewards_copy[ending_state,ending_state] = 999
</code></pre>
<p>Learning is a continuous process, hence we will let the robot to explore the environment for a while and we will do it by simply looping it through 1000 times. We will then pick a state randomly from the set of states we defined above and we will call it <strong>current_state</strong>.</p><pre><code>for i in range(1000):
 # Pick up a state randomly
 current_state = np.random.randint(0,9)
</code></pre>
<p>We then, being inside the loop, iterate through the rewards matrix to get the states that are directly reachable from the randomly chosen current state and we will assign those state in a list named <strong>playable_actions</strong>.</p><p><strong>Note</strong> that so far we have not bothered about the starting location yet.</p><pre><code>playable_actions = []
# Iterate through the new rewards matrix and get the actions &gt; 0
for j in range(9):
 if rewards_copy[current_state,j] &gt; 0:
 playable_actions.append(j)
</code></pre>
<p>We will now choose a state randomly from the <strong>playable_actions</strong>.</p><pre><code># Pick an action randomly from the list of playable actions leading us to the next 
# state
 next_state = np.random.choice(playable_actions)
</code></pre>
<p>We can now compute the temporal difference and update the Q-values accordingly. Here is the formula of temporal difference for your convenience:</p><figure></figure><p>Here is the way to update the Q-values using the temporal difference:</p><p>$$Q_{t}(s, a)=Q_{t-1}(s, a)+\alpha T D_{t}(a, s)$$</p><pre><code># Compute the temporal difference
# The action here exactly refers to going to the next state
TD = rewards_copy[current_state,next_state] + gamma * Q[next_state, np.argmax(Q[next_state,])] - Q[current_state,next_state]
# Update the Q-Value using the Bellman equation
Q[current_state,next_state] += alpha * TD
</code></pre>
<p>The good news is we are done implementing the most critical part of the process and up until now the definition of <strong>get_optimal_route() </strong>should look like:</p><pre><code>def get_optimal_route(start_location,end_location):
 # Copy the rewards matrix to new Matrix
 rewards_new = np.copy(rewards)
 
 # Get the ending state corresponding to the ending location as given
 ending_state = location_to_state[end_location]
 
 # With the above information automatically set the priority of the given ending 
 # state to the highest one
 rewards_new[ending_state,ending_state] = 999
 # -----------Q-Learning algorithm-----------
 
 # Initializing Q-Values
 Q = np.array(np.zeros([9,9]))
 # Q-Learning process
 for i in range(1000):
 # Pick up a state randomly
 current_state = np.random.randint(0,9) # Python excludes the upper bound
 # For traversing through the neighbor locations in the maze
 playable_actions = []
 # Iterate through the new rewards matrix and get the actions &gt; 0
 for j in range(9):
 if rewards_new[current_state,j] &gt; 0:
 playable_actions.append(j)
 # Pick an action randomly from the list of playable actions 
 # leading us to the next state
 next_state = np.random.choice(playable_actions)
 # Compute the temporal difference
 # The action here exactly refers to going to the next state
 TD = rewards_new[current_state,next_state] + gamma * Q[next_state, np.argmax(Q[next_state,])] - Q[current_state,next_state]
 # Update the Q-Value using the Bellman equation
 Q[current_state,next_state] += alpha * TD
</code></pre>
<p>We will now start the other half of finding the optimal route. We will first initialize the optimal route with the starting location.</p><pre><code># Initialize the optimal route with the starting location
 route = [start_location]
</code></pre>
<p>We currently do not know about the next move of the robot. Thus we will set the next location to also be the starting location.</p><p><code>next_location = start_location</code></p>
<p>Since we do not know the exact number of iterations the robot will take in order to find out the optimal route, we will simply loop the next set of processes until the next location is not equal to the ending location. This is where we will terminate the loop.</p><pre><code># We don't know about the exact number of iterations needed to reach to the final 
# location hence while loop will be a good choice for iteratiing
 while(next_location != end_location):
 # Fetch the starting state
 starting_state = location_to_state[start_location]
 # Fetch the highest Q-value pertaining to starting state
 next_state = np.argmax(Q[starting_state,])
 # We got the index of the next state. But we need the corresponding letter. 
 next_location = state_to_location[next_state]
 route.append(next_location)
 # Update the starting location for the next iteration
 start_location = next_location
 
 return route
</code></pre>
<p>Finally, we return the route.</p><p><code>return route</code></p>
<p>Here is the whole<strong> get_optimal_route() </strong>function:</p><pre><code>def get_optimal_route(start_location,end_location):
 # Copy the rewards matrix to new Matrix
 rewards_new = np.copy(rewards)
 
 # Get the ending state corresponding to the ending location as given
 ending_state = location_to_state[end_location]
 
 # With the above information automatically set the priority of 
 # the given ending state to the highest one
 rewards_new[ending_state,ending_state] = 999
 # -----------Q-Learning algorithm-----------
 
 # Initializing Q-Values
 Q = np.array(np.zeros([9,9]))
 # Q-Learning process
 for i in range(1000):
 # Pick up a state randomly
 current_state = np.random.randint(0,9) # Python excludes the upper bound
 
 # For traversing through the neighbor locations in the maze
 playable_actions = []
 
 # Iterate through the new rewards matrix and get the actions &gt; 0
 for j in range(9):
 if rewards_new[current_state,j] &gt; 0:
 playable_actions.append(j)
 
 # Pick an action randomly from the list of playable actions 
 # leading us to the next state
 next_state = np.random.choice(playable_actions)
 
 # Compute the temporal difference
 # The action here exactly refers to going to the next state
 TD = rewards_new[current_state,next_state] + gamma * Q[next_state, np.argmax(Q[next_state,])] - Q[current_state,next_state]
 
 # Update the Q-Value using the Bellman equation
 Q[current_state,next_state] += alpha * TD
 # Initialize the optimal route with the starting location
 route = [start_location]
 # We do not know about the next location yet, so initialize with the value of 
 # starting location
 next_location = start_location
 
 # We don't know about the exact number of iterations
 # needed to reach to the final location hence while loop will be a good choice 
 # for iteratiing
 
 while(next_location != end_location):
 # Fetch the starting state
 starting_state = location_to_state[start_location]
 
 # Fetch the highest Q-value pertaining to starting state
 next_state = np.argmax(Q[starting_state,])
 
 # We got the index of the next state. But we need the corresponding letter. 
 next_location = state_to_location[next_state]
 route.append(next_location)
 
 # Update the starting location for the next iteration
 start_location = next_location
 
 return route
</code></pre>
<p>If we call <strong>print(get_optimal_route('L9', 'L1'))</strong>, we should get: </p><p><code>['L9', 'L8', 'L5', 'L2', 'L1']</code></p>
<p>Note how the program considers the barriers that are present in the environment. For fun, you can change the ɑ and 𝜸 parameters to see how the learning process changes. With this, let me move to the conclusion section where we will be discussing how you can take your reinforcement learning journey further. </p><p>Let’s refactor the code a little bit to conform to <strong>OOP </strong>paradigm. We will define a class named <strong>QAgent() </strong>containing the following two methods apart from <em>init</em>:</p><ul><li><strong>training(self, start_location, end_location, iterations)</strong> which will help the robot to obtain the Q-values from the environment</li><li><strong>get_optimal_route(self, start_location, end_location, next_location, route, Q)</strong> which will get the robot an optimal route from a point to another</li></ul><p>Let’s first define the <strong>__init__()</strong> method which would initialize the class constructor:</p><pre><code>def __init__(self, alpha, gamma, location_to_state, actions, rewards, state_to_location, Q):
 
 self.gamma = gamma 
 self.alpha = alpha 
 self.location_to_state = location_to_state
 self.actions = actions
 self.rewards = rewards
 self.state_to_location = state_to_location
 self.Q = Q
</code></pre>
<p>Now comes the <strong>training() </strong>method:</p><pre><code>def training(self, start_location, end_location, iterations):
 rewards_new = np.copy(self.rewards)
 ending_state = self.location_to_state[end_location]
 rewards_new[ending_state, ending_state] = 999
 for i in range(iterations):
 current_state = np.random.randint(0,9) 
 playable_actions = []
 for j in range(9):
 if rewards_new[current_state,j] &gt; 0:
 playable_actions.append(j)
 next_state = np.random.choice(playable_actions)
 TD = rewards_new[current_state,next_state] + 
 self.gamma * self.Q[next_state, np.argmax(self.Q[next_state,])] - self.Q[current_state,next_state]
 self.Q[current_state,next_state] += self.alpha * TD
 route = [start_location]
 next_location = start_location
 # Get the route 
 self.get_optimal_route(start_location, end_location, next_location, route, self.Q)
</code></pre>
<p>Finally the <strong>get_optimal_route()</strong> method.</p><pre><code>def get_optimal_route(self, start_location, end_location, next_location, route, Q):
 while(next_location != end_location):
 starting_state = self.location_to_state[start_location]
 next_state = np.argmax(Q[starting_state,])
 next_location = self.state_to_location[next_state]
 route.append(next_location)
 start_location = next_location
 print(route)
</code></pre>
<p>The entire class definition should look like:</p><pre><code>class QAgent():
 
 # Initialize alpha, gamma, states, actions, rewards, and Q-values
 def __init__(self, alpha, gamma, location_to_state, actions, rewards, state_to_location, Q):
 
 self.gamma = gamma 
 self.alpha = alpha 
 
 self.location_to_state = location_to_state
 self.actions = actions
 self.rewards = rewards
 self.state_to_location = state_to_location
 
 self.Q = Q
 
 # Training the robot in the environment
 def training(self, start_location, end_location, iterations):
 
 rewards_new = np.copy(self.rewards)
 
 ending_state = self.location_to_state[end_location]
 rewards_new[ending_state, ending_state] = 999
 
 for i in range(iterations):
 current_state = np.random.randint(0,9) 
 playable_actions = []
 for j in range(9):
 if rewards_new[current_state,j] &gt; 0:
 playable_actions.append(j)
 
 next_state = np.random.choice(playable_actions)
 TD = rewards_new[current_state,next_state] + \
 self.gamma * self.Q[next_state, np.argmax(self.Q[next_state,])] - self.Q[current_state,next_state]
 
 self.Q[current_state,next_state] += self.alpha * TD
 route = [start_location]
 next_location = start_location
 
 # Get the route 
 self.get_optimal_route(start_location, end_location, next_location, route, self.Q)
 
 # Get the optimal route
 def get_optimal_route(self, start_location, end_location, next_location, route, Q):
 
 while(next_location != end_location):
 starting_state = self.location_to_state[start_location]
 next_state = np.argmax(Q[starting_state,])
 next_location = self.state_to_location[next_state]
 route.append(next_location)
 start_location = next_location
 
 print(route)
</code></pre>
<p>Once the class is compiled, you should be able to create a class object and call the <strong>training() </strong>method like so:</p><pre><code>qagent = QAgent(alpha, gamma, location_to_state, actions, rewards, state_to_location, Q)
qagent.training('L9', 'L1', 1000)
</code></pre>
<p>Notice that every is exactly similar to previous chunk of code but the refactored version indeed looks more elegant and modular. And yes, the output will also be same.</p><p><code>['L9', 'L8', 'L5', 'L2', 'L1']</code></p>
<h2><strong>Conclusion and further steps</strong></h2><p>We finally have come to the very end of the article. We covered a lot of preliminary grounds of reinforcement learning that will be useful if you are planning to further strengthen your knowledge of reinforcement learning. We also implemented the simplest reinforcement learning just by using <strong>Numpy</strong>. These base scratch implementations are not only for just fun but also they help tremendously to know the nuts and bolts of an algorithm. </p><p>Reinforcement learning has given solutions to many problems from a wide variety of different domains. One that I particularly like is <a href="https://arxiv.org/abs/1707.07012">Google’s <strong>NasNet</strong></a> which uses <em>deep reinforcement learning</em> for finding an optimal neural network architecture for a given dataset.  </p><p>Let’s now review some of the best resources for breaking into reinforcement learning in a serious manner: </p><ul><li><a href="https://blog.floydhub.com/best-deep-learning-books-updated-for-2019/#reinforcement-learning-intro">Reinforcement Learning, Second Edition: An Introduction</a> by <strong>Richard S. Sutton</strong> and <strong>Andrew G. Barto </strong>which is considered to be the textbook of reinforcement learning<br /></li><li><a href="https://www.coursera.org/learn/practical-rl?specialization=aml">Practical Reinforcement Learning</a> a course designed by the National Research University Higher School of Economics offered by Coursera<br /></li><li><a href="https://www.udacity.com/course/reinforcement-learning--ud600">Reinforcement Learning</a> a course designed by the Georgia  University and offered by Udacity<br /></li><li>If you are interested in the conjunction of meta-learning and reinforcement learning then you may follow <a href="https://blog.floydhub.com/meta-rl/">this article</a><br /></li><li>How about combining deep learning + reinforcement learning? Check out <a href="https://sites.google.com/view/deep-rl-bootcamp/lectures">Deep RL Bootcamp</a>.<br /></li><li><a href="https://blog.floydhub.com/best-deep-learning-books-updated-for-2019/#deep-reinforcement-learning-hands-on">Deep Reinforcement Learning Hands-On</a> a book by Maxim Lapan which covers many cutting edge RL concepts like deep Q-networks, value iteration, policy gradients and so on.<br /></li><li><a href="https://blog.floydhub.com/best-deep-learning-courses-updated-for-2019/#intro-to-rl">MIT Deep Learning</a> a course taught by Lex Fridman which teaches you how different deep learning applications are used in autonomous vehicle systems and more<br /></li><li><a href="https://blog.floydhub.com/best-deep-learning-courses-updated-for-2019/#adv-dl-rl">Introduction to Reinforcement Learning</a> a course taught by one of the main leaders in the game of reinforcement learning - David Silver</li><li><a href="https://blog.floydhub.com/best-deep-learning-courses-updated-for-2019/#spinning-up">Spinning Up in Deep RL</a> a course offered from the house of OpenAI which serves  as your guide to connecting the dots between theory and practice in deep reinforcement learning<br /></li><li><a href="https://blog.floydhub.com/robotic-arm-control-deep-reinforcement-learning/">Controlling a 2D Robotic Arm with Deep Reinforcement Learning</a> an article which shows how to build your own robotic arm best friend by diving into deep reinforcement learning<br /></li><li><a href="https://blog.floydhub.com/spinning-up-with-deep-reinforcement-learning/">Spinning Up a Pong AI With Deep Reinforcement Learning</a> an article which shows you to code a vanilla policy gradient model that plays the beloved early 1970s classic video game Pong in a step-by-step manner</li></ul><p>The list is kind of handpicked for those who really want to step up their game in reinforcement learning. So if you are one among them, don’t forget to check out the resources. </p><p>I hope you enjoyed the article and you will take it forward to make applications that can adapt with respect to the environment they are employed to.</p><hr /><p>Thanks to <a href="https://www.linkedin.com/in/alessio-gozzoli-530aa2109/">Alessio</a> and <a href="https://www.linkedin.com/in/sridharanbharath/">Bharath</a> of FloydHub for sharing their valuable feedback on the article. Big thanks to the entire FloydHub team for letting me run the accompanying notebook on their platform. If you haven’t checked FloydHub yet, give FloydHub a spin for your Machine Learning and Deep Learning projects. There is no going back once you’ve learned how easy they make it.</p><hr /><h4><strong>FloydHub Call for AI writers</strong></h4><p>Want to write amazing articles like Sayak and play your role in the long road to Artificial General Intelligence? <a href="https://blog.floydhub.com/write-for-floydhub/?utm_source=floydhub&amp;utm_medium=banner&amp;utm_campaign=call_for_writers_2019">We are looking for passionate writers</a>, to build the world's best blog for practical applications of groundbreaking A.I. techniques. FloydHub has a large reach within the AI community and with your help, we can inspire the next wave of AI. <a href="https://goo.gl/forms/PbOw0VmUnOfO1Lxp1">Apply now</a> and join the crew!</p><hr /><p><strong>About Sayak Paul</strong><br />Sayak loves everything deep learning. He goes by the motto of understanding complex things and helping people understand them as easily as possible. Sayak is an extensive blogger and all of his blogs can be found <a href="https://www.sayak.dev/authoring">here</a>. He is also working with his friends on the application of deep learning in Phonocardiogram classification. Sayak is also a <a href="https://blog.floydhub.com/write-for-floydhub/?utm_source=floydhub&amp;utm_medium=banner&amp;utm_campaign=call_for_writers_2019">FloydHub AI Writer</a>. He is always open to discussing novel ideas and taking them forward to implementations. You can connect with Sayak on <a href="https://www.linkedin.com/in/sayak-paul/">LinkedIn</a> and <a href="https://twitter.com/RisingSayak">Twitter</a>.</p>
 </section>
</article>
</div>||||I|||| Sign in Subscribe
 Reinforcement Learning

 An introduction to Q-Learning: Reinforcement Learning

 Learn about the basic concepts of reinforcement learning and implement a simple RL algorithm called Q-Learning.

 * 

 Sayak Paul

 May 15, 2019 • 27 min read

 Have you ever trained a pet and rewarded it for every correct command you asked for? Do you know that this simple way of rewarding behavior can be modeled in a robot or a software program to make it do useful things? In this article, we are going to step into the world of reinforcement learning, another beautiful branch of artificial intelligence, which lets machines learn on their own in a way different from traditional machine learning. Particularly, we will be covering the simplest reinforcement learning algorithm i.e. the Q-Learning algorithm in great detail.
 In the first half of the article, we will be discussing reinforcement learning in general with examples where reinforcement learning is not just desired but also required. We will then study the Q-Learning algorithm along with an implementation in Python using Numpy.
 Let’s start, shall we?

 Ready to build, train, and deploy AI?

 Get started with FloydHub's collaborative AI platform for free

 Try FloydHub for free

 The brass tacks: What is reinforcement learning?

 A rough framework of reinforcement learning

 Throughout our lives, we perform a number of actions to pursue our dreams. Some of them bring us good rewards and others do not. Along the way, we keep exploring different paths and try to figure out which action might lead to better rewards. We work hard towards our dreams utilizing the feedback we get based on our actions to improve our strategies. They help us determine how close we are to achieving our goals. Our mental states change continuously to representing this closeness.

 In that description of how we pursue our goals in daily life, we framed for ourselves a representative analogy of reinforcement learning. Let me summarize the above example reformatting the main points of interest.

 Our reality contains environments in which we perform numerous actions. Sometimes we get good or positive rewards for some of these actions in order to achieve goals. During the entire course of life, our mental and physical states evolve. We strengthen our actions in order to get as many rewards as possible.

 The key entities of interest are - environment, action, reward and state. Let’s give ourselves a name as well - we are agents in this whole game of life. This whole paradigm of exploring our lives and learning from it through actions, rewards and states establishes the foundation of reinforcement learning. In fact, this is almost how we act in any given circumstance in our lives, isn’t it?

 It turns out that the whole idea of reinforcement learning is pretty empirical in nature. Now, think for a moment, how would we train robots and machines to do the kind of useful tasks we humans do. Be it switching off the television, or moving things around, or organizing bookshelves. Fundamentally, these tasks are not about finding a function mapping inputs to outputs or finding hidden representations within input data. These are a completely different set of tasks and require a different learning paradigm for a computer to be able to perform these tasks. Andriy Burkov in his The Hundred Page Machine Learning Book describes reinforcement learning as:

 Reinforcement learning solves a particular kind of problem where decision making is sequential, and the goal is long-term, such as game playing, robotics, resource management, or logistics.

 For a robot, an environment is a place where it has been put to use. Remember this robot is itself the agent. For example, a textile factory where a robot is used to move materials from one place to another. We will come to its states, actions, rewards later. The tasks we discussed just now, have a property in common - these tasks involve an environment and expect the agents to learn from that environment. This is where traditional machine learning fails and hence the need for reinforcement learning.

 So, the headline AI Bots Join Forces To Beat Top Human Dota 2 Team that shook the gaming world is a direct byproduct of reinforcement learning. Beyond beating humans in game-playing, there are some other marvelous use cases of reinforcement learning:

 * Optimizing business processes
 * Minimizing energy costs
 * Maximizing revenue shares of a company

 and lots more.

 In the next section, we will begin by defining the problem which is very relevant in the field of reinforcement learning. After that, we will study its agents, environment, states, actions and rewards. We will then directly proceed towards the Q-Learning algorithm.

 Recipes for reinforcement learning

 It is good to have an established overview of the problem that is to be solved using reinforcement learning, Q-Learning in this case. It helps to define the main components of a reinforcement learning solution i.e. agents, environment, actions, rewards and states.

 Defining the problem statement

 We are to build a few autonomous robots for a guitar building factory. These robots will help the guitar luthiers by conveying them the necessary guitar parts that they would need in order to craft a guitar. These different parts are located at nine different positions within the factory warehouse. Guitars parts include polished wood stick for the fretboard, polished wood for the guitar body, guitar pickups and so on. The luthiers have prioritized the location that contains body woods to be the topmost. They have provided the priorities for other locations as well which we will look in a moment. These locations within the factory warehouse look like -

 A sample environment

 As we can see there are little obstacles present (represented with smoothed lines) in between the locations. L6 is the top-priority location that contains the polished wood for preparing guitar bodies. Now, the task is to enable the robots so that they can find the shortest route from any given location to another location on their own.

 The agents, in this case, are the robots. The environment is the guitar factory warehouse.

 The states

 The states are the locations. The location in which a particular robot is present in the particular instance of time will denote its state. Machines understand numbers rather than letters. So, let’s map the location codes to numbers.

 Locations mapped as states

 The actions

 In our example, the actions will be the direct locations that a robot can go to from a particular location. Consider, a robot is at the L8 location and the direct locations to which it can move are L5, L7 and L9. The below figure may come in handy in order to visualize this.

 Sample actions


 As you might have already guessed the set of actions here is nothing but the set of all possible states of the robot. For each location, the set of actions that a robot can take will be different. For example, the set of actions will change if the robot is in L1.

 The rewards

 By now, we have the following two sets:

 * A set of states: $$S = {0, 1, 2, 3, 4, 5, 6, 7, 8}$$
 * A set of actions: $$A = {0, 1, 2, 3, 4, 5, 6, 7, 8}$$

 The rewards, now, will be given to a robot if a location (read it state) is directly reachable from a particular location. Let’s take an example:

 L9 is directly reachable from L8. So, if a robot goes from L8 to L9 and vice-versa, it will be rewarded by 1. If a location is not directly reachable from a particular location, we do not give any reward (a reward of 0). Yes, the reward is just a number here and nothing else. It enables the robots to make sense of their movements helping them in deciding what locations are directly reachable and what are not. With this cue, we can construct a reward table which contains all the reward values mapping between all the possible states (locations).

 Table of rewards

 In the above table, we have all the possible rewards that a robot can get by moving in between the different states.

 Now comes an interesting decision. Remember from the previous sections that the luthier prioritized L6 to be of the topmost? So, how do we incorporate this fact in the above table? This is done by associating the topmost priority location with a very higher reward than the usual ones. Let’s put 999 in the cell (L6, L6):

 Table of rewards with a higher reward for the topmost location

 We have now formally defined all the vital components for the solution we are aiming for the problem discussed above. We will shift gears a bit and study some of the fundamental concepts that prevail in the world of reinforcement learning. We will start with the Bellman Equation.

 The Bellman Equation

 Consider the following square of rooms which is analogous to the actual environment from our original problem but without the barriers.

 An empty environment

 Now suppose a robot needs to go to the room, marked in green from its current position (A) using the specified direction.

 Sample environment, agent and directions to proceed

 How can we enable the robot to do this programmatically? One idea would be to introduce some kind of footprint which the robot would be able to follow like below.

 An environment with value footprints

 Here, a constant value is specified in each of the rooms which will come along the robot’s way if it follows the direction specified above. In this way, if it starts at location A, it will be able to scan through this constant value and will move accordingly. But this would only work if the direction is prefixed and the robot always starts at location A. Now, consider the robot starts at the following location:

 An environment with value footprint (contd.)

 The robot now sees footprints in two different directions. It is, therefore, unable to decide which way to go in order to get to the destination (green room). It happens primarily because the robot does not have a way to remember the directions to proceed. So, our job now is to enable the robot with a memory. This is where the Bellman Equation comes into play:

 $$V(s)=\max _{a}\left(R(s, a) + \gamma V\left(s^{\prime}\right)\right)$$

 where,

 * s = a particular state (room)
 * a = action (moving between the rooms)
 * s′ = state to which the robot goes from s
 * 𝜸 = discount factor (we will get to it in a moment)
 * R(s, a) = a reward function which takes a state s and action a and outputs a reward value
 * V(s) = value of being in a particular state (the footprint)

 We consider all the possible actions and take the one that yields the maximum value.

 There is one constraint however regarding the value footprint i.e. the room, marked in yellow just below the green room will always have a value of 1 to denote that it is one of the nearest room adjacent to the green room. This is also to ensure that a robot gets a reward when it goes from the yellow room to the green room. Let’s now see how to make sense of the above equation here.

 We will assume a discount factor of 0.9.

 An environment with value footprint (contd.)

 * For this room (read state) what will be V(s)? Let’s put the values into the equation straightly:

 $$V(s)=\max _{a}(0 + 0.9 * 1)=0.9$$

 Here, the robot will not get any reward for going to the state (room) marked in yellow, hence R(s, a) = 0 here. The robot knows the values of being in the yellow room hence V(s′) is 1. Following this for the other states, we should get:

 An environment with some value footprints computed from the Bellman equation

 A couple of things to notice here:

 * The max() function helps the robot to always choose the state that gives it the maximum value of being in that state.
 * The discount factor 𝜸 notifies the robot about how far it is from the destination. This typically specified by the developer of the algorithm that would be instilled in the robot.

 The other states can also be given their respective values in a similar way:

 An environment with all the value footprints computed from the Bellman equation

 The robot now can proceed its way through the green room utilizing these value footprints even if it is dropped at any arbitrary room in the above square. Now, if a robot lands up in the highlighted (sky blue) room, it will still find two options to choose from. But eventually, either of the paths will be good enough for the robot to take because of the way the value footprints are now laid out.

 Notice how the main task of reaching a destination from a particular source got broken down to similar subtasks. It turns out that there is a particular programming paradigm developed especially for solving problems that have repetitive subproblems in them. It is referred to as dynamic programming. It was invented by Richard Bellman in 1954 who also coined the equation we just studied (hence the name, Bellman Equation). Note that this is one of the key equations in the world of reinforcement learning.

 If we think realistically, our surroundings do not always work in the way we expect. There is always a bit of stochasticity involved in it. This applies to a robot as well. Sometimes, it might so happen that the robot’s inner machinery got corrupted. Sometimes, the robot may come across some hindrances on its way which may not be known to it beforehand. Sometimes, even if the robot knows that it needs to take the right turn, it will not. So, how do we introduce this stochasticity in our case? Markov Decision Processes.

 Modeling stochasticity: Markov Decision Processes

 Consider the robot is currently in the red room and it needs to go to the green room.

 An environment with an agent

 Let’s now consider, the robot has a slight chance of dysfunctioning and might take the left or right or bottom turn instead of taking the upper turn in order to get to the green room from where it is now (red room). Now, the question is how do we enable the robot to handle this when it is out there in the above environment?

 An environment with an agent (with stochasticity)

 This is a situation where the decision making regarding which turn is to be taken is partly random and partly under the control of the robot. Partly random because we are not sure when exactly the robot might dysfunction and partly under the control of the robot because it is still making a decision of taking a turn on its own and with the help of the program embedded into it. Here is the definition of Markov Decision Processes (collected from Wikipedia):

 A Markov decision process (MDP) is a discrete time stochastic control process. It provides a mathematical framework for modeling decision making in situations where outcomes are partly random and partly under the control of a decision maker.

 You may focus only on the highlighted part. We have the exact same situation here in our case.

 We have now introduced ourselves with the concept of partly random and partly controlled decision making. We need to give this concept a mathematical shape (most likely an equation) which then can be taken further. You might be surprised to see that we can do this with the help of the Bellman Equation with a few minor tweaks.

 Here is the original Bellman Equation, again:

 $$V(s)=\max _{a}\left(R(s, a) + \gamma V\left(s^{\prime}\right)\right)$$

 What needs to be changed in the above equation so that we can introduce some amount of randomness here? As long as we are not sure when the robot might not take the expected turn, we are then also not sure in which room it might end up in which is nothing but the room it moves from its current room. At this point, according to the above equation, we are not sure of s′ which is the next state (room, as we were referring them). But we do know all the probable turns the robot might take! In order to incorporate each of these probabilities into the above equation, we need to associate a probability with each of the turns to quantify that the robot has got x% chance of taking this turn. If we do so, we get:

 $$V(s)=\max _{a}\left(R(s, a) + \gamma \sum{s^{\prime}} P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$

 Taking the new notations step by step:

 * P(s, a, s′) - the probability of moving from room s to room s′ with action a
 * $$\sum_{s^{\prime}} P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)$$ - expectation of the situation that the robot incurs randomness

 Notice, everything else from the above two points is exactly the same. Let’s assume the following probabilities are associated with each of the turns the robot might take while being in that red room (to go to the green room).

 An environment with an agent (with probabilities)

 When we associate probabilities to each of these turns, we essentially mean that there is an 80% chance that the robot will take the upper turn. If we put all the required values in our equation, we get:

 $$V(s)=\max _{a} (R(s, a) + \gamma((0.8V(room_{up})) +( 0.1V(room_{down})) + ...))$$

 Note that the value footprints will now change due to the fact that we are incorporating stochasticity here. But this time, we will not calculate those value footprints. Instead, we will let the robot to figure it out (more on this in a moment).

 Up until this point, we have not considered about rewarding the robot for its action of going into a particular room. We are only rewarding the robot when it gets to the destination. Ideally, there should be a reward for every action the robot takes to help it better assess the quality of its actions. The rewards need not be always the same. But it is much better than having some amount reward for the actions than having no rewards at all. This idea is known as the living penalty. In reality, the rewarding system can be very complex and particularly modeling sparse rewards is an active area of research in the domain reinforcement learning. If you would like to give a spin to this topic then following resources might come in handy:

 * Curiosity-driven Exploration by Self-supervised Prediction
 * Curiosity and Procrastination in Reinforcement Learning
 * Learning to Generalize from Sparse and Underspecified Rewards

 In the next section, we will introduce the notion of the quality of an action rather than looking at the value of going into a particular room (V(s)).

 Transitioning to Q-Learning

 By now, we have got the following equation which gives us a value of going to a particular state (form now on, we will refer to the rooms as states) taking the stochasticity of the environment into the account:

 $$V(s)=\max {a}\left(R(s, a) + \gamma \sum{s^{\prime}} P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$

 We have also learned very briefly about the idea of living penalty which deals with associating each move of the robot with a reward.

 Q-Learning poses an idea of assessing the quality of an action that is taken to move to a state rather than determining the possible value of the state (value footprint) being moved to.

 Earlier, we had:

 An environment with an agent (with possible value footprints)

 If we incorporate the idea of assessing the quality of actions for moving to a certain state s′.

 An environment with an agent (with quality of actions)

 The robot now has four different states to choose from and along with that, there are four different actions also for the current state it is in. So how do we calculate Q(s, a) i.e. the cumulative quality of the possible actions the robot might take? Let’s break down.

 From this equation $$V(s)=\max {a}\left(R(s, a) + \gamma \sum{s^{\prime}} P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$, if we discard the max() function, we get:

 $$R(s, a) + \gamma \sum_{s^{\prime}}\left(P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$

 Essentially, in the equation that produces V(s), we are considering all possible actions and all possible states (from the current state the robot is in) and then we are taking the maximum value caused by taking a certain action. The above equation produces a value footprint is for just one possible action. In fact, we can think of it as the quality of the action:

 $$Q(s, a)=R(s, a) + \gamma \sum_{s^{\prime}}\left(P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$

 Now that we have got an equation to quantify the quality of a particular action we are going to make a little adjustment in the above equation. We can now say that V(s) is the maximum of all the possible values of Q(s, a). Let’s utilize this fact and replace V(s′) as a function of Q():

 $$Q(s, a)=R(s, a) + \gamma \sum_{s^{\prime}}\left(P\left(s, a, s^{\prime}\right) \max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime}\right)\right)$$

 But why would we do that?

 To ease our calculations. Because now, we have only one function Q() (which is also at the core of the dynamic programming paradigm) to calculate and R(s, a) is a quantified metric which produces rewards of moving to a certain state. The qualities of the actions are called Q-values. And from now on, we will refer the value footprints as the Q-values.

 We now have the last piece of the puzzle remaining i.e. temporal difference before we jump to the implementation part and we are going to study that in the next section.

 The last piece of the puzzle: Temporal difference

 Recollect the statement from a previous section:

 But this time, we will not calculate those value footprints. Instead, we will let the robot to figure it out (more on this in a moment).

 Temporal Difference is the component that will help the robot to calculate the Q-values with respect to the changes in the environment over time. Consider our robot is currently in the marked state and it wants to move to the upper state. Note that the robot already knows the Q-value of making the action i.e. moving to the upper state.

 An environment with an agent

 We know that the environment is stochastic in nature and the reward that the robot will get after moving to the upper state might be different from an earlier observation. So how do we capture this change (read difference)? We recalculate the new Q(s, a) with the same formula and subtract the previously known Q(s, a) from it.

 The equation that we just derived gives the temporal difference in the Q-values which further helps to capture the random changes that the environment may impose. The new Q(s, a) is updated as the following:

 $$Q_{t}(s, a)=Q_{t-1}(s, a)+\alpha T D_{t}(a, s)$$

 where,

 * ɑ is the learning rate which controls how quickly the robot adopts to the random changes imposed by the environment
 * $$Q_t (s, a)$$ is the current Q-value
 * $$Q_{t-1} (s, a)$$ is the previously recorded Q-value

 If we replace $$TD_t(s, a)$$ with its full-form equation, we should get:

 $$Q_{t}(s, a)=Q_{t-1}(s, a)+\alpha\left(R(s, a)+\gamma \max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime}\right)-Q_{t-1}(s, a)\right)$$

 We now have all the little pieces of Q-Learning together to move forward to its implementation part. Feel free to review the problem statement once which we discussed in the very beginning.

 Implementing Q-Learning in Python with Numpy

 If you do not have a local setup, you can run this notebook directly on FloydHub by just clicking on the below button -

 To implement the algorithm, we need to understand the warehouse locations and how that can be mapped to different states. Let’s start by recollecting the sample environment shown earlier:

 A sample environment

 Let’s map each of the above locations in the warehouse to numbers (states). It will ease our calculations.

 # Define the states
 location_to_state = {
 'L1' : 0,
 'L2' : 1,
 'L3' : 2,
 'L4' : 3,
 'L5' : 4,
 'L6' : 5,
 'L7' : 6,
 'L8' : 7,
 'L9' : 8
 }
 

 We will only be using Numpy as our dependency. So let’s import that aliased as np:

 import numpy as np

 The next step is to define the actions which as mentioned above represents the transition to the next state:

 # Define the actions
 actions = [0,1,2,3,4,5,6,7,8]
 

 Now the reward table:

 # Define the rewards
 rewards = np.array([[0,1,0,0,0,0,0,0,0],
 [1,0,1,0,1,0,0,0,0],
 [0,1,0,0,0,1,0,0,0],
 [0,0,0,0,0,0,1,0,0],
 [0,1,0,0,0,0,0,1,0],
 [0,0,1,0,0,0,0,0,0],
 [0,0,0,1,0,0,0,1,0],
 [0,0,0,0,1,0,1,0,1],
 [0,0,0,0,0,0,0,1,0]])
 

 If you understood it correctly, there isn't any real barrier limitation as depicted in the image. For example, the transition L4 to L1 is allowed but the reward will be zero to discourage that path.

 In the above code snippet, we took each of the states and put ones in the respective state that are directly reachable from a certain state. Refer to the reward table once again. The above array construction will be easy to understand then. Note that, we did not consider the top-priority location (L6) yet.

 Table of rewards (rewards matrix)

 We would also need an inverse mapping from the states back to original location indicators. It will be clearer when we reach to the utter depths of the algorithm. The following line of code will do this for us.

 # Maps indices to locations
 state_to_location = dict((state,location) for location,state in location_to_state.items())
 

 Before we write a helper function which will yield the optimal route for going from one location to another, let’s specify the two parameters of the Q-Learning algorithm - ɑ (learning rate) and 𝜸 (discount factor).

 # Initialize parameters
 gamma = 0.75 # Discount factor 
 alpha = 0.9 # Learning rate 
 

 We will now define a function get_optimal_route() which will:

 * Take two arguments:
 * starting location in the warehouse and
 * end location in the warehouse respectively
 * And will return the optimal route for reaching the end location from the starting location in the form of an ordered list containing the letters

 We will start defining the function by initializing the Q-values to be all zeros.

 # Initializing Q-Values
 Q = np.array(np.zeros([9,9]))
 

 For convenience, we will copy the rewards matrix rewards to a separate variable and will operate on that.

 # Copy the rewards matrix to new Matrix
 rewards_copy = np.copy(rewards)
 

 [9,9] as we have a total of 9 locations. The function will take a starting location and an ending location. So, let’s set the priority of the ending location to a larger integer like 999 and get the ending state from the location letter (such as L1, L2 and so on).

 # Get the ending state corresponding to the ending location as given
 ending_state = location_to_state[end_location]
 
 # With the above information automatically set the priority of the
 # given ending state to the highest one
 rewards_copy[ending_state,ending_state] = 999
 

 Learning is a continuous process, hence we will let the robot to explore the environment for a while and we will do it by simply looping it through 1000 times. We will then pick a state randomly from the set of states we defined above and we will call it current_state.

 for i in range(1000):
 # Pick up a state randomly
 current_state = np.random.randint(0,9)
 

 We then, being inside the loop, iterate through the rewards matrix to get the states that are directly reachable from the randomly chosen current state and we will assign those state in a list named playable_actions.

 Note that so far we have not bothered about the starting location yet.

 playable_actions = []
 # Iterate through the new rewards matrix and get the actions > 0
 for j in range(9):
 if rewards_copy[current_state,j] > 0:
 playable_actions.append(j)
 

 We will now choose a state randomly from the playable_actions.

 # Pick an action randomly from the list of playable actions leading us to the next 
 # state
 next_state = np.random.choice(playable_actions)
 

 We can now compute the temporal difference and update the Q-values accordingly. Here is the formula of temporal difference for your convenience:

 Here is the way to update the Q-values using the temporal difference:

 $$Q_{t}(s, a)=Q_{t-1}(s, a)+\alpha T D_{t}(a, s)$$

 # Compute the temporal difference
 # The action here exactly refers to going to the next state
 TD = rewards_copy[current_state,next_state] + gamma * Q[next_state, np.argmax(Q[next_state,])] - Q[current_state,next_state]
 
 # Update the Q-Value using the Bellman equation
 Q[current_state,next_state] += alpha * TD
 

 The good news is we are done implementing the most critical part of the process and up until now the definition of get_optimal_route() should look like:

 def get_optimal_route(start_location,end_location):
 # Copy the rewards matrix to new Matrix
 rewards_new = np.copy(rewards)
 
 # Get the ending state corresponding to the ending location as given
 ending_state = location_to_state[end_location]
 
 # With the above information automatically set the priority of the given ending 
 # state to the highest one
 rewards_new[ending_state,ending_state] = 999
 
 # -----------Q-Learning algorithm-----------
 
 # Initializing Q-Values
 Q = np.array(np.zeros([9,9]))
 
 # Q-Learning process
 for i in range(1000):
 # Pick up a state randomly
 current_state = np.random.randint(0,9) # Python excludes the upper bound
 # For traversing through the neighbor locations in the maze
 playable_actions = []
 # Iterate through the new rewards matrix and get the actions > 0
 for j in range(9):
 if rewards_new[current_state,j] > 0:
 playable_actions.append(j)
 # Pick an action randomly from the list of playable actions 
 # leading us to the next state
 next_state = np.random.choice(playable_actions)
 # Compute the temporal difference
 # The action here exactly refers to going to the next state
 TD = rewards_new[current_state,next_state] + gamma * Q[next_state, np.argmax(Q[next_state,])] - Q[current_state,next_state]
 # Update the Q-Value using the Bellman equation
 Q[current_state,next_state] += alpha * TD
 

 We will now start the other half of finding the optimal route. We will first initialize the optimal route with the starting location.

 # Initialize the optimal route with the starting location
 route = [start_location]
 

 We currently do not know about the next move of the robot. Thus we will set the next location to also be the starting location.

 next_location = start_location

 Since we do not know the exact number of iterations the robot will take in order to find out the optimal route, we will simply loop the next set of processes until the next location is not equal to the ending location. This is where we will terminate the loop.

 # We don't know about the exact number of iterations needed to reach to the final 
 # location hence while loop will be a good choice for iteratiing
 while(next_location != end_location):
 # Fetch the starting state
 starting_state = location_to_state[start_location]
 # Fetch the highest Q-value pertaining to starting state
 next_state = np.argmax(Q[starting_state,])
 # We got the index of the next state. But we need the corresponding letter. 
 next_location = state_to_location[next_state]
 route.append(next_location)
 # Update the starting location for the next iteration
 start_location = next_location
 
 return route
 

 Finally, we return the route.

 return route

 Here is the whole get_optimal_route() function:

 def get_optimal_route(start_location,end_location):
 # Copy the rewards matrix to new Matrix
 rewards_new = np.copy(rewards)
 
 # Get the ending state corresponding to the ending location as given
 ending_state = location_to_state[end_location]
 
 # With the above information automatically set the priority of 
 # the given ending state to the highest one
 rewards_new[ending_state,ending_state] = 999
 
 # -----------Q-Learning algorithm-----------
 
 # Initializing Q-Values
 Q = np.array(np.zeros([9,9]))
 
 # Q-Learning process
 for i in range(1000):
 # Pick up a state randomly
 current_state = np.random.randint(0,9) # Python excludes the upper bound
 
 # For traversing through the neighbor locations in the maze
 playable_actions = []
 
 # Iterate through the new rewards matrix and get the actions > 0
 for j in range(9):
 if rewards_new[current_state,j] > 0:
 playable_actions.append(j)
 
 # Pick an action randomly from the list of playable actions 
 # leading us to the next state
 next_state = np.random.choice(playable_actions)
 
 # Compute the temporal difference
 # The action here exactly refers to going to the next state
 TD = rewards_new[current_state,next_state] + gamma * Q[next_state, np.argmax(Q[next_state,])] - Q[current_state,next_state]
 
 # Update the Q-Value using the Bellman equation
 Q[current_state,next_state] += alpha * TD
 
 # Initialize the optimal route with the starting location
 route = [start_location]
 # We do not know about the next location yet, so initialize with the value of 
 # starting location
 next_location = start_location
 
 # We don't know about the exact number of iterations
 # needed to reach to the final location hence while loop will be a good choice 
 # for iteratiing
 
 while(next_location != end_location):
 # Fetch the starting state
 starting_state = location_to_state[start_location]
 
 # Fetch the highest Q-value pertaining to starting state
 next_state = np.argmax(Q[starting_state,])
 
 # We got the index of the next state. But we need the corresponding letter. 
 next_location = state_to_location[next_state]
 route.append(next_location)
 
 # Update the starting location for the next iteration
 start_location = next_location
 
 return route
 

 If we call print(get_optimal_route('L9', 'L1')), we should get:

 ['L9', 'L8', 'L5', 'L2', 'L1']

 Note how the program considers the barriers that are present in the environment. For fun, you can change the ɑ and 𝜸 parameters to see how the learning process changes. With this, let me move to the conclusion section where we will be discussing how you can take your reinforcement learning journey further.

 Let’s refactor the code a little bit to conform to OOP paradigm. We will define a class named QAgent() containing the following two methods apart from init:

 * training(self, start_location, end_location, iterations) which will help the robot to obtain the Q-values from the environment
 * get_optimal_route(self, start_location, end_location, next_location, route, Q) which will get the robot an optimal route from a point to another

 Let’s first define the __init__() method which would initialize the class constructor:

 def __init__(self, alpha, gamma, location_to_state, actions, rewards, state_to_location, Q):
 
 self.gamma = gamma 
 self.alpha = alpha 
 
 self.location_to_state = location_to_state
 self.actions = actions
 self.rewards = rewards
 self.state_to_location = state_to_location
 
 self.Q = Q
 

 Now comes the training() method:

 def training(self, start_location, end_location, iterations):
 
 rewards_new = np.copy(self.rewards)
 
 ending_state = self.location_to_state[end_location]
 rewards_new[ending_state, ending_state] = 999
 
 for i in range(iterations):
 current_state = np.random.randint(0,9) 
 playable_actions = []
 
 for j in range(9):
 if rewards_new[current_state,j] > 0:
 playable_actions.append(j)
 
 next_state = np.random.choice(playable_actions)
 TD = rewards_new[current_state,next_state] + 
 self.gamma * self.Q[next_state, np.argmax(self.Q[next_state,])] - self.Q[current_state,next_state]
 
 self.Q[current_state,next_state] += self.alpha * TD
 
 route = [start_location]
 next_location = start_location
 
 # Get the route 
 self.get_optimal_route(start_location, end_location, next_location, route, self.Q)
 

 Finally the get_optimal_route() method.

 def get_optimal_route(self, start_location, end_location, next_location, route, Q):
 
 while(next_location != end_location):
 starting_state = self.location_to_state[start_location]
 next_state = np.argmax(Q[starting_state,])
 next_location = self.state_to_location[next_state]
 route.append(next_location)
 start_location = next_location
 
 print(route)
 

 The entire class definition should look like:

 class QAgent():
 
 # Initialize alpha, gamma, states, actions, rewards, and Q-values
 def __init__(self, alpha, gamma, location_to_state, actions, rewards, state_to_location, Q):
 
 self.gamma = gamma 
 self.alpha = alpha 
 
 self.location_to_state = location_to_state
 self.actions = actions
 self.rewards = rewards
 self.state_to_location = state_to_location
 
 self.Q = Q
 
 # Training the robot in the environment
 def training(self, start_location, end_location, iterations):
 
 rewards_new = np.copy(self.rewards)
 
 ending_state = self.location_to_state[end_location]
 rewards_new[ending_state, ending_state] = 999
 
 for i in range(iterations):
 current_state = np.random.randint(0,9) 
 playable_actions = []
 
 for j in range(9):
 if rewards_new[current_state,j] > 0:
 playable_actions.append(j)
 
 next_state = np.random.choice(playable_actions)
 TD = rewards_new[current_state,next_state] + \
 self.gamma * self.Q[next_state, np.argmax(self.Q[next_state,])] - self.Q[current_state,next_state]
 
 self.Q[current_state,next_state] += self.alpha * TD
 
 route = [start_location]
 next_location = start_location
 
 # Get the route 
 self.get_optimal_route(start_location, end_location, next_location, route, self.Q)
 
 # Get the optimal route
 def get_optimal_route(self, start_location, end_location, next_location, route, Q):
 
 while(next_location != end_location):
 starting_state = self.location_to_state[start_location]
 next_state = np.argmax(Q[starting_state,])
 next_location = self.state_to_location[next_state]
 route.append(next_location)
 start_location = next_location
 
 print(route)
 

 Once the class is compiled, you should be able to create a class object and call the training() method like so:

 qagent = QAgent(alpha, gamma, location_to_state, actions, rewards, state_to_location, Q)
 qagent.training('L9', 'L1', 1000)
 

 Notice that every is exactly similar to previous chunk of code but the refactored version indeed looks more elegant and modular. And yes, the output will also be same.

 ['L9', 'L8', 'L5', 'L2', 'L1']

 Conclusion and further steps

 We finally have come to the very end of the article. We covered a lot of preliminary grounds of reinforcement learning that will be useful if you are planning to further strengthen your knowledge of reinforcement learning. We also implemented the simplest reinforcement learning just by using Numpy. These base scratch implementations are not only for just fun but also they help tremendously to know the nuts and bolts of an algorithm.

 Reinforcement learning has given solutions to many problems from a wide variety of different domains. One that I particularly like is Google’s NasNet which uses deep reinforcement learning for finding an optimal neural network architecture for a given dataset.

 Let’s now review some of the best resources for breaking into reinforcement learning in a serious manner:

 * Reinforcement Learning, Second Edition: An Introduction by Richard S. Sutton and Andrew G. Barto which is considered to be the textbook of reinforcement learning
 * Practical Reinforcement Learning a course designed by the National Research University Higher School of Economics offered by Coursera
 * Reinforcement Learning a course designed by the Georgia University and offered by Udacity
 * If you are interested in the conjunction of meta-learning and reinforcement learning then you may follow this article
 * How about combining deep learning + reinforcement learning? Check out Deep RL Bootcamp.
 * Deep Reinforcement Learning Hands-On a book by Maxim Lapan which covers many cutting edge RL concepts like deep Q-networks, value iteration, policy gradients and so on.
 * MIT Deep Learning a course taught by Lex Fridman which teaches you how different deep learning applications are used in autonomous vehicle systems and more
 * Introduction to Reinforcement Learning a course taught by one of the main leaders in the game of reinforcement learning - David Silver
 * Spinning Up in Deep RL a course offered from the house of OpenAI which serves as your guide to connecting the dots between theory and practice in deep reinforcement learning
 * Controlling a 2D Robotic Arm with Deep Reinforcement Learning an article which shows how to build your own robotic arm best friend by diving into deep reinforcement learning
 * Spinning Up a Pong AI With Deep Reinforcement Learning an article which shows you to code a vanilla policy gradient model that plays the beloved early 1970s classic video game Pong in a step-by-step manner

 The list is kind of handpicked for those who really want to step up their game in reinforcement learning. So if you are one among them, don’t forget to check out the resources.

 I hope you enjoyed the article and you will take it forward to make applications that can adapt with respect to the environment they are employed to.

 Thanks to Alessio and Bharath of FloydHub for sharing their valuable feedback on the article. Big thanks to the entire FloydHub team for letting me run the accompanying notebook on their platform. If you haven’t checked FloydHub yet, give FloydHub a spin for your Machine Learning and Deep Learning projects. There is no going back once you’ve learned how easy they make it.

 FloydHub Call for AI writers

 Want to write amazing articles like Sayak and play your role in the long road to Artificial General Intelligence? We are looking for passionate writers, to build the world's best blog for practical applications of groundbreaking A.I. techniques. FloydHub has a large reach within the AI community and with your help, we can inspire the next wave of AI. Apply now and join the crew!

 About Sayak Paul
 Sayak loves everything deep learning. He goes by the motto of understanding complex things and helping people understand them as easily as possible. Sayak is an extensive blogger and all of his blogs can be found here. He is also working with his friends on the application of deep learning in Phonocardiogram classification. Sayak is also a FloydHub AI Writer. He is always open to discussing novel ideas and taking them forward to implementations. You can connect with Sayak on LinkedIn and Twitter.

 Sign up for more like this.

 Enter your email
 Subscribe
 Featured

 FloydHub has shut down

 FloydHub - our ML platform used by thousands of Data Scientists and AI enthusiasts was shut down on August 20, 2021.
 Aug 21, 2021 1 min read

 NLP Datasets: How good is your deep learning model?

 With the rapid advance in NLP models we have outpaced out ability to measure just how good they are at human level language tasks. We need better NLP datasets now more than ever to both evaluate how good these models are and to be able to tweak them for out own business domains.
 Jun 10, 2020 31 min read

 The Future of AI is Open

 This Humans of ML interview with Han Xiao covers the ethics of AI, open-source entrepreneurship, how writing made Han a better coder, and more.
 Jun 2, 2020 18 min read
 FloydHub Blog © 2023
 Powered by Ghost
Learn about the basic concepts of reinforcement learning and implement a simple RL algorithm called Q-Learning.
 
 
 
 
 
 
 Have you ever trained a pet and rewarded it for every correct command you asked for? Do you know that this simple way of rewarding behavior can be modeled in a robot or a software program to make it do useful things? In this article, we are going to step into the world of reinforcement learning, another beautiful branch of artificial intelligence, which lets machines learn on their own in a way different from traditional machine learning. Particularly, we will be covering the simplest reinforcement learning algorithm i.e. the Q-Learning algorithm in great detail. In the first half of the article, we will be discussing reinforcement learning in general with examples where reinforcement learning is not just desired but also required. We will then study the Q-Learning algorithm along with an implementation in Python using Numpy.   Let’s start, shall we? The brass tacks: What is reinforcement learning?   A rough framework of reinforcement learning   Throughout our lives, we perform a number of actions to pursue our dreams. Some of them bring us good rewards and others do not. Along the way, we keep exploring different paths and try to figure out which action might lead to better rewards. We work hard towards our dreams utilizing the feedback we get based on our actions to improve our strategies. They help us determine how close we are to achieving our goals. Our mental states change continuously to representing this closeness.  In that description of how we pursue our goals in daily life, we framed for ourselves a representative analogy of reinforcement learning. Let me summarize the above example reformatting the main points of interest.  Our reality contains environments in which we perform numerous actions. Sometimes we get good or positive rewards for some of these actions in order to achieve goals. During the entire course of life, our mental and physical states evolve. We strengthen our actions in order to get as many rewards as possible.  The key entities of interest are - environment, action, reward and state. Let’s give ourselves a name as well - we are agents in this whole game of life. This whole paradigm of exploring our lives and learning from it through actions, rewards and states establishes the foundation of reinforcement learning. In fact, this is almost how we act in any given circumstance in our lives, isn’t it? It turns out that the whole idea of reinforcement learning is pretty empirical in nature. Now, think for a moment, how would we train robots and machines to do the kind of useful tasks we humans do. Be it switching off the television, or moving things around, or organizing bookshelves. Fundamentally, these tasks are not about finding a function mapping inputs to outputs or finding hidden representations within input data. These are a completely different set of tasks and require a different learning paradigm for a computer to be able to perform these tasks.   Andriy Burkov in his The Hundred Page Machine Learning Book  describes reinforcement learning as:  Reinforcement learning solves a particular kind of problem where decision making is sequential, and the goal is long-term, such as game playing, robotics, resource management, or logistics.   For a robot, an environment is a place where it has been put to use. Remember this robot is itself the agent. For example, a textile factory where a robot is used to move materials from one place to another. We will come to its states, actions, rewards later. The tasks we discussed just now, have a property in common -  these tasks involve an environment and expect the agents to learn from that environment. This is where traditional machine learning fails and hence the need for reinforcement learning.  So, the headline AI Bots Join Forces To Beat Top Human Dota 2 Team that shook the gaming world is a direct byproduct of reinforcement learning. Beyond beating humans in game-playing, there are some other marvelous use cases of reinforcement learning:   Optimizing business processes   Minimizing energy costs   Maximizing revenue shares of a company   and lots more.  In the next section, we will begin by defining the problem which is very relevant in the field of reinforcement learning. After that, we will study its agents, environment, states, actions and rewards. We will then directly proceed towards the Q-Learning algorithm. Recipes for reinforcement learning It is good to have an established overview of the problem that is to be solved using reinforcement learning, Q-Learning in this case. It helps to define the main components of a reinforcement learning solution i.e. agents, environment, actions, rewards and states.  Defining the problem statement  We are to build a few autonomous robots for a guitar building factory. These robots will help the guitar luthiers by conveying them the necessary guitar parts that they would need in order to craft a guitar. These different parts are located at nine different positions within the factory warehouse. Guitars parts include polished wood stick for the fretboard, polished wood for the guitar body, guitar pickups and so on. The luthiers have prioritized the location that contains body woods to be the topmost. They have provided the priorities for other locations as well which we will look in a moment. These locations within the factory warehouse look like -    A sample environment   As we can see there are little obstacles present (represented with smoothed lines) in between the locations. L6 is the top-priority location that contains the polished wood for preparing guitar bodies. Now, the task is to enable the robots so that they can find the shortest route from any given location to another location on their own.  The agents, in this case, are the robots. The environment is the guitar factory warehouse.  The states  The states are the locations. The location in which a particular robot is present in the particular instance of time will denote its state. Machines understand numbers rather than letters. So, let’s map the location codes to numbers.   Locations mapped as states    The actions  In our example, the actions will be the direct locations that a robot can go to from a particular location. Consider, a robot is at the L8 location and the direct locations to which it can move are L5, L7 and L9. The below figure may come in handy in order to visualize this.   Sample actions    As you might have already guessed the set of actions here is nothing but the set of all possible states of the robot. For each location, the set of actions that a robot can take will be different. For example, the set of actions will change if the robot is in L1.  The rewards  By now, we have the following two sets:  A set of states: $$S = {0, 1, 2, 3, 4, 5, 6, 7, 8}$$ A set of actions: $$A = {0, 1, 2, 3, 4, 5, 6, 7, 8}$$  The rewards, now, will be given to a robot if a location (read it state) is directly reachable from a particular location. Let’s take an example:  L9 is directly reachable from L8. So, if a robot goes from L8 to L9 and vice-versa, it will be rewarded by 1. If a location is not directly reachable from a particular location, we do not give any reward (a reward of 0). Yes, the reward is just a number here and nothing else. It enables the robots to make sense of their movements helping them in deciding what locations are directly reachable and what are not. With this cue, we can construct a reward table which contains all the reward values mapping between all the possible states (locations).   Table of rewards   In the above table, we have all the possible rewards that a robot can get by moving in between the different states.  Now comes an interesting decision. Remember from the previous sections that the luthier prioritized L6 to be of the topmost? So, how do we incorporate this fact in the above table? This is done by associating the topmost priority location with a very higher reward than the usual ones. Let’s put 999 in the cell (L6, L6):   Table of rewards with a higher reward for the topmost location   We have now formally defined all the vital components for the solution we are aiming for the problem discussed above. We will shift gears a bit and study some of the fundamental concepts that prevail in the world of reinforcement learning. We will start with the Bellman Equation.  The Bellman Equation  Consider the following square of rooms which is analogous to the actual environment from our original problem but without the barriers.   An empty environment   Now suppose a robot needs to go to the room, marked in green from its current position (A) using the specified direction.   Sample environment, agent and directions to proceed   How can we enable the robot to do this programmatically? One idea would be to introduce some kind of footprint which the robot would be able to follow like below.   An environment with value footprints   Here, a constant value is specified in each of the rooms which will come along the robot’s way if it follows the direction specified above. In this way, if it starts at location A, it will be able to scan through this constant value and will move accordingly. But this would only work if the direction is prefixed and the robot always starts at location A. Now, consider the robot starts at the following location:   An environment with value footprint (contd.)   The robot now sees footprints in two different directions. It is, therefore, unable to decide which way to go in order to get to the destination (green room). It happens primarily because the robot does not have a way to remember the directions to proceed. So, our job now is to enable the robot with a memory. This is where the Bellman Equation comes into play: $$V(s)=\max _{a}\left(R(s, a) + \gamma V\left(s^{\prime}\right)\right)$$ where,  s = a particular state (room) a = action (moving between the rooms) s′ = state to which the robot goes from s 𝜸 = discount factor (we will get to it in a moment) R(s, a) = a reward function which takes a state s and action a and outputs a reward value V(s) = value of being in a particular state (the footprint)  We consider all the possible actions and take the one that yields the maximum value.  There is one constraint however regarding the value footprint i.e. the room, marked in yellow just below the green room will always have a value of 1 to denote that it is one of the nearest room adjacent to the green room. This is also to ensure that a robot gets a reward when it goes from the yellow room to the green room. Let’s now see how to make sense of the above equation here. We will assume a discount factor of 0.9.   An environment with value footprint (contd.)     * For this room (read state) what will be V(s)? Let’s put the values into the equation straightly: $$V(s)=\max _{a}(0 + 0.9 * 1)=0.9$$ Here, the robot will not get any reward for going to the state (room) marked in yellow, hence R(s, a) = 0 here. The robot knows the values of being in the yellow room hence V(s′) is 1. Following this for the other states, we should get:   An environment with some value footprints computed from the Bellman equation   A couple of things to notice here:  The max() function helps the robot to always choose the state that gives it the maximum value of being in that state. The discount factor 𝜸 notifies the robot about how far it is from the destination. This typically specified by the developer of the algorithm that would be instilled in the robot.   The other states can also be given their respective values in a similar way:   An environment with all the value footprints computed from the Bellman equation   The robot now can proceed its way through the green room utilizing these value footprints even if it is dropped at any arbitrary room in the above square. Now, if a robot lands up in the highlighted (sky blue) room, it will still find two options to choose from. But eventually, either of the paths will be good enough for the robot to take because of the way the value footprints are now laid out.  Notice how the main task of reaching a destination from a particular source got broken down to similar subtasks. It turns out that there is a particular programming paradigm developed especially for solving problems that have repetitive subproblems in them. It is referred to as dynamic programming. It was invented by Richard Bellman in 1954 who also coined the equation we just studied (hence the name, Bellman Equation). Note that this is one of the key equations in the world of reinforcement learning.  If we think realistically, our surroundings do not always work in the way we expect. There is always a bit of stochasticity involved in it. This applies to a robot as well. Sometimes, it might so happen that the robot’s inner machinery got corrupted. Sometimes, the robot may come across some hindrances on its way which may not be known to it beforehand. Sometimes, even if the robot knows that it needs to take the right turn, it will not. So, how do we introduce this stochasticity in our case? Markov Decision Processes. Modeling stochasticity: Markov Decision Processes Consider the robot is currently in the red room and it needs to go to the green room.   An environment with an agent   Let’s now consider, the robot has a slight chance of dysfunctioning and might take the left or right or bottom turn instead of taking the upper turn in order to get to the green room from where it is now (red room). Now, the question is how do we enable the robot to handle this when it is out there in the above environment?   An environment with an agent (with stochasticity)   This is a situation where the decision making regarding which turn is to be taken is partly random and partly under the control of the robot. Partly random because we are not sure when exactly the robot might dysfunction and partly under the control of the robot because it is still making a decision of taking a turn on its own and with the help of the program embedded into it. Here is the definition of Markov Decision Processes (collected from Wikipedia):   A Markov decision process (MDP) is a discrete time stochastic control process. It provides a mathematical framework for modeling decision making in situations where outcomes are partly random and partly under the control of a decision maker. You may focus only on the highlighted part. We have the exact same situation here in our case.  We have now introduced ourselves with the concept of partly random and partly controlled decision making. We need to give this concept a mathematical shape (most likely an equation) which then can be taken further.  You might be surprised to see that we can do this with the help of the Bellman Equation with a few minor tweaks.  Here is the original Bellman Equation, again: $$V(s)=\max _{a}\left(R(s, a) + \gamma V\left(s^{\prime}\right)\right)$$ What needs to be changed in the above equation so that we can introduce some amount of randomness here? As long as we are not sure when the robot might not take the expected turn, we are then also not sure in which room it might end up in which is nothing but the room it moves from its current room. At this point, according to the above equation, we are not sure of s′ which is the next state (room, as we were referring them). But we do know all the probable turns the robot might take! In order to incorporate each of these probabilities into the above equation, we need to associate a probability with each of the turns to quantify that the robot has got x% chance of taking this turn. If we do so, we get: $$V(s)=\max _{a}\left(R(s, a) + \gamma \sum{s^{\prime}} P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$ Taking the new notations step by step:   P(s, a, s′) - the probability of moving from room s to room s′ with action a  $$\sum_{s^{\prime}} P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)$$ - expectation of the situation that the robot incurs randomness  Notice, everything else from the above two points is exactly the same. Let’s assume the following probabilities are associated with each of the turns the robot might take while being in that red room (to go to the green room).   An environment with an agent (with probabilities)   When we associate probabilities to each of these turns, we essentially mean that there is an 80% chance that the robot will take the upper turn. If we put all the required values in our equation, we get: $$V(s)=\max _{a} (R(s, a) + \gamma((0.8V(room_{up})) +( 0.1V(room_{down})) + ...))$$  Note that the value footprints will now change due to the fact that we are incorporating stochasticity here. But this time, we will not calculate those value footprints. Instead, we will let the robot to figure it out (more on this in a moment).  Up until this point, we have not considered about rewarding the robot for its action of going into a particular room. We are only rewarding the robot when it gets to the destination. Ideally, there should be a reward for every action the robot takes to help it better assess the quality of its actions. The rewards need not be always the same. But it is much better than having some amount reward for the actions than having no rewards at all. This idea is known as the living penalty. In reality, the rewarding system can be very complex and particularly modeling sparse rewards is an active area of research in the domain reinforcement learning. If you would like to give a spin to this topic then following resources might come in handy:    Curiosity-driven Exploration by Self-supervised Prediction   Curiosity and Procrastination in Reinforcement Learning   Learning to Generalize from Sparse and Underspecified Rewards   In the next section, we will introduce the notion of the quality of an action rather than looking at the value of going into a particular room (V(s)).  Transitioning to Q-Learning  By now, we have got the following equation which gives us a value of going to a particular state (form now on, we will refer to the rooms as states) taking the stochasticity of the environment into the account: $$V(s)=\max {a}\left(R(s, a)  + \gamma \sum{s^{\prime}} P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$ We have also learned very briefly about the idea of living penalty which deals with associating each move of the robot with a reward.  Q-Learning poses an idea of assessing the quality of an action that is taken to move to a state rather than determining the possible value of the state (value footprint) being moved to.  Earlier, we had:   An environment with an agent (with possible value footprints)   If we incorporate the idea of assessing the quality of actions for moving to a certain state s′.   An environment with an agent (with quality of actions)   The robot now has four different states to choose from and along with that, there are four different actions also for the current state it is in. So how do we calculate Q(s, a) i.e. the cumulative quality of the possible actions the robot might take? Let’s break down.  From this equation $$V(s)=\max {a}\left(R(s, a) +  \gamma \sum{s^{\prime}} P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$, if we discard the max() function, we get: $$R(s, a) + \gamma \sum_{s^{\prime}}\left(P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$ Essentially, in the equation that produces V(s), we are considering all possible actions and all possible states (from the current state the robot is in) and then we are taking the maximum value caused by taking a certain action. The above equation produces a value footprint is for just one possible action. In fact, we can think of it as the quality of the action: $$Q(s, a)=R(s, a) + \gamma \sum_{s^{\prime}}\left(P\left(s, a, s^{\prime}\right) V\left(s^{\prime}\right)\right)$$ Now that we have got an equation to quantify the quality of a particular action we are going to make a little adjustment in the above equation. We can now say that V(s) is the maximum of all the possible values of Q(s, a) . Let’s utilize this fact and replace  V(s′) as a function of Q():  $$Q(s, a)=R(s, a) + \gamma \sum_{s^{\prime}}\left(P\left(s, a, s^{\prime}\right) \max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime}\right)\right)$$ But why would we do that?   To ease our calculations. Because now, we have only one function Q() (which is also at the core of the dynamic programming paradigm) to calculate and R(s, a) is a quantified metric which produces rewards of moving to a certain state. The qualities of the actions are called Q-values. And from now on, we will refer the value footprints as the Q-values.  We now have the last piece of the puzzle remaining i.e. temporal difference before we jump to the implementation part and we are going to study that in the next section.  The last piece of the puzzle: Temporal difference  Recollect the statement from a previous section:  But this time, we will not calculate those value footprints. Instead, we will let the robot to figure it out (more on this in a moment).   Temporal Difference is the component that will help the robot to calculate the Q-values with respect to the changes in the environment over time. Consider our robot is currently in the marked state and it wants to move to the upper state. Note that the robot already knows the Q-value of making the action i.e. moving to the upper state.     An environment with an agent   We know that the environment is stochastic in nature and the reward that the robot will get after moving to the upper state might be different from an earlier observation. So how do we capture this change (read difference)? We recalculate the new Q(s, a) with the same formula and subtract the previously known Q(s, a) from it.   The equation that we just derived gives the temporal difference in the Q-values which further helps to capture the random changes that the environment may impose. The new Q(s, a) is updated as the following: $$Q_{t}(s, a)=Q_{t-1}(s, a)+\alpha T D_{t}(a, s)$$ where,  ɑ is the learning rate which controls how quickly the robot adopts to the random changes imposed by the environment  $$Q_t (s, a)$$ is the current Q-value  $$Q_{t-1} (s, a)$$ is the previously recorded Q-value   If we replace $$TD_t(s, a)$$ with its full-form equation, we should get: $$Q_{t}(s, a)=Q_{t-1}(s, a)+\alpha\left(R(s, a)+\gamma \max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime}\right)-Q_{t-1}(s, a)\right)$$ We now have all the little pieces of Q-Learning together to move forward to its implementation part. Feel free to review the problem statement once which we discussed in the very beginning.  Implementing Q-Learning in Python with Numpy  If you do not have a local setup, you can run this notebook directly on FloydHub by just clicking on the below button -
 
To implement the algorithm, we need to understand the warehouse locations and how that can be mapped to different states. Let’s start by recollecting the sample environment shown earlier:  A sample environment  Let’s map each of the above locations in the warehouse to numbers (states). It will ease our calculations.   # Define the states
location_to_state = {
 'L1' : 0,
 'L2' : 1,
 'L3' : 2,
 'L4' : 3,
 'L5' : 4,
 'L6' : 5,
 'L7' : 6,
 'L8' : 7,
 'L9' : 8
}
 
We will only be using Numpy as our dependency. So let’s import that aliased as np:   import numpy as np 
The next step is to define the actions which as mentioned above represents the transition to the next state:   # Define the actions
actions = [0,1,2,3,4,5,6,7,8]
 
Now the reward table:  # Define the rewards
rewards = np.array([[0,1,0,0,0,0,0,0,0],
 [1,0,1,0,1,0,0,0,0],
 [0,1,0,0,0,1,0,0,0],
 [0,0,0,0,0,0,1,0,0],
 [0,1,0,0,0,0,0,1,0],
 [0,0,1,0,0,0,0,0,0],
 [0,0,0,1,0,0,0,1,0],
 [0,0,0,0,1,0,1,0,1],
 [0,0,0,0,0,0,0,1,0]])
 
If you understood it correctly, there isn't any real barrier limitation as depicted in the image. For example, the transition L4 to L1 is allowed but the reward will be zero to discourage that path. In the above code snippet, we took each of the states and put ones in the respective state that are directly reachable from a certain state. Refer to the reward table once again. The above array construction will be easy to understand then. Note that, we did not consider the top-priority location (L6) yet.   Table of rewards (rewards matrix)   We would also need an inverse mapping from the states back to original location indicators. It will be clearer when we reach to the utter depths of the algorithm. The following line of code will do this for us.  # Maps indices to locations
state_to_location = dict((state,location) for location,state in location_to_state.items())
 
Before we write a helper function which will yield the optimal route for going from one location to another, let’s specify the two parameters of the Q-Learning algorithm - ɑ (learning rate) and 𝜸 (discount factor).  # Initialize parameters
gamma = 0.75 # Discount factor 
alpha = 0.9 # Learning rate 
 
We will now define a function get_optimal_route() which will:  Take two arguments: starting location in the warehouse and end location in the warehouse respectively And will return the optimal route for reaching the end location from the starting location in the form of an ordered list containing the letters  We will start defining the function by initializing the Q-values to be all zeros.  # Initializing Q-Values
Q = np.array(np.zeros([9,9]))
 
For convenience, we will copy the rewards matrix rewards to a separate variable and will operate on that.  # Copy the rewards matrix to new Matrix
rewards_copy = np.copy(rewards)
 
 [9,9] as we have a total of 9 locations. The function will take a starting location and an ending location. So, let’s set the priority of the ending location to a larger integer like 999 and get the ending state from the location letter (such as L1, L2 and so on).  # Get the ending state corresponding to the ending location as given
ending_state = location_to_state[end_location]
# With the above information automatically set the priority of the
# given ending state to the highest one
rewards_copy[ending_state,ending_state] = 999
 
Learning is a continuous process, hence we will let the robot to explore the environment for a while and we will do it by simply looping it through 1000 times. We will then pick a state randomly from the set of states we defined above and we will call it current_state.  for i in range(1000):
 # Pick up a state randomly
 current_state = np.random.randint(0,9)
 
We then, being inside the loop, iterate through the rewards matrix to get the states that are directly reachable from the randomly chosen current state and we will assign those state in a list named playable_actions.  Note that so far we have not bothered about the starting location yet.  playable_actions = []
# Iterate through the new rewards matrix and get the actions &gt; 0
for j in range(9):
 if rewards_copy[current_state,j] &gt; 0:
 playable_actions.append(j)
 
We will now choose a state randomly from the playable_actions.  # Pick an action randomly from the list of playable actions leading us to the next 
# state
 next_state = np.random.choice(playable_actions)
 
We can now compute the temporal difference and update the Q-values accordingly. Here is the formula of temporal difference for your convenience:   Here is the way to update the Q-values using the temporal difference: $$Q_{t}(s, a)=Q_{t-1}(s, a)+\alpha T D_{t}(a, s)$$  # Compute the temporal difference
# The action here exactly refers to going to the next state
TD = rewards_copy[current_state,next_state] + gamma * Q[next_state, np.argmax(Q[next_state,])] - Q[current_state,next_state]
# Update the Q-Value using the Bellman equation
Q[current_state,next_state] += alpha * TD
 
The good news is we are done implementing the most critical part of the process and up until now the definition of get_optimal_route() should look like:  def get_optimal_route(start_location,end_location):
 # Copy the rewards matrix to new Matrix
 rewards_new = np.copy(rewards)
 
 # Get the ending state corresponding to the ending location as given
 ending_state = location_to_state[end_location]
 
 # With the above information automatically set the priority of the given ending 
 # state to the highest one
 rewards_new[ending_state,ending_state] = 999
 # -----------Q-Learning algorithm-----------
 
 # Initializing Q-Values
 Q = np.array(np.zeros([9,9]))
 # Q-Learning process
 for i in range(1000):
 # Pick up a state randomly
 current_state = np.random.randint(0,9) # Python excludes the upper bound
 # For traversing through the neighbor locations in the maze
 playable_actions = []
 # Iterate through the new rewards matrix and get the actions &gt; 0
 for j in range(9):
 if rewards_new[current_state,j] &gt; 0:
 playable_actions.append(j)
 # Pick an action randomly from the list of playable actions 
 # leading us to the next state
 next_state = np.random.choice(playable_actions)
 # Compute the temporal difference
 # The action here exactly refers to going to the next state
 TD = rewards_new[current_state,next_state] + gamma * Q[next_state, np.argmax(Q[next_state,])] - Q[current_state,next_state]
 # Update the Q-Value using the Bellman equation
 Q[current_state,next_state] += alpha * TD
 
We will now start the other half of finding the optimal route. We will first initialize the optimal route with the starting location.  # Initialize the optimal route with the starting location
 route = [start_location]
 
We currently do not know about the next move of the robot. Thus we will set the next location to also be the starting location.  next_location = start_location 
Since we do not know the exact number of iterations the robot will take in order to find out the optimal route, we will simply loop the next set of processes until the next location is not equal to the ending location. This is where we will terminate the loop.  # We don't know about the exact number of iterations needed to reach to the final 
# location hence while loop will be a good choice for iteratiing
 while(next_location != end_location):
 # Fetch the starting state
 starting_state = location_to_state[start_location]
 # Fetch the highest Q-value pertaining to starting state
 next_state = np.argmax(Q[starting_state,])
 # We got the index of the next state. But we need the corresponding letter. 
 next_location = state_to_location[next_state]
 route.append(next_location)
 # Update the starting location for the next iteration
 start_location = next_location
 
 return route
 
Finally, we return the route.  return route 
Here is the whole get_optimal_route() function:  def get_optimal_route(start_location,end_location):
 # Copy the rewards matrix to new Matrix
 rewards_new = np.copy(rewards)
 
 # Get the ending state corresponding to the ending location as given
 ending_state = location_to_state[end_location]
 
 # With the above information automatically set the priority of 
 # the given ending state to the highest one
 rewards_new[ending_state,ending_state] = 999
 # -----------Q-Learning algorithm-----------
 
 # Initializing Q-Values
 Q = np.array(np.zeros([9,9]))
 # Q-Learning process
 for i in range(1000):
 # Pick up a state randomly
 current_state = np.random.randint(0,9) # Python excludes the upper bound
 
 # For traversing through the neighbor locations in the maze
 playable_actions = []
 
 # Iterate through the new rewards matrix and get the actions &gt; 0
 for j in range(9):
 if rewards_new[current_state,j] &gt; 0:
 playable_actions.append(j)
 
 # Pick an action randomly from the list of playable actions 
 # leading us to the next state
 next_state = np.random.choice(playable_actions)
 
 # Compute the temporal difference
 # The action here exactly refers to going to the next state
 TD = rewards_new[current_state,next_state] + gamma * Q[next_state, np.argmax(Q[next_state,])] - Q[current_state,next_state]
 
 # Update the Q-Value using the Bellman equation
 Q[current_state,next_state] += alpha * TD
 # Initialize the optimal route with the starting location
 route = [start_location]
 # We do not know about the next location yet, so initialize with the value of 
 # starting location
 next_location = start_location
 
 # We don't know about the exact number of iterations
 # needed to reach to the final location hence while loop will be a good choice 
 # for iteratiing
 
 while(next_location != end_location):
 # Fetch the starting state
 starting_state = location_to_state[start_location]
 
 # Fetch the highest Q-value pertaining to starting state
 next_state = np.argmax(Q[starting_state,])
 
 # We got the index of the next state. But we need the corresponding letter. 
 next_location = state_to_location[next_state]
 route.append(next_location)
 
 # Update the starting location for the next iteration
 start_location = next_location
 
 return route
 
If we call print(get_optimal_route('L9', 'L1')), we should get:   ['L9', 'L8', 'L5', 'L2', 'L1'] 
Note how the program considers the barriers that are present in the environment. For fun, you can change the ɑ and 𝜸 parameters to see how the learning process changes. With this, let me move to the conclusion section where we will be discussing how you can take your reinforcement learning journey further.  Let’s refactor the code a little bit to conform to OOP paradigm. We will define a class named QAgent() containing the following two methods apart from init:   training(self, start_location, end_location, iterations) which will help the robot to obtain the Q-values from the environment  get_optimal_route(self, start_location, end_location, next_location, route, Q) which will get the robot an optimal route from a point to another  Let’s first define the __init__() method which would initialize the class constructor:  def __init__(self, alpha, gamma, location_to_state, actions, rewards, state_to_location, Q):
 
 self.gamma = gamma 
 self.alpha = alpha 
 self.location_to_state = location_to_state
 self.actions = actions
 self.rewards = rewards
 self.state_to_location = state_to_location
 self.Q = Q
 
Now comes the training() method:  def training(self, start_location, end_location, iterations):
 rewards_new = np.copy(self.rewards)
 ending_state = self.location_to_state[end_location]
 rewards_new[ending_state, ending_state] = 999
 for i in range(iterations):
 current_state = np.random.randint(0,9) 
 playable_actions = []
 for j in range(9):
 if rewards_new[current_state,j] &gt; 0:
 playable_actions.append(j)
 next_state = np.random.choice(playable_actions)
 TD = rewards_new[current_state,next_state] + 
 self.gamma * self.Q[next_state, np.argmax(self.Q[next_state,])] - self.Q[current_state,next_state]
 self.Q[current_state,next_state] += self.alpha * TD
 route = [start_location]
 next_location = start_location
 # Get the route 
 self.get_optimal_route(start_location, end_location, next_location, route, self.Q)
 
Finally the get_optimal_route() method.  def get_optimal_route(self, start_location, end_location, next_location, route, Q):
 while(next_location != end_location):
 starting_state = self.location_to_state[start_location]
 next_state = np.argmax(Q[starting_state,])
 next_location = self.state_to_location[next_state]
 route.append(next_location)
 start_location = next_location
 print(route)
 
The entire class definition should look like:  class QAgent():
 
 # Initialize alpha, gamma, states, actions, rewards, and Q-values
 def __init__(self, alpha, gamma, location_to_state, actions, rewards, state_to_location, Q):
 
 self.gamma = gamma 
 self.alpha = alpha 
 
 self.location_to_state = location_to_state
 self.actions = actions
 self.rewards = rewards
 self.state_to_location = state_to_location
 
 self.Q = Q
 
 # Training the robot in the environment
 def training(self, start_location, end_location, iterations):
 
 rewards_new = np.copy(self.rewards)
 
 ending_state = self.location_to_state[end_location]
 rewards_new[ending_state, ending_state] = 999
 
 for i in range(iterations):
 current_state = np.random.randint(0,9) 
 playable_actions = []
 for j in range(9):
 if rewards_new[current_state,j] &gt; 0:
 playable_actions.append(j)
 
 next_state = np.random.choice(playable_actions)
 TD = rewards_new[current_state,next_state] + \
 self.gamma * self.Q[next_state, np.argmax(self.Q[next_state,])] - self.Q[current_state,next_state]
 
 self.Q[current_state,next_state] += self.alpha * TD
 route = [start_location]
 next_location = start_location
 
 # Get the route 
 self.get_optimal_route(start_location, end_location, next_location, route, self.Q)
 
 # Get the optimal route
 def get_optimal_route(self, start_location, end_location, next_location, route, Q):
 
 while(next_location != end_location):
 starting_state = self.location_to_state[start_location]
 next_state = np.argmax(Q[starting_state,])
 next_location = self.state_to_location[next_state]
 route.append(next_location)
 start_location = next_location
 
 print(route)
 
Once the class is compiled, you should be able to create a class object and call the training() method like so:  qagent = QAgent(alpha, gamma, location_to_state, actions, rewards, state_to_location, Q)
qagent.training('L9', 'L1', 1000)
 
Notice that every is exactly similar to previous chunk of code but the refactored version indeed looks more elegant and modular. And yes, the output will also be same.  ['L9', 'L8', 'L5', 'L2', 'L1'] 
 Conclusion and further steps  We finally have come to the very end of the article. We covered a lot of preliminary grounds of reinforcement learning that will be useful if you are planning to further strengthen your knowledge of reinforcement learning. We also implemented the simplest reinforcement learning just by using Numpy. These base scratch implementations are not only for just fun but also they help tremendously to know the nuts and bolts of an algorithm.  Reinforcement learning has given solutions to many problems from a wide variety of different domains. One that I particularly like is Google’s NasNet  which uses deep reinforcement learning for finding an optimal neural network architecture for a given dataset.   Let’s now review some of the best resources for breaking into reinforcement learning in a serious manner:    Reinforcement Learning, Second Edition: An Introduction by Richard S. Sutton and Andrew G. Barto which is considered to be the textbook of reinforcement learning   Practical Reinforcement Learning a course designed by the National Research University Higher School of Economics offered by Coursera   Reinforcement Learning a course designed by the Georgia  University and offered by Udacity  If you are interested in the conjunction of meta-learning and reinforcement learning then you may follow this article   How about combining deep learning + reinforcement learning? Check out Deep RL Bootcamp.   Deep Reinforcement Learning Hands-On a book by Maxim Lapan which covers many cutting edge RL concepts like deep Q-networks, value iteration, policy gradients and so on.   MIT Deep Learning a course taught by Lex Fridman which teaches you how different deep learning applications are used in autonomous vehicle systems and more   Introduction to Reinforcement Learning a course taught by one of the main leaders in the game of reinforcement learning - David Silver  Spinning Up in Deep RL a course offered from the house of OpenAI which serves  as your guide to connecting the dots between theory and practice in deep reinforcement learning   Controlling a 2D Robotic Arm with Deep Reinforcement Learning an article which shows how to build your own robotic arm best friend by diving into deep reinforcement learning   Spinning Up a Pong AI With Deep Reinforcement Learning an article which shows you to code a vanilla policy gradient model that plays the beloved early 1970s classic video game Pong in a step-by-step manner  The list is kind of handpicked for those who really want to step up their game in reinforcement learning. So if you are one among them, don’t forget to check out the resources.  I hope you enjoyed the article and you will take it forward to make applications that can adapt with respect to the environment they are employed to.  Thanks to Alessio and Bharath of FloydHub for sharing their valuable feedback on the article. Big thanks to the entire FloydHub team for letting me run the accompanying notebook on their platform. If you haven’t checked FloydHub yet, give FloydHub a spin for your Machine Learning and Deep Learning projects. There is no going back once you’ve learned how easy they make it.   FloydHub Call for AI writers  Want to write amazing articles like Sayak and play your role in the long road to Artificial General Intelligence? We are looking for passionate writers, to build the world's best blog for practical applications of groundbreaking A.I. techniques. FloydHub has a large reach within the AI community and with your help, we can inspire the next wave of AI. Apply now and join the crew!   About Sayak Paul Sayak loves everything deep learning. He goes by the motto of understanding complex things and helping people understand them as easily as possible. Sayak is an extensive blogger and all of his blogs can be found here. He is also working with his friends on the application of deep learning in Phonocardiogram classification. Sayak is also a FloydHub AI Writer. He is always open to discussing novel ideas and taking them forward to implementations. You can connect with Sayak on LinkedIn and Twitter.
An introduction to Q-Learning: Reinforcement Learning
https://blog.floydhub.com/an-introduction-to-q-learning-reinforcement-learning/