<p>You could also choose to make use of the R package "<a href="https://github.com/Nth-iteration-labs/contextual" rel="nofollow noreferrer">contextual</a>", which aims to ease the implementation and evaluation of both context-free (as described in Sutton &amp; Barto) and contextual (such as for <a href="https://github.com/Nth-iteration-labs/contextual/blob/master/R/policy_cmab_lin_ucb_disjoint.R" rel="nofollow noreferrer">example</a> <a href="https://arxiv.org/abs/1003.0146" rel="nofollow noreferrer">LinUCB</a>) Multi-Armed Bandit policies. </p>

<p>The package actually <a href="https://nth-iteration-labs.github.io/contextual/articles/sutton_barto.html" rel="nofollow noreferrer">offers a vignette</a> on how to replicate all Sutton &amp; Barto bandit plots. For example, to generate the ε-greedy plots, just simulate <a href="https://github.com/Nth-iteration-labs/contextual/blob/master/R/policy_mab_epsilon_greedy.R" rel="nofollow noreferrer">EpsilonGreedy</a> policies against a <a href="https://github.com/Nth-iteration-labs/contextual/blob/master/R/bandit_basic_gaussian.R" rel="nofollow noreferrer">Gaussian bandit</a> :</p>

<pre><code>library(contextual)

set.seed(2)
mus             &lt;- rnorm(10, 0, 1)
sigmas          &lt;- rep(1, 10)
bandit          &lt;- BasicGaussianBandit$new(mu_per_arm = mus, sigma_per_arm = sigmas)

agents          &lt;- list(Agent$new(EpsilonGreedyPolicy$new(0),    bandit, "e = 0, greedy"),
                        Agent$new(EpsilonGreedyPolicy$new(0.1),  bandit, "e = 0.1"),
                        Agent$new(EpsilonGreedyPolicy$new(0.01), bandit, "e = 0.01"))

simulator       &lt;- Simulator$new(agents = agents, horizon = 1000, simulations = 2000)
history         &lt;- simulator$run()

plot(history, type = "average", regret = FALSE, lwd = 1, legend_position = "bottomright")
plot(history, type = "optimal", lwd = 1, legend_position = "bottomright")
</code></pre>

<p><a href="https://i.stack.imgur.com/fAuaV.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/fAuaV.png" alt="EpsilonGreedy, average rewards"></a></p>

<p><a href="https://i.stack.imgur.com/huGHb.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/huGHb.png" alt="EpsilonGreedy, optimal arms"></a></p>

<p>Full disclosure: I am one of the developers of the package.</p>
n-armed bandit simulation in R
<p>I'm using Sutton &amp; Barto's ebook <em>Reinforcement Learning: An Introduction</em> to study reinforcement learning. I'm having some issues trying to emulate the results (plots) on the <a href="http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node16.html" rel="nofollow noreferrer">action-value page</a>.</p>

<p>More specifically, how can I simulate the <code>greedy</code> value for each task? The book says:</p>

<blockquote>
  <p>...we can plot the performance and behavior of various methods as
  they improve with experience over 1000 plays...</p>
</blockquote>

<p>So I guess I have to keep track of the <em>exploratory</em> values as better ones are found. The issue is how to do this using the <em>greedy</em> approach - since there are no exploratory moves, how do I know <em>what is a greedy behavior</em>?</p>

<p>Thanks for all the comments and answers!</p>

<p>UPDATE: See code on my answer.</p>