algorithms. Our experiments show that the exploration strategy achieves improved learning performance on the majority of Atari games.

Hyperparameters {#app:param}
===============

We tabulate the hyperparameters in our well-tuned implementation of double DQN in Table \[table:param\]:

[p[3cm]{}lp[4.8cm]{}]{} hyperparameter & value & descriptions\
total training frames & $40$ million& Length of training for each game.\
\
minibatch size & 32 & Size of minibatch samples for each parameter update.\
\
replay buffer size & 1000000 & The number of most recent frames stored in replay buffer.\
\
agent history length & 4 & The number of most recent frames concatenated as input to the Q network. Total number of iterations = total training frames / agent history length.\
\
target network update frequency & 10000 & The frequency of updating target network, in the number of parameter updates.\
\
discount factor & 0.99 & Discount factor for Q value.\
\
action repeat & 4 & Repeat each action selected by the agent this many times. A value of 4 means the agent sees every 4th frame.\
\
update frequency & 4 & The number of actions between successive parameter updates.\
\
optimizer & Adam & Optimizer for parameter updates.\
\
$\beta_1$ & 