<p>In the original paper, the author pushes one sample to the experience replay buffer and randomly samples 32 transitions to train the model in the minibatch fashion. The samples took from interacting with the environment is not directly feeding to the model. To increase the speed of training, the author store samples every step but updates the model every four steps. </p>

<p>Use OpenAI's <a href="https://github.com/openai/baselines" rel="nofollow noreferrer">Baseline project</a>; this single-process method can master easy games like Atari Pong (Pong-v4) about 2.5 hours using a single GPU. Of course, training in this kind of single process way makes multi-core, multi-GPU (or single-GPU) system's resource underutilised. So in new publications had decoupled action-selection and model optimisation. They use multiple "Actors" to interact with environments simultaneously and a single GPU "Leaner" to optimise the model or multiple Leaners with multiple models on various GPUs. The multi-actor-single-learner is described in Deepmind's Apex-DQN (<a href="https://arxiv.org/abs/1803.00933" rel="nofollow noreferrer">Distributed Prioritized Experience Replay, D. Horgan et al., 2018</a>) method and the multi-actor-multi-learner described in (<a href="https://arxiv.org/abs/1803.00933" rel="nofollow noreferrer">Accelerated Methods for Deep Reinforcement Learning, Stooke and Abbeel, 2018</a>). When using multiple learners, the parameter sharing across processes becomes essential. The old trail described in Deepmind's PDQN (<a href="https://arxiv.org/abs/1507.04296" rel="nofollow noreferrer">Massively Parallel Methods for Deep Reinforcement Learning, Nair et al., 2015</a>) which was proposed in the period between DQN and A3C. However, the work was performed entirely on CPUs, so it looks using massive resources, the result can be easy outperformed by PPAC's batched action-selection on GPU method.  </p>

<p>You can't optimise on each episode end, because the episode length isn't fixed, the better the model usually results in the longer episode steps. The model's learning capability will decrease when they perform a little better. The learning progress will be instable.  </p>

<p>We also don't train the model only on target model clone, because the introduction of the target is to stabilise the training process by keeping an older set of parameters. If you update only on parameter clones, the target model's parameters will be the same as the model and this cause instability. Because, if we use the same parameters, one model update will cause the next state to have a higher value. </p>

<p>In Deepmind's 2015 Nature paper, it states that:</p>

<blockquote>
  <p>The second modification to online Q-learning aimed at further improving the stability of our method with neural networks is to use a separate network for generating the target yj in the Q-learning update. More precisely, every C updates we clone the network Q to obtain a target network Q' and use Q' for generating the Q-learning targets y<sub>j</sub> for the following C updates to Q. 
  This modification makes the algorithm more stable compared to standard online Q-learning, where an update that increases Q(s<sub>t</sub>,a<sub>t</sub>) often also increases Q(s<sub>t+1</sub>, a) for all a and hence also increases the target y<sub>j</sub>, possibly leading to oscillations or divergence of the policy. Generating the targets using the older set of parameters adds a delay between the time an update to Q is made and the time the update affects the targets y<sub>j</sub>, making divergence or oscillations much more unlikely.  </p>
</blockquote>

<p><a href="https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf" rel="nofollow noreferrer">Human-level control through deep reinforcement
learning, Mnih et al., 2015</a> </p>
Fitting step in Deep Q Network
<p>I am confused why dqn with experience replay algorithm would perform gradient descent step for every step in a given episode? This will fit only one step, right? This would make it extremely slow. Why not after each episode ends or every time the model is cloned?</p>