<p>This is Q-learning (the version with the max operator) vs SARSA (without the max). </p>

<p>In short, you collect samples using the e-greedy policy: this is your behavior (or exploration) policy. The policy you want to learn is called "target" and can be different.<br>
With Q-learning, you use the max operator, so your target is chosen according to the greedy (target) policy. This is called off-policy learning, because you learn a policy (target) with the samples collected by a different one (behavior).
<br>
With SARSA, there is no max, so in practice you just use the action from the samples, that was selected by the behavior policy. This is on-policy, because the target and the behavior are the same.</p>

<p>Which one to prefer is up to you, but I think that Q-learning is more common (and DQN uses Q-learning).</p>

<p>More reading about this</p>

<p><a href="https://stackoverflow.com/questions/6848828/what-is-the-difference-between-q-learning-and-sarsa">What is the difference between Q-learning and SARSA?</a></p>

<p><a href="https://stackoverflow.com/questions/32846262/are-q-learning-and-sarsa-with-greedy-selection-equivalent">Are Q-learning and SARSA with greedy selection equivalent?</a></p>

<p><a href="https://stats.stackexchange.com/questions/184657/what-is-the-difference-between-off-policy-and-on-policy-learning">https://stats.stackexchange.com/questions/184657/what-is-the-difference-between-off-policy-and-on-policy-learning</a></p>

<p><a href="http://incompleteideas.net/book/RLbook2018.pdf" rel="nofollow noreferrer">http://incompleteideas.net/book/RLbook2018.pdf</a></p>

<p><strong>EDIT FOR DDQN</strong></p>

<p>SARSA and Q-learning are two separate algorithms.
<br>
In DDQN you have two target Q, and two target policies, so the algorithm is still off-policy (sampling policy is e-greedy, target policies are greedy), while SARSA is on-policy (target policy = sampling policy).
<br>
The trick in DDQN is that you use the max operator over Q2 (second critic) in the TD target for updating Q1 (first critic), and viceversa. <strong>But there still is the max, so it's still off-policy. SARSA, instead, is on-policy.</strong></p>

<p>There are multiple versions of DDQN, some use the mininum over Q1 and Q2 for instance. Here are some references</p>

<p><a href="https://arxiv.org/pdf/1509.06461.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1509.06461.pdf</a></p>

<p><a href="https://arxiv.org/pdf/1802.09477.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1802.09477.pdf</a></p>
How to get Q Values in RL - DDQN
<p>I'm not sure how to get the Q Values for a DDQN. </p>

<p>DQN is the normal network, TAR the target network.</p>

<pre><code>    q_values = self.DQN.predict(c_states) # DQN batch predict Q on states
    dqn_next = self.DQN.predict(n_states) # DQN batch predict Q on next_states
    tar_next = self.TAR.predict(n_states) # TAR batch predict Q on next_states
</code></pre>

<p>I mainly found 2 versions:</p>

<p><strong>Version 1:</strong></p>

<pre><code>q_values[i][actions[i]] = (rewards[i] + (GAMMA * np.amax(tar_next[i])))
</code></pre>

<p><strong>Version 2:</strong></p>

<pre><code>act = np.argmax(dqn_next[i])
q_values[i][actions[i]] = (rewards[i] + (GAMMA * tar_next[i][act]))
</code></pre>

<p>Which one is correct? And why?</p>

<p><strong>Version 1 Links:</strong></p>

<p><a href="https://github.com/keon/deep-q-learning/blob/master/ddqn.py" rel="nofollow noreferrer">https://github.com/keon/deep-q-learning/blob/master/ddqn.py</a></p>

<p><a href="https://pythonprogramming.net/training-deep-q-learning-dqn-reinforcement-learning-python-tutorial" rel="nofollow noreferrer">https://pythonprogramming.net/training-deep-q-learning-dqn-reinforcement-learning-python-tutorial</a></p>

<p><strong>Version 2 Links:</strong></p>

<p><a href="https://github.com/germain-hug/Deep-RL-Keras/blob/master/DDQN/ddqn.py" rel="nofollow noreferrer">https://github.com/germain-hug/Deep-RL-Keras/blob/master/DDQN/ddqn.py</a></p>

<p><a href="https://github.com/rlcode/reinforcement-learning/blob/master/2-cartpole/2-double-dqn/cartpole_ddqn.py" rel="nofollow noreferrer">https://github.com/rlcode/reinforcement-learning/blob/master/2-cartpole/2-double-dqn/cartpole_ddqn.py</a></p>

<p><a href="https://jaromiru.com/2016/11/07/lets-make-a-dqn-double-learning-and-prioritized-experience-replay/" rel="nofollow noreferrer">https://jaromiru.com/2016/11/07/lets-make-a-dqn-double-learning-and-prioritized-experience-replay/</a></p>

<hr>

<p><strong>EDIT:</strong>
Many thanks, to clarify this</p>

<pre><code>Q-learning: 
q_values[i][actions[i]] = (rewards[i] + (GAMMA * np.amax(tar_next[i])))

SARSA: 
act = np.argmax(dqn_next[i])
q_values[i][actions[i]] = (rewards[i] + (GAMMA * tar_next[i][act]))
</code></pre>

<p><strong>EDIT: re-open 03/2020</strong></p>

<p>I'm sorry but i have to re-open that question. Maybe I misunderstood something, but the following sources show that my Version 2 (SARSA) is Double Q Learning?</p>

<p><strong>Page 158 : Double Q-learning</strong>
<a href="http://incompleteideas.net/book/RLbook2018.pdf" rel="nofollow noreferrer">http://incompleteideas.net/book/RLbook2018.pdf</a></p>

<p><a href="https://adventuresinmachinelearning.com/double-q-reinforcement-learning-in-tensorflow-2/" rel="nofollow noreferrer">adventuresinML</a></p>

<p><a href="https://github.com/adventuresinML/adventures-in-ml-code/blob/e661eeb5db86d2d0aa21621b68b5186d80e3d8b6/double_q_tensorflow2.py#L86" rel="nofollow noreferrer">adventuresinML source</a></p>