<p>Thanks to your help and the informations here <a href="https://medium.com/@leosimmons/double-dqn-implementation-to-solve-openai-gyms-cartpole-v-0-df554cd0614d" rel="nofollow noreferrer">leosimmons</a>, I found the source of my confusion:</p>

<p>The Bellman equation used here <a href="https://arxiv.org/pdf/1509.06461.pdf" rel="nofollow noreferrer">Bellman equation - link 3</a> follows the equation:</p>

<pre><code>value = reward + discount_factor * target_network.predict(next_state)[argmax(online_network.predict(next_state))]
</code></pre>

<p>The Bellman equation in the <strong>original</strong> (vanilla) DQN <a href="https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf" rel="nofollow noreferrer">Bellman equation - link 2</a> is:</p>

<pre><code>value = reward + discount_factor * max(target_network.predict(next_state))
</code></pre>

<p><a href="https://medium.com/@leosimmons/double-dqn-implementation-to-solve-openai-gyms-cartpole-v-0-df554cd0614d" rel="nofollow noreferrer">leosimmons</a> </p>

<blockquote>
  <p>The difference is that, using the terminology of the field, the second
  equation uses the target network for both SELECTING and EVALUATING the
  action to take whereas the first equation uses the online network for
  SELECTING the action to take and the target network for EVALUATING the
  action. Selection here means choosing which action to take, and
  evaluation means getting the projected Q value for that action. This
  form of the Bellman equation is what makes this agent a Double DQN and
  not just a DQN and was introduced in <a href="https://arxiv.org/pdf/1509.06461.pdf" rel="nofollow noreferrer">3</a>.</p>
</blockquote>

<p><a href="https://medium.com/@leosimmons/double-dqn-implementation-to-solve-openai-gyms-cartpole-v-0-df554cd0614d" rel="nofollow noreferrer">1</a> <a href="https://medium.com/@leosimmons/double-dqn-implementation-to-solve-openai-gyms-cartpole-v-0-df554cd0614d" rel="nofollow noreferrer">https://medium.com/@leosimmons/double-dqn-implementation-to-solve-openai-gyms-cartpole-v-0-df554cd0614d</a></p>

<p><a href="https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf" rel="nofollow noreferrer">2</a> <a href="https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf" rel="nofollow noreferrer">https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf</a></p>

<p><a href="https://arxiv.org/pdf/1509.06461.pdf" rel="nofollow noreferrer">3</a> <a href="https://arxiv.org/pdf/1509.06461.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1509.06461.pdf</a></p>

<p>Very well explained here:
<a href="https://youtu.be/ILDLT97FsNM?t=331" rel="nofollow noreferrer">https://youtu.be/ILDLT97FsNM?t=331</a></p>
How to get Q Values in RL - DDQN
<p>I'm not sure how to get the Q Values for a DDQN. </p>

<p>DQN is the normal network, TAR the target network.</p>

<pre><code>    q_values = self.DQN.predict(c_states) # DQN batch predict Q on states
    dqn_next = self.DQN.predict(n_states) # DQN batch predict Q on next_states
    tar_next = self.TAR.predict(n_states) # TAR batch predict Q on next_states
</code></pre>

<p>I mainly found 2 versions:</p>

<p><strong>Version 1:</strong></p>

<pre><code>q_values[i][actions[i]] = (rewards[i] + (GAMMA * np.amax(tar_next[i])))
</code></pre>

<p><strong>Version 2:</strong></p>

<pre><code>act = np.argmax(dqn_next[i])
q_values[i][actions[i]] = (rewards[i] + (GAMMA * tar_next[i][act]))
</code></pre>

<p>Which one is correct? And why?</p>

<p><strong>Version 1 Links:</strong></p>

<p><a href="https://github.com/keon/deep-q-learning/blob/master/ddqn.py" rel="nofollow noreferrer">https://github.com/keon/deep-q-learning/blob/master/ddqn.py</a></p>

<p><a href="https://pythonprogramming.net/training-deep-q-learning-dqn-reinforcement-learning-python-tutorial" rel="nofollow noreferrer">https://pythonprogramming.net/training-deep-q-learning-dqn-reinforcement-learning-python-tutorial</a></p>

<p><strong>Version 2 Links:</strong></p>

<p><a href="https://github.com/germain-hug/Deep-RL-Keras/blob/master/DDQN/ddqn.py" rel="nofollow noreferrer">https://github.com/germain-hug/Deep-RL-Keras/blob/master/DDQN/ddqn.py</a></p>

<p><a href="https://github.com/rlcode/reinforcement-learning/blob/master/2-cartpole/2-double-dqn/cartpole_ddqn.py" rel="nofollow noreferrer">https://github.com/rlcode/reinforcement-learning/blob/master/2-cartpole/2-double-dqn/cartpole_ddqn.py</a></p>

<p><a href="https://jaromiru.com/2016/11/07/lets-make-a-dqn-double-learning-and-prioritized-experience-replay/" rel="nofollow noreferrer">https://jaromiru.com/2016/11/07/lets-make-a-dqn-double-learning-and-prioritized-experience-replay/</a></p>

<hr>

<p><strong>EDIT:</strong>
Many thanks, to clarify this</p>

<pre><code>Q-learning: 
q_values[i][actions[i]] = (rewards[i] + (GAMMA * np.amax(tar_next[i])))

SARSA: 
act = np.argmax(dqn_next[i])
q_values[i][actions[i]] = (rewards[i] + (GAMMA * tar_next[i][act]))
</code></pre>

<p><strong>EDIT: re-open 03/2020</strong></p>

<p>I'm sorry but i have to re-open that question. Maybe I misunderstood something, but the following sources show that my Version 2 (SARSA) is Double Q Learning?</p>

<p><strong>Page 158 : Double Q-learning</strong>
<a href="http://incompleteideas.net/book/RLbook2018.pdf" rel="nofollow noreferrer">http://incompleteideas.net/book/RLbook2018.pdf</a></p>

<p><a href="https://adventuresinmachinelearning.com/double-q-reinforcement-learning-in-tensorflow-2/" rel="nofollow noreferrer">adventuresinML</a></p>

<p><a href="https://github.com/adventuresinML/adventures-in-ml-code/blob/e661eeb5db86d2d0aa21621b68b5186d80e3d8b6/double_q_tensorflow2.py#L86" rel="nofollow noreferrer">adventuresinML source</a></p>