<p>The part that is confusing you is the <code>Bellman approximation</code> which is used to update the <code>Q-values</code> of a state that is defined as <code>s</code> given an action <code>a</code> is taken.</p>
<p><a href="https://i.stack.imgur.com/SqF75.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/SqF75.png" alt="enter image description here" /></a></p>
<p><code>Q</code> for this state, <code>s</code>, and action, <code>a</code>, equals the expected immediate reward and the discounted long-term reward of the destination state.</p>
<p>We take the maximum of the values of the <code>Q-values(or the value of the action)</code> of being at state <code>s'</code> which is the next state going from state <code>s</code>, with an action <code>a'</code>, as the actions we can take when going from a state <code>s</code> to a state <code>s'</code> are a set of mutually exclusive discrete set (i.e., your environment allows you to move in the direction up, left, right or down) and the most optimal action would therefore be the action which results in the highest value of the action.</p>
<p><a href="https://i.stack.imgur.com/Oj5rX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Oj5rX.png" alt="enter image description here" /></a></p>
<p>Take the image above as an example. The agent starts at state, <code>s0</code> and is able to move up, left, right, or down which are the actions. The actions that the agent can take are stochastic in nature and not deterministic but it i.e., when the agent intends to go up there is a <code>0.33%</code> chance that the agent might instead go to the left or the right. I will just assign a value of 1 to gamma here.</p>
<p>This is how you calculate the <code>Q-values</code> for the state, <code>s0</code> and action, <code>up</code> with the values of going to the state being the immediate reward received by the agent, <code>V1 = 1, V2 = 2, V3 = 3, V4 = 4</code>.</p>
<pre><code>Q(s0,up) = 0.33 * V1 + 0.33 * V2  0.33 * V4  
         = 0.33 * 1 + 0.33 * 2 + 0.33 * 4 
         = 2.31
</code></pre>
<p>Next, if you calculate Q-values for all the other possible states and their actions you would get the following:</p>
<pre><code>Q(s0,left) = 1.98
Q(s0,right) = 2.64
Q(s0,down) = 2.97
</code></pre>
<p>Therefore the final value for the state, <code>s0</code> is the <code>maximum</code> of those actions' values which is <code>2.97</code>. That is all you really trying to do there in the code.</p>
<p>As for what does <code>currentQ[action] = newQ;</code> do, it is performing an update on the current <code>Q-values</code> for an action from its old value to the new updated value at the end of an episode.</p>
<p>One thing you have to understand as to why does it do this, is that the agent updates its <code>Q-values</code> after an episode, then training is done once again and the process is repeated up until the agent manages to complete its goal(for the Atari paper that this algorithm was introduced from, that goal was having a mean score of I think 19 which is equivalent to winning 19 games out of 21 games).</p>
<p>You can read more about the entire process from the <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwjRqeDI55DuAhVn4jgGHWo0DDAQFjADegQIAxAC&amp;url=https%3A%2F%2Farxiv.org%2Fabs%2F1312.5602&amp;usg=AOvVaw1wUW9fyPY7pUHTVhWXfO4h" rel="nofollow noreferrer">original paper</a>.</p>
<p>But I think you need more of an understanding of Bellmans equation before that as it is extremely important in understanding Reinforcement Learning. DeepMind has an excellent Youtube series about this that can be <a href="https://www.youtube.com/watch?v=hMbxmRyDw5M" rel="nofollow noreferrer">found here</a>.</p>
<p>Even better there is a <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;ved=2ahUKEwiq2oD_6ZDuAhWWgtgFHSXeA88QFjABegQIARAC&amp;url=https%3A%2F%2Fwww.andrew.cmu.edu%2Fcourse%2F10-703%2Ftextbook%2FBartoSutton.pdf&amp;usg=AOvVaw0AIbsm2D6IhSbHdz9RPK2i" rel="nofollow noreferrer">free book</a> on Reinforcement Learning from the founding fathers of it, Richard Sutton and Andrew Barto. I believe they go in detail about this in Chapter 4.</p>
<p>Edit:</p>
<p>I am not too sure what you mean by how it affects training but I will outline the entire process for you to understand how training works for this:</p>
<p><a href="https://i.stack.imgur.com/itOaQ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/itOaQ.png" alt="enter image description here" /></a></p>
How does work this implementation of DQN algorithm on TensorFlowJs?
<p>devs,</p>
<p>I found a bunch of examples of DQN implementations, but because I'm no TensorFlow expert, I'm a little bit confused.</p>
<p>Let's see <a href="https://dumpz.org/c77HNAA4XxGF" rel="nofollow noreferrer">here</a> is one one of them.</p>
<p>I can understand, on the 73rd line, we slice some batch of stored data <code>[{state, action, reward, newState, done}]</code> exactly, then we get <code>currentStates</code> which is <code>[[s1, s2, ...]]</code>, then on 75 we use the model to get <code>currentQs</code> which should be, how I understand, <code>[[act1, act2, ...]]</code>, because our model is used to get action from env's state. The same happens to <code>newCurrentStates</code> and <code>futureQs</code>.</p>
<p>But then on 88, we see <code>let maxFutureQ = Math.max(futureQs);</code>. What happened here? <code>futureQs</code> is an array of arrays with actions probabilities for each futureState? And then <code>maxFutureQ</code> should be an action probability, why then we add this to reward? This part is confusing me.</p>
<p>Also I cannot understand why we need to do <code>currentQ[action] = newQ;</code> on 94.</p>
<p>Please, could someone help me to understand what is going on here and leave comments for lines, maybe?</p>
<p>Thanks in advance.</p>
<p>edit:</p>
<p>discussed code:
<a href="https://i.stack.imgur.com/yfgZu.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/yfgZu.png" alt="discussed code" /></a></p>