<p>You should estimate <strong>Q(s,a)</strong> using the <strong>critic</strong>, not the <strong>actor</strong>.</p>

<p>Remember that in the <strong>actor-critic</strong> setting (e.g. A2C), the actor(s) will <strong>output</strong> the probability distribution over all your actions at state <code>s</code>. From this distribution, you'll sample an action <code>a</code> to take in the environment. Then, the environment will give you a reward <code>r</code> and the next state <code>s'</code>. </p>

<p>After <code>N</code> steps, you'll use the <strong>critic</strong> to estimate the <strong>state value</strong> <code>V(s)</code> and will calculate the <strong>advantage</strong> to point out how much better your actions were than the average for example. With the advantage, you'll update your policy (<strong>actor</strong>) to increase/decrease the probability of taking the action <code>a</code> at state <code>s</code>.</p>

<p>Therefore, to use your <strong>advantage function</strong> in this framework, you could use the critic to estimate <code>Q(s,a)</code>, which is the value for each pair of action-state. Then, you can estimate <code>V(s)</code> with:</p>

<p><a href="https://i.stack.imgur.com/ru0vc.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ru0vc.png" alt="enter image description here"></a></p>

<p>You can take a look at this <a href="https://datascience.stackexchange.com/a/31791">answer</a> and at this <a href="https://danieltakeshi.github.io/2017/04/02/notes-on-the-generalized-advantage-estimation-paper/" rel="nofollow noreferrer">post</a> to have a better idea. Note that, to estimate <code>Q(s,a)</code> your critic network should have <code>|A|</code> output units, instead of just one as in the case of <code>V(s)</code>. There are also other options to try as your advantage function.</p>

<p>Remember that the only purpose of the advantage function is to tell your model how much to increase/decrease the probabilities of taking action <code>a</code> at state <code>s</code>. If it's better than average you increase otherwise you decrease.</p>

<p>This <a href="https://arxiv.org/pdf/1506.02438.pdf" rel="nofollow noreferrer">paper</a> is a very good reference.</p>
Could anyone explain clearly how to compute the advantage function in reinforcement learning?
<p>I have troubles understanding how to get data to compute the advantage in actor-critic settings.</p>

<p>I know that <code>A(s,a) = Q(s,a) - V(s)</code>. It seems straightforward to get the state value estimate <code>V(s)</code>, but how can we estimate <code>Q(s,a)</code> given that the policy only outputs probabilities?  </p>

<p>Thanks!   </p>