title: Could anyone explain clearly how to compute the advantage function in reinforcement learning?
body: <p>I have troubles understanding how to get data to compute the advantage in actor-critic settings.</p>
  
  <p>I know that <code>A(s,a) = Q(s,a) - V(s)</code>. It seems straightforward to get the state value estimate <code>V(s)</code>, but how can we estimate <code>Q(s,a)</code> given that the policy only outputs probabilities?  </p>
  
  <p>Thanks!   </p>