<p>You can do it by sharing some (or all) layers of their network. If you do, however, you are assuming that there is a common state representation (the intermediate layer output) that is optimal w.r.t. both. This is a very strong assumption and it usually doesn't hold. It has been shown to work for learning from image, where you put (for instance) an autoencoder on the top both the actor and the critic network and train it using the sum of their loss function. </p>

<p>This is mentioned in <a href="https://arxiv.org/pdf/1707.06347.pdf" rel="noreferrer">PPO paper</a> (just before Eq. (9)). However, they just say that they share layers only for learning Atari games, not for continuous control problems. They don't say why, but this can be explained as I said above: Atari games have a low-dimensional state representation that is optimal for both the actor and the critic (e.g., the encoded image learned by an autoencoder), while for continuous control you usually pass directly a low-dimensional state (coordinates, velocities, ...).</p>

<p>A3C, which you mentioned, was also used mostly for games (Doom, I think).</p>

<p>From my experience, in control sharing layers never worked if the state is already compact.</p>
How do shared parameters in actor-critic models work?
<p>Hello StackOverflow Community!</p>

<p>I have a question about Actor-Critic Models in Reinforcement Learning.</p>

<p>While listening policy gradient methods classes of Berkeley University, it is said in the lecture that in the actor-critic algorithms where we both optimize our policy with some policy parameters and our value functions with some value function parameters, we use same parameters in both optimization problems(i.e. policy parameters = value function parameters) in some algorithms (e.g. A2C/A3C)</p>

<p>I could not understand how this works. I was thinking that we should optimize them separately. How does this shared parameter solution helps us?</p>

<p>Thanks in advance :)</p>