<p>PPO has a built-in mechanism(surrogate clipping objective function) to prevent large gradient updates &amp; generally outperforms A3C on most continuous control environments. </p>

<p>In order for PPO to enjoy the benefits of parallel computing like A3C, Distributed PPO(DPPO) is the way to go.</p>

<p>Check out the links below to find out more information about DPPO.</p>

<p><a href="https://i.stack.imgur.com/AYjQN.png" rel="noreferrer">Pseudo code from the original DeepMind paper</a></p>

<p><a href="https://arxiv.org/pdf/1707.02286.pdf" rel="noreferrer">Original DeepMind paper: Emergence of Locomotion Behaviours in Rich Environments</a></p>

<p>If you plan to implement your DPPO code in Python with Tensorflow, I will suggest you to try <a href="https://ray.readthedocs.io/en/latest/index.html#" rel="noreferrer">Ray</a> for the part on distributed execution.</p>
What are the similarities between A3C and PPO in reinforcement learning policy gradient methods?
<p>Is there any easy way to merge properties of PPO with an A3C method? A3C methods run a number of parrel actors and optimize the parameters. I am trying to merge PPO with A3C.</p>