<p>Too large changes to the current policy is the main cause of instability of A3C algorithm. There are methods to stabilize it, e.g. <a href="https://arxiv.org/abs/1502.05477" rel="nofollow noreferrer">TRPO</a> or <a href="https://arxiv.org/abs/1707.06347" rel="nofollow noreferrer">PPO</a>. I'd suggest you to look at PPO - it is very easy to implement, although less effective.</p>

<p>In PPO, you would simply change the loss function (based on the blog's notation) to:</p>

<p><a href="https://i.stack.imgur.com/6Wetc.gif" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6Wetc.gif" alt="math formula"></a></p>

<p>Where it is recommended to use <code>e=0.2</code>. Good luck with your implementation!</p>
Convergence issues in a3c
<p>I've built an A3C implementation in keras using this as referance: <a href="https://jaromiru.com/2017/03/26/lets-make-an-a3c-implementation/" rel="nofollow noreferrer">https://jaromiru.com/2017/03/26/lets-make-an-a3c-implementation/</a>
And I'm using custom environment, where an agent has a choise of purchasing some items, selling or exchanging them given their price as state. And it is given positive rewards for good deals and negative rewards for bad deals. I have tested it on DQN in the past and it sucessfully converged showing really good results. But when I use the same environment in A3C, it results in model just choosing the same action over and over. I tried changing some hyper-parametrs, but no result. I also tried using target model and updating it every n episodes, which resulted in better convergence with gym CartPole environment, but still no effect on performance of my model in my custom environment. I have found a few discussions on reddit about the same problem, but none of them were answered. </p>