<p>These kind of problems happen pretty often and you shouldn't give up. First, of course, you should do another one or two checks if the code is all right - try to compare your code to other implementations, see how the loss function behave etc. If you are pretty sure your code is all fine - and, as you say that model can learn the task from time to time, it probably is - you should start experimenting with the hyper-parameters.</p>

<p>Your problems seem to be connected to hyper-parameters like exploration technique, learning rate, the way you are updating the target networks and to the experience replay memory. I would not play around with the hidden layer sizes - find the values for which the model learned once and keep them fixed. </p>

<ul>
<li><strong>Exploration technique:</strong> I assume you use epsilon-greedy strategy. My advice would be to start with a high epsilon value (I usually start with 1.0) and decay it after each step or episode, but define an epsilon_min too. Starting with a low epsilon value may be the problem of different learning speeds and success rates - if you go full random, you always populate your memory with similar kind of transitions at the beginning. With lower epsilon rates at the start, there is a bigger chance for your model to not explore enough before the exploitation phase begins. </li>
<li><strong>Learning rate:</strong> Make sure it is not too big. Smaller rate may lower the learning speed, but helps a learned model to not escape back from global minima to some local, worse ones. Also, adaptive learning rates such as these calculated with <a href="https://arxiv.org/abs/1412.6980" rel="noreferrer">Adam</a> might help you. Of course the batch size have an impact as well, but I would keep it fixed and worry about it only if the other hyper-parameter changes won't work. </li>
<li><strong>Target network update (rate and value):</strong> This is an important one as well. You have to experiment a bit - not only how often do you perform the update, but also how much of the primary values you copy into the target ones. People often do a hard update each episode or so, but try doing soft updates instead if the first technique does not work. </li>
<li><strong>Experience replay</strong>: Do you use it? You should. How big is your memory size? This is very important factor and the memory size can influence the stability and success rate (<a href="https://arxiv.org/abs/1712.01275" rel="noreferrer">A Deeper Look at Experience Replay</a>). Basically, if you notice instability of your algorithm, try a bigger memory size, and if it affects your learning curve a lot, try out the technique proposed in the mentioned paper.</li>
</ul>
Why is my Deep Q Net and Double Deep Q Net unstable?
<p>I am trying to implement DQN and DDQN(both with experience reply) to solve OpenAI AI-Gym Cartpole Environment. Both of the approaches are able to learn and solve this problem sometimes, but not always.</p>

<p>My network is simply a feed forward network(I've tried using 1 and 2 hidden layers). In DDQN I created one network in DQN, and two networks in DDQN, a target network to evaluate the Q value and a primary network to choose the best action, train the primary network, and copy it to target network after some episodes.</p>

<p>The problem in DQN is: </p>

<ul>
<li>Sometimes it can achieve the perfect 200 score within 100 episodes, but sometimes it gets stuck and only achieves 10 score no matter how long it is trained. </li>
<li>Also, in case of successful learning, the learning speed differ.</li>
</ul>

<p>The problem in DDQN is:</p>

<ul>
<li>It can learn to achieve 200 score, but then it seems to forget what's learned and the score drops dramatically.</li>
</ul>

<p>I've tried tuning batch size, learning rate, number of neurons in the hidden layer, the number of hidden layers, exploration rate, but instability persists. </p>

<p>Are there any rule of thumb on the size of network and batch size? I think reasonably larger network and larger batch size will increase stability. </p>

<p>Is it possible to make the learning stable? Any comments or references are appreciated!</p>