a serious problem for DRL and often leads to instabilities that cause complete breakdowns of such learning approaches. In particular, overestimation of values has shown to draw learned control approaches towards such over-estimated actions and often learning never recovers from such bad selections. Different solutions have been proposed, for example using separate and more conservative estimates for action selection [@hasselt_deep_2016]. One alternative would be to always require gathering new experience which, unfortunately, would be very costly and unfeasible for many problems. Even worse, and here we turn towards the distinction of different types of problems: in many real world settings, agents not only have to deal with a novel, given situation. But part of it are other agents that are learning themselves. This means that in these cases the response of the environment might change by itself and independently of the changes in the learning agent itself. Therefore, the agentÕs current policy is already rendered in a way off-policy as he is trying to optimize towards responses of the environment which have changed themselves. From the experience in DRL it is known that this destabilizes learning an estimator and that this makes it difficult to handle or even intractable. Current 