Trying to capture the variability becomes increasingly difficult through simple exploration of inherently variable responses from the environment. This becomes even more of a problem when there are other active agents present which might learn to react towards our actions or simply vary how to act and what goals to follow.

In the following, we want to explain why considering a permanently dynamically changing environment (its response characteristic) is problematic for the employed conceptualization as a simple Markov Decision Problem (or a semi-MDP in the case of a hierarchical model) and why therefore following such a route might not scale as well as currently hoped for.

One important insight for DRL approaches was the idea of how to use past experience to train a value function using TD-learning. Such value functions are supposed to provide an estimate of a next state (or a chosen action), but this value is always necessarily conditioned on an assumption how one would proceed from there on. Whenever one starts to change the estimator for a value function which is exploited for subsequent action selection, this might interfere with this estimated assumption of what would follow. This problem of off-policy learning has shown to be 