approaches trying to alleviate this problem, for example, similar to the DDQN approach, a form of correction has been applied in continuous learning settings [@fujimoto_addressing_2018]. This allows to push the border further and deal with some more variability of behavior. But as soon as the environment is allowed to change, such approaches run into trouble.

The fundamental problem of a dynamically changing environment is not addressed at its core. Going back to the simple maze problem: consider a change of the maze, like adding a wall that closes the shortest path in a maze or adding a novel high reward at a different location. For reinforcement learning, all the paths that cross through that passage would be destroyed and all values that depend on paths (in q-learning through the max operation) through this passage would be misleading. Closing the passage “revivesÕÕ the relevance of experiences from alternative paths whose values — in the presence of the passage — were dominated by the better values of paths through the passage. However, standard formulations of RL have no memory for such suboptimal paths; their contributions thus become overwritten as soon as a better alternative is detected. They thus cannot become “revivedÕÕ when 