the goal, decisions for a current state can be based on neighboring stepping stones that are associated towards objective values. Approaches following such an idea have now been applied in DRL as well. For example, in one hierarchical DRL approach [@kulkarni_hierarchical_2016] a similar representation of a positional space in a computer game has been employed as an intermediate level in a hierarchy. Here, the learning agent explored new behaviors for their own sake rather than for immediately solving problems. This approach, reminiscent of “intrinsic motivation” in biological systems, later helped the agent to deal with situations when feedback from the environment was either too sparse for a standard reinforcement approach to benefit from it, or when environmental feedback was delayed in time.

Representations for Stepping Stones — A Behavior Space
======================================================

The above examples have demonstrated in an exemplary fashion that the search for “stepping stones” — either through hierarchical RL or EAs — can be an efficient strategy for coping with situations in which standard (deep) reinforcement learning approaches fail. Therefore, the question arises if and how such intermediate representations can be found in general and how they would be structured? Here we start by providing again an example 