We introduce CBF, an exploration method that works in the absence of rewards or end of episode signal. CBF is based on intrinsic reward derived from the error of a dynamics model operating in feature space. It was inspired by (Pathak et al., 2017), is easy to implement, and can achieve results such as passing four levels of Super Mario Bros, navigating VizDoom mazes and passing two levels of SpaceInvaders. We investigated the effect of combining the method with several auxiliary tasks, but find inconsistent improvements over the CBF baseline.
 Modern reinforcement learning methods work well for tasks with dense reward functions, but in many environments of interest the reward function may be sparse, require considerable human effort to specify, be misspecified, or be prohibitively costly to evaluate. In general it is much easier to find environments that we could train an agent to act in than it is to find sensible reward functions to train it with. It is therefore desirable to find ways to learn interesting behaviors from environments without specified reward functions. BID19 introduced an exploration strategy that leads to sophisticated behavior in several games in the absence of any extrinsic reward. The strategy involves 1. Learning features using an inverse dynamics prediction task, 2. training a forward dynamics model in the feature space, and 3. using the error of the forward model as an intrinsic reward for an exploration agent.Inspired by this result we wondered if it was possible to improve the method by using a different task for learning the features in step 1. To our surprise we found that the choice of feature-learning task didn't matter much. In fact when skipping step 1 altogether, we often obtained comparable or better results. As a result we obtained a method that is simple to implement, and shows purposeful behavior on a range of games, including passing over four levels of Super Mario Bros without any extrinsic rewards or end of episode signal (see video here). Previous work reported making significant progress on the first level of this game. In addition we report the results of using our method on VizDoom maze environment, and a range of Atari games. Our experiments have shown that any of the joint training methods can work well for exploration. The fact that a method as simple as CBF performs so well, however, suggests that the success of the method of BID19 comes to a great extent from a feature-bootstrapping effect, and the utility of an auxiliary task, if any, is to stabilize this process.Some immediate future research directions include trying CBF on environments with continuous action spaces, and investigating feature-bootstrapping for count-based exploration methods such as BID17 . We would also like to research exploration of environments with greater amounts of stochasticity. 9 APPENDIX: EXPERIMENTAL DETAILS PREPROCESSING We followed the standard preprocessing for Atari games (see wrappers used in the DQN implementation in BID10 ) for all of our experiments, except for not using the automatic "press fire" in the beginning of the episode wrapper. For Mario and VizDoom we downscaled the observations to 84 by 84 pixels, converted them to grayscale, stacked sequences of four frames as four channels of the observation, and used a frame skip of four. We also used an action wrapper replicating the action space used in BID19 .
A simple intrinsic motivation method using forward dynamics model error in feature space of the policy.