    Reinforcement learning (RL) has led to increasingly complex looking behavior in recent years. However, such complexity can be misleading and hides over-fitting. We find that visual representations may be a useful metric of complexity, and both correlates well objective optimization and causally effects reward optimization. We then propose curious representation learning (CRL) which allows us to use better visual representation learning algorithms to correspondingly increase visual representation in policy through an intrinsic objective on both simulated environments and transfer to real images. Finally, we show better visual representations induced by CRL allows us to obtain better performance on Atari without any reward than other curiosity objectives. In recent years, reinforcement learning(RL) has lead to increasingly complex behavior from simulated environments (Silver et al., 2016; OpenAI, 2018; Mnih et al., 2013; Andrychowicz et al., 2018 ). Yet despite this, there lacks a quantitative measure of intelligence in these agents. Qualitative measures can be deceptive. Consider agent Alice and Bob in Minecraft. Alice is capable of a constructing a house while Bob appears to only be able to navigate around the world. While at face value it may then appear that Alice is more complex, upon closer inspection we may find that Alice has simply memorized a set of actions to construct a house in that particular environment! How can we be certain that our agents are not simply not memorizing a set of moves? One hypothesis is that the more intelligent an agent is, the more likely the inner representations in its policy will exhibit disentangled properties of the world. Towards this end, we investigate the emergent visual representations that occur in RL policies. We investigate on various objectives and environment conditions, and find that the quality of visual representation learning correlates well with progress in reward optimization. Similarily, we find improved visual representations help agents perform better reward optimization. Thus, another natural question to ask is, how can we enable our agents to have better visual representations? While there are ways to hardcode reward functions to enable agents perform well, can we come up with a generic objective that our agents can optimize that will directly lead them to have good representations? One idea towards this is to use recent work in curiosity. In curiosity, agents are typically given rewards corresponding to surprisal of state. But another view of curiosity is that of a minimax game where a curious agent is seeking to maximize the surprisal of an uncertainty model, while the uncertainty model seeks become less surprised about new states. Thus, to enable a policy to learn good visual representations, we can treat the uncertainty model as a representation learning model. We then seek a policy that wants to lower the loss of the representation learning objective, while the model itself tries to optimize this loss. Under this objective, a policy must learn good visual representations, so that it is able to find visually surprising inputs for the vision model. We call this overall objective, Curious Representation Learning (CRL). By coupling policy learning with representation learning, we find that CRL allows us to get better policy visual representations simply by applying better visual representation learning algorithms to the model. As a result, we find that CRL obtains consistently good representations in policies across environment size and type, often beating many hard-coded domain specific objectives. As an added bonus, we find that CRL is also able to achieve better visual representation learning than other data collection methods, as it actively sees diverse inputs that surprise it. In this paper, we have shown visual representations correspond and help reward optimization. Motivated by this insight, we propose a new method, CRL, that allows us to get improved visual representations in policies through better visual representations in model. We further illustrate that these better visual representation can provide incentives to explore more in no reward scenarios. We hope that our results will inspire further exploration on both better visual representation learning models/policies and better reward optimization. We further show nearest neighbor images on VizDoom in Figure 8 . The leftmost column is the query image while the other 4 columns are the 4 nearest neighbors in embedding space. Training through CRL allows clustering of various doom objects.
We present a formulation of curiosity as a visual representation learning problem and show that it allows good visual representations in agents. This paper formulates curiosity based RL training as learning a visual representation model, arguing that focusing on better LR and maximising model loss for novel scenes will get better overall performance.