Environments in Reinforcement Learning (RL) are usually only partially observable. To address this problem, a possible solution is to provide the agent with information about past  observations. While common methods represent this history using a Recurrent Neural Network (RNN), in this paper we propose an alternative representation which is based on the record of the past events observed in a given episode. Inspired by the human memory, these events describe only important changes in the environment and, in our approach, are automatically discovered using self-supervision.
  We evaluate our history representation method using two challenging RL benchmarks: some games of the Atari-57 suite and the 3D environment Obstacle Tower. Using these benchmarks we show the advantage of our solution with respect to common RNN-based approaches. Deep Reinforcement Learning (RL) algorithms have been successfully applied to a range of challenging domains, from computer games (Mnih et al., 2013) to robotic control (OpenAI et al., 2018) . These approaches use a Neural Network (NN) both to represent the current observation of the environment and to learn the agent's optimal policy, used to choose the next action. For instance, the state observation can be the current game frame or an image of the robot camera, and a Convolutional Neural Network (CNN) may be used to obtain a compact feature vector from it. However, often RL environments are only partially observable and having a significant representation of the past may be crucial for the agent (Kapturowski et al., 2019) . For instance, in the Atari 2600 Pong game, the state can be represented as two consecutive frames. In this way, the agent can determine both the position of the ball and the direction of its movement. More complex partially observable domains require a longer state history input to the agent. For instance, when navigating inside a 1-st person-view 3D labyrinth, the agent obtains little information from the current scene observation and needs several previous frames to localise itself. A common solution to represent the observation history is based on the use of Recurrent Neural Networks (RNNs), where the RNN hidden-layer activation vector is input to the agent, possibly together with the current state observation (Mnih et al., 2016) . However, RL is characterized by highly nonstationary data, which makes training unstable (Schulman et al., 2016 ) and this instability is exacerbated when a recurrent network needs to be simultaneously trained to extract the agent's input representation. We show in Sec. 5.2 that in some common cases an RNN-based history representation struggles to improve the agent's results over an input representation composed of only the instantaneous observations. In this paper we propose a different direction, in which the agent's observation history is represented using a set of discrete events, which describe important changes in the state of the world (Orr et al., 2018) . Environment-specific events are automatically discovered during training by clustering past observations and are then used as landmarks in our history representation. Intuitively, this is inspired by the common human behaviour: when making decisions, humans do not keep detailed visual information of the previous steps. For instance, while navigating through the halls of a building, it is sufficient to recall a few significant landmarks seen during the walk, e.g. specific doors or furniture. Following this idea, in this paper we propose an Event Discovery History Representation (EDHR) approach, composed of 3 iterative stages: experience collection, policy optimisation and event discovery. We discover events by clustering past state observations. In more detail, we maximize the Mutual Information (MI) between the latent representations of temporally-close frames to cluster the frame semantics. This is a form of self-supervision, in which no additional annotation is needed for the event discovery: the higher the predictability of one frame with respect to the other, the larger the semantic information shared by the two frames (van den Oord et al., 2018; Anand et al., 2019; Ji et al., 2019) . Once clusters have been formed, the probability distribution of a given frame F t for the set of current clusters is used as the semantic representation of the state observation at time t and is recorded in a longer history. Finally, the history is input to the agent together with an instantaneous observation representation, obtained, following (Mnih et al., 2013) , as a stack of the last 4 frames. This information is now used for policy optimisation. Note that our proposed history representation is independent of the specific RL approach. However, in all our experiments we use the PPO algorithm (Schulman et al., 2017) for the policy and the value function optimisation. The 3 stages are iterated during training, and thus past clusters can be modified and adapted to address new observations, while the agent is progressing through the environment. We summarize the contribution of this paper below. First, we use a modified version of the Invariant Information Clustering (IIC) algorithm (Ji et al., 2019) to discover significant events in the agent's past observations. Second, we propose to replace common RNN-based history representations with a time-dependent probability distribution of the observations for the events so far discovered. We evaluate our history representation (EDHR) using the PPO algorithm on several environments of the Atari-57 benchmark (Bellemare et al., 2012) and on the 3D environment Obstacle Tower (Juliani et al., 2019) , showing that it provides a significant boost with respect to the most common input representation methods. Specifically, we show that EDHR outperforms plain PPO in cases, where history can increase observability, and it outperforms RNN-based methods in several common cases, while simultaneously requiring 2× less wall-clock time for training. The source code of the method and all the experiments is publicly available at the anonymous link: https://github.com/iclr2020anon/EDHR and will be published after acceptance. We presented a method for history representation in RL which is an alternative to common solutions based on RNNs. Our EDHR is based on the idea that important information about the past can be "compressed" using events. Specifically, these events are automatically discovered using a modification of the ICC clustering method (Ji et al., 2019) , which is performed jointly with the agent training and iterated through time, to adapt the discovered events to the new observations. In EDHR, visual information is represented using two different networks: Φ and Ψ. The latter is trained using a reward signal, so it presumably extracts task-specific information from the observations. On the other hand, the encoder Φ is trained using self-supervision, and thus it focuses on patterns which are repeated in the data stream, potentially leveraging a larger quantity of supervision signal. Although self-supervision has been explored in other RL and non-RL works, this is the first work to show how the discovered information can be exploited in the form of discrete events used for history representation. Our results, based on the challenging deep RL benchmarks, ALE and Obstacle Tower, show that EDHR can more effectively represent information about the past in comparison with task-oriented representations such as those used in PPO and PPO-RNN. A HYPERPARAMETERS In Tab. 3 we list all our hyperparameters which are different from (Schulman et al., 2017) .
event discovery to represent the history for the agent in RL The authors study the problem of RL under partially observed settings, and propose a solution that uses a FFNN but provides a history representation, outperforming PPO. This paper proposes a new way to represent past history as input to an RL agent, showing to perform better than PPO and an RNN variant of PPO.