We humans have an innate understanding of the asymmetric progression of time, which we use to efficiently and safely perceive and manipulate our environment. Drawing inspiration from that, we approach the problem of learning an arrow of time in a Markov (Decision) Process. We illustrate how a learned arrow of time can capture salient information about the environment, which in turn can be used to measure reachability, detect side-effects and to obtain an intrinsic reward signal. Finally, we propose a simple yet effective algorithm to parameterize the problem at hand and learn an arrow of time with a function approximator (here, a deep neural network). Our empirical results span a selection of discrete and continuous environments, and demonstrate for a class of stochastic processes that the learned arrow of time agrees reasonably well with a well known notion of an arrow of time due to Jordan, Kinderlehrer and Otto (1998). The asymmetric progression of time has a profound effect on how we, as agents, perceive, process and manipulate our environment. Given a sequence of observations of our familiar surroundings (e.g. as video frames), we possess the innate ability to predict whether the said observations are ordered correctly. We use this ability not just to perceive, but also to act: for instance, we know to be cautious about dropping a vase, guided by the intuition that the act of breaking a vase cannot be undone. This profound intuition reflects some fundamental properties of the world in which we dwell, and in this work we ask whether and how these properties can be exploited to learn a representation that functionally mimics our understanding of the asymmetric nature of time. The term Arrow of Time was coined by the British astronomer Eddington (1929) to denote this inherent asymmetry, which he attributed to the non-decreasing nature of the total thermodynamic entropy of an isolated system, as required by the second law of thermodynamics. Since then, the notion of an arrow of time has been formalized and explored in various contexts, spanning not only physics, but also algorithmic information theory (Zurek, 1989) , causal inference (Janzing et al., 2016) and time-series analysis (Janzing, 2010; Bauer et al., 2016) . Broadly, an arrow of time can be thought of as a function that monotonously increases as a system evolves in time. Expectedly, the notion of irreversibility plays a central role in the discourse. In statistical physics, it is posited that the arrow of time (i.e. entropy production) is driven by irreversible processes (Prigogine, 1978; Seifert, 2012) . To understand how a notion of an arrow of time can be useful in the reinforcement learning context, consider the example of a cleaning robot tasked with moving a box across a room (Amodei et al., 2016) . The optimal way of successfully completing the task might involve the robot doing something disruptive, like knocking a vase over (Fig 1) . Now on the one hand, such disruptions -or side-effects -might be difficult to recover from. In the extreme case, they might be virtually irreversible -say when the vase is broken. On the other hand, irreversibility implies that states with a larger number of broken vases tend to occur in the future, and one should therefore expect an arrow of time (as a scalar function of the state) to assign larger values to states with larger number of broken vases. An arrow of time should therefore quantify the amount of disorder in the environment, analogous to the entropy for isolated thermodynamical systems. Now, one possible application could be to detect and preempt such side-effects, for instance by penalizing policies that significantly increment the arrow of time by executing difficult-to-reverse transitions. But the utility of an arrow of time is more general: it serves as a directed measure of The agent (in orange) is tasked with reaching its goal, the checkered flag (middle frame). It may take the shorter path (right frame), which entails breaking the vases in its way, or it may prefer the safer path (left frame) which is longer but keeps the vases intact. The former path is irreversible, and the initial state is unreachable from the final state (red arrow). On the contrary, the latter path is completely reversible, and the initial state remains reachable from the final state. Now, an arrow of time (pink) measures the disorder, which might help a safe agent decide which path to take. reachability. This can be seen by observing that it is more difficult to obtain order from disorder: it is, after all, difficult to reach a state with a vase intact from one with it broken, rather than vice versa. In this sense, we may say that a state is relatively unreachable from another state if an arrow of time assigns a lower value to the former. Further, a directed measure of reachability afforded by an arrow of time can be utilized for deriving an intrinsic reward signal to enable agents to learn complex skills in the absence of external rewards. To see how, consider that an agent tasked with reversing the arrow of time (by creating order from disorder) must in general learn complex skills to achieve its goal. Indeed, gluing together a broken vase will require the agent to learn an array of complex planning and motor skills, which is the ultimate goal of such intrinsic rewards. In summary, our contributions are the following. (a) We propose a simple objective to learn an arrow of time for a Markov (Decision) Process in a self-supervised manner, i.e. entirely from sampled environment trajectories and without external rewards. We call the resulting function (acting on the state) the h-potential, and demonstrate its utility and caveats for a selection of discrete and continuous environments. Moreover, we compare the learned h-potential to the free-energy functional of stochastic processes -the latter being a well-known notion of an arrow of time (Jordan et al., 1998) . While there exist prior work on detecting the arrow of time in videos (Pickup et al., 2014; Wei et al., 2018) and time-series data (Peters et al., 2009; Bauer et al., 2016) , we believe our work to be the first towards measuring it in the context of reinforcement learning. (b) We critically and transparently discuss the conceptually rich subtleties that arise before an arrow of time can be practically useful in the RL context. (c) We expose how the notions of reachability, safety and curiosity can be unified under the common framework afforded by a learned arrow of time. In this work, we approached the problem of learning an arrow of time in a Markov (Decision) Processes. We defined the arrow of time (h-potential) as a solution to an optimization problem and laid out the conceptual roadblocks that must be cleared before it can be useful in the RL context. But once these roadblocks have been cleared, we demonstrated how the notions of reachability, safety and curiosity can be bridged by a common framework of a learned arrow of time. Finally, we empirically investigated the strengths and shortcomings of our method on a selection of discrete and continuous environments. Future work could draw connections to algorithmic independence of cause and mechanism (Janzing et al., 2016) and explore applications in causal inference (Janzing, 2010; Peters et al., 2017
We learn the arrow of time for MDPs and use it to measure reachability, detect side-effects and obtain a curiosity reward signal.  This work proposes the h-potential as a solution to an objective that measures state-transition asymmetry in an MDP.