discrete or continuous actions, has recently led to the development of expert systems for guiding autonomous cars [@sallab2017deep; @isele2018navigating], predicting the stock exchange impact [@ponomarev2019using; @meng2019reinforcement], and coordinating a swarm of robots to protect the environment [@haksar2018distributed; @yu2018reinforcement], to name a few examples.

As their popularity grows, the need for providing stable and trustworthy solutions for real-world problems also increases. When deployed in production environments, these RL models, although designed to achieve impressive performance, are not easily understandable, and thus need experts to address the problems that may arise. The robustness of the RL solutions is thus limited by the general inability of explaining easily how these models learn and derive their state-action mapping [@cruz2019memory]. This has led to the fast development of eXplainable AI (XAI) [@gunning2017explainable] as a complementary field for deep reinforcement learning.

The community around XAI has been investigating how to approach the understanding of reinforcement learning when applied to different problems. Problems that involve the processing of human-based data are somehow the easiest ones to address, as demonstrated by explaining recommendation systems [@wang2018reinforcement], applications on the medical domain [@holzinger2017we], and robotics applications based on known behavior [@anjomshoae2019explainable]. However, it is much more difficult to explain the 