agents’ behavior in scenarios where the environment cannot be easily modeled [@goebel2018explainable], or the solutions are unknown a priori [@sheh2017did]. In this regard, recent applications derive human-level explanation based on the agent’s own knowledge of the situation [@cruz2020explainable]. In particular, transforming the selected Q-values for each action into a confidence metric, using a re-shaping function based on the logarithmic transformation [@cruz2020explainable], improved the understanding of a robot trying to solve a grid-based navigation task. The confidence metric measures how a specific action contributes to the robot reaching its goal however it does not carry any temporal correlation between the actions selected by the agent during the navigation. This means that the metric cannot be used to explain the behavior of the agent within the entire simulation, but only for each action.

This approach functions quite well when the end state of the task is clearly defined, and it is possible to measure the distance between the current state and the end state. However, it does not deal well with competitive scenarios where a set of agents have to learn decisions that a) maximize their goal, and b) minimize their adversaries’ goals. Besides considering the dynamic of the scenarios, they usually 