Reinforcement learning and evolutionary algorithms can be used to create sophisticated control solutions. Unfortunately explaining how these solutions work can be difficult to due to their "black box" nature. In addition, the time-extended nature of control algorithms often prevent direct applications of explainability techniques used for standard supervised learning algorithms. This paper attempts to address explainability of blackbox control algorithms through six different techniques: 1) Bayesian rule lists, 2) Function analysis, 3) Single time step integrated gradients, 4) Grammar-based decision trees, 5) Sensitivity analysis combined with temporal modeling with LSTMs, and 6) Explanation templates. These techniques are tested on a simple 2d domain, where a simulated rover attempts to navigate through obstacles to reach a goal. For control, this rover uses an evolved multi-layer perception that maps an 8d field of obstacle and goal sensors to an action determining where it should go in the next time step. Results show that some simple insights in explaining the neural network are possible, but that good explanations are difficult. Explanation of machine learning algorithms is a challenging and important field of research. Most techniques to date have focused on supervised learning algorithms, such as image processing, text processing and medical diagnosis BID10 BID5 . Instead of supervised learning, this paper focuses on reward based machine learning such as reinforcement learning and evolutionary algorithms, where rewards are given to measure performance instead of using examples of what is correct. The nature of reward learning and supervised learning is different in both problem domains and learning tools used to solve these problems. In this paper we look at explainability techniques that have been designed for supervised learning problems and apply them to reward learning problems. Reinforcement learning and evolutionary algorithms can be used to automatically learn high performance control systems for complex problems BID4 BID3 BID1 . This is particularly the case in the context of autonomy where control may involve many variables and need to dynamically adapt to different environments and situations.A common form of machine learning is to train a set of weights of a neural network-based control policy. Based on inputs (such as sensors) the control policy can command control actions (such as speed and direction of a vehicle). Training is typically done with a simulator, where the learning algorithm attempts to improve the performance of the control policy through a long series of trials. The goal of this training process is to produce a high-performance non-linear control policy that takes inputs and produces controls.While a successful training will produce a control policy that achieves high performance in simulation, how the control policy actually works will typically be unclear to its programmers, let alone its end-users. Due to this fact, machine learning algorithms are often referred to as "blackbox": their inputs and outputs can be viewed, but there is no knowledge of their internal workings.Even when machine learning achieves high performance, it can be difficult to trust for two reasons: 1) coverage, and 2) generalizability. In terms of coverage, while an algorithm may have performed well in scenarios that were tested, there may be other likely scenarios where it would have performed very poorly. In addition since coverage of machine learning algorithms is largely dependent on the data set, the user may not even be aware of the algorithm's coverage and can easily overlook large gaps in the data sets. In terms of generalizability, while the algorithm performed well in the simulator it may not perform well in the real world or in environments that are slightly different than the simulated one. These problems can be exacerbated by the blackbox nature of these learning algorithms, where reward hacking, poorly defined utility functions or simple errors in the simulator can lead to unrealistically high levels of performance that cannot be achieved when deployed. In addition, machine learning algorithms have many unintuitive parameters that have no obvious relation to the underlying control problem, such as number of hidden nodes and learning rates. Yet poor choices of these parameters can lead to poor generalization.Improving explainability of these blackbox algorithms can help improve trust that they will behave as expected when deployed (Gunning ) . If a control decision is backed up by a meaningful and understandable rationale, then one can trust that the decision is not made "by chance", and therefore the system can be expected to behave well in other similar circumstances. Additionally, if we understand a learned control algorithm, we can see if there are any clear gaps in coverage, or if there are any obvious flaws that would prevent it from generalizing outside of the simulated environment. On the other hand, what constitutes a meaningful, understandable explanation?Providing explanations of machine learning is a very active research field. Several approaches have been proposed for standard supervised learning algorithms. Despite this fact, it is still unclear what types of explanations may be suitable in practice. Control further complicates the picture, because control strategies develop over time, and are typically not evaluated over snapshots. How can such strategies be captured in explanations and what type of explanations would those be?To address this problem, we have experimented with a variety of techniques to provide explanations in the context of a very simple machine learning algorithm that we developed for navigating a rover towards a goal while avoiding obstacles. We decided to build the algorithm from scratch in order to evaluate the pitfalls and errors that may occur in developing such systems, as well as how/what explanations may assist in detecting those. We used six techniques in order to develop explanations: 1) Bayesian rule lists, 2) Function analysis, 3) Single time step integrated gradients , 4) Grammarbased decision trees, 5) Sensitivity analysis combined with temporal modeling with LSTMs, and 6) Explanation templates. This set of techniques was chosen as it represents a diverse set of explanations that could be readily applied to control data. In particular, it includes both local and global explanations . These local attempt to explain a single control action in a particular state. To form a big picture of a control policy with local explanations , we would want many local explanations covering many different states. In contrast global explanations try to explain an overall action policy over all states.The remainder of the paper is organized as follows. We first present the example obstacle avoidance problem we use throughout the paper. Then we describe the neural network controller and the Monte Carlo algorithm used to determine the weights of the neural network. We subsequently discuss the need for explainability and how simple analysis of algorithm performance may be insufficient. To address this we present six different explainability algorithms applied to the example problem and discuss their relative merits. While several explanation algorithms have been successfully used on supervised learning problems, direct application to reward based controls learning is somewhat illusive. A large part of this is due to the time-extended property of control policies. An action taken at a particular time step may seem sub-optimal at that particular time step but has benefits for future time steps. This limits a lot of direct application of supervised learning explanation as these explanations will tend to explain the superficial benefit of the action for the immediate time step and will likely miss the explanations of the future benefits. Our use of grammar-Based decision trees and temporal modeling attempt to address this issue, but they also lead to another problem: Control policies that need to optimize for future time steps are performing operations that are inherently complex and are difficult to summarize with simple explanations. In our test-domain the explanation algorithms are able to expose a major flaw in the operation of our learned neural network controller. However, it seems unlikely that they would be able to reveal more subtle issues or would be able to scale to more complex learned controllers. In addition the explanations do not seem as convincing or as useful as the explanations the same algorithms provide for their original supervised learning domain. Explaining a control algorithm based on machine learning is difficult due to the black-box nature of machine learning algorithms and the time-extended properties of control problems. In this paper we attempt to explain such a controller used on a simple obstacle avoidance problem: a neural network trained using a Monte Carlo algorithm. We do this by applying a number of explainability algorithms to this problem. These algorithms look at the inputs and outputs of the controller and based on these values attempt to explain what the controller is trying to do. The explanation algorithms proved useful in revealing a potential hazard in the controller, where it tries to head towards an obstacle and then turn to avoid it. However beyond this flaw it was difficult to gain deep insights into these explanations.
Describes a series of explainability techniques applied to a simple neural network controller used for navigation. This paper provides insights and explanations for the problem of providing explanations for a multilayer perceptron used as an inverse controller for rover movement, and ideas on how to explain a black-box model.