In this paper, I discuss some varieties of explanation that can arise
 in intelligent agents. I distinguish between process accounts, which
 address the detailed decisions made during heuristic search, and
 preference accounts, which clarify the ordering of alternatives
 independent of how they were generated. I also hypothesize 
 which types of users will appreciate which types of explanation.
 In addition, I discuss three facets of multi-step decision making
 -- conceptual inference, plan generation, and plan execution --
 in which explanations can arise. I also consider alternative ways
 to present questions to agents and for them provide their answers.
 Intelligent systems are becoming more widely adopted for critical tasks like driving cars and controlling military robots. Our increased reliance on such devices has led to concerns about the interpretability of their complex behavior. Before we can fully trust such autonomous agents, they must be able to explain their decisions so that we can gain insight into their operation. There is now a substantial literature on explanation in systems that learn from experience, but it has focused on tasks like object recognition and reactive control, typically using opaque encodings of expertise.However, we also need research on explanation for more complex tasks that involve multi-step decision making, such as the generation and execution of plans. Approaches to these problems rely on high-level representations that are themselves easily interpreted, but challenges arise in communicating solutions that combine these elements and the reasons they were chosen. In this paper, I focus on such settings. Some work on explanation, especially with opaque models, has dealt with post hoc rationalizations of behavior, rather than the actual reasons for it. In the pages that follow, I limit my discussion to the latter. Moreover, I will focus on self explanations, that is, the reasons the explaining agent carried out a certain activity. Elsewhere BID8 , I have referred to this ability as explainable agency. This problem is arguably less challenging than postulating the reasons that another agent behaved as it did, sometimes called plan recognition, as the system can store and access traces of its own decision making. We can specify the task of explainable agency in generic terms. Given domain knowledge for generating task solutions and criteria for evaluating candidates, the agent carries out search to find one or more solutions. After generating, and possibly executing, these solutions, a human asks the agent to justify its decisions, at which point it must clarify its reasoning in comprehensible terms. One example involves an intelligent robot that plans and executes a reconnaissance mission, after which it takes part in an 'after-action review' where it answers questions from a human supervisor. There has been some research on such explainable planning (Fox et al., 2017; Smith, 2012; BID15 ), but we need more effort devoted this important topic.In the next section, I distinguish between two forms of self explanation, identify component abilities they require, and citing relevant research. I also propose two hypotheses about when each type of account will be most useful. After this, I discuss three types of content over which one can generate explanations, along with alternative ways to pose questions and present answers. In closing, I review the essay's main points and reiterate the need for substantially additional research on the topic of explainable agency.
This position paper analyzes different types of self explanation that can arise in planning and related systems.  Discusses different aspects of explanations, particularly in the context of sequential decision making. 