# Modeling Data Analytic Iteration With
Probabilistic Outcome Sets

Roger D. Peng
Department of Statistics and Data Sciences, University of of Texas, Austin
and
Stephanie C. Hicks
Department of Biostatistics and Department of Biomedical Engineering
Johns Hopkins University  The authors gratefully acknowledge Lucy D’Agostino
McGowan, Tiffany Timbers, Alyssa Columbus, and Boyi Guo for their constructive
feedback and suggestions to improve this manuscript.

###### Abstract

In 1977 John Tukey described how in exploratory data analysis, data analysts
use tools, such as data visualizations, to separate their expectations from
what they observe. In contrast to statistical theory, an underappreciated
aspect of data analysis is that a data analyst must make decisions by
comparing the observed data or output from a statistical tool to what the
analyst previously expected from the data. However, there is little formal
guidance for how to make these data analytic decisions as statistical theory
generally omits a discussion of who is using these statistical methods. Here,
we extend the basic idea of comparing an analyst’s expectations to what is
observed in a data visualization to more general analytic situations. In this
paper, we propose a model for the iterative process of data analysis based on
the analyst’s expectations, using what we refer to as expected and anomaly
probabilistic outcome sets, and the concept of statistical information gain.
Our model posits that the analyst’s goal is to increase the amount of
information the analyst has relative to what the analyst already knows,
through successive analytic iterations. We introduce two criteria–expected
information gain and anomaly information gain–to provide guidance about
analytic decision-making and ultimately to improve the practice of data
analysis. Finally, we show how our framework can be used to characterize
common situations in practical data analysis.

Keywords: data analysis, Shannon entropy, information, exploratory data
analysis, confirmatory data analysis

##  1 Introduction

An important goal of statistical theory is to characterize the behavior of
data analytic tools in the presence of random variation in the data. These
theoretical properties are derived by making some assumptions about the data
generating mechanism and by analyzing the mechanisms of the analytic tools
themselves. The properties of statistical methods are typically presented in a
manner that is independent of any specific person applying the methods.
Therefore, two data analysts employing the same model on the same data (making
the same assumptions) should agree about the method’s properties, which is
useful when objectively evaluating or comparing a set of methods.

In the practice of data analysis, data analysts apply numerous tools to data
and must be familiar with their theoretical properties. In addition, analysts
make decisions in response to observing the data or the output obtained from
applying statistical and computational methods to the data. The decisions made
by the analyst drive an iterative analytic process that ultimately leads to
new knowledge or evidence regarding a scientific, business, or policy question
(Kim & Hardin, 2021; Peng & Parker, 2022). It is natural to assume that
different analysts, over the course of an entire analysis, might make
different decisions regarding what tools to apply to the data and how to react
to the output the tools generate (Breznau et al., 2022). These decisions may
be influenced by the individual analyst’s (a) familiarity with data collection
procedures; (b) knowledge of the properties of statistical techniques; and (c)
experience with computing, all of which might affect the analyst’s
expectations for observing the output of a given tool. This sense-making
process is critically dependent on the analyst’s perspective and previous
understanding of the underlying phenomena being studied and the data
(Grolemund & Wickham, 2014).

For example, an analyst who fits a linear regression model and obtains a
negative slope when they were expecting a positive slope can react to that
output in a variety of ways. An experienced practitioner might be familiar
with possible reasons for why this can occur (such as outliers) and choose a
diagnostic tool as a next step. A novice practitioner might not be familiar
with the possibility of outliers and might choose to accept the result as part
of the expected variation in the data. It may not be clear who is correct in
this situation, but it is clear that different analysts can take different
paths at this juncture.

Others have noted that there is relatively little guidance about how to carry
out a data analysis or how to make data analytic decisions (Unwin, 2001; Wild,
1994; Peng, 2017). A contributor to this reality is that statistical theory
generally omits a discussion of who is using these statistical methods and
applying them to the data. Such an omission allows for an objective discussion
of statistical methods and their properties, but it also omits any information
or background that an individual analyst might bring to a problem.

The iterative nature of data analysis, which is driven forward by the numerous
decisions made by the analyst, is another characteristic of practical data
science that is often missing from statistical theory frameworks (Peng et al.,
2021; Peng & Parker, 2022). Real-world data analyses will typically involve
long sequences of data operations, some of which may involve statistical
models and some of which may involve basic data wrangling operations. At the
end of each iteration, the analyst must decide what to do next, if anything:
What tool to apply, what new information to gather, or perhaps what expertise
to consult. While statistical theory is helpful in the areas involving the
application of statistical models, what is largely missing is a general
framework for describing how to make decisions when analyzing data. While
previous work has suggested the outlines of what this framework might look
like (Wild & Pfannkuch, 1999; Grolemund & Wickham, 2014; Greenhouse & Seltman,
2018; Lovett & Greenhouse, 2000), details of how to operationalize it, in
practice for analyzing data, are sparse.

John Tukey wrote in his landmark book Exploratory Data Analysis that “The
greatest value of a picture is when it forces us to notice what we never
expected to see.” (Tukey, 1977). While his comment was in reference to data
visualization in particular, it is possible to take the ideas embedded in his
comment, such as the dichotomy between expectations and observations, to build
a more general framework for data analytic iteration. In this paper, we
present a model for characterizing the iterative application of analytic tools
to data and the information that is obtained. Our approach incorporates the
unique perspective of the analyst conducting the analysis by considering what
the analyst expects to observe before applying a tool to the data. A key
question that we aim to address here is how does an analyst decide what to do
next in an analytic iteration?

We propose a model for the iterative process of data analysis based on the
analyst’s expectations and the concept of statistical information gain. Our
model posits that the analyst’s goal is to increase the amount of information
the analyst has, relative to what the analyst already knows, through
successive analytic iterations. We introduce the notion of probabilistic
outcome sets to formalize what the analyst expects to see in an analysis
output and what would be considered unexpected. We then propose two
criteria—expected information gain and anomaly information gain—for modeling
how an analyst might choose the next step in an analysis. Finally, we discuss
how this model can be used to describe common data science practices and
outline some of the possibilities of using this model in more complex data
analyses.

##  2 Data Generation Mechanisms in Analytic Iteration

Analytic iteration is defined here as the process by which an analyst (a)
plans a data operation by selecting an analytic tool (e.g. calculate the
empirical mean) to apply to a given dataset, (b) sets their expectations about
the potential outcomes that could arise from the data operation, (c) executes
the data operation, (d) compares the observed result to what the analyst
expected prior to executing the operation, and (e) decides which tool to use
in the next iteration. Each step in the analytic iteration is designed to
produce information from the data, and perhaps, other useful output such as
numerical summaries, graphical summaries, or fitted models.

The analyst’s general goal in a data analysis is to learn about the mechanism
that produced the data while asking some scientific, policy, or business
question related to that dataset. This data generating mechanism is
represented by the cumulative distribution function $F$ and is assumed to be
unknown. The observed data $\mathbf{X}$ are sampled from population
$\mathcal{X}$ according to distribution $F$. The analyst’s best estimate of
$F$ at a specific moment in an analytic iteration is called $\widehat{F}$. We
assume the densities for these distribution functions exist, so that $f$ is
the density associated with $F$ and $\hat{f}$ is the density associated with
$\widehat{F}$.

While a data analyst can examine the raw data directly, typically such
examination will occur through the output of some tool $T$ applied to
$\mathbf{X}$. For example, the distribution of a variable may be examined by
looking at a histogram. The output of $T$ is $\mathbf{Y}=T(\mathbf{X})$ and we
will be concerned with the probability distribution of this output, which will
be denoted by $\mathbb{P}$. Let the set $A$ represent a collection of possible
values for $\mathbf{Y}$ (we will be more precise about this set in Section 4).
The probability of the event that the output $\mathbf{Y}$ falls in $A$ is
denoted as $\mathbb{P}(\mathbf{Y}\in
A)\stackrel{{\scriptstyle\Delta}}{{=}}\mathbb{P}(\mathbf{X}\in
T^{-1}(A))=\int_{T^{-1}(A)}\,dF$. Here, $T^{-1}$ is the inverse mapping of $T$
and $T^{-1}(A)$ represents the various configurations of $\mathbf{X}$ that
result in $\mathbf{Y}$ when $T$ is applied to $\mathbf{X}$. The analyst’s best
estimate of the probability that $\mathbf{Y}$ falls in a set $A$ is denoted as
$\widehat{\mathbb{P}}(\mathbf{Y}\in
A)\stackrel{{\scriptstyle\Delta}}{{=}}\widehat{\mathbb{P}}(\mathbf{X}\in
T^{-1}(A))=\int_{T^{-1}(A)}\,d\widehat{F}$.

We propose that analytic iteration is a sequential process by which the
analyst learns about $F$ while asking some question, and investigating that
question using the observed data $\mathbf{X}$ from $F$. This process is
constrained by the requirements of the scientific, policy, or business
question being asked. By asking a sequence of questions and applying a
corresponding sequence of informative analytic tools to the data, the analyst
gains information about $F$ at each step of the iteration. The goal for the
analyst is for $\widehat{F}$ to be close enough to $F$ to serve the analyst’s
interests as well as the interests of any other stakeholders.

For the remainder of the paper we will assume that we are in the middle of a
single iteration of a larger data analysis. We will use $\widehat{F}$ to
indicate the analyst’s current best estimate of $F$ and $\widehat{P}$ to
indicate the analyst’s current best estimate of $\mathbb{P}$, the probability
distribution associated with the output of some tool $T$.

##  3 Informative Tools for Analytic Iteration

At each stage of an analytic iteration, an analyst applies a tool $T$ to the
data $\mathbf{X}$ to produce output $\mathbf{Y}$. The tool $T$ could range
from a basic computation, such as computing the sample mean of a column in a
table, to a complex statistical model. An initial problem to address is
determining whether a given tool $T$ is informative for a given problem when
it is applied to $\mathbf{X}$. In the following section, we formally describe
what is meant by an informative tool.

For a given question or problem, we begin by considering a collection of
competing hypotheses in $\mathcal{H}$ about $\mathbf{X}$, which we refer to as
the hypothesis set. For example, the hypotheses in the hypothesis set could be
defined on whether there are no missing values ($H_{1}$) or there are some
missing values ($H_{2}$) in a datasest $\mathbf{X}$. Similarly, a set of
hypotheses could be defined on whether the data $\mathbf{X}$ follow a
symmetric ($H_{1}$) or right-skewed ($H_{2}$) distribution. Alternatively, a
set of hypotheses could involve a statement about a population parameter
$\theta$, where one hypothesis is $H_{1}:\theta=\theta_{1}$ for some
$\theta_{1}$, a second hypothesis is $H_{2}:\theta=\theta_{2}$, and third
hypothesis is $H_{3}:\theta=\theta_{3}$.

For simplicity, in the definition of an informative tool below, we consider
the scenario where the hypothesis set $\mathcal{H}$ only contains two
competing hypotheses $\mathcal{H}=\\{H_{1},H_{2}\\}$. The basic concept here
is that tool is an informative tool if the expected outcome of applying that
tool to the data is different under $H_{1}$ relative to $H_{2}$.

######  Definition 1 (Informative tool in analytic iteration).

A tool $T$ applied to a given dataset $\mathbf{X}\sim F$ is an informative
tool for discriminating between competing hypotheses $H_{1}$ and $H_{2}$ if
$\mathbb{E}_{F_{H_{1}}}[T(\mathbf{X})]\neq\mathbb{E}_{F_{H_{2}}}[T(\mathbf{X})]$.

The property of informativeness is not an inherent or intrinsic property of
any given data analytic element or tool. Informativeness is something that can
only be determined in the context of the hypothesis set $\mathcal{H}$ under
consideration. We cannot say whether a boxplot or histogram is inherently
informative or not for all types of questions asked about $\mathbf{X}$. For
example, a histogram may be informative for discriminating between hypotheses
about the symmetry or skewness of the data. However, the histogram will not be
informative for discriminating between hypotheses concerning the correlation
of a variable with another variable. For more general hypothesis sets
$\mathcal{H}$, which may for example contain an infinite number of hypotheses,
an informative tool is expected to produce distinct output from the data under
each distinct hypothesis.

In the remainder of this paper, we will assume that the tools applied to the
data are informative in the sense of Definition 1.

##  4 Potential Outcome Sets for Analytic Iteration

We begin by assuming a data operation has been planned, so that the analyst
has chosen a tool to apply to a dataset. We will return to how the analyst
chooses this tool in Section 6.1. In this section, we define three sets of
events that could occur related to the potential (or possible) outcome of
applying a given analytic tool to a dataset in a single analytic iteration. We
note that all three of these potential outcome sets (Definitions 2-4) are
characterizing events before the analyst applies a given analytic tool to the
data.

###  4.1 Complete Potential Outcome Set

Once an analyst has chosen an analytic tool to apply to the data, it is
usually possible to characterize in some way the set in which the output might
fall, before the tool is actually applied to the data. In the following, we
consider the unobserved data as a random variable $\mathbf{X}$. For a given
tool, $T$ we aim to describe the entire set of possible outcomes from the tool
when applied to $\mathbf{X}$. We call this set the complete potential outcome
set.

######  Definition 2 (Complete potential outcome set).

Let $\mathbf{X}\in\mathcal{X}$ be a random variable with a cumulative
distribution function $F$. We consider $\mathbf{Y}=T(\mathbf{X})$, defined as
a mapping from the original sample space of $\mathbf{X}$,
$\mathcal{X}=\\{\mathbf{x}:f(\mathbf{x})>0\\}$, to a new sample space
$\mathcal{Y}$, the sample space for the random variable $\mathbf{Y}$, i.e.
$T:\mathcal{X}\mapsto\mathcal{Y}$. We associate $T$ with an inverse mapping,
denoted by $T^{-1}$, which maps subsets of $\mathcal{Y}$ to subsets of
$\mathcal{X}$, and is defined by

$T^{-1}(A)=\\{\mathbf{x}\in\mathcal{X}:T(\mathbf{x})\in A\\}.$

We also assume the mathematical domain (or input) of $T$ is valid for the data
type $\mathbf{X}$. Then, the sample space
$\mathcal{Y}=\\{\mathbf{y}:\mathbf{y}=T(\mathbf{x})\text{ for some
}\mathbf{x}\in\mathcal{X}\\}$, which is the image of $\mathcal{X}$ under $T$,
is the complete potential outcome set that describes all the potential
outcomes of the analytic step for applying tool $T$ to the data $\mathbf{X}$.

For example, consider a set of observations $\mathbf{x}=(x_{1},\ldots,x_{n})$
that comes in the form of a collection of $n$ non-negative integers with
sample space $\mathcal{X}=\mathbb{Z}^{+}_{0}=\\{0,1,\ldots,\\}$. If we
consider the sample mean as a tool $T$,
$\mathbf{y}=T(\mathbf{x})=\frac{1}{n}\sum_{i=1}^{n}x_{i}$, then the complete
potential outcome set is the positive half of the real line $[0,\infty)$, or
$\mathcal{Y}=\\{\mathbf{y}:\mathbf{y}=T(\mathbf{x}),\mathbf{x}\in\mathcal{X}\\}=\mathbb{R}^{+}_{0}$.

For analytic tools that summarize numerical data, such as calculating a sample
mean, it is straightforward to specify their complete potential outcome sets
with some precision. However, with other output, such as graphical or high-
dimensional output, it may be more difficult to specify the complete set of
potential outcomes. Nevertheless, it may still be possible to describe in a
general sense what output could be produced, and there is value in developing
such a description. For example, when making a scatter plot of pairs of $n$
data points $(x_{11},x_{21}),\dots,(x_{1n},x_{2n})$, where $x_{1i}$, $x_{2i}$
$\in\mathbb{R}$, then the complete potential outcome set is the set of all
possible combinations of $n$ points arranged on the plane $\mathbb{R}^{2}$.

The complete potential outcome set follows from the nature of the data and the
known properties of the analytical tool. Therefore, the complete potential
outcome set is common to any analysis meeting the same specifications (e.g.
computing the arithmetic mean on non-negative integers). An implicit
assumption that we make is that most analysts would agree on the
characteristics of the complete potential outcome set, conditional on the data
and the chosen analytic tool. Thus, the specification of the complete
potential outcome set does not incorporate an analyst’s unique perspective or
experience, aside from the fact that the analyst chose the analytic tool.

###  4.2 Probability Structure for Outcome Sets

Next, in order to further model the analytic process, we need to introduce a
probability structure for modeling complete potential outcome sets. Then, this
probability structure will allow us to introduce and model two additional
outcome sets below, namely the expected and anomaly potential outcome sets.
The probabilities here characterize the likelihood of the output $\mathbf{Y}$
from the analytic tool $T$ on the data $\mathbf{X}$ falling in a given
potential outcome set. Because the true data generating mechanism $F$ is
presumably not known to the analyst, the analyst will have to estimate the
probability that the outcome of an analytic tool falls anywhere in a potential
outcome set. To distinguish between the analyst’s estimate of the data
generating mechanism and the true mechanism, we denote
$\widehat{\mathbb{P}}(A)$ rather than $\mathbb{P}(B)$ for some event $B$, as
described in Section 2.

The randomness that gives rise to the probability structure that we propose
has two primary sources. The first is the traditional statistical variation
that would cause two different sampled datasets to be different in their
values. The second source of variation arises from “upstream uncertainty”,
which is the series of activities that occurs between data collection and the
generation of data analytic output. For example, in some contexts, batch
effects can be introduced in data collection that can affect inferences in
downstream analyses. Similarly, data recording errors can produce outliers in
certain variables. In both cases, the analyst may not initially be aware of
these features of the data. If the analyst is not able to completely
characterize these upstream processes, then it may be reasonable to consider
this aspect of the data generation as random. Of course, once the analytic
step has been taken and the method is applied to the data, we observe the
result and there is nothing random. Hence, all discussions of probability in
this section are to be considered in the context prior to applying a method to
the data.

Next, we describe three assumptions about complete potential outcome sets.
First, a basic, but also an important assumption that we make regarding the
data $\mathbf{X}$, the tool $T$, and the output $\mathbf{Y}$, is that the
complete potential outcome set $\mathcal{Y}$ contains all of the outcomes that
we could observe.

######  Assumption 1 (Complete potential outcome set contains all outcomes).

Suppose the analyst has chosen $T$ to apply to the data $\mathbf{X}$ and let
$\mathbf{Y}$ represent the output from that method (as yet unobserved). Let
$\mathcal{Y}$ represent the complete potential outcome set for $\mathbf{Y}$
and let $\\{\mathbf{Y}\in\mathcal{Y}\\}$ represent the event that the output
$\mathbf{Y}$ is observed to fall in $\mathcal{Y}$. Let $\mathbb{P}$ represent
the true, but unknown, probability distribution governing the output
$\mathbf{Y}$. Then, $\mathbb{P}(\mathbf{Y}\in\mathcal{Y})=1$.

The second assumption we make states that although an analyst cannot know
$\mathbb{P}$, the probability distribution governing the output $\mathbf{Y}$,
the analyst’s best estimate of $\mathbb{P}$, i.e. $\widehat{\mathbb{P}}$,
satisfies the same assumption.

######  Assumption 2 (Complete potential outcome set contains all of analyst’s
outcomes).

Let $\widehat{\mathbb{P}}$ be an analyst’s best estimate of the true
probability $\mathbb{P}$, which is unknown in general. Then, we also assume
that $\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{Y})=1$.

Third, we note that $\mathbb{P}$ and the complete potential outcome set
$\mathcal{Y}$ are universal in the sense that they are independent of the
analyst looking at the data. However, $\widehat{\mathbb{P}}$ is specific to an
analyst and will likely differ from analyst to analyst.

######  Assumption 3 (Universality of the complete potential outcome set).

Suppose two analysts labeled A and B look at the same dataset $\mathbf{X}$
independently of each other and apply the same tool $T$ to produce output
$\mathbf{Y}$. Denote $\widehat{\mathbb{P}}_{A}$ and $\widehat{\mathbb{P}}_{B}$
as the estimates of $\mathbb{P}$ for analysts A, and B, respectively. Then, we
assume that
$\widehat{\mathbb{P}}_{A}(\mathbf{Y}\in\mathcal{Y})=\widehat{\mathbb{P}}_{B}(\mathbf{Y}\in\mathcal{Y})=1$.

While it is likely that
$\widehat{\mathbb{P}}_{A}\neq\widehat{\mathbb{P}}_{B}$, so that two different
analysts might disagree on how the probability mass of the output is
distributed across $\mathcal{Y}$, they should still agree on the overall
structure of $\mathcal{Y}$ and the boundaries of where $\mathbf{Y}$ might
fall, regardless of how likely or unlikely. In what follows, we will use
$\widehat{\mathbb{P}}$ to denote an arbitrary analyst’s estimate of the
probability distribution.

Continuing the example from the previous section with non-negative integer
data $\mathbf{X}=(X_{1},\dots,X_{n})$, if our analytic method $T$ were the
sample mean such that $\mathbf{Y}=T(\mathbf{X})$, then our complete potential
outcome set is $\mathcal{Y}=\mathbb{R}_{0}^{+}$. Under this setup, it would be
reasonable to assume that $\mathbb{P}(\mathbf{Y}\in\mathcal{Y})=1$ and that an
analyst’s best estimate $\widehat{\mathbb{P}}$ would also satisfy
$\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{Y})=1$.

Next, we partition the complete potential outcome set $\mathcal{Y}$ into two
subsets, namely the expected potential outcome set (Section 4.3) and the
anomaly potential outcome set (Section 4.4).

###  4.3 Expected Potential Outcome Set

In this section, we introduce a second potential outcome set called the
expected potential outcome set, which is a subset of the complete potential
outcome set and is unique to the data analyst. In our analytic iteration
model, before applying $T$ to the data, the analyst often makes a (explicit or
implicit) prediction as to what the output $\mathbf{Y}=T(\mathbf{X})$ is
expected to be, or the range that the output will fall in. Such predictions
can range from being vaguely characterized to being very well specified,
perhaps based on some sort of previously developed hypothesis or preliminary
data. The predictions made by the analyst incorporate previous knowledge about
the data and the underlying data generation process.

The expected potential outcome set is the set, according to the best judgment
of the analyst, in which the output $\mathbf{Y}=T(\mathbf{X})$ is expected to
fall $\mathcal{E}$, which is a subset of the complete potential outcome set
$\mathcal{Y}$. Specifying the expected potential outcome set is important for
interpreting the output from the analytic step and for deciding on any next
steps in the analytic iteration. In particular, the expected potential outcome
set indirectly specifies which outcomes are unexpected, which we will discuss
in the next section.

######  Definition 3 (Expected potential outcome set).

Let $\mathbf{Y}=T(\mathbf{X})$ be the output of applying an analytic tool $T$
to the data $\mathbf{X}$ and let $\mathcal{E}$ be a subset of the complete
potential outcome set $\mathcal{Y}$. $\mathcal{E}$ is an expected potential
outcome set for $\mathbf{Y}$ if it satisfies:

  1. 1.

$\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})>0$;

  2. 2.

$\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})<1$; and

  3. 3.

$\frac{\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})}{1-\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})}>1$

where $\widehat{\mathbb{P}}$ is the analyst’s best estimate of the true
probability structure $\mathbb{P}$.

A defining characteristic of $\mathcal{E}$ is that the analyst believes that
the outcome is expected to fall in that set, by which we mean that
$\mathbf{Y}$ is “more likely than not” to fall in $\mathcal{E}$, in the best
judgment of the analyst.

The expected potential outcome set depends on many factors, including the
analyst’s current state of knowledge of the science, assumptions about the
underlying data generation process, the ability to model the data using
statistical or computational tools, knowledge of the theoretical properties of
those tools, and the understanding of the uncertainty or variability of the
observed data across multiple samples. Hence, even if the underlying truth is
thought of as fixed, it is reasonable to assume that different analysts might
develop different sets of expected potential outcomes, reflecting differing
levels of familiarity with the various factors involved and different
weightings of existing evidence or data.

####  4.3.1 Example: Reading in a data file

A common data science activity involves reading in data into a statistical or
analytical software package. Consider a scenario where an analyst receives a
data file and is told by the project scientist that the data file contains
1,000 observations. The analyst’s first analytic step might be to read in the
data and verify that the resulting data object is a table with 1,000 rows.
Here, the tool $T$ might be a function that counts the number of rows or
perhaps a printed summary of the number of rows provided by the software. The
“data” $\mathbf{X}$ in this case is the dataset file being read in, and the
output $\mathbf{Y}$ is the number of rows. The complete potential outcome set
$\mathcal{Y}$ is the set of non-negative integers and the expected potential
outcome set might be $\mathcal{E}=\\{1000\\}$, i.e. the set containing a
single integer. Finally, the analyst might estimate that
$\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})=0.99$ based on the fact that
the project scientist confirmed beforehand that there were 1,000 rows.

###  4.4 Anomaly Potential Outcome Set

The anomaly potential outcome set contains potential outcomes for $\mathbf{Y}$
that are unexpected by the analyst for data $\mathbf{X}$ using analytic tool
$T$. The outcomes characterized by the anomaly potential outcome set are not
impossible, but are deemed unlikely to occur in the eyes of the analyst. The
anomaly potential outcome set will typically not be specified directly, but
rather indirectly via the specification of the complete potential outcome set
and the expected potential outcome set.

######  Definition 4 (Anomaly potential outcome set).

Given a complete potential outcome set $\mathcal{Y}$ and expected potential
outcome set $\mathcal{E}\subset\mathcal{Y}$, the anomaly potential outcome set
is $\mathcal{E}^{c}=\mathcal{Y}\setminus\mathcal{E}$.

Using the definitions of $\mathcal{Y}$ and $\mathcal{E}$, it follows that for
an analytic output $\mathbf{Y}$,
$\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E}^{c})>0$ and
$\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E}^{c})=1-\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})<0.5$.
As $\mathcal{E}^{c}$ is the complement of $\mathcal{E}$ in $\mathcal{Y}$, the
interpretation of $\mathcal{E}^{c}$ is that the output $\mathbf{Y}$ may fall
in $\mathcal{E}^{c}$, but is “less likely than not” to do so.

Continuing the data file example from the previous section, because
$\mathcal{E}=\\{1000\\}$ and $\mathcal{Y}=\mathbb{R}^{+}_{0}$, then the
anomaly set is $\mathcal{E}^{c}=\\{0,1,\dots,999,1001,1002,\dots\\}$. Given
that the analyst specified
$\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})=0.99$, we have that
$\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E}^{c})=0.01$. Hence, from the
perspective of the analyst, it is highly unlikely that the number of rows
presented in the output will be anything other than 1,000.

###  4.5 From Outcome Sets to Information

Dividing the complete potential outcome set into the complementary expected
and anomaly potential outcome sets is a task that separates data analysis
problems from traditional statistical problems. In particular, this division
must be done by a person, namely the data analyst, and every analyst will
likely devise a different configuration. The expected and anomaly potential
outcome sets are dependent on the analyst’s judgment and are not inherent
aspects of the science or the data. Therefore, as the analyst’s information
about the data and the data generating mechanism accumulates, it is possible
that the expected potential outcome sets for various outputs will evolve.

Developing the different potential outcome sets in an analytic step, as well
as their probability structure, allows us to characterize what we see as the
goal of data analysis, which is to gain information. Information cannot be
obtained without risk, and in our model, information is gained from resolving
the uncertainty about the output $\mathbf{Y}$ obtained from applying $T$ to
$\mathbf{X}$. By observing the result of applying an analytic tool $T$ and
seeing whether it falls in $\mathcal{E}$ or $\mathcal{E}^{c}$, this leads to a
formal framework for how to determine what is learned in a step of an analytic
iteration, which we describe in the next section.

##  5 Information Gain From A Single Analytic Step

Upon observing $\mathbf{Y}$, the output from a single analytic step, we can
determine whether one of the mutually exclusive events
$\\{\mathbf{y}\in\mathcal{E}\\}$ or $\\{\mathbf{y}\in\mathcal{E}^{c}\\}$ has
occurred. At this point, the analyst must make a decision about what to do
next, using the observation that has just been realized. In particular, we
need a framework for choosing amongst a collection of informative tools that
we can apply to the data in the next iteration. In order to aid this choice,
we first characterize the information that is gained from observing the output
from an analytic step. We then define information-based criteria that can be
used to prioritize the available tools based on an analyst’s goals or current
knowledge of the data. We propose two possible criteria in the next section.
The first involves maximizing the expected amount of new information obtained
from applying $T$ to $\mathbf{X}$ and the second guides the analyst to choose
a tool such that the potential for strong evidence is maximized.

###  5.1 Information Gain

For a given tool $T$, we describe below how the information gained from
applying $T$ to the data $\mathbf{X}$ is determined by the expected and
anomaly potential outcome sets $\mathcal{E}$ and $\mathcal{E}^{c}$ constructed
by the analyst and $\widehat{\mathbb{P}}$, the analyst’s best estimate of the
probability distribution for the output $\mathbf{Y}$. Because $\mathcal{E}$ is
defined by the analyst using $\widehat{\mathbb{P}}$, the information gained
from observing the potential outcome is similarly defined in terms of
$\widehat{\mathbb{P}}$. First, we introduce the quantitative characterization
of information in this context, which is equivalent to Shannon information
(Shannon, 1948). Then, we introduce properties of information in this context.

######  Definition 5 (Information gain from an analytic step).

Let $\mathbf{X}$ be a dataset and $T$ be any analytic tool that can be applied
to $\mathbf{X}$ to produce output $\mathbf{Y}=T(\mathbf{X})$. Let $S$ be any
subset of the complete potential outcome set $\mathcal{Y}$. Then, the
information gained by the analyst from observing the event $\\{\mathbf{Y}\in
S\\}$ in a single analytic iteration step is
$-\log\widehat{\mathbb{P}}(\mathbf{Y}\in S)$.

The information structure of an analytic step is introduced by the analyst via
the specification of $\mathcal{E}$ and $\mathcal{E}^{c}$ within $\mathcal{Y}$.
It follows from Section 4.2 Assumption 1 (namely complete potential outcome
set $\mathcal{Y}$ contains all outcomes) and the definitions of $\mathcal{E}$
and $\mathcal{E}^{c}$ in Sections 4.3-4.4 that the outcome $\mathbf{Y}$ must
fall into one of those two sets. If we observe the event
$\\{\mathbf{Y}\in\mathcal{E}\\}$, we say the result is as-expected; if we
observe the event $\\{\mathbf{Y}\in\mathcal{E}^{c}\\}$, we say the result is
unexpected. Note that without the specification of $\mathcal{E}$ and
$\mathcal{E}^{c}$, there is no possibility of information gain.

######  Theorem 1 (No information gain without specifying expected potential
outcome set).

Without any specification of the expected potential outcome set $\mathcal{E}$
(and the anomaly potential outcome set $\mathcal{E}^{c}$), the information
gained from an analytic step is $0$.

###### Proof.

If all that is specified is that $\mathbf{Y}$ can fall in the complete
potential outcome set $\mathcal{Y}$, then it follows from Assumption 2
(namely, $\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{Y})=1$) and Definition 5
that the information gained from that outcome is
$-\log\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{Y})=-\log 1=0$. ∎

By Definition 5, the information gained by the analyst upon observing
$\\{\mathbf{Y}\in\mathcal{E}\\}$ or $\\{\mathbf{Y}\in\mathcal{E}^{c}\\}$ is
then $-\log\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})$ or
$-\log\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E}^{c})$, respectively. The
next property shows that no matter what potential outcome set the output
$\mathbf{Y}$ ultimately falls in, there will be some non-zero information
gain.

######  Theorem 2 (Information positivity).

Upon specification of the expected potential outcome set $\mathcal{E}$ (and
the anomaly potential outcome set $\mathcal{E}^{c}$), the information gained
from an analytic step is strictly positive and finite.

###### Proof.

By Definitions 3 and 4, we know that
$0<\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})<1$ and
$0<\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E}^{c})<1$, respectively.
Therefore, it follows that
$-\log\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})>0$ and
$-\log\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E}^{c})>0$. ∎

###  5.2 Expected Information Gain

When evaluating different analytic tools to apply to the data, one
consideration is the amount of information that we expect to obtain from
applying the tool to the data regardless of whether the outcome is as-expected
or unexpected. First, we can define the observed information gained from
applying a tool $T$ to the data $\mathbf{X}$ in a single analytic step. Let
$\mathbf{X}$ be a dataset and $T$ any analytic tool to be applied to
$\mathbf{X}$ to produce output $\mathbf{Y}=T(\mathbf{X})$. Then the observed
information gained by the analyst after observing $\mathbf{Y}$ is

$G=\mathbf{1}\\{\mathbf{Y}\in\mathcal{E}\\}\left[-\log\widehat{\mathbb{P}}(\\{\mathbf{Y}\in\mathcal{E}\\})\right]+\mathbf{1}\\{\mathbf{Y}\in\mathcal{E}^{c}\\}\left[-\log\widehat{\mathbb{P}}(\\{\mathbf{Y}\in\mathcal{E}^{c}\\})\right].$

The value of $G$ cannot be computed until after $T$ has been applied to
$\mathbf{X}$ and the output $\mathbf{Y}$ has been observed, so it is not
useful for planning purposes. However, we can compute its expectation with
respect to $\widehat{\mathbb{P}}$, which is the average amount of information
gained from applying a tool with a given expected potential outcome set.

######  Definition 6 (Expected information gain from an analytic step).

Let $\mathcal{E}$ and $\mathcal{E}^{c}$ be the expected and anomaly potential
outcome sets, respectively, associated with a tool $T$ and dataset
$\mathbf{X}$. Let $\mathbf{Y}$ be the output obtained from applying $T$ to
$\mathbf{X}$. The expected information gained by the analyst from observing
the output $\mathbf{Y}$ is

$\displaystyle\widehat{H}$ $\displaystyle=$
$\displaystyle\mathbb{E}_{\widehat{\mathbb{P}}}[G]$ $\displaystyle=$
$\displaystyle\mathbb{E}_{\widehat{\mathbb{P}}}[\mathbf{1}\\{\mathbf{Y}\in\mathcal{E}\\}]\left[-\log\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})\right]+\mathbb{E}_{\widehat{\mathbb{P}}}[\mathbf{1}\\{\mathbf{Y}\in\mathcal{E}^{c}\\}]\left[-\log\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E}^{c})\right]$
$\displaystyle=$
$\displaystyle\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})\left[-\log\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})\right]+\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E}^{c})\left[-\log\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E}^{c})\right].$

We note that the expectation is taken with respect to $\widehat{\mathbb{P}}$,
which is the analyst’s estimate of the probability distribution for the
output, not the true distribution. Therefore, it is possible that the expected
information gain computed here will diverge from what might have been computed
under the true $\mathbb{P}$. We will discuss this divergence further in
Section 6.3.

###  5.3 Anomaly Information Gain

The expected information gain $\widehat{H}$ characterizes the average
information that would be obtained in a step regardless of whether the outcome
is as-expected or unexpected. However, it might also be worthwhile to consider
an alternate criterion that focuses on the information obtained solely when an
unexpected outcome is observed.

######  Definition 7 (Anomaly information gain from an analytic step).

Let $\mathcal{E}$ and $\mathcal{E}^{c}$ be the expected and anomaly potential
outcome sets, respectively, associated with a tool $T$ and dataset
$\mathbf{X}$. Let $\mathbf{Y}$ be the output obtained from applying $T$ to
$\mathbf{X}$. The anomaly information gained by the analyst from observing
output $\mathbf{Y}$ is

$\widehat{M}=-\log\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E}^{c}).$

It is useful to note that because we define $\mathcal{E}^{c}$ such that
$\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E}^{c})>0$ (Definition 4),
$\widehat{M}$ is always finite.

######  Theorem 3.

$\widehat{M}$ represents the maximum possible information gain that can be
obtained from applying the tool $T$ to the data $\mathbf{X}$.

###### Proof.

Because the $-\log$ function is monotonically decreasing on $(0,1)$ and
Definitions 3 and 4 restrict $\mathcal{E}$ so that
$\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E}^{c})<\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})$,
it follows that
$-\log\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E}^{c})>-\log\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})$.
∎

###  5.4 Example: Estimating the Sign of a Correlation

A common data analytic task is to determine whether two variables are
positively or negatively related to each other. For the sake of this example,
we do not consider this task as an inferential one, but rather as a
computational step that is perhaps part of a larger analysis. With two
(continuous) numerical variables $\mathbf{x}_{1}=x_{11},\dots,x_{1n}$ and
$\mathbf{x}_{2}=x_{21},\dots,x_{2n}$, we can compute a Pearson correlation
coefficient $r(\mathbf{x}_{1},\mathbf{x}_{2})$ and observe the sign of the
output value. In this case, the tool we have chosen is
$T=\text{sign}(r(\cdot,\cdot))$ and the output is
$\mathbf{Y}=\text{sign}(r(\mathbf{x}_{1},\mathbf{x}_{2}))$. The complete
potential outcome set for $\mathbf{Y}$ here is $\mathcal{Y}=\\{1,0,-1\\}$.

An analyst may expect that the two variables are positively related and so the
expected potential outcome set is $\mathcal{E}=\\{1\\}$ and the anomaly
potential outcome set is $\mathcal{E}^{c}=\\{0,-1\\}$. Furthermore, the
analyst may be quite confident about this problem and may assign the event
$\\{\mathbf{Y}\in\mathcal{E}\\}$ a probability of
$\widehat{\mathbb{P}}(\mathbf{Y}\in\mathcal{E})=0.95$. Note that for this
example we will use $\log_{2}$ so that $\widehat{H}$ ranges from 0 to 1. From
the analyst’s perspective, the expected information from this single analytic
iteration is $\widehat{H}=0.29$ bits. The expected information is relatively
low here given the perceived high likelihood of the result being as-expected.
Stated another way, if an analyst places high confidence on the event
$\\{\mathbf{Y}\in\mathcal{E}\\}$, then the expected information gain is small.
In contrast, the anomaly information gain in this scenario is
$\widehat{M}=4.3$ bits, which is the information gained in the event that the
observed sign is either $0$ or $-1$. Given the high confidence in the
correlation being positive, we gain a substantial amount of information from
observing an anomaly in this case.

##  6 Applications

The model for analytic iteration that we have developed in the previous
sections aims to provide a generalizable approach to characterizing a broad
range of data analytic steps that would be taken over the course of a data
analysis. These steps include data wrangling, cleaning or processing,
exploratory data analysis and visualization, and statistical modeling. The
approach we have taken parallels the traditional statistical paradigm where an
observed output is decomposed into a “model” predicted value plus a “residual”
value. However in our proposed framework, we have decomposed each analytic
step into expected and anomaly potential outcome sets. Next, we outline some
applications of this framework to formally characterize some common data
analysis activities.

###  6.1 Choosing Between Tools in Analytic Iteration

A fundamental problem that must be addressed at each step of an analytic
iteration is the question of what tool to apply to the data. This question may
be driven by the output from the previous iteration, which might have been as-
expected or unexpected. Assume that for the next step the analyst can choose
from a collection of tools informative $T_{j}$, $j=1,\dots,J$ to apply to the
data $\mathbf{X}$ in order to produce potential outcomes
$\mathbf{Y}_{j}=T_{j}(\mathbf{X})$. Given $T_{j}$ and $\mathbf{X}$, there is
an associated complete potential outcome set $\mathcal{Y}_{j}$ in which
$\mathbf{Y}_{j}$ can fall. Furthermore, the analyst can construct expected and
anomaly potential outcome sets $\mathcal{E}_{j}$ and $\mathcal{E}^{c}_{j}$
based on their own knowledge of $\mathbf{X}$ and $T_{j}$. The choice the
analyst must then make is between the available triples
$(T_{j},\mathcal{E}_{j},\mathcal{E}^{c}_{j})$, $j=1,\dots,J$.

In the previous section, we proposed two criteria—expected information gain
$\widehat{H}$, and maximum potential information gain $\widehat{M}$—as metrics
for characterizing the potential information gain from taking an analytic
step. We suggest here that these criteria could be used to prioritize
different triples $(T_{j},\mathcal{E}_{j},\mathcal{E}^{c}_{j})$ before
observing the outcome when analyzing data in practice. Here, we describe how
these criteria might be applied and how the different criteria may result in
different choices.

Let $\widehat{H}_{j}$ ($j=1,\dots,J)$ be the expected information gain
associated with the triple $(T_{j},\mathcal{E}_{j},\mathcal{E}^{c}_{j})$. This
is the expected information gained from applying $T_{j}$ to the dataset
$\mathbf{X}$ with the expected and anomaly potential outcome sets
$\mathcal{E}_{j}$ and $\mathcal{E}^{c}_{j}$. If $\widehat{\mathbb{P}}$ is the
analyst’s probability model for the data, then we have

$\widehat{H}_{j}=\widehat{\mathbb{P}}(\mathbf{Y}_{j}\in\mathcal{E}_{j})\left[-\log\widehat{\mathbb{P}}(\mathbf{Y}_{j}\in\mathcal{E}_{j})\right]+\widehat{\mathbb{P}}(\mathbf{Y}_{j}\in\mathcal{E}^{c}_{j})\left[-\log\widehat{\mathbb{P}}(\mathbf{Y}_{j}\in\mathcal{E}^{c}_{j})\right].$

Given $\widehat{\mathbb{P}}$ and the triple
$(T_{j},\mathcal{E}_{j},\mathcal{E}^{c}_{j})$, the expected information gain
$\widehat{H}_{j}$ can be computed before applying $T_{j}$ to the data.
Therefore, each of the $J$ triples
$(T_{j},\mathcal{E}_{j},\mathcal{E}^{c}_{j})$ can be evaluated based on
$\widehat{H}_{j}$ and the analyst could choose the triple
$(T_{j},\mathcal{E}_{j},\mathcal{E}^{c}_{j})$ that maximizes $\widehat{H}_{j}$
over $j=1,\dots,J$. If the anomaly information gain criteria were used, then
we could similarly define $\widehat{M}_{j}$ as the anomaly information gain
for triple $(T_{j},\mathcal{E}_{j},\mathcal{E}^{c}_{j})$, i.e.
$\widehat{M}_{j}=-\log\widehat{\mathbb{P}}(\mathbf{Y}_{j}\in\mathcal{E}^{c}_{j}).$
The analyst could then choose the triple
$(T_{j},\mathcal{E}_{j},\mathcal{E}^{c}_{j})$ that maximizes $\widehat{M}_{j}$
over $j=1,\dots,J$.

The definitions of $\widehat{H}_{j}$ and $\widehat{M}_{j}$ suggest that each
criterion is likely to favor different methods to apply to the data. The
expected information gain criterion ($\widehat{H}_{j}$) will favor method
triples that have $\widehat{\mathbb{P}}(\mathbf{Y}_{j}\in\mathcal{E}_{j})$
close to $0.5$ because $\widehat{H}_{j}$ is maximized when
$\widehat{\mathbb{P}}(\mathbf{Y}_{j}\in\mathcal{E}_{j})$ is exactly equal to
0.5 (although that is not allowed given the definition of $\mathcal{E}$). The
anomaly information gain ($\widehat{M}_{j}$) will favor triples where
$\widehat{\mathbb{P}}(\mathbf{Y}_{j}\in\mathcal{E}_{j})$ is very near $1$,
implying that $\widehat{\mathbb{P}}(\mathbf{Y}_{j}\in\mathcal{E}^{c}_{j})$ is
near zero and
$-\log\widehat{\mathbb{P}}(\mathbf{Y}_{j}\in\mathcal{E}^{c}_{j})$ is large. In
the next section, we attempt to characterize common data analysis scenarios in
which the different criteria might be more relevant for choosing an analytic
tool.

###  6.2 Exploratory vs. Confirmatory Analytic Steps

Maximizing the two information criteria we have specified here—expected
information gain $\widehat{H}$ and anomaly information gain $\widehat{M}$—can
lead to two different types of analytic steps when exploring the data. We
propose that maximizing expected information gain leads to exploratory
analytic steps while maximizing anomaly information gain leads to confirmatory
analytic steps.

In an exploratory situation, the analyst will likely have very little a priori
knowledge of the data generating mechanism. To the extent that the analyst has
hypotheses about how the data are generated, none of the hypotheses in the
hypothesis set is considered much more likely than any of the others. In this
scenario, the aim is to apply a tool that uses the data to learn as much as
possible, so that any outcome will provide a reasonably large information
gain. In a confirmatory situation, the analyst will have much more knowledge
about certain aspects of the data generating mechanism and will be more
confident in the output of certain analytic tools. Nevertheless, there may
still be value in producing such output in order to provide further support
for the analyst’s understanding. Should the output be unexpected in this
scenario, it would provide significant information to the analyst and may
motivate substantial follow up work.

As an example, consider the task of learning about the distribution of a
continuous one-dimensional random variable from a sample $X_{1},\dots,X_{n}$.
Initially, an analyst might intend to learn about the mean of the
distribution. If there were substantial previous information or preliminary
data, the analyst might be confident of the location of the mean, but would
seek to confirm it by computing the sample mean $\bar{X}$. Given the previous
information, the analyst might confident that the sample mean would fall in
the interval $\mathcal{E}_{\bar{X}}=[a,b]$ with probability
$\widehat{\mathbb{P}}(\bar{X}\in\mathcal{E}_{\bar{X}})=0.9$. An alternative
tool to investigate the underlying mean could be a graphical tool such as
histogram. However, because the histogram provides the location and the shape
of the distribution, the analyst might be less confident about all of the
details of the histogram. In this case the expected potential outcome set
might be $\mathcal{E}_{hist}=$ “an approximately symmetric histogram centered
near the interval $[a,b]$” with perhaps probability
$\widehat{\mathbb{P}}(\text{histogram}\in\mathcal{E}_{hist})=0.6$. Maximizing
anomaly information gain in this case would result in using the sample mean
because $\widehat{M}_{\bar{X}}=3.3$ bits while $\widehat{M}_{hist}=1.3$ bits.

Suppose that instead of investigating the mean of the distribution, the
analyst wants to investigate whether the distribution is skewed or symmetric
and has relatively little previous information about this aspect. One tool to
investigate this property would be the same histogram as before. Another quick
tool could be a comparison of the sample mean $\bar{X}$ and the sample median
$\tilde{X}$. If the sample mean and median were close to each other, that
might suggest rough symmetry in the distribution. Given previous knowledge
about the mean (and lack of specific information about skewness), the analyst
might also expect that $\mathcal{E}_{\tilde{X}}=[a,b]$ and that
$\widehat{\mathbb{P}}(\tilde{X}\in\mathcal{E}_{\tilde{X}})=0.9$. Maximizing
expected information gain in this case would lead to choosing the histogram
because $\widehat{H}_{hist}=0.97$ bits and $\widehat{H}_{\tilde{X}}=0.47$
bits.

This example highlights an important distinction between different analytic
tools, which is that analytic tools that have greater degrees of freedom and
can capture more features of the data are more difficult to specify in
advance. While it may be straightforward to suggest that a sample mean might
fall in a certain interval, it is more challenging to specify every aspect of
a histogram. We would therefore hypothesize that traditional exploratory data
analysis techniques, which tend to have greater degrees of freedom, would tend
to have expected potential outcome sets with lower probabilities from the
perspective of the analyst, and therefore tend to have larger expected
information gain.

The nature of exploratory analytic steps is to gain information regardless of
whether the outcome is as-expected or unexpected. Such a scenario favors
maximizing the expected information gain, even if the observed information
gain will be modest. Furthermore, there is generally little consequence for
observations being as-expected or unexpected. With confirmatory analytic
steps, we likely have much more previous knowledge of the data generating
process, and so our focus is on confirming what we already understand. In
addition, there may be more serious consequences to observing a result that is
outside of our expectations. Therefore, should a result be unexpected, we want
to ensure that the evidence that we generate is strong and unambiguous, so
that the observed information under an unexpected outcome is high. Ultimately,
determining which information criteria to use would require external knowledge
about the costs or benefits of as-expected and unexpected observations.

###  6.3 Evaluating Analytic Progress

In Section 5.2 we described two numerical quantities, the observed information
gain from an analytic step $G$, and the expected information gain
$\widehat{H}$ (from the analyst’s perspective). The value of $\widehat{H}$ is
entirely determined by the analyst’s estimate of the data generation
mechanism, while the value of $G$ is determined in part by the true underlying
data generation mechanism. Therefore, it may be of interest to compare the
observed $G$ after a sequence of analytic steps to the analyst’s stated value
of $\widehat{H}$, particularly if the analyst is using $\widehat{H}$ to guide
the choice of tools. If the analyst were to make their specification of
$\mathcal{E}$ and $\mathcal{E}^{c}$ known, one could compare the observed
information gain $G$ to the expected information gain $\widehat{H}$ to assess
the reasonableness of the analyst’s perspective. If
$\widehat{\mathbb{P}}\approx\mathbb{P}$, so that the analyst has a reasonable
estimate of the true probability distribution governing $\mathbf{Y}$, then $G$
and $\widehat{H}$ should be close on average. For a single iteration, any
deviation between $G$ and $\widehat{H}$ might be difficult to interpret.
However, with many steps executed over the course of an analysis, a pattern
may emerge of large or small deviations.

Consider a sequence of analytic steps $t=1,2,\dots$ where at each step the
analyst must specify a value for $\widehat{H}_{t}$ based on their current
understanding of the data generating mechanism and their choice of analytic
tool. Upon executing the analytic step we observe $G_{t}$. Over the course of
the sequence of analytic steps, we can compute the cumulative expected
information $S_{\widehat{H}}=\sum_{t}\widehat{H}_{t}$ and the cumulative
observed information gain $S_{G}=\sum_{t}G_{t}$. If the analyst has a good
understanding of the data generating mechanism, it might be expected that
$S_{G}$ and $S_{\widehat{H}}$ would be close.

Consider a scenario where an analyst is highly confident of the outcome and
predicts it with 95% probability. Should an anomaly be observed, so that
$\mathbf{Y}$ is observed to fall in the anomaly potential outcome set
$\mathcal{E}^{c}$, then this single observation might be attributed to random
chance. However, over the course many analytic steps, if anomalies were to
continue to occur, despite the analyst’s confidence, then we would expect that
$S_{G}$ would deviate substantially from $S_{\widehat{H}}$. In this scenario,
we could potentially use the deviation between $S_{G}$ and $S_{\widehat{H}}$
as a measure of how far off is the analyst’s understanding of the data.

###  6.4 Motivating the Next Step in Analytic Iteration

Determining the next step in any analytic iteration will typically require
general consideration of the high-level question being considered as well as
any immediate data-related phenomena being observed. One way that our model
can motivate the next step is through the classification of the output
$T(\mathbf{X})$ as as-expected or unexpected, depending on whether the output
fell in $\mathcal{E}$ or $\mathcal{E}^{c}$, respectively.

Unexpected results are in a sense more straightforward to consider because in
that situation the data directly contradict what was initially expected. A
logical next step might be for the analyst to diagnose this anomaly and
determine its cause, whether it be some unforeseen aspect of the data (e.g. an
outlier or missing data) or a problem with the analyst’s initial expectation.
The goal of such a diagnosis would be generate new hypotheses about the data
that could then be tested by applying new tools in a subsequent round of the
analytic iteration.

Results that are as-expected need to be handled with care because they confirm
what we already suspected to be true and the analyst may be encouraged to
accept the results and end the analysis. However, experienced analysts will
often exhibit skepticism in these situations and continue to interrogate the
data by checking various assumptions that are required for the conclusions to
be true. Similar to the situation with unexpected results, hypotheses can be
generated about the data that can be tested directly by applying different
tools. While the development of formal approaches to executing diagnostic or
skeptical thinking within analytic iteration is beyond the scope of this
paper, it may be possible to borrow techniques from other fields such as
systems engineering (e.g. Stamatelatos et al., 2002).

##  7 Discussion

Substantial previous work has described data visualizations as a tool in
exploratory data analyses that could be used to guide data analysts on how to
separate their expectations from what they observe in the practice of data
analysis (Tukey, 1977; Anscombe, 1973). In this paper, we extend the basic
idea of comparing expectations to observations in data visualizations to more
general analytic situations. Specifically, we propose a model for the
iterative process of data analysis based on the analyst’s expectations using
what we refer to as probabilistic outcome sets and statistical information
gain. Each analytic step induces a set of potential outcomes which is then
partitioned by the analyst into an expected potential outcome set and an
anomaly potential outcome set. These sets are then assigned probabilities
which can be used to quantify the information and evidence that can be
obtained from the analytic step.

It is important to note that the concept of developing expectations and
comparing them to observed data is fundamental to all of statistics and has
previously been applied in the context of data analysis (Grolemund & Wickham,
2014). Statistical models themselves are formal representations of an
analyst’s or scientist’s expectations for the data. However, the revolution in
data science in recent years has shined a light on the less formal, yet
critically important, aspects of data analysis that have yet to admit a
mathematical model. In fact, many have argued that data analysis is a skill
that can be learned only through experience (Greenhouse & Seltman, 2018). If
so, it should be unsatisfying to the statistical community that there is an
important part of our field that can only be learned by experiencing it. It is
only logical then to ask what is it that we learn through the experience of
doing data analysis and what can we generalize across our accumulated
experiences?

Here, we argue that analytic iteration, the process of developing
expectations, reconciling output with our expectations, and determining next
steps, is a generalizable aspect of data analysis that deserves more
attention. We propose a model for this cyclic process in order to provide some
formal structure to the analytic process without being prescriptive.
Furthermore, we introduce some language for describing this process when
teaching it to others. This is important and relevant because as the
popularity and importance of data science grows in all areas of society, the
role of data analysis will be highlighted. The benefit of developing a formal
characterization of the analytic process is a greater ability to scale
training as well as an opportunity to make advances and develop tools to a
improve the general quality of data analysis in all areas.

The framework we have presented here is a first step that admits a number of
new directions to explore. One such direction involves the process by which we
set up the next iteration. The end of each iteration produces a contrast
between the analyst’s expectations and the observed output. The analyst must
then react to that contrast and generate new ideas to pursue with the data.
While a formal framework for how to think about that process is not yet clear,
an obvious starting point would be to consider whether the output of the
iteration is unexpected or as-expected. Unexpected results, in particular, may
require an investigation within the data to determine their cause, and
developing tools to accelerate this process could be an interesting area of
future work. Another direction to explore could examine the minimal
requirements for stopping the iterative process. In practice, analytic
iterations may stop for a variety of reasons, but it may be possible to define
a more formal stopping criterion to ensure that specific goals are achieved or
a minimal amount of information is gained.

The area of data analysis is lacking in knowledge that can be applied in a
range of applications and analytic situations often have unique circumstances.
While our model of analytic iteration is just that—a model, the benefit of
having some approximation to this process is that it allows for the
development of new tools, approaches, or advice that can potentially be
applied broadly.

## References

  * (1)
  * Anscombe (1973) Anscombe, F. J. (1973), ‘Graphs in statistical analysis’, The american statistician 27(1), 17–21.
  * Breznau et al. (2022) Breznau, N., Rinke, E. M., Wuttke, A., Nguyen, H. H., Adem, M., Adriaans, J., Alvarez-Benjumea, A., Andersen, H. K., Auer, D., Azevedo, F. et al. (2022), ‘Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty’, Proceedings of the National Academy of Sciences 119(44), e2203150119.
  * Greenhouse & Seltman (2018) Greenhouse, J. B. & Seltman, H. J. (2018), ‘On teaching statistical practice: From novice to expert’, The American Statistician 72(2), 147–154.
  * Grolemund & Wickham (2014) Grolemund, G. & Wickham, H. (2014), ‘A cognitive interpretation of data analysis’, International Statistical Review 82(2), 184–204.
  * Kim & Hardin (2021) Kim, A. Y. & Hardin, J. (2021), ‘“Playing the whole game”: A data collection and analysis exercise with google calendar’, Journal of Statistics and Data Science Education 29(sup1), S51–S60.
  * Lovett & Greenhouse (2000) Lovett, M. C. & Greenhouse, J. B. (2000), ‘Applying cognitive theory to statistics instruction’, The American Statistician 54(3), 196–206.
  * Peng (2017) Peng, R. D. (2017), ‘Comment on “50 years of data science”’, Journal of Computational and Graphical Statistics 26(4), 767–767.
  * Peng et al. (2021) Peng, R. D., Chen, A., Bridgeford, E., Leek, J. T. & Hicks, S. C. (2021), ‘Diagnosing data analytic problems in the classroom’, Journal of Statistics and Data Science Education 29(3), 267–276.
  * Peng & Parker (2022) Peng, R. D. & Parker, H. S. (2022), ‘Perspective on data science’, Annual Review of Statistics and Its Application 9, 1–20.
  * Shannon (1948) Shannon, C. E. (1948), ‘A mathematical theory of communication’, The Bell System Technical Journal 27, 379–423.
http://plan9.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf

  * Stamatelatos et al. (2002) Stamatelatos, M., Vesely, W., Dugan, J., Fragola, J., Minarick, J. & Railsback, J. (2002), ‘Fault tree handbook with aerospace applications’.
  * Tukey (1977) Tukey, J. W. (1977), Exploratory Data Analysis, Vol. 2, Reading, MA.
  * Unwin (2001) Unwin, A. (2001), ‘Patterns of data analysis’, Journal of the Korean Statistical Society 30(2), 219–230.
  * Wild (1994) Wild, C. J. (1994), ‘Embracing the “wider view” of statistics’, The American Statistician 48(2), 163–171.
  * Wild & Pfannkuch (1999) Wild, C. J. & Pfannkuch, M. (1999), ‘Statistical thinking in empirical enquiry’, International statistical review 67(3), 223–248.