# Learning-From-Disagreement: A Model Comparison and Visual Analytics
Framework

Junpeng Wang, Liang Wang, Yan Zheng, Chin-Chia Michael Yeh, Shubham Jain, and
Wei Zhang J. Wang, L. Wang, Y. Zheng, C.-C. M. Yeh, S. Jain, and W. Zhang are
with Visa Research, Palo Alto, CA, 94301.
E-mail: {junpenwa, liawang, yazheng, miyeh, shubhjai,
<EMAIL_ADDRESS>received April xx, 20xx; revised August xx, 20xx.

###### Abstract

With the fast-growing number of classification models being produced every
day, numerous model interpretation and comparison solutions have also been
introduced. For example, LIME [1] and SHAP [2] can interpret what input
features contribute more to a classifier’s output predictions. Different
numerical metrics (e.g., accuracy) can be used to easily compare two
classifiers. However, few works can interpret the contribution of a data
feature to a classifier in comparison with its contribution to another
classifier. This comparative interpretation can help to disclose the
fundamental difference between two classifiers, select classifiers in
different feature conditions, and better ensemble two classifiers. To
accomplish it, we propose a learning-from-disagreement (LFD) framework to
visually compare two classification models. Specifically, LFD identifies data
instances with disagreed predictions from two compared classifiers and trains
a discriminator to learn from the disagreed instances. As the two classifiers’
training features may not be available, we train the discriminator through a
set of meta-features proposed based on certain hypotheses of the classifiers
to probe their behaviors. Interpreting the trained discriminator with the SHAP
values of different meta-features, we provide actionable insights into the
compared classifiers. Also, we introduce multiple metrics to profile the
importance of meta-features from different perspectives. With these metrics,
one can easily identify meta-features with the most complementary behaviors in
two classifiers, and use them to better ensemble the classifiers. We focus on
binary classification models in the financial services and advertising
industry to demonstrate the efficacy of our proposed framework and
visualizations.

###### Index Terms:

Learning from disagreement, model comparison, feature visualization, visual
analytics, explainable AI.

##  1 Introduction and Motivation

Classification, i.e., predicting the likelihood of given data instances to be
different categories, is a fundamental problem in machine learning (ML).
Numerous classification models have been proposed for it, including
conventional models [3], ensemble learning models [4], and deep learning
models [5]. The outstanding performance of these classifiers has made them
widely adopted in many real-world applications, e.g., spam filtering and
click-through rate (CTR) predictions. Often, for the same problem/application,
two or multiple models could be developed. Comparing them and identifying the
best one to use in a given context become an increasingly more important
problem.

Although many solutions have been proposed to interpret features’ importance
to a single model [1, 2] or compare two models’ prediction outcomes [6, 7],
few attempts try to tackle both problems simultaneously (Challenge 1), i.e.,
comparing two models by interpreting which feature in what condition is more
important to one model over the other. Considering the following scenario in
spam filtering, where ML practitioners need to choose between two spam
classifiers (A and B). Using LIME [1], they found the number of URLs in an
email (n_url) is an important feature to model A. Applying LIME to B, n_url is
also important. Comparing A and B with aggregated metrics (e.g., accuracy),
both show similar overall performance with small differences. Now, a new email
with a large n_url comes (target feature: n_url, condition: large), which
classifier will behave better? As one can see here, the interpretation of
individual models (i.e., LIME’s output) does not help to select models in this
scenario, as n_url is important to both. The small difference revealed by the
aggregated metrics is not sufficient to make any decisions either. We try to
address this challenge by comparatively interpreting two classifiers based on
their behavior discrepancy in different feature conditions, such that one can
select between them in the corresponding conditions.

Another challenge for ML practitioners is that they often do not have full
details of the compared models (Challenge 2). This is very common in industry,
as model training and comparison could be conducted by different teams. Fig. 1
(top) shows a general model development pipeline, where the training features
and architecture details of the two compared models (in gray) are not
available. Targeting to serve ML practitioners with sufficient knowledge on
their domain data, our work relies on users to propose new features to probe
the compared classifiers’ behaviors. We call these new features meta-features,
which can be generated based on: (a) the target application scenario, (b) the
general assumptions of the compared models (the two dashed arrows in Fig. 1).
The spam filtering scenario is a good example for case (a), where users are
interested in selecting models based on their behaviors on n_url. They can
then derive such a meta-feature, even though n_url may not be used when
training either of the two models. For case (b), although users have little
details of the compared models, they often have certain hypotheses on the
general behaviors of models in the same type. For example, if one of the
compared classifiers is an RNN, then sequence-related meta-features (e.g.,
sequence length) can be proposed to see if the RNN really behaves better on
instances with longer sequences. Note that, the meta-features here are
generated for interpretations, which differ from the training features (that
target to improve models’ performance but may be less interpretable). The
meta-features could overlap with the training features.

Figure 1: A general model training, comparison, and deployment pipeline (top).
Model-agnostic and feature-agnostic LFD comparisons (bottom).

Combining our ways of tackling the two challenges, we propose learning-from-
disagreement (LFD) (Fig. 1, bottom), a model-agnostic and feature-agnostic
framework to compare two classifiers. LFD works in three steps. First (Fig.
1a), it identifies the prediction disagreement from the two compared
classifiers (A and B) and uses it to label instances. For example, we can give
the instances that are captured (highly scored) by A but missed by B, denoted
as $\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}$, as negative. Similarly,
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}$ instances will have positive
labels. Second (Fig. 1b), LFD uses the meta-features of the disagreed
instances and their disagreement labels to train a binary classifier, which is
named the discriminator. This discriminator leverages the power of ML to
automatically learn which meta-features contribute more to the disagreement
between A and B. Third (Fig. 1c), we use SHAP [2] to conduct feature-based
interpretations on the trained discriminator. Our improved SHAP visualizations
and scalable meta-feature explorations in this step effectively disclose which
model behaves better in what feature conditions.

LFD helps to select models in given feature conditions and directly leads to a
better way to combine them. In the spam filtering case, if model A outperforms
B when n_url is larger and vice versa, then we can take scores from $A$ for
emails with large n_url and scores from $B$ for emails with small n_url to
generate a superior ensemble model. Feature-weighted linear stacking (FWLS)
[8], an advanced model ensembling approach, is based on this idea. It
dynamically weights up models that behave better in the current instances’
feature conditions and weights down the rest. However, when the number of
meta-features is large, FWLS becomes hard to train. The meta-features with
more complementary behaviors in the two models would ensemble them better,
thus should be considered more. To fill the current gap in comprehensive
feature-ordering, this work also proposes multiple metrics to rank the meta-
features.

To demonstrate the power of LFD and the efficacy of our feature-ordering
metrics, we conduct comprehensive case studies on two real-world problems with
front-line ML practitioners, (1) merchant category verifications and (2) CTR
predictions. Numerous ML classifiers are involved in the comparisons. The
insights gained from these studies, the quantitative results, and the feedback
from the domain users together validate the efficacy of our approach.

In summary, the contributions of our work include:

  1. 1.

We propose LFD and facilitate it with visual feature analysis to comparatively
interpret a pair of classifiers.

  2. 2.

We introduce multiple metrics to prioritize/rank a large number of meta-
features from different perspectives.

  3. 3.

We present multiple cases of using LFD and the corresponding visualizations to
better compare and ensemble a pair of classifiers from real-world
applications.

##  2 Related Works

Interpreting classification models is attracting increasingly more attention
in recent years and numerous solutions have been proposed [9, 10, 11, 12].
Roughly, model interpretations can be categorized into model-specific and
model-agnostic. Model-specific interpretations consider classification models
as “white-boxes”, where people have access to all internal details. For
example, most interpretations for deep learning models [13, 14, 15, 16, 17,
18, 19, 20, 21, 22] visualize and investigate the internal neurons’ activation
to disclose how data were transformed internally. Model-agnostic
interpretations regard predictive models as “black-boxes”, where only the
models’ input and output are available [23, 24, 25, 6, 26, 7]. These
approaches often employ an interpretable surrogate model to mimic or probe the
behavior of the interpreted models locally or globally. For example, LIME [1]
uses a linear model as a surrogate to simulate the local behavior of the
complicated classifier to be interpreted. DeepVID [27] trains an interpretable
model using the knowledge distilled from the original classifier for
interpretation. RuleMatrix [28] converts classification models as a set of
standardized IF-THEN-ELSE rules using only the models’ input-output behaviors.
For most of these interpretation solutions, an important goal is to answer
what input features are more important to the models’ output. There are also
solutions that statistically quantify features’ importance. One outstanding
example is the SHapley Additive exPlanation (SHAP) [2, 29], which we will
explain with details in Sec. 3.2. Our work is also model-agnostic (and even
feature-agnostic), i.e., we assume no knowledge of the compared models or
their training features. Moreover, we interpret a model’s behavior in contrast
to another model, which is more useful when selecting between them.

Comparing classifiers is often conducted through numerical metrics, e.g.,
accuracy and LogLoss. Multiple general model-building and visualization
toolkits, e.g., TensorFlow [30] and scikit-learn [31], provide built-in APIs
for these metrics. However, these aggregated metrics often fall short to
provide sufficient details in model comparison and selection, e.g., two models
could achieve the same accuracy in very different ways and the underlying
details are often of more interest when comparing them. Many visual analytics
works [32, 6, 33, 34] have tried to go beyond these aggregated metrics for
more comprehensive model comparisons. For example, Manifold [6] compares two
models by disclosing the agreed and disagreed predictions. The comparison is
model-agnostic, and for user-selected instances, Manifold can identify the
features contributing to their prediction discrepancy. DeepCompare[33]
compares deep learning models with incomparable architectures (i.e., CNN v.s.
RNN) through their activation patterns. CNNComparator [34] compares the same
CNN from different training stages to reveal the model’s evolution.
Deconvolution techniques have also been adopted to compare CNNs [35]. These
existing comparison works mostly rely on humans’ visual comprehension to
identify models’ behavior differences. In contrast, our LFD framework is a
learning approach, which leverages the power of ML to compare ML models.

Feature visualization in ML focuses either on (1) revealing what features have
been captured by predictive models, or (2) prioritizing features based on
their importance to limit the scope of analysis. The former is often conducted
on image data and a typical example is from Olah et al. [36], where the
authors used “visualization by optimization” to produce feature maps that
activate different neurons to interpret deep learning models. Different
saliency-map generation algorithms [18, 37, 38] also share the same goal of
highlighting the captured features to better understand deep neural networks.
The latter focus of feature prioritization is often conducted on tabular data,
where different metrics are used to order the contributions of different data
features [6, 39, 29, 40]. For example, when interpreting tree-based models,
the number of times that each feature was used to split a tree node is often
used to rank the features [41, 42, 43]. LIME [1] and SHAP [2, 29] also provide
quantitative metrics to order different data features. As the data of our
focused applications are in tabular format, we study more on the feature
ordering part in this work, and we contribute several new metrics (Sec. 6.2.2)
to prioritize features based on their complementary-level in two compared
classifiers.

##  3 Background and Motivation

Our objective is to (1) compare two classifiers; (2) reveal how different
features behave in them; and (3) use the insights from (1) and (2) to better
ensemble the models. This section explains the necessary background for these
three topics.

###  3.1 Comparing Classification Models

Comparing two classifiers is a very common and frequent operation in industry.
For example, replacing an old production model with a new model often comes
with significant business impacts. Therefore, thorough studies need to be
conducted beforehand to compare the two models. These studies are often
referred to as “swap analysis”, deciding whether the old model should be
replaced/swapped by the new one. We focus on this application scenario in this
work and thus target on the comparison between two models.

To date, most of the model comparisons are limited to numerical metrics, e.g.,
the area under the receiver operating characteristic (ROC) curve (AUC) and
LogLoss. However, these metrics cannot tell in what feature conditions one
model outperforms the other, which is a critical question for model selections
(e.g., the spam filtering scenario in Sec. 1). Furthermore, as very limited
information is available during comparisons, model-agnostic and feature-
agnostic comparisons are often more preferable.

###  3.2 Feature Analysis with SHAP

Feature analysis is to reveal how an input feature impacts the output of a
classifier and the magnitude of the impact. We use one of the most popular
solutions, i.e., SHAP, for this analysis, as it is consistent and additive [2,
29, 43].

A tabular dataset with $n$ instances and $m$ features can be considered as a
matrix of size $n{\times}m$. Fig. 2a (left) presents a fake dataset for a spam
classifier. The dataset has 5 instances (emails), each with 3 features: the
word count ($n\\_wd$), the number of URLs ($n\\_url$), and the count of
numerical values ($n\\_num$) in an email. To interpret the classifier’s
behavior on this dataset, SHAP generates an interpretation matrix, which also
has the size of $n{\times}m$ (Fig. 2a, right). Each element of this matrix,
$[i,j]$, denotes the contribution of the $j$th feature to the prediction of
the $i$th instance. In other words, the sum of all values from the $i$th row
of the SHAP matrix is the classifier’s final prediction for the $i$th instance
(i.e., the $log(odds)$, which will go through a $sigmoid()$ to become the
final probability value). The above explanation considers SHAP as a black-box,
as the detail of generating the SHAP matrix is out of the scope of this paper.

The SHAP summary-plot [29] (Fig. 2b, read labels in the blue boxes) is
designed to visualize the effect of individual features to a classifier’s
prediction. For example, to show the impact of $n\\_url$ to the spam
classifier, the summary-plot encodes each email as a point. The color and
horizontal position of the point reflect the corresponding feature- and SHAP-
value respectively. The five points with black stroke in Fig. 2b show the five
emails in Fig. 2a. Extending the plot to more emails/instances, we can
conclude the feature’s impact from the instances’ collective behaviors, i.e.,
emails with more URLs (the red points with larger positive SHAP) are more
likely to be spam. The impact of other features can be visualized and
vertically aligned with this feature (along the $SHAP{=}0$ line, see Fig. 7).
These features will be ordered based on their importance to the classifier,
which is computed by $mean(|SHAP|){=}\frac{1}{n}\sum_{i=1}^{n}|SHAP_{i}|$.

Figure 2: (a) Interpreting a classifier’s behavior on a dataset with SHAP will
generate an interpretation matrix sharing the same size with the input. (b)
Visualizing the contribution of a feature using the summary-plot [29] (read
labels in the blue boxes). The collective behavior of all instances reflects
that emails with higher n_url are more likely to be spam. The summary-plot can
also interpret the discriminator from LFD (read labels in the green boxes),
i.e., emails with higher n_cap (a meta-feature) are more likely to be captured
by model B but missed by A (i.e.,
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}$).  Figure 3: LFD compares two
classifiers (A & B, Step 1) by extracting data instances captured by each
classifier ($\text{{A}}^{\textit{+}}$ and $\text{{B}}^{\text{+}}$, Step 2),
and joining them into both-captured
($\text{{A}}^{\text{+}}\text{{B}}^{\text{+}}$) and captured-only instances
($\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}$ and
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}$, Step 3). Based on the data
labels, the three sets of instances can further be divided into true-positive
and false-positive groups (Step 4, see the six cells’ legend at the bottom-
right corner). For each group, a discriminator (binary classifier) is trained,
considering $\text{{A}}^{\text{-}}\textit{\text{B}}^{\text{+}}$ and
$\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}$ as positive and negative samples,
and using a set of meta-features (Step 5). Interpreting the discriminator
through SHAP (Step 6), we derive insights to effectively interpret which
classifier prefers which features in what value ranges.

###  3.3 Model Ensembling

Model ensembling [44] is the research of combining multiple pre-trained models
to achieve better performance than individuals. The naïve way of ensembling
two models is to train a linear regressor to fit scores from the two pre-
trained models, which is known as linear stacking [45]. Considering the data
instance $x$ and two pre-trained models $M_{a}$ and $M_{b}$, the linear
stacking result $LS(x)$ is computed by:

$\footnotesize LS(x)=w_{1}{\cdot}M_{a}(x)+w_{2}{\cdot}M_{b}(x).$ (1)

Feature-weighted linear stacking (FWLS) [8], a state-of-the-art model
ensembling solution, claims that the weights ($w_{1}$ and $w_{2}$ in Eq. 1)
should not be fixed but vary based on different feature-values. Because the
two models may have varying performances in different feature value ranges.
FWLS (Eq. 2), therefore, combines data features with models’ scores by
feature-crossing and trains the regressor on the crossed features. The weights
$w_{1}$ and $w_{2}$ become $m$ pairs of weights ($w_{1}^{i}$ and $w_{2}^{i}$
in Eq. 2), where $m$ is the number of features and $F_{i}(x)$ retrieves the
$i$th feature value of $x$.

$\scriptsize
FWLS(x)=\sum_{i=1}^{m}[w_{1}^{i}{\cdot}(M_{a}(x){\cdot}F_{i}(x))+w_{2}^{i}{\cdot}(M_{b}(x){\cdot}F_{i}(x))].$
(2)

As the number of features may be large, FWLS can be conducted using a subset
of the features. However, incautiously selected features may impair the
ensembling result. Therefore, properly ranking the features and choosing the
most important ones for ensembling become a critical problem. We propose
different metrics to rank features in Sec. 6.2.2.

##  4 Learning-From-Disagreement (LFD)

This section concretizes LFD into six execution steps (Fig. 3). Step 1$\sim$4
identify the disagreement (Fig. 1a). Step 5 trains a discriminator to learn
from the disagreement based on user-proposed meta-features (Fig. 1b). Step 6
interprets the discriminator to answer which model behaves better in what
feature conditions (Fig. 1c). In detail, the six steps are:

  1. 1.

Feed data into the compared classifiers (A & B) and get the two classifiers’
scores for individual data instances.

  2. 2.

Sort instances by the two sets of scores decreasingly, and set a threshold as
the score cutoff (e.g., 5% of all instances). Instances with scores above it
are those captured by individual models ($\text{{A}}^{\text{+}}$ and
$\text{{B}}^{\text{+}}$). The threshold often depends on applications, e.g.,
for loan eligibility predictions, it will be decided by a bank’s budget.

  3. 3.

Join the two sets of captured instances (i.e., $\text{{A}}^{\text{+}}$ and
$\text{{B}}^{\text{+}}$) into three cells of the disagreement matrix, i.e., A
captured B missed ($\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}$), A missed B
captured ($\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}$), and both captured
($\text{{A}}^{\text{+}}\text{{B}}^{\text{+}}$). For comparison purposes, the
$\text{{A}}^{\text{+}}\text{{B}}^{\text{+}}$ instances are of less interest.
Also, we do not have $\text{{A}}^{\text{-}}\text{{B}}^{\text{-}}$ instances,
since the filtered instances from Step 2 are at least captured by one model.

  4. 4.

Based on the true label of the captured instances, we divide the disagreement
matrix into two matrices: one for the TP instances (i.e., correctly captured),
the other for the FP instances (i.e., mis-captured).

  5. 5.

Train the TP and FP discriminators (two binary classifiers) to differentiate
the $\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}$ (negative) and
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}$ (positive) instances from the TP
and FP sides, respectively. The training uses a set of meta-features, as the
features of model A and B may not be available.

  6. 6.

Interpret the trained discriminators with SHAP to provide insights into the
fundamental difference between A and B. The insights also help to rank the
meta-features and pick the best ones to ensemble A and B.

Meta-features came into the picture, as the features used to train classifier
A and B may not be available during comparison (i.e., our second motivating
challenge in Sec. 1). Nevertheless, if the training features are readily
available, they can directly work as meta-features. For example, in Fig. 2a,
$n\\_wd$, $n\\_url$, and $n\\_num$ are three features derived from raw emails
(email title, address, and body) for model training. They can also work as
meta-features if users are interested to know how the compared models will
behave on them (e.g., which model performs better when $n\\_url$ is large?).
In cases where the training features are not available, users can derive meta-
features based on their interested application scenario. For example, in Fig.
2b, $n\\_cap$ (the number of capitalized words) is proposed to probe how the
compared spam classifiers would be impacted by this meta-feature (though this
feature was not used to train either of the classifiers). Meta-features can
also be derived based on the type of the compared models. For example, when
comparing RNNs with tree-based models, one can generate sequence-related meta-
features to verify if the RNNs really benefit from being aware of sequential
behaviors. To compare GNNs with RNNs, one can propose neighbor-related meta-
features (e.g., nodes’ degree) to reveal how much the GNNs can take advantage
of neighbors’ information. As LFD targets to serve users with sufficient
knowledge on their domain data, generating a sufficient number of meta-
features to train the discriminator is usually not a bottleneck.

The discriminator can be any binary classifier, as long as it is SHAP-friendly
(this work uses XGBoost [42]). Based on the SHAP of different meta-features,
we provide insights into the two compared classifiers. For example, if the
discriminator shows the SHAP view in Fig. 2b (use labels in the green boxes),
we can conclude that “compared to A, classifier B tends to capture emails
containing more capitalized words ($n\\_cap$ is a meta-feature)”. If the
discriminator is the TP discriminator, we know that B outperforms A when this
feature is large (correctly captures more). However, if the discriminator is
the FP discriminator, B is not as good as A, as it mis-captures more when the
feature value is large.

LFD has the following direct advantages:

  * •

Model-Agnostic: LFD needs only the input and output from the two compared
models, making it a generally applicable solution to compare any types of
classifiers.

  * •

Feature-Agnostic: LFD compares classifiers based on newly proposed meta-
features, which are independent of the original model-training features.

  * •

Avoiding Data-Imbalance: For many real-world applications (e.g., anomaly
detections and CTR predictions), data-imbalance, i.e., positive instances are
much fewer than negative instances, brings a big challenge to model training
and interpretation. LFD smartly avoids this, as it compares the difference
between two models, i.e., the two “captured-only” cells usually have similar
sizes.

##  5 Domain Experts and Design Requirements

We worked with eight ML experts, specialized in various classification models,
to study the model comparison problem. All are ML researchers and front-line
model designers in an industrial research lab. Among them, five are research
scientists with model training and feature engineering experience for 5+
years. The other three are senior researchers with decades of experience on
predictive models. Two experts ($E_{1}$, $E_{2}$) intensively participated in
the design of LFD, as the framework was originated from their desire to help
the “swap analysis” between a to-be-launched model and a production model. Two
experts ($E_{3}$, $E_{4}$) helped to prepare the transaction/CTR data and the
ML models we used in Sec. 7, but did not use our LFD system until we conduct
case studies with them. These four experts co-authored this work. The rest
four experts ($E_{5}{\sim}E_{8}$) participated in the usability studies only
to provide objective feedback.

While working with $E_{1}$ and $E_{2}$, we identified multiple requirements
for LFD. For example, they mentioned that knowing the distribution of the
disagreed instances is where they want to get started. When building different
models and analyzing features’ contributions, accurate feature importance
order is crucial. Through intensive discussions in multiple iterations, we
distilled the following requirements focused on model comparison ( MC) and
feature analysis ( FA):

  * •

MC1

Present the distribution of data instances across the six cells of the two
disagreement matrices (Step 4 of LFD) under different thresholds (Step 2 of
LFD).

  * •

MC2

Interpret the TP and FP discriminators to explain the impact of different
meta-features on the captured and mis-captured instances (Step 6 of LFD).

  * •

MC3

Be able to compare the contribution of the same meta-feature to the TP and FP
discriminators, and interpret the difference with as little as possible
information.

  * •

FA1

Disclose the behavior difference of the two compared classifiers on a user-
proposed meta-feature.

  * •

FA2

Effectively rank meta-features from different perspectives and use the more
important ones (e.g., more complementary ones) to improve model ensembling.

Figure 4: (a1-a8) The Disagreement Distribution View presents the size of the
six cells (i.e., $\text{{A}}^{\text{+}}\text{{B}}^{\text{+}}$,
$\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}$, and
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}$ from both the TP and FP sides,
Step 4 of LFD) across different thresholds. The Feature View presents the
important meta-features to the TP and FP discriminators as two feature lists
(b5). Users can select a meta-feature to check its details (b2, b3). (b6)
shows the ranks of the selected meta-feature in different metrics.

##  6 Visual Interface

Following the design requirements, two parts of LFD need to be visually
presented: (1) the disagreement matrices under different cutoffs (Step
2$\sim$4) and (2) the meta-features and their SHAP values (Step 6). These two
parts require humans’ interactions and visual interpretations on the LFD
outcomes. We present them with the Disagreement Distribution View and the
Feature View respectively, in two subsections.

###  6.1 Disagreement Distribution View

The Disagreement Distribution View (Fig. 4, left) is designed to meet MC1.
Given a threshold at Step 2 of LFD, we filter out two sets of instances
captured by model A and B respectively. These instances are then joined into
three cells at Step 3, i.e., $\text{{A}}^{\text{+}}\text{{B}}^{\text{+}}$,
$\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}$, and
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}$. The height of red bars in Fig.
4-a1 denotes the size of the three cells under the current threshold. When
changing the threshold, the size of the three cells will change accordingly.
Thus, we show a sequence of green bars inside each cell, denoting the cell
sizes across all possible thresholds (i.e., the horizontal axis of each bar
chart represents different thresholds and the vertical axis denotes the number
of instances). This distribution overview guides users to select a proper
threshold to maximize the size of disagreed data for learning.

To better encode the data joining process, we use a white and a gray
background rectangle to represent the instances captured by model A and B
respectively, i.e., $\text{{A}}^{\text{+}}$ and $\text{{B}}^{\text{+}}$ (Fig.
4-a2). Their overlapped region reflects the instances captured by both
($\text{{A}}^{\text{+}}\text{{B}}^{\text{+}}$) and the non-overlapped regions
on the two sides are the instances captured by one only
($\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}$ and
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}$). The width of the rectangles
(and the overlapped region) is proportional to the number of instances.

Coming to Step 4 of LFD, the disagreement matrix in Fig. 4-a2 is divided into
two matrices when considering the label of the instances. Meanwhile, we rotate
the three green bar-charts 90 degrees and present the current threshold value
in the middle of the two disagreement matrices. In Fig. 4-a4, the threshold is
15% and the corresponding bars in the six disagreement matrix cells are
highlighted in red, Fig. 4-a3 (TP side), Fig. 4-a5 (FP side).

The two triangles in Fig. 4-a4 help to flexibly adjust the threshold (Step 2
of LFD). The current threshold is 15% and the cutoff scores for the two
compared models, Tree (A) and RNN (B), are 0.2516 and 0.2682 respectively.
Instances with respective scores larger than these are captured by individual
models. When increasing the threshold to 50% (Fig. 4-a7), the overlapped cell
becomes larger but the two “captured-only” cells become smaller (Fig. 4-a6,
4-a8). The white and gray rectangles will be overlapped completely, when the
threshold becomes 100% (i.e., both models consider all instances as captured).
Note that, the Disagreement Distribution View is hidden from the interface by
default, but can be enabled by clicking the button in Fig. 4-b1.

We have also explored several alternative designs to present the disagreement
matrices. For example, we tried to lay out the three cells in the way they are
placed in Fig. 4-a1, initially. The layout is intuitive, but it is not space-
efficient, as the bottom-right corner is always empty. Although our current
design may not be the best choice, it sufficiently meets MC1. The ML experts
commented that the design is easy to understand, and they like the way we
metaphorize the joining process through two overlapped rectangles, and
inherently, encode the cell size into the width of the overlapped/non-
overlapped regions.

###  6.2 Feature View

The Feature View (Fig. 4, right) serves Step 6 of LFD by presenting the meta-
features from both the TP and FP discriminators through an “Overview+Details”
exploration. It is designed to meet MC2 and MC3, and the design starts from
the traditional summary-plot [29] (Fig. 2b), where we found two limitations of
the plot on its visualization accuracy (Sec. 6.2.1) and feature importance
order (Sec. 6.2.2).

####  6.2.1 Visualization Accuracy

The accuracy of the summary-plot could be undermined if the number of
instances is large. As explained in Fig. 2, each instance is represented as a
point. Within the limited space, numerous points will be overplotted and
stacked vertically, which may result in misleading visualizations. Fig. 5a
shows a real example (generated using the Python package shap). At first
glance, it seems large feature values (the red points on the right of the
dashed line) always contribute positively to the prediction. However, there
are also red points being plotted under the blue ones on the left but are less
visible.

Figure 5: (a) The overlap of points in a summary-plot may cause misleading
visualizations. (b) The visualization on right [43] avoids overplotting.

This overplotting issue has been discovered by Wang et al. [43], and they
fixed it by visualizing data distributions as bubbles, rather than individual
instances as points. As shown in Fig. 6a, Wang et al. [43] construct a 2D
histogram of feature- and SHAP-values (i.e., based on the two matrices in Fig.
2a). Non-empty cells of the histogram are represented with bubbles whose size
denotes the number of instances. These bubbles are then packed along the
$x$-axis (without overlap) based on their SHAP values, using circle packing or
force-directed layouts [46, 47, 43] (Fig. 6b). In detail, the circle packing
algorithm sequentially places bubbles in tangent to each other while striving
to maintain their $x$-position [46]. Similarly, the force-directed algorithm
resolves the overlap among bubbles by iteratively adjusting the bubbles’
position, while also trying to retain their $x$-position. The final
visualization (Fig. 5b) resolves the overplotting issue (e.g., many large
value instances on the left are visible now). Note that, the number of bubbles
is bounded by the product of the number of feature-bins and SHAP-bins in the
2D histogram (Fig. 6a). Therefore, one can control the number of bubbles by
adjusting the number of bins.

The new design, however, cannot accurately reflect the data distribution for
two reasons. First, it cannot guarantee to position a bubble at its exact
$x$-position, as circles may get shifted to be packed tightly. Second, the
size mapping between the number of instances and the bubble size can also
cause issues. For example, to make sure bubbles are not too big or too small,
size clipping is often applied. If the biggest bubble represents $100$
instances, bubbles with $1000$ instances will have the same size and the
accumulated area of bubbles cannot accurately reflect the data distribution.
To fix this, we draw the data distribution as a set of horizontally stacked
rectangles (Fig. 6c), whose height accurately reflects the number of instances
in the corresponding SHAP bin (Fig. 6-c1). The color of a rectangle is blended
through a weighted-sum of all bubbles in the SHAP bin (Fig. 6-c2). Increasing
the granularity of the SHAP bins, we get the final area-plot visualization for
one feature, e.g., Fig. 4-b2.

To guarantee interpretation accuracy ( FA1) and enable interactivity, we
employ both designs in Fig. 6b, 6c through an “Overview+Details” design. All
meta-features from the TP and FP discriminators are presented as two columns
of area-plots for an overview, where the same features are connected across
columns (Fig. 4-b5, MC3). When clicking a meta-feature, its details will be
shown using the bubble-plot (Fig. 4-b3). We also improve the color legend as
an interactive “transfer function”. Users can add/drag/delete the control-
points on the legend to change the color mapping. Brushing the legend will
select bubbles with feature values in the brushed range (an example is shown
in Fig. 11c).

Figure 6: Visualizing the 2D histogram between feature- and SHAP-values (a)
through a group of packed bubbles [43] (b) or an area-plot (c).

####  6.2.2 Feature Importance Order

In real-world applications, it is very common to work with hundreds of
features. Effectively identifying the important ones becomes critical ( FA2).
The summary-plot prioritizes features by $mean(|SHAP|)$ [29], which is an
effective metric in reflecting the Magnitude of features’ SHAP values, i.e.,

$\scriptsize Magnitude{=}\frac{1}{n}\sum_{i=1}^{n}|SHAP_{i}|,\ \ n\ is\ the\
number\ of\ instances.$ (3)

However, we found this metric often fails to bring the most important features
up, as an example shown in Fig. 7. Feature $F_{1}$ has a larger absolute
magnitude than $F_{2}$, as the points are distributed more widely in the
horizontal direction. However, the contribution of $F_{1}$ is less consistent
than $F_{2}$. For example, within the two dashed lines, both small and large
$F_{1}$ values (blue and red points) could have positive contributions
($SHAP{>}0$) to the final prediction. In contrast, only large $F_{2}$ values
contribute positively. Fig. 8a shows a real large-magnitude feature. Although
the feature’s contribution magnitude is large, its contribution is not
consistent, as the blue and orange bubbles are mixed within any horizontal
range. Apparently, measuring features’ importance by their magnitude only is
not sufficient.

To accommodate the above issue, we propose the Consistency metric. It is
computed by (1) calculating the entropy of the feature-values in each SHAP bin
(a column of cells in Fig. 6a), (2) summing up the entropy from all SHAP bins,
using the number of instances in each bin as the weight, (3) taking the
inverse of the sum value. Mathematically,

$\scriptsize
Consistency{=}\Big{(}\frac{1}{n}\sum_{i=1}^{m}\pazocal{H}\big{(}F(x),\forall
x{\in}bin_{i}\big{)}{\times}|bin_{i}|\Big{)}^{-1},\
n{=}\sum_{i=1}^{m}|bin_{i}|,$ (4)

where $F(x)$ retrieves the feature-value of data instance $x$, $|bin_{i}|$
denotes the size of a bin, $\pazocal{H}()$ computes the entropy, and $m$ is
the number of SHAP bins. Fig. 8b presents a real feature with high
consistency, reflected by the homogeneous color of bubbles within different
SHAP bins (horizontal ranges). However, this feature is not very useful.
Because small values (in blue) are largely distributed on both sides of the
$SHAP{=}0$ line, and could contribute both positively and negatively.
Consequently, solely relying on the consistency metric cannot effectively
identify important features either.

Figure 7: $F_{1}$ is more important than $F_{2}$ as its magnitude is larger.
However, $F_{2}$ is more consistent and more contrast compared to $F_{1}$.
Figure 8: Important features identified by different metrics: (a) Magnitude;
(b) Consistency; (c) Contrast; and (d) Correlation.

To capture features with clear contribution differences, we propose the
Contrast metric, which is computed by the Jensen-Shannon divergence (JSD [48])
between two normalized distributions formed by the feature-values with
positive and non-positive SHAP values, i.e.,

$\scriptsize Contrast{=}JSD\Big{(}\pazocal{D}\big{(}F(x)|SHAP_{x}{<=}0\big{)}\
{||}\ \pazocal{D}\big{(}F(x)|SHAP_{x}{>}0\big{)}\Big{)}$ (5)

where $\pazocal{D}()$ denotes the operation of forming a normalized
distribution from a set of values. Fig. 8c demonstrates a typical contrast
feature. This feature is a very good one as it has very clear feature
contributions. Large feature values (orange bubbles) always contribute
positively and small values (blue bubbles) always contribute negatively.

Lastly, we also compute the absolute Pearson Correlation between the SHAP-
values and feature-values. This metric further enhances the contrast metric by
revealing if the feature-values are linearly correlated with their
contributions or not. Fig 8d shows a feature with a large correlation. Smaller
feature-values (in darker blue) contribute more positively to the prediction
and the contribution is roughly monotonic (i.e., from left to right, the color
changes from dark orange, light orange, light blue, to dark blue).

We claim that features’ importance should be evaluated from multiple
perspectives, and thus, integrate the four metrics with a weighted-sum to
generate the fifth metric, i.e., Overall (Eq. 6). Note that, the weights here
are derived based on preliminary studies with the Avazu dataset [49]. However,
they could be different for different datasets.

$\scriptsize
Overall{=}2{\times}Magnitude{+}Consistency{+}2{\times}Contrast{+}Correlation$
(6)

The TP and FP discriminators have the same set of meta-features, and they are
presented as two columns, which can be easily ordered by the five metrics
(Fig. 4-b5, FA2). The same features across columns are linked by a curve for
tracking ( MC3). The visualization in Fig. 4-b6 demonstrates the ranks of the
selected meta-feature in all five metrics.

##  7 Case Studies

This section demonstrates cases of using LFD in two real-world applications,
(1) merchant category verifications from the payment industry; (2) CTR
predictions for advertising. The case studies were conducted with the eight ML
experts introduced in Sec. 5. We used the “guided exploration + think-aloud
discussions” as the protocol to conduct the studies in three steps, (1)
explaining the design goals of LFD and our visualizations; (2) guiding the
experts to explore the cases, our system, and explain the corresponding
findings; (3) open-ended interview to collect their feedback (Sec. 8.3).

###  7.1 Models for Merchant Category Verification

In the financial services industry, each registered merchant has a category
reflecting its service type, e.g., restaurants, pharmacies, and casino gaming.
The category of a merchant may get misreported for various reasons, e.g.,
high-risk merchants may report a fake category with lower risk to avoid high
processing fees [50]. Therefore, it is important to verify the category
reported by each merchant. The credit card transactions of a merchant,
depicting the characteristic of its provided service, are often used to solve
this problem.

For simplicity, we compare binary classifiers for restaurant verification only
in this work (positive: restaurant, negative: non-restaurant). Four
classifiers have been introduced by the ML experts. Their details are as
follows.

  * •

The Tree model is an XGBoost, which consumes data in tabular format (rows:
merchants, columns: features).

  * •

The CNN takes sequential features of individual merchants as input (each
sequence denotes the values of a merchant’s feature across time). It captures
temporal behaviors through 1D convolutions in residual blocks.

  * •

The RNN also captures the merchants’ temporal behaviors, but through gated
recurrent units (GRUs).

  * •

The GNN takes both the temporal and affinity information of a merchant into
consideration. The temporal part is managed by 1D convolutions, whereas the
affinity part is derived from a graph of merchants. Two merchants are
connected if at least one card-holder visited both, and the strength of the
connection is proportional to the number of shared card-holders. A GNN is then
built to learn from this weighted graph of merchants.

More details on the models’ architectures can be found from [50]. However, LFD
does not need these details since it is model-agnostic. From the training data
(raw transactions), the four models use different feature extraction methods
to derive their respective training features in different formats, e.g., some
are in tabular form and some are in sequences. Even we have little knowledge
about the features used by different classifiers, we can still compare them
with LFD (i.e., feature-agnostic). For the comparisons, we have $\sim$3.8
million merchants and their raw transactions in 2.5 years.

Since the experts build these models, they have sufficient knowledge on them.
We compare the models using LFD and verify the derived insights with the
experts for sanity checks in Sec. 7.1.1. Sec. 7.1.2 presents new insights to
deepen the experts’ understanding of the models.

TABLE I: Using LFD to compare the Tree and RNN models (top row), the RNN and GNN models (bottom row) for sanity checks (see details in Sec. 7.1.1). The action items at each step start with “$\triangleright$”, and the output from each step (which serves as the input to the next step) starts with “$\Rightarrow$”. Step 1 |  Step 2 |  Step 3 |  Step 4 |  Step 5 |  Step 6
---|---|---|---|---|---
$\triangleright$ A: Tree $\triangleright$ B: RNN $\Rightarrow$scores |  $\triangleright$ sort merchants by their scores from A and B $\triangleright$ filter out highly scored ones with a threshold $\Rightarrow$two sets of merchants, i.e., $\text{{A}}^{\text{+}}$ and $\text{{B}}^{\text{+}}$ |  $\triangleright$ join the two sets of merchants into three cells of the disagreement matrix $\Rightarrow$three sets of merchants, i.e., $\text{{A}}^{\text{+}}\text{{B}}^{\text{+}}$, $\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}$, $\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}$ |  $\triangleright$ divide each set of merchants into two subsets based on merchants’ true label $\Rightarrow$six sets of merchants, i.e., $\text{{A}}^{\text{+}}\text{{B}}^{\text{+}}\text{(TP)}$, $\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}\text{(TP)}$, $\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}\text{(TP)}$, $\text{{A}}^{\text{+}}\text{{B}}^{\text{+}}\text{(FP)}$, $\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}\text{(FP)}$, $\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}\text{(FP)}$ |  $\triangleright$ generate meta-features $\triangleright$ use $\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}\text{(TP)}$ and $\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}\text{(TP)}$ to train the TP discriminator $\triangleright$ use $\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}\text{(FP)}$ and $\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}\text{(FP)}$ to train the FP discriminator $\Rightarrow$TP and FP discriminators |  $\triangleright$ visualizations and insights $\Rightarrow$Fig. 9
$\triangleright$ A: RNN $\triangleright$ B: GNN $\Rightarrow$scores |  $\triangleright$ visualizations and insights $\Rightarrow$Fig. 10

####  7.1.1 Sanity Checks: [Tree v.s. RNN] and [RNN v.s. GNN]

We first compare the Tree (A) and RNN (B). The Tree has a higher AUC (it
outperforms the RNN), but the RNN has a lower LogLoss (the RNN outperforms the
Tree). The performance reflected by the two metrics conflicts with each other,
and it is hard to choose the models based on them. Also, the metrics reveal
nothing about the models’ behavior discrepancy on different features and
feature conditions.

Tab. I (top row) shows the LFD comparison process for the two models. Step 1
feeds the $\sim$3.8 million merchants to the two models and generates two
scores for each merchant. Step 2 uses these scores to sort the merchants
decreasingly. Based on a given threshold, we filter out the merchants being
predicted as restaurants (i.e., captured) by model A and B respectively, i.e.,
$\text{{A}}^{\text{+}}$ and $\text{{B}}^{\text{+}}$. Step 3 joins these two
sets and separates the merchants into
$\text{{A}}^{\text{+}}\text{{B}}^{\text{+}}$,
$\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}$, and
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}$. Based on the merchants’ true
label, each cell can further be divided into two smaller cells at Step 4
(i.e., six cells in total).

Step 5 generates 70 meta-features for each merchant from its raw transactions
over the past 2.5 years. It is not necessary to explain all of them, but the
ones used in this case and mentioned later in Sec. 7.1.2 include:

  * •

nonzero_numApprTrans: the number of days that a merchant has at least one
approved transaction in the 2.5 years. For time-series data, this meta-feature
reflects the number of meaningful points in a sequence. It is derived based on
the prior knowledge that RNNs often behave better on instances with richer
sequential information. Its value range is [0, 912] (2.5 years=912 days),
reflecting the active level of a merchant.

  * •

nonzero_amtDeclTrans: the number of days that a merchant has at least one
declined transaction (with a nonzero dollar amount) in the 2.5 years.

  * •

mean_avgAmtAppr: the mean of the average approved amount per day, over the 2.5
years, for each merchant.

  * •

mean_rateTxnAppr: the mean of the daily transaction approval rate, over the
2.5 years, for each merchant.

  * •

mean_numApprTrans: the mean of the number of daily approved transactions, over
the 2.5 years, per merchant.

Using all 70 meta-features of the merchants in the
$\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}$ and
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}$ cells from the TP side, we train
the TP discriminator. The FP discriminator is trained using the same meta-
features, but the merchants from the corresponding FP cells.

Step 6 interprets the discriminators to derive insights. For example, from the
visualization of nonzero_numApprTrans in the TP side (Fig. 9), we notice that
active merchants in orange bubbles are more likely to be from the
$\text{{Tree}}^{\text{-}}\text{{RNN}}^{\text{+}}$ (i.e.,
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}(\text{TP})$) cell, indicating they
will be correctly recognized (captured) by the RNN, but missed by the Tree
(i.e., the RNN outperforms the Tree on active merchants). In contrast, the
less active merchants in blue bubbles are more likely to be from the
$\text{{Tree}}^{\text{+}}\text{{RNN}}^{\text{-}}$ (i.e.,
$\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}(\text{TP})$) cell, indicating the
Tree outperforms the RNN in correctly recognizing less active merchants. The
insights here directly guide model selections based on the merchants’ service
frequency, i.e., active or not.

Figure 9: Compare Tree v.s. RNN, meta-feature: nonzero_numApprTrans.

Next, we compare the RNN (A) and GNN (B) (Tab. I, bottom row). Step 1$\sim$4
of the comparison are conducted very similarly to the previous case. The only
difference is that we sampled around $\sim$70K merchants from the $\sim$3.8M
for this comparison to reduce the merchant graph construction cost.

At Step 5, we generate several affinity-related meta-features to probe the two
models, because the experts expect the GNN to leverage the information from a
merchant’s neighbors more than the RNN. For example, the entropy of a merchant
reflects how diverse its neighbors’ category is. The n_connection denotes the
degree of each merchant in the merchant graph. Using these new meta-features,
we train the two discriminators of LFD. From the visualization at Step 6, we
found that entropy is a very differentiable meta-feature (Fig. 10). When a
merchant has more diverse neighbors (i.e., from the orange bubbles with larger
entropy), the GNN tends to correctly capture it whereas the RNN is more likely
to miss it (i.e., the orange bubbles mostly fall into the
$\text{{RNN}}^{\text{-}}\text{{GNN}}^{\text{+}}\text{(TP)}$ cell). This
indicates the GNN outperforms RNN on these merchants and the neighbors’
information indeed contributes to the predictions. The observation here
clearly reveals the value of affinity information between merchants, verifying
the improvement from the GNN [50].

These two brief cases use prior knowledge on the compared models to derive
meta-features. The models’ behaviors on the meta-features, in turn, verify the
experts’ expectations, i.e., our sanity checks succeeded.

####  7.1.2 Deeper Insights: [CNN v.s. RNN]

This section presents deeper insights when comparing the CNN (A) and RNN (B)
using LFD. The insights go beyond what was known by the experts and deepen
their understandings of the models. The CNN and RNN have similar performance
and both capture the temporal behaviors of the merchants, but in different
ways. The CNN uses 1D convolutions, whereas the RNN employs the GRU structure.

Step 12

Feeding the $\sim$3.8M merchants to the two models (i.e., two black-boxes), we
get two sets of scores, which are used to sort the merchants and identify the
two sets of merchants captured by individual models (i.e.,
$\text{{A}}^{\text{+}}$ and $\text{{B}}^{\text{+}}$). Joining the two sets, we
get merchants in the three cells of the disagreement matrix (i.e.,
$\text{{A}}^{\text{+}}\text{{B}}^{\text{+}}$,
$\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}$, and
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}$). This matrix is then divided
into two matrices of six cells based on the merchants’ true category label,
i.e., $\text{{A}}^{\text{+}}\text{{B}}^{\text{+}}\text{(TP)}$,
$\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}\text{(TP)}$,
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}\text{(TP)}$,
$\text{{A}}^{\text{+}}\text{{B}}^{\text{+}}\text{(FP)}$,
$\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}\text{(FP)}$, and
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}\text{(FP)}$.

Step 5

For the “learning” part of LFD, we use the merchants from the
$\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}\text{(TP)}$ and
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}\text{(TP)}$ cells to train the TP
discriminator; the merchants from
$\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}\text{(FP)}$ and
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}\text{(FP)}$ to train the FP
discriminator. For meta-features, we continue to use the 70 meta-features
derived when comparing the Tree and RNN models in Sec. 7.1.1.

Figure 10: Compare the RNN with GNN, meta-feature: entropy. Figure 11: Compare
the CNN and RNN. (a) TP (left) and FP (right) meta-feature lists. (b)
nonzero_numApprTrans ranks the third in the TP list. The RNN performs better
when this meta-feature has a smaller value. (c) is the same with (b) but
highlights merchants that are less active than 2 years (730 days). (d)
nonzero_numApprTrans on the FP side shows a reversed pattern with (b). All
instances here are non-restaurants. The RNN still behaves better than CNN on
less-active merchants, i.e., the RNN will NOT mis-capture less-active
merchants (the blue bubbles on the left). Figure 12: The ranks of meta-
features on the FP side (top-8 only). (a) mean_numApprTrans shows large
Magnitude but very small Contrast and Correlation (not in the top-8), thus its
Overall ranking becomes lower. (b) nonzero_amtDeclTrans ranks the 7th in
Magnitude. The good Contrast and Correlation bring it up in the Overall
ranking. (c, d) The same summary-plot with different color mappings. (d, e)
The same summary-plot with different granularities, i.e., different numbers of
bins in the 2D histogram of feature- and SHAP-values (details explained in
Fig. 6a).

Step 6

Fig. 11a shows the important meta-features to the TP and FP discriminators,
both sides are ranked by the Overall metric (Eq. 6, MC3). On the TP side
(left), nonzero_numApprTrans ranks the third and its detail is shown in Fig.
11b. The RNN correctly captures more when the merchants are relatively less-
active (i.e., the blue bubbles on right are more likely to be
$\text{{CNN}}^{\text{-}}\text{{RNN}}^{\text{+}}$), whereas the CNN correctly
captures more when the merchants are very active in the past 2.5 years (the
orange bubbles). With some analysis, the models’ behavior matched the experts’
expectations and they commented that the CNN has a limited receptive field
(limited by the number of convolutional layers), and focuses more on local
patterns, whereas the RNN can memorize longer history through its internal
hidden states. When a merchant has a long and active history, there are rich
local patterns as well, and the CNN outperforms the RNN. However, when the
merchant is less-active, the RNN performs better in capturing the sparser
temporal behaviors. Fig. 11c shows the same result with user-interactions,
i.e., by dragging the color control-points and brushing the legend, we can
easily identify merchants that the RNN outperforms the CNN (i.e., the ones
with approved transactions less than 730 days). Using this meta-feature, LFD
successfully reveals the subtle difference between the two models ( MC2, FA1).

Another meta-feature, i.e., mean_rateTxnAppr (ranked the second on the TP
side), demonstrates the same trend, indicating the RNN outperforms CNN in
correctly identifying restaurants with lower approval rates as well. In
contrast, the mean_avgAmtAppr (ranked the first) shows unnoticeable difference
for the merchants on the two sides of the $SHAP{=}0$ line (the areas on both
sides are in blue), implying this meta-feature cannot differentiate the two
models. However, the large magnitude of this feature still makes it rank the
first.

On the FP side (Fig. 11a, right), nonzero_numApprTrans ranks the second, and
its detail is shown in Fig. 11d. All merchants here are non-restaurants but
mis-captured as restaurants (i.e., FP), and the bubbles’ pattern is reversed
comparing to Fig. 11b (i.e., the blue bubbles come to the left in Fig. 11d).
The less-active merchants in blue bubbles are less likely to be mis-captured
by the RNN, but more likely to be mis-captured by the CNN (i.e., from the
$\text{{CNN}}^{\text{+}}\text{{RNN}}^{\text{-}}$ cell of the FP side),
indicating the RNN still outperforms the CNN on merchants with sparser
temporal behaviors.

We have also explored different metrics in ranking the meta-features. Fig. 12
shows the top-8 features ranked on the FP side. mean_numApprTrans (Fig. 12a)
has the largest Magnitude and it is identified as the most important meta-
feature using the traditional order (i.e., $mean(|SHAP|)$). However, it ranks
the 7th in the Consistency order and is not in the top-8 in the Contrast and
Correlation orders (tracking the red curve). In contrast, nonzero_amtDeclTrans
(Fig. 12b, tracking the green curve) is the 7th important in the Magnitude
list but has very large Contrast and Correlation values. Compared to the
traditional Magnitude order, our Overall metric improves the rank of
nonzero_amtDeclTrans (ranked the 3rd) and decreases the rank of
mean_numApprTrans (ranked the 4th), by considering multiple aspects of the
meta-features ( FA2).

Note that, the visual appearance of the area-plot and bubble-plot depends on
the color mapping, which can be adjusted by the “transfer function” widget.
Fig. 12c shows the detailed view of mean_numApprTrans with the default color
legend. It can be seen that there are very few orange bubbles and their
contribution to the aggregated color of the area-plot is unnoticeable. We can
change the color mapping, as shown by the legend in Fig. 12d, to map a larger
range of values to orange. However, most of the bubbles are still in blue, and
the meta-feature still has small Contrast.

The layout and size of the bubbles can also be flexibly adjusted to reflect
different levels of details for a meta-feature. Fig. 12d-e demonstrate this by
increasing the number of feature- and SHAP-bins when computing the 2D
histogram (explained in Fig. 6a), e.g., Fig. 12e uses more bins and presents
the feature in finer granularity. This also reflects that the accumulated area
of the bubble-plot (the Detail part) cannot accurately present the data
distribution, and verifies the need of the area-plot (the Overview part), see
Sec. 6.2.1.

###  7.2 Models for CTR Prediction

CTR prediction is to predict if a user will click an advertisement or not,
which is a critical problem in the advertising industry. Avazu [49] is a
public CTR dataset, containing more than 40 million instances across 10 days.
Each instance is an impression (advertisement view) and has 21 anonymized
categorical features, e.g., site_id, device_id (see details in [51]).

Step 1

Using Avazu, we compare a tree-based model (A, short for Tree) and an RNN
model (B). Both are trained to differentiate click from non-click instances.
The following data partition was used in both models’ training:

  * •

Day 1: reserved to generate historical features.

  * •

Day 2-8: used for model training

  * •

Day 9: used for model testing and comparison

  * •

Day 10: held-out for model ensembling experiments

As the models may use some historical features, e.g., the number of active
hours in the past day, we reserve Day 1 data for this purpose. Also, we do not
touch Day 10 data, and leave it for our later quantitative evaluation
experiments. Note that, there are also works that partition Avazu by shuffling
the 10 days data [52] and dividing them into folds. Our way of partitioning
data by time follows our industrial practice and is more realistic, i.e., we
should not leak any future data into the training process.

The Tree model takes individual data instances (advertisement views) as input,
whereas the RNN connects instances into viewing sequences and takes them as
input. We followed the winning solution of the Avazu CTR Challenge to form the
sequences [53]. The details of individual models’ training features and
architectures are not needed, as LFD is feature-agnostic and model-agnostic.
The final AUCs of the Tree and RNN are 0.7468 and 0.7396 respectively.

Step 23

These steps are similar to what we have described in earlier cases. Fig. 4
(left) shows the Disagreement Distribution View for this case, from which, we
know that the size of the disagreed instances from the TP side reaches the
peak if the cutoff/threshold is between 15%$\sim$20% (Fig 4-a3). We use 15% as
the cutoff to maximize the size of training data with the most disagreed
predictions ( MC1). One can also choose other cutoffs based on the FP data
distributions. However, we care more about TP instances in this case.

Step 5

This step generates meta-features from the raw data in two steps. First, the
Avazu raw data has 21 categorical features, and we extend them by
concatenating the features’ categorical values, e.g., device_id_ip is a
feature combined from device_id and device_ip. Based on the experts’ knowledge
on the data, we extend Avazu to have 42 features (and feature combinations).
Second, as CTR is the ratio between the number of clicks (n_clicks) and
impressions (n_impressions), i.e., CTR=n_clicks/n_impressions, we propose
meta-features by profiling the frequency of the 42 features from these three
dimensions. For example, for the raw feature device_id, we generate meta-
feature n_clicks_device_id, denoting the number of clicks per device_id value.
Also, as the RNN is not as good as the Tree (see the AUCs at Step 1), we want
to probe the models’ sequence-related behaviors. Thus, we generate meta-
features from a fourth dimension to reflect the active level of the features
(i.e., n_active_hours_*). In summary, generating meta-features from the 42
features in the following four dimensions, we get 168 ($42{\times}4$) meta-
features (their details are included in our Appendix),

  * •

n_impressions_* denotes the number of impressions per value of * (* represents
one of the 42 features).

  * •

n_clicks_* indicates the number of clicks per value of *.

  * •

ctr_* denotes the CTR for each value of *. For example, if * is device_id and
the CTR for a certain device is $x$, then the value of ctr_device_id will be
$x$ for all impressions happened on this device.

  * •

n_active_hours_* reflects the number of hours (in the past day) that each
value of * appeared in the data.

Using the 168 meta-features, we train the TP and FP discriminators to learn
the difference between the Tree and RNN.

Step 6

After training, we use our LFD system to rank the meta-features from the TP
and FP sides (Fig. 4-b5). Meta-features n_active_hours_* never appear in the
top differentiable ones, implying the RNN may not benefit more from the
sequential information (compared to the Tree). This insight provides clues to
further diagnose the RNN ( MC2).

After some analysis, the ML experts tend to believe the relatively worse
performance of RNN is from the data. First, there is no unique sequence id in
Avazu, and the solution we adopted to connect instances into sequences (from
[53]) may not be optimal. Second, the sequence length is very short and the
only 10 days data may not be able to well capture users’ temporal behaviors
(detailed sequence statistics are in our Appendix). In contrast, in Sec. 7.1,
we used 2.5 years of data and each merchant has a unique sequence id. The
interpretations provide some explanations on why RNN is not one of the top
solutions for this CTR challenge [54].

From the TP side (Fig. 4-b5, left), we can see ctr_site_id is a very
differentiable feature. Although the RNN has a smaller AUC, it outperforms the
Tree in capturing clicks when this meta-feature is large, i.e., the orange
bubbles in Fig. 4-b3 are more likely to be from the
$\text{{Tree}}^{\text{-}}\text{{RNN}}^{\text{+}}$ cell. This makes sense as
the RNN tends to remember the site visiting history. From the meta-features
ordered by the Overall metric, we can easily identify other important ones.
For example, ctr_site_app_id (ranked the second) denotes the CTR of the
feature combined from site_id and app_id. It shows the same trend with the
meta-feature ctr_site_id, i.e., the RNN behaves better than the Tree if an
impression is from the site-application pair with a higher click rate. ctr_c14
(ranked the third) is another important meta-feature. The RNN would be more
accurate in capturing clicks when this feature has small values (i.e.,
indicated by the blue area on the right of Fig. 4-b7). Although we do not know
what c14 is (an anonymized raw feature of Avazu), we can infer it is an
important click-related feature from LFD.

For the FP side (Fig. 4-b5, right), ctr_site_id is also the most important
(Fig. 4-b4, tracking the curves between the two meta-feature lists, MC3). Its
behavior is consistent to the TP side (Fig. 4-b2), i.e., orange regions are
still on the right, indicating the RNN tends to mis-capture instances if the
value of ctr_site_id is large. In contrast, the good feature we saw from Sec.
7.1.2 (i.e., nonzero_numApprTrans) shows reversed patterns from the TP and FP
sides (Fig. 11b, 11d). This reveals a potential bias of the model, i.e., the
RNN here tends to give high scores to instances with large ctr_site_id, no
matter they are real click instances or not. After thoroughly examining and
comparing the two cases, we found that the observation actually reflects
different signal strengths of the merchant category verification and CTR
problems.

As explained by the ML experts, the signal strength of a dataset reflects how
distinguishable the positive instances are from the negative ones. For
merchant category verification, the signal strength is very strong, i.e.,
merchants with falsified categories have certain on-purpose behaviors that
regular merchants cannot accidentally conduct. However, for CTR, the signal
strength is much weaker. Randomness widely exists when users choose to click
or not. As a result, two records with similar values across all features may
have different click labels. Consequently, some instances from the
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}(\text{TP})$ and
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}(\text{FP})$ cells are similar, so
as to some instances from the
$\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}(\text{TP})$ and
$\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}(\text{FP})$ cells. In turn, the
trained TP and FP discriminators behave similarly (due to their similar
training data), which explains the similar patterns of ctr_site_id on the TP
and FP sides (Fig. 4-b2, b4).

##  8 Evaluation and Feedback

This section quantitatively evaluates the feature-ordering metrics (using
FWLS) and presents an ablation study to profile how the number and quality of
meta-features affect the training of the discriminators at Step 5 of LFD.
Also, we summarize the feedback and suggestions provided by the ML experts
during the open-ended interviews with them.

###  8.1 Quantitative Evaluation with Model Ensembling

As explained in Sec. 3.3, FWLS ensembles models by considering the models’
behaviors in different features. What features should be used in FWLS is a
critical problem and the ensembling result can quantitatively reflect the
quality of the used features. We use the top-15 meta-features ranked by the
five metrics of LFD to generate five FWLS models, and compare their
performance to quantify the rankings’ quality. Note that, the ensembling
experiments here can be fully automatic and do not rely on any visual inputs.
However, to be consistent with our earlier writing style and simplify the
logic, we still describe them following the six steps of LFD and explain what
is needed/expected in individual steps.

Step 1

We use two state-of-the-art and publicly available CTR models to conduct this
experiment, so that people can reproduce it. One is the logistic regression
(LR) implemented by “follow the proximally regularized leader (FPRL-Proximal)”
[55]. The other is the “feature interaction graph neural network” (Fi-GNN,
short for GNN) [52], currently ranking the first for this challenge [54]. Both
models are trained using the 21 raw features of Avazu (Day 2$\sim$8).

Step 235

Using scores of the two models, we sort data instances (i.e., test data from
Day 9), identify the ones in the
$\text{{LR}}^{\text{+}}\text{{GNN}}^{\text{-}}$ and
$\text{{LR}}^{\text{-}}\text{{GNN}}^{\text{+}}$ cell (Step 3), and train a
discriminator to differentiate them using the 168 meta-features proposed in
Sec. 7.2 (Step 5). These meta-features have no overlap with the training
features of the LR or GNN. Note that, Step 4 of LFD is not needed here, since
we need a single order of meta-features considering both TP and FP instances,
i.e., no need to separate the TP and FP instances.

Step 6

After getting the discriminator, we use the five metrics from Sec. 6.2.2 to
rank the 168 meta-features into five orders, and pick the top-15 from each to
conduct a FWLS. As a result, there will be five models ensembled from the
original LR and GNN, each uses a set of 15 meta-features that are considered
as the most important by one metric. The performance of the five FWLS models
reflects how useful the five sets of meta-features are, which in turn,
reflects the quality of the five metrics. All the FWLS models are trained on
Day 9 data to learn the values of $w_{1}^{i}$ and $w_{2}^{i}$ in Eq. 2, and we
test their performance on both Day 9 and Day 10 data.

TABLE II: Performance of the original LR, GNN, and the five models ensembled from them (using the top-15 meta-features from different metrics). Model |  AUC (Day 9) |  LogLoss (9) |  AUC (10) |  LogLoss (10)
---|---|---|---|---
LR |  0.747349 |  0.379716 |  0.739194 |  0.401167
GNN |  0.753254 |  0.377235 |  0.739651 |  0.400147
$\text{{Esb}}_{\text{{Magnitude}}}$ |  0.756016 |  0.375891 |  0.745535 |  0.414997
$\text{{Esb}}_{\text{{Consistency}}}$ |  0.752616 |  0.383943 |  0.744574 |  0.410970
$\text{{Esb}}_{\text{{Contrast}}}$ |  0.756796 |  0.375351 |  0.746868 |  0.397326
$\text{{Esb}}_{\text{{Correlation}}}$ |  0.756187 |  0.375575 |  0.745156 |  0.398449
$\text{{Esb}}_{\text{{Overall}}}$ |  0.757125 |  0.375096 |  0.745980 |  0.397525

Tab. II shows the performance of the original and ensembled models. We have
three findings. First, all ensembled models (except
$\text{{Esb}}_{\text{{Consistency}}}$ for Day 9) are better than the original
LR and GNN, reflecting the efficacy of FWLS. Second, for both Day 9 and Day 10
data, $\text{{Esb}}_{\text{{Overall}}}$ is better than
$\text{{Esb}}_{\text{{Magnitude}}}$, indicating our Overall metric that
profiling features’ importance from multiple perspectives is better than the
traditional Magnitude metric. Third, for Day 10 data,
$\text{{Esb}}_{\text{{Contrast}}}$ generates the best performance, indicating
that the weights in our Overall metric (Eq. 6) may need to be adjusted for
different datasets.

###  8.2 Ablation Study on Meta-Features

Meta-features are dominating factors controlling the quality of the
discriminator (Step 5 of LFD). Therefore, using the discriminator from Sec.
8.1, we conduct ablation studies to investigate how the number and quality of
meta-features affect the discriminator’s performance.

Figure 13: Ablation study to investigate how the Discriminator’s performance
is affected by the number and quality of of meta-features.

To reveal how many meta-features are sufficient to train a good discriminator,
we remove the 168 meta-features one-by-one, retrain the discriminator from
scratch after each removal, and measure its performance by AUC. What order to
use when removing the meta-features is an important question. To simulate the
possible choices of meta-features with different qualities, we remove them in
a random order but repeat the entire ablation process 100 times. The red curve
in Fig. 13a shows how the average AUC (averaged over the 100 runs) changes
along with the decreasing number of meta-features. The green band surrounding
the red curve shows the standard deviation of the 100 runs. For example, the
right-most point on the red curve shows the average AUC of the discriminator
(when being trained with one meta-feature only) is 0.621, and the standard
deviation is 0.118. The upper/lower bound of the green band reflects the case
of using high/low-quality meta-features. The blue curve in Fig. 13a shows the
AUC of the discriminator when removing meta-features following our Overall
order. From the figure, we can draw the following conclusions.

  * •

Our Overall metric successfully prioritizes important meta-features, as
removing features following it makes the AUC drop much earlier and more
significantly.

  * •

Good meta-features, even only 5 of them (Fig. 13a, point $A$), can train a
good discriminator to differentiate the two compared models. Low-quality meta-
features, as many as 28 (Fig. 13a, point $B$), are less effective to train the
discriminator. So, meta-features’ quality matters.

  * •

Some meta-features or their combinations may have similar effects. For
example, combing meta-feature $\beta$ and $\gamma$ may produce the equivalent
effect of feature $\alpha$. Consequently, removing $\alpha$ will not
significantly affect the discriminator’s quality. This could be the reason for
the very similar AUC at point $C$ and $D$ in Fig. 13a. More analysis regarding
this can be found in our Appendix.

When training the discriminator (i.e., an XGBoost), we set the training
iterations to be very large but terminate the training if the AUC has no
improvement in 10 consecutive iterations. This mechanism makes sure the model
is sufficiently trained. Following the encoding in Fig. 13a, Fig. 13b shows
the number of training iterations. It can be seen that the number of meta-
features does not obviously affect the number of training iterations if the
discriminator can be successfully trained (i.e., mostly ${\sim}1000$
iterations). However, when the discriminator cannot be successfully trained
(due to the insufficient number or low quality of meta-features), the number
of training iterations drops significantly.

From this study, we notice that it is important to reveal the quality of the
discriminator. For different cases, however, it is hard to know how many meta-
features are sufficient and what qualities the meta-features have. Therefore,
we reflect the AUC of the trained discriminators in the system (i.e., the
numbers in Fig. 4-b8 show the AUC of the TP and FP discriminators). This
quality indicator helps users get to know if their meta-features are
sufficient or good enough before they can interpret the SHAP visualizations.

How hard to train the discriminators? Theoretically, the discriminators’ job
is much easier than model A and B, as both can clearly separate the
$\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}$ and
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}$ instances using their score
cutoff, e.g., for model $A$, all $\text{{A}}^{\text{+}}\text{{B}}^{\text{-}}$
instances are from $\text{{A}}^{\text{+}}$ (above the cutoff) and all
$\text{{A}}^{\text{-}}\text{{B}}^{\text{+}}$ instances are from
$\text{{A}}^{\text{-}}$ (below the cutoff). Practically, this can also be
verified from the much better AUCs of the discriminators (${>}0.99$ in Fig.
4-b8) than the two compared models (${<}0.76$ in Tab. II).

###  8.3 Domain Experts’ Feedback

We conduct multiple case studies (Sec. 7) with the eight ML experts
($E_{1}{\sim}E_{8}$) introduced in Sec. 5. The four merchant category
verification models in Sec. 7.1 are proposed by the experts and they have
sufficient knowledge on their differences. As a sanity check, the insights
derived from LFD match well with the experts’ expectations, e.g., the RNN is
better than the Tree in capturing temporal behaviors, the affinity information
helps the GNN outperform the RNN. The comparison of CTR models also makes
sense to the experts and the proposed meta-features (e.g., n_active_hours_*)
help to reveal the RNN’s deficiency on short sequences. We conclude the
studies with open-ended interviews and think-aloud discussions, and summarize
the experts’ feedback from the following perspectives.

LFD Idea and Its Understandability. In general, all experts felt positive
about the idea to learn from two models’ disagreement. $E_{1}$ liked the idea
a lot and considered LFD as an “offline adversarial learning” (in analogy to
the online adversarial learning of GAN [56]), where the two compared models
are used to explicitly identify where to learn and our discriminator is
similar to the discriminator of GAN. He also identified the advantage of LFD
in “smartly avoiding the data imbalance issue”. $E_{6}$ stated that LFD is
“similar to existing Explainable AI works” that “train a surrogate model by
distilling knowledge from original models”. He considered our discriminator as
the surrogate which distills disagreement knowledge from the comparison.
$E_{7}$ commented that the “LFD pipeline is smooth without any gap”. $E_{8}$
echoed that the novel part of LFD is to use “supervised learning” to relate
users’ interested model-behaviors (externalized by meta-features) with the
models’ prediction discrepancy. Regarding the difficulty level of LFD, most
experts agreed that the overall idea is straightforward and Fig. 1 is a good
illustration. $E_{6}$ and $E_{8}$ expected the logic of Step 4 to be difficult
for novices to comprehend, and commented that users do need some deeper
thoughts to understand the complicated disagreement scenarios in real-world
applications.

Meta-Features and Feature-Ordering. Some experts understood the concept of
meta-features very easily. For example, $E_{3}$ worked on FWLS and immediately
accepted this concept, as it was also used in FWLS. The concept was new to
some other experts, e.g., $E_{6}$. After understanding it, $E_{6}$
differentiated meta-features from training features based on their purpose,
i.e., “training features are to improve models’ performance” whereas “meta-
features are to improve models’ interpretability”. About how hard it is to
generate meta-features, $E_{7}{\sim}E_{8}$ commented that they often have
thousands of ideas to generate features, as long as they are familiar with the
data. However, they were concerned about whether the quality of their meta-
features would be good enough. We comforted them that they can blindly throw
all their meta-features into LFD, use the feature-ordering metrics to sort
them, and compare models based on the important meta-features only. $E_{7}$
advocated this idea and considered it as a good feature filtering mechanism.
About our feature-ordering metrics, all experts supported our consideration to
more comprehensively rank features. $E_{2}$ has decades of ML experience. He
commented that “the feature order is extremely important” in his model
building process and appreciated the new metrics. $E_{7}$ felt the Contrast
order should be the best. We presented her a meta-feature with large Contrast
but small Magnitude to show how features’ magnitude could also limit their
contribution. $E_{8}$ worked on similar feature ordering problems and felt our
metrics “very inspiring”.

Visual Interface and SHAP Interpretation. For the Disagreement Distribution
View, most experts could easily get our design philosophy after watching the
associated video illustrations. However, some experts tended to ignore the
disagreement distributions, but chose the threshold (for Step 2 of LFD) based
on their experience, to make the analysis more realistic. For the Feature
View, the interactions were mostly ordering the meta-features and clicking
individual ones to check their details in the bubble-plot. These interactions
and the “overview+details” exploration were intuitive to the experts. $E_{6}$
focused on the color legend (in Fig. 4-b3) during explorations and liked the
flexible color mapping function (i.e., by dragging the control points). He
found this widget very useful, especially when the contrast between the left
and right half of the bubble-plot is not obvious. He also recommended
algorithms to automatically find the best color mapping. $E_{7}$ felt the
numerical values in Fig. 4-b6 are sufficient to show the orders and suggested
removing the blue bars (with orange triangles) to save space. Although our
eight users are ML experts, not all of them are experts in SHAP. However, as
LFD uses SHAP as a “black-box", all experts felt comfortable when interpreting
the outcomes. Also, we would like to admit the SHAP dependency of our
framework and the corresponding limitations (see Sec. 9).

Limitations. First, there was a discussion on the possibility of extending LFD
to compare more than two classifiers. The straightforward extension is to
train a multi-class discriminator at Step 5 of LFD. $E_{6}$ pointed out the
complexity would likely come from the disagreement matrix, where the number of
cells is roughly $2^{n}$ ($n$ is the number of compared models), which may
make the logic complicated. Another limitation identified by $E_{2}$ and
$E_{6}$ is that the two compared models must have disagreements before LFD can
be executed. If $\text{{A}}^{\text{+}}{\subset}\text{{B}}^{\text{+}}$ or
$\text{{B}}^{\text{+}}{\subset}\text{{A}}^{\text{+}}$ (at Step 2), LFD can
learn nothing. Third, $E_{7}$ and $E_{8}$ discussed how different types of
discriminators would affect the interpretability of LFD. We currently used
XGBoost but other SHAP-friendly models could also be used, and we agreed it is
a good future direction to compare the interpretation outcomes from different
discriminators. Lastly, $E_{8}$ also mentioned the possibility of
contradictory feature conditions. He continued our spam filtering example and
explained that the coming emails may have large $n\\_url$ and large $n\\_cap$.
From LFD, we may know that model A outperforms B when $n\\_url$ is large,
whereas B outperforms A when $n\\_cap$ is large. In this case, it is hard to
choose between A and B as LFD would recommend both. We did not see such cases
in our current experiments but were glad to notice such possible scenarios. A
quick remedy is to vote models based on the meta-features. We would like to
conduct more studies in this direction in the future.

Suggested Improvements. We asked the experts to provide suggestions or desired
functions to improve LFD at the end. $E_{1}{\sim}E_{2}$ encouraged us to
integrate more feature interpretation methods to make LFD more general.
$E_{3}{\sim}E_{4}$ wanted to see the precision of the two compared models at
different thresholds in the Disagreement Distribution View (Fig. 4-a4).
$E_{5}$ asked us to extend LFD by enabling “the negative side comparisons”
(i.e., comparing true-negative and false-negative instances by sorting
instances increasingly at Step 2). $E_{6}$ recommended us weighting the
disagreed instances based on their disagreement-level before learning from
them. This is similar to LIME on weighting instances based on their locality
to the interpreted instance. $E_{7}{\sim}E_{8}$ preferred to employ learning-
to-rank algorithms to generate the best weights in the Overall metric. These
suggestions provide promising directions for us to improve LFD.

##  9 Discussion, Limitations, and Future Work

The $\text{{A}}^{\text{-}}\text{{B}}^{\text{-}}$ cell. We focused on positive
predictions (i.e., TP and FP instances) of the two compared classifiers, and
thus, the instances are at least captured by one model. As a result, we do not
have the $\text{{A}}^{\text{-}}\text{{B}}^{\text{-}}$ cell in LFD. However, as
suggested by $E_{5}$ (Sec. 8.3), one can also compare two classifiers from the
negative predictions, i.e., sorting the scores increasingly at Step 2 of LFD.
This is less common and out of the scope of this paper. Also, we want to
emphasize that the $\text{{A}}^{\text{-}}\text{{B}}^{\text{-}}$ (as well as
the $\text{{A}}^{\text{+}}\text{{B}}^{\text{+}}$) cell is of less interest for
the purpose of comparison, as it is where the two classifiers agree on.

Dependency on SHAP. LFD depends on SHAP, so does the interpretation accuracy.
Consequently, it has an inherent limitation on cases where SHAP cannot provide
accurate interpretations. For this limitation, we want to emphasize two
points. First, SHAP is widely applicable to most ML models and comes with
solid theoretical supports [29, 2]. So, we do not expect considerably
inaccurate interpretations to occur very often. Second, as the six steps of
LFD have been very well modularized, we can easily replace SHAP with other
interpretation methods at Step 6.

There are three scalability concerns of LFD. First, LFD is limited to the
comparison between two binary classifiers only due to our focused problem
(i.e., “swap analysis”). We discussed the possibility of extending LFD to
compare more classifiers (Sec. 8.3) and considered it as an important future
work. For multi-class classifiers, one can still use LFD to compare their
difference on a single class at a time. Second, our system currently supports
hundreds of meta-features. However, for cases with thousands of meta-features,
our visualization (the overview part, Fig. 4-b5) may not scale well.
Fortunately, using the proposed new metrics, we can eliminate the less
important meta-features from visualization to reduce the cost. Lastly, the
SHAP computations may also be a bottleneck when the data is large. However,
SHAP can be computed offline and one may consider other more efficient model
interpretation methods for a replacement.

We believe LFD is useful from at least two perspectives. First, it provides
feature-level interpretations when comparing two classifiers (i.e.,
comparative interpretations), providing unique insights into model selections.
We want to emphasize that LFD is not designed to completely replace or compete
against any existing model comparison solutions. Existing solutions, e.g.,
comparing models with aggregated metrics, still have their advantages (e.g.,
simpler and more generalizable). We expect LFD to be used together with them
and reveal models’ discrepancy from a different angle. Second, LFD provides
more effective metrics in prioritizing meta-features, leading to better model
ensembling. As explained, the importance of features is often measured by
their contribution Magnitude, which depicts features’ importance from one
perspective only. Our Overall metric profiles features from multiple
perspectives and could more comprehensively reflect features’ importance.

##  10 Conclusion

This work introduces LFD, a model comparison and visualization framework. LFD
compares two classification models by identifying data instances with
disagreed predictions, and learning from their disagreement using a set of
user-proposed meta-features. Based on the SHAP interpretation of the learning
outcomes, we can interpret the two models’ behavior discrepancy on different
feature conditions. Also, we propose multiple metrics to prioritize the meta-
features and use model ensembling to evaluate the metrics’ quality. Through
concrete cases conducted with ML experts on real-world problems, we validate
the efficacy of LFD.

## References

  * [1] M. T. Ribeiro, S. Singh, and C. Guestrin, “"why should i trust you?" explaining the predictions of any classifier,” in _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2016, pp. 1135–1144.
  * [2] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in _Advances in neural information processing systems_ , 2017, pp. 4765–4774.
  * [3] C. M. Bishop, _Pattern recognition and machine learning_. springer, 2006.
  * [4] O. Sagi and L. Rokach, “Ensemble learning: A survey,” _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , vol. 8, no. 4, p. e1249, 2018.
  * [5] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [6] J. Zhang, Y. Wang, P. Molino, L. Li, and D. S. Ebert, “Manifold: A model-agnostic framework for interpretation and diagnosis of machine learning models,” _IEEE transactions on visualization and computer graphics_ , vol. 25, no. 1, pp. 364–373, 2018.
  * [7] D. Ren, S. Amershi, B. Lee, J. Suh, and J. D. Williams, “Squares: Supporting interactive performance analysis for multiclass classifiers,” _IEEE transactions on visualization and computer graphics_ , vol. 23, no. 1, pp. 61–70, 2016.
  * [8] J. Sill, G. Takács, L. Mackey, and D. Lin, “Feature-weighted linear stacking,” _arXiv preprint arXiv:0911.0460_ , 2009.
  * [9] J. Yuan, C. Chen, W. Yang, M. Liu, J. Xia, and S. Liu, “A survey of visual analytics techniques for machine learning,” _Computational Visual Media_ , vol. 7, no. 1, pp. 3–36, 2021. [Online]. Available: https://doi.org/10.1007/s41095-020-0191-7
  * [10] S. Liu, X. Wang, M. Liu, and J. Zhu, “Towards better analysis of machine learning models: A visual analytics perspective,” _Visual Informatics_ , vol. 1, no. 1, pp. 48–56, 2017.
  * [11] F. Hohman, M. Kahng, R. Pienta, and D. H. Chau, “Visual analytics in deep learning: An interrogative survey for the next frontiers,” _IEEE transactions on visualization and computer graphics_ , vol. 25, no. 8, pp. 2674–2693, 2018.
  * [12] C. Molnar, _Interpretable machine learning_. Lulu. com, 2020.
  * [13] H. Strobelt, S. Gehrmann, H. Pfister, and A. M. Rush, “Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks,” _IEEE transactions on visualization and computer graphics_ , vol. 24, no. 1, pp. 667–676, 2017.
  * [14] J. Wang, L. Gou, H. Yang, and H.-W. Shen, “GANViz: A visual analytics approach to understand the adversarial game,” _IEEE transactions on visualization and computer graphics_ , vol. 24, no. 6, pp. 1905–1917, 2018.
  * [15] M. Kahng, P. Y. Andrews, A. Kalro, and D. H. P. Chau, “ActiVis: Visual exploration of industry-scale deep neural network models,” _IEEE transactions on visualization and computer graphics_ , vol. 24, no. 1, pp. 88–97, 2017.
  * [16] N. Pezzotti, T. Höllt, J. Van Gemert, B. P. Lelieveldt, E. Eisemann, and A. Vilanova, “DeepEyes: Progressive visual analytics for designing deep neural networks,” _IEEE transactions on visualization and computer graphics_ , vol. 24, no. 1, pp. 98–108, 2017.
  * [17] M. Liu, J. Shi, Z. Li, C. Li, J. Zhu, and S. Liu, “Towards better analysis of deep convolutional neural networks,” _IEEE transactions on visualization and computer graphics_ , vol. 23, no. 1, pp. 91–100, 2016.
  * [18] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2921–2929.
  * [19] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all convolutional net,” _arXiv preprint arXiv:1412.6806_ , 2014.
  * [20] J. Wang, L. Gou, H.-W. Shen, and H. Yang, “DQNViz: A visual analytics approach to understand deep q-networks,” _IEEE transactions on visualization and computer graphics_ , vol. 25, no. 1, pp. 288–298, 2018.
  * [21] K. Cao, M. Liu, H. Su, J. Wu, J. Zhu, and S. Liu, “Analyzing the noise robustness of deep neural networks,” _IEEE transactions on visualization and computer graphics_ , 2020.
  * [22] Y. Ming, S. Cao, R. Zhang, Z. Li, Y. Chen, Y. Song, and H. Qu, “Understanding hidden memories of recurrent neural networks,” in _2017 IEEE Conference on Visual Analytics Science and Technology (VAST)_. IEEE, 2017, pp. 13–24.
  * [23] F. Hohman, A. Head, R. Caruana, R. DeLine, and S. M. Drucker, “Gamut: A design probe to understand how data scientists understand machine learning models,” in _Proceedings of the 2019 CHI conference on human factors in computing systems_ , 2019, pp. 1–13.
  * [24] J. Wexler, M. Pushkarna, T. Bolukbasi, M. Wattenberg, F. Viégas, and J. Wilson, “The what-if tool: Interactive probing of machine learning models,” _IEEE transactions on visualization and computer graphics_ , vol. 26, no. 1, pp. 56–65, 2019.
  * [25] J. Krause, A. Perer, and K. Ng, “Interacting with predictions: Visual inspection of black-box machine learning models,” in _Proceedings of the 2016 CHI conference on human factors in computing systems_ , 2016, pp. 5686–5697.
  * [26] S. Amershi, M. Chickering, S. M. Drucker, B. Lee, P. Simard, and J. Suh, “Modeltracker: Redesigning performance analysis tools for machine learning,” in _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ , 2015, pp. 337–346.
  * [27] J. Wang, L. Gou, W. Zhang, H. Yang, and H.-W. Shen, “DeepVID: Deep visual interpretation and diagnosis for image classifiers via knowledge distillation,” _IEEE transactions on visualization and computer graphics_ , vol. 25, no. 6, pp. 2168–2180, 2019.
  * [28] Y. Ming, H. Qu, and E. Bertini, “RuleMatrix: Visualizing and understanding classifiers with rules,” _IEEE transactions on visualization and computer graphics_ , vol. 25, no. 1, pp. 342–352, 2018.
  * [29] S. M. Lundberg, G. G. Erion, and S.-I. Lee, “Consistent individualized feature attribution for tree ensembles,” _arXiv:1802.03888_ , 2018.
  * [30] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard _et al._ , “Tensorflow: A system for large-scale machine learning,” in _12th $\\{$USENIX$\\}$ symposium on operating systems design and implementation ($\\{$OSDI$\\}$ 16)_, 2016, pp. 265–283.
  * [31] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg _et al._ , “Scikit-learn: Machine learning in python,” _the Journal of machine Learning research_ , vol. 12, pp. 2825–2830, 2011.
  * [32] H. Piringer, W. Berger, and J. Krasser, “Hypermoval: Interactive visual validation of regression models for real-time simulation,” in _Computer Graphics Forum_ , vol. 29, no. 3. Wiley Online Library, 2010, pp. 983–992.
  * [33] S. Murugesan, S. Malik, F. Du, E. Koh, and T. M. Lai, “Deepcompare: Visual and interactive comparison of deep learning model performance,” _IEEE computer graphics and applications_ , vol. 39, no. 5, pp. 47–59, 2019.
  * [34] H. Zeng, H. Haleem, X. Plantaz, N. Cao, and H. Qu, “Cnncomparator: Comparative analytics of convolutional neural networks,” _arXiv preprint arXiv:1710.05285_ , 2017.
  * [35] W. Yu, K. Yang, Y. Bai, T. Xiao, H. Yao, and Y. Rui, “Visualizing and comparing alexnet and vgg using deconvolutional layers,” in _Proceedings of the 33 rd International Conference on Machine Learning_ , 2016\.
  * [36] C. Olah, A. Mordvintsev, and L. Schubert, “Feature visualization,” _Distill_ , 2017, https://distill.pub/2017/feature-visualization.
  * [37] G. Montavon, S. Lapuschkin, A. Binder, W. Samek, and K.-R. Müller, “Explaining nonlinear classification decisions with deep taylor decomposition,” _Pattern Recognition_ , vol. 65, pp. 211–222, 2017.
  * [38] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.
  * [39] S. Liu, J. Xiao, J. Liu, X. Wang, J. Wu, and J. Zhu, “Visual diagnosis of tree boosting methods,” _IEEE transactions on visualization and computer graphics_ , vol. 24, no. 1, pp. 163–173, 2017.
  * [40] T. Mühlbacher and H. Piringer, “A partition-based framework for building and validating regression models,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 19, no. 12, pp. 1962–1971, 2013.
  * [41] W.-Y. Loh, “Classification and regression trees,” _Wiley interdisciplinary reviews: data mining and knowledge discovery_ , vol. 1, no. 1, pp. 14–23, 2011.
  * [42] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , 2016, pp. 785–794.
  * [43] J. Wang, W. Zhang, L. Wang, and H. Yang, “Investigating the evolution of tree boosting models with visual analytics,” in _2021 IEEE Pacific Visualization Symposium (PacificVis)_. IEEE, 2021.
  * [44] Z.-H. Zhou, _Ensemble methods: foundations and algorithms_. CRC press, 2012.
  * [45] L. Breiman, “Stacked regressions,” _Machine learning_ , vol. 24, no. 1, pp. 49–64, 1996.
  * [46] J. Zhao, N. Cao, Z. Wen, Y. Song, Y.-R. Lin, and C. Collins, “# fluxflow: Visual analysis of anomalous information spreading on social media,” _IEEE transactions on visualization and computer graphics_ , vol. 20, no. 12, pp. 1773–1782, 2014.
  * [47] W. Wang, H. Wang, G. Dai, and H. Wang, “Visualization of large hierarchical data by circle packing,” in _Proceedings of the SIGCHI conference on Human Factors in computing systems_ , 2006, pp. 517–520.
  * [48] J. Lin, “Divergence measures based on the shannon entropy,” _IEEE Transactions on Information theory_ , vol. 37, no. 1, pp. 145–151, 1991.
  * [49] AnyAI, “OpenCTR,” https://zenodo.org/record/2002072, 2018.
  * [50] C.-C. M. Yeh, Z. Zhuang, Y. Zheng, L. Wang, J. Wang, and W. Zhang, “Merchant category identification using credit card transactions,” in _IEEE International Conference on Big Data (Big Data)_ , 2020, pp. 1736–1744.
  * [51] “Avazu features,” https://github.com/xue-pai/OpenCTR/tree/master/data#Avazu, accessed: 2021-02-01.
  * [52] Z. Li, Z. Cui, S. Wu, X. Zhang, and L. Wang, “Fi-gnn: Modeling feature interactions via graph neural networks for ctr prediction,” in _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , 2019, pp. 539–548.
  * [53] “Winning solution of the Avazu CTR problem,” https://github.com/ycjuan/kaggle-avazu, accessed: 2021-02-01.
  * [54] “Click through rate prediction on avazu,” https://paperswithcode.com/sota/click-through-rate-prediction-on-avazu, accessed: 2021-02-01.
  * [55] H. B. McMahan, G. Holt, D. Sculley, M. Young, D. Ebner, J. Grady, L. Nie, T. Phillips, E. Davydov, D. Golovin, S. Chikkerur, D. Liu, M. Wattenberg, A. M. Hrafnkelsson, T. Boulos, and J. Kubica, “Ad click prediction: A view from the trenches,” in _Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2013, p. 1222–1230.
  * [56] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2_ , ser. NIPS’14. Cambridge, MA, USA: MIT Press, 2014, p. 2672–2680.

|  Junpeng Wang is a staff research scientist at Visa Research. He received
his B.E. degree in software engineering from Nankai University in 2011, M.S.
degree in computer science from Virginia Tech in 2015, and Ph.D. degree in
computer science from the Ohio State University in 2019. His research
interests are broadly in visualization, visual analytics, and explainable AI.
---|---
|  Liang Wang is a principal research scientist in the Data Analytics team at
Visa Research. His research interests are in data mining, machine learning,
and fraud analytics. Liang received his Ph.D. degree in Computer Science from
Faculté Polytechnique de Mons, Mons, Belgium with highest honors, and his BS
degree in Electrical Engineering & Automation and his MS degree in Systems
Engineering, both from Tianjin University. Prior to joining Visa, He was a
senior principal data scientist at Yahoo!, responsible for building traffic
protection solutions for Yahoo!’s advertising platforms. Before Yahoo!, he was
a distinguished scientist at eBay/PayPal, leading projects on risk detection
for PayPal’s core payment system. Liang also worked at FICO as a senior
scientist, focusing on bankcard fraud detection. He is the inventor of over 20
patents and has published over 30 papers in international journals and
conferences.
---|---
|  Yan Zheng is currently a Senior Staff Research Scientist at Visa Research.
Yan received her Ph.D. in Computer Science from University of Utah in 2017.
She has published papers in top venues, including KDD, SIGMOD, ICDM and
others. Her research interests are in data mining, machine learning, and
representation learning.
---|---
|  Chin-Chia Michael Yeh is currently a Staff Research Scientist at Visa
Research. Michael received his Ph.D. in Computer Science from University of
California, Riverside. His Ph.D. thesis “Toward a Near Universal Time Series
Data Mining Tool: Introducing the Matrix Profile,” received Doctoral
Dissertation Award Honorable Mention at KDD 2019. He has published papers in
top venues, including KDD, VLDB, ICDM and others. His research interests are
in data mining, machine learning, and time series analysis.
---|---
|  Shubham Jain is a Staff Software Engineer at Visa Research. He received his
M.S. in Computer Science from University of Illinois at Urbana-Champaign in
2019, and received his B.T. in Computer Science and Engineering from Indian
Institute of Technology Kanpur in 2017. His master’s thesis focused on using
landmarks for enhancing cloth retrieval. Prior to joining Visa, Shubham was a
Software Engineer Intern at NVIDIA, working on deep learning research. Before
working at NVIDIA, he was a Research Intern at Adobe and a Visiting Student
Researcher at Montreal Institute of Learning Algorithms (MILA) under Prof.
Yoshua Bengio. His interests broadly lies in Computer Vision, Recurrent Neural
Networks and Machine Learning.
---|---
|  Wei Zhang is a principal research scientist and research manager at Visa
Research and interested in big data modeling and advanced machine learning
technologies for payment industry. Prior to joining Visa Research, Wei worked
as a Research Scientist in Facebook, R&D manager in Nuance Communications and
also worked in IBM research over 10 years. Wei received his Bachelor and
Master degrees from Department of Computer Science, Tsinghua University.
---|---