MAGDiff: Covariate Data Set Shift Detection via Activation Graphs of
  Deep Neural Networks
Despite their successful application to a variety of tasks, neural networks
remain limited, like other machine learning methods, by their sensitivity to
shifts in the data: their performance can be severely impacted by differences
in distribution between the data on which they were trained and that on which
they are deployed. In this article, we propose a new family of representations,
called MAGDiff, that we extract from any given neural network classifier and
that allows for efficient covariate data shift detection without the need to
train a new model dedicated to this task. These representations are computed by
comparing the activation graphs of the neural network for samples belonging to
the training distribution and to the target distribution, and yield powerful
data- and task-adapted statistics for the two-sample tests commonly used for
data set shift detection. We demonstrate this empirically by measuring the
statistical powers of two-sample Kolmogorov-Smirnov (KS) tests on several
different data sets and shift types, and showing that our novel representations
induce significant improvements over a state-of-the-art baseline relying on the
network output.
Charles Arnal, Felix Hensel, Mathieu Carrière, Théo Lacombe, Hiroaki Kurihara, Yuichi Ike, Frédéric Chazal
# MAGDiff: Covariate Data Set Shift Detection via Activation Graphs of Deep Neural Networks

###### Abstract

Despite their successful application to a variety of tasks, neural networks remain limited, like other machine learning methods, by their sensitivity to shifts in the data: their performance can be severely impacted by differences in distribution between the data on which they were trained and that on which they are deployed. In this article, we propose a new family of representations, called MAGDiff, that we extract from any given neural network classifier and that allows for efficient covariate data shift detection without the need to train a new model dedicated to this task. These representations are computed by comparing the activation graphs of the neural network for samples belonging to the training distribution and to the target distribution, and yield powerful data- and task-adapted statistics for the two-sample tests commonly used for data set shift detection. We demonstrate this empirically by measuring the statistical powers of two-sample Kolmogorov-Smirnov (KS) tests on several different data sets and shift types, and showing that our novel representations induce significant improvements over a state-of-the-art baseline relying on the network output.

## 1 Introduction

During the last decade, neural networks (NN) have become immensely popular, reaching state-of-the-art performances in a wide range of situations. Nonetheless, once deployed in real-life settings, NN can face various challenges such as being subject to adversarial attacks [13], being exposed to out-of-distributions samples (samples that were not presented at training time) [11], or more generally being exposed to a _distribution shift_: when the distribution of inputs differs from the training distribution (_e.g._, input objects are exposed to a corruption due to deterioration of measure instruments such as cameras or sensors). Such distribution shifts are likely to degrade performances of presumably well-trained models [31], and being able to detect such shifts is a key challenge in monitoring NN once deployed in real-life applications. Though shift detection for univariate variables is a well-studied problem, the task gets considerably harder with high-dimensional data, and seemingly reasonable methods often end up performing poorly [23].

In this work, we introduce the Mean Activation Graph Difference (MAGDiff), a new approach that harnesses the powerful dimensionality reduction capacity of deep neural networks in a data- and task-adapted way. The key idea, further detailed in Section 4, is to consider the activation graphs generated by inputs as they are processed by a neural network that has already been trained for a classification task, and to compare such graphs to those associated to samples from the training distribution. The
method can thus be straightforwardly added as a diagnostic tool on top of preexisting classifiers without requiring any further training ; it is easy to implement, and computationally inexpensive. As the activation graphs depend on the network weights, which in turn have been trained for the data and task at hand, one can also hope for them to capture information that is most relevant to the context. Hence, our method can easily support, and benefit from, any improvements in deep learning.

Our approach is to be compared to _Black box shift detection_ (BBSD), a method introduced in [20; 22] that shares a similar philosophy. BBSD uses the output of a trained classifier to efficiently detect various types of shifts (see also Section 4); in their experiments, BBSD generally beats other methods, the runner-up being a much more complex and computationally costly multivariate two-sample test combining an autoencoder and the Maximum Mean Discrepancy statistic [7].

Our contributions are summarized as follows.

1. Given any neural network classifier, we introduce a new family of representations MAGDiff, that is obtained by comparing the activation graphs of samples to the mean activation graph of each class in the training set.
2. We propose to use MAGDiff as a statistic for data set shift detection. More precisely, we combine our representations with the statistical method that was proposed and applied to the Confidence Vectors (CV) of classifiers in [20], yielding a new method for shift detection.
3. We experimentally show that our shift detection method with MAGDiff outperforms the state-of-the-art BBSD with CV on a variety of datasets, covariate shift types and shift intensities, often by a wide margin. Our code is publicly available1. Footnote 1: [https://github.com/hensel-f/MAGDiff_experiments](https://github.com/hensel-f/MAGDiff_experiments)

## 2 Related Work

Detecting changes or outliers in data can be approached from the angle of anomaly detection, a well-studied problem [5], or out-of-distribution (OOD) sample detection [26]. Among techniques that directly frame the problem as shift detection, kernel-based methods such as Maximum Mean Discrepancy (MMD) [7; 34] and Kernel Mean Matching (KMM) [8; 35] have proved popular, though they scale poorly with the dimensionality of the data [23]. Using classifiers to test whether samples coming from two distributions can be correctly labeled, hence whether the distributions can be distinguished, has also been attempted; see, _e.g._, [16]. The specific cases of covariate shift [14; 29; 22] and label shift [28; 20] have been further investigated, from the point of view of causality and anticausality [25]. Moreover, earlier investigations of similar questions have arisen from the fields of economics [10] and epidemiology [24].

Among the works cited above, [20] and [22] are of particular interest to us. In [20], the authors detect label shifts using shifts in the distribution of the outputs of a well-trained classifier; they call this method Black Box Shift Detection (BBSD). In [22], the authors observe that BBSD tends to generalize very well to covariate shifts, though without the theoretical guarantees it enjoys in the label shift case. Our method is partially related to BBSD. Roughly summarized, we apply similar statistical tests--combined univariate Kolmogorov-Smirnov tests--to different features--Confidence Vectors (CV) in the case of BBSD, distances to mean activation graphs (MAGDiff) in ours. Similar statistical ideas have also been explored in [3] and [4], while neural network activation graph features have been studied in, _e.g._, [18] and [12]. The related issue of the robustness of various algorithms to diverse types of shifts has been recently investigated in [32].

## 3 Background

### Shift Detection with Two-Sample Tests

There can often be a shift between the distribution \(\mathbb{P}_{0}\) of data on which a model has been trained and tested and the distribution \(\mathbb{P}_{1}\) of the data on which it is used after deployment; many factors can cause such a shift, _e.g._, a change in the environment, in the data acquisition process, or the training set being unrepresentative. Detecting shifts is crucial to understanding, and possibly correcting, potential losses in performance; even shifts that do not strongly impact accuracy can be important symptoms of inaccurate assumptions or changes in deployment conditions.
Additional assumptions can sometimes be made on the nature of the shift. In the context of a classification task, where data points are of the shape \((x,y)\) with \(x\) the feature vector and \(y\) the label, a shift that preserves the conditional distribution \(p(x|y)\) (but allows the proportion of each label to vary) is called _label shift_. Conversely, a _covariate shift_ occurs when \(p(y|x)\) is preserved, but the distribution of \(p(x)\) is allowed to change. In this article, we focus on the arguably harder case of covariate shifts. See Section 5 for examples of such shifts in numerical experiments.

Shifts can be detected using _two-sample tests_: that is, a statistical test that aims at deciding between the two hypotheses

\[H_{0}:\mathbb{P}_{0}=\mathbb{P}_{1}\text{ and }H_{1}:\mathbb{P}_{0}\neq\mathbb{ P}_{1},\]

given two random sets of samples, \(X_{0}\) and \(X_{1}\), independently drawn from two distributions \(\mathbb{P}_{0}\) and \(\mathbb{P}_{1}\). To do so, many statistics have been derived, depending on the assumptions made on \(\mathbb{P}_{0}\) and \(\mathbb{P}_{1}\).

In the case of distributions supported on \(\mathbb{R}\), one such test is the _univariate Kolmogorov-Smirnov (KS) test_, of which we make use in this article. Given, as above, two sets of samples \(X_{0},X_{1}\subset\mathbb{R}\), consider the empirical distribution functions \(F_{i}(z)\coloneqq\frac{1}{\text{Card}(X_{i})}\sum_{x\in X_{1}}1_{x\leq z}\) for \(i=0,1\) and \(z\in\mathbb{R}\). Then the statistic associated with the KS test and the samples is \(T\coloneqq\sup_{z\in\mathbb{R}}|F_{0}(z)-F_{1}(z)|\). If \(\mathbb{P}_{0}=\mathbb{P}_{1}\), the distribution of \(T\) is independent of \(\mathbb{P}_{0}\) and converges to a known distribution when the sizes of the samples tend to infinity (under mild assumptions) [27]. Hence approximate \(p\)-values can be derived.

The KS test can also be used to compare multivariate distributions: if \(\mathbb{P}_{0}\) and \(\mathbb{P}_{1}\) are distributions on \(\mathbb{R}^{D}\), a \(p\)-value \(p_{i}\) can be computed from the samples by comparing the \(i\)-th entries of the vectors of \(X_{0},X_{1}\subset\mathbb{R}^{D}\) using the univariate KS test, for \(i=1,\ldots,D\). A standard and conservative way of combining those \(p\)-values is to reject \(H_{0}\) if \(\min(p_{1},\ldots,p_{D})<\alpha/D\), where \(\alpha\) is the significance level of the test. This is known as the _Bonferroni correction_[30]. Other tests tackle the multidimensionality of the problem more directly, such as the _Maximum Mean Discrepancy (MMD) test_, though not necessarily with greater success (see, _e.g._, [23]).

### Neural Networks

We now recall the basics of _neural networks_ (NN), which will be our main object of study. We define a neural network2 as a (finite) sequence of functions called _layers_\(f_{1},\ldots,f_{L}\) of the form \(f_{\ell}\colon\mathbb{R}^{n_{\ell}}\to\mathbb{R}^{n_{\ell+1}},x\mapsto \sigma_{\ell}(W_{\ell}\cdot x+b_{\ell})\), where the parameters \(W_{\ell}\in\mathbb{R}^{n_{\ell+1}\times n_{\ell}}\) and \(b_{\ell}\in\mathbb{R}^{n_{\ell+1}}\) are called the weight matrix and the bias vector respectively, and \(\sigma_{\ell}\) is an (element-wise) activation map (_e.g._, sigmoid or ReLU). The neural network encodes a map \(F\colon\mathbb{R}^{d}\to\mathbb{R}^{D}\) given by \(F=f_{L}\circ\cdots\circ f_{1}\). We sometimes use \(F\) to refer to the neural network as a whole, though it has more structure.

Footnote 2: While our exposition is restricted to sequential neural networks for the sake of concision, our representations are well-defined for other types of neural nets (_e.g._, recurrent neural nets).

When the neural network is used as a classifier, the last activation function \(\sigma_{L}\) is often taken to be the _softmax_ function, so that \(F(x)_{i}\) can be interpreted as the confidence that the network has in \(x\) belonging to the \(i\)-th class, for \(i=1,\ldots,D\). For this reason, we use the terminology _confidence vector_ (CV) for the output \(F(x)\in\mathbb{R}^{D}\). The true class of \(x\) is represented by a label \(y=(0,\ldots,0,1,0,\ldots,0)\in\mathbb{R}^{D}\) that takes value \(1\) at the coordinate indicating the correct class and \(0\) elsewhere. The parameters of each layer \((W_{\ell},b_{\ell})\) are typically learned from a collection of training observations and labels \(\{(x_{n},y_{n})\}_{n=1}^{N}\) by minimizing a cross-entropy loss through gradient descent, in order to make \(F(x_{n})\) as close to \(y_{n}\) as possible on average over the training set. The _prediction_ of the network on a new observation \(x\) is then given by \(\arg\max_{i=1,\ldots,D}F(x)_{i}\), and its (test) _accuracy_ is the proportion of correct predictions on a new set of observations \(\{(x^{\prime}_{n},y^{\prime}_{n})\}_{n=1}^{N^{\prime}}\), that is assumed to have been independently drawn from the same distribution as the training observations.

In this work, we consider NN classifiers that have already been trained on some training data and that achieve reasonable accuracies on test data following the same distribution as training data.

### Activation Graphs

Given an instance \(x=x_{0}\in\mathbb{R}^{d}\), a trained neural network \(f_{1},\ldots,f_{L}\) with \(x_{\ell+1}=f_{\ell}(x_{\ell})=\sigma_{\ell}(W_{\ell}\cdot x_{\ell}+b_{\ell})\) and a layer \(f_{\ell}\colon\mathbb{R}^{n_{\ell}}\longrightarrow\mathbb{R}^{n_{\ell+1}}\), we can define a weighted graph, called the _activation graph_\(G_{\ell}(x)\) of \(x\) for the layer \(f_{\ell}\), as follows. We let \(V\coloneqq V_{\ell}\sqcup V_{\ell+1}\) for two sets \(V_{\ell}\) and \(V_{\ell+1}\) of cardinality \(n_{\ell}\) and \(n_{\ell+1}\) respectively. The edges are defined as \(E\coloneqq V_{\ell}\times V_{\ell+1}\). To each edge
\((i,j)\in E_{\ell}\), we associate the weight \(w_{i,j}(x)\coloneqq W_{\ell}(j,i)\cdot x_{\ell}(i)\), where \(x_{\ell}(i)\) (resp. \(W_{\ell}(j,i)\)) denotes the \(i\)-th coordinate of \(x_{\ell}\in\mathbb{R}^{n_{\ell}}\) (resp. entry \((j,i)\) of \(W_{\ell}\in\mathbb{R}^{n_{\ell+1}\times n_{\ell}}\)). The activation graph \(G_{\ell}(x)\) is the weighted graph \((V,E,\{w_{i,j}(x)\})\), which can be conveniently represented as a \(n_{\ell}\times n_{\ell+1}\) matrix whose entry \((i,j)\) is \(w_{i,j}(x)\). Intuitively, these activation graphs--first considered in [6]--represent how the network "reacts" to a given observation \(x\) at inner-level, rather than only considering the network output (_i.e._, the Confidence Vector).

## 4 Two-Sample Statistical Tests using MAGDiff

### The MAGDiff representations

Let \(\mathbb{P}_{0}\) and \(\mathbb{P}_{1}\) be two distributions for which we want to test \(H_{0}:\mathbb{P}_{0}=\mathbb{P}_{1}\). As mentioned above, two-sample statistical tests tend to underperform when used directly on high-dimensional data. It is thus common practice to extract lower-dimensional representations \(\Psi(x)\) from the data \(x\sim\mathbb{P}_{i}\), where \(\Psi\colon\operatorname{supp}\,\mathbb{P}_{0}\cup\operatorname{supp}\, \mathbb{P}_{1}\to\mathbb{R}^{N}\). Given a classification task with classes \(1,\ldots,D\), we define a family of such representations as follows. Let \(T\colon\operatorname{supp}\,\mathbb{P}_{0}\cup\operatorname{supp}\,\mathbb{P }_{1}\to V\) be any map whose codomain \(V\) is a Banach space with norm \(\|\cdot\|_{V}\). For each class \(i\in\{1,\ldots,D\}\), let \(\mathbb{P}_{0,i}\) be the conditional distribution of data points from \(\mathbb{P}_{0}\) in class \(i\). We define

\[\Psi_{i}(x)\coloneqq\|T(x)-\mathbb{E}_{\mathbb{P}_{0,i}}[T(x^{\prime})]\|_{V}\]

for \(x\in\operatorname{supp}\,\mathbb{P}_{0}\cup\operatorname{supp}\,\mathbb{P}_{1}\). Given a fixed finite dataset \(x_{1},\ldots,x_{m}\stackrel{{\text{iid}}}{{\sim}}\mathbb{P}_{0}\), we similarly define the approximation

\[\tilde{\Psi}_{i}(x)\coloneqq\left\|T(x)-\frac{1}{m_{i}}\sum_{j=1}^{m_{i}}T(x _{j}^{i})\right\|_{V}\,,\]

where \(x_{1}^{i},\ldots,x_{m_{i}}^{i}\) are the points of the dataset whose class is \(i\). This defines a map \(\tilde{\Psi}\colon\operatorname{supp}\,\mathbb{P}_{0}\cup\operatorname{supp} \,\mathbb{P}_{1}\to\mathbb{R}^{D}\).

The map \(T\colon\operatorname{supp}\,\mathbb{P}_{0}\cup\operatorname{supp}\,\mathbb{P }_{1}\to V\) could _a priori_ take many shapes. In this article, we assume that we are provided with a neural network \(F\) that has been trained for the classifying task at hand, as well as a training set drawn from \(\mathbb{P}_{0}\). We let \(T\) be the activation graph \(G_{\ell}\) of the layer \(f_{\ell}\) of \(F\) represented as a matrix, so that the expected values \(\mathbb{E}_{\mathbb{P}_{0,i}}[G_{\ell}(x^{\prime})]\) (for \(i=1,\ldots,D\)) are simply mean matrices, and the norm \(\|\cdot\|_{V}\) is the Frobenius norm \(\|\cdot\|_{2}\). We call the resulting \(D\)-dimensional representation _Mean Activation Graph Difference_ (MAGDiff):

\[\texttt{MAGDiff}(x)_{i}\coloneqq\left\|G_{\ell}(x)-\frac{1}{m_{i}}\sum_{j=1}^ {m_{i}}G_{\ell}(x_{j}^{i})\right\|_{2}\,,\]

for \(i=1,\ldots,D\), where \(x_{1}^{i},\ldots,x_{m_{i}}^{i}\) are, as above, samples of the training set whose class is \(i\). Therefore, for a given new observation \(x\), we derive a vector MAGDiff\((x)\in\mathbb{R}^{D}\) whose \(i\)-th coordinate indicates whether \(x\) activates the chosen layer of the network in a similar way "as training observations of the class \(i\)".

Many variations are possible within that framework. One could, _e.g._, consider the activation graph of several consecutive layers, use another matrix norm, or apply Topological Data Analysis techniques to compute a more compact representation of the graphs, such as the _topological uncertainty_[18]. In this work, we focus on MAGDiff for dense layers, though it could be extended to other types.

### Comparison of distributions of features with multiple KS tests

Given as above a (relatively low-dimensional) representation \(\Psi\colon\operatorname{supp}\,\mathbb{P}_{0}\cup\operatorname{supp}\, \mathbb{P}_{1}\to\mathbb{R}^{N}\) and samples \(x_{1},\ldots,x_{n}\stackrel{{\text{iid}}}{{\sim}}\mathbb{P}_{0}\) and \(x_{1}^{\prime},\ldots,x_{m}^{\prime}\stackrel{{\text{iid}}}{{\sim}} \mathbb{P}_{1}\), one can apply multiple univariate (coordinate-wise) KS tests with Bonferroni correction to the sets \(\Psi(x_{1}),\ldots,\Psi(x_{n})\) and \(\Psi(x_{1}^{\prime}),\ldots,\Psi(x_{m}^{\prime})\), as described in Section 3. If \(\Psi\) is well-chosen, a difference between the distributions \(\mathbb{P}_{0}\) and \(\mathbb{P}_{1}\) (hard to test directly due to the dimensionality of the data) will translate to a difference between the distributions \(\Psi(x)\) and \(\Psi(x^{\prime})\) for \(x\sim\mathbb{P}_{0}\) and \(x^{\prime}\sim\mathbb{P}_{1}\) respectively. Detecting such a difference serves as a proxy for testing \(H_{0}:\mathbb{P}_{0}=\mathbb{P}_{1}\). In our experiments, we apply this procedure to the
MAGDiff representations defined above (see Section 5.1 for a step-by-step description). This is a reasonable approach, as it is a simple fact that a generic shift in the distribution of the random variable \(x\sim\mathbb{P}_{0}\) will in turn induce a shift in the distribution of \(\Psi(x)\), as long as \(\Psi\) is not constant3; however, this does not give us any true guarantee, as it does not provide any quantitative result regarding the shift in the distribution of \(\Psi(x)\). Such results are beyond the scope of this paper, in which we focus on the good experimental performance of the MAGDiff statistic.

Footnote 3: See Appendix C for an elementary proof.

### Differences from BBSD and motivations

The BBSD method described in [20] and [22] is defined in a similar manner, except that the representations \(\Psi\) on which the multiple univariate KS tests are applied are simply the Confidence Vectors (CV) \(F(x)\in\mathbb{R}^{D}\) of the neural network \(F\) (or of any other classifier that outputs confidence vectors), rather than our newly proposed MAGDiff representations. In other words, they detect shifts in the distribution of the inputs by testing for shifts in the distribution of the outputs of a given classifier4.

Footnote 4: This corresponds to the best-performing variant of their method, denoted as _BBSDs_ (as opposed to, _e.g._, _BBSDh_) in [22].

Both our method and theirs share advantages: the features are task- and data-driven, as they are derived from a classifier that was trained for the specific task at hand. They do not require the design or the training of an additional model specifically geared towards shift detection, and they have favorable algorithmic complexity, especially compared to some kernel-based methods. In particular, combining the KS tests with the Bonferroni correction spaces us from having to calibrate our statistical tests with a permutation test, which can be costly as shown in [22]. A common downside is that the Bonferroni correction can be overly conservative; other tests might offer higher power. The main focus of this article is the relevance of the MAGDiff representations, rather than the statistical tests that we apply to them, and it has been shown in [22] that KS tests yield state-of-the-art performances; as such, we did not investigate alternatives, though additional efforts in that direction might produce even better results.

The nature of the construction of the MAGDiff representations is geared towards shift detection since it is directly based on encoding differences (_i.e._, deviations) from the mean activation graphs (of \(\mathbb{P}_{0}\)). Moreover, they are based on representations from deeper within the NN, which are less compressed than the confidence vectors - passing through each layer leads to a potential loss of information regarding the differences between the training and the test distributions. Hence, we can hope for the MAGDiff to encode more information from the input data than the CV representations used in [22] which focus on the class to which a sample belongs to, while sharing the same favorable dimensionality reduction properties. Therefore, we expect MAGDiff to perform particularly well with covariate shifts, where shifts in the distribution of the data do not necessarily translate to strong shifts in the distribution of the CV. Conversely, we do not hope for our representations to bring significant improvements over CV in the specific case of label shifts; all the information relative to labels available to the network is, in a sense, best summarized in the CV, as this is the main task of the NN. These expectations were confirmed in our experiments.

## 5 Experiments

This experimental section is devoted to showcasing the use of the MAGDiff representations and its benefits over the well-established baseline CV when it comes to performing covariate shift detection. As detailed in Section 5.1, we combine coordinate-wise KS tests for both these representations. Note that in the case of CV, this corresponds exactly to the method termed _BBSDs_ in [22]. Our code is publicly available5.

Footnote 5: [https://github.com/hensel-f/MAGDiff_experiments](https://github.com/hensel-f/MAGDiff_experiments)

### Experimental Settings

To illustrate the versatility of our approach, we ran our experiments on various datasets, for various network architectures, various type of shifts along with various levels of intensity. A comprehensive presentation of these datasets and parameters can be found in the Appendix.
Datasets.We consider the standard datasets MNIST [19], FashionMNIST (FMNIST) [33], CIFAR-10 [17], SVHN [21], as well as a lighter version of ImageNet (restricted to \(10\) classes) called Imagenette [1].

Architectures.For MNIST and FMNIST, we used a simple CNN architecture consisting of 3 convolutional layers followed by 4 dense layers. For CIFAR-10 and SVHN, we considered (a slight modification, to account for input images of size \(32\times 32\), of) the ResNet18 architecture [9]. For Imagenette, we used a pretrained ResNet18 model provided by Pytorch [2]. With these architectures, we reached a test accuracy of \(98.6\%\) on MNIST, \(91.1\%\) on FMNIST, \(94.1\%\) on SVHN, \(81\%\) on CIFAR-10 and \(99.2\%\) for Imagenette, validating the "well-trained" assumption mentioned in Section 4. Note that we used simple architectures, without requiring the networks to achieve state-of-the-art accuracy.

Shifts.We applied three types of shift to our datasets: Gaussian noise (additive white noise), Gaussian blur (convolution by a Gaussian distribution), and Image shift (random combination of rotation, translation, zoom and shear), for six different levels of increasing intensities (denoted by I, II,...,VI), and a fraction of shifted data \(\delta\in\{0.25,0.5,1.0\}\). For each dataset and shift type, we chose the shift intensities in such a manner that the shift detection for the lowest intensities and low \(\delta\) is almost indetectable for both methods (MAGDiff and CV), and very easily detectable for high intensities and values of \(\delta\). Details (including the impact of the shifts on model accuracy) and illustrations can be found in Appendix B.

Sample size.We ran the shift detection tests with sample sizes6\(\{10,20,50,100,200,500,1000\}\) to assess how many samples a given method requires to reliably detect a distribution shift. A good method should be able to detect a shift with as few samples as possible.

Footnote 6: That is, the number of elements from the clean and shifted sets on which the statistical tests are performed; see the paragraph **Experimental protocol** for more details.

Experimental protocol.In all of the experiments below, we start with a neural network that is pre-trained on the training set of a given dataset. The test set will be referred to as the _clean set_ (CS). We then apply the selected shift (type, intensity, and proportion \(\delta\)) to the clean set and call the resulting set the _shifted set_\(SS\); it represents the target distribution \(\mathbb{P}_{1}\) in the case where \(\mathbb{P}_{1}\neq\mathbb{P}_{0}\).

As explained in Section 4, for each of the classes \(i=1,\ldots,D\) (for all of our datasets, \(D=10\)), we compute the mean activation graph of a chosen dense layer \(f_{\ell}\) of (a random subset of size \(1000\) of all) samples in the training set whose class is \(i\); this yields \(D\) mean activation graphs \(G_{1},\ldots,G_{D}\). We compute for each sample \(x\) in \(CS\) and each sample in \(SS\) the representation MAGDiff\((x)\), where MAGDiff\((x)_{i}=\|G_{\ell}(x)-G_{i}\|_{2}\) for \(i=1,\ldots,D\) and \(G_{\ell}(x)\) is the activation graph of \(x\) for the layer \(f_{\ell}\) (as explained in Section 4). Doing so, we obtain two sets \(\{\texttt{MAGDiff}(x)\mid x\in CS\}\) and \(\{\texttt{MAGDiff}(x^{\prime})\mid x^{\prime}\in SS\}\) of \(D\)-dimensional features; both have the same cardinality as the test set.

Now, we estimate the power of the test for a given sample size7\(m\) and for a type I error of at most \(0.05\); in other words, the probability that the test rejects \(H_{0}\) when \(H_{1}\) is true and when it has access to only \(m\) samples from the respective datasets, and under the constraint that it does not falsely reject \(H_{0}\) in more than \(5\%\) of cases. To do so, we randomly sample (with replacement) \(m\) elements \(x^{\prime}_{1},\ldots,x^{\prime}_{m}\) from \(SS\), and consider for each class \(i=1,\ldots,D\) the discrete empirical univariate distribution \(q_{i}\) of the values MAGDiff\((x^{\prime}_{1})_{i},\ldots,\texttt{MAGDiff}(x^{\prime}_{m})_{i}\). Similarly, by randomly sampling \(m\) elements from \(CS\), we obtain another discrete univariate distribution \(p_{i}\) (see Figure 1 for an illustration). Then, for each \(i=1,\ldots,D\), the KS test is used to compare \(p_{i}\) and \(q_{i}\) to obtain a \(p\)-value \(\lambda_{i}\), and reject \(H_{0}\) if \(\min(\lambda_{1},\ldots,\lambda_{D})>\alpha/D\), where \(\alpha\) is the threshold for the univariate KS test at confidence \(0.05\) (_cf._ Section 3.1). Following standard bootstrapping protocol, we repeat that experiment (independently sampling \(m\) points from \(CS\) and \(SS\), computing \(p\)-values, and possibly rejecting \(H_{0}\)) \(1500\) times; the percentage of rejection of \(H_{0}\) is the estimated _power_ of the statistical test (since \(H_{0}\) is false in this scenario). We use the asymptotic normal distribution of the standard Central Limit Theorem to compute approximate \(95\%\)-confidence intervals on our estimate.

Footnote 7: The same sample size that is mentioned in the **Sample size** paragraph.

To illustrate that the test is well calibrated, we repeat the same procedure while sampling twice \(m\) elements from \(CS\) (rather than \(m\) elements from \(SS\) and \(m\) elements from \(CS\)), which allows us to
estimate the type I error (_i.e._, the percentage of incorrect rejections of \(H_{0}\)) and assert that it remains below the significance level of \(5\%\) (see, _e.g._, Figure 2).

We experimented with a few variants of the MAGDiff representations: we tried reordering the coordinates of each vector \(\texttt{MAGDiff}(x)\in\mathbb{R}^{D}\) in increasing order of the value of the associated confidence vectors. We also tried replacing the matrix norm of the difference to the mean activation graph by either its Topological Uncertainty (TU) [18], or variants thereof. Early analysis suggested that these variations did not bring increased performances, despite their increased complexity. Experiments also suggested that MAGDiff representations brought no improvement over CV in the case of label shift. We also tried to combine (_i.e._, concatenate) the CV and MAGDiff representations, but the results were unimpressive, which we attribute to the Bonferroni correction getting more conservative the higher the dimension. We thus only report the results for the standard MAGDiff.

Competitor.We used multiple univariate KS tests applied to CV (the method BBSDs from [22]) as the baseline, which we denote by "CV" in the figures and tables, in contrast to our method denoted by "MAGDiff". The similarity in the statistical testing between BBSDs and MAGDiff allows us to easily assess the relevance of the MAGDiff features. We chose them as our sole competitors as it has been convincingly shown in [22] that they outperform on average all other standard methods, including the use of dedicated dimensionality reduction models, such as autoencoders, or of multivariate kernel tests. Many of these methods are also either computationally more costly (to the point where they cannot be practically applied to more than a thousand samples) or harder to implement (as they require an additional neural network to be implemented) than both BBSDs and MAGDiff.

### Experimental results and influence of parameters.

We now showcase the power of shift detection using our MAGDiff representations in various settings and compare it to the state-of-the-art competitor CV. Since there were a large number of hyper-parameters in our experiments (datasets, shift types, shift intensities, etc.), we started with a standard set of hyper-parameters that yielded representative and informative results according to our observations (MNIST and Gaussian noise, as in [22], \(\delta=0.5\), sample size \(100\), MAGDiff computed with the last layer of the network) and let some of them vary in the successive experiments. We focus on the well-known MNIST dataset to allow for easy comparisons, and refer to Appendix B for additional experimental results that confirm our findings on other datasets.

Sample size.The first experiment consists of estimating the power of the shift detection test as a function of the sample size (a common way of measuring the performance of such a test) using either the MAGDiff or the baseline CV representations. Figure 2 shows the powers of the KS tests using the MAGDiff (red curve) and CV (green curve) representations with respect to the sample size for the MNIST dataset. Here, we choose to showcase the results for Gaussian noise of intensities II, IV and IV with shift proportion \(\delta=0.5\).

Figure 1: Empirical distributions of \(\texttt{MAGDiff}_{1}\) for the \(10,000\) samples of the clean and shifted sets (MNIST, Gaussian noise, \(\delta=0.5\), last dense layer). For the clean set, the distribution of the component MAGDiff1 of \(\texttt{MAGDiff}\) exhibits a peak close to \(0\). This corresponds to those samples whose distance to the mean activation graph of (training) samples belonging to the associated class is very small, _i.e._, these are samples that presumably belong to the same class as well. Note that, for the shifted set, this peak close to \(0\) is substantially diminished, which indicates that the activation graph of samples affected by the shift is no longer as close to the mean activation graph of their true class.
It can clearly be seen that MAGDiff consistently and significantly outperformed the CV representations. While in both cases, the tests achieved a power of \(1.0\) for large sample sizes (\(m\approx 1000\)) and/or high shift intensity (VI), MAGDiff was capable of detecting the shift even with much lower sample sizes. This was particularly striking for the low intensity level II, where the test with CV was completely unable to detect the shift, even with the largest sample size, while MAGDiff was capable of reaching non-trivial power already for a medium sample size of \(100\) and exceptional power for large sample size. Note that the tests were always well-calibrated. That is, the type I error remained below the significance level of \(0.05\), indicated by the horizontal dashed black line in the figures.

To further support our claim that MAGDiff outperforms CV on average in other scenarios, we provide, in Table 1, averaged results over all parameters except the sample size. Though the precise values obtained are not particularly informative (due to the aggregation over very different sets of hyper-parameters), the comparison between the two rows remains relevant. In Appendix B, a more comprehensive experimental report (including, in particular, the CIFAR-10 and Imagenette datasets) further supports our claims.

Shift intensity.The first experiment suggests that MAGDiff representations perform particularly well when the shift is hard to detect. In the second experiment, we further investigate the influence of the shift intensity level and \(\delta\) (which is, in a sense, another measure of shift intensity) on the power of the tests. We chose a fixed sample size of \(100\), which was shown to make for a challenging yet

\begin{table}
\begin{tabular}{l c c c c c c c} \multicolumn{8}{c}{Averaged power (\%)} \\ \hline Sample size & \(10\) & \(20\) & \(50\) & \(100\) & \(200\) & \(500\) & \(1000\) \\ \hline \hline MAGDiff & \(7.4\) & \(17.1\) & \(27.6\) & \(40.7\) & \(54.7\) & \(71.4\) & \(80.4\) \\ CV & \(4.0\) & \(9.8\) & \(15.6\) & \(24.7\) & \(35.3\) & \(49.7\) & \(59.2\) \\ \hline \end{tabular}
\end{table}
Table 1: Averaged test power of MAGDiff and CV over all hyper-parameters except sample size (dataset, shift type, \(\delta\), shift intensity). A \(95\%\)-confidence interval for the averaged powers has been estimated via bootstrapping and is, in all cases, strictly contained in a \(\pm 0.1\%\) interval.

Figure 3: Power and type I error of the test with MAGDiff (red) and CV (green) features w.r.t. the shift intensity for Gaussian noise on the MNIST dataset with sample size \(100\) and \(\delta=0.25\) (left), \(\delta=0.5\) (middle), \(\delta=1.0\) (right), for the last dense layer. The estimated \(95\%\)-confidence intervals are displayed around the curves.

Figure 2: Power and type I error of the statistical test with MAGDiff (red) and CV (green) representations w.r.t. sample size (on a log-scale) for three different shift intensities (II, IV, VI) and fixed \(\delta=0.5\) for the MNIST dataset, Gaussian noise and last layer of the network, with estimated \(95\%\)-confidence intervals.
doable task. The results in Figure 3 confirm that our representations were much more sensitive to weak shifts than the CV, with differences in power greater than \(80\%\) for some intensities.

Shift type.The previous experiments focused on the case of Gaussian noise; in this experiment, we investigate whether the results hold for other shift types. As detailed in Table 2, the test with MAGDiff representations reacted to the shifts even for low intensities of I, II, and III for all shift types (Gaussian blur being the most difficult case), while the KS test with CV was unable to detect anything. For medium to high intensities III, IV, V and VI, MAGDiff again significantly outperformed the baseline and reaches powers close to \(1\) for all shift types. For the Gaussian blur, the shift remained practically undetectable using CV.

MAGDiff with respect to different layers.The NN architect we used with MNIST and FMNIST consisted had several dense layers before the output. As a variation of our method, we investigate the effect on the shift detection when computing our MAGDiff representations with respect to different layers8. More precisely, we consider the last three dense layers denoted by \(\ell_{-1}\), \(\ell_{-2}\) and \(\ell_{-3}\), ordered from the closest to the network output (\(\ell_{-1}\)) to the third from the end (\(\ell_{-3}\)). The averaged results over all parameters and noise types are reported in Table 3.

Footnote 8: Since ResNet18 only has a single dense layer after its convolutional layers, there is no choice to be made in the case of CIFAR-10, SVHN and Imagenette.

In the case of MNIST we only observe a slight increase in power when considering layer \(\ell_{-3}\) further from the output of the NN. In the case of FMNIST, on the other hand, we clearly see a much more pronounced improvement when switching from \(\ell_{-1}\) to \(\ell_{-3}\). This hints at the possibility that features derived from encodings further within the NN can, in some cases, be more pertinent to the task of shift detection than those closer to the output.

## 6 Conclusion

In this article, we introduce novel types of representations, called MAGDiff, which are derived from the activation graphs of a trained NN classifier. We empirically show that using MAGDiff representations for the task of data set shift detection via coordinate-wise KS tests (with Bonferroni correction) significantly outperforms the baseline given by using confidence vectors (CV) of the NN established in [20], while remaining equally fast and easy to implement. Thus, our MAGDiff representations are an effective method for encoding data- and task-driven information of the input data that is highly relevant for the critical challenge of dataset shift detection in data science.

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \multicolumn{8}{c}{**Impact of Shift Type**} \\ \hline \multicolumn{8}{c}{Power of the test (\%)} \\ \hline \multirow{2}{*}{Shift} & \multirow{2}{*}{Feat.} & \multicolumn{4}{c}{Shift intensity} \\ \cline{3-8}  & & I & II & III & IV & V & VI \\ \hline \hline \multirow{2}{*}{GN} & \(\mathbb{MD}\) & \(7.2\pm 1.3\) & \(29.3\pm 2.3\) & \(61.9\pm 2.5\) & \(93.3\pm 1.3\) & \(98.9\pm 0.5\) & \(100.0-0.0\) \\  & CV & \(0.0+0.2\) & \(0.1\pm 0.1\) & \(1.5\pm 0.6\) & \(17.6\pm 1.9\) & \(72.3\pm 2.3\) & \(98.9\pm 0.5\) \\ \hline \multirow{2}{*}{GB} & \(\mathbb{MD}\) & \(3.7\pm 1.0\) & \(4.3\pm 1.0\) & \(27.7\pm 2.3\) & \(63.1\pm 2.4\) & \(85.0\pm 1.8\) & \(92.4\pm 1.3\) \\  & CV & \(0.0+0.2\) & \(0.0+0.2\) & \(0.0+0.2\) & \(0.4\pm 0.3\) & \(1.3\pm 0.6\) & \(5.3\pm 1.1\) \\ \hline \multirow{2}{*}{IS} & \(\mathbb{MD}\) & \(10.3\pm 1.5\) & \(32.7\pm 2.4\) & \(53.5\pm 2.5\) & \(78.5\pm 2.1\) & \(90.6\pm 1.5\) & \(98.9\pm 0.5\) \\  & CV & \(0.0+0.2\) & \(0.1\pm 0.2\) & \(2.1\pm 0.7\) & \(11.5\pm 1.6\) & \(37.0\pm 2.4\) & \(86.3\pm 1.7\) \\ \hline \end{tabular}
\end{table}
Table 2: Power of the two methods (our method, denoted as MD, and CV) as a function of the shift intensity for the shift types Gaussian noise (GN), Gaussian blur (GB) and Image shift (IS) on the MNIST dataset with \(\delta=0.5\), sample size \(100\), for the last dense layer. Red indicates that the estimated power is below \(10\%\), blue that it is above \(50\%\). The \(95\%\)-confidence intervals have been estimated as mentioned in Section 5.

\begin{table}
\begin{tabular}{c c c c c} \multicolumn{4}{c}{**Choice of Layer**} \\ \multicolumn{4}{c}{Averaged power (\%)} \\ \hline \multirow{2}{*}{Dataset} & \multicolumn{4}{c}{Features} \\ \cline{2-5}  & CV & \(\ell_{-1}\) & \(\ell_{-2}\) & \(\ell_{-3}\) \\ \hline \hline MNIST & \(25.1\) & \(51.9\) & \(53.0\) & \(56.4\) \\ \hline FMNIST & \(46.2\) & \(44.9\) & \(47.6\) & \(53.7\) \\ \hline \end{tabular}
\end{table}
Table 3: Averaged performance of the various layers for MAGDiff over all other parameters (for MNIST and FMNIST), compared to BBSD with CV. A \(95\%\) confidence interval for the averaged powers has be estimated and is in all cases strictly contained in a \(\pm 0.1\%\) interval.
Our findings open many avenues for future investigations. We focused on classification of image data in this work, but our method is a general one and can be applied to other settings. Moreover, adapting our method to regression tasks as well as to settings where shifts occur gradually is feasible and a starting point for future work. Finally, exploring variants of the MAGDiff representations--considering several layers of the network at once, extending it to other types of layers, extracting finer topological information from the activation graphs, weighting the edges of the graph by backpropagating their contribution to the output, etc.--could also result in increased performance.

Acknowledgments.The authors are grateful to the OPAL infrastructure from Universite Cote d'Azur for providing resources and support. FC thanks the ANR TopAI chair (ANR-19-CHIA-0001) for financial support and the DATAIA Institute.
