Quantifying Overfitting: Evaluating Neural Network Performance through
  Analysis of Null Space
Machine learning models that are overfitted/overtrained are more vulnerable
to knowledge leakage, which poses a risk to privacy. Suppose we download or
receive a model from a third-party collaborator without knowing its training
accuracy. How can we determine if it has been overfitted or overtrained on its
training data? It's possible that the model was intentionally over-trained to
make it vulnerable during testing. While an overfitted or overtrained model may
perform well on testing data and even some generalization tests, we can't be
sure it's not over-fitted. Conducting a comprehensive generalization test is
also expensive. The goal of this paper is to address these issues and ensure
the privacy and generalization of our method using only testing data. To
achieve this, we analyze the null space in the last layer of neural networks,
which enables us to quantify overfitting without access to training data or
knowledge of the accuracy of those data. We evaluated our approach on various
architectures and datasets and observed a distinct pattern in the angle of null
space when models are overfitted. Furthermore, we show that models with poor
generalization exhibit specific characteristics in this space. Our work
represents the first attempt to quantify overfitting without access to training
data or knowing any knowledge about the training samples.
Hossein Rezaei, Mohammad Sabokrou
# Quantifying Overfitting: Evaluating Neural Network Performance through Analysis of Null Space

###### Abstract

Machine learning models that are overfitted/overtrained are more vulnerable to knowledge leakage, which poses a risk to privacy. Suppose we download or receive a model from a third-party collaborator without knowing its training accuracy. How can we determine if it has been overfitted or overtrained on its training data? It's possible that the model was intentionally over-trained to make it vulnerable during testing. While an overfitted or overtrained model may perform well on testing data and even some generalization tests, we can't be sure it's not over-fitted. Conducting a comprehensive generalization test is also expensive. The goal of this paper is to address these issues and ensure the privacy and generalization of our method using only testing data. To achieve this, we analyze the null space in the last layer of neural networks, which enables us to quantify overfitting without access to training data or knowledge of the accuracy of those data. We evaluated our approach on various architectures and datasets and observed a distinct pattern in the angle of null space when models are overfitted. Furthermore, we show that models with poor generalization exhibit specific characteristics in this space. Our work represents the first attempt to quantify overfitting without access to training data or knowing any knowledge about the training samples. 1

Footnote 1: The source code will be available after the review.

## 1 Introduction

Deep learning models have been very successful in many applications such as computer vision, natural language processing, and speech recognition. These models are trained on large amounts of data and have demonstrated outstanding performance in tasks such as image classification, object detection, and language translation [1; 2]. However, despite their effectiveness, ensuring the privacy and trustworthiness of deep learning models remains a significant challenge [3; 4; 5].

In today's data-driven world, accessing pre-trained models has become increasingly common, whether obtained from the internet or delivered by third-party companies. However, it is crucial to ensure that these models uphold privacy standards and do not possess knowledge leakages. A key factor in determining the vulnerability of a model to membership inference attacks is the presence of overfitting. Generally, the more overfitted a model is, the more susceptible it becomes to such attacks. However, assessing this characteristic becomes challenging when we lack information about the
model's training accuracy or training data. In this paper, we aim to address this critical question and explore potential solutions for evaluating model vulnerability in situations where these crucial details are unavailable (see Fig. 1).

Generally, one of the key concerns in deep learning is that the models often memorize the training data [6; 7]. This means that the models may overfit the training data, resulting in poor generalization to new data. Additionally, the models may memorize sensitive information from the training data, which can pose a risk to privacy [8]. For instance, if a model has a privacy leakage, attackers may be able to extract sensitive information from the model during inference [9; 10]. If an attacker gains access to such information, it can have severe consequences for individuals or organizations.

A primary factor that makes a deep learning model vulnerable to privacy breaches is overfitting [9]. Researchers have proposed various methods to address this issue and enhance the privacy and trustworthiness of deep learning models. For example, [11; 12] leveraged differentially private training, [13; 14] exploited gradient clipping, and [15; 16] utilized machine unlearning to improve privacy and prevent knowledge leakage.

One of the simplest ways to detect overfitting is by comparing the accuracy of the model on the training and testing datasets. If the model achieves high accuracy on the training data but low accuracy on the testing data (low bias and high variance), it may be overfitting [7; 17]. However, obtaining the accuracy of the training data requires access to the training dataset, which may not always be feasible or ethical. Some papers, like [18; 10; 9], attempt to measure forgetting and memorization by conducting attacks while relying on training data to accomplish this task.

Another approach to detecting overfitting is by performing a generalization/robustness evaluation [19; 20; 21]. This test evaluates how well the model can generalize to new data by measuring its performance on a separate dataset that it has not seen before. If the model performs well on the generalization test, it is less likely to suffer from overfitting. However, this approach has some drawbacks. Firstly, it can be costly to implement, as it requires collecting several separate datasets for the generalization test and it takes too much time to perform multiple inferences from the model. Secondly, since generalization tests are widely known, an attacker could overtrain/overfit the model on those tests, making the model robust to those specific tests and potentially opening the door for privacy breaches.

In addition to the aforementioned methods, another simple approach to investigate the issue of overfitting is to examine the uncertainty of the model by analyzing the soft-max output or logits [22]. The idea is that probably there is a direct relationship between the uncertainty of the model and the overfitting. However, in Section 4.2, we demonstrate that this argument does not always hold true.

To address these challenges, we propose a novel method to detect overfitting and ensure the privacy and generalization of deep learning models using only a small amount of test data. The proposed method involves analyzing the null space in the last layer of neural networks, which enables us to quantify overfitting without access to the training data or knowledge of its accuracy. The null space is the set of all vectors that the neural network maps to zero. We interestingly find that by analyzing the angle between the null space of weights and the representation, can give us supervision to detect overfitting and determine the generalization performance of the model. The proposed method has been evaluated on various architectures and datasets, and the results show that there is a distinct pattern in the angle of the null space when models are overfitted. Furthermore, we illustrate that

Figure 1: We possess two downloaded models with no information regarding their training data or training accuracy. Additionally, we only have access to a limited subset of test data. Despite model 1 exhibiting higher test accuracy, it appears to suffer from overfitting, and existing current methods are unable to effectively discern this issue.
models exhibiting poor generalization display specific characteristics within this space. _The proposed method represents one of the first attempts to quantify overfitting without access to training data or any knowledge about the training samples_. The method is easy to implement and can be applied to various architectures and datasets, making it a promising tool to enhance the privacy of deep learning models.

## 2 Related Work

In this section, we provide a review delving into the literature attempting to measure the overfitting and generalization capability of machine learning models. Furthermore, we explore several works that leverage null space across various applications using neural networks.

### Overfitting & Generalization

Overfitting arises when a model becomes too complex and memorizes the training data instead of learning the representative patterns, resulting in failure to generalize well to unseen datasets. To address this issue, Werbachowski et al. [23] introduce a non-intrusive statistical test using adversarial examples to detect test set overfitting in machine learning models. Yet, one notable challenge they highlighted is accurately measuring test set overfitting due to shifts in data distribution. Moreover, Jagielski et al. [9] explored memorization, forgetting, and their impact on overfitting via a privacy attack method. They train two models with extra data to measure forgetting, using the success rate to identify retained sensitive information and discarded irrelevant or noisy information. Carlini et al. [10] consider a testing approach that evaluates the level of risk associated with generative sequence models inadvertently memorizing infrequent or distinct training data sequences. To assess the level of overfitting in convolutional neural networks (CNNs), PHOM [24] employs trained network weights to create clique complexes on CNN layers. By examining co-adaptations among neurons via one-dimensional persistent homology (PH), it detects overfitting without relying on training data. PHOM differs from our work in terms of efficiency and complexity.

Generalization and Out-Of-Distribution (OOD) generalization refer to our model's ability to adapt appropriately to new, previously unseen data drawn from either the same distribution or a different distribution as the training data, respectively. To address this issue, Neyshabur et al. [25] connect sharpness to PAC-Bayes theory and show that expected sharpness, a measure of network output change with input change, along with weight norms, can capture neural network generalization behavior effectively. Some other works attempt to evaluate the generalization of deep networks by defining bounds. For instance, Liang et al. [26] introduce the Fisher-Rao norm, an invariant measure based on information geometry. It quantifies the local inner product on positive probability density functions (PDFs) and relates the loss function to the negative logarithm of conditional probability, with Fisher information as the gradient. Kuang et al. [27], and Shen et al. [28] use the average accuracy to measure OOD generalization. While Duchi et al. [29], and Esfahani et al. [30] measure the OOD generalization using worst-case accuracy.

Unlike current approaches, we propose to measure overfitting and generalization without access to the training data or training accuracy, utilizing only a small subset of the test set to determine the degree of overfitting and generalization capability.

### Null Space

The concept of the null space is important across diverse domains of mathematics, such as linear algebra, differential equations, and control theory. In the context of neural networks, the null space of a weight matrix can be used for various applications. Most research work is focused on analyzing null space for out-of-distribution detection.

In novelty detection, Bodesheim et al. [31] use null space for detecting samples from unknown classes in object recognition by mapping training samples to a single point, enabling joint treatment of multiple classes and novelty detection in one model. IKNDA [32] addressed the demanding computational burden caused by kernel matrix eigendecomposition in this method and performed novelty detection by extracting new information from newly-added samples, integrating it with the existing model, and updating the null space basis to add a single point to the subspace. For outlier detection, Null Space Analysis (NuSA) [33] is proposed to detect outliers in neural networks using
weight matrix null spaces in each layer. It provides competency awareness in ANNs and tackles adversarial data points by controlling null space projection. Likewise, Wang et al. [34] utilize null space to measure out-of-distribution (OOD) degree. by decomposing feature vectors, generating confident outlier images, and subsequently calculating angle-based OOD score. Additionally, Idnani et al. [35] explore the null space's impact on OOD generalization, introducing null space occupancy as a failure mode in neural networks. They optimize network weights using orthogonal gradient descent to reduce null space occupancy, which enhances generalization.

Drawing inspiration from the application of null space in out-of-distribution detection, we employ null space properties to assess both the degree of overfitting and the generalization capacity of machine learning models.

## 3 Proposed Method

The aim of this study is to explore how to detect overfitting in machine learning models without prior knowledge of the training samples or accuracy. We discovered a close correlation between the weights associated with each class and the representation. If the weight for each class is orthogonal to the representation, it means that the input does not belong to that class. Conversely, if the weight is in the same direction as the representation, the angle between them is close to zero, indicating that the input belongs to that class. Although the main concepts of different classes are different, there are some common/shared characteristics between them. Therefore, the angle between the representation and the weight of targeted class should be close to zero, but there should be some gap/angles that reflect the relationship with other classes. We found that when a model is over-fitted or over-trained, it loses of relationship with other classes (the angle between the target class weight and the representation is very close to zero) and less generalization, which can result in overfitting. To apply these findings, we propose monitoring the angle between the weights and the representation during model training. Our approach provides a simple and effective method for detecting overfitting in machine learning models. By using our proposed method, we can detect models that generalize well to new data and avoid overfitting, even without prior knowledge of the training data or accuracy.

Formally speaking, We investigate the concepts of overfitting from a null space perspective. As mentioned, the goal is to determine whether models are overfitted or not and analyze their generalization capability. To accomplish this, suppose \(\mathcal{M}\) is a set of models \(\mathcal{M}=\{m_{1},m_{2},..,m_{k}\}\), and access the test data (or validation samples) \(\mathcal{X}=\{x_{1},x_{2},..,x_{n}\}\). \(\mathcal{X}\) feeds into the network aiming to represent it i.e., \(\mathcal{R}=\{r_{1},r_{2},..,r_{n}\}\) (\(r_{i}\) corresponds the \(x_{i}\) samples). Then we leverage the angle between \(r_{i}\) and the null space of weights that are not associated with ground truth, as well as the angle between \(r_{i}\) and the weights associated with ground truth, to establish two scores for measuring overfitting and generalization. These Scores are defined as follows:

\[\mathcal{O}=\alpha+\beta,\]

\[\mathcal{G}=\frac{\alpha}{max(\alpha)}+\frac{|\beta|}{max(|\beta|)},\]

Where, \(\mathcal{O}\) denotes the degree of overfitting, while \(\mathcal{G}\) represents the amount of generalization capability. \(\alpha\) denotes the average of the angles between the \(\mathcal{R}\) and weight vector of target classes, while \(\beta\) defines the average angles between \(\mathcal{R}\) and the null space of weight vectors (column space) of false classes.

### Null Space & Column Space

In linear algebra, the null space and column space are two fundamental subspaces associated with a matrix. The null space is sometimes called the kernel of the matrix, while the column space is sometimes called the range of the matrix. The null space of an m x n matrix A is a subspace of \(\mathcal{R}^{n}\), written as \(Nul(A)\), and defined by:

\[Nul(A)=\{x\in\mathcal{R}^{n}\,|\,Ax=0\}\]

Where A refers to a linear mapping. Geometrically, the null space represents all the directions in which the matrix A "collapses" to zero. The column space of an m x n matrix A, written as \(Col(A)\)
is a subspace of \(\mathcal{R}^{m}\), and is the set of all linear combinations of the columns of A. In other words, The column space of a matrix A is the span of its columns. Geometrically, the column space represents the "shadow" of the matrix A, as cast onto a lower-dimensional subspace. If \(A=[a_{1},...,a_{n}]\), then

\[Col(A)=Span\{a_{1},...,a_{n}\}\]

Now, it is true that the null space and column space of a matrix are orthogonal complements of each other. This means that any vector in the null space is orthogonal to any vector in the column space, and vice versa. In fact, the left null space is orthogonal to the column space of A. To see why this is true, consider the Appendix A.

### Over-fitting & Generalization Measurement

To analyze overfitting, we split the weight vectors into two groups: The first group for the weights vector of the true class (target class), and the second group for the weights vectors of the false classes. We then analyze the behavior of the representation vector toward these two groups.

Null Space Angle.In deep learning models, we use the inner product. Specifically, we use the following formula:

\[y=w^{T}\cdot x\]

where \(<\cdot>\) represents the inner product between the representation vector and the transpose of weight vectors, and y represents the logits. Since the vectors in the second group represent false classes, their logits should have low values and should not significantly influence the output decision. Therefore, the angle between the weight vectors of false classes (group 2) and \(x\) should be close to 90 degrees. Furthermore, as discussed in [34], the dimensions of the representation vector are typically larger than the dimensions of the logits. This can result in some information loss when the representation vector is fed into the MLP layers. By leveraging the null space, and the behavior of the representation vector toward this space we can potentially recover some of this lost information, which may be useful for analyzing overfitting and generalization.

The space spanned by the vectors in the second group is known as the column space. As previously mentioned, the null space is orthogonal to this space. For the aforementioned reasons, to analyze

Figure 2: Our method is based on a simple framework. We randomly select a subset of the test data to compare the degree of overfitting of the two models. The weights vector of the target class is denoted by \(W_{1}\), while the Null Space plane corresponds to the Null Space associated with the weights vectors of the false classes (\(W_{2}\&W_{3}\)). To compute the degree of overfitting, we first pass the test samples through the encoder to obtain their representation vectors. Next, we measure the angle between the representation vector and the null space plane (\(\beta\)) and the angle between the representation vector and the true class weight vector (\(\alpha\)) ( we perform this process for all samples and ultimately calculate the average). Finally, we compute the sum of \(\alpha\) and \(\beta\), which serves as a quantitative measure of the degree of overfitting.
the relationship between the representation vector and the vectors in the second group, we utilize the concept of null space. In other words, we measured the angle between the null space and the representation vector. In this way, We found that this angle provides us with useful information for analyzing overfitting. In Fig. 2, \(\beta\) represents this angle.

True AngleAs previously discussed in the Null Space Angle section, deep learning models use the inner product to predict the output. Since the output of the network depends on the argmax of Logits/SoftMax, the corresponding logit for the inner product of the representation vector and the vector from group 1 (the weight vector of the target class) should have the maximum value among the other logits. To ensure this, the angle between the representation vector and the vector from group 1 should be close to zero.

We analyzed this angle and found that it provides us with some information about overfitting and generalization. Therefore, we measured it to determine the degree of overfitting and generalization capability. In Fig. 2, \(\alpha\) represents this angle.

OverfittingWe have observed that when the network exhibits good forgetting (the network was not overfitted), it tends to optimize two things simultaneously. Firstly, it minimizes the angle between the representation vector and the correct class weight vector (i.e., the vector associated with group 1) to ensure that the representation vector to what extent is aligned with the correct class. Secondly, it maximizes the angle between the representation vector and the null space.

In other words, the network is trained to adjust its learnable weights and parameters in a way that moves the representation vector away from the null space while simultaneously maximizing its projection onto the vector of group 1. This process leads to a decrease in the value of \(\alpha\) and an increase in the absolute value of \(\beta\) (or a decrease in \(\beta\) itself), resulting in an overall decrease in the sum of \(\alpha\) and \(\beta\). \(\alpha\) represents to what extent the network correctly predicts the label, while \(\beta\) indicates how likely the network considers the representation vector to be similar to other classes.

Therefore, the sum of \(\alpha\) and \(\beta\) serves as an indicator of the degree of overfitting. The lower this value is, the less overfitting the model is.

**Important note:** In fact, the angle between the representation vector and the weights vector of the target class (denoted as \(\alpha\) in Fig. 2) indicates the relationship between the input image and the target class. Meanwhile, the angle between the representation vector and the null space (denoted as \(\beta\) in Fig. 2) reflects the average behavior of the input image towards false classes. For instance, let's consider an example using CIFAR10, where our input sample is a cat. In this case, the representation vector should be close to the weights vector of the cat target class. Furthermore, since there is another class called "dog" in CIFAR10, the representation vector of the cat should be slightly closer to the weights vector of the "dog" class (since the cat and dog bear some resemblance to each other; slightly less than 90 degrees).

On the other hand, classes like "ship" and "truck" have no similarity to the "cat" class. Hence, the angle between the representation vector and the weights vectors of these classes should be slightly greater than 90 degrees. It is worth noting that the angle between the representation vector and the null space (denoted as \(\beta\) in Fig. 2) represents the average behavior of the input image towards false classes. Therefore, on average, the angle between the representation vector and this space should be more than 90 degrees. Due to this reason, as this angle is greater than 90 degrees, it falls on the right side of the space, and we consider this angle as negative.

GeneralizationWe have discovered that when the network possesses good generalization capabilities, it attempts to reduce the projection of the representation vector onto both the null space and the weight vector of the target class. This behavior resembles the concept of overfitting, wherein the network strives to move the representation vector away from the null space. However, in terms of moving the representation vector away from the weight vector of the target class (true class), it opposes overfitting. It is important to note, as demonstrated by the results of various corruptions (as shown in Fig. 4), that a model with less overfitting does not necessarily exhibit superior generalization ability. Hence, we have come to understand that if a model aims to possess both high generalization ability and reduced overfitting While increasing the angle of the representation vector with the null space, it should establish a balance between a small or large angle of the representation vector and
the weights vector of the target class. Essentially, the projection should neither be excessively high nor too low.

## 4 Experiments

We evaluate the performance of our method on several widely-used convolutional neural network architectures, including ResNet18, ResNet34, ResNet50, DenseNet121, VGG19, and MobileNetV2, using three different datasets: CIFAR10, SVHN, and CIFAR100. To conserve space, we present the results for the ResNet18 architecture on CIFAR10 in the main text, with additional experiments provided in Appendix B.

### Setup

CIFAR10.The CIFAR10 dataset is a widely-used image classification dataset comprising 60,000 32x32 color images across 10 classes. To evaluate the performance of our method, we trained 11 different ResNet18 models with and without data augmentation and dropout on this dataset (with different epochs). This allowed us to obtain models with varying generalization and overfitting capabilities. For instance, in Model 8 (as shown in Fig. 3 and Table 1), the first two layers of ResNet18 were utilized with data augmentation and dropout techniques (obtained in epoch 148).

Svhn.The SVHN dataset is another commonly used dataset for image classification tasks consisting of 600,000 labeled digit images. For this dataset, we followed the same methodology as for CIFAR10. Results are available in Appendix B.

Cifar100.The CIFAR100 dataset is similar to CIFAR10, but it differs in the number of classes and images per class. Specifically, it consists of 100 classes, with each class containing 600 images. We trained this data set on ResNet34, ResNet50, DenseNet121, VGG19, and MobileNetV2. And then evaluated our method on these models. The results are available in Appendix B.

### Results

OverfittingAfter training the 11 different ResNet18 models on CIFAR10 we randomly selected a small subset of the test data CIFAR10 and fed it into these models. To assess overfitting, we measured the values of \(\alpha\) and \(\beta\) for these samples and then calculated their average. The results are presented in Table 1 and also plotted in Fig. 3. Our analysis shows that as the degree of overfitting increases, the sum of \(\alpha\) and \(\beta\) also increases, which is reflected in the "\(\mathcal{O}\)" column of Table 1. As shown in Fig. 3, the size of the circles (angles) increases as we move from the bottom to the top or from right to left, indicating an increase in overfitting. Conversely, when we move diagonally from the bottom left to the top right, the size of the circles (angles) decreases (\(\mathcal{O}\) decrease), indicating a decrease in overfitting.

The results for the SVHN and CIFAR100 are available in Appendix B.

Figure 3: (a) shows the results of our method applied to the 11 different ResNet18 models that were trained on CIFAR10. The size of the circles in the plot corresponds to the degree of overfitting, as measured by the ”\(\mathcal{O}\)” values in Table 1. (b) and (c) show the SoftMax and Logit outputs of these models, respectively.
GeneralizationAfter completing the training process for 11 distinct ResNet18 models on CIFAR10, we proceeded to randomly select a small subset of the CIFAR10 test data. This subset of data was then used as input for these models to assess their generalization capability. To quantify this capability, we calculated the \(\alpha\) and \(\beta\) values for these samples and computed their average. Subsequently, we normalized the \(\alpha\) value by dividing it by the maximum value, denoted as \(\alpha^{\prime}\) in Table 2. Similarly, for \(\beta\), we first calculated its absolute value and then normalized it by dividing it by the maximum value, denoted as \(\beta^{\prime}\) in Table 2. Our analysis revealed that as the generalization capability increased, the sum of \(\alpha^{\prime}\) and \(\beta^{\prime}\) also increased, as indicated in the \(\mathcal{G}\) column of Table 2. This trend is further visualized in Fig. 4.

To validate the effectiveness of our proposed method for analyzing generalization, we conducted a series of generalization tests. Specifically, we applied various data corruptions, including adjust_sharpness, adjust_brightness, gaussian_blur, perspective, adjust_hue, and rotate, to the test data. We then evaluated the performance of the models on these distributional shifts. The accuracy values were averaged, and are presented as the "\(Corruption\)" in Table 2. Additionally, we visualized our metric and the accuracy under these corruptions in Fig. 4. As you can observe, our metric aligns with the accuracy of data corruption, demonstrating the effectiveness of our method.

The results for the SVHN are available in Appendix B.

SoftMax & Logits.A common method that comes to mind for detecting overfitting is to examine the softmax or logit outputs of the model. This involves calculating the softmax or logit values for the ground truth label of each sample and then averaging these values across all samples. However, as shown in Fig. 3 and Table 1, there are cases where this method fails to detect overfitting. For example, consider models 1 and 2, where the softmax or logit outputs do not indicate overfitting despite evidence of overfitting based on other metrics.

Angles of deeper architectures.During our analysis of deeper architectures on CIFAR100, we discovered that increasing the number of layers in a model (creating a deeper architecture) leads to a greater range of angles the model can achieve. For instance, let's consider a ResNet18 model and a DenseNet121 model, both exhibiting identical training and testing accuracy. In this case, the DenseNet121 model will have a higher sum of \(\alpha\) and \(\beta\). We believe that this is because deeper models strive to attain superior generalization ability while mitigating the issue of overfitting.

The results supporting this observation can be found in Appendix B.

Ablation study.We have analyzed the size of the test set in relation to our proposed methods. This analysis reveals that the size of the test data has a minimal impact on the measures of overfitting and generalizability that we have put forward.

Figure 4: (a) illustrates the average accuracy across 5 different data corruptions for 11 distinct ResNet18 models trained on CIFAR10. Meanwhile, (b) showcases the outcomes of our generalization analysis method, which closely aligns with the accuracy observed for the data corruption.
The evidence substantiating this observation is available in Appendix C.

## 5 Conclusion

This paper addresses the issue of determining if a downloaded or received model has been overfitted without knowledge of its training accuracy or data. Overfitted models are more vulnerable to knowledge leakage, posing privacy risks. The proposed method analyzes the null space in the last layer of neural networks, quantifying overfitting and generalization using only a small subset of the testing data. The approach was evaluated on different architectures and datasets, revealing distinct patterns in the null space angle for overfitted models and poor generalization characteristics. This novel method provides insights into model vulnerability without training data, enhancing privacy and trustworthiness in deep learning models.

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{Our Method} & \multicolumn{3}{c}{Conference} & \multicolumn{3}{c}{Accuracy} \\ \cline{2-9} Model & \(\alpha\) & \(\beta\) & \(\mathcal{O}\) & \(SoftMax\) & \(Logits\) & \(Train\) & \(Test\) & \(Difference\) \\ \hline
1 & 59.61 & -27.28 & **32.32** & **0.8906** & 7.88 & 99.99 & 94.78 & **5.21** \\
2 & 78.82 & -11.55 & **67.27** & **0.8078** & 7.47 & 100.0 & 84.04 & **15.96** \\
3 & 60.58 & -26.42 & 34.17 & 0.8649 & 8.09 & 97.73 & 92.42 & 5.31 \\
4 & 70.47 & -19.00 & 51.47 & 0.8373 & 9.53 & 97.69 & 90.06 & 7.63 \\
5 & 64.72 & -23.68 & 41.04 & 0.8377 & 9.16 & 95.75 & 90.02 & 5.73 \\
6 & 70.32 & -19.40 & 50.92 & 0.8361 & 9.71 & 95.66 & 88.67 & 6.99 \\
7 & 61.75 & -25.13 & 36.62 & 0.8240 & 7.29 & 91.68 & 88.87 & 2.81 \\
8 & 70.24 & -18.70 & 51.54 & 0.7834 & 7.61 & 86.31 & 84.24 & 2.07 \\
9 & 63.77 & -22.17 & 41.60 & 0.7782 & 6.28 & 82.77 & 82.52 & 0.25 \\
10 & 70.10 & -18.08 & 52.02 & 0.7786 & 7.50 & 82.52 & 80.31 & 2.21 \\
11 & 64.87 & -21.18 & 43.68 & 0.7978 & 6.55 & 79.77 & 79.52 & 0.25 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Results of training various ResNet18 on CIFAR10. The ’\(\alpha\)’ value indicates the angle between the representation vector and the true class weight vector, while ’\(\beta\)’ shows the angle between the representation vector and the null space. The sum of ’\(\alpha\)’ and ’\(\beta\)’, denoted as ’\(\mathcal{O}\)’, indicates the degree of overfitting.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{3}{c}{Our Method} & \multicolumn{3}{c}{Accuracy} \\ \cline{2-7} Model & \(\alpha^{\prime}\) & \(\beta^{\prime}\) & \(\mathcal{G}\) & \(Train\) & \(Test\) & \(Corruption\) \\ \hline
1 & 0.7563 & 1.0 & 1.7563 & 99.99 & 94.78 & 55.128 \\
2 & 1.0 & 0.4234 & 1.4234 & 100.0 & 84.04 & 35.666 \\
3 & 0.7686 & 0.9685 & 1.7371 & 97.73 & 92.42 & 51.896 \\
4 & 0.8941 & 0.6965 & 1.5906 & 97.69 & 90.06 & 46.446 \\
5 & 0.8211 & 0.8680 & 1.6891 & 95.75 & 90.02 & 45.584 \\
6 & 0.8922 & 0.7111 & **1.6033** & 95.66 & 88.67 & **46.716** \\
7 & 0.7834 & 0.9212 & **1.7046** & 91.68 & 88.87 & **47.51** \\
8 & 0.8911 & 0.6855 & 1.5766 & 86.31 & 84.24 & 40.33 \\
9 & 0.8091 & 0.8127 & 1.6218 & 82.77 & 82.52 & 42.136 \\
10 & 0.8894 & 0.6628 & 1.5522 & 82.52 & 80.31 & 41.76 \\
11 & 0.8230 & 0.7764 & 1.5994 & 79.77 & 79.52 & 45.066 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The variable \(\alpha^{\prime}\) represents the normalized angle between the representation vector and the target weights vector, while \(\beta^{\prime}\) indicates the normalized absolute value of the angle between the representation vector and null space. The sum of \(\alpha^{\prime}\) and \(\beta^{\prime}\), denoted by \(\mathcal{G}\), provides an indication of the model’s generalization capability. The \(Corruption\) represents the average accuracy across 5 different data corruptions for 11 distinct ResNet18 models trained on CIFAR10. As demonstrated, our metric aligns with the corruption accuracies observed.