# EPA: Neural Collapse Inspired Robust Out-of-Distribution Detector

###### Abstract

Out-of-distribution (OOD) detection plays a crucial role in ensuring the
security of neural networks. Existing works have leveraged the fact that In-
distribution (ID) samples form a subspace in the feature space, achieving
state-of-the-art (SOTA) performance. However, the comprehensive
characteristics of the ID subspace still leave under-explored. Recently, the
discovery of Neural Collapse ($\mathcal{NC}$) sheds light on novel properties
of the ID subspace. Leveraging insight from $\mathcal{NC}$, we observe that
the Principal Angle between the features and the ID feature subspace forms a
superior representation for measuring the likelihood of OOD. Building upon
this observation, we propose a novel $\mathcal{NC}$-inspired OOD scoring
function, named Entropy-enhanced Principal Angle (EPA), which integrates both
the global characteristic of the ID subspace and its inner property. We
experimentally compare EPA with various SOTA approaches, validating its
superior performance and robustness across different network architectures and
OOD datasets.

Index Terms—  Out-of-distribution detection, subspace principal angle, neural
collapse.

##  1 Introduction

Neural networks confront security challenges arising from the presence of Out-
of-distribution (OOD) samples. A reliable model should not only precisely
classify In-Distribution (ID) samples but also effectively discern and reject
unknown samples. This highlights the significance of the OOD detection task,
which aims to discriminate whether a given sample pertains to the ID category
or not.

Existing methods aim to design scoring functions to distinguish between ID and
OOD samples based on potential disparities originating from three sources:
softmax probabilities [1, 2], logits [3, 4, 5], and feature space [6, 7, 8, 9,
10, 11, 12]. Among them, the majority of recent approaches primarily leverage
feature space attributes in the design of scoring functions. It has been
observed that ID features form a subspace within the feature space, giving
rise to the development of several competitive methods [6, 7, 8]. However,
these works often utilize the subspace property without delving into a
corresponding explanation and more comprehensive characteristics which may
assist in identifying more interpretable and discriminative attributes.

Recently, the discovery of the Neural Collapse ($\mathcal{NC}$) phenomenon and
its associated theories have revealed four distinct characteristics of ID
features during the terminal phase of training (TPT) [13]. Viewed from a
“bottom-up” perspective, $\mathcal{NC}$ elucidates the prevalent operational
paradigms underlying feature engineering in neural networks and propels the
advancement of neural network interpretability with effective and novel
insights to the subdomains of machine learning [14, 15]. For instance,
researchers have quantitatively observed that $\mathcal{NC}$ consistently
occurs during the transportation of a model to novel samples or categories,
leading to a predictable indicator of the performance of transfer learning and
the development of novel incremental learning methods [16, 17]. The geometric
properties of training features elucidated by $\mathcal{NC}$ motivate us to
delve deeper into the comprehensive attributes of the ID subspace for more
effective OOD detection methods.

Drawing inspiration from $\mathcal{NC}$, we shed light on a vital attribute of
ID features: as training progresses, the features of training samples
gradually depart from the global feature mean and collapse towards the mean of
their respective categories, leading to a considerable distance from the ID
features to the global feature mean. This insight indicates that ID features
have significant components within the corresponding subspace and lie farther
from the global feature mean, which highlights that the Principal Angle (PA)
between the feature and the ID subspace could be employed as a discriminative
indicator for OOD detection. Moreover, while PA characteristics the global
property of the subspace, its inner structure obtained from $\mathcal{NC}$
induces a higher concentration of the output probabilities of ID samples,
which could be measured through the widely employed Entropy method [12, 18,
19]. This motivates us to further enhance the performance of PA by integrating
softmax Entropy to fuse the global and the inner information of the subspace.

The application of $\mathcal{NC}$ in OOD detection tasks is a novel and
promising perspective. To the best of our knowledge, only one relevant study
currently enhances the performance of OOD detection by inducing $\mathcal{NC}$
using L2 regularization on features during the training process [20].
Nevertheless, non-post-hoc methods may have limited applicability to pre-
trained models, and a corresponding explanation of the association between
$\mathcal{NC}$ and OOD detection has yet to be explored. Furthermore, the
emergence of $\mathcal{NC}$ often coincides with the enhancement of
generalization capability and robustness [13]. This propels us to further
explore whether the $\mathcal{NC}$-inspired OOD detection method exhibits
superior robustness compared to existing methods, since real-world
applications request consistent effectiveness across various models and
datasets while reliably addressing tasks of varying difficulties.

In this paper, we provide novel insights to bridge the gap in the relationship
between $\mathcal{NC}$ and OOD detection. Building upon this exploration, we
propose an $\mathcal{NC}$-inspired OOD scoring function named Entropy enhanced
Principal Angle (EPA), which not only considers the overarching subspace
structure of ID features but also encompasses the inner property within the
subspace. We validate the outstanding performance and robustness of EPA by
comprehensive comparison with various state-of-the-art (SOTA) methods on
diverse network architectures and datasets, demonstrating the potential of
reconsidering the OOD detection problem from the perspective of $\mathcal{NC}$
for future research.

##  2 Preliminaries

In this section, we present the notations and fundamental concepts of OOD
detection and Neural Collapse, laying the necessary groundwork for our
approach.

###  2.1 Notations

Denote the training distribution as $\mathbb{X}_{\text{id}}\times\mathbb{Y}$,
where $\mathbb{X}_{\text{id}}$ represents the data distribution of the
training set, and $\mathbb{Y}$ represents the corresponding categories with
the number of categories $|\mathbb{Y}|=C$. Given a neural network classifier
$\mathbf{f}(\mathbf{x}):\mathbb{X}_{\text{id}}\rightarrow\mathbb{R}^{|\mathbb{Y}|}$
trained on $\mathbb{X}_{\text{id}}\times\mathbb{Y}$, assume
$\mathbf{f}(\mathbf{x})=\mathbf{W}^{\mathrm{T}}\mathbf{g}(\mathbf{x})+\mathbf{b}$,
where $\mathbf{z}=\mathbf{g}(\mathbf{x})\in\mathbb{R}^{n}$ is the penultimate
layer feature corresponding to the sample $\mathbf{x}$.
$\mathbf{W}=(\mathbf{w}_{1},\mathbf{w}_{2},\cdots,\mathbf{w}_{C})\in\mathbb{R}^{n\times
C}$ and $\mathbf{b}\in\mathbb{R}^{C}$ are the weights and bias of the last
linear layer in the network.

Let $\mathbf{x}_{i,c}$ be the $i$th training sample in category $c$, with its
corresponding penultimate layer feature as
$\mathbf{z}_{i,c}=\mathbf{g}(\mathbf{x}_{i,c})$. The feature mean for category
$c$ and the global feature mean are denoted as
$\bm{\mu}_{c}=\mathbb{E}_{i}(\mathbf{z}_{i,c})$ and
$\bm{\mu}_{g}=\mathbb{E}_{i,c}(\mathbf{z}_{i,c})$ respectively.

###  2.2 OOD Detection

The goal of OOD detection is to determine whether a new sample $\mathbf{x}$
belongs to or lies outside of $\mathbb{X}_{\text{id}}$. The mainstream OOD
detection methods aims to design a scoring function $h(\mathbf{x})$, which
quantifies the likelihood that a sample $\mathbf{x}$ is classified as an OOD
sample. Thus, OOD detection can be formulated as a binary classification
problem as follows.

$\displaystyle
S(\mathbf{x})=\begin{cases}\text{OOD}&h(\mathbf{x})\geq\gamma\\\
\text{ID}&h(\mathbf{x})<\gamma\end{cases}$ (1)

where $\gamma$ is a threshold. Commonly used evaluation metrics include the
Area Under the Receiver Operating Characteristic curve (AUROC) and the False
Positive Rate when the True Positive Rate reaches 95% (FPR@95). AUROC and
FPR@95 values range from 0 to 1. A higher AUROC value and a lower FPR@95 value
indicate better performance of the scoring function.

###  2.3 Neural Collapse

In [13], four phenomena related to the behavior of the penultimate layer
features of the neural networks are observed during the terminal phase of
training: ($\mathcal{NC}1$) The variance of the features within the same
category tends to be zero. ($\mathcal{NC}2$) The category means of features
converge to a general simplex equiangular tight frame (ETF). ($\mathcal{NC}3$)
The difference between the rescaled weight matrix of the last layer and the
rescaled category feature mean converges to zero. ($\mathcal{NC}4$) The
network simplifies to a nearest class center classifier.

##  3 Methodology

Neural Collapse reveals both the global and inner structures of the subspace
formed by features in a neural network during the late stages of training.
While some prior literature [6, 7, 8] has also considered the use of subspace
properties of features, further investigation is still required to delve into
a corresponding explanation and more comprehensive properties of the subspace.
In this section, we begin by introducing our novel observations based on
Neural Collapse and presenting guidelines for designing the scoring functions.
Then we provide details on the implementation of the proposed EPA method.

###  3.1 Insights from $\mathcal{NC}$

We draw the following conclusions from $\mathcal{NC}1$ to $\mathcal{NC}4$:
When $\mathcal{NC}$ happens, we have

  * (a)

The features of training samples (after mean subtraction) form a subspace
$\mathbb{V}_{\text{id}}$, and the distance between training features and the
origin of $\mathbb{V}_{\text{id}}$ is approximately a constant value
$\alpha>0$, where $\alpha$ is the distance between the vertices and the center
of the general simplex ETF.

  * (b)

The difference between
$\mathbf{o}^{\prime}=-\left(\mathbf{W}^{\mathrm{T}}\right)^{\dagger}\mathbf{b}\in\mathbb{R}^{n}$
and the global feature mean $\bm{\mu}_{g}$ is approximately orthogonal to the
feature subspace $\mathbb{V}_{\text{id}}$. Here,
$\left(\mathbf{W}^{\mathrm{T}}\right)^{\dagger}$ denotes the Moore-Penrose
generalized inverse of $\mathbf{W}^{\mathrm{T}}$.

  * (c)

The outputs of the training samples exhibit high concentration.

Conclusion (a) is derived from $\mathcal{NC}1$ and $\mathcal{NC}2$, as the
vertices of the ETF constitute a subspace in $\mathbb{R}^{n}$. Conclusion (c)
originates from the equiangular property of the ETF and the nearest class
center classifier. Conclusion (b) comes from $\mathcal{NC}3$ and
$\mathcal{NC}4$. Consider

$\displaystyle\mathbf{y}=\mathbf{W}^{\mathrm{T}}\mathbf{z}+\mathbf{b}\approx\mathbf{W}^{\mathrm{T}}\left(\mathbf{z}-\mathbf{o}^{\prime}\right)=\mathbf{W}^{\mathrm{T}}\left(\mathbf{z}-\bm{\mu}_{g}+\bm{\mu}_{g}-\mathbf{o}^{\prime}\right)$
(2)

Since $\mathbf{w}_{c}$ is parallel to $\bm{\mu}_{c}-\bm{\mu}_{g}$ and
$||\mathbf{w}_{c}||_{2}\approx||\mathbf{w}_{c^{\prime}}||_{2}$ (which can be
obtained from $\mathcal{NC}3$), for any two categories $c,c^{\prime}$, to
ensure the nearest neighbor classifier’s characteristics, it is required that
$\bm{\mu}_{g}-\mathbf{o}^{\prime}$ is orthogonal to
$\bm{\mu}_{c}-\bm{\mu}_{c^{\prime}}$. By iterating over all $c,c^{\prime}$,
$\bm{\mu}_{g}-\mathbf{o}^{\prime}$ needs to be orthogonal to the subspace
$\mathbb{V}_{\text{id}}$.

The above observations inspire us to utilize the Principal Angle (abbreviated
as PA) between the features and the subspace to measure the likelihood of a
sample belonging to the ID category. Conclusion (a) indicates that features of
ID samples are likely to be in a subspace and far from the origin, so they
enjoy small PAs, while features of OOD samples may not possess the same
characteristics as ID samples. In practice, biasing the features by
$o^{\prime}$ can introduce favorable improvement in the performance. We
explain this by conclusion (b), as when redefining $o^{\prime}$ as the origin,
the distance of ID features from the origin will be enlarged to
$d\approx\sqrt{\alpha^{2}+||\bm{\mu}_{g}-o^{\prime}||^{2}}$, which leads to
smaller PAs.

PA leverages the global information of the subspace by measuring the affinity
between the feature and the subspace, while the inner structure induced from
$\mathcal{NC}$ remains underutilized. We note that conclusion (c) actually
delves into the equiangular property of the ETF and provides complementary
information for PA. This motivates the further enhancement of PA by
quantifying the concentration through softmax Entropy and fusing information
from both perspectives. Below we provide the specific form of the proposed EPA
method, which consists of the computation of PA and a training set-dependent
integrating procedure with Entropy.

###  3.2 The proposed EPA

Input: Training subset $\mathbb{X}_{s}$, Network weights $\mathbf{g}(\cdot)$,
$\mathbf{W}$, $\mathbf{b}$, Hyperparameter $D$, Test sample $\mathbf{x}$.

Output: EPA score of $\mathbf{x}$.

1\. Calculate
$\mathbf{o}^{\prime}=-(\mathbf{W}^{\mathrm{T}})^{\dagger}\mathbf{b}$.

2\. Denote the orthogonal decomposition as
$\mathbb{E}_{\mathbf{x}_{s}\in\mathbb{X}_{s}}\left(\overline{\mathbf{g}}(\mathbf{x}_{s})\overline{\mathbf{g}}(\mathbf{x}_{s})^{T}\right)=\mathbf{P}\mathbf{\Lambda}\mathbf{P}^{\mathrm{T}}$
with
$\mathbf{\overline{g}}(\mathbf{x}_{s})=\mathbf{{g}}(\mathbf{x}_{s})-o^{\prime}$.
Take the first $D$ columns of $\mathbf{P}$ as $\mathbf{R}$.

3\. Calculate the scaling factor
$\beta=\frac{\max_{\mathbf{x}_{s}\in\mathbb{X}_{s}}H(\mathbf{x}_{s})}{\min_{\mathbf{x}_{s}\in\mathbb{X}_{s}}\theta(\mathbf{x}_{s})}$,
where $\theta(\cdot)$ and $H(\cdot)$ are defined in (4) and (5).

4\. The EPA score of $\mathbf{x}$ is
$\text{EPA}(\mathbf{x})=\beta\theta(\mathbf{x})+H(\mathbf{x})$.

return $\text{EPA}(\mathbf{x})$

Algorithm 1 The implementation of EPA

Computation of PA. Firstly, we sample a subset $\mathbb{X}_{s}$ from the
training set to compute the subspace. Denote
$\overline{\mathbf{g}}(\mathbf{x})=\mathbf{g}(\mathbf{x})-\mathbf{o}^{\prime}$,
and consider the orthogonal decomposition of
$\mathbb{E}_{\mathbf{x}_{s}\in\mathbb{X}_{s}}\left(\overline{\mathbf{g}}(\mathbf{x}_{s})\overline{\mathbf{g}}(\mathbf{x}_{s})^{T}\right)$
as $\mathbf{P}\mathbf{\Lambda}\mathbf{P}^{\mathrm{T}}$, assuming that
$\mathbf{\Lambda}$ is arranged in descending order of eigenvalues. Let
$\mathbf{R}$ be the matrix composed of the first $D$ columns of $\mathbf{P}$,
then the projection matrix onto the subspace formed by the first $D$
eigenvectors of $\mathbf{\Sigma}_{s}$ is
$\mathbf{Q}=\mathbf{R}\mathbf{R}^{\mathrm{T}}$.

Given a test sample $\mathbf{x}$, we can calculate the length of the parallel
component of $\mathbf{\overline{g}}(\mathbf{x})$ within the subspace, which is
given by

$||\mathbf{Q}\overline{\mathbf{g}}(\mathbf{x})||_{2}=||\mathbf{R}^{\mathrm{T}}\overline{\mathbf{g}}(\mathbf{x})||_{2}$
(3)

Hence, the PA between $\overline{\mathbf{g}}(\mathbf{x})$ and the feature
subspace is computed as follows:

$\theta(\mathbf{x})=\arccos\frac{||\mathbf{R}^{\mathrm{T}}\overline{\mathbf{g}}(\mathbf{x})||_{2}}{||\overline{\mathbf{g}}(\mathbf{x})||_{2}}$
(4)

The integration with Entropy. Note that
$\mathbf{y}=\mathbf{W}^{T}\mathbf{g}(\mathbf{x})+\mathbf{b}$, we normalize
$\mathbf{y}$ into probabilities using softmax:
$p_{i}=e^{y_{i}}/\sum_{j=1}^{C}e^{y_{j}}$, where $y_{j}$ represents the $j$-th
component of $\mathbf{y}$. The softmax entropy of sample $\mathbf{x}$ is

$H(\mathbf{x})=-\sum_{i=1}^{C}p_{i}\ln p_{i}$ (5)

Integrating PA and Entropy through simple addition does not effectively
harness their respective advantages. Therefore, we introduce a training set-
dependent scaling factor $\beta$ to strike a balance between PA and Entropy,
calculated by

$\beta=\frac{\max_{\mathbf{x}_{s}\in\mathbb{X}_{s}}H(\mathbf{x}_{s})}{\min_{\mathbf{x}_{s}\in\mathbb{X}_{s}}\theta(\mathbf{x}_{s})}$
(6)

where $\mathbb{X}_{s}$ is the sampled subset of the training set. Now we
present the complete EPA score by weighting and adding PA and Entropy, given
by

$\text{EPA}(\mathbf{x})=\beta\theta(\mathbf{x})+H(\mathbf{x})$ (7)

The specific implementation of EPA score is shown in Algorithm 1. A larger
$\text{EPA}(\mathbf{x})$ implies a higher likelihood of being an OOD sample,
while a smaller value suggests a higher likelihood of being an ID sample.

##  4 Experiments

Fig. 1: Average AUROC% and FPR@95% for various $\beta$. The solid lines
represent the performance of various $\beta$ range in $[0,150]$. The dashed
lines represent the performance using the adaptive $\beta$ in Section 3.2.

###  4.1 Experimental setup

We use the validation set of ImageNet-1k as ID dataset and employ three
different classification models: the CNN-based RepVGG [21], ResNet-50d [22],
and the transformer-based Swin Transformer [23], with their weights obtained
from the timm repo111https://github.com/huggingface/pytorch-image-models. We
test on five different OOD datasets: Openimage-O [6], Texture [24],
iNaturalist [25], ImageNet-O [26], and ImageNet-c [27]. For experiments on
ImageNet-c, we use two subsets, “blur” and “digital” as the OOD dataset,
encompassing a total of eight different corruptions and five corruption
levels. AUROC and FPR@95 are used to evaluate the OOD detection methods. All
results are reported in percentage.

We compare the performance of our proposed EPA score with various post-hoc
methods, including classic OOD scores such as MSP [2], Energy [3], and
MaxLogit [4], as well as recent SOTA methods like ReAct [9], ViM [6], KNN
[10], DML [11], and GEN [12]. We pre-sample 200,000 images from the ImageNet
training set for all methods requiring training samples. For KNN, we set the
number of nearest neighbors, $k$, to 150, which is tuned within the range
[100, 300]. $\lambda$ is set to 10 for DML. Regarding GEN, we report the
performance of the best-performing method in [12], GEN+Residual. For the EPA
score proposed in this work, we set the subspace dimensionality D to 1000 for
RepVGG and ResNet-50d, which is consistent with the total number of ID
categories $C$. For Swin Transformer, we set $D=512$.

Table 1: Ablation study across four datasets. $\theta$ and $H$ refer to PA and
Entropy respectively. ‘w/o $o^{\prime}$’ refers to replacing $o^{\prime}$ by
global mean $\mu_{g}$.

Dataset | Metrics | EPA | $\theta$ | $H$ |  w/o $o^{\prime}$
---|---|---|---|---|---
Openimage-O |  AUROC$\uparrow$ | 93.56 | 89.77 | 88.03 | 91.70
FPR95$\downarrow$ | 31.19 | 44.74 | 50.70 | 38.68
Texture |  AUROC$\uparrow$ | 96.73 | 96.47 | 83.06 | 95.22
FPR95$\downarrow$ | 14.44 | 15.78 | 60.53 | 20.58
iNaturalist |  AUROC$\uparrow$ | 94.99 | 90.69 | 90.98 | 91.81
FPR95$\downarrow$ | 26.63 | 43.15 | 41.15 | 39.84
ImageNet-O |  AUROC$\uparrow$ | 85.57 | 86.56 | 67.83 | 82.68
FPR95$\downarrow$ | 55.23 | 53.67 | 77.8 | 60.55
Average |  AUROC$\uparrow$ | 92.71 | 90.87 | 82.48 | 90.35
FPR95$\downarrow$ | 31.87 | 39.34 | 57.55 | 39.91

###  4.2 Results and Analysis

Table 2: OOD detection performance of EPA and eight other algorithms on
RepVGG, ResNet-50d, and Swin Transformer. OOD datasets include Openimage-O,
Texture, iNaturalist, and ImageNet-O. AUROC% and FPR@95% metrics are reported.
EPA demonstrates excellent performance across different models and OOD
datasets.

|  | Openimage-O | Texture | iNaturalist | ImageNet-O | Average
---|---|---|---|---|---|---
Model | Method | AUROC$\uparrow$FPR95$\downarrow$ | AUROC$\uparrow$FPR95$\downarrow$ | AUROC$\uparrow$FPR95$\downarrow$ | AUROC$\uparrow$FPR95$\downarrow$ | AUROC$\uparrow$FPR95$\downarrow$
| MSP [2] | 85.06 63.36 | 78.58 72.62 | 87.11 54.93 | 61.65 91.30 | 78.10 70.55
| Energy [3] | 83.64 69.92 | 74.53 82.97 | 83.92 75.31 | 63.36 87.75 | 76.36 78.99
| MaxLogit [4] | 84.81 65.04 | 76.33 76.86 | 86.22 62.20 | 62.87 89.90 | 77.56 73.50
| ReAct [9] | 46.08 99.65 | 54.56 97.66 | 47.18 99.88 | 48.76 98.65 | 49.14 98.96
| ViM [6] | 89.27 52.40 | 93.69 23.76 | 91.35 46.79 | 76.93 79.05 | 87.81 50.50
| KNN [10] | 88.33 54.54 | 94.98 14.69 | 89.33 62.57 | 83.38 62.45 | 89.01 48.56
| DML [11] | 84.19 72.53 | 84.07 70.45 | 85.61 73.33 | 71.26 92.50 | 81.28 77.20
| GEN+Res [12] | 88.99 53.84 | 92.74 27.87 | 92.17 42.74 | 76.11 82.10 | 87.50 51.64
RepVGG | EPA (ours) | 91.16 38.84 | 97.19 9.65 | 93.21 34.63 | 82.11 63.50 | 90.92 36.66
| MSP [2] | 84.50 63.53 | 82.75 64.40 | 88.58 50.05 | 56.13 93.85 | 77.99 67.96
| Energy [3] | 75.95 76.83 | 73.93 75.31 | 80.50 71.32 | 53.95 90.10 | 71.08 78.39
| MaxLogit [4] | 81.50 65.50 | 79.25 66.20 | 86.42 53.00 | 54.39 92.65 | 75.39 69.34
| ReAct [9] | 85.30 60.79 | 91.12 39.26 | 87.27 56.03 | 68.02 78.45 | 82.93 58.63
| ViM [6] | 90.76 50.45 | 95.84 20.58 | 89.26 64.59 | 81.02 74.80 | 89.22 52.61
| KNN [10] | 80.40 76.91 | 95.95 20.68 | 79.61 84.50 | 84.05 65.75 | 85.00 61.96
| DML [11] | 86.21 67.58 | 88.11 57.98 | 87.15 66.68 | 71.96 91.80 | 83.36 71.01
|  GEN+Res [12] | 90.17 53.34 | 95.24 23.47 | 90.65 58.19 | 80.23 78.40 | 89.07 53.35
ResNet-50d | EPA (ours) | 92.07 40.47 | 97.57 11.92 | 92.40 42.43 | 81.97 66.55 | 91.00 40.34
| MSP [2] | 91.35 34.96 | 85.21 51.90 | 94.76 23.19 | 78.97 63.70 | 87.57 43.44
| Energy [3] | 90.93 27.58 | 82.62 51.57 | 95.22 15.47 | 82.29 45.70 | 87.77 35.08
| MaxLogit [4] | 91.91 26.79 | 84.67 47.42 | 95.72 15.41 | 81.28 51.50 | 88.40 35.28
| ReAct [9] | 93.58 23.07 | 85.51 47.91 | 97.51  9.98 | 84.09 44.50 | 90.17 31.36
| ViM [6] | 96.04 23.88 | 92.34 38.49 | 99.28 2.60 | 88.78 59.20 | 94.11 31.04
| KNN [10] | 92.86 41.19 | 91.13 38.00 | 95.46 27.51 | 86.79 58.40 | 91.56 41.28
| DML [11] | 95.09 25.39 | 89.21 45.97 | 97.75 11.33 | 86.20 57.25 | 92.06 34.99
| GEN+Res [12] | 95.70 25.42 | 92.18 38.33 | 99.12  3.16 | 88.10 62.05 | 93.78 32.24
Swin | EPA (ours) | 97.44 14.26 | 95.44 21.76 | 99.36 2.84 | 92.64 35.65 | 96.22 18.63

Ablation study. Table 1 showcases the results of ablation experiments. We
compare the performance of the proposed EPA score, the PA $\theta$, and the
Entropy $H$. We also compare the effect of biasing the feature by $o^{\prime}$
and by global mean $\mu_{g}$ (referred to as w/o $o^{\prime}$). All values are
averaged across three different models. The results demonstrate that PA is a
discriminative OOD score, and the enhancement of PA by Entropy in this paper
leads to a significant improvement in performance. Moreover, replacing
$\mu_{g}$ by $o^{\prime}$ to bias the features results in a 2.36% improvement
in AUROC and an 8.04% reduction in FPR@95.

Fig. 2: Average AUROC% of different algorithms on ImageNet-c, averaged across
RepVGG, ResNet-50d, and Swin Transformer. The horizontal axis represents
different corruption levels. EPA demonstrates excellent performance across
different corruption levels.

We further validate the effectiveness of the selection of $\beta$ outlined in
Section 3.2. The solid line in Figure 1 illustrates the average AUROC and
FPR@95 across three models and four datasets (Openimage-O, Texture,
iNaturalist, ImageNet-O) for various $\beta$ values, while the dashed line
represents the average performance using the $\beta$ in Section 3.2. The
results demonstrate that the proposed selection of $\beta$, as an adaptive
criterion that requires no manual tuning, achieves competitive performance.

Comparison with other methods. Table 2 presents the performance of EPA and
other methods on three different models (RepVGG, ResNet-50d, Swin Transformer)
and four OOD datasets (Openimage-O, Texture, iNaturalist, ImageNet-O). The
last column displays the average AUROC and FPR@95 values across different OOD
datasets. EPA exhibits highly competitive performance across various models
and OOD datasets, and consistently achieves the highest average AUROC and the
lowest average FPR@95 across three models. This demonstrates the robustness
and adaptability of EPA to different OOD datasets and various deep neural
network architectures, including CNN-based and Transformer-based models.

We also evaluate EPA’s performance in detecting distribution shifts. Figure 2
displays the average AUROC across RepVGG, ResNet-50d, and Swin Transformer of
nine methods when using ImageNet-c’s “blur” and “digital” subsets as OOD
dataset. The corruptions are categorized into five levels, with level 1 having
minimal corruption (indicating OOD data is closer to ID data and OOD detection
is the toughest) and level 5 having the highest corruption (indicating the
easiest OOD detection). Figure 2 showcases that EPA consistently achieves the
highest AUROC across different corruption levels, confirming the robustness of
our EPA score.

##  5 Conclusion

In this paper, we propose EPA, a novel method for OOD detection based on new
observations drawn from Neural Collapse. We conduct a comprehensive ablation
study and compare the proposed method with various algorithms on multiple
models and OOD datasets, confirming the outstanding performance and robustness
of our EPA method. Future work may delve into the distribution of OOD samples
and explore further improvements to the $\mathcal{NC}$-inspired methods.

## References

  * [1] Shiyu Liang, Yixuan Li, and R Srikant,  “Enhancing the reliability of out-of-distribution image detection in neural networks,”  in International Conference on Learning Representations, 2018.

  * [2] Dan Hendrycks and Kevin Gimpel,  “A baseline for detecting misclassified and out-of-distribution examples in neural networks,”  in International Conference on Learning Representations, 2016.
  * [3] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li,  “Energy-based out-of-distribution detection,”  Advances in Neural Information Processing Systems, vol. 33, pp. 21464–21475, 2020.
  * [4] Steven Basart, Mazeika Mantas, Mostajabi Mohammadreza, Steinhardt Jacob, and Song Dawn,  “Scaling out-of-distribution detection for real-world settings,”  in International Conference on Machine Learning, 2022.
  * [5] Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li,  “Mitigating neural network overconfidence with logit normalization,”  in International Conference on Machine Learning. PMLR, 2022, pp. 23631–23644.
  * [6] Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang,  “Vim: Out-of-distribution with virtual-logit matching,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4921–4930.
  * [7] Amanda Rios, Nilesh Ahuja, Ibrahima Ndiour, Utku Genc, Laurent Itti, and Omesh Tickoo,  “incdfm: Incremental deep feature modeling for continual novelty detection,”  in European Conference on Computer Vision. Springer, 2022, pp. 588–604.
  * [8] Ibrahima Ndiour, Nilesh Ahuja, and Omesh Tickoo,  “Out-of-distribution detection with subspace techniques and probabilistic modeling of features,”  arXiv preprint arXiv:2012.04250, 2020.
  * [9] Yiyou Sun, Chuan Guo, and Yixuan Li,  “React: Out-of-distribution detection with rectified activations,”  Advances in Neural Information Processing Systems, vol. 34, pp. 144–157, 2021.
  * [10] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li,  “Out-of-distribution detection with deep nearest neighbors,”  in International Conference on Machine Learning. PMLR, 2022, pp. 20827–20840.
  * [11] Zihan Zhang and Xiang Xiang,  “Decoupling maxlogit for out-of-distribution detection,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 3388–3397.
  * [12] Xixi Liu, Yaroslava Lochman, and Christopher Zach,  “Gen: Pushing the limits of softmax-based out-of-distribution detection,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 23946–23955.
  * [13] Vardan Papyan, XY Han, and David L Donoho,  “Prevalence of neural collapse during the terminal phase of deep learning training,”  Proceedings of the National Academy of Sciences, vol. 117, no. 40, pp. 24652–24663, 2020.
  * [14] XY Han, Vardan Papyan, and David L Donoho,  “Neural collapse under mse loss: Proximity to and dynamics on the central path,”  in International Conference on Learning Representations, 2021.
  * [15] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu,  “A geometric analysis of neural collapse with unconstrained features,”  in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, Eds., 2021, vol. 34, pp. 29820–29834.
  * [16] Tomer Galanti, András György, and Marcus Hutter,  “On the role of neural collapse in transfer learning,”  in International Conference on Learning Representations, 2022.
  * [17] Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao,  “Neural collapse inspired feature-classifier alignment for few-shot class-incremental learning,”  in The Eleventh International Conference on Learning Representations, 2022.
  * [18] Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L Willke,  “Out-of-distribution detection using an ensemble of self supervised leave-out classifiers,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 550–564.
  * [19] Robin Chan, Matthias Rottmann, and Hanno Gottschalk,  “Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 5128–5137.
  * [20] Jarrod Haas, William Yolland, and Bernhard T Rabus,  “Linking neural collapse and l2 normalization with improved out-of-distribution detection in deep neural networks,”  Transactions on Machine Learning Research, 2022.
  * [21] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun,  “Repvgg: Making vgg-style convnets great again,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 13733–13742.
  * [22] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li,  “Bag of tricks for image classification with convolutional neural networks,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 558–567.
  * [23] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo,  “Swin transformer: Hierarchical vision transformer using shifted windows,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10012–10022.
  * [24] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi,  “Describing textures in the wild,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 3606–3613.
  * [25] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie,  “The inaturalist species classification and detection dataset,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8769–8778.
  * [26] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song,  “Natural adversarial examples,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 15262–15271.
  * [27] Dan Hendrycks and Thomas Dietterich,  “Benchmarking neural network robustness to common corruptions and perturbations,”  in International Conference on Learning Representations, 2018.


* [131] S. M. Kuzenko and G. Tartaglino-Mazzucchelli, “Different representations for the action principle in 4D N = 2 supergravity,” JHEP 04 (2009), 007 [arXiv:0812.3464 [hep-th]].
  * [132] L. Castellani, R. D’Auria and P. Fre, Supergravity and superstrings: A Geometric perspective. Vol. 2: Supergravity, World Scientific, Singapore, 1991, pp. 680–684.
  * [133] S. J. Gates Jr., S. M. Kuzenko and G. Tartaglino-Mazzucchelli, “Chiral supergravity actions and superforms,” Phys. Rev. D 80 (2009), 125015 [arXiv:0909.3918 [hep-th]].
  * [134] D. Butter and J. Novak, “Component reduction in N=2 supergravity: the vector, tensor, and vector-tensor multiplets,” JHEP 05 (2012), 115 [arXiv:1201.5431 [hep-th]].
  * [135] D. Butter and S. M. Kuzenko, “A dual formulation of supergravity-matter theories,” Nucl. Phys. B 854, 1 (2012) [arXiv:1106.3038 [hep-th]].
  * [136] P. S. Howe and R. W. Tucker, “Scale invariance in superspace,” Phys. Lett. B 80, 138 (1978).
  * [137] S. M. Kuzenko, U. Lindström, M. Roček and G. Tartaglino-Mazzucchelli, “On conformal supergravity and projective superspace,” JHEP 0908, 023 (2009) [arXiv:0905.0063 [hep-th]].
  * [138] S. M. Kuzenko, U. Lindström, M. Roček and G. Tartaglino-Mazzucchelli, “4D N=2 supergravity and projective superspace,” JHEP 0809, 051 (2008) [arXiv:0805.4683].
  * [139] M. Müller, Consistent Classical Supergravity Theories, (Lecture Notes in Physics, Vol. 336), Springer, Berlin, 1989.
  * [140] W. D. Linch, III and G. Tartaglino-Mazzucchelli, “Six-dimensional Supergravity and Projective Superfields,” JHEP 08, 075 (2012) [arXiv:1204.4195 [hep-th]].
  * [141] D. Butter, J. Novak and G. Tartaglino-Mazzucchelli, “The component structure of conformal supergravity invariants in six dimensions,” JHEP 05, 133 (2017) [arXiv:1701.08163 [hep-th]].
  * [142] S. M. Kuzenko, “Symmetries of curved superspace,” JHEP 03, 024 (2013) [arXiv:1212.6179 [hep-th]].
  * [143] P. S. Howe and U. Lindström, “Some remarks on (super)-conformal Killing-Yano tensors,” JHEP 11, 049 (2018) [arXiv:1808.00583 [hep-th]].
  * [144] P. S. Howe and U. Lindström, “Notes on super Killing tensors,” JHEP 03, 078 (2016) [arXiv:1511.04575 [hep-th]].
  * [145] S. J. Gates Jr., S. M. Kuzenko and A. G. Sibiryakov, “Towards a unified theory of massless superfields of all superspins,” Phys. Lett. B 394, 343 (1997) [hep-th/9611193].
  * [146] I. L. Buchbinder and S. M. Kuzenko, “On the off-shell massive hypermultiplets,” Class. Quant. Grav. 14, L157 (1997) [arXiv:hep-th/9704002 [hep-th]].
  * [147] F. Gonzalez-Rey, M. Roček, S. Wiles, U. Lindström and R. von Unge, “Feynman rules in N = 2 projective superspace. I: Massless hypermultiplets,” Nucl. Phys. B 516, 426 (1998) [arXiv:hep-th/9710250].
  * [148] M. F. Sohnius, “Supersymmetry and central charges,” Nucl. Phys. B 138, 109 (1978).
  * [149] P. Fayet, “Fermi-Bose hypersymmetry,” Nucl. Phys. B 113, 135 (1976).
  * [150] S. Ferrara, J. Wess and B. Zumino, “Supergauge multiplets and superfields,” Phys. Lett. B 51, 239 (1974).
  * [151] E. I. Buchbinder, J. Hutomo and S. M. Kuzenko, “Higher spin supercurrents in anti-de Sitter space,” JHEP 09, 027 (2018) [arXiv:1805.08055 [hep-th]].
  * [152] M. F. Sohnius, “The multiplet of currents for $N=2$ extended supersymmetry,” Phys. Lett. B 81, 8-10 (1979)
  * [153] T. Curtright, “Generalized gauge fields,” Phys. Lett. B 165, 304 (1985).
  * [154] S. J. Gates Jr. and W. Siegel, “(3/2, 1) superfield of O(2) supergravity,” Nucl. Phys. B164, 484 (1980).
  * [155] W. Siegel and S. J. Gates Jr. “Superfield supergravity,” Nucl. Phys. B 147, 77 (1979).
  * [156] S. M. Kuzenko and M. Ponds, “Generalised conformal higher-spin fields in curved backgrounds,” JHEP 04, 021 (2020) [arXiv:1912.00652 [hep-th]].
  * [157] I. Bandos, K. Lechner, D. Sorokin and P. K. Townsend, “A non-linear duality-invariant conformal extension of Maxwell’s equations,” Phys. Rev. D 102, 121703 (2020) [arXiv:2007.09092 [hep-th]].
  * [158] B. P. Kosyakov, “Nonlinear electrodynamics with the maximum allowable symmetries,” Phys. Lett. B 810, 135840 (2020) [arXiv:2007.13878 [hep-th]].
  * [159] I. Bandos, K. Lechner, D. Sorokin and P. K. Townsend, “ModMax meets Susy,” JHEP 10, 031 (2021) [arXiv:2106.07547 [hep-th]].
  * [160] S. M. Kuzenko, “Superconformal duality-invariant models and $\mathcal{N}=4$ SYM effective action,” [arXiv:2106.07173 [hep-th]].
  * [161] M. K. Gaillard and B. Zumino, “Self-duality in nonlinear electromagnetism,” in Supersymmetry and Quantum Field Theory, J. Wess and V. P. Akulov (Eds.), Springer Verlag, 1998, pp. 121–129 [arXiv:hep-th/9705226].
  * [162] M. K. Gaillard and B. Zumino, “Nonlinear electromagnetic self-duality and Legendre transformations,” in Duality and Supersymmetric Theories, D. I. Olive and P. C. West (Eds.), Cambridge University Press, 1999, pp. 33–48 [hep-th/9712103].
  * [163] S. M. Kuzenko and S. Theisen, “Nonlinear self-duality and supersymmetry,” Fortsch. Phys. 49, 273 (2001) [arXiv:hep-th/0007231].
  * [164] I. Bialynicki-Birula, “Nonlinear electrodynamics: Variations on a theme by Born and Infeld,” in Quantum Theory of Particles and Fields, B. Jancewicz and J. Lukierski (Eds.), World Scientific, 1983, pp. 31–48.
  * [165] G. W. Gibbons and D. A. Rasheed, “Electric-magnetic duality rotations in nonlinear electrodynamics,” Nucl. Phys. B454, 185 (1995) [arXiv:hep-th/9506035].
  * [166] M. K. Gaillard and B. Zumino, “Duality rotations for interacting fields,” Nucl. Phys. B193, 221 (1981).
  * [167] M. Born and L. Infeld, “Foundations of the new field theory,” Proc. Roy. Soc. Lond. A 144, no.852, 425-451 (1934)
  * [168] I. Bandos, K. Lechner, D. Sorokin and P. K. Townsend, “On p-form gauge theories and their conformal limits,” JHEP 03, 022 (2021) [arXiv:2012.09286 [hep-th]].
  * [169] E. A. Ivanov and B. M. Zupnik, “N=3 supersymmetric Born-Infeld theory,” Nucl. Phys. B 618, 3 (2001) [hep-th/0110074].
  * [170] E. A. Ivanov and B. M. Zupnik, “New representation for Lagrangians of self-dual nonlinear electrodynamics,” in Supersymmetries and Quantum Symmetries. Proceedings of the 16th Max Born Symposium, SQS’01: Karpacz, Poland, September 21–25, 2001, E. Ivanov (Ed.), Dubna, 2002, pp. 235–250 [hep-th/0202203].
  * [171] E. A. Ivanov and B. M. Zupnik, “New approach to nonlinear electrodynamics: Dualities as symmetries of interaction,” Phys. Atom. Nucl. 67, 2188 (2004) [Yad. Fiz. 67, 2212 (2004)] [hep-th/0303192].
  * [172] R. Penrose and W. Rindler, Spinors and Space-Time: Volume 2, Spinor and Twistor Methods in Space-Time Geometry, Cambridge University Press, Cambridge, 1986.
  * [173] S. M. Kuzenko and S. Theisen, “Supersymmetric duality rotations,” JHEP 0003, 034 (2000) [arXiv:hep-th/0001068].
  * [174] S. M. Kuzenko and S. A. McCarthy, “Nonlinear self-duality and supergravity,” JHEP 0302, 038 (2003) [hep-th/0212039].
  * [175] S. M. Kuzenko, “Nonlinear self-duality in N=2 supergravity,” JHEP 1206, 012 (2012) [arXiv:1202.0126 [hep-th]].
  * [176] S. M. Kuzenko, “Duality rotations in supersymmetric nonlinear electrodynamics revisited,” JHEP 1303, 153 (2013) [arXiv:1301.5194 [hep-th]].
  * [177] E. Ivanov, O. Lechtenfeld and B. Zupnik, “Auxiliary superfields in N=1 supersymmetric self-dual electrodynamics,” JHEP 05, 133 (2013) [arXiv:1303.5962 [hep-th]].
  * [178] X. Bekaert, E. Joung and J. Mourad, “Effective action in a higher-spin background,” JHEP 02, 048 (2011) [arXiv:1012.2103 [hep-th]].
  * [179] N. S. Craigie, V. K. Dobrev and I. T. Todorov, “Conformally covariant composite operators in quantum chromodynamics,” Annals Phys. 159, 411 (1985).
  * [180] Y. M. Makeenko, “Conformal operators in quantum chromodynamics,” Sov. J. Nucl. Phys. 33, 440 (1981) [Yad. Fiz. 33, 842 (1981)].
  * [181] M. Beccaria and A. A. Tseytlin, “On induced action for conformal higher spins in curved background,” Nucl. Phys. B 919, 359 (2017) [arXiv:1702.00222 [hep-th]].
  * [182] M. Grigoriev and A. A. Tseytlin, “On conformal higher spins in curved background,” J. Phys. A 50, no. 12, 125401 (2017) [arXiv:1609.09381 [hep-th]].
  * [183] M. Henneaux and C. Teitelboim, Quantization of gauge systems, Princeton University Press, Princeton, 1992.
  * [184] S. Ferrara and B. Zumino, “Transformation properties of the supercurrent,” Nucl. Phys. B 87, 207 (1975).
  * [185] M. Magro, I. Sachs and S. Wolf, “Superfield Noether procedure,” Annals Phys. 298, 123-166 (2002) [arXiv:hep-th/0110131 [hep-th]].
  * [186] R. Grimm, M. Sohnius and J. Wess, “Extended supersymmetry and gauge theories,” Nucl. Phys. B 133, 275 (1978).
  * [187] P. Breitenlohner and M. F. Sohnius, “Superfields, auxiliary fields, and tensor calculus for N=2 extended supergravity,” Nucl. Phys. B 165, 483 (1980).
  * [188] M. F. Sohnius, K. S. Stelle and P. C. West, “Representations of extended supersymmetry,” in Superspace and Supergravity, S. W. Hawking and M. Roček (Eds.), Cambridge University Press, Cambridge, 1981, p. 283.
  * [189] L. Mezincescu, “On the superfield formulation of O(2) supersymmetry,” Dubna preprint JINR-P2-12572 (June, 1979).
  * [190] C. M. Hull and E. Witten, “Supersymmetric Sigma Models and the Heterotic String,” Phys. Lett. B 160, 398 (1985).
  * [191] M. Ademollo et al., “Supersymmetric strings and color confinement,” Phys. Lett. B 62, 105 (1976); “Dual String with U(1) Color Symmetry,” Nucl. Phys. B 111, 77 (1976); “Dual String Models with Nonabelian Color and Flavor Symmetries,” Nucl. Phys. B 114, 297 (1976).
  * [192] P. Ramond and J. H. Schwarz, “Classification of dual model gauge algebras,” Phys. Lett. B 64, 75 (1976).
  * [193] R. Gastmans, A. Sevrin, W. Troost and A. Van Proeyen, “Infinite dimensional extended superconformal Lie algebras,” Int. J. Mod. Phys. A 2, 195 (1987).
  * [194] K. Schoutens, “Structure of $d=2$ Conformal Supergravity and Covariant Actions for Strings,” Nucl. Phys. B 292, 150 (1987).
  * [195] P. S. Howe and G. Papadopoulos, “N=2, D=2 supergeometry,” Class. Quant. Grav. 4, 11 (1987).
  * [196] M. T. Grisaru and M. E. Wehlau, “Prepotentials for (2,2) supergravity,” Int. J. Mod. Phys. A 10, 753 (1995) [arXiv:hep-th/9409043 [hep-th]].
  * [197] M. T. Grisaru and M. E. Wehlau, “Superspace measures, invariant actions, and component projection formulae for (2,2) supergravity,” Nucl. Phys. B 457, 219 (1995) [arXiv:hep-th/9508139 [hep-th]].
  * [198] S. J. Gates Jr., M. T. Grisaru and M. E. Wehlau, “A Study of general 2-D, N=2 matter coupled to supergravity in superspace,” Nucl. Phys. B 460, 579 (1996) [arXiv:hep-th/9509021 [hep-th]].
  * [199] G. Tartaglino-Mazzucchelli, “2D N = (4,4) superspace supergravity and bi-projective superfields,” JHEP 04, 034 (2010) [arXiv:0911.2546 [hep-th]].
  * [200] S. Bellucci and E. Ivanov, “N=(4,4), 2-D supergravity in SU(2) x SU(2) harmonic superspace,” Nucl. Phys. B 587, 445-480 (2000) [arXiv:hep-th/0003154 [hep-th]].
  * [201] M. Evans and B. A. Ovrut, “The World Sheet Supergravity of the Heterotic String,” Phys. Lett. B 171, 177 (1986).
  * [202] R. Brooks, F. Muhammad and S. J. Gates Jr., “Unidexterous D=2 Supersymmetry in Superspace,” Nucl. Phys. B 268, 599 (1986)
  * [203] S. Govindarajan and B. A. Ovrut, “The Anomaly structure of (2,0) heterotic world sheet supergravity with gauged R invariance,” Nucl. Phys. B 385, 251 (1992).
  * [204] M. Brown and S. J. Gates Jr., “Superspace Bianchi Identities and the Supercovariant Derivative,” Annals Phys. 122, 443 (1979).
  * [205] E. J. Martinec, “Superspace Geometry of Fermionic Strings,” Phys. Rev. D 28, 2604 (1983).
  * [206] S. J. Gates Jr. and H. Nishino, “$D=2$ superfield supergravity, local (supersymmetry)**2 and nonlinear $\Sigma$ models,” Class. Quant. Grav. 3, 391 (1986).
  * [207] M. Roček, P. van Nieuwenhuizen and S. C. Zhang, “Superspace Path Integral Measure of the $N=1$ Spinning String,” Annals Phys. 172, 348 (1986).
  * [208] S. J. Gates Jr., M. T. Grisaru, L. Mezincescu and P. K. Townsend, “(1,0) Supergraphity,” Nucl. Phys. B 286, 1 (1987).
  * [209] M. Beccaria, H. Jiang and A. A. Tseytlin, “Supersymmetric Liouville theory in AdS2 and AdS/CFT,” JHEP 11, 051 (2019) [arXiv:1909.10255 [hep-th]].
  * [210] S. M. Kuzenko, U. Lindström and G. Tartaglino-Mazzucchelli, “Three-dimensional (p,q) AdS superspaces and matter couplings,” JHEP 1208, 024 (2012) [arXiv:1205.4622 [hep-th]].
  * [211] E. S. Fradkin and A. A. Tseytlin, “Quantum string theory effective action,” Nucl. Phys. B 261, 1 (1985) [erratum: Nucl. Phys. B 269, 745 (1986)].
  * [212] S. J. Gates, Jr., Y. Hassoun and P. van Nieuwenhuizen, “Auxiliary Fields for $d=2$, $N=4$ Supergravity,” Nucl. Phys. B 317, 302 (1989).
  * [213] S. J. Gates, Jr., L. Lu and R. N. Oerter, “Simplified SU(2) spinnig string superspace supergravity,” Phys. Lett. B 218, 33 (1989).
  * [214] S. M. Kuzenko and G. Tartaglino-Mazzucchelli, “Supertwistor realisations of AdS superspaces,” Eur. Phys. J. C 82, no.2, 146 (2022) [arXiv:2108.03907 [hep-th]].
  * [215] G. Barnich and F. Brandt, “Covariant theory of asymptotic symmetries, conservation laws and central charges,” Nucl. Phys. B 633, 3-82 (2002) [arXiv:hep-th/0111246 [hep-th]].
  * [216] F. A. Berends, G. J. H. Burgers and H. van Dam, “On the theoretical problems in constructing interactions involving higher spin massless particles,” Nucl. Phys. B 260, 295-322 (1985).
  * [217] X. Bekaert, E. Joung and J. Mourad, “On higher spin interactions with matter,” JHEP 05, 126 (2009) [arXiv:0903.3338 [hep-th]].
  * [218] S. M. Kuzenko and E. S. N. Raptakis, work in progress.
  * [219] A. Karlhede, U. Lindström and M. Roček, “Self-interacting tensor multiplets in N=2 superspace,” Phys. Lett. B 147, 297 (1984).
  * [220] U. Lindström and M. Roček, “N=2 super Yang-Mills theory in projective superspace,” Commun. Math. Phys. 128, 191 (1990).
  * [221] S. J. Gates Jr., C. M. Hull and M. Roček, “Twisted multiplets and new supersymmetric nonlinear sigma models,” Nucl. Phys. B 248, 157 (1984).
  * [222] U. Lindström, M. Roček, R. von Unge and M. Zabzine, “Generalized Kahler manifolds and off-shell supersymmetry,” Commun. Math. Phys. 269, 833 (2007) [arXiv:hep-th/0512164 [hep-th]].
  * [223] C. Hull and U. Lindström, “The generalised complex geometry of $(p,q)$ Hermitian geometries,” Commun. Math. Phys. 375, no.1, 479 (2019) [arXiv:1810.06489 [hep-th]].
  * [224] U. Lindström, “$2d$ Sigma Models and Geometry,” [arXiv:2203.02974 [hep-th]].
  * [225] C. M. Hull, “Lectures On nonlinear sigma models and strings,” Lectures given at the Vancouver Advanced Research Workshop, published in Super Field Theories (Plenum, New York, 1988), edited by H. Lee and G. Kunstatter.