Knowing When to Stop: Delay-Adaptive Spiking Neural Network Classifiers
  with Reliability Guarantees
Spiking neural networks (SNNs) process time-series data via internal
event-driven neural dynamics. The energy consumption of an SNN depends on the
number of spikes exchanged between neurons over the course of the input
presentation. Typically, decisions are produced after the entire input sequence
has been processed. This results in latency and energy consumption levels that
are fairly uniform across inputs. However, as explored in recent work, SNNs can
produce an early decision when the SNN model is sufficiently ``confident'',
adapting delay and energy consumption to the difficulty of each example.
Existing techniques are based on heuristic measures of confidence that do not
provide reliability guarantees, potentially exiting too early. In this paper,
we introduce a novel delay-adaptive SNN-based inference methodology that,
wrapping around any pre-trained SNN classifier, provides guaranteed reliability
for the decisions produced at input-dependent stopping times. The approach,
dubbed SpikeCP, leverages tools from conformal prediction (CP). It entails
minimal complexity increase as compared to the underlying SNN, requiring only
additional thresholding and counting operations at run time. SpikeCP is also
extended to integrate a CP-aware training phase that targets delay performance.
Variants of CP based on alternative confidence correction schemes, from
Bonferroni to Simes, are explored, and extensive experiments are described
using the MNIST-DVS data set, DVS128 Gesture dataset, and CIFAR-10 dataset.
Jiechen Chen, Sangwoo Park, Osvaldo Simeone
# Knowing When to Stop:

###### Abstract

Spiking neural networks (SNNs) process time-series data via internal event-driven neural dynamics whose energy consumption depends on the number of spikes exchanged between neurons over the course of the input presentation. Typically, decisions are produced after the entire input sequence has been processed, resulting in latency and energy consumption levels that are fairly uniform across inputs. However, as explored in recent work, SNNs can produce an early decision when the SNN model is sufficiently "confident", adapting delay and energy consumption to the difficulty of each example. Existing techniques are based on heuristic measures of confidence that do not provide reliability guarantees, potentially exiting too early. In this paper, we introduce a novel delay-adaptive SNN-based inference methodology that, wrapping around any pre-trained SNN classifier, provides guaranteed reliability for the decisions produced at input-dependent stopping times. The approach, dubbed _SpikeCP_, leverages tools from conformal prediction (CP), and it entails minimal complexity increase as compared to the underlying SNN, requiring only additional thresholding and counting operations at run time. SpikeCP is also extended to integrate a CP-aware training phase that targets delay performance. Variants of CP based on alternative confidence correction schemes, from Bonferroni to Simes, are explored, and extensive experiments are described using the MNIST-DVS data set.

 Spiking neural networks, conformal prediction, delay adaptivity, reliability, neuromorphic computing.

## I Introduction

### _Motivation_

Spiking neural networks (SNNs) have emerged as efficient models for the processing of time series data, particularly in settings characterized by sparse inputs [1]. SNNs implement recurrent, event-driven, neural dynamics whose energy consumption depends on the number of spikes exchanged between neurons over the course of the input presentation. As shown in Fig. 1(a), an SNN-based classifier processes input time series to produce spiking signals - one for each possible class - with the spiking rate of each output signal typically quantifying the _confidence_ the model has in the corresponding labels. Typically, decisions are produced after the entire input sequence has been processed, resulting in latency and energy consumption levels that are fairly uniform across inputs.

The online operation of SNNs, along with their in-built adaptive measures of confidence derived from the output spikes, suggest an alternative operating principle, whereby inference latency and energy consumption are tailored to the difficulty of each example. Specifically, as proposed in [2, 3], _delay-adaptive_ SNN classifiers produce an _early decision_ when the SNN model is sufficiently confident. In practice, however, the confidence levels output by an SNN, even when adjusted with limited data as in [4], are not well _calibrated_, in the sense that they do not precisely reflect the underlying accuracy of the corresponding decisions (see Fig. 1). As a result, relying on its output confidence signals may cause the SNN to stop prematurely, failing to meet target accuracy levels.

To illustrate this problem, Fig. 1(b) shows the test accuracy and confidence level (averaged over test inputs) that are produced by a pre-trained SNN for an image classification task (on the MNIST-DVS dataset [5]) as a function of time \(t\). It is observed that the SNN's classification decisions tend to be first _under-confident_ and then _over-confident_ with respect to the decision's ground-truth, unknown, test accuracy. Therefore, using the SNN's confidence levels to decide when to make a decision generally causes a _reliability gap_ between the true test accuracy and the target accuracy. This problem can be mitigated by relying on _calibration data_ to re-calibrate the SNN's confidence level, but only if one has enough calibration data [4] (see Sec. VI for experimental evidence, e.g., in Fig. 4).

### _SpikeCP_

In this paper, we introduce a novel delay-adaptive SNN solution that (_i_) provides _guaranteed_ reliability - and hence a zero (or non-positive) reliability gap; while (_ii_) supporting a tunable trade-off between latency and inference energy, on the one hand, and informativeness of the decision, on the other hand. The proposed method, referred to as _SpikeCP_, builds on _conformal prediction_ (CP), a statistical framework for calibration that is currently experiencing a surge of interest in the machine learning community [6, 7].

SpikeCP uses local or global information produced by the output layer of SNN model (see Fig. 1(a)), along with _calibration data_, to produce at each time \(t\) a _subset of labels_ as its decision. By the properties of CP, the predictive set produced by SpikeCP includes the ground-truth label with a target accuracy level at any stopping time. A stopping decision is then made by SpikeCP not based on a reliability requirement - which is always satisfied - but rather based on the desired
size of the predicted set. As illustrated in Fig. 1, the desired set size provides a novel degree of freedom that can be used to control the trade-off between latency, or energy consumption, and _informativeness_ of the decision, as measured by the set size.

SpikeCP wraps around any pre-trained SNN classifier, providing guaranteed reliability for the decisions produced at input-dependent stopping times. It does so with a minimal complexity increase as compared to the underlying SNN, requiring only additional thresholding and counting operations at run time. At a technical level, SpikeCP applies a Bonferroni correction of the target accuracy that scales with the number of possible stopping times in order to ensure a zero (or non-positive) reliability gap. Heuristics based on Simes correction [8, 9] are also explored via numerical results.

The approach is finally extended to integrate a CP-aware training phase that targets minimization of the delay via a reduction of the average predicted set size. Unlike conventional training methods for SNNs [10], the proposed method adds an explicit regularizer that controls the average number of labels included in the predicted set.

### _Related Work_

_Training SNNs._ Typical training algorithms for SNNs are based on direct conversions from trained artificial neural networks [2], on heuristic local rules such as spike-timing-dependent plasticity (STDP) [11], or approximations of backpropagation-through-time that simplify credit assignment and address the non-differentiability of the spiking mechanism [1, 12]. Another approach that targets the direct training of SNNs is based on modelling the spiking mechanism as a stochastic process, which enables the use of likelihood-based methods [13], as well as of Bayesian rules [14]. SpikeCP works as a wrapper around any training scheme.

_Calibration and delay-adaptivity for SNN._ Calibration is a subject of extensive research for artificial neural networks [15, 16] but is still an underexplored subject for SNNs. SNN calibration is carried out by leveraging a pre-trained ANN in [4]; while [14] applies Bayesian learning to reduce the calibration error. As discussed in the previous sections, adaptivity for rate decoding was studied in [2, 3]. Other forms of adaptivity may leverage _temporal decoding_, whereby, for instance, as soon as one output neuron spikes a decision is made [17].

_Early exit in conventional deep learning._ The idea of delay-adaptivity in SNNs is related to that of _early-exit_ decisions in feedforward neural networks. In neural networks with an early exit option, confidence levels are evaluated at intermediate layers, and a decision is made when the confidence level passes a threshold [18, 19, 20]. The role of calibration for early-exit neural networks was studied in [21].

_Prediction cascades._ Another related concept is that of prediction cascades, which apply a sequence of classifiers, ranging from light-weight to computationally expensive [22], to a static input. The goal is to apply the more expensive classifiers only when the difficulty of the input requires it.

Fig. 1: (a) SNN \(C\)-class classification model: At time \(t\), real-valued discrete-time time-series data \(\mathbf{x}^{t}\) are fed to the input neurons of an SNN and processed by internal spiking neurons, whose spikes feed \(C\) readout neurons. Each output neuron \(c\in\{1,...,C\}\) evaluates the _local spike count_ variable \(r_{c}(\mathbf{x}^{t})\) by accumulating the number of spikes it produces. The spike rates may be aggregated across all output neurons to produce the _predictive probability vector_\(\{p_{c}(\mathbf{x}^{t})\}_{C=1}^{C}\). (b) Evolution of confidence and accuracy as a function of time \(t\) for a conventional pre-trained SNN. As illustrated, SNN classifiers tend to be first under-confident and then over-confident with respect to the true accuracy, which may cause a positive _reliability gap_, i.e., a shortfall in accuracy, when the confidence level is used as an inference-stopping criterion. (c) Evolution of the (test-averaged) predicted set size (normalized by the number of classes \(C=10\)) and of the set accuracy as a function of time \(t\) for the same pre-trained SNN when used in conjunction with the proposed SpikeCP method. The set accuracy is the probability that the true label lies inside the predicted set. It is observed that, irrespective of the stopping time, the set accuracy is always guaranteed to exceed the target accuracy level. Therefore, the inference-stopping criterion can be designed to control the trade-off between latency, and hence also energy consumption, and the size of the predicted set.
The application of CP to prediction cascades was investigated in [8].

_CP-aware training._ CP provides a general methodology to turn a pre-trained probabilistic predictors into a reliable set predictor [7, 23]. Applications of CP range from healthcare [24] to control [25], large language models [26], and wireless systems [23]. References [27, 28, 29] have observed that the efficiency of the set-valued predictions produced via CP can be improved by training the underlying predictor in a _CP-aware_ manner that targets directly the predicted set size. Specifically, the authors of [27] propose to minimize a loss functions that penalizes large prediction set sizes when used in conjunction with CP. It explored strategies to differentiate through CP during training with the goal of training model, with the conformal wrapper end-to-end. Related work in [29] has leveraged differentiation through CP to design meta-learning strategies targeting the predictive set size (see also [30]). To the best of our knowledge, no prior work has applied the idea of CP-aware training to the design of delay-adaptive classifiers.

### _Main Contributions and Paper Organization_

The main contributions of this paper are summarized as follows.

\(\bullet\) We introduce _SpikeCP_, a novel inference framework that turns any pre-trained SNN into a reliable and delay-adaptive set predictor, irrespective of the quality of the pre-trained SNN and of the number of calibration points. The performance of the pre-trained SNN determines the achievable trade-off curve between latency and energy efficiency, on the one hand, and informativeness of the decision, as measured by the set size, on the other. SpikeCP requires minimal changes to the underlying SNN, adding only counting and thresholding operations. Furthermore, it can be implemented using different measures of confidence at the output of the SNN, such as spiking rates and softmax-modulated signals.

\(\bullet\)_Theoretical guarantees_ are proved by leveraging a modification of the confidence levels based on Bonferroni correction [31]. Heuristic alternatives based on Simes correction are also considered [32].

\(\bullet\) In order to improve the performance in terms of attainable trade-offs between delay/energy consumption and predictive set sizes, we introduce a _SpikeCP-aware training_ strategy that targets directly the performance of the SNN when used in conjunction with SpikeCP. The approach is based on regularizing the classical cross-entropy loss [33, 34] with a differentiable approximation of the predicted set size.

\(\bullet\) Extensive numerical results are provided that demonstrate the advantages of the proposed SpikeCP algorithms over conventional point predictors in terms of reliability, latency, and energy consumption metrics.

The remainder of the paper is organized as follows. Section II presents the multi-class classification problem via SNNs. Adaptive point classification schemes are reviewed for reference in Section III. The SpikeCP algorithm is proposed in Section IV, while Section V presents a training strategy that targets directly the performance of the SNN when used in conjunction with SpikeCP. Experimental setting and results are described in Section VI. Finally, Section VII concludes the paper.

## II Problem Definition

In this paper, we consider the problem of efficiently and reliably classifying time series data via SNNs by integrating adaptive-latency decision rules [2, 3] with CP [6, 7]. The proposed scheme, SpikeCP, produces _adaptive SNN-based set classifiers_ with _formal reliability guarantees_. In this section, we start by defining the problem under study, along with the main performance metrics of interest, namely reliability, latency, and inference energy. We also review the conventional model of SNNs adopted in this study that is based on leaky integrate-and-fire (LIF) neurons [35].

### _Multi-Class Time Series Classification_

We focus on the problem of classifying real-valued vector time series data \(\mathbf{x}=(\mathbf{x}_{1},...,\mathbf{x}_{T})\), with \(N\times 1\) vector samples \(\mathbf{x}_{t}\) over time index \(t=1,...,T\), into \(C\) classes, using _dynamic_ classifiers implemented via SNNs. As illustrated in Fig. 1(a), the SNN model has \(N\) input neurons, an arbitrary number of internal spiking neurons, and \(C\) output neurons in the readout layer. Each output neuron is associated with one of the \(C\) class labels in set \(\mathcal{C}=\{1,...,C\}\).

At each time \(t\), the SNN takes as input the real-valued vector \(\mathbf{x}_{t}\), and produces sequentially the binary, "spiking", output vector \(\mathbf{y}_{t}=[y_{t,1},...,y_{t,C}]\) of size \(C\), with \(y_{t,c}\in\{0,1\}\), as a function of the samples

\[\mathbf{x}^{t}=(\mathbf{x}_{1},...,\mathbf{x}_{t}), \tag{1}\]

observed so far. Accordingly, if \(y_{t,c}=1\), output neuron \(c\in\mathcal{C}\) emits a spike, while, if \(y_{t,c}=0\), output neuron \(c\) is silent. Using conventional _rate decoding_, each output neuron \(c\in\mathcal{C}\) maintains the sum of spikes evaluated so far, i.e.,

\[r_{c}(\mathbf{x}^{t})=\sum_{t^{\prime}=1}^{t}y_{t^{\prime},c}, \tag{2}\]

along the time axis \(t=1,...,T\).

Each _spike count_ variable \(r_{c}(\mathbf{x}^{t})\) may be used as an estimate of the degree of confidence of the SNN in class \(c\) being the correct one. In order to obtain predictive probabilities, the spike count vector \(\mathbf{r}(\mathbf{x}^{t})=[r_{1}(\mathbf{x}^{t}),...,r_{C}(\mathbf{x}^{t})]\) can be passed through a softmax function to yield a probability for class \(c\) as \(p_{c}(\mathbf{x}^{t})=e^{r_{c}(\mathbf{x}^{t})}/\sum_{c^{\prime}=1}^{C}e^{r_{c^{\prime }}(\mathbf{x}^{t})}\) (see Fig. 1(a)). The resulting _predictive probability vector_

\[\mathbf{p}(\mathbf{x}^{t})=[p_{1}(\mathbf{x}^{t}),...,p_{C}(\mathbf{x}^{t})], \tag{3}\]

quantifies the _normalized_ confidence levels of the classifier in each class \(c\) given the observations up to time \(t\). We emphasize that evaluating the vector (3) requires coordination among all output neurons, since each probability value \(p_{c}(\mathbf{x}^{t})\) depends on the spike counts of all output spiking neurons.

A classifier is said to be _well calibrated_ if the confidence vector \(\mathbf{p}(\mathbf{x}^{t})\) provides a close approximation of the true, test, accuracy of each decision \(c\in\mathcal{C}\). Machine learning models based on deep learning are well known to be typically
over-confident_, resulting in confidence vectors \(\mathbf{p}(\mathbf{x}^{t})\) that are excessively skewed towards a single class \(c\), dependent on the input \(\mathbf{x}^{t}\)[15, 36]. As discussed in Sec. I, SNN models also tend to provide over-confident decisions as time \(t\) increases.

Following the conventional supervised learning formulation of the problem, multi-class time series classification data consist of pairs \((\mathbf{x},c)\) of input sequence \(\mathbf{x}\) and true class index \(c\in\mathcal{C}\). All data points are generated from a _ground-truth distribution_\(p(\mathbf{x},c)\) in an independent and identically distributed (i.i.d.) manner. We focus on _pre-trained_ SNN classification models, on which we make no assumptions in terms of accuracy or calibration. Furthermore, we assume the availability of a, typically small, _calibration data set_

\[\mathcal{D}^{\mathrm{cal}}=\{\mathbf{z}[i]=(\mathbf{x}[i],c[i])\}_{i=1}^{|\mathcal{D} ^{\mathrm{cal}}|}. \tag{4}\]

In practice, a new calibration data set may be produced periodically at test time to be reused across multiple test points \((\mathbf{x},c)\)[2, 6, 7].

### _Taxonomy of SNN Classifiers_

As detailed in Table I and Fig. 2, we distinguish SNN classifiers along two axes, namely adaptivity and decision type.

_Adaptivity_: As shown in Fig. 2(a) and Fig. 2(c), a _non-adaptive_ classifier, having observed all the \(T\) samples of the input sequence \(\mathbf{x}\), makes a decision on the basis of the spike count vector \(\mathbf{r}(\mathbf{x}^{T})=\mathbf{r}(\mathbf{x})\) or of the predictive probability vector \(\mathbf{p}(\mathbf{x}^{T})=\mathbf{p}(\mathbf{x})\). In contrast, as seen in Fig. 2(b) and Fig. 2(d), an _adaptive_ classifier allows for the time \(T_{s}(\mathbf{x})\) at which a classification decision is produced, to be adapted to the difficulty of the input \(\mathbf{x}\). For any given input \(\mathbf{x}\), the _stopping time_\(T_{s}(\mathbf{x})\) and the final decision produced at time \(T_{s}(\mathbf{x})\) depend on either the spike count vector \(\mathbf{r}(\mathbf{x}^{t})\) or on the predictive distribution vector \(\mathbf{p}(\mathbf{x}^{t})\) produced by the SNN classifier after having observed the first \(t=T_{s}(\mathbf{x})\) input samples \(\mathbf{x}^{t}\) in (1).

_Decision type_: As illustrated in Fig. 2(a) and Fig. 2(b), for any given input \(\mathbf{x}\), a conventional _point classifier_ produces as output a single estimate \(\hat{c}(\mathbf{x})\) of the label \(c\) in a non-adaptive (Fig. 2(a)) or adaptive (Fig. 2(b)) way. In contrast, as seen in Fig. 2(c) and Fig. 2(d), a _set classifier_ outputs a decision in the form of a _subset_\(\Gamma(\mathbf{x})\subseteq\mathcal{C}\) of the \(C\) classes [6, 7], with the decision being non-adaptive (Fig. 2(c)) or adaptive (Fig. 2(d)). The _predicted set_\(\Gamma(\mathbf{x})\) describes the classifier's estimate of the most likely candidate labels for input \(\mathbf{x}\). Accordingly, a predicted set \(\Gamma(\mathbf{x})\) with a larger cardinality \(|\Gamma(\mathbf{x})|\) is less _informative_ than one with a smaller (but non-zero) cardinality.

### _Reliability, Latency, and Inference Energy_

In this work, we study the performance of adaptive classifiers on the basis of the following metrics.

_Reliability_: Given a _target accuracy level_\(p_{\mathrm{targ}}\in(0,1)\), an adaptive _point_ classifier is said to be _reliable_ if the accuracy of its decision is no smaller than the target level \(p_{\mathrm{targ}}\). This condition is stated as

\[\Pr\bigl{(}c=\hat{c}(\mathbf{x})\bigr{)}\geq p_{\mathrm{targ}},\] \[\text{i.e., }\Delta R=p_{\mathrm{targ}}-\Pr\bigl{(}c=\hat{c}(\mathbf{x}) \bigr{)}\leq 0, \tag{5}\]

where \(\hat{c}(\mathbf{x})\) is the decision made by the adaptive point classifier at time \(T_{s}(\mathbf{x})\) (see Fig. 2(b)). In (5), we have defined the _reliability gap_\(\Delta R\), which is positive for _unreliable_ classifiers and non-positive for _reliable_ ones (see Fig. 1(b)). In a similar manner, an adaptive _set_ predictor \(\Gamma(\mathbf{x})\) is reliable at the target accuracy level \(p_{\mathrm{targ}}\) if the true class \(c\) is included in the predicted set \(\Gamma(\mathbf{x})\), produced at the stopping time \(T_{s}(\mathbf{x})\), with probability no smaller than the desired accuracy level \(p_{\mathrm{targ}}\). This is written as

\[\Pr\bigl{(}c\in\Gamma(\mathbf{x})\bigr{)}\geq p_{\mathrm{targ}},\] \[\text{i.e. }\Delta R=p_{\mathrm{targ}}-\Pr\bigl{(}c=\Gamma(\mathbf{x}) \bigr{)}\leq 0, \tag{6}\]

where \(\Gamma(\mathbf{x})\) is the decision made by the adaptive set classifier at time \(T_{s}(\mathbf{x})\) (see Fig. 2(d)). The probabilities in (5) and (6) are taken over the distribution of the test data point \((\mathbf{x},c)\) and of the calibration data (4).

_Latency_: Latency is defined as the average stopping time \(\mathbb{E}[T_{s}(\mathbf{x})]\), where the expectation is taken over the same distribution as for (5) and (6).

_Inference energy_: As a proxy for the energy consumption of the SNN classifier at inference time, we follow the standard approach also adopted in, e.g., [33, 38], of counting the average number of spikes, denoted as \(\mathbb{E}[S(\mathbf{x})]\), that are produced internally by the SNN classifier prior to producing a decision.

### _Spiking Neural Network Model_

In this work, we adopt the standard LIF neural model known as _spike response model_ (SRM) [34]. Consider a set of spiking neurons indexed via integers in set \(\mathcal{K}\). Each spiking neuron \(k\in\mathcal{K}\) outputs a binary signal \(b_{k,t}\in\{0,1\}\) at time \(t=1,...,T\), with \(b_{k,t}=1\) representing the firing of the spike and \(b_{k,t}=0\) an idle neuron at time \(t\). It receives inputs from a subset of neurons \(\mathcal{N}_{k}\) through directed links, known as _synapses_. Accordingly, neurons in set \(\mathcal{N}_{k}\) are referred to as _pre-synaptic_ with respect to neuron \(k\); while neuron \(k\) is said to be _post-synaptic_ for any neuron \(j\in\mathcal{N}_{k}\). For a fully-connected layered SNN, as assumed in the experiments of this paper, the set of pre-synaptic neurons, \(\mathcal{N}_{k}\), for a neuron \(k\) in a given layer consists of the entire set of indices of the neurons in the previous layer.

Following the SRM, each neuron \(k\) maintains an internal analog state variable \(o_{k,t}\), known as the _membrane potential_, over time \(t\). The membrane potential \(o_{k,t}\) evolves as the sum of the responses of the synapses to the incoming spikes produced by the pre-synaptic neurons, as well as of the response of the neuron itself to the spikes it produces. Mathematically, the evolution of the membrane potential is given as

\[o_{k,t}=\sum_{j\in\mathcal{N}_{k}}w_{k,j}\cdot(\alpha_{t}*b_{j,t})+\beta_{t}*b_{ k,t}, \tag{7}\]
where \(w_{k,j}\) is a learnable synaptic weight between neuron \(j\in\mathcal{N}_{k}\) and neuron \(k\); \(\alpha_{t}\) represents a filter applied to the spiking signals produced by each pre-synaptic neurons; \(\beta_{t}\) is the filter applied to its own spiking output; and "\(*\)" denotes the convolution operator.

Typical choices for synaptic filters include the first-order feedback filter \(\beta_{t}=\exp(-t/\tau_{\mathrm{ref}})\), and the second-order synaptic filter \(\alpha_{t}=\exp(-t/\tau_{\mathrm{mem}})-\exp(-t/\tau_{\mathrm{syn}})\), for \(t=1,2,...\), with finite positive constants \(\tau_{\mathrm{ref}}\), \(\tau_{\mathrm{mem}}\), and \(\tau_{\mathrm{syn}}\)[12]. Each neuron \(k\) outputs a spike at time step \(t\) whenever its membrane potential crosses a fixed threshold \(\vartheta\), i.e.,

\[b_{k,t}=\Theta(o_{k,t}-\vartheta), \tag{8}\]

where \(\Theta(\cdot)\) is the Heaviside step function.

The synaptic weights \(w_{k,j}\) in (7) between any neurons \(k\in\mathcal{K}\) and the corresponding pre-synaptic neurons \(j\in\mathcal{N}_{k}\) constitute the model parameters to be optimized during training. Accordingly, we write as \(\mathbf{\theta}=\{\{w_{k,j}\}_{j\in\mathcal{N}_{k}}\}_{k\in\mathcal{K}}\) the vector of model parameters of the SNN.

## III Adaptive Point Classification

In this section, we review, for reference, the adaptive point classifiers introduced in [2] and [3], which are referred to as _dynamic-confidence SNN_ (DC-SNN) and _stopping-policy SNN_ (SP-SNN), respectively.

_DC-SNN_[2]: As illustrated in Fig. 2(b), DC-SNN produces a decision at the first time \(t\) for which the maximum confidence level across all possible classes is larger than a fixed _target confidence level_\(p_{\text{th}}\in(0,1)\). Accordingly, the stopping time is given by

\[T_{s}(\mathbf{x})=\min_{t\in\{1,...,T\}}t\ \ \text{s.t.}\ \ \max_{c\in\mathcal{C}}p_{c}(\mathbf{x}^{t}) \geq p_{\text{th}}, \tag{9}\]

if there is a time \(t<T\) that satisfies the constraint; and \(T_{s}(\mathbf{x})=T\) otherwise. The rationale for this approach is that, by (9), if \(T_{s}(\mathbf{x})<T\), the classifier has a confidence level no smaller than \(p_{\text{th}}\) on the decision

\[\hat{c}(\mathbf{x})=\arg\max_{c\in\mathcal{C}}p_{c}(\mathbf{x}^{T_{s}(\mathbf{x})}). \tag{10}\]

If the SNN classifier is _well calibrated_, the confidence level coincides with the true accuracy of the decision given by the class \(\arg\max_{c\in\mathcal{C}}p_{c}(\mathbf{x}^{t})\) at all times \(t\). Therefore, setting the target confidence level \(p_{\text{th}}\) to be equal to the target accuracy \(p_{\text{aug}}\), i.e., \(p_{\text{th}}=p_{\text{aug}}\), guarantees a zero, or negative, reliability gap for the adaptive decision (10) when \(T_{s}(\mathbf{x})<T\). However, as discussed in Sec. I, the assumption of calibration is typically not valid (see Fig. 1(b)). To address this problem, reference [2] introduced a solution based on the use of a calibration data set.

Specifically, DC-SNN evaluates the empirical accuracy of the decision (10), i.e., \(\hat{\mathcal{A}}^{\text{cal}}(p_{\text{th}})=|\mathcal{D}^{\text{cal}}|^{-1} \sum_{i=1}^{|\mathcal{D}^{\text{cal}}|}\mathbb{1}\left(\hat{c}(\mathbf{x}[i])=c[i]\right)\), where \(\mathbb{1}(\cdot)\) is the indicator function, for a grid of possible values of the target confidence level \(p_{\text{th}}\). Then, it chooses the minimum value \(p_{\text{th}}\) that ensures the inequality \(\hat{\mathcal{A}}^{\text{cal}}(p_{\text{th}})\geq p_{\text{aug}}\), so that the calibration accuracy exceeds the target accuracy level \(p_{\text{aug}}\); or the smallest value \(p_{\text{th}}\) that maximizes \(\hat{\mathcal{A}}^{\text{cal}}(p_{\text{th}})\) if the constraint \(\hat{\mathcal{A}}^{\text{cal}}(p_{\text{th}})\geq p_{\text{aug}}\) cannot be met.

_SP-SNN_[3]: SP-SNN defines a parameterized _policy_\(\mathbf{\pi}(\mathbf{x}|\mathbf{\phi})\), implemented using a separate artificial neural network (ANN), that maps the input sequence \(\mathbf{x}\) to a probability distribution \(\mathbf{\pi}(\mathbf{x}|\mathbf{\phi})=[\pi_{1}(\mathbf{x}|\mathbf{\phi}),...,\pi_{T}(\mathbf{x}|\mathbf{ \phi})]\) over the \(T\) time steps, where \(\mathbf{\phi}\) is the trainable parameter vector of the ANN. Accordingly, given input \(\mathbf{x}\), the stopping time is drawn using the policy \(\mathbf{\pi}(\mathbf{x}|\mathbf{\phi})\) as \(T_{s}(\mathbf{x})\sim\mathbf{\pi}(\mathbf{x}|\mathbf{\phi})\).

Unlike DC-SNN, which uses a pre-trained SNN, the policy in SP-SNN is optimized jointly with the SNN based on an available training data set

\[\mathcal{D}^{\text{tr}}=\{(\mathbf{x}^{\text{tr}}[i],c^{\text{tr}}[i])\}_{i=1}^{| \mathcal{D}^{\text{tr}}|} \tag{11}\]

of \(|\mathcal{D}^{\text{tr}}|\) examples, whose data points are i.i.d. as for the calibration data set (4) and for the test data. Furthermore, unlike DC-SNN, SP-SNN does not make use of calibration data.

Optimization in SP-SNN targets an objective function that depends on a combination of latency and accuracy. To be

Fig. 2: (a) A _non-adaptive point_ classifier outputs a point decision \(\hat{c}(\mathbf{x})\) after having observed the entire time series \(\mathbf{x}\). (b) An _adaptive_ point classifier stops when the confidence level of the classifier passes a given threshold \(p_{\text{th}}\), producing a classification decision at an input-dependent time \(T_{s}(\mathbf{x})\). (c) A _non-adaptive set_ classifier produces a predicted set \(\Gamma(\mathbf{x})\) consisting of a subset of the class labels after having observed the entire time series \(\mathbf{x}\). (d) The _adaptive set_ classifiers presented in this work stop at the earliest time \(T_{s}(\mathbf{x})\) when the predicted set \(\Gamma(\mathbf{x}^{T_{s}(\mathbf{x})})\) is sufficiently informative, in the sense that its cardinality is below a given threshold \(d_{\text{th}}\) (in the figure we set \(d_{\text{th}}=2\)). The proposed SpikCP method can guarantee that the predicted set \(\Gamma(\mathbf{x})=\Gamma(\mathbf{x}^{T_{s}(\mathbf{x})})\) at the stopping time \(T_{s}(\mathbf{x})\) includes the true label with probability no smaller than the target probability \(p_{\text{aug}}\).
specific, given a training example \((\mathbf{x},c)\in\mathcal{D}^{\mathbf{x}}\), SP-SNN takes an action \(T_{s}(\mathbf{x})\) derived by the policy, from which a _reward_

\[R\big{(}T_{s}(\mathbf{x})\big{)}=\begin{cases}1/2^{T_{s}(\mathbf{x})},&\hat{c}(\mathbf{x})=c, \\ -\zeta,&\text{otherwise},\end{cases} \tag{12}\]

is provided to SP-SNN to optimize the policy ANN, where \(\zeta\) is a positive constant. Accordingly, if the prediction is correct, i.e., if \(\hat{c}(\mathbf{x})=c\), the reward (12) favors lower latencies by assigning a larger reward to a policy that produces a decision at an earlier time \(T_{s}(\mathbf{x})\). Conversely, if the prediction is wrong, a penalty \(\zeta\) is applied.

Accuracy in SP-SNN is accounted for via the standard _cross-entropy loss_. For an example \((\mathbf{x},c)\) at stopping time \(T_{s}(\mathbf{x})\), this is defined as

\[L(\mathbf{x}^{T_{s}(\mathbf{x})})=-\log p_{c}(\mathbf{x}^{T_{s}(\mathbf{x})}), \tag{13}\]

where probability \(p_{c}(\mathbf{x}^{T_{s}(\mathbf{x})})\) is defined in (3). Accordingly, SP-SNN jointly optimizes the SNN parameters \(\mathbf{\theta}\) (see Sec. VI) and the policy network parameters \(\mathbf{\phi}\) by addressing the problem

\[\min_{\mathbf{\theta},\mathbf{\phi}}\sum_{(\mathbf{x},c)\in\mathcal{D}^{\mathbf{x}}}\mathbb{E }[-R\big{(}T_{s}(\mathbf{x})\big{)}+L(\mathbf{x}^{T_{s}(\mathbf{x})})], \tag{14}\]

where the expectation is taken over the probability distribution \(\mathbf{\pi}(\mathbf{x}|\mathbf{\phi})\). The problem is tackled via an alternate application of reinforcement learning for the optimization of parameters \(\mathbf{\phi}\) and of supervised learning for the optimization of parameters \(\mathbf{\theta}\).

## IV SpikeCP: Reliable Adaptive Set Classification

The adaptive point classifiers reviewed in the previous section are generally characterized by a positive reliability gap (see Fig. 1(a)), unless the underlying SNN classifier is well calibrated or unless the calibration data set is large enough to ensure a reliable estimate of the true accuracy. In this section, we introduce _SpikeCP_, a novel inference methodology for adaptive classification that wraps around any pre-trained SNN model, guaranteeing the reliability requirement (6) - and hence a zero, or negative, reliability gap - irrespective of the quality of the SNN classifier and of the amount of calibration data. In the next section we discuss how to potentially improve the performance of SpikeCP by training tailored SNN models.

### _Stopping Time_

SpikeCP pre-determines a subset of possible stopping times, referred to as _checkpoints_, in set \(\mathcal{T}_{s}\subseteq\{1,...,T\}\). Set \(\mathcal{T}_{s}\subseteq\{1,...,T\}\) always includes the last time \(T\), and adaptivity is only possible if the cardinality of set \(\mathcal{T}_{s}\) is strictly larger than one. At each time \(t\in\mathcal{T}_{s}\), using the local spike count variables \(\mathbf{r}(\mathbf{x}^{t})\) or the global predictive probabilities \(\mathbf{p}(\mathbf{x}^{t})\), SpikeCP produces a candidate predicted set \(\Gamma(\mathbf{x}^{t})\subseteq\mathcal{C}\). Then, as illustrated in Fig. 2(d), the cardinality \(|\Gamma(\mathbf{x}^{t})|\) of the candidate predicted set \(\Gamma(\mathbf{x}^{t})\) is compared with a threshold \(I_{\rm th}\). If we have the inequality

\[|\Gamma(\mathbf{x}^{t})|\leq I_{\rm th}, \tag{15}\]

the predicted set is deemed to be sufficiently _informative_, and SpikeCP stops processing the input to produce set \(\Gamma(\mathbf{x}^{t})\) as the final decision \(\Gamma(\mathbf{x})\). As we detail next and as illustrated in Fig. 1(c), the candidate predicted sets \(\Gamma(\mathbf{x}^{t})\) are constructed in such a way to ensure a non-positive reliability gap simultaneously for _all_ checkpoints, and hence also at the stopping time. The overall procedure of SpikeCP is summarized in Algorithm 1.

To construct the candidate predicted set \(\Gamma(\mathbf{x}^{t})\) at a checkpoint \(t\in\mathcal{T}_{s}\), SpikeCP follows the _split_, or _validation-based_, CP procedure proposed in [6] and reviewed in [7, 39]. Accordingly, using the local rate counts \(\mathbf{r}(\mathbf{x}^{t})\) or the global probabilities \(\mathbf{p}(\mathbf{x}^{t})\), SpikeCP produces a so-called _non-continuity (NC) score_ vector \(\mathbf{s}(\mathbf{x}^{t})=[s_{1}(\mathbf{x}^{t}),...,s_{C}(\mathbf{x}^{t})]\). Each entry \(s_{c}(\mathbf{x}^{t})\) of this vector is a measure of the _lack of confidence_ of the SNN classifier in label \(c\) given input \(\mathbf{x}^{t}\). The candidate predicted set \(\Gamma(\mathbf{x}^{t})\) is then obtained by including all labels \(c\in\mathcal{C}\) whose NC score \(s_{c}(\mathbf{x}^{t})\) is no larger than a threshold \(s_{\rm th}^{t}\), i.e.,

\[\Gamma(\mathbf{x}^{t})=\{c\in\mathcal{C}:s_{c}(\mathbf{x}^{t})\leq s_{\rm th}^{t}\}. \tag{16}\]

As described in Sec. IV-B, the threshold \(s_{\rm th}^{t}\) is evaluated as a function of the target accuracy level \(p_{\rm targ}\), of the the calibration set \(\mathcal{D}^{\rm cal}\), and of the number of checkpoints \(|\mathcal{T}_{s}|\).

We consider two NC scores, one locally computable at the output neurons and one requiring coordination among the output neurons. The _local NC score_ is defined as

\[s_{c}(\mathbf{x}^{t})=t-r_{c}(\mathbf{x}^{t}). \tag{17}\]

Intuitively, class \(c\) is assigned a lower NC score (17) - and hence a higher degree of confidence - if the spike count variable \(r_{c}(\mathbf{x}^{t})\) is larger. In contrast, the _global NC score_ is given by the standard _log-loss_

\[s_{c}(\mathbf{x}^{t})=-\log p_{c}(\mathbf{x}^{t}). \tag{18}\]

### _Evaluation of the Threshold_

As we detail in this subsection, the threshold \(s_{\rm th}^{t}\) in (16) is evaluated based on the calibration data set \(\mathcal{D}^{\rm cal}\) with the goal of ensuring the reliability condition (6) for a target accuracy level \(p_{\rm arg}\). The general methodology follows CP, with the important caveat that, in order to ensure a non-positive reliability gap simultaneously at all checkpoints, a form of _Bonferroni correction_ is applied. Alternative, heuristic, corrections are also described at the end of this section.

Let us define as \(1-\alpha\), with \(\alpha\in(0,1)\), an auxiliary _per-checkpoint accuracy level_. Suppose that we can guarantee the _per-checkpoint reliability condition_

\[\Pr(c\in\Gamma(\mathbf{x}^{t}))\geq 1-\alpha \tag{19}\]

Fig. 3: CP meets condition (19) by choosing the threshold \(s_{\rm th}^{t}\) in (16) as the \(\lceil(1-\alpha)(|\mathcal{D}^{\rm cal}|+1)\rceil\)-th smallest value among the NC scores evaluated in the calibration set.
for all checkpoints \(t\in\mathcal{T}_{s}\). In (19), the probability is taken over the distribution of the test and calibration data. We will see below that this condition can be guaranteed by leveraging the toolbox of CP. Then, by the union bound, we also have the reliability condition

\[\Pr(c\in\Gamma(\mathbf{x}^{t})\text{ for all }t\in\mathcal{T}_{s})\geq 1-|\mathcal{T}_{s}|\alpha, \tag{20}\]

which applies simultaneously across all checkpoints. This inequality implies that we can guarantee the condition (6) by setting \(\alpha=(1-p_{\text{targ}})/|\mathcal{T}_{s}|\), since the stopping point \(T_{s}(\mathbf{x})\) is in set \(\mathcal{T}_{s}\) by construction. This is a form of _Bonferroni correction_, whereby the target accuracy for the test carried out at each checkpoint is increased in order to ensure reliability simultaneously for the tests at all checkpoints [32]. This increase is linear in the number of checkpoints \(|\mathcal{T}_{s}|\), and it guarantees the desired reliability condition irrespective of the underlying distribution of the data, as long as the per-checkpoint inequality (19) is satisfied.

The remaining open question is how to ensure the per-checkpoint reliability condition (19). To address this goal, we follow the standard CP procedure. Accordingly, during an _offline_ phase, for each calibration data point \((\mathbf{x}[i],c[i])\), with \(i=1,...,|\mathcal{D}^{\text{cal}}|\), SpikeCP computes the NC score \(s^{t}[i]=s_{c[i]}(\mathbf{x}^{t}[i])\) at each checkpoint \(t\in\mathcal{T}_{s}\). The calibration NC scores \(\{s^{t}[i]\}_{i=1}^{|\mathcal{D}^{\text{cal}}|}\) are ordered from smallest to largest, with ties broken arbitrarily, separately for each checkpoint \(t\). Finally, the threshold \(s_{\text{th}}^{t}\) is selected to be approximately equal to the smallest value that is larger than a fraction \((1-\alpha)\) of the calibration NC scores (see Fig. 3). More precisely, assuming \(\alpha\geq 1/(|\mathcal{D}^{\text{cal}}|+1)\) we set [6, 7]

\[s_{\text{th}}^{t}=\lceil(1-\alpha)(|\mathcal{D}^{\text{cal}}|+1)\rceil \text{-th smallest value}\\ \text{in the set }\{s^{t}[i]\}_{i=1}^{|\mathcal{D}^{\text{cal}}|}, \tag{21}\]

while for \(\alpha<1/(|\mathcal{D}^{\text{cal}}|+1)\) we set \(s_{\text{th}}^{t}=\infty\). This is illustrated in Fig. 3.

### _Reliability Guarantees of SpikeCP_

In this subsection, we show that SpikeCP, as summarized in Algorithm 1, satisfies the reliability condition (6).

**Theorem 1** (Reliability of SpikeCP).: _The adaptive decision \(\Gamma(\mathbf{x})=\Gamma(\mathbf{x}^{T_{s}(\mathbf{x})})\) produced by SpikeCP, as described in Algorithm 1, satisfies the reliability condition (6), and hence has a non-positive reliability gap, i.e., \(\triangle R\leq 0\)._

Proof.: By the properties of CP, the threshold (21) ensures the per-checkpoint reliability condition (19) (see, e.g., [40, Theorem 1] and [6, 41, 42]). As proved in the last subsection, this is sufficient to conclude that the reliability condition (6) is satisfied. We refer to Appendix for further details. 

### _An Alternative Heuristic Threshold Selection_

The theoretical guarantees of SpikeCP in Theorem 1 rely on the Bonferroni correction that sets the per-checkpoint target accuracy level to \(1-\alpha=1-(1-p_{\text{targ}})/|\mathcal{T}_{s}|\). This requirement becomes increasingly stricter, and hence harder to satisfy, as the number of checkpoints \(|\mathcal{T}_{s}|\) increases. However, having a large number of checkpoints may be advantageous by enhancing the granularity of delay adaptivity.

In this subsection, we introduce an alternative, heuristic, choice for the per-checkpoint reliability condition based on Simes correction [9]. The approach sets a different target \(1-\alpha_{t}\) for each checkpoint \(t\in\mathcal{T}_{s}\), by imposing the constraint

\[\Pr(c\in\Gamma(\mathbf{x}^{t}))\geq 1-\alpha_{t} \tag{22}\]

in lieu of the constant-target condition (19). For each time step \(t\in\mathcal{T}_{s}\), let us define \(i_{t}\) for the index that runs across the checkpoints as \(i_{t}=\sum_{t^{\prime}\in\mathcal{T}_{s}}\mathbb{1}(t^{\prime}\leq t)\). Then, the target reliability for the checkpoint at time \(t\in\mathcal{T}_{s}\) is set to \(1-\alpha_{t}\) with

\[\alpha_{t}=i_{t}\cdot\frac{(1-p_{\text{targ}})}{|\mathcal{T}_{s}|}. \tag{23}\]

Accordingly, for the first checkpoint \(t\), with \(i_{1}=1\), the target coincides with that obtained from Bonferroni correction, i.e., \(\alpha_{t}=(1-p_{\text{targ}})/|\mathcal{T}_{s}|\); while for the last checkpoint, with \(i_{t}=|\mathcal{T}_{s}|\), it corresponds to the target accuracy level, i.e., \(\alpha_{t}=1-p_{\text{targ}}\).

Using Simes correction (23) in step 3 in Algorithm 1 in lieu of \(\alpha_{t}=(1-p_{\text{targ}})/|\mathcal{T}_{s}|\), yields an alternative version of SpikeCP that is guaranteed to meet the reliability condition (5) only under additional assumptions that are hard to verify in practice (see Appendix). One of such assumptions is that the accuracy of SNN never decreases with increased time steps, as posited, e.g., in [3, Assumption 3.1]. Given this limitation, we propose Simes correction here merely as a heuristic, which may yield some practical gains as demonstrated in Sec. VI-C (see Fig. 8).
## V SpikeCP-Based Training

While SpikeCP provides guarantees on the reliability of its set-valued decisions irrespective of the quality of the pre-trained SNN (see Theorem 1), the achievable trade-offs between average delay and energy consumption, on the one hand, and informativeness of the set predictor, on the other, generally depend on the performance of the underlying SNN-based classifier. In this section, we introduce a training strategy - referred to as _SpikeCP-based training_ - that, unlike conventional learning algorithms for SNNs (see, e.g., [10, 34]), targets directly the performance of the SNN when used in conjunction with SpikeCP.

### _Training Objective_

In order to describe the training objective of SpikeCP-based training, we start by recalling from Sec. IV-A that the stopping time of SpikeCP is determined by the size \(|\Gamma(\mathbf{x}^{t})|\) of the predicted set \(\Gamma(\mathbf{x}^{t})\) for input \(\mathbf{x}\) as per the threshold rule (15) with target set size \(I_{\rm th}\). Therefore, to reduce the average latency, one can train the SNN with the aim at minimizing the sizes \(|\Gamma(\mathbf{x}^{t})|\) of the predicted sets \(\Gamma(\mathbf{x}^{t})\) in (16) produced by SpikeCP over time instants \(t\) with the set \(\mathcal{T}_{s}\) of candidates checkpoints.

To this end, the model parameters \(\mathbf{\theta}\) are optimized on the basis of the training set (11). Specifically, in order to mimic the test-time distinction between calibration and test data leveraged by SpikeCP, we randomly partition the training set \(\mathcal{D}^{\rm tr}\) into two disjoint subsets \(\mathcal{D}^{\rm t,cal}\) and \(\mathcal{D}^{\rm tr,le}\) with \(\mathcal{D}^{\rm t,cal}\cap\mathcal{D}^{\rm tr,le}=\emptyset\) and \(\mathcal{D}^{\rm tr,cal}\cup\mathcal{D}^{\rm tr,le}=\mathcal{D}^{\rm tr}\).

Given a data set split \((\mathcal{D}^{\rm tr,cal},\mathcal{D}^{\rm tr,le})\), we run SpikeCP (Algorithm 1) with \(\mathcal{D}^{\rm tr,cal}\) in lieu of the calibration data \(\mathcal{D}^{\rm cal}\), and with the input parts of the data points in the set \(\mathcal{D}^{\rm tr,le}\) as the test inputs \(\mathbf{x}\). For each such test input \(\mathbf{x}\) in \(\mathcal{D}^{\rm tr,le}\), SpikeCP returns the predictive set \(\Gamma(\mathbf{x}^{t})\) for all times \(t\in\mathcal{T}_{s}\). In line with the motivation explained in the previous paragraph, we consider the set sizes \(|\Gamma(\mathbf{x}^{t})|\) for all times \(t\) in the checkpoint set \(\mathcal{T}_{s}\) as the target of the training process.

To quantify the mentioned predictive set sizes using training data, we define the _efficiency training loss_

\[\mathcal{L}^{E}(\mathbf{\theta})=\sum_{\begin{subarray}{c}\mathcal{D}^{\rm real} \subset\mathcal{D}^{\rm tr}\\ \mathcal{D}^{\rm tr,le}=\mathcal{D}^{\rm tr}\setminus\mathcal{D}^{\rm real} \end{subarray}}\sum_{(\mathbf{x},c)\in\mathcal{D}^{\rm min}}\sum_{t\in\mathcal{T }_{s}}|\Gamma(\mathbf{x}^{t})|. \tag{24}\]

The outer sum in (24) is over a number of splits realized by randomly sampling the subset \(\mathcal{D}^{\rm tr,cal}\subset\mathcal{D}^{\rm tr}\) for a fixed given number of calibration data points \(|\mathcal{D}^{\rm tr,cal}|<|\mathcal{D}^{\rm tr}|\); the middle sum is over the test data points in set \(\mathcal{D}^{\rm tr,le}=\mathcal{D}^{\rm tr}\setminus\mathcal{D}^{\rm tr,cal}\); and the inner sum is over the time instants in the checkpoint set \(\mathcal{T}_{s}\).

The efficiency training loss \(\mathcal{L}^{E}(\mathbf{\theta})\) in (24) does not make use of the labels of the test data sets, and it does not directly target the accuracy of the SNN classifier. In a manner somewhat similar to the criterion (14) used by SP-SNN, we hence propose to complement the efficiency training loss with the standard _cross-entropy training loss_ as

\[\mathcal{L}^{C}(\mathbf{\theta})=-\sum_{\begin{subarray}{c}\mathcal{D}^{\rm real} \subset\mathcal{D}^{\rm tr}\\ \mathcal{D}^{\rm tr,le}=\mathcal{D}^{\rm tr}\setminus\mathcal{D}^{\rm real} \end{subarray}}\sum_{(\mathbf{x},c)\in\mathcal{D}^{\rm min}}\sum_{t\in\mathcal{T }_{s}}\log p_{c}(\mathbf{x}^{t}), \tag{25}\]

where \(p_{c}(\mathbf{x}^{t})\) is the probability value assigned by the model to input \(\mathbf{x}^{t}\) for class \(c\) using (3). The sums in (25) are evaluated as for the efficiency training loss (24).

Overall, we propose to optimize the parameter vector \(\mathbf{\theta}\) of SNN by addressing the problem

\[\min_{\mathbf{\theta}}\mathcal{L}^{C}(\mathbf{\theta})+\lambda\mathcal{L}^{E}(\mathbf{ \theta}) \tag{26}\]

for a hyperparameter \(\lambda\geq 0\) that dictates the trade-off between cross-entropy and efficiency criteria. With \(\lambda=0\) and \(\mathcal{T}_{s}=\{T\}\), this training objective recovers the conventional cross-entropy evaluated at the last time instant adopted in most of the literature on SNN-based classification (see, e.g., [10, 38]).

### _Training_

The gradient of the standard cross-entropy objective \(\mathcal{L}^{C}(\mathbf{\theta})\) can be approximated via well-established surrogate gradient methods that apply the straight-through estimator of the gradient [34, 35]. Accordingly, when applying backpropagation, while the forward pass uses the actual non-differentiable activation model (8) of the SRM neurons, the backward pass replaces the non-differentiable spiking threshold function (8) with a smooth sigmoidal function [34, 35]. For each neuron \(k\) at time \(t\), this yields the differentiable activation

\[\hat{b}_{k,t}=\sigma(o_{k,t}-\vartheta), \tag{27}\]

where the Heaviside step function \(\Theta(\cdot)\) in (8) is replaced by the sigmoid function \(\sigma(x)=1/(1+e^{-x})\).

Given the availability of surrogate gradient methods, the main new challenge in tackling problem (26) lies in the evaluation of the gradient of the criterion \(\mathcal{L}^{E}(\mathbf{\theta})\). The rest of this section focuses on this problem.

The efficiency training loss \(\mathcal{L}^{E}(\mathbf{\theta})\) in (24) depends on the cardinality \(|\Gamma(\mathbf{x}^{t})|\), which is a non-differentiable function of the model parameters \(\mathbf{\theta}\), even when considering the surrogate SNN model with activation function in (27). In fact, the NC scores \(s_{c}(\mathbf{x}^{t})\) in (17) or (18) are differentiable in \(\mathbf{\theta}\) under the surrogate model (27), but this is not the case for the cardinality \(|\Gamma(\mathbf{x}^{t})|\) of the predicted set.

To see this, observe that cardinality \(|\Gamma(\mathbf{x}^{t})|\) is obtained via a cascade of two non-differentiable functions of the scores \(s^{t}[i]\), \(i=1,...,|\mathcal{D}^{\rm cal}|\): (_i) Sorting_: By Algorithm 1, SpikeCP sorts the calibration scores \(s^{t}[i]\), \(i=1,...,|\mathcal{D}^{\rm cal}|\), to obtain the threshold \(s^{t}_{\text{th}}\) via (21) at each checkpoint time \(t\in\mathcal{T}_{s}\); (_ii) Counting_: The cardinality \(|\Gamma(\mathbf{x}^{t})|\) of the set predictor is obtained by counting the number of labels \(c\) whose score \(s_{c}(\mathbf{x}^{t})\) is no larger than the threshold \(s^{t}_{\text{th}}\), i.e., \(|\Gamma(\mathbf{x}^{t})|=\sum_{t=1}^{C}\mathbbm{1}\left(s_{c}(\mathbf{x}^{t})\leq s^{t}_{ \text{th}}\right)\).

In the next subsection, we introduce a differentiable approximation \(|\Gamma(\mathbf{x}^{t})|\) of the cardinality function \(|\Gamma(\mathbf{x}^{t})|\) under the smooth activation (31). The approach follows prior art on CP-aware training [27, 28, 29].
### _Differentiable Threshold and Set Cardinality_

The threshold \(s_{\text{th}}^{t}\) in (21) amounts to the \((1-\alpha)\)-empirical quantile of the calibration scores \(s^{t}[i]\), \(i=1,...,|\mathcal{D}^{\text{cal}}|\). Given \(\mathcal{D}^{\text{tr,cal}}\), this can be obtained as the solution of the problem

\[s_{\text{th}}^{t}=\arg\min_{s\in\{s^{t}[i]\}_{i=1}^{|\mathcal{D}^{\text{cal}}| }\big{(}\rho_{1-\alpha}(s|\{s^{t}[i]\}_{i=1}^{|\mathcal{D}^{\text{cal}}|}\cup \{\infty\})\big{)} \tag{28}\]

where we have defined the _pinball loss_ as

\[\rho_{1-\alpha}(a|\{a[i]\}_{i=1}^{M}) =\alpha\sum_{i=1}^{M}\text{ReLU}(a-a[i])+\] \[(1-\alpha)\sum_{i=1}^{M}\text{ReLU}(a[i]-a), \tag{29}\]

for \(M\) real numbers \(\{a[i]\}_{i=1}^{M}\) with \(\text{ReLU}(a)=\max(0,a)\).

The solution of problem (28) can be approximated by replacing the minimum with a _soft minimum_ function \(\delta(x_{i})=e^{-x_{i}}/\sum_{j}e^{-x_{j}}\). Accordingly, a differentiable estimate of the threshold \(s_{\text{th}}^{t}\) can be written as [29]

\[\hat{s}_{\text{th}}^{t}=\sum_{i=1}^{|\mathcal{D}^{\text{tr,cal}}|+1}s^{t}[i] \delta\big{(}\frac{\rho_{1-\alpha}(s^{t}[i]\{s^{t}[i]\}_{i=1}^{|\mathcal{D}^{ \text{tr,cal}}|+1})}{c_{Q}}\big{)}, \tag{30}\]

where we have defined \(s^{t}[|\mathcal{D}^{\text{tr,cal}}|+1]=\max(\{s^{t}[i]\}_{i=1}^{|\mathcal{D}^ {\text{tr,cal}}|})+\beta\) for some sufficiently large parameter \(\beta>0\). In (30), the hyperparameter \(c_{Q}>0\) dictates the trade-off between smoothness and accuracy of the approximation. With small enough \(c_{Q}\), the smoothed threshold \(\hat{s}_{\text{th}}^{t}\) recovers the original value \(s_{\text{th}}^{t}\), in the sense that we have the limit \(\lim_{c_{Q}\to 0}\hat{s}_{\text{th}}^{t}=s_{\text{th}}^{t}\).

Based on the differentiable approximation \(\hat{s}_{\text{th}}^{t}\) introduced in the previous subsection, we can approximate the cardinality \(|\Gamma(\mathbf{x}^{t})|\) as the sum \(\sum_{c=1}^{C}\mathbb{1}\left(s_{c}(\mathbf{x}^{t})\leq\hat{s}_{\text{th}}^{t}\right)\). Since the indicator function \(\mathbb{1}(\cdot)\) is also not differentiable, we replace the indicator function with the sigmoid function \(\sigma(x)\) to obtain the following final differentiable approximation of the size of the set predictor

\[|\hat{\Gamma}(\mathbf{x}^{t})|=\sum_{c=1}^{C}\sigma\big{(}\hat{s}_{\text{th}}^{t} -s_{c}(\mathbf{x}^{t})\big{)}. \tag{31}\]

## VI Experiments

In this section, we provide experimental results to compare the performance of the adaptive point classifier DC-SNN [2], described in Sec. III, and of the proposed set classifier SpikeCP. We also provide insights into the trade-off between delay/energy and informativeness enabled by SpikeCP, as well as into the benefits of SpikeCP-based training. Finally, we offer a numerical comparison between the performance levels obtained by SpikeCP with Bonferroni and Simes corrections. All the experiments were run over a GPU server with single NVIDIA A100 card.

### _Setting_

We consider the MNIST-DVS dataset [5], which contains labelled \(26\times 26\) spiking signals of duration \(T=80\) samples. Each data point contains \(26\times 26=676\) spiking signals, which are recorded from a DVS camera that is shown moving handwritten digits from \(``0"\) to \(``9"\) on a screen. The data set contains \(8,000\) training examples, as well as \(2,000\) examples used for calibration and testing. We adopt a fully connected SNN with one hidden layer having \(1,000\) neurons, which is trained via the surrogate gradient method as in [37]. Except for the SpikeCP-based training, all the results reported in this section adopt a pre-trained SNN that is trained by assuming \(\lambda=0\) and \(\mathcal{T}_{s}=\{T\}\) as discussed in Sec V, given training data set \(\mathcal{D}^{\text{tr}}\) that is consisted of \(8,000\) examples, with each class containing \(800\) examples. The calibration data set \(\mathcal{D}^{\text{cal}}\) is obtained by randomly sampling \(|\mathcal{D}^{\text{cal}}|\) examples from the \(2,000\) data points allocated for calibration and testing, with the rest used for testing (see, e.g., [25]). We average the performance measures introduced in Sec. II-C over \(50\) different realizations of calibration and test data set. For SpikeCP, we assume the set of possible checkpoints as \(\mathcal{T}_{s}=\{20,40,60,80\}\), and use the global NC score (18) for SpikeCP, and we set the target set size to \(I_{\text{th}}=5\), unless specified otherwise.

In this work, we implement the policy network of SP-SNN as a recurrent neural network (RNN) with one hidden layer having \(500\) hidden neurons equipped with Tanh activation, followed by \(T=80\) output neurons with a softmax activation function. The RNN takes the time series data \(\mathbf{x}=\{\mathbf{x}_{t}\}_{t=1}^{T}\) as input, and outputs a probability vector \(\mathbf{\pi}(\mathbf{x}|\mathbf{\phi})\). The stopping time is chosen as \(T_{s}(\mathbf{x})=\arg\max_{t\in\{1,...,T\}}\pi_{t}(\mathbf{x}|\mathbf{\phi})\) during the testing phase. The choice of a light-weight RNN architecture for policy network is dictated by the principle of ensuring that the size of the additional ANN is comparable to that of the SNN classifier [3].

For SpikeCP-based training, we assume data is split by considering the actual number of calibration data, i.e., \(|\mathcal{D}^{\text{tr,cal}}|=\min\{|\mathcal{D}^{\text{cal}}|,|\mathcal{D}^{ \text{tr}}|/2\}\), which also ensures a non-empty set \(\mathcal{D}^{\text{tr,be}}\). The hyperparameters \(c_{Q}\) and \(\beta\) are set to \(0.001\) and \(1\), respectively. The weight factor \(\lambda\) is set to 0.01, and the target accuracy level is set to \(0.9\), i.e., \(\alpha=0.1\).

### _Performance Analysis of SpikeCP with a Pre-Trained SNN_

We start by evaluating the performance with the same pre-trained SNN model for all schemes. Fig. 4 reports accuracy - \(\Pr(c=\hat{c}(\mathbf{x}))\) for DC-SNN and \(\Pr(c\in\Gamma(\mathbf{x}))\) for SpikeCP - and normalized latency \(\mathbb{E}[T_{s}(\mathbf{x})]/T\) as a function of the target accuracy \(p_{\text{targ}}\) for different sizes \(|\mathcal{D}^{\text{cal}}|\) of the calibration data set. The accuracy plots highlight the regime in which we have a positive reliability gap \(\Delta R\) in (5) and (6), which corresponds to _unreliable_ decisions.

For reference, in Fig. 4(a), we show the performance obtained by setting the threshold \(p_{\text{th}}\) in (9) to the accuracy target \(p_{\text{targ}}\). Following the results reported in Fig. 1(b), this approach yields unreliable decisions as soon as the target accuracy level is sufficiently large, here larger than \(0.7\). By leveraging calibration data, DC-SNN can address this problem, suitably increasing the decision latency as \(p_{\text{targ}}\) increases.
However, reliability - i.e., a non-positive reliability gap - is only approximately guaranteed when the number of calibration data points is sufficiently large, here \(|\mathcal{D}^{\text{cal}}|=100\).

In contrast, as shown in Fig. 4(b) and proved in Theorem 1, SpikeCP is always reliable, achieving a non-positive reliability gap irrespective of the number of calibration data points. With a fixed threshold \(I_{\text{th}}\), as in this example, increasing the size \(|\mathcal{D}^{\text{cal}}|\) of the calibration data set has the effect of significantly reducing the average latency.

The trade-off supported by SpikeCP between latency and energy, on the one hand, and informativeness, i.e., set size, on the other hand, is investigated in Fig. 5 by varying the target set size \(I_{\text{th}}\), with target accuracy level \(p_{\text{arg}}=0.9\) and \(|\mathcal{D}^{\text{cal}}|=200\) calibration examples. Note that the reliability gap is always negative as in Fig. 4(b), and is hence omitted in the figure to avoid clutter. Increasing the target set size, \(I_{\text{th}}\), causes the final predicted set size, shown in the figure normalized by the number of classes \(C=10\), to increase, yielding less informative decisions. On the flip side, sacrificing informativeness entails a lower (normalized) latency, as well as, correspondingly, a lower inference energy, with the latter shown in the figure as the average number of spikes per sample and per hidden neuron, \(\mathbb{E}[S(\mathbf{x})]/(1000T)\).

In Fig. 6, we show the performance of SpikeCP when using either local NC scores (17) or global NC scores (18) (see Sec. IV), as well as the performance of the DC-SNN and SP-SNN point predictors, as a function of the number of checkpoints \(|\mathcal{T}_{s}|\), for \(p_{\text{targ}}=0.9\), \(|\mathcal{D}^{\text{cal}}|=200\), and \(I_{\text{th}}=5\). The checkpoints are equally spaced among the \(T\) time steps, and hence the checkpoint set is \(\mathcal{T}_{s}=\{T/|\mathcal{T}_{s}|,2T/|\mathcal{T}_{s}|,...,T\}\). The metrics displayed in the four panels are the accuracy - probability \(\Pr(c=\hat{c}(\mathbf{x}))\) for point predictors and probability \(\Pr(c\in\Gamma(\mathbf{x}))\) for set predictors - along with normalized latency \(\mathbb{E}[T_{s}(\mathbf{x})]/T\) and normalized, per-neuron and per-time step, inference energy \(\mathbb{E}[S(\mathbf{x})]/(1000T)\). Note that the operation of SP-SNN and DC-SNN does not depend on the number of checkpoints, and hence the performance of these schemes is presented as a constant function.

By Theorem 1, SpikeCP always achieves negative reliability gap, while SP-SNN and DC-SNN fall short of the target reliability \(p_{\text{arg}}\) in this example. Using global NC scores with SpikeCP yields better performance in terms of informativeness, and hence the performance of the SP-SNN and DC-SNN are comparable.

Fig. 4: (a) Accuracy \(\Pr(c=\hat{c}(\mathbf{x}))\) and normalized latency \(\mathbb{E}[T_{s}(\mathbf{x})]/T\) for the DC-SNN point classifier [2]. (b) Accuracy \(\Pr(c\in\Gamma(\mathbf{x}))\) and normalized latency \(\mathbb{E}[T_{s}(\mathbf{x})]/T\) for the proposed SpikeCP set predictor given the target set size \(I_{\text{th}}=5\). The shaded error bars correspond to intervals covering \(95\%\) of the realized values, obtained from \(50\) different draws of calibration data.

Fig. 5: Normalized latency, inference energy, and set size (informativeness) as a function of target set size \(I_{\text{th}}\) for SpikeCP, assuming \(p_{\text{arg}}=0.9\) and \(|\mathcal{D}^{\text{cal}}|=200\) under the same conditions as Fig. 4.

Fig. 6: Accuracy, normalized latency, normalized set size (informativeness), and normalized inference energy as a function of number of checkpoints \(|\mathcal{T}_{s}|\) for SpikeCP with local and global scores, as well as for DC-SNN and SP-SNN point classifiers, with \(p_{\text{targ}}=0.9\), \(|\mathcal{D}^{\text{cal}}|=200\), and \(I_{\text{th}}=5\).
ness, i.e., set size, as well as latency and inference energy. The performance gap between the two choices of NC scores increases with the number of checkpoints, demonstrating that local NC scores are more sensitive to the Bonferroni correction applied by SpikeCP (see Sec. 4). This is due to the lower discriminative power of local confidence levels, which yield less informative NC scores (see, e.g., [8]). That said, moderate values of latency and inference energy can also be obtained with local NC scores, without requiring any coordination among the readout neurons. This can be considered to be one of the advantages of the calibration afforded by the use of SpikeCP.

With global NC scores, the number of checkpoints \(|\mathcal{T}_{\mathrm{s}}|\) is seen to control the trade-off between latency and informativeness for SpikeCP. In fact, a larger number of checkpoints improves the resolution of the stopping times, while at the same time yielding more conservative set-valued decision at each time step due to the mentioned Bonferroni correction.

In Fig. 7, we show the performance of SpikeCP with global NC scores, DC-SNN, and SP-SNN as a function of the number, \(|\mathcal{D}^{\mathrm{cal}}|\), of calibration data points, with \(p_{\mathrm{targ}}=0.9\), \(|\mathcal{T}_{\mathrm{s}}|=4\), and \(I_{\mathrm{th}}=5\). The general conclusions around the comparisons among the different schemes are aligned with those presented above for Fig. 6. The figure also reveals that SP-SNN outperforms DC-SNN when the calibration data set is small, while DC-SNN is preferable in the presence of a sufficiently large data set. Finally, with a larger calibration data set, SpikeCP is able to increase the informativeness of the predicted set, while also decreasing latency and inference energy.

### _Comparing Bonferroni and Simes Corrections_

In Fig. 8, we study the performance of SpikeCP, which uses Bonferroni correction (see Sec. IV-B), with a heuristic variant of SpikeCP that uses Simes correction (see Sec. IV-D) with \(p_{\mathrm{targ}}=0.8\) and \(p_{\mathrm{targ}}=0.9\). Fig. 8 plots accuracy and normalized latency as a function of the number of checkpoints. As discussed in Sec. IV-D, the Bonferroni correction applied by SpikeCP becomes increasingly strict as the number of checkpoints increases. Accordingly, alternative correction factors, such as Simes, may become advantageous in the regime of large number of checkpoints. Confirming this argument, the figures show that indeed Simes correction can yield some advantage in terms of latency, while still satisfying, despite its lack of theoretical guarantees, the reliability requirement (5).

### _Performance Analysis of SpikeCP-based Training_

We finally turn to analyzing the potential benefits of SikeCP-based training, as introduced in Sec. V. Accordingly, the SNN classifier is trained by maximizing the objective in (26), with hyperparameter \(\lambda\) dictating the relative weight given to the prediction set efficiency over the conventional cross-entropy performance metric. With \(\lambda=0\), we recover the same SNN model assumed throughout the rest of the section, while larger values of \(\lambda>0\) ensure that the training model is increasingly tailored to the use of SpikeCP during inference by targeting the predictive set inefficiency.

In order to elaborate on the choice of hyperparameter \(\lambda\), in Fig. 9 we plot the normalized latency of SpikeCP as a function of \(\lambda\). For both target accuracy values \(p_{\mathrm{targ}}=0.8\) and \(p_{\mathrm{targ}}=0.9\), it is observed that there is an optimal value of \(\lambda\) that

Fig. 8: Accuracy and normalized latency as a function of number of checkpoints \(|\mathcal{T}_{\mathrm{s}}|\) for SpikeCP, which uses the Bonferroni correction, as well as for a variant that applies Simes correction (see Sec. IV-D), with \(p_{\mathrm{targ}}=0.8\) and \(p_{\mathrm{targ}}=0.9\).

Fig. 7: Accuracy, normalized latency, normalized set size (informativeness), and normalized inference energy as a function of number \(|\mathcal{D}^{\mathrm{cal}}|\) of calibration data points for SpikeCP with global NC scores, as well as for DC-SNN and SP-SNN point classifiers, with \(p_{\mathrm{targ}}=0.9\), \(|\mathcal{T}_{\mathrm{s}}|=4\), and \(I_{\mathrm{th}}=5\).

Fig. 9: Normalized latency as a function of the weight factor \(\lambda\) in the training objective (26) for training-based SpikeCP under target accuracy \(p_{\mathrm{targ}}=0.8\) and \(p_{\mathrm{targ}}=0.9\), assuming \(|\mathcal{D}^{\mathrm{cal}}|=200\) calibration data points with the same other conditions as in Fig. 4.
balances the inefficiency and accuracy (cross-entropy) criteria. Increasing \(\lambda\) is initially beneficial, yielding smaller predictive sets and hence smaller latencies. However, larger values of \(\lambda\) eventually downweigh excessively the accuracy criterion, producing worse performance. Furthermore, the optimal value of \(\lambda\) is seen to be decreasing with growing target reliability levels \(p_{\text{targ}}\), which call for more emphasis on the cross-entropy criterion.

We now turn to comparing the performance of SpikeCP-based training with conventional SpikeCP (with \(\lambda=0\)), DC-SNN and SP-SNN. Specifically, Fig. 10 plots accuracy and normalized latency as a function of the number of training data points \(|\mathcal{D}^{\text{tr}}|\). The point classifiers DC-SNN and SP-SNN exhibit an increasing accuracy level as the training data set size increases, while still failing to meet the reliability target \(p_{\text{targ}}=0.9\). In contrast, SpikeCP schemes meet the reliability requirement for any number of training data points. More training data translate into a lower latency, with SpikeCP-based training, here run with \(\lambda=0.01\), proving an increasingly sizeable latency reduction.

## VII Conclusions

In this work, we have introduced SpikeCP, a delay-adaptive SNN set predictor with provable reliability guarantees. SpikeCP wraps around any pre-trained SNN classifier, producing a set classifier with a tunable trade-off between informativeness of the decision - i.e., size of the predicted set - and latency, or inference energy as measured by the number of spikes. Unlike prior art, the reliability guarantees of SpikeCP hold irrespective of the quality of the pre-trained SNN and of the number of calibration points, with minimal added complexity. SpikeCP was also integrated with a CP-aware training strategy that complements the conventional cross-entropy criterion with a regularizer accounting for the informativeness of the predicted set.

Among directions for future work, we highlight extensions of SpikeCP that take into account time decoding or Bayesian learning [43] in order to further reduce the number of spikes and enhance the reliability of confidence estimates.

## Appendix: CP and Hypothesis Testing

As detailed in Sec. IV, SpikeCP relies on the use of the Bonferroni, or Simes, corrections, which are tools introduced in the literature on hypothesis testing [44]. In this appendix, we elaborate on the connection between CP and multiple-hypothesis testing.

Conventional CP effectively applies a binary hypothesis test for each possible label \(c\), testing the null hypothesis that the label \(c\) is the correct one. With the notation of this paper, for any fixed time \(t\), CP considers the null hypothesis

\[\mathcal{H}_{t}(\mathbf{x}^{t},c):(\mathbf{x}^{t},c)\text{ and the calibration data }\mathcal{D}^{t,\text{cal}}\text{ are i.i.d.},\]

where we have defined \(\mathcal{D}^{t,\text{cal}}=\{(\mathbf{x}^{t}[i],c[i])\}_{i=1}^{|\mathcal{D}^{ \text{cal}}|}\). In fact, if this hypothesis holds true, label \(c\) is the ground-truth label for input \(\mathbf{x}^{t}\).

Suppose that we have a valid \(p\)-variable \(p_{t}(\mathbf{x}^{t},c)\) for this hypothesis, i.e., a random variable - which may be also a function of the calibration data - that satisfies the inequality \(\Pr(p_{t}(\mathbf{x}^{t},c)\leq\alpha^{\prime}|\mathcal{H}_{t}(\mathbf{x}^{t},c))\leq \alpha^{\prime}\) for all \(\alpha^{\prime}\in[0,1]\), where the probability is conditioned over the hypothesis being correct. Then, constructing the predictive set as \(\Gamma(\mathbf{x}^{t})=\{c\in\mathcal{C}:p_{t}(\mathbf{x}^{t},c)>\alpha^{\prime}\}\) would guarantee the reliability condition \(\Pr\left(c\in\Gamma(\mathbf{x}^{t})\right)\geq 1-\alpha^{\prime}\) for the given fixed time \(t\).

The key underlying technical result in the theory of CP is that the variable

\[p_{t}(\mathbf{x}^{t},c)=\frac{1+\sum_{i=1}^{|\mathcal{D}^{\text{cal}}|}\mathbb{1} \left(s_{c}(\mathbf{x}^{t})\leq s_{c[i]}(\mathbf{x}^{t}[i])\right)}{|\mathcal{D}^{ \text{cal}}|+1}, \tag{32}\]

is a valid \(p\)-variable for time \(t\), where \(s_{c}(\mathbf{x}^{t})\) is an NC score. The predictive set constructed by \(p\)-value \(\Gamma(\mathbf{x}^{t})=\{c\in\mathcal{C}:p_{t}(\mathbf{x}^{t},c)>\alpha\}\) is equivalent to the expression of (16) since _excluding_ the \(\alpha\)-fraction (\(p_{t}(\mathbf{x}^{t},c)>\alpha\)) is equivalent to _including_ the \((1-\alpha)\)-fraction (\(s_{c}(\mathbf{x}^{t})\leq s_{\text{th}}^{t}\)).

In SpikeCP, the time \(t=T_{s}(\mathbf{x})\) at which a decision is made depends on the input \(\mathbf{x}\), and hence the reliability guarantees described above do not apply directly. What is needed, instead, are _corrected_\(p\)-variables \(\tilde{p}_{t}(\mathbf{x}^{t},c)\) satisfying the property \(\Pr\left(\tilde{p}_{t}(\mathbf{x}^{t},c)>\alpha^{\prime}\text{ for all }t\in\mathcal{T}_{s}|\mathcal{H}(\mathbf{x},c)\right)\geq 1- \alpha^{\prime}\) for all \(\alpha^{\prime}\in[0,1]\), where, under the composite null hypothesis \(\mathcal{H}(\mathbf{x},c)\), the pair \((\mathbf{x},c)\) and the calibration data \(\mathcal{D}^{\text{cal}}\) are i.i.d. Note that the hypothesis \(\mathcal{H}(\mathbf{x},c)\) implies all hypotheses \(\mathcal{H}_{t}(\mathbf{x}^{t},c)\) for \(t\in\mathcal{T}_{s}\).

To find such corrected \(p\)-variables, it is sufficient to identify a valid \(p\)-variable \(p(\mathbf{x},c)\) for the composite hypothesis \(\mathcal{H}(\mathbf{x},c)\) such that, with probability 1, we have the inequalities \(\tilde{p}_{t}(\mathbf{x}^{t},c)\geq p(\mathbf{x},c)\) for suitable functions \(\tilde{p}_{t}(\mathbf{x}^{t},c)\) of the original \(p\)-values \(p_{t}(\mathbf{x}^{t},c)\). Bonferroni's method provides one such \(p\)-variable, namely \(p^{\text{B}}(\mathbf{x},c)=\min_{t\in\mathcal{T}_{s}}\{|\mathcal{T}_{s}|p_{t}(\mathbf{ x}^{t},c)\}\) with corrected \(p\)-variables \(\tilde{p}_{t}(\mathbf{x}^{t},c)=|\mathcal{T}_{s}|p_{t}(\mathbf{x}^{t},c)\)[45, Appendix 2]. It can be checked that this selection yields the SpikeCP procedure in Algorithm 1.

Alternatively, the composite \(p\)-value produced by Simes correction is \(p^{\text{S}}(\mathbf{x},c)=\min_{t\in\mathcal{T}_{s}}\{|\mathcal{T}_{s}|p_{t}(\mathbf{ x}^{t},c)/r(t)\}\), where \(r(t)\) is the ranking of \(p_{t}(\mathbf{x}^{t},c)\) among \(\{p_{t}(\mathbf{x}^{t},c)\}_{t\in\mathcal{T}_{s}}\). This yields corrected \(p\)-variables \(\tilde{p}_{t}(\mathbf{x}^{t},c)=|\mathcal{T}_{s}|p_{t}(\mathbf{x}^{t},c)/r(t)\). Reference [9] proved that this approach provides a valid \(p\)-value as long as the joint distribution over the \(|\mathcal{T}_{s}|\)\(p\)-values \(p_{t}(\mathbf{x}^{t},c)\) have the _multivariate totally positive of order_\(2\) (MTP\({}_{2}\)) property as defined in [9]. Together with the assumption of increasing \(p\)-value, Simes corrected \(p\)-values yield the heuristic SpikeCP variant discussed in Sec. IV-D.

Fig. 10: Accuracy and normalized latency as a function of the number of training data \(|\mathcal{D}^{\text{tr}}|\), assuming \(p_{\text{targ}}=0.9\), and \(|\mathcal{D}^{\text{cal}}|=100\) under the same conditions as Fig. 4.