Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our contribution in this work is threefold:

The review comment to be evaluated is:

**Final recommendation**
I support accepting this paper. While I am a bit disappointed the authors did not add in the paper the results they discussed during the rebuttal, I think the paper is interesting and clearer than at submission.

**Summary**
stochastic subset selection (SSS) is a method to learn to compress a set $D$ by selecting a subset $D_s$ such that the loss of a task performed on $D_s$ is as close as possible to the loss if the task had been performed on the original $D$. The paper show how this general method can be applied to several tasks. For example, $D$ can contain the pixels of an image and the task can be to reconstruct the original image from the subset. This corresponds to learning to compress. As another example, with the same $D$ but a label prediction task, SSS learns to dynamically select sparse features for a classification task. The other tasks considered are dataset distillation (compressing a data set) and selecting prototypes for few-shot classification. $D_s$ is constructed in two steps. A candidate set is first constructed by considering elements of $D$ independently and in a second step elements of $D_c$ are included in $D_s$ iteratively by considering other elements in $D_c$ and in $D_s$. Experiments are performed for one example of each task previously discussed and SSS is shown to outperform baselines. SSS is motivated by the need to reduce bandwidth, computation and memory footprint of deep learning in edge devices.

---
**Strong points**

1) Code is provided (I did not try to run it).

2) The proposed approach is applied to four different tasks. This convinced me of its applicability.

3) Empirical experiments seem well conducted (with some caveat, see weak point 3) and results are good.

4) The problems tackled are interesting.

**Weak points**

1) The paper is very dense, which makes it hard to follow. Related to that, I think that some parts of the paper are too concise to understand them well. The supplementary material partly compensates that.

2) While the experiments show that the proposed approach works well, I think that some experiments should compare full data pipelines. As an example, one experiment shows that using SSS to dynamically select pixels of an image and using this sparse representation as input leads to good classification accuracy. The paper implicitely suggests that this reduces inference cost if classification is performed on the device and bandwidth if inference is performed on another device. However, the cost of the pixel selection by SSS is never analyzed. I think comparing the memory footprint, the number of operations, bandwidth used and accuracy for all steps of the following pipelines would give a much clearer picture of the interest of the approach for deep learning on mobile devices:
    - classification is done directly on the device, using an optimized network.
    - pixels are randomly selected and classification is performed on the device.
    - pixels are randomly selected, transmitted and classification is performed on another device.
    - pixels are selected by SSS, which involves computing multiple embeddings, and classification is performed on the device.
    - pixels are selected by SSS, transmitted and classification is performed on another device.

---
**Recommendation**
Overall, I vote for accepting. I am on the fence with this paper. The current results are good and I think the proposed approach has some potential and will be interesting for the community. On the other hand the paper is hard to read and the experiments, while good to show the strength of the approach, do not cover usefullness for the mobile use case. I think the former can be adressed in time and that the latter, while important, does not prevent the paper to be published.

---
**Details**

I like that each task is first formally described and then evaluated in a specific case with the experiments. However I only understood most tasks when reading the experiment sections. I would suggest to either provide a short illustration of an actual instance of the task when describing the task formally, to directly provide the experimental result after describing the task or to direct the reader to the relevant experimental section(s) for illustration.

The paper states that model descriptions are available in appendix A, but I could not find them.

For the *Set Classification/Prediction* task, I would suggest clarifying that for this task a set typically contains the features of a single example, so $D=x$. I only understood this when reading the experimental section.

The *Dataset Distillation: Instance Selection* task was not clear to me. Here are a few points that puzzled me.
- The text suggest the proposed approach is applied to each $D_i$, but figure 3(c) has no $D_i$ and $i$ seems to be used as an index of $x$ rather than $D$. Furthermore, in figure 3(c) a single D_s is constructed whereas I was expecting one per $D_i$.
- The paper states that *It is important to note that $c$ is computed using only $D_i$ for every element in $D$*. However $c$ also depends on *a query $\alpha$ which is computed using $d_i$*, which, if I understood correctly, is the element of $D$ and can be outside $D_i$. So it seems that $c$ is not computed using only $D_i$.

For the *Dataset Distillation: Classification* task, in figure 3d, the labels are not observed. How can the model learn?

1) For this task, if the label is observed, how is this different from the first task?

2) For the Image Reconstruction experiment, why is $x$ 2 dimensional? Are the inputs black and white images?

I think that the image reconstruction experiment should include a comparison to compression algorithms. I agree with the paper that the proposed approach is applicable to many other types of data than images, but it would be very interesting to see how the approach compares to expert defined methods. Furthermore, implementation in relation to mobile devices is the motivation of this work and as far as I know classical compression algorithms are still used on mobile devices.

---
**Questions**

Questions 1 and 2 in *details* above.

I would also be interested to hear authors' opinions on weak point 2 and a comparison to compression.

For classification, pixels not selected are set to 0. Wouldn't the inference cost be the same when using convolutional networks?

---
**Minor details**

For the *set reconstruction* task, would it also be interesting to learn a model on $D_s$ and use it to predict $Y$ from $X$?

*typos*
foro

The evidence to be evaluated is:Firstly, our model can be applied to any type of sets (and instances represented as sets), while the aforementioned models mainly work for images represented in tensor form. Furthermore, our method can be applied both at the instance and dataset level.

In this work, we consider data of the type \(D=\{d_{1},\ldots,d_{n}\}\) where individual \(d_{i}\)'s are possibly represented as _input_\(x_{i}\) and _target_\(y_{i}\). \(D\) is the complete set and we assume that within \(D\), there exists a subset \(D_{s}=\{s_{i},\ldots,s_{k}\}\subset D\) such that \(k\ll n\) and that for an arbitrarily defined loss function \(\ell(.,D)\) that we are interested in optimizing over the full set \(D\), \(D_{s}\) can be used as a proxy for \(D\) such that \(\ell(.,D)\approx\ell(.,D_{s})\). In what follows, we present a method that learns the conditional distribution \(p(D_{s}|D)\) of the subset \(D_{s}\) via a two stage selection procedure dubbed _candidate selection_ and _autoregressive subset selection_. The overall objective is then to minimize the loss function with respect to the subset \(D_{s}\), \(\mathbb{E}_{p(D,|D)}[\ell(.,D_{s})]\). When the set \(D\) itself follows a distribution of sets as in the _meta-learning_ framework, then the objective becomes \(\mathbb{E}_{D}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]]\). In essence, we seek to construct a subset \(D_{s}\) that is optimally representative of the full set \(D\) w.r.t \(\ell(.)\).

As such, we first present the _candidate selection_ procedure used to construct a smaller set, \(D_{c}\), without considering inter-sample dependencies. This is then followed by the _autoregressive subset selection_ procedure used to construct \(D_{s}\) from \(D_{c}\) by modeling inter-sample dependencies.

We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\). We implement both \(r(D)\) and \(\rho(d_{i},r(D))\) as neural networks and specifically for \(r(D)\), we use Deep Sets (Zaheer et al., 2017). Since _Ber_ is non-differentiable, we use the continuous relaxations of the Bernoulli distribution introduced in (Maddison et al., 2016; Jang et al., 2016; Gal et al., 2017). Specifically, to sample \(z_{i}\), we execute the following computational routine: \[z_{i}=\sigma\Big{(}\frac{1}{\tau}\Big{(}\log\frac{\pi_{i}}{1-\pi_{i}}+\log \frac{u}{1-u}\Big{)}\Big{)},\quad\pi_{i}=\rho(d_{i},r(D)),\quad u\sim\text{ Unif}(0,1), \tag{2}\] where \(\sigma\) is the Sigmoid function, \(\tau\) is the temperature for the continuous relaxation and \(u\) is sampled from the uniform distribution. \(\tau\) is set to \(0.05\) in all our experiments.

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

The key to avoiding samples with redundant information in \(D_{s}^{(k)}\) lies in the fact that for each element added to \(D_{s}\), it's selection is conditioned on both \(D_{c}\) and all elements in \(D_{s}^{(i-1)}\). We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

The inference complexity depends heavily on the choice of the function \(f\). If \(f\) considers the pairwise interactions between all candidate elements and the selected elements, the inference complexity is \(O(n)+O(k^{2}d/q)\) where \(n,d,k\) correspond to \(|D|,|D_{c}|\) and \(|D_{s}|\) respectively. In our experiments, for the choice of the function \(f\), we utilize either a Set Transformer(Lee et al., 2018) or DeepSets(Zaheer et al., 2017) to model the pairwise interactions between the elements of a given set.

We can also opt to train the network to predict a single target \(y_{D}\) for the set \(D\). For instance, the target could be the class of an image(classification) or the statistics of the set(regression problem). Here, \(p_{\theta}(y_{D}|D_{s})\) is a neural network that predicts the target \(y_{D}\). A set in this task, may be the features from a single example like an image and experimental results can be found in Section 4.2. The model for this task is depicted in Figure 2(b).

We describe a model capable of taking as input \(D_{i}\in\mathcal{D}\) to perform a task such as the reconstruction of all elements in the given dataset. For a single dataset \(D_{i}\in\mathcal{D}\), we apply the subset construction method already described to a \(D_{s}\) that can be used to reconstruct all the elements in \(D_{i}\). In essence, \(D_{i}\) is _distilled_ into a new dataset \(D_{s}\) with \(k<|D_{i}|\) elements.

As a first step, we represent \(D_{s}\) as a unique representative vector \(c\) for each element in the dataset akin to the statistics network used in the Neural Statistician (Edwards and Storkey, 2016) model. Specifically, to reconstruct an element \(d_{i}\in D_{i}\) given \(D_{s}\), \(c\) is computed by applying a stochastic cross-attention mechanism on \(D_{s}\) where the stochasticity is supplied by a query \(\alpha\) which is computed using \(d_{i}\). To obtain varying styles in the generated images, we additionally learn a latent variable \(w\) used to perturb \(c\) and both are combined to obtain a new element \(x\).

Additionally, to ensure that \(c\) is properly learnt, we add an informativity loss by reconstructing \(c\) from the generated samples from the given dataset. The objective for the model depicted in Figure 2(c) for a single dataset \(D\) is : \[\begin{split}\mathcal{L}(\theta,\phi,\psi)=\sum_{d_{i}\in D}[ \mathbb{E}_{q_{\phi}(w_{i}|d_{i})}[p_{\theta}(d_{i}|w_{i},c_{i})]-\text{KL}[q_ {\phi}(w_{i}|d_{i})||p_{\phi}(w)]\\ -\text{KL}[q_{\phi}(\alpha_{i}|d_{i})||p_{\phi}(\alpha)]-\text{ KL}[q_{\phi}(c_{i}|D_{s},\alpha_{i})||p_{\phi}(c)]\end{split} \tag{7}\] where \(p_{\psi}(\cdot)\) are priors on their respective latent variables and \(q_{\phi}(\cdot)\)'s are implemented with neural networks.

In addition to Equation 7 and the informativity loss, the model is optimized together with the subset selection model already described. When the model is fully optimized, it is applied to the instance selection task on the given dataset. In summary, the purpose of the generative model introduced is to train the subset selection module for the instance selection task.

Finally in the dataset distillation task, we consider the problem of selecting prototypes to be used for few-shot classification. Here, we adopt Prototypical Networks (Snell et al., 2017) and apply the subset selection model to the task of selecting representative prototypes from each class to be used for classifying new instances. By learning to select the prototypes, we can remove outliers that would otherwise change the class prediction boundaries in the classification task. The complete graphical model for this task is given in Figure 2(d) where again \(D_{s}\) corresponds to the selected prototypes and \(x_{*}\) and \(y_{*}\) correspond to query and class label respectively.

Our first experiment is on 1D function reconstruction. Suppose that we have a function \(f:[a,b]\rightarrow\mathbb{R}\). We first construct a set of data points of that function: \(D=\{(x_{1},y_{1}=f(x_{1})),(x_{2},y_{2}=f(x_{2})),\ldots,(x_{n},y_{n}=f(x_{n }))\}\) where \((x_{1},x_{2},\ldots,x_{n})\) are uniformly distributed along the x-axis within the interval \([a,b]\). Now if we have a family of functions \((f^{(1)},f^{(2)},\ldots,f^{(N)})\), this will lead to a family of sets \((D^{(1)},D^{(2)},\ldots,D^{(N)})\). We train our model which consists of the subset selection model \(p(D_{s}|D)\) and a task network \(p(Y|X,D_{s})\) (e.g. ANP), on this data set and report the reconstruction loss, which is the negative log-likelihood. SSS out-performs Random Select (RS), verifying that the subset selection model \(p(D_{s}|D)\) can learn a meaningful distribution for the selected elements. Our model also out-performs the Learning to Sample (LTS) baseline. Through the visualization of the selected points in Figure 5, we can see that out model tends to pick out more points (presented as red dots) in the drifting parts of the curve, which is reasonable since these parts are harder to reconstruct.

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

An ANP is then used to reconstruct the remaining pixels from a set of context elements (selected subset in our case). We conduct this experiment on the CelebA dataset (Liu et al., 2018). Figure 3(b) shows that our model significantly outperforms ANP with RS (as in the original ANP paper) and the LTS baseline. Figure 6 shows the reconstruction samples of our model which are visually better than the reconstruction of the baselines for the same number of pixels.

We again use the CelebA dataset, but this time the selected pixels are used to give predictions for 40 attributes of a celebrity's face (in a multi-task learning setting). For our proposed model, only the selected pixels are used for prediction (other pixels' values are set to zeros). Table 1 shows that using only 500 pixels (\(\sim\)1.3% of total pixels in an image), we can achieve a mean AUC of 0.9093 (99.3% of the accuracy obtained with the full image).

The AUC with selected pixels learned from our SSS is significantly higher than that of the random pixels baseline, showing the effectiveness of our subset selection method.

This is our stochastic subset selection model trained for reconstruction, but then later used for classification. Our model outperforms this variant, showing the effectiveness of training with the target task.

Since our method is stochastic, the predictive distribution can be written as \(\mathbb{E}_{p(D_{s}|D)}\left[p_{\theta}(y_{D}|D_{s})\right]\), and we can use Monte Carlo sampling to get the prediction in practice. However, throughout the experiment section, we only reported the result with one sampled subset, since it gives the best reduction in memory and computational cost. This can be seen as MC sampling with one sample. We compare it against another variant: SSS(MC) with MC sampling (5 samples). It should be noted that by doing MC sampling with 5 samples, the computational cost (inference) is increased by 5 times, and the memory requirement can be increased by up to 5 times too. Table 1 shows that our model achieves comparable performance with that variant, thus justifying that it can achieve good performance for target tasks, while reducing memory and computation requirement. **Instance Selection** We present results on the instance selection task applied to a whole dataset. In this task, we use the CelebA dataset since it has an imbalance both in terms of gender and race. A dataset is constructed by sampling 200 random images from the full dataset. In this experiment, we seek to select only a few(5-30) representative images from these generated datasets. On this task, our subset selection module is trained via the procedure detailed in Section 3.6 on instance selection. To evaluate the effectiveness of the SSS model, we evaluate the model in terms of the diversity in the selected subset using the Frechet Inception Distance(FID Score) (Heusel et al., 2017) which measures the similarity and diversity between two datasets. We compare our model with the model that randomly samples instances from the full dataset. Additionally, we compare our method with the Farthest Point Sampling(FPS) algorithm which selects \(k\) points from a given set by computing distances on a metric space between all elements and selecting those elements that are furthest from each other. FPS in general seeks to obtain a wide coverage over a given set and hence is a suitable baseline. The results of this experiment is presented in Table 4 where our selection method achieves a lower FID score compared to FPS and Random Sampling. Additionally, given that the dataset is highly imbalanced, FPS performs worst since by selecting the furthest elements in the given set it cannot capture the true representation of the whole dataset even when compared with Random Sampling. Also for small sample selection, our method outperforms FPS and Random Sampling significantly since our method is able to model the interactions within the full dataset and hence can select the most representative subset.

The results for this experiment is shown in Table 3, where it can be observed that SSS can learn to select more representative prototypes compared to the other methods especially in the few-shot problems where the choice of prototypes matters more.

In this paper, we have proposed a stochastic subset selection method to reduce the size of an arbitrary set while preserving performance on a target task. Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account. Hence, it can select a compact set that avoids samples with redundant information. By using the compact subset in place of the original set for a target task, we can save memory, communication and computational cost.

1:\(k\)(max subset size) 2:\(q\)(# elements selected at each iteration) 3:\(p(D)\) (distribution of sets) 4:\(\alpha\) (learning rate) 5: a target task with loss function \(\ell(\cdot,\cdot)\) 6:trained model with converged \(\theta\) and \(\phi\) 7:\(\theta,\phi\leftarrow\) initialization 8:while not converged do 9: Sample a minibatch with \(m\) sets \(D^{(1)},D^{(2)},\ldots,D^{(m)}\) from \(p(D)\)

Input: Figure 5: 1D Function Reconstruction.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input: Table 4: . FID Score for varying Instance Selection

Input: Figure 6: CelebA reconstruction samples with varying number of pixels.

Input: Figure 3: Graphical Models: (a) Feature Selection for Reconstruction. (b) Feature Selection for Prediction Task. (c) Instance Selection model. (d) Instance Selection model for Classification

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence, I find that while the authors have demonstrated the broad applicability of their SSS method across multiple tasks (reconstruction, classification, instance selection, few-shot learning), they don't explicitly state what their three key contributions are, making it impossible to evaluate a "threefold contribution" claim. The evidence shows strong experimental results and theoretical development of the method, but without the specific three contributions being enumerated, the claim itself is incomplete and cannot be properly assessed.

<score>1</score>

The claim appears to be a sentence fragment beginning with "Our contribution in this work is threefold:" but doesn't complete the thought. While the paper clearly has multiple valuable contributions based on the extensive evidence provided, I cannot evaluate an incomplete claim statement. The most appropriate score is 1 since there isn't actually an overstatement - rather there's an incomplete statement that needs to be completed before it can be properly assessed.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our contribution in this work is threefold:

The review comment to be evaluated is:

This work introduces a method to select instances from any set (stochastic subset selection, or SSS). The experiments demonstrate a diverse set of use-cases, including feature selection and core-set selection. The proposed approach is a two-stage method involving candidate selection (learning a function $\rho$ to determine a Bernoulli probability for each input) and AutoRegressive subset selection (learning a function $f$ to generate probabilities for sampling elements from a reduced set); both stages use the Concrete distribution to ensure differentiability.

The paper includes a broad set of experiments, but I have some questions and concerns regarding the baselines and the proposed SSS method.

1. Using the Concrete distribution enables differentiability, but Algorithm 1 seems to implement the candidate selection and AutoRegressive Subset selection steps in ways that eliminate differentiability. Line 4 assigns elements to $D_c$ based on $z_i$, but when is $z_i$ differentiated? Similarly, $Q$ is selected based on sampling using the $p_i$ parameters (line 13), but at what point are the samples differentiated after assigning elements to $Q$? The method apparently works so I'm sure there's a good answer, but it is not well explained.

2. Some aspects of SSS seem arbitrary, and the authors could make a more compelling case for this particular approach with a more serious ablation study. A couple ideas: (i) Running SSS just using the first stage (candidate selection). (ii) Running SSS using the second stage directly (if it's not too slow). (iii) Removing aspects of the models, such as removing $r(D)$ as an input to $\rho$, or $D_C$ as an input to $f$.

3. As a further baseline for SSS, could the authors simply have implemented an algorithm that learns separate Bernoulli probabilities for each example (without the model $\rho$)? This seems far simpler than SSS and it would be interesting to see how it compares.

4. When attempting to constrain the size of $D_C$, why use a KL divergence penalty and not simply penalize each $z_i$'s probability, or the sampled $z_i$'s themselves? The latter could be viewed as a version of an $\ell_0$ penalty on the subset size $|D_C|$, and both would remove the need to choose a hyperparameter for $r(Z)$.

5. Obtaining $q$ examples by independently sampling from Bernoulli distributions with probabilities given by $q \times p_j$ is odd. You can't guarantee that you get $q$ samples, and you can't even guarantee that $q \times p_j$ is a valid probability. There must be a better solution to this problem; [1, 2] both use clever sampling tricks with the Concrete distribution, perhaps they suggest a better way.

6. The paper is missing a couple citations and comparisons when it comes to feature selection using the Concrete distribution. Concrete Autoencoders (CAEs) [1] are a method for performing differentiable feature selection using a similar trick with the Concrete distribution; while CAEs are used for global feature selection (the same features for every instance), Learning to Explain (L2X) [2] is a method for instance-wise feature selection that also uses the Concrete distribution. Both should be mentioned, and L2X could be used a comparison in certain experiments.

The motivation around reducing communication overhead and allowing for faster training is interesting, but these claims could be more effectively grounded in the experiments performed in the paper. When accounting for the overhead involved in running SSS, is this method faster than conventional training, and does it require less communication overhead? The clearest application from these experiments is feature selection, but the comparisons with baselines are not extensive.

Overall, the method seems promising but I hope the authors can clarify several points and offer more competitive baselines.

[1] Abid et al. "Concrete Autoencoders for Differentiable Feature Selection and Reconstruction" (2019)

[2] Chen et al. "Learning to Explain: An Information-Theoretic Perspective on Model Interpretation" (2018)

The evidence to be evaluated is:Firstly, our model can be applied to any type of sets (and instances represented as sets), while the aforementioned models mainly work for images represented in tensor form. Furthermore, our method can be applied both at the instance and dataset level.

In this work, we consider data of the type \(D=\{d_{1},\ldots,d_{n}\}\) where individual \(d_{i}\)'s are possibly represented as _input_\(x_{i}\) and _target_\(y_{i}\). \(D\) is the complete set and we assume that within \(D\), there exists a subset \(D_{s}=\{s_{i},\ldots,s_{k}\}\subset D\) such that \(k\ll n\) and that for an arbitrarily defined loss function \(\ell(.,D)\) that we are interested in optimizing over the full set \(D\), \(D_{s}\) can be used as a proxy for \(D\) such that \(\ell(.,D)\approx\ell(.,D_{s})\). In what follows, we present a method that learns the conditional distribution \(p(D_{s}|D)\) of the subset \(D_{s}\) via a two stage selection procedure dubbed _candidate selection_ and _autoregressive subset selection_. The overall objective is then to minimize the loss function with respect to the subset \(D_{s}\), \(\mathbb{E}_{p(D,|D)}[\ell(.,D_{s})]\). When the set \(D\) itself follows a distribution of sets as in the _meta-learning_ framework, then the objective becomes \(\mathbb{E}_{D}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]]\). In essence, we seek to construct a subset \(D_{s}\) that is optimally representative of the full set \(D\) w.r.t \(\ell(.)\).

As such, we first present the _candidate selection_ procedure used to construct a smaller set, \(D_{c}\), without considering inter-sample dependencies. This is then followed by the _autoregressive subset selection_ procedure used to construct \(D_{s}\) from \(D_{c}\) by modeling inter-sample dependencies.

We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\). We implement both \(r(D)\) and \(\rho(d_{i},r(D))\) as neural networks and specifically for \(r(D)\), we use Deep Sets (Zaheer et al., 2017). Since _Ber_ is non-differentiable, we use the continuous relaxations of the Bernoulli distribution introduced in (Maddison et al., 2016; Jang et al., 2016; Gal et al., 2017). Specifically, to sample \(z_{i}\), we execute the following computational routine: \[z_{i}=\sigma\Big{(}\frac{1}{\tau}\Big{(}\log\frac{\pi_{i}}{1-\pi_{i}}+\log \frac{u}{1-u}\Big{)}\Big{)},\quad\pi_{i}=\rho(d_{i},r(D)),\quad u\sim\text{ Unif}(0,1), \tag{2}\] where \(\sigma\) is the Sigmoid function, \(\tau\) is the temperature for the continuous relaxation and \(u\) is sampled from the uniform distribution. \(\tau\) is set to \(0.05\) in all our experiments.

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

The key to avoiding samples with redundant information in \(D_{s}^{(k)}\) lies in the fact that for each element added to \(D_{s}\), it's selection is conditioned on both \(D_{c}\) and all elements in \(D_{s}^{(i-1)}\). We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

The inference complexity depends heavily on the choice of the function \(f\). If \(f\) considers the pairwise interactions between all candidate elements and the selected elements, the inference complexity is \(O(n)+O(k^{2}d/q)\) where \(n,d,k\) correspond to \(|D|,|D_{c}|\) and \(|D_{s}|\) respectively. In our experiments, for the choice of the function \(f\), we utilize either a Set Transformer(Lee et al., 2018) or DeepSets(Zaheer et al., 2017) to model the pairwise interactions between the elements of a given set.

We can also opt to train the network to predict a single target \(y_{D}\) for the set \(D\). For instance, the target could be the class of an image(classification) or the statistics of the set(regression problem). Here, \(p_{\theta}(y_{D}|D_{s})\) is a neural network that predicts the target \(y_{D}\). A set in this task, may be the features from a single example like an image and experimental results can be found in Section 4.2. The model for this task is depicted in Figure 2(b).

We describe a model capable of taking as input \(D_{i}\in\mathcal{D}\) to perform a task such as the reconstruction of all elements in the given dataset. For a single dataset \(D_{i}\in\mathcal{D}\), we apply the subset construction method already described to a \(D_{s}\) that can be used to reconstruct all the elements in \(D_{i}\). In essence, \(D_{i}\) is _distilled_ into a new dataset \(D_{s}\) with \(k<|D_{i}|\) elements.

As a first step, we represent \(D_{s}\) as a unique representative vector \(c\) for each element in the dataset akin to the statistics network used in the Neural Statistician (Edwards and Storkey, 2016) model. Specifically, to reconstruct an element \(d_{i}\in D_{i}\) given \(D_{s}\), \(c\) is computed by applying a stochastic cross-attention mechanism on \(D_{s}\) where the stochasticity is supplied by a query \(\alpha\) which is computed using \(d_{i}\). To obtain varying styles in the generated images, we additionally learn a latent variable \(w\) used to perturb \(c\) and both are combined to obtain a new element \(x\).

Additionally, to ensure that \(c\) is properly learnt, we add an informativity loss by reconstructing \(c\) from the generated samples from the given dataset. The objective for the model depicted in Figure 2(c) for a single dataset \(D\) is : \[\begin{split}\mathcal{L}(\theta,\phi,\psi)=\sum_{d_{i}\in D}[ \mathbb{E}_{q_{\phi}(w_{i}|d_{i})}[p_{\theta}(d_{i}|w_{i},c_{i})]-\text{KL}[q_ {\phi}(w_{i}|d_{i})||p_{\phi}(w)]\\ -\text{KL}[q_{\phi}(\alpha_{i}|d_{i})||p_{\phi}(\alpha)]-\text{ KL}[q_{\phi}(c_{i}|D_{s},\alpha_{i})||p_{\phi}(c)]\end{split} \tag{7}\] where \(p_{\psi}(\cdot)\) are priors on their respective latent variables and \(q_{\phi}(\cdot)\)'s are implemented with neural networks.

In addition to Equation 7 and the informativity loss, the model is optimized together with the subset selection model already described. When the model is fully optimized, it is applied to the instance selection task on the given dataset. In summary, the purpose of the generative model introduced is to train the subset selection module for the instance selection task.

Finally in the dataset distillation task, we consider the problem of selecting prototypes to be used for few-shot classification. Here, we adopt Prototypical Networks (Snell et al., 2017) and apply the subset selection model to the task of selecting representative prototypes from each class to be used for classifying new instances. By learning to select the prototypes, we can remove outliers that would otherwise change the class prediction boundaries in the classification task. The complete graphical model for this task is given in Figure 2(d) where again \(D_{s}\) corresponds to the selected prototypes and \(x_{*}\) and \(y_{*}\) correspond to query and class label respectively.

Our first experiment is on 1D function reconstruction. Suppose that we have a function \(f:[a,b]\rightarrow\mathbb{R}\). We first construct a set of data points of that function: \(D=\{(x_{1},y_{1}=f(x_{1})),(x_{2},y_{2}=f(x_{2})),\ldots,(x_{n},y_{n}=f(x_{n }))\}\) where \((x_{1},x_{2},\ldots,x_{n})\) are uniformly distributed along the x-axis within the interval \([a,b]\). Now if we have a family of functions \((f^{(1)},f^{(2)},\ldots,f^{(N)})\), this will lead to a family of sets \((D^{(1)},D^{(2)},\ldots,D^{(N)})\). We train our model which consists of the subset selection model \(p(D_{s}|D)\) and a task network \(p(Y|X,D_{s})\) (e.g. ANP), on this data set and report the reconstruction loss, which is the negative log-likelihood. SSS out-performs Random Select (RS), verifying that the subset selection model \(p(D_{s}|D)\) can learn a meaningful distribution for the selected elements. Our model also out-performs the Learning to Sample (LTS) baseline. Through the visualization of the selected points in Figure 5, we can see that out model tends to pick out more points (presented as red dots) in the drifting parts of the curve, which is reasonable since these parts are harder to reconstruct.

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

An ANP is then used to reconstruct the remaining pixels from a set of context elements (selected subset in our case). We conduct this experiment on the CelebA dataset (Liu et al., 2018). Figure 3(b) shows that our model significantly outperforms ANP with RS (as in the original ANP paper) and the LTS baseline. Figure 6 shows the reconstruction samples of our model which are visually better than the reconstruction of the baselines for the same number of pixels.

We again use the CelebA dataset, but this time the selected pixels are used to give predictions for 40 attributes of a celebrity's face (in a multi-task learning setting). For our proposed model, only the selected pixels are used for prediction (other pixels' values are set to zeros). Table 1 shows that using only 500 pixels (\(\sim\)1.3% of total pixels in an image), we can achieve a mean AUC of 0.9093 (99.3% of the accuracy obtained with the full image).

The AUC with selected pixels learned from our SSS is significantly higher than that of the random pixels baseline, showing the effectiveness of our subset selection method.

This is our stochastic subset selection model trained for reconstruction, but then later used for classification. Our model outperforms this variant, showing the effectiveness of training with the target task.

Since our method is stochastic, the predictive distribution can be written as \(\mathbb{E}_{p(D_{s}|D)}\left[p_{\theta}(y_{D}|D_{s})\right]\), and we can use Monte Carlo sampling to get the prediction in practice. However, throughout the experiment section, we only reported the result with one sampled subset, since it gives the best reduction in memory and computational cost. This can be seen as MC sampling with one sample. We compare it against another variant: SSS(MC) with MC sampling (5 samples). It should be noted that by doing MC sampling with 5 samples, the computational cost (inference) is increased by 5 times, and the memory requirement can be increased by up to 5 times too. Table 1 shows that our model achieves comparable performance with that variant, thus justifying that it can achieve good performance for target tasks, while reducing memory and computation requirement. **Instance Selection** We present results on the instance selection task applied to a whole dataset. In this task, we use the CelebA dataset since it has an imbalance both in terms of gender and race. A dataset is constructed by sampling 200 random images from the full dataset. In this experiment, we seek to select only a few(5-30) representative images from these generated datasets. On this task, our subset selection module is trained via the procedure detailed in Section 3.6 on instance selection. To evaluate the effectiveness of the SSS model, we evaluate the model in terms of the diversity in the selected subset using the Frechet Inception Distance(FID Score) (Heusel et al., 2017) which measures the similarity and diversity between two datasets. We compare our model with the model that randomly samples instances from the full dataset. Additionally, we compare our method with the Farthest Point Sampling(FPS) algorithm which selects \(k\) points from a given set by computing distances on a metric space between all elements and selecting those elements that are furthest from each other. FPS in general seeks to obtain a wide coverage over a given set and hence is a suitable baseline. The results of this experiment is presented in Table 4 where our selection method achieves a lower FID score compared to FPS and Random Sampling. Additionally, given that the dataset is highly imbalanced, FPS performs worst since by selecting the furthest elements in the given set it cannot capture the true representation of the whole dataset even when compared with Random Sampling. Also for small sample selection, our method outperforms FPS and Random Sampling significantly since our method is able to model the interactions within the full dataset and hence can select the most representative subset.

The results for this experiment is shown in Table 3, where it can be observed that SSS can learn to select more representative prototypes compared to the other methods especially in the few-shot problems where the choice of prototypes matters more.

In this paper, we have proposed a stochastic subset selection method to reduce the size of an arbitrary set while preserving performance on a target task. Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account. Hence, it can select a compact set that avoids samples with redundant information. By using the compact subset in place of the original set for a target task, we can save memory, communication and computational cost.

1:\(k\)(max subset size) 2:\(q\)(# elements selected at each iteration) 3:\(p(D)\) (distribution of sets) 4:\(\alpha\) (learning rate) 5: a target task with loss function \(\ell(\cdot,\cdot)\) 6:trained model with converged \(\theta\) and \(\phi\) 7:\(\theta,\phi\leftarrow\) initialization 8:while not converged do 9: Sample a minibatch with \(m\) sets \(D^{(1)},D^{(2)},\ldots,D^{(m)}\) from \(p(D)\)

Input: Figure 5: 1D Function Reconstruction.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input: Table 4: . FID Score for varying Instance Selection

Input: Figure 6: CelebA reconstruction samples with varying number of pixels.

Input: Figure 3: Graphical Models: (a) Feature Selection for Reconstruction. (b) Feature Selection for Prediction Task. (c) Instance Selection model. (d) Instance Selection model for Classification

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence and reviewer comments, this evaluation focuses on the claim "Our contribution in this work is threefold." Let me assess whether this specific statement is well-stated or overstated.

The evidence shows that the work does introduce novel components: (1) A general subset selection method applicable to any type of sets, (2) A two-stage selection process using candidate selection and autoregressive subset selection, and (3) Multiple demonstrated applications across feature selection, instance selection, and dataset distillation tasks.

However, while the reviewer acknowledges these contributions, they raise important questions about the novelty and uniqueness of some aspects, particularly noting missing comparisons to similar methods like Concrete Autoencoders and L2X that use similar techniques. Some technical choices are described as "arbitrary" without sufficient justification.

<score>2</score>

The claim leans toward being well-stated, as the evidence demonstrates three distinct contributions. However, it could be more precise in acknowledging related work and better justifying technical decisions. The extensive experimental results and mathematical formalization provide reasonable support for the claimed contributions.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our first contribution is to benchmark eight recent novelty detection methods in the near-ND setting, which consists of ND problems whose normal and anomaly classes are either naturally semantically close, or else synthetically forced to be close.

The review comment to be evaluated is:

The paper investigates the near anomalies detection and points a significant decrease in performance of state of the art approaches on near anomalies cases. A new solution is proposed: 1) using SDE models trained to generate data from the training distribution. 2) generate a dataset from the trained SDE. 3) train a binary classifier on top of a feature extractor to distinguish between the generated data and real data. 4) use nearest neighbour for detecting anomalies.
Extensive experiments and empirical analysis are carried on. The method shows good results.

The evidence to be evaluated is:In order to generate near anomalous samples, a basic diffusion model, called Stochastic Differential Equation (SDE) (43) is adopted. In the diffusion models, the gradient of the logarithmic probability density of the data with respect to the input is called the score. A SDE model considers a continuum of distributions that develop over time in accordance with a diffusion process, to corrupt a sample and turn it into pure noise.

The model is trained to learn how to gradually transform the noisy image back to its original form through the score fucntion (backward process).

\[x_{n-1}-x_{n}=[f(x,t)-g^{2}(t)s_{\theta}(x,t)]dt+g(t)d\hat{w},\quad x_{T}\sim N (0,1), \tag{1}\] where \(x_{n}\) is the data sample at time \(n\), \(f=0\) and \(g=\sigma^{t}\) are called the drift and diffusion coefficients, respectively. \(s_{\theta}(x,t)\) is called the score function, as is expected to estimate \(\nabla_{x}\log p_{t}(x)\).

In this section, we compare our method against the SOTA in the ND and near-ND setups. All the experiments are performed 10 times and the average result is reported.

We compare our approach with current top self-supervised and pre-trained feature adaptation methods in the literature (44; 34; 33; 7; 16).

Our experiments are organized into three categories and are presented separately in sub-tables.

To evaluate the performance on near-ND benchmark, we use 4 different datasets that naturally show slight differences between their class distributions. These datasets are not commonly used in ND but used in fine-grained image classification or segmentation, containing hard-to-distinguish object classes. Thus they are suitable for benchmarking near-ND. The datasets that are tested in this setting include FGVC-Aircraft (25), Stanford-Cars (22), White Blood Cell (WBC) (49), and Weather (10). Table 1(b) provides the results in this category, which again shows that our method outperforms SOTA by roughly 8 percentage points of AUROC in the mentioned near-ND setup.

In this setup, each class of CIFAR-10 (\(C_{i}^{10}\)) and its closest class in CIFAR-100 (\(C_{j}^{100}\)) are selected, to represent normal and abnormal classes, respectively, where \(i\) and \(j\) are class numbers that are less than 10 and 100, respectively. Then, the model is trained on the training samples of (\(C_{i}^{10}\)) and tested against the aggregation of (\(C_{j}^{100}\)) and (\(C_{i}^{10}\)) test sets.

While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting. To be specific, most of the methods significantly fail to detect generated fake samples, which is particularly the case for PANDA (34) with 40 percentage points performance drop. Surprisingly, DeepSAD with roughly 30 percentage points performance decrease does not work decently despite being exposed to real outliers during the training process.

All of these results highlight the need of benchmarking previous ND methods in the near-ND setup.

The results are provided for both the ND and near-ND evaluation setups.

The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting. Moreover, ViT-B_16 and R50+ViT-B_16 perform roughly 8% and 16% better after the fine-tuning in the ND setup, showing the _generality_ of our approach regardless of the setting or backbone.

Evidently, employing our fake samples generated by the SDE is almost always beneficial as opposed the ones generated by the other models. Interestingly, other models generated samples are harmful to the performance in Weather and MvTecAD datasets. Having artifacts, mode-collapse, and undiversified generated samples could be the reasons behind this observation, which are shown in Appendix10. This highlights that not all the generative models are beneficial for the ND and near-ND tasks, and they need to be carefully selected based on the constraints defined in such domains.

Utilizing the fake data for the novelty detection task has previously been considered. The general idea is to employ synthetic images, which may be generated by GANs, to augment the training set (6; 26; 50; 29; 11; 1). In the case of open-set recognition, (21) proposed OpenGAN, which adverserially generates fake open-set images. The discriminator is then utilized at the test time for the novelty detection. The key point in OpenGAN is that a tiny additional dataset, containing both in- and outliers, is used as a validation set for the model selection. This, also known as outlier exposure, is necessary due to the GAN unstable training. Even though these models can work well on simple datasets, they cannot handle more complex datasets and cannot detect hard near-distribution anomalies.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting.

To provide a standard benchmark for the near-ND task there is a need to define a one-class distribution closeness score. Suppose a \(K\)-class training dataset is given from which a normal class \(C_{i}\) is randomly sampled and used to train the model \(\mathcal{M}\). The closest abnormal distribution with respect to the selected normal class for \(\mathcal{M}\) can be defined as \(C_{j}\neq C_{i}\) that minimizes the test time performance when it is considered as the abnormal distribution. We call the performance in the mentioned scenario the bottom-1 score of class \(C_{i}\) with the backbone \(\mathcal{M}\). However, this score depends on the choice of \(\mathcal{M}\), making it specific for every problem setup and method. Therefore, inspired by the CLP criterion introduced in (47) as a measurement of dataset distance, we could use CLP as an alternative closeness score for the novelty detection task. Given a \(K\)-class training dataset, one category is randomly selected as the normal distribution. Then, a supervised classifier \(\mathcal{P}\) is trained on the rest \(K-1\) abnormal categories.

The closeness score of each abnormal class \(i\) with respect to the normal class is obtained as follows: \[\text{Closeness score}_{i}=\sum_{x\in\text{Normal Class}}\mathcal{P}(\hat{y}=i|x). \tag{8}\] The higher the closeness score of an abnormal category, the more similar it is to the normal class.

In section 8, we proposed the closeness score (CLP) to find the closest abnormal class for each class of the normal dataset. In our experiments, we used the ViT-B_16 (pretrained on ImageNet-21K) as the backbone used for extracting the closest abnormal classes based on this criterion. The bottom-1 abnormal class is the class that has the lowest novelty detection performance during the testing phase. Note that the abnormal classes chosen based on the CLP criterion do not necessarily have the worst novelty detection performance; therefore, it is expected for the novelty detection methods to perform better on the CLP criterion. This section aims to investigate how well these two criteria match each other. In the Table 5, for each novelty detection model and every class of the CIFAR-10 dataset, the bottom-1 class is shown. Furthermore, the last row indicates the closest abnormal class selected based on the CLP criterion. As shown in the Table 5, the extracted classes based on the CLP criterion are the same or conceptually similar to the respected bottom-1 classes. Figure 2 also shows a decent correlation between the bottom-1 score and the CLP score, which means both these criteria could be used to extract near-distribution abnormal classes, and that the CLP could be used as a proxy to the bottom-1 criterion. Also, The Pearson correlation coefficient between these two results is 0.843, indicating a strong positive relationship.

Because in our setup, only one class (the normal class) is available for the training of the generative model. As the sample size becomes limited in such a setting, model convergence and generation of high quality real samples become a challenge. Even under such circumstances, the SDE-based models converge. Furthermore, such models can eventually generate high quality real images. However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images. This makes it a better choice for the near-ND setup.

As can be seen, the images are clearly different semantically from the normal class. Despite this semantic difference, most novelty detection models have a poor performance in the detection of these images as anomalous.

To further extend our results, we compared the methods on a number of small datasets from different domains: 102 Category Flowers and Caltech-UCSD Birds 200. For each of these datasets, we evaluated the methods using only each of the first 20 classes as normal, and using the entire test set for the evaluation. For FGVC-Aircraft, due to the extreme similarity of the classes to each other, we randomly selected a subset of ten classes from the entire dataset, such that no two classes have the same Manufacturer.

MvTecAD: This dataset contains 15 different industrial products, with normal images of the proper products for training and 1 to 9 types of manufacturing errors as anomalies. The anomalies in MvTecAD are in-class, i.e. the anomalous images come from the same class of the normal images with subtle variations. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 2: The performance of novelty detection methods (AUROC %) in the near-ND and ND settings. In the near-ND setting, results are reported according to both criteria, i.e. CLP and bottom-1.

Input: Figure 2: Overview of our framework for near distribution novelty detection. In the first step, the SDE diffusion model generates semantically close outliers, after its training is early-stopped. As it is shown, they have very subtle yet semantic differences with the normal data distribution (downleft). After that, using a linear layer, a pre-trained feature extractor is fine-tuned to solve a binary classification between the normal and abnormal inputs. This modifies the normal boundaries toward including more distinctive features. Finally, normal embeddings are stored and used to compute the k-NN distance at the test time as the novelty score.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 2: The performance of novelty detection methods (AUROC %) in the near-ND and ND settings. In the near-ND setting, results are reported according to both criteria, i.e. CLP and bottom-1.

Input: Figure 2: Overview of our framework for near distribution novelty detection. In the first step, the SDE diffusion model generates semantically close outliers, after its training is early-stopped. As it is shown, they have very subtle yet semantic differences with the normal data distribution (downleft). After that, using a linear layer, a pre-trained feature extractor is fine-tuned to solve a binary classification between the normal and abnormal inputs. This modifies the normal boundaries toward including more distinctive features. Finally, normal embeddings are stored and used to compute the k-NN distance at the test time as the novelty score.

Input: Table 5: For each model and every class of the CIFAR-10, the anomalous class that minimizes the anomaly detection performance among 100 classes of the CIFAR-100 dataset is reported. The last row is the CIFAR-100 class that achieves the highest CLP score. Note that the latter does not depend on the anomaly detection method.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 4: For each class of CIFAR-10, nearest class from CIFAR-100 has been provided, according to the closeness score. In each 3-row panel, the first row is assumed as normal, the second one is assumed the nearest abnormal class, and the third row indicates the corresponding generated fake images using the SDE model.

Input: Figure 5: For each class of CIFAR-10, nearest class from CIFAR-100 has been provided, according to the closeness score. In each 3-row panel, the first row is assumed as normal, the second one is assumed the nearest abnormal class, and the third row indicates the corresponding generated fake images using the SDE model.

Input: Figure 3: Left: Unlike GAN-based models, the diffusion SDE model performs much better in consistently improving the fidelity and diversity of the generated samples along the training. The metrics “density” and “coverage” (27) are used to reliably measure these concepts. Thus, obtaining high-quality and diverse anomalous samples is feasible by early stopping in the SDE model. Right: Some generated samples by various methods in different datasets as well as their corresponding normal samples are shown. As can be seen SDE samples, in the third row, are authentic (differ from the normal class), high-quality, and have minimal differences to the normal class, while other methods produce low quality samples that are far away from the normal distribution.

Input: Figure 3: By increasing i, the AUROCs of the models improve due to the presence of more anomalies that are further from the boundary. As expected, in most cases, the gap between the performance of models has become smaller. Furthermore, in the case of i = 1, in which anomalies constitute the near distribution ones, the models’ performances vary greatly. In this case, our proposed method achieves SOTA results by a large margin. 16

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, the claim about benchmarking eight novelty detection methods in near-ND settings appears well-supported. The paper presents comprehensive experimental results across multiple datasets (FGVC-Aircraft, Stanford-Cars, WBC, Weather) showing performance comparisons between different methods. Tables 1(a-c) clearly demonstrate the benchmarking results, with detailed AUROC scores. The evidence includes mathematical formulations, extensive visualizations, and systematic comparisons. The claim is modest and accurately reflects what was accomplished without overstating the findings. The review comment also confirms the thoroughness of the benchmarking efforts.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our first contribution is to benchmark eight recent novelty detection methods in the near-ND setting, which consists of ND problems whose normal and anomaly classes are either naturally semantically close, or else synthetically forced to be close.

The review comment to be evaluated is:

* There is a large room for improvement in clarity.
    * The paper should be clearer on how exactly the generative model is trained and used. The most important fact, in my opinion, is that the score-based generative model is trained on in-distribution data and terminated prematurely. However, the fact is only briefly mentioned in Page 2, and then mentioned in Figure 2. Given the importance of this information, this point should be described in detail in Section 2.1. Also, other details and design choices, for example, how early the training of the score-based model should be stopped, needs to be elaborated.
    * The experimental setup described in Section 3.2 is difficult to follow. There are several comparison settings but are not well organized.
* Discussion on the differences among diffusion models, GANs, and VAEs in Section 2.2 are not very persuasive. It does not have concrete examples or quantitative arguments. Plus, the argument of Section 3.2 is mostly about generative modeling itself and not about its application to generating auxiliary data in novelty detection.
* When performing CIFAR-10 vs CIFAR-100 experiment, it is more standard and more challenging to incorporate all the classes, instead of using only one class from each dataset. Overall, I am not convinced that the experiment settings used in the paper are standard.
* Replacing a GAN with a diffusion model in the data generator for novelty detection itself is not a very novel contribution to the community.

The evidence to be evaluated is:In order to generate near anomalous samples, a basic diffusion model, called Stochastic Differential Equation (SDE) (43) is adopted. In the diffusion models, the gradient of the logarithmic probability density of the data with respect to the input is called the score. A SDE model considers a continuum of distributions that develop over time in accordance with a diffusion process, to corrupt a sample and turn it into pure noise.

The model is trained to learn how to gradually transform the noisy image back to its original form through the score fucntion (backward process).

\[x_{n-1}-x_{n}=[f(x,t)-g^{2}(t)s_{\theta}(x,t)]dt+g(t)d\hat{w},\quad x_{T}\sim N (0,1), \tag{1}\] where \(x_{n}\) is the data sample at time \(n\), \(f=0\) and \(g=\sigma^{t}\) are called the drift and diffusion coefficients, respectively. \(s_{\theta}(x,t)\) is called the score function, as is expected to estimate \(\nabla_{x}\log p_{t}(x)\).

In this section, we compare our method against the SOTA in the ND and near-ND setups. All the experiments are performed 10 times and the average result is reported.

We compare our approach with current top self-supervised and pre-trained feature adaptation methods in the literature (44; 34; 33; 7; 16).

Our experiments are organized into three categories and are presented separately in sub-tables.

To evaluate the performance on near-ND benchmark, we use 4 different datasets that naturally show slight differences between their class distributions. These datasets are not commonly used in ND but used in fine-grained image classification or segmentation, containing hard-to-distinguish object classes. Thus they are suitable for benchmarking near-ND. The datasets that are tested in this setting include FGVC-Aircraft (25), Stanford-Cars (22), White Blood Cell (WBC) (49), and Weather (10). Table 1(b) provides the results in this category, which again shows that our method outperforms SOTA by roughly 8 percentage points of AUROC in the mentioned near-ND setup.

In this setup, each class of CIFAR-10 (\(C_{i}^{10}\)) and its closest class in CIFAR-100 (\(C_{j}^{100}\)) are selected, to represent normal and abnormal classes, respectively, where \(i\) and \(j\) are class numbers that are less than 10 and 100, respectively. Then, the model is trained on the training samples of (\(C_{i}^{10}\)) and tested against the aggregation of (\(C_{j}^{100}\)) and (\(C_{i}^{10}\)) test sets.

While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting. To be specific, most of the methods significantly fail to detect generated fake samples, which is particularly the case for PANDA (34) with 40 percentage points performance drop. Surprisingly, DeepSAD with roughly 30 percentage points performance decrease does not work decently despite being exposed to real outliers during the training process.

All of these results highlight the need of benchmarking previous ND methods in the near-ND setup.

The results are provided for both the ND and near-ND evaluation setups.

The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting. Moreover, ViT-B_16 and R50+ViT-B_16 perform roughly 8% and 16% better after the fine-tuning in the ND setup, showing the _generality_ of our approach regardless of the setting or backbone.

Evidently, employing our fake samples generated by the SDE is almost always beneficial as opposed the ones generated by the other models. Interestingly, other models generated samples are harmful to the performance in Weather and MvTecAD datasets. Having artifacts, mode-collapse, and undiversified generated samples could be the reasons behind this observation, which are shown in Appendix10. This highlights that not all the generative models are beneficial for the ND and near-ND tasks, and they need to be carefully selected based on the constraints defined in such domains.

Utilizing the fake data for the novelty detection task has previously been considered. The general idea is to employ synthetic images, which may be generated by GANs, to augment the training set (6; 26; 50; 29; 11; 1). In the case of open-set recognition, (21) proposed OpenGAN, which adverserially generates fake open-set images. The discriminator is then utilized at the test time for the novelty detection. The key point in OpenGAN is that a tiny additional dataset, containing both in- and outliers, is used as a validation set for the model selection. This, also known as outlier exposure, is necessary due to the GAN unstable training. Even though these models can work well on simple datasets, they cannot handle more complex datasets and cannot detect hard near-distribution anomalies.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting.

To provide a standard benchmark for the near-ND task there is a need to define a one-class distribution closeness score. Suppose a \(K\)-class training dataset is given from which a normal class \(C_{i}\) is randomly sampled and used to train the model \(\mathcal{M}\). The closest abnormal distribution with respect to the selected normal class for \(\mathcal{M}\) can be defined as \(C_{j}\neq C_{i}\) that minimizes the test time performance when it is considered as the abnormal distribution. We call the performance in the mentioned scenario the bottom-1 score of class \(C_{i}\) with the backbone \(\mathcal{M}\). However, this score depends on the choice of \(\mathcal{M}\), making it specific for every problem setup and method. Therefore, inspired by the CLP criterion introduced in (47) as a measurement of dataset distance, we could use CLP as an alternative closeness score for the novelty detection task. Given a \(K\)-class training dataset, one category is randomly selected as the normal distribution. Then, a supervised classifier \(\mathcal{P}\) is trained on the rest \(K-1\) abnormal categories.

The closeness score of each abnormal class \(i\) with respect to the normal class is obtained as follows: \[\text{Closeness score}_{i}=\sum_{x\in\text{Normal Class}}\mathcal{P}(\hat{y}=i|x). \tag{8}\] The higher the closeness score of an abnormal category, the more similar it is to the normal class.

In section 8, we proposed the closeness score (CLP) to find the closest abnormal class for each class of the normal dataset. In our experiments, we used the ViT-B_16 (pretrained on ImageNet-21K) as the backbone used for extracting the closest abnormal classes based on this criterion. The bottom-1 abnormal class is the class that has the lowest novelty detection performance during the testing phase. Note that the abnormal classes chosen based on the CLP criterion do not necessarily have the worst novelty detection performance; therefore, it is expected for the novelty detection methods to perform better on the CLP criterion. This section aims to investigate how well these two criteria match each other. In the Table 5, for each novelty detection model and every class of the CIFAR-10 dataset, the bottom-1 class is shown. Furthermore, the last row indicates the closest abnormal class selected based on the CLP criterion. As shown in the Table 5, the extracted classes based on the CLP criterion are the same or conceptually similar to the respected bottom-1 classes. Figure 2 also shows a decent correlation between the bottom-1 score and the CLP score, which means both these criteria could be used to extract near-distribution abnormal classes, and that the CLP could be used as a proxy to the bottom-1 criterion. Also, The Pearson correlation coefficient between these two results is 0.843, indicating a strong positive relationship.

Because in our setup, only one class (the normal class) is available for the training of the generative model. As the sample size becomes limited in such a setting, model convergence and generation of high quality real samples become a challenge. Even under such circumstances, the SDE-based models converge. Furthermore, such models can eventually generate high quality real images. However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images. This makes it a better choice for the near-ND setup.

As can be seen, the images are clearly different semantically from the normal class. Despite this semantic difference, most novelty detection models have a poor performance in the detection of these images as anomalous.

To further extend our results, we compared the methods on a number of small datasets from different domains: 102 Category Flowers and Caltech-UCSD Birds 200. For each of these datasets, we evaluated the methods using only each of the first 20 classes as normal, and using the entire test set for the evaluation. For FGVC-Aircraft, due to the extreme similarity of the classes to each other, we randomly selected a subset of ten classes from the entire dataset, such that no two classes have the same Manufacturer.

MvTecAD: This dataset contains 15 different industrial products, with normal images of the proper products for training and 1 to 9 types of manufacturing errors as anomalies. The anomalies in MvTecAD are in-class, i.e. the anomalous images come from the same class of the normal images with subtle variations. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 2: The performance of novelty detection methods (AUROC %) in the near-ND and ND settings. In the near-ND setting, results are reported according to both criteria, i.e. CLP and bottom-1.

Input: Figure 2: Overview of our framework for near distribution novelty detection. In the first step, the SDE diffusion model generates semantically close outliers, after its training is early-stopped. As it is shown, they have very subtle yet semantic differences with the normal data distribution (downleft). After that, using a linear layer, a pre-trained feature extractor is fine-tuned to solve a binary classification between the normal and abnormal inputs. This modifies the normal boundaries toward including more distinctive features. Finally, normal embeddings are stored and used to compute the k-NN distance at the test time as the novelty score.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 2: The performance of novelty detection methods (AUROC %) in the near-ND and ND settings. In the near-ND setting, results are reported according to both criteria, i.e. CLP and bottom-1.

Input: Figure 2: Overview of our framework for near distribution novelty detection. In the first step, the SDE diffusion model generates semantically close outliers, after its training is early-stopped. As it is shown, they have very subtle yet semantic differences with the normal data distribution (downleft). After that, using a linear layer, a pre-trained feature extractor is fine-tuned to solve a binary classification between the normal and abnormal inputs. This modifies the normal boundaries toward including more distinctive features. Finally, normal embeddings are stored and used to compute the k-NN distance at the test time as the novelty score.

Input: Table 5: For each model and every class of the CIFAR-10, the anomalous class that minimizes the anomaly detection performance among 100 classes of the CIFAR-100 dataset is reported. The last row is the CIFAR-100 class that achieves the highest CLP score. Note that the latter does not depend on the anomaly detection method.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 4: For each class of CIFAR-10, nearest class from CIFAR-100 has been provided, according to the closeness score. In each 3-row panel, the first row is assumed as normal, the second one is assumed the nearest abnormal class, and the third row indicates the corresponding generated fake images using the SDE model.

Input: Figure 5: For each class of CIFAR-10, nearest class from CIFAR-100 has been provided, according to the closeness score. In each 3-row panel, the first row is assumed as normal, the second one is assumed the nearest abnormal class, and the third row indicates the corresponding generated fake images using the SDE model.

Input: Figure 3: Left: Unlike GAN-based models, the diffusion SDE model performs much better in consistently improving the fidelity and diversity of the generated samples along the training. The metrics “density” and “coverage” (27) are used to reliably measure these concepts. Thus, obtaining high-quality and diverse anomalous samples is feasible by early stopping in the SDE model. Right: Some generated samples by various methods in different datasets as well as their corresponding normal samples are shown. As can be seen SDE samples, in the third row, are authentic (differ from the normal class), high-quality, and have minimal differences to the normal class, while other methods produce low quality samples that are far away from the normal distribution.

Input: Figure 3: By increasing i, the AUROCs of the models improve due to the presence of more anomalies that are further from the boundary. As expected, in most cases, the gap between the performance of models has become smaller. Furthermore, in the case of i = 1, in which anomalies constitute the near distribution ones, the models’ performances vary greatly. In this case, our proposed method achieves SOTA results by a large margin. 16

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence:

The claim about benchmarking eight novelty detection methods in near-ND settings is supported by comprehensive experimental results shown in Table 1, which presents comparative performance across multiple datasets and methods. The evidence demonstrates thorough evaluation using both natural and synthetic near-distribution cases (FGVC-Aircraft, Stanford-Cars, WBC, Weather datasets, and CIFAR10-vs-CIFAR100). The reviewer comments don't dispute the benchmarking contribution but rather focus on other aspects of the paper. The claim accurately represents the scope and nature of the benchmarking without exaggeration.

<score>1</score>