Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our contribution in this work is threefold:

The review comment to be evaluated is:

### Summary

The authors present a stochastic algorithm for selecting a subset of a large dataset, while trying to preserve statistics from the original dataset. The algorithm is a two-step process, first selecting a set of "candidates" based on individual features, then filtering to a final subset which may account for interactions between candidate items.

### Strengths

The paper provides empirical experimental results which suggest the superiority of the proposed method against two baselines: Learning to Sample (LTS) and random selection (RS).

Very few grammatical mistakes.

### Concerns and Issues that Need Clarification

1. A significant amount of active learning literature deals with subset selection. However, this paper fails to discuss or even cite any results from the active learning literature. It would be helpful for the authors to compare/contrast their proposed method with existing active learning results. For example, consider
   - [*Active Learning for Convolutional Neural Networks: A Core-Set Approach* by Sener and Savarese (2018)](https://openreview.net/forum?id=H1aIuk-RW)
   - [*Selection via Proxy: Efficient Data Selection for Deep Learning* by Coleman et al. (2019)](https://openreview.net/forum?id=HJg2b0VYDr)
   - [*Submodularity in Data Subset Selection and Active Learning* by Wei et al. (2015)](http://proceedings.mlr.press/v37/wei15.html)
   - [*On Statistical Bias In Active Learning: How and When to Fix It* (2020)](https://openreview.net/forum?id=JiYq3eqTKY) - this paper is a submission to ICLR 2020, so obviously there is no expectation that the authors would be familiar with this work. However, I bring this paper up because it discusses a similar topic of trying to select data points from a larger dataset that provide a reasonable estimate of a loss function on the "population" vs. a "sample."

2. The algorithm, as described, states that individual elements of a dataset $D$ are "possibly represented" as a pair $(x_i, y_i)$. What does "possibly represented" mean? Assuming that the dataset elements are given as a $(x_i, y_i)$ pair, the stochastic subset selection algorithm seems to describe instance selection. However, the paper also claims that the algorithm works for feature selection. What are $D$, $x_i$, and $y_i$ in the feature selection tasks? The paper makes an attempt at clarifying this by stating, "Here, x is 2-dimensional and y is 3-dimensional for RGB images." But this still leaves the reader clueless - why is $x$ only 2-dimensional? What are those 2 dimensions?

3. The end of the introduction claims that the proposed subset selection method "learns to sample a subset from a larger set with linear time complexity," yet Section 3.4 claims that the inference complexity is $O(n) + O(k^2 d/q)$, which is nonlinear if either $k$ or $d$ is chosen to be dependent on $n$. Could the authors please clarify?

4. The authors claim that their method uses meta-learning, but no mention of meta learning is made beyond the "Preliminaries" subsection of the "Approach" section. How is this method a example of meta-learning?

5. The paper provides experimental evidence that the proposed method outperforms other subset selection baselines, such as Learning to Select (LTS) and random selection (RS). However, the paper provides little (if any) intuition or formal justification for why the proposed method performs better. The authors need to explain the motivation behind the proposed algorithm.

6. The writing and figures lack clarity and are often difficult to follow. Some examples:
   - The context for the mathematical expression in (4) is very unclear. Is (4) an objective function? If so, is this expression supposed to be minimized or maximized? And what is the random variable $Z$ supposed to represent? And why is there a period in the middle of the expression?
   - Figure 2 is poorly explained and difficult to understand. I'm assuming that the blue vertical vector is supposed to represent $D_c$. Is that correct? And are the dark shaded boxes supposed to represent the "candidates" and the lightly shaded boxes supposed to represent the items that were not selected? As a reader, I should not have to guess what the colors mean. They should be explained clearly in the caption, or at the very least, in the paper text. Furthermore, it seems to me (again it's never made explicit) that Figure 2b is depicting the Autoregressive Subset Selection ("AutoSelect") routine. In this case, why is there a white row vector of $D = d_1, \dotsc, d_n$? In Algorithm 1, it seems as though AutoSelect only takes as input $D_s$ and $D_c$, which are subsets of $D$ and not $D$ itself.

### Original Rating and Confidence

**Rating** - 4: Ok but not good enough - rejection

**Confidence** - 4: The reviewer is confident but not absolutely certain that the evaluation is correct

### Updated Review after Author Response

Thank you for the thoughtful responses to my questions, and apologies for the delay in updating my review. In general the responses have addressed my questions, and I have updated my rating accordingly. However, there are still several issues in my mind:

1. **Connection to active learning** - The authors correctly point out that active learning does not assume access to labels, whereas the proposed method does. However, I believe the comparison against selection algorithms such as k-center greedy should be included in the paper for completeness sake, and not just left as a comment in this forum.

2. **Dataset Distillation: Instance Selection** - Upon re-reading the paper carefully, I found this subsection in Section 3.6 to be confusing. Why are there multiple $D_i$? The section seems to only talk about one $D_i$.

3. **Set vs. Tensor**: In the new "representation learning" subsection of Section 2, the paper stresses that the model learns a representation of a set, instead of an individual data point. However, many tasks (e.g., image reconstruction) do impose some sort of ordering. The paper lacks clarity about where permutation-invariance is used vs. not used.

4. **Other places that need clarification**:
  - Equation (2): presumably you threshold the output of the sigmoid function, right? Because in Algorithm 1, it seems like $z_i$ are supposed to be boolean-valued.
  - Appendix: there are several missing closing parentheses

The evidence to be evaluated is:Firstly, our model can be applied to any type of sets (and instances represented as sets), while the aforementioned models mainly work for images represented in tensor form. Furthermore, our method can be applied both at the instance and dataset level.

In this work, we consider data of the type \(D=\{d_{1},\ldots,d_{n}\}\) where individual \(d_{i}\)'s are possibly represented as _input_\(x_{i}\) and _target_\(y_{i}\). \(D\) is the complete set and we assume that within \(D\), there exists a subset \(D_{s}=\{s_{i},\ldots,s_{k}\}\subset D\) such that \(k\ll n\) and that for an arbitrarily defined loss function \(\ell(.,D)\) that we are interested in optimizing over the full set \(D\), \(D_{s}\) can be used as a proxy for \(D\) such that \(\ell(.,D)\approx\ell(.,D_{s})\). In what follows, we present a method that learns the conditional distribution \(p(D_{s}|D)\) of the subset \(D_{s}\) via a two stage selection procedure dubbed _candidate selection_ and _autoregressive subset selection_. The overall objective is then to minimize the loss function with respect to the subset \(D_{s}\), \(\mathbb{E}_{p(D,|D)}[\ell(.,D_{s})]\). When the set \(D\) itself follows a distribution of sets as in the _meta-learning_ framework, then the objective becomes \(\mathbb{E}_{D}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]]\). In essence, we seek to construct a subset \(D_{s}\) that is optimally representative of the full set \(D\) w.r.t \(\ell(.)\).

As such, we first present the _candidate selection_ procedure used to construct a smaller set, \(D_{c}\), without considering inter-sample dependencies. This is then followed by the _autoregressive subset selection_ procedure used to construct \(D_{s}\) from \(D_{c}\) by modeling inter-sample dependencies.

We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\). We implement both \(r(D)\) and \(\rho(d_{i},r(D))\) as neural networks and specifically for \(r(D)\), we use Deep Sets (Zaheer et al., 2017). Since _Ber_ is non-differentiable, we use the continuous relaxations of the Bernoulli distribution introduced in (Maddison et al., 2016; Jang et al., 2016; Gal et al., 2017). Specifically, to sample \(z_{i}\), we execute the following computational routine: \[z_{i}=\sigma\Big{(}\frac{1}{\tau}\Big{(}\log\frac{\pi_{i}}{1-\pi_{i}}+\log \frac{u}{1-u}\Big{)}\Big{)},\quad\pi_{i}=\rho(d_{i},r(D)),\quad u\sim\text{ Unif}(0,1), \tag{2}\] where \(\sigma\) is the Sigmoid function, \(\tau\) is the temperature for the continuous relaxation and \(u\) is sampled from the uniform distribution. \(\tau\) is set to \(0.05\) in all our experiments.

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

The key to avoiding samples with redundant information in \(D_{s}^{(k)}\) lies in the fact that for each element added to \(D_{s}\), it's selection is conditioned on both \(D_{c}\) and all elements in \(D_{s}^{(i-1)}\). We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

The inference complexity depends heavily on the choice of the function \(f\). If \(f\) considers the pairwise interactions between all candidate elements and the selected elements, the inference complexity is \(O(n)+O(k^{2}d/q)\) where \(n,d,k\) correspond to \(|D|,|D_{c}|\) and \(|D_{s}|\) respectively. In our experiments, for the choice of the function \(f\), we utilize either a Set Transformer(Lee et al., 2018) or DeepSets(Zaheer et al., 2017) to model the pairwise interactions between the elements of a given set.

We can also opt to train the network to predict a single target \(y_{D}\) for the set \(D\). For instance, the target could be the class of an image(classification) or the statistics of the set(regression problem). Here, \(p_{\theta}(y_{D}|D_{s})\) is a neural network that predicts the target \(y_{D}\). A set in this task, may be the features from a single example like an image and experimental results can be found in Section 4.2. The model for this task is depicted in Figure 2(b).

We describe a model capable of taking as input \(D_{i}\in\mathcal{D}\) to perform a task such as the reconstruction of all elements in the given dataset. For a single dataset \(D_{i}\in\mathcal{D}\), we apply the subset construction method already described to a \(D_{s}\) that can be used to reconstruct all the elements in \(D_{i}\). In essence, \(D_{i}\) is _distilled_ into a new dataset \(D_{s}\) with \(k<|D_{i}|\) elements.

As a first step, we represent \(D_{s}\) as a unique representative vector \(c\) for each element in the dataset akin to the statistics network used in the Neural Statistician (Edwards and Storkey, 2016) model. Specifically, to reconstruct an element \(d_{i}\in D_{i}\) given \(D_{s}\), \(c\) is computed by applying a stochastic cross-attention mechanism on \(D_{s}\) where the stochasticity is supplied by a query \(\alpha\) which is computed using \(d_{i}\). To obtain varying styles in the generated images, we additionally learn a latent variable \(w\) used to perturb \(c\) and both are combined to obtain a new element \(x\).

Additionally, to ensure that \(c\) is properly learnt, we add an informativity loss by reconstructing \(c\) from the generated samples from the given dataset. The objective for the model depicted in Figure 2(c) for a single dataset \(D\) is : \[\begin{split}\mathcal{L}(\theta,\phi,\psi)=\sum_{d_{i}\in D}[ \mathbb{E}_{q_{\phi}(w_{i}|d_{i})}[p_{\theta}(d_{i}|w_{i},c_{i})]-\text{KL}[q_ {\phi}(w_{i}|d_{i})||p_{\phi}(w)]\\ -\text{KL}[q_{\phi}(\alpha_{i}|d_{i})||p_{\phi}(\alpha)]-\text{ KL}[q_{\phi}(c_{i}|D_{s},\alpha_{i})||p_{\phi}(c)]\end{split} \tag{7}\] where \(p_{\psi}(\cdot)\) are priors on their respective latent variables and \(q_{\phi}(\cdot)\)'s are implemented with neural networks.

In addition to Equation 7 and the informativity loss, the model is optimized together with the subset selection model already described. When the model is fully optimized, it is applied to the instance selection task on the given dataset. In summary, the purpose of the generative model introduced is to train the subset selection module for the instance selection task.

Finally in the dataset distillation task, we consider the problem of selecting prototypes to be used for few-shot classification. Here, we adopt Prototypical Networks (Snell et al., 2017) and apply the subset selection model to the task of selecting representative prototypes from each class to be used for classifying new instances. By learning to select the prototypes, we can remove outliers that would otherwise change the class prediction boundaries in the classification task. The complete graphical model for this task is given in Figure 2(d) where again \(D_{s}\) corresponds to the selected prototypes and \(x_{*}\) and \(y_{*}\) correspond to query and class label respectively.

Our first experiment is on 1D function reconstruction. Suppose that we have a function \(f:[a,b]\rightarrow\mathbb{R}\). We first construct a set of data points of that function: \(D=\{(x_{1},y_{1}=f(x_{1})),(x_{2},y_{2}=f(x_{2})),\ldots,(x_{n},y_{n}=f(x_{n }))\}\) where \((x_{1},x_{2},\ldots,x_{n})\) are uniformly distributed along the x-axis within the interval \([a,b]\). Now if we have a family of functions \((f^{(1)},f^{(2)},\ldots,f^{(N)})\), this will lead to a family of sets \((D^{(1)},D^{(2)},\ldots,D^{(N)})\). We train our model which consists of the subset selection model \(p(D_{s}|D)\) and a task network \(p(Y|X,D_{s})\) (e.g. ANP), on this data set and report the reconstruction loss, which is the negative log-likelihood. SSS out-performs Random Select (RS), verifying that the subset selection model \(p(D_{s}|D)\) can learn a meaningful distribution for the selected elements. Our model also out-performs the Learning to Sample (LTS) baseline. Through the visualization of the selected points in Figure 5, we can see that out model tends to pick out more points (presented as red dots) in the drifting parts of the curve, which is reasonable since these parts are harder to reconstruct.

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

An ANP is then used to reconstruct the remaining pixels from a set of context elements (selected subset in our case). We conduct this experiment on the CelebA dataset (Liu et al., 2018). Figure 3(b) shows that our model significantly outperforms ANP with RS (as in the original ANP paper) and the LTS baseline. Figure 6 shows the reconstruction samples of our model which are visually better than the reconstruction of the baselines for the same number of pixels.

We again use the CelebA dataset, but this time the selected pixels are used to give predictions for 40 attributes of a celebrity's face (in a multi-task learning setting). For our proposed model, only the selected pixels are used for prediction (other pixels' values are set to zeros). Table 1 shows that using only 500 pixels (\(\sim\)1.3% of total pixels in an image), we can achieve a mean AUC of 0.9093 (99.3% of the accuracy obtained with the full image).

The AUC with selected pixels learned from our SSS is significantly higher than that of the random pixels baseline, showing the effectiveness of our subset selection method.

This is our stochastic subset selection model trained for reconstruction, but then later used for classification. Our model outperforms this variant, showing the effectiveness of training with the target task.

Since our method is stochastic, the predictive distribution can be written as \(\mathbb{E}_{p(D_{s}|D)}\left[p_{\theta}(y_{D}|D_{s})\right]\), and we can use Monte Carlo sampling to get the prediction in practice. However, throughout the experiment section, we only reported the result with one sampled subset, since it gives the best reduction in memory and computational cost. This can be seen as MC sampling with one sample. We compare it against another variant: SSS(MC) with MC sampling (5 samples). It should be noted that by doing MC sampling with 5 samples, the computational cost (inference) is increased by 5 times, and the memory requirement can be increased by up to 5 times too. Table 1 shows that our model achieves comparable performance with that variant, thus justifying that it can achieve good performance for target tasks, while reducing memory and computation requirement. **Instance Selection** We present results on the instance selection task applied to a whole dataset. In this task, we use the CelebA dataset since it has an imbalance both in terms of gender and race. A dataset is constructed by sampling 200 random images from the full dataset. In this experiment, we seek to select only a few(5-30) representative images from these generated datasets. On this task, our subset selection module is trained via the procedure detailed in Section 3.6 on instance selection. To evaluate the effectiveness of the SSS model, we evaluate the model in terms of the diversity in the selected subset using the Frechet Inception Distance(FID Score) (Heusel et al., 2017) which measures the similarity and diversity between two datasets. We compare our model with the model that randomly samples instances from the full dataset. Additionally, we compare our method with the Farthest Point Sampling(FPS) algorithm which selects \(k\) points from a given set by computing distances on a metric space between all elements and selecting those elements that are furthest from each other. FPS in general seeks to obtain a wide coverage over a given set and hence is a suitable baseline. The results of this experiment is presented in Table 4 where our selection method achieves a lower FID score compared to FPS and Random Sampling. Additionally, given that the dataset is highly imbalanced, FPS performs worst since by selecting the furthest elements in the given set it cannot capture the true representation of the whole dataset even when compared with Random Sampling. Also for small sample selection, our method outperforms FPS and Random Sampling significantly since our method is able to model the interactions within the full dataset and hence can select the most representative subset.

The results for this experiment is shown in Table 3, where it can be observed that SSS can learn to select more representative prototypes compared to the other methods especially in the few-shot problems where the choice of prototypes matters more.

In this paper, we have proposed a stochastic subset selection method to reduce the size of an arbitrary set while preserving performance on a target task. Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account. Hence, it can select a compact set that avoids samples with redundant information. By using the compact subset in place of the original set for a target task, we can save memory, communication and computational cost.

1:\(k\)(max subset size) 2:\(q\)(# elements selected at each iteration) 3:\(p(D)\) (distribution of sets) 4:\(\alpha\) (learning rate) 5: a target task with loss function \(\ell(\cdot,\cdot)\) 6:trained model with converged \(\theta\) and \(\phi\) 7:\(\theta,\phi\leftarrow\) initialization 8:while not converged do 9: Sample a minibatch with \(m\) sets \(D^{(1)},D^{(2)},\ldots,D^{(m)}\) from \(p(D)\)

Input: Figure 5: 1D Function Reconstruction.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input: Table 4: . FID Score for varying Instance Selection

Input: Figure 6: CelebA reconstruction samples with varying number of pixels.

Input: Figure 3: Graphical Models: (a) Feature Selection for Reconstruction. (b) Feature Selection for Prediction Task. (c) Instance Selection model. (d) Instance Selection model for Classification

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, the claim "Our contribution in this work is threefold" appears incomplete as the actual three contributions are not stated in the claim itself. However, assessing this as an overarching claim, the evidence demonstrates that the authors have presented:

1. A novel stochastic subset selection method with a two-stage process (Bernoulli and Categorical)
2. Applications across multiple domains (function reconstruction, image reconstruction, feature selection, instance selection)
3. Empirical evidence showing superior performance compared to baselines

The evidence comprehensively supports these contributions through detailed methodology, mathematical formulations, experimental results, and comparisons. While the reviewer raises some valid concerns about clarity and comparisons to active learning literature, the core contributions are well-supported by the extensive technical details and experimental validation provided.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We propose a novel two-stage stochastic subset selection method that **learns to sample** a subset from a larger set with linear time complexity, with minimal loss of accuracy at the downstream task.

The evidence to be evaluated is:Firstly, our model can be applied to any type of sets (and instances represented as sets), while the aforementioned models mainly work for images represented in tensor form. Furthermore, our method can be applied both at the instance and dataset level.

Our instance-sampling model is also related to the Variational Auto Encoder (VAE) (Kingma and Welling, 2013). However, while VAE learns a compact representation of a _data point_, our model learns a compact representation of a _set_.

Our method also falls in this category.

In this work, we consider data of the type \(D=\{d_{1},\ldots,d_{n}\}\) where individual \(d_{i}\)'s are possibly represented as _input_\(x_{i}\) and _target_\(y_{i}\). \(D\) is the complete set and we assume that within \(D\), there exists a subset \(D_{s}=\{s_{i},\ldots,s_{k}\}\subset D\) such that \(k\ll n\) and that for an arbitrarily defined loss function \(\ell(.,D)\) that we are interested in optimizing over the full set \(D\), \(D_{s}\) can be used as a proxy for \(D\) such that \(\ell(.,D)\approx\ell(.,D_{s})\). In what follows, we present a method that learns the conditional distribution \(p(D_{s}|D)\) of the subset \(D_{s}\) via a two stage selection procedure dubbed _candidate selection_ and _autoregressive subset selection_.

In essence, we seek to construct a subset \(D_{s}\) that is optimally representative of the full set \(D\) w.r.t \(\ell(.)\).

As such, we first present the _candidate selection_ procedure used to construct a smaller set, \(D_{c}\), without considering inter-sample dependencies. This is then followed by the _autoregressive subset selection_ procedure used to construct \(D_{s}\) from \(D_{c}\) by modeling inter-sample dependencies.

We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\). We implement both \(r(D)\) and \(\rho(d_{i},r(D))\) as neural networks and specifically for \(r(D)\), we use Deep Sets (Zaheer et al., 2017).

Specifically, to sample \(z_{i}\), we execute the following computational routine: \[z_{i}=\sigma\Big{(}\frac{1}{\tau}\Big{(}\log\frac{\pi_{i}}{1-\pi_{i}}+\log \frac{u}{1-u}\Big{)}\Big{)},\quad\pi_{i}=\rho(d_{i},r(D)),\quad u\sim\text{ Unif}(0,1), \tag{2}\] where \(\sigma\) is the Sigmoid function, \(\tau\) is the temperature for the continuous relaxation and \(u\) is sampled from the uniform distribution. \(\tau\) is set to \(0.05\) in all our experiments.

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

If \(f\) considers the pairwise interactions between all candidate elements and the selected elements, the inference complexity is \(O(n)+O(k^{2}d/q)\) where \(n,d,k\) correspond to \(|D|,|D_{c}|\) and \(|D_{s}|\) respectively. In our experiments, for the choice of the function \(f\), we utilize either a Set Transformer(Lee et al., 2018) or DeepSets(Zaheer et al., 2017) to model the pairwise interactions between the elements of a given set.

11:procedureAutoSelect(\(q,D_{s},D_{c}\)) 12:\(C=\{w_{1},w_{2},\dots,w_{m}\}\gets D_{c}\setminus D_{s}\) 13:\((p_{1},p_{2},\dots,p_{m})\leftarrow(f(w_{1},D_{c},D_{s}),f(w_{2},D_{c},D_{s}), \dots,f(w_{m},D_{c},D_{s}))\) 14:\((p_{1},p_{2},\dots,p_{m})\leftarrow(p_{1},p_{2},\dots,p_{m})/\sum_{j=1}^{m}p_{j}\) 15:\(Q\leftarrow\text{Select }q\) elements from \(C\) with probability \((p_{1},p_{2},\dots,p_{m})\)

For computational efficiency, we may desire to restrict the size of \(D_{c}\) to save computational cost when constructing \(D_{s}\). We adopt the idea of Information Bottleneck and constrain the distribution of \(Z\) for \(D_{c}\).

\[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]+\beta\text{KL}[p(Z|D) ||r(Z)]] \tag{4}\]where \(r(Z)\) is a sparse prior. In our experiments, we set the parameter of the Bernoulli spares prior \(r(Z)\) to either \(0.1\) or \(0.01\) for different levels of sparsity and \(\beta\) is set to \(1.0\) or \(0.001\).

\[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[-\log p_{\theta}(Y|X,D_{s})]+\beta \text{KL}[p(Z|D)||r(Z)]] \tag{5}\] Minimizing this objective ensures that we learn a compact subset(\(D_{s}\)) most representative of \(D\) and \(D_{s}\) can then be used for other tasks.

For a single dataset \(D_{i}\in\mathcal{D}\), we apply the subset construction method already described to a \(D_{s}\) that can be used to reconstruct all the elements in \(D_{i}\). In essence, \(D_{i}\) is _distilled_ into a new dataset \(D_{s}\) with \(k<|D_{i}|\) elements.

As a first step, we represent \(D_{s}\) as a unique representative vector \(c\) for each element in the dataset akin to the statistics network used in the Neural Statistician (Edwards and Storkey, 2016) model. Specifically, to reconstruct an element \(d_{i}\in D_{i}\) given \(D_{s}\), \(c\) is computed by applying a stochastic cross-attention mechanism on \(D_{s}\) where the stochasticity is supplied by a query \(\alpha\) which is computed using \(d_{i}\).

In summary, the purpose of the generative model introduced is to train the subset selection module for the instance selection task.

Finally in the dataset distillation task, we consider the problem of selecting prototypes to be used for few-shot classification. Here, we adopt Prototypical Networks (Snell et al., 2017) and apply the subset selection model to the task of selecting representative prototypes from each class to be used for classifying new instances. By learning to select the prototypes, we can remove outliers that would otherwise change the class prediction boundaries in the classification task.

We train our model which consists of the subset selection model \(p(D_{s}|D)\) and a task network \(p(Y|X,D_{s})\) (e.g. ANP), on this data set and report the reconstruction loss, which is the negative log-likelihood. SSS out-performs Random Select (RS), verifying that the subset selection model \(p(D_{s}|D)\) can learn a meaningful distribution for the selected elements. Our model also out-performs the Learning to Sample (LTS) baseline. Through the visualization of the selected points in Figure 5, we can see that out model tends to pick out more points (presented as red dots) in the drifting parts of the curve, which is reasonable since these parts are harder to reconstruct.

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

An ANP is then used to reconstruct the remaining pixels from a set of context elements (selected subset in our case).

Figure 3(b) shows that our model significantly outperforms ANP with RS (as in the original ANP paper) and the LTS baseline. Figure 6 shows the reconstruction samples of our model which are visually better than the reconstruction of the baselines for the same number of pixels.

The goal is to learn to select a subset for a target task such as classification or regression.

Table 1 shows that using only 500 pixels (\(\sim\)1.3% of total pixels in an image), we can achieve a mean AUC of 0.9093 (99.3% of the accuracy obtained with the full image).

The AUC with selected pixels learned from our SSS is significantly higher than that of the random pixels baseline, showing the effectiveness of our subset selection method.

Our model outperforms this variant, showing the effectiveness of training with the target task.

Since our method is stochastic, the predictive distribution can be written as \(\mathbb{E}_{p(D_{s}|D)}\left[p_{\theta}(y_{D}|D_{s})\right]\), and we can use Monte Carlo sampling to get the prediction in practice. However, throughout the experiment section, we only reported the result with one sampled subset, since it gives the best reduction in memory and computational cost.

We compare it against another variant: SSS(MC) with MC sampling (5 samples). It should be noted that by doing MC sampling with 5 samples, the computational cost (inference) is increased by 5 times, and the memory requirement can be increased by up to 5 times too. Table 1 shows that our model achieves comparable performance with that variant, thus justifying that it can achieve good performance for target tasks, while reducing memory and computation requirement.

On this task, our subset selection module is trained via the procedure detailed in Section 3.6 on instance selection. To evaluate the effectiveness of the SSS model, we evaluate the model in terms of the diversity in the selected subset using the Frechet Inception Distance(FID Score) (Heusel et al., 2017) which measures the similarity and diversity between two datasets.

Also for small sample selection, our method outperforms FPS and Random Sampling significantly since our method is able to model the interactions within the full dataset and hence can select the most representative subset.

In this paper, we have proposed a stochastic subset selection method to reduce the size of an arbitrary set while preserving performance on a target task. Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account. Hence, it can select a compact set that avoids samples with redundant information. By using the compact subset in place of the original set for a target task, we can save memory, communication and computational cost.

The idea of the greedy training algorithm is to train the auto-regressive model to select the best next \(q\) elements from the candidate set to minimize the target loss on the selected samples. By doing this, we do not have to run the auto-regressive model \(k/q\) time during training, thus reducing the computational cost.

5: a target task with loss function \(\ell(\cdot,\cdot)\) 6:trained model with converged \(\theta\) and \(\phi\) 7:\(\theta,\phi\leftarrow\) initialization 8:while not converged do 9: Sample a minibatch with \(m\) sets \(D^{(1)},D^{(2)},\ldots,D^{(m)}\) from \(p(D)\) 10:\(D^{(j)}_{\theta}\sim p(D^{(j)}_{\theta}|D^{(j)})\) for \(j=1\ldots m\) 11:\(i\sim\) random sample from \((0,\ldots,k-q)\) 12:\(I^{(j)}\sim\) random \(i\)-element subset of \(D^{(j)}_{\theta}\) for \(j=1\ldots m\) 13:\(Q^{(j)}\sim\) select a \(q\)-element subset from \(D^{(j)}_{\theta}\setminus I^{(j)}\) (with the auto-regressive model) 14:\(\theta\leftarrow\theta-\alpha\nabla_{\theta}\frac{1}{m}\sum_{j=1}^{m}\ell( \cdot,I^{(j)}\cup Q^{(j)}),\phi\leftarrow\phi-\alpha\nabla_{\phi}\frac{1}{m} \sum_{j=1}^{m}\ell(\cdot,I^{(j)}\cup Q^{(j)})\)

Further, we provide qualitative results on a single dataset in Figure 10 where we show how our model picks 5 instances from the full set of 200 images face images.

SSS consists of \(r(D)\), \(\rho(d_{i},r(D))\) and \(f(d,D_{c},D_{s}^{(i-1)}\). We describe the models in this section. For all experiments, \(r(D)\) is implemented as DeepSets. This means that we take the mean of all the samples in a set to obtain the set representation. \(\rho(d_{i},r(D))\) is implemented as a neural network with the following specifications: there are 3 Linear layers each followed by ReLU activation. Also, all inputs are projected into feature space using 3 Linear layers, each followed by ReLU activation.

Input: Figure 5: 1D Function Reconstruction.

Input: Figure 10: Visualization of a set with 200 images for instance selection. The two stage selection method in SSS is visualized as Candidate Set and SSS. A coreset of size 5 is visualized.

Input: Figure 6: CelebA reconstruction samples with varying number of pixels.

Input: Figure 3: Graphical Models: (a) Feature Selection for Reconstruction. (b) Feature Selection for Prediction Task. (c) Instance Selection model. (d) Instance Selection model for Classification

Input: Figure 4: Models’ performance on set reconstruction and classification task (lighter-color areas present the standard deviations). (a) 1D Function reconstruction. (b) CelebA (reconstruction). (c) CelebA (classification)

Input: Figure 11: Sample visualization of prototype selection for the miniImagenet dataset on the few-shot classification task. Each row represents a set that corresponds to the support from which a prototype is selected for the few-shot classification task.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input: Figure 8: Reconstruction samples of 1D functions with different selection methods

Input: Figure 9: Selected pixels for different tasks on CelebA.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears to be somewhat overstated. While the paper does present a novel two-stage stochastic subset selection method that operates with linear time complexity, the "learns to sample" phrasing implies more capability than demonstrated. The evidence shows a carefully engineered approach using Bernoulli and Categorical distributions with neural networks, rather than a method that truly "learns" sampling behavior. The experimental results demonstrate good performance but with some limitations - notably that they only report results with one sampled subset rather than multiple samples, and the method requires careful tuning of hyperparameters.

<score>3</score>

Justification:
- The claim's wording about "learning to sample" overstates the method's capabilities - it uses standard probabilistic sampling techniques rather than learning sampling behavior
- The "minimal loss of accuracy" part is supported by empirical results (e.g., 99.3% accuracy with only 1.3% of pixels)
- Time complexity claim is supported by the O(n)+O(k^2d/q) analysis
- Several limitations and implementation details are not acknowledged in the claim
- Experimental validation shows good but not universal superiority over baselines

The claim would be better stated by being more specific about the probabilistic sampling approach used and acknowledging key limitations/assumptions.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We propose a novel two-stage stochastic subset selection method that **learns to sample** a subset from a larger set with linear time complexity, with minimal loss of accuracy at the downstream task.

The review comment to be evaluated is:

**Final recommendation**
I support accepting this paper. While I am a bit disappointed the authors did not add in the paper the results they discussed during the rebuttal, I think the paper is interesting and clearer than at submission.

**Summary**
stochastic subset selection (SSS) is a method to learn to compress a set $D$ by selecting a subset $D_s$ such that the loss of a task performed on $D_s$ is as close as possible to the loss if the task had been performed on the original $D$. The paper show how this general method can be applied to several tasks. For example, $D$ can contain the pixels of an image and the task can be to reconstruct the original image from the subset. This corresponds to learning to compress. As another example, with the same $D$ but a label prediction task, SSS learns to dynamically select sparse features for a classification task. The other tasks considered are dataset distillation (compressing a data set) and selecting prototypes for few-shot classification. $D_s$ is constructed in two steps. A candidate set is first constructed by considering elements of $D$ independently and in a second step elements of $D_c$ are included in $D_s$ iteratively by considering other elements in $D_c$ and in $D_s$. Experiments are performed for one example of each task previously discussed and SSS is shown to outperform baselines. SSS is motivated by the need to reduce bandwidth, computation and memory footprint of deep learning in edge devices.

---
**Strong points**

1) Code is provided (I did not try to run it).

2) The proposed approach is applied to four different tasks. This convinced me of its applicability.

3) Empirical experiments seem well conducted (with some caveat, see weak point 3) and results are good.

4) The problems tackled are interesting.

**Weak points**

1) The paper is very dense, which makes it hard to follow. Related to that, I think that some parts of the paper are too concise to understand them well. The supplementary material partly compensates that.

2) While the experiments show that the proposed approach works well, I think that some experiments should compare full data pipelines. As an example, one experiment shows that using SSS to dynamically select pixels of an image and using this sparse representation as input leads to good classification accuracy. The paper implicitely suggests that this reduces inference cost if classification is performed on the device and bandwidth if inference is performed on another device. However, the cost of the pixel selection by SSS is never analyzed. I think comparing the memory footprint, the number of operations, bandwidth used and accuracy for all steps of the following pipelines would give a much clearer picture of the interest of the approach for deep learning on mobile devices:
    - classification is done directly on the device, using an optimized network.
    - pixels are randomly selected and classification is performed on the device.
    - pixels are randomly selected, transmitted and classification is performed on another device.
    - pixels are selected by SSS, which involves computing multiple embeddings, and classification is performed on the device.
    - pixels are selected by SSS, transmitted and classification is performed on another device.

---
**Recommendation**
Overall, I vote for accepting. I am on the fence with this paper. The current results are good and I think the proposed approach has some potential and will be interesting for the community. On the other hand the paper is hard to read and the experiments, while good to show the strength of the approach, do not cover usefullness for the mobile use case. I think the former can be adressed in time and that the latter, while important, does not prevent the paper to be published.

---
**Details**

I like that each task is first formally described and then evaluated in a specific case with the experiments. However I only understood most tasks when reading the experiment sections. I would suggest to either provide a short illustration of an actual instance of the task when describing the task formally, to directly provide the experimental result after describing the task or to direct the reader to the relevant experimental section(s) for illustration.

The paper states that model descriptions are available in appendix A, but I could not find them.

For the *Set Classification/Prediction* task, I would suggest clarifying that for this task a set typically contains the features of a single example, so $D=x$. I only understood this when reading the experimental section.

The *Dataset Distillation: Instance Selection* task was not clear to me. Here are a few points that puzzled me.
- The text suggest the proposed approach is applied to each $D_i$, but figure 3(c) has no $D_i$ and $i$ seems to be used as an index of $x$ rather than $D$. Furthermore, in figure 3(c) a single D_s is constructed whereas I was expecting one per $D_i$.
- The paper states that *It is important to note that $c$ is computed using only $D_i$ for every element in $D$*. However $c$ also depends on *a query $\alpha$ which is computed using $d_i$*, which, if I understood correctly, is the element of $D$ and can be outside $D_i$. So it seems that $c$ is not computed using only $D_i$.

For the *Dataset Distillation: Classification* task, in figure 3d, the labels are not observed. How can the model learn?

1) For this task, if the label is observed, how is this different from the first task?

2) For the Image Reconstruction experiment, why is $x$ 2 dimensional? Are the inputs black and white images?

I think that the image reconstruction experiment should include a comparison to compression algorithms. I agree with the paper that the proposed approach is applicable to many other types of data than images, but it would be very interesting to see how the approach compares to expert defined methods. Furthermore, implementation in relation to mobile devices is the motivation of this work and as far as I know classical compression algorithms are still used on mobile devices.

---
**Questions**

Questions 1 and 2 in *details* above.

I would also be interested to hear authors' opinions on weak point 2 and a comparison to compression.

For classification, pixels not selected are set to 0. Wouldn't the inference cost be the same when using convolutional networks?

---
**Minor details**

For the *set reconstruction* task, would it also be interesting to learn a model on $D_s$ and use it to predict $Y$ from $X$?

*typos*
foro

The evidence to be evaluated is:Firstly, our model can be applied to any type of sets (and instances represented as sets), while the aforementioned models mainly work for images represented in tensor form. Furthermore, our method can be applied both at the instance and dataset level.

Our instance-sampling model is also related to the Variational Auto Encoder (VAE) (Kingma and Welling, 2013). However, while VAE learns a compact representation of a _data point_, our model learns a compact representation of a _set_.

Our method also falls in this category.

In this work, we consider data of the type \(D=\{d_{1},\ldots,d_{n}\}\) where individual \(d_{i}\)'s are possibly represented as _input_\(x_{i}\) and _target_\(y_{i}\). \(D\) is the complete set and we assume that within \(D\), there exists a subset \(D_{s}=\{s_{i},\ldots,s_{k}\}\subset D\) such that \(k\ll n\) and that for an arbitrarily defined loss function \(\ell(.,D)\) that we are interested in optimizing over the full set \(D\), \(D_{s}\) can be used as a proxy for \(D\) such that \(\ell(.,D)\approx\ell(.,D_{s})\). In what follows, we present a method that learns the conditional distribution \(p(D_{s}|D)\) of the subset \(D_{s}\) via a two stage selection procedure dubbed _candidate selection_ and _autoregressive subset selection_.

In essence, we seek to construct a subset \(D_{s}\) that is optimally representative of the full set \(D\) w.r.t \(\ell(.)\).

As such, we first present the _candidate selection_ procedure used to construct a smaller set, \(D_{c}\), without considering inter-sample dependencies. This is then followed by the _autoregressive subset selection_ procedure used to construct \(D_{s}\) from \(D_{c}\) by modeling inter-sample dependencies.

We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\). We implement both \(r(D)\) and \(\rho(d_{i},r(D))\) as neural networks and specifically for \(r(D)\), we use Deep Sets (Zaheer et al., 2017).

Specifically, to sample \(z_{i}\), we execute the following computational routine: \[z_{i}=\sigma\Big{(}\frac{1}{\tau}\Big{(}\log\frac{\pi_{i}}{1-\pi_{i}}+\log \frac{u}{1-u}\Big{)}\Big{)},\quad\pi_{i}=\rho(d_{i},r(D)),\quad u\sim\text{ Unif}(0,1), \tag{2}\] where \(\sigma\) is the Sigmoid function, \(\tau\) is the temperature for the continuous relaxation and \(u\) is sampled from the uniform distribution. \(\tau\) is set to \(0.05\) in all our experiments.

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

If \(f\) considers the pairwise interactions between all candidate elements and the selected elements, the inference complexity is \(O(n)+O(k^{2}d/q)\) where \(n,d,k\) correspond to \(|D|,|D_{c}|\) and \(|D_{s}|\) respectively. In our experiments, for the choice of the function \(f\), we utilize either a Set Transformer(Lee et al., 2018) or DeepSets(Zaheer et al., 2017) to model the pairwise interactions between the elements of a given set.

11:procedureAutoSelect(\(q,D_{s},D_{c}\)) 12:\(C=\{w_{1},w_{2},\dots,w_{m}\}\gets D_{c}\setminus D_{s}\) 13:\((p_{1},p_{2},\dots,p_{m})\leftarrow(f(w_{1},D_{c},D_{s}),f(w_{2},D_{c},D_{s}), \dots,f(w_{m},D_{c},D_{s}))\) 14:\((p_{1},p_{2},\dots,p_{m})\leftarrow(p_{1},p_{2},\dots,p_{m})/\sum_{j=1}^{m}p_{j}\) 15:\(Q\leftarrow\text{Select }q\) elements from \(C\) with probability \((p_{1},p_{2},\dots,p_{m})\)

For computational efficiency, we may desire to restrict the size of \(D_{c}\) to save computational cost when constructing \(D_{s}\). We adopt the idea of Information Bottleneck and constrain the distribution of \(Z\) for \(D_{c}\).

\[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]+\beta\text{KL}[p(Z|D) ||r(Z)]] \tag{4}\]where \(r(Z)\) is a sparse prior. In our experiments, we set the parameter of the Bernoulli spares prior \(r(Z)\) to either \(0.1\) or \(0.01\) for different levels of sparsity and \(\beta\) is set to \(1.0\) or \(0.001\).

\[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[-\log p_{\theta}(Y|X,D_{s})]+\beta \text{KL}[p(Z|D)||r(Z)]] \tag{5}\] Minimizing this objective ensures that we learn a compact subset(\(D_{s}\)) most representative of \(D\) and \(D_{s}\) can then be used for other tasks.

For a single dataset \(D_{i}\in\mathcal{D}\), we apply the subset construction method already described to a \(D_{s}\) that can be used to reconstruct all the elements in \(D_{i}\). In essence, \(D_{i}\) is _distilled_ into a new dataset \(D_{s}\) with \(k<|D_{i}|\) elements.

As a first step, we represent \(D_{s}\) as a unique representative vector \(c\) for each element in the dataset akin to the statistics network used in the Neural Statistician (Edwards and Storkey, 2016) model. Specifically, to reconstruct an element \(d_{i}\in D_{i}\) given \(D_{s}\), \(c\) is computed by applying a stochastic cross-attention mechanism on \(D_{s}\) where the stochasticity is supplied by a query \(\alpha\) which is computed using \(d_{i}\).

In summary, the purpose of the generative model introduced is to train the subset selection module for the instance selection task.

Finally in the dataset distillation task, we consider the problem of selecting prototypes to be used for few-shot classification. Here, we adopt Prototypical Networks (Snell et al., 2017) and apply the subset selection model to the task of selecting representative prototypes from each class to be used for classifying new instances. By learning to select the prototypes, we can remove outliers that would otherwise change the class prediction boundaries in the classification task.

We train our model which consists of the subset selection model \(p(D_{s}|D)\) and a task network \(p(Y|X,D_{s})\) (e.g. ANP), on this data set and report the reconstruction loss, which is the negative log-likelihood. SSS out-performs Random Select (RS), verifying that the subset selection model \(p(D_{s}|D)\) can learn a meaningful distribution for the selected elements. Our model also out-performs the Learning to Sample (LTS) baseline. Through the visualization of the selected points in Figure 5, we can see that out model tends to pick out more points (presented as red dots) in the drifting parts of the curve, which is reasonable since these parts are harder to reconstruct.

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

An ANP is then used to reconstruct the remaining pixels from a set of context elements (selected subset in our case).

Figure 3(b) shows that our model significantly outperforms ANP with RS (as in the original ANP paper) and the LTS baseline. Figure 6 shows the reconstruction samples of our model which are visually better than the reconstruction of the baselines for the same number of pixels.

The goal is to learn to select a subset for a target task such as classification or regression.

Table 1 shows that using only 500 pixels (\(\sim\)1.3% of total pixels in an image), we can achieve a mean AUC of 0.9093 (99.3% of the accuracy obtained with the full image).

The AUC with selected pixels learned from our SSS is significantly higher than that of the random pixels baseline, showing the effectiveness of our subset selection method.

Our model outperforms this variant, showing the effectiveness of training with the target task.

Since our method is stochastic, the predictive distribution can be written as \(\mathbb{E}_{p(D_{s}|D)}\left[p_{\theta}(y_{D}|D_{s})\right]\), and we can use Monte Carlo sampling to get the prediction in practice. However, throughout the experiment section, we only reported the result with one sampled subset, since it gives the best reduction in memory and computational cost.

We compare it against another variant: SSS(MC) with MC sampling (5 samples). It should be noted that by doing MC sampling with 5 samples, the computational cost (inference) is increased by 5 times, and the memory requirement can be increased by up to 5 times too. Table 1 shows that our model achieves comparable performance with that variant, thus justifying that it can achieve good performance for target tasks, while reducing memory and computation requirement.

On this task, our subset selection module is trained via the procedure detailed in Section 3.6 on instance selection. To evaluate the effectiveness of the SSS model, we evaluate the model in terms of the diversity in the selected subset using the Frechet Inception Distance(FID Score) (Heusel et al., 2017) which measures the similarity and diversity between two datasets.

Also for small sample selection, our method outperforms FPS and Random Sampling significantly since our method is able to model the interactions within the full dataset and hence can select the most representative subset.

In this paper, we have proposed a stochastic subset selection method to reduce the size of an arbitrary set while preserving performance on a target task. Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account. Hence, it can select a compact set that avoids samples with redundant information. By using the compact subset in place of the original set for a target task, we can save memory, communication and computational cost.

The idea of the greedy training algorithm is to train the auto-regressive model to select the best next \(q\) elements from the candidate set to minimize the target loss on the selected samples. By doing this, we do not have to run the auto-regressive model \(k/q\) time during training, thus reducing the computational cost.

5: a target task with loss function \(\ell(\cdot,\cdot)\) 6:trained model with converged \(\theta\) and \(\phi\) 7:\(\theta,\phi\leftarrow\) initialization 8:while not converged do 9: Sample a minibatch with \(m\) sets \(D^{(1)},D^{(2)},\ldots,D^{(m)}\) from \(p(D)\) 10:\(D^{(j)}_{\theta}\sim p(D^{(j)}_{\theta}|D^{(j)})\) for \(j=1\ldots m\) 11:\(i\sim\) random sample from \((0,\ldots,k-q)\) 12:\(I^{(j)}\sim\) random \(i\)-element subset of \(D^{(j)}_{\theta}\) for \(j=1\ldots m\) 13:\(Q^{(j)}\sim\) select a \(q\)-element subset from \(D^{(j)}_{\theta}\setminus I^{(j)}\) (with the auto-regressive model) 14:\(\theta\leftarrow\theta-\alpha\nabla_{\theta}\frac{1}{m}\sum_{j=1}^{m}\ell( \cdot,I^{(j)}\cup Q^{(j)}),\phi\leftarrow\phi-\alpha\nabla_{\phi}\frac{1}{m} \sum_{j=1}^{m}\ell(\cdot,I^{(j)}\cup Q^{(j)})\)

Further, we provide qualitative results on a single dataset in Figure 10 where we show how our model picks 5 instances from the full set of 200 images face images.

SSS consists of \(r(D)\), \(\rho(d_{i},r(D))\) and \(f(d,D_{c},D_{s}^{(i-1)}\). We describe the models in this section. For all experiments, \(r(D)\) is implemented as DeepSets. This means that we take the mean of all the samples in a set to obtain the set representation. \(\rho(d_{i},r(D))\) is implemented as a neural network with the following specifications: there are 3 Linear layers each followed by ReLU activation. Also, all inputs are projected into feature space using 3 Linear layers, each followed by ReLU activation.

Input: Figure 5: 1D Function Reconstruction.

Input: Figure 10: Visualization of a set with 200 images for instance selection. The two stage selection method in SSS is visualized as Candidate Set and SSS. A coreset of size 5 is visualized.

Input: Figure 6: CelebA reconstruction samples with varying number of pixels.

Input: Figure 3: Graphical Models: (a) Feature Selection for Reconstruction. (b) Feature Selection for Prediction Task. (c) Instance Selection model. (d) Instance Selection model for Classification

Input: Figure 4: Models’ performance on set reconstruction and classification task (lighter-color areas present the standard deviations). (a) 1D Function reconstruction. (b) CelebA (reconstruction). (c) CelebA (classification)

Input: Figure 11: Sample visualization of prototype selection for the miniImagenet dataset on the few-shot classification task. Each row represents a set that corresponds to the support from which a prototype is selected for the few-shot classification task.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input: Figure 8: Reconstruction samples of 1D functions with different selection methods

Input: Figure 9: Selected pixels for different tasks on CelebA.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, I'll evaluate the claim: "We propose a novel two-stage stochastic subset selection method that learns to sample a subset from a larger set with linear time complexity, with minimal loss of accuracy at the downstream task."

The evidence strongly supports this claim through:
- Mathematical formulation showing two-stage selection (Bernoulli + Categorical)
- Complexity analysis showing O(n)+O(k²d/q) which is linear in input size n
- Experimental results demonstrating minimal accuracy loss (e.g., 99.3% of full image accuracy using only 1.3% of pixels)
- Reviewer confirmation of good empirical results and clear applicability
- Visual examples showing effective subset selection across multiple tasks

The claim appears well-stated and accurately reflects the evidence without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We propose a novel two-stage stochastic subset selection method that **learns to sample** a subset from a larger set with linear time complexity, with minimal loss of accuracy at the downstream task.

The review comment to be evaluated is:

This work introduces a method to select instances from any set (stochastic subset selection, or SSS). The experiments demonstrate a diverse set of use-cases, including feature selection and core-set selection. The proposed approach is a two-stage method involving candidate selection (learning a function $\rho$ to determine a Bernoulli probability for each input) and AutoRegressive subset selection (learning a function $f$ to generate probabilities for sampling elements from a reduced set); both stages use the Concrete distribution to ensure differentiability.

The paper includes a broad set of experiments, but I have some questions and concerns regarding the baselines and the proposed SSS method.

1. Using the Concrete distribution enables differentiability, but Algorithm 1 seems to implement the candidate selection and AutoRegressive Subset selection steps in ways that eliminate differentiability. Line 4 assigns elements to $D_c$ based on $z_i$, but when is $z_i$ differentiated? Similarly, $Q$ is selected based on sampling using the $p_i$ parameters (line 13), but at what point are the samples differentiated after assigning elements to $Q$? The method apparently works so I'm sure there's a good answer, but it is not well explained.

2. Some aspects of SSS seem arbitrary, and the authors could make a more compelling case for this particular approach with a more serious ablation study. A couple ideas: (i) Running SSS just using the first stage (candidate selection). (ii) Running SSS using the second stage directly (if it's not too slow). (iii) Removing aspects of the models, such as removing $r(D)$ as an input to $\rho$, or $D_C$ as an input to $f$.

3. As a further baseline for SSS, could the authors simply have implemented an algorithm that learns separate Bernoulli probabilities for each example (without the model $\rho$)? This seems far simpler than SSS and it would be interesting to see how it compares.

4. When attempting to constrain the size of $D_C$, why use a KL divergence penalty and not simply penalize each $z_i$'s probability, or the sampled $z_i$'s themselves? The latter could be viewed as a version of an $\ell_0$ penalty on the subset size $|D_C|$, and both would remove the need to choose a hyperparameter for $r(Z)$.

5. Obtaining $q$ examples by independently sampling from Bernoulli distributions with probabilities given by $q \times p_j$ is odd. You can't guarantee that you get $q$ samples, and you can't even guarantee that $q \times p_j$ is a valid probability. There must be a better solution to this problem; [1, 2] both use clever sampling tricks with the Concrete distribution, perhaps they suggest a better way.

6. The paper is missing a couple citations and comparisons when it comes to feature selection using the Concrete distribution. Concrete Autoencoders (CAEs) [1] are a method for performing differentiable feature selection using a similar trick with the Concrete distribution; while CAEs are used for global feature selection (the same features for every instance), Learning to Explain (L2X) [2] is a method for instance-wise feature selection that also uses the Concrete distribution. Both should be mentioned, and L2X could be used a comparison in certain experiments.

The motivation around reducing communication overhead and allowing for faster training is interesting, but these claims could be more effectively grounded in the experiments performed in the paper. When accounting for the overhead involved in running SSS, is this method faster than conventional training, and does it require less communication overhead? The clearest application from these experiments is feature selection, but the comparisons with baselines are not extensive.

Overall, the method seems promising but I hope the authors can clarify several points and offer more competitive baselines.

[1] Abid et al. "Concrete Autoencoders for Differentiable Feature Selection and Reconstruction" (2019)

[2] Chen et al. "Learning to Explain: An Information-Theoretic Perspective on Model Interpretation" (2018)

The evidence to be evaluated is:Firstly, our model can be applied to any type of sets (and instances represented as sets), while the aforementioned models mainly work for images represented in tensor form. Furthermore, our method can be applied both at the instance and dataset level.

Our instance-sampling model is also related to the Variational Auto Encoder (VAE) (Kingma and Welling, 2013). However, while VAE learns a compact representation of a _data point_, our model learns a compact representation of a _set_.

Our method also falls in this category.

In this work, we consider data of the type \(D=\{d_{1},\ldots,d_{n}\}\) where individual \(d_{i}\)'s are possibly represented as _input_\(x_{i}\) and _target_\(y_{i}\). \(D\) is the complete set and we assume that within \(D\), there exists a subset \(D_{s}=\{s_{i},\ldots,s_{k}\}\subset D\) such that \(k\ll n\) and that for an arbitrarily defined loss function \(\ell(.,D)\) that we are interested in optimizing over the full set \(D\), \(D_{s}\) can be used as a proxy for \(D\) such that \(\ell(.,D)\approx\ell(.,D_{s})\). In what follows, we present a method that learns the conditional distribution \(p(D_{s}|D)\) of the subset \(D_{s}\) via a two stage selection procedure dubbed _candidate selection_ and _autoregressive subset selection_.

In essence, we seek to construct a subset \(D_{s}\) that is optimally representative of the full set \(D\) w.r.t \(\ell(.)\).

As such, we first present the _candidate selection_ procedure used to construct a smaller set, \(D_{c}\), without considering inter-sample dependencies. This is then followed by the _autoregressive subset selection_ procedure used to construct \(D_{s}\) from \(D_{c}\) by modeling inter-sample dependencies.

We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\). We implement both \(r(D)\) and \(\rho(d_{i},r(D))\) as neural networks and specifically for \(r(D)\), we use Deep Sets (Zaheer et al., 2017).

Specifically, to sample \(z_{i}\), we execute the following computational routine: \[z_{i}=\sigma\Big{(}\frac{1}{\tau}\Big{(}\log\frac{\pi_{i}}{1-\pi_{i}}+\log \frac{u}{1-u}\Big{)}\Big{)},\quad\pi_{i}=\rho(d_{i},r(D)),\quad u\sim\text{ Unif}(0,1), \tag{2}\] where \(\sigma\) is the Sigmoid function, \(\tau\) is the temperature for the continuous relaxation and \(u\) is sampled from the uniform distribution. \(\tau\) is set to \(0.05\) in all our experiments.

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

If \(f\) considers the pairwise interactions between all candidate elements and the selected elements, the inference complexity is \(O(n)+O(k^{2}d/q)\) where \(n,d,k\) correspond to \(|D|,|D_{c}|\) and \(|D_{s}|\) respectively. In our experiments, for the choice of the function \(f\), we utilize either a Set Transformer(Lee et al., 2018) or DeepSets(Zaheer et al., 2017) to model the pairwise interactions between the elements of a given set.

11:procedureAutoSelect(\(q,D_{s},D_{c}\)) 12:\(C=\{w_{1},w_{2},\dots,w_{m}\}\gets D_{c}\setminus D_{s}\) 13:\((p_{1},p_{2},\dots,p_{m})\leftarrow(f(w_{1},D_{c},D_{s}),f(w_{2},D_{c},D_{s}), \dots,f(w_{m},D_{c},D_{s}))\) 14:\((p_{1},p_{2},\dots,p_{m})\leftarrow(p_{1},p_{2},\dots,p_{m})/\sum_{j=1}^{m}p_{j}\) 15:\(Q\leftarrow\text{Select }q\) elements from \(C\) with probability \((p_{1},p_{2},\dots,p_{m})\)

For computational efficiency, we may desire to restrict the size of \(D_{c}\) to save computational cost when constructing \(D_{s}\). We adopt the idea of Information Bottleneck and constrain the distribution of \(Z\) for \(D_{c}\).

\[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]+\beta\text{KL}[p(Z|D) ||r(Z)]] \tag{4}\]where \(r(Z)\) is a sparse prior. In our experiments, we set the parameter of the Bernoulli spares prior \(r(Z)\) to either \(0.1\) or \(0.01\) for different levels of sparsity and \(\beta\) is set to \(1.0\) or \(0.001\).

\[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[-\log p_{\theta}(Y|X,D_{s})]+\beta \text{KL}[p(Z|D)||r(Z)]] \tag{5}\] Minimizing this objective ensures that we learn a compact subset(\(D_{s}\)) most representative of \(D\) and \(D_{s}\) can then be used for other tasks.

For a single dataset \(D_{i}\in\mathcal{D}\), we apply the subset construction method already described to a \(D_{s}\) that can be used to reconstruct all the elements in \(D_{i}\). In essence, \(D_{i}\) is _distilled_ into a new dataset \(D_{s}\) with \(k<|D_{i}|\) elements.

As a first step, we represent \(D_{s}\) as a unique representative vector \(c\) for each element in the dataset akin to the statistics network used in the Neural Statistician (Edwards and Storkey, 2016) model. Specifically, to reconstruct an element \(d_{i}\in D_{i}\) given \(D_{s}\), \(c\) is computed by applying a stochastic cross-attention mechanism on \(D_{s}\) where the stochasticity is supplied by a query \(\alpha\) which is computed using \(d_{i}\).

In summary, the purpose of the generative model introduced is to train the subset selection module for the instance selection task.

Finally in the dataset distillation task, we consider the problem of selecting prototypes to be used for few-shot classification. Here, we adopt Prototypical Networks (Snell et al., 2017) and apply the subset selection model to the task of selecting representative prototypes from each class to be used for classifying new instances. By learning to select the prototypes, we can remove outliers that would otherwise change the class prediction boundaries in the classification task.

We train our model which consists of the subset selection model \(p(D_{s}|D)\) and a task network \(p(Y|X,D_{s})\) (e.g. ANP), on this data set and report the reconstruction loss, which is the negative log-likelihood. SSS out-performs Random Select (RS), verifying that the subset selection model \(p(D_{s}|D)\) can learn a meaningful distribution for the selected elements. Our model also out-performs the Learning to Sample (LTS) baseline. Through the visualization of the selected points in Figure 5, we can see that out model tends to pick out more points (presented as red dots) in the drifting parts of the curve, which is reasonable since these parts are harder to reconstruct.

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

An ANP is then used to reconstruct the remaining pixels from a set of context elements (selected subset in our case).

Figure 3(b) shows that our model significantly outperforms ANP with RS (as in the original ANP paper) and the LTS baseline. Figure 6 shows the reconstruction samples of our model which are visually better than the reconstruction of the baselines for the same number of pixels.

The goal is to learn to select a subset for a target task such as classification or regression.

Table 1 shows that using only 500 pixels (\(\sim\)1.3% of total pixels in an image), we can achieve a mean AUC of 0.9093 (99.3% of the accuracy obtained with the full image).

The AUC with selected pixels learned from our SSS is significantly higher than that of the random pixels baseline, showing the effectiveness of our subset selection method.

Our model outperforms this variant, showing the effectiveness of training with the target task.

Since our method is stochastic, the predictive distribution can be written as \(\mathbb{E}_{p(D_{s}|D)}\left[p_{\theta}(y_{D}|D_{s})\right]\), and we can use Monte Carlo sampling to get the prediction in practice. However, throughout the experiment section, we only reported the result with one sampled subset, since it gives the best reduction in memory and computational cost.

We compare it against another variant: SSS(MC) with MC sampling (5 samples). It should be noted that by doing MC sampling with 5 samples, the computational cost (inference) is increased by 5 times, and the memory requirement can be increased by up to 5 times too. Table 1 shows that our model achieves comparable performance with that variant, thus justifying that it can achieve good performance for target tasks, while reducing memory and computation requirement.

On this task, our subset selection module is trained via the procedure detailed in Section 3.6 on instance selection. To evaluate the effectiveness of the SSS model, we evaluate the model in terms of the diversity in the selected subset using the Frechet Inception Distance(FID Score) (Heusel et al., 2017) which measures the similarity and diversity between two datasets.

Also for small sample selection, our method outperforms FPS and Random Sampling significantly since our method is able to model the interactions within the full dataset and hence can select the most representative subset.

In this paper, we have proposed a stochastic subset selection method to reduce the size of an arbitrary set while preserving performance on a target task. Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account. Hence, it can select a compact set that avoids samples with redundant information. By using the compact subset in place of the original set for a target task, we can save memory, communication and computational cost.

The idea of the greedy training algorithm is to train the auto-regressive model to select the best next \(q\) elements from the candidate set to minimize the target loss on the selected samples. By doing this, we do not have to run the auto-regressive model \(k/q\) time during training, thus reducing the computational cost.

5: a target task with loss function \(\ell(\cdot,\cdot)\) 6:trained model with converged \(\theta\) and \(\phi\) 7:\(\theta,\phi\leftarrow\) initialization 8:while not converged do 9: Sample a minibatch with \(m\) sets \(D^{(1)},D^{(2)},\ldots,D^{(m)}\) from \(p(D)\) 10:\(D^{(j)}_{\theta}\sim p(D^{(j)}_{\theta}|D^{(j)})\) for \(j=1\ldots m\) 11:\(i\sim\) random sample from \((0,\ldots,k-q)\) 12:\(I^{(j)}\sim\) random \(i\)-element subset of \(D^{(j)}_{\theta}\) for \(j=1\ldots m\) 13:\(Q^{(j)}\sim\) select a \(q\)-element subset from \(D^{(j)}_{\theta}\setminus I^{(j)}\) (with the auto-regressive model) 14:\(\theta\leftarrow\theta-\alpha\nabla_{\theta}\frac{1}{m}\sum_{j=1}^{m}\ell( \cdot,I^{(j)}\cup Q^{(j)}),\phi\leftarrow\phi-\alpha\nabla_{\phi}\frac{1}{m} \sum_{j=1}^{m}\ell(\cdot,I^{(j)}\cup Q^{(j)})\)

Further, we provide qualitative results on a single dataset in Figure 10 where we show how our model picks 5 instances from the full set of 200 images face images.

SSS consists of \(r(D)\), \(\rho(d_{i},r(D))\) and \(f(d,D_{c},D_{s}^{(i-1)}\). We describe the models in this section. For all experiments, \(r(D)\) is implemented as DeepSets. This means that we take the mean of all the samples in a set to obtain the set representation. \(\rho(d_{i},r(D))\) is implemented as a neural network with the following specifications: there are 3 Linear layers each followed by ReLU activation. Also, all inputs are projected into feature space using 3 Linear layers, each followed by ReLU activation.

Input: Figure 5: 1D Function Reconstruction.

Input: Figure 10: Visualization of a set with 200 images for instance selection. The two stage selection method in SSS is visualized as Candidate Set and SSS. A coreset of size 5 is visualized.

Input: Figure 6: CelebA reconstruction samples with varying number of pixels.

Input: Figure 3: Graphical Models: (a) Feature Selection for Reconstruction. (b) Feature Selection for Prediction Task. (c) Instance Selection model. (d) Instance Selection model for Classification

Input: Figure 4: Models’ performance on set reconstruction and classification task (lighter-color areas present the standard deviations). (a) 1D Function reconstruction. (b) CelebA (reconstruction). (c) CelebA (classification)

Input: Figure 11: Sample visualization of prototype selection for the miniImagenet dataset on the few-shot classification task. Each row represents a set that corresponds to the support from which a prototype is selected for the few-shot classification task.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input: Figure 8: Reconstruction samples of 1D functions with different selection methods

Input: Figure 9: Selected pixels for different tasks on CelebA.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find this claim is somewhat overstated based on several key points:

1. While the method does achieve linear time complexity (O(n) + O(k^2d/q)), the paper doesn't thoroughly demonstrate "minimal loss of accuracy" across all tasks. Some experiments show good performance (e.g., 99.3% accuracy with 1.3% pixels), but comprehensive comparisons and ablation studies were noted as missing by the reviewer.

2. The claim of "learning to sample" implies full differentiability, but the reviewer highlights potential issues with differentiation in Algorithm 1's implementation that aren't well explained.

3. The evidence shows the method works but doesn't definitively prove it maintains minimal accuracy loss universally or conclusively demonstrate learning occurs throughout the sampling process.

<score>3</score>