Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Specifically, we learn the sampling rate for individual samples in two stages.

The review comment to be evaluated is:

### Summary

The authors present a stochastic algorithm for selecting a subset of a large dataset, while trying to preserve statistics from the original dataset. The algorithm is a two-step process, first selecting a set of "candidates" based on individual features, then filtering to a final subset which may account for interactions between candidate items.

### Strengths

The paper provides empirical experimental results which suggest the superiority of the proposed method against two baselines: Learning to Sample (LTS) and random selection (RS).

Very few grammatical mistakes.

### Concerns and Issues that Need Clarification

1. A significant amount of active learning literature deals with subset selection. However, this paper fails to discuss or even cite any results from the active learning literature. It would be helpful for the authors to compare/contrast their proposed method with existing active learning results. For example, consider
   - [*Active Learning for Convolutional Neural Networks: A Core-Set Approach* by Sener and Savarese (2018)](https://openreview.net/forum?id=H1aIuk-RW)
   - [*Selection via Proxy: Efficient Data Selection for Deep Learning* by Coleman et al. (2019)](https://openreview.net/forum?id=HJg2b0VYDr)
   - [*Submodularity in Data Subset Selection and Active Learning* by Wei et al. (2015)](http://proceedings.mlr.press/v37/wei15.html)
   - [*On Statistical Bias In Active Learning: How and When to Fix It* (2020)](https://openreview.net/forum?id=JiYq3eqTKY) - this paper is a submission to ICLR 2020, so obviously there is no expectation that the authors would be familiar with this work. However, I bring this paper up because it discusses a similar topic of trying to select data points from a larger dataset that provide a reasonable estimate of a loss function on the "population" vs. a "sample."

2. The algorithm, as described, states that individual elements of a dataset $D$ are "possibly represented" as a pair $(x_i, y_i)$. What does "possibly represented" mean? Assuming that the dataset elements are given as a $(x_i, y_i)$ pair, the stochastic subset selection algorithm seems to describe instance selection. However, the paper also claims that the algorithm works for feature selection. What are $D$, $x_i$, and $y_i$ in the feature selection tasks? The paper makes an attempt at clarifying this by stating, "Here, x is 2-dimensional and y is 3-dimensional for RGB images." But this still leaves the reader clueless - why is $x$ only 2-dimensional? What are those 2 dimensions?

3. The end of the introduction claims that the proposed subset selection method "learns to sample a subset from a larger set with linear time complexity," yet Section 3.4 claims that the inference complexity is $O(n) + O(k^2 d/q)$, which is nonlinear if either $k$ or $d$ is chosen to be dependent on $n$. Could the authors please clarify?

4. The authors claim that their method uses meta-learning, but no mention of meta learning is made beyond the "Preliminaries" subsection of the "Approach" section. How is this method a example of meta-learning?

5. The paper provides experimental evidence that the proposed method outperforms other subset selection baselines, such as Learning to Select (LTS) and random selection (RS). However, the paper provides little (if any) intuition or formal justification for why the proposed method performs better. The authors need to explain the motivation behind the proposed algorithm.

6. The writing and figures lack clarity and are often difficult to follow. Some examples:
   - The context for the mathematical expression in (4) is very unclear. Is (4) an objective function? If so, is this expression supposed to be minimized or maximized? And what is the random variable $Z$ supposed to represent? And why is there a period in the middle of the expression?
   - Figure 2 is poorly explained and difficult to understand. I'm assuming that the blue vertical vector is supposed to represent $D_c$. Is that correct? And are the dark shaded boxes supposed to represent the "candidates" and the lightly shaded boxes supposed to represent the items that were not selected? As a reader, I should not have to guess what the colors mean. They should be explained clearly in the caption, or at the very least, in the paper text. Furthermore, it seems to me (again it's never made explicit) that Figure 2b is depicting the Autoregressive Subset Selection ("AutoSelect") routine. In this case, why is there a white row vector of $D = d_1, \dotsc, d_n$? In Algorithm 1, it seems as though AutoSelect only takes as input $D_s$ and $D_c$, which are subsets of $D$ and not $D$ itself.

### Original Rating and Confidence

**Rating** - 4: Ok but not good enough - rejection

**Confidence** - 4: The reviewer is confident but not absolutely certain that the evaluation is correct

### Updated Review after Author Response

Thank you for the thoughtful responses to my questions, and apologies for the delay in updating my review. In general the responses have addressed my questions, and I have updated my rating accordingly. However, there are still several issues in my mind:

1. **Connection to active learning** - The authors correctly point out that active learning does not assume access to labels, whereas the proposed method does. However, I believe the comparison against selection algorithms such as k-center greedy should be included in the paper for completeness sake, and not just left as a comment in this forum.

2. **Dataset Distillation: Instance Selection** - Upon re-reading the paper carefully, I found this subsection in Section 3.6 to be confusing. Why are there multiple $D_i$? The section seems to only talk about one $D_i$.

3. **Set vs. Tensor**: In the new "representation learning" subsection of Section 2, the paper stresses that the model learns a representation of a set, instead of an individual data point. However, many tasks (e.g., image reconstruction) do impose some sort of ordering. The paper lacks clarity about where permutation-invariance is used vs. not used.

4. **Other places that need clarification**:
  - Equation (2): presumably you threshold the output of the sigmoid function, right? Because in Algorithm 1, it seems like $z_i$ are supposed to be boolean-valued.
  - Appendix: there are several missing closing parentheses

The evidence to be evaluated is:In what follows, we present a method that learns the conditional distribution \(p(D_{s}|D)\) of the subset \(D_{s}\) via a two stage selection procedure dubbed _candidate selection_ and _autoregressive subset selection_.

As such, we first present the _candidate selection_ procedure used to construct a smaller set, \(D_{c}\), without considering inter-sample dependencies. This is then followed by the _autoregressive subset selection_ procedure used to construct \(D_{s}\) from \(D_{c}\) by modeling inter-sample dependencies.

We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

4:\((\pi_{1},\pi_{2},\dots,\pi_{n})\leftarrow(\rho(d_{1},r(D)),\dots,\rho(d_{n},r(D)))\) 5:\(z_{i}\sim\text{Ber}(\pi_{i})\) for \(i=1,\dots,n\).

8:for\(i=1,\dots,k/q\)do\(\triangleright\) AutoRegressive Subset Selection 9:\(D_{s}\gets D_{s}\cup\text{AutoSelect}(q,D_{s},D_{c})\)\(\triangleright\) Select \(q\) elements 10:return\(D_{s}\)

Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account.

12:\(I^{(j)}\sim\) random \(i\)-element subset of \(D^{(j)}_{\theta}\) for \(j=1\ldots m\) 13:\(Q^{(j)}\sim\) select a \(q\)-element subset from \(D^{(j)}_{\theta}\setminus I^{(j)}\) (with the auto-regressive model)

Input: Figure 10: Visualization of a set with 200 images for instance selection. The two stage selection method in SSS is visualized as Candidate Set and SSS. A coreset of size 5 is visualized.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "Specifically, we learn the sampling rate for individual samples in two stages."

1. Summary of evaluation:
The evidence strongly supports this claim through multiple sources:
- The text explicitly describes a two-stage selection process: candidate selection followed by autoregressive subset selection
- Figure 2 visually demonstrates the two stages with (a) Candidate Selection using Bernoulli distribution and (b) Core Subset Selection using Categorical distributions
- The algorithmic details show first selecting candidates (lines 4-5) and then performing autoregressive subset selection (lines 8-10)
- The figure shows the progression from full set to candidate set to final coreset

The claim is well-stated and accurately reflects the methodology without exaggeration. It clearly maps to the implemented two-stage sampling process demonstrated in both text and figures.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

First we learn a Bernoulli sampling rate for individual sample to efficiently screen out less important elements.

The evidence to be evaluated is:We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\).

Specifically, to sample \(z_{i}\), we execute the following computational routine: \[z_{i}=\sigma\Big{(}\frac{1}{\tau}\Big{(}\log\frac{\pi_{i}}{1-\pi_{i}}+\log \frac{u}{1-u}\Big{)}\Big{)},\quad\pi_{i}=\rho(d_{i},r(D)),\quad u\sim\text{ Unif}(0,1), \tag{2}\] where \(\sigma\) is the Sigmoid function, \(\tau\) is the temperature for the continuous relaxation and \(u\) is sampled from the uniform distribution. \(\tau\) is set to \(0.05\) in all our experiments.

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

4:\((\pi_{1},\pi_{2},\dots,\pi_{n})\leftarrow(\rho(d_{1},r(D)),\dots,\rho(d_{n},r(D)))\) 5:\(z_{i}\sim\text{Ber}(\pi_{i})\) for \(i=1,\dots,n\). 6:\(D_{c}\leftarrow\{d_{i}\text{ for }i=1:n\text{ if }z_{i}\}\)\(\triangleright\) Candidate Selection

13:\((p_{1},p_{2},\dots,p_{m})\leftarrow(f(w_{1},D_{c},D_{s}),f(w_{2},D_{c},D_{s}), \dots,f(w_{m},D_{c},D_{s}))\) 14:\((p_{1},p_{2},\dots,p_{m})\leftarrow(p_{1},p_{2},\dots,p_{m})/\sum_{j=1}^{m}p_{j}\) 15:\(Q\leftarrow\text{Select }q\) elements from \(C\) with probability \((p_{1},p_{2},\dots,p_{m})\)

For computational efficiency, we may desire to restrict the size of \(D_{c}\) to save computational cost when constructing \(D_{s}\).

\[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]+\beta\text{KL}[p(Z|D) ||r(Z)]] \tag{4}\]where \(r(Z)\) is a sparse prior. In our experiments, we set the parameter of the Bernoulli spares prior \(r(Z)\) to either \(0.1\) or \(0.01\) for different levels of sparsity and \(\beta\) is set to \(1.0\) or \(0.001\).

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

An ANP is then used to reconstruct the remaining pixels from a set of context elements (selected subset in our case).

Table 1 shows that using only 500 pixels (\(\sim\)1.3% of total pixels in an image), we can achieve a mean AUC of 0.9093 (99.3% of the accuracy obtained with the full image).

The AUC with selected pixels learned from our SSS is significantly higher than that of the random pixels baseline, showing the effectiveness of our subset selection method.

Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account. Hence, it can select a compact set that avoids samples with redundant information.

The idea of the greedy training algorithm is to train the auto-regressive model to select the best next \(q\) elements from the candidate set to minimize the target loss on the selected samples. By doing this, we do not have to run the auto-regressive model \(k/q\) time during training, thus reducing the computational cost.

12:\(I^{(j)}\sim\) random \(i\)-element subset of \(D^{(j)}_{\theta}\) for \(j=1\ldots m\) 13:\(Q^{(j)}\sim\) select a \(q\)-element subset from \(D^{(j)}_{\theta}\setminus I^{(j)}\) (with the auto-regressive model) 14:\(\theta\leftarrow\theta-\alpha\nabla_{\theta}\frac{1}{m}\sum_{j=1}^{m}\ell( \cdot,I^{(j)}\cup Q^{(j)}),\phi\leftarrow\phi-\alpha\nabla_{\phi}\frac{1}{m} \sum_{j=1}^{m}\ell(\cdot,I^{(j)}\cup Q^{(j)})\)

Input: Figure 10: Visualization of a set with 200 images for instance selection. The two stage selection method in SSS is visualized as Candidate Set and SSS. A coreset of size 5 is visualized.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input: Figure 9: Selected pixels for different tasks on CelebA.

Input: Figure 6: CelebA reconstruction samples with varying number of pixels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim "First we learn a Bernoulli sampling rate for individual sample to efficiently screen out less important elements" against the provided evidence, I find this claim is well-stated and accurately reflects the methodology described in the evidence.

The evidence clearly shows that:
1. The method uses a Bernoulli process for candidate selection, as demonstrated in equations (1) and (2)
2. The process is efficient, using a single-pass sampling method
3. Figure 2 visually confirms the Bernoulli-based candidate selection process
4. The screening effectiveness is demonstrated through experimental results (like the 99.3% accuracy with only 1.3% of pixels)

The claim avoids exaggeration and precisely describes the first stage of the two-stage selection process.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

First we learn a Bernoulli sampling rate for individual sample to efficiently screen out less important elements.

The review comment to be evaluated is:

**Final recommendation**
I support accepting this paper. While I am a bit disappointed the authors did not add in the paper the results they discussed during the rebuttal, I think the paper is interesting and clearer than at submission.

**Summary**
stochastic subset selection (SSS) is a method to learn to compress a set $D$ by selecting a subset $D_s$ such that the loss of a task performed on $D_s$ is as close as possible to the loss if the task had been performed on the original $D$. The paper show how this general method can be applied to several tasks. For example, $D$ can contain the pixels of an image and the task can be to reconstruct the original image from the subset. This corresponds to learning to compress. As another example, with the same $D$ but a label prediction task, SSS learns to dynamically select sparse features for a classification task. The other tasks considered are dataset distillation (compressing a data set) and selecting prototypes for few-shot classification. $D_s$ is constructed in two steps. A candidate set is first constructed by considering elements of $D$ independently and in a second step elements of $D_c$ are included in $D_s$ iteratively by considering other elements in $D_c$ and in $D_s$. Experiments are performed for one example of each task previously discussed and SSS is shown to outperform baselines. SSS is motivated by the need to reduce bandwidth, computation and memory footprint of deep learning in edge devices.

---
**Strong points**

1) Code is provided (I did not try to run it).

2) The proposed approach is applied to four different tasks. This convinced me of its applicability.

3) Empirical experiments seem well conducted (with some caveat, see weak point 3) and results are good.

4) The problems tackled are interesting.

**Weak points**

1) The paper is very dense, which makes it hard to follow. Related to that, I think that some parts of the paper are too concise to understand them well. The supplementary material partly compensates that.

2) While the experiments show that the proposed approach works well, I think that some experiments should compare full data pipelines. As an example, one experiment shows that using SSS to dynamically select pixels of an image and using this sparse representation as input leads to good classification accuracy. The paper implicitely suggests that this reduces inference cost if classification is performed on the device and bandwidth if inference is performed on another device. However, the cost of the pixel selection by SSS is never analyzed. I think comparing the memory footprint, the number of operations, bandwidth used and accuracy for all steps of the following pipelines would give a much clearer picture of the interest of the approach for deep learning on mobile devices:
    - classification is done directly on the device, using an optimized network.
    - pixels are randomly selected and classification is performed on the device.
    - pixels are randomly selected, transmitted and classification is performed on another device.
    - pixels are selected by SSS, which involves computing multiple embeddings, and classification is performed on the device.
    - pixels are selected by SSS, transmitted and classification is performed on another device.

---
**Recommendation**
Overall, I vote for accepting. I am on the fence with this paper. The current results are good and I think the proposed approach has some potential and will be interesting for the community. On the other hand the paper is hard to read and the experiments, while good to show the strength of the approach, do not cover usefullness for the mobile use case. I think the former can be adressed in time and that the latter, while important, does not prevent the paper to be published.

---
**Details**

I like that each task is first formally described and then evaluated in a specific case with the experiments. However I only understood most tasks when reading the experiment sections. I would suggest to either provide a short illustration of an actual instance of the task when describing the task formally, to directly provide the experimental result after describing the task or to direct the reader to the relevant experimental section(s) for illustration.

The paper states that model descriptions are available in appendix A, but I could not find them.

For the *Set Classification/Prediction* task, I would suggest clarifying that for this task a set typically contains the features of a single example, so $D=x$. I only understood this when reading the experimental section.

The *Dataset Distillation: Instance Selection* task was not clear to me. Here are a few points that puzzled me.
- The text suggest the proposed approach is applied to each $D_i$, but figure 3(c) has no $D_i$ and $i$ seems to be used as an index of $x$ rather than $D$. Furthermore, in figure 3(c) a single D_s is constructed whereas I was expecting one per $D_i$.
- The paper states that *It is important to note that $c$ is computed using only $D_i$ for every element in $D$*. However $c$ also depends on *a query $\alpha$ which is computed using $d_i$*, which, if I understood correctly, is the element of $D$ and can be outside $D_i$. So it seems that $c$ is not computed using only $D_i$.

For the *Dataset Distillation: Classification* task, in figure 3d, the labels are not observed. How can the model learn?

1) For this task, if the label is observed, how is this different from the first task?

2) For the Image Reconstruction experiment, why is $x$ 2 dimensional? Are the inputs black and white images?

I think that the image reconstruction experiment should include a comparison to compression algorithms. I agree with the paper that the proposed approach is applicable to many other types of data than images, but it would be very interesting to see how the approach compares to expert defined methods. Furthermore, implementation in relation to mobile devices is the motivation of this work and as far as I know classical compression algorithms are still used on mobile devices.

---
**Questions**

Questions 1 and 2 in *details* above.

I would also be interested to hear authors' opinions on weak point 2 and a comparison to compression.

For classification, pixels not selected are set to 0. Wouldn't the inference cost be the same when using convolutional networks?

---
**Minor details**

For the *set reconstruction* task, would it also be interesting to learn a model on $D_s$ and use it to predict $Y$ from $X$?

*typos*
foro

The evidence to be evaluated is:We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\).

Specifically, to sample \(z_{i}\), we execute the following computational routine: \[z_{i}=\sigma\Big{(}\frac{1}{\tau}\Big{(}\log\frac{\pi_{i}}{1-\pi_{i}}+\log \frac{u}{1-u}\Big{)}\Big{)},\quad\pi_{i}=\rho(d_{i},r(D)),\quad u\sim\text{ Unif}(0,1), \tag{2}\] where \(\sigma\) is the Sigmoid function, \(\tau\) is the temperature for the continuous relaxation and \(u\) is sampled from the uniform distribution. \(\tau\) is set to \(0.05\) in all our experiments.

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

4:\((\pi_{1},\pi_{2},\dots,\pi_{n})\leftarrow(\rho(d_{1},r(D)),\dots,\rho(d_{n},r(D)))\) 5:\(z_{i}\sim\text{Ber}(\pi_{i})\) for \(i=1,\dots,n\). 6:\(D_{c}\leftarrow\{d_{i}\text{ for }i=1:n\text{ if }z_{i}\}\)\(\triangleright\) Candidate Selection

13:\((p_{1},p_{2},\dots,p_{m})\leftarrow(f(w_{1},D_{c},D_{s}),f(w_{2},D_{c},D_{s}), \dots,f(w_{m},D_{c},D_{s}))\) 14:\((p_{1},p_{2},\dots,p_{m})\leftarrow(p_{1},p_{2},\dots,p_{m})/\sum_{j=1}^{m}p_{j}\) 15:\(Q\leftarrow\text{Select }q\) elements from \(C\) with probability \((p_{1},p_{2},\dots,p_{m})\)

For computational efficiency, we may desire to restrict the size of \(D_{c}\) to save computational cost when constructing \(D_{s}\).

\[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]+\beta\text{KL}[p(Z|D) ||r(Z)]] \tag{4}\]where \(r(Z)\) is a sparse prior. In our experiments, we set the parameter of the Bernoulli spares prior \(r(Z)\) to either \(0.1\) or \(0.01\) for different levels of sparsity and \(\beta\) is set to \(1.0\) or \(0.001\).

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

An ANP is then used to reconstruct the remaining pixels from a set of context elements (selected subset in our case).

Table 1 shows that using only 500 pixels (\(\sim\)1.3% of total pixels in an image), we can achieve a mean AUC of 0.9093 (99.3% of the accuracy obtained with the full image).

The AUC with selected pixels learned from our SSS is significantly higher than that of the random pixels baseline, showing the effectiveness of our subset selection method.

Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account. Hence, it can select a compact set that avoids samples with redundant information.

The idea of the greedy training algorithm is to train the auto-regressive model to select the best next \(q\) elements from the candidate set to minimize the target loss on the selected samples. By doing this, we do not have to run the auto-regressive model \(k/q\) time during training, thus reducing the computational cost.

12:\(I^{(j)}\sim\) random \(i\)-element subset of \(D^{(j)}_{\theta}\) for \(j=1\ldots m\) 13:\(Q^{(j)}\sim\) select a \(q\)-element subset from \(D^{(j)}_{\theta}\setminus I^{(j)}\) (with the auto-regressive model) 14:\(\theta\leftarrow\theta-\alpha\nabla_{\theta}\frac{1}{m}\sum_{j=1}^{m}\ell( \cdot,I^{(j)}\cup Q^{(j)}),\phi\leftarrow\phi-\alpha\nabla_{\phi}\frac{1}{m} \sum_{j=1}^{m}\ell(\cdot,I^{(j)}\cup Q^{(j)})\)

Input: Figure 10: Visualization of a set with 200 images for instance selection. The two stage selection method in SSS is visualized as Candidate Set and SSS. A coreset of size 5 is visualized.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input: Figure 9: Selected pixels for different tasks on CelebA.

Input: Figure 6: CelebA reconstruction samples with varying number of pixels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "First we learn a Bernoulli sampling rate for individual sample to efficiently screen out less important elements."

Based on the evidence provided:
The claim appears to be well-stated and accurately reflects the technical implementation described in the paper. The evidence shows a clear two-stage selection process where Bernoulli sampling is used in the first stage for candidate selection. This is demonstrated in both the mathematical formulation (Equations 1-2) and visually in Figure 2(a). The reviewer comments support that the method works well in practice. The claim avoids overstating the capabilities and clearly describes the first step of the selection process.

<score>1</score>

Justification:
1. The evidence directly supports the claim through:
- Equations 1-2 showing the Bernoulli sampling formulation
- Figure 2(a) visualizing the Bernoulli-based candidate selection
- Algorithm lines 4-6 detailing the Bernoulli sampling implementation
- Visual results in Figures 9-10 demonstrating effective element screening

2. The claim is precise and limited in scope:
- Only describes the first stage of selection
- Uses measured language ("learn", "sampling rate")
- Accurately characterizes the purpose ("screen out less important elements")
- Does not overstate performance or capabilities

3. The reviewer comments confirm the method's effectiveness while noting its good empirical performance, supporting that this is a well-grounded claim.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

First we learn a Bernoulli sampling rate for individual sample to efficiently screen out less important elements.

The review comment to be evaluated is:

This work introduces a method to select instances from any set (stochastic subset selection, or SSS). The experiments demonstrate a diverse set of use-cases, including feature selection and core-set selection. The proposed approach is a two-stage method involving candidate selection (learning a function $\rho$ to determine a Bernoulli probability for each input) and AutoRegressive subset selection (learning a function $f$ to generate probabilities for sampling elements from a reduced set); both stages use the Concrete distribution to ensure differentiability.

The paper includes a broad set of experiments, but I have some questions and concerns regarding the baselines and the proposed SSS method.

1. Using the Concrete distribution enables differentiability, but Algorithm 1 seems to implement the candidate selection and AutoRegressive Subset selection steps in ways that eliminate differentiability. Line 4 assigns elements to $D_c$ based on $z_i$, but when is $z_i$ differentiated? Similarly, $Q$ is selected based on sampling using the $p_i$ parameters (line 13), but at what point are the samples differentiated after assigning elements to $Q$? The method apparently works so I'm sure there's a good answer, but it is not well explained.

2. Some aspects of SSS seem arbitrary, and the authors could make a more compelling case for this particular approach with a more serious ablation study. A couple ideas: (i) Running SSS just using the first stage (candidate selection). (ii) Running SSS using the second stage directly (if it's not too slow). (iii) Removing aspects of the models, such as removing $r(D)$ as an input to $\rho$, or $D_C$ as an input to $f$.

3. As a further baseline for SSS, could the authors simply have implemented an algorithm that learns separate Bernoulli probabilities for each example (without the model $\rho$)? This seems far simpler than SSS and it would be interesting to see how it compares.

4. When attempting to constrain the size of $D_C$, why use a KL divergence penalty and not simply penalize each $z_i$'s probability, or the sampled $z_i$'s themselves? The latter could be viewed as a version of an $\ell_0$ penalty on the subset size $|D_C|$, and both would remove the need to choose a hyperparameter for $r(Z)$.

5. Obtaining $q$ examples by independently sampling from Bernoulli distributions with probabilities given by $q \times p_j$ is odd. You can't guarantee that you get $q$ samples, and you can't even guarantee that $q \times p_j$ is a valid probability. There must be a better solution to this problem; [1, 2] both use clever sampling tricks with the Concrete distribution, perhaps they suggest a better way.

6. The paper is missing a couple citations and comparisons when it comes to feature selection using the Concrete distribution. Concrete Autoencoders (CAEs) [1] are a method for performing differentiable feature selection using a similar trick with the Concrete distribution; while CAEs are used for global feature selection (the same features for every instance), Learning to Explain (L2X) [2] is a method for instance-wise feature selection that also uses the Concrete distribution. Both should be mentioned, and L2X could be used a comparison in certain experiments.

The motivation around reducing communication overhead and allowing for faster training is interesting, but these claims could be more effectively grounded in the experiments performed in the paper. When accounting for the overhead involved in running SSS, is this method faster than conventional training, and does it require less communication overhead? The clearest application from these experiments is feature selection, but the comparisons with baselines are not extensive.

Overall, the method seems promising but I hope the authors can clarify several points and offer more competitive baselines.

[1] Abid et al. "Concrete Autoencoders for Differentiable Feature Selection and Reconstruction" (2019)

[2] Chen et al. "Learning to Explain: An Information-Theoretic Perspective on Model Interpretation" (2018)

The evidence to be evaluated is:We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\).

Specifically, to sample \(z_{i}\), we execute the following computational routine: \[z_{i}=\sigma\Big{(}\frac{1}{\tau}\Big{(}\log\frac{\pi_{i}}{1-\pi_{i}}+\log \frac{u}{1-u}\Big{)}\Big{)},\quad\pi_{i}=\rho(d_{i},r(D)),\quad u\sim\text{ Unif}(0,1), \tag{2}\] where \(\sigma\) is the Sigmoid function, \(\tau\) is the temperature for the continuous relaxation and \(u\) is sampled from the uniform distribution. \(\tau\) is set to \(0.05\) in all our experiments.

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

4:\((\pi_{1},\pi_{2},\dots,\pi_{n})\leftarrow(\rho(d_{1},r(D)),\dots,\rho(d_{n},r(D)))\) 5:\(z_{i}\sim\text{Ber}(\pi_{i})\) for \(i=1,\dots,n\). 6:\(D_{c}\leftarrow\{d_{i}\text{ for }i=1:n\text{ if }z_{i}\}\)\(\triangleright\) Candidate Selection

13:\((p_{1},p_{2},\dots,p_{m})\leftarrow(f(w_{1},D_{c},D_{s}),f(w_{2},D_{c},D_{s}), \dots,f(w_{m},D_{c},D_{s}))\) 14:\((p_{1},p_{2},\dots,p_{m})\leftarrow(p_{1},p_{2},\dots,p_{m})/\sum_{j=1}^{m}p_{j}\) 15:\(Q\leftarrow\text{Select }q\) elements from \(C\) with probability \((p_{1},p_{2},\dots,p_{m})\)

For computational efficiency, we may desire to restrict the size of \(D_{c}\) to save computational cost when constructing \(D_{s}\).

\[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]+\beta\text{KL}[p(Z|D) ||r(Z)]] \tag{4}\]where \(r(Z)\) is a sparse prior. In our experiments, we set the parameter of the Bernoulli spares prior \(r(Z)\) to either \(0.1\) or \(0.01\) for different levels of sparsity and \(\beta\) is set to \(1.0\) or \(0.001\).

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

An ANP is then used to reconstruct the remaining pixels from a set of context elements (selected subset in our case).

Table 1 shows that using only 500 pixels (\(\sim\)1.3% of total pixels in an image), we can achieve a mean AUC of 0.9093 (99.3% of the accuracy obtained with the full image).

The AUC with selected pixels learned from our SSS is significantly higher than that of the random pixels baseline, showing the effectiveness of our subset selection method.

Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account. Hence, it can select a compact set that avoids samples with redundant information.

The idea of the greedy training algorithm is to train the auto-regressive model to select the best next \(q\) elements from the candidate set to minimize the target loss on the selected samples. By doing this, we do not have to run the auto-regressive model \(k/q\) time during training, thus reducing the computational cost.

12:\(I^{(j)}\sim\) random \(i\)-element subset of \(D^{(j)}_{\theta}\) for \(j=1\ldots m\) 13:\(Q^{(j)}\sim\) select a \(q\)-element subset from \(D^{(j)}_{\theta}\setminus I^{(j)}\) (with the auto-regressive model) 14:\(\theta\leftarrow\theta-\alpha\nabla_{\theta}\frac{1}{m}\sum_{j=1}^{m}\ell( \cdot,I^{(j)}\cup Q^{(j)}),\phi\leftarrow\phi-\alpha\nabla_{\phi}\frac{1}{m} \sum_{j=1}^{m}\ell(\cdot,I^{(j)}\cup Q^{(j)})\)

Input: Figure 10: Visualization of a set with 200 images for instance selection. The two stage selection method in SSS is visualized as Candidate Set and SSS. A coreset of size 5 is visualized.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input: Figure 9: Selected pixels for different tasks on CelebA.

Input: Figure 6: CelebA reconstruction samples with varying number of pixels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim "First we learn a Bernoulli sampling rate for individual sample to efficiently screen out less important elements."

This claim appears to be well-stated based on the available evidence. The paper clearly demonstrates a two-stage selection process where:
1. The first stage uses a Bernoulli process to select candidates (Equation 1 and 2)
2. The process is learnable through a differentiable function ρ
3. The method efficiently screens elements as shown in the visualizations in Figure 2 and experimental results
4. The screening is effective - for example, using only 1.3% of pixels achieves 99.3% accuracy

While there are some concerns raised in the review about implementation details, the fundamental claim about learning Bernoulli sampling rates is accurate and supported by the mathematical formulation and experimental results.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

First we learn a Bernoulli sampling rate for individual sample to efficiently screen out less important elements.

The review comment to be evaluated is:

This paper proposes a stochastic subset selection method for reducing the storage / transmission cost of datasets. The proposes method minimizes the expected loss over selected datasets. The data selection algorithm consists a candidate selection stage and an autoregressive selection stage, parameterized with neural networks, and are trainable by gradient methods. The authors formulate and tested their approach on four tasks. The problem formulation and methodology are technically sound. The proposed method also seems to be more general than competing methods, such as coreset.

However, I think the experimental results do not support the paper well.
1. In Table 1 and Table 2, SSS doesn't seem to have big advantage over RS. From the selected images (e.g., Fig. 10 and 11), I also don't gain an intuition on what does SSS actually select, and how do the selected instances differ from random subsampling.
2. The paper claims to address the dataset redundancy. From to my understanding, that is, removing similar instances. However, this claim is not explicitly supported by the experiments.

The evidence to be evaluated is:We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\).

Specifically, to sample \(z_{i}\), we execute the following computational routine: \[z_{i}=\sigma\Big{(}\frac{1}{\tau}\Big{(}\log\frac{\pi_{i}}{1-\pi_{i}}+\log \frac{u}{1-u}\Big{)}\Big{)},\quad\pi_{i}=\rho(d_{i},r(D)),\quad u\sim\text{ Unif}(0,1), \tag{2}\] where \(\sigma\) is the Sigmoid function, \(\tau\) is the temperature for the continuous relaxation and \(u\) is sampled from the uniform distribution. \(\tau\) is set to \(0.05\) in all our experiments.

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

4:\((\pi_{1},\pi_{2},\dots,\pi_{n})\leftarrow(\rho(d_{1},r(D)),\dots,\rho(d_{n},r(D)))\) 5:\(z_{i}\sim\text{Ber}(\pi_{i})\) for \(i=1,\dots,n\). 6:\(D_{c}\leftarrow\{d_{i}\text{ for }i=1:n\text{ if }z_{i}\}\)\(\triangleright\) Candidate Selection

13:\((p_{1},p_{2},\dots,p_{m})\leftarrow(f(w_{1},D_{c},D_{s}),f(w_{2},D_{c},D_{s}), \dots,f(w_{m},D_{c},D_{s}))\) 14:\((p_{1},p_{2},\dots,p_{m})\leftarrow(p_{1},p_{2},\dots,p_{m})/\sum_{j=1}^{m}p_{j}\) 15:\(Q\leftarrow\text{Select }q\) elements from \(C\) with probability \((p_{1},p_{2},\dots,p_{m})\)

For computational efficiency, we may desire to restrict the size of \(D_{c}\) to save computational cost when constructing \(D_{s}\).

\[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]+\beta\text{KL}[p(Z|D) ||r(Z)]] \tag{4}\]where \(r(Z)\) is a sparse prior. In our experiments, we set the parameter of the Bernoulli spares prior \(r(Z)\) to either \(0.1\) or \(0.01\) for different levels of sparsity and \(\beta\) is set to \(1.0\) or \(0.001\).

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

An ANP is then used to reconstruct the remaining pixels from a set of context elements (selected subset in our case).

Table 1 shows that using only 500 pixels (\(\sim\)1.3% of total pixels in an image), we can achieve a mean AUC of 0.9093 (99.3% of the accuracy obtained with the full image).

The AUC with selected pixels learned from our SSS is significantly higher than that of the random pixels baseline, showing the effectiveness of our subset selection method.

Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account. Hence, it can select a compact set that avoids samples with redundant information.

The idea of the greedy training algorithm is to train the auto-regressive model to select the best next \(q\) elements from the candidate set to minimize the target loss on the selected samples. By doing this, we do not have to run the auto-regressive model \(k/q\) time during training, thus reducing the computational cost.

12:\(I^{(j)}\sim\) random \(i\)-element subset of \(D^{(j)}_{\theta}\) for \(j=1\ldots m\) 13:\(Q^{(j)}\sim\) select a \(q\)-element subset from \(D^{(j)}_{\theta}\setminus I^{(j)}\) (with the auto-regressive model) 14:\(\theta\leftarrow\theta-\alpha\nabla_{\theta}\frac{1}{m}\sum_{j=1}^{m}\ell( \cdot,I^{(j)}\cup Q^{(j)}),\phi\leftarrow\phi-\alpha\nabla_{\phi}\frac{1}{m} \sum_{j=1}^{m}\ell(\cdot,I^{(j)}\cup Q^{(j)})\)

Input: Figure 10: Visualization of a set with 200 images for instance selection. The two stage selection method in SSS is visualized as Candidate Set and SSS. A coreset of size 5 is visualized.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input: Figure 9: Selected pixels for different tasks on CelebA.

Input: Figure 6: CelebA reconstruction samples with varying number of pixels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "First we learn a Bernoulli sampling rate for individual sample to efficiently screen out less important elements."

The evidence shows that the method does use a Bernoulli sampling process in the first stage to select candidate elements, as demonstrated by equations (1) and (2) and Figure 2's architecture diagram. However, the claim is somewhat understated and could be more precise, as it doesn't fully capture that:

1. The Bernoulli sampling rates are conditioned on both individual elements and the set representation
2. The process involves a two-stage selection (Bernoulli followed by Categorical)
3. The sampling includes a temperature parameter and continuous relaxation

The claim is actually quite conservative in describing just one aspect of the selection process, making it well-stated rather than overstated.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

First we learn a Bernoulli sampling rate for individual sample to efficiently screen out less important elements.

The review comment to be evaluated is:

### Summary

The authors present a stochastic algorithm for selecting a subset of a large dataset, while trying to preserve statistics from the original dataset. The algorithm is a two-step process, first selecting a set of "candidates" based on individual features, then filtering to a final subset which may account for interactions between candidate items.

### Strengths

The paper provides empirical experimental results which suggest the superiority of the proposed method against two baselines: Learning to Sample (LTS) and random selection (RS).

Very few grammatical mistakes.

### Concerns and Issues that Need Clarification

1. A significant amount of active learning literature deals with subset selection. However, this paper fails to discuss or even cite any results from the active learning literature. It would be helpful for the authors to compare/contrast their proposed method with existing active learning results. For example, consider
   - [*Active Learning for Convolutional Neural Networks: A Core-Set Approach* by Sener and Savarese (2018)](https://openreview.net/forum?id=H1aIuk-RW)
   - [*Selection via Proxy: Efficient Data Selection for Deep Learning* by Coleman et al. (2019)](https://openreview.net/forum?id=HJg2b0VYDr)
   - [*Submodularity in Data Subset Selection and Active Learning* by Wei et al. (2015)](http://proceedings.mlr.press/v37/wei15.html)
   - [*On Statistical Bias In Active Learning: How and When to Fix It* (2020)](https://openreview.net/forum?id=JiYq3eqTKY) - this paper is a submission to ICLR 2020, so obviously there is no expectation that the authors would be familiar with this work. However, I bring this paper up because it discusses a similar topic of trying to select data points from a larger dataset that provide a reasonable estimate of a loss function on the "population" vs. a "sample."

2. The algorithm, as described, states that individual elements of a dataset $D$ are "possibly represented" as a pair $(x_i, y_i)$. What does "possibly represented" mean? Assuming that the dataset elements are given as a $(x_i, y_i)$ pair, the stochastic subset selection algorithm seems to describe instance selection. However, the paper also claims that the algorithm works for feature selection. What are $D$, $x_i$, and $y_i$ in the feature selection tasks? The paper makes an attempt at clarifying this by stating, "Here, x is 2-dimensional and y is 3-dimensional for RGB images." But this still leaves the reader clueless - why is $x$ only 2-dimensional? What are those 2 dimensions?

3. The end of the introduction claims that the proposed subset selection method "learns to sample a subset from a larger set with linear time complexity," yet Section 3.4 claims that the inference complexity is $O(n) + O(k^2 d/q)$, which is nonlinear if either $k$ or $d$ is chosen to be dependent on $n$. Could the authors please clarify?

4. The authors claim that their method uses meta-learning, but no mention of meta learning is made beyond the "Preliminaries" subsection of the "Approach" section. How is this method a example of meta-learning?

5. The paper provides experimental evidence that the proposed method outperforms other subset selection baselines, such as Learning to Select (LTS) and random selection (RS). However, the paper provides little (if any) intuition or formal justification for why the proposed method performs better. The authors need to explain the motivation behind the proposed algorithm.

6. The writing and figures lack clarity and are often difficult to follow. Some examples:
   - The context for the mathematical expression in (4) is very unclear. Is (4) an objective function? If so, is this expression supposed to be minimized or maximized? And what is the random variable $Z$ supposed to represent? And why is there a period in the middle of the expression?
   - Figure 2 is poorly explained and difficult to understand. I'm assuming that the blue vertical vector is supposed to represent $D_c$. Is that correct? And are the dark shaded boxes supposed to represent the "candidates" and the lightly shaded boxes supposed to represent the items that were not selected? As a reader, I should not have to guess what the colors mean. They should be explained clearly in the caption, or at the very least, in the paper text. Furthermore, it seems to me (again it's never made explicit) that Figure 2b is depicting the Autoregressive Subset Selection ("AutoSelect") routine. In this case, why is there a white row vector of $D = d_1, \dotsc, d_n$? In Algorithm 1, it seems as though AutoSelect only takes as input $D_s$ and $D_c$, which are subsets of $D$ and not $D$ itself.

### Original Rating and Confidence

**Rating** - 4: Ok but not good enough - rejection

**Confidence** - 4: The reviewer is confident but not absolutely certain that the evaluation is correct

### Updated Review after Author Response

Thank you for the thoughtful responses to my questions, and apologies for the delay in updating my review. In general the responses have addressed my questions, and I have updated my rating accordingly. However, there are still several issues in my mind:

1. **Connection to active learning** - The authors correctly point out that active learning does not assume access to labels, whereas the proposed method does. However, I believe the comparison against selection algorithms such as k-center greedy should be included in the paper for completeness sake, and not just left as a comment in this forum.

2. **Dataset Distillation: Instance Selection** - Upon re-reading the paper carefully, I found this subsection in Section 3.6 to be confusing. Why are there multiple $D_i$? The section seems to only talk about one $D_i$.

3. **Set vs. Tensor**: In the new "representation learning" subsection of Section 2, the paper stresses that the model learns a representation of a set, instead of an individual data point. However, many tasks (e.g., image reconstruction) do impose some sort of ordering. The paper lacks clarity about where permutation-invariance is used vs. not used.

4. **Other places that need clarification**:
  - Equation (2): presumably you threshold the output of the sigmoid function, right? Because in Algorithm 1, it seems like $z_i$ are supposed to be boolean-valued.
  - Appendix: there are several missing closing parentheses

The evidence to be evaluated is:We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\).

Specifically, to sample \(z_{i}\), we execute the following computational routine: \[z_{i}=\sigma\Big{(}\frac{1}{\tau}\Big{(}\log\frac{\pi_{i}}{1-\pi_{i}}+\log \frac{u}{1-u}\Big{)}\Big{)},\quad\pi_{i}=\rho(d_{i},r(D)),\quad u\sim\text{ Unif}(0,1), \tag{2}\] where \(\sigma\) is the Sigmoid function, \(\tau\) is the temperature for the continuous relaxation and \(u\) is sampled from the uniform distribution. \(\tau\) is set to \(0.05\) in all our experiments.

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

4:\((\pi_{1},\pi_{2},\dots,\pi_{n})\leftarrow(\rho(d_{1},r(D)),\dots,\rho(d_{n},r(D)))\) 5:\(z_{i}\sim\text{Ber}(\pi_{i})\) for \(i=1,\dots,n\). 6:\(D_{c}\leftarrow\{d_{i}\text{ for }i=1:n\text{ if }z_{i}\}\)\(\triangleright\) Candidate Selection

13:\((p_{1},p_{2},\dots,p_{m})\leftarrow(f(w_{1},D_{c},D_{s}),f(w_{2},D_{c},D_{s}), \dots,f(w_{m},D_{c},D_{s}))\) 14:\((p_{1},p_{2},\dots,p_{m})\leftarrow(p_{1},p_{2},\dots,p_{m})/\sum_{j=1}^{m}p_{j}\) 15:\(Q\leftarrow\text{Select }q\) elements from \(C\) with probability \((p_{1},p_{2},\dots,p_{m})\)

For computational efficiency, we may desire to restrict the size of \(D_{c}\) to save computational cost when constructing \(D_{s}\).

\[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]+\beta\text{KL}[p(Z|D) ||r(Z)]] \tag{4}\]where \(r(Z)\) is a sparse prior. In our experiments, we set the parameter of the Bernoulli spares prior \(r(Z)\) to either \(0.1\) or \(0.01\) for different levels of sparsity and \(\beta\) is set to \(1.0\) or \(0.001\).

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

An ANP is then used to reconstruct the remaining pixels from a set of context elements (selected subset in our case).

Table 1 shows that using only 500 pixels (\(\sim\)1.3% of total pixels in an image), we can achieve a mean AUC of 0.9093 (99.3% of the accuracy obtained with the full image).

The AUC with selected pixels learned from our SSS is significantly higher than that of the random pixels baseline, showing the effectiveness of our subset selection method.

Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account. Hence, it can select a compact set that avoids samples with redundant information.

The idea of the greedy training algorithm is to train the auto-regressive model to select the best next \(q\) elements from the candidate set to minimize the target loss on the selected samples. By doing this, we do not have to run the auto-regressive model \(k/q\) time during training, thus reducing the computational cost.

12:\(I^{(j)}\sim\) random \(i\)-element subset of \(D^{(j)}_{\theta}\) for \(j=1\ldots m\) 13:\(Q^{(j)}\sim\) select a \(q\)-element subset from \(D^{(j)}_{\theta}\setminus I^{(j)}\) (with the auto-regressive model) 14:\(\theta\leftarrow\theta-\alpha\nabla_{\theta}\frac{1}{m}\sum_{j=1}^{m}\ell( \cdot,I^{(j)}\cup Q^{(j)}),\phi\leftarrow\phi-\alpha\nabla_{\phi}\frac{1}{m} \sum_{j=1}^{m}\ell(\cdot,I^{(j)}\cup Q^{(j)})\)

Input: Figure 10: Visualization of a set with 200 images for instance selection. The two stage selection method in SSS is visualized as Candidate Set and SSS. A coreset of size 5 is visualized.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input: Figure 9: Selected pixels for different tasks on CelebA.

Input: Figure 6: CelebA reconstruction samples with varying number of pixels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim "First we learn a Bernoulli sampling rate for individual sample to efficiently screen out less important elements."

The evidence strongly supports this claim as a well-stated description of the first stage of the method. Specifically:

1. The evidence shows a clear Bernoulli sampling process through equations (1) and (2), where z_i is sampled from a Bernoulli distribution with logits conditioned on individual elements and the set representation.

2. Figure 2(a) and Algorithm 1 lines 4-6 demonstrate this initial candidate selection stage using Bernoulli sampling.

3. The purpose of efficient screening is supported by the evidence stating "For computational efficiency, we may desire to restrict the size of D_c to save computational cost."

4. The figures show how this initial Bernoulli sampling effectively reduces the full set to a smaller candidate set.

The claim accurately describes this first stage without exaggeration or overstatement.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Then, to select the most important elements out of this candidate set considering relative importance, we use a Categorical distribution to model the conditional distribution of sampling each element given a set of selected elements.

The evidence to be evaluated is:Our instance-sampling model is also related to the Variational Auto Encoder (VAE) (Kingma and Welling, 2013). However, while VAE learns a compact representation of a _data point_, our model learns a compact representation of a _set_.

Chen et al. (2018) learns instancewise feature selection with the goal of model interpretation by extracting subset of features most informative for a given sample. Our method also falls in this category.

In what follows, we present a method that learns the conditional distribution \(p(D_{s}|D)\) of the subset \(D_{s}\) via a two stage selection procedure dubbed _candidate selection_ and _autoregressive subset selection_. The overall objective is then to minimize the loss function with respect to the subset \(D_{s}\), \(\mathbb{E}_{p(D,|D)}[\ell(.,D_{s})]\). When the set \(D\) itself follows a distribution of sets as in the _meta-learning_ framework, then the objective becomes \(\mathbb{E}_{D}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]]\).

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

13:\((p_{1},p_{2},\dots,p_{m})\leftarrow(f(w_{1},D_{c},D_{s}),f(w_{2},D_{c},D_{s}), \dots,f(w_{m},D_{c},D_{s}))\) 14:\((p_{1},p_{2},\dots,p_{m})\leftarrow(p_{1},p_{2},\dots,p_{m})/\sum_{j=1}^{m}p_{j}\) 15:\(Q\leftarrow\text{Select }q\) elements from \(C\) with probability \((p_{1},p_{2},\dots,p_{m})\)

Here, we adopt Prototypical Networks (Snell et al., 2017) and apply the subset selection model to the task of selecting representative prototypes from each class to be used for classifying new instances. By learning to select the prototypes, we can remove outliers that would otherwise change the class prediction boundaries in the classification task.

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

An ANP is then used to reconstruct the remaining pixels from a set of context elements (selected subset in our case).

On this task, our subset selection module is trained via the procedure detailed in Section 3.6 on instance selection.

Additionally, we compare our method with the Farthest Point Sampling(FPS) algorithm which selects \(k\) points from a given set by computing distances on a metric space between all elements and selecting those elements that are furthest from each other.

Also for small sample selection, our method outperforms FPS and Random Sampling significantly since our method is able to model the interactions within the full dataset and hence can select the most representative subset.

Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set.

The idea of the greedy training algorithm is to train the auto-regressive model to select the best next \(q\) elements from the candidate set to minimize the target loss on the selected samples. By doing this, we do not have to run the auto-regressive model \(k/q\) time during training, thus reducing the computational cost.

12:\(I^{(j)}\sim\) random \(i\)-element subset of \(D^{(j)}_{\theta}\) for \(j=1\ldots m\) 13:\(Q^{(j)}\sim\) select a \(q\)-element subset from \(D^{(j)}_{\theta}\setminus I^{(j)}\) (with the auto-regressive model) 14:\(\theta\leftarrow\theta-\alpha\nabla_{\theta}\frac{1}{m}\sum_{j=1}^{m}\ell( \cdot,I^{(j)}\cup Q^{(j)}),\phi\leftarrow\phi-\alpha\nabla_{\phi}\frac{1}{m} \sum_{j=1}^{m}\ell(\cdot,I^{(j)}\cup Q^{(j)})\)

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim to be relatively well-stated but slightly imprecise. The evidence shows that the method indeed uses a Categorical distribution for selecting elements from the candidate set, as demonstrated in Figure 2(b) and explained in the text through equation (3). The iterative nature of the selection process is clearly shown with multiple Categorical distributions stacked for sequential element selection. However, the original claim oversimplifies by not mentioning the two-stage process that includes initial Bernoulli candidate selection before the Categorical distribution is applied.

<score>2</score>

Justification:
1. The claim is mostly accurate in describing the use of Categorical distribution for element selection.
2. The evidence supports this through equation (3) and Figure 2(b) showing the stacked Categorical distributions.
3. However, the claim omits the crucial first stage of Bernoulli candidate selection shown in Figure 2(a).
4. The text evidence explicitly states "Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset."
5. While not severely overstated, the claim could be more precise by acknowledging the complete two-stage selection process.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Then, to select the most important elements out of this candidate set considering relative importance, we use a Categorical distribution to model the conditional distribution of sampling each element given a set of selected elements.

The review comment to be evaluated is:

**Final recommendation**
I support accepting this paper. While I am a bit disappointed the authors did not add in the paper the results they discussed during the rebuttal, I think the paper is interesting and clearer than at submission.

**Summary**
stochastic subset selection (SSS) is a method to learn to compress a set $D$ by selecting a subset $D_s$ such that the loss of a task performed on $D_s$ is as close as possible to the loss if the task had been performed on the original $D$. The paper show how this general method can be applied to several tasks. For example, $D$ can contain the pixels of an image and the task can be to reconstruct the original image from the subset. This corresponds to learning to compress. As another example, with the same $D$ but a label prediction task, SSS learns to dynamically select sparse features for a classification task. The other tasks considered are dataset distillation (compressing a data set) and selecting prototypes for few-shot classification. $D_s$ is constructed in two steps. A candidate set is first constructed by considering elements of $D$ independently and in a second step elements of $D_c$ are included in $D_s$ iteratively by considering other elements in $D_c$ and in $D_s$. Experiments are performed for one example of each task previously discussed and SSS is shown to outperform baselines. SSS is motivated by the need to reduce bandwidth, computation and memory footprint of deep learning in edge devices.

---
**Strong points**

1) Code is provided (I did not try to run it).

2) The proposed approach is applied to four different tasks. This convinced me of its applicability.

3) Empirical experiments seem well conducted (with some caveat, see weak point 3) and results are good.

4) The problems tackled are interesting.

**Weak points**

1) The paper is very dense, which makes it hard to follow. Related to that, I think that some parts of the paper are too concise to understand them well. The supplementary material partly compensates that.

2) While the experiments show that the proposed approach works well, I think that some experiments should compare full data pipelines. As an example, one experiment shows that using SSS to dynamically select pixels of an image and using this sparse representation as input leads to good classification accuracy. The paper implicitely suggests that this reduces inference cost if classification is performed on the device and bandwidth if inference is performed on another device. However, the cost of the pixel selection by SSS is never analyzed. I think comparing the memory footprint, the number of operations, bandwidth used and accuracy for all steps of the following pipelines would give a much clearer picture of the interest of the approach for deep learning on mobile devices:
    - classification is done directly on the device, using an optimized network.
    - pixels are randomly selected and classification is performed on the device.
    - pixels are randomly selected, transmitted and classification is performed on another device.
    - pixels are selected by SSS, which involves computing multiple embeddings, and classification is performed on the device.
    - pixels are selected by SSS, transmitted and classification is performed on another device.

---
**Recommendation**
Overall, I vote for accepting. I am on the fence with this paper. The current results are good and I think the proposed approach has some potential and will be interesting for the community. On the other hand the paper is hard to read and the experiments, while good to show the strength of the approach, do not cover usefullness for the mobile use case. I think the former can be adressed in time and that the latter, while important, does not prevent the paper to be published.

---
**Details**

I like that each task is first formally described and then evaluated in a specific case with the experiments. However I only understood most tasks when reading the experiment sections. I would suggest to either provide a short illustration of an actual instance of the task when describing the task formally, to directly provide the experimental result after describing the task or to direct the reader to the relevant experimental section(s) for illustration.

The paper states that model descriptions are available in appendix A, but I could not find them.

For the *Set Classification/Prediction* task, I would suggest clarifying that for this task a set typically contains the features of a single example, so $D=x$. I only understood this when reading the experimental section.

The *Dataset Distillation: Instance Selection* task was not clear to me. Here are a few points that puzzled me.
- The text suggest the proposed approach is applied to each $D_i$, but figure 3(c) has no $D_i$ and $i$ seems to be used as an index of $x$ rather than $D$. Furthermore, in figure 3(c) a single D_s is constructed whereas I was expecting one per $D_i$.
- The paper states that *It is important to note that $c$ is computed using only $D_i$ for every element in $D$*. However $c$ also depends on *a query $\alpha$ which is computed using $d_i$*, which, if I understood correctly, is the element of $D$ and can be outside $D_i$. So it seems that $c$ is not computed using only $D_i$.

For the *Dataset Distillation: Classification* task, in figure 3d, the labels are not observed. How can the model learn?

1) For this task, if the label is observed, how is this different from the first task?

2) For the Image Reconstruction experiment, why is $x$ 2 dimensional? Are the inputs black and white images?

I think that the image reconstruction experiment should include a comparison to compression algorithms. I agree with the paper that the proposed approach is applicable to many other types of data than images, but it would be very interesting to see how the approach compares to expert defined methods. Furthermore, implementation in relation to mobile devices is the motivation of this work and as far as I know classical compression algorithms are still used on mobile devices.

---
**Questions**

Questions 1 and 2 in *details* above.

I would also be interested to hear authors' opinions on weak point 2 and a comparison to compression.

For classification, pixels not selected are set to 0. Wouldn't the inference cost be the same when using convolutional networks?

---
**Minor details**

For the *set reconstruction* task, would it also be interesting to learn a model on $D_s$ and use it to predict $Y$ from $X$?

*typos*
foro

The evidence to be evaluated is:Our instance-sampling model is also related to the Variational Auto Encoder (VAE) (Kingma and Welling, 2013). However, while VAE learns a compact representation of a _data point_, our model learns a compact representation of a _set_.

Chen et al. (2018) learns instancewise feature selection with the goal of model interpretation by extracting subset of features most informative for a given sample. Our method also falls in this category.

In what follows, we present a method that learns the conditional distribution \(p(D_{s}|D)\) of the subset \(D_{s}\) via a two stage selection procedure dubbed _candidate selection_ and _autoregressive subset selection_. The overall objective is then to minimize the loss function with respect to the subset \(D_{s}\), \(\mathbb{E}_{p(D,|D)}[\ell(.,D_{s})]\). When the set \(D\) itself follows a distribution of sets as in the _meta-learning_ framework, then the objective becomes \(\mathbb{E}_{D}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]]\).

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

13:\((p_{1},p_{2},\dots,p_{m})\leftarrow(f(w_{1},D_{c},D_{s}),f(w_{2},D_{c},D_{s}), \dots,f(w_{m},D_{c},D_{s}))\) 14:\((p_{1},p_{2},\dots,p_{m})\leftarrow(p_{1},p_{2},\dots,p_{m})/\sum_{j=1}^{m}p_{j}\) 15:\(Q\leftarrow\text{Select }q\) elements from \(C\) with probability \((p_{1},p_{2},\dots,p_{m})\)

Here, we adopt Prototypical Networks (Snell et al., 2017) and apply the subset selection model to the task of selecting representative prototypes from each class to be used for classifying new instances. By learning to select the prototypes, we can remove outliers that would otherwise change the class prediction boundaries in the classification task.

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

An ANP is then used to reconstruct the remaining pixels from a set of context elements (selected subset in our case).

On this task, our subset selection module is trained via the procedure detailed in Section 3.6 on instance selection.

Additionally, we compare our method with the Farthest Point Sampling(FPS) algorithm which selects \(k\) points from a given set by computing distances on a metric space between all elements and selecting those elements that are furthest from each other.

Also for small sample selection, our method outperforms FPS and Random Sampling significantly since our method is able to model the interactions within the full dataset and hence can select the most representative subset.

Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set.

The idea of the greedy training algorithm is to train the auto-regressive model to select the best next \(q\) elements from the candidate set to minimize the target loss on the selected samples. By doing this, we do not have to run the auto-regressive model \(k/q\) time during training, thus reducing the computational cost.

12:\(I^{(j)}\sim\) random \(i\)-element subset of \(D^{(j)}_{\theta}\) for \(j=1\ldots m\) 13:\(Q^{(j)}\sim\) select a \(q\)-element subset from \(D^{(j)}_{\theta}\setminus I^{(j)}\) (with the auto-regressive model) 14:\(\theta\leftarrow\theta-\alpha\nabla_{\theta}\frac{1}{m}\sum_{j=1}^{m}\ell( \cdot,I^{(j)}\cup Q^{(j)}),\phi\leftarrow\phi-\alpha\nabla_{\phi}\frac{1}{m} \sum_{j=1}^{m}\ell(\cdot,I^{(j)}\cup Q^{(j)})\)

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "Then, to select the most important elements out of this candidate set considering relative importance, we use a Categorical distribution to model the conditional distribution of sampling each element given a set of selected elements."

1. Summary of evaluation:
The evidence strongly supports this claim. Figure 2(b) explicitly shows the iterative core subset selection using Categorical distributions. The technical details in the text explain how elements are selected using categorical distributions with probabilities computed based on relative importance (equation 3). The review comments don't dispute this aspect. The claim accurately reflects the methodology without exaggeration and is well-grounded in the provided implementation details.

<score>1</score>

Justification:
1. The claim is technically precise and matches the mathematical formulation shown in equation 3
2. The visual evidence in Figure 2(b) directly illustrates the described process
3. The description aligns with the actual implementation details provided in the algorithms
4. No overstatement of capabilities or results
5. The language used is appropriately measured and technical
6. The reviewer comments don't raise any concerns about this specific claim

The claim is well-stated as it accurately describes the technical approach without embellishment or exaggeration.